Diving into Data Cheat Sheet by djjang2

Big Data: refers to the large, diverse sets of inform­ation that grow at ever-i­ncr­easing rates. It encomp­asses the volume of inform­ation, the velocity or speed at which it is created and collected, and the variety or scope of the data points being covered (known as the "­three v's" of big data). Big data often comes from data mining and arrives in multiple formats.

Big data can be structured (often numeric, easily formatted and stored) or unstru­ctured (more free-form, less quanti­fia­ble).

Big data can be collected from publicly shared comments on social networks and websites, volunt­arily gathered from personal electr­onics and apps, through questi­onn­aires, product purchases, and electronic check-ins.

Data analysts look at the relati­onship between different types of data, such as demogr­aphic data and purchase history, to determine whether a correl­ation exists. Such assess­ments may be done in-house or externally by a third-­party that focuses on processing big data into digestible formats. Businesses often use the assessment of big data by such experts to turn it into actionable inform­ation.

Economic Data: Data regarding Interest rates, Asset prices, Exchange rates, and the Consumer Price Index; and other info about the global, national, or regional economy.

Unstru­ctured Data: Data that is not organized into predet­ermined formats, such as databases, and often consists of text, images, or other nontra­dit­ional media.

External Data: Facts and figures available in locations outside a company. Refers to published data from outside the business.

EDA is different from initial data analysis (IDA), which focuses more narrowly on checking assump­tions required for model fitting and hypothesis testing, and handling missing values and making transf­orm­ations of variables as needed. EDA encomp­asses IDA.
EDA Techniques incl: Scatter Plot & Bubble Plot

Text Mining: Obtains info through language recogn­ition; more difficult than w/ other models b/c there are no organized fields & no numerical values.

Data loss occurs when valuable or sensitive inform­ation on a computer is compro­mised due to theft, human error, viruses, malware, or power failure. It may also occur due to physical damage or mechanical failure or equipment of an edifice.

Data Mining: is a process used by companies to turn raw data into useful inform­ation. By using software to look for patterns in large batches of data, businesses can learn more about their customers to develop more effective marketing strate­gies, increase sales and decrease costs. Data mining depends on effective data collec­tion, wareho­using, and computer processing.
Data mining programs break down patterns and connec­tions in data based on what inform­ation users request or provide.

Disruptive Innovation: Disruptive Innovation refers to a technology whose applic­ation signif­icantly affects the way a market or industry functions. An example of modern disruptive innovation is the Internet, which signif­icantly altered the way companies did business and which negatively impacted companies that were unwilling to adapt to it.
Disruptive innovation refers to a new develo­pment that dramat­ically changes the way a structure or industry functions.

Electronic Commerce (e-com­merce): Electronic commerce or e-commerce (sometimes written as eCommerce) is a business model that lets firms and indivi­duals buy and sell things over the internet. E-commerce operates in all four of the following major market segments: *Business to business; * Business to consumer; *Consumer to consumer; and *Consumer to business

Scatter Plot: A graphed cluster of dots, each of which represents the values of two variables. The slope of the points suggests the direction of the relati­onship between the two variables. The amount of scatter suggests the strength of the correl­ation.
two dimens­ional plot of point values

Correl­ation Matrix: A table that summarizes a series of correl­ations among several variables.
rectan­gular display of all the correl­ations between all pairs of data sets with a key (such as color coding) that indicates the strength of the correl­ation

Regression Analysis: A set of statis­tical processes for estimating the relati­onships between a dependent variable (often called the 'outco­me/­target variable') and one or more indepe­ndent variables (often called 'predi­ctors', 'covar­iates', or 'featu­res').
Primarily used for (2) concep­tually distinct purposes.
First, widely used for prediction and foreca­sting, where its use has substa­ntial overlap with the field of machine learning; Second, in some situat­ions, can be used to infer causal relati­onships between the indepe­ndent and dependent variables.

Linear Regression: Falls into 1 of 2 catego­ries:
If the goal is predic­tion, foreca­sting, or error reduction, linear regression can be used to fit a predictive model to an observed data set of values of the response and explan­atory variables. After developing such a model, if additional values of the explan­atory variables are collected without an accomp­anying response value, the fitted model can be used to make a prediction of the response.
If the goal is to explain variation in the response variable that can be attributed to variation in the explan­atory variables, linear regression analysis can be applied to quantify the strength of the relati­onship between the response and the explan­atory variables, and in particular to determine whether some explan­atory variables may have no linear relati­onship with the response at all, or to identify which subsets of explan­atory variables may contain redundant inform­ation about the response.

GLM consists of (3) elements:
1. An expone­ntial family of probab­ility distri­but­ions.
2. A linear predictor - the quantity which incorp­orates the inform­ation about the indepe­ndent variables into the model.
3. A link function - provides the relati­onship between the linear predictor and the mean of the distri­bution function.

Data-D­riven Decision Making: gives reference to the collection and analysis of data to guide decisions that improve success.
Data-I­nformed Decision Making (DIDM): (2) basic approa­che­s:D­esc­riptive & Predictive approach.

Data Wareho­using: the electronic storage of a large amount of inform­ation by a business or organi­zation. Data wareho­using is a vital component of business intell­igence that employs analytical techniques on business data.
A data warehouse is designed to run query and analysis on historical data derived from transa­ctional sources for business intell­igence and data mining purposes.

Data Analytics Process: involves several different steps:
1. The first step is to determine the data requir­ements or how the data is grouped. Data may be separated by age, demogr­aphic, income, or gender. Data values may be numerical or be divided by category.
2. The second step in data analytics is the process of collecting it. This can be done through a variety of sources such as computers, online sources, cameras, enviro­nmental sources, or through personnel.
3. Once the data is collected, it must be organized so it can be analyzed. Organi­zation may take place on a spread­sheet or other form of software that can take statis­tical data.
4. The data is then cleaned up before analysis. This means it is scrubbed and checked to ensure there is no duplic­ation or error, and that it is not incomp­lete. This step helps correct any errors before it goes on to a data analyst to be analyzed.

Neural Network: Disadv­antages: The processes for developing the rules and logic may not be transp­arent.
a neural network can be overtr­ained if it reviews data in such detail that it can not then operate in a larger framework with other types of data

Social Network Analysis (Network analysis): Studies the connec­tions and relati­onships among people in a social network.
Useful tool for making predic­tions based on trends
Social Network - group of indivi­duals or entities who share relati­onships and the flow of commun­ication

Centrality Measures: In a social network context, the quanti­fic­ation of a node's relati­onship to other nodes in the same network.
Determines the efficiency of the flow btwn Social Network commec­tions.
indicators of centrality identify the most important vertices within a graph.

Artificial Intell­igence (AI): Artificial intell­igence (AI) refers to the simulation of human intell­igence in machines that are programmed to think like humans and mimic their actions. The term may also be applied to any machine that exhibits traits associated with a human mind such as learning and proble­m-s­olving.

Machine Learning: Machine learning is the concept that a computer program can learn and adapt to new data without human interv­ention. Machine learning is a field of artificial intell­igence (AI) that keeps a computer’s built-in algorithms current regardless of changes in the worldwide economy.
Machine learning is useful in parsing the immense amount of inform­ation that is consis­tently and readily available in the world to assist in decision making.

Disruptive Technology: Disruptive technology is an innovation that signif­icantly alters the way that consumers, indust­ries, or businesses operate. A disruptive technology sweeps away the systems or habits it replaces because it has attributes that are recogn­izably superior. Recent disruptive technology examples include e-comm­erce, online news sites, ride-s­haring apps, and GPS systems.
A disruptive technology supersedes an older process, product, or habit.

Process of Associ­ation Rule Generation is usually split up into two separate steps:
1. A minimum support threshold is applied to find all frequent itemsets in a database.
2. A minimum confidence constraint is applied to these frequent itemsets in order to form rules.

Decision Tree Analysis: (5) Steps:
1. Define the problem with a statement of the decision being considered
2. Create pathways (sequence of events) for each altern­ative, with each pathway leading to an outcome
3. Assign a probab­ility to each event on a pathway and estimate the value (cost or gain) of the outcome of each pathway
4. Multiply the probab­ility of each event by the value of its outcome to determine the expected value of each pathway
5. Compare expected values to determine the pathway with the highest expected value

Analysis: Analyzes the conseq­uences, costs, and gains of decisions to compare altern­ative decisions.

Features: The process can be used to analyze both negative and positive conseq­uences.

Outputs: Decision tree analysis produces an analysis of risk for each pathway with options and an expected value for each pathway.

Disadv­antages: Can be compli­cated and difficult to explain
also suscep­tible to oversi­mpl­ifi­cation, which can result in less accurate decision making

1. Identify the initiating event (first accidental event that could result in unwanted conseq­uences)
2. Determine conseq­uences of events that could follow the accidental event
3. Construct an event tree diagram that lists barriers in the sequence that would be activated if the designated event occurred
4. Design each pathway to fork at each barrier depending on whether the barrier succeeds or fails
5. Assign an estimated probab­ility to the likelihood of success of failure of each barrier 6. Calculate the frequency of outcomes for each pathway

Event Tree Analysis: Analyzes the conseq­uences of accidental events rather than decisions.

Features - Process typically analyzes only negative conseq­uences.

- List of potential problems, with estimated values for outcomes and freque­ncies
- Recomm­end­ations regarding the effect­iveness of barriers

Advantages:
- offers a visual portrayal of sequences of events following an accident
- shows the effect­iveness of control systems
- provides both quanti­tative and qualit­ative inform­ation

Economic Data: Data regarding interest rates, asset prices, exchange rates, the Consumer Price Index, and other inform­ation about the global, the national, or a regional economy.

Cluster Analysis: A model that determines previously unknow groupings of data.

Centrality Measures: In a social network context, the quanti­fic­ation of a node's relati­onship to other nodes in the same network.

Data Science: An interd­isc­ipl­inary field involving the design and the use of techniques to process very large amounts of data from a variety of courses to provide knowledge based on data.