Data science and Big Data: definitions and common myths

II. Data Misconceptions

Data are quickly becoming a new form of capital, a different coin, and an innovative source of value. It is extremely important to learn how to channel the power of big data into an efficient strategy to manage and grow a business. A well-set data strategy is becoming fundamental to every business, regardless the actual size of the datasets used. However, in order to establish a data framework that works, there are a few misconceptions that need to be clarified:

i) More data means higher accuracy. Not all data are good quality data, and tainting a dataset with dirty data could compromise the final products. It is similar to a blood transfusion: if a non-compatible blood type is used, the outcome can be catastrophic for the whole body.

Secondly, there is always the risk of overfitting data to the model, yet not derive any further insight — “if you torture the data enough, nature will always confess” (Coase, 2012). In all applications of big data, you want to avoid striving for perfection: too many variables increase the complexity of the model without necessarily increasing accuracy or efficiency. More data always implies higher costs and not necessarily higher accuracy. Costs include higher maintenance costs — both for the physical storage and for model retention; greater difficulties in calling the shots and interpreting the results; more burdensome data collection and time opportunity costs.

Undoubtedly the data used do not have to be conventional or used in a standard way — and this is where the real gain is locked in — and they may challenge the general wisdom, but they have to be proven and validated. In summary, smart data strategies always start from analyzing internal datasets, before integrating them with public or external sources.

Do not store and process data just for the sake of having them, because, with the amount of data being generated daily, the noise increases faster than the signal (Silver, 2013). Pareto’s 80/20 rule applies: the 80% of the phenomenon could be probably explained by the 20% of the data owned.

ii) If you want to do big data, you have to start big. A good practice before investing heavily in technology and infrastructures for big data is to start with few high-value problems that validate whether big data may be of any value to your organization. Once the proof of concept demonstrates the impact of big data, the process can be scaled up.

iii) Data equals Objectivity. The interpretation of data is the quintessence of its value to business. Ultimately, different types of data could provide different insights to different observers due to relative problem frameworks or interpretation abilities (i.e., the framing effect).

Let’s also not forget that people are affected by a wide range of behavioral biases that may invalidate the objectivity of the analysis. The most common ones between both scientists and managers are: apophenia (finding patterns where there are no patterns at all), narrative fallacy (the need to fit pattern to series of disconnected facts), confirmation bias (the tendency to use only information that confirms some priors – and the corollary according to which the search for evidence will eventually end up with evidence discovery), and selection bias (the propensity to use always some type of data, possibly those that are best known).

A final interesting big data curse to be pointed out is nowadays becoming known as “Hathaway’s effect”: it appeared that when the famous actress appeared positively in the news, stock prices in Warren Buffett’s Berkshire Hathaway company increased. This suggests that sometimes there exist correlations that are either spurious or completely meaningless and groundless.