Many of us would like to believe that we base our important business decisions on well grounded facts. It does not take much looking around to realise that the data do not necessarily support the decisions being made. David Brooks wrote an op-ed piece in the New York Times, “What Data Can’t Do” that outlines five areas data science struggles with.
- Data struggles with context – that data is bad at narrative and emergent thinking
- Data creates bigger haystacks – that we gather more and more findings as the data gets bigger but most of these are spurious
- Big data has trouble with big problems – that few are convinced to come to an answer by analytics on the big questions of life and society
- Data favours memes over masterpieces – That data finds it much more difficult to spot specialness before it becomes popular
- Data obscures values – that data is gather and structured according to a researcher’s worldview of the problem
Nevertheless, decision sciences are all about data. GilPress in “A Very Short History of Data Science” states that data science emerged as a discipline in 1974 when Peter Naur stated that it is “The science of dealing with data, once they have been established, while the relation of the data to what they represent is delegated to other fields and sciences.”
There has never really been a shortage of data for companies to analyse but there have been limitations to what could be accomplished by reasonable computing power that businesses had at their disposal. According to some sources, the volume of data captured is almost doubling every year. This is the world of big data where the issue is that the volume of data is so large that regular statistical packages and even regular statistics need to be adjusted. This issue is being addressed by data science which according to Drew Conway is the confluence of three areas: hacking skills, math and stats knowledge, and substantive expertise. See his blog for his explanations of these three skills.
Brooks states that “data can be used to make sense of mind-bogglingly complex situations. Data can help compensate for our overconfidence in our own intuitions and can help reduce the extent to which our desires distort our perceptions.” Data science will not, of course, answer everything to the satisfaction of business leaders all the time. This is especially true of the big strategic questions. In the commercial arena where executives are looking for an edge in their decision making, analytics is a useful must-have tool but ultimately it will never provide the complete picture on the big questions.
While data science, big data and all that moves forward, there is another more qualitative factor that also needs to be addressed in order for business leaders to make use of all that data science has to offer. I call it an emotional overlay. As Brooks states, “Commerce depends on trust. Trust is reciprocity coated by emotion. People and companies that behave well in tough times earn affection and self-respect that is extremely valuable, even if it is hard to capture in data.”
Trust is built on the back of our perceptions and emotions in a situation, environment or relationship. Our perceptions and emotions are a vital part of the decision making process and no amount of analytics will or should eliminate these completely. Thus, I add a perceptual/ emotional overlay to Drew Conway’s data science Venn diagram as it relates to management decision making. That overlay may help explain why certain mathematically robust data insights are accepted in the decision making process and others not.
What is a possible antidote for the emotional overlay that may sometimes cloud the robust computationally accurate findings of data science? Storytelling. Infographics are headed in this direction but storytelling in its full glory allows analytics to come alive in an emotional context that matters to decision makers. In the end, we never escape the full experience which includes rational and emotional parts even when the experience is analytics and data science itself.