Recently, there has been a flurry of articles in serious forums about the promise of BIG DATA. The Aspen Institute authored a report on the Promise and Perils of Big Data and McKinsey Quarterly ran an article on the Challenge and Opportunity of Big Data. There is even a conference for data scientists, a newly found species. The most recent issue of Fortune calls data science the “hottest new gig in tech”.
In the midst of all this hype, it was refreshing to read a recent advisory from the Corporate Executive Board titled “Overcoming the Insight Deficit: Big Judgement in the Era of Big Data”. Its premise is that, despite the fact that companies are investing eight to nine figure sums on capturing data from suppliers, operations and customers, less than 40% of their employees have the skills and ability to turn this into useful or valuable insight. It calls for “big judgment” as a necessary complement to realize the promise of “big data”, no matter how comprehensive or well analyzed. Lacking this good judgment, big data can lead to bad decisions and bigger disasters. You can delve into the details of their recommendations and approach by clicking on the link above and downloading the entire article, but their contention rings very true to someone like me who has been in the field of analytics (before it became BIG) all of my career.
There is a natural tendency when talking about or tackling “big data” to focus on the technological challenges involved in capturing, cleansing, linking, maintaining and analysing the data. The underlying assumption is that this process will automatically and magically yield business insights that are tranformational. This is far from the truth. Analysis of data has always been a challenge. It is becoming more so because of the ubiquity of data around us today but emerging tools and technology make it possible to sift through the deluge of information. Those are significant technological challenges that can and are being solved.
The bigger issue has always been finding people with enough business background, maturity, understanding and, above all, intuition and insight, to actually take this information and turn it into unique new opportunities that provide true competitive advantage in the marketplace. This is not just about individuals either, it is about company culture. Among recent companies that have done this effectively, Amazon, Capital One and Google come to mind. These companies do have exemplary technical skills in capturing and harnessing large quantities of data, but what truly distinguishes them is a culture that encourages analytics and fact based decision making. Many companies still have a long way to go in this regard. Unless they deal with their culture and change their “Insight IQ” those big investments in big data will just be a really big waste.
