After all the hype about big data and its mental cousin Hadoop, some CIOs are getting skittish about investing additional money in a big data program without a clear business case. Indeed, in terms of big data it’s OK to step back and think critically about what you’re doing, pause your programs for a time if necessary, and—yes, even change your mind about big data.
Courtesy of Flickr. Creative Commons. By Steven Depolo
Economist and Federal Reserve Chairman, Alan Greenspan, has changed his mind many times. In aFinancial Times article, columnist Gillian Tett, chronicles Greenspan’s multiple positions on the value of gold. Tett says in his formative years, Greenspan was fascinated with the idea of the gold standard (i.e. pegging the value of a currency to a given amount of gold), but later was a staunch defender of fiat currencies. And now, in his sunset years, Greenspan has shifted again saying; “Gold is a currency. It is still, by all evidence, a premier currency. No fiat currency, including the dollar, can match it.”
To me at least, Greenspan’s fluctuating positions on gold reflect a mind that continually adapts to new information. Some would view Greenspan as “waffler”, or someone who cannot make up his mind. I don’t see it that way. Changing your mind isn’t a sign of weakness; rather it shows pragmatic and adaptive thinking that mutates as market or business conditions shift.
So what does any of this have to do with the concept of big data? While big data and associated big data technologies have enjoyed plenty of hype, there’s a new reality setting in regarding getting more value from big data investments.
Take for example a Barclays survey where a large percentage of CIOs were “uncertain”—thus far—as to the value of Hadoop because of the ongoing costs of support, training, hiring hard to find operations and development staff, and the necessary work to make Hadoop integrate with existing enterprise systems.
In another survey of 111 U.S. data scientists sponsored by Paradigm4, twenty-two percent of those surveyed said Hadoop and Spark were not well-suited to their analytics. And in the same survey, thirty-five percent of data scientists who tried Hadoop or Spark have stopped using it.
And earlier in the year, Gartner analyst Svetlana Sicular noted that big data has fallen into Gartner’s trough of disillusionment by commenting; “My most advanced with Hadoop clients are also getting disillusioned…these organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.”
With all this in mind, I think it makes sense to take a step back and assess your big data progress. If you are one of those early Hadoop adopters, it’s a good time to examine your current program, report on results, and test against any return on investment (hard dollar or soft benefits) projections you’ve made. Or maybe you have never formalized a business case for big data? Here’s your chance to work up that business case, because future capital investments will likely depend on it.
In fact, now’s the perfect opportunity for deeper thinking on your big data investments. It’s time to go beyond the big data pilot and put effort into strategies for integrating these pilots with the rest of your enterprise systems. And it’s also time to think long and hard about how to make your analytics “consumable by the masses”, or in other words, making your analytics accessible to many more business users than those currently using your systems.
And maybe you are in the camp of charting a different course for big data investments. Perhaps business conditions aren’t just right at the current moment, or there’s an executive shift that warrants a six month reprieve to focus on other core items. If this is your situation, it might not be a bad idea to let an ever changing big data technology and vendor landscape shake out a bit before jumping back in.
To be clear, there’s no suggestion—whatsoever—to abandon your plans to harness big data. Now that would be dumb. But much like Alan Greenspan’s shifting opinions on gold, it’s also perfectly OK to re-assess your current position, and chart a more pragmatic and flexible course towards big data results.
