Big Data: you don’t need a data scientist, you need a data plumber


Share on LinkedIn

Data is like water. It flows from one place to another. Big Data on the other hand is like the ocean, and you can very easily drown in it. It’s another reason why the ‘relevance paradox‘ will be the make or break of business decisions made from the informational tidal wave.

A quick Twitter exchange with Ray Wang from Constellation on information brokering lead me to the idea of data plumbing rather than what some analysts are calling for which is information brokering and data scientists. Ray posited that with big data you’ll be able to subscribe to streams like cable channels, and in a way this is already happening in your every day life: you follow people on Twitter and subscribe to their info-stream, you connect with people on Facebook and LinkedIn and do the same. However too many connections can lead to data overload, or in some cases just a load of garbage that you don’t want to read. But you want to remain connected with these people because now and again they do say something relevant.

So why a plumber, why not a scientist or even a cable engineer ?

Because data is alive; it grows, ebbs and flows as it moves and collates information. Data is in fact non-linear despite the pictures painted of information highways, it doesn’t get into it’s battered Ford Taurus and drives from A to B, it’s very much like a stream meandering from one point to the next, collecting stuff along the way, carving out new channels.

Scientists do the number crunching and make sense of the flood of information, but it’s the handy plumber that makes sure you get the information you need, when you need it, and by how much you need, routing the flow of a constant stream of acquired knowledge and making sure there are no leaks. You want to understand or make a decision on something you turn on the tap and the information flows to you, instantly. It’s not there when you don’t need it. It’s not constantly dripping and distracting you.

The scientist maintains the water tank, the plumber maintains the system itself.

Who do you entrust your data to more ?

Republished with author's permission from original post.

Theo Priestley
Theo Priestley is Vice President and Chief Evangelist at Software AG, responsible for enabling the marketing and voice of the industry's leading Business Process, Big Data/ In-Memory/ Complex Event Processing, Integration and Transaction suite of platforms. Theo writes for several technology and business related sites including his own successful blog IT Redux. When he isn't evangelizing he's playing videogames, collecting comics and takes the odd photo now and then. Theo was previously an independent industry analyst and successful enterprise transformation consultant.


Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here