In the News: Big Data

I read an article this morning about the Salesforce acquisition of Implisit Insights, a data-mining/big data company. Salesforce prices appear to have jumped on the news.

This is likely because investors believe that the technology acquired will help Salesforce improve it’s CRM software. The ability to take vast amounts of input variables and condense them into easy to understand, actionable summary stats is important to the future of statistical analysis in business.

But as recent troubles at Palantir show,  this industry is much younger and more distantly useful then currently portrayed. Now the technology is not obscure, at the core it is about embracing the higher processing power of modern computers to perform analysis on huge data sets. But the ability for companies to use these tools accurately is not there yet.

Data-mining in huge data sets is dependent on complex algorithms crunching many variables. The quality of the output depends on a ton of uncertainties, from putting in unbiased inputs to simply asking the right question when looking for the output.  Currently in the small data world, accountability is already a challenge. So how  does this accountability scale up when managers are asked to understand results from systems that are much more complicated then simple averages or VAR?

This is where I think the big-data movement is lacking. When a company like Coca-Cola no longer believes the insights gained from big data are worth listening to, you can be sure that it wasn’t a lack of resources that drove the decision. More likely, it was an inability to create a working relationship where managers understood how to use the services to maximize value. And there is no one to fault, because being a big data scientists almost requires doctorates in the fields of statistics, computer science, mathematics, as well as in the industry which is meant to be analyzed. And until these sorts of people become widely available, big data will have limited applications for most companies.

The Salesforce acquisition may limit the downsides of outsourcing your data-mining insights, but value creation may be a long ways and many hires away.

Leave a Comment