Bring on the Baby Quants: Making Sense of the Growing Complexity of Enterprise Data

I’ve been spending a lot of time thinking about the pending data revolution in the enterprise, and I finally hit on the solution to a problem that has frankly bedeviled the industry for years, and is only getting worse. The problem is this: as more and more data are accumulated across the extended, networked supply chains companies are building even as – or because – the global recession is growing unabated, the problem of what to do with all this data looms large.

More and more I’m confronted with vendors and customers who have unprecedented amounts of data – much of it unstructured, and originating outside their four walls – about demand, consumption, pricing, the competition, and other salient aspects of the interconnected business world we live. The problem is that the growing accumulation of these data are not matched by a growing accumulation of knowledge about what to do with these data to be more reactive, predictive, supportive, and otherwise proactive with respect to revenues, customer and partner satisfaction, and new business opportunities.

Sure, there are tons of tools out there – and some relatively good vertical industry solutions that target various parts of the problem. But fundamentally, both user and vendor are struggling with understanding how to proactively use all these data to greatest advantage, and, quite frankly, both sides are stumped.

Users are largely outgunned when it comes to understanding new ways in which to use new types of data. In part, this is a training issue – getting extremely creative with very complex data is tricky, even for relatively sophisticated users. Many lack the background in mathematics and modeling necessary for this new level of analysis. And in part this is a responsibility issue: too often there is an understanding that some new data analysis would be of value – such as, something as relatively simple as identifying secondary markets and buyers for excess inventory – but there is no one with the mandate to actually do anything about the problem, or opportunity, that the new analysis presents.

On the other side of the fence, vendors are limited either by a tools approach that over-emphasizes a do-it-yourself analytical model that smacks right into the aforementioned user ignorance about how to do the necessary modeling and analysis, or the vendors are so busy building out the technology that can capture and aggregate these new data types that they’ve sidestepped the issue of what to do with all the data they’re now able to bring to bear in the business.

So, here’s my modest solution to the problem. According to the AP, over 200,000 Wall Street denizens have been plowed under by the current downturn in financial services, many of them quantitative analysts, or quants. Which means a lot of these uber-geeks are on the street, and hopefully, looking for gainful employment. Meanwhile, the graduate programs that hone the next generation of “baby quants” are having trouble placing their students in the rapidly declining financial sector. So, here’s my plan. Give some of these baby quants access to all this new data pouring into the enterprise and see what they can make of it. My guess is that, as long as we keep an eye on their moral compasses – who the hell needs a supply chain bubble-cum-meltdown in the current economic climate? – these number crunchers would be able to do some wonderful things with all this new data.

The quants might have to be content with slightly lower salaries – or at least bonuses – than they were used to getting in the bubble days. But hopefully the prospect of gainful, honest employment – nurtured with some decent stock options – might be enough to tempt these geeks to come over to our line of business and start creating real value. If they play their cards right, this could a ground floor opportunity into one of the more interesting data revolutions in our lifetimes. It could also be a boon to the customers and vendors who are increasingly adrift in a sea of data – and are starting to drown in it, despite their desperate thirst for more knowledge.