Why you might need to torture data

TORONTO – Although an enterprise resource planning (ERP) system is great for integrating data across a large organization, this type of software is not well-suited for producing predictive analytics, said a SAS data expert at a recent lecture at the Richard Ivey School of Business in Toronto.

As the amount of data produced by businesses has expanded, the traditional response has been to produce more reports. But according to Mark Moorman, data expert and advisor to Office of the CTO at Cary, N.C.-based SAS Institute Inc., there “is way too much data and not enough information about that data.”

Reports tend to conceal important factors like economics, geography and meteorology, he said. But predictive analytics (along with an assessment of how accurate it is), can add needed value to an organization and help it make better business decisions.

Furthermore, not everyone within an organization wants to be made into an analyst by having to go through a plethora of reports that an ERP system produces, said Moorman. “What they really want is the answer.”

The lecture, the first in a series by SAS and the Richard Ivey School of Business to help Canadian executives deal with today’s business challenges, focused on the advantages of predictive analytics to the business.

But before a business can begin to reap the benefits of predictive analytics, some buy-in may be required. Getting commitment from the executive level could be a challenge especially if the organization has always based decisions on intuition, said Moorman.

But the IT department could be another hurdle considering the tradition of controlling, managing and keeping data intact and “now we are asking them to be a little more free, and be a little more loose, and allow us to torture that data a little.”

IT has historically been in the business of producing ad hoc reports, but when it found itself inundated with requests, IT provided users with tools to run queries on the data, Moorman explained. When that approach, in turn, inundated the user with data, IT resorted to providing alerts on particular data. But the business wants predictions, not reports and alerts.

Once IT and business are merged, “you’ll get a more streamlined understanding of what technology can do and what are the business applications you can apply them to,” said Moorman. To that end, SAS encourages organizations to create a Business Intelligence Competency Centre (BICC), a cross-functional team designed to cater to the organization’s needs, not solely those of IT.

The other speaker at the lecture was Ian Ayres, Yale professor and author of Super Crunchers: Why Thinking-By-Numbers is the New Way to be Smart. Super crunching, explained Ayres, is performing analytics on large amounts of data sets and “is data-driven decision-making, it’s having impact on decisions with the speed and scale we haven’t seen before.”

Techniques – regression and randomization – used in super crunching have been used for decades, said Ayres. The former is the mining of historical data to make predictions. The latter approach is data mined from observing groups impacted by different variables as in medical research when a drug is administered to one group and a placebo to another.

Drilling down further still, businesses can conduct randomization trials on subgroups of data to get an even more nuanced strategy. In fact, Ayres thinks that e-commerce businesses not engaged in these tools are “really making a big mistake.”

But accepting super crunching as a business tool may not be simple. Regression analysis is little understood and “ends up being trust me statistics”. Randomization trials, on the other hand, have transparency and therefore garner more trust. “One of the ways to get organizations moving toward becoming super crunching predictive analytics organizations is to use randomized trials as a starter drug,” said Ayres.

That’s one approach to changing the leadership mindset around predictive analytics. Companies can also build upon existing investments. In an interview with ComputerWorld Canada, Moorman explained that businesses who’ve invested in ERP systems can still demonstrate its value by deploying high-end analytics platforms on top to further take advantage of the data.

Although ERP systems can solve many problems, they are historical databases with little ability to formulate predictions on the data. The issue is they have been erroneously marketed and sold “as a panacea, a silver bullet,” said Moorman.

“Unfortunately,” he said, “the platform itself isn’t conducive to super crunching in and of itself.” But he does note that performing predictive analytics on top of an ERP system makes acquiring and cleaning data more efficient.

But the use of predictive analytics is probably more conducive to some industries than others. Financial services has been using predictive analytics because that is the nature of their business, said Moorman. But “laggards” like many retailers, are no longer perceiving themselves as merely “dollar-based” transactional businesses, and are increasingly turning to super crunching techniques.

Super crunching, said Ayres, doesn’t mean the death of intuition in the business. “But increasingly the best decision makers have to be willing to put their intuitions to the test.”