With every failed project, IT managers lose a little more credibility. But it could be worse. At least they’re not social scientists.
In the summer issue of City Journal, Jim Manzi takes a long, hard look at why the predictions made by economists and others routinely prove false, yet are still justified by the original forecasters. For example, a social scientist may insist that implementing a literacy program in a troubled neighbourhood will radically improve the lives of at-risk youth, only to see crime and violence skyrocket a year or two after the program has been in place. The social scientist, however, could point to variables like increased drug use in the area, a faltering local economy or rise in the general unemployment rate. Manzi explains that it’s the “counterfactuals” that pose a significant problem:
We have no reliable way to measure counterfactuals—that is, to know what would have happened had we not executed some policy—because so many other factors influence the outcome. This seemingly narrow problem is central to our continuing inability to transform social sciences into actual sciences. Unlike physics or biology, the social sciences have not demonstrated the capacity to produce a substantial body of useful, nonobvious, and reliable predictive rules about what they study—that is, human social behavior, including the impact of proposed government programs.
Or, you could argue, the impact of IT. The original deployments of enterprise resource planning (ERP) systems about 20 years ago were supposed to revolutionize the way companies operate and lead to major improvements in the process and bottom line. Many got there eventually, but not without a lot of pain, and probably taking much more time than the vendors originally said they would need. Productivity tools often breed little more than time-wasting, customer relationship management has been an ongoing joke, and lots of CIOs are still wondering when they’re going to get more value out of the data warehouses they have spent so much money to build. And that’s just the IT operation. Lord knows what senior management is thinking.
There are ways, of course, to fudge the numbers and try to outline what would happen were a company not to move forward with a business-driven technology initiative. Most IT departments don’t bother doing that, though. They don’t need to. Organizations realize they have to set up portals, do something to manage their data, secure mission-critical information. Vendors simply hint, in their white papers and press releases, about the risk and high-cost of “doing nothing” without ever attempting to quantify it.
Maybe because technology seems like a hard science – with products that can be measured on feeds and speeds – we overlook the counterfactuals that could actually help avoid unsatisfactory outcomes. If we think of IT as more of a social science, and truly accept the premise that people issues are at the heart of many failed projects, counterfactual thinking becomes much more appropriate. Manzi charts the history of social scientists learning to experiment with control groups: one that might try a course of treatment, for instance, and one that doesn’t. Some smart IT managers do the same thing with pilot projects, but take it a step further: Think of what the enterprise might look like without the BI software, or the service-oriented architecture. What factors might make conditions better, or worse, than if the technology were deployed as planned? We may never be able to measure what didn’t happen. But by working harder to imagine those alternate realities, IT managers could have a bigger, more positive impact on what does happen.
