You could sum up the goal of many big data projects — whether it has been spelled out or not — as an attempt to answer the question, “What on Earth is going on?”
This is probably a little too concise in many instances, of course, but CIOs who have been exposed to big data analytics initiatives could easily spell out a more nuanced version of this question. It could be a matter of understanding why sales are down, why a blog post went viral, or what customers are doing today that indicates how they will behave tomorrow.
In other words, whether you’re analyzing the past or trying to be predictive, big data projects should leave you knowing more than you did before. Right?
A recent interview with someone worlds away from the realm of analytics recently got me thinking otherwise. Yuval Hoah Harari is a scholar who, in some respects, took on the biggest of big data projects in his book Sapiens: A Brief History of Humankind. This is on my reading list, because I’m interested not merely in how technology can boost analytical thinking, but how our ability to develop our collective understanding of the world has evolved over time.
On Edge.org, Daniel Kahneman (Thinking, Fast and Slow) begins their video discussion by referencing a chapter in Harari’s book about the discovery of ignorance, which, the author reasons, was when science was first developed. In other words, you have to know what you don’t know before you can know anything else. Here’s a bit more on this theory, in Harari’s own words:
I often tell my students at the University that my aim is that after three years, you basically know less than when you first got here. When you first got here, you thought you knew what the world is like and what is war and what is a state, and so forth. After three years, my hope is that you will understand that you actually know far, far less, and you come out with a much broader view of the present and of the future.
Even if he’s never set foot in a data centre, I think Harari has captured one of the most common unintended consequences of so many big data projects, and IT projects in general. Start working with unstructured data and, even with the best products and services at your disposal, you may have many more new questions than answers to those originally posed.
What’s different, in Harari’s case, is that he goes into his research with this reality in mind. You could almost call it an active pursuit of ignorance, which in turn provides its own potential value. Here’s another quote:
I’m trying to do something that is the opposite of predicting the future. I’m trying to identify what are the possibilities, what is the horizon of possibilities that we are facing? And what will happen from among these possibilities? We still have a lot of choice in this regard.
Perhaps it’s unfair to compare the kind of work Harari is doing than the data science of large enterprises, but depending on the quality of data you’re working with, the existing level of insight the organization has had and the business goals, the active pursuit of ignorance might be a more realistic starting point for analytics. After all, as Harari might be the first to point out, we’re only human beings.