Greg makes a great recommendation on “Hard Facts, Dangerous Half-Truths, and Total Nonsense” by Stanford Business School Professors Jeffrey Pfeffer and Robert Sutton. I’ve been doing some thinking on this myself. As always, I’m speaking for me, #include <std_disclaimer.h>.
Recently, I’ve seen a number of companies adopt a “data-driven decision process.” This appears to come mostly from Google saying they use data, not politics, and a lot of other companies are now echoing that as they believe (a) deciding with data (vs intuition, common sense, etc.) is a good thing, and (b) if they say they’re using the same strategy as Company A, and Company A is kicking butt, people will believe their strategy will also kick some butt.
I’ve now come to really hate this strategy, and cringe when people say they’re “data-driven.” It’s not that it’s completely without merit, but the devil is in the details, and those are often what’s lacking.
First, the biggest example of why I think data-driven tends to be a bad decision model: New Coke. For those that don’t remember, New Coke was a sweeter, more Pepsi-like drink Coca-Cola introduced to replace it’s much-loved Coke, aka Coke Classic. This was done as a reaction largely to Pepsi’s Pepsi Challenge, where they had people on commercials try both Coke and Pepsi and say they liked Pepsi better. Well, after a lot of research done by the Coca-Cola company, and lots of data, they decided to go ahead and change the formula from Coke to New Coke. This then turned into one of the greatest marketing disasters people can remember.
The Coke folks had data. In fact, they made the decision largely on data. They weren’t about to change the formula that drastically unless they had lots of data saying their move would be a success. And yet, it wasn’t.
And this brings me to the reasons I hate data-driven decision theory. A data-driven approach has to start with the right question, followed by experiments that provide data that is properly interpreted to provide the answer. Typically, people fail in either starting out with the wrong question, or by conducting poor experiments that produce flawed data. An evil twin of flawed data is what I call Executive Data Bias. A decision-maker will have a certain bias on what to do, and is looking for data to back up that decision. Thus, flawed data that backs up the decision is accepted without much probing, while good data and the implications are rejected, typically by asking for “more experiments” or “more data,” or questioning assumptions made in the experiment or question.
Data-driven can work. But it requires knowing the right experiments, understanding the data, and being prepared for an unpopular outcome. For example, in television, most decisions to kill programs are data driven. If a program doesn’t do well, meaning isn’t watched, the executives try to move it to a better slot. If that doesn’t work, they kill it. It doesn’t matter as to the quality of the show (although they’ll certainly try more things to keep what they believe are quality assets), just whether or not people are watching. I remember at one point that Charles S. Dutton, who played Roc Emerson on Roc back in the early 90s, made an interesting statement to that effect. When Fox canceled the critically acclaimed Roc, a number of fans were extremely grumpy and vocal, especially as at the time it was one of the few shows to put black families in a positive light. However, Dutton came right out and said that it didn’t matter, what mattered were numbers, and they weren’t good enough, sadly. Update: actually, he blasted Fox as deciding to cancel the show because of network racism versus numbers. I must have misremembered, but luckily the Internet keeps old quotes!
However, most decisions aren’t as cut and dry as whether or not to cancel a program, and thus getting the data and using it to inform is often very, very tricky. This isn’t to say it’s worthless, but rather that data is one component in what should be a thoughtful decision making process.