Monday, October 30, 2006

Paralysis by Analysis: The Fatal Flaws of Reliance on Data

Alas, the need to quantify marketing efforts has become a mandate that today exists at most major corporations. CMO teams are consistently measured on their ability to move a brand metric needle or in some cases bottom line. There have been countless articles that have addressed the varying measurement practices, so I won't go too far down this path as it deserves its own blog, but I will raise the reality, that the quantification of marketing efforts has become a norm that drives a fair majority of marketing and advertising decisions; and that we live largely in a "cover-your-ass" world as a result.


Here are 3 reasons why this is a bad thing:


#1 The extinction of going with your gut. When I was younger, I cheered for the Philadelphia 76ers and Larry Bird used to kill my team, thus, forever earning a place on my shit list. As I used to watch Bird light up the scoreboard everytime the Celtics and 76ers played, what I remember hearing about the most was his "feel" for the game, his "court sense" and why these things made him such a good player. Of course these references drove me crazy, but they were fairly accurate. He wasn't the fastest guy on the court, he could hardly jump, but he had such amazing vision and sense of how players would react to his action, that he became one of the greatest players of all time.



Switching gears ever so slightly, when I worked at Hal Riney, I would hear countless stories of Hal breaking the spirit of young creatives by conveying his opinions of their work in a a less than polite manner, but equal to the number of these stories were those that spoke of his natural insight and the ability to act upon it. How Hal could sell a campaign to Ernest Gallo over the course of a cigar session by articulating a feeling he had and how best to bring it to life. These were just stories, but the idea that strong instincts and the ability to go with a feeling and succeed were themes I found truly compelling.



I don't believe its a coincidence that two individuals from such different disciplines were able to rise to the pinnacle of their professions by commonly employing instincts and intuition, nor do I believe these are the only reasons why these individuals succeeded (Larry Bird could shoot the lights out). What I do suggest is that instincts and intuition can be valuable tools, especially as they are honed through experiences to provide a degree of expertise.



What concerns me in this day and age, is that without a series of data supports an idea derived from good judgement, insight and expertise can be quickly dismissed. As individuals look to protect themselves in cya fashion, they increasingly look to the numbers for the defense of their decisions. "What does tracking tell us?", "Where are sales numbers this week?", "How does the campaign compare to norms?" These questions are common in practices that rely too heavily on numbers as rationale for decisions, practices that are widespread and in my opinion quickly undermining the value provided by a gut feeling. Honestly, if numbers really told us everything we needed to know, wouldn't more campaigns be succeeding? Would we be seeing the third Sprint campaign in less than a year? Wouldn't Ford and GM's sales woes be on the mend? In all seriousness, numbers can be helpful guides, but they are not the end all be all.




Which brings me to reason #2, numbers can contradict common sense- There is a great quote that I ran across sometime back that states the mathematician's blunder: "if you tell a mathematician that 3 people are in a room and 5 happen to walk out of that same room, a mathematician will tell you 2 people have to go back in the room for it to be empty." To me, this sums up the approach applied to all too many marketing challenges. A number may say that one campaign will likely peform better than another or that a certain tagline is likely to resonate more favorably among the consumer masses, but its is not a substitute for applying good judgement and common sense.

We may not all have the instincts of Hal Riney, but most of us have a degree of reason inherent to our very nature as humans who live among other humans. When numbers tell us something that contradicts a feeling or instinct, a red flag should go up. But all too often decision makers put their complete faith in a number, a study or a data point. As I mentioned earlier, if numbers were 100% reliable, wouldn't marketers be reaping the kudos for their successes rather than defending their actions.

Which gets us to my next point of concern (#3 for those keeping track at home). Numbers are not 100% reliable, because they are commonly misinterpreted. Methodologies are frequently misapplied, complete faith is often placed in the pov of a non-marketing employee and all too common is the sourcing of a number without appropriate context. When I first became and account planner, my boss at the time warned, "a number without the appropriate context is a dangerous thing." Over the course of time I would quickly understand why this was all too true. I observed clients making decisions based on data without truly understanding what the data was telling them. They assumed because the number for one cell was higher than another, then that cell must contain a stronger response. Little/to know concern was paid to statistical significance.

Statistical significance is a term that very few in our business understand yet its a foundational element in basic quantitative interpretation. It ultimately takes into account the size of a sample and the degree of variance required for a number to be statistically different from another. Clients were choosing one campaign over another because they interpreted it as better performing according to the data when in reality this was not necessarily the case. This is almost forgivable in comparison to projecting the response of 8-10 people in a focus group to a national level. Believe it or not, I've seen this too...the idea that if 6 out of 8 said one thing then 75% of the real target must feel the same way...so the client chooses the campaign that 6 out of 8 like versus the campaign that 5 out of 8 liked.

If a marketer does not understand what a number truly represents, how can they rely on a number to aid in decisions let alone serve as a decision making variable of significant influence?

No comments: