Nobody likes to see their idea lose. Everybody wants straight A’s. Look good for the boss. It’s the Lake Wobegon effect: all the children are above average. (If you don’t appreciate that joke, take this pop quiz.)
So at some level it’s not surprising that survey data of Fortune 1,000 marketers from the CEB Marketing Leadership Council shows that only 50% of marketers agree or strongly agree with the statement: “My team accepts that some experiments must fail in order for us to learn from them.”
But if your experiments have no risk of failure, then one of two things is true:
- You’re not actually testing new ideas, you’re just polishing the fenders of your old jalopy.
- You’re delusional and should seek the advice of a doctor.
Failure is an inherent part of real experimentation. “We have an idea. Let’s try it. Works? Great. Doesn’t work? Good to know, let’s try something different.” You do your best to construct your hypotheses with reason and logic, but also with imagination and creativity. You test those hypotheses, and whether they’re proven right or wrong, you’ve learned something of value.
Jim Manzi reported in his book Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society that Google ran approximately 12,000 experiments in 2009, with only about 10% of them successfully leading to business changes. That’s what experimentation looks like — and in the end, it netted them a large number of wins by absolute numbers, but only from trying a lot of things that didn’t pan out.
This is one of the key tenants of agile marketing: continuously running experiments in rapid iterations. Try ideas on a small scale, quickly toss the ones that don’t pan out, and eagerly scale the ones that do.
Unfortunately, it hasn’t been a tenant of legacy marketing culture, which could better be described as: have an idea, go big fast, pray you end up being right (or that no one can measure if you were wrong), and let your career ride on the outcome. (Good time for a review of survivorship bias in Fooled by Randomness.)
And, sorry, but avoiding innovation is not an option.
The only viable way forward is for marketing to embrace a test-and-learn culture. According to the CEB, about 25% of marketers are doing just that, rating their teams as effective or very effective at test-and-learn experiments. Bravo!
As for the rest… well, at least half aren’t above average yet.
Thanks to Anna Bird at the CEB for sharing this data with me and giving me permission to share it with you.
