Article by Ernan Roman
Featured on CMO.com
Dan Keir is senior director of direct marketing at Comcast. He has extensive experience optimizing campaigns for both B2C and B2B companies, with a particular focus on test and learn, direct response, and campaign analytics.
Keir’s key priority? Disciplined and innovation-focused testing, as he explains in this edition of “4 Questions For Digital Innovators.”
1. What is the one marketing topic that is most important to you as an innovator?
ABT—always be testing! Testing removes the guesswork and grounds your marketing plans in data to drive the strategy. To continue innovating, you have to understand which changes drive lift in performance, and testing provides that insight. A strong test-and-learn practice is critical to drive improvement in the customer experience and your results. No matter where your company is on the testing journey, all companies can benefit from a test-and-learn tune-up.
2. Why is this so important?
Testing allows companies to innovate faster because they can lean into what is working and quickly shift from something that is not. From simple A/B testing to more complex multivariate tests, it is imperative companies embrace a culture of test and learn as part of their DNA. At Comcast, we really gained traction once we shared our results across teams in a consistent, regularly scheduled manner.
3. How will this improve the customer experience?
When companies leverage continuous and advanced testing techniques, they are able to iterate faster to improve the customer experience. At Comcast, our Xfinity X1 platform is continuously updated. A part of this development cycle is testing new features on a small subset of customers before rolling out the changes to everybody. This ensures an improved customer experience everybody benefits from.
All companies have an opportunity to identify testing areas where they can make improvements and track the results of these efforts.
4. How will this improve the effectiveness of marketing?
A strong test-and-learn practice can dramatically increase results since it fuels a playbook of best practices and innovation. Often a person designated as a champion of testing within each functional area encourages the broader team to test and take calculated risks. Test and learn truly is a win-win for customers by improving the customer experience and for companies as they drive better results.
Bonus: Keir’s six tips to optimize your test-and-learn practice:
Establish a testing road map: Look ahead at the immediate needs of the business to develop what you will tackle over the next few months. Then take a longer view of six to 18 months to align around future insights needed to address opportunities further out.
Gain consensus on your KPIs up-front: Before the test is in market, align stakeholders to how you will measure success once the test is complete.
Build a consistent view of results: To gain stakeholders’ trust that results are not cherry-picked, establish a consistent view of results that is mirrored for each campaign. Add in any additional metrics needed based on the prior KPI discussion.
Prepare to fail: If you’re not failing, you’re not being aggressive enough in your testing. The biggest gains are often driven from the tests where you change the status-quo the most. Be calculated in the size of your test cells to mitigate risk.
Conduct multivariate testing: A fractional factorial test design allows you to test hundreds of combinations with a fraction of the test cells. We have calculated that one multivariate test gains us the learnings of 10 years of traditional A/B testing.
Share the results: Publish your learnings internally, and share those insights to keep everybody moving forward on the goals and KPIs you established.
