5 reasons why you should never forget to A/B test


Share on LinkedIn

Gone are the days when CRM teams obsessed over email open rates. Today, our focus is spread across a number of business objectives – from engagement to revenue – and a multitude of digital channels.

In fast-paced CRM teams, it’s easy to feel the pressure when it comes to rolling out highly personalised and engaging campaigns globally, across email, SMS, push notifications, in-app messages, web messages (the list goes on…). It’s also incredibly easy to lose focus on those ever-important business objectives, especially with so much change at play.

But one thing remains consistent and will help you whatever your channel or objective is – A/B testing. As tempting as it may be to skip an A/B test to quickly roll out a campaign you’re about to miss a deadline for, testing is still a CRM team’s best friend.

To help you nail your A/B tests, here are five things that you need to bear in mind and why:

1. Create a data dictionary

In order to run a successful CRM campaign, you need to have the right data points. Without these, you will struggle to segment and personalise – let alone run A/B tests. My top tip for anyone working in CRM is to create a data dictionary (if you don’t already have one) and outline all the data you have available to personalise and segment your campaigns. Once you have this you will quickly see what data you’re missing, as well as what experiments you can run easily and quickly.

2. Right time, right place, right message

The holy grail of CRM. Easy to say, less easy to do. This is something that takes time and continual testing to get right. This is especially important in a global business, where personalising campaigns should go beyond just sending them with local delivery.

To start off, segment your audience by market. In the past, I have seen some big changes in performance across markets by just tweaking times and copy. China is a great example where behaviour can be vastly different from western markets and where testing is important; bolder email designs, localised copy and driving more mobile campaigns have all helped increase engagement considerably.

This is something that should always be tested and explored for any CRM team.

3. Personalisation beyond the {fname} tag

We all know that personalisation increases performance, but how do you test this and what’s the real benefit?

Start off small! It’s important to keep iterating and understand which personalisation is actually improving performance. If you change too much too quickly, it’s hard to understand if it was personalising the content or changing a colour or word that caused the uplift or downlift in performance.

Also, don’t personalise just for the sake of personalising! Does it make sense to the message? Does adding the personalisation enrich your message?
The point of personalisation is to increase an emotive reaction and make them feel connected to your brand. However, used incorrectly, personalisation can feel intrusive and creepy – you might know that someone last visited your site exactly 37 hours ago, but is it helpful to tell them so? Or just a bit creepy?

Personalisation can be a CRM’s best friend, but can also have its downfalls. Make sure you test and see what works for your audience and brand.

4. Push the boundaries

If you work in an app-based company, then push notifications are just as, if not more important a channel than email. They also open up a lot of opportunities for testing; copy length, emojis, rich push vs standard push, buttons etc, All things which are also relatively quick and easy to test, but can have a massive impact on performance.

It’s also important to understand that the behaviour of Android and iOS users changes quite significantly, so testing across both is important, as you’re likely to have different results.

A great example of this is the use of rich push, does it work better than a standard push?
The display of a rich push differs quite significantly between the two, therefore it’s no surprise when the behaviour also differs. I’ve found using rich push boosts performance considerably on iOS, with direct opens increasing 22% on one test, yet saw no difference on Android.

Moral of the story, test across all platforms and also test multiple times, the first result isn’t necessarily the consistent result.

5. Subject lines – an oldie, but a goodie

Sounds simple enough, because it is, but you’d be surprised how many CRM teams forget to run simple subject line A/B tests. With your average office worker receiving over 121 emails per day, with average open rates of just 15.7%, it’s never been more important to make sure your email campaign stands out amongst all the noise.

Running subject line tests on language tone, style and use of emojis is something quick and easy for all CRM teams to incorporate into campaigns.

And don’t forget pre-headers! See these as an extension of your subject line, to grab the attention of your customers.

And to finish off: make sure you’re continuously testing and not falling behind by planning your tests in an experimentation calendar. With every test analyse the performance and take action on the results. It’s easy to neglect this, especially in a fast-paced CRM team. But remember, A/B testing isn’t just ticking a box, it’s something to learn and build from. Make sure you find the time to do so as a CRM team, and you’ll quickly see the positive results in your campaigns and revenue.

Collette Mosca
Collette Mosca is the CRM Lead at Busuu, the world's largest language learning community. Her team is responsible for growth and engagement CRM activities, communicating with more than 90 million users from over 190 different countries. Prior to joining Busuu, she was a Senior Marketing Manager at DHI Group and before that, a Marketing Manager at eFinancialCareers.


Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here