English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
Submit post
Go to Blog

7 Ways You’re Screwing Up Your Email A/B Tests

The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.

7 Ways You’re Screwing Up Your Email A/B Tests

Yury Kanyuka
7 Ways You’re Screwing Up Your Email A/B Tests

FACT: Email is an amazing channel for attracting, winning and retaining customers. Not to mention generating repeat business.

But there is one problem; it is really hard to get emails right for the first time.

What you consider an engaging subject line, recipients see as a dud. A call to action you believe would compel them to click, spurred no action whatsoever.

That is why it is important to A/B test your emails to find new techniques or elements that improve conversions.

Unfortunately, many companies launch split tests hoping for the best. They disregard the rules of A/B testing and commit some major mistakes rendering their efforts useless.

So, if you have been split testing emails but see no viable results, keep on reading. I am going to show you the most common A/B testing mistakes sabotaging your efforts.

Before we begin though…

Why You Should Always Start by Testing Concepts Not Elements

As it turns out before you even begin testing various email elements, you should identify a general strategy your audience responds best to.

In other words, before you start fine-tuning the template, testing subject lines or modifying the call to action, you should first test two different marketing strategies against each other.

For instance, you could test two different ways to convert recipients – via email form or social media login. Or sending people to a landing page vs. allowing them to purchase the product directly via email.

And only when you have identified the winning strategy, should you start testing individual elements to improve conversions.

However, when you do, make sure you don’t commit any of the mistakes below:

Mistake #1. Testing More Than One Element at Once

By far and away, this is the most common mistake of them all.

You have so many ideas on how to improve email conversions. But the last thing you want is spending weeks to test every one of them in turn. And so, to speed things up a little, you decide to analyze them all at once.

You send different template variations under various sender names, using different subject lines, and including different copy in each test.

This results in so many email variations that, in the end, you can’t even tell if any of your ideas worked.

Time-consuming as it may be, you should always test only one element at a time.

Mistake #2. Checking Results Too Early

Since the majority of email platforms start delivering campaign results within 2 hours after sending, it is tempting to start analyzing a test’s performance right away, right?

However, by doing so, you miss out on some important data.

For one, users have different reading habits. Some open the email right away, flick it and either act on it or forget about it. Others put important messages aside to check out later. And as a result, might come back to your email a couple of days later.

And so, by analyzing results too early, you might miss important traffic and usage patterns, affecting the actual test results.

From personal experience, I can attest that the best time to start going through test results is about 2 weeks after launching the campaign.

Mistake #3. Ignoring Statistical Significance

80% of your test results are worthless. It is no different for almost anyone else split testing their emails.

And so, the challenge is to draw conclusions based only on the remaining 20%.

One way to achieve it is by identifying statistically significant results and weeding out those caused by pure chance.

One way to do it is to use a statistical significance calculator. Personally, I use the one developed by Visual Website Optimizer, but you could use just about any similar app out there.


Mistake #4. Focusing on Too Small Sample Size

The number of recipients you include in the test affects the outcome. The smaller change you want to test, the greater sample size you might need.

For instance, let’s assume that you developed a hypothesis stating that using emojis in the subject line should help increase the existing 15% open rate by 10%. To conduct such analysis, you need to test this hypothesis on 8,000 people.

Using a smaller sample size will deliver statistically insignificant results.

So, before you launch the test, you need to calculate what sample size you need to receive viable feedback from. To do so, use the Optimizely’s sample size calculator.


Mistake #5. Failing to Develop a Proper Hypothesis

A hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation. I am sure you have heard this definition already.

However, in email split testing, a hypothesis must have one other characteristic:

  • It must be applicable to different campaigns.

So for instance, a statement such as “emails with animated pictures generate a higher CTR” would work as a hypothesis. Once proven right, it could be applied to many different campaigns.

On the other hand, assuming that a particular subject line will fare better than another would not work. It applies to a specific campaign only and cannot scale to your other email efforts.

The lack of understanding of this important email hypothesis characteristic leads to conducting tests de facto without a hypothesis at all. Or in the best case scenario, using a weak hypothesis to try and improve conversions.

To avoid making this mistake, use the industry’s approved hypotheses. Jordie van Rijn collected 150 of them in this post.

Mistake #6. Not Testing Segments

We all know that different audience segments might respond to your message in their own unique way. And thus, a hypothesis improving conversions in one segment might deliver no results in another.

Just take cultural differences as an example. Spanish recipients might have no problem with a high frequency of emails. However, emailing a couple of times a week might prompt subscribers from other countries to abandon your list.

And so, segment your tests to analyze different user behaviors.

Mistake #7. Sending Each Variation at a Different Time

To receive viable results, you should analyze no more than one variable at a time.

And yet, I see many companies unknowingly adding another factor to the mix: time. How? By sending each variation at a different time.

With this method, half of the subscribers might receive one variation at a time they are not busy and thus, susceptible to opening marketing messages. The other half, however, might get it in the middle of a busy day, resulting in many ignoring or even overlooking it.

As a result, the data gets skewed by different recipient behavior, depending on the time at which they received the email.

So, to guarantee the validity of the test, always send both variations at the same time to ensure that no other factor interferes with the test.

What About You?

Are there any other email testing mistakes you made? Share them with us in the comments below.

Yury Kanyuka

Provides valuable insights and adds depth to the conversation.

Email marketing strategist at SEMrush. Yury Kanyuka has five years of email marketing and marketing automation experience.
With his team he increased SEMrush's email-driven revenue by 200% in one year.
Share this post



Either just recently joined or is too shy to say something.

So this is only for marketers with ginormous lists because #4 says I need at least 8000 people and #6 says I should separate my tests in smaller segments. Someone please rewrite this article for those of us who are small/medium biz marketers. Or say upfront who this actually applies to.

Either just recently joined or is too shy to say something.

Hm... My searches for my wants led me here...
My facility ( a motel)
is having a dispute of massive division of opinion, and I am looking for a mail service that will allow A/B testing through the course of six separate emailings mailings..
divide the entire database into a/b for all campaigns for six messages.. with reporting on 'destination clicks'

A half gets folksy personally written and casual marketing messages (think bed & breakfast down homey)
B half gets corporate speak and clarity and concise marketing messages (think Marriott or Days Inn)

opens, clicks, and unsubscribes follows the database through all 6 campaigns.
And regardless of outcome, for the next round of messages, the A's stay A's, and B's Stay B's
Hopefully, as names are added, they just get randomly assigned/split down the middle to both campaigns.

Anyone? can I get a fraction of what I want?
Anastasia Armeeva

Either just recently joined or is too shy to say something.

Marke Johnston

You mean you need to randomly assign one of 2 sequences in real time?

We use marketing cloud called Mindbox for this purpose. However, it is the local product, thus, English localization is poor.

Also this logic can be implemented in Marketo, here’s the thread on their forum: https://nation.marketo.com/thread/21281

Probably, it also can be done in Pardot by SalesForce campaigns or tagging the audience randomly. That then could kick off 2 engagement streams/automation rules.
Domain Registration .co.uk

Occasionally takes part in conversations.

Brilliantly defined. Thanks buddy.
Comani ‘12‘

Either just recently joined or is too shy to say something.

Nice one mate! Love your way of thinking
aboadam demerdash

Occasionally takes part in conversations.

Thank you for the great article

An experienced member who is always happy to help.

"To receive viable results, you should analyze no more than one variable at a time."
Simple, Actionable Tips like this are drizzled all over this high value article. If you've only read it once you missed more than one piece of game-changing advice from Yury Kanyuka.

Either just recently joined or is too shy to say something.


Either just recently joined or is too shy to say something.

This article is informative. Thanks for sharing.

Either just recently joined or is too shy to say something.

Thank you for the information.

Send feedback

Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!