42% of Marketers Are A/B Testing The Wrong Thing

Talia Wolf

Jul 10, 20147 min read
42% of Marketers Are A/B Testing The Wrong Thing

A lot has been written about conversion optimization in the past few years. The importance of A/B testing and optimizing websites has become clearer and has never been more interesting. As many companies add conversion optimization methods into their routine marketing efforts we see an increase in many tests and a decrease in their analysis.

A recent study by Adobe showed 40 percent of marketers say they see positive results while A/B testing. However, 42 percent of marketers say analyzing the A/B test results is the hardest part of conversion optimization. More and more companies are relying on their test results to evaluate the tactics used, and recommend next best actions. The problem being of course that 42 percent of them aren’t sure they’re analyzing the results correctly or are choosing the right way to react.

Testing elements

The reason for this disparity is the flawed strategies marketers use for A/B testing. In recent years, the majority of conversion optimization efforts have been tuned to the automatic choice — behavioral targeting. The majority of marketers are using behavioral targeting to determine their user’s behavior on their site and introduce more personalized content.

This means detecting what country users are coming from, their browser, device, the time, and many other important metrics that can help site owners personalize their message are often being ignored. Behavioral targeting focuses almost exclusively on features and pricing — adjusting both these elements is supposed to bring the change marketers look for.

AB Testing Schema

The most common way of using behavioral targeting in conversion testing is the process of testing elements. During these type of tests marketers duplicate their landing pages and test different elements such as call-to-action buttons (color or text), a different title or maybe a different image. An additional way of testing elements is “flipping” the landing page and moving the same elements around on the page.

The emphasis being, of course, on having the exact same landing page as a control and being able to distinguish once a test has finished what made the difference.

AB Testing Schema 2

Why testing elements won’t help

So, here’s why 42 percent of marketers find analyzing their tests difficult: their tests mean nothing.

Now, before you jump at me and tell me you’ve seen great results (which I’m sure you have) hear me out. Yes, these tests can result in an increase in conversion and yes these tests can increase revenue BUT, to what extent?

Testing elements can only take you so far, and here’s why.

When testing elements you’re making small changes to your landing page, such as changing the color of a button, moving the image from one side of the page to another and generally making very small changes that may generate small results. Say, for example, you’re testing your call-to-action button, and you have one that leads to a landing page with an orange button and the other to a page with a blue button. You run your test and see that the blue button won by 3 percent — great.

Now what does that tell us? Does this mean your audience prefers the color blue? Should you change your site to blue? Or should you just keep using this blue button? What would be your next step? This test didn’t teach us anything actionable, so we have no idea what to do next.

Another issue with these small-scale A/B tests is that they are inherently limited, you can only test so far and by constantly tinkering with these small changes you’re not really able to build a lasting strategy that will keep conversion increasing. Once you’ve tested that blue button, that’s all the actionable data you have, the path into the testing future is unclear.

Incremental changes like these can only lead to small increases in conversion rates, leaving the majority of your tests with uninteresting results and the inability to scale.

The case for testing strategies

Since the majority of brands today focus exclusively on their features and pricing, most tests are based on exactly these behavioral elements. Trying to put as many features of a product or a service up-front is a very common way brands distinguish themselves from their competitors. Pricing is yet another element that is used frequently by companies to mark the difference between their competitors and themselves. But, as we've seen, it can lead to incremental changes and results that are far from dramatic.

So what does work? Emotional Targeting, a methodology we’ve developed over the past six years focuses on something entirely different than elements, focuses on strategy and concepts. There are several reasons why emotional targeting works better than behavioral targeting, the most important one being that emotional targeting focuses on why people buy a product or a service but not why they should. Behavioral targeting that generally focuses on features and pricing, whereas emotional targeting focuses on the emotional reasons people purchase products/services.

Our purchasing habits are determined by what we’re looking for emotionally; we buy products because of what they make us feel about ourselves. In fact, we often don’t remember where we bought a product, it’s price or why we actually chose it, we remember the experience. We buy products because we see ourselves as better versions of ourselves with these services/products. Emotional targeting is all about understanding why my customers are purchasing my product — and no, it’s not because I have more features than my competitors or I’m cheaper/more expensive, it’s because of what my product/service makes them feel about themselves.

The idea of using emotional targeting means you’re testing strategies and concepts — not elements. This means you don’t have the same landing page with different titles, but you actually have different landing pages altogether — each landing page represents a different concept and idea.

Emotional targeting (testing concepts) works better because you learn more. Since each landing page is a whole concept of its own and represents different emotional triggers, when one landing page wins you can understand why better, you can understand your audience better, you can tap into your product and why people actually buy it and most importantly — you can scale. With testing strategies you can continue optimizing and once you have your strategy set, you can scale down to changes such as the of a color of the button.

How to test strategies

A few months ago we concluded our first round of tests for a large e-card company. This company has a huge audience of customers who use the product to create invitations, cards, slideshows, videos and much more.

Their original landing page looked like this:

Original Website Design

With over 100k visitors a month to this landing page they had a pretty good conversion rate. The funnel was as follows: Users arrive on the landing page and need to download the product, then they need to sign up, enter their credit card (as there’s a seven-day trial) and only then can they start creating their e-card.

With this setup, the company has many different obstacles facing them. For one, they are much more expensive than their competitors and they do not have a one-time payment for a single card (the minimum you can pay for is six months of access). Additionally, it’s a download product meaning people have to download it to their computers before they can start using it.

When we set out to the test we ran an in-depth research to determine what the emotional triggers are and why people create e-cards. We determined we needed to focus on the promise of an amazing event, an unforgettable event that will represent their best party ever. People will be talking about their event for a long time to come.

We created two variations:

Website Version A


Website Version B

Each variation (completely different from each other) was designed to create an experience that enhances fun, love and the cherished moments that will come. Using different colors — green vs. pink, for example, enhanced different emotions that made it easier on the user to use the product. Each part of the landing page was designed establish wealth, relaxation, fun, surprise and a cool event.

The results introduced an immediate 65 percent increase in revenue during the first round of testing. Downloads increased by 12 percent. So, you can see how an emotional elements test — as opposed to an elements test — had an immediate dramatic impact on the conversion rate of the site in all sections of the conversion funnel.

Where to start testing

One important lessons that can be taken from this test is to do with where to start testing. A common “mistake” that marketers make when trying to increase revenue online is test the checkout page or the billing page. It may seem like it makes more sense to go straight to the end of the funnel to try to make a bigger impact,as opposed to working your way down, But from our continuous testing, we have learned time and time again that this isn’t always the case. Emotional testing strategies can help increase revenue right from the top part of the funnel (a well-targeted banner ad for example) and there’s no need to rush to the end of the funnel. Starting at the top and testing your way down can lead to better and more substantial results.

Testing for results that matter

In order to avoid wasted time and tests it is important to build the right strategy for your tests. Before starting your test answer the following questions:

1. What are my test goals? What do I want people to do?

2. Why do people want my product or service?

3. How do people feel about themselves once they’ve purchased my product/service?

The answers to these three questions will help you establish the first part of your strategy. The key is to understand your audience better, and to be able to scale from one test to another.

Author Photo
Talia WolfAs founder of GetUplift, Talia and her team provide conversion optimization services for high-growth companies using customer-centric methods, emotional targeting strategies and in-depth data. Talia has been invited to teach conversion optimization on hundreds of stages such as Google, MozCon, Call To Action Conference, SearchLove, and many more and was recently listed as one of the most influential experts in conversion optimization.
Subscribe to learn more about SEO
By clicking “Subscribe” you agree to Semrush Privacy Policy and consent to Semrush using your contact data for newsletter purposes