Running small tests that compare different parts of your website or marketing can make a big difference to how your business performs.
In this guide, you’ll learn everything you need to know to get started with split testing, including steps to plan and run tests to get meaningful results.
Let’s get started.
What Is Split Testing?
Split testing (also called A/B testing) is the process of comparing two or more versions of website pages, emails, ads, or other digital marketing content to find out which performs best.
Split testing is an effective way to learn what connects with your audience to improve their experience and increase conversions (i.e., completing a desired goal, like clicking a button).
How Split Testing Works
In a split test, you create two versions of your content (like a webpage, email, ad, or social media post):
- Version A is the original. This is called the “control.”
- Version B is a modified version of the control. Tweaks might include changes to messaging, images, colors, buttons, or design. This version is called the “variant.”
50% of your test audience sees version A, and 50% of your test audience sees version B.
The version that produces the best results (leads, demos, purchases, etc.) wins.
Say you’re testing two different versions of a checkout page and more people buy using the variant. This is the page you’d move forward with.
In this example, SplitSignal details a test SEO experiment involving changing a product page’s breadcrumbs.
The winning version then becomes the control and the testing process starts again. You design a new version to experiment with ways to make the page even better.
You can also test more than one variant against the control. This is known as A/B/n testing, or multivariate testing.
So instead of half the audience seeing the control and the other half seeing the variant, you might show 25% the control, 25% variant one, 25% variant two, and 25% variant three.
However, the more tests you run, the more website visitors you’ll need. So this methodology is best reserved for websites with high traffic.
A Quick Note on Split Testing Types
There are distinct differences between split testing for conversions, known as CRO (conversion rate optimization) testing, and SEO (search engine optimization) split testing.
CRO split tests involve duplicating content, such as two versions of a webpage or email, and changing elements to see which version performs best.
SEOsplit testing involves changing an SEO element on your site or page, then comparing those SEO metrics over time.
While CRO aims to increase conversions on a page or element, SEO testing aims to increase organic traffic overall.
Duplicating a page in SEOsplit testing is considered cloaking and violates Google’s webmaster guidelines.
Pro tip: Learn more about how SEOsplit testing compares to CRO testing.
Why Is Split Testing Beneficial?
Split testing helps you understand why websites or campaigns aren’t as effective as you’d like. And what you can make changes to get better results.
Learning what your customers prefer can result in:
- Better user engagement
- Lower bounce rates
- Higher conversion rates
- Low-risk changes
Let’s explore these benefits in more detail.
Better User Engagement
Split testing helps you improve your website or content marketing to increase user interaction. Each test informs your strategy for current and future campaigns.
Say you want more people to click on the links in its emails. MailerLite tested two versions of an email—one with a GIF at the beginning and one without:
The email without the GIF performed best, so this version became the control. Mailer Lite then ran another test, replacing the GIF with an image.
The email without the image continued to generate more clicks:
The company found that, for its audience, images can be distracting and slow-loading graphics can result in a poor user experience.
Knowing how this small change affects engagement helps MailerLite create more effective content.
Lower Bounce Rates
Your bounce rate is the percentage of people who visit a page but leave without performing a specific action. Like staying for a certain amount of time, clicking a link, or making a purchase.
A high bounce rate can be a sign that your webpages aren’t meeting a visitor’s needs. This might be due to several issues, including:
- Slow loading pages
- Low-quality design
- Irrelevant content
Split testing helps you learn why people are leaving your site, so you can make changes that keep them around for longer.
You can test components such as headlines, messaging, and image placement to find the best combination that keeps visitors on the page.
For example, online gallery MedaliaArt set up split tests for its holiday sale. The business experimented with a banner displayed prominently on the homepage vs. a banner displayed more discreetly in the sidebar.
From this test, they learned that the more prominent messaging helped reduce bounce rates. Furthermore, with people spending more time on the websites, sales also increased.
Higher Conversion Rates
Your conversion rate is the percentage of visitors who come to your website and perform an action. Like signing up for your email list or buying a product.
Getting people to visit your website requires time, effort, and resources. Split testing lets you make the most of your efforts by improving content to turn more visitors into customers.
For example, game maker Ubisoft experimented with a streamlined variant of a product page that made the buying process less tedious.
Here’s how the control looked:
And here’s the variant:
The variant reduces the amount of scrolling a visitor needs to do, simplifying the buying process.
With a more user-friendly design, conversions increased by 12%.
It’s a great example of asking your life: What changes will make a customer’s life easier?
Low-risk Changes
Split testing lets you make low-risk tweaks to improve performance without harming current conversions.
For example, you won’t know how your audience will react to a feature or design elements until you launch it. Perform a split test to understand how your audience will receive the change.
If the audience doesn’t respond positively to the change, experiment with different ways to communicate it.
Getting better results doesn’t require broad-brush transformations. Small edits can make a big difference.
4 Common Split Testing Mistakes and How to Avoid Them
To uncover meaningful insights, split tests need to follow certain principles. Most importantly, you need to run the right tests with the right intention.
Before we cover how to run experiments, it’s important to know how not to run them. Here are four common mistakes and what to do to prevent them from ruining your tests.
1. Running Complex Tests
Keep things simple. Test one element at a time and use results to decide on your next test.
Why only one?
Because running multiple tests at the same time can distort results. Let’s take SplitSignal’s story about Drizly.
Drizly’s SEO Manager was spending lots of time doing manual SEO testing. And realized the results were unreliable and unscalable.
With SplitSignal, Drizly focused their efforts on one SEO element to reflect customers’ search intent for one category of pages—“beer brands.”
By focusing on one element, Drizly saw reliable results.
And from there, Drizly could implement the change to the rest of their pages with confidence.
Don’t rush the process. Start with straightforward tests to get more reliable results.
2. Not Testing with a Hypothesis
Before you test anything, you need a solid reason for doing so. Testing random ideas can waste time and resources.
“Version B” performing better than “Version A” means nothing if you can’t learn from it. Instead, come up with a hypothesis.
A hypothesis is a statement that focuses on a specific problem and provides a starting point for further investigation.
Create your hypothesis in three steps: find the problem, propose a solution, create your hypothesis.
Step 1. Find the Problem
Look at your webpage or content from the perspective of your visitors. What’s stopping them from taking an action (e.g., filling out a form or completing a purchase)?
For example, you might run a survey on your site and learn that users felt there wasn’t enough information about the product to consider moving forward.
When you’ve found why you believe visitors aren’t converting, you can propose a solution.
Step 2. Propose the Solution
Decide what test you want to run to fix the problem. For example, if your issue is “there’s not enough information about the product,” you might come up with the following proposals:
- Create a video explainer
- Add an FAQ section
- Add a section covering the product benefits
- Use images to detail product features
When you have your proposal, you can articulate its impact.
Step 3. Create Your Hypothesis
Your hypothesis is an impact statement that covers what you’ll be testing and how it will make a difference. Create it by combining your proposed solution with your problem.
Say your problem is: “There’s not enough information about the product.”
And the proposed solution is: “Add an FAQ section.”
Your hypothesis might be: “By adding an FAQ section, there will be more information about the product and users will be more likely to convert.”
3. Stopping Tests Too Early
Early test results might confirm your hypothesis. The variant outperforms the control as you expected.
So, why waste any more time running a test that’s telling you what you already know?
Because results can have nothing to do with your changes. All kinds of factors impact a test:
- Time of day, week, month, or year
- Seasonal trends (more people shopping at Christmas time or fewer people shopping in January or February)
- Visitors from different sources (search engines, email newsletters, or social media)
- Changes in mood or circumstance (tests at the start of the month might be picking up consumers who’ve just been paid)
Run tests until they’ve attained statistical significance, or the probability that your result is not due to random chance.
In other words, be sure that enough people have seen and clicked on a change to make it the biggest deciding factor.
Semrush’s SplitSignal split testing tool, for example, displays significance as Confidence level.
The generally accepted industry standard for statistical significance is 95%. If your test achieves this score or higher, it’s a sign that your change likely caused the increase or decrease in clicks.
If your test score is lower than 95%, the change is not large enough to consider it a conclusive probability.
In addition to reaching statistical significance, your test needs to meet your predetermined sample size. This is the number of people who need to see your control and variant.
Fortunately, there’s an easy way to calculate this number. Open up a sample size calculator like this one by Optimizely:
Enter the conversion rate of the control in the space under “Baseline Conversion Rate”. If you’re unsure of how to calculate your conversion rate, check out our beginner’s guide to CRO.
Under “Minimum Detectable Effect,” enter the change in conversion rate you want to see. The calculator will then work out the required sample size to use in your tests.
In this example, the control conversion rate is set to 3%, and the relative change is set to 20%. To achieve statistical significance, the test needs 13,000 visitors per variation, so 26,000 in total.
It’s also important to run split tests for full-week cycles. So if you’re running your test for one month and starting on Monday, finish it on a Sunday.
Visitor numbers will fluctuate depending on the day and time, so you’ll want to include each day of the week.
4. Focusing on the Wrong Areas
You can run tests on almost every page element on your website. But that doesn’t mean you should. By focusing on all elements, you risk getting bogged down by insignificant tests.
Let’s look at the results that Google returns for “TikTok video dimensions”:
The information we need to make videos the correct size is easily available. We don’t need to run a test to find out.
Spend time testing the parts of your website or content that impact conversions the most. Here are some common components to test.
Headlines
On average, only 2 out of 10 people will read past the headline (e.g., page title or email subject line). The right headline makes a big difference in how people engage with your content.
For example, adding the more persuasive “Guaranteed Lowest Prices” to the end of a page title helped Locomotive Agency earn 11.4% more clicks for its client:
Test different versions of headlines based on:
- Length. Test short, snappy headlines vs. longer descriptive headlines. Or direct headlines that get straight to the point vs. indirect headlines that raise curiosity (like “Free Guide: SEO Simplified” vs. “Want to Win With Search?”).
- Language. Test power adjectives to convey emotion (like “free,” “best,” “killer,” “ultimate,” “essential”), headlines with and without numbers, and odd vs. even numbers.
- Capitalization. Test title case vs. sentence case (e.g. “7 Reasons Change Can Fail” vs. “7 reasons change can fail”). Title case can add authority and help headlines stand out from the body text to grab attention.
To get started, learn how to write headlines that increase website traffic.
Calls to Action
A call to action What (CTA) encourages your audience to take deliberate action.
For example, the headline on this Semrush landing page explains the value of the product. The call-to-action button encourages visitors to act on it: “Start your free trial.”
The design, placement, and wording of buttons or links that ask people to take action can determine how many clicks they get. A small change can boost conversions.
For example, optimizing emails with a clear CTA at the end rather than the start helped Process Street’s Benjamin Brandall increase open rates and click-through rates.
Experiment with different elements of CTAs:
- Button color. Test colors that contrast with other colors on your website to see which grab attention and get clicks.
- Text. Change the wording of CTAs to see if they become more persuasive (e.g. “Get Started” vs. “Start Your Free Trial”).
- Size. Test small, medium, and large buttons. Does size impact how many clicks a CTA gets?
- Position. Try placing CTAs at the top of the page vs. further down the page. Or moving them closer to headlines or benefit copy.
- Shape. Test round button corners vs. square button corners. Does the change help draw attention to the CTA?
- Links vs. buttons. Test CTAs as plain links vs. buttons with colored backgrounds. If your page already has several buttons, a muted tone for less important CTAs might have a positive effect.
Read our guide to CTAs for further information.
Visuals
People like to see what they are buying. Images and videos make your product come alive.
For example, SplitSignal details a case study where they tested a landing page against a variation that included testimonials and images. Brookdale Living saw conversions increase 3.92% with this change.
To find out what works best, test:
- Text-only content vs. posts with images or videos
- Regular images vs. animated GIFs or videos
- Product photos vs. graphics
- Team images vs. stock photography
- Video length
Product Descriptions
How you describe your product helps communicate benefits. Descriptions need to tell visitors:
- What problems your product solves
- How it helps them
- What makes it the best choice
Experiment with:
- Formats. Move descriptions up or down the page. Use bullet points to draw attention to important benefits and features. Test bullet points vs. icons that display benefits.
- Visuals. Test images showing products in use (e.g., on a model) vs. on a plain background. Test videos vs. zoomable photos.
- Reviews. Do star ratings improve results? Test ratings vs. single customer testimonials.
These small changes don’t drastically change a page, but the results can be telling.
For example, adding a review star rating to ecommerce product pages helped CRO agency Growth Rockincrease conversions by 15% and revenue by 17%.
How to Prepare and Plan Your Next Split Test
Split testing is a four-stage process:
- Determine what to improve
- Hypothesize your test
- Create variation and run the experiment
- Measure results
Let’s look at each stage in detail.
1. Determine What to Improve
Before you start a split test, you need to know what to test. Testing different headlines, for example, won’t improve performance if the design is what’s bothering visitors.
To find out where to focus your attention, conduct research on how your website is performing.
You can do so with Google Analytics. In this case, we’ll use GA4.
To see your top pages, head to “Reports” > “Lifecycle” > “Engagement” > “Pages and screens.”
You’ll see pages with the most number of views, average engagement time, and conversions.
To see where your traffic is coming from, head to “Reports” > “Lifecycle” > “Acquisition” > “Traffic acquisition.”
You’ll see a table sorting your traffic by channel group (e.g., Direct, Organic Search, Display, etc.). But you can also see traffic by “Session source / medium” to see what URLs funneled traffic to your site.
You can supplement your analysis with additional data from customer interviews and heatmaps.
A heat map is a tool that tracks the motions of a mouse so you can see how people use your website. Some popular options are Hotjar and SessionCam.
By logging mouse movements, you can see what attracts visitors’ eyes and whether they click what you want them to click.
Learn more about heat map options and how the technology benefits your business.
2. Hypothesize Your Test
Your data will likely give you many different start points. To work out where to start, prioritize ideas using an impact matrix:
Fill out this matrix to determine where each potential test falls:
- Quick wins: High impact and low effort
- Big bets: High impact and high effort
- Fill-ins: Low impact and low effort
- Thankless tasks: High impact and low effort
Based on this data and your available resources, choose what you want to test first.
Then hypothesize your test using the three-step framework we outlined earlier: find the problem, propose the solution, create the hypothesis.
3. Create the Variation
To set up your split test, choose a split testing tool or service. Here are four to try:
- SplitSignal. Semrush’s own split testing tool is the best option for running SEO-specific tests. You will need to have at least 100K visitors a month for a significant test, making SplitSignal a better option as your business grows.
- Google Optimize. Free to use and beginner friendly. Lets you create and run simple A/B tests, split URL tests, and multivariate tests.
- VWO. Features a WYSIWYG (what-you-see-is-what-you-get) editor for beginners. Plans also come with heat maps and site surveys to help you plan and measure the effectiveness of changes.
- Optimizely. Makes it easy to run small tests without technical expertise. Its Stats Engine also makes analyzing results simple. It is, however, more expensive than Google Optimize or VWO.
4. Measure Split Test Results
Split testing tools will show you the results of your test, including the winner, confidence level, and uplift (the difference between performance of a variation and the control group).
But they won’t analyze results for you. This is something you’ll need to do yourself.
Statistical significance is the biggest sign that your test has had a positive impact. But don’t stop your analysis there.
Look at the audience breakdown of groups that are meaningful to your business (e.g. new visitors, returning visitors, social media visitors).
This will help you answer questions like:
- How did traffic from different sources perform (like search engines or social media platforms)?
- Which option worked best with new users?
- Which option won on mobile or desktop?
Why is this important? Because even if your test isn’t a huge success, there’s a chance your hypothesis is right with certain audience segments.
Even if the overall result doesn’t go the way you expected, now you’ll know what to avoid. Running a deeper analysis on why it happened will give you even more insights to use in future tests.
Get Started with Split Testing
Split testing is a learning process and every test is a chance to understand something about your audience, what works, and why.
Never stop testing. Every new test you run should be based on research and a good hypothesis. The more you can test, the more insights you’ll have to improve your marketing.
Most successful split tests will give marginal gains, like 1%, 3%, or 5%. But don’t see these low numbers as a failure; any uplift is positive.
Look at results from a 12-month perspective. If testing helps you increase your conversion rate by 3% each month, your sales will triple over a year.
Focus on small gains. They all add up.