In this post, we will discuss what outliers are in SEO A/B split testing. We will shed light on some of the issues that cause outliers to occur while making recommendations on dealing with them.
Let’s dive right in.
Sometimes we see traffic spikes within an SEO A/B split test; it is an indication that there are outlier URLs within the data set. The traffic spike is occurring as there is a misalignment of data.
There are several types of outliers in SEO A/B split testing:
How Does Organic Traffic Affect SEO A/B Split Testing With Outliers?
SEO A/B tests require a group of pages to be split into two equitable groups (control and variant) based upon the organic traffic.
It also required 100-days of pre-test organic traffic before splitting URLs into control and variant groups.
Each test requires a minimum of 100,000 clicks to the set of URLs being split tested to have a statistically significant test result.
When traffic anomalies within the group are split, these traffic outliers are excluded from the control and variant groups.
An Example of Organic Traffic Outlier Spikes:
*The 100 day model in the above example runs between Jan 30th, 2022, and May 9th, 2022.
While the testing period of 30 days has not yet been completed, we can see there are issues with the prior 100-day model through traffic spikes.
When we have spikes in the 100 day model between the control and variant groups, it is an indication that the model is not suitable for SEO A/B split testing due to the traffic not matching up.
In addition, the broad light blue shading shows us the traffic model is not good.
Read our knowledge base article on How to read SplitSignal test results.
Topical Relevance in URLs and Traffic Modeling
Some sites have traffic differences based upon topics, even if the testing page template is the same. The difference in traffic can create outliers.
When this happens, we recommend using URLs from similar topics. These URLs can then be uploaded into SplitSignal, where you would then re-run your test.
How Can Outliers Occur With Page Elements?
Outliers in page elements can occur when testing a change in page elements. You should ensure that all the URLs in your control and variant groups have the element on the page you want to change. Pages that are included that do not have the element will be seen as an outlier.
Page Elements Outlier Example:
If we were looking to change an h3 tag in blog articles to an h2 tag in our test, we would not assume all pages have the h3 tag.
Instead, your internal team or an approved SplitSignal agency should crawl the site with a tool such as Screaming Frog or Deep Crawl to grab the exact set of URLs within the site structure you are looking to do your test that has the desired on-page element.
In the example above, we would only look to find the blog pages with an h3 heading within the site structure.
What Causes Seasonality Outliers in SEO A/B Split Testing?
When you run an SEO split test before, during, or after a holiday, you may find that your test results have a traffic spike within them.
This is due to URL(s) associated with the holiday. This spike in traffic throws off the test results.
The key here is to identify the URL(s) in question, review them to see if they are seasonal due to the holiday and remove them from the test group.
SplitSignal allows you to filter your pages to exclude pages that are outliers. You can then re-run the test. If you still are seeing a spike of traffic, you will need to review your URL(s) further as there are additional pages that are outliers. Repeat the above process.
Seasonality Outlier Example:
In this example, we will show three tests of a global marketplace for creating unique designs and products with customization.
Test 1:
Timeframe: July 29th, 2021–Nov 2nd, 2021
Hypothesis: By adding personalized to the h1, we will increase clicks.
Analysis:
The test had a positive effect size of 30.7%.
Because the effect size was so large, we decided to rerun the test to ensure accuracy.
The test showed a positive change in total clicks within the variant group.
Test 2:
Timeframe: Oct 8th, 2021–Jan 12th, 2022.
Hypothesis: By removing personalized to the h1, we will increase clicks.
Analysis:
The test had a positive effect size of 8.6%.
The test showed a positive change in predicted clicks within the control group.
Our testing only changed because Test # 1 included the text: personalized to the h1 tag. In Test #2, we were removing it from the h1.
Why did we analyze both tests further?
- Both SEO A/B split tests had positive results.
- Test # 1 has positive results for the variant group.
- Test # 2 has positive results for the control group.
- There were significant holidays between both timeframes: Christmas, Hanukkah, and New Year.
Here is what we found:
Both split test groups had a positive test that led us to look for a URL with a larger amount of traffic in the split testing period, but not in the 100-day model prior to the test.
We were able to isolate a URL: /ornaments, and this URL received a larger number of clicks than other URLs within both groups, and it was holiday-related.
In addition, in Test # 1, it resided in the variant group, and in Test # 2, it resided in the control group.
Since the traffic irregularities happened in testing and not within the prior 100-day model, SplitSignal did not exclude the URL.
We manually excluded this URL and re-ran the test one more time.
Test 3:
Timeframe: Nov 10th, 2021–Feb 16th, 2022
Hypothesis: By removing personalized to the h1, we will increase clicks.
Additional Clicks
Pages Tested
*We manually removed the outlier URL before running this test.
URLs visited by Googlebot:
Analysis:
This test had a positive effect size of 3.7%.
While this test is still running, we can bring you these results as Googlebot has visited all of the 128 URLs in the variant group.
These test results have normalized now that the seasonal outlier URL has been removed.
We also know that the test is statistically significant because the confidence level is at 96%, and a confidence level of 95% or above is required for statistical significance.
Tests 1 and 2 were also statistically significant. However, their results pointed to additional factors that required further analysis to confirm those results. These factors ultimately allowed us to see the seasonal outlier URL. Tests # 1 and Test # 2 were skewed due to a seasonal outlier URL.
URL Exclusions That SplitSignal Automatically Removes
SplitSignal excludes URLs from the prior 100-day’s organic traffic:
- URLs With Traffic Inconsistency—URLs with inconsistent click traffic with the other URLs within the pre-test 100-day traffic model are excluded.
SplitSignal does a great job at handling traffic inconsistency exclusions, but occasionally URLs that are included do not have traffic inconsistencies in the 100-day pre-traffic model.
- Redirected URLs—URLs with redirects are excluded from inclusion.
- Not Found URLs—URLs that are not found are excluded from inclusion.
- Blocked URLs—URLs that are blocked through page meta robots or a robots.txt file are excluded from inclusion.
SplitSignal respects the directives from a robots.txt file and page meta robots as this is the same behavior that Google bot would have.
