Before you start: if you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we’re suggesting you start here or request a demo of SplitSignal.
First, we asked our Twitter followers to vote:
This time the majority of our followers were RIGHT! Read further to find out why.
The Case Study
One important element that is often tested amongst our customers is the H1 tag, which is often used to provide a keyword-rich title for a web page. However, it’s important to note that changes to the H1 tag may also affect the title displayed in search engine results pages (SERPs).
In August 2021, Google announced an update to its algorithm for generating web page titles in the SERPs. Google’s documentation states that they are using text that humans can visually see when they arrive at a web page. Google considers the main visual title or headline showed on a page, content that site owners often place within <H1> tags or other heading tags, and content that’s large and prominent through the use of style treatments.
As an SEO, it is important to understand that the titles of web pages listed in the SERPs are one of the primary ways people determine which results may be relevant to their search query. Therefore, anything displayed in the SERP snippet can have a positive or negative impact on organic traffic to the website. It is important to monitor how the H1 tag changes impact the title displayed in SERP and how it affects the click-through rate (CTR) and, thus, organic traffic to the website.
Considering this, our customer, a travel website, decided to conduct a split test to validate the potential impact of adding additional departure location and destination keywords to the H1 tag on their SEO.
The Hypothesis
The travel website in question, conducted an analysis of user search behavior and developed a hypothesis that incorporating keywords pertaining to both the departure location and the destination in the H1 tag would result in an increase in organic traffic to the tested pages.
As we know, Google tends to use the <H1> heading to rewrite the title displayed in the SERPs. This means that changing the H1 heading, as part of this test, would likely also result in changes to the title displayed in the SERPs. It is important to consider this when interpreting and analyzing the results of the test, as it can have a significant impact on the click-through rate (CTR) and organic traffic to the website.
The Test
To evaluate the impact of adding additional keywords to their detail pages, our customer conducted a split test using the tool SplitSignal. A total of approximately 1,800 detail pages were selected as either control or variant.
The test was conducted over a period of 22 days, during which Googlebot visited around 94% of the tested pages. A script was used to track if the changes made to the H1 tags were being reflected in the titles displayed in the SERPs. This helped in monitoring the impact of changes on the click-through rate (CTR) and organic traffic to the website and confirming if the changes made to the H1 tags were being reflected in the titles displayed in the SERPs.
The Results
The results of the split test indicated that incorporating additional keywords into the H1 tag resulted in a notable increase in clicks, estimated at 8.3%. The analysis revealed that this increase was statistically significant at the 95% level, as demonstrated by the blue-shaded area falling above the x=0 axis in the analysis. Furthermore, by the end of the test, the results reached 99% significance.
These findings suggest that incorporating keywords pertaining to both departure location and destination in the H1 tag can have a positive impact on the organic traffic to the detail pages of the website. The statistically significant results indicate that this effect can confidently be attributed to the changes made to the H1 tags. Overall, these results provide strong evidence for the effectiveness of incorporating additional keywords into the H1 tag to improve the website’s SEO performance.
Note that we are not comparing the actual control group pages to our tested pages. but rather a forecast based on historical data. The model predicts the counterfactual response that would have occurred had no intervention occurred. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into the true impact of an SEO change.
Why
The results of the split test indicate that incorporating additional keywords pertaining to both departure location and destination in the H1 tag had a positive impact on the performance of the tested pages. By analyzing the data, we observed a significant increase in both clicks and impressions, indicating that the pages had become more relevant to the search queries for which they were ranked.
As expected, we also found that this change to the H1 tag had an impact on the title displayed in the SERPs. The combination of enhancing the H1 tag with relevant keywords people tend to use when searching and the corresponding change in the SERP title more closely aligned with the user’s search intent, resulting in increased traffic.
It’s important to keep in mind that changes made to headings, such as the H1 tag, can also affect the way the site is displayed in the SERP. Google tends to use the H1 heading to rewrite the title displayed in the SERPs, which means that changes to the H1 tag can have a significant impact on the click-through rate (CTR) and organic traffic to the website. Therefore, it is crucial to take into account the potential impact on the title displayed in the SERPs when making changes to the H1 tag and to thoroughly analyze the results of any changes to understand their impact fully.
Something that works for one website may not work for another. So if you feel inspired to start split testing, check out this step-by-step guide to test and analyze SEO changes (yourself). The only way to know for sure is to test what works for you!