Before you start: if you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we’re suggesting you start here or request a demo of SplitSignal.
First, we asked our Twitter followers to vote:
This time the majority of our followers were WRONG! Read further to find out why.
But first, let’s see what SEO professionals had to say about the result of this test:
Geoff Jackson, CEO of Clubnet Digital:
I am guessing it improved the CTR slightly however, I'm not sure it would necessarily be good for organic rankings if it just strengthens competing pages with the main category page.
— Geoff Jackson (blue tick) (@zigojacko) December 1, 2022
Joshua Jarvis, SEO Director & PPC Manager at 4rd Marketing:
I'd remove the image if you want to make this a hot take. Clearly, adding the category gives more context to a page when the page title is a brand or seemingly random collection of words. Adding context will almost always improve ecommerce performance in these situations.
— Joshua Jarvis (@jarvatar) December 1, 2022
Follow us on LinkedIn to share your thoughts on the next test.
The Case Study
Page titles are often the most prominent element a user sees in the search results. In search, you are always competing with other search results. Therefore, it is essential to write the most relevant page title. Finding the most optimal page title often requires a lot of testing and trade-offs between including relevant keywords and having search results that grab users’ attention to improve click-through rates (CTR).
Product pages on an ecommerce website are often the most important pages when it comes to monetization. Running tests on these types of pages to find the best possible answer for both search engines and users can be very rewarding when it comes to optimizing your organic performance and revenue. For a large ecommerce company, we added the name of the product category to the titles of their product pages to see if it made their search results more relevant and effective.
The Hypothesis
We hypothesized that adding the category name would add relevance as it gives users and search engines more insight into what the product type is. In addition, by adding the category name, we added (relevant) keywords related to the product, which we assumed could increase visibility for relevant searches.
We wanted to validate whether adding the product category name to the product page titles would have a positive effect on organic traffic, as shown in the example:
The Test
We used SplitSignal to change the page title for the tested product pages. In total, approximately 5,000 product pages were selected as either control or variant. We started the test and ran it for 21 days. We found that Googlebot visited 91% of the pages tested.
We had a script to track whether Google would show the desired page title changes for the tested pages. We were able to determine that Google displayed the changes in the search results.
The Results
Adding the category name to product page titles resulted in an estimated −8.2% loss in clicks to the tested pages.
Within a week, we were able to determine that the effect we saw was significant. When the blue shaded area performs below or above the x=0 axis, the test is statistically significant at the 95% level. This means that we can be confident that adding the product category name to the title of the product page will have a negative effect on organic traffic to the product pages of the website.
Note that we are not comparing the actual control group pages to our tested pages. but rather a forecast based on historical data. The model predicts the counterfactual response that would have occurred had no intervention taken place. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into the true impact of an SEO change.
Why
This test is a perfect example of why you should validate changes before deploying them to your website. Testing can be a great ‘safety net’ when it comes to changes that negatively affect your organic performance. Naturally, we wanted to dig deeper into why this led to a negative outcome.
We concluded that many page titles were getting too long, resulting in truncated snippets of search results. Since the website in question is a well-known brand, this may have contributed to the negative result of the test.
We also concluded that adding more descriptive information about the product type is probably unnecessary for both Google and users. We found that Google generally showed the same search results for queries like “bose QuietComfort 45 black” and “bose QuietComfort 45 black headphones,” meaning that Google already has a good understanding of what the product is and for which queries the product page is a relevant search result. In addition, users who perform these types of long-tail searches already know that they are searching for “headphones,” for example. This is why adding the category name doesn’t add any extra value to the title of the product page.
To further validate these findings, we ran the same test setup for multiple websites, all of which resulted in a negative result. Adding the category name to an online music store’s product page titles resulted in a −7.2% organic traffic loss. For another ecommerce website, the test resulted in a loss of −5.4%. This doesn't necessarily mean that this is a negative change by default, but it should definitely be tested before being rolled out on your website.
Keep in mind that something that works for one website may not work for another. The only way to know for sure is to test what works for you!