Submit a post
Go to Blog

Step-By-Step Guide: SEO Pruning with SEMrush

82
Wow-Score
The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.
Learn more

Step-By-Step Guide: SEO Pruning with SEMrush

Kevin Indig
Step-By-Step Guide: SEO Pruning with SEMrush

Sometimes you make something better not by adding to it, but by taking away.

When I got into the SEO game about 10 years ago, I remember being taught that it is important to pump as many pages into Google’s index as possible. That really, really doesn’t work anymore nowadays.

Panda and other Google algorithms have taught us that quality - not quantity - is the name of the game. An effective approach to keeping quality high is “SEO pruning”: cutting off or editing underperforming pages to make a site stronger. You would rather have little but great content.

That’s what SEO pruning is all about.

The Idea Behind SEO Pruning

The concept of pruning is taken from Bonsai trees. They tend to grow to the top, which means the outer branches grow out of proportion, while the inner branches get crummy and die off. Pruning can get a Bonsai tree into the desired shape by making sure growth is distributed more equally.

The same applies to websites and SEO performance.

Most sites follow a power curve: a small number of pages gets most of the traffic. That in itself is not bad, but it results in a long-tail of underperforming pages that don’t get enough “light”. They are the crummy branches. What you want instead is a high ratio of indexed pages vs. pages with top 10 rankings. Sites with a low amount of underperforming pages seem to get a sort of extra boost.

We don’t know what the exact mechanics behind pruning are, but it seems to be a mix of optimizing crawl budget, quality score, and user experience. The original idea of a quality score comes from paid search, but the concept also exists in relation to a site’s link graph or brand combination searches. For content, Google seems to measure the quality of all pages in its index and accrue it to a "domain quality score”.

Now, the question is how can you prune your site and what tools should you use?

Let’s take a look!

How to Do SEO pruning with SEMrush

The goal of this step-by-step guide is to create a spreadsheet that tells you which URLs are underperforming, so you can prune them. I am using one of my personal projects in Germany to lead through the tutorial.

spreadsheet-finished-overview.jpegWhat the final result should look like. Hat tip to conditional formatting.

Before we get started, make sure you connected Search Console and Google Analytics (or your web analytics tool of choice) to SEMrush and have a project set up for the domain you want to prune.

Step 1: Crawl Your Site with “Site Audit”

First, we need to crawl our site, so go to “Site Audit” under “Projects” and get one started.

semrush-site-audit.pngThe “Site Audit” feature in SEMrush: Dashboard -> Projects -> Site Audit.

Once the crawl ran through, you get a report that lets you export the crawled pages (on “crawled pages”, click on the export button in the top right).

semrush-site-audit-crawled-pages.jpegCrawled pages report in the Site Audit feature.

Export and save the data.

Step 2: Extract User Behavior and Backlink Data

Now we have a basic index of pages, but we still need the data to assess which ones to prune. We can use many different criteria to assess the performance of a URL:

  • Organic entrances
  • % bounced entrances / bounce rate
  • Total pageviews/sessions
  • Number + quality of backlinks
  • Crawl frequency 
  • # of ranking keywords
  • CTR
  • Backlinks
  • Social shares

In my example, I used # of ranking keywords, backlinks (from SEMrush and Search Console), traffic, user signals. “Organic Traffic Insights” has all of that (can be found in the projects section).

semrush-organic-traffic-insights.jpegThe “Organic Traffic Insights” report in SEMrush.

Don’t forget to define the time range before you export. By default, it is set to 7 days, but that is not very representative. Instead, choose the last 3-6 months.

semrush-organic-traffic-insights-time-range.pngPick the right time range for the data export.

What you get out should look something like this:

semrush-organic-traffic-insights-export-overview.jpegExport from “Organic Traffic Insights” in SEMrush

What is nice is that the export gives you get average user behavior metrics across your site to benchmark. Plus, below the URL overview, you get all ranking keywords for each URL*.

*(With the latter you can create a pivot table and check whether several URLs rank for the same keyword to assess potential keyword cannibalization. But that is a topic for another blog article.)

We also need to export backlinks, so head over to “Backlink Analytics” -> “Backlinks” and export ‘em all. Just make sure you exclude lost backlinks before the export, otherwise, your list will be noisy.

semrush-backlinks.png“Backlinks” in SEMrush

Additionally, you can add backlinks from any tool you like, e.g., Search Console. The goal is to get a number of backlinks per URL, or a backlink quality metric.

In Search Console, just click on “Links” -> “Top linked pages - externally” and export that list.

search-console-links.jpegExternal links report in Google Search Console: Status -> Links -> Top linked pages - externally.

Once you downloaded all the data, import the CSVs into the spreadsheet from the crawl in step 1. Then, use VLOOKUP to group all data per URL. Create a pivot table for the backlink data, so you can group the sum (or average strength if you use a proprietary metric) of backlinks per URL.

You should now have a list of URLs with information about rankings, traffic, and user behavior performance.

I indicated which data comes from SEMrush (orange) and which from Search Console (grey) in my spreadsheet:

pruning-spreadsheet-tools.jpegThe final pruning table sorted after links from search console.

Step 3: Identify Underperforming Pages

From here on it’s all Excel magic.

Apply conditional formatting to each column and sort the table after traffic to see why certain URLs are not performing as well. Sometimes, a page has a high bounce rate, other times not enough links, or it was never optimized for a keyword.

What you want to do is take the whole list and group URLs into three buckets.

Performers (do nothing)

The first bucket is for pages that are performing well. They rank (well) for keywords, bring in traffic, have good user signals and backlinks. So, you obviously don’t want to prune those. Pages that cannot be removed also fall into this bucket, like homepage, Terms of Service, feature landing pages, etc.

Slight Underperformers (edit)

Slight underperformers bring in some traffic but don’t rank for any keyword in the top 10. They could get on the first page on Google with some optimization. They might also have brought in a lot of traffic once but dropped in rankings over time.

Definite Underperformers (merge or redirect)

The last group is reserved for pages that never got any traffic or backlinks. They are sometimes duplicates or overlap with other content, e.g., two articles targeting the same topic/keyword. Often, they are just pages that are part of the infrastructure and have never been optimized for a (good) keyword, e.g., archive, author, tag pages.

Pages that often fall into this bucket are:

  • Author/tag/archive pages
  • Blog articles
  • Hub pages (from topic clusters)
  • Empty inventory (products that aren’t available anymore)

It is best to redirect those pages either to another one that is similar (maybe targets the same keyword) or to the next higher page in the hierarchy (could be the blog overview page or a category page). You can also merge the content with another page, depending on how well that is feasible.

Step 4: Determine What to Do with Underperformers

Pruning doesn’t mean deleting URLs. You have a couple of options for what to do with the 2nd and 3rd bucket:

  • Merge the content of two or more URLs into one. Make sure to 301-redirect the old URLs to the new one.
  • Redirect a URL to another or the page next higher in the hierarchy. 
  • Edit/rework/rewrite/update the content.

Start by setting all URLs in the underperformer bucket to meta=noindex. You might not be able to change all pages, for example when it is an author page that you need people to be able to click on. But you can prevent Google from indexing it, which already helps.

Then, go through the list of underperformers (bucket 3) and decide what to do with each of the pages you can change (merge, redirect, or edit).

Only then would I start working on the second bucket, the slight underperformers. In some cases, it might not even be necessary to prune slight underperformers.

Step 5: Release Changes in Stages

At this point, you just have to make the changes and roll them out.

I recommend releasing in stages: start with underperformers and wait for a bit (3-6 weeks). See what happens and make a data-informed decision. If the change is positive for your site, you can try to continue with the slight underperformers.

You don’t want to prune too much at once. Try to find the optimum, instead.

What results can you achieve with pruning?

One site we pruned at Atlassian gained significantly in top 20 rankings, and we gained roughly 25% in organic traffic about a month after we pruned.

atlassian-site-pruning-page-1-and-2.jpeg# of keywords over the last 12 months in SEMrush.

How often should you prune?

I recommend 1-2x/year; otherwise, you risk “over pruning”. It is like cutting a tree too short. Sometimes, the results need a bit of time to show. It often depends on the size of the site; a larger site will show results faster.

Who should prune?

Everybody, but your site shouldn’t be “too small”. If you have 100 pages, it probably doesn’t make sense, but you can start thinking about it from 1,000 pages on. Keep in mind that these are arbitrary values that I’m providing to give a rough idea of what “small” and “large” is. You need some content to prune. You can’t cut branches off that aren’t there, yet.

What else can you prune?

You can (and should) prune backlinks and inventory after the same principle: disavow spammy backlinks and get rid of products nobody buys to improve the overall quality of your profile. It is part of keeping things in order.

Sometimes, less is more.

Like this post? Follow us on RSS and read more interesting posts:

RSS
I run technical SEO @ Atlassian in the Silicon Valley and mentor start-ups in and outside the German Accelerator. I'm very passionate about technical marketing, data crunching, and critical thinking.
Share this post
or

Comments

2000 symbols remain
Thank you for that article Kevin, I'm actually trying to prune some of the underperforming articles, and when I tried to export via the Organic traffic insights, I've noticed that I can export maximum of 50 URLs (even less if I have the Pro pack). Plus there's no URL filtering option in that view (OTI). Is it possible this method is more relevant to smaller scale website? (I'm talking about 200-300 aritcle pages I wanted to inspect).
Thanks again!
Kevin Indig
Shachar Blum
Hey Shachar,

the method should be doable for sites with thousands of pages.

Did you choose "Landing pages / SEMrush" or the other option? It worked beyond 50 URLS for me.

Cheers,
Kevin
Jack F
SEM rush now gives you 'Content Sets' which recommend which pages to remove/rewrite etc when you perform a 'Content Audit'. How accurate do you think these suggestions are? Its criteria are if a page gets less than 15 pageviews in 30 days, and the page was published 2 years or more ago. My site gets a lot of traffic but only has around 280 pages. It's suggesting that about 70 pages need to be rewritten or removed. A lot of them are pretty standard articles, not particularly spammy or anything. What are your thoughts on this? Many thanks!
Kevin Indig
Jack F
Hey Jack, I honestly have to play around more with content sets to give you a good answer.

> Its criteria are if a page gets less than 15 pageviews in 30 days, and the page was published 2 years or more ago.

I like that idea and think it applies to almost every site. However, for more nuanced recommendations these values have to be customized to the site.

> My site gets a lot of traffic but only has around 280 pages. It's suggesting that about 70 pages need to be rewritten or removed. A lot of them are pretty standard articles, not particularly spammy or anything. What are your thoughts on this?

Hard to say without looking at the site and data. How is the engagement for the 70 articles?
Jack F
Kevin Indig
Many thanks for your response.
Most of these pages have about 1-10 sessions a month. Those visits often have 0 for session duration, but some will have a few minutes. A lot of the pages have one or two backlinks from articles on industry sites or Quora, but I'm not sure if these backlinks have any value if they only send maybe one visitor a month...
A bit hair-raising culling pages as I don't want ranking to suddenly take a nosedive! lol
Any thoughts welcomed, thanks
Kevin Indig
Jack F
I don't think you're in immediate danger to drop in rankings/traffic, but I'd still recommend to work on these articles over time.
Best Women’s CrossFit Shoes
Really excellent stuff... The first opportunity you only mentioned that it would be important to add a response to the question, then for the second one you emphasized that not only should we respond to the question thoroughly, but we should also provide a link to an article on our blog that is an even better response to the question! Excellent idea!
Solomon
I'm only learning this backlink pruning strategy for the first time, although I don't know how effective it would be since I haven't tried it, but it was a really comprehensive and well explanatory post, I really find it useful in this season of constant change in SEO
Nikola Roza
SEO prunning is interesting, it makes sense that it would work since quality is the name of the game now.

When you say that big sites get results faster, did you mean to say that since they get traffic traffic, it's easier to see what happens to it after pruning?

Also, will pruning a site that gets little traffic, and that only has few pages that are weighing it down, help it pick up steam quicker?

Thanks Kevin:)
Kevin Indig
Nikola Roza
Hey Nikola,

> When you say that big sites get results faster, did you mean to say that since they get traffic, it's easier to see what happens to it after pruning?

I mean that you seem to need a certain size for pruning to be effective. That could have something to do with crawl budget or another threshold Google measures that we're not aware of. If your blog has only 50 pages, for example, the effects might not be as strong as if you had 500 pages. It doesn't mean that pruning doesn't work at all, just that the effects are less significant.

> Also, will pruning a site that gets little traffic, and that only has few pages that are weighing it down, help it pick up steam quicker?

Do you mean a site that "only has a few pages" or a site that "has a few underperforming pages"?

Cheers,
Kevin
Nikola Roza
Kevin Indig
Hi Kevin,
Thanks for your helpful comment.

I guessed that having a decent sized website opens it up to all potential benefits of SEO pruning, but that it also works on smaller websites.

> Do you mean a site that "only has a few pages" or a site that "has a few under-performing pages"?

I meant a site that has a few pages (not under-performers since they all are- the site is barely getting any traffic).
I ask you this because I recently listened to Brian Dean on OkDork podcast, and he said that one of the reasons Backlinko was able to compete with the giants is that he kept his site lean and that every link he got counted way more for him that it would to, for example, Moz (they have over 60 000 indexed pages)

To sum it up:
is it worth to prune a very small site while building high-quality links?

Will this concentrated link equity help the site rank faster and higher than it normally would (for a site that is not pruned, when authority is spread across both useful and dead-weight pages)?
Thanks for your reply,
Cheers!

Nikola Roza
Kevin Indig
Nikola Roza
Hey Nikola,

you're welcome. I think what Brian Dean does with his site is pretty much the same pruning. He creates few but super high-quality and long-form content. I don't think there's much to prune on his site because he keeps his quality so high.

> Will this concentrated link equity help the site rank faster and higher than it normally would (for a site that is not pruned, when authority is spread across both useful and dead-weight pages)?

I think the concentration of link equity on a site with few but high-quality pages plays a role but it's not the only reason the site ranks so well. The good content is more important, I'd say.

Cheers,
Kevin
White Rabbit
This is a great post. Thanks Kevin!
Rajeev Dave
Hi Kevin

Thanks for sharing informative Tips for Step-By-Step Guide: SEO.

Awesome
I am a new user of this tool. just love this.
Thanks for the wonderful Article
Dylan Howell
Big site w/ only recently fixed crawl issues (js).

We put out quite a bit of new content every day (1000 product pages, user generated).

So far in my crawl, 80% of the pages on the site have no search console data (zero impressions).
Only ~3.5% of the pages have 1 or more click in the past 90 days.

Would you start with just pruning the bottom 80% don't even have impressions? Then get the remaining 16.5% after watching for a few weeks?

I'm sure it will take weeks for the pages to fall out of the index anyways due to the number and crawl restraints.
Kevin Indig
Dylan Howell
Yes, I would start with the 80% and set them to meta=noindex. Also, monitor your crawl rate. You might want to build a logic into your site that automatically sets a new post to noindex until there's some more activity on it.
Once you set the posts to noindex, try putting them all in an XML sitemap and ping it to Google. don't keep it long-term, just unti Google crawled the pages.
Really great article on off-page SEO Technique bro Thanks for sharing my question is about backlinks how many backlinks i should create if 1 keyword difficulty is 2 ?
Kevin, I have client who runs a recipe and food blog. I've talked to her about pruning, but she thinks more is better. In some sense I agree, more recipes, even underperforming ones, give the site more potential for traffic. Thoughts?
Kevin Indig
James Shea
Hey James, good point! You can prune a site by leaving content that's important but doesn't perform as well online but setting it to meta=noindex. It very much seems like Google counts pages that are indexed - not crawled - for the overall quality score.

There is also the lens you look through: if a recipe is underperforming in terms of traffic, it's a strong point for fully pruning it. If it's just underperforming in terms of rankings/organic traffic but gets lot's of shares, for example, then meta=noindex might be a good try. The good thing about noindexing is that it's reversible.

Hope that helps,
Kevin
Shehryar  Aziz
James Shea
One thought; why not edit the recipes that are underperforming?

If it was my site, I must have given it a try: before pruning or no-indexing:

- Do an effective keyword research for the underperforming recipes.
- List down the perfect keywords that you think can get you traffic.
- Re-edit the content, make them better, better than the competitors.
- Add the keywords and publish it.

Re-editing the old content can give you a ranking boost.
Kathleen Garvin
This is such a helpful article. Working on our own pruning project now and resources like this are invaluable. Are you able to disclose the amount and/or percentage of content you pruned that produced the results shown in the chart with the first batch of pruning? Curious about the amount you started with. Thanks!!
Kevin Indig
Kathleen Garvin
Thank you, Kathleen! We pruned about 30% in the first batch. The following batches were much smaller.

Good luck,
Kevin
Konnie De Oliveira
This is awesome! I've only in SEO for about a year and a half now, and while I have finding quality links down, I need to focus more on the pruning aspect. This article is an amazing help! Thanks for sharing!!
Kevin Indig
Konnie De Oliveira
Thanks, Konnie! Happy you like it :-).
SEO Pruning new tools for the toolbox of tricks for SEO/SEM Tools, I always wonder how to clean up old Backlinks that has expired domain links also pages with no SEO impact. I run a digital agency [link removed by moderator] and I'm looking to test some of these pruning tools for my much older clients.
Harsh Agrawal
Hey Kevin
That's an interesting read and I'm actually in the midst of content audit and pruning for [link removed by moderator]. I was doing it category wise and it required quite a manual work. Still got to learn VLOOKUP to group all data per URL.
Thanks for the guide.
Kevin Indig
Harsh Agrawal
Thanks, Harsh! You should find VLOOKUP tutorials all over the net ;-).
Anand Joshi
I am a new user of semrush. I have learned many things from your blog. thank you so much for sharing
Gian
Thank you for sharing!
Kevin Indig
Gian
Thanks for reading ;-)
C. Alex Velazquez
Nice share. Super valuable stuff here! ;)
Kevin Indig
C. Alex Velazquez
Thank you C. Alex! Happy it's useful to you ;-).
Bais Naqvi
Hi, Kevin Indig i'm very thankful to you for uploading a valuable blog post. i have learn more knowledge for the help of your bloc post. keep it up ...
Kevin Indig
Bais Naqvi
Thank you, Bais!

Send feedback

Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!