logo-small
Features Prices
News 0
Latest News See All

Temporarily unavailable. Please come back later.

See All
Webinars 0
Upcoming Webinars See All
Upcoming Webinars

Sorry, we could not find any upcoming webinars.

See recorded webinars
Blog 0
Recent Posts See All

Temporarily unavailable. Please come back later.

See All
Jamie White

Does Google Crawl rel:"nofollow" at All?

Jamie White
Does Google Crawl rel:"nofollow" at All?

Adding a rel=”nofollow” tag onto internal links used to be very common practice back in the ‘dark days’ of SEO where directory links, article spinning and keyword-rich anchor texts were all the rage. The nofollow tag could be used to prevent Google crawling and passing PageRank through to the pages on a site that weren’t central to SEO performance, and in turn ensuring that more valuable pages received a greater portion of the authority.

‘PageRank sculpting’ is a term that was coined to describe the process of manipulating the way PageRank would be distributed throughout a site. The purpose of this manipulation was to guarantee that an SEO campaign’s most valuable landing pages received the greatest portion of link authority. However, in around 2008/2009, Google clamped down on this by changing the way that PageRank was distributed to links on a page.

Pre-change: In very basic terms, if you had ten links on a page, then each link would receive one-tenth of that page’s link authority. If you applied the nofollow tag to five of these links, then the remaining five would instead receive one-fifth of the link authority (put another way, the authority of the nofollow links would be redistributed to the remaining links)

Post-change: This is no longer the case, as any link authority going to nofollow links is simply lost and not repurposed.

This change completely killed the process of PageRank sculpting, as webmasters could no longer control the flow of link authority through their website by using the nofollow tag. As a result, it would be easy to think that there’s little to no reason for using the nofollow tag internally.

Hypothesis

The nofollow tag was always used as a method to say to Google: “Don’t pass link authority through this link,” and that’s why it became such a powerful tool to use internally. However, Google has never been clear about whether it even crawls the link to discover new content. If it doesn’t, then this could mean that rel=nofollow is still an awesomely powerful tool to use internally in order to maximise SEO performance.

Google’s official stance on the matter is this:

Google nofollow link description

But that’s pretty ambiguous – it doesn’t actually say whether content linked with a nofollow tag will be discovered.

On top of this, Rand Fishkin shared his opinion in a Moz Whiteboard Friday that Google does crawl these links:

Google shares some nofollow links

Google may share nofollow links

We were intrigued, so decided to conduct a little experiment. Here at Search Laboratory we are always putting SEO theory to the test to ensure that our clients receive the best possible recommendations in order to drive increased organic search performance.

Methodology

Our test started with building two brand new websites using simple WordPress installs. We then analysed their crawl logs to determine whether content behind nofollow links could be found by search engines. The following method was used on two sites in order to gain a bigger sample size and statistically significant results:

  1. Block the site to search engine crawlers prior to commencing the test. We didn’t want any pre-test crawl activity skewing our results
  2. Set up three pages – the homepage, a second page (Page B) and a third page (Page C)
  3. Build a link from the homepage to Page B, and apply the rel=”nofollow” tag
  4. Build a link from the homepage to Page C, without the rel=”nofollow” tag (this was done to ensure that Google was actually crawling further than the homepage)
  5. Unblock the site and open it up to search engine crawlers:
    1. Remove the Disallow: / command from robots.txt
    2. Use the Fetch As Google command in Google Search Console – selecting ‘Crawl only this URL’ to make sure that Google would find the nofollow link naturally and without us helping it

Google nofollow test

  1. Monitor Googlebot activity on the site using raw access log files

Results

After ten days of crawling, we saw no crawl activity on Page B despite numerous hits on the robots.txt file, homepage and Page C. We were a little sceptical that Google might have simply not detected the link to Page B, so we went back into Google Search Console to use ‘Fetch As Google’ once again. This time we told Google to ‘Crawl this URL and its direct links,’ leaving no stone unturned:

Google test results

We then gathered a month’s worth of crawl data to truly determine whether Google was crawling our nofollow linked content:

SITE 1

URL Crawl instances 
robots.txt 113
Homepage 69
/page-b/ 0
/page-c/ 8

 

SITE 2

URL Crawl instances 
robots.txt 75
Homepage 39
/page-b/ 0
/page-c/ 4

 

Conclusion

What we have drawn from this test is that Google will completely honour the nofollow tag when used internally and will not use it to discover content. This has huge implications for webmasters, as they can use this functionality to moderate crawl budget within their site. This is especially relevant for websites that utilize faceted navigation, something that can be a real Pandora’s Box of crawling and content discovery if not handled correctly.

Do you utilize nofollow tags on your website? Let us know what you think about it given our new findings.

Jamie White is the Head of Technical SEO at Search Laboratory. Jamie has worked in SEO for over six years in a number of top UK digital agencies, and joined Search Laboratory in 2013. Follow him @jammy8891 on Twitter.

Comments

2000 symbols remain
Eli Schwartz
Great test! What about the bias that might come from one domain? Have you tried on authoritative domains?
Jamie White
Jamie White
Eli Schwartz
Hi Eli

We haven't tried this on any authoritative domains yet but we are looking to expand this test to try and verify our results. I personally am pretty reassured by the results we've received so far, especially as the second internal page (/page-c/) was crawled a handful of times despite being linked-to in exactly the same way as /page-b/ but simply without the nofollow tag
Alistair Lattimore
Alistair Lattimore
For those that are interested in more of this type of research, I put together a series of mobile SEO tests to understand how Googlebot, Googlebot (smartphone) and dynamic serving are handled at http://www.convergentmedia.co/.... I was surprised by the findings, if you've got questions about the results - let me know - happy to answer them.
Jim Hodson
Jim Hodson
I have heard Matt Cutts state emphatically on numerous occasions at Pubcon and SMX that Google does NOT crawl links that are flagged NOFOLLOW. Nice to see someone run a little test to prove it.
Rich
Rich
Does this become a moot point if the page that's at the receiving end of a "nofollow" internal link gets a follow link from an external source? And what happens when someone else in the company forgets to use nofollow when they link to that page from some piece of content they create down the road?
Jim Hodson
Jim Hodson
Rich
You are correct Rich. This is EXACTLY why Google has always told webmasters that NOFOLLOW is NOT a reliable method to prevent a URL from being crawled/indexed. Robots.txt is equally unreliable to prevent a URL from being shown in the SERPs as Google can still show a blocked/disallowed URL in the SERPs if they think (based on inbound links) that it is relevant to the query.
Jamie White
Jamie White
Rich
Hi Rich

That's right. If the page is discovered via a regular (followed) link then Google will crawl this as normal and will likely index the content.

Same applies to your second point, any regular internal link without the nofollow tag would result in that content being crawled.
Have a Suggestion?