Why Is SEMrush Crawling More Pages Than My Sitemap? Question

When conducting a Site Audit project, you may find that our user agent may crawl more pages than what is displayed within your sitemap. This can be due to a couple of reasons:

  1. You did not include all of your webpages within your sitemap
  2. You have certain webpages publicly available that were intended to be private

Our user agent crawls websites based on what pages are made publicly available. We do not use sitemaps as a determinant for pages to crawl. If your Site Audit project is crawling more pages than your sitemap, you may need to double check with your webmaster to see if any pages you intended to be private are not available publicly.

It’s possible you created pages on your website previously and forgot to add them to your sitemap. If there is a certain page you would like to allow or disallow from your Site Audit project, please take a look at our Configuration manual.

Log in

By clicking on "Create my account", you are agreeing to our Terms of Use.