Site Audit: The New User Agent Feature

Today we're happy to announce a minor, yet extremely important update to our Site Audit tool - a new feature that allows users to crawl their website with either of the two new user agents: Googlebot mobile and Googlebot desktop. This is particularly helpful if the website you're trying to audit blocks the default SEMrush crawler, or if you want to see if your website handles requests from Google's crawlers properly. In other words, you can now see what issues the Google search engine finds with your website and resolve them to improve your search ranking!

Please note that this is only a mimic of the actual Googlebot, designed to help you see how your website responds to it specifically.

Changing the user agent is easy and only takes five simple steps: First click on the gear icon in the top-right corner of the Site Audit campaign interface(1) and select the 'User Agent' line in the dropdown menu(2). Alternatively, you can also click the agent icon, as shown in the picture below.

Then, simply select any user agent you like from the menu. Finally, click on the "Save" button, and that's it.

In order for your changes to take effect, please re-run your Site Audit by clicking the corresponding button in the interface.

SEMrush will now use the user agent you have chosen to crawl your website. You can always switch back to using the previous user agent in the Site Audit settings, as shown in the example above.

Please let us know what you think of this update by dropping us a line at [email protected]. Your feedback is most appreciated!