We are proud to announce that we’ve implemented two new crawling features in the Site Audit tool: AJAX crawling and Sitemap links crawling.
You can now crawl your entire website without missing a piece of dynamically generated content or a URL, and immediately take action on technical issues.
This is a huge step in our tool development, since only a few tools on the market provide this functionality and now the SEMrush Site Audit tool is one of them.
Let’s get straight to the details.
Fortunately since 2015 Google started to render and understand the pages with dynamic content like modern browsers and eventually deprecated the old AJAX crawling scheme.
However, it’s still crucial for SEO professionals to conduct the full-scale analysis to check if Google sees these pages correctly. Now with the new AJAX crawling feature in Site Audit, you can track your dynamic content and your internal linking health without extra efforts!
To crawl your AJAX pages, you don’t have to change your settings or follow any special instructions, just re-run your campaign.
If you don’t have a project with Site Audit setup, head to the Projects dashboard, create a new project and set up the Site Audit tool. From now on, your AJAX pages will be crawled automatically.
With this update, you can now send our crawler directly to your sitemap file to make it crawl your website exactly in accordance with your sitemap the same way as you recommend Google to do it.
In order to crawl all links of your sitemap, first go to your project with Site Audit setup, or create a new one. Click on the gear wheel and select ‘Crawl sources’.
Now you can change your crawling settings — click on ‘Crawl source’ and select:
Finally, click on the ‘Save’ button and re-run your campaign. Your website is now crawled in accordance with your sitemap.
Here at SEMrush, we are sure that this new functionality will help you conduct the most complete SEO analysis of your website and not a single page will be left unnoticed by our crawlers. And this is one more step in making the Site Audit tool the best site crawler on market.
What do you think about this update? Share your thoughts and ideas at
[email protected], and stay tuned!