How SemrushBot Crawls Your Site
A bot, also known as a web robot, web spider or web crawler, is a software application designed to automatically perform simple and repetitive tasks in a more effective, structured, and concise manner than any human can ever do. The most common use of bots is in web spidering or web crawling.
SemrushBot is the search bot software that Semrush sends out to discover and collect new and updated web data. A crawl process starts with a list of webpage URLs. When SemrushBot visits these URLs, it saves hyperlinks from the page for further crawling. This list, also known as the "crawl frontier", is repeatedly visited according to a set of Semrush policies to effectively map a site for updates: content changes, new pages, and dead links.
Frequently asked questions
- Available historical data
- What is Live Update?
- What is Search Volume in Semrush?
- What is KD%?
- What Does Results Mean?
- What Does SERP Features Mean For My Keywords?
- Why Does No Data Show For My Keyword/Domain?
- What mobile databases are available on Semrush?
- How many databases does Semrush now have?
- How can I access more databases?
- How SemrushBot Crawls Your Site
- How is Keyword Intent calculated?
Back to Semrush Data & Metrics category