en
English Español Deutsch Français Italiano Português (Brasil) Русский 中文 日本語
Go to Blog
Erika Varagouli

An In-Depth Guide to Log File Analysis for SEO

...
Wow-Score
collecting now
The Wow-Score shows how engaging a blog post is. It is calculated based on the correlation between users’ active reading time, their scrolling speed and the article’s length.
Erika Varagouli
Log File Analysis for SEO

Your website’s log file records every single request made to your server, and analyzing this information can reveal insights about how search engines are crawling your site and its web pages.

In this guide, we’re going to take a deep dive into how to carry out a log file analysis and what it can be used for in SEO, specifically looking at:

What Is Log File Analysis?

Log file analysis is a technical SEO task that lets you see exactly how Googlebot (and other web crawlers and users) interacts with your website. A log file gives you valuable insights that can inform your SEO strategy or solve problems surrounding the crawling and indexing of your web pages. 

But before we look at the main insights you can gain by carrying out a log file analysis, let’s take a moment to understand what a log file is and a little more about the information that it contains.

Perform a Log File Analysis

with Semrush Log File Analyzer

Please specify a valid domain, e.g., www.example.com

What Is A Log File and What Information Does It Contain?

Your website’s log file is stored on your server and records information about the performed requests. 

Every time a user or bot visits a web page on your site, an entry is recorded in your log file for every resource that’s loaded. The log shows exactly how users, search engines, and other crawlers are interacting with your site.

Here’s an example of what a log file looks like:

Log File screenshot Image Credit: Fourth P

Within a log file, you’ll find data including:

  • The URL of the page or resource being requested 

  • The HTTP status code of the request 

  • The IP address of the request server

  • A timestamp of the hit (time and date)

  • The user agent making the request (e.g., Googlebot)

  • The method of the request (GET/POST)

You might also find the client IP, the time taken to download the resource, and the referrer are included too.

If you’re looking at a log file the first time, there’s no denying that it can be confusing. However, by understanding what log file analysis is used for and how to do it, you’ll be in a position to gain some truly valuable insights.

What Is Log File Analysis Used For in SEO?

There are several different insights that you can get from your site’s log file as an SEO, with some of the main ones you need to know about being:

  • How frequently Googlebot is crawling your site, and it’s most important pages (and whether they’re being crawled at all) and identifying pages that aren’t often crawled 

  • Identifying your most commonly crawled pages and folders 

  • Whether your site’s crawl budget is being wasted on irrelevant pages 

  • Find URLs with parameters that are being crawled unnecessarily 

  • If your site has moved over to mobile-first indexing

  • The specific status code served for each of your site’s pages and finding areas of concern

  • If a page is unnecessarily large or slow

  • Finding static resources that are being crawled too frequently

  • Finding frequently crawled redirect chains 

  • Spotting sudden increases or decreases in crawler activity 

How To Do a Log File Analysis

Now that we've taken a look at some of the insights that can be gained with log file analysis let's look at how to do it. 

You’ll need:

While you can convert a .log to a .csv simply by renaming the file, meaning that this can then be opened and analyzed in Excel or Google Sheets, using a dedicated tool makes the analysis easier and quicker. This means you can spend more time actioning solutions from any issues you find rather than interpreting the data manually.

That said, if you do want to run a manual analysis, you'll need to be familiar with advanced usage of either of these, including creating pivot tables. If you're not familiar with how to do this in Excel, you can read this guide or take a look at this one to learn how to do it in Google Sheets.

Alternatively, use the Semrush Log File Analyzer to avoid the need to learn how to do this (however, pivot tables come in handy for a whole host of tasks, and it's worth learning more about).

Where To Get Your Log File

Before you can analyze your site's log file, you need to obtain a copy of it. 

Log files are stored on your web server, and you'll need access to this to download a copy. If you don't have this level of access, speak with your web developer or IT team, ask them to either provide it or share a copy of the log file. 

To access the log file, you'll need to use either your server control panels' file manager, via the command line or using an FTP client (if you've not already got one, Filezilla is free and comes recommended).

We'll assume you're accessing your server via FTP, given that this is most common.

Once you've connected to the server, you will need to navigate to the location of the server log file. Common server setups can find this in the following locations:

  • Apache: /var/log/access_log
  • Nginx: logs/access.log
  • IIS: %SystemDrive%\inetpub\logs\LogFiles

However, it's important to be aware that retrieving your site's log file isn't always so simple, and common challenges that can be faced include:

  • Finding that log files have been disabled by a server admin and are not available
  • Clients or other internal teams unwilling to provide log files or access to retrieve them
  • Huge file sizes
  • Log files that only store recent data (based either on a number of days or hits)
  • Issues caused by CDNs
  • Custom formats

That said, these issues all have solutions that can usually be worked on alongside a developer or server admin.

Introducing the Semrush Log File Analyzer

The Semrush Log File Analyzer is an ideal way to gain an insight into how search engines are crawling your site without having extensive experience manually auditing these. 

We've already shared how it can be complex and confusing for those who have never done a log file analysis before. However, our tool can help you to get the same insights simply and straightforwardly. 

In fact, we recommend using the tool for the following reasons: 

Analyzing a log file manually is tiresome. Unless you’re highly trained in technical website analysis, it can be an arduous task that leaves you cross-eyed and confused. If you want the quickest way to read an access log and understand how bots from Google interact with your website, this is the tool for you.

Using the tool makes log file analysis quick and easy to understand, and here's a simple and straightforward step-by-step process:

1. Make Sure Your Log File Is in the Correct Format

Before using the tool, you'll need to make sure that your log file is in the correct format; that's the proper access.log format. The tool also supports W3C, Kinsta, and Combined log format variations. 

The proper file format is “Combined Log Format," and it uses the following structure:

%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"

Where:

  • h — the host / IP address from which the request was made to the server
  • l — client id, usually stays blank (represented by a hyphen (-) in the file)
  • u — username, usually stays blank (represented by a hyphen (-) in the file)
  • t — the time and time zone of the request to server
  • r — the type of the request, its content, and version
  • s — the HTTP status code
  • b — the size of the object requested (in bytes)
  • Referer — the URL source of the request (previous page) often stays blank (represented by a hyphen (-) in the file)
  • User-Agent — the HTTP header containing information about the request (client application, language, etc.)

There's also a 1GB maximum upload size you need to be aware of. 

2. Upload Your Log File to the Tool

Semrush Log File Analyzer

You can either drag and drop your log file(s) into the tool or browse your machine for the file to get ready to run the analysis. 

Please make sure that your log files do not contain any personal data as recommended at this stage. It's important to note that log file analysis for SEO will only require GET data and not POST data (sensitive data can be found within this).

3. Start The Log File Analyzer

start log file analyzer

Once your log file has been uploaded, either add another file (in instances where the data you want to analyze is split across two or more files) or hit 'Start Log File Analyzer' to begin the analysis. 

There may be a short wait while the tool runs, depending upon the size of the file.

4. Analyze Your Log File Data

Once your log file has been analyzed, you'll see two main reports within the tool:

  • Googlebot Activity
  • Hits by Pages

Looking first at Googlebot Activity, you can gain insights into the daily number of hits for different bots (bots), see the breakdown of different status codes (Status Codes), and view the frequency that different file types have been requested (File Type).

Bots:

bots

Status Codes:

status codes

File Types:

file types

You can use these insights to understand:

  • How many requests different search engine bots are making to your site each day (Bots)
  • The breakdown of different HTTP status codes found per day (Status Codes)
  • A breakdown of the different file types crawled each day (File Types)

These insights are sitewide, but to gain specific information about how Googlebot and other crawlers are accessing your sites' content, look at the Hits by Pages report. 

hits by pages

Here you'll see insights for specific pages (Filter by path), see which of your site's pages or folders has the most or least bot hits (sort by the Bot Hits column), or see which are crawled most frequently.


Using the tool makes log file analysis simple and straightforward. There's no reason why SEOs need to manually analyze their log file when they can use a tool to speed up the process. 

Spend time optimizing your site, not analyzing data.

Perform a Log File Analysis

with Semrush Log File Analyzer

Please specify a valid domain, e.g., www.example.com

Erika Varagouli
SEMrush

SEMrush employee.

I create local and global content marketing strategies here at SEMrush. I started my career as a journalist, but gradually the world of SEO and content marketing won me over. I am obsessed with creating content people love, Nick Cave, and Italian films. Fueled by caffeine.
Send feedback
Your feedback must contain at least 3 words (10 characters).

We will only use this email to respond to you on your feedback. Privacy Policy

Thank you for your feedback!