Your website’s log file records every single request made to your server, and analyzing this information can reveal insights about how search engines are crawling your site and its web pages.
In this guide, we’re going to take a deep dive into how to carry out a log file analysis and what it can be used for in SEO, specifically looking at:
What Is Log File Analysis? What Is Log File Analysis Used For in SEO? How To Do a Log File AnalysisWhat Is Log File Analysis?
Log file analysis is a technical SEO task that lets you see exactly how Googlebot (and other web crawlers and users) interacts with your website. A log file gives you valuable insights that can inform your SEO strategy or solve problems surrounding the crawling and indexing of your web pages.
But before we look at the main insights you can gain by carrying out a log file analysis, let’s take a moment to understand what a log file is and a little more about the information that it contains.What Is A Log File and What Information Does It Contain?
Your website’s log file is stored on your server and records information about the performed requests.
Every time a user or bot visits a web page on your site, an entry is recorded in your log file for every resource that’s loaded. The log shows exactly how users, search engines, and other crawlers are interacting with your site.
Here’s an example of what a log file looks like:
Image Credit: Fourth PWithin a log file, you’ll find data including:
The URL of the page or resource being requested
The HTTP status code of the request
The IP address of the request server
A timestamp of the hit (time and date)
The user agent making the request (e.g., Googlebot)
The method of the request (GET/POST)
You might also find the client IP, the time taken to download the resource, and the referrer are included too.
If you’re looking at a log file the first time, there’s no denying that it can be confusing. However, by understanding what log file analysis is used for and how to do it, you’ll be in a position to gain some truly valuable insights.
What Is Log File Analysis Used For in SEO?
There are several different insights that you can get from your site’s log file as an SEO, with some of the main ones you need to know about being:
How frequently Googlebot is crawling your site, and it’s most important pages (and whether they’re being crawled at all) and identifying pages that aren’t often crawled
Identifying your most commonly crawled pages and folders
Whether your site’s crawl budget is being wasted on irrelevant pages
Find URLs with parameters that are being crawled unnecessarily
If your site has moved over to mobile-first indexing
The specific status code served for each of your site’s pages and finding areas of concern
If a page is unnecessarily large or slow
Finding static resources that are being crawled too frequently
Finding frequently crawled redirect chains
Spotting sudden increases or decreases in crawler activity
How To Do a Log File Analysis
Now that we've taken a look at some of the insights that can be gained with log file analysis let's look at how to do it.
You’ll need:
Your website's server log file
Access to the Semrush Log File Analyzer
While you can convert a .log to a .csv simply by renaming the file, meaning that this can then be opened and analyzed in Excel or Google Sheets, using a dedicated tool makes the analysis easier and quicker. This means you can spend more time actioning solutions from any issues you find rather than interpreting the data manually.
That said, if you do want to run a manual analysis, you'll need to be familiar with advanced usage of either of these, including creating pivot tables. If you're not familiar with how to do this in Excel, you can read this guide or take a look at this one to learn how to do it in Google Sheets.
Alternatively, use the Semrush Log File Analyzer to avoid the need to learn how to do this (however, pivot tables come in handy for a whole host of tasks, and it's worth learning more about).
Where To Get Your Log File
Before you can analyze your site's log file, you need to obtain a copy of it.
Log files are stored on your web server, and you'll need access to this to download a copy. If you don't have this level of access, speak with your web developer or IT team, ask them to either provide it or share a copy of the log file.
To access the log file, you'll need to use either your server control panels' file manager, via the command line or using an FTP client (if you've not already got one, Filezilla is free and comes recommended).
We'll assume you're accessing your server via FTP, given that this is most common.
Once you've connected to the server, you will need to navigate to the location of the server log file. Common server setups can find this in the following locations:
Apache: /var/log/access_log Nginx: logs/access.log IIS: %SystemDrive%\inetpub\logs\LogFilesHowever, it's important to be aware that retrieving your site's log file isn't always so simple, and common challenges that can be faced include:
Finding that log files have been disabled by a server admin and are not available Clients or other internal teams unwilling to provide log files or access to retrieve them Huge file sizes Log files that only store recent data (based either on a number of days or hits) Issues caused by CDNs Custom formatsThat said, these issues all have solutions that can usually be worked on alongside a developer or server admin.
Introducing the Semrush Log File Analyzer
The Semrush Log File Analyzer is an ideal way to gain an insight into how search engines are crawling your site without having extensive experience manually auditing these.
We've already shared how it can be complex and confusing for those who have never done a log file analysis before. However, our tool can help you to get the same insights simply and straightforwardly.
In fact, we recommend using the tool for the following reasons:
Analyzing a log file manually is tiresome. Unless you’re highly trained in technical website analysis, it can be an arduous task that leaves you cross-eyed and confused. If you want the quickest way to read an access log and understand how bots from Google interact with your website, this is the tool for you.
Using the tool makes log file analysis quick and easy to understand, and here's a simple and straightforward step-by-step process:
1. Make Sure Your Log File Is in the Correct Format
Before using the tool, you'll need to make sure that your log file is in the correct format; that's the proper access.log format. The tool also supports W3C, Kinsta, and Combined log format variations.
The proper file format is “Combined Log Format," and it uses the following structure:
%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"Where:
h — the host / IP address from which the request was made to the server l — client id, usually stays blank (represented by a hyphen (-) in the file) u — username, usually stays blank (represented by a hyphen (-) in the file) t — the time and time zone of the request to server r — the type of the request, its content, and version s — the HTTP status code b — the size of the object requested (in bytes) Referer — the URL source of the request (previous page) often stays blank (represented by a hyphen (-) in the file) User-Agent — the HTTP header containing information about the request (client application, language, etc.)There's also a 1GB maximum upload size you need to be aware of.
2. Upload Your Log File to the Tool
You can either drag and drop your log file(s) into the tool or browse your machine for the file to get ready to run the analysis.
Please make sure that your log files do not contain any personal data as recommended at this stage. It's important to note that log file analysis for SEO will only require GET data and not POST data (sensitive data can be found within this).
3. Start The Log File Analyzer
Once your log file has been uploaded, either add another file (in instances where the data you want to analyze is split across two or more files) or hit 'Start Log File Analyzer' to begin the analysis.
There may be a short wait while the tool runs, depending upon the size of the file.
4. Analyze Your Log File Data
Once your log file has been analyzed, you'll see two main reports within the tool:
Googlebot Activity Hits by PagesLooking first at Googlebot Activity, you can gain insights into the daily number of hits for different bots (bots), see the breakdown of different status codes (Status Codes), and view the frequency that different file types have been requested (File Type).
Bots:
Status Codes:
File Types:
You can use these insights to understand:
How many requests different search engine bots are making to your site each day (Bots) The breakdown of different HTTP status codes found per day (Status Codes) A breakdown of the different file types crawled each day (File Types)These insights are sitewide, but to gain specific information about how Googlebot and other crawlers are accessing your sites' content, look at the Hits by Pages report.
Here you'll see insights for specific pages (Filter by path), see which of your site's pages or folders has the most or least bot hits (sort by the Bot Hits column), or see which are crawled most frequently.
Using the tool makes log file analysis simple and straightforward. There's no reason why SEOs need to manually analyze their log file when they can use a tool to speed up the process.
Spend time optimizing your site, not analyzing data.Innovative SEO services
SEO is a patience game; no secret there. We`ll work with you to develop a Search strategy focused on producing increased traffic rankings in as early as 3-months.
A proven Allinclusive. SEO services for measuring, executing, and optimizing for Search Engine success. We say what we do and do what we say.
Our company as Semrush Agency Partner has designed a search engine optimization service that is both ethical and result-driven. We use the latest tools, strategies, and trends to help you move up in the search engines for the right keywords to get noticed by the right audience.
Today, you can schedule a Discovery call with us about your company needs.
Source:
