How to analyze server log files for SEO

Technical SEO · 10 November 2026 · Updated 19 November 2026 · 5 min read

A step-by-step method for SEO log analysis: getting the logs, verifying Googlebot and AI bots, the eight checks that matter, and the tools to use.

A computer screen showing system information in a terminal window

Server logs record every request your site receives, including every visit from Googlebot, Bingbot and the AI crawlers. Analysing them shows what the bots did, which no crawler simulation or Search Console report can show you in full. This is the method I use. If you're still deciding whether you need it, read the role of log file analysis in technical SEO first.

Step 1: Get the right logs

Ask your developers or hosting team for raw access logs. Where they live depends on your setup:

  • Your own server: Nginx and Apache write access logs, usually in the "combined" format.
  • A CDN in front of the site: Cloudflare, Akamai, Fastly and CloudFront see requests that may never reach your server. Use their log exports (Cloudflare Logpush, for example).
  • A hosting platform: Vercel, Netlify and similar platforms stream logs through log drains to a storage or analytics service.

Get at least 30 days. For large sites or slow-changing sections, 60 to 90 days shows patterns better. Each line needs the client IP, timestamp, request method and URL, status code, bytes sent, referrer and user agent. Response time is a bonus.

A combined-format line looks like this:

66.249.66.1 - - [03/Nov/2026:10:15:32 +0530] "GET /services/aeo/ HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Logs contain IP addresses, which count as personal data in many places, so agree with your team how they will be stored and deleted.

Step 2: Keep only real bots

Anyone can put "Googlebot" or "GPTBot" in a user agent. Scanners do it all the time.

  • Googlebot and Bingbot: run a reverse DNS lookup on the IP, check that it ends in googlebot.com, google.com or search.msn.com, then run a forward lookup to confirm it points back to the same IP. Google also publishes its crawler IP ranges as JSON files.
  • AI crawlers: OpenAI, Perplexity and Anthropic publish the IP ranges their bots use. Match requests against those lists.
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Tools like Screaming Frog Log File Analyser do this verification for you. Expect to discard a real share of "AI bot" traffic: one SEO who checked a week of his own logs in 2026 found that 80% of requests calling themselves GPTBot were fake.

Step 3: Run the eight checks

1. Status codes by bot

What share of Googlebot's requests get 200, 301, 404 and 5xx? Errors on pages that should work are the first thing to fix. A spike of 5xx responses means the server struggled while being crawled.

grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

2. Crawl by section

Group URLs by folder or template (product, category, blog, filters, account) and compare the crawl each one gets with its value to the business. If filter pages take half the crawl and product pages a tenth, you have a priority problem.

grep "Googlebot" access.log | awk '{print $7}' | cut -d'?' -f1 | cut -d'/' -f2 | sort | uniq -c | sort -rn

3. Priority pages: crawled or not

Take the list of pages that earn revenue or leads. Check when Googlebot last fetched each one. Pages not crawled in weeks are slow to pick up changes, and often have weak internal links.

4. Crawl waste

Look for URLs with parameters (sorting, tracking, session IDs), endless filter combinations, internal search results and calendar pages. Count how much of the crawl they take.

grep "Googlebot" access.log | awk '{print $7}' | grep -c "?"

5. Orphan URLs

Compare the URLs bots request with the URLs a site crawl finds through links. URLs that bots visit but your crawl can't reach are orphans: old pages, removed sections, pages only in the sitemap. Link the useful ones, and redirect or remove the rest.

6. Redirects and chains

Count 301 and 302 responses, and find old URLs that bots keep requesting. Update internal links and sitemaps so bots go straight to the final URL, and collapse chains to a single hop.

7. Response time

If your logs record it, compare response times for bots across sections. Slow templates get crawled less, and Google's documentation says it reduces crawling when a server responds slowly or with errors.

8. AI bot access

Separate the AI crawlers by job: training (GPTBot, ClaudeBot), search indexing (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user-triggered fetches (ChatGPT-User, Perplexity-User, Claude-User). Check that the search and user bots reach your important pages with 200 responses, and compare their coverage with Googlebot's. My post on crawl parity between search bots and LLM bots covers this check and the tools that do it.

Step 4: Choose a log file analyzer that fits the site

Site sizeToolNotes
Up to a few hundred thousand log lines a dayScreaming Frog Log File AnalyserDesktop app with a free version limited to 1,000 log events; the paid licence adds bot verification and AI bot presets
Small to mid-size sites, quick lookSemrush Log File AnalyzerUpload a log file in the browser and see bot hits and status codes; included with a Semrush account
Mid-size to large sitesJetOctopusCloud crawler and log analyzer with live log streaming
Large sites, many millions of linesBotify, OncrawlCloud platforms that join logs with crawl data and keep history
Any size, technical teamCommand line, BigQuery, ClickHouseCheapest at scale, needs someone who writes queries

Whatever you use, join the logs with a site crawl and your sitemap. Logs alone tell you what bots did. The join tells you what they missed.

Step 5: Fix, then compare

Each finding should become a ticket with an expected result:

  • Waste on parameters: add rules to robots.txt, fix the internal links that create the URLs, set canonicals
  • Priority pages crawled rarely: add internal links from frequently crawled pages, include them in the sitemap with accurate lastmod
  • Errors: fix or redirect, and correct the links pointing at them
  • Slow templates: hand the response time data to the developers

Then pull the next 30 days and compare the same checks. That before-and-after view is what makes log work easy to report.

Key takeaways

  • Ask for 30 to 90 days of raw logs from wherever requests are handled, including the CDN.
  • Verify bots by IP before counting anything; a lot of bot traffic is fake.
  • Run the eight checks: status codes, sections, priority pages, waste, orphans, redirects, speed and AI bots.
  • Join logs with a site crawl and the sitemap to see what bots missed.
  • Turn each finding into a ticket, then compare the next 30 days against the first.

FAQ Questions

Questions people ask

What is the best tool for SEO log file analysis?

For most small and mid-size sites, Screaming Frog Log File Analyser. Large sites usually need a cloud platform such as Botify or Oncrawl, or a data warehouse like BigQuery.

How many days of logs do I need?

At least 30. For large sites, or to see how often deep pages get crawled, 60 to 90 days is better.

Can I do log analysis without developer help?

Usually you need a developer or hosting admin to export the logs. The analysis itself can be done in a log tool without coding.

Can I see AI crawlers like GPTBot in my logs?

Yes, if their requests reach your server or CDN logs. Filter by user agent, then verify against the IP ranges each company publishes.

Chaitanya Kore in round sunglasses, facing the camera

Author About the writer

Chaitanya Kore

AI & SEO Strategist · Pune · Since 2017

I have worked in SEO since 2017, starting as an analyst. I have led an SEO team of 19 analysts across more than 70 projects for clients in India and Sweden. Today I run SEO strategy, including AEO and GEO, for international SaaS, eCommerce and B2B clients at WSI ALM Corp.

I also work directly with businesses in Pune on a 12-month SEO, AEO and GEO contract, planned one quarter at a time. If anything on a quarter's list isn't delivered, the extra hours are on me.

BBA, Marketing, Pune University · Certified: Google Analytics, HubSpot

Next Get in touch

Plan your first quarter

Based in Pune and working with companies across India. If anything on a quarter's list isn't delivered, the extra hours are on me.