Server logs record every request your site receives, including every visit from Googlebot, Bingbot and the AI crawlers. Analysing them shows what the bots did, which no crawler simulation or Search Console report can show you in full. This is the method I use. If you're still deciding whether you need it, read the role of log file analysis in technical SEO first.
Step 1: Get the right logs
Ask your developers or hosting team for raw access logs. Where they live depends on your setup:
- Your own server: Nginx and Apache write access logs, usually in the "combined" format.
- A CDN in front of the site: Cloudflare, Akamai, Fastly and CloudFront see requests that may never reach your server. Use their log exports (Cloudflare Logpush, for example).
- A hosting platform: Vercel, Netlify and similar platforms stream logs through log drains to a storage or analytics service.
Get at least 30 days. For large sites or slow-changing sections, 60 to 90 days shows patterns better. Each line needs the client IP, timestamp, request method and URL, status code, bytes sent, referrer and user agent. Response time is a bonus.
A combined-format line looks like this:
66.249.66.1 - - [03/Nov/2026:10:15:32 +0530] "GET /services/aeo/ HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Logs contain IP addresses, which count as personal data in many places, so agree with your team how they will be stored and deleted.
Step 2: Keep only real bots
Anyone can put "Googlebot" or "GPTBot" in a user agent. Scanners do it all the time.
- Googlebot and Bingbot: run a reverse DNS lookup on the IP, check that it ends in googlebot.com, google.com or search.msn.com, then run a forward lookup to confirm it points back to the same IP. Google also publishes its crawler IP ranges as JSON files.
- AI crawlers: OpenAI, Perplexity and Anthropic publish the IP ranges their bots use. Match requests against those lists.
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com
Tools like Screaming Frog Log File Analyser do this verification for you. Expect to discard a real share of "AI bot" traffic: one SEO who checked a week of his own logs in 2026 found that 80% of requests calling themselves GPTBot were fake.
Step 3: Run the eight checks
1. Status codes by bot
What share of Googlebot's requests get 200, 301, 404 and 5xx? Errors on pages that should work are the first thing to fix. A spike of 5xx responses means the server struggled while being crawled.
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
2. Crawl by section
Group URLs by folder or template (product, category, blog, filters, account) and compare the crawl each one gets with its value to the business. If filter pages take half the crawl and product pages a tenth, you have a priority problem.
grep "Googlebot" access.log | awk '{print $7}' | cut -d'?' -f1 | cut -d'/' -f2 | sort | uniq -c | sort -rn
3. Priority pages: crawled or not
Take the list of pages that earn revenue or leads. Check when Googlebot last fetched each one. Pages not crawled in weeks are slow to pick up changes, and often have weak internal links.
4. Crawl waste
Look for URLs with parameters (sorting, tracking, session IDs), endless filter combinations, internal search results and calendar pages. Count how much of the crawl they take.
grep "Googlebot" access.log | awk '{print $7}' | grep -c "?"
5. Orphan URLs
Compare the URLs bots request with the URLs a site crawl finds through links. URLs that bots visit but your crawl can't reach are orphans: old pages, removed sections, pages only in the sitemap. Link the useful ones, and redirect or remove the rest.
6. Redirects and chains
Count 301 and 302 responses, and find old URLs that bots keep requesting. Update internal links and sitemaps so bots go straight to the final URL, and collapse chains to a single hop.
7. Response time
If your logs record it, compare response times for bots across sections. Slow templates get crawled less, and Google's documentation says it reduces crawling when a server responds slowly or with errors.
8. AI bot access
Separate the AI crawlers by job: training (GPTBot, ClaudeBot), search indexing (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user-triggered fetches (ChatGPT-User, Perplexity-User, Claude-User). Check that the search and user bots reach your important pages with 200 responses, and compare their coverage with Googlebot's. My post on crawl parity between search bots and LLM bots covers this check and the tools that do it.
Step 4: Choose a log file analyzer that fits the site
| Site size | Tool | Notes |
|---|---|---|
| Up to a few hundred thousand log lines a day | Screaming Frog Log File Analyser | Desktop app with a free version limited to 1,000 log events; the paid licence adds bot verification and AI bot presets |
| Small to mid-size sites, quick look | Semrush Log File Analyzer | Upload a log file in the browser and see bot hits and status codes; included with a Semrush account |
| Mid-size to large sites | JetOctopus | Cloud crawler and log analyzer with live log streaming |
| Large sites, many millions of lines | Botify, Oncrawl | Cloud platforms that join logs with crawl data and keep history |
| Any size, technical team | Command line, BigQuery, ClickHouse | Cheapest at scale, needs someone who writes queries |
Whatever you use, join the logs with a site crawl and your sitemap. Logs alone tell you what bots did. The join tells you what they missed.
Step 5: Fix, then compare
Each finding should become a ticket with an expected result:
- Waste on parameters: add rules to
robots.txt, fix the internal links that create the URLs, set canonicals - Priority pages crawled rarely: add internal links from frequently crawled pages, include them in the sitemap with accurate
lastmod - Errors: fix or redirect, and correct the links pointing at them
- Slow templates: hand the response time data to the developers
Then pull the next 30 days and compare the same checks. That before-and-after view is what makes log work easy to report.

