How to use the Log Analyzer
What it does
The Log Analyzer parses your web server's access logs to show how search engine and AI crawlers actually interact with your site. Unlike a simulated crawl, log data reflects real bot behavior: which pages they hit, how often, which status codes they receive, and how much crawl activity each bot spends. This reveals crawl-budget waste, ignored pages, and errors that only bots encounter.
How to use it
- Open Log Analyzer under Tools.
- Export an access log file from your server (common formats such as Apache or Nginx combined logs are supported).
- Upload the file using the Upload Log control.
Reading the results
The report groups hits by user agent, so you can see how much attention Googlebot, Bingbot, and AI crawlers give your site. The status code breakdown highlights how many bot requests hit 404 or 5xx errors, which waste crawl budget. The top URLs view shows which pages bots crawl most and least, helping you spot important pages being ignored or unimportant pages being over-crawled. Use these findings to tune internal linking, your sitemap, and robots directives.
Credits
The Log Analyzer is free and costs 0 credits. Tip: analyze at least a few weeks of logs for a reliable picture, since crawl patterns vary day to day.
Was this helpful?