Log File Analysis & Bot Behavior

Your client's hosting provider stores log files on the server — they're typically accessible via cPanel, Plesk, or SSH. For most shared hosting setups, the raw log files can be downloaded directly from the control panel. For managed or cloud hosting (WP Engine, Kinsta, AWS, etc.), logs are usually available through the hosting dashboard or via a support request. We'll walk your team through exactly where to find them and what format we need. All files are handled securely and covered under our standard NDA.

For the Basic and Standard tiers, a minimum of 30 days is sufficient to identify crawl patterns and issues. 60–90 days gives a more reliable picture and is what we recommend where available. The Deep Dive tier is built for multi-month trend analysis — ideally 3–6 months of logs — which allows us to identify seasonal crawl patterns, track the impact of previous changes, and build a much stronger reallocation strategy. If logs have been rotated and only a shorter window is available, we work with what's there and note any limitations in the report.

Yes, in a number of situations. Crawl budget is most critical for large sites, but log analysis provides insights that are valuable regardless of size — particularly for sites experiencing unexplained indexation drops, sites that have undergone a migration, or sites where Search Console is showing crawl anomalies. If the client is seeing pages that should be indexed but aren't, or if they've had a ranking drop without an obvious cause, log files often surface the reason. For smaller sites with no known issues and stable performance, the return on investment is lower and we'll tell you so honestly.

A site crawl (Screaming Frog, Sitebulb, etc.) simulates how a bot sees the site at a single point in time. Log file analysis shows you what Google's bot actually did over a period of weeks or months — which pages it visited, how often, in what sequence, and what it received back. The two tools answer different questions. A crawl tells you what's there; logs tell you what Google is actually doing with it. Pages that appear fine in a crawl can show problematic crawl patterns in the logs — visited rarely, returning errors intermittently, or being crawled at the expense of higher-value pages.

Both — but with a clear priority on actionable outcomes. Every finding in the report is paired with a specific recommendation: whether that's a robots.txt change to stop Googlebot wasting crawl budget on low-value URLs, an internal linking adjustment to direct more crawl attention to important pages, or a technical fix for pages returning 404s or redirect chains to the bot. The Standard and Deep Dive tiers include explicit prioritization so your team or client's developer knows exactly what to tackle first and why.