Kicked off /full-site-audit on a fairly big site (roughly 900 pages) and the progress bar has been sitting on 'crawling' for a long while — much longer than my smaller sites, which finish in a couple of minutes. Is it actually stuck and I should retry, or is it just slow on a site this size? Don't want to burn credits re-running if it's going to eventually complete. Screenshot of the stuck state if it helps.
Full Site Audit stuck on 'crawling' for a large site
3 Replies
Not stuck — genuinely slower, and this is expected on heavy sites. The crawl is polite by design (it doesn't hammer your server), so a 900-page site legitimately takes considerably longer than a 60-page one. On large sites the 'crawling' phase can run for a good while before it flips to analysis, and it will complete.
A couple of tips:
- Don't re-run — you'd just spend credits on a duplicate crawl. The original will finish.
- If it ever truly hangs (no progress for a very long time), it's usually server-side rate limiting or a crawl trap (infinite calendar/filter URLs) rather than our tool dying.
- The result saves to Analysis History when done, so you can close the tab and come back.
We do show a slowness caveat for heavy sites for exactly this reason. Let it run.
You were right — came back after lunch and it had completed, health score and all, no re-run needed. The 'close the tab, it saves to history' bit is the reassurance I needed. Maybe make that message more prominent on big crawls?
Good feedback — making the 'this is safe to leave, we'll save it' messaging more prominent during long crawls is a fair UX ask. Passing it to the product team. Glad it completed cleanly.