Crawl budget matters most for large sites — those with hundreds of thousands of URLs. Small sites are usually crawled comprehensively.
The two components
- Crawl capacity limit — how many requests Googlebot can make without degrading the site's performance.
- Crawl demand — how much Googlebot wants to crawl based on URL popularity and staleness.
Example
A marketplace with two million product URLs and a slow server may see Googlebot only recrawl a fraction of the catalogue each week, so newly listed products take days to be indexed.
How SEM Optimiser reports it
The audit surfaces crawl-budget waste by detecting duplicate parameter URLs, infinite calendar loops, and redirect chains that consume budget without adding value.
Related terms
- Index Coverage
Index coverage is the report in Google Search Console that shows which URLs on a site are indexed, which are excluded, and why.
- robots.txt
robots.txt is a plain-text file at the root of a domain that instructs crawlers which paths they are allowed or disallowed to fetch.
- XML Sitemap
An XML sitemap is a file that lists a website's important URLs to help search engines discover and crawl them efficiently.
- JavaScript SEO
JavaScript SEO is the practice of ensuring content generated or modified by client-side JavaScript is discoverable, crawlable, and indexable by search engines.