Glossary · Search technical
Crawl budget
Crawl budget is the number of URLs and the frequency a search engine is willing to fetch from one site over a period.
- By
- OSKIT team (Afformation)
- Updated
On this page
Definition
Crawl budget is the number of URLs and the frequency a search engine is willing to fetch from one site over a period.
Crawl budget
How it works
Google’s documentation says it mainly matters for large sites with hundreds of thousands of pages or very frequent changes.
Duplicate URLs, endless filter combinations, redirect chains and slow responses consume it. Daily-publishing sites should set sitemap lastmod to the real edit date so new posts are fetched first.
What to check
- 01Redirects resolve in one hop
- 02Parameter URLs are not indexed
- 03Sitemap lastmod is the real edit date
See also: XML sitemap · robots.txt · 301 redirect · Canonical URL
Sources
- [1]Google Search Central — Large site owner's guide to managing crawl budgethttps://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract