Skip to content

Glossary · Search technical

Crawl budget

Crawl budget is the number of URLs and the frequency a search engine is willing to fetch from one site over a period.

By
OSKIT team (Afformation)
Updated
On this page
  1. 01Definition
  2. 02How it works
  3. 03What to check

Definition

Crawl budget is the number of URLs and the frequency a search engine is willing to fetch from one site over a period.

Crawl budget

How it works

Google’s documentation says it mainly matters for large sites with hundreds of thousands of pages or very frequent changes.

Duplicate URLs, endless filter combinations, redirect chains and slow responses consume it. Daily-publishing sites should set sitemap lastmod to the real edit date so new posts are fetched first.

What to check

  1. 01Redirects resolve in one hop
  2. 02Parameter URLs are not indexed
  3. 03Sitemap lastmod is the real edit date

See also: XML sitemap · robots.txt · 301 redirect · Canonical URL

Sources

  1. [1]Google Search Central — Large site owner's guide to managing crawl budgethttps://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Summarize in an AI chat

Only the page URL is sent.

A free check shows your search and AI answer visibility

Enter your website address and see search engine and AI bot access and index status right away.

3 minutes · Free · No contract