Skip to content

Blog · Crawling & indexing

11 AI crawlers and what to allow in robots.txt

We split OpenAI, Perplexity, Anthropic and Google bots into search, user-triggered and training bots, then set a robots.txt default and a way to check it.

By
OSKIT team (Afformation)
Updated
2 min read
On this page
  1. 01Bots fall into 3 purposes
  2. 02Bots by company (checked 18 November 2026)
  3. 03Google works differently
  4. 04The OSKIT default
  5. 05When robots.txt is not enough
  6. 06Counting real bot visits
  7. 07Checklist

Summary

  • AI company bots serve 3 purposes: search answers, user requests and training.
  • Blocking a search bot can remove you from that assistant’s search answers. Each company says blocking training bots does not affect search answers.
  • User-Agent strings can be spoofed, so visits are counted against published IP lists.

Bots fall into 3 purposes

An AI crawler is rarely a single bot per company. OpenAI runs 3, Perplexity 2 and Anthropic 3, each with a separate purpose[1][2][3]. The purposes are:

  1. 01Search answers: fetched in advance so the site can appear in AI search results. Blocking it can remove you from that assistant’s search answers.
  2. 02User requests: a visit at the moment a user asks a question or shares a link. Because it is not an automatic crawl, robots.txt may not apply.
  3. 03Training: collects model training data. Each company says blocking it does not affect search answers.

Bots by company (checked 18 November 2026)

CompanyUA tokenPurposerobots.txt
OpenAIOAI-SearchBotChatGPT searchObeyed, about 24 h to apply
OpenAIChatGPT-UserUser-triggered visitsMay not apply
OpenAIGPTBotTrainingObeyed
PerplexityPerplexityBotPerplexity searchObeyed
PerplexityPerplexity-UserUser-triggered visitsMostly ignored
AnthropicClaude-SearchBotClaude search indexObeyed
AnthropicClaude-UserUser-triggered visitsObeyed
AnthropicClaudeBotTrainingObeyed, incl. Crawl-delay
GoogleGooglebotSearch, AI Overviews, AI ModeObeyed
GoogleGoogle-ExtendedControl token for Gemini training and groundingObeyed (not a crawler)
OpenAIOAI-AdsBotChatGPT ad landing reviewOnly submitted ad pages

OpenAI and Perplexity were checked in their official documentation[1][2]. The three Anthropic bots were checked in a support article and a secondary report[3][4].

Google works differently

Google has no separate crawler for AI features. AI Overviews and AI Mode are part of Search, so Googlebot rules apply[5]. Google-Extended is a control token. Blocking it stops Gemini and other systems from using your content for training and grounding. It does not affect Google Search or AI Overviews[5].

The costliest mistake is blocking Googlebot itself. A staging Disallow: / that ships to production is the classic case.

The OSKIT default

Sites built by OSKIT’s Site Engine generate robots.txt with these rules:

  • Allow search and user-triggered bots: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot, Yeti (Naver), Daumoa (Daum).
  • Allow training bots (GPTBot, ClaudeBot, Google-Extended) by default. Clients can switch them off in settings.
  • Block admin, API and login paths for every bot.
txt
User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

When robots.txt is not enough

A bot can still be blocked when robots.txt allows it. CDN and firewall bot protection often blocks AI bots, and robots.txt will not show it. Request the home URL with each UA string and log the status code. A 403 or 503 points to a firewall rule. OSKIT runs this check every week.

Counting real bot visits

Anyone can fake a User-Agent string. OpenAI and Perplexity publish IP lists per bot in JSON[1][2]. Download them once a day and count only requests where UA and IP both match. We use the count for two things: how many days it takes a search bot to fetch a new post, and how often bots request llms.txt.

Checklist

  1. 01Googlebot is not blocked on any path.
  2. 02The 4 search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot) are allowed.
  3. 03The training-bot decision is written down as company policy.
  4. 04Each UA gets a 200 on the home URL, logged weekly.
  5. 05Bot visits are counted by UA plus published IP lists.

Questions this page answers

If I block GPTBot, will I disappear from ChatGPT search?

According to OpenAI, ChatGPT search visibility depends on OAI-SearchBot. GPTBot is for training, so blocking it is separate from search visibility.

Does blocking Google-Extended remove me from AI Overviews?

Google says AI Overviews and AI Mode follow Googlebot rules. Blocking Google-Extended only applies to training and grounding use by Gemini and other systems.

How fast do robots.txt changes apply?

OpenAI says OAI-SearchBot changes take about 24 hours. Other companies do not publish a time.

Sources

  1. [1]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
  2. [2]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots
  3. [3]Anthropic — Does Anthropic crawl data from the webhttps://support.claude.com/en/articles/8896518
  4. [4]Search Engine Land — Anthropic Claude botshttps://searchengineland.com/anthropic-claude-bots-470171
  5. [5]Google Search Central — AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features

Summarize in an AI chat

Only the page URL is sent.

A free check shows your search and AI answer visibility

Enter your website address and see search engine and AI bot access and index status right away.

3 minutes · Free · No contract