Blog · Crawling & indexing
11 AI crawlers and what to allow in robots.txt
We split OpenAI, Perplexity, Anthropic and Google bots into search, user-triggered and training bots, then set a robots.txt default and a way to check it.
- By
- OSKIT team (Afformation)
- Updated
- 2 min read
On this page
Summary
- AI company bots serve 3 purposes: search answers, user requests and training.
- Blocking a search bot can remove you from that assistant’s search answers. Each company says blocking training bots does not affect search answers.
- User-Agent strings can be spoofed, so visits are counted against published IP lists.
Bots fall into 3 purposes
An AI crawler is rarely a single bot per company. OpenAI runs 3, Perplexity 2 and Anthropic 3, each with a separate purpose[1][2][3]. The purposes are:
- 01Search answers: fetched in advance so the site can appear in AI search results. Blocking it can remove you from that assistant’s search answers.
- 02User requests: a visit at the moment a user asks a question or shares a link. Because it is not an automatic crawl, robots.txt may not apply.
- 03Training: collects model training data. Each company says blocking it does not affect search answers.
Bots by company (checked 18 November 2026)
| Company | UA token | Purpose | robots.txt |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT search | Obeyed, about 24 h to apply |
| OpenAI | ChatGPT-User | User-triggered visits | May not apply |
| OpenAI | GPTBot | Training | Obeyed |
| Perplexity | PerplexityBot | Perplexity search | Obeyed |
| Perplexity | Perplexity-User | User-triggered visits | Mostly ignored |
| Anthropic | Claude-SearchBot | Claude search index | Obeyed |
| Anthropic | Claude-User | User-triggered visits | Obeyed |
| Anthropic | ClaudeBot | Training | Obeyed, incl. Crawl-delay |
Googlebot | Search, AI Overviews, AI Mode | Obeyed | |
Google-Extended | Control token for Gemini training and grounding | Obeyed (not a crawler) | |
| OpenAI | OAI-AdsBot | ChatGPT ad landing review | Only submitted ad pages |
OpenAI and Perplexity were checked in their official documentation[1][2]. The three Anthropic bots were checked in a support article and a secondary report[3][4].
Google works differently
Google has no separate crawler for AI features. AI Overviews and AI Mode are part of Search, so Googlebot rules apply[5]. Google-Extended is a control token. Blocking it stops Gemini and other systems from using your content for training and grounding. It does not affect Google Search or AI Overviews[5].
The costliest mistake is blocking Googlebot itself. A staging Disallow: / that ships to production is the classic case.
The OSKIT default
Sites built by OSKIT’s Site Engine generate robots.txt with these rules:
- Allow search and user-triggered bots:
OAI-SearchBot,ChatGPT-User,PerplexityBot,Perplexity-User,Claude-SearchBot,Claude-User,Googlebot,Bingbot,Yeti(Naver),Daumoa(Daum). - Allow training bots (
GPTBot,ClaudeBot,Google-Extended) by default. Clients can switch them off in settings. - Block admin, API and login paths for every bot.
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlWhen robots.txt is not enough
A bot can still be blocked when robots.txt allows it. CDN and firewall bot protection often blocks AI bots, and robots.txt will not show it. Request the home URL with each UA string and log the status code. A 403 or 503 points to a firewall rule. OSKIT runs this check every week.
Counting real bot visits
Anyone can fake a User-Agent string. OpenAI and Perplexity publish IP lists per bot in JSON[1][2]. Download them once a day and count only requests where UA and IP both match. We use the count for two things: how many days it takes a search bot to fetch a new post, and how often bots request llms.txt.
Checklist
- 01Googlebot is not blocked on any path.
- 02The 4 search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot) are allowed.
- 03The training-bot decision is written down as company policy.
- 04Each UA gets a 200 on the home URL, logged weekly.
- 05Bot visits are counted by UA plus published IP lists.
Questions this page answers
If I block GPTBot, will I disappear from ChatGPT search?
According to OpenAI, ChatGPT search visibility depends on OAI-SearchBot. GPTBot is for training, so blocking it is separate from search visibility.
Does blocking Google-Extended remove me from AI Overviews?
Google says AI Overviews and AI Mode follow Googlebot rules. Blocking Google-Extended only applies to training and grounding use by Gemini and other systems.
How fast do robots.txt changes apply?
OpenAI says OAI-SearchBot changes take about 24 hours. Other companies do not publish a time.
Sources
- [1]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
- [2]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots
- [3]Anthropic — Does Anthropic crawl data from the webhttps://support.claude.com/en/articles/8896518
- [4]Search Engine Land — Anthropic Claude botshttps://searchengineland.com/anthropic-claude-bots-470171
- [5]Google Search Central — AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract