Guides · Crawling & indexing
Setting up llms.txt and robots.txt: order and check commands
Write robots.txt with AI search bots and training bots separated, build llms.txt and verify status codes with curl.
- By
- OSKIT team (Afformation)
- Updated
- 1 min read
On this page
Summary
- robots.txt sets what may be crawled. llms.txt lists pages worth reading and cannot block access.
- Allow search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot). Decide training bots by company policy.
- After setup, request the home page with each bot’s UA and confirm a 200.
Steps
- 1
Check the current robots.txt
Open https://domain/robots.txt and look for lines blocking Googlebot or everything (*).
- 2
Decide bots and policy
Split search, user-triggered and training bots and decide on training bots.
- 3
Write robots.txt
Add Allow per bot, Disallow for admin and API paths, and a Sitemap line.
- 4
Write llms.txt
H1 site name, summary blockquote, H2 link lists, canonical URLs only.
- 5
Check status codes
Send curl requests with each bot UA. A 403 or 503 points to a firewall rule.
- 6
Log the change
Record date and change. OAI-SearchBot changes apply after about 24 hours.
The short answer
Allow AI search bots in robots.txt, decide training bots by company policy, list canonical URLs in llms.txt, then request the site with each bot’s UA and confirm a 200. It takes 1–2 hours.
1. Check the current state
curl -s https://example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txtLook for Disallow: / under User-agent: *, any rule blocking Googlebot, and staging rules left in production.
2. Bot policy
| Group | Bots | Default | If blocked |
|---|---|---|---|
| Search | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Yeti, Daumoa | Allow | May drop out of that engine’s answers |
| User-triggered | ChatGPT-User, Perplexity-User, Claude-User | Allow | May fail to read links users share |
| Training | GPTBot, ClaudeBot, Google-Extended | Company policy | Excluded from training and grounding, search unaffected |
3. robots.txt
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlrobots.txt does not remove pages from search. Use noindex on the page, and do not block that page in robots.txt or the crawler cannot read the noindex.
4. llms.txt
# Clinic name
> Dermatology clinic, exit 3 of OO Station, Gangnam, Seoul. Japanese and English spoken.
## Treatments
- [Lifting](https://example.com/en/treatments/lifting): types, duration, price range
## Clinic
- [Doctors](https://example.com/en/doctors): 3 board-certified dermatologistsUse canonical URLs only, describe facts, leave out private paths and regenerate when pages change. Google says no special file is needed for AI features, so we promise no effect.
5. Check status codes
for ua in "OAI-SearchBot" "PerplexityBot" "Claude-SearchBot" "GPTBot" "Googlebot"; do
code=$(curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/)
echo "$ua $code"
doneAll 200 means pass. A 403 or 503 points to CDN or firewall bot rules.
6. Log the change
Record the date and change. OpenAI says OAI-SearchBot changes take about 24 hours.
In OSKIT
Site Engine sites ship with these defaults. Clients switch training bots in settings, and a weekly check logs status codes per bot in Signal Engine.
Sources
- [1]Google Search Central — Introduction to robots.txthttps://developers.google.com/search/docs/crawling-indexing/robots/intro
- [2]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
- [3]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots
- [4]llmstxt.orghttps://llmstxt.org/
- [5]Google Search Central — AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract