Skip to content

Guides · Crawling & indexing

Setting up llms.txt and robots.txt: order and check commands

Write robots.txt with AI search bots and training bots separated, build llms.txt and verify status codes with curl.

By
OSKIT team (Afformation)
Updated
1 min read
On this page
  1. 01Steps
  2. 02The short answer
  3. 031. Check the current state
  4. 042. Bot policy
  5. 053. robots.txt
  6. 064. llms.txt
  7. 075. Check status codes
  8. 086. Log the change
  9. 09In OSKIT

Summary

  • robots.txt sets what may be crawled. llms.txt lists pages worth reading and cannot block access.
  • Allow search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot). Decide training bots by company policy.
  • After setup, request the home page with each bot’s UA and confirm a 200.

Steps

  1. 1

    Check the current robots.txt

    Open https://domain/robots.txt and look for lines blocking Googlebot or everything (*).

  2. 2

    Decide bots and policy

    Split search, user-triggered and training bots and decide on training bots.

  3. 3

    Write robots.txt

    Add Allow per bot, Disallow for admin and API paths, and a Sitemap line.

  4. 4

    Write llms.txt

    H1 site name, summary blockquote, H2 link lists, canonical URLs only.

  5. 5

    Check status codes

    Send curl requests with each bot UA. A 403 or 503 points to a firewall rule.

  6. 6

    Log the change

    Record date and change. OAI-SearchBot changes apply after about 24 hours.

The short answer

Allow AI search bots in robots.txt, decide training bots by company policy, list canonical URLs in llms.txt, then request the site with each bot’s UA and confirm a 200. It takes 1–2 hours.

1. Check the current state

bash
curl -s https://example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt

Look for Disallow: / under User-agent: *, any rule blocking Googlebot, and staging rules left in production.

2. Bot policy

GroupBotsDefaultIf blocked
SearchOAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Yeti, DaumoaAllowMay drop out of that engine’s answers
User-triggeredChatGPT-User, Perplexity-User, Claude-UserAllowMay fail to read links users share
TrainingGPTBot, ClaudeBot, Google-ExtendedCompany policyExcluded from training and grounding, search unaffected

3. robots.txt

txt
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

robots.txt does not remove pages from search. Use noindex on the page, and do not block that page in robots.txt or the crawler cannot read the noindex.

4. llms.txt

md
# Clinic name

> Dermatology clinic, exit 3 of OO Station, Gangnam, Seoul. Japanese and English spoken.

## Treatments
- [Lifting](https://example.com/en/treatments/lifting): types, duration, price range

## Clinic
- [Doctors](https://example.com/en/doctors): 3 board-certified dermatologists

Use canonical URLs only, describe facts, leave out private paths and regenerate when pages change. Google says no special file is needed for AI features, so we promise no effect.

5. Check status codes

bash
for ua in "OAI-SearchBot" "PerplexityBot" "Claude-SearchBot" "GPTBot" "Googlebot"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/)
  echo "$ua $code"
done

All 200 means pass. A 403 or 503 points to CDN or firewall bot rules.

6. Log the change

Record the date and change. OpenAI says OAI-SearchBot changes take about 24 hours.

In OSKIT

Site Engine sites ship with these defaults. Clients switch training bots in settings, and a weekly check logs status codes per bot in Signal Engine.

Sources

  1. [1]Google Search Central — Introduction to robots.txthttps://developers.google.com/search/docs/crawling-indexing/robots/intro
  2. [2]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
  3. [3]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots
  4. [4]llmstxt.orghttps://llmstxt.org/
  5. [5]Google Search Central — AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features

Summarize in an AI chat

Only the page URL is sent.

A free check shows your search and AI answer visibility

Enter your website address and see search engine and AI bot access and index status right away.

3 minutes · Free · No contract