Blog · Crawling & indexing
Before you add llms.txt: format, official positions and what you can measure
We compare the llmstxt.org format with Google Search documentation and describe how to measure llms.txt without claiming effects.
- By
- OSKIT team (Afformation)
- Updated
- 2 min read
On this page
Summary
- llms.txt is a Markdown file where only an H1 with the site name is required.
- Google Search documentation says no special file is needed to appear in AI features.
- It costs little to generate, so we generate it and report only how often AI bots request it.
What llms.txt is and who proposed it
llms.txt is a Markdown file at /llms.txt. The format was proposed at llmstxt.org to help AI models read a site’s key pages in one pass[1].
It has five parts:
- 01An H1 with the site name (the only required part)
- 02A one-paragraph blockquote summary
- 03Optional free text
- 04Link lists under H2 headings:
- [Name](URL): description - 05An “Optional” H2 for links that can be skipped
# OSKIT
> A growth engine service run by Afformation: websites, daily blog posts, search and AI visibility analytics.
## Pages
- [Pricing](https://oskit.afformation.co.kr/en/pricing): build fee, monthly plan, usage
- [Glossary](https://oskit.afformation.co.kr/en/wiki): 46 search and AI termsllmstxt.org says thousands of sites publish the file, including the developer docs of OpenAI, Anthropic and Google Gemini[1]. It also recommends a Markdown version of each page linked with rel="alternate" type="text/markdown".
What Google Search documentation says
Google’s page on AI features says you do not need new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode[2]. AI features use the same index as Search and follow Googlebot rules.
| Item | What is confirmed |
|---|---|
| Standard status | Proposed by llmstxt.org. Not a W3C or IETF standard |
| Google AI features | No special file needed (Google docs) |
| Developer doc sites | OpenAI, Anthropic and Google Gemini docs publish one (llmstxt.org) |
| Visibility effect | No official measurement confirms one |
Why we still generate it
It costs almost nothing. If you already have structured page data, the file can be built automatically and stays in sync. It also helps people: a user can hand one URL to an assistant to describe the whole site.
Measuring without claiming effects
OSKIT does not say “llms.txt raises AI visibility” because there is no evidence. We report only what we can count:
- llms.txt requests by AI bots, counting only requests where UA and published IP list match[3].
- Which bots requested it: search bots such as OAI-SearchBot versus training bots such as GPTBot.
- Later bot visits to listed URLs compared with unlisted URLs. This is correlation and proves no cause. Reports say so.
Rules when you build it
- 01Use canonical URLs, never redirecting ones.
- 02Describe facts on each page. Adjectives give an AI nothing to quote.
- 03Regenerate it when pages are added or removed.
- 04Leave out private paths. llms.txt cannot block access. Use robots.txt and authentication for that.
- 05For multilingual sites, split by language section or file.
In OSKIT
Every Site Engine site generates llms.txt from its page structure. Post Engine updates it on each publish. Signal Engine shows AI bot requests by day. This site’s llms.txt is built the same way.
Questions this page answers
Does llms.txt make a site appear more in AI answers?
No official source confirms that. Google says no special file is needed. OSKIT reports request counts and promises no effect.
Do I also need llms-full.txt?
It is optional. It concatenates key page content and is mostly used on documentation sites.
How is llms.txt different from robots.txt?
robots.txt sets what may be crawled. llms.txt lists pages worth reading. llms.txt cannot block access.
Sources
- [1]llmstxt.org — The /llms.txt filehttps://llmstxt.org/
- [2]Google Search Central — AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features
- [3]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract