Glossary · Search technical
robots.txt
robots.txt is a text file at the site root that tells each crawler which paths it may fetch.
- By
- OSKIT team (Afformation)
- Updated
Definition
robots.txt is a text file at the site root that tells each crawler which paths it may fetch.
robots.txt
How it works
User-agent lines select crawlers and Allow or Disallow set paths. It does not block indexing. Use noindex for that.
Blocking an AI search bot can remove the site from that assistant’s search answers. OpenAI says changes for OAI-SearchBot take about 24 hours.
What to check
- 01AI search bots and training bots are configured separately
- 02Admin and API paths are blocked
- 03No CDN or firewall rule blocks bots separately
Where OSKIT handles it
Site Engine
OSKIT sites allow search and user-triggered bots by default. Clients switch training bots on or off in settings.
See the featureSee also: AI crawler · Google-Extended · llms.txt · XML sitemap
Sources
- [1]Google Search Central — Introduction to robots.txthttps://developers.google.com/search/docs/crawling-indexing/robots/intro
- [2]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract