Skip to content

Glossary · Search technical

robots.txt

robots.txt is a text file at the site root that tells each crawler which paths it may fetch.

By
OSKIT team (Afformation)
Updated
On this page
  1. 01Definition
  2. 02How it works
  3. 03What to check
  4. 04Where OSKIT handles it

Definition

robots.txt is a text file at the site root that tells each crawler which paths it may fetch.

robots.txt

How it works

User-agent lines select crawlers and Allow or Disallow set paths. It does not block indexing. Use noindex for that.

Blocking an AI search bot can remove the site from that assistant’s search answers. OpenAI says changes for OAI-SearchBot take about 24 hours.

What to check

  1. 01AI search bots and training bots are configured separately
  2. 02Admin and API paths are blocked
  3. 03No CDN or firewall rule blocks bots separately

Where OSKIT handles it

Site Engine

OSKIT sites allow search and user-triggered bots by default. Clients switch training bots on or off in settings.

See the feature

See also: AI crawler · Google-Extended · llms.txt · XML sitemap

Sources

  1. [1]Google Search Central — Introduction to robots.txthttps://developers.google.com/search/docs/crawling-indexing/robots/intro
  2. [2]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots

Summarize in an AI chat

Only the page URL is sent.

A free check shows your search and AI answer visibility

Enter your website address and see search engine and AI bot access and index status right away.

3 minutes · Free · No contract