Skip to content

Glossary · AI search

AI crawler

An AI crawler is a bot an AI company uses to fetch pages for search answers, user requests or model training.

By
OSKIT team (Afformation)
Updated
On this page
  1. 01Definition
  2. 02How it works
  3. 03What to check
  4. 04Where OSKIT handles it

Definition

An AI crawler is a bot an AI company uses to fetch pages for search answers, user requests or model training.

AI crawler

How it works

They are split by purpose. OpenAI runs OAI-SearchBot (search), ChatGPT-User (user requests) and GPTBot (training). Perplexity runs PerplexityBot and Perplexity-User. Anthropic runs Claude-SearchBot, Claude-User and ClaudeBot.

User-Agent strings can be spoofed. OpenAI and Perplexity publish IP lists in JSON, so real bot visits are counted by matching both UA and IP.

What to check

  1. 01Each bot is allowed or blocked by purpose
  2. 02Server logs count visits by UA plus IP
  3. 03Firewall bot rules were checked

Where OSKIT handles it

Signal Engine

Signal Engine counts AI bot visits from request logs by matching UA against published IP lists.

See the feature

See also: robots.txt · Google-Extended · llms.txt · AI referral traffic

Sources

  1. [1]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
  2. [2]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots

Summarize in an AI chat

Only the page URL is sent.

A free check shows your search and AI answer visibility

Enter your website address and see search engine and AI bot access and index status right away.

3 minutes · Free · No contract