Glossary · AI search
AI crawler
An AI crawler is a bot an AI company uses to fetch pages for search answers, user requests or model training.
- By
- OSKIT team (Afformation)
- Updated
Definition
An AI crawler is a bot an AI company uses to fetch pages for search answers, user requests or model training.
AI crawler
How it works
They are split by purpose. OpenAI runs OAI-SearchBot (search), ChatGPT-User (user requests) and GPTBot (training). Perplexity runs PerplexityBot and Perplexity-User. Anthropic runs Claude-SearchBot, Claude-User and ClaudeBot.
User-Agent strings can be spoofed. OpenAI and Perplexity publish IP lists in JSON, so real bot visits are counted by matching both UA and IP.
What to check
- 01Each bot is allowed or blocked by purpose
- 02Server logs count visits by UA plus IP
- 03Firewall bot rules were checked
Where OSKIT handles it
Signal Engine
Signal Engine counts AI bot visits from request logs by matching UA against published IP lists.
See the featureSee also: robots.txt · Google-Extended · llms.txt · AI referral traffic
Sources
- [1]OpenAI — Overview of OpenAI crawlershttps://developers.openai.com/api/docs/bots
- [2]Perplexity — Perplexity crawlershttps://docs.perplexity.ai/guides/bots
Summarize in an AI chat
Only the page URL is sent.
A free check shows your search and AI answer visibility
Enter your website address and see search engine and AI bot access and index status right away.
3 minutes · Free · No contract