AI crawlers
AI crawlers are automated programs that AI companies use to fetch web pages for model training, for search indexes, or in response to a person's request.
Source: https://contentcucumber.com/aeo-glossary/ai-crawlers/
Topic: Getting found. Also called: AI bots, LLM crawlers
Why it matters
Each crawler has its own user agent name, which means a robots.txt rule that blocks a training crawler can leave a search crawler allowed.
Example (illustrative, not a client result)
A publisher blocks GPTBot to keep pages out of training data and allows OAI-SearchBot so the same pages can appear in ChatGPT search results.
What to check
Read your robots.txt file and your server logs for the user agents each AI company documents, then confirm the rules match what you intend to allow.
What this does not tell you
OpenAI notes that robots.txt rules may not apply to a fetch a person starts. Treat robots.txt as a rule for crawlers, and expect an assistant to reach a page in other ways too.
Sources
- OpenAI: overview of OpenAI crawlers (Platform documentation)
- Anthropic: how site owners can block the crawler (Platform documentation)
Evidence type: Documented concept. Sources checked on 2026-11-21