AI crawlers

AI crawlers are automated programs that AI companies use to fetch web pages for model training, for search indexes, or in response to a person's request.

Source: https://contentcucumber.com/aeo-glossary/ai-crawlers/

Topic: Getting found. Also called: AI bots, LLM crawlers


Why it matters

Each crawler has its own user agent name, which means a robots.txt rule that blocks a training crawler can leave a search crawler allowed.

Example (illustrative, not a client result)

A publisher blocks GPTBot to keep pages out of training data and allows OAI-SearchBot so the same pages can appear in ChatGPT search results.

What to check

Read your robots.txt file and your server logs for the user agents each AI company documents, then confirm the rules match what you intend to allow.

What this does not tell you

OpenAI notes that robots.txt rules may not apply to a fetch a person starts. Treat robots.txt as a rule for crawlers, and expect an assistant to reach a page in other ways too.

Sources

Evidence type: Documented concept. Sources checked on 2026-11-21

Related terms