# AI crawlers

> AI crawlers are automated programs that AI companies use to fetch web pages for model training, for search indexes, or in response to a person's request.

*Source: [https://contentcucumber.com/aeo-glossary/ai-crawlers/](https://contentcucumber.com/aeo-glossary/ai-crawlers/)*

*Topic: Getting found. Also called: AI bots, LLM crawlers*

---

## Why it matters

Each crawler has its own user agent name, which means a robots.txt rule that blocks a training crawler can leave a search crawler allowed.

## Example (illustrative, not a client result)

A publisher blocks GPTBot to keep pages out of training data and allows OAI-SearchBot so the same pages can appear in ChatGPT search results.

## What to check

Read your robots.txt file and your server logs for the user agents each AI company documents, then confirm the rules match what you intend to allow.

## What this does not tell you

OpenAI notes that robots.txt rules may not apply to a fetch a person starts. Treat robots.txt as a rule for crawlers, and expect an assistant to reach a page in other ways too.

## Sources

- [OpenAI: overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (Platform documentation)
- [Anthropic: how site owners can block the crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Platform documentation)

*Evidence type: Documented concept. Sources checked on 2026-11-21*

## Related terms

- [robots.txt](https://llms.contentcucumber.com/aeo-glossary/robots-txt.md)
- [Crawlability](https://llms.contentcucumber.com/aeo-glossary/crawlability.md)
- [llms.txt](https://llms.contentcucumber.com/aeo-glossary/llms-txt.md)
