# robots.txt

> robots.txt is a website file that tells cooperating crawlers which paths they may request.

*Source: [https://contentcucumber.com/aeo-glossary/robots-txt/](https://contentcucumber.com/aeo-glossary/robots-txt/)*

*Topic: Getting found. Also called: Robots exclusion, Disallow*

---

## Why it matters

A rule can prevent a crawler from fetching a page you intended it to read.

## Example (illustrative, not a client result)

A Disallow rule for an old staging directory remains after that directory becomes a public resource library.

## What to check

Review the rules for the specific crawler and path, and coordinate changes with the person responsible for security and publishing.

## What this does not tell you

robots.txt is not a privacy control or a reliable removal method. A blocked URL can still appear in search, and a blocked crawler cannot read a page’s noindex tag.

## Sources

- [Google Search Central: robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (Platform documentation)
- [Google Search Central: robots meta tags](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) (Platform documentation)

*Evidence type: Documented concept. Sources checked on 2026-11-19*

## Related terms

- [Crawlability](https://llms.contentcucumber.com/aeo-glossary/crawlability.md)
- [Indexability](https://llms.contentcucumber.com/aeo-glossary/indexability.md)
