robots.txt

robots.txt is a website file that tells cooperating crawlers which paths they may request.

Source: https://contentcucumber.com/aeo-glossary/robots-txt/

Topic: Getting found. Also called: Robots exclusion, Disallow


Why it matters

A rule can prevent a crawler from fetching a page you intended it to read.

Example (illustrative, not a client result)

A Disallow rule for an old staging directory remains after that directory becomes a public resource library.

What to check

Review the rules for the specific crawler and path, and coordinate changes with the person responsible for security and publishing.

What this does not tell you

robots.txt is not a privacy control or a reliable removal method. A blocked URL can still appear in search, and a blocked crawler cannot read a page’s noindex tag.

Sources

Evidence type: Documented concept. Sources checked on 2026-11-19

Related terms