# The robots.txt and llms.txt Punch List | Content Cucumber

> The full robots.txt and llms.txt punch list for AI search visibility. Five steps, the AI crawlers to allow or block, and copy-paste templates for a spec-compliant llms.txt. Read it here or download the PDF, no form.

*Source: [https://contentcucumber.com/research/llms-txt-punch-list/](https://contentcucumber.com/research/llms-txt-punch-list/)*

*By Brent Peterson, CEO, Content Cucumber. Updated April 2026*

---

**Checklist**

Five steps and copy-paste templates that get a site's robots.txt and llms.txt right in an afternoon, including which AI crawlers to allow and which to block. This is the same punch list Content Cucumber works through before publishing either file for a client. All of it is on this page, and the PDF needs no form.

[Download the PDF](https://contentcucumber.com/wp-content/uploads/2026/09/llms-txt-punch-list.pdf)

## Introduction

AI systems are already answering questions about businesses. Right now those answers come from whatever a website's homepage says and whatever meta tags a crawler can parse. A site's **robots.txt** and **llms.txt** are the two files that change who writes the script.
**robots.txt** controls where crawlers can go. Every site has one.
**llms.txt** tells AI systems who the business is and what matters. Most sites don't have one.
This punch list walks through both files. Follow the five steps, copy the templates, and by the end of the afternoon any site is AI-ready.

## Step 1: Check What You Have

Before fixing anything, find out where the site stands.
Checklist
Open yourdomain.com/robots.txt in a browser. Does it load?
Open yourdomain.com/llms.txt in a browser. Does it load?
Check for SSL errors on both URLs.
Note which AI crawlers are mentioned (or not mentioned) in robots.txt.
Score Yourself
**Both files exist and load** — ahead of 90% of sites.
**robots.txt exists, no llms.txt** — normal, most sites are here.
**Neither exists** — fix robots.txt first, then llms.txt.

## Step 2: robots.txt Essentials

robots.txt controls where crawlers can go. Every site needs one.
Required Entries
User-agent: * with Allow: / (let crawlers in)
Disallow: for admin paths (/wp-admin/, /admin/, /api/)
Sitemap: pointing to the XML sitemap URL
Verify the sitemap URL returns a valid XML file
AI Crawler Decisions
The choice: allow search crawlers, block training crawlers, or both.
AI Search Crawlers (recommended: ALLOW)
These power the search and answer experiences that reference the business.
**GPTBot** — ChatGPT search answers
**OAI-SearchBot** — OpenAI web search
**ChatGPT-User** — ChatGPT browsing
**ClaudeBot** — Claude search and citations
**anthropic-ai** — Anthropic AI systems
**PerplexityBot** — Perplexity AI search
**Amazonbot** — Amazon Alexa and product search
**Applebot-Extended** — Apple intelligence features
AI Training Crawlers (recommended: BLOCK)
These take content to train future models. No attribution, no traffic back.
**CCBot** — Common Crawl dataset
**Bytespider** — ByteDance/TikTok model training
**Google-Extended** — Google Gemini model training
**cohere-ai** — Cohere model training
robots.txt Template
Copy and adapt. Replace yourdomain.com with the real site.
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://yourdomain.com/sitemap.xml # AI Search Crawlers - Allowed User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Amazonbot Allow: / # Training Crawlers - Blocked User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Google-Extended Disallow: / User-agent: cohere-ai Disallow: /

## Step 3: llms.txt Setup

llms.txt tells AI systems who a business is and what matters on a site. Think of it as a business card for machines.
Format Rules
Plain text file at the site root (/llms.txt)
H1 (#) with the company or site name
Blockquote (>) with a one-line summary immediately after H1
Optional paragraph with more detail
H2 (##) sections for each category of pages
Markdown links: - [Page Title](https://url): Brief description
Required Sections
**Company name** as H1
**One-line summary** as blockquote (what the business does, who it serves)
**Core Pages** section with the most important URLs
**Services** or **Products** section with links to key offerings
Recommended Sections
**About / Team** with key people and credentials
**Blog** or **Resources** linking to best content
**Contact** with website and location
**Integrations** or **Platforms** if applicable
**Brand Family** if there are related brands or parent companies
What to Include
Pages AI should cite and reference
Services associated with the brand name
Key people with their credentials (AI attributes expertise)
Locations (helps with local and regional queries)
Links to downloadable resources (research, guides, tools)
What to Leave Out
Marketing copy and superlatives
Every page on the site (curate, don't dump)
Pricing details that change frequently
Internal pages, login portals, utility pages
Anything that wouldn't go on a business card
llms.txt Template
Copy and adapt. Replace the placeholders with real content.
# Your Company Name > One-line description of what you do and who you serve. Brief paragraph with additional context. Keep it factual. Two to three sentences. ## Core Pages - [Homepage](https://yourdomain.com/): What visitors find on the homepage. - [About](https://yourdomain.com/about): Company background and team. - [Services](https://yourdomain.com/services): Overview of what you offer. - [Blog](https://yourdomain.com/blog): Articles and insights. - [Contact](https://yourdomain.com/contact): How to get in touch. ## Services - [Service One](https://yourdomain.com/service-one): Brief description. - [Service Two](https://yourdomain.com/service-two): Brief description. ## Contact - Website: yourdomain.com - Location: City, State

## Step 4: Verify After Deployment

Once both files are deployed, check they're working.
Load yourdomain.com/robots.txt in browser. Confirm it shows the rules.
Load yourdomain.com/llms.txt in browser. Confirm it shows structured content (not a 404 or the homepage).
Check SSL. Both files must load over HTTPS without certificate errors.
Test with an AI. Ask ChatGPT, Claude, or Perplexity *"What is [your company]?"* Compare the answer before and after.
Validate robots.txt syntax at [Google's robots.txt tester](https://www.google.com/webmasters/tools/robots-testing-tool).

## Step 5: Ongoing Maintenance

These files aren't set-and-forget.
Review llms.txt **quarterly**. Update when new services, pages, or team members appear.
Review robots.txt when new AI crawlers appear. The list grows every few months.
Monitor AI crawler traffic in Google Analytics 4 (check server logs or GA4 for bot user agents).
Re-audit after any major site restructure, rebrand, or domain change.
Watch the llms.txt spec. It's still evolving. Check [llmstxt.org](https://llmstxt.org) for updates.

## Quick Reference: robots.txt vs llms.txt

**robots.txt** = *"Where can you go?"* Controls crawler access. The front door.
**llms.txt** = *"Who are we?"* Describes the business for AI systems. The business card.
Both are needed. One lets AI in. The other tells AI who's inside.

## Six Common Mistakes

**1. Allowing training crawlers by default.** If robots.txt doesn't mention CCBot or Bytespider, they have access. Block what you don't want.
**2. Writing llms.txt like marketing copy.** AI systems want facts and links, not superlatives. *"Award-winning industry-leading solutions"* tells an AI nothing.
**3. Using AI-flagged language.** The file represents the brand. Run it through an AI content checker. Words like *"approach," "capture,"* and *"leverage"* are heavily overused by AI and make the file sound robotic. If a brand prides itself on human writing, the file introducing it to AI systems sounds human too.
**4. Listing every page.** Curate 10 to 15 most important pages. An AI map, not a sitemap.
**5. Forgetting SSL.** If llms.txt throws a certificate error, no AI system reads it.
**6. Setting and forgetting.** New AI crawlers appear regularly. Review quarterly.

## Ready for the Next Step?

This checklist gets the foundation right. The harder work is the writing inside llms.txt. **Content Cucumber writes llms.txt files for brands that want AI systems to represent them accurately.**
If the checklist surfaces work, Content Cucumber can handle it.
**Get in touch:** [contentcucumber.com/contact](https://contentcucumber.com/contact)

## Questions about this document

### What is this punch list?

Five steps and copy-paste templates for getting a site's robots.txt and llms.txt right, including which AI crawlers to allow and which to block. It is the same list Content Cucumber works through before publishing either file for a client.

### Is the full punch list on this page?

Yes, all five steps, both templates, the quick reference, and the six common mistakes. The PDF is the designed version with the same content, and it downloads with no form.

### What is llms.txt?

A plain text file at the root of a site that tells AI systems who the business is and which pages matter, in a format they can read. The setup step on this page covers the format rules and what to leave out.

### Which AI crawlers does the list say to allow?

The search and answer crawlers that send visibility back, such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Amazonbot. Training-only crawlers like CCBot and Bytespider are the ones it says to block.