---
type: "Article"
title: "Block AI Bots in robots.txt? Why I Did the Opposite"
description: "A scanner told me to block AI bots in my robots.txt. I allowed them all and declared my terms with Content Signals instead. Here's the file and the trap."
resource: "https://toddmorourke.com/learn/robots-txt-ai-crawlers/"
tags: ["AEO"]
status: "stable"
generated:
  by: "okf-publisher/0.3"
  at: "2026-07-15T15:50:26+00:00"
verified:
  - by: "human:todd-orourke"
    at: "2026-07-15T15:50:26+00:00"
stale_after: "2027-01-15"
sources:
  - resource: "https://www.rfc-editor.org/rfc/rfc9309"
  - resource: "https://blog.cloudflare.com/content-signals-policy/"
  - resource: "https://github.com/ai-robots-txt/ai.robots.txt"
  - resource: "https://contentsignals.org"
timestamp: 2026-07-15T15:50:26+00:00
---

# Block AI Bots in robots.txt? Why I Did the Opposite

The same scanner that flagged my site for [a missing markdown setup](https://toddmorourke.com/learn/markdown-for-agents/) came back with two more complaints about my robots.txt: no rules for AI crawlers, and no Content Signals.

Search the topic and every guide tells you the same thing: block the AI bots. Block GPTBot, block CCBot, turn on the "block AI" toggle. I did the opposite. I allowed every AI crawler on the list and instead *declared* how my content may be used. I want to be cited by AI, so walling off the crawlers that feed AI answers is the last thing I'd do. This is the next entry in an unofficial series where I test things on the site to see what breaks before applying it to client websites. I previously wrote about Google's [Agentic Resource Discovery](https://toddmorourke.com/learn/agentic-resource-discovery/) and how I applied it on this very site.

This time, we're looking at the robots.txt file updates, including the policy I chose, the exact file, and a WordPress trap that silently breaks robots.txt on a CMS without a single error.

## Key Takeaways

- A bare `User-agent: *` block isn't a policy. Scanners now flag the absence of explicit AI-bot rules because silence isn't a decision.
- The whole internet says "block." For most businesses that's the wrong default: blocking doesn't stop bad actors, and it removes you from the AI answers where discovery now happens.
- I allowed the crawl and declared terms with one Content Signal: `search=yes, ai-train=no, ai-input=yes`. Index me, cite me, but don't train on me for free.
- robots.txt is a request, not a fence. And per RFC 9309, a named bot ignores your `*` block, so every directive has to be repeated in each bot's own group.
- On WordPress, a physical robots.txt file silently overrides the virtual one your plugins write to. If it exists, it's your only robots.txt.

## The Two Things My robots.txt Was Missing

My file had a wildcard rule and a crawl-delay, and I'd assumed that was enough. It wasn't a policy. It was silence. The scanner named two specific gaps.

### No AI-specific crawler rules

A single `User-agent: *` block says nothing about AI crawlers on purpose or otherwise. It's ambiguous by design, and per the protocol a named bot can ignore it entirely (more on that below). That's why scanners now flag the absence of explicit AI-bot groups: silence isn't consent, refusal, or a documented choice. It's just an unanswered question. Name the bots, even to allow them, and you've put a decision on the record instead of leaving it to inference.

### No Content Signals

Content Signals are a separate, newer layer: a way to declare *how* your content may be used, on top of the old allow-and-deny crawl rules. Cloudflare introduced them on September 24, 2025, and automatically added them to the robots.txt of the millions of domains on its managed service. ([Cloudflare](https://blog.cloudflare.com/content-signals-policy/)) My file had none. So I was granting crawl permission while stating no usage preference at all, which is exactly the gap between "you can read this" and "here's what you may do with it."

## Block, Allow, or Declare: Picking an AI-Crawler Policy

Every site owner is making this decision whether they realize it or not. There are three real options, and I rejected the popular one on purpose.

### Why the whole internet says "block" (and why I didn't)

Look at the search results for this topic and they're a wall of blocking: Cloudflare, Netlify, and Akamai selling toggles, a community repo shipping blocklists, publishers framing it as protecting content for future licensing deals. That's a legitimate stance for a large media site sitting on licensable archives. It's the wrong default for almost everyone else, for two reasons.

First, blocking in robots.txt doesn't actually stop a determined scraper. It's a request, not a wall (I'll come back to that). Second, it *does* pull you out of the AI answers and search features where discovery is increasingly happening. For a business that wants to be found and cited, blocking the crawlers that feed those answers is self-sabotage dressed up as caution. The most-referenced community resource on the topic is a repo that ships ready-made blocklists for robots.txt, Apache, and nginx and recommends blocking outright. ([ai-robots-txt](https://github.com/ai-robots-txt/ai.robots.txt))

### The stance I chose: allow the crawl, declare the terms

I allowed every AI crawler and set one Content Signal across the file: `search=yes, ai-train=no, ai-input=yes`. In plain English, that's index me, cite me in AI answers, but I'm declining to hand my content to model training for free.

Those three dimensions are the whole point of Content Signals. `search` covers traditional and AI search indexing, the thing you've always wanted. `ai-input` covers real-time use in AI answers, which is the citation you're actively chasing if you care about AI search. `ai-train` covers using your content to fine-tune model weights, and that's the one most publishers reserve, because it's the use with no link back and no traffic. Allowing the crawl keeps me discoverable; the signal states the terms of use. You don't have to choose between visibility and a position on training.

### Can robots.txt actually stop AI crawlers?

No, not on its own, and it's worth being honest about that. robots.txt, and Content Signals with it, is a *request* that compliant operators honor. The major crawlers, GPTBot and ClaudeBot among them, do obey it. But a `Disallow` or an `ai-train=no` does nothing to a crawler that ignores the rules. Stopping those needs a firewall or an edge rule, not a line in a text file. So treat these directives as a stated preference with real weight among the companies that matter, not as DRM. Anyone who tells you a robots.txt line "blocks" a bad actor is overselling it.

## The AI Crawlers Worth Naming

You can't set a policy for bots you can't name. There are two buckets that matter for your decision, plus one entry that almost everyone misreads.

### Training crawlers vs. answer engines

Sort the zoo into two buckets, because your policy can differ by what the bot is *for*.

Answer and search engines fetch pages to cite in real time: OAI-SearchBot and ChatGPT-User from OpenAI, PerplexityBot, Claude-Web and Claude-User, Applebot. These are the citations you want, so allow them. Training and dataset crawlers collect content to train models: GPTBot (OpenAI), ClaudeBot and the legacy anthropic-ai (Anthropic), CCBot (Common Crawl), Bytespider, Applebot-Extended, Meta-ExternalAgent. Allow the crawl if you like, but this is the bucket your `ai-train=no` signal is talking to. You don't need to memorize the list; a community-maintained master list tracks the tokens as they change.

### The one everyone gets wrong: Google-Extended

`Google-Extended` isn't a crawler and doesn't fetch anything. It's an opt-out token that governs whether your already-crawled content feeds Google's generative AI, Gemini and Vertex. Googlebot still crawls for Search no matter what you set. The catch is that it's a single coarse switch covering both AI training and AI grounding, so you can't tell Google "ground yes, train no" through robots.txt alone. I left it allowed and let the Content Signal carry the nuance it can't. That's the honest limitation nobody spells out.

## Writing the Rules (and Why Most Guides Get the Groups Wrong)

The mechanics are simple, except for one rule in the protocol that quietly breaks naive robots.txt files. Get this part right and the rest is copy-paste.

### The most-specific-group rule that trips everyone up

A crawler obeys only the single most-specific `User-agent` group that matches it. It doesn't merge your `User-agent: *` block with its own named block. That's the rule in RFC 9309, and it has a consequence most guides skip: if you put a `Content-Signal` or a `Disallow` only under `User-agent: *`, then GPTBot, which matches its own `GPTBot` group, never sees it.

So every directive you want a named bot to honor has to be repeated inside that bot's own group. It looks redundant when you write it out, twenty groups each carrying the same Content-Signal line. It's also the only way the rules actually apply to the bots you're targeting: under the protocol, a crawler obeys the group matching its own name and falls back to the `*` group only when no named group matches, so the two are never merged. ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309))

### Setting Content Signals (search / ai-train / ai-input)

The syntax is one line inside a user-agent group: `Content-Signal:` followed by `search`, `ai-train`, and `ai-input`, each set to `yes` or `no`. Because of the group rule above, it goes in the `*` block, for search engines and unlisted bots, and again in each named AI-bot group. The canonical line I used is `Content-Signal: search=yes, ai-train=no, ai-input=yes`, and it layers on top of your `Allow` and `Disallow` rules rather than replacing them. Crawl permission and usage preference are two different statements living in the same group. ([Content Signals Policy](https://contentsignals.org))

### The file I actually shipped

The structure is a `*` group with the Content Signal and open crawling, then a group for each answer engine and each training crawler, every one repeating the signal with `Allow: /`, and finally the file-level `Sitemap:`, `Agentmap:`, and `Entitymap:` lines. You can read the whole thing live: `curl https://toddmorourke.com/robots.txt`.

Two deliberate calls in there. I dropped `Crawl-delay`, because Google ignores it and it only throttles the honest bots that bother to read it. And I list the AI bots explicitly even though I allow all of them, because allowing them on the record is itself the policy. An empty wildcard says nothing; a named `Allow` says I looked at this and decided.

## The WordPress Trap: Physical vs. Virtual robots.txt

Here's the part that made my robots.txt quietly wrong for months, and it's invisible on any CMS. Nothing errors. The file just stops telling the truth.

### A file on disk silently overrides your CMS

WordPress serves a *virtual* robots.txt it builds at request time, and plugins add lines to it through the `robots_txt` filter. That's how SEO plugins inject a `Sitemap:` line, and how my own plugins add `Agentmap:` and `Entitymap:` directives. It all works, right up until a *physical* `robots.txt` file exists in the web root.

The moment that file exists, the server returns it directly. WordPress never boots for that request, and every one of those filters silently no-ops. No warning, no log line. The file just quietly stops reflecting whatever your plugins think they're publishing. Anyone who has ever hand-edited robots.txt on WordPress has probably done this without knowing it.

### How I caught it: my Entitymap line was missing

I'd hand-added the Agentmap directive to that same file months earlier, so it was live and I stopped thinking about it. Later I shipped a plugin to advertise [my machine-readable entity map](https://toddmorourke.com/entitymap/) with an `Entitymap:` line through the `robots_txt` filter. It never showed up in production.

Same cause: the physical file was winning, so the filter never ran. The identical trap silently drops the `Sitemap:` line most SEO plugins rely on. The lesson is blunt. On WordPress, if a physical robots.txt exists, it is now your *only* robots.txt, and every directive has to live in that one file, added by hand.

## Deploying and Verifying

Ship it, then prove it two ways, because passing a scan and actually being reachable are different things.

### Edit the physical file and re-scan

Because the physical file wins, I edited it directly in the web root. It's a static file, so there's no cache layer or PHP to fight, which is a rare bit of simplicity on this host. Then I re-ran the scan. Eyeball it live with `curl https://toddmorourke.com/robots.txt`, and the two checks, one for AI-bot rules and one for Content Signals, both flip to pass. Just remember the manual reality: no plugin will manage this file once it's physical. Treat it like any hand-deployed asset and keep a version-controlled copy so the next edit doesn't start from a guess.

### Policy is not reachability, so check both

robots.txt states what a crawler *may* do. It says nothing about whether the crawler can physically *reach* your pages. A WAF, an aggressive bot rule, or a 403 to unfamiliar user-agents can block the very AI crawlers you just invited, and your friendly robots.txt won't save them. So the second check is reachability. I built a tool to [check whether an AI crawler can even reach your pages](https://toddmorourke.com/tools/ai-crawler-access-checker/), per user-agent, because policy and reachability are two separate gates. Set only the first and the permissive robots.txt is theater.

## Conclusion

A lone wildcard in your robots.txt looks hands-off, but the tools reading your site now read it as an unanswered question, and they treat it accordingly. The useful work isn't deciding to block or not. It's making a real choice, writing it so the named bots actually honor it, and making sure the file you edited is the file the server serves. On WordPress that last part is where most people, me included, quietly go wrong.

### Next Steps

- Curl your own robots.txt and confirm whether it's WordPress-virtual or a physical file sitting on disk.
- Decide an actual policy: block, allow, or allow-and-declare. Pick one on purpose.
- Write named `User-agent` groups for the AI bots you care about, and repeat your Content Signal in each of them.
- If the file is physical, put every directive in it by hand and keep it version-controlled.
- Verify with a re-scan for the rules and signals, then confirm the bots can actually reach you.

If you want your site both legible and correctly gated for AI search and agents, [answer engine optimization services](https://toddmorourke.com/services/ai-search/).

## AI-Crawler robots.txt Checklist

1. Audit your current robots.txt: curl it and confirm whether it's WordPress-virtual or a physical file on disk.
2. Decide your policy: block, allow, or allow-and-declare (allow the crawl, declare usage terms).
3. Add explicit `User-agent` groups for the AI bots you care about, both answer engines and training crawlers, not just `*`.
4. Set a `Content-Signal` line (search, ai-train, ai-input) and repeat it in every group, because named bots ignore the `*` block.
5. Handle `Google-Extended` knowingly: it's a coarse opt-out token, not a crawler.
6. If a physical robots.txt exists, edit that file directly and keep a version-controlled copy.
7. Verify: re-run an agent-readiness scan for the rules and signals, then confirm the bots can actually reach you.

## Frequently Asked Questions

 Should you block AI bots from your website?

For most businesses, no. Blocking removes you from AI answers and search features, and it doesn't stop non-compliant scrapers anyway. Allowing the crawl while declaring usage terms with Content Signals is the stronger default. Large publishers protecting licensable archives are the main exception.

  Can robots.txt actually stop AI crawlers?

Not on its own. It's a request that compliant crawlers like GPTBot and ClaudeBot honor, not a technical barrier. A crawler that ignores the rules keeps crawling. Stopping those requires a firewall or an edge rule, so treat robots.txt as a stated preference, not a lock.

  How do you block GPTBot, ChatGPT, Claude, or Perplexity in robots.txt?

Add a named `User-agent` group for each bot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and so on, with `Disallow: /`. Because of RFC 9309, the rule has to sit in each bot's own group, not just under the `*` wildcard, or the named bots won't apply it.

  What are Content Signals in robots.txt?

A 2025 standard for declaring how your content may be used: `search`, `ai-train`, and `ai-input`, each set to yes or no, layered on top of your crawl rules. Compliant AI operators read them as usage preferences, separate from whether they're allowed to crawl the page at all.

  What's the difference between ai-train, search, and ai-input?

`search` is indexing for search, including AI search. `ai-input` is real-time use in AI answers, which is the citation you generally want. `ai-train` is using your content to train model weights, the use with no link back, which is the one most publishers decline while allowing the other two.

  Does blocking or allowing AI bots affect SEO or Google rankings?

Blocking Googlebot hurts Search, but `Google-Extended` only governs Gemini use and has no effect on ranking. Allowing the AI answer crawlers doesn't hurt SEO, and it can win you [citations in AI search](https://toddmorourke.com/learn/llm-seo/). The two systems are separate, so a training opt-out won't cost you rankings.

  llms.txt vs robots.txt: which one controls AI crawlers?

They do different jobs. robots.txt is your access and usage policy: who may crawl, and how the content may be used. [llms.txt does a different job](https://toddmorourke.com/learn/open-knowledge-format/), a curated content and discovery file that points LLMs at your best material. Neither replaces the other; they work together.

  Sources

- [https://www.rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)
- [https://blog.cloudflare.com/content-signals-policy/](https://blog.cloudflare.com/content-signals-policy/)
- [https://github.com/ai-robots-txt/ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt)
- [https://contentsignals.org](https://contentsignals.org)