GPTBot, ClaudeBot, PerplexityBot: the AI crawler and robots.txt guide

You see GPTBot in your logs and the question follows: should I block it? That cannot be answered until three things the phrase "AI crawler" routinely conflates are pulled apart.

Three families, three separate decisions

1. Training crawlers. They collect text to build datasets. They send you no traffic and cite you nowhere today; they feed the memory of future models. Blocking protects your content and steps out of the slow path.
2. Indexing crawlers. They build the index an assistant searches when it is composing an answer. Blocking them is asking not to be found at the moment an answer is written.
3. Human-triggered fetches. A user pastes your link, or asks a question your page is a candidate answer to, and the assistant goes to get it right then. That is not a scraper, that is your prospect with an intermediary. Blocking it is almost always a mistake.

The reference table

The operators marked as documented publish their own agent lists and control instructions: OpenAI, Anthropic, Perplexity and Google.

Two clarifications that prevent blind decisions. Google-Extended is not a crawler: it is a usage-control token — the content is still fetched by Googlebot, but its use in certain Google AI products is withheld, and Search ranking does not depend on it. And CCBot feeds a public archive many organisations reuse: blocking it reaches far further than the name suggests.

How robots.txt actually behaves

The format is standardised by RFC 9309. Three rules prevent nearly every mistake we see:

• A robot obeys exactly one group: the one whose User-agent matches its name most specifically. If GPTBot has its own group it ignores the * group entirely — including the rules you thought were global.
• Matching is case-insensitive and applies to a token, not to the whole User-Agent string.
• The file has no enforcement power. It is a polite request. Serious operators honour it; an anonymous scraper will not even read it. Real blocking happens at the server or CDN.

Three configurations to copy

Pick by the decision you are making, not by fashion.

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

# Human-triggered fetches: always allowed.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Indexing for answer engines: allowed — this is how citation happens.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training collection: refused.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap.xml

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Configuration C carries a cost you should take on knowingly: you are no longer a possible answer, including for a user asking for you by name. It makes sense where the value of the content is its exclusivity — paid research libraries, licensed archives — and rarely for a commercial site.

Content-Signal: stating permitted use, not just access

robots.txt expresses a binary permission: come in, or don't. It says nothing about what may be done with the content once read. The Content-Signal header, proposed to fill that gap, separates three uses and lives in the same file:

Content-Signal: search=yes, ai-input=yes, ai-train=no

# search=yes    — index it and link to it
# ai-input=yes  — quote it in a generated answer, with attribution
# ai-train=no   — do not use this content as training data

An honest caveat: this is a convention, not an adopted standard, and it carries no enforcement. Its value is twofold — it states an unambiguous intent, and it leaves a dated record of that intent. That is little, but it is more than silence. Our own robots.txt carries it, because a product that audits other people for this had better hold it itself.

Verifying a robot is who it claims to be

A User-Agent is a free-text string: any script can announce itself as GPTBot to slip past a filter, and aggressive scrapers frequently do. Before drawing any conclusion from a log line — and certainly before blocking an address — verify the origin. Serious operators publish either their IP ranges or a verifiable reverse DNS record.

# 1. Which hostname owns the address seen in the log?
host 203.0.113.42
# 2. Does that hostname resolve back to the same address?
host the-hostname-you-got.example.com
# Both must agree. If they do not, the User-Agent is forged.

This two-step check is the same one long recommended for authenticating Googlebot. It avoids the most expensive symmetric mistake: blocking a genuine indexing crawler on the strength of a log line, or letting a scraper in because it typed the right name.

When robots.txt is not enough

Against a client that ignores the file, the only effective answer sits at the server or CDN. On nginx a User-Agent refusal is three lines; it runs before any application code, so it costs almost nothing.

map $http_user_agent $block_ai {
    default 0;
    ~*(Bytespider|ClaudeBot|GPTBot) 1;
}

server {
    if ($block_ai) { return 403; }
}

Two cautions before generalising this. A block that is too broad always ends up catching an agent you meant to keep — reread the list before applying it. And a refusal served with a 200 status and a styled error page is not a refusal: return a real 403, or both sides will be wrong about what happened.

The mistakes that cost the most

• A staging Disallow: / pushed to production. Always the first suspect when visibility collapses overnight.
• Blocking CSS and JavaScript. Engines that render the page can no longer understand it, and you lose on both fronts.
• Blocking .md files. Those are precisely the clean version you wanted agents to read.
• Believing a block removes what already exists. robots.txt affects future visits; it deletes nothing already collected and nothing a model has already memorised.
• Forbidding access, then wondering about the absence. The most frequent case we see at audit: the site forbids exactly what it wants cited, often through a setting left behind by a plugin.