# GPTBot, ClaudeBot, PerplexityBot: the AI crawler and robots.txt guide > Three families of robot visit your site on behalf of AI systems, and confusing them is expensive: block the wrong one and you have shut the door on the most qualified visitor you get. Here is who is who, and what to write. Published: 2026-08-22 · Updated: 2026-08-26 · 7 min read Source: https://indexonic.com/en/blog/ai-crawlers-robots-txt/ You see `GPTBot` in your logs and the question follows: should I block it? That cannot be answered until three things the phrase "AI crawler" routinely conflates are pulled apart. ## Three families, three separate decisions 1. **Training crawlers.** They collect text to build datasets. They send you no traffic and cite you nowhere today; they feed the memory of future models. Blocking protects your content and steps out of the slow path. 2. **Indexing crawlers.** They build the index an assistant searches when it is composing an answer. Blocking them is asking not to be found at the moment an answer is written. 3. **Human-triggered fetches.** A user pastes your link, or asks a question your page is a candidate answer to, and the assistant goes to get it right then. That is not a scraper, that is **your prospect with an intermediary**. Blocking it is almost always a mistake. > **The short rule** — A name ending in `-User` almost always means a fetch triggered by a real person: `ChatGPT-User`, `Claude-User`, `Perplexity-User`. Let those three in. ## The reference table The operators marked as documented publish their own agent lists and control instructions: [OpenAI](https://platform.openai.com/docs/bots), [Anthropic](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/guides/bots) and [Google](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers). | User-Agent | Operator | Family | Effect of blocking | | --- | --- | --- | --- | | GPTBot | OpenAI | Training | Excluded from future training sets | | OAI-SearchBot | OpenAI | Indexing | Absent from ChatGPT search | | ChatGPT-User | OpenAI | Human-triggered | The user gets an error instead of you | | ClaudeBot | Anthropic | Training | Excluded from future training sets | | Claude-SearchBot | Anthropic | Indexing | Absent from Claude search | | Claude-User | Anthropic | Human-triggered | The user's request fails | | PerplexityBot | Perplexity | Indexing | Absent from the Perplexity index | | Perplexity-User | Perplexity | Human-triggered | The user's request fails | | Google-Extended | Google | Usage control | Content withheld from some Google AI uses; no ranking effect | | Googlebot | Google | Indexing | Disappearance from Google Search — do not touch | | Applebot-Extended | Apple | Usage control | Withholds content from Apple's AI uses | | Bytespider | ByteDance | Training | Excluded from ByteDance collection | | CCBot | Common Crawl | Public archive | Excluded from a corpus many parties reuse | Two clarifications that prevent blind decisions. `Google-Extended` is not a crawler: it is a usage-control token — the content is still fetched by Googlebot, but its use in certain Google AI products is withheld, and Search ranking does not depend on it. And `CCBot` feeds a public archive many organisations reuse: blocking it reaches far further than the name suggests. ## How robots.txt actually behaves The format is standardised by [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). Three rules prevent nearly every mistake we see: - **A robot obeys exactly one group**: the one whose `User-agent` matches its name most specifically. If `GPTBot` has its own group it ignores the `*` group entirely — including the rules you thought were global. - **Matching is case-insensitive** and applies to a token, not to the whole User-Agent string. - **The file has no enforcement power.** It is a polite request. Serious operators honour it; an anonymous scraper will not even read it. Real blocking happens at the server or CDN. ## Three configurations to copy Pick by the decision you are making, not by fashion. *A — Maximum visibility: everyone welcome* ``` User-agent: * Allow: / Sitemap: https://your-domain.com/sitemap.xml ``` *B — The common compromise: citation yes, training no* ``` # Human-triggered fetches: always allowed. User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / # Indexing for answer engines: allowed — this is how citation happens. User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Training collection: refused. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Sitemap: https://your-domain.com/sitemap.xml ``` *C — Closed to AI, classic search kept* ``` User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: / ``` Configuration C carries a cost you should take on knowingly: you are no longer a possible answer, including for a user asking for you by name. It makes sense where the value of the content is its exclusivity — paid research libraries, licensed archives — and rarely for a commercial site. ## Content-Signal: stating permitted use, not just access robots.txt expresses a binary permission: come in, or don't. It says nothing about what may be _done_ with the content once read. The `Content-Signal` header, proposed to fill that gap, separates three uses and lives in the same file: *A machine-readable statement of use* ``` Content-Signal: search=yes, ai-input=yes, ai-train=no # search=yes — index it and link to it # ai-input=yes — quote it in a generated answer, with attribution # ai-train=no — do not use this content as training data ``` An honest caveat: this is a convention, not an adopted standard, and it carries no enforcement. Its value is twofold — it states an unambiguous intent, and it leaves a dated record of that intent. That is little, but it is more than silence. Our own `robots.txt` carries it, because a product that audits other people for this had better hold it itself. ## Verifying a robot is who it claims to be A User-Agent is a free-text string: any script can announce itself as `GPTBot` to slip past a filter, and aggressive scrapers frequently do. Before drawing any conclusion from a log line — and certainly before blocking an address — verify the origin. Serious operators publish either their IP ranges or a verifiable reverse DNS record. *Reverse lookup, then forward lookup* ``` # 1. Which hostname owns the address seen in the log? host 203.0.113.42 # 2. Does that hostname resolve back to the same address? host the-hostname-you-got.example.com # Both must agree. If they do not, the User-Agent is forged. ``` This two-step check is the same one long recommended for authenticating Googlebot. It avoids the most expensive symmetric mistake: blocking a genuine indexing crawler on the strength of a log line, or letting a scraper in because it typed the right name. ## When robots.txt is not enough Against a client that ignores the file, the only effective answer sits at the server or CDN. On nginx a User-Agent refusal is three lines; it runs before any application code, so it costs almost nothing. *nginx — an actual refusal rather than a polite request* ``` map $http_user_agent $block_ai { default 0; ~*(Bytespider|ClaudeBot|GPTBot) 1; } server { if ($block_ai) { return 403; } } ``` Two cautions before generalising this. A block that is too broad always ends up catching an agent you meant to keep — reread the list before applying it. And a refusal served with a 200 status and a styled error page is not a refusal: return a real 403, or both sides will be wrong about what happened. ## The mistakes that cost the most - **A staging `Disallow: /` pushed to production.** Always the first suspect when visibility collapses overnight. - **Blocking CSS and JavaScript.** Engines that render the page can no longer understand it, and you lose on both fronts. - **Blocking `.md` files.** Those are precisely the clean version you wanted agents to read. - **Believing a block removes what already exists.** robots.txt affects future visits; it deletes nothing already collected and nothing a model has already memorised. - **Forbidding access, then wondering about the absence.** The most frequent case we see at audit: the site forbids exactly what it wants cited, often through a setting left behind by a plugin. > **One-command check** — `curl -sS https://your-domain.com/robots.txt` — read the whole thing. Indexonic's audit goes further: it checks each agent group, detects the soft 404, and tells you which of your own settings contradicts what you say you want. ## Frequently asked questions ### Does blocking GPTBot keep me out of ChatGPT? Not directly. GPTBot feeds training; ChatGPT's search goes through OAI-SearchBot and on-demand fetches through ChatGPT-User. Blocking the first while allowing the other two is coherent: refuse training, accept citation. ### Is robots.txt legally binding? No. It is a technical convention honoured by serious operators. Effective blocking requires server or CDN filtering — by User-Agent, by IP, or by verifying that the robot is genuine. ### How do I find out who really visits my site? Your access logs. A single grep across agent names gives you a dated, counted list. It is the one source that depends on nobody's interpretation. ### Should I block CCBot? It depends on your tolerance for reuse. CCBot feeds a public archive used by a very wide range of parties, researchers included. The block therefore reaches far, and it does not undo collection already done. --- Indexonic — https://indexonic.com · https://agent.indexonic.com