GPTBot, ClaudeBot & PerplexityBot: The Complete robots.txt Guide.

The Three Jobs Every AI Bot Does

This taxonomy is the whole guide. Get it right and every configuration decision becomes obvious.

Job one: training collection

These crawlers harvest content that may be used in future model training runs. Their visits produce no immediate citations — they shape what next year's models know from memory rather than what this year's models retrieve today. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, cohere-ai, and AI2Bot live here.

Blocking these has a real but delayed and diffuse cost, and essentially no short-term visibility cost. Publishers with licensing leverage increasingly block them. Most ecommerce brands conclude their marginal contribution to a training corpus is worth nothing to them and leave them open. Both positions are defensible.

Job two: search index crawling

These build the retrieval indexes that AI answers are assembled from at query time. OAI-SearchBot populates ChatGPT's search, Claude-SearchBot supports Claude's web answers, PerplexityBot feeds Perplexity's index, Amazonbot feeds Amazon's systems.

Blocking any of these removes you from that engine's answers. Not reduces — removes. If citation visibility matters to you at all, these should be open.

Job three: user-triggered fetching

These fire when a specific person asks a question and the assistant goes to retrieve a particular page to answer it. ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, and MistralAI-User are in this group.

Blocking these accomplishes almost nothing defensively and breaks the exact moment a real human being tries to hand your page to their assistant. There is very little argument for it.

The Complete 2026 Bot Reference

Every user-agent an ecommerce brand needs to have an opinion about, grouped by operator.

Roughly twelve of these genuinely matter for an ecommerce brand, operated across six organizations. The full annotated list with crawl behavior and IP verification detail lives in our AI crawler list for ecommerce.

Config A: Maximum AI Visibility

This is the right configuration for the large majority of ecommerce brands. You want to be found, cited, and recommended, and you have no licensing leverage to protect. Everything open, private paths excluded.

Note the wildcard block at the top handles the paths no bot should crawl. The named blocks below it are explicit allows, which matter because some hosting platforms and CMS defaults inject restrictive rules you may not have written.

Config B: Block Training, Keep Citations

This is the nuanced position, and it is only available because training and retrieval run on separate tokens. You opt out of contributing to model training while remaining fully eligible to be cited in AI answers.

It makes sense if your content is your product — original research, proprietary data, editorial work you license — rather than marketing for something else you sell.

Config C: Full Block

Included for completeness. This makes you invisible in AI answers. For an ecommerce brand it is almost always the wrong call, and you should be able to articulate exactly why you are choosing it before you deploy it.

Even in the full-block configuration, Googlebot stays open. Section six explains why that is not optional.

The Google Trap Nobody Warns You About

Here is the constraint that surprises people: you cannot opt out of Google's AI Overviews without opting out of Google Search.

Both run through Googlebot. There is no separate token for AI Overviews or AI Mode. If Googlebot can crawl your page, that page is eligible to appear in an AI Overview, and the only way to prevent it is to block Googlebot — which removes you from Google Search entirely. For an ecommerce brand that is not a trade-off, it is self-destruction.

What Google-Extended actually controls

Google-Extended is frequently misunderstood as the AI Overviews opt-out. It is not. It governs whether your content is used for Gemini and Vertex AI model development. Blocking it has no effect on Googlebot's crawling, on your Search rankings, or on whether you appear in AI Overviews.

The practical read: there is no configuration that gets you Google Search without AI Overview eligibility. Anyone selling you one is mistaken. The detail on how those surfaces differ is in our Google AI Mode versus AI Overviews guide.

Control Tokens That Are Not Crawlers

Two of the most-discussed entries in every AI robots.txt article do not correspond to any bot that will ever appear in your server logs.

Google-Extended and Applebot-Extended are permission tokens. They exist purely so publishers can express a data-use preference through robots.txt. No user-agent by either name fetches pages. If you go looking for Google-Extended in your access logs to confirm your rule is working, you will find nothing, and nothing is the expected result.

This trips people up in a specific way: they add the token, see no log entries, conclude the rule is broken, and start changing things that were fine.

The corresponding real crawlers are Googlebot and Applebot, which do appear in logs and which you should generally leave alone.

The Bots That Ignore robots.txt

robots.txt is a norm, not an enforcement mechanism. It works because well-behaved crawlers choose to read and honor it. Some do not.

Compliance is opt-in

The major labs — OpenAI, Anthropic, Google, Apple, Perplexity — document their user-agents and state that their crawlers respect robots.txt directives. In practice they largely do, and you can verify it from your own logs.

Three caveats matter and none of them are optional to understand:

• A directive only works if the bot reads it. Bytespider in particular has a poor record. HAProxy reported in 2024 that a very large share of AI crawler traffic across their customer base came from Bytespider alone, much of it disregarding disallow rules. CCBot and Diffbot have mixed records.
• User-agents can be spoofed. Anything can claim to be anything. The major operators publish IP ranges precisely so you can verify that a request claiming to be GPTBot actually originated from OpenAI infrastructure.
• Real-time agent fetches are a gray zone. When an AI browsing agent operating on behalf of a logged-in human fetches a page, whether that is a crawler subject to robots.txt or a user-agent acting as a proxy for a person is genuinely unsettled.

Server-Level Enforcement

When you genuinely need a block rather than a request, it has to happen above robots.txt. The pattern is to return a 403 based on user-agent matching.

Apache

Nginx

Cloudflare

A WAF custom rule matching on User Agent with a Block action is the cleanest option if you are already behind Cloudflare, and it applies before the request reaches your origin. Cloudflare also ships managed AI-bot controls, which are convenient and are also the single most common cause of accidentally blocking bots you wanted — see the next section.

The Silent Blockers Costing You Citations

This is the section that matters most for brands who believe their configuration is fine. A perfect robots.txt can be completely overridden by something else in your stack, and none of these announce themselves.

• Cloudflare bot management. Managed rules and bot-fight settings can challenge or block AI crawlers regardless of what your robots.txt says. This is the most common cause of unexplained invisibility, and it is often enabled by default or turned on by a well-meaning developer.
• WAF rules matching by ASN or rate. Aggressive rate limiting looks reasonable until an indexing crawler trips it and quietly stops coming back.
• Security plugins. Several popular WordPress security plugins ship with AI-crawler blocking as a default-on feature.
• Hosting platform defaults. Some managed hosts and CMS platforms inject their own robots rules or bot filtering above your file.
• Meta directives. A page-level noarchive or nosnippet can suppress the snippet an engine would have quoted, even though the crawl succeeded.
• JavaScript rendering. If your content only exists after client-side rendering, a crawler that does not execute JavaScript retrieves an empty page. It was allowed, it fetched, and it got nothing.
• Login walls and interstitials. Age gates, cookie consent blockers, and geo-redirects can serve a crawler something entirely different from what a user sees.

Diagnosing these requires looking at server logs rather than at your robots.txt file, which is exactly why so many brands never find them. Our AI crawler technical audit walks the full diagnostic process for finding what is actually blocking you.

Verifying the Config Actually Works

Writing the file is the easy part. Confirming it does what you think requires three checks.

Check one: fetch your own robots.txt as a bot

Request your robots.txt while presenting an AI crawler user-agent string. If a CDN or WAF is interfering, you will frequently see it here first — a 403 or a challenge page rather than your file.

Check two: read your server logs

Filter access logs for each user-agent token over the last thirty days. You are looking for two things: which bots are visiting at all, and what status codes they receive. A bot you allowed that has never visited is a signal. A bot receiving 403s is a definite problem.

Remember section seven — Google-Extended and Applebot-Extended will never appear, and their absence means nothing.

Check three: ask the engines directly

The end-to-end test is behavioral. Ask each assistant a question your content should answer, then ask it what sources it used. If your pages are being retrieved, they will show up. This is slower than log analysis but it tests the whole chain rather than one link. Several AI visibility tracking tools automate the sampling.

robots.txt vs llms.txt vs Meta Directives

Three mechanisms, frequently conflated, controlling genuinely different things.

On llms.txt specifically

It is worth being straight about this. Research through 2026 suggests llms.txt has minimal measurable impact on citation rates today. What actually drives citations is domain authority, structured data, review platform presence, and third-party mentions. That does not make llms.txt worthless — it is trivially cheap to implement and may matter more later — but it is not where the gains are, and anyone presenting it as the primary lever is selling you the easy thing rather than the effective thing. Our llms.txt guide for ecommerce covers implementation and the honest evidence.

The maintenance cadence

Audit quarterly. The bot landscape produces two to four significant changes a year, and a robots.txt written eighteen months ago is missing tokens that did not exist when you wrote it. Put a recurring ninety-day reminder on it, re-check against the current bot list, and re-run the curl tests in section eleven.

For the wider technical picture, our AI visibility audit guide covers the full diagnostic, and the schema markup stack handles what happens after a crawler successfully reaches your page.