Cloudflare Settings That Block Scrapers but Keep AI Crawlers

Two Problems That Look Like One

The scrapers are the problem: they consume bandwidth, hit login and XML-RPC endpoints, copy content, and pollute GA4 badly enough that sitewide engagement rates are meaningless. The AI crawlers are the opportunity: they are the retrieval agents behind ChatGPT, Claude and Perplexity answers, and the visitors they send engage at organic-search rates. The should-you-block-AI-crawlers guide makes the case for letting them in; this post assumes you agree and shows how to do it without letting the scrapers in with them.

What Cloudflare Blocks By Default

The first row is the one that catches people. A brand that moved to Cloudflare after July 2025, or spun up a new zone for a redesign, has the AI block on and does not know it, and the technical audit for AI crawler blockers finds this more often than any other cause of zero AI citations.

Step 1: Turn Off The AI Block, Set AI Crawl Control

The Search / Agent / Training split, if it is live on your zone, is a real improvement: it lets a publisher allow search-style crawlers while declining training, which is the compromise many ad-funded sites want. For an ecommerce brand or an agency that wants to be cited, all three categories are on, because the retrieval agents (Agent) are what fetch your page when a shopper asks an assistant a question, and blocking them means the assistant answers from memory or from a competitor. The AI crawler list has the per-crawler purposes.

Step 2: Leave Bot Fight Mode Off

Bot Fight Mode is the free plan's one-toggle bot defense, and the temptation is to turn it on and be done. Two reasons not to, from Cloudflare's own documentation: it cannot be customized, and it cannot be bypassed by WAF custom rules. So if it decides to challenge a request you wanted through, there is no rule you can write to fix it. Verified bots are supposed to be exempt, and in practice the reports of it interfering with legitimate crawlers and with monitoring services are common enough that the safer position is off, with the targeted rules below doing the work.

The exception is an active attack, where a blunt instrument is what you want for an afternoon. Turn it on, ride it out, turn it off. For steady-state protection, it is the wrong tool for a site that wants to be crawled.

Step 3: The Skip Rule For Verified Bots

Custom rules run in order, and the first one should exempt every verified bot from everything that follows, so that no rule you write later can accidentally catch Googlebot or ClaudeBot.

This rule is the reason the rest can be aggressive. Everything below applies only to traffic Cloudflare could not verify, which is exactly the population you want to be strict with.

Step 4: Block And Challenge The Junk

Order matters: 1 skip, 2 block, 3 challenge, 4 challenge. A verified crawler never reaches 2 through 4. A real customer in Singapore gets a challenge page once and then browses normally. A datacenter session that cannot solve the challenge stops existing in your analytics, which is the point of section 11.

One more note on the free plan: the number of custom rules is capped (five at the time of writing), so the four rules above plus one rate limit use most of it. If you need more, combine rules 3 and 4 into one expression with an "or" between the geo clause and the user-agent clause; the action is the same Managed Challenge, and the Skip rule still protects the crawlers.

Step 5: Rate Limits On The Paths Scrapers Hit

The rate limit is what catches the scraper that passed rule 4 by sending a real browser's user agent. A person reads a page every twenty seconds; a scraper requests three a second. The threshold in the example is generous on purpose; tighten it after a week of reading Security Events.

Paid Plans: Super Bot Fight Mode And Bot Scores

On Pro and above, Super Bot Fight Mode replaces rule 4 with something better, because it uses Cloudflare's detections rather than a user-agent list.

On Bot Management (Enterprise, or as an add-on), you get the numeric bot score and Cloudflare's own recommended rules become available as expressions: block cf.bot_management.score eq 1, challenge score between 2 and 29, both with "and not cf.bot_management.verified_bot". Cloudflare's documentation recommends exactly that pair, and it is what the free-plan rules in sections 6 and 7 approximate. The only thing the paid tiers add for this purpose is precision; the architecture is the same.

Robots.txt Still Does Its Job

Cloudflare and robots.txt are two layers, and the mistake in either direction is thinking one replaces the other.

• Robots.txt governs the polite crawlers. GPTBot, ClaudeBot, PerplexityBot, Googlebot and Bingbot read it and obey it. A Disallow in robots.txt stops them before Cloudflare ever sees a request. If you want them in, robots.txt has to say so, and ours does: a comment line welcoming AI crawlers, Allow for everything except admin, search and API paths, and a pointer to llms.txt.
• Cloudflare governs the impolite ones. Scrapers, headless browsers and datacenter fetchers do not read robots.txt. Only a rule at the edge stops them.
• Cloudflare's managed robots.txt can prepend directives to yours (it is the mechanism behind its content-signals and pay-per-crawl initiatives). If it is on, your published robots.txt is not the one you wrote; check the live file, and turn the managed option off if the prepended lines conflict with your policy.
• A robots.txt Allow with a Cloudflare block is the worst state. The crawler is invited, requests the page, and gets a 403. Some crawlers back off for weeks after that. Section 10 catches it.

The robots.txt guide for AI crawlers covers the file itself; the llms.txt guide covers the companion file the polite crawlers may also read.

Verifying You Did Not Block The Good Bots

The last check confuses people, so to be explicit: a fake GPTBot from your laptop being challenged is the system working. The real one, from OpenAI's verified ranges, is what check 02 confirms is getting through.

What Changed In Our Analytics

Before: about half of GA4 users in thirty days from one country at one second of engagement; sitewide engagement rate 20.6%, 7 seconds; every channel-versus-sitewide comparison wrong by about a factor of two. After the geo challenge and the rate limit went in, the datacenter sessions could not complete a challenge, so they stopped reaching the page and stopped firing GA4. We will publish the after numbers once there is a clean month; the honest expectation from the arithmetic in the AI visitor conversion post is a sitewide engagement rate near 40%, which is roughly what organic search already showed.

What did not change: verified crawler traffic, checked in AI Crawl Control and Search Console, and the AI assistant referral channel, which continued to grow. That is the whole test of the configuration. If bot-filtered sitewide engagement rises and crawler traffic holds, you blocked the right things.

The Full Configuration

The counterweight, since we sell the visibility work that this protects: this is an hour in the Cloudflare dashboard and it is free. Nobody needs an agency to do it, and an agency that charges to "unblock AI crawlers" is charging for switching off one managed rule. What is worth paying for is the content that makes the crawlers worth letting in; the configuration above just opens the door.