[01] Free Tool

AI Crawler Checker: See Which AI Bots Can Read Your Store

Q: How do I check if my site blocks AI crawlers?

Three places: your firewall or bot-management settings, your robots.txt (Disallow rules under GPTBot, ClaudeBot, PerplexityBot, and the others), and your server's response to a request that identifies as one of those bots. This tool does all three from the outside in one pass.

Enter a website. We read its robots.txt and llms.txt, then request the homepage as GPTBot, ClaudeBot, PerplexityBot, and Googlebot to catch firewall blocks that robots.txt never shows. You get a per-bot report in about ten seconds.

What gets sent: one request from our server to the site you enter for /robots.txt, /llms.txt, /llms-full.txt, and five requests for its homepage under different user agents. Nothing else. Results are cached for 10 minutes. Public sites only.

[02] What It Checks

Three Layers of Crawler Access

1. Firewall and bot management

Cloudflare's "Block AI bots" toggle, Super Bot Fight Mode, hosting WAF rules, and plugins like Wordfence can return 403 or a challenge page to crawlers. robots.txt is irrelevant if the request never gets through. The checker requests your homepage as each major AI crawler and reports the status code it got.

2. robots.txt

The file that tells well-behaved crawlers what they may read. The checker parses every User-agent group and reports, per bot, whether it is allowed site-wide, blocked site-wide, or restricted on some paths. Many stores copied a "block all AI" template in 2023 and forgot it. Full list of bots in the 2026 AI crawler list.

3. llms.txt and llms-full.txt

The plain-text files that tell assistants what your site is and which pages matter. Not a ranking factor, but cheap, and a growing number of tools read them. The checker reports whether they exist and shows the first lines. Format in the llms.txt guide.

What it cannot see

Rules inside your server config (nginx, .htaccess) that only fire on specific IP ranges, crawl-rate limits that block after many requests, and JavaScript-rendered content that a crawler fetches but cannot read. A 200 here means the door is open; it does not mean the room is readable. That is what the full audit covers.

[03] The Bots

Which Crawler Feeds Which Assistant

Blocking a training crawler does not always block the search or browsing agent from the same company. These are the ones the checker reports on.

GPTBot, OAI-SearchBot, ChatGPT-User: OpenAI training, ChatGPT search index, and live browsing when a user asks ChatGPT to open a page. Block GPTBot alone and ChatGPT search can still cite you.

ClaudeBot, Claude-SearchBot, Claude-User: Anthropic training, Claude's search index, and user-initiated fetches. Same split as OpenAI.

PerplexityBot, Perplexity-User: Perplexity's index and user-initiated fetches. Perplexity is the assistant most likely to cite a page directly.

Google-Extended, Googlebot: Google-Extended controls Gemini training and grounding. Blocking it does not remove you from Google Search or AI Overviews, which use Googlebot.

Applebot, Applebot-Extended: Siri, Spotlight, and Safari suggestions use Applebot; Applebot-Extended is Apple's AI training opt-out. Local businesses should keep Applebot allowed.

Bingbot, Amazonbot, meta-externalagent, CCBot, Bytespider: Copilot, Alexa, Meta AI, Common Crawl (which many models train on), and ByteDance. Most stores allow the first two and decide case by case on the rest.

Should You Block AI Crawlers at All? The Technical Audit Method

[04] FAQ

AI Crawler Checker Questions

How do I check if my site blocks AI crawlers?

Three places: your firewall or bot-management settings (Cloudflare has a one-click AI bot block), your robots.txt (look for Disallow rules under GPTBot, ClaudeBot, PerplexityBot, and the others), and your server's response to a request that identifies as one of those bots. This tool does all three from the outside in one pass.

Why does the tool request my homepage five times?

Once as a normal browser and once each as GPTBot, ClaudeBot, PerplexityBot, and Googlebot. If the browser gets a 200 and a bot gets a 403 or a challenge page, something upstream of robots.txt is blocking that crawler. It is the most common cause of "we are not in ChatGPT" and the least visible.

Is a 200 for every bot the same as being cited?

No. Access is the first gate, not the last. Assistants also need clear entity information, structured data, and content that answers questions. The full audit measures those; this tool only tells you whether the door is open.

Should I block AI crawlers?

For an ecommerce brand that wants to be recommended, usually no. Blocking training bots while allowing search and user agents is a defensible middle path. We wrote up the tradeoffs in should you block AI crawlers.

What does the tool send to my site?

Eight GET requests from our server: robots.txt, llms.txt, llms-full.txt, and the homepage under five user agents. No cookies, no form submissions, response bodies capped at 200 KB. Results are cached for 10 minutes, and the tool is rate-limited so it cannot be used to flood anyone.

Can I check a competitor's site?

Yes. Everything the tool reads is public. It is a quick way to see whether the brands being cited in your category have simply left the door open while you have not.

The tool says my hostname does not resolve or is not allowed. Why?

It only checks public websites on standard ports. Local addresses, private networks, staging hosts behind a VPN, and non-standard ports are refused by design.

Free · 30 minutes · No pitch deck

Door Open? Now Find Out if Anyone Walks Through.

Book a free call and we will run a few live prompts to see whether the assistants that can read your site actually recommend it.

Book a Free Call → Get the Free Ecom Profit Box