robots.txt tester for AI bots: GPTBot, ClaudeBot and more

This robots.txt tester shows which AI crawlers your file lets in. Paste your robots.txt, type the page you care about, and it gives a verdict for 16 bots, with the group and the line that decided each one.

It follows RFC 9309, the robots.txt standard. It runs in your browser, and your file never leaves your device. It starts with an example that blocks AI training and keeps AI search.

How to test your robots.txt for AI bots

1. Open your own robots.txt in a browser. The standard puts it at the top level of your site, as in https://www.example.com/robots.txt.
2. Copy all of it and paste it into the box, over the example.
3. Type the page you want to check, either a path like /blog/ or a full address. A path ending in /robots.txt is always allowed, because every bot has to read that file.
4. Read the table. Each row says Allowed or Blocked, which group of rules the bot read, and the line that decided it.
5. To test a bot that isn't in the list, type its name in "Another bot". Bot names in robots.txt use only letters, hyphens and underscores.

The four buttons above the box load the policies from my AI crawlers guide: allow every bot, block AI training, block AI training and search, and a file with a common mistake in it.

What the verdicts mean

Allowed means the bot may crawl that page under your file. Blocked means a rule tells it not to. The "Why" column names the group the bot read: its own group, when a User-agent: line names it, or the * group, which covers every bot without a group of its own.

A bold line under a verdict is a warning. "It has its own group" appears when your * group has Disallow rules and this bot skips them, because a group names it. A rule you wrote for every bot doesn't reach it. Load the override mistake example to see it: OAI-SearchBot gets into /cart/, and GPTBot, which has no group, stays out.

Three bots also carry a note when they're blocked. Their operators say they may not follow robots.txt, because a person asked for the page: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores" them, and Meta says Meta-ExternalFetcher may bypass them.

If some lines in your file had no effect, a box under the table lists them with the reason: a misspelled record, a rule that comes before any User-agent: line, or a path without its leading slash.

The bots it checks

Fourteen of the 16 are AI bots, from the table in my AI crawlers guide, where each one is described from its operator's own documentation. I re-read every operator's page on 2026-10-24 before publishing this tool.

Operator	Bots
OpenAI	GPTBot, OAI-SearchBot, ChatGPT-User
Anthropic	ClaudeBot, Claude-SearchBot, Claude-User
Perplexity	PerplexityBot, Perplexity-User
Google	Google-Extended, Googlebot
Apple	Applebot-Extended, Applebot
Meta	meta-externalagent, Meta-WebIndexer, Meta-ExternalFetcher
Common Crawl	CCBot

Googlebot and Applebot are the two companies' search crawlers. They're in the list so you can see whether a policy aimed at AI also touches Google Search or Apple's search in Spotlight, Siri and Safari. Google-Extended and Applebot-Extended don't crawl anything: they're names that tell Google and Apple whether pages their normal crawlers collect may be used for AI.

How the tester reads your file

It follows the rules RFC 9309 sets for crawlers:

• A group is one or more User-agent: lines followed by rules. Names are matched without regard to case, and two groups naming the same bot are combined.
• A bot reads only the groups that name it. The * group applies only when no group names the bot, and with no groups at all, nothing is blocked.
• Of the rules that match the page, the longest one wins. When an Allow and a Disallow are the same length, Allow wins. With no matching rule, the page is allowed.
• * in a rule matches any run of characters, and $ at the end means the page address must end there. Paths are case-sensitive, so /Private and /private are different pages.
• Characters outside plain ASCII are percent-encoded the same way in the rule and the page before comparing, as the standard requires.
• Rules before the first User-agent: line are ignored, and a Sitemap: line doesn't end a group.

I tested the matcher against the examples in RFC 9309 itself, the three policies and the mistake from the guide, whose verdicts I had recorded with an earlier matcher, and edge cases such as percent-encoding and misplaced rules. All 35 checks passed.

What this tester can't tell you

Whether the bots can reach your site. It reads only the text you paste. A firewall, a bot-blocking setting at your host or a CDN rule can turn a bot away before it ever reads robots.txt. TJ Robertson, who founded TJ Digital, explains how to check whether your website is blocking AI bots, including Cloudflare and hosting settings.

Whether a bot obeys. robots.txt is a request, and RFC 9309 says it "is not a substitute for valid content security measures." A page that must stay private needs a login.

Every crawler's quirks. The standard lets crawlers accept common misspellings of a record, so a misspelled line may work for one bot and not another. A name written with a version number, such as GPTBot/1.4, isn't a valid name under the standard. The tester reads it as the letters, hyphens and underscores before the slash, and lists the line under the table so you can fix it.

Your live file. It doesn't fetch robots.txt from your site, so nothing you test passes through a server. Pasting takes one copy.

Your file stays on your device

The tester is a small script that loads with this page. After that, it makes no network requests: I recorded every request the page made while I loaded each example, pasted a file, changed the page and added a bot name, and there were none. It doesn't store anything either, so a reload starts from the example again unless your browser restores what you typed.

The tool needs JavaScript to check your own file. Without it, the page shows the example file's verdicts and this explanation.