Free tool
The AI crawler checker.
Paste your robots.txt and see which AI crawlers you allow, which you block, and which line of your file decided it. Eleven crawlers, split into the two kinds that behave completely differently.
No email, no signup, no account. It runs in your browser and nothing you paste leaves this page.
The checker
Paste your robots.txt.
The distinction that matters
Two kinds of crawler, and only one of them decides whether you are in today's answer.
Almost every guide treats "AI crawlers" as one thing. They are not one thing, and confusing them is how a business ends up invisible in AI answers while believing it made a principled stand about training data.
Kind one
Training crawlers
These collect pages to build a future model. If you block one, the next model is trained without you, and absolutely nothing changes about the answers people get today.
Two of the five are not crawlers at all. Google-Extended and Applebot-Extended are permission tokens: they govern what may be done with pages that Googlebot and Applebot already fetched.
- GPTBot
- Google-Extended
- ClaudeBot
- CCBot
- Applebot-Extended
Kind two
Retrieval and search crawlers
These fetch pages while an assistant is answering a live question, or build the index it searches. Block one and you cannot be cited in those answers. Not ranked lower. Absent.
This is why blocking GPTBot and blocking OAI-SearchBot are not the same decision, even though both belong to OpenAI and both are one line in the same file.
- OAI-SearchBot
- ChatGPT-User
- Claude-SearchBot
- PerplexityBot
- Perplexity-User
- Bingbot
Google is the exception that proves the split. Its AI Overviews are built from Google's ordinary search index, so the crawler that governs them is Googlebot, not Google-Extended. Blocking Google-Extended keeps you out of Gemini's training and grounding and leaves Search and AI Overviews exactly as they were. Googlebot is a general search crawler rather than an AI-specific one, so it is not in the list above, but if you block it you have blocked far more than AI.
Under the hood
How the checker decides, and why a search for the word "GPTBot" would get it wrong.
Most quick robots.txt checks look for a bot name and a nearby Disallow. That is not how the file works, and it produces answers that are not just imprecise but backwards. A file can contain the line Disallow: / and still allow GPTBot everywhere. A file can name GPTBot and block it without the word Disallow appearing anywhere near it.
This tool follows the rules crawlers actually apply, in four steps.
1. The file is read as groups, not lines
Consecutive User-agent lines form one group header. The first Allow or Disallow closes the header, and the next User-agent line after that starts a new group. Two groups that name the same crawler are merged into one, which is what the standard requires and what surprises people whose file names GPTBot twice.
2. One group applies, and it is the specific one
A crawler uses the group that names it, matched without regard to case. If no group names it, and only then, it uses the * group. Every other group in the file is invisible to that crawler, however strict it is. This is the step that a keyword search cannot do, and it is the reason a blanket Disallow: / under User-agent: * means nothing to a crawler that has its own group.
3. Inside that group, the longest matching rule wins
Both Allow and Disallow are considered together. The rule whose path is longest wins, whether it permits or forbids. When two matching rules are exactly the same length, Allow wins. An empty Disallow: is the standard way of saying "nothing is forbidden", so it is treated as a permission of length zero that any real rule beats.
4. Wildcards are honoured
A * in a rule path matches any run of characters, and a $ at the end anchors the match to the end of the path. Specificity is still measured on the written rule, so /*.pdf$ counts as seven characters against /'s one.
Nothing here is proprietary. It is the behaviour described in RFC 9309 and in Google's published robots.txt specification, implemented honestly rather than approximated.
Worked example 1
User-agent: * Disallow: / User-agent: GPTBot Disallow:
Testing /: GPTBot is allowed. It has its own group, so the blanket block above it never applies to it. Every other crawler in this checker is blocked.
Worked example 2
User-agent: * Disallow: /blog/ Allow: /blog/pricing
Testing /blog/pricing: allowed. Both rules match, and the Allow path is thirteen characters against the Disallow's six, so the longer rule wins.
Worked example 3
User-agent: PerplexityBot Disallow: /*.pdf$ Allow: /
Testing /reports/2026.pdf: blocked. The wildcard rule is seven characters and the Allow is one. Testing /reports/: allowed, because the anchored rule does not match.
Worked example 4
User-agent: ChatGPT Disallow: /
Testing /: nothing is blocked. ChatGPT is not a crawler name. OpenAI's agents are GPTBot, OAI-SearchBot and ChatGPT-User, and a group naming something else is simply never matched. The checker flags this.
Honest framing
Blocking is a legitimate choice, not a mistake.
Plenty of publishers block training crawlers deliberately and for good reasons. Their archive is the asset. Being ingested into a model they were not paid for, and then competed with by that model, is a poor trade. News organisations, photographers, membership bodies and paid-research businesses all have a defensible case for the blanket block, and this tool is not going to tell them they got it wrong.
In our own scan of 300 search-visible Australian businesses, 13% blocked AI crawlers outright in robots.txt. For some of them that was a decision. For others it was a plugin default, a copied file or a line nobody has read since the site was built, and they were unaware they had opted out of anything.
The point of this checker is to tell you which of those two you are. It reports what your file currently does. It does not tell you what it should do.
One thing worth knowing before you decide: robots.txt is a request, not a lock. The named crawlers here publish their agents and state that they honour it, and in practice they do. But nothing in the file enforces anything, and it cannot recall content that has already been collected. If you want enforcement rather than a request, that is a server or firewall rule.
The full study is published with its dataset and its method, including the denominator behind every figure. The 13% is counted over all 300 businesses.
Questions
Common questions about AI crawlers and robots.txt.
Where do I find my robots.txt file?
yourdomain.com/robots.txt. Open that address in a browser, select everything on the page and paste it into the checker. If the address returns a 404, you do not have a robots.txt file, which means nothing is blocked and every crawler is allowed everywhere.Why can't this tool fetch my robots.txt for me?
Does blocking GPTBot remove my business from ChatGPT?
What is the difference between Google-Extended and Googlebot?
When an Allow rule and a Disallow rule both match, which one wins?
Disallow: /blog/ and Allow: /blog/pricing both match /blog/pricing, but the Allow path is longer, so the page is crawlable. When two matching rules are exactly the same length, Allow wins. Rules in a group that does not apply to the crawler are not considered at all, however specific they are.Does robots.txt actually stop AI crawlers?
Keep reading
Your file is one input. It is not the lever.
Across the 137 businesses in our study with complete records from both engines, the correlation between on-site AI readiness and per-engine visibility was r = 0.116. A crawlable site is necessary and nowhere near sufficient.
The research
State of AI Visibility 2026
300 Australian businesses scanned. Open dataset, published method.
Read more →Reference
AI visibility glossary
32 terms defined in plain English, crawlers included.
Read more →Learn
Why doesn't ChatGPT mention my business?
Six reasons, ranked by how often we found each one.
Read more →Learn
Show up in Google AI answers
How AI Overviews choose which businesses to surface.
Read more →Crawlable is the floor, not the finish line.
Your robots.txt decides whether AI is allowed to read you. It does not decide whether AI names you. A free scan asks ChatGPT and Gemini real buyer questions and counts who they actually recommend.