Should I block AI crawlers from my website?

How to check your own robots.txt in sixty seconds

Type your domain followed by /robots.txt into a browser and read the file. It is plain text, it is public on every site that has one, and you do not need a tool to read it.

You are looking for any User-agent line naming an AI crawler, followed by a Disallow: /. A blanket User-agent: * with Disallow: / blocks everything, including Google. If the file does not exist at all, nothing is blocked, which for this purpose is fine.

Two things that catch people out. Plenty of sites block AI crawlers without anyone deciding to, because a security plugin, a CDN bot-management rule or a theme default did it. And a rule at the edge, in a firewall or bot filter, can block a crawler that robots.txt appears to allow, so a clean robots.txt is necessary rather than conclusive.

The six crawlers worth knowing, and what each one does

These are the six our own site check looks for in robots.txt, published on The Receipts Engine, plus the one that is missing from that list and matters most for ChatGPT.

The GPTBot and OAI-SearchBot purposes are as stated in OpenAI's own crawler documentation. The first six are the set our readiness check tests, as published on The Receipts Engine. OAI-SearchBot is not in that six, and we would rather say so here than let a readiness score imply a completeness it does not have.

The one legitimate reason to block, and who it applies to

You do not want your work used to train a commercial model, and your business does not depend on being recommended by one. That is a real position and it is not the same question as visibility.

It fits publishers, photographers, illustrators, writers, course authors, anyone whose product is the content itself. For that case, block the training crawlers and leave the retrieval crawlers alone. Blocking GPTBot and setting Google-Extended keeps your work out of training. Leaving OAI-SearchBot allowed keeps you present in ChatGPT's search answers. OpenAI documents these as separate controls precisely so the two decisions can be made separately.

What does not fit is a service business that wants customers. If a plumber blocks GPTBot to protect intellectual property, the property being protected is a page of suburb names.

What blocking actually costs, from our own 300 sites

The 39 sites in our scan that blocked at least one of the six crawlers averaged 21.3 out of 100 for AI visibility. The 261 that allowed all six averaged 40.9. That is the whole sample of 300, scanned on 30 July 2026.

Recomputed from data.csv, the open dataset behind our 2026 study. The 39 are 13% of the 300, which is the figure published in the study.

Narrow it to the businesses we have complete records for from both engines and it gets starker: 21 of the blockers are in that group of 137, and 14 of those 21 were never named by either engine.

Now the caveat, because it matters. This is a correlation across one day, not a controlled experiment. Blocking is rarely the only thing wrong with a site that blocks: 27 of the 39 blockers also had no JSON-LD structured data. A site that has told the crawlers to go away is usually a site nobody is tending. Read this as a strong reason not to block, not as proof that unblocking alone will lift a score.

Google-Extended is a separate decision from Googlebot

Google-Extended does not control whether you appear in Google Search, in AI Overviews or in AI Mode. Google's own AI features documentation says the robots.txt directives for Googlebot are the control for how sites are crawled for Search, and points site owners to Google-Extended separately for limiting AI training and grounding in Google's other systems.

So the two decisions come apart cleanly. Setting Google-Extended keeps your content out of training and grounding for those other systems. It does not hide you from the AI answers sitting on top of Google Search. Blocking Googlebot does that, and it also removes you from ordinary search results, which almost nobody intends.

Our own readiness check counts Google-Extended among the six crawlers, and it is worth being precise about why: we read it as a signal of how legible a site is trying to be to machines, not as a claim that setting it removes you from AI Overviews. The practical side of Google's AI answers is covered in how to show up in Google AI answers.

Where to start

Reading your own robots.txt is free and takes a minute, and if it is clean you have ruled out the single most mechanical cause of being invisible. Our free scan checks it along with the rest of the site and shows you what both engines actually say.