Why crawler access decides whether AI can cite you
AI assistants answer from pages they can fetch. When someone asks ChatGPT which tools to consider in your category, the model either has your pages available to it or it does not — and if robots.txt disallows GPTBot, it does not. No amount of content quality changes that. Access is the precondition for everything else.
This is worth checking rather than assuming, because the file being served is not always the file you wrote.
Most blocks were never a decision
Several CDN and hosting platforms now add AI crawler blocking as a default, and they inject those rules aboveyour own in the served file. Because crawlers apply the first matching group, the operator's block wins and your rules below it never apply. We hit this on our own domain: the repository said every AI crawler was allowed, and the live file said the opposite. Reading the deployed robots.txt is the only reliable check.
The crawlers, and what each one governs
They are not interchangeable, and blocking one does not imply blocking another.
- GPTBot crawls in bulk for OpenAI, while ChatGPT-User fetches a single page live when a user asks ChatGPT to browse, and OAI-SearchBot feeds ChatGPT Search. Three different paths to being cited.
- ClaudeBot and Claude-User split the same way for Anthropic — bulk crawling versus in-conversation retrieval.
- Google-Extended is not a crawler at all. Google crawls with Googlebot regardless; this token controls whether that content may be used in Gemini and AI Overviews. Blocking it does not affect your search rankings.
- PerplexityBot matters more than its size suggests, because Perplexity leans heavily on live retrieval and cites sources explicitly.
Blocking is a legitimate choice
Publishers who sell access to their archive block these crawlers on purpose, and they are right to. The distinction worth drawing is between training and retrieval: Google-Extended and Applebot-Extended govern model training and AI answers rather than search indexing, so some operators let you allow one while restricting the other. The problem is not blocking — it is blocking without knowing.
Once the crawlers are in
Access gets you eligible, not cited. The next question is whether your pages are quotable — whether they answer a buyer's question in a passage a model can lift. If you want the file that tells assistants which of your pages matter most, our llms.txt generator builds one in a browser.
Frequently asked questions
- Why does blocking an AI crawler matter?
- AI assistants answer from pages they can fetch. If GPTBot is disallowed in your robots.txt, ChatGPT has nothing of yours to read and cannot cite you — however strong your content is. Access is the precondition for visibility in AI answers.
- Am I blocking AI crawlers without knowing?
- It is common. Several CDNs and hosting platforms add AI crawler blocks by default, injected into robots.txt above your own rules — so the file you wrote is not the file being served. Checking the live robots.txt is the only reliable way to know.
- What is the difference between GPTBot and ChatGPT-User?
- GPTBot crawls pages in bulk to train and ground the model. ChatGPT-User fetches a specific page in the moment, when a user asks ChatGPT to browse. Blocking either one removes a different path to being cited.
- Does allowing crawlers mean my content trains their models?
- It can, and that is a real trade-off. Google-Extended and Applebot-Extended govern training and AI answers rather than search indexing, so you can allow retrieval while restricting training with some operators. Decide deliberately rather than by default.