Definition
AI crawler
An AI crawler is a bot such as GPTBot, ClaudeBot or PerplexityBot that fetches web pages either to train a language model or to retrieve sources for a generated answer.
The distinction that matters is training versus retrieval. GPTBot and Google-Extended collect content for model training. ChatGPT-User, OAI-SearchBot and PerplexityBot fetch pages to answer a question someone is asking right now. Each uses its own user agent, so a site can allow retrieval and refuse training, and many sites mean to do exactly that without realizing the two are separable.
Access is decided by more than robots.txt. A named group in robots.txt is only consulted by crawlers that read it, the most-specific matching group wins so a permissive `User-agent: *` is irrelevant once a named group exists, and a WAF or bot-fighting rule that returns 403 to an AI user agent has the same effect as a Disallow while being invisible in robots.txt.
Most AI crawlers do not execute JavaScript. A page whose primary copy only exists after client-side rendering is, to them, an empty shell, which is why server-rendered HTML is a readiness prerequisite rather than a performance preference. Checking access means fetching the page with the crawler's user agent and reading what comes back, not reading your own configuration.
Frequently asked questions
Should I block AI crawlers?
It depends on which ones. Blocking retrieval crawlers removes your pages from the AI answers your buyers are already reading. Blocking training crawlers has no effect on those answers, so the two decisions are separate and should be made separately.
How do I know if an AI crawler can reach my site?
Request your own pages with the crawler's user agent and check the status code and the body. A 403 from a CDN rule and a Disallow in robots.txt look identical to the crawler and completely different in your configuration.