AI crawler

An AI crawler is software that fetches web content for an AI service, such as search, answer retrieval, or model training.

Providers use different agents and controls for different purposes. Allowing content to be retrieved for answers is a separate policy decision from allowing training use. Check the provider's documentation before changing either.

robots.txt is one part of access. A CDN, firewall, login requirement, or browser challenge can also prevent a request from reaching the page. Review the response a request receives, not only the rule you intended to publish.

Some fetchers read the HTML response without running the site's JavaScript. Confirm that important content is present there. A request with a crawler user agent can reveal a block, but it does not prove that a provider's real crawler can access the page from its own network.

Frequently asked questions

Should I block AI crawlers?

Decide separately for search, answer retrieval, and training. Review which controls each provider offers and how those choices fit your site's policy.

How can I check crawler access?

Inspect robots.txt, request the page with relevant user agents, and review server logs for actual crawler requests. A simulated request is useful evidence, not a guarantee of provider access.

Related

More terms