Can AI crawlers access your website?
Check robots.txt rules and test how your server responds to requests using each crawler identity.
Questions
Does an allowed robots.txt rule mean a crawler can read my site?
Not necessarily. A firewall, challenge or page error can still block the request. The tool reports policy and observed fetch results separately.
Is this a request from a real AI provider?
No. The tool sends requests using crawler user-agent names. A provider may use different network addresses or request behavior.
Can I allow search crawlers and block training crawlers?
You can set separate rules for named crawler identities. Check each provider's documentation before changing the policy, because crawler purposes differ.