Can AI crawlers access your website?

Check robots.txt rules and test how your server responds to requests using each crawler identity.

Checks public pages only. Your result is not published.

Questions

Does an allowed robots.txt rule mean a crawler can read my site?

Not necessarily. A firewall, challenge or page error can still block the request. The tool reports policy and observed fetch results separately.

Is this a request from a real AI provider?

No. The tool sends requests using crawler user-agent names. A provider may use different network addresses or request behavior.

Can I allow search crawlers and block training crawlers?

You can set separate rules for named crawler identities. Check each provider's documentation before changing the policy, because crawler purposes differ.