Free tools / AI crawler access checker
Can ChatGPT, Claude and Perplexity read your site?
robots.txt is not the whole story: a CDN or WAF rule can answer 403 to a crawler while a browser gets the page. This check fetches the page as each crawler and reports what each received, by bot class, the way OpenAI and Anthropic document them.
Questions
Which crawlers does it fetch as?
Googlebot and bingbot (search); OAI-SearchBot, Claude-SearchBot and PerplexityBot (AI search); ChatGPT-User, Claude-User and Perplexity-User (user-triggered fetchers); GPTBot, ClaudeBot and CCBot (training). Each class is reported separately because blocking each costs something different.
What does blocking a training crawler cost?
Nothing a user sees. OpenAI documents GPTBot as training-only and OAI-SearchBot as the crawler whose opt-out removes a site from ChatGPT search answers; Anthropic documents the same split. Disallowing the training crawlers is a policy choice, and the check reports it as one.
Why did a crawler get a 403 when robots.txt allows it?
Bot protection at the edge. Cloudflare blocks AI crawlers by default on new zones, and many WAF rules challenge anything that is not a browser. The fix is a rule for the crawlers you want, not a robots.txt line.
The other free checks
- robots.txt AI crawler checker
- llms.txt checker
- Sitemap checker
- Structured data checker
- Security headers checker
- Full SEO and AI-search audit
- AI visibility checker
All of them run locally too: npx @rankcli/cli audit --url yoursite.com.