Free tools / robots.txt AI crawler checker
What does your robots.txt tell the AI crawlers?
A robots.txt group for "*" with Disallow: / blocks every crawler that has no group of its own, including OAI-SearchBot and Claude-SearchBot. This check parses the file as each bot does and names the crawlers that are disallowed, by class.
Questions
Does a crawler obey robots.txt?
The search and AI search crawlers from Google, Microsoft, OpenAI, Anthropic and Perplexity document that they do. robots.txt is necessary and not sufficient: a crawler the file allows can still be refused at the edge, which the AI crawler access checker tests.
Should I disallow GPTBot?
That is a policy choice about training, not visibility: GPTBot is training-only and OAI-SearchBot is the crawler behind ChatGPT search answers. Disallow GPTBot if you do not want your content used for training; keep OAI-SearchBot allowed if you want to appear in answers.
What is the right robots.txt for AI crawlers?
One group per class: allow the search and AI search crawlers and the user-triggered fetchers; decide the training group as policy. RankCLI generates that file from the audit, and the MCP server's seo_generate_robots tool writes it.
The other free checks
- AI crawler access checker
- llms.txt checker
- Sitemap checker
- Structured data checker
- Security headers checker
- Full SEO and AI-search audit
- AI visibility checker
All of them run locally too: npx @rankcli/cli audit --url yoursite.com.