Free tool · Technical SEO
AI Bot robots.txt Inspector
Analyze your robots.txt and control with a switch whether GPTBot, ClaudeBot, PerplexityBot and other AI bots can crawl your site. Free and no sign-up.
What does blocking AI bots control?
robots.txt is the file at the root of your domain (yoursite.com/robots.txt) where each crawler declares its user-agent, and you declare what it can crawl. AI bots like GPTBot (OpenAI), ClaudeBot (Anthropic) or PerplexityBot have their own user-agent, distinct from Googlebot's. That means you can let Google index you normally while deciding, bot by bot, which ones can use your content to train AI models or generate answers.
If you've never touched your robots.txt, these bots very likely have full access by default: the absence of a rule is not the same as a block.
Why control AI bot access?
- You'll know who's crawling you: GPTBot, ClaudeBot, PerplexityBot and other AI bots often have full access by default if you never blocked them.
- You decide case by case, without touching code: one switch per bot instead of editing the file by hand.
- It doesn't affect your traditional SEO: robots.txt rules for AI bots are independent of Googlebot and your search engine rankings.
How it works
Paste or upload your robots.txt
From the textarea or by uploading the .txt file directly from your computer.
Check the bot matrix
Each bot shows its status (Allowed, Blocked or Partial) and which rule determines it.
Adjust with a switch
Allowing or blocking regenerates the robots.txt instantly, ready to copy or download.
Try the tool
100% free, no sign-up. Everything runs in your browser, no data is sent to a server.
Paste or upload your robots.txt
The analysis updates automatically as you type, paste, or upload a file.
AI bot matrix
GPTBot
OpenAI
Trains the GPT models on web content.
No applicable rules: full access.
ChatGPT-User
OpenAI
Browses the web in real time when a user requests it inside ChatGPT.
No applicable rules: full access.
PerplexityBot
Perplexity AI
Indexes content to generate AI-powered search answers.
No applicable rules: full access.
ClaudeBot
Anthropic
Trains and updates the Claude models.
No applicable rules: full access.
Google-Extended
Controls the use of your content to train Gemini (doesn't affect Google Search).
No applicable rules: full access.
Bytespider
ByteDance
Collects content for ByteDance's (TikTok) AI models.
No applicable rules: full access.
CCBot
Common Crawl
Open dataset used to train multiple third-party LLMs.
No applicable rules: full access.
FacebookBot
Meta
Collects content for Meta's AI services.
No applicable rules: full access.
Resulting robots.txt
editableFrequently asked questions
What is GPTBot and why does it show up in my robots.txt?
GPTBot is OpenAI's crawler that collects web content to train its AI models. If you don't have an explicit rule for it (or a User-agent: * block restricting it), it can crawl your site freely by default.
Does blocking these bots affect my Google rankings?
No. robots.txt has no effect on Googlebot or your Google Search rankings — they're independent rules per user-agent. Blocking GPTBot, ClaudeBot or similar only stops those AI crawlers from using your content.
What's the difference between an AI bot and a traditional search engine?
Traditional search engines (Googlebot, Bingbot) index your content to show it in search results, driving traffic to your site. AI bots like GPTBot or CCBot typically collect content to train models, without necessarily citing the source or generating traffic back.
What happens if I have no rule for these bots?
If a bot has no specific rule and there's no User-agent: * block restricting it, it has full access to your site by default.
Does this tool modify my site automatically?
No. It runs 100% in your browser: it analyzes the text you paste or upload and gives you back an updated robots.txt for you to upload manually to your server's root.
Does it cost anything?
No, it's free and requires no sign-up.
Next step
Ready to scale your organic visibility?
We talk to you, understand your business and put together a tailored proposal. No fine print.