AI Crawler robots.txt Generator
Decide which AI crawlers may read your site: training, AI search or user requests.
Runs in your browser. Nothing is uploaded.
Product tokens separated by commas or spaces, exactly as the crawler documents them.
More options
One per line. / blocks the whole site; /blog/ blocks one folder.
Paste it to merge: your other rules and sitemaps stay, and any old rules for the bots above are replaced by your choices.
Replaced old rules for .
Before you upload
- The file must sit at the root:
https://your-site.com/robots.txt. Rules in a subfolder are ignored. - robots.txt is a request, not a lock. Well-behaved crawlers follow it; it does not stop scrapers that ignore it. Use your firewall or CDN for that.
- Google-Extended and Applebot-Extended are control tokens: Googlebot and Applebot keep crawling for search, and the token only tells them how the content may be used.
Bot list checked against vendor documentation on .
| Status and rule used | |
|---|---|
|
Other user agents in this file
How to use the generator
- Pick a preset. Allow AI search, block training keeps your pages available to AI answer engines that link to their sources, while asking model makers not to train on them. Block all AI bots opts out of every crawler in the list.
- Adjust single bots if you need to. Each one is labeled with its operator, what it does and a link to the vendor's documentation.
- Under More options, paste your current robots.txt to merge the rules into it. Your other rules and sitemap lines stay; old rules for the listed bots are replaced by your choices.
- Copy or download the result and upload it to the root of your site, so it opens at
/robots.txt.
Training, search and user fetches
AI companies run several crawlers, and blocking each one means something different.
| Purpose | Examples | What blocking does |
|---|---|---|
| AI training | GPTBot, ClaudeBot, CCBot, Google-Extended | Asks the vendor not to use your pages to train models. It does not remove you from search. |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages can stop appearing, or being cited, in that AI search product. |
| User fetches | ChatGPT-User, Claude-User, Perplexity-User | The assistant may not open your page when a person asks it to. |
A typical "allow search, block training" file looks like this. Bots without a group of their own follow the User-agent: * rules:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Google-Extended and Applebot-Extended
These two are control tokens, not crawlers. Googlebot and Applebot keep crawling your site for search either way; the token only tells Google and Apple whether the content they crawl may be used for their AI models. Google says Google-Extended does not affect inclusion or ranking in Google Search, and Apple says pages that disallow Applebot-Extended can still appear in its search results.
The checker
Paste any robots.txt into Check a robots.txt to see which listed bots it lets in, for the home page or a path you choose. It applies the rules of RFC 9309, the robots.txt standard: a crawler follows the group that names its token and falls back to User-agent: * only when no group names it; the longest matching rule wins, and Allow wins a tie. That is why a group like User-agent: GPTBot with Allow: / cancels all your * rules for GPTBot, a common surprise.
The list contains only tokens we could confirm in each vendor's own documentation, on the date shown next to the output. Some well-known crawlers, such as ByteDance's Bytespider, have no public documentation we could check, so they are not listed; add them as custom user agents.
Want AI assistants to find the right pages once they are allowed in? Write an llms.txt file, or browse security and SEO tools in our directory.
Will blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search relies on OAI-SearchBot, so block that one too if you want to stay out of it. OpenAI notes that an opted-out site can still appear as a plain navigational link.
Does blocking Google-Extended hurt my Google rankings?
No. Google says the token does not affect inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini models and grounding.
Do AI crawlers really obey robots.txt?
The vendors in this list document robots.txt opt-outs for their crawlers. Fetchers that act on a user's request are the exception: OpenAI, Perplexity, Meta and Amazon all say theirs may not follow every rule. Independent reports have also caught some bots ignoring the file, so use your firewall for anything that must stay private.
How long until the new rules take effect?
Crawlers cache robots.txt, and the standard asks them not to rely on a cached copy for more than 24 hours. OpenAI says changes for OAI-SearchBot take about 24 hours; DuckDuckGo says DuckAssistBot stops visiting within 72 hours.
Does blocking training bots remove content they already collected?
No. robots.txt only affects future crawling. It does not remove your pages from data that was gathered before you added the rule.
SEO tools from our directory
Browse: SEO, Security, Websites
Transforms complex search data into actionable insights for improvement.
Create professional landing pages and sales funnels effortlessly.
