Free robots.txt generator for AI crawlers
Many AI crawlers read websites today. Some collect pages to train AI models. Others read a page to answer a question in an AI search engine, often with a link to you. If you block the wrong one, your site can drop out of those answers.
Start from a preset, or choose Allow or Block for each crawler. Add the paths to keep out and your sitemap. You get a robots.txt file to copy. If your site already has one, add these lines to it. It runs in your browser.
Paths to keep out and your sitemap (optional)
One path a line, starting with /.
Runs in your browser. Nothing you type is sent to us.
Your robots.txt
# Rules for AI crawlers, made with Probeli AI's free robots.txt generator User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: FacebookBot Allow: / User-agent: Amazonbot Allow: / User-agent: Diffbot Allow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: /
How to use this file
- Open the
robots.txtat the root of your site, or make a new one. - Add these lines. Replace any older lines for the same crawlers.
- Keep your other rules, such as the ones for Googlebot and your sitemap.
Probeli AI writes articles and publishes them to your own site.Get started
Once AI crawlers can read your site, give them pages worth quoting.
Probeli AI writes articles that answer your buyers' questions and publishes them.
Free · No account needed
Frequently asked questions
Should I block AI crawlers?
It depends on the crawler. Some crawlers collect pages to train AI models. Others read pages to answer a person's question in an AI search engine, often with a link to the source.
Blocking a training crawler keeps your pages out of future training. Blocking a search or answer crawler can keep your site out of that engine's answers.
What is the difference between GPTBot and ChatGPT-User?
GPTBot collects pages that OpenAI may use to train its models. ChatGPT-User reads a page when a person asks ChatGPT about it. OAI-SearchBot builds the index that ChatGPT search uses. You can allow or block each one separately.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls whether Google may use your pages to train its AI models, such as Gemini. Googlebot, which ranks your pages in Google Search, is a different crawler, and this file doesn't block it.
Do AI crawlers follow robots.txt?
The large AI companies say their crawlers follow it. But robots.txt is a request, not a lock: a crawler chooses to follow it. To block a crawler for sure, your host or firewall has to refuse it.
Where do I put the file or code I made?
Put robots.txt and llms.txt at the root of your site, so they open at yoursite.com/robots.txt and yoursite.com/llms.txt.
Paste the schema markup into the HTML of the page it describes, in the head or the body. Most website builders have a field for custom code in each page's settings.