Skip to content
Free AI search tool · By Probeli AI

Free robots.txt generator for AI crawlers

Many AI crawlers read websites today. Some collect pages to train AI models. Others read a page to answer a question in an AI search engine, often with a link to you. If you block the wrong one, your site can drop out of those answers.

Start from a preset, or choose Allow or Block for each crawler. Add the paths to keep out and your sitemap. You get a robots.txt file to copy. If your site already has one, add these lines to it. It runs in your browser.

Quick start

14 allowed · 5 blocked

AI crawlers

OpenAI

  • GPTBotOpenAI

    Collects pages that OpenAI may use to train its models.

  • ChatGPT-UserOpenAI

    Reads a page when a ChatGPT user asks about it.

  • OAI-SearchBotOpenAI

    Builds the index behind ChatGPT search.

Anthropic

  • ClaudeBotAnthropic

    Collects pages that Anthropic may use to train Claude.

  • Claude-SearchBotAnthropic

    Builds the index Claude uses to search the web.

  • Claude-UserAnthropic

    Reads a page when a Claude user asks about it.

  • anthropic-aiAnthropic

    An older Anthropic name. Some sites still list it.

Perplexity

  • PerplexityBotPerplexity

    Builds the index behind Perplexity's answers.

  • Perplexity-UserPerplexity

    Reads a page when a Perplexity user asks about it.

Google

  • Google-ExtendedGoogle

    Lets Google use your pages for Gemini, in training and in its answers.

    Blocking this doesn't remove you from Google Search.

Common Crawl

  • CCBotCommon Crawl

    Collects pages for Common Crawl, a free copy of the web.

    Many AI models learn from this crawler's data.

ByteDance

  • BytespiderByteDance

    Collects pages for ByteDance's AI models.

Apple

  • Applebot-ExtendedApple

    Lets Apple use your pages to train its AI models.

Meta

  • Meta-ExternalAgentMeta

    Collects pages that Meta may use to train its AI.

  • FacebookBotMeta

    Collects pages to improve Meta's speech recognition.

Amazon

  • AmazonbotAmazon

    Reads pages for Alexa answers and other Amazon services.

Diffbot

  • DiffbotDiffbot

    Collects pages for a data service that AI products use.

ImageSift

  • ImagesiftBotImageSift

    Collects images from the web for AI training.

Webz.io

  • OmgilibotWebz.io

    Collects news and forum pages for a data service that sells them.

Paths to keep out and your sitemap (optional)

One path a line, starting with /.

Runs in your browser. Nothing you type is sent to us.

Your robots.txt

# Rules for AI crawlers, made with Probeli AI's free robots.txt generator

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Omgilibot
Disallow: /

How to use this file

  1. Open the robots.txt at the root of your site, or make a new one.
  2. Add these lines. Replace any older lines for the same crawlers.
  3. Keep your other rules, such as the ones for Googlebot and your sitemap.

Probeli AI writes articles and publishes them to your own site.Get started

One of the jobs Probeli AI does for you

Once AI crawlers can read your site, give them pages worth quoting.

Probeli AI writes articles that answer your buyers' questions and publishes them.

Free · No account needed

Frequently asked questions

Should I block AI crawlers?

It depends on the crawler. Some crawlers collect pages to train AI models. Others read pages to answer a person's question in an AI search engine, often with a link to the source.

Blocking a training crawler keeps your pages out of future training. Blocking a search or answer crawler can keep your site out of that engine's answers.

What is the difference between GPTBot and ChatGPT-User?

GPTBot collects pages that OpenAI may use to train its models. ChatGPT-User reads a page when a person asks ChatGPT about it. OAI-SearchBot builds the index that ChatGPT search uses. You can allow or block each one separately.

Do AI crawlers follow robots.txt?

The large AI companies say their crawlers follow it. But robots.txt is a request, not a lock: a crawler chooses to follow it. To block a crawler for sure, your host or firewall has to refuse it.

Where do I put the file or code I made?

Put robots.txt and llms.txt at the root of your site, so they open at yoursite.com/robots.txt and yoursite.com/llms.txt.

Paste the schema markup into the HTML of the page it describes, in the head or the body. Most website builders have a field for custom code in each page's settings.