---
title: "Free robots.txt Generator for AI Crawlers – Probeli AI"
description: "Choose which AI crawlers may read your site, from GPTBot to ClaudeBot and PerplexityBot. Add the paths to keep out and your sitemap, then copy it."
url: "https://probeli.ai/tools/robots-txt-generator"
---

Free AI search tool · By Probeli AI

# Free robots.txt generator for AI crawlers

Many AI crawlers read websites today. Some collect pages to train AI models. Others read a page to answer a question in an AI search engine, often with a link to you. If you block the wrong one, your site can drop out of those answers.

Start from a preset, or choose Allow or Block for each crawler. Add the paths to keep out and your sitemap. You get a robots.txt file to copy. If your site already has one, add these lines to it. It runs in your browser.

One of the jobs Probeli AI does for you

## Once AI crawlers can read your site, give them pages worth quoting.

Probeli AI writes articles that answer your buyers' questions and publishes them.

[Get a free audit of your site](https://probeli.ai/free-ai-site-audit?utm_source=tools&utm_campaign=robots-txt-generator)

Free · No account needed

## Frequently asked questions

**Should I block AI crawlers?**

It depends on the crawler. Some crawlers collect pages to train AI models. Others read pages to answer a person's question in an AI search engine, often with a link to the source.

Blocking a training crawler keeps your pages out of future training. Blocking a search or answer crawler can keep your site out of that engine's answers.

**What is the difference between GPTBot and ChatGPT-User?**

GPTBot collects pages that OpenAI may use to train its models. ChatGPT-User reads a page when a person asks ChatGPT about it. OAI-SearchBot builds the index that ChatGPT search uses. You can allow or block each one separately.

**Does blocking Google-Extended remove me from Google Search?**

No. Google-Extended only controls whether Google may use your pages to train its AI models, such as Gemini. Googlebot, which ranks your pages in Google Search, is a different crawler, and this file doesn't block it.

**Do AI crawlers follow robots.txt?**

The large AI companies say their crawlers follow it. But robots.txt is a request, not a lock: a crawler chooses to follow it. To block a crawler for sure, your host or firewall has to refuse it.

**Where do I put the file or code I made?**

Put robots.txt and llms.txt at the root of your site, so they open at yoursite.com/robots.txt and yoursite.com/llms.txt.

Paste the schema markup into the HTML of the page it describes, in the head or the body. Most website builders have a field for custom code in each page's settings.

## More free tools

- [AI recommendation checker](https://probeli.ai/tools/ai-recommendation-checker)
- [llms.txt generator](https://probeli.ai/tools/llms-txt-generator)
- [Schema markup generator](https://probeli.ai/tools/schema-markup-generator)

[All free tools](https://probeli.ai/tools)
