Skip to content
Free SEO tool · By Probeli AI

Free robots.txt validator and tester

Paste your robots.txt. We read every line the way crawlers do and flag the mistakes. That includes the ones that block your whole site, your sitemap or AI search. Then test any address for any crawler, such as Googlebot or OAI-SearchBot.

One wrong line in this file can keep Google out of your site. It runs in your browser, and nothing you paste is sent to us.

Probeli AI does this for you

Probeli AI writes your articles and tells search engines about each one as it goes live.

Each new article goes out through IndexNow.

Free · No account needed

Frequently asked questions

What is robots.txt?

A plain text file at the root of your site, such as example.com/robots.txt. It tells crawlers which parts of the site they may read. A mistake in it can stop Google from reading your pages, however good they are.

What are the most common robots.txt mistakes?

Three come up most. Disallow: / under User-agent: *, which blocks your whole site. A Sitemap line without the full https:// address. And a rule placed before any User-agent line, which crawlers ignore. The validator flags all three, and blocks of AI crawlers too.

Does robots.txt keep a page out of Google?

No. It only stops crawling. A blocked page can still show in Google if other sites link to it. To keep a page out, let Google read it and add a noindex tag, or remove the page.

Do I need a robots.txt if I don't block anything?

You don't have to have one. Without the file, crawlers may read every page. A short file is still useful: a Sitemap line in it helps every crawler find your sitemap.

Do you keep the file I paste?

No. The validator runs in your browser. Nothing you paste is sent to us or saved.