What our crawler reads, and how to block it.
ProbeliAIBot fetches public web pages for Probeli AI. Probeli AI writes and publishes articles for businesses.
How to recognize it
Every request ProbeliAIBot sends carries this user agent:
Mozilla/5.0 (compatible; ProbeliAIBot/1.0; +https://probeli.ai/bot)
In robots.txt, its token is probeliaibot (in any capitalization). It downloads pages without running their JavaScript, sends no cookies, never signs in or submits forms, and never requests private or internal network addresses.
What it reads, and when
ProbeliAIBot fetches a page only when a Probeli AI customer's workspace needs it:
- Pages that AI answers cite. When an answer from ChatGPT, Gemini, Google AI Mode and Perplexity cites a page, ProbeliAIBot reads that page to find the brands it mentions. It doesn't follow the links it finds there.
- A few key pages of a customer's own website, for its business brief. When a workspace's brief is generated, ProbeliAIBot checks that website's robots.txt (reading it again unless it has a recent copy), then reads its homepage and up to 4 key pages it links to (such as about, products, pricing), or that its sitemap lists when the homepage links to too few. It doesn't follow links to other websites. A workspace's brief is written when its website is first added and whenever a manager asks for a new one; where Probeli AI has turned that on, a brief nobody has edited is also written again once it is more than 90 days old, so it keeps up with the website.
- A website's AI site audit. When a customer runs the audit for the website their workspace tracks, ProbeliAIBot reads its robots.txt, homepage, sitemaps, llms.txt and llms-full.txt, a few common API description paths (such as
/openapi.json), up to 40 pages, and a sample of the internal links on them (aHEADrequest, or a shortGET). - The top pages of Google results for a keyword, when a customer generates an article. ProbeliAIBot reads up to 8 of the pages ranking for that keyword, and, when the article is written for a topic of related keywords, for the topic's main keyword; when the article compares two products, the pages ranking for each product's name count toward the same limit, and so do the pages ranking for at most three more searches an article's research may run: one for the official page its topic needs (a product's own page, its pricing and limits pages, its release notes, a government's page, a vendor's documentation), one for a figure the results quote when the article asks for data, for a news article whose release notes it didn't find, for the product's "what's new" page or, for a list of picks or cases whose results are mostly old, for the keyword with the current year, and one for the keyword without the sites of firms that sell what the customer sells, when too few of the pages it read were by others. When a government's page for a trip's topic refuses the read, it tries up to four other official travel pages of the article's market instead (such as a health agency's destination list, an airport security agency's rules, or the EU's or the UK's entry-system pages), within the same limits. It reads PDF documents among those pages too, for their text. Each time an article's research runs, it tries at most 12 of them and requests each one once, or twice for one of the top three results whose first read timed out or came back too short (the second request counts as a try). Trying an article again reuses its research while it is recent and the article's keyword and language are unchanged; the research runs again, and can request the same pages again, when it didn't finish (the generation stopped or failed during it), when the earlier generation stopped at its cost limit, or when it is no longer recent. It skips sites such as video and social networks. Of the customer's own website it reads, within the same limits, up to 2 pages on the article's topic, and the writer reads up to 5 of its pages on the article's subject: a product's pages when the keyword names one of the customer's products, so the article describes it as the customer's own page does, and in the slots left (always one at least) the customer's own posts, guides, case studies or research on that subject, or its documentation, glossary, help or care pages when they answer the article (a term it defines, a product it reviews, a task it explains), among them those its sitemaps list, so the article doesn't contradict them. It never reads these for a refresh of a published article or a rewrite of an existing page. Before an article's brief is written, it also requests once each of up to 14 pages of the customer's own website the article may link (never the homepage, one at a time, within the same robots.txt rules) only to check that the page still exists and, for a single product's page, whether it says the product is a final sale or out of stock, and keeps nothing of it but those answers, so the article never links a page that is gone or points its call to action at a product that can't be bought. Of the links it finds on the pages it reads, it follows only those to the official site of a product the keyword names (among them its release notes), on a product's own page that the article reviews or compares, to that product's pricing and limits pages on the same site (two at most for each article), to a code repository on the article's topic (and, on a repository's page, to its license file), on a page collecting other people's statistics, to the studies it credits and to the research publishers, public bodies and studies it links, and, on a page collecting case studies, to the pages of the cases it lists (two such links at most for each article, six for an article that rounds up statistics), within the same limits. It keeps each page's address, title, headings, word count and dates, and short quotes with their link (among them the sentences the article's figures come from), as the article's sources, not the full page text.
- A customer's own sitemaps, when they generate an article. Before an article's brief is written, ProbeliAIBot reads the robots.txt, the sitemaps and the RSS or Atom feed of the website the customer's workspace tracks, then the sitemaps of its documentation (
/docs/sitemap.xmlon the same website, and those of itsdocs.subdomain after that subdomain's robots.txt), to find a post of that website that already covers the article's keyword, so the customer can update that post instead of publishing a second one. It reads the lists, not the pages they name, once per article (trying the article again while its research is recent doesn't read them again), and not at all for a week after it read all of them for an earlier article of the same workspace, whose list of addresses it uses instead (when it couldn't read them all, the next article carries on with the sitemap files the earlier one hadn't reached). For an article about case studies, when the lists name the website's case-study folder (such as/work/) but few cases in it, it also reads that folder's own page once to see which cases it links, and keeps their addresses and link text, not the page. It keeps only the address and title of each matching page, and the addresses of a few of its research, case-study, guide and post pages on the article's subject, from which the article's writer picks the pages it reads (never their text); when a matching post is in the sitemap, it is also listed in the workspace's existing pages (its address and the sitemap file that lists it), not read. When the website turns ProbeliAIBot away (a 401, 403 or 429, or a bot challenge, on its sitemaps, or a robots.txt that answers 429 or 5xx), it reads nothing more there: Probeli AI runs one Google search of the keyword limited to that website instead (it reads Google's results, not the website; for a week after Google answered such a search mostly with other websites' pages, it runs none for that workspace), the article's writer reads none of the pages it finds unless ProbeliAIBot could read the website's homepage, and the customer is told that a page may not have been checked. When the lists can't all be read in time, Probeli AI runs that Google search too, for the pages the lists it read didn't reach. - A page a customer marks as published. When a customer marks an article as published on the website it was written for (the website their workspace tracks, or tracked when the article was written), or asks to check it again, ProbeliAIBot reads the robots.txt of each address (scheme and host) of that website it reaches (unless it has a recent copy) and loads that one page to confirm the article is there. It also does this by itself about 10 minutes after a page is published, also when Probeli AI published it to the customer's site. If the page isn't there yet, it looks once more about 30 minutes later. It follows redirects only within the website the article was written for, and doesn't store the page's text. When the page is the article, Probeli AI also compares its text with the version it gave the customer, in memory during that same read, to suggest writing settings to the customer (a workspace can turn this off). It keeps only the suggestions, which hold at most 3 words of the page's own wording, never the page's text. This sends no extra request.
- The sources of a published article, once a month. When a customer's article is published, Probeli AI checks its prices, versions and dates against the pages it cites: about once a month (and when a customer asks to check it now), ProbeliAIBot reads up to 8 of those pages again, the customer's own pages through the same rules. It compares them with the article in memory during that read and keeps no page text, only a fingerprint (a hash) of each page to see whether it changed. A customer can turn this off for a workspace or an article.
- A customer's own posts and pages, when they scan or improve them. When a customer scans the website their workspace tracks to optimize its existing pages, ProbeliAIBot reads its robots.txt and its sitemaps, then up to 25 of its pages (the ones its sitemap or the customer's Search Console lists, never another website's). It reads one page again when the customer asks ("Read again"), and once more when the customer asks for refresh suggestions for that page. It keeps each page's title, headings and counts (words, lists, links) for the page's report, not its text. Refresh suggestions keep only short excerpts of the parts a suggestion would change, never the whole page; "Read again" compares the page with the open suggestions in memory and keeps nothing of its text. It never changes the page itself. It doesn't follow the links it finds there.
- The products a list article recommends, when a customer generates one. Before such an article is planned, ProbeliAIBot checks each product it may recommend on the product's own page: a page of the maker's website that the article's research or one of its searches found, then, when that website is a Shopify or WooCommerce store, the product's public data there (Shopify's
/products/<name>.js, a collection's/products.json, WooCommerce's public Store API) or the page's own structured data, only to confirm the product's current name, price, stock and return terms. For a software tool or service it may recommend, it reads that maker's pricing page (one the research or a search found on the maker's website, else the pricing link of the maker's homepage, which it loads once), only to note its lowest paid plan and whether it has a free plan or trial. It also reads the customer's own store's public product list (Shopify's/products.jsonor WooCommerce's Store API, a few pages of it) or, without one, a few of the customer's own product pages, so the article recommends only the customer's products that are in stock and can be returned, and the customer's own product pages, its own category page for the article's subject and its own guide on it, once each, so the article describes the customer's products by what their pages list. It loads once each of up to six list pages the article's research already read, only to see whether that website sells the same kind of product itself (its own cart or its own pricing page and accounts), so the article never takes a seller's word about a competitor. For a theme, template, starter or plugin it may recommend, it loads once each the live demo and the source repository that the product's page, the customer's own guide or one of those list pages links for it (and the product's page on a theme directory such as astro.build that they link), only to confirm each one is up before the article links it; for a repository that is a copy (a fork) of another, it loads the original instead and links that one. For a list of alternatives to a product, it checks that product's own homepage and pricing page the same way, only to state its current price beside the others, and reads up to three of the customer's own pages whose address names a need the article's research asks about (such as self-hosting), for their title and description. At most 60 requests for each article, one at a time for each website, within the same robots.txt rules, and never a redirect to another website. It keeps each product's name, price, stock and return terms, not the page. - Images for a customer's article. Refresh suggestions for a customer's own page download none of its images. When a customer's article lists products or tools, ProbeliAIBot loads the start of each listed item's own page (up to the end of its
<head>; only an address the article already cites or links, for up to 10 items) to read its preview image (og:image), and downloads that image; the article credits and links that page under the image. In the same way it reads the preview image of a product's own page on the customer's site when the article names that product (for up to 3 products), and that site's homepage once, to leave out the image every page shares. It keeps a resized copy of each image, not the page, and doesn't follow the links it finds there. - A page of a customer's own website, when they ask for a better title for it. On the Traffic page, a customer can ask Probeli AI to suggest a title and description for a page of the website their workspace tracks that Google shows often but few people click. ProbeliAIBot checks that website's robots.txt (unless it has a recent copy) and loads that one page to read its current title and description. It follows redirects only within that website and keeps only the title and description; a read from the last day is reused instead of loading the page again.
- A page someone checks with a free tool. On our free tools, anyone can check one page's meta tags, check a website's sitemap, or ask for blog post ideas from a website. ProbeliAIBot checks that website's robots.txt (unless it has a recent copy), then loads that one page, or for a sitemap check the sitemap the robots.txt names (else
/sitemap.xml). It reads at most 1 MB, follows at most 5 redirects, each only when that address's own robots.txt allows it, and reads nothing a robots.txt blocks. It doesn't follow the links it finds there and keeps only the tool's answer (a list of checks, a count of sitemap entries, or post ideas), not the page. - A website someone checks with the free site audit. On our free site audit, anyone can check one website. ProbeliAIBot checks that website's robots.txt (unless it has a recent copy), then reads its homepage and at most one sitemap file: the first one its robots.txt names, else
/sitemap.xml. It reads nothing a robots.txt blocks, at most 1 MB per file, and doesn't follow the links it finds. It keeps the homepage's title and description, a few page checks and up to 10 addresses from the sitemap, not the page text. - A homepage, when a customer enters a website or asks to read it again. Creating a workspace, changing its website or adding a competitor loads the top of that site's homepage, up to the end of its
<head>, to name the brand (a competitor's only when its robots.txt allows it). When a new workspace's setup searches for its competitors, ProbeliAIBot loads the top of each one's homepage the same way (at most ten), to check that the site is that brand's and sells what the customer sells before it is tracked. For the website a workspace tracks, Probeli AI also uses that read to propose the site's language and country, and ProbeliAIBot reads on, at most 200 KB past the<head>, for the page's headings and to check whether its text shows without JavaScript. A customer can ask for that same read again from their workspace's settings, to see whether ProbeliAIBot can read the site now; Probeli AI keeps only the outcome of the last read (when, whether it worked and why not), not the page.
How often
- Cited pages: one page at a time from each site, and the next page at least 1 second after the previous one (from each of our servers), or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds. The spacing is between pages: the requests for one page go out right after each other (your robots.txt, when ProbeliAIBot has no recent copy of it, then the page and the redirects it follows, to your site or another). A page it has read, or found blocked, isn't requested again for 7 days, and after that only if an AI answer cites it again. When a page fails for a reason that can pass (a server error, a rate limit or a timeout), ProbeliAIBot tries again later, never sooner than aRetry-Afterheader asks and 3 tries in all, then waits until an AI answer cites the page again. - Audits: only when a customer starts one, and each workspace can start only a few a day. An audit sends at most 4 requests at a time to your site (from each of our servers), plus its robots.txt once per address. When your robots.txt sets a
Crawl-delay, an audit sends one request at a time to that address instead, the next that long after the previous one ended (up to 10 seconds), and reads fewer pages and links: what fits in a few minutes at that pace. - Brief pages: each read loads one page at a time, the next at least 1 second after the previous one, or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds. The spacing is between pages (robots.txt, when ProbeliAIBot has no recent copy, and a page's redirects go out right after each other), and one workspace's brief doesn't read the site again within 10 minutes. Without anyone asking, a workspace's brief is written again at most once every 90 days, and only when nobody has edited it and its last homepage read worked. - Article research: one request at a time to each site (from each of our servers; a site is a domain with its subdomains), and the next page at least 1 second after the previous one ended, or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds. The spacing is between pages: the requests for one page go out right after each other (your robots.txt, when ProbeliAIBot has no recent copy of it, then the page and the redirects it follows on your site). A redirect to another site waits for that site's turn like a page, and isn't followed while that site is busy. A page waiting too long for its turn is left out of that article. - Own sitemaps: only while a customer's article is being written, once per article: one request at a time to your site (from each of our servers, shared with article research), the next at least 1 second after the previous one ended, robots.txt included when ProbeliAIBot has no recent copy, or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds. It reads at most 6 sitemap files and keeps at most 2,500 addresses from them, then one feed (/feed, else/rss.xml), for at most 30 seconds; when that time runs out, it carries on from where it stopped for at most 45 seconds more, never requesting again a file it already read (what it hasn't read by then is left out). Then it reads at most 5 sitemap files of your documentation in all (/docs/sitemap.xml, and yourdocs.subdomain's, after that subdomain's robots.txt: the sitemaps it declares on that subdomain, else/sitemap.xml,/sitemap-index.xmland/sitemap_index.xmlin turn until one lists a page) and keeps at most 500 addresses from them, for at most 10 seconds. For an article about case studies it may then read your case-study folder's own page once (one page, tried once more at most when it doesn't load, within the same rules,Crawl-delayincluded; when it couldn't load it, a later article about case studies may try once more). Within a week of reading all your lists for one article, it reads none of them for the next articles of the same workspace. A file that doesn't load is tried once more at most, a few seconds later. - Published pages: one page load per check, plus at most 5 redirects within the website the article was written for (each one request), and the robots.txt of each address (scheme and host) of that website the check reaches (such as
http://example.com, thenhttps://www.example.comafter a redirect), each with at most 5 redirects within the website, only when ProbeliAIBot has no recent copy; only when a customer marks an article as published or asks to check it again, and each workspace can run at most 30 checks an hour and 100 a day. A check doesn't followCrawl-delayand isn't retried. A check of a page on your own website also reads your robots.txt for itsSitemap:lines and at most 2 of those sitemap files, only to tell the customer whether your sitemap lists the new page yet; it keeps no page text, reuses one website's answer for a day (from each of our servers, as above), and never reads a sitemap of a blog Probeli AI hosts itself. - IndexNow key files: one request to
/<key>.txton the address you give us, plus that address's robots.txt when ProbeliAIBot has no recent copy, only when a customer asks us to check the file; each workspace can ask at most 10 times an hour and 30 times a day. On a blog Probeli AI hosts itself, where the address points at our own servers and the file is ours to serve, nobody has to ask: we look for it without being asked, every 7 days once it is there, and at most once an hour while it isn't. On your own website it follows at most 5 redirects, over https and only within that website (such as fromwww.to the bare address), and reads the robots.txt of each address it reaches; a redirect to another website is never followed, and on a blog Probeli AI hosts it follows none. We keep whether the file held the key and, when it didn't, the last address read, its status code and the first line of what it returned. - Sitemap choices: your robots.txt only, and only when ProbeliAIBot has no recent copy of it, when an owner or admin opens the sitemap setting in Settings · Website — to list the sitemaps you already declare there. No sitemap file is read, and nothing else is requested from your site.
- Fact checks: the same pace as article research (one request at a time to each site, at least 1 second apart,
Crawl-delayfollowed up to 10 seconds), at most 8 pages per published article, about once a month per article, and a page that doesn't load is tried again the next month, not sooner. - Existing pages: only when a customer starts a scan, and each workspace can start at most 3 scans an hour and 10 a day. A scan reads one page at a time, the next at least 1 second after the previous one, or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds, and doesn't read a page it read in the last 7 days again. A page read again on request, or for refresh suggestions, is one page load. - Free tools: one page load per use (a sitemap check of a website reads its robots.txt, then one sitemap file), plus robots.txt when ProbeliAIBot has no recent copy. Each visitor can use each tool 20 times a day, and the same address's answer is reused for 10 minutes. A use isn't retried and doesn't follow
Crawl-delay. - Free site audit: at most two reads per audit, the homepage and then one sitemap file, plus robots.txt when ProbeliAIBot has no recent copy. When your robots.txt sets a
Crawl-delay, the second read waits that long after the first one ended (up to 10 seconds). Each visitor can run 5 audits a day, an audit of the same website in the same language is reused for 24 hours, and an audit isn't retried. - Article images: only while a customer's article is being written, once per article: one request at a time to each address, the next at least 1 second after the previous one, or longer when your robots.txt sets a
Crawl-delay, which it follows up to 10 seconds; at most 30 requests (pages and images) within 90 seconds per article, plus the robots.txt of each address when ProbeliAIBot has no recent copy. It isn't retried. - Title reads: one page load, plus the redirects it follows within the website and the website's robots.txt when ProbeliAIBot has no recent copy, only when a customer asks for a suggested title; each workspace can read at most 20 pages an hour and 60 a day this way. It doesn't follow
Crawl-delayand isn't retried. - Homepage reads: one page load, following redirects (for a competitor's, plus its robots.txt when ProbeliAIBot has no recent copy), each time a customer enters your website or asks to read it again (a workspace can ask again 1 minute after its last read at the earliest), and once when a workspace's competitor search finds your website.
How it follows robots.txt
ProbeliAIBot reads the robots.txt group for probeliaibot, or the * group when there is none, with Allow, Disallow and the * and $ wildcards (RFC 9309).
- Cited pages: a page your robots.txt disallows is never requested, and neither is a redirect to one. A missing robots.txt allows everything; one that can't be read (a server error, a rate limit or a timeout) stops ProbeliAIBot from reading any of your pages until it can. ProbeliAIBot keeps a site's robots.txt for up to 1 day, so a change can take that long to apply.
- Audits: robots.txt is read again for every audit, for each address (scheme and host) the audit reaches, such as
example.comandwww.example.com. Every request an audit sends (pages, links, sitemaps, llms.txt and llms-full.txt, API description files, on your site or another) and every redirect it follows is one that address's robots.txt allows. If it disallows your homepage, or can't be read, the audit is limited: it reads nothing else, and the checks of the files it couldn't read show as not checked. - Brief pages: robots.txt decides the homepage, each key page, each sitemap and each redirect it follows. A key page it disallows is skipped; if it disallows your homepage, or can't be read, nothing else is read and no brief is written. As for cited pages, ProbeliAIBot keeps a site's robots.txt for up to 1 day, so a change can take that long to apply.
- Article research: robots.txt decides each page and each redirect it follows. A page it disallows, or whose robots.txt can't be read or redirects to another website, isn't read and isn't a source of the article. As for cited pages, ProbeliAIBot keeps a site's robots.txt for up to 1 day.
- Own sitemaps: robots.txt decides each sitemap, the feed and each redirect it follows; a file it disallows isn't requested, and a robots.txt that can't be read stops it from requesting anything. ProbeliAIBot uses the same copy of your robots.txt as for cited pages, kept for up to 1 day.
- Fact checks: as for article research, robots.txt decides each page and each redirect; a page it disallows isn't read, and its facts are no longer checked against it.
- Existing pages: robots.txt decides each sitemap, each page and each redirect it follows (only within your website); a page it disallows isn't read, and a robots.txt that can't be read stops the scan from reading any sitemap or page. ProbeliAIBot uses the same copy of your robots.txt as for cited pages, kept for up to 1 day.
- Competitor homepages: robots.txt decides the homepage and each redirect it follows; when it disallows them, or can't be read, nothing is loaded and the competitor is named after its domain (a competitor the setup's search found is then only suggested, not tracked). ProbeliAIBot uses the same copy of your robots.txt as for cited pages, kept for up to 1 day.
- Article images: robots.txt of each address (scheme and host) decides each page, each image and each redirect it follows; a page or image it disallows, or whose robots.txt can't be read, isn't requested, and the article goes without that image. ProbeliAIBot uses the same copy of your robots.txt as for cited pages, kept for up to 1 day.
- Title reads: robots.txt decides the page and each redirect it follows; a page it disallows, or whose robots.txt can't be read, isn't read, and the suggestion is written without its current title.
- Published pages: as for cited pages, a page your robots.txt disallows is never requested, and neither is a redirect to one; a robots.txt that can't be read stops the check. A redirect to another website isn't requested at all, not even that website's robots.txt. ProbeliAIBot uses the same copy of your robots.txt as for cited pages, kept for up to 1 day.
Two requests aren't subject to robots.txt:
- robots.txt itself;
- the homepage read of a workspace's own website, when a customer enters it (creating a workspace, changing its website, an agency adding a client) or asks to read it again: one page load, at that customer's request, like a person opening the page.
How to block it
To block ProbeliAIBot on your whole website, add this to your robots.txt:
User-agent: probeliaibot
Disallow: /
To keep it out of part of your website, disallow only that path, for example Disallow: /private/. To slow it down, add a Crawl-delay line (in seconds) to its group.
When your robots.txt blocks it, Probeli AI can't find mentions on your pages, the audit of your website is limited, the business brief of your website can't be written from its pages, your pages aren't read as sources for articles or for their monthly fact checks, your sitemaps can't show whether one of your posts already covers an article's keyword, an article published on your website can't be confirmed as live, and your images aren't copied into articles. To refuse those homepage reads too, block requests whose user agent contains ProbeliAIBot on your server or firewall. We don't publish a list of IP addresses.
Questions and requests
Write to hi@probeli.ai with questions about ProbeliAIBot, a problem with how it reads your website, or a request to delete a page from our cache. Our Privacy Policy explains what we store from the pages it reads and for how long.
Want articles that bring in customers?
Probeli AI finds what your buyers search, writes an article for each search and publishes it to your blog.
7-day money-back guarantee · Cancel anytime