Your site may be blocking ChatGPT without you knowing

Our agency sells a service that helps companies get cited by artificial intelligence. On November 11, 2026, we opened the rules file of our own website, and it told the ChatGPT, Claude and Gemini bots to stay out. We hadn't written that order: it was in a block that Cloudflare places at the top of the file. Below, you'll find what we discovered, how to check your own site in two minutes, and how to decide whether to allow these bots.

What we found on our site

Every website can have a file called robots.txt. It's a note on the door, addressed to the robots that read the internet: "you may enter" or "you may not." Ours had two notes, one contradicting the other.

The first began with the line "BEGIN Cloudflare Managed content" and set Disallow: / (do not enter any page) for a list of bots: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Amazonbot, Bytespider, meta-externalagent and more than twenty others. Right below came the block we had written, setting Allow: / (you may enter) for GPTBot, ClaudeBot, Google-Extended and CCBot. The same names, with opposite orders, in the same file.

The Cloudflare dashboard showed the practical result over the previous seven days:

• ClaudeBot (Anthropic): 0 requests allowed, 273 unsuccessful.
• GPTBot (OpenAI): 0 allowed, 201 unsuccessful.
• CCBot (Common Crawl): 0 allowed, 207 unsuccessful.
• Bytespider and Amazonbot: 0 allowed, 199 and 193 unsuccessful.

In the same period, the AI search bots got through normally: 202 allowed requests from PerplexityBot, 174 from OAI-SearchBot and 109 from Claude-SearchBot. That difference is the heart of the story, and the next section explains it.

One honest caveat: "unsuccessful" in the dashboard includes any request that didn't end well, such as a page that doesn't exist. The number alone doesn't prove blocking. What stands out is the zero in the allowed column, for every bot on the disallowed list and only for them.

Three types of AI bot, three different consequences

"AI bot" is not one thing. Each company uses at least three, with separate functions, and blocking one doesn't block the others. OpenAI's and Anthropic's documentation describe them like this:

• Training bot (GPTBot, ClaudeBot, CCBot). Reads pages so the content can be used to train the next models. Blocking it means asking for your material to be kept out of that training.
• Search bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Builds the index the AI consults when it searches the internet. OpenAI is direct about it: a site that blocks OAI-SearchBot won't appear in ChatGPT's search answers, although it may still show up as a navigation link.
• User-requested bot (ChatGPT-User, Claude-User, Perplexity-User). Opens a specific page when someone asks the AI to read that address. OpenAI notes that, because this is an action initiated by a person, robots.txt rules may not apply to it.

Google is a case of its own. Google-Extended is not a bot but a name used only in robots.txt to say whether your content can train Gemini and ground its answers. Google's documentation states that this name does not affect a site's inclusion in Search and is not used as a ranking signal. Blocking Google-Extended doesn't drop anyone from Google, and allowing it doesn't lift anyone up.

In our case, the disallowed list was the training bots. The search bots were never on it, which is why they kept getting in. If your worry is "I vanished from ChatGPT," the first thing to look at is whether OAI-SearchBot is allowed. We explain the full path in our post on how to appear in ChatGPT and other AIs.

How the order got there

Cloudflare is a service that sits in front of a website: it speeds it up, protects it from attacks and, since 2024, offers controls over AI bots. The timeline, based on the company's own announcements:

• September 2024: a one-click option to block AI bots appears. More than a million customers chose it.
• July 1, 2025: the company announces that it will block, by default, AI bots that access content without permission or payment, and that every new domain will be asked at sign-up whether it wants to allow these bots. On the same day it launches managed robots.txt, available on all plans, including the free one.
• September 24, 2025: Cloudflare reports that managed robots.txt is active on more than 3.8 million domains and begins adding to them the line Content-Signal: search=yes, ai-train=no, which means "may be used for search, may not be used for training."

The documentation says the feature doesn't come switched on: someone has to enable the "Enable Bot Preference Sync" toggle. No one at our agency remembers enabling it. The most likely explanation is one of those setup screens where the suggested answer is protection and we accept it without reading to the end. That's why the title says "without you knowing": the choice exists, but it's made in a one-second click by someone thinking about something else.

It's worth saying that Cloudflare's intention is defensible. The company argues that AIs read a lot and send back few visits, and it wants content producers to be able to charge for that. For a newspaper, that makes complete sense. For a small business that wants to be found, the math is usually different.

Check your site in 2 minutes

1. In your browser, type your site's address followed by /robots.txt. Example: www.suaempresa.com.br/robots.txt.
2. Search for "Cloudflare Managed." If it appears, there is a block you didn't write.
3. Search for "GPTBot," "ClaudeBot" and "Google-Extended." Look at the line below each: Disallow: / blocks the whole site, Allow: / permits it.
4. Search for "OAI-SearchBot" and "PerplexityBot." If either has Disallow: /, the problem is more serious, because those are search bots.
5. If the same name appears twice with opposite orders, you have the same flaw we had.

About the double order: the technical standard for robots.txt (RFC 9309) says to merge repeated rules for the same bot and, in a tie, use the most permissive one. But following robots.txt is voluntary, and not every bot interprets a tie the same way. A contradictory file is a gamble; a clean file is certainty.

If your site doesn't use Cloudflare, the test is the same. Any service in front of the site, or anyone who has ever worked on it, may have left a block there.

Where to turn it off, if your site uses Cloudflare

In our dashboard, the block was in two places, and we had to change both:

1. AI Crawl Control → Overview → "Enable Bot Preference Sync." This toggle is what writes the block at the top of robots.txt. With it off, the file goes back to being only yours.
2. Security → Settings → Bot traffic → "AI Crawler and Agent access policies." There are three lines: Search, Training and Agents (user-requested bots). Each accepts "Allow," "Disallow" (asks via robots.txt), "Block on pages with ads" (blocks only where there are ads) or "Block" (actually stops them). Ours had Search on Allow, Training on Disallow and Agents on Block on pages with ads.

We switched all three to "Allow" and checked the live file: the Cloudflare block was gone. Minutes later, ClaudeBot logged its first allowed request of the week. GPTBot and CCBot had not come back by the time we published; OpenAI says its systems can take about 24 hours to adjust to a change in robots.txt.

A warning: these menu names change often. If you can't find them, search for "AI" in the dashboard's search box.

Allow or keep blocked? The criterion

There is no single answer. The question that decides it is: is your content the product, or is it the storefront?

Keep training bots blocked if the content is what you sell: a news portal, a course, an image bank, teaching material, literary text. In those cases, entering a model's training means giving away for free what pays the bills. There is an ongoing legal debate about this, which we summarized in our post on training AI on protected works.

Allow them if the content exists to bring in customers: the bakery that explains how to order a cake, the clinic that describes its treatments, the accounting firm that publishes answers to MEI (Brazilian micro-entrepreneur) questions. Nobody pays to read those pages. They exist so the company can be found, and an AI that knows your business is one more place where that can happen.

In either case, don't block the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot), unless the decision really is to stay out of AI answers.

The gain from allowing training is small and slow. Models are trained from time to time, and no company promises that your site will be included or that you'll be cited because of it. Among the world's 10,000 largest sites, only 7.8% blocked GPTBot in mid-2025, according to Cloudflare: the vast majority were already allowing it, and that didn't make them all famous in AI. Allowing removes an obstacle. What makes AI speak well of a company is still what we described in what AI says about your company: a clear site, correct business listings, and answered reviews.

What doesn't help

• Thinking robots.txt is a lock. It's a request. Cloudflare itself writes that content signals "are not technical countermeasures" and that some companies may ignore them. Anyone who needs to really keep bots out uses firewall blocking.
• Writing "Allow" under a "Disallow" and considering the matter settled. That was our mistake. The file that counts is the one that's live, and it may have sections you didn't write.
• Testing by pretending to be the bot. Changing the browser's identification to "GPTBot" proves nothing: Cloudflare checks the origin address of the real bot. The test that counts is the dashboard's allowed-requests column, a few days later.
• Expecting results the same week. The effect of training shows up in the next generation of models, when it shows up at all.

What this means for your business

The episode taught us a lesson worth more than the configuration: a company's website has layers that no one at the company wrote. The hosting service, the protection service and the tools installed over the years make decisions on the owner's behalf, and some of them contradict what the company wants. We sell AI visibility, and we spent months with a "no entry" sign on the door for AIs.

Checking takes two minutes and costs nothing. If you found something strange in your robots.txt and don't know whether to touch it, our SEO and AI-answer optimization team can run this audit: we read the file, the dashboard of whatever protects your site and what each AI says today about your company, and we deliver a list of what to change.

Sources

• Cloudflare robots.txt file and AI Crawl Control dashboard for the Agência Primeira Página website, read on November 11, 2026.
• Cloudflare, July 1, 2025 announcement on blocking AI bots by default.
• Cloudflare, blog, "Control content use for AI training" (July 1, 2025), with data on the 10,000 largest domains.
• Cloudflare, blog, "Content Signals Policy" (September 24, 2025).
• Cloudflare, documentation on managed robots.txt and AI Crawl Control.
• OpenAI, bot documentation (GPTBot, OAI-SearchBot and ChatGPT-User).
• Anthropic, help center, description of ClaudeBot, Claude-User and Claude-SearchBot.
• Google, crawler documentation, Google-Extended entry.
• IETF, RFC 9309, Robots Exclusion Protocol.