On September 15, Blocking AI Bots Starts Blocking Google Too. Your Dealership Site May Already Be Set to Do It.

On July 1, Cloudflare replaced its single "Block AI bots" switch with three separate categories. Site owners can now manage crawlers by what the crawler is actually doing. Cloudflare defines Search as "any behavior that collects or indexes your content, so it can answer questions about it later," Agent as "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now," and Training as "a crawler taking your content to train or fine-tune a model."

More control is good. Then comes the sentence that should make you open a browser tab.

"Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

That takes effect September 15, 2026. Read the parenthetical twice, because it is doing all the work. It includes the legacy setting. If anyone at your store, or at your website vendor, or at the agency that touched your site two years ago, ever ticked "Block AI bots," Cloudflare's own post says Googlebot gets blocked.

The checkbox was built for a web that no longer exists

To understand why this is about to catch people, you have to remember what that checkbox was for.

Through 2024 and 2025, website owners were told over and over that AI companies were taking their content for free. The industry's answer was a switch. One click, block the AI bots, done. Cloudflare describes what that preset actually covered: "The managed preset to 'Block AI bots' that we've announced in the past included single-purpose bots that crawled data for model training."

Single-purpose. That was the whole design assumption. A bot showed up, it was either a search crawler or a training scraper, and you sorted it into a bucket.

That assumption broke. Googlebot now feeds a system that both ranks your inventory pages and generates answers about your store. Cloudflare CEO Matthew Prince has criticized Google for bundling its search crawler with its AI crawler, which is the friction sitting underneath this change. When one crawler does two jobs and you have written a rule against one of those jobs, somebody has to decide what happens.

Cloudflare decided. The most restrictive rule wins.

That is the correct default for a firewall. Security engineering has long worked that way, and for good reason: when a rule is ambiguous, you fail closed. The trouble is that failing closed on Googlebot means failing closed on Google Search. Google's documentation is blunt about what Googlebot's crawling touches: "Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover."

A firewall rule written in 2024 to protect your content from a chatbot is about to be applied to the crawler that decides whether your dealership appears when somebody searches your own name.

What this means for your dealership

Start with the uncomfortable part: you probably do not know what this setting is on your site.

Dealership websites are rarely administered by the dealership. The domain is often registered by one party, the site is built by a platform vendor, the DNS was moved to Cloudflare by a developer who is no longer involved, and the security settings were configured once during launch and never reviewed. That is normal. It is also exactly the condition under which a switch gets flipped during an industry panic and then sits there for two years.

If your site is behind Cloudflare and Training is blocked, here is the sequence starting September 15. Googlebot requests a page. Cloudflare evaluates it as a multi-purpose crawler, finds the Training block, applies the most restrictive rule, and returns a 403. Google does not get the page. It does not get your sitemap either, which is how the whole site becomes invisible rather than just one URL.

Nothing about this looks like an outage. Your website stays up. Human shoppers browse normally. Your ads keep serving. The only thing that changes is that the crawler stops getting through, and organic listings decay quietly over the following weeks while everyone assumes it is a ranking fluctuation.

There is a version of this already being reported, and it is worth being precise about its status. On August 4, Search Engine Journal wrote up a post from r/TechSEO in which a site owner said that "When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap." Google's John Mueller replied asking to look at it directly. Search Engine Journal is careful to say "it's unclear whether this is a fluke or user error," and that caveat is fair. Treat that report as a single unconfirmed data point rather than proof. Act on this because Cloudflare published the behavior itself, with a date attached. The Reddit thread is just the first person to notice.

One more honest caveat, because the coverage is muddled here. Cloudflare's new default for Training and Agent crawlers is scoped to "the pages that display ads," and it applies to "all new domains onboarding to Cloudflare." Some trade coverage has carried that ad-page qualifier over to the multi-purpose crawler change as well. Cloudflare's own sentence about Googlebot does not include it, and nobody has published a clarification from Cloudflare either way. Most dealership sites do not display third-party advertising, so if the ad-page limit does apply, the exposure is smaller than it looks. That ambiguity is a reason to go check your own settings rather than a reason to relax, because the check takes ten minutes and resolves the question for your specific site.

What to do about it

Find out whether you are on Cloudflare at all. Run curl -sI https://www.yourdealership.com and look for a cf-ray header in the response, or just look up your domain's nameservers. If you would rather not touch a terminal, email your website vendor one sentence: "Are we behind Cloudflare, and if so what is our AI bot blocking setting currently set to?" Ask for a screenshot.

Find out who owns the Cloudflare account. This matters more than the setting itself. Cloudflare says it will "continue to notify customers of the upcoming change to defaults as we approach September 15." Those notifications go to whoever is on the account. If that is a former vendor's shared inbox, the warning arrives somewhere nobody reads.

Check the Training setting before September 15. Cloudflare's opt-out path is in the dashboard: "if a website owner wants to opt out of these new default configurations, they can easily mark this in their Security settings any time leading up to September 15, which will confirm that they want no changes on Training crawlers that also crawl for Search purposes." The new controls are available on every plan, "including customers on our Free tier," so a free-plan dealer site is fully in scope.

If you genuinely want out of AI training, use the tool that was built for it. Google publishes a separate token for exactly this. Google-Extended is documented as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models," and Google states plainly that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." A robots.txt directive costs you nothing in search visibility. An edge-level block on a multi-purpose crawler costs you the crawler. Those are two very different decisions that have been sold as the same decision.

Do not confuse this with the AI Overviews control. Opting out of Gemini training is a separate lever from removing your dealership from AI Overviews and AI Mode, which is a Search Console toggle we covered on July 29. Three different switches, three different consequences. Know which one you are touching.

Set a calendar reminder for late September. Open Search Console, check Crawl Stats and the sitemap report, and look for 403 responses. A crawl error report that was clean in August and shows 403s in October is the signature of this exact problem.

The larger point

The instinct behind the AI bot block was reasonable. We argued a while back that managing a hand-curated list of approved bots was a losing strategy for dealers, and this is the same lesson arriving from the opposite direction. Blanket blocking fails for the same reason blanket allowing fails. The bots stopped being sortable into neat categories, and any rule built on the assumption that they are will eventually catch something you meant to keep.

Most of the dealers who get hurt here will have made a perfectly reasonable decision in 2024 under a different set of facts, then never revisited it, on a system they do not log into, administered by a vendor they no longer work with. The fix is a ten-minute audit. The cost of skipping it is your organic search presence going dark in the middle of Q4 while everyone hunts for a ranking algorithm that never changed.

Go look at the setting. Then write down who owns the account.

If you want a second set of eyes on your site's crawler configuration before the deadline, book a call and we will walk your setup with you.