Cloudflare is drawing a new line in the sand against AI data scraping, preparing to automatically block crawlers from both search engines and AI companies by default. The move, announced yesterday, aims to give website owners decisive control over how their content is used to train artificial intelligence models.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," stated CEO and co-founder Matthew Prince. He positioned the new tools as a way to provide website owners with "increased visibility and commercial opportunities."

A New Default for Web Crawlers

Starting September 15, 2026, Cloudflare will activate a new setting for all new customers and websites. This setting is designed to "allow for search but block training and agent use for pages with ads."

The policy directly targets "mixed-use" crawlers—bots that index for search while simultaneously harvesting data for AI. Any such crawlers that fail to offer a way to separate these functions will be blocked on ad-supported pages by default. Free plan users will also be switched to the new setting unless they opt-out.

From 'Pay Per Crawl' to 'Pay Per Use'

The Implicit Challenge to Google

While the policy is broad, its sharpest implications are aimed squarely at Google. The search giant's primary crawler, Googlebot, indexes the web for search and collects data to train its Gemini models, which power features like AI Overviews.

Cloudflare notes that "the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI."

Google does offer a separate crawler, Google-Extended, for traditional search only. However, this forces a difficult choice: to appear in generative AI features, publishers must also allow their content to be used for model training. Cloudflare's new default policy is a clear attempt to force Google and others to unbundle these functions, giving publishers genuine control over their intellectual property.