For two decades, the bargain was simple: let Google crawl your content, get traffic in return. Nobody questioned it because nobody could afford to. Now USA Today CEO Mike Reed is reportedly preparing to delist from Google from handling content without a value exchange within six to twelve months if no acceptable value exchange emerges. That’s not a tantrum — it’s a negotiating position backed by real infrastructure. AI search features blurred the line between indexing and training, and publishers finally have tools to draw it back.
Cloudflare announced that on September 15, 2026, new and free-tier customers on ad-supported pages will have “multi-purpose crawlers” blocked by default. The company now classifies crawlers into three buckets:
- Search (allowed)
- Training (blocked)
- Agentic (blocked)
When a crawler like Googlebot serves multiple purposes, the most restrictive rule applies. Beehiiv has already partnered with Cloudflare to give newsletter creators the same blocking capability.
That’s the technical scaffolding. The regulatory scaffolding arrived from the U.K.
The U.K. Competition and Markets Authority imposed conduct requirements forcing Google to let publishers block content from AI features — AI Overviews and generative results — without penalizing traditional search rankings. Google is testing a Search Console control for exactly this. Publishers who opt out reportedly won’t appear in AI-driven results, but organic rankings are said to stay intact.
Here’s what publishers now have access to:
- Cloudflare’s crawler classification system, defaulting multi-purpose crawlers to the most restrictive setting on ad-supported pages
- Google-Extended, an opt-out specifically for AI training
- A Search Console control (currently testing in the U.K.) to block AI Overviews and generative features
- Pay-per-crawl, a Cloudflare private beta using HTTP 402 Payment Required responses to charge AI crawlers
- U.K. CMA requirements mandating proper attribution with clear links in AI features
“We want a technical solution that allows you to be discoverable without having to give your content away for free.” — Stephanie Cohen, Chief Strategy Officer, Cloudflare
The Math Is Changing
The leverage publishers never had before is suddenly sitting right in their hands.
Publishers who have watched referral traffic erode for three years will feel this shift acutely. SEO consultant Lily Ray noted that abandoning Google Search was historically commercially unthinkable — declining traffic has made some publishers willing to reconsider. The strategic question is whether withholding content now generates more value as leverage than the visibility Google provides.
Google frames its controls as generous. Neither Google-Extended nor the Search Console opt-out affects traditional rankings, the company says. But Google hasn’t struck publisher licensing deals at the scale Meta, Microsoft, or Amazon reportedly have with some media companies — a gap made more striking given the scale of AI infrastructure investment reshaping the industry. That gap is the quiet part publishers keep saying out loud.
This isn’t a protest. It’s a renegotiation — and for the first time, publishers came to the table with leverage.
