Article

Block, allow, or charge: the new controls for AI crawlers

Author

Oleksandr Kotliarov

Date

August 1, 2026

Reading Time

9 min

For about thirty years the web ran on a handshake. You let search crawlers read your pages, and in exchange they sent you readers. Nobody signed anything. The deal held because both sides came out ahead: Google indexed you, and a slice of its traffic landed on your site. robots.txt was the whole contract — a voluntary text file that well-behaved bots agreed to respect.

AI crawlers kept the first half of the deal and quietly dropped the second. They read everything and send almost nothing back, because an answer generated from your content is an answer the reader never has to visit you to get. That is the shift worth understanding before you touch a single setting: the economics that made open crawling a fair trade stopped being true, and the tooling is now catching up to that fact. Cloudflare’s move — flipping the default and adding a way to charge — is the most concrete version of the response, and it is a useful lens even if you never use their product.

The deal broke on the numbers

The clearest way to see what changed is the crawl-to-referral ratio: how many pages a crawler reads for every visitor it sends you. For a traditional search engine that number was survivable. For AI crawlers it is not close.

Cloudflare, which sits in front of roughly a fifth of web traffic, published the figures when it made its case. By its accounting, Anthropic’s ClaudeBot made about 71,000 requests for every single referral it sent back, and getting traffic out of OpenAI’s crawlers was on the order of 750 times harder than the old Google model. The company’s own summary of the break is blunt: the deal between crawlers and site owners that had held up for thirty years — crawl you, and you get referrals — was no longer true.

You do not need to trust one vendor’s numbers to feel the direction. The reason a page gets crawled has decoupled from the reason a person shows up. A model that ingests your documentation to answer questions about your product is not a funnel to your product; it is a substitute for reading your docs. The crawl still costs you bandwidth and compute. The visit that used to pay for it does not arrive.

What Cloudflare actually shipped

The response arrived in stages, and it helps to separate what is generally available from what is still a beta promise.

The visible pivot came on 1 July 2025, when Cloudflare began blocking known AI training and agent crawlers by default on new domains — the first time a major network treated AI crawling as opt-in rather than opt-out. Around it sits a set of controls that had been landing over the previous year:

AI Audit / Crawl Control — analytics that show which AI bots hit your site and how often, with a one-click block. This is the part you should reach for first, because it turns the decision from a guess into a measurement.
Managed robots.txt and a content-signal header — a maintained policy file plus a header that states intent (index yes, train no), so your preference is machine-readable instead of buried in a text file you last edited in 2019.
AI Labyrinth — instead of blocking a misbehaving crawler, feed it a maze of plausible, useless generated pages to waste its budget. A tarpit, not a wall.
Pay-per-crawl — the new lever. A crawler that requests a page it has not paid for gets an HTTP 402 Payment Required response instead of the content, and can be granted access once it settles. This is the third option, and as of writing it is still a private beta, not a shipped market.

Notice the shape. Four of these are refinements of “block or allow” — better measurement, better signalling, a nastier way to say no. Pay-per-crawl is the only one that adds a genuinely new choice, and it is also the least proven. Keep that asymmetry in mind: the mature controls are the ones that let you say no cleanly, and the exciting one is still an experiment.

Three levers, and when to pull each

Strip it back and you have three positions a site can take toward any given AI bot.

Block is the right default for content that is your product or your revenue. If people pay for your research, your structured data, or your writing, and an AI answer built on it means they never arrive, letting the crawler in is subsidising your own disintermediation. Most publishers have landed here, and fast: by mid-2026 roughly 45% of prominent websites were blocking at least one AI crawler, and among top news sites the figure for training bots was near 79%. This is no longer a fringe stance.

Allow is the right call more often than the block-everything reflex admits, and the reason is discoverability. AI-mediated search — the assistant that answers a question and cites its sources — is becoming a real referral channel, and you only appear in it if its retrieval crawler can read you. Blocking indiscriminately can wall you out of the exact surface where buyers now start. The distinction that matters is which bot: a training crawler that ingests you to improve a model is a different animal from a retrieval bot that fetches you to answer a live question with a link back. Blocking the first while allowing the second is the position most product companies actually want, and it is only possible if you stop treating “AI bots” as one undifferentiated class.

Charge is the interesting new option and the one to be most sober about. If your content is genuinely licensable and a crawler wants it badly enough to pay, HTTP 402 lets you name a price instead of choosing between free and forbidden. That is a real improvement to the menu. But the marketplace is early, the named participants are few, and — the point that undercuts a lot of the excitement — a price is only a price if the other side agrees to pay it.

Block is a policy, not a guarantee

Here is the part the announcements underplay. Every one of these controls inherits the same weakness robots.txt always had: it works on bots that choose to be identified and choose to comply.

Cloudflare can block a crawler it recognises, and it can maintain a registry of verified bots. What it cannot do is force a crawler to tell the truth about who it is. When a bot rotates its IP addresses, spoofs a browser’s user-agent, and routes around your rules, a network-level block is trying to stop something it can no longer reliably see. This is not hypothetical: Cloudflare removed Perplexity from its verified-bot listing after documenting exactly this behaviour — stealth crawling that ignored site directives and disguised itself as ordinary browser traffic. The block worked on the bots that agreed to be blocked. The one that didn’t agree kept going.

Pay-per-crawl inherits the same gap, only sharper. A 402 is a request to pay, and a crawler that already ignores robots.txt has shown you exactly how much it weighs a request. The thing that would close the gap is cryptographic — a way for a bot to prove its identity with a signature you can verify, rather than a header you have to trust. That work is underway as an IETF effort (Web Bot Auth), with the relevant pieces still moving through standardisation. Until something like it is widely adopted and enforced, “we blocked the AI bots” and “the AI bots stopped reading us” are two different sentences, and the honest ones will keep them separate.

None of this makes the controls useless. Raising the cost and friction of unwanted crawling is worth doing even when it is not airtight, the same way a lock deters without being unpickable. It does mean you should size your expectations to a deterrent, not a fence.

What to do Monday morning

The mistake is to treat this as one switch. It is a per-bot-class policy decision, and it starts with looking before you block.

Measure first. Turn on crawl analytics and find out which AI bots actually hit you and how hard, before you decide anything. Most teams have never looked, and the answer is frequently surprising — either a crawler you assumed was negligible is your top consumer, or the one you were worried about barely shows up.
Split the bots into classes. Training crawlers, retrieval/search bots, and autonomous agents want different things from you and deserve different answers. Decide per class, not per “AI”.
Block what substitutes for you; allow what refers to you. If a bot’s job is to make your content reachable without you, that is a block candidate. If its job is to send a cited reader your way, think hard before you wall it out — you may be blocking your own distribution.
Treat charging as a pilot, not a plan. If your content is licensable, experiment with pay-per-crawl. Do not build a revenue forecast on a private beta whose enforcement depends on the goodwill of the parties you are trying to charge.
Keep “blocked” and “stopped” as separate words. Verify with your logs that the policy is having the effect you think. A rule that a determined crawler ignores is a false sense of security, which is worse than none.

The larger picture is that the open web’s oldest assumption — that letting machines read you is the same as being discovered by people — has come apart, and the tooling to price the difference is arriving faster than the norms to govern it. Cloudflare gave the market a third button. Whether the button does anything depends on who agrees to press it.