A crawler is not one thing, and a publisher who treats it as one thing will make one bad decision instead of three deliberate ones.
On 15 September, Cloudflare split the choice. Website owners can now set search, AI training, and AI agents separately. A new control, Disallow AI Training, is meant to keep a site in search results while refusing to let that crawl become training data. Apple, Google, and Microsoft are in the group Cloudflare calls Accountable, meaning they meet, or have dated, the conditions for honouring that split. The old single switch for “AI bots” is on the way out.
For a new site that makes money from ads, the suggested defaults are specific. Search stays on. Training is refused. Agents are blocked on pages that carry an ad. Everyone else gets a different preset. Cloudflare’s own post is frank that agents are not the same tradeoff as mixed-use crawlers, and that the web does not yet have a settled way to tell an agent “no.” Blocking them on the pages where the ads are is a commercial hedge: do not let a machine skip the business model while reading the journalism.
This is the audience question in infrastructure form. Search is still how many humans arrive. Training is a one-time extraction of your archive into someone else’s model. An agent is a live reader, acting for a person who asked a question, and it may or may not send that person to you. Each one creates a different kind of demand and deserves a different price, from free, to licensed, to blocked.
Newsrooms do not have to become CDN engineers to use the distinction. They do have to decide, with their commercial colleagues, which stories are the public record, which are the subscription, and which may be quoted by a machine on the way to an answer. Agents will keep asking. The new part is that “yes to all machines” and “no to all machines” are both lazier than the traffic now requires.
The journalism people will still pay for is the journalism a summary cannot replace, delivered in a form a person can finish. Let the machines fetch a clean version of that, or do not let them fetch it. Do not make them guess.