“This policy token is not a directive for crawling behavior.” That line from the IETF draft written by Anthropic engineer Kevin Kelley explains most of what is new. The signal doesn’t stop bots visiting a site. It only states whether the content may be used to train AI models.
It works as a pseudo robot name, “__usage_ai-training”, in the User-agent line of robots.txt. A Disallow under it refuses training, an Allow grants it, and rules can be set path by path.
Google, Microsoft, OpenAI and Anthropic have committed to honour it. Meta spokesperson Andy Stone called it “an important step” without committing. Anthropic already applies it, OpenAI promises the coming months, Microsoft starts on 1 February 2027 and Google sometime in 2027.
The caveats are significant. With no line in place, training stays allowed. Content already collected is untouched, private deals can override the signal, and compliance is voluntary. It doesn’t govern AI-generated answers either: Microsoft has a separate AI Grounding setting, while Google will treat the token as a Google-Extended block, which TechRadar notes also covers Gemini grounding with no way to separate the two.
It is an individual draft with no formal IETF standing and lapses on 3 May 2027 unless revised. Google says it fits the EU code of practice for general-purpose AI, which treats robots.txt as a machine-readable opt-out.
The backdrop includes the New York Times lawsuit against OpenAI and Microsoft, Anthropic’s $1.5 billion settlement with authors and Cloudflare’s default blocking of AI crawlers since July 2025. For a small publisher, refusing training won’t remove a site from Google or Bing, and won’t guarantee its articles stay out of AI answers either.

