One opt-out for AI training: what the shared robots.txt token changes, and what it does not

What was agreed

OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 that they will honour one shared robots.txt token for opting out of AI training, so a single rule can cover all four companies' crawlers; the token is described in an IETF Internet-Draft¹⁴. It builds on the IETF's wider work on a shared vocabulary for AI usage preferences⁵.

The details matter more than the headline. Reporting says Anthropic has begun honouring the token, Microsoft is targeting early 2027 and separates training from grounding with a different tag, Google plans to treat the opt-out like its existing Google-Extended control, which also covers grounding for Gemini answers, and OpenAI has said it aims to implement it in the coming months¹³. Sources differ on some timings. The signal is voluntary and does not remove content already collected or models already trained².

Training, grounding and search are three different decisions

The most important practical point is that "AI use" is not one thing. Training puts your content into a model. Grounding fetches your content at answer time and may cite and link you. Search indexing sends people to your site. Opting out of one may or may not affect the others, depending on the company.

For most businesses, the right answer differs by content type. Marketing pages and documentation usually benefit from being cited in AI answers. Paid research, proprietary datasets and member content usually should not train anyone's model for free. That is the same tiering logic as in what Reddit's logged-in move means for content.

A practical rollout

Keep the old rules for now. Until every company has implemented the shared token, keep existing company-specific crawler rules in place; reporting says those continue to work².

Do not rely on signals for valuable content. Voluntary signals are honoured by companies that choose to honour them. Content that is your competitive asset needs technical controls at your edge, as described in the front-end controls and gateway patterns elsewhere on this site.

Verify, do not assume. Log requests by user agent and path at your edge or in your traces, and check that crawlers stop fetching opted-out sections after the relevant rollout dates. Enriching traces with request headers makes this a simple filter rather than a log-parsing project⁶.

Who should act

• Publishers, marketplaces and data businesses: decide per content tier now, so the token goes in with a clear policy behind it.
• B2B SaaS and documentation-heavy companies: think twice before opting out of grounding; being cited in AI answers is increasingly how buyers find you. See structured data for AI search.
• Regulated organisations in the EU, UK and elsewhere: record your choices; they may matter for copyright and AI transparency discussions. See the Europe page.

CTO take

A single opt-out signal backed by four major AI companies is genuinely useful: it lowers the cost of saying no to training. But it arrives in stages, it does not cover the same things at each company, and it remains voluntary. Treat it as one part of a content access policy that separates training, grounding and search, enforces what matters at your edge, and verifies behaviour with your own data. Read alongside the Reddit analysis and where I work.

How I can help

I help content, marketplace and SaaS businesses design content access policies, edge controls and AI search strategies. I am based in India, the United States and the UAE, working globally. See fractional CTO work or analyse your challenge and bring the result to a call.