OpenAI, Google, Microsoft and Anthropic back one robots.txt signal to opt your website out of AI training
Announced · Published · By Contractor AI Shop
OpenAI, Google, Microsoft and Anthropic say they'll honor one robots.txt policy token, from an IETF AIPREF draft, to opt a site out of AI training. Anthropic supports it now; Microsoft Feb 1, 2027; Google early 2027; OpenAI in coming months. Google says search visibility isn't affected.
Key facts
- Google says site owners will be able to use one shared 'policy token' in robots.txt to set AI-training preferences across participating providers, instead of separate rules for each AI company. Google Search Central (John Mueller)
- Google says it will treat an opt-out via the shared token as if the site had disallowed Google-Extended, its existing AI-training control; company-specific controls keep working alongside it. Google Search Central (John Mueller)
- Google says it aims to support the control in 2027. Per Engadget, Anthropic already supports it, Microsoft starts Feb 1, 2027, Google in early 2027, and OpenAI 'in the coming months.' Engadget (Mariella Moon)
- The proposal was written by Anthropic's Kevin Kelley at the IETF. Microsoft says sites will still appear in Bing results regardless of their preference, per The Verge. The Verge (Emma Roth)
- Engadget prints the group as 'User-Agent: __usage_ai-training' followed by Disallow/Allow lines; that matches the IETF draft (draft-kelley-aipref-training-preference-signal-00), which we checked Nov 2. Engadget (Mariella Moon)
- Engadget notes the signal can't undo past scraping: content already crawled stays with the companies. Engadget (Mariella Moon)
Our take: what it means for contractors
If your company site has project photos, blog posts or pricing pages you'd rather not feed to AI models, this will eventually be one block in robots.txt instead of a rule per crawler. It doesn't hide you from Google or Bing search. It's not live everywhere yet, so there's no rush; whoever manages your site can add it when providers switch on.
What we don't know yet
- It's an individual IETF draft, not a ratified standard, and may change
- Meta signaled support but didn't commit to honoring it (The Verge)
- OpenAI hasn't given a date
- It can't undo scraping that already happened
Sources
- Google Search Central (John Mueller) · Supporting a shared AI training control for websites · 2026-10-30 · Primary
- The Verge (Emma Roth) · The biggest names in AI are backing a new way to opt out of scraping · 2026-10-30 · Secondary
- Engadget (Mariella Moon) · Google joins OpenAI and Anthropic in promising new scraping controls · 2026-10-31 · Secondary