Four AI companies agree on one robots.txt opt-out for training
Publishers could use one robots.txt setting to keep their material out of chatbot model development.
By Standard Topic. AI-generated from the reporting listed under Coverage. Published . Updated .
This story is no longer being updated. It reflects reporting up to .
What happened
Google, Microsoft, OpenAI and Anthropic have agreed to honour one shared robots.txt entry that tells AI companies not to train models on a website's content. Google and Microsoft will start honouring it in 2027, and Google's version will also cover Gemini grounding. Sites using NoArchive or NoCache will lose their Microsoft AI opt-out on March 1, 2027 unless they switch.
Why it matters
Publishers say they should not have to choose between their content and being found by AI services, and the single setting is meant to let them opt out of training across several companies at once.
Timeline
- Oct 30, 10:00 PM ET: Google and other AI firms said they would support a new AI crawling opt-out standard.
- Oct 31, 4:01 AM ET: Sites using NoArchive or NoCache will lose their Microsoft AI opt-out on March 1, 2027 unless they switch.
- Oct 31, 8:24 AM ET: Google and Microsoft are set to start honouring the shared opt-out in 2027.
Coverage
- How a new standard could make it easier for publishers to opt out of AI training (Indian Express)
- AI training opt-out: publishers ‘should not have to choose’ (Ana Maria Constantin)
- One robots.txt line will opt your site out of AI training at Google and OpenAI (martincid.com)
More from Technology & AI
- China puts humanoid robots to work serving food and sorting parcels
- Newsom proposes sweeping AI safety rules for Washington
- Google releases Gemini 4 Neon AI model with million-token context
- Microsoft and Intel widen their cloud chip partnership
- Google DeepMind rolls out Genie 4 to select subscribers
- AI helped AMD shorten chip design work, OpenAI says