Technology & AI

Four AI companies agree on one robots.txt opt-out for training

Publishers could use one robots.txt setting to keep their material out of chatbot model development.

By Standard Topic. AI-generated from the reporting listed under Coverage. Published . Updated .

This story is no longer being updated. It reflects reporting up to .

What happened

Google, Microsoft, OpenAI and Anthropic have agreed to honour one shared robots.txt entry that tells AI companies not to train models on a website's content. Google and Microsoft will start honouring it in 2027, and Google's version will also cover Gemini grounding. Sites using NoArchive or NoCache will lose their Microsoft AI opt-out on March 1, 2027 unless they switch.

Why it matters

Publishers say they should not have to choose between their content and being found by AI services, and the single setting is meant to let them opt out of training across several companies at once.

Timeline

  1. Oct 30, 10:00 PM ET: Google and other AI firms said they would support a new AI crawling opt-out standard.
  2. Oct 31, 4:01 AM ET: Sites using NoArchive or NoCache will lose their Microsoft AI opt-out on March 1, 2027 unless they switch.
  3. Oct 31, 8:24 AM ET: Google and Microsoft are set to start honouring the shared opt-out in 2027.

Coverage

More on this story

More from Technology & AI

All Technology & AI news · Technology & AI story archive