Google and Microsoft back OpenAI's robots.txt AI-training opt-out
The move is a proposed web standard letting sites tell AI companies not to use their content for training models.
By Standard Topic. AI-generated from the reporting listed under Coverage. Published . Updated .
This story is no longer being updated. It reflects reporting up to .
What happened
Google and Microsoft have signed up to OpenAI's plan for a new website crawling standard. Anthropic has also pledged to honor a shared robots.txt preference about using website content to train AI models. The signal does not block bots from crawling websites, and the proposal has not been endorsed by the IETF.
Why it matters
The standard could soon let website owners stop AI scraping with a single robots.txt line. It remains a pledge and is not yet an endorsed standard.
Coverage
- Google and Microsoft join OpenAI and Anthropic in backing a shared AI-training signal for robots.txt (Mezha)
- Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping (techradar.com)
More from Technology & AI
- China puts humanoid robots to work serving food and sorting parcels
- Newsom proposes sweeping AI safety rules for Washington
- Google releases Gemini 4 Neon AI model with million-token context
- Microsoft and Intel widen their cloud chip partnership
- Google DeepMind rolls out Genie 4 to select subscribers
- AI helped AMD shorten chip design work, OpenAI says