LIVE
BREAKING
COP31 Final Deal Close as Presidency Tables New PackageOcean Catalyst Group Launches at COP31 to Protect Coastal TourismBayelsa Boat Disaster Death Toll Rises to 24 as Search ContinuesASEAN Leaders Back One Billion Trees Plan to Restore Southeast Asia’s ForestsWorld’s Best Destinations for 2026: Santorini to Samarkand‘Hunger Games’ Prequel Scores $56 Million Opening DayHoplite Raises $5.5 Million From Stripe’s Collison BrothersGold Price Crashes Below $4,000 in Worst Week Since MarchRussian Helicopter Suspected of Violating Finnish AirspaceWolbachia Mosquitoes Cut Dengue Infections by 69% in Brazil StudyEU Bans Plastic Waste Exports to Non-OECD Countries From TodayCOP31 Risks Ending With No Deal as Saturday Overtime Drags OnIsraeli Strike Kills Three in Aid Vehicle in Gaza CityBasra Power Station Fire Kills at Least Four and Injures DozensClimate News Is Losing Its Audience — What the Research Says Can Win Them BackSouth Africa’s First Woman Deputy President Phumzile Mlambo-Ngcuka Dies Aged 71US Denies Iran Claim of Missile Hit on F-16 Over GulfEuropean Week for Waste Reduction Opens Today: Living Well With Less Under ‘Material Sufficiency’France and US Draft Plan for Life After UNIFIL in LebanonSecond Tanker Hit Near Strait of Hormuz on Saturday as Iran Claims US Drone DownedCOP31 Final Deal Close as Presidency Tables New PackageOcean Catalyst Group Launches at COP31 to Protect Coastal TourismBayelsa Boat Disaster Death Toll Rises to 24 as Search ContinuesASEAN Leaders Back One Billion Trees Plan to Restore Southeast Asia’s ForestsWorld’s Best Destinations for 2026: Santorini to Samarkand‘Hunger Games’ Prequel Scores $56 Million Opening DayHoplite Raises $5.5 Million From Stripe’s Collison BrothersGold Price Crashes Below $4,000 in Worst Week Since MarchRussian Helicopter Suspected of Violating Finnish AirspaceWolbachia Mosquitoes Cut Dengue Infections by 69% in Brazil StudyEU Bans Plastic Waste Exports to Non-OECD Countries From TodayCOP31 Risks Ending With No Deal as Saturday Overtime Drags OnIsraeli Strike Kills Three in Aid Vehicle in Gaza CityBasra Power Station Fire Kills at Least Four and Injures DozensClimate News Is Losing Its Audience — What the Research Says Can Win Them BackSouth Africa’s First Woman Deputy President Phumzile Mlambo-Ngcuka Dies Aged 71US Denies Iran Claim of Missile Hit on F-16 Over GulfEuropean Week for Waste Reduction Opens Today: Living Well With Less Under ‘Material Sufficiency’France and US Draft Plan for Life After UNIFIL in LebanonSecond Tanker Hit Near Strait of Hormuz on Saturday as Iran Claims US Drone Downed
Technology

Google, Microsoft, OpenAI and Anthropic Back One Shared AI Training Opt-Out for Publishers

News Desk 2Published November 1, 2026
Google, Microsoft, OpenAI and Anthropic Back One Shared AI Training Opt-Out for Publishers
Share this story

Four of the world’s biggest artificial intelligence companies — Google, Microsoft, OpenAI and Anthropic — will honour a single shared signal that lets a website opt out of having its content used to train generative AI models. The move, announced on Friday, 30 October, answers a complaint publishers have pressed for years: keeping content out of training sets currently means tracking down each company’s crawler and writing a blocking rule for it by name.

FAKTA

  • Google, Microsoft, OpenAI and Anthropic will honour a shared robots.txt signal, __usage_ai-training.
  • A publisher adds one group to robots.txt; participating companies read it the same way.
  • The token comes from an individual Internet-Draft by Anthropic’s Kevin Kelley, submitted to the IETF’s AI Preferences working group.
  • Anthropic has already started support; Microsoft starts 1 February 2027; Google aims for early 2027; OpenAI aims to follow within months.
  • The signal covers future training only: it does not remove already-collected content, stop crawling, or remove sites from search results.

AI training opt-out: software developers at work

AI training opt-out: how the shared signal works

Under the shared approach, a publisher adds a policy entry to the robots.txt file — the decades-old file that governs web crawlers — stating a preference about training use. The entry uses the token __usage_ai-training as a user-agent group, followed by Disallow lines marking what is off limits, with Allow lines able to make exceptions, for example for a press kit.

Until now, a publisher had to list GPTBot, Google-Extended, ClaudeBot and a growing pile of other crawlers one by one, then keep that list current as new crawlers appeared. The shared token is designed to replace that patchwork with one rule that every participating company reads identically.

Different timelines for the same pledge

The four companies have not promised the same start date. Anthropic said it has already started rolling out support. Microsoft will begin honouring the preferences from 1 February 2027 as its Bingbot crawler revisits pages, and will stop recognising its NoArchive and NoCache tags as training or grounding signals after 1 March 2027.

Google aims to support the shared control in 2027, with its crawler documentation narrowing that to early 2027; where a site opts out through the shared line without mentioning Google-Extended, Google plans to read it as if the publisher had disallowed Google-Extended. OpenAI said it aims to honour the signal within the next few months while its current robots.txt policy stays in effect.

AI training opt-out: developer working with code on a large screen

What the signal does not do

The limits are as specific as the promise. The setting applies only to future training of generative models, and it does not prevent bots from crawling a site otherwise. Microsoft said its controls can distinguish AI training from AI grounding, in which a model is pointed to verified sources to improve answers, and that blocking grounding will significantly reduce a site’s visibility in AI-generated answers.

The draft is also not yet an official standard: it is an individual Internet-Draft that the IETF has not endorsed and that has no formal standing in the standards process. Microsoft stressed that the new preferences do not establish licensing or payment arrangements, and the reports note the signal says nothing about deleting existing datasets or paying publishers.

Conclusion

For publishers, the practical effect will develop gradually through 2027 as each company’s crawler adopts the shared rule. The direction, though, is clear: after years of fragmented opt-outs, the biggest AI companies have agreed that one line in robots.txt should be enough to say no to training.

Related Articles

Frequently Asked Questions

Related Stories

Planet & environment

Climate & Sustainability

News, stories and solutions driving change towards a greener and more sustainable planet for future generations.

COP31 Final Deal Close as Presidency Tables New Package
Featured story

COP31 Final Deal Close as Presidency Tables New Package

COP31 talks in Antalya approached a final deal early on Sunday after the presidency announced a new package, with delegates given until 03:00 GMT to raise objections ahead of an approval session at 06:00 GMT.

Read full story

GREEN JOURNALISM

Every Action Counts

Small steps today, big impact tomorrow.

Explore more stories