How to Opt Out of AI Training With robots.txt (2026)

On 30 October 2026, Google, Microsoft, OpenAI and Anthropic lined up behind a single shared robots.txt signal for AI training. Instead of writing a separate rule for each company’s crawler, a site owner can now write one group in robots.txt and have every participating provider read the same answer: do not use this content to train generative AI models. This guide shows you how to add that rule, what it does and does not do, and the dates that decide when it actually starts working.

Quick answer: Add User-Agent: __usage_ai-training with Disallow: / to robots.txt to tell Google, Microsoft, OpenAI and Anthropic not to train on it. Search keeps working.

What is the __usage_ai-training rule?

The signal is a policy token, not a new crawler name. In robots.txt you write an ordinary rule group, but the User-Agent line names a use — AI training — rather than a bot. The token is __usage_ai-training, with two leading underscores. Under it, Disallow: / means do not train on anything on the site, while an Allow line can carve out exceptions. The idea comes from an Internet-Draft titled “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley, a technical lead for standards at Anthropic, and filed with the Internet Engineering Task Force (IETF) on 30 October 2026. It is an individual draft, not an approved standard: it carries no formal IETF standing yet and expires on 3 May 2027 unless it is revised or adopted. That matters for expectations. You are publishing a stated preference that four large providers have promised to honour, not flipping a switch that is enforced today. “Training” here has a specific meaning, taken from the IETF AI Preferences (AIPREF) working group’s vocabulary work: using content to modify the learned parameters of a generative AI model. It is not the same as crawling, indexing, ranking, or quoting a page in an answer. Keep that distinction in mind for every decision below, because the most common mistake with this rule is assuming it does more than it does.

Add the site-wide opt-out in 3 steps

Step 1 — Find your robots.txt file. It lives at the root of your site, at yourdomain.com/robots.txt. On WordPress, check whether an SEO plugin or your host manages it before editing by hand; if a plugin generates the file, edit it through the plugin so your change is not overwritten on the next save.
Step 2 — Add this group, on its own lines, at the end of the file:
```
User-Agent: __usage_ai-training
Disallow: /
```
Step 3 — Test the file. Load yourdomain.com/robots.txt in a browser and confirm the new group appears exactly as written, with two underscores before “usage” and no extra spaces in the token. Then check your site still loads normally and that search engines can still reach it: this token states a training preference, and the draft says plainly that it “is not a directive for crawling behavior.” It does not block crawlers from visiting, and Google and Microsoft both say it does not change ordinary search rankings.

One trap to avoid: your existing wildcard group does not cover this. A group starting User-agent: * must not be interpreted as matching the policy token under the draft, so a site with only a wildcard rule has expressed no training preference at all. The token group has to be present in its own right.

Opt out of part of your site only

You do not have to choose all or nothing. The draft’s own example opens one folder while closing the rest:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That pattern says: no training anywhere except the press kit. The same longest-match path rules that govern ordinary robots.txt apply, so put the exception you care about in an Allow line and keep the broad Disallow above it. A practical use is a publisher that wants its marketing material to remain available for training while keeping its reporting, photography or paid archive out. Whatever you carve out, name it deliberately — an Allow line is a permission, and it will be read as one.

Keep your existing per-company rules for now

This is the most important practical point in the whole guide: do not delete your current rules yet. None of the four providers reads the shared token everywhere today. Anthropic says it has already started implementing support; OpenAI says support comes “in the coming months” with no firm date; Microsoft starts on 1 February 2027; Google aims for 2027, with its crawler documentation narrowing that to early 2027. Until each provider’s date arrives, the old controls are the ones doing the work: GPTBot rules for OpenAI, Google-Extended for Google, and ClaudeBot rules for Anthropic. Run the shared token alongside them, not instead of them, and treat the shared token as the forward-looking rule that takes over as each rollout lands.

What changes at Google — and the Gemini grounding catch

Google’s announcement came from John Mueller, Google Search Advocate, on the Google Search Central blog on 30 October 2026. Google says it aims to support the shared control along with other AI providers in 2027. The catch is how Google reads the token. If a site opts out of training with the shared token and does not explicitly mention Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Google-Extended is not only a training switch: Google’s crawler documentation says it covers training future Gemini models and grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. So at Google, the same robots.txt group that opts you out of training can also take you out of Gemini’s grounded answers, where a page is handed to the model at prompt time and can earn a citation and a click. Google also says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal, and its separate Search Console control for AI Overviews and AI Mode is a different switch again — the shared token does not control those features. If Gemini grounding traffic matters to you, this is the trade-off to weigh before you paste the site-wide rule, and the reason the decision checklist below exists.

What changes at Bing — training split from grounding

Microsoft’s Bing announcement of 30 October 2026 goes further than the shared token, and it is the clearest of the four. Bing splits two things its older NoArchive and NoCache tags had bundled together since 2023: AI Training (using a page to change a model’s parameters) and AI Grounding (using a page as input to an AI-generated answer in Bing, Copilot and Microsoft 365). Both preferences default to Allow, both are honoured as Bing crawls from 1 February 2027, and neither changes eligibility for ordinary Bing search results. Training is set site-wide or per path in robots.txt with the shared token group, and can be overridden on a single page. Grounding is page-level only — there is no site-wide robots.txt form — set with a meta tag or an X-Robots-Tag response header naming Bingbot, in the reported form ai-training:disallow and ai-grounding:disallow. Grounding also has a middle setting, DisallowExtended, which keeps content available to Microsoft’s own products while withholding it from third-party grounding services Microsoft sells. Microsoft’s own example makes the point: you can set AI Training to Disallow and AI Grounding to Allow, so refusing to train a model no longer means disappearing from the answer. Two deadlines sit around the rollout: temporary grounding transition preferences open in Bing Webmaster Tools on 15 December 2026 for sites using NoArchive or NoCache on that date, and after 1 March 2027 Bing stops reading NoArchive and NoCache as training or grounding signals at all. If your site still relies on those tags, migrate before March. Check Bing’s own AI preferences help page for the final value names before deploying the page-level tags, since this rollout has not started yet.

When each provider starts honouring the rule

Rollout table — provider and status as announced 30 October 2026:

Provider	Status
Anthropic	Already started implementing support, per the draft’s author.
OpenAI	Support in the “coming months”; no firm date published.
Microsoft (Bing)	Honours the new training and grounding preferences from 1 February 2027; NoArchive and NoCache stop counting after 1 March 2027.
Google	Aims to support the shared control in 2027; crawler documentation says early 2027.

Two further facts belong in any honest version of this table. First, Meta is not a signatory in this coverage: the four companies are Google, Microsoft, OpenAI and Anthropic, and Meta’s absence means the token says nothing about Meta’s training. Second, the underlying specification is a version 00 individual Internet-Draft that expires on 3 May 2027 unless revised — the shared signal only stays shared if the draft is kept alive and adopted. Treat the dates above as vendor plans, not guarantees; plans slip, and the draft’s expiry is a real deadline to watch.

What this rule does not do

It does not block crawling. Crawlers can still visit and index your pages. It does not remove your site from Google or Bing search results, and it is not a ranking signal. It does not delete or unlearn content already collected and used in existing training data or models — it covers future training only. It does not stop a page being cited in an AI answer today, except through the separate grounding controls described above (Google-Extended at Google, ai-grounding at Bing). And it does not bind companies that have not adopted it. If a provider is not on the list, this token is just text in a file to them. Anyone selling this rule as a delete button for past training, or as a way to vanish from search, is overselling it.

Our take: five questions before you opt out

This is the article’s original element: a decision checklist built from the rollout split above, because the right answer differs by site. Work through it before you edit robots.txt.

Do you want your pages cited in AI answers? If yes, check the Google catch first: a site-wide training opt-out is read at Google as a Google-Extended disallow, which can cost Gemini grounding. At Bing you can split the two, but at Google you cannot with this token alone.
Do you still rely on NoArchive or NoCache for Bing? If yes, diary 1 March 2027 now and migrate to the new preferences before that date, or your restriction silently stops being read.
Is any part of your site worth keeping available for training? Press kits, documentation and public research are common carve-outs; use an Allow line for them rather than leaving the whole site open.
Who else trains on your content? Meta is not covered, and smaller crawlers are not bound by this token. If a specific crawler matters to you, keep its own rule in place.
When will you re-check? Diary early 2027 for Google’s rollout, 1 February 2027 for Bing, and May 2027 for the draft’s expiry. A preference nobody re-checks is how a site ends up opted out of something it wanted, or open to something it refused.

Worked example: a small independent blog that earns traffic when Gemini cites its guides, but does not want its archive used for training, faces a genuine conflict at Google — the shared token alone cannot give it both. The honest options are to hold the site-wide opt-out until Google clarifies a split control, to opt out and accept the Gemini grounding trade-off, or to carve out only the archive paths that matter most and leave the guides open. There is no version of this rule that quietly delivers both at Google today, and any guide that claims otherwise is guessing.

Frequently Asked Questions

Does adding this rule remove my site from Google Search?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and Microsoft says its new preferences do not change eligibility for ordinary Bing results. The token states a training preference, not a crawl block.

Does it delete data already used to train a model?

No. The signal covers future training only. It says nothing about content already collected or incorporated into existing models, and it creates no deletion or payment obligation in the draft.

Do I still need my GPTBot, Google-Extended and ClaudeBot rules?

Yes, for now. The shared token rolls out through 2027, with Bing firmest at 1 February 2027 and Google aiming for early 2027. Until a provider’s date arrives, its existing control is the one it reads.

Can I protect only some pages?

Yes. Use Disallow: / with an Allow line for the folder you want to keep open, following the draft’s press-kit example. Path rules follow ordinary robots.txt longest-match precedence, so write the exception explicitly.

Is Meta covered by this opt-out?

No, not in the coverage of the 30 October 2026 announcements. The four participating companies are Google, Microsoft, OpenAI and Anthropic. Meta is not a signatory, so this token states no preference to Meta.

Sources and methodology

Sources:

Methodology: claims in this article rest on the 30 October 2026 announcements by Google (Search Central blog, John Mueller), Microsoft (Bing blog), OpenAI and Anthropic (Internet-Draft draft-kelley-aipref-training-preference-signal-00), as reported 30 October–2 November 2026 and re-verified 4 November 2026. Rollout dates are vendor plans and may change; the Microsoft page-level tag syntax is the reported form and should be checked against Bing’s own help page before deployment. This guide is desk-researched: vendor support for the shared token does not start until 2027, so it could not be tested against live crawlers, and that limit is stated plainly here.