How to Use Robots.txt to Manage AI Training of Your Brand Data

What changed in robots.txt AI training controls?

Google and Microsoft announced support for a shared AI training preference in robots.txt on 30 October 2026. The preference uses the policy token `__usage_ai-training`, which lets a publisher disallow AI training for an entire site or a path without relying only on provider-specific controls.

The whole-site example published by Microsoft is simple: `User-Agent: __usage_ai-training` followed by `Disallow: /`. A site can also disallow training only for a directory, such as `/research/`. Microsoft says allowing training is the default, so a site does not need a robots.txt directive to permit it.

Google describes the signal as a shared policy token that can express broad preferences across participating AI providers. When a site opts out through the token and does not explicitly specify Google-Extended, Google plans to read that preference as though Google-Extended were disallowed. Google also says its existing provider-specific controls will continue to work alongside the shared signal.

This is an important change because the directive is about the use of content after collection, not ordinary search access. Google says the shared signal concerns AI training. Microsoft says its new preferences do not change eligibility for traditional Bing search results, subject to ordinary crawl and indexing requirements. A publisher can therefore preserve search visibility while making a separate decision about model training.

The control remains early-stage. The Internet Engineering Task Force lists the underlying document as an active individual Internet-Draft, updated on 30 October 2026, and explicitly says it is not endorsed by the IETF or part of its formal standards process. Treat the syntax as a live industry implementation rather than a final web standard.

1. Shared signal: `__usage_ai-training` can express a training preference in robots.txt.
2. Google: plans support in 2027 and retains Google-Extended as a separate control.
3. Microsoft: published the robots.txt syntax and will honor new preferences as it crawls from 1 February 2027.
4. IETF status: the cited document is an active Internet-Draft, not an endorsed standard.

Who does the new AI training preference affect?

The change affects any publisher whose pages contain material they may want AI systems to train on, including brands, publishers, software companies, research organizations and ecommerce sites. It matters most when a team has previously treated crawl access as a single yes-or-no decision.

A brand that wants content cited in generated answers but does not want it used to train future models now has a clearer choice in Microsoft products. Microsoft distinguishes AI training from AI grounding. Training changes the learned parameters of a generative model. Grounding uses web content as input to help inform a generated response.

That distinction matters for Citation Share, the percentage of relevant AI answers in a category that cite a brand. A page that is unavailable for grounding may have less opportunity to inform a generated response or receive a citation path. Microsoft says explicitly that disallowing grounding can reduce brand visibility in AI experiences.

The new shared training signal does not mean that every AI provider has adopted identical behavior. Google says it aims to support the control in 2027. Microsoft has set an effective date for its own new preferences. The IETF draft remains a draft. Each publisher should identify which providers crawl its content, which existing directives are present and which content uses need different treatment.

This affects technical SEO, content, legal and brand teams together. The technical team controls implementation. The content team needs to know whether key evidence pages remain available for grounding. Legal or policy owners may need to decide how the organization treats model training, especially for proprietary research, customer documentation or paid content.

1. Publishers with proprietary or licensed material need a training-use decision.
2. Brands seeking citations need a grounding-use decision.
3. Teams using Google-Extended need to understand how it interacts with the shared signal.
4. Sites with legacy NoArchive or NoCache directives need a transition plan before Microsoft's deadline.

How do training and grounding differ?

Training and grounding are separate uses of the same page. Training uses content to modify a model's learned parameters. Grounding uses content at answer time to help inform a generated response. Microsoft says that indexing and ranking a page in traditional search results is not grounding.

Microsoft allows the shared AI training preference in robots.txt or at page level. It says AI grounding is page-level only. A publisher can add a Bingbot meta tag or an X-Robots-Tag HTTP header to control grounding for an individual page. Microsoft documents `ai-grounding:disallow` for no grounding and `ai-grounding:disallowextended` for grounding in Microsoft-operated experiences while excluding grounding services Microsoft provides to third parties.

The practical implication is direct. A blanket robots.txt decision about model training does not answer whether a page should remain available to support generated answers. A content team that wants its authoritative pages considered for citations should avoid treating training and grounding as interchangeable settings.

Microsoft says both preferences default to allow when no applicable preference is published. It also says neither choice guarantees inclusion, traffic or payment. Allowing grounding can make content eligible to inform responses, but it does not guarantee that Bing, Copilot or another experience will select the page for an answer.

For a brand publishing original research, product documentation or category explainers, page-level grounding controls can be more precise than a sitewide move. Keep publicly useful source pages available where that matches policy. Apply more restrictive settings to pages whose content should not be used in generated responses. The decision should follow the actual content and business policy, not a generic fear of AI crawling.

1. AI training: affects future generative model training.
2. AI grounding: affects whether content can help inform generated responses.
3. Traditional Bing results: Microsoft says the new preferences do not change ordinary search eligibility.
4. Citations: allowing grounding creates eligibility, not a promise of inclusion or traffic.

What is the before-and-after timeline for publishers?

The before-and-after is clear for publishers using Microsoft's old NoArchive or NoCache preferences. Before the new controls take effect, those legacy directives may still be part of a site's existing approach to training and grounding preferences. After Microsoft detects the new preferences, it says it will disregard remaining NoArchive and NoCache preferences for those uses.

Microsoft will begin honoring the new AI training and AI grounding preferences as it crawls from 1 February 2027. It asks publishers that use NoArchive or NoCache to complete the corresponding transition steps before 1 March 2027. After that date, Microsoft says those older directives will no longer be read as preferences for training or grounding.

Google's timing is less specific. Its 30 October 2026 announcement says Google aims to support the shared training control with other providers in 2027. That makes the next period one for inventory and policy work, rather than assumptions that the shared signal has already reached every crawler.

The correct response is to audit first. Locate the production robots.txt file, capture current Google-Extended, NoArchive and NoCache directives, then identify which pages use Bingbot meta tags or X-Robots-Tag headers. A directive that blocks a crawler can affect discoverability differently from a usage preference. Do not add broad disallow rules before confirming their crawler and product effects.

1. Before 1 February 2027: inventory existing directives and decide training and grounding policy.
2. From 1 February 2027: Microsoft says it will honor new preferences as it crawls.
3. Before 1 March 2027: replace NoArchive and NoCache as Microsoft-directed training or grounding controls.
4. During 2027: monitor Google's implementation update for the shared training signal.

How should a brand respond without losing AI visibility?

Start by separating the policy question from the visibility question. Decide whether the organization permits AI training on each class of public content. Then decide separately whether those pages should remain eligible to ground generated responses. A brand that wants to be cited should protect the evidence quality, coverage and freshness of the pages it leaves available.

For training, use the shared robots.txt signal only after confirming the desired scope. Microsoft documents whole-site and path-level examples. A path-level preference can suit a site that wants to protect a defined research area while allowing a public resource library to remain under the default training setting. Google says provider-specific controls can remain alongside the shared signal, so existing Google-Extended choices should be reviewed rather than silently removed.

For grounding, review critical pages one by one. Product pages, research explainers and category comparisons can be candidates to remain eligible if they are public and support the brand's citation strategy. Pages with restricted-use material may require a different policy. Microsoft says a page-level AI training preference overrides a robots.txt setting, which adds precision but also makes implementation inventory essential.

Check deployment in production. Confirm the robots.txt response, page source and relevant HTTP headers at the live URL. Record what each directive is intended to govern. Then observe citation outcomes across ChatGPT, Perplexity, Gemini, Copilot and AI Overviews rather than equating a technical control with a guaranteed result.

The goal is not to game an answer engine. It is to make a deliberate publishing choice while keeping authoritative material ready to be cited where the brand permits it. Rankings got you found. Citations get you chosen.

1. Audit live robots.txt, meta robots directives and X-Robots-Tag headers.
2. Classify pages by training policy and grounding policy.
3. Replace Microsoft legacy controls before the stated deadline where applicable.
4. Verify live deployment after release.
5. Track Citation Share and Answer Presence after changes, without promising a citation outcome.

Sources

Source: Google Search Central Blog

Google announced plans to support a shared AI training control in robots.txt and said it aims to support it in 2027. Google Search Central Blog, 2026-10-30

Source: Microsoft Bing Blog

Microsoft published separate AI training and AI grounding preferences, effective crawling behavior from 1 February 2027, and a 1 March 2027 transition deadline for NoArchive and NoCache. Microsoft Bing Blog, 2026-10-30

Source: IETF Datatracker

The Machine-Readable Training Preference Signal is an active individual Internet-Draft, last updated on 30 October 2026, and is not endorsed by the IETF. IETF Datatracker, 2026-10-30