How Does the New Robots.txt Rule Affect AI Training Opt-Outs?

What changed in the AI training opt-out rule?

The change is a shared robots.txt signal for communicating AI-content preferences to Google and Bing. Before this move, publishers generally had to rely on crawler-specific user-agent rules, such as Google-Extended, or accept that a general search-crawling rule could also affect visibility in search.

The new approach separates common AI uses into explicit preferences. A publisher can state whether content may be used for search, as input to AI answers, or for AI training. That is more precise than a broad Disallow rule because it tells a participating engine what kind of use the publisher permits or declines.

The source report describes Google and Bing backing one robots.txt rule for training opt-outs. The practical shift is interoperability: publishers can express a preference once rather than maintain an expanding collection of engine-specific instructions. It is a meaningful improvement in communication, not proof that all AI crawlers use the same policy.

Robots.txt remains a voluntary crawl directive. It tells a compliant crawler how the publisher wants content handled. It does not remove pages already collected, prevent a person from copying public material, or compel an AI company that has not adopted the signal. The Internet Engineering Task Force describes the Robots Exclusion Protocol as a way for website owners to communicate crawler access rules, while noting that it is not an access-control system.

For AI search teams, that distinction matters. Blocking every crawler can also block discovery, index updates, and the evidence that supports AI search visibility. A training preference should be set deliberately, with a record of the exact directive and the crawlers that recognize it.

1. Before: publishers often managed separate user-agent rules for each crawler or used a broad block that could affect search crawling.
2. After: participating Google and Bing systems can read one declared preference for AI-related use.
3. Still true: robots.txt communicates a preference to compliant systems. It is not a universal technical barrier.

How does the new rule compare with crawler-specific opt-outs?

The shared rule improves portability, but crawler-specific controls still matter. Google documents Google-Extended as a standalone token that publishers can disallow in robots.txt to control whether Google may use content to improve Gemini and other generative AI products. Google also says that Google-Extended does not affect a site's inclusion or ranking in Google Search.

That separation is useful for publishers that want search traffic while declining Google model training. The new common signal makes the same policy easier to express across engines that support it, but it should not be treated as a substitute until an engine's documentation says it honors the signal for the relevant product.

Bing's search crawler and its AI systems also have their own documented policies and user agents. A site owner should read the current Bing guidance before removing an existing rule. If a legacy directive covers a system that the new shared rule does not explicitly cover, deleting it creates uncertainty rather than simplifying governance.

The clearest setup is layered. Use the shared signal for the preference that Google and Bing now recognize. Retain a crawler-specific directive where official documentation still calls for one. Then test the live robots.txt file from the site's root domain, because a correct line in a staging file has no effect on the production crawler.

Do not confuse training with answer inclusion. A publisher may care about whether a model trains on content, whether an AI answer retrieves or summarizes a page, and whether the page remains searchable. Those are different uses. Keeping the distinction clear prevents an AI training opt-out from becoming an accidental ban on the visibility work behind Citation Share.

1. Use the shared signal when an engine documents support for it.
2. Keep Google-Extended or another crawler-specific rule when its official documentation still requires it.
3. Review search, AI-answer input, and training as separate permissions.
4. Record the policy decision, the live URL, and the date it was checked.

Who does the new robots.txt rule affect?

The rule affects any public website that has content a participating crawler can request. Publishers, SaaS companies, newsrooms, marketplaces, documentation sites, local businesses, and agencies all need a decision because each site can publish a preference even if it has no dedicated AI policy today.

It matters most for teams that have already added multiple AI-crawler rules. Those teams have a maintenance problem: every new crawler name can produce another robots.txt edit, a new review cycle, and a chance of blocking a search bot by mistake. A shared signal can reduce that sprawl where Google and Bing honor it.

It also affects businesses that depend on being cited in AI answers. Omnicite measures whether a brand appears across ChatGPT, Perplexity, Gemini, Copilot, and AI Overviews. A decision to opt out of training does not automatically answer the separate question of whether content should be available as input for AI answers. Teams should decide both policies rather than assume one setting controls both outcomes.

Content owners with contractual or licensing restrictions have a stronger reason to document their preference. The robots.txt signal can show a clear technical instruction at the domain level, but legal, licensing, and access-control decisions may require separate work. The directive does not change rights in the underlying content.

Small sites should not assume they are outside the change. The file is public, the implementation is low effort, and a clean policy now is easier to maintain than a rushed response after a crawler or vendor policy changes.

1. Publishers with existing AI bot blocks should map old rules before changing them.
2. Brands seeking AI citations should separate training choices from answer-retrieval choices.
3. Sites with licensed, regulated, or restricted content should involve the relevant legal or policy owner.
4. Any site using robots.txt should check that the production file is reachable at /robots.txt.

How should you respond to the new AI training opt-out rule?

Start with an inventory, then make one explicit policy decision for each AI use. List every existing robots.txt user-agent rule, the crawler it targets, and the business purpose behind it. Do not remove lines simply because the new rule exists. The correct response depends on which engines visit the site and what each current rule controls.

Next, decide whether the organisation permits search crawling, AI-answer input, and AI training. These choices should come from the content owner, legal policy, product team, or editorial lead. A growth team can explain the effect on discoverability, but it should not guess at licensing intent or create an opt-out policy on behalf of the owner.

Publish the shared signal in the production robots.txt file according to the participating engines' current documentation. Preserve existing crawler-specific rules until you have confirmed that the shared signal covers the same system and use case. Use a source-controlled change, because robots.txt is operational infrastructure and a small syntax error can change crawler access across the whole domain.

Verify the response in two ways. First, request the public robots.txt URL and confirm the new line appears on the canonical host, including the www or non-www host that crawlers use. Second, compare the file with the current official Google and Bing documentation. A syntax checker can confirm formatting, but it cannot tell you whether an engine supports a newly introduced policy.

Finally, monitor outcomes rather than promise one. A training opt-out does not guarantee changes in rankings, traffic, citations, or citations lost. Track crawler logs where available, organic search health, and AI-answer presence on the prompts that matter to the business. If visibility changes after a policy edit, examine the exact bot rules and crawl data before attributing the change to AI training preferences.

This is also a governance moment. Add a quarterly review for robots.txt and AI-crawler policy. Engine names, documentation, and product boundaries change faster than traditional search rules. A written owner and review date protect the site from a patchwork of old directives.

1. Inventory every current robots.txt rule and identify its targeted crawler.
2. Choose separate policies for search crawling, AI-answer input, and AI training.
3. Add the shared signal only from current official syntax documentation.
4. Keep un-replaced crawler-specific directives in place.
5. Verify the production file and review crawler logs after deployment.
6. Set a named owner and a future policy-review date.

Does the shared rule make AI training consent enforceable?

No. The shared rule makes a publisher's preference clearer to participating crawlers. It does not make robots.txt an enforcement mechanism, convert a preference into a license, or prove that content was never used in previous training.

That boundary should shape internal communications. Say that the organisation has published an AI training preference that Google and Bing support, not that all AI models are blocked. The more precise statement is also easier to defend when a crawler's behavior, product terms, or documentation change.

The strongest practical control depends on the risk. A public robots.txt preference may be appropriate for ordinary web publishing. Content that must remain private needs access controls. Licensed material needs contract and rights-management work. A training preference is one policy layer among several.

For teams focused on citation outcomes, the operational question is equally specific: which crawling and retrieval permissions support the intended visibility strategy? Keep that question separate from the question of model training. Clear permissions make it easier to audit both.

1. Robots.txt states a crawl preference for compliant systems.
2. It does not control past collection, private redistribution, or every AI provider.
3. Use authentication and access control for content that must not be public.
4. Use contracts and rights processes when licensing terms must be enforced.

What should an AI visibility team document after making the change?

Document the policy choice, the exact live robots.txt URL, the directive added or retained, the approving owner, and the official documentation used. That record makes the next review faster and gives marketing, legal, and engineering one factual source of truth.

Add the decision to the same operating record as your AI visibility measurement. If the site tracks Answer Presence, citation counts, or competitive Share of Voice, preserve the dates of the robots.txt change beside the measurement history. Correlation is not causation, but a dated change log stops teams from reconstructing critical context from memory.

The new rule lowers the burden of expressing an opt-out across Google and Bing. It does not remove the need for careful configuration. The useful outcome is a policy that is explicit, deployed, verified, and reviewed when engine documentation changes.

1. Save the production robots.txt URL and a copy of the deployed rule.
2. Name the policy owner and approver.
3. Link the current Google and Bing documentation used for the decision.
4. Record the deployment date beside AI visibility measurements.
5. Review the rule when either engine updates its crawler guidance.

Sources

Source: Google Search Central

Google documents Google-Extended as a standalone product token that publishers can disallow to control use in Gemini and other generative AI products, without affecting Google Search inclusion or ranking. Google Search Central, 2023-09-28

Source: PPC Land

Google and Bing back a shared robots.txt rule for communicating AI training opt-out preferences. PPC Land, 2026-10-30

Source: Internet Engineering Task Force

RFC 9309 defines the Robots Exclusion Protocol as a method for website owners to communicate crawler access rules. Internet Engineering Task Force, 2022-09-01