What Does the New AI Crawling Standard Mean for Your Brand's Visibility?

What changed with the AI crawling standard?

The change is a proposed shared preference signal for AI training, written in an ordinary robots.txt file. The October 30, 2026 IETF Internet-Draft introduces the policy token __usage_ai-training, which lets a site operator express an allow or disallow preference for a site or a path.

Before this draft, a publisher that wanted to express a training preference had to manage crawler-specific rules and product controls. That can mean separate entries for different companies, separate names for different crawlers, and separate documentation to check when a provider changes its approach. The proposed token is designed to give participating AI developers one common signal to read.

The draft does not add a new robots.txt directive or alter the underlying robots.txt grammar. It uses a User-Agent group keyed to a policy token. A whole-site training preference can be expressed with the following logic: the token identifies the purpose, while Disallow: / expresses a preference against AI training across the site.

That distinction matters for brands. The proposal addresses content use for future model training. It does not create a universal control over every form of AI retrieval, every search index, or every AI-generated answer. A site can make a clear training preference and still need a separate decision about whether its content should remain available to inform answers that cite web sources.

The draft is also work in progress, not a finished IETF standard. It was published as Internet-Draft draft-kelley-aipref-training-preference-signal-00 on October 30, 2026 and is scheduled to expire on May 3, 2027 unless it is updated, replaced, or otherwise advanced through the IETF process. Treat it as an emerging interoperability mechanism, not as a settled compliance requirement.

1. Read the shared signal as a preference about future generative-AI training.
2. Keep crawler access controls separate from training preferences.
3. Keep answer-grounding and citation visibility separate from training preferences.
4. Recheck provider documentation before changing production robots.txt rules.

Does the new standard stop AI bots from crawling your website?

No. The proposed standard does not tell a crawler to stop fetching pages. The IETF draft says that policy-token groups are independent from crawler rules, so existing robots.txt traffic-management rules continue to work as they did before.

That is the most important correction to make when evaluating the news. A training-preference signal describes a permitted or disallowed use of content after collection. Crawl restrictions govern access. They overlap operationally because both may appear in robots.txt, but they answer different questions.

A crawler dedicated only to collecting training data may decide not to fetch pages it cannot use under a published preference. That practical outcome does not turn the training signal into a crawl block. A crawler that has another legitimate function under its own product rules may still fetch pages unless its crawler-specific access rule says otherwise.

For a visibility team, this creates a necessary inventory. Identify which controls currently exist in robots.txt, which ones relate to search crawling, which ones relate to named AI crawlers, and which ones express a use preference. Do not collapse them into one policy simply because they live in the same file.

Removing or broadening an existing restriction without mapping its effect can create a visibility problem. Adding a broad restriction without mapping its effect can do the same. The correct objective is a documented policy that preserves the discovery and answer surfaces your brand wants while expressing the content-use boundaries your organization has approved.

1. Crawl control asks whether a bot may fetch a URL.
2. Training preference asks whether fetched content may be used for future generative-model training.
3. Grounding control asks whether content may help inform a generated response.
4. Citation visibility depends on an engine deciding that a page is useful and eligible for a particular answer.

Who does the new AI crawling standard affect?

The proposed signal affects any site owner that publishes robots.txt and has a view on how generative-AI developers may use its content. That includes B2B software companies, publishers, retailers, service businesses, marketplaces, research organizations, and local brands with pages that could appear in AI answers.

It matters most to organizations that currently maintain several crawler-specific policies. A shared signal can make the training-preference portion of that policy easier to understand and audit if providers adopt it consistently. The value is operational clarity, not an automatic gain in visibility.

Content teams also need to care because training and answer use are different decisions. A company may prefer to disallow training while still allowing its strongest guides, service pages, documentation, and comparison pages to inform answers. Another company may decide that certain premium, licensed, or customer-only materials need tighter handling than publicly available educational pages.

Microsoft's October 30, 2026 announcement shows how this separation can work at a provider level. Microsoft describes independent preferences for AI training and AI grounding, and says a publisher can disallow training while allowing content to help inform generated responses. Microsoft also says that disallowing grounding can reduce brand visibility in AI experiences, because grounding is how web content can inform responses that include links and citations.

This creates an editorial question as much as a technical one. Which pages is your brand accurately enough to be used in an answer? Pages that are thin, stale, unsupported, or inconsistent create weak citation candidates even if they remain accessible. A preference setting cannot compensate for incomplete coverage or weak evidence.

1. Website owners need a documented policy for training use and answer use.
2. Technical teams need to map current robots.txt rules before making changes.
3. Editorial teams need pages with clear answers, current evidence, and direct sourcing.
4. Leadership needs to decide which content categories require distinct treatment.

What did the policy look like before, and what should brands do after the change?

Before the draft, training preferences were fragmented across provider-specific crawler names and controls. After the draft, the shared __usage_ai-training token has one proposed place to express a training preference for participating consumers. The practical response is to simplify only after confirming how each engine documents and honors the signal.

The following before-and-after is the citable operational change. It separates the common training-preference signal from the controls that remain necessary for crawl access and answer visibility.

Do not copy an example into production because it appears in a news story. Robots.txt has broad consequences, and a rule can apply more widely than intended. Confirm the current production file, keep a rollback copy, validate the syntax, and test the relevant provider tools or documentation after deployment.

Use the rollout to establish an owner. Someone should be accountable for the robots.txt file, the record of why each AI-related rule exists, and the periodic review of provider behavior. Without that ownership, a short file can become a collection of old assumptions that nobody can safely change.

1. Audit the current robots.txt file and all AI-related controls.
2. Classify every rule as crawl access, training preference, grounding preference, or another purpose.
3. Set an approved default for public editorial pages, product pages, documentation, and restricted content.
4. Monitor citations and answer presence after any visibility-affecting change.

How should you respond if AI citations matter to your brand?

Respond by protecting access to the pages you want AI systems to cite, then make the training decision separately. Citation visibility depends on more than a crawler setting. An answer engine needs a page that directly addresses the question, supports its claims, and remains available under that engine's current rules.

Start with your highest-intent question set. For a B2B software brand, that may include category questions, alternatives, implementation questions, comparisons, pricing-context questions, and integration questions. For a local business, it may include service-and-location questions, availability questions, qualification questions, and booking questions. Check whether the pages behind those answers are current, accessible, and precise.

Next, establish a content-use policy with legal, editorial, and technical owners. The policy should answer whether public content may be used for future generative training, whether it may inform answer grounding, and whether some paths deserve separate handling. The new training signal supports path-level preferences, so the decision does not have to be one blunt sitewide rule.

Then measure the outcome that matters. Omnicite calls the percentage of relevant AI answers in a category that cite your brand Citation Share. Track it alongside Answer Presence and the actual questions where a competitor is cited instead. That gives your team a way to distinguish a policy preference from a visibility result.

Finally, keep the editorial work moving. A clear robots.txt policy does not make a brand citable by itself. Publish pages that answer specific questions, show source-backed evidence, state the scope of their claims, and stay fresh when facts change. Rankings got you found. Citations get you chosen.

1. Preserve the technical eligibility of pages that support commercial questions.
2. Make training and grounding decisions independently where provider controls allow it.
3. Use path-level treatment for content that has different business or rights requirements.
4. Measure Citation Share on the prompt set your buyers actually use.

What are the limits and risks of the new AI crawling standard?

The main limit is adoption. The Internet-Draft defines a mechanism, but each AI developer must decide whether and how it will implement the signal. The draft itself says consumers that do not implement the mechanism behave as before. A published preference does not prove that every crawler or product will interpret it.

The second limit is scope. The draft addresses future generative-AI training. It does not establish a license, create payment terms, settle rights questions, or govern content already collected. It also does not make a page eligible for a citation, a featured answer, or a traditional search result.

The third limit is provider-specific behavior. Microsoft's announcement says its training and grounding preferences are independent, applies its new preferences from February 1, 2027, and recommends transition steps by March 1, 2027 for sites using its older NoArchive or NoCache preferences. Other providers can document different products, dates, controls, and limitations.

The risk is treating one configuration change as an AI visibility strategy. A broad restriction can reduce the ability of an engine to use your content in answers. A broad allowance does not ensure a citation. Your policy must match your rights position, while your content operation must earn citation eligibility through quality, coverage, and freshness.

1. A draft is not a final standard or a universal technical guarantee.
2. A training preference is not a crawler block.
3. A training preference is not an answer-grounding preference.
4. An allowed page is not guaranteed to receive a citation.

What should your team do next?

Your next move is an AI-access audit, not an immediate rewrite of robots.txt. Document every existing rule, name the provider or purpose it affects, identify the public pages that drive qualified demand, and decide whether training and answer use need different treatment.

For brands pursuing AI search visibility, preserve a clean route from question to evidence. Make the best public pages easy to crawl where that matches your approved policy, keep their facts current, and review whether answer engines cite them for the questions that matter. If you adopt the shared training-preference signal, record the exact change and the provider documentation that justified it.

The new AI crawling standard can make one part of web policy less fragmented. It does not remove the strategic work. Brands still need a deliberate approach to crawl access, content use, answer grounding, and Citation Share.

1. Assign an owner for AI-related robots.txt controls.
2. Audit rules before changing them.
3. Document the approved training and grounding policy.
4. Measure Citation Share before and after material changes.

Sources

Source: Internet Engineering Task Force

The proposed mechanism uses the __usage_ai-training policy token in ordinary robots.txt groups and does not change robots.txt grammar or crawler rules. Internet Engineering Task Force, 2026-10-30

Source: Internet Engineering Task Force

Internet-Drafts are work in progress, may be updated or replaced, and this draft expires on May 3, 2027 unless advanced or revised. Internet Engineering Task Force, 2026-10-30

Source: Microsoft Bing Blog

Microsoft describes independent AI training and AI grounding preferences, says it will honor new preferences from February 1, 2027, and says grounding restrictions can reduce AI-experience visibility. Microsoft Bing Blog, 2026-10-30

Source: TechRadar

Google, Microsoft, OpenAI, and Anthropic were reported as participating around a shared training-preference proposal, while implementation timing and interpretation varied. TechRadar, 2026-11-09