What News Sites Are Blocking AI Crawlers and Why It Matters for Your Brand?

What changed in AI crawler policies at news sites?

Major news sites are setting different access rules for the crawlers behind AI products. That creates a fragmented source environment: a publisher may permit one engine to fetch its pages while blocking another. The policy decision is usually expressed in robots.txt, a file that tells automatic crawlers which URLs they can access.

Five Blocks reviewed robots.txt files and page tags for 44 major news outlets on October 23, 2026. Its November 5 report found that 25 of the 44 outlets blocked Google's Gemini app, and the same number blocked Perplexity's crawler. Twelve blocked ChatGPT's search bot. The study also found that none of the sampled outlets blocked Google AI Overviews or AI Mode.

Those figures describe a dated sample of large news outlets, not a permanent map of the web. A robots.txt file can change, and each crawler has a separate user agent. The durable lesson is that access is no longer a single search visibility setting. Brands need to know which bot a publisher permits, which AI product uses that bot, and whether the resulting answer actually cites the source.

Google describes robots.txt as a way to manage crawler access and traffic. It also cautions that robots.txt is not a reliable method for keeping a web page out of Google Search. That distinction matters because a publisher can make a crawl decision without creating a clean, universal removal from every search or answer surface.

1. Check policies by named crawler, not by a generic label such as AI bot.
2. Treat robots.txt as a current policy signal, then recheck it before acting on the result.
3. Separate access, indexing, answer inclusion, and citation as different outcomes.

Which news sites and AI engines does the research affect?

The research affects brands whose visibility depends on earned coverage, industry reporting, executive mentions, and category narratives. If an outlet that covers your company is blocked for a given crawler, that outlet may contribute less directly to that engine's accessible source pool. It can still matter to readers, journalists, traditional search, and other engines.

Five Blocks tested 14,392 answers about Fortune 100 companies and their CEOs across eight AI engines. In that study, the Gemini app cited zero articles from the 25 outlets that blocked it. Google AI Mode cited those same outlets in 69% of news answers. This is a measured difference in one research design, not proof that crawler rules alone determine every citation.

The contrast is useful because it stops a common mistake: treating an AI answer as one unified channel. An outlet's terms can interact with engine-specific crawling, retrieval systems, product choices, licensed material, user-triggered browsing, and the answer system's own selection process. A mention in one answer engine does not prove the same coverage is usable in another.

OpenAI documents this separation explicitly. OAI-SearchBot is used for ChatGPT search results, while GPTBot relates to content that may be used to train OpenAI's generative AI foundation models. A site can allow one and disallow the other. Blocking GPTBot does not, by itself, describe whether a site can appear in ChatGPT search answers.

1. News publishers control their own crawler policies.
2. AI products do not necessarily share the same crawler or source behavior.
3. Brands should compare visibility by engine and by query set.

What does the dated before-and-after evidence show?

The dated evidence shows a gap between declared access policy and observed answer citations. Five Blocks recorded publisher crawler policies on October 23, 2026, then tested answer behavior in its 14,392-answer dataset. That sequence does not establish a universal causal rule, but it gives brands a concrete way to investigate citation access.

The strongest observed result concerned the Gemini app. Five Blocks reported zero cited articles from the 25 outlets that blocked it. In the same report, Google AI Mode cited those outlets in 69% of news answers. The practical implication is not to assume that a block applies everywhere. The implication is to test the exact engine and answer experience your buyer uses.

The study also reported a freshness concern. Only 30% to 60% of links in a news answer came back the next day. This finding is about link persistence in the study's answers, not a claim that any particular publisher disappeared or that every engine has the same turnover rate. It does make recurring monitoring sensible when a brand relies on fast-moving news coverage.

A dated policy record plus an answer-level citation test is more useful than a generic traffic report. It tells you whether the outlets that mention your brand are accessible to a specified crawler, whether the engine cites them for relevant questions, and whether the links remain available when checked again.

1. Before: Five Blocks read robots.txt files and page tags on October 23, 2026.
2. After: Five Blocks evaluated citations in 14,392 answers for Fortune 100 companies and CEOs.
3. Observed result: Gemini cited zero articles from sampled outlets that blocked its app.
4. Observed result: Google AI Mode cited those same outlets in 69% of news answers.

Why do AI crawler policies matter for your brand?

AI crawler policies matter because your brand can earn coverage that one answer engine can use less directly than another. Rankings got you found. Citations get you chosen. If a buyer asks an engine for the best provider in a category, the sources that engine can read and cite help define the answer set.

That does not make publisher outreach less important. A strong story in a respected outlet can influence readers, referral traffic, journalist research, conventional search, and future coverage. It does mean that earned media alone is an incomplete plan for AI search visibility. You need authoritative material on domains and pages that remain accessible to the engines relevant to your market.

The Five Blocks results also show why brands should avoid treating a citation as a permanent asset. A cited URL can disappear from a later answer. A crawler policy can change. An engine can alter how it retrieves news. Your reporting should therefore distinguish Citation Share, the percentage of relevant AI answers in a category that cite you, from Citation Count per day and Answer Presence across the question universe.

Owned content gives you a controllable source layer. Publish direct answers to real buyer questions, support them with dated evidence, make ownership and update dates clear, and keep pages technically accessible to the crawlers you choose to allow. Omnicite calls this Citation Engineering: building quality, coverage, and freshness at a scale AI systems can trust.

1. Map earned coverage to the engines that matter to your buyers.
2. Build owned evidence that does not depend on a single publisher's policy.
3. Monitor Citation Share and Answer Presence across separate engines.
4. Refresh source pages when facts, policies, or buyer questions change.

How should your brand respond to blocked AI crawlers?

Start with an audit of your own crawler permissions and your citation sources. List the pages that explain your category, product, proof, and comparisons. Then record which bots can access them. Do not copy a block or allow rule from another company without understanding its purpose, because OpenAI and Google document separate agents for different functions.

Next, map the publishers that drive your earned visibility. For each outlet, record the article URL, publication date, mention context, relevant crawler policy at the time checked, and the AI engines that cite it for priority prompts. Keep the policy observation separate from the citation result. A block is evidence about crawler access, while a citation test is evidence about an answer output.

Then close gaps with source-worthy owned pages. A page should answer a specific question early, show who wrote it, link to original evidence, and carry a meaningful update date. Do not manufacture statistics or write thin pages to chase a crawler. Quality, coverage, and freshness are the mechanism. The objective is content an answer engine can examine and a buyer can verify.

Finally, rerun the test on a schedule. Test category prompts, comparison prompts, problem prompts, and local prompts where relevant. Review citations across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews rather than declaring success from one result. When a source stops appearing, investigate the current page, its crawler access, the query, competing citations, and answer volatility before changing strategy.

1. Audit your own robots.txt and named crawler rules.
2. Track earned-media URLs alongside current crawler policy checks.
3. Publish direct, sourced answers on owned pages.
4. Measure citations by engine, prompt, date, and competitor set.
5. Recheck results because policies and answer outputs can change.

Should you block AI crawlers on your own site?

Whether to block an AI crawler is a business and governance decision, not a default growth tactic. The right answer depends on what the named crawler does, which products matter to your audience, your content rights policy, server capacity, and the degree of access you want to grant.

OpenAI's documentation provides a useful example of why broad decisions fail. It says OAI-SearchBot is used to surface websites in ChatGPT search features, while GPTBot is used to crawl content that may be used in training generative AI foundation models. A site that wants visibility in ChatGPT search but does not want its content used for foundation-model training can apply different rules to those agents.

Google likewise distinguishes crawlers from user-triggered fetchers. Its documentation says common crawlers respect robots.txt for automatic crawls. It also explains that a robots.txt file primarily manages crawler traffic and should not be used as the only way to hide a web page from Google search results.

Write down the objective before changing a rule. If the objective is citation visibility, measure the citation outcome after the change. If the objective is training control, verify the relevant agent and policy. If the objective is server protection, assess crawl load and use the least disruptive method that solves that problem. A single disallow line can affect more than the team expects.

1. Identify the specific user agent before changing access.
2. Separate search visibility from model-training preferences.
3. Document the business reason and the date of every policy change.
4. Test the effect on priority answer engines after a policy update.

Sources

Source: Five Blocks

Five Blocks reviewed robots.txt files and page tags for 44 major news outlets on October 23, 2026, then reported crawler blocks and citation outcomes across 14,392 answers. Five Blocks, 2026-11-05

Source: OpenAI

OpenAI documents separate roles for OAI-SearchBot, GPTBot, OAI-AdsBot, and ChatGPT-User, including the distinction between search and potential foundation-model training use. OpenAI, 2026-11-07

Source: Google Search Central

Google explains that robots.txt manages crawler access and traffic, and that it is not a reliable method for keeping a web page out of Google Search. Google Search Central, 2026-11-07

Source: Google for Developers

Google distinguishes automatic crawlers from user-triggered fetchers and says common crawlers respect robots.txt rules for automatic crawls. Google for Developers, 2026-11-07