How to Optimize for AI Crawlers: Training vs. Answering

What changed for AI crawlers on September 15, 2026?

Cloudflare retired its single 'Block AI Bots' toggle and replaced it with three independent controls: one for search crawlers, one for AI training crawlers, and one for AI agents acting on a user's behalf. The change is live now, not proposed.

Ad-supported sites get the stricter default: search crawlers stay allowed, while training and agent bots are blocked unless the site owner turns them back on. Sites without ads keep everything allowed until the owner changes it. Mixed-use crawlers such as Googlebot, Applebot and Bingbot get evaluated under both policies at once, so a training block can catch a crawler a site owner assumed was safe for search.

The timing lines up with a second shift. Optimization Theory's report, published the same week, found the gap between ranking and citation widening: only 17 to 38% of AI Overview citations now come from a page ranking in the organic top 10, down from roughly 76% in mid-2024. Two changes landing together turn bot-level access control into a ranking-adjacent decision, not a security afterthought handled once and forgotten.

Who does the training-versus-answering split affect?

Everyone on Cloudflare's free plan, everyone launching a new site there, and everyone who never touched their AI bot settings gets the new defaults automatically. That covers a wide slice of the web without a single click from the site owner.

For B2B SaaS and tech growth teams, the stakes are direct: a site that never explicitly allowed OAI-SearchBot or PerplexityBot could now sit outside the exact 'best [category] tool' answers that Citation Share is built to measure. For local and service businesses, the same accident quietly removes them from '[service] in [city]' answers, and it never shows up as a lost keyword ranking, only as a citation that stops appearing.

Publishers and ad-supported marketplaces face the sharpest edge, since ad-supported pages inherit the stricter default. Sites without ads keep the old open posture, which is its own risk if keeping training crawlers out was ever the goal in the first place.

How should you respond to the change?

Audit robots.txt bot by bot instead of trusting a blanket rule, then check the defaults Cloudflare just applied actually match what the site wants.

1. Pull the current robots.txt and check whether an old wildcard rule blocks OAI-SearchBot, Claude-SearchBot or PerplexityBot along with the training bots it was written for years ago.
2. Confirm ad-supported pages are not accidentally blocking ChatGPT-User or Perplexity-User if the goal is to have pages fetched live when someone asks a direct question.
3. Check that key pages render content server-side, since several AI crawlers execute little to no JavaScript, and content that only appears after client-side hydration is effectively invisible to them.
4. Refresh cornerstone pages on a real cadence: pages updated within 30 days earn roughly 3.2 times more citations, per Optimization Theory's 2026 report.
5. Track citation share by engine rather than organic rank alone, since the two now move independently of each other.

What is the actual difference between a training crawler and an answering crawler?

A training crawler collects content to build or refine a model. An answering crawler, sometimes called a search or retrieval crawler, fetches content specifically to generate or index a response shown to someone right now. Blocking the first costs a site nothing in citations. Blocking the second can remove that site from an answer entirely.

OpenAI runs three separate bots for exactly this reason: GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for fetching a page live when a person asks about it directly. Anthropic and Perplexity follow a similar pattern, most visibly ClaudeBot for training against Claude-SearchBot for retrieval, with PerplexityBot crawling purely to answer.

What happens when a site blocks the wrong bot?

The most common mistake is inherited, not chosen. A wildcard disallow rule written years ago to stop scraping now silently catches bots that did not exist when the rule was written. Optimization Theory's report puts training-focused crawler traffic at 51.8% of the total, mixed-purpose at 35.7%, and search-only at 9.3%, so a rule that blocks anything with 'bot' in the name is overwhelmingly likely to hit a training crawler, but it still catches enough search-only traffic to matter.

The cost of that mistake is rising. The share of AI bot requests met with a 403 Forbidden response climbed to 9.64% in July 2026, up from 5.67% a year earlier, according to TechnologyChecker.io's analysis of Cloudflare Radar traffic against more than 4,000 robots.txt files. Crawl-to-referral ratios tell the rest of the story: GPTBot returns one referral for every 251 pages it crawls, PerplexityBot for every 289, and ClaudeBot for every 1,917. A single misapplied disallow rule affects those ratios directly, and it never announces itself as a lost ranking.

Is this good news or bad news for citation share?

Neither, exactly. The split makes it possible to keep training crawlers out without losing citations for the first time, which is a genuine improvement over the all-or-nothing switch it replaced. It also raises the cost of getting the configuration wrong, because the defaults now do the blocking automatically instead of waiting for a site owner to flip a switch.

There is no page two in an AI answer, and there is no partial credit for a page an answering bot was never allowed to fetch. For a team already tracking citation share across ChatGPT, Perplexity, Gemini and AI Overviews, the fix is administrative: match the allow list to intent, not to sentiment about AI training. For everyone else, September 15 is a reasonable day to find out what robots.txt has actually been doing since it was last edited.

Sources

Source: Cloudflare

Cloudflare replaced its single AI bot toggle with separate controls for search, training and AI agents, blocking training and agent bots by default on ad-supported sites Cloudflare, 2026-09-15

Source: TechnologyChecker.io

The share of AI bot requests met with a 403 Forbidden response rose from 5.67% in July 2025 to 9.64% in July 2026, with GPTBot, PerplexityBot and ClaudeBot showing crawl-to-referral ratios of 251:1, 289:1 and 1,917:1 TechnologyChecker.io, 2026-09-03

Source: Optimization Theory

Only 17 to 38% of AI Overview citations now come from a top-10 organic result, down from roughly 76% in mid-2024, and pages refreshed within 30 days earn about 3.2 times more citations Optimization Theory, 2026-09-15