[The Engines]
How Can You Optimize for AI Engines Like ChatGPT and Perplexity?
ChatGPT and Perplexity quietly split their crawlers into a citation bot and a training bot, and most sites are still blocking the wrong one.
Explore this article with AI
Open a source-aware analysis with this article as the primary source.The short answer
ChatGPT and Perplexity each run two separate crawlers now: one that retrieves pages for live citations and one that only feeds model training. Block the training bot and you lose nothing. Block the citation bot, whether on purpose or by accident, and you disappear from both engines' answers even if you rank fine on Google.
What changed in how ChatGPT and Perplexity crawl the web?
ChatGPT and Perplexity now treat search retrieval and model training as two separate jobs, run by two separate crawlers, and most sites still write robots.txt rules as if one bot does both. A robots.txt audit from Pepper, verified September 10, 2026, calls this mismatch the single most common technical fault behind a site vanishing from AI answers while its rankings on Google stay untouched.
OAI-SearchBot is the crawler that lets ChatGPT retrieve and cite a page during a live search answer. GPTBot is a different crawler entirely, and it only feeds OpenAI's training data. A site can block GPTBot without losing a single citation, and plenty of sites do exactly that on purpose. The trouble starts when a blanket rule written to catch 'anything with GPT in the name' catches OAI-SearchBot too, which quietly removes the site from ChatGPT's citation pool.
Perplexity works differently again. PerplexityBot is the crawler that fetches pages to answer queries, and there is no separate training-only counterpart to route around it. There is also Perplexity-User, an agent triggered by an actual person asking a question inside the app rather than a scheduled crawl, and it commonly ignores robots.txt on those live, user-triggered fetches. Google, for its part, still uses one ranking system, Googlebot, for both classic search and AI Overviews, with Google-Extended sitting alongside it as the training-only opt-out.
Who does this affect?
It affects any site whose robots.txt was written defensively rather than deliberately. Plenty of marketing and legal teams added a blanket disallow rule for anything resembling an AI crawler once GPTBot and its peers showed up in server logs, without knowing that OpenAI and Perplexity had split search access from training access. That single rule, meant to opt out of training, can also opt the site out of being cited.
The cost is not evenly spread. ChatGPT cites fewer sources per answer than Perplexity does, and each citation carries more weight as a result, so a blocked domain loses proportionally more of what Omnicite calls Citation Share, the percentage of relevant AI answers in a category that cite a given site. Perplexity spreads citations across more sources with less weight per link, which softens the damage from a single blocked domain but does not erase it.
The damage concentrates on the two groups fighting hardest for AI visibility right now: B2B SaaS teams competing to be the answer for 'best [category] tool' prompts, and local or multi-location service businesses trying to show up when someone asks an engine for the best provider in a city. Both groups depend on being retrievable at the exact moment a prompt fires, and a misconfigured robots.txt removes that option before content quality ever gets a chance to matter.
| Engine | Citation crawler (allow) | Training-only crawler (safe to block) | Common before-state mistake |
|---|---|---|---|
| ChatGPT | OAI-SearchBot | GPTBot | A wildcard rule blocking anything with 'GPT' in the name blocks both bots at once |
| Perplexity | PerplexityBot | None dedicated; Perplexity-User (live, user-triggered) often ignores robots.txt entirely | Treating Perplexity as one bot and disallowing it outright |
| Googlebot (shared by classic search and AI Overviews) | Google-Extended | Blocking Google-Extended and assuming it also removes the site from AI Overviews |
How should you respond right now?
Start with the file that takes ten minutes to check and costs nothing to fix. Then move through rendering, page structure, and the media strategy that keeps you eligible for citation once the crawlers can actually reach you.
- Open robots.txt and search for every crawler token by name (OAI-SearchBot, GPTBot, PerplexityBot, Googlebot, Google-Extended). Allow the search and citation bots even where the training bots stay blocked.
- Confirm your key pages render server-side. If a page's core content only appears after JavaScript executes, a crawler that does not run a full browser sees a blank shell and has nothing to cite.
- Restructure your highest-value pages so a single passage answers a single question cleanly. A dense paragraph that buries the answer under three qualifiers is much harder for a model to lift and quote than a direct, self-contained sentence.
- Invest in earned coverage, not paid placement. Third-party mentions carry far more weight with these engines than anything published on your own domain, so a citation strategy that stops at your blog is an incomplete one.
What does the before-and-after actually look like?
The clearest way to see the change is side by side: what a typical robots.txt rule blocked before the split was widely understood, against what it should allow now that the citation and training bots are known to be distinct.
The pattern repeats across all three engines. The fix is not to open the site to every crawler. It is to know which crawler does which job, and to write the rule for the job you actually want to opt out of.
What does the earned-media data mean for AI optimization?
Fixing robots.txt gets a crawler through the door. It does not make the model want to cite you once it is inside. Muck Rack's Generative Pulse study, published in May 2026 and built on more than 25 million links cited by ChatGPT, Gemini and Claude, found that earned media accounts for 84 percent of AI citations, journalism alone accounts for 27 percent, and paid or advertorial content accounts for just 0.3 percent.
That split has held steady since Muck Rack's first edition of the study in mid-2025, ranging between 82 and 89 percent across every update. Read next to the robots.txt problem, the two findings describe one system: access controls the floor, and third-party coverage controls the ceiling. A site with a clean robots.txt and no earned coverage is retrievable but rarely worth quoting. A site with strong earned coverage and a blocked citation bot is unreachable no matter how often other outlets mention it.
This is the layer Omnicite's Citation Engineering approach is built to track: not just whether a crawler can reach a page, but whether the resulting content, at the volume and freshness an in-house team struggles to sustain, actually earns a place in the answer. Rankings got a page found. Citations get it chosen, and that choice runs through both the crawler rule and the coverage behind it.
Key takeaways
- ChatGPT and Perplexity now run separate crawlers for live citation retrieval and for model training, and the two should not share a robots.txt rule.
- OAI-SearchBot handles ChatGPT citations, GPTBot handles OpenAI training, and blocking the second does not touch the first.
- PerplexityBot fetches pages for live answers, and Perplexity-User, triggered by an actual person's query, often ignores robots.txt on those requests.
- A blanket disallow rule written defensively against 'AI bots' is the most common way a site loses citation eligibility without anyone noticing.
- Earned media drives 84 percent of AI citations versus 0.3 percent for paid content, so fixing crawler access is necessary but not sufficient on its own.
- Check robots.txt first, confirm server-side rendering second, then restructure pages so a single passage answers a single question cleanly.
Omnicite Editorial. "Optimize for ChatGPT and Perplexity: What Changed" The Citation Report, Omnicite. https://omnicite.co/blog/how-can-you-optimize-for-ai-engines-like-chatgpt/
Sources
Source: Pepper
AI engines run separate crawlers for search citation and model training, verified crawler tokens as of September 10, 2026 Pepper, 2026-09-10
Source: Muck Rack
Earned media accounts for 84 percent of AI citations across ChatGPT, Gemini and Claude, versus 0.3 percent for paid content Muck Rack, 2026-05-07
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot only feeds training data for OpenAI's models and has no role in live answers. OAI-SearchBot is the crawler ChatGPT uses to retrieve and cite pages during an actual search answer. Blocking GPTBot does not affect citation eligibility. Blocking OAI-SearchBot does.
Does blocking PerplexityBot stop Perplexity from citing my site?
Mostly, yes, for scheduled crawling. But Perplexity-User, the agent triggered when a real person asks a question inside the app, frequently fetches pages live and can ignore robots.txt on that request, since a human, not a bot, initiated it.
How do I check which crawlers my site currently blocks?
Open your robots.txt file directly in a browser and search for the exact tokens: OAI-SearchBot, GPTBot, PerplexityBot, Googlebot and Google-Extended. Any wildcard or partial-match rule (like disallowing anything containing 'GPT') needs a second look, since it may be catching a bot you never meant to block.
Does fixing robots.txt guarantee my content gets cited?
No. It only restores the possibility of being retrieved. Muck Rack's Generative Pulse study found earned media drives 84 percent of AI citations, so third-party coverage still does most of the work of earning the citation itself, once a crawler can reach the page at all.
Why does Google use the same crawler for search and AI Overviews?
Google runs one ranking system, Googlebot, across both classic search results and AI Overviews, while Google-Extended is the separate opt-out for training data only. A site cannot appear in AI Overviews while blocking Googlebot itself.
What should a marketing team do this week?
Audit robots.txt for the exact crawler tokens above, confirm key pages render without requiring JavaScript execution, and restructure the highest-traffic pages so each core question gets one direct, liftable answer rather than a buried one.