Step-by-Step Guide to Optimizing for ChatGPT and Perplexity

What changed in AI engine optimization?

The change is that AI search access must be managed engine by engine, not through a single broad decision about AI crawlers. OpenAI documents `OAI-SearchBot` as the crawler used to surface sites in ChatGPT search features, while `GPTBot` is used for content that may be used to train OpenAI foundation models. Perplexity documents `PerplexityBot` as the crawler that surfaces and links websites in its search results, and says it is not used to crawl content for AI foundation models.

That distinction changes the first technical question for publishers. A site can block a training crawler while allowing a search crawler, or do the reverse. The first choice preserves a route to citations in answer engines. The second can remove that route before content quality, topical coverage, or authority have a chance to matter.

The practical before-and-after is simple. Before, a team might treat `GPTBot` as the control for ChatGPT visibility and make a single robots.txt decision. After, the decision must be split: `OAI-SearchBot` controls automatic search eligibility for ChatGPT, while `GPTBot` controls whether content may be used for foundation-model training. Perplexity has the same operational separation between `PerplexityBot` and its user-triggered fetcher.

This is not a claim that a robots.txt allowance earns a citation. It does not. It creates the condition for discovery and retrieval. Citation Engineering begins after access, with pages that answer a defined question clearly, show their evidence, and stay current enough to deserve reuse.

1. Check `yourdomain.com/robots.txt` for `OAI-SearchBot`, `GPTBot`, and `PerplexityBot`.
2. Confirm that a CDN, web application firewall, or bot-management layer is not blocking the same verified crawler after robots.txt permits it.
3. Record the current rule and the date of the check so later changes have an auditable baseline.

Who does this affect most?

This affects any team whose buyers ask ChatGPT or Perplexity for a recommendation, a comparison, an explanation, or a local provider. B2B SaaS teams are exposed when prospects ask questions such as 'best [category] tool' or compare named products. Service businesses face the same problem when people ask for a provider in a city or region.

The biggest risk sits with teams that made a privacy or scraping decision without separating automatic search retrieval from training use. They may have blocked `OAI-SearchBot` while trying to block `GPTBot`, or blocked `PerplexityBot` through a generic bot policy. The result is not a penalty. It is lost eligibility for the relevant automatic crawler to reach the page.

Technical ownership also matters. Marketing can publish the strongest guide in its category and still lose the opportunity if a platform rule blocks the crawler. Infrastructure teams can allow access and still produce little citation presence if the published material does not give an engine a direct, attributable answer. AI engine optimization is shared work, with different failure modes on each side.

Teams already investing in search content should not throw away their existing process. They should extend it. The useful unit is no longer only the ranking page. It is also the passage, table, definition, source link, and update date that can stand up inside an AI answer.

1. B2B SaaS growth teams tracking recommendations and comparison prompts.
2. Multi-location and service businesses tracking service-and-location questions.
3. Publishers that use restrictive bot policies, CDNs, or web application firewalls.
4. Content teams that have traffic reporting but no view of citations across answer engines.

How should you verify crawler access before changing content?

Verify access in this order: inspect robots.txt, inspect delivery-layer rules, then confirm the crawler identity. OpenAI recommends allowing `OAI-SearchBot` in robots.txt and allowing requests from its published IP ranges for sites that want to appear in ChatGPT Search. Perplexity similarly recommends allowing `PerplexityBot` and its published IP ranges.

Do not allow a user-agent header on its own when a security control can validate the vendor's published IP ranges as well. Headers can be impersonated. OpenAI and Perplexity publish IP-range data for their respective search crawlers, which gives an infrastructure team a second condition for an allow rule.

The change window needs patience. OpenAI says systems can take about 24 hours to adjust after a robots.txt update. Perplexity says changes may take up to 24 hours to reflect. That period is a systems update window, not evidence that a page will be cited within a day.

After access is confirmed, test representative pages rather than only the home page. Product pages, comparison pages, help content, regional pages, and newly published articles can pass through different templates or protection rules. A single successful fetch does not prove every high-value template is reachable.

1. Open the live robots.txt file and save the exact relevant directives.
2. Review web application firewall, CDN, and bot-management logs for verified crawler requests.
3. Allow the correct search crawler and its verified IP ranges where policy permits.
4. Recheck after the vendor's stated adjustment window.
5. Document which page templates were tested and which still need remediation.

What should a ChatGPT-focused page do differently?

A ChatGPT-focused page should make one important question easy to answer, then support that answer with evidence a reader can inspect. OpenAI states that sites opted out of `OAI-SearchBot` will not be shown in ChatGPT search answers, though they can still appear as navigational links. That makes eligibility a clear prerequisite, not a substitute for editorial quality.

Start with the answer near the top. State the scope, the conditions, and the conclusion in plain language. Follow it with the proof: dated first-party documentation, a transparent methodology, a comparison table when the question requires one, and links that let the reader inspect the source. Do not bury the actual answer beneath a brand story.

Build for a narrow set of commercially meaningful questions before expanding coverage. A credible implementation might begin with category questions, alternative questions, implementation questions, and objection questions. Each page should own one primary answer while linking naturally to nearby work. That creates coverage without publishing near-duplicates.

Keep the language precise. Separate what a vendor documents from what you observe in your own citation tracking. Do not call an optimization a guarantee. A good page gives ChatGPT a useful source to cite. It cannot compel the model to select that source in every answer.

1. Lead with a concise answer that matches the page question.
2. Use dated primary sources for claims about platforms, products, or regulations.
3. Add a comparison table when a buyer must distinguish options or approaches.
4. Show author, publication, and update context where editorially appropriate.
5. Track Citation Share on a stable prompt set instead of treating one answer as a verdict.

What should a Perplexity-focused page do differently?

A Perplexity-focused page should be easy to retrieve, easy to link, and broad enough to answer related questions without becoming vague. Perplexity says `PerplexityBot` is designed to surface and link sites in its search results and is not used for foundation-model training. That gives teams a direct technical control to check before they evaluate content performance.

Perplexity's own documentation also distinguishes `Perplexity-User`, which supports user actions and generally ignores robots.txt because a user requested the fetch. That behavior should not be mistaken for automatic search eligibility. A direct user-triggered visit and a search crawler are different routes with different controls.

The editorial response is to make source paths obvious. Give the reader a direct conclusion, label the limits of the conclusion, and cite the original material. Use descriptive headings that match the decisions people actually make. A strong page can answer the main question, define its terms, compare alternatives, and point to the next question without pretending every issue has a universal answer.

Breadth has a role here, but coverage should remain deliberate. Map adjacent questions that a prospect would ask before and after the main question. Then create distinct pages or sections when the answer, evidence, or audience changes materially. Repeating the same claim under slightly altered headings creates little new citation surface.

1. Allow `PerplexityBot` where the site's policy permits it.
2. Verify crawler requests with Perplexity's published IP data as well as the user agent.
3. Publish clear answers with primary-source links close to the relevant claim.
4. Cover adjacent questions only when they require distinct evidence or guidance.
5. Measure answers across repeatable prompts and revisit pages that lose relevance.

How do you turn access into citation-worthy content?

Turn access into citations by treating every important page as an answer asset. The page should resolve a question that exists in the market, make its conclusion visible, and show where its factual claims came from. That is more demanding than adding a generic section about AI to an existing article.

Use a short research brief before drafting. Define the exact question, the audience, the claim that can be supported, the primary sources, and the decision the reader should be able to make. If the evidence does not support a precise claim, narrow the claim rather than filling the gap with an attractive statistic.

Freshness is part of the work. Platform documentation and crawler behavior can change. Put a review date on the editorial calendar for pages that describe vendor controls, user agents, or citation behavior. When an official source changes, update the page, source list, and measurement notes together.

Finally, measure what the business needs. Citation Count per day shows volume. Answer Presence shows whether a brand appears across the relevant question universe. Citation Share shows the percentage of relevant AI answers in a category that cite the brand. Those measures answer different questions, so do not collapse them into a single success claim.

1. Select a defined prompt set tied to category demand or customer decisions.
2. Audit access before briefing writers or commissioning research.
3. Publish answer-first pages with real sources, dates, and useful comparisons.
4. Review source-dependent pages when vendors change technical guidance.
5. Track Citation Share, Citation Count per day, Answer Presence, and Share of Voice separately.

What is the step-by-step response plan?

The response plan is to fix access first, publish the best-supported answers next, then measure citation presence over time. This sequence avoids the common waste of improving content that the relevant search crawler cannot automatically reach.

Week one should be a technical baseline. Capture robots.txt, delivery-layer policies, verified bot logs, representative URL tests, and a prompt set for the questions that matter. The output is a record of access and an honest starting point for Citation Share and Answer Presence.

The next phase is editorial. Prioritize pages where the company can provide firsthand knowledge, clear product documentation, or well-sourced analysis. Add an answer-first summary, question-shaped headings, dated sources, and a comparison table when it helps someone decide. Then create internal paths between pages that answer the same buyer journey.

Measurement closes the loop. Re-run the same prompt set at a defined cadence, record citations and competitors, and inspect losses before reacting. A drop can come from access, freshness, weak evidence, changed prompts, or a competitor publishing a better answer. Diagnose the cause before changing the page.

1. Establish a crawler-access baseline for ChatGPT and Perplexity.
2. Fix only the search-crawler blocks that conflict with the site's policy.
3. Prioritize high-intent questions with evidence the business can publish responsibly.
4. Publish answer-first content with citation-grade sourcing.
5. Track citation outcomes on the same prompt set and investigate meaningful changes.

Sources

Source: OpenAI

OpenAI distinguishes OAI-SearchBot for ChatGPT Search from GPTBot for content that may be used to train foundation models, and says search adjustments can take about 24 hours after a robots.txt update. OpenAI, 2026-09-29

Source: Perplexity

Perplexity distinguishes PerplexityBot for search-result surfacing and linking from Perplexity-User for user-triggered fetches, and says crawler-setting changes may take up to 24 hours to reflect. Perplexity, 2026-09-29

Source: Google for Developers

Google explains the distinction between automatic crawlers and user-triggered fetchers, and states that common crawlers such as Googlebot respect robots.txt for automatic crawls. Google for Developers, 2026-09-29

Source: Pepper Content

The dated engine-by-engine analysis frames the technical change as separating search-crawler access from training-crawler controls across major answer engines. Pepper Content, 2026-09-10