How to Optimize Your Site for AI Crawlers: A Guide to Semrush's New 'AI Search Health' Feature

What changed in Semrush AI Search Health?

Semrush Site Audit now includes an AI Search Health category that evaluates whether a site is configured for AI crawler access. The July 18, 2026 Semrush Site Audit walkthrough by Marco Diversi describes checks for pages blocked from AI user agents and for the presence and structure of llms.txt.

This changes the audit conversation from a broad question about AI visibility to a testable technical question: can the relevant crawler reach the page? A normal organic crawl can look healthy while an AI crawler is blocked by a rule that targets a specific user agent or by a wider robots.txt rule.

The has belongs in Site Audit rather than in a citation tracker. It can identify access conditions that may prevent crawling. It cannot prove that a model will cite a page, recommend a brand, or send traffic. Citation outcomes still depend on the quality, coverage, freshness, and usefulness of the material a system can retrieve.

The practical value is speed. Teams can surface crawler blocks alongside redirect errors, broken links, indexability issues, and other technical findings. That makes a previously easy-to-miss configuration decision visible in an existing technical workflow.

1. Check the AI Search Health category after each major robots.txt, CDN, firewall, or hosting change.
2. Treat a blocked crawler as a configuration finding, not proof of lost citations.
3. Confirm the affected URL set before editing a sitewide rule.
4. Re-run the audit after the change and retain the result with the deployment record.

What did crawler access look like before and after the change?

Before AI Search Health, a team could review robots.txt manually, search server logs, or maintain its own crawler checklist. Those methods remain useful, especially for verifying request behavior, but they require someone to know which user agents matter and to inspect every relevant rule.

After the change, Semrush surfaces AI crawler access as a dedicated audit category. The dated shift is operational rather than a change to crawler standards: the same robots.txt file can now be reviewed through a named AI Search Health report during a Site Audit.

That distinction matters. Semrush has added a diagnostic layer. It has not created a universal permission system for AI search, and it has not made llms.txt a substitute for crawl access, source quality, or page structure. Use the report to find conditions worth checking, then validate each condition against the crawler owner's documentation.

1. Before July 18, 2026: crawler access required manual robots.txt review or separate monitoring.
2. After July 18, 2026: Semrush Site Audit includes an AI Search Health category for AI crawler and llms.txt checks.
3. What to do now: assign one owner to review the report after every crawl and route intended blocks separately from accidental blocks.

Which sites does AI Search Health affect most?

AI Search Health affects any site that wants to make an intentional decision about access by AI crawlers. That includes B2B software companies publishing category and comparison pages, local businesses seeking discovery for service questions, publishers, marketplaces, and documentation-heavy sites.

It is especially relevant where robots.txt is maintained by a CDN, a security product, a hosting template, or a deployment pipeline. A team may correctly allow Googlebot while unintentionally denying a search-specific AI crawler through a catch-all rule. The error is often procedural: nobody has assigned ownership of the rule or tested the effect of a security change.

Sites that intentionally restrict training crawlers also need the report. A company may choose to deny training access while still allowing search crawlers. OpenAI documents this separation directly: OAI-SearchBot supports ChatGPT search features, while GPTBot concerns potential use in training generative AI foundation models.

Anthropic makes a similar distinction. Its April 7, 2026 documentation describes ClaudeBot for potential training data, Claude-User for user-initiated retrieval, and Claude-SearchBot for search-result quality. A single blanket rule can collapse those decisions into one without anyone meaning to do so.

1. B2B SaaS sites with comparison, use-case, and documentation pages.
2. Local and multi-location businesses whose service pages answer location-specific questions.
3. Publishers that want search discovery but have separate training-access policies.
4. Large sites where robots.txt is generated by infrastructure rather than edited by the content team.

Which AI crawlers should you review separately?

Review crawler purpose before you allow or deny it. User-agent names are not interchangeable, and the impact of a block depends on what the crawler does. OpenAI says a site opted out of OAI-SearchBot will not appear in ChatGPT search answers, although it may still appear as a navigational link.

OAI-SearchBot is the OpenAI agent to review for ChatGPT search visibility. GPTBot addresses potential model training. ChatGPT-User handles certain user-directed visits and is not the automatic crawler used to determine whether content may appear in Search, according to OpenAI.

Anthropic documents Claude-SearchBot as its search crawler, Claude-User as the agent used for user-initiated visits, and ClaudeBot as a model-development crawler. Anthropic says disabling Claude-SearchBot may reduce a site's visibility and accuracy in user search results.

Google also documents that common crawlers respect robots.txt rules for automatic crawls. Its crawler documentation provides the foundation for a cautious operating rule: do not assume that a generic allow or deny statement has the same effect for every engine. Read the current documentation for each crawler you are deciding about.

1. Search crawler: review when the goal is appearance in an AI search experience.
2. Training crawler: review when the goal is controlling potential model-training access.
3. User-triggered fetcher: review when the goal is controlling retrieval requested by an end user.
4. Generic wildcard rule: review because it can apply more widely than the author intended.

How should you respond to an AI Search Health warning?

Respond by classifying the warning before changing a file. First identify the user agent, the URLs affected, the exact robots.txt directive, and the business reason for the current rule. A warning may be correct, accidental, or too broad to interpret without a second check.

Then decide whether the crawler's purpose matches the site's access policy. If the business wants visibility in ChatGPT search, inspect access for OAI-SearchBot. If it wants visibility in Claude search, inspect Claude-SearchBot. If it does not want content used for model training, that is a separate decision for GPTBot or ClaudeBot.

Make the smallest safe change. Do not replace a precise deny rule with a broad allow rule because an audit score looks better. Keep protected areas protected, preserve crawl controls required for server stability, and ask the security or legal owner to approve a policy change where relevant.

Finally, validate the deployment. Re-fetch robots.txt, verify that the syntax matches the intended agent and path, re-run the Semrush audit, and monitor server logs for actual requests where the site has that capability. The audit report is the start of verification, not the final proof.

1. Identify the agent, blocked URLs, directive, and owner of the rule.
2. Confirm whether the block is intended for search, training, user retrieval, or server protection.
3. Change only the relevant rule after the policy owner approves it.
4. Re-test robots.txt, re-run Site Audit, and check logs when available.

Does llms.txt solve AI crawler access?

No. llms.txt does not override robots.txt, firewall rules, HTTP errors, authentication, or crawl restrictions. Semrush can audit its presence and structure, but that check should sit behind crawler access and page availability in the priority order.

The useful question is whether the site provides a clear, maintained source for machine-readable orientation after public pages are accessible. An outdated llms.txt file can create confusion. A missing file does not by itself establish that a page cannot be crawled or cited.

Keep the effort proportional. First make sure the pages you want discovered return successfully, are internally linked, and are not unintentionally blocked. Then review whether an llms.txt file reflects the current structure of your public content. Do not treat the file as a citation guarantee.

This is where content and technical work meet. A crawler needs access, but access alone does not make a page quote-worthy. Pages still need direct answers, dated evidence, clear definitions, and enough coverage for the question universe the business wants to address.

1. Do not use llms.txt as a replacement for robots.txt review.
2. Keep any llms.txt file aligned with live public URLs.
3. Prioritize successful page retrieval and clear internal linking.
4. Publish direct, sourced answers after access is confirmed.

How should teams measure the impact after fixing access?

Measure the technical fix and the visibility outcome as separate things. The technical result is whether the intended crawler is no longer blocked and whether the audit warning clears. The visibility result is whether the brand appears or is cited for a defined prompt set over time.

For Omnicite's measurement vocabulary, Citation Share is the percentage of relevant AI answers in a category that cite you. Citation Count per day measures volume. Answer Presence measures breadth across the question universe, while Share of Voice compares a brand with named competitors. None of those metrics should be inferred from a robots.txt edit alone.

Set a baseline before changing access. Record the affected pages, the original directive, the date of the change, and the current Citation Share or Answer Presence for the relevant prompts. Review the same prompt set after a consistent interval, while noting other content releases and site changes that could affect the result.

The strongest operating habit is modest: keep crawler policy deliberate, keep technical evidence attached to each change, and keep citation reporting honest about what was observed. A cleared audit warning proves a condition changed. It does not prove a model selected the page.

1. Record the original robots.txt directive and affected URL set.
2. Save the before-and-after audit result with the deployment date.
3. Track Citation Share and Answer Presence against the same prompt set.
4. Report crawler access as a technical outcome and citations as a separate observed outcome.

Sources

Source: Marco Diversi

Semrush Site Audit includes AI-search checks for blocked AI user agents and llms.txt. Marco Diversi, 2026-07-18

Source: OpenAI Developers

OAI-SearchBot is used for ChatGPT search, while GPTBot addresses potential model-training use. OpenAI Developers, 2026-10-26

Source: Claude Help Center

Anthropic distinguishes ClaudeBot, Claude-User, and Claude-SearchBot, including their separate purposes. Claude Help Center, 2026-04-07

Source: Google for Developers

Google documents that its common crawlers respect robots.txt rules for automatic crawls. Google for Developers, 2026-10-26