Getting Cited by ChatGPT, Claude and Perplexity: The Setup We Actually Use

A growing share of the people who could hire our clients never see a results page. They ask an assistant, they get an answer with two or three citations, and they click one of those.

Getting into that set of citations is a different problem from ranking, and it is mostly not a content problem. It is a plumbing problem. Before an assistant can cite you it has to be allowed to fetch your page, able to parse it, and confident about which URL represents it.

Here is what we set up on client sites, and which parts of it we think are actually load bearing.

Step one: stop blocking the crawlers

The most common reason a site is invisible to AI assistants is that somebody blocked them, usually years ago, usually on advice that treated AI crawlers as a threat.

That framing made some sense when the only crawlers were training crawlers. It makes much less sense now, because the retrieval crawlers and the training crawlers are different user agents, and blocking the retrieval ones removes you from the citation set without protecting anything.

The distinction matters, so here it is by vendor.

OpenAI documents three. GPTBot crawls content that may be used to train models. OAI-SearchBot indexes content so it can surface in ChatGPT search. ChatGPT-User fetches a page when a user's question sends the assistant to it. OpenAI is explicit that these controls are independent, so you can allow search indexing while disallowing training if that is your position.

Anthropic documents a similar split. ClaudeBot is the training crawler. Claude-SearchBot discovers and refreshes content for Claude's search. Claude-User fetches pages in response to a user's request.

Perplexity documents PerplexityBot, which surfaces and links websites in Perplexity's search results and is explicitly not used to crawl content for foundation models, and Perplexity-User, which visits a page because a user asked something.

Google runs Google-Extended separately from Googlebot. It controls whether your content may be used for training Gemini models and for grounding in Gemini apps and Vertex AI. Google states plainly that Google-Extended does not affect your inclusion in Google Search and is not a ranking signal, which is worth knowing before anyone argues that allowing it is an SEO risk. It is not. It is a separate decision.

The decision we recommend to clients, and the one we run by default: allow the search and retrieval agents. The training agents are a genuine business decision and reasonable people take either side. But blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from citations while doing nothing about training, which is the worst of both outcomes.

A minimal allow block looks like this, sitting alongside your normal rules:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Two practical notes from doing this a lot. Robots.txt matching is by longest matching user agent group, so a specific group for an agent overrides a general one, and a permissive User-agent: * block does not reliably cover an agent you also named elsewhere. And the file is frequently regenerated by your build tooling, so hand-editing it on the server is wasted work. Fix it where it is generated. We have found clients' AI crawler rules quietly reverted by a deploy more than once, which is why checking robots.txt after deployment is on our release checklist rather than in somebody's memory.

Step two: the llms.txt question, answered honestly

llms.txt is a proposed convention, introduced in 2024 by Jeremy Howard of Answer.AI. The idea is a markdown file at /llms.txt giving a language model a curated map of your most useful content, with an optional /llms-full.txt containing the content itself. The reasoning is sound: HTML pages are full of navigation, scripts and chrome, and a clean map removes ambiguity.

Here is the honest status. It is a proposal, not a standard, and the major AI vendors have not committed to consuming it. Adoption among documentation platforms is real and growing. Adoption by the assistants themselves is not something anyone can currently demonstrate.

So our position: publish it if it is cheap, do not build a strategy on it. For a documentation site or a site with a clear content hierarchy, generating an llms.txt is a small amount of work with a plausible future payoff. For a twelve page local business site it is close to pointless. Anybody selling llms.txt as the thing that will get you into AI answers is selling something they cannot evidence.

The effort is better spent on the next two sections, which have measurable effects today.

Step three: make the page machine-readable

Retrieval systems are extracting facts, not admiring your layout. Three things help materially.

Structured data. Schema.org markup in JSON-LD gives an assistant unambiguous, typed facts instead of inferred ones. For a local business that means LocalBusiness with real address, service area, hours and phone. For an article, Article with author, publication date and a meaningful headline. For a service page, Service. For a page that answers questions, FAQPage where the questions are ones people actually ask rather than ones you invented to fill a schema block.

Semantic HTML with honest headings. An H2 that names the question the section answers is a retrieval target. An H2 that says "Our Approach" is not. This is the cheapest win available and most sites do not take it.

Server-rendered content. If the substance of the page only appears after JavaScript runs, some fetchers will see an empty shell. Assistants fetching a page in response to a user question are generally not running a full browser. Anything that matters should be in the initial HTML response.

These are the same fundamentals that carry conventional organic rankings in a niche, which is why we treat this work as an extension of the programme in SEO for mobile home investors rather than as a separate channel.

Step four: canonical hygiene, which matters more than people think

This is the part that gets skipped, and in our experience it is the part that most often breaks retrieval quietly.

A retrieval system has to decide which URL represents a document. If your site gives it several answers, the signal splits and the system may pick one that is not the one you promote, or may treat the variants as separate lower-confidence documents.

Three failures we have actually found and fixed on live sites:

Two canonical tags on one page. When a page carries two canonicals, Google discards both and the page ends up with no canonical signal at all, which is worse than one imperfect tag. This happens easily when a template injects one and a plugin or build tool injects another.

A canonical pointing at a URL that does not exist. We found a set of posts each naming a canonical target that had never resolved. The pages were live and had real content, and they were telling every crawler "do not index me, index this other page," where the other page was a 404.

A canonical pointing at a URL that redirects. On a site configured to force trailing slashes, several posts named the non-slash form, so every canonical resolved through a 308. A canonical that only resolves after a redirect hop is a weak and ambiguous signal.

The check is boring and takes minutes. Fetch every canonical URL and confirm it returns 200 directly, with no redirect hop. Then confirm your sitemap lists the same final URLs the canonicals name, because a sitemap entry is itself a claim about which URL is canonical, and the two disagreeing sends two different answers about the same document.

What this does not fix

All of the above makes you retrievable. None of it makes you worth citing.

An assistant picks sources that contain a specific answer to a specific question. Pages built from your own operational knowledge, real constraints, real process, real numbers you measured, get selected because they contain something the model cannot produce on its own. Pages assembled from the same summarised material as everyone else do not, because there is no reason to pick yours.

This is also why retrieval belongs inside the wider system rather than bolted on at the end. A citation is a first touch, and it is worth nothing if nothing catches it, which is the case we make in the complete guide to a lead generation system.

That is the same property that determines whether a page survives a search spam update, which we wrote about in Google's August 2026 spam update. Two different systems, one underlying requirement: publish something only you know.

All of this makes you retrievable. None of it makes you worth citing. An assistant picks sources that contain a specific answer it cannot produce on its own.

Caveats

Crawler names, behaviours and vendor policies change. Anthropic, OpenAI and Perplexity have all revised their crawler documentation within the last year, and new agents appear. Check the vendor documentation before you finalise a robots.txt rather than copying a block from a blog post, including this one.

We are describing a setup we run on client sites. We are not claiming a specific outcome from it, because citation behaviour is not something any publisher controls and none of these vendors expose the mechanism.

Sources: OpenAI, Overview of OpenAI crawlers; Anthropic, Does Anthropic crawl data from the web; Perplexity, PerplexityBot and Perplexity-User; Google, common crawlers including Google-Extended; The /llms.txt proposal; Answer.AI, the original llms.txt post

Keep reading

• Google's August 2026 spam update and what it means for agency SEO
• SEO for mobile home investors
• The complete guide to a lead generation system