TL;DR
AI search decomposes queries into intent and entities, retrieves 10-30 candidate sources, synthesizes an answer, then cites sources that provided the clearest answers. Their crawlers cannot execute JavaScript.
Google trained you to think in links. AI thinks in answers.
For 25 years, search returned ranked link lists. AI search returns synthesized answers with citations. The question is no longer "Where do I rank?" — it's "Am I being cited?"
The anatomy of an AI search response
When a user queries Perplexity, four steps happen: query understanding (intent + entity decomposition), source retrieval (10-30 candidates from web index), answer synthesis (combining insights from multiple sources), and citation selection (attributing to the clearest, most authoritative sources).
The crawlers: GPTBot, ClaudeBot, PerplexityBot
All AI crawlers share one critical limitation: they do not execute JavaScript. When GPTBot visits your page, it reads raw HTML only. Client-side rendered content is invisible.
Check your robots.txt — many CDN providers block AI crawlers by default.
1# Allow AI crawlers
2User-agent: GPTBot
3Allow: /
4
5User-agent: ClaudeBot
6Allow: /The shift from keywords to entities
Traditional SEO optimizes around keywords (text strings). AI search optimizes around entities (real-world concepts). The AI builds an entity graph of your brand — what you do, what you build, who you serve. Richer and more consistent entity data means more authoritative citations.