Crawlers fetch pages so their owners can index, train on, or answer questions from the content. Alongside Googlebot, there is now a fleet of AI crawlers — GPTBot (OpenAI), PerplexityBot, ClaudeBot (Anthropic), Google-Extended, and CCBot — each feeding a different AI system.
How easily a crawler can read your site matters: heavy JavaScript, slow responses, or blocked paths can leave parts of your content unseen, which means unseen by the AI built on that crawler.
When PerplexityBot visits your site, it reads the HTML it receives. If key content only appears after JavaScript runs, the bot may never see it — and Perplexity answers without you.
Why this matters for AI findability
Crawlers are the eyes of AI systems. Making your pages easy for them to fetch and parse is the entry ticket to agentic findability — if the bot reads you cleanly, the AI can represent you.