robots.txt is the gatekeeper. It lists user-agents (including AI crawlers like GPTBot, PerplexityBot, Google-Extended, and ClaudeBot) and what each is allowed to fetch. Block the wrong bot and you make yourself invisible to that AI system entirely.
In an agentic world this file carries new weight: it is where you explicitly welcome — or accidentally exclude — the very agents you want to be found by. It controls access, while llms.txt adds meaning.
Adding "User-agent: GPTBot / Allow: /" signals that OpenAI's crawler is welcome; a careless "Disallow: /" for all bots would quietly cut you out of AI answers.
Why this matters for AI findability
If AI crawlers can't reach your pages, none of your other optimisation matters. A correct robots.txt is the precondition for agentic findability — the difference between being readable and being locked out.