framework
robots.txt
The /robots.txt file telling crawlers what they can fetch — the 2026 retrieval-vs-training distinction matters most.
Definition
robots.txt is a plain-text file at the root of a domain that tells web crawlers which paths they can access. The critical 2026 distinction is between RETRIEVAL bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) which drive citation, and TRAINING bots (GPTBot, ClaudeBot, Google-Extended) which inform future models. Blocking retrieval bots kills your AEO; blocking training bots is a content-policy decision.
Example
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
(Allows ChatGPT citations; blocks OpenAI training.)How to optimize
Explicitly Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User. Decide separately whether to Allow GPTBot, ClaudeBot. VectorCite's ai-bot-access signal flags blocked retrieval bots.