engines
AI agent crawler
Crawlers used by AI coding agents (Cursor, Claude Code, Copilot) to fetch documentation on demand.
Definition
AI agent crawlers are user-agents that IDE coding agents (Cursor, Claude Code, GitHub Copilot, Windsurf) use when fetching documentation or context from the web during a coding session. Distinct from search retrieval bots — these are on-demand fetches triggered by a developer prompt. They preferentially fetch llms.txt + structured docs.
Example
A developer asks Cursor to 'use the Stripe Payments API'. Cursor's agent fetches stripe.com/llms.txt + relevant docs URLs, includes them in the model context, writes code grounded in current API.How to optimize
Publish llms.txt at site root (use VectorCite's generator at /audit/llms-txt). Mark docs URLs with clean canonical structure. Keep docs versioned and dated.