schema
X-Robots-Tag
HTTP header equivalent of <meta name=robots> — controls crawler behavior on non-HTML files (PDFs, images).
Definition
X-Robots-Tag is an HTTP response header that controls crawler indexing rules for any URL — including non-HTML files like PDFs, images, and JSON APIs where meta tags aren't available. Same directives as <meta name='robots'>: noindex, nofollow, noarchive, nosnippet, max-snippet, max-image-preview.
Example
Apache: Header set X-Robots-Tag 'noindex' for *.pdf. Now PDF files at /docs/*.pdf are crawlable but not indexed.How to optimize
Use X-Robots-Tag for PDFs, JSON files, generated reports. Combine with robots.txt to control crawl behavior on file types. Test via curl -I.