X-robots tags
X-Robots-Tag is a type of meta directive used in the HTTP header to control how search engines crawl and index specific files on a website. Unlike regular meta tags, which are added to a webpage’s HTML, X-Robots-Tag is applied at the server level.
This tag is especially useful for controlling non-HTML files like PDFs, images, or videos. For example, you can use it to tell search engines not to index a PDF by adding the directive X-Robots-Tag: noindex to the file’s HTTP header.
Common directives for X-Robots-Tag include noindex (to prevent a file from being indexed), nofollow (to prevent links on the page from being followed), and noarchive (to stop search engines from storing a cached version).
X-Robots-Tag gives website owners more flexibility and control compared to standard meta tags. For instance, you can use it in server configurations like .htaccess files to apply rules to multiple files or file types at once.
It’s particularly useful for managing large sites or ensuring that sensitive or irrelevant content doesn’t appear in search results.
While robots.txt restricts access to specific sections of a website, it doesn’t prevent indexed content from appearing in SERP if it’s already linked elsewhere. In contrast, X-Robots-Tag can directly tell search engines not to index or display certain files, offering finer control.
Common directives include:
noindex (to block indexing),
nofollow (to prevent following links on the page),
noarchive (to stop caching), and
nosnippet (to block snippets in search results).
