X-robots tags

X Robots Tag

X-Robots-Tag is a type of meta directive used in the HTTP header to control how search engines crawl and index specific files on your website. Unlike regular meta tags, which are added to a webpage’s HTML, X-Robots-Tag is applied at the server level.

The X-Robots-Tag is a part of the HTTP response header and allows you to control how search engines index your pages. You use this tag in the HTTP header of a web page to control the page's indexing by search engines.

This tag is especially useful for controlling non-HTML files like PDFs, images, or videos. For example, you can use it to tell search engines not to index a PDF by adding the directive X-Robots-Tag: noindex to the file’s HTTP header.

Meta robots vs. X-robots

Unlike meta robots, the X-Robots-Tag can control indexing for different file types, not just HTML pages. You can use it to manage XML, PDF, and even images in the same way you manage HTML with the Robots-Tags.

In short, you apply X-Robots-Tags on non-HTML files. On HTML pages, you apply meta robots.

X-Robots-Tag gives website owners more flexibility and control compared to standard meta tags. For instance, you can use it in server configurations like .htaccess files to apply rules to multiple files or file types at once.

It’s particularly useful for managing large sites or ensuring that sensitive or irrelevant content doesn’t appear in search results.

While robots.txt restricts access to specific sections of a website, it doesn’t prevent indexed content from appearing in SERP if it’s already linked elsewhere. In contrast, X-Robots-Tag can directly tell search engines not to index or display certain files, offering finer control.

X-Robots-Tag directives

Some common directives used in X-Robots-Tags include:

  • all: Allows all search engine indexing and behavior (first must remove it).

  • none: Equivalent to noindex, nofollow, nosnippet, noarchive, and notranslate all together.

  • index: Index the page/file.

  • noindex: Don't index the page/file.

  • nofollow: Don't follow links on the page or file.

  • nosnippet: Prevents a text snippet from being shown in search results.

  • noarchive: Stops cached links from appearing in search results.

  • unavailable_after: A date after which the page should not be shown in search results.

How to check x-robots-tag?

To see if a page has an X-Robots-Tag set up, try these easy methods:

1. Use Chrome Developer tools

  1. Right-click on the webpage and pick “Inspect.”

  2. Go to the “Network” tab.

  3. Hit F5 to refresh the page. You'll see a list of files loading.

  4. Click on the top file (usually the name of the webpage), then look at the “Headers” section.

  5. In the headers, look for “X-Robots-Tag.”

2. Use cURL on the command line

Open your Command Prompt or Terminal and enter:

Change the URL to your file link. Look for a line in the result that says “X-Robots-Tag.”

3. Use online tools

There are online tools that check X-Robots-Tags. Just search “X-Robots-Tag checker” and use a trusted site. Enter your URL to see the tag information.

4. Check your server files

If you can access your server, check the .htaccess (for Apache) or nginx.conf (for Nginx) files for X-Robots-Tag rules. But be careful—changing these files can affect your site.

5. Use SEO tools for big sites

SEO tools like Screaming Frog can check X-Robots-Tags across many pages. Run a site crawl to see tag data for all your pages.