HTTP Headers

X Robots Tag

HTTP Header

The X-Robots-Tag response header specifies how crawlers should index URLs.
Although not officially part of any formal specification, it has become a widely accepted standard method for communicating with search bots, web crawlers, and similar user agents.
Search engine crawlers utilize the directives from the X-Robots-Tag header to determine how web pages and other resources are displayed in search results.

Indexing rules set through <meta name="robots"> elements and X-Robots-Tag headers are identified when a URL is crawled.
Specifying indexing instructions via an HTTP header is especially useful for non-HTML documents like images, PDFs, or other media files.

Syntax

Multiple indexing rules can be combined as a comma-separated list to control how web pages are handled by robots or crawlers.

To specify rules for different user agents, you can include the <bot-name> before the rules, indicating which user agent the rules apply to.

For example, you might set rules like:

Basic indexing rule:
X-Robots-Tag: <indexing-rule>
Multiple rules:
X-Robots-Tag: <indexing-rule>, <indexing-ruleN>

To target specific user agents, you can specify the bot name along with rules:

Example:
X-Robots-Tag: <indexing-rule>, <bot-name>: <indexing-rule>
Multiple rules for a user agent:
X-Robots-Tag: <bot-name>: <indexing-rule>, <bot-name>: <indexing-ruleN>

Directives

Any of the following indexing rules may be used:

all

No restrictions for indexing or serving in search results. This rule is the default value and has no effect if explicitly listed.

noindex

Do not show this page, media, or resource in search results. If omitted, the page, media, or resource may be indexed and shown in search results.

nofollow

Do not follow the links on this page. If omitted, search engines may use the links on the page to discover those linked pages.

none

Equivalent to noindex, nofollow.

nosnippet

Do not show a text snippet or video preview in the search results for this page. A static image thumbnail (if available) may still be visible. If omitted, search engines may generate a text snippet and video preview based on information found on the page. To exclude certain sections of your content from appearing in search result snippets, use the data-nosnippet HTML attribute.

indexifembedded

A search engine is allowed to index the content of a page if it’s embedded in another page through iframes or similar HTML elements, in spite of a noindex rule. indexifembedded only has an effect if it’s accompanied by noindex.

max-snippet: <number>

Use a maximum of <number> characters as a textual snippet for this search result. Ignored if no valid <number> is specified.

max-image-preview: <setting>

The maximum size of an image preview for this page in search results. If omitted, search engines may show an image preview of the default size. If you don’t want search engines to use larger thumbnail images, specify a max-image-preview value of standard or none. Values include:

none

No image preview is to be shown.

standard

A default image preview may be shown.

large

A larger image preview, up to the width of the viewport, may be shown.

max-video-preview: <number>

Use a maximum of <number> seconds as a video snippet for videos on this page in search results. If omitted, search engines may show a video snippet in search results, and the search engine decides how long a preview may be. Ignored if no valid <number> is specified. Special values are as follows:

0

At most, a static image may be used, in accordance to the max-image-preview setting.

-1

No video length limit.

notranslate

Don’t offer translation of this page in search results. If omitted, search engines may translate the search result title and snippet into the language of the search query.

noimageindex

Do not index images on this page. If omitted, images on the page may be indexed and shown in search results.

unavailable_after: <date/time>

Requests not to show this page in search results after the specified <date/time>. Ignored if no valid <date/time> is specified. A date must be specified in a format such as RFC 822, RFC 850, or ISO 8601.

By default there is no expiration date for content. If omitted, this page may be shown in search results indefinitely. Crawlers are expected to considerably decrease the crawl rate of the URL after the specified date and time.

Example

Indexing rules through the <meta name="robots"> tag and the X-Robots-Tag HTTP header are identified when a URL is crawled. Many crawlers support rules specified in the X-Robots-Tag HTTP header, which can be set using a <meta name="robots"> element. If there are conflicting robot rules within the X-Robots-Tag or between the X-Robots-Tag HTTP header and the <meta name="robots"> element, the more restrictive rule takes precedence. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will be applied. Indexing directives will not be discovered or enforced if paths are blocked by a robots.txt file. Certain directives, such as index versus noindex or follow versus nofollow, are mutually exclusive. In cases where these conflict, the behavior of crawlers may be unpredictable and vary between implementations.

Note: Always ensure your robots.txt file does not block essential resources or pages you want to be indexed. Use X-Robots-Tag directives carefully to control indexing and crawling.

Blocking a resource via robots.txt prevents crawlers from access, and as a result, they will ignore any indexing or serving rules set by <meta name="robots"> or X-Robots-Tag headers. Even if a page is blocked from crawling, it might still appear in search results if it is referenced elsewhere on the web (especially if nofollow is used). To ensure a page is removed from search indices, applying X-Robots-Tag: noindex usually works, although the page must be revisited by a crawler to recognize this directive.

Below are some examples demonstrating the use of X-Robots-Tag:

The following header instructs crawlers not to index the page or display media and resources associated with it in search results:

HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 17:08:49 GMT
X-Robots-Tag: noindex

When a response contains multiple X-Robots-Tag headers, each specifying different rules, all are considered together. For example:

HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 17:08:49 GMT
X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: Wed, 03 Dec 2025 13:09:53 GMT

It’s also possible to specify rules for specific user agents. For example, the following headers instruct particular bots not to follow links or index the page:

HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 17:08:49 GMT
X-Robots-Tag: BadBot: noindex, nofollow
X-Robots-Tag: googlebot: nofollow

Alternatively, these rules can be combined into a single header, where each rule applies to the specified user agent:

HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 17:08:49 GMT
X-Robots-Tag: BadBot: noindex, nofollow, googlebot: nofollow

When multiple crawlers are targeted with different directives, the crawler will interpret the combined rules as a set of negative instructions for each user agent. For example, a header with:

X-Robots-Tag: nofollow
X-Robots-Tag: googlebot: noindex

means that Googlebot should not index or follow links on the page, while other bots might only follow links unless explicitly instructed otherwise.

These directives are not part of any current formal specification but are widely supported and used in practice.

How to Modify Header using Requestly

Requestly is a powerful Chrome extension that allows you to modify HTTP headers, including the X-Robots-Tag header. This can help you control how search engines index or follow links on your web pages during testing or troubleshooting. Steps to Modify the X-Robots-Tag Header:

Install and open the Requestly Chrome extension. You can find it on the Chrome Web Store.
Create a new rule: Click on “Create Rule” and choose “Modify Headers” from the list of available rule types.
Add a new header modification:
- Under “Action”, select “Add” or “Override”.
- In the “Header Name” field, enter X-Robots-Tag.
- In the “Header Value” field, enter your preferred directive (e.g., noindex, nofollow).
Set the URL condition: Specify the URL or pattern where this header change should apply (e.g., https://your-website.com/*).
Save the rule.

Once set up, Requestly will inject the X-Robots-Tag: noindex, nofollow header into all matching requests, allowing you to control how search engines treat your content during development or testing.

You might need to modify the X-Robots-Tag header to test how search engines handle indexing or following links on your site without changing your actual server settings. This helps ensure your site SEO behaves as expected before going live.