Precedence of X-Robots-Tag header vs robots meta tag

Question

I've placed the following Header in my vhost config:

Header set X-Robots-Tag "noindex, nofollow"

The goal here is to just disable search engines from indexing my testing environment. The site is Wordpress and there is a plugin installed to manage per-page the meta robots settings. For example:

<meta name="robots" content="index, follow" />

So my question is, which directive will take precedence over the other since both are being set on every page?

Answer 1

I am not sure if a definitive answer can be given to the question, as the behavior may be implementation-dependent (on the robot side).

However, I think there is reasonable evidence that X-Robots-Tag will take precedence over <meta name="robots" .... See :

One significant difference between the X-Robots-Tag and the robots meta directive is:

• X-Robots-Tag is part of the HTTP protocol header.
• <meta name="robots" ... is part of the HTML document header.

Therefore the the X-Robots-Tag belongs to HTTP protocol layer, while <meta name="robots" ... belongs to the HTML protocol layer.

As they belong to a different protocol layer, they will not be parsed simultaneously by the (robot) client getting the page: The HTTP layer will be parsed first, and the HTML in a later step.

(Also, it should be noted that X-Robots-Tag and <meta name="robots" ... are not suppported by all robots. Google and Yahoo/Bing suppport both, but according to this some support only <meta name="robots" ..., others support neither.)

Summary :

• if supported by the robot, X-Robots-Tag will be processed first ; restrictions (noindex, nofollow) apply (and <meta name="robots" ... is ignored).
• else, <meta name="robots" ... directive applies.

Answer 2

In my recent experience, when Google sees mixed-messages it prefers positive action by default - ie - it favours indexing - meanwhile will flag the issue as a critical error/warning in your webmaster tools console if you have one.

see your site's status in google here: https://www.google.com/webmasters/

see you site's status in bing here: http://www.bing.com/toolbox/webmaster (note that yahoo search is now powered by bing)

Google takes this positive-by-default action because lots of site owners unwittingly have a dodgy cms semi-blocking robots and we know how google loves to accumulate as much data as it can - any excuse!

if the technical settings are erroneous they're liable to be totally disregarded, and we know how search engines index and follow by default when no settings are specified.

Answer 3

Just an update to Dan's experience, I also have both the

Header set X-Robots-Tag "noindex, nofollow"

and

<meta name="robots" content="index, follow" />

on my one of my Wordpress sites, and a check in Google Search Console confirmed that the noindex in X-Robots-Tag is taking precedence as the pages have been crawled and but aren't indexed. So the logic in the correct answer is indeed, correct.