Comments on: Handling Google’s neat X-Robots-Tag - Sending REP header tags with PHP

By: Jeff

Jeff — Fri, 05 Feb 2010 15:55:41 +0000

This works really simply, and it keeps other headers (like Content-Length) that Apache applies to PDFs:

Header set X-Robots-Tag “index, noarchive”

You can put it in the server config, a Virtual Host section, or an .htaccess, for fine control of what PDFs it applies to (and/or adjust the regex).

Jeff

By: Houston SEO

Houston SEO — Wed, 16 Dec 2009 18:15:06 +0000

Thanks for this interesting article, Just discovered this.. can be quite vicious

[Are you aware that links dropped by ‘Houston SEO’ get removed across the boards, in case the comments submitted by you guys aren’t catched by spam filters beforehand?]

By: WordPress Hacked? Total Security Lockdown

WordPress Hacked? Total Security Lockdown — Tue, 08 Dec 2009 05:46:46 +0000

[…] be useful as wellGetting URLs outta Google – the good, the popular, and the definitive way Handling Google’s neat X-Robots-Tag – Sending REP header tags with PHPNasty Bots & UsersA lot of security relies on identifying nasty bots, detecting rogue activity […]

By: Alphane Moon

Alphane Moon — Wed, 19 Dec 2007 09:11:17 +0000

Hi Sebastian,
thank you for this article on the X-Robots-Tag, it is very useful and was precisely what I was looking for.

By: Noobliminal

Noobliminal — Thu, 13 Sep 2007 18:35:09 +0000

I’m not .. or .. am I? %\
Regards.

By: Sebastian

Sebastian — Thu, 13 Sep 2007 18:29:50 +0000

Noobliminal, your method works. To prevent you from such cheats by sneaky link mongers there’s only one route: don’t do link exchanges.

By: Noobliminal

Noobliminal — Thu, 13 Sep 2007 13:39:53 +0000

I have figured an ‘alternative’ use for this new http header tag. Might be worth a check

By: Unavailable_After is totally and utterly useless

Unavailable_After is totally and utterly useless — Tue, 21 Aug 2007 07:10:52 +0000

[…] Dan Crow from Google announced a pretty neat thing in the same post: With the X-Robots-Tag you can now apply crawler directives valid in robots meta tags to non-HTML documents like PDF files […]

By: NOPREVIEW - The missing X-Robots-Tag

NOPREVIEW - The missing X-Robots-Tag — Mon, 20 Aug 2007 22:04:17 +0000

[…] malware, etcetera. However, under some circumstances it would make sound sense to have a NOPREVIEW X-Robots-Tag, but unfortunately Google forgot to introduce it […]

By: Hamlet Batista

Hamlet Batista — Thu, 02 Aug 2007 15:06:00 +0000

Sebastian - That is very clever. Good job!

I thought about doing something similar, but decided the .htaccess solution might be easier for non-programmers.

I thought about using ScriptAliasMatch to map files to a cgi script that would add the header and using PATH_INFO to find the file on disk.