Comments on: Q&A: An undocumented robots.txt crawler directive from Google http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/ If you've read my articles somewhere on the Internet, expect something different here. Thu, 24 Jul 2008 16:24:31 +0000 http://wordpress.org/?v=2.2.3 By: XML Sitemaps and Competitive Intelligence - Pocket SEO http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-887 XML Sitemaps and Competitive Intelligence - Pocket SEO Wed, 26 Dec 2007 09:34:55 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-887 [...] engines can’t index your sitemap files you might be able to block them with robots.txt (e.g., noindex directive), and/or send an x-robots-tag HTTP header telling Google and Yahoo not to index them. This entry [...]

[…] engines can’t index your sitemap files you might be able to block them with robots.txt (e.g., noindex directive), and/or send an x-robots-tag HTTP header telling Google and Yahoo not to index them. This entry […]

]]> By: Sebastian http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-743 Sebastian Mon, 26 Nov 2007 15:46:04 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-743 Jordan, thanks and give <a href="http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/">this</a> a try. Jordan, thanks and give this a try.

]]> By: Advantages of a smart robots.txt file http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-741 Advantages of a smart robots.txt file Mon, 26 Nov 2007 15:16:29 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-741 [...] loyal reader of my pamphlets asked me: I foresee many new capabilities with robots.txt in the future due to this [Google’s [...] […] loyal reader of my pamphlets asked me: I foresee many new capabilities with robots.txt in the future due to this [Google’s […]

]]> By: Utah SEO Pro http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-737 Utah SEO Pro Sun, 25 Nov 2007 22:01:12 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-737 This discovery is quite the milestone. I foresee many new capabilities with robots.txt in the future due to this. However, how the hell can a webmaster hide their robots.txt from the public while serving it up to bots without doing anything shady? Do you have any example .htaccess code? Cuz I know you're the pimp master at that kinda shit. Way to lay it thick on Igor. I love how you're so forward. haha This discovery is quite the milestone. I foresee many new capabilities with robots.txt in the future due to this. However, how the hell can a webmaster hide their robots.txt from the public while serving it up to bots without doing anything shady? Do you have any example .htaccess code? Cuz I know you’re the pimp master at that kinda shit.

Way to lay it thick on Igor. I love how you’re so forward. haha

]]> By: Sebastian http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-718 Sebastian Wed, 21 Nov 2007 10:00:09 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-718 Probably a trust issue, no SEO-voodoo involved. It seems mine is one of very few sites still allowing you to leave your nick. I don't try to outrank your absurd home page, why should I? You've created your nick yourself, and it's suitable. At Google I just told you that trolling is not appreciated. Igor, you were warned that insane and childish off-topic comments will get you banned eventually. I've deleted a few of them and if you don't behave you're history. That's my last warning. Behave yourself or get the fuck outta here. Probably a trust issue, no SEO-voodoo involved. It seems mine is one of very few sites still allowing you to leave your nick. I don’t try to outrank your absurd home page, why should I?

You’ve created your nick yourself, and it’s suitable. At Google I just told you that trolling is not appreciated.

Igor, you were warned that insane and childish off-topic comments will get you banned eventually. I’ve deleted a few of them and if you don’t behave you’re history. That’s my last warning. Behave yourself or get the fuck outta here.

]]> By: Igor The Troll http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-716 Igor The Troll Wed, 21 Nov 2007 09:20:49 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-716 Sebasian, why do you rank number 1 for Igor The Troll in Google? Is Sebstain Igor The Troll? http://www.google.com/search?hl=en&q=Igor+The+Troll I know you created me in GWHG, but now you are taking over my Nick...are you some Alien being that sucks the souls out of people and their Websites? NP, Everyone knows that Sabestian is the father of Igor The Troll Or, are you gaming Google with Black Hat Seo? Sebasian, why do you rank number 1 for Igor The Troll in Google?

Is Sebstain Igor The Troll?
http://www.google.com/search?hl=en&q=Igor+The+Troll

I know you created me in GWHG, but now you are taking over my Nick…are you some Alien being that sucks the souls out of people and their Websites?

NP, Everyone knows that Sabestian is the father of Igor The Troll

Or, are you gaming Google with Black Hat Seo?

]]> By: Validate your robots.txt - Googlebot becomes smarter http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-714 Validate your robots.txt - Googlebot becomes smarter Tue, 20 Nov 2007 18:58:27 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-714 [...] week I reported that Google experiments with new crawler directives for use in robots.txt. Today Google has confirmed that Googlebot understands experimental REP syntax like [...] […] week I reported that Google experiments with new crawler directives for use in robots.txt. Today Google has confirmed that Googlebot understands experimental REP syntax like […]

]]> By: JohnMu http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-712 JohnMu Tue, 20 Nov 2007 12:12:28 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-712 Wildcards should be ok, as they are for the normal disallow: and allow: directives. I just want to remind everyone again that this is something that may still change over time. Be careful when playing with things like this :) . Wildcards should be ok, as they are for the normal disallow: and allow: directives. I just want to remind everyone again that this is something that may still change over time. Be careful when playing with things like this :) .

]]> By: Sebastian http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-711 Sebastian Tue, 20 Nov 2007 11:59:22 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-711 Thanks for your confirmation John! Googlebot has fetched the <a href="http://sebastians-pamphlets.com/noindex/">crawler trap</a> on 2007-11-14 07:23:03 and it's not <a href="http://www.google.com/search?q=noindex+crawler+trap+site:sebastians-pamphlets.com&num=100&hl=en&safe=off&filter=0">indexed</a>. Maybe the page makes it in the index if you at the time of crawling and indexing had a cached robots.txt without the Noindex: statement, but I doubt it. I'll watch that for a while and then launch another experiment w/o telling the URI. ;) Do you support wildcards with "Noindex:" too, for example Noindex: /*.txt$ Noarchive: /cloaked/*.html$ Nofollow: /links/*.php$ Nopreview: /scholar/*.pdf$ or so? That would be exciting! Please LMK :) Thanks for your confirmation John! Googlebot has fetched the crawler trap on 2007-11-14 07:23:03 and it’s not indexed. Maybe the page makes it in the index if you at the time of crawling and indexing had a cached robots.txt without the Noindex: statement, but I doubt it. I’ll watch that for a while and then launch another experiment w/o telling the URI. ;)
Do you support wildcards with “Noindex:” too, for example
Noindex: /*.txt$
Noarchive: /cloaked/*.html$
Nofollow: /links/*.php$
Nopreview: /scholar/*.pdf$
or so? That would be exciting! Please LMK :)

]]> By: JohnMu http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-710 JohnMu Tue, 20 Nov 2007 11:23:51 +0000 http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-710 Good catch, Sebastian. How is your experiment going? At the moment we will usually accept the "noindex" directive in the robots.txt, but we are not yet at a point where we are willing to set it into stone and announce full support. Good catch, Sebastian. How is your experiment going? At the moment we will usually accept the “noindex” directive in the robots.txt, but we are not yet at a point where we are willing to set it into stone and announce full support.

]]>