Comments on: Validate your robots.txt - Googlebot becomes smarter http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/ If you've read my articles somewhere on the Internet, expect something different here. Wed, 23 May 2012 06:51:27 +0000 http://wordpress.org/?v=2.2.3 By: Noindex dans robots.txt est-elle prise en compte ? http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-2840 Noindex dans robots.txt est-elle prise en compte ? Tue, 29 Mar 2011 11:53:36 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-2840 [...] from Google et encore plus avec les résultats de ces tests et une annonce de John Müller Validate your robots.txt – Googlebot becomes smarter. Je décide donc de mettre cela en place sur quelques sites et de vérifier la véracité de ces [...] […] from Google et encore plus avec les résultats de ces tests et une annonce de John Müller Validate your robots.txt – Googlebot becomes smarter. Je décide donc de mettre cela en place sur quelques sites et de vérifier la véracité de ces […]

]]> By: vande bali news http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-1753 vande bali news Tue, 25 Aug 2009 22:56:19 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-1753 how to make that sitemap is allow in robot.txt, i try to put : User-agent: * allow:/sitemap.xml but google says that my robots.txt is block googlebots, [Try User-agent: * Sitemap: http:// the bali pages . com (without the spaces)] how to make that sitemap is allow in robot.txt, i try to put :

User-agent: *
allow:/sitemap.xml

but google says that my robots.txt is block googlebots,

[Try
User-agent: *
Sitemap: http:// the bali pages . com
(without the spaces)]

]]> By: My plea to Google - Please sanitize your REP revamps http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-916 My plea to Google - Please sanitize your REP revamps Fri, 04 Jan 2008 00:03:12 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-916 [...] supports a Noindex: directive in robots.txt. It seems Google’s Noindex: blocks crawling like Disallow:, but additionally prevents URLs blocked with [...] […] supports a Noindex: directive in robots.txt. It seems Google’s Noindex: blocks crawling like Disallow:, but additionally prevents URLs blocked with […]

]]> By: Utah SEO Pro http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-777 Utah SEO Pro Sun, 02 Dec 2007 20:37:12 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-777 Oh my god, block level robots.txt commands is ingenious! That'd be great. I always thought there should be an inline directive for that. Oh my god, block level robots.txt commands is ingenious! That’d be great. I always thought there should be an inline directive for that.

]]> By: Google to change the Robots Exclusion Protocol again http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-775 Google to change the Robots Exclusion Protocol again Sat, 01 Dec 2007 09:42:54 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-775 [...] Google is working on new robots.txt syntax, and I am, politely put, not amused. Here is why I fear that Google is going to totally mess up the [...] […] Google is working on new robots.txt syntax, and I am, politely put, not amused. Here is why I fear that Google is going to totally mess up the […]

]]> By: Sebastian http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-764 Sebastian Wed, 28 Nov 2007 18:55:40 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-764 Looking at the <a href="http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#robots-txt-test-results-2007-11-28">preliminary test results</a>, I was not amused. Looking at the preliminary test results, I was not amused.

]]> By: david deangelo http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-734 david deangelo Sat, 24 Nov 2007 07:57:41 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-734 Good article. My main use of robot.txt is to disallow certain bots. Good article. My main use of robot.txt is to disallow certain bots.

]]> By: 5 ways to de-index your site • Tim Nash UK SEO Blog http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-731 5 ways to de-index your site • Tim Nash UK SEO Blog Fri, 23 Nov 2007 08:36:56 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-731 [...] in 94 now in the nauties we can simply screw up our site with the line Noindex: / How cool is that! Sebastian as always has come up with some more creative examples of robot.txt directives which may or may not [...] […] in 94 now in the nauties we can simply screw up our site with the line Noindex: / How cool is that! Sebastian as always has come up with some more creative examples of robot.txt directives which may or may not […]

]]> By: Search Engine Land: News About Search Engines & Search Marketing http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-727 Search Engine Land: News About Search Engines & Search Marketing Wed, 21 Nov 2007 17:41:00 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-727 <strong>SearchCap: The Day In Search, November 21, 2007...</strong> Below is what happened in search today, as reported on Search Engine Land and from other places across the web....... SearchCap: The Day In Search, November 21, 2007…

Below is what happened in search today, as reported on Search Engine Land and from other places across the web…….

]]> By: kelvin newman http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-724 kelvin newman Wed, 21 Nov 2007 13:30:27 +0000 http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#comment-724 I really ought to get to grips with Robots.txt stuff, I find it intriguing even though it's not an area of expertise I really ought to get to grips with Robots.txt stuff, I find it intriguing even though it’s not an area of expertise

]]>