Comments on: In need of a "Web-Robot Directives Standard" http://sebastians-pamphlets.com/in-need-of-a-web-robot-directives-standard/ If you've read my articles somewhere on the Internet, expect something different here. Fri, 25 Jul 2008 11:08:12 +0000 http://wordpress.org/?v=2.2.3 By: Google to change the Robots Exclusion Protocol again http://sebastians-pamphlets.com/in-need-of-a-web-robot-directives-standard/#comment-771 Google to change the Robots Exclusion Protocol again Fri, 30 Nov 2007 18:34:36 +0000 http://sebastians-pamphlets.com/in-need-of-a-web-robot-directives-standard/#comment-771 [...] in the Robots Exclusion Protocol (REP), evolved since 1994. Nowadays we’ve to deal with a conglomerate of not binding de facto standards and microformats, all of them extended by various organizations. All search engines claim that they obey “the [...]

[…] in the Robots Exclusion Protocol (REP), evolved since 1994. Nowadays we’ve to deal with a conglomerate of not binding de facto standards and microformats, all of them extended by various organizations. All search engines claim that they obey “the […]

]]> By: Validate your robots.txt - Googlebot becomes smarter http://sebastians-pamphlets.com/in-need-of-a-web-robot-directives-standard/#comment-729 Validate your robots.txt - Googlebot becomes smarter Wed, 21 Nov 2007 18:38:50 +0000 http://sebastians-pamphlets.com/in-need-of-a-web-robot-directives-standard/#comment-729 [...] Here is the live test of suspected respectively desired new crawler directives for robots.txt. I’ve added a few unusual statements to my robots.txt [...] […] Here is the live test of suspected respectively desired new crawler directives for robots.txt. I’ve added a few unusual statements to my robots.txt […]

]]>