Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?

Posted on 12 March, 2007

Google states

Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.

A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.

Anyone?


Share/bookmark this: del.icio.us • Google • ma.gnolia • Mixx • Netscape • reddit • Sphinn • Squidoo • StumbleUpon • Yahoo MyWeb
Subscribe to Entries Entries Comments Comments All Comments All Comments 		 

Sebastian | Google, Uncategorized | Related posts

1 Comment to "Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?"

  1. Rob on 22 April, 2009  #link

    ftp://ftp.utexas.edu/robots.txt is different from http://ftp.utexas.edu/robots.txt - the robots.txt probably needs to be in the ftp root to block control google’s indexing of the ftp server.

Leave a reply


[If you don't do the math, or the answer is wrong, you'd better have saved your comment before hitting submit. Here is why.]

Be nice and feel free to link out when a link adds value to your comment. More in my comment policy.


Find related posts in Google, Uncategorized