Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?

Posted on 12 March, 2007

Google states

Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.

A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.

Anyone?


Share/bookmark this: del.icio.us • Google • ma.gnolia • Mixx • Netscape • reddit • Sphinn • Squidoo • StumbleUpon • Yahoo MyWeb
Subscribe to Entries Entries Comments Comments All Comments All Comments 		 

Sebastian | Google, Uncategorized | Related posts

Be the first to add a Comment to "Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?"

Leave a reply


[If you don't do the math, or the answer is wrong, you'd better have saved your comment before hitting submit. Here is why.]

Be nice and feel free to link out when a link adds value to your comment. More in my comment policy.


Find related posts in Google, Uncategorized