Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?
Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.
A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.
Anyone?
Share/bookmark this: del.icio.us • Google • ma.gnolia • Mixx • Netscape • reddit • Sphinn • Squidoo • StumbleUpon • Yahoo MyWeb Subscribe to ![]() ![]() ![]() |
Sebastian | Google, Uncategorized | Related posts
ftp://ftp.utexas.edu/robots.txt is different from http://ftp.utexas.edu/robots.txt - the robots.txt probably needs to be in the ftp root to block control google’s indexing of the ftp server.