Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?

Google states

Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.

A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.


Share/bookmark this: del.icio.usGooglema.gnoliaMixxNetscaperedditSphinnSquidooStumbleUponYahoo MyWeb
Subscribe to      Entries Entries      Comments Comments      All Comments All Comments

1 Comment to "Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?"

  1. Rob on 22 April, 2009  #link

    ftp://ftp.utexas.edu/robots.txt is different from http://ftp.utexas.edu/robots.txt - the robots.txt probably needs to be in the ftp root to block control google’s indexing of the ftp server.

Leave a reply

[If you don't do the math, or the answer is wrong, you'd better have saved your comment before hitting submit. Here is why.]

Be nice and feel free to link out when a link adds value to your comment. More in my comment policy.