However, with that googlebot-nosnippet command, you would take away the snippets also from websearch results.

I only want to remove the snippets from Google News….(only for googlebot-news) and even my contact at Google News USA believes, the news-bot does not accept the command…

If I have a chance, I´ll try…

Usually you want Google to display both the snippet and the cached link. However, there are some cases where you might want to disable one or both of these. For example, say you were a newspaper publisher, and you have a page whose content changes several times a day. It may take longer than a day for us to reindex a page, so users may have access to a cached copy of the page that is not the same as the one currently on your site. In this case, you probably don’t want the cached link appearing in our results.

Again, the Robots Exclusion Protocol comes to your aid. Add the NOARCHIVE tag to a web page and Google won’t cache copy of a web page in search results:

<META NAME=”GOOGLEBOT” CONTENT=”NOARCHIVE”>

Similarly, you can tell Google not to display a snippet for a page. The NOSNIPPET tag achieves this:

<META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”>

Adding NOSNIPPET also has the effect of preventing a cache link from being shown, so if you specify NOSNIPPET you automatically get NOARCHIVE too.

So I’d guess that works fine with the news crawler. Didn’t test it myself, tough, since I don’t run CNN or so.

So will google accept this?

Trying to get my head round some of it, but the “noindex” stuff has been implemented on my robots.txt.

Yous blog is extremely valuable, and I appreciate that!

Keep up the good work

Christopher

What you need is an apparatus that detects nasty bot behavior analyzing each and every HTTP request, sometimes triggering a ban of an IP, host, user agent, or a combination of those values on the nth request.

Unusual (IOW fraudulent) requests of /robots.txt are just one source that feeds such a black list. For example when you request the robots.txt file of a protected site pretending to be Googlebot from your Cox Communications IP addy, all your future requests run into a 403. The same will happen when you request the root index page, or any other resource (page, image, …), regardless whether you’ve fetched robots.txt before or not.

Legit bots sent out from major search engines provide a procedure to validate their requests. Any request with a search engine crawler user agent string and an IP addy that can’t be verified as belonging to the particular search engine’s crawling engine is fraudulent, and should be blocked. Instead of responding with a 403-GFY you can serve ads or anything else except your content.

If you make use of somewhat shady optimization techniques, such as some variants of IP delivery, the procedure outlined above is not safe enough, but for most sites it will serve you well.

Does this refer to your Disallow in the smart robots.txt file? Or does it refer to some other function? Because it’s a little confusing…as the root of the problem is still in the fact that bad bots generally ignore robots.txt.