Since August 2007 MSN runs a bogus bot faking a human visitor coming from a search results page, that follows their crawler. This spambot downloads everything from a page, that is images and other objects, external CSS/JS files, and ad blocks rendering even contextual advertising from Google and Yahoo. It fakes MSN SERP referrers diluting the search term stats with generic and unrelated keywords. Webmasters running non-adult sites wondered why a database tutorial suddenly ranks for [oral sex] and why MSN sends visitors searching for [MILF pix] to a teenager’s diary. Webmasters assumed that MSN is after deceitful cloaking, and laughed out loud because their webspam detection method was that primitive and easy to fool.

Now MSN admits all their sins –except the launch of a porn affiliate program– and posted a vague excuse on their Webmaster Blog telling the world that they discovered the evil cloakers and their index is somewhat spam free now. Donna has chatted with the MSN spam team about their spambot and reports that blocking its IP addresses is a bad idea, even for sites that don’t cloak. Vanessa Fox summarized MSN’s poor man’s cloaking detection at Search Engine Land:

And one has to wonder how effective methods like this really are. Those savvy enough to cloak may be able to cloak for this new cloaker detection bot as well.

They say that they no longer spam sites that don’t cloak, but reverse this statement telling Donna

we need to be able to identify the legitimate and illegitimate content

and Vanessa

sites that are cloaking may continue to see some amount of traffic from this bot. This tool crawls sites throughout the web — both those that cloak and those that don’t — but those not found to be cloaking won’t continue to see traffic.

Here is an excerpt from yesterdays referrer log of a site that does not cloak, and never did:
http://search.live.com/results.aspx?q=webmaster&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=smart&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=search&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=progress&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=google&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=google&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=domain&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=database&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=content&mrt=en-us&FORM=LIVSOP
http://search.live.com/results.aspx?q=business&mrt=en-us&FORM=LIVSOP
Why can’t the MSN dudes tell the truth, not even when they apologize?

Another lie is “we obey robots.txt”. Of course the spambot doesn’t request it to bypass bot traps, but according to MSN it uses a copy served to the LiveSearch crawler “msnbot”:

Yes, this robot does follow the robots.txt file. The reason you don’t see it download it, is that we use a fresh copy from our index. The tool does respect the robots.txt the same way that MSNBot does with a caveat; the tool behaves like a browser and some files that a crawler would ignore will be viewed just like real user would.

In reality, it doesn’t help to block CSS/JS files or images in robots.txt, because MSN’s spambot will download them anyway. The long winded statement above translates to “We promise to obey robots.txt, but if it fits our needs we’ll ignore it”.

Well, MSN is not the only search engine running stealthy bots to detect cloaking, but they aren’t clever enough to do it in a less abusive and detectable way.

Their insane spambot led all cloaking specialists out there to their not that obvious spam detection methods. They may have caught a few cloaking sites, but considering the short life cycle of Webspam on throwaway domains they shot themselves in both feet. What they really have achieved is that the cloaking scripts are MSN spam detection immune now.

Was it really necessary to annoy and defraud the whole Webmaster community and to burn huge amounts of bandwidth just to catch a few cloakers who launched new scripts on new throwaway domains hours after the first appearance of the MSN spam bot?

Can cosmetic changes with regard to their useless spam activities restore MSN’s lost reputation? I doubt it. They’ve admitted their miserable failure five months too late. Instead of dumping the spambot, they announce that they’ll spam away for the foreseeable future. How silly is that? I thought Microsoft is somewhat profit orientated, why do they burn their and our money with such amateurish projects?

Besides all this crap MSN has good news too. Microsoft Live Search told Search Engine Roundtable that they’ll spam our sites with keywords related to our content from now on, at least they’ll try it. And they have a forum and a contact form to gather complaints. Crap on, so much bureaucratic efforts to administer their ridiculous spam fighting funeral. They’d better build a search engine that actually sends human traffic.

All that comes with common hassles. I want to use these social gadgets and services without jumps thru unintended hoops, that is I consider all the above mentioned methods to tell friends that I still love them diverting those services from their intended use. Also, not every friend of mine makes use of all these geeky tools, so I need to digg posts of A., to delicious articles by B., to share posts of C., and to visit the blogs of D., E. and F. just to show that I’ve read their stuff in my feed reader.

I can’t do that, at least not in a reliable manner, especially not when I’m swamped and just try to catch up after 12 or more hours of dealing with legacy applications or other painful tasks like meetings with wannabe-geeks (unexperienced controllers or chiefs of whichever-useless-service-center) respectively anti-geeks (know-it-all but utterly-clueless and dangerous-to-the-company’s-safety IT managers). Doh!

So when I’m not able to send my friends a twitter-great-job-message or IM, and don’t have the time to link to their stuff, should I feel bad? Probably. Penalties are well deserved. Actually, the consequence is that nice guys like Nick Wilson @Metaversed unfriend me (among other well-meaning followers) at Twitter coz “I didn’t provide useful input for a while”, not knowing that I follow them with interest, read their posts and all that, but just can’t contribute at the moment because their actual field of interest doesn’t match my time schedule respectively my todays-hot-topic-list, nor my current centre of gravity, so to say. That does not mean I’m not interested in whatever they do and output, I just can’t process it ATM but I know that’ll change at some point in the future. Hey, geeks usually hop from today’s hot thing to tomorrow’s hot thing, and flashbacks are rather natural, so why expect continuousness?

Bugger, I wrote four paragraphs and didn’t come to the point expectable from the post’s title. And I bored you dear readers with lots of title bait recently. Sorry, but I did enjoy it. Ok, here’s the message:

Everybody monitors referrer stats. Don’t say you don’t do it because that’s first a lie and second a natural thing to do. That applies to ego searches too by the way. So why don’t we make use of referrer spoofing to send a signal to our friends? It’s that easy. Just add the referrer-spoofing widget to your PrefBar, enter your URL, and surf on. Well, technically that’s referrer spamming, so if you wear a tinfoil hat use a non-indexable server like example.com. I’m currently surfing with the HTTP_REFERER “http://www.example.com/gofuckyourself” but I’m going to change that to this blog’s URL. Funny folks visiting my blog provide bogus referrers like “http://spamteam.google.com/” and “http://corp.google.com:8080/webspam/watchlist.py”, so why the fuck shouldn’t I use my actual address? This will tell my friends that I still love them. And real geeks shouldn’t expect unforged referrer stats, since many nice guys surf without spamming the server logs with a referrer.

What do you think?

I love PrefBar, a neat FireFox plug-in, which provides me with a pretty useful customizable toolbar. With PrefBar you can switch JavaScript, Flash, colors, images, cookies… on and off with one mouse click, and you can enter a list of user agent names to choose the user agent while browsing.

So I’ve asked Manuel Reimer to create a referrer spoofer widget, and kindly he created it with PrefBar 3.4.1. Thank you Manuel!

To activate referrer spoofing in your PrefBar toolbar install or update Prefbar to 3.4.1, then download the Referer Spoof Menulist 1.0, click “Customize” on the toolbar and import the file. Then click on “Edit” to add all the referrer URLs you need for testing purposes, and enjoy. It works great.