A pragmatic defence against Google’s anti paid links campaign

Google’s recent shot across the bows of a gazillion sites handling paid links, advertising, or internal cross links not compliant to Google’s imagination of a natural link is a call for action. Google’s message is clear: “condomize your commercial links or suffer” (from deducted toolbar PageRank, links without the ability to pass real PageRank and relevancy signals, or perhaps even penalties).

Paid links: good versus evilOf course that’s somewhat evil, because applying nofollow values to all sorts of links is not exactly a natural thing to do; visitors don’t care about invisible link attributes and sometimes they’re even pissed when they get redirected to an URL not displayed in their status bar. Also, this requirement forces Webmasters to invest enormous efforts in code maintenance for the sole purpose of satisfying search engines. The argument “if Google doesn’t like these links, then they can discount them in their system, without bothering us” has its merits, but unfortunately that’s not the way Google’s cookie crumbles for various reasons. Hence lets develop a pragmatic procedure to handle those links.

The problem

Google thinks that uncondomized paid links as well as commercial links to sponsors or affiliated entities aren’t natural, because the terms “sponsor|pay for review|advertising|my other site|sign-up|…” and “editorial vote” are not compatible in the sense of Google’s guidelines. This view at the Web’s linkage is pretty black vs. white.

Either you link out because a sponsor bought ads, or you don’t sell ads and link out for free because you honestly think your visitors will like a page. Links to sponsors without condom are black, links to sites you like and which you don’t label “sponsor” are white.

There’s nothing in between, respectively gray areas like links to hand picked sponsors on a page with a gazillion of links count as black. Google doesn’t care whether or not your clean links actually pass a reasonable amount of PageRank to link destinations which buy ad space too, the sole possibility that those links could  influence search results is enough to qualify you as sort of a link seller.

The same goes for paid reviews on blogs and whatnot, see for example Andy’s problem with his honest reviews which Google classifies as paid links, and of course all sorts of traffic deals, affiliate links, banner ads and stuff like that.

You don’t even need to label a clean link as advert or sponsored. If the link destination matches a domain in Google’s database of on-line advertisers, link buyers, e-commerce sites / merchants etcetera, or Google figures out that you link too much to affiliated sites or other sites you own or control, then your toolbar PageRank is toast and most probably your outgoing links will be penalized. Possibly these penalties have impact on your internal links too, what results in less PageRank landing on subsidiary pages. Less PageRank gathered by your landing pages means less crawling, less ranking, less SERP referrers, less revenue.

The solution

You’re absolutely right when you say that such search engine nitpicking should not force you to throw nofollow crap on your links like confetti. From your and my point of view condomizing links is wrong, but sometimes it’s better to pragmatically comply to such policies in order to stay in the game.

Although uncrawlable redirect scripts have advantages in some cases, the simplest procedure to condomize a link is the rel-nofollow microformat. Here is an example of a googlified affiliate link:
<a href="http://sponsor.com/?affID=1" rel="nofollow">Sponsor</a>

Why serve your visitors search engine crawler directives?

Complying to Google’s laws does not mean that you must deliver crawler directives like rel=”nofollow” to your visitors. Since Google is concerned about search engine rankings influenced by uncondomized links with commercial intent, serving crawler directives to crawlers and clean links to users is perfectly in line with Google’s goals. Actually, initiatives like the X-Robots-Tag make clear that hiding crawler directives from users is fine with Google. To underline that, here is a quote from Matt Cutts:

[…] If you want to sell a link, you should at least provide machine-readable disclosure for paid links by making your link in a way that doesn’t affect search engines. […]

The other best practice I’d advise is to provide human readable disclosure that a link/review/article is paid. You could put a badge on your site to disclose that some links, posts, or reviews are paid, but including the disclosure on a per-post level would better. Even something as simple as “This is a paid review” fulfills the human-readable aspect of disclosing a paid article. […]

Google’s quality guidelines are more concerned with the machine-readable aspect of disclosing paid links/posts […]

To make sure that you’re in good shape, go with both human-readable disclosure and machine-readable disclosure, using any of the methods [uncrawlable redirects, rel-nofollow] I mentioned above.
[emphasis mine]

Since Google devalues paid links anyway, search engine friendly cloaking of rel-nofollow for Googlebot is a non-issue with advertisers, as long as this fact is disclosed. I bet most link buyers look at the magic green pixels anyway, but that’s their problem.

How to cloak rel-nofollow for search engine crawlers

I’ll discuss a PHP/Apache example, but this method is adaptable to other server sided scripting languages like ASP or so with ease. If you’ve a static site and PHP is available on your (*ix) host, you need to tell Apache that you’re using PHP in .html (.htm) files. Put this statement in your root’s .htaccess file:
AddType application/x-httpd-php .html .htm

Next create a plain text file, insert the code below, and upload it as “funct_nofollow.php” or so to your server’s root directory (or a subdirectory, but then you need to change some code below).
<?php
function makeRelAttribute ($linkClass) {
$numargs = func_num_args();
// optional 2nd input parameter: $relValue
if ($numargs >= 2) {
$relValue = func_get_arg(1) ." ";
}
$referrer = $_SERVER["HTTP_REFERER"];
$refUrl = parse_url($referrer);
$isSerpReferrer = FALSE;
if (stristr($refUrl[host], "google.") ||
stristr($refUrl[host], "yahoo."))
$isSerpReferrer = TRUE;
$userAgent = $_SERVER["HTTP_USER_AGENT"];
$isCrawler = FALSE;
if (stristr($userAgent, "Googlebot") ||
stristr($userAgent, "Slurp"))
$isCrawler = TRUE;
if ($isCrawler /*|| $isSerpReferrer*/ ) {
if ("$linkClass" == "ad") $relValue .= "advertising nofollow";
if ("$linkClass" == "paid") $relValue .= "sponsored nofollow";
if ("$linkClass" == "own") $relValue .= "affiliated nofollow";
if ("$linkClass" == "vote") $relValue .= "editorial dofollow";
}
if (empty($relValue))
return "";
return " rel=\"" .trim($relValue) ."\" ";
} // end function makeRelValue
?>

Next put the code below in a PHP file you’ve included in all scripts, for example header.php. If you’ve static pages, then insert the code at the very top.
<?php
@include($_SERVER["DOCUMENT_ROOT"] ."/funct_nofollow.php");
?>

Do not paste the function makeRelValue itself! If you spread code this way you’ve to edit tons of files when you need to change the functionality later on.

Now you can use the function makeRelValue($linkClass,$relValue) within the scripts or HTML pages. The function has an input parameter $linkClass and knows the (self-explanatory) values “ad”, “paid”, “own” and “vote”. The second (optional) input parameter is a value for the A element’s REL attribute itself. If you provide it, it gets appended, or, if makeRelValue doesn’t detect a spider, it creates a REL attribute with this value. Examples below. You can add more user agents, or serve rel-nofollow to visitors coming from SERPs by enabling the || $isSerpReferrer condition (remove the bold /*&*/).

When you code a hyperlink, just add the function to the A tag. Here is a PHP example:
print "<a href=\"http://google.com/\"" .makeRelAttribute("ad") .">Google</a>";

will output
<a href="http://google.com/" rel="advertising nofollow" >Google</a>
when the user agent is Googlebot, and
<a href="http://google.com/">Google</a>
to a browser.

If you can’t write nice PHP code, for example because you’ve to follow crappy guidelines and worst practices with a WordPress blog, then you can mix HTML and PHP tags:
<a href="http://search.yahoo.com/"<?php print makeRelAttribute("paid"); ?>>Yahoo</a>

Please note that this method is not safe with search engines or unfriendly competitors when you want to cloak for other purposes. Also, the link condoms are served to crawlers only, that means search engine staff reviewing your site with a non-crawler user agent name won’t spot the nofollow’ed links unless they check the engine’s cached page copy. An HTML comment in HEAD like “This site serves machine-readable disclosures, e.g. crawler directives like rel-nofollow applied to links with commercial intent, to Web robots only.” as well as a similar comment line in robots.txt would certainly help to pass reviews by humans.

A Google-friendly way to handle paid links, affiliate links, and cross linking

Load this page with different user agents and referrers. You can do this for example with a FireFox extension like PrefBar. For testing purposes you can use these user agent names:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

and these SERP referrer URLs:
http://google.com/search?q=viagra
http://search.yahoo.com/search?p=viagra&ei=utf-8&iscqry=&fr=sfp

Just enter these values in PrefBar’s user agent respectively referrer spoofing options (click “Customize” on the toolbar, select “User Agent” / “Referrerspoof”, click “Edit”, add a new item, label it, then insert the strings above). Here is the code above in action:

Referrer URL:
User Agent Name: CCBot/2.0 (http://commoncrawl.org/faq/)
Ad makeRelAttribute(”ad”): Google
Paid makeRelAttribute(”paid”): Yahoo
Own makeRelAttribute(”own”): Sebastian’s Pamphlets
Vote makeRelAttribute(”vote”): The Link Condom
External makeRelAttribute(”", “external”): W3C rel="external"
Without parameters makeRelAttribute(”"): Sphinn

When you change your browser’s user agent to a crawler name, or fake a SERP referrer, the REL value will appear in the right column.

When you’ve developed a better solution, or when you’ve a nofollow-cloaking tutorial for other programming languages or platforms, please let me know in the comments. Thanks in advance!



Share/bookmark this: del.icio.usGooglema.gnoliaMixxNetscaperedditSphinnSquidooStumbleUponYahoo MyWeb
Subscribe to      Entries Entries      Comments Comments      All Comments All Comments
 

20 Comments to "A pragmatic defence against Google's anti paid links campaign"

  1. JLH on 26 October, 2007  #link

    Sphunn.

    Plus it has the added benefit of screwing the people that think they are buying a link for PageRank, as long as they don’t check their link with another user agent. :)

    Just kidding, I know that’s not your intent.

  2. Michael VanDeMar on 26 October, 2007  #link

    Sphunn, because I do feel the discussion has merit, but I honestly have to disagree with you. Google is behaving like an over emotional 14 year old girl who is changing the rules at her whim. They pretty much stated that the rule of thumb behind paid links without condomization (ie. Yahoo Directory) is editorial discretion. If they were to simply factor in a Negative PageRank attribute for linking to crappy sites, then they would easily be able to enforce this. As it is, they have apparently decided that we cannot decide for ourselves who is and who is not worthy of linking to, and this latest bout of penalties is their temper-tantrum response. Feeding in to it does neither ourselves nor Google any favors. What they need to do is to grow up.

    They have lost the vision of indexing the worlds knowledge, and now are trying to dictate the web’s content instead. That is ludicrous.

  3. Sebastian on 26 October, 2007  #link
  4. Sebastian on 26 October, 2007  #link

    Michael, we both know that Google forcing Webmasters to condomize links with at least slightly commercial intent is, well, a mistake. The point is, that Google will not change this policy. The whole Web could cry foul, that would not change Google’s opinion that paid links are not good for ranking algos and therefore must vanish completely. They decided to police the Web, and they’ve the power to enforce it. Actually, they took away a few opportunities to manipulate search engines. That’s not a big loss, because every dog and his fleas took advantage of them. Think positive, they just opened the market for ninjas. Frankly, we all knew since a few years that this will happen. At the latest after the Searchking debacle or the great O’Reilly debate we began to change our methods. ;)

  5. fantomaster on 26 October, 2007  #link

    Nice hack.
    What you’re not addressing, however, is the fundamental issue of how Google it to actually determine a link’s status (well, “intention” to all practical purposes, actually).
    Ok, if you believe they can do that (and that, by the way, is EXACTLY what they want to make you believe…), your policy may make sense. (Though more realistically it’ll simply change linking contracts to rule it out - else, no payola, simple as that.) As a site selling links you’d better go full disclosure on this or you’ll need a heck of a legal department to save your derriere.
    IMV there’s more viable ways to get around this - strategies, moreover, that won’t devalue paid links and screw the entire market.

  6. Sebastian on 27 October, 2007  #link

    Ralph, judging from the few sites suffering from deducted toolbar PR I visited recently, I could write the primitive algos to detect most of the paid links myself. I mean when a blog has blinking banners linked to hosting services and well known advertisers like that, links to text link brokers masked with tinyURL only, parts of the blogroll duplicated on tons of affiliated sites, pay for review buttons nofollow’ed or not …, then a blind, deaf and dumb monkey can detect link intents. I’m sure Google can do that algorithmically, and for this sort of links I posted the nofollow hack.

    When a site sells links via a public service leaving footprints, and all those brokers put trackable links, then this site’s owner, Webmaster or SEO slept the last years. In this case refunds caused by subsequently applied link condoms are unavoidable, and probably Google will approve a reconsideration request. By the way, the brokers deal with condomized links too, and lots of sites sell nofollow’ed links. Google has devalued paid links since 2003, probably even longer, hence dofollow’ed link trades dealt with zilch PageRank anyway. IANAL, but I doubt that it’s possible to win a case when the commodity both sites agreed over is a fiction and this fact was public knowledge at the time the contract was signed. Of course, full disclosure is a no-brainer.

    There will always be paid links that Google can’t detect, and smart link building can at least reduce the chances that competitors turn in such linkage. But that’s another league. I don’t think that Joe Webmaster has the skills to accomplish that, nor the budget to pay a SEO consultant who can do it. My posts addresses folks not able or willing to take the risks involved with paid links.

    I doubt Google has screwed the entire market. I agree that the mass market is quite dead now, but deep pockets and smart approaches still allow successful link trades. Google was after the obvious paid links (80), not the cleverly hidden intent (20). Many Webmasters and even SEOs who argued “Google can never detect all paid links” didn’t understand that, and now they suffer from penalties because they were arrogant or ignorant enough to not cover their ass with regard to easy detectable paid links.

    I don’t like it, but it’s reality. Google changed a part of the Web’s economy.

  7. david deangelo on 27 October, 2007  #link

    This is an brilliant article, and very clever. The set back though is that google could consider this like an invisible link (as this technique is showing the googlebot different content to that of the user). If they find out, the webmaster using this technique could be penalised.

  8. Sebastian on 28 October, 2007  #link

    David, hidden links work the other way round: show the link to crawlers, obfuscate or hide it from visitors. This nofollow hack shows nofollow to crawlers, so that search engines don’t count it for ranking purposes. In my book that’s a perfect implementation of machine-readable disclosure for the engines and Google should like it. Cloaking at all is not penalized by Google and other engines, “deceitful” cloaking is a problem when you get caught. By the way, judging from your link nofollow confuses you.

  9. fantomaster on 28 October, 2007  #link

    Your point about “primitive paid linking” (my term, of course) is well taken. However, paid linking won’t go away and in my view it’s not much of a business model to offer paid links at half-value. (Ok, so many such links probably aren’t about influencing ranking at all - they’re merely intended to pull traffic, and your solution doesn’t impede that in any way.)
    What’s more likely is that paid linking will become a lot sneakier (following the technological lead of those speculative “20%” more sophisticated, savvy linkers you mention) - which will land Google in a nice impasse again.
    In any case the irony of actually deploying cloaking in order to be IN COMPLIANCE rather than at odds with Google’s TOS/webmaster guidelines is rather stunning.

    From a technical point of view I would seriously recommend going for full IP delivery, however. Since both hosts and UserAgents can be spoofed with preciously little effort, anyone who implements your solution will otherwise be liable for bigtime outing by competitors denigrating them for “selling useless links”, especially those who choose not to go for full disclosure because of (probably very reasonable) fears that their revenues will suffer.
    (Lots of business opportunities come to mind as well - e.g. portals selling lists of advertisers who won’t use link condoms, specialized brokerage etc. etc.)

    In any case it’s a mere workaround (albeit an admittedly brilliant one) addressing a problem the Google crowd have created for themselves and which IMV should be their responsibility, not the webmasters’, to resolve.

  10. Sebastian on 28 October, 2007  #link

    I agree that IP delivery is a cleaner way to cloak nofollow or anything else, and for non-cloaking readers I’d like to add that this method should come with a “noarchive” X-Robots-Tag in the header, or a “noarchive” value in the conventional robots meta tag, because otherwise competitors would find the nofollow’ed links in the cached copy. However, maintaining IP lists is not for everybody, and you won’t share yours, won’t you? Of course, in this case Matt’s Googlebot identification method would do the trick too, although without caching it slows crawling down. Replacing user agent detection by IP testing should be an easy hack in the function above.

    Probably Google can live with the few savvy linkers (actually way less than 20% IMO) better than with the current amount of paid links, at least for a while. Once they’ve wiped out the majority, that is all primitive paid links, they’ll dedicate their efforts to sophisticated linkage, however. Respectively I doubt they wait that long, I believe that they aren’t so much behind of most ninja tactics applied by smart link builders, thanks to paid links reports and other factors.

    As for the brokers and large sites selling links traffic, there is no apparent reason to lower the prices, because these primitive paid links didn’t pass any juice for ages now. Not that the link buyers will get that. It’s way easier to live with fictions and shiny expectations than reality and hard facts. Also, lets not forget the vast majority of link sellers as well as link buyers who trade links and don’t bother reading Google’s messages, SEO blogs, or Webmaster forums. It will be interesting to watch the market …

  11. […] here is a pragmatic answer to Google’s paid links requirements: Cloak the hell out of your links with commercial intent! Share/bookmark this: del.icio.us • Google • ma.gnolia • Netscape • reddit […]

  12. […] no following links. That is, making the paid links look like they “follow” except when viewed by Google’s spiders. These type of links would be called “cloaked […]

  13. The day the routers died on 31 October, 2007  #link

    […] whilst Matt’s gang in building 43 intones “No mercy, smack paid links, no place to hide for TLA links” … Enjoy this video, it’s friggin’ […]

  14. 5ubliminal on 31 October, 2007  #link

    Such a fuss over virtually nothing. Any link resulting over a financial transaction should not pass any link juice and Google is entitled to nuke the hell of those who offer paid reviews.
    Wtf is a paid review? Before I got to know the blogging community I had no idea of such thing. Paid reviews can’t be honest.
    Any review which does not involve money can be considered sincere but any paid review has doubt written all over it.
    If I pay you and you say cr4p about my product I send the big boyz for you. It’s the way it works. One ain’t gonna ruin my product’s reputation paid by myself.

    So one of the few things I agree with Google upon is the retirement of link-PIMPs. Remove the green pixels from their toolbars and let them keep selling links.

    PS: No follow the link on my comment ;)

    [Dunno why, but here’s your nofollow.]

  15. […] offers his idea for keeping Google’s paid-link police at arm’s […]

  16. […] A pragmatic defence against Google’s anti paid links campaign […]

  17. Gerry on 3 November, 2007  #link

    Sebastian,

    I do not understand exactly how you do it but here is what happened.

    I searched [dog comment “rel=nofollow”] on google and this page appears first. However there is no “dog” in this page. The only place I found this word is in the Referrer URL on this page.

    I still wonder how this page showed up first in Google search… but I’ll add your site to my links in the next few hours.

  18. Sebastian on 3 November, 2007  #link

    Context is a ranking factor as well as anchor text. Google’s algo is a sly old dog. ;)

  19. Magic on 4 March, 2008  #link

    This is an brilliant article, and very clever. The set back though is that google could consider this like an invisible link (as this technique is showing the googlebot different content to that of the user). If they find out, the webmaster using this technique could be penalised.

  20. Sebastian on 4 March, 2008  #link

    Nope Magic, that’s not deceitful cloaking, hence not penalized by Google.

Leave a reply


[If you don't do the math, or the answer is wrong, you'd better have saved your comment before hitting submit. Here is why.]

Be nice and feel free to link out when a link adds value to your comment. More in my comment policy.