Google says you must manage your affiliate links in order to get indexed

Screwing affiliates recommended by Google ;=)I’ve worked hard to overtake the SERP positions of a couple merchants allowing me to link to them with an affiliate ID, and now the allmighty Google tells the sponsors they must screw me with internal 301 redirects to rescue their rankings. Bugger. Since I read the shocking news on Google’s official Webmaster blog this morning I worked on a counter strategy, with success. Affiliate programs will not screw me, not even with Google’s help. They’ll be hoist by their own petard. I’ll strike back with nofollow and I’ll take no prisoners.

Seriously, the story reads a little different and is not breaking news at all. Maile Ohye from Google just endorsed best practices I’ve recommended for ages. Here is my recap.

The problem

Actually, there are problems on both sides of an affiliate link. The affiliate needs to hide these links from Google to avoid a so called “thin affiliate site penalty”, and the affiliate program suffers from duplicate content issues, link juice dilution, and often even URL hijacking by affiliate links.

Diligent affiliates gathering tons of PageRank on their pages can “unintentionally” overtake URLs on the SERPs by fooling the canonicalization algos. When Google discovers lots of links from strong pages on different hosts pointing to http://sponsor.com/?affid=me and this page adds ?affid=me to its internal links, my URL on the sponsor’s site can “outrank” the official home page, or landing page, http://sponsor.com/. When I choose the right anchor text, Google will feed my affiliate page with free traffic, whilst the affiliate program’s very own pages don’t exist on the SERPs.

Managing incoming affiliate links (merchants)

The best procedure is capturing all incoming traffic before a single byte of content is sent to the user agent, extracting the affiliate ID from the URL, storing it in a cookie, then 301-redirecting the user agent to the canonical version of the landing page, that is a page without affiliate or user specific parameters in the URL. That goes for all user agents (humans accepting the cookie and Web robots which don’t accept cookies and start a new session with every request).

Users not accepting cookies are redirected to a version of the landing page blocked by robots.txt, the affiliate ID sticks with the URLs in this case. Search engine crawlers, identified by their user agent name or whatever, are treated as users and shall never see (internal) links to URLs with tracking parameters in the query string.

This 301 redirect passes all the link juice, that is PageRank & Co. as well as anchor text, to the canonical URL. Search engines can no longer index page versions owned by affiliates. (This procedure doesn’t prevent you from 302 hijacking where your content gets indexed under the affiliate’s URL.)

Putting safe affiliate links (online marketers)

Honestly, there’s no such thing as a safe affiliate link, at least not safe with regard to picky search engines. Masking complex URLs with redirect services like tinyurl.com or so doesn’t help, because the crawlers get the real URL from the redirect header and will leave a note in the record of the original link on the page carrying the affiliate link. Anyways, the tiny URL will fool most visitors, and if you own the redirect service it makes managing affiliate links easier.

Of course you can cloak the hell out of your thin affiliate pages by showing the engines links to authority pages whilst humans get the ads, but then better forget the Google traffic (I know, I know … cloaking still works if you can handle it properly, but not everybody can handle the risks so better leave that to the experts).

There’s only one official approach to make a page plastered with affiliate links safe with search engines: replace it with a content rich page, of course Google wants unique and compelling content and checks its uniqueness, then sensibly work in the commercial links. Best link within the content to the merchants, apply rel-nofollow to all affiliate links, and avoid banner farms in the sidebars and above the fold.

Update: I’ve sanitized the title, “Google says you must screw your affiliates in order to get indexed” was not one of my best title baits.



Share/bookmark this: del.icio.usGooglema.gnoliaMixxNetscaperedditSphinnSquidooStumbleUponYahoo MyWeb
Subscribe to      Entries Entries      Comments Comments      All Comments All Comments
 

23 Comments to "Google says you must manage your affiliate links in order to get indexed"

  1. JLH on 12 September, 2007  #link

    The only saving grace of this is that anyone slick enough to implement this has probably already done it and doesn’t need Google to tell them to, on the other hand only people in the know probably read the google webmaster blog anyway.

    I kind of wonder how much of their message actually gets to the general public. If Matt Cutts said on his blog to have “asjklasdfjklajkl” in the bottom right hand of each page in order to be indexed quickly, how many sites would do that? A small percentage I’d bet. The same fun test could be done with Google’s blog, webmaster’s group, or even searchenginelandmozworldtable…

  2. Sebastian on 12 September, 2007  #link

    Yep, I figured writing about “screwing affiliates” could help to get the word out.

  3. john andrews on 12 September, 2007  #link

    There are TONS of affiliate links on hosted blog services where the user has no access server side, and cannot properly redirect or cloak or robots.txt or noindex. Those are obviously “expendible” players, and this trimming action aids the commercial entities quite a bit.

    Aside from the legitimate players, I think this is more power to the capable spammers and will hurt the web overall.

  4. Sebastian on 12 September, 2007  #link

    Adding affiliate links on free hosts of any kind usually allows throwing rel-nofollow crap on all links not inserted by the host. Replacing the URLs by tiny-URLs is also possible, and clever free hosted marketers own a redirect service or ad server capable to handle crawler requests to their advantage. Hence both legitimate players as well as folks playing on the shady side can make use of free hosts to spread affiliate links.

    The trimming takes place at the affiliate program’s site, and I’ve not yet seen a legit free hosted affiliate program.

    As for the capable spammers, they’re way ahead of trivial things like that.

  5. Matt Cutts on 12 September, 2007  #link

    “Seriously, the story reads a little different and is not breaking news at all. Maile Ohye from Google just endorsed best practices I’ve recommended for ages.”

    Sebastian, don’t scare me with headlines like that. :)

  6. Chris from Detroit on 13 September, 2007  #link

    I’m not very knowledgeable about Robots.txt but could you robot.txt some of these websites?

  7. Sebastian on 13 September, 2007  #link

    Oops Matt. If it works with you then it will reach the merchants and their affiliates. ;)

  8. Sebastian on 13 September, 2007  #link

    Chris, as much as I’d love to deindex a few sites on the Web that’s not the way the cookie crumbles. The robots.txt file is controlled by each site’s Webmaster and tells search engine crawlers how they have to behave. Since I’ve no access to holymoleephoto.com for example, I can’t robots.txt you out of the game. If you’re interested in learning more about robots.txt and other crawler directives, then feel free to browse these categories here: robots.txt, robots meta tags, x-robots-tags, microformats and crawler directives.

  9. JLH on 13 September, 2007  #link

    Here’s my simplistic interpretation.

    Back in 2005 a site of mine that has affiliates that link to it with the ?partner=something method used to have duplicate pages shown in the index all the time, then sometimes the affiliate linked page would out rank the base page.

    Back to today…now the same site has the base URLs listed only and the affiliate links are gone. Even for deeper product pages where the only links are internal or from affiliates.

    With this in mind, I tend to believe that what Maile said,

    “1. When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.

    2. We select what we think is the “best” URL to represent the cluster in search results.

    3. We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.”

    Is very true and for a well indexed site implementing the 301 redirect will only server to help Google in this discovery. Perhaps when a new affiliate signs up and links to some new pages stripping the ID and 301 will speed up Google’s understanding of the site structure.

    In the long run I think the effect is neutral as Google is already consolidating URLs for sites.

    As far as outranking your merchants, I’d imagine with your knowledge you can do that just fine. Most merchants are experts in selling widgets and only look at the web as a publishing medium, if they have any search assistance it’s part time, and only used when there is a problem. You on the other hand more than likely are adapting daily to the changes. In order for the 301′d and non-301′d link to have any effect on your ranking versus their would lead us down a path where linking can be used to harm a sites ranking, which I know we aren’t going to go down.

    You could say that by linking to an affiliate link is a way of not boosting the target base page, but as I said earlier, I think Google is already consolidating them. So you truly are outranking them already with your link to the page. If you want to punish your merchant, then the nofollow is a way of pulling that link juice. I just don’t see you doing that though. :)

  10. Sebastian on 13 September, 2007  #link

    I didn’t intend to doubt Maile’s helpful post. Google does a pretty awesome job with their canonicalization. I think that in most cases they choose the right canonical URL. Doh … “right canonical” … However, I think helping the search engines to get it right without educated guessing cannot be a bad thing. And on the other side condomizing affiliate links is what Google wants us to do. Although that’s pretty redundant because Google can identify most affiliate links without rel=”advertising nofollow” hints from the Webmaster. As for the first paragraph, you’ll find background info in the archives of a Googler we both love.

  11. […] them, and some of the ones I found aren’t real recent. But when people like Everett Sizemore, Sebastian, Joost de Valk, Hamlet Batista, John Chow, Shoemoney, and Dave Taylor all think that the subject […]

  12. Heapseo » I love Sphinn on 13 September, 2007  #link

    […] Google says you must screw your affiliates in order to get indexed (sebastians-pamphlets.com) […]

  13. […] Ok, that’s a geeky issue and you don’t need to take it very seriously. Request your BlogRush affiliate link with a plain user agent not accepting cookies or executing client sided scripting, then read the headers. BlogRush does a 302 redirect to their home page rescuing your affiliate ID in an unclosed base href directive. Chances are you’ll never get the promised credits from upsold visitors using uncommon user agents respectively browser settings, because they don’t manage their affiliate traffic properly. […]

  14. Melanie Phung on 19 September, 2007  #link

    Good post Sebastian.

    Let’s say I hypothetically work as an in-house SEO for an e-commerce company. This company does a very significant amount of volume via partners and affiliates. I spent at least a year and a half begging, pleading, ranting about the need to strip tracking id’s out of our URLs and doing a 301 to the canonical version, but my puppet masters always come back with “but some people don’t accept cookies and we have to make sure the sale gets credited.”

    So the fact that we put a ton of work into creating this brilliantly SEO’d site and our affiliates are ranking on Page 1 for competitive keyword terms with OUR page, while the non-affiliate version of the page is in Supplemental Hell, doesn’t bother you?

    “Your problem, not ours”

    Btw: we pay our affiliates up to $100 per sale. You’d think this would make it someone’s problem besides just mine.

    So, if you’re an affiliate of my company’s and you suspect we’ve been trying to screw you out of commissions… rest assured, to my big distress and against my SEO-biased advice, we’ve never ever done this. Hope you’re happy puny little affiliates because you made my life hell. Hypothetically.

    But of course, at some point during the summer, as if by magic, Google seemed to have figured it out and the free ride was over.

  15. Melanie Phung on 19 September, 2007  #link

    FWIW: Yahoo never seemed to have a problem understand the deal with our affiliate IDs. It was only Google that had me pulling my hair out over those damn tracking variables.

  16. Melanie Phung on 19 September, 2007  #link

    understanding

  17. Sebastian on 19 September, 2007  #link

    Thanks Melanie, that’s the point exactly. Google has improved the canonicalization esp. with affiliate links, and now that they see they’re on track they recommend that merchants fine tune their affiliate traffic management, what’s a sensible thing to do because besides Google and Yahoo there are other not that smart search engines out there. By the way, as an affiliate I’m not really offended when I lose rankings for pages on a merchants site, because it wasn’t fair in the first place.

  18. Melanie Phung on 19 September, 2007  #link

    Search engines besides Google? I do not know this thing of which you speak.

    ps. arrr!

  19. […] his honest reviews which Google classifies as paid links, and of course all sorts of traffic deals, affiliate links, banner ads and stuff like […]

  20. Sebastian on 2 November, 2007  #link

    If you’re after nice early bird discounts get your pass for SMX West today (Santa Clara, CA, February/26-28/2008).

    I don’t broadcast affiliate links. This is a test for some fancy things search engine crawlers should face when they try to crawl the URL. Of course it’s a good idea to book early, so the link isn’t completely worthless.

  21. […] = document.getElementById(domid); if (handle) handle.style.display = “none”; } My recent posts on managing affiliate links and nofollow cloaking paid links led to so many reactions from my readers that I thought explaining […]

  22. Corporate SEO Services on 21 September, 2009  #link

    document.getElementById(domid);
    does it still works in 2009 ?

    [You comment spammers don’t really believe that JavaScript syntax gets changed each new year’s day, eh?]

  23. Al Sefati on 8 November, 2010  #link

    hey thanks for the post. It seems that you are talking more as a publisher but what if you are an advertiser that pays for affiliate links. Should I worry that Google may ding me for buying links if affiliate doesn’t put nofollow on their links?

    Thanks

Leave a reply


[If you don't do the math, or the answer is wrong, you'd better have saved your comment before hitting submit. Here is why.]

Be nice and feel free to link out when a link adds value to your comment. More in my comment policy.