# robots.txt EXAMPLE, # see: http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/ User-agent: Mediapartners-Google Disallow: Allow: / Sitemap: http://sebastians-pamphlets.com/repstuff/repstuff-sitemap.txt # UA string: CCBot/1.0 (+http://www.commoncrawl.org/bot.html)