# robots.txt EXAMPLE, # see: http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/ User-agent: Googlebot Disallow: Allow: / Sitemap: http://sebastians-pamphlets.com/repstuff/repstuff-sitemap.txt # UA string: CCBot/2.0 (https://commoncrawl.org/faq/)