All Posts Tagged 'Crawler-directives'
- List all categories with links to all posts
- CRAWLER DIRECTIVES
Telling search engine crawlers how they have to behave. Not all crawler directives are standardized in the REP (robots exclusion protocol). Crap like rel-nofollow for example is defined as the Web’s plague. - Crawler Directives
26 posts.
- Get yourself a smart robots.txt (22 comments)
- How do Majestic and LinkScape get their raw data? (56 comments)
- How brain-amputated developers created the social media plague (10 comments)
- URI canonicalization with an X-Canonical-URI HTTP header (11 comments)
- The perfect robots.txt for News Corp (7 comments)
- Debugging robots.txt with Google Webmaster Tools (2 comments)
- Vaporize yourself before Google burns your linking power (18 comments)
- Crawling vs. Indexing (15 comments)
- @ALL: Give Google your feedback on NOINDEX, but read this pamphlet beforehand! (6 comments)
- Update your crawler detection: MSN/Live Search announces msnbot/1.1 (7 comments)
- Get a grip on the Robots Exclusion Protocol (REP) (2 comments)
- Getting URLs outta Google - the good, the popular, and the definitive way (26 comments)
- My plea to Google - Please sanitize your REP revamps (21 comments)
- Google to change the Robots Exclusion Protocol again (8 comments)
- Validate your robots.txt - Googlebot becomes smarter (12 comments)
- Microsoft funding bankrupt Live Search experiment with porn spam (9 comments)
- Q&A: An undocumented robots.txt crawler directive from Google (20 comments)
- Act out your sophisticated affiliate link paranoia (11 comments)
- A pragmatic defence against Google’s anti paid links campaign (20 comments)
- Google and Yahoo accept undelayed meta refreshs as 301 redirects (25 comments)
- NOPREVIEW - The missing X-Robots-Tag (4 comments)
- Handling Google’s neat X-Robots-Tag - Sending REP header tags with PHP (12 comments)
- Unavailable_After is totally and utterly useless (6 comments)
- Is XML Sitemap Autodiscovery for Everyone? (2 comments)
- In need of a "Web-Robot Directives Standard" (2 comments)
- XML sitemap auto-discovery
|
Share/bookmark this: del.icio.us • Google • ma.gnolia • Mixx • Netscape • reddit • Sphinn • Squidoo • StumbleUpon • Yahoo MyWeb Subscribe to  Entries Comments All Comments
|