Comments on: Advantages of a smart robots.txt file http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/ If you've read my articles somewhere on the Internet, expect something different here. Sat, 17 May 2008 03:30:57 +0000 http://wordpress.org/?v=2.2.3 By: Save bandwidth costs: Dynamic pages can support If-Modified-Since too http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-1169 Save bandwidth costs: Dynamic pages can support If-Modified-Since too Tue, 19 Feb 2008 19:35:57 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-1169 [...] can test your Web server’s conditional GET support with your robots.txt, or, if even your robots.txt is a script, create a tiny HTML page with a text editor and upload it via FTP. Another neat tool to check HTTP [...]

[…] can test your Web server’s conditional GET support with your robots.txt, or, if even your robots.txt is a script, create a tiny HTML page with a text editor and upload it via FTP. Another neat tool to check HTTP […]

]]> By: Zak Nicola http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-797 Zak Nicola Tue, 11 Dec 2007 12:59:49 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-797 Thanks for all the great info, in the comments as well (g1smd). I've made some mistakes with my own robots.txt in the past that, had I read this before... well, no use in crying over spilled milk now. [Author link removed coz it points to an empty page] Thanks for all the great info, in the comments as well (g1smd).

I’ve made some mistakes with my own robots.txt in the past that, had I read this before… well, no use in crying over spilled milk now.

[Author link removed coz it points to an empty page]

]]> By: Utah SEO Pro http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-776 Utah SEO Pro Sun, 02 Dec 2007 20:18:39 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-776 Sebastian, I appreciate you addressing my question. Thank you for this post. I'm going to try to implement this strategy tonight. Sebastian, I appreciate you addressing my question. Thank you for this post. I’m going to try to implement this strategy tonight.

]]> By: Andy http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-760 Andy Wed, 28 Nov 2007 05:38:24 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-760 Interesting post. There's plenty to think about there. I've bookmarked it on del.icio.us for future reference. Interesting post. There’s plenty to think about there. I’ve bookmarked it on del.icio.us for future reference.

]]> By: Sebastian http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-758 Sebastian Tue, 27 Nov 2007 13:03:08 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-758 I've clients with Windows boxes too, you don't try to convince them to switch to *ix when some AppServers, databases and stuff like that can't run on Unix boxes, or when their staff isn't capable to operate non-Windows stuff. ;) I don't agree that *ix/Apache is for entry level only. I know a few pretty decent large scale systems running under Apache on Linux or BSD servers. I’ve clients with Windows boxes too, you don’t try to convince them to switch to *ix when some AppServers, databases and stuff like that can’t run on Unix boxes, or when their staff isn’t capable to operate non-Windows stuff. ;)

I don’t agree that *ix/Apache is for entry level only. I know a few pretty decent large scale systems running under Apache on Linux or BSD servers.

]]> By: Jeremy Luebke http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-757 Jeremy Luebke Tue, 27 Nov 2007 12:07:28 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-757 Actually it's getting easier. The only real issues with IIS has been SE friendly URLs and redirects along with documentation for SEO using IIS & .Net. That's changing as there are some great rewrite packages out now for free or fee. There is also an amazing CMS framework in Umbraco thats SE frindly out of the box. That said, windows hosting / program is suited more for enterprise level situations. Virtual hosting just doesn't work, you need at least a VPS or dedicated server to get started so the barrier for entry is much higher and that's the reason most small webmasters start out with *nix/Apache. Actually it’s getting easier. The only real issues with IIS has been SE friendly URLs and redirects along with documentation for SEO using IIS & .Net. That’s changing as there are some great rewrite packages out now for free or fee. There is also an amazing CMS framework in Umbraco thats SE frindly out of the box.

That said, windows hosting / program is suited more for enterprise level situations. Virtual hosting just doesn’t work, you need at least a VPS or dedicated server to get started so the barrier for entry is much higher and that’s the reason most small webmasters start out with *nix/Apache.

]]> By: Sebastian http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-753 Sebastian Tue, 27 Nov 2007 05:05:58 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-753 Jeremy, I believe that a developer forced to work with IIS/ASP/.NET must be extremely smart, because there are so many pitfalls (case issues, lack of request_uri, canonicalization nightmares ... just to name a few). It's easier with Apache under Windows, though. However, from years of Webmaster/Web developer support I've learned that it's better not to run Web sites under Windows. I know, sometimes it's unavoidable, but IMHO it's not recommendable in general. Jeremy, I believe that a developer forced to work with IIS/ASP/.NET must be extremely smart, because there are so many pitfalls (case issues, lack of request_uri, canonicalization nightmares … just to name a few). It’s easier with Apache under Windows, though. However, from years of Webmaster/Web developer support I’ve learned that it’s better not to run Web sites under Windows. I know, sometimes it’s unavoidable, but IMHO it’s not recommendable in general.

]]> By: Tony Adam http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-751 Tony Adam Tue, 27 Nov 2007 02:20:18 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-751 Great post explaining how to put together a detailed robots.txt. I've del.icio.us'ed this in case I need it in the future! Great post explaining how to put together a detailed robots.txt. I’ve del.icio.us’ed this in case I need it in the future!

]]> By: Jeremy Luebke http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-750 Jeremy Luebke Mon, 26 Nov 2007 23:46:32 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-750 I know your not ignorant so please don't say ignorant things. "Here is my step by step “how to create a smart robots.txt” guide. As always: if you suffer from IIS/ASP go search for reliable hosting (*ix/Apache)." Some people are smart enough to program using the IIS/.NET platform ;) I know your not ignorant so please don’t say ignorant things.

“Here is my step by step “how to create a smart robots.txt” guide. As always: if you suffer from IIS/ASP go search for reliable hosting (*ix/Apache).”

Some people are smart enough to program using the IIS/.NET platform ;)

]]> By: Search Engine Land: News About Search Engines & Search Marketing http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-749 Search Engine Land: News About Search Engines & Search Marketing Mon, 26 Nov 2007 19:01:48 +0000 http://sebastians-pamphlets.com/cloak-the-hell-out-of-your-robots-txt/#comment-749 <strong>SearchCap: The Day In Search, November 26, 2007...</strong> Below is what happened in search today, as reported on Search Engine Land and from other places across the web....... SearchCap: The Day In Search, November 26, 2007…

Below is what happened in search today, as reported on Search Engine Land and from other places across the web…….

]]>