Below is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it. The working theory is, that all crawlers except Ms. Googlebot ignore the robots.txt statement above. Whether the possibly crawled page /repstuff/index-nofollow.php linking here passed the indexer or not can be checked by requesting Google's cache of the referring page. Whether this page made it in the search index via the link on /repstuff/index-nofollow.php or not reveals this link to Google's cached copy of this page. Perhaps all this leads to nothing but fun. We'll see. Here is the log file:
| IP Address | TimeStamp | Response | UserAgent | Requested URI |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) | /repstuff/index-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Googlebot/2.1 (+http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 66.249.72.209 | 2007-11-21 14:56:58 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2007-11-28 21:54:49 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2007-12-06 18:03:42 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.72.209 | 2007-12-07 06:21:10 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2007-12-13 18:06:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2007-12-20 18:07:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2007-12-27 18:07:48 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-01-04 00:09:04 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-01-11 00:14:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-01-18 00:36:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.72.5 | 2008-01-24 08:12:21 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-01-25 00:37:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-01-28 06:32:50 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 66.249.72.5 | 2008-01-29 19:26:10 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-01-31 12:24:16 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-02-01 12:11:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-02-03 05:32:37 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-02-06 09:34:35 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 66.249.72.5 | 2008-02-07 06:51:31 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-02-08 12:19:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.72.5 | 2008-02-13 08:48:35 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 66.249.72.5 | 2008-02-14 20:33:25 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-02-15 12:21:02 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-02-22 20:38:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.72.74 | 2008-02-25 23:04:46 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-02-29 20:57:10 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.72.74 | 2008-03-03 10:39:49 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-03-05 04:20:17 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 66.249.66.239 | 2008-03-11 12:20:20 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-12 00:31:15 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-13 01:29:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-15 07:18:07 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-18 08:19:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-03-22 16:56:07 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-23 09:45:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.66.236 | 2008-03-28 08:04:00 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-03-30 09:47:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-04-04 22:20:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.66.44 | 2008-04-05 05:19:57 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-04-07 04:58:40 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-04-08 06:43:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 65.55.209.149 | 2008-04-08 21:27:58 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-04-12 05:29:32 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.66.44 | 2008-04-14 00:50:20 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 74.6.27.100 | 2008-04-15 06:51:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 67.195.37.91 | 2008-04-15 13:25:13 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-04-19 05:32:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.121 | 2008-04-20 12:15:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.121 | 2008-04-21 16:11:38 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.67.234 | 2008-04-25 03:01:54 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-04-26 05:34:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 67.195.37.91 | 2008-05-03 05:28:13 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-05-03 10:21:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.121 | 2008-05-05 19:08:19 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.70.170 | 2008-05-08 01:52:39 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 67.195.37.115 | 2008-05-10 05:28:49 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.121 | 2008-05-10 10:21:32 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 66.249.67.165 | 2008-05-10 17:37:55 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/index-nofollow-linked.php |
| 67.195.51.152 | 2008-05-12 07:12:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 74.6.27.121 | 2008-05-14 19:52:26 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
| 65.55.212.242 | 2008-05-15 05:32:54 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/index-nofollow-linked.php |
| 67.195.37.105 | 2008-05-17 05:33:59 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/index-nofollow-linked.php |
The first five entries are samples. An empty table wouldn't look nice.