Below is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it. The working theory is, that all crawlers except Ms. Googlebot ignore the robots.txt statement above. Whether the possibly crawled page /repstuff/noindex-follow.php linking here passed the indexer or not can be checked by requesting Google's cache of the referring page. Whether this page made it in the search index via the link on /repstuff/noindex-follow.php or not reveals this link to Google's cached copy of this page. Perhaps all this leads to nothing but fun. We'll see. Here is the log file:
| IP Address | TimeStamp | Response | UserAgent | Requested URI |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-follow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) | /repstuff/noindex-follow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Googlebot/2.1 (+http://www.google.com/bot.html) | /repstuff/noindex-follow-linked.php |
| 66.249.72.209 | 2007-11-21 14:56:19 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-11-23 17:42:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-11-30 18:03:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2007-12-03 11:23:29 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2007-12-05 18:17:55 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-12-07 18:03:45 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-12-14 18:05:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2007-12-22 20:06:29 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-12-23 01:31:45 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2007-12-30 01:41:38 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-01-06 02:55:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-01-08 23:41:25 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 65.55.212.244 | 2008-01-11 03:36:21 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-01-13 02:55:28 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-01-20 02:57:58 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-01-27 03:00:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-02-03 03:00:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-02-05 08:58:45 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-02-10 03:01:26 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-02-17 04:48:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-02-24 05:03:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-02-26 19:49:11 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.22.229 | 2008-03-02 05:05:50 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-03-11 11:08:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-03-15 12:24:17 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-03-18 07:10:28 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-03-23 05:37:16 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-03-30 05:38:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-04-02 07:48:46 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-02 21:04:54 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-04-04 21:14:19 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.91 | 2008-04-08 15:53:26 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-09 01:10:26 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-04-11 21:23:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-13 20:05:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-17 19:38:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-04-18 21:23:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.209.145 | 2008-04-19 11:01:35 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-22 12:15:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-04-25 21:26:17 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-26 14:52:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-04-29 04:31:28 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.91 | 2008-04-29 22:14:55 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.91 | 2008-05-01 20:16:45 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-05-02 21:32:59 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-02 23:44:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-06 12:37:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.115 | 2008-05-09 01:07:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-09 12:53:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-05-09 21:33:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-12 20:57:17 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.105 | 2008-05-16 01:26:39 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-05-16 21:36:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-16 23:39:16 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-21 12:02:54 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-05-23 21:37:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.11.217 | 2008-05-26 18:18:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.8.98 | 2008-05-30 17:11:14 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-05-30 21:41:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.8.98 | 2008-06-03 14:09:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.87 | 2008-06-05 17:39:53 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-06-06 21:42:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.17.166 | 2008-06-09 23:51:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.105 | 2008-06-12 01:51:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-06-13 21:43:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.8.105 | 2008-06-14 12:17:22 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.8.105 | 2008-06-19 14:04:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 65.55.212.242 | 2008-06-19 16:38:47 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-06-20 21:44:07 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.17.173 | 2008-06-26 05:42:47 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.17.173 | 2008-07-02 08:44:07 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.17.173 | 2008-07-07 10:10:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.52.84 | 2008-07-10 21:41:41 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.105 | 2008-07-14 10:42:55 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.105 | 2008-07-18 23:36:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 74.6.22.105 | 2008-07-23 07:36:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
| 67.195.37.165 | 2008-07-23 10:06:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-follow-linked.php |
The first five entries are samples. An empty table wouldn't look nice.