Below is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it. The working theory is, that all crawlers except Ms. Googlebot ignore the robots.txt statement above. Whether the possibly crawled page /repstuff/noindex.php linking here passed the indexer or not can be checked by requesting Google's cache of the referring page. Whether this page made it in the search index via the link on /repstuff/noindex.php or not reveals this link to Google's cached copy of this page. Perhaps all this leads to nothing but fun. We'll see. Here is the log file:
| IP Address | TimeStamp | Response | UserAgent | Requested URI |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) | /repstuff/noindex-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Googlebot/2.1 (+http://www.google.com/bot.html) | /repstuff/noindex-linked.php |
| 66.249.72.209 | 2007-11-21 14:57:19 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2007-11-29 20:31:22 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2007-12-06 23:48:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2007-12-14 05:32:41 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2007-12-23 11:53:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2007-12-30 20:16:42 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-01-07 08:25:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.212.243 | 2008-01-13 18:30:13 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-01-14 11:13:20 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-01-22 00:05:12 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-01-26 05:39:40 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-01-29 00:07:39 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-01-30 02:59:27 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-02-05 01:06:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-02-12 01:08:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-02-17 07:31:14 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-02-19 01:12:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-02-26 01:14:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.23.23 | 2008-03-04 11:51:48 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-03-05 10:21:52 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-06 05:52:04 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-07 16:03:43 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-09 18:50:19 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-13 14:31:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-17 16:43:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-23 02:25:32 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-03-23 05:09:55 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-27 05:55:12 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-03-29 06:47:40 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-01 03:45:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-04-04 08:54:54 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-04 17:14:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.209.146 | 2008-04-09 07:17:55 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-09 13:33:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-04-11 16:58:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-16 01:09:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-20 04:03:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-23 19:06:43 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-04-25 17:08:13 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-04-27 14:42:05 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-02 02:14:39 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-05-02 17:08:12 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 65.55.212.242 | 2008-05-03 08:47:46 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-05 11:47:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-06 11:48:51 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-08 12:38:08 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.115 | 2008-05-09 11:50:41 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-05-10 00:46:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-11 16:46:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-12 16:49:45 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-14 17:54:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-05-17 00:46:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-18 15:03:51 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-23 04:40:49 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-05-24 00:49:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.27.152 | 2008-05-27 04:28:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.105 | 2008-05-28 02:14:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.8.98 | 2008-05-30 17:20:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-05-31 01:33:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.105 | 2008-06-04 07:38:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.22.87 | 2008-06-05 03:32:28 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.52.105 | 2008-06-07 02:05:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.17.166 | 2008-06-09 00:40:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.17.166 | 2008-06-13 11:21:04 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.8.105 | 2008-06-18 08:19:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.105 | 2008-06-18 17:27:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.54.27 | 2008-06-19 06:58:53 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.17.155 | 2008-06-25 08:20:27 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.17.173 | 2008-06-29 04:42:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.165 | 2008-07-03 02:55:09 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.17.173 | 2008-07-06 04:43:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.54.27 | 2008-07-09 07:10:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.22.105 | 2008-07-13 04:45:42 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 74.6.22.105 | 2008-07-19 21:43:59 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
| 67.195.37.165 | 2008-07-23 09:46:38 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-linked.php |
The first five entries are samples. An empty table wouldn't look nice.