Below is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it. The working theory is, that all crawlers except Ms. Googlebot ignore the robots.txt statement above. Whether the possibly crawled page /repstuff/noindex-nofollow.php linking here passed the indexer or not can be checked by requesting Google's cache of the referring page. Whether this page made it in the search index via the link on /repstuff/noindex-nofollow.php or not reveals this link to Google's cached copy of this page. Perhaps all this leads to nothing but fun. We'll see. Here is the log file:
| IP Address | TimeStamp | Response | UserAgent | Requested URI |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) | /repstuff/noindex-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Googlebot/2.1 (+http://www.google.com/bot.html) | /repstuff/noindex-nofollow-linked.php |
| 66.249.72.209 | 2007-11-21 14:59:11 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2007-11-28 15:21:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2007-12-02 01:09:08 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2007-12-04 05:44:23 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2007-12-05 16:54:30 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.145 | 2007-12-06 09:24:17 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2007-12-20 02:51:58 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2007-12-23 12:09:09 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2007-12-27 06:05:48 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-01-03 15:27:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-01-09 15:58:15 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-01-10 15:31:52 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-01-17 15:37:47 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-01-24 15:38:33 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-01-31 15:39:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-02-05 10:41:51 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-02-07 13:57:44 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.23.168 | 2008-02-07 15:40:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-02-13 15:21:22 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-02-20 15:23:43 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-02-26 21:21:55 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-02-27 15:25:53 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-03-08 06:46:46 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-03-15 01:21:34 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-03-15 16:33:09 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-03-22 23:07:08 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-03-30 03:54:05 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.209.146 | 2008-04-01 02:16:08 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-04-04 07:04:28 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-04-08 05:13:05 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-04-11 09:03:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.91 | 2008-04-15 13:11:02 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-04-17 10:36:20 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-04-18 09:04:59 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-04-21 15:17:00 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-04-25 09:06:41 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-04-27 19:21:22 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-04-30 13:33:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-05-02 09:08:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.115 | 2008-05-06 04:27:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-05-07 13:34:16 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-05-09 09:12:50 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.115 | 2008-05-13 10:07:17 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-05-14 13:37:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-05-16 09:13:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.25.166 | 2008-05-18 05:12:44 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.105 | 2008-05-20 10:13:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.22.121 | 2008-05-22 14:13:11 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.22.121 | 2008-05-26 11:26:38 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.105 | 2008-05-27 10:22:15 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-05-28 10:59:54 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.22.90 | 2008-05-30 05:34:34 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.8.98 | 2008-06-03 06:19:14 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.105 | 2008-06-04 05:56:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-06-04 11:00:54 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-06-11 11:01:16 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-06-18 17:53:36 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 65.55.212.242 | 2008-06-22 16:02:48 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.165 | 2008-06-24 08:07:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.17.155 | 2008-06-24 18:07:02 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.165 | 2008-07-03 21:42:27 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.52.111 | 2008-07-08 17:52:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 67.195.37.165 | 2008-07-15 20:39:58 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
| 74.6.22.105 | 2008-07-18 14:31:42 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/noindex-nofollow-linked.php |
The first five entries are samples. An empty table wouldn't look nice.