Below is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it. The working theory is, that all crawlers except Ms. Googlebot ignore the robots.txt statement above. Whether the possibly crawled page /repstuff/nofollow.php linking here passed the indexer or not can be checked by requesting Google's cache of the referring page. Whether this page made it in the search index via the link on /repstuff/nofollow.php or not reveals this link to Google's cached copy of this page. Perhaps all this leads to nothing but fun. We'll see. Here is the log file:
| IP Address | TimeStamp | Response | UserAgent | Requested URI |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html) | /repstuff/nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| Faked | 2007-11-21 00:00:01 | 200-OK | Googlebot/2.1 (+http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2007-11-28 17:03:48 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2007-12-05 17:05:40 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.209 | 2007-12-07 06:45:48 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2007-12-12 19:12:08 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2007-12-19 19:48:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.209 | 2007-12-21 21:09:41 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2007-12-26 19:49:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.164 | 2007-12-28 00:43:47 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-01-03 13:51:32 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.164 | 2008-01-03 21:15:09 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 66.249.66.164 | 2008-01-09 11:09:59 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-01-10 17:58:58 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 65.55.212.244 | 2008-01-11 01:27:17 | 200-OK | msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 66.249.67.236 | 2008-01-15 00:32:33 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-01-17 18:01:18 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-01-20 12:44:23 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-01-24 18:02:23 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-01-26 03:19:22 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-01-31 18:03:58 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-01-31 20:37:48 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-02-06 22:34:01 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-02-07 18:07:09 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-02-09 00:45:05 | 200-OK | msnbot/1.0 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-02-09 02:53:03 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-02-14 20:31:37 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-02-15 00:50:21 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.5 | 2008-02-21 00:26:56 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-02-22 00:53:24 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.74 | 2008-02-28 10:08:28 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-02-28 23:53:35 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-02-29 00:54:52 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.72.74 | 2008-03-05 19:44:14 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-03-09 10:28:55 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.239 | 2008-03-13 20:56:13 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-03-17 04:03:01 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-03-17 04:51:02 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.239 | 2008-03-19 14:17:03 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-03-24 04:52:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-03-31 04:58:15 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.44 | 2008-04-01 03:00:26 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-02 05:50:50 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-04-03 05:38:18 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-03 05:57:40 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-04-04 19:06:37 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.44 | 2008-04-08 06:25:48 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-04-09 10:31:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-10 06:03:29 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-11 12:27:25 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-13 13:51:45 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.37.91 | 2008-04-16 01:08:10 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-04-16 10:32:32 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.66.44 | 2008-04-17 23:33:48 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 65.55.209.148 | 2008-04-20 07:54:22 | 200-OK | msnbot/1.1 (+http://search.msn.com/msnbot.htm) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-21 00:44:53 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-04-24 00:51:57 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.67.91 | 2008-04-27 19:43:52 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-04-28 00:46:20 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.37.91 | 2008-04-30 01:53:15 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.37.91 | 2008-04-30 03:20:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-05-01 04:20:55 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-05-03 08:09:03 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.73.84 | 2008-05-05 10:29:03 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 67.195.37.115 | 2008-05-07 03:26:02 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-05-07 14:17:39 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 66.249.70.170 | 2008-05-08 03:27:32 | 200-OK | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | /repstuff/nofollow-linked.php |
| 67.195.51.152 | 2008-05-08 23:47:05 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
| 74.6.28.49 | 2008-05-11 15:56:35 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/nofollow-linked.php |
The first five entries are samples. An empty table wouldn't look nice.