This page logs suspect requests of /repstuff/robots.txt

Choose a log file:
Here is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it.
This is a list of other user agents which requested it, but weren't verified as crawlers from major search engines.

/repstuff/robots.txt is just an example I've used in this post on cloaking robots.txt files. Of course search engines expect to find your robots.txt on root level.

Here is the log file:

220.243.135.15 2018-10-11 20:17:00 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a
52.210.122.6 2018-10-11 21:49:59 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=Slurp
144.202.193.40 2018-10-11 23:12:54 200-OK Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) /repstuff/robots.txt
52.210.122.6 2018-10-11 23:44:31 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=Mediapartners-Google
46.4.94.226 2018-10-12 01:33:55 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=msnbot
62.210.247.93 2018-10-12 03:34:56 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=*
62.210.247.93 2018-10-12 03:38:36 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=Adsbot-Google
144.202.193.40 2018-10-12 05:37:29 200-OK Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) /repstuff/robots.txt?crawlerName=Googlebot
62.210.247.93 2018-10-12 05:49:33 200-OK Riddler (http://riddler.io/about) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
220.243.136.231 2018-10-12 20:43:43 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=adsbot-google
220.243.135.105 2018-10-12 20:43:43 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a
220.243.135.231 2018-10-12 20:43:43 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=msnbot
220.243.135.105 2018-10-12 20:43:43 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-image
220.243.136.43 2018-10-12 20:43:45 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=slurp
220.243.135.43 2018-10-12 20:43:46 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=teoma
220.243.136.225 2018-10-12 20:43:46 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-mobile
220.243.135.225 2018-10-12 20:43:46 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=mediapartners-google
220.243.136.91 2018-10-12 20:43:47 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot
178.255.215.78 2018-10-13 21:23:51 200-OK Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
220.243.135.253 2018-10-14 03:42:08 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=slurp
220.243.136.199 2018-10-14 03:42:12 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot
220.243.136.199 2018-10-14 03:42:13 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=mediapartners-google
220.243.135.136 2018-10-14 03:42:13 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=teoma
220.243.136.10 2018-10-14 03:42:13 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-image
220.243.135.73 2018-10-14 03:42:13 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=msnbot
220.243.136.199 2018-10-14 03:42:13 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=adsbot-google
64.62.252.174 2018-10-14 22:07:40 200-OK The Knowledge AI /repstuff/robots.txt
64.62.252.174 2018-10-14 22:07:41 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.174 2018-10-14 22:07:41 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.174 2018-10-14 22:07:41 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.174 2018-10-14 22:07:56 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.174 2018-10-14 22:08:08 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.174 2018-10-14 22:08:08 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Slurp
64.62.252.174 2018-10-14 22:08:08 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=msnbot
64.62.252.174 2018-10-14 22:08:09 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Teoma
64.62.252.174 2018-10-14 22:08:09 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=*
220.243.135.6 2018-10-15 04:17:58 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-mobile
220.243.135.59 2018-10-15 04:18:19 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a
54.36.148.127 2018-10-16 11:17:00 200-OK Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) /repstuff/robots.txt
178.255.215.78 2018-10-16 19:44:56 200-OK Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) /repstuff/robots.txt?crawlerName=Slurp
220.243.135.252 2018-10-16 20:11:00 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=slurp
220.243.135.252 2018-10-16 20:11:03 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot
220.243.136.189 2018-10-16 20:11:05 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=mediapartners-google
220.243.135.16 2018-10-16 20:11:05 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=teoma
66.160.140.187 2018-10-17 04:47:46 200-OK The Knowledge AI /repstuff/robots.txt
66.160.140.187 2018-10-17 04:47:46 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot
66.160.140.187 2018-10-17 04:47:47 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Mobile
66.160.140.187 2018-10-17 04:48:02 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Image
66.160.140.187 2018-10-17 04:48:02 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Mediapartners-Google
66.160.140.187 2018-10-17 04:48:03 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Adsbot-Google
66.160.140.187 2018-10-17 04:48:16 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Slurp
66.160.140.187 2018-10-17 04:48:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=msnbot
66.160.140.187 2018-10-17 04:48:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Teoma
66.160.140.187 2018-10-17 04:48:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=*
69.13.39.34 2018-10-17 11:10:26 200-OK Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0 /repstuff/robots.txt
64.62.252.176 2018-10-17 16:06:52 200-OK The Knowledge AI /repstuff/robots.txt
64.62.252.176 2018-10-17 16:06:56 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.176 2018-10-17 16:07:06 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.176 2018-10-17 16:07:07 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.176 2018-10-17 16:07:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.176 2018-10-17 16:07:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.176 2018-10-17 16:07:17 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Slurp
64.62.252.176 2018-10-17 16:07:18 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=msnbot
64.62.252.176 2018-10-17 16:07:18 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=Teoma
64.62.252.176 2018-10-17 16:07:28 200-OK The Knowledge AI /repstuff/robots.txt?crawlerName=*
95.163.255.77 2018-10-17 22:00:45 200-OK Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) /repstuff/robots.txt
220.243.135.106 2018-10-17 22:29:05 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-mobile
220.243.136.232 2018-10-17 22:29:09 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-image
220.243.136.217 2018-10-17 22:29:10 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt
220.243.135.28 2018-10-17 22:29:11 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=msnbot
220.243.135.149 2018-10-18 04:01:10 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=adsbot-google
220.243.136.63 2018-10-18 08:30:20 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a
123.125.71.46 2018-10-18 16:50:43 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
220.243.136.243 2018-10-19 10:56:19 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=mediapartners-google
220.243.135.139 2018-10-19 10:56:20 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=teoma
220.243.135.96 2018-10-19 10:56:20 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a
220.243.136.139 2018-10-19 10:56:21 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-mobile
220.243.136.48 2018-10-19 10:56:25 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=slurp
220.243.135.159 2018-10-19 10:56:25 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-image
144.202.193.40 2018-10-19 21:03:17 200-OK Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) /repstuff/robots.txt?crawlerName=Slurp
144.202.193.40 2018-10-20 09:26:04 200-OK Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) /repstuff/robots.txt?crawlerName=Mediapartners-Google
220.243.135.123 2018-10-20 21:43:18 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=slurp
178.154.244.164 2018-10-21 01:00:57 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Googlebot-Image
220.243.135.197 2018-10-21 19:15:52 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-mobile
220.243.136.97 2018-10-21 19:15:53 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=teoma
220.243.135.182 2018-10-21 23:37:18 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=mediapartners-google
220.243.135.245 2018-10-21 23:37:19 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot
220.243.136.62 2018-10-22 05:59:16 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=msnbot
220.243.135.62 2018-10-22 05:59:16 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=googlebot-image
220.243.136.117 2018-10-22 05:59:20 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt
220.243.135.39 2018-10-22 09:08:30 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=adsbot-google
220.243.135.34 2018-10-22 11:28:53 200-OK Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 /repstuff/robots.txt?crawlername=%2a

The first five entries are samples. An empty table wouldn't look nice.