Choose a log file:
Here is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it.
This is a list of other user agents which requested it, but weren't verified as crawlers from major search engines.
/repstuff/robots.txt is just an example I've used in this post on cloaking robots.txt files. Of course search engines expect to find your robots.txt on root level.
Here is the log file:
| 220.243.135.15 | 2018-10-11 20:17:00 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
| 52.210.122.6 | 2018-10-11 21:49:59 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=Slurp |
| 144.202.193.40 | 2018-10-11 23:12:54 | 200-OK | Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) | /repstuff/robots.txt |
| 52.210.122.6 | 2018-10-11 23:44:31 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 46.4.94.226 | 2018-10-12 01:33:55 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=msnbot |
| 62.210.247.93 | 2018-10-12 03:34:56 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=* |
| 62.210.247.93 | 2018-10-12 03:38:36 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 144.202.193.40 | 2018-10-12 05:37:29 | 200-OK | Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) | /repstuff/robots.txt?crawlerName=Googlebot |
| 62.210.247.93 | 2018-10-12 05:49:33 | 200-OK | Riddler (http://riddler.io/about) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 220.243.136.231 | 2018-10-12 20:43:43 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=adsbot-google |
| 220.243.135.105 | 2018-10-12 20:43:43 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
| 220.243.135.231 | 2018-10-12 20:43:43 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=msnbot |
| 220.243.135.105 | 2018-10-12 20:43:43 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-image |
| 220.243.136.43 | 2018-10-12 20:43:45 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=slurp |
| 220.243.135.43 | 2018-10-12 20:43:46 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=teoma |
| 220.243.136.225 | 2018-10-12 20:43:46 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-mobile |
| 220.243.135.225 | 2018-10-12 20:43:46 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=mediapartners-google |
| 220.243.136.91 | 2018-10-12 20:43:47 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot |
| 178.255.215.78 | 2018-10-13 21:23:51 | 200-OK | Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 220.243.135.253 | 2018-10-14 03:42:08 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=slurp |
| 220.243.136.199 | 2018-10-14 03:42:12 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot |
| 220.243.136.199 | 2018-10-14 03:42:13 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=mediapartners-google |
| 220.243.135.136 | 2018-10-14 03:42:13 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=teoma |
| 220.243.136.10 | 2018-10-14 03:42:13 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-image |
| 220.243.135.73 | 2018-10-14 03:42:13 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=msnbot |
| 220.243.136.199 | 2018-10-14 03:42:13 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=adsbot-google |
| 64.62.252.174 | 2018-10-14 22:07:40 | 200-OK | The Knowledge AI | /repstuff/robots.txt |
| 64.62.252.174 | 2018-10-14 22:07:41 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.174 | 2018-10-14 22:07:41 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.174 | 2018-10-14 22:07:41 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.174 | 2018-10-14 22:07:56 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.174 | 2018-10-14 22:08:08 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.174 | 2018-10-14 22:08:08 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.174 | 2018-10-14 22:08:08 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.174 | 2018-10-14 22:08:09 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.174 | 2018-10-14 22:08:09 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=* |
| 220.243.135.6 | 2018-10-15 04:17:58 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-mobile |
| 220.243.135.59 | 2018-10-15 04:18:19 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
| 54.36.148.127 | 2018-10-16 11:17:00 | 200-OK | Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) | /repstuff/robots.txt |
| 178.255.215.78 | 2018-10-16 19:44:56 | 200-OK | Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) | /repstuff/robots.txt?crawlerName=Slurp |
| 220.243.135.252 | 2018-10-16 20:11:00 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=slurp |
| 220.243.135.252 | 2018-10-16 20:11:03 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot |
| 220.243.136.189 | 2018-10-16 20:11:05 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=mediapartners-google |
| 220.243.135.16 | 2018-10-16 20:11:05 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=teoma |
| 66.160.140.187 | 2018-10-17 04:47:46 | 200-OK | The Knowledge AI | /repstuff/robots.txt |
| 66.160.140.187 | 2018-10-17 04:47:46 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot |
| 66.160.140.187 | 2018-10-17 04:47:47 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 66.160.140.187 | 2018-10-17 04:48:02 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 66.160.140.187 | 2018-10-17 04:48:02 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 66.160.140.187 | 2018-10-17 04:48:03 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 66.160.140.187 | 2018-10-17 04:48:16 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Slurp |
| 66.160.140.187 | 2018-10-17 04:48:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=msnbot |
| 66.160.140.187 | 2018-10-17 04:48:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Teoma |
| 66.160.140.187 | 2018-10-17 04:48:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=* |
| 69.13.39.34 | 2018-10-17 11:10:26 | 200-OK | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0 | /repstuff/robots.txt |
| 64.62.252.176 | 2018-10-17 16:06:52 | 200-OK | The Knowledge AI | /repstuff/robots.txt |
| 64.62.252.176 | 2018-10-17 16:06:56 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.176 | 2018-10-17 16:07:06 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.176 | 2018-10-17 16:07:07 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.176 | 2018-10-17 16:07:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.176 | 2018-10-17 16:07:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.176 | 2018-10-17 16:07:17 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.176 | 2018-10-17 16:07:18 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.176 | 2018-10-17 16:07:18 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.176 | 2018-10-17 16:07:28 | 200-OK | The Knowledge AI | /repstuff/robots.txt?crawlerName=* |
| 95.163.255.77 | 2018-10-17 22:00:45 | 200-OK | Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) | /repstuff/robots.txt |
| 220.243.135.106 | 2018-10-17 22:29:05 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-mobile |
| 220.243.136.232 | 2018-10-17 22:29:09 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-image |
| 220.243.136.217 | 2018-10-17 22:29:10 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt |
| 220.243.135.28 | 2018-10-17 22:29:11 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=msnbot |
| 220.243.135.149 | 2018-10-18 04:01:10 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=adsbot-google |
| 220.243.136.63 | 2018-10-18 08:30:20 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
| 123.125.71.46 | 2018-10-18 16:50:43 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 220.243.136.243 | 2018-10-19 10:56:19 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=mediapartners-google |
| 220.243.135.139 | 2018-10-19 10:56:20 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=teoma |
| 220.243.135.96 | 2018-10-19 10:56:20 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
| 220.243.136.139 | 2018-10-19 10:56:21 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-mobile |
| 220.243.136.48 | 2018-10-19 10:56:25 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=slurp |
| 220.243.135.159 | 2018-10-19 10:56:25 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-image |
| 144.202.193.40 | 2018-10-19 21:03:17 | 200-OK | Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) | /repstuff/robots.txt?crawlerName=Slurp |
| 144.202.193.40 | 2018-10-20 09:26:04 | 200-OK | Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 220.243.135.123 | 2018-10-20 21:43:18 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=slurp |
| 178.154.244.164 | 2018-10-21 01:00:57 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 220.243.135.197 | 2018-10-21 19:15:52 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-mobile |
| 220.243.136.97 | 2018-10-21 19:15:53 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=teoma |
| 220.243.135.182 | 2018-10-21 23:37:18 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=mediapartners-google |
| 220.243.135.245 | 2018-10-21 23:37:19 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot |
| 220.243.136.62 | 2018-10-22 05:59:16 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=msnbot |
| 220.243.135.62 | 2018-10-22 05:59:16 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=googlebot-image |
| 220.243.136.117 | 2018-10-22 05:59:20 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt |
| 220.243.135.39 | 2018-10-22 09:08:30 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=adsbot-google |
| 220.243.135.34 | 2018-10-22 11:28:53 | 200-OK | Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36 | /repstuff/robots.txt?crawlername=%2a |
The first five entries are samples. An empty table wouldn't look nice.