Choose a log file:
Here is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it.
This is a list of other user agents which requested it, but weren't verified as crawlers from major search engines.
/repstuff/robots.txt is just an example I've used in this post on cloaking robots.txt files. Of course search engines expect to find your robots.txt on root level.
Here is the log file:
| 158.69.249.18 | 2017-08-25 18:04:14 | 200-OK | Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.163 | 2017-03-13 12:05:23 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.163 | 2017-03-13 12:05:24 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.163 | 2017-03-13 12:05:24 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.163 | 2017-03-13 12:05:24 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.163 | 2017-03-13 12:05:24 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.163 | 2017-03-13 12:05:40 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.163 | 2017-03-13 12:05:40 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.163 | 2017-03-13 12:05:40 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 141.8.143.138 | 2017-03-13 14:14:46 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 141.8.143.138 | 2017-03-13 15:02:52 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 141.8.143.138 | 2017-03-14 02:24:13 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.164 | 2017-03-14 02:30:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.164 | 2017-03-14 02:30:47 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.164 | 2017-03-14 02:30:57 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.164 | 2017-03-14 02:30:57 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.164 | 2017-03-14 02:30:57 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.164 | 2017-03-14 02:30:57 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.164 | 2017-03-14 02:30:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.164 | 2017-03-14 02:30:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.164 | 2017-03-14 02:31:08 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.164 | 2017-03-14 02:31:08 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 77.88.47.33 | 2017-03-14 05:41:54 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=msnbot |
| 82.76.87.6 | 2017-03-14 09:06:09 | 200-OK | Java/1.6.0_04 | /repstuff/robots.txt |
| 141.8.143.138 | 2017-03-14 23:24:03 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt |
| 79.143.180.232 | 2017-03-15 01:51:22 | 200-OK | Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0 | /repstuff/robots.txt |
| 64.62.252.162 | 2017-03-16 06:42:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.162 | 2017-03-16 06:42:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.162 | 2017-03-16 06:43:01 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.162 | 2017-03-16 06:43:20 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.162 | 2017-03-16 06:43:21 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 64.62.252.164 | 2017-03-16 08:35:16 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.164 | 2017-03-16 08:35:16 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.164 | 2017-03-16 08:35:16 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.164 | 2017-03-16 08:35:16 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.164 | 2017-03-16 08:35:16 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.164 | 2017-03-16 08:35:17 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.164 | 2017-03-16 08:35:17 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.164 | 2017-03-16 08:35:31 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.164 | 2017-03-16 08:35:31 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.164 | 2017-03-16 08:35:32 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 104.154.58.95 | 2017-03-16 08:50:41 | 200-OK | ltx71 - (http://ltx71.com/) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 173.208.157.186 | 2017-03-16 10:23:50 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt |
| 173.208.157.186 | 2017-03-16 10:23:52 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt?crawlerName=Googlebot |
| 192.162.227.236 | 2017-03-17 07:38:57 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt?crawlerName=* |
| 192.162.227.236 | 2017-03-17 07:39:00 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 192.162.227.236 | 2017-03-17 07:39:02 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 192.162.227.236 | 2017-03-17 07:39:04 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 192.162.227.236 | 2017-03-17 07:39:06 | 200-OK | Mozilla/5.0 (compatible; MJ12bot/v1.4.7; t?crawlerName=Mediapartners-Google | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 144.76.198.133 | 2017-10-02 13:27:55 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=* |
| 144.76.198.133 | 2017-10-02 13:27:58 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=* |
| 144.76.198.133 | 2017-10-02 13:28:06 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Teoma |
| 144.76.198.133 | 2017-10-02 13:28:11 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Teoma |
| 144.76.198.133 | 2017-10-02 13:28:17 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=msnbot |
| 144.76.198.133 | 2017-10-02 13:28:18 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=msnbot |
| 144.76.198.133 | 2017-10-02 13:28:23 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Slurp |
| 144.76.198.133 | 2017-10-02 13:28:25 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Slurp |
| 144.76.198.133 | 2017-10-02 13:28:28 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 144.76.198.133 | 2017-10-02 13:28:32 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 144.76.198.133 | 2017-10-02 13:28:39 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 144.76.198.133 | 2017-10-02 13:28:45 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 144.76.198.133 | 2017-10-02 13:28:51 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 144.76.198.133 | 2017-10-02 13:28:52 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 144.76.198.133 | 2017-10-02 13:28:56 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 144.76.198.133 | 2017-10-02 13:28:58 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 144.76.198.133 | 2017-10-02 13:29:01 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot |
| 144.76.198.133 | 2017-10-02 13:29:05 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt?crawlerName=Googlebot |
| 144.76.198.133 | 2017-10-02 13:29:11 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt |
| 144.76.198.133 | 2017-10-02 13:29:17 | 200-OK | Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) | /repstuff/robots.txt |
| 35.184.189.105 | 2017-10-02 15:01:02 | 200-OK | ltx71 - (http://ltx71.com/) | /repstuff/robots.txt |
| 180.76.15.10 | 2017-10-03 23:45:02 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 217.182.132.77 | 2017-10-04 16:18:08 | 200-OK | Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) | /repstuff/robots.txt |
| 180.76.15.26 | 2017-10-04 21:34:17 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 180.76.15.140 | 2017-10-05 01:08:03 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 136.243.36.89 | 2017-10-05 04:46:49 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt |
| 136.243.36.89 | 2017-10-05 04:46:56 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=* |
| 136.243.36.89 | 2017-10-05 04:47:03 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 136.243.36.89 | 2017-10-05 04:47:11 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot |
| 136.243.36.89 | 2017-10-05 04:47:18 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 136.243.36.89 | 2017-10-05 04:47:23 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 136.243.36.89 | 2017-10-05 04:47:30 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 136.243.36.89 | 2017-10-05 04:47:36 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Slurp |
| 136.243.36.89 | 2017-10-05 04:47:40 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Teoma |
| 136.243.36.89 | 2017-10-05 04:47:49 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=msnbot |
| 180.76.15.156 | 2017-10-06 08:45:08 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 141.8.143.185 | 2017-10-06 17:38:49 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 35.184.189.105 | 2017-10-08 01:27:59 | 200-OK | ltx71 - (http://ltx71.com/) | /repstuff/robots.txt?crawlerName=msnbot |
| 35.184.189.105 | 2017-10-08 14:42:14 | 200-OK | ltx71 - (http://ltx71.com/) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 68.180.231.49 | 2017-10-09 09:48:31 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/robots.txt |
| 167.114.219.41 | 2017-10-09 10:49:02 | 200-OK | Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/) | /repstuff/robots.txt |
| 220.181.108.149 | 2017-10-09 22:05:07 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.164 | 2017-10-10 02:34:17 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.164 | 2017-10-10 02:34:34 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.164 | 2017-10-10 02:34:35 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.164 | 2017-10-10 02:34:35 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.164 | 2017-10-10 02:34:35 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.164 | 2017-10-10 02:34:35 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.164 | 2017-10-10 02:34:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.164 | 2017-10-10 02:34:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.164 | 2017-10-10 02:34:46 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.164 | 2017-10-10 02:34:47 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 141.8.143.185 | 2017-10-11 09:02:20 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt |
| 64.62.252.163 | 2017-10-11 12:17:31 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.163 | 2017-10-11 12:17:32 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.163 | 2017-10-11 12:17:42 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.163 | 2017-10-11 12:17:42 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.163 | 2017-10-11 12:17:43 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.163 | 2017-10-11 12:17:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.163 | 2017-10-11 12:17:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.163 | 2017-10-11 12:17:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.163 | 2017-10-11 12:17:58 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.163 | 2017-10-11 12:17:59 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 191.101.127.222 | 2017-10-11 15:48:54 | 200-OK | Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 | /repstuff/robots.txt |
| 191.101.125.242 | 2017-10-11 15:48:54 | 200-OK | Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 | /repstuff/robots.txt?crawlerName=Slurp |
| 191.101.127.222 | 2017-10-11 15:48:55 | 200-OK | Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 | /repstuff/robots.txt?crawlerName=msnbot |
| 136.243.36.89 | 2017-10-11 22:28:40 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt |
| 136.243.36.89 | 2017-10-11 22:28:46 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=* |
| 136.243.36.89 | 2017-10-11 22:28:53 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 136.243.36.89 | 2017-10-11 22:28:57 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot |
| 136.243.36.89 | 2017-10-11 22:29:02 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 136.243.36.89 | 2017-10-11 22:29:07 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 136.243.36.89 | 2017-10-11 22:29:11 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 136.243.36.89 | 2017-10-11 22:29:16 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Slurp |
| 136.243.36.89 | 2017-10-11 22:29:20 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=Teoma |
| 136.243.36.89 | 2017-10-11 22:29:26 | 200-OK | Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) | /repstuff/robots.txt?crawlerName=msnbot |
| 141.8.143.185 | 2017-10-12 23:07:14 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 141.8.143.185 | 2017-10-13 01:23:37 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.162 | 2017-10-13 06:39:34 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Mediapartners-Google |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Slurp |
| 64.62.252.162 | 2017-10-13 06:39:45 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=msnbot |
| 64.62.252.162 | 2017-10-13 06:39:56 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=Teoma |
| 64.62.252.162 | 2017-10-13 06:39:56 | 200-OK | BUbiNG (+http://law.di.unimi.it/BUbiNG.html) | /repstuff/robots.txt?crawlerName=* |
| 123.125.71.108 | 2017-10-13 13:04:25 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Adsbot-Google |
| 220.181.108.105 | 2017-10-13 14:44:25 | 200-OK | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 169.60.128.99 | 2017-10-14 23:40:24 | 200-OK | Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)/Nutch-1.14-SNAPSHOT | /repstuff/robots.txt?crawlerName=Googlebot-Mobile |
| 169.60.128.125 | 2017-10-15 02:04:52 | 200-OK | Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)/Nutch-1.14-SNAPSHOT | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 68.180.231.49 | 2017-10-15 09:42:52 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/robots.txt |
| 141.8.143.185 | 2017-10-16 02:04:02 | 200-OK | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | /repstuff/robots.txt?crawlerName=Googlebot-Image |
| 68.180.231.49 | 2017-10-17 12:58:56 | 200-OK | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | /repstuff/robots.txt |
The first five entries are samples. An empty table wouldn't look nice.