This page logs suspect requests of /repstuff/robots.txt

Choose a log file:
Here is a list of search engine crawlers from Google, Yahoo, MSN and Ask which requested it.
This is a list of other user agents which requested it, but weren't verified as crawlers from major search engines.

/repstuff/robots.txt is just an example I've used in this post on cloaking robots.txt files. Of course search engines expect to find your robots.txt on root level.

Here is the log file:

158.69.249.18 2017-08-25 18:04:14 200-OK Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.163 2017-03-13 12:05:23 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.163 2017-03-13 12:05:24 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.163 2017-03-13 12:05:24 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.163 2017-03-13 12:05:24 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.163 2017-03-13 12:05:24 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.163 2017-03-13 12:05:40 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.163 2017-03-13 12:05:40 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.163 2017-03-13 12:05:40 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
141.8.143.138 2017-03-13 14:14:46 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Googlebot-Image
141.8.143.138 2017-03-13 15:02:52 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Adsbot-Google
141.8.143.138 2017-03-14 02:24:13 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.164 2017-03-14 02:30:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.164 2017-03-14 02:30:47 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.164 2017-03-14 02:30:57 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.164 2017-03-14 02:30:57 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.164 2017-03-14 02:30:57 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.164 2017-03-14 02:30:57 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.164 2017-03-14 02:30:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.164 2017-03-14 02:30:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.164 2017-03-14 02:31:08 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.164 2017-03-14 02:31:08 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
77.88.47.33 2017-03-14 05:41:54 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=msnbot
82.76.87.6 2017-03-14 09:06:09 200-OK Java/1.6.0_04 /repstuff/robots.txt
141.8.143.138 2017-03-14 23:24:03 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt
79.143.180.232 2017-03-15 01:51:22 200-OK Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0 /repstuff/robots.txt
64.62.252.162 2017-03-16 06:42:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.162 2017-03-16 06:42:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.162 2017-03-16 06:43:01 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.162 2017-03-16 06:43:20 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.162 2017-03-16 06:43:21 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
64.62.252.164 2017-03-16 08:35:16 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.164 2017-03-16 08:35:16 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.164 2017-03-16 08:35:16 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.164 2017-03-16 08:35:16 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.164 2017-03-16 08:35:16 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.164 2017-03-16 08:35:17 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.164 2017-03-16 08:35:17 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.164 2017-03-16 08:35:31 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.164 2017-03-16 08:35:31 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.164 2017-03-16 08:35:32 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
104.154.58.95 2017-03-16 08:50:41 200-OK ltx71 - (http://ltx71.com/) /repstuff/robots.txt?crawlerName=Adsbot-Google
173.208.157.186 2017-03-16 10:23:50 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt
173.208.157.186 2017-03-16 10:23:52 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt?crawlerName=Googlebot
192.162.227.236 2017-03-17 07:38:57 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt?crawlerName=*
192.162.227.236 2017-03-17 07:39:00 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt?crawlerName=Adsbot-Google
192.162.227.236 2017-03-17 07:39:02 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt?crawlerName=Googlebot-Image
192.162.227.236 2017-03-17 07:39:04 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
192.162.227.236 2017-03-17 07:39:06 200-OK Mozilla/5.0 (compatible; MJ12bot/v1.4.7; t?crawlerName=Mediapartners-Google /repstuff/robots.txt?crawlerName=Mediapartners-Google
144.76.198.133 2017-10-02 13:27:55 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=*
144.76.198.133 2017-10-02 13:27:58 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=*
144.76.198.133 2017-10-02 13:28:06 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Teoma
144.76.198.133 2017-10-02 13:28:11 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Teoma
144.76.198.133 2017-10-02 13:28:17 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=msnbot
144.76.198.133 2017-10-02 13:28:18 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=msnbot
144.76.198.133 2017-10-02 13:28:23 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Slurp
144.76.198.133 2017-10-02 13:28:25 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Slurp
144.76.198.133 2017-10-02 13:28:28 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Adsbot-Google
144.76.198.133 2017-10-02 13:28:32 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Adsbot-Google
144.76.198.133 2017-10-02 13:28:39 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Mediapartners-Google
144.76.198.133 2017-10-02 13:28:45 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Mediapartners-Google
144.76.198.133 2017-10-02 13:28:51 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot-Image
144.76.198.133 2017-10-02 13:28:52 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot-Image
144.76.198.133 2017-10-02 13:28:56 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
144.76.198.133 2017-10-02 13:28:58 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
144.76.198.133 2017-10-02 13:29:01 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot
144.76.198.133 2017-10-02 13:29:05 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt?crawlerName=Googlebot
144.76.198.133 2017-10-02 13:29:11 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt
144.76.198.133 2017-10-02 13:29:17 200-OK Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) /repstuff/robots.txt
35.184.189.105 2017-10-02 15:01:02 200-OK ltx71 - (http://ltx71.com/) /repstuff/robots.txt
180.76.15.10 2017-10-03 23:45:02 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Teoma
217.182.132.77 2017-10-04 16:18:08 200-OK Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) /repstuff/robots.txt
180.76.15.26 2017-10-04 21:34:17 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
180.76.15.140 2017-10-05 01:08:03 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
136.243.36.89 2017-10-05 04:46:49 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt
136.243.36.89 2017-10-05 04:46:56 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=*
136.243.36.89 2017-10-05 04:47:03 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Adsbot-Google
136.243.36.89 2017-10-05 04:47:11 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot
136.243.36.89 2017-10-05 04:47:18 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot-Image
136.243.36.89 2017-10-05 04:47:23 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
136.243.36.89 2017-10-05 04:47:30 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Mediapartners-Google
136.243.36.89 2017-10-05 04:47:36 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Slurp
136.243.36.89 2017-10-05 04:47:40 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Teoma
136.243.36.89 2017-10-05 04:47:49 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=msnbot
180.76.15.156 2017-10-06 08:45:08 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Googlebot
141.8.143.185 2017-10-06 17:38:49 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
35.184.189.105 2017-10-08 01:27:59 200-OK ltx71 - (http://ltx71.com/) /repstuff/robots.txt?crawlerName=msnbot
35.184.189.105 2017-10-08 14:42:14 200-OK ltx71 - (http://ltx71.com/) /repstuff/robots.txt?crawlerName=Mediapartners-Google
68.180.231.49 2017-10-09 09:48:31 200-OK Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) /repstuff/robots.txt
167.114.219.41 2017-10-09 10:49:02 200-OK Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/) /repstuff/robots.txt
220.181.108.149 2017-10-09 22:05:07 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.164 2017-10-10 02:34:17 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.164 2017-10-10 02:34:34 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.164 2017-10-10 02:34:35 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.164 2017-10-10 02:34:35 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.164 2017-10-10 02:34:35 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.164 2017-10-10 02:34:35 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.164 2017-10-10 02:34:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.164 2017-10-10 02:34:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.164 2017-10-10 02:34:46 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.164 2017-10-10 02:34:47 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
141.8.143.185 2017-10-11 09:02:20 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt
64.62.252.163 2017-10-11 12:17:31 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.163 2017-10-11 12:17:32 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.163 2017-10-11 12:17:42 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.163 2017-10-11 12:17:42 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.163 2017-10-11 12:17:43 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.163 2017-10-11 12:17:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.163 2017-10-11 12:17:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.163 2017-10-11 12:17:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.163 2017-10-11 12:17:58 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.163 2017-10-11 12:17:59 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
191.101.127.222 2017-10-11 15:48:54 200-OK Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 /repstuff/robots.txt
191.101.125.242 2017-10-11 15:48:54 200-OK Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 /repstuff/robots.txt?crawlerName=Slurp
191.101.127.222 2017-10-11 15:48:55 200-OK Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 /repstuff/robots.txt?crawlerName=msnbot
136.243.36.89 2017-10-11 22:28:40 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt
136.243.36.89 2017-10-11 22:28:46 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=*
136.243.36.89 2017-10-11 22:28:53 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Adsbot-Google
136.243.36.89 2017-10-11 22:28:57 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot
136.243.36.89 2017-10-11 22:29:02 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot-Image
136.243.36.89 2017-10-11 22:29:07 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
136.243.36.89 2017-10-11 22:29:11 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Mediapartners-Google
136.243.36.89 2017-10-11 22:29:16 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Slurp
136.243.36.89 2017-10-11 22:29:20 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=Teoma
136.243.36.89 2017-10-11 22:29:26 200-OK Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) /repstuff/robots.txt?crawlerName=msnbot
141.8.143.185 2017-10-12 23:07:14 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Mediapartners-Google
141.8.143.185 2017-10-13 01:23:37 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.162 2017-10-13 06:39:34 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Mobile
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Mediapartners-Google
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Slurp
64.62.252.162 2017-10-13 06:39:45 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=msnbot
64.62.252.162 2017-10-13 06:39:56 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=Teoma
64.62.252.162 2017-10-13 06:39:56 200-OK BUbiNG (+http://law.di.unimi.it/BUbiNG.html) /repstuff/robots.txt?crawlerName=*
123.125.71.108 2017-10-13 13:04:25 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Adsbot-Google
220.181.108.105 2017-10-13 14:44:25 200-OK Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) /repstuff/robots.txt?crawlerName=Googlebot-Image
169.60.128.99 2017-10-14 23:40:24 200-OK Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)/Nutch-1.14-SNAPSHOT /repstuff/robots.txt?crawlerName=Googlebot-Mobile
169.60.128.125 2017-10-15 02:04:52 200-OK Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)/Nutch-1.14-SNAPSHOT /repstuff/robots.txt?crawlerName=Googlebot-Image
68.180.231.49 2017-10-15 09:42:52 200-OK Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) /repstuff/robots.txt
141.8.143.185 2017-10-16 02:04:02 200-OK Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) /repstuff/robots.txt?crawlerName=Googlebot-Image
68.180.231.49 2017-10-17 12:58:56 200-OK Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) /repstuff/robots.txt

The first five entries are samples. An empty table wouldn't look nice.