What kind of robots visit my site?

F

Friend2016-12-14 15:38:35

Computer networks

Friend, 2016-12-14 15:38:35

108.162.229.104 - - [14/Dec/2016:14:19:20 +0200] "GET /wp-content/uploads/2015/09/IMG_5847.jpg/ HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.1; + ahrefs.com/robot/) "
54.90.173.39 - - [14/Dec/2016:14:21:09 +0200] "HEAD / HTTP/1.1" 200 0 "-" "Go -http-client/1.1"
141.101.98.163 - - [14/Dec/2016:14:27:02 +0200] "GET /catalog/hudisvitshotykurtki?sort=price&page=2 HTTP/1.1" 404 152 "-" "Mozilla /5.0 (compatible; YandexBot/3.0; + yandex.com/bots) "
141.101.80.182 - - [14/Dec/2016:14:27:13 +0200] "GET /product-category/hit-prodazh/?v =f9308c5d0596 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_1_1 like Mac OS X) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0 Mobile/14B100 Safari/602.1"
141.101.81.174 - - [14/Dec/2016:14:27:14 +0200] "GET /apple-touch-icon-120x120-precomposed.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1 .4 Darwin/16.1.0"
141.101.81.168 - - [14/Dec/2016:14:27:14 +0200] "GET /apple-touch-icon-120x120.png HTTP/1.1" 404 152 "-" " MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
141.101.81.157 - - [14/Dec/2016:14:27:14 +0200] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
141.101.80.157 - - [14/Dec/2016:14:27:14 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
141.101.80.165 - - [14/Dec/2016:14:27:14 +0200] "GET /favicon.ico HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
173.245.51.12 - - [14/Dec/2016:14:27:47 +0200] "GET /robots.txt HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; + www.google .com/bot.html) "
173.245.50.202 - - [14/Dec/2016:14:27:47 +0200] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (compatible; Googlebot/2.1 ; + www.google.com/bot.html) "
108.162.229.104 - - [14/Dec/2016:14:32:19 +0200] "GET /kombinezon-angel/ HTTP/1.1" 404 152 "-" " Mozilla/5.0 (compatible; AhrefsBot/5.1; + ahrefs.com/robot/) "
172.68.51.54 - - [14/Dec/2016:14:33:31 +0200] "GET /catalog/bad-girls HTTP/1.1 " 404 152 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML,like Gecko) Version/10.0 Mobile/14A456 Safari/602.1"
172.68.51.72 - - [14/Dec/2016:14:33:31 +0200] "GET /apple-touch-icon-120x120-precomposed.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.0 .2 Darwin/16.0.0"
172.68.51.72 - - [14/Dec/2016:14:33:31 +0200] "GET /apple-touch-icon-120x120.png HTTP/1.1" 404 152 "-" " MobileSafari/602.1 CFNetwork/808.0.2 Darwin/16.0.0"
172.68.51.55 - - [14/Dec/2016:14:33:31 +0200] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.0.2 Darwin/16.0.0"
172.68.51.53 - - [14/Dec/2016:14:33:32 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.0.2 Darwin/16.0.0"
162.158.78.220 - - [14/Dec/2016:14:33:40 +0200] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPad; CPU OS 10_1_1 like Mac OS X) AppleWebKit/602.2 .14 (KHTML, like Gecko) Version/10.0 Mobile/14B100 Safari/602.1"
172.68.65.54 - - [14/Dec/2016:14:33:41 +0200] "GET /apple-touch-icon-152x152-precomposed .png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
172.68.65.18 - - [14/Dec/2016:14:33:41 +0200] "GET /apple -touch-icon-152x152.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
162.158.78.40 - - [14/Dec/2016:14:33:42 + 0200] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/16.1.0"
162.158.79.215 - - [14/Dec/2016:14:33:42 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/ 16.1.0"
162.158.79.167 - - [14/Dec/2016:14:33:43 +0200] "GET /favicon.ico HTTP/1.1" 404 152 "-" "MobileSafari/602.1 CFNetwork/808.1.4 Darwin/ 16.1.0"
There are also yandex and google, but transitions seem to start with /shop/.....
Why are there such strange transitions (links) that I don't have on my site?
Should they be blocked?

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

A

Alexey, 2016-12-14
@alsopub

In general, these are search robots.
If you block them, you will disappear from the search engines.
What you see 404 is the standard files that browsers are trying to download (favicon, picture for apple devices).
If there is a real problem with robots (site crash), then you need to look - either optimize the site or close part of the site through robots.txt