How fast are search bots?

B

Baxterok2014-08-15 17:59:41

Google

Baxterok, 2014-08-15 17:59:41

I wonder why search bots are not blocked on popular sites as a possible DDoS attack? Is it because of their slow work or the servers detect them and give them the opportunity to work?

Reply

Answer the question

In order to leave comments, you need to log in

3 answer(s)

S

Sergey, 2014-08-15
Protko @Fesor

just for big sites it's zilch. Just like another user who climbs the pages.

P

Puma Thailand, 2014-08-15
@opium

99 percent of sites do not block any bots in any way

M

Myateznik, 2014-08-17
@Myateznik

Most search bots have a rule: 1 page = 1 second. It should also be taken into account that bots have some kind of queue, where pages of different sites are mixed up. Due to the rule and peculiarities of the work of search bots, they are not a DDOS threat, just like a person who opened 2 pages of one site at once in ~ 2 seconds.
(Indexing of thousands of pages over a period of time is provided by the number of bots. 1000 pages per minute = 16-17 parallel bots)