F
F
Fiamma2019-04-02 20:17:57
Parsing
Fiamma, 2019-04-02 20:17:57

Why is the parsing speed with DigitalOcean much slower than with AWS?

There is a parser that runs on a DigitalOcean minimal virtual machine and collects 10 pages per second. This is too small, I tried to change the type of virtual machines, deployed optimized ones with a bunch of processors and memory, the result is about the same.
I tried to put it on AWS with the cheapest virtual machine for comparison, and the parsing speed soared from 10 pages to 500 without any strain.
I can’t understand what is the reason, it’s too expensive to switch to AWS. I would like to understand and correct the situation ((
Parsing is done using PHP and Guzzle.
PS.
==================
Ping to DO - 150
Ping to AWS - 256
=== ================

Answer the question

In order to leave comments, you need to log in

4 answer(s)
#
#, 2019-04-02
@mindtester

have you tried tracing and pinging to the target sites?
.. offhand, it would be absolutely adequate if the target site is on AWS, and even in the same DC
ps but in general, the process brings profit? if so, what are the problems to calculate the benefit from the gain in speed, in comparison with the cost of hosting?
.. if not - do not suffer, let yourself drip slowly

R
Roman Mirilaczvili, 2019-04-02
@2ord

In my opinion, the AWS instance was located more optimally geographically than DO. That is, it is a matter of distance and speed of individual routes.

F
Fiamma, 2019-09-16
@Fiamma

On DO - 20 requests per second, on AWS - 800. From observations, I can say that AWS works great for the first day / two - then the speed drops to 30 requests, and as I understand it, this is due to the power points that AWS provides.
In general, I solved my problem by optimizing the parser. Everything got much better.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question