V
V
Vitaly2021-07-24 23:24:36
linux
Vitaly, 2021-07-24 23:24:36

How to receive responses from an overloaded server with a certain frequency?

Given:
There is a website in apache + php that is being parsed by a lot of people. The server is physically one, no balancers, cloud services, etc. no. At different times, the load on the server is different, and the response from it can come in 0.5 seconds, 30 seconds, or not come at all during the peak load period.

Question:
How to maximize the probability of successfully receiving data from the server (page parsing) at least once every 10 seconds? Those. I need to parse data from the server and get a response at least every 10 seconds.

If you send requests sequentially, then at peak load, you can wait several minutes before the server returns data.

What are the options? Bomb every 5 seconds, counting on the fact that at least 50% of requests will reach and the average time to receive a response will be no more than 30 seconds? Plus add proxies that are geographically closer to the server?
I am using aiohttp in python3. Switching to something faster with a modern approach to asynchrony / multi-threading like Rust / Go, probably does not make much sense?

60fc75fcc9ec1645158657.png

Answer the question

In order to leave comments, you need to log in

1 answer(s)
K
ky0, 2021-07-24
@ky0

The server is physically one, no balancers, cloud services, etc. no.

Horizontally scale, add balancers, cloud services, etc.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question