V
V
Vitaly B2019-08-02 08:21:01
Parsing
Vitaly B, 2019-08-02 08:21:01

What is the optimal solution for quickly parsing 600 servers on a weak VDS?

Task: Searching for a user (BF4) on game servers, getting the server id and requesting round statistics.
Required: bypass each url, get json with a list of players, search for a player in this list, return the server id if found.
Given: json with 600 server id and player's nickname.
What I tried: multicurl in 100 requests - the result is from 3 to 30 seconds, but on the local machine. On a DO droplet with 1 core and 512GB RAM, it loads until it freezes.
And if using a synchronous method, then the result can be expected up to 3 minutes even on a local machine.
What are the options for PHP, NODE.JS? Or other languages?
It is required to get the search result as quickly as possible, but at the same time not to spend money on buying large virtual machine resources.
Keeping and updating all players in the database also seems like a dubious solution.
Options with pthreads and alternative options for PHP, as I understand it, will also require an increase in resources.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
K
Kirill Gorelov, 2019-08-02
@Kirill-Gorelov

And for how much do you want to collect data, in 1 second, or less, or 5 seconds will be fine??
Or increase not 100 requests, but for example 500, play with these settings.

A
Alex, 2019-08-02
@mr_ko

For this task, though, Node.js is better. Well, or try ReactPHP or something similar.
For Node.js, I don't see any problems, but the numbers don't look big.
You can drop (mail in profile or write here) a detailed description of the task (I mean the address where they are knocking, etc. if it's not a secret :)) I'll try to do it on the weekend.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question