Answer the question
In order to leave comments, you need to log in
How to organize a queue of http requests for a parser?
I am making a parser with the ability to send a lot of http requests in asynchronous mode.
I am using laravel 6, mysql, guzzle, dom crawler.
There is a table that stores visits with url, rule_id, scheduled_visit_at, visited_at, response_body for example. rule_id is the ID of the rule that stores settings for parsing specific links. These settings have a condition, for example, a limit of 10 parallel requests to one resource, 50 to another, and so on ... And there is also a separate setting to limit parallel requests to the entire server, for example, 200.
Running 200 queue worker processes is not an option, since this is ideas are costly in terms of resources.
I would like to write a thread of a script like a demon so that it spins constantly in the background, in which there would be something like a pool of parallel requests, if at the moment the pool is less than the limit, then we check whether there is something to take into circulation now.
This is real in general, if so, what libraries to take?
Or some other way to solve this problem?
I read about asynchronous frameworks, such as react, amphp, they have non-blocking I / O libraries for http and mysql. are they suitable for this?
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question