B
B
Bone2014-11-04 17:52:19
PHP
Bone, 2014-11-04 17:52:19

What is the best way to implement an infinite job queue?

The bottom line is this - I need to check certain pages on different sites. Continuously and sequentially. When the queue reaches the end, it starts again from the first task. It should be possible to add new tasks, swap tasks and delete them. Accordingly, the question is, how best to implement it? I know PHP and ActionScript (more or less close to Java), but as far as I understand, they are not suitable for such purposes. The tasks themselves are quite simple, I think that it will not be a big problem to implement them in any language, so you need to choose the most suitable one. Should work on Debian. Please advise any specific bundles that you consider the most suitable for such a task.

Answer the question

In order to leave comments, you need to log in

4 answer(s)
A
algebris, 2014-11-04
@Bone

you can use Node.JS and check not only sequentially, but also in parallel, that is, asynchronously :) If you know PHP, then I can assume that you are familiar with Javascript :)
https://github.com/request/request - here it is you can make a request to the page
https://github.com/cheeriojs/cheerio - this is how to parse it

A
Alexander Aksentiev, 2014-11-04
@Sanasol

"Bundle": crontab, php
And a database with tasks, then you yourself will figure out how it should be.

X
xmoonlight, 2014-11-04
@xmoonlight

Start +1 php process in infinite loop with include. Further, at any time, edit the file-include. Include multi-threaded curl in it. (see documentation), etc.

A
Alexander Zelenin, 2014-11-04
@zelenin

e.g.
$currentTask = 0;
$allTasks = 1000;
$currentTask++;
if($currentTask == $allTasks) {
$currentTask = 0;
}

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question