I
I
Ilya Erokhin2016-03-24 06:55:30
Highload
Ilya Erokhin, 2016-03-24 06:55:30

Working with queues (queue) in Node.js, and notifications about the completion of tasks, how?

A server on the Node sticks out on the web, clients come and ask for answers. Multiple clients often ask for the same answer at the same time.
The formation of each response is a very expensive task (we quickly run into the CPU), tasks are performed in child processes, for a long time. For maximum performance, from 2 to a dozen (s) of task-processes should be running simultaneously - how many specifically - is set in the settings.
That is, when the web server receives a task - it queues it for execution, and waits for it to be ready - it subscribes to a notification about the completion of this task. Several http request handlers can subscribe to one task. When it is ready, they all give the answer to their clients.
External queue managers such as Celery are not suitable - everything should spin inside one Node process (not counting child processes that perform tasks).
Task states - failed / completed successfully / completed with an error + an error object (like a promise). There are no other properties, subscribers know where to get the results of a successfully completed task.
The pattern, in principle, is quite standard, and there should be ready-made best practices for the Node, or at least share your thoughts - where to dig at all?
UPD: it seems that the understanding has finally taken shape:

task = queue.push(taskData(req));
task.on('complete', () => {
// задача выполнена, делаем полезное
}).on('error', err => {
// ошибка, делаем неприятное
});

That is, this is a queue of EventEmitter'ov, the queue itself resolves what to do with tasks - add as a new one or return an existing one. A fixed number of workers receive tasks from the queue sequentially.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
I
ilejn, 2016-03-24
@ilejn

memcache has already been invented https://habrahabr.ru/post/200060/

N
Night, 2016-03-30
@maxtm

Look towards AMQP. Rabbit is perfect .
It has a powerful system of task providers and subscribers to them.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question