Is there a lightweight alternative to rq that doesn't use Redis?

Question

Python

There is a need to regularly score a large queue. Redis stores information in RAM. Sometimes it may not be enough. Sometimes it may be necessary to restart the computer before all the tasks in the queue have finished executing. (I don't know if Redis can save the base between runs.) I would like to have something lightweight like rq, but insensitive to the amount of RAM.

Answer 1

Redis can write to disk, and there are several write strategies, more details here .

Answer 2

Write to text files, in append mode.
Rotate them every N seconds/minutes for as long as you need. Rotated files are sorted by the worker, and all new incoming files go to the current file.
Very simple and stupid =)