What is the correct way to implement a "heavy" internal service in Django?

Question

Django

Problem:
My website has some functionality with objects that take a long time to initialize. So I initialize them once and store them in memory. However, they also take up quite a lot of memory by themselves. Therefore, now it turns out that each worker from Apache creates its own set of such objects, and the web hoster's memory limit is reached very quickly.
How I imagine the solution:
I want all Apache workers to access the same object - so as not to create a different one in each thread. Those. there should be one separate thread in which all these objects will be stored, and the threads that service requests to the server will send requests to this thread. This functionality is fast enough, so no queue should be created.
Question:
How to correctly implement such an architecture in Django? Should this be done at the Python level or using some Django mechanisms? Should it be a standalone application with a REST API?

Answer 1

You need to move heavy objects outside the web server into a separate process, provide an interface to them via RPC on any transport (for example, ZeroMQ or bare TCP as a transport). It will be more architecturally correct. At the same time, the web server will remain lightweight, multithreaded (what do you have under Apache? Gunicorn? UWSGI? - multithreading is their task) and, most importantly, stateless (as it should be). And the process encapsulating a heavy object will become an RPC server, pre-initialized, optimally cached, not causing memory overruns (the consumption is almost constant and does not depend on the load from web clients) and, most importantly, statefull (which is what you most likely need ).