What is faster than 10 threads in 10 processes, or individual 100 threads?

Question

Python

Good afternoon. Testing multithreading and multiprocessing. The question arose: what is faster than 10 threads in 10 processes or individual 100 threads? Regarding resource consumption, 10 processes are more expensive, I understand. But what about speed?

Answer 1

There is little difference between threads and processes at the OS level.
Processes have isolated memory, while threads share shared memory within a process.
And that's basically the difference.
In terms of performance, there is a big difference between threads (processes) and coroutines. You can run 100_000 coroutines and everything will work, but 100_000 threads will most likely fall from Out of memory.

Answer 2

It all depends on how long the context of the thread / process will rise and how long everyone will wait for the task to complete. Roughly speaking, if each thread makes a request to the site and receives a response in 20ms, then there will be no profit from the processes, but if everyone downloads data for 10 seconds and then processes them, then the profit can be significant.
If the task is not clear whether it will be profitable, you can use
https://joblib.readthedocs.io/en/latest/generated/...
and try in each of the modes (in this library, the mode changes by simply changing the settings)
It is also worth looking at side of celery, perhaps this will give more control over the process.