Why is Python multiprocessing unstable?

Question

Python

Hello community. Attempts to deal with multiprocessing in python led to a serious snag. I would be very grateful for any help in parsing the problem, or for recommendations on how to fix the code.
In the program for mathematical calculations I use the part with multiprocessing. The parallel block of code looks like this:

worker_count = multiprocessing.cpu_count()
        jobs = []
        print "---> Starting multiprocessing #", series_number
        for i in xrange(worker_count):
            s = solver.get_solution(copied_system)
            p = multiprocessing.Process(target=combinatorial_set.find_nearest_set_point, args=(s, result_queue))
            jobs.append(p)
            p.start()
            print p
        for w in jobs:
            w.join()
            print w
        results = []
        while not result_queue.empty():
            results.append(result_queue.get())
        for i in xrange(len(res)):
            print results[i], is_solution(copied_system, results[i])
            if is_solution(copied_system, results[i]):
                func_value = f(results[i])
                experiment_valid_points[func_value] = results[i]

        #End of parallel
        print "---> End of multiprocessing #", series_number

Here's what the code was supposed to do:
A series of experiments are being carried out. Within the framework of each experiment, the following actions are performed
: 1. Starting the experiment.
2. Generation of points (s) in some area. Number of dots = number of cores.
3. Run for each point of finding the minimum of the function. This part is done in parallel.
4. Collection of results.
5. Select from the generated points the best one.
6. Recalculation of the area for generation.
7. Transition to a new experiment.
And here's what happens:
In practice, this code works very unstable. The following situations can occur on the same data.
1. The first "ideal" one: within the framework of each experiment, 4 python.exe are launched, each loading ~ 25% of the CPU. After the end of the experiment, 3 of these four processes die and everything starts from the beginning.
2. The second one is "worse": within each experiment, 4 python.exe are launched, but they are executed in pseudo-parallel. The resource monitor shows 4 python.exe processes, one thread each. At one point in time, only one is always working, the rest are interrupted until their turn reaches them.
3. The third "worst": in the console, all 4 processes are unsubscribed by Process(Process-i, started), but neither in the task manager nor in the resource monitor are there 4 processes. There is only 1 python.exe that uses ~25% CPU and it's not clear what it counts. Naturally, the work does not go further, since you need to wait for the completion of all four processes, but they simply do not exist.
Thanks in advance for your replies.

Answer 1

Good thing you tagged Windows, that explains it all. Under Windows, there is no easy way to "fork" a process when called multiprocessing.Process, so a very complex emulation of this behavior is performed. In this case, the function is targettorn out of the module, launched in a separate interpreter, and all parameters are serealized, passed and de-realized before calling target, while the module initialization in the new interpreter is performed partially (only the global context is initialized). More about this, for example, here , there is another very good article where this mechanism is discussed in detail, but now I can’t find a link.
Briefly about how to prepare multiprocessing under Windows:

1. Separate processes (call multiprocessing.Process()) as early as possible in the code.
   
2. If possible, avoid initializing any resources and global variables prior to splitting. Keep in mind that this code is executed in all processes independently and can have a bunch of side effects.
   
3. Do not pass through argsany complex objects with "behavior" (except for objects from multiprocessing itself, it knows how to pass them correctly), only bare data (primitives or objects consisting only of primitives), which are serialized without side effects.
   
4. Create child processes once, and work with them throughout the entire time through messaging through Pipe / Queue. Do not spawn new processes in the calculation cycle at the moment "when needed".
   
5. Queue, when trying to write / read, may block the process if it is being written / read to / from it in another process. (I think this is exactly what is happening in the code in the question).
   
6. It is better to use Pipe, which in the worst case blocks one process, and not all, like Queue.
   
7. When creating a process, you can pass two Pipes to it (input of one + output of the other), store the corresponding connectors in the calling process and only use them to communicate with the child process.
   
8. You can not do it process.join(), but just read the results from the output Pipe, they will be read only after they get there, what happens next with the process is no longer important (you can put it returnafter writing to the Pipe in the child process).
   

Answer 2

3. Third "worst"

1. It seems to me that in this target=combinatorial_set.find_nearest_set_point - you should have a critical error that just kills processes.
2. Why are you doing this in Python? At the same time, you lose 100 times in performance compared to languages ​​that compile to native code (C, C ++, Go). Moreover, there you would have the opportunity to use lightweight threads instead of heavy parallel processes.