H
H
haDDah2021-05-26 14:15:50
Python
haDDah, 2021-05-26 14:15:50

How to pass complex objects between processes in python?

There is such task: It is
necessary to create and fill a certain hierarchy of classes, in different processes (for acceleration) and to return to the main process.

import multiprocessing
import multiprocessing.managers
 
class ChildClass:
    name = "child name"
    def __init__(self):
        pass
 
class ParentClass:
    name = "parent name"
    child_class = []
 
    def __init__(self):
        num_of_childs = 3
        for i in range(num_of_childs):
            self.child_class.append(ChildClass())
 
def func(shared_array):
    shared_array.append(ParentClass())
 
if __name__ == '__main__':
    mngr = multiprocessing.Manager()
    shared_array = mngr.list()
 
    num_iterations = 3
    processes = []
    for i in range(3):
        processes.append(multiprocessing.Process(target=func, args=(shared_array,)))
 
    for process in processes:
        process.start()
    for process in processes:
        process.join()
 
    for obj in shared_array:
        print(obj.name)
        print(obj.child_class)

Result:
parent name
[]
parent name
[]
parent name
[]

The problem is the array of ChildClass objects is empty.

In fact, the class hierarchy is much larger and each class can have many arrays containing other classes in the hierarchy.
I understand that the problem is that I create a class in another process and cannot read it in the main one.

Googling, I tried to use multiprocessing.managers.BaseManager to register my class, but the problem is that creating this BaseManager is not a fast process and with the number of objects 1000+ without parallelization, the program runs faster ...

It is not necessary to use the multiprocessing lib, I will accept any ideas on any libs.
Thank you all in advance, otherwise I have been suffering for more than 1 day)

Answer the question

In order to leave comments, you need to log in

2 answer(s)
R
rPman, 2021-05-26
@rPman

Don't make yourself a problem, use threads, not processes.
Sharing data within one process is much easier than trying to do it between processes, not only in terms of logic and code, but also organization.
ps python and speed? These two words cannot stand side by side... and even if you need to use shared memory even more so, look for crap for yourself.

A
Andy_U, 2021-05-26
@Andy_U

Won't go? shared memory for direct access across processes

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question