A
A
Alya2019-01-15 17:28:28
Python
Alya, 2019-01-15 17:28:28

How to make multiprocessing work?

Task: make 3 processes (independent) that access the same function with different arguments.
I did this using threading, but then I realized that it uses only 1 core and the processes are executed sequentially. I need multiple cores to work.
I tried to do it like this:

Phrase1 = Process(target = Noun_Phrase, name = 'pr_1', args = ('./wiki/wikido1.txt', './wiki/file_1.txt'))
Phrase2 = Process(target = Noun_Phrase, name = 'pr_2',args = ('./wiki/wikiot1do2.txt', './wiki/file_2.txt'))
Phrase3 = Process(target = Noun_Phrase, name = 'pr_3',args = ('./wiki/wikiot2.txt', './wiki/file_3.txt'))
Phrase1.start()
Phrase2.start()
Phrase3.start()
Phrase1.join()
Phrase2.join()
Phrase3.join()

but when executed, either 1 process is launched, or an error in general (each time different processes are launched) and I can’t understand what my mistake is and how to fix it
, help me:

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
Sergey Gornostaev, 2019-01-15
@sergey-gornostaev

import multiprocessing as mp

data = [
    ('./wiki/wikido1.txt', './wiki/file_1.txt'),
    ('./wiki/wikiot1do2.txt', './wiki/file_2.txt'),
    ('./wiki/wikiot2.txt', './wiki/file_3.txt'),
]

with mp.Pool(mp.cpu_count()) as pool:
    pool.starmap(Noun_Phrase, data)

M
Mikhail Trainin, 2019-01-15
@Stormx480

There is a good article on multi-threading in Python here.
I will add my example, I just tested it.

spoiler
import multiprocessing
import os

def worker():
    #worker function
    print ('Worker')
    x = 0
    while x < 1000:
        print(x)
        x += 1
    return

if __name__ == '__main__':
    jobs = []
    for i in range(int(os.environ["NUMBER_OF_PROCESSORS"])):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

Using the os module, we find out the number of processes, and tell the program to work with their number.

R
Roman Kitaev, 2019-01-15
@deliro

Do not touch multiprocessing unless absolutely necessary. Use ProcessPoolExecutor and asyncio (loop.run_in_executor)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question