Answer the question
In order to leave comments, you need to log in
How to make multiprocessing work?
Task: make 3 processes (independent) that access the same function with different arguments.
I did this using threading, but then I realized that it uses only 1 core and the processes are executed sequentially. I need multiple cores to work.
I tried to do it like this:
Phrase1 = Process(target = Noun_Phrase, name = 'pr_1', args = ('./wiki/wikido1.txt', './wiki/file_1.txt'))
Phrase2 = Process(target = Noun_Phrase, name = 'pr_2',args = ('./wiki/wikiot1do2.txt', './wiki/file_2.txt'))
Phrase3 = Process(target = Noun_Phrase, name = 'pr_3',args = ('./wiki/wikiot2.txt', './wiki/file_3.txt'))
Phrase1.start()
Phrase2.start()
Phrase3.start()
Phrase1.join()
Phrase2.join()
Phrase3.join()
Answer the question
In order to leave comments, you need to log in
import multiprocessing as mp
data = [
('./wiki/wikido1.txt', './wiki/file_1.txt'),
('./wiki/wikiot1do2.txt', './wiki/file_2.txt'),
('./wiki/wikiot2.txt', './wiki/file_3.txt'),
]
with mp.Pool(mp.cpu_count()) as pool:
pool.starmap(Noun_Phrase, data)
There is a good article on multi-threading in Python here.
I will add my example, I just tested it.
import multiprocessing
import os
def worker():
#worker function
print ('Worker')
x = 0
while x < 1000:
print(x)
x += 1
return
if __name__ == '__main__':
jobs = []
for i in range(int(os.environ["NUMBER_OF_PROCESSORS"])):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
Do not touch multiprocessing unless absolutely necessary. Use ProcessPoolExecutor and asyncio (loop.run_in_executor)
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question