A
A
Andrey2018-07-30 21:37:55
Python
Andrey, 2018-07-30 21:37:55

How to correctly and simply implement multithreading in Python?

You need to run 100+ threads in parallel.
Example:

file = open('file.txt')
lines = file.readlines()
def just():
  for line in lines:
    sleep(1)
    print(line)

Tried like this, but each thread starts the loop again:
for i in range(0, 100):
    t = threading.Thread(target=just)
    t.start()

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
Sergey Gornostaev, 2018-07-30
@tellst1

def just(lines):
  for line in lines:
    sleep(1)
    print(line)


file = open('file.txt')
lines = file.readlines()
step = len(lines) // 100

threads = []
for i in range(0, len(lines), step):
    t = threading.Thread(target=just, args=(lines[i:i+step],))
    t.start()
    threads.append(t)

for t in threads:
    t.join()

D
Dvvarreyn, 2018-07-31
@Dvvarreyn

Python uses Global Interpreter Lock and therefore does not have efficient parallel multithreading.
Pythonic multithreading can only be used to organize any controlling threads.
For parallelization, you need to use multiprocessing.
Reading from a file is not parallelized (at least on the same machine, there will be no threads on different machines), since the file is not random access data.
Scattering reading between threads leads to a quadratic algorithm in python, instead of a linear one. Hence the slowdown.
I would recommend picking a smarter task to parallelize. And then start with an example

import multiprocessing 

def proc(a):
  <…>
  return something

pool = multiprocessing.Pool(3) # ну может 4. 

result = pool.map(proc, list(range(100)))

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question