N
N
neotmirnyj2022-02-28 00:50:07
Python
neotmirnyj, 2022-02-28 00:50:07

Multiprocessing read every line from txt file on every processor?

Please tell me how to do it right, for the second day I can’t set it up.

txt file was read on all processors simultaneously and repeatedly, but it is necessary that each processor read a separate line. I did not repeat everything several times.

For example, the first line is displayed like this
111
111
111
111
111
111
111
111

the second is like this

222
222
222
222
222
222
333
333

, etc.

Each processor reads lines repeatedly!

def main():
.......

file = open("txt.txt", 'r')

if __name__ == '__main__':
    for r in range(cores):
        multiprocessing.Process(target=main).start()

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dr. Bacon, 2022-02-28
@bacon

And it is better to make only one process read from the file and put it in the queue, from which all the others would take it.

S
shurshur, 2022-02-28
@shurshur

One of the easiest ways to do this is to assign a process number to each N process and have the first one read lines 1, N+1, 2N+1, ..., the second one 2, N+2, 2N+2, ... and etc. The lines of other processes are simply skipped.

# number - номер процесса от 0 до total-1
# total - всего процессов
def process(number, total):
   for i, l in enumerate(f):
      if i % total == number:
          ...

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question