D
D
delkov2016-12-05 06:24:55
linux
delkov, 2016-12-05 06:24:55

How to do parallel reading from a file, Python+Linux(GNU_parallel)?

Good day, dear ones!
There is a primitive bash script that reads a file:
cat -v /dev/pts/2 | python3 'process.py'
Where is the python3 script:

import sys
  for string in sys.stdin:
    some_calculation(string)

My question is how can I parallelize reading and processing? To run two process.py scripts at the same time
I found GNU-parallel, but something didn't start:
cat -v /dev/pts/2 | parallel  --pipe --recstart '>' python3 process.py

How to fix it, or are there other ways to do it?
Thanks in advance!
solution found, just play around with the --block option:
cat -v /dev/pts/2 | parallel --j 3 --pipe --block 12M python3 process.py

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
delkov, 2016-12-05
@delkov

solution found, just play around with the --block option:
cat -v /dev/pts/2 | parallel --j 3 --pipe --block 12M python3 process.py

V
Vladimir, 2016-12-05
@vintello

problem in setting the task. it is not known what kind of text file we have and why it must be read in several streams.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question