D
D
Darkzenon2018-02-28 19:10:24
Python
Darkzenon, 2018-02-28 19:10:24

How to get the sequence number of a process?

Good afternoon.
Question about multiprocessing and tqdm in python on windows.
The code works but there are 2 problems.
1 - tqdm needs to set the position number, otherwise, instead of fixed progressbars, a scrollbar is obtained, new lines are simply added.
2 - for multiprocessing. how to get process sequence number so that it can be inserted into tqdm position number
idea with generator

def gen():
  a=[1,2,3,4,5,6,7,8,9,10]
  yield a

does not work, the whole generator is returned.
The only idea I don't really like is the list being passed to map. The first position is to pass the value of the iterator and then parse this list into functions.
spoiler
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool

def GetData(part_hrefs):
    df = []
    for item in tqdm(part_hrefs, position=2, desc="h", leave=False):

    return(df)



def main():
    with open('file.txt', 'r') as file:
        TotalLinks =file.readlines()

    parts = [round(len(TotalLinks)/10)*i for i in range(10)]
    parts.append(len(TotalLinks))
    names = [TotalLinks[parts[i]:parts[i+1]] for i in range(10)]

    pool = ThreadPool(10)
    l = pool.map(GetData, names)
    itog_data=Reduce_2(l)
    pd.DataFrame.to_csv(itog_data,'output.csv')

if __name__=='__main__':
    print ("start")
    main()
    print ("finish")

Answer the question

In order to leave comments, you need to log in

1 answer(s)
P
power mind, 2018-03-01
@dkdoter

Generator idea tip:
Instead of yield a, you need to yield from a

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question