P
P
PtrGrd2021-11-26 21:03:52
Python
PtrGrd, 2021-11-26 21:03:52

How to write the result of each process into a separate corresponding file (Multiprocessing, Python)?

Briefly:
There is a script that splits a large array of data into groups. Each such group of data runs in a separate process. It is necessary to make sure that each such group of data is written to the target file for it.

I need to make the code find the file where the data needs to be written

Simplified version of my code (for clarity)

# Разбили массив данных на три группы. Каждому процессу - одну группу из списка
data_group = 

# Функция записи данных в конкретный файл
def write_data(data, file_name):

  with open(file_name, mode='w', encoding="utf-8") as file:
    names = ["Колонка1", "Колонка2"]
    file = csv.DictWriter(file, delimiter = ",", lineterminator = "\r",
      fieldnames = names)
    # Запись данных в колонку
    for item in data:
      file.writerow(item)

# Функция анализа данных
def func(data_group):
  data = []
  # Какой-то анализ группы данных с последующей записью в файл
  for item in data_group:
    first = item[0]
    second = item[1]
    third = item[2]
    data.append(first, second, third)
    write_data(data, file_name)

# Запуск в 3 процессах
with Pool(3) as pool:
  pool.starmap(func, product(data_group, file_name))


It is necessary to somehow transfer a separate file to each separate process. Please help in advance, thanks a lot!

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vindicar, 2021-11-26
@PtrGrd

for i,item in enumerate(data_group):
    file_name = f"{i}.txt"
    #ну и далее по тексту

Something like this? I seriously don't see the problem.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question