How to make Keras and Multiprocessing friends?

Question

Python

Hello.
I have a database that stores pictures. There is a neuron collected and saved using Keras . The script should get the rows from the table and process the image in each of them in order using the neuron, and return the processing result to the row. In order not to take too much time, I immediately decided to use the Pool method from the Multiprocessing package . In such an implementation, if pools_number > 1 and multiprocessing is used , the script hangs on keras_model.predict([x]) . If pools_number = 1 , then everything works as it should:

pools_number = 1

from keras.models import load_model
from keras.preprocessing import image as keras_image

# Загружаем сохраненную сеточку в память
keras_model = load_model('/work_model.h5')

def getResult(image):

  print('Обрабатываем {}...'.format(image['name']))

  start = time.monotonic()

  # Открываем картинку
  img_keras = keras_image.load_img(image['path'], target_size=(150, 150))
  # Конвертируем в np массив
  x = keras_image.img_to_array(face_img)
  # Добавляем ось
  x = np.expand_dims(x, axis=0)  
  x /= 255.

  # Получаем результат
  predict_result = keras_model.predict([x])
  result1 = predict_result[0][0]
  result2 = predict_result[0][1]

  result_time = time.monotonic() - start
  print('Выполнено за {:>.3f} секунд'.format(result_time))

rows = [image1, image2]

if pools_number == 1:

  for idx, row in enumerate(rows):

    getResult(row)

else:

  with multiprocessing.Pool(pools_number) as pool:

    pool.map(getResult, rows)
    pool.close()
    pool.join()

I read articles and threw in the second option, when the mesh model from the file is loaded inside the getResult () function:

pools_number = 1

def getResult(image):

  from keras.models import load_model
  from keras.preprocessing import image as keras_image

  # Загружаем сохраненную сеточку в память
  keras_model = load_model('/work_model.h5')

  print('Обрабатываем {}...'.format(image['name']))

  start = time.monotonic()

  # Открываем картинку
  img_keras = keras_image.load_img(image['path'], target_size=(150, 150))
  # Конвертируем в np массив
  x = keras_image.img_to_array(face_img)
  # Добавляем ось
  x = np.expand_dims(x, axis=0)  
  x /= 255.

  # Получаем результат
  predict_result = keras_model.predict([x])
  result1 = predict_result[0][0]
  result2 = predict_result[0][1]

  result_time = time.monotonic() - start
  print('Выполнено за {:>.3f} секунд'.format(result_time))

rows = [image1, image2]

if pools_number == 1:

  for idx, row in enumerate(rows):

    getResult(row)

else:

  with multiprocessing.Pool(pools_number) as pool:

    pool.map(getResult, rows)
    pool.close()
    pool.join()

In this case, everything works great for any values ​​of pools_number , but since the work_model.h5 file weighs almost gigabytes, processing one image takes almost 30 seconds, and the server memory requirements (to support, for example, 10-14 threads for 12 core processor) skyrocket. I can’t refuse to use multithreading, since there are more than 8 million rows in the database, and the time to complete the task is limited.
Question: how to make Keras and Multiprocessing friends ?

Answer 1

0) Use a GPU.
1) Use batches, i.e. process images in batches.
2) Image loading-preprocessing procedures should be measured by execution time and replaced with fast ones - for example, from OpenCV (cv2.imread, cv2.warpAffine, cv2.normalize). Also, prepare buffers for the result in advance, and not allocate memory for the output array anew each time.
3) Image loading can be moved to asynchronous code.
4) Tensorflow (or whatever you have under the hood) already uses all the cores.
5) Have you optimized the neuron for inference? Frozen graph, merge BN layers, TensorRT, bit depth reduction?
6) Why does the model weigh almost a gigabyte? Lighter models did not fit? Google MobileNetV2 for example?

Answer 2

In general, Keras is designed in such a way that the only way to combine it with Multiprocessing is to load the model separately for each process. If processes are created for each "small" code, as, for example, in my case for processing one image, then you can rewrite the script so that the process is created not for a single image, but for a group of them (in my case, I simultaneously uploaded 20 images ). Thus, the memory requirements will become more loyal and can be put into the available resources.