How to take into account the color in the photo recommendation system?

Question

Neural networks

Hello. I am working on a product recommendation system based on photos. It was possible to get good recommendations using a neural network and class classification of goods. The problem is that the recommendations are too far in color from each other. If someone faced this problem, tell me how you can take into account the color in the recommendations?
Already used: convolutional network with color classification and calculation of Euclidean detuning from the product color vector to the recommended one, search for the main color of the product and its angle between the color of the recommended product in Lab