Algorithm for converting image colors to reference ones

Question

Image processing

As an example, the image search service in Yandex or Google - you choose a color, you are shown the corresponding pictures.
Applied task: there are pictures of wallpapers and fabrics, it is necessary to automatically select the primary and secondary colors and bring them to the reference ones. What are the reference colors: the admin can set a list in the admin panel: white, pink, orange, etc. Naturally, the codes of these colors (RGB) are indicated.
The problem is that there are a lot of shades. And you need to determine which reference color is closer to the shade.
The following method has been tried: RGB=XYZ. Each reference color is a point inside a cube with a face length of 256. The color of an image pixel is also a point inside this cube. The pixel distance to all reference colors is calculated, the smallest distance is the required reference color.
This thing works. But, it works too well. For example, a dirty green color, such as khaki, she quite correctly relates to yellow. But visually it is green. This is not mathematics, but a feature of human perception.
Attempts to introduce adjustments, such as moving yellow outside the cube, did not give much improvement.
Where to dig, comrades?

Answer 1

Try to compare not in RBG, but in Lab.
There are RGB -> XYZ -> CIE-L*ab translation algorithms here, and here is the color difference formula in CIE-Lab.