How to automatically create multiple images by overlaying one image on top of others and transforming the shape of that image in the process?

Question

Automation

There are many online stores that sell printed clothing, and they usually offer several different types of products with the same print. As an example: https://www.redbubble.com/i/t-shirt/Visit-Tatooine... The question is how to automate the creation of images for this kind of products?

What is clear at the moment.

How to do this in Photoshop in semi-automatic mode is clear - the designer creates a smart object, on which all manipulations are performed and which is replaced with a print image. It is assumed that the process of imposing prints will take place on the server at the time of creation of goods through the CMS Magento admin panel.

Reproducing the image overlay in the right place at the right size and mixing the colors of the print and layout so that the uniform is visible through the print can be done using a library like OpenCV.

The problem is in this step, in converting the shape of the print so that it follows the shape of the clothes.




I was told that I should look in the direction of 2D spline deformation, a couple of libraries that I looked at seem to be too low-level in order to be able to set complex deformations for layouts in a sane time.

Using GIMP in console mode seems like a possible option, but it's not yet clear if it can reproduce all the required steps automatically. Those. put the print in the right place in the right size on the layout, mix the colors, change the shape of the print depending on the layout and get the output image in JPEG and, ideally, PSD with layers. For example, there are no smart objects in it, that is, it will definitely not work to reproduce the approach used in Photoshop 1 in 1.

An alternative approach, about which everything is still completely unclear, is to set up a separate virtual machine on Windows and try to automate Photoshop on it, but this option certainly looks more complicated than using the console GIMP or OpenCV.

Maybe someone knows how they usually approach solving such problems? For some reason, it was not possible to google anything on this topic at all (maybe, of course, I was looking for the wrong thing), despite the fact that this is probably not done manually and the task as a whole looks typical.

Answer 1

There is a 100% ready-made library, but I didn’t google it, since the task is not so difficult, because everything can be done in advance.
The initial photo with the mask should not be taken with Malevich's white square, but with a specially prepared image based on a chessboard with small colored squares (the same colors should not be side by side, just a regular pattern), so that too large folds do not interfere with determining which square it is.
The resulting photo needs to be compared with the original image, and the difference will be the changes that need to be made to the target images before overlaying. We associate each square (the area painted with one color so as not to butt with shadows, you need to take a photo with good lighting) in the photo with a square in the image (search for the nearest of the same color) and then either look for the center or try to detect the corners of the squares and take them by points, for each point we build a shift vector (dx, dy) to which we need to correct the corresponding square, then we consider these vectors as splines for each point on the image, we use the resulting matrix to edit all the photos.
Look for libraries with warping / morphing effect, those that exist work on key points, just those that you calculate on the centers of the square in my description above.