Search for a colored frame in an image by color. How to take into account the change in color at different angles of inclination?

Question

Image processing

I welcome everyone!
Started learning computer vision using C# and EMGUCV.
Based on an example from the site, I'm trying to track a colored frame in the video stream from the webcam of the laptop and, having determined the image plane, draw a normal to the frame.
(Example: www.emgu.com/wiki/index.php/Shape_ (Triangle,_Rectangle,_Circle,_Line)_Detection_in_CSharp)
Problem: if you rotate the frame, then at certain tilt angles it stops being tracked, as the camera starts to perceive its color differently.
Refinement of the experiment:
I'm trying to find a square nested within another square. I cut out a square from colored paper, cut a smaller square in the center and pasted it all on an a4 landscape sheet, the camera sees only the sheet (therefore, there is no interference in the background).
The logic of the program is as follows:

• Got a BGR frame
  
• transferred to HSV
  
• function inRange (h, s, v) set the color of interest to us
  
• dilate and erode to remove noise from the resulting binary image
  
• got contours with canny
  
• filtered out from the list of contours open and with too small an area
  
• Hough transform for lines
  
• drawn the found lines on the original image.
  

(Not written yet) - find two nested squares from the list of lines, calculate the coordinates of two vectors, find the third vector, draw everything on the screen.
On the first image, you can see that in general everything works fine - the found contour is drawn in green on the original image from the webcam (upper left). On the bottom left picture (shows the result of hsv transformation and morphological operations) you can see that the frame is generally visible well, there is almost no interference. There is no way to insert a video, but while working on the frame, small interference periodically occurs, no more than 1/10 of the frame thickness.
The second image shows the moment of failure - the sheet is tilted so much that the frame is no longer detected. As far as I figured out, the reason is this - the camera sees the color differently -> the color is cut off inRange -> dilate and erode cut out too much part -> the contour is damaged and cut off according to the closure condition (see above) The
question is simple - what to do? What approach to correct the situation or how exactly to refine my method if it is erroneous (and what is the error)?
Additional question - is there an adaptivethreshold method for color images? Googling didn't turn up anything clear.
In general, I found a few hints on how to fix the situation:
1 - morphological transformations with a self-made core, it is necessary to align the contour inside the frame. For corners, use a different core. I don’t understand how to make it so that it also works when the frame is rotated, because it’s not so easy to set the core for the line, and if we also tilt it, we generally work with a rhombus.
2 - if you find the histogram of the frame, then by stretching it, you can theoretically refine the inRange method. How exactly - I do not understand. Also, using the histogram, in theory, you can search for the frame itself in the image. But how then to find the data on which to build a histogram?
3 - find feature point.
4 - find all angles in the original image. then run the search for the frame, determine the largest shaded area (probable candidate for the frame), then go through all the angles to find the equations of the lines that make up our frame.
I would be glad for any advice, if something is not clear or clarification of the task is required, I will unsubscribe in the course of the discussion.
Thank you all in advance!

Answer 1

And if here's how to try: calculate the Laplacian of the Gaussian from the hue in HSV, and then binarize the fields with a Laplacian close to zero by thresholding?
In this case, smooth color changes (due to uneven illumination) should go away, while sharp changes (borders) should remain.