ORB descriptors how to find similar images?

Question

Java

There are 2 images, I find ORB descriptors for each of them. Next, I filter the descriptors by distance and by the RANSAC algorithm, but it turns out that the ear is not pointing to the logo, and so on. What is the problem?

Below code:

public static void compareFeature(String filename1, String filename2) {
        ORB orb = ORB.create();

        DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);

        Mat img1 = Imgcodecs.imread(filename1, Imgcodecs.CV_LOAD_IMAGE_GRAYSCALE);
        Mat descriptors1 = new Mat();
        MatOfKeyPoint keypoints1 = new MatOfKeyPoint();
        orb.detectAndCompute(img1, new Mat(), keypoints1, descriptors1);

// second image
        Mat img2 = Imgcodecs.imread(filename2, Imgcodecs.CV_LOAD_IMAGE_GRAYSCALE);
        Mat descriptors2 = new Mat();
        MatOfKeyPoint keypoints2 = new MatOfKeyPoint();
        orb.detectAndCompute(img2, new Mat(), keypoints2, descriptors2);


// MATCHING
// match these two keypoints sets
        MatOfDMatch matches = new MatOfDMatch();
        matcher.match(descriptors1, descriptors2, matches);

        List<DMatch> matchesList = matches.toList();
        // ratio test
        LinkedList<DMatch> good_matches = new LinkedList<>();
        Double max_dist = 0.0;
        Double min_dist = 100.0;
        for (DMatch aMatchesList1 : matchesList) {
            Double dist = (double) aMatchesList1.distance;
            if (dist < min_dist)
                min_dist = dist;
            if (dist > max_dist)
                max_dist = dist;
        }
        for (DMatch aMatchesList : matchesList) {
            if (aMatchesList.distance <= (1.5 * min_dist))
                good_matches.addLast(aMatchesList);
        }
        // get keypoint coordinates of good matches to find homography and remove outliers using ransac
        List<Point> pts1 = new ArrayList<>();
        List<Point> pts2 = new ArrayList<>();
        for (DMatch good_matche : good_matches) {
            pts1.add(keypoints1.toList().get(good_matche.queryIdx).pt);
            pts2.add(keypoints2.toList().get(good_matche.trainIdx).pt);
        }

        // convertion of data types - there is maybe a more beautiful way
        Mat outputMask = new Mat();
        MatOfPoint2f pts1Mat = new MatOfPoint2f();
        pts1Mat.fromList(pts1);
        MatOfPoint2f pts2Mat = new MatOfPoint2f();
        pts2Mat.fromList(pts2);

        // Find homography - here just used to perform match filtering with RANSAC, but could be used to e.g. stitch images
        // the smaller the allowed reprojection error (here 15), the more matches are filtered
        System.out.println("pts1mat " + pts1Mat.size());
        System.out.println("pts2mat " + pts2Mat.size());
        Mat Homog = Calib3d.findHomography(pts1Mat, pts2Mat, Calib3d.RANSAC, 15, outputMask, 2000, 0.995);
        System.out.println("pts1mat " + pts1Mat.size());
        System.out.println("pts2mat " + pts2Mat.size());
        // outputMask contains zeros and ones indicating which matches are filtered
        LinkedList<DMatch> better_matches = new LinkedList<DMatch>();
        for (int i = 0; i < good_matches.size(); i++) {
            if (outputMask.get(i, 0)[0] != 0.0) {
                better_matches.add(good_matches.get(i));
            }
        }
        // DRAWING OUTPUT
        Mat outputImg = new Mat();
        // this will draw all matches, works fine
        MatOfDMatch better_matches_mat = new MatOfDMatch();
        better_matches_mat.fromList(better_matches);
        Features2d.drawMatches(img1, keypoints1, img2, keypoints2, better_matches_mat, outputImg);
        // save image
        Imgcodecs.imwrite("result.jpg", outputImg);
        System.out.println("better_matches_mat " + better_matches_mat.size());
    }

Answer 1

what it should do at this stage is to select similar features located in the pictures approximately equally relative to each other. to do this, it tries to choose such a projection matrix of one picture onto another so that the maximum number of similar features matches (in order to discard mismatched ones). and this is done on the assumption that such a projection exists.
there are cases when it works: for example, stereopairs of images, adjacent frames from a video, or heavily overlapping images in photogrammetry - it will be able to calculate the overlay and everything will be fine.
there are cases when it does not work at all: in pictures with repeating textures, or for example, if in one picture an object from the second occurs several times in different places and poses - it will try to fit a picture with a single object so that it is in the right places hit the best matched parts from different objects.
your pictures are initially different, so the best thing it came up with was to compress one of the pictures diagonally almost into a line, and stick it upside down with the back side to the second picture. if you transform the picture with the resulting matrix and draw over the second one, the signs will almost certainly match there, that is, it works as it should. another question is that it is obviously useless to you.
what you can do with it depends on what you need.
let's say if you're trying to look specifically for logos on T-shirts, it makes sense to collect a small visual dictionary of the features that appear in them, and before trying to match, filter out from all the features of the pictures only those that are embedded in the logos. so that there are more features from logos and fewer features from fingers-ears.
you can also try to rewrite the selection of the matrix so that it does not consider degenerate options. I was doing this for a long time (in 11-12 years), there were no orb descriptors then (I had surf-sift), but this selection of homography by Ransak never gave me normal results - it constantly tried to turn the picture inside out (match internal corners instead of outside). I remember that I tried to prop it up with a crutch so that it would not put negative coefficients in the matrices, but everything was complicated there (it seems that it supports several algorithms except ransac and the logic is smeared into layers at random). in general, I didn’t finish it - before it worked, for my specific task, I came up with a simplified way to verify the relative position of features without these matrices.
therefore, the last sentence: try to do without this homography. although, in my opinion, the basic simple options will not work for your pictures.