Quantitative testing?

Question

Software testing

There is a certain algorithm that is constantly being improved (or not). And I want to test it on several reference samples after each improvement. As I understand it, there are Continuous Integration systems for this, which assemble the project, test it and report on the results.
And I understand how it works in the case of classical testing - the algorithm can either work or not work (1 or 0).
In my case, this is a computer vision algorithm. And he can work at 100%, maybe at 0%, or maybe at 66% or 45.6%.
I have not found any means that take into account such quantitative results. Maybe I'm not understanding something, or I'm missing something.
Please help with advice or some best practices on this topic.
Thanks

Answer 1

Only hardcore, only developing your own metrics, classifying test data into groups. Expert evaluation of data by people, comparison of the result with expert evaluation, comparison of the obtained metrics with the previous and best result, separately for each data set and for the class. At the same time, the framework should be able to recalculate all the metrics for previous versions according to the new algorithm, because the quality assessment system will be regularly updated. An attempt to reduce the estimate to a binary form will give the "average temperature for the hospital." I am almost sure that you will not find a ready-made solution, it is too atypical task for a mass solution. At least I didn't find it at the time.