P
P
pavelkolodin2016-02-01 20:04:18
A/b testing
pavelkolodin, 2016-02-01 20:04:18

A/B - testing. Correctness of calculation of results?

There are many objects, the algorithm glues them into pairs. The steam burns out with the release of energy. The energy value depends on the known and unknown parameters of the objects in the pair.
That is, there are a bunch of objects. A stream from this heap falls at the input of the algorithm (it is not a fact that the entire heap has time to go through). The algorithm attaches another object from the same heap to the next object. Both will not come to the entrance. The steam immediately burns out with the release of energy and the burnt objects are no longer accessible.
There is a new algorithm and a desire to compare its "energy efficiency" with the old one. (How much more energy does it generate by bonding pairs).The entire set of objects is virtually split (according to the hash function of their ID + remainder) into groups A and B. We write a log of hits of an object in group B -- i.e. we have a "list" of group B objects at the end of the experiment. There is a log of incoming objects, according to which we can restore group A ( which came to the input and did not get into log B ). There is also protection against theft of objects from other groups.
Comparison of algorithms: we compare 2 values:
* divide the sum of energies of pairs from list A by the number of objects in list A
* divide the sum of energies of pairs from list B by the number of objects in list B
There is a feeling that the group sizes should be a constant reflected in the separation function (hash(id) % percent), and not calculated "after the fact" from some lists. But I can't justify this feeling logically. I don’t understand why it’s impossible to consider the function of dividing into groups as a stable black box (assigning one object stably to its group, but how - we don’t know and can only find out from logs) and, accordingly, why it’s impossible to calculate the sizes of groups after (from logs) - during summarizing.