Gentlemen, what is my mistake?

Question

Mathematics

We need to compare the probabilities of success in two binomial distributions, having some sample of trials for both options. Let's call the option that has more sample mean option A, the second option B.
Pa is the probability of success in option A., Pb in option B. Let's denote Pa>Pb as A>B.

We build confidence intervals for both options. We select alpha (confidence probability) so that the lower limit of the confidence interval of option A is strictly greater than the upper limit of option B. We get two independent events with a probability of alpha. If they both attack, then the probability that A>B is 100%. Therefore, the probability of A>B is not less than alpha squared.

When the sample means are not equal and the number of successes is greater than zero in both cases, then we get that the alpha is non-zero. Those. the probability is strictly greater than zero.

However, we do not know how the probability of success is distributed among the options. And we can choose a distribution such that the probability of A>B is zero. For example, the probability of success is always 0.5. Then we get that the probability that A > B = 0. Although before that we estimated it as strictly greater than zero.

What's my mistake?

- update: I'll give an example.

We need to test the registration form on the site. Which of these options is better in terms of the likelihood that the user will fill it out. We create two options and send traffic to them randomly. A month later, we see that 105 users have registered on option A. Option B has 99 users. Both options were visited by 1000 users. And we calculate the confidence interval. We select the alpha parameter so that the intervals do not intersect. We are of little interest in specific values, the main thing is that the minimum of interval A is greater than the maximum of B, and alpha is not equal to zero.
Option A lies in its range with probability alpha. Option B is also in its range with probability alpha.
The probability that both options lie in their ranges at the same time is equal to alpha squared. If they lie in their ranges, then option A is 100% better. Those. the probability that option A is indeed better is greater than alpha squared.

Let's say we have an alpha of 0.5. As a result, we get that the probability that A is better than B is at least 0.25.
We return to the site to stop the experiment and it turns out that we made a stupid mistake in the code and users were shown option A instead of option B. That is. two identical options. And A=B. Although before that we got a non-zero probability that A is strictly greater than B.

What is my mistake?

Answer 1

maybe I didn't understand all of the above correctly, but still. To estimate two unknown parameters of the binomial distribution of a random variable, you specify the level of alpha, choose a statistic with which you will evaluate the unknown parameter, and build a gamma confidence interval for both random variables. the constructed intervals will be an estimate of the unknown distribution parameters, in a specific case, the probability of success in a series of independent Bernoulli trials. where do you get that you have two events with a given probability and even more independent? you are trying to estimate probabilities not from your set of conditions at all.