When to stop A/B testing

Question

Google Analytics

How much traffic, in your experience, is needed to reliably decide which option is best?

I'm currently doing A/A testing with Google Analytics. After 600 visits, he shows that one of the options is 2 times better than the other. Although this is impossible, because options are the same. How long does it take for the conversion rate to equalize?

Answer 1

I think it depends on your site. If you have a very low conversion rate, then let's say in your experiment option A scored 2 conversions, and B - 1 with 600 visits, so it turns out "twice". In fact, this is your control group. Look at what number of visits they consistently equal, this will be approximately the size of your minimum statistically significant sample.

Answer 2

I learned from experience that it is impossible to draw any meaningful conclusions about the quality of a page until it has generated more than 25-30 conversions. In most cases, by this time it is already quite confident to talk about the conversion rate of the page.
Although in some cases the result was quite stable already at 10-15 sales.

Answer 3

I recommend multi-armed bandits, they even in their simplest form solve this problem automatically.