Video Bonus: Conversion Rates Meet Distributions with Tim Wilson

July 11, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

One comment on “Video Bonus: Conversion Rates Meet Distributions with Tim Wilson

  1. Georgi Georgiev Jul 12, 2018

    Hey Tim,

    That’s a really nice try to delve into the inner-workings of the stats and I’m glad my writings were useful to you 🙂

    One addition or clarification, if you’d allow me. The video continuously depicts the two distributions: of the conversion rate of the control and the variant, as if that is what we most often try to measure/estimate in an A/B test. In reality (most of the time) one performs statistical significance tests and builds confidence intervals for the absolute difference between these two conversion rates (delta = pB – pA), not for the conversion rates per se (pA, pB). The two are, of course, directly related, but are not the same thing. The absolute difference has its own distribution with its own mean, variance, etc.

    Going even further, if one wants to make conclusions about percentage lift, then we have a different metric: the relative change (percentage change): delta = (pB – pA) / pA for relative change and delta = (pB – pA) / pA * 100% for percentage change, assuming A is the control.

    Due to the division by pA you get higher variance for % lift than for absolute difference, requiring different computations for statistical significance and confidence intervals, as I’ve recently discussed: .

    Keep up the good work!


The Digital Analytics Power Hour © 2021