Experimentation and A/B Testing
What an online experiment reports when it is too small, watched too often, or read across too many metrics: an effect inflated 2.4 times, a false positive rate of 19% instead of 5%, and a winning segment in almost half of all experiments where nothing happened.
Sign in to take quizzes, earn XP, and unlock stages as you reach 90% mastery.
Power and the Winner’s Curse
25 min · 100 XPWhy the sample size has to be fixed before the test starts, what an underpowered experiment reports when it does reach significance, and the reason halving the effect you care about quadruples the traffic you need.
Open lesson →Sign in to take the 3-question quiz.
Peeking and the Dashboard of Metrics
30 min · 120 XPA test with no effect at all, watched ten times and stopped when it looks good, comes out significant 19% of the time; the same arithmetic applied to twenty independent metrics gives 64%. Both, and what to do instead.
Open lesson →Sign in to take the 3-question quiz.
Variance Reduction and Reading the Result
30 min · 120 XPUsing what you already knew about each user to halve the traffic an experiment needs, why slicing a null result by segment finds a winner 46% of the time, and turning a measured effect into a decision.
Open lesson →Sign in to take the 3-question quiz.