Tagged “statistical-power”

2 articles.

11 September 20268 min readExperimentation

The Experiment That Was Going to Win Anyway

A test with 2,000 users per arm reports effects 2.4 times too large. An A/A test checked ten times comes out significant 19% of the time. Twenty independent null metrics produce a winner 64% of the time and twelve null segments 46%. Four numbers, one cause, and the decisions that have to be made before the data arrives.

StatisticsMachine Learning

10 September 202610 min readStatistical Inference

What a Sample Can and Cannot Tell You

Estimators as random variables with distributions of their own, the case where the unbiased estimator is the worse one, what a confidence interval actually promises and the standard interval that delivers 87% where it advertises 95%, and what a p-value is a probability of - every figure computed exactly or by fixed-seed simulation.

StatisticsProbability