The 95% Interval That Covers 81% of the Time

A 95% confidence interval promises one thing: over repeated samples, the interval contains the true parameter 95% of the time. For a proportion this is checkable exactly. There are only $n + 1$ possible samples, so you can build the interval for each, ask whether it contains $p$, and add up the binomial probabilities of the ones that do. No simulation, no approximation.

Do that for the standard interval, $\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/n}$, at $n = 30$ and $p = 0.10$:

$$\text{coverage} = 0.8085.$$

One in five samples produces an interval that does not contain the truth. The interval is labelled 95%.

A. Not a small-sample caveat

The usual defence is that 30 is a small sample. Here is the same calculation across a range of settings, with two alternatives alongside:

A hundred observations of a 5% event still gives 0.8775. And the last row is worth noting on its own: at the most favourable proportion there is, with a hundred observations, the interval still covers 94.31% rather than 95%. It is never quite right, and in the corner of the parameter space where most real questions live - rare events, moderate samples - it is not close.

B. More data does not fix it monotonically

Coverage does not climb as $n$ grows. At $p = 0.10$:

Going from 25 observations to 30 takes the coverage from 0.9187 down to 0.8085. Going from 45 to 50 takes it from 0.9356 down to 0.8789.

In the figure, pick 0.10 beside true p and drag sample size n to 30: Coverage at n reads 80.9%, the 0.8085 above.

The oscillation is not noise, because there is no noise here: every entry is an exact sum. It happens because the set of achievable $\hat{p}$ values is discrete, and as $n$ changes, one of the $n + 1$ possible outcomes moves across the boundary of containing $p$ and takes its whole binomial probability with it. Any statement of the form "the interval is fine once $n$ is large enough" has to survive this table, and the usual rules of thumb ($n\hat{p} > 5$, and so on) do not.

C. The degenerate case

Take $n = 40$ and $p = 0.001$. The coverage of the standard interval is

$$0.0392.$$

A 95% interval that contains the truth in 4% of samples. The mechanism is immediate: $P(k = 0) = 0.9608$, and when $k = 0$ the estimate is $\hat{p} = 0$, the standard error is $\sqrt{0 \cdot 1 / n} = 0$, and the interval is the single point $[0, 0]$, which does not contain $0.001$. The formula reports perfect certainty in exactly the situation where it has seen nothing.

This is the case that turns up in production: the rare failure, the rare click, the rare adverse event.

D. What to use instead

Both alternatives in the table are one line and neither needs a new idea.

• Wilson. Invert the test rather than the estimate: keep the values of $p$ the data would not reject. The interval is no longer centred on $\hat{p}$ and it never collapses to a point.
• Agresti-Coull. Add two successes and two failures, then use the ordinary formula on the adjusted counts. It is the Wald interval with the estimate pulled off the boundary, and in the table above it is the more conservative of the two.

The general lesson is the one the exact calculation makes available: coverage is computable, so compute it. For a proportion it is a finite sum. For anything else, simulating the sampling distribution and counting how often the interval contains the value you generated from takes a few lines, and it is the only way to know whether the number on the label is the number you get.