Confidence Interval

An estimate on its own says nothing about its own precision. A confidence interval attaches that missing information by reporting a range instead of a point, together with a promise about how the range was built: apply this procedure to sample after sample, and the stated fraction of the resulting intervals will contain the true value.

The promise is about the procedure, which is the part most easily misread. Once the data are in hand and the numbers are fixed, the interval either covers the truth or it does not, and no probability remains to be assigned. Saying there is a 95% chance the parameter lies inside a computed interval treats a fixed unknown as if it were random. The usable reading is the frequency one: intervals built this way are right 95% of the time, and this is one of them.

Two sources of uncertainty enter when the population spread is unknown, and using the normal quantile accounts for only the first. At a sample of 10 the normal quantile 1.959964 gives an exact coverage of 0.918351 rather than 0.95, because the estimate of the spread is itself uncertain; the t quantile 2.262157 is the correction, and for normally distributed observations it is exact rather than conservative. The gap narrows as data accumulate: at 29 degrees of freedom the quantile is already 2.045230.

For a proportion, the textbook interval fails in a way that is worth seeing in full, because the same unlucky sample both moves the centre and mis-sizes the width. At n = 50, exact enumeration of all 51 outcomes gives coverage 0.878917 at p = 0.1 and 0.394848 at p = 0.01 - the latter because 0.605006 of such samples contain no successes at all, making the estimated standard error exactly zero and collapsing the interval to a single point. The Wilson interval, which does not estimate the width from the same slip, gives 0.970308 where the standard one gives 0.878917.

Twenty thousand simulated samples of size 10 from a normal population, each giving a 95% t-interval for the mean, cover the true mean 0.95 of the time. Replacing the t quantile with the normal one drops the exact coverage to 0.918351.

The standard interval for a proportion at n = 50, evaluated exactly over all 51 possible outcomes: 0.919867 at p = 0.05, 0.878917 at p = 0.1, 0.937531 at p = 0.2 and 0.935091 at p = 0.5. The Wilson interval on the same points gives 0.962224, 0.970308, 0.950701 and 0.935091 - identical at p = 0.5, materially better where the standard one fails.

Coverage is not monotone in the sample size. At p = 0.2 the standard interval covers 0.951214 at n = 23 and 0.872862 at n = 24: one additional observation costs nearly eight points. Coverage falls from one size to the next at 29 of the 180 steps between 20 and 200, though only three of those falls exceed five points.