Confidence Intervals and Precision

#An estimate is not an exact answer

Studies usually observe a sample rather than every person in a population. The result, such as an average difference or risk ratio, is therefore an estimate. Another sample studied in the same way could produce a different number simply because of the variation involved in sampling.

#What an interval describes

A confidence interval accompanies an estimate with a range derived using a statistical method. Under that method's assumptions, it helps show which effect sizes are more compatible with the observed data. A narrow interval indicates greater precision; a wide interval leaves a broader range of possibilities.

For a standard 95% confidence interval, the 95% describes the long-run performance of the procedure: across repeated studies conducted under its assumptions, about 95% of the resulting intervals would contain the true value. It is not a 95% probability statement about that particular calculated interval.

#Precision is not the same as importance

Larger studies and more informative measurements often improve precision. For some outcomes, the number of events matters more than the total number of participants. Precision can also depend on variation between people, missing observations and the study design, including whether participants were studied in groups.

An interval should be read in relation to effects that would matter in practice. If it includes both an important benefit and an important harm, the evidence is inconclusive. If it excludes a no-effect value but contains only very small differences, the result may have little practical importance.

#Statistical uncertainty has limits

Confidence intervals generally do not account for all bias, unmeasured confounding or problems applying findings elsewhere. A very precise estimate can still be systematically wrong. Conversely, a result that is not statistically significant does not prove there is no effect; the interval may simply be too wide.

#Common misunderstandings

A 95% confidence interval does not mean that 95% of patients will have an outcome within that range. It describes uncertainty around an estimated quantity, such as an average difference or a relative risk, rather than the spread of individual experiences.

It also does not mean there is a 95% probability that the true value lies inside this particular interval. In the usual statistical interpretation, the method would produce intervals containing the true value in 95% of repeated studies, provided its assumptions hold.

An interval that includes “no difference” does not prove that treatments are equivalent. It may still include meaningful benefit and meaningful harm. Likewise, an interval that excludes “no difference” does not establish that the effect matters clinically.

Finally, a narrow interval is not a guarantee of accuracy. A large study can produce a precise estimate while still being affected by biased measurement, missing data or other design problems.

#Questions worth asking a clinician

• What does this confidence interval tell us about the range of effects compatible with the data and the analysis assumptions?
• Which assumptions does this confidence interval rely on, and how might violations change its interpretation?
• Could bias make this estimate misleading even though its confidence interval is narrow?
• Does this statistically significant result represent a meaningful health benefit, or does the confidence interval include effects too small to matter?
• For this non-significant result, does the confidence interval still include clinically important benefit or harm?