Type I vs Type II Error: Alpha, Beta, Power & Examples

A type I error is a false positive: rejecting a null hypothesis that is actually true, such as concluding a drug works when it does not. A type II error is a false negative: failing to reject a null hypothesis that is actually false, such as missing a drug's real benefit. Their probabilities are α (the significance level, conventionally 0.05) and β (conventionally 0.10 to 0.20), and power = 1 − β, the chance of detecting an effect that really exists. A larger sample raises power, and lowering α cuts false positives at the price of more false negatives unless the sample grows.

The 2 × 2 table: your decision vs the truth

The null hypothesis (H₀) says there is no difference or association. A study either rejects it ("statistically significant") or fails to reject it.

Stem algorithm: (1) Did the study reject H₀? (2) Was H₀ actually true? Rejected a true null = type I. Kept a false null = type II. If the study reached the right answer, there is no error, whatever the p-value.

Memory hooks:

• The boy who cried wolf: the villagers first make a type I error (they believe in a wolf that isn't there), then a type II error (they ignore the wolf that is).
• α = false alarm, β = blind spot.
• Diagnostic-test analogy: α is the study's false-positive rate (like 1 − specificity), power is its sensitivity for a real effect, and β is its miss rate (like 1 − sensitivity).

Alpha and the p-value

α is chosen before the study: it is the type I error rate you agree to tolerate. ICH E9, the international guideline for clinical trial statistics, says it is conventionally set at 5% or less. The p-value is computed after the study: the probability of a test statistic at least as extreme as the one observed, assuming H₀ is true. If $p < \alpha$, reject H₀.

What a p-value is not, per the American Statistical Association's 2016 statement:

• Not the probability that H₀ is true, and not the probability that the data were produced by chance alone.
• Not a measure of the size or importance of an effect. A huge trial can make a trivial effect "significant," so statistical significance is not clinical significance.
• Not a verdict on its own: conclusions should not rest only on whether p crosses a threshold, and the Cochrane Handbook calls the 0.05 cutoff itself arbitrary. More misreadings are drilled in the p-value interpretation guide.

Multiple comparisons inflate type I error. Test $k$ independent true null hypotheses at $\alpha = 0.05$ and the chance of at least one false positive is $1 - (1 - 0.05)^k$. With 20 tests that is $1 - 0.95^{20} \approx 0.64$. The Bonferroni correction tests each at $\alpha / k$ (here $0.05 / 20 = 0.0025$), which protects against false positives but is conservative and lowers power. Stem clue: many subgroups or outcomes analyzed, one of them "significant."

Beta, power and what raises power

β is the probability of a type II error; power = 1 − β is the probability of rejecting H₀ when a real effect of the specified size exists. ICH E9 describes β as conventionally set at 10% to 20%, which is why trials are typically powered at 80% to 90%. Power is planned before enrollment, through the sample size calculation.

The sample size formula (NIST) shows every lever at once:

$n \propto \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2 \sigma^2}{\delta^2}$

where $\sigma$ is the standard deviation and $\delta$ the difference you want to detect. Three consequences worth knowing:

• Halve the difference you want to detect and you need 4 times the sample ($n \propto 1/\delta^2$).
• Raising power from 80% to 90% at two-sided $\alpha = 0.05$: $(1.96 + 1.28)^2 / (1.96 + 0.84)^2 \approx 10.50 / 7.84 \approx 1.34$, about 34% more participants.
• Tightening α from 0.05 to 0.01 at 80% power: $(2.58 + 0.84)^2 / (1.96 + 0.84)^2 \approx 11.70 / 7.84 \approx 1.49$, about 49% more participants. At a fixed sample size, the same change raises β instead.

Type II error is the classic failure of small studies. Anything that dilutes the observed effect, including nonadherence in an intention-to-treat analysis, lowers power too. "Not significant" means absence of evidence, not evidence of absence: look at the confidence interval before concluding that a treatment does nothing.

Confidence intervals: significance and precision together

A 95% confidence interval is the range of values compatible with the data. Strictly, if the study were repeated many times, 95% of such intervals would contain the true effect. It pairs with a two-sided $\alpha = 0.05$: the 95% CI excludes the null value if and only if p < 0.05.

• Null value = 0 for differences (mean difference, risk difference) and 1 for ratios (relative risk, odds ratio, hazard ratio).
• Width = precision. Larger studies give narrower intervals.
• A wide interval that includes the null and clinically important effects means an inconclusive, likely underpowered study, the fingerprint of a possible type II error.

Remember that p-values and CIs deal only with random error. A tight, "significant" interval can still be wrong because of bias or confounding, which no amount of statistics fixes.

Rapid-fire exam scenarios

Practice question

Vignette. A randomized trial assigns 60 adults with hypertension to a new drug or placebo. At 12 weeks, systolic blood pressure falls 6 mm Hg more with the drug than with placebo (95% CI −2 to 14 mm Hg; p = 0.14). The authors conclude that the drug does not lower blood pressure. A later trial of 2,400 patients finds a 5 mm Hg greater reduction with the drug (95% CI 4 to 6 mm Hg). Which of the following best describes the first trial's conclusion?

A. Type I error B. Type II error C. A correct conclusion, because p > 0.05 D. An error caused by setting α too high E. Confounding by baseline blood pressure

Answer: B. Type II error. The first trial failed to reject a null hypothesis that was false. With 60 patients, its power to detect a 5 to 6 mm Hg difference was low, and its interval, which runs from a small increase to a 14 mm Hg reduction, was compatible with a clinically important benefit.

Why not the others?

• Type I error means rejecting a true null. The first trial rejected nothing, and the larger trial shows the effect is real.
• "Correct because p > 0.05" confuses absence of evidence with evidence of absence.
• α set too high would make rejection easier and raise the type I error rate. The problem here is too little power, not too much α.
• Confounding is a systematic bias, which randomization guards against. The first trial's problem is imprecision (a wide interval from a small sample), a random error that a larger sample fixes.

Key takeaways

• Type I = false positive (reject a true H₀), probability α. Type II = false negative (fail to reject a false H₀), probability β.
• Power = 1 − β, conventionally 80% to 90%. It rises with a larger sample, a larger effect, less variability and a higher α.
• Lowering α reduces type I errors but raises type II errors unless the sample grows.
• p < α means reject H₀; the p-value is not the probability that H₀ is true and says nothing about effect size.
• 95% CI excludes the null value (0 for differences, 1 for ratios) if and only if p < 0.05; width reflects precision.
• Multiple comparisons inflate false positives: $1 - 0.95^{20} \approx 0.64$ for 20 independent tests; Bonferroni uses $\alpha / k$.

Sources

• Outcomes and the Type I and Type II Errors (Introductory Statistics 2e, section 9.2) — OpenStax (2023)
• What are statistical tests? (section 7.1.3) — NIST/SEMATECH e-Handbook of Statistical Methods
• Critical values and p values (section 7.1.3.1) — NIST/SEMATECH e-Handbook of Statistical Methods
• Confidence interval approach (section 7.2.2.1) — NIST/SEMATECH e-Handbook of Statistical Methods
• Sample sizes required (section 7.2.2.2) — NIST/SEMATECH e-Handbook of Statistical Methods
• ICH E9: Statistical Principles for Clinical Trials — European Medicines Agency (1998)
• Statement on Statistical Significance and P-Values — American Statistical Association (2016)
• Schünemann HJ, et al. Chapter 15: Interpreting results and drawing conclusions — Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024)
• Bland JM, Altman DG. Multiple significance tests: the Bonferroni method — BMJ (1995)
• Altman DG, Bland JM. Absence of evidence is not evidence of absence — BMJ (1995)