Everything You Need to Know About P-value interpretation for Step 1

P-values show up on Step 1 the same way they show up in real life: as a single number people treat like a “truth stamp.” Your job is to interpret what it actually means, recognize common traps (especially “$p<0.05$ = important”), and connect it to study design, error types, and confidence intervals.

---

What a P-value is (and what it is not)

Definition (memorize this)

A p-value is the probability of observing results at least as extreme as what you observed, assuming the null hypothesis is true.

• In symbols: $p = P(\text{data as/extremer} \mid H_0 \text{ true})$
• Null hypothesis ($H_0$): typically “no difference” or “no association”
• Alternative hypothesis ($H_1$): “there is a difference/association”

What a p-value is not

Common USMLE traps:

---

Step 1 “pathophysiology”: what generates a p-value?

Not pathophysiology in the disease sense—but Step 1 loves mechanisms. Think of p-values as coming from a hypothesis test pipeline:

1. Assume $H_0$ is true
2. Choose a test statistic (e.g., $t$, $\chi^2$, $z$)
3. Use its sampling distribution under $H_0$
4. Compute the probability of your statistic being as extreme (tail area) → p-value

One-tailed vs two-tailed (frequent exam pitfall)

• Two-tailed: detects difference in either direction (most common in clinical research)
• One-tailed: detects difference in one direction only (rare; must be pre-specified)

High-yield: A two-tailed p-value is basically “both tails combined.” If a question hints the researchers “looked for any difference,” think two-tailed.

---

The $0.05$ cutoff: what “statistical significance” actually means

Alpha ($\alpha$) is the threshold, set before you look

• $\alpha$ = probability of Type I error you’re willing to tolerate
• Commonly: $\alpha=0.05$

If $p < \alpha$, you reject $H_0$.

Type I vs Type II errors (must-know table)

High-yield connection:

• Lowering $\alpha$ (e.g., from 0.05 to 0.01) → decreases Type I error but tends to increase Type II error unless sample size increases.

---

Clinical “presentation”: how p-values show up on test day

You’ll usually see p-values in these contexts:

1) “Is this result statistically significant?”

• If $p < 0.05$ → “significant” (unless they specify a different $\alpha$)

2) Comparing p-value with confidence interval (CI)

This is a classic integration point.

• If the 95% CI excludes the null value, then typically $p<0.05$
  • For differences of means: null value is 0
  • For ratios (RR, OR, HR): null value is 1

Example logic:

• OR = 0.7, 95% CI (0.4–1.2) → includes 1 → not significant (often $p>0.05$)
• Mean difference = 5 mmHg, 95% CI (2–8) → excludes 0 → significant

3) “Big study, tiny p-value, tiny effect”

USMLE loves the idea that:

• Large $n$ can make trivial effects “significant”
• Clinical relevance needs effect size + patient-centered outcome

---

Diagnosis: a Step-wise approach to interpreting a p-value vignette

When you see a p-value in a question stem, run this quick checklist:

Step 1: Identify the null and alternative

• $H_0$: no difference/no association
• $H_1$: difference/association exists

Step 2: Compare p to $\alpha$

• If $p < \alpha$ → reject $H_0$
• If $p \ge \alpha$ → fail to reject $H_0$

Step 3: Translate into plain English

• Avoid saying “accept $H_0$”
• Better: “There is/ isn’t sufficient evidence…”

Step 4: Check design and context

• Randomization? Blinding?
• Confounding? Bias?
• Multiple comparisons?

---

Treatment: how to “fix” p-value problems (study design moves)

You can’t “treat” a p-value, but you can treat the research scenario that creates misleading p-values.

If p is non-significant but you suspect a real effect:

Consider low power (high $\beta$). How to increase power:

• Increase sample size
• Increase effect size (hard in real life; sometimes by enriching high-risk populations)
• Reduce variability (better measurement, narrower inclusion criteria)
• Use a higher $\alpha$ (rarely justified; increases Type I error)

If p is significant but you’re worried it’s misleading:

Think false positives, bias, and p-hacking:

• Multiple comparisons inflate Type I error
  • Example: test 20 outcomes at $\alpha=0.05$ → expect ~1 false positive by chance
• Solutions:
  • Pre-specify primary outcome
  • Correct for multiple testing (e.g., Bonferroni; often beyond Step 1 details but concept is fair game)
  • Replication studies

---

High-yield associations: p-values + common Step 1 biostats pairings

P-value vs confidence interval (CI)

• CI communicates range and precision
• p-value communicates compatibility with $H_0$
• USMLE prefers that you know:
  • Wide CI → imprecise estimate (often small sample)
  • Narrow CI → precise estimate (often large sample)

P-values and sample size ($n$)

• Larger $n$ → smaller standard error → easier to get small p-values
• But: does not guarantee clinical importance

P-value and effect size

• Effect size examples:
  • Mean difference
  • Relative risk (RR)
  • Odds ratio (OR)
  • Hazard ratio (HR)
• Best practice: report effect size + CI; p-value is supplemental

P-values in common tests (you don’t need calculations, but know matching)

---

Common “gotchas” that Step 1 tests

1) “Fail to reject $H_0$” ≠ “prove no difference”

Non-significant findings can happen with:

• Underpowered study
• High variance
• Wrong population
• Poor measurement

2) Statistical significance ≠ clinical significance

Example:

• Drug lowers SBP by 1 mmHg with $p<0.001$ in 50,000 patients
  → statistically significant, clinically meh.

3) Multiple comparisons

If they measured tons of endpoints and celebrate one significant p-value, suspect:

• Increased Type I error
• Need for correction / skepticism about the finding

4) P-values don’t correct bias

A perfectly “significant” p-value can come from:

• Confounding (no randomization)
• Selection bias
• Information bias (misclassification)
• Loss to follow-up (attrition bias)

---

Quick clinical-style vignettes (USMLE flavor)

Vignette 1: “p=0.04”

A randomized trial shows mortality 10% vs 13%, $p=0.04$.

• Interpretation: If no true mortality difference existed, there’s a 4% chance of seeing a difference this large or larger.
• Conclusion: Reject $H_0$ at $\alpha=0.05$ (statistically significant).

Vignette 2: “p=0.20”

Small pilot study: symptom score improves, but $p=0.20$.

• Interpretation: Insufficient evidence to reject $H_0$.
• Next best step: Consider increasing sample size / power rather than claiming “no effect.”

Vignette 3: “CI crosses the null”

OR 0.6, 95% CI 0.3–1.1.

• Interpretation: CI includes 1 → not statistically significant at 0.05 (typically).

---

First Aid cross-references (Biostatistics)

Page numbers vary by edition, but these topics are consistently in the Biostatistics section of First Aid:

• Type I vs Type II error, $\alpha$, $\beta$, power
• Confidence intervals and relationship to statistical significance
• t-test, chi-square, ANOVA (test selection)
• Sensitivity/specificity and predictive values (often tested near the same cluster—even though not p-values, students mix them up)

How to use First Aid here: treat p-values as part of the “hypothesis testing bundle” with $\alpha$, $\beta$, and power, then connect to CI interpretation and basic test selection.

---

Ultra-high-yield takeaways (write these on your mental whiteboard)

• p-value = $P(\text{data} \mid H_0)$, not $P(H_0 \mid \text{data})$
• $p<\alpha$ → reject $H_0$; $p\ge\alpha$ → fail to reject $H_0$
• 95% CI excludes null value (0 for differences, 1 for ratios) ↔ often $p<0.05$
• Large sample sizes can make tiny effects “significant”
• p-values do not fix bias, confounding, or bad design
• Multiple comparisons inflate false positives