A result is called statistically significant when it's unlikely to have happened by random chance alone. That's the whole idea in one sentence — but the details are where people trip up.
The logic in three steps
- You assume there's no real effect — the null hypothesis (e.g. the two groups are actually the same).
- You calculate the p-value: the probability of seeing data at least as extreme as yours if the null were true.
- If that probability is below a threshold you set in advance — the significance level, α (usually 0.05) — you call the result significant and reject the null.
Significant = "if nothing were really going on, data like this would be rare (rarer than our α cutoff), so we don't think nothing is going on."
What α = 0.05 actually commits you to
Setting α = 0.05 means you accept a 5% chance of a false positive — declaring an effect that isn't real (a Type I error). It's a convention, not a natural law; stricter fields use 0.01 or lower. "Significant at the 0.05 level" simply means your p-value came in under 0.05.
The trap: significant ≠ important
This is the single most misunderstood point. Statistical significance says an effect probably exists — it says nothing about how big or useful it is. With a large enough sample, a difference so tiny it's meaningless in practice will still come out "significant." That's why you should always pair significance with an effect size and a confidence interval.
| Statistical significance | Practical significance |
|---|---|
| "The effect is probably not zero" | "The effect is big enough to matter" |
| Measured by the p-value | Measured by effect size |
| Depends heavily on sample size | Independent of sample size |
What a non-significant result means
Failing to reach significance (p > α) does not prove there's no effect. It means you didn't gather enough evidence — possibly because the effect is small, or because your sample was too small to detect it (low statistical power). "Not significant" and "no effect" are not the same claim.
Example. A new fertilizer raises yield by 0.1% with p = 0.001 across a million plots — highly significant, commercially pointless. Another raises yield 20% with p = 0.08 across 12 plots — not significant, but very promising and worth a bigger trial. Significance alone would rank these backwards.
Test your own data
Run a t-test →Get the p-value and a plain-English significance verdict — then check the effect size too.Related reading: Interpreting p-values · Effect size explained · Type I & II errors
Sources & further reading: NIST/SEMATECH e-Handbook of Statistical Methods · OpenStax Introductory Statistics. How we compute & validate: methodology. Last reviewed August 2026.