Picking the wrong statistical test is one of the most common mistakes in student projects, dissertations, and even published papers. The good news: for the vast majority of real analyses, the correct test is determined by just three questions about your data. This guide walks through them and links you straight to a free calculator for each test.
The three questions that decide almost everything:
- What kind of outcome are you measuring — a number, or a category?
- How many groups or variables are you comparing?
- Is your data roughly normally distributed, or skewed / ordinal?
Step 1 — Is your outcome numeric or categorical?
A numeric (continuous) outcome is something you measure on a scale: height, reaction time, test score, blood pressure, revenue. A categorical outcome is a label or count: pass/fail, yes/no, which of three brands a person prefers.
This single distinction splits the entire decision tree. Numeric outcomes lead to means-based tests (t-tests, ANOVA, correlation, regression). Categorical outcomes lead to proportion- and frequency-based tests (chi-square, two-proportions z-test, binomial).
Step 2 — How many groups or variables?
Once you know your outcome type, count what you're comparing:
- One group vs a known value — e.g. "is our class average different from the national average of 70?"
- Two groups — e.g. treatment vs control, men vs women.
- Three or more groups — e.g. three teaching methods.
- Two numeric variables measured together — e.g. hours studied vs exam score (a relationship, not groups).
Step 3 — Is the data normal?
Many classic tests assume the outcome is approximately normally distributed. When that assumption is badly violated — heavily skewed data, ordinal scales like Likert responses, small samples with outliers — you switch to a non-parametric alternative that ranks the data instead of using raw values. More on this below.
The decision table
Find the row that matches your situation. Each links to a calculator that runs the test and interprets the result for you.
| Your situation | Use this test |
|---|---|
| One numeric sample vs a known/target mean | One-sample t-test |
| Two independent numeric groups (normal) | Independent t-test |
| Two independent groups (skewed / ordinal) | Mann-Whitney U test |
| Same subjects measured twice (before/after) | Paired t-test |
| Three or more numeric groups | One-way ANOVA |
| Relationship between two numeric variables | Correlation / Regression |
| Association between two categorical variables | Chi-square test |
| Compare two proportions / conversion rates | Two-proportions z-test |
| Probability of X successes in N trials | Binomial distribution |
| Where a value sits in a distribution | Z-score / Normal distribution |
| Estimate a range for a mean or proportion | Confidence interval |
| How many participants do I need? | Sample size calculator |
Parametric vs non-parametric: the most common fork
The decision people agonize over most is t-test vs its non-parametric cousin. Here's the short version:
- Use the independent t-test when your two groups are roughly normal and reasonably sized (say 30+ per group). It's slightly more powerful.
- Use the Mann-Whitney U test when the data is ordinal, clearly skewed, or the sample is small and you can't verify normality. It compares ranks, so outliers barely affect it.
When in doubt with small or messy data, the non-parametric test is the safer default — it makes fewer assumptions and rarely misleads you.
Don't stop at the p-value
A statistically significant result (small p-value) tells you an effect probably exists — not that it's large or important. Always report an effect size alongside your test, and consider a confidence interval to show the range of plausible values. A significant result with a trivial effect size is usually not worth acting on.
Quick worked example. You ran an A/B test: 45 of 100 visitors converted on version A, 35 of 100 on version B. Outcome is categorical (converted / didn't), you have two groups, so you want the two-proportions z-test — not a t-test. It will tell you whether the 10-point gap is real or just noise.