Three questions decide the test
Almost every choice between common statistical tests comes down to three questions, which is what the chooser above asks:
- What is your goal? Comparing groups, measuring a relationship between two variables, or predicting one variable from others.
- What type is your outcome variable? Continuous measurements (height, score, time), categories (yes/no, brand chosen), or ranks.
- How are the groups related? Independent groups (different people in each) or paired data (the same people measured twice, or matched pairs).
Quick reference
| Your situation | Usual test | If assumptions fail |
|---|---|---|
| Compare one group's mean with a known value | One-sample t-test | Wilcoxon signed-rank test |
| Compare means of two independent groups | Two-sample t-test (Welch) | Mann–Whitney U test |
| Compare the same people before and after | Paired t-test | Wilcoxon signed-rank test |
| Compare means of three or more groups | One-way ANOVA | Kruskal–Wallis test |
| Compare two proportions (e.g. conversion rates) | Two-proportion z-test | Fisher's exact test (small samples) |
| Is there an association between two categorical variables? | Chi-square test of independence | Fisher's exact test (small expected counts) |
| Strength of a relationship between two continuous variables | Pearson correlation | Spearman rank correlation |
| Predict a continuous outcome from one or more predictors | Linear regression | Transform the outcome or use robust regression |
| Is an observed success count consistent with a known rate? | Binomial test | — |
| Compare two classifiers on the same test set | McNemar's test | Paired bootstrap |
Parametric or nonparametric?
Parametric tests such as the t-test and ANOVA compare means and assume the data in each group are roughly normal. With moderate samples (around 30 or more per group) they are robust to mild skew because of the Central Limit Theorem. Nonparametric tests such as Mann–Whitney and Kruskal–Wallis compare ranks instead, so they cope with strong skew, outliers and ordinal data such as Likert ratings, at the cost of a little power when the data really are normal. If your sample is small and clearly skewed, or your outcome is a rating scale, the nonparametric option is usually safer.
Two common traps
- Running many t-tests instead of one ANOVA. With four groups there are six pairwise comparisons, and the chance of at least one false positive rises from 5% to about 26%. See ANOVA vs t-test.
- Treating paired data as independent. Before-and-after scores on the same people are correlated; an independent-samples test ignores this and usually loses power.
For a longer walkthrough with a flowchart, read the guide to choosing a statistical test.