How do I choose which statistical test to use?

Test selection flowcharts are useful and they hide the important point, which is that the test follows from the design rather than the other way round. If you find yourself choosing a test after data collection and hoping one fits, the difficulty started earlier, at the point where the analysis was not planned alongside the design.

Working through the four questions is usually enough. A continuous outcome, two independent groups, roughly normal residuals gives you an independent samples t test. The same outcome measured twice on the same people gives you a paired test or a mixed model. A binary outcome with several predictors gives you logistic regression. Most of applied statistics is this narrowing.

Worked examples make the narrowing concrete. To compare diastolic blood pressure across three age groups, one-way ANOVA fits when the values are normally distributed, and the Kruskal-Wallis H test or a median test fits when they are not. To compare mean arterial pressure between a treatment group and a control group, the independent samples t test fits normally distributed values and the Mann-Whitney U test fits non-normal ones. Pairing is a separate question from normality: measurements taken from the same subjects at two times or by two methods call for a paired samples t test, and different subjects in each group call for the independent samples test (Mishra et al., 2019, p. 298). The same authors point out that SPSS, R, Stata, and SAS make the analysis easy to run without solving the selection problem, and that running an unpaired t test on paired data or a parametric test on non-normal data are common errors in published biomedical research (Mishra et al., 2019, p. 297).

The question people skip is independence. Data collected in clusters, classrooms, clinics, households, repeated occasions, violates the independence assumption of the simple tests, and ignoring the clustering produces standard errors that are too small and results that look more significant than they are. If your data has structure, the model needs to know about it.

Where the decision genuinely is not obvious, ask a statistician before collecting data rather than after. Most institutions have a consulting service, and the conversation is short when the design is still changeable and long when it is not.

Which questions decide the answer?

Outcome type first, since continuous, count, binary, and ordinal outcomes lead to different families. Then the number and structure of groups. Then independence, because repeated measures on the same participants require paired or mixed models. Then assumptions, which decide between the parametric test and its alternative.

Write your answers to the four questions into the methods chapter as a short paragraph. It makes the choice legible to a reader and it is the passage an examiner will use to check that you understood why the test fits.

For a categorical outcome, the number of categories and the dependence structure decide together. Pearson's chi-square or Fisher's exact test covers proportions and associations across two or more independent groups, McNemar's test covers two paired groups, and Cochran's Q test covers three or more paired groups. When you are predicting rather than comparing, binary logistic regression handles a two-category outcome and multinomial logistic regression handles an outcome with three or more categories (Mishra et al., 2019, pp. 298-300). Zitomer et al. extend the same logic to distributional form rather than the categorical label alone, naming logistic regression and Fisher's exact test for binary data, Poisson or negative binomial regression for counts, and Dirichlet regression for proportions (Zitomer et al., 2022, p. 5).

Note also what you are estimating, not just what you are testing. A test gives you a p value, and a reader usually wants the effect size and its confidence interval, which comes from the model rather than the test.

Decide all of this while you are still planning, because sample size is set by the effect you expect rather than by the test you pick. There is no fixed minimum sample size attached to any particular test. The number follows from the effect size, the confidence level, and the power you target, with 95% confidence and 80% power the usual settings, and a larger expected effect needing a smaller sample (Mishra et al., 2019, p. 300). Power rises with sample size and effect size and falls when variability is high, so a large enough sample will return a significant p value for an effect too small to matter (Zitomer et al., 2022, p. 6). Report the estimate instead. One of their examples states that a species was 3.2 times more likely to use a treatment site than a control site, with a 95% confidence interval of 1.3 to 4.6, which tells a reader far more than p < .01 (Zitomer et al., 2022, p. 6).

What do I do when the assumptions fail?

You have three routes: transform the outcome, use the nonparametric equivalent, or use a model that assumes the distribution you actually have. Report which you chose and why. Running the parametric test anyway and not mentioning the violation is the option that causes trouble in review.

Check assumptions with plots rather than only with significance tests of normality, since those tests are oversensitive in large samples and underpowered in small ones. A residual plot tells you more than a p value about whether a model fits.

Treat the parametric and nonparametric options as a conditional choice rather than a ranking. Parametric methods are generally better at detecting group differences, and they ask more of the data in return, including approximate normality and an adequate sample size. Means are also strongly affected by outliers, while medians and mean ranks are affected much less (Mishra et al., 2019, p. 299). Zitomer et al. agree that nonparametric methods carry fewer assumptions and add the qualification that they usually have lower power and that every alternative has assumptions of its own (Zitomer et al., 2022, p. 5). Kass et al. extend the list of assumption violations beyond normality, to missing data, measurement bias, nonlinearity, and dependence, and conclude that you should read data displays, residuals, and measures of fit rather than accept what the software prints (Kass et al., 2016, pp. 5-6).

Sensitivity analysis is the strongest response when you are unsure. Run the analysis both ways and report that the conclusion did or did not change. If it did not, the assumption question stops mattering. If it did, that is something a reader needs to know.