You ran an ANOVA—and it was significant! Now, of course, you want to know exactly which groups differ. The first impulse: Simply run a few t-tests between the groups. But beware!
Alpha error inflation
When you conduct multiple hypothesis tests simultaneously, the probability of making at least one Type I error (a falsely significant result) increases—that is, the so-called alpha error.
A simple example:
You compare three groups (A, B, C). To test all possible differences, you need three t-tests:
- A vs. B
- A vs. C
- B vs. C
If you choose a significance level of $\alpha = 0{,}05$ for each test, the probability that at least one of these tests is falsely significant is clearly above 5%.
More precisely:
Here, k is the number of tests. For three tests:
So there is a 14% probability that you will get a randomly significant result, even though there is no genuine difference!
How can alpha error inflation be avoided?
That is why there are post hoc tests that control the accumulation of alpha errors, for example:
- Tukey HSD (for all pairwise comparisons with equal group sizes)
- Bonferroni correction (conservative and easy to implement)
- Scheffé test (for arbitrary contrasts)
These tests adjust the significance level or calculate adjusted p-values so that you do not draw false conclusions.
In short:
If you want to compare several groups, run an ANOVA first.
If it is significant: use post hoc tests, but please do not simply run lots of t-tests! Otherwise, you will fall into the trap of inflating the Type I error rate.