I just submitted a paper on hidden multiplicity (i.e., a problem of multiple comparisons) in multiway ANOVA. In brief, when a researcher uses a multiway ANOVA without specifying hypotheses a priori, we argue that this researcher is in what Adriaan de Groot (1969) called the “guess” phase of the empirical cycle in which hypotheses are formed rather than tested. And in that case, for a design with, say, two factors, we argue that the family of hypotheses encompasses all hypotheses implied by the design (in this case three: two main effects and one interaction). This inflates the family-wise Type I error to 14% instead of 5% as a result of which the alpha level should be corrected accordingly (multiple methods exist to do this, for example the sequential Bonferroni procedure). By reviewing the 2010 volumes of six flagship journals in psychology, we show that applied researchers are generally not aware of this problem; that they, as a result, do not correct the alpha level; and that applying the sequential Bonferroni correction to a random subsample of articles alters at least one of the substantive conclusions in 45 out of 60 articles considered. We conclude that researchers either have to correct alpha levels when using multiway ANOVA to “see what we can find” (de Groot, 1969), or should preregister their hypotheses. In the latter case, one enters the “predict” phase of the empirical cycle in which hypotheses are tested. In such a confirmative setting, each hypothesis could be argued to constitute its separate family; hence obviating the need to correct the alpha level.
Paper on hidden multiplicity in multiway ANOVA
Reply