Combining Dependent p-values: Methods and Properties from a Practical Perspective

Testing the global null hypothesis, that all of a collection of m null hypotheses are true, is a central task in statistical inference. This paper reviews recent and established tests that approach this task by combining the m hypotheses' p-values, focusing on the situation that these p-values are dependent and that the precise form of this dependence is unknown. We provide guidelines for choosing p-value combination functions based on theoretical results and numerical experiments. Bonferroni and Simes represent the classical approaches; the former offers uniform validity against any dependence structure, while the latter is robust to many practical dependencies and uniformly more powerful. We also discuss the merits and promises of novel approaches, such as the Cauchy and harmonic mean p-value combination function, twice the median and the recent uniform improvement of Hommel's procedure. The permutation approach is recommended whenever the null distribution is invariant to permutations of the data. Finally, we emphasize that global testing is often just a single but important step in a bigger inferential framework, and illustrate the utility of such tests within modern statistical practice.