Science is currently facing a ‘replication crisis’ – a concern that many scientific findings reported are difficult or impossible to reproduce. A major cause of this is the availability of technology that permits the exploration and testing of very large numbers of hypotheses, some of which will almost certainly show large or significant effects by chance, even when no real effects are present: this is the ‘multiplicity’ or ‘multiple testing’ problem. The statistical tools available to address this problem include:
• the False Discovery Rate (FDR), which is specified in relation to the subset of the m hypotheses tested for which the discovery of an effect is reported, and which indicates the proportion of these ‘discoveries’ that is expected to be false; and
• shrunk estimates, which reduce the estimated effect, in relation to every individual hypothesis, from the observed value towards the null value.
This talk will first examine the conceptual basis for each of these tools, then consider how they are connected. Though the FDR and shrunk estimates are both conventionally presented in the frequentist statistical framework, they can both also be presented in empirical-Bayesian terms, the prior probability distribution being calculated from the data relating to the m hypotheses tested, as follows:
• in the case of the FDR, from the proportion of the m significance tests conducted that give a p-value at or below the specified significance threshold, and that are therefore announced as ‘discoveries’; and
• in the case of shrunk estimates, from the distribution of the observed effect sizes over the m hypotheses.
Based on this connection, a formal relationship between FDR values and shrunk estimates will be presented. It will be argued that these two tools can profitably be used in conjunction, and their combined application, both to real (human gene expression) data and to simulated data, will be illustrated.
