Computation of measures of effect size for neuroscience data sets

Abstract

The overwhelming majority of research in the neurosciences employs P-values stemming from tests of statistical significance to decide on the presence or absence of an effect of some treatment variable. Although a continuous variable, the P-value is commonly used to reach a dichotomous decision about the presence of an effect around an arbitrary criterion of 0.05. This analysis strategy is widely used, but has been heavily criticized in the past decades. To counter frequent misinterpretations of P-values, it has been advocated to complement or replace P-values with measures of effect size (MES). Many psychological, biological and medical journals now recommend reporting appropriate MES. One hindrance to the more frequent use of MES may be their scarcity in standard statistical software packages. Also, the arguably most widespread data analysis software in neuroscience, matlab, does not provide MES beyond correlation and receiver-operating characteristic analysis. Here we review the most common criticisms of significance testing and provide several examples from neuroscience where use of MES conveys insights not amenable through the use of P-values alone. We introduce an open-access matlab toolbox providing a wide range of MES to complement the frequently used types of hypothesis tests, such as t-tests and analysis of variance. The accompanying documentation provides calculation formulae, intuitive explanations and example calculations for each measure. The toolbox described is usable without sophisticated statistical knowledge and should be useful to neuroscientists wishing to enhance their repertoire of statistical reporting.