Animal pilot studies should not be used to estimate sample size if effect size and population variance are unknown
Article Sidebar
Main Article Content
Abstract
Reducing the number of animals required for a given experiment is part of the 3Rs strategies for animal welfare. Sample size estimation is a critical step in efficient and ethical experimental design. It is generally believed that pilot studies can be used to estimate sample sizes, which could lead to an overall reduction in the number of animals used. As part of the standard approach to ensuring that a planned animal experiment has sufficient statistical power, estimates of effect size and population variance are required. Here we derive the distribution of the sample size estimator when both effect size and variance are unknown. We show that, in this case, it is not feasible to conduct a preliminary pilot study to estimate the required sample size. Our analysis indicates that the sample size of a useful pilot study will often be much larger than that of the main study itself when the effect size is unknown. Therefore, we conclude that performing pilot studies with the aim of estimating sample size will not help to minimize the overall number of animal experiments in basic or pre-clinical research. A practical example is given, and alternative approaches are proposed and discussed.
Plain language summary
When conducting an experiment, researchers attempt to achieve a high probability of getting conclusive data regarding a treatment effect, i.e., to have high statistical power. This power depends on, among other things, the unknown true effect size of a treatment, the unknown population variability, and the sample size. In animal experiments, ethics require using the smallest sample size necessary to produce adequate power. Estimating this sample size requires assumptions about the unknown true effect size and the population variability. We show mathematically that these assumptions are poorly constrained by data from preliminary pilot studies. We conclude that additional pilot studies should not be conducted as part of the estimation process as the data they provide cannot lead to a reduction in the total number of animal test subjects.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles are distributed under the terms of the Creative Commons Attribution 4.0 International license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is appropriately cited (CC-BY). Copyright on any article in ALTEX is retained by the author(s).
Arain, M., Campbell, M. J., Cooper, C. L. et al. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 10, 67. doi:10.1186/1471-2288-10-67
Arifin, W. N. and Zahiruddin, W. M. (2017). Sample size calculation in animal studies using resource equation approach. Malays J Med Sci. 24, 101-105. doi:10.21315/mjms2017.24.5.11
de Boo, J., Hendriksen, C. (2005). Reduction strategies in animal research: A review of scientific approaches at the intra-experimental., supra-experimental and extra-experimental levels. Altern Lab Anim. 33, 369-377. doi:10.1177/026119290503300404
Browne, R. H. (1995). On the use of a pilot sample for sample size determination. Stat Med 14, 1933-1940. doi:10.1002/sim.4780141709
Bulgren, W. (1971). On representations of the doubly non-central F-distribution. Biometrika 66, 184-186. doi:10.2307/2284870
Cochran, W. (1940). The distribution of quadratic forms in a normal system, with applications to the analysis of covariance. Math Proc Camb Philos Soc. 30, 178-191. doi:10.1017/S0305004100016595
Cocks, K., Torgerson, D. J. (2012). Sample size calculations for pilot randomized trials: A confidence interval approach. J Clin Epidemiol 66, 197-201. doi:10.1016/j.jclinepi.2012.09.002
Fenwick, N., Danielson, P., Griffin, G. (2011). Survey of Canadian animal-based researchers’ views on the three Rs: Replacement, Reduction and Refinement. PLoS One 6, e22478. doi:10.1371/journal.pone.0022478
Festing, M. F. W., Altman, D. G. (2002). Guidelines for the design and statistical analysis of experiments using laboratory animals. ILAR J 43, 244-58. doi:10.1093/ilar.43.4.244
Festing, M. F. W. (2018). On determining sample size in experiments involving laboratory animals. Lab Anim 52, 341-350. doi:10.1177/0023677217738268
Furusawa, C., Suzuki, T., Kashiwagi, A. et al. (2005). Ubiquity of log-normal distributions in intra-cellular reaction dynamics. Biophysics 21, 25-31. doi:10.2142/biophysics.1.25
Johnson, N., Welch, B. (1940). Applications of the non-central t-distribution. Biometrika 31, 362-389. doi:10.1093/biomet/31.3-4.362
Julious, S. A. (2005). Sample size of 12 per group rule of thumb for a pilot study? Pharm Stat 4, 287-291. doi:10.1002/pst.185
Kendall, L. V., Owiny, J. R., Dohm, E. D. et al. (2018). Replacement, Refinement, and Reduction in animal studies with biohazardous agents. Comp Med 59, 177-194. doi:10.1093/ilar/ily021
Koch, A. (1966). The logarithm in biology 1: Mechanisms generating the log-normal distribution exactly. J Theor Biol 12, 276-290. doi:10.1016/0022-5193(66)90119-6
Kraemer, H. C., Mintz, J., Noda, A. et al. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry 63, 484-489. doi:10.1001/archpsyc.63.5.484
Lancaster, G. A., Dodd, S. and Williamson, P. R. (2004). Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 10, 307-312. doi:10.1111/j..2002.384.doc.x
Lehr, R. (1992). Sixteen S-squared over D-squared: A relation for crude sample size estimates. Stat Med 11, 1099-1102. doi:10.1002/sim.4780110811
Mead, R. (1988). The design of experiments: Statistical principles for practical applications. Cambridge: Cambridge University Press. doi:10.1016/0006-8993(71)90358-1
Patnaik, P. B. (1949). The non-central χ^2- and F-distributions and their application. Biometrika 36, 202–232. doi:10.1093/biomet/36.1-2.202
Reynolds, P. S. (2023). Why pilot studies? In: A guide to sample size for animal-based studies. Hoboken: Wiley. doi:10.1002/9781119800002.ch4
Rößler, N., Jungenitz, T., Sigler, A. et al. (2023). Skewed distribution of spines is independent of presynaptic transmitter release and synaptic plasticity, and emerges early during adult neurogenesis. Open Biol 13, 230063. doi:10.1098/rsob.230063
Russell, W. M. S. and Burch, R. L. (1959). The principles of humane experimental technique. London: Methuen. doi:10.5694/j.1326-5377.1960.tb73127.x
Sim, J. and Lewis, M. (2012). The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. J Clin Epidemiol. 65, 301-308. doi:10.1016/j.jclinepi.2011.07.011
Sommer, N., Alebrahimdehkordi, N., Pak, O. et al. (2020). Bypassing mitochondrial complex III using alternative oxidase inhibits acute pulmonary oxygen sensing. Sci Adv 6, eaba0694. doi:10.1126/sciadv.aba0694
Teare, M. D., Dimairo, M., Shephard, N. et al. (2014). Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: A simulation study. Trials 15, 264. doi:10.1186/1745-6215-15-264
Thabane, L., Ma, J., Chu, R. et al. (2010). A tutorial on pilot studies: The what, why and how. BMC Med Res Methodol 10, 1. doi:10.1186/1471-2288-10-1
West, R. M. (2022). Best practice in statistics: The use of log transformation. Ann Clin Biochem. 59, 162-165. doi:10.1177/00045632211050531
Woods, C. G., Fu, J., Xue, P. et al. (2009). Dose-dependent transitions in Nrf2-mediated adaptive response and related stress responses to hypochlorous acid in mouse macrophages. Toxicol Appl Pharmacol. 238, 27-36. doi:10.1016/j.taap.2009.04.007
Yin, Y. (2002). Sample size calculation for a proof of concept study. Biopharm Drug Dispos 12, 267-276. doi:10.1081/BIP-120015748