Systematic review in evidence-based risk assessment
Article Sidebar
Main Article Content
Abstract
Systematic reviews provide a structured framework for summarizing the available evidence in a comprehensive, objective, and transparent manner. They inform evidence-based guidelines in medicine, public policy, and more recently, in environmental health and toxicology. Many regulatory agencies have extended and adapted the well-established systematic review methods, initially developed for clinical studies, for their assessment needs. The use of systematic reviews to summarize evidence from existing human, animal, and mechanistic studies can reduce reliance on animal test data in risk assessment and can help avoid unnecessary duplication of animal experiments that have already been conducted. As alternative test methods can be expected to play an increasing role in human health risk assessment in the future, systematic reviews can be particularly helpful in validating these alternatives. The field of evidence-based toxicology has undergone extensive development since its first meeting in 2007 as a result of collaborative efforts among international experts and public health agencies, particularly with respect to the use of mechanistic data and evidence integration. The continued development and wider adoption of systematic review methodology can lead to better 3R implementation. As undertaking a systematic review can be a complex and lengthy process, it is important to understand the main steps involved. Key steps, along with current best practices, are described with references to guidance from organizations with expertise in evidence synthesis. Applications of systematic reviews in clinical, observational, and experimental studies are presented. Finally, software tools available to facilitate and increase the efficiency of completing a systematic review are described.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles are distributed under the terms of the Creative Commons Attribution 4.0 International license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is appropriately cited (CC-BY). Copyright on any article in ALTEX is retained by the author(s).
AHRQ (2014). Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Publication No. 10(14)-EHC063-EF. Rockville, MD, USA: Agency for Healthcare Research and Quality.
Andersen, M. E., McMullen, P. D., Phillips, M. B. et al. (2019). Developing context appropriate toxicity testing approaches using new alternative methods (NAMs). ALTEX 36, 523-534. doi:10.14573/altex.1906261
Arbyn, M., Xu, L., Simoens, C. et al. (2018). Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev 5, CD009069. doi:10.1002/14651858.CD009069.pub3
Aromataris, E. and Munn, Z. (2017). Joanna Briggs Institute Reviewer’s Manual. The Joanna Briggs Institute. https://reviewersmanual.joannabriggs.org/
Banwell, V., Sena, E. S. and Macleod, M. R. (2009). Systematic review and stratified meta-analysis of the efficacy of interleukin-1 receptor antagonist in animal models of stroke. J Stroke Cerebrovasc Dis 18, 269-276. doi:10.1016/j.jstrokecerebrovasdis.2008.11.009
Beronius, A., Molander, L., Zilliacus, J. et al. (2018). Testing and refining the science in risk assessment and policy (SciRAP) web-based platform for evaluating the reliability and relevance of in vivo toxicity studies. J Appl Toxicol 38, 1460-1470. doi:10.1002/jat.3648
Bronsveld, H. K., ter Braak, B., Karlstad, Ø. et al. (2015). Treatment with insulin (analogues) and breast cancer risk in diabetics; a systematic review and meta-analysis of in vitro, animal and human evidence. Breast Cancer Res 17, 100. doi:10.1186/s13058-015-0611-2
Buscemi, N., Hartling, L., Vandermeer, B. et al. (2006). Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol 59, 697-703 doi:10.1016/j.jclinepi.2005.11.010
Carver, J. C., Hassler, E., Hernandes, E. et al. (2013). Identifying barriers to the systematic literature review process. Paper presented at the 2013 ACM / IEEE international symposium on empirical software engineering and measurement. https://doi.org/10.1109/esem.2013.28
Centre for Reviews and Dissemination (2008). Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. Centre for Reviews and Dissemination, University of York. https://www.york.ac.uk/crd/guidance/
Counsell, C. (1997). Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 127, 380-387. doi:10.7326/0003-4819-127-5-199709010-00008
Crispo, J., Farhat, N., Fortin, Y. et al. (submitted). Non-ergot dopamine agonists and the risk of heart failure and other adverse cardiovascular reactions in Parkinson’s disease.
Currie, G. L., Angel-Scott, H. N., Colvin, L. et al. (2019). Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis. PLoS Biol 17, e3000243. doi:10.1371/journal.pbio.3000243
de Vries, R. B., Hooijmans, C. R., Tillema, A. et al. (2011). A search filter for increasing the retrieval of animal studies in Embase. Lab Anim 45, 268-270. doi:10.1258/la.2011.011056
de Vries, R. B., Hooijmans, C. R., Tillema, A. et al. (2014a). Updated version of the Embase search filter for animal studies. Lab Anim 48, 88. doi:10.1177/0023677213494374
de Vries, R. B. M., Wever, K. E., Avey, M. T. et al. (2014b). The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR J 55, 427-437. doi:10.1093/ilar/ilu043
de Vries, R. B. M., Hooijmans, C. R., Langendam, M. W. et al. (2015). A protocol format for the preparation, registration and publication of systematic reviews of animal intervention studies. Evid Based Preclin Med 2, e00007. doi:10.1002/ebm2.7
Downs, S. H. and Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 52, 377-384. doi:10.1136/jech.52.6.377
EFSA (2010). Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J 8, 1637. doi:10.2903/j.efsa.2010.1637
EFSA (2018). EFSA scientific colloquium 23 – Joint European food safety authority and evidence-based toxicology collaboration colloquium evidence integration in risk assessment: The science of combining apples and oranges 25-26 October 2017 Lisbon, Portugal. EFSA Supporting Publications 15, 1396E. doi:10.2903/sp.efsa.2018.EN-1396
Egger, M. and Smith, G. D. (2001). Systematic Reviews in Health Care: Meta-Analysis in Context. 2nd edition. London, UK: BMJ Books.
Fontelo, P. and Liu, F. (2018). A review of recent publication trends from top publishing countries. Syst Rev 7, 147. doi:10.1186/s13643-018-0819-1
Fox, D. (2010). The Convergence of Science and Governance: Research, Health Policy, and American States. Berkeley, CA, USA: University of California Press.
Gartlehner, G., Wagner, G., Lux, L. et al. (2019). Assessing the accuracy of machine-assisted abstract screening with distillerai: A user study. Syst Rev 8, 277. doi:10.1186/s13643-019-1221-3
Gates, A., Guitard, S., Pillay, J. et al. (2019). Performance and usability of machine learning for screening in systematic reviews: A comparative evaluation of three tools. Syst Rev 8, 278-278. doi:10.1186/s13643-019-1222-2
Goodman, J. E., Petito Boyce, C., Sax, S. N. et al. (2015). Rethinking meta-analysis: Applications for air pollution data and beyond. Risk Anal 35, 1017-1039. doi:10.1111/risa.12405
Griesinger, C., Hoffmann, S., Kinsner, A. et al. (2009). Preface – Proceedings of the 1st international forum towards evidence-based toxicology. Hum Exp Toxicol 28, 83-86. doi:10.1177/0960327109105753
Guyatt, G., Oxman, A. D., Akl, E. A. et al. (2011). Grade guidelines: 1. Introduction-grade evidence profiles and summary of findings tables. J Clin Epidemiol 64, 383-394. doi:10.1016/j.jclinepi.2010.04.026
Guyatt, G. H., Oxman, A. D., Santesso, N. et al. (2013). Grade guidelines: 12. Preparing summary of findings tables – Binary outcomes. J Clin Epidemiol 66, 158-172. doi:10.1016/j.jclinepi.2012.01.012
Hersi, M., Quach, P., Wang, M. D. et al. (2017). Systematic reviews of factors associated with the onset and progression of neurological conditions in humans: A methodological overview. Neurotoxicology 61, 12-18. doi:10.1016/j.neuro.2016.06.017
Higgins, J. and Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions.Version 5.1.0. The Cochrane Collaboration and John Wiley & Sons Ltd.
Higgins, J. P., Altman, D. G., Gotzsche, P. C. et al. (2011). The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343, d5928. doi:10.1136/bmj.d5928
Hoffmann, S. and Hartung, T. (2005). Diagnosis: Toxic! – Trying to apply approaches of clinical diagnostics and prevalence in toxicology considerations. Toxicol Sci 85, 422-428. doi:10.1093/toxsci/kfi099
Hoffmann, S., Griesinger, C., Coecke, S. et al. (2007). First international forum towards evidence based toxicology. ALTEX 24, 354-355. https://www.altex.org/index.php/altex/article/view/740/756
Hoffmann, S., Stephens, M. and Hartung, T. (2014). Evidence-based toxicology. In P. Wexler (ed.), Encyclopedia of Toxicology (565-567). Elsevier Inc. doi:10.1016/b978-0-12-386454-3.01060-5
Hoffmann, S., de Vries, R. B. M., Stephens, M. L. et al. (2017). A primer on systematic reviews in toxicology. Arch Toxicol 91, 2551-2575. doi:10.1007/s00204-017-1980-3
Hooijmans, C. R., Tillema, A., Leenaars, M. et al. (2010). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Lab Anim 44, 170-175. doi:10.1258/la.2010.009117
Hooijmans, C. R., IntHout, J., Ritskes-Hoitinga, M. et al. (2014a). Meta-analyses of animal studies: An introduction of a valuable instrument to further improve healthcare. ILAR J 55, 418-426. doi:10.1093/ilar/ilu042
Hooijmans, C. R., Rovers, M. M., de Vries, R. B. et al. (2014b). SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol 14, 43. doi:10.1186/1471-2288-14-43
Hooijmans, C. R., de Vries, R. B. M., Ritskes-Hoitinga, M. et al. (2018). Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies. PLoS One 13, e0187271. doi:10.1371/journal.pone.0187271
IARC (2019). IARC Monographs on the Identification of Carcinogenic Hazards to Humans – Preamble.
Institute of Medicine (2011). Clinical Practice Guidelines We Can Trust. Washington, DC, USA: National Academies Press. doi:10.17226/13058
Jorgensen, L., Paludan-Muller, A. S., Laursen, D. R. et al. (2016). Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: Overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev 5, 80. doi:10.1186/s13643-016-0259-8
Judson, R., Kavlock, R., Martin, M. et al. (2013). Perspectives on validation of high-throughput assays supporting 21st century toxicity testing. ALTEX 30, 51-56. doi:10.14573/altex.2013.1.051
Kemppinen, A. K., Kaprio, J., Palotie, A. et al. (2011). Systematic review of genome-wide expression studies in multiple sclerosis. BMJ Open 1, e000053. doi:10.1136/bmjopen-2011-000053
Khan, K., Kunz, R., Kelijnen, J. et al. (2011). Systematic Reviews to Support Evidence Based Medicine: How to Review and Apply Findings of Healthcare Research. 2nd edition. London, UK: Hodder & Stoughton Ltd.
Krauth, D., Anglemyer, A., Philipps, R. et al. (2014). Nonindustry-sponsored preclinical studies on statins yield greater efficacy estimates than industry-sponsored studies: A meta-analysis. PLoS Biol 12, e1001770. doi:10.1371/journal.pbio.1001770
Krewski, D., Saunders-Hastings, P., Baan, R. et al. (2022). Workshop report: Development of an evidence-based risk assessment framework. ALTEX, in press.
Lam, J., Koustas, E., Sutton, P. et al. (2014). The navigation guide – Evidence-based medicine meets environmental health: Integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect 122, 1040-1051. doi:10.1289/ehp.1307923
Leenaars, M., Hooijmans, C. R., van Veggel, N. et al. (2012). A step-by-step guide to systematically identify all relevant animal studies. Lab Anim 46, 24-31. doi:10.1258/la.2011.011087
McCann, S. K., Cramond, F., Macleod, M. R. et al. (2016). Systematic review and meta-analysis of the efficacy of interleukin-1 receptor antagonist in animal models of stroke: An update. Transl Stroke Res 7, 395-406. doi:10.1007/s12975-016-0489-z
Moher, D., Liberati, A., Tetzlaff, J. et al. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 6, e1000097. doi:10.1371/journal.pmed.1000097
Moher, D., Shamseer, L., Clarke, M. et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4, 1. doi:10.1186/2046-4053-4-1
Molander, L., Ågerstrand, M., Beronius, A. et al. (2015). Science in risk assessment and policy (SciRAP): An online resource for evaluating and reporting in vivo (eco)toxicity studies. Hum Ecol Risk Assess 21, 753-762. doi:10.1080/10807039.2014.928104
Morgan, R. L., Baack, B., Smith, B. D. et al. (2013). Eradication of hepatitis C virus infection and the development of hepatocellular carcinoma: A meta-analysis of observational studies. Ann Intern Med 158, 329-337. doi:10.7326/0003-4819-158-5-201303050-00005
Morgan, R. L., Thayer, K. A., Bero, L. et al. (2016). Grade: Assessing the quality of evidence in environmental and occupational health. Environ Int 92-93, 611-616. doi:10.1016/j.envint.2016.01.004
Morgan, R. L., Thayer, K. A., Santesso, N. et al. (2018a). Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the ‘target experiment’ concept in studies of exposures: Rationale and preliminary instrument development. Environ Int 120, 382-387. doi:10.1016/j.envint.2018.08.018
Morgan, R. L., Whaley, P., Thayer, K. A. et al. (2018b). Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes. Environ Int 121, 1027-1031. doi:10.1016/j.envint.2018.07.015
Morgan, R. L., Thayer, K. A., Santesso, N. et al. (2019). A risk of bias instrument for non-randomized studies of exposures: A users’ guide to its application in the context of GRADE. Environ Int 122, 168-184. doi:10.1016/j.envint.2018.11.004
Morton, S., Murad, M., O’Connor, E. et al. (2018). Quantitative Synthesis – An Update. Methods Guide for Comparative Effectiveness Reviews. Rockville, MD, USA: Agency for Healthcare Research and Quality. doi:10.23970/ahrqepcmethguide3
Mücke, M., Phillips, T., Radbruch, L. et al. (2018). Cannabis‐based medicines for chronic neuropathic pain in adults. Cochrane Database Syst Rev 3, CD012182. doi:10.1002/14651858.CD012182.pub2
NTP – National Toxicology Program (2015). OHAT Risk of Bias Rating Tool for Human and Animal Studies. Office of Health Assessment and Translation. https://ntp.niehs.nih.gov/ntp/ohat/pubs/riskofbiastool_508.pdf
NTP (2016). Systematic Literature Review on the Effects of Fluoride on Learning and Memory in Animal Studies. NTP Research Report 1. https://ntp.niehs.nih.gov/ntp/results/pubs/rr/reports/rr01_508.pdf
NTP (2019). Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. Office of Health Assessment and Translation. https://ntp.niehs.nih.gov/ntp/ohat/pubs/handbookmarch2019_508.pdf
Ouzzani, M., Hammady, H., Fedorowicz, Z. et al. (2016). Rayyan – A web and mobile app for systematic reviews. Syst Rev 5, 210. doi:10.1186/s13643-016-0384-4
Ritskes-Hoitinga, M., Leenaars, M., Avey, M. et al. (2014). Systematic reviews of preclinical animal studies can make significant contributions to health care and more transparent translational medicine. Cochrane Database Syst Rev 3, ED000078. doi:10.1002/14651858.ed000078
Ritskes-Hoitinga, M. and Wever, K. (2018). Improving the conduct, reporting, and appraisal of animal research. BMJ 360, j4935. doi:10.1136/bmj.j4935
Ritskes-Hoitinga, M. and van Luijk, J. (2019). How can systematic reviews teach us more about the implementation of the 3Rs and animal welfare? Animals (Basel) 9, doi:10.3390/ani9121163
Samuel, G. O., Hoffmann, S., Wright, R. A. et al. (2016). Guidance on assessing the methodological and reporting quality of toxicologically relevant studies: A scoping review. Environ Int 92-93, 630-646. doi:10.1016/j.envint.2016.03.010
Schünemann, H., Brożek, J., Guyatt, G. et al. (2013a). The Grade Handbook. https://gdt.gradepro.org/app/handbook/handbook.html
Schünemann, H. J., Tugwell, P., Reeves, B. C. et al. (2013b). Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Res Synth Methods 4, 49-62. doi:10.1002/jrsm.1078
Schünemann, H. J., Cuello, C., Akl, E. A. et al. (2019). Grade guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Cin Epidemiol 111, 105-114. doi:10.1016/j.jclinepi.2018.01.012
Sena, E. S., Briscoe, C. L., Howells, D. W. et al. (2010). Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: Systematic review and meta-analysis. J Cereb Blood Flow Metab 30, 1905-1913. doi:10.1038/jcbfm.2010.116
Shea, B. J., Reeves, B. C., Wells, G. et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358, j4008. doi:10.1136/bmj.j4008
Silbergeld, E. and Scherer, R. W. (2013). Evidence-based toxicology: Strait is the gate, but the road is worth taking. ALTEX 30, 67-73. doi:10.14573/altex.2013.1.067
Stephens, M. L., Andersen, M., Becker, R. A. et al. (2013). Evidence-based toxicology for the 21st century: Opportunities and challenges. ALTEX 30, 74-103. doi:10.14573/altex.2013.1.074
Stephens, M. L., Betts, K., Beck, N. B. et al. (2016). The emergence of systematic review in toxicology. Toxicol Sci 152, 10-16. doi:10.1093/toxsci/kfw059
Stephens, M. L., Akgun-Olmez, S. G., Hoffmann, S. et al. (2018). Adaptation of the systematic review framework to the assessment of toxicological test methods: Challenges and lessons learned with the zebrafish embryotoxicity test. Toxicol Sci 171, 56-68. doi:10.1093/toxsci/kfz128
Sterne, J., Hernán, M., Reeves, B. et al. (2016a). ROBINS-I: A tool for assessing risk of bias in non-randomized studies of interventions. BMJ 355, i4919. doi:10.1136/bmj.i4919
Sterne, J., Higgins, J., Elbers, R. et al. (2016b). Risk of Bias in Non-Randomized Studies of Interventions (ROBINS-I): Detailed Guidance, Updated 12 October, 2016. http://www.riskofbias.info
US EPA (2018a). Strategic Plan to Promote the Development and Implementation of Alternative Test Methods Within the TSCA Program. US EPA. EPA-740-R1-8004. https://www.epa.gov/sites/production/files/2018-06/documents/epa_alt_strat_plan_6-20-18_clean_final.pdf
US EPA (2018b). Application of Systematic Review in TSCA Risk Evaluations. Document# 740-P1-8001. https://www.epa.gov/sites/production/files/2018-06/documents/final_application_of_sr_in_tsca_05-31-18.pdf
Van der Mierden, S., Tsaioun, K., Bleich, A. et al. (2019). Software tools for literature screening in systematic reviews in biomedical research. ALTEX 36, 508-517. doi:10.14573/altex.1902131
Vesterinen, H. M., Sena, E. S., Egan, K. J. et al. (2014). Meta-analysis of data from animal studies: A practical guide. J Neurosci Methods 221, 92-102. doi:10.1016/j.jneumeth.2013.09.010
Viswanathan, M., Patnode, C., Berkman, N. et al. (2017). Assessing the risk of bias in systematic reviews of health care interventions. Methods guide for comparative effectiveness reviews. (prepared by the scientific resource center under contract no. 290-2012-0004-c). AHRQ Publication No. 17(18)-ehc036-ef. doi:10.23970/ahrqepcmethguide2
Whaley, P., Aiassa, E., Beausoleil, C. et al. (in preparation). A code of practice for the conduct of systematic reviews in toxicology and environmental health research (COSTER).
Whiting, P., Savović, J., Higgins, J. P. et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69, 225-234. doi:10.1016/j.jclinepi.2015.06.005
Wolffe, T., Vidler, J., Halsall, C. et al. (2020). A survey of systematic evidence mapping practice and the case for knowledge graphs in environmental health and toxicology. Toxicol Sci 175, 35-49. doi:10.1093/toxsci/kfaa025
Woodruff, T. J. and Sutton, P. (2014). The navigation guide systematic review methodology: A rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect 122, 1007-1014. doi:10.1289/ehp.1307175
Wright, R. W., Brand, R. A., Dunn, W. et al. (2007). How to write a systematic review. Clin Orthop Relat Res 455, 23-29. doi:10.1097/BLO.0b013e31802c9098
Yauw, S. T., Wever, K. E., Hoesseini, A. et al. (2015). Systematic review of experimental studies on intestinal anastomosis. Br J Surg 102, 726-734. doi:10.1002/bjs.9776