Systematic review in evidence-based risk assessment

Main Article Content

Nawal Farhat
Katya Tsaioun
Patrick Saunders-Hastings
Rebecca L Morgan
Siva Ramoju
Thomas Hartung
Daniel Krewski


Systematic reviews provide a structured framework for summarizing the available evidence in a comprehensive, objective, and transparent manner. They inform evidence-based guidelines in medicine, public policy, and more recently, in environmental health and toxicology. Many regulatory agencies have extended and adapted the well-established systematic review methods, initially developed for clinical studies, for their assessment needs. The use of systematic reviews to summarize evidence from existing human, animal, and mechanistic studies can reduce reliance on animal test data in risk assessment and can help avoid unnecessary duplication of animal experiments that have already been conducted. As alternative test methods can be expected to play an increasing role in human health risk assessment in the future, systematic reviews can be particularly helpful in validating these alternatives. The field of evidence-based toxicology has undergone extensive development since its first meeting in 2007 as a result of collaborative efforts among international experts and public health agencies, particularly with respect to the use of mechanistic data and evidence integration. The continued development and wider adoption of systematic review methodology can lead to better 3R implementation. As undertaking a systematic review can be a complex and lengthy process, it is important to understand the main steps involved. Key steps, along with current best practices, are described with references to guidance from organizations with expertise in evidence synthesis. Applications of systematic reviews in clinical, observational, and experimental studies are presented. Finally, software tools available to facilitate and increase the efficiency of completing a systematic review are described.

Article Details

How to Cite
Farhat, N., Tsaioun, K., Saunders-Hastings, P., Morgan, R. L., Ramoju, S., Hartung, T. and Krewski, D. (2021) “Systematic review in evidence-based risk assessment”, ALTEX - Alternatives to animal experimentation. doi: 10.14573/altex.2004111.

AHRQ (2014). Methods guide for effectiveness and comparative effectiveness reviews. Publication no. 10(14)-ehc063-ef Vol. Publication No. 10(14)-EHC063-EF Rockville, MD: Agency for Healthcare Research and Quality.

Andersen, M. E., McMullen, P. D., Phillips, M. B. et al. (2019). Developing context appropriate toxicity testing approaches using new alternative methods (nams). ALTEX 36, 523-534. doi:10.14573/altex.1906261

Arbyn, M., Xu, L., Simoens, C. et al. (2018). Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database of Systematic Reviews doi:10.1002/14651858.CD009069.pub3

Aromataris, E. and Munn, Z. (2017). Joanna briggs institute reviewer's manual. Vol. The Joanna Briggs Institute.

Banwell, V., Sena, E. S. and Macleod, M. R. (2009). Systematic review and stratified meta-analysis of the efficacy of interleukin-1 receptor antagonist in animal models of stroke. J Stroke Cerebrovasc Dis 18, 269-276. doi:10.1016/j.jstrokecerebrovasdis.2008.11.009

Beronius, A., Molander, L., Zilliacus, J. et al. (2018). Testing and refining the science in risk assessment and policy (scirap) web-based platform for evaluating the reliability and relevance of in vivo toxicity studies. J Appl Toxicol 38, 1460-1470. doi:10.1002/jat.3648

Bronsveld, H. K., ter Braak, B., Karlstad, Ø. et al. (2015). Treatment with insulin (analogues) and breast cancer risk in diabetics; a systematic review and meta-analysis of in vitro, animal and human evidence. Breast Cancer Res 17, 100. doi:10.1186/s13058-015-0611-2

Buscemi, N., Hartling, L., Vandermeer, B. et al. (2006). Single data extraction generated more errors than double data extraction in systematic reviews. . J Clin Epidemiol 59, 697-703 doi:10.1016/j.jclinepi.2005.11.010

Carver, J. C., Hassler, E., Hernandes, E., et al. (2013). Identifying barriers to the systematic literature review process. Paper presented at the 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

Centre for Reviews and Dissemination (2008). Systematic reviews: Crd’s guidance for undertaking reviews in health care. Vol. Centre for Reviews and Dissemination, University of York.

Counsell, C. (1997). Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 127, 380-387. doi:10.7326/0003-4819-127-5-199709010-00008

Crispo, J., Farhat, N., Fortin, Y. et al. (submitted). Non-ergot dopamine agonists and the risk of heart failure and other adverse cardiovascular reactions in parkinson’s disease.

Currie, G. L., Angel-Scott, H. N., Colvin, L. et al. (2019). Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis. PLoS Biol 17, e3000243. doi:10.1371/journal.pbio.3000243

de Vries, R. B., Hooijmans, C. R., Tillema, A. et al. (2011). A search filter for increasing the retrieval of animal studies in embase. Lab Anim 45, 268-270. doi:10.1258/la.2011.011056

de Vries, R. B., Hooijmans, C. R., Tillema, A. et al. (2014a). Updated version of the embase search filter for animal studies. Lab Anim 48, 88. doi:10.1177/0023677213494374

de Vries, R. B. M., Wever, K. E., Avey, M. T. et al. (2014b). The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR Journal 55, 427-437. doi:10.1093/ilar/ilu043

de Vries, R. B. M., Hooijmans, C. R., Langendam, M. W. et al. (2015). A protocol format for the preparation, registration and publication of systematic reviews of animal intervention studies. Evidence-based Preclinical Medicine 2, doi:10.1002/ebm2.7

Downs, S. H. and Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 52, 377-384. doi:10.1136/jech.52.6.377

EFSA (2010). Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 8, 1637. doi:10.2903/j.efsa.2010.1637

EFSA (2018). Efsa scientific colloquium 23 – joint european food safety authority and evidence-based toxicology collaboration colloquium evidence integration in risk assessment: The science of combining apples and oranges 25–26 october 2017 lisbon, portugal. EFSA Supporting Publications 15, 1396E. doi:10.2903/sp.efsa.2018.EN-1396

Egger, M., and Smith, G. D. (2001). Systematic Reviews in Health Care: Meta-Analysis in Context. 2nd ed. London: BMJ Books

Fontelo, P. and Liu, F. (2018). A review of recent publication trends from top publishing countries. Systematic Reviews 7, 147. doi:10.1186/s13643-018-0819-1

Fox, D. (2010). The convergence of science and governance: Research, health policy, and american states. . Vol. Berkeley, CA: University of California Press.

Gartlehner, G., Wagner, G., Lux, L. et al. (2019). Assessing the accuracy of machine-assisted abstract screening with distillerai: A user study. Systematic Reviews 8, 277. doi:10.1186/s13643-019-1221-3

Gates, A., Guitard, S., Pillay, J. et al. (2019). Performance and usability of machine learning for screening in systematic reviews: A comparative evaluation of three tools. Systematic reviews 8, 278-278. doi:10.1186/s13643-019-1222-2

Goodman, J. E., Petito Boyce, C., Sax, S. N. et al. (2015). Rethinking meta-analysis: Applications for air pollution data and beyond. Risk Anal 35, 1017-1039. doi:10.1111/risa.12405

Griesinger, C., Hoffmann, S., Kinsner, A. et al. (2009). Preface - proceedings of the 1st international forum towards evidence-based toxicology. Human and Experimental Toxicology 28, 83-86. doi:doi:10.1177/0960327109105753

Guyatt, G., Oxman, A. D., Akl, E. A. et al. (2011). Grade guidelines: 1. Introduction-grade evidence profiles and summary of findings tables. J Clin Epidemiol 64, 383-394. doi:10.1016/j.jclinepi.2010.04.026

Guyatt, G. H., Oxman, A. D., Santesso, N. et al. (2013). Grade guidelines: 12. Preparing summary of findings tables-binary outcomes. J Clin Epidemiol 66, 158-172. doi:10.1016/j.jclinepi.2012.01.012

Hersi, M., Quach, P., Wang, M. D. et al. (2017). Systematic reviews of factors associated with the onset and progression of neurological conditions in humans: A methodological overview. Neurotoxicology 61, 12-18. doi:10.1016/j.neuro.2016.06.017

Higgins, J. and Green, S. (2011). Cochrane handbook for systematic reviews of interventions version 5.1.0. Vol. The Cochrane Collaboration and John Wiley & Sons Ltd.

Higgins, J. P., Altman, D. G., Gotzsche, P. C. et al. (2011). The cochrane collaboration's tool for assessing risk of bias in randomised trials. Bmj 343, d5928. doi:10.1136/bmj.d5928

Hoffmann, S. and Hartung, T. (2005). Diagnosis: Toxic!--trying to apply approaches of clinical diagnostics and prevalence in toxicology considerations. Toxicol Sci 85, 422-428. doi:10.1093/toxsci/kfi099

Hoffmann, S., Griesinger, C., Coecke, S. et al. (2007). First international forum towards evidence based toxicology. ALTEX 24, 354-355.

Hoffmann, S., Stephens, M. and Hartung, T. (2014). Evidence-based toxicology. In (eds.), Encyclopedia of toxicology. Elsevier Inc. doi:10.1016/b978-0-12-386454-3.01060-5

Hoffmann, S., de Vries, R. B. M., Stephens, M. L. et al. (2017). A primer on systematic reviews in toxicology. Arch Toxicol 91, 2551-2575. doi:10.1007/s00204-017-1980-3

Hooijmans, C. R., Tillema, A., Leenaars, M. et al. (2010). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in pubmed. Lab Anim 44, 170-175. doi:10.1258/la.2010.009117

Hooijmans, C. R., IntHout, J., Ritskes-Hoitinga, M. et al. (2014a). Meta-analyses of animal studies: An introduction of a valuable instrument to further improve healthcare. Ilar j 55, 418-426. doi:10.1093/ilar/ilu042

Hooijmans, C. R., Rovers, M. M., de Vries, R. B. et al. (2014b). Syrcle's risk of bias tool for animal studies. BMC Med Res Methodol 14, 43. doi:10.1186/1471-2288-14-43

Hooijmans, C. R., de Vries, R. B. M., Ritskes-Hoitinga, M. et al. (2018). Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies. PLOS ONE 13, e0187271. doi:10.1371/journal.pone.0187271

IARC (2019). IARC monographs on the identification of carcinogenic hazards to humans - preamble. IARC.

Institute of Medicine (2011). Clinical practice guidelines we can trust. Vol. Washington (DC): National Academies Press (US).

Jorgensen, L., Paludan-Muller, A. S., Laursen, D. R. et al. (2016). Evaluation of the cochrane tool for assessing risk of bias in randomized clinical trials: Overview of published comments and analysis of user practice in cochrane and non-cochrane reviews. Syst Rev 5, 80. doi:10.1186/s13643-016-0259-8

Judson, R., Kavlock, R., Martin, M. et al. (2013). Perspectives on validation of high-throughput assays supporting 21st century toxicity testing. Altex 30, 51-56. doi:10.14573/altex.2013.1.051

Kemppinen, A. K., Kaprio, J., Palotie, A. et al. (2011). Systematic review of genome-wide expression studies in multiple sclerosis. BMJ Open 1, e000053. doi:10.1136/bmjopen-2011-000053

Khan, K., Kunz, R., Kelijnen, J. et al. (2011). Systematic reviews to support evidence based medicine: How to review and apply findings of healthcare research. Vol. 2nd. London: Hodder & Stoughton Ltd.

Krauth, D., Anglemyer, A., Philipps, R. et al. (2014). Nonindustry-sponsored preclinical studies on statins yield greater efficacy estimates than industry-sponsored studies: A meta-analysis. PLoS Biol 12, e1001770. doi:10.1371/journal.pbio.1001770

Krewski, D., Saunders-Hastings, P., Baan, R. et al. (2020). Workshop report: Development of an evidence-based risk assessment framework. ALTEX, under review.

Lam, J., Koustas, E., Sutton, P. et al. (2014). The navigation guide - evidence-based medicine meets environmental health: Integration of animal and human evidence for pfoa effects on fetal growth. Environ Health Perspect 122, 1040-1051. doi:10.1289/ehp.1307923

Leenaars, M., Hooijmans, C. R., van Veggel, N. et al. (2012). A step-by-step guide to systematically identify all relevant animal studies. Laboratory animals 46, 24-31. doi:10.1258/la.2011.011087

McCann, S. K., Cramond, F., Macleod, M. R. et al. (2016). Systematic review and meta-analysis of the efficacy of interleukin-1 receptor antagonist in animal models of stroke: An update. Transl Stroke Res 7, 395-406. doi:10.1007/s12975-016-0489-z

Moher, D., Liberati, A., Tetzlaff, J. et al. (2009). Preferred reporting items for systematic reviews and meta-analyses: The prisma statement. PLoS Med 6, e1000097. doi:10.1371/journal.pmed.1000097

Moher, D., Shamseer, L., Clarke, M. et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement. Systematic Reviews 4, 1-9. doi:10.1186/2046-4053-4-1

Molander, L., Ågerstrand, M., Beronius, A. et al. (2015). Science in risk assessment and policy (scirap): An online resource for evaluating and reporting in vivo (eco)toxicity studies. Human and Ecological Risk Assessment: An International Journal 21, 753-762. doi:10.1080/10807039.2014.928104

Morgan, R. L., Baack, B., Smith, B. D. et al. (2013). Eradication of hepatitis c virus infection and the development of hepatocellular carcinoma: A meta-analysis of observational studies. Annals of internal medicine 158, 329-337. doi:10.7326/0003-4819-158-5-201303050-00005

Morgan, R. L., Thayer, K. A., Bero, L. et al. (2016). Grade: Assessing the quality of evidence in environmental and occupational health. Environ Int 92-93, 611-616. doi:10.1016/j.envint.2016.01.004

Morgan, R. L., Thayer, K. A., Santesso, N. et al. (2018a). Evaluation of the risk of bias in non-randomized studies of interventions (robins-i) and the ‘target experiment’concept in studies of exposures: Rationale and preliminary instrument development. Environment international 120, 382-387. doi:10.1016/j.envint.2018.08.018

Morgan, R. L., Whaley, P., Thayer, K. A. et al. (2018b). Identifying the peco: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes. Environ Int doi:10.1016/j.envint.2018.07.015

Morgan, R. L., Thayer, K. A., Santesso, N. et al. (2019). A risk of bias instrument for non-randomized studies of exposures: A users' guide to its application in the context of grade. Environment international 122, 168-184. doi:10.1016/j.envint.2018.11.004

Morton, S., Murad, M., O’Connor, E. et al. (2018). Quantitative synthesis—an update. Methods guide for comparative effectiveness reviews. . Vol. Rockville, MD: Agency for Healthcare Research and Quality. doi:10.23970/AHRQEPCMETHGUIDE3

Mücke, M., Phillips, T., Radbruch, L. et al. (2018). Cannabis‐based medicines for chronic neuropathic pain in adults. Cochrane Database of Systematic Reviews doi:10.1002/14651858.CD012182.pub2

NTP (2015). Ohat risk of bias rating tool for human and animal studies Office of Health Assessment and Translation, National Toxicology Program

NTP (2016). Systematic literature review on the effects of fluoride on learning and memory in animal studies NTP Research Report 1. National Toxicology Program.

NTP (2019). Handbook for conducting a literature-based health assessment using ohat approach for systematic review and evidence integration.

Ouzzani, M., Hammady, H., Fedorowicz, Z. et al. (2016). Rayyan—a web and mobile app for systematic reviews. Systematic Reviews 5, 210. doi:10.1186/s13643-016-0384-4

Ritskes-Hoitinga, M., Leenaars, M., Avey, M. et al. (2014). Systematic reviews of preclinical animal studies can make significant contributions to health care and more transparent translational medicine. The Cochrane database of systematic reviews 3, ED000078. doi:10.1002/14651858.ed000078

Ritskes-Hoitinga, M. and Wever, K. (2018). Improving the conduct, reporting, and appraisal of animal research. BMJ 360, j4935. doi:10.1136/bmj.j4935

Ritskes-Hoitinga, M. and van Luijk, J. (2019). How can systematic reviews teach us more about the implementation of the 3rs and animal welfare? Animals (Basel) 9, doi:10.3390/ani9121163

Samuel, G. O., Hoffmann, S., Wright, R. A. et al. (2016). Guidance on assessing the methodological and reporting quality of toxicologically relevant studies: A scoping review. Environ Int 92-93, 630-646. doi:10.1016/j.envint.2016.03.010

Saunders-Hastings, P., Rhomberg, L. and Krewski, D. (2020). Best practices in weight of evidence: A review. ALTEX, under review.

Schünemann, H., Brożek, J., Guyatt, G. et al. (2013a). The grade handbook. Vol.

Schünemann, H. J., Tugwell, P., Reeves, B. C. et al. (2013b). Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods 4, 49-62. doi:10.1002/jrsm.1078

Schünemann, H. J., Cuello, C., Akl, E. A. et al. (2019). Grade guidelines: 18. How robins-i and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. Journal of clinical epidemiology 111, 105-114. doi:10.1016/j.jclinepi.2018.01.012

Sena, E. S., Briscoe, C. L., Howells, D. W. et al. (2010). Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: Systematic review and meta-analysis. J Cereb Blood Flow Metab 30, 1905-1913. doi:10.1038/jcbfm.2010.116

Shea, B. J., Reeves, B. C., Wells, G. et al. (2017). Amstar 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj 358, j4008. doi:10.1136/bmj.j4008

Silbergeld, E. and Scherer, R. W. (2013). Evidence-based toxicology: Strait is the gate, but the road is worth taking. Altex 30, 67-73. doi:10.14573/altex.2013.1.067

Stephens, M. L., Andersen, M., Becker, R. A. et al. (2013). Evidence-based toxicology for the 21st century: Opportunities and challenges. Altex 30, 74-103. doi:10.14573/altex.2013.1.074

Stephens, M. L., Betts, K., Beck, N. B. et al. (2016). The emergence of systematic review in toxicology. Toxicol Sci 152, 10-16. doi:10.1093/toxsci/kfw059

Stephens, M. L., Akgun-Olmez, S. G., Hoffmann, S. et al. (2018). Adaptation of the systematic review framework to the assessment of toxicological test methods: Challenges and lessons learned with the zebrafish embryotoxicity test. Toxicol Sci doi:10.1093/toxsci/kfz128

Sterne, J., Hernán, M., Reeves, B. et al. (2016a). Robins-i: A tool for assessing risk of bias in non-randomized studies of interventions. BMJ 355, i4919. doi:10.1136/bmj.i4919

Sterne, J., Higgins, J., Elbers, R. et al. (2016b). Risk of bias in non-randomized studies of interventions (robins-i): Detailed guidance, updated 12 october.

The Cochrane Collaboration Cochrane handbook for systematic reviews of diagnostic test accuracy Vol. The Cochrane Collaboration.

US EPA (2018a). Strategic plan to promote the development and implementation of alternative test methods within the tsca program. US EPA. EPA-740-R1-8004.

US EPA (2018b). Application of systematic review in tsca risk evaluations. Document# 740-P1-8001.

Van der Mierden, S., Tsaioun, K., Bleich, A. et al. (2019). Software tools for literature screening in systematic reviews in biomedical research. Altex 36, 508-517. doi:10.14573/altex.1902131

Vesterinen, H. M., Sena, E. S., Egan, K. J. et al. (2014). Meta-analysis of data from animal studies: A practical guide. J Neurosci Methods 221, 92-102. doi:10.1016/j.jneumeth.2013.09.010

Viswanathan, M., Patnode, C., Berkman, N. et al. (2017). Assessing the risk of bias in systematic reviews of health care interventions. Methods guide for comparative effectiveness reviews. (prepared by the scientific resource center under contract no. 290-2012-0004-c). Ahrq publication no. 17(18)-ehc036-ef. A. f. H. R. a. Quality, doi:10.23970/AHRQEPCMETHGUIDE2

Whaley, P., Aiassa, E., Beausoleil, C. et al. (draft). A code of practice for the conduct of systematic reviews in toxicology and environmental health research (coster). In preparation

Whiting, P., Savović, J., Higgins, J. P. et al. (2016). Robis: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69, 225-234. doi:10.1016/j.jclinepi.2015.06.005

Wolffe, T., Vidler, J., Halsall, C. et al. (2020). A survey of systematic evidence mapping practice and the case for knowledge graphs in environmental health & toxicology. Toxicological Sciences : An Official Journal of the Society of Toxicology doi:10.1093/toxsci/kfaa025

Woodruff, T. J. and Sutton, P. (2014). The navigation guide systematic review methodology: A rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect 122, 1007-1014. doi:10.1289/ehp.1307175

Wright, R. W., Brand, R. A., Dunn, W. et al. (2007). How to write a systematic review. Clin Orthop Relat Res 455, 23-29. doi:10.1097/BLO.0b013e31802c9098

Yauw, S. T., Wever, K. E., Hoesseini, A. et al. (2015). Systematic review of experimental studies on intestinal anastomosis. Br J Surg 102, 726-734. doi:10.1002/bjs.9776

Most read articles by the same author(s)

<< < 1 2 3 4 5 6 7 8 9 10 > >>