Statistical derivation of cut-off values for in vitro assays
Article Sidebar
Main Article Content
Abstract
Chemical substances and mixtures are classified based on their toxicological hazard. Today, in vitro methods are more frequently applied for this purpose. The regulatory validation process assesses their relevance by comparing them to standard in vivo test data, which includes transforming continuous read-out data into ordinal data (hazard classes). Existing strategies for developing new methods overlook the constraints associated with small data sets omitting the use of contemporary statistical techniques, such as uncertainty quantification and bootstrapping. To overcome these limitations, we apply bootstrapping, estimates for the out-of-sample error, and uncertainty quantification to the validation dataset of Kaluzhny et al. (2011) and a dataset of plant protection products (PPPs) previously published by Kolle et al. (2015), which have been tested for eye irritation in vitro (OECD TG492) and in vivo (OECD TG 405). Assessment criteria for sensitivity, specificity, and accuracy are proposed, considering uncertainty quantification and estimation of the out-of-sample error. The cut-off value for plant protection products based on the available set of in vitro-in vivo data pairs can be improved by the application of modern cut-off approaches. For PPPs, the OECD recommended cut-off of 60% mean tissue viability based on single substances leads to lower sensitivity than the newly derived cut-off value of 67%. For liquid single substances, the OECD recommended cut-off is confirmed. This case study demonstrates that modern statistical methods for small datasets improve the reliability of in vitro cut-off values and should therefore be used to revise and derive cut-off values for hazard classification in future.
Plain language summary
In vitro experiments using tissue cultures can replace tests on animals (in vivo experiments). The mean tissue viability of the in vitro experiment is used to predict the toxicological hazard of a chemical. In order to classify chemicals, a cut-off for the mean tissue viability is needed. The cut-off has to be validated by comparison to existing in vivo data. Current standard validation approaches do not consider the limitations of small data sets and lack the application of modern statistical methods, including uncertainty quantification of the application of the cut-off to new data. For the classification of chemicals which might cause eye irritation or eye damage two datasets are analysed one with single substances and one with plant protection products. Application of modern statistical methods yields a more protective cut-off for mixtures. It is shown that different cut-offs for single substances and mixtures are optimal.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles are distributed under the terms of the Creative Commons Attribution 4.0 International license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is appropriately cited (CC-BY). Copyright on any article in ALTEX is retained by the author(s).
Adler-Flindt, S. and Martin, S. (2019). Comparative cytotoxicity of plant protection products and their active ingredients. Toxicology in Vitro 54, 354-366, doi:10.1016/j.tiv.2018.10.020
Carpenter, J. and Bithell J. (2000). Bootstrap confidence intervals: When, which, what? A practical guide for medical statisticians. Stat Med 19, 1141-1164. doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F
Choksi N., Latorre, A., Catalano, S. et al. (2024). Retrospective evaluation of the eye irritation potential of agrochemical formulations, Regul Toxicol Pharmacol 146, 105543. doi:10.1016/j.yrtph.2023.105543
Clippinger, A. J., Raabe, H. A., Allen, D. G. et al. (2021). Human-relevant approaches to assess eye corrosion/irritation potential of agrochemical formulations. Cutan Ocul Toxicol 40, 145-167. doi:10.1080/15569527.2021.1910291
Corvaro, M., Gehen S., Andrews, K. et al. (2017). A retrospective analysis of in vivo eye irritation, skin irritation and skin sensitisation studies with agrochemical formulations: Setting the scene for development of alternative strategies. Regul Toxicol Pharmacol 89, 131-147. doi:10.1016/j.yrtph.2017.06.014
Curren R. D. and Harbell J. W. (2002). Ocular safety: A silent (in vitro) success story. Altern Lab Anim 30, 69-74. doi:10.1177/026119290203002S10
Davila, J. C., Rodriguez, R. J., Melchert, R. B. et al. (1998). Predictive value of in vitro model systems in toxicology. Annu Rev Pharmacol Toxicol 38, 63-96. doi:10.1146/annurev.pharmtox.38.1.63
Efron, B. and Tibshirani R. (1997). Improvements on cross-validation: The .632+ bootstrap method. Journal of the American Statistical Association 92, 548–60. http://www.jstor.org/stable/2965703
EFSA SC – EFSA Scientific Committee, Hardy A., Benford D., et al. (2017). Update: Guidance on the use of the benchmark dose approach in risk assessment. EFSA J 15, 4658. doi:10.2903/j.efsa.2017.4658
EFSA SC, More, S. J., Bampidis, V. et al. (2022). Guidance on the use of the benchmark dose approach in risk assessment. EFSA J 20, e07584. doi:10.2903/j.efsa.2022.7584
EC – European Commission, Joint Research Centre, McNamee, P., Cole, T. et al. (2014). The EURL ECVAM: cosmetics Europe prospective validation study of reconstructed human cornea-like epithelium (RhCE)-based test methods for identifying chemicals not requiring classification and labelling for serious eye damage/eye irritation: validation study report. Publications Office. doi:10.2787/41680
Ewald, B. (2006). Post hoc choice of cut points introduced bias to diagnostic research. J Clin Epidemiol 59, 798-801. doi:10.1016/j.jclinepi.2005.11.025
Feiertag, K., Karaca, M., Fischer, B. et al. (2023). Mixture effects of co-formulants and two plant protection products in a liver cell line. EXCLI J 22, 221-236. doi:10.17179/excli2022-5648
Fluss, R., Faraggi D., and Reiser B. (2005). Estimation of the Youden index and its associated cutoff point. Biometrical Journal 47, 458-72. doi:10.1002/bimj.200410135
Gabbert S., Mathea M., Kolle S. N. et al. (2022). Accounting for precision uncertainty of toxicity testing: Methods to define borderline ranges and implications for hazard assessment of chemicals. Risk Anal 42, 224-238. doi:10.1111/risa.13648
Jester, J. V., Li, L., Molai A. et al. (2001). Extent of initial corneal injury as a basis for alternative eye irritation tests. Toxicol in Vitro 15, 115-130. doi:10.1016/S0887-2333(00)00065-5
Kaluzhny, Y., Kandárová H., Hayden P. et al. (2011). Development of the EpiOcular(TM) Eye Irritation Test for Hazard Identification and Labelling of Eye Irritating Chemicals in Response to the Requirements of the EU Cosmetics Directive and REACH Legislation. Altern Lab Anim 39, 339-64. doi:10.1177/026119291103900409
Karaca, M., Fischer, B. C., Willenbockel, C. T. et al. (2021). Effects of co-formulants on the absorption and secretion of active substances in plant protection products in vitro. Arch Toxicol 95, 3205-3221. doi:10.1007/s00204-021-03140-x
Kolle, S. N., Rey Moreno M. C., Mayer W. et al. (2015). The EpiOcular™ Eye Irritation Test Is the Method of Choice for the in Vitro Eye Irritation Testing of Agrochemical Formulations: Correlation Analysis of EpiOcular Eye Irritation Test and BCOP Test Data According to the UN GHS, US EPA and Brazil ANVISA Classification Schemes. Alternat Lab Anim 43, 181-198. doi:10.1177/026119291504300307
Kolle, S. N., Van Cott, A., van Ravenzwaay, B. et al. (2017). Lacking applicability of in vitro eye irritation methods to identify seriously eye irritating agrochemical formulations: Results of bovine cornea opacity and permeability assay, isolated chicken eye test and the EpiOcular™ ET-50 method to classify according to UN GHS. Regul Toxicol Pharmacol 85, 33-47. doi:10.1016/j.yrtph.2017.01.013
Kolle, S., Mathea, M., Natsch, A. et al. (2021). Assessing Experimental Uncertainty in Defined Approaches: Borderline Ranges for In Chemico and In Vitro Skin Sensitization Methods Determined from Ring Trial Data. Applied In Vitro Toxicology 7, 102-111. doi:10.1089/aivt.2021.0003
Kuhn, M., and Johnson K. (2013). Applied Predictive Modeling. Springer New York, NY. doi:10.1007/978-1-4614-6849-3
Leeflang, M. M. G., Moons K. G. M., Reitsma J. B. et al., (2008). Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutoff Values: Mechanisms, Magnitude, and Solutions. Clinical Chemistry 54, 729-737. doi:10.1373/clinchem.2007.096032
McKim J. M., Jr (2010). Building a tiered approach to in vitro predictive toxicity screening: a focus on assays with in vivo relevance. Combinatorial chemistry & high throughput screening 13, 188-206. doi:10.2174/138620710790596736
OECD (2015), Test No. 404: Acute Dermal Irritation/Corrosion, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/9789264242678-en
OECD (2018), Guidance Document on Good In Vitro Method Practices (GIVIMP), OECD Series on Testing and Assessment, No. 286, OECD Publishing, Paris, doi:10.1787/9789264304796-en
OECD (2019), Second Edition - Guidance Document on Integrated Approaches to Testing and Assessment (IATA) for Serious Eye Damage and Eye Irritation, OECD Series on Testing and Assessment, No. 263, OECD Publishing, Paris, doi:10.1787/84b83321-en
OECD (2021), Test No. 439: In Vitro Skin Irritation: Reconstructed Human Epidermis Test Method, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/9789264242845-en
OECD (2023a), Test No. 437: Bovine Corneal Opacity and Permeability Test Method for Identifying i) Chemicals Inducing Serious Eye Damage and ii) Chemicals Not Requiring Classification for Eye Irritation or Serious Eye Damage, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/9789264203846-en
OECD (2023b), Test No. 405: Acute Eye Irritation/Corrosion, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/9789264185333-en
OECD (2024a), Test No. 492: Reconstructed human Cornea-like Epithelium (RhCE) test method for identifying chemicals not requiring classification and labelling for eye irritation or serious eye damage, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/9789264242548-en
OECD (2024b), Test No. 467: Defined Approaches for Serious Eye Damage and Eye Irritation, OECD Guidelines for the Testing of Chemicals, Section 4, OECD Publishing, Paris, doi:10.1787/28fe2841-en
Prinsen M. K. (2006). The Draize Eye Test and in vitro alternatives; a left-handed marriage? Toxicol in Vitro 20, 78-81. doi:10.1016/j.tiv.2005.06.030
Schrage, A., Gamer A. O., van Ravenzwaay B. et al. (2010). Experience with the HET-CAM Method in the Routine Testing of a Broad Variety of Chemicals and Formulations. Altern Lab Anim 38, 39-52. doi:10.1177/026119291003800109
Scott, L., Eskes, C., Hoffmann, S. et al. (2010). A proposed eye irritation testing strategy to reduce and replace in vivo studies using Bottom-Up and Top-Down approaches. Toxicol in Vitro 24, 1-9. doi:10.1016/j.tiv.2009.05.019
Settivari, R. S., Amado R. A., Corvaro, M. et al. (2016). Tiered application of the neutral red release and EpiOcular™ assays for evaluating the eye irritation potential of agrochemical formulations. Regul Toxicol Pharmacol 81, 407-420. doi:10.1016/j.yrtph.2016.09.028
Strickland J., Truax J., Corvaro M. et al. (2022). Application of defined approaches for skin sensitization to agrochemical products. Front Toxicol 4, 2673-3080. doi:10.3389/ftox.2022.852856
Thiele, C. and Hirschfeld G. (2021). Cutpointr: Improved estimation and validation of optimal cutpoints in R. Journal of Statistical Software 98, 1-27. doi:10.18637/jss.v098.i11
van der Zalm, A. J., Daniel, A. B., Raabe, H.A. et al. (2023). Defined approaches to classify agrochemical formulations into EPA hazard categories developed using EpiOcularTM reconstructed human corneal epithelium and bovine corneal opacity and permeability assays. Cutan Ocul Toxicol 43, 58-68. doi:10.1080/15569527.2023.2275029
Wickham, H. (2016). Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer 3, 32-35. doi:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3