Artificial intelligence-based incident analysis and learning system to enhance patient safety and improve treatment quality – npj Digital Medicine

1. Patient Safety: Achieving a New Standard for Care. (National Academies Press, Washington, D.C., 2004). https://doi.org/10.17226/10863.

2. Lee, S. E. et al. Safety culture, patient safety, and quality of care outcomes: A literature review. West J. Nurs. Res. 41, 279–304 (2019).

   Google Scholar 

3. DiCuccio, M. H. The relationship between patient safety culture and patient outcomes: A systematic review. J. Patient Saf. 11, 135–142 (2015).

   Google Scholar 

4. Institute of Medicine (US) Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. (National Academies Press (US), Washington (DC), 2000).

5. Zarei, M., Gershan, V. & Holmberg, O. Safety in radiation oncology (SAFRON): Learning about incident causes and safety barriers in external beam radiotherapy. Phys. Med. 111, 102618 (2023).

   Google Scholar 

6. Nelson, C., Roy, L. A. & Wallace, H. J. Radiation oncology incident learning system (RO-ILS): Increasing stakeholder participation for safety and quality improvement. J. Clin. Oncol. 37, 232–232 (2019).

   Google Scholar 

7. Milosevic, M. et al. The Canadian National System for Incident Reporting in Radiation Treatment (NSIR-RT) taxonomy. Pr. Radiat. Oncol. 6, 334–341 (2016).

   Google Scholar 

8. Ford, E. C. & Evans, S. B. Incident learning in radiation oncology: A review. Med Phys. 45, e100–e119 (2018).

   Google Scholar 

9. Ford, E. C., Fong de Los Santos, L., Pawlicki, T., Sutlief, S. & Dunscombe, P. Consensus recommendations for incident learning database structures in radiation oncology. Med Phys. 39, 7272–7290 (2012).

   Google Scholar 

10. Thomadsen, B. et al. AAPM task group report 288: Recommendations for guiding radiotherapy event narratives. Med Phys. 51, 5858–5872 (2024).

    Google Scholar 

11. Huq, M. S. et al. The report of Task Group 100 of the AAPM: Application of risk analysis methods to radiation therapy quality management. Med. Phys. 43, 4209–4262 (2016).

    Google Scholar 

12. Yang, F. et al. Validating FMEA output against incident learning data: A study in stereotactic body radiation therapy. Med. Phys. 42, 2777–2785 (2015).

    Google Scholar 

13. Paradis, K. C. et al. The fusion of incident learning and failure mode and effects analysis for data-driven patient safety improvements. Pr. Radiat. Oncol. 11, e106–e113 (2021).

    Google Scholar 

14. Shappell, S. A. & Wiegmann, D. A. The Human Factors Analysis and Classification System—HFACS. (2000).

15. Jalali, M., Dehghan, H., Habibi, E. & Khakzad, N. Application of ‘Human Factor Analysis and Classification System’ (HFACS) model to the prevention of medical errors and adverse events: A systematic review. Int J. Prev. Med. 14, 127 (2023).

    Google Scholar 

16. Diller, T. et al. The human factors analysis classification system (HFACS) applied to health care. Am. J. Med. Qual. 29, 181–190 (2014).

    Google Scholar 

17. Lee, J.-Y. et al. Applying the human factors analysis and classification system (HFACS) within root cause analysis (RCA) to prevent medical errors and enhancing patient safety culture: Insights from a medical center. Int. J. Quality Health Care https://doi.org/10.1093/intqhc/mzaf009 10.1093/intqhc/mzaf009. (2025).

18. Mosaly, P. R. et al. Application of human factors analysis and classification system model to event analysis in radiation oncology. Pract. Radiat. Oncol. 5, 113–119 (2015).

    Google Scholar 

19. Chan, A. J. et al. The use of human factors methods to identify and mitigate safety issues in radiation therapy. Radiother. Oncol. 97, 596–600 (2010).

    Google Scholar 

20. Judy, G. D. et al. Incorporating human factors analysis and classification system (HFACS) into analysis of reported near misses and incidents in radiation oncology. Pr. Radiat. Oncol. 10, e312–e321 (2020).

    Google Scholar 

21. McGurk, R. et al. Multi-institutional stereotactic body radiation therapy incident learning: Evaluation of safety barriers using a human factors analysis and classification system. J. Patient Saf. 19, e18–e24 (2023).

    Google Scholar 

22. Weintraub, S. M., Salter, B. J., Chevalier, C. L. & Ransdell, S. Human factor associations with safety events in radiation therapy. J. Appl Clin. Med Phys. 22, 288–294 (2021).

    Google Scholar 

23. RO-ILS and Clarity PSO. Aggregate Data Report Quarter 3, 2024. https://www.astro.org/getmedia/49e5d239-14e3-4e93-b148-75b9f3ce178e/ROILS_2024_Q3.pdf (2024).

24. Savova, G. K. et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): Architecture, component evaluation and applications. J. Am. Med. Inform. Assoc. 17, 507–513 (2010).

    Google Scholar 

25. Yalamanchili, A. et al. Quality of large language model responses to radiation oncology patient care questions. JAMA Netw. Open 7, e244630 (2024).

    Google Scholar 

26. Wang, P. et al. Fine-tuning large language models for radiation oncology, a specialized health care domain. Int. J. Radiat. Oncol. *Biol. *Phys. 120, e664 (2024).

    Google Scholar 

27. Ganguly, I., Buhrman, G., Kline, E., Mun, S. K. & Sengupta, S. Automated error labeling in radiation oncology via statistical natural language processing. Diagnostics 13, 1215 (2023).

    Google Scholar 

28. Mathew, F., Wang, H., Montgomery, L. & Kildea, J. Natural language processing and machine learning to assist radiation oncology incident learning. J. Appl Clin. Med Phys. 22, 172–184 (2021).

    Google Scholar 

29. Syed, K. et al. Automatic incident triage in radiation oncology incident learning system. Healthcare 8, 272 (2020).

    Google Scholar 

30. Chen, H., Cohen, E., Wilson, D. & Alfred, M. A machine learning approach with human-AI collaboration for automated classification of patient safety event reports: Algorithm development and validation study. JMIR Hum. Factors 11, e53378 (2024).

    Google Scholar 

31. Tabaie, A., Sengupta, S., Pruitt, Z. M. & Fong, A. A natural language processing approach to categorise contributing factors from patient safety event reports. BMJ Health Care Inf. 30, e100731 (2023).

    Google Scholar 

32. Fong, A. Realizing the power of text mining and natural language processing for analyzing patient safety event narratives: The challenges and path forward. J. Patient Saf. 17, e834–e836 (2021).

    Google Scholar 

33. Aronson, A. R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp 17–21 (2001).

34. Dubey, A. et al. The Llama 3 Herd of Models. http://arxiv.org/abs/2407.21783 (2024).

35. Ezzell, G. et al. Common error pathways seen in the RO-ILS data that demonstrate opportunities for improving treatment safety. Pr. Radiat. Oncol. 8, 123–132 (2018).

    Google Scholar 

36. Hurkmans, C. et al. A joint ESTRO and AAPM guideline for development, clinical validation and reporting of artificial intelligence models in radiation therapy. Radiother. Oncol. 197, 110345 (2024).

    Google Scholar 

37. RLDatix. Event Reporting. https://rldatix.com/en-nam/solutions/how-we-help/risk/event-reporting/.

38. Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. http://arxiv.org/abs/1904.03323 (2019).

39. Liu, Y. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. http://arxiv.org/abs/1907.11692 (2019).

40. Kailas, P., Goto, S., Homilius, M., MacRae, C. A. & Deo, R. C. Robust DeID: De-Identification of Medical Notes using Transformer Architectures. https://pypi.org/project/robust-deid/ (2022).

41. Fan, J.-W. & Friedman, C. Word sense disambiguation via semantic type classification. AMIA Annu Symp. Proc. 2008, 177–181 (2008).

    Google Scholar 

42. Rios, A. M. PYMETAMAP. https://github.com/AnthonyMRios/pymetamap.

43. Hugging Face. PEFT (Parameter-Efficient Fine-Tuning). https://huggingface.co/docs/peft/index.

44. Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. https://doi.org/10.48550/arXiv.2106.09685 (2021).

45. Nvidia Corporation. NVIDIA A100 Tensor Core GPU. https://www.nvidia.com/en-us/data-center/a100/?srsltid=AfmBOoqVawK32CuqvYk3XmDuvfXewcDMEEAWkXbjyb4hF8O3XVe459UR.

46. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. http://arxiv.org/abs/1810.04805 (2019).

47. Li, M., Gao, Q. & Yu, T. Kappa statistic considerations in evaluating inter-rater reliability between two raters: Which, when and context matters. BMC Cancer 23, 799 (2023).

    Google Scholar 

Download references

Authors and Affiliations

1. Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, USA

   Abbas J. Jinia, Katherine Chapman, Shi Liu, Cesar Della Biancia, Elizabeth Hipp, Eric Lin, Anyi Li & Jean M. Moran

2. Division of Quality and Safety, Memorial Sloan Kettering Cancer Center, New York, NY, USA

   Robin Moulder

3. Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, USA

   Dhwani Parikh, Jason Cordero, Caralaina Pistone & Mary Gil

4. Department of Nursing, Memorial Sloan Kettering Cancer Center, New York, NY, USA

   John Ford

Contributions

A.J.J., K.C., A.L., and J.M.M. conceptualized the study, directly accessed, and verified the data, conducted the formal analysis and methodology, administered the project, created the software, visualized the data, wrote the original draft of the manuscript, and reviewed and edited the manuscript. S.L., C.D.B., E.H., E.L., R.M., D.P., J.C., C.P., M.G., and J.F. contributed to data curation, methodology, and review and editing of the manuscript. All authors had full access to all data in the study and had final responsibility for the decision to submit for publication.

Corresponding authors

Correspondence to Abbas J. Jinia, Anyi Li or Jean M. Moran.

Competing interests

The authors declare no competing interests related to this work. EH declares the following disclosures: Volunteering position on the executive board of the Knickerbocker Chapter NSDAR. J.M.M. declares the following disclosures: Grants from Varian received during my time at the University of Michigan. Honoraria paid by the Connecticut Area Medical Physics Society. Patient issued for Combined radiation acoustics and ultrasound for radiotherapy guidance and cancer targeting. Co-founder and board member at Prexient, Inc. Chair of Work Group on Science Council EDI at the American Association of Physicists in Medicine. Chair of Work Group on Report Writing at the American Association of Physicists in Medicine. Vice Chair of Research Committee at the Radiation Oncology Institute. Co-Chair at the Radiation Oncology Institute Radiation Oncology Safety Stakeholders Initiative. FuseOncology (Copyright), which has been licensed by my previous institution (University of Michigan) for which I made a contribution. Consultant (unfunded) to the Michigan Radiation Oncology Quality Consortium.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions