Development and validation of a multimodal AI-agent system for prognosis analysis of bladder urothelial carcinoma – npj Precision Oncology

The source code is available online (https://github.com/hqh1997/MMS_AI_agent).

1. Nadal, R., Valderrama, B. P. & Bellmunt, J. Progress in systemic therapy for advanced-stage urothelial carcinoma. Nat. Rev. Clin. Oncol. 21, 8–27 (2024).

   Google Scholar 

2. Hemenway, G. et al. Advancements in Urothelial Cancer Care: Optimizing Treatment for Your Patient. Am. Soc. Clin. Oncol. Educ. Book 44, e432054 (2024).

   Google Scholar 

3. Lopez-Beltran, A., Cookson, M. S., Guercio, B. J. & Cheng, L. Advances in diagnosis and treatment of bladder cancer. BMJ 384, e076743 (2024).

   Google Scholar 

4. Soualhi, A. et al. The incidence and prevalence of upper tract urothelial carcinoma: a systematic review. BMC Urol. 21, 110 (2021).

   Google Scholar 

5. Compérat, E. et al. Current best practice for bladder cancer: a narrative review of diagnostics and treatments. Lancet 400, 1712–1721 (2022).

   Google Scholar 

6. Shen, J., Li, Z., Wang, R., Ding, G. & Zhang, Y. Bladder cancer diagnostic and prognostic models from DNA methylation by multi algorithm machine learning. NPJ Precis. Oncol. https://doi.org/10.1038/s41698-025-01195-y (2025).

7. Raman, S. P. & Fishman, E. K. Bladder malignancies on CT: the underrated role of CT in diagnosis. AJR Am. J. Roentgenol. 203, 347–354 (2014).

8. Compérat, E. et al. Updated pathology reporting standards for bladder cancer: biopsies, transurethral resections and radical cystectomies. World J. Urol. 40, 915–927 (2022).

   Google Scholar 

9. Perez-Lopez, R., Ghaffari Laleh, N., Mahmood, F. & Kather, J. N. A guide to artificial intelligence for cancer researchers. Nat. Rev. Cancer https://doi.org/10.1038/s41568-024-00694-7 (2024).

10. McGenity, C. et al. Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy. npj Dig. Med. 7, 114 (2024).

    Google Scholar 

11. He, J. et al. Development of a deep learning model for T1N0 gastric cancer diagnosis using 2.5D radiomic data in preoperative CT images. NPJ Precis. Oncol. 9, 249 (2025).

    Google Scholar 

12. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).

    Google Scholar 

13. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (eds Sebastien Ourselin et al) 424–432 (Springer International Publishing, 2016).

14. Shao, Z. TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In Proc. 35th International Conference on Neural Information Processing Systems 2136–2147 (ACM, 2021).

15. Chen, R. J. et al. in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24. 339–349 (Springer, 2021).

16. Saldanha, O. L. et al. Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology. NPJ Precis. Oncol. 7, 35 (2023).

    Google Scholar 

17. Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nat. Med. 30, 863–874 (2024).

    Google Scholar 

18. Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850–862 (2024).

    Google Scholar 

19. Yuan, Y. et al. Cell graph analysis in hepatocellular carcinoma: predicting local recurrence and identifying spatial relationship biomarkers. NPJ Precis. Oncol. 9, 261 (2025).

    Google Scholar 

20. Pisula, J. I. et al. Explainable, federated deep learning model predicts disease progression risk of cutaneous squamous cell carcinoma. NPJ Precis. Oncol. 9, 205 (2025).

    Google Scholar 

21. Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021).

    Google Scholar 

22. Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878.e866 (2022).

    Google Scholar 

23. Wang, G. et al. DeepIGeoS: a deep interactive geodesic framework for medical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1559–1572 (2018).

24. Luo, X. et al. MIDeepSeg: minimally interactive segmentation of unseen objects from medical images using deep learning. Med. Image Anal. 72, 102102 (2021).

25. Liang, J. et al. Deep learning supported discovery of biomarkers for clinical prognosis of liver cancer. Nat. Mach. Intell. 5, 408–420 (2023).

    Google Scholar 

26. He, Q. et al. Integrated multicenter deep learning system for prognostic prediction in bladder cancer. NPJ Precis. Oncol. 8, 233 (2024).

    Google Scholar 

27. Lee, Y., Ferber, D., Rood, J. E., Regev, A. & Kather, J. N. How AI agents will change cancer research and oncology. Nat. Cancer 5, 1765–1767 (2024).

    Google Scholar 

28. Ferber, D. et al. Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology. Nat. Cancer https://doi.org/10.1038/s43018-025-00991-6 (2025).

29. Liu, X. et al. A generalist medical language model for disease diagnosis assistance. Nat. Med. 31, 932–942 (2019).

    Google Scholar 

30. Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision 128, 336–359 (2020).

31. Suzuki, H. et al. Exploring drug resistance via intercellular crosstalk using spatial transcriptomics in high-grade serous ovarian carcinoma. NPJ Precis. Oncol. 9, 345 (2025).

    Google Scholar 

32. Li, T. et al. Computational pathology annotation enhances the resolution and interpretation of breast cancer spatial transcriptomics data. NPJ Precis. Oncol. 9, 310 (2025).

    Google Scholar 

33. Hatamizadeh, A. et al. in International MICCAI brainlesion workshop. 272–284 (Springer).

34. Kirillov, A. et al. Segment anything. in Proc. IEEE/CVF International Conference on Computer Vision 4015–4026 (IEEE, 2023).

35. Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. in Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5987–5995 (IEEE, 2017).

36. Ganesan, K. ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks. Preprint at https://arxiv.org/abs/1803.01937 (2018).

37. Zhang, T. et al. BERTScore: Evaluating text generation with BERT. Int. Conf. Learn. Represent. (2020).

38. Chen, R. J. et al. in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) 3995–4005 (IEEE, 2021).

39. Chen, R. J. et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41, 757–770 (2022).

    Google Scholar 

40. Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. Preprint at https://doi.org/10.48550/arXiv.2009.07896 (2020).

41. Zadeh, S. G. & Schmid, M. Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3126–3137 (2021).

Download references

Authors and Affiliations

1. Department of Urology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China

   Quanhao He, Hao Tan, Bangxin Xiao, Xiang Peng, Canjie Peng, Weiyang He & Mingzhao Xiao

2. Department of Pathology, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, China

   Yiwen Tan

3. Department of Pathology, Yongchuan Hospital of Chongqing Medical University, Chongqing, China

   YingJia Liu

4. Department of Pathology, College of Basic Medicine, Chongqing Medical University, Chongqing, China

   Youde Cao

5. Department of Clinical Pathololgy Laboratory of Pathology Diagnostic Center, Chongqing Medical University, Chongqing, China

   Youde Cao

6. Molecular Medicine Diagnostic and Testing Center, Chongqing Medical University, Chongqing, China

   Youde Cao

7. Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China

   Fa Jin Lv

8. College of Artificial Intelligence Medicine, Chongqing Medical University, Chongqing, China

   Wenlong Zhao

9. College of Biomedical Engineering, Chongqing Medical University, Chongqing, China

   Wenlong Zhao

10. Department of Urology, The Third Affiliated Hospital of Chongqing Medical University, Chongqing, China

    Xiaofeng Yue

Contributions

Q.H.H., H.T., B.X.X., Y.W.T., X.P., W.Y.H., and M.Z.X. conceived and designed the study; H.T., C.J.P., X.F.Y., X.P., and X.Z. collected the data. Q.H.H., H.T., and C.J.P. evaluated images. Y.J.L., Y.W.T., and D.Y.C. labeled the pathological slide images. F.J.L. supervised and annotated the radiographic images. Q.H.H., W.L.Z., and X.P. trained and developed the AI system. Q.H.H., B.X.X., and Y.W.T. analyzed and interpreted the data and wrote the original draft of the manuscript. Q.H.H. and X.F.Y. were responsible for revising the manuscript and performing supplementary experiments. W.L.Z., X.F.Y., W.Y.H., and M.Z.X. supervised and directed the study.

Corresponding authors

Correspondence to Wenlong Zhao, Xiaofeng Yue, Weiyang He or Mingzhao Xiao.

Competing interests

The authors declare no competing interests.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions