Blog
multimodal-deep-learning-with-clinical-metadata-for-skin-lesion-classification-–-scientific-reports

Multimodal deep learning with clinical metadata for skin lesion classification – Scientific Reports

Abstract

Accurate skin lesion classification often requires integration of lesion appearance with patient clinical information. Although multimodal artificial intelligence (AI) approaches have shown promise, many existing methods rely on computationally intensive fusion mechanisms or simple integration strategies that may inadequately capture interactions between imaging and clinical data. This study aimed to develop and evaluate a multimodal deep learning model that integrates skin images and structured clinical metadata for skin lesion classification and to compare its performance with image-based and prior multimodal baseline models. This retrospective diagnostic study evaluated dataset-specific model performance using 2 public datasets (PAD-UFES-20 and ISIC 2019) and 1 institutional dataset (PKU-Skin). PAD-UFES-20 included 2,298 clinical images with 21 structured clinical variables across 6 lesion classes. ISIC 2019 included 25,331 dermoscopic images across 8 diagnostic categories. PKU-Skin was an institutional dataset of 800 clinical images (643 patients) with corresponding clinical metadata collected at Peking University Shenzhen Hospital and represented a binary classification task involving psoriasis and basal cell carcinoma. Model development and evaluation were performed separately within each dataset. The primary outcome was correct lesion classification. The primary evaluation metric was balanced accuracy. Secondary metrics included sensitivity, specificity, precision, F1 score, and overall accuracy. On PAD-UFES-20, the proposed multimodal model achieved a balanced accuracy of 85.6%, compared with 75.1% for a prior multimodal baseline model, with sensitivity of 76.1%, specificity of 95.2%, F1 score of 78.1%, and overall accuracy of 78.3%. On ISIC 2019, the model achieved a balanced accuracy of 86.0%, F1 score of 84.3%, and overall accuracy of 84.5%. On PKU-Skin, the model achieved a balanced accuracy of 99.6% and overall accuracy of 99.2%; because this dataset represented a separate binary classification task, these results should be interpreted as supportive evidence from an institutional setting rather than formal external validation of a single fixed model. In these retrospective dataset-specific experiments, integrating structured clinical metadata with skin images was associated with improved skin lesion classification performance relative to comparator models. These findings support further evaluation of multimodal approaches for dermatologic decision support, particularly in prospectively collected and clinically matched external datasets.

Acknowledgements

We would like to thank the participants and their families for their support of this project.

Funding

This study was supported by grants from Shenzhen Science and Technology Program (No.: JCYJ20240813120002004), Shenzhen High-level Hospital Construction Fund & Peking University Shenzhen Hospital Scientific Research Fund (Nos.: KYQD2025471, KYQD2025472), “Hengjing” Skin Immunity Research Foundation (No.: HJ-DER-2024-39), and Training Action Project for Young and Middle-aged Teachers in Colleges and Universities of Anhui Province (No.: JWFX2026007).

Author information

Author notes

  1. Ruibo Wang, Dongying Luo, Ziyi Shen and Zhaoping Lin are contributed equally to this work.

Authors and Affiliations

  1. Research Center for Brain Computer Interface, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518067, Guangdong, China

    Ruibo Wang, Ziyi Shen & Kun Shang

  2. Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, 2600 AA, Delft, Netherlands

    Ruibo Wang & Feiyang Liu

  3. Department of Dermatology, Peking University Shenzhen Hospital, Shenzhen, 518036, Guangdong, China

    Dongying Luo, Zhaoping Lin, Hui Li, Ning Xu, Tao Tao, Weiwei Tian, Cong Huang, Bo Yu & Changbing Shen

  4. Shenzhen Key Laboratory for Translational Medicine of Dermatology, Institute of Dermatology, Shenzhen Peking University – The Hong Kong University of Science and Technology Medical Center, Shenzhen, 518036, Guangdong, China

    Dongying Luo, Zhaoping Lin, Hui Li, Ning Xu, Tao Tao, Weiwei Tian, Cong Huang, Bo Yu & Changbing Shen

  5. PKU-Shenzhen Clinical Institute of Shantou University Medical College, Shenzhen, 518036, Guangdong, China

    Dongying Luo, Zhaoping Lin, Tao Tao, Bo Yu & Changbing Shen

  6. School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, Guangdong, China

    Ziyi Shen

  7. Department of Dermatology, Shenzhen Xinhua Hospital, Shenzhen, 518131, Guangdong, China

    Jieyi Wang

  8. Department of Dermatology and Venereology, The Second Affiliated Hospital of Anhui Medical University, Hefei, 230601, Anhui, China

    Xiaoting Wu & Jing Gao

  9. Joint Laboratory for Plasma Clinical Applications, The Second Affiliated Hospital of Anhui Medical University, Hefei, 230601, Anhui, China

    Xiaoting Wu & Jing Gao

  10. Guangdong Provincial Key Laboratory of Multimodality Non-Invasive Brain-Computer Interfaces, Shenzhen, 518067, China

    Kun Shang

  11. Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518067, China

    Kun Shang

Authors

  1. Ruibo Wang
  2. Dongying Luo
  3. Ziyi Shen
  4. Zhaoping Lin
  5. Hui Li
  6. Feiyang Liu
  7. Ning Xu
  8. Jieyi Wang
  9. Tao Tao
  10. Xiaoting Wu
  11. Weiwei Tian
  12. Cong Huang
  13. Bo Yu
  14. Jing Gao
  15. Changbing Shen
  16. Kun Shang

Corresponding authors

Correspondence to Jing Gao, Changbing Shen or Kun Shang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, R., Luo, D., Shen, Z. et al. Multimodal deep learning with clinical metadata for skin lesion classification. Sci Rep (2026). https://doi.org/10.1038/s41598-026-78124-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41598-026-78124-x

Keywords

colind88

Related Posts

nvidia-ai-day-seoul-highlights-korea’s-advances-in-sovereign,-agentic-and-physical-ai-—-live-updates
NVIDIA AI Day Seoul Highlights Korea’s Advances in Sovereign, Agentic and Physical AI — Live Updates

NVIDIA AI Day Seoul opens at COEX, bringing together more than 2,000 developers, startups, enterprises and researchers together for sessions, hands-on NVIDIA Deep Learning Institute workshops and special events. The event celebrates Korea’s developer ecosystem and the latest advancements across open models and agents for sovereign AI, physical AI and industrial transformation, AI for science

ai-model-classifies-arrhythmia-risk-in-three-clinical-levels
AI Model Classifies Arrhythmia Risk in Three Clinical Levels

Researchers from a collaborative team including Hua Zhang, Jing Li, and Haiyan Wang have developed a deep learning algorithm that stratifies cardiac arrhythmias into three distinct risk levels: normal, not life-threatening, and life-threatening. Published in the journal BioData Mining in July 2026, the study introduces a model named PaliConvLSTM, which utilizes a palindromic structure of

Back To Top