Multimodal AI for Precision Cancer Screening

This presentation from the Northwestern Medicine Healthcare AI Forum features two speakers from UCLA: Dr. William Hsu and PhD candidate Luoting Zhuang. Dr. Hsu begins by motivating the need for multimodal data in cancer screening, using a car analogy to illustrate how combining diverse data sources (imaging, clinical, molecular) can improve diagnostic accuracy. He discusses challenges in building robust multimodal models, including data unpairedness and incomplete EHRs, and highlights his lab’s integrated diagnostics workflow for collecting imaging, blood, nasal swabs, and biopsy tissues. He explains different fusion strategies (early, intermediate, late) and presents preliminary results on combining CT imaging and methylation data for pulmonary nodule malignancy prediction, showing intermediate fusion performs best. He emphasizes the importance of rigorous validation to avoid shortcuts and hallucinations in AI models. Luoting Zhuang then presents her work on a vision-language model for lung nodule malignancy prediction, which integrates semantic features from radiology reports with imaging features to improve explainability and performance. She details the model architecture and results, showing that the multimodal approach outperforms image-only models. The talk concludes with a discussion of future directions, including longitudinal monitoring and environmental factors.