Day 2: Mohammad Yaqub - The Medical Multimodal Mind Building Foundation Models for Health

Day 2: Mohammad Yaqub - The Medical Multimodal Mind Building Foundation Models for Health

by Mohammad Yaqub

Applied Sciences & Engineering AI — Models & Research UYQArtificial intelligence (AI)UYQMMachine learning
🎙 Mohammad Yaqub 👥 824 📅 November 5, 2025 ⏱ 42 min 👁 103 📄 expert opinion 🧭 2026-08-16
Available in: English (current) Français

Keywords

multimodalfoundation modelszero-shotmedical imaginghealthcare

Summary

Mohammad Yaqub, Associate Professor of Computer Vision at MBZUAI, presents a talk on building multimodal foundation models for health, emphasizing the shift from single-modality AI to models that jointly understand medical images and clinical language. He introduces the concept of the ‘Medical Multimodal Mind’ and discusses zero-shot diagnosis, where models can recognize conditions they were not explicitly trained on. He illustrates the limitations of single-modality approaches and the need for joint representations. He presents two research projects: FetalCLIP, a foundation model for fetal ultrasound, which achieves high accuracy in view classification and biometrics, and a project on echocardiography benchmarking, showing that current foundation models often fail on out-of-distribution data. He argues that while foundation models show promise, they are not yet reliable for clinical use, and highlights challenges such as generalizability and the need for human-AI collaboration. The talk concludes with a call for more robust evaluation and development of these models.

152 words

Critical Evaluation

Value of the Information & Strength of the Argument

The talk provides valuable insights into the evolution of medical AI, from handcrafted features to deep learning and now foundation models. The argumentation is coherent, using concrete examples from fetal health and echocardiography to illustrate the potential and limitations. The speaker effectively communicates the importance of multimodal data and the challenges of zero-shot diagnosis. However, some claims, such as the 50% detection rate for congenital heart disease, are stated without citing specific sources, which weakens the scientific rigor. The argument that foundation models are not yet reliable is well-supported by the benchmarking results mentioned, but the lack of detailed data and methodology limits the depth of the analysis.

Scientific Rigor, Source Quality, Title Accuracy

The speaker is a credible expert in the field, and the talk references his own research projects (FetalCLIP, echocardiography benchmarking) and general concepts like CLIP. However, no specific external sources are cited, and the description only provides a link to the ADIA Lab symposium page. The title accurately reflects the content, which focuses on building multimodal foundation models for health. The talk is more of an expert opinion and overview of ongoing research rather than a detailed scientific presentation with rigorous citations. The lack of specific references and data makes it difficult to verify the claims independently.

220 words

Title / Content Match

The title accurately reflects the content, which focuses on building multimodal foundation models for health.

Quality & Reliability

7/10

The speaker is an associate professor with extensive research experience, and the talk presents plausible arguments and examples. However, specific data and results are mentioned without detailed citations, and some claims (e.g., detection rates) are presented without sources.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Concurring Sources

Contribution & Novelties

The talk provides a clear overview of the transition from single-modality to multimodal foundation models in medical AI, with concrete examples from fetal health and echocardiography. It highlights the potential of zero-shot diagnosis while also pointing out its limitations, particularly in out-of-distribution settings. The speaker’s perspective as a researcher actively building these models adds practical insight.

Pour aller plus loin :

  • CLIP (Contrastive Language-Image Pre-training) — The foundational model architecture mentioned in the talk for aligning images and text.
  • FetalCLIP — The speaker’s foundation model for fetal ultrasound (note: this is a placeholder URL; actual paper may be under review).
  • Zero-shot learning in medical imaging — General concept of zero-shot learning, relevant to the talk’s discussion.
  • Out-of-distribution generalization — Key challenge discussed in the talk.

125 words

Radar Profile

The radar profile shows high scores in information quantity and quality, reflecting the speaker's expertise and the breadth of topics covered. The technical level is moderate, suitable for a general audience, and the reliability is good but not perfect due to lack of detailed citations. Overall, the talk is informative and credible, with minor weaknesses in source rigor.

Reliability 7/10