HAI Seminar with Sheng Wang: Generative AI for Multimodal Biomedicine

Item: HAI Seminar with Sheng Wang: Generative AI for Multimodal Biomedicine
Rating: 4
Author: PostSingularity Institute

by Sheng Wang

Applied Sciences & Engineering AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence

🎙 Sheng Wang 👥 34K 📅 November 8, 2024 ⏱ 65 min 👁 1K 📄 expert opinion 🧭 2026-08-06

Available in: English (current) Français

Keywords

GigaPathOCTCubeBiomedParsemultimodalfoundation model

Summary

In this seminar, Sheng Wang presents three recent works from his lab on multimodal biomedicine foundation models. He begins by outlining four paradigms of AI in medicine, from deep learning to multi-agent models, and emphasizes the inherent multimodality of medicine. The first work, GigaPath, is a whole-slide pathology foundation model that handles gigapixel images using long-context modeling, similar to ChatGPT. It achieves state-of-the-art performance on cancer subtyping and biomarker prediction across 15 cancer types. The second work, OCTCube, is a 3D retinal imaging foundation model that processes 3D OCT scans, addressing the challenge of analyzing volumetric data. The third work, BiomedParse, integrates nine major biomedical imaging modalities by projecting them into text space, enabling unified analysis. Wang also discusses the use of chain-of-thought reasoning guided by clinical guidelines to improve treatment prediction, and highlights the potential for AI models to act as clinical lab tests. The talk concludes with a Q&A session.

152 words

Critical Evaluation

The seminar provides a comprehensive overview of recent advances in multimodal biomedical AI, with a focus on three specific models developed by the speaker’s lab. The presentation is well-structured, starting with a clear framework of AI paradigms in medicine, which helps contextualize the work. The technical depth is appropriate for an academic audience, with explanations of long-context modeling, chain-of-thought, and multimodal integration. The speaker demonstrates strong command of the subject, drawing on published research in Nature and collaborations with Microsoft Research. The claims are supported by empirical results, such as performance on 25 out of 26 tasks and 17 out of 18 biomarkers, which adds credibility. However, the talk is primarily a presentation of the speaker’s own work, and while it references clinical guidelines, it does not provide a critical comparison with alternative approaches or discuss limitations in depth. The discussion of future directions, such as multi-agent frameworks and integration with multi-omics, is forward-looking but speculative. The Q&A segment, though not transcribed, likely addressed some concerns. Overall, the information is reliable and valuable for researchers in the field, but the lack of independent verification and potential bias towards the speaker’s own models should be considered. The title accurately reflects the content, and the presentation is well-suited for an academic audience.

210 words

Title / Content Match

The title accurately reflects the content, focusing on generative AI for multimodal biomedicine, with three specific models presented.

Quality & Reliability

8/10

The speaker is an assistant professor at UW, with publications in Nature and collaborations with Microsoft Research. The talk presents original research with clear methodology and results, though it is a seminar presentation rather than a peer-reviewed publication.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

GigaPath paper in Nature — Published a few months ago, collaboration with Microsoft Research, UW, and Providence Genomics.

Concurring Sources

GigaPath paper in Nature — The paper is the primary source for the claims about GigaPath's performance.

Contribution & Novelties

The seminar presents three novel foundation models for biomedical imaging: GigaPath for whole-slide pathology, OCTCube for 3D retinal imaging, and BiomedParse for integrating multiple imaging modalities. The key innovation is the application of generative AI techniques, such as long-context modeling and chain-of-thought, to medical imaging, enabling analysis of gigapixel images and multimodal data. The work demonstrates that a single model can generalize across cancer types and biomarkers, and that clinical guidelines can be used as a chain-of-thought to improve treatment prediction.

Pour aller plus loin :

Vision Transformer (ViT) — The underlying architecture for many foundation models, including those discussed.
Chain-of-Thought Prompting — The technique used to decompose complex tasks, applied here to clinical guidelines.
Multimodal Learning — The broader field of integrating multiple data types, relevant to BiomedParse.

129 words

Radar Profile

The radar profile shows high scores in information quantity, quality, and reliability, with a slightly lower technical level, indicating a presentation that is rich in content and well-supported, but not overly technical for a general academic audience.

Reliability 8/10

💬 No comments were provided for analysis.