Lorin Crawford - When more isn't better: Rethinking Scale in Single-cell Foundation Models

Value of the Information & Strength of the Argument

The talk provides valuable insights into the limitations of current single-cell foundation models, backed by systematic experiments. The argumentation is solid, with clear hypotheses and rigorous evaluation. The finding that performance plateaus with dataset size is counterintuitive and important for the field. The speaker also highlights the issue of batch effects in embeddings, which is a critical concern. The discussion of data composition adds further depth, showing that simply adding more data, especially from disease states, does not necessarily improve generalization. The talk is well-structured and the speaker is transparent about the negative results, which is valuable for the community.

Scientific Rigor, Source Quality, Title Accuracy

The talk is based on original research, and the speaker references several models and datasets (e.g., scGPT, Geneformer, CELLxGENE). However, specific citations are not provided in the talk itself. The description includes a link to the workshop page, which may contain additional resources. The title accurately reflects the content. The speaker does not cite specific papers, but the work appears to be rigorous and is presented by a credible researcher. The talk is part of a workshop, so the audience is likely familiar with the field.

The title accurately reflects the content, focusing on the counterintuitive finding that larger pre-training datasets do not necessarily improve performance in single-cell foundation models.

8/10

The talk presents original research from a reputable group (Microsoft Research, Broad, Dana-Farber) with a systematic evaluation of single-cell foundation models. The methodology is described in detail, and the results are presented with appropriate caveats. However, the talk is a conference presentation and not a peer-reviewed publication, and some claims are based on unpublished work.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction to Project Xvivo and cell state concept
• 05:00Explanation of foundation models and zero-shot vs fine-tuning
• 10:00Overview of existing single-cell foundation models and their training data sizes
• 15:00Evaluation of models: they predict the mean and embeddings capture batch effects
• 20:00Question about feature selection and model evaluation
• 25:00Experiments on dataset size: performance plateaus at a fraction of the data
• 30:00Experiments on data composition: generalization to unseen cell types is poor
• 35:00Discussion of implications and future directions
• 40:00Conclusion and call for collaboration

• Mathematics of Cancer: Open Mathematical Problems Workshop — Workshop page where the talk was presented

• scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI — The scGPT paper, which the talk evaluates and finds limitations in.
• Geneformer: Transfer learning in transcriptomics enables robust prediction of cell type and disease — The Geneformer paper, another model evaluated in the talk.

• scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI — The scGPT paper claims strong performance, but the talk shows that it may not generalize well and often predicts the mean.