ICL Ciphers: Quantifying "Learning" in In-Context Learning via Substitution Ciphers - EMNLP 2025 main
Keywords
Summary
148 words
Critical Evaluation
Value of the Information & Strength of the Argument
The value of the information lies in proposing a novel, principled method to disentangle task retrieval and task learning in ICL, a fundamental problem in understanding LLMs. The argumentation is solid: the use of bijective vs. non-bijective ciphers provides a controlled comparison, isolating the effect of learnable patterns. The empirical results across multiple models and datasets strengthen the claim. However, the presentation is brief and does not delve into potential limitations or alternative interpretations, which could be a weakness.
Scientific Rigor, Source Quality, Title Accuracy
The work is based on a peer-reviewed paper (EMNLP 2025) and the presentation follows a clear scientific structure. The source is the arXiv paper link provided in the description. The title accurately reflects the content. No comments were provided, so no analysis of public reception is possible.
141 words
Title / Content Match
The title accurately reflects the content, which focuses on quantifying in-context learning via substitution ciphers.
Quality & Reliability
8/10
The presentation is based on a peer-reviewed paper (EMNLP 2025) and provides a clear methodology with empirical results across multiple models and datasets. The speaker explains the approach and findings systematically, though the talk is concise and lacks deep technical details.
Key Moments
Markers derived by PSI from the transcript: the creator did not define chapters.
- Introduction to In-Context Learning (ICL) and its dual modes: task retrieval and task learning.
- Explanation of the challenge in disentangling task retrieval and task learning, and previous approaches.
- Introduction to ICL Ciphers: definition and types (bijective and non-bijective).
- How to quantify task learning using the performance gap between bijective and non-bijective ciphers.
- Presentation of results: consistent gaps across models and datasets, effect of shuffle rate and number of demonstrations.
- Conclusion and future work directions.
Cited Sources
- ICL Ciphers: Quantifying "Learning" in In-Context Learning via Substitution Ciphers — The paper presenting the ICL Ciphers method and its results.
Concurring Sources
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — This paper discusses the mechanisms of ICL and the role of demonstrations, supporting the dual-mode hypothesis.
Dissenting Sources
- No, Language Models are not Emergent Reasoners — This work questions the extent of emergent abilities in LLMs, which could be seen as contrasting with the idea of task learning in ICL.
Contribution & Novelties
The main contribution is a novel method to quantify task learning in ICL by using substitution ciphers, which are unlikely to have been seen during pre-training. This allows for a more reliable disentanglement of task retrieval and task learning compared to previous label-manipulation approaches. The method is simple yet effective, providing consistent evidence across multiple models and datasets.
Pour aller plus loin :
- In-context learning — Background on ICL.
- Substitution cipher — Classic cryptography concept used.
- Large language models — Context on LLMs.
83 words
Radar Profile
The radar profile shows a balanced performance across all dimensions, with slightly higher scores in quality and reliability, indicating a well-structured and credible presentation. The lower score in quantity suggests the talk is concise, but the technical depth is adequate for the target audience.