Generative AI L26: LLM jail breaks, DAN jail breaks. Post training and fine-tuning LLMs

Item: Generative AI L26: LLM jail breaks, DAN jail breaks. Post training and fine-tuning LLMs
Rating: 4
Author: PostSingularity Institute

by Agha Ali Raza

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence

🎙 Agha Ali Raza 👥 3K 📅 May 23, 2026 ⏱ 70 min 👁 49 📄 lecture 🧭 2026-08-15

Available in: English (current) Français

Keywords

jailbreakDANprompt injectionalignmentfine-tuning

Summary

This lecture from a graduate course on Generative AI discusses the limitations of pre-trained language models and the need for alignment. It covers LLM jailbreaks, specifically the DAN (Do Anything Now) attack, and prompt injection attacks. The instructor explains how models trained on internet data can produce harmful content if not properly aligned. He then discusses post-training and fine-tuning methods to align models, including catastrophic forgetting and the trade-offs of single-task vs multi-task fine-tuning. The lecture emphasizes the importance of data hygiene and the risks of sharing personal information with AI systems.

92 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture provides valuable insights into the practical challenges of deploying LLMs, with concrete examples of jailbreaks and prompt injection attacks. The argumentation is solid, drawing on a peer-reviewed paper and real-world incidents. The instructor effectively explains complex concepts in an accessible manner, making the content valuable for both beginners and experienced practitioners.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is scientifically rigorous, referencing a peer-reviewed paper from ACM CCS 2024 and a CNBC article. The title accurately reflects the content. The instructor also mentions his own experience with prompt injection in peer review, adding credibility. The content is well-structured and aligns with the course objectives.

116 words

Title / Content Match

The title accurately reflects the content, covering LLM jailbreaks and post-training/fine-tuning.

Quality & Reliability

8/10

Lecture by a university professor, part of a graduate course, with references to a peer-reviewed paper and a news article. The content is well-structured and based on established knowledge in the field.

Chapters

Cited Sources

Course materials and assessments (CSaLT) — Slides and assessments for the course.
Full playlist of lectures — All lecture videos for the course.

Concurring Sources

ACM CCS 2024 paper on jailbreaks — Referenced in the lecture as a comprehensive analysis of jailbreak attacks.

Contribution & Novelties

The lecture provides a comprehensive overview of LLM jailbreaks and alignment, with practical examples and references. It highlights the ongoing cat-and-mouse game between jailbreak techniques and defenses.

Pour aller plus loin :

Prompt injection attacks — Overview of prompt injection attacks.
Adversarial attacks on LLMs — Survey of adversarial attacks on language models.
Catastrophic forgetting — Explanation of catastrophic forgetting in neural networks.

63 words

Radar Profile

The radar profile shows high scores in information quantity and quality, with moderate technical depth and reliability. This indicates a well-balanced lecture that is both informative and accessible.

Reliability 8/10