Qwen 3.8 27B Just Hit 200 Token/s on Mac M5 (Lithos Metal)
Title history
- Qwen 3.8 27B Just Hit 200 Token/s on Mac M5 (Lithos Metal) — observed on 20261031-2309 UTC (YouTube)
Keywords
Summary
161 words
Critical Evaluation
Value of the Information & Strength of the Argument
The video provides substantial value by dissecting the 200 t/s claim, separating projected from measured performance, and offering context on acceptance rates and task types. The argumentation is solid, backed by specific data from the Lithos Metal repository and independent benchmarks. The author maintains a balanced tone, acknowledging the engine’s strengths in code generation while exposing its weaknesses in English text and prefill speed. The reasoning is logical and well-structured, guiding viewers through the technical concepts step by step.
Scientific Rigor, Source Quality, Title Accuracy
The scientific rigor is high: the video relies on primary sources from the Lithos Metal repo, including README, studies, and issues, and compares with independent benchmarks. It clearly distinguishes between projected and measured values, and cites the relevant research paper. The title adequacy is good, as it directly addresses the claim and then provides a critical analysis. The video also mentions the lack of transparency in some claims, enhancing its credibility.
165 words
Title / Content Match
The title accurately captures the video's focus on the 200 token/s claim, though the video critically examines and contextualizes that claim.
Quality & Reliability
9/10
The video provides a rigorous, well-sourced analysis distinguishing measured performance from projections, with clear explanations of benchmarking contexts.
Chapters
Cited Sources
- Lithos Metal repository — Main repository for the Lithos Metal engine.
- Lithos Metal README — Provides the performance projections and caveats.
- Measured DSpark study — Details the actual measured run showing ~80 t/s.
- Backend comparison study — Comparison methodology and projection details.
- LithosAI blog post — Original blog post claiming 200 t/s.
- DSpark paper (DeepSeek) — Foundational paper on speculative decoding used in Lithos Metal.
- Qwen3.8-27B model card — Details about the Qwen model used in the tests.
- MacBook Pro specifications — Official memory bandwidth specs for M5 Max variants.
- Reddit benchmark on plain 4-bit M5 Max — Independent benchmark showing 30.5 t/s baseline.
- AgentPerf Local benchmark — Benchmark for measuring real agent performance.
Concurring Sources
- Lithos Metal measured study — Shows measured average acceptance rate and ~80 t/s, supporting the video's analysis.
- Backend verification — Provides context on benchmarking methodologies.
Dissenting Sources
- LithosAI blog — Claims 200 t/s without clarifying it's a projection, contradicting the video's measured findings.
External References
Contribution & Novelties
The video’s original contribution is its critical examination of a viral performance claim, offering a nuanced perspective that is often missing in reaction videos. It introduces the concept of speculative decoding and explains the importance of acceptance rates and task types. It also highlights the prefill bottleneck in agentic workflows, a factor often overlooked. The video encourages viewers to measure end-to-end task times rather than relying on headline numbers.
Pour aller plus loin :
- DSpark paper on speculative decoding — The foundational paper introducing the technique.
- AgentPerf Local benchmark — A benchmark for measuring real agent performance.
- Apple MacBook Pro specifications — Official memory bandwidth specs for M5 Max variants.
110 words
Radar Profile
The video exhibits high scores across all dimensions, reflecting a well-balanced, informative, and technically sound analysis.