llama.cpp Just Made 125B MoE Models 4× Faster!

llama.cpp Just Made 125B MoE Models 4× Faster!

Title history
  • llama.cpp Just Made 125B MoE Models 4× Faster! — observed on 20261031-2304 UTC (YouTube)

by Kai · View the channel on PSI

Applied Sciences & Engineering AI — News & Applications UYComputer scienceUYQArtificial intelligence (AI)
🎙 Kai 👥 65K 📅 October 22, 2026 ⏱ 28 min 👁 60K 📄 news review 🧭 2026-10-31
Available in: English (current) Français

Keywords

MoEexpert cachehit ratePCIelocal LLM

Summary

The video analyzes llama.cpp’s new MoE expert cache (released in v0.7.0) that claims up to 4× speedup for 125B parameter models on 24GB GPUs. The host explains the technical background of Mixture-of-Experts models, where only 2% of experts are active per token, making VRAM usage inefficient. The cache stores recently used experts in VRAM, using an LRU policy, and achieves gains depending on hit rate and PCIe bandwidth. The 4× headline is misleading as it compares against a baseline with zero experts in VRAM; a more realistic improvement is 1.55–2.2×. Small VRAM cards (8–12GB) are slowed down due to insufficient cache size, with a break-even hit rate of ~85%. The video also covers Windows pinned-memory fixes, prompt processing trade-offs, speculative decoding effects, and compares llama.cpp with the Strata runtime. It ends with a 15-minute testing procedure and stresses the importance of individual benchmarking.

143 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video provides high value by demystifying the performance numbers and giving practical advice. It argues that the speedup is real but conditional on hardware and workload, using concrete examples and data from multiple sources. The reasoning is logical and well-supported, with clear explanations of underlying concepts like hit rate, pinned memory, and routing overhead. It also owns up to the complexity and provides a test method for users to find their own results.

Scientific Rigor, Source Quality, Title Accuracy

The video cites numerous authoritative sources, including GitHub PRs, issues, release notes, model cards, and Reddit threads with benchmark data. The sources are directly relevant and verifiable. The title is slightly sensational but the content clarifies the scope, and the video explicitly discusses the discrepancy between the headline and the fair comparison. The presentation is rigorous, with careful attention to detail and caveats.

152 words

Title / Content Match

The title is attention-grabbing but accurately represents the core claim, though the video explains the 4× comes from a specific baseline, providing a balanced view.

Quality & Reliability

8/10

The video is well-structured, uses numerous primary sources (GitHub PRs, issues, Reddit threads), and provides clear reasoning with benchmark numbers. It acknowledges limitations and nuances, making it reliable overall.

Chapters

Cited Sources

Concurring Sources

Contribution & Novelties

The video provides a clear, evidence-based analysis of a recent llama.cpp feature, clarifying the conditions under which the expert cache provides real benefits. It bridges the gap between marketing headlines and practical user experiences, offering a reproducible testing method. The inclusion of PCIe bandwidth as a critical factor and the break-even hit-rate threshold are particularly valuable for the community.

Pour aller plus loin :

105 words

Radar Profile

The radar shows high scores in information quantity and technical level, with moderate reliability reflecting the dependence on user-reported benchmarks. The video is well-balanced but may require background knowledge to fully appreciate, positioning it as a deep-dive for enthusiasts.

Reliability 8/10

💬 Positive — across the 30 comments analyzed, viewers express appreciation for the clear explanations and the inclusion of Strata comparisons, though a few note minor issues like chapter timestamp mismatches and suggest more in-depth comparisons.