Is This the End of NVIDIA's Memory Chip (NCM)

Is This the End of NVIDIA's Memory Chip (NCM)

Title history
  • Is This the End of NVIDIA's Memory Chip (NCM) — observed on 20261109-1536 UTC (YouTube)
🎙 Kai 👥 71K 📅 November 9, 2026 ⏱ 24 min 👁 118 📄 expert opinion 🧭 2026-11-09
Available in: English (current) Français

Keywords

NCMHBMKV cachememory bandwidthAI inference

Summary

The video analyzes the Next Concept Modeling (NCM) paper from Stanford and Together AI, which claims to generate 1,024 tokens in 32 passes instead of 1,024, potentially reducing the need for NVIDIA’s HBM memory. The creator, Kai, reads the actual paper and explains that while NCM is a clever architecture that writes in parallel levels, it primarily reduces bandwidth requirements, not memory capacity. The video breaks down the two roles of memory in AI systems: feeding (bandwidth) and holding (capacity), showing that in modern servers, capacity dominates (e.g., ~260 GB of 288 GB on a Rubin GPU holds KV cache for conversations). Data from OpenRouter shows agents use ~90% cached tokens and only ~1% output, so NCM optimizes the 1% while the 99% memory footprint remains. The paper’s speed claims are compared against diffusion models, not standard LLMs, and independent tests show NCM preserves only 8% of JSON structures. The video notes that NCM also requires a KV cache, and lacks training code. The main benefit is local inference on single GPUs (like RTX 5090) where bandwidth is the bottleneck, but capacity limits remain. The video concludes NCM is not the end of NVIDIA’s memory chips, as memory capacity and market dynamics (SK hynix sold out through 2029, DRAM prices quadrupling) are driven by AI demand. The creator also touches on NVIDIA’s Rubin CPX and NVHBM, reinforcing that memory demands persist.

231 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video provides substantial value by demystifying the NCM paper and countering hyperbolic headlines with clear technical reasoning. The argumentation is solid: it uses concrete numbers (e.g., H100 bandwidth, KV cache sizes, OpenRouter statistics) and logical distinctions (bandwidth vs. capacity, batching, agent usage) to demonstrate that NCM’s impact is limited. The creator fairly credits NCM’s genuine cleverness (Sudoku performance) while critiquing the comparisons and missing training code. The reasoning is structured, addressing the paper’s claims step-by-step, and concludes with a nuanced view that NCM could benefit local inference but not large-scale memory demands.

Scientific Rigor, Source Quality, Title Accuracy

The video demonstrates strong scientific rigor by referencing the original arXiv paper, GitHub issues and code, NVIDIA specifications, TrendForce market reports, and OpenRouter data. It distinguishes between the paper’s actual comparisons and the headlines, and highlights independent community experiments. The title ‘Is This the End of NVIDIA’s Memory Chip (NCM)’ is appropriately provocative yet accurately reflects the content, as the video directly addresses that question. The sources are reputable and current, and the video correctly interprets the data without overclaiming. Overall, the rigor is high, and the title-content fit is good.

199 words

Title / Content Match

The title asks a provocative question and the video answers it directly with a well-reasoned 'no', so it matches the content well.

Quality & Reliability

8/10

The video reads the original paper thoroughly, cross-references multiple primary sources (arXiv, GitHub, TrendForce, OpenRouter), distinguishes bandwidth from capacity with concrete numbers, and presents a balanced critical analysis of NCM's claims.

Chapters

Cited Sources

Concurring Sources

Contribution & Novelties

The video’s contribution lies in clarifying the bandwidth vs. capacity distinction for memory in AI systems and using real data (OpenRouter, TrendForce) to assess NCM’s actual impact. It debunks exaggerated claims by grounding the discussion in technical and market realities. The analysis highlights that while NCM could reduce bandwidth in local inference, it does not address the dominating capacity needs from KV caches, especially in multi-user and agent scenarios.

Pour aller plus loin :

144 words

Radar Profile

The radar profile shows high scores across all dimensions, with particularly strong performance in quantity and quality of information, reflecting the video's comprehensive and well-sourced analysis. The technical level is slightly lower, indicating that while the content is deep, it remains accessible to a well-informed audience. The overall reliability is high, supported by robust citations and careful reasoning. This profile is typical of a high-quality educational technology analysis.

Reliability 9/10