Efficient Memory Management for LLM serving

This meetup talk, led by Neha from West Coast Machine Learning, provides a thorough walkthrough of the paper ‘Efficient Memory Management for LLM Serving’ (PagedAttention). The discussion begins with the importance of batching for high-throughput LLM serving and the unique memory challenges posed by the KV cache, which grows dynamically per request. The presenter explains how traditional inference systems suffer from internal and external memory fragmentation due to static allocation. The core solution, PagedAttention, draws inspiration from virtual memory paging in operating systems, dividing the KV cache into fixed-size blocks and mapping logical to physical memory non-contiguously. This reduces fragmentation and enables memory sharing across requests. The talk covers the memory profile of LLMs, the distinction between prefill and decode phases, and the impact of KV cache size on GPU capacity. The presenter also discusses the vLLM serving system and its improvements, including kernel optimizations. The session includes interactive Q&A with participants, clarifying concepts like internal vs. external fragmentation and the analogy to demand paging. The talk concludes with a comparison of vLLM’s throughput versus existing systems, highlighting significant gains in requests served per second.