LLM Inference: A Comparative Guide to Modern Open-Source Runtimes

LLM Inference: A Comparative Guide to Modern Open-Source Runtimes

by Aleksandr Shirokov

Applied Sciences & Engineering AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence
🎙 Aleksandr Shirokov 👥 5K 📅 October 20, 2025 ⏱ 51 min 👁 1K 📄 expert opinion 🧭 2026-08-15
Available in: English (current) Français

Keywords

LLM inferencevLLMTGITritonSGLang

Summary

In this technical talk from MLOps World | GenAI Summit 2025, Aleksandr Shirokov, Team Lead MLOps Engineer at Wildberries, presents a comparative analysis of modern open-source LLM inference runtimes. He begins by outlining key performance metrics for LLM serving, including latency, throughput, and GPU memory usage, and emphasizes the importance of custom benchmarking to evaluate frameworks in production. The talk covers four main frameworks: vLLM, TGI, Triton TensorRT-LLM, and SGLang, discussing their evolution, strengths, and weaknesses. Shirokov then dives into optimization techniques such as attention backends (FlashAttention, FlashInfer), speculative decoding, KV cache management, and weight loading strategies. He shares practical production insights from Wildberries, including GPU sharing with HAM, autoscaling with HPA, and using Kong as an API gateway for rate limiting and observability. The talk concludes with lessons from deploying DeepSeek R1-0528, highlighting the trade-offs between performance, cost, and complexity. The presentation is aimed at MLOps engineers and technical practitioners looking to deploy LLMs at scale.

157 words

Critical Evaluation

Value of the Information & Strength of the Argument

The talk provides valuable, practical insights from real-world production experience at Wildberries. The speaker systematically compares four major LLM runtimes, offering concrete advice on when to use each based on workload characteristics. The argumentation is solid, grounded in internal benchmarks and hands-on testing. However, the lack of external validation and the reliance on anecdotal evidence from a single company’s experience limit the generalizability. The speaker acknowledges trade-offs and avoids overpromising, which enhances credibility. The discussion of optimization techniques is informative, but some claims, such as the performance of FlashInfer, are presented without detailed evidence. Overall, the value lies in the practical, actionable guidance for practitioners.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates a strong understanding of the technical landscape, but the scientific rigor is moderate. The speaker references internal benchmarks and personal experience rather than citing external studies or papers. The sources cited are limited to the conference website (mlopsworld.com) and the frameworks’ GitHub repositories, which are not explicitly mentioned in the talk. The title accurately reflects the content, as the talk is indeed a comparative guide. The speaker’s expertise is evident, but the lack of external references and the focus on a single company’s experience may limit the generalizability of the findings. No comments were provided for analysis.

220 words

Title / Content Match

The title accurately reflects the content: a comparative guide to modern open-source LLM runtimes, focusing on vLLM, TGI, Triton, and SGLang.

Quality & Reliability

8/10

Talk by an experienced MLOps engineer from Wildberries, presenting practical insights from production deployments. The content is based on hands-on experience and internal benchmarks, but lacks peer-reviewed sources and some claims are anecdotal. The speaker provides specific technical details and acknowledges trade-offs, enhancing credibility.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Concurring Sources

  • vLLM GitHub Repository — The speaker mentions vLLM as one of the frameworks they use; the repository provides documentation and benchmarks.
  • TensorRT-LLM GitHub Repository — The speaker discusses Triton TensorRT-LLM; the repository is the official source for this framework.

Dissenting Sources

  • No discordant sources identified — The talk does not present conflicting viewpoints or sources.

Contribution & Novelties

This talk provides a practical, production-focused comparison of four major LLM inference runtimes, offering insights from Wildberries’ experience. It highlights the importance of custom benchmarking and shares specific techniques for GPU sharing, autoscaling, and API management. The talk’s novelty lies in its hands-on perspective, though it does not introduce new theoretical concepts.

Pour aller plus loin :

  • vLLM — Official repository for vLLM, a high-throughput LLM serving engine.
  • TensorRT-LLM — NVIDIA’s TensorRT-LLM for optimized inference.
  • Text Generation Inference — Hugging Face’s TGI framework.
  • SGLang — SGLang runtime for LLM serving.
  • FlashAttention — Efficient attention kernels.
  • PagedAttention — Paper on PagedAttention, the basis for vLLM’s memory management.

106 words

Radar Profile

The radar profile shows high scores in information quantity, quality, and technical level, indicating a dense and technically rich presentation. The lower reliability score reflects the reliance on anecdotal evidence and lack of external citations. Overall, the talk is highly informative for practitioners but may not meet strict academic standards.

Reliability 7/10