The Real World

Theoretical stacking is one thing. The real question is: do these optimizations survive contact with production?

The evidence says yes. The major inference serving frameworks — vLLM, TensorRT-LLM, SGLang, TGI — all support multi-layer optimization stacks. They combine KV cache management, continuous batching, prefix caching, and speculative decoding in a single system. Each layer was built by different teams, for different use cases, and they work together because they were designed with clean interfaces.

The most advanced systems (and the most secretive, since this is where competitive advantage comes from) add custom layers: learned prompt compression, adaptive quantization based on token importance, dynamic draft tree selection based on prompt difficulty.

The result is that the same GPU that served 10 concurrent users at acceptable quality in 2023 can serve 1,000 concurrent users today — a 100x improvement in capacity, driven almost entirely by algorithmic workarounds for the memory wall.