The Concurrency Trap

Here's the limitation that matters most for practical deployment: NLD's advantage is concentrated at low concurrency, which is where most production systems are not.

The big numbers — 3.3x, 4x, 850 tok/s — come from single-user benchmarks. At 1-8 concurrent streams, self-speculation is transformative. At 64+ streams, the advantage disappears, and NVIDIA just switches to AR mode.

This is fine if you're building a personal AI assistant, an interactive agent, or a single-user development tool. It's much less relevant if you're running a high-traffic API endpoint serving thousands of requests per second.

The concurrency inflection point varies by hardware, but the pattern is consistent: the benefit is largest where you need it least (low utilization) and smallest where you need it most (high utilization). This is a structural feature of any technique that trades compute for memory bandwidth. When the GPU is already busy, extra compute per request hurts.

This doesn't make NLD less valuable. It just means the value is concentrated in a specific deployment pattern — the one that's growing fastest (personal AI, on-device inference, agentic systems) but still represents a minority of total inference compute today.