The Concurrency Table

The most important table in the paper isn't about throughput. It's about concurrency.

At low concurrency (1-64 concurrent streams), self-speculation dominates because the bottleneck is memory bandwidth. The GPU spends most of its time loading model weights; any technique that generates more tokens per load wins.

At high concurrency (>64 streams), the model becomes compute-bound. The GPU is processing enough requests simultaneously that its arithmetic units are fully utilized. In this regime, self-speculation's advantage shrinks — the extra compute per forward pass starts to hurt rather than help.

NVIDIA's solution is elegant: switch to AR mode. At high concurrency, the standard causal attention mask gives the best throughput because the GPU is already saturated. No separate deployment, no model reload — just a different attention pattern on the same weights.

This means the optimal serving strategy is now multi-modal by default:

| Concurrency | Best Mode | Why |

|-------------|-----------|-----|

| 1-8 (personal AI) | Diffusion or Self-Spec | Memory-bound, maximize tokens per load |

| 8-64 (team/API) | Self-Speculation | Still memory-bound, draft/verify wins |

| 64+ (datacenter) | Autoregressive | Compute-bound, AR has lowest overhead |

The inflection point varies by hardware (DGX Spark hits compute-bound at lower concurrency than GB200), but the principle holds across all GPUs.

This is the tangible takeaway for practitioners: your concurrency profile determines your ideal mode. One-size-fits-all AR serving is no longer optimal at any concurrency level. If you're building a personal AI assistant that serves one user at a time, you should be using diffusion or self-speculation mode. If you're serving a high-traffic API, AR mode is still fine — but you gain nothing from the diffusion capability at that scale.

---

Chapter 2: The Hard Parts

Now, the questions the press release doesn't answer.