The Concurrency-Characterized Serving Model

The single most important concept NLD introduces isn't a technical innovation — it's a deployment pattern: switch modes by load.

Every inference serving system today uses one generation strategy (AR, always). Maybe it adds speculative decoding as an optimization, but the fundamental approach is constant. NLD formalizes the idea that the optimal strategy changes with load, and provides a clean mechanism (attention mask switching) to make that practical.

This shifts the design problem from "find the single best strategy" to "characterize your concurrency profile and choose accordingly." It's a small conceptual shift with large practical consequences:

• Personal AI applications (1 user, long sessions) → diffusion or self-speculation, always
• Team tools (10-50 users) → self-speculation during work hours, AR at peak
• API services (1000s of users) → AR most of the time, self-speculation during off-peak

The serving framework of the future will monitor queue depth and switch modes automatically, without operator intervention. The attention mask becomes a configurable parameter like batch size or precision.