The Decision Framework

Here are the tangible questions to ask for any inference deployment:

Question 1: What's your typical concurrency?

• 1-8 streams: Self-speculation or diffusion model is a must-evaluate
• 8-64 streams: Worth testing, may see 2-3x improvement
• 64+ streams: Diffusion adds little; focus on batching and AR optimization

Question 2: Can you tolerate training cost?

• NLD-style training costs significantly more than standard fine-tuning
• If you're building on a pre-trained model, you need the open-weight NLD (available now on Hugging Face)
• If you need to train your own, budget for 1.3T tokens of continued pretraining

Question 3: Is this for an application where latency matters?

• Personal AI, interactive agents, real-time chatbots: huge win
• Batch processing, offline generation, non-interactive: AR is fine, diffusion adds complexity

Question 4: Do you need the framework ecosystem?

• SGLang: support incoming (merge request pending)
• vLLM, TensorRT-LLM, TGI: not yet supported
• Custom inference stack: feasible but requires kernel work

Question 5: What's your hardware refresh cycle?

• Buying this quarter: B200 over H200 if diffusion is in your 12-month plan
• Buying next year: wait for diffusion-optimized hardware, possibly with dedicated attention mask switching