Why Not 70B?

NLD exists at 3B, 8B, and 14B. Not 70B, not 400B, not a 671B MoE. This isn't an accident, and it's not because NVIDIA chose not to.

Training a joint AR + diffusion model at 70B+ is genuinely harder than at 8B for several reasons:

Training stability. The joint loss requires careful balancing — the diffusion loss weight (0.3) and the AR loss weight (1.0) are tuned for the specific model scale and data distribution. At 70B+, the optimization landscape changes. Gradient variance increases. The interaction between the two loss functions becomes harder to predict. The paper's techniques — global loss averaging, DP-rank varying masking — are designed to mitigate this, but they were validated at 8B. Whether they generalize is an open question.

The 1.3T token training cost. 1T tokens of AR + 300B tokens of joint continued pretraining. At 8B parameters, this is expensive but feasible — a few hundred thousand dollars of compute, depending on hardware. At 70B, the same training run costs 10x more (more parameters per token) and requires more tokens to converge (larger models need more data to saturate). Rough estimate: $3-5 million for the 70B version. For a 400B or MoE model, the cost is in the tens of millions. This is a bet, not an experiment.

The attention mask complexity. The clean-stream design — where clean tokens attend strictly causally while noisy tokens attend bidirectionally within their block — requires careful implementation. At 70B with Flash Attention and tensor parallelism, this interaction gets significantly more complex. The paper mentions "Megatron Bridge" for training infrastructure, which suggests this wasn't trivial even at 8B.

The data quality requirement. The 1.3T tokens aren't random web text. They're high-quality continued pretraining data, curated for the joint objective. Scaling this curation pipeline to 70B requires proportionally more data, and the data quality requirements may differ at larger scales.

The honest answer: nobody knows if this works at 70B. The team probably attempted a small-scale test and committed to publishing at 8B before scaling further. If the 8B recipe generalizes, a 70B NLD model is likely in training right now. If it doesn't, the architecture may have a fundamental scale ceiling.