The Training Cost Nobody Quotes

The 1.3T token training figure sounds modest compared to the 15T tokens used to train Ministral3 from scratch. But this is continued pretraining, not full training. The cost structure is different:

• 1T tokens of AR continued pretraining: This is essentially a long fine-tuning run on a frozen base. The model already knows language; it's being tuned for the joint objective. But AR continued pretraining at this scale is non-trivial — it's more expensive than fine-tuning by a significant margin.

• 300B tokens of joint training: This is where the real innovation is, and it's also where the cost is hardest to amortize. The joint forward pass needs to compute both losses simultaneously, which requires storing two sets of gradients and managing two optimizers. The DP-rank varying masking technique means the effective batch size per GPU varies — complicating the standard data-parallel training pipeline.

The paper mentions that AR loss alone contributes 7.48% accuracy improvement to diffusion decoding. This means the 1T AR stage isn't optional — it's a critical enabler of the joint training that follows.

Bottom line: this training run cost NVIDIA significantly more than a standard SFT or continued pretraining run at the same parameter count. The joint training isn't an add-on; it's a fundamental change to the training pipeline.