Beyond the Waiting Game

How AI is Learning to Work Around the Memory Wall

The sequel exploring what comes after inference economics — architectures and techniques that reshape how models think.

Contents

• Introduction: The Kitchen is Still Waiting
• 1Speculative Decoding and the Art of the Good Guess
• 2The Draft Model: The Junior Chef
• 3MEDUSA: Train Extra Heads Instead of a Second Model
• 4EAGLE: From Token Guessing to Feature Guessing
• 5EAGLE-3: Six Tokens at Once, Losslessly
• 6Multi-Token Prediction: What If the Main Model Did the Drafting?
• 7The Common Thread
• 8The Limits of Drafting
• 9Discrete Diffusion and Parallel Generation
• 10The Sculptor vs. The Potter
• 11From Images to Language
• 12TIDAR: Thinking in Diffusion, Talking in Autoregression
• 13Zyphra ZAYA1: The First MoE Diffusion Model
• 14The Meaning of Diffusion
• 15Compressing the KV Cache
• 16GQA and MLA: Fewer Storage Shelves, Same Information
• 17CCA: Compressed Convolutional Attention
• 18KV Cache Quantization: Less Precision, More Tokens
• 19Prefix Caching and Shared Memories
• 20The Combined Effect
• 21Optimizing the Inbound Trip
• 22Prompt Compression: The TL;DR Approach
• 23Speculative Prefill: Guessing the Input
• 24PFlash and DFlash: A Real System, Real Numbers
• 25The Prefill Future
• 26Every Trick, All at Once
• 27The Stacking Principle
• 28But Don't They Interfere?
• 29The Real World
• 30The End of the Waiting Game?