speculative decoding
a way to make an AI model write faster
Speculative decoding speeds up how fast a model writes. A small, quick draft model guesses the next few words, and the main model checks them all at once. Correct guesses are kept, wrong ones replaced. The output is the same as without it, just produced faster.
Think of it as
A typist's autocomplete suggesting the next few words, which the typist accepts or corrects in one glance.