Decode
The second phase: generating output tokens one at a time, sequentially.
The second phase: generating output tokens one at a time, sequentially. Memory-bound; sets TPOT; the KV cache grows here.
The second phase: generating output tokens one at a time, sequentially.
The second phase: generating output tokens one at a time, sequentially. Memory-bound; sets TPOT; the KV cache grows here.