KV offloading / swapping
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.