Latency–throughput trade-off
Bigger batches raise throughput but can raise per-user latency.
Bigger batches raise throughput but can raise per-user latency. The central tension of serving; the goal is max throughput subject to a latency SLO.
Bigger batches raise throughput but can raise per-user latency.
Bigger batches raise throughput but can raise per-user latency. The central tension of serving; the goal is max throughput subject to a latency SLO.