Tensor parallelism (TP)

Appears in 1 tutorial

Splitting each layer's math across GPUs; makes a big model fit and adds bandwidth, but needs constant fast communication — keep within one NVLink node.

As used in LLM Infrastructure →

Splitting each layer's math across GPUs; makes a big model fit and adds bandwidth, but needs constant fast communication — keep within one NVLink node. (vLLM: --tensor-parallel-size.)

LLM Infrastructure →