Pipeline parallelism (PP)

Appears in 1 tutorial

Splitting the model by layers across GPUs/nodes, assembly-line style; less communication, so it can cross nodes.

As used in LLM Infrastructure →

Splitting the model by layers across GPUs/nodes, assembly-line style; less communication, so it can cross nodes. Risk: pipeline "bubbles." (vLLM: --pipeline-parallel-size.)