GPU collective communication moves tensors among a group of GPU ranks. A collective is a coordinated operation where participating ranks exchange or combine tensors as one logical step. This note assumes Distributed Training for Embedding-Heavy Recommenders for the training-step story and owns the communication vocabulary: all-to-all, all-reduce, send/recv, NCCL, and profiler interpretation.
NCCL, the NVIDIA Collective Communications Library, is the common NVIDIA backend for high-throughput GPU-to-GPU collectives used by frameworks such as PyTorch distributed and TorchRec. NCCL is not a model architecture. It is the communication layer underneath the training framework.
Two Collective Shapes In One Step
This diagram shows the two collective shapes that matter for the recommender training step: all-to-all for remote embedding lookups, and all-reduce for synchronizing replicated dense weights.
All-to-all sends different payloads from every rank to every other rank. In embedding lookup, rank 0 may need board rows owned by rank 2, while rank 2 may need Pin rows owned by rank 1. The payload is irregular because each batch touches different ids.
all-to-all lookup requests:
rank 0 -> rank 1: pin_77, pin_12
rank 0 -> rank 2: board_789, board_555
rank 1 -> rank 0: user_51
rank 2 -> rank 1: pin_88All-reduce combines tensors from all ranks and returns the combined result to all ranks. In data-parallel training, replicated dense layers often use all-reduce so every replica applies the same gradient update.
dense gradient before all-reduce:
rank 0: dW0
rank 1: dW1
rank 2: dW2
after all-reduce:
rank 0: dW0 + dW1 + dW2
rank 1: dW0 + dW1 + dW2
rank 2: dW0 + dW1 + dW2Send/recv operations are point-to-point tensor transfers. A profiler may show ncclKernel_SendRecv even when the framework-level operation is part of a larger all-to-all. The kernel name tells you NCCL moved tensors; the surrounding model trace tells you whether the cause was embedding lookup, gradient synchronization, checkpointing, or another stage.
Physical Route Matters
The same logical collective has different cost depending on where the ranks live:
| Route | Cost shape |
|---|---|
| GPU to GPU inside one node | fastest path when GPUs have local high-bandwidth links |
| GPU to GPU across nodes | slower path through network devices and distributed communication libraries |
| CPU-mediated transfer | usually worst for training-critical tensor exchange |
Training code talks in ranks. Operators pay for physical routes. An all-to-all among eight local GPUs is different from an all-to-all among eight GPUs spread across multiple machines.
Why Utilization Can Mislead
GPU utilization can be high while model math stalls. NCCL collectives launch GPU kernels, so monitoring can report the GPU as busy. If those kernels spend the step moving tensors or waiting on peers, the streaming multiprocessors are not doing the dense math the model needs.
In the Pinterest article, profiler traces showed large NCCL send/recv spans during distributed embedding lookup. The next useful optimization was not a faster dense layer. It was reducing bytes, balancing row ownership, or changing which ranks exchanged lookup vectors.
Sources
- NVIDIA NCCL User Guide - Collective Operations
- PyTorch distributed communication package
- Pinterest Engineering - Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models