Quantized Communication for Distributed Training

Quantized communication sends selected tensors in a smaller wire format while ranks communicate during distributed training. This note assumes GPU Collective Communication and owns one optimization question: what changes when the bottleneck is bytes on the wire?

The model does not have to store all weights or run all local arithmetic in the same precision used for communication. A job can train with FP32 or BF16 compute while sending selected lookup vectors or gradients in FP8 or another lower-precision format.

Where Quantization Sits

Communication quantization wraps the payload around a collective:

sender rank:
  tensor in training precision
  -> quantize for wire
  -> collective sends fewer bytes
 
receiver rank:
  receive smaller payload
  -> dequantize or consume lower-precision tensor
  -> continue training step

For embedding-heavy recommenders, the natural target is the lookup exchange from Distributed Training for Embedding-Heavy Recommenders. The number of ids does not change. The number of values per vector does not change. The bytes per value change.

Numeric Example

Suppose one distributed step returns 1,000,000 embedding vectors and each vector has 128 coordinates.

FP32 wire payload:
  1,000,000 lookups x 128 values x 4 bytes = 512 MB
 
FP8 wire payload:
  1,000,000 lookups x 128 values x 1 byte = 128 MB

The useful mental model is:

$$\text{bytes moved}\approx \text{lookups}\times \text{embedding dimension}\times \text{bytes per value}$$

Quantized communication changes the last term. It does not fix too many lookups, too-wide vectors, bad sharding, or a topology that sends all-to-all traffic across slow links.

Pinterest QComms Example

In the Pinterest article, FBGEMM’s QComms compressed embedding communication payloads from FP32 to FP8 before NCCL collectives. FBGEMM provides optimized kernels and quantization support. TorchRec routes distributed embedding lookups. NCCL moves tensors between GPU ranks. QComms sits between the TorchRec lookup path and the NCCL payload.

The article reports the largest NCCL send/recv operation shrinking by more than 75%. Throughput scaling improved, but full training jobs still had to converge equivalently before the optimization was useful.

What Remains Slow

Quantizing the payload reduces bytes. It does not remove every bottleneck:

Remaining bottleneck	Why quantization does not remove it
Load imbalance	one row owner can still receive more hot lookups than peers
Topology mismatch	a smaller all-to-all can still cross slow inter-node links
Input stalls	ranks still wait if the data path cannot feed the GPUs
Quality sensitivity	lower-precision communication can hurt convergence if error is too high

The Pinterest optimization sequence shows the layering. QComms reduced payload volume enough to expose sharding imbalance and topology problems that needed separate fixes.

Sources

• Pinterest Engineering - Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models
• FBGEMM GPU documentation

See also

• Quantization Strategies
• GPU Collective Communication
• Distributed Training for Embedding-Heavy Recommenders
• Training Scalability for Distributed ML