2026-06-30-recommender-systems-feedback

Recommender Systems Feedback Checklist

Scope: 2 - Areas/Tech/AI and ML/Recommender Systems/.

Use this as the rewrite queue for MR 88. Each unchecked item is a concrete note-quality problem found during review. Line references point to the file state at the time this feedback was written.

Cluster-Level

• Establish one shared lifecycle spine across the cluster: serving request → retrieval returns candidate ids → ranking scores candidate rows → feed composition/allocation chooses shown items → event logs become training examples → distributed training updates model state.
• Keep serving-time ranking, retrieval/vector search, and training-time distributed lookup exchange separate until each has been defined with its owner, input, output, and state mutation.
• Add or revise diagrams so they carry lifecycle structure instead of showing isolated boxes; use serving diagrams for request-time ranking and swimlane-style training diagrams for rank/GPU communication.

Recommendation Ranking Models.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:1 Rewrite the opening so it defines recommendation ranking positively before contrasting it with retrieval; introduce inventory/catalog/candidate set only after the product object is clear.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:3 State the unit of ranker invocation directly: one request produces many candidate items, the ranker builds one feature row per (request, candidate) pair, and inference usually scores those rows as a batch.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:7 Resolve the granularity conflict between “one candidate’s path” and candidate-set ranking; make the diagram and prose follow either one candidate row inside a batch or the whole request lifecycle.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:21-34 Move or reframe the retrieval boundary section so serving-time approximate nearest-neighbor search is introduced as upstream retrieval, not as a sudden correction to “whole catalog” search.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:25-34 Replace ads-shaped vocabulary such as no-fill and diagram text about auction rules with organic feed-composition terms, or explicitly switch the example to ads before using auction language.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Recommendation Ranking Models.md:48-58 Define a logged opportunity and distinguish shown impressions, retrieved-but-not-shown candidates, sampled negatives, and later labels before using them as training-row inputs.

Categorical Feature Embeddings in Recommender Systems.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Categorical Feature Embeddings in Recommender Systems.md:1-11 Separate the serving lookup path from the training update path; the opening and diagram currently move from ranker input to training row updates without naming which system is reading versus mutating the table.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Categorical Feature Embeddings in Recommender Systems.md:25-35 Turn the feature table into one end-to-end ranking example with an output score, not just possible learned signals, so the reader sees how user_42, pin_77, and board_789 flow into one prediction.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Categorical Feature Embeddings in Recommender Systems.md:55-61 Label failure modes by lifecycle stage: cold-start learning, distributed-training communication, and model-quality memorization are currently adjacent but not clearly scoped.

Embedding-Heavy Models.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Embedding-Heavy Models.md:1-3 Split the definition into model property versus training-system consequence; optimizer state, sparse gradients, and network traffic are training concerns, while serving has exported tables plus lookup latency and memory pressure.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Embedding-Heavy Models.md:5-26 Add a concrete recommender example before the parameter and consequence tables, then connect each consequence to that example’s lookup/update path instead of listing outcomes before mechanism.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Embedding-Heavy Models.md:46-58 Make the comparison three-way: trainable categorical table during training, exported categorical table during ranking serving, and vector-search index during retrieval. The current wording risks implying categorical embeddings are only training-time.

Distributed Training for Embedding-Heavy Recommenders.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Distributed Training for Embedding-Heavy Recommenders.md:5-20 Rewrite the base step as a staged lifecycle: forward lookup request all-to-all, vector return, local dense compute, backward sparse-gradient return to owning shards, dense-gradient synchronization, optimizer update.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Distributed Training for Embedding-Heavy Recommenders.md:13-18 Replace vague “synchronized dense state” with the exact synchronization operation and owner: which ranks hold dense replicas, which gradients are reduced, and where optimizer state is updated.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Distributed Training for Embedding-Heavy Recommenders.md:40-51 Define SM efficiency on first use and tie each profiling signal to the lifecycle stage it implicates; otherwise the table reports symptoms without teaching the diagnosis path.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Distributed Training for Embedding-Heavy Recommenders.md:55-64 Define EFA, FP8 QComms, and 2D parallelism at first mention or move them into a Pinterest-specific lever table with one sentence explaining the variable each changes.

GPU Collective Communication.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/GPU Collective Communication.md:1-3 Define rank and collective before using them as primitives; the note starts with both terms but only explains NCCL.
• 2 - Areas/Tech/AI and ML/Recommender Systems/GPU Collective Communication.md:5-15 Add one concrete training-step spine showing where all-to-all and all-reduce occur, so the diagram is grounded in forward embedding lookup versus backward gradient synchronization.
• 2 - Areas/Tech/AI and ML/Recommender Systems/GPU Collective Communication.md:15 Rewrite “product-level cause” into a concrete distinction between profiler kernel name, framework collective, and model stage that triggered the communication.
• 2 - Areas/Tech/AI and ML/Recommender Systems/GPU Collective Communication.md:35-39 Define or briefly inline TorchRec DistributedModelParallel, ncclKernel_SendRecv, 2D-parallel design, and replicated groups; this section currently depends on terms introduced only by linked notes or source context.

Pinterest Foundation Model Training Scalability.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:1-18 Rewrite the opening around one concrete training batch before the link cluster; define FRM, rank, sharded embedding rows, lookup exchange, and NCCL all-to-all in place.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:20-26 Fix the diagram setup: either add the pre-EFA “~5x slower” baseline to the diagram or state that the diagram starts after EFA.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:30-40 Replace or supplement the diagnosis table with a forward-pass trace showing ids, owning ranks, exchanged tensors, and why NCCL spans grow.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:63-65 Define “hash partitions” and show how aligning partitions with GPUs changes row ownership and workload balance.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:71-79 Use a concrete before/after embedding shape to separate row-count capacity from vector width and explain why bytes per lookup fall.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Pinterest Foundation Model Training Scalability.md:103-112 Separate training-time teacher models from serving-time student models and add the measurement/deployment handoff before claiming product impact.

Quantized Communication for Distributed Training.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Quantized Communication for Distributed Training.md:1-16 Add a concrete sender/receiver lifecycle and define rank, collective, and wire format before the stage table.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Quantized Communication for Distributed Training.md:7-14 Split forward all-to-all embedding payloads from backward all-reduce gradients; the generic model-update row blurs training stages.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Quantized Communication for Distributed Training.md:20-28 Add a numeric worked example for lookups x embedding_dimension x bytes_per_value before/after FP32 to FP8.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Quantized Communication for Distributed Training.md:30-34 Define QComms, FBGEMM, TorchRec, and NCCL ownership: who provides each piece and where quantization sits in the communication path.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Quantized Communication for Distributed Training.md:38-47 Expand the boundaries table into mechanisms for load imbalance and topology mismatch; the table lists bottlenecks without showing what remains slow.

Training Scalability for Distributed ML.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/Training Scalability for Distributed ML.md:1-3 Define the measurement contract: same model, data path, batch semantics, and quality target; distinguish examples/sec from time-to-quality.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Training Scalability for Distributed ML.md:7-23 Add strong-versus-weak scaling before the speedup formula, then state which interpretation the Pinterest numbers use.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Training Scalability for Distributed ML.md:27-37 Replace the cost table with a training-step lifecycle, then map each cost to input, forward exchange, backward sync, optimizer, or checkpointing.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Training Scalability for Distributed ML.md:31-32 Define or inline-link replicas, updated parameters, model-parallel shards, and lookup exchange before relying on them in the mechanism table.
• 2 - Areas/Tech/AI and ML/Recommender Systems/Training Scalability for Distributed ML.md:47-57 Qualify near-linear throughput with convergence and quality; throughput scaling alone does not prove faster time-to-good-model if global batch or optimizer behavior changes.

2D Parallelism for Recommendation Model Training.md

• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:1-3 Rework the opening around a concrete 4-node or 8-GPU example and define group, replica, model-parallel sharding, and data-parallel replication.
• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:7-13 Add a caption after the diagram and make the visual show both local intra-node all-to-all and cross-node replica synchronization.
• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:13-21 Replace “serving every embedding lookup” and “request-shaped” with training-specific wording, or explicitly distinguish training-batch lookups from online serving requests.
• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:17-24 Expand the payload table into a two-stage training trace; “natural collective” and “more regular” label the difference but do not teach payload shape or timing.
• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:28-30 Show the memory tradeoff with numbers: model size per replica, GPUs per group, and number of replica groups.
• 2 - Areas/Tech/AI and ML/Recommender Systems/2D Parallelism for Recommendation Model Training.md:34-36 Clarify what changed between the all-reduce-optimized and all-to-all-optimized layouts; the current sequence can read as contradicting the earlier claim that 2D already keeps all-to-all local.