Categorical feature embeddings are learned lookup-table rows for ids inside a ranking model. They are one form of trainable parameter state in the Machine Learning Model Shapes map. This note follows one candidate row from serving-time ranking lookup to later training update.
The running row is user_42 viewing pin_77 in the context of board_789. The ranker cannot feed the string board_789 into a dense layer. It turns that id into an integer table address, reads the board row, and uses the returned vector as one input to the model.
One Id Through Serving And Training
This diagram follows one board id through the lifecycle. Serving reads a published model version. Training updates a mutable checkpoint. A later export is what makes changed rows visible to serving.
Serving Lookup
The ranking model is a collection of trainable arrays. Some arrays are dense-layer matrices, biases, attention weights, projection weights, or prediction-head weights. An embedding table is another trainable array: a large two-dimensional table whose rows are addressed by ids.
For a board feature, the table shape is num_board_ids x embedding_dimension. A table with 200 million board ids and 64 learned numbers per row stores 12.8 billion numbers, before the extra values that the optimizer tracks while training.
One serving lookup is just an indexed read:
candidate row contains:
user_id: user_42
candidate_id: pin_77
context_board_id: board_789
lookup:
board_index = id_mapping["board_789"]
board_vector = board_embedding_table[board_index]The board vector is then joined with the rest of the ranker input: user vector, candidate Pin vector, recent-action vectors, query or surface context, and numeric features such as candidate age or quality scores. Multiple Embedding Tables in Recommender Rankers follows that full candidate-row fan-out. The interaction layers consume the assembled tensor and might emit p(save)=0.18 and p(hide)=0.02 for pin_77.
The board vector is not read from the board product record. The board record may store title, owner, privacy state, Pins, and timestamps. The embedding row is learned by the model from historical examples involving that board.
Training Update
Embedding rows are learned by the same loss-and-gradient loop as other model weights. At initialization, board_embedding_table[board_index] starts from random or otherwise initialized numbers. It becomes useful because examples containing board_789 create prediction errors, and the optimizer changes the row to reduce future errors.
A tiny training example looks like this:
features:
user_id: user_42
candidate_id: pin_77
context_board_id: board_789
model_score_at_serving: p(save)=0.18
label observed later:
save = 1During the forward pass, training gathers the same rows the serving model would gather. During the backward pass, the loss sends gradients to the dense weights and to the embedding rows touched by this example: user row 42, Pin row 77, board row 789, and any action or topic rows in the candidate row.
Only rows present in the batch receive embedding-row gradients:
batch ids:
board_789
board_555
board_789
rows updated this step:
board_789
board_555
rows not updated this step:
board_812
board_901Popular ids can move often because they appear in many minibatches. New or rare ids move slowly because the model has little label evidence for them. Serving still reads a coherent published model version until a trained checkpoint is exported and rolled out.
One-Hot Is The Math View
One-hot encoding is a mathematical way to describe the lookup. If there are 1,000 possible board ids, board_789 can be represented as a length-1,000 vector whose only nonzero cell is the position assigned to board_789. Multiplying that mostly-zero vector by the board embedding table selects the same row as board_embedding_table[board_index].
Implementations use gather instead of materializing the one-hot vector. Creating millions of zeros just to select one row would waste memory bandwidth.
Why This Is Called Sparse
The word sparse describes how the feature addresses the model, not the contents of the returned vector. One candidate row touches a tiny subset of a large id space. The vector returned from the table is usually dense: most coordinates can be nonzero.
| Input feature | Operation | Why it is or is not sparse |
|---|---|---|
board_789 | gather one row from the board table | one id selects one row from a huge table |
recent actions [save, click, hide] | gather several action rows, then pool or sequence them | a short list selects a few rows |
candidate age 3 days | normalize or bucketize a measured value | the value enters as a measurement, not as a table address |
This distinction matters because sparse id features create huge tables and sparse updates. Numeric features create different problems: normalization, calibration, missing values, and drift.
Relation To Transformer Token Embeddings
Token embeddings in a Transformer use the same learned lookup-table pattern. Tokenization maps text to token ids, the model gathers token rows, and training sends gradients back into those rows plus the attention, MLP, layer-norm, and output-head weights.
Recommender systems can also use separate embedding models. A text encoder, image encoder, or multimodal model might embed Pin content; a retrieval system might store item vectors in a vector-search index. The categorical id table inside the ranker answers a narrower question: given this exact id in this candidate row, what learned vector should the ranking model use?
Edge Cases
New ids need a fallback because their exact row has little or no label history. Common fallbacks include content features, parent entities, topic ids, default rows, hashed buckets, or a separate content encoder.
Hot ids can dominate training traffic because they appear in many batches. That becomes a distributed-training problem when embedding tables are sharded across GPUs.
Overmemorized ids can explain historical accidents that content, policy, or freshness should dominate. Ranking systems use content features, regularization, sampling rules, and evaluation checks so the model does not rely only on id memory.
Sources
- Pinterest Engineering - Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models
- PyTorch - Embedding