Multiple Embedding Tables in Recommender Rankers

A recommender ranker usually stores a set of embedding tables. Each table belongs to a feature family: users, candidate items, boards, topics, queries, actions, creators, or other categorical ids. One candidate row references a few rows from several of those tables, and the ranker uses the returned vectors plus dense numeric features to build the tensor consumed by its interaction layers.

The distributed-training bottleneck comes from that fan-out. A rank does not need every row from every table. It needs the rows touched by its local batch, and some of those rows may be owned by other ranks.

One Candidate Row Fans Out

A ranker row starts as structured product facts. The model cannot do dense math over raw ids such as user_42 or board_789; it first maps each id to an integer row address in the matching table.

Feature family	Value in one row	Table read	Returned object
User	user_42	user_embedding_table[42]	user vector
Candidate Pin	pin_77	pin_embedding_table[77]	candidate vector
Board context	board_789	board_embedding_table[789]	board vector
Topics	[topic_12, topic_48]	lookup_many(topic_embedding_table, [12, 48])	topic vectors, often pooled
Recent actions	[save, click, hide]	action_embedding_table[...]	action vectors, often pooled or sequenced
Numeric features	age, quality score, freshness	no embedding table	normalized numbers

The model then assembles those pieces into its local input:

candidate_row = {
    "user_id": 42,
    "pin_id": 77,
    "board_id": 789,
    "topic_ids": [12, 48],
    "recent_actions": ["save", "click", "hide"],
    "pin_age_days": 3.0,
    "quality_score": 0.82,
}
 
looked_up_vectors = {
    "user": user_embedding_table[42],
    "pin": pin_embedding_table[77],
    "board": board_embedding_table[789],
    "topics": pool(lookup_many(topic_embedding_table, [12, 48])),
    "actions": encode_sequence(lookup_many(action_embedding_table, ["save", "click", "hide"])),
}
 
dense_features = normalize([candidate_row["pin_age_days"], candidate_row["quality_score"]])
ranker_input = assemble(looked_up_vectors, dense_features)
scores = interaction_layers(ranker_input)

assemble might concatenate vectors, project them to a common dimension, add feature-family embeddings, pool repeated features, or feed sequences into attention. The exact architecture varies. The invariant is that the interaction layers cannot run until the required vectors for that candidate row have arrived.

Why There Are Multiple Tables

Separate tables keep unrelated id spaces separate. user_42, pin_42, and board_42 can all have numeric id 42, but they do not mean the same thing and should not share one learned row.

Feature families also have different shapes and operating constraints:

Reason	Example
Cardinality differs	there may be far more Pins than action types
Dimensions differ	Pin ids may deserve wider vectors than country ids
Update frequency differs	hot Pins and users appear in batches more often than rare boards
Fallbacks differ	a new Pin can use content features; a new action type may use a default row
Aggregation differs	topic ids may be pooled, while recent actions may keep sequence order

This is why a recommender can be embedding-heavy even when each example touches only a small number of rows. The full model stores rows for huge id spaces, while each forward pass reads a sparse subset.

Serving Versus Training

Serving and training both perform feature-family lookups, but they use different model state.

Path	Table state	What happens
Serving	published model version	read the rows needed for one request’s candidate rows
Training	mutable checkpoint plus optimizer state	read touched rows, compute loss, send gradients back to touched rows

Serving wants stable low-latency reads. Training wants mutable rows and optimizer bookkeeping. A row that changes during training does not affect serving until a trained checkpoint is exported and rolled out.

Why Distributed Training Needs Row Movement

In distributed training, one rank can own the local batch examples while other ranks own some of the embedding rows referenced by those examples.

rank 0 local batch:
  row A: user_42, pin_77, board_789
  row B: user_99, pin_12, board_555
 
row ownership:
  user rows live on rank 0
  pin rows live on rank 5
  board rows live on rank 12

Rank 0 can read the user rows locally, but it cannot build the full ranker input until rank 5 returns the Pin vectors and rank 12 returns the board vectors:

rank 0 -> rank 5:  lookup pin_77, pin_12
rank 0 -> rank 12: lookup board_789, board_555
 
rank 5  -> rank 0: vector(pin_77), vector(pin_12)
rank 12 -> rank 0: vector(board_789), vector(board_555)

The forward pass waits for those returned vectors. The backward pass sends sparse gradients for pin_77, pin_12, board_789, and board_555 back to the ranks that own those rows. That is the lookup exchange explained in Distributed Training for Embedding-Heavy Recommenders.

Batch Path

The batch path is:

one candidate row
  -> many feature-family lookups
  -> one assembled ranker input tensor
  -> local interaction layers
  -> per-candidate scores

Distributed training keeps the same model math but changes where the rows live. The bottleneck appears when the rank with the examples must wait for vectors from remote row owners before it can do its local compute.

Sources

• Pinterest Engineering - Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models
• PyTorch - Embedding
• TorchRec documentation

See also

• Recommendation Ranking Models
• Categorical Feature Embeddings in Recommender Systems
• Embedding-Heavy Models
• Distributed Training for Embedding-Heavy Recommenders