Ads indexing and retrieval is the serving-time lookup layer that turns advertiser configuration into candidate ads for one request. The advertiser-facing model says “RunCo targets runners in New York with this shoe creative.” The retrieval layer needs request-shaped records: keys, posting lists, overlays, creative references, and compact fields that can answer “which ads could plausibly serve here?” under tight latency.
Retrieval sits between Advertiser Control Plane and Ad Serving Request Lifecycle. The control plane owns rich domain objects. The request path owns one ad opportunity. Indexing is the translation layer that makes the request path fast enough to run for every opportunity.
The Flow
The diagram separates the offline/streaming indexing path from request-time retrieval. Ranking and auction only see the candidates that survive retrieval and cheap state overlays.
RunCo creates a campaign for running shoes, with one ad group targeting runners in California and one image creative allowed in feed placements. Before any user request arrives, the platform derives serving-facing records from that advertiser configuration.
- Advertiser objects change. Campaigns, ad groups, ads, creatives, product groups, targeting specs, budgets, bids, review state, and measurement config are created or updated.
- Indexing validates and expands them. The indexing path normalizes fields, expands targeting into lookup keys, attaches creative/render constraints, joins policy state, and publishes serving records.
- Indexes answer stable lookup problems. Targeting keys, keyword keys, product category keys, audience segment ids, creative format keys, and vector embeddings can all live in different physical index shapes.
- Fresh overlays handle fast-changing state. Pauses, budget exhaustion, policy blocks, product stock, review outcomes, and emergency blocks often change faster than full index rebuilds. Serving reads these overlays during retrieval or eligibility.
- Request-time retrieval returns a bounded candidate set. The request has a surface, placement, geography, privacy state, content context, and allowed user identifiers. Retrieval uses those facts to fetch candidate IDs and lightweight records.
- Ranking and auction run later. A candidate that never appears in retrieval cannot win, no matter how high its eventual score would have been.
Index Shapes
There is no single “ad index” data structure. Ads systems use several shapes because exact targeting, text matching, product catalogs, mutable state, and embedding search have different access patterns.
| Shape | What it stores | Example lookup |
|---|---|---|
| Inverted index / posting lists | A key maps to many candidate ids. | keyword:running shoes → ad groups matching that keyword. |
| Bitmap or compressed set index | Candidate sets that can be intersected quickly. | geo:US-CA AND surface:feed AND format:image. |
| Key-value serving record | Fields for one campaign, ad group, creative, or product. | ad_group:456 → bid controls, creative ids, serving version. |
| Forward index | Precomputed fields stored with a candidate for fast hydration. | Candidate id → targeting flags, creative constraints, quality priors. |
| Fresh state overlay | Small mutable blocks layered over slower indexes. | campaign:123 → paused or budget exhausted. |
| Vector index | Embeddings for approximate nearest-neighbor retrieval. | User/context embedding → related products, ads, or creatives. |
The physical choice is implementation-dependent. The concept is stable: indexing precomputes request-shaped lookup records from advertiser-shaped objects.
Retrieval Is An Economic Gate
Retrieval controls which advertisers can enter the later auction. That makes it a revenue and fairness mechanism, not a neutral database lookup.
Missed retrieval means a campaign had no chance to compete. The cause can be stale indexing, missing targeting expansion, privacy-state key removal, an out-of-date product feed, a broken candidate source, or an overly narrow retrieval rule.
Overbroad retrieval means too many candidates reach ranking. The platform pays for more feature hydration and model inference, and may hit latency limits. If the request times out, some demand loses even though the system found it.
Candidate trimming is the resource-control step between retrieval and ranking. It applies caps, per-source quotas, per-advertiser limits, cheap pre-scores, or exploration rules so the expensive ranker sees a bounded set.
| Retrieval choice | System effect |
|---|---|
| Add a new candidate source | More recall, more auction pressure, more feature/inference cost. |
| Lower candidate cap | Lower latency and cost, but more risk of dropping high-value demand before ranking. |
| Use a fresh budget overlay | Fewer wasted candidates from exhausted campaigns, but more online state reads. |
| Use contextual keys when user identifiers are unavailable | Preserves some demand under privacy limits, but loses audience/retargeting precision. |
| Add vector retrieval | Better broad-match and exploration, but harder debugging and more recall-quality evaluation. |
What To Debug First
When an advertiser asks why an ad did not serve, retrieval questions come after control-plane validity and before ranking.
| Question | What it distinguishes |
|---|---|
| Is the campaign/ad group/creative active and approved? | Control-plane or policy problem. |
| Did the latest config reach the serving index? | Indexing propagation problem. |
| Which retrieval key should have matched this request? | Targeting expansion or request-context problem. |
| Did the candidate source return the ad? | Retrieval-source problem. |
| Did a fresh overlay remove it? | Budget, policy, stock, pause, or emergency block. |
| Did trimming remove it before ranking? | Candidate-control problem, not auction loss. |
Do not start with auction theory if the candidate never reached ranking.