Pinterest Cloud Capacity Operating Model

Pinterest’s cloud capacity process behaves more like datacenter planning than pure elastic cloud usage. The important distinction is that money commitments and launch guarantees are separate:

• Savings Plans and Reserved Instances are financial commitments: they reduce the hourly rate for matching usage, but they do not make instances launchable.
• On-Demand Capacity Reservations (ODCRs) are launch guarantees: they reserve a specific instance type in a specific AZ, but they do not create a discount by themselves.
• Instance Guarantees (IG / FOOB) are provider-coordinated holds for large or forecastable capacity increases. AWS expects Pinterest to consume delivered capacity quickly, because held capacity blocks other customers too.

The practical job is coordinating three flows:

1. Demand: teams forecast launches, migrations, seasonal peaks, failover needs, training jobs, and serving growth.
2. Supply: Capacity Engineering and provider account teams make sure the right instance families exist in the right AZs at the right time.
3. Platform admission: PinCompute, Training Compute Platform (TCP), quotas, ODCR matching, and autoscaling decide which workloads can actually consume that supply.

Related concepts

Read Cloud Capacity Reservations for the generic ODCR mechanism, Event-Driven Capacity Planning for launch/runbook mechanics, Kubernetes Capacity Management for quota and scheduling behavior, and GPU Capacity Planning / Instance Rightsizing for GPU benchmarking and CPU/memory right-typing.

Who does what

Actor	Owns	Provides	Can change directly
Service / product teams	Demand signal and workload behavior	Forecasts, launch dates, traffic estimates, model sizes, batch sizes, SLOs	Application autoscaling settings, workload requests, quota requests
Spend Captains	Budget and priority gate	Approval, ROI context, org priorities	Approve or reject capacity requests before provider ordering
Capacity Engineering (CapEng)	EC2 supply, ODCR/IG operations, scarcity response	Capacity request triage, AWS coordination, ODCR acceptance, audits	ODCR creation/acceptance, capacity request coordination, exception handling
Platform / PinCompute	Kubernetes admission and node capacity	Namespace/project quotas, Reserved tier behavior, node-pool controls	Quota config PRs, node-pool settings, Karpenter / Auto Scaling Group (ASG) integration
ML Platform / TCP	Shared GPU training capacity	GPU quota model, project allocations, training fleet guidance	TCP quota, GPU capacity planning inputs, reserved GPU pools
Cloud provider account team / TAM	Provider-side delivery	Supply availability, constrained-family guidance, IG/FOOB delivery	Provider-side capacity holds and delivery commitments

The handoff is the hard part. A service team does not simply ask “can I spend more?” It has to specify demand shape: instance family, count, AZ/Region, deadline, duration, fallback options, and whether the workload can tolerate queueing or degradation. CapEng then turns that into provider supply work. PinCompute or TCP turns delivered supply into quota and scheduling behavior.

How demand becomes capacity

Demand enters through capacity requests, quota requests, planning cycles, and event-specific runbooks.

Typical triggers:

• large net-new host footprint;
• migrations, especially hundreds of hosts or more;
• launches, seasonal peaks, and marketing events;
• incident-driven surges, which may be filed retroactively after stabilizing;
• GPU training or serving growth;
• sustained queueing or quota pressure in TCP / PinCompute.

The rough operating cadence from the research:

Horizon	Meaning
6 months	Rolling capacity forecast: 3 months hard signal, 3 months soft signal refreshed through Adaptive Governance
8-12 weeks	Normal lead time for low-volume, hot, or constrained instance families
About 15 weeks	Forward signal expected for AWS GPU / accelerator capacity
7-8 weeks	Large migration planning window, such as 500-host moves
2 business days	Typical triage SLA for generic capacity requests, with high-cost requests taking longer

The failure mode is late demand. If a team discovers a launch needs scarce capacity two weeks before the event, the cloud may not have enough physical supply even if Pinterest has budget and quota.

Capacity request intake

The Capacity Request Tool (CRT) is the user-facing entrypoint for large EC2 footprint changes. Submitting a request creates a capacity request and an automatic Slack thread in #capacity-request-notifications.

The useful fields are not just “how much capacity”:

Field	Why it matters
Nimbus project / Above-Below project	Ties the ask to the funded project and approval context
Platform	Tells CapEng whether the capacity will be consumed by Teletraan, PinCompute, TCP, or another platform
Instance type and size	Defines the exact supply shape AWS must provide
Count and AZ distribution	Shows whether the request needs balanced multi-AZ capacity or a pinned locality
Need-by date	Determines whether self-service, ODCR, future-dated ODCR, or IG/FOOB is plausible
Migration source and release date	Lets CapEng avoid double-provisioning old and new families indefinitely
Launch / migration / incident links	Provides the evidence Spend Captains and CapEng use to judge priority

Large migrations need the full shape: total host count, testing batch, production batch, AZ distribution, and release plan for the old instance type. GPU or hot-family asks need longer lead time because provider supply has to be planned, not merely approved.

Incidents are treated differently. Teams should mitigate first, then file a retroactive capacity request once stable so the temporary demand becomes visible to CapEng and Spend Captains.

ODCR and IG decisions

ODCRs are used when Pinterest needs launch certainty for a specific shape: account, AZ, instance type, platform, and tenancy. IG/FOOB is used when the ask is large, forecastable, or constrained enough that AWS must plan supply manually.

The decision is not “do we want a discount?” That is Savings Plans / Reserved Instances. The ODCR question is:

If this capacity is unavailable at the moment we need it, what breaks, and is that worse than paying for idle reserved slots?

Immediate ODCR

Use an immediate ODCR when the start time is unpredictable but the capacity must be launchable on demand.

Examples:

• offline fleet restart capacity after maintenance;
• failover capacity during an AZ evacuation;
• migration capacity where jobs may restart or reschedule unpredictably;
• constrained families where opportunistic users could consume the pool before the funded workload arrives.

Immediate ODCRs are expensive when unused. CapEng audits for reduction opportunities and tracks exceptions because an idle ODCR is a real capacity claim, not just a line in a cost report.

Future-dated ODCR or IG

Use future-dated ODCR or IG/FOOB when the need has a known window and the lead time is long enough.

Examples:

• holiday peak;
• product launch;
• large migration;
• planned GPU serving rollout;
• training capacity for forecasted model work.

The provider-side delivery matters operationally: once AWS delivers capacity, CapEng accepts ODCRs only when requestors confirm they are ready to consume. Otherwise Pinterest pays for idle capacity and may block other customers from using scarce supply.

Open vs targeted reservations

Open ODCRs allow any matching instance in the account to consume the reservation. They are operationally simple but can be grabbed by the wrong workload.

Targeted ODCRs are tied to a resource group or workload. They are used when a specific domain, such as serving or batch, funded the capacity and needs protection from cross-workload contention.

Karpenter and PinCompute increasingly consume ODCRs programmatically. The practical issue is fallback: a cluster that strictly matches only reserved capacity can fail to scale when the reservation shape is wrong, while a cluster that falls back too freely can double-spend by leaving ODCRs idle and launching ordinary on-demand instances elsewhere.

Operational trace: CR #2990

CR #2990 is a concrete example of how demand becomes held capacity.

The demand came from Ads RecGPT for shopping, which needed more capacity on the ro-wc-theta-use1 Galaxy KVStore cluster. The forecast was tied to incremental traffic: about +22k queries per second (QPS) to GSS and about 1.32M keys per second (KPS) to partitions_pin_features_6, on top of existing cluster load. KV and Ads first iterated through an earlier request, then re-filed against the correct Nimbus project.

GSS is an internal serving dependency in this request path; the important capacity signal is the extra QPS/KPS, not the service acronym itself.

The final ask was 33 i7ie.2xlarge hosts. The approval path was:

1. Capacity request filed in CapEng tooling.
2. CapacityGuru posted the request into #capacity-request-notifications.
3. InfraGov / budget-transfer context was added for the relevant Ads project.
4. Spend Captain approval landed.
5. CapacityGuru marked the request approved and pending delivery.
6. CapEng found or held ODCR capacity for the request.
7. CapacityGuru marked the request delivered.
8. KVStore planned a ramp: roughly 50% first, then 100% after validation.

The ODCR Exceptions Log recorded a 33-instance i7ie.2xlarge block held for the request. That exception matters because a generic ODCR-reduction audit would otherwise see idle i7ie.2xlarge capacity and recommend shrinking it. The log tells the next on-call that the idle capacity is intentional until the request is consumed.

The useful lesson: delivery is not the end of the workflow. The capacity has to be consumed, then later released or folded into the steady-state ODCR plan. Without that last step, a successful capacity request becomes stranded idle capacity.

ODCR audit loop

CapEng on-call manages ODCRs as an active inventory, not a one-time purchase. The daily loop from the research is:

1. Check Statsboard ODCR dashboards for missing data, unexpected drops, and available capacity by instance family.
2. Ask HelixBot for short-term ODCR reduction opportunities.
3. Run fuller ODCR audits when handing off or preparing weekly changes.
4. Compare reduction suggestions against the ODCR Exceptions Log.
5. Reduce reservations that are persistently unused and not exception-protected.
6. Record new exceptions when capacity is intentionally idle for a migration, launch, supply-risk buffer, or request like CR #2990.

The pattern is important: automation finds candidates, but humans preserve context. A reservation can be idle because it is waste, or idle because it is insurance for a known future action.

Supply management

CapEng tracks supply through a mix of dashboards, alerts, provider conversations, and ODCR inventory controls.

Signal	What it answers
Running Instances Statsboard	What is actually running by instance family and AZ?
ODCR Available Capacity Statsboard	How many reserved slots are still unused by family and AZ?
ICE alerts	Are launches failing because AWS lacks physical supply?
EC2 quota alerts	Is the blocker a service quota rather than physical supply?
HelixBot ODCR audits	Which reservations look persistently unused and reducible?
ODCR Exceptions Log	Which idle reservations are intentionally held and should not be reduced?

InsufficientInstanceCapacity (ICE) and quota failures require different fixes. Quota failure means Pinterest is not allowed to ask for more of a resource until the service quota is raised. ICE means the provider lacks suitable physical capacity in that location at that moment. A team can have budget, quota, and Savings Plan coverage and still fail to launch if physical capacity is missing.

The supply playbook is ordered:

1. Avoid scarce shapes if the workload does not need them.
2. Spread across more AZs where data locality and service design allow it.
3. Add fallback families or architectures.
4. Use ODCRs when the shape is necessary and failure impact justifies idle cost.
5. Use IG/FOOB and TAM escalation when the ask is large or provider-constrained.

The ODCR Exceptions Log is the operational memory for “idle but intentional” capacity. Entries record user, date, instance type, count, AZ, reason, and completion status. Without this log, one on-call can preserve capacity for a launch while the next on-call releases it during an unused-capacity cleanup.

PinCompute quota and capacity

PinCompute turns underlying EC2 capacity into Kubernetes namespace/project quotas. For normal requests, teams use PinConsole or K8sDashboard to generate a quota config PR and a CMP Jira ticket.

Large changes require capacity planning, not just quota editing:

• more than 120 CPU;
• more than 300 Gi memory;
• any GPU capacity;
• Reserved tier requests;
• large migrations or launch-driven increases.

The key operational rule is that quota is not supply. A namespace can have quota but fail to schedule if the cluster lacks matching nodes. Conversely, EC2 supply can exist but remain unavailable to a project until PinCompute quota and scheduling rules expose it.

Reserved tier quotas are stricter because they represent funded or protected capacity. Requests and limits are usually set equal so pods receive Guaranteed Quality of Service (QoS, Kubernetes’ scheduling and eviction class) and capacity accounting stays tractable.

Node capacity also has platform overhead. System daemons such as metrics agents, logging agents, and platform services consume CPU and memory on every node, so raw instance capacity is not equal to allocatable workload quota.

The quota-to-node translation has to subtract daemon overhead. One runbook example uses roughly 2.95 vCPU and about 3.8 GiB memory reserved per node for platform daemons. If a team reserves 10 c5.9xlarge nodes, the user-visible quota is not 10 * 36 vCPU and 10 * 72 GiB. The configured quota is closer to:

CPU quota    = (36 - 3) * 10 = 330 vCPU
Memory quota = (72 - 5) * 10 = 670 GiB

The exact daemon reservation changes over time, but the mechanism is stable: capacity planning starts from allocatable node capacity, not sticker capacity.

Reserved-tier GPU quota flow

GPU Reserved tier quota requires more than increasing namespace quota. The platform has to prove that funded GPU supply exists and that the namespace can consume it without stealing from unrelated teams.

reservedQuotaConfig is PinCompute’s quota-reservation record. Upstream Kubernetes has no field with that name. The PinCompute CRD models it as a QuotaReservation attached to a namespace quota config: instanceSelector chooses the instance type or ASG that can satisfy the reservation, zones constrains where the reserved capacity can land, strategy controls zone distribution, and quota declares the reserved-tier resource list (requests.cpu, requests.memory, requests.nvidia.com/gpu, and pods). The newer reservedQuotaConfigs field is the plural form for multiple reservations; admission rejects configs that set both singular and plural fields. PinCompute’s quota reservation manager reads those records, matches the selector to available node groups, simulates how many unit pods fit on the selected shape, and turns the result into concrete node reservation requests. That field is the bridge between “this namespace has funded Reserved GPU quota” and “these nodes or zones should be held for it.”

The runbook flow from the research:

1. The requester files a quota change through PinConsole or K8sDashboard with a reservedQuotaConfig: instance selector, zones, CPU, memory, GPU, and pod quota.
2. If the request needs new GPU supply, the requester also files a capacity request / IG ticket.
3. Spend Captains approve funding and business priority.
4. CapEng confirms underlying EC2 GPU capacity exists or is being delivered.
5. PinCompute on-call verifies quota ratios against the selected instance type after subtracting daemon overhead.
6. PinCompute updates pinkube quota artifacts and QoS configuration.
7. Strict tiering and fit thresholds may be configured so reserved workloads use the intended GPU pool.
8. If capacity is partial, migration is staged: run only the secured portion on Reserved tier and leave the remainder on OnDemand until more supply arrives.

The failure mode is granting abstract quota before physical capacity exists. That creates schedulable intent but not schedulable pods. The opposite failure also happens: EC2 GPU capacity can exist, but the namespace cannot use it until quota and scheduling policy expose it.

GPU capacity and right-typing

GPU capacity is not just “how many GPUs?” It is “which GPU, for which model, at which batch size, with which latency or training target, in which cluster, and with what lead time?”

Pinterest distinguishes several GPU demand shapes:

• online inference: user-facing serving with latency targets;
• offline / batch inference: throughput-oriented scoring or generation;
• training and fine-tuning: jobs that consume GPUs for model updates;
• evaluation and experimentation: lower-priority or preemptable work;
• large multi-node jobs: workloads sensitive to networking and cluster placement.

Training Compute Platform

The Training Compute Platform (TCP) is the shared GPU training fleet. Jobs are registered under Nimbus projects for cost ownership and quota control.

Each project has:

• a MinGPU estimate, which is the approximate guaranteed share of cluster GPUs;
• a hard quota, which caps maximum GPU consumption;
• queueing behavior that reveals when demand exceeds allocation.

Teams request more GPU capacity when queueing stays high, quota is hit frequently, or upcoming projects increase concurrent workloads, GPUs per workload, model size, or hyperparameter sweeps.

Two simple sizing formulas from the research:

GPUs for concurrency = total_concurrent_workloads * GPUs_per_workload
GPUs for queue relief = expected_pending_workload * GPUs_per_workload

Those formulas are only intake estimates. The real request still needs instance family, AZ distribution, timeline, and budget approval.

GPU capacity request flow

The GPU request flow has two linked halves: provider supply and platform quota.

1. The team notices sustained queueing, repeated quota hits, or upcoming model work that changes GPU demand.
2. The team estimates GPUs needed for concurrency or queue relief.
3. The team records the demand in the GPU planning sheet and files a CRT / IG request with instance type, count, AZ preferences, timeline, and approved project context.
4. Spend Captains approve funding and priority.
5. CapEng coordinates delivery with AWS, especially for P-family and newer GPU generations.
6. After capacity is delivered, the team gives the CRT ID and supply details to PinCompute / TCP owners.
7. Platform owners add nodes or node pools, configure quota, and wire the project into Reserved or OnDemand tier behavior.
8. The team validates utilization and queueing. If utilization stays low, the next action is job or batch-size tuning, not automatically asking for more GPUs.

This is why GPU capacity is slower than ordinary quota. A quota change alone does not create p4d, p5, g6e, or p6 supply in the right AZ.

Operational trace: llms GPU request

The llms GPU request shows how a GPU ask changes as CapEng, Spend Captains, and ML users negotiate cost, supply, and fit.

The original ask was 14 p4de.24xlarge instances for offline VLM (vision-language model) training and corpus-cleaning work, with projected annual cost around $677k. The request did not simply pass through as “team asked for GPUs, buy GPUs.” CapEng pushed back on growing the p4de footprint and suggested alternatives such as p5en or p6-b200. After discussion, the request moved toward g6e: first g6e.4xlarge, then g6e.12xlarge once the smaller shape looked insufficient for training.

The final operational shape was four g6e.12xlarge instances in us-east-1a, backed by ODCR and then wired into PinCompute for the llms project.

The flow:

1. Requestor files GPU capacity request for a Nimbus project.
2. Spend Captains and CapEng ask for planning context and business approval.
3. CapEng challenges the requested GPU family based on supply and strategy.
4. Request is revised from p4de to g6e.12xlarge.
5. CapEng delivers ODCR-backed GPU capacity in the chosen AZ.
6. Requestor files the PinCompute quota request.
7. PinCompute / Cloud Runtime updates pinkube quota and cluster config so the project can schedule onto those GPUs.

The useful lesson: GPU capacity approval is not only budget approval. It is also instance-family governance. CapEng may reject a technically valid request if it deepens exposure to a scarce or strategically discouraged family.

GPU tiering

PinCrease introduces tiered GPU behavior:

Tier	Meaning
Reserved	Project-specific reserved pools with minimum guarantees and dedicated queues
OnDemand	Shared global pool; best effort, but above preemptable work
Preemptable	Spare capacity only; evicted when higher-priority work needs GPUs

Reserved GPU quota is coordinated across users, Spend Captains, CapEng, Cloud Runtime, and GPU capacity leads. It is not just a self-serve quota bump because the underlying GPU supply is scarce and often provider-constrained.

Benchmarking GPU choices

The research says GPU instance choices are benchmark-driven. The useful benchmark is cost per unit of work under the workload’s real constraints, not instance hourly price.

For inference, the benchmark should vary:

• GPU family and instance type, such as G5/G6e/P4d/P5/P5en/P6;
• model size and memory footprint;
• precision or quantization;
• ML batch size, meaning prompts/tokens/images grouped into one GPU pass;
• context length or sequence length;
• concurrency;
• p50/p95/p99 latency target;
• tokens/sec, requests/sec, GPU utilization, and queueing delay.

For training or fine-tuning, the benchmark should vary:

• training batch size, meaning examples per optimizer step;
• gradient accumulation, which splits one logical batch across smaller GPU passes when the full batch does not fit;
• activation memory and optimizer state;
• dataloader throughput;
• checkpoint overhead;
• network bandwidth for multi-GPU or multi-node jobs;
• time to convergence, not just examples/sec.

The practical lesson is that a more expensive GPU can be cheaper per unit of work if it finishes faster, fits the model without contortions, or enables larger batches without breaking latency or memory limits.

Representative decisions from the research:

• migrate some G5 fleets to P4d/A100 when per-GPU cost is similar but throughput is higher;
• evaluate P5/P5en for 7-10B and 34B parameter models where memory and bandwidth matter more than hourly price;
• use G5/G6e for smaller training, batch inference, or workloads where P-family capacity is unnecessary;
• keep GPU memory headroom instead of running close to the limit, because small model or batch changes can turn into scheduling failures or OOMs.

Operational trace: VLM batch-inference benchmark

The VLM batch-inference benchmark is the clearest example in the research of GPU right-typing by measurement rather than hourly price. The workload was Qwen2-VL-7B offline batch inference for alt-text backfill, replacing a 50-node g5.12xlarge fleet.

The benchmark compared g6e.12xlarge, p4d.24xlarge, and p4de.24xlarge using vLLM. It varied max_num_seqs and batch_size to stress KV cache and GPU memory.

Run	Shape	Tuning	Result
p4d.24xlarge	6 nodes	max_num_seqs=768, batch_size=2048	About 2,600 rows/sec, about $2.82 per million inferences, GPU memory around 82.5%
g6e.12xlarge undertuned	10 nodes	max_num_seqs=384, batch_size=1024	About 1,200 rows/sec, about $6.33 per million inferences, GPU memory around 79%
g6e.12xlarge pushed	10 nodes	max_num_seqs=768, batch_size=2048	About 1,700 rows/sec before CPU OOM after 1.5 hours
p4de.24xlarge	6 nodes	max_num_seqs=1536, batch_size=4096	About 3,600 rows/sec, about $2.56 per million inferences, GPU memory around 82.6%

The conclusion was not “newer is better” or “cheaper hourly is better.” p4de.24xlarge was best for this workload, p4d.24xlarge was a strong fallback, and g6e.12xlarge should not replace the fleet until the CPU/input pipeline bottleneck is fixed.

The useful lesson: a GPU benchmark has to include the whole serving or batch pipeline. A GPU can have available memory and still lose because CPU memory, preprocessing, or batching prevents the fleet from reaching efficient throughput.

Event capacity runbook

For launches and seasonal peaks, the operating model is scheduled capacity work, not blind autoscaling.

The generic sequence:

1. Forecast demand and identify constrained families.
2. File capacity requests with enough lead time.
3. Get Spend Captain approval and provider delivery confirmation.
4. Accept ODCR/IG delivery when requestors are ready to consume.
5. Raise ASG or node-pool minimums before traffic arrives.
6. Warm caches, connection pools, database replicas, and queue consumers.
7. Run synthetic probes against the full dependency path.
8. Monitor the event with manual levers ready.
9. Scale down, release temporary reservations, and audit idle capacity.

For GPU serving, the same shape applies, but the inputs are model-specific: expected QPS, token length, latency target, model memory footprint, and benchmark-derived instance counts.

Scarcity and incidents

Pinterest monitors InsufficientInstanceCapacity (ICE) errors and quota/capacity alerts. When an ICE alert fires, CapEng on-call checks:

• instance type and AZ;
• whether the affected fleet should be ODCR-backed;
• whether Karpenter, Auto Scaling Groups, or node pools are pinned too tightly;
• whether the issue is physical capacity, service quota, or scheduling policy.

Mitigations include:

• spreading node pools across more AZs;
• using fallback instance families;
• launching in unconstrained AZs when data locality allows it;
• stopping low-priority jobs;
• lowering priority or moving experiments to preemptable capacity;
• escalating constrained families to AWS TAMs;
• filing quota increases when the blocker is service quota rather than supply.

The operational trap is confusing quota, capacity, and commitment:

• quota lets Pinterest ask for resources;
• capacity means AWS has physical supply;
• ODCR/IG means Pinterest has a claim on that supply;
• Savings Plans mean Pinterest has a discounted spend commitment.

Incidents happen when one of those is missing and the system assumes another one covers it.

Operational trace: p6 reallocation for Homefeed

The Homefeed p6 case is not a normal capacity purchase. It is shortage triage: find idle reserved GPU supply and move it to the highest-priority workload.

During a broader GPU shortage, CapEng and ML Platform found 10 p6-b200 instances sitting unused in us-east-1a: ODCR-backed capacity existed, but it was not yet fully consumed by active workloads. Homefeed needed production p6 capacity and could not wait for a new AWS delivery.

The decision was to move eight p6 instances to Homefeed and leave two in the OnDemand pool for benchmarking or future shared needs.

The flow:

1. Statsboard showed unused p6 ODCR / OnDemand capacity.
2. CapEng and ML Platform reviewed the shortage and priority.
3. Infra leadership and relevant owners were tagged in Slack so the move was a coordinated allocation, not an ad hoc team grab.
4. Homefeed filed the PinCompute quota request for the eight p6 instances.
5. PinCompute / Cloud Runtime added the nodes to the shared cluster and raised Homefeed’s GPU quota.
6. Two p6 instances were intentionally left as a shared OnDemand buffer.

The cost/reliability tradeoff is straightforward: no new AWS order was needed, so reallocation improved ODCR utilization. Reliability improved for a critical recommender workload, but the platform preserved a small shared buffer instead of allocating every idle GPU to one team.

The useful lesson: during GPU scarcity, ODCR utilization is not just a cost metric. It is a priority-routing signal. Idle reserved GPUs should either be kept intentionally for a named future use or moved to current high-priority work.

Operational trace: AMI build ICE incident

The AMI build incident is a small but useful example because it shows that scarcity affects internal tooling, not just production serving. AMI validation builds repeatedly failed when the pipeline tried to launch d2.4xlarge instances and AWS returned InsufficientInstanceCapacity.

The immediate mitigation was not buying a reservation. The team removed d2.4xlarge from the validation path and moved tests away from that constrained legacy family.

The long-term lesson was that low-volume or legacy instance families are poor dependencies for automation. If an internal tool can use a more available family, that is usually better than carrying special-case capacity risk for a family the provider has little supply of.

That incident illustrates the supply-management hierarchy:

1. Avoid scarce shapes when the workload does not need them.
2. Use fallback families when possible.
3. Reserve capacity only when the shape is necessary and the failure impact justifies idle-cost risk.
4. Escalate to provider capacity requests when the shape is both necessary and too scarce for self-service.

How to read the internal artifacts

The internal source list in the Glean synthesis points to four clusters of documents:

Artifact cluster	What to learn from it
Instance and ODCR Strategy, ODCR Monitoring and Management, CapEng On-Call	ODCR ownership, audits, acceptance, exceptions, and scarcity handling
Capacity Requests, Capacity Forecasting, Holiday Capacity Planning, Adaptive Infrastructure Governance	Demand intake, lead times, planning cadence, Spend Captain approval
PinCompute quota docs, Kubernetes capacity planning tools, daemon right-sizing, whole-node occupation	How EC2 supply becomes Kubernetes quota and schedulable capacity
TCP, GPU Capacity Planning, PinCrease, Training Efficiency, ML Benchmarking Handbook, Navigator/SMS serving docs	GPU quotas, model benchmarks, instance selection, serving/training differences

Read them with these questions in mind:

1. What decision does this artifact support?
2. Who is allowed to act on it?
3. What is the lead time?
4. What happens if the forecast is wrong?
5. What is the fallback when supply is unavailable?
6. What cost is intentionally idle versus accidentally stranded?

Open questions for the next research pass

The current research is enough to understand the operating model, but several details still need source-backed examples:

1. What exact fields are mandatory in a capacity request, and which fields are tribal knowledge?
2. What is the current decision threshold for immediate ODCR vs future-dated ODCR vs IG/FOOB?
3. Which ODCR exceptions are still active, and how old are they?
4. Which dashboards are trusted for ODCR utilization versus finance variance tracking?
5. How does Karpenter’s reserved-capacity behavior differ by cluster?
6. For GPU Reserved tier, which projects are strict-tiered today?
7. Which GPU benchmark artifacts are current enough to use for 2026 capacity planning?
8. How do Entitlements change the handoff between budget approval, quota, and physical supply?

See also

• Cloud Capacity Reservations
• Event-Driven Capacity Planning
• Cloud Capacity Management
• Kubernetes Capacity Management
• Instance Rightsizing
• Compute Fleet Strategy