Cloud Capacity Management
Capacity management is where cost and reliability collide. Too little capacity causes incidents, throttling, and failed launches. Too much capacity creates idle cost and inaccurate forecasts. The discipline is making the tradeoff between availability risk, cost efficiency, and governance explicit and measurable.
The three-layer model
When reasoning about cloud cost, separate the problem into three independent layers — each with different owners, tools, and levers.
Layer 1: Usage optimization — “Are we using the right amount?”
Actual consumption of resources:
- EC2/VM hours, vCPU-hours, memory-hours, GPU-hours
- Storage GB-months, request counts, data transfer
- Kubernetes pod requests, job resource consumption
- Idle nodes, unnecessary replicas, over-requested CPU/memory
Question: Can this workload use fewer resources while maintaining its SLO?
Levers: Instance Rightsizing, pod request tuning, deleting unused resources, compacting storage, fixing inefficient jobs.
Owner: Service owners, platform teams.
Layer 2: Capacity management — “Do we have enough supply in the right place?”
Ensuring resources are available when needed:
- Enough instances in the right Region/AZ
- Enough failover headroom
- Enough GPU capacity for training/inference
- Enough autoscaler headroom (ASG max, node pool max)
- Enough quota for teams
Question: Will we have capacity during peak / failover / launch?
Levers: ODCRs, capacity requests, ASG max tuning, node pool sizing, quota policy.
Owner: Capacity engineering, platform teams, service owners for demand signals.
Capacity management is not automatically cost optimization
Sometimes it intentionally increases cost to reduce risk. The point is making the tradeoff explicit: “We are paying $X/month in idle capacity to guarantee we can survive an AZ failure.”
Layer 3: Rate optimization — “Are we paying the right price?”
Applying discount instruments to the usage that exists:
- Savings Plans / CUDs
- Reserved Instances
- Spot / preemptible
- Graviton / ARM price-performance
- Storage tiers, private pricing
Question: Given our usage, are we on the cheapest pricing model?
Levers: Commitment purchases, Spot adoption, architecture migration (ARM, serverless), storage tiering.
Owner: Finance, procurement, capacity engineering, FinOps team.
Why separation matters
| Question | Layer | Wrong answer if layers are confused |
|---|---|---|
| ”Why did spend go up?” | Usage/allocation | ”We bought Savings Plans” (rate, not usage) |
| “Will we have capacity at peak?” | Capacity | ”We have Savings Plans” (discount ≠ capacity) |
| “Should we buy more commitments?” | Rate | ”Our nodes are underutilized” (that’s usage, not rate) |
Key distinctions people confuse
Demand has shape, not just magnitude. Average demand is irrelevant for capacity planning — incidents happen at the tail (P95/P99), during failover (AZ loss shifts 50% more traffic to survivors), and during step-functions (migration lands, backfill kicks off). Planning from averages is the #1 cause of capacity incidents.
Capacity is location-bound. A g5.12xlarge in us-east-1a is not
fungible with one in us-east-1d if your ODCR or data is pinned to one
AZ. Every capacity conversation must specify region, AZ, family, and
architecture.
Accelerator capacity is workload-shaped. GPU demand is not just a count of devices. The right instance depends on model size, inference batch size, training batch size, memory headroom, latency target, and whether the job can queue or degrade. See GPU Capacity Planning.
Headroom is not waste — it’s priced insurance. The job is not to eliminate headroom but to make it intentional, visible, and costed. “We pay $X/month in idle capacity to survive single-AZ failure” is a valid engineering decision. “We have extra capacity because nobody reviewed the ASG max in 6 months” is not.
Quota ≠ capacity. A team may have quota but no physical instances available. Capacity may exist but quota blocks a team from using it. Quota is a fairness and governance mechanism. Capacity is physical supply. Conflating them causes “I have quota, why can’t I launch?” confusion.
The capacity planning loop
The loop is: Observe → Forecast → Decide → Execute → Validate → Institutionalize → repeat. See Cloud Capacity Forecasting for the forecasting mechanics.
The non-obvious part is what makes each step fail:
- Observe fails when you measure requested resources instead of actual usage (Kubernetes clusters look “full” at 30% real utilization because requests are inflated), or when you lack AZ-granularity on saturation signals.
- Forecast fails when you capture only baseline trend and miss step-functions (a migration landing, a backfill kicking off, a feature launch). See The fatal mistake average-based planning.
- Decide fails when capacity and rate decisions are conflated (“we have Savings Plans, so capacity is handled” — no, those are different layers), or when the decision doesn’t account for commitment portfolio impact.
- Execute fails when lead times are not respected (large GPU reservations need 6–8 weeks; a request filed 2 weeks before a launch will not land in time).
- Validate fails when you check only cost OR reliability, not both. A capacity change that improves cost but degrades P99 latency is not a success.