Instance Rightsizing
Rightsizing is matching instance types and sizes to actual workload needs. The basic idea (look at utilization, pick something smaller) is obvious. What’s non-obvious is the mechanism: how do you make a safe recommendation when workloads are bursty, memory behavior is non-linear, and CPU throttling creates latency problems invisible in average metrics?
Prerequisites
- Kubernetes Capacity Management — QoS classes, cgroups, OOM behavior
- Cloud Compute Pricing Primitives — why instance family matters, not just size
The CPU limits debate
The most contentious question in Kubernetes rightsizing is: should you set CPU limits at all?
How CFS throttling works
Linux’s Completely Fair Scheduler (CFS) enforces CPU limits via a bandwidth controller. The mechanism:
- Each cgroup gets a quota (microseconds of CPU time) and a period (default: 100ms).
- A pod with
limits.cpu: 2gets 200ms of CPU time per 100ms period. - If the pod exhausts its quota within a period, all threads are throttled until the next period begins — even if the node has idle CPUs.
The non-obvious failure: a pod averaging 30% CPU utilization can still be throttled. Multi-threaded applications can consume their entire quota in a burst at the start of a period (e.g., handling a batch of requests simultaneously), then sit throttled for the remaining 60–80ms. This shows up as P99 latency spikes with no corresponding CPU saturation signal.
The two schools
| Approach | Who uses it | Rationale |
|---|---|---|
| No CPU limits (only requests) | Google’s internal practice, many SREs | Avoids throttling; relies on requests for scheduling fairness. Noisy neighbors are handled by monitoring, not kernel enforcement. |
| CPU limits | Organizations needing hard isolation, multi-tenant clusters | Prevents one pod from starving others. Required for Guaranteed QoS class. |
Diagnosing CFS throttling
Check the container’s cpu.stat in cgroup v2 (or cpu,cpuacct in v1):
nr_throttled — number of times the cgroup was throttled
throttled_time — total nanoseconds spent throttled
In Kubernetes, exposed via container_cpu_cfs_throttled_periods_total
and container_cpu_cfs_throttled_seconds_total in cAdvisor/Prometheus.
Rule of thumb: if nr_throttled / nr_periods > 0.05 (5% of periods
experience throttling), the workload is suffering even if average CPU
looks fine.
Memory: the hard constraint
CPU throttling degrades performance. Memory exhaustion kills the process. This asymmetry means memory rightsizing carries more risk and requires higher confidence margins.
How OOM kill priority works in Kubernetes
When a node approaches memory exhaustion, the kernel’s OOM killer selects
a victim based on oom_score_adj (set by kubelet based on QoS class):
| QoS class | oom_score_adj | Kill priority |
|---|---|---|
| BestEffort | 1000 | Killed first |
| Burstable | 2–999 (proportional to request/limit ratio) | Middle — lower request ratio = killed sooner |
| Guaranteed | -997 | Killed last (effectively immune unless node itself is dying) |
The formula for Burstable:
oom_score_adj = 1000 - 1000 × (memory_request / memory_limit)
A pod requesting 512Mi with limit 1Gi gets oom_score_adj = 500. A pod
requesting 900Mi with limit 1Gi gets oom_score_adj = 100 (harder to kill).
Implication for rightsizing: setting requests close to limits (high request/limit ratio) makes your pod harder to OOM-kill at the cost of blocking more cluster capacity. Setting requests low gives you burst room but makes you an early OOM target.
The confidence interval question
How much headroom above observed peak should memory requests include?
There’s no universal answer, but the tradeoffs:
| Strategy | Request setting | Risk | Waste |
|---|---|---|---|
| Tight (P95 of observed) | Low | OOM kill on spikes above P95 | Low |
| Conservative (P99 + 20% margin) | High | Rarely OOM’d | High — blocks capacity |
| VPA default | P90 + safety margin (varies by container history) | Moderate | Moderate |
For online services where OOM kill = dropped requests: use P99 + margin. For batch jobs where OOM kill = retry: P95 is often sufficient.
Memory behavior is not like CPU
- CPU is elastic — throttling is reversible, pod recovers next period
- Memory is inelastic — once RSS (resident set size, the process memory currently resident in physical RAM) grows, it typically doesn’t shrink until the process exits or explicitly frees memory
- Many runtimes (JVM, Go, Python) hold allocated memory and only return it to OS lazily or never
- This means “peak memory observed in the last 2 weeks” is the minimum safe request, not a generous one
How VPA computes recommendations
The Vertical Pod Autoscaler doesn’t just average your usage. The algorithm (as implemented in the recommender component):
- Maintains a decaying histogram of CPU and memory usage samples over time (recent samples weighted more heavily via half-life decay).
- Targets a percentile — default is P90 for CPU, P90 for memory
(configurable via
--target-cpu-percentileand--target-memory-percentile). - Applies a safety margin — multiplier above the target percentile (default: 15% for CPU, 20% for memory).
- Applies a confidence multiplier — wider margin when few samples exist (shrinks as observation window grows).
- Bounds the recommendation between
minAllowedandmaxAllowedfrom the VPA spec.
The result: recommendation = percentile_value × safety_margin × confidence_factor.
VPA modes
| Mode | Behavior | Use case |
|---|---|---|
Off | Only generates recommendations (viewable via API) | Safest; human reviews |
Initial | Applies recommendations only at pod creation | Conservative; no live changes |
Auto | Evicts and recreates pods with new requests | Aggressive; requires PDB for safety |
VPA and HPA conflict
VPA adjusts pod requests; HPA adjusts replica count based on utilization relative to requests. If VPA raises requests, utilization drops, and HPA may scale down replicas — the opposite of what you want. Use VPA on requests/memory and HPA on custom metrics (not CPU utilization) when combining them.
Instance family selection: the non-obvious cases
The family table (CPU-bound → c-family, memory-bound → r-family) is basic. The hard cases:
Network/storage-bound workloads masquerading as CPU-bound
Symptoms: CPU utilization is moderate (40–60%), but latency is high. The bottleneck is network bandwidth or EBS IOPS hitting instance-level ceilings. Upsizing to a larger instance within the same family gives disproportionate improvement because AWS network and EBS limits scale with instance size (not linearly — usually step functions).
Check: compare NetworkIn/Out to the instance’s documented baseline
bandwidth. If you’re hitting the baseline, you need a larger size or a
network-optimized family.
The burstable trap (t-family)
T3/T4g instances use CPU credits. When credits deplete, the instance throttles to baseline (often 20–30% of max). This creates a bimodal performance profile: great during low-traffic hours (accumulating credits), terrible during peaks (credits exhausted).
Only use t-family if: traffic is genuinely bursty with long idle periods, AND you’ve verified credit balance never hits zero during peak. Otherwise the instance is cheaper per-hour but delivers less work per dollar during the hours that matter.
GPU right-typing (not just rightsizing)
GPU instance selection is fundamentally different from CPU instances:
- GPU memory is the hard constraint (model must fit)
- GPU compute utilization varies wildly by ML batch size (how many prompts, images, or training examples are processed in one GPU forward pass), not just by request rate
- The gap between instance types is enormous (g5.xlarge at $1/hr vs p4d.24xlarge at $32/hr)
- Inference vs training have completely different profiles
The question is usually “which GPU type?” not “how many GPUs?” The answer requires benchmarking your specific model with your specific batch sizes.
See GPU Capacity Planning for the deeper capacity-planning and benchmarking workflow.
For inference, batch size usually means grouping multiple user requests or generated tokens together so the GPU does more parallel work per kernel launch. Larger batches improve throughput but can hurt latency because each request waits for the batch to fill or finish.
For training, batch size means the number of training examples used for one optimizer step. Larger batches increase activation memory and can require gradient accumulation (splitting one logical batch across several smaller GPU passes) when the full batch does not fit in memory. A GPU that is perfect for low-latency inference may be a bad training choice if model weights, activations, and optimizer state do not fit.
Measuring cost per unit of work
The correct rightsizing metric is never instance cost alone:
cost_efficiency = infrastructure_cost / business_units_produced
A c5.xlarge at $0.17/hr serving 1,000 req/s is more efficient than a c5.large at $0.085/hr serving 400 req/s:
- c5.xlarge: $0.00017 per 1,000 requests
- c5.large: $0.00021 per 1,000 requests
Downsizing only “saves money” if the smaller instance can handle the same throughput. If it can’t, you need more instances, and the net cost may increase.