Kubernetes Capacity Management

The basic model — pods declare requests, scheduler places them, autoscaler adds nodes — is straightforward. The hard problems emerge at the boundaries: how the kernel enforces resource limits, how overcommit creates hidden risk, how system services compete with application pods, and how the QoS hierarchy decides who dies first.

Prerequisites

• Cloud Capacity Management — three-layer model (usage, capacity, rate)
• Instance Rightsizing — CFS throttling, VPA algorithm

QoS classes and the OOM kill hierarchy

Kubernetes assigns a QoS class to every pod based on how requests and limits are set. The class determines who gets killed first when the node runs out of memory.

QoS class	Condition	oom_score_adj	Behavior
Guaranteed	Every container has requests == limits for both CPU and memory	-997	Last to be killed. Gets CPU bandwidth guarantee (no throttling beyond limit).
Burstable	At least one container has requests < limits	2–999 (calculated per container)	Killed after BestEffort. Can burst above request but risk OOM if node is full.
BestEffort	No requests or limits set	1000	Killed first. Gets whatever is left on the node.

The kernel’s OOM killer selects the process with the highest oom_score, which combines oom_score_adj (set by kubelet) with actual memory consumption (larger RSS, or resident set size — process memory currently resident in physical RAM — means a higher score within the same adjustment range).

Design decision: Guaranteed QoS gives maximum safety but minimum flexibility. If every pod is Guaranteed, you lose all overcommit benefit and the cluster behaves like dedicated VMs. If most pods are BestEffort, you save money but accept that anything can be killed at any time.

Most production clusters use Burstable for the majority of workloads, with Guaranteed reserved for truly critical services (databases, control plane components).

cgroup memory enforcement: what happens at the boundary

Kubernetes programs a cgroup hierarchy. On Linux nodes, kubelet and the container runtime normally create a pod-level cgroup or systemd slice for the pod sandbox, then child cgroups for the containers. The pod cgroup gives kubelet a unit for pod accounting, QoS, and eviction. Newer pod-level resource settings can also place controls at that pod layer when enabled, while the child container cgroups remain in place. The container cgroups are where the usual container resources.limits become hard kernel controller settings. Exact paths differ between cgroup v1 and v2 and between the systemd and cgroupfs drivers, but the ownership model is stable: kubelet calculates the resource settings, the runtime creates the cgroups, and the kernel enforces limits against processes in those cgroups.

The grouping boundary is the running Pod, not the Deployment or ReplicaSet that created it. An app container and its sidecar share the same pod-level cgroup because they run in the same Pod, while each container has its own child cgroup under that Pod. Two replicas from the same ReplicaSet get two separate pod-level cgroups, even if the scheduler places both Pods on the same node.

Resource sameness comes from the Pod template, not from the ReplicaSet sharing a cgroup. A ReplicaSet normally creates each new Pod from the same template, so two app-container replicas start with the same requests and limits if no admission policy or runtime resize changes them. Containers inside the same Pod can still have different resources: the app container might request 2 CPU and 4Gi, while a logging sidecar requests 100m CPU and 128Mi. Live Pods can also drift from the original template if a mutating admission policy, defaulting rule, or in-place resource resize changes one Pod at creation time or runtime.

In the common container-level memory model on cgroup v2:

• memory.max = container’s memory limit (hard ceiling)
• the memory request is mainly a scheduling, QoS, and eviction input; on some cgroup v2 setups the runtime can also use it as a soft protection hint via memory.min or memory.low, but the request is not the hard kill boundary.

For an ordinary containers[].resources.limits.memory setting, memory.max lives on the child container cgroup. If the app container exceeds its limit, the kernel reclaims memory and then chooses a victim inside the app container’s cgroup; the sidecar’s child cgroup is not the pool being charged for that limit. If a cluster uses pod-level memory controls, then the parent pod cgroup can also have its own limit, and a pod-level OOM can choose from processes under that pod boundary.

When a container hits memory.max:

1. Kernel attempts memory reclaim (page cache eviction, compaction)
2. If reclaim fails → OOM kill of a process within that cgroup
3. The OOM killer picks the highest-oom_score process within the cgroup (usually the main process)
4. Kubelet records the terminated container state with reason OOMKilled and restart policy determines next action

If the killed process is the sidecar’s main process, kubelet normally restarts that container in the same Pod on the same node. The Pod stays bound to that node; Kubernetes does not deschedule it just because one sidecar died. In a Deployment, the Pod restart policy is Always, so both app containers and sidecar-as-app-container patterns restart in place. Native Kubernetes sidecars are initContainers with container-level restartPolicy: Always, so they also restart independently of the main container. Repeated sidecar crashes put that container into restart backoff and can make the Pod unready if the sidecar has a readiness probe, but the Deployment controller does not create a replacement Pod unless the existing Pod is deleted, evicted, or reaches a terminal phase.

The page cache complication

Linux aggressively uses free memory for page cache. A container reading large files can push RSS + page cache close to memory.max without the application actually allocating that memory. When the application then tries to allocate, it can trigger OOM because page cache eviction doesn’t happen fast enough.

Practical implication: containers doing heavy file I/O (log writing, data processing) often need memory limits higher than their “real” RSS would suggest, because the kernel counts page cache against the cgroup’s memory usage.

CPU CFS throttling: the hidden latency killer

Covered in detail in The CPU limits debate, but the capacity management implication:

If your cluster runs pods with CPU limits and you see unexplained P99 latency spikes, check container_cpu_cfs_throttled_periods_total. The fix might be:

1. Remove CPU limits entirely (keep only requests)
2. Or increase limits to 2–3× observed P99 CPU usage within a period
3. Or switch to Guaranteed QoS (limits = requests) at a higher level

Option 1 means the pod can burst on idle CPUs. Option 3 means the pod has a guaranteed CPU allocation regardless of neighbors. Each has different capacity implications.

System service protection

Nodes run kubelet, containerd/docker, journald, systemd services, and OS-level daemons alongside application pods. If pods consume all resources, system services starve and the node becomes unresponsive (the “node NotReady” failure mode).

Reserved resource budgets

Kubelet allocates resources in layers:

Total node capacity
  - kube-reserved       (kubelet, container runtime)
  - system-reserved     (OS daemons, journald, sshd)
  - eviction-threshold  (buffer before hard eviction)
  = allocatable         (what pods can use)

Configuration flags:

• --kube-reserved=cpu=200m,memory=256Mi — reserves for kubelet + runtime
• --system-reserved=cpu=100m,memory=128Mi — reserves for OS services
• --eviction-hard=memory.available<500Mi — triggers pod eviction when free memory drops below threshold

Common mistake: not setting these at all. Default is zero reservation, meaning pods can consume 100% of node memory and CPU. Under memory pressure, system services get OOM-killed, kubelet crashes, and the node drops out of the cluster.

Eviction signals and thresholds

Kubelet continuously monitors node resources and evicts pods when thresholds are breached:

Signal	Condition	What kubelet does
memory.available < hard threshold	Node free memory critically low	Evicts pods (BestEffort first, then Burstable by most over-request)
nodefs.available < threshold	Node filesystem filling up	Evicts pods using most local storage
imagefs.available < threshold	Image filesystem full	Garbage-collects unused images
pid.available < threshold	PID exhaustion	Evicts pods

Eviction order follows QoS class priority, then resource consumption relative to requests (pods most exceeding their requests are evicted first among pods of the same QoS class).

Memory overcommit: how to do it safely

Overcommit means the sum of pod memory requests exceeds node allocatable memory. This works because not all pods hit their peak simultaneously. But it creates a risk: if enough pods spike concurrently, the node runs out and OOM kills begin.

The overcommit ratio

overcommit_ratio = sum(pod_memory_requests) / node_allocatable_memory

Ratio	Risk profile
1.0	No overcommit. Every pod guaranteed. Expensive.
1.0–1.3	Mild. Safe for most production clusters.
1.3–1.8	Moderate. Acceptable for batch/dev clusters. Needs monitoring.
>2.0	Aggressive. Expect OOM kills under load. Only for BestEffort workloads.

Making overcommit safe

1. Use Burstable QoS with conservative requests — set requests to P95 of observed usage, limits to 2× requests. Pods have burst room without blocking excessive capacity.
2. Monitor memory.available on nodes — alert at soft threshold (e.g., 1Gi free) before hard eviction kicks in.
3. Pod priority classes — ensure critical workloads survive eviction:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-service
value: 1000000
preemptionPolicy: PreemptLowerPriority

4. Pod disruption budgets — prevent eviction from taking down too many replicas of a service simultaneously.

Pod priority and preemption

When a high-priority pod can’t be scheduled (no capacity), the scheduler can preempt lower-priority pods to make room:

1. Scheduler identifies pending high-priority pod
2. Finds nodes where evicting low-priority pods would create enough room
3. Evicts those pods (respecting PDBs when possible)
4. Schedules the high-priority pod

System critical priorities (built-in):

• system-node-critical (2000001000) — kubelet, kube-proxy, CNI
• system-cluster-critical (2000000000) — CoreDNS, metrics-server

If your platform runs both serving and batch workloads on the same cluster, priority classes are how you ensure serving pods always get resources even when batch pods fill the cluster.

The HPA + node scaling coordination gap

The well-known problem: HPA wants to add pods, but no node has room, so pods sit Pending until the autoscaler provisions a new node (60–120s).

Less obvious problems:

Scale-down thrashing

Node autoscaler removes underutilized nodes. But removing a node forces pod rescheduling. If the rescheduled pods trigger HPA scale-up (because they’re now colocated and utilization increases), HPA adds more pods, which may trigger a scale-up of new nodes. Cycle repeats.

Mitigation: --scale-down-utilization-threshold and cool-down periods. Karpenter handles this more gracefully via consolidation (moves pods to fewer nodes without intermediate Pending state).

Over-provisioning with balloon pods

A pattern for latency-sensitive services: deploy low-priority “balloon” pods that occupy node capacity. When real pods need space, balloon pods get preempted instantly (they do nothing, so no graceful shutdown needed). The node already exists and is warm — no 60–120s provisioning wait.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: balloon
spec:
  replicas: 3
  template:
    spec:
      priorityClassName: balloon-priority  # very low (e.g., -1)
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          requests:
            cpu: "4"
            memory: "8Gi"

Cost: you pay for the extra nodes, but they’re “instant headroom” when demand spikes. The balloon pods consume capacity that would otherwise sit idle-but-allocatable.

Shared cluster cost attribution

The hard question isn’t “how to split cost” but request-based vs usage-based attribution:

Method	How it works	Fair?
Request-based	Team pays for what they request, regardless of actual usage	Encourages tight requests, penalizes over-provisioning
Usage-based	Team pays for actual CPU/memory consumed	Doesn’t account for reserved headroom; “free” to over-request
Hybrid	Request-based for memory (because it blocks capacity), usage-based for CPU (because unused CPU is available to others)	Most accurate model of actual cost causation

The hybrid approach reflects reality: in Kubernetes, unused CPU is shared (other pods can use it), but memory requests are “hard” — they block scheduling capacity whether used or not.

See also

• Instance Rightsizing
• Spot Instances
• Cloud Capacity Management
• Cloud Cost Attribution