GPU Capacity Planning

GPU capacity planning is harder than CPU capacity planning because the resource is both scarce and workload-shaped. A CPU service usually asks for more cores or more instances. A GPU service has to answer a tighter question:

Which accelerator, in which Region/AZ, at which batch size, for which model, under which latency or training target?

The wrong GPU can be expensive even when it is cheap per hour. A smaller GPU may force low batch sizes, spill to CPU, or fail to fit the model. A larger GPU may cost more per hour but finish the work faster enough to be cheaper per unit of work.

Prerequisites

• Cloud Capacity Management - supply, demand, and capacity-risk framing
• Cloud Capacity Reservations - capacity reservations and scarce-instance guarantees
• Instance Rightsizing - CPU, memory, and instance-family right-sizing

Workload shapes

GPU demand should be classified before picking an instance type.

Workload shape	Primary metric	Common constraint	Planning question
Online inference	p95/p99 latency, throughput per dollar	latency, memory, queueing	How many requests/tokens can this model serve within the SLO?
Offline / batch inference	rows, images, or tokens per hour per dollar	throughput, input pipeline, memory	Which GPU completes the backfill cheapest before the deadline?
Training	time to useful checkpoint or convergence	memory, interconnect, dataloader, optimizer state	Which cluster shape reaches the target model quality cheapest?
Fine-tuning	step time and time to convergence	activation memory, optimizer state	Can the model train safely on smaller GPUs, or does it need larger memory?
Experimentation	queue time and iteration speed	quota, preemption tolerance	Should this run on reserved, on-demand, or spare/preemptable capacity?

The same model can need different capacity for different modes. An instance type that is good for low-latency serving can be a poor training choice if optimizer state and activations do not fit. A GPU that is overkill for one request can be efficient for batched offline inference.

Batch size means different things

For inference, batch size is the number of prompts, images, requests, or tokens grouped into one GPU pass. Larger batches usually improve throughput because the GPU does more parallel work per kernel launch. They can hurt latency because requests wait for the batch to fill or finish.

For training, batch size is the number of examples used for one optimizer step. Larger batches increase activation memory and can change model behavior. When the desired logical batch does not fit in memory, teams may use gradient accumulation: split one logical batch across several smaller GPU passes, accumulate gradients, then run one optimizer step.

Do not compare inference and training batch sizes as if they were the same quantity. They answer different questions.

What to benchmark

GPU right-typing should benchmark the workload’s real shape, not just a synthetic matrix multiply or vendor peak FLOPS (floating-point operations per second) number.

For inference, vary:

• instance family and GPU generation;
• model size and memory footprint;
• precision or quantization;
• context length / sequence length;
• inference batch size;
• concurrency;
• request mix;
• input pipeline and CPU preprocessing;
• latency target.

Measure:

• requests/sec, rows/sec, images/sec, or tokens/sec;
• time to first token when serving generated text;
• p50/p95/p99 latency;
• cost per million requests or tokens;
• GPU utilization;
• GPU memory headroom;
• CPU memory and CPU saturation;
• queueing delay.

For training and fine-tuning, vary:

• training batch size;
• gradient accumulation steps;
• precision;
• checkpoint frequency;
• dataloader parallelism;
• number of GPUs per job;
• single-node vs multi-node layout;
• interconnect and network placement.

Measure:

• step time;
• examples/sec or tokens/sec;
• time to target quality or convergence;
• GPU utilization;
• GPU memory headroom;
• activation memory;
• optimizer-state memory;
• dataloader throughput;
• checkpoint overhead;
• network bandwidth and all-reduce time (time spent combining gradients across GPUs) for distributed jobs.

Decision rule

Pick the instance type using cost per unit of useful work:

inference_cost = cost_per_hour / useful_inferences_per_hour
training_cost  = cost_per_hour * hours_to_target_checkpoint

The cheap hourly instance loses if it needs many more nodes, runs smaller batches, misses the latency target, or fails from CPU/memory bottlenecks. The expensive hourly instance wins if it reduces total nodes, finishes sooner, or keeps enough memory headroom to avoid retries and operational churn.

The benchmark must show both cost and constraint:

Claim	Evidence needed
”This GPU is cheaper”	Cost per unit of work, not hourly rate
”This GPU is fast enough”	p95/p99 latency or training step/convergence time
”This GPU fits”	Model weights + activations + KV cache (transformer key/value cache) / optimizer state + headroom
”This fleet is reliable”	Capacity availability, fallback shape, quota, and reservation plan

Capacity procurement

GPU planning has longer lead times than ordinary CPU planning because supply is thin and provider-specific. The capacity request should state:

• workload type: inference, training, fine-tuning, evaluation, experimentation;
• model or model family;
• instance type candidates;
• count and duration;
• Region/AZ constraints;
• required start date;
• fallback instance types;
• whether queueing, lower batch size, lower context length, or degraded model quality is acceptable;
• whether capacity must be reserved or can run opportunistically.

The capacity decision and the right-typing decision interact. Benchmarking may show that one GPU is best, but the provider may not have enough of it in the needed AZ. A practical plan includes the preferred shape and fallback shapes.

Failure modes

Failure mode	What it looks like	Fix
Benchmark ignores batch size	Fleet looks expensive after launch	Re-run with realistic inference/training batches
Benchmark ignores CPU/input pipeline	GPU appears underutilized	Measure CPU memory, dataloading, preprocessing, and queueing
Chosen GPU barely fits	Small model or context change causes OOM	Keep explicit GPU memory headroom
Capacity request arrives late	Provider cannot deliver scarce GPUs	Forecast earlier or choose fallback families
Quota exists but nodes do not	Jobs are admitted but cannot schedule	Reconcile quota with physical supply and reservations
Reservation exists but workload misses it	Reserved GPUs sit idle while on-demand launches elsewhere	Fix targeting, resource groups, scheduler integration, or labels

See also

• Cloud Capacity Management
• Cloud Capacity Reservations
• Instance Rightsizing
• Kubernetes Capacity Management
• Pinterest Cloud Capacity Operating Model