Event-Driven Capacity Planning
Event-driven capacity planning handles predictable demand spikes that exceed normal autoscaling headroom: holiday shopping seasons, product launches, marketing campaigns, live events, and annual peaks. These events are foreseeable (unlike incidents) but their magnitude often exceeds what steady-state capacity can absorb.
“Event-driven” means the capacity plan is driven by a known business event, not that the infrastructure necessarily scales from an event bus message. The execution can be:
- Manual runbook — an operator raises ASG (Auto Scaling Group) minimums, database replica counts, cache capacity, and queue-consumer counts at planned times.
- Scheduled automation — a CI (continuous integration) job, Argo Workflow, Terraform pipeline, Kubernetes CronJob, or EventBridge scheduled rule applies the same changes with approvals and validation.
- Hybrid — automation prepares reversible changes, but humans approve the final scale-up, failover test, or scale-down.
The mature version is usually a scheduled runbook encoded as automation: scale the fleet before the event, warm dependencies, validate health, monitor during the event, then scale down in controlled steps.
Why autoscaling alone isn’t enough
For most organizations, steady-state autoscaling (HPA, or Horizontal Pod Autoscaler, plus the node autoscaler) handles daily/weekly traffic patterns. Event-driven peaks break this because:
- Scale exceeds headroom — ASG max or node pool max is too low
- Provisioning latency — new nodes take 2–5 minutes; a traffic spike measured in seconds can saturate before scaling kicks in
- Capacity availability — cloud provider may not have enough supply of your instance type in your AZ during high-demand periods (everyone scales up for Black Friday)
- Dependency coordination — databases, caches, and downstream services also need to scale; they have longer warm-up times
- Supply chain lead times — reserving GPU or constrained capacity requires weeks of advance notice
Planning timeline
| Lead time | Action |
|---|---|
| 8–12 weeks | Identify capacity needs; submit capacity requests for constrained types |
| 6–8 weeks | Cloud provider confirms ODCR (On-Demand Capacity Reservation) / instance guarantee delivery |
| 4 weeks | Load test at expected peak; validate autoscaler limits |
| 2 weeks | Pre-scale ASG min / node pool min; warm caches and connections |
| 1 week | Final capacity validation; runbook review; on-call staffing |
| Event day | Monitor; manual scaling levers ready; war room for anomalies |
| Post-event | Scale down; release temporary capacity; run retrospective |
Capacity instruments for events
| Instrument | When to use | Lead time |
|---|---|---|
| ASG max increase | Known peak within existing instance types | Minutes (config change) |
| Pre-scaling (increase ASG min) | Ensure warm capacity before spike | Hours |
| ODCR (immediate) | Hold empty capacity for an unpredictable start time, such as failover or sudden queue burst | Hours–days |
| ODCR (future-dated) | Known event date where capacity should start later, not today | Weeks |
| Provider capacity request / instance guarantee | Short-term hold coordinated with the provider account team, often for constrained capacity | 2–8 weeks |
| Capacity Block | Scheduled GPU/accelerator access | Varies by availability |
| Spot fleet expansion | Non-critical workloads, cost-sensitive burst | Minutes |
Pre-scaling strategy
Pre-scaling means raising the minimum instance count before traffic arrives, so the system is already warm when demand hits.
What to pre-scale
- Application instances (pods / ASG min)
- Cache warm-up (pre-populate connection pools, cache entries)
- Database read replicas
- CDN (Content Delivery Network) / edge capacity (if self-managed)
- Queue consumers
Example event runbook
A retailer expects a product drop at 10:00.
| Time | Action | Why |
|---|---|---|
| 08:00 | Raise ASG min from 30 to 120 and Kubernetes node-pool min from 20 to 80 | Nodes and pods are already registered before traffic arrives |
| 08:30 | Pre-warm application caches and connection pools | First real users do not pay the cold-start cost |
| 09:00 | Increase queue consumers from 20 to 100, but keep risky workers paused if they would process jobs too early | Capacity is ready without accidentally draining business queues |
| 09:30 | Run synthetic checkout/search/payment probes at elevated load | Confirms the whole dependency chain works, not just the web tier |
| 10:00 | Event starts; monitor saturation, error rate, queue depth, and autoscaler behavior | Manual levers are ready if forecast was wrong |
| 12:00 | Step down to 75%, then 50%, then normal capacity with monitoring gates | Avoids keeping expensive event capacity alive for days |
This can be executed by a person, a script, or a workflow engine. The important property is not the tool; it is that each capacity change has a planned time, owner, validation check, and rollback.
Timing
Scale up well before the event — not at the moment traffic arrives. Reasons:
- Health checks take time (ELB registration, readiness probes)
- Connection pool warming
- JIT compilation warm-up (JVM services)
- Cache filling
Scale-down plan
Equally important: when and how to reduce capacity after the event. Leaving pre-scaled capacity running for days after a 4-hour event is waste. But scaling down too aggressively can cause issues if there’s a tail of demand.
Typical approach: step down in phases (75% → 50% → normal) with monitoring gates between each step.
Load testing
Before any major event, validate capacity at expected peak:
- Define target load — expected peak QPS/RPS with safety margin (1.5–2×)
- Test in production or production-mirror — staging often doesn’t reflect real capacity constraints
- Identify breaking points — which service saturates first? Where’s the bottleneck?
- Validate autoscaler behavior — does it scale fast enough? Does it hit ASG max?
- Test failover under load — simulate AZ failure during peak traffic
Coordination across teams
Event planning requires cross-team coordination because services have dependencies:
Frontend → API gateway → Application → Cache → Database
→ ML inference
→ Payment provider
If the application tier scales but the database can’t handle the connection count, the system fails. Event planning must trace the dependency chain and ensure every component can handle peak load.
Checklist for each service in the dependency chain
- Can handle expected peak load (proven by load test)?
- Autoscaler limits raised?
- Pre-scaled if needed?
- ODCR / capacity guarantee in place (if constrained)?
- Runbook for manual scaling exists?
- On-call staffed and aware of the event?
- Rollback / circuit breaker ready?
Post-event operations
After the event:
- Release temporary capacity — cancel short-term ODCRs, reduce ASG min, terminate extra nodes
- Measure actual vs predicted — was the forecast accurate? Over or under?
- Calculate cost — what did the event capacity cost? Was it justified by business value?
- Retrospective — what worked, what broke, what to do differently next time
- Update models — feed actual event data back into next year’s forecast