Spot Instances
Spot instances (AWS), preemptible VMs (GCP), and Spot VMs (Azure) use spare cloud provider capacity at steep discounts (typically 60–90% off on-demand) in exchange for the provider being able to reclaim them when capacity is needed.
How interruption works
| Provider | Notice period | Signal |
|---|---|---|
| AWS Spot | 2 minutes before termination | Instance metadata event + EventBridge |
| GCP Preemptible | 30 seconds (ACPI G2 soft-off) | Preemption notice in metadata |
| GCP Spot VM | 30 seconds | Same as preemptible, but no 24-hour max lifetime |
| Azure Spot | 30 seconds (configurable eviction policy: deallocate or delete) | Scheduled Events metadata |
AWS also emits an EC2 Spot interruption warning via EventBridge two minutes before termination (except for hibernation, which begins immediately).
Pricing model
Spot pricing is market-driven — the price fluctuates based on spare capacity in each AZ for each instance type. Key properties:
- You set a max price (optional on AWS; defaults to on-demand price)
- If spot price exceeds your max, instance is interrupted
- In practice, AWS Spot prices have been relatively stable since 2017 (they moved away from the volatile auction model)
- GCP preemptible/Spot VMs have fixed discount (~60–91%) without bidding
When Spot works and when it doesn’t
The rule is simple: workloads must be interruptible and retryable. Batch jobs, CI/CD, queue consumers, ML training with checkpointing, and stateless web workers behind load balancers are natural fits. Singletons, stateful services without graceful shutdown, and anything where 2-minute drain exceeds the SLO are not.
The non-obvious case: online serving on Spot. It works if you have sufficient replica count (N+2 minimum), graceful connection draining, health-check-based routing, and the service can tolerate brief capacity dips. The savings are large enough (60–90%) that many organizations run a fraction of their serving fleet on Spot with on-demand fallback.
Best practices for reliability
Diversify across instance types and AZs
The #1 Spot best practice. Spot capacity shortage in one type/AZ doesn’t affect others. Use allocation strategies that maximize diversity:
- AWS:
capacity-optimizedorprice-capacity-optimizedstrategies in EC2 Fleet / ASG (prefer pools with most available capacity) - GCP: use multiple machine types and zones in Managed Instance Groups
- Azure: use multiple VM sizes in Scale Sets
Graceful interruption handling
- Monitor the interruption signal (instance metadata endpoint)
- Drain connections / stop accepting new work
- Checkpoint state if applicable
- Signal orchestrator that instance is going away
Fallback to on-demand
Configure Auto Scaling Groups / fleets with mixed instance policies:
Base capacity: On-demand (guaranteed baseline)
Scaling capacity: Spot (cheap burst)
Fallback: On-demand if Spot unavailable
This hybrid approach gets Spot savings during normal operation while ensuring capacity during Spot shortages.
Over-provision slightly
Since any instance can disappear with 2-minute notice, maintain slightly more capacity than needed (e.g., N+1 or N+2). The per-instance cost is so low that mild over-provisioning is still cheaper than on-demand.
Spot + commitment interaction
Important
Savings Plans and Regional RIs do not apply to Spot instances. Spot has its own (already discounted) pricing. If a workload moves from on-demand to Spot, any Savings Plans that were covering it become unused (reducing SP utilization).
This means Spot adoption must be coordinated with commitment strategy:
- Move a workload to Spot → ensure something else consumes the freed SP coverage
- Or reduce SP commitment at next renewal
Spot in Kubernetes
Karpenter (AWS)
Karpenter natively supports Spot provisioning:
- Configurable instance type diversification
- Automatic fallback to on-demand when Spot unavailable
- Consolidation: replaces expensive on-demand nodes with Spot when available
- Interruption handling: cordons and drains nodes on Spot termination notice
Cluster Autoscaler
Supports Spot via mixed ASG configurations. Less sophisticated than Karpenter for Spot-specific optimization.
Pod disruption budgets (PDBs)
Set PDBs to control how many pods can be interrupted simultaneously. Prevents Spot termination from taking down too many replicas at once.
Metrics to track
| Metric | What it tells you |
|---|---|
| Spot savings | on-demand equivalent cost - actual Spot cost |
| Interruption frequency | how often instances are reclaimed (per pool) |
| Fallback rate | how often Spot-eligible workloads run on on-demand instead |
| Spot capacity score | AWS-provided signal of Spot availability per pool |
| Time to replacement | how quickly interrupted instances are replaced |