Stop over-provisioning cloud block storage on guesswork: watch queue depth, p99 latency, and burst balance to tell whether you're IOPS-, throughput-, or capacity-bound, then scale only the knob that's actually throttling with the cost math to prove it.
The latest PostgreSQL beta shows how the database is becoming easier to tune, observe and scale without abandoning its relational core.
Every cloud bill has the same line item everyone pretends not to see: over-provisioned block storage. We buy 3x the IOPS "to be safe," set 2x the volume size "for growth," and then get paged at 2 AM anyway not because the disk was full, but because we were throttled on a burst credit we didn't know existed.
I work on telemetry infrastructure and stateful orchestration for large cloud-native environments. The pattern I see is consistent: storage decisions are made at provisioning time with almost no production feedback. Observability exists, but it lives in a different team's dashboard. This essay is about closing that gap making storage a runtime decision driven by the same telemetry you already collect.
Cloud block storage isn't a disk. It's a networked service with its own control plane, and it fails in ways local disks don't:
1. Burst credits, not steady state. gp3/EBS-style volumes give you baseline IOPS plus a burst bucket. Your p99 looks fine for weeks, then the bucket empties during a compaction and latency cliffs. 2. Throughput vs IOPS vs size are independent knobs. Teams size for capacity (GB) and get throttled on throughput (MB/s) or vice versa. The bottleneck is rarely the knob you tuned. 3. Noisy neighbors are real, but invisible. Multi-tenant storage means your tail latency includes someone else's GC pause. Without per-volume queue-depth metrics, you'll blame the app.
The fix isn't a bigger volume. It's knowing which of the three limits you're actually hitting, and having the telemetry to prove it before you pay for more.
Forget the 40 metrics your CSI driver exports. For block storage decisions, I watch three:
node_disk_io_time_weighted or cloud VolumeQueueLength): sustained queue > 1 per 1k IOPS of provisioned performance means you're throttling.BurstBalance, GCP has throttled_read_ops_count. If you don't alert on these, you're flying blind.A minimal Prometheus setup that actually answers the question:
# prometheus-rules.yaml — storage throttle alerts that page on cause, not symptom
groups:
- name: storage
rules:
- alert: VolumeBurstExhaustionRisk
expr: |
aws_ebs_burst_balance_percent < 20
and
rate(node_disk_reads_completed_total[5m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Volume {{ $labels.volume }} burst balance below 20%"
- alert: VolumeQueueSaturation
expr: |
rate(node_disk_io_time_seconds_total[5m])
/
rate(node_disk_reads_completed_total[5m] + node_disk_writes_completed_total[5m])
> 0.008
for: 10m
labels:
severity: critical
annotations:
summary: "Volume {{ $labels.device }} queue saturated — throttling likely"These are starting points, not universal thresholds. Baseline them against your own workload for a week before paging anyone. (All numbers in this essay are illustrative from typical web-serving and database-adjacent workloads measure your own.)
When an alert fires, the question is which knob to turn. Here's the decision tree I use:
def recommend_storage_change(metrics) -> dict:
"""
metrics: dict with p99_read_latency_ms, p99_write_latency_ms,
queue_depth, burst_balance_pct, throughput_util_pct,
iops_util_pct, size_util_pct
Returns a recommendation with reasoning. Illustrative thresholds.
"""
if metrics["burst_balance_pct"] < 20 and metrics["queue_depth"] > 1.0:
return {
"action": "increase_baseline_iops",
"reason": "Burst exhausted and queue building — baseline IOPS too low",
}
if metrics["throughput_util_pct"] > 85 and metrics["iops_util_pct"] < 60:
return {
"action": "increase_throughput_not_iops",
"reason": "Throughput-bound (large sequential IO) — don't pay for IOPS you won't use",
}
if metrics["size_util_pct"] > 80:
return {
"action": "expand_volume",
"reason": "Capacity-bound — but check if it's logs/snapshots before paying",
}
if metrics["p99_read_latency_ms"] > 20 and metrics["queue_depth"] < 0.5:
return {
"action": "investigate_not_scale",
"reason": "High latency without queue — likely noisy neighbor or AZ issue, not your volume",
}
return {"action": "no_change", "reason": "No clear bottleneck — keep observing"}The last branch is the most valuable. In my experience, roughly a third of "slow disk" pages are not the disk. Scaling the volume in those cases just increases the bill and hides the real problem for another month.
Cloud storage pricing punishes guessing. A quick break-even sketch I run before any scale-up:
# Illustrative cost check — plug in your provider's actual rates
def monthly_cost_provisioned(iops, throughput_mbps, size_gb,
iops_rate=0.005, tput_rate=0.04, gb_rate=0.08):
return iops * iops_rate + throughput_mbps * tput_rate + size_gb * gb_rate
before = monthly_cost_provisioned(iops=16000, throughput_mbps=1000, size_gb=2000)
after = monthly_cost_provisioned(iops=8000, throughput_mbps=1000, size_gb=2000)
print(f"Before: ${before:.2f}/mo After: ${after:.2f}/mo "
f"Saved: ${before - after:.2f}/mo")
# Before: $280.00/mo After: $240.00/mo Saved: $40.00/mo per volume
# At 200 volumes, that's $8k/mo — for a config change, not an architecture change.These rates are illustrative - use your actual price list. The point is that right-sizing IOPS independently of size (which gp3 and equivalents allow) is often a 10-15% saving per volume with zero performance impact, if your telemetry shows you're not IOPS-bound.
One more cost trap: snapshots. I've seen teams with 30% of their storage bill in forgotten snapshots of deleted volumes. A monthly job that lists unattached snapshots older than 30 days and tags the owner pays for itself immediately.
# Find expensive forgetfulness: unattached EBS snapshots (illustrative)
aws ec2 describe-snapshots --owner-ids self \
--query 'Snapshots[?StartTime<`2026-09-01`].[SnapshotId,VolumeSize,StartTime]' \
--output table
# Pair with Cost Explorer's EBS snapshot line item before deleting anything.The cultural fix matters more than the PromQL. I treat storage configuration like any other infrastructure decision:
Checklist if you're starting from zero:
This isn't "just use local NVMe" or "just move to object storage." Those are valid architectures for specific workloads, but most teams are running stateful workloads on networked block storage today and need it to behave. This is about operating what you have with evidence instead of superstition.
It also isn't a replacement for database-level tuning. If your Postgres is doing full sequential scans because of a missing index, no amount of provisioned IOPS will save you and your storage dashboard will tell you the volume is idle while the app is slow. That's the signal to look up the stack, not scale the disk.
The durable lesson: storage is a systems problem, not a shopping problem. The teams that spend the least on it are the ones that watch it the closest.
node_exporter disk stats and cloud provider CloudWatch/Cloud Monitoring volume metrics
Comments0