Same Database. Same Queries. Different Universe.
Storage Series — Part 3
We have talked about the wire. We have talked about the architecture. Now we make it concrete.
PostgreSQL. A production schema. Real mixed workload — reads, writes, concurrent connections, background autovacuum, index scans running alongside bulk inserts. The kind of load that does not show up in synthetic benchmarks but does show up in your application at 9am on a Monday when everyone logs in at once.
Two environments. One running on EBS gp3, the standard choice for most AWS-hosted databases. One running on a Dell PowerEdge R750, four Samsung 9100 PRO NVMe drives in ZFS RAID-10, 128GB ECC RAM with ARC configured to use 64GB as hot read cache.
Same PostgreSQL version. Same configuration. Same data. Different storage.
What EBS Gives You
A 16TB gp3 volume provisioned with 16,000 IOPS — the maximum — and 1,000 MB/s throughput, which is also the maximum. You are at the ceiling before the workload starts. Single-digit millisecond latency on a good day, climbing under contention. Consistent until it is not, and when it is not, your queries feel it immediately.
Cost: $1,345/month. Every month.
What Local NVMe Gives You
IO that never leaves the machine. PCIe 5.0 latency measured in microseconds, not milliseconds. ZFS ARC serving hot pages directly from RAM — for a well-tuned PostgreSQL workload the majority of your most frequent queries never touch the drives at all. When they do touch the drives, those drives are doing 4 million IOPS across the RAID-10 array.
There is no ceiling to provision up to. There is no throughput limit to purchase. There is no Monday morning contention spike because the storage tier cannot keep up with demand.
The Latency Gap
A typical OLTP query hitting a cached page on EBS: 1–3ms storage latency. The same query served from ZFS ARC: under 100 microseconds. An order of magnitude, on the queries that matter most — the ones your application makes thousands of times per minute.
For queries that do hit disk: EBS under mixed load sits at 500 microseconds to 2ms depending on queue depth and contention. Local NVMe RAID-10 under the same load: 50–150 microseconds. Consistently. Without provisioning anything.
That gap is not a benchmark curiosity. It is the difference between a dashboard that loads in 200ms and one that loads in 800ms. Users do not read storage benchmarks. They feel 600 milliseconds. And they remember it.
PostgreSQL Specifically
PostgreSQL rewards fast local storage more than almost any other database. Its shared buffer cache works alongside ZFS ARC to create two layers of intelligent caching. WAL writes — the sequential IO path that every transaction touches — are blazing fast on local NVMe, which directly improves write throughput and replication lag. Autovacuum, which runs continuously in the background on busy tables, stops competing with your query workload for IO bandwidth because there is enough bandwidth for both.
A PostgreSQL instance that struggles on EBS under load — query latency climbing, autovacuum falling behind, checkpoint warnings in the logs — frequently runs without issue on local NVMe. Not because the database changed. Because the storage stopped being the bottleneck.
The Honest Caveat
Local NVMe wins decisively on latency, throughput, and cost at this scale. EBS wins on operational simplicity — snapshots are one API call, volume resizing is online, and you do not manage the hardware. Those are real advantages for teams without infrastructure expertise.
But if you have that expertise, or are building it, the performance and cost argument is not close. You are not trading a little convenience for a little savings. You are trading a managed abstraction for a storage platform that is faster by an order of magnitude and pays for itself in six months.
The wire costs $1,345 every month. Cutting it is a one-time decision.
Next and final: when bare-metal storage makes sense and when it does not — the honest decision framework for CTOs who want the right answer, not the loudest opinion.