Siddhartha Bariker
---
The SSD That Cried Etcd
A homelab debugging story about why marketing IOPS lie and etcd will absolutely tell you about it
August 2, 2026
tech, homelab, kubernetes
---
TL;DR: one flaky node in my k3s cluster turned into an all-day debugging adventure that ended with `fio` teaching me that SSDs have a lot of feelings about how you write to them.
The setup
Three-node k3s cluster. Two nodes are physical machines with Samsung 870 SATA SSDs. The third node lives as a VM on a Proxmox host, with its virtual disk on an LVM-thin pool backed by a WD Red SA500. All consumer SATA SSDs, all “fast enough,” right?
Right?
The symptom
I asked a Tekton pipeline to build an app. It failed. I asked again. It failed. I looked at the Tekton dashboard to see why. The dashboard was slow.
Slow how? Every API call was making a 300 ms round-trip through my auth proxy. Fine, that’s known. But underneath, the actual API server was answering in five whole seconds. Not fine.
The rabbit hole
Peeling the onion, in order:
1. Pipelines failing to push to Harbor with `500 internal server error` 2. Harbor DB (Postgres) crashlooping with `Input/output error` on its own config file 3. `fsck` the Longhorn volume, clean, back it goes, minutes later same corruption 4. The volume’s replicas keep dying on the VM node, but the running replicas are on the healthy physical nodes. Why is that killing them too? 5. Longhorn replication to the VM node is crawling. Volumes that should sync in a couple of minutes are still going after twenty. Rebuild progress bars barely move. That’s the moment I stop suspecting Kubernetes and start suspecting the disk under the VM. 6. Kernel is filling `dmesg` with `aborted journal` and `critical medium error` on the VM’s iSCSI targets 7. The VM’s load average is 35 on 12 cores
At this point I’ve gone from “why is my build failing” to “why is my whole node crying.”
Me, five hours into `kubectl describe`, convinced I was one grep away from cracking the case. (HIMYM S6E2, where Robin says this about Ted before Ted has done anything remotely detective-like. Yes. That’s the energy.)
The reveal
I fired up `fio` for a workload matching what etcd actually does: 4 KiB random writes, queue depth 1, with `fsync` after every one. This is the “hard” workload nobody puts on a datasheet.
Here’s what I actually measured across the drives in my cluster:
| Drive | Environment | Marketed 4K IOPS | QD1 fsync | QD32 no-sync | | --------------------- | ----------------------------- | ---------------- | ------------- | ------------ | | Samsung 870 EVO 500GB | Physical, bare-metal | 98,000 | 248 IOPS | 19,200 IOPS | | Samsung 870 EVO 500GB | Physical, bare-metal | 98,000 | 311 IOPS | 27,400 IOPS | | WD Red SA500 1TB | Proxmox VM, virtio + LVM-thin | 85,000 | 134 IOPS | 6,076 IOPS |
The marketing number is real. The drive can do that under specific lab conditions. But it’s measured at queue depth 32 with no sync forced. That’s a benchmark that shows the drive’s theoretical throughput when it can batch and reorder to its heart’s content. It’s the number that fits on a box.
But when your database says “put this on real disk right now, I’ll wait,” the SSD can’t batch. It can’t cache. If it doesn’t have power-loss protection (PLP), it has to actually push the bits into NAND before returning. Consumer SSDs, including the WD Red NAS-branded ones, do that at 100 to 500 IOPS.
Etcd’s docs say it needs 500+ IOPS for sync writes. The “fast enough” NAS SSD sitting in my Proxmox host was, by the numbers, not fast enough to be an etcd master. The 85,000 IOPS on the box was measured at a queue depth and sync mode nothing like what etcd actually asks of a disk.
The compounding factor
The pool that SSD sat in was also shared with another VM doing its own heavy sync-write workload. Two `fsync`\-happy processes queuing behind SSDs with no DRAM cache is a bad time. When Longhorn started rebuilding replicas on top of that, the queue exploded, latencies climbed to 89 ms, and everyone downstream started returning EIO to each other in a chain of increasingly grumpy processes.
The fix
The final bug was simple: the SSD was too slow for the job I was asking it to do.
The fix was to stop asking it. Load rebalanced, cluster went green, all the “mysterious” errors upstream (Harbor 500s, corrupt Postgres configs, failing pipelines) evaporated as soon as the underlying disk stopped saying “I’ll get to it in 89 ms.”
What I learned
SSD performance is a workload story, not a spec
The number on the box is the drive’s best day: big blocks, deep queue, no forced sync. The workload that decides whether a database, etcd, or a distributed storage engine feels snappy or dies is the exact opposite: tiny blocks, queue depth one, sync forced on every write. Nobody prints that number on the box, so you have to measure it yourself. Ten seconds of `fio` will tell you more than an hour of reading datasheets:
Unsupported language.If that number is below 500 IOPS, the drive is not a good home for anything stateful. Below 200, it’s a good home for cold storage and not much else.
Power-loss protection is the difference between consumer and enterprise SSDs
PLP means the drive has onboard capacitors that guarantee an in-flight write actually reaches NAND even if the power drops mid-write. Because the drive can promise that, it’s allowed to acknowledge `fsync` the moment the write hits its DRAM cache. That’s how enterprise SSDs get to 20,000+ QD1 fsync IOPS on the same NAND that a consumer drive struggles to push past 500.
Consumer drives can’t cheat, so they can’t be fast. That’s not a bug, it’s the whole design. If you need fsync throughput, you need PLP.
The BX500s that made this whole story possible are the third panel. Don’t be the third panel.
Every database is really a bet on fsync latency
Postgres, MySQL, SQLite, etcd, Kafka, all of them make the same trade: they promise durability by calling `fsync` at commit time, and they pay for it in tail latency. On a good drive, a commit takes half a millisecond. On a slow one, it takes 90. A “slow database” investigation almost always ends at fsync latency, not at query planners or index tuning. If your p99 write latency is bad, look at the disk before you look at the query.
etcd is picky for a specific reason
Etcd’s docs recommend a disk that can do 50 sequential IOPS with fsync latency under 10 ms consistently, and 500+ IOPS under any real workload. That’s not because etcd writes a lot, it’s because everything in Kubernetes writes through etcd on the hot path: kubelet heartbeats, leader election leases, controller work queues, scheduler decisions. When etcd slows down, the whole cluster starts feeling sick in ways that don’t obviously point at etcd. Pods stuck pending, nodes flipping NotReady, “why is my Tekton dashboard taking five seconds,” corrupted-looking database pods. All symptoms of one starving etcd.
Distributed replicated storage amplifies slow disks, it doesn’t hide them
I used to think Longhorn (or Ceph, or any n-way replication system) would mask a slow node the way a load balancer masks a slow server. It doesn’t. A synchronous replicated write has to be acknowledged by every replica before the client sees success. If one replica lives on a slow disk, that replica’s latency becomes the whole volume’s latency. The distributed layer only makes you resilient to failure, not to slowness. Slow is worse than dead here, because dead gets you failed over out of the write path and slow just sits there dragging.
Related, on Longhorn specifically:
- Replica rebuilds are sync-heavy and will hammer the target node’s disk. There’s a knob for it (`concurrent-replica-rebuild-per-node-limit`) and it saved me during recovery.
- Longhorn stores each replica as a raw file on the node’s filesystem, and exposes the volume over iSCSI. So when the underlying disk is slow, you don’t get “slow volume” errors, you get kernel `critical medium error` and `aborted journal` messages, because the block layer decides the iSCSI target has stopped talking. Very confusing the first time.
- Keep running consumer drives where they fit (bulk storage, media, backups, worker nodes).
- When I need real fsync performance, watch the used market for a proper enterprise SSD instead of buying new.
- Benchmark every drive that comes in with the `fio` command above before trusting it with anything stateful.
- Accept that some nodes will end up as compute-only, and that’s OK.
Kubernetes doesn’t warn you when your disk is the bottleneck
It’ll happily tell you a pod is CrashLoopBackOff, or a PVC is stuck attaching, or a node is NotReady. None of those diagnostics will say “your disk is too slow.” You have to look at kernel `dmesg` on the node, or `iostat`, or the load average, to see what’s actually going on. A load average significantly higher than CPU count with low actual CPU use is almost always the disk. That was my one honest smoking gun in the whole investigation.
You can’t fsck your way out of hardware slowness
I ran `fsck` on the corrupted Postgres volume, it came back clean, Postgres started, and minutes later it corrupted again on the same file. That’s a hint that the corruption isn’t the bug, it’s the symptom of an ongoing storage problem. If a filesystem corrupts, you fix it once; if it re-corrupts, stop fixing symptoms and go find whatever’s timing out underneath.
Virtualization tax on IOPS is real
The same disk technology on bare metal versus inside a VM via virtio plus LVM-thin lost roughly half its QD1-fsync performance in my measurements, even before any contention. If you’re running stateful workloads inside a VM, budget for that tax. It’s not free.
RAID 0 across two storage pools ties their fates together
Every write to the RAID device stripes across both pools, so a slow drive in one pool pulls the other one down. Fine for expendable bulk storage; risky when the array holds anything the rest of your cluster is waiting on.
Am I switching to enterprise SSDs?
Realistically, no. Between homelab budget and current storage prices, brand-new PLP-equipped enterprise SSDs are hard to justify for a hobby cluster.
I also have a personal rule for the homelab: buy used whenever possible. Facebook Marketplace, eBay, decommissioned data-center gear, that guy on Reddit selling a lot of ten drives from a retired server. That’s where my hardware comes from. It’s cheap, it keeps working parts out of e-waste, and it lets me actually afford the right class of hardware for a workload when I do go looking for one. A used enterprise SSD with 90 % life left often costs less than a new consumer SSD of the same capacity, and it will absolutely destroy it at 4k QD1 fsync.
So the plan is:
I’ll keep this table updated as I add new drives or try different brands. If you’ve got a consumer SSD you’re happy with for sync-heavy workloads, or a good used-enterprise deal to share, I’d love to hear about it. Bring the `fio` numbers.
If your homelab is choking, benchmark 4k QD1 fsync before you look at anything else. The disk knows.
“Wasn’t it naive to put consumer SSDs on a production-shaped workload?”
If that thought crossed your mind reading this: yeah, fair. But that’s kind of the whole point.
I run this homelab for fun. Cheap, off-the-shelf, mostly used hardware, doing things it was never designed for. The goal isn’t to build a mini-AWS. The goal is to break things, chase them down, and see how far the tape holds. Every stupid decision I make on a Saturday night is a mistake I don’t have to make on a Tuesday morning at work.
Over the years this has quietly paid for itself many times over in my day job. When you’re running things at real scale, most bugs don’t come from bad code. They come from the physics of the underlying system doing something you didn’t expect. Disk fsync latency spiking under load. Networks getting reordered. Caches lying. Filesystems corrupting quietly and only telling you three hops downstream. You don’t learn that from a book. You learn it by having a Saturday night where your homelab kills itself, then spending the next five hours figuring out why.
The muscle you build isn’t “how to fix this specific bug.” It’s the intuition: where to sniff first, what looks off, what’s a red herring, what’s the actual smoking gun. Faster debugging, better instinct for what could go wrong before it does, deeper understanding of how the whole stack fits together.
Knowing what not to do is at least as valuable as knowing what to do, and the only reliable way to build that catalog is to do the wrong thing occasionally, on purpose or by accident, in a place where nobody’s paycheck depends on it.
And this matters more now than it ever has. In the age of AI, writing code is cheap. Anyone can generate a working-looking prototype in an afternoon. What isn’t cheap is figuring out why the AI-generated thing falls over at 2 a.m. under real load, or why the “obviously correct” migration script corrupts one row in a million, or why a healthy-looking pod is quietly returning EIO to its downstream. AI can write the code. It’s on us to have the intuition for when the code is lying to us, and the debugging muscle to catch it before customers do. That muscle isn’t AI-generatable. It’s earned, hour by hour, one confusing bug at a time.
So yeah, consumer SSDs into a k3s etcd master was a bad call. But it was my bad call, in my lab, and it taught me more about storage than any datasheet ever will.
Homelabs aren’t miniature production. They’re gyms.