Blog
5 min read
Turning Homeschool Coaching Into a Bookable Product: International School of America
Jun 12, 2026
The most dangerous response in a distributed system is 200 OK. An error gets retried, logged, alarmed on, fixed. A success that didn't actually happen gets trusted — and everything downstream builds confidently on a write that never landed. This is a field guide to that bug class, drawn from hardening our own content-publishing pipeline: five distinct ways "success" lied to us, one verification pattern that caught all of them, and why these bugs survive in well-tested systems.
TL;DR: A write API's status code reports that the request succeeded, not that the state changed. Under concurrency, every layer between accepting a write and durably serving it can drop the payload while still returning success. The fix that actually worked: after every write, read back what the consumer would read — content, not status — and retry in a bounded loop until the read matches intent.
The system, abstractly
The shape will be familiar to anyone who's built content infrastructure: authors (human and AI) write through an API into a versioned content store; a sync layer watches for changes and materializes them into the serving layer that readers actually hit. Multiple writers, sometimes concurrent. Each hop acknowledges the previous one. Every component is individually reasonable.
The API returns success when the first hop accepts the write. And that's the whole story of this bug class: acceptance was the only thing being reported, while durability was the thing being assumed.
Five ways "success" lied
1. The swallowed race
Two writers push concurrently; one lands, the other is rejected by the version store as a conflicting update — correctly! — but the layer in between caught the rejection, logged it nowhere useful, and returned success anyway. The losing write existed only in a local working copy. Symptom at a distance: content that "published" but never appeared, with nothing in the logs but green.
2. The no-op that passes its own check
We added a verification: after pushing, confirm the store's head matches the version the push reported. Then we found the write that had nothing to commit — a prior step had quietly clobbered the staged change — so the push "succeeded" by doing nothing and reported the current head. Which our check compared to the current head. A verification that the system agrees with itself is not a verification that your change is in it.
3. The empty write under contention
Under storage contention, a commit was created whose file list was empty — the staging step lost its files between accept and commit. Every identifier was real: a commit existed, it had a hash, the API returned it proudly. The downstream change-event fired with an empty change set, so the sync layer correctly did nothing, forever. The write had become a ghost with valid papers.
4. The partial downstream
A multi-artifact write (think: a document, its metadata, its index entry) landed partially — some artifacts in the commit, some lost. The sync layer processed what arrived and produced a half-materialized record in the serving layer: present, but broken in ways that only surfaced when a reader hit it. Worse than absence, because it passed existence checks.
5. The dedup that was too clever
The sync layer deduplicated by content hash — sane, until a logically meaningful write produced byte-identical content at the artifact it hashed (the change lived in a sibling artifact). The sync skipped the whole unit as "unchanged." Every layer behaved as designed; the design just disagreed with our intent at one seam.
The fix: verify at the read path
We tried verifying at each layer and kept getting outwitted — every layer's self-report had a blind spot (that's why the bugs existed). The fix that ended the class was embarrassingly direct:
After every write, fetch each artifact the way a consumer would — raw, from the authoritative store — and compare content to what you intended to write. Not the status code. Not the version hash. The bytes. If they don't match, re-stage and re-push, bounded retries, loud failure at the bound.
Content read-back defeats all five liars at once: the swallowed race (your bytes aren't there), the no-op (your change isn't in the head it reports), the empty commit (the artifact 404s), the partial write (one of N artifacts mismatches — so check per artifact, never just one), and the clever dedup (the serving layer's record doesn't reflect your intent, which a final end-to-end check catches). The cost is one read per artifact per write — milliseconds, against failure modes that each cost us hours of archaeology. The asymmetry is the entire argument: verification is cheap precisely because it's mechanical; silent failure is expensive precisely because it isn't.
Patterns worth stealing
- Write → read-back → bounded retry as the default shape for any write you'd be sad to lose. Make retries idempotent (re-staging the same content must be safe) so the loop can't make things worse.
- Verify intent, not agreement. Any check of the form "does the system match the system?" can pass vacuously. The comparison must include something only the writer knows: the content it meant to write.
- Check the final state a consumer sees, at least for the last hop — the intermediate layers can all be green while the serving layer is wrong.
- Learn each failure's tell. Ours each had a signature once we knew to look — the no-op returns a head identical to pre-write; the empty commit fires a change event with no files; the truncated artifact has a malformed tail. Documenting the tells turned future incidents from mysteries into lookups.
- Make no-ops loud. "Nothing to commit" was the root enabler twice. A write API that did no work should say so in a way the caller can't mistake for success — a distinct status, not a shared one.
Why these bugs survive good engineering
None of this was sloppy code — and that's the uncomfortable lesson. Each bug required concurrency, contention, or a precise interleaving to trigger; each returned success, so no alarm fired; and they hid behind each other, the second emerging only after we'd fixed the first. Tests passed because tests run one writer at a time on a quiet machine — the exact conditions under which this class is unreproducible. The honest conclusion we took away: for distributed writes, you don't test your way out of this class; you verify your way out, in production, on every write, forever. The verification loop isn't scaffolding you remove when things stabilize. It's the part of the system that makes "success" mean something.
Key takeaways
- Status codes report acceptance, not durability: every layer between accepting a write and serving it can drop the payload while returning success.
- Five liars, one family: swallowed races, self-agreeing no-ops, empty writes under contention, partial multi-artifact landings, and over-clever dedup — all green, all wrong.
- Verify intent at the read path: fetch each artifact as a consumer would and compare bytes to what you meant to write — per artifact, not per batch.
- Bounded, idempotent retry loops: write, read back, retry, and fail loudly at the bound — re-staging must be safe for the loop to be safe.
- Make no-ops unmistakable: "did no work" must not share a status with "did the work" — that one design choice enabled half this list.
- You verify your way out, not test your way out: the trigger conditions don't exist in test environments — the read-back loop is permanent infrastructure, not scaffolding.
Frequently asked questions
Isn't read-after-write verification expensive at scale?
It's one read per artifact per write, against your own store — for content pipelines, that's milliseconds on an operation humans initiate a few hundred times a day. If you're doing millions of writes a second, you'd reach for different machinery (write-ahead acknowledgment, quorum reads). For the very large class of systems shaped like ours — important writes at human scale — the read-back is rounding error, and the engineering time it saves is not.
Why not fix the layers instead of verifying around them?
We did both — each root cause got fixed. But the verification stayed, for an epistemic reason: every one of these bugs was in code that had been reviewed and believed correct, which means our belief in the next layer's correctness is exactly as good as it was before the first incident. The read-back is how the system stays honest about the gap between believed-correct and verified-correct. Fixes shrink the class; verification bounds it.
How do you handle the verification itself failing — false alarms?
Distinguish "mismatch" from "couldn't check": a read that errors or times out is an unknown, not a failure — retry the read before retrying the write. For mismatches, the bounded loop absorbs transients (replication lag, eventual consistency) by re-checking before re-writing. In practice, our false-alarm rate after tuning the read path was effectively zero — the authoritative store either has your bytes or it doesn't, and that question has a crisp answer.
What's the minimum version of this for a small team?
One function: writeAndVerify(artifacts) that wraps your existing write, fetches each artifact back, compares, retries twice, and throws loudly. An afternoon of work, and every future write inherits it. The mistake to avoid is the partial adoption — verifying only the writes that have burned you already. The class is bigger than your incident history; that's the one prediction from this whole experience we'd bet on.
We build Faster so small businesses get infrastructure with these lessons already baked in — including the publishing pipeline this post is quietly about. More engineering notes as we earn them: the engineering blog.