# 33 · Production Incidents, Root-Caused

**Rank 7 of 34** · Tier A · `app/GlobalChatService/service/kafka.go`, `app/AuthGateway`, `app/BillboardML/billboardml/model/FINGERPRINT`, `app/workers/billboard_users`, `app/web-platform/app/boxoffice/sagas`, `Portfolio/DesignByDuhart/site/scripts/verify.mjs`
**Stack:** Go (sarama, gocql), Kafka, Cassandra, Redis, Cloudflare Workers + Hyperdrive, Python (numpy memory maps), IndexedDB, systemd, the Linux OOM killer

> **Where this sits.** The other case studies describe systems. This one
> describes what happened when they broke. Seven incidents from the last six
> weeks, each written the same way: the symptom, the theory everyone reached for
> first, the evidence that actually decided it, the fix, and the rule it left
> behind. In most of them the first theory was wrong. That is the point of
> writing them down.

---

## The pattern across all seven

| # | Symptom | First theory | What it was |
|---|---|---|---|
| 1 | A Kafka handler's retries never fired | "It's retrying, just slowly" | The consumer loop skipped failed offsets forever |
| 2 | Users logged out with "Invalid refresh token" | A restart invalidated the tokens | A rotation response lost in a tunnel drop, replayed past the leeway |
| 3 | Every login failed for sixteen minutes | Broken web client, then a bad deploy | Cassandra stalled; the gateway stayed "active" and its logs went silent |
| 4 | Tunnel timeouts on four edge IPs at once | Cloudflare, DNS, the network | A thrashing box that could not answer keepalives |
| 5 | The recognition API served garbage after a rebuild | A bad model, bad data | The index was rewritten in place under a live memory map |
| 6 | One upload hung for fifty minutes, deterministically | Network, CORS, the file | A stale IndexedDB Blob that sends nothing and raises nothing |
| 7 | A build gate reported PASS for three days | Nothing. It was green. | It was scanning a directory the build had stopped writing |

Six of the seven share a shape: **the failing component stayed quiet.** No
crash, no error, no alert. Silence was the symptom.

## 1. The Kafka retry that could never happen

**Symptom.** A new consumer in GlobalChat logged `attempt 2 of 5` once, and
then nothing. Ever.

**First theory.** The handler returns an error, the offset stays uncommitted,
Kafka redelivers. I had written exactly that into a comment and a commit
message.

**Evidence.** Read the loop. `ConsumeClaim` logged the handler error and called
`continue`. That did not mark the failed message, which looks like "leave it for
redelivery". But the loop also moved on, and the next message that succeeded
called `MarkMessage` on a **higher offset**. Committing offset N+1 commits
everything below it. The failed message was skipped permanently. The "attempt 2"
in the log had come from a process restart, not a redelivery. The match bridge's
bounded retry and dead-letter path could only ever reach attempt 1.

**Fix.** `consumeOne` now retries in place: six attempts, backoff from 500 ms
doubling to a 4 s cap, chosen so a handler's own five-attempt budget completes
inside one session. Then it marks and moves on. A session that ends mid-retry
returns **without** marking, because a rebalance genuinely does redeliver.

**Rule.** "Returning an error leaves the offset uncommitted" is a claim about
your consumer loop, not a Kafka guarantee. Offsets are a high-water mark.

## 2. The refresh token that was "reused"

**Symptom.** Users bounced to a generic 401, "Invalid refresh token", after an
outage. A spec was written blaming the restart for invalidating tokens, and
proposing a Postgres token store and key stabilisation to fix it.

**Evidence.** The logs disproved every premise. Refresh tokens live in Redis
with AOF on. JWT signing keys load from files and are never generated at boot. A
60 second `RefreshReuseLeeway` already existed, with its own test. The real
chain: memory pressure, a thrashing box, a hard reboot. Minutes before it, a
`RefreshTokens` call **succeeded server-side** while the tunnel died. The
browser never received the new token. Twenty minutes later it replayed the old
one, far past the leeway, and reuse detection did exactly what it is designed to
do: revoke the whole session chain.

**Fix.** The frontend stopped hanging on it. The real bug there was a gate,
`!bootstrapped || !currentUser`, that rendered nothing forever after bootstrap
finished signed out. Now an expired session redirects to login with a reason in
under a second, with a 10 s bootstrap watchdog and a 4 s auth RPC timeout. The
backend change is named for what it is: a **reuse policy decision** (treat a
replay of R1 as a lost response while R2 is still unused), not a storage
project.

**Rule.** Any outage longer than the leeway that swallows a rotation response
kills that session, and no client retry can fix it. Rotation plus reuse
detection has a failure mode you must decide on, not discover.

## 3. The gateway that was "active" and silent

**Symptom.** Every web login failed for sixteen minutes. The site itself
served 200 throughout.

**First theory.** The web client. Then a deploy.

**Evidence.** AuthGateway's identity store is Cassandra via gocql. When
Cassandra stopped answering, the process stayed `active`. Nothing exited,
systemd restarted nothing, and the structured log went **completely quiet**.
Only the error log showed `gocql: no response received from cassandra within
timeout period`. A restart then crash-looped on `unable to create session` until
Cassandra came back.

**Fix.** A triage order, written down and reused on the recurrence two weeks
later: count gocql errors in the error log since the last boot line; read the
hourly synthetic logins as a success monitor, where a gap in cadence *is* the
outage window; only then open the browser. The recurrence was blamed on a feed
deploy. Checking the deployment that was actually serving, not the branch head,
showed that deploy had never gone live.

**Rule.** A liveness check that only asks "is the process up?" will report a
wedged dependency as healthy. Silence in a structured log is a signal.

## 4. The network outage that was memory

**Symptom.** `cloudflared` reported no recent network activity on all four edge
connections and DNS lookups timing out. Production login returned 530, then on
another night a 1033. Chat, match and upload all logged connection errors in the
same window.

**First theory.** Cloudflare. DNS. The tunnel.

**Evidence.** Four independent edge IPs failing at the same instant means the
common factor is local. `journalctl -k` showed the OOM killer. The box carries a
standing load of roughly half its memory (Cassandra alone is about 7.6 GB) and
two concurrent builds tipped it. A box that is thrashing cannot service UDP
keepalives or resolve a name, so it presents as a network failure.

**Fix.** Serialise heavy builds across sessions, cap Node heaps, run batch jobs
under `systemd-run` scopes with hard memory limits, and protect the datastore
and auth gateway with OOM score adjustments so a batch job is reaped first.

**Rule.** When several unrelated services fail at once, check `free` and the
kernel log before the network.

## 5. The index rewritten under a live reader

**Symptom.** After an index rebuild grew from 457 M to 781 M fingerprint pairs,
the running recognition API answered with nonsense until restarted.

**Evidence.** The API memory-maps the index arrays at startup and records their
lengths and bucket offsets. The builder truncated and rewrote **the same
inode**. The running process kept its old offsets over new bytes. Separately, a
single query expanded to about 9 GB of candidate rows and the OOM killer took
the API under its 3 GB cap.

**Fix.** `finalise_index()` writes `.tmp` siblings and renames them into place,
metadata last. A rename keeps the old inode alive for the running reader and the
new files appear whole. Queries are now looked up in 4,096-pair slices, which
bounds the expansion to roughly 4 M rows.

**Rule.** Never mutate a file another process has mapped. Publish by rename.

## 6. The upload Blob that sends nothing

**Symptom.** One track of an album upload never sent a byte. Twice.
Deterministic. Worker logs showed the CORS preflight succeed and then no PUT.

**Evidence.** Two stacked faults. The part upload was a bare `fetch` with no
deadline, so the hung request never errored, never retried, and held its saga
lane for fifty minutes. Underneath it: every checkpoint re-put the whole
IndexedDB row, **including the file Blobs**. A Blob the saga had already read
went stale, and a stale Blob handed to XHR sends nothing and fires no error.

**Fix.** An XHR with a 45 s idle timeout on progress, plus an opt-in stall
watchdog in the shared saga runtime. Storage v2 keeps files in their own store,
written once and never rewritten. Each part is read into memory under a deadline
before sending, and an unreadable file makes the saga ask for it again.

**Rule.** Debug "upload hangs" by looking for a preflight with no request after
it. Never rewrite Blob-holding rows on a hot path.

## 7. The gate that passed because it stopped looking

**Symptom.** None. That was the problem.

**Evidence.** The site's post-build gate scans for redaction leaks, broken
links and external requests. It read `out/`. The site stopped writing `out/`
when it moved from static export to Workers, but the stale export stayed on disk
and satisfied the gate's only sanity check, "does the directory exist". For
three days and about forty commits it reported the same counts and PASS without
reading a byte of the shipping site.

**Fix.** Repointing alone would leave the same hole one directory over. The gate
now reconciles the files it finds against the build's own prerender manifest, in
both directions, so pointing it at the wrong tree can no longer produce a PASS. A
companion script plants a leak, a broken link, an external resource and a stray
file and asserts the gate refuses each.

**Rule.** A missing gate gets noticed. A gate that passes because it stopped
looking gets trusted. Make every gate cross-check its input against something
the build emits, and watch it fail at least once.

## Two more, briefly

- **A Worker that hung forever on Postgres.** An unreachable Hyperdrive left
  `connect()` pending until the runtime cancelled the request with an error
  carrying no CORS headers, which a browser renders as "Failed to fetch". Every
  Postgres client now has a kill-switch guard, an `'error'` listener and a
  timeout, so a repeat degrades to a 503 that names its cause. Also learned:
  check for the configuration *variable*, never the comment block warning you
  not to remove it.
- **A benchmark ceiling that belonged to the harness.** A runtime comparison
  reported "80 rps, Postgres-bound" for two contenders. The real ceiling was the
  fault-injection proxy, pinned at 100% of its half-CPU budget. Every run was
  invalidated and repeated. Check the measurement path's own CPU before you
  believe any ceiling.

## Honest gaps

- **No paging.** Most of these were found by a person reading logs. There is no
  alerting on consumer lag, structured-log silence or OOM events yet.
- **The reuse-policy change in incident 2 is not shipped.** The frontend
  degrades cleanly; the backend still revokes on a late replay.
- **The memory ceiling is mitigated, not solved.** The box is still shared by
  production datastores and development work.
- **Incident 4 has one unexplained reboot** on record whose cause was never
  established.