# Interview Stories Eight stories, in situation → action → result form, each with the follow-up questions it invites and the answer you should already have ready. These are drawn from committed code, so every one survives a "walk me through it." --- ## 1. The phantom Cassandra partition **Situation.** Users were seeing nameless cards in their discovery deck — real rows, no display name. **Action.** Traced it back through the sorting hat to the ranking pipelines. `match_cards` is partitioned by `(service_region, geohash4)`. The pipelines iterate a *configured* list of regions, and the region in the loop is the pool being ranked — not necessarily the pool the user's card lives in. Because Cassandra `UPDATE` is an upsert, writing with the loop's region did not fail and did not update: it **created a new partition**, one per user per configured region, holding only score columns. The sorting hat then indexed those into real decks. **Result.** Added a partition-lookup layer that resolves the user's true partition before any write, so partition identity is never inferred from iteration context. Fix and reasoning committed in `user_partition.go`. **Follow-ups to expect** - *Why didn't this error?* — Cassandra has no "row must exist" write. Upsert is the only write. - *How would you catch it earlier?* — a read-back assertion in the write path in non-prod, or a partition-count invariant per user in monitoring. - *Would a different partition key have avoided it?* — only by making the key derivable from the row itself; the real fix is not letting the caller supply a key it does not own. --- ## 2. The 952 orphaned messages **Situation.** A Redis Streams feed index was silently falling behind. **Action.** A Redis consumer group moves a message to the consumer's **pending-entries list** on delivery and removes it only on `XACK`. Both feed consumers `continue` past a processing failure without acking — which is correct, since acking work you failed to do destroys it. But nothing ever came back for those entries. `feed_index:writes` had accumulated **952 pending messages**, the oldest long past useful. **Result.** Implemented an aged-entry reclaim path so failed work is retried rather than orphaned. `stream_reclaim.go`. **Follow-ups to expect** - *Kafka vs Redis Streams here?* — Kafka's group offsets make this failure look like lag (visible); Redis's PEL makes it look like nothing (invisible). That difference is the whole story. - *When do you give up on a message?* — you need a delivery-count threshold and a dead-letter destination; unbounded retry is the same bug wearing a different hat. - *How do you alert on it?* — PEL depth and oldest-pending age, both as gauges. --- ## 3. The Worker that could not speak Kafka **Situation.** The directive was "feed materialization is a Kafka consumer writing to Cassandra." The service producing the activity is a Cloudflare Worker. **Action.** Established first that Workers cannot open raw TCP sockets — so both the Kafka wire protocol and CQL are permanently unreachable, not merely awkward. That ruled out producing from the Worker at all. It also ruled out a dual write: a Worker cannot enlist Kafka in a Postgres transaction, so producing directly leaves a window where a post exists with no event, or an event survives a rolled-back post. **Result.** A **transactional outbox**: the event row is written inside the same transaction as the post; a host-resident Go relay claims, produces, and stamps `published_at`. The relay produces *before* stamping — a crash between the two re-delivers, which is deliberate, because **a lost event cannot be recovered and a duplicate can be absorbed**. Each of the three consumers absorbs it with a named mechanism. **Follow-ups to expect** - *Why not CDC / Debezium?* — a legitimate alternative; it trades an application-owned relay for an infrastructure dependency and a schema-coupled connector. Say that, don't dodge it. - *Exactly-once?* — no, and you should say so flatly. At-least-once plus idempotent sinks. Effectively-once at the sink is the honest phrasing. - *What's the idempotency key?* — the outbox row id, stable across re-delivery by construction. --- ## 4. Three groups, not three stages **Situation.** Three things must happen when a user posts: it enters followers' feeds, it generates notifications, and it fans out to realtime listeners. **Action.** The tempting build is one pipeline doing all three in sequence. That means a Cassandra outage stops notifications, and a notification failure keeps posts out of feeds. Instead: **one topic, three independent consumer groups**, each reading every event, each failing and retrying alone. **Result.** All four processes run in one binary today because they share a database pool and a Cassandra session — **not** because they are coupled. Splitting them across hosts is a deployment change, not a rewrite. **Follow-ups to expect** - *Why not three topics?* — one event, three readers. Three topics means three producers to keep consistent. - *Ordering guarantees?* — per-partition, keyed by actor, so one actor's events stay ordered relative to each other. Cross-actor ordering is not promised and is not needed. - *Hot partition risk?* — yes, a very high-volume actor concentrates on one partition. Mitigation is a composite key or a spill partition for outliers. --- ## 5. The event that deliberately carries almost nothing **Situation.** The outbox table carries ~40 denormalized display columns for the activity-feed UI. The obvious move is to put them in the Kafka event so consumers don't have to join. **Action.** Kept the event minimal — id, type, actor, target type, target id, timestamp. Consumers that need the actor's avatar join for it. **Result.** Display fields are not frozen at write time. A user renaming themselves does not leave a trail of events carrying their old name forever. **Follow-ups to expect** - *Isn't the join expensive at fan-out time?* — yes, and it's cached. The alternative is unbounded incorrectness, which has no cache. - *When would you denormalize?* — when the value is genuinely immutable at event time (the amount of a transaction, the version of a document), not when it's a mutable profile attribute. --- ## 6. The gate that stops a model shipping **Situation.** A trained checkpoint is not a deployable model. **Action.** Built promotion as a **9-gate evaluation** — perplexity, BPC, token accuracy, distinct-2, repetition rate, LAMBADA, HellaSwag, serving p95, and throughput — with a data card recording contamination checks against the eval suites themselves. Only when all nine pass is the checkpoint copied to `serving.pt`, the image rebuilt against it, and the rollout performed. **Result.** All 9 green; test PPL 90.0, p95 454 ms, 116 tok/s. The image on the cluster is provably the checkpoint that passed, via a SHA256 manifest. **Follow-ups to expect** - *Why include latency in a quality gate?* — because a model that is accurate and too slow is not shippable, and discovering that after rollout is the expensive way. - *Contamination checking?* — the corpus was checked against the eval prompt suites; otherwise the gate measures memorization and passes a bad model. - *How do you roll back?* — the previous `serving.pt` and its manifest are retained; rollback is an image tag, not a retrain. --- ## 7. The bug that takes down the host **Situation.** Auditing the WebRTC call path in a Go service. **Action.** Found a **concurrent map access** in the call path. In Go this is not a panic you can recover — it is `fatal error: concurrent map writes`, which terminates the runtime. So it does not degrade one call; it drops **every connection on the host**. **Result.** Documented as G-07 in the gap register with severity and a remediation entry, rather than being papered over. **Follow-ups to expect** - *How do you fix it?* — `sync.Map` for the read-mostly case; better, a single owning goroutine with a channel so the map is never shared at all. - *How do you find these before production?* — `-race` in CI, and the fact that this codebase has dedicated race tests (`calling_race_test.go`) is the answer to "do you actually do that." - *Why is a fatal worse than a panic?* — no recover, no graceful shutdown, no draining. Every in-flight request dies with it. --- ## 8. Reproducing bugs on purpose **Situation.** Porting a large iOS app to Kotlin/Compose, seven pre-existing defects surfaced in the original. **Action.** **Reproduced them deliberately in the port** and wrote them down, instead of fixing them in passing. **Result.** The two platforms stay behaviorally identical during the migration, so parity testing means something. The defects get fixed once, on purpose, on both platforms, as their own change. **Follow-ups to expect** - *Isn't that shipping known bugs?* — for the duration of a migration, yes, and it is the cheaper error. A port that silently diverges is untestable against the original. - *What if one is a security bug?* — then it is not a parity question, it comes out of the queue and gets fixed on both immediately. Have this distinction ready. --- ## Questions to ask them Ask these; they signal the level you're interviewing at. 1. What is your delivery guarantee at the boundary between services, and where do you pay for it? 2. When a consumer group falls behind, what page fires and who gets it? 3. How do you keep client and server contracts from drifting — is it enforced, or reviewed? 4. What is the last unrecoverable production failure, and what changed structurally afterward? 5. For a fully remote role at this level: how are design decisions made and recorded when nobody is in a room?