# 1 — Billboard Activity Graph

**Rank 1 of 28** · Tier S · `app/BillboardActivityService`, `app/workers/billboard_users`
**Stack:** Go 1.25, Kafka (segmentio/kafka-go), Cassandra (gocql), PostgreSQL, Cloudflare Workers
**Status:** Built end-to-end and running. Commit `9d31b7d28`.

> **Why this is ranked first.** This is the system design interview. Not
> *like* it — it is the exact question ("design a social feed / activity
> stream") with the exact primitives (outbox, log, consumer groups,
> idempotency, hybrid fan-out) implemented in production Go against real
> Kafka and real Cassandra. Every follow-up an interviewer can ask has an
> answer already written down in the commit message.

---

## The problem

Playlists, follows, feed, and posts are one social graph. The naive build gives
each of them its own event pipeline, its own fan-out, and its own consistency
story — four systems that drift apart. The directive was one activity log and
one materializer.

Complication: the service that writes activity, `billboard-users`, is a
**Cloudflare Worker**. Workers cannot open raw TCP sockets — not with a
library, not with a shim. The Kafka wire protocol and CQL are both permanently
unreachable from a Worker.

## The architecture

```
billboard-users (Cloudflare Worker)
      │  writes billboard_feed_events in the SAME transaction as the post
      ▼
Postgres billboard_feed_events                    ← the activity log / outbox
      │  relay: claim → produce → stamp published_at
      ▼
Kafka  billboard.activity.v1   (6 partitions, keyed by actor)
      │
      ├─ group feed-materializer   → Cassandra billboard_feed.feed_timeline
      ├─ group notification-worker → Postgres billboard_notifications
      └─ group fanout              → realtime delivery
```

Four processes in one binary because they share a database pool and a Cassandra
session — **not** because they are coupled. Each consumer owns its group, so
splitting them across hosts later is a deployment change, not a rewrite.

## The four decisions worth defending

### 1. Outbox, not direct produce

A Worker cannot enlist Kafka in a Postgres transaction. Producing directly
leaves a window where either a post exists with no event, or an event survives
a rolled-back post. Writing the event **inside the post's transaction** and
relaying it afterwards closes that window.

### 2. At-least-once, with the cost paid explicitly at every sink

The relay produces *before* stamping `published_at`. A crash between the two
re-delivers. That is deliberate: **a lost event cannot be recovered, a
duplicate can be absorbed.** Each consumer absorbs it differently:

| Consumer | Idempotency mechanism |
|---|---|
| feed-materializer | `billboard_feed.applied_activity`, Cassandra, 7-day TTL |
| notification-worker | unique index on `activity_id` + `ON CONFLICT DO NOTHING` |
| fanout | repeated realtime frame dropped client-side |

The idempotency key is the outbox row id — stable across re-delivery by
construction.

### 3. Three groups, not three stages

A Cassandra outage must not stop notifications. A notification failure must not
keep posts out of feeds. A single shared pipeline does exactly that. Three
independent groups off one topic means each can fail, lag, and retry alone.

### 4. Thin events, not denormalized ones

`billboard_feed_events` carries ~40 denormalized display columns for the
activity-feed UI. **None of them are in the Kafka event.** A consumer that
needs the actor's avatar joins for it. Baking display fields into the event
freezes every one of them at write time — and they are wrong the moment a user
renames themselves.

## Related design directives implemented

- **Fractional / LexoRank ordering**, not integer positions — reordering one
  track must not rewrite every row after it.
- **System playlists are fixed-ID playlists.** "Liked Songs" is a row with a
  reserved id, not a parallel table with duplicate code paths.
- **One `follows` table discriminated by `target_type`**, not one per
  followable kind.
- **Hybrid fan-out** — push below ~10k–50k followers, pull above. The threshold
  is a tuning constant, not a fork in the architecture.

## What was verified before building

Directive 6 ("feed materialization is a Kafka consumer writing to Cassandra")
required a Kafka and a Cassandra reachable from this host. Both were confirmed
present before a line was written. Directive 2 was identified as a *migration*,
not a new column — `/v1/playlists/{id}/shuffle` already rewrites an integer
`position` holding live data.

Where a directive would have required infrastructure that was not present, it
was written up as blocked rather than quietly substituted.

## Scope

| Metric | Value |
|---|---|
| Go source | 977 lines across 5 internal packages |
| Kafka partitions | 6, keyed by actor id |
| Consumer groups | 3, independent |
| Datastores written | Cassandra, Postgres, realtime channel |
| Delivery semantics | At-least-once, deduped at every sink |

## Interview surface this opens

- Exactly-once vs at-least-once, and why you never got to claim the former
- Outbox pattern vs CDC vs dual-write, and what each costs
- Partition key choice (actor id) and the hot-partition risk it carries
- Consumer group rebalancing and lag monitoring
- Push/pull fan-out crossover and where the celebrity problem bites
- LexoRank and why integer positions do not survive reordering at scale