Designs by Duhart ← All infographics

Designs by Duhart · project 1 of 28

The activity graph

One log, one topic, three consumers. A transactional outbox because the producer is a Cloudflare Worker and Workers cannot open TCP sockets — so Kafka and CQL are permanently out of reach from the edge.

The Billboard activity pipeline A Cloudflare Worker writes a post and an outbox event in one Postgres transaction. A relay claims outbox rows, produces to a Kafka topic with six partitions keyed by actor, then stamps published_at. Three independent Kafka consumer groups read the same topic: a feed materializer writing to Cassandra, a notification worker writing to Postgres, and a fanout consumer delivering realtime frames. CLOUDFLARE EDGE — no TCP sockets billboard-users (Worker) writes the post AND the outbox row in the same Postgres transaction ORIGIN HOST — BillboardActivityService (Go), beside GlobalChatService one transaction Postgres · billboard_feed_events the activity log / outbox ~40 display columns stay HERE relay claim → produce → stamp published_at Kafka · billboard.activity.v1 6 partitions, keyed by actor id thin events — id, type, actor, target at-least-once from here on group: feed-materializer → Cassandra billboard_feed.feed_timeline dedupe: applied_activity, 7-day TTL group: notification-worker → Postgres billboard_notifications dedupe: unique(activity_id) + ON CONFLICT group: fanout → realtime delivery dedupe: repeat frame dropped client-side Three groups, not three stages A Cassandra outage must not stop notifications. A notification failure must not keep posts out of feeds.

Each consumer reads every event independently and can fail, lag, and retry alone. All four processes run in one binary today because they share a database pool and a Cassandra session — not because they are coupled. Splitting them across hosts is a deployment change, not a rewrite.

Four decisions, and what each costs

Decision 1

Outbox, not direct produce

A Worker cannot enlist Kafka in a Postgres transaction. Producing directly leaves a window where a post exists with no event, or an event survives a rolled-back post. Writing the event inside the post's transaction closes it.

Decision 2

At-least-once, deliberately

The relay produces before stamping published_at. A crash between the two re-delivers. That is the chosen direction: a lost event cannot be recovered, a duplicate can be absorbed. The reverse order would lose events instead.

Decision 3

Three groups, not three stages

One shared pipeline couples every sink's availability to every other's. Three independent consumer groups off one topic means a Cassandra outage stops feeds and nothing else.

Decision 4

Thin events

The outbox row carries ~40 denormalized display columns. None reach the event. Baking display fields in freezes them at write time — and they are wrong the moment a user changes their name.

The cost of at-least-once, paid at every sink

Consumer groupWrites toIdempotency mechanismWhat a duplicate would look like
feed-materializerCassandra feed_timeline applied_activity, 7-day TTLThe same post twice in a feed
notification-workerPostgres billboard_notifications unique index on activity_id + ON CONFLICT DO NOTHING Two buzzes on someone's phone
fanoutrealtime channel repeated frame dropped client-sideA duplicate row appearing live

The idempotency key is the outbox row id — stable across re-delivery by construction. There is no exactly-once claim anywhere in this design, and there should not be.

The directives this was built to

Fractional ordering

LexoRank, not integer positions — reordering one track must not rewrite every row after it.

System playlists are playlists

"Liked Songs" is a row with a reserved id, not a separate table with a parallel code path.

One follows table

Discriminated by target_type, not one table per followable kind.

Hybrid fan-out

Push below ~10k–50k followers, pull above. The threshold is a tuning constant, not a fork in the architecture.

What was verified before any of it was written

Directive 6 required a Kafka and a Cassandra reachable from this host — both confirmed present first. Directive 2 was identified as a migration, not a new column, because /v1/playlists/{id}/shuffle already rewrites an integer position holding live data, and the standing rule is that nothing holding data gets renamed. Where a directive would have required infrastructure that was not present, it was written up as blocked rather than quietly substituted.