28 — Platform Operations, Deployment & CI/CD

Additional project 4 of 4 · deploy/, app/deploy/systemd, .github/, .gitlab-ci.yml, app/gateways/docker-compose.yml, */RUNBOOK.md Stack: systemd (units + timers), Docker Compose, k3s/Helm, GitLab CI, GitHub Actions, wrangler, cloudflared

Where this sits. The connective tissue. On its own it is not a project you lead with; in aggregate it is the evidence that everything else in this portfolio actually runs somewhere rather than existing as a repository.

Infrastructure in the tree

deploy/
  kafka/      docker-compose + k8s manifests
  postgres/   redis/   coturn/   k8s/ingress.yaml   J8/
app/deploy/systemd/
  theone-backup.service     theone-backup.timer
  theone-health.service     theone-health.timer
  theone-restore-verify.service  theone-restore-verify.timer
app/gateways/docker-compose.yml   Makefile
app/GlobalChatService/deploy/     systemd unit, local bring-up, CQL schema fix
Solomon/k8s/helm/                 8 chart templates
Solomon/Dockerfile  Makefile  mk/  ci/

The three timers are the tell

theone-backup, theone-health, and — the one that matters — theone-restore-verify.

A backup you have never restored is a hypothesis. A scheduled timer that verifies the restore is the difference between claiming you have backups and knowing it. Most candidates have the first two units. The third is the one to point at.

Local reproducibility

app/gateways/docker-compose.yml + Makefile — the whole three-gateway stack, Kafka and Cassandra included, on a laptop
app/GlobalChatService/deploy/local/ — bringup.sh, docker-compose.yml, chat.local.env, and a README
deploy/kafka, deploy/postgres, deploy/redis, deploy/coturn — each dependency stood up independently

Being able to run a multi-datastore distributed system locally is an underrated answer to "how do you develop against this?"

CI, and the fact that it was new

Two systems: .gitlab-ci.yml (with secret detection) and .github/ workflows. The Billboard campaign's [M-3] commit is titled "CI that runs typecheck, tests and a real end-to-end pass on every push" and [CI] is "Contract diff and shadow-type gates, and the first CI Billboard has had."

Saying "this service had no CI and I gave it one" is a stronger claim than "we had CI."

Gates enforced:

Gate	What it catches
Typecheck	Drift between code and types
Unit tests	Regression
Real end-to-end pass	The thing that only breaks when wired together
Contract diff	Server drifting from `openapi.yaml`
Shadow types	Hand-written types shadowing generated ones
Secret detection	Credentials in a commit

The shadow-type gate exists because of a real defect: [F-5] Remove the fake R2 types, and fix the three bugs they were hiding. Hand-written types that shadow the real SDK's types make the compiler agree with code that cannot work. The gate stops it recurring — fix the bug, then close the door behind it.

Runbooks and edge deployment

RUNBOOK.md in GlobalChatService, docs/deployment/DEPLOYMENT-GUIDE.md, MASTER-CONTROL.md, k8s/INGRESS.md, and per-session DEPLOY_QUEUE.md / BLOCKED.md records. Edge deploys go through wrangler; origin traffic arrives via cloudflared tunnels.

Interview surface this opens

Backup verification as a scheduled job, not a quarterly fire drill
systemd timers vs cron for service-adjacent scheduled work
Designing CI gates that encode a specific past defect
Reproducing a multi-datastore distributed system locally
Tunnels and edge-to-origin routing without exposing an origin