# 4 — Solomon: LLM Training & Serving Platform

**Rank 5 of 28** · Tier A · `Solomon/`
**Stack:** PyTorch (no HuggingFace), SentencePiece, FastAPI, MLflow, Docker, k3s + Helm + HPA, Prometheus, Go (retrieval tier), Kafka, GitLab CI
**Status:** Trained, eval-gated, promoted, containerized, deployed to k3s, scaled to 3 replicas under load.

> **Why this ranks here.** It is the ML-platform / infrastructure story, and it
> is complete end to end: corpus → tokenizer → training → gated evaluation →
> promotion → container → Kubernetes → autoscaling → streaming inference →
> clients. For Google (Cloud/ML infra) and any bank building internal model
> serving, this is directly on-target. It ranks below the Tier S three only
> because it is single-node training, not a distributed training story.

---

## What was built

A decoder-only causal language model implemented **in pure PyTorch from the
Annotated Transformer reference**, with every component either verbatim or with
its deviation justified in writing, then carried all the way to a deployed,
autoscaled inference service.

Explicitly **no HuggingFace anywhere** — model, tokenizer, data loading, and
the LAMBADA/HellaSwag eval sets were all sourced and implemented directly.

### Model
6.84M params · d_model 256 · 6 layers · 8 heads · d_ff 1024 · vocab 8192 ·
weight-tied · pre-norm · KV-cached decode **proven exactly equal to full
recompute** · greedy + beam + temperature/top-k/top-p.

### Data
WikiText-103 raw (CC BY-SA), cleaned, deduped: 130M train / 275K val / 314K
test tokens. Data card with **contamination checks against the eval suites**.
SentencePiece BPE 8192 with byte fallback: 4.02 chars/token, 0 UNK.

### Training
Noam schedule + label smoothing, grad clipping, early stopping,
checkpoint/resume, MLflow tracking, `torch.compile`. 8,000 steps, 3.0 h wall,
**val PPL 40.0**. Report with loss/LR curves, hparams, and a SHA256 manifest.

### Evaluation — a promotion gate, not a report

All **9 gates green**, and only then is the checkpoint promoted to
`artifacts/checkpoints/serving.pt`:

| Gate | Result |
|---|---|
| Test perplexity | 90.0 |
| Bits per character | 1.62 |
| Token accuracy | 24.9 % |
| Distinct-2 | 0.95 |
| Repetition rate | 0.5 % |
| LAMBADA | 1.3 % |
| HellaSwag | 24.7 % |
| Serving p95 | 454 ms |
| Throughput | 116 tok/s |

*This gate is the part that matters for hiring.* It is the difference between
"I trained a model" and "I built the pipeline that decides whether a model is
allowed into production."

### Serving
FastAPI with **SSE and WebSocket streaming**, continuous batching, KV cache,
Prometheus metrics, OpenAPI, and RS256 platform JWT verification (401 path
verified).

### Kubernetes
k3s + Helm: inference deployment with **HPA, observed scaling to 3 replicas
under load**, MLflow in-cluster, manifests passing `kubectl apply
--dry-run=server`, image rebuilt against the promoted checkpoint and rolled
out. Chart templates cover a multi-phase topology (orchestrator, members,
retrieval, search, network policy).

### Retrieval tier — Go, 4,757 lines
`Solomon/retrieval/` is a separate Go service: chunking, embedding, search,
store, an agent loop with tool calling and a **grounding guard** (with its own
test), RSS/fetcher ingestion, migrations, auth, and a `kafkax` package. This is
where the ML platform becomes a distributed system.

### Testing
Python 62/62 at **86 % coverage** — including a causal-mask leak test, a
positional-encoding formula test, an overfit-one-batch test, greedy
determinism, and contract tests. Web vitest 11/11, Playwright 2/2 including a
live e2e with no mocks. Load: **198 requests, 0 failures, p95 2.7 s against a
6 s SLO.** GitLab CI wired.

## Intellectual honesty on display

`CODE_REVIEW.md` and `BLOCKERS.md` list what is still wrong, unprompted:
SSE per-token decode dropping inter-word spaces; the engine swallowing
generation exceptions into an empty 200; disconnect-cancellation dead code
leaking generations; `make deploy` sudo stripping `KUBECONFIG`; a `--resume`
shuffle restart; a localStorage seed-clobber race.

**Bring this file to the interview.** A candidate who ships a written list of
their own unfixed bugs, ranked, is signalling exactly the thing senior loops
are trying to detect.

## Scope

| Metric | Value |
|---|---|
| Python + Go source | 22,050 lines, 163 files |
| Go retrieval tier | 4,757 lines |
| Helm templates | 8 |
| Eval gates | 9, all green, promotion-blocking |
| Test coverage | 86 % Python; vitest 11/11; Playwright 2/2 |
| Load result | 198 reqs, 0 fail, p95 2.7 s vs 6 s SLO |

## Interview surface this opens

- Transformer internals at the tensor level, defended component by component
- KV caching and how you *prove* it is equivalent to full recompute
- Continuous batching and the latency/throughput trade-off
- Model promotion gates, contamination checking, and reproducibility manifests
- HPA on a stateful-ish inference workload: what metric do you scale on?
- Why no HuggingFace — and what you learned by not having it