22 — Solomon: Evaluation Gates & Model Promotion
Solomon 4 of 6 · Solomon/solomon/evaluation, Solomon/reports/evaluation
Stack: PyTorch, custom LAMBADA + HellaSwag loaders (from original sources, no HF)
Result: all 9 gates green → checkpoint promoted to artifacts/checkpoints/serving.pt
Where this sits. The single most transferable piece of Solomon. It is not "I evaluated my model" — it is a promotion pipeline where evaluation has the authority to block a release, which is exactly what a bank or a platform team needs and rarely gets.
The nine gates
A checkpoint is promoted only if all nine pass. Not most. All.
| # | Gate | Result | What it protects against |
|---|---|---|---|
| 1 | Test perplexity | 90.0 | The model is worse than the last one |
| 2 | Bits per character | 1.62 | Tokenizer-independent sanity check |
| 3 | Token accuracy | 24.9 % | Next-token prediction actually works |
| 4 | Distinct-2 | 0.95 | Output has lexical variety |
| 5 | Repetition rate | 0.5 % | The classic degenerate-loop failure |
| 6 | LAMBADA | 1.3 % | Long-range context is being used |
| 7 | HellaSwag | 24.7 % | Commonsense continuation |
| 8 | Serving p95 | 454 ms | It is fast enough to ship |
| 9 | Throughput | 116 tok/s | It is cheap enough to ship |
Two of those gates are not quality gates — and that is the design
Gates 8 and 9 are latency and throughput. A model that is accurate and too slow is not shippable, and discovering that after rollout is the expensive way to learn it. Putting serving performance inside the promotion gate means "good enough" is defined once, covering both axes, before anything reaches a cluster.
Diversity and degeneration gates
Distinct-2 (0.95) and repetition rate (0.5 %) catch the failure mode perplexity is blind to: a model can score well on likelihood while producing loops. Two independent metrics, one failure mode, both required.
Benchmarks loaded from original sources
LAMBADA and HellaSwag are loaded from their original distributions, not from a dataset hub. That is more work and it is the correct choice: it means the eval set is the one the paper describes, and the contamination check against the training corpus is meaningful.
What promotion actually does
- All 9 gates evaluate → green.
- A model card and an HTML report are written to
reports/evaluation/20260610-024614/. - The checkpoint is copied to
artifacts/checkpoints/serving.pt. - The container image is rebuilt against the promoted checkpoint.
- The rollout is performed.
The image on the cluster is therefore provably the checkpoint that passed, traceable back through the SHA256 manifest to the training run and the corpus.
Rollback is an image tag, not a retrain — the previous serving.pt and its
manifest are retained.
Interview surface this opens
- Designing release gates that can actually block, and who is allowed to override
- Why latency belongs in a quality gate
- Perplexity's blind spots, and which metrics cover them
- Artifact lineage: corpus → run → manifest → checkpoint → image → traffic
- Model cards as a release artifact rather than a compliance chore