# 22 — Solomon: Evaluation Gates & Model Promotion

**Solomon 4 of 6** · `Solomon/solomon/evaluation`, `Solomon/reports/evaluation`
**Stack:** PyTorch, custom LAMBADA + HellaSwag loaders (**from original sources, no HF**)
**Result:** all 9 gates green → checkpoint promoted to `artifacts/checkpoints/serving.pt`

> **Where this sits.** The single most transferable piece of Solomon. It is not
> "I evaluated my model" — it is **a promotion pipeline where evaluation has the
> authority to block a release**, which is exactly what a bank or a platform
> team needs and rarely gets.

---

## The nine gates

A checkpoint is promoted only if **all nine** pass. Not most. All.

| # | Gate | Result | What it protects against |
|---|---|---|---|
| 1 | Test perplexity | 90.0 | The model is worse than the last one |
| 2 | Bits per character | 1.62 | Tokenizer-independent sanity check |
| 3 | Token accuracy | 24.9 % | Next-token prediction actually works |
| 4 | Distinct-2 | 0.95 | Output has lexical variety |
| 5 | Repetition rate | 0.5 % | The classic degenerate-loop failure |
| 6 | LAMBADA | 1.3 % | Long-range context is being used |
| 7 | HellaSwag | 24.7 % | Commonsense continuation |
| 8 | **Serving p95** | **454 ms** | It is fast enough to ship |
| 9 | **Throughput** | **116 tok/s** | It is cheap enough to ship |

### Two of those gates are not quality gates — and that is the design

Gates 8 and 9 are **latency and throughput**. A model that is accurate and too
slow is not shippable, and discovering that after rollout is the expensive way
to learn it. Putting serving performance inside the promotion gate means
"good enough" is defined once, covering both axes, before anything reaches a
cluster.

## Diversity and degeneration gates

Distinct-2 (0.95) and repetition rate (0.5 %) catch the failure mode perplexity
is blind to: a model can score well on likelihood while producing loops. Two
independent metrics, one failure mode, both required.

## Benchmarks loaded from original sources

LAMBADA and HellaSwag are loaded from their **original distributions**, not
from a dataset hub. That is more work and it is the correct choice: it means
the eval set is the one the paper describes, and the
[contamination check](20-solomon-corpus-and-tokenizer.md) against the training
corpus is meaningful.

## What promotion actually does

1. All 9 gates evaluate → green.
2. A **model card** and an HTML report are written to
   `reports/evaluation/20260610-024614/`.
3. The checkpoint is copied to `artifacts/checkpoints/serving.pt`.
4. The container image is **rebuilt against the promoted checkpoint**.
5. The rollout is performed.

The image on the cluster is therefore provably the checkpoint that passed,
traceable back through the SHA256 manifest to the
[training run](21-solomon-training-pipeline.md) and the corpus.

**Rollback is an image tag, not a retrain** — the previous `serving.pt` and its
manifest are retained.

## Interview surface this opens

- Designing release gates that can actually block, and who is allowed to override
- Why latency belongs in a quality gate
- Perplexity's blind spots, and which metrics cover them
- Artifact lineage: corpus → run → manifest → checkpoint → image → traffic
- Model cards as a release artifact rather than a compliance chore