# 21 — Solomon: Training Loop & Experiment Tracking

**Solomon 3 of 6** · `Solomon/solomon/training`, `Solomon/reports/training`
**Stack:** PyTorch, MLflow (self-hosted), `torch.compile`, GitLab CI
**Result:** 8,000 steps · 3.0 h wall · **validation perplexity 40.0** · CPU only

> **Where this sits.** The reproducibility artifact. Anyone can run a training
> script; this is the pipeline that makes a run *reproducible, resumable, and
> auditable months later*.

---

## The loop

**Noam schedule + label smoothing** as the baseline, plus gradient
accumulation, gradient clipping at 1.0, early stopping, checkpoint/resume,
sample generations at every eval, and `torch.compile`.

Seeds pinned. Requirements locked. Dockerfile environment fixed.

## CPU-only, and said so

No GPU on the host. Mixed precision is therefore N/A — and that is **written
down as a documented constraint** rather than quietly omitted. The model was
sized so a real 3-hour run fits the machine's actual spare RAM after the
production services on the same box take theirs.

> Naming your constraints is more credible than implying you had a cluster.

## MLflow — self-hosted, both ways

File/SQLite backend for local runs; a server **in-cluster** for the deployed
case (`Solomon/k8s/helm/templates/mlflow.yaml`). Same tracking API, two
deployment shapes.

## The run report — the artifact that matters

Every run emits a report to `reports/training/<run-id>/` containing:

- Loss and learning-rate curves (HTML + Markdown)
- Full hyperparameter record
- The data card for the corpus used
- A **SHA256 manifest**

The manifest is the point. It is what lets you say, six months later, *"the
model serving traffic right now is exactly the checkpoint produced by this run,
from this corpus, under these hyperparameters"* — and prove it. That chain is
what the [promotion gate](22-solomon-evaluation-gates.md) consumes and what the
[deployed image](23-solomon-serving-and-kubernetes.md) is built from.

## Results

| Metric | Value |
|---|---|
| Steps | 8,000 |
| Wall time | 3.0 h (CPU, 16 cores) |
| Validation perplexity | **40.0** |
| Run id | `5ef54b53bfec41b6a639d22cec3abdba` |

## Known defects, recorded not hidden

From `CODE_REVIEW.md`: **`--resume` restarts the shuffle.** A resumed run does
not continue the same data order it would have seen, which makes a resumed run
subtly not identical to an uninterrupted one. It is listed as an open defect
rather than left to be discovered.

## Interview surface this opens

- The Noam warmup schedule: why warmup, and what happens without it
- Label smoothing and its effect on calibration vs raw accuracy
- Gradient accumulation as a substitute for batch size you cannot afford
- Checkpoint/resume correctness — including the data-order problem named above
- Experiment tracking and artifact lineage: run → manifest → image → traffic