Solomon 2 of 6 · Solomon/solomon/data, Solomon/solomon/tokenizer, Solomon/artifacts/corpus
Stack: SentencePiece (BPE), custom extractors, no HuggingFace datasets
Where this sits. The unglamorous half of ML that separates people who have shipped a model from people who have run a notebook. Licensing, cleaning, deduplication, contamination checking, and a written data card.
Licence recorded rather than assumed, deduplicated, and checked for contamination against the evaluation sets before a single training step ran. The tokenizer reaches zero unknown tokens on held-out text, a measurement, not a hope.
WikiText-103 raw, CC BY-SA — license recorded, not assumed. Cleaned, exact-line deduplicated, and split:
| Split | Tokens |
|---|---|
| train | 130M |
| validation | 275K |
| test | 314K |
A documented fallback existed if the mirror was unreachable: Simple English Wikipedia (CC BY-SA, own extractor) plus Project Gutenberg (public domain). Planning the fallback before needing it is the operational habit worth noting.
artifacts/corpus/DATA_CARD.mdProvenance, license, cleaning steps, split sizes, and contamination notes.
The corpus was checked against the evaluation prompt suites and against LAMBADA and HellaSwag. Without that check, the 9-gate promotion pipeline is measuring memorization and cheerfully promoting a bad model.
An eval number is worthless if the eval set leaked into training. Here the contamination check is written down as an artifact, not assumed.
SentencePiece BPE, vocab 8192, byte fallback. Trained here, not downloaded.
| Metric | Value |
|---|---|
| Compression | 4.02 chars/token |
| UNK tokens | 0 |
Recorded in artifacts/tokenizer/tokenizer_stats.json.
Byte fallback is the reason UNK is zero: any codepoint the BPE merges do not cover decomposes into bytes rather than becoming an unknown. That is a design choice with a measurable consequence, and the measurement is in the repo.
8192 was chosen against a 6.84M-parameter model with tied embedding and
output weights. The embedding matrix is vocab × d_model; at vocab 32k it
would dominate the parameter budget and starve the layers. This is the
vocab/depth trade-off, made explicitly rather than by default.