30 · BillboardML, Audio Recognition & Recommendation

The shape of it

cached audio ──► signal engine (DSP) ──► stores: CLAP, AST, MERT, CNN, hybrid, fused
      │                                         │
      ▼                                         ▼
 landmark fingerprint ──► bucketed index    heads, towers, JamzFinder
      │                       │                 │
      ▼                       ▼                 ▼
          /v1/identify      /v1/jamzfinder/related      Track Intelligence record

A batch CLI and a daily cycle produce everything; a small API on loopback serves it. Nothing here is a daemon that owns state except the API, and the API only reads artefacts the batch side wrote. That split matters for the incident further down.

Fingerprinting: a fourth implementation, held bit-for-bit

Jamz is the app's "what is playing" feature. Three implementations of its landmark fingerprint already had to agree exactly: the C compiled to wasm that the Worker's Durable Object runs over the catalogue, the Swift engine on the phone, and the browser's TypeScript. If one constant drifts, the hashes never collide. Recognition then answers "no match" forever, with no error anywhere.

So BillboardML did not invent a fifth hash space. It ported the contract: 4096-point frames at 44.1 kHz, hop 2048, a Hann window carrying the vDSP normalisation, the x2 correction for vDSP's FFT scaling, power dB not amplitude dB, strict interior maxima above -60 dB, five peaks a frame, six targets per anchor within 500 ms, and a 32-bit packed hash whose top bit is set for high anchor bins. The port is tested against a naive-DFT transcription of the wasm harness's reference on its seven signals, peaks bin-for-bin and dB within 0.05. Transcribed, not imported, so a coding error on either side gets caught. A shared misreading would not, and the note says so.

The payoff is that the Worker's fingerprint table and this index describe the same recordings in the same hashes. One client capture can be answered by either side.

Thresholds measured, not inherited

Every decision rule in identify() came from a gate run, and each run changed one rule. On a 35-track smoke index:

1. The count rule alone is not a rule. Jamz accepts at five matching pairs. A held detuned chord matched a real track at 651 votes, white noise at 7. A 10 s clip carries thousands of pairs, so five agreeing somewhere is chance.
2. The upstream confidence is always 1.0. The Worker's SQL divides each score by the window's maximum and then takes LIMIT 1, so the top row divides by itself and the MIN_CONFIDENCE 0.75 check can never reject anything. Here confidence is the margin over the best other track. Recorded for the Worker's owner, not fixed from this repository.
3. The spike straddles two bins. The query's frame grid lands anywhere inside a hop relative to the catalogue's, so a true match fills the neighbouring offset bin nearly as full. Coherence is the bin plus its neighbours against the median far spike, because looped music earns real secondary spikes eight bars later.
4. Two true clips lost to themselves. The runner-up was the same recording under a second id. A coherent runner-up within half the spike is now a reported near_duplicate, which doubles as a duplicate detector.

Then the full index: 3,158 tracks, 457 million pairs, 4.57 GB. Three more gate rounds moved the rules again (query phases, even sampling of common hashes, a floor on distinct coherent votes). The confirmatory gate in data/fingerprint/gate.json: 57 of 60 clips identified at 10 dB SNR, the weakest true clip at 156 votes, every foreign clip refused at 42 or fewer, 99 of 99 byte-identical duplicate pairs found. The three misses are shared audio, the same loop under another id, not chance.

Flat memory on a box that has none

The host is a shared 31 GB machine that is usually within a few GB of the OOM killer. The index was designed around that:

• fingerprints.jsonl is append-only and its line order is the track number, so a killed build resumes without renumbering.
• Rows stream into 256 bucket files as each track finishes. Finalising sorts one bucket at a time into three memory-mapped arrays.
• A query is a binary search inside one bucket per hash.

Two OOMs still happened, both read and fixed. A whole query expanded at once was 9 GB and was killed under the API's 3 GB cap, so lookups now run in 4,096-pair slices and keep only deduplicated votes between slices. And a build worker died alongside Cassandra in a global OOM wave; because the log is append-only, the build lost nothing.

The incident: a writer that truncated a live reader's mmap

The next morning the daily cycle rebuilt the index to 5,362 tracks and 781 million pairs. finalise_index opened the arrays with "wb". That truncates and rewrites the same inodes the running API had memory-mapped the night before, under the old length and the old bucket offsets. Nothing crashed. The API kept answering, against a mismatched index: two known clips scored 1,014 and 1,279 votes in-process on the fresh files, and 8 and 5 through the API. Both refused.

The fix is the old Unix rule. Write .tmp siblings, os.replace them into place, metadata last. A running process keeps its old inode until it restarts, so a reader sees either the whole old index or the whole new one. Tests went to 16 of 16, the API was restarted, and four clips (two old tracks, two newly indexed) identified themselves at 1,014 to 1,520 votes.

Recommendation, and the discipline that keeps models out

The yardstick is same-artist recall@10 with album-excluded queries and artist-grouped folds, against a stated chance level (0.52 on the current catalogue), because random splits leak the same album into train and test. Every learned store must match a frozen CLAP reference under a paired bootstrap before it serves.

So far nothing does, and the notes say so. The best student, a pretrained Cnn14 distilled from CLAP over five folds, reads album-excluded R@10 0.711 [0.684, 0.736] against the teacher's 0.825, a paired difference of -0.114. Better than ResNet18 and the from-scratch CNN, and still out. The two-tower retriever loses to the centroid baseline and is gated out. JamzFinder's fused list (0.7 cosine, 0.3 a written-down musical distance) does not beat the plain embedding (CI [-0.037, 0.020]) and is withheld, served only when the lower bound clears zero. Along the way one "music" CLAP checkpoint on the Hub turned out to be untrained, found from its own evidence chain, and replaced by its trained sibling.

Metadata coverage, where the gate is the unit's exit status

A separate batch sweep keeps artist and genre filled across the catalogue. The owner's rule: at least 80 percent coverage, fill blanks only, never overwrite. The sweep runs from a systemd timer, and the unit's exit status is the gate: a failed unit means coverage fell below 80 percent or a step raised.

The finding that moved coverage: the pipeline had only ever read filenames. The files' own ID3 tags named the artist for most of the "unidentifiable" rows. Tags and the web crawler agreed only 46 percent of the time, and in 9 of 10 hand-checked disagreements the tag was right, the crawler having picked a cover act. Tags now outrank the crawler. Measured 2026-09-24: artist 97.8 percent, genre 87.8 percent of the music tracks, audiobooks excluded from the denominator. Provenance lives in an append-only ledger, and training excludes the model's own earlier guesses so it never learns from itself.

Honest gaps

• Nothing learned beats CLAP yet. The serving stores are frozen backbones.
• The fingerprint duplicates migration is written and not applied, and the catalogue-wide duplicate pass has never run to completion.
• The first full daily cycle ended with a FAIL verdict, on a two-tower regression gate that compares across catalogues of different sizes; whether that gate is right is an open decision.
• Nobody consumes the learned stack in the apps yet. /v1/identify and JamzFinder are live on loopback, not wired into Jamz.
• Genre CV numbers from the coverage sweep used random splits and are optimistic. There is no independent genre ground truth.
• The Worker's confidence defect is recorded, not fixed.