29 · FeedBench: One Feed Contract, Three Runtimes, Under Load and Chaos

The setup

The system under test is the hot path of the Global Feed Gateway: a mode-scoped social feed with profiles, posts, reactions and comments. Every read passes a six-clause visibility projection. A hidden item returns a 404 that is byte-identical to one that never existed. The feed page is assembled from three sections (own, followed, suggested), each with a 400 ms budget. A section that overruns is dropped and named in degraded, and the page still returns.

Three implementations, one written contract:

contract/CONTRACT.md fixes nine routes, the schema, the wire format, the error table, the cursor scheme, the transaction boundary of every write, and the operational behaviour: readiness, a graceful drain on SIGTERM, per-request deadlines carried into the database, and bounded pool waits that fail fast with a 503. Each port carries the projection agreement test (the SQL predicate and the row check must agree) plus a mutation test per clause. A port that fails them is not benchmarked. That rule is what makes "three implementations" mean one system three times, rather than three systems.

The testbed

One host, hard cgroup caps, one contender at a time. Two gateway replicas at 1.5 CPU and 1 GB each behind HAProxy, Postgres at 2 CPU, an open-model k6 load generator, Toxiproxy on every database and object-store link, and a memory guard that aborts the run below 1.5 GB available. A separate arm ran a bench Worker on Cloudflare's real network with its own D1, R2 and KV.

Experiments: a step ramp to saturation, a 5-minute baseline at 70% of each contender's own saturation, then six faults injected at that load. Replica kill, graceful termination, CPU theft by a co-tenant, +100 ms on the database link, a 15 s database cut, and +150 ms on the object store. Plus a 24-hour diurnal curve compressed to 24 minutes and a 20-minute soak.

Sections 1 to 3 of the paper were committed on 2026-10-01, before any run. Where the study departed from them, a deviations section says so, item by item, instead of quietly editing the plan.

What the numbers say

Under a 500 ms p99 and 1% error objective:

The runtime did not decide throughput. At the first failing step, Postgres was at 96 to 98% of its 2-core cap for both ports while the gateways averaged 18% (Go) and 37% (Spring) of theirs. The Workers store host was running D1's single SQLite thread at one full core. None of the three gateways was the bottleneck.

It did decide cost and recovery. Go served about 30% more requests per core than Spring in a twentieth of the memory. A Spring replica took 8 s to answer its first readiness probe after a restart and 17 to 18 s to get back inside 10% of control. Go was back in under a second. A JVM fleet needs warm-up traffic or balancer slow-start for deploys to be invisible, and now I have the number that says how much.

The finding nobody asked for: the production schema full-scans the feed

With the production indexes, Go met the objective only up to 4 arrivals per second. Spring never met it (p99 10.0 s at 2 arrivals per second). The Workers ramp aborted during warm-up with 70.7% of requests failing.

The cause: every feed section filters on mode AND audience IN (...) AND archived_at IS NULL AND (ownerBranch OR visitorBranch), and the existing partial indexes require state = 'ready' or visibility = 'public'. The query does not imply either, so SQLite never uses them and Postgres picks a generic plan that takes 1.6 to 1.8 s on the suggested section.

Two indexes, no query change, no code change: Go went from 4 to 80, Spring and Workers from never meeting the objective to 90 and 15. That is a 20x effect, larger than anything the runtimes did. A runtime comparison on the production schema would have been a benchmark of the query planner. The production Billboard feed has the same predicate shape, so this went to the owner as a finding. The study did not touch production.

Caching moves the bottleneck, and leaves a tail

Go's per-replica cache (singleflight, 256 MiB) raised saturation from 80 to 190. Then a 1-core object store became the ceiling and failed within minutes at 70% of the new rate, with S3 SlowDown in the logs. The cache also made restarts expensive in a new way: a killed replica came back cold, and recovery took 84 s against 0.4 s without the cache. A stateless replica is cheap to lose. A replica holding a warm cache is not.

On Cloudflare's edge, the Cache API cut median latency by 37% at 35 arrivals per second and pushed the error knee from 60 to 80. D1 still capped the edge Worker below both Postgres contenders. And the production gateway marks every feed response Cache-Control: private, no-store, so none of those cache gains exist in production today.

Faults: fail fast worked, database latency hurt everyone

On a replica kill nobody returned an error, because HAProxy's retries 1 and redispatch re-sent the in-flight requests. Latency did suffer: for the 3.8 to 4.5 s before the dead replica was marked down, requests waited out the 1 s connect timeout first.

On a 15 s database cut, Go answered in about 2 ms with a 503 upstream_unavailable instead of holding connections, readiness pulled the replicas, and it was back within a second of the link returning. That is the contract doing its job.

+100 ms on the database link separated the contenders most. Go absorbed it with zero errors and dropped 62 feed sections, naming each in degraded. Spring, at its own operating point, failed 41.6% of requests. With 32 pooled connections per replica and every statement holding one, Little's law puts the limit right around the offered load. I state plainly that the contenders ran at different loads, so this shows where each broke, not which runtime is more robust.

The most reusable lesson: measure the benchmark first

The first set of Go and Spring runs all "saturated near 80, Postgres-bound". That was false. The fault proxy sat between every gateway and its database at 0.5 CPU and 128 MB, ran at 100% CPU in every run, ran out of memory in the cache arm, and when it died the gateways logged hostname resolution errors. I found it only by reading the proxy's own CPU in the cgroup samples.

Fourteen runs were moved to results/_invalid_toxiproxy_cap/ and rerun at 2 CPU and 512 MB, where the proxy peaked at 23%. Other invalidated sets have their own folders too: a k6 bug, a baseline default that ignored saturation, and 52 runs lost after MinIO's upload prefixes filled the disk at 57 GB. The R analysis excludes them by name. The paper reports the CPU of the load generator, proxy, balancer and object store for every analysed run, so a reader can check that the measurement path did not bind.

How the numbers are computed

The R analysis (analysis/R/, ~900 lines) reproduces the harness's Python saturation rule exactly in all nine ramps, and the two agree on every RTO and MTTD to the millisecond. Saturation is reported under five definitions side by side so the choice is visible. RTO uses a rolling 10 s window, because at tens of requests per second a one-second p99 is just the single slowest request and a per-second rule almost never passes. Efficiency integrates cgroup samples over their own timestamps, which fixed an NA that a sum without na.rm had produced for Spring.

Honest gaps

• Single runs. Three repetitions were planned and only pass 1 completed. H0 cannot be rejected and no difference is called significant.
• 27 planned configurations were not run, including the Spring and Workers cache arms, most Workers faults, and the runtime-bound profile designed to push the gateways (not the database) to saturation. So the regime where a language difference in throughput would show was never reached.
• Local workerd is not Workers on Cloudflare, and D1 against Postgres compares platforms, not languages. The edge 500s are probably D1 overload, unconfirmed.
• The host ran other production services throughout. The caps limit interference, they do not remove it.