An open engineering notebook: real benchmarks, real costs, real failures — including the results that went against us. Every note states its method, links its evidence, and says what changed because of it.
The notebook
Opus 4.8 solo, Opus-in-Alepou, and an Opus-supervised GLM 5.2 worker build the same app from the same frozen spec. The cheap model wasn't cheap, the durable layer cost 9%, and our own orchestrator lost on cost.
The product is developed with its own memory, task board, supervisor runs, and task-to-commit audit trail. A curated tour of that history — resets, recoveries, and all.
The harder benchmark: adding a real feature to a large, convention-heavy codebase, where decomposition and coordination are part of the difficulty.
Field Note 002 · 2026-07-08
Two-fold. First: how much cheaper is an open-weight worker (GLM 5.2 via OpenRouter) than Opus 4.8 on the same real task? Sticker pricing says roughly 6×. Second: does an Opus supervisor measurably improve a cheaper worker's reliability, completion, and polish?
One frozen spec — "Notare", a local-first markdown knowledge base with deliberately awkward requirements (malformed-frontmatter recovery, duplicate-title slugs, wikilinks and backlinks, autosave state, ranked search, export). A seeded 12-task board, a 22-test acceptance suite, and a scoring script, all frozen before any arm ran. Each arm got an identical clone and the same working rules. Claude runs are priced at API-equivalent rates (they ran on subscription); GLM figures are actual OpenRouter spend.
| Arm | Configuration | Cost | Wall clock | Acceptance | UI rubric |
|---|---|---|---|---|---|
| A | Opus 4.8 solo — bare terminal | $7.77 | ~20 min | 22/22 | 6/6 |
| E | Opus 4.8 solo — inside Alepou | $8.45 | comparable | 22/22, self-added a 13th task + extra tests | 6/6 |
| C | Opus 4.8 supervisor + GLM 5.2 worker (orchestrated) | $17.23 | ~1h 40m | 22/22, zero rework cycles | 6/6, "a little plain" |
Every configuration shipped a fully working app. The differentiation was entirely in cost, time, and polish margins — which is exactly where it gets interesting.
Same model, same task, same output quality — bare terminal versus inside Alepou — differed by $0.68 (+8.8%). For that premium, the Alepou run produced a live task board with a commit and SHA per task (a clickable audit trail of the entire build), session continuity state, and — unprompted — a 13th task it identified and completed, with extra tests. The bare run left a working app. The Alepou run left a working app plus a legible, reversible record of how it was built.
GLM 5.2 lists ~6× cheaper than Opus 4.8. It cost 49% more ($11.60 vs $7.77) to do the same work, slower. The worker's per-turn token records show why: the prompt cache through the aggregator kept dying, and turns alternated between healthy (~800 fresh tokens + ~222k cached) and total misses that re-sent 147–224k tokens at full price. Across 286 turns: 8.27M uncached input tokens, 19.2M cached reads — and just 62.5k output tokens. Output price, the number everyone compares, was about 1% of the bill.
The cost model for agentic work is input volume × cache stability, not output sticker price. A first-party runtime with stable turn-over-turn caching beat a nominally 6×-cheaper model on real dollars.
The supervised run was our flagship configuration, and it came last on cost (2.2× solo). Reliability worked exactly as designed: zero rework cycles, every task accepted first pass, and the watchdog correctly identified a false "waiting for input" state without sending phantom input. The economics failed for identifiable reasons: the worker re-verified everything the supervisor also verified (a full test suite per task), the unstable worker-side cache multiplied every one of those outputs, and one watchdog wake shipped a ~150KB terminal buffer into the supervisor's context — a real bug.
What changed because of this note: the buffer bug is fixed (supervisor wakes are now hard-capped at ~4KB of sanitized tail), we removed the "1+1 is the sweet spot" claim from this very website, and orchestration is being redesigned around where it actually earns its keep — queues that outlive one context window, parallel workers for wall-clock speed, and unattended recovery — rather than per-token worker arbitrage.
Screen recordings of all three runs (including the full supervised run) land here shortly.
Per-arm scorecards, git logs with per-task commits, final board states, the worker session export with per-turn token accounting, OpenRouter spend records, and screen recordings. The benchmark harness (spec, fixtures, acceptance suite, scorer) is frozen at tag bench-v1.