Cost Series 06 — Cost per Success
Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: The Bill.
The finale: even cost is a proxy. The real objective is cost per success — and that's where this series meets the Fleet Series thesis.
The hook
A cheap model that fails, retries three times, and still needs a human fix is more expensive than one correct pass on the frontier model. Optimizing cost-per-call can quietly raise cost-per-outcome. The terminal metric isn't cost — it's cost per unit of useful, correct work.
What the viewer walks away with
- Cost per success > cost per call: retries, rework, and human cleanup are hidden costs the token bill doesn't show
- You can't optimize cost-per-success without an evaluator to define "success" — the billing-lens view of "the evaluator is the product"
- Cheap-first-then-escalate (episode 04) only works because a checker catches the cheap failures
- Optimize cost subject to a quality/latency constraint — not cost alone
Beats
- The trap: a cost-per-call win that's a cost-per-outcome loss (retries + human fix)
- Redefine the denominator: successes, not calls — and who decides success (the evaluator)
- Tie the series together: routing + caching + output discipline, all governed by "did it actually work?"
- Bridge to the Fleet Series: this is Fleet 05 (gates / evaluator) seen from the invoice; close on "price the work, and define the work"
Demo ideas
- Two pipelines to the same done-state: cheap-with-retries vs right-first-time — total cost per landed task
- Overlay the gate pass/fail on the cost chart — cost only counts when the work passes
Source material
- Fleet Series 05 — Gates: the Evaluator Is the Product — the sibling thesis
- The usage ledger + A/B baton — cost-per-landed-task measurement
- The coding pipeline's gate/verdict data — successes vs attempts
Open questions
- Does this ship as the finale or as the opener (reframing the whole series around success up front)? Leaning finale — it lands after the mechanics
- How to instrument "success" cheaply enough to divide by it routinely