Cost Series 06 — Cost per Success

Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: The Bill.

The finale: even cost is a proxy. The real objective is cost per success — and that's where this series meets the Fleet Series thesis.

The hook

A cheap model that fails, retries three times, and still needs a human fix is more expensive than one correct pass on the frontier model. Optimizing cost-per-call can quietly raise cost-per-outcome. The terminal metric isn't cost — it's cost per unit of useful, correct work.

What the viewer walks away with

• Cost per success > cost per call: retries, rework, and human cleanup are hidden costs the token bill doesn't show
• You can't optimize cost-per-success without an evaluator to define "success" — the billing-lens view of "the evaluator is the product"
• Cheap-first-then-escalate (episode 04) only works because a checker catches the cheap failures
• Optimize cost subject to a quality/latency constraint — not cost alone

Beats

• The trap: a cost-per-call win that's a cost-per-outcome loss (retries + human fix)
• Redefine the denominator: successes, not calls — and who decides success (the evaluator)
• Tie the series together: routing + caching + output discipline, all governed by "did it actually work?"
• Bridge to the Fleet Series: this is Fleet 05 (gates / evaluator) seen from the invoice; close on "price the work, and define the work"

Demo ideas

• Two pipelines to the same done-state: cheap-with-retries vs right-first-time — total cost per landed task
• Overlay the gate pass/fail on the cost chart — cost only counts when the work passes

Source material

• Fleet Series 05 — Gates: the Evaluator Is the Product — the sibling thesis
• The usage ledger + A/B baton — cost-per-landed-task measurement
• The coding pipeline's gate/verdict data — successes vs attempts

Open questions

• Does this ship as the finale or as the opener (reframing the whole series around success up front)? Leaning finale — it lands after the mechanics
• How to instrument "success" cheaply enough to divide by it routinely