Cost Series 02 — Measure the Right Thing

Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: The Bill.

You can't optimize what you don't measure — and a single "tokens used" number hides everything that matters.

The hook

Most LLM dashboards show one number: tokens. That's like measuring a road trip in "distance" while ignoring whether you were on a toll road, in a taxi, or on foot. The useful unit isn't tokens — it's cost per task, broken down by axis and by model.

What the viewer walks away with

• Instrument the four axes separately: fresh input, cache writes, cache reads, output — they price 50× apart
• Attribute cost to the unit of work (a solved task / a call), not the calendar month
• Track cache hit rate as a first-class metric — it's the difference between 1× and 0.1×
• Once you can see cost-per-task by model, the routing decision makes itself

Beats

• The "one big number" dashboard and why it's useless for optimizing
• Build the real breakdown: per-task, per-axis, per-model — from actual usage logs
• The metric that changes behavior: cache hit rate, and watching it climb as you restructure prompts
• Hand off to the levers: now that we can measure, we can move each one (episodes 03–05)

Demo ideas

• Live usage-ledger view: one epic's spend split by axis and by model, not just a total
• A/B baton run: same task, model A vs model B, cost-per-task side by side

Source material

• The usage ledger — per-run/per-epic token + spend accounting from the fleet's loop hardening
• The A/B baton — the rig for same-state cost comparisons
• The token-usage / cached-read analysis — the source charts

Open questions

• Cost-per-task needs a task boundary — lean on the task-tree leaf / wave as the unit?
• How to surface cache hit rate in the existing dashboards without adding noise