Cost Series 03 — Caching as Architecture
Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: The Bill.
Caching isn't a flag you flip — it's a way of structuring prompts. Get the structure right and most of your input goes to 0.1×.
The hook
A cached token costs a tenth of a fresh one — but only if it's a prefix hit. That one constraint (cache matches from the front) turns prompt design into cache design: put what's stable first, what's volatile last, and keep the window warm.
What the viewer walks away with
- Prefix-first prompt design: stable system/context/tools up front, volatile user turn last — so the expensive part is the reused part
- The write break-even: a cache write costs ~1.25× (5-min) or ~2× (1-hr); it pays back after ~2 reads, so know your reuse count and TTL
- Warm-cache patterns: batch turns inside the TTL window; the 1-hr TTL as a lever for long/overnight sessions
- Cache is why "send more tokens" can be cheaper — the counterintuitive result from episode 01, explained
Beats
- The prefix-hit rule, drawn simply — why reordering a prompt changes its price
- Break-even math: writes are an investment; when it's worth it and when it isn't
- TTL as a knob: 5-min vs 1-hr, and matching it to session shape
- The A/B baton tie-in: identical cached prefix, so two runs start from the same state and only the variable differs
Demo ideas
- Reorder a prompt (volatile-first → stable-first) and watch the cached-read fraction jump
- A long session on the 1-hr TTL: cost per turn falling as the prefix stays warm
Source material
- The series index — the write/read/TTL multipliers
- The A/B baton — the same-prefix comparison rig
- The token-usage / cached-read analysis — real hit-rate curves
Open questions
- How much provider-specific caching mechanics (explicit breakpoints vs automatic) to cover vs keep general
- Show the break-even as a formula or just a rule of thumb ("reused twice → cache it")