Cost Series 03 — Caching as Architecture

Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: The Bill.

Caching isn't a flag you flip — it's a way of structuring prompts. Get the structure right and most of your input goes to 0.1×.

The hook

A cached token costs a tenth of a fresh one — but only if it's a prefix hit. That one constraint (cache matches from the front) turns prompt design into cache design: put what's stable first, what's volatile last, and keep the window warm.

What the viewer walks away with

• Prefix-first prompt design: stable system/context/tools up front, volatile user turn last — so the expensive part is the reused part
• The write break-even: a cache write costs ~1.25× (5-min) or ~2× (1-hr); it pays back after ~2 reads, so know your reuse count and TTL
• Warm-cache patterns: batch turns inside the TTL window; the 1-hr TTL as a lever for long/overnight sessions
• Cache is why "send more tokens" can be cheaper — the counterintuitive result from episode 01, explained

Beats

• The prefix-hit rule, drawn simply — why reordering a prompt changes its price
• Break-even math: writes are an investment; when it's worth it and when it isn't
• TTL as a knob: 5-min vs 1-hr, and matching it to session shape
• The A/B baton tie-in: identical cached prefix, so two runs start from the same state and only the variable differs

Demo ideas

• Reorder a prompt (volatile-first → stable-first) and watch the cached-read fraction jump
• A long session on the 1-hr TTL: cost per turn falling as the prefix stays warm

Source material

• The series index — the write/read/TTL multipliers
• The A/B baton — the same-prefix comparison rig
• The token-usage / cached-read analysis — real hit-rate curves

Open questions

• How much provider-specific caching mechanics (explicit breakpoints vs automatic) to cover vs keep general
• Show the break-even as a formula or just a rule of thumb ("reused twice → cache it")