Fleet Series 09 — The LLM Router: Local Models as a Utility

Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: Building the Fleet.

Deep dive on the LLM router: the homelab model fleet behind one OpenAI-compatible endpoint — the power supply the rest of the fleet plugs into.

The hook

Every agent in the fleet talks to "the model" through one URL on the local network. Behind that URL: a router, a rack of local models, aliases, and auto-routing. Swap a model, nothing upstream changes. That one seam is what makes an always-on agent fleet economically sane.

What the viewer walks away with

• The shape: one OpenAI-compatible endpoint on the local network, model aliases (auto, coder, …) instead of hardcoded model names, auto-routing picking the backend per request
• Why it exists: always-on loops burn tokens; ensembles multiply that by N; metered-API economics don't survive contact with a fleet
• The migration story: agents moved off hosted APIs onto the router one at a time, with evals deciding readiness — not vibes
• The eval harness: frozen baseline, distill targets, few-shot prompt distillation — how you know a local model is good enough for a given role
• Hard lessons: alias tiers matter (auto can resolve to a text-only model when an agent needs tool calls — use coder for pipeline leaves), and a Go rewrite of the always-on parts of the stack
• Honest split: local models for the high-volume routine work, frontier models where the ceiling matters

Beats

• Open with the bill that motivates everything (even hypothetically): what an ensemble-heavy day costs at API prices vs. watts
• Trace one request: agent → router → alias resolution → backend model → response, with the dashboard live
• The eval segment: run the eval suite against a new model, compare to the frozen baseline, promote or reject
• The "auto vs. coder" war story as the cautionary tale about aliases and capability tiers
• Close: the router as the reason every other episode's demos are affordable to run in a loop

Demo ideas

• Same prompt through auto vs. coder aliases, show the routing decision and capability difference
• Live dashboard during a pipeline wave: watch the fleet light up per leaf

Source material

• The router build itself: the Go rewrite, the tool proxy, the dashboard
• The eval/distillation work — frozen baseline and next-round targets
• The agent-migration log: which agents moved local, in what order, and what the evals said

Open questions

• Hardware tour: how much rack/GPU detail belongs here vs. a separate homelab video?
• Voice/RAG side quests — probably out of scope for this episode, maybe teased as adjacent content

Borrowed line — cost acceleration

The loop-engineering crash course makes the Jevons-paradox point crisply: better loops + cheaper tokens = more runs = faster burn. Cheap inference doesn't shrink the bill — it changes what you dare to run in a loop.

• Slots into the economics beat: the router is what makes ensemble-heavy days affordable, and exactly why caps still matter afterward — and the caps that matter are token/spend budgets, not just retry counts.
• The Cost Series picks this thread up and runs a whole track on it.