Fleet Series 05 — Gates: the Evaluator Is the Product

Working notes for a planned episode — an outline, not an article. Details will shift when it gets scripted. Series index: Building the Fleet.

Deep dive on the gates: the fail-closed checks standing between "a model did a thing" and "we keep it". This is the thesis episode — the series' one idea, given a full treatment.

The hook

A model will cheerfully write the bug, then write the test that asserts the bug. "The tests pass" is not an evaluator. The real engineering in an autonomous fleet is deciding what may happen unsupervised — and making every one of those decisions fail closed.

What the viewer walks away with

• Gates, not autonomy: the hype word is "autonomous"; the actual work is the permission boundary
• Fail-closed as a design rule: any gate that can't run (suite missing, reviewer down, diff unreadable) blocks the merge rather than waving it through
• The layered defense: gate ("don't merge bad code") → human merge ("the line the agent doesn't cross") → sandbox ("even bad code can't hurt you") — three independent layers, each catching what the others miss
• Gates are cheap to specialize: the same sign-off pattern backs wave verification, testing auto-merge, and dependency-bump auto-merge
• Real catches beat theory: I keep a running log of gate catches (and escapes) precisely because "trust me, it works" is not a talk

Beats

• Open with the best gate catch from the log (candidate: the epic gate catching a no-op bug — code that ran green and did nothing)
• The evaluator-quality ladder: tests pass → suite gate with coverage of the actual change → multi-model review quorum → human sign-off, and what each rung costs
• The layered-defense day: one working day (July 11, 2026) where every layer earned its keep — walk it hour by hour
• Escapes matter too: what got through, and which layer should have caught it
• Close: the two questions to ask any agentic demo — what's your evaluator, and what happens when it's wrong?

Demo ideas

• Deliberately hand the pipeline a poisoned leaf (subtle bug + accomplice test) and watch which gate trips
• Show a gate failing closed for boring reasons (suite won't run) and why that's a feature

Source material

• The gate-catches anecdote log — the receipts; pull 3–4 best entries
• The layered-defense-day walkthrough notes (July 11, 2026)
• Lightning Talk — The Loop Is the Easy Part — two of its slides are this episode in miniature
• The testing auto-merge build — the 3-gate design and the shared auto-merge gate module

Open questions

• Does this ship as its own episode or fold into 01? Leaning own episode — it's the thesis and deserves the runtime
• Keep appending to the anecdote log at the moment catches happen (standing convention) so this episode stays fresh

Borrowed sharpenings — the checker law and the third stopping condition

From the loop-engineering crash course; both slot straight into existing beats.

• The checker ladder, stated as a law: a passing test is proof, mechanical checks are partial proof, a rubric score is a claim — and "the weaker your checker, the more work passes through the human gate." Checker strength is the knob that sets human-gate width; this upgrades the evaluator-quality-ladder beat from a list into a design law with a slider.
• Stopping conditions come in threes: success condition, limit, no-progress check. The named failure — the "Ralph loop" — has a success condition and an attempt cap but no stuck-check, so it repeats the same mistake until the budget dies. Great teachable failure; pair it with a gate-catch anecdote.
• Where the human sits: the human gate isn't a bolt-on — it's the feedback loop (hours) wrapping the coding loop (minutes). Spec and evals are how decisions made there travel inward.