2026-08-18 — south_san_juans_3day: what we did by hand, and why none of it is a method¶

This build is not a template. It is an incident report.

south_san_juans_3day reached publication through five research passes, six prose passes, three manually-killed orchestrator runs, ten hand edits to files the tooling owns, and three tooling fixes written during the live build. It then shipped with three content defects that every gate passed. Kyle found all three by reading the page.

The report is correct now because a person read it, not because the pipeline produced it. A build that needs this much hand-steering has not succeeded; it has been carried. Every hand intervention below is a missing rail, a missing gate, or a missing contract — recorded here so the next build does not need any of them.

Part 1 — Hand edits to files the tooling owns¶

Each of these was Claude editing an artifact or input directly because no rail existed.

M-1. Kyle's route decision written into `peaks.yml` days[]¶

Did: wrote "KYLE'S DECISION 2026-08-18" as a comment in the bennett day block. Why it failed: days[] is the research agent's output, regenerated every pass. The next agent overwrote it and explicitly rejected the claim — "that entry was not in deviations.yml and was invalid" — which was correct behaviour. Cost: one full research pass (~14 min). Durable fix: a rail — scripts/record_decision.py --slug <s> --kind <k> --quote <...> — that writes deviations.yml and materializes it. Today the only way to record a Kyle decision is to hand-author YAML in the right file, and the wrong file looks identical.

M-2. `deviations.yml` entries appended by shell heredoc¶

Did: cat >> deviations.yml with hand-written YAML, three times, then deviation_kinds.py --materialize. Why it's wrong: the deviation registry has a validator and a materializer but no writer. Hand-authored YAML in the one file that overrides safety gates is the highest- stakes text in the repo to get wrong, and there is no schema help, no dry run, no diff. Durable fix: same rail as M-1. --materialize should be implicit in writing.

M-3. `class_source` URLs hand-added¶

Did: rewrote research.class_source in peaks.yml to turn climb13ers/colorado-13ers/bennett-peak into full https:// URLs. Why: check_beta_provenance requires a URL and fails at the prose gate — two phases after the research agent wrote the field, in a phase whose agent cannot fix it. The agent's formatting also varied run to run: pass 1 reported this gate clean, pass 4 failed it. Durable fix: run the provenance token/URL check at the research exit gate, and put the URL requirement in the research exit criteria. The agent has the beta cache with canonical URLs; it should never emit a bare path.

M-4. `status_source` hand-corrected to cite peak_db¶

Did: replaced peakbagger URLs with the peak_db resolution preflight actually used. Why: the research agent cited peakbagger peak.aspx pages for climbed status, but peakbagger only exposes a climber's ascents when logged in, and the headless agent reads a cached page with no session. The citation could not have been true. Durable fix: climbed status should be stamped by the tooling from peak_db (it is already resolved at preflight), never written by an agent. Same class as write_report.py's peak_db enrichment landed this session.

M-5. Trailhead elevation — caught by tooling, listed as the counter-example¶

The research agent carried ele_ft: 11837 from the old SE trailhead onto the new FS 265 one, 631 ft wrong. dem_elevation.py --apply corrected it to 11,206 automatically ("a trailhead's elevation is measured, not typed", daf1901). This is what every item above should look like. No human touched it; no one had to notice.

Part 2 — Manual diagnosis the tooling should have reported¶

M-6. Liveness determined with `ps aux`¶

Did: repeatedly ran ps aux | grep to prove the run was alive, because status.html said it had exited. Why: orchestrator_exited is stamped ~67 s into a subagent phase that runs 15–20 minutes. Kyle asked "is it frozen?" twice. A live run and a dead run produce byte- identical state. Durable fix: heartbeat from the spawned agent; status must distinguish running / stalled / exited, and say how long the current phase has been going.

M-7. Gain arithmetic checked by hand-sampling the DEM¶

Did: sampled dem_elevation.py at the trailhead and summit to prove a route's reported gain was below its own net rise. Why: nothing asserts the floor. It found a real defect (a route stopping short of the summit) and also produced a wrong conclusion I acted on — see M-12. Durable fix: a gate — a route's reported gain may not be less than summit_elevation − trailhead_elevation. Arithmetically impossible for a real walk.

M-8. Track shape determined by grepping GPX endpoints¶

Did: grepped first/last <trkpt> of four candidate files to learn which were one-way and which were round trips. Why: list_source_tracks.py reports length and coverage but not shape. Quoting a one-way length beside round-trip lengths sent Kyle a misleading comparison, and he chose a route on it. Durable fix: list_source_tracks.py reports shape (out-and-back / point-to-point / loop) and, for one-way lines, the implied round-trip distance.

M-9. Forensics for `validated_tracks` computed by hand¶

Did: counted trackpoints, measured step spacing, checked <time>/<ele> presence to build the evidence block admitting a timestamp-less track. Why: validated_tracks requires forensics but nothing produces them. The drawn- line test is a single binary signal — timestamps present or not — with no forensic tier between "recording" and "rejected". Durable fix: scripts/track_forensics.py <file> emitting the signature (point cadence, step distribution, jitter, which fields are absent together) as a paste-ready evidence block; and a recording-like classification the drawn-line gate can cite so a converted export is not indistinguishable from a freehand sketch.

M-10. Gates run out-of-band to pre-empt failures¶

Did: ran check_route_summits, check_beta_provenance, check_report_stats by hand mid-build to catch problems before the finalize gate did. Why: gates run at the end. A defect written in research surfaces two phases later, after ~40 minutes of work, in a phase that cannot fix it. Durable fix: run each gate at the earliest phase whose owner can fix what it checks.

Part 3 — Manual control of the orchestrator¶

M-11. Three runs killed by hand (`TaskStop`), then `--stopped` / `--resumed`¶

Did: killed the orchestrator three times: once to stop a build of a rejected route, once to break a non-terminating prose loop, once to apply a composer fix. Why the loop existed: stale-prose cleanup deletes the report on every prose re-entry, not just the first after a rewind. With the skeleton generator crashing, each cycle was: agent writes report (~10 min) → cleanup deletes it → generator crashes → gate says "report missing" → spawn agent → repeat. It would have run until the 60-minute timeouts, forever, producing nothing. Durable fix: clear the stale flag once the prose phase has run; and detect non-progress — same phase, same blocker, N consecutive agent runs — and halt with that stated, rather than spending agent runs indefinitely.

M-12. A wrong conclusion acted on, then caught by a gate¶

Did: told Kyle a gain shortfall was explained by a deliberate CalTopo-matching filter and was "not worth holding for". It was not: check_route_gap_loss found the route was losing 1.27 mi of 3.14 (40%) in segment breaks. Why it matters here: the reasoning was plausible and wrong, and the only thing that caught it was a gate. Every hand-diagnosis above carries this risk. This is the argument against manual verification as a method — not that it is slow, but that it is unreliable in a way gates are not.

Part 4 — Tooling changed during a live build¶

Three fixes were written while a build was in flight: write_report.py (peak_db enrichment), build_recommended_route.py (mapped-OSM compose), check_report_stats.py (day-bound stat lines). All were correct and all shipped with tests — but the build that produced the live report was running against a working tree that was not committed, so for several hours the published report could not be reproduced from a clean clone.

Durable fix: the orchestrator should refuse to start, or loudly mark the run, when scripts/ is dirty; and the build provenance should record the tooling commit it ran against. A report built by uncommitted code is not reproducible, and nothing said so.

Part 5 — What shipped broken anyway¶

Three defects passed every gate and were found by Kyle reading the published page:

Day 1's heading carried the trip total (22.0 mi / 7,539 ft) instead of Conejos' 6.0 / 2,008. check_report_stats allowed it because a body stat line need only match "the total or any day" — it could not tell which day a heading belonged to. Its own docstring named this exact failure as "never the fix" and then permitted it. Fixed this session: stat lines now bind to the day heading above them.
Every day's "Getting there" gave directions from Boulder, when a three-trailhead trip needs trailhead-to-trailhead legs. build_drive_route.py computed those legs correctly in this very run and printed them to the data log, where they were dropped.
The headline drive went to the wrong trailhead entirely — Treasure Creek, which is Day 3's start, 15 mi from Day 1's. build_context takes the first trailhead in the landmarks list, not Day 1's.

All three share one root cause: the generator has no concept of which trailhead belongs to which day. It emits a single Getting-there section for an arbitrarily-chosen trailhead, so the prose agent improvised the rest from the one example it could see.

Claude also dismissed defect 3 to Kyle twice as "a headline-TH artifact, not a route change" at the publish hold. It was not cosmetic: it is the direction line a person follows on the morning of the trip. Same failure mode as homestake_peak (check_drive_directions, 2026-08-16) — "I'm doing homestake today and the directions were wrong."

The standing rule Kyle set from this incident¶

A gate failure rewinds to the phase that can fix it. (Kyle, 2026-08-18: "Every gate failure should cause a rewind for a fix unless it's a judgment call.")

The orchestrator today re-runs the phase it is in when a gate fails. That is how the prose loop formed: status_source is written in RESEARCH and checked at the PROSE gate, so a research-phase defect re-spawned the prose agent — which cannot edit a research field. It reported nothing to do, burned a full agent run, and hit the same gate again.

Required behaviour:

Each gate declares the phase that OWNS the artifact it checks.
On failure, the orchestrator rewinds to that phase, carries the gate's verdict into that phase's task file, and re-runs.
It escalates to Kyle only when the fix is a judgment call — route choice, terrain grade, day grouping, a marker tie — or when there is no legitimate on-rails fix (CLAUDE.md defer trigger 1). A bounded, reasoned accept_* is a legitimate fix; sweeping knobs until green is not.
Non-progress is a halt: same phase, same blocker, N consecutive runs → stop and say so.

This subsumes two earlier items on this list — running each gate at the earliest phase whose owner can fix it, and carrying the blocker text into the re-spawned agent — and it is the single change that turns a silent 10-minute-per-cycle loop into a bounded failure.

The rule this incident buys¶

A report is finished when the pipeline produces it correctly, not when a person has corrected it enough times to look right. Every item above is a place where a human substituted for a rail. The measure of the fixes is simple:

Rebuild south_san_juans_3day from scratch with no hand edits, no killed runs, no out-of-band gate runs, and no deviations — and get this report.

Until that passes, this file stands and the route_shape deviation on this slug stays flagged as scaffolding, per Kyle: "I want us later to fix what didn't see this route as a legit route so we can remove the deviation and it builds more or less this route on its on."