materializer + replay ports + vectord substrate fix verified at scale

Two threads landing together — the doc edits interleave so they ship
in a single commit.

1. **vectord substrate fix verified at original scale** (closes the
   2026-05-01 thread). Re-ran multitier 5min @ conc=50: 132,211
   scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix).
   Throughput dropped 1,115 → 438/sec because previously-broken
   scenarios now do real HNSW Add work — honest cost of correctness.
   The fix (i.vectors side-store + safeGraphAdd recover wrappers +
   smallIndexRebuildThreshold=32 + saveTask coalescing) holds at the
   footprint that originally surfaced the bug.

2. **Materializer port** — internal/materializer + cmd/materializer +
   scripts/materializer_smoke.sh. Ports scripts/distillation/transforms.ts
   (12 transforms) + build_evidence_index.ts (idempotency, day-partition,
   receipt). On-wire JSON shape matches TS so Bun and Go runs are
   interchangeable. 14 tests green.

3. **Replay port** — internal/replay + cmd/replay +
   scripts/replay_smoke.sh. Ports scripts/distillation/replay.ts
   (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL
   phase 7 live invocation on the Go side. Both runtimes append to the
   same data/_kb/replay_runs.jsonl (schema=replay_run.v1). 14 tests green.

Side effect on internal/distillation/types.go: EvidenceRecord gained
prompt_tokens, completion_tokens, and metadata fields to mirror the TS
shape the materializer transforms produce.

STATE_OF_PLAY refreshed to 2026-05-02; ARCHITECTURE_COMPARISON decisions
tracker moves the materializer + replay items from _open_ to DONE and
adds the substrate-fix scale verification row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

reports/cutover/multitier_100k.md

@@ -173,17 +173,26 @@ Add to `docs/ARCHITECTURE_COMPARISON.md` Decisions tracker:
 | Date | Decision | Effect |
 |---|---|---|
 | 2026-05-01 | playbook_record under load triggers coder/hnsw v0.6.1 nil-deref | **Recover guard added** in BatchAdd; daemon stays up. **Real fix open**: upstream patch OR small-index custom Add path OR alternate store. |
+| 2026-05-01 (later) | **Real fix landed.** vectord lifts source-of-truth out of coder/hnsw via `i.vectors map[string][]float32` side store; `safeGraphAdd`/`safeGraphDelete` recover panics; warm-path Add falls back to rebuild on failure; `rebuildGraphLocked` reads from the panic-safe side map. Re-ran multitier 60s/conc=50: **0 failures across 19,622 scenarios** (was 96-98% on 2/6). p50 on previously-failing scenarios moves 5ms (instant fail) → 551ms (real Add work — honest cost of correctness). Memory cost: ~2× for vectors. STATE_OF_PLAY captures the architecture invariant. |
+| 2026-05-02 | **Full-scale verification.** Re-ran multitier at the original failure-surfacing footprint (5min @ conc=50). Result: **132,211 scenarios at 438.5/sec, 0 failures across all 6 classes.** Throughput dropped from pre-fix 1,115/sec → 438/sec because previously-broken scenarios (96-98% fail) now do real HNSW Add work instead of fast nil-deref panics. Healthy tails: `surge_fill_validate` p50=28.9ms / p99=1.53s, `playbook_record_replay` p50=504ms / p99=2.32s — small-index rebuild kicking in under sustained churn, working as designed. **Substrate fix scales beyond the 19.6k-scenario probe; closing the open thread.** |
 
 ## Conclusion
 
-The Go substrate handles **335,257 multi-tier scenarios in 5 minutes**
-against a 100k corpus, with **4 of 6 scenario classes at 0% failure**
-and the remaining 2 exposing a real coder/hnsw v0.6.1 substrate bug
-that operators can recover from via DELETE + recreate.
+**Pre-fix (2026-05-01):** 335,257 scenarios in 5min, 4/6 classes at 0%
+failure, 2/6 hit a coder/hnsw v0.6.1 nil-deref under playbook record
+churn. Operator recovery via DELETE + recreate.
+
+**Post-fix (2026-05-02):** 132,211 scenarios in 5min @ conc=50,
+**6/6 classes at 0% failure**. Throughput moved 1,115/sec → 438/sec
+because the formerly fast-failing scenarios are now doing real HNSW
+Add work — that's the honest cost of correctness, not a regression.
+The fix (i.vectors side-store + safeGraphAdd recover wrappers +
+small-index rebuild threshold of 32 + saveTask write coalescing)
+shifts vectord's source-of-truth out of coder/hnsw so panics can't
+lose data and the daemon recovers automatically.
 
 This is the most production-shape test we've run. The harness mixes
 search, validator calls (in-process), HTTP cross-daemon round-trips,
-playbook recording (where the bug surfaces), and cache exercise. The
-result is more honest than a single-endpoint load test: 4 workflows
-work cleanly at scale, 1 has a bounded substrate issue with a known
-recovery path.
+playbook recording, and cache exercise. The result is more honest
+than a single-endpoint load test, and post-fix all six workflows
+work cleanly at scale.