materializer + replay ports + vectord substrate fix verified at scale

Two threads landing together — the doc edits interleave so they ship
in a single commit.

1. **vectord substrate fix verified at original scale** (closes the
   2026-05-01 thread). Re-ran multitier 5min @ conc=50: 132,211
   scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix).
   Throughput dropped 1,115 → 438/sec because previously-broken
   scenarios now do real HNSW Add work — honest cost of correctness.
   The fix (i.vectors side-store + safeGraphAdd recover wrappers +
   smallIndexRebuildThreshold=32 + saveTask coalescing) holds at the
   footprint that originally surfaced the bug.

2. **Materializer port** — internal/materializer + cmd/materializer +
   scripts/materializer_smoke.sh. Ports scripts/distillation/transforms.ts
   (12 transforms) + build_evidence_index.ts (idempotency, day-partition,
   receipt). On-wire JSON shape matches TS so Bun and Go runs are
   interchangeable. 14 tests green.

3. **Replay port** — internal/replay + cmd/replay +
   scripts/replay_smoke.sh. Ports scripts/distillation/replay.ts
   (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL
   phase 7 live invocation on the Go side. Both runtimes append to the
   same data/_kb/replay_runs.jsonl (schema=replay_run.v1). 14 tests green.

Side effect on internal/distillation/types.go: EvidenceRecord gained
prompt_tokens, completion_tokens, and metadata fields to mirror the TS
shape the materializer transforms produce.

STATE_OF_PLAY refreshed to 2026-05-02; ARCHITECTURE_COMPARISON decisions
tracker moves the materializer + replay items from _open_ to DONE and
adds the substrate-fix scale verification row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

STATE_OF_PLAY.md

@@ -1,7 +1,7 @@
 # STATE OF PLAY — Lakehouse-Go
 
-**Last verified:** 2026-04-30 ~16:42 CDT
-**Verified by:** live probes + `just verify` PASS + multi-coord stress run #011 (full 9-phase scenario, 67 captured events, 1 Langfuse trace + 111 child observations covering every phase + every external call), not memory.
+**Last verified:** 2026-05-02 ~03:00 CDT
+**Verified by:** live probes + `just verify` PASS + multitier_100k **full-scale re-run on persistent stack** (132,211 scenarios across 5min @ conc=50, 0 failures across all 6 classes — was 4/6 at 0% pre-fix). Substrate fix (i.vectors side-store + safeGraphAdd + smallIndexRebuildThreshold=32 + saveTask coalescing) holds at original failure-surfacing footprint.
 
 > **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo); the Rust legacy at `/home/profit/lakehouse/` is maintenance-only. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
 
@@ -11,7 +11,7 @@
 
 ### Substrate (G0 + G1 family)
 
-13 service binaries under `cmd/` plus 2 driver scripts under `scripts/staffing_*` build into `bin/`. **18 smoke scripts all PASS.** `just verify` (vet + 30 packages × short tests + 9 core smokes) green in ~31s wall.
+13 service binaries under `cmd/` plus 2 driver scripts (`scripts/staffing_*`) and 3 distillation tools (`cmd/audit_full`, `cmd/materializer`, `cmd/replay`) build into `bin/`. **20 smoke scripts all PASS** (added `materializer_smoke.sh` + `replay_smoke.sh` 2026-05-02). `just verify` (vet + 32 packages × short tests + 9 core smokes) green in ~32s wall.
 
 | Binary | Port | What |
 |---|---|---|
@@ -50,6 +50,8 @@ Full ADR-004 surface shipped. **Cycle-detection + retired-trace exclusion proven
 
 - **E (partial)** at `57d0df1` — scorer + contamination firewall ported from Rust v1.0.0 (logic only per ADR-001 §1.4; not bit-identical).
 - **F (first slice)** at `be65f85` — drift quantification, scorer drift first.
+- **Materializer port** (2026-05-02) — `internal/materializer` + `cmd/materializer`. Ports `scripts/distillation/transforms.ts` (12 transforms) + `build_evidence_index.ts` (idempotency, day-partition, receipt). On-wire JSON shape matches TS so Bun and Go runs are interchangeable. 14 tests + `materializer_smoke.sh`.
+- **Replay port** (2026-05-02) — `internal/replay` + `cmd/replay`. Ports `scripts/distillation/replay.ts` (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL phase 7 live invocation on the Go side. Both runtimes append to the same `data/_kb/replay_runs.jsonl` (`schema=replay_run.v1`). 14 tests + `replay_smoke.sh`.
 
 ### chatd — Phase 4 (shipped 2026-04-30, scrum-hardened same day)
 
@@ -211,6 +213,8 @@ Verbatim verdicts at `reports/scrum/_evidence/2026-04-30/verdicts/`. Disposition
 - `temperature` is **omitted** for Anthropic 4.7 (handled by `Request.Temperature *float64`); don't re-add it.
 - chatd-smoke runs with **all cloud providers disabled** intentionally so the suite doesn't depend on API keys; that's why it can't catch B-3-class bugs (those need a fake-server fixture, see Sprint 0 follow-up).
 - **Langfuse Go-side client lives at `internal/langfuse/`** with best-effort fail-open posture. URL+creds from `/etc/lakehouse/langfuse.env`. Don't propose to "wire Langfuse on Go side" — it's wired; multi_coord_stress is the proof.
+- **vectord's source-of-truth is `i.vectors`, NOT the coder/hnsw graph.** The `Index` struct holds a parallel `vectors map[string][]float32` updated on every successful Add/Delete; the graph is a derived, replaceable view. `safeGraphAdd`/`safeGraphDelete` wrap the library's panic-prone ops; `rebuildGraphLocked` reads from `i.vectors` (graph-state-independent). Don't propose to "drop the side map for memory" — it's the load-bearing piece that makes Add panic-recoverable past the small-index threshold (closes the multitier_100k 277884b 96-98% fail). The prior `i.ids` set was folded into `i.vectors` keys.
+- **vectord saves are coalesced async, not synchronous.** `cmd/vectord/main.go` runs a per-index `saveTask` that single-flights through `Persistor.Save` — at most one in-flight + one pending. Add returns OK before the save completes; an Add-then-crash can lose ~1 save's worth of data, matching ADR-005's fail-open posture. Don't propose to "make saves synchronous for durability" — that re-introduces the lock-contention bottleneck (1-2.5s tail at conc=50, observed 2026-05-01) without fixing a real durability hole (in-memory state is the source of truth in flight).
 
 ---
 
@@ -276,6 +280,8 @@ a steady state. Future items will land here as production triggers fire.
 | (g5-slice) | **G5 cutover slice LIVE** (2026-05-01). First real Bun-frontend traffic reaching the Go substrate end-to-end. Bun mcp-server (`/home/profit/lakehouse/mcp-server/index.ts`) gains opt-in `/_go/*` pass-through to `$GO_LAKEHOUSE_URL` (set to `http://127.0.0.1:4110` via systemd drop-in). `/_go/v1/embed` returns nomic-embed-text-v2-moe vectors via Go embedd; `/_go/v1/matrix/search` returns 3/3 Forklift Operators against the persistent 200-worker corpus. Fully additive (no existing Bun tool modified) + fully reversible (unset env). `/api/*` (Rust gateway) path unchanged. See `reports/cutover/g5_first_slice_live.md`. |
 | (close-3) | **OPEN #3: distribution drift via PSI** — `internal/drift/drift.go`: `ComputeDistributionDrift` returns Population Stability Index + verdict tier (stable < 0.10, minor 0.10–0.25, major ≥ 0.25). Equal-width bucketing over combined min/max range, epsilon-clamping for empty buckets, per-bucket breakdown for drilldown. 7 new tests including identical-is-stable, hard-shift-is-major, moderate-detected-not-stable, empty-inputs-safe, all-identical-safe, bucket-counts-conserved, num-buckets-clamping. |
 | (close-4) | **OPEN #4: ops nice-to-haves** — (a) Real-time wall-clock for stress harness: per-phase elapsed time logged to stdout as it runs (`[stress] phase NAME starting (T+12.3s)` + `[stress] phase NAME done — 8.5s (T+20.8s)`); `Output.PhaseTimings` + `Output.TotalElapsedMs` written to JSON; (b) chatd fixture-mode S3 mock + (c) liberal-paraphrase calibration: not actioned — no fired trigger yet, would be speculative. Documented as deferred-until-need rather than ignored. |
+| (close-bug) | **coder/hnsw v0.6.1 panic — REAL FIX landed** (2026-05-01 ~22:25). The 277884b multitier_100k run hit 96-98% fail on 2/6 scenarios from a v0.6.1 nil-deref (`layerNode.search`) that fires when the graph transitions through degenerate states post-Delete. Initial recover() guard caught panics but returned errors at the same rate. **Real fix**: lift the source-of-truth out of coder/hnsw — `i.vectors map[string][]float32` side store maintained alongside the graph, panic-safe `safeGraphAdd`/`safeGraphDelete` wrappers, `rebuildGraphLocked` reads from `i.vectors` (independent of graph state), warm-path Add falls back to rebuild on panic. Side effect: `i.ids` collapsed into `i.vectors` keys; `Len()` reads from `len(i.vectors)`. Memory cost: ~2x for vectors. Verification: 7 new regression tests in `index_test.go` (`TestAdd_PastThreshold_SustainedReAdd` reproduces the multitier shape — 64-entry index, 800 upserts, 0 errors), `just verify` PASS, multitier_100k re-run on persistent stack 19,622 scenarios / 0 failures across all 6 classes. p50 on previously-failing scenarios went 5ms (instant fail) → 551ms (real Add work — honest cost of correctness). |
+| (perf-fix) | **Save coalescing — write-path lock contention closed** (2026-05-01 ~22:50). The panic fix exposed a second bottleneck: every successful Add called `Persistor.Save` synchronously, which takes the index RLock for `Encode` (~6MB JSON for 1942-entry × 768d) — blocking concurrent Add Lock acquisitions. 5min sustained run showed playbook scenario p50 climbing 551ms→1398ms as the index grew. **Fix**: `saveTask` per-index single-flight coalescer in `cmd/vectord/main.go` — `saveAfter` now triggers an async save; concurrent triggers during an in-flight save mark "pending" so N triggers collapse into ≤2 actual saves. RPO trade: Add returns OK before save completes (~1 save's worth of crash-loss exposure; same fail-open posture as ADR-005). Verification: 3 new tests in `cmd/vectord/main_test.go` (50-trigger pile-up → 2 saves; single → 1; error doesn't stall). Re-run: surge_fill_validate p50 1296ms→**47ms** (~28× faster), playbook_record_replay 1398ms→**385ms** (~3.6× faster), throughput 144→**668 scen/sec** at 0% fail. Restart-rehydrate verified — playbook_memory 4041 entries persisted to MinIO and round-tripped cleanly. |
 
 Plus on Rust side (`8de94eb`, `3d06868`): qwen2.5 → qwen3.5:latest backport in active defaults; distillation acceptance reports regenerated (run_hash refresh, reproducibility property still holds).