real_006 diagnosis: Q43 leak is cross-city, not cross-role

Traced w-279's path through the substrate. The leak source is Q49
(Packers in Indianapolis IN for Midway Distribution), NOT Q18 as
the initial reading suspected.

Q49 recorded w-279 with role=Packers, client=Midway Distribution,
city=Indianapolis. Q43 (Packer in Chicago IL for Midway Distribution)
ran later. roleEqual("Packer","Packers") → both normalize to
"packer" → role gate passes (correctly, by design — they ARE the
same role under plural-strip). Cosine distance between Q49's
recorded query and Q43's query is small enough to fit inside the
0.20 inject threshold because role + client + count + time-token
dominate the embedding (only the city and singular/plural noun
differ). Inject fires, w-279 surfaces at Q43's warm top-1 in
Chicago, judge correctly rates 2/5 — wrong city.

The role gate IS working. What's missing is a CITY gate. Real_002's
fix targeted cross-role bleed (Forklift → CNC). real_006 surfaced
cross-city bleed within same role + same client — a hole prior
tests structurally couldn't reach because they all sourced from
rows 0-9 where no such pair existed.

Concrete fix surface documented (1 new field, 2 gate checks, 1
regex, ~5 tests). Half a session of work, same shape as real_002.
Not implementing tonight — diagnosis only.

The 18 unit-level role-gate tests still pass, confirming the gate
is doing what it was specified to do. The bug is a missing
specification, not a broken implementation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,37 @@
+# Match what `go build` actually needs; everything else bloats the
+# build context and slows COPY . . to a crawl.
+
+bin/
+out/
+.git/
+.github/
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Reality test outputs — regenerable, not needed in the image.
+reports/
+data/
+
+# Per-run smoke logs in /tmp obviously don't end up here, but defensive.
+*.log
+
+# Local config + secrets — must NEVER ship in an image.
+lakehouse.toml
+secrets-go.toml
+auth.env
+*.env
+
+# Editor swap files
+.DS_Store
+Thumbs.db
+
+# Existing systemd unit files don't belong in the container — the
+# image deploys via docker-compose, not systemd-inside-container.
+deploy/systemd/
+
+# Reality-test JSON corpora are large + the multi_coord_stress
+# binary will reference them as files, but they get mounted at
+# runtime, not baked in.
+tests/reality/contracts/*.json

.gitignore

@@ -34,7 +34,25 @@ vendor/
 /data/lance/
 /exports/
 /logs/
-/reports/
+# /reports/ holds runtime artifacts by default (matches Rust lakehouse
+# convention) — but reports/scrum/ is intentional audit documentation.
+# Use /reports/* + un-ignore so git can traverse into reports/.
+/reports/*
+!/reports/scrum/
+!/reports/reality-tests/
+!/reports/cutover/
+# Inside the audit directory, the per-run _evidence/ dump (smoke logs,
+# command output) IS runtime — track the dir, ignore its contents.
+/reports/scrum/_evidence/*
+!/reports/scrum/_evidence/.gitkeep
+# Reality-test JSON evidence is runtime — track the dir + MD reports
+# (committed deliberately as outcome record), ignore per-run JSON.
+/reports/reality-tests/*.json
+
+# Proof harness runtime output — same pattern as reports/scrum/_evidence.
+# Track the directory but ignore per-run subdirs.
+/tests/proof/reports/*
+!/tests/proof/reports/.gitkeep
 
 # Secrets — never commit. Resolved via SecretsProvider per ADR-001 §1.x.
 *.env

Dockerfile

@@ -0,0 +1,89 @@
+# syntax=docker/dockerfile:1.6
+#
+# Multi-stage Dockerfile for Lakehouse-Go.
+#
+# Single image carries all 11 daemon binaries; docker-compose runs
+# one container per daemon (matches the systemd unit topology in
+# deploy/systemd/). Operators can also `docker run lakehouse-go
+# /usr/local/bin/lakehouse/<daemon>` to invoke any one daemon
+# directly.
+#
+# Builder uses golang:1.25-bookworm (DuckDB cgo needs gcc + glibc;
+# alpine's musl doesn't link the official duckdb-go bindings cleanly).
+# Runtime is debian:bookworm-slim — same libc, much smaller surface.
+#
+# Build:
+#   docker build -t lakehouse-go:latest .
+# Or with a tag:
+#   docker build -t lakehouse-go:$(git rev-parse --short HEAD) .
+
+# ── Stage 1: builder ────────────────────────────────────────────
+FROM golang:1.25-bookworm AS builder
+
+# build-essential pulls gcc + make + libc-dev — DuckDB cgo needs all three.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      build-essential \
+      ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /src
+
+# Copy go.mod + go.sum first so module download is cacheable across
+# source-only changes.
+COPY go.mod go.sum ./
+RUN go mod download
+
+# Source.
+COPY . .
+
+# Build all 11 daemon binaries + the staffing_workers script (used
+# by the multi_coord_stress harness; ships in the same image so
+# operators can run reality tests against a deployed stack).
+RUN go build -trimpath -o /out/ \
+      ./cmd/storaged ./cmd/catalogd ./cmd/ingestd ./cmd/queryd \
+      ./cmd/embedd ./cmd/vectord ./cmd/pathwayd ./cmd/observerd \
+      ./cmd/matrixd ./cmd/gateway ./cmd/chatd \
+      ./scripts/staffing_workers ./scripts/playbook_lift ./scripts/multi_coord_stress
+
+# ── Stage 2: runtime ────────────────────────────────────────────
+FROM debian:bookworm-slim
+
+# CA certs for outbound HTTPS (Ollama Cloud, OpenRouter, OpenCode,
+# Kimi). curl + jq for in-container health checks + smoke probes.
+# tini handles PID 1 signal forwarding so docker stop sends SIGTERM
+# to the actual daemon, not just to a wrapper.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      ca-certificates \
+      curl \
+      jq \
+      tini \
+    && rm -rf /var/lib/apt/lists/*
+
+# Non-root runtime user — same name as the systemd User= directive
+# in deploy/systemd/, so file ownership stays consistent across
+# deployment modes (docker-compose vs systemd).
+RUN groupadd --system --gid 999 lakehouse \
+ && useradd --system --uid 999 --gid 999 \
+            --no-create-home --shell /usr/sbin/nologin lakehouse
+
+# Layout matches /usr/local/bin/lakehouse/<daemon> from REPLICATION.md
+# so docs apply equally to systemd + docker deployments.
+COPY --from=builder /out/* /usr/local/bin/lakehouse/
+
+# /var/lib/lakehouse for pathway/observer JSONLs; /var/log/lakehouse
+# in case operators want file logs in addition to docker logs.
+RUN mkdir -p /var/lib/lakehouse/pathway /var/lib/lakehouse/observer /var/log/lakehouse \
+ && chown -R lakehouse:lakehouse /var/lib/lakehouse /var/log/lakehouse
+
+USER lakehouse
+WORKDIR /var/lib/lakehouse
+
+# No default CMD — operators (or docker-compose) MUST specify which
+# daemon. Forces explicit topology rather than implicit "run
+# everything in one container."
+ENTRYPOINT ["/usr/bin/tini", "--"]
+
+# Default healthcheck targets gateway's port. Per-service compose
+# overrides land per their own port.
+HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --retries=3 \
+  CMD curl -sSf http://127.0.0.1:3110/health || exit 1

README.md

@@ -38,6 +38,28 @@ real-scale validation + G1/G1P/G2 pointer at the bottom).
 | `queryd` | 3214 | DuckDB SELECT over registered Parquets via httpfs |
 | `vectord` | 3215 | HNSW vector search (+ optional persistence to storaged) |
 | `embedd` | 3216 | Text → vector via Ollama (default `nomic-embed-text` 768-d) |
+| `mcpd`    | stdio | Model Context Protocol server (Claude Desktop / Code consumers) |
+
+## MCP server
+
+`bin/mcpd` exposes Lakehouse capabilities as MCP tools over stdio:
+`list_datasets`, `get_manifest`, `query_sql`, `embed_text`, `search_vectors`.
+All tools proxy to the gateway, so the gateway must be up first.
+
+Wire into Claude Desktop / Claude Code by adding to the MCP config:
+```json
+{
+  "mcpServers": {
+    "lakehouse": {
+      "command": "/path/to/golangLAKEHOUSE/bin/mcpd",
+      "args": ["--gateway", "http://127.0.0.1:3110"]
+    }
+  }
+}
+```
+
+Replaces the Bun `mcp-server.ts` MCP-tool surface from the Rust system.
+HTTP demo routes (the staffing co-pilot UI) stay Bun until G5.
 
 ## Acceptance smokes
 
@@ -53,20 +75,38 @@ scripts/g1p_smoke.sh  # vectord state survives kill+restart via storaged
 scripts/g2_smoke.sh   # embed → vectord add → search round-trip
 ```
 
-Run them all in any order:
+Or run the full gate via the task runner (see below):
 ```
-for s in scripts/{d1,d2,d3,d4,d5,d6,g1,g1p,g2}_smoke.sh; do "$s" || break; done
+just verify     # vet + tests + 9 smokes; ~33s wall
 ```
 
+## Task runner
+
+```
+just                 # show available recipes
+just verify          # full Sprint 0 gate (vet + tests + 9 smokes)
+just smoke <day>     # single smoke (d1..d6, g1, g1p, g2)
+just doctor          # check cold-start deps; --json for CI
+just install-hooks   # install pre-push hook that runs just verify
+```
+
+After a fresh clone, run `just install-hooks` once so `git push` is
+gated on the same green chain that ran here. Hook lives in
+`.git/hooks/pre-push` (not tracked; recreated by the recipe).
+
 ## Cold-start dependencies
 
 - Go 1.25+ at `/usr/local/go/bin` (arrow-go pulled the 1.25 floor)
 - `gcc` + `libc-dev` for the DuckDB cgo binding (ADR-001 §1.1)
+- `just` task runner (`apt install just` on Debian 13+)
 - MinIO running on `:9000` with bucket `lakehouse-go-primary`
 - Ollama running on `:11434` with `nomic-embed-text` loaded (G2)
 - `/etc/lakehouse/secrets-go.toml` with `[s3.primary]` credentials
   (storaged + queryd both read this)
 
+`just doctor` probes all of the above and reports the fix command
+for each missing dep. CI / scripts can use `just doctor --json`.
+
 ## Layout
 
 ```

REPLICATION.md

@@ -0,0 +1,264 @@
+# Lakehouse-Go — Replication Runbook
+
+How to deploy Lakehouse-Go onto a fresh Linux host. Mirrors the layout
+the dev box uses; covers prereqs, secrets, systemd units, validation.
+
+## Prereqs
+
+The host needs these external services reachable BEFORE the Lakehouse
+daemons can usefully start. None are managed by Lakehouse-Go's own
+units; they're operator infrastructure.
+
+| Service | Purpose | Reachability |
+|---|---|---|
+| **Go 1.25+** | builds the binaries | `go version` returns ≥ 1.25 |
+| **gcc** | DuckDB cgo (queryd) | `gcc --version` |
+| **MinIO** (or AWS S3) | storaged backing store | `curl http://localhost:9000/minio/health/live` returns 200; bucket `lakehouse-go-primary` exists |
+| **Ollama** | embedd + chatd LLM dispatch | `curl http://localhost:11434/api/tags` returns 200 with `nomic-embed-text-v2-moe` (or whatever `[embedd].default_model` names) loaded |
+| **Langfuse** *(optional)* | trace + span observability | `curl http://localhost:3001/api/public/health` returns 200 |
+| **PostgreSQL** *(optional)* | only if Langfuse is wanted | bundled with the Langfuse docker compose |
+
+Bind ports the daemons use (G0 dev defaults; shifted by 10 from the
+Rust legacy on 3100/3201–3204 so both stacks coexist):
+
+| Daemon | Port |
+|---|---:|
+| gateway | 3110 |
+| storaged | 3211 |
+| catalogd | 3212 |
+| ingestd | 3213 |
+| queryd | 3214 |
+| vectord | 3215 |
+| embedd | 3216 |
+| pathwayd | 3217 |
+| matrixd | 3218 |
+| observerd | 3219 |
+| chatd | 3220 |
+
+## Bootstrap
+
+### 1. User + directories
+
+```bash
+sudo useradd --system --no-create-home --shell /usr/sbin/nologin lakehouse
+sudo mkdir -p /var/lib/lakehouse/{pathway,observer} /var/log/lakehouse \
+              /usr/local/bin/lakehouse /etc/lakehouse
+sudo chown -R lakehouse:lakehouse /var/lib/lakehouse /var/log/lakehouse
+```
+
+### 2. Build + install binaries
+
+From a clone of the repo:
+
+```bash
+git clone https://git.agentview.dev/profit/golangLAKEHOUSE.git
+cd golangLAKEHOUSE
+just verify    # vet + tests + 9 core smokes — ~31s
+go build -o bin/ ./cmd/...   # 11 binaries land in ./bin/
+sudo cp bin/{gateway,storaged,catalogd,ingestd,queryd,vectord,embedd,pathwayd,observerd,matrixd,chatd} /usr/local/bin/lakehouse/
+sudo chmod 755 /usr/local/bin/lakehouse/*
+```
+
+### 3. Config + secrets
+
+```bash
+# Main config — edit ports/URLs/model tier as needed
+sudo cp lakehouse.toml /etc/lakehouse/lakehouse.toml
+
+# S3 credentials — fill in real keys
+sudo cp deploy/etc-lakehouse/secrets-go.toml.example /etc/lakehouse/secrets-go.toml
+sudo chown root:lakehouse /etc/lakehouse/secrets-go.toml
+sudo chmod 0640 /etc/lakehouse/secrets-go.toml
+sudo $EDITOR /etc/lakehouse/secrets-go.toml  # set [s3.primary] keys
+
+# Auth token — required ONLY if any daemon binds non-loopback
+sudo cp deploy/etc-lakehouse/auth.env.example /etc/lakehouse/auth.env
+sudo chown root:lakehouse /etc/lakehouse/auth.env
+sudo chmod 0640 /etc/lakehouse/auth.env
+# For non-loopback deploys, set:
+#   AUTH_TOKEN=<generate via `openssl rand -hex 32`>
+sudo $EDITOR /etc/lakehouse/auth.env
+
+# Optional: Langfuse traces. When set, every authenticated HTTP
+# request to every daemon emits a trace + span (production
+# observability per OPEN item #2 closure). Missing file = no
+# traces, no warnings.
+sudo cp deploy/etc-lakehouse/langfuse.env.example /etc/lakehouse/langfuse.env
+sudo chown root:lakehouse /etc/lakehouse/langfuse.env
+sudo chmod 0640 /etc/lakehouse/langfuse.env
+sudo $EDITOR /etc/lakehouse/langfuse.env  # set URL + PUBLIC_KEY + SECRET_KEY
+
+# Optional: chatd cloud provider keys, one file per provider
+# (each is its own EnvironmentFile so rotations don't restart all chatd)
+for provider in ollama_cloud openrouter opencode kimi; do
+  echo "${provider^^}_API_KEY=" | sudo tee /etc/lakehouse/$provider.env > /dev/null
+  sudo chown root:lakehouse /etc/lakehouse/$provider.env
+  sudo chmod 0640 /etc/lakehouse/$provider.env
+done
+sudo $EDITOR /etc/lakehouse/openrouter.env  # etc per provider you need
+```
+
+### 4. systemd units
+
+```bash
+sudo cp deploy/systemd/*.service deploy/systemd/*.target /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable lakehouse-go.target
+sudo systemctl start lakehouse-go.target
+```
+
+### 5. Validation
+
+```bash
+# All 11 daemons should be active
+systemctl status 'lakehouse-*.service' --no-pager | grep -E "Active|●"
+
+# Health endpoints respond on each port
+for port in 3110 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220; do
+  printf "%5d: " "$port"
+  curl -sS --max-time 2 "http://127.0.0.1:$port/health" || echo "FAIL"
+done
+
+# Through the gateway: all chatd providers register (cloud keys present)
+curl -sS http://127.0.0.1:3110/v1/chat/providers | jq
+
+# End-to-end: ingest a tiny CSV → queryd SELECT → matrix.search
+echo -e "id,name,role\n1,Alice,Forklift Operator" > /tmp/probe.csv
+curl -sS -F "file=@/tmp/probe.csv" "http://127.0.0.1:3110/v1/ingest?name=probe"
+curl -sS -X POST http://127.0.0.1:3110/v1/sql \
+  -H 'content-type: application/json' \
+  -d '{"sql":"SELECT COUNT(*) FROM probe"}' | jq
+```
+
+## Auth posture
+
+Per ADR-006:
+
+- **Loopback-only deploy** (every daemon binds 127.0.0.1): no auth needed. Empty `AUTH_TOKEN` is fine. Network is the boundary.
+- **Non-loopback deploy** (gateway exposed beyond loopback, daemons internal-private): set `AUTH_TOKEN` in `/etc/lakehouse/auth.env`. The mechanical gate at startup refuses to bind without one.
+- **Multi-host deploy** (gateway + daemons on separate machines): set `AUTH_TOKEN` *and* `[auth].allowed_ips` in lakehouse.toml to the gateway's address. Both layers gate.
+- **TLS**: terminate at nginx/Caddy in front of the gateway. The Go daemons speak HTTP; in-process TLS is explicitly out of scope per ADR-006 Decision 6.6.
+
+## Token rotation
+
+Per ADR-006 Decision 6.5 — dual-token window:
+
+```bash
+# 1. Generate new token
+NEW=$(openssl rand -hex 32)
+
+# 2. Add as secondary, keep old as primary
+sudo sed -i "s|^AUTH_SECONDARY_TOKEN=.*|AUTH_SECONDARY_TOKEN=$NEW|" /etc/lakehouse/auth.env
+sudo systemctl restart lakehouse-go.target
+
+# 3. Update every caller to use NEW token
+# 4. Promote: NEW becomes primary, secondary clears
+sudo sed -i "s|^AUTH_TOKEN=.*|AUTH_TOKEN=$NEW|" /etc/lakehouse/auth.env
+sudo sed -i "s|^AUTH_SECONDARY_TOKEN=.*|AUTH_SECONDARY_TOKEN=|" /etc/lakehouse/auth.env
+sudo systemctl restart lakehouse-go.target
+```
+
+## Docker / docker-compose deploy (alternative to systemd)
+
+The single-image `Dockerfile` carries all 11 daemons; `docker-compose.yml`
+runs one container per daemon with the same dependency graph as the
+systemd units. Useful when the host doesn't have systemd (Mac dev
+boxes, remote VMs without root) or when you want all of Lakehouse-Go
+isolated to a private docker network.
+
+```bash
+# Build the image (multi-stage; ~3 min on first build, ~30s with
+# cached go module download).
+docker build -t lakehouse-go:latest .
+
+# Place config + secrets next to docker-compose.yml. The compose file
+# bind-mounts these into every container at /etc/lakehouse/.
+cp lakehouse.toml lakehouse.toml          # already in repo; edit if needed
+cp deploy/etc-lakehouse/secrets-go.toml.example secrets-go.toml
+chmod 0600 secrets-go.toml
+cp deploy/etc-lakehouse/auth.env.example auth.env
+chmod 0600 auth.env
+# Per-provider chatd keys (each its own file so missing == provider
+# unregistered, NOT chatd startup failure):
+for p in ollama_cloud openrouter opencode kimi; do
+  echo "${p^^}_API_KEY=" > $p.env
+  chmod 0600 $p.env
+done
+
+# $EDITOR each file to fill in real values...
+
+# Bring up the stack.
+docker compose up -d
+docker compose ps    # all 11 services Healthy
+docker compose logs -f gateway
+
+# Validate via the gateway like the systemd path.
+curl -sS http://127.0.0.1:3110/v1/chat/providers | jq
+
+# Tear down.
+docker compose down
+# State volume (pathway/observer JSONLs) survives `down`. To wipe:
+docker compose down -v
+```
+
+### Key docker-vs-systemd differences
+
+| Concern | systemd | docker-compose |
+|---|---|---|
+| Process supervision | systemd | tini + docker daemon |
+| Logs | journald | `docker logs` (or routed to a sink via logging driver) |
+| Restarts on failure | `Restart=on-failure` | `restart: unless-stopped` |
+| File ownership | `User=lakehouse` (uid varies) | `user: 999:999` (uid is fixed in the image) |
+| Reaches MinIO/Ollama | host network | host's address from inside the bridge network — typically `host.docker.internal` (Mac/Win) or `172.17.0.1` (Linux). Set `[s3].endpoint` + `[embedd].provider_url` accordingly. |
+| Backup target | `/var/lib/lakehouse/` on host | the `lakehouse-state` named volume; bind to a host path via the commented-out `driver_opts` in compose if needed |
+
+## Logs
+
+systemd routes everything to journald with per-daemon SyslogIdentifier:
+
+```bash
+journalctl -u lakehouse-gateway.service -f
+journalctl -u 'lakehouse-*.service' --since '5 min ago'
+```
+
+## Stopping
+
+```bash
+sudo systemctl stop lakehouse-go.target  # cascades to all 11 daemons
+```
+
+## Backup / state preservation
+
+| Path | What | Backup priority |
+|---|---|---|
+| `/var/lib/lakehouse/pathway/state.jsonl` | Mem0 trace store (append-only) | high |
+| `/var/lib/lakehouse/observer/ops.jsonl` | observer ring's persistor backup | medium |
+| MinIO `lakehouse-go-primary` bucket | parquets, vector LHV1 indexes, catalog manifests | high |
+| `/etc/lakehouse/lakehouse.toml` | service config | medium |
+| `/etc/lakehouse/secrets-go.toml` + `*.env` | secrets | high (in your secrets manager, not on disk) |
+
+## Troubleshooting
+
+**Daemon refuses to start with "refuse non-loopback bind without auth.token"**
+ADR-006 6.1 mechanical gate. Set `AUTH_TOKEN` in `/etc/lakehouse/auth.env` or bind back to loopback.
+
+**Daemon refuses to start with "refusing non-loopback bind ... see audit R-001"**
+The previous loopback-bind gate. For dev: `LH_<NAME>_ALLOW_NONLOOPBACK=1` overrides. For prod: set `AUTH_TOKEN` AND keep the override (or move to loopback + reverse-proxy).
+
+**catalogd 500 / NoSuchBucket**
+storaged is pointing at a bucket that doesn't exist. Either create the bucket in MinIO or fix `[s3].bucket` in lakehouse.toml.
+
+**embedd 502 on /v1/embed**
+Ollama not running OR `[embedd].default_model` not loaded. `ollama list` to verify; `ollama pull nomic-embed-text-v2-moe` to load.
+
+**chatd `/v1/chat/providers` shows `false` for cloud providers**
+The provider's env file is missing or empty. Check `/etc/lakehouse/<provider>.env`.
+
+**queryd unable to read parquet**
+Check `[queryd].secrets_path` points at the right secrets-go.toml AND the file's owner+mode allow the lakehouse user to read.
+
+## Related docs
+
+- `STATE_OF_PLAY.md` — verified-working snapshot
+- `docs/DECISIONS.md` — all ADRs, especially ADR-003 (auth substrate) + ADR-006 (auth posture)
+- `docs/SPEC.md` §1 — component table

STATE_OF_PLAY.md

@@ -0,0 +1,351 @@
+# STATE OF PLAY — Lakehouse-Go
+
+**Last verified:** 2026-05-02 ~11:30 CDT
+**Verified by:** **production-readiness gauntlet** — 21/21 smoke chain green, per-component scrum across 4 bundles, **6 cross-runtime parity probes all green: 38/38 byte-identical assertions** (validator 6/6 after wire-format alignment, materializer 2/2 after omitempty fix, extract_json 12/12, session_log 4/4, embed 8/8 cosine, subject_audit 6/6 after the three runtime-divergence-class fixes documented at `lakehouse/STATE_OF_PLAY.md` lines 36-39). All findings surfaced by the parity probes have been actioned. Disposition: `reports/cutover/gauntlet_2026-05-02/disposition.md`. Subject-audit probe added 2026-05-03 (`262a77a`).
+
+> **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo) but ASK if ambiguous — the Rust system at `/home/profit/lakehouse/` is also receiving real work as of 2026-05-02 (Lance backend hardening, observability parity, sidecar drop — see `lakehouse/STATE_OF_PLAY.md`). The cross-runtime decisions tracker at `docs/ARCHITECTURE_COMPARISON.md` is the single source of truth for what's open across both. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
+
+---
+
+## VERIFIED WORKING RIGHT NOW
+
+### Substrate (G0 + G1 family)
+
+14 service binaries under `cmd/` plus 2 driver scripts (`scripts/staffing_*`) and 3 distillation tools (`cmd/audit_full`, `cmd/materializer`, `cmd/replay`) build into `bin/`. **21 smoke scripts all PASS** (added `validatord_smoke.sh` 2026-05-02). `just verify` (vet + 33 packages × short tests + 9 core smokes) green in ~32s wall.
+
+| Binary | Port | What |
+|---|---|---|
+| `gateway` | 3110 | reverse proxy, single OpenAI-compat-style edge |
+| `storaged` | 3211 | S3 GET/PUT/LIST/DELETE w/ per-prefix PUT cap (ADR-002) |
+| `catalogd` | 3212 | Parquet manifests, ADR-020 idempotent register |
+| `ingestd` | 3213 | CSV → Parquet → catalogd, content-addressed keys |
+| `queryd` | 3214 | DuckDB SELECT over Parquet via httpfs |
+| `vectord` | 3215 | HNSW indexes (coder/hnsw), persistence to storaged |
+| `embedd` | 3216 | Ollama-backed embedder w/ LRU cache |
+| `pathwayd` | 3217 | Mem0 ops (Add/Update/Revise/Retire/History/Search) |
+| `matrixd` | 3218 | Multi-corpus retrieve+merge + relevance + downgrade + playbook |
+| `observerd` | 3219 | Witness loop, workflow runner with DAG executor |
+| `chatd` | 3220 | LLM dispatcher: ollama / ollama_cloud / openrouter / opencode / kimi |
+| `validatord` | 3221 | `/validate` (FillValidator + EmailValidator + PlaybookValidator) + `/iterate` (gen→validate→correct loop). Roster from JSONL. |
+| `mcpd` | — | MCP SDK port (Bun mcp-server replacement) |
+| `fake_ollama` | — | Test fixture (used by `g2_smoke_fixtures.sh`) |
+
+### Matrix indexer — all 5 SPEC §3.4 components shipped
+
+1. **Corpus builders** (`internal/corpusingest`)
+2. **Multi-corpus retrieve+merge** (`matrixd /matrix/search`)
+3. **Relevance filter** (`internal/matrix/relevance.go` 376 LoC + 289 LoC test)
+4. **Strong-model downgrade gate** (`internal/matrix/downgrade.go`, reads `cfg.Models.WeakModels` after Phase 2)
+5. **Playbook memory: boost + Shape B inject** (`internal/matrix/playbook.go`, learning loop). Shape B (`InjectPlaybookMisses`, `154a72e`) appends recorded answers to results when regular retrieval misses them — closed the paraphrase recovery gap exposed by run #002 (0/2) and validated by run #003 (6/6).
+
+### Pathway memory (Mem0 substrate)
+
+Full ADR-004 surface shipped. **Cycle-detection + retired-trace exclusion proven by tests:** `TestHistory_CycleDetected`, `TestRetire_ExcludedFromSearch`, `TestRevise_ChainOfThree_BackwardWalk`. JSONL append-only persistence with corruption tolerance.
+
+### Observer + workflow runner
+
+- `observerd` ring buffer + JSONL persistence
+- Workflow DAG executor (Archon-style) with 5 native modes wired: `matrix.relevance`, `matrix.downgrade`, `matrix.search`, `distillation.score`, `drift.scorer`. Plus `fixture.echo` / `fixture.upper` for runner mechanics smokes.
+
+### Distillation + drift
+
+- **E (partial)** at `57d0df1` — scorer + contamination firewall ported from Rust v1.0.0 (logic only per ADR-001 §1.4; not bit-identical).
+- **F (first slice)** at `be65f85` — drift quantification, scorer drift first.
+- **Materializer port** (2026-05-02) — `internal/materializer` + `cmd/materializer`. Ports `scripts/distillation/transforms.ts` (12 transforms) + `build_evidence_index.ts` (idempotency, day-partition, receipt). On-wire JSON shape matches TS so Bun and Go runs are interchangeable. 14 tests + `materializer_smoke.sh`.
+- **Replay port** (2026-05-02) — `internal/replay` + `cmd/replay`. Ports `scripts/distillation/replay.ts` (retrieve → bundle → /v1/chat → validate → log). Closes audit-FULL phase 7 live invocation on the Go side. Both runtimes append to the same `data/_kb/replay_runs.jsonl` (`schema=replay_run.v1`). 14 tests + `replay_smoke.sh`.
+
+### chatd — Phase 4 (shipped 2026-04-30, scrum-hardened same day)
+
+Multi-provider LLM dispatcher routing `/v1/chat` by model-name prefix or `:cloud` suffix:
+
+| Prefix / suffix | Provider | Auth |
+|---|---|---|
+| `ollama/<m>` or bare | `ollama` (local) | none |
+| `ollama_cloud/<m>` or `<m>:cloud` | `ollama_cloud` | Bearer (OLLAMA_CLOUD_KEY) |
+| `openrouter/<v>/<m>` | `openrouter` | Bearer (OPENROUTER_API_KEY) |
+| `opencode/<m>` | `opencode` | Bearer (OPENCODE_API_KEY) |
+| `kimi/<m>` | `kimi` | Bearer (KIMI_API_KEY) |
+
+All 5 keys live in `/etc/lakehouse/{ollama_cloud,openrouter,opencode,kimi}.env` files (mode 0600). Empty/missing files leave that provider unregistered (404 at first call instead of 503). Test request: `POST /v1/chat {"model":"opencode/claude-opus-4-7","messages":[{"role":"user","content":"hi"}],"max_tokens":8}`.
+
+`Request.Temperature` is `*float64` (pointer) — Anthropic 4.7 deprecates `temperature` entirely, so we omit the field when caller doesn't set it.
+
+### Model tier registry
+
+`lakehouse.toml [models]` names model IDs by tier so swaps are 1-line:
+
+```toml
+local_fast       = "qwen3.5:latest"
+local_judge      = "qwen2.5:latest"   # NOT qwen3.5 — vision-SSM 256K-ctx ran 30s/judge in lift loop
+cloud_judge      = "kimi-k2.6:cloud"
+cloud_review     = "qwen3-coder:480b"
+frontier_review  = "openrouter/anthropic/claude-opus-4-7"
+frontier_arch    = "openrouter/moonshotai/kimi-k2-0905"
+frontier_free    = "opencode/claude-opus-4-7"
+weak_models      = ["qwen3.5:latest", "qwen3:latest"]   # matrix.downgrade bypass
+```
+
+Callers read `cfg.Models.LocalJudge` etc. instead of literal strings. `playbook_lift` harness, `matrix.downgrade`, and observerd's `MatrixDowngradeWithWeakList` factory all migrated.
+
+### Code health
+
+- `go vet ./...` → **0 warnings, 0 errors**
+- `go test -short ./...` → **all green**, 349 test functions
+- `just verify` → PASS (vet + tests + 9 smokes) in ~31s
+- 18 smoke scripts (9 core gating verify + 9 domain smokes for new daemons)
+
+### Latest scrum: 2026-04-30 cross-lineage wave
+
+Composite **50/60** at scrum2 head `c7e3124` (was 35 baseline → 43 R1 → 50 R2). Today's chatd wave reviewed by Opus + Kimi + Qwen3-coder via the chatd's own `/v1/chat`; **2 BLOCKs + 2 WARNs landed as fixes** (`0efc736`); reusable driver at `scripts/scrum_review.sh`.
+
+### Reality tests #001–#003 — load-bearing gate verified (2026-04-30 ~05:50–07:05 CDT)
+
+The 5-loop substrate's load-bearing gate (per `project_small_model_pipeline_vision.md`: *"the playbook + matrix indexer must give the results we're looking for"*) is verified for both **verbatim replay** and **paraphrase queries**.
+
+| Run | Stance | Verbatim lift | Paraphrase recovery | What it proved |
+|---|---|---|---|---|
+| `playbook_lift_001` | boost-only | **7/8 (87.5%)** | not tested | Cosine + boost re-rank works for verbatim replay. Substrate live. |
+| `playbook_lift_002` | boost-only | 2/2 | **0/2** | Boost can't promote answers OUT of regular top-K — paraphrase gap exposed. |
+| `playbook_lift_003` | Shape B (loose 0.5) | 2/6 | 6/6 → top-1 | Shape B injects, but cross-pollinates: w-4435 surfaces as warm top-1 for unrelated OOD queries (dental/RN/SWE). |
+| `playbook_lift_004` | **Shape B + split threshold (0.5 boost / 0.20 inject)** | **6/8 (75%)** | **6/8 (75%)** | OOD cross-pollination GONE; system refuses to inject when it's not confident. The honest configuration. |
+
+**Shape B** (`InjectPlaybookMisses` in `internal/matrix/playbook.go`): when warm-pass retrieval doesn't already include a playbook hit's answer, append a synthetic Result with distance = `playbook_hit_distance × BoostFactor`. Caller re-sorts + truncates. Documented at `playbook.go:22-27` since v0; v3 shipped the implementation. v4 added the split-threshold defense (`DefaultPlaybookMaxInjectDistance = 0.20` while boost stays at 0.50) — boost is safe at loose thresholds because it only re-ranks results already in retrieval; inject is structurally riskier so its threshold is tighter.
+
+OOD honesty (dental hygienist / RN / software engineer queries) holds across all three runs — judge rates them 1, system doesn't fabricate matches. Cross-corpus boosts (e- ↔ w- swaps) confirmed in v1 + v3.
+
+Evidence: `reports/reality-tests/playbook_lift_{001,002,003}.{json,md}`. Per the report's rubric (lift ≥ 50% = matrix doing real work), 6/6 paraphrase recovery is the validation that matters — verbatim replay is structurally the easy case.
+
+**v3 → v4 is the configuration evolution.** v3 ran with one threshold (0.5) for both boost and inject — paraphrase recovery hit 6/6 but a single strong recording (w-4435 for "OSHA-30 forklift Wisconsin") cross-pollinated to OOD queries (dental hygienist / RN / software engineer) because their text vectors fell within 0.5 cosine. v4 split: boost stays at 0.5 (safe — only re-ranks existing results), inject tightens to 0.20 (true paraphrase territory). Result: OOD cross-pollination eliminated, paraphrase recovery 6/8, verbatim lift 6/8. The two paraphrases that missed (Q9, Q15) were liberally rephrased — drift > 0.20 cosine.
+
+### Multi-coordinator stress test (Phase 1 → 3) — VERIFIED end-to-end
+
+Reality test #2 catalog. New harness `scripts/multi_coord_stress.{sh,go}` simulates 3 coordinators (alice/bob/carol) handling 3 distinct contracts (Milwaukee distribution / Indianapolis manufacturing / Chicago construction), each with their own playbook namespace. 9-phase operational narrative across simulated Hours 0–48: baseline → fresh-resume injection → inbox burst → mid-day surge → 200-worker swap → contract merge → handover (verbatim + paraphrase) → split → reissue.
+
+| Capability | Verified | Where |
+|---|---|---|
+| Per-coordinator playbook isolation | ✓ | `playbook_alice` / `playbook_bob` / `playbook_carol` corpora |
+| Same-role-across-contracts diversity | Jaccard 0.026 (n=9) — 97% workers differ per region | Phase 1 baseline |
+| Different-roles-same-contract diversity | Jaccard 0.070 (n=18) — 93% differ per role | Phase 1 baseline |
+| HNSW retrieval determinism | Jaccard 1.000 (n=12) | Phase 6 reissue |
+| Verbatim handover (Bob runs Alice's queries with Alice's playbook) | 4/4 | Phase 4 |
+| Paraphrase handover (Bob runs qwen2.5-paraphrased queries) | 4/4 | Phase 4b |
+| 200-worker swap with `ExcludeIDs` | Jaccard 0.000 — 8/8 placed workers fully replaced | Phase 2b |
+| Fresh-resume injection (two-tier `fresh_workers` index) | 3/3 fresh workers at top-1 | Phase 1b |
+| Inbox endpoint `/v1/observer/inbox` (email + SMS, priority weighting) | 6/6 events recorded | Phase 1c |
+| LLM demand parsing (qwen2.5 format=json on inbox bodies) | 6/6 parsed cleanly into structured `{role, count, location, certs, skills, shift}` | Phase 1c |
+| Judge re-rates inbox top-1 against ORIGINAL body | catches tight-distance-but-wrong (Q3 crane case: dist 0.23 → rating 1) | Phase 1c |
+| Langfuse Go-side tracing | 111 observations on a single run trace, browseable at http://localhost:3001 | Run #011 |
+
+**Substrate gains added by this wave:**
+- `internal/matrix/playbook.go` Shape B + split inject threshold (commit `67d1957` from earlier wave; verified in multi-coord too)
+- `internal/matrix/retrieve.go` `ExcludeIDs` field on `SearchRequest` — filters worker IDs at retrieve, boost, AND inject (so excluded answers can't sneak back via playbook). Real-world driver: coordinator placed N workers, client asks for replacements.
+- `internal/observer/types.go` `SourceInbox` taxonomy alongside `SourceMCP / SourceScenario / SourceWorkflow`
+- `cmd/observerd` `POST /observer/inbox` route — accepts `{type, sender, subject, body, priority, tag}` and records as `ObservedOp`. Type must be `email` or `sms`; body required; priority defaults to medium.
+- `internal/langfuse/client.go` — minimal Go-side Trace+Span client, best-effort posture (logs on error, never blocks calling path; same fail-open semantics as ADR-005 Decision 5.1).
+- `embedd default_model` bumped from `nomic-embed-text` (137M) → `nomic-embed-text-v2-moe` (475M, MoE, drop-in 768d). Same-role-across-contracts diversity went from 0.080 → 0.000 with the upgrade.
+- Two-tier index pattern: fresh content goes to `fresh_workers` (a small "hot" corpus); main queries include it in the corpora list. Solves the HNSW post-build add recall issue (incremental adds to a populated graph land in poorly-connected regions and disappear from search). Canonical NRT pattern; Lucene works the same way.
+
+### Harness expansion (2026-04-30 ~05:30 CDT)
+
+`scripts/playbook_lift.sh` rewritten from a 5-daemon stripped harness to the **full 10-daemon prod-realistic stack** (chatd stays up independently). The 5-daemon version was structurally hiding bugs; expanding the daemon set surfaced 7 distinct fixes:
+
+| # | Fix | Lock |
+|---|---|---|
+| 1 | driver→matrixd: `query` → `query_text` field name | `cmd/matrixd/main_test.go` TestPlaybookRecord_OldFieldNameRejected |
+| 2 | harness toml missing `[s3]` block | inline comment in `scripts/playbook_lift.sh` |
+| 3 | harness→queryd: `q` → `sql` field name | `cmd/queryd/main_test.go` TestHandleSQL_WrongFieldName_400 |
+| 4 | 5→10 daemon boot order | inline comment + dep-ordered launch |
+| 5 | SQL surface probe (3-row CSV → COUNT=3) | `[lift] ✓ SQL surface probe passed` assertion |
+| 6 | `candidates` corpus was SWE-tech, not staffing | swapped to `ethereal_workers.parquet` (10K rows, real staffing schema, "e-" id prefix) |
+| 7 | `qwen3.5:latest` is vision-SSM 256K-ctx → 30s/judge | reverted `local_judge` to `qwen2.5:latest` (1s/judge, 30× faster) |
+
+### R-005 closed (2026-04-30 ~05:35 CDT)
+
+Four new `cmd/<bin>/main_test.go` files — chi router-level contract tests:
+
+- `cmd/matrixd/main_test.go` (123 lines) — playbook record drift detector + score bounds + 6 routes mounted
+- `cmd/queryd/main_test.go` (extended) — wrong-field-name drift detector
+- `cmd/pathwayd/main_test.go` (102 lines) — 9 routes + add round-trip + retire-nonexistent
+- `cmd/observerd/main_test.go` (98 lines) — 4 routes + invalid-op 400 + unknown-mode 400
+
+`go test ./cmd/{matrixd,queryd,pathwayd,observerd}` all green. R-005 from prior STATE OPEN list is closed.
+
+---
+
+## DO NOT RELITIGATE
+
+### Ratified ADRs (`docs/DECISIONS.md`)
+
+- **ADR-001**: DuckDB via cgo, HTMX UI, Gitea hosting, distillation rebuilt-not-ported, pathway memory clean start, auditor longitudinal signal restarts. **6 sub-decisions, all final.**
+- **ADR-002**: storaged per-prefix PUT cap (4 GiB for `_vectors/`, 256 MiB elsewhere) — implemented at `423a381`. Operator-config bump rather than constant change is the documented path if 4 GiB ever insufficient.
+- **ADR-003**: Inter-service auth = Bearer + IP allowlist, opt-in via `cfg.Auth.Token`. Wiring deferred to Sprint 1 but **the design is locked** — alternatives (mTLS, JWT, OAuth2, IP-only) all considered + rejected.
+- **ADR-004**: Pathway memory = Mem0 versioned traces, JSONL append-only persistence, opaque `json.RawMessage` content. Implemented in `internal/pathway/`.
+
+### Today's scrum dispositions (2026-04-30)
+
+Verbatim verdicts at `reports/scrum/_evidence/2026-04-30/verdicts/`. Disposition table: `reports/scrum/_evidence/2026-04-30/disposition.md`.
+
+**Real findings, all fixed in `0efc736`:**
+- B-1 (Opus+Kimi convergent): `ResolveKey` 3-arg API → 2-arg
+- B-2 (Opus+Kimi convergent): `handleProviders` direct map lookup, drop synthesis-via-Resolve
+- B-3 (Opus single, trace-verified): `OllamaCloud.Chat` strips `ollama_cloud/` prefix correctly
+- B-4 (Opus single): Ollama `done_reason` surfaced to FinishReason
+
+**False positives dismissed (3, documented):**
+- FP-A1: Kimi misread `TestMaybeDowngrade_WithConfigList` assertion
+- FP-A2: Qwen claimed nil-deref in `MaybeDowngrade` that doesn't exist
+- FP-C1: Opus claimed `qwen3.5:latest` doesn't exist on Ollama hub (it does on this box's local install)
+
+### Session frame (don't redo)
+
+- The Rust system is **NOT maintenance-only** as of 2026-05-02 — it's receiving parity work + infrastructure (Lance gauntlet, sidecar drop, observability parity). Don't propose ports of components already shipped on the Go side, but do expect Rust commits to land in parallel for cross-runtime fabric.
+- The matrix indexer **5/5 components** are shipped. Don't propose to "build the matrix indexer" — it's done.
+- The 5-loop substrate's load-bearing gate is **PASSED**. v3 (`154a72e`) showed 6/6 paraphrase recovery via Shape B. Don't propose to "test if the playbook learns" — it does.
+- **Shape B is the playbook stance now.** When `use_playbook=true`, both `ApplyPlaybookBoost` (re-rank in place) AND `InjectPlaybookMisses` (insert recorded answers not in regular top-K) run. Don't revert to boost-only; v2 proved that path can't recover paraphrase queries.
+- **Boost / inject use SEPARATE thresholds.** Boost stays at `DefaultPlaybookMaxDistance = 0.5` (safe — only re-ranks results already in retrieval). Inject uses tighter `DefaultPlaybookMaxInjectDistance = 0.20` (Shape B forces results in, so loose match cross-pollinates wrong-domain). Don't merge them.
+- **Multi-coord product theory is empirically VALIDATED** by run #011 (Phase 3). Per-coordinator playbook namespaces (`playbook_alice` etc.) with cross-coordinator handover (Bob takes Alice's contract using `playbook_corpus=playbook_alice`) work end-to-end including paraphrase handover. Don't propose to "test if multi-coord works" — it does.
+- **Auth posture is locked per ADR-006.** Non-loopback bind requires `auth.token` (mechanical gate at `shared.Run`). Operators set the token via `token_env` (defaults to `AUTH_TOKEN`) loaded by systemd `EnvironmentFile=/etc/lakehouse/auth.env`, NOT in the committed TOML. Internal services use `AllowedIPs`; external boundary uses Bearer. Token rotation is dual-token via `secondary_tokens`. TLS terminates at the edge (nginx/Caddy), not in-process. Don't re-litigate.
+- **Shape B inject has a judge-gate substrate.** `InjectPlaybookMisses` takes an optional `InjectGate` (interface) that approves each candidate before the rank insert. `LLMJudgeGate` (Ollama-shape /api/chat client) is the default impl; nil gate = pre-judge-gating distance-only behavior preserved for backward compat. Caller wires via `SearchRequest.{JudgeURL, JudgeModel, JudgeMinRating}`. Closes the lift-suite tail issues (Q6↔Q7 adjacent-query swap + Q9/Q15 paraphrase drift) at substrate level.
+- **Fresh content uses two-tier indexing.** Fresh resumes go to `fresh_workers` corpus, not the main `workers` index. coder/hnsw incremental adds to a populated 5K+ graph have recall issues; the small hot index has no such crowding. Periodic merge (post-G3) consolidates fresh→main. Canonical NRT pattern.
+- **`embedd.default_model = "nomic-embed-text-v2-moe"`** (475M MoE, 768d). Don't bump to `nomic-embed-text` (137M) "for speed" — diversity scores fell from 0.000 → 0.080 same-role-across-contracts on the smaller model. Cost (~5× slower ingest) is acceptable for once-per-deploy work.
+- **Inbox flow: parse + search + judge + trace.** `/v1/observer/inbox` records the body; coordinator/driver parses via LLM (qwen2.5 format=json), runs `matrix.search` on the parsed query, judges top-1 against the ORIGINAL body, emits Langfuse spans through it all. Don't replace the judge re-rate with a distance-only gate — tight distance + low rating is the load-bearing honesty signal.
+- `local_judge = "qwen2.5:latest"` for the lift loop — qwen3.5:latest is a vision-SSM 256K-ctx build that ran 30s/judge. Don't bump to qwen3.5 again "for cleanliness." Different model in the same tier has different cost geometry.
+- `qwen3.5:latest` IS available locally on this box. Opus's hub-only knowledge is a known-stale signal; the chatd_smoke uses it daily.
+- `temperature` is **omitted** for Anthropic 4.7 (handled by `Request.Temperature *float64`); don't re-add it.
+- chatd-smoke runs with **all cloud providers disabled** intentionally so the suite doesn't depend on API keys; that's why it can't catch B-3-class bugs (those need a fake-server fixture, see Sprint 0 follow-up).
+- **Langfuse Go-side client lives at `internal/langfuse/`** with best-effort fail-open posture. URL+creds from `/etc/lakehouse/langfuse.env`. Don't propose to "wire Langfuse on Go side" — it's wired; multi_coord_stress is the proof.
+- **vectord's source-of-truth is `i.vectors`, NOT the coder/hnsw graph.** The `Index` struct holds a parallel `vectors map[string][]float32` updated on every successful Add/Delete; the graph is a derived, replaceable view. `safeGraphAdd`/`safeGraphDelete` wrap the library's panic-prone ops; `rebuildGraphLocked` reads from `i.vectors` (graph-state-independent). Don't propose to "drop the side map for memory" — it's the load-bearing piece that makes Add panic-recoverable past the small-index threshold (closes the multitier_100k 277884b 96-98% fail). The prior `i.ids` set was folded into `i.vectors` keys.
+- **vectord saves are coalesced async, not synchronous.** `cmd/vectord/main.go` runs a per-index `saveTask` that single-flights through `Persistor.Save` — at most one in-flight + one pending. Add returns OK before the save completes; an Add-then-crash can lose ~1 save's worth of data, matching ADR-005's fail-open posture. Don't propose to "make saves synchronous for durability" — that re-introduces the lock-contention bottleneck (1-2.5s tail at conc=50, observed 2026-05-01) without fixing a real durability hole (in-memory state is the source of truth in flight).
+- **`X-Lakehouse-Trace-Id` header propagates Langfuse parent traces across daemon boundaries.** When validatord's `/v1/iterate` calls chatd's `/v1/chat`, it forwards the header so chatd's middleware reuses the parent trace id instead of minting a new one. Each iteration attempt also emits a child span (`iterate.attempt[N]`) carrying the prompt, raw model output, and validator verdict. Result: an iterate session with N retries shows in Langfuse as ONE trace tree, not N+1 disconnected traces. Don't propose to "wire trace-id propagation" — it's wired; the test at `internal/shared/langfuse_middleware_test.go::TestLangfuseMiddleware_HonorsTraceIDHeader` is the proof. Closes J's 2026-05-02 multi-call observability concern.
+- **Coordinator session JSONL exists at `[validatord].session_log_path`.** One row per `/v1/iterate` session, schema=`session.iterate.v1`, append-only. Captures session_id (= Langfuse trace_id), kind, model, iterations, attempts[], final_verdict, **grounded_in_roster** (the load-bearing forensic property — always true on accepted fills, false on phantom-worker artifacts that somehow slip past FillValidator). Empty path = disabled. DuckDB queries documented at `docs/SESSION_LOG.md`. Don't propose to "build offline session analysis" — it's built; the schema is locked at v1; rotate the file via logrotate when it gets large.
+
+---
+
+## OPEN — what's not done yet
+
+The list is intentionally short. Items move to closed when the work demands them, not on a calendar. Ordered by leverage on the active product theory (multi-coord staffing co-pilot via the 5-loop substrate), not by effort.
+
+**All 4 prior OPEN items closed (substrate or fully) in the 2026-04-30
+"fix the other 4" wave.** No new items pending; the substrate is in
+a steady state. Future items will land here as production triggers fire.
+
+---
+
+## RECENT VERIFIED WAVE (2026-04-30)
+
+`05273ac..e4ee002` — 4 phases + scrum + tooling, all gate-tested.
+
+| SHA | What |
+|---|---|
+| `ec1d031` | Phase 1: `[models]` tier config (additive, no callers migrate) |
+| `622e124` | Phase 2: `matrix.downgrade` reads `cfg.Models.WeakModels` |
+| `848cbf5` | Phase 3: `playbook_lift` harness defaults from config |
+| `05273ac` | Phase 4: chatd + 5 providers (1,624 LoC) |
+| `0efc736` | Scrum: 4 fixes (B-1..B-4) + 2 INFOs from cross-lineage review |
+| `e4ee002` | `scripts/scrum_review.sh` — reusable 3-lineage driver |
+| `b2e45f7` | playbook_lift harness expansion + reality test #001 (7/8 lift, 87.5%) |
+| `6c02c90` | scrum lift_001: 4 fixes (sleep→polling SQL probe, JUDGE_SOURCE template, -id-prefix validation, chi.Router cast) |
+| `2c71d1c` | ADR-005: observer fail-safe semantics |
+| `9ce067b` | observerd: test that locks ADR-005 5.3 (provenance recorded post-run) |
+| `e9822f0` | playbook_lift v2: paraphrase pass — exposed boost-only limit (0/2 paraphrase recovery) |
+| `154a72e` | matrix: Shape B (`InjectPlaybookMisses`) — 6/6 paraphrase recovery in run #003 |
+| `94fc3b6` | STATE_OF_PLAY: capture v3 reality test + Shape B + cross-pollination |
+| `67d1957` | matrix: split boost/inject thresholds (0.5 / 0.20) — kills cross-pollination, run #004 6/8+6/8 |
+| `b13b5cd` | playbook_lift v4 metric: warm-top-1 re-judge → quality lift +24%/-14% (5 lifted / 13 neutral / 3 regressed) |
+| `61c7b55` | multi-coord stress harness (Phase 1) — 3 coords / 3 contracts / 7-phase scenario |
+| `0fa42a0` | multi-coord stress Phase 1.5 — shared-role contracts + paraphrase handover |
+| `84a32f0` | multi-coord stress Phase 2 — `ExcludeIDs`, 200-worker swap, fresh-resume |
+| `4da32ad` | embedd: bump default to nomic-embed-text-v2-moe (475M MoE, 768d drop-in) |
+| `e7fc63b` | observerd `/observer/inbox` + multi-coord stress phase 1c (priority-ordered events) |
+| `186d209` | multi_coord_stress: LLM-parsed inbox demands (qwen2.5 format=json) |
+| `ce940f4` | multi_coord_stress: judge re-rates inbox top-1 against original body — recovers OOD honesty signal |
+| `7e6431e` | langfuse: Go-side client + Phase 1c instrumentation |
+| `08a0867` | multi_coord_stress: fresh_workers two-tier index — fresh-resume now top-1 (3/3) |
+| `5d49967` | multi_coord_stress: full Langfuse coverage — every phase + every call (111 observations) |
+| `68d9e55` | shared: auto-emit Langfuse trace+span per HTTP request — closes OPEN #2 |
+| `a2fa9a2` | scrum_review: pipe diff via temp files — fixes argv overflow on large bundles |
+| (prep)    | G5 cutover prep: `embed_parity` probe — Rust `/ai/embed` ↔ Go `/v1/embed` 5/5 cos=1.000 (both v1 and v2-moe). Verdict + drift catalog in `reports/cutover/SUMMARY.md`. Wire-format remap (`embeddings`/`vectors`, `dimensions`/`dimension`) is the only real cutover work; math is provably equivalent. |
+| (probe)   | Reality test real_001: 10 real-shape queries from `fill_events.parquet` through lift harness. 8/10 cold-pass top-1 = judge-best (substrate works on real distribution). Surfaced **same-client+city cross-role bleed** — Shape A boost from Forklift-Operator playbook landed on CNC-Operator query, demoting the cold-pass-correct worker. Findings: `reports/reality-tests/real_001_findings.md`. |
+| (fix)     | Cross-role gate: `Role` on `PlaybookEntry`, `QueryRole` on `SearchRequest`, gate fires in both `ApplyPlaybookBoost` + `InjectPlaybookMisses`. `roleEqual` handles case + plural. Backward-compat: empty role on either side = gate disabled (preserves lift suite + free-form callers). 5 new unit tests use exact real_001 distance + role values. Re-run real_002: bleed closed (Q#5 Pickers, Q#10 CNC Operator stay at cold-pass top-1; same-role lifts still fire). Closes OPEN #1. Findings: `reports/reality-tests/real_002_findings.md`. |
+| (probe)   | Reality test real_003: 40 queries (10 fill_events rows × 4 styles — `need` / `client_first` / `looking` / `shorthand`). Confirmed shorthand-vs-shorthand bleed is real: CNC Operator shorthand recording leaked `w-2404` onto Forklift Operator shorthand query (both empty role, gate disabled). Extended `extractRoleFromNeed` to handle `client_first` + `looking` patterns; shorthand stays empty (regex can't separate role from city without anchor). Re-run real_003b: bleed closed across all 4 styles in this dataset. 10 new sub-tests in `scripts/playbook_lift/main_test.go` lock the patterns + the documented shorthand limitation. Findings: `reports/reality-tests/real_003_findings.md`. |
+| (fix)     | LLM-based role extractor (real_004): `roleExtractor` struct with regex-first → qwen2.5 format=json fallback → per-process cache. Opt-in via `-llm-role-extract` flag + `LLM_ROLE_EXTRACT=1` env. Off-by-default preserves real_003b shipping config. 8 new tests including `TestRoleExtractor_ClosesCrossRoleShorthandBleed` — the load-bearing witness pairing with the matrix-side `TestInjectPlaybookMisses_RoleGateRejectsCrossRole` to prove the extraction-layer + gate-layer compose correctly on the exact real_003 failure mode. Findings: `reports/reality-tests/real_004_findings.md`. |
+| (scrum)   | 3-lineage scrum review on `7f2f112..0331288` (Opus + Kimi + Qwen3-coder via `scripts/scrum_review.sh`). Convergent finding (3/3): `roleNormalize` plural-stripper mangled non-plural-s tokens (Sales → Sale, Logistics → Logistic). **Fixed**: `nonPluralSWords` allowlist + `-ss` ending check + `strings.ToLower`/`TrimSpace` cleanup. New tests `TestRoleNormalize_NonPluralS` + `TestRoleEqual_NonPluralS` lock the edge cases. Kimi 2 BLOCKs were false positives (model-truncation artifacts per `feedback_cross_lineage_review.md`). Disposition: `reports/scrum/_evidence/2026-04-30/verdicts/role_gate_v1_disposition.md` (local). |
+| (probe)   | Negation reality test real_005: 5 explicit-negation queries ("NOT in Detroit", "excluding Cornerstone roster", etc.). Confirmed substrate has **zero negation handling** — cosine treats "NOT X" as "X" + noise. Judge IS the safety net (Q1/Q3/Q4 rated all top-10 results 1-2/5 — operator-visible honesty signal). **No code change needed**: production UI should handle exclusion via `ExcludeIDs` (already supported, added in multi-coord stress 200-worker swap), not via NL-negation. Findings: `reports/reality-tests/real_005_findings.md`. |
+| (wire-up) | Multi-coord stress role wire-through: `Demand.Role` was already extracted at every call site (44 occurrences) but never threaded into matrix retrieve or playbook record. Cross-role gate was bypassed for the entire multi-coord harness. **Fixed** by extending `tracedSearch`, `matrixSearch`, and `playbookRecord` signatures with `role string` and updating all 14 call sites — passing `d.Role` (demand loops), `parsed.Role` (LLM-parsed inbox path), `warehouseDemand.Role` (swap path), `ev.Role` (reissue path), `""` (fresh-verify resume snippet — no clean role). Build + vet + tests green; multi-coord stress now honors role gate end-to-end. |
+| (close-1) | **OPEN #1: vectord merge endpoint** — `POST /v1/vectors/index/{src}/merge` with body `{dest, clear_source}`. Idempotent on re-runs (existing-in-dest items skipped). New `Index.IDs()` snapshot method backs it; new `i.ids` tracker field is the canonical ID set (independent of meta map's nil-vs-{} sparseness). 4 cmd-level tests + 1 unit test. |
+| (close-2) | **OPEN #2: distillation SFT export substrate** — `internal/distillation/sft_export.go`: `IsSftNever` predicate + `ListScoredRunFiles` (data/scored-runs/YYYY/MM/DD walk) + `LoadScoredRunsFromFile` + partial `ExportSft` that wires the firewall but leaves synthesis (instruction/input/response generation) as the next wave. Firewall pinning test fails if `SftNever` set changes without review. 5 new tests. The synthesis port remains on Rust at `scripts/distillation/export_sft.ts`. |
+| (close-2 full) | **OPEN #2 fully ported** (2026-05-01): `SynthesizeSft` + `LoadEvidenceByRunID` + `buildInstruction` ported byte-for-byte from `scripts/distillation/export_sft.ts`. All 8 source-class instruction templates (scrum_reviews / mode_experiments / auto_apply / audits / observer_reviews / contract_analyses / outcomes / default) match Rust output exactly so a/b validation between runtimes can diff JSONL byte-for-byte. `ExportSft` writes to `data/distilled/sft/sft_export.jsonl`. 5 additional tests including per-source-class template verification, extraction-rejection, empty-text-rejection, context-assembly, end-to-end fixture write. |
+| (close-2 lineage) | **Audit-baselines lineage ported** (2026-05-01): `internal/distillation/audit_baseline.go` mirrors Rust `audit_full.ts`'s LoadBaseline/AppendBaseline/buildDriftTable. `LoadLastBaseline` reads the most recent JSON line from `data/_kb/audit_baselines.jsonl`; `AppendBaseline` appends append-only with bufio. `BuildAuditDriftTable` flags drift `>20%` (configurable); zero-baseline and new-metric edge cases handled (no division-by-zero, no false-stable on zero→nonzero). `FormatAuditDriftTable` for stdout dumps. Generic on metric names so callers running both runtimes can pin Rust-compat names (`AuditBaselineRustCompat` constant lists them). 13 tests including last-line-wins, trailing-blank-tolerance, malformed-line-errors, threshold-boundary, zero-baseline-handling, sort-stability. |
+| (scrum) | 3-lineage scrum on `434f466..0d4f033` (post_role_gate_v1). Convergent finding (Opus + Kimi): `DecodeIndex` lost nil-meta items across persistence. **Fixed** by bumping envelope version 1→2 with explicit `IDs []string` field; v1 envelopes still load via meta-key fallback. Opus-only real bugs also actioned: `handleMerge` non-`ErrIndexNotFound` nil-deref, `mathLog` dead wrapper removed, bubble sort → `sort.Slice`. False positives rejected after verification (Kimi rollback misreading + Opus stale-comment claim). 2 new regression tests lock the v2 round-trip + v1 backward-compat. Disposition: `reports/scrum/_evidence/2026-05-01/verdicts/post_role_gate_v1_disposition.md`. |
+| (audit-full port) | **Audit-FULL pipeline** (phases 0/3/4) ported from `scripts/distillation/audit_full.ts`. `internal/distillation/audit_full.go` + `cmd/audit_full` CLI. 6 ported required-check classes; 4 phases (1, 2, 5, 6, 7) deferred — depend on broader Rust pieces (materializer / replay / run-summaries) not yet ported. **Cross-runtime byte-equal verdict on live data**: Go-side audit-full against `/home/profit/lakehouse` produced p3_*/p4_* metrics IDENTICAL to the last Rust-emitted `audit_baselines.jsonl` entry (all 8 metrics match: p3_accepted=386, p3_partial=132, p3_rejected=57, p3_human=480, p4_sft_rows=353, p4_rag_rows=448, p4_pref_pairs=83, p4_total_quarantined=1325). 6 new tests + the live-data probe captured in `reports/cutover/audit_full_go_vs_rust.md`. |
+| (audit-full skips fixed) | **Phases 1/2/5/7 unskipped** (2026-05-01) — port reduced from 4 deferred phases to 1. **Phase 1**: invokes `go test ./internal/distillation/...` via exec.Command (Go equivalent of Rust's `bun test`). **Phase 2**: reads `data/evidence/` and tallies rows + tier-1 source hits as an observer (doesn't re-run the materializer; emits `p2_evidence_rows`/`p2_evidence_skips` metrics). **Phase 5**: reads `reports/distillation/{run_id}/summary.json` + 5 stage receipts; validates schema_version + run_hash sha256 + git_commit hex. **Phase 7**: reads `data/_kb/replay_runs.jsonl`; tail-row JSON parse check. Only **Phase 6** remains skipped (Rust `acceptance.ts` is a TS-only fixture harness; porting fixtures + invariant runner is its own ADR). Live-data probe: 12/12 required checks PASS, `p2_evidence_rows=1055` byte-equal to Rust `summary.json` `collect.records_out`. 6 new tests. |
+| (lets-go) | **Persistent Go stack live** (2026-05-01). All 11 daemons (storaged/catalogd/ingestd/queryd/embedd/vectord/pathwayd/observerd/matrixd/gateway/chatd) up as long-running processes on :3110+:3211-:3220 → later moved to :4110+:4211-:4219+:3220 for smoke isolation. First time the Go side runs as production-shape daemons rather than per-harness transient processes. Brought up via `scripts/cutover/start_go_stack.sh`. Gateway proxies `/v1/embed` correctly to embedd; all 5 chatd providers loaded. **First Go-side entry written to `data/_kb/audit_baselines.jsonl`** (entry #8, git_commit=`ee2a40c`, golangLAKEHOUSE SHA distinguishable from Rust's `ca7375ea`); the longitudinal log now mixes runtimes. |
+| (g5-slice) | **G5 cutover slice LIVE** (2026-05-01). First real Bun-frontend traffic reaching the Go substrate end-to-end. Bun mcp-server (`/home/profit/lakehouse/mcp-server/index.ts`) gains opt-in `/_go/*` pass-through to `$GO_LAKEHOUSE_URL` (set to `http://127.0.0.1:4110` via systemd drop-in). `/_go/v1/embed` returns nomic-embed-text-v2-moe vectors via Go embedd; `/_go/v1/matrix/search` returns 3/3 Forklift Operators against the persistent 200-worker corpus. Fully additive (no existing Bun tool modified) + fully reversible (unset env). `/api/*` (Rust gateway) path unchanged. See `reports/cutover/g5_first_slice_live.md`. |
+| (close-3) | **OPEN #3: distribution drift via PSI** — `internal/drift/drift.go`: `ComputeDistributionDrift` returns Population Stability Index + verdict tier (stable < 0.10, minor 0.10–0.25, major ≥ 0.25). Equal-width bucketing over combined min/max range, epsilon-clamping for empty buckets, per-bucket breakdown for drilldown. 7 new tests including identical-is-stable, hard-shift-is-major, moderate-detected-not-stable, empty-inputs-safe, all-identical-safe, bucket-counts-conserved, num-buckets-clamping. |
+| (close-4) | **OPEN #4: ops nice-to-haves** — (a) Real-time wall-clock for stress harness: per-phase elapsed time logged to stdout as it runs (`[stress] phase NAME starting (T+12.3s)` + `[stress] phase NAME done — 8.5s (T+20.8s)`); `Output.PhaseTimings` + `Output.TotalElapsedMs` written to JSON; (b) chatd fixture-mode S3 mock + (c) liberal-paraphrase calibration: not actioned — no fired trigger yet, would be speculative. Documented as deferred-until-need rather than ignored. |
+| (close-bug) | **coder/hnsw v0.6.1 panic — REAL FIX landed** (2026-05-01 ~22:25). The 277884b multitier_100k run hit 96-98% fail on 2/6 scenarios from a v0.6.1 nil-deref (`layerNode.search`) that fires when the graph transitions through degenerate states post-Delete. Initial recover() guard caught panics but returned errors at the same rate. **Real fix**: lift the source-of-truth out of coder/hnsw — `i.vectors map[string][]float32` side store maintained alongside the graph, panic-safe `safeGraphAdd`/`safeGraphDelete` wrappers, `rebuildGraphLocked` reads from `i.vectors` (independent of graph state), warm-path Add falls back to rebuild on panic. Side effect: `i.ids` collapsed into `i.vectors` keys; `Len()` reads from `len(i.vectors)`. Memory cost: ~2x for vectors. Verification: 7 new regression tests in `index_test.go` (`TestAdd_PastThreshold_SustainedReAdd` reproduces the multitier shape — 64-entry index, 800 upserts, 0 errors), `just verify` PASS, multitier_100k re-run on persistent stack 19,622 scenarios / 0 failures across all 6 classes. p50 on previously-failing scenarios went 5ms (instant fail) → 551ms (real Add work — honest cost of correctness). |
+| (perf-fix) | **Save coalescing — write-path lock contention closed** (2026-05-01 ~22:50). The panic fix exposed a second bottleneck: every successful Add called `Persistor.Save` synchronously, which takes the index RLock for `Encode` (~6MB JSON for 1942-entry × 768d) — blocking concurrent Add Lock acquisitions. 5min sustained run showed playbook scenario p50 climbing 551ms→1398ms as the index grew. **Fix**: `saveTask` per-index single-flight coalescer in `cmd/vectord/main.go` — `saveAfter` now triggers an async save; concurrent triggers during an in-flight save mark "pending" so N triggers collapse into ≤2 actual saves. RPO trade: Add returns OK before save completes (~1 save's worth of crash-loss exposure; same fail-open posture as ADR-005). Verification: 3 new tests in `cmd/vectord/main_test.go` (50-trigger pile-up → 2 saves; single → 1; error doesn't stall). Re-run: surge_fill_validate p50 1296ms→**47ms** (~28× faster), playbook_record_replay 1398ms→**385ms** (~3.6× faster), throughput 144→**668 scen/sec** at 0% fail. Restart-rehydrate verified — playbook_memory 4041 entries persisted to MinIO and round-tripped cleanly. |
+
+Plus on Rust side (`8de94eb`, `3d06868`): qwen2.5 → qwen3.5:latest backport in active defaults; distillation acceptance reports regenerated (run_hash refresh, reproducibility property still holds).
+
+---
+
+## RUNTIME CHEATSHEET
+
+```bash
+# Verify everything green
+cd /home/profit/golangLAKEHOUSE
+just verify                                            # vet + tests + 9 core smokes (~31s)
+just doctor                                            # dep probe (go/gcc/minio/ollama/secrets)
+
+# Boot the chat dispatcher (Phase 4)
+nohup ./bin/chatd   -config lakehouse.toml > /tmp/chatd.log   2>&1 & disown
+nohup ./bin/gateway -config lakehouse.toml > /tmp/gateway.log 2>&1 & disown
+curl -sf http://127.0.0.1:3110/v1/chat/providers | jq  # all 5 providers should report true
+
+# Test a chat call to each lineage
+for m in "qwen3.5:latest" "opencode/claude-opus-4-7" "openrouter/moonshotai/kimi-k2-0905"; do
+  curl -sS -X POST http://127.0.0.1:3110/v1/chat \
+    -H 'Content-Type: application/json' \
+    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"reply: OK\"}],\"max_tokens\":8}" \
+    | jq -c '{model,provider,content}'
+done
+
+# Run the scrum on a diff
+./scripts/scrum_review.sh path/to/bundle.diff bundle_label
+ls reports/scrum/_evidence/$(date +%Y-%m-%d)/verdicts/
+
+# Domain smokes (not in `just verify`)
+for s in chatd matrix observer pathway playbook relevance downgrade workflow; do
+  bash scripts/${s}_smoke.sh > /tmp/${s}.log 2>&1 && echo "$s ✓" || echo "$s ✗"
+done
+```
+
+---
+
+## VISION — what we're actually building
+
+J's framing (canonical at `/root/.claude/projects/-home-profit/memory/project_small_model_pipeline_vision.md`): a small-model-driven autonomous pipeline that gets better with each run. Frontier APIs (Opus, Kimi, GPT-5) are too expensive + rate-limited for the inner loop — they live in audit/oversight via `frontier_*` tier. The hot path runs on local `qwen3.5:latest` given:
+
+1. **Pathway memory** — what we tried before, how it went (Mem0 substrate ✓)
+2. **Matrix indexer** — multi-corpus retrieve+merge giving the small model the right slice for this task (5/5 components ✓)
+3. **Observer** — watches each run, refines configs (not prompts) toward good pathways
+
+Successful runs get **rated and distilled back into the playbook**. Each iteration the playbook gets denser, runs get cheaper, results get better. **Drift** in the distilled playbook is a measured signal, not vibes.
+
+**The single load-bearing gate:** *"the playbook + matrix indexer must give the results we're looking for."* Throughput, scaling, code elegance are all secondary. The `playbook_lift` reality test is the regression gate before Enterprise cutover (where real contracts + live profile updates land).
+
+When evaluating any Go workstream, ask: which of the 5 loops does this advance? Strong workstreams advance ≥1; weak workstreams sit in infra-for-its-own-sake.
+
+---
+
+## SIBLING TOOLS (separate repos, intentional integration target later)
+
+**`local-review-harness`** at `git.agentview.dev/profit/local-review-harness` (also SMB-mounted at `/home/profit/share/local-review-harness-full-md/`). Local-first code review harness — 12 evidence-bearing static analyzers, Scrum-style reports, no cloud deps. Phase A + B (MVP) shipped 2026-04-30. Phases C–E (Ollama LLM review, validation, memory) pending.
+
+**Cross-pollination plan when both stabilize:**
+- Replace harness's `internal/llm/ollama.go` with a chatd `/v1/chat` client → frontier judges via config toggle
+- Feed harness findings into Lakehouse pathway memory as a drift signal
+- Treat harness's `.memory/known-risks.json` as a matrix-indexer corpus
+
+Detail at `docs/SPEC.md` §3.10. Don't re-port harness functionality into Lakehouse-Go — the standalone tool is the design.

cmd/audit_full/main.go

@@ -0,0 +1,105 @@
+// audit_full — Go-side audit-full runner. Calls into
+// internal/distillation.RunAuditFull, dumps the Markdown report to
+// stdout (or a file), and optionally appends an AuditBaseline entry
+// to data/_kb/audit_baselines.jsonl for the longitudinal log.
+//
+// Usage:
+//   audit_full                                      # report only
+//   audit_full -root /home/profit/lakehouse         # custom root
+//   audit_full -append-baseline                     # also append to audit_baselines.jsonl
+//   audit_full -out reports/distillation/run.md     # write report file
+//
+// Designed to live alongside the Rust scripts/distillation/audit_full.ts
+// — operators can run either runtime against the same root and the
+// audit_baselines.jsonl entries are interchangeable.
+package main
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"strings"
+	"time"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+func main() {
+	root := flag.String("root", "", "lakehouse data root (defaults to $LH_DISTILL_ROOT or /home/profit/lakehouse)")
+	out := flag.String("out", "", "write Markdown report to this path (default: stdout)")
+	appendBaseline := flag.Bool("append-baseline", false, "append an AuditBaseline entry to data/_kb/audit_baselines.jsonl after the run")
+	jsonOut := flag.Bool("json", false, "emit the full PhaseCheckReport as JSON instead of Markdown")
+	flag.Parse()
+
+	gitHEAD := resolveGitHEAD()
+	report := distillation.RunAuditFull(distillation.AuditFullOptions{
+		Root:    *root,
+		GitHEAD: gitHEAD,
+	})
+
+	var body []byte
+	if *jsonOut {
+		body = mustJSON(report)
+	} else {
+		body = []byte(distillation.FormatAuditFullReport(report))
+	}
+
+	if *out == "" {
+		_, _ = os.Stdout.Write(body)
+	} else {
+		if err := os.WriteFile(*out, body, 0o644); err != nil {
+			log.Fatalf("write %s: %v", *out, err)
+		}
+		fmt.Fprintf(os.Stderr, "wrote %s (%d bytes)\n", *out, len(body))
+	}
+
+	if *appendBaseline {
+		// Resolve the same path the Rust pipeline uses so both
+		// runtimes share the audit_baselines.jsonl log.
+		resolvedRoot := *root
+		if resolvedRoot == "" {
+			if env := os.Getenv("LH_DISTILL_ROOT"); env != "" {
+				resolvedRoot = env
+			} else {
+				resolvedRoot = "/home/profit/lakehouse"
+			}
+		}
+		bp := distillation.DefaultBaselinePath(resolvedRoot)
+		err := distillation.AppendBaseline(bp, distillation.AuditBaseline{
+			RecordedAt: time.Now().UTC().Format(time.RFC3339),
+			GitCommit:  gitHEAD,
+			Metrics:    report.Metrics,
+		})
+		if err != nil {
+			log.Fatalf("append baseline: %v", err)
+		}
+		fmt.Fprintf(os.Stderr, "appended baseline to %s\n", bp)
+	}
+
+	if report.Failed > 0 {
+		os.Exit(1)
+	}
+}
+
+// resolveGitHEAD returns the current commit SHA if the Go repo is a
+// git checkout. Falls back to "" rather than failing — the audit
+// runs even on a fresh clone without git.
+func resolveGitHEAD() string {
+	cmd := exec.Command("git", "rev-parse", "HEAD")
+	bs, err := cmd.Output()
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(bs))
+}
+
+func mustJSON(v any) []byte {
+	bs, err := json.MarshalIndent(v, "", "  ")
+	if err != nil {
+		log.Fatalf("json marshal: %v", err)
+	}
+	return append(bs, '\n')
+}

cmd/catalogd/main.go

@@ -49,7 +49,7 @@ func main() {
 
 	h := newHandlers(registry)
 
-	if err := shared.Run("catalogd", cfg.Catalogd.Bind, h.register); err != nil {
+	if err := shared.Run("catalogd", cfg.Catalogd.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}

cmd/catalogd/main_test.go

@@ -0,0 +1,150 @@
+package main
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/catalogd"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/storeclient"
+)
+
+// Closes R-005 for catalogd: cmd-level tests for route mounting,
+// body-cap rejection, malformed JSON handling, and the decode-error
+// paths in handleRegister. Deeper Registry semantics live in
+// internal/catalogd/registry_test.go.
+
+func newTestHandlers(t *testing.T) (*handlers, *httptest.Server) {
+	t.Helper()
+	// Stub storaged so the registry can hydrate (it needs nothing
+	// initially). Empty server = 404 on any GET; that's fine for
+	// these tests because we don't exercise storaged paths here.
+	stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusNotFound)
+	}))
+	t.Cleanup(stub.Close)
+
+	store := storeclient.New(stub.URL)
+	reg := catalogd.NewRegistry(store)
+	return newHandlers(reg), stub
+}
+
+func mountedRouter(h *handlers) chi.Router {
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	want := map[string]string{
+		"POST /catalog/register":     "register endpoint",
+		"GET /catalog/manifest/*":    "manifest endpoint",
+		"GET /catalog/list":          "list endpoint",
+	}
+	got := map[string]bool{}
+	chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		got[method+" "+route] = true
+		return nil
+	})
+	for sig := range want {
+		if !got[sig] {
+			t.Errorf("expected route %q mounted; got %v", sig, got)
+		}
+	}
+}
+
+func TestHandleRegister_BodyTooLarge(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	// 5 MiB body — over the 4 MiB cap.
+	big := bytes.Repeat([]byte("x"), 5<<20)
+	resp, err := http.Post(srv.URL+"/catalog/register", "application/json", bytes.NewReader(big))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	// MaxBytesReader trips during JSON decode → 400 with "body too large"
+	// in the message, OR 413 if Content-Length up-front cap is added.
+	// Today the path returns 400 via decode error; lock that contract.
+	if resp.StatusCode < 400 || resp.StatusCode >= 500 {
+		t.Errorf("expected 4xx on oversize body, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleRegister_MalformedJSON(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/catalog/register",
+		"application/json", strings.NewReader("not json"))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on malformed JSON, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleRegister_EmptyName_400(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	body := `{"name":"","schema_fingerprint":"sha256:x","objects":[{"key":"k","size":1}]}`
+	resp, err := http.Post(srv.URL+"/catalog/register", "application/json", strings.NewReader(body))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty name, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleGetManifest_404(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Get(srv.URL + "/catalog/manifest/nonexistent")
+	if err != nil {
+		t.Fatalf("GET: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404 for missing manifest, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleList_EmptyShape(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Get(srv.URL + "/catalog/list")
+	if err != nil {
+		t.Fatalf("GET: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("expected 200, got %d", resp.StatusCode)
+	}
+	if ct := resp.Header.Get("Content-Type"); !strings.HasPrefix(ct, "application/json") {
+		t.Errorf("Content-Type = %q, want application/json", ct)
+	}
+}

cmd/chatd/main.go

@@ -0,0 +1,144 @@
+// chatd is the LLM chat dispatcher service (Phase 4 — small-model
+// pipeline tier abstraction). Routes POST /chat to the right
+// provider based on the model-name prefix or :cloud suffix:
+//
+//	ollama/<m>          → local Ollama (no auth)
+//	ollama_cloud/<m>    → Ollama Cloud (Bearer auth)
+//	<m>:cloud           → Ollama Cloud (suffix variant)
+//	openrouter/<v>/<m>  → OpenRouter (Bearer auth)
+//	opencode/<m>        → OpenCode unified Zen+Go (Bearer auth)
+//	kimi/<m>            → Kimi For Coding (Bearer auth)
+//	bare names          → local Ollama (default)
+//
+// Provider keys come from env vars (or /etc/lakehouse/<provider>.env
+// fallback files). Providers with empty keys stay unregistered, so
+// requests for them 404 cleanly instead of 503-ing at call time.
+//
+// Routes:
+//
+//	POST /chat       — dispatch a chat request to the resolved provider
+//	GET  /providers  — list registered providers (telemetry / health)
+//	GET  /health     — readiness (always 200 — sub-providers are
+//	                   independently checked via /providers)
+package main
+
+import (
+	"encoding/json"
+	"flag"
+	"errors"
+	"log/slog"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/chat"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
+)
+
+const maxRequestBytes = 4 << 20 // 4 MiB cap on /chat bodies
+
+func main() {
+	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
+	flag.Parse()
+
+	cfg, err := shared.LoadConfig(*configPath)
+	if err != nil {
+		slog.Error("config", "err", err)
+		os.Exit(1)
+	}
+
+	timeout := time.Duration(cfg.Chatd.TimeoutSecs) * time.Second
+	registry := chat.BuildRegistry(chat.BuilderInput{
+		OllamaURL:      cfg.Chatd.OllamaURL,
+		OllamaCloudKey: chat.ResolveKey(cfg.Chatd.OllamaCloudKeyEnv, cfg.Chatd.OllamaCloudKeyFile),
+		OpenRouterKey:  chat.ResolveKey(cfg.Chatd.OpenRouterKeyEnv, cfg.Chatd.OpenRouterKeyFile),
+		OpenCodeKey:    chat.ResolveKey(cfg.Chatd.OpenCodeKeyEnv, cfg.Chatd.OpenCodeKeyFile),
+		KimiKey:        chat.ResolveKey(cfg.Chatd.KimiKeyEnv, cfg.Chatd.KimiKeyFile),
+		Timeout:        timeout,
+	})
+
+	h := &handlers{registry: registry}
+	if err := shared.Run("chatd", cfg.Chatd.Bind, h.register, cfg.Auth); err != nil {
+		slog.Error("server", "err", err)
+		os.Exit(1)
+	}
+}
+
+type handlers struct {
+	registry *chat.Registry
+}
+
+func (h *handlers) register(r chi.Router) {
+	// Routes mirror what the gateway proxies: /v1/chat → /chat (POST)
+	// and /v1/chat/providers → /chat/providers (GET). Keeping providers
+	// under /chat/ avoids a separate /providers root route that would
+	// need its own gateway proxy entry.
+	r.Post("/chat", h.handleChat)
+	r.Get("/chat/providers", h.handleProviders)
+}
+
+func (h *handlers) handleChat(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	defer r.Body.Close()
+
+	var req chat.Request
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest)
+		return
+	}
+	if req.Model == "" || len(req.Messages) == 0 {
+		http.Error(w, "model and messages are required", http.StatusBadRequest)
+		return
+	}
+
+	resp, err := h.registry.Chat(r.Context(), req)
+	if err != nil {
+		writeChatError(w, err)
+		return
+	}
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// handleProviders lists registered providers + per-provider Available()
+// status. Phase 4 scrum fix B-2: looks up by name directly, skipping
+// the prior synthetic-model-name route through Resolve. The endpoint
+// reports registry membership, not routing rules — the latter is
+// covered by /v1/chat itself.
+func (h *handlers) handleProviders(w http.ResponseWriter, _ *http.Request) {
+	names := h.registry.Names()
+	statuses := make(map[string]bool, len(names))
+	for _, n := range names {
+		statuses[n] = h.registry.Available(n)
+	}
+	writeJSON(w, http.StatusOK, map[string]any{
+		"providers": statuses,
+	})
+}
+
+// writeChatError maps chat sentinel errors to HTTP status codes.
+// Unknown errors map to 500.
+func writeChatError(w http.ResponseWriter, err error) {
+	switch {
+	case errors.Is(err, chat.ErrProviderNotFound):
+		http.Error(w, err.Error(), http.StatusNotFound)
+	case errors.Is(err, chat.ErrProviderDisabled):
+		http.Error(w, err.Error(), http.StatusServiceUnavailable)
+	case errors.Is(err, chat.ErrUpstream):
+		http.Error(w, err.Error(), http.StatusBadGateway)
+	case errors.Is(err, chat.ErrTimeout):
+		http.Error(w, err.Error(), http.StatusGatewayTimeout)
+	default:
+		slog.Error("chat", "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+	}
+}
+
+func writeJSON(w http.ResponseWriter, status int, body any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	if err := json.NewEncoder(w).Encode(body); err != nil {
+		slog.Error("encode", "err", err)
+	}
+}

cmd/embedd/main.go

@@ -41,11 +41,22 @@ func main() {
 		os.Exit(1)
 	}
 
-	h := &handlers{
-		provider: embed.NewOllama(cfg.Embedd.ProviderURL, cfg.Embedd.DefaultModel),
+	// Wrap the upstream provider in an LRU cache so repeat queries
+	// (the staffing co-pilot replays many of the same texts) bypass
+	// the ~50ms Ollama round-trip. Cache size 0 = pass-through.
+	base := embed.NewOllama(cfg.Embedd.ProviderURL, cfg.Embedd.DefaultModel)
+	cached, err := embed.NewCachedProvider(base, cfg.Embedd.DefaultModel, cfg.Embedd.CacheSize)
+	if err != nil {
+		slog.Error("embed cache", "err", err)
+		os.Exit(1)
 	}
+	slog.Info("embed cache",
+		"size", cfg.Embedd.CacheSize,
+		"default_model", cfg.Embedd.DefaultModel,
+		"enabled", cfg.Embedd.CacheSize > 0)
+	h := &handlers{provider: cached, cache: cached}
 
-	if err := shared.Run("embedd", cfg.Embedd.Bind, h.register); err != nil {
+	if err := shared.Run("embedd", cfg.Embedd.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}
@@ -53,10 +64,36 @@ func main() {
 
 type handlers struct {
 	provider embed.Provider
+	// cache is the same instance as provider when caching is enabled,
+	// kept as a typed pointer so /v1/embed/stats can expose hit-rate
+	// without type-asserting through the Provider interface. nil when
+	// CacheSize=0 (pass-through mode).
+	cache *embed.CachedProvider
 }
 
 func (h *handlers) register(r chi.Router) {
 	r.Post("/embed", h.handleEmbed)
+	r.Get("/embed/stats", h.handleStats)
+}
+
+// handleStats reports cache hits/misses + hit rate + size. Operators
+// use this to confirm the cache is doing its job (high hit rate) or
+// is sized wrong (low hit rate + many misses on a workload that
+// should have repeats).
+func (h *handlers) handleStats(w http.ResponseWriter, _ *http.Request) {
+	w.Header().Set("Content-Type", "application/json")
+	if h.cache == nil {
+		_ = json.NewEncoder(w).Encode(map[string]any{"enabled": false})
+		return
+	}
+	hits, misses := h.cache.Stats()
+	_ = json.NewEncoder(w).Encode(map[string]any{
+		"enabled":   true,
+		"hits":      hits,
+		"misses":    misses,
+		"hit_rate":  h.cache.HitRate(),
+		"size":      h.cache.Len(),
+	})
 }
 
 // embedRequest is the POST /embed body. Texts is the list to

cmd/embedd/main_test.go

@@ -0,0 +1,227 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/embed"
+)
+
+// Closes R-005 for embedd: cmd-level tests for the /embed handler's
+// decode + validation paths (empty texts → 400, body cap → 413,
+// upstream error → 502). Provider semantics live in
+// internal/embed/ollama_test.go.
+
+// stubProvider implements embed.Provider with deterministic stubs.
+type stubProvider struct {
+	result embed.Result
+	err    error
+}
+
+func (s *stubProvider) Embed(_ context.Context, _ []string, _ string) (embed.Result, error) {
+	return s.result, s.err
+}
+
+func mountWithProvider(p embed.Provider) chi.Router {
+	h := &handlers{provider: p}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	r := mountWithProvider(&stubProvider{})
+	found := false
+	chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		if method == "POST" && route == "/embed" {
+			found = true
+		}
+		return nil
+	})
+	if !found {
+		t.Error("POST /embed not mounted")
+	}
+}
+
+func TestHandleEmbed_BodyTooLarge(t *testing.T) {
+	// MaxBytesReader trips during JSON decode. Depending on whether
+	// the decoder unwrapping surfaces MaxBytesError or wraps it as a
+	// generic decode error, the response is either 413 or 400. Both
+	// are valid "client error, fails loud" contracts; the harness's
+	// proof_assert_status_4xx covers either at the integration level.
+	r := mountWithProvider(&stubProvider{})
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	big := bytes.Repeat([]byte("x"), maxRequestBytes+(1<<20))
+	resp, err := http.Post(srv.URL+"/embed", "application/json", bytes.NewReader(big))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 400 || resp.StatusCode >= 500 {
+		t.Errorf("expected 4xx on oversize, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleEmbed_MalformedJSON_400(t *testing.T) {
+	r := mountWithProvider(&stubProvider{})
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/embed", "application/json", strings.NewReader("not json"))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on malformed, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleEmbed_EmptyTextRejected_400(t *testing.T) {
+	// Per scrum O-W3 (Opus): reject empty strings up front.
+	r := mountWithProvider(&stubProvider{})
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/embed", "application/json",
+		strings.NewReader(`{"texts":["valid",""]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty text in batch, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleEmbed_UpstreamError_502(t *testing.T) {
+	// Provider returns a generic error → handler maps to 502 (the
+	// "embedding backend was wrong" case, distinct from 400 = your
+	// input was wrong).
+	r := mountWithProvider(&stubProvider{err: errors.New("ollama is down")})
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/embed", "application/json",
+		strings.NewReader(`{"texts":["hello"]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadGateway {
+		t.Errorf("expected 502 on provider error, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleEmbed_HappyPath_ProviderEcho(t *testing.T) {
+	stub := &stubProvider{result: embed.Result{
+		Model:     "test-model",
+		Dimension: 3,
+		Vectors:   [][]float32{{0.1, 0.2, 0.3}},
+	}}
+	r := mountWithProvider(stub)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/embed", "application/json",
+		strings.NewReader(`{"texts":["hello"],"model":"test-model"}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("expected 200 happy path, got %d", resp.StatusCode)
+	}
+}
+
+func TestStatsEndpoint_CacheDisabled(t *testing.T) {
+	r := mountWithProvider(&stubProvider{})
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Get(srv.URL + "/embed/stats")
+	if err != nil {
+		t.Fatalf("GET: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("status = %d, want 200", resp.StatusCode)
+	}
+	if !strings.Contains(getBody(t, resp), `"enabled":false`) {
+		t.Errorf("expected enabled:false when no cache wired")
+	}
+}
+
+func TestStatsEndpoint_CacheEnabled_TracksHitsAndMisses(t *testing.T) {
+	stub := &stubProvider{result: embed.Result{
+		Model:     "test-model",
+		Dimension: 3,
+		Vectors:   [][]float32{{0.1, 0.2, 0.3}},
+	}}
+	cache, err := embed.NewCachedProvider(stub, "test-model", 100)
+	if err != nil {
+		t.Fatalf("NewCachedProvider: %v", err)
+	}
+	h := &handlers{provider: cache, cache: cache}
+	r := chi.NewRouter()
+	h.register(r)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	// First call → miss.
+	http.Post(srv.URL+"/embed", "application/json",
+		strings.NewReader(`{"texts":["hello"],"model":"test-model"}`))
+	// Second call same text → hit.
+	http.Post(srv.URL+"/embed", "application/json",
+		strings.NewReader(`{"texts":["hello"],"model":"test-model"}`))
+
+	resp, err := http.Get(srv.URL + "/embed/stats")
+	if err != nil {
+		t.Fatalf("stats GET: %v", err)
+	}
+	defer resp.Body.Close()
+	body := getBody(t, resp)
+	if !strings.Contains(body, `"enabled":true`) {
+		t.Errorf("expected enabled:true, body=%s", body)
+	}
+	if !strings.Contains(body, `"hits":1`) {
+		t.Errorf("expected hits:1, body=%s", body)
+	}
+	if !strings.Contains(body, `"misses":1`) {
+		t.Errorf("expected misses:1, body=%s", body)
+	}
+}
+
+func getBody(t *testing.T, resp *http.Response) string {
+	t.Helper()
+	buf := make([]byte, 4096)
+	n, _ := resp.Body.Read(buf)
+	return string(buf[:n])
+}
+
+func TestItoa(t *testing.T) {
+	cases := []struct {
+		in  int
+		out string
+	}{
+		{0, "0"},
+		{1, "1"},
+		{42, "42"},
+		{1000, "1000"},
+		{99, "99"},
+	}
+	for _, tc := range cases {
+		if got := itoa(tc.in); got != tc.out {
+			t.Errorf("itoa(%d) = %q, want %q", tc.in, got, tc.out)
+		}
+	}
+}

cmd/fake_ollama/main.go

@@ -0,0 +1,102 @@
+// fake_ollama is a minimal Ollama-API-compatible fake for proof harness
+// fixture-mode smokes (R-006 partial). Implements just enough of the
+// Ollama API surface for embedd to drive end-to-end without a real
+// Ollama installation:
+//
+//   GET  /api/tags         — returns a fixed model list including
+//                            nomic-embed-text:latest
+//   POST /api/embeddings   — returns a deterministic dim-D vector
+//                            derived from sha256(prompt). Same prompt
+//                            → bit-identical vector across runs.
+//
+// Vectors are NOT semantically meaningful (the value of similarity
+// search against these is undefined). The fake is for proving the
+// EMBED CONTRACT — dimension echo, response shape, status codes —
+// not for proving real semantic ranking. That requires real Ollama.
+//
+// Why this exists: the proof harness's contract tier already runs
+// against real Ollama (when present). For CI / fresh-clone reviewers
+// without Ollama, this fake unblocks the chain.
+//
+// Usage:
+//   bin/fake_ollama --bind 127.0.0.1:11435 --dim 768
+package main
+
+import (
+	"crypto/sha256"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+)
+
+func main() {
+	bind := flag.String("bind", "127.0.0.1:11435", "bind addr")
+	dim := flag.Int("dim", 768, "embedding dimension to return")
+	model := flag.String("model", "nomic-embed-text", "model name to echo back")
+	flag.Parse()
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/api/tags", func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"models": []map[string]any{
+				{
+					"name":  *model + ":latest",
+					"model": *model + ":latest",
+				},
+			},
+		})
+	})
+	mux.HandleFunc("/api/embeddings", func(w http.ResponseWriter, r *http.Request) {
+		var req struct {
+			Model  string `json:"model"`
+			Prompt string `json:"prompt"`
+		}
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			http.Error(w, "decode: "+err.Error(), http.StatusBadRequest)
+			return
+		}
+		// Reject unknown models so embedd's bad-model→502 contract
+		// path is exercisable. The fake recognizes the configured
+		// model name only.
+		if req.Model != "" && req.Model != *model && req.Model != *model+":latest" {
+			http.Error(w, fmt.Sprintf("model %q not found", req.Model), http.StatusNotFound)
+			return
+		}
+		vec := deterministicVector(req.Prompt, *dim)
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"embedding": vec,
+		})
+	})
+	mux.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(`{"status":"ok","service":"fake_ollama"}`))
+	})
+
+	slog.Info("fake_ollama starting", "bind", *bind, "dim", *dim, "model", *model)
+	srv := &http.Server{Addr: *bind, Handler: mux}
+	if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+		slog.Error("fake_ollama serve", "err", err)
+		os.Exit(1)
+	}
+}
+
+// deterministicVector returns a fixed dim-d float64 vector derived
+// from sha256(prompt). Same prompt → same vector across runs and
+// across machines, so smoke assertions can compare to fixtures.
+func deterministicVector(prompt string, dim int) []float64 {
+	h := sha256.Sum256([]byte(prompt))
+	vec := make([]float64, dim)
+	for i := range vec {
+		// Spread the 32 hash bytes across `dim` positions; map to
+		// [-1, 1] so cosine distance is well-defined and the result
+		// looks vaguely like a real embedding.
+		b := h[i%len(h)]
+		vec[i] = (float64(b) - 128.0) / 128.0
+	}
+	return vec
+}

cmd/gateway/main.go

@@ -44,6 +44,11 @@ func main() {
 		"queryd_url":   cfg.Gateway.QuerydURL,
 		"vectord_url":  cfg.Gateway.VectordURL,
 		"embedd_url":   cfg.Gateway.EmbeddURL,
+		"pathwayd_url": cfg.Gateway.PathwaydURL,
+		"matrixd_url":  cfg.Gateway.MatrixdURL,
+		"observerd_url": cfg.Gateway.ObserverdURL,
+		"chatd_url":    cfg.Gateway.ChatdURL,
+		"validatord_url": cfg.Gateway.ValidatordURL,
 	}
 	for k, v := range upstreams {
 		if v == "" {
@@ -63,6 +68,11 @@ func main() {
 	querydURL := mustParseUpstream("queryd_url", cfg.Gateway.QuerydURL)
 	vectordURL := mustParseUpstream("vectord_url", cfg.Gateway.VectordURL)
 	embeddURL := mustParseUpstream("embedd_url", cfg.Gateway.EmbeddURL)
+	pathwaydURL := mustParseUpstream("pathwayd_url", cfg.Gateway.PathwaydURL)
+	matrixdURL := mustParseUpstream("matrixd_url", cfg.Gateway.MatrixdURL)
+	observerdURL := mustParseUpstream("observerd_url", cfg.Gateway.ObserverdURL)
+	chatdURL := mustParseUpstream("chatd_url", cfg.Gateway.ChatdURL)
+	validatordURL := mustParseUpstream("validatord_url", cfg.Gateway.ValidatordURL)
 
 	storagedProxy := gateway.NewProxyHandler(storagedURL)
 	catalogdProxy := gateway.NewProxyHandler(catalogdURL)
@@ -70,8 +80,14 @@ func main() {
 	querydProxy := gateway.NewProxyHandler(querydURL)
 	vectordProxy := gateway.NewProxyHandler(vectordURL)
 	embeddProxy := gateway.NewProxyHandler(embeddURL)
+	pathwaydProxy := gateway.NewProxyHandler(pathwaydURL)
+	matrixdProxy := gateway.NewProxyHandler(matrixdURL)
+	observerdProxy := gateway.NewProxyHandler(observerdURL)
+	chatdProxy := gateway.NewProxyHandler(chatdURL)
+	validatordProxy := gateway.NewProxyHandler(validatordURL)
 
 	if err := shared.Run("gateway", cfg.Gateway.Bind, func(r chi.Router) {
+
 		// Storage / catalog have multi-segment paths under their
 		// prefix (e.g. /v1/storage/get/<key>). chi's `*` wildcard
 		// captures the rest of the path.
@@ -87,7 +103,19 @@ func main() {
 		r.Handle("/v1/vectors/*", vectordProxy)
 		// Embedding service — /v1/embed
 		r.Handle("/v1/embed", embeddProxy)
-	}); err != nil {
+		// Pathway memory — /v1/pathway/*
+		r.Handle("/v1/pathway/*", pathwaydProxy)
+		// Matrix indexer — /v1/matrix/*  (multi-corpus retrieve+merge per SPEC §3.4)
+		r.Handle("/v1/matrix/*", matrixdProxy)
+		// Observer — /v1/observer/*  (autonomous-iteration witness loop)
+		r.Handle("/v1/observer/*", observerdProxy)
+		// Chat — /v1/chat (LLM dispatcher) + /v1/chat/providers
+		r.Handle("/v1/chat", chatdProxy)
+		r.Handle("/v1/chat/*", chatdProxy)
+		// Validator — /v1/validate (single-shot) + /v1/iterate (loop)
+		r.Handle("/v1/validate", validatordProxy)
+		r.Handle("/v1/iterate", validatordProxy)
+	}, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}

cmd/gateway/main_test.go

@@ -0,0 +1,102 @@
+package main
+
+import (
+	"net/url"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+)
+
+// Closes R-005 for gateway: cmd-level test for mustParseUpstream.
+// The proxy mounts themselves are exercised end-to-end by the
+// proof harness's GOLAKE-003 case (gateway proxy passthrough).
+//
+// mustParseUpstream calls os.Exit on bad input — testing it directly
+// would kill the test process. The standard Go pattern for testing
+// os.Exit-calling code: re-exec the test binary with a flag and
+// observe the subprocess exit status. We exercise the helper that
+// way for the failure paths and inline-check the success path.
+
+func TestMustParseUpstream_HappyPaths(t *testing.T) {
+	// Success paths can be exercised inline — only failure exits.
+	cases := []string{
+		"http://127.0.0.1:3211",
+		"https://example.com:443",
+		"http://catalogd:3212",
+	}
+	for _, raw := range cases {
+		t.Run(raw, func(t *testing.T) {
+			u := mustParseUpstream("test", raw)
+			if u.Scheme == "" || u.Host == "" {
+				t.Errorf("mustParseUpstream(%q) returned empty scheme/host: %+v", raw, u)
+			}
+		})
+	}
+}
+
+func TestMustParseUpstream_FailureExits(t *testing.T) {
+	if os.Getenv("GATEWAY_TEST_EXIT") == "1" {
+		// Subprocess: invoke mustParseUpstream with a bad value;
+		// expect os.Exit(1). url.Parse is permissive — schemes can
+		// be missing without a parse error — so the assertion in
+		// mustParseUpstream catches the empty-Host case.
+		mustParseUpstream("storaged_url", "127.0.0.1:3211")
+		// If we reach here, the function failed to fail.
+		os.Exit(0)
+	}
+
+	cmd := exec.Command(os.Args[0], "-test.run=TestMustParseUpstream_FailureExits")
+	cmd.Env = append(os.Environ(), "GATEWAY_TEST_EXIT=1")
+	err := cmd.Run()
+
+	if err == nil {
+		t.Fatal("expected subprocess to exit non-zero on bad upstream URL")
+	}
+	exitErr, ok := err.(*exec.ExitError)
+	if !ok {
+		t.Fatalf("expected ExitError, got %T: %v", err, err)
+	}
+	if exitErr.ExitCode() == 0 {
+		t.Fatal("subprocess returned 0 — mustParseUpstream did not fail")
+	}
+}
+
+func TestMustParseUpstream_GarbageInput_Exits(t *testing.T) {
+	if os.Getenv("GATEWAY_TEST_EXIT_GARBAGE") == "1" {
+		mustParseUpstream("queryd_url", "https://%zz")
+		os.Exit(0)
+	}
+	cmd := exec.Command(os.Args[0], "-test.run=TestMustParseUpstream_GarbageInput_Exits")
+	cmd.Env = append(os.Environ(), "GATEWAY_TEST_EXIT_GARBAGE=1")
+	err := cmd.Run()
+	if err == nil {
+		t.Fatal("expected subprocess to exit non-zero on garbage URL")
+	}
+}
+
+// TestUpstreamConfigKeys documents the upstream config field names
+// the gateway expects. A future refactor that renames a field would
+// fail this test; operators eyeballing systemctl status see the
+// failure before traffic does.
+func TestUpstreamConfigKeys_DocumentedShape(t *testing.T) {
+	// This test is shape documentation. main() iterates a map with
+	// these exact keys; if any are renamed, all gateway deployments
+	// silently break.
+	expected := []string{
+		"storaged_url",
+		"catalogd_url",
+		"ingestd_url",
+		"queryd_url",
+		"vectord_url",
+		"embedd_url",
+	}
+	for _, k := range expected {
+		if !strings.HasSuffix(k, "_url") {
+			t.Errorf("upstream key %q does not end in _url — convention break", k)
+		}
+		if _, err := url.Parse("http://" + k); err != nil {
+			t.Errorf("key %q failed url-test parse: %v", k, err)
+		}
+	}
+}

cmd/ingestd/main.go

@@ -65,7 +65,7 @@ func main() {
 		maxBytes:    maxBytes,
 	}
 
-	if err := shared.Run("ingestd", cfg.Ingestd.Bind, h.register); err != nil {
+	if err := shared.Run("ingestd", cfg.Ingestd.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}

cmd/ingestd/main_test.go

@@ -0,0 +1,188 @@
+package main
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/catalogclient"
+)
+
+// Closes R-005 for ingestd: cmd-level tests for the cmd-shape
+// of /ingest — name query param, body cap, multipart parsing,
+// missing form file. CSV→Parquet logic is tested in internal/ingestd.
+
+func newTestHandlers(t *testing.T) (*handlers, *httptest.Server) {
+	t.Helper()
+	// Stub catalogd so we can run end-to-end happy paths without the
+	// real catalogd up. The stub returns a 200-shaped registerResponse.
+	stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/catalog/register":
+			w.Header().Set("Content-Type", "application/json")
+			w.Write([]byte(`{"manifest":{"name":"x","dataset_id":"d","schema_fingerprint":"sha256:x","objects":[]},"existing":false}`))
+		default:
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	t.Cleanup(stub.Close)
+
+	storaged := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"status":"ok"}`))
+	}))
+	t.Cleanup(storaged.Close)
+
+	h := &handlers{
+		storagedURL: strings.TrimRight(storaged.URL, "/"),
+		catalogd:    catalogclient.New(stub.URL),
+		hc:          &http.Client{},
+		maxBytes:    256 << 20,
+	}
+	return h, stub
+}
+
+func mountedRouter(h *handlers) chi.Router {
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	found := false
+	chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		if method == "POST" && route == "/ingest" {
+			found = true
+		}
+		return nil
+	})
+	if !found {
+		t.Error("POST /ingest not mounted")
+	}
+}
+
+func TestHandleIngest_MissingNameQueryParam(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/ingest",
+		"multipart/form-data; boundary=x", strings.NewReader(""))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on missing name param, got %d", resp.StatusCode)
+	}
+	body, _ := io.ReadAll(resp.Body)
+	if !strings.Contains(string(body), "name") {
+		t.Errorf("error body should mention 'name', got %q", body)
+	}
+}
+
+func TestHandleIngest_MalformedMultipart(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/ingest?name=test",
+		"multipart/form-data; boundary=xyz", strings.NewReader("garbage not multipart"))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on malformed multipart, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleIngest_MissingFormFile(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	// Valid multipart with no "file" field.
+	body := bytes.NewReader([]byte(
+		"--xyz\r\n" +
+			"Content-Disposition: form-data; name=\"other\"\r\n" +
+			"\r\n" +
+			"value\r\n" +
+			"--xyz--\r\n",
+	))
+	resp, err := http.Post(srv.URL+"/ingest?name=test",
+		"multipart/form-data; boundary=xyz", body)
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on missing form file, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleIngest_BodyTooLarge(t *testing.T) {
+	h, _ := newTestHandlers(t)
+	h.maxBytes = 1024 // tiny cap so we hit it without huge upload
+	r := mountedRouter(h)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	big := bytes.Repeat([]byte("x"), 4096)
+	resp, err := http.Post(srv.URL+"/ingest?name=test",
+		"multipart/form-data; boundary=xyz", bytes.NewReader(big))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 400 || resp.StatusCode >= 500 {
+		t.Errorf("expected 4xx on oversize body, got %d", resp.StatusCode)
+	}
+}
+
+func TestEscapeKeyPath(t *testing.T) {
+	cases := []struct {
+		in   string
+		want string
+	}{
+		{"a/b/c.parquet", "a/b/c.parquet"},
+		{"data sets/x.parquet", "data%20sets/x.parquet"},
+		{"O'Reilly/key", "O%27Reilly/key"},
+		{"datasets/proof/abc.parquet", "datasets/proof/abc.parquet"},
+		{"", ""},
+		{"/", "/"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.in, func(t *testing.T) {
+			got := escapeKeyPath(tc.in)
+			if got != tc.want {
+				t.Errorf("escapeKeyPath(%q) = %q, want %q", tc.in, got, tc.want)
+			}
+		})
+	}
+}
+
+func TestParquetKeyPath_Format(t *testing.T) {
+	// Lock the content-addressed key shape per scrum C-DRIFT.
+	// Failure here means a dataset's parquet would land at an
+	// unexpected key, breaking schema-drift idempotency.
+	if !strings.Contains(parquetKeyPath, "%s") {
+		t.Errorf("parquetKeyPath should be a fmt template, got %q", parquetKeyPath)
+	}
+	if !strings.HasPrefix(parquetKeyPath, "datasets/") {
+		t.Errorf("parquetKeyPath should be under datasets/, got %q", parquetKeyPath)
+	}
+	if !strings.HasSuffix(parquetKeyPath, ".parquet") {
+		t.Errorf("parquetKeyPath should end with .parquet, got %q", parquetKeyPath)
+	}
+}

cmd/materializer/main.go

@@ -0,0 +1,78 @@
+// materializer — Go-side build_evidence_index runner. Reads source
+// JSONL streams in `data/_kb/`, transforms each row to an
+// EvidenceRecord, writes day-partitioned output under `data/evidence/`
+// + an audit-grade receipt under `reports/distillation/<ts>/`.
+//
+// Mirrors the Bun runner at scripts/distillation/build_evidence_index.ts
+// — both runtimes can run against the same root and produce
+// interoperable outputs (per ADR-001 #4: same logic, on-wire
+// JSON shape preserved).
+//
+// Usage:
+//
+//	materializer                                        # full run, write outputs
+//	materializer -dry-run                               # count, no writes
+//	materializer -root /home/profit/lakehouse           # custom repo root
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"os"
+	"time"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/materializer"
+)
+
+func main() {
+	root := flag.String("root", defaultRoot(), "lakehouse repo root (defaults to $LH_DISTILL_ROOT or current dir)")
+	dryRun := flag.Bool("dry-run", false, "count rows but do not write outputs")
+	flag.Parse()
+
+	recordedAt := time.Now().UTC().Format(time.RFC3339Nano)
+
+	res, err := materializer.MaterializeAll(materializer.MaterializeOptions{
+		Root:       *root,
+		Transforms: materializer.Transforms,
+		RecordedAt: recordedAt,
+		DryRun:     *dryRun,
+	})
+	if err != nil {
+		log.Fatalf("materializer: %v", err)
+	}
+
+	suffix := ""
+	if *dryRun {
+		suffix = " (DRY RUN)"
+	}
+	fmt.Printf("[evidence_index] %d read · %d written · %d skipped · %d deduped%s\n",
+		res.Totals.RowsRead, res.Totals.RowsWritten, res.Totals.RowsSkipped, res.Totals.RowsDeduped, suffix)
+	for _, s := range res.Sources {
+		if !s.RowsPresent {
+			fmt.Printf("  %s: (missing — skipped)\n", s.SourceFileRelPath)
+			continue
+		}
+		fmt.Printf("  %s: read=%d wrote=%d skip=%d dedup=%d\n",
+			s.SourceFileRelPath, s.RowsRead, s.RowsWritten, s.RowsSkipped, s.RowsDeduped)
+	}
+
+	if !*dryRun {
+		fmt.Printf("[evidence_index] receipt: %s\n", res.ReceiptPath)
+		fmt.Printf("[evidence_index] validation_pass=%v\n", res.Receipt.ValidationPass)
+	}
+
+	if !res.Receipt.ValidationPass {
+		os.Exit(1)
+	}
+}
+
+func defaultRoot() string {
+	if r := os.Getenv("LH_DISTILL_ROOT"); r != "" {
+		return r
+	}
+	if cwd, err := os.Getwd(); err == nil {
+		return cwd
+	}
+	return "."
+}

cmd/matrixd/main.go

@@ -0,0 +1,306 @@
+// matrixd is the matrix indexer service. Wraps internal/matrix's
+// Retriever with HTTP routes per docs/SPEC.md §3.4.
+//
+// Routes:
+//   POST /matrix/search             — multi-corpus retrieve+merge,
+//                                      with optional playbook boost
+//   GET  /matrix/corpora            — list known vectord indexes
+//   POST /matrix/relevance          — adjacency-pollution filter
+//   POST /matrix/downgrade          — strong-model downgrade gate
+//   POST /matrix/playbooks/record   — record a single (query → answer)
+//                                      success for the learning loop
+//   POST /matrix/playbooks/bulk     — bulk-record N successes; useful
+//                                      for backfilling historical
+//                                      placement data into the
+//                                      playbook substrate
+//
+// matrixd talks to embedd (for query-text embedding) and vectord
+// (for per-corpus search) via HTTP. Both URLs come from
+// [matrixd] config; gateway sets them to its own upstream URLs so
+// matrixd inherits the same provider topology.
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"flag"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/matrix"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
+)
+
+const maxRequestBytes = 4 << 20 // 4 MiB cap on request bodies
+
+func main() {
+	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
+	flag.Parse()
+
+	cfg, err := shared.LoadConfig(*configPath)
+	if err != nil {
+		slog.Error("config", "err", err)
+		os.Exit(1)
+	}
+	if cfg.Matrixd.EmbeddURL == "" || cfg.Matrixd.VectordURL == "" {
+		slog.Error("matrixd: embedd_url and vectord_url required in [matrixd]")
+		os.Exit(1)
+	}
+
+	retriever := matrix.New(cfg.Matrixd.EmbeddURL, cfg.Matrixd.VectordURL)
+	h := &handlers{r: retriever, weakModels: cfg.Models.WeakModels}
+
+	if err := shared.Run("matrixd", cfg.Matrixd.Bind, h.register, cfg.Auth); err != nil {
+		slog.Error("server", "err", err)
+		os.Exit(1)
+	}
+}
+
+type handlers struct {
+	r *matrix.Retriever
+	// weakModels comes from cfg.Models.WeakModels at startup. Threaded
+	// into every DowngradeInput so the gate uses the configured list
+	// instead of matrix.DefaultWeakModels. nil/empty falls back to
+	// the package default — matches pre-Phase 2 behavior.
+	weakModels []string
+}
+
+func (h *handlers) register(r chi.Router) {
+	r.Post("/matrix/search", h.handleSearch)
+	r.Get("/matrix/corpora", h.handleCorpora)
+	r.Post("/matrix/relevance", h.handleRelevance)
+	r.Post("/matrix/downgrade", h.handleDowngrade)
+	r.Post("/matrix/playbooks/record", h.handlePlaybookRecord)
+	r.Post("/matrix/playbooks/bulk", h.handlePlaybookBulk)
+}
+
+func (h *handlers) handleSearch(w http.ResponseWriter, r *http.Request) {
+	var req matrix.SearchRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	resp, err := h.r.Search(r.Context(), req)
+	if err != nil {
+		writeMatrixError(w, err)
+		return
+	}
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// relevanceRequest is the POST /matrix/relevance body. Threshold
+// defaults to matrix.DefaultRelevanceThreshold when zero.
+type relevanceRequest struct {
+	Focus     matrix.FocusFile        `json:"focus"`
+	Chunks    []matrix.CandidateChunk `json:"chunks"`
+	Threshold float64                 `json:"threshold,omitempty"`
+}
+
+func (h *handlers) handleRelevance(w http.ResponseWriter, r *http.Request) {
+	var req relevanceRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if len(req.Chunks) == 0 {
+		http.Error(w, "chunks must be non-empty", http.StatusBadRequest)
+		return
+	}
+	threshold := req.Threshold
+	if threshold == 0 {
+		threshold = matrix.DefaultRelevanceThreshold
+	}
+	res := matrix.FilterChunks(req.Focus, req.Chunks, threshold)
+	writeJSON(w, http.StatusOK, res)
+}
+
+// playbookRecordRequest is the POST /matrix/playbooks/record body.
+// Corpus is optional; defaults to matrix.DefaultPlaybookCorpus.
+// Role is optional; when set, gates Shape A boost + Shape B inject
+// at retrieve time so this entry only fires for queries with the
+// same role (real_001 cross-role bleed fix). Empty Role = current
+// behavior (no gate).
+type playbookRecordRequest struct {
+	QueryText    string   `json:"query_text"`
+	Role         string   `json:"role,omitempty"`
+	AnswerID     string   `json:"answer_id"`
+	AnswerCorpus string   `json:"answer_corpus"`
+	Score        float64  `json:"score"`
+	Tags         []string `json:"tags,omitempty"`
+	Corpus       string   `json:"corpus,omitempty"`
+}
+
+func (h *handlers) handlePlaybookRecord(w http.ResponseWriter, r *http.Request) {
+	var req playbookRecordRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	entry := matrix.NewPlaybookEntryWithRole(req.QueryText, req.Role, req.AnswerID, req.AnswerCorpus, req.Score, req.Tags)
+	if err := entry.Validate(); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	pbID, err := h.r.Record(r.Context(), entry, req.Corpus)
+	if err != nil {
+		slog.Warn("playbook record", "err", err)
+		http.Error(w, err.Error(), http.StatusBadGateway)
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{
+		"playbook_id":   pbID,
+		"query_text":    entry.QueryText,
+		"answer_id":     entry.AnswerID,
+		"answer_corpus": entry.AnswerCorpus,
+		"score":         entry.Score,
+	})
+}
+
+// playbookBulkRequest is the POST /matrix/playbooks/bulk body —
+// component C (operational rating wiring). Used to backfill
+// historical placement data, or batch-record a session's worth of
+// coordinator click-tracking. Each Entry is recorded independently;
+// failures are reported per-entry without aborting the batch.
+type playbookBulkRequest struct {
+	Entries []playbookRecordRequest `json:"entries"`
+	Corpus  string                  `json:"corpus,omitempty"` // applies to all if entry-level not set
+}
+
+// playbookBulkResult reports per-entry outcomes plus the aggregate
+// count. Errors include the entry index so callers can locate the
+// offending record without diffing.
+type playbookBulkResult struct {
+	Recorded int                      `json:"recorded"`
+	Failed   int                      `json:"failed"`
+	Results  []playbookBulkItemResult `json:"results"`
+}
+
+type playbookBulkItemResult struct {
+	Index      int    `json:"index"`
+	PlaybookID string `json:"playbook_id,omitempty"`
+	Error      string `json:"error,omitempty"`
+}
+
+func (h *handlers) handlePlaybookBulk(w http.ResponseWriter, r *http.Request) {
+	var req playbookBulkRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if len(req.Entries) == 0 {
+		http.Error(w, "entries must be non-empty", http.StatusBadRequest)
+		return
+	}
+
+	out := playbookBulkResult{
+		Results: make([]playbookBulkItemResult, len(req.Entries)),
+	}
+	for i, item := range req.Entries {
+		corpus := item.Corpus
+		if corpus == "" {
+			corpus = req.Corpus
+		}
+		entry := matrix.NewPlaybookEntryWithRole(item.QueryText, item.Role, item.AnswerID, item.AnswerCorpus, item.Score, item.Tags)
+		if err := entry.Validate(); err != nil {
+			out.Results[i] = playbookBulkItemResult{Index: i, Error: err.Error()}
+			out.Failed++
+			continue
+		}
+		pbID, err := h.r.Record(r.Context(), entry, corpus)
+		if err != nil {
+			out.Results[i] = playbookBulkItemResult{Index: i, Error: err.Error()}
+			out.Failed++
+			continue
+		}
+		out.Results[i] = playbookBulkItemResult{Index: i, PlaybookID: pbID}
+		out.Recorded++
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// downgradeRequest is the POST /matrix/downgrade body. Mirrors
+// matrix.DowngradeInput. When ForceFullOverride is omitted from
+// the body, the value falls back to matrixd's process env
+// (LH_FORCE_FULL_ENRICHMENT) — an opinionated default that lets
+// operators set the env var on the matrixd unit and have every
+// gate decision honor it without per-request changes. Per
+// 2026-04-29 cross-lineage scrum (Opus WARN): callers that want
+// deterministic gate behavior independent of matrixd's env should
+// pass ForceFullOverride explicitly in the body.
+type downgradeRequest struct {
+	Mode              string `json:"mode"`
+	Model             string `json:"model"`
+	ForcedMode        bool   `json:"forced_mode,omitempty"`
+	ForceFullOverride *bool  `json:"force_full_override,omitempty"`
+}
+
+func (h *handlers) handleDowngrade(w http.ResponseWriter, r *http.Request) {
+	var req downgradeRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if req.Mode == "" || req.Model == "" {
+		http.Error(w, "mode and model are required", http.StatusBadRequest)
+		return
+	}
+	in := matrix.NewDowngradeInputFromEnv(req.Mode, req.Model, req.ForcedMode)
+	if req.ForceFullOverride != nil {
+		// Explicit body override beats env, useful for tooling that
+		// wants to ask "what would the gate do under these conditions"
+		// without env pollution.
+		in.ForceFullOverride = *req.ForceFullOverride
+	}
+	in.WeakModels = h.weakModels
+	writeJSON(w, http.StatusOK, matrix.MaybeDowngrade(in))
+}
+
+func (h *handlers) handleCorpora(w http.ResponseWriter, r *http.Request) {
+	names, err := h.r.Corpora(r.Context())
+	if err != nil {
+		slog.Error("matrix corpora", "err", err)
+		http.Error(w, "vectord unavailable", http.StatusBadGateway)
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{"corpora": names, "count": len(names)})
+}
+
+func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {
+	defer r.Body.Close()
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	if err := json.NewDecoder(r.Body).Decode(v); err != nil {
+		var maxErr *http.MaxBytesError
+		if errors.As(err, &maxErr) || strings.Contains(err.Error(), "http: request body too large") {
+			http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
+			return false
+		}
+		http.Error(w, "decode body: "+err.Error(), http.StatusBadRequest)
+		return false
+	}
+	return true
+}
+
+func writeJSON(w http.ResponseWriter, code int, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(code)
+	if err := json.NewEncoder(w).Encode(v); err != nil {
+		slog.Warn("matrix write json", "err", err)
+	}
+}
+
+// writeMatrixError maps internal/matrix sentinels to HTTP statuses.
+// Corpus / embed failures bubble up as 502 (the upstream service is
+// what's wrong); validation errors are 400.
+func writeMatrixError(w http.ResponseWriter, err error) {
+	switch {
+	case errors.Is(err, matrix.ErrEmptyCorpora),
+		errors.Is(err, matrix.ErrEmptyQuery):
+		http.Error(w, err.Error(), http.StatusBadRequest)
+	case errors.Is(err, matrix.ErrCorpus),
+		errors.Is(err, matrix.ErrEmbed):
+		slog.Warn("matrix upstream", "err", err)
+		http.Error(w, err.Error(), http.StatusBadGateway)
+	default:
+		slog.Error("matrix", "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+	}
+}

cmd/matrixd/main_test.go

@@ -0,0 +1,139 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/matrix"
+)
+
+// newTestRouter builds the matrixd router with a Retriever pointing at
+// unreachable URLs. Contract-drift assertions in this file fire BEFORE
+// any retriever call, so the unreachable-upstream behavior only matters
+// for tests that exercise the success path (none here).
+func newTestRouter(t *testing.T) http.Handler {
+	t.Helper()
+	h := &handlers{r: matrix.New("http://127.0.0.1:0", "http://127.0.0.1:0")}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+// TestPlaybookRecord_OldFieldNameRejected locks against a regression of
+// the 2026-04-30 driver/matrixd contract drift: the playbook_lift driver
+// briefly sent `{"query": ...}` while matrixd parsed `{"query_text": ...}`.
+// Empty QueryText fails Validate() with "query_text required", which is
+// the exact 400 the harness saw. If anyone renames the JSON tag, this
+// test catches it before the harness has to.
+func TestPlaybookRecord_OldFieldNameRejected(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"query":"x","answer_id":"y","answer_corpus":"z","score":1.0}`)
+	req := httptest.NewRequest("POST", "/matrix/playbooks/record", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Fatalf("expected 400 for old field name, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	if !strings.Contains(w.Body.String(), "query_text required") {
+		t.Errorf("expected validation error to mention query_text, got %q", w.Body.String())
+	}
+}
+
+// TestPlaybookRecord_CurrentFieldName proves the right field name parses
+// and reaches the retriever. We can't assert 200 without a live retriever,
+// but we CAN assert the response is NOT a 400 from the validate step —
+// which is the drift-detector counterpart to the test above.
+func TestPlaybookRecord_CurrentFieldName(t *testing.T) {
+	r := newTestRouter(t)
+	body, _ := json.Marshal(map[string]any{
+		"query_text":    "forklift operator OSHA-30",
+		"answer_id":     "worker_42",
+		"answer_corpus": "workers",
+		"score":         1.0,
+		"tags":          []string{"reality-test"},
+	})
+	req := httptest.NewRequest("POST", "/matrix/playbooks/record", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	// Retriever will fail (unreachable upstream); expected outcomes are
+	// 502 (bad gateway, mapped from upstream HTTP error) or 500 (network
+	// error). Anything that's NOT a 400 means we cleared validation.
+	if w.Code == http.StatusBadRequest {
+		t.Errorf("valid request rejected at validation step: %d %s", w.Code, w.Body.String())
+	}
+}
+
+// TestPlaybookRecord_ScoreOutOfRange locks the score-bounds invariant
+// from internal/matrix/playbook.go. Negative or >1.0 scores must 400.
+func TestPlaybookRecord_ScoreOutOfRange(t *testing.T) {
+	r := newTestRouter(t)
+	for _, s := range []float64{-0.1, 1.1, 99} {
+		body, _ := json.Marshal(map[string]any{
+			"query_text":    "x",
+			"answer_id":     "y",
+			"answer_corpus": "z",
+			"score":         s,
+		})
+		req := httptest.NewRequest("POST", "/matrix/playbooks/record", bytes.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		w := httptest.NewRecorder()
+		r.ServeHTTP(w, req)
+		if w.Code != http.StatusBadRequest {
+			t.Errorf("score=%v should be rejected, got %d", s, w.Code)
+		}
+	}
+}
+
+// TestRelevance_EmptyChunks locks the explicit empty-chunks 400 in
+// handleRelevance. Keeps callers from silently getting an empty result
+// when their request was malformed.
+func TestRelevance_EmptyChunks(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"focus":{},"chunks":[]}`)
+	req := httptest.NewRequest("POST", "/matrix/relevance", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty chunks, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+// TestRoutesMounted asserts that every route in handlers.register(r)
+// resolves to a handler — i.e. none of them would 404 against a request.
+// Closes R-005 for matrixd (router-level wiring test).
+func TestRoutesMounted(t *testing.T) {
+	r := newTestRouter(t)
+	cases := []struct {
+		method, path string
+	}{
+		{"POST", "/matrix/search"},
+		{"GET", "/matrix/corpora"},
+		{"POST", "/matrix/relevance"},
+		{"POST", "/matrix/downgrade"},
+		{"POST", "/matrix/playbooks/record"},
+		{"POST", "/matrix/playbooks/bulk"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.method+" "+tc.path, func(t *testing.T) {
+			req := httptest.NewRequest(tc.method, tc.path, bytes.NewReader([]byte(`{}`)))
+			req.Header.Set("Content-Type", "application/json")
+			w := httptest.NewRecorder()
+			r.ServeHTTP(w, req)
+			if w.Code == http.StatusNotFound {
+				t.Errorf("%s %s returned 404 — route not mounted", tc.method, tc.path)
+			}
+			if w.Code == http.StatusMethodNotAllowed {
+				t.Errorf("%s %s returned 405 — wrong method registered", tc.method, tc.path)
+			}
+		})
+	}
+}

cmd/mcpd/main.go

@@ -0,0 +1,268 @@
+// mcpd is the Model Context Protocol server that exposes Lakehouse
+// capabilities as MCP tools. Replaces the Bun mcp-server.ts surface
+// (the MCP-tool-only subset; HTTP demo routes stay Bun until G5).
+//
+// Tools exposed:
+//   list_datasets     GET  /v1/catalog/list
+//   get_manifest      GET  /v1/catalog/manifest/<name>
+//   query_sql         POST /v1/sql
+//   embed_text        POST /v1/embed
+//   search_vectors    POST /v1/vectors/index/<name>/search
+//
+// Transport: StdioTransport (the universal MCP transport — Claude
+// Desktop, Claude Code, MCP CLI all speak this). Other transports
+// (SSE, HTTP) can be added later by changing the Run call.
+//
+// Setup for Claude Desktop / Claude Code:
+//   bin/mcpd --gateway http://127.0.0.1:3110
+//   (configure your client to spawn this binary as an MCP server)
+//
+// Why not in cmd/gateway: separation of concerns. Gateway is HTTP
+// for direct-API callers; mcpd is stdio for MCP consumers. Keeping
+// them separate means each can be deployed / restarted / monitored
+// without affecting the other.
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+)
+
+func main() {
+	gatewayURL := flag.String("gateway", "http://127.0.0.1:3110",
+		"Gateway URL (where mcpd proxies all tool calls)")
+	flag.Parse()
+
+	srv := buildServer(*gatewayURL)
+
+	if err := srv.Run(context.Background(), &mcp.StdioTransport{}); err != nil {
+		log.Fatalf("mcpd: %v", err)
+	}
+}
+
+// buildServer assembles the MCP server with all tools wired against
+// the given gateway URL. Extracted from main() so tests can build
+// the server with a test gateway URL.
+func buildServer(gatewayURL string) *mcp.Server {
+	srv := mcp.NewServer(&mcp.Implementation{
+		Name:    "lakehouse",
+		Version: "v0.1.0",
+	}, nil)
+
+	gw := newGatewayClient(gatewayURL)
+
+	mcp.AddTool(srv, &mcp.Tool{
+		Name: "list_datasets",
+		Description: "List all datasets registered in the catalog. " +
+			"Returns dataset_id, name, schema_fingerprint, row_count " +
+			"per dataset.",
+	}, gw.listDatasets)
+
+	mcp.AddTool(srv, &mcp.Tool{
+		Name: "get_manifest",
+		Description: "Fetch the manifest for a single dataset by name. " +
+			"Includes schema fingerprint, parquet object keys, row count, " +
+			"created_at_unix_ns.",
+	}, gw.getManifest)
+
+	mcp.AddTool(srv, &mcp.Tool{
+		Name: "query_sql",
+		Description: "Execute a SQL query against the registered datasets. " +
+			"Returns columns + rows. SQL is interpreted by DuckDB; standard " +
+			"SQL plus DuckDB-specific functions (read_parquet, etc.) work.",
+	}, gw.querySQL)
+
+	mcp.AddTool(srv, &mcp.Tool{
+		Name: "embed_text",
+		Description: "Embed one or more texts via the configured embedding " +
+			"model (default: nomic-embed-text). Returns one vector per text " +
+			"in the same order as the input.",
+	}, gw.embedText)
+
+	mcp.AddTool(srv, &mcp.Tool{
+		Name: "search_vectors",
+		Description: "Find the top-K nearest neighbors of a query vector " +
+			"in the named HNSW index. K defaults to 10 if omitted.",
+	}, gw.searchVectors)
+
+	return srv
+}
+
+// gatewayClient holds the HTTP client + base URL for proxying tool
+// calls to the Go gateway. Per-tool latency is on the order of a
+// gateway round-trip; the 30s timeout accommodates the slowest
+// expected SQL query without holding stdio sessions indefinitely.
+type gatewayClient struct {
+	baseURL string
+	hc      *http.Client
+}
+
+func newGatewayClient(baseURL string) *gatewayClient {
+	return &gatewayClient{
+		baseURL: strings.TrimRight(baseURL, "/"),
+		hc:      &http.Client{Timeout: 30 * time.Second},
+	}
+}
+
+// ── tool argument structs (jsonschema tags drive schema generation) ──
+
+type listDatasetsArgs struct{}
+
+type getManifestArgs struct {
+	Name string `json:"name" jsonschema:"the dataset name to fetch"`
+}
+
+type querySQLArgs struct {
+	SQL string `json:"sql" jsonschema:"the SQL query to execute"`
+}
+
+type embedTextArgs struct {
+	Texts []string `json:"texts" jsonschema:"the texts to embed"`
+	Model string   `json:"model,omitempty" jsonschema:"optional model name (defaults to embedd's configured default)"`
+}
+
+type searchVectorsArgs struct {
+	IndexName string    `json:"index_name" jsonschema:"the index to search"`
+	Vector    []float32 `json:"vector" jsonschema:"the query vector"`
+	K         int       `json:"k,omitempty" jsonschema:"top-K to return (default 10)"`
+}
+
+// ── tool handlers ──
+
+func (g *gatewayClient) listDatasets(ctx context.Context, _ *mcp.CallToolRequest, _ listDatasetsArgs) (*mcp.CallToolResult, any, error) {
+	body, err := g.proxy(ctx, http.MethodGet, "/v1/catalog/list", nil)
+	if err != nil {
+		return errorResult(err), nil, nil
+	}
+	return jsonResult(body), nil, nil
+}
+
+func (g *gatewayClient) getManifest(ctx context.Context, _ *mcp.CallToolRequest, args getManifestArgs) (*mcp.CallToolResult, any, error) {
+	if args.Name == "" {
+		return errorResult(fmt.Errorf("name is required")), nil, nil
+	}
+	path := "/v1/catalog/manifest/" + url.PathEscape(args.Name)
+	body, err := g.proxy(ctx, http.MethodGet, path, nil)
+	if err != nil {
+		return errorResult(err), nil, nil
+	}
+	return jsonResult(body), nil, nil
+}
+
+func (g *gatewayClient) querySQL(ctx context.Context, _ *mcp.CallToolRequest, args querySQLArgs) (*mcp.CallToolResult, any, error) {
+	if strings.TrimSpace(args.SQL) == "" {
+		return errorResult(fmt.Errorf("sql is required")), nil, nil
+	}
+	reqBody, _ := json.Marshal(map[string]string{"sql": args.SQL})
+	body, err := g.proxy(ctx, http.MethodPost, "/v1/sql", reqBody)
+	if err != nil {
+		return errorResult(err), nil, nil
+	}
+	return jsonResult(body), nil, nil
+}
+
+func (g *gatewayClient) embedText(ctx context.Context, _ *mcp.CallToolRequest, args embedTextArgs) (*mcp.CallToolResult, any, error) {
+	if len(args.Texts) == 0 {
+		return errorResult(fmt.Errorf("texts is required")), nil, nil
+	}
+	payload := map[string]any{"texts": args.Texts}
+	if args.Model != "" {
+		payload["model"] = args.Model
+	}
+	reqBody, _ := json.Marshal(payload)
+	body, err := g.proxy(ctx, http.MethodPost, "/v1/embed", reqBody)
+	if err != nil {
+		return errorResult(err), nil, nil
+	}
+	return jsonResult(body), nil, nil
+}
+
+func (g *gatewayClient) searchVectors(ctx context.Context, _ *mcp.CallToolRequest, args searchVectorsArgs) (*mcp.CallToolResult, any, error) {
+	if args.IndexName == "" {
+		return errorResult(fmt.Errorf("index_name is required")), nil, nil
+	}
+	if len(args.Vector) == 0 {
+		return errorResult(fmt.Errorf("vector is required")), nil, nil
+	}
+	payload := map[string]any{"vector": args.Vector}
+	if args.K > 0 {
+		payload["k"] = args.K
+	}
+	reqBody, _ := json.Marshal(payload)
+	path := "/v1/vectors/index/" + url.PathEscape(args.IndexName) + "/search"
+	body, err := g.proxy(ctx, http.MethodPost, path, reqBody)
+	if err != nil {
+		return errorResult(err), nil, nil
+	}
+	return jsonResult(body), nil, nil
+}
+
+// proxy makes a request to the gateway and returns the response body
+// on success. Non-2xx status codes return an error with the body
+// preview in the message — surfaced to the MCP client as a tool error
+// rather than a transport-level failure.
+func (g *gatewayClient) proxy(ctx context.Context, method, path string, body []byte) ([]byte, error) {
+	var rdr io.Reader
+	if body != nil {
+		rdr = bytes.NewReader(body)
+	}
+	req, err := http.NewRequestWithContext(ctx, method, g.baseURL+path, rdr)
+	if err != nil {
+		return nil, fmt.Errorf("build request: %w", err)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/json")
+	}
+	resp, err := g.hc.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("call gateway: %w", err)
+	}
+	defer resp.Body.Close()
+	respBody, _ := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) // 16 MiB tool-response cap
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		preview := respBody
+		if len(preview) > 512 {
+			preview = preview[:512]
+		}
+		return nil, fmt.Errorf("gateway %s %s: status %d: %s",
+			method, path, resp.StatusCode, string(preview))
+	}
+	return respBody, nil
+}
+
+// errorResult wraps an error as an MCP tool error result. The MCP
+// protocol distinguishes "tool ran but reported failure" (returned
+// in CallToolResult.IsError + content) from "tool threw" (returned
+// as the third return value). We use the former so the LLM caller
+// sees the error text and can decide how to react, rather than
+// surfacing the error as transport noise.
+func errorResult(err error) *mcp.CallToolResult {
+	return &mcp.CallToolResult{
+		IsError: true,
+		Content: []mcp.Content{
+			&mcp.TextContent{Text: err.Error()},
+		},
+	}
+}
+
+// jsonResult wraps a JSON byte slice as a successful tool result.
+// The content is text — MCP clients render it; LLMs parse it as
+// JSON when their tool config indicates so.
+func jsonResult(body []byte) *mcp.CallToolResult {
+	return &mcp.CallToolResult{
+		Content: []mcp.Content{
+			&mcp.TextContent{Text: string(body)},
+		},
+	}
+}

cmd/mcpd/main_test.go

@@ -0,0 +1,282 @@
+package main
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+)
+
+// Tests the MCP tool surface end-to-end without a subprocess: spin
+// up a fake gateway via httptest, build the MCP server pointed at
+// it, connect a client via in-memory transports, call each tool.
+
+// fakeGateway returns an httptest.Server that responds to the routes
+// mcpd proxies. Each route's handler is configurable via the routes map.
+func fakeGateway(t *testing.T, routes map[string]http.HandlerFunc) *httptest.Server {
+	t.Helper()
+	mux := http.NewServeMux()
+	for path, h := range routes {
+		mux.HandleFunc(path, h)
+	}
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		t.Errorf("fakeGateway: unexpected route %s %s", r.Method, r.URL.Path)
+		http.NotFound(w, r)
+	})
+	srv := httptest.NewServer(mux)
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+// connect builds the mcpd server pointed at gatewayURL, connects an
+// in-memory client, and returns a ready-to-use ClientSession.
+func connect(t *testing.T, gatewayURL string) *mcp.ClientSession {
+	t.Helper()
+	ctx := context.Background()
+
+	srv := buildServer(gatewayURL)
+	client := mcp.NewClient(&mcp.Implementation{Name: "test-client", Version: "v0.0.1"}, nil)
+
+	t1, t2 := mcp.NewInMemoryTransports()
+	if _, err := srv.Connect(ctx, t1, nil); err != nil {
+		t.Fatalf("server connect: %v", err)
+	}
+	session, err := client.Connect(ctx, t2, nil)
+	if err != nil {
+		t.Fatalf("client connect: %v", err)
+	}
+	t.Cleanup(func() { _ = session.Close() })
+	return session
+}
+
+func callTool(t *testing.T, session *mcp.ClientSession, name string, args any) *mcp.CallToolResult {
+	t.Helper()
+	res, err := session.CallTool(context.Background(), &mcp.CallToolParams{
+		Name:      name,
+		Arguments: args,
+	})
+	if err != nil {
+		t.Fatalf("CallTool %s: %v", name, err)
+	}
+	return res
+}
+
+func resultText(t *testing.T, res *mcp.CallToolResult) string {
+	t.Helper()
+	if len(res.Content) == 0 {
+		t.Fatal("result has no content")
+	}
+	tc, ok := res.Content[0].(*mcp.TextContent)
+	if !ok {
+		t.Fatalf("first content is %T, want *mcp.TextContent", res.Content[0])
+	}
+	return tc.Text
+}
+
+func TestListTools(t *testing.T) {
+	gw := fakeGateway(t, nil)
+	session := connect(t, gw.URL)
+
+	res, err := session.ListTools(context.Background(), nil)
+	if err != nil {
+		t.Fatalf("ListTools: %v", err)
+	}
+	want := map[string]bool{
+		"list_datasets":  false,
+		"get_manifest":   false,
+		"query_sql":      false,
+		"embed_text":     false,
+		"search_vectors": false,
+	}
+	for _, tool := range res.Tools {
+		if _, ok := want[tool.Name]; ok {
+			want[tool.Name] = true
+		}
+	}
+	for name, found := range want {
+		if !found {
+			t.Errorf("expected tool %q exposed by mcpd, not in ListTools", name)
+		}
+	}
+}
+
+func TestListDatasets_HappyPath(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/catalog/list": func(w http.ResponseWriter, _ *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			_, _ = w.Write([]byte(`{"manifests":[{"name":"workers","row_count":500000}],"count":1}`))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "list_datasets", listDatasetsArgs{})
+	if res.IsError {
+		t.Fatalf("expected success, got IsError with: %s", resultText(t, res))
+	}
+	body := resultText(t, res)
+	if !strings.Contains(body, "workers") || !strings.Contains(body, "500000") {
+		t.Errorf("response missing expected fields: %s", body)
+	}
+}
+
+func TestGetManifest_HappyPath(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/catalog/manifest/workers": func(w http.ResponseWriter, _ *http.Request) {
+			_, _ = w.Write([]byte(`{"name":"workers","row_count":500000,"schema_fingerprint":"sha256:abc"}`))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "get_manifest", getManifestArgs{Name: "workers"})
+	if res.IsError {
+		t.Fatalf("expected success, got: %s", resultText(t, res))
+	}
+	body := resultText(t, res)
+	if !strings.Contains(body, "workers") {
+		t.Errorf("missing manifest fields in response: %s", body)
+	}
+}
+
+func TestGetManifest_EmptyName_IsError(t *testing.T) {
+	gw := fakeGateway(t, nil) // no handlers — tool should error before hitting gateway
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "get_manifest", getManifestArgs{Name: ""})
+	if !res.IsError {
+		t.Fatal("expected IsError on empty name")
+	}
+}
+
+func TestQuerySQL_HappyPath(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/sql": func(w http.ResponseWriter, r *http.Request) {
+			if r.Method != "POST" {
+				t.Errorf("/v1/sql got %s, want POST", r.Method)
+			}
+			_, _ = w.Write([]byte(`{"columns":[{"name":"n","type":"BIGINT"}],"rows":[[5]],"row_count":1}`))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "query_sql", querySQLArgs{SQL: "SELECT count(*) FROM workers"})
+	if res.IsError {
+		t.Fatalf("expected success, got: %s", resultText(t, res))
+	}
+	if !strings.Contains(resultText(t, res), `"row_count":1`) {
+		t.Errorf("response missing row_count field: %s", resultText(t, res))
+	}
+}
+
+func TestQuerySQL_EmptySQL_IsError(t *testing.T) {
+	gw := fakeGateway(t, nil)
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "query_sql", querySQLArgs{SQL: "   "})
+	if !res.IsError {
+		t.Fatal("expected IsError on whitespace SQL")
+	}
+}
+
+func TestEmbedText_HappyPath(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/embed": func(w http.ResponseWriter, _ *http.Request) {
+			_, _ = w.Write([]byte(`{"model":"nomic-embed-text","dimension":768,"vectors":[[0.1,0.2]]}`))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "embed_text", embedTextArgs{Texts: []string{"hello"}})
+	if res.IsError {
+		t.Fatalf("expected success, got: %s", resultText(t, res))
+	}
+	if !strings.Contains(resultText(t, res), `"dimension":768`) {
+		t.Errorf("missing dimension in response: %s", resultText(t, res))
+	}
+}
+
+func TestEmbedText_EmptyTexts_IsError(t *testing.T) {
+	gw := fakeGateway(t, nil)
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "embed_text", embedTextArgs{Texts: nil})
+	if !res.IsError {
+		t.Fatal("expected IsError on empty texts")
+	}
+}
+
+func TestSearchVectors_HappyPath(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/vectors/index/test_idx/search": func(w http.ResponseWriter, _ *http.Request) {
+			_, _ = w.Write([]byte(`{"results":[{"id":"v1","distance":0.001}]}`))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "search_vectors", searchVectorsArgs{
+		IndexName: "test_idx",
+		Vector:    []float32{1, 0, 0, 0},
+		K:         5,
+	})
+	if res.IsError {
+		t.Fatalf("expected success, got: %s", resultText(t, res))
+	}
+	if !strings.Contains(resultText(t, res), `"id":"v1"`) {
+		t.Errorf("missing top-1 in response: %s", resultText(t, res))
+	}
+}
+
+func TestSearchVectors_MissingIndex_IsError(t *testing.T) {
+	gw := fakeGateway(t, nil)
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "search_vectors", searchVectorsArgs{
+		Vector: []float32{1, 0, 0, 0},
+	})
+	if !res.IsError {
+		t.Fatal("expected IsError on missing index_name")
+	}
+}
+
+func TestSearchVectors_MissingVector_IsError(t *testing.T) {
+	gw := fakeGateway(t, nil)
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "search_vectors", searchVectorsArgs{
+		IndexName: "test_idx",
+	})
+	if !res.IsError {
+		t.Fatal("expected IsError on missing vector")
+	}
+}
+
+func TestGatewayError_PropagatesAsIsError(t *testing.T) {
+	gw := fakeGateway(t, map[string]http.HandlerFunc{
+		"/v1/sql": func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusBadRequest)
+			_, _ = w.Write([]byte("syntax error: no such table"))
+		},
+	})
+	session := connect(t, gw.URL)
+
+	res := callTool(t, session, "query_sql", querySQLArgs{SQL: "SELECT * FROM nope"})
+	if !res.IsError {
+		t.Fatal("expected IsError on gateway 4xx")
+	}
+	body := resultText(t, res)
+	if !strings.Contains(body, "400") {
+		t.Errorf("error result should mention status 400, got: %s", body)
+	}
+}
+
+func TestGatewayUnreachable_PropagatesAsIsError(t *testing.T) {
+	// Point at a non-listening port — connect will fail.
+	session := connect(t, "http://127.0.0.1:1") // reserved port
+
+	res := callTool(t, session, "list_datasets", listDatasetsArgs{})
+	if !res.IsError {
+		t.Fatal("expected IsError on unreachable gateway")
+	}
+}

cmd/observerd/main.go

@@ -0,0 +1,364 @@
+// observerd is the autonomous-iteration witness service. Port of
+// the load-bearing pieces of mcp-server/observer.ts (Rust system).
+//
+// Routes (all under /observer):
+//   GET  /observer/health        — service liveness + ring size
+//   GET  /observer/stats         — aggregate counters + recent scenarios
+//   POST /observer/event         — record one observed op
+//
+// Deferred to follow-up commits (see internal/observer doc):
+//   - POST /observer/review (cloud-LLM hand review fall-back)
+//   - background loops (analyzeErrors, consolidatePlaybooks,
+//     tailOverseerCorrections)
+//   - failure-cluster escalation to LLM Team
+//
+// /relevance was already ported to internal/matrix in 9588bd8 and is
+// not duplicated here.
+
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/observer"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/workflow"
+)
+
+const maxRequestBytes = 4 << 20 // 4 MiB cap on request bodies
+
+func main() {
+	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
+	flag.Parse()
+
+	cfg, err := shared.LoadConfig(*configPath)
+	if err != nil {
+		slog.Error("config", "err", err)
+		os.Exit(1)
+	}
+
+	// Persistence is optional — empty path = ephemeral (matches the
+	// pathwayd pattern). Production sets a stable path under
+	// /var/lib/lakehouse/observer/ops.jsonl.
+	var persistor *observer.Persistor
+	if cfg.Observerd.PersistPath != "" {
+		persistor, err = observer.NewPersistor(cfg.Observerd.PersistPath)
+		if err != nil {
+			slog.Error("observer persistor", "err", err)
+			os.Exit(1)
+		}
+	}
+
+	store := observer.NewStore(persistor)
+	if persistor != nil {
+		n, err := store.Load()
+		if err != nil {
+			slog.Warn("observer load", "err", err, "loaded", n)
+		} else {
+			slog.Info("observer loaded", "ops", n, "path", cfg.Observerd.PersistPath)
+		}
+	}
+
+	runner := workflow.NewRunner()
+	// matrixd URL: prefer explicit observerd config field, fall back
+	// to gateway's matrixd_url so a single-toml deploy works without
+	// duplicating the address.
+	matrixdURL := cfg.Gateway.MatrixdURL
+	registerBuiltinModes(runner, matrixdURL, cfg.Models.WeakModels)
+
+	h := &handlers{store: store, runner: runner}
+	if err := shared.Run("observerd", cfg.Observerd.Bind, h.register, cfg.Auth); err != nil {
+		slog.Error("server", "err", err)
+		os.Exit(1)
+	}
+}
+
+type handlers struct {
+	store  *observer.Store
+	runner *workflow.Runner
+}
+
+func (h *handlers) register(r chi.Router) {
+	r.Get("/observer/stats", h.handleStats)
+	r.Post("/observer/event", h.handleEvent)
+	r.Post("/observer/workflow/run", h.handleWorkflowRun)
+	r.Get("/observer/workflow/modes", h.handleWorkflowModes)
+	r.Post("/observer/inbox", h.handleInbox)
+}
+
+// inboxMessage is the POST /observer/inbox body — an incoming
+// real-world signal (email or SMS) that a coordinator would receive
+// and act on. The handler only RECORDS it as an ObservedOp; whether
+// to trigger a downstream matrix.search or workflow is the caller's
+// concern. Keeps observer's witness role pure.
+type inboxMessage struct {
+	Type     string `json:"type"`     // "email" | "sms"
+	Sender   string `json:"sender"`
+	Subject  string `json:"subject,omitempty"`
+	Body     string `json:"body"`
+	Priority string `json:"priority"` // "urgent" | "high" | "medium" | "low"
+	Tag      string `json:"tag,omitempty"`
+}
+
+// validInboxPriorities is the closed set of priority values. Per
+// scrum (Kimi WARN): a free-form priority string was accepted, so
+// downstream consumers parsing on exact enum could see "nonsense"
+// values. Now rejected at the boundary.
+var validInboxPriorities = map[string]bool{
+	"":       true, // empty defaults to medium below
+	"urgent": true,
+	"high":   true,
+	"medium": true,
+	"low":    true,
+}
+
+// inboxMaxBodyChars caps the inbox body at a sane size so a large
+// payload doesn't bloat observer's ring buffer or JSONL log. Per
+// scrum (Opus WARN): only emptiness was checked, so a multi-MB body
+// would be accepted unconditionally.
+const inboxMaxBodyChars = 8 * 1024
+
+// inboxMaxFieldChars caps subject/sender/tag at a much tighter size.
+// These appear inside the InputSummary string + JSONL log entry, so
+// long values bloat every record and embedded newlines corrupt
+// downstream line-based parsers.
+const inboxMaxFieldChars = 256
+
+func sanitizeInboxField(s string) string {
+	// Strip newlines so multi-line attacker-controlled strings don't
+	// corrupt the JSONL log (one-line-per-op invariant).
+	s = strings.ReplaceAll(s, "\n", " ")
+	s = strings.ReplaceAll(s, "\r", " ")
+	if len(s) > inboxMaxFieldChars {
+		s = s[:inboxMaxFieldChars]
+	}
+	return s
+}
+
+func (h *handlers) handleInbox(w http.ResponseWriter, r *http.Request) {
+	var msg inboxMessage
+	if !decodeJSON(w, r, &msg) {
+		return
+	}
+	if msg.Type != "email" && msg.Type != "sms" {
+		http.Error(w, "type must be 'email' or 'sms'", http.StatusBadRequest)
+		return
+	}
+	if strings.TrimSpace(msg.Body) == "" {
+		http.Error(w, "body required", http.StatusBadRequest)
+		return
+	}
+	if len(msg.Body) > inboxMaxBodyChars {
+		http.Error(w, fmt.Sprintf("body exceeds %d chars", inboxMaxBodyChars), http.StatusRequestEntityTooLarge)
+		return
+	}
+	if !validInboxPriorities[msg.Priority] {
+		http.Error(w, "priority must be urgent|high|medium|low", http.StatusBadRequest)
+		return
+	}
+	if msg.Priority == "" {
+		msg.Priority = "medium"
+	}
+	// Sanitize the fields that go into the InputSummary string so
+	// newlines don't corrupt the JSONL log.
+	sender := sanitizeInboxField(msg.Sender)
+	subject := sanitizeInboxField(msg.Subject)
+	tag := sanitizeInboxField(msg.Tag)
+	op := observer.ObservedOp{
+		Endpoint:      "/observer/inbox/" + msg.Type,
+		InputSummary:  fmt.Sprintf("from=%s priority=%s tag=%s subject=%s", sender, msg.Priority, tag, subject),
+		OutputSummary: msg.Body,
+		Source:        observer.SourceInbox,
+		Success:       true,
+	}
+	if err := h.store.Record(op); err != nil {
+		if errors.Is(err, observer.ErrInvalidOp) {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		slog.Error("observer record inbox", "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+		return
+	}
+	stats := h.store.Stats()
+	writeJSON(w, http.StatusOK, map[string]any{
+		"accepted":  true,
+		"type":      msg.Type,
+		"priority":  msg.Priority,
+		"ring_size": stats.Total,
+	})
+}
+
+func (h *handlers) handleStats(w http.ResponseWriter, _ *http.Request) {
+	writeJSON(w, http.StatusOK, h.store.Stats())
+}
+
+func (h *handlers) handleEvent(w http.ResponseWriter, r *http.Request) {
+	var op observer.ObservedOp
+	if !decodeJSON(w, r, &op) {
+		return
+	}
+	if err := h.store.Record(op); err != nil {
+		if errors.Is(err, observer.ErrInvalidOp) {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		slog.Error("observer record", "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+		return
+	}
+	stats := h.store.Stats()
+	writeJSON(w, http.StatusOK, map[string]any{
+		"accepted":  true,
+		"ring_size": stats.Total,
+	})
+}
+
+// workflowRunRequest is the POST /observer/workflow/run body — a
+// Workflow definition in JSON form (matches Archon's YAML shape but
+// JSON-serialized for the HTTP path).
+type workflowRunRequest struct {
+	Workflow workflow.Workflow `json:"workflow"`
+}
+
+func (h *handlers) handleWorkflowRun(r http.ResponseWriter, req *http.Request) {
+	var body workflowRunRequest
+	if !decodeJSON(r, req, &body) {
+		return
+	}
+	res, err := h.runner.Run(req.Context(), body.Workflow)
+	// Record per-node provenance into the observer ring AS the
+	// workflow runs — same shape as any other ObservedOp so the
+	// existing /observer/stats aggregation surfaces workflow ops
+	// alongside scenario ops without a schema change.
+	for _, n := range res.Nodes {
+		op := observer.ObservedOp{
+			Endpoint:      "/observer/workflow/run/" + body.Workflow.Name + "/" + n.NodeID,
+			InputSummary:  fmt.Sprintf("workflow=%s node=%s mode=%s", body.Workflow.Name, n.NodeID, n.Mode),
+			Success:       n.Error == "",
+			DurationMs:    n.DurationMs,
+			OutputSummary: summarizeOutput(n.Output),
+			Source:        observer.SourceWorkflow,
+			Error:         n.Error,
+			Timestamp:     n.StartedAt.UTC().Format(time.RFC3339Nano),
+		}
+		if recErr := h.store.Record(op); recErr != nil {
+			slog.Warn("workflow run: provenance record failed", "err", recErr)
+		}
+	}
+	if err != nil {
+		// Aborting errors (cycle, missing dep, unknown mode) — surface
+		// as 4xx because the workflow definition itself is wrong.
+		slog.Warn("workflow run aborted", "err", err)
+		writeJSON(r, http.StatusBadRequest, map[string]any{
+			"error":  err.Error(),
+			"result": res,
+		})
+		return
+	}
+	writeJSON(r, http.StatusOK, res)
+}
+
+func (h *handlers) handleWorkflowModes(w http.ResponseWriter, _ *http.Request) {
+	modes := h.runner.Modes()
+	writeJSON(w, http.StatusOK, map[string]any{
+		"modes": modes,
+		"count": len(modes),
+	})
+}
+
+// summarizeOutput renders a workflow node's output map for the
+// ObservedOp's OutputSummary string. Best-effort — long values get
+// truncated rather than ballooning the ring buffer's memory.
+func summarizeOutput(output map[string]any) string {
+	if output == nil {
+		return "(nil)"
+	}
+	bs, err := json.Marshal(output)
+	if err != nil {
+		return fmt.Sprintf("(marshal err: %v)", err)
+	}
+	if len(bs) > 256 {
+		return string(bs[:256]) + "...(truncated)"
+	}
+	return string(bs)
+}
+
+// registerBuiltinModes wires the modes the runner knows about. The
+// pure-function wrappers (matrix.relevance, matrix.downgrade,
+// distillation.score, drift.scorer) are direct Go calls. matrix.search
+// is HTTP-backed, pointed at the configured matrixd_url so workflows
+// can compose retrieval into multi-pass measurement chains.
+//
+// Fixture modes (fixture.echo, fixture.upper) stay registered for
+// the workflow_smoke that proves the runner mechanics independently
+// of the real modes' availability.
+//
+// Real-mode follow-ups still pending:
+//   - playbook.record (HTTP to matrixd)
+//   - playbook.lookup (HTTP to matrixd)
+//   - llm.chat (HTTP to gateway /v1/chat)
+func registerBuiltinModes(r *workflow.Runner, matrixdURL string, weakModels []string) {
+	// Fixture modes for runner mechanics smokes.
+	r.RegisterMode("fixture.echo", func(_ workflow.Context, input map[string]any) (map[string]any, error) {
+		out := make(map[string]any, len(input))
+		for k, v := range input {
+			out[k] = v
+		}
+		return out, nil
+	})
+	r.RegisterMode("fixture.upper", func(_ workflow.Context, input map[string]any) (map[string]any, error) {
+		prompt, _ := input["prompt"].(string)
+		return map[string]any{"upper": strings.ToUpper(prompt)}, nil
+	})
+
+	// Real modes — pure-function wrappers (no I/O).
+	r.RegisterMode("matrix.relevance", workflow.MatrixRelevance)
+	// matrix.downgrade reads weakModels from config — Phase 2.
+	// nil/empty falls back to matrix.DefaultWeakModels per the
+	// MatrixDowngradeWithWeakList factory contract.
+	r.RegisterMode("matrix.downgrade", workflow.MatrixDowngradeWithWeakList(weakModels))
+	r.RegisterMode("distillation.score", workflow.DistillationScore)
+	r.RegisterMode("drift.scorer", workflow.DriftScorer)
+
+	// HTTP-backed modes — only register when their backend URL is set.
+	// matrixd_url defaults to a known address but tests/dev may run
+	// without matrixd.
+	if matrixdURL != "" {
+		hc := &http.Client{Timeout: 30 * time.Second}
+		r.RegisterMode("matrix.search", workflow.MatrixSearch(matrixdURL, hc))
+	}
+}
+
+func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {
+	defer r.Body.Close()
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	if err := json.NewDecoder(r.Body).Decode(v); err != nil {
+		var maxErr *http.MaxBytesError
+		if errors.As(err, &maxErr) || strings.Contains(err.Error(), "http: request body too large") {
+			http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
+			return false
+		}
+		http.Error(w, "decode body: "+err.Error(), http.StatusBadRequest)
+		return false
+	}
+	return true
+}
+
+func writeJSON(w http.ResponseWriter, code int, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(code)
+	if err := json.NewEncoder(w).Encode(v); err != nil {
+		slog.Warn("observer write json", "err", err)
+	}
+}

cmd/observerd/main_test.go

@@ -0,0 +1,258 @@
+package main
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/observer"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/workflow"
+)
+
+// newTestRouter builds the observerd router with an in-memory store
+// and a workflow runner with no modes registered. Closes R-005 for
+// observerd.
+//
+// Returns chi.Router (not http.Handler) so chi.Walk works without a
+// type assertion that would panic if a future refactor wraps the
+// router in plain net/http middleware.
+func newTestRouter(t *testing.T) chi.Router {
+	t.Helper()
+	h := &handlers{
+		store:  observer.NewStore(nil),
+		runner: workflow.NewRunner(),
+	}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	r := newTestRouter(t)
+	want := map[string]bool{
+		"GET /observer/stats":           false,
+		"POST /observer/event":          false,
+		"POST /observer/workflow/run":   false,
+		"GET /observer/workflow/modes":  false,
+		"POST /observer/inbox":          false,
+	}
+	_ = chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		key := method + " " + route
+		if _, ok := want[key]; ok {
+			want[key] = true
+		}
+		return nil
+	})
+	for k, mounted := range want {
+		if !mounted {
+			t.Errorf("route not mounted: %s", k)
+		}
+	}
+}
+
+func TestStats_GET(t *testing.T) {
+	r := newTestRouter(t)
+	req := httptest.NewRequest("GET", "/observer/stats", nil)
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Errorf("expected 200, got %d", w.Code)
+	}
+}
+
+func TestWorkflowModes_GET(t *testing.T) {
+	r := newTestRouter(t)
+	req := httptest.NewRequest("GET", "/observer/workflow/modes", nil)
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Errorf("expected 200, got %d", w.Code)
+	}
+}
+
+// TestEvent_InvalidOp locks the validation path: an ObservedOp with
+// missing required fields must 400, not 500. Without this assertion,
+// observer.ErrInvalidOp could silently slip into the 500 branch on a
+// future refactor and clients would see "internal" instead of the
+// actual validation error.
+func TestEvent_InvalidOp(t *testing.T) {
+	r := newTestRouter(t)
+	// Empty body — no endpoint, no source — fails ObservedOp validation.
+	body := []byte(`{}`)
+	req := httptest.NewRequest("POST", "/observer/event", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on invalid op, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+// TestWorkflowRun_AllProvenanceRecordedPostRun proves the gap ratified
+// in ADR-005 Decision 5.3: handleWorkflowRun calls runner.Run
+// synchronously and only records ObservedOps from the returned
+// RunResult AFTER Run completes. A crash mid-Run would lose ALL
+// provenance for that workflow.
+//
+// The test pauses inside a node, samples observer state (must be 0),
+// unblocks, then samples again (must be N). If a future commit adds
+// per-node streaming (e.g. runner.NodeHook firing before Run returns),
+// the first assertion fires — that's the intentional test-as-spec
+// lock so the behavior change is visible in `go test` instead of
+// surfacing under load.
+func TestWorkflowRun_AllProvenanceRecordedPostRun(t *testing.T) {
+	pauseCh := make(chan struct{})
+
+	runner := workflow.NewRunner()
+	runner.RegisterMode("test.pause", func(_ workflow.Context, _ map[string]any) (map[string]any, error) {
+		<-pauseCh
+		return map[string]any{"unpaused": true}, nil
+	})
+
+	h := &handlers{
+		store:  observer.NewStore(nil),
+		runner: runner,
+	}
+	r := chi.NewRouter()
+	h.register(r)
+
+	// Two-node serial workflow so we have something to record post-run.
+	body := []byte(`{"workflow":{"name":"adr_005_5_3","nodes":[
+        {"id":"n1","mode":"test.pause"},
+        {"id":"n2","mode":"test.pause","depends_on":["n1"]}
+    ]}}`)
+
+	// Send the request in a goroutine — it'll block until pauseCh closes.
+	done := make(chan int)
+	go func() {
+		req := httptest.NewRequest("POST", "/observer/workflow/run", bytes.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		w := httptest.NewRecorder()
+		r.ServeHTTP(w, req)
+		done <- w.Code
+	}()
+
+	// Wait briefly for the runner to enter n1 and block on pauseCh.
+	// 50ms is conservative; the goroutine + chi routing + topo sort
+	// take well under that on this hardware.
+	time.Sleep(50 * time.Millisecond)
+
+	// LOCK: store MUST be empty while runner.Run is paused.
+	// If a future change adds streaming-record-as-each-node-finishes,
+	// n1's record would land here as soon as n1 returns — but n1
+	// hasn't returned yet (we're paused before it does), so the
+	// only way this assertion passes is if recording is post-run-only.
+	if got := h.store.Stats().Total; got != 0 {
+		t.Errorf("expected 0 observer ops during paused run, got %d "+
+			"(if non-zero, ADR-005 Decision 5.3 must be updated — recording "+
+			"is no longer post-run-only)", got)
+	}
+
+	// Unblock all paused nodes (channel close broadcasts to all receivers).
+	close(pauseCh)
+
+	// Wait for the handler to return + record post-run.
+	if code := <-done; code != http.StatusOK {
+		t.Errorf("workflow run failed: HTTP %d", code)
+	}
+
+	// LOCK: store MUST have 2 ops after run completes.
+	if got := h.store.Stats().Total; got != 2 {
+		t.Errorf("expected 2 observer ops after run, got %d", got)
+	}
+}
+
+// TestInbox_AcceptsValidEmail locks the happy-path contract for the
+// /observer/inbox route — accepts an email message with required
+// fields, records as ObservedOp, returns 200 with ring-size.
+func TestInbox_AcceptsValidEmail(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"type":"email","sender":"client@northstar.com","subject":"URGENT: 50 forklift ops","body":"Need 50 forklift operators in Cleveland OH for next week. Day shift.","priority":"urgent","tag":"alpha-surge"}`)
+	req := httptest.NewRequest("POST", "/observer/inbox", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	if !strings.Contains(w.Body.String(), `"accepted":true`) {
+		t.Errorf("expected accepted=true, got %s", w.Body.String())
+	}
+}
+
+// TestInbox_RejectsBadType locks the validation: type must be
+// "email" or "sms", anything else is 400.
+func TestInbox_RejectsBadType(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"type":"smoke-signal","sender":"x","body":"y","priority":"high"}`)
+	req := httptest.NewRequest("POST", "/observer/inbox", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on bad type, got %d", w.Code)
+	}
+}
+
+// TestInbox_RejectsEmptyBody locks the body-required invariant.
+func TestInbox_RejectsEmptyBody(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"type":"email","sender":"x","body":"","priority":"high"}`)
+	req := httptest.NewRequest("POST", "/observer/inbox", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty body, got %d", w.Code)
+	}
+}
+
+// TestInbox_RejectsBadPriority locks the priority enum (per
+// scrum Kimi WARN — free-form priority strings were accepted).
+func TestInbox_RejectsBadPriority(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"type":"email","sender":"x","body":"y","priority":"nonsense"}`)
+	req := httptest.NewRequest("POST", "/observer/inbox", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on bad priority, got %d", w.Code)
+	}
+}
+
+// TestInbox_RejectsOversizedBody locks the body-size cap (per scrum
+// Opus WARN — only emptiness was checked before).
+func TestInbox_RejectsOversizedBody(t *testing.T) {
+	r := newTestRouter(t)
+	bigBody := strings.Repeat("x", inboxMaxBodyChars+1)
+	body := []byte(`{"type":"email","sender":"x","body":"` + bigBody + `","priority":"high"}`)
+	req := httptest.NewRequest("POST", "/observer/inbox", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusRequestEntityTooLarge {
+		t.Errorf("expected 413 on oversized body, got %d", w.Code)
+	}
+}
+
+// TestWorkflowRun_UnknownMode locks the 400 path on workflow definitions
+// that reference modes not registered with the runner. The harness's
+// reality test runs depend on this so an unknown-mode misconfiguration
+// surfaces as a definition error, not a server error.
+func TestWorkflowRun_UnknownMode(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"workflow":{"name":"t","nodes":[{"id":"n1","mode":"does.not.exist"}]}}`)
+	req := httptest.NewRequest("POST", "/observer/workflow/run", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Errorf("expected 400 on unknown mode, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}

cmd/pathwayd/main.go

@@ -0,0 +1,278 @@
+// pathwayd is the pathway memory service. Wraps internal/pathway's
+// Store with HTTP routes for the Mem0-style operations defined in
+// ADR-004.
+//
+// Routes (all under /pathway):
+//   POST /pathway/add              — new trace with fresh UID
+//   POST /pathway/add_idempotent   — UID-keyed add or replay-bump
+//   POST /pathway/update           — replace content for an existing UID
+//   POST /pathway/revise           — new revision linked to predecessor
+//   POST /pathway/retire           — mark trace retired (excluded from search)
+//   GET  /pathway/get/{uid}        — fetch one trace (incl. retired)
+//   GET  /pathway/history/{uid}    — backward chain via predecessor links
+//   POST /pathway/search           — filter-based listing
+//   GET  /pathway/stats            — total/active/retired counters
+//
+// Persistence: optional. Empty [pathwayd].persist_path = in-memory
+// only (matches vectord G1's pattern). Set a path for durable
+// per-trace JSONL append.
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"flag"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/pathway"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
+)
+
+const maxRequestBytes = 4 << 20 // 4 MiB cap on request bodies
+
+func main() {
+	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
+	flag.Parse()
+
+	cfg, err := shared.LoadConfig(*configPath)
+	if err != nil {
+		slog.Error("config", "err", err)
+		os.Exit(1)
+	}
+
+	// Persistence is optional — empty path = in-memory ephemeral.
+	var persistor *pathway.Persistor
+	if cfg.Pathwayd.PersistPath != "" {
+		persistor, err = pathway.NewPersistor(cfg.Pathwayd.PersistPath)
+		if err != nil {
+			slog.Error("pathway persistor", "err", err)
+			os.Exit(1)
+		}
+	}
+
+	store := pathway.NewStore(persistor)
+	if persistor != nil {
+		n, err := store.Load()
+		if err != nil {
+			slog.Warn("pathway load", "err", err, "loaded", n)
+		} else {
+			slog.Info("pathway loaded", "events", n, "path", cfg.Pathwayd.PersistPath)
+		}
+	}
+
+	h := &handlers{store: store}
+
+	if err := shared.Run("pathwayd", cfg.Pathwayd.Bind, h.register, cfg.Auth); err != nil {
+		slog.Error("server", "err", err)
+		os.Exit(1)
+	}
+}
+
+type handlers struct {
+	store *pathway.Store
+}
+
+func (h *handlers) register(r chi.Router) {
+	r.Post("/pathway/add", h.handleAdd)
+	r.Post("/pathway/add_idempotent", h.handleAddIdempotent)
+	r.Post("/pathway/update", h.handleUpdate)
+	r.Post("/pathway/revise", h.handleRevise)
+	r.Post("/pathway/retire", h.handleRetire)
+	r.Get("/pathway/get/{uid}", h.handleGet)
+	r.Get("/pathway/history/{uid}", h.handleHistory)
+	r.Post("/pathway/search", h.handleSearch)
+	r.Get("/pathway/stats", h.handleStats)
+}
+
+// ── request shapes ───────────────────────────────────────────────
+
+type addRequest struct {
+	Content json.RawMessage `json:"content"`
+	Tags    []string        `json:"tags,omitempty"`
+}
+
+type addIdempotentRequest struct {
+	UID     string          `json:"uid"`
+	Content json.RawMessage `json:"content"`
+	Tags    []string        `json:"tags,omitempty"`
+}
+
+type updateRequest struct {
+	UID     string          `json:"uid"`
+	Content json.RawMessage `json:"content"`
+}
+
+type reviseRequest struct {
+	PredecessorUID string          `json:"predecessor_uid"`
+	Content        json.RawMessage `json:"content"`
+	Tags           []string        `json:"tags,omitempty"`
+}
+
+type retireRequest struct {
+	UID string `json:"uid"`
+}
+
+type searchRequest struct {
+	Tag             string `json:"tag,omitempty"`
+	ContentContains string `json:"content_contains,omitempty"`
+	CreatedAfterNs  int64  `json:"created_after_ns,omitempty"`
+	CreatedBeforeNs int64  `json:"created_before_ns,omitempty"`
+	IncludeRetired  bool   `json:"include_retired,omitempty"`
+}
+
+// ── handlers ────────────────────────────────────────────────────
+
+func (h *handlers) handleAdd(w http.ResponseWriter, r *http.Request) {
+	var req addRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	tr, err := h.store.Add(req.Content, req.Tags...)
+	if writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusCreated, tr)
+}
+
+func (h *handlers) handleAddIdempotent(w http.ResponseWriter, r *http.Request) {
+	var req addIdempotentRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	tr, err := h.store.AddIdempotent(req.UID, req.Content, req.Tags...)
+	if writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusOK, tr)
+}
+
+func (h *handlers) handleUpdate(w http.ResponseWriter, r *http.Request) {
+	var req updateRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if err := h.store.Update(req.UID, req.Content); writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{"status": "updated"})
+}
+
+func (h *handlers) handleRevise(w http.ResponseWriter, r *http.Request) {
+	var req reviseRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	tr, err := h.store.Revise(req.PredecessorUID, req.Content, req.Tags...)
+	if writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusCreated, tr)
+}
+
+func (h *handlers) handleRetire(w http.ResponseWriter, r *http.Request) {
+	var req retireRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if err := h.store.Retire(req.UID); writeStoreError(w, err) {
+		return
+	}
+	w.WriteHeader(http.StatusNoContent)
+}
+
+func (h *handlers) handleGet(w http.ResponseWriter, r *http.Request) {
+	uid := chi.URLParam(r, "uid")
+	tr, err := h.store.Get(uid)
+	if writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusOK, tr)
+}
+
+func (h *handlers) handleHistory(w http.ResponseWriter, r *http.Request) {
+	uid := chi.URLParam(r, "uid")
+	chain, err := h.store.History(uid)
+	if writeStoreError(w, err) {
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]any{
+		"chain":  chain,
+		"length": len(chain),
+	})
+}
+
+func (h *handlers) handleSearch(w http.ResponseWriter, r *http.Request) {
+	var req searchRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	results := h.store.Search(pathway.SearchFilter{
+		Tag:             req.Tag,
+		ContentContains: req.ContentContains,
+		CreatedAfterNs:  req.CreatedAfterNs,
+		CreatedBeforeNs: req.CreatedBeforeNs,
+		IncludeRetired:  req.IncludeRetired,
+	})
+	writeJSON(w, http.StatusOK, map[string]any{
+		"results": results,
+		"count":   len(results),
+	})
+}
+
+func (h *handlers) handleStats(w http.ResponseWriter, _ *http.Request) {
+	writeJSON(w, http.StatusOK, h.store.Stats())
+}
+
+// ── helpers ────────────────────────────────────────────────────
+
+func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {
+	defer r.Body.Close()
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	if err := json.NewDecoder(r.Body).Decode(v); err != nil {
+		var maxErr *http.MaxBytesError
+		if errors.As(err, &maxErr) || strings.Contains(err.Error(), "http: request body too large") {
+			http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
+			return false
+		}
+		http.Error(w, "decode body: "+err.Error(), http.StatusBadRequest)
+		return false
+	}
+	return true
+}
+
+func writeJSON(w http.ResponseWriter, code int, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(code)
+	if err := json.NewEncoder(w).Encode(v); err != nil {
+		slog.Warn("pathway write json", "err", err)
+	}
+}
+
+// writeStoreError maps internal/pathway sentinel errors to HTTP
+// status codes. Returns true if a response was written (caller
+// should return). Returns false on success (caller continues).
+func writeStoreError(w http.ResponseWriter, err error) bool {
+	if err == nil {
+		return false
+	}
+	switch {
+	case errors.Is(err, pathway.ErrNotFound):
+		http.Error(w, err.Error(), http.StatusNotFound)
+	case errors.Is(err, pathway.ErrPredecessorMissing):
+		http.Error(w, err.Error(), http.StatusNotFound)
+	case errors.Is(err, pathway.ErrEmptyUID),
+		errors.Is(err, pathway.ErrInvalidContent):
+		http.Error(w, err.Error(), http.StatusBadRequest)
+	case errors.Is(err, pathway.ErrCycle):
+		http.Error(w, err.Error(), http.StatusConflict)
+	default:
+		slog.Error("pathway store", "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+	}
+	return true
+}

cmd/pathwayd/main_test.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/pathway"
+)
+
+// newTestRouter builds the pathwayd router with an in-memory store
+// (nil persistor). Closes R-005 for pathwayd: 9 routes mounted with
+// no router-level test prior to this file.
+//
+// Returns chi.Router (not http.Handler) so chi.Walk works without a
+// type assertion that would panic if a future refactor wraps the
+// router in plain net/http middleware.
+func newTestRouter(t *testing.T) chi.Router {
+	t.Helper()
+	h := &handlers{store: pathway.NewStore(nil)}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	r := newTestRouter(t)
+	want := map[string]string{
+		"POST /pathway/add":             "",
+		"POST /pathway/add_idempotent":  "",
+		"POST /pathway/update":          "",
+		"POST /pathway/revise":          "",
+		"POST /pathway/retire":          "",
+		"GET /pathway/get/{uid}":        "",
+		"GET /pathway/history/{uid}":    "",
+		"POST /pathway/search":          "",
+		"GET /pathway/stats":            "",
+	}
+	got := map[string]bool{}
+	_ = chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		got[method+" "+route] = true
+		return nil
+	})
+	for k := range want {
+		if !got[k] {
+			t.Errorf("route not mounted: %s", k)
+		}
+	}
+}
+
+// TestAdd_RoundTrip locks the happy-path contract: POST a content blob,
+// receive a 201 with a trace, GET it back at /pathway/get/{uid}.
+// Catches drift in either the add response shape or the get path.
+func TestAdd_RoundTrip(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"content":{"hello":"world"},"tags":["test"]}`)
+	req := httptest.NewRequest("POST", "/pathway/add", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusCreated {
+		t.Fatalf("expected 201 on add, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+func TestStats_GET(t *testing.T) {
+	r := newTestRouter(t)
+	req := httptest.NewRequest("GET", "/pathway/stats", nil)
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Errorf("expected 200 on stats, got %d", w.Code)
+	}
+}
+
+// TestAddIdempotent_MissingUID locks the validation: empty UID must
+// 4xx rather than silently accepting (which would defeat the
+// idempotency contract).
+func TestAddIdempotent_MissingUID(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"content":{"x":1}}`)
+	req := httptest.NewRequest("POST", "/pathway/add_idempotent", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code/100 != 4 {
+		t.Errorf("missing uid should 4xx, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+// TestRetire_NonexistentUID locks the not-found path. The store rejects
+// retiring traces that don't exist; the handler must surface that as a
+// 4xx, not a 5xx.
+func TestRetire_NonexistentUID(t *testing.T) {
+	r := newTestRouter(t)
+	body := []byte(`{"uid":"does-not-exist"}`)
+	req := httptest.NewRequest("POST", "/pathway/retire", bytes.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code/100 != 4 {
+		t.Errorf("retire of nonexistent uid should 4xx, got %d", w.Code)
+	}
+}

cmd/queryd/main.go

@@ -99,7 +99,7 @@ func main() {
 
 	h := &handlers{db: db}
 
-	if err := shared.Run("queryd", cfg.Queryd.Bind, h.register); err != nil {
+	if err := shared.Run("queryd", cfg.Queryd.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}

cmd/queryd/main_test.go

@@ -0,0 +1,154 @@
+package main
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+)
+
+// Closes R-005 for queryd: cmd-level tests for the /sql handler's
+// pre-DB paths (decode, body cap, empty SQL). The actual SQL execution
+// path needs DuckDB so it lives in the smoke chain + proof harness.
+//
+// We construct handlers with a nil *sql.DB — the tests only exercise
+// paths that return early before db.QueryContext. Tests that would
+// reach the db are covered by GOLAKE-040 in the proof harness.
+
+func mountedRouter() chi.Router {
+	h := &handlers{db: nil}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	r := mountedRouter()
+	found := false
+	chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		if method == "POST" && route == "/sql" {
+			found = true
+		}
+		return nil
+	})
+	if !found {
+		t.Error("POST /sql not mounted")
+	}
+}
+
+func TestHandleSQL_BodyTooLarge(t *testing.T) {
+	// 4xx range — see embedd's TestHandleEmbed_BodyTooLarge for the
+	// 413-vs-400 detail. The contract is "client error, fails loud."
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	big := bytes.Repeat([]byte("x"), maxSQLBodyBytes+1024)
+	resp, err := http.Post(srv.URL+"/sql", "application/json", bytes.NewReader(big))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 400 || resp.StatusCode >= 500 {
+		t.Errorf("expected 4xx on oversize, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleSQL_MalformedJSON_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/sql", "application/json", strings.NewReader("not json"))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on malformed, got %d", resp.StatusCode)
+	}
+}
+
+// TestHandleSQL_WrongFieldName_400 locks the JSON tag on sqlRequest.SQL
+// against drift. The 2026-04-30 playbook_lift harness sent {"q": "..."}
+// — the Go decoder ignores unknown fields by default, so req.SQL stays
+// empty and the empty-check fires with "sql is empty". If anyone renames
+// the JSON tag, callers POSTing the new (wrong) shape would hit this
+// same path; this test makes the contract explicit so the failure mode
+// is documented rather than discovered during a reality run.
+func TestHandleSQL_WrongFieldName_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	cases := []string{
+		`{"q":"SELECT 1"}`,        // the actual 2026-04-30 harness shape
+		`{"query":"SELECT 1"}`,    // matrixd-style drift in the other direction
+		`{"statement":"SELECT 1"}`,
+	}
+	for _, body := range cases {
+		t.Run(body, func(t *testing.T) {
+			resp, err := http.Post(srv.URL+"/sql", "application/json", strings.NewReader(body))
+			if err != nil {
+				t.Fatalf("POST: %v", err)
+			}
+			defer resp.Body.Close()
+			if resp.StatusCode != http.StatusBadRequest {
+				t.Errorf("expected 400 on wrong field name, got %d", resp.StatusCode)
+			}
+			rb, _ := io.ReadAll(resp.Body)
+			if !strings.Contains(string(rb), "sql is empty") {
+				t.Errorf("expected 'sql is empty' to anchor the contract, got %q", string(rb))
+			}
+		})
+	}
+}
+
+func TestHandleSQL_EmptySQL_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	cases := []string{
+		`{"sql":""}`,
+		`{"sql":"   "}`,
+		`{"sql":"\n\t  \n"}`,
+	}
+	for _, body := range cases {
+		t.Run(body, func(t *testing.T) {
+			resp, err := http.Post(srv.URL+"/sql", "application/json", strings.NewReader(body))
+			if err != nil {
+				t.Fatalf("POST: %v", err)
+			}
+			defer resp.Body.Close()
+			if resp.StatusCode != http.StatusBadRequest {
+				t.Errorf("expected 400 on empty/whitespace SQL, got %d", resp.StatusCode)
+			}
+		})
+	}
+}
+
+func TestMaxSQLBodyBytes_Reasonable(t *testing.T) {
+	// SQL strings shouldn't be huge — 64 KiB is generous for queryd
+	// (DuckDB statements above 64 KiB are pathological). Locking the
+	// constant prevents an accidental refactor from blowing this open.
+	if maxSQLBodyBytes < 16<<10 {
+		t.Errorf("maxSQLBodyBytes=%d below sane SQL minimum (16 KiB)", maxSQLBodyBytes)
+	}
+	if maxSQLBodyBytes > 1<<20 {
+		t.Errorf("maxSQLBodyBytes=%d above sane SQL maximum (1 MiB)", maxSQLBodyBytes)
+	}
+}
+
+func TestPrimaryBucket_Constant(t *testing.T) {
+	// Locks the logical bucket name — secrets provider lookup keys
+	// against this. Refactor that flips this would silently fail
+	// secret resolution for queryd at startup.
+	if primaryBucket != "primary" {
+		t.Errorf("primaryBucket = %q, want %q", primaryBucket, "primary")
+	}
+}

cmd/replay/main.go

@@ -0,0 +1,87 @@
+// replay — Go-side distillation replay runner. Closes audit-FULL
+// phase 7 live invocation on the Go side. Mirrors
+// scripts/distillation/replay.ts; both runtimes append to the same
+// `data/_kb/replay_runs.jsonl` shape (schema=replay_run.v1).
+//
+// Usage:
+//
+//	replay -task "rebuild evidence index"
+//	replay -task "..."  -allow-escalation
+//	replay -task "..."  -no-retrieval                # baseline mode
+//	replay -task "..."  -dry-run                     # synthetic, no LLM
+//	replay -task "..."  -root /home/profit/lakehouse # custom repo root
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"os"
+	"strings"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/replay"
+)
+
+func main() {
+	task := flag.String("task", "", "input task to replay")
+	localOnly := flag.Bool("local-only", false, "never escalate; record validation result only")
+	allowEscalation := flag.Bool("allow-escalation", false, "fall back to the bigger model when local validation fails")
+	noRetrieval := flag.Bool("no-retrieval", false, "baseline mode: skip retrieval bundle (still logs)")
+	dryRun := flag.Bool("dry-run", false, "synthesize a deterministic response — no LLM call")
+	root := flag.String("root", replay.DefaultRoot(), "lakehouse repo root (defaults to $LH_DISTILL_ROOT or cwd)")
+	gateway := flag.String("gateway", "", "override gateway URL (default: $LH_GATEWAY_URL or http://localhost:3110)")
+	localModel := flag.String("local-model", "", "override local model name")
+	escalationModel := flag.String("escalation-model", "", "override escalation model name")
+	flag.Parse()
+
+	if *task == "" {
+		fmt.Fprintln(os.Stderr, `usage: replay -task "<input>" [-local-only] [-allow-escalation] [-no-retrieval] [-dry-run]`)
+		os.Exit(2)
+	}
+
+	res, err := replay.Replay(context.Background(), replay.ReplayRequest{
+		Task:            *task,
+		LocalOnly:       *localOnly,
+		AllowEscalation: *allowEscalation,
+		NoRetrieval:     *noRetrieval,
+		DryRun:          *dryRun,
+		GatewayURL:      *gateway,
+		LocalModel:      *localModel,
+		EscalationModel: *escalationModel,
+	}, *root)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "replay: %v\n", err)
+		os.Exit(1)
+	}
+
+	fmt.Printf("[replay] run_id=%s\n", res.RecordedRunID)
+	if res.ContextBundle == nil {
+		fmt.Println("[replay] retrieval: DISABLED")
+	} else {
+		fmt.Printf("[replay] retrieval: %d playbooks\n", len(res.ContextBundle.RetrievedPlaybooks))
+	}
+	fmt.Printf("[replay] escalation_path: %s\n", strings.Join(res.EscalationPath, " → "))
+	fmt.Printf("[replay] model_used: %s · %dms\n", res.ModelUsed, res.DurationMs)
+	verdict := "PASS"
+	if !res.ValidationResult.Passed {
+		verdict = "FAIL"
+	}
+	suffix := ""
+	if len(res.ValidationResult.Reasons) > 0 {
+		suffix = " (" + strings.Join(res.ValidationResult.Reasons, "; ") + ")"
+	}
+	fmt.Printf("[replay] validation: %s%s\n", verdict, suffix)
+	fmt.Println()
+	fmt.Println("─── response ───")
+	body := res.ModelResponse
+	if len(body) > 1500 {
+		fmt.Println(body[:1500])
+		fmt.Printf("... [%d more chars]\n", len(body)-1500)
+	} else {
+		fmt.Println(body)
+	}
+
+	if !res.ValidationResult.Passed {
+		os.Exit(1)
+	}
+}

cmd/storaged/main.go

@@ -24,12 +24,41 @@ import (
 )
 
 const (
-	maxPutBytes      = 256 << 20 // 256 MiB per Qwen Q1 fix
-	maxConcurrentPut = 4         // 4-slot semaphore on in-flight PUTs
-	retryAfterSecs   = "5"       // Retry-After header on 503
+	// Default per-PUT body cap. Most callers (ingest parquets, catalog
+	// manifests) live well under this. The cap is a safety gate, not a
+	// memory bottleneck — manager.Uploader streams; this just refuses
+	// reads past the limit so a runaway client can't drain the daemon.
+	maxPutBytesDefault = 256 << 20 // 256 MiB
+
+	// Vector-persistence prefix gets a much larger cap because vectord
+	// persists single-file LHV1 indexes that exceed 256 MiB above
+	// ~150K vectors at d=768 (the 500K staffing test's documented gap).
+	// 4 GiB covers >700K vectors at d=768 with HNSW graph overhead and
+	// keeps the simple-Put torn-write guarantee from G1P intact (memory
+	// project_golang_lakehouse.md: "Single Put eliminates the torn-write
+	// class").
+	maxPutBytesVectors = 4 << 30 // 4 GiB
+
+	// Vector-persistence prefix matched against incoming PUT keys. Keep
+	// this in sync with internal/vectord/persistor.go's lhv1KeyFor.
+	vectorsPrefix = "_vectors/"
+
+	maxConcurrentPut = 4   // 4-slot semaphore on in-flight PUTs
+	retryAfterSecs   = "5" // Retry-After header on 503
 	primaryBucket    = "primary"
 )
 
+// maxPutBytesFor returns the body-cap to apply to a PUT for the given
+// key. Vectord LHV1 persistence (under "_vectors/") gets the larger
+// cap; everything else stays at the default. Function-level so a
+// future operator-driven cap (env, config) can drop in cleanly.
+func maxPutBytesFor(key string) int64 {
+	if strings.HasPrefix(key, vectorsPrefix) {
+		return maxPutBytesVectors
+	}
+	return maxPutBytesDefault
+}
+
 func main() {
 	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
 	secretsPath := flag.String("secrets", "/etc/lakehouse/secrets-go.toml", "path to secrets TOML (Go-side; Rust uses /etc/lakehouse/secrets.toml)")
@@ -49,7 +78,7 @@ func main() {
 
 	h := newHandlers(registry)
 
-	if err := shared.Run("storaged", cfg.Storaged.Bind, h.register); err != nil {
+	if err := shared.Run("storaged", cfg.Storaged.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}
@@ -181,15 +210,14 @@ func (h *handlers) handlePut(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	// Up-front Content-Length cap. Per Opus C3 review: the
-	// manager.Uploader's multipart path runs body reads in goroutines
-	// and wraps errors in its own types, so *http.MaxBytesError can be
-	// buried by the time it reaches us — meaning bodies just over the
-	// 5 MiB multipart threshold could surface as 500 instead of 413.
-	// Catching Content-Length up front returns 413 deterministically
-	// when the client honestly declares size; MaxBytesReader + the
-	// string-match fallback below cover chunked / lying-CL cases.
-	if r.ContentLength > maxPutBytes {
+	// Per-key body cap. Vectord LHV1 persistence under "_vectors/"
+	// gets a 4 GiB cap; everything else keeps 256 MiB. Up-front
+	// Content-Length check first per Opus C3 review (manager.Uploader's
+	// multipart path can bury *http.MaxBytesError in its own error
+	// types); MaxBytesReader + string-match fallback below cover
+	// chunked / lying-CL cases.
+	cap := maxPutBytesFor(key)
+	if r.ContentLength > cap {
 		w.Header().Set("Retry-After", retryAfterSecs)
 		http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
 		return
@@ -208,11 +236,11 @@ func (h *handlers) handlePut(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	// 256 MiB per-request body cap. Reads beyond this surface as
-	// *http.MaxBytesError; for chunked-encoding bodies that's the only
-	// signal we get. Defer LIFO order: r.Body.Close fires before
-	// <-h.putSem, so the body is fully drained before the slot frees.
-	r.Body = http.MaxBytesReader(w, r.Body, maxPutBytes)
+	// Per-key body cap as MaxBytesReader so chunked-encoding bodies
+	// also fail-loud at the limit. Defer LIFO order: r.Body.Close
+	// fires before <-h.putSem, so the body is fully drained before
+	// the slot frees.
+	r.Body = http.MaxBytesReader(w, r.Body, cap)
 	defer r.Body.Close()
 
 	bucket, err := h.reg.Resolve(primaryBucket)

cmd/storaged/main_test.go

@@ -1,6 +1,10 @@
 package main
 
-import "testing"
+import (
+	"testing"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/vectord"
+)
 
 func TestValidateKey(t *testing.T) {
 	cases := []struct {
@@ -36,3 +40,57 @@ func TestValidateKey(t *testing.T) {
 		})
 	}
 }
+
+func TestMaxPutBytesFor(t *testing.T) {
+	// Vectord LHV1 persistence gets the larger cap so single-file
+	// indexes above ~150K vectors at d=768 (the 500K staffing test
+	// gap) can survive a Save without 413. Default cap stays at
+	// 256 MiB for everything else (parquets, manifests, etc.).
+	cases := []struct {
+		name string
+		key  string
+		want int64
+	}{
+		{"vectord LHV1 file", "_vectors/workers_500k.lhv1", maxPutBytesVectors},
+		{"vectord nested", "_vectors/some/nested/index.lhv1", maxPutBytesVectors},
+		{"parquet under datasets/", "datasets/workers/abc.parquet", maxPutBytesDefault},
+		{"catalog manifest", "_catalog/manifests/foo.bin", maxPutBytesDefault},
+		{"plain key", "x.txt", maxPutBytesDefault},
+		{"empty key", "", maxPutBytesDefault},
+		{"key with vectors-ish substring (not prefix)", "datasets/_vectors/x", maxPutBytesDefault},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := maxPutBytesFor(tc.key)
+			if got != tc.want {
+				t.Errorf("maxPutBytesFor(%q) = %d, want %d", tc.key, got, tc.want)
+			}
+		})
+	}
+}
+
+// TestVectorPrefixSyncWithVectord locks the prefix value to vectord's
+// VectorPrefix constant. If a future refactor renames either side,
+// this test surfaces the drift before vectord saves silently bypass
+// the larger cap.
+func TestVectorPrefixSyncWithVectord(t *testing.T) {
+	if vectorsPrefix != vectord.VectorPrefix {
+		t.Errorf("storaged vectorsPrefix=%q out of sync with vectord.VectorPrefix=%q",
+			vectorsPrefix, vectord.VectorPrefix)
+	}
+}
+
+// TestVectorCapAccommodates500KStaffingTest reads the documented gap
+// (memory project_golang_lakehouse.md): "storaged 256 MiB PUT cap
+// blocks single-file LHV1 persistence above ~150K vectors at d=768."
+// 500K vectors at d=768 with HNSW graph overhead is approximately
+// 4.5 GiB resident, ~700 MiB on disk after compression.
+// 4 GiB cap covers more than the documented production workload.
+func TestVectorCapAccommodates500KStaffingTest(t *testing.T) {
+	const fiveHundredKVectorsAtD768Disk = 700 << 20 // ~700 MiB conservative estimate
+	if maxPutBytesVectors < fiveHundredKVectorsAtD768Disk {
+		t.Errorf("maxPutBytesVectors=%d (%.0f MiB) below documented production "+
+			"workload of ~700 MiB for 500K vectors at d=768",
+			maxPutBytesVectors, float64(maxPutBytesVectors)/(1<<20))
+	}
+}

cmd/validatord/main.go

@@ -0,0 +1,485 @@
+// validatord is the staffing-validator service daemon. Hosts:
+//
+//	POST /validate   — dispatch a single artifact to FillValidator,
+//	                   EmailValidator, or PlaybookValidator
+//	POST /iterate    — generate→validate→correct loop (Phase 43 PRD).
+//	                   Calls chatd for the LLM hop and runs the
+//	                   validator in-process for the gate.
+//	GET  /health     — readiness (always 200; roster status reported
+//	                   in /validate responses)
+//
+// Per docs/SPEC.md and architecture_comparison.md "Go primary path":
+// this closes the last bounded item — the now-Go-side validators get
+// a network surface so any caller (TS code path, other daemons, agents)
+// can validate artifacts via gateway /v1/validate or /v1/iterate.
+//
+// The roster (worker existence + city/state/role/blacklist) loads
+// from a JSONL file at startup. Empty path = no roster, worker-existence
+// checks fail Consistency. Production points this at a roster that's
+// regenerated from workers_500k.parquet on a schedule.
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/langfuse"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/validator"
+)
+
+const maxRequestBytes = 4 << 20 // 4 MiB
+
+func main() {
+	configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
+	flag.Parse()
+
+	cfg, err := shared.LoadConfig(*configPath)
+	if err != nil {
+		slog.Error("config", "err", err)
+		os.Exit(1)
+	}
+
+	lookup, err := validator.LoadJSONLRoster(cfg.Validatord.RosterPath)
+	if err != nil {
+		slog.Error("roster load", "path", cfg.Validatord.RosterPath, "err", err)
+		os.Exit(1)
+	}
+	slog.Info("validatord roster",
+		"path", cfg.Validatord.RosterPath,
+		"records", lookup.Len(),
+	)
+
+	chatTimeout := time.Duration(cfg.Validatord.ChatTimeoutSecs) * time.Second
+	if chatTimeout <= 0 {
+		chatTimeout = 240 * time.Second
+	}
+
+	h := &handlers{
+		lookup:     lookup,
+		chatdURL:   cfg.Validatord.ChatdURL,
+		chatClient: &http.Client{Timeout: chatTimeout},
+		iterCfg: validator.IterateConfig{
+			DefaultMaxIterations: cfg.Validatord.DefaultMaxIterations,
+			DefaultMaxTokens:     cfg.Validatord.DefaultMaxTokens,
+		},
+		// Same env loader the middleware uses — getting our own
+		// reference lets us emit child spans of the per-request trace.
+		// nil when Langfuse env isn't set; the iterate handler skips
+		// span emission gracefully in that case.
+		lf:         shared.LoadLangfuseFromEnv(),
+		sessionLog: validator.NewSessionLogger(cfg.Validatord.SessionLogPath),
+	}
+	if h.sessionLog != nil {
+		slog.Info("validatord session log", "path", cfg.Validatord.SessionLogPath)
+	}
+
+	if err := shared.Run("validatord", cfg.Validatord.Bind, h.register, cfg.Auth); err != nil {
+		slog.Error("server", "err", err)
+		os.Exit(1)
+	}
+}
+
+type handlers struct {
+	lookup     validator.WorkerLookup
+	chatdURL   string
+	chatClient *http.Client
+	iterCfg    validator.IterateConfig
+	// lf is the Langfuse client (nil when unconfigured — best-effort
+	// posture matching the rest of the stack). Per-attempt iteration
+	// spans get emitted via lf when set. The HTTP middleware that
+	// creates the parent trace lives in shared.Run; this client lets
+	// validatord's handler emit application-level child spans.
+	lf *langfuse.Client
+	// sessionLog appends one SessionRecord per /v1/iterate session to
+	// `coordinator_sessions.jsonl` for offline DuckDB analysis. nil
+	// when the session log is unconfigured (best-effort posture).
+	sessionLog *validator.SessionLogger
+}
+
+func (h *handlers) register(r chi.Router) {
+	r.Post("/validate", h.handleValidate)
+	r.Post("/iterate", h.handleIterate)
+}
+
+// validateRequest is the request body for POST /validate. Mirrors
+// Rust's ValidateRequest in `crates/gateway/src/v1/validate.rs`.
+type validateRequest struct {
+	Kind     string         `json:"kind"`     // "fill" | "email" | "playbook"
+	Artifact map[string]any `json:"artifact"`
+	Context  map[string]any `json:"context,omitempty"`
+}
+
+func (h *handlers) handleValidate(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	defer r.Body.Close()
+
+	var req validateRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest)
+		return
+	}
+	if req.Kind == "" {
+		http.Error(w, "kind is required", http.StatusBadRequest)
+		return
+	}
+	if req.Artifact == nil {
+		http.Error(w, "artifact is required", http.StatusBadRequest)
+		return
+	}
+
+	report, vErr, kindErr := h.runValidator(req.Kind, req.Artifact, req.Context)
+	switch {
+	case kindErr != nil:
+		http.Error(w, kindErr.Error(), http.StatusBadRequest)
+	case vErr != nil:
+		writeJSON(w, http.StatusUnprocessableEntity, vErr)
+	default:
+		writeJSON(w, http.StatusOK, report)
+	}
+}
+
+// runValidator dispatches by kind. Returns (Report, ValidationError, kindErr).
+// kindErr is non-nil only for unknown kind strings (400).
+func (h *handlers) runValidator(kind string, artifact, ctx map[string]any) (*validator.Report, *validator.ValidationError, error) {
+	merged := mergeContext(artifact, ctx)
+	a, kindErr := buildArtifact(kind, merged)
+	if kindErr != nil {
+		return nil, nil, kindErr
+	}
+	v, vErr := pickValidator(kind, h.lookup)
+	if vErr != nil {
+		return nil, nil, vErr
+	}
+	report, err := v.Validate(a)
+	if err != nil {
+		var ve *validator.ValidationError
+		if errors.As(err, &ve) {
+			return nil, ve, nil
+		}
+		// Validators only ever return ValidationError; an "any other
+		// error" path means the validator violated its own contract.
+		// Surface as 500 rather than silently coercing.
+		return nil, &validator.ValidationError{
+			Kind:   validator.ErrSchema,
+			Reason: "internal validator error: " + err.Error(),
+		}, nil
+	}
+	return &report, nil, nil
+}
+
+// buildArtifact maps the kind string to the right Artifact union arm.
+// Unknown kinds return a 400-friendly error.
+func buildArtifact(kind string, body map[string]any) (validator.Artifact, error) {
+	switch kind {
+	case "fill":
+		return validator.Artifact{FillProposal: body}, nil
+	case "email":
+		return validator.Artifact{EmailDraft: body}, nil
+	case "playbook":
+		return validator.Artifact{Playbook: body}, nil
+	default:
+		return validator.Artifact{}, fmt.Errorf("unknown kind %q — expected fill | email | playbook", kind)
+	}
+}
+
+func pickValidator(kind string, lookup validator.WorkerLookup) (validator.Validator, error) {
+	switch kind {
+	case "fill":
+		return validator.NewFillValidator(lookup), nil
+	case "email":
+		return validator.NewEmailValidator(lookup), nil
+	case "playbook":
+		return validator.PlaybookValidator{}, nil
+	default:
+		return nil, fmt.Errorf("unknown kind %q", kind)
+	}
+}
+
+// mergeContext folds `context` into `artifact._context` so validators
+// pull contract metadata uniformly. Caller-supplied artifact._context
+// wins on key collision (caller knows their own contract).
+func mergeContext(artifact, ctx map[string]any) map[string]any {
+	if ctx == nil {
+		return artifact
+	}
+	out := make(map[string]any, len(artifact)+1)
+	for k, v := range artifact {
+		out[k] = v
+	}
+	existing, _ := out["_context"].(map[string]any)
+	merged := make(map[string]any, len(ctx)+len(existing))
+	for k, v := range ctx {
+		merged[k] = v
+	}
+	for k, v := range existing {
+		merged[k] = v // existing wins
+	}
+	out["_context"] = merged
+	return out
+}
+
+func (h *handlers) handleIterate(w http.ResponseWriter, r *http.Request) {
+	r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
+	defer r.Body.Close()
+
+	var req validator.IterateRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest)
+		return
+	}
+	if req.Kind == "" || req.Prompt == "" || req.Provider == "" || req.Model == "" {
+		http.Error(w, "kind, prompt, provider, and model are required", http.StatusBadRequest)
+		return
+	}
+
+	// Resolve the parent trace id, in priority order:
+	//   1. req.TraceID (caller put it in the JSON body)
+	//   2. X-Lakehouse-Trace-Id header (read DIRECTLY here so the path
+	//      works even when Langfuse is off and the middleware is a
+	//      passthrough — matches Rust's iterate handler shape)
+	//   3. context value set by the langfuse middleware (when Langfuse
+	//      is configured)
+	//   4. fallback to a locally-minted time-ordered hex id
+	//
+	// Surfaced 2026-05-02 cross-runtime test: Go was minting fallback
+	// even when caller forwarded the header, because middleware
+	// passthrough mode never read it. Reading the header here closes
+	// the gap and makes Go behavior match Rust regardless of
+	// Langfuse configuration.
+	if req.TraceID == "" {
+		req.TraceID = r.Header.Get(shared.TraceIDHeader)
+	}
+	if req.TraceID == "" {
+		req.TraceID = shared.TraceIDFromCtx(r.Context())
+	}
+	if req.TraceID == "" {
+		req.TraceID = newFallbackTraceID()
+	}
+
+	chat := h.chatCaller()
+	validate := func(kind string, artifact map[string]any) (validator.Report, error) {
+		report, vErr, kindErr := h.runValidator(kind, artifact, req.Context)
+		if kindErr != nil {
+			return validator.Report{}, &validator.ValidationError{
+				Kind:   validator.ErrSchema,
+				Reason: kindErr.Error(),
+			}
+		}
+		if vErr != nil {
+			return validator.Report{}, vErr
+		}
+		return *report, nil
+	}
+
+	// Wire per-attempt span emission when Langfuse is configured —
+	// each chat→validate cycle becomes a child span of the iterate
+	// trace, with prompt/raw/verdict captured for inspection.
+	cfg := h.iterCfg
+	if h.lf != nil {
+		cfg.Tracer = h.iterTracer(r.Context())
+	}
+
+	t0 := time.Now()
+	resp, fail, err := validator.Iterate(r.Context(), req, cfg, chat, validate)
+	durationMs := time.Since(t0).Milliseconds()
+
+	// Emit one session row regardless of success/failure/infra-error.
+	// Best-effort: a failing session log never blocks the response.
+	rosterCheck := h.rosterCheckFor(req.Kind)
+	rec := validator.BuildSessionRecord(req, resp, fail, err, rosterCheck, durationMs)
+	h.sessionLog.Append(rec)
+
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadGateway)
+		return
+	}
+	if fail != nil {
+		writeJSON(w, http.StatusUnprocessableEntity, fail)
+		return
+	}
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// rosterCheckFor returns a closure that verifies an accepted artifact's
+// candidate IDs all exist in the roster. Used by BuildSessionRecord to
+// stamp `grounded_in_roster` on each session row — answers the "did
+// the model produce real workers, not phantoms?" question at offline-
+// query time.
+//
+// Only the `fill` kind has worker IDs to verify. Other kinds return
+// nil (omits the field from the session record).
+//
+// Note: FillValidator already enforces this constraint on every
+// successful iteration (phantom IDs raise ValidationError::Consistency).
+// Stamping the bool here surfaces the verified-grounded property as
+// queryable data, not relying on validator semantics being recalled
+// during analysis. A future scrum may want to query "show me sessions
+// where the model guessed real worker IDs without ever needing a
+// retry" — that's a `iterations=1 AND grounded_in_roster=true` query.
+func (h *handlers) rosterCheckFor(kind string) func(map[string]any) *bool {
+	if kind != "fill" {
+		return nil
+	}
+	return func(artifact map[string]any) *bool {
+		fills, ok := artifact["fills"].([]any)
+		if !ok {
+			f := false
+			return &f
+		}
+		for _, raw := range fills {
+			fill, ok := raw.(map[string]any)
+			if !ok {
+				f := false
+				return &f
+			}
+			id, _ := fill["candidate_id"].(string)
+			if id == "" {
+				f := false
+				return &f
+			}
+			if _, found := h.lookup.Find(id); !found {
+				f := false
+				return &f
+			}
+		}
+		t := true
+		return &t
+	}
+}
+
+// chatCaller wires the iteration loop to chatd via HTTP. Builds the
+// chat.Request shape, posts to ${chatdURL}/chat, returns the content
+// string (no choices wrapper — chatd's response is already flat).
+//
+// When traceID is non-empty, forwards it as the `X-Lakehouse-Trace-Id`
+// header so chatd's middleware attaches its trace to the same
+// Langfuse parent. Closes the multi-call observability gap: a
+// /v1/iterate session shows in Langfuse as one trace with N child
+// chat spans, instead of N+1 disconnected traces.
+func (h *handlers) chatCaller() validator.ChatCaller {
+	return func(ctx context.Context, system, user, _, model string, temp *float64, maxTokens int, traceID string) (string, error) {
+		messages := make([]map[string]string, 0, 2)
+		if system != "" {
+			messages = append(messages, map[string]string{"role": "system", "content": system})
+		}
+		messages = append(messages, map[string]string{"role": "user", "content": user})
+		body := map[string]any{
+			"model":      model,
+			"messages":   messages,
+			"max_tokens": maxTokens,
+		}
+		if temp != nil {
+			body["temperature"] = *temp
+		}
+		buf, err := json.Marshal(body)
+		if err != nil {
+			return "", fmt.Errorf("marshal chat req: %w", err)
+		}
+		req, err := http.NewRequestWithContext(ctx, "POST", h.chatdURL+"/chat", bytes.NewReader(buf))
+		if err != nil {
+			return "", fmt.Errorf("build chat req: %w", err)
+		}
+		req.Header.Set("Content-Type", "application/json")
+		if traceID != "" {
+			req.Header.Set(shared.TraceIDHeader, traceID)
+		}
+		resp, err := h.chatClient.Do(req)
+		if err != nil {
+			return "", fmt.Errorf("chat hop: %w", err)
+		}
+		defer resp.Body.Close()
+		raw, _ := io.ReadAll(resp.Body)
+		if resp.StatusCode >= 400 {
+			return "", fmt.Errorf("chat %d: %s", resp.StatusCode, trim(string(raw), 300))
+		}
+		var parsed struct {
+			Content string `json:"content"`
+		}
+		if err := json.Unmarshal(raw, &parsed); err != nil {
+			return "", fmt.Errorf("parse chat resp: %w", err)
+		}
+		return parsed.Content, nil
+	}
+}
+
+// iterTracer adapts validator.Tracer → langfuse.Client.Span. Each
+// iteration attempt becomes one Langfuse span on the parent trace
+// with input={prompt, model, provider}, output={raw, verdict}.
+// Operators reading Langfuse can scroll the iterate session and see
+// the full retry chain — which prompt produced which raw, which
+// validator verdict landed, which retry the model recovered on.
+//
+// Returns the span id so it lands in IterateAttempt.SpanID — the
+// /v1/iterate response carries that id back to the caller for log
+// correlation.
+//
+// Errors are silent (best-effort posture per package langfuse). A
+// dropped span never blocks the iterate loop.
+func (h *handlers) iterTracer(ctx context.Context) validator.Tracer {
+	return func(s validator.AttemptSpan) string {
+		level := "DEFAULT"
+		if s.Verdict.Kind != "accepted" {
+			level = "WARNING"
+		}
+		return h.lf.Span(ctx, langfuse.SpanInput{
+			TraceID: s.TraceID,
+			Name:    fmt.Sprintf("iterate.attempt[%d]", s.Iteration),
+			Input: map[string]any{
+				"iteration": s.Iteration,
+				"model":     s.Model,
+				"provider":  s.Provider,
+				"prompt":    trim(s.Prompt, 4000),
+			},
+			Output: map[string]any{
+				"verdict": s.Verdict.Kind,
+				"error":   s.Verdict.Error,
+				"raw":     trim(s.Raw, 4000),
+			},
+			Metadata: map[string]any{
+				"verdict_error": s.Verdict.Error,
+			},
+			StartTime: s.StartTime,
+			EndTime:   s.EndTime,
+			Level:     level,
+		})
+	}
+}
+
+// newFallbackTraceID generates a time-ordered hex id used when no
+// upstream trace id was forwarded AND the langfuse middleware didn't
+// mint one (Langfuse unconfigured). Same shape Langfuse accepts so
+// future Langfuse-enabled deployments don't break correlation. Avoids
+// pulling in a uuid crate dep; nanosecond precision + a randomness
+// suffix is unique enough at the rates iterate runs.
+func newFallbackTraceID() string {
+	ns := time.Now().UTC().UnixNano()
+	rand := uint32(time.Now().UnixNano() % (1 << 32))
+	return fmt.Sprintf("%016x-%08x", ns, rand)
+}
+
+func writeJSON(w http.ResponseWriter, status int, body any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	if err := json.NewEncoder(w).Encode(body); err != nil {
+		slog.Error("encode", "err", err)
+	}
+}
+
+func trim(s string, n int) string {
+	if len(s) <= n {
+		return s
+	}
+	return s[:n]
+}

cmd/validatord/main_test.go

@@ -0,0 +1,427 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/validator"
+)
+
+// readSessionLog parses the JSONL session log into row maps. Used by
+// the iterate-handler tests to verify per-session entries.
+func readSessionLog(t *testing.T, path string) []map[string]any {
+	t.Helper()
+	f, err := os.Open(path)
+	if err != nil {
+		t.Fatalf("open session log: %v", err)
+	}
+	defer f.Close()
+	var out []map[string]any
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 1<<16), 1<<24)
+	for sc.Scan() {
+		line := sc.Bytes()
+		if len(line) == 0 {
+			continue
+		}
+		var row map[string]any
+		if err := json.Unmarshal(line, &row); err != nil {
+			t.Fatalf("parse session log line: %v", err)
+		}
+		out = append(out, row)
+	}
+	return out
+}
+
+// newTestRouter builds the validatord router with an explicit lookup
+// + a fake chatd URL. Tests that exercise /iterate need a live mock
+// chatd (constructed inline per-test).
+func newTestRouter(lookup validator.WorkerLookup, chatdURL string) http.Handler {
+	h := newTestHandlers(lookup, chatdURL, nil)
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+// newTestHandlers exposes the underlying handlers struct so tests can
+// inspect post-call state (e.g. session log contents). sessionLog is
+// optional — pass nil to disable the session log.
+func newTestHandlers(lookup validator.WorkerLookup, chatdURL string, sessionLog *validator.SessionLogger) *handlers {
+	return &handlers{
+		lookup:     lookup,
+		chatdURL:   chatdURL,
+		chatClient: &http.Client{Timeout: 5 * time.Second},
+		iterCfg: validator.IterateConfig{
+			DefaultMaxIterations: 3,
+			DefaultMaxTokens:     4096,
+		},
+		sessionLog: sessionLog,
+	}
+}
+
+// ─── /validate ─────────────────────────────────────────────────
+
+func TestValidate_RejectsUnknownKind(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "")
+	body := []byte(`{"kind":"unknown","artifact":{}}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Fatalf("expected 400 for unknown kind, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+func TestValidate_RejectsMissingArtifact(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "")
+	body := []byte(`{"kind":"playbook"}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Fatalf("expected 400 for missing artifact, got %d", w.Code)
+	}
+}
+
+func TestValidate_PlaybookHappyPath(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "")
+	body := []byte(`{
+		"kind": "playbook",
+		"artifact": {
+			"operation": "fill: Welder x2 in Toledo, OH",
+			"endorsed_names": ["W-1","W-2"],
+			"target_count": 2,
+			"fingerprint": "abc123"
+		}
+	}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	var report validator.Report
+	if err := json.Unmarshal(w.Body.Bytes(), &report); err != nil {
+		t.Fatalf("decode response: %v", err)
+	}
+	if report.ElapsedMs < 0 {
+		t.Errorf("elapsed_ms negative: %d", report.ElapsedMs)
+	}
+}
+
+func TestValidate_PlaybookSchemaErrorReturns422(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "")
+	body := []byte(`{
+		"kind": "playbook",
+		"artifact": {
+			"operation": "wrong_prefix: foo",
+			"endorsed_names": ["a"],
+			"fingerprint": "x"
+		}
+	}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusUnprocessableEntity {
+		t.Fatalf("expected 422, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	var ve validator.ValidationError
+	if err := json.Unmarshal(w.Body.Bytes(), &ve); err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	if ve.Kind != validator.ErrSchema {
+		t.Errorf("kind = %v, want schema", ve.Kind)
+	}
+}
+
+func TestValidate_FillRoutesThroughLookup(t *testing.T) {
+	city := "Toledo"
+	lookup := validator.NewInMemoryWorkerLookup([]validator.WorkerRecord{
+		{CandidateID: "W-1", Name: "Ada", Status: "active", City: &city},
+	})
+	r := newTestRouter(lookup, "")
+
+	// Candidate that doesn't exist in lookup → consistency failure.
+	body := []byte(`{
+		"kind": "fill",
+		"artifact": {
+			"fills": [{"candidate_id":"W-PHANTOM","name":"Nobody"}]
+		},
+		"context": {"target_count": 1, "city": "Toledo", "client_id": "C-1"}
+	}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusUnprocessableEntity {
+		t.Fatalf("expected 422 for phantom candidate, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+func TestValidate_ContextMergedIntoArtifactContext(t *testing.T) {
+	// _context.target_count from the request `context` block must
+	// reach the FillValidator's completeness check. Without the
+	// merge, target_count would default to 0 and any non-empty fills
+	// list would fail Completeness.
+	city := "Toledo"
+	role := "Welder"
+	lookup := validator.NewInMemoryWorkerLookup([]validator.WorkerRecord{
+		{CandidateID: "W-1", Name: "Ada", Status: "active", City: &city, Role: &role},
+	})
+	r := newTestRouter(lookup, "")
+	body := []byte(`{
+		"kind": "fill",
+		"artifact": {"fills":[{"candidate_id":"W-1","name":"Ada"}]},
+		"context": {"target_count": 1, "city": "Toledo", "role": "Welder", "client_id": "C-1"}
+	}`)
+	req := httptest.NewRequest("POST", "/validate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200 with context merged, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}
+
+// ─── /iterate ──────────────────────────────────────────────────
+
+// fakeChatd returns a stand-in chatd HTTP server that emits the given
+// content string for every /chat call. Caller closes the server.
+func fakeChatd(t *testing.T, content string) *httptest.Server {
+	t.Helper()
+	mux := chi.NewRouter()
+	mux.Post("/chat", func(w http.ResponseWriter, _ *http.Request) {
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"model":     "test-model",
+			"content":   content,
+			"provider":  "test",
+			"latency_ms": 1,
+		})
+	})
+	return httptest.NewServer(mux)
+}
+
+func TestIterate_RejectsMissingFields(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "")
+	body := []byte(`{"kind":"playbook","prompt":"x"}`) // missing provider+model
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadRequest {
+		t.Fatalf("expected 400, got %d", w.Code)
+	}
+}
+
+func TestIterate_HappyPath_ReturnsAcceptedArtifact(t *testing.T) {
+	server := fakeChatd(t, `{"operation":"fill: Welder x1 in Toledo, OH","endorsed_names":["W-1"],"target_count":1,"fingerprint":"abc"}`)
+	defer server.Close()
+
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), server.URL)
+	body, _ := json.Marshal(map[string]any{
+		"kind":     "playbook",
+		"prompt":   "produce a playbook artifact",
+		"provider": "ollama",
+		"model":    "qwen3.5:latest",
+	})
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	var resp validator.IterateResponse
+	if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	if resp.Iterations != 1 {
+		t.Errorf("iterations = %d, want 1", resp.Iterations)
+	}
+	if resp.Artifact["operation"] != "fill: Welder x1 in Toledo, OH" {
+		t.Errorf("artifact.operation: %v", resp.Artifact["operation"])
+	}
+}
+
+func TestIterate_MaxIterReturns422WithHistory(t *testing.T) {
+	// Always returns a no-JSON response, so iterate exhausts retries.
+	server := fakeChatd(t, "no json here, just prose")
+	defer server.Close()
+
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), server.URL)
+	body, _ := json.Marshal(map[string]any{
+		"kind":           "playbook",
+		"prompt":         "produce X",
+		"provider":       "ollama",
+		"model":          "x",
+		"max_iterations": 2,
+	})
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusUnprocessableEntity {
+		t.Fatalf("expected 422, got %d (body=%s)", w.Code, w.Body.String())
+	}
+	var fail validator.IterateFailure
+	if err := json.Unmarshal(w.Body.Bytes(), &fail); err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	if fail.Iterations != 2 {
+		t.Errorf("iterations = %d, want 2", fail.Iterations)
+	}
+	for _, h := range fail.History {
+		if h.Status.Kind != "no_json" {
+			t.Errorf("expected all attempts to be no_json, got %v", h.Status.Kind)
+		}
+	}
+}
+
+// TestIterate_WritesSessionRowOnSuccess locks the offline-analysis
+// contract: every successful /iterate call appends one SessionRecord
+// to the configured JSONL with grounded_in_roster=true (because the
+// model emitted real candidate IDs that exist in the roster).
+func TestIterate_WritesSessionRowOnSuccess(t *testing.T) {
+	city, role := "Toledo", "Welder"
+	lookup := validator.NewInMemoryWorkerLookup([]validator.WorkerRecord{
+		{CandidateID: "W-1", Name: "Ada", Status: "active", City: &city, Role: &role},
+	})
+	server := fakeChatd(t, `{"fills":[{"candidate_id":"W-1","name":"Ada"}]}`)
+	defer server.Close()
+
+	dir := t.TempDir()
+	logPath := filepath.Join(dir, "sessions.jsonl")
+	h := newTestHandlers(lookup, server.URL, validator.NewSessionLogger(logPath))
+	r := chi.NewRouter()
+	h.register(r)
+
+	body, _ := json.Marshal(map[string]any{
+		"kind":     "fill",
+		"prompt":   "produce fill",
+		"provider": "ollama",
+		"model":    "qwen3.5:latest",
+		"context":  map[string]any{"target_count": 1, "city": "Toledo", "role": "Welder", "client_id": "C-1"},
+	})
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200, got %d (body=%s)", w.Code, w.Body.String())
+	}
+
+	rows := readSessionLog(t, logPath)
+	if len(rows) != 1 {
+		t.Fatalf("expected 1 session row, got %d", len(rows))
+	}
+	row := rows[0]
+	if row["schema"] != validator.SessionRecordSchema {
+		t.Errorf("schema: %v", row["schema"])
+	}
+	if row["final_verdict"] != "accepted" {
+		t.Errorf("final_verdict: %v", row["final_verdict"])
+	}
+	if row["grounded_in_roster"] != true {
+		t.Errorf("grounded_in_roster: expected true (W-1 is in roster), got %v", row["grounded_in_roster"])
+	}
+	if row["kind"] != "fill" {
+		t.Errorf("kind: %v", row["kind"])
+	}
+}
+
+// TestIterate_GroundedFalseWhenArtifactReferencesPhantom is the
+// load-bearing forensic property J flagged 2026-05-02: phantom
+// worker IDs in the final accepted artifact must surface as
+// grounded_in_roster=false in the session log. This shouldn't happen
+// in practice (FillValidator catches phantoms before they reach this
+// row), but the explicit check guards against a future bug where the
+// validator weakens or a different validator path skips the roster
+// check. Defense-in-depth at the observability layer.
+func TestIterate_GroundedFalseWhenArtifactReferencesPhantom(t *testing.T) {
+	// Empty roster — every candidate ID is "phantom". Plus we use
+	// the playbook kind so the validator doesn't reject the artifact
+	// upfront (PlaybookValidator doesn't consult the roster), letting
+	// the artifact reach the session log. We then build a custom
+	// rosterCheckFor "fill" closure and exercise it directly.
+	lookup := validator.NewInMemoryWorkerLookup(nil)
+	h := newTestHandlers(lookup, "", nil)
+	check := h.rosterCheckFor("fill")
+	if check == nil {
+		t.Fatal("expected non-nil checker for fill kind")
+	}
+	got := check(map[string]any{
+		"fills": []any{
+			map[string]any{"candidate_id": "W-PHANTOM-NEVER-EXISTS"},
+		},
+	})
+	if got == nil || *got {
+		t.Errorf("expected grounded=false for phantom candidate, got %v", got)
+	}
+	// Other kinds: no roster concept → checker returns nil.
+	if h.rosterCheckFor("email") != nil {
+		t.Error("email kind should not have a roster checker")
+	}
+	if h.rosterCheckFor("playbook") != nil {
+		t.Error("playbook kind should not have a roster checker")
+	}
+}
+
+// TestIterate_WritesSessionRowOnFailure: max-iter exhaustion still
+// produces a row (omits artifact, captures the retry chain).
+func TestIterate_WritesSessionRowOnFailure(t *testing.T) {
+	server := fakeChatd(t, "no json here, just prose")
+	defer server.Close()
+
+	dir := t.TempDir()
+	logPath := filepath.Join(dir, "sessions.jsonl")
+	h := newTestHandlers(validator.NewInMemoryWorkerLookup(nil), server.URL,
+		validator.NewSessionLogger(logPath))
+	r := chi.NewRouter()
+	h.register(r)
+
+	body, _ := json.Marshal(map[string]any{
+		"kind":           "playbook",
+		"prompt":         "produce X",
+		"provider":       "ollama",
+		"model":          "x",
+		"max_iterations": 2,
+	})
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+
+	if w.Code != http.StatusUnprocessableEntity {
+		t.Fatalf("expected 422, got %d", w.Code)
+	}
+
+	rows := readSessionLog(t, logPath)
+	if len(rows) != 1 {
+		t.Fatalf("expected 1 session row, got %d", len(rows))
+	}
+	if rows[0]["final_verdict"] != "max_iter_exhausted" {
+		t.Errorf("final_verdict: %v", rows[0]["final_verdict"])
+	}
+	if _, hasArtifact := rows[0]["artifact"]; hasArtifact {
+		t.Errorf("artifact should be omitted on failure: %v", rows[0]["artifact"])
+	}
+}
+
+func TestIterate_ChatdDownReturns502(t *testing.T) {
+	r := newTestRouter(validator.NewInMemoryWorkerLookup(nil), "http://127.0.0.1:1") // unroutable
+	body, _ := json.Marshal(map[string]any{
+		"kind":     "playbook",
+		"prompt":   "X",
+		"provider": "ollama",
+		"model":    "x",
+	})
+	req := httptest.NewRequest("POST", "/iterate", bytes.NewReader(body))
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, req)
+	if w.Code != http.StatusBadGateway {
+		t.Fatalf("expected 502, got %d (body=%s)", w.Code, w.Body.String())
+	}
+}

cmd/vectord/main.go

@@ -17,6 +17,7 @@ import (
 	"os"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/go-chi/chi/v5"
@@ -62,7 +63,7 @@ func main() {
 		}
 	}
 
-	if err := shared.Run("vectord", cfg.Vectord.Bind, h.register); err != nil {
+	if err := shared.Run("vectord", cfg.Vectord.Bind, h.register, cfg.Auth); err != nil {
 		slog.Error("server", "err", err)
 		os.Exit(1)
 	}
@@ -71,6 +72,73 @@ func main() {
 type handlers struct {
 	reg     *vectord.Registry
 	persist *vectord.Persistor // nil when persistence is disabled
+
+	// saversMu guards lazy initialization of per-index save tasks.
+	// Each task coalesces synchronous Save calls into single-flight
+	// async saves so high-write-rate indexes (playbook_memory under
+	// multitier_100k load) don't pay one MinIO PUT per Add. See the
+	// saveTask docstring for the coalescing semantics.
+	saversMu sync.Mutex
+	savers   map[string]*saveTask
+}
+
+// saveTask coalesces saves for one index into a single-flight async
+// goroutine. While a save is in-flight, additional triggers mark
+// "pending" — the in-flight goroutine reruns the save after it
+// finishes, collapsing N concurrent triggers into at most 2 saves
+// (the current in-flight + one catch-up).
+//
+// Why: pre-2026-05-01 each successful Add called Persistor.Save
+// synchronously inside the request handler. For playbook_memory at
+// 1900-entry / 768-d, Encode + MinIO PUT cost 100-300ms. With 50
+// concurrent writers, end-to-end Add latency hit 2-2.5s purely from
+// save serialization (Save takes the index RLock for Encode, which
+// blocks new Adds taking the Lock).
+//
+// Trade-off: RPO. Add now returns OK before the save completes, so
+// a crash can lose up to ~1 save's worth of data. Acceptable for
+// the playbook-memory shape (learning loop — lost trace re-recorded
+// on next run) and consistent with ADR-005's fail-open posture.
+type saveTask struct {
+	mu       sync.Mutex
+	inflight bool
+	pending  bool
+}
+
+// trigger schedules a save. If a save is already in-flight, marks
+// pending and returns. If none in-flight, starts a goroutine that
+// runs save and any queued pending saves.
+//
+// save is the actual save operation (parameterized for testability).
+// Errors are logged via slog and not returned — same fail-open
+// posture as the prior synchronous saveAfter.
+func (s *saveTask) trigger(save func() error) {
+	s.mu.Lock()
+	if s.inflight {
+		s.pending = true
+		s.mu.Unlock()
+		return
+	}
+	s.inflight = true
+	s.mu.Unlock()
+
+	go func() {
+		for {
+			if err := save(); err != nil {
+				slog.Warn("persist save", "err", err)
+			}
+			s.mu.Lock()
+			if !s.pending {
+				s.inflight = false
+				s.mu.Unlock()
+				return
+			}
+			s.pending = false
+			s.mu.Unlock()
+			// Loop: re-run save to capture changes that arrived
+			// while we were saving.
+		}
+	}()
 }
 
 // rehydrate enumerates persisted indexes and loads each into the
@@ -103,19 +171,38 @@ func (h *handlers) rehydrate(ctx context.Context) (int, error) {
 	return loaded, nil
 }
 
-// saveAfter is the post-write persistence hook. Logs-not-fatal:
-// in-memory state is the source of truth in flight; a failed save
-// gets re-attempted on the next mutation, and the operator log
-// shows the storaged outage.
+// saveAfter triggers a coalesced async persistence for the index.
+// In-memory state is the source of truth in flight; a failed save
+// re-runs on the next mutation, and the operator log shows the
+// storaged outage.
+//
+// Coalescing semantics (added 2026-05-01 after multitier_100k
+// follow-up): rapid concurrent writes collapse into at most two
+// MinIO PUTs per index (current + one catch-up), instead of one
+// per Add. See the saveTask docstring.
 func (h *handlers) saveAfter(idx *vectord.Index) {
 	if h.persist == nil {
 		return
 	}
-	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-	defer cancel()
-	if err := h.persist.Save(ctx, idx); err != nil {
-		slog.Warn("persist save", "name", idx.Params().Name, "err", err)
+	name := idx.Params().Name
+	h.saversMu.Lock()
+	if h.savers == nil {
+		h.savers = make(map[string]*saveTask)
 	}
+	s, ok := h.savers[name]
+	if !ok {
+		s = &saveTask{}
+		h.savers[name] = s
+	}
+	h.saversMu.Unlock()
+	s.trigger(func() error {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		if err := h.persist.Save(ctx, idx); err != nil {
+			return err
+		}
+		return nil
+	})
 }
 
 // deleteAfter mirrors saveAfter for the Delete path.
@@ -137,6 +224,7 @@ func (h *handlers) register(r chi.Router) {
 	r.Delete("/vectors/index/{name}", h.handleDelete)
 	r.Post("/vectors/index/{name}/add", h.handleAdd)
 	r.Post("/vectors/index/{name}/search", h.handleSearch)
+	r.Post("/vectors/index/{name}/merge", h.handleMerge)
 }
 
 // createRequest mirrors POST /vectors/index body.
@@ -274,21 +362,18 @@ func (h *handlers) handleAdd(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 	}
+	// Pre-validation above is exhaustive (id, dim, finite, zero-norm),
+	// so BatchAdd takes the write-lock once and pushes the whole batch
+	// into coder/hnsw via one variadic Graph.Add. Saves N-1 lock
+	// acquisitions per HTTP batch.
+	batch := make([]vectord.BatchItem, len(req.Items))
 	for j, it := range req.Items {
-		if err := idx.Add(it.ID, it.Vector, it.Metadata); err != nil {
-			// Vector-validation errors (NaN/Inf, zero-norm under
-			// cosine) only surface here; pre-validation is intentional
-			// minimal scope (id + dim only).
-			if errors.Is(err, vectord.ErrDimensionMismatch) ||
-				strings.Contains(err.Error(), "non-finite") ||
-				strings.Contains(err.Error(), "zero-norm") {
-				http.Error(w, "items["+strconv.Itoa(j)+"]: "+err.Error(), http.StatusBadRequest)
-				return
-			}
-			slog.Error("add", "name", name, "id", it.ID, "err", err)
-			http.Error(w, "internal", http.StatusInternalServerError)
-			return
-		}
+		batch[j] = vectord.BatchItem{ID: it.ID, Vector: it.Vector, Metadata: it.Metadata}
+	}
+	if err := idx.BatchAdd(batch); err != nil {
+		slog.Error("batch add", "name", name, "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+		return
 	}
 	// One save per batch (post-loop), not per item. Per scrum
 	// O-W4-style discipline: HTTP-batch boundary is the natural unit.
@@ -327,6 +412,119 @@ func (h *handlers) handleSearch(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, searchResponse{Results: hits})
 }
 
+// mergeRequest body for POST /vectors/index/{name}/merge:
+//
+//	{"dest": "workers", "clear_source": true}
+//
+// "name" in the URL is the SOURCE index. Every item from source is
+// added to dest with the same id + vector + metadata. clear_source=
+// true (default false) deletes each successfully-merged item from
+// the source after add — leaves source empty when merge succeeds in
+// full. clear_source=false leaves source untouched (useful for dry-
+// run or "copy not move" semantics).
+//
+// Closes OPEN #1: periodic fresh→main index merge. The fresh_workers
+// two-tier index pattern grows monotonically; this endpoint is the
+// drain that operators (or a cron) call when fresh_workers crosses
+// the operational ceiling (~500 items per the original gating
+// criterion).
+//
+// Returns counts: {merged, skipped_already_present, failed, length_dest, length_source}.
+type mergeRequest struct {
+	Dest        string `json:"dest"`
+	ClearSource bool   `json:"clear_source,omitempty"`
+}
+
+type mergeResponse struct {
+	Merged                 int    `json:"merged"`
+	SkippedAlreadyPresent  int    `json:"skipped_already_present"`
+	Failed                 int    `json:"failed"`
+	LengthDest             int    `json:"length_dest"`
+	LengthSource           int    `json:"length_source"`
+	FirstError             string `json:"first_error,omitempty"`
+}
+
+func (h *handlers) handleMerge(w http.ResponseWriter, r *http.Request) {
+	name := chi.URLParam(r, "name")
+	src, err := h.reg.Get(name)
+	if errors.Is(err, vectord.ErrIndexNotFound) {
+		http.Error(w, "source not found", http.StatusNotFound)
+		return
+	} else if err != nil {
+		// Per scrum post_role_gate_v1 (Opus): non-ErrIndexNotFound
+		// errors must NOT fall through — src would be nil and the
+		// next deref panics.
+		slog.Error("merge: get source", "name", name, "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+		return
+	}
+	var req mergeRequest
+	if !decodeJSON(w, r, &req) {
+		return
+	}
+	if req.Dest == "" || req.Dest == name {
+		http.Error(w, "dest must be set and differ from source", http.StatusBadRequest)
+		return
+	}
+	dest, err := h.reg.Get(req.Dest)
+	if errors.Is(err, vectord.ErrIndexNotFound) {
+		http.Error(w, "dest not found", http.StatusNotFound)
+		return
+	} else if err != nil {
+		// Same fix as source — non-ErrIndexNotFound dest errors
+		// can't reach the merge body with a nil dest pointer.
+		slog.Error("merge: get dest", "name", req.Dest, "err", err)
+		http.Error(w, "internal", http.StatusInternalServerError)
+		return
+	}
+	// Dimension match is non-negotiable — silently moving a 768-d
+	// vector into a 384-d index would corrupt search forever.
+	if src.Params().Dimension != dest.Params().Dimension {
+		http.Error(w, "source/dest dimension mismatch", http.StatusBadRequest)
+		return
+	}
+
+	resp := mergeResponse{}
+	for _, id := range src.IDs() {
+		vec, meta, ok := src.Lookup(id)
+		if !ok {
+			// Vanished between IDs() snapshot and Lookup — concurrent
+			// delete. Treat as skip; not a failure.
+			continue
+		}
+		// Skip if dest already has the id (idempotent re-runs don't
+		// double-add). Operators expect "merge again" to be safe.
+		if _, _, exists := dest.Lookup(id); exists {
+			resp.SkippedAlreadyPresent++
+			if req.ClearSource {
+				src.Delete(id)
+			}
+			continue
+		}
+		if err := dest.Add(id, vec, meta); err != nil {
+			resp.Failed++
+			if resp.FirstError == "" {
+				resp.FirstError = "add " + id + ": " + err.Error()
+			}
+			continue
+		}
+		resp.Merged++
+		if req.ClearSource {
+			src.Delete(id)
+		}
+	}
+	// Persist both. Saving in-line under the merge endpoint is fine
+	// here because operators run this as a deliberate one-shot job,
+	// not a hot-path batch.
+	h.saveAfter(dest)
+	if req.ClearSource {
+		h.saveAfter(src)
+	}
+	resp.LengthDest = dest.Len()
+	resp.LengthSource = src.Len()
+	writeJSON(w, http.StatusOK, resp)
+}
+
 // decodeJSON reads + decodes a JSON body with a body-size cap.
 // Returns false (and writes the error response) on failure.
 func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {

cmd/vectord/main_test.go

@@ -0,0 +1,525 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/go-chi/chi/v5"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/vectord"
+)
+
+// Closes R-005 for vectord: cmd-level tests for the 6 routes.
+// Persistence-disabled mode (h.persist == nil) is the test config —
+// keeps tests pure-in-memory; persistence is covered by g1p_smoke +
+// proof GOLAKE-070.
+
+func mountedRouter() chi.Router {
+	h := &handlers{reg: vectord.NewRegistry()}
+	r := chi.NewRouter()
+	h.register(r)
+	return r
+}
+
+func TestRoutesMounted(t *testing.T) {
+	r := mountedRouter()
+	want := map[string]bool{
+		"POST /vectors/index":                 false,
+		"GET /vectors/index":                  false,
+		"GET /vectors/index/{name}":           false,
+		"DELETE /vectors/index/{name}":        false,
+		"POST /vectors/index/{name}/add":      false,
+		"POST /vectors/index/{name}/search":   false,
+		"POST /vectors/index/{name}/merge":    false,
+	}
+	chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		key := method + " " + route
+		if _, ok := want[key]; ok {
+			want[key] = true
+		}
+		return nil
+	})
+	for sig, found := range want {
+		if !found {
+			t.Errorf("expected route %q mounted", sig)
+		}
+	}
+}
+
+func TestHandleCreate_HappyPath_201(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader(`{"name":"test_idx","dimension":4}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusCreated {
+		t.Errorf("expected 201 on create, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleCreate_MissingDim_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader(`{"name":"missing_dim"}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on missing dim, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleCreate_MalformedJSON_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader("not json"))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on malformed, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleCreate_BodyTooLarge(t *testing.T) {
+	// 4xx range — see embedd's TestHandleEmbed_BodyTooLarge for the
+	// 413-vs-400 unwrap nuance. Contract is "client error, fails loud."
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	big := bytes.Repeat([]byte("x"), maxRequestBytes+(1<<20))
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json", bytes.NewReader(big))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 400 || resp.StatusCode >= 500 {
+		t.Errorf("expected 4xx on oversize, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleGetIndex_NotFound_404(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Get(srv.URL + "/vectors/index/nonexistent")
+	if err != nil {
+		t.Fatalf("GET: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleAdd_IndexNotFound_404(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/vectors/index/missing/add", "application/json",
+		strings.NewReader(`{"items":[{"id":"v1","vector":[1,2,3,4]}]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404 on add to missing index, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleAdd_EmptyItems_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	// Create index first.
+	http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader(`{"name":"empty_test","dimension":4}`))
+
+	resp, err := http.Post(srv.URL+"/vectors/index/empty_test/add", "application/json",
+		strings.NewReader(`{"items":[]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty items, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleAdd_DimMismatch_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader(`{"name":"dim_test","dimension":3}`))
+
+	resp, err := http.Post(srv.URL+"/vectors/index/dim_test/add", "application/json",
+		strings.NewReader(`{"items":[{"id":"x","vector":[1,2,3,4]}]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on dim mismatch, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleAdd_EmptyID_400(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	http.Post(srv.URL+"/vectors/index", "application/json",
+		strings.NewReader(`{"name":"id_test","dimension":4}`))
+
+	resp, err := http.Post(srv.URL+"/vectors/index/id_test/add", "application/json",
+		strings.NewReader(`{"items":[{"id":"","vector":[1,2,3,4]}]}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on empty id, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleSearch_IndexNotFound_404(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Post(srv.URL+"/vectors/index/missing/search", "application/json",
+		strings.NewReader(`{"vector":[1,2,3,4],"k":5}`))
+	if err != nil {
+		t.Fatalf("POST: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404 on search of missing index, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleDelete_NotFound_404(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	req, _ := http.NewRequest(http.MethodDelete, srv.URL+"/vectors/index/missing", nil)
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		t.Fatalf("DELETE: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404 deleting missing index, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleList_EmptyShape(t *testing.T) {
+	r := mountedRouter()
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	resp, err := http.Get(srv.URL + "/vectors/index")
+	if err != nil {
+		t.Fatalf("GET: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("expected 200, got %d", resp.StatusCode)
+	}
+}
+
+// TestHandleMerge end-to-end via mountedRouter (no external HTTP):
+// create source + dest indexes, populate source, merge with
+// clear_source=true, assert dest gained the items, source emptied.
+// Closes OPEN #1 — locks the merge contract at unit level so a
+// future regression on the IDs/Lookup/Add/Delete chain fails here
+// before any operator hits "merge again" and silently moves nothing.
+func TestHandleMerge_HappyPath_DrainAndClear(t *testing.T) {
+	h := &handlers{reg: vectord.NewRegistry()}
+	r := chi.NewRouter()
+	h.register(r)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	// Create both indexes (4-d for test simplicity).
+	for _, name := range []string{"fresh_test", "main_test"} {
+		body := `{"name":"` + name + `","dimension":4,"distance":"cosine"}`
+		resp, err := http.Post(srv.URL+"/vectors/index", "application/json", strings.NewReader(body))
+		if err != nil {
+			t.Fatalf("create %s: %v", name, err)
+		}
+		resp.Body.Close()
+	}
+
+	// Populate fresh_test with 3 items.
+	addBody := `{"items":[
+		{"id":"f-1","vector":[1,0,0,0],"metadata":{"name":"fresh-001"}},
+		{"id":"f-2","vector":[0,1,0,0],"metadata":{"name":"fresh-002"}},
+		{"id":"f-3","vector":[0,0,1,0],"metadata":{"name":"fresh-003"}}
+	]}`
+	resp, err := http.Post(srv.URL+"/vectors/index/fresh_test/add", "application/json", strings.NewReader(addBody))
+	if err != nil || resp.StatusCode != http.StatusOK {
+		t.Fatalf("add to fresh_test: status=%d err=%v", resp.StatusCode, err)
+	}
+	resp.Body.Close()
+
+	// Pre-seed main_test with one item that ALSO exists in fresh
+	// (collision) so we exercise the skipped_already_present path.
+	preBody := `{"items":[{"id":"f-1","vector":[1,0,0,0],"metadata":{"name":"main-collision"}}]}`
+	resp, err = http.Post(srv.URL+"/vectors/index/main_test/add", "application/json", strings.NewReader(preBody))
+	if err != nil || resp.StatusCode != http.StatusOK {
+		t.Fatalf("add collision to main_test: status=%d err=%v", resp.StatusCode, err)
+	}
+	resp.Body.Close()
+
+	// Merge fresh_test → main_test, clearing source.
+	mergeBody := `{"dest":"main_test","clear_source":true}`
+	resp, err = http.Post(srv.URL+"/vectors/index/fresh_test/merge", "application/json", strings.NewReader(mergeBody))
+	if err != nil {
+		t.Fatalf("merge: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		t.Errorf("expected 200 on merge, got %d", resp.StatusCode)
+	}
+	var out mergeResponse
+	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
+		t.Fatalf("decode merge resp: %v", err)
+	}
+	if out.Merged != 2 {
+		t.Errorf("expected 2 merged (f-2 + f-3), got %d", out.Merged)
+	}
+	if out.SkippedAlreadyPresent != 1 {
+		t.Errorf("expected 1 skipped (f-1 collision), got %d", out.SkippedAlreadyPresent)
+	}
+	if out.LengthSource != 0 {
+		t.Errorf("expected source emptied, got len=%d", out.LengthSource)
+	}
+	if out.LengthDest != 3 {
+		t.Errorf("expected dest len=3 after merge, got %d", out.LengthDest)
+	}
+}
+
+func TestHandleMerge_DimensionMismatch_400(t *testing.T) {
+	h := &handlers{reg: vectord.NewRegistry()}
+	r := chi.NewRouter()
+	h.register(r)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	for _, c := range []struct{ name string; dim int }{
+		{"src_4d", 4},
+		{"dst_8d", 8},
+	} {
+		body := `{"name":"` + c.name + `","dimension":` + strconv.Itoa(c.dim) + `,"distance":"cosine"}`
+		resp, err := http.Post(srv.URL+"/vectors/index", "application/json", strings.NewReader(body))
+		if err != nil {
+			t.Fatalf("create %s: %v", c.name, err)
+		}
+		resp.Body.Close()
+	}
+
+	resp, err := http.Post(srv.URL+"/vectors/index/src_4d/merge", "application/json",
+		strings.NewReader(`{"dest":"dst_8d"}`))
+	if err != nil {
+		t.Fatalf("merge: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 on dim mismatch, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleMerge_DestNotFound_404(t *testing.T) {
+	h := &handlers{reg: vectord.NewRegistry()}
+	r := chi.NewRouter()
+	h.register(r)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	body := `{"name":"only_src","dimension":4}`
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json", strings.NewReader(body))
+	if err != nil {
+		t.Fatalf("create: %v", err)
+	}
+	resp.Body.Close()
+
+	resp, err = http.Post(srv.URL+"/vectors/index/only_src/merge", "application/json",
+		strings.NewReader(`{"dest":"missing_dest"}`))
+	if err != nil {
+		t.Fatalf("merge: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusNotFound {
+		t.Errorf("expected 404 for missing dest, got %d", resp.StatusCode)
+	}
+}
+
+func TestHandleMerge_SameSourceDest_400(t *testing.T) {
+	h := &handlers{reg: vectord.NewRegistry()}
+	r := chi.NewRouter()
+	h.register(r)
+	srv := httptest.NewServer(r)
+	defer srv.Close()
+
+	body := `{"name":"self","dimension":4}`
+	resp, err := http.Post(srv.URL+"/vectors/index", "application/json", strings.NewReader(body))
+	if err != nil {
+		t.Fatalf("create: %v", err)
+	}
+	resp.Body.Close()
+
+	resp, err = http.Post(srv.URL+"/vectors/index/self/merge", "application/json",
+		strings.NewReader(`{"dest":"self"}`))
+	if err != nil {
+		t.Fatalf("merge: %v", err)
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Errorf("expected 400 for self-merge, got %d", resp.StatusCode)
+	}
+}
+
+func TestSearchK_DefaultsAndMax(t *testing.T) {
+	if defaultK <= 0 {
+		t.Errorf("defaultK = %d, must be > 0", defaultK)
+	}
+	if maxK < defaultK {
+		t.Errorf("maxK=%d < defaultK=%d", maxK, defaultK)
+	}
+	// Sanity bounds.
+	if maxK > 100_000 {
+		t.Errorf("maxK=%d unreasonably large", maxK)
+	}
+}
+
+// TestSaveTask_Coalesces locks the multitier_100k follow-up: a
+// burst of triggers must collapse into at most 2 actual saves
+// (the in-flight one + one catch-up). Without coalescing, every
+// trigger would yield a save and concurrent writers would
+// serialize on the index RLock during Encode (the original
+// 1-2.5s tail-latency cause).
+func TestSaveTask_Coalesces(t *testing.T) {
+	var (
+		s        saveTask
+		saveCnt  atomic.Int32
+		started  = make(chan struct{}, 1)
+		release  = make(chan struct{})
+	)
+	save := func() error {
+		// First save blocks until released so we can pile up
+		// triggers behind it. Subsequent saves return fast so the
+		// catch-up logic completes promptly.
+		n := saveCnt.Add(1)
+		if n == 1 {
+			started <- struct{}{}
+			<-release
+		}
+		return nil
+	}
+	// Trigger first save and wait for it to enter the blocked region.
+	s.trigger(save)
+	<-started
+	// Pile up triggers while the first is blocked. None of these
+	// should start their own goroutines — they should mark "pending".
+	for i := 0; i < 50; i++ {
+		s.trigger(save)
+	}
+	// Release the first save. The trigger logic should run ONE
+	// catch-up save for all 50 piled-up triggers, then return.
+	close(release)
+	// Wait for the goroutine to drain.
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		s.mu.Lock()
+		idle := !s.inflight && !s.pending
+		s.mu.Unlock()
+		if idle {
+			break
+		}
+		time.Sleep(5 * time.Millisecond)
+	}
+	got := saveCnt.Load()
+	if got != 2 {
+		t.Errorf("save count = %d, want 2 (one in-flight + one catch-up)", got)
+	}
+}
+
+// TestSaveTask_RunsOnce — single trigger fires exactly one save.
+func TestSaveTask_RunsOnce(t *testing.T) {
+	var s saveTask
+	var n atomic.Int32
+	done := make(chan struct{})
+	s.trigger(func() error {
+		n.Add(1)
+		close(done)
+		return nil
+	})
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("trigger goroutine never ran")
+	}
+	// Wait briefly for the goroutine to mark inflight=false.
+	time.Sleep(20 * time.Millisecond)
+	if got := n.Load(); got != 1 {
+		t.Errorf("save count = %d, want 1", got)
+	}
+}
+
+// TestSaveTask_LogsSaveError — a save error doesn't break the
+// coalescing state machine; subsequent triggers still work.
+func TestSaveTask_LogsSaveError(t *testing.T) {
+	var s saveTask
+	var n atomic.Int32
+	wantErr := errors.New("boom")
+	var wg sync.WaitGroup
+	wg.Add(1)
+	s.trigger(func() error {
+		defer wg.Done()
+		n.Add(1)
+		return wantErr
+	})
+	wg.Wait()
+	// State must reset so the next trigger fires another save.
+	time.Sleep(20 * time.Millisecond)
+	wg.Add(1)
+	s.trigger(func() error {
+		defer wg.Done()
+		n.Add(1)
+		return nil
+	})
+	wg.Wait()
+	if got := n.Load(); got != 2 {
+		t.Errorf("save count = %d, want 2 (failure must not stall the task)", got)
+	}
+}

deploy/etc-lakehouse/auth.env.example

@@ -0,0 +1,23 @@
+# /etc/lakehouse/auth.env — inter-service auth token per ADR-006.
+#
+# Mode 0600, owned by the lakehouse user (or root if systemd reads it
+# before dropping privileges via User=). Loaded by every daemon's
+# systemd unit via EnvironmentFile=-/etc/lakehouse/auth.env (the `-`
+# prefix means "missing file is OK" so loopback-only deploys can skip
+# this entirely).
+#
+# When the daemon binds non-loopback (anything other than 127.0.0.0/8
+# or ::1), AUTH_TOKEN MUST be set — otherwise shared.Run refuses to
+# start (R-001 + R-007 mechanical gate). Loopback-only deploys can
+# leave this empty.
+#
+# Token rotation (ADR-006 Decision 6.5):
+#   1. Generate new secret
+#   2. Set AUTH_SECONDARY_TOKEN to new secret while AUTH_TOKEN stays
+#      on old (lakehouse.toml [auth].secondary_tokens reads this)
+#   3. Update every caller to use new secret
+#   4. Promote: AUTH_TOKEN=<new>, clear AUTH_SECONDARY_TOKEN
+#   5. Restart daemons (or SIGHUP once hot-reload lands)
+
+AUTH_TOKEN=
+# AUTH_SECONDARY_TOKEN=  # only set during rotation windows

deploy/etc-lakehouse/langfuse.env.example

@@ -0,0 +1,19 @@
+# /etc/lakehouse/langfuse.env — Langfuse trace observability.
+#
+# Mode 0640, root:lakehouse. Loaded by every daemon's systemd unit
+# via EnvironmentFile=-/etc/lakehouse/langfuse.env. The `-` prefix
+# means "missing file is OK" — Langfuse is optional. If the file is
+# missing or any of the three env vars below is empty, every daemon
+# runs without Langfuse and with no logged warnings (best-effort).
+#
+# When all three are set, every authenticated HTTP request to every
+# daemon emits one Langfuse trace + span, browseable at LANGFUSE_URL.
+# /health is exempt to keep LB probes from polluting traces.
+#
+# Same env vars + same shape as the file the multi_coord_stress
+# driver reads — operators ship one langfuse.env across the whole
+# deploy.
+
+LANGFUSE_URL=http://localhost:3001
+LANGFUSE_PUBLIC_KEY=pk-lf-staffing
+LANGFUSE_SECRET_KEY=sk-lf-staffing-secret

deploy/etc-lakehouse/secrets-go.toml.example

@@ -0,0 +1,25 @@
+# /etc/lakehouse/secrets-go.toml — per-bucket S3 credentials.
+#
+# Mode 0600, root-owned (storaged + queryd both need to read it; they
+# run as the lakehouse user, so chgrp lakehouse + 0640 is fine if the
+# group is restricted).
+#
+# Schema is one [s3.<bucket-logical-name>] block per bucket the
+# storaged BucketRegistry should serve. G0 is single-bucket
+# ("primary" — see cmd/storaged/main.go primaryBucket const). G2
+# multi-bucket federation will add more entries here.
+#
+# This file is DELIBERATELY NOT in version control. Operators copy
+# this template, fill in real credentials, and place at
+# /etc/lakehouse/secrets-go.toml. The committed lakehouse.toml [s3]
+# block has bucket=<logical-name> + endpoint=<url> + region=<region>
+# — only the credentials live here.
+
+[s3.primary]
+access_key_id     = "REPLACE_ME"
+secret_access_key = "REPLACE_ME"
+
+# Future G2 example — multiple buckets:
+# [s3.archive]
+# access_key_id     = "..."
+# secret_access_key = "..."

deploy/systemd/lakehouse-catalogd.service

@@ -0,0 +1,31 @@
+[Unit]
+Description=Lakehouse-Go catalogd — Parquet manifest registry
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target lakehouse-storaged.service
+Wants=network-online.target
+Requires=lakehouse-storaged.service
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/catalogd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-catalogd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-chatd.service

@@ -0,0 +1,42 @@
+[Unit]
+Description=Lakehouse-Go chatd — multi-provider LLM dispatcher
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+# Operator prereq: Ollama on localhost:11434 for the bare/ollama/
+# providers; cloud providers (ollama_cloud, openrouter, opencode,
+# kimi) read keys from /etc/lakehouse/<provider>.env per chatd
+# config. Missing key files leave that provider unregistered (404
+# at first call, never 503).
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/chatd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+# chatd reads provider key files via paths in lakehouse.toml [chatd]
+# (ollama_cloud_key_file etc.) — each is its own EnvironmentFile so
+# operators can rotate one provider without restarting others.
+EnvironmentFile=-/etc/lakehouse/ollama_cloud.env
+EnvironmentFile=-/etc/lakehouse/openrouter.env
+EnvironmentFile=-/etc/lakehouse/opencode.env
+EnvironmentFile=-/etc/lakehouse/kimi.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-chatd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-embedd.service

@@ -0,0 +1,34 @@
+[Unit]
+Description=Lakehouse-Go embedd — text→vector via Ollama
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+# Operator prereq: Ollama running at the URL in lakehouse.toml
+# [embedd].provider_url, with default_model loaded (e.g.
+# `ollama pull nomic-embed-text-v2-moe`). Not a systemd unit we
+# control; embedd surfaces unreachable-Ollama as 502 at request time.
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/embedd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-embedd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-gateway.service

@@ -0,0 +1,33 @@
+[Unit]
+Description=Lakehouse-Go gateway — single OpenAI-compat-shaped edge proxy
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target lakehouse-storaged.service lakehouse-catalogd.service lakehouse-ingestd.service lakehouse-queryd.service lakehouse-vectord.service lakehouse-embedd.service lakehouse-pathwayd.service lakehouse-observerd.service lakehouse-matrixd.service lakehouse-chatd.service
+Wants=network-online.target lakehouse-storaged.service lakehouse-catalogd.service lakehouse-ingestd.service lakehouse-queryd.service lakehouse-vectord.service lakehouse-embedd.service lakehouse-pathwayd.service lakehouse-observerd.service lakehouse-matrixd.service lakehouse-chatd.service
+# gateway is the public-facing edge — if any upstream is down, the
+# proxy returns 502 at request time. Wants= (not Requires=) so a
+# single upstream restart doesn't cascade-restart the gateway.
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/gateway -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-gateway
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-go.target

@@ -0,0 +1,11 @@
+[Unit]
+Description=Lakehouse-Go — all 11 daemons (gateway + 10 backing services)
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+# A single target operators can `systemctl start` / `stop` / `enable`
+# instead of touching each daemon individually. Per-daemon units list
+# this target in their [Install].WantedBy=, so `systemctl enable
+# lakehouse-go.target` enables every daemon's unit, and a target
+# start cascades through Wants=.
+
+[Install]
+WantedBy=multi-user.target

deploy/systemd/lakehouse-ingestd.service

@@ -0,0 +1,31 @@
+[Unit]
+Description=Lakehouse-Go ingestd — CSV → Parquet → catalogd registration
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target lakehouse-storaged.service lakehouse-catalogd.service
+Wants=network-online.target
+Requires=lakehouse-storaged.service lakehouse-catalogd.service
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/ingestd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-ingestd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-matrixd.service

@@ -0,0 +1,31 @@
+[Unit]
+Description=Lakehouse-Go matrixd — multi-corpus retrieve+merge with Shape B playbook
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target lakehouse-embedd.service lakehouse-vectord.service
+Wants=network-online.target
+Requires=lakehouse-embedd.service lakehouse-vectord.service
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/matrixd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-matrixd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-observerd.service

@@ -0,0 +1,35 @@
+[Unit]
+Description=Lakehouse-Go observerd — witness ring + workflow runner + inbox
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+# observerd CAN call matrixd (workflow modes that hit matrix.search)
+# but doesn't strictly require it — modes that fail at startup are
+# logged and the daemon keeps running. So no Requires= here.
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/observerd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+# observerd's [observerd].persist_path defaults under
+# /var/lib/lakehouse/observer/ for ops.jsonl persistence.
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-observerd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-pathwayd.service

@@ -0,0 +1,32 @@
+[Unit]
+Description=Lakehouse-Go pathwayd — Mem0-style versioned trace store
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/pathwayd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+# pathwayd's [pathwayd].persist_path defaults under /var/lib/lakehouse/pathway/
+# in production for JSONL append-only persistence.
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-pathwayd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-queryd.service

@@ -0,0 +1,34 @@
+[Unit]
+Description=Lakehouse-Go queryd — DuckDB-over-S3 SQL surface
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target lakehouse-catalogd.service
+Wants=network-online.target
+Requires=lakehouse-catalogd.service
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+# queryd's DuckDB httpfs path needs S3 credentials; secrets_path
+# in lakehouse.toml [queryd] points at /etc/lakehouse/secrets-go.toml
+# which storaged also reads.
+ExecStart=/usr/local/bin/lakehouse/queryd -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-queryd
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-storaged.service

@@ -0,0 +1,40 @@
+[Unit]
+Description=Lakehouse-Go storaged — S3-backed object store gateway
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+# Operator prereq: MinIO (or AWS S3) reachable at the URL in
+# /etc/lakehouse/secrets-go.toml [s3.primary]. Not a systemd unit
+# we control, so we just wait for network and let the bind+probe
+# in storaged surface unreachable-bucket errors.
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/storaged -config /etc/lakehouse/lakehouse.toml -secrets /etc/lakehouse/secrets-go.toml
+Restart=on-failure
+RestartSec=5
+
+# Per ADR-006 Decision 6.2: auth token from env, not committed TOML.
+# Empty AUTH_TOKEN is fine for loopback-only deploys (matches
+# requireAuthOnNonLoopback gate at startup).
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+# Hardening — minimum needed for the daemon to read its config
+# + write its log + open its bind port.
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+# Log routing — JSON to journald, structured per slogRequest middleware.
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-storaged
+
+[Install]
+WantedBy=lakehouse-go.target

deploy/systemd/lakehouse-vectord.service

@@ -0,0 +1,33 @@
+[Unit]
+Description=Lakehouse-Go vectord — HNSW vector index registry
+Documentation=https://git.agentview.dev/profit/golangLAKEHOUSE
+After=network-online.target
+Wants=network-online.target
+# storaged is OPTIONAL for vectord — when [vectord].storaged_url is
+# empty, indexes are in-memory-only and don't survive restart. When
+# set, vectord persists LHV1 files via storaged. Operator's call.
+
+[Service]
+Type=simple
+User=lakehouse
+Group=lakehouse
+WorkingDirectory=/var/lib/lakehouse
+ExecStart=/usr/local/bin/lakehouse/vectord -config /etc/lakehouse/lakehouse.toml
+Restart=on-failure
+RestartSec=5
+
+EnvironmentFile=-/etc/lakehouse/auth.env
+EnvironmentFile=-/etc/lakehouse/langfuse.env
+
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths=/var/lib/lakehouse /var/log/lakehouse
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=lakehouse-vectord
+
+[Install]
+WantedBy=lakehouse-go.target

docker-compose.yml

@@ -0,0 +1,180 @@
+# Lakehouse-Go — 11-daemon docker-compose deployment.
+#
+# Same dependency graph as deploy/systemd/, mapped to Compose's
+# `depends_on` + healthchecks. Operators MUST provide:
+#   - lakehouse.toml mounted at /etc/lakehouse/lakehouse.toml
+#   - secrets-go.toml mounted at /etc/lakehouse/secrets-go.toml
+#     (mode 0600 on host)
+#   - auth.env mounted at /etc/lakehouse/auth.env (per ADR-006 —
+#     required if any service binds non-loopback)
+#   - MinIO + Ollama reachable from the lakehouse network
+#
+# Bring up:  docker compose up -d
+# Tear down: docker compose down
+# Logs:      docker compose logs -f gateway
+# Status:    docker compose ps
+
+x-lakehouse-base: &lakehouse-base
+  image: lakehouse-go:latest
+  restart: unless-stopped
+  networks: [lakehouse]
+  user: "999:999"  # matches the lakehouse user baked into the image
+  volumes:
+    - ./lakehouse.toml:/etc/lakehouse/lakehouse.toml:ro
+    - lakehouse-state:/var/lib/lakehouse
+  env_file:
+    - ./auth.env
+
+x-healthcheck: &lakehouse-healthcheck
+  test: ["CMD", "curl", "-sSf", "http://127.0.0.1:$$PORT/health"]
+  interval: 10s
+  timeout: 2s
+  start_period: 5s
+  retries: 3
+
+services:
+
+  storaged:
+    <<: *lakehouse-base
+    container_name: lakehouse-storaged
+    command: ["/usr/local/bin/lakehouse/storaged", "-config", "/etc/lakehouse/lakehouse.toml", "-secrets", "/etc/lakehouse/secrets-go.toml"]
+    ports: ["3211:3211"]
+    volumes:
+      - ./lakehouse.toml:/etc/lakehouse/lakehouse.toml:ro
+      - ./secrets-go.toml:/etc/lakehouse/secrets-go.toml:ro
+      - lakehouse-state:/var/lib/lakehouse
+    environment: [PORT=3211]
+    healthcheck: *lakehouse-healthcheck
+
+  catalogd:
+    <<: *lakehouse-base
+    container_name: lakehouse-catalogd
+    command: ["/usr/local/bin/lakehouse/catalogd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3212:3212"]
+    environment: [PORT=3212]
+    healthcheck: *lakehouse-healthcheck
+    depends_on:
+      storaged: { condition: service_healthy }
+
+  ingestd:
+    <<: *lakehouse-base
+    container_name: lakehouse-ingestd
+    command: ["/usr/local/bin/lakehouse/ingestd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3213:3213"]
+    environment: [PORT=3213]
+    healthcheck: *lakehouse-healthcheck
+    depends_on:
+      storaged: { condition: service_healthy }
+      catalogd: { condition: service_healthy }
+
+  queryd:
+    <<: *lakehouse-base
+    container_name: lakehouse-queryd
+    command: ["/usr/local/bin/lakehouse/queryd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3214:3214"]
+    volumes:
+      - ./lakehouse.toml:/etc/lakehouse/lakehouse.toml:ro
+      - ./secrets-go.toml:/etc/lakehouse/secrets-go.toml:ro
+      - lakehouse-state:/var/lib/lakehouse
+    environment: [PORT=3214]
+    healthcheck: *lakehouse-healthcheck
+    depends_on:
+      catalogd: { condition: service_healthy }
+
+  vectord:
+    <<: *lakehouse-base
+    container_name: lakehouse-vectord
+    command: ["/usr/local/bin/lakehouse/vectord", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3215:3215"]
+    environment: [PORT=3215]
+    healthcheck: *lakehouse-healthcheck
+
+  embedd:
+    <<: *lakehouse-base
+    container_name: lakehouse-embedd
+    command: ["/usr/local/bin/lakehouse/embedd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3216:3216"]
+    environment: [PORT=3216]
+    healthcheck: *lakehouse-healthcheck
+    # No depends_on — Ollama is operator infra, not a compose service.
+    # embedd surfaces unreachable Ollama as 502 at request time.
+
+  pathwayd:
+    <<: *lakehouse-base
+    container_name: lakehouse-pathwayd
+    command: ["/usr/local/bin/lakehouse/pathwayd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3217:3217"]
+    environment: [PORT=3217]
+    healthcheck: *lakehouse-healthcheck
+
+  observerd:
+    <<: *lakehouse-base
+    container_name: lakehouse-observerd
+    command: ["/usr/local/bin/lakehouse/observerd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3219:3219"]
+    environment: [PORT=3219]
+    healthcheck: *lakehouse-healthcheck
+
+  matrixd:
+    <<: *lakehouse-base
+    container_name: lakehouse-matrixd
+    command: ["/usr/local/bin/lakehouse/matrixd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3218:3218"]
+    environment: [PORT=3218]
+    healthcheck: *lakehouse-healthcheck
+    depends_on:
+      embedd: { condition: service_healthy }
+      vectord: { condition: service_healthy }
+
+  chatd:
+    <<: *lakehouse-base
+    container_name: lakehouse-chatd
+    command: ["/usr/local/bin/lakehouse/chatd", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3220:3220"]
+    environment: [PORT=3220]
+    # chatd's per-provider key files — each as its own env_file so a
+    # missing file is just "this provider unregistered" not a startup
+    # failure. Same pattern as the systemd unit's EnvironmentFile=- list.
+    env_file:
+      - ./auth.env
+      - ./ollama_cloud.env
+      - ./openrouter.env
+      - ./opencode.env
+      - ./kimi.env
+    healthcheck: *lakehouse-healthcheck
+
+  gateway:
+    <<: *lakehouse-base
+    container_name: lakehouse-gateway
+    command: ["/usr/local/bin/lakehouse/gateway", "-config", "/etc/lakehouse/lakehouse.toml"]
+    ports: ["3110:3110"]
+    environment: [PORT=3110]
+    healthcheck: *lakehouse-healthcheck
+    # Wants= equivalent in compose: depends_on without
+    # condition: service_healthy, so a single upstream restart
+    # doesn't cascade-restart the gateway.
+    depends_on:
+      - storaged
+      - catalogd
+      - ingestd
+      - queryd
+      - vectord
+      - embedd
+      - pathwayd
+      - observerd
+      - matrixd
+      - chatd
+
+networks:
+  lakehouse:
+    driver: bridge
+
+volumes:
+  lakehouse-state:
+    # /var/lib/lakehouse persisted across container restarts. Bind
+    # to a host path if backups need to live outside the volume:
+    #   driver: local
+    #   driver_opts:
+    #     type: none
+    #     o: bind
+    #     device: /opt/lakehouse-state

docs/ARCHITECTURE_COMPARISON.md

@@ -0,0 +1,346 @@
+# Lakehouse: Rust vs Go architecture comparison
+
+> **Status**: Living document · primary source for the parallel-runtime
+> comparison.
+> **Owner**: J. Update this when either side ships a fix that changes
+> the table values, or when a new architectural axis surfaces.
+> **Last meaningful refresh**: 2026-05-02 (post-Lance-gauntlet + observability parity)
+
+This document compares the two parallel implementations of the lakehouse
+substrate — Rust at `/home/profit/lakehouse/` (production today), Go at
+`/home/profit/golangLAKEHOUSE/` (cutover-prep, Bun `/_go/*` slice live).
+The goal of running both lines is to find where each architecture is
+weak vs strong, address those gaps, and make the keep/maintain
+decision based on real evidence rather than preference.
+
+A snapshot of this document at any point in time is also captured at
+`reports/cutover/architecture_comparison.md`. The version in `docs/`
+is the source of truth; `reports/cutover/` is the historical record.
+
+---
+
+## How to update this doc
+
+Three triggers:
+
+1. **A fix lands on either side that moves a table value.** Update the
+   number, append a one-line entry to the change log at the bottom,
+   commit alongside the fix.
+2. **A new architectural axis surfaces.** Add a section. Match the
+   shape of existing sections (table + read paragraph).
+3. **A keep/maintain decision is made.** Update the Recommendation
+   section + change log.
+
+Don't:
+- Delete sections without recording the reason in the change log.
+- Embed unverified claims — every "Rust is X" or "Go is X" should
+  point to either a load-test number, a code reference (`crate/file:line`),
+  or an explicit "asserted, not measured" caveat.
+
+---
+
+## Decisions tracker
+
+| Date | Decision | Effect |
+|---|---|---|
+| 2026-05-01 | Add LRU embed cache to Rust aibridge | Closes 236× perf gap. **DONE** (commit `150cc3b` in lakehouse). |
+| 2026-05-01 | Port FillValidator + EmailValidator to Go | Production safety net Go was missing. **DONE** (commit `b03521a` in golangLAKEHOUSE). |
+| 2026-05-01 | Multi-tier load test against 100k corpus | 335k scenarios in 5min, 4/6 at 0% fail. Surfaced coder/hnsw v0.6.1 bug. Recover guard added. **DONE** (multitier_100k.md). |
+| 2026-05-01 | **coder/hnsw v0.6.1 panic — REAL FIX landed** | Lifted source-of-truth out of coder/hnsw via `i.vectors map[string][]float32` side store + `safeGraphAdd`/`safeGraphDelete` recover wrappers + warm-path rebuild fallback. Re-run: 0 failures across 19,622 scenarios (was 96-98% on 2/6). **DONE.** Architecture invariant in STATE_OF_PLAY "DO NOT RELITIGATE". |
+| 2026-05-02 | **Substrate fix verified at original failure scale** | Re-ran multitier 5min @ conc=50 (the footprint that originally surfaced the bug at 96-98% fail). Result: 132,211 scenarios at 438/sec, **6/6 classes at 0% failure**. Throughput dropped 1,115/sec → 438/sec because broken scenarios now do real HNSW Add work. Tails healthy: surge_fill_validate p99=1.53s, playbook_record_replay p99=2.32s. **Fix scales — closing the open thread.** |
+| 2026-05-02 | **Drop Python sidecar from Rust aibridge — DONE** | `crates/aibridge/src/client.rs` rewrite (commit `ba928b1` in lakehouse). AiClient now talks Ollama directly: per-text `/api/embed` loop, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `std::process::Command nvidia-smi` for vram_snapshot. Public API unchanged — 0 callers updated. Verification: cargo test -p aibridge 32/32 PASS, live smoke `/ai/embed` returns 768-dim vector + `/v1/chat` returns "OK". Sidecar's `lab_ui.py` + `pipeline_lab.py` (~888 LOC dev-only UIs) keep running; only the hot-path embed/generate/rerank/admin routes are retired. Process count drops from "mega-binary + sidecar" to "mega-binary alone". |
+| 2026-05-02 | **Rust observability parity (trace-id + session JSONL) — DONE** | Mirrors the Go-side wave (commits `d6d2fdf` + `1a3a82a` in golangLAKEHOUSE). `crates/gateway/src/v1/iterate.rs` now reads `X-Lakehouse-Trace-Id` header, forwards to `/v1/chat` + `/v1/validate` hops, emits per-attempt Langfuse spans (`iterate.attempt[N]`), and writes one `SessionRecord` JSONL row per session via the new `crates/gateway/src/v1/session_log.rs` writer. New `crates/gateway/src/bin/parity_session_log` helper enables the cross-runtime parity probe at `scripts/cutover/parity/session_log_parity.sh` — **4/4 fixtures byte-equal** (including unicode prompts) after normalizing the 2 fields that legitimately differ (timestamp, daemon name). Both runtimes can write to the same path and DuckDB queries see them as one stream. 90/90 Rust unit tests PASS, 33 Go packages PASS. |
+| 2026-05-02 | **Cross-runtime parity gap closure — DONE** | Two follow-ups from the deploy + reality wave: (1) Go validatord didn't honor `X-Lakehouse-Trace-Id` when Langfuse middleware was a passthrough (no env config) — fixed by reading the header DIRECTLY in the iterate handler before the ctx fallback (Go commit `6847bbc`). (2) Rust `IterateResponse`/`IterateFailure` didn't echo `trace_id` back to callers — added the field with skip-serializing-if-none for backward compat (Rust commit `98b6647`). Plus enabled `[gateway].session_log_path` on live Rust pointing at the SAME path Go writes to, so DuckDB queries see one unified longitudinal log. **Verified end-to-end**: identical /v1/iterate to both runtimes lands in one sessions.jsonl tagged `daemon=gateway` (Rust) vs `daemon=validatord` (Go); response bodies both echo the forwarded trace_id. 24/24 parity assertions (validator+extract_json+session_log+materializer) hold post-restart. |
+| 2026-05-02 | **Port Rust materializer to Go (transforms.ts) — DONE** | `internal/materializer` + `cmd/materializer` + `materializer_smoke.sh`. Ports `transforms.ts` (12 transforms) + `build_evidence_index.ts`. Idempotency, day-partition, receipt. 14 tests green; on-wire JSON matches TS so both runtimes interoperate. |
+| 2026-05-02 | **Port Rust replay tool to Go — DONE** | `internal/replay` + `cmd/replay` + `replay_smoke.sh`. Ports `replay.ts` retrieve → bundle → /v1/chat → validate → log. Closes audit-FULL phase 7 live invocation on Go side. 14 tests green; same `data/_kb/replay_runs.jsonl` shape (schema=replay_run.v1) as TS. |
+| 2026-05-02 | **`/v1/validate` + `/v1/iterate` HTTP surface — DONE** | `cmd/validatord` (port 3221) hosts both endpoints. `internal/validator` gains `PlaybookValidator` (3rd kind), JSONL roster loader, and the `Iterate` orchestrator + `ExtractJSON` helper. Gateway proxies `/v1/validate` + `/v1/iterate` to validatord. Closes the last "Go-primary" backlog item (architecture_comparison.md item #7). 30+ tests + `validatord_smoke.sh` 5/5 PASS. |
+| 2026-05-02 | **Cross-runtime validator parity probe — surfaced wire-format gap** | New `scripts/cutover/parity/validator_parity.sh` runs 6 identical /v1/validate cases against Rust :3100 AND Go :4110, compares status + body. Result: **6/6 status codes match (logic-level equivalence holds), 5/6 body shapes diverge.** Rust returns serde-tagged enum `{"Schema":{"field":"x","reason":"y"}}`; Go returns flat struct `{"Kind":"schema","Field":"x","Reason":"y"}`. Any caller parsing the error envelope would break in cutover. **Open**: pick a target shape (Go matching Rust is the cutover-friendly direction) and align via custom `MarshalJSON` on `ValidationError`. |
+| 2026-05-02 | **Materializer parity probe — caught + fixed real bug** | New `scripts/cutover/parity/materializer_parity.sh` runs Bun + Go materializer on identical synthetic root, diffs output JSONL. Result on first run: **0/2 match** — Go's `Provenance.LineOffset` had `json:",omitempty"` and stripped the field on first-row records (line_offset=0 is semantically meaningful, not absent). 1-line fix (drop `omitempty` + comment explaining why). Re-run: **2/2 match**. Real cross-runtime gap surfaced + closed in same wave. |
+| 2026-05-02 | **extract_json parity probe — 12/12 match across edge cases** | New `scripts/cutover/parity/extract_json_parity.sh` runs identical model-output strings through Rust `gateway::v1::iterate::extract_json` AND Go `validator.ExtractJSON`. 12 fixtures: fenced/unfenced blocks, nested objects, unicode, escaped quotes, top-level array, malformed JSON. Substrate gate: `cargo test -p gateway extract_json` PASS before probe. Result: **12/12 match.** Algorithms genuinely equivalent. Rust side gained `pub` on `extract_json` + new `bin/parity_extract_json` (~30 LOC). |
+| 2026-05-02 | **Validator wire-format alignment — DONE** | Custom `MarshalJSON`/`UnmarshalJSON` on Go's `validator.ValidationError` emits the Rust serde-tagged-enum shape `{"Schema":{"field":"x","reason":"y"}}`. UnmarshalJSON also accepts the legacy flat shape (migration safety) and rejects unknown variants (drift guard for future Rust enum additions). 4 new pinning tests in `types_test.go`. Re-run validator parity probe: **6/6 match** (was 1/6). |
+| 2026-05-02 | **Lance backend gauntlet (4-pack + root-cause fix) — DONE** | Lance crate had zero tests + no smoke when audited this morning. Shipped: (a) `sanitize_lance_err` over all 5 routes (search/doc/index/append/migrate) — missing-index now 404 not 500, no `/home/` or `/root/.cargo/` paths leaked; (b) 7 unit tests in `crates/vectord-lance` with synth Parquet helper; (c) 9-probe `scripts/lance_smoke.sh` against live `:3100`; (d) 10M re-bench (`reports/lance_10m_rebench_2026-05-02.md`) — search warm ~20ms, search cold ~46ms median. Bench surfaced doc-fetch p50 ~100ms (300x slower than ADR-019 100K projection); root-caused to lance-bench bypassing IndexMeta → warming auto-build never fired → no `doc_id` btree. **Fix shipped (commit `5d30b3d`)**: `lance_migrate` HTTP handler now auto-builds the btree inline (1.2s on 10M, +269MB), drops doc-fetch to ~5ms (20x). Live verified 9/9 smoke + post-restart doc-fetch 4-15ms. |
+| 2026-05-03 | **Subject manifests + per-subject HMAC audit log — DONE on Rust + Go** | Local-first compliance substrate per `lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8. Rust shipped: `SubjectManifest` type + Registry CRUD (`crates/catalogd/src/registry.rs`), `SubjectAuditWriter` with HMAC-SHA256 chain + per-subject Mutex serialization + canonical-JSON via BTreeMap (`subject_audit.rs`), backfill ETL (`bin/backfill_subjects`), gateway tool dispatch + validator decorator wiring, legal-tier `/audit/subject/{id}` endpoint with constant-time-eq token + tampering detection, daily `bin/retention_sweep` (BIPA-aware, idempotent, no auto-mutation). Go shipped: identical `internal/catalogd/subject.go` reader + `VerifyChain` over RAW LINE BYTES (avoids time-precision drift), 11 unit tests. **6th cross-runtime parity probe**: `scripts/cutover/parity/subject_audit_parity.sh` — 6/6 byte-identical assertions across known-answer fixture + 5 real production audit logs. Surfaced + closed three drift classes in authoring loop: (1) Go `omitempty` stripping `trace_id:""`; (2) `time.RFC3339Nano` truncating trailing-zero nanoseconds where chrono AutoSi keeps 9 digits; (3) Go `json.Marshal` HTML-escaping `<>&` where serde keeps literal — fixed via `marshalNoEscapeHTML` + raw-bytes canonicalization. Two cross-lineage scrums caught real bugs each round (chain corruption race, schema-evolution HMAC drift, hardcoded "success" classifier, token min length, chain_root from windowed slice, tampering detection, HTML escape divergence). |
+| _open_ | Decide Lance vs Parquet+HNSW for primary | Lance verified production-ready at 10M (this morning's gauntlet). HNSW at 10M doesn't fit RAM (~60GB for vectors+graph), so the comparison is between Lance and Parquet+HNSW-with-spilling. Decide once we have a 10M ingest scenario where the Parquet path is bottlenecked. |
+| _open_ | Pick Go primary vs Rust primary | Both viable. Go has perf edge after today; Rust has production deploy + producer-side completeness. |
+
+---
+
+## Code volume
+
+| | Lines | Last verified |
+|---|---:|---|
+| Rust `crates/` (15 crates) | 35,447 | 2026-05-01 |
+| Rust `sidecar/` (Python) | 1,237 | 2026-05-01 |
+| Go `internal/` (20 packages) | 11,896 (+ validator 1190) | 2026-05-01 |
+| Go `cmd/` (14 binaries) | 3,232 | 2026-05-01 |
+| **Go total** | **~16,300** | 2026-05-01 |
+
+Go is ~46% the size of Rust on like-for-like surface (post-validator-port).
+The gap is largely `vectord` (Rust 11,005 lines vs Go 804) — Rust's
+vectord implements HNSW + Lance-format storage + benchmarking; Go's
+wraps `coder/hnsw` and stops there.
+
+---
+
+## Process model
+
+| | Rust | Go |
+|---|---|---|
+| Binaries running | **1** mega-process (gateway PID 1241, 14.9G RSS, 374% CPU under load) | **11** dedicated daemons (~100-300MB RSS each) |
+| Inter-component comms | In-process axum.nest (no network) | HTTP between daemons |
+| Crash blast radius | Whole system if any subsystem panics | One daemon dies, rest survive |
+| Horizontal scale | One unit only — can't scale individual components | Each daemon scales independently |
+| Deploy unit | Single binary | 11 systemd units |
+
+**Reading**: Rust's mega-binary is simpler ops at small scale (one
+thing to start, one log to tail). Go's daemons are simpler ops at
+production scale (kill the misbehaving one, restart it, others stay
+up). Go also lets you tune per-daemon resource limits via systemd.
+
+---
+
+## Python dependency (the load-bearing axis)
+
+This was the architectural difference that drove the original perf gap.
+
+```
+Pre-2026-05-02:
+  Rust embed:  gateway → HTTP → Python sidecar :3200 → HTTP → Ollama :11434
+  Go embed:    gateway → HTTP → Go embedd      :4216 → HTTP → Ollama :11434
+
+Post-2026-05-02 (commit ba928b1):
+  Rust embed:  gateway → HTTP → Ollama :11434                (sidecar dropped)
+  Go embed:    gateway → HTTP → Go embedd :4216 → HTTP → Ollama :11434
+```
+
+The hot-path Python sidecar (~120 LOC across embed.py / generate.py
+/ rerank.py / admin.py) was pure pass-through to Ollama and added no
+logic — just translation. AiClient was rewritten 2026-05-02 to call
+Ollama directly. `lab_ui.py` + `pipeline_lab.py` (~888 LOC of
+Streamlit-shape dev UIs) remain as ad-hoc tooling, not on the
+runtime hot path.
+
+### Performance impact (load-tested 2026-05-01, 6 rotating bodies, 10 concurrency, 30s)
+
+| Path | Pre-cache | Post-cache (`150cc3b`) | Δ |
+|---|---:|---:|---:|
+| **Rust /ai/embed** (via gateway) | 128 RPS · p50 78ms · p99 124ms | **30,279 RPS · p50 129µs · p99 5ms** | +236× RPS |
+| **Go /v1/embed** (via gateway → embedd) | 8,119 RPS · p50 0.79ms · p99 3ms | _unchanged_ | (already cached) |
+
+Rust now beats Go ~3.7× on cache-warm workloads. The cache being
+in-process inside Rust's gateway (no HTTP hop to a separate daemon)
+gives it the edge once both sides have caching.
+
+### What the cache fix did NOT do
+
+The Python sidecar is still in the Rust path on cache misses. Cold
+queries pay the full Python+Ollama tax. Dropping the sidecar
+(rewriting aibridge to call Ollama directly) is the next universal-win
+item — open in the Decisions tracker.
+
+---
+
+## Vector storage
+
+| | Rust | Go |
+|---|---|---|
+| HNSW lib | `hnsw_rs` (mature) | `coder/hnsw` (newer, smaller) |
+| Code size | 11,005 lines (`vectord` + `vectord-lance`) | 804 lines |
+| Lance-format storage | Yes (`vectord-lance` crate) | No |
+| Persistence | LanceDB or in-memory | MinIO + JSON envelope (v2 envelope as of `eb0dfdf`) |
+| Distance functions | cosine, euclidean, dot product | cosine, euclidean |
+
+**Reading**: Rust has the deeper substrate. Lance-format gives columnar
+persistence + zero-copy reads + Apache Arrow integration. For
+staffing-domain corpus sizes (5K-500K vectors) both work fine; for
+multi-million-row indexes Rust would have a real edge. **Defer the Go
+Lance port until corpus growth demands it.**
+
+---
+
+## Distillation pipeline (porting status)
+
+| Phase | Rust source | Go port |
+|---|---|---|
+| Materializer (transforms.ts) | TS, full | ❌ NOT YET PORTED |
+| Scorer | TS + Go | ✅ Ported |
+| Score categories + firewall | Pinned | ✅ Ported (`SftNever`) |
+| SFT export (synthesis) | TS, full (8 source classes) | ✅ Fully ported, 4-decimal byte-equal |
+| RAG export | TS | ❌ NOT YET PORTED |
+| Preference export | TS | ❌ NOT YET PORTED |
+| Audit-baselines | TS | ✅ Fully ported, byte-equal verified |
+| Audit-FULL phase 0/3/4 | TS | ✅ Ported |
+| Audit-FULL phase 1 (schema) | bun test | ✅ Via `go test` exec |
+| Audit-FULL phase 2 (materializer) | TS | ✅ Observer mode (read-only) |
+| Audit-FULL phase 5 (run summaries) | TS | ✅ Observer mode (read-only) |
+| Audit-FULL phase 6 (acceptance) | TS fixture harness | ❌ Skipped (TS-only deps) |
+| Audit-FULL phase 7 (replay) | TS | ✅ Observer mode (read-only) |
+| Replay tool | TS | ❌ NOT YET PORTED |
+| Quarantine writer | TS | ❌ NOT YET PORTED |
+
+**Reading**: Go has the substrate for everything observable (read
+paths) and SFT export end-to-end. The producer side (materializer,
+replay) is still Rust-only. To run the full pipeline from Go alone,
+the materializer + replay need porting.
+
+---
+
+## Production validators
+
+| | Rust | Go |
+|---|---|---|
+| FillValidator | `crates/validator/src/staffing/fill.rs` (12 unit tests) | ✅ **Ported 2026-05-01** (`internal/validator/fill.go` + 13 tests) |
+| EmailValidator | `crates/validator/src/staffing/email.rs` (12 tests) | ✅ **Ported 2026-05-01** (`internal/validator/email.go` + 11 tests) |
+| `/v1/validate` endpoint | Yes | ❌ NOT YET PORTED (validator network surface) |
+| `/v1/iterate` endpoint | Yes (gen→validate→correct→retry loop) | ❌ NOT YET PORTED |
+| Production validators load `workers_500k.parquet` at startup | Yes (75MB resident) | N/A — Go uses WorkerLookup interface; in-memory or adapter |
+
+**Reading**: With today's port, Go has the load-bearing validators.
+The network surface (`/v1/validate`, `/v1/iterate`) is the next
+piece — the in-memory validators work in-process; turning them into
+HTTP endpoints adds the production-shape access pattern.
+
+---
+
+## Substrate features unique to each side
+
+### Go has, Rust doesn't
+
+- **chatd 5-provider dispatcher** (kimi / opencode / openrouter / ollama_cloud / ollama).
+- **Cross-role gate** in matrix retrieve (real_001 fix). Verified by reality tests real_001..005.
+- **Multi-corpus matrix indexer** (Spec §3.4 component 2).
+- **Pathway memory** (Mem0-style versioned traces).
+- **Observer fail-safe semantics** (ADR-005 Decision 5.1).
+- **In-process embed cache** (CachedProvider + LRU). _Note: Rust got this 2026-05-01 too._
+- **LLM-based role extractor** (regex + qwen2.5 fallback).
+- **Persistent stack 3-layer isolation** (`scripts/cutover/start_go_stack.sh`).
+- **Cutover slice** (Bun `/_go/*` route, opt-in via systemd drop-in).
+- **Production load test** (`scripts/cutover/loadgen/`) with Bun-frontend + direct comparison.
+
+### Rust has, Go doesn't
+
+- **Lance-format vector storage** (vectord-lance crate, 605 lines).
+- **`truth` crate** (970 lines). Cross-source claim reconciliation.
+- **`journald` crate** (455 lines). Structured event journal.
+- **`/v1/validate` + `/v1/iterate` endpoints** (network surface).
+- **`ui` crate (Dioxus, 1,509 lines)**. Native desktop/web UI.
+- **Materializer + replay tools** (the "produce evidence" side).
+- **Acceptance harness** (22 invariants over fixtures, TS).
+- **Production deployment** (devop.live/lakehouse/* serves through Rust today).
+
+---
+
+## Strengths and weaknesses
+
+### Rust strengths
+
+- Mature, in production, serving real demo traffic.
+- Single deploy unit; one binary, one systemd service, one log.
+- Type system + memory safety; fewer runtime bugs in hot paths.
+- Mature library ecosystem (axum, tokio, polars, arrow, hnsw_rs, lance).
+- Native distillation pipeline; Go is the porter.
+- Production validators (now also in Go but Rust authored them).
+- Lance vector storage scales beyond 5M rows.
+- **In-process embed cache (post-`150cc3b`) makes Rust the fastest path on warm workloads.**
+
+### Rust weaknesses
+
+- **Python sidecar dependency** — every cache-miss AI call goes through Python. Adds 1 runtime + 1 process to ops. ~200 LOC to fix.
+- **Mega-binary blast radius** — gateway at 14.9G RSS means any panic kills the whole production system.
+- **Tail latency cliff under uncached load** — single async runtime serializes I/O completions.
+- **Compile times** — slow iteration vs Go's per-package builds.
+- **Coupling** — adding a feature touches gateway/v1/ and ripples across crates.
+
+### Go strengths
+
+- **Process isolation** — daemons crash independently; ops can `systemctl restart vectord` without touching gateway.
+- **Per-daemon scale** — embed cache lives in embedd; vectord shards independently. Hot daemons scale horizontally.
+- **No Python dependency** — every daemon talks to peers in HTTP/JSON. Native Go down to Ollama.
+- **In-process embed cache** at the daemon level (was the perf lever pre-Rust-cache).
+- **Smaller, denser code** — 16,300 lines vs Rust's 35,447 + 1,237 sidecar (~46% the size).
+- **Faster iteration** — `go build` of all 14 binaries is ~3-5s; Rust full rebuild is minutes.
+- **Cross-runtime artifact compatibility verified** — audit_baselines.jsonl, scored-runs JSONL, sft_export.jsonl all round-trip byte-equal.
+
+### Go weaknesses
+
+- **Distillation pipeline incomplete** — materializer + replay + RAG export + preference export still Rust-only.
+- **Validator network surface missing** — in-memory validators work, but `/v1/validate` HTTP endpoint not yet ported. Operators can't call validators over the wire from Go.
+- **Vector storage HNSW-only** — no Lance equivalent. Fine for current scale.
+- **Less production-tested** — cutover slice live but no real coordinator traffic yet.
+- **HTTP between daemons** — every cross-daemon call is a network round-trip. Latency fine on localhost (microseconds) but tail-latency contributes more than Rust's in-process composition.
+- **`coder/hnsw` is newer** than Rust's `hnsw_rs`. Less battle-tested.
+
+---
+
+## Cross-cutting abstracts to address
+
+The list below is a working backlog. Move items to "Decisions tracker"
+(at top) when actioned with a commit reference.
+
+### Universal wins (apply regardless of primary line)
+
+1. ✅ **Embed cache in Rust aibridge** — DONE 2026-05-01 (`150cc3b`).
+2. ✅ **FillValidator + EmailValidator in Go** — DONE 2026-05-01 (`b03521a`).
+3. ✅ **Drop Python sidecar from Rust** — DONE 2026-05-02 (`ba928b1`).
+4. **Cross-runtime contract tests** — Pin shared JSONL schemas (audit_baselines, scored_run, sft_sample) as canonical specs in `auditor/schemas/` with Go-side validators consuming the same definitions.
+
+### If keeping Go primary
+
+5. ✅ **Port materializer** — DONE 2026-05-02 (`cmd/materializer`).
+6. ✅ **Port replay tool** — DONE 2026-05-02 (`cmd/replay`).
+7. ✅ **Port `/v1/validate` + `/v1/iterate` HTTP surface** — DONE 2026-05-02 (`cmd/validatord`).
+8. **Skip Lance** until corpus growth demands it (>5M rows).
+9. **Keep chatd, observer fail-safe, role gate, multi-corpus matrix** — real Go wins worth preserving.
+
+### If keeping Rust primary
+
+10. **Port chatd's 5-provider dispatcher to Rust** — unified cloud LLM access.
+11. **Port the cross-role gate to Rust matrix retrieve** — production safety on the matrix layer (verified by Go reality tests real_001..005).
+12. **Consider process splitting** — even partial decomposition (split out vectord into its own process) would help with the mega-binary blast radius.
+
+---
+
+## Recommendation (working hypothesis)
+
+**Go for the primary line, Rust for production-bridge maintenance.**
+
+Reasons:
+1. **Operations** — process isolation is genuinely simpler at production scale than a 14.9G mega-binary.
+2. **Code volume** — Go does the same job in ~46% the lines.
+3. **Cross-runtime parity verified** — every artifact round-trips byte-equal between runtimes.
+4. **The 4 missing pieces are bounded** — materializer + replay + validators-network + RAG/preference exports are concrete porting targets, not research questions.
+5. **Performance is no longer a deciding factor** post-`150cc3b` — Rust is faster on warm cache, but both are well above staffing-domain demand levels (<1 RPS typical).
+
+But **don't abandon Rust**:
+1. devop.live/lakehouse/ runs through Rust today; cutover is multi-week.
+2. Several Go improvements would be downstream of Rust patterns. Keeping Rust live means anything new there is a porting opportunity for Go.
+3. The Python sidecar drop + cross-role gate port are valuable Rust improvements regardless of which line is primary.
+
+---
+
+## Change log
+
+Append entries here when this doc gets updated. One-line entries; link to commits.
+
+- 2026-05-01 — Initial draft (`b3ad148` golangLAKEHOUSE).
+- 2026-05-01 — Recorded Rust embed cache shipping (`150cc3b` lakehouse), updated Python-dependency section + table.
+- 2026-05-01 — Recorded Go validator port shipping (`b03521a` golangLAKEHOUSE), updated production-validators section.
+- 2026-05-01 — Reframed as living document in `docs/`, added Decisions tracker + Update guidance + Change log sections.
+- 2026-05-01 — Multi-tier 100k load test ran (335k scenarios @ 1,115/sec, 4/6 at 0% fail), surfaced coder/hnsw v0.6.1 nil-deref on small playbook_memory index. Recover guard added; real fix open.
+- 2026-05-01 (later) — coder/hnsw v0.6.1 panic real fix landed: vectord lifts source-of-truth out of coder/hnsw via `i.vectors` side store + recover wrappers + rebuild fallback. Re-run multitier 60s/conc=50: 0 failures across 19,622 scenarios. STATE_OF_PLAY invariant added to "DO NOT RELITIGATE".
+- 2026-05-02 — Substrate fix verified at original failure-surfacing scale. Multitier 5min @ conc=50: 132,211 scenarios at 438/sec, 6/6 classes at 0% failure (was 4/6 pre-fix). Throughput drop (1,115 → 438/sec) is the honest cost of the formerly-broken scenarios doing real HNSW Add work. STATE_OF_PLAY refreshed to 2026-05-02.
+- 2026-05-02 — Materializer + replay tool ported from Rust legacy to Go (`internal/materializer` + `internal/replay`, both with CLI + smoke + tests). Both runtimes now produce the same `data/evidence/YYYY/MM/DD/*.jsonl` and `data/_kb/replay_runs.jsonl` shapes; Go side no longer needs Bun for these phases.
+- 2026-05-02 — `/v1/validate` + `/v1/iterate` HTTP surface ported as `cmd/validatord` on `:3221`. Closes the last "If keeping Go primary" backlog item — Go now owns the entire validator path end-to-end (no Rust dep for staffing safety net). 5/5 smoke probes via gateway :3110.
+- 2026-05-02 — Python sidecar dropped from Rust hot path (`ba928b1`). AiClient now talks Ollama directly. Process count: mega-binary + sidecar → mega-binary alone. Public API unchanged, 0 callers updated. Cargo tests 32/32 PASS, live `/ai/embed` + `/v1/chat` smokes green on test gateway. Lab UI Python remains as ad-hoc dev tooling.
+
+---
+
+## See also
+
+- **`reports/cutover/architecture_comparison.md`** — historical snapshot (matched this doc as of the date stamp at top).
+- **`docs/SPEC.md`** — Go-side architectural spec.
+- **`docs/DECISIONS.md`** — Go-side ADRs.
+- **`/home/profit/lakehouse/docs/DECISIONS.md`** — Rust-side ADRs.
+- **`/home/profit/lakehouse/docs/go-rewrite/`** — Rust-side notes on the rewrite.
+- **`reports/cutover/SUMMARY.md`** — running log of cross-runtime parity probes.
+- **`reports/cutover/g5_load_test.md`** — load-test methodology + numbers.

docs/CLAUDE_REFACTOR_GUARDRAILS.md

@@ -0,0 +1,281 @@
+# Claude Refactor Guardrails — Go Lakehouse
+
+## Mission
+
+Continue the Go refactor without recreating the Rust-era complexity.
+
+The Go rewrite exists to make the lakehouse operationally legible:
+
+- small binaries
+- clear service boundaries
+- gateway-fronted APIs
+- smoke-testable behavior
+- fast build/run/debug loop
+- no accidental framework bureaucracy
+
+The Rust repo had maturity, but also accumulated control-plane weight: validator, auditor, provider routing, iteration loops, UI, truth layers, and agent-era scaffolding. Do not blindly port all of that back.
+
+## Prime Directive
+
+Preserve the Go spine:
+
+```text
+gateway
+  → storaged
+  → catalogd
+  → ingestd
+  → queryd
+  → vectord
+  → embedd
+Only add complexity when there is measured evidence, a failing smoke, or a documented feature-parity requirement.
+
+Refactor Rules
+1. No silent behavior changes
+
+Before changing any service behavior, identify:
+
+current route
+current request schema
+current response schema
+current status-code behavior
+current smoke test covering it
+
+If no smoke exists, add one before or with the refactor.
+
+2. Keep service boundaries hard
+
+Do not let services reach into each other’s internals.
+
+Allowed:
+
+HTTP client calls
+shared request/response structs when stable
+small internal packages for config/secrets/logging
+
+Avoid:
+
+importing another service’s implementation package
+hidden global state
+“just for now” shared mutable registries
+circular service knowledge
+3. Go is not Rust
+
+Do not imitate Rust patterns mechanically.
+
+Prefer Go-native clarity:
+
+simple structs
+explicit errors
+small interfaces at the consumer side
+context-aware HTTP handlers
+table-driven tests
+boring package names
+no abstraction tax unless repeated 3+ times
+4. Validation replaces the borrow checker
+
+Rust caught many problems at compile time. Go will not.
+
+Therefore every refactor must preserve or improve:
+
+input validation
+dimension checks
+duplicate handling
+restart persistence checks
+schema drift detection
+error status mapping
+smoke coverage
+5. Performance work must be measured
+
+Do not optimize by vibes.
+
+For each performance change, record:
+
+baseline command
+baseline result
+changed code path
+new result
+regression risk
+rollback plan
+
+Current known bottleneck:
+
+vectord Add is RWMutex-serialized.
+500K vectors: ~35m36s, ~234/sec avg.
+GPU around 65%, so embedding is not the only bottleneck.
+
+Do not claim concurrency improvements unless the HNSW library thread-safety is audited or writes are safely batched/sharded.
+
+File/Package Expectations
+cmd/
+
+One binary per service. Keep main files thin.
+
+Main should only:
+
+load config
+construct dependencies
+wire routes
+start server
+handle shutdown
+internal/
+
+Shared code belongs here only when it is genuinely shared.
+
+Good internal packages:
+
+config
+secrets
+storeclient
+catalogclient
+gateway routing helpers
+logging
+request/response contracts
+
+Bad internal packages:
+
+vague “utils”
+giant “common”
+cross-service god objects
+hidden dependency containers
+scripts/
+
+Every major behavior needs a runnable smoke.
+
+Smokes are not decoration. They are the replacement nervous system.
+
+Existing smoke pattern must remain:
+
+d1 skeleton/health/gateway
+d2 storaged
+d3 catalogd
+d4 ingestd
+d5 queryd
+d6 full ingest/query
+g1 vectord
+g1p vectord persistence
+g2 embed → vector add → search
+
+New functionality needs a new smoke or an extension to the closest existing one.
+
+Refactor Checklist
+
+Before editing:
+
+ Read README.md
+ Read docs/PRD.md
+ Read docs/SPEC.md
+ Read docs/DECISIONS.md
+ Read docs/PHASE_G0_KICKOFF.md
+ Identify affected services
+ Identify affected smokes
+
+During editing:
+
+ Keep public API stable unless explicitly changing it
+ Keep errors explicit
+ Keep logs useful but not noisy
+ Avoid package sprawl
+ Avoid premature generic interfaces
+ Preserve restart behavior
+ Preserve gateway-only acceptance path
+
+After editing:
+
+ Run go test ./...
+ Run relevant smoke script
+ Run full smoke loop when service contracts changed
+ Record evidence in a short refactor note
+
+Full smoke loop:
+
+for s in scripts/{d1,d2,d3,d4,d5,d6,g1,g1p,g2}_smoke.sh; do
+  "$s" || break
+done
+Refactor Note Format
+
+Create or update:
+
+docs/refactor-notes/YYYYMMDD-<short-name>.md
+
+Use this structure:
+
+# Refactor Note: <name>
+
+## Goal
+
+## Files changed
+
+## Behavior changed
+
+## Behavior preserved
+
+## Tests run
+
+## Smoke results
+
+## Performance before/after
+
+## Risks
+
+## Rollback
+Anti-Patterns To Reject
+Reject these unless specifically requested:
+
+porting Rust modules 1:1
+adding orchestration before service parity
+adding AI/agent logic inside core services
+making gateway business-aware
+hiding failures behind retries
+swallowing errors
+“temporary” global maps
+changing route contracts without smoke updates
+adding dependencies for trivial code
+optimizing vector ingestion without measurement
+rebuilding the Rust bureaucracy in Go clothing
+Preferred Next Targets
+
+Prioritize in this order:
+
+Stability of existing Go service contracts
+Better smoke coverage
+Persistence limits and large object handling
+vectord ingestion bottleneck analysis
+gateway observability
+feature parity with Rust only where needed
+UI/agent/auditor layers later, not now
+Architectural Position
+
+The Go rewrite should remain the production spine.
+
+The Rust system remains historical reference and possible source for:
+
+validation ideas
+audit semantics
+provider-routing lessons
+prior acceptance criteria
+edge cases
+
+But Rust is not the shape to copy.
+
+Go owns the clean operational path.
+Rust owns historical scar tissue and high-performance lessons.
+
+Do not confuse the two.
+
+
+Also add a shorter command prompt when you hand this to Claude Code:
+
+```md
+Read `docs/CLAUDE_REFACTOR_GUARDRAILS.md` first.
+
+Then inspect the current Go lakehouse repo and produce a refactor plan only. Do not edit code yet.
+
+Your plan must identify:
+1. affected services
+2. affected routes
+3. affected request/response contracts
+4. affected smoke scripts
+5. risks of accidentally reintroducing Rust-era complexity
+6. exact tests/smokes you will run after changes
+
+Do not port Rust structure blindly. Preserve the Go service spine.

docs/DECISIONS.md

@@ -121,6 +121,538 @@ historical record.
 
 ---
 
-(Future ADRs from ADR-002 onward will be added as the Go
-implementation accrues design decisions — e.g. HNSW parameter
-choices, pathway-memory hash function, auditor model rotation, etc.)
+## ADR-002: storaged per-prefix PUT cap (vectord _vectors/ → 4 GiB)
+**Date:** 2026-04-29
+**Decided by:** J
+**Status:** Implemented (commit `423a381`)
+
+`storaged` enforces a 256 MiB per-PUT body cap as DoS protection
+(`MaxBytesReader` + Content-Length check). Keys under `_vectors/`
+(vectord LHV1 persistence) get a raised cap of 4 GiB; everything
+else stays at 256 MiB.
+
+**Rationale:** the 500K staffing test surfaced that single-file LHV1
+above ~150K vectors at d=768 hits the 256 MiB cap. `manager.Uploader`
+already streams on the outbound side, so the cap is a safety gate
+not a memory bottleneck — raising it for the vector path doesn't
+introduce new memory pressure. Per-prefix preserves the safety
+gate for routine traffic while opening the documented production
+path. Splitting LHV1 across multiple keys was rejected because G1P
+specifically shipped the single-Put framed format to eliminate
+torn-write — multi-key would re-introduce that failure mode.
+
+**Follow-up:** if production workloads exceed 4 GiB single-file
+LHV1, refactor to operator-driven config (env/TOML) rather than
+bumping the constant. The function-level `maxPutBytesFor(key)` in
+`cmd/storaged/main.go` keeps that drop-in clean.
+
+---
+
+## ADR-003: Inter-service auth posture — Bearer token + IP allowlist
+**Date:** 2026-04-29
+**Decided by:** J + Claude
+**Status:** Decided — wiring deferred to Sprint 1
+
+**Decision:** When inter-service auth is needed (the moment any
+binary binds non-loopback or the deployment crosses a trust
+boundary), the auth model is **a Bearer token loaded from
+`secrets-go.toml` plus a configurable IP allowlist**. Both layers
+required: the token authenticates the caller; the allowlist
+narrows the network surface.
+
+**Status today (G0):** zero auth middleware. Every binary binds
+`127.0.0.1` by default; commit `6af0520` (R-001 partial fix) refuses
+non-loopback bind unless the per-service `LH_<SVC>_ALLOW_NONLOOPBACK=1`
+env override is set. The override-and-no-auth combination is the
+worst case — this ADR locks in what we'll require before any
+production override fires.
+
+### What gets implemented when auth lands
+
+1. **`secrets-go.toml` adds a `[auth]` section:**
+   ```toml
+   [auth]
+   token = "..."          # 32+ random bytes, hex-encoded
+   allowed_ips = ["10.0.0.0/8", "127.0.0.1/32"]   # CIDR list
+   ```
+
+2. **`internal/shared/auth.go`** ships a single chi middleware:
+   ```go
+   func RequireAuth(cfg AuthConfig) func(http.Handler) http.Handler
+   ```
+   - Empty `cfg.Token` → middleware is a no-op (G0 dev mode).
+   - Non-empty token → reject 401 unless request has
+     `Authorization: Bearer <token>` matching constant-time.
+   - Non-empty `allowed_ips` → reject 403 unless `r.RemoteAddr` (or
+     `X-Forwarded-For` first hop, configurable) is in CIDR set.
+   - `/health` exempt — load balancers + monitors need it open.
+
+3. **Every `cmd/<svc>/main.go` adds one line:**
+   ```go
+   r.Use(shared.RequireAuth(cfg.Auth))
+   ```
+   Mounted before `register(r)` so it covers every route the binary
+   exposes after `/health`.
+
+4. **`shared.Run` startup gate:** if bind is non-loopback AND
+   `cfg.Auth.Token == ""`, refuse to start. The implicit
+   "localhost is the auth layer" guarantee becomes explicit when
+   crossing the loopback boundary.
+
+### Alternatives considered
+
+| Option | Why rejected |
+|---|---|
+| **mTLS** | Strongest but heaviest — every binary needs cert provisioning, rotation tooling, and cert-aware client wiring. Overkill for inter-service traffic that already passes through a single gateway. Reconsider when Lakehouse-Go runs across machines. |
+| **JWT with short TTL** | Buys nothing over Bearer here — there's no third-party identity provider, no claim hierarchy worth modelling. Pure token has the same security properties at half the wire complexity. |
+| **No auth, IP-allowlist only** | One stolen IP allowlist entry → full access. Token + IP is defense in depth; either alone is too weak. |
+| **OAuth2 via external IdP** | Rejected for G0–G3 timeline. No external IdP commitment. Revisit if Lakehouse-Go ever serves end-user requests directly (today everything fronts through the staffing co-pilot which has its own session model). |
+
+### Constant-time comparison + token hygiene
+
+Token comparison must use `crypto/subtle.ConstantTimeCompare` —
+naive `==` is vulnerable to timing attacks against an attacker who
+can issue many requests and measure round-trip. Token rotation is
+operator-driven via `secrets-go.toml` edit + restart; G0 doesn't
+need rotate-without-restart.
+
+### What this ADR does NOT do
+
+- **Does not implement the middleware.** Code lands in Sprint 1.
+- **Does not require token in G0 dev.** Empty token → no-op. Smokes
+  + proof harness keep working without setting tokens.
+- **Does not address gateway → end-user auth.** Gateway terminates
+  inter-service auth at its inbound; if end-users hit gateway from
+  a browser, that's a different ADR (likely cookie/session, fronted
+  by a reverse proxy that handles user auth).
+
+### How this closes audit findings
+
+- **R-001 (queryd /sql RCE-equivalent off-loopback):** the bind
+  gate prevents accidental exposure today; this ADR specifies the
+  guardrail when intentional exposure is needed.
+- **R-007 (zero auth middleware):** answered by the design above;
+  R-007 stays open until the middleware is implemented but is no
+  longer "design TBD."
+- **R-010 (no CORS posture):** orthogonal to inter-service auth,
+  but the `RequireAuth` middleware sits at the right layer to add
+  CORS handling later (browsers don't reach inter-service routes
+  in the current design, so CORS is also Sprint 1+ when end-user
+  requests start landing).
+
+---
+
+## ADR-004: Pathway memory data model — Mem0-style versioned traces
+**Date:** 2026-04-29
+**Decided by:** J + Claude
+**Status:** Decided — substrate landing in `internal/pathway/`
+
+**Decision:** Pathway memory is an append-only event log of opaque
+traces with Mem0-style semantics: Add / Update / Revise / Retire /
+History / Search. Each trace has a UID; revisions chain backward
+via `predecessor_uid` so the full history is reconstructible.
+Persistence is JSONL append-only with full-replay on load;
+corruption recovery skips bad lines without halting startup.
+
+### Operations
+
+| Op | Effect |
+|---|---|
+| `Add(content, tags...)` | New UID, stored fresh, replay_count=1. |
+| `AddIdempotent(uid, content, tags...)` | If UID exists → replay_count++. Else → Add with that UID. |
+| `Update(uid, content)` | In-place content replacement (same UID). Bumps `updated_at_ns`. NOT a revision — same trace, new content. |
+| `Revise(predecessorUID, content, tags...)` | New UID with `predecessor_uid` set. Old trace stays accessible via History. Failure modes: predecessor missing → error; predecessor retired → still allowed (revisions of retired traces are valid). |
+| `Retire(uid)` | Sets `retired=true`. Excluded from `Search` by default; still accessible via `Get` and `History`. |
+| `Get(uid)` | Returns the trace (including if retired); error on missing. |
+| `History(uid)` | Walks `predecessor_uid` chain backward, returns slice [self, parent, grandparent, ...]. Cycle-detected via visited-set; returns error on cycle (which only happens if persistence file was hand-edited). |
+| `Search(filter)` | Returns matching traces. Default excludes retired; opt in via `IncludeRetired: true`. Filters: tag-match, content-substring, time range. |
+
+### Why Mem0-style + Why these specific ops
+
+- **Mem0** (memory pattern from the OpenAI Memories paper / Mem0 lib)
+  is the canonical "agent memory" interface for the same reason
+  Markdown is the canonical text format: it's the lowest-common-
+  denominator that the entire ecosystem assumes. Adopting it lets
+  agent loops written against any Mem0-aware substrate work here.
+- Update vs Revise are deliberately separate. Update is "I noticed
+  a typo in my note." Revise is "I now believe something different
+  than I did when I wrote this; preserve the old belief for audit."
+  Conflating them loses the audit trail.
+- Retire vs Delete is deliberate. Retire stops a trace from
+  surfacing in search but preserves it for history reconstruction.
+  Delete (which we don't expose) would break references.
+
+### Trace data shape
+
+```go
+type Trace struct {
+    UID            string          // UUID v4 unless caller provides one
+    Content        json.RawMessage // opaque, schema is caller's contract
+    PredecessorUID string          // empty if root revision
+    CreatedAtNs    int64
+    UpdatedAtNs    int64
+    Retired        bool
+    ReplayCount    int             // ≥1 for any stored trace
+    Tags           []string        // for Search
+}
+```
+
+`Content` is opaque JSON (not a struct) so callers can store any
+shape — the data model doesn't constrain semantics. Callers add
+their own validators on top.
+
+### Persistence
+
+JSONL append-only log under `_pathway/<store_name>.jsonl`. Each
+mutation appends one JSON line:
+
+```
+{"op":"add",     "trace":{...}}
+{"op":"update",  "uid":"…",   "content":"…"}
+{"op":"revise",  "trace":{…}}    # trace.PredecessorUID is set
+{"op":"retire",  "uid":"…"}
+{"op":"replay",  "uid":"…"}     # idempotent re-add hit
+```
+
+On startup, replay every line in order, building in-memory state.
+A malformed line logs a warn and is skipped; load continues.
+Corruption tolerance is non-optional — partial state is better
+than no state for an agent substrate.
+
+Compaction is a future concern. A 100K-trace log replays in
+seconds; below that scale, JSONL append is the simplest correct
+choice. When compaction lands, the format will be: snapshot file
+(full state JSON) + tail JSONL since snapshot. Detect snapshot,
+load it, then replay tail.
+
+### Cycle safety
+
+UIDs are generated server-side via `uuid.New()` (existing dep —
+catalogd uses it). New UID for every Add and Revise. The data
+model itself can't form cycles — every Revise points at an
+EXISTING uid, and the new uid didn't exist a moment ago.
+
+History walks defensively anyway: visited-set tracks UIDs seen
+this walk; if we encounter a duplicate, return error. Protects
+against corruption (manual edit, bug in a future op) without
+constraining the happy path.
+
+### Storage location
+
+JSONL file path is configurable per store. Default:
+`/var/lib/lakehouse/pathway/<name>.jsonl` for prod; tests use
+`t.TempDir()`. Persistence is OPTIONAL — empty path means
+in-memory only (matches vectord G1's pattern).
+
+### What this ADR does NOT do
+
+- **No HTTP surface decision.** Whether `cmd/pathwayd` is its own
+  binary or routes get added to `cmd/vectord` is the next ADR's
+  concern. The substrate is a pure library either way.
+- **No vector index integration.** Pathway traces can carry a
+  vector embedding in `Content` (caller decides), but this ADR
+  doesn't define how the substrate integrates with `vectord`'s
+  HNSW indexes. That's the staffing co-pilot's design problem
+  when those layers compose.
+- **No agent-loop semantics.** "When does an agent ADD vs
+  REVISE?" is a workflow decision, not a substrate decision.
+
+---
+
+## ADR-005: Observer fail-safe semantics
+
+**Date:** 2026-04-30
+**Status:** RATIFIED
+**Scope:** `internal/observer` (Store, Persistor) + `internal/workflow` (Runner) + `cmd/observerd`
+
+The Rust legacy had a documented "verdict:accept on crash" anti-pattern:
+when the observer crashed mid-evaluation, the upstream interpreted the
+missing verdict as implicit acceptance. Several silent regressions traced
+to it. The Go observer's role is structurally different — it is a
+**witness** (records what happened) rather than a **gate** (decides
+accept/reject) — but adjacent fail-safe decisions still need locking
+now that observerd is on the prod-realistic stack via the lift harness
+(commit `b2e45f7`, 2026-04-30). This ADR ratifies the current behavior
+and locks the rationale so future consumers don't break the invariant
+by flipping the defaults.
+
+### Decision 5.1 — Persist failure is logged-not-fatal; ring is the in-flight source of truth
+
+Already implemented (`internal/observer/store.go:60-67`). Locked:
+
+- If `persistor.Append` fails, log a warning and continue. Do NOT
+  return an error to the caller of `Store.Record`.
+- The in-memory ring buffer is the source of truth in flight; the
+  JSONL is a best-effort durability shadow.
+- Operators who need fail-closed audit-grade trails configure that
+  mode through a future opt-in (deferred to a later ADR; not the
+  G0/G1/G2 default).
+
+**Why fail-open here:** the observer's job is to keep recording even
+when the disk hiccups. A `persist-fail-fatal` mode would translate
+every transient I/O blip into an observer-blackout, which is strictly
+worse for the witness role than missing a few persisted entries — the
+ring still has them, and operators can drain it on restart.
+
+**Why this isn't the Rust anti-pattern:** the Go observer doesn't
+emit verdicts. A persist failure here means "we recorded fewer rows
+on disk than in memory," not "we accepted something we shouldn't have."
+
+### Decision 5.2 — Mode failure in workflow.Runner: `Success = (Error == "")`, no panic-swallow path
+
+Already implemented (`internal/workflow/runner.go`). Locked:
+
+- Mode errors are caught by the runner and surfaced via the node's
+  `Error` field; `Success` is the boolean derived from `Error == ""`.
+- `observerd` records an `ObservedOp` per node with `Success: false`
+  and the error string when a mode fails.
+- Cycles, missing-deps, and unknown modes are aborting errors → 4xx
+  from `/observer/workflow/run` with the failure encoded in the JSON
+  response.
+
+**Why this is the explicit anti-Rust:** allowing a mode to silently
+swallow its panic and report `Success: true` is exactly how the Rust
+"verdict:accept on crash" pattern manifests. Forcing the runner to
+record `Success: false` on error makes the failure observable to
+downstream consumers (observerd queries, scrum review, distillation
+selection) instead of laundering it into a fake success.
+
+### Decision 5.3 — Provenance is one-row-per-node, recorded post-run
+
+Already implemented (`cmd/observerd/main.go:140-154`). Locked:
+
+- `runner.Run` returns the full `RunResult` with per-node Success/Error;
+  `handleWorkflowRun` then iterates `res.Nodes` and `store.Record`s an
+  `ObservedOp` per node.
+- One row per node, NOT a single per-workflow catch-all. A workflow with
+  N nodes produces N audit rows.
+- Crash semantics:
+  - Crash *during* `runner.Run` → no provenance recorded; queries see
+    absence, not a false acceptance.
+  - Crash *during* the recording loop → some nodes recorded, some
+    absent; queries see partial provenance, again not a false
+    acceptance.
+- Recovery: re-run the whole workflow. No incremental resume in G0/G1/G2.
+
+**Why one row per node:** debugging a partial workflow is a one-grep
+operation when each node has its own row. A single catch-all row would
+be exactly the Rust anti-pattern surface — "we accepted this workflow"
+records that survive partial crashes look identical to genuine
+acceptances. Per-node-row makes that structurally impossible.
+
+**Known gap, not yet a follow-up ADR:** recording happens after
+`runner.Run` returns, not as each node completes. A long workflow with
+late-stage failure currently records nodes that already finished only
+once the runner returns. For G0/G1/G2 substrate this is fine —
+workflows are short. When workflows get long enough that mid-run
+visibility matters, a streaming-record callback is the right shape.
+
+### Decision 5.4 — `/observer/event` accepts even when the ring is full
+
+Already implemented via `Store.Record`'s shift-left eviction. Locked:
+
+- Ring overflow is normal operation: oldest evicted, newest accepted.
+- 200 OK from `/observer/event` means "we accepted into the ring"; it
+  does NOT promise "we persisted." Persistence remains best-effort
+  per Decision 5.1.
+- 4xx is reserved for malformed `ObservedOp` payloads (validation
+  failures).
+
+**Why accept-on-full:** treating a full ring as a 503 would translate
+every brief activity burst into client errors, which is exactly the
+wrong direction for an audit witness — the witness's job is to never
+refuse to write, only to lose oldest data when capacity binds.
+
+### Alternatives considered
+
+- **Persist-required mode** — caller-configurable fail-closed for
+  audit-grade workloads. The right approach when this lands is an
+  opt-in on `Store` construction, leaving the default fail-open.
+  Deferred to a future ADR.
+- **Distributed ring with WAL** — persist before accept-into-ring,
+  sync semantics. Too heavy for G0/G1 and breaks the ring's "in-flight
+  source of truth" property.
+- **Mode-result schema with explicit verdict field** — would force
+  every mode to declare accept/reject. Overengineered for the witness
+  role and reintroduces the gate-vs-witness confusion this ADR is
+  trying to avoid.
+
+### What this ADR does NOT do
+
+- **No retention policy.** "How long do we keep observer entries on
+  disk?" is a separate operations decision.
+- **No mode-level retry.** If a mode fails, the runner records that
+  and moves on. Whether to retry is a workflow-definition concern
+  (Archon-style retry policies in the YAML), not the runner's.
+- **No cross-process recovery.** A crashed observerd loses the ring;
+  the persistor preserves what it managed to write. Operators read the
+  JSONL after restart, not query a dead daemon.
+- **No persist-required opt-in.** Mentioned in alternatives; lands in
+  a separate ADR when an audit-grade consumer requires it.
+
+### How this closes the OPEN list
+
+STATE_OF_PLAY listed ADR-005 as a doc-only gate before observer wired
+into production paths. The 2026-04-30 lift run wired observerd into the
+prod-realistic harness boot, which means observer is now on the data
+path for every reality test workflow. This ADR locks the fail-safe
+invariants before the next consumer (scrum runner, distillation rebuild,
+or a real production workflow) takes a hard behavioral dependency.
+
+---
+
+## ADR-006: Auth posture for non-loopback deploy
+
+**Date:** 2026-04-30
+**Status:** RATIFIED
+**Scope:** `internal/shared/auth.go` + `internal/shared/bind.go` + every `cmd/<bin>/main.go`'s `shared.Run` call site
+
+ADR-003 locked the substrate (Bearer token + IP allowlist, opt-in via
+`cfg.Auth.Token`/`cfg.Auth.AllowedIPs`, `/health` exempt). ADR-006
+ratifies the **operator playbook + deploy-time invariants** — what
+gets enforced when, what operators set where, what happens when keys
+rotate. Required because Sprint 4 deployment work (REPLICATION.md,
+systemd units, Dockerfile) needs a locked auth posture before it
+touches production-shaped configs.
+
+### Decision 6.1 — Non-loopback bind requires `auth.token`; the gate is mechanical
+
+Already implemented in `requireAuthOnNonLoopback` (`internal/shared/bind.go:58-67`).
+Locked:
+
+- Any binary that binds anything other than `127.0.0.0/8` / `::1` /
+  `localhost` MUST have `cfg.Auth.Token != ""`. Empty-token +
+  non-loopback-bind = startup error, not silent insecure mode.
+- The check fires in `shared.Run` BEFORE `http.Server.Serve`, so a
+  misconfigured binary fails fast at startup rather than serving
+  one request.
+- Pairs with `requireLoopbackOrOverride`: that gate refuses any
+  non-loopback bind without `LH_<NAME>_ALLOW_NONLOOPBACK=1`. Together
+  they make the audit's R-001+R-007 worst case (queryd `/sql` =
+  RCE-equivalent off-loopback with no auth) mechanically impossible.
+
+**Why mechanical, not policy:** policy gates rely on operator
+discipline. The substrate gates work even when an operator copies a
+dev `lakehouse.toml` into prod and forgets to set the token —
+binary refuses to start, error message names the env override.
+
+### Decision 6.2 — Token comes from `cfg.Auth.Token` populated by env or secrets file
+
+Locked:
+
+- Operators do NOT put the production token in `lakehouse.toml`
+  directly. The TOML field is empty in the committed file; the
+  daemon's systemd unit sets `AUTH_TOKEN` (or whatever
+  `cfg.Auth.TokenEnv` names) via `EnvironmentFile=` pointing at
+  `/etc/lakehouse/auth.env` (mode 0600, root-owned).
+- Same pattern as `chatd`'s provider keys (`OPENROUTER_API_KEY` etc.):
+  TOML names the env var, systemd loads the env file.
+- Justification: keeps secrets out of git + out of the running
+  process's command line + audit-able via filesystem ACLs.
+
+### Decision 6.3 — `AllowedIPs` is the inter-service gate; `Token` is the cross-trust-boundary gate
+
+Locked:
+
+- Same-box deploys (10 daemons all on one host, all on `127.0.0.0/8`
+  or a private subnet) use **`AllowedIPs` only**. Each daemon's
+  `cfg.Auth.AllowedIPs` lists the gateway's address (and any other
+  daemon that legitimately calls it). No token shared between
+  internal services.
+- Gateway-to-external traffic (a coordinator UI in another VPC,
+  a user's browser, an external integrator) goes through
+  **Bearer token**. The token is per-tenant; rotation is per-tenant.
+- Mixed: a service can require BOTH (allowlist AND token) — the
+  middleware logic is `allowed = ip_allowed && token_valid` when
+  both are set. Use this for the gateway when binding non-loopback.
+
+**Why split:** token rotation is operationally expensive (every
+caller updates a secret). IP allowlist rotation is free if the
+network topology is stable. Splitting them by trust boundary lets
+internal services treat allowlist drift as a network change while
+external callers handle token rotation as a credential change.
+
+### Decision 6.4 — `/health` is unauthenticated; everything else under `shared.Run` is gated
+
+Already implemented (`internal/shared/server.go:84-92`). Locked:
+
+- Load balancers + monitor probes hit `/health` without a token.
+  The route returns `{"status":"ok","service":"<name>"}` and nothing
+  about service state — no version, no commit, no internal counts.
+- Every other route registered via `shared.Run`'s `register`
+  callback lives inside the auth-gated chi.Group. New routes
+  inherit auth automatically; new daemons inherit it via `shared.Run`.
+- A daemon that needs a public route MUST add it to the outer router
+  before the `register` group, with a code comment explaining the
+  exemption. There are no others today.
+
+### Decision 6.5 — Token rotation is operator-staged; old + new accepted during the window
+
+Not yet implemented; locked as a Sprint 4 follow-up:
+
+- Operators stage a rotation by adding a second token to
+  `cfg.Auth.SecondaryTokens []string`. Both primary and secondary
+  pass auth during the window.
+- After every caller is updated to the new token, operators
+  promote secondary → primary and clear secondary. A second
+  rotation can begin.
+- Rolling restart not required; daemons reload `cfg.Auth` on
+  SIGHUP (also a Sprint 4 follow-up — currently they re-read on
+  restart only).
+
+**Why dual-token instead of just single-rotation:** caller pool can be
+large (gateway + observerd + scrum runner + UI + external integrators).
+A single-token rotation forces a flag-day. Dual-token windows let
+operators rotate gradually and abort on failure.
+
+### Decision 6.6 — TLS is the network operator's job, not ours
+
+Locked:
+
+- Daemons speak HTTP, not HTTPS. TLS termination happens at the
+  network edge (nginx / Caddy / cloud LB), not in the Go process.
+- Internal daemon-to-daemon traffic stays on plaintext HTTP because
+  it's all on `127.0.0.0/8` or a private subnet (per Decision 6.3).
+- Justification: TLS in-process means cert management, rotation,
+  reload — all undifferentiated lift that nginx already solves
+  better. The Bearer token + allowlist gates are sufficient when
+  combined with a TLS-terminating reverse proxy.
+
+### Alternatives considered
+
+- **mTLS for inter-service auth** — every daemon issues + verifies
+  certs. Solves token-rotation pain but adds cert lifecycle as a
+  problem. Allowlist + plaintext on the private network is cheaper
+  and gets the same threat-model coverage.
+- **JWT-only** — JWTs let callers carry richer claims (tenant id,
+  expiry, scopes). Overkill for the current threat model; the
+  Bearer token + allowlist split is honest about what each layer
+  actually defends against. Revisit when multi-tenant gateway
+  features land.
+- **No auth, network is the boundary** — works for G0 dev and the
+  current single-box deployment. ADR-006 explicitly does NOT
+  recommend this for non-loopback prod (the mechanical gate
+  refuses it).
+
+### What this ADR does NOT do
+
+- **Does not specify how the gateway authenticates external callers.**
+  Token-vs-mTLS-vs-OAuth at the public edge is a separate decision
+  driven by who-calls-us. ADR-006 is about the inter-service +
+  same-trust-domain posture.
+- **Does not implement token rotation hot-reload.** Decision 6.5
+  documents the design; the implementation is Sprint 4 work.
+- **Does not lock TLS termination details.** Where + how nginx/Caddy
+  goes is ops infrastructure, not ADR territory.
+
+### How this closes the OPEN list
+
+STATE_OF_PLAY listed ADR-006 as the gate before any Go binary binds
+non-loopback in prod. The substrate gates were already present (R-001
++ R-007 enforced via `requireLoopbackOrOverride` +
+`requireAuthOnNonLoopback`); this ADR locks the operator playbook
+that turns those gates into a deployable posture. Sprint 4 can now
+write systemd units that set `AUTH_TOKEN` from `EnvironmentFile=`
+without re-litigating the design.
+
+---

docs/PRD.md

@@ -9,6 +9,61 @@ estimates, library choices, and acceptance gates.
 
 ---
 
+## Product vision — what we're actually building
+
+**The Go refactor isn't the goal. The goal is a small-model-driven autonomous pipeline that gets better with each run, with frontier models in audit/oversight and humans triaged in only for the genuinely abstract cases.**
+
+The Rust Lakehouse already has most of the pieces:
+- **Pathway memory** (`internal/pathway` in Go, 88 Rust traces preserved) — what we tried, what worked
+- **Matrix indexer** (SPEC §3.4) — multi-corpus retrieve+merge that gives the small model the right knowledge slice for *this* task
+- **Observer** — watches runs, refines configs, escalates
+- **Distillation v1.0.0** (`e7636f2`) — turns successful runs into denser playbooks
+- **Auditor cross-lineage fabric** — Kimi/Haiku/Opus oversight on small-model outputs
+
+What the Go refactor is FOR: a second-language pass surfaces architectural weaknesses that Rust hid. The pipeline has to pull together cleanly *as a pipeline* — not as 15 crates that happen to interact.
+
+### The five-loop substrate
+
+1. **Knowledge pathway loop** — pathway memory + matrix indexer give the small model context for the task. Pathway answers "what worked last time?"; matrix answers "what's relevant now?"
+2. **Execution loop** — small model runs on focused context. Frontier API calls are reserved for audit/escalation, not the inner loop. Cost + rate limits stay sane.
+3. **Observer loop** — watches each run, refines the configs (matrix corpus picks, downgrade gate, prompt mold) that got the model to a good pathway. Outputs new config, not new prompt.
+4. **Rating + distillation loop** — successful outcomes get scored and folded back into the playbook substrate. The playbook gets denser; the next run starts smarter.
+5. **Drift loop** — quantify when the distilled playbook stops matching reality (codebase changed, contracts shifted, profiles updated). Drift is a *measured* signal, not "hope nothing broke."
+
+### The gate
+
+**The playbook + matrix indexer must produce the results we're looking for.** That's the single load-bearing acceptance criterion. Throughput, scaling, code elegance — all secondary. If a deep-field reality test on the 500K corpus surfaces wrong answers, the loop isn't working and we fix that before adding anything else.
+
+### Observer as system resource (clarified 2026-04-29)
+
+The observer is not a service among services — it's a *system
+resource*. Its job is to be objective about the process: watch
+everything, record measurements, surface what worked vs what
+didn't, feed the KB so the playbook substrate can decide the
+right pathway to the correct outcome.
+
+The bare-bones observerd shipped in `bc9ab93` (event ingest +
+stats) is the substrate for this. The architectural pattern
+that grows it into the full "objective measurement engine" is
+the **multi-pass workflow runner** documented in SPEC §3.8 —
+inspired by Archon (`/home/profit/external/Archon`) and proven
+in the Rust `observer-kb` branch's Python prototypes (`deep_analysis.py`,
+`extract_knowledge.py`, `process_knowledge.py`).
+
+The pipeline mode-chain (extract → validator → hallucination →
+consensus → redteam → pipeline → render) IS how the observer
+makes actionable decisions: each mode pass is a deterministic
+measurement; what survives the gauntlet is what feeds the KB.
+
+### Triage / human-in-loop
+
+Most cases are abstract enough that small-model + pathway + matrix can complete them. Some can't — they need a human. The system's job is to **identify which is which** and only escalate the second class. Frontier models partially solve this internally with their thinking loops; we're externalizing it so:
+- Small models are swappable (vendor independence)
+- Drift is measurable (quantitative signal, not vibes)
+- Each loop iteration is auditable (the pathway memory IS the audit trail)
+
+This is what the auditor cross-lineage fabric proves out in Rust — Opus auto-promote on diffs >100k chars is the same pattern: triage by signal, not by guesswork.
+
 ## Direction pivot — why this PRD exists
 
 The Rust-first Lakehouse (15 crates, ~24 unmerged commits past PR #11,

docs/SCRUM.md

@@ -0,0 +1,214 @@
+# Scrum Test: Matrix Agent Validated Hardening Sprint
+
+## Mission
+
+Run a Scrum-style technical validation against this repository:
+
+https://git.agentview.dev/profit/matrix-agent-validated.git
+
+Do not add features first. Treat the codebase as a validated prototype that now needs production-hardening pressure.
+
+The goal is to produce a hard evidence report and a prioritized sprint backlog.
+
+## Core Questions
+
+1. Can the repo be cloned, built, and smoke-tested from a clean environment?
+2. Are the claimed validated paths actually covered by repeatable tests?
+3. Where does the system rely on demo assumptions, hardcoded paths, permissive fallbacks, or unsafe string construction?
+4. Which failures would corrupt trust in the agent loop?
+5. What must be fixed before this becomes a reusable agent-memory substrate?
+
+## Required Inspection Areas
+
+### 1. Build and Test Surface
+
+Inspect:
+
+- Cargo workspace
+- Rust crates
+- Bun/TypeScript MCP server
+- Python sidecar
+- tests/
+- justfile
+- REPLICATION.md
+- systemd units
+- scripts/
+
+Run or prepare the following commands where possible:
+
+```bash
+just --list
+cargo check --workspace
+cargo test --workspace
+cd mcp-server && bun install && bun test || true
+bun run tests/agent_test/agent_harness.ts || true
+
+If heavy data or external services are missing, do not fake success. Record the blocker and define a mock/minimal fixture path.
+
+2. Security and Trust Boundary Review
+
+Search for:
+
+raw SQL interpolation
+shell command execution
+open CORS
+unauthenticated mutation endpoints
+pass-through proxy routes
+hardcoded absolute paths
+secrets in repo
+fail-open review behavior
+unbounded file reads/writes
+unsafe JSON parsing assumptions
+
+Pay special attention to:
+
+mcp-server/index.ts
+mcp-server/observer.ts
+crates/vectord/src/pathway_memory.rs
+crates/vectord/src/playbook_memory.rs
+scripts/
+sidecar/
+3. Agent Validation Review
+
+Verify whether the following claims are actually enforced by tests:
+
+vector retrieval across corpora
+observer hand-review gates candidates
+successful playbooks are sealed
+retrieval surfaces prior playbooks on later runs
+Mem0-style ADD / UPDATE / REVISE / RETIRE / HISTORY behavior works
+retired traces are excluded from retrieval
+history chains are cycle-safe
+agent claims can be verified against SQL truth
+cloud-only adaptation works without local Ollama
+
+Create a table:
+
+Claim	Code Location	Existing Test	Missing Test	Risk
+4. Scrum Backlog Output
+
+Create a prioritized backlog using this format:
+
+Sprint 0 — Reproducibility Gate
+
+Goal: make the repo provably runnable.
+
+Stories:
+
+As an operator, I can run one command and know which dependencies are missing.
+As an operator, I can run a minimal fixture test without the 470MB data payload.
+As an operator, I can verify gateway, sidecar, observer, and MCP health with one command.
+
+Acceptance:
+
+just verify exists.
+just smoke runs without large datasets.
+failure output is structured JSON.
+no test claims success when dependencies are missing.
+Sprint 1 — Trust Boundary Gate
+
+Goal: prevent agent trust collapse.
+
+Stories:
+
+Replace raw SQL string interpolation with validated query builders or parameterized calls.
+Change observer /review failure from fail-open accept to explicit degraded/cycle verdict.
+Add auth or localhost-only guardrails for mutation endpoints.
+Add schema validation for every public endpoint.
+
+Acceptance:
+
+SQL injection tests fail before fix and pass after fix.
+observer crash cannot auto-accept unsafe candidate output.
+mutation endpoints require configured token or local-only mode.
+Sprint 2 — Memory Correctness Gate
+
+Goal: prove Mem0/pathway memory cannot poison itself.
+
+Stories:
+
+Add tests for ADD, UPDATE, REVISE, RETIRE, HISTORY.
+Add cycle detection tests.
+Add retired-trace retrieval exclusion tests.
+Add duplicate trace replay_count tests.
+Add corrupted memory row recovery test.
+
+Acceptance:
+
+deterministic fixture corpus
+all memory operations covered
+every memory mutation emits audit/event receipt
+Sprint 3 — Agent Loop Reality Gate
+
+Goal: prove the agent loop works across actual workflows.
+
+Stories:
+
+Build deterministic mini corpus.
+Run search → verify → observer review → playbook seal → second-run retrieval.
+Add negative case where observer rejects hallucinated claim.
+Add regression for health endpoint content-type mismatch.
+
+Acceptance:
+
+single command proves the full loop
+generated report includes input hash, output hash, verdict, and memory mutation receipt
+Sprint 4 — Deployment Gate
+
+Goal: turn REPLICATION.md into executable deployment validation.
+
+Stories:
+
+Convert REPLICATION.md validation section into scripts.
+Add env var template.
+Add config validation.
+Remove hardcoded /home/profit/lakehouse paths.
+Add systemd readiness checks.
+
+Acceptance:
+
+fresh clone can run just doctor
+missing env vars are reported clearly
+no absolute path assumptions remain unless configured
+Required Final Deliverables
+
+Create:
+
+reports/scrum/matrix-agent-scrum-test.md
+reports/scrum/risk-register.md
+reports/scrum/claim-coverage-table.md
+reports/scrum/sprint-backlog.md
+reports/scrum/acceptance-gates.md
+
+Do not rewrite the system yet.
+
+First produce the reports only.
+
+Scoring Model
+
+Use this scoring:
+
+Reproducibility: 0–10
+Test Coverage: 0–10
+Trust Boundary Safety: 0–10
+Agent Memory Correctness: 0–10
+Deployment Readiness: 0–10
+Maintainability: 0–10
+
+Mark each score with evidence.
+
+Final Rule
+
+No vibes. No “appears to work.” Every claim must point to:
+
+file path
+line/function
+command output
+test result
+missing evidence
+
+That’s the move: **don’t refactor yet. Put the repo under oath first.**
+::contentReference[oaicite:5]{index=5}
+
+
+

docs/SESSION_LOG.md

@@ -0,0 +1,158 @@
+# Coordinator session log — `coordinator_sessions.jsonl`
+
+**Last updated:** 2026-05-02 · **Schema:** `session.iterate.v1` · **Writer:** `internal/validator.SessionLogger` · **Producer:** validatord `/v1/iterate`
+
+## Why
+
+The Langfuse trace tree is the live view: per-session, you can scroll
+the retry chain and inspect every sub-call. But for **longitudinal
+forensics** ("show me every session in the last week where the model
+guessed a real worker without a retry," or "find sessions where
+validation rejected three times in a row"), Langfuse's UI doesn't
+scale — you need a queryable data plane.
+
+This JSONL is that plane. One row per `/v1/iterate` session, append-
+only, DuckDB-friendly.
+
+## Where
+
+Configurable via `[validatord].session_log_path` in `lakehouse.toml`.
+Empty = disabled (best-effort posture; a missing log never blocks an
+iterate request). Production:
+
+```toml
+[validatord]
+session_log_path = "/var/lib/lakehouse/validator/sessions.jsonl"
+```
+
+## Schema (v1)
+
+```jsonc
+{
+  "schema":          "session.iterate.v1",
+  "session_id":      "<Langfuse trace_id>",      // join key to Langfuse
+  "timestamp":       "2026-05-02T07:30:00.123456Z",
+  "daemon":          "validatord",
+  "kind":            "fill | email | playbook",
+  "model":           "qwen3.5:latest",
+  "provider":        "ollama",
+  "prompt":          "produce a fill artifact ...",  // truncated to 4000 chars
+  "iterations":      3,                              // attempts spent
+  "max_iterations":  3,                              // cap per request
+  "final_verdict":   "accepted | max_iter_exhausted | infra_error",
+  "attempts": [
+    { "iteration": 0, "verdict_kind": "validation_failed",
+      "error": "consistency: candidate_id W-X not in roster",
+      "span_id": "abc..." },
+    { "iteration": 1, "verdict_kind": "validation_failed",
+      "error": "consistency: city mismatch", "span_id": "def..." },
+    { "iteration": 2, "verdict_kind": "accepted", "span_id": "ghi..." }
+  ],
+  "artifact":        { /* final accepted artifact, omitted on failure */ },
+  "grounded_in_roster": true,                       // null when N/A (email/playbook)
+  "duration_ms":     2840
+}
+```
+
+### Field semantics
+
+| Field | When set | What it means |
+|---|---|---|
+| `session_id` | always | Langfuse trace id. Pivot to live trace tree by URL: `${LANGFUSE_URL}/trace/<session_id>`. |
+| `final_verdict=accepted` | success | Loop converged within `max_iterations`. `artifact` is non-null. |
+| `final_verdict=max_iter_exhausted` | failure | Loop hit the cap without passing validation. `artifact` is omitted. |
+| `final_verdict=infra_error` | failure | Chat hop or other infra crashed. Single attempt with `verdict_kind=infra_error`. |
+| `grounded_in_roster=true` | fill kind, success | Every `candidate_id` in the artifact exists in `WorkerLookup`. |
+| `grounded_in_roster=false` | fill kind, anomaly | Phantom or otherwise-invalid candidate IDs (shouldn't happen — FillValidator catches these — but the explicit check defends against future validator weakening). |
+| `grounded_in_roster=null` (omitted) | non-fill kinds, or failure | The roster check doesn't apply or wasn't run. |
+
+## DuckDB queries
+
+```sql
+-- Read the log directly via DuckDB's read_json_auto.
+ATTACH ':memory:' AS sessions;
+SELECT * FROM read_json_auto(
+  '/var/lib/lakehouse/validator/sessions.jsonl', format='newline_delimited'
+) LIMIT 10;
+```
+
+### "Did the validator catch every phantom worker?"
+```sql
+-- Sessions where iteration 0's verdict was validation_failed AND
+-- the error mentions 'phantom' or 'consistency'. If grounded=true on
+-- the same session's final state, the model recovered.
+SELECT session_id, model, iterations, grounded_in_roster, final_verdict
+FROM read_json_auto('sessions.jsonl', format='newline_delimited')
+WHERE final_verdict = 'accepted'
+  AND iterations > 1
+  AND list_contains(
+        list_transform(attempts,
+          x -> x.error LIKE '%consistency%' OR x.error LIKE '%phantom%'),
+        true
+      );
+```
+
+### "First-shot success rate per model" (the "did the corpus give it enough" gate)
+```sql
+SELECT model,
+       COUNT(*) AS sessions,
+       SUM(CASE WHEN iterations = 1 AND final_verdict = 'accepted' THEN 1 ELSE 0 END) AS first_shot,
+       ROUND(100.0 * SUM(CASE WHEN iterations = 1 AND final_verdict = 'accepted' THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct
+FROM read_json_auto('sessions.jsonl', format='newline_delimited')
+WHERE kind = 'fill'
+GROUP BY model
+ORDER BY pct DESC;
+```
+
+### "Sessions that were never grounded" (the alarm query)
+```sql
+-- Should always be empty. If it isn't, FillValidator has a hole or
+-- a different code path is bypassing the roster check.
+SELECT session_id, model, iterations, attempts
+FROM read_json_auto('sessions.jsonl', format='newline_delimited')
+WHERE kind = 'fill'
+  AND final_verdict = 'accepted'
+  AND grounded_in_roster = false;
+```
+
+### "Average retry depth per model"
+```sql
+SELECT model, AVG(iterations) AS avg_iter, COUNT(*) AS n
+FROM read_json_auto('sessions.jsonl', format='newline_delimited')
+WHERE kind = 'fill' AND final_verdict = 'accepted'
+GROUP BY model
+ORDER BY avg_iter ASC;
+```
+
+### "What did validation reject?" (failure mode breakdown)
+```sql
+-- Pull each rejected attempt's error string, classify by prefix.
+WITH errors AS (
+  SELECT session_id,
+         model,
+         unnest(attempts) AS att
+  FROM read_json_auto('sessions.jsonl', format='newline_delimited')
+)
+SELECT model,
+       split_part(att.error, ':', 1) AS kind,
+       COUNT(*) AS n
+FROM errors
+WHERE att.verdict_kind = 'validation_failed'
+GROUP BY model, kind
+ORDER BY n DESC;
+```
+
+## Operational notes
+
+- **Append-only.** No row is ever updated; storage grows linearly with iterate calls. Operators rotate via cron when the file gets unwieldy (logrotate-style).
+- **Best-effort posture.** Every write goes through `slog.Warn` on failure but never blocks the iterate handler. A full disk silently drops session rows; the iterate response still ships.
+- **Schema versioning.** `schema=session.iterate.v1` is the contract. Future incompatible changes bump the version; consumers should branch on the field.
+- **PII consideration.** `prompt` is captured truncated to 4000 chars and the final `artifact` (when present) is captured verbatim. Operators handling PII-bearing prompts should set the path under a restricted-access volume or filter before retention.
+- **Cross-runtime parity.** The Rust gateway's `/v1/iterate` does NOT yet write this file. If you want a unified longitudinal log across runtimes, port the writer to Rust (`crates/gateway/src/v1/iterate.rs`) and target the same JSONL path. ~50 LOC.
+
+## See also
+
+- `internal/validator/session_log.go` — writer + record types
+- `internal/validator/iterate.go` — `Tracer` callback + Langfuse span emission
+- `internal/shared/langfuse_middleware.go` — `X-Lakehouse-Trace-Id` header propagation (the `session_id` join key)
+- `data/_kb/replay_runs.jsonl` — the *replay* tool's own JSONL (different shape, different producer); these two streams are siblings, not duplicates

docs/SPEC.md

@@ -28,9 +28,11 @@ Effort scale (one engineer-week = ~40h focused work):
 | `queryd` | datafusion, arrow | `cmd/queryd` | **`duckdb/duckdb-go/v2`** (cgo, official) | **HARD** | high — see §3 |
 | `ingestd` | csv, json, lopdf, postgres | `cmd/ingestd` | stdlib `encoding/csv`, `encoding/json`, `pdfcpu/pdfcpu`, `jackc/pgx/v5` | **L** | low |
 | `vectord` | hora, arrow, hnsw | `cmd/vectord` | `coder/hnsw`, `apache/arrow-go/v18` | **L** | medium — re-validate HNSW recall |
+| **matrix indexer** (emergent in Rust — `mode.rs` + `build_*_corpus.ts` + observer `/relevance`) | scripts/build_*_corpus.ts, crates/gateway/src/v1/mode.rs, mcp-server/observer.ts | `internal/matrix/` + gateway routes (`/v1/matrix/*`) | stdlib + vectord client | **L** | medium — see §3.4. Corpus-as-shard composer; relevance filter; strong-model downgrade gate; multi-corpus retrieve+merge. The learning-loop layer that lifts vectord from "static index" to "meta-index that learns from playbooks." |
 | `vectord-lance` | lance | **DROPPED** | n/a | n/a | n/a — Parquet+HNSW only |
 | `journald` | parquet, arrow | `cmd/journald` | `apache/arrow-go/v18` | **M** | low |
 | `aibridge` | reqwest | library | `net/http` + connection pool · `anthropics/anthropic-sdk-go` available for direct Claude calls (currently routed via opencode) | **S** | low |
+| **chatd** (Phase 4 — multi-provider LLM dispatcher) | crates/gateway/src/v1/{ollama_cloud,openrouter,opencode}.rs | `cmd/chatd` + `internal/chat/` | stdlib `net/http` only | **DONE** | medium — see §3.9. 5-provider routing (ollama / ollama_cloud / openrouter / opencode / kimi) by model-name prefix or `:cloud` suffix. Replaces the Rust gateway's `v1::` adapters. |
 | `validator` | parquet, custom | library | `apache/arrow-go/v18` parquet reader | **M** | low — port the 24 unit tests as gates |
 | `truth` | tomli, custom DSL | library | `pelletier/go-toml/v2` | **M** | low |
 | `proto` | tonic-build | `proto/` + `protoc-gen-go` | `buf` + `protoc-gen-go-grpc` | **S** | low |
@@ -116,6 +118,371 @@ needs revisiting in Go to confirm the sidecar format we ship.
 - G3.2.C — Recall@10 within 2% of Rust baseline on
   `lakehouse_arch_v1`
 
+### §3.4 — Matrix indexer (corpus-as-shard composer)
+
+**What it is.** The matrix indexer is the layer above `vectord` that
+turns a fleet of single-corpus HNSW indexes into a learning meta-index.
+In the Rust system this is emergent — split between corpus builders
+(`scripts/build_*_corpus.ts`), the mode runner (`crates/gateway/src/v1/mode.rs`),
+the observer relevance endpoint (`mcp-server/observer.ts`), and the
+strong-model downgrade gate (`mode.rs::execute`). In Go we name it
+explicitly so future sessions don't reduce it to "vectord."
+
+**Why corpus-as-shard, not shard-by-id.** Sharding a single index by
+hash(id) is a pure throughput hack with a recall tax. Sharding by
+corpus is the existing retrieval shape — `lakehouse_arch_v1`,
+`lakehouse_symbols_v1`, `scrum_findings_v1`, `lakehouse_answers_v1`,
+`kb_team_runs_v1`, `successful_playbooks_live`, etc. — each with
+distinct topology and a distinct retrieval intent. Concurrent Adds
+parallelize naturally because they go to different corpora; the
+matrix layer's job is to retrieve+merge across them, filter for
+relevance, and downgrade composition when strong models prove the
+matrix is anti-additive.
+
+**Components to port (in dependency order):**
+
+1. **Corpus builders** — Go equivalents of `scripts/build_*_corpus.ts`.
+   For each named corpus, a builder that reads source, splits into
+   chunks per the corpus's schema, embeds via `/v1/embed`, and adds
+   to a vectord index of the same name. Effort: **M** for the first
+   builder, **S** for each subsequent.
+
+2. **Multi-corpus retrieve+merge** (`internal/matrix/retrieve.go`) —
+   given a query and a list of corpus names, search each at top_k=K,
+   merge by score, return top N globally. Match Rust's pattern:
+   top_k=6 per corpus, top 8 globally before relevance filter.
+
+3. **Relevance filter** (`internal/matrix/relevance.go`) — port the
+   threshold-based filter from `mcp-server/observer.ts:/relevance`.
+   Drops adjacency-pollution chunks that share a corpus with the hit
+   but aren't actually about the query. `LH_RELEVANCE_FILTER` /
+   `LH_RELEVANCE_THRESHOLD` env knobs preserved.
+
+4. **Strong-model downgrade gate** (`internal/matrix/downgrade.go`) —
+   port `is_weak_model` + the `codereview_lakehouse → codereview_isolation`
+   flip from `mode.rs::execute`. Pass5 proved composed corpora lose
+   5/5 vs isolation on grok-4.1-fast (p=0.031); the gate is
+   load-bearing for paid-model retrieval quality.
+
+5. **Learning-loop integration** — write outcomes back to a
+   playbook-memory corpus (probably `lakehouse_answers_v1` analogue).
+   This is what makes the matrix INDEX a learning system rather than
+   static retrieval. Per `feedback_meta_index_vision.md`: this is the
+   north star, not the data structure.
+
+**Gateway routes:** `/v1/matrix/search` (multi-corpus retrieve+merge),
+`/v1/matrix/corpora` (list + metadata), `/v1/matrix/relevance` (filter
+endpoint, used by both internal callers and external tooling).
+
+**Acceptance gates:**
+- G3.4.A — `/v1/matrix/search` against ≥3 corpora returns merged top-N
+  with corpus attribution per result.
+- G3.4.B — Relevance filter drops at least the threshold-margin chunks
+  on a known adjacency-pollution test case.
+- G3.4.C — Strong-model downgrade gate flips composed→isolation when
+  the model is non-weak; bypassed when caller sets `force_mode`.
+- G3.4.D — Concurrent Adds across N=4 corpora parallelize (no shared
+  write-lock); Add throughput scales near-linearly with corpus count.
+
+**Persistence:** each corpus's vectord index persists via the existing
+G1P LHV1 format. The matrix layer is stateless above that — corpus
+list lives in catalog, retrieval params in config.
+
+**Why this is its own §3.x:** in Rust the matrix indexer was emergent
+and got reduced to "we have vectord" in earlier port-planning. The
+SPEC names it explicitly so the port preserves the multi-corpus
+retrieval shape AND the learning loop, not just the HNSW substrate.
+
+### §3.5 — Drift quantification (loop 5 of the PRD)
+
+**What it is.** PRD names "drift" as the 5th loop: quantify when
+historical decisions stop matching current reality. Distinct from
+the rating+distillation loop because drift is MEASUREMENT, not
+LEARNING. The learning loop says "this match worked, remember it";
+the drift loop says "this 4-month-old playbook entry — does it
+still match what the substrate would surface today?"
+
+**What's shipped (commit `be65f85`):**
+  - SCORER drift: re-runs current `distillation.ScoreRecord` over
+    historical (EvidenceRecord, persisted_category) pairs and
+    reports mismatches + a sorted shift matrix
+  - `internal/drift/drift.go` — pure-function `ComputeScorerDrift`
+  - 6 unit tests covering no-drift, shift detection, multi-shift
+    sorted-by-count, includeEntries flag, empty input, scorer-version
+    stamping
+
+**Future drift shapes (not shipped):**
+  - PLAYBOOK drift: re-run playbook queries through current
+    matrix-search; recorded answer not in top-K = drift
+  - EMBEDDING drift: KS-test on vector distribution at T1 vs T2
+  - AUDIT BASELINE drift: matches Rust `audit_baselines.jsonl`
+    longitudinal signal
+
+**Acceptance gates:**
+- G3.5.A — A scorer-version bump triggers a non-zero `Drifted` count
+  on a corpus of historical ScoredRuns where the new logic produces
+  different categories than the persisted ones.
+- G3.5.B — `ScorerDriftReport.ShiftMatrix` is deterministic-ordered
+  (count desc, ties broken alphabetically) so JSON output is stable
+  across runs.
+
+### §3.6 — Staffing-side structured filter
+
+**What it is.** Reality tests on the candidates + workers corpora
+(commits `0d1553c`, `a97881d`) surfaced that pure semantic retrieval
+can't gate by location/status/availability — the matrix indexer
+returns Production Workers for a Forklift+OSHA-30 query because
+nomic-embed-text's geometry doesn't separate the role labels well.
+Structured filtering is the addressable piece: pre-filter the
+candidate set on metadata fields BEFORE semantic ranking.
+
+**What's shipped (commit `b199093`):**
+  - `SearchRequest.MetadataFilter` — `map[string]any` of metadata
+    field → expected value (single value or list-of-values for OR
+    semantics within a key, AND across keys)
+  - Post-retrieval filter applied before top-K truncation in
+    `internal/matrix/retrieve.go`
+  - `SearchResponse.MetadataFilterDropped` for telemetry on filter
+    aggressiveness
+  - 7 unit tests covering nil filter, missing metadata, exact match,
+    AND across keys, OR within list, bool match, malformed JSON
+
+**Deferred:**
+  - Pre-retrieval SQL gate via `queryd` (the actual hybrid). The
+    post-retrieval filter is an MVP that helps when the candidate
+    set is mostly relevant; for aggressive filters that drop most
+    results, a SQL pre-filter into matrix retrieval would surface
+    the right candidates with less wasted embedding work.
+  - Filter language richer than equality (e.g. range, prefix, regex).
+
+**Acceptance gates:**
+- G3.6.A — `MetadataFilter: {"state": "IL"}` against a mixed-state
+  corpus drops every non-IL result; `MetadataFilterDropped` reports
+  the count.
+- G3.6.B — List filter `{"state": ["IL", "WI"]}` keeps both states,
+  drops the rest (OR within key).
+- G3.6.C — Multi-key filter is AND: a result missing any key is
+  dropped, no exception.
+
+### §3.7 — Operational rating wiring
+
+**What it is.** PRD loop 4 (rating + distillation) needs real
+inflows to be a learning system rather than a substrate. The
+playbook-record endpoint (`06e7152`) takes one (query, answer,
+score) per call; productizing it into actual signal sources is what
+makes the system get smarter with use.
+
+**What's shipped (commit `6392772`):**
+  - `POST /v1/matrix/playbooks/bulk` — bulk-record N successes;
+    per-entry success/failure response so callers can see which of
+    a 4,701-row historical placement import succeeded vs which
+    failed validation.
+  - Single-record path from `06e7152` unchanged.
+
+**Deferred:**
+  - UI shim for click-tracking (no Go demo UI yet — the Bun demo at
+    `devop.live/lakehouse/` is still serving the public surface).
+    When the Go UI lands or a feedback API is added to the Bun UI,
+    every coordinator click → bulk-batched POST → playbook entry.
+  - Negative feedback (this match didn't work). Currently only
+    positive scores are recorded; a rejection signal would help the
+    learning loop avoid pushing bad matches.
+  - Time-decay on playbook scores so stale recommendations attenuate.
+
+**Acceptance gates:**
+- G3.7.A — Bulk POST of N entries returns `{recorded, failed,
+  results[]}` with per-entry IDs/errors, no single-entry failure
+  aborting the batch.
+- G3.7.B — Each recorded entry surfaces in `/v1/matrix/search` with
+  `use_playbook=true` after a re-query.
+
+### §3.8 — Observer-KB workflow runner (Archon-style multi-pass)
+
+**What it is.** The architectural pattern documented in the Rust
+`observer-kb` branch (10 commits ahead of main, never merged) and
+proven by `/home/profit/external/Archon`'s workflow engine. Multiple
+mode passes processing data, with each pass an objective measurement
+that contributes to the KB:
+
+```
+Raw data
+   ↓ Mode: EXTRACT       structured facts/entities/relationships
+   ↓ Mode: VALIDATOR     fact-check, confidence 1-10
+   ↓ Mode: HALLUCINATION verify each claim, flag likely fabrications
+   ↓ Mode: CONSENSUS     multiple passes until extraction converges
+   ↓ Mode: REDTEAM       attack what survived, patch what fails
+   ↓ Mode: PIPELINE      clean → Q&A structure → topic group → rank
+   ↓ RENDER              curated doc anchored on questions
+```
+
+This is the *orchestrator* missing from §3.4 components 1-5: each
+SPEC §3.4 piece (relevance, downgrade, scorer, drift) is a "mode";
+what's missing is the workflow engine that chains them.
+
+**Why it matters.** Per the PRD's product vision: the observer
+should make actionable decisions based on watching what's
+successful. The workflow runner is how observers compose modes
+into multi-pass pipelines that score outcomes rigorously enough
+to feed the KB and inform the playbook substrate.
+
+**Reference materials on the system:**
+  - `/home/profit/lakehouse/.archon/workflows/lakehouse-architect-review.yaml`
+    (committed `69919d9` in main) — proves Archon-via-Lakehouse
+    works with a 3-node `shape → weakness → improvement` workflow
+  - `/home/profit/external/Archon` — the upstream workflow engine
+    (cloned 2026-04-26); `packages/providers/src/community/pi/provider.ts`
+    has the local Lakehouse-routing mod committed locally as
+    `3f2afc8` (not pushed to upstream `coleam00/Archon`)
+  - Rust `observer-kb` branch (10 commits, +4338/-55506 LoC) —
+    `apps/observer-kb/docs/PRD.md` documents the multi-pass
+    architecture; `scripts/{deep_analysis,extract_knowledge,process_knowledge}.py`
+    are the Python prototypes that proved it on real ChatGPT/Claude
+    PDF data (496 topics, 300 decisions, 100 insights extracted)
+
+**Components to port (in dependency order):**
+
+1. **Workflow definition** (`internal/workflow/types.go`) — YAML
+   schema matching Archon's shape: `name`, `description`, `provider`,
+   `model`, list of `nodes` each with `id`, `prompt`, `allowed_tools`,
+   `effort`, `idle_timeout`, `depends_on`. The depends_on edges form
+   a DAG; the runner resolves topologically.
+
+2. **Node executor** (`internal/workflow/runner.go`) — given a
+   workflow and a starting context, walks the DAG, executes each
+   node by dispatching to the configured backend (matrix.Search,
+   distillation.ScoreRecord, drift.ComputeScorerDrift, or a generic
+   prompt-against-LLM via gateway `/v1/chat`), captures per-node
+   output, makes it available as `$<node_id>.output` in subsequent
+   nodes.
+
+3. **Provenance recording** — every node execution lands an
+   ObservedOp (via the observerd substrate from `bc9ab93`) with
+   `source: "workflow"`, the workflow name + node ID, input/output
+   summaries, and timing. The ring buffer + JSONL log become the
+   substrate for the rating+distillation loop's KB feed.
+
+4. **Mode catalog** (`internal/workflow/modes.go`) — registry of
+   the modes the runner can dispatch to. Each mode is a Go function
+   matching a uniform `func(ctx, input map[string]any) (map[string]any, error)`
+   signature so workflows can compose them. Initial modes from
+   §3.4: `matrix.search`, `matrix.relevance`, `matrix.downgrade`,
+   `playbook.record`, `playbook.lookup`, `distillation.score`,
+   `drift.scorer`. Plus `llm.chat` for free-form mode prompts.
+
+5. **HTTP surface** — `POST /v1/observer/workflow/run` accepts a
+   workflow YAML body + a starting context; returns the per-node
+   results + the chain of ObservedOps generated. `GET
+   /v1/observer/workflow/list` lists workflows in a known directory
+   for operator discoverability.
+
+**Why integrate into observerd, not a new service.** The observer
+is the system resource that watches and records. Workflows ARE
+observation patterns — multi-step processes whose every step is
+recorded. Putting the runner inside observerd keeps the
+"measurement → KB feed" wiring tight; a separate service would
+re-implement the recording layer.
+
+**Acceptance gates:**
+- G3.8.A — Load a workflow YAML matching the Archon `lakehouse-architect-review.yaml`
+  shape; runner executes the 3-node DAG topologically.
+- G3.8.B — Each node execution lands an ObservedOp with
+  `source: "workflow"` and the node's input/output. Stats endpoint
+  shows the workflow ops.
+- G3.8.C — A node referencing `$<prior_node>.output` in its prompt
+  resolves correctly; missing reference is a clear error not a
+  silent empty string.
+- G3.8.D — Mode catalog dispatches `matrix.search` invocation to
+  the matrixd backend without going through HTTP (in-process
+  function call when matrixd is co-resident).
+
+**Status:** PORT TARGET, not yet started. SPEC commits the design;
+implementation is its own wave (estimated **L** effort given the
+DAG runner + mode dispatch + provenance recording).
+
+### §3.9 — chatd (multi-provider LLM dispatcher) — SHIPPED 2026-04-30
+
+**Status:** done at commit `05273ac` (Phase 4 wave) + scrum-hardened
+at `0efc736`. Composite port: 1,624 LoC, 19+ tests, 6/6 chatd_smoke.
+
+**What:** `cmd/chatd` on `:3220` routes `POST /chat` to a provider
+selected by model-name prefix or `:cloud` suffix:
+
+```
+ollama/<m>            → local Ollama at :11434 (no auth)
+ollama_cloud/<m>      → ollama.com /api/generate (Bearer)
+<m>:cloud             → ollama_cloud (suffix variant)
+openrouter/<v>/<m>    → openrouter.ai (OpenAI-compat, Bearer)
+opencode/<m>          → opencode.ai/zen/v1 (OpenAI-compat, Bearer)
+kimi/<m>              → api.kimi.com/coding/v1 (OpenAI-compat, Bearer)
+bare names            → ollama (default)
+```
+
+**Provider key resolution:** env var first (`OPENROUTER_API_KEY`,
+`OPENCODE_API_KEY`, `KIMI_API_KEY`, `OLLAMA_CLOUD_KEY`); then
+`/etc/lakehouse/<provider>.env` fallback (mode 0600). Empty key →
+provider stays unregistered (404 at first call instead of 503).
+
+**Companion to the model tier registry** in `lakehouse.toml [models]`
+(local_fast / local_judge / cloud_judge / frontier_review / etc.),
+which maps tier names to model IDs. Callers reference
+`cfg.Models.LocalJudge` instead of literal strings. Bumping a tier
+is a 1-line config edit.
+
+**Quirks captured by today's scrum:**
+- `Request.Temperature` is `*float64` (pointer) — Anthropic 4.7 (via
+  OpenCode) rejects the field entirely with "temperature is deprecated."
+  Pointer lets us omit when caller didn't set explicitly.
+- Local Ollama defaults `think=false` — qwen3.5:latest is reasoning-
+  capable but the inner-loop hot path wants direct answers, not
+  reasoning traces consuming the token budget.
+
+**Replaces the Rust adapters:**
+- `crates/gateway/src/v1/ollama_cloud.rs` → `internal/chat/ollama_cloud.go`
+- `crates/gateway/src/v1/openrouter.rs` → `internal/chat/openai_compat.go` (shared with opencode + kimi)
+- `crates/gateway/src/v1/opencode.rs` → same shared helper
+- `crates/aibridge/*` → `internal/chat/` (cleaner abstraction)
+
+**Reusable downstream:** `scripts/scrum_review.sh` runs a 3-lineage
+cross-review (Opus + Kimi + Qwen3-coder) by POST-ing each diff to
+chatd's `/v1/chat`. Same vehicle the harness's own scrum-hardening
+used to find its own bugs.
+
+### §3.10 — local-review-harness (sibling tool, separate repo)
+
+**Where:** `git.agentview.dev/profit/local-review-harness` (also
+SMB-mounted at `/home/profit/share/local-review-harness-full-md/`).
+
+**What:** a local-first code review harness — walks a target repo,
+runs evidence-bearing static checks (12 analyzers covering hardcoded
+paths, raw SQL, wildcard CORS, secrets, exec/spawn, large files,
+TODO/FIXME, missing tests, .env files committed, exposed mutation
+endpoints, hardcoded private IPs), produces Scrum-style markdown
+reports + JSON receipts. **No cloud deps.** Single static Go binary.
+
+**Status:** Phase A (skeleton) + Phase B (MVP — static-only path)
+shipped at first commit `f3ee472`. 5 acceptance gates green plus
+self-review (the harness reviews its own repo).
+
+**Phases C–E pending:** local-Ollama LLM review, validation cross-
+check, append-only `.memory/`, diff/rules subcommands.
+
+**Why a sibling tool, not a Lakehouse module:** PROMPT.md "Strategic
+Goal" — the harness eventually plugs into OpenClaw / MCP tools /
+Lakehouse memory / playbook sealing / observer review loop. But
+first it has to be reliable, inspectable, and evidence-driven on
+its own. Lakehouse-Go integration is post-Phase-E (probably E+1:
+write the harness's findings into Lakehouse's playbook substrate
+via `/v1/matrix/playbooks/record`).
+
+**Cross-pollination opportunities (post-MVP):**
+- Replace `internal/llm/ollama.go` in the harness with a thin client
+  pointed at chatd's `/v1/chat` — frontier judges become a config
+  toggle.
+- Feed harness findings into Lakehouse-Go's pathway memory as a
+  drift signal (which static checks fired this run vs last).
+- Use the harness's `.memory/known-risks.json` as a corpus the
+  matrix indexer can retrieve from when the same risk pattern appears.
+
 ### §3.3 — UI (HTMX)
 
 **Approach:** server-rendered Go templates using `html/template`,

docs/TEST_PROOF_SCOPE.md

@@ -0,0 +1,258 @@
+Create `docs/TEST_PROOF_SCOPE.md`.
+
+Purpose: design a serious proof harness for the Go lakehouse refactor.
+
+You are not writing production features yet. You are designing and implementing a claims-verification test suite that proves or disproves what this system currently claims.
+
+## System Claims To Prove
+
+The Go lakehouse claims:
+
+1. Gateway-fronted services work as a coherent system.
+2. CSV data can ingest into Parquet.
+3. Catalog manifests remain consistent.
+4. DuckDB query path returns correct results.
+5. Embedding path works through Ollama or configured embedding backend.
+6. Vector add/search works.
+7. Vector persistence survives restart.
+8. Service contracts are stable.
+9. Refactor preserved behavior.
+10. Performance claims are measurable, not vibes.
+
+## Required Output
+
+Create a proof harness under:
+
+```text
+tests/proof/
+tests/proof/
+  README.md
+  claims.yaml
+  run_proof.sh
+  lib/
+    env.sh
+    http.sh
+    assert.sh
+    metrics.sh
+  cases/
+    00_health.sh
+    01_storage_roundtrip.sh
+    02_catalog_manifest.sh
+    03_ingest_csv_to_parquet.sh
+    04_query_correctness.sh
+    05_embedding_contract.sh
+    06_vector_add_search.sh
+    07_vector_persistence_restart.sh
+    08_gateway_contracts.sh
+    09_failure_modes.sh
+    10_perf_baseline.sh
+  fixtures/
+    csv/
+    expected/
+  reports/
+    .gitkeep
+
+Test Design Requirements
+
+Each test must produce evidence, not just pass/fail.
+
+For every case, record:
+
+claim tested
+service routes called
+input fixture hash
+output hash
+expected result
+actual result
+pass/fail
+latency
+status codes
+logs location
+timestamp
+git commit hash
+
+Write results to:
+
+tests/proof/reports/proof-YYYYMMDD-HHMMSS/
+
+Each run must produce:
+
+summary.md
+summary.json
+raw/
+  http/
+  logs/
+  outputs/
+  metrics/
+Claims File
+
+Create tests/proof/claims.yaml.
+
+Each claim should have:
+
+id: GOLAKE-001
+name: Gateway health routes respond
+type: contract
+services:
+  - gateway
+routes:
+  - GET /health
+evidence:
+  - status_code
+  - response_body
+  - latency_ms
+required: true
+
+Include claims for:
+
+gateway health
+each service health
+storage put/get/list/delete if supported
+catalog create/read/update/list if supported
+ingest job creation
+Parquet output existence
+query correctness against known CSV fixture
+embedding vector dimension
+vector add/search nearest-neighbor correctness
+vector restart persistence
+invalid request rejection
+missing object behavior
+duplicate vector ID behavior
+malformed CSV behavior
+unavailable downstream service behavior
+latency baseline
+throughput baseline
+Fixtures
+
+Create deterministic fixtures.
+
+Minimum CSV fixture:
+
+id,name,role,city,score
+1,Ada,welder,Chicago,91
+2,Grace,electrician,Detroit,88
+3,Linus,operator,Chicago,77
+4,Ken,pipefitter,Houston,84
+5,Barbara,safety,Houston,95
+
+Expected query assertions:
+
+count rows = 5
+city Chicago = 2
+max score = 95
+role safety belongs to Barbara
+Houston average score = 89.5
+
+For vector tests, use deterministic text fixtures:
+
+doc-001: industrial staffing for welders in Chicago
+doc-002: safety compliance for warehouse crews
+doc-003: electrical contractors assigned to Detroit
+doc-004: pipefitters and heavy equipment operators in Houston
+
+Search assertions should verify that semantically related queries return expected top candidates where embeddings are enabled.
+
+If embeddings are not deterministic enough, support a contract-only mode that verifies:
+
+vector dimension
+non-empty vector
+add succeeds
+search returns known inserted IDs
+persistence survives restart
+Modes
+
+Support three modes:
+
+tests/proof/run_proof.sh --mode contract
+tests/proof/run_proof.sh --mode integration
+tests/proof/run_proof.sh --mode performance
+Contract mode
+
+Fast. No massive data. Verifies APIs, schemas, status codes, basic correctness.
+
+Integration mode
+
+Runs full gateway → service chain.
+
+Must prove:
+
+CSV fixture → storaged → ingestd → catalogd → queryd
+text fixture → embedd → vectord → search
+Performance mode
+
+Measures baseline only. Do not fake claims.
+
+Record:
+
+rows ingested/sec
+vectors added/sec
+p50/p95 query latency
+p50/p95 vector search latency
+memory usage if available
+CPU usage if available
+service restart time if available
+Failure-Mode Tests
+
+Add tests proving the system fails cleanly.
+
+Required:
+
+malformed JSON
+missing required field
+invalid vector dimension
+missing object
+bad SQL query
+duplicate vector ID
+downstream service unavailable if easy to simulate
+restart before persistence load completes if relevant
+
+Do not hide failures behind retries unless the system explicitly documents retry behavior.
+
+Hard Rules
+Do not add production features unless needed to expose testable behavior.
+Do not change public route contracts without documenting it.
+Do not write tests that merely check “HTTP 200” unless the claim is health-only.
+Do not use random data unless seeded and recorded.
+Do not make performance claims without before/after metrics.
+Do not assume Ollama is available; detect it and mark embedding tests skipped or degraded with explanation.
+Do not let skipped tests appear as passed.
+Do not silently ignore missing services.
+Do not make the proof harness depend on external cloud services.
+Final Deliverables
+
+After implementation, produce:
+
+tests/proof/README.md
+tests/proof/claims.yaml
+tests/proof/run_proof.sh
+tests/proof/cases/*.sh
+tests/proof/reports/<latest>/summary.md
+tests/proof/reports/<latest>/summary.json
+Final Report Must Answer
+
+At the end, write a clear report:
+
+Which claims are proven?
+Which claims are partially proven?
+Which claims failed?
+Which claims were skipped and why?
+What evidence supports each claim?
+What bottlenecks were measured?
+What contract drift was found?
+What refactor risks remain?
+What should be fixed first?
+Execution Plan
+
+First inspect the repo.
+
+Then produce a short implementation plan.
+
+Then build the proof harness.
+
+Then run contract mode.
+
+Then run integration mode if services can be started.
+
+Then run performance mode only if contract and integration pass.
+
+Do not declare success without evidence files.

go.mod

@@ -10,8 +10,12 @@ require (
 	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.16
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.100.0
 	github.com/aws/smithy-go v1.25.0
+	github.com/coder/hnsw v0.6.1
+	github.com/duckdb/duckdb-go/v2 v2.10502.0
 	github.com/go-chi/chi/v5 v5.2.5
 	github.com/google/uuid v1.6.0
+	github.com/hashicorp/golang-lru/v2 v2.0.7
+	github.com/modelcontextprotocol/go-sdk v1.5.0
 	github.com/pelletier/go-toml/v2 v2.3.0
 )
 
@@ -33,26 +37,29 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/sts v1.42.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/chewxy/math32 v1.10.1 // indirect
-	github.com/coder/hnsw v0.6.1 // indirect
 	github.com/duckdb/duckdb-go-bindings v0.10502.0 // indirect
 	github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10502.0 // indirect
 	github.com/duckdb/duckdb-go-bindings/lib/darwin-arm64 v0.10502.0 // indirect
 	github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10502.0 // indirect
 	github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10502.0 // indirect
 	github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10502.0 // indirect
-	github.com/duckdb/duckdb-go/v2 v2.10502.0 // indirect
 	github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
 	github.com/goccy/go-json v0.10.6 // indirect
 	github.com/google/flatbuffers v25.12.19+incompatible // indirect
+	github.com/google/jsonschema-go v0.4.2 // indirect
 	github.com/google/renameio v1.0.1 // indirect
 	github.com/klauspost/compress v1.18.5 // indirect
 	github.com/klauspost/cpuid/v2 v2.3.0 // indirect
 	github.com/pierrec/lz4/v4 v4.1.26 // indirect
+	github.com/segmentio/asm v1.1.3 // indirect
+	github.com/segmentio/encoding v0.5.4 // indirect
 	github.com/viterin/partial v1.1.0 // indirect
 	github.com/viterin/vek v0.4.2 // indirect
+	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
 	github.com/zeebo/xxh3 v1.1.0 // indirect
 	golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
 	golang.org/x/net v0.52.0 // indirect
+	golang.org/x/oauth2 v0.35.0 // indirect
 	golang.org/x/sync v0.20.0 // indirect
 	golang.org/x/sys v0.43.0 // indirect
 	golang.org/x/text v0.35.0 // indirect

go.sum

@@ -74,26 +74,38 @@ github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPE
 github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
 github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
+github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
+github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
 github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
 github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs=
 github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/jsonschema-go v0.4.2 h1:tmrUohrwoLZZS/P3x7ex0WAVknEkBZM46iALbcqoRA8=
+github.com/google/jsonschema-go v0.4.2/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
 github.com/google/renameio v1.0.1 h1:Lh/jXZmvZxb0BBeSY5VKEfidcbcbenKjZFzM/q0fSeU=
 github.com/google/renameio v1.0.1/go.mod h1:t/HQoYBZSsWSNK35C6CO/TpPLDVWvxOHboWUAweKUpk=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
+github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
 github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
 github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
+github.com/modelcontextprotocol/go-sdk v1.5.0 h1:CHU0FIX9kpueNkxuYtfYQn1Z0slhFzBZuq+x6IiblIU=
+github.com/modelcontextprotocol/go-sdk v1.5.0/go.mod h1:gggDIhoemhWs3BGkGwd1umzEXCEMMvAnhTrnbXJKKKA=
 github.com/pelletier/go-toml/v2 v2.3.0 h1:k59bC/lIZREW0/iVaQR8nDHxVq8OVlIzYCOJf421CaM=
 github.com/pelletier/go-toml/v2 v2.3.0/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
 github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY=
 github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc=
+github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg=
+github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0=
+github.com/segmentio/encoding v0.5.4/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0=
 github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
 github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
 github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
@@ -104,6 +116,8 @@ github.com/viterin/vek v0.4.2 h1:Vyv04UjQT6gcjEFX82AS9ocgNbAJqsHviheIBdPlv5U=
 github.com/viterin/vek v0.4.2/go.mod h1:A4JRAe8OvbhdzBL5ofzjBS0J29FyUrf95tQogvtHHUc=
 github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
 github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
+github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
+github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
 github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
 github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
 github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
@@ -124,12 +138,16 @@ golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7
 golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU=
 golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
 golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
+golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
+golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
 golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
 golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
 golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
 golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
+golang.org/x/tools v0.43.0 h1:12BdW9CeB3Z+J/I/wj34VMl8X+fEXBxVR90JeMX5E7s=
+golang.org/x/tools v0.43.0/go.mod h1:uHkMso649BX2cZK6+RpuIPXS3ho2hZo4FVwfoy1vIk0=
 gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
 gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260120221211-b8f7ae30c516 h1:sNrWoksmOyF5bvJUcnmbeAmQi8baNhqg5IWaI3llQqU=

internal/catalogd/subject.go

@@ -0,0 +1,371 @@
+// Subject manifests + per-subject audit-log chain verification.
+//
+// Specification: /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
+//
+// This file is the Go side of Step 8 (cross-runtime parity). The Rust
+// side is the writer (crates/catalogd/src/subject_audit.rs); Go is a
+// READER + VERIFIER only — both sides serialize / hash with identical
+// algorithms so a chain written by Rust verifies under Go.
+//
+// Algorithm (must match Rust crates/catalogd/src/subject_audit.rs exactly):
+//   row_hmac = HMAC-SHA256(key, prev_chain_hash_bytes || canonical_row_bytes)
+//
+// where:
+//   - prev_chain_hash_bytes is the ASCII bytes of the previous row's
+//     row_hmac field, OR the literal ASCII string "GENESIS" for the
+//     first row (no hex decode!)
+//   - canonical_row_bytes is the row JSON with the row_hmac field
+//     dropped, with all object keys sorted alphabetically at every
+//     nesting depth, with no insignificant whitespace
+//   - the final hash is rendered as 64 lowercase hex characters
+package catalogd
+
+import (
+	"bytes"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"os"
+	"sort"
+	"strings"
+	"time"
+)
+
+const GenesisHash = "GENESIS"
+
+// SubjectManifest mirrors crates/shared/src/types.rs::SubjectManifest.
+// Keep field tags byte-identical so manifest JSON written by Rust round-trips.
+//
+// Vertical: omitempty matches Rust's serde-default deserializer accepting
+// missing field (default Unknown). For the OTHER fields we deliberately
+// avoid omitempty since Rust always emits them.
+type SubjectManifest struct {
+	Schema              string                `json:"schema"`
+	CandidateID         string                `json:"candidate_id"`
+	CreatedAt           time.Time             `json:"created_at"`
+	UpdatedAt           time.Time             `json:"updated_at"`
+	Status              string                `json:"status"`
+	Vertical            string                `json:"vertical,omitempty"`
+	Consent             SubjectConsent        `json:"consent"`
+	Retention           SubjectRetention      `json:"retention"`
+	Datasets            []SubjectDatasetRef   `json:"datasets"`
+	SafeViews           []string              `json:"safe_views"`
+	AuditLogPath        string                `json:"audit_log_path"`
+	AuditLogChainRoot   string                `json:"audit_log_chain_root"`
+	BiometricCollection *BiometricCollection  `json:"biometric_collection,omitempty"`
+}
+
+// BiometricCollection mirrors crates/shared/src/types.rs::BiometricCollection.
+// Written by the Phase 1.6 Gate 3 photo-upload endpoint. None until the
+// first biometric upload; cleared on BIPA erasure.
+type BiometricCollection struct {
+	DataPath            string          `json:"data_path"`
+	TemplateHash        string          `json:"template_hash"`
+	CollectedAt         time.Time       `json:"collected_at"`
+	ConsentVersionHash  string          `json:"consent_version_hash,omitempty"`
+	Classifications     json.RawMessage `json:"classifications,omitempty"`
+}
+
+type SubjectConsent struct {
+	GeneralPii GeneralPiiConsent `json:"general_pii"`
+	Biometric  BiometricConsent  `json:"biometric"`
+}
+
+type GeneralPiiConsent struct {
+	Status       string     `json:"status"`
+	Version      string     `json:"version,omitempty"`
+	GivenAt      *time.Time `json:"given_at,omitempty"`
+	WithdrawnAt  *time.Time `json:"withdrawn_at,omitempty"`
+}
+
+type BiometricConsent struct {
+	Status         string     `json:"status"`
+	RetentionUntil *time.Time `json:"retention_until,omitempty"`
+}
+
+type SubjectRetention struct {
+	GeneralPiiUntil time.Time `json:"general_pii_until"`
+	Policy          string    `json:"policy,omitempty"`
+}
+
+type SubjectDatasetRef struct {
+	Name      string `json:"name"`
+	KeyColumn string `json:"key_column"`
+	KeyValue  string `json:"key_value"`
+}
+
+// SubjectAuditRow mirrors crates/shared/src/types.rs::SubjectAuditRow.
+// JSON field order does NOT matter (canonicalizer sorts at hash time)
+// but field names + types MUST match for round-trip.
+type SubjectAuditRow struct {
+	Schema         string         `json:"schema"`
+	Ts             time.Time      `json:"ts"`
+	CandidateID    string         `json:"candidate_id"`
+	Accessor       AuditAccessor  `json:"accessor"`
+	FieldsAccessed []string       `json:"fields_accessed"`
+	Result         string         `json:"result"`
+	PrevChainHash  string         `json:"prev_chain_hash"`
+	RowHmac        string         `json:"row_hmac"`
+}
+
+// AuditAccessor mirrors crates/shared/src/types.rs::AuditAccessor.
+//
+// IMPORTANT: trace_id has NO omitempty — Rust's serde always writes
+// the field (even empty) because `#[serde(default)]` only affects
+// READS. Stripping it here produces different canonical bytes than
+// the writer used → HMAC mismatch on real production rows. (Caught
+// 2026-05-03 by subject_audit_parity.sh on a live WORKER-1 row with
+// trace_id="".)
+type AuditAccessor struct {
+	Kind    string `json:"kind"`
+	Daemon  string `json:"daemon"`
+	Purpose string `json:"purpose"`
+	TraceID string `json:"trace_id"`
+}
+
+// LoadSubjectManifest reads + parses a subject manifest JSON file.
+func LoadSubjectManifest(path string) (*SubjectManifest, error) {
+	bytes, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("read manifest %s: %w", path, err)
+	}
+	var m SubjectManifest
+	if err := json.Unmarshal(bytes, &m); err != nil {
+		return nil, fmt.Errorf("parse manifest %s: %w", path, err)
+	}
+	return &m, nil
+}
+
+// AuditLogEntry is one parsed audit row PLUS the raw line bytes from
+// the file. The raw bytes are load-bearing: we MUST canonicalize from
+// them (not from the re-marshaled struct) to avoid time-precision drift.
+//
+// Why: Rust's chrono RFC3339-AutoSi serializes nanoseconds with 9 digits
+// (e.g. "461439210"). Go's time.RFC3339Nano strips trailing zeros (e.g.
+// "46143921"). Round-tripping through time.Time changes the byte sequence
+// and breaks the HMAC. Read once, hash from the original bytes.
+// (Caught 2026-05-03 by subject_audit_parity.sh on a real WORKER-5 row
+// whose nanoseconds happened to end in 0.)
+type AuditLogEntry struct {
+	Row SubjectAuditRow
+	Raw []byte
+}
+
+// ReadAuditLog parses an audit JSONL file. Returns one AuditLogEntry
+// per non-empty line. Defensive: blank lines are skipped, unparseable
+// lines are skipped (matches Rust read_rows_in_range).
+func ReadAuditLog(path string) ([]AuditLogEntry, error) {
+	bytes, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil
+		}
+		return nil, fmt.Errorf("read audit log %s: %w", path, err)
+	}
+	var entries []AuditLogEntry
+	for _, line := range strings.Split(string(bytes), "\n") {
+		trimmed := strings.TrimSpace(line)
+		if trimmed == "" {
+			continue
+		}
+		var row SubjectAuditRow
+		if err := json.Unmarshal([]byte(trimmed), &row); err != nil {
+			continue
+		}
+		entries = append(entries, AuditLogEntry{Row: row, Raw: []byte(trimmed)})
+	}
+	return entries, nil
+}
+
+// canonicalRowBytesFromRaw is the production canonicalizer used by
+// VerifyChain. It accepts the raw line bytes from the audit JSONL,
+// drops the row_hmac key, sorts all keys alphabetically at every
+// nesting depth, and re-emits compact JSON. Numbers + strings pass
+// through verbatim (json.Number preserves number text, strings are
+// re-JSON-escaped identically to Rust's serde_json).
+//
+// CRITICAL: hashing must always go through this function. NEVER hash
+// the bytes produced by canonicalRowBytesFromStruct against a real
+// production chain — the time-precision drift will give wrong HMACs
+// for any row whose nanoseconds end in 0.
+func canonicalRowBytesFromRaw(rawLine []byte) ([]byte, error) {
+	dec := json.NewDecoder(strings.NewReader(string(rawLine)))
+	dec.UseNumber()
+	var v any
+	if err := dec.Decode(&v); err != nil {
+		return nil, fmt.Errorf("decode raw line: %w", err)
+	}
+	if obj, ok := v.(map[string]any); ok {
+		delete(obj, "row_hmac")
+	}
+	var buf strings.Builder
+	if err := writeCanonical(&buf, v); err != nil {
+		return nil, err
+	}
+	return []byte(buf.String()), nil
+}
+
+// canonicalRowBytesFromStruct is for the parity probe's known-answer
+// vector + tests where we WANT to canonicalize a Go-built row from
+// scratch (no original bytes exist yet). Production verification uses
+// canonicalRowBytesFromRaw against the file bytes; this function is
+// only safe for synthetic inputs whose ts has no trailing-zero nanos.
+func canonicalRowBytesFromStruct(row *SubjectAuditRow) ([]byte, error) {
+	raw, err := json.Marshal(row)
+	if err != nil {
+		return nil, fmt.Errorf("marshal row: %w", err)
+	}
+	return canonicalRowBytesFromRaw(raw)
+}
+
+// marshalNoEscapeHTML wraps json.Encoder with HTML escaping disabled.
+//
+// Why: Go's json.Marshal escapes `<`, `>`, `&` to `<`, `>`,
+// `&` by default. Rust's serde_json::to_vec keeps them literal.
+// Any string field containing one of those characters would produce
+// different canonical bytes across runtimes → broken HMAC chain.
+// (Caught 2026-05-03 by opus scrum WARN on parity_subject_audit.rs:
+//  canonical_json — initially undetected because no production audit
+//  field contained `<>&`, but realistic for purpose strings like
+//  "error & retry" or trace_id "<HTTP-Request-Id>".)
+//
+// Also strips the trailing newline json.Encoder appends — that newline
+// is meaningful to JSONL consumers but is junk for hash input.
+func marshalNoEscapeHTML(v any) ([]byte, error) {
+	var buf bytes.Buffer
+	enc := json.NewEncoder(&buf)
+	enc.SetEscapeHTML(false)
+	if err := enc.Encode(v); err != nil {
+		return nil, err
+	}
+	out := buf.Bytes()
+	return bytes.TrimRight(out, "\n"), nil
+}
+
+// writeCanonical recursively writes v as canonical JSON: object keys
+// sorted alphabetically, no insignificant whitespace. Arrays preserve
+// element order (semantically significant per spec §3).
+//
+// All scalar emission goes through marshalNoEscapeHTML so the byte
+// sequence matches Rust's serde_json output character-for-character.
+func writeCanonical(buf *strings.Builder, v any) error {
+	switch t := v.(type) {
+	case map[string]any:
+		keys := make([]string, 0, len(t))
+		for k := range t {
+			keys = append(keys, k)
+		}
+		sort.Strings(keys)
+		buf.WriteByte('{')
+		for i, k := range keys {
+			if i > 0 {
+				buf.WriteByte(',')
+			}
+			ks, err := marshalNoEscapeHTML(k)
+			if err != nil {
+				return fmt.Errorf("marshal key: %w", err)
+			}
+			buf.Write(ks)
+			buf.WriteByte(':')
+			if err := writeCanonical(buf, t[k]); err != nil {
+				return err
+			}
+		}
+		buf.WriteByte('}')
+	case []any:
+		buf.WriteByte('[')
+		for i, elem := range t {
+			if i > 0 {
+				buf.WriteByte(',')
+			}
+			if err := writeCanonical(buf, elem); err != nil {
+				return err
+			}
+		}
+		buf.WriteByte(']')
+	default:
+		bs, err := marshalNoEscapeHTML(v)
+		if err != nil {
+			return fmt.Errorf("marshal scalar: %w", err)
+		}
+		buf.Write(bs)
+	}
+	return nil
+}
+
+// CanonicalAndHmac is a helper for the parity probe: canonicalizes the
+// row from scratch (struct → JSON → canonical), computes the HMAC
+// against the given prev_hash, returns both. Used only for the
+// known-answer fixture; production verification uses VerifyChain
+// (which canonicalizes from the original file bytes to dodge time
+// precision drift).
+func CanonicalAndHmac(row *SubjectAuditRow, key []byte, prevHash string) (canonical []byte, hmacHex string, err error) {
+	canonical, err = canonicalRowBytesFromStruct(row)
+	if err != nil {
+		return nil, "", err
+	}
+	hmacHex = computeRowHMAC(key, prevHash, canonical)
+	return canonical, hmacHex, nil
+}
+
+// computeRowHMAC = HMAC-SHA256(key, prev_hash_ascii_bytes || canonical_row_bytes)
+// rendered as 64-char lowercase hex.
+func computeRowHMAC(key []byte, prevHash string, canonical []byte) string {
+	mac := hmac.New(sha256.New, key)
+	mac.Write([]byte(prevHash))
+	mac.Write(canonical)
+	return hex.EncodeToString(mac.Sum(nil))
+}
+
+// VerifyChain replays the chain for a subject's audit log. Returns the
+// number of rows verified, the chain tip (last row's row_hmac, or
+// GENESIS when log is empty), and an error describing the first chain
+// break found.
+//
+// CRITICAL: canonicalizes from each entry's RAW LINE BYTES, not from
+// the parsed struct. Round-tripping through encoding/json + time.Time
+// strips trailing zeros from RFC3339 nanos and produces different
+// canonical bytes than the Rust writer used (which used chrono's
+// AutoSi 9-digit format). Catches: 2026-05-03 WORKER-5 nanos
+// 461439210 → Rust emits 9 digits, Go's time.RFC3339Nano emits 8.
+//
+// Special case: empty/missing log returns (0, GENESIS, nil).
+func VerifyChain(entries []AuditLogEntry, key []byte) (count int, chainTip string, err error) {
+	prev := GenesisHash
+	for i, entry := range entries {
+		if entry.Row.PrevChainHash != prev {
+			return i, prev, fmt.Errorf("chain break at row %d: prev_chain_hash=%q expected=%q",
+				i+1, entry.Row.PrevChainHash, prev)
+		}
+		claimed := entry.Row.RowHmac
+		canonical, cerr := canonicalRowBytesFromRaw(entry.Raw)
+		if cerr != nil {
+			return i, prev, fmt.Errorf("canonicalize row %d: %w", i+1, cerr)
+		}
+		recomputed := computeRowHMAC(key, prev, canonical)
+		if recomputed != claimed {
+			return i, prev, fmt.Errorf("hmac mismatch at row %d: stored=%s recomputed=%s",
+				i+1, claimed, recomputed)
+		}
+		prev = claimed
+		count = i + 1
+	}
+	chainTip = prev
+	return count, chainTip, nil
+}
+
+// LoadKeyFile reads a signing key file. Refuses keys shorter than 32
+// bytes (matches Rust: 2026-05-03 kimi BLOCK fix lifted the minimum
+// from 16 to 32 to align with HMAC-SHA256 best practice).
+func LoadKeyFile(path string) ([]byte, error) {
+	key, err := os.ReadFile(path)
+	if err != nil {
+		return nil, fmt.Errorf("read key file %s: %w", path, err)
+	}
+	if len(key) < 32 {
+		return nil, fmt.Errorf("signing key is %d bytes; recommend ≥32 bytes for HMAC-SHA256", len(key))
+	}
+	return key, nil
+}

internal/catalogd/subject_test.go

@@ -0,0 +1,364 @@
+package catalogd
+
+import (
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+)
+
+// deterministicKey is the same fixture key the Rust tests use:
+// (0u8..32).collect() — so a Rust-written chain verifies under Go.
+func deterministicKey() []byte {
+	k := make([]byte, 32)
+	for i := range k {
+		k[i] = byte(i)
+	}
+	return k
+}
+
+func mkRow(candidateID string, fields []string, prevHash, ts string) SubjectAuditRow {
+	t, _ := time.Parse(time.RFC3339Nano, ts)
+	return SubjectAuditRow{
+		Schema:      "subject_audit.v1",
+		Ts:          t,
+		CandidateID: candidateID,
+		Accessor: AuditAccessor{
+			Kind:    "gateway_lookup",
+			Daemon:  "gateway",
+			Purpose: "fill_validation",
+			TraceID: "",
+		},
+		FieldsAccessed: fields,
+		Result:         "success",
+		PrevChainHash:  prevHash,
+		RowHmac:        "", // computed below
+	}
+}
+
+// TestCanonicalJSON_KeysSortedAlphabetically asserts the same property
+// the Rust unit test asserts (subject_audit::tests::canonical_json_sorts_keys_alphabetically).
+func TestCanonicalJSON_KeysSortedAlphabetically(t *testing.T) {
+	v := map[string]any{
+		"z": 1,
+		"a": 2,
+		"m": map[string]any{"y": 1, "b": 2},
+	}
+	var buf strings.Builder
+	if err := writeCanonical(&buf, v); err != nil {
+		t.Fatalf("canonical: %v", err)
+	}
+	s := buf.String()
+	a, m, z := strings.Index(s, "\"a\""), strings.Index(s, "\"m\""), strings.Index(s, "\"z\"")
+	if !(a < m && m < z) {
+		t.Fatalf("top-level keys out of order: %s", s)
+	}
+	b, y := strings.Index(s, "\"b\""), strings.Index(s, "\"y\"")
+	if !(b < y) {
+		t.Fatalf("nested keys out of order: %s", s)
+	}
+}
+
+// TestCanonicalJSON_ArraysPreserveOrder asserts arrays are NOT sorted —
+// matches Rust subject_audit::tests::canonical_json_arrays_preserve_order.
+func TestCanonicalJSON_ArraysPreserveOrder(t *testing.T) {
+	v := map[string]any{"k": []any{"c", "a", "b"}}
+	var buf strings.Builder
+	if err := writeCanonical(&buf, v); err != nil {
+		t.Fatalf("canonical: %v", err)
+	}
+	if !strings.Contains(buf.String(), "\"c\",\"a\",\"b\"") {
+		t.Fatalf("array order altered: %s", buf.String())
+	}
+}
+
+// buildEntry produces an AuditLogEntry by computing the HMAC against
+// the row's struct-derived canonical bytes, then storing the resulting
+// row JSON as the raw bytes. Test-only — production reads raw bytes
+// straight from disk so the time-precision drift doesn't apply.
+func buildEntry(row SubjectAuditRow, key []byte, prev string) AuditLogEntry {
+	canon, err := canonicalRowBytesFromStruct(&row)
+	if err != nil {
+		panic(err)
+	}
+	row.RowHmac = computeRowHMAC(key, prev, canon)
+	raw, err := json.Marshal(row)
+	if err != nil {
+		panic(err)
+	}
+	return AuditLogEntry{Row: row, Raw: raw}
+}
+
+// TestVerifyChain_ReplaysAndReachesTip writes 3 rows with HMACs computed
+// the same way Rust would, then verifies they chain. This is the local
+// half of the parity contract — the cross-runtime half (Rust writes,
+// Go verifies) is covered by scripts/cutover/parity/subject_audit_parity.sh.
+func TestVerifyChain_ReplaysAndReachesTip(t *testing.T) {
+	key := deterministicKey()
+	r1 := mkRow("CAND-PARITY", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
+	e1 := buildEntry(r1, key, GenesisHash)
+	r2 := mkRow("CAND-PARITY", []string{"phone"}, e1.Row.RowHmac, "2026-05-03T12:00:01Z")
+	e2 := buildEntry(r2, key, e1.Row.RowHmac)
+	r3 := mkRow("CAND-PARITY", []string{"email"}, e2.Row.RowHmac, "2026-05-03T12:00:02Z")
+	e3 := buildEntry(r3, key, e2.Row.RowHmac)
+
+	count, tip, err := VerifyChain([]AuditLogEntry{e1, e2, e3}, key)
+	if err != nil {
+		t.Fatalf("verify failed: %v", err)
+	}
+	if count != 3 {
+		t.Fatalf("expected 3 rows verified, got %d", count)
+	}
+	if tip != e3.Row.RowHmac {
+		t.Fatalf("chain tip wrong: tip=%s expected=%s", tip, e3.Row.RowHmac)
+	}
+}
+
+// TestVerifyChain_EmptyLogIsTriviallyValid mirrors Rust's empty-log
+// special case: 0 rows, GENESIS tip, no error.
+func TestVerifyChain_EmptyLogIsTriviallyValid(t *testing.T) {
+	count, tip, err := VerifyChain(nil, deterministicKey())
+	if err != nil {
+		t.Fatalf("empty log returned error: %v", err)
+	}
+	if count != 0 {
+		t.Fatalf("expected 0 rows on empty log, got %d", count)
+	}
+	if tip != GenesisHash {
+		t.Fatalf("expected GENESIS tip on empty log, got %q", tip)
+	}
+}
+
+// TestVerifyChain_TamperDetected: tamper the raw line's `result` field
+// (the canonicalizer sees the new bytes; HMAC mismatches the stored hash).
+func TestVerifyChain_TamperDetected(t *testing.T) {
+	key := deterministicKey()
+	r1 := mkRow("CAND-T", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
+	e1 := buildEntry(r1, key, GenesisHash)
+	// Tamper: replace "success" with "denied" in the raw bytes ONLY.
+	// The struct's row_hmac (used as the "stored" comparator) stays put.
+	e1.Raw = []byte(strings.Replace(string(e1.Raw), `"success"`, `"denied"`, 1))
+	_, _, err := VerifyChain([]AuditLogEntry{e1}, key)
+	if err == nil {
+		t.Fatal("expected hmac mismatch after tamper, got nil")
+	}
+	if !strings.Contains(err.Error(), "hmac mismatch") {
+		t.Fatalf("expected hmac mismatch, got: %v", err)
+	}
+}
+
+// TestVerifyChain_BadKeyRejectsValidRows: same rows + wrong key = mismatch.
+func TestVerifyChain_BadKeyRejectsValidRows(t *testing.T) {
+	good := deterministicKey()
+	r1 := mkRow("CAND-BK", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
+	e1 := buildEntry(r1, good, GenesisHash)
+	bad := make([]byte, 32)
+	for i := range bad {
+		bad[i] = 0xff
+	}
+	_, _, err := VerifyChain([]AuditLogEntry{e1}, bad)
+	if err == nil {
+		t.Fatal("expected hmac mismatch with wrong key")
+	}
+}
+
+// TestComputeRowHMAC_StableAcrossRuns: same row + same key always = same hash.
+func TestComputeRowHMAC_StableAcrossRuns(t *testing.T) {
+	key := deterministicKey()
+	r := mkRow("CAND-S", []string{"a", "b"}, GenesisHash, "2026-05-03T12:00:00Z")
+	c1, _ := canonicalRowBytesFromStruct(&r)
+	c2, _ := canonicalRowBytesFromStruct(&r)
+	if string(c1) != string(c2) {
+		t.Fatalf("canonical bytes unstable across runs:\n  c1=%s\n  c2=%s", c1, c2)
+	}
+	h1 := computeRowHMAC(key, GenesisHash, c1)
+	h2 := computeRowHMAC(key, GenesisHash, c2)
+	if h1 != h2 {
+		t.Fatalf("hmac unstable across runs: %s vs %s", h1, h2)
+	}
+	if len(h1) != 64 {
+		t.Fatalf("hmac wrong length %d", len(h1))
+	}
+	// Sanity: hex-decodable.
+	if _, err := hex.DecodeString(h1); err != nil {
+		t.Fatalf("hmac not hex: %v", err)
+	}
+}
+
+// TestKnownAnswerVector matches a Go-computed reference. The same
+// inputs must produce this exact byte string under Rust as well —
+// scripts/cutover/parity/subject_audit_parity.sh runs the Rust helper
+// against this exact fixture and asserts byte-identical output.
+//
+// If you change the fixture, rebuild Rust's parity_subject_audit + Go's
+// helper and update both sides together.
+func TestKnownAnswerVector(t *testing.T) {
+	key := deterministicKey()
+	r := SubjectAuditRow{
+		Schema:      "subject_audit.v1",
+		Ts:          time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC),
+		CandidateID: "WORKER-FIXED",
+		Accessor: AuditAccessor{
+			Kind:    "gateway_lookup",
+			Daemon:  "gateway",
+			Purpose: "parity_test",
+			TraceID: "trace-fixed",
+		},
+		FieldsAccessed: []string{"name"},
+		Result:         "success",
+		PrevChainHash:  GenesisHash,
+		RowHmac:        "",
+	}
+	canon, err := canonicalRowBytesFromStruct(&r)
+	if err != nil {
+		t.Fatalf("canonical: %v", err)
+	}
+	t.Logf("canonical bytes: %s", canon)
+	hmacHex := computeRowHMAC(key, GenesisHash, canon)
+	t.Logf("hmac: %s", hmacHex)
+	// Sanity: round-trip through encoding/json + canonicalization is stable.
+	again, err := canonicalRowBytesFromStruct(&r)
+	if err != nil {
+		t.Fatalf("canonical 2: %v", err)
+	}
+	if string(canon) != string(again) {
+		t.Fatalf("canonical drift: %s vs %s", canon, again)
+	}
+	// Sanity: real HMAC against the canonical bytes.
+	mac := hmac.New(sha256.New, key)
+	mac.Write([]byte(GenesisHash))
+	mac.Write(canon)
+	expected := hex.EncodeToString(mac.Sum(nil))
+	if hmacHex != expected {
+		t.Fatalf("computeRowHMAC drift: %s vs %s", hmacHex, expected)
+	}
+}
+
+// TestVerifyChain_HtmlChars_NotEscaped is the regression test for the
+// 2026-05-03 opus scrum WARN: Go's json.Marshal escapes `<`, `>`, `&`
+// to `<`, `>`, `&` by default; Rust's serde_json keeps
+// them literal. Audit rows with these chars in any string field would
+// silently break the chain across runtimes. Fix is in writeCanonical's
+// marshalNoEscapeHTML helper. This test asserts canonical bytes contain
+// the literal `<`, `>`, `&` (proving the fix is in place).
+func TestVerifyChain_HtmlChars_NotEscaped(t *testing.T) {
+	r := mkRow("CAND-HTML", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
+	r.Accessor.Purpose = "error & retry"   // & must NOT be &
+	r.Accessor.TraceID = "<HTTP-Req-Id>"   // < and > must NOT be < / >
+	canon, err := canonicalRowBytesFromStruct(&r)
+	if err != nil {
+		t.Fatalf("canonical: %v", err)
+	}
+	s := string(canon)
+	// FAIL if the bytes contain Go's HTML-safe < / > / &
+	// escape sequences (six raw chars each: backslash, u, 0, 0, hex, hex).
+	// Those wouldn't match Rust's literal-char output and would silently
+	// break the cross-runtime HMAC chain. Note: the strings below are
+	// raw-string literals — the backslash + u006xx is six literal bytes,
+	// NOT a Go-source unicode escape.
+	if strings.Contains(s, "\\u003c") || strings.Contains(s, "\\u003e") || strings.Contains(s, "\\u0026") {
+		t.Fatalf("canonical bytes contain Go HTML-escape sequences (would diverge from Rust):\n%s", s)
+	}
+	// PASS only if the literal chars survived round-trip.
+	if !strings.Contains(s, "\"<HTTP-Req-Id>\"") || !strings.Contains(s, "\"error & retry\"") {
+		t.Fatalf("canonical bytes missing literal <>&:\n%s", s)
+	}
+}
+
+// TestVerifyChain_RawBytesPreserveTimePrecision is the regression test
+// for the 2026-05-03 WORKER-5 finding: when a row's nanoseconds end in
+// 0, time.RFC3339Nano strips the trailing zero on re-marshal, producing
+// different canonical bytes than Rust's chrono AutoSi (which always
+// emits 9 digits). VerifyChain MUST canonicalize from the raw line
+// bytes to avoid this drift. Test feeds a hand-crafted raw line whose
+// ts has a trailing-zero nano value and asserts verify succeeds when
+// the chain hash was computed against THOSE EXACT bytes.
+func TestVerifyChain_RawBytesPreserveTimePrecision(t *testing.T) {
+	key := deterministicKey()
+	// Hand-crafted raw line exactly as Rust would write it, with
+	// nanoseconds=461439210 (trailing zero present).
+	rawNoHmac := `{"schema":"subject_audit.v1","ts":"2026-05-03T09:12:47.461439210Z","candidate_id":"WORKER-5","accessor":{"kind":"validator_lookup","daemon":"gateway","purpose":"validator_worker_lookup","trace_id":""},"fields_accessed":["exists"],"result":"not_found","prev_chain_hash":"GENESIS"}`
+	canonical, err := canonicalRowBytesFromRaw([]byte(rawNoHmac))
+	if err != nil {
+		t.Fatalf("canonicalize raw: %v", err)
+	}
+	hmacHex := computeRowHMAC(key, GenesisHash, canonical)
+	// Compose the full row by injecting row_hmac at the end (matches
+	// what the Rust writer produces — declaration order + appended hmac).
+	rawFull := strings.TrimSuffix(rawNoHmac, "}") + `,"row_hmac":"` + hmacHex + `"}`
+
+	var row SubjectAuditRow
+	if err := json.Unmarshal([]byte(rawFull), &row); err != nil {
+		t.Fatalf("unmarshal: %v", err)
+	}
+	entry := AuditLogEntry{Row: row, Raw: []byte(rawFull)}
+	count, tip, err := VerifyChain([]AuditLogEntry{entry}, key)
+	if err != nil {
+		t.Fatalf("verify failed (regression: time-precision drift): %v", err)
+	}
+	if count != 1 {
+		t.Fatalf("expected 1 row verified, got %d", count)
+	}
+	if tip != hmacHex {
+		t.Fatalf("tip mismatch: %s vs %s", tip, hmacHex)
+	}
+}
+
+// TestSubjectManifest_RoundTripJSON: parse a fixture JSON identical in
+// shape to what crates/catalogd/src/registry.rs::put_subject writes to
+// data/_catalog/subjects/<id>.json. If this fails, the Go reader is
+// out of sync with the Rust writer (a Step 8 contract violation).
+func TestSubjectManifest_RoundTripJSON(t *testing.T) {
+	src := `{
+  "schema": "subject_manifest.v1",
+  "candidate_id": "WORKER-1",
+  "created_at": "2026-05-03T08:22:24.571647177Z",
+  "updated_at": "2026-05-03T08:22:24.571647177Z",
+  "status": "active",
+  "vertical": "unknown",
+  "consent": {
+    "general_pii": {
+      "status": "pending_backfill_review",
+      "version": ""
+    },
+    "biometric": {
+      "status": "never_collected"
+    }
+  },
+  "retention": {
+    "general_pii_until": "2030-05-02T08:22:24.571647177Z",
+    "policy": "4_year_default"
+  },
+  "datasets": [
+    {"name": "workers_500k", "key_column": "worker_id", "key_value": "1"}
+  ],
+  "safe_views": ["workers_safe"],
+  "audit_log_path": "_catalog/subjects/WORKER-1.audit.jsonl",
+  "audit_log_chain_root": ""
+}`
+	var m SubjectManifest
+	if err := json.Unmarshal([]byte(src), &m); err != nil {
+		t.Fatalf("unmarshal: %v", err)
+	}
+	if m.CandidateID != "WORKER-1" {
+		t.Fatalf("candidate_id wrong: %s", m.CandidateID)
+	}
+	if m.Status != "active" {
+		t.Fatalf("status wrong: %s", m.Status)
+	}
+	if m.Consent.GeneralPii.Status != "pending_backfill_review" {
+		t.Fatalf("general_pii.status wrong: %s", m.Consent.GeneralPii.Status)
+	}
+	if m.Consent.Biometric.Status != "never_collected" {
+		t.Fatalf("biometric.status wrong: %s", m.Consent.Biometric.Status)
+	}
+	if m.Retention.Policy != "4_year_default" {
+		t.Fatalf("retention.policy wrong: %s", m.Retention.Policy)
+	}
+	if len(m.Datasets) != 1 || m.Datasets[0].Name != "workers_500k" {
+		t.Fatalf("datasets wrong: %+v", m.Datasets)
+	}
+}

internal/chat/builder.go

@@ -0,0 +1,111 @@
+package chat
+
+import (
+	"bufio"
+	"log/slog"
+	"os"
+	"strings"
+	"time"
+)
+
+// BuilderInput drives provider construction. Each field maps to one
+// provider; empty fields mean "skip" (the registry won't have that
+// provider — :cloud suffix or openrouter/* prefixes will 404 cleanly).
+type BuilderInput struct {
+	OllamaURL       string        // local Ollama, no auth (typically http://localhost:11434)
+	OllamaCloudKey  string        // OLLAMA_CLOUD_KEY
+	OpenRouterKey   string        // OPENROUTER_API_KEY
+	OpenCodeKey     string        // OPENCODE_API_KEY
+	KimiKey         string        // KIMI_API_KEY
+	Timeout         time.Duration // default 180s
+}
+
+// BuildRegistry constructs a Registry from the input. Logs which
+// providers were registered (for operator confidence at boot).
+func BuildRegistry(in BuilderInput) *Registry {
+	if in.Timeout == 0 {
+		in.Timeout = 180 * time.Second
+	}
+
+	var providers []Provider
+	registered := []string{}
+
+	// Local Ollama always registered if URL given (no auth needed).
+	if in.OllamaURL != "" {
+		providers = append(providers, NewOllama(in.OllamaURL, in.Timeout))
+		registered = append(registered, "ollama")
+	}
+	if in.OllamaCloudKey != "" {
+		providers = append(providers, NewOllamaCloud(in.OllamaCloudKey, in.Timeout))
+		registered = append(registered, "ollama_cloud")
+	}
+	if in.OpenRouterKey != "" {
+		providers = append(providers, NewOpenRouter(in.OpenRouterKey, in.Timeout))
+		registered = append(registered, "openrouter")
+	}
+	if in.OpenCodeKey != "" {
+		providers = append(providers, NewOpenCode(in.OpenCodeKey, in.Timeout))
+		registered = append(registered, "opencode")
+	}
+	if in.KimiKey != "" {
+		providers = append(providers, NewKimi(in.KimiKey, in.Timeout))
+		registered = append(registered, "kimi")
+	}
+
+	r := NewRegistry(providers...)
+	slog.Info("chat registry built", "providers", registered)
+	return r
+}
+
+// ResolveKey reads an API key with the priority chain:
+//   1. Explicit env var (named by envVar)
+//   2. .env file at envFilePath, looking for "<envVar>=<value>" line
+//   3. "" if neither set
+//
+// Mirrors the Rust adapter's resolve_*_key() pattern. Empty key
+// means the provider stays unregistered — operators see one fewer
+// entry in the boot log instead of a 503 at first request.
+//
+// Phase 4 scrum fix B-1 (Opus + Kimi convergent): collapsed from
+// 3-arg API to 2-arg. Previous version forced callers to pass the
+// env var name twice (once for env-lookup, once for file-key-lookup),
+// which let operator renames silently miss the file fallback. Now
+// both lookups use the same canonical name. Operators who want a
+// different file-key-name can use a different env var altogether.
+func ResolveKey(envVar, envFilePath string) string {
+	if envVar != "" {
+		if v := strings.TrimSpace(os.Getenv(envVar)); v != "" {
+			return v
+		}
+	}
+	if envFilePath != "" && envVar != "" {
+		if v := readEnvFileVar(envFilePath, envVar); v != "" {
+			return v
+		}
+	}
+	return ""
+}
+
+// readEnvFileVar reads a KEY=value style env file and returns the
+// value of `name`. Returns "" on any error or missing key. Stops at
+// first match. No quoting/escaping — same simple shape that systemd
+// EnvironmentFile= reads.
+func readEnvFileVar(path, name string) string {
+	f, err := os.Open(path)
+	if err != nil {
+		return ""
+	}
+	defer f.Close()
+	scanner := bufio.NewScanner(f)
+	prefix := name + "="
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		if strings.HasPrefix(line, prefix) {
+			return strings.Trim(strings.TrimPrefix(line, prefix), `"'`)
+		}
+	}
+	return ""
+}

internal/chat/ollama.go

@@ -0,0 +1,157 @@
+package chat
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// Ollama (local) provider — calls /api/chat on the local Ollama
+// server. No auth needed; default URL http://localhost:11434.
+//
+// Bare model names route here by default (registry.defaultName=ollama),
+// so "qwen3.5:latest" → ollama. Explicit "ollama/qwen3.5:latest" also
+// works (prefix stripped).
+type Ollama struct {
+	baseURL    string
+	httpClient *http.Client
+}
+
+// NewOllama returns a local Ollama provider. baseURL defaults to
+// http://localhost:11434 when empty. timeout 0 → 180s.
+func NewOllama(baseURL string, timeout time.Duration) *Ollama {
+	if baseURL == "" {
+		baseURL = "http://localhost:11434"
+	}
+	if timeout == 0 {
+		timeout = 180 * time.Second
+	}
+	return &Ollama{
+		baseURL:    strings.TrimRight(baseURL, "/"),
+		httpClient: &http.Client{Timeout: timeout},
+	}
+}
+
+func (o *Ollama) Name() string { return "ollama" }
+
+// Available pings /api/tags. Cached negative result would be a
+// premature optimization for G0 — Ollama is typically up. If down,
+// next call gets ErrUpstream which is the right signal anyway.
+func (o *Ollama) Available() bool {
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	req, _ := http.NewRequestWithContext(ctx, "GET", o.baseURL+"/api/tags", nil)
+	resp, err := o.httpClient.Do(req)
+	if err != nil {
+		return false
+	}
+	defer resp.Body.Close()
+	return resp.StatusCode/100 == 2
+}
+
+// Chat translates Request to Ollama's /api/chat shape and back.
+// Strips the optional "ollama/" prefix from req.Model.
+func (o *Ollama) Chat(ctx context.Context, req Request) (*Response, error) {
+	model := StripPrefix(req.Model, "ollama")
+
+	options := map[string]any{}
+	// Pointer-valued temperature so "not set" (nil) doesn't overwrite
+	// Ollama's default. Only forward when caller set it explicitly.
+	if req.Temperature != nil {
+		options["temperature"] = *req.Temperature
+	}
+	if req.MaxTokens > 0 {
+		options["num_predict"] = req.MaxTokens
+	}
+	body := map[string]any{
+		"model":    model,
+		"messages": req.Messages,
+		"stream":   false,
+		// Local hot path: skip reasoning by default. qwen3 / qwen3.5 are
+		// thinking-capable but the inner-loop use case wants direct
+		// answers, not reasoning traces. Without this, low max_tokens
+		// budgets get consumed by thinking before any content is
+		// produced. Cloud tier (Ollama Cloud) inherits the same default
+		// — see ollama_cloud.go.
+		"think":   false,
+		"options": options,
+	}
+	if req.Format == "json" {
+		body["format"] = "json"
+	}
+
+	bs, _ := json.Marshal(body)
+	httpReq, err := http.NewRequestWithContext(ctx, "POST", o.baseURL+"/api/chat", bytes.NewReader(bs))
+	if err != nil {
+		return nil, err
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+
+	resp, err := o.httpClient.Do(httpReq)
+	if err != nil {
+		if errors.Is(ctx.Err(), context.DeadlineExceeded) {
+			return nil, fmt.Errorf("%w: %s", ErrTimeout, "ollama")
+		}
+		return nil, fmt.Errorf("ollama: %w", err)
+	}
+	defer resp.Body.Close()
+
+	rb, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode/100 != 2 {
+		return nil, fmt.Errorf("%w: ollama %d: %s", ErrUpstream, resp.StatusCode, abbrev(string(rb), 200))
+	}
+
+	var ollamaResp struct {
+		Model   string `json:"model"`
+		Message struct {
+			Content string `json:"content"`
+		} `json:"message"`
+		Done            bool   `json:"done"`
+		DoneReason      string `json:"done_reason"` // "stop", "length", ...
+		PromptEvalCount int    `json:"prompt_eval_count"`
+		EvalCount       int    `json:"eval_count"`
+	}
+	if err := json.Unmarshal(rb, &ollamaResp); err != nil {
+		return nil, fmt.Errorf("ollama decode: %w (body=%s)", err, abbrev(string(rb), 200))
+	}
+
+	return &Response{
+		Model:        model,
+		Content:      ollamaResp.Message.Content,
+		InputTokens:  ollamaResp.PromptEvalCount,
+		OutputTokens: ollamaResp.EvalCount,
+		FinishReason: finishReasonFromOllama(ollamaResp.Done, ollamaResp.DoneReason),
+	}, nil
+}
+
+// finishReasonFromOllama prefers Ollama's done_reason when present
+// (newer Ollama 0.4+ exposes this on /api/chat). Falls back to the
+// done bool for older versions. Phase 4 scrum fix B-4 (Opus WARN):
+// previous logic mapped `done==true` → "stop" unconditionally, which
+// hid truncations that Ollama reports as `done=true, done_reason="length"`.
+// The playbook_lift judge needs this signal to detect when max_tokens
+// budget ran out before the answer completed.
+func finishReasonFromOllama(done bool, doneReason string) string {
+	if doneReason != "" {
+		return doneReason
+	}
+	if done {
+		return "stop"
+	}
+	return "length"
+}
+
+// abbrev shortens long error bodies for log/error messages without
+// pulling fmt's truncation flags everywhere.
+func abbrev(s string, n int) string {
+	if len(s) <= n {
+		return s
+	}
+	return s[:n] + "…"
+}

internal/chat/ollama_cloud.go

@@ -0,0 +1,164 @@
+package chat
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// OllamaCloud — Ollama Cloud (Pro plan as of 2026-04-28). Bearer
+// auth via OLLAMA_CLOUD_KEY. Wire format mirrors the Rust adapter
+// (crates/gateway/src/v1/ollama_cloud.rs) — system/prompt split
+// against /api/generate with cloud-friendly num_predict floor.
+//
+// Model routing: "ollama_cloud/<model>" (prefix) and "<model>:cloud"
+// (suffix) both route here. Registry.Resolve handles the suffix case;
+// the prefix case strips "ollama_cloud/" before the upstream call so
+// ollama.com sees the bare model name. Suffix-form models pass through
+// StripPrefix unchanged (no leading prefix to strip — exactly what the
+// upstream wants since "kimi-k2.6:cloud" is the canonical name on
+// ollama.com).
+type OllamaCloud struct {
+	apiKey     string
+	baseURL    string
+	httpClient *http.Client
+}
+
+func NewOllamaCloud(apiKey string, timeout time.Duration) *OllamaCloud {
+	if timeout == 0 {
+		timeout = 180 * time.Second
+	}
+	return &OllamaCloud{
+		apiKey:     apiKey,
+		baseURL:    "https://ollama.com",
+		httpClient: &http.Client{Timeout: timeout},
+	}
+}
+
+func (o *OllamaCloud) Name() string    { return "ollama_cloud" }
+func (o *OllamaCloud) Available() bool { return o.apiKey != "" }
+
+func (o *OllamaCloud) Chat(ctx context.Context, req Request) (*Response, error) {
+	// Strip "ollama_cloud/" prefix (Phase 4 scrum fix B-3 — Opus BLOCK).
+	// Pre-fix used "cloud" which never matched, so upstream got the
+	// prefixed model name and 400'd. ":cloud" suffix models pass
+	// through unchanged — that's the canonical name on ollama.com.
+	model := StripPrefix(req.Model, "ollama_cloud")
+
+	system, prompt := flattenMessages(req.Messages)
+
+	body := map[string]any{
+		"model":  model,
+		"prompt": prompt,
+		"stream": false,
+		// Mirror local Ollama — skip reasoning by default. Callers that
+		// need thinking explicitly (e.g. for Kimi K2 long reasoning)
+		// will get a future Request.Think field; the v0 default keeps
+		// outputs short and predictable.
+		"think": false,
+		"options": map[string]any{
+			// Cloud reasoning models need headroom — 400 floor matches
+			// the Rust adapter's policy for kimi-k2.6 / gpt-oss:120b.
+			"num_predict": maxInt(req.MaxTokens, 400),
+			// Cloud-tier default 0.3 when caller didn't set; honor an
+			// explicit value otherwise. Anthropic 4.7 deprecation
+			// doesn't apply here (Ollama Cloud is its own server),
+			// so always send.
+			"temperature": tempPtrOr(req.Temperature, 0.3),
+		},
+	}
+	if system != "" {
+		body["system"] = system
+	}
+	if req.Format == "json" {
+		body["format"] = "json"
+	}
+
+	bs, _ := json.Marshal(body)
+	httpReq, err := http.NewRequestWithContext(ctx, "POST", o.baseURL+"/api/generate", bytes.NewReader(bs))
+	if err != nil {
+		return nil, err
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Authorization", "Bearer "+o.apiKey)
+
+	resp, err := o.httpClient.Do(httpReq)
+	if err != nil {
+		if errors.Is(ctx.Err(), context.DeadlineExceeded) {
+			return nil, fmt.Errorf("%w: ollama_cloud", ErrTimeout)
+		}
+		return nil, fmt.Errorf("ollama_cloud: %w", err)
+	}
+	defer resp.Body.Close()
+
+	rb, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode/100 != 2 {
+		return nil, fmt.Errorf("%w: ollama_cloud %d: %s", ErrUpstream, resp.StatusCode, abbrev(string(rb), 300))
+	}
+
+	var cloudResp struct {
+		Model           string `json:"model"`
+		Response        string `json:"response"`
+		Done            bool   `json:"done"`
+		DoneReason      string `json:"done_reason"`
+		PromptEvalCount int    `json:"prompt_eval_count"`
+		EvalCount       int    `json:"eval_count"`
+	}
+	if err := json.Unmarshal(rb, &cloudResp); err != nil {
+		return nil, fmt.Errorf("ollama_cloud decode: %w (body=%s)", err, abbrev(string(rb), 200))
+	}
+
+	return &Response{
+		Model:        model,
+		Content:      cloudResp.Response,
+		InputTokens:  cloudResp.PromptEvalCount,
+		OutputTokens: cloudResp.EvalCount,
+		FinishReason: finishReasonFromOllama(cloudResp.Done, cloudResp.DoneReason),
+	}, nil
+}
+
+// flattenMessages splits a Message list into Ollama Cloud's
+// /api/generate shape (single system + concatenated prompt).
+// Mirrors the Rust adapter's flatten_messages_public.
+func flattenMessages(messages []Message) (system, prompt string) {
+	var sysParts []string
+	var promptParts []string
+	for _, m := range messages {
+		switch m.Role {
+		case "system":
+			sysParts = append(sysParts, m.Content)
+		case "assistant":
+			promptParts = append(promptParts, "Assistant: "+m.Content)
+		default: // "user" or anything else
+			promptParts = append(promptParts, "User: "+m.Content)
+		}
+	}
+	return strings.Join(sysParts, "\n\n"), strings.Join(promptParts, "\n\n")
+}
+
+func maxInt(a, b int) int {
+	if a > b {
+		return a
+	}
+	return b
+}
+
+func defaultFloat(v, fallback float64) float64 {
+	if v == 0 {
+		return fallback
+	}
+	return v
+}
+
+func tempPtrOr(v *float64, fallback float64) float64 {
+	if v == nil {
+		return fallback
+	}
+	return *v
+}

internal/chat/ollama_test.go

@@ -0,0 +1,128 @@
+package chat
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+// ollamaServer mocks /api/chat + /api/tags. Captures last request body.
+func ollamaServer(t *testing.T, status int, respBody string) (*httptest.Server, *string) {
+	t.Helper()
+	captured := ""
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/api/tags":
+			w.WriteHeader(200)
+			_, _ = w.Write([]byte(`{"models":[{"name":"qwen3.5:latest"}]}`))
+			return
+		case "/api/chat":
+			bs, _ := io.ReadAll(r.Body)
+			captured = string(bs)
+			w.WriteHeader(status)
+			_, _ = w.Write([]byte(respBody))
+			return
+		default:
+			w.WriteHeader(404)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv, &captured
+}
+
+func TestOllama_ChatHappyPath(t *testing.T) {
+	resp := `{
+		"model": "qwen3.5:latest",
+		"message": {"content": "ok"},
+		"done": true,
+		"prompt_eval_count": 7,
+		"eval_count": 3
+	}`
+	srv, captured := ollamaServer(t, 200, resp)
+	o := NewOllama(srv.URL, 5*time.Second)
+
+	out, err := o.Chat(context.Background(), Request{
+		Model:    "qwen3.5:latest",
+		Messages: []Message{{Role: "user", Content: "hi"}},
+	})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	if out.Content != "ok" {
+		t.Errorf("Content = %q, want ok", out.Content)
+	}
+	if out.InputTokens != 7 || out.OutputTokens != 3 {
+		t.Errorf("tokens = (%d, %d), want (7, 3)", out.InputTokens, out.OutputTokens)
+	}
+
+	// Verify captured request shape: stream=false, options.temperature
+	// surfaced.
+	var sent map[string]any
+	if err := json.Unmarshal([]byte(*captured), &sent); err != nil {
+		t.Fatalf("parse captured: %v", err)
+	}
+	if sent["stream"] != false {
+		t.Errorf("stream should be false, got %v", sent["stream"])
+	}
+}
+
+func TestOllama_StripsExplicitPrefix(t *testing.T) {
+	srv, captured := ollamaServer(t, 200, `{"message":{"content":""},"done":true}`)
+	o := NewOllama(srv.URL, 5*time.Second)
+	_, err := o.Chat(context.Background(), Request{Model: "ollama/qwen3.5:latest"})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	var sent map[string]any
+	_ = json.Unmarshal([]byte(*captured), &sent)
+	if sent["model"] != "qwen3.5:latest" {
+		t.Errorf("upstream model = %v, want qwen3.5:latest (prefix stripped)", sent["model"])
+	}
+}
+
+func TestOllama_FormatJSON(t *testing.T) {
+	srv, captured := ollamaServer(t, 200, `{"message":{"content":"{}"},"done":true}`)
+	o := NewOllama(srv.URL, 5*time.Second)
+	_, _ = o.Chat(context.Background(), Request{Model: "qwen3.5:latest", Format: "json"})
+	if !strings.Contains(*captured, `"format":"json"`) {
+		t.Errorf("Format=json should set top-level format=json; captured=%s", *captured)
+	}
+}
+
+func TestOllama_Available(t *testing.T) {
+	srv, _ := ollamaServer(t, 200, "{}")
+	o := NewOllama(srv.URL, 5*time.Second)
+	if !o.Available() {
+		t.Errorf("server is up; Available should be true")
+	}
+}
+
+func TestOllama_UpstreamError(t *testing.T) {
+	srv, _ := ollamaServer(t, 500, `{"error":"out of memory"}`)
+	o := NewOllama(srv.URL, 5*time.Second)
+	_, err := o.Chat(context.Background(), Request{Model: "qwen3.5:latest"})
+	if !errors.Is(err, ErrUpstream) {
+		t.Errorf("500 should ErrUpstream; got %v", err)
+	}
+}
+
+func TestOllamaCloud_FlattenMessages(t *testing.T) {
+	system, prompt := flattenMessages([]Message{
+		{Role: "system", Content: "You are helpful."},
+		{Role: "user", Content: "hi"},
+		{Role: "assistant", Content: "hello!"},
+		{Role: "user", Content: "how are you?"},
+	})
+	if system != "You are helpful." {
+		t.Errorf("system = %q, want 'You are helpful.'", system)
+	}
+	if !strings.Contains(prompt, "User: hi") || !strings.Contains(prompt, "Assistant: hello!") || !strings.Contains(prompt, "User: how are you?") {
+		t.Errorf("prompt missing role tags: %q", prompt)
+	}
+}

internal/chat/openai_compat.go

@@ -0,0 +1,149 @@
+package chat
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// openaiCompat implements the OpenAI Chat Completions wire format,
+// which OpenRouter / OpenCode / Kimi all speak. Differences between
+// those three providers are limited to:
+//   - base URL (e.g. /v1/chat/completions vs /zen/v1/chat/completions)
+//   - prefix stripped from req.Model
+//   - bearer token source (different env vars)
+//
+// Each concrete provider wraps this with its own constructor pinning
+// those values.
+type openaiCompat struct {
+	name        string
+	baseURL     string
+	apiKey      string
+	prefix      string
+	httpClient  *http.Client
+}
+
+func newOpenAICompat(name, baseURL, apiKey, prefix string, timeout time.Duration) *openaiCompat {
+	if timeout == 0 {
+		timeout = 180 * time.Second
+	}
+	return &openaiCompat{
+		name:       name,
+		baseURL:    strings.TrimRight(baseURL, "/"),
+		apiKey:     apiKey,
+		prefix:     prefix,
+		httpClient: &http.Client{Timeout: timeout},
+	}
+}
+
+func (c *openaiCompat) Name() string    { return c.name }
+func (c *openaiCompat) Available() bool { return c.apiKey != "" }
+
+func (c *openaiCompat) Chat(ctx context.Context, req Request) (*Response, error) {
+	model := StripPrefix(req.Model, c.prefix)
+
+	body := map[string]any{
+		"model":    model,
+		"messages": req.Messages,
+		"stream":   false,
+	}
+	// Anthropic 4.7 (via OpenCode → Anthropic) rejects the temperature
+	// field entirely with "temperature is deprecated for this model".
+	// Send only when caller explicitly set it. Nil = use upstream
+	// default. Keep this true for ALL openai-compat providers — being
+	// permissive about field omission is better than maintaining a
+	// per-model deprecation table.
+	if req.Temperature != nil {
+		body["temperature"] = *req.Temperature
+	}
+	if req.MaxTokens > 0 {
+		body["max_tokens"] = req.MaxTokens
+	}
+	if req.Format == "json" {
+		// OpenAI-compat response_format. OpenRouter passes through;
+		// OpenCode + Kimi accept it. Worst case the upstream ignores it.
+		body["response_format"] = map[string]any{"type": "json_object"}
+	}
+
+	bs, _ := json.Marshal(body)
+	url := c.baseURL + "/chat/completions"
+	httpReq, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(bs))
+	if err != nil {
+		return nil, err
+	}
+	httpReq.Header.Set("Content-Type", "application/json")
+	httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
+	// HTTP-Referer + X-Title are OpenRouter-specific but harmless on
+	// other providers — they pass them through to upstream attribution.
+	httpReq.Header.Set("HTTP-Referer", "https://golanglakehouse.local")
+	httpReq.Header.Set("X-Title", "Lakehouse-Go")
+
+	resp, err := c.httpClient.Do(httpReq)
+	if err != nil {
+		if errors.Is(ctx.Err(), context.DeadlineExceeded) {
+			return nil, fmt.Errorf("%w: %s", ErrTimeout, c.name)
+		}
+		return nil, fmt.Errorf("%s: %w", c.name, err)
+	}
+	defer resp.Body.Close()
+
+	rb, _ := io.ReadAll(resp.Body)
+	if resp.StatusCode/100 != 2 {
+		return nil, fmt.Errorf("%w: %s %d: %s", ErrUpstream, c.name, resp.StatusCode, abbrev(string(rb), 300))
+	}
+
+	var oaResp struct {
+		Model   string `json:"model"`
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+			FinishReason string `json:"finish_reason"`
+		} `json:"choices"`
+		Usage struct {
+			PromptTokens     int `json:"prompt_tokens"`
+			CompletionTokens int `json:"completion_tokens"`
+		} `json:"usage"`
+	}
+	if err := json.Unmarshal(rb, &oaResp); err != nil {
+		return nil, fmt.Errorf("%s decode: %w (body=%s)", c.name, err, abbrev(string(rb), 200))
+	}
+	if len(oaResp.Choices) == 0 {
+		return nil, fmt.Errorf("%w: %s returned 0 choices: %s", ErrUpstream, c.name, abbrev(string(rb), 200))
+	}
+
+	return &Response{
+		Model:        model,
+		Content:      oaResp.Choices[0].Message.Content,
+		InputTokens:  oaResp.Usage.PromptTokens,
+		OutputTokens: oaResp.Usage.CompletionTokens,
+		FinishReason: oaResp.Choices[0].FinishReason,
+	}, nil
+}
+
+// NewOpenRouter returns a provider for openrouter.ai. apiKey
+// resolved at construction (typically from OPENROUTER_API_KEY env or
+// /etc/lakehouse/openrouter.env).
+func NewOpenRouter(apiKey string, timeout time.Duration) *openaiCompat {
+	return newOpenAICompat("openrouter", "https://openrouter.ai/api/v1", apiKey, "openrouter", timeout)
+}
+
+// NewOpenCode returns a provider for OpenCode's unified Zen+Go endpoint.
+// One key reaches Anthropic Opus, GPT-5, Gemini 3.1, Kimi K2.6,
+// DeepSeek, GLM, Qwen, plus 4 free-tier models.
+func NewOpenCode(apiKey string, timeout time.Duration) *openaiCompat {
+	return newOpenAICompat("opencode", "https://opencode.ai/zen/v1", apiKey, "opencode", timeout)
+}
+
+// NewKimi returns a provider for the direct Kimi For Coding endpoint.
+// `api.kimi.com` is a separate account system from api.moonshot.ai —
+// keys are NOT interchangeable.
+func NewKimi(apiKey string, timeout time.Duration) *openaiCompat {
+	return newOpenAICompat("kimi", "https://api.kimi.com/coding/v1", apiKey, "kimi", timeout)
+}

internal/chat/openai_compat_test.go

@@ -0,0 +1,227 @@
+package chat
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+// openaiServer returns an httptest server that mocks an OpenAI Chat
+// Completions endpoint. Handler captures the last request body for
+// assertion.
+func openaiServer(t *testing.T, status int, respBody string) (*httptest.Server, *string) {
+	t.Helper()
+	captured := ""
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		bs, _ := io.ReadAll(r.Body)
+		captured = string(bs)
+		// Ensure Authorization header was set (Bearer prefix).
+		if auth := r.Header.Get("Authorization"); !strings.HasPrefix(auth, "Bearer ") {
+			t.Errorf("missing Bearer auth header; got %q", auth)
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(status)
+		_, _ = w.Write([]byte(respBody))
+	}))
+	t.Cleanup(srv.Close)
+	return srv, &captured
+}
+
+func TestOpenAICompat_HappyPath(t *testing.T) {
+	resp := `{
+		"model": "anthropic/claude-opus-4-7",
+		"choices": [{"message":{"content":"hello world"}, "finish_reason":"stop"}],
+		"usage": {"prompt_tokens": 10, "completion_tokens": 5}
+	}`
+	srv, captured := openaiServer(t, 200, resp)
+
+	p := newOpenAICompat("openrouter", srv.URL+"/v1", "test-key", "openrouter", 5*time.Second)
+	out, err := p.Chat(context.Background(), Request{
+		Model:    "openrouter/anthropic/claude-opus-4-7",
+		Messages: []Message{{Role: "user", Content: "hi"}},
+	})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	if out.Content != "hello world" {
+		t.Errorf("Content = %q, want hello world", out.Content)
+	}
+	// Provider strips its prefix before sending to upstream.
+	var sent map[string]any
+	if err := json.Unmarshal([]byte(*captured), &sent); err != nil {
+		t.Fatalf("parse captured: %v", err)
+	}
+	if sent["model"] != "anthropic/claude-opus-4-7" {
+		t.Errorf("upstream got model = %v, want anthropic/claude-opus-4-7 (prefix stripped)", sent["model"])
+	}
+	// Token accounting carried through.
+	if out.InputTokens != 10 || out.OutputTokens != 5 {
+		t.Errorf("tokens = (%d, %d), want (10, 5)", out.InputTokens, out.OutputTokens)
+	}
+	if out.FinishReason != "stop" {
+		t.Errorf("FinishReason = %q, want stop", out.FinishReason)
+	}
+}
+
+func TestOpenAICompat_FormatJSON(t *testing.T) {
+	resp := `{"choices":[{"message":{"content":"{\"a\":1}"},"finish_reason":"stop"}],"usage":{}}`
+	srv, captured := openaiServer(t, 200, resp)
+	p := newOpenAICompat("opencode", srv.URL+"/zen/v1", "test-key", "opencode", 5*time.Second)
+
+	_, err := p.Chat(context.Background(), Request{
+		Model:    "opencode/claude-opus-4-7",
+		Messages: []Message{{Role: "user", Content: "give me JSON"}},
+		Format:   "json",
+	})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	// Format=json should set response_format on the upstream call.
+	if !strings.Contains(*captured, `"response_format"`) || !strings.Contains(*captured, `"json_object"`) {
+		t.Errorf("Format=json should set response_format json_object; captured=%s", *captured)
+	}
+}
+
+func TestOpenAICompat_UpstreamError(t *testing.T) {
+	srv, _ := openaiServer(t, 429, `{"error":"rate limited"}`)
+	p := newOpenAICompat("openrouter", srv.URL+"/v1", "test-key", "openrouter", 5*time.Second)
+	_, err := p.Chat(context.Background(), Request{Model: "openrouter/x"})
+	if !errors.Is(err, ErrUpstream) {
+		t.Errorf("429 should be ErrUpstream; got %v", err)
+	}
+}
+
+func TestOpenAICompat_ZeroChoices(t *testing.T) {
+	srv, _ := openaiServer(t, 200, `{"choices":[],"usage":{}}`)
+	p := newOpenAICompat("openrouter", srv.URL+"/v1", "test-key", "openrouter", 5*time.Second)
+	_, err := p.Chat(context.Background(), Request{Model: "openrouter/x"})
+	if !errors.Is(err, ErrUpstream) {
+		t.Errorf("zero-choices should ErrUpstream; got %v", err)
+	}
+}
+
+func TestOpenAICompat_EmptyKeyUnavailable(t *testing.T) {
+	p := newOpenAICompat("openrouter", "https://example.com", "", "openrouter", 5*time.Second)
+	if p.Available() {
+		t.Errorf("empty key should make provider unavailable")
+	}
+}
+
+// TestOllamaCloud_StripsCorrectPrefix locks in scrum fix B-3.
+// Pre-fix: StripPrefix(req.Model, "cloud") never matched "ollama_cloud/<m>"
+// because the prefix string was "cloud/" not "ollama_cloud/". Result was
+// the upstream got the prefixed model name and 400'd. This test would
+// have caught it had it existed.
+func TestOllamaCloud_StripsCorrectPrefix(t *testing.T) {
+	captured := ""
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		bs, _ := io.ReadAll(r.Body)
+		captured = string(bs)
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"model":"deepseek-v3.2","response":"ok","done":true,"done_reason":"stop"}`))
+	}))
+	t.Cleanup(srv.Close)
+
+	// Inject the test server URL via reflection-free constructor —
+	// just build the struct directly with our overrides since
+	// NewOllamaCloud hardcodes ollama.com.
+	o := &OllamaCloud{
+		apiKey:     "test-key",
+		baseURL:    srv.URL,
+		httpClient: srv.Client(),
+	}
+
+	cases := []struct {
+		input    string
+		expected string // model the upstream should see
+	}{
+		// Prefix form (Phase 4 scrum fix B-3): strip "ollama_cloud/"
+		{"ollama_cloud/deepseek-v3.2", "deepseek-v3.2"},
+		{"ollama_cloud/kimi-k2.6:cloud", "kimi-k2.6:cloud"},
+		// Suffix form: nothing to strip (already canonical)
+		{"kimi-k2.6:cloud", "kimi-k2.6:cloud"},
+		{"qwen3-coder:480b", "qwen3-coder:480b"},
+	}
+	for _, c := range cases {
+		captured = ""
+		_, err := o.Chat(context.Background(), Request{
+			Model:    c.input,
+			Messages: []Message{{Role: "user", Content: "hi"}},
+		})
+		if err != nil {
+			t.Errorf("Chat(%q): %v", c.input, err)
+			continue
+		}
+		var sent map[string]any
+		if err := json.Unmarshal([]byte(captured), &sent); err != nil {
+			t.Fatalf("parse captured: %v", err)
+		}
+		got, _ := sent["model"].(string)
+		if got != c.expected {
+			t.Errorf("Chat(%q): upstream got model=%q, want %q", c.input, got, c.expected)
+		}
+	}
+}
+
+func TestFinishReasonFromOllama_PrefersDoneReason(t *testing.T) {
+	// Phase 4 scrum fix B-4: done_reason takes precedence when present.
+	cases := []struct {
+		done       bool
+		doneReason string
+		want       string
+	}{
+		{true, "stop", "stop"},
+		{true, "length", "length"}, // truncation surfaced — pre-fix lost this
+		{true, "", "stop"},          // legacy fallback
+		{false, "", "length"},
+		{false, "stop", "stop"}, // done_reason wins regardless of done
+	}
+	for _, c := range cases {
+		got := finishReasonFromOllama(c.done, c.doneReason)
+		if got != c.want {
+			t.Errorf("finishReasonFromOllama(%v, %q) = %q, want %q", c.done, c.doneReason, got, c.want)
+		}
+	}
+}
+
+func TestRegistry_AvailableLooksUpDirectly(t *testing.T) {
+	// Phase 4 scrum fix B-2: Available() reports registry membership,
+	// not routing. Unregistered name → false; registered + Available()
+	// → true; registered but empty key → false.
+	registered := newFake("openrouter", true)
+	disabled := newFake("opencode", false) // "registered" but no key
+	r := NewRegistry(registered, disabled)
+	if !r.Available("openrouter") {
+		t.Errorf("openrouter should be Available")
+	}
+	if r.Available("opencode") {
+		t.Errorf("opencode has no key — Available should be false")
+	}
+	if r.Available("nonexistent") {
+		t.Errorf("nonexistent should be false")
+	}
+}
+
+func TestNewProviderConstructors(t *testing.T) {
+	// Smoke: each public constructor produces a working provider with
+	// the right name/prefix and respects empty-key=unavailable.
+	or := NewOpenRouter("test-key", 0)
+	if or.Name() != "openrouter" || !or.Available() {
+		t.Errorf("openrouter constructor wrong: %s avail=%v", or.Name(), or.Available())
+	}
+	oc := NewOpenCode("test-key", 0)
+	if oc.Name() != "opencode" || !oc.Available() {
+		t.Errorf("opencode constructor wrong: %s avail=%v", oc.Name(), oc.Available())
+	}
+	km := NewKimi("", 0) // empty key → unavailable
+	if km.Available() {
+		t.Errorf("kimi with empty key should be unavailable")
+	}
+}

internal/chat/registry.go

@@ -0,0 +1,153 @@
+package chat
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strings"
+	"time"
+)
+
+// Registry resolves a model name to its Provider. Lookup is by the
+// first slash-delimited prefix; bare names (no slash) fall through to
+// the configured default provider — typically `ollama` so local
+// model names like `qwen3.5:latest` work without a prefix.
+//
+// Mirrors the Rust gateway's resolve_provider() pattern from
+// crates/gateway/src/v1/mod.rs.
+type Registry struct {
+	providers      map[string]Provider // name → provider
+	defaultName    string              // resolved when no prefix matches
+}
+
+// NewRegistry builds a registry from a list of providers. The first
+// "ollama" provider becomes the bare-name default; callers can
+// override via SetDefault.
+func NewRegistry(providers ...Provider) *Registry {
+	r := &Registry{providers: make(map[string]Provider, len(providers))}
+	for _, p := range providers {
+		r.providers[p.Name()] = p
+		if r.defaultName == "" && p.Name() == "ollama" {
+			r.defaultName = "ollama"
+		}
+	}
+	return r
+}
+
+// Register adds or replaces a provider. Used after construction (e.g.
+// for tests injecting fakes).
+func (r *Registry) Register(p Provider) {
+	r.providers[p.Name()] = p
+}
+
+// SetDefault sets the provider used when no prefix matches. Empty
+// model names always 404 — the default only kicks in for unprefixed
+// non-empty names.
+func (r *Registry) SetDefault(name string) {
+	r.defaultName = name
+}
+
+// Names returns the registered provider names, sorted (deterministic
+// output for /v1/chat/providers listing).
+func (r *Registry) Names() []string {
+	out := make([]string, 0, len(r.providers))
+	for n := range r.providers {
+		out = append(out, n)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// Available reports whether the named provider is registered AND
+// its Available() method returns true. Returns false for unregistered
+// names — used by /v1/chat/providers (Phase 4 scrum fix B-2: replaces
+// the prior synthetic-route probe that mis-routed via Resolve).
+func (r *Registry) Available(name string) bool {
+	p, ok := r.providers[name]
+	if !ok {
+		return false
+	}
+	return p.Available()
+}
+
+// Resolve returns the Provider for a model name. Resolution rules:
+//
+//   1. Empty model → ErrProviderNotFound
+//   2. Suffix ":cloud" → ollama_cloud (e.g. "kimi-k2.6:cloud")
+//   3. Prefix match (e.g. "openrouter/...") → that provider
+//   4. No prefix or unknown prefix → default provider (typically ollama)
+//   5. No default registered → ErrProviderNotFound
+//
+// The suffix rule mirrors the Rust gateway and the Ollama Cloud
+// upstream's own naming convention — kimi-k2.6:cloud, qwen3-coder:480b
+// (when on cloud) etc. Without it, every cloud model would need a
+// "cloud/" prefix in lakehouse.toml, which clashes with the Ollama
+// upstream that wants the bare suffix-named model.
+func (r *Registry) Resolve(model string) (Provider, error) {
+	if model == "" {
+		return nil, fmt.Errorf("%w: empty model name", ErrProviderNotFound)
+	}
+	// Suffix detection — `:cloud` always means Ollama Cloud.
+	if strings.HasSuffix(model, ":cloud") {
+		if p, ok := r.providers["ollama_cloud"]; ok {
+			return p, nil
+		}
+		// :cloud suffix with no ollama_cloud provider → 404. Don't
+		// silently fall through to local Ollama; that would burn the
+		// model name on a provider that doesn't have it.
+		return nil, fmt.Errorf("%w: %q has :cloud suffix but ollama_cloud provider is not registered", ErrProviderNotFound, model)
+	}
+	// Prefix match: "openrouter/anthropic/claude-opus-4-7" splits on
+	// first "/". Multi-segment provider names not supported (none
+	// shipped use them).
+	if idx := strings.Index(model, "/"); idx > 0 {
+		prefix := model[:idx]
+		if p, ok := r.providers[prefix]; ok {
+			return p, nil
+		}
+		// Unknown prefix — falls through to default. Lets bare model
+		// names with slashes (e.g. "anthropic/claude-3.5") still hit
+		// ollama if that's how the operator named local models.
+	}
+	if r.defaultName == "" {
+		return nil, fmt.Errorf("%w: %q", ErrProviderNotFound, model)
+	}
+	p, ok := r.providers[r.defaultName]
+	if !ok {
+		return nil, fmt.Errorf("%w: default provider %q not registered", ErrProviderNotFound, r.defaultName)
+	}
+	return p, nil
+}
+
+// Chat is the dispatcher entry point: resolve provider, dispatch,
+// stamp telemetry on the response. Returns ErrProviderDisabled when
+// the resolved provider isn't Available() (caller should map to 503).
+func (r *Registry) Chat(ctx context.Context, req Request) (*Response, error) {
+	p, err := r.Resolve(req.Model)
+	if err != nil {
+		return nil, err
+	}
+	if !p.Available() {
+		return nil, fmt.Errorf("%w: %s", ErrProviderDisabled, p.Name())
+	}
+	t0 := time.Now()
+	resp, err := p.Chat(ctx, req)
+	if err != nil {
+		return nil, err
+	}
+	resp.LatencyMs = time.Since(t0).Milliseconds()
+	resp.Provider = p.Name()
+	return resp, nil
+}
+
+// StripPrefix removes the leading "<provider>/" from model when
+// present. Helpers for upstream calls — providers that need the bare
+// model name (e.g. OpenRouter sees "anthropic/claude-opus-4-7", not
+// "openrouter/anthropic/claude-opus-4-7") use this.
+func StripPrefix(model, prefix string) string {
+	want := prefix + "/"
+	if strings.HasPrefix(model, want) {
+		return model[len(want):]
+	}
+	return model
+}

internal/chat/registry_test.go

@@ -0,0 +1,168 @@
+package chat
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+)
+
+// fakeProvider is a minimal Provider for registry testing — no HTTP,
+// just records what model name reached it.
+type fakeProvider struct {
+	name      string
+	available bool
+	got       Request // last request
+}
+
+func (f *fakeProvider) Name() string    { return f.name }
+func (f *fakeProvider) Available() bool { return f.available }
+func (f *fakeProvider) Chat(_ context.Context, req Request) (*Response, error) {
+	f.got = req
+	return &Response{Model: req.Model, Content: "ok from " + f.name}, nil
+}
+
+func newFake(name string, available bool) *fakeProvider {
+	return &fakeProvider{name: name, available: available}
+}
+
+func TestRegistry_ResolveByPrefix(t *testing.T) {
+	ollama := newFake("ollama", true)
+	openrouter := newFake("openrouter", true)
+	opencode := newFake("opencode", true)
+	kimi := newFake("kimi", true)
+
+	r := NewRegistry(ollama, openrouter, opencode, kimi)
+
+	cases := []struct {
+		model string
+		want  string
+	}{
+		{"openrouter/anthropic/claude-opus-4-7", "openrouter"},
+		{"opencode/claude-opus-4-7", "opencode"},
+		{"kimi/kimi-for-coding", "kimi"},
+		{"ollama/qwen3.5:latest", "ollama"},  // explicit prefix
+		{"qwen3.5:latest", "ollama"},          // bare → default
+		{"unknown/foo/bar", "ollama"},         // unknown prefix → default
+	}
+	for _, c := range cases {
+		p, err := r.Resolve(c.model)
+		if err != nil {
+			t.Errorf("Resolve(%q): unexpected error: %v", c.model, err)
+			continue
+		}
+		if p.Name() != c.want {
+			t.Errorf("Resolve(%q) = %s, want %s", c.model, p.Name(), c.want)
+		}
+	}
+}
+
+func TestRegistry_ResolveCloudSuffix(t *testing.T) {
+	ollama := newFake("ollama", true)
+	cloud := newFake("ollama_cloud", true)
+	r := NewRegistry(ollama, cloud)
+
+	// :cloud suffix routes to ollama_cloud regardless of any prefix.
+	p, err := r.Resolve("kimi-k2.6:cloud")
+	if err != nil {
+		t.Fatalf("Resolve kimi-k2.6:cloud: %v", err)
+	}
+	if p.Name() != "ollama_cloud" {
+		t.Errorf("kimi-k2.6:cloud should route to ollama_cloud, got %s", p.Name())
+	}
+
+	// Without ollama_cloud registered, :cloud → ErrProviderNotFound
+	// (don't silently fall through to local).
+	rNoCloud := NewRegistry(ollama)
+	if _, err := rNoCloud.Resolve("kimi-k2.6:cloud"); !errors.Is(err, ErrProviderNotFound) {
+		t.Errorf("missing ollama_cloud should ErrProviderNotFound; got %v", err)
+	}
+}
+
+func TestRegistry_ResolveErrors(t *testing.T) {
+	r := NewRegistry()
+
+	// Empty model
+	if _, err := r.Resolve(""); !errors.Is(err, ErrProviderNotFound) {
+		t.Errorf("empty model should ErrProviderNotFound; got %v", err)
+	}
+
+	// No providers registered, any model → 404
+	if _, err := r.Resolve("openrouter/foo"); !errors.Is(err, ErrProviderNotFound) {
+		t.Errorf("unregistered openrouter should 404; got %v", err)
+	}
+	if _, err := r.Resolve("bare-model"); !errors.Is(err, ErrProviderNotFound) {
+		t.Errorf("bare with no default should 404; got %v", err)
+	}
+}
+
+func TestRegistry_ChatStampsTelemetry(t *testing.T) {
+	ollama := newFake("ollama", true)
+	r := NewRegistry(ollama)
+
+	resp, err := r.Chat(context.Background(), Request{Model: "qwen3.5:latest", Messages: []Message{{Role: "user", Content: "hi"}}})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	if resp.Provider != "ollama" {
+		t.Errorf("Provider should be stamped to %q, got %q", "ollama", resp.Provider)
+	}
+	if resp.LatencyMs < 0 {
+		t.Errorf("LatencyMs negative: %d", resp.LatencyMs)
+	}
+}
+
+func TestRegistry_ChatProviderUnavailable(t *testing.T) {
+	openrouter := newFake("openrouter", false) // no key
+	r := NewRegistry(openrouter)
+
+	_, err := r.Chat(context.Background(), Request{Model: "openrouter/foo"})
+	if !errors.Is(err, ErrProviderDisabled) {
+		t.Errorf("unavailable provider should ErrProviderDisabled; got %v", err)
+	}
+}
+
+func TestStripPrefix(t *testing.T) {
+	cases := []struct {
+		model, prefix, want string
+	}{
+		{"openrouter/anthropic/claude", "openrouter", "anthropic/claude"},
+		{"opencode/claude-opus-4-7", "opencode", "claude-opus-4-7"},
+		{"qwen3.5:latest", "ollama", "qwen3.5:latest"},          // no prefix to strip
+		{"ollama/qwen3.5:latest", "ollama", "qwen3.5:latest"},   // explicit ollama prefix
+		{"kimi-k2.6:cloud", "cloud", "kimi-k2.6:cloud"},          // suffix doesn't trigger strip
+	}
+	for _, c := range cases {
+		if got := StripPrefix(c.model, c.prefix); got != c.want {
+			t.Errorf("StripPrefix(%q, %q) = %q, want %q", c.model, c.prefix, got, c.want)
+		}
+	}
+}
+
+func TestRegistry_Names(t *testing.T) {
+	r := NewRegistry(
+		newFake("zz", true),
+		newFake("aa", true),
+		newFake("mm", true),
+	)
+	names := r.Names()
+	if len(names) != 3 || names[0] != "aa" || names[1] != "mm" || names[2] != "zz" {
+		t.Errorf("Names() = %v, want sorted [aa mm zz]", names)
+	}
+}
+
+// Time-stamp sanity — the dispatcher should never produce LatencyMs
+// in the past.
+func TestRegistry_LatencyMonotonic(t *testing.T) {
+	ollama := newFake("ollama", true)
+	r := NewRegistry(ollama)
+	t0 := time.Now()
+	resp, err := r.Chat(context.Background(), Request{Model: "qwen3.5:latest"})
+	if err != nil {
+		t.Fatalf("Chat: %v", err)
+	}
+	elapsed := time.Since(t0).Milliseconds()
+	if resp.LatencyMs > elapsed+1 {
+		t.Errorf("LatencyMs %d > elapsed %d (impossible)", resp.LatencyMs, elapsed)
+	}
+}

internal/chat/types.go

@@ -0,0 +1,114 @@
+// Package chat provides the LLM chat abstraction over multiple
+// providers (local Ollama, Ollama Cloud, OpenRouter, OpenCode, Kimi).
+//
+// Architecture per docs/SPEC.md §1 (chatd) + the small-model pipeline
+// vision (project_small_model_pipeline_vision.md):
+//
+//   - Hot path runs on local Ollama via cheap `qwen3.5:latest` repeats.
+//   - Oversight tier reaches Ollama Cloud (Pro plan) for tasks needing
+//     more capacity (kimi-k2.6:cloud, qwen3-coder:480b, deepseek-v3.2).
+//   - Frontier escalation goes through OpenRouter (claude-opus-4-7,
+//     gpt-5, kimi-k2-0905) or OpenCode (free-tier opus-4-7) for
+//     blockers that need full-scope reasoning.
+//
+// Provider resolution is by model-name prefix (mirroring the Rust
+// gateway's pattern from crates/gateway/src/v1/mod.rs):
+//
+//   ollama/<model>          → ollama (or bare names like "qwen3.5:latest")
+//   ollama_cloud/<model>    → ollama_cloud
+//   openrouter/<vendor>/<m> → openrouter
+//   opencode/<model>        → opencode
+//   kimi/<model>            → kimi
+//
+// The prefix is stripped before the upstream call.
+package chat
+
+import (
+	"context"
+	"errors"
+)
+
+// Sentinel errors for the upper layers (gateway response codes,
+// observability, retry decisions).
+var (
+	// ErrProviderNotFound — model name didn't resolve to any registered
+	// provider. Maps to 404 at the HTTP layer.
+	ErrProviderNotFound = errors.New("provider not found for model")
+
+	// ErrProviderDisabled — the provider was registered but its key
+	// resolved empty. Maps to 503 at the HTTP layer (operator can fix
+	// by setting the env var; not a client bug).
+	ErrProviderDisabled = errors.New("provider disabled (no auth key)")
+
+	// ErrUpstream — the provider returned a non-2xx. Body included in
+	// the wrapped error message. Maps to 502.
+	ErrUpstream = errors.New("provider upstream error")
+
+	// ErrTimeout — provider call exceeded the configured timeout.
+	// Maps to 504.
+	ErrTimeout = errors.New("provider timeout")
+)
+
+// Message is one entry in a Chat request's conversation. Role is
+// "system", "user", or "assistant" — matches OpenAI/Anthropic shapes.
+type Message struct {
+	Role    string `json:"role"`
+	Content string `json:"content"`
+}
+
+// Request is the provider-neutral chat request. Providers translate
+// to their wire format (OpenAI Chat Completions, Ollama /api/chat,
+// Ollama Cloud /api/generate, etc.). Format="json" asks the provider
+// to return JSON-only output (constrained decoding when supported).
+//
+// Temperature is *float64 (pointer) so callers can distinguish "not
+// set" (nil) from "set to 0" (deterministic). Anthropic 4.7 rejects
+// the temperature field entirely, so providers omit it from the
+// upstream call when this is nil.
+type Request struct {
+	Model       string    `json:"model"`
+	Messages    []Message `json:"messages"`
+	Temperature *float64  `json:"temperature,omitempty"`
+	MaxTokens   int       `json:"max_tokens,omitempty"`
+	Format      string    `json:"format,omitempty"` // "" or "json"
+	Stream      bool      `json:"stream,omitempty"` // ignored in G0 — chatd is synchronous
+}
+
+// Temp is a convenience for callers building a Request literal:
+//
+//	chat.Request{Model: "...", Temperature: chat.Temp(0.0), ...}
+func Temp(v float64) *float64 { return &v }
+
+// Response is the provider-neutral chat response. LatencyMs is filled
+// by the dispatcher (Provider implementations don't track it).
+// Provider names the resolved provider for telemetry / debug.
+type Response struct {
+	Model        string `json:"model"`
+	Content      string `json:"content"`
+	InputTokens  int    `json:"input_tokens,omitempty"`
+	OutputTokens int    `json:"output_tokens,omitempty"`
+	FinishReason string `json:"finish_reason,omitempty"`
+	Provider     string `json:"provider"`
+	LatencyMs    int64  `json:"latency_ms"`
+}
+
+// Provider is the contract every backend must implement. Implementations
+// must be safe for concurrent calls — the dispatcher shares one
+// instance across all incoming requests.
+//
+// Chat receives the full Request including the prefixed model name
+// (e.g. "openrouter/anthropic/claude-opus-4-7"). Implementations
+// strip their prefix before the upstream call.
+type Provider interface {
+	// Name returns the provider's short identifier (matches the prefix
+	// in model names). Used for telemetry and Response.Provider.
+	Name() string
+
+	// Chat performs one round-trip. Should respect ctx cancellation.
+	Chat(ctx context.Context, req Request) (*Response, error)
+
+	// Available reports whether the provider is ready to serve. False
+	// means missing auth key or unreachable upstream — dispatcher
+	// returns ErrProviderDisabled for callers using this provider.
+	Available() bool
+}

internal/corpusingest/ingest.go

@@ -0,0 +1,437 @@
+// Package corpusingest is the generalized text→vector ingestion
+// pipeline. Originally extracted from scripts/staffing_500k/main.go;
+// reusable by any corpus-builder script that needs to embed a stream
+// of (id, text, metadata) rows and push them into a vectord index.
+//
+// Design: per-corpus Source impls own the parsing/column-mapping;
+// this package owns the parallel-embed dispatcher, batching, vectord
+// index lifecycle, and progress reporting. Adding a corpus is one
+// Source struct + one main.go that calls Run; no copy-pasted pipeline.
+//
+// Per docs/SPEC.md §3.4 component 1 (corpus builders): this is the
+// substrate the rest of the matrix indexer's value depends on. Get
+// the pipeline right, then iterate on builders.
+package corpusingest
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+// Row is one logical document in a corpus. Metadata may be any
+// JSON-marshalable value (struct, map, json.RawMessage); the library
+// marshals once per row before pushing to vectord.
+type Row struct {
+	ID       string
+	Text     string
+	Metadata any
+}
+
+// Source produces a stream of rows. Source lifecycle (open/close) is
+// owned by the caller; this package only consumes Next() until io.EOF.
+type Source interface {
+	// Next returns the next row or io.EOF when the source is drained.
+	// Other errors cause Run to abort with the error wrapped.
+	Next() (Row, error)
+}
+
+// Config drives one Run. Defaults match the Ollama-on-A4000 sweet
+// spot from the 500K validation; override per-deployment if needed.
+type Config struct {
+	GatewayURL   string // default "http://127.0.0.1:3110"
+	IndexName    string // required
+	Dimension    int    // required, must match the embed model output
+	Distance     string // default "cosine"
+	EmbedModel   string // optional; empty = embedd's default
+	EmbedBatch   int    // default 16, texts per /v1/embed call
+	EmbedWorkers int    // default 8, parallel embed goroutines
+	AddBatch     int    // default 1000, items per /v1/vectors/index/add call
+	Limit        int    // 0 = no limit (process all rows)
+	DropExisting bool   // true = DELETE index first; false = idempotent reuse
+	HTTPClient   *http.Client
+	// LogProgress is the interval between progress logs. 0 disables.
+	LogProgress time.Duration
+}
+
+// Stats reports run outcomes. FailedBatches counts embed-or-add
+// batches that errored out and were skipped (partial-failure
+// semantics). When non-zero, Run returns ErrPartialFailure so
+// callers can't accidentally treat "1 of 313 batches succeeded"
+// as a successful run.
+type Stats struct {
+	Scanned       int64
+	Embedded      int64
+	Added         int64
+	Wall          time.Duration
+	FailedBatches int64
+}
+
+// ErrPartialFailure signals that one or more batches errored during
+// Run. Stats.FailedBatches has the count; the caller decides
+// whether to retry / log / abort. Per 2026-04-29 cross-lineage
+// scrum (Opus WARN): the original behavior returned nil even when
+// 100% of batches failed silently, making "embedded=0/scanned=N"
+// look like an empty corpus rather than a broken pipeline.
+var ErrPartialFailure = errors.New("corpusingest: one or more batches failed")
+
+// Run executes the ingest pipeline. Returns on source EOF after all
+// in-flight jobs drain, on context cancellation, or on the first
+// embed/add error (errors are logged via slog and the pipeline
+// continues — partial-failure semantics; see comment inside).
+func Run(ctx context.Context, cfg Config, src Source) (Stats, error) {
+	cfg = applyDefaults(cfg)
+	if err := validateConfig(cfg); err != nil {
+		return Stats{}, err
+	}
+
+	t0 := time.Now()
+	if err := prepareIndex(ctx, cfg); err != nil {
+		return Stats{}, fmt.Errorf("prepare index: %w", err)
+	}
+
+	jobs := make(chan job, cfg.EmbedWorkers*2)
+
+	var (
+		totalEmbedded int64
+		totalAdded    int64
+		failedBatches int64
+	)
+
+	var wg sync.WaitGroup
+	for i := 0; i < cfg.EmbedWorkers; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for j := range jobs {
+				vecs, err := embedBatch(ctx, cfg, j.texts)
+				if err != nil {
+					// Partial-failure semantics: log + continue. A wedged
+					// embed batch shouldn't kill 8 workers' worth of
+					// progress; Run returns ErrPartialFailure on any
+					// failure so callers can't miss the signal.
+					slog.Warn("corpusingest: embed batch failed",
+						"index", cfg.IndexName, "items", len(j.texts), "err", err)
+					atomic.AddInt64(&failedBatches, 1)
+					continue
+				}
+				// Defense against a degraded embed backend that returns
+				// fewer vectors than texts: vecs[i] would panic in
+				// addBatch otherwise. Caught by ContextCancel unit test.
+				if len(vecs) != len(j.ids) {
+					slog.Warn("corpusingest: embed returned wrong count",
+						"index", cfg.IndexName, "want", len(j.ids), "got", len(vecs))
+					atomic.AddInt64(&failedBatches, 1)
+					continue
+				}
+				atomic.AddInt64(&totalEmbedded, int64(len(vecs)))
+				if err := addBatch(ctx, cfg, j.ids, vecs, j.metas); err != nil {
+					slog.Warn("corpusingest: add batch failed",
+						"index", cfg.IndexName, "items", len(j.ids), "err", err)
+					atomic.AddInt64(&failedBatches, 1)
+					continue
+				}
+				atomic.AddInt64(&totalAdded, int64(len(j.ids)))
+			}
+		}()
+	}
+
+	stopProgress := make(chan struct{})
+	progressDone := make(chan struct{})
+	if cfg.LogProgress > 0 {
+		go func() {
+			defer close(progressDone)
+			ticker := time.NewTicker(cfg.LogProgress)
+			defer ticker.Stop()
+			for {
+				select {
+				case <-ticker.C:
+					slog.Info("corpusingest: progress",
+						"index", cfg.IndexName,
+						"embedded", atomic.LoadInt64(&totalEmbedded),
+						"added", atomic.LoadInt64(&totalAdded))
+				case <-stopProgress:
+					return
+				case <-ctx.Done():
+					return
+				}
+			}
+		}()
+	} else {
+		close(progressDone)
+	}
+
+	scanned, err := drainSource(ctx, cfg, src, jobs)
+	close(jobs)
+	wg.Wait()
+	close(stopProgress) // tell the progress goroutine to exit; would otherwise hang Run forever (caught by candidates e2e 2026-04-29)
+	<-progressDone
+
+	stats := Stats{
+		Scanned:       scanned,
+		Embedded:      atomic.LoadInt64(&totalEmbedded),
+		Added:         atomic.LoadInt64(&totalAdded),
+		Wall:          time.Since(t0),
+		FailedBatches: atomic.LoadInt64(&failedBatches),
+	}
+	if err != nil {
+		return stats, err
+	}
+	if stats.FailedBatches > 0 {
+		return stats, fmt.Errorf("%w: %d batches failed (embedded=%d added=%d scanned=%d)",
+			ErrPartialFailure, stats.FailedBatches, stats.Embedded, stats.Added, stats.Scanned)
+	}
+	return stats, nil
+}
+
+// drainSource pulls rows, batches them, and dispatches into jobs.
+// Returns when source EOFs, ctx cancels, or limit is hit.
+func drainSource(ctx context.Context, cfg Config, src Source, jobs chan<- job) (int64, error) {
+	curIDs := make([]string, 0, cfg.EmbedBatch)
+	curTexts := make([]string, 0, cfg.EmbedBatch)
+	curMetas := make([]json.RawMessage, 0, cfg.EmbedBatch)
+
+	flush := func() {
+		if len(curIDs) == 0 {
+			return
+		}
+		jobs <- job{ids: curIDs, texts: curTexts, metas: curMetas}
+		curIDs = make([]string, 0, cfg.EmbedBatch)
+		curTexts = make([]string, 0, cfg.EmbedBatch)
+		curMetas = make([]json.RawMessage, 0, cfg.EmbedBatch)
+	}
+
+	var scanned int64
+	for {
+		if ctx.Err() != nil {
+			flush()
+			return scanned, ctx.Err()
+		}
+		row, err := src.Next()
+		if err == io.EOF {
+			flush()
+			return scanned, nil
+		}
+		if err != nil {
+			flush()
+			return scanned, fmt.Errorf("source row %d: %w", scanned, err)
+		}
+		if row.ID == "" {
+			return scanned, fmt.Errorf("source row %d: empty id", scanned)
+		}
+		// Empty Text would 400 at embedd; skip-with-warn rather than
+		// abort the whole run — a stray empty row shouldn't kill 500K.
+		if row.Text == "" {
+			slog.Warn("corpusingest: skipping row with empty text",
+				"index", cfg.IndexName, "id", row.ID)
+			scanned++
+			continue
+		}
+		meta, err := marshalMeta(row.Metadata)
+		if err != nil {
+			return scanned, fmt.Errorf("row %s: marshal metadata: %w", row.ID, err)
+		}
+		curIDs = append(curIDs, row.ID)
+		curTexts = append(curTexts, row.Text)
+		curMetas = append(curMetas, meta)
+		scanned++
+
+		if len(curIDs) >= cfg.EmbedBatch {
+			flush()
+		}
+		if cfg.Limit > 0 && scanned >= int64(cfg.Limit) {
+			flush()
+			return scanned, nil
+		}
+	}
+}
+
+// job is the unit of work between drainSource and the embed workers.
+// Internal type; kept small so the channel buffer doesn't bloat.
+type job struct {
+	ids   []string
+	texts []string
+	metas []json.RawMessage
+}
+
+func marshalMeta(v any) (json.RawMessage, error) {
+	if v == nil {
+		return nil, nil
+	}
+	if rm, ok := v.(json.RawMessage); ok {
+		return rm, nil
+	}
+	return json.Marshal(v)
+}
+
+// prepareIndex creates the vectord index, optionally dropping a
+// preexisting one. Idempotent on matching params: 409 from create is
+// treated as "already exists, reuse." If DropExisting is set, DELETE
+// fires first to give a clean slate.
+func prepareIndex(ctx context.Context, cfg Config) error {
+	if cfg.DropExisting {
+		if err := httpDelete(ctx, cfg.HTTPClient,
+			cfg.GatewayURL+"/v1/vectors/index/"+cfg.IndexName); err != nil {
+			// 404 (not found) is fine — drop-existing is idempotent.
+			slog.Debug("corpusingest: drop existing", "err", err)
+		}
+	}
+	body, _ := json.Marshal(map[string]any{
+		"name":      cfg.IndexName,
+		"dimension": cfg.Dimension,
+		"distance":  cfg.Distance,
+	})
+	code, msg, err := httpPost(ctx, cfg.HTTPClient, cfg.GatewayURL+"/v1/vectors/index", body)
+	if err != nil {
+		return err
+	}
+	switch code {
+	case http.StatusCreated:
+		slog.Info("corpusingest: created index",
+			"name", cfg.IndexName, "dim", cfg.Dimension, "distance", cfg.Distance)
+	case http.StatusConflict:
+		// Already exists — vectord didn't change params on conflict.
+		// Caller's responsibility to ensure existing dim/distance match.
+		slog.Info("corpusingest: index already exists, reusing", "name", cfg.IndexName)
+	default:
+		return fmt.Errorf("create index %d: %s", code, msg)
+	}
+	return nil
+}
+
+func embedBatch(ctx context.Context, cfg Config, texts []string) ([][]float32, error) {
+	body := map[string]any{"texts": texts}
+	if cfg.EmbedModel != "" {
+		body["model"] = cfg.EmbedModel
+	}
+	bs, _ := json.Marshal(body)
+	code, msg, raw, err := httpPostRaw(ctx, cfg.HTTPClient, cfg.GatewayURL+"/v1/embed", bs)
+	if err != nil {
+		return nil, err
+	}
+	if code != http.StatusOK {
+		return nil, fmt.Errorf("embed status %d: %s", code, msg)
+	}
+	var er struct {
+		Vectors [][]float32 `json:"vectors"`
+	}
+	if err := json.Unmarshal(raw, &er); err != nil {
+		return nil, fmt.Errorf("embed decode: %w", err)
+	}
+	return er.Vectors, nil
+}
+
+func addBatch(ctx context.Context, cfg Config, ids []string, vecs [][]float32, metas []json.RawMessage) error {
+	type addItem struct {
+		ID       string          `json:"id"`
+		Vector   []float32       `json:"vector"`
+		Metadata json.RawMessage `json:"metadata,omitempty"`
+	}
+	// Add-batch may exceed cfg.AddBatch when EmbedBatch divides into it
+	// non-evenly; vectord handles that fine. Keep one HTTP per job.
+	items := make([]addItem, len(ids))
+	for i := range ids {
+		items[i] = addItem{ID: ids[i], Vector: vecs[i], Metadata: metas[i]}
+	}
+	bs, _ := json.Marshal(map[string]any{"items": items})
+	code, msg, err := httpPost(ctx, cfg.HTTPClient,
+		cfg.GatewayURL+"/v1/vectors/index/"+cfg.IndexName+"/add", bs)
+	if err != nil {
+		return err
+	}
+	if code != http.StatusOK {
+		return fmt.Errorf("add status %d: %s", code, msg)
+	}
+	return nil
+}
+
+// ── HTTP helpers — small, no extra deps ─────────────────────────
+
+func httpPost(ctx context.Context, hc *http.Client, url string, body []byte) (int, string, error) {
+	code, msg, _, err := httpPostRaw(ctx, hc, url, body)
+	return code, msg, err
+}
+
+func httpPostRaw(ctx context.Context, hc *http.Client, url string, body []byte) (int, string, []byte, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
+	if err != nil {
+		return 0, "", nil, err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	resp, err := hc.Do(req)
+	if err != nil {
+		return 0, "", nil, err
+	}
+	defer resp.Body.Close()
+	raw, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return resp.StatusCode, "", nil, err
+	}
+	preview := raw
+	if len(preview) > 256 {
+		preview = preview[:256]
+	}
+	return resp.StatusCode, string(preview), raw, nil
+}
+
+func httpDelete(ctx context.Context, hc *http.Client, url string) error {
+	req, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := hc.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	io.Copy(io.Discard, resp.Body)
+	if resp.StatusCode >= 400 && resp.StatusCode != http.StatusNotFound {
+		return fmt.Errorf("delete status %d", resp.StatusCode)
+	}
+	return nil
+}
+
+// ── config validation + defaults ────────────────────────────────
+
+func applyDefaults(cfg Config) Config {
+	if cfg.GatewayURL == "" {
+		cfg.GatewayURL = "http://127.0.0.1:3110"
+	}
+	if cfg.Distance == "" {
+		cfg.Distance = "cosine"
+	}
+	if cfg.EmbedBatch <= 0 {
+		cfg.EmbedBatch = 16
+	}
+	if cfg.EmbedWorkers <= 0 {
+		cfg.EmbedWorkers = 8
+	}
+	if cfg.AddBatch <= 0 {
+		cfg.AddBatch = 1000
+	}
+	if cfg.HTTPClient == nil {
+		cfg.HTTPClient = &http.Client{Timeout: 5 * time.Minute}
+	}
+	if cfg.LogProgress < 0 {
+		cfg.LogProgress = 0
+	}
+	return cfg
+}
+
+func validateConfig(cfg Config) error {
+	if cfg.IndexName == "" {
+		return errors.New("corpusingest: IndexName is required")
+	}
+	if cfg.Dimension <= 0 {
+		return errors.New("corpusingest: Dimension must be > 0")
+	}
+	return nil
+}

internal/corpusingest/ingest_test.go

@@ -0,0 +1,455 @@
+package corpusingest
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+// fakeGateway records the embed + add calls corpusingest fires and
+// returns canned responses. The whole point of the unit test is to
+// validate the pipeline shape (request payloads, batching, stats)
+// without needing live embedd/vectord.
+type fakeGateway struct {
+	mu             sync.Mutex
+	embedCalls     int
+	embedTexts     [][]string  // texts per call
+	addCalls       int
+	addItems       [][]addItem // items per call
+	createCalled   bool
+	deleteCalled   bool
+	indexConflict  bool        // simulate "index already exists" → 409
+	embedDimension int
+}
+
+type addItem struct {
+	ID       string          `json:"id"`
+	Vector   []float32       `json:"vector"`
+	Metadata json.RawMessage `json:"metadata,omitempty"`
+}
+
+func newFakeGateway(dim int) *fakeGateway {
+	return &fakeGateway{embedDimension: dim}
+}
+
+func (f *fakeGateway) handler() http.Handler {
+	mux := http.NewServeMux()
+
+	mux.HandleFunc("/v1/vectors/index", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodPost {
+			http.Error(w, "wrong method", http.StatusMethodNotAllowed)
+			return
+		}
+		f.mu.Lock()
+		f.createCalled = true
+		conflict := f.indexConflict
+		f.mu.Unlock()
+		if conflict {
+			http.Error(w, "exists", http.StatusConflict)
+			return
+		}
+		w.WriteHeader(http.StatusCreated)
+	})
+
+	mux.HandleFunc("/v1/embed", func(w http.ResponseWriter, r *http.Request) {
+		var req struct {
+			Texts []string `json:"texts"`
+		}
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		// Synthesize deterministic vectors: vector[i] = float32(i+1).
+		vecs := make([][]float32, len(req.Texts))
+		for i := range vecs {
+			v := make([]float32, f.embedDimension)
+			for j := range v {
+				v[j] = float32(i + j + 1)
+			}
+			vecs[i] = v
+		}
+		f.mu.Lock()
+		f.embedCalls++
+		// Copy because we'll release the slice after returning.
+		texts := append([]string(nil), req.Texts...)
+		f.embedTexts = append(f.embedTexts, texts)
+		f.mu.Unlock()
+		w.Header().Set("Content-Type", "application/json")
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"vectors":   vecs,
+			"dimension": f.embedDimension,
+			"model":     "fake-embed",
+		})
+	})
+
+	mux.HandleFunc("/v1/vectors/index/", func(w http.ResponseWriter, r *http.Request) {
+		// /v1/vectors/index/{name}/add
+		if !strings.HasSuffix(r.URL.Path, "/add") {
+			if r.Method == http.MethodDelete {
+				f.mu.Lock()
+				f.deleteCalled = true
+				f.mu.Unlock()
+				w.WriteHeader(http.StatusNoContent)
+				return
+			}
+			http.Error(w, "unhandled "+r.URL.Path, http.StatusNotFound)
+			return
+		}
+		var req struct {
+			Items []addItem `json:"items"`
+		}
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		f.mu.Lock()
+		f.addCalls++
+		f.addItems = append(f.addItems, append([]addItem(nil), req.Items...))
+		f.mu.Unlock()
+		_, _ = io.WriteString(w, `{"added":`+fmt.Sprint(len(req.Items))+`}`)
+	})
+
+	return mux
+}
+
+// staticSource yields a fixed slice of rows.
+type staticSource struct {
+	rows []Row
+	i    int
+}
+
+func (s *staticSource) Next() (Row, error) {
+	if s.i >= len(s.rows) {
+		return Row{}, io.EOF
+	}
+	r := s.rows[s.i]
+	s.i++
+	return r, nil
+}
+
+func TestRun_PipelineShapeAndStats(t *testing.T) {
+	const dim = 4
+	fg := newFakeGateway(dim)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	rows := make([]Row, 50)
+	for i := range rows {
+		rows[i] = Row{
+			ID:       fmt.Sprintf("r-%03d", i),
+			Text:     fmt.Sprintf("row %d text", i),
+			Metadata: map[string]any{"i": i, "kind": "test"},
+		}
+	}
+
+	stats, err := Run(context.Background(), Config{
+		GatewayURL:   srv.URL,
+		IndexName:    "test_corpus",
+		Dimension:    dim,
+		Distance:     "cosine",
+		EmbedBatch:   16,
+		EmbedWorkers: 4,
+		HTTPClient:   srv.Client(),
+		LogProgress:  0,
+	}, &staticSource{rows: rows})
+	if err != nil {
+		t.Fatalf("Run: %v", err)
+	}
+
+	if stats.Scanned != 50 {
+		t.Errorf("Scanned: want 50, got %d", stats.Scanned)
+	}
+	if stats.Embedded != 50 {
+		t.Errorf("Embedded: want 50, got %d", stats.Embedded)
+	}
+	if stats.Added != 50 {
+		t.Errorf("Added: want 50, got %d", stats.Added)
+	}
+	if !fg.createCalled {
+		t.Error("expected create-index to be called")
+	}
+	// 50 rows / 16 batch = ceil(50/16) = 4 batches → 4 embed calls + 4 add calls
+	if fg.embedCalls != 4 {
+		t.Errorf("embedCalls: want 4 (50 rows / 16 batch), got %d", fg.embedCalls)
+	}
+	if fg.addCalls != 4 {
+		t.Errorf("addCalls: want 4, got %d", fg.addCalls)
+	}
+
+	// Sum of texts across embed calls must be 50, and IDs across add
+	// calls must be every r-NNN exactly once.
+	totalTexts := 0
+	for _, ts := range fg.embedTexts {
+		totalTexts += len(ts)
+	}
+	if totalTexts != 50 {
+		t.Errorf("total embedded texts: want 50, got %d", totalTexts)
+	}
+	seen := make(map[string]bool)
+	for _, items := range fg.addItems {
+		for _, it := range items {
+			if seen[it.ID] {
+				t.Errorf("duplicate id in add stream: %s", it.ID)
+			}
+			seen[it.ID] = true
+			if len(it.Vector) != dim {
+				t.Errorf("vector dim: want %d, got %d", dim, len(it.Vector))
+			}
+		}
+	}
+	if len(seen) != 50 {
+		t.Errorf("unique ids added: want 50, got %d", len(seen))
+	}
+}
+
+func TestRun_DropExistingFiresDelete(t *testing.T) {
+	fg := newFakeGateway(4)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	_, err := Run(context.Background(), Config{
+		GatewayURL:   srv.URL,
+		IndexName:    "drops_first",
+		Dimension:    4,
+		DropExisting: true,
+		HTTPClient:   srv.Client(),
+	}, &staticSource{rows: []Row{{ID: "x", Text: "y", Metadata: nil}}})
+	if err != nil {
+		t.Fatalf("Run: %v", err)
+	}
+	if !fg.deleteCalled {
+		t.Error("expected delete-index to fire when DropExisting=true")
+	}
+}
+
+func TestRun_IndexAlreadyExistsIsReused(t *testing.T) {
+	fg := newFakeGateway(4)
+	fg.indexConflict = true // first POST /v1/vectors/index → 409
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	stats, err := Run(context.Background(), Config{
+		GatewayURL:   srv.URL,
+		IndexName:    "exists_already",
+		Dimension:    4,
+		HTTPClient:   srv.Client(),
+		EmbedWorkers: 1,
+	}, &staticSource{rows: []Row{{ID: "x", Text: "y", Metadata: nil}}})
+	if err != nil {
+		t.Fatalf("Run with existing index should succeed: %v", err)
+	}
+	if stats.Added != 1 {
+		t.Errorf("Added: want 1, got %d", stats.Added)
+	}
+}
+
+func TestRun_LimitStopsEarly(t *testing.T) {
+	fg := newFakeGateway(4)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	rows := make([]Row, 100)
+	for i := range rows {
+		rows[i] = Row{ID: fmt.Sprintf("r-%d", i), Text: "t", Metadata: nil}
+	}
+
+	stats, err := Run(context.Background(), Config{
+		GatewayURL:   srv.URL,
+		IndexName:    "limited",
+		Dimension:    4,
+		Limit:        25,
+		EmbedBatch:   8,
+		EmbedWorkers: 2,
+		HTTPClient:   srv.Client(),
+	}, &staticSource{rows: rows})
+	if err != nil {
+		t.Fatalf("Run: %v", err)
+	}
+	if stats.Scanned != 25 {
+		t.Errorf("Scanned: want 25 (limit), got %d", stats.Scanned)
+	}
+}
+
+func TestRun_EmptyTextSkipped(t *testing.T) {
+	fg := newFakeGateway(4)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	rows := []Row{
+		{ID: "a", Text: "real text", Metadata: nil},
+		{ID: "b", Text: "", Metadata: nil}, // skipped
+		{ID: "c", Text: "more text", Metadata: nil},
+	}
+
+	stats, err := Run(context.Background(), Config{
+		GatewayURL: srv.URL, IndexName: "skip", Dimension: 4,
+		HTTPClient: srv.Client(),
+	}, &staticSource{rows: rows})
+	if err != nil {
+		t.Fatalf("Run: %v", err)
+	}
+	if stats.Scanned != 3 {
+		t.Errorf("Scanned: want 3 (b is skipped but counted as scanned), got %d", stats.Scanned)
+	}
+	if stats.Added != 2 {
+		t.Errorf("Added: want 2 (b excluded from embed), got %d", stats.Added)
+	}
+}
+
+// TestRun_ProgressLoggerExits guards the bug caught 2026-04-29 in
+// the candidates e2e: when LogProgress > 0, the progress goroutine's
+// only exit was ctx.Done(). With context.Background() in the
+// production driver, Run hung forever after the pipeline finished.
+// This test bounds Run's wall to a few hundred ms — if it regresses,
+// the test deadline kicks in.
+func TestRun_ProgressLoggerExits(t *testing.T) {
+	fg := newFakeGateway(4)
+	srv := httptest.NewServer(fg.handler())
+	defer srv.Close()
+
+	rows := []Row{
+		{ID: "a", Text: "x", Metadata: nil},
+		{ID: "b", Text: "y", Metadata: nil},
+	}
+
+	done := make(chan error, 1)
+	go func() {
+		_, err := Run(context.Background(), Config{
+			GatewayURL:  srv.URL,
+			IndexName:   "progress_test",
+			Dimension:   4,
+			HTTPClient:  srv.Client(),
+			LogProgress: 50 * time.Millisecond,
+		}, &staticSource{rows: rows})
+		done <- err
+	}()
+
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatalf("Run: %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("Run did not return within 2s — progress goroutine likely hanging")
+	}
+}
+
+// TestRun_NonzeroFailedBatchesReturnsError guards the 2026-04-29
+// scrum WARN: original behavior returned nil even when 100% of
+// batches failed, making "embedded=0/scanned=N" look like an empty
+// corpus rather than a broken pipeline.
+func TestRun_NonzeroFailedBatchesReturnsError(t *testing.T) {
+	// Fake gateway that fails every embed call.
+	mux := http.NewServeMux()
+	mux.HandleFunc("/v1/vectors/index", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusCreated)
+	})
+	mux.HandleFunc("/v1/embed", func(w http.ResponseWriter, r *http.Request) {
+		http.Error(w, "embed failure injected", http.StatusBadGateway)
+	})
+	mux.HandleFunc("/v1/vectors/index/", func(w http.ResponseWriter, r *http.Request) {
+		// shouldn't reach here since embed fails first
+		http.Error(w, "should not be called", http.StatusInternalServerError)
+	})
+	srv := httptest.NewServer(mux)
+	defer srv.Close()
+
+	rows := make([]Row, 5)
+	for i := range rows {
+		rows[i] = Row{ID: fmt.Sprintf("r-%d", i), Text: "x"}
+	}
+
+	stats, err := Run(context.Background(), Config{
+		GatewayURL: srv.URL, IndexName: "fail_only", Dimension: 4,
+		EmbedBatch: 1, EmbedWorkers: 1, HTTPClient: srv.Client(),
+	}, &staticSource{rows: rows})
+
+	if !errors.Is(err, ErrPartialFailure) {
+		t.Errorf("want ErrPartialFailure, got %v", err)
+	}
+	if stats.FailedBatches == 0 {
+		t.Error("FailedBatches should be > 0 when embeds fail")
+	}
+	if stats.Added != 0 {
+		t.Errorf("Added: want 0 (all failed), got %d", stats.Added)
+	}
+}
+
+func TestRun_RequiresIndexName(t *testing.T) {
+	_, err := Run(context.Background(), Config{Dimension: 4},
+		&staticSource{rows: nil})
+	if err == nil || !strings.Contains(err.Error(), "IndexName") {
+		t.Errorf("want IndexName-required error, got %v", err)
+	}
+}
+
+func TestRun_RequiresDimension(t *testing.T) {
+	_, err := Run(context.Background(), Config{IndexName: "x"},
+		&staticSource{rows: nil})
+	if err == nil || !strings.Contains(err.Error(), "Dimension") {
+		t.Errorf("want Dimension-required error, got %v", err)
+	}
+}
+
+// TestRun_ContextCancel verifies the pipeline drains cleanly when
+// ctx is cancelled mid-run. Source returns rows fast enough that
+// without ctx the run would complete; cancelling early should stop
+// well before all 1000 rows are processed.
+func TestRun_ContextCancel(t *testing.T) {
+	fg := newFakeGateway(4)
+	// Slow embed handler: each call sleeps 50ms.
+	mux := http.NewServeMux()
+	mux.HandleFunc("/v1/vectors/index", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusCreated)
+	})
+	mux.HandleFunc("/v1/embed", func(w http.ResponseWriter, r *http.Request) {
+		var req struct {
+			Texts []string `json:"texts"`
+		}
+		_ = json.NewDecoder(r.Body).Decode(&req)
+		// Simulate slow-but-valid backend so we test ctx cancel, not
+		// degraded-payload handling (that's covered in production by
+		// the len-mismatch guard in Run's worker).
+		time.Sleep(50 * time.Millisecond)
+		_ = fg
+		vecs := make([][]float32, len(req.Texts))
+		for i := range vecs {
+			vecs[i] = []float32{1, 2, 3, 4}
+		}
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"vectors":   vecs,
+			"dimension": 4,
+			"model":     "x",
+		})
+	})
+	mux.HandleFunc("/v1/vectors/index/", func(w http.ResponseWriter, r *http.Request) {
+		_, _ = io.WriteString(w, `{}`)
+	})
+	srv := httptest.NewServer(mux)
+	defer srv.Close()
+
+	rows := make([]Row, 1000)
+	for i := range rows {
+		rows[i] = Row{ID: fmt.Sprintf("r-%d", i), Text: "t"}
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
+	defer cancel()
+
+	stats, err := Run(ctx, Config{
+		GatewayURL: srv.URL, IndexName: "cancel_me", Dimension: 4,
+		EmbedBatch: 1, EmbedWorkers: 1, HTTPClient: srv.Client(),
+	}, &staticSource{rows: rows})
+	// Either an error or a partial stats; the point is "didn't process all 1000."
+	if stats.Scanned >= 1000 {
+		t.Errorf("ctx cancel did not stop early: scanned=%d err=%v", stats.Scanned, err)
+	}
+}

internal/distillation/audit_baseline.go

@@ -0,0 +1,251 @@
+package distillation
+
+// Audit-baseline lineage — the longitudinal signal that distinguishes
+// "metrics shifted because the world changed" from "metrics shifted
+// because we broke something." Mirrors the Rust audit_full.ts
+// LoadBaseline/AppendBaseline/buildDriftTable shape so a Go-side
+// audit run can be compared against Rust-side baselines and
+// vice-versa during the migration.
+//
+// Storage: data/_kb/audit_baselines.jsonl, one AuditBaseline per
+// line, append-only. The LAST line is the most recent. New runs
+// read the prior baseline, compute drift vs current metrics, then
+// append a fresh entry.
+//
+// Why generic on metric name (vs Rust's pinned p2_evidence_rows
+// etc.): the Rust phase numbering (p0..p7) doesn't translate to Go
+// directly. Operators with mixed Rust+Go pipelines should use the
+// SAME metric names on both sides so the drift table compares
+// like-for-like. Helper constants below pin the Rust-compat names
+// for callers running both runtimes.
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+)
+
+// AuditBaseline is one entry in the audit_baselines.jsonl
+// longitudinal log. Schema-stable; new metrics land as new keys
+// in the Metrics map (additive — readers tolerate unknown keys).
+type AuditBaseline struct {
+	RecordedAt string           `json:"recorded_at"`         // ISO 8601 UTC
+	GitCommit  string           `json:"git_commit,omitempty"` // sha of the run's HEAD
+	Metrics    map[string]int64 `json:"metrics"`
+}
+
+// AuditBaselineRustCompat lists the metric names the Rust pipeline
+// emits at audit_full.ts. Go-side callers running an equivalent
+// audit should use these names so drift compares across runtimes.
+// Adding new names here requires the Rust side to mint them too.
+var AuditBaselineRustCompat = []string{
+	"p2_evidence_rows",
+	"p2_evidence_skips",
+	"p3_accepted",
+	"p3_partial",
+	"p3_rejected",
+	"p3_human",
+	"p4_rag_rows",
+	"p4_sft_rows",
+	"p4_pref_pairs",
+	"p4_total_quarantined",
+}
+
+// DefaultBaselinePath returns the canonical audit baselines path
+// rooted at the lakehouse data dir. Match Rust's BASELINE_PATH_FOR.
+func DefaultBaselinePath(root string) string {
+	return filepath.Join(root, "data", "_kb", "audit_baselines.jsonl")
+}
+
+// LoadLastBaseline reads audit_baselines.jsonl and returns the
+// most recent entry — i.e. the LAST non-empty JSON line. Missing
+// file or empty file returns (nil, nil), not an error: a fresh
+// pipeline has no baseline yet, and the caller should treat that
+// as "first run" via BuildAuditDriftTable's nil-prior handling.
+//
+// Malformed last line returns an error (rather than silently
+// skipping to the previous line) so operators don't lose drift
+// signal under partial-write corruption.
+func LoadLastBaseline(path string) (*AuditBaseline, error) {
+	data, err := os.ReadFile(path)
+	if os.IsNotExist(err) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, fmt.Errorf("read baselines: %w", err)
+	}
+	lines := strings.Split(string(data), "\n")
+	// Walk back to the last non-empty line.
+	for i := len(lines) - 1; i >= 0; i-- {
+		s := strings.TrimSpace(lines[i])
+		if s == "" {
+			continue
+		}
+		var b AuditBaseline
+		if err := json.Unmarshal([]byte(s), &b); err != nil {
+			return nil, fmt.Errorf("decode last baseline (line %d): %w", i+1, err)
+		}
+		return &b, nil
+	}
+	return nil, nil
+}
+
+// AppendBaseline appends one AuditBaseline as a JSON line to
+// audit_baselines.jsonl. Creates the parent directory if missing.
+// Atomic write at the line level: a partial write on disk-full or
+// crash leaves the file with at most one truncated trailing line,
+// which LoadLastBaseline will surface as a decode error.
+func AppendBaseline(path string, b AuditBaseline) error {
+	if b.RecordedAt == "" {
+		return errors.New("audit_baseline: RecordedAt required")
+	}
+	if b.Metrics == nil {
+		return errors.New("audit_baseline: Metrics required (use empty map for zero-metric run)")
+	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+		return fmt.Errorf("mkdir baseline dir: %w", err)
+	}
+	line, err := json.Marshal(b)
+	if err != nil {
+		return fmt.Errorf("encode baseline: %w", err)
+	}
+	f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		return fmt.Errorf("open baselines: %w", err)
+	}
+	defer f.Close()
+	w := bufio.NewWriter(f)
+	if _, err := w.Write(line); err != nil {
+		return fmt.Errorf("write baseline: %w", err)
+	}
+	if err := w.WriteByte('\n'); err != nil {
+		return fmt.Errorf("write newline: %w", err)
+	}
+	return w.Flush()
+}
+
+// AuditDriftFlag categorizes a single metric's drift verdict.
+// Mirrors the Rust DriftRow.flag values exactly.
+type AuditDriftFlag string
+
+const (
+	AuditDriftFlagFirstRun AuditDriftFlag = "first_run" // no prior baseline → can't compute change
+	AuditDriftFlagOK       AuditDriftFlag = "ok"        // |Δ%| ≤ threshold
+	AuditDriftFlagWarn     AuditDriftFlag = "warn"      // |Δ%| > threshold
+)
+
+// DefaultDriftWarnThreshold is 20% — matches Rust's hard-coded
+// `Math.abs(pct) > 0.20`. Operators tuning sensitivity per metric
+// can pass a different value to BuildAuditDriftTable.
+const DefaultDriftWarnThreshold = 0.20
+
+// AuditDriftRow is one metric's drift verdict. PctChange is nil
+// when prior baseline was zero (division-by-zero) OR when this is
+// the first run. Encoded as *float64 so JSON omits the field
+// rather than emitting 0.0 for "unknowable" cases.
+type AuditDriftRow struct {
+	Metric    string         `json:"metric"`
+	Baseline  *int64         `json:"baseline"`
+	Current   int64          `json:"current"`
+	PctChange *float64       `json:"pct_change"`
+	Flag      AuditDriftFlag `json:"flag"`
+}
+
+// BuildAuditDriftTable computes per-metric drift between a prior
+// baseline (nil = first run) and the current metric snapshot. The
+// result is sorted by metric name for stable display.
+//
+// Threshold is the absolute percent-change above which a metric is
+// flagged "warn". Pass DefaultDriftWarnThreshold (0.20 = 20%) to
+// match Rust audit_full.ts. Use a per-metric threshold map by
+// calling BuildAuditDriftTable once per metric subset.
+func BuildAuditDriftTable(prior *AuditBaseline, current map[string]int64, threshold float64) []AuditDriftRow {
+	if threshold <= 0 {
+		threshold = DefaultDriftWarnThreshold
+	}
+	// Union of metric names so a metric that disappeared from
+	// current still surfaces as "current=0, drifted -100%".
+	names := make(map[string]struct{}, len(current))
+	for k := range current {
+		names[k] = struct{}{}
+	}
+	if prior != nil {
+		for k := range prior.Metrics {
+			names[k] = struct{}{}
+		}
+	}
+	rows := make([]AuditDriftRow, 0, len(names))
+	for name := range names {
+		row := AuditDriftRow{Metric: name, Current: current[name]}
+		if prior == nil {
+			row.Flag = AuditDriftFlagFirstRun
+			rows = append(rows, row)
+			continue
+		}
+		priorVal, hadPrior := prior.Metrics[name]
+		if !hadPrior {
+			// New metric in current — treat as first-run for THIS metric.
+			row.Flag = AuditDriftFlagFirstRun
+			rows = append(rows, row)
+			continue
+		}
+		row.Baseline = &priorVal
+		if priorVal == 0 {
+			// Division-by-zero: leave PctChange nil. If current is
+			// also 0 → ok (no change). Otherwise → warn (the metric
+			// went from zero to non-zero, which is always notable).
+			if current[name] == 0 {
+				row.Flag = AuditDriftFlagOK
+			} else {
+				row.Flag = AuditDriftFlagWarn
+			}
+			rows = append(rows, row)
+			continue
+		}
+		pct := float64(current[name]-priorVal) / float64(priorVal)
+		row.PctChange = &pct
+		if math.Abs(pct) > threshold {
+			row.Flag = AuditDriftFlagWarn
+		} else {
+			row.Flag = AuditDriftFlagOK
+		}
+		rows = append(rows, row)
+	}
+	// Sort for stable display + deterministic JSON output. Per
+	// scrum post_role_gate_v1 (Opus INFO): use sort.Slice — already
+	// imported in the package, idiomatic, and shorter.
+	sort.Slice(rows, func(i, j int) bool { return rows[i].Metric < rows[j].Metric })
+	return rows
+}
+
+// FormatAuditDriftTable renders a drift table as a fixed-width
+// text grid — useful for stdout dumps in audit-full runs. Matches
+// the Rust output shape so an operator can grep across runtimes
+// without re-learning the layout.
+func FormatAuditDriftTable(rows []AuditDriftRow) string {
+	if len(rows) == 0 {
+		return "(no metrics)\n"
+	}
+	var buf bytes.Buffer
+	fmt.Fprintf(&buf, "%-26s %12s %12s %10s %s\n", "metric", "baseline", "current", "Δ%", "flag")
+	for _, r := range rows {
+		baseline := "-"
+		if r.Baseline != nil {
+			baseline = fmt.Sprintf("%d", *r.Baseline)
+		}
+		pct := "-"
+		if r.PctChange != nil {
+			pct = fmt.Sprintf("%+.1f%%", *r.PctChange*100)
+		}
+		fmt.Fprintf(&buf, "%-26s %12s %12d %10s %s\n",
+			r.Metric, baseline, r.Current, pct, r.Flag)
+	}
+	return buf.String()
+}

internal/distillation/audit_baseline_test.go

@@ -0,0 +1,278 @@
+package distillation
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestLoadLastBaseline_MissingFile: no baselines file yet → (nil, nil),
+// not an error. First-run pipelines have no baseline.
+func TestLoadLastBaseline_MissingFile(t *testing.T) {
+	tmp := t.TempDir()
+	got, err := LoadLastBaseline(filepath.Join(tmp, "nonexistent.jsonl"))
+	if err != nil {
+		t.Fatalf("expected nil error on missing file, got %v", err)
+	}
+	if got != nil {
+		t.Errorf("expected nil baseline on missing file, got %+v", got)
+	}
+}
+
+// TestLoadLastBaseline_LastLineWins locks the "last line is most
+// recent" semantic — append-only log read backward.
+func TestLoadLastBaseline_LastLineWins(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "baselines.jsonl")
+	jsonl := `{"recorded_at":"2026-04-01T00:00:00Z","metrics":{"a":1}}
+{"recorded_at":"2026-04-15T00:00:00Z","metrics":{"a":2}}
+{"recorded_at":"2026-04-30T00:00:00Z","metrics":{"a":3}}
+`
+	if err := os.WriteFile(path, []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	got, err := LoadLastBaseline(path)
+	if err != nil {
+		t.Fatalf("LoadLastBaseline: %v", err)
+	}
+	if got == nil || got.RecordedAt != "2026-04-30T00:00:00Z" {
+		t.Errorf("expected last-line baseline (2026-04-30), got %+v", got)
+	}
+	if got.Metrics["a"] != 3 {
+		t.Errorf("expected metrics.a=3, got %d", got.Metrics["a"])
+	}
+}
+
+// TestLoadLastBaseline_TolerateTrailingBlankLines: writers append
+// "\n" — a stray blank line at the end mustn't trigger malformed-
+// JSON errors.
+func TestLoadLastBaseline_TolerateTrailingBlankLines(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "baselines.jsonl")
+	jsonl := `{"recorded_at":"2026-04-30T00:00:00Z","metrics":{"a":3}}
+
+`
+	if err := os.WriteFile(path, []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	got, err := LoadLastBaseline(path)
+	if err != nil {
+		t.Fatalf("LoadLastBaseline: %v", err)
+	}
+	if got == nil || got.Metrics["a"] != 3 {
+		t.Errorf("expected baseline with metrics.a=3, got %+v", got)
+	}
+}
+
+// TestLoadLastBaseline_MalformedLastLineErrors: partial-write on
+// disk-full would leave a truncated last line. Rather than
+// silently skip back to a stale baseline, surface the error so
+// operators don't lose drift signal.
+func TestLoadLastBaseline_MalformedLastLineErrors(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "baselines.jsonl")
+	jsonl := `{"recorded_at":"2026-04-15T00:00:00Z","metrics":{"a":2}}
+{"recorded_at":"2026-04-30T0` // truncated
+	if err := os.WriteFile(path, []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	_, err := LoadLastBaseline(path)
+	if err == nil {
+		t.Errorf("expected decode error on truncated last line, got nil")
+	}
+}
+
+// TestAppendBaseline_RoundTrip: append one + read back via
+// LoadLastBaseline. Critical for the "first audit-full run on a
+// fresh box" path.
+func TestAppendBaseline_RoundTrip(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "data", "_kb", "audit_baselines.jsonl")
+	b := AuditBaseline{
+		RecordedAt: "2026-05-01T12:00:00Z",
+		GitCommit:  "deadbeef",
+		Metrics: map[string]int64{
+			"p3_accepted":   42,
+			"p4_sft_rows":   17,
+		},
+	}
+	if err := AppendBaseline(path, b); err != nil {
+		t.Fatalf("AppendBaseline: %v", err)
+	}
+	got, err := LoadLastBaseline(path)
+	if err != nil {
+		t.Fatalf("LoadLastBaseline: %v", err)
+	}
+	if got == nil {
+		t.Fatalf("expected non-nil baseline after append")
+	}
+	if got.RecordedAt != b.RecordedAt || got.GitCommit != b.GitCommit {
+		t.Errorf("baseline header mismatch:\n  got:  %+v\n  want: %+v", got, b)
+	}
+	if got.Metrics["p3_accepted"] != 42 || got.Metrics["p4_sft_rows"] != 17 {
+		t.Errorf("metrics roundtrip: got %+v", got.Metrics)
+	}
+}
+
+// TestAppendBaseline_AppendsToExisting: existing file gets an
+// extra line, prior contents preserved.
+func TestAppendBaseline_AppendsToExisting(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "baselines.jsonl")
+	if err := os.WriteFile(path,
+		[]byte(`{"recorded_at":"2026-04-30T00:00:00Z","metrics":{"a":1}}`+"\n"),
+		0o644); err != nil {
+		t.Fatalf("seed: %v", err)
+	}
+	if err := AppendBaseline(path, AuditBaseline{
+		RecordedAt: "2026-05-01T00:00:00Z",
+		Metrics:    map[string]int64{"a": 2},
+	}); err != nil {
+		t.Fatalf("AppendBaseline: %v", err)
+	}
+	data, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatalf("read: %v", err)
+	}
+	lines := strings.Split(strings.TrimRight(string(data), "\n"), "\n")
+	if len(lines) != 2 {
+		t.Errorf("expected 2 lines after append, got %d", len(lines))
+	}
+}
+
+// TestAppendBaseline_RejectsEmptyRecordedAt: schema invariant —
+// every entry must carry a timestamp for ordering.
+func TestAppendBaseline_RejectsEmptyRecordedAt(t *testing.T) {
+	tmp := t.TempDir()
+	path := filepath.Join(tmp, "baselines.jsonl")
+	err := AppendBaseline(path, AuditBaseline{Metrics: map[string]int64{"a": 1}})
+	if err == nil {
+		t.Errorf("expected error on empty RecordedAt")
+	}
+}
+
+// TestBuildAuditDriftTable_FirstRun: no prior → every metric flagged
+// "first_run", no PctChange.
+func TestBuildAuditDriftTable_FirstRun(t *testing.T) {
+	rows := BuildAuditDriftTable(nil, map[string]int64{"a": 10, "b": 20}, 0)
+	if len(rows) != 2 {
+		t.Fatalf("expected 2 rows, got %d", len(rows))
+	}
+	for _, r := range rows {
+		if r.Flag != AuditDriftFlagFirstRun {
+			t.Errorf("metric %s: expected first_run flag, got %s", r.Metric, r.Flag)
+		}
+		if r.PctChange != nil {
+			t.Errorf("metric %s: PctChange should be nil on first run", r.Metric)
+		}
+	}
+}
+
+// TestBuildAuditDriftTable_ThresholdBoundary: exactly threshold = OK,
+// just over = WARN. Locks the >|threshold| (strict) semantic.
+func TestBuildAuditDriftTable_ThresholdBoundary(t *testing.T) {
+	prior := &AuditBaseline{Metrics: map[string]int64{"a": 100, "b": 100}}
+	current := map[string]int64{
+		"a": 120, // +20% — exactly at threshold → OK
+		"b": 121, // +21% — over threshold → WARN
+	}
+	rows := BuildAuditDriftTable(prior, current, 0.20)
+	byMetric := map[string]AuditDriftRow{}
+	for _, r := range rows {
+		byMetric[r.Metric] = r
+	}
+	if byMetric["a"].Flag != AuditDriftFlagOK {
+		t.Errorf("metric a (+20%% exactly): expected OK, got %s", byMetric["a"].Flag)
+	}
+	if byMetric["b"].Flag != AuditDriftFlagWarn {
+		t.Errorf("metric b (+21%%): expected warn, got %s", byMetric["b"].Flag)
+	}
+}
+
+// TestBuildAuditDriftTable_ZeroBaseline: prior=0 means we can't
+// compute pct (div-by-0). PctChange stays nil; current=0 stays
+// OK; current>0 escalates to WARN (zero→nonzero is always
+// notable).
+func TestBuildAuditDriftTable_ZeroBaseline(t *testing.T) {
+	prior := &AuditBaseline{Metrics: map[string]int64{"stayed_zero": 0, "went_nonzero": 0}}
+	current := map[string]int64{
+		"stayed_zero":   0,
+		"went_nonzero":  5,
+	}
+	rows := BuildAuditDriftTable(prior, current, 0.20)
+	byMetric := map[string]AuditDriftRow{}
+	for _, r := range rows {
+		byMetric[r.Metric] = r
+	}
+	if byMetric["stayed_zero"].Flag != AuditDriftFlagOK {
+		t.Errorf("0→0 should be OK, got %s", byMetric["stayed_zero"].Flag)
+	}
+	if byMetric["went_nonzero"].Flag != AuditDriftFlagWarn {
+		t.Errorf("0→5 should be warn, got %s", byMetric["went_nonzero"].Flag)
+	}
+	if byMetric["stayed_zero"].PctChange != nil || byMetric["went_nonzero"].PctChange != nil {
+		t.Errorf("zero-baseline rows must have nil PctChange (no division by zero)")
+	}
+}
+
+// TestBuildAuditDriftTable_NewMetricInCurrent: a metric present in
+// current but not in prior is flagged first_run, not "0%-change".
+func TestBuildAuditDriftTable_NewMetricInCurrent(t *testing.T) {
+	prior := &AuditBaseline{Metrics: map[string]int64{"old_only": 5}}
+	current := map[string]int64{"old_only": 5, "brand_new": 10}
+	rows := BuildAuditDriftTable(prior, current, 0)
+	byMetric := map[string]AuditDriftRow{}
+	for _, r := range rows {
+		byMetric[r.Metric] = r
+	}
+	if byMetric["brand_new"].Flag != AuditDriftFlagFirstRun {
+		t.Errorf("new metric should be first_run, got %s", byMetric["brand_new"].Flag)
+	}
+	if byMetric["old_only"].Flag != AuditDriftFlagOK {
+		t.Errorf("unchanged metric should be OK, got %s", byMetric["old_only"].Flag)
+	}
+}
+
+// TestBuildAuditDriftTable_SortedByMetric: deterministic JSON
+// output requires stable sort — drift tables in CI runs need to
+// diff cleanly.
+func TestBuildAuditDriftTable_SortedByMetric(t *testing.T) {
+	prior := &AuditBaseline{Metrics: map[string]int64{"zoo": 1, "alpha": 1, "midway": 1}}
+	current := map[string]int64{"zoo": 1, "alpha": 1, "midway": 1}
+	rows := BuildAuditDriftTable(prior, current, 0)
+	want := []string{"alpha", "midway", "zoo"}
+	for i, r := range rows {
+		if r.Metric != want[i] {
+			t.Errorf("rows[%d]: got %q, want %q", i, r.Metric, want[i])
+		}
+	}
+}
+
+// TestFormatAuditDriftTable_RendersFlags: stdout dump shape — we
+// don't pin every byte but verify the metric names + flags appear.
+func TestFormatAuditDriftTable_RendersFlags(t *testing.T) {
+	rows := []AuditDriftRow{
+		{Metric: "p3_accepted", Current: 50, Flag: AuditDriftFlagFirstRun},
+		{Metric: "p4_sft_rows", Baseline: int64Ptr(10), Current: 13, PctChange: float64Ptr(0.30), Flag: AuditDriftFlagWarn},
+	}
+	out := FormatAuditDriftTable(rows)
+	for _, want := range []string{"p3_accepted", "p4_sft_rows", "first_run", "warn", "+30.0%"} {
+		if !strings.Contains(out, want) {
+			t.Errorf("expected %q in output:\n%s", want, out)
+		}
+	}
+}
+
+// TestFormatAuditDriftTable_EmptyHeader: empty rows yields a
+// single-line "(no metrics)" — operators see something instead of
+// blank output.
+func TestFormatAuditDriftTable_EmptyHeader(t *testing.T) {
+	out := FormatAuditDriftTable(nil)
+	if !strings.Contains(out, "no metrics") {
+		t.Errorf("expected 'no metrics' notice on empty input, got %q", out)
+	}
+}
+
+func int64Ptr(v int64) *int64       { return &v }
+func float64Ptr(v float64) *float64 { return &v }

internal/distillation/audit_full.go

@@ -0,0 +1,807 @@
+package distillation
+
+// Audit-FULL pipeline — Go port of scripts/distillation/audit_full.ts
+// (Rust legacy). Runs the metric-collection passes that produce
+// audit_baselines.jsonl entries. Pure observability: never modifies
+// pipeline data, only reads and tallies.
+//
+// Phase coverage in this port:
+//   - Phase 0 (file presence)            ✓ ported
+//   - Phase 1 (schema validators)        ✓ ported (invokes `go test`
+//                                          on internal/distillation)
+//   - Phase 2 (evidence materialization) ✓ ported as observer — reads
+//                                          existing data/evidence/
+//                                          and tallies rows. Doesn't
+//                                          re-run the materializer
+//                                          (which is Rust-side); the
+//                                          audit-FULL discipline is
+//                                          OBSERVATION, not re-execution.
+//   - Phase 3 (scored-runs distribution) ✓ ported
+//   - Phase 4 (contamination firewall)   ✓ ported
+//   - Phase 5 (receipts validation)      ✓ ported as observer — reads
+//                                          reports/distillation/{run_id}/
+//                                          summary.json + 5 stage
+//                                          receipts (any-runtime artifacts).
+//   - Phase 6 (acceptance gate)          ✗ skipped — TS-only fixture
+//                                          harness at scripts/distillation/
+//                                          acceptance.ts with bun-
+//                                          specific deps. Porting the
+//                                          fixtures + invariant runner
+//                                          to Go is its own ADR-worth
+//                                          of work; out of scope.
+//   - Phase 7 (replay log shape)         ✓ ported as observer — reads
+//                                          data/_kb/replay_runs.jsonl
+//                                          and checks shape, doesn't
+//                                          re-run replay (Rust-side
+//                                          replay.ts is the producer).
+//
+// Output: a structured PhaseCheckReport plus a Markdown summary.
+// Operators run this from cmd/audit_full to validate a Go-side
+// distillation pipeline run produced sane outputs.
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// PhaseCheck is one observable check within a phase. Mirrors the
+// Rust shape exactly — Markdown rendering uses the same column
+// layout so cross-runtime diff'ing is meaningful.
+type PhaseCheck struct {
+	Phase    int      `json:"phase"`
+	Name     string   `json:"name"`
+	Expected string   `json:"expected"`
+	Actual   string   `json:"actual"`
+	Passed   bool     `json:"passed"`
+	Required bool     `json:"required"` // false → informational only, doesn't fail audit
+	Notes    []string `json:"notes,omitempty"`
+}
+
+// PhaseCheckReport is the aggregate result of one audit-full run.
+// Metrics is the AuditBaseline-shape metric snapshot that the
+// caller can pass to AppendBaseline to grow the longitudinal log.
+type PhaseCheckReport struct {
+	Checks  []PhaseCheck     `json:"checks"`
+	Metrics map[string]int64 `json:"metrics"`
+	Failed  int              `json:"failed"`           // count of REQUIRED checks that failed
+	Skipped int              `json:"deferred_phases"`  // phases not yet ported
+	GitHEAD string           `json:"git_head,omitempty"`
+}
+
+// AuditFullOptions controls a single audit-full run. Root is the
+// data dir (defaults to LH_DISTILL_ROOT or /home/profit/lakehouse
+// to keep operators running both runtimes hitting the same paths).
+type AuditFullOptions struct {
+	Root    string
+	GitHEAD string // optional — caller resolves and passes through
+	// GoTestModule is the package-pattern Phase 1 invokes via
+	// `go test`. Defaults to "./internal/distillation/..." when
+	// empty. Tests pass an empty path to disable the live
+	// `go test` invocation (which would recurse).
+	GoTestModule string
+}
+
+// RunAuditFull orchestrates the ported phases (0, 3, 4) and
+// returns the aggregated report. Each phase is independent; a
+// phase that errors is recorded as a failed check rather than
+// aborting the run, matching Rust's "always emit a report" stance.
+func RunAuditFull(opts AuditFullOptions) PhaseCheckReport {
+	if opts.Root == "" {
+		if env := os.Getenv("LH_DISTILL_ROOT"); env != "" {
+			opts.Root = env
+		} else {
+			opts.Root = "/home/profit/lakehouse"
+		}
+	}
+	report := PhaseCheckReport{
+		Metrics: make(map[string]int64),
+		GitHEAD: opts.GitHEAD,
+		Skipped: 1, // only phase 6 (TS-only acceptance harness) deferred
+	}
+	auditPhase0(opts.Root, &report)
+	auditPhase1(opts.Root, &report, opts.GoTestModule)
+	auditPhase2(opts.Root, &report)
+	auditPhase3(opts.Root, &report)
+	auditPhase4(opts.Root, &report)
+	auditPhase5(opts.Root, &report)
+	// phase 6 intentionally skipped — see header comment
+	auditPhase7(opts.Root, &report)
+	for _, c := range report.Checks {
+		if c.Required && !c.Passed {
+			report.Failed++
+		}
+	}
+	return report
+}
+
+// ── Phase 0: file presence ─────────────────────────────────────────
+
+func auditPhase0(root string, report *PhaseCheckReport) {
+	// The recon doc is Rust-specific (docs/recon/local-distillation-
+	// recon.md); a Go-side equivalent would live in the
+	// golangLAKEHOUSE repo. For audit-full's purposes, we treat its
+	// presence as informational rather than required when running
+	// against a non-Rust root.
+	reconPath := filepath.Join(root, "docs", "recon", "local-distillation-recon.md")
+	exists := fileExists(reconPath)
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 0, Name: "recon doc exists",
+		Expected: "docs/recon/local-distillation-recon.md present",
+		Actual:   fmt.Sprintf("%v", exists),
+		Passed:   exists, Required: false, // informational on Go-side runs
+	})
+
+	tier1 := []string{
+		"data/_kb/distilled_facts.jsonl",
+		"data/_kb/scrum_reviews.jsonl",
+		"data/_kb/audit_facts.jsonl",
+		"data/_kb/mode_experiments.jsonl",
+	}
+	missing := []string{}
+	for _, p := range tier1 {
+		if !fileExists(filepath.Join(root, p)) {
+			missing = append(missing, p)
+		}
+	}
+	notes := []string{}
+	if len(missing) > 0 {
+		notes = append(notes, "fresh-clone or post-rotation environment — Phase 2 will tally as rows_present=false; not a hard fail")
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 0, Name: "tier-1 source streams present",
+		Expected: "all 4 tier-1 jsonls on disk",
+		Actual: func() string {
+			if len(missing) == 0 {
+				return "all present"
+			}
+			return "missing: " + strings.Join(missing, ", ")
+		}(),
+		Passed: len(missing) == 0, Required: false,
+		Notes: notes,
+	})
+}
+
+// ── Phase 1: schema validators ─────────────────────────────────────
+
+// auditPhase1 invokes `go test` on the distillation package — the Go
+// equivalent of Rust's `bun test auditor/schemas/distillation/`. The
+// audit-FULL semantic: "do the schema validators still pass on
+// fixtures?" When module == "" (test mode) the phase records a
+// skipped-with-rationale check rather than recursing into itself.
+func auditPhase1(root string, report *PhaseCheckReport, module string) {
+	if module == "" {
+		// Test-disabled mode: record but don't invoke (would recurse
+		// when called from a `go test` already in progress).
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 1, Name: "schema validators (skipped — test invocation disabled)",
+			Expected: "go test ./internal/distillation/...",
+			Actual:   "skipped",
+			Passed:   true, Required: false,
+			Notes: []string{"caller passed empty GoTestModule — typically because we're already inside a test run"},
+		})
+		return
+	}
+	cmd := exec.Command("go", "test", "-count=1", module)
+	cmd.Dir = root // run from go module root if caller supplied it; otherwise cwd
+	out, err := cmd.CombinedOutput()
+	passed := err == nil
+	actual := "PASS"
+	if !passed {
+		actual = "FAIL — " + abbrevOutput(string(out), 200)
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 1, Name: "schema validators pass on fixtures",
+		Expected: "go test ./internal/distillation/... → exit 0",
+		Actual:   actual,
+		Passed:   passed, Required: true,
+	})
+}
+
+// abbrevOutput truncates noisy command-output to a stable preview.
+// Long stack traces would blow out the report Markdown without this.
+func abbrevOutput(s string, max int) string {
+	s = strings.TrimSpace(s)
+	if len(s) <= max {
+		return s
+	}
+	return s[:max] + "...(truncated)"
+}
+
+// ── Phase 2: evidence materialization (observer) ───────────────────
+
+// auditPhase2 reads data/evidence/ and tallies rows + skipped
+// markers. Mirrors the Rust phase 2's "materializer dry-run
+// completes / tier-1 sources each materialize ≥1 row" checks but
+// in OBSERVER mode — doesn't re-run the materializer (which is
+// Rust-side); instead reads what the Rust side already produced.
+//
+// Records p2_evidence_rows + p2_evidence_skips metrics that match
+// the Rust shape, so a Go-side audit-full producing baselines is
+// drop-in-comparable to a Rust-side run.
+func auditPhase2(root string, report *PhaseCheckReport) {
+	evidenceDir := filepath.Join(root, "data", "evidence")
+	if !fileExists(evidenceDir) {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 2, Name: "evidence materialization output present",
+			Expected: "data/evidence/ populated",
+			Actual:   "missing",
+			Passed:   false, Required: true,
+			Notes: []string{"run materializer (Rust: ./scripts/distill collect; Go-side materializer not yet ported) before audit-full"},
+		})
+		return
+	}
+	rows := int64(0)
+	skips := int64(0)
+	bySource := map[string]int64{}
+	tier1Hits := map[string]bool{
+		"distilled_facts":  false,
+		"scrum_reviews":    false,
+		"audit_facts":      false,
+		"mode_experiments": false,
+	}
+
+	walkErr := filepath.Walk(evidenceDir, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return nil
+		}
+		if info.IsDir() || !strings.HasSuffix(path, ".jsonl") {
+			return nil
+		}
+		data, err := os.ReadFile(path)
+		if err != nil {
+			return nil
+		}
+		// Tally per-source via the ev.provenance.source_file field on
+		// each evidence row. Match Rust's "by_source" map shape.
+		for _, line := range strings.Split(string(data), "\n") {
+			line = strings.TrimSpace(line)
+			if line == "" {
+				continue
+			}
+			rows++
+			var rec struct {
+				Provenance struct {
+					SourceFile string `json:"source_file"`
+				} `json:"provenance"`
+				SuccessMarkers []string `json:"success_markers,omitempty"`
+				FailureMarkers []string `json:"failure_markers,omitempty"`
+			}
+			if err := json.Unmarshal([]byte(line), &rec); err != nil {
+				skips++
+				continue
+			}
+			stem := stemFromSourceFile(rec.Provenance.SourceFile)
+			bySource[stem]++
+			if _, ok := tier1Hits[stem]; ok {
+				tier1Hits[stem] = true
+			}
+		}
+		return nil
+	})
+	if walkErr != nil {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 2, Name: "evidence walk",
+			Expected: "no error", Actual: walkErr.Error(),
+			Passed: false, Required: true,
+		})
+		return
+	}
+
+	report.Metrics["p2_evidence_rows"] = rows
+	report.Metrics["p2_evidence_skips"] = skips
+
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 2, Name: "evidence materialization output non-empty",
+		Expected: ">=1 row across all sources",
+		Actual:   fmt.Sprintf("%d rows · %d skipped", rows, skips),
+		Passed:   rows >= 1, Required: true,
+	})
+
+	tier1Found := []string{}
+	for src, hit := range tier1Hits {
+		if hit {
+			tier1Found = append(tier1Found, src)
+		}
+	}
+	sort.Strings(tier1Found)
+	notes := []string{}
+	if len(tier1Found) < 4 {
+		notes = append(notes, "fresh-environment OK; expect lower count when source streams are absent")
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 2, Name: "tier-1 sources each materialize ≥1 row",
+		Expected: "4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments",
+		Actual:   fmt.Sprintf("%d/4 hit (%s)", len(tier1Found), strings.Join(tier1Found, ", ")),
+		Passed:   len(tier1Found) >= 1, Required: false,
+		Notes: notes,
+	})
+}
+
+// ── Phase 3: scored-runs distribution ──────────────────────────────
+
+func auditPhase3(root string, report *PhaseCheckReport) {
+	scoredDir := filepath.Join(root, "data", "scored-runs")
+	if !fileExists(scoredDir) {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 3, Name: "scored-runs on disk",
+			Expected: "data/scored-runs/ populated",
+			Actual:   "missing",
+			Passed:   false, Required: true,
+			Notes: []string{"run scoring before audit-full (Go: scripts/distillation/score; Rust: ./scripts/distill score)"},
+		})
+		return
+	}
+
+	counts := map[string]int64{
+		"accepted":           0,
+		"partially_accepted": 0,
+		"rejected":           0,
+		"needs_human_review": 0,
+	}
+	files, err := ListScoredRunFiles(root)
+	if err != nil {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 3, Name: "scored-runs walk",
+			Expected: "no error", Actual: err.Error(),
+			Passed: false, Required: true,
+		})
+		return
+	}
+	for _, f := range files {
+		runs, _, err := LoadScoredRunsFromFile(f)
+		if err != nil {
+			continue
+		}
+		for _, r := range runs {
+			if _, ok := counts[string(r.Category)]; ok {
+				counts[string(r.Category)]++
+			}
+		}
+	}
+	total := counts["accepted"] + counts["partially_accepted"] + counts["rejected"] + counts["needs_human_review"]
+
+	report.Metrics["p3_accepted"] = counts["accepted"]
+	report.Metrics["p3_partial"] = counts["partially_accepted"]
+	report.Metrics["p3_rejected"] = counts["rejected"]
+	report.Metrics["p3_human"] = counts["needs_human_review"]
+
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 3, Name: "on-disk scored-runs distribution non-empty",
+		Expected: ">=1 accepted",
+		Actual:   fmt.Sprintf("acc=%d part=%d rej=%d hum=%d", counts["accepted"], counts["partially_accepted"], counts["rejected"], counts["needs_human_review"]),
+		Passed:   counts["accepted"] >= 1, Required: true,
+	})
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 3, Name: "scored-runs distribution sums positive",
+		Expected: ">0 total", Actual: fmt.Sprintf("%d total", total),
+		Passed: total > 0, Required: false,
+	})
+}
+
+// ── Phase 4: contamination firewall + provenance ───────────────────
+
+// sigHashRe pre-compiled match for the canonical sig_hash shape:
+// 64 lowercase hex characters (sha256 hex). Used per-row in the
+// provenance check.
+var sigHashRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
+
+func auditPhase4(root string, report *PhaseCheckReport) {
+	sftPath := filepath.Join(root, "exports", "sft", "instruction_response.jsonl")
+	ragPath := filepath.Join(root, "exports", "rag", "playbooks.jsonl")
+	prefPath := filepath.Join(root, "exports", "preference", "chosen_rejected.jsonl")
+
+	sftRows := readJSONLLines(sftPath)
+	ragRows := readJSONLLines(ragPath)
+	prefRows := readJSONLLines(prefPath)
+
+	report.Metrics["p4_sft_rows"] = int64(len(sftRows))
+	report.Metrics["p4_rag_rows"] = int64(len(ragRows))
+	report.Metrics["p4_pref_pairs"] = int64(len(prefRows))
+
+	// SFT contamination firewall: 0 forbidden quality_scores. The
+	// only legal SFT quality scores are accepted + partially_accepted.
+	sftForbidden := 0
+	for _, line := range sftRows {
+		var r struct {
+			QualityScore string `json:"quality_score"`
+		}
+		if err := json.Unmarshal([]byte(line), &r); err != nil {
+			continue // tolerate malformed (matches Rust)
+		}
+		if r.QualityScore != "accepted" && r.QualityScore != "partially_accepted" {
+			sftForbidden++
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 4, Name: "SFT contamination firewall: 0 forbidden quality_scores",
+		Expected: "0", Actual: fmt.Sprintf("%d", sftForbidden),
+		Passed: sftForbidden == 0, Required: true,
+		Notes: []string{"this is the spec non-negotiable — rejected/needs_human_review must NEVER appear in SFT"},
+	})
+
+	// RAG firewall: 0 rejected leaks
+	ragRejected := 0
+	for _, line := range ragRows {
+		var r struct {
+			SuccessScore string `json:"success_score"`
+		}
+		if err := json.Unmarshal([]byte(line), &r); err != nil {
+			continue
+		}
+		if r.SuccessScore == "rejected" {
+			ragRejected++
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 4, Name: "RAG firewall: 0 rejected leaks",
+		Expected: "0", Actual: fmt.Sprintf("%d", ragRejected),
+		Passed: ragRejected == 0, Required: true,
+	})
+
+	// Preference: 0 self-pairs + 0 identical-text pairs.
+	prefSelfPairs, prefIdenticalText := 0, 0
+	for _, line := range prefRows {
+		var r struct {
+			ChosenRunID   string `json:"chosen_run_id"`
+			RejectedRunID string `json:"rejected_run_id"`
+			Chosen        string `json:"chosen"`
+			Rejected      string `json:"rejected"`
+		}
+		if err := json.Unmarshal([]byte(line), &r); err != nil {
+			continue
+		}
+		if r.ChosenRunID == r.RejectedRunID {
+			prefSelfPairs++
+		}
+		if r.Chosen == r.Rejected {
+			prefIdenticalText++
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 4, Name: "Preference: 0 self-pairs (chosen_run_id != rejected_run_id)",
+		Expected: "0", Actual: fmt.Sprintf("%d", prefSelfPairs),
+		Passed: prefSelfPairs == 0, Required: true,
+	})
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 4, Name: "Preference: 0 identical-text pairs",
+		Expected: "0", Actual: fmt.Sprintf("%d", prefIdenticalText),
+		Passed: prefIdenticalText == 0, Required: true,
+	})
+
+	// Provenance check: every export row must carry a 64-char hex
+	// sig_hash. Walks sft + rag + pref together since the contract
+	// is uniform across all three.
+	noProv := 0
+	checkProv := func(line string) {
+		var r struct {
+			Provenance struct {
+				SigHash string `json:"sig_hash"`
+			} `json:"provenance"`
+		}
+		if err := json.Unmarshal([]byte(line), &r); err != nil {
+			return
+		}
+		if r.Provenance.SigHash == "" || !sigHashRe.MatchString(r.Provenance.SigHash) {
+			noProv++
+		}
+	}
+	for _, line := range sftRows {
+		checkProv(line)
+	}
+	for _, line := range ragRows {
+		checkProv(line)
+	}
+	for _, line := range prefRows {
+		checkProv(line)
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 4, Name: "every export row carries valid sha256 provenance.sig_hash",
+		Expected: "0 missing", Actual: fmt.Sprintf("%d missing", noProv),
+		Passed: noProv == 0, Required: true,
+	})
+
+	// Quarantine totals (informational — feeds the p4_total_quarantined
+	// metric used by the longitudinal drift signal).
+	totalQuar := int64(0)
+	for _, qp := range []string{
+		"exports/quarantine/sft.jsonl",
+		"exports/quarantine/rag.jsonl",
+		"exports/quarantine/preference.jsonl",
+	} {
+		totalQuar += int64(len(readJSONLLines(filepath.Join(root, qp))))
+	}
+	report.Metrics["p4_total_quarantined"] = totalQuar
+}
+
+// ── Phase 5: receipts validation (observer) ────────────────────────
+
+// runSummaryShape mirrors the Rust RunSummary just enough to
+// validate the file's shape — schema_version, run_hash sha256,
+// git_commit hex, and the 5 stage names. Full schema validation
+// is intentionally NOT ported (it would require porting the
+// schemas/distillation/ TS validators); we check the load-bearing
+// invariants and call it good.
+type runSummaryShape struct {
+	SchemaVersion int    `json:"schema_version"`
+	RunID         string `json:"run_id"`
+	GitCommit     string `json:"git_commit"`
+	RunHash       string `json:"run_hash"`
+	Stages        []struct {
+		Stage string `json:"stage"`
+	} `json:"stages"`
+}
+
+func auditPhase5(root string, report *PhaseCheckReport) {
+	reportsDir := filepath.Join(root, "reports", "distillation")
+	if !fileExists(reportsDir) {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 5, Name: "receipts directory exists",
+			Expected: "reports/distillation/", Actual: "MISSING",
+			Passed: false, Required: true,
+		})
+		return
+	}
+	// Find the most recent run_id directory with a summary.json.
+	// Mirrors the Rust mtime-sort behavior — ordering matters when
+	// both Rust + Go runs land in the same directory.
+	type cand struct {
+		id    string
+		mtime int64
+	}
+	var cands []cand
+	entries, err := os.ReadDir(reportsDir)
+	if err != nil {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 5, Name: "scan reports/distillation",
+			Expected: "no error", Actual: err.Error(),
+			Passed: false, Required: true,
+		})
+		return
+	}
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		sumPath := filepath.Join(reportsDir, e.Name(), "summary.json")
+		st, err := os.Stat(sumPath)
+		if err != nil {
+			continue
+		}
+		cands = append(cands, cand{id: e.Name(), mtime: st.ModTime().UnixMilli()})
+	}
+	if len(cands) == 0 {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 5, Name: "≥1 run with summary.json",
+			Expected: "≥1", Actual: "0",
+			Passed: false, Required: false,
+			Notes: []string{"no Phase 5 run-all has executed yet — Rust: ./scripts/distill run-all"},
+		})
+		return
+	}
+	sort.Slice(cands, func(i, j int) bool { return cands[i].mtime > cands[j].mtime })
+	latest := cands[0]
+	runDir := filepath.Join(reportsDir, latest.id)
+
+	// All 5 stage receipts present.
+	expected := []string{"collect", "score", "export-rag", "export-sft", "export-preference"}
+	missing := []string{}
+	for _, s := range expected {
+		if !fileExists(filepath.Join(runDir, s+".json")) {
+			missing = append(missing, s)
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 5, Name: fmt.Sprintf("latest run (%s) has all 5 stage receipts", latest.id),
+		Expected: strings.Join(expected, ","),
+		Actual: func() string {
+			if len(missing) == 0 {
+				return "all present"
+			}
+			return "missing: " + strings.Join(missing, ",")
+		}(),
+		Passed: len(missing) == 0, Required: true,
+	})
+
+	// Each receipt parses as JSON. Full schema validation (StageReceipt
+	// schema) is Rust-side only; we check basic decodability here.
+	invalid := 0
+	for _, s := range expected {
+		path := filepath.Join(runDir, s+".json")
+		data, err := os.ReadFile(path)
+		if err != nil {
+			continue
+		}
+		var anyShape any
+		if err := json.Unmarshal(data, &anyShape); err != nil {
+			invalid++
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 5, Name: "every stage receipt parses as JSON",
+		Expected: "0 invalid", Actual: fmt.Sprintf("%d invalid", invalid),
+		Passed: invalid == 0, Required: true,
+	})
+
+	// RunSummary shape: schema_version=1, run_hash sha256, git_commit
+	// 40-char hex.
+	summaryPath := filepath.Join(runDir, "summary.json")
+	data, err := os.ReadFile(summaryPath)
+	if err != nil {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 5, Name: "summary.json readable",
+			Expected: "ok", Actual: err.Error(),
+			Passed: false, Required: true,
+		})
+		return
+	}
+	var sum runSummaryShape
+	if err := json.Unmarshal(data, &sum); err != nil {
+		report.Checks = append(report.Checks, PhaseCheck{
+			Phase: 5, Name: "summary.json decodable",
+			Expected: "ok", Actual: err.Error(),
+			Passed: false, Required: true,
+		})
+		return
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 5, Name: "summary.schema_version == 1",
+		Expected: "1", Actual: fmt.Sprintf("%d", sum.SchemaVersion),
+		Passed: sum.SchemaVersion == 1, Required: true,
+	})
+	gitHEADRe := regexp.MustCompile(`^[0-9a-f]{40}$`)
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 5, Name: "summary.git_commit is 40-char hex",
+		Expected: "/^[0-9a-f]{40}$/", Actual: shortHash(sum.GitCommit),
+		Passed: gitHEADRe.MatchString(sum.GitCommit), Required: false,
+	})
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 5, Name: "run_hash is sha256",
+		Expected: "/^[0-9a-f]{64}$/", Actual: shortHash(sum.RunHash),
+		Passed: sigHashRe.MatchString(sum.RunHash), Required: true,
+	})
+}
+
+func shortHash(h string) string {
+	if len(h) <= 16 {
+		return h
+	}
+	return h[:16] + "..."
+}
+
+// ── Phase 7: replay log shape (observer) ───────────────────────────
+
+// auditPhase7 checks data/_kb/replay_runs.jsonl exists and contains
+// well-shaped records. Mirrors Rust phase 7's "persisted log shape"
+// check but skips the live-replay invocation (which would require
+// porting Rust replay.ts, a substantial effort). The full Rust
+// phase 7 also runs 3 dry-run replays — operators wanting that
+// signal continue to invoke the Rust audit-full.
+func auditPhase7(root string, report *PhaseCheckReport) {
+	logPath := filepath.Join(root, "data", "_kb", "replay_runs.jsonl")
+	lines := readJSONLLines(logPath)
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 7, Name: "replay_runs.jsonl exists",
+		Expected: "exists with ≥1 row",
+		Actual: func() string {
+			if !fileExists(logPath) {
+				return "missing"
+			}
+			return fmt.Sprintf("%d rows total", len(lines))
+		}(),
+		Passed: fileExists(logPath), Required: false,
+	})
+	if !fileExists(logPath) {
+		return
+	}
+	// Validate shape on a sample of rows — full validation across
+	// thousands of lines isn't worth the cost, and a structural
+	// problem will show up in any sample.
+	sample := lines
+	if len(sample) > 50 {
+		sample = sample[len(sample)-50:]
+	}
+	malformed := 0
+	for _, line := range sample {
+		var anyShape any
+		if err := json.Unmarshal([]byte(line), &anyShape); err != nil {
+			malformed++
+		}
+	}
+	report.Checks = append(report.Checks, PhaseCheck{
+		Phase: 7, Name: "replay_runs.jsonl tail rows parse as JSON",
+		Expected: "0 malformed in last 50", Actual: fmt.Sprintf("%d malformed", malformed),
+		Passed: malformed == 0, Required: true,
+	})
+}
+
+// ── helpers ────────────────────────────────────────────────────────
+
+func fileExists(p string) bool {
+	_, err := os.Stat(p)
+	return err == nil
+}
+
+// readJSONLLines reads a JSONL file and returns non-empty lines.
+// Returns nil on missing file (matches Rust's existsSync ? read : []).
+func readJSONLLines(path string) []string {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	out := make([]string, 0)
+	for _, line := range strings.Split(string(data), "\n") {
+		if strings.TrimSpace(line) != "" {
+			out = append(out, line)
+		}
+	}
+	return out
+}
+
+// FormatAuditFullReport renders a Markdown report mirroring the
+// Rust phase8-full-audit-report.md shape so operators reading
+// across runtimes don't have to re-learn the layout.
+func FormatAuditFullReport(report PhaseCheckReport) string {
+	var b strings.Builder
+	fmt.Fprintln(&b, "# Audit-FULL report (Go)")
+	fmt.Fprintln(&b)
+	if report.GitHEAD != "" {
+		fmt.Fprintf(&b, "**git HEAD:** `%s`\n\n", report.GitHEAD)
+	}
+	failed := report.Failed
+	total := 0
+	for _, c := range report.Checks {
+		if c.Required {
+			total++
+		}
+	}
+	verdict := "PASS"
+	if failed > 0 {
+		verdict = "FAIL"
+	}
+	fmt.Fprintf(&b, "**Verdict:** %s — %d/%d required checks passed; %d phase(s) deferred.\n\n",
+		verdict, total-failed, total, report.Skipped)
+
+	fmt.Fprintln(&b, "## Checks")
+	fmt.Fprintln(&b)
+	fmt.Fprintln(&b, "| phase | name | expected | actual | required | passed |")
+	fmt.Fprintln(&b, "|---|---|---|---|---|---|")
+	for _, c := range report.Checks {
+		req := "no"
+		if c.Required {
+			req = "**yes**"
+		}
+		passed := "✗"
+		if c.Passed {
+			passed = "✓"
+		}
+		fmt.Fprintf(&b, "| %d | %s | %s | %s | %s | %s |\n",
+			c.Phase, c.Name, c.Expected, c.Actual, req, passed)
+		for _, n := range c.Notes {
+			fmt.Fprintf(&b, "| | _note_ | %s | | | |\n", n)
+		}
+	}
+
+	if len(report.Metrics) > 0 {
+		fmt.Fprintln(&b)
+		fmt.Fprintln(&b, "## Metrics")
+		fmt.Fprintln(&b)
+		fmt.Fprintln(&b, "| metric | value |")
+		fmt.Fprintln(&b, "|---|---:|")
+		// Stable order for diffs.
+		names := make([]string, 0, len(report.Metrics))
+		for k := range report.Metrics {
+			names = append(names, k)
+		}
+		sort.Strings(names)
+		for _, k := range names {
+			fmt.Fprintf(&b, "| %s | %d |\n", k, report.Metrics[k])
+		}
+	}
+	return b.String()
+}

internal/distillation/audit_full_test.go

@@ -0,0 +1,394 @@
+package distillation
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestRunAuditFull_EmptyRoot: missing data directories yield
+// failures on required checks but doesn't error out the run.
+// Operator running on a fresh box sees the report with the
+// expected "missing" actuals.
+func TestRunAuditFull_EmptyRoot(t *testing.T) {
+	tmp := t.TempDir()
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	if len(report.Checks) == 0 {
+		t.Fatalf("expected check rows even on empty root, got %d", len(report.Checks))
+	}
+	// Phase 3's "scored-runs on disk" must fail (required); the
+	// failure count rises by at least 1.
+	if report.Failed < 1 {
+		t.Errorf("expected ≥1 required failure on empty root, got %d", report.Failed)
+	}
+}
+
+// TestPhase2_EvidenceTallyFromOnDisk seeds data/evidence/ and
+// asserts phase 2 reads + tallies the rows correctly. The
+// observer-mode port (no live materializer invocation) means the
+// check works against any-runtime-emitted evidence files.
+func TestPhase2_EvidenceTallyFromOnDisk(t *testing.T) {
+	tmp := t.TempDir()
+	dir := filepath.Join(tmp, "data", "evidence", "2026", "05", "01")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	// 3 records: 2 from scrum_reviews (a tier-1 source), 1 from
+	// "other_source" (not in tier-1 list). Phase 2 should tally
+	// 3 rows total + flag 1/4 tier-1 sources hit.
+	jsonl := `{"run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"a","recorded_at":"2026-05-01T00:00:00Z"}}
+{"run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"b","recorded_at":"2026-05-01T00:00:00Z"}}
+{"run_id":"r3","provenance":{"source_file":"data/_kb/other_source.jsonl","sig_hash":"c","recorded_at":"2026-05-01T00:00:00Z"}}
+`
+	if err := os.WriteFile(filepath.Join(dir, "evidence.jsonl"), []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp}) // GoTestModule empty disables phase 1
+	if report.Metrics["p2_evidence_rows"] != 3 {
+		t.Errorf("p2_evidence_rows: got %d, want 3", report.Metrics["p2_evidence_rows"])
+	}
+	if report.Metrics["p2_evidence_skips"] != 0 {
+		t.Errorf("p2_evidence_skips: got %d, want 0", report.Metrics["p2_evidence_skips"])
+	}
+	// Find the tier-1 hit count check.
+	for _, c := range report.Checks {
+		if c.Phase == 2 && c.Name == "tier-1 sources each materialize ≥1 row" {
+			if !c.Passed {
+				t.Errorf("expected tier-1 check to pass with 1/4 sources hit (≥1 = ok), got %+v", c)
+			}
+			if !strings.Contains(c.Actual, "1/4") || !strings.Contains(c.Actual, "scrum_reviews") {
+				t.Errorf("tier-1 actual missing expected counts: %s", c.Actual)
+			}
+		}
+	}
+}
+
+// TestPhase5_FullSummaryFlow seeds reports/distillation/{run_id}/
+// with summary.json + 5 stage receipts and asserts phase 5 passes
+// all required checks.
+func TestPhase5_FullSummaryFlow(t *testing.T) {
+	tmp := t.TempDir()
+	runID := "test-run-id"
+	runDir := filepath.Join(tmp, "reports", "distillation", runID)
+	if err := os.MkdirAll(runDir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	// 5 stage receipts (parse-as-JSON only — full schema validation
+	// is Rust-side).
+	for _, s := range []string{"collect", "score", "export-rag", "export-sft", "export-preference"} {
+		if err := os.WriteFile(filepath.Join(runDir, s+".json"), []byte(`{}`), 0o644); err != nil {
+			t.Fatalf("write %s: %v", s, err)
+		}
+	}
+	// summary.json with valid schema_version, 40-char git_commit, 64-char run_hash.
+	summary := `{
+  "schema_version": 1,
+  "run_id": "test-run-id",
+  "git_commit": "0123456789abcdef0123456789abcdef01234567",
+  "run_hash":   "a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0",
+  "stages": [{"stage":"collect"},{"stage":"score"},{"stage":"export-rag"},{"stage":"export-sft"},{"stage":"export-preference"}]
+}`
+	if err := os.WriteFile(filepath.Join(runDir, "summary.json"), []byte(summary), 0o644); err != nil {
+		t.Fatalf("write summary: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	for _, c := range report.Checks {
+		if c.Phase == 5 && c.Required && !c.Passed {
+			t.Errorf("phase 5 required check failed: %s — actual=%q", c.Name, c.Actual)
+		}
+	}
+}
+
+// TestPhase5_ShortRunHashCaught: a run_hash that isn't 64-char hex
+// must fail the required check.
+func TestPhase5_ShortRunHashCaught(t *testing.T) {
+	tmp := t.TempDir()
+	runDir := filepath.Join(tmp, "reports", "distillation", "id")
+	if err := os.MkdirAll(runDir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	for _, s := range []string{"collect", "score", "export-rag", "export-sft", "export-preference"} {
+		_ = os.WriteFile(filepath.Join(runDir, s+".json"), []byte(`{}`), 0o644)
+	}
+	bad := `{"schema_version":1,"run_id":"id","git_commit":"0123456789abcdef0123456789abcdef01234567","run_hash":"too_short","stages":[]}`
+	_ = os.WriteFile(filepath.Join(runDir, "summary.json"), []byte(bad), 0o644)
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	hashFailed := false
+	for _, c := range report.Checks {
+		if c.Phase == 5 && c.Name == "run_hash is sha256" && !c.Passed {
+			hashFailed = true
+		}
+	}
+	if !hashFailed {
+		t.Errorf("expected run_hash sha256 check to fail on too_short")
+	}
+}
+
+// TestPhase7_ReplayLogReadsFromDisk seeds a replay_runs.jsonl and
+// asserts phase 7 reports the correct row count.
+func TestPhase7_ReplayLogReadsFromDisk(t *testing.T) {
+	tmp := t.TempDir()
+	dir := filepath.Join(tmp, "data", "_kb")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	jsonl := `{"task":"a","passed":true}
+{"task":"b","passed":true}
+{"task":"c","passed":false}
+`
+	if err := os.WriteFile(filepath.Join(dir, "replay_runs.jsonl"), []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	for _, c := range report.Checks {
+		if c.Phase == 7 && c.Name == "replay_runs.jsonl exists" {
+			if !c.Passed {
+				t.Errorf("expected pass, got %+v", c)
+			}
+			if !strings.Contains(c.Actual, "3 rows") {
+				t.Errorf("expected '3 rows' in actual, got %s", c.Actual)
+			}
+		}
+	}
+}
+
+// TestPhase7_MalformedTailRowsCaught seeds a replay log with a
+// trailing malformed row and asserts the structural check fires.
+func TestPhase7_MalformedTailRowsCaught(t *testing.T) {
+	tmp := t.TempDir()
+	dir := filepath.Join(tmp, "data", "_kb")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	jsonl := `{"task":"a"}
+{"task":"b"}
+not valid json garbage
+`
+	_ = os.WriteFile(filepath.Join(dir, "replay_runs.jsonl"), []byte(jsonl), 0o644)
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	parseFailed := false
+	for _, c := range report.Checks {
+		if c.Phase == 7 && c.Name == "replay_runs.jsonl tail rows parse as JSON" && !c.Passed {
+			parseFailed = true
+		}
+	}
+	if !parseFailed {
+		t.Errorf("expected tail-row parse check to fail on malformed line")
+	}
+}
+
+// TestRunAuditFull_FullFixtureFlow seeds a complete data layout
+// and verifies all phases produce the expected metrics + a clean
+// PASS verdict. Locks the end-to-end orchestration.
+func TestRunAuditFull_FullFixtureFlow(t *testing.T) {
+	tmp := t.TempDir()
+	// scored-runs: one accepted record (passes phase 3 required check)
+	scoredDir := filepath.Join(tmp, "data", "scored-runs", "2026", "05", "01")
+	if err := os.MkdirAll(scoredDir, 0o755); err != nil {
+		t.Fatalf("mkdir scored: %v", err)
+	}
+	scoredJSONL := `{"category":"accepted","evidence_run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0","recorded_at":"2026-05-01T00:00:00Z"}}
+{"category":"partially_accepted","evidence_run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff1","recorded_at":"2026-05-01T00:00:00Z"}}
+{"category":"rejected","evidence_run_id":"r3","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff2","recorded_at":"2026-05-01T00:00:00Z"}}
+`
+	if err := os.WriteFile(filepath.Join(scoredDir, "run.jsonl"), []byte(scoredJSONL), 0o644); err != nil {
+		t.Fatalf("write scored: %v", err)
+	}
+
+	// SFT export: only legal quality scores, valid sig_hash on every row.
+	sftDir := filepath.Join(tmp, "exports", "sft")
+	if err := os.MkdirAll(sftDir, 0o755); err != nil {
+		t.Fatalf("mkdir sft: %v", err)
+	}
+	sftJSONL := `{"quality_score":"accepted","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+{"quality_score":"partially_accepted","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff1"}}
+`
+	if err := os.WriteFile(filepath.Join(sftDir, "instruction_response.jsonl"), []byte(sftJSONL), 0o644); err != nil {
+		t.Fatalf("write sft: %v", err)
+	}
+
+	// RAG: no rejected leaks
+	ragDir := filepath.Join(tmp, "exports", "rag")
+	if err := os.MkdirAll(ragDir, 0o755); err != nil {
+		t.Fatalf("mkdir rag: %v", err)
+	}
+	ragJSONL := `{"success_score":"accepted","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+`
+	if err := os.WriteFile(filepath.Join(ragDir, "playbooks.jsonl"), []byte(ragJSONL), 0o644); err != nil {
+		t.Fatalf("write rag: %v", err)
+	}
+
+	// Preference: distinct chosen vs rejected, no self-pairs
+	prefDir := filepath.Join(tmp, "exports", "preference")
+	if err := os.MkdirAll(prefDir, 0o755); err != nil {
+		t.Fatalf("mkdir pref: %v", err)
+	}
+	prefJSONL := `{"chosen_run_id":"a","rejected_run_id":"b","chosen":"good","rejected":"bad","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+`
+	if err := os.WriteFile(filepath.Join(prefDir, "chosen_rejected.jsonl"), []byte(prefJSONL), 0o644); err != nil {
+		t.Fatalf("write pref: %v", err)
+	}
+
+	// Phase 2: evidence directory with at least one row.
+	evidenceDir := filepath.Join(tmp, "data", "evidence", "2026", "05", "01")
+	if err := os.MkdirAll(evidenceDir, 0o755); err != nil {
+		t.Fatalf("mkdir evidence: %v", err)
+	}
+	evidenceJSONL := `{"run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"a","recorded_at":"2026-05-01T00:00:00Z"}}
+`
+	if err := os.WriteFile(filepath.Join(evidenceDir, "evidence.jsonl"), []byte(evidenceJSONL), 0o644); err != nil {
+		t.Fatalf("write evidence: %v", err)
+	}
+
+	// Phase 5: reports/distillation/{run_id}/ with summary + 5 receipts.
+	runDir := filepath.Join(tmp, "reports", "distillation", "test-run")
+	if err := os.MkdirAll(runDir, 0o755); err != nil {
+		t.Fatalf("mkdir runDir: %v", err)
+	}
+	for _, s := range []string{"collect", "score", "export-rag", "export-sft", "export-preference"} {
+		_ = os.WriteFile(filepath.Join(runDir, s+".json"), []byte(`{}`), 0o644)
+	}
+	summaryJSON := `{"schema_version":1,"run_id":"test-run","git_commit":"0123456789abcdef0123456789abcdef01234567","run_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0","stages":[]}`
+	_ = os.WriteFile(filepath.Join(runDir, "summary.json"), []byte(summaryJSON), 0o644)
+
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	if report.Failed != 0 {
+		t.Errorf("clean fixture should have 0 required failures, got %d", report.Failed)
+		for _, c := range report.Checks {
+			if c.Required && !c.Passed {
+				t.Logf("  failed: phase=%d name=%q actual=%q", c.Phase, c.Name, c.Actual)
+			}
+		}
+	}
+	// Metrics populated correctly
+	if report.Metrics["p3_accepted"] != 1 {
+		t.Errorf("p3_accepted: got %d, want 1", report.Metrics["p3_accepted"])
+	}
+	if report.Metrics["p3_partial"] != 1 {
+		t.Errorf("p3_partial: got %d, want 1", report.Metrics["p3_partial"])
+	}
+	if report.Metrics["p3_rejected"] != 1 {
+		t.Errorf("p3_rejected: got %d, want 1", report.Metrics["p3_rejected"])
+	}
+	if report.Metrics["p4_sft_rows"] != 2 {
+		t.Errorf("p4_sft_rows: got %d, want 2", report.Metrics["p4_sft_rows"])
+	}
+	if report.Metrics["p4_rag_rows"] != 1 {
+		t.Errorf("p4_rag_rows: got %d, want 1", report.Metrics["p4_rag_rows"])
+	}
+	if report.Metrics["p4_pref_pairs"] != 1 {
+		t.Errorf("p4_pref_pairs: got %d, want 1", report.Metrics["p4_pref_pairs"])
+	}
+}
+
+// TestPhase4_SftFirewallCatchesRejected: contamination must never
+// leak into SFT export. Test seeds a row with a forbidden
+// quality_score and asserts the firewall flags it.
+func TestPhase4_SftFirewallCatchesRejected(t *testing.T) {
+	tmp := t.TempDir()
+	sftDir := filepath.Join(tmp, "exports", "sft")
+	if err := os.MkdirAll(sftDir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	bad := `{"quality_score":"rejected","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+`
+	if err := os.WriteFile(filepath.Join(sftDir, "instruction_response.jsonl"), []byte(bad), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	found := false
+	for _, c := range report.Checks {
+		if c.Phase == 4 && strings.Contains(c.Name, "SFT contamination firewall") {
+			if c.Passed {
+				t.Errorf("firewall should fail on rejected SFT row, but check passed")
+			}
+			if c.Actual != "1" {
+				t.Errorf("firewall actual: got %q, want '1'", c.Actual)
+			}
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("firewall check not present in report")
+	}
+}
+
+// TestPhase4_PreferenceSelfPairCaught: same chosen + rejected run_id
+// is structural noise and must be flagged.
+func TestPhase4_PreferenceSelfPairCaught(t *testing.T) {
+	tmp := t.TempDir()
+	prefDir := filepath.Join(tmp, "exports", "preference")
+	if err := os.MkdirAll(prefDir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	bad := `{"chosen_run_id":"X","rejected_run_id":"X","chosen":"a","rejected":"b","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+`
+	if err := os.WriteFile(filepath.Join(prefDir, "chosen_rejected.jsonl"), []byte(bad), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	found := false
+	for _, c := range report.Checks {
+		if c.Phase == 4 && strings.Contains(c.Name, "self-pairs") {
+			if c.Passed {
+				t.Errorf("self-pair check should fail, but passed")
+			}
+			found = true
+		}
+	}
+	if !found {
+		t.Errorf("self-pair check not present in report")
+	}
+}
+
+// TestPhase4_ProvenanceRequiresValidSha256: bad sig_hash must be
+// flagged. Locks the regex shape — only 64-char lowercase hex.
+func TestPhase4_ProvenanceRequiresValidSha256(t *testing.T) {
+	tmp := t.TempDir()
+	sftDir := filepath.Join(tmp, "exports", "sft")
+	if err := os.MkdirAll(sftDir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	// Three rows: one valid, one wrong-length, one wrong-charset (uppercase).
+	bad := `{"quality_score":"accepted","provenance":{"sig_hash":"a1b2c3d4e5f60718293a4b5c6d7e8f900112233445566778899aabbccddeeff0"}}
+{"quality_score":"accepted","provenance":{"sig_hash":"too_short"}}
+{"quality_score":"accepted","provenance":{"sig_hash":"A1B2C3D4E5F60718293A4B5C6D7E8F900112233445566778899AABBCCDDEEFF0"}}
+`
+	if err := os.WriteFile(filepath.Join(sftDir, "instruction_response.jsonl"), []byte(bad), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	report := RunAuditFull(AuditFullOptions{Root: tmp})
+	for _, c := range report.Checks {
+		if c.Phase == 4 && strings.Contains(c.Name, "sig_hash") {
+			if c.Actual != "2 missing" {
+				t.Errorf("provenance check: got actual=%q, want '2 missing'", c.Actual)
+			}
+			if c.Passed {
+				t.Errorf("provenance check should fail with 2 bad sig_hashes")
+			}
+		}
+	}
+}
+
+// TestFormatAuditFullReport_RendersCheckTable: smoke-test the
+// Markdown formatter — operators should see the right verdict +
+// per-phase rows.
+func TestFormatAuditFullReport_RendersCheckTable(t *testing.T) {
+	report := PhaseCheckReport{
+		GitHEAD: "deadbeef",
+		Checks: []PhaseCheck{
+			{Phase: 0, Name: "test check", Expected: "x", Actual: "x", Passed: true, Required: true},
+			{Phase: 4, Name: "fail check", Expected: "0", Actual: "5", Passed: false, Required: true},
+		},
+		Metrics: map[string]int64{"p3_accepted": 42, "p4_sft_rows": 17},
+		Failed:  1,
+		Skipped: 4,
+	}
+	out := FormatAuditFullReport(report)
+	for _, want := range []string{"FAIL", "deadbeef", "test check", "fail check", "p3_accepted", "42", "deferred"} {
+		if !strings.Contains(out, want) {
+			t.Errorf("expected %q in formatted report:\n%s", want, out)
+		}
+	}
+}

internal/distillation/scorer.go

@@ -0,0 +1,410 @@
+package distillation
+
+// scorer.go — pure deterministic Success Scorer (port of Rust
+// scripts/distillation/scorer.ts at e7636f2).
+//
+// Takes one EvidenceRecord, returns category + reasons + sub_scores.
+// NO I/O, NO LLM, NO clock reads, NO mutable state. Identical input
+// → identical output forever. Same contract as the Rust source —
+// future scoring-rule changes bump ScorerVersion atomically with
+// the logic.
+//
+// Three-class strategy mirrors the Rust source taxonomy
+// (docs/recon/local-distillation-recon.md + data/_kb/evidence_health.md):
+//
+//   CLASS A — verdict-bearing
+//     scrum_reviews, observer_reviews, audits, contract_analyses
+//     Direct scoring from existing markers / observer_verdict
+//
+//   CLASS B — telemetry-rich
+//     auto_apply, outcomes, mode_experiments
+//     Markers exist but partial; needs_human_review fills the gap
+//
+//   CLASS C — pure-extraction (no native scoring signal)
+//     distilled_*, audit_facts, observer_escalations
+//     Default needs_human_review; v2 will JOIN to parent verdict
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// sourceClass categorizes an EvidenceRecord's source_file for the
+// scorer's three-class dispatch.
+type sourceClass string
+
+const (
+	classVerdict    sourceClass = "verdict"
+	classTelemetry  sourceClass = "telemetry"
+	classExtraction sourceClass = "extraction"
+)
+
+// sourceClassFor maps a source_file (from provenance) to a class.
+// Centralized so adding a new source is a one-line change. Mirrors
+// the Rust switch on the stem (data/_kb/X.jsonl → X).
+func sourceClassFor(sourceFile string) sourceClass {
+	stem := strings.TrimSuffix(strings.TrimPrefix(sourceFile, "data/_kb/"), ".jsonl")
+	switch stem {
+	case "scrum_reviews", "observer_reviews", "audits", "contract_analyses":
+		return classVerdict
+	case "auto_apply", "outcomes", "mode_experiments":
+		return classTelemetry
+	case "distilled_facts", "distilled_procedures", "distilled_config_hints",
+		"audit_facts", "observer_escalations":
+		return classExtraction
+	default:
+		// Unknown source → most conservative path (forces
+		// needs_human_review until a transform is added).
+		return classExtraction
+	}
+}
+
+// stemOf extracts the stable corpus identifier from a source_file.
+// E.g. "data/_kb/scrum_reviews.jsonl" → "scrum_reviews".
+func stemOf(sourceFile string) string {
+	return strings.TrimSuffix(strings.TrimPrefix(sourceFile, "data/_kb/"), ".jsonl")
+}
+
+// ScoreOutput is the scorer's return shape — category + reasons +
+// the captured sub-signals. Reasons is always non-empty (validator
+// requires it).
+type ScoreOutput struct {
+	Category  ScoreCategory
+	Reasons   []string
+	SubScores *SubScores
+}
+
+// ScoreRecord dispatches an EvidenceRecord to the appropriate class
+// scorer and returns the verdict + reasons + sub-scores. Pure
+// function. Caller wraps the output in a ScoredRun via BuildScoredRun
+// for the on-wire shape.
+func ScoreRecord(rec EvidenceRecord) ScoreOutput {
+	cls := sourceClassFor(rec.Provenance.SourceFile)
+	stem := stemOf(rec.Provenance.SourceFile)
+
+	switch cls {
+	case classVerdict:
+		switch stem {
+		case "scrum_reviews":
+			return scoreScrumReview(rec)
+		case "observer_reviews":
+			return scoreObserverReview(rec)
+		case "audits":
+			return scoreAudit(rec)
+		case "contract_analyses":
+			return scoreContractAnalysis(rec)
+		}
+	case classTelemetry:
+		switch stem {
+		case "auto_apply":
+			return scoreAutoApply(rec)
+		case "outcomes":
+			return scoreOutcomes(rec)
+		case "mode_experiments":
+			return scoreModeExperiment(rec)
+		}
+	}
+	return scoreExtraction()
+}
+
+// BuildScoredRun composes a complete ScoredRun for persistence.
+// Caller supplies recorded_at + the source file path/line offset.
+// SigHash is computed deterministically from the EvidenceRecord
+// JSON; ScoredRun traces to the materialized evidence row.
+func BuildScoredRun(rec EvidenceRecord, sourceFile string, lineOffset int64, recordedAt string) (ScoredRun, error) {
+	out := ScoreRecord(rec)
+	sig, err := canonicalSha256(rec)
+	if err != nil {
+		return ScoredRun{}, fmt.Errorf("scoredrun sig hash: %w", err)
+	}
+	return ScoredRun{
+		SchemaVersion:  ScoredRunSchemaVersion,
+		EvidenceRunID:  rec.RunID,
+		EvidenceTaskID: rec.TaskID,
+		Category:       out.Category,
+		Reasons:        out.Reasons,
+		ScoredAt:       recordedAt,
+		ScorerVersion:  ScorerVersion,
+		SubScores:      out.SubScores,
+		Provenance: Provenance{
+			SourceFile: sourceFile,
+			LineOffset: lineOffset,
+			SigHash:    sig,
+			RecordedAt: recordedAt,
+		},
+	}, nil
+}
+
+// canonicalSha256 hashes a value's canonical JSON encoding. Used
+// for ScoredRun.Provenance.SigHash. Matches the Rust pattern of
+// "hash the structured object, not the raw source bytes" so
+// re-materialization with same logic produces same hash.
+func canonicalSha256(v any) (string, error) {
+	bs, err := json.Marshal(v)
+	if err != nil {
+		return "", err
+	}
+	sum := sha256.Sum256(bs)
+	return hex.EncodeToString(sum[:]), nil
+}
+
+// ─── Class A: verdict-bearing ────────────────────────────────────
+
+func scoreScrumReview(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	successMarker := findPrefix(r.SuccessMarkers, "accepted_on_attempt_")
+	if successMarker == "" {
+		return ScoreOutput{
+			Category: CategoryNeedsHumanReview,
+			Reasons:  []string{"scrum_review missing accepted_on_attempt_* success marker"},
+			SubScores: subs,
+		}
+	}
+	attemptStr := strings.TrimPrefix(successMarker, "accepted_on_attempt_")
+	attempt, err := strconv.Atoi(attemptStr)
+	if err != nil {
+		return ScoreOutput{
+			Category: CategoryNeedsHumanReview,
+			Reasons:  []string{"scrum_review accepted_on_attempt_* marker has non-integer suffix: " + attemptStr},
+			SubScores: subs,
+		}
+	}
+	subs.AcceptedOnAttempt = &attempt
+	switch {
+	case attempt == 1:
+		return ScoreOutput{
+			Category: CategoryAccepted,
+			Reasons:  []string{"scrum: accepted on first attempt"},
+			SubScores: subs,
+		}
+	case attempt <= 3:
+		return ScoreOutput{
+			Category: CategoryPartiallyAccepted,
+			Reasons:  []string{fmt.Sprintf("scrum: accepted after %d attempts", attempt)},
+			SubScores: subs,
+		}
+	default:
+		return ScoreOutput{
+			Category: CategoryPartiallyAccepted,
+			Reasons:  []string{fmt.Sprintf("scrum: accepted only after %d attempts (high-cost path)", attempt)},
+			SubScores: subs,
+		}
+	}
+}
+
+func scoreObserverReview(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	switch r.ObserverVerdict {
+	case VerdictAccept:
+		subs.ObserverVerdict = VerdictAccept
+		return ScoreOutput{
+			Category: CategoryAccepted,
+			Reasons:  []string{"observer accepted the reviewed attempt"},
+			SubScores: subs,
+		}
+	case VerdictReject:
+		subs.ObserverVerdict = VerdictReject
+		return ScoreOutput{
+			Category: CategoryRejected,
+			Reasons:  []string{"observer rejected the reviewed attempt"},
+			SubScores: subs,
+		}
+	case VerdictCycle:
+		subs.ObserverVerdict = VerdictCycle
+		return ScoreOutput{
+			Category: CategoryPartiallyAccepted,
+			Reasons:  []string{"observer flagged the attempt as cycling — partial signal"},
+			SubScores: subs,
+		}
+	default:
+		return ScoreOutput{
+			Category: CategoryNeedsHumanReview,
+			Reasons:  []string{fmt.Sprintf("observer_verdict missing or unrecognized: %q", r.ObserverVerdict)},
+			SubScores: subs,
+		}
+	}
+}
+
+func scoreAudit(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	succ := r.SuccessMarkers
+	fail := r.FailureMarkers
+
+	// Legacy markers (back-compat with pre-fix materializations).
+	if contains(succ, "approved") {
+		return ScoreOutput{Category: CategoryAccepted,
+			Reasons: []string{"audit overall=approved (legacy marker)"}, SubScores: subs}
+	}
+	if contains(fail, "blocked") {
+		return ScoreOutput{Category: CategoryRejected,
+			Reasons: []string{"audit overall=block (legacy marker)"}, SubScores: subs}
+	}
+	if contains(fail, "request_changes") {
+		return ScoreOutput{Category: CategoryPartiallyAccepted,
+			Reasons: []string{"audit overall=request_changes (legacy marker)"}, SubScores: subs}
+	}
+
+	// Severity-derived markers (Phase 2 transform).
+	sevSucc := findPrefix(succ, "audit_severity_")
+	sevFail := findPrefix(fail, "audit_severity_")
+	if sevSucc != "" {
+		return ScoreOutput{Category: CategoryAccepted,
+			Reasons: []string{sevSucc + " → minor finding"}, SubScores: subs}
+	}
+	if sevFail == "audit_severity_medium" {
+		return ScoreOutput{Category: CategoryPartiallyAccepted,
+			Reasons: []string{"audit_severity_medium → finding warrants review"}, SubScores: subs}
+	}
+	if sevFail == "audit_severity_high" || sevFail == "audit_severity_critical" {
+		return ScoreOutput{Category: CategoryRejected,
+			Reasons: []string{sevFail + " → blocking finding"}, SubScores: subs}
+	}
+	return ScoreOutput{Category: CategoryNeedsHumanReview,
+		Reasons: []string{"audit row has no severity or overall marker"}, SubScores: subs}
+}
+
+func scoreContractAnalysis(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	// failure_markers takes precedence: explicit rejection beats absent verdict.
+	if contains(r.FailureMarkers, "observer_rejected") || r.ObserverVerdict == VerdictReject {
+		subs.ObserverVerdict = VerdictReject
+		return ScoreOutput{Category: CategoryRejected,
+			Reasons: []string{"contract analysis: observer rejected"}, SubScores: subs}
+	}
+	switch r.ObserverVerdict {
+	case VerdictAccept:
+		subs.ObserverVerdict = VerdictAccept
+		return ScoreOutput{Category: CategoryAccepted,
+			Reasons: []string{"contract analysis: observer accepted"}, SubScores: subs}
+	case VerdictCycle:
+		subs.ObserverVerdict = VerdictCycle
+		return ScoreOutput{Category: CategoryPartiallyAccepted,
+			Reasons: []string{"contract analysis: observer cycled (partial)"}, SubScores: subs}
+	}
+	return ScoreOutput{Category: CategoryNeedsHumanReview,
+		Reasons: []string{"contract analysis: no observer verdict signal"}, SubScores: subs}
+}
+
+// ─── Class B: telemetry-rich ─────────────────────────────────────
+
+func scoreAutoApply(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	if contains(r.SuccessMarkers, "committed") {
+		t := true
+		subs.CargoGreen = &t
+		return ScoreOutput{Category: CategoryAccepted,
+			Reasons: []string{"auto_apply: patch committed (cargo green + warning baseline + rationale alignment passed)"},
+			SubScores: subs}
+	}
+	reverted := findContaining(r.FailureMarkers, "reverted")
+	if reverted != "" {
+		if strings.Contains(reverted, "build_red") {
+			f := false
+			subs.CargoGreen = &f
+		}
+		return ScoreOutput{Category: CategoryRejected,
+			Reasons: []string{"auto_apply: " + reverted}, SubScores: subs}
+	}
+	return ScoreOutput{Category: CategoryNeedsHumanReview,
+		Reasons: []string{"auto_apply: no commit + no revert (likely no_patches or dry_run)"},
+		SubScores: subs}
+}
+
+func scoreOutcomes(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	if contains(r.SuccessMarkers, "all_events_ok") {
+		return ScoreOutput{Category: CategoryAccepted,
+			Reasons: []string{"outcomes: all events ok"}, SubScores: subs}
+	}
+	if gap := numericFromMap(r.ValidationResults, "gap_signals"); gap > 0 {
+		return ScoreOutput{Category: CategoryPartiallyAccepted,
+			Reasons: []string{fmt.Sprintf("outcomes: %d gap signal(s) detected", int(gap))},
+			SubScores: subs}
+	}
+	return ScoreOutput{Category: CategoryNeedsHumanReview,
+		Reasons: []string{"outcomes: no decisive marker — defer to human"},
+		SubScores: subs}
+}
+
+func scoreModeExperiment(r EvidenceRecord) ScoreOutput {
+	subs := &SubScores{}
+	if strings.TrimSpace(r.Text) == "" {
+		return ScoreOutput{Category: CategoryRejected,
+			Reasons: []string{"mode_experiment: empty response text"}, SubScores: subs}
+	}
+	if r.LatencyMs > 120_000 {
+		return ScoreOutput{Category: CategoryPartiallyAccepted,
+			Reasons: []string{fmt.Sprintf("mode_experiment: latency %dms exceeds 2-minute soft cap", r.LatencyMs)},
+			SubScores: subs}
+	}
+	return ScoreOutput{Category: CategoryNeedsHumanReview,
+		Reasons: []string{"mode_experiment: response present, latency within bounds; verdict not yet wired"},
+		SubScores: subs}
+}
+
+// ─── Class C: pure-extraction ────────────────────────────────────
+
+func scoreExtraction() ScoreOutput {
+	return ScoreOutput{
+		Category: CategoryNeedsHumanReview,
+		Reasons:  []string{"extraction-class source has no native scoring signal — JOIN to parent verdict pending Phase 3 v2"},
+		SubScores: &SubScores{},
+	}
+}
+
+// ─── Internal helpers ────────────────────────────────────────────
+
+func contains(slice []string, want string) bool {
+	for _, s := range slice {
+		if s == want {
+			return true
+		}
+	}
+	return false
+}
+
+func findPrefix(slice []string, prefix string) string {
+	for _, s := range slice {
+		if strings.HasPrefix(s, prefix) {
+			return s
+		}
+	}
+	return ""
+}
+
+func findContaining(slice []string, sub string) string {
+	for _, s := range slice {
+		if strings.Contains(s, sub) {
+			return s
+		}
+	}
+	return ""
+}
+
+func numericFromMap(m map[string]any, key string) float64 {
+	if m == nil {
+		return 0
+	}
+	v, ok := m[key]
+	if !ok {
+		return 0
+	}
+	switch n := v.(type) {
+	case int:
+		return float64(n)
+	case int64:
+		return float64(n)
+	case float32:
+		return float64(n)
+	case float64:
+		return n
+	case json.Number:
+		f, _ := n.Float64()
+		return f
+	}
+	return 0
+}

internal/distillation/scorer_test.go

@@ -0,0 +1,375 @@
+package distillation
+
+import (
+	"errors"
+	"strings"
+	"testing"
+)
+
+func mkRecord(sourceFile string) EvidenceRecord {
+	return EvidenceRecord{
+		RunID:         "run-1",
+		TaskID:        "task-1",
+		Timestamp:     "2026-04-29T12:00:00Z",
+		SchemaVersion: EvidenceSchemaVersion,
+		Provenance: Provenance{
+			SourceFile: sourceFile,
+			SigHash:    "deadbeef",
+			RecordedAt: "2026-04-29T12:00:01Z",
+		},
+	}
+}
+
+func TestSourceClassFor(t *testing.T) {
+	cases := []struct {
+		path string
+		want sourceClass
+	}{
+		{"data/_kb/scrum_reviews.jsonl", classVerdict},
+		{"data/_kb/observer_reviews.jsonl", classVerdict},
+		{"data/_kb/audits.jsonl", classVerdict},
+		{"data/_kb/contract_analyses.jsonl", classVerdict},
+		{"data/_kb/auto_apply.jsonl", classTelemetry},
+		{"data/_kb/outcomes.jsonl", classTelemetry},
+		{"data/_kb/mode_experiments.jsonl", classTelemetry},
+		{"data/_kb/distilled_facts.jsonl", classExtraction},
+		{"data/_kb/audit_facts.jsonl", classExtraction},
+		{"data/_kb/observer_escalations.jsonl", classExtraction},
+		{"data/_kb/wholly_unknown.jsonl", classExtraction}, // unknown → extraction (conservative)
+	}
+	for _, c := range cases {
+		got := sourceClassFor(c.path)
+		if got != c.want {
+			t.Errorf("sourceClassFor(%q): want %q, got %q", c.path, c.want, got)
+		}
+	}
+}
+
+func TestScoreScrumReview(t *testing.T) {
+	cases := []struct {
+		name           string
+		successMarkers []string
+		wantCategory   ScoreCategory
+		wantReasonSub  string
+	}{
+		{
+			name:           "first attempt → accepted",
+			successMarkers: []string{"accepted_on_attempt_1"},
+			wantCategory:   CategoryAccepted,
+			wantReasonSub:  "first attempt",
+		},
+		{
+			name:           "second attempt → partial",
+			successMarkers: []string{"accepted_on_attempt_2"},
+			wantCategory:   CategoryPartiallyAccepted,
+			wantReasonSub:  "after 2 attempts",
+		},
+		{
+			name:           "fourth attempt → partial (high-cost)",
+			successMarkers: []string{"accepted_on_attempt_4"},
+			wantCategory:   CategoryPartiallyAccepted,
+			wantReasonSub:  "high-cost",
+		},
+		{
+			name:           "missing marker → needs_human_review",
+			successMarkers: []string{},
+			wantCategory:   CategoryNeedsHumanReview,
+			wantReasonSub:  "missing accepted_on_attempt",
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			rec := mkRecord("data/_kb/scrum_reviews.jsonl")
+			rec.SuccessMarkers = c.successMarkers
+			out := ScoreRecord(rec)
+			if out.Category != c.wantCategory {
+				t.Errorf("category: want %q, got %q (reasons=%v)", c.wantCategory, out.Category, out.Reasons)
+			}
+			if !reasonsContain(out.Reasons, c.wantReasonSub) {
+				t.Errorf("reasons missing %q: %v", c.wantReasonSub, out.Reasons)
+			}
+		})
+	}
+}
+
+func TestScoreObserverReview(t *testing.T) {
+	cases := []struct {
+		verdict ObserverVerdict
+		want    ScoreCategory
+	}{
+		{VerdictAccept, CategoryAccepted},
+		{VerdictReject, CategoryRejected},
+		{VerdictCycle, CategoryPartiallyAccepted},
+		{"", CategoryNeedsHumanReview},
+		{"weird-verdict", CategoryNeedsHumanReview},
+	}
+	for _, c := range cases {
+		rec := mkRecord("data/_kb/observer_reviews.jsonl")
+		rec.ObserverVerdict = c.verdict
+		out := ScoreRecord(rec)
+		if out.Category != c.want {
+			t.Errorf("verdict=%q: want %q, got %q", c.verdict, c.want, out.Category)
+		}
+	}
+}
+
+func TestScoreAudit_LegacyAndSeverityMarkers(t *testing.T) {
+	cases := []struct {
+		name string
+		succ []string
+		fail []string
+		want ScoreCategory
+	}{
+		{"legacy approved", []string{"approved"}, nil, CategoryAccepted},
+		{"legacy blocked", nil, []string{"blocked"}, CategoryRejected},
+		{"legacy request_changes", nil, []string{"request_changes"}, CategoryPartiallyAccepted},
+		{"severity_low → accepted", []string{"audit_severity_low"}, nil, CategoryAccepted},
+		{"severity_info → accepted", []string{"audit_severity_info"}, nil, CategoryAccepted},
+		{"severity_medium fail → partial", nil, []string{"audit_severity_medium"}, CategoryPartiallyAccepted},
+		{"severity_high → rejected", nil, []string{"audit_severity_high"}, CategoryRejected},
+		{"severity_critical → rejected", nil, []string{"audit_severity_critical"}, CategoryRejected},
+		{"no markers", nil, nil, CategoryNeedsHumanReview},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			rec := mkRecord("data/_kb/audits.jsonl")
+			rec.SuccessMarkers = c.succ
+			rec.FailureMarkers = c.fail
+			out := ScoreRecord(rec)
+			if out.Category != c.want {
+				t.Errorf("want %q, got %q (reasons=%v)", c.want, out.Category, out.Reasons)
+			}
+		})
+	}
+}
+
+func TestScoreAutoApply(t *testing.T) {
+	cases := []struct {
+		name string
+		succ []string
+		fail []string
+		want ScoreCategory
+	}{
+		{"committed → accepted", []string{"committed"}, nil, CategoryAccepted},
+		{"reverted_build_red → rejected", nil, []string{"reverted_build_red"}, CategoryRejected},
+		{"reverted other → rejected", nil, []string{"reverted_warning_count_up"}, CategoryRejected},
+		{"no signal → needs_human", nil, nil, CategoryNeedsHumanReview},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			rec := mkRecord("data/_kb/auto_apply.jsonl")
+			rec.SuccessMarkers = c.succ
+			rec.FailureMarkers = c.fail
+			out := ScoreRecord(rec)
+			if out.Category != c.want {
+				t.Errorf("want %q, got %q", c.want, out.Category)
+			}
+		})
+	}
+}
+
+func TestScoreOutcomes(t *testing.T) {
+	rec := mkRecord("data/_kb/outcomes.jsonl")
+	rec.SuccessMarkers = []string{"all_events_ok"}
+	if out := ScoreRecord(rec); out.Category != CategoryAccepted {
+		t.Errorf("all_events_ok: want accepted, got %q", out.Category)
+	}
+
+	rec2 := mkRecord("data/_kb/outcomes.jsonl")
+	rec2.ValidationResults = map[string]any{"gap_signals": float64(2)}
+	if out := ScoreRecord(rec2); out.Category != CategoryPartiallyAccepted {
+		t.Errorf("gap_signals=2: want partial, got %q (reasons=%v)", out.Category, out.Reasons)
+	}
+
+	rec3 := mkRecord("data/_kb/outcomes.jsonl")
+	if out := ScoreRecord(rec3); out.Category != CategoryNeedsHumanReview {
+		t.Errorf("no signal: want needs_human, got %q", out.Category)
+	}
+}
+
+func TestScoreModeExperiment(t *testing.T) {
+	rec := mkRecord("data/_kb/mode_experiments.jsonl")
+	rec.Text = ""
+	if out := ScoreRecord(rec); out.Category != CategoryRejected {
+		t.Errorf("empty text: want rejected, got %q", out.Category)
+	}
+
+	rec.Text = "real response"
+	rec.LatencyMs = 130_000
+	if out := ScoreRecord(rec); out.Category != CategoryPartiallyAccepted {
+		t.Errorf("over latency cap: want partial, got %q", out.Category)
+	}
+
+	rec.LatencyMs = 5000
+	if out := ScoreRecord(rec); out.Category != CategoryNeedsHumanReview {
+		t.Errorf("normal: want needs_human (verdict not yet wired), got %q", out.Category)
+	}
+}
+
+func TestScoreExtraction_Defaults(t *testing.T) {
+	for _, src := range []string{
+		"data/_kb/distilled_facts.jsonl",
+		"data/_kb/distilled_procedures.jsonl",
+		"data/_kb/audit_facts.jsonl",
+		"data/_kb/observer_escalations.jsonl",
+	} {
+		rec := mkRecord(src)
+		out := ScoreRecord(rec)
+		if out.Category != CategoryNeedsHumanReview {
+			t.Errorf("%s: want needs_human_review, got %q", src, out.Category)
+		}
+	}
+}
+
+// ─── Contamination firewall — the safety-critical guarantee ───────
+
+func TestValidateSftSample_RejectsContaminationCategories(t *testing.T) {
+	for _, contaminated := range []SftQualityScore{
+		SftQualityScore("rejected"),
+		SftQualityScore("needs_human_review"),
+	} {
+		s := goodSftSample()
+		s.QualityScore = contaminated
+		err := ValidateSftSample(s)
+		if err == nil {
+			t.Errorf("contaminated quality_score=%q should fail validation", contaminated)
+			continue
+		}
+		if !errors.Is(err, ErrSftContamination) {
+			t.Errorf("contaminated %q: want errors.Is(err, ErrSftContamination), got %v", contaminated, err)
+		}
+	}
+}
+
+func TestValidateSftSample_AcceptsLegalCategories(t *testing.T) {
+	for _, legal := range []SftQualityScore{SftQualityAccepted, SftQualityPartiallyAccepted} {
+		s := goodSftSample()
+		s.QualityScore = legal
+		if err := ValidateSftSample(s); err != nil {
+			t.Errorf("legal quality_score=%q failed: %v", legal, err)
+		}
+	}
+}
+
+func TestValidateSftSample_RejectsTypoCategory(t *testing.T) {
+	s := goodSftSample()
+	s.QualityScore = "approved" // close to "accepted" but wrong
+	err := ValidateSftSample(s)
+	if err == nil {
+		t.Fatal("typo category should fail validation")
+	}
+	// Typo is NOT contamination — should be a regular ValidationError,
+	// not the firewall sentinel. This distinguishes "you typo'd" from
+	// "you broke the spec."
+	if errors.Is(err, ErrSftContamination) {
+		t.Error("typo should not surface as ErrSftContamination")
+	}
+}
+
+func TestValidateSftSample_RejectsEmptyPair(t *testing.T) {
+	s := goodSftSample()
+	s.Instruction = "  "
+	if err := ValidateSftSample(s); err == nil {
+		t.Error("whitespace-only instruction should fail")
+	}
+
+	s2 := goodSftSample()
+	s2.Response = ""
+	if err := ValidateSftSample(s2); err == nil {
+		t.Error("empty response should fail")
+	}
+}
+
+func TestValidateScoredRun_ReasonsRequired(t *testing.T) {
+	r := ScoredRun{
+		SchemaVersion:  ScoredRunSchemaVersion,
+		EvidenceRunID:  "x",
+		EvidenceTaskID: "y",
+		Category:       CategoryAccepted,
+		Reasons:        nil, // empty — must fail
+		ScoredAt:       "2026-04-29T12:00:00Z",
+		ScorerVersion:  ScorerVersion,
+		Provenance: Provenance{
+			SourceFile: "data/_kb/scrum_reviews.jsonl",
+			SigHash:    "abc",
+			RecordedAt: "2026-04-29T12:00:00Z",
+		},
+	}
+	err := ValidateScoredRun(r)
+	if err == nil {
+		t.Fatal("empty reasons should fail")
+	}
+	if !strings.Contains(err.Error(), "reasons") {
+		t.Errorf("error should mention reasons: %v", err)
+	}
+}
+
+func TestBuildScoredRun_DeterministicSigHash(t *testing.T) {
+	rec := mkRecord("data/_kb/scrum_reviews.jsonl")
+	rec.SuccessMarkers = []string{"accepted_on_attempt_1"}
+
+	r1, err := BuildScoredRun(rec, "data/scored-runs/2026/04/29/x.jsonl", 0, "2026-04-29T12:00:00Z")
+	if err != nil {
+		t.Fatal(err)
+	}
+	r2, err := BuildScoredRun(rec, "data/scored-runs/2026/04/29/x.jsonl", 0, "2026-04-29T12:00:00Z")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if r1.Provenance.SigHash != r2.Provenance.SigHash {
+		t.Errorf("identical EvidenceRecord should produce identical sig_hash: %s vs %s",
+			r1.Provenance.SigHash, r2.Provenance.SigHash)
+	}
+	if r1.Category != CategoryAccepted {
+		t.Errorf("scored category: %q", r1.Category)
+	}
+	if r1.ScorerVersion != ScorerVersion {
+		t.Errorf("scorer version stamped wrong: %q", r1.ScorerVersion)
+	}
+}
+
+func TestScoreRecord_PureFunction_NoMutationOfInput(t *testing.T) {
+	// Belt-and-braces: the contract says "NO mutable state, identical
+	// input → identical output forever." Verify by scoring the same
+	// record twice and ensuring the input hasn't been touched.
+	rec := mkRecord("data/_kb/scrum_reviews.jsonl")
+	rec.SuccessMarkers = []string{"accepted_on_attempt_2"}
+	original := rec
+	out1 := ScoreRecord(rec)
+	out2 := ScoreRecord(rec)
+	if rec.RunID != original.RunID || len(rec.SuccessMarkers) != 1 {
+		t.Error("ScoreRecord mutated its input")
+	}
+	if out1.Category != out2.Category {
+		t.Error("ScoreRecord is non-deterministic")
+	}
+}
+
+// ─── Helpers ─────────────────────────────────────────────────────
+
+func goodSftSample() SftSample {
+	return SftSample{
+		SchemaVersion: SftSampleSchemaVersion,
+		ID:            "sft-1",
+		Instruction:   "summarize the diff",
+		Context:       "diff body...",
+		Response:      "the diff adds a function",
+		SourceRunID:   "run-1",
+		QualityScore:  SftQualityAccepted,
+		CreatedAt:     "2026-04-29T12:00:00Z",
+		Provenance: Provenance{
+			SourceFile: "data/scored-runs/2026/04/29/x.jsonl",
+			SigHash:    "deadbeef",
+			RecordedAt: "2026-04-29T12:00:01Z",
+		},
+	}
+}
+
+func reasonsContain(reasons []string, sub string) bool {
+	for _, r := range reasons {
+		if strings.Contains(r, sub) {
+			return true
+		}
+	}
+	return false
+}

internal/distillation/sft_export.go

@@ -0,0 +1,399 @@
+package distillation
+
+// SFT (Supervised Fine-Tuning) export pipeline. Closes the SUBSTRATE
+// half of OPEN #2 — types, contamination firewall, file-listing
+// helper. The actual synthesis (turning EvidenceRecord + ScoredRun
+// into instruction/input/response triples) is still on the Rust
+// side at scripts/distillation/export_sft.ts and will land in a
+// follow-up wave.
+//
+// Why ship substrate without synthesis: the firewall constants and
+// types are the load-bearing contamination guarantees. Once they're
+// pinned in Go (with tests proving the firewall set is exactly
+// {rejected, needs_human_review} and never expands), the synthesis
+// port becomes a translation exercise rather than a design one.
+//
+// Per the project_distillation_substrate.md note: SFT_NEVER is one
+// of the "what NOT to touch casually" knobs. Replicating it here in
+// Go preserves the cross-runtime invariant — the contamination
+// firewall fires even if the SFT export is run from the Go side.
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+)
+
+// SftNever is declared in types.go (the load-bearing contamination
+// firewall — pinned at the type-level so every consumer reads the
+// same source of truth). IsSftNever below is the predicate
+// helper; it lives here because it's specific to the SFT export
+// path, not a property of the type system.
+//
+// IsSftNever returns true if a scored run's category is on the
+// contamination firewall list. Inlinable; called per-record in the
+// hot synthesis loop.
+func IsSftNever(c ScoreCategory) bool {
+	for _, blocked := range SftNever {
+		if c == blocked {
+			return true
+		}
+	}
+	return false
+}
+
+// ExportSftOptions mirrors the TS shape so callers porting from
+// Rust have an identity-translation surface. Root is the lakehouse
+// data root (default $LH_DISTILL_ROOT or /home/profit/lakehouse).
+// RecordedAt is the timestamp stamped on emitted SFT samples for
+// lineage. IncludePartial toggles "emit even when evidence record
+// is missing some optional fields"; DryRun skips file writes.
+type ExportSftOptions struct {
+	Root           string
+	RecordedAt     string
+	IncludePartial bool
+	DryRun         bool
+}
+
+// ExportSftResult mirrors the TS result shape exactly so a
+// callable swap between sides doesn't break consumers reading the
+// JSON.
+type ExportSftResult struct {
+	ScoredFilesRead    int    `json:"scored_files_read"`
+	RecordsRead        int    `json:"records_read"`
+	RecordsExported    int    `json:"records_exported"`
+	RecordsQuarantined int    `json:"records_quarantined"`
+	OutputPath         string `json:"output_path"`
+	QuarantineSummary  string `json:"quarantine_summary"`
+}
+
+// ListScoredRunFiles walks {root}/data/scored-runs/YYYY/MM/DD/*.jsonl
+// and returns the sorted list. Empty when the dir doesn't exist
+// (matches Rust behavior — caller should treat zero-files as a
+// no-op, not an error).
+func ListScoredRunFiles(root string) ([]string, error) {
+	if root == "" {
+		return nil, errors.New("distillation: empty root")
+	}
+	base := filepath.Join(root, "data", "scored-runs")
+	if _, err := os.Stat(base); os.IsNotExist(err) {
+		return nil, nil
+	} else if err != nil {
+		return nil, fmt.Errorf("stat %s: %w", base, err)
+	}
+	var out []string
+	years, err := os.ReadDir(base)
+	if err != nil {
+		return nil, fmt.Errorf("read %s: %w", base, err)
+	}
+	sortDirEntries(years)
+	for _, y := range years {
+		if !y.IsDir() {
+			continue
+		}
+		months, err := os.ReadDir(filepath.Join(base, y.Name()))
+		if err != nil {
+			continue
+		}
+		sortDirEntries(months)
+		for _, m := range months {
+			if !m.IsDir() {
+				continue
+			}
+			days, err := os.ReadDir(filepath.Join(base, y.Name(), m.Name()))
+			if err != nil {
+				continue
+			}
+			sortDirEntries(days)
+			for _, d := range days {
+				if !d.IsDir() {
+					continue
+				}
+				files, err := os.ReadDir(filepath.Join(base, y.Name(), m.Name(), d.Name()))
+				if err != nil {
+					continue
+				}
+				sortDirEntries(files)
+				for _, f := range files {
+					if strings.HasSuffix(f.Name(), ".jsonl") {
+						out = append(out, filepath.Join(base, y.Name(), m.Name(), d.Name(), f.Name()))
+					}
+				}
+			}
+		}
+	}
+	return out, nil
+}
+
+// sortDirEntries sorts dir entries by name in-place. Stable
+// alphabetical so the directory walk is deterministic — important
+// for the audit_baselines longitudinal signal which expects the
+// same order across runs.
+func sortDirEntries(entries []os.DirEntry) {
+	sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
+}
+
+// LoadScoredRunsFromFile reads a JSONL of ScoredRun records.
+// Returns the slice + the count of malformed lines (skipped).
+// This is the read-half — the synthesis half (turn ScoredRun +
+// EvidenceRecord into SftSample) is the not-yet-ported piece.
+func LoadScoredRunsFromFile(path string) ([]ScoredRun, int, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil, 0, err
+	}
+	lines := strings.Split(string(data), "\n")
+	out := make([]ScoredRun, 0, len(lines))
+	skipped := 0
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+		var sr ScoredRun
+		if err := json.Unmarshal([]byte(line), &sr); err != nil {
+			skipped++
+			continue
+		}
+		out = append(out, sr)
+	}
+	return out, skipped, nil
+}
+
+// LoadEvidenceByRunID reads {evidence_path}'s JSONL and returns a
+// map[run_id]EvidenceRecord. evidence_path is derived by replacing
+// /scored-runs/ with /evidence/ in the scored-runs path — the
+// convention the Rust pipeline uses to colocate evidence + scores.
+//
+// Cache is the per-call memoization map: callers iterating multiple
+// scored-runs files in the same directory benefit from shared
+// loads. Pass nil to disable caching (useful in tests).
+func LoadEvidenceByRunID(scoredPath string, cache map[string]map[string]EvidenceRecord) (map[string]EvidenceRecord, error) {
+	evidencePath := strings.Replace(scoredPath, "/scored-runs/", "/evidence/", 1)
+	if cache != nil {
+		if hit, ok := cache[evidencePath]; ok {
+			return hit, nil
+		}
+	}
+	out := make(map[string]EvidenceRecord)
+	data, err := os.ReadFile(evidencePath)
+	if os.IsNotExist(err) {
+		if cache != nil {
+			cache[evidencePath] = out
+		}
+		return out, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	for _, line := range strings.Split(string(data), "\n") {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+		var r EvidenceRecord
+		if err := json.Unmarshal([]byte(line), &r); err != nil {
+			continue // tolerate malformed lines per Rust convention
+		}
+		out[r.RunID] = r
+	}
+	if cache != nil {
+		cache[evidencePath] = out
+	}
+	return out, nil
+}
+
+// SynthesizeSft turns a (ScoredRun, EvidenceRecord) pair into an
+// SftSample. Returns nil when the record is unsuitable for SFT:
+//   - model_role isn't executor/reviewer/applier (extraction-class
+//     records have no instruction→response shape)
+//   - text is empty (nothing to learn from)
+//
+// Mirrors the Rust synthesizeSft exactly so byte-identical SFT
+// JSONL output is achievable across both runtimes (operators
+// running a/b validation between Rust + Go pipelines depend on this).
+func SynthesizeSft(scored ScoredRun, ev EvidenceRecord, recordedAt, sftID string) *SftSample {
+	if ev.ModelRole != RoleExecutor && ev.ModelRole != RoleReviewer && ev.ModelRole != RoleApplier {
+		return nil
+	}
+	text := ev.Text
+	if strings.TrimSpace(text) == "" {
+		return nil
+	}
+
+	stem := stemFromSourceFile(ev.Provenance.SourceFile)
+	firstSourceFile := ""
+	if len(ev.SourceFiles) > 0 {
+		firstSourceFile = ev.SourceFiles[0]
+	}
+	if firstSourceFile == "" {
+		firstSourceFile = "<file>"
+	}
+	instruction := buildInstruction(stem, ev.TaskID, firstSourceFile)
+
+	// Context — what the model could see. Keep terse; matches Rust's
+	// space-separated " · " join.
+	var ctxParts []string
+	if ev.RetrievedContext != nil {
+		if len(ev.RetrievedContext.MatrixCorpora) > 0 {
+			ctxParts = append(ctxParts, "matrix="+strings.Join(ev.RetrievedContext.MatrixCorpora, ","))
+		}
+		if ev.RetrievedContext.PathwayFingerprintsSeen > 0 {
+			ctxParts = append(ctxParts, fmt.Sprintf("pathway_fingerprints=%d", ev.RetrievedContext.PathwayFingerprintsSeen))
+		}
+	}
+	if ev.ModelName != "" {
+		ctxParts = append(ctxParts, "model="+ev.ModelName)
+	}
+	context := strings.Join(ctxParts, " · ")
+
+	return &SftSample{
+		SchemaVersion: SftSampleSchemaVersion,
+		ID:            sftID,
+		Instruction:   instruction,
+		Context:       context,
+		Response:      text,
+		SourceRunID:   scored.EvidenceRunID,
+		QualityScore:  SftQualityScore(scored.Category),
+		CreatedAt:     recordedAt,
+		Provenance: Provenance{
+			SourceFile: scored.Provenance.SourceFile,
+			LineOffset: scored.Provenance.LineOffset,
+			SigHash:    scored.Provenance.SigHash,
+			RecordedAt: recordedAt,
+		},
+	}
+}
+
+// stemFromSourceFile mirrors the Rust path-cleanup: strip the
+// `data/_kb/` prefix and the `.jsonl` suffix. Used to dispatch
+// the per-source-class instruction templates below.
+func stemFromSourceFile(path string) string {
+	s := strings.TrimPrefix(path, "data/_kb/")
+	s = strings.TrimSuffix(s, ".jsonl")
+	return s
+}
+
+// buildInstruction returns the per-source-class instruction
+// template. Mirrors the Rust switch statement byte-for-byte so
+// trained-model behavior is unchanged across runtimes.
+func buildInstruction(stem, taskID, firstSourceFile string) string {
+	switch stem {
+	case "scrum_reviews":
+		return fmt.Sprintf("Review the file '%s' against the PRD + change-proposal context. Produce a forensic audit with findings, severity, confidence, patches.", firstSourceFile)
+	case "mode_experiments":
+		return fmt.Sprintf("Run task_class='%s' for file '%s'. Produce the mode-runner's expected output shape.", taskID, firstSourceFile)
+	case "auto_apply":
+		return fmt.Sprintf("Auto-apply: emit a 6-line surgical patch for '%s' based on the latest scrum review findings.", firstSourceFile)
+	case "audits":
+		phase := strings.TrimPrefix(taskID, "phase:")
+		return fmt.Sprintf("Audit phase '%s' and report findings with severity.", phase)
+	case "observer_reviews":
+		return fmt.Sprintf("Observer-review the latest attempt on '%s'. Verdict: accept | reject | cycle.", firstSourceFile)
+	case "contract_analyses":
+		permit := strings.TrimPrefix(taskID, "permit:")
+		// The Rust version reads ev.metadata.contractor when present.
+		// Go's EvidenceRecord doesn't carry a free-form metadata bag
+		// today, so we always emit the no-contractor form. Operators
+		// who need contractor-aware instructions can extend this once
+		// the metadata field lands in EvidenceRecord (separate ADR).
+		return fmt.Sprintf("Analyze permit '%s'. Recommend with risk markers.", permit)
+	case "outcomes":
+		return "Run scenario; report per-event outcome with citations."
+	default:
+		return fmt.Sprintf("Source '%s' run; produce the appropriate output for this task type.", stem)
+	}
+}
+
+// ExportSft is the now-COMPLETE port (not just substrate). Walks
+// scored-runs files, loads paired evidence, applies the firewall,
+// synthesizes SFT samples for surviving records, writes the
+// combined JSONL output.
+//
+// Behavior:
+//   - DryRun=true: counts but doesn't write files.
+//   - IncludePartial=true: emit samples even when synthesis returns
+//     "skip" reasons that aren't on the firewall (model_role
+//     mismatch, empty text). Default false.
+//   - Empty Root or missing scored-runs dir: returns zero-counts,
+//     no error.
+//
+// Output JSONL format: one SftSample per line, UTF-8, LF-terminated,
+// matching the Rust convention so a/b validation between runtimes
+// can diff the files directly.
+func ExportSft(opts ExportSftOptions) (ExportSftResult, error) {
+	if opts.RecordedAt == "" {
+		return ExportSftResult{}, errors.New("distillation: ExportSft requires RecordedAt")
+	}
+	res := ExportSftResult{
+		OutputPath: filepath.Join(opts.Root, "data", "distilled", "sft", "sft_export.jsonl"),
+	}
+	files, err := ListScoredRunFiles(opts.Root)
+	if err != nil {
+		return res, fmt.Errorf("list scored runs: %w", err)
+	}
+	res.ScoredFilesRead = len(files)
+
+	evCache := make(map[string]map[string]EvidenceRecord)
+	var output []byte
+	skippedNotInstructable := 0
+
+	for _, f := range files {
+		runs, _, err := LoadScoredRunsFromFile(f)
+		if err != nil {
+			continue
+		}
+		evidence, err := LoadEvidenceByRunID(f, evCache)
+		if err != nil {
+			continue
+		}
+		res.RecordsRead += len(runs)
+		for i, r := range runs {
+			if IsSftNever(r.Category) {
+				res.RecordsQuarantined++
+				continue
+			}
+			ev, ok := evidence[r.EvidenceRunID]
+			if !ok {
+				// Evidence missing — without it we can't synthesize.
+				// IncludePartial=true would let us emit a sample with
+				// just scored-side fields, but the contract says
+				// SftSample.Response must be non-empty (no evidence
+				// = no text), so skip regardless.
+				skippedNotInstructable++
+				continue
+			}
+			sftID := fmt.Sprintf("%s:%d", filepath.Base(f), i)
+			sample := SynthesizeSft(r, ev, opts.RecordedAt, sftID)
+			if sample == nil {
+				skippedNotInstructable++
+				continue
+			}
+			if !opts.DryRun {
+				line, err := json.Marshal(sample)
+				if err != nil {
+					skippedNotInstructable++
+					continue
+				}
+				output = append(output, line...)
+				output = append(output, '\n')
+			}
+			res.RecordsExported++
+		}
+	}
+
+	res.QuarantineSummary = fmt.Sprintf("firewall=%d, not-instructable=%d", res.RecordsQuarantined, skippedNotInstructable)
+	if !opts.DryRun && len(output) > 0 {
+		if err := os.MkdirAll(filepath.Dir(res.OutputPath), 0o755); err != nil {
+			return res, fmt.Errorf("mkdir output: %w", err)
+		}
+		if err := os.WriteFile(res.OutputPath, output, 0o644); err != nil {
+			return res, fmt.Errorf("write output: %w", err)
+		}
+	}
+	return res, nil
+}

internal/distillation/sft_export_test.go

@@ -0,0 +1,385 @@
+package distillation
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestIsSftNever_Firewall locks the contamination firewall set:
+// the predicate fires for "rejected" and "needs_human_review" and
+// no others. Per project_distillation_substrate.md: this is one of
+// the substrate's load-bearing knobs — touching the firewall set
+// requires explicit sign-off.
+func TestIsSftNever_Firewall(t *testing.T) {
+	mustBlock := []ScoreCategory{
+		CategoryRejected,
+		CategoryNeedsHumanReview,
+	}
+	for _, c := range mustBlock {
+		if !IsSftNever(c) {
+			t.Errorf("firewall must block %q", c)
+		}
+	}
+	// Anything else should NOT be blocked. Read every category
+	// constant in this package and assert non-blocked unless it's
+	// in mustBlock.
+	allKnown := []ScoreCategory{
+		CategoryAccepted,
+		CategoryPartiallyAccepted,
+		CategoryRejected,
+		CategoryNeedsHumanReview,
+	}
+	for _, c := range allKnown {
+		shouldBlock := false
+		for _, b := range mustBlock {
+			if c == b {
+				shouldBlock = true
+				break
+			}
+		}
+		if got := IsSftNever(c); got != shouldBlock {
+			t.Errorf("IsSftNever(%q) = %v, want %v", c, got, shouldBlock)
+		}
+	}
+	// Unknown category is NOT blocked — that's the safe default
+	// (operators bumping ScoreCategory enum should explicitly add
+	// to firewall if they want it gated).
+	if IsSftNever(ScoreCategory("custom_future_category")) {
+		t.Errorf("unknown category must not be blocked by firewall")
+	}
+}
+
+// TestSftNever_PinsExpectedSet locks the firewall slice contents.
+// If a future commit adds or removes categories from SftNever, this
+// test fails — forcing the change through review.
+func TestSftNever_PinsExpectedSet(t *testing.T) {
+	want := map[ScoreCategory]bool{
+		CategoryRejected:         true,
+		CategoryNeedsHumanReview: true,
+	}
+	if len(SftNever) != len(want) {
+		t.Fatalf("SftNever has %d entries, want %d (firewall set changed without review?)",
+			len(SftNever), len(want))
+	}
+	for _, c := range SftNever {
+		if !want[c] {
+			t.Errorf("SftNever contains %q, which is not in the expected firewall set", c)
+		}
+	}
+}
+
+// TestListScoredRunFiles_Empty: missing root → no files, no error.
+// Matches Rust behavior; operators running ExportSft on a fresh box
+// shouldn't see an error before any scored runs have landed.
+func TestListScoredRunFiles_Empty(t *testing.T) {
+	tmp := t.TempDir()
+	files, err := ListScoredRunFiles(tmp)
+	if err != nil {
+		t.Fatalf("ListScoredRunFiles: %v", err)
+	}
+	if len(files) != 0 {
+		t.Errorf("empty root: expected 0 files, got %d", len(files))
+	}
+}
+
+// TestListScoredRunFiles_WalksYearMonthDay locks the directory walk
+// pattern: data/scored-runs/YYYY/MM/DD/*.jsonl. Subset of full
+// Rust-side test coverage but proves the walk visits the right
+// nesting.
+func TestListScoredRunFiles_WalksYearMonthDay(t *testing.T) {
+	tmp := t.TempDir()
+	// Create the expected nested structure.
+	dirs := []string{
+		filepath.Join(tmp, "data", "scored-runs", "2026", "04", "30"),
+		filepath.Join(tmp, "data", "scored-runs", "2026", "05", "01"),
+	}
+	for _, d := range dirs {
+		if err := os.MkdirAll(d, 0o755); err != nil {
+			t.Fatalf("mkdir: %v", err)
+		}
+	}
+	// Drop a JSONL in each + a non-JSONL we should skip.
+	for i, d := range dirs {
+		jsonlPath := filepath.Join(d, "run.jsonl")
+		if err := os.WriteFile(jsonlPath, []byte("{}\n"), 0o644); err != nil {
+			t.Fatalf("write %s: %v", jsonlPath, err)
+		}
+		// Non-JSONL — must be skipped.
+		other := filepath.Join(d, "skip.txt")
+		if err := os.WriteFile(other, []byte("ignore me"), 0o644); err != nil {
+			t.Fatalf("write %s: %v", other, err)
+		}
+		_ = i
+	}
+	files, err := ListScoredRunFiles(tmp)
+	if err != nil {
+		t.Fatalf("ListScoredRunFiles: %v", err)
+	}
+	if len(files) != 2 {
+		t.Errorf("expected 2 .jsonl files, got %d (%v)", len(files), files)
+	}
+	// Sort order: 2026-04-30 before 2026-05-01. Critical for audit
+	// baselines — the longitudinal signal depends on stable order.
+	if len(files) >= 2 {
+		if files[0] >= files[1] {
+			t.Errorf("files not sorted ascending: %q vs %q", files[0], files[1])
+		}
+	}
+	// Non-JSONL must be skipped.
+	for _, f := range files {
+		if filepath.Ext(f) != ".jsonl" {
+			t.Errorf("listing returned non-.jsonl: %q", f)
+		}
+	}
+}
+
+// TestSynthesizeSft_PerSourceClass locks the per-source-class
+// instruction templates byte-for-byte against the Rust source.
+// If a future commit changes a template, this test fails — the
+// trained-model behavior shifts under our feet.
+func TestSynthesizeSft_PerSourceClass(t *testing.T) {
+	cases := []struct {
+		name        string
+		sourceFile  string
+		taskID      string
+		sourceFiles []string
+		wantPrefix  string
+	}{
+		{
+			"scrum_reviews",
+			"data/_kb/scrum_reviews.jsonl",
+			"any",
+			[]string{"src/foo.rs"},
+			"Review the file 'src/foo.rs' against",
+		},
+		{
+			"mode_experiments",
+			"data/_kb/mode_experiments.jsonl",
+			"task_42",
+			[]string{"src/bar.go"},
+			"Run task_class='task_42' for file 'src/bar.go'.",
+		},
+		{
+			"auto_apply",
+			"data/_kb/auto_apply.jsonl",
+			"any",
+			[]string{"src/baz.ts"},
+			"Auto-apply: emit a 6-line surgical patch for 'src/baz.ts'",
+		},
+		{
+			"audits with phase: prefix stripped",
+			"data/_kb/audits.jsonl",
+			"phase:G2",
+			nil,
+			"Audit phase 'G2' and report findings",
+		},
+		{
+			"observer_reviews",
+			"data/_kb/observer_reviews.jsonl",
+			"any",
+			[]string{"f.rs"},
+			"Observer-review the latest attempt on 'f.rs'.",
+		},
+		{
+			"contract_analyses with permit: prefix",
+			"data/_kb/contract_analyses.jsonl",
+			"permit:ABC123",
+			nil,
+			"Analyze permit 'ABC123'. Recommend with risk markers.",
+		},
+		{
+			"outcomes",
+			"data/_kb/outcomes.jsonl",
+			"any",
+			nil,
+			"Run scenario; report per-event outcome with citations.",
+		},
+		{
+			"unknown source falls back to default",
+			"data/_kb/something_new.jsonl",
+			"any",
+			nil,
+			"Source 'something_new' run; produce the appropriate output",
+		},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			scored := ScoredRun{
+				EvidenceRunID: "rid",
+				Category:      CategoryAccepted,
+				Provenance: Provenance{
+					SourceFile: c.sourceFile,
+					SigHash:    "abc",
+					RecordedAt: "2026-04-30T00:00:00Z",
+				},
+			}
+			ev := EvidenceRecord{
+				RunID:       "rid",
+				TaskID:      c.taskID,
+				ModelRole:   RoleExecutor,
+				Text:        "model response text",
+				SourceFiles: c.sourceFiles,
+				Provenance:  Provenance{SourceFile: c.sourceFile, SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
+			}
+			sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "test-id")
+			if sample == nil {
+				t.Fatalf("expected non-nil sample for %s", c.name)
+			}
+			if !strings.HasPrefix(sample.Instruction, c.wantPrefix) {
+				t.Errorf("instruction prefix mismatch:\n  got:  %q\n  want: %q...", sample.Instruction, c.wantPrefix)
+			}
+		})
+	}
+}
+
+// TestSynthesizeSft_RejectsExtraction: extraction-class records
+// have no instruction→response shape (they're pure data extraction,
+// not model-output-as-training-target). Synthesis must return nil.
+func TestSynthesizeSft_RejectsExtraction(t *testing.T) {
+	ev := EvidenceRecord{
+		RunID:     "rid",
+		ModelRole: RoleExtractor,
+		Text:      "extracted data",
+		Provenance: Provenance{SourceFile: "data/_kb/anything.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
+	}
+	scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
+	if sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id"); sample != nil {
+		t.Errorf("extraction record must produce nil sample, got %+v", sample)
+	}
+}
+
+// TestSynthesizeSft_RejectsEmptyText: text is the response side of
+// the SFT pair; empty text means nothing to learn.
+func TestSynthesizeSft_RejectsEmptyText(t *testing.T) {
+	ev := EvidenceRecord{
+		RunID:     "rid",
+		ModelRole: RoleExecutor,
+		Text:      "   \n\t",
+		Provenance: Provenance{SourceFile: "data/_kb/scrum_reviews.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
+	}
+	scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
+	if sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id"); sample != nil {
+		t.Errorf("empty-text record must produce nil sample, got %+v", sample)
+	}
+}
+
+// TestSynthesizeSft_ContextAssembly verifies the terse " · "-joined
+// context string carries matrix corpora + pathway fingerprints +
+// model name in the documented order.
+func TestSynthesizeSft_ContextAssembly(t *testing.T) {
+	ev := EvidenceRecord{
+		RunID:     "rid",
+		ModelRole: RoleReviewer,
+		Text:      "verdict",
+		ModelName: "qwen3.5",
+		RetrievedContext: &RetrievedContext{
+			MatrixCorpora:           []string{"workers", "candidates"},
+			PathwayFingerprintsSeen: 88,
+		},
+		Provenance: Provenance{SourceFile: "data/_kb/scrum_reviews.jsonl", SigHash: "abc", RecordedAt: "2026-04-30T00:00:00Z"},
+	}
+	scored := ScoredRun{EvidenceRunID: "rid", Category: CategoryAccepted, Provenance: ev.Provenance}
+	sample := SynthesizeSft(scored, ev, "2026-04-30T00:00:00Z", "id")
+	if sample == nil {
+		t.Fatalf("expected non-nil sample")
+	}
+	want := "matrix=workers,candidates · pathway_fingerprints=88 · model=qwen3.5"
+	if sample.Context != want {
+		t.Errorf("context mismatch:\n  got:  %q\n  want: %q", sample.Context, want)
+	}
+}
+
+// TestExportSft_FullPort_WritesJSONL covers the fully-ported path:
+// scored runs + paired evidence both present, synthesis produces
+// SftSamples, output JSONL is written. Locks the end-to-end
+// contract that next-wave changes (synthesis tweaks, output layout)
+// have to preserve.
+func TestExportSft_FullPort_WritesJSONL(t *testing.T) {
+	tmp := t.TempDir()
+	scoredDir := filepath.Join(tmp, "data", "scored-runs", "2026", "04", "30")
+	evidenceDir := filepath.Join(tmp, "data", "evidence", "2026", "04", "30")
+	for _, d := range []string{scoredDir, evidenceDir} {
+		if err := os.MkdirAll(d, 0o755); err != nil {
+			t.Fatalf("mkdir %s: %v", d, err)
+		}
+	}
+	scoredJSONL := `{"category":"accepted","evidence_run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
+{"category":"rejected","evidence_run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
+`
+	evidenceJSONL := `{"run_id":"r1","model_role":"executor","text":"some review output","source_files":["src/foo.rs"],"provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
+{"run_id":"r2","model_role":"executor","text":"another output","source_files":["src/bar.rs"],"provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
+`
+	if err := os.WriteFile(filepath.Join(scoredDir, "run.jsonl"), []byte(scoredJSONL), 0o644); err != nil {
+		t.Fatalf("write scored: %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(evidenceDir, "run.jsonl"), []byte(evidenceJSONL), 0o644); err != nil {
+		t.Fatalf("write evidence: %v", err)
+	}
+	res, err := ExportSft(ExportSftOptions{
+		Root:       tmp,
+		RecordedAt: "2026-04-30T00:00:00Z",
+	})
+	if err != nil {
+		t.Fatalf("ExportSft: %v", err)
+	}
+	if res.RecordsRead != 2 || res.RecordsExported != 1 || res.RecordsQuarantined != 1 {
+		t.Errorf("counts: read=%d exported=%d quarantined=%d (want 2/1/1)",
+			res.RecordsRead, res.RecordsExported, res.RecordsQuarantined)
+	}
+	out, err := os.ReadFile(res.OutputPath)
+	if err != nil {
+		t.Fatalf("read output: %v", err)
+	}
+	if !strings.Contains(string(out), "Review the file 'src/foo.rs'") {
+		t.Errorf("output missing expected scrum_reviews instruction; got:\n%s", string(out))
+	}
+	if strings.Contains(string(out), "src/bar.rs") {
+		t.Errorf("output contains rejected record's source_file — firewall leak")
+	}
+}
+
+// TestExportSft_FirewallFiresBeforeEvidenceLoad locks the order-of-
+// operations: even if evidence records are missing, the
+// firewall counts records as quarantined so the contamination
+// guarantee never depends on side data being present. Records
+// that pass the firewall but lack evidence get the more honest
+// "not instructable" label rather than being silently exported.
+func TestExportSft_FirewallFiresBeforeEvidenceLoad(t *testing.T) {
+	tmp := t.TempDir()
+	dir := filepath.Join(tmp, "data", "scored-runs", "2026", "04", "30")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	jsonl := `{"category":"accepted","evidence_run_id":"r1","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h1","recorded_at":"2026-04-30T00:00:00Z"}}
+{"category":"rejected","evidence_run_id":"r2","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h2","recorded_at":"2026-04-30T00:00:00Z"}}
+{"category":"partially_accepted","evidence_run_id":"r3","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h3","recorded_at":"2026-04-30T00:00:00Z"}}
+{"category":"needs_human_review","evidence_run_id":"r4","provenance":{"source_file":"data/_kb/scrum_reviews.jsonl","sig_hash":"h4","recorded_at":"2026-04-30T00:00:00Z"}}
+`
+	if err := os.WriteFile(filepath.Join(dir, "run.jsonl"), []byte(jsonl), 0o644); err != nil {
+		t.Fatalf("write: %v", err)
+	}
+	// No evidence directory created → records that pass the firewall
+	// land in "not instructable" since synthesis can't proceed.
+	res, err := ExportSft(ExportSftOptions{
+		Root:       tmp,
+		RecordedAt: "2026-04-30T00:00:00Z",
+		DryRun:     true,
+	})
+	if err != nil {
+		t.Fatalf("ExportSft: %v", err)
+	}
+	if res.RecordsRead != 4 {
+		t.Errorf("RecordsRead: got %d, want 4", res.RecordsRead)
+	}
+	if res.RecordsQuarantined != 2 {
+		t.Errorf("RecordsQuarantined (firewall-blocked): got %d, want 2", res.RecordsQuarantined)
+	}
+	if res.RecordsExported != 0 {
+		t.Errorf("RecordsExported with no evidence: got %d, want 0", res.RecordsExported)
+	}
+	if !strings.Contains(res.QuarantineSummary, "not-instructable=2") {
+		t.Errorf("expected quarantine summary to flag 2 not-instructable, got %q", res.QuarantineSummary)
+	}
+}

internal/distillation/types.go

@@ -0,0 +1,498 @@
+// Package distillation is the Go port of the Rust v1.0.0 distillation
+// substrate (frozen at e7636f2). Per ADR-001 #4: port LOGIC, not
+// bit-identical reproducibility.
+//
+// What this package owns (this commit):
+//   - The deterministic scorer: EvidenceRecord → ScoredRun
+//   - Score categories + scorer version constant
+//   - SftSample type + validator with the contamination firewall
+//     (the safety-critical piece — rejected/needs_human_review must
+//     NEVER ship to SFT)
+//
+// What's deferred to follow-up commits:
+//   - Materialization layer (file iteration, jsonl read/write,
+//     date-partitioned storage) — operational tooling on top of
+//     the scorer logic
+//   - export_preference, export_rag (other export shapes)
+//   - acceptance harness (the gate that locks v1.0.0)
+//   - replay, receipts, evidence-index builders
+//
+// The scorer + SftSample validator are the LOAD-BEARING pieces
+// per project_distillation_substrate.md memory. The rest is plumbing
+// that can land incrementally without changing the logic the
+// downstream learning loop depends on.
+
+package distillation
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// ScoreCategory is one of the 4 deterministic verdicts. Matches Rust
+// SCORE_CATEGORIES exactly.
+type ScoreCategory string
+
+const (
+	CategoryAccepted          ScoreCategory = "accepted"
+	CategoryPartiallyAccepted ScoreCategory = "partially_accepted"
+	CategoryRejected          ScoreCategory = "rejected"
+	CategoryNeedsHumanReview  ScoreCategory = "needs_human_review"
+)
+
+// AllScoreCategories lists every legal category — used by validators.
+var AllScoreCategories = []ScoreCategory{
+	CategoryAccepted,
+	CategoryPartiallyAccepted,
+	CategoryRejected,
+	CategoryNeedsHumanReview,
+}
+
+// ScorerVersion is hardcoded — the deterministic-output contract
+// requires this. Bump the literal in the same commit as any scoring-
+// rule change so the version stamp moves atomically with logic.
+// Mirrors the Rust SCORER_VERSION (also v1.0.0 at e7636f2).
+const ScorerVersion = "v1.0.0"
+
+// SftQualityScore enumerates the categories LEGAL in SFT exports.
+// SFT_NEVER (defined below) is the inverse — categories that NEVER
+// ship to SFT under any flag combination. The contamination firewall
+// is enforced at the schema layer (ValidateSftSample) AND by the
+// exporter; defense in depth.
+type SftQualityScore string
+
+const (
+	SftQualityAccepted          SftQualityScore = "accepted"
+	SftQualityPartiallyAccepted SftQualityScore = "partially_accepted"
+)
+
+// SftQualityScores lists quality scores legal in SFT samples.
+// Default is SftQualityAccepted only; --include-partial CLI flag
+// expands to both. rejected and needs_human_review are NEVER legal.
+var SftQualityScores = []SftQualityScore{
+	SftQualityAccepted,
+	SftQualityPartiallyAccepted,
+}
+
+// SftNever is the contamination firewall: ScoreCategories that NEVER
+// ship to SFT under ANY caller flag. Enforced at the schema layer
+// (ValidateSftSample) AND at the exporter layer. Per the Rust
+// e7636f2 spec: "Hard non-negotiable: this set never expands. If you
+// find yourself adding 'needs_human_review' or 'rejected' here, stop
+// — that's the contamination the spec forbids."
+//
+// Exported so callers AND the validator share the same source of
+// truth. Modifying this constant changes the contract; reviewers
+// should treat any commit that touches it as a security review.
+var SftNever = []ScoreCategory{
+	CategoryRejected,
+	CategoryNeedsHumanReview,
+}
+
+// SftSampleSchemaVersion bumps when the on-wire SftSample shape
+// changes incompatibly. Match the Rust SFT_SAMPLE_SCHEMA_VERSION.
+const SftSampleSchemaVersion = 1
+
+// ScoredRunSchemaVersion bumps when the on-wire ScoredRun shape
+// changes incompatibly. Match the Rust SCORED_RUN_SCHEMA_VERSION.
+const ScoredRunSchemaVersion = 1
+
+// EvidenceSchemaVersion mirrors the Rust EVIDENCE_SCHEMA_VERSION.
+// This package consumes EvidenceRecord; producing it is a separate
+// concern (the materialization layer not yet ported).
+const EvidenceSchemaVersion = 1
+
+// ModelRole categorizes the kind of model output represented by an
+// EvidenceRecord. Used by the SFT exporter to filter "real model
+// output" from pure-extraction rows.
+type ModelRole string
+
+const (
+	RoleExecutor    ModelRole = "executor"
+	RoleReviewer    ModelRole = "reviewer"
+	RoleExtractor   ModelRole = "extractor"
+	RoleVerifier    ModelRole = "verifier"
+	RoleCategorizer ModelRole = "categorizer"
+	RoleTiebreaker  ModelRole = "tiebreaker"
+	RoleApplier     ModelRole = "applier"
+	RoleEmbedder    ModelRole = "embedder"
+	RoleOther       ModelRole = "other"
+)
+
+// Provenance is the source-linkage every distillation record carries.
+// SourceFile is required (no record without source linkage); other
+// fields are best-effort for de-duplication and trace-back.
+//
+// LineOffset is intentionally NOT `omitempty`: line offset 0 is the
+// first row of a source file — a real, semantically-meaningful value.
+// Stripping it on first-row records would silently diverge from the
+// Bun materializer (which always emits the field). Surfaced by the
+// 2026-05-02 cross-runtime parity probe.
+type Provenance struct {
+	SourceFile string `json:"source_file"`
+	LineOffset int64  `json:"line_offset"`
+	SigHash    string `json:"sig_hash"`
+	RecordedAt string `json:"recorded_at"` // ISO 8601
+}
+
+// ObserverVerdict is what an observer returned for an executor's
+// output. Matches the Rust enum but as a string type for JSON
+// flexibility.
+type ObserverVerdict string
+
+const (
+	VerdictAccept ObserverVerdict = "accept"
+	VerdictReject ObserverVerdict = "reject"
+	VerdictCycle  ObserverVerdict = "cycle"
+)
+
+// EvidenceRecord is one row in the canonical evidence stream.
+// Producing it (transforms from raw KB streams) is a separate
+// concern; this package consumes it.
+//
+// Fields mirror the Rust EvidenceRecord at e7636f2. Optional fields
+// use Go pointers / slices so missing-vs-empty stays distinguishable
+// for the scorer's heuristics.
+type EvidenceRecord struct {
+	RunID         string `json:"run_id"`
+	TaskID        string `json:"task_id"`
+	Timestamp     string `json:"timestamp"`
+	SchemaVersion int    `json:"schema_version"`
+
+	Provenance Provenance `json:"provenance"`
+
+	ModelName     string    `json:"model_name,omitempty"`
+	ModelProvider string    `json:"model_provider,omitempty"`
+	ModelRole     ModelRole `json:"model_role,omitempty"`
+
+	InputHash  string `json:"input_hash,omitempty"`
+	OutputHash string `json:"output_hash,omitempty"`
+
+	SourceFiles []string `json:"source_files,omitempty"`
+	CommandsRun []string `json:"commands_run,omitempty"`
+
+	RetrievedContext *RetrievedContext `json:"retrieved_context,omitempty"`
+
+	ObserverNotes      []string         `json:"observer_notes,omitempty"`
+	ObserverVerdict    ObserverVerdict  `json:"observer_verdict,omitempty"`
+	ObserverConfidence float64          `json:"observer_confidence,omitempty"`
+	ScratchpadSummary  string           `json:"scratchpad_summary,omitempty"`
+
+	SuccessMarkers []string `json:"success_markers,omitempty"`
+	FailureMarkers []string `json:"failure_markers,omitempty"`
+
+	ValidationResults map[string]any `json:"validation_results,omitempty"`
+
+	HumanOverride *HumanOverride `json:"human_override,omitempty"`
+
+	CostUSD          float64 `json:"cost_usd,omitempty"`
+	LatencyMs        int64   `json:"latency_ms,omitempty"`
+	PromptTokens     int64   `json:"prompt_tokens,omitempty"`
+	CompletionTokens int64   `json:"completion_tokens,omitempty"`
+	Text             string  `json:"text,omitempty"`
+
+	// Domain-specific bucket for source-row fields that don't earn a
+	// top-level slot. e.g. contract_analyses carries `contractor` here.
+	// Typed scalar values only — keep this small or it becomes a junk
+	// drawer. Mirrors EvidenceRecord.metadata in evidence_record.ts.
+	Metadata map[string]any `json:"metadata,omitempty"`
+}
+
+// RetrievedContext captures what the model saw via retrieval. Matches
+// the Rust shape exactly so the JSON round-trips byte-identical (per
+// ADR-001 #4 "logic, not bit-identical" — but on-wire compatibility
+// is desirable for tooling that consumes EvidenceRecord JSONL).
+type RetrievedContext struct {
+	MatrixCorpora             []string `json:"matrix_corpora,omitempty"`
+	MatrixHits                int      `json:"matrix_hits,omitempty"`
+	MatrixChunksKept          int      `json:"matrix_chunks_kept,omitempty"`
+	MatrixChunksDropped       int      `json:"matrix_chunks_dropped,omitempty"`
+	PathwayFingerprintsSeen   int      `json:"pathway_fingerprints_seen,omitempty"`
+}
+
+// HumanOverride captures a human-in-the-loop decision overriding the
+// scorer's verdict. Recorded but doesn't change the scorer's output;
+// downstream consumers (UI, distillation acceptance) decide how to
+// treat it.
+type HumanOverride struct {
+	Overrider     string `json:"overrider"`
+	Decision      string `json:"decision"` // accept|reject|needs_review
+	Reason        string `json:"reason"`
+	OverriddenAt  string `json:"overridden_at"`
+}
+
+// SubScores carries the deterministic scorer's intermediate signals
+// alongside the final ScoreCategory. Persisted on every ScoredRun
+// so a downstream UI can show "why" without re-running the scorer.
+type SubScores struct {
+	CargoGreen              *bool  `json:"cargo_green,omitempty"`
+	AnchorGrounding         *float64 `json:"anchor_grounding,omitempty"`
+	SchemaValid             *bool  `json:"schema_valid,omitempty"`
+	PathwayReplaySucceeded  *bool  `json:"pathway_replay_succeeded,omitempty"`
+	ObserverVerdict         ObserverVerdict `json:"observer_verdict,omitempty"`
+	AcceptedOnAttempt       *int   `json:"accepted_on_attempt,omitempty"`
+	// Extra fields the Rust schema accepted as `[key: string]: unknown`.
+	// Captured here as a free-form map so future signals don't require
+	// type-system changes.
+	Extras map[string]any `json:"-"`
+}
+
+// ScoredRun is the deterministic scorer's output. One per
+// EvidenceRecord. Provenance ties back to the materialized evidence
+// row (not the raw source stream).
+type ScoredRun struct {
+	SchemaVersion    int           `json:"schema_version"`
+	EvidenceRunID    string        `json:"evidence_run_id"`
+	EvidenceTaskID   string        `json:"evidence_task_id"`
+	Category         ScoreCategory `json:"category"`
+	Reasons          []string      `json:"reasons"` // non-empty
+	ScoredAt         string        `json:"scored_at"`
+	ScorerVersion    string        `json:"scorer_version"`
+	SubScores        *SubScores    `json:"sub_scores,omitempty"`
+	Provenance       Provenance    `json:"provenance"`
+}
+
+// SftSample is one entry in exports/sft/instruction_response.jsonl.
+// The contamination firewall lives in ValidateSftSample.
+type SftSample struct {
+	SchemaVersion int             `json:"schema_version"`
+	ID            string          `json:"id"`
+	Instruction   string          `json:"instruction"`
+	Context       string          `json:"context"` // empty allowed; null/missing not
+	Response      string          `json:"response"`
+	SourceRunID   string          `json:"source_run_id"`
+	QualityScore  SftQualityScore `json:"quality_score"`
+	CreatedAt     string          `json:"created_at"`
+	Provenance    Provenance      `json:"provenance"`
+}
+
+// ─── Validators ──────────────────────────────────────────────────
+
+// ValidationError is a single field-level violation.
+type ValidationError struct {
+	Field   string
+	Message string
+}
+
+func (e ValidationError) Error() string {
+	return fmt.Sprintf("%s: %s", e.Field, e.Message)
+}
+
+// ValidationErrors is the joinable error returned by the validators
+// when one or more fields violate the schema.
+type ValidationErrors []ValidationError
+
+func (es ValidationErrors) Error() string {
+	if len(es) == 0 {
+		return "no errors"
+	}
+	parts := make([]string, len(es))
+	for i, e := range es {
+		parts[i] = e.Error()
+	}
+	return strings.Join(parts, "; ")
+}
+
+// HasErrors returns true when one or more errors are present.
+func (es ValidationErrors) HasErrors() bool { return len(es) > 0 }
+
+// ValidateScoredRun mirrors the Rust validateScoredRun. Returns nil
+// on success or a ValidationErrors with the field-level violations.
+func ValidateScoredRun(r ScoredRun) error {
+	var errs ValidationErrors
+	if r.SchemaVersion != ScoredRunSchemaVersion {
+		errs = append(errs, ValidationError{
+			"schema_version",
+			fmt.Sprintf("expected %d, got %d", ScoredRunSchemaVersion, r.SchemaVersion),
+		})
+	}
+	if r.EvidenceRunID == "" {
+		errs = append(errs, ValidationError{"evidence_run_id", "must be non-empty"})
+	}
+	if r.EvidenceTaskID == "" {
+		errs = append(errs, ValidationError{"evidence_task_id", "must be non-empty"})
+	}
+	if !validISOTimestamp(r.ScoredAt) {
+		errs = append(errs, ValidationError{"scored_at", "must be ISO 8601 timestamp"})
+	}
+	if r.ScorerVersion == "" {
+		errs = append(errs, ValidationError{"scorer_version", "must be non-empty"})
+	}
+	if len(r.Reasons) == 0 {
+		errs = append(errs, ValidationError{"reasons", "must be non-empty (every score needs a reason)"})
+	}
+	if !isValidCategory(r.Category) {
+		errs = append(errs, ValidationError{"category", fmt.Sprintf("must be one of %v, got %q", AllScoreCategories, r.Category)})
+	}
+	if err := validateProvenance(r.Provenance, "provenance"); err != nil {
+		errs = append(errs, err...)
+	}
+	if r.SubScores != nil && r.SubScores.AnchorGrounding != nil {
+		ag := *r.SubScores.AnchorGrounding
+		if ag < 0 || ag > 1 {
+			errs = append(errs, ValidationError{"sub_scores.anchor_grounding", "must be in [0, 1]"})
+		}
+	}
+	if errs.HasErrors() {
+		return errs
+	}
+	return nil
+}
+
+// ValidateSftSample is the contamination firewall. Returns ErrSftContamination
+// (wrapped) when quality_score is in SftNever — which is the safety-critical
+// guarantee the spec calls non-negotiable.
+//
+// Other field violations come back as ValidationErrors.
+func ValidateSftSample(s SftSample) error {
+	var errs ValidationErrors
+	if s.SchemaVersion != SftSampleSchemaVersion {
+		errs = append(errs, ValidationError{
+			"schema_version",
+			fmt.Sprintf("expected %d, got %d", SftSampleSchemaVersion, s.SchemaVersion),
+		})
+	}
+	if s.ID == "" {
+		errs = append(errs, ValidationError{"id", "must be non-empty"})
+	}
+	if strings.TrimSpace(s.Instruction) == "" {
+		errs = append(errs, ValidationError{"instruction", "must be non-whitespace (no empty pairs)"})
+	}
+	if strings.TrimSpace(s.Response) == "" {
+		errs = append(errs, ValidationError{"response", "must be non-whitespace (no empty pairs)"})
+	}
+	// Context is required-string but empty is allowed.
+	// (Field is always typed as string in Go, so the only way to
+	// distinguish "set" from "missing" was via the JSON layer; here
+	// empty is fine.)
+	if s.SourceRunID == "" {
+		errs = append(errs, ValidationError{"source_run_id", "must be non-empty"})
+	}
+	if !validISOTimestamp(s.CreatedAt) {
+		errs = append(errs, ValidationError{"created_at", "must be ISO 8601 timestamp"})
+	}
+	if err := validateProvenance(s.Provenance, "provenance"); err != nil {
+		errs = append(errs, err...)
+	}
+
+	// Contamination firewall. Hard non-negotiable per the spec.
+	if !isLegalSftQualityScore(s.QualityScore) {
+		// If it's in SftNever, surface the firewall sentinel — callers
+		// can errors.Is(err, ErrSftContamination) to reliably detect
+		// "the spec said never" as opposed to "you typo'd a category."
+		if isContaminationCategory(s.QualityScore) {
+			return fmt.Errorf("%w: quality_score %q in SftNever (rejected/needs_human_review never legal in SFT)",
+				ErrSftContamination, s.QualityScore)
+		}
+		errs = append(errs, ValidationError{
+			"quality_score",
+			fmt.Sprintf("must be one of %v, got %q", SftQualityScores, s.QualityScore),
+		})
+	}
+
+	if errs.HasErrors() {
+		return errs
+	}
+	return nil
+}
+
+// ErrSftContamination is the firewall sentinel — when ValidateSftSample
+// rejects a sample because its quality_score is in SftNever, callers
+// can errors.Is(err, ErrSftContamination) to reliably distinguish
+// "spec violation" from "typo'd category."
+var ErrSftContamination = errors.New("distillation: SFT contamination — quality_score in SftNever")
+
+// ─── Internal helpers ────────────────────────────────────────────
+
+func isValidCategory(c ScoreCategory) bool {
+	for _, v := range AllScoreCategories {
+		if c == v {
+			return true
+		}
+	}
+	return false
+}
+
+func isLegalSftQualityScore(q SftQualityScore) bool {
+	for _, v := range SftQualityScores {
+		if q == v {
+			return true
+		}
+	}
+	return false
+}
+
+func isContaminationCategory(q SftQualityScore) bool {
+	// Compare as ScoreCategory — the on-wire string is the same; this
+	// just guards the firewall against typos that happen to match
+	// SftNever string-wise.
+	for _, v := range SftNever {
+		if string(v) == string(q) {
+			return true
+		}
+	}
+	return false
+}
+
+func validISOTimestamp(s string) bool {
+	if s == "" {
+		return false
+	}
+	// time.Parse with RFC3339 covers most ISO 8601. We accept both
+	// the basic and nano variants since the Rust producers vary.
+	if _, err := time.Parse(time.RFC3339, s); err == nil {
+		return true
+	}
+	if _, err := time.Parse(time.RFC3339Nano, s); err == nil {
+		return true
+	}
+	return false
+}
+
+func validateProvenance(p Provenance, field string) ValidationErrors {
+	var errs ValidationErrors
+	if p.SourceFile == "" {
+		errs = append(errs, ValidationError{field + ".source_file", "must be non-empty"})
+	}
+	if p.SigHash == "" {
+		errs = append(errs, ValidationError{field + ".sig_hash", "must be non-empty"})
+	}
+	if !validISOTimestamp(p.RecordedAt) {
+		errs = append(errs, ValidationError{field + ".recorded_at", "must be ISO 8601 timestamp"})
+	}
+	return errs
+}
+
+// MarshalSubScores is a shim — Go's encoding/json doesn't merge a
+// "rest" map into the struct's JSON output by default. Callers that
+// need Extras serialized into the same object can use this helper.
+func MarshalSubScores(s *SubScores) ([]byte, error) {
+	if s == nil {
+		return []byte("null"), nil
+	}
+	// First marshal the typed fields normally.
+	type alias SubScores
+	base, err := json.Marshal((*alias)(s))
+	if err != nil {
+		return nil, err
+	}
+	if len(s.Extras) == 0 {
+		return base, nil
+	}
+	// Decode back to a map, merge Extras, re-encode. Less efficient
+	// but keeps the field semantics correct (typed fields override
+	// extras on collision — first-write-wins for known keys).
+	var combined map[string]any
+	if err := json.Unmarshal(base, &combined); err != nil {
+		return nil, err
+	}
+	for k, v := range s.Extras {
+		if _, exists := combined[k]; !exists {
+			combined[k] = v
+		}
+	}
+	return json.Marshal(combined)
+}

internal/drift/drift.go

@@ -0,0 +1,359 @@
+// Package drift quantifies when historical decisions stop matching
+// current reality. Per the PRD's 5-loop substrate, this is loop 5
+// (drift) — distinct from the rating+distillation loop because
+// drift is about MEASUREMENT, not learning. The learning loop says
+// "this match worked, remember it"; the drift loop says "the
+// playbook entry from 4 months ago — does it still match what the
+// substrate would surface today?"
+//
+// First-shipped drift shape: SCORER drift. When the deterministic
+// scorer's logic changes (ScorerVersion bumped), historical
+// ScoredRuns may no longer match what the current scorer would
+// produce on the same EvidenceRecord. ComputeScorerDrift re-runs
+// the current scorer over a slice of (EvidenceRecord, persisted
+// category) pairs and reports mismatches.
+//
+// Why this matters: the rating+distillation loop only learns
+// forward. Without a drift quantifier, a scorer-rule change
+// silently invalidates the historical training data feeding the
+// loop. With drift quantification, a rule change surfaces a
+// concrete number ("847 of 4701 historical scoredruns now
+// disagree") that triggers a re-score-and-retrain cycle rather
+// than letting the substrate quietly rot.
+//
+// Future drift shapes (not in this commit):
+//   - PLAYBOOK drift: for each playbook entry, re-run its query
+//     through current matrix-search; if the recorded answer is no
+//     longer in top-K, the world has moved.
+//   - EMBEDDING drift: KS-test on the distribution of embedding
+//     vectors at T1 vs T2; large shifts = the corpus has changed
+//     materially.
+//   - AUDIT BASELINE drift: track how PR audit verdicts shift over
+//     scorer/auditor versions; matches the Rust audit_baselines.jsonl
+//     longitudinal signal.
+
+package drift
+
+import (
+	"math"
+	"sort"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+// ScorerDriftEntry is one mismatch — a historical (record, category)
+// pair where the current scorer disagrees with the persisted
+// verdict. Reasons captures the current scorer's explanation so
+// operators can see WHY the verdict changed.
+type ScorerDriftEntry struct {
+	EvidenceRunID    string                       `json:"evidence_run_id"`
+	EvidenceTaskID   string                       `json:"evidence_task_id"`
+	PersistedCategory distillation.ScoreCategory  `json:"persisted_category"`
+	CurrentCategory   distillation.ScoreCategory  `json:"current_category"`
+	CurrentReasons    []string                    `json:"current_reasons"`
+	SourceFile        string                      `json:"source_file"`
+}
+
+// CategoryShift is one cell in the drift matrix — "X persisted
+// records that NOW classify as Y." e.g. "12 records that were
+// 'rejected' yesterday are 'partially_accepted' today."
+type CategoryShift struct {
+	From  distillation.ScoreCategory `json:"from"`
+	To    distillation.ScoreCategory `json:"to"`
+	Count int                        `json:"count"`
+}
+
+// ScorerDriftReport is the summary returned by ComputeScorerDrift.
+// The shape is intentionally machine-readable so a downstream
+// dashboard / alerting layer can threshold on Drifted / TotalChecked
+// without parsing the entries list.
+type ScorerDriftReport struct {
+	ScorerVersion string             `json:"scorer_version"` // current scorer's version
+	TotalChecked  int                `json:"total_checked"`
+	Matched       int                `json:"matched"`        // current == persisted
+	Drifted       int                `json:"drifted"`        // current != persisted
+	DriftRate     float64            `json:"drift_rate"`     // Drifted / TotalChecked
+	ShiftMatrix   []CategoryShift    `json:"shift_matrix,omitempty"`
+	Entries       []ScorerDriftEntry `json:"entries,omitempty"` // mismatches only
+}
+
+// ScorerDriftInput is one (record, persisted_category) pair to check.
+// Caller is responsible for materializing these from disk; this
+// package is pure compute.
+type ScorerDriftInput struct {
+	Record            distillation.EvidenceRecord
+	PersistedCategory distillation.ScoreCategory
+}
+
+// ComputeScorerDrift re-runs distillation.ScoreRecord over each
+// input and reports mismatches. Pure function — no I/O. The caller
+// supplies the inputs (typically by reading a directory of
+// scored-runs JSONL alongside the corresponding evidence JSONL).
+//
+// IncludeEntries controls whether the per-mismatch detail list is
+// populated. For large corpora (e.g. 4,701 fill events) the
+// summary numbers may be all the caller needs; setting this to
+// false avoids allocating the entries slice.
+func ComputeScorerDrift(inputs []ScorerDriftInput, includeEntries bool) ScorerDriftReport {
+	report := ScorerDriftReport{
+		ScorerVersion: distillation.ScorerVersion,
+		TotalChecked:  len(inputs),
+	}
+
+	shiftCounts := make(map[[2]distillation.ScoreCategory]int)
+
+	for _, in := range inputs {
+		out := distillation.ScoreRecord(in.Record)
+		if out.Category == in.PersistedCategory {
+			report.Matched++
+			continue
+		}
+		report.Drifted++
+		shiftCounts[[2]distillation.ScoreCategory{in.PersistedCategory, out.Category}]++
+		if includeEntries {
+			report.Entries = append(report.Entries, ScorerDriftEntry{
+				EvidenceRunID:     in.Record.RunID,
+				EvidenceTaskID:    in.Record.TaskID,
+				PersistedCategory: in.PersistedCategory,
+				CurrentCategory:   out.Category,
+				CurrentReasons:    out.Reasons,
+				SourceFile:        in.Record.Provenance.SourceFile,
+			})
+		}
+	}
+
+	if report.TotalChecked > 0 {
+		report.DriftRate = float64(report.Drifted) / float64(report.TotalChecked)
+	}
+
+	if len(shiftCounts) > 0 {
+		report.ShiftMatrix = make([]CategoryShift, 0, len(shiftCounts))
+		for k, v := range shiftCounts {
+			report.ShiftMatrix = append(report.ShiftMatrix, CategoryShift{
+				From: k[0], To: k[1], Count: v,
+			})
+		}
+		// Sort: largest shifts first, then alphabetical-ish for ties.
+		// Stable ordering matters for downstream display and JSON
+		// determinism in tests.
+		sort.Slice(report.ShiftMatrix, func(i, j int) bool {
+			a, b := report.ShiftMatrix[i], report.ShiftMatrix[j]
+			if a.Count != b.Count {
+				return a.Count > b.Count
+			}
+			if a.From != b.From {
+				return string(a.From) < string(b.From)
+			}
+			return string(a.To) < string(b.To)
+		})
+	}
+
+	return report
+}
+
+// ─────────────────────────────────────────────────────────────────
+// Distribution drift via Population Stability Index (PSI).
+//
+// Closes OPEN #3's "concrete distribution-drift signal" — pairs with
+// ComputeScorerDrift's "categorical drift" to give callers a
+// continuous-value drift metric for things like cosine distances,
+// judge ratings, query top-1 distance distributions over time.
+//
+// Why PSI: standard in finance/risk for distribution-shift monitoring,
+// well-defined verdict tiers (< 0.1 stable, 0.1-0.25 minor, > 0.25
+// major), tolerant of moderate sample sizes, no assumption about
+// distribution shape. Alternatives (KS-test, KL-divergence) all
+// have failure modes when one bucket has zero observations; PSI's
+// epsilon-clamping handles that gracefully.
+//
+// Math:  PSI = Σᵢ (actualᵢ - expectedᵢ) × ln(actualᵢ / expectedᵢ)
+// where actualᵢ and expectedᵢ are the proportion of observations
+// falling into bucket i (zero values are clamped to a small
+// epsilon so the log doesn't blow up).
+//
+// First-shipped use case: comparing the distribution of cosine top-1
+// distances at T0 vs T1 to detect when the embedder's behavior
+// against the corpus has shifted materially (e.g. after a model
+// upgrade or a corpus refresh).
+// ─────────────────────────────────────────────────────────────────
+
+// DistributionDriftTier is a verdict bucket for the PSI value.
+// Standard PSI thresholds from finance/risk; documented in the
+// function-level comment for ComputeDistributionDrift.
+type DistributionDriftTier string
+
+const (
+	DriftTierStable DistributionDriftTier = "stable" // PSI < 0.1
+	DriftTierMinor  DistributionDriftTier = "minor"  // 0.1 ≤ PSI < 0.25
+	DriftTierMajor  DistributionDriftTier = "major"  // PSI ≥ 0.25
+)
+
+// DistributionDriftBucket is one bucket's contribution to the PSI sum.
+// Useful for drilldown — operators can see WHICH part of the
+// distribution shifted (e.g. "the [0.3, 0.4) bucket gained 12% of
+// observations from T0 to T1").
+type DistributionDriftBucket struct {
+	Lower         float64 `json:"lower"`
+	Upper         float64 `json:"upper"`
+	BaselineCount int     `json:"baseline_count"`
+	CurrentCount  int     `json:"current_count"`
+	BaselineRatio float64 `json:"baseline_ratio"` // [0, 1]
+	CurrentRatio  float64 `json:"current_ratio"`  // [0, 1]
+	PSIPart       float64 `json:"psi_part"`       // contribution to PSI
+}
+
+// DistributionDriftReport is the aggregate output of a PSI run.
+type DistributionDriftReport struct {
+	PSI            float64                   `json:"psi"`
+	Tier           DistributionDriftTier     `json:"tier"`
+	Buckets        []DistributionDriftBucket `json:"buckets"`
+	BaselineN      int                       `json:"baseline_n"`
+	CurrentN       int                       `json:"current_n"`
+	NumBuckets     int                       `json:"num_buckets"`
+	Min            float64                   `json:"min"`
+	Max            float64                   `json:"max"`
+}
+
+// DistributionDriftInput is the input shape for ComputeDistributionDrift.
+// Baseline is the reference distribution (T0); Current is the new
+// distribution (T1). NumBuckets defaults to 10 when zero, capped at
+// 100 to keep the per-bucket math stable on small samples.
+type DistributionDriftInput struct {
+	Baseline   []float64
+	Current    []float64
+	NumBuckets int
+}
+
+// driftEpsilon clamps zero-bucket ratios so log doesn't blow up.
+// 1e-4 is the standard PSI choice — represents "less than 0.01% of
+// observations." Anything smaller would let one rare-bucket shift
+// dominate the PSI sum unfairly.
+const driftEpsilon = 1e-4
+
+// ComputeDistributionDrift returns a PSI-based drift report between
+// the baseline and current distributions. Buckets span [min, max] of
+// the COMBINED data (so neither side falls outside) with
+// equal-frequency buckets on the baseline (so each baseline bucket
+// has roughly the same count — robust to skewed distributions like
+// cosine distances which cluster near 0).
+//
+// Empty inputs return PSI=0, tier=stable. Caller should also check
+// BaselineN/CurrentN before trusting the verdict — PSI on tiny
+// samples is statistical noise.
+//
+// Thresholds (industry-standard, citable):
+//   - PSI < 0.10  → stable: distributions are operationally equivalent
+//   - 0.10 ≤ PSI < 0.25 → minor: investigate but not alarming
+//   - PSI ≥ 0.25 → major: distributions have shifted materially;
+//     downstream decisions trained on baseline may be invalid
+func ComputeDistributionDrift(in DistributionDriftInput) DistributionDriftReport {
+	report := DistributionDriftReport{
+		BaselineN: len(in.Baseline),
+		CurrentN:  len(in.Current),
+	}
+	if report.BaselineN == 0 || report.CurrentN == 0 {
+		report.Tier = DriftTierStable
+		return report
+	}
+	n := in.NumBuckets
+	if n <= 0 {
+		n = 10
+	}
+	if n > 100 {
+		n = 100
+	}
+	report.NumBuckets = n
+
+	// Combined min/max so both distributions fit the bucket range.
+	minV, maxV := in.Baseline[0], in.Baseline[0]
+	for _, v := range in.Baseline {
+		if v < minV {
+			minV = v
+		}
+		if v > maxV {
+			maxV = v
+		}
+	}
+	for _, v := range in.Current {
+		if v < minV {
+			minV = v
+		}
+		if v > maxV {
+			maxV = v
+		}
+	}
+	report.Min = minV
+	report.Max = maxV
+	if maxV == minV {
+		// All values identical — trivially stable, no meaningful PSI.
+		report.Tier = DriftTierStable
+		return report
+	}
+
+	// Equal-WIDTH bucketing. Equal-frequency would be more robust on
+	// skewed distributions, but it complicates the math (need to
+	// quantile-sort the baseline) for marginal gain on the symmetric
+	// distributions PSI is most useful for. Operators with skewed
+	// data can pre-bucket their inputs into normalized scores.
+	width := (maxV - minV) / float64(n)
+	bucketIdx := func(v float64) int {
+		if v >= maxV {
+			return n - 1 // right-edge inclusive
+		}
+		idx := int((v - minV) / width)
+		if idx < 0 {
+			idx = 0
+		}
+		if idx >= n {
+			idx = n - 1
+		}
+		return idx
+	}
+
+	bCounts := make([]int, n)
+	cCounts := make([]int, n)
+	for _, v := range in.Baseline {
+		bCounts[bucketIdx(v)]++
+	}
+	for _, v := range in.Current {
+		cCounts[bucketIdx(v)]++
+	}
+
+	report.Buckets = make([]DistributionDriftBucket, 0, n)
+	bN := float64(report.BaselineN)
+	cN := float64(report.CurrentN)
+	psi := 0.0
+	for i := 0; i < n; i++ {
+		bRatio := float64(bCounts[i]) / bN
+		cRatio := float64(cCounts[i]) / cN
+		// Epsilon clamp so empty buckets don't blow up log.
+		if bRatio < driftEpsilon {
+			bRatio = driftEpsilon
+		}
+		if cRatio < driftEpsilon {
+			cRatio = driftEpsilon
+		}
+		part := (cRatio - bRatio) * math.Log(cRatio/bRatio)
+		psi += part
+		report.Buckets = append(report.Buckets, DistributionDriftBucket{
+			Lower:         minV + float64(i)*width,
+			Upper:         minV + float64(i+1)*width,
+			BaselineCount: bCounts[i],
+			CurrentCount:  cCounts[i],
+			BaselineRatio: float64(bCounts[i]) / bN,
+			CurrentRatio:  float64(cCounts[i]) / cN,
+			PSIPart:       part,
+		})
+	}
+	report.PSI = psi
+	switch {
+	case psi < 0.10:
+		report.Tier = DriftTierStable
+	case psi < 0.25:
+		report.Tier = DriftTierMinor
+	default:
+		report.Tier = DriftTierMajor
+	}
+	return report
+}
+

internal/drift/drift_test.go

@@ -0,0 +1,289 @@
+package drift
+
+import (
+	"testing"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+func mkInput(sourceFile string, persisted distillation.ScoreCategory, succ []string) ScorerDriftInput {
+	return ScorerDriftInput{
+		Record: distillation.EvidenceRecord{
+			RunID:         "run-x",
+			TaskID:        "task-x",
+			Timestamp:     "2026-01-01T00:00:00Z",
+			SchemaVersion: distillation.EvidenceSchemaVersion,
+			Provenance: distillation.Provenance{
+				SourceFile: sourceFile,
+				SigHash:    "abc",
+				RecordedAt: "2026-01-01T00:00:01Z",
+			},
+			SuccessMarkers: succ,
+		},
+		PersistedCategory: persisted,
+	}
+}
+
+func TestComputeScorerDrift_NoDrift(t *testing.T) {
+	// All inputs have persisted=accepted matching what the current
+	// scrum_review scorer produces on accepted_on_attempt_1.
+	inputs := []ScorerDriftInput{
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_1"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_1"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_1"}),
+	}
+	r := ComputeScorerDrift(inputs, true)
+	if r.TotalChecked != 3 || r.Matched != 3 || r.Drifted != 0 {
+		t.Errorf("no-drift case: total=%d matched=%d drifted=%d",
+			r.TotalChecked, r.Matched, r.Drifted)
+	}
+	if r.DriftRate != 0 {
+		t.Errorf("drift_rate: want 0, got %v", r.DriftRate)
+	}
+	if len(r.Entries) != 0 {
+		t.Errorf("entries: want 0, got %d", len(r.Entries))
+	}
+}
+
+func TestComputeScorerDrift_ShiftDetected(t *testing.T) {
+	// Simulate a historical labeling where the persisted scorer
+	// thought attempt-2 acceptances were "accepted" but the current
+	// scorer (this code) categorizes them as "partially_accepted".
+	// Drift should fire on those.
+	inputs := []ScorerDriftInput{
+		// Match: attempt 1 → accepted (still)
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_1"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_1"}),
+		// Drift: persisted thought attempt-2 was accepted, today's scorer says partial
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_2"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_3"}),
+		// Drift: persisted thought attempt-5 was accepted, today's scorer says partial (high-cost)
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_5"}),
+	}
+	r := ComputeScorerDrift(inputs, true)
+	if r.TotalChecked != 5 {
+		t.Errorf("total: want 5, got %d", r.TotalChecked)
+	}
+	if r.Matched != 2 {
+		t.Errorf("matched: want 2, got %d", r.Matched)
+	}
+	if r.Drifted != 3 {
+		t.Errorf("drifted: want 3, got %d", r.Drifted)
+	}
+	wantRate := 3.0 / 5.0
+	if r.DriftRate < wantRate-1e-9 || r.DriftRate > wantRate+1e-9 {
+		t.Errorf("drift_rate: want %v, got %v", wantRate, r.DriftRate)
+	}
+	if len(r.Entries) != 3 {
+		t.Errorf("entries: want 3 mismatches, got %d", len(r.Entries))
+	}
+	// Shift matrix should show one shift: accepted → partially_accepted, count=3
+	if len(r.ShiftMatrix) != 1 {
+		t.Errorf("shift matrix: want 1 shift, got %d (%+v)", len(r.ShiftMatrix), r.ShiftMatrix)
+	} else {
+		s := r.ShiftMatrix[0]
+		if s.From != distillation.CategoryAccepted ||
+			s.To != distillation.CategoryPartiallyAccepted ||
+			s.Count != 3 {
+			t.Errorf("shift: got %+v", s)
+		}
+	}
+}
+
+func TestComputeScorerDrift_MultipleShiftsSortedByCount(t *testing.T) {
+	inputs := []ScorerDriftInput{
+		// 3× accepted→partial
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_2"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_2"}),
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_2"}),
+		// 1× rejected→needs_human (no marker)
+		{
+			Record: distillation.EvidenceRecord{
+				RunID: "r1", TaskID: "t1",
+				Timestamp:     "2026-01-01T00:00:00Z",
+				SchemaVersion: distillation.EvidenceSchemaVersion,
+				Provenance: distillation.Provenance{
+					SourceFile: "data/_kb/scrum_reviews.jsonl",
+					SigHash:    "x", RecordedAt: "2026-01-01T00:00:01Z",
+				},
+				// no markers → needs_human_review
+			},
+			PersistedCategory: distillation.CategoryRejected,
+		},
+	}
+	r := ComputeScorerDrift(inputs, false)
+	if r.Drifted != 4 {
+		t.Errorf("drifted: want 4, got %d", r.Drifted)
+	}
+	if len(r.ShiftMatrix) != 2 {
+		t.Errorf("shift matrix: want 2 distinct shifts, got %d", len(r.ShiftMatrix))
+	}
+	// Sorted by count desc, so accepted→partial (3) before rejected→needs_human (1)
+	if r.ShiftMatrix[0].Count != 3 || r.ShiftMatrix[1].Count != 1 {
+		t.Errorf("shift order wrong: got %+v", r.ShiftMatrix)
+	}
+}
+
+func TestComputeScorerDrift_IncludeEntriesFalse(t *testing.T) {
+	inputs := []ScorerDriftInput{
+		mkInput("data/_kb/scrum_reviews.jsonl", distillation.CategoryAccepted, []string{"accepted_on_attempt_2"}),
+	}
+	r := ComputeScorerDrift(inputs, false)
+	if r.Drifted != 1 {
+		t.Errorf("drifted: want 1, got %d", r.Drifted)
+	}
+	if len(r.Entries) != 0 {
+		t.Errorf("entries: want 0 when includeEntries=false, got %d", len(r.Entries))
+	}
+}
+
+func TestComputeScorerDrift_EmptyInput(t *testing.T) {
+	r := ComputeScorerDrift(nil, true)
+	if r.TotalChecked != 0 || r.Drifted != 0 || r.Matched != 0 {
+		t.Errorf("empty: want all-zero, got %+v", r)
+	}
+	if r.DriftRate != 0 {
+		t.Errorf("drift_rate on empty: want 0, got %v", r.DriftRate)
+	}
+}
+
+func TestComputeScorerDrift_ScorerVersionStamped(t *testing.T) {
+	r := ComputeScorerDrift(nil, false)
+	if r.ScorerVersion != distillation.ScorerVersion {
+		t.Errorf("scorer_version: want %q, got %q", distillation.ScorerVersion, r.ScorerVersion)
+	}
+}
+
+// ── Distribution drift (PSI) tests ────────────────────────────────
+
+// TestDistributionDrift_IdenticalIsStable: same data on both sides
+// should yield PSI ≈ 0 and tier=stable. Anchors the lower bound.
+func TestDistributionDrift_IdenticalIsStable(t *testing.T) {
+	data := []float64{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline:   data,
+		Current:    data,
+		NumBuckets: 5,
+	})
+	if r.PSI > 0.001 {
+		t.Errorf("identical distributions: expected PSI ≈ 0, got %f", r.PSI)
+	}
+	if r.Tier != DriftTierStable {
+		t.Errorf("expected stable tier, got %q", r.Tier)
+	}
+}
+
+// TestDistributionDrift_HardShiftIsMajor: distribution moved
+// completely to a different range — should yield major tier.
+func TestDistributionDrift_HardShiftIsMajor(t *testing.T) {
+	baseline := []float64{0.1, 0.1, 0.2, 0.2, 0.3, 0.3, 0.4, 0.4}
+	current := []float64{0.7, 0.7, 0.8, 0.8, 0.9, 0.9, 1.0, 1.0}
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline:   baseline,
+		Current:    current,
+		NumBuckets: 10,
+	})
+	if r.PSI < 0.25 {
+		t.Errorf("hard distribution shift: expected PSI ≥ 0.25, got %f", r.PSI)
+	}
+	if r.Tier != DriftTierMajor {
+		t.Errorf("expected major tier, got %q", r.Tier)
+	}
+}
+
+// TestDistributionDrift_DetectsModerateShift: distribution shifted
+// noticeably but not catastrophically — PSI must be > 0 (some drift
+// detected) and tier must NOT be stable. Whether the tier is minor
+// vs major depends on bucketing granularity; we don't pin that here
+// because PSI thresholds are sensitive to bucket count.
+func TestDistributionDrift_DetectsModerateShift(t *testing.T) {
+	// Baseline: many around 0.5, some spread.
+	baseline := []float64{0.4, 0.45, 0.5, 0.5, 0.5, 0.5, 0.5, 0.55, 0.6, 0.6,
+		0.45, 0.5, 0.5, 0.55, 0.5, 0.5, 0.5, 0.55, 0.5, 0.6}
+	// Current: same range, slight rightward shift (still overlapping).
+	current := []float64{0.45, 0.5, 0.5, 0.55, 0.55, 0.55, 0.6, 0.6, 0.65, 0.7,
+		0.5, 0.55, 0.55, 0.55, 0.6, 0.6, 0.55, 0.6, 0.65, 0.65}
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline:   baseline,
+		Current:    current,
+		NumBuckets: 10,
+	})
+	if r.PSI < 0.01 {
+		t.Errorf("moderate shift should produce PSI > 0.01, got %f", r.PSI)
+	}
+	if r.Tier == DriftTierStable {
+		t.Errorf("moderate shift should NOT be stable tier, got PSI=%f tier=%q", r.PSI, r.Tier)
+	}
+}
+
+// TestDistributionDrift_EmptyInputs: empty baseline OR current
+// returns PSI=0, stable tier — caller must check N before trusting.
+func TestDistributionDrift_EmptyInputs(t *testing.T) {
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline: []float64{},
+		Current:  []float64{1, 2, 3},
+	})
+	if r.PSI != 0 || r.Tier != DriftTierStable {
+		t.Errorf("empty baseline: expected PSI=0 stable, got psi=%f tier=%q", r.PSI, r.Tier)
+	}
+	r = ComputeDistributionDrift(DistributionDriftInput{
+		Baseline: []float64{1, 2, 3},
+		Current:  []float64{},
+	})
+	if r.PSI != 0 || r.Tier != DriftTierStable {
+		t.Errorf("empty current: expected PSI=0 stable, got psi=%f tier=%q", r.PSI, r.Tier)
+	}
+}
+
+// TestDistributionDrift_AllIdenticalValues: degenerate case where
+// everything's the same value (e.g., all zeros). Should not panic;
+// returns stable.
+func TestDistributionDrift_AllIdenticalValues(t *testing.T) {
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline: []float64{0.5, 0.5, 0.5},
+		Current:  []float64{0.5, 0.5, 0.5},
+	})
+	if r.Tier != DriftTierStable {
+		t.Errorf("expected stable on identical-singleton, got %q", r.Tier)
+	}
+}
+
+// TestDistributionDrift_BucketCounts: per-bucket counts must sum to
+// the input N. If they don't, we're losing observations to bucket
+// boundary issues.
+func TestDistributionDrift_BucketCounts(t *testing.T) {
+	baseline := []float64{0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}
+	current := []float64{0.5, 0.5, 0.5, 0.5}
+	r := ComputeDistributionDrift(DistributionDriftInput{
+		Baseline:   baseline,
+		Current:    current,
+		NumBuckets: 5,
+	})
+	totalB := 0
+	totalC := 0
+	for _, b := range r.Buckets {
+		totalB += b.BaselineCount
+		totalC += b.CurrentCount
+	}
+	if totalB != len(baseline) {
+		t.Errorf("baseline bucket counts sum to %d, expected %d", totalB, len(baseline))
+	}
+	if totalC != len(current) {
+		t.Errorf("current bucket counts sum to %d, expected %d", totalC, len(current))
+	}
+}
+
+// TestDistributionDrift_NumBucketsClamping: 0 → default 10; > 100 → 100.
+func TestDistributionDrift_NumBucketsClamping(t *testing.T) {
+	in := DistributionDriftInput{Baseline: []float64{1, 2}, Current: []float64{1, 2}}
+	in.NumBuckets = 0
+	r := ComputeDistributionDrift(in)
+	if r.NumBuckets != 10 {
+		t.Errorf("0 should default to 10 buckets, got %d", r.NumBuckets)
+	}
+	in.NumBuckets = 500
+	r = ComputeDistributionDrift(in)
+	if r.NumBuckets != 100 {
+		t.Errorf("500 should clamp to 100 buckets, got %d", r.NumBuckets)
+	}
+}

internal/embed/cached.go

@@ -0,0 +1,180 @@
+// cached.go — LRU caching wrapper around a Provider.
+//
+// Memoizes (effective_model, sha256(text)) → []float32. Repeat
+// queries return the stored vector without round-tripping to the
+// upstream embedding service. Per-text caching: a batch with
+// mixed hit/miss only fetches the misses, then merges the result
+// preserving the caller's input order.
+//
+// Memory budget: ~3 KB per entry at d=768. 10K-entry default ≈ 30 MB
+// — small enough for any realistic embedd deployment. Operators
+// raising the cap should weigh memory headroom against expected
+// hit rate.
+//
+// Why this exists: 500K staffing test (memory project_golang_lakehouse)
+// showed that the staffing co-pilot replays many of the same query
+// texts ("forklift driver", "welder Chicago", etc.). Caching them
+// drops repeat-query cost from ~50ms (Ollama round-trip) to <1µs
+// (LRU hit). Real production win documented in feedback_meta_index_vision.
+
+package embed
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"sync/atomic"
+
+	lru "github.com/hashicorp/golang-lru/v2"
+)
+
+// CachedProvider wraps a Provider with an LRU cache keyed on
+// (effective_model, sha256(text)). Thread-safe.
+type CachedProvider struct {
+	inner        Provider
+	defaultModel string
+	cache        *lru.Cache[string, []float32]
+	hits         atomic.Int64
+	misses       atomic.Int64
+}
+
+// NewCachedProvider wraps inner with an LRU cache of the given size.
+//
+// defaultModel is used to resolve cache keys when a request leaves
+// the model field empty: a request for model="" is treated as
+// model=defaultModel for cache-key purposes, so callers that mix
+// "" and the explicit model name still hit the same cache entry.
+//
+// Caller-side panic protection: size <= 0 is treated as "no cache"
+// — every Embed call passes through. Avoids forcing operators to
+// understand LRU sizing to disable.
+func NewCachedProvider(inner Provider, defaultModel string, size int) (*CachedProvider, error) {
+	if size <= 0 {
+		// Sentinel: nil cache means pass-through. NewCachedProvider
+		// stays callable so the wiring layer can always wrap.
+		return &CachedProvider{inner: inner, defaultModel: defaultModel}, nil
+	}
+	cache, err := lru.New[string, []float32](size)
+	if err != nil {
+		return nil, fmt.Errorf("embed cache init: %w", err)
+	}
+	return &CachedProvider{
+		inner:        inner,
+		defaultModel: defaultModel,
+		cache:        cache,
+	}, nil
+}
+
+// Embed returns vectors for texts, memoizing per (model, text). On
+// a batch with mixed hits/misses, only the misses round-trip to
+// inner; the result preserves caller ordering.
+func (c *CachedProvider) Embed(ctx context.Context, texts []string, model string) (Result, error) {
+	// Pass-through when caching disabled.
+	if c.cache == nil {
+		return c.inner.Embed(ctx, texts, model)
+	}
+
+	if len(texts) == 0 {
+		return Result{}, ErrEmptyTexts
+	}
+
+	effectiveModel := model
+	if effectiveModel == "" {
+		effectiveModel = c.defaultModel
+	}
+
+	// Pass 1: cache lookup; collect misses preserving original index
+	// so we can write the upstream result back into the right slots.
+	out := make([][]float32, len(texts))
+	missTexts := make([]string, 0, len(texts))
+	missIdx := make([]int, 0, len(texts))
+	for i, t := range texts {
+		key := cacheKey(effectiveModel, t)
+		if v, ok := c.cache.Get(key); ok {
+			out[i] = v
+			c.hits.Add(1)
+			continue
+		}
+		missTexts = append(missTexts, t)
+		missIdx = append(missIdx, i)
+		c.misses.Add(1)
+	}
+
+	// All hits — synthesize the result without an upstream call.
+	// Use the effective model + the first cached vector's length
+	// for the response. Every cached vector for the same model has
+	// the same dimension by construction (Provider guarantees it).
+	if len(missTexts) == 0 {
+		dim := 0
+		if len(out) > 0 && len(out[0]) > 0 {
+			dim = len(out[0])
+		}
+		return Result{
+			Model:     effectiveModel,
+			Dimension: dim,
+			Vectors:   out,
+		}, nil
+	}
+
+	// Pass 2: fetch the misses, populate cache, merge.
+	res, err := c.inner.Embed(ctx, missTexts, model)
+	if err != nil {
+		return Result{}, err
+	}
+	if len(res.Vectors) != len(missTexts) {
+		return Result{}, fmt.Errorf("embed cache: provider returned %d vectors for %d miss texts",
+			len(res.Vectors), len(missTexts))
+	}
+	// Cache under effectiveModel (request-derived), NOT res.Model
+	// (upstream-derived). Future requests with the same input shape
+	// — same explicit model OR the same "" — must hit the same key.
+	// Only ours is predictable from the request; upstream's resolution
+	// can drift if the upstream's default changes.
+	for j, t := range missTexts {
+		out[missIdx[j]] = res.Vectors[j]
+		c.cache.Add(cacheKey(effectiveModel, t), res.Vectors[j])
+	}
+
+	return Result{
+		Model:     res.Model,
+		Dimension: res.Dimension,
+		Vectors:   out,
+	}, nil
+}
+
+// Stats returns lifetime hit + miss counts for this cache. Atomic
+// reads — no locking. Useful for /health, /metrics, or operator
+// dashboards.
+func (c *CachedProvider) Stats() (hits, misses int64) {
+	return c.hits.Load(), c.misses.Load()
+}
+
+// HitRate returns the fraction of requests served from cache.
+// Returns 0.0 when no requests have been served (avoids NaN).
+func (c *CachedProvider) HitRate() float64 {
+	h := c.hits.Load()
+	m := c.misses.Load()
+	total := h + m
+	if total == 0 {
+		return 0.0
+	}
+	return float64(h) / float64(total)
+}
+
+// Len returns the current entry count in the cache. Returns 0 when
+// caching is disabled.
+func (c *CachedProvider) Len() int {
+	if c.cache == nil {
+		return 0
+	}
+	return c.cache.Len()
+}
+
+// cacheKey is "<model>:<sha256(text)>". sha256 collapses long texts
+// to a fixed-size key; model prefix scopes cache to one model so
+// callers using multiple models don't get cross-contamination.
+func cacheKey(model, text string) string {
+	h := sha256.Sum256([]byte(text))
+	return model + ":" + hex.EncodeToString(h[:])
+}

internal/embed/cached_test.go

@@ -0,0 +1,350 @@
+package embed
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+)
+
+// stubProvider records every call so tests can verify caching
+// behavior (call counts, request texts) without spinning up Ollama.
+type stubProvider struct {
+	mu        sync.Mutex
+	model     string
+	dim       int
+	callCount int
+	lastTexts []string
+	err       error
+	// vectorFn returns a deterministic vector for a given text so
+	// cache hits assert byte-equality, not just shape.
+	vectorFn func(text string) []float32
+}
+
+func (s *stubProvider) Embed(_ context.Context, texts []string, model string) (Result, error) {
+	s.mu.Lock()
+	s.callCount++
+	s.lastTexts = append([]string(nil), texts...)
+	s.mu.Unlock()
+	if s.err != nil {
+		return Result{}, s.err
+	}
+	out := make([][]float32, len(texts))
+	for i, t := range texts {
+		out[i] = s.vectorFn(t)
+	}
+	resolvedModel := model
+	if resolvedModel == "" {
+		resolvedModel = s.model
+	}
+	return Result{Model: resolvedModel, Dimension: s.dim, Vectors: out}, nil
+}
+
+func deterministicVector(t string) []float32 {
+	v := make([]float32, 4)
+	for i := range v {
+		v[i] = float32(int(t[0]) + i)
+	}
+	return v
+}
+
+func newStub() *stubProvider {
+	return &stubProvider{
+		model:    "test-model",
+		dim:      4,
+		vectorFn: deterministicVector,
+	}
+}
+
+func TestCachedProvider_PassThroughWhenSizeZero(t *testing.T) {
+	stub := newStub()
+	c, err := NewCachedProvider(stub, "test-model", 0)
+	if err != nil {
+		t.Fatalf("NewCachedProvider: %v", err)
+	}
+	for i := 0; i < 3; i++ {
+		_, err := c.Embed(context.Background(), []string{"hello"}, "")
+		if err != nil {
+			t.Fatalf("Embed: %v", err)
+		}
+	}
+	if stub.callCount != 3 {
+		t.Errorf("size=0 should pass through every call, got callCount=%d", stub.callCount)
+	}
+	if c.Len() != 0 {
+		t.Errorf("Len()=%d, want 0 when caching disabled", c.Len())
+	}
+}
+
+func TestCachedProvider_HitOnRepeatText(t *testing.T) {
+	stub := newStub()
+	c, err := NewCachedProvider(stub, "test-model", 100)
+	if err != nil {
+		t.Fatalf("NewCachedProvider: %v", err)
+	}
+
+	// First call — miss, hits upstream.
+	r1, err := c.Embed(context.Background(), []string{"hello"}, "test-model")
+	if err != nil {
+		t.Fatalf("first: %v", err)
+	}
+	if stub.callCount != 1 {
+		t.Errorf("first call should hit upstream, callCount=%d", stub.callCount)
+	}
+
+	// Second call — hit, no upstream call.
+	r2, err := c.Embed(context.Background(), []string{"hello"}, "test-model")
+	if err != nil {
+		t.Fatalf("second: %v", err)
+	}
+	if stub.callCount != 1 {
+		t.Errorf("second call should hit cache, callCount=%d (want unchanged 1)", stub.callCount)
+	}
+
+	// Result should be byte-equal.
+	if len(r1.Vectors[0]) != len(r2.Vectors[0]) {
+		t.Errorf("vector dim differs: %d vs %d", len(r1.Vectors[0]), len(r2.Vectors[0]))
+	}
+	for i := range r1.Vectors[0] {
+		if r1.Vectors[0][i] != r2.Vectors[0][i] {
+			t.Errorf("vector[%d]: %v vs %v", i, r1.Vectors[0][i], r2.Vectors[0][i])
+		}
+	}
+
+	hits, misses := c.Stats()
+	if hits != 1 || misses != 1 {
+		t.Errorf("hits/misses = %d/%d, want 1/1", hits, misses)
+	}
+}
+
+func TestCachedProvider_MixedBatchOnlyFetchesMisses(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 100)
+
+	// Prime cache with one text.
+	c.Embed(context.Background(), []string{"alpha"}, "test-model")
+	if stub.callCount != 1 {
+		t.Fatalf("priming should produce 1 upstream call, got %d", stub.callCount)
+	}
+
+	// Mixed batch: alpha cached, beta + gamma fresh.
+	r, err := c.Embed(context.Background(),
+		[]string{"alpha", "beta", "gamma"}, "test-model")
+	if err != nil {
+		t.Fatalf("Embed: %v", err)
+	}
+
+	// Provider should have been called only with the misses.
+	if len(stub.lastTexts) != 2 {
+		t.Errorf("upstream called with %d texts, want 2 (only misses)", len(stub.lastTexts))
+	}
+	for _, lt := range stub.lastTexts {
+		if lt == "alpha" {
+			t.Errorf("upstream got 'alpha' but it was cached")
+		}
+	}
+
+	// Result preserves order: alpha first (cached), beta + gamma in
+	// their original positions.
+	if len(r.Vectors) != 3 {
+		t.Fatalf("got %d vectors, want 3", len(r.Vectors))
+	}
+	for i, txt := range []string{"alpha", "beta", "gamma"} {
+		want := deterministicVector(txt)
+		for j := range want {
+			if r.Vectors[i][j] != want[j] {
+				t.Errorf("vec[%d (%s)][%d] = %v, want %v",
+					i, txt, j, r.Vectors[i][j], want[j])
+			}
+		}
+	}
+}
+
+func TestCachedProvider_ModelKeyIsolation(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "default-model", 100)
+
+	// Same text, different models → distinct cache entries.
+	c.Embed(context.Background(), []string{"shared"}, "model-A")
+	c.Embed(context.Background(), []string{"shared"}, "model-B")
+	if stub.callCount != 2 {
+		t.Errorf("different models should NOT share cache, got callCount=%d", stub.callCount)
+	}
+
+	// Repeat with model-A → hits cache.
+	c.Embed(context.Background(), []string{"shared"}, "model-A")
+	if stub.callCount != 2 {
+		t.Errorf("repeat of model-A should hit cache, got callCount=%d", stub.callCount)
+	}
+}
+
+func TestCachedProvider_EmptyModelResolvesToDefault(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "the-default", 100)
+
+	// First call with model="" — caches under "the-default".
+	c.Embed(context.Background(), []string{"text"}, "")
+	if stub.callCount != 1 {
+		t.Fatalf("first call should hit upstream, got %d", stub.callCount)
+	}
+
+	// Second call with model="the-default" — should hit cache because
+	// the first call resolved "" → "the-default" for the key.
+	c.Embed(context.Background(), []string{"text"}, "the-default")
+	if stub.callCount != 1 {
+		t.Errorf("explicit default-model should hit cache populated by '' request, got callCount=%d", stub.callCount)
+	}
+}
+
+func TestCachedProvider_LRUEviction(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 2) // cap=2
+
+	// Fill: a, b
+	c.Embed(context.Background(), []string{"a"}, "")
+	c.Embed(context.Background(), []string{"b"}, "")
+	// c arrives — evicts a (LRU).
+	c.Embed(context.Background(), []string{"c"}, "")
+	if c.Len() > 2 {
+		t.Errorf("Len()=%d, want ≤2 with cap=2", c.Len())
+	}
+	if stub.callCount != 3 {
+		t.Errorf("callCount=%d, want 3 after a/b/c distinct", stub.callCount)
+	}
+
+	// Re-request a → miss (was evicted).
+	c.Embed(context.Background(), []string{"a"}, "")
+	if stub.callCount != 4 {
+		t.Errorf("evicted a should miss, callCount=%d want 4", stub.callCount)
+	}
+}
+
+func TestCachedProvider_PropagatesUpstreamError(t *testing.T) {
+	stub := &stubProvider{
+		model:    "test-model",
+		dim:      4,
+		vectorFn: deterministicVector,
+		err:      errors.New("upstream broken"),
+	}
+	c, _ := NewCachedProvider(stub, "test-model", 10)
+
+	_, err := c.Embed(context.Background(), []string{"hello"}, "")
+	if err == nil {
+		t.Fatal("expected upstream error to propagate")
+	}
+	// Cache should NOT have stored anything on error.
+	if c.Len() != 0 {
+		t.Errorf("error path should not populate cache, Len()=%d", c.Len())
+	}
+}
+
+func TestCachedProvider_AllHitsSynthesized(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 100)
+
+	// Prime
+	c.Embed(context.Background(), []string{"x", "y"}, "")
+	stub.mu.Lock()
+	stub.callCount = 0 // reset for clarity
+	stub.mu.Unlock()
+
+	// All-hits batch → no upstream call.
+	r, err := c.Embed(context.Background(), []string{"x", "y"}, "")
+	if err != nil {
+		t.Fatalf("Embed: %v", err)
+	}
+	if stub.callCount != 0 {
+		t.Errorf("all-hits batch should not hit upstream, callCount=%d", stub.callCount)
+	}
+	if r.Dimension != 4 {
+		t.Errorf("all-hits Dimension=%d, want 4 (from cached vector len)", r.Dimension)
+	}
+	if r.Model != "test-model" {
+		t.Errorf("all-hits Model=%q, want resolved 'test-model'", r.Model)
+	}
+}
+
+func TestCachedProvider_HitRate(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 100)
+
+	if c.HitRate() != 0.0 {
+		t.Errorf("initial HitRate=%v, want 0.0", c.HitRate())
+	}
+
+	c.Embed(context.Background(), []string{"a"}, "")    // miss
+	c.Embed(context.Background(), []string{"a"}, "")    // hit
+	c.Embed(context.Background(), []string{"a"}, "")    // hit
+
+	got := c.HitRate()
+	want := 2.0 / 3.0
+	if got < want-0.01 || got > want+0.01 {
+		t.Errorf("HitRate=%v, want %v", got, want)
+	}
+}
+
+func TestCachedProvider_EmptyTextsRejected(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 100)
+
+	_, err := c.Embed(context.Background(), []string{}, "")
+	if !errors.Is(err, ErrEmptyTexts) {
+		t.Errorf("empty texts should return ErrEmptyTexts, got %v", err)
+	}
+}
+
+func TestCachedProvider_ConcurrentSafe(t *testing.T) {
+	stub := newStub()
+	c, _ := NewCachedProvider(stub, "test-model", 100)
+
+	// 50 goroutines × 100 calls each, mostly the same text.
+	var wg sync.WaitGroup
+	for g := 0; g < 50; g++ {
+		wg.Add(1)
+		go func(id int) {
+			defer wg.Done()
+			for i := 0; i < 100; i++ {
+				_, err := c.Embed(context.Background(), []string{"shared"}, "")
+				if err != nil {
+					t.Errorf("concurrent embed err: %v", err)
+					return
+				}
+			}
+		}(g)
+	}
+	wg.Wait()
+
+	hits, misses := c.Stats()
+	if hits+misses != 5000 {
+		t.Errorf("total calls = %d, want 5000", hits+misses)
+	}
+	// First call is a guaranteed miss; subsequent should mostly be
+	// hits. With 50 concurrent goroutines hitting an empty cache,
+	// up to ~50 first-arrivers may miss before the first one writes.
+	if misses > 100 {
+		t.Errorf("misses=%d unreasonably high; cache concurrency broken?", misses)
+	}
+}
+
+func TestCacheKey_StableAndDistinct(t *testing.T) {
+	a := cacheKey("m1", "text-a")
+	a2 := cacheKey("m1", "text-a")
+	b := cacheKey("m1", "text-b")
+	c := cacheKey("m2", "text-a")
+
+	if a != a2 {
+		t.Errorf("cache key not stable: %q != %q", a, a2)
+	}
+	if a == b {
+		t.Errorf("different texts produced same key: %q", a)
+	}
+	if a == c {
+		t.Errorf("different models produced same key: %q", a)
+	}
+	// Key shape: model + ":" + 64-hex-chars sha256.
+	const sha256HexLen = 64
+	want := len("m1") + 1 + sha256HexLen
+	if len(a) != want {
+		t.Errorf("key len=%d, want %d", len(a), want)
+	}
+}

internal/langfuse/client.go

@@ -0,0 +1,222 @@
+// Package langfuse is a minimal Go-side client for the Langfuse v2
+// ingestion API. Mirrors the surface area we need from the Rust
+// crates/gateway/src/v1/langfuse_trace.rs emitter — Trace + Span,
+// nothing else yet (no scores, no observations, no datasets).
+//
+// Auth is Basic over public_key:secret_key. URL + creds come from
+// /etc/lakehouse/langfuse.env in production; tests can pass any URL.
+//
+// Best-effort transport: errors are logged but don't fail the calling
+// path. Lakehouse's internal services should never go down because
+// Langfuse is unreachable.
+package langfuse
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"sync"
+	"time"
+)
+
+// Client posts traces + spans to Langfuse's ingestion endpoint.
+// Events are buffered and flushed in batches. Always call Flush
+// before exit; Close also flushes.
+type Client struct {
+	url       string
+	auth      string // pre-encoded "Basic ..."
+	hc        *http.Client
+	mu        sync.Mutex
+	pending   []event
+	maxBatch  int
+}
+
+// New constructs a Client. URL like "http://localhost:3001"; creds
+// from langfuse.env. nil hc → uses default with 5s timeout.
+func New(url, publicKey, secretKey string, hc *http.Client) *Client {
+	if hc == nil {
+		hc = &http.Client{Timeout: 5 * time.Second}
+	}
+	auth := "Basic " + base64.StdEncoding.EncodeToString([]byte(publicKey+":"+secretKey))
+	return &Client{
+		url:      url,
+		auth:     auth,
+		hc:       hc,
+		maxBatch: 50,
+	}
+}
+
+// NewID returns a hex string suitable as a trace/span id. Langfuse
+// accepts arbitrary strings; a 16-byte random hex is unambiguous.
+func NewID() string {
+	b := make([]byte, 16)
+	_, _ = rand.Read(b)
+	return hex.EncodeToString(b)
+}
+
+// event is one Langfuse ingestion envelope. Body shape varies by
+// type (trace-create vs span-create); we use map[string]any to
+// keep the wire shape declarative.
+type event struct {
+	ID        string         `json:"id"`
+	Type      string         `json:"type"` // "trace-create" | "span-create"
+	Timestamp string         `json:"timestamp"`
+	Body      map[string]any `json:"body"`
+}
+
+// TraceInput is what callers fill in when starting a trace.
+type TraceInput struct {
+	Name     string
+	UserID   string
+	Input    any
+	Metadata map[string]any
+	Tags     []string
+}
+
+// Trace records a top-level trace. Returns the trace id so callers
+// can attach spans. Best-effort: errors are logged and the trace
+// id is still returned so callers don't need error-handling for the
+// common case.
+func (c *Client) Trace(ctx context.Context, t TraceInput) string {
+	id := NewID()
+	body := map[string]any{
+		"id":   id,
+		"name": t.Name,
+	}
+	if t.UserID != "" {
+		body["userId"] = t.UserID
+	}
+	if t.Input != nil {
+		body["input"] = t.Input
+	}
+	if t.Metadata != nil {
+		body["metadata"] = t.Metadata
+	}
+	if len(t.Tags) > 0 {
+		body["tags"] = t.Tags
+	}
+	c.queue(event{
+		ID:        NewID(),
+		Type:      "trace-create",
+		Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
+		Body:      body,
+	})
+	return id
+}
+
+// SpanInput is what callers fill in when recording a span.
+type SpanInput struct {
+	TraceID    string
+	ParentID   string // optional — for nested spans
+	Name       string
+	Input      any
+	Output     any
+	Metadata   map[string]any
+	StartTime  time.Time
+	EndTime    time.Time
+	StatusCode int    // 0 = success, anything else = error code
+	Level      string // "DEBUG" | "DEFAULT" | "WARNING" | "ERROR"
+}
+
+// Span records one span attached to a trace. Returns the span id.
+func (c *Client) Span(ctx context.Context, s SpanInput) string {
+	id := NewID()
+	body := map[string]any{
+		"id":      id,
+		"traceId": s.TraceID,
+		"name":    s.Name,
+	}
+	if s.ParentID != "" {
+		body["parentObservationId"] = s.ParentID
+	}
+	if s.Input != nil {
+		body["input"] = s.Input
+	}
+	if s.Output != nil {
+		body["output"] = s.Output
+	}
+	if s.Metadata != nil {
+		body["metadata"] = s.Metadata
+	}
+	if !s.StartTime.IsZero() {
+		body["startTime"] = s.StartTime.UTC().Format(time.RFC3339Nano)
+	}
+	if !s.EndTime.IsZero() {
+		body["endTime"] = s.EndTime.UTC().Format(time.RFC3339Nano)
+	}
+	if s.Level != "" {
+		body["level"] = s.Level
+	}
+	if s.StatusCode != 0 {
+		body["statusMessage"] = fmt.Sprintf("status_code=%d", s.StatusCode)
+	}
+	c.queue(event{
+		ID:        NewID(),
+		Type:      "span-create",
+		Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
+		Body:      body,
+	})
+	return id
+}
+
+func (c *Client) queue(e event) {
+	c.mu.Lock()
+	c.pending = append(c.pending, e)
+	shouldFlush := len(c.pending) >= c.maxBatch
+	c.mu.Unlock()
+	if shouldFlush {
+		// Fire-and-forget so the calling goroutine isn't blocked by
+		// the 5s HTTP timeout when Langfuse hiccups. Per scrum (Opus
+		// WARN): synchronous flush from queue defeated the
+		// "best-effort, never blocks calling path" guarantee in the
+		// package doc.
+		go func() { _ = c.Flush(context.Background()) }()
+	}
+}
+
+// Flush sends all queued events in one batch. Best-effort: returns
+// the error but also logs; callers can ignore.
+func (c *Client) Flush(ctx context.Context) error {
+	c.mu.Lock()
+	if len(c.pending) == 0 {
+		c.mu.Unlock()
+		return nil
+	}
+	batch := c.pending
+	c.pending = nil
+	c.mu.Unlock()
+
+	body, err := json.Marshal(map[string]any{"batch": batch})
+	if err != nil {
+		slog.Warn("langfuse: marshal batch", "err", err, "n", len(batch))
+		return err
+	}
+	req, err := http.NewRequestWithContext(ctx, "POST", c.url+"/api/public/ingestion", bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Authorization", c.auth)
+	req.Header.Set("Content-Type", "application/json")
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		slog.Warn("langfuse: post", "err", err, "n", len(batch))
+		return err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode/100 != 2 && resp.StatusCode != 207 {
+		slog.Warn("langfuse: non-2xx", "status", resp.StatusCode, "n", len(batch))
+		return fmt.Errorf("langfuse ingestion: HTTP %d", resp.StatusCode)
+	}
+	return nil
+}
+
+// Close flushes any remaining events. Idempotent.
+func (c *Client) Close() error {
+	return c.Flush(context.Background())
+}

internal/materializer/canonical.go

@@ -0,0 +1,93 @@
+// Package materializer ports scripts/distillation/transforms.ts +
+// build_evidence_index.ts to Go. Source rows in data/_kb/*.jsonl are
+// transformed into EvidenceRecord rows under data/evidence/YYYY/MM/DD/.
+//
+// Per ADR-001 #4: port LOGIC, not bit-identical reproducibility — but
+// on-wire JSON layout matches the TS shape so Bun and Go runs stay
+// interchangeable for tooling that reads either output.
+package materializer
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"sort"
+)
+
+// CanonicalSha256 returns the hex SHA-256 of `obj` after sorting all
+// object keys recursively. Matches the TS canonicalSha256 in
+// auditor/schemas/distillation/types.ts so a row hashed by either
+// runtime gets the same sig_hash.
+//
+// Determinism contract: identical input → identical hash, regardless
+// of the producer's serialization order.
+func CanonicalSha256(obj any) (string, error) {
+	ordered := orderKeys(obj)
+	buf, err := json.Marshal(ordered)
+	if err != nil {
+		return "", fmt.Errorf("canonical marshal: %w", err)
+	}
+	sum := sha256.Sum256(buf)
+	return hex.EncodeToString(sum[:]), nil
+}
+
+// orderKeys recursively sorts every map's keys. For arrays we keep the
+// element order (arrays are inherently ordered). Scalars pass through.
+func orderKeys(v any) any {
+	switch t := v.(type) {
+	case map[string]any:
+		keys := make([]string, 0, len(t))
+		for k := range t {
+			keys = append(keys, k)
+		}
+		sort.Strings(keys)
+		out := make(orderedMap, 0, len(keys))
+		for _, k := range keys {
+			out = append(out, kvPair{Key: k, Value: orderKeys(t[k])})
+		}
+		return out
+	case []any:
+		out := make([]any, len(t))
+		for i, e := range t {
+			out[i] = orderKeys(e)
+		}
+		return out
+	default:
+		return v
+	}
+}
+
+// orderedMap preserves insertion order on JSON marshal. We populate it
+// in sorted-key order so the produced bytes are stable.
+type orderedMap []kvPair
+
+type kvPair struct {
+	Key   string
+	Value any
+}
+
+func (om orderedMap) MarshalJSON() ([]byte, error) {
+	if len(om) == 0 {
+		return []byte("{}"), nil
+	}
+	out := []byte{'{'}
+	for i, kv := range om {
+		if i > 0 {
+			out = append(out, ',')
+		}
+		k, err := json.Marshal(kv.Key)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, k...)
+		out = append(out, ':')
+		v, err := json.Marshal(kv.Value)
+		if err != nil {
+			return nil, err
+		}
+		out = append(out, v...)
+	}
+	out = append(out, '}')
+	return out, nil
+}

internal/materializer/canonical_test.go

@@ -0,0 +1,45 @@
+package materializer
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestCanonicalSha256_StableAcrossMapOrder(t *testing.T) {
+	a := map[string]any{"b": 2, "a": 1, "c": map[string]any{"y": "Y", "x": "X"}}
+	b := map[string]any{"a": 1, "c": map[string]any{"x": "X", "y": "Y"}, "b": 2}
+	hashA, err := CanonicalSha256(a)
+	if err != nil {
+		t.Fatalf("hash a: %v", err)
+	}
+	hashB, err := CanonicalSha256(b)
+	if err != nil {
+		t.Fatalf("hash b: %v", err)
+	}
+	if hashA != hashB {
+		t.Fatalf("identical objects produced different hashes:\n  a=%s\n  b=%s", hashA, hashB)
+	}
+	if len(hashA) != 64 || strings.Trim(hashA, "0123456789abcdef") != "" {
+		t.Fatalf("hash isn't a 64-char hex string: %q", hashA)
+	}
+}
+
+func TestCanonicalSha256_DistinctsDifferentInputs(t *testing.T) {
+	a := map[string]any{"k": "v"}
+	b := map[string]any{"k": "v2"}
+	hashA, _ := CanonicalSha256(a)
+	hashB, _ := CanonicalSha256(b)
+	if hashA == hashB {
+		t.Fatalf("different inputs collided: %s", hashA)
+	}
+}
+
+func TestCanonicalSha256_ArrayOrderMatters(t *testing.T) {
+	a := map[string]any{"k": []any{1, 2, 3}}
+	b := map[string]any{"k": []any{3, 2, 1}}
+	hashA, _ := CanonicalSha256(a)
+	hashB, _ := CanonicalSha256(b)
+	if hashA == hashB {
+		t.Fatal("array order should change the hash, but did not")
+	}
+}

internal/materializer/materializer.go

@@ -0,0 +1,515 @@
+package materializer
+
+import (
+	"bufio"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// MaterializeOptions drives MaterializeAll. Tests construct this with
+// a temp Root and override Transforms; the CLI uses defaults.
+type MaterializeOptions struct {
+	Root       string         // repo root; sources + outputs are relative
+	Transforms []TransformDef // override for tests
+	RecordedAt string         // ISO 8601 — fixed for the run
+	DryRun     bool           // count but don't write
+}
+
+// SourceResult mirrors TS SourceResult.
+type SourceResult struct {
+	SourceFileRelPath string   `json:"source_file_relpath"`
+	RowsPresent       bool     `json:"rows_present"`
+	RowsRead          int      `json:"rows_read"`
+	RowsWritten       int      `json:"rows_written"`
+	RowsSkipped       int      `json:"rows_skipped"`
+	RowsDeduped       int      `json:"rows_deduped"`
+	OutputFiles       []string `json:"output_files"`
+}
+
+// MaterializeResult is what MaterializeAll returns. Receipt is the
+// authoritative "did the run succeed" surface — the rest is plumbing.
+type MaterializeResult struct {
+	Sources     []SourceResult `json:"sources"`
+	Totals      Totals         `json:"totals"`
+	Receipt     Receipt        `json:"receipt"`
+	ReceiptPath string         `json:"receipt_path"`
+	EvidenceDir string         `json:"evidence_dir"`
+	SkipsPath   string         `json:"skips_path"`
+}
+
+// Totals — flat sum across sources.
+type Totals struct {
+	RowsRead    int `json:"rows_read"`
+	RowsWritten int `json:"rows_written"`
+	RowsSkipped int `json:"rows_skipped"`
+	RowsDeduped int `json:"rows_deduped"`
+}
+
+// Receipt mirrors auditor/schemas/distillation/receipt.ts. Schema
+// version pinned to match the TS producer so consumers see the same
+// shape regardless of which runtime generated the run.
+const ReceiptSchemaVersion = 1
+
+type Receipt struct {
+	SchemaVersion   int             `json:"schema_version"`
+	Command         string          `json:"command"`
+	GitSHA          string          `json:"git_sha"`
+	GitBranch       string          `json:"git_branch,omitempty"`
+	GitDirty        bool            `json:"git_dirty"`
+	StartedAt       string          `json:"started_at"`
+	EndedAt         string          `json:"ended_at"`
+	DurationMs      int64           `json:"duration_ms"`
+	InputFiles      []FileReference `json:"input_files"`
+	OutputFiles     []FileReference `json:"output_files"`
+	RecordCounts    RecordCounts    `json:"record_counts"`
+	ValidationPass  bool            `json:"validation_pass"`
+	Errors          []string        `json:"errors"`
+	Warnings        []string        `json:"warnings"`
+}
+
+type FileReference struct {
+	Path   string `json:"path"`
+	SHA256 string `json:"sha256"`
+	Bytes  int64  `json:"bytes"`
+}
+
+type RecordCounts struct {
+	In      int `json:"in"`
+	Out     int `json:"out"`
+	Skipped int `json:"skipped"`
+	Deduped int `json:"deduped"`
+}
+
+// SkipRecord is one row in distillation_skips.jsonl. Operators read
+// this stream when a run reports rows_skipped > 0.
+type SkipRecord struct {
+	SourceFile string   `json:"source_file"`
+	LineOffset int64    `json:"line_offset"`
+	Errors     []string `json:"errors"`
+	SigHash    string   `json:"sig_hash,omitempty"`
+	RecordedAt string   `json:"recorded_at"`
+}
+
+// MaterializeAll iterates Transforms[], reads each source JSONL,
+// transforms each row, validates, writes to date-partitioned output.
+// Returns a Receipt whose ValidationPass tells the caller whether all
+// rows survived validation.
+func MaterializeAll(opts MaterializeOptions) (MaterializeResult, error) {
+	if opts.RecordedAt == "" {
+		return MaterializeResult{}, errors.New("MaterializeOptions.RecordedAt required")
+	}
+	if opts.Root == "" {
+		return MaterializeResult{}, errors.New("MaterializeOptions.Root required")
+	}
+	if !validISOTimestamp(opts.RecordedAt) {
+		return MaterializeResult{}, fmt.Errorf("RecordedAt not ISO 8601: %s", opts.RecordedAt)
+	}
+	transforms := opts.Transforms
+	if transforms == nil {
+		transforms = Transforms
+	}
+
+	evidenceDir := filepath.Join(opts.Root, "data", "evidence")
+	skipsPath := filepath.Join(opts.Root, "data", "_kb", "distillation_skips.jsonl")
+	reportsDir := filepath.Join(opts.Root, "reports", "distillation")
+
+	startedMs := time.Now().UnixMilli()
+	sources := make([]SourceResult, 0, len(transforms))
+	for _, t := range transforms {
+		sr, err := processSource(t, opts, evidenceDir, skipsPath)
+		if err != nil {
+			return MaterializeResult{}, fmt.Errorf("processSource %s: %w", t.SourceFileRelPath, err)
+		}
+		sources = append(sources, sr)
+	}
+
+	totals := Totals{}
+	for _, s := range sources {
+		totals.RowsRead += s.RowsRead
+		totals.RowsWritten += s.RowsWritten
+		totals.RowsSkipped += s.RowsSkipped
+		totals.RowsDeduped += s.RowsDeduped
+	}
+
+	endedAt := time.Now().UTC().Format(time.RFC3339Nano)
+	durationMs := time.Now().UnixMilli() - startedMs
+
+	inputFiles := make([]FileReference, 0)
+	for _, s := range sources {
+		if !s.RowsPresent {
+			continue
+		}
+		path := filepath.Join(opts.Root, s.SourceFileRelPath)
+		ref, err := fileReferenceAt(path, s.SourceFileRelPath)
+		if err == nil {
+			inputFiles = append(inputFiles, ref)
+		}
+	}
+	outputFiles := make([]FileReference, 0)
+	for _, s := range sources {
+		for _, p := range s.OutputFiles {
+			rel := strings.TrimPrefix(p, opts.Root+string(os.PathSeparator))
+			ref, err := fileReferenceAt(p, rel)
+			if err == nil {
+				outputFiles = append(outputFiles, ref)
+			}
+		}
+	}
+
+	var (
+		errs     []string
+		warnings []string
+	)
+	for _, s := range sources {
+		if !s.RowsPresent {
+			warnings = append(warnings, fmt.Sprintf("%s: source file not found (skipped)", s.SourceFileRelPath))
+		}
+		if s.RowsSkipped > 0 {
+			warnings = append(warnings, fmt.Sprintf("%s: %d rows skipped (validation/parse errors)", s.SourceFileRelPath, s.RowsSkipped))
+		}
+	}
+
+	receipt := Receipt{
+		SchemaVersion: ReceiptSchemaVersion,
+		Command:       commandLineOf(opts),
+		GitSHA:        getGitSHA(opts.Root),
+		GitBranch:     getGitBranch(opts.Root),
+		GitDirty:      getGitDirty(opts.Root),
+		StartedAt:     opts.RecordedAt,
+		EndedAt:       endedAt,
+		DurationMs:    durationMs,
+		InputFiles:    inputFiles,
+		OutputFiles:   outputFiles,
+		RecordCounts: RecordCounts{
+			In:      totals.RowsRead,
+			Out:     totals.RowsWritten,
+			Skipped: totals.RowsSkipped,
+			Deduped: totals.RowsDeduped,
+		},
+		ValidationPass: totals.RowsSkipped == 0,
+		Errors:         emptyToNil(errs),
+		Warnings:       emptyToNil(warnings),
+	}
+
+	stamp := strings.NewReplacer(":", "-", ".", "-").Replace(endedAt)
+	receiptDir := filepath.Join(reportsDir, stamp)
+	receiptPath := filepath.Join(receiptDir, "receipt.json")
+	if !opts.DryRun {
+		if err := os.MkdirAll(receiptDir, 0o755); err != nil {
+			return MaterializeResult{}, fmt.Errorf("mkdir receipt dir: %w", err)
+		}
+		buf, err := json.MarshalIndent(receipt, "", "  ")
+		if err != nil {
+			return MaterializeResult{}, fmt.Errorf("marshal receipt: %w", err)
+		}
+		buf = append(buf, '\n')
+		if err := os.WriteFile(receiptPath, buf, 0o644); err != nil {
+			return MaterializeResult{}, fmt.Errorf("write receipt: %w", err)
+		}
+	}
+
+	return MaterializeResult{
+		Sources:     sources,
+		Totals:      totals,
+		Receipt:     receipt,
+		ReceiptPath: receiptPath,
+		EvidenceDir: evidenceDir,
+		SkipsPath:   skipsPath,
+	}, nil
+}
+
+// processSource reads, transforms, validates, and writes a single
+// source JSONL.
+func processSource(t TransformDef, opts MaterializeOptions, evidenceDir, skipsPath string) (SourceResult, error) {
+	srcPath := filepath.Join(opts.Root, t.SourceFileRelPath)
+	res := SourceResult{SourceFileRelPath: t.SourceFileRelPath}
+
+	info, err := os.Stat(srcPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return res, nil
+		}
+		return res, fmt.Errorf("stat %s: %w", srcPath, err)
+	}
+	if info.IsDir() {
+		return res, fmt.Errorf("%s is a directory, not a file", srcPath)
+	}
+	res.RowsPresent = true
+
+	partition := isoDatePartition(opts.RecordedAt)
+	stem := stemFor(t.SourceFileRelPath)
+	outDir := filepath.Join(evidenceDir, partition)
+	outPath := filepath.Join(outDir, stem+".jsonl")
+	if !opts.DryRun {
+		if err := os.MkdirAll(outDir, 0o755); err != nil {
+			return res, fmt.Errorf("mkdir output dir: %w", err)
+		}
+	}
+
+	seen, err := loadSeenHashes(outPath)
+	if err != nil {
+		return res, fmt.Errorf("load seen hashes: %w", err)
+	}
+
+	f, err := os.Open(srcPath)
+	if err != nil {
+		return res, fmt.Errorf("open %s: %w", srcPath, err)
+	}
+	defer f.Close()
+
+	var (
+		rowsToWrite  []byte
+		skipsToWrite []byte
+	)
+
+	scanner := bufio.NewScanner(f)
+	scanner.Buffer(make([]byte, 0, 1<<16), 1<<24)
+	lineOffset := int64(-1)
+	for scanner.Scan() {
+		lineOffset++
+		raw := scanner.Bytes()
+		if len(raw) == 0 {
+			continue
+		}
+		res.RowsRead++
+
+		var row map[string]any
+		if err := json.Unmarshal(raw, &row); err != nil {
+			res.RowsSkipped++
+			skipsToWrite = appendSkip(skipsToWrite, SkipRecord{
+				SourceFile: t.SourceFileRelPath,
+				LineOffset: lineOffset,
+				Errors:     []string{"JSON.parse failed: " + trim(err.Error(), 200)},
+				RecordedAt: opts.RecordedAt,
+			})
+			continue
+		}
+
+		sigHash, err := CanonicalSha256(row)
+		if err != nil {
+			res.RowsSkipped++
+			skipsToWrite = appendSkip(skipsToWrite, SkipRecord{
+				SourceFile: t.SourceFileRelPath,
+				LineOffset: lineOffset,
+				Errors:     []string{"sig_hash compute failed: " + trim(err.Error(), 200)},
+				RecordedAt: opts.RecordedAt,
+			})
+			continue
+		}
+		if _, dup := seen[sigHash]; dup {
+			res.RowsDeduped++
+			continue
+		}
+		seen[sigHash] = struct{}{}
+
+		rec := t.Transform(TransformInput{
+			Row:               row,
+			LineOffset:        lineOffset,
+			SourceFileRelPath: t.SourceFileRelPath,
+			RecordedAt:        opts.RecordedAt,
+			SigHash:           sigHash,
+		})
+		if rec == nil {
+			res.RowsSkipped++
+			skipsToWrite = appendSkip(skipsToWrite, SkipRecord{
+				SourceFile: t.SourceFileRelPath,
+				LineOffset: lineOffset,
+				Errors:     []string{"transform returned nil"},
+				SigHash:    sigHash,
+				RecordedAt: opts.RecordedAt,
+			})
+			continue
+		}
+
+		if vErrs := ValidateEvidenceRecord(*rec); len(vErrs) > 0 {
+			res.RowsSkipped++
+			skipsToWrite = appendSkip(skipsToWrite, SkipRecord{
+				SourceFile: t.SourceFileRelPath,
+				LineOffset: lineOffset,
+				Errors:     vErrs,
+				SigHash:    sigHash,
+				RecordedAt: opts.RecordedAt,
+			})
+			continue
+		}
+
+		buf, err := json.Marshal(rec)
+		if err != nil {
+			res.RowsSkipped++
+			skipsToWrite = appendSkip(skipsToWrite, SkipRecord{
+				SourceFile: t.SourceFileRelPath,
+				LineOffset: lineOffset,
+				Errors:     []string{"marshal output: " + trim(err.Error(), 200)},
+				SigHash:    sigHash,
+				RecordedAt: opts.RecordedAt,
+			})
+			continue
+		}
+		rowsToWrite = append(rowsToWrite, buf...)
+		rowsToWrite = append(rowsToWrite, '\n')
+		res.RowsWritten++
+	}
+	if err := scanner.Err(); err != nil {
+		return res, fmt.Errorf("scan %s: %w", srcPath, err)
+	}
+
+	if !opts.DryRun {
+		if len(rowsToWrite) > 0 {
+			if err := appendBytes(outPath, rowsToWrite); err != nil {
+				return res, fmt.Errorf("append output: %w", err)
+			}
+			res.OutputFiles = append(res.OutputFiles, outPath)
+		}
+		if len(skipsToWrite) > 0 {
+			if err := os.MkdirAll(filepath.Dir(skipsPath), 0o755); err != nil {
+				return res, fmt.Errorf("mkdir skips dir: %w", err)
+			}
+			if err := appendBytes(skipsPath, skipsToWrite); err != nil {
+				return res, fmt.Errorf("append skips: %w", err)
+			}
+		}
+	}
+
+	return res, nil
+}
+
+// loadSeenHashes reads sig_hashes from an existing day-partition output
+// file. Idempotency: a re-run that produces the same hash is a dedup
+// not a duplicate write.
+func loadSeenHashes(outPath string) (map[string]struct{}, error) {
+	seen := map[string]struct{}{}
+	f, err := os.Open(outPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return seen, nil
+		}
+		return nil, err
+	}
+	defer f.Close()
+	scanner := bufio.NewScanner(f)
+	scanner.Buffer(make([]byte, 0, 1<<16), 1<<24)
+	for scanner.Scan() {
+		raw := scanner.Bytes()
+		if len(raw) == 0 {
+			continue
+		}
+		var rec struct {
+			Provenance struct {
+				SigHash string `json:"sig_hash"`
+			} `json:"provenance"`
+		}
+		if err := json.Unmarshal(raw, &rec); err != nil {
+			continue // malformed line; ignore
+		}
+		if rec.Provenance.SigHash != "" {
+			seen[rec.Provenance.SigHash] = struct{}{}
+		}
+	}
+	return seen, scanner.Err()
+}
+
+func appendSkip(buf []byte, sk SkipRecord) []byte {
+	out, err := json.Marshal(sk)
+	if err != nil {
+		// Should never happen for the well-typed SkipRecord — fall back
+		// to a sentinel so the materializer doesn't drop the skip silently.
+		return append(buf, []byte(fmt.Sprintf(`{"source_file":%q,"line_offset":%d,"errors":["marshal_skip_failed:%s"],"recorded_at":%q}`+"\n",
+			sk.SourceFile, sk.LineOffset, err.Error(), sk.RecordedAt))...)
+	}
+	buf = append(buf, out...)
+	buf = append(buf, '\n')
+	return buf
+}
+
+func appendBytes(path string, data []byte) error {
+	f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	_, err = f.Write(data)
+	return err
+}
+
+func isoDatePartition(iso string) string {
+	t, err := time.Parse(time.RFC3339Nano, iso)
+	if err != nil {
+		t, err = time.Parse(time.RFC3339, iso)
+	}
+	if err != nil {
+		// Defense-in-depth only — MaterializeAll calls validISOTimestamp
+		// on RecordedAt before processSource ever invokes us, so this
+		// branch is unreachable through the public surface. Kept so a
+		// future direct caller doesn't get a panic; "0000/00/00" is at
+		// least a valid path.
+		return "0000/00/00"
+	}
+	t = t.UTC()
+	return fmt.Sprintf("%04d/%02d/%02d", t.Year(), int(t.Month()), t.Day())
+}
+
+func fileReferenceAt(path, relpath string) (FileReference, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return FileReference{}, err
+	}
+	defer f.Close()
+	hasher := sha256.New()
+	n, err := io.Copy(hasher, f)
+	if err != nil {
+		return FileReference{}, err
+	}
+	return FileReference{
+		Path:   relpath,
+		SHA256: hex.EncodeToString(hasher.Sum(nil)),
+		Bytes:  n,
+	}, nil
+}
+
+func getGitSHA(root string) string {
+	out, err := exec.Command("git", "-C", root, "rev-parse", "HEAD").Output()
+	if err != nil {
+		return strings.Repeat("0", 40)
+	}
+	return strings.TrimSpace(string(out))
+}
+
+func getGitBranch(root string) string {
+	out, err := exec.Command("git", "-C", root, "rev-parse", "--abbrev-ref", "HEAD").Output()
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(out))
+}
+
+func getGitDirty(root string) bool {
+	out, err := exec.Command("git", "-C", root, "status", "--porcelain").Output()
+	if err != nil {
+		return false
+	}
+	return strings.TrimSpace(string(out)) != ""
+}
+
+func commandLineOf(opts MaterializeOptions) string {
+	cmd := "go run ./cmd/materializer"
+	if opts.DryRun {
+		cmd += " --dry-run"
+	}
+	return cmd
+}
+
+func emptyToNil(s []string) []string {
+	if len(s) == 0 {
+		return []string{}
+	}
+	return s
+}

internal/materializer/materializer_test.go

@@ -0,0 +1,218 @@
+package materializer
+
+import (
+	"bufio"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestMaterializeAll_RoundTrip writes a fixture source jsonl, runs the
+// materializer, and checks every contract: receipt, output rows,
+// idempotency on second run.
+func TestMaterializeAll_RoundTrip(t *testing.T) {
+	root := t.TempDir()
+	mustWriteFixture(t, root, "data/_kb/distilled_facts.jsonl",
+		`{"run_id":"r1","source_label":"lab-a","created_at":"2026-04-26T00:00:00Z","extractor":"qwen3.5:latest","text":"first"}
+{"run_id":"r2","source_label":"lab-b","created_at":"2026-04-26T01:00:00Z","extractor":"qwen3.5:latest","text":"second"}`)
+
+	transforms := []TransformDef{
+		{SourceFileRelPath: "data/_kb/distilled_facts.jsonl", Transform: extractorTransform},
+	}
+
+	first, err := MaterializeAll(MaterializeOptions{
+		Root:       root,
+		Transforms: transforms,
+		RecordedAt: "2026-05-02T00:00:00Z",
+	})
+	if err != nil {
+		t.Fatalf("first run: %v", err)
+	}
+	if !first.Receipt.ValidationPass {
+		t.Errorf("first run should pass validation. errors=%v warnings=%v", first.Receipt.Errors, first.Receipt.Warnings)
+	}
+	if first.Totals.RowsRead != 2 || first.Totals.RowsWritten != 2 || first.Totals.RowsSkipped != 0 {
+		t.Errorf("first run counts wrong: %+v", first.Totals)
+	}
+	if first.Totals.RowsDeduped != 0 {
+		t.Errorf("first run should have 0 dedupes, got %d", first.Totals.RowsDeduped)
+	}
+
+	outPath := filepath.Join(root, "data/evidence/2026/05/02/distilled_facts.jsonl")
+	rows := readJSONL(t, outPath)
+	if len(rows) != 2 {
+		t.Fatalf("expected 2 output rows, got %d", len(rows))
+	}
+	for _, r := range rows {
+		if r["schema_version"].(float64) != 1 {
+			t.Errorf("schema_version wrong: %v", r["schema_version"])
+		}
+		prov := r["provenance"].(map[string]any)
+		if prov["source_file"] != "data/_kb/distilled_facts.jsonl" {
+			t.Errorf("provenance.source_file: %v", prov["source_file"])
+		}
+		if prov["recorded_at"] != "2026-05-02T00:00:00Z" {
+			t.Errorf("provenance.recorded_at: %v", prov["recorded_at"])
+		}
+	}
+
+	// Second run with identical input + RecordedAt → all rows should
+	// dedup, nothing newly written.
+	second, err := MaterializeAll(MaterializeOptions{
+		Root:       root,
+		Transforms: transforms,
+		RecordedAt: "2026-05-02T00:00:00Z",
+	})
+	if err != nil {
+		t.Fatalf("second run: %v", err)
+	}
+	if second.Totals.RowsRead != 2 || second.Totals.RowsWritten != 0 || second.Totals.RowsDeduped != 2 {
+		t.Errorf("idempotency broken; second run counts: %+v", second.Totals)
+	}
+	rows2 := readJSONL(t, outPath)
+	if len(rows2) != 2 {
+		t.Fatalf("output file grew on idempotent rerun: %d rows", len(rows2))
+	}
+}
+
+func TestMaterializeAll_BadJSONLineGoesToSkips(t *testing.T) {
+	root := t.TempDir()
+	mustWriteFixture(t, root, "data/_kb/distilled_facts.jsonl",
+		`{"run_id":"r1","source_label":"a","created_at":"2026-04-26T00:00:00Z","extractor":"q","text":"t"}
+not-json
+{"run_id":"r2","source_label":"b","created_at":"2026-04-26T01:00:00Z","extractor":"q","text":"t2"}`)
+
+	transforms := []TransformDef{
+		{SourceFileRelPath: "data/_kb/distilled_facts.jsonl", Transform: extractorTransform},
+	}
+	res, err := MaterializeAll(MaterializeOptions{
+		Root:       root,
+		Transforms: transforms,
+		RecordedAt: "2026-05-02T00:00:00Z",
+	})
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+	if res.Totals.RowsWritten != 2 {
+		t.Errorf("good rows should still pass through; written=%d", res.Totals.RowsWritten)
+	}
+	if res.Totals.RowsSkipped != 1 {
+		t.Errorf("bad-json row should be in skipped bucket; got %d", res.Totals.RowsSkipped)
+	}
+	if res.Receipt.ValidationPass {
+		t.Errorf("validation_pass should be false when any row was skipped")
+	}
+
+	skipsPath := filepath.Join(root, "data/_kb/distillation_skips.jsonl")
+	skips := readJSONL(t, skipsPath)
+	if len(skips) != 1 {
+		t.Fatalf("expected 1 skip record, got %d", len(skips))
+	}
+	if !strings.Contains(toJSON(t, skips[0]), "JSON.parse failed") {
+		t.Errorf("skip record should mention parse failure: %v", skips[0])
+	}
+}
+
+func TestMaterializeAll_DryRunWritesNothing(t *testing.T) {
+	root := t.TempDir()
+	mustWriteFixture(t, root, "data/_kb/distilled_facts.jsonl",
+		`{"run_id":"r1","source_label":"a","created_at":"2026-04-26T00:00:00Z","extractor":"q","text":"t"}`)
+
+	transforms := []TransformDef{
+		{SourceFileRelPath: "data/_kb/distilled_facts.jsonl", Transform: extractorTransform},
+	}
+	res, err := MaterializeAll(MaterializeOptions{
+		Root:       root,
+		Transforms: transforms,
+		RecordedAt: "2026-05-02T00:00:00Z",
+		DryRun:     true,
+	})
+	if err != nil {
+		t.Fatalf("dry run: %v", err)
+	}
+	if res.Totals.RowsRead != 1 || res.Totals.RowsWritten != 1 {
+		t.Errorf("dry run should still count, got %+v", res.Totals)
+	}
+	outPath := filepath.Join(root, "data/evidence/2026/05/02/distilled_facts.jsonl")
+	if _, err := os.Stat(outPath); !os.IsNotExist(err) {
+		t.Errorf("dry run wrote output file (should not): err=%v", err)
+	}
+	if _, err := os.Stat(res.ReceiptPath); !os.IsNotExist(err) {
+		t.Errorf("dry run wrote receipt (should not): err=%v", err)
+	}
+}
+
+func TestMaterializeAll_MissingSourceTalliedAsWarning(t *testing.T) {
+	root := t.TempDir()
+	transforms := []TransformDef{
+		{SourceFileRelPath: "data/_kb/distilled_facts.jsonl", Transform: extractorTransform},
+	}
+	res, err := MaterializeAll(MaterializeOptions{
+		Root:       root,
+		Transforms: transforms,
+		RecordedAt: "2026-05-02T00:00:00Z",
+	})
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+	if res.Sources[0].RowsPresent {
+		t.Errorf("expected rows_present=false")
+	}
+	if !res.Receipt.ValidationPass {
+		t.Errorf("missing source ≠ validation failure; got pass=%v warnings=%v", res.Receipt.ValidationPass, res.Receipt.Warnings)
+	}
+	if len(res.Receipt.Warnings) == 0 {
+		t.Errorf("missing source should produce a warning")
+	}
+}
+
+// ─── Helpers ─────────────────────────────────────────────────────
+
+func mustWriteFixture(t *testing.T, root, relpath, content string) {
+	t.Helper()
+	full := filepath.Join(root, relpath)
+	if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+	if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
+		t.Fatalf("write fixture: %v", err)
+	}
+}
+
+func readJSONL(t *testing.T, path string) []map[string]any {
+	t.Helper()
+	f, err := os.Open(path)
+	if err != nil {
+		t.Fatalf("open %s: %v", path, err)
+	}
+	defer f.Close()
+	var out []map[string]any
+	sc := bufio.NewScanner(f)
+	sc.Buffer(make([]byte, 0, 1<<16), 1<<24)
+	for sc.Scan() {
+		line := sc.Bytes()
+		if len(line) == 0 {
+			continue
+		}
+		var row map[string]any
+		if err := json.Unmarshal(line, &row); err != nil {
+			t.Fatalf("parse %s: %v", path, err)
+		}
+		out = append(out, row)
+	}
+	if err := sc.Err(); err != nil {
+		t.Fatalf("scan %s: %v", path, err)
+	}
+	return out
+}
+
+func toJSON(t *testing.T, v any) string {
+	t.Helper()
+	b, err := json.Marshal(v)
+	if err != nil {
+		t.Fatalf("marshal: %v", err)
+	}
+	return string(b)
+}

internal/materializer/transforms.go

@@ -0,0 +1,653 @@
+package materializer
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+// TransformInput is what each TransformFn receives. Mirrors the TS
+// TransformInput shape — every field is supplied by the materializer
+// driver, not by the transform.
+type TransformInput struct {
+	Row                 map[string]any
+	LineOffset          int64
+	SourceFileRelPath   string // relative to repo root
+	RecordedAt          string // ISO 8601, caller's "now"
+	SigHash             string // canonical sha256 of row, pre-computed
+}
+
+// TransformFn maps a single source row to an EvidenceRecord. Returning
+// nil signals "skip this row" — the materializer logs a deterministic
+// skip with no record produced.
+//
+// Transforms must be pure: no I/O, no clock reads, no model calls.
+// Any time component must come from the row itself or RecordedAt.
+type TransformFn func(in TransformInput) *distillation.EvidenceRecord
+
+// TransformDef binds a source-file path to its TransformFn. Order in
+// Transforms[] has no effect (each runs against its own SourceFile).
+type TransformDef struct {
+	SourceFileRelPath string
+	Transform         TransformFn
+}
+
+// ─── Transforms — one per source-file. Ports of TRANSFORMS[] in
+// scripts/distillation/transforms.ts. Tier 1 first (validated), Tier 2
+// second (untested but in-shape). ────────────────────────────────────
+
+// Transforms is the canonical list. CLI passes this to MaterializeAll.
+// Adding a new source: append a TransformDef.
+var Transforms = []TransformDef{
+	// ── Tier 1: validated 100% in Phase 1 ─────────────────────────
+	{SourceFileRelPath: "data/_kb/distilled_facts.jsonl", Transform: extractorTransform},
+	{SourceFileRelPath: "data/_kb/distilled_procedures.jsonl", Transform: extractorTransform},
+	{SourceFileRelPath: "data/_kb/distilled_config_hints.jsonl", Transform: extractorTransform},
+	{SourceFileRelPath: "data/_kb/contract_analyses.jsonl", Transform: contractAnalysesTransform},
+	{SourceFileRelPath: "data/_kb/mode_experiments.jsonl", Transform: modeExperimentsTransform},
+	{SourceFileRelPath: "data/_kb/scrum_reviews.jsonl", Transform: scrumReviewsTransform},
+	{SourceFileRelPath: "data/_kb/observer_escalations.jsonl", Transform: observerEscalationsTransform},
+	{SourceFileRelPath: "data/_kb/audit_facts.jsonl", Transform: auditFactsTransform},
+
+	// ── Tier 2: untested streams that still belong in EvidenceRecord ──
+	{SourceFileRelPath: "data/_kb/auto_apply.jsonl", Transform: autoApplyTransform},
+	{SourceFileRelPath: "data/_kb/observer_reviews.jsonl", Transform: observerReviewsTransform},
+	{SourceFileRelPath: "data/_kb/audits.jsonl", Transform: auditsTransform},
+	{SourceFileRelPath: "data/_kb/outcomes.jsonl", Transform: outcomesTransform},
+}
+
+// TransformByPath returns the TransformDef for a given source path,
+// or nil if no transform is registered. Matches the TS helper.
+func TransformByPath(relpath string) *TransformDef {
+	for i := range Transforms {
+		if Transforms[i].SourceFileRelPath == relpath {
+			return &Transforms[i]
+		}
+	}
+	return nil
+}
+
+// ─── Per-source transform implementations ─────────────────────────
+
+// extractorTransform powers the three distilled_* sources. Same shape:
+// LLM-extracted text with a model_name from `extractor`.
+func extractorTransform(in TransformInput) *distillation.EvidenceRecord {
+	stem := stemFor(in.SourceFileRelPath)
+	rec := distillation.EvidenceRecord{
+		RunID:         strDefault(in.Row, "run_id", fmt.Sprintf("%s:%d", stem, in.LineOffset)),
+		TaskID:        strDefault(in.Row, "source_label", fmt.Sprintf("%s:%d", stem, in.LineOffset)),
+		Timestamp:     getString(in.Row, "created_at"),
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelName:     getString(in.Row, "extractor"),
+		ModelRole:     distillation.RoleExtractor,
+		ModelProvider: "ollama",
+		Text:          getString(in.Row, "text"),
+	}
+	return &rec
+}
+
+// contractAnalysesTransform: per-permit executor with observer signals,
+// retrieval telemetry, and cost in micro-units that gets converted to
+// USD. Carries `contractor` in metadata.
+func contractAnalysesTransform(in TransformInput) *distillation.EvidenceRecord {
+	permitID := getString(in.Row, "permit_id")
+	tsStr := getString(in.Row, "ts")
+	tsMs := timeToMS(tsStr)
+
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("contract_analysis:%s:%d", permitID, tsMs),
+		TaskID:        fmt.Sprintf("permit:%s", permitID),
+		Timestamp:     tsStr,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleExecutor,
+		Text:          getString(in.Row, "analysis"),
+	}
+
+	if rc := buildRetrievedContext(map[string]any{
+		"matrix_corpora": objectKeys(in.Row, "matrix_corpora"),
+		"matrix_hits":    in.Row["matrix_hits"],
+	}); rc != nil {
+		rec.RetrievedContext = rc
+	}
+
+	if notes := flattenNotes(in.Row, "observer_notes"); len(notes) > 0 {
+		rec.ObserverNotes = notes
+	}
+	if v, ok := in.Row["observer_verdict"].(string); ok && v != "" {
+		rec.ObserverVerdict = distillation.ObserverVerdict(v)
+	}
+	if c, ok := numFloat(in.Row, "observer_conf"); ok {
+		rec.ObserverConfidence = c
+	}
+	if ok, present := boolField(in.Row, "ok"); present && ok {
+		rec.SuccessMarkers = []string{"matrix_hits_above_threshold"}
+	}
+	verdict := getString(in.Row, "observer_verdict")
+	okPresent, _ := boolField(in.Row, "ok")
+	if !okPresent || verdict == "reject" {
+		rec.FailureMarkers = []string{"observer_rejected"}
+	}
+	if cost, ok := numFloat(in.Row, "cost"); ok {
+		rec.CostUSD = cost / 1_000_000.0
+	}
+	if d, ok := numInt(in.Row, "duration_ms"); ok {
+		rec.LatencyMs = d
+	}
+	if contractor := getString(in.Row, "contractor"); contractor != "" {
+		rec.Metadata = map[string]any{"contractor": contractor}
+	}
+	return &rec
+}
+
+// modeExperimentsTransform: mode_runner per-call traces. Provider
+// derived from model name shape ("/" → openrouter, else ollama_cloud).
+func modeExperimentsTransform(in TransformInput) *distillation.EvidenceRecord {
+	tsStr := getString(in.Row, "ts")
+	tsMs := timeToMS(tsStr)
+	filePath := getString(in.Row, "file_path")
+	keySuffix := filePath
+	if keySuffix == "" {
+		keySuffix = fmt.Sprintf("%d", in.LineOffset)
+	}
+	model := getString(in.Row, "model")
+	provider := "ollama_cloud"
+	if strings.Contains(model, "/") {
+		provider = "openrouter"
+	}
+
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("mode_exec:%d:%s", tsMs, keySuffix),
+		TaskID:        getString(in.Row, "task_class"),
+		Timestamp:     tsStr,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelName:     model,
+		ModelRole:     distillation.RoleExecutor,
+		ModelProvider: provider,
+		Text:          getString(in.Row, "response"),
+	}
+	if d, ok := numInt(in.Row, "latency_ms"); ok {
+		rec.LatencyMs = d
+	}
+	if filePath != "" {
+		rec.SourceFiles = []string{filePath}
+	}
+	if sources, ok := in.Row["sources"].(map[string]any); ok {
+		rec.RetrievedContext = buildRetrievedContext(map[string]any{
+			"matrix_corpora":            sources["matrix_corpus"],
+			"matrix_chunks_kept":        sources["matrix_chunks_kept"],
+			"matrix_chunks_dropped":     sources["matrix_chunks_dropped"],
+			"pathway_fingerprints_seen": sources["bug_fingerprints_count"],
+		})
+	}
+	return &rec
+}
+
+// scrumReviewsTransform: per-file scrum review traces. Success marker
+// captures the attempt number when accepted.
+func scrumReviewsTransform(in TransformInput) *distillation.EvidenceRecord {
+	reviewedAt := getString(in.Row, "reviewed_at")
+	tsMs := timeToMS(reviewedAt)
+	file := getString(in.Row, "file")
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("scrum:%d:%s", tsMs, file),
+		TaskID:        fmt.Sprintf("scrum_review:%s", file),
+		Timestamp:     reviewedAt,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelName:     getString(in.Row, "accepted_model"),
+		ModelRole:     distillation.RoleExecutor,
+		Text:          getString(in.Row, "suggestions_preview"),
+	}
+	if file != "" {
+		rec.SourceFiles = []string{file}
+	}
+	if a, ok := numInt(in.Row, "accepted_on_attempt"); ok && a > 0 {
+		rec.SuccessMarkers = []string{fmt.Sprintf("accepted_on_attempt_%d", a)}
+	}
+	return &rec
+}
+
+// observerEscalationsTransform: reviewer-class trace; carries token
+// counts so the SFT exporter sees real usage signals.
+func observerEscalationsTransform(in TransformInput) *distillation.EvidenceRecord {
+	tsStr := getString(in.Row, "ts")
+	tsMs := timeToMS(tsStr)
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("obs_esc:%d:%s", tsMs, getString(in.Row, "sig_hash")),
+		TaskID:        fmt.Sprintf("observer_escalation:%s", strDefault(in.Row, "cluster_endpoint", "?")),
+		Timestamp:     tsStr,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleReviewer,
+		Text:          getString(in.Row, "analysis"),
+	}
+	if pt, ok := numInt(in.Row, "prompt_tokens"); ok {
+		rec.PromptTokens = pt
+	}
+	if ct, ok := numInt(in.Row, "completion_tokens"); ok {
+		rec.CompletionTokens = ct
+	}
+	return &rec
+}
+
+// auditFactsTransform: per-PR auditor extraction. Text is a compact
+// JSON summary of array lengths (facts/entities/relationships).
+func auditFactsTransform(in TransformInput) *distillation.EvidenceRecord {
+	headSHA := getString(in.Row, "head_sha")
+	prNumber := getString(in.Row, "pr_number")
+	body, _ := json.Marshal(map[string]any{
+		"facts":         arrayLen(in.Row, "facts"),
+		"entities":      arrayLen(in.Row, "entities"),
+		"relationships": arrayLen(in.Row, "relationships"),
+	})
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("audit_facts:%s:%d", headSHA, in.LineOffset),
+		TaskID:        fmt.Sprintf("pr:%s", prNumber),
+		Timestamp:     getString(in.Row, "extracted_at"),
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelName:     getString(in.Row, "extractor"),
+		ModelRole:     distillation.RoleExtractor,
+		Text:          string(body),
+	}
+	return &rec
+}
+
+// autoApplyTransform: applier traces. Pure metadata — no text payload.
+// Deterministic ts fallback to RecordedAt when the row lacks one
+// (matches TS comment about wall-clock leak fix).
+func autoApplyTransform(in TransformInput) *distillation.EvidenceRecord {
+	ts := getString(in.Row, "ts")
+	if ts == "" {
+		ts = in.RecordedAt
+	}
+	tsMs := timeToMS(ts)
+	action := strDefault(in.Row, "action", "unknown")
+	file := getString(in.Row, "file")
+	keySuffix := file
+	if keySuffix == "" {
+		keySuffix = fmt.Sprintf("%d", in.LineOffset)
+	}
+
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("auto_apply:%d:%s", tsMs, keySuffix),
+		TaskID:        fmt.Sprintf("auto_apply:%s", strDefault(in.Row, "file", "?")),
+		Timestamp:     ts,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleApplier,
+	}
+	if file != "" {
+		rec.SourceFiles = []string{file}
+	}
+	if action == "committed" {
+		rec.SuccessMarkers = []string{"committed"}
+	}
+	if strings.Contains(action, "reverted") {
+		rec.FailureMarkers = []string{action}
+	}
+	return &rec
+}
+
+// observerReviewsTransform: reviewer-class. Falls back from `ts` to
+// `reviewed_at`. Mirrors observer_escalations but carries verdict +
+// confidence + free-form notes.
+func observerReviewsTransform(in TransformInput) *distillation.EvidenceRecord {
+	ts := getString(in.Row, "ts")
+	if ts == "" {
+		ts = getString(in.Row, "reviewed_at")
+	}
+	tsMs := timeToMS(ts)
+	file := getString(in.Row, "file")
+
+	keySuffix := file
+	if keySuffix == "" {
+		keySuffix = fmt.Sprintf("%d", in.LineOffset)
+	}
+	taskID := fmt.Sprintf("observer_review:%s", keySuffix)
+	if file == "" {
+		taskID = fmt.Sprintf("observer_review:%d", in.LineOffset)
+	}
+
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("obs_rev:%d:%s", tsMs, keySuffix),
+		TaskID:        taskID,
+		Timestamp:     ts,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleReviewer,
+	}
+	if v, ok := in.Row["verdict"].(string); ok && v != "" {
+		rec.ObserverVerdict = distillation.ObserverVerdict(v)
+	}
+	if c, ok := numFloat(in.Row, "confidence"); ok {
+		rec.ObserverConfidence = c
+	}
+	if notes := flattenNotes(in.Row, "notes"); len(notes) > 0 {
+		rec.ObserverNotes = notes
+	}
+	if text := getString(in.Row, "notes"); text != "" {
+		rec.Text = text
+	} else if review := getString(in.Row, "review"); review != "" {
+		rec.Text = review
+	}
+	return &rec
+}
+
+// auditsTransform: per-finding auditor stream. Severity drives the
+// success/failure marker shape — info/low → success, medium →
+// non-fatal failure, high/critical → blocking failure.
+//
+// Note on determinism: the TS port falls back to `new Date().toISOString()`
+// when `ts` is missing, which is non-deterministic. The Go port uses
+// RecordedAt as the deterministic fallback (matches the
+// auto_apply fix pattern).
+func auditsTransform(in TransformInput) *distillation.EvidenceRecord {
+	sev := strings.ToLower(strDefault(in.Row, "severity", "unknown"))
+	minor := sev == "info" || sev == "low"
+	blocking := sev == "high" || sev == "critical"
+	medium := sev == "medium"
+
+	findingID := getString(in.Row, "finding_id")
+	keySuffix := findingID
+	if keySuffix == "" {
+		keySuffix = fmt.Sprintf("%d", in.LineOffset)
+	}
+	phase := getString(in.Row, "phase")
+	taskID := "audit_finding"
+	if phase != "" {
+		taskID = fmt.Sprintf("phase:%s", phase)
+	}
+
+	ts := getString(in.Row, "ts")
+	if ts == "" {
+		ts = in.RecordedAt
+	}
+
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("audit_finding:%s", keySuffix),
+		TaskID:        taskID,
+		Timestamp:     ts,
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleReviewer,
+	}
+	if minor {
+		rec.SuccessMarkers = []string{fmt.Sprintf("audit_severity_%s", sev)}
+	}
+	if blocking {
+		rec.FailureMarkers = []string{fmt.Sprintf("audit_severity_%s", sev)}
+	} else if medium {
+		rec.FailureMarkers = []string{"audit_severity_medium"}
+	}
+	if ev, ok := in.Row["evidence"].(string); ok && ev != "" {
+		rec.Text = ev
+	} else {
+		rec.Text = getString(in.Row, "resolution")
+	}
+	return &rec
+}
+
+// outcomesTransform: command-runner outcome stream. Latency from
+// elapsed_secs (× 1000), success when all events ok.
+func outcomesTransform(in TransformInput) *distillation.EvidenceRecord {
+	rec := distillation.EvidenceRecord{
+		RunID:         fmt.Sprintf("outcome:%s", strDefault(in.Row, "run_id", fmt.Sprintf("%d", in.LineOffset))),
+		Timestamp:     getString(in.Row, "created_at"),
+		SchemaVersion: distillation.EvidenceSchemaVersion,
+		Provenance:    provenance(in),
+		ModelRole:     distillation.RoleExecutor,
+	}
+	if sigHash := getString(in.Row, "sig_hash"); sigHash != "" {
+		rec.TaskID = fmt.Sprintf("outcome_sig:%s", sigHash)
+	} else {
+		rec.TaskID = fmt.Sprintf("outcome:%d", in.LineOffset)
+	}
+	if elapsed, ok := numFloat(in.Row, "elapsed_secs"); ok {
+		rec.LatencyMs = int64(elapsed*1000 + 0.5) // rounded
+	}
+	if okEv, ok1 := numInt(in.Row, "ok_events"); ok1 {
+		if total, ok2 := numInt(in.Row, "total_events"); ok2 {
+			if total > 0 && okEv == total {
+				rec.SuccessMarkers = []string{"all_events_ok"}
+			}
+		}
+	}
+	if g, ok := numInt(in.Row, "total_gap_signals"); ok {
+		vr := map[string]any{"gap_signals": g}
+		if c, ok2 := numInt(in.Row, "total_citations"); ok2 {
+			vr["citation_count"] = c
+		}
+		rec.ValidationResults = vr
+	}
+	return &rec
+}
+
+// ─── Helpers — coercion + extraction patterns shared by transforms ──
+
+func provenance(in TransformInput) distillation.Provenance {
+	return distillation.Provenance{
+		SourceFile: in.SourceFileRelPath,
+		LineOffset: in.LineOffset,
+		SigHash:    in.SigHash,
+		RecordedAt: in.RecordedAt,
+	}
+}
+
+// stemFor extracts "distilled_facts" from "data/_kb/distilled_facts.jsonl".
+func stemFor(relpath string) string {
+	idx := strings.LastIndex(relpath, "/")
+	base := relpath
+	if idx >= 0 {
+		base = relpath[idx+1:]
+	}
+	return strings.TrimSuffix(base, ".jsonl")
+}
+
+// getString returns row[key] as a string, or "" if missing/wrong-type.
+func getString(row map[string]any, key string) string {
+	v, ok := row[key]
+	if !ok || v == nil {
+		return ""
+	}
+	switch t := v.(type) {
+	case string:
+		return t
+	case float64:
+		return fmt.Sprintf("%v", t)
+	case bool:
+		return fmt.Sprintf("%t", t)
+	default:
+		return fmt.Sprintf("%v", t)
+	}
+}
+
+// strDefault returns row[key] coerced to string, or fallback if empty/missing.
+func strDefault(row map[string]any, key, fallback string) string {
+	if s := getString(row, key); s != "" {
+		return s
+	}
+	return fallback
+}
+
+// numInt returns row[key] as int64. JSON numbers come in as float64.
+// Returns (val, true) when present and finite, else (0, false).
+func numInt(row map[string]any, key string) (int64, bool) {
+	v, ok := row[key]
+	if !ok || v == nil {
+		return 0, false
+	}
+	switch t := v.(type) {
+	case float64:
+		return int64(t), true
+	case int:
+		return int64(t), true
+	case int64:
+		return t, true
+	}
+	return 0, false
+}
+
+// numFloat returns row[key] as float64.
+func numFloat(row map[string]any, key string) (float64, bool) {
+	v, ok := row[key]
+	if !ok || v == nil {
+		return 0, false
+	}
+	switch t := v.(type) {
+	case float64:
+		return t, true
+	case int:
+		return float64(t), true
+	case int64:
+		return float64(t), true
+	}
+	return 0, false
+}
+
+// boolField returns (value, present). present=false when key missing
+// or non-bool.
+func boolField(row map[string]any, key string) (bool, bool) {
+	v, ok := row[key]
+	if !ok {
+		return false, false
+	}
+	if b, isBool := v.(bool); isBool {
+		return b, true
+	}
+	return false, false
+}
+
+// arrayLen returns len(row[key]) if it's an array, else 0.
+func arrayLen(row map[string]any, key string) int {
+	if a, ok := row[key].([]any); ok {
+		return len(a)
+	}
+	return 0
+}
+
+// objectKeys returns sorted keys of row[key] when it's a map. Returns
+// nil when missing or non-map (so callers can treat empty corpus list
+// as "field absent").
+func objectKeys(row map[string]any, key string) []string {
+	m, ok := row[key].(map[string]any)
+	if !ok || len(m) == 0 {
+		return nil
+	}
+	keys := make([]string, 0, len(m))
+	for k := range m {
+		keys = append(keys, k)
+	}
+	// Sort for determinism — TS Object.keys() order is insertion-order
+	// in modern engines but Go map iteration is randomized.
+	sortInPlace(keys)
+	return keys
+}
+
+// flattenNotes coerces row[key] from string OR []string into a clean
+// non-empty []string. TS form `[x].flat().filter(Boolean)` — Go does
+// it explicitly.
+func flattenNotes(row map[string]any, key string) []string {
+	v, ok := row[key]
+	if !ok || v == nil {
+		return nil
+	}
+	switch t := v.(type) {
+	case string:
+		if t == "" {
+			return nil
+		}
+		return []string{t}
+	case []any:
+		out := make([]string, 0, len(t))
+		for _, e := range t {
+			if s, ok := e.(string); ok && s != "" {
+				out = append(out, s)
+			}
+		}
+		if len(out) == 0 {
+			return nil
+		}
+		return out
+	}
+	return nil
+}
+
+// timeToMS parses an ISO 8601 string and returns milliseconds since
+// epoch, matching TS `new Date(iso).getTime()`. Returns 0 on parse
+// failure (matches TS NaN coerced to 0 by Number(...) in run_id paths,
+// although there it'd produce "NaN" — the Go behavior is more useful).
+func timeToMS(iso string) int64 {
+	if iso == "" {
+		return 0
+	}
+	for _, layout := range []string{time.RFC3339Nano, time.RFC3339} {
+		if t, err := time.Parse(layout, iso); err == nil {
+			return t.UnixMilli()
+		}
+	}
+	return 0
+}
+
+// buildRetrievedContext assembles RetrievedContext from a flat map of
+// already-coerced fields. Returns nil when nothing meaningful is set,
+// so transforms can attach the field conditionally without wrapping
+// the call site.
+func buildRetrievedContext(fields map[string]any) *distillation.RetrievedContext {
+	rc := distillation.RetrievedContext{}
+	any := false
+	if v, ok := fields["matrix_corpora"].([]string); ok && len(v) > 0 {
+		rc.MatrixCorpora = v
+		any = true
+	}
+	if v, ok := numFromAny(fields["matrix_hits"]); ok {
+		rc.MatrixHits = int(v)
+		any = true
+	}
+	if v, ok := numFromAny(fields["matrix_chunks_kept"]); ok {
+		rc.MatrixChunksKept = int(v)
+		any = true
+	}
+	if v, ok := numFromAny(fields["matrix_chunks_dropped"]); ok {
+		rc.MatrixChunksDropped = int(v)
+		any = true
+	}
+	if v, ok := numFromAny(fields["pathway_fingerprints_seen"]); ok {
+		rc.PathwayFingerprintsSeen = int(v)
+		any = true
+	}
+	if !any {
+		return nil
+	}
+	return &rc
+}
+
+func numFromAny(v any) (float64, bool) {
+	if v == nil {
+		return 0, false
+	}
+	switch t := v.(type) {
+	case float64:
+		return t, true
+	case int:
+		return float64(t), true
+	case int64:
+		return float64(t), true
+	}
+	return 0, false
+}
+
+func sortInPlace(s []string) {
+	// Tiny insertion sort — corpus lists are typically <10 entries.
+	for i := 1; i < len(s); i++ {
+		for j := i; j > 0 && s[j-1] > s[j]; j-- {
+			s[j-1], s[j] = s[j], s[j-1]
+		}
+	}
+}

internal/materializer/transforms_test.go

@@ -0,0 +1,287 @@
+package materializer
+
+import (
+	"encoding/json"
+	"testing"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+const fixedRecordedAt = "2026-05-02T00:00:00Z"
+const fixedSigHash = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
+
+func ti(row map[string]any, source string, lineOffset int64) TransformInput {
+	return TransformInput{
+		Row:               row,
+		LineOffset:        lineOffset,
+		SourceFileRelPath: source,
+		RecordedAt:        fixedRecordedAt,
+		SigHash:           fixedSigHash,
+	}
+}
+
+func TestExtractorTransform_DistilledFacts(t *testing.T) {
+	in := ti(map[string]any{
+		"run_id":       "run-1",
+		"source_label": "lab-3",
+		"created_at":   "2026-04-01T00:00:00Z",
+		"extractor":    "qwen3.5:latest",
+		"text":         "Hello.",
+	}, "data/_kb/distilled_facts.jsonl", 0)
+	rec := extractorTransform(in)
+	if rec == nil {
+		t.Fatal("nil record")
+	}
+	if rec.RunID != "run-1" || rec.TaskID != "lab-3" {
+		t.Fatalf("ids: %+v", rec)
+	}
+	if rec.ModelRole != distillation.RoleExtractor {
+		t.Errorf("role=%v, want extractor", rec.ModelRole)
+	}
+	if rec.ModelProvider != "ollama" {
+		t.Errorf("provider=%q, want ollama", rec.ModelProvider)
+	}
+	if rec.Provenance.SigHash != fixedSigHash {
+		t.Errorf("provenance.sig_hash mismatch: %q", rec.Provenance.SigHash)
+	}
+	if rec.Text != "Hello." {
+		t.Errorf("text=%q", rec.Text)
+	}
+}
+
+func TestExtractorTransform_FallbackIDs(t *testing.T) {
+	in := ti(map[string]any{
+		"created_at": "2026-04-01T00:00:00Z",
+		"text":       "row without ids",
+	}, "data/_kb/distilled_procedures.jsonl", 7)
+	rec := extractorTransform(in)
+	if rec.RunID != "distilled_procedures:7" || rec.TaskID != "distilled_procedures:7" {
+		t.Fatalf("fallback ids wrong: %+v", rec)
+	}
+}
+
+func TestContractAnalysesTransform_Fields(t *testing.T) {
+	in := ti(map[string]any{
+		"permit_id":         "P-001",
+		"ts":                "2026-04-26T12:00:00Z",
+		"matrix_corpora":    map[string]any{"workers": 1, "candidates": 1},
+		"matrix_hits":       3.0,
+		"observer_notes":    []any{"good", "spec match"},
+		"observer_verdict":  "accept",
+		"observer_conf":     85.0,
+		"ok":                true,
+		"cost":              2_500_000.0, // micro-units
+		"duration_ms":       1234.0,
+		"contractor":        "Acme",
+		"analysis":          "Looks good.",
+	}, "data/_kb/contract_analyses.jsonl", 0)
+	rec := contractAnalysesTransform(in)
+	if rec.RunID == "" || rec.TaskID != "permit:P-001" {
+		t.Fatalf("ids: %+v", rec)
+	}
+	if rec.ModelRole != distillation.RoleExecutor {
+		t.Errorf("role=%v", rec.ModelRole)
+	}
+	if rec.RetrievedContext == nil || len(rec.RetrievedContext.MatrixCorpora) != 2 || rec.RetrievedContext.MatrixHits != 3 {
+		t.Errorf("retrieved_context wrong: %+v", rec.RetrievedContext)
+	}
+	if len(rec.ObserverNotes) != 2 {
+		t.Errorf("observer_notes=%v", rec.ObserverNotes)
+	}
+	if string(rec.ObserverVerdict) != "accept" || rec.ObserverConfidence != 85 {
+		t.Errorf("observer fields: %+v", rec)
+	}
+	if rec.CostUSD != 2.5 {
+		t.Errorf("cost should convert micro→USD; got %v", rec.CostUSD)
+	}
+	if rec.LatencyMs != 1234 {
+		t.Errorf("latency: %v", rec.LatencyMs)
+	}
+	if rec.Metadata == nil || rec.Metadata["contractor"] != "Acme" {
+		t.Errorf("metadata.contractor missing: %v", rec.Metadata)
+	}
+	if len(rec.SuccessMarkers) != 1 || rec.SuccessMarkers[0] != "matrix_hits_above_threshold" {
+		t.Errorf("success_markers: %v", rec.SuccessMarkers)
+	}
+	if len(rec.FailureMarkers) != 0 {
+		t.Errorf("expected no failure_markers when ok=true and verdict=accept, got %v", rec.FailureMarkers)
+	}
+}
+
+func TestContractAnalysesTransform_FailureMarkers(t *testing.T) {
+	in := ti(map[string]any{
+		"permit_id":        "P-002",
+		"ts":               "2026-04-26T12:00:00Z",
+		"observer_verdict": "reject",
+		"ok":               false,
+		"analysis":         "Issues found.",
+	}, "data/_kb/contract_analyses.jsonl", 1)
+	rec := contractAnalysesTransform(in)
+	if len(rec.FailureMarkers) != 1 || rec.FailureMarkers[0] != "observer_rejected" {
+		t.Errorf("failure_markers: %v", rec.FailureMarkers)
+	}
+}
+
+func TestModeExperimentsTransform_ProviderInference(t *testing.T) {
+	openrouter := ti(map[string]any{
+		"ts":         "2026-04-26T12:00:00Z",
+		"task_class": "scrum_review",
+		"model":      "anthropic/claude-opus-4-7",
+		"file_path":  "src/foo.rs",
+		"sources":    map[string]any{"matrix_corpus": []any{"docs"}, "matrix_chunks_kept": 4.0},
+		"latency_ms": 200.0,
+		"response":   "ok",
+	}, "data/_kb/mode_experiments.jsonl", 0)
+	rec := modeExperimentsTransform(openrouter)
+	if rec.ModelProvider != "openrouter" {
+		t.Errorf("provider=%q, want openrouter", rec.ModelProvider)
+	}
+
+	cloud := ti(map[string]any{
+		"ts":         "2026-04-26T12:00:00Z",
+		"task_class": "scrum_review",
+		"model":      "qwen3-coder:480b",
+		"sources":    map[string]any{"matrix_corpus": []any{"docs"}},
+		"response":   "ok",
+	}, "data/_kb/mode_experiments.jsonl", 1)
+	rec2 := modeExperimentsTransform(cloud)
+	if rec2.ModelProvider != "ollama_cloud" {
+		t.Errorf("provider=%q, want ollama_cloud", rec2.ModelProvider)
+	}
+	if len(rec2.SourceFiles) != 0 {
+		t.Errorf("source_files should be empty when file_path missing; got %v", rec2.SourceFiles)
+	}
+}
+
+func TestObserverEscalationsTransform_Tokens(t *testing.T) {
+	in := ti(map[string]any{
+		"ts":                "2026-04-26T12:00:00Z",
+		"sig_hash":          "abc",
+		"cluster_endpoint":  "/v1/chat",
+		"prompt_tokens":     100.0,
+		"completion_tokens": 50.0,
+		"analysis":          "review",
+	}, "data/_kb/observer_escalations.jsonl", 0)
+	rec := observerEscalationsTransform(in)
+	if rec.PromptTokens != 100 || rec.CompletionTokens != 50 {
+		t.Errorf("tokens: prompt=%d completion=%d", rec.PromptTokens, rec.CompletionTokens)
+	}
+	if rec.TaskID != "observer_escalation:/v1/chat" {
+		t.Errorf("task_id=%q", rec.TaskID)
+	}
+}
+
+func TestAuditFactsTransform_TextIsSummary(t *testing.T) {
+	in := ti(map[string]any{
+		"head_sha":      "abc123",
+		"pr_number":     11.0,
+		"extracted_at":  "2026-04-26T12:00:00Z",
+		"extractor":     "qwen2.5",
+		"facts":         []any{"f1", "f2"},
+		"entities":      []any{"e1"},
+		"relationships": []any{},
+	}, "data/_kb/audit_facts.jsonl", 0)
+	rec := auditFactsTransform(in)
+	var summary map[string]any
+	if err := json.Unmarshal([]byte(rec.Text), &summary); err != nil {
+		t.Fatalf("text not JSON: %v", err)
+	}
+	if summary["facts"].(float64) != 2 || summary["entities"].(float64) != 1 || summary["relationships"].(float64) != 0 {
+		t.Errorf("counts wrong: %+v", summary)
+	}
+}
+
+func TestAutoApplyTransform_DeterministicTimestampFallback(t *testing.T) {
+	in := ti(map[string]any{
+		"action": "committed",
+		"file":   "src/x.rs",
+	}, "data/_kb/auto_apply.jsonl", 0)
+	rec := autoApplyTransform(in)
+	if rec.Timestamp != fixedRecordedAt {
+		t.Errorf("expected fallback to RecordedAt %q, got %q", fixedRecordedAt, rec.Timestamp)
+	}
+	if len(rec.SuccessMarkers) != 1 || rec.SuccessMarkers[0] != "committed" {
+		t.Errorf("success_markers: %v", rec.SuccessMarkers)
+	}
+
+	revertedIn := ti(map[string]any{
+		"ts":     "2026-04-26T12:00:00Z",
+		"action": "auto_reverted_after_test_fail",
+		"file":   "src/x.rs",
+	}, "data/_kb/auto_apply.jsonl", 1)
+	rec2 := autoApplyTransform(revertedIn)
+	if len(rec2.FailureMarkers) != 1 || rec2.FailureMarkers[0] != "auto_reverted_after_test_fail" {
+		t.Errorf("failure_markers: %v", rec2.FailureMarkers)
+	}
+}
+
+func TestAuditsTransform_SeverityRouting(t *testing.T) {
+	cases := []struct {
+		sev      string
+		success  bool
+		blocking bool
+		medium   bool
+	}{
+		{"info", true, false, false},
+		{"low", true, false, false},
+		{"medium", false, false, true},
+		{"high", false, true, false},
+		{"critical", false, true, false},
+	}
+	for _, c := range cases {
+		t.Run(c.sev, func(t *testing.T) {
+			in := ti(map[string]any{
+				"finding_id": "F-1",
+				"phase":      "G2",
+				"severity":   c.sev,
+				"ts":         "2026-04-26T12:00:00Z",
+				"evidence":   "details",
+			}, "data/_kb/audits.jsonl", 0)
+			rec := auditsTransform(in)
+			hasSuccess := len(rec.SuccessMarkers) > 0
+			hasFailure := len(rec.FailureMarkers) > 0
+			if hasSuccess != c.success {
+				t.Errorf("severity=%s success=%v wanted %v", c.sev, hasSuccess, c.success)
+			}
+			if hasFailure != (c.blocking || c.medium) {
+				t.Errorf("severity=%s failure=%v wanted %v", c.sev, hasFailure, c.blocking || c.medium)
+			}
+		})
+	}
+}
+
+func TestOutcomesTransform_LatencyAndSuccess(t *testing.T) {
+	in := ti(map[string]any{
+		"run_id":            "r-1",
+		"created_at":        "2026-04-26T12:00:00Z",
+		"sig_hash":          "abc",
+		"elapsed_secs":      1.234,
+		"ok_events":         5.0,
+		"total_events":      5.0,
+		"total_gap_signals": 2.0,
+		"total_citations":   3.0,
+	}, "data/_kb/outcomes.jsonl", 0)
+	rec := outcomesTransform(in)
+	if rec.LatencyMs != 1234 {
+		t.Errorf("latency=%d", rec.LatencyMs)
+	}
+	if len(rec.SuccessMarkers) != 1 || rec.SuccessMarkers[0] != "all_events_ok" {
+		t.Errorf("success: %v", rec.SuccessMarkers)
+	}
+	if g, ok := rec.ValidationResults["gap_signals"].(int64); !ok || g != 2 {
+		t.Errorf("gap_signals: %v", rec.ValidationResults)
+	}
+	if c, ok := rec.ValidationResults["citation_count"].(int64); !ok || c != 3 {
+		t.Errorf("citation_count: %v", rec.ValidationResults)
+	}
+}
+
+func TestTransformByPath_Found(t *testing.T) {
+	td := TransformByPath("data/_kb/distilled_facts.jsonl")
+	if td == nil {
+		t.Fatal("expected to find distilled_facts transform")
+	}
+	if TransformByPath("data/_kb/never_existed.jsonl") != nil {
+		t.Fatal("expected nil for unknown path")
+	}
+}

internal/materializer/validate.go

@@ -0,0 +1,131 @@
+package materializer
+
+import (
+	"fmt"
+	"regexp"
+	"strings"
+	"time"
+
+	"git.agentview.dev/profit/golangLAKEHOUSE/internal/distillation"
+)
+
+// ValidateEvidenceRecord ports validateEvidenceRecord from
+// auditor/schemas/distillation/evidence_record.ts. Returns nil on
+// success or a slice of human-readable error messages — the
+// materializer logs the slice into distillation_skips.jsonl so an
+// operator can see why a row was rejected without diff'ing logic.
+//
+// The validator is intentionally separate from
+// distillation.ValidateScoredRun: scoring runs and evidence records
+// have different shapes and the scorer's validator only covers the
+// scored-run side.
+func ValidateEvidenceRecord(r distillation.EvidenceRecord) []string {
+	var errs []string
+
+	if r.RunID == "" {
+		errs = append(errs, "run_id: must be non-empty")
+	}
+	if r.TaskID == "" {
+		errs = append(errs, "task_id: must be non-empty")
+	}
+	if !validISOTimestamp(r.Timestamp) {
+		errs = append(errs, fmt.Sprintf("timestamp: not a valid ISO 8601 timestamp: %s", trim(r.Timestamp, 60)))
+	}
+	if r.SchemaVersion != distillation.EvidenceSchemaVersion {
+		errs = append(errs, fmt.Sprintf("schema_version: expected %d, got %d", distillation.EvidenceSchemaVersion, r.SchemaVersion))
+	}
+	errs = append(errs, validateProvenanceFields(r.Provenance)...)
+
+	if r.ModelRole != "" && !isValidModelRole(r.ModelRole) {
+		errs = append(errs, fmt.Sprintf("model_role: must be a known role, got %q", r.ModelRole))
+	}
+	if r.InputHash != "" && !isHexSha256(r.InputHash) {
+		errs = append(errs, "input_hash: must be hex sha256 when present")
+	}
+	if r.OutputHash != "" && !isHexSha256(r.OutputHash) {
+		errs = append(errs, "output_hash: must be hex sha256 when present")
+	}
+	if r.ObserverConfidence < 0 || r.ObserverConfidence > 100 {
+		errs = append(errs, "observer_confidence: must be in [0, 100]")
+	}
+	if r.HumanOverride != nil {
+		if r.HumanOverride.Overrider == "" {
+			errs = append(errs, "human_override.overrider: must be non-empty")
+		}
+		if r.HumanOverride.Reason == "" {
+			errs = append(errs, "human_override.reason: must be non-empty")
+		}
+		if !validISOTimestamp(r.HumanOverride.OverriddenAt) {
+			errs = append(errs, "human_override.overridden_at: must be ISO 8601")
+		}
+		switch r.HumanOverride.Decision {
+		case "accept", "reject", "needs_review":
+		default:
+			errs = append(errs, "human_override.decision: must be accept|reject|needs_review")
+		}
+	}
+
+	if len(errs) == 0 {
+		return nil
+	}
+	return errs
+}
+
+func validateProvenanceFields(p distillation.Provenance) []string {
+	var errs []string
+	if p.SourceFile == "" {
+		errs = append(errs, "provenance.source_file: must be non-empty")
+	}
+	if !isHexSha256(p.SigHash) {
+		errs = append(errs, fmt.Sprintf("provenance.sig_hash: not a valid hex sha256: %s", trim(p.SigHash, 80)))
+	}
+	if !validISOTimestamp(p.RecordedAt) {
+		errs = append(errs, "provenance.recorded_at: must be ISO 8601")
+	}
+	return errs
+}
+
+var (
+	// Permissive ISO 8601 (matches TS regex):
+	//   YYYY-MM-DDTHH:MM:SS(.fraction)?(Z|±HH:MM)?
+	isoTimestampRE = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?$`)
+	hexSha256RE    = regexp.MustCompile(`^[0-9a-f]{64}$`)
+)
+
+func validISOTimestamp(s string) bool {
+	if s == "" {
+		return false
+	}
+	if !isoTimestampRE.MatchString(s) {
+		return false
+	}
+	// Belt-and-suspenders: confirm it's actually parseable too.
+	if _, err := time.Parse(time.RFC3339, s); err == nil {
+		return true
+	}
+	if _, err := time.Parse(time.RFC3339Nano, s); err == nil {
+		return true
+	}
+	return false
+}
+
+func isHexSha256(s string) bool {
+	return hexSha256RE.MatchString(s)
+}
+
+func isValidModelRole(role distillation.ModelRole) bool {
+	switch role {
+	case distillation.RoleExecutor, distillation.RoleReviewer, distillation.RoleExtractor,
+		distillation.RoleVerifier, distillation.RoleCategorizer, distillation.RoleTiebreaker,
+		distillation.RoleApplier, distillation.RoleEmbedder, distillation.RoleOther:
+		return true
+	}
+	return false
+}
+
+func trim(s string, n int) string {
+	if len(s) <= n {
+		return s
+	}
+	return strings.ReplaceAll(s[:n], "\n", " ")
+}

internal/matrix/downgrade.go

@@ -0,0 +1,163 @@
+package matrix
+
+// Strong-model auto-downgrade gate. Port of mode.rs::execute's
+// downgrade block (Rust system, 2026-04-26 pass5).
+//
+// What it does: if the caller resolves `codereview_lakehouse` against
+// a strong model and didn't force the mode, flip to
+// `codereview_isolation` so we don't pollute the prompt with matrix
+// chunks the model demonstrably does better without.
+//
+// Why: pass5 variance test on x-ai/grok-4.1-fast — composing matrix
+// corpora into codereview_lakehouse LOST 5/5 head-to-head reps
+// against matrix-free codereview_isolation, p=0.031. Strong models
+// have enough native capacity that bug fingerprints + adversarial
+// framing + file content carry them; matrix chunks displace
+// depth-of-analysis.
+//
+// Defaults: assume "strong" (downgrade matrix off). The explicit
+// IsWeakModel predicate keeps the weak-list small — anything
+// matching `:free` (OpenRouter free tier) or the local last-resort
+// rungs (qwen3.5/qwen3) stays on the full lakehouse path where
+// matrix demonstrably helped during the 2026-04-26 free-tier
+// bake-off.
+
+import (
+	"os"
+	"strings"
+)
+
+// Mode constants — exported so callers don't string-literal them.
+const (
+	ModeCodeReviewLakehouse = "codereview_lakehouse"
+	ModeCodeReviewIsolation = "codereview_isolation"
+)
+
+// EnvForceFullEnrichment is the env var that bypasses the gate for
+// diagnostic runs ("LH_FORCE_FULL_ENRICHMENT=1" or "true").
+const EnvForceFullEnrichment = "LH_FORCE_FULL_ENRICHMENT"
+
+// DefaultWeakModels is the fallback weak-model list used when no
+// config-driven list is provided. Matches the pre-config hardcoded
+// switch — preserved for backward compatibility with callers that
+// haven't migrated to the [models] tier config (Phase 1).
+var DefaultWeakModels = []string{"qwen3.5:latest", "qwen3:latest"}
+
+// IsWeakModel returns true for models matrix-corpus composition
+// demonstrably helped during the 2026-04-26 pass5 bake-off. Strong
+// models (default) get matrix dropped to avoid the "composed lost
+// 5/5 vs isolation" effect.
+//
+// Weak signals:
+//   - `:free` suffix (OpenRouter free tier, e.g. `gpt-oss-120b:free`)
+//   - `:free/` infix (handles routing-prefixed names like `or:free/x`)
+//   - DefaultWeakModels list — local-hot-path eligible.
+//
+// For config-driven lists, use IsWeakModelInList.
+func IsWeakModel(model string) bool {
+	return IsWeakModelInList(model, DefaultWeakModels)
+}
+
+// IsWeakModelInList is the config-aware variant. Callers pass the
+// configured weak-model list (typically cfg.Models.WeakModels) and
+// the function applies the same rule-based checks (`:free` suffix /
+// infix) plus the literal list match.
+//
+// Empty list still applies the rule-based checks — only the literal
+// match becomes a no-op. That preserves "free-tier always weak"
+// regardless of what the operator configured.
+func IsWeakModelInList(model string, weakList []string) bool {
+	if strings.HasSuffix(model, ":free") || strings.Contains(model, ":free/") {
+		return true
+	}
+	for _, w := range weakList {
+		if w == model {
+			return true
+		}
+	}
+	return false
+}
+
+// DowngradeInput is what MaybeDowngrade evaluates.
+//
+// ForcedMode: caller explicitly set their mode (mirrors Rust's
+// req.force_mode.is_some()) — treated as opt-in to the chosen mode,
+// skips the downgrade. Experiments need exact-mode control.
+//
+// ForceFullOverride: the LH_FORCE_FULL_ENRICHMENT escape hatch —
+// usually populated from the env var via NewDowngradeInputFromEnv,
+// but the field is explicit so callers can pass it from a config or
+// test deterministically.
+type DowngradeInput struct {
+	Mode              string
+	Model             string
+	ForcedMode        bool
+	ForceFullOverride bool
+	// WeakModels overrides the default weak-model list (DefaultWeakModels)
+	// when non-nil. Typically populated from cfg.Models.WeakModels at
+	// startup. nil = use DefaultWeakModels (backward compat).
+	WeakModels []string
+}
+
+// DowngradeDecision is the output. DowngradedFrom is non-empty
+// only when a downgrade fired — callers should record it for audit
+// (matches the Rust EnrichmentSources.downgraded_from field).
+//
+// Reason is a short human-readable string for logs/responses;
+// useful for debugging "why did/didn't the gate fire."
+type DowngradeDecision struct {
+	Mode           string `json:"mode"`
+	DowngradedFrom string `json:"downgraded_from,omitempty"`
+	Reason         string `json:"reason"`
+}
+
+// MaybeDowngrade applies the strong-model auto-downgrade gate.
+// Pure function; no env reads. For env-driven callers see
+// NewDowngradeInputFromEnv.
+func MaybeDowngrade(in DowngradeInput) DowngradeDecision {
+	out := DowngradeDecision{Mode: in.Mode}
+	if in.Mode != ModeCodeReviewLakehouse {
+		out.Reason = "mode is not " + ModeCodeReviewLakehouse + "; gate not applicable"
+		return out
+	}
+	if in.ForcedMode {
+		out.Reason = "caller forced mode; skip downgrade"
+		return out
+	}
+	if in.ForceFullOverride {
+		out.Reason = EnvForceFullEnrichment + " bypass"
+		return out
+	}
+	weakList := in.WeakModels
+	if weakList == nil {
+		weakList = DefaultWeakModels
+	}
+	if IsWeakModelInList(in.Model, weakList) {
+		out.Reason = "weak model; matrix composition demonstrably helped (2026-04-26 free-tier bake-off)"
+		return out
+	}
+	// Downgrade fires.
+	out.Mode = ModeCodeReviewIsolation
+	out.DowngradedFrom = ModeCodeReviewLakehouse
+	out.Reason = "strong model; matrix composes anti-additively (pass5: composed lost 5/5 vs isolation on grok-4.1-fast, p=0.031)"
+	return out
+}
+
+// NewDowngradeInputFromEnv is a convenience that reads
+// LH_FORCE_FULL_ENRICHMENT from the process environment and returns
+// a populated DowngradeInput. Most production callers want this;
+// tests should construct DowngradeInput directly to avoid env
+// pollution.
+func NewDowngradeInputFromEnv(mode, model string, forcedMode bool) DowngradeInput {
+	return DowngradeInput{
+		Mode:              mode,
+		Model:             model,
+		ForcedMode:        forcedMode,
+		ForceFullOverride: envForceFullEnrichment(),
+	}
+}
+
+func envForceFullEnrichment() bool {
+	v := strings.ToLower(strings.TrimSpace(os.Getenv(EnvForceFullEnrichment)))
+	return v == "1" || v == "true"
+}

internal/matrix/downgrade_test.go

@@ -0,0 +1,162 @@
+package matrix
+
+import "testing"
+
+func TestIsWeakModel(t *testing.T) {
+	cases := []struct {
+		model string
+		weak  bool
+	}{
+		// :free suffix → weak
+		{"openai/gpt-4o:free", true},
+		{"meta-llama/llama-3-8b:free", true},
+		// :free/ infix (routing-prefixed names)
+		{"openrouter:free/anthropic/claude-3.5-sonnet", true},
+		// Local last-resort rungs
+		{"qwen3.5:latest", true},
+		{"qwen3:latest", true},
+		// Strong by default
+		{"x-ai/grok-4.1-fast", false},
+		{"opencode/claude-opus-4-7", false},
+		{"openai/gpt-5", false},
+		{"qwen3-coder:480b", false}, // not the :latest tag
+		{"", false},
+	}
+	for _, c := range cases {
+		got := IsWeakModel(c.model)
+		if got != c.weak {
+			t.Errorf("IsWeakModel(%q): want %v, got %v", c.model, c.weak, got)
+		}
+	}
+}
+
+func TestMaybeDowngrade_TruthTable(t *testing.T) {
+	cases := []struct {
+		name string
+		in   DowngradeInput
+		want DowngradeDecision
+	}{
+		{
+			name: "downgrade fires: lakehouse mode + strong model + no force",
+			in:   DowngradeInput{Mode: ModeCodeReviewLakehouse, Model: "x-ai/grok-4.1-fast"},
+			want: DowngradeDecision{
+				Mode:           ModeCodeReviewIsolation,
+				DowngradedFrom: ModeCodeReviewLakehouse,
+			},
+		},
+		{
+			name: "no downgrade: forced mode bypasses gate",
+			in:   DowngradeInput{Mode: ModeCodeReviewLakehouse, Model: "x-ai/grok-4.1-fast", ForcedMode: true},
+			want: DowngradeDecision{Mode: ModeCodeReviewLakehouse},
+		},
+		{
+			name: "no downgrade: env override bypasses gate",
+			in:   DowngradeInput{Mode: ModeCodeReviewLakehouse, Model: "x-ai/grok-4.1-fast", ForceFullOverride: true},
+			want: DowngradeDecision{Mode: ModeCodeReviewLakehouse},
+		},
+		{
+			name: "no downgrade: weak model keeps lakehouse",
+			in:   DowngradeInput{Mode: ModeCodeReviewLakehouse, Model: "openai/gpt-4o:free"},
+			want: DowngradeDecision{Mode: ModeCodeReviewLakehouse},
+		},
+		{
+			name: "no downgrade: non-lakehouse mode (gate not applicable)",
+			in:   DowngradeInput{Mode: "codereview_isolation", Model: "x-ai/grok-4.1-fast"},
+			want: DowngradeDecision{Mode: "codereview_isolation"},
+		},
+	}
+	for _, c := range cases {
+		got := MaybeDowngrade(c.in)
+		if got.Mode != c.want.Mode {
+			t.Errorf("%s: Mode want %q, got %q", c.name, c.want.Mode, got.Mode)
+		}
+		if got.DowngradedFrom != c.want.DowngradedFrom {
+			t.Errorf("%s: DowngradedFrom want %q, got %q", c.name, c.want.DowngradedFrom, got.DowngradedFrom)
+		}
+		if got.Reason == "" {
+			t.Errorf("%s: Reason should be non-empty", c.name)
+		}
+	}
+}
+
+// TestIsWeakModelInList covers the config-driven variant added in
+// Phase 2 — callers pass cfg.Models.WeakModels instead of relying on
+// the package-level DefaultWeakModels.
+func TestIsWeakModelInList(t *testing.T) {
+	custom := []string{"my-custom-7b:latest", "qwen3:latest"}
+	cases := []struct {
+		model string
+		list  []string
+		weak  bool
+	}{
+		// Custom literal hits (config-driven)
+		{"my-custom-7b:latest", custom, true},
+		{"qwen3:latest", custom, true},
+		// qwen3.5:latest is in DefaultWeakModels but NOT in custom list →
+		// not weak when caller supplies custom list (intentional — operator
+		// owns the list).
+		{"qwen3.5:latest", custom, false},
+		// :free/free-tier rules apply regardless of list
+		{"openai/gpt-4o:free", custom, true},
+		{"openrouter:free/anthropic/claude", custom, true},
+		// Empty list → only rule-based checks apply
+		{"qwen3.5:latest", []string{}, false},
+		{"openai/gpt-4o:free", []string{}, true},
+		// nil list behaves like empty (caller hasn't migrated yet)
+		{"qwen3.5:latest", nil, false},
+	}
+	for _, c := range cases {
+		got := IsWeakModelInList(c.model, c.list)
+		if got != c.weak {
+			t.Errorf("IsWeakModelInList(%q, %v): want %v, got %v", c.model, c.list, c.weak, got)
+		}
+	}
+}
+
+// TestMaybeDowngrade_WithConfigList verifies the DowngradeInput.WeakModels
+// field overrides DefaultWeakModels when populated, so callers reading
+// cfg.Models.WeakModels at startup get the configured behavior end-to-end.
+func TestMaybeDowngrade_WithConfigList(t *testing.T) {
+	// Config says "only my-custom is weak". qwen3.5:latest is now strong
+	// (operator opted out of the local-hot-path treatment).
+	configList := []string{"my-custom:latest"}
+	in := DowngradeInput{
+		Mode:       ModeCodeReviewLakehouse,
+		Model:      "qwen3.5:latest",
+		WeakModels: configList,
+	}
+	got := MaybeDowngrade(in)
+	if got.Mode != ModeCodeReviewIsolation {
+		t.Errorf("qwen3.5:latest with config list excluding it should downgrade; got Mode=%q", got.Mode)
+	}
+	if got.DowngradedFrom != ModeCodeReviewLakehouse {
+		t.Errorf("DowngradedFrom should be lakehouse; got %q", got.DowngradedFrom)
+	}
+
+	// Same input but weak by config — gate must NOT fire.
+	in.Model = "my-custom:latest"
+	got = MaybeDowngrade(in)
+	if got.Mode != ModeCodeReviewLakehouse {
+		t.Errorf("my-custom:latest is in config list — should stay lakehouse; got %q", got.Mode)
+	}
+}
+
+// TestMaybeDowngrade_ForcedTrumpsOthers verifies precedence: when
+// multiple bypass conditions hit, ForcedMode wins (explicit caller
+// intent always overrides). Caught a subtle ordering bug in the
+// original Rust code where this was tested only by happy path.
+func TestMaybeDowngrade_ForcedTrumpsOthers(t *testing.T) {
+	in := DowngradeInput{
+		Mode:              ModeCodeReviewLakehouse,
+		Model:             "qwen3.5:latest", // weak — would otherwise hit weak-bypass
+		ForcedMode:        true,
+		ForceFullOverride: true,
+	}
+	got := MaybeDowngrade(in)
+	if got.Mode != ModeCodeReviewLakehouse {
+		t.Errorf("forced mode should keep mode: got %q", got.Mode)
+	}
+	if got.DowngradedFrom != "" {
+		t.Errorf("no downgrade expected; got DowngradedFrom=%q", got.DowngradedFrom)
+	}
+}

internal/matrix/filter_test.go

@@ -0,0 +1,95 @@
+package matrix
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func TestMatchesMetadataFilter_NoFilter_KeepsAll(t *testing.T) {
+	meta := json.RawMessage(`{"role":"Forklift Operator","state":"IL"}`)
+	if !matchesMetadataFilter(meta, nil) {
+		t.Error("nil filter should match everything")
+	}
+	if !matchesMetadataFilter(meta, map[string]any{}) {
+		t.Error("empty filter should match everything")
+	}
+}
+
+func TestMatchesMetadataFilter_NoMetadata_AlwaysFails(t *testing.T) {
+	if matchesMetadataFilter(nil, map[string]any{"x": "y"}) {
+		t.Error("missing metadata should fail any filter")
+	}
+}
+
+func TestMatchesMetadataFilter_SingleValueExactMatch(t *testing.T) {
+	meta := json.RawMessage(`{"state":"IL","status":"active","years":5}`)
+	cases := []struct {
+		filter map[string]any
+		want   bool
+	}{
+		{map[string]any{"state": "IL"}, true},
+		{map[string]any{"state": "TX"}, false},
+		{map[string]any{"status": "active"}, true},
+		{map[string]any{"status": "inactive"}, false},
+		// JSON normalizes both sides, so 5 matches 5.0
+		{map[string]any{"years": 5.0}, true},
+		{map[string]any{"years": 5}, true},
+		// Missing key = fail
+		{map[string]any{"city": "Chicago"}, false},
+	}
+	for _, c := range cases {
+		got := matchesMetadataFilter(meta, c.filter)
+		if got != c.want {
+			t.Errorf("filter %v on %s: want %v, got %v", c.filter, meta, c.want, got)
+		}
+	}
+}
+
+func TestMatchesMetadataFilter_AllKeysAND(t *testing.T) {
+	meta := json.RawMessage(`{"state":"IL","status":"active","role":"Forklift Operator"}`)
+	if !matchesMetadataFilter(meta, map[string]any{
+		"state":  "IL",
+		"status": "active",
+	}) {
+		t.Error("both keys match: should pass")
+	}
+	if matchesMetadataFilter(meta, map[string]any{
+		"state":  "IL",
+		"status": "inactive", // mismatch
+	}) {
+		t.Error("one key mismatches: should fail (AND across keys)")
+	}
+}
+
+func TestMatchesMetadataFilter_ListValueOR(t *testing.T) {
+	meta := json.RawMessage(`{"state":"IL"}`)
+	// state in {"IL","WI","IN"} → match
+	if !matchesMetadataFilter(meta, map[string]any{
+		"state": []any{"IL", "WI", "IN"},
+	}) {
+		t.Error("list with matching element: should pass")
+	}
+	// state in {"TX","CA"} → fail
+	if matchesMetadataFilter(meta, map[string]any{
+		"state": []any{"TX", "CA"},
+	}) {
+		t.Error("list with no matching element: should fail")
+	}
+}
+
+func TestMatchesMetadataFilter_BoolMatch(t *testing.T) {
+	meta := json.RawMessage(`{"available":true,"placed":false}`)
+	if !matchesMetadataFilter(meta, map[string]any{"available": true}) {
+		t.Error("bool true match")
+	}
+	if matchesMetadataFilter(meta, map[string]any{"available": false}) {
+		t.Error("bool true should not match false filter")
+	}
+}
+
+func TestMatchesMetadataFilter_MalformedMetadataFails(t *testing.T) {
+	meta := json.RawMessage(`{not valid json}`)
+	if matchesMetadataFilter(meta, map[string]any{"x": "y"}) {
+		t.Error("malformed metadata should fail")
+	}
+}

internal/matrix/judge.go

@@ -0,0 +1,152 @@
+package matrix
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// LLMJudgeGate is an InjectGate implementation that uses an Ollama-
+// compatible chat endpoint (or chatd's /v1/chat) to rate the
+// (query, candidate) pair on a 1-5 rubric, then approves the
+// injection iff rating >= MinRating.
+//
+// The HTTP path is intentionally generic — works against any
+// endpoint that speaks Ollama's /api/chat shape: bare Ollama,
+// chatd's /v1/chat, or anything else honoring the same JSON.
+// Per-call timeout is bounded by the parent ctx + the http.Client.
+//
+// Best-effort posture: a judge call that fails (network, JSON
+// decode, anything) returns Approve=false. Same fail-closed default
+// as the inject path's distance gate — when the judge can't speak,
+// don't inject (better silent miss than confident wrong-domain).
+//
+// Usage from retrieve.go:
+//   gate := matrix.NewLLMJudgeGate(req.JudgeURL, req.JudgeModel,
+//       req.JudgeMinRating, hc)
+//   results, injected = matrix.InjectPlaybookMisses(req.QueryText,
+//       results, hits, maxInjectDist, gate)
+type LLMJudgeGate struct {
+	URL        string
+	Model      string
+	MinRating  int
+	HTTPClient *http.Client
+}
+
+// NewLLMJudgeGate is the constructor. Defaults: minRating 3, 10s
+// HTTP timeout. URL must include the path (e.g.
+// "http://localhost:11434/api/chat" for bare Ollama). Returns nil
+// when URL or Model is empty — caller treats nil InjectGate as
+// "no judge configured, default-approve" per InjectPlaybookMisses
+// contract.
+func NewLLMJudgeGate(url, model string, minRating int, hc *http.Client) *LLMJudgeGate {
+	if url == "" || model == "" {
+		return nil
+	}
+	if minRating <= 0 {
+		minRating = 3
+	}
+	if hc == nil {
+		hc = &http.Client{Timeout: 10 * time.Second}
+	}
+	return &LLMJudgeGate{
+		URL:        url,
+		Model:      model,
+		MinRating:  minRating,
+		HTTPClient: hc,
+	}
+}
+
+// Approve calls the LLM judge with a query+candidate prompt; returns
+// true iff the judge's rating meets MinRating. Errors return false
+// (fail-closed — see type doc).
+func (g *LLMJudgeGate) Approve(query string, hit PlaybookHit) bool {
+	if g == nil || query == "" {
+		// No judge or no query to judge against — treat as approve.
+		// Empty-query case mirrors InjectPlaybookMisses' contract:
+		// callers without a query string can't usefully judge.
+		return true
+	}
+	rating := g.rate(query, hit)
+	return rating >= g.MinRating
+}
+
+func (g *LLMJudgeGate) rate(query string, hit PlaybookHit) int {
+	system := `You rate retrieval results for a staffing co-pilot.
+Rate the result 1-5 against the query:
+  5 = perfect match (this person/role IS what was asked for)
+  4 = strong match (right field, right level, minor mismatches)
+  3 = adjacent match (related field or partial overlap)
+  2 = weak/tangential match
+  1 = irrelevant
+Output JSON only: {"rating": N, "reason": "<one sentence>"}.`
+	// We pass the recorded query text + answer ID to give the judge
+	// minimal context. Production might also fetch the answer's
+	// metadata, but that requires a second HTTP hop; the recorded
+	// query is usually enough to sniff wrong-domain matches.
+	user := fmt.Sprintf("Query: %q\n\nCandidate playbook entry:\n  recorded_query: %q\n  answer_id: %s\n  answer_corpus: %s\n  recorded_score: %.2f",
+		query, hit.Entry.QueryText, hit.Entry.AnswerID, hit.Entry.AnswerCorpus, hit.Entry.Score)
+
+	body, _ := json.Marshal(map[string]any{
+		"model":  g.Model,
+		"stream": false,
+		"format": "json",
+		"messages": []map[string]string{
+			{"role": "system", "content": system},
+			{"role": "user", "content": user},
+		},
+		"options": map[string]any{"temperature": 0},
+	})
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, "POST", g.URL, bytes.NewReader(body))
+	if err != nil {
+		slog.Warn("matrix.judge: build request", "err", err)
+		return 0
+	}
+	req.Header.Set("Content-Type", "application/json")
+	resp, err := g.HTTPClient.Do(req)
+	if err != nil {
+		slog.Warn("matrix.judge: HTTP", "err", err, "url", g.URL)
+		return 0
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode/100 != 2 {
+		slog.Warn("matrix.judge: non-2xx", "status", resp.StatusCode, "url", g.URL)
+		return 0
+	}
+	rb, _ := io.ReadAll(resp.Body)
+	var ollamaResp struct {
+		Message struct {
+			Content string `json:"content"`
+		} `json:"message"`
+	}
+	if err := json.Unmarshal(rb, &ollamaResp); err != nil {
+		slog.Warn("matrix.judge: decode envelope", "err", err)
+		return 0
+	}
+	var v struct {
+		Rating int `json:"rating"`
+	}
+	// Some chat endpoints wrap content in markdown code fences even
+	// with format=json. Strip leading/trailing whitespace + fences.
+	content := strings.TrimSpace(ollamaResp.Message.Content)
+	content = strings.TrimPrefix(content, "```json")
+	content = strings.TrimPrefix(content, "```")
+	content = strings.TrimSuffix(content, "```")
+	content = strings.TrimSpace(content)
+	if err := json.Unmarshal([]byte(content), &v); err != nil {
+		slog.Warn("matrix.judge: decode rating", "err", err, "content", content)
+		return 0
+	}
+	if v.Rating < 1 || v.Rating > 5 {
+		return 0
+	}
+	return v.Rating
+}

internal/matrix/playbook.go

@@ -0,0 +1,492 @@
+package matrix
+
+// Playbook memory — SPEC §3.4 component 5 (learning-loop integration).
+//
+// Concept: every time an external system confirms "(query → answer_id)
+// was a successful match," record it. Future similar queries get that
+// answer's score boosted, so the matrix indexer learns from outcomes
+// rather than relying solely on the base embedder's geometry.
+//
+// Per feedback_meta_index_vision.md: this is the north star — a
+// meta-index that LEARNS from playbooks over time, not a static
+// hybrid search engine.
+//
+// Storage shape: a vectord index named DefaultPlaybookCorpus where:
+//   - The vector is embed(query_text)
+//   - The metadata is a serialized PlaybookEntry
+// Retrieval shape: at /matrix/search time, when use_playbook=true,
+// matrixd searches the playbook corpus with the same query vector,
+// looks up each hit's answer_id, and if that answer is in the current
+// matrix-search results, applies a boost to its distance.
+//
+// Composition: this layer is additive on top of the existing
+// retrieve+merge — when use_playbook=false, behavior is unchanged.
+// The boost only re-ranks results that ALREADY surfaced from the
+// regular retrieval. A v1 enhancement would inject playbook hits
+// directly even when they weren't in the top-K (Shape B from the
+// design conversation), but v0 keeps the safer "boost-only" stance.
+
+import (
+	"encoding/json"
+	"errors"
+	"sort"
+	"strings"
+	"time"
+)
+
+// DefaultPlaybookCorpus is the vectord index name where playbook
+// entries land by default. Callers can override per-request, but
+// having one default makes the system observable from the outside
+// (operator hits /vectors/index and sees this corpus in the list).
+const DefaultPlaybookCorpus = "playbook_memory"
+
+// DefaultPlaybookTopK is how many similar past queries to consider
+// when applying boost. 3 keeps the influence focused — we want the
+// boost to reward consistent matches, not let one stale playbook
+// dominate. Caller can override.
+const DefaultPlaybookTopK = 3
+
+// DefaultPlaybookMaxDistance is the cosine ceiling for "this past
+// query is similar enough to count." 0.5 lets in genuinely related
+// queries while excluding pure-coincidence neighbors. Caller can
+// override per-request as we learn what works for staffing data.
+//
+// This threshold gates the BOOST path (re-rank in place), which is
+// safe at loose thresholds because boost only modifies results already
+// in regular retrieval. The INJECT path uses a tighter ceiling — see
+// DefaultPlaybookMaxInjectDistance.
+const DefaultPlaybookMaxDistance = 0.5
+
+// DefaultPlaybookMaxInjectDistance is the SHAPE B cosine ceiling for
+// "this past query is similar enough to FORCE its answer into the
+// result set." Tighter than DefaultPlaybookMaxDistance because inject
+// is structurally riskier than boost: it adds a result the embedding
+// didn't surface, so a loose match can cross-pollinate the wrong
+// answer into unrelated queries.
+//
+// Empirical motivation (playbook_lift_003): Q2's recording for an
+// OSHA-30 forklift operator surfaced as warm top-1 for the dental
+// hygienist / RN / software engineer OOD queries because their text
+// vectors fell within 0.5 cosine of "OSHA-30 forklift Wisconsin."
+// 0.20 would have rejected those (implied playbook distances 0.38-0.46)
+// while keeping all 6 paraphrase recoveries (≤ 0.30 implied).
+//
+// Boost path stays at 0.5 — re-ranking results that already retrieved
+// by their own merits is safe even when the playbook match is loose.
+const DefaultPlaybookMaxInjectDistance = 0.20
+
+// PlaybookEntry is what gets stored as metadata on each playbook
+// vector. RecordedAt is captured at write time; callers should not
+// set it (the recorder fills it in).
+//
+// Role is an optional caller-provided hint that names the query's
+// staffing role at record time (e.g. "Forklift Operator", "CNC
+// Operator"). When set, the matrix retrieve layer gates Shape A boost
+// and Shape B inject so a recorded entry only fires for queries with
+// the same role. Empty Role = no gate (caller didn't know the role,
+// or the query has no clean role concept — e.g. lift-suite multi-
+// constraint queries). Reality test real_001 surfaced same-client+city
+// queries bleeding across roles when the embedder collapsed them
+// within the inject distance threshold; the role gate closes that
+// without rejecting legitimate intra-role boosts.
+type PlaybookEntry struct {
+	QueryText    string   `json:"query_text"`
+	Role         string   `json:"role,omitempty"`
+	AnswerID     string   `json:"answer_id"`
+	AnswerCorpus string   `json:"answer_corpus"`
+	Score        float64  `json:"score"`         // 0..1; higher = better outcome
+	RecordedAtNs int64    `json:"recorded_at_ns"`
+	Tags         []string `json:"tags,omitempty"`
+}
+
+// Validate returns an error if the entry is missing required fields.
+// Callers should validate before storage so bad data doesn't pollute
+// the corpus.
+func (p PlaybookEntry) Validate() error {
+	if p.QueryText == "" {
+		return errors.New("playbook: query_text required")
+	}
+	if p.AnswerID == "" {
+		return errors.New("playbook: answer_id required")
+	}
+	if p.AnswerCorpus == "" {
+		return errors.New("playbook: answer_corpus required")
+	}
+	if p.Score < 0 || p.Score > 1 {
+		return errors.New("playbook: score must be in [0, 1]")
+	}
+	return nil
+}
+
+// BoostFactor returns the multiplier applied to a result's distance
+// when this playbook entry matches it. Lower is better:
+//
+//	score = 0   → 1.0  (no boost)
+//	score = 0.5 → 0.75 (mild boost)
+//	score = 1.0 → 0.5  (halve the distance — strong boost)
+//
+// Math: 1 - 0.5*score. Capped to [0.5, 1.0] for safety.
+//
+// Why halving as the maximum boost: a perfect-confidence playbook
+// entry shouldn't completely override the base embedding (that
+// invites runaway feedback loops where one early playbook
+// dominates forever). Halving is enough to move a mid-rank result
+// to the top in most cases without erasing the base ranking
+// signal.
+func (p PlaybookEntry) BoostFactor() float64 {
+	score := p.Score
+	if score < 0 {
+		score = 0
+	}
+	if score > 1 {
+		score = 1
+	}
+	return 1.0 - 0.5*score
+}
+
+// MarshalMetadata serializes the entry as the JSON RawMessage that
+// vectord stores per item. Convenience for the recorder.
+func (p PlaybookEntry) MarshalMetadata() (json.RawMessage, error) {
+	return json.Marshal(p)
+}
+
+// UnmarshalPlaybookMetadata is the inverse — used when fetching
+// playbook hits to decode their metadata back into entries.
+func UnmarshalPlaybookMetadata(raw json.RawMessage) (PlaybookEntry, error) {
+	var e PlaybookEntry
+	if len(raw) == 0 {
+		return e, errors.New("playbook: empty metadata")
+	}
+	if err := json.Unmarshal(raw, &e); err != nil {
+		return e, err
+	}
+	return e, nil
+}
+
+// NewPlaybookEntry stamps RecordedAtNs to now and returns the entry.
+// Validation happens at storage; this is just construction.
+//
+// Role-aware overload is NewPlaybookEntryWithRole — callers that don't
+// know the role can keep using this constructor and the role gate
+// will degrade to current (un-gated) behavior.
+func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
+	return PlaybookEntry{
+		QueryText:    query,
+		AnswerID:     answerID,
+		AnswerCorpus: answerCorpus,
+		Score:        score,
+		RecordedAtNs: time.Now().UnixNano(),
+		Tags:         tags,
+	}
+}
+
+// NewPlaybookEntryWithRole is the role-aware variant of NewPlaybookEntry.
+// role should be a stable role token (see roleEqual / role-token notes
+// in the cross-role gate); empty disables the gate for this entry.
+func NewPlaybookEntryWithRole(query, role, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
+	e := NewPlaybookEntry(query, answerID, answerCorpus, score, tags)
+	e.Role = role
+	return e
+}
+
+// PlaybookHit is one similarity-search result from the playbook
+// corpus, paired with its decoded entry. Distance is the cosine
+// distance between the current query and this past playbook's
+// query vector — used by the caller to filter out "too far"
+// matches via PlaybookMaxDistance.
+type PlaybookHit struct {
+	PlaybookID string        `json:"playbook_id"`
+	Distance   float32       `json:"distance"`
+	Entry      PlaybookEntry `json:"entry"`
+}
+
+// roleEqual is the cross-role gate's match function. Returns true when
+// EITHER side is empty (gate disabled — current behavior preserved) OR
+// the role tokens compare equal under a normalization that ignores
+// case + trailing 's' (so "Forklift Operator" matches "forklift
+// operator" matches "Forklift Operators"). Robust enough for the
+// staffing-domain plurals seen in fill_events; tighter normalization
+// (Levenshtein, role-graph) is a separate concern and not blocking
+// the bleed fix.
+//
+// Why this lives near PlaybookEntry: the gate is a property of the
+// entry/query pair, not of the boost or inject mechanism — both call
+// sites in this file delegate to the same function so they can't
+// drift apart.
+func roleEqual(a, b string) bool {
+	if a == "" || b == "" {
+		return true
+	}
+	return roleNormalize(a) == roleNormalize(b)
+}
+
+// nonPluralSWords is the allowlist of staffing-domain tokens that end
+// in 's' but are NOT plurals — naïve trailing-'s' stripping would
+// mangle these into wrong tokens ("Sales" → "Sale", "Logistics" →
+// "Logistic"). Per scrum review role_gate_v1 (Opus + Kimi + Qwen
+// convergent finding 2026-04-30): expand this list as new false-
+// positives surface from production traffic.
+var nonPluralSWords = map[string]bool{
+	"sales":      true,
+	"logistics":  true,
+	"operations": true,
+	"facilities": true,
+	"premises":   true,
+	"news":       true,
+	"physics":    true,
+	"economics":  true,
+	"mathematics":true,
+	"analytics":  true,
+}
+
+// roleNormalize lowercases + trims + best-effort-singularizes the
+// last word so "Forklift Operators" matches "Forklift Operator" but
+// "Sales Associate" doesn't lose its "s".
+//
+// Only the LAST word is plural-checked because staffing-domain role
+// names follow English head-noun-last conventions ("Sales Associate"
+// → head is "Associate"; "Forklift Operators" → head is "Operators").
+//
+// Singularization rules (in order):
+//  1. Last word ∈ nonPluralSWords → return as-is (e.g. "Logistics
+//     Coordinator" stays "logistics coordinator").
+//  2. Last word ends in "ss" → return as-is ("Press Operator" stays;
+//     "Boss" stays).
+//  3. Last word ends in "es" after "x"/"s"/"ch"/"sh" → strip "es"
+//     ("Boxes" → "Box", "Dishes" → "Dish").
+//  4. Last word ends in "s" → strip "s" ("Operators" → "Operator").
+//  5. Otherwise return as-is.
+func roleNormalize(s string) string {
+	s = strings.ToLower(strings.TrimSpace(s))
+	if s == "" {
+		return ""
+	}
+	// Find the last word.
+	lastSpace := strings.LastIndex(s, " ")
+	prefix, last := "", s
+	if lastSpace >= 0 {
+		prefix, last = s[:lastSpace+1], s[lastSpace+1:]
+	}
+	if nonPluralSWords[last] {
+		return s
+	}
+	if strings.HasSuffix(last, "ss") {
+		return s
+	}
+	n := len(last)
+	if n >= 3 && last[n-1] == 's' && last[n-2] == 'e' &&
+		(last[n-3] == 'x' || last[n-3] == 's' || last[n-3] == 'h') {
+		return prefix + last[:n-2]
+	}
+	if strings.HasSuffix(last, "s") {
+		return prefix + last[:n-1]
+	}
+	return s
+}
+
+// InjectPlaybookMisses appends synthetic Results for playbook hits
+// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
+// This is "Shape B" from the doc comment at the top of this file:
+// the v0 boost-only stance (ApplyPlaybookBoost) can't promote a
+// recorded answer that wasn't already in the regular retrieval's
+// top-K. Paraphrase queries broke this — different embedding ⇒
+// different top-K ⇒ recorded answer drops out ⇒ no boost can save
+// it. Reality test playbook_lift_002 showed 0/2 paraphrase top-1
+// lifts because of exactly that.
+//
+// Synthetic distance = playbook_hit_distance × BoostFactor — same
+// formula as ApplyPlaybookBoost, applied to the playbook hit's own
+// distance instead of a result's. Lower playbook hit distance
+// (current query is similar to recorded query) AND higher score
+// (recorded outcome was strong) push the injection toward top-1.
+//
+// fetchPlaybookHits has already filtered hits to those within
+// DefaultPlaybookMaxDistance (0.5), so injected results land in the
+// same distance range as regular retrieval — they don't dominate
+// top-K from out-of-distribution playbooks.
+//
+// Returns the (possibly extended) results slice and how many synthetic
+// rows were appended. Caller MUST re-sort + truncate to K afterwards.
+//
+// maxInjectDist filters which hits qualify for injection — hits whose
+// playbook-corpus cosine distance exceeds it are skipped (the boost
+// path may still re-rank them in place). Pass 0 (or any non-positive
+// value) to use DefaultPlaybookMaxInjectDistance.
+//
+// gate is an optional approval callback called once per CANDIDATE
+// (post-distance-filter, post-dedup) before injection. Returning
+// false rejects that candidate. Use nil for the historical "all
+// distance-eligible candidates inject" behavior.
+//
+// Multi-coord run #008's judge re-rating proved that distance + LLM
+// rating disagree often enough to matter (Q3 crane: dist 0.23 looks
+// confident, judge says 1/5 = irrelevant). Lift-suite tail issues
+// (Q6↔Q7 swap, Q9/Q15 paraphrase drift) are exactly this shape —
+// embedding-tight but wrong-domain. The gate parameter lets callers
+// route those candidates through a judge before the inject lands.
+//
+// query is the current search's query text — passed to the gate so
+// it can score (query, candidate) pairs without re-deriving from
+// SearchRequest. Empty when the caller doesn't have it (gate
+// implementations should treat empty query as "skip judge, allow").
+//
+// queryRole, when non-empty, gates injection on role match: a playbook
+// hit whose Entry.Role is set and DIFFERENT from queryRole is rejected
+// before the judge gate runs. Closes the same-client+city cross-role
+// bleed (real_001 finding) where the inject distance threshold (0.20)
+// wasn't tight enough to prevent Forklift-Operator playbook entries
+// from injecting on CNC-Operator queries that happened to embed close
+// (same city + client + count-token + time-token dominate cosine).
+// Empty queryRole OR empty hit.Entry.Role = no gate (preserves
+// behavior for callers that don't supply role).
+func InjectPlaybookMisses(query, queryRole string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
+	if len(hits) == 0 {
+		return results, 0
+	}
+	if maxInjectDist <= 0 {
+		maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
+	}
+	present := make(map[string]bool, len(results))
+	for _, r := range results {
+		present[r.Corpus+"|"+r.ID] = true
+	}
+
+	// For each (corpus, id) NOT in results, keep the playbook hit
+	// with the largest boost (lowest BoostFactor = highest score).
+	// Multiple hits to the same answer collapse to one injection.
+	bestForKey := make(map[string]PlaybookHit)
+	for _, h := range hits {
+		// Inject-specific tighter threshold (boost path's threshold is
+		// looser; this prevents cross-pollination of wrong-domain
+		// answers into queries whose text happens to fall within
+		// boost-distance of an unrelated recording).
+		if h.Distance > maxInjectDist {
+			continue
+		}
+		// Cross-role gate (real_001 bleed fix). Reject before the
+		// judge gate so we don't burn a slow LLM call to immediately
+		// reject the candidate on a structural mismatch.
+		if !roleEqual(queryRole, h.Entry.Role) {
+			continue
+		}
+		key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
+		if present[key] {
+			continue
+		}
+		if existing, ok := bestForKey[key]; !ok || h.Entry.BoostFactor() < existing.Entry.BoostFactor() {
+			bestForKey[key] = h
+		}
+	}
+
+	injected := 0
+	for _, h := range bestForKey {
+		// Judge gate (per OPEN item #1, closed by this commit):
+		// post-distance-filter, ask the gate whether the candidate
+		// actually fits the current query before letting it inject.
+		// Closes the lift-suite tail issues where embedding said
+		// "tight" but a judge said "wrong domain."
+		if gate != nil && !gate.Approve(query, h) {
+			continue
+		}
+		injectedDist := h.Distance * float32(h.Entry.BoostFactor())
+		// Synthesize metadata that flags the injection so callers
+		// (driver/UI/observer) can distinguish "regular retrieval"
+		// from "playbook injection." Production consumers needing
+		// the actual worker metadata can fetch from vectord by
+		// (Corpus, ID) — synthetic results carry only provenance.
+		meta, _ := json.Marshal(map[string]any{
+			"playbook_injected":      true,
+			"playbook_id":            h.PlaybookID,
+			"playbook_score":         h.Entry.Score,
+			"playbook_query_text":    h.Entry.QueryText,
+			"playbook_recorded_at_ns": h.Entry.RecordedAtNs,
+			"playbook_hit_distance":  h.Distance,
+		})
+		results = append(results, Result{
+			ID:       h.Entry.AnswerID,
+			Corpus:   h.Entry.AnswerCorpus,
+			Distance: injectedDist,
+			Metadata: meta,
+		})
+		injected++
+	}
+
+	return results, injected
+}
+
+// InjectGate is the optional approval callback for Shape B inject.
+// Called once per candidate (after distance filter, after dedup).
+// Returning false rejects that candidate. Implementations:
+//   - LLMJudgeGate (this package, see judge.go): Ollama LLM rates the
+//     (query, candidate) pair against a 1-5 rubric.
+//   - InjectGateFunc (this package): zero-deps adapter for arbitrary
+//     caller logic — useful in tests + when callers want non-LLM
+//     gating (e.g. metadata-only filters).
+//
+// nil InjectGate = pre-judge-gating behavior (all distance-eligible
+// candidates inject); preserves backward compatibility.
+type InjectGate interface {
+	Approve(query string, hit PlaybookHit) bool
+}
+
+// InjectGateFunc adapts a plain function to the InjectGate interface.
+// Used heavily in tests; production callers usually use LLMJudgeGate.
+type InjectGateFunc func(query string, hit PlaybookHit) bool
+
+// Approve makes InjectGateFunc satisfy InjectGate.
+func (f InjectGateFunc) Approve(q string, h PlaybookHit) bool { return f(q, h) }
+
+// ApplyPlaybookBoost re-ranks results in place using matched
+// playbook hits. For each hit whose (AnswerID, AnswerCorpus)
+// matches a result, multiply that result's distance by the hit's
+// BoostFactor. If multiple hits match the same result, the highest-
+// score one wins (greatest reduction in distance).
+//
+// queryRole, when non-empty, gates the boost on role match — same
+// semantics as InjectPlaybookMisses' queryRole. Defense in depth:
+// even if a recorded entry's answer happens to surface in the
+// regular retrieve top-K for a different-role query, role-mismatch
+// shouldn't fire a boost. Empty role on either side disables the gate.
+//
+// After applying boosts, results are re-sorted ascending by
+// distance.
+//
+// Returns the number of distinct results that received a boost.
+// Callers can log this as a signal of "how much the playbook
+// influenced this query."
+func ApplyPlaybookBoost(queryRole string, results []Result, hits []PlaybookHit) int {
+	if len(hits) == 0 || len(results) == 0 {
+		return 0
+	}
+
+	// For each result, find the hit with the lowest BoostFactor
+	// (= largest boost = highest score, since BoostFactor is
+	// 1-0.5*score and we minimize).
+	bestBoost := make(map[int]float64, len(results))
+	for i, r := range results {
+		for _, h := range hits {
+			if h.Entry.AnswerID != r.ID || h.Entry.AnswerCorpus != r.Corpus {
+				continue
+			}
+			// Cross-role gate (real_001 bleed fix). Same semantics as
+			// InjectPlaybookMisses — see comment there.
+			if !roleEqual(queryRole, h.Entry.Role) {
+				continue
+			}
+			bf := h.Entry.BoostFactor()
+			if cur, ok := bestBoost[i]; !ok || bf < cur {
+				bestBoost[i] = bf
+			}
+		}
+	}
+
+	for i, bf := range bestBoost {
+		results[i].Distance = float32(float64(results[i].Distance) * bf)
+	}
+
+	sort.SliceStable(results, func(i, j int) bool {
+		return results[i].Distance < results[j].Distance
+	})
+
+	return len(bestBoost)
+}