feat(gateway/v1): real SSE streaming on the openrouter lane; honest 400 elsewhere

The gateway accepted `stream: true` and returned a single JSON body — the code
said 'Phase 39+ wires real streaming' and it never happened. A streaming client
waits for SSE frames that never arrive and reports 'no response': that is what
broke openclaw's gateway lane today and cost six orchestrator rounds before the
cause was found.

Now: openrouter/openrouter_free forward the upstream SSE byte-for-byte; every
other lane rejects stream=true with a 400 that names the lane, instead of
pretending to have streamed. Verified live: chat.completion.chunk frames from
:3100, and the non-streaming path unchanged. Streamed calls skip usage metering
and Langfuse (stated in the code and logged per call) — metering needs the final
usage chunk and belongs in its own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.archon/workflows/lakehouse-architect-review.yaml

@@ -0,0 +1,42 @@
+# Real Archon workflow on the Lakehouse repo, fully via our gateway.
+# Three Pi nodes, each fires LLM → /v1/chat/completions → OpenRouter,
+# every call lands a Langfuse trace + observer event.
+#
+# Read-only (allowed_tools: [read]). Don't pass --branch / leave
+# --no-worktree at runtime so Archon doesn't try to create a worktree.
+name: lakehouse-architect-review
+description: 'Pi reviews Lakehouse architecture in 3 turns through our gateway.'
+provider: pi
+model: openrouter/x-ai/grok-4.1-fast
+
+nodes:
+  - id: shape
+    prompt: |
+      Read these files and answer in 3 short bullets describing the
+      architectural shape of Lakehouse:
+      - /home/profit/lakehouse/Cargo.toml
+      - /home/profit/lakehouse/lakehouse.toml
+      - /home/profit/lakehouse/docs/MODE_RUNNER_TUNING_PLAN.md
+      Be terse. No preamble.
+    allowed_tools: ["read"]
+    effort: low
+    idle_timeout: 90000
+
+  - id: weakness
+    prompt: |
+      Read /home/profit/lakehouse/crates/gateway/src/v1/mod.rs and
+      identify ONE real weakness or risk. Cite file:line. One paragraph.
+    allowed_tools: ["read"]
+    effort: low
+    idle_timeout: 90000
+    depends_on: [shape]
+
+  - id: improvement
+    prompt: |
+      Based on the prior weakness ($weakness.output), propose ONE
+      surgical improvement (≤6 lines of Rust). Show the patch as
+      `old_string` and `new_string` in markdown code blocks.
+    allowed_tools: []
+    effort: low
+    idle_timeout: 90000
+    depends_on: [weakness]

.gitignore

@@ -4,3 +4,74 @@
 .env
 __pycache__/
 *.pyc
+
+# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
+# (synthetic StyleGAN, ~580MB for 1000 faces). Manifest + fetch script are tracked.
+data/headshots/face_*.jpg
+data/headshots/_thumbs/
+# ComfyUI on-demand generated portraits (per-worker unique). Cached on first
+# request; fully regeneratable via /headshots/generate/:key.
+data/headshots_gen/
+
+# Runtime data — all regeneratable from inputs or accumulated by daemons.
+# Anything under data/_<name>/ is internal state (auditor outputs, KB caches,
+# pathway memory snapshots, HNSW trial results, etc.). Anything under
+# data/datasets/ or data/vectors/ is generated by ingest/index pipelines.
+data/_*/
+data/lance/
+data/datasets/
+data/vectors/
+data/demo/
+data/evidence/
+data/face_test/
+data/headshots_role_pool/
+data/icons_pool/
+data/scored-runs/
+data/workspaces/
+data/catalog/
+data/**/*.bak-*
+data/**/*.pre-*-bak
+
+# Logs
+logs/
+
+# Build artifacts
+node_modules/
+exports/
+mcp-server/data/
+
+# Per-run distillation reports (timestamp-named); keep the parent dir tracked
+# via .gitkeep if needed but don't carry every batch's report set.
+reports/distillation/[0-9]*/
+reports/distillation/*-*-*-*-*/
+
+# Test scratch — scratchpads, traces, sessions are regenerated each run.
+# PRD/scenario fixtures stay tracked (they ARE the test).
+tests/agent_test/_*
+tests/agent_test/sessions/
+tests/real-world/runs/
+
+# BIPA-quarantined photo uploads — Phase 1.6 Gate 3a writes to
+# data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext> with mode 0700/0600.
+# This is regulated subject-of-record data and must NEVER enter git.
+data/biometric/
+
+# Local-only scrum review evidence. Per `feedback_audit_findings_log.md`
+# scrum runs fold fixes into a batch commit; the verdict / disposition
+# files stay local for forensics.
+reports/scrum/
+
+# Per-event biometric verification reports (timestamp-named, regenerated
+# per `verify_biometric_erasure.sh` invocation). Source-of-truth is the
+# audit chain itself; these reports are derived views.
+reports/biometric/
+
+# Counsel transmission tarballs + manifests are regenerated by
+# `bundle_counsel_packet.sh` from the tracked `docs/counsel/` source.
+# The bundle is transmittable, not source-of-truth.
+reports/counsel/
+
+# Local experiments scratchpad — per the "Test code in main is ACTIVELY
+# being cleaned out" policy (commits 6aafd41 + f4ebd22), one-off
+# experiments stay out of the tracked tree.
+experiments/

Cargo.lock

@@ -48,6 +48,7 @@ version = "0.1.0"
 dependencies = [
  "async-trait",
  "axum",
+ "lru",
  "reqwest",
  "serde",
  "serde_json",
@@ -1364,10 +1365,13 @@ dependencies = [
  "axum",
  "bytes",
  "chrono",
+ "hmac",
  "object_store",
+ "parquet 55.2.0",
  "proto",
  "serde",
  "serde_json",
+ "sha2",
  "shared",
  "storaged",
  "tokio",
@@ -4086,12 +4090,14 @@ dependencies = [
  "shared",
  "storaged",
  "tokio",
+ "toml",
  "tonic",
  "tower-http",
  "tracing",
  "tracing-opentelemetry",
  "tracing-subscriber",
  "truth",
+ "validator",
  "vectord",
 ]
 
@@ -8911,6 +8917,8 @@ dependencies = [
 name = "validator"
 version = "0.1.0"
 dependencies = [
+ "arrow 55.2.0",
+ "parquet 55.2.0",
  "serde",
  "serde_json",
  "thiserror 2.0.18",

Cargo.toml

@@ -36,6 +36,7 @@ datafusion = "47"
 bytes = "1"
 futures = "0.3"
 sha2 = "0.10"
+hmac = "0.12"
 url = "2"
 tonic = "0.13"
 prost = "0.13"

STATE_OF_PLAY.md

@@ -0,0 +1,492 @@
+# STATE OF PLAY — Lakehouse
+
+**Last verified:** 2026-06-30 (search index-alignment fix live on :3700) · biometric substrate 2026-05-05
+**Verified by (2026-06-30):** live probe after mcp-server restart — `/match` + `/search` with `state='IL'` return only IL workers (new PID on :3700). Not memory.
+**Verified by (2026-05-05):** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory.
+
+> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
+
+---
+
+## WHAT LANDED 2026-06-30 (search index-alignment sweep + P1-already-done finding)
+
+**Fixed a systemic hybrid-search bug.** Every `/vectors/hybrid` call that paired the
+`ethereal_workers` SQL dataset (10K rows) with the `workers_500k_v1` vector index (50K,
+a *different* worker set) returned wrong-entity results: same `worker_id` = different
+person in each index, so a `state='IL'` query surfaced a **KY** worker flagged
+`sql_verified:true`. Root cause: the default index didn't share an id-space with the
+default dataset.
+
+Swept **8 call sites** in `mcp-server/index.ts` to `ethereal_workers_v1` (10K, same
+id-space as the SQL table). Aligned sites (`filter_dataset: workers_500k`) and pure-RAG
+sites (`/vectors/rag`) were correctly left alone.
+
+| Commit | Sites |
+|---|---|
+| `423400e` | `/search` HTTP handler |
+| `35a80a7` | `search_workers` tool · `match_contract` tool · `/match` HTTP handler |
+| `1570306` | 4 intelligence routes (similar-workers, role-discovery, smart_search, scenario-find) |
+
+**Verified live** after an mcp-server restart (new PID on :3700): `/match` and `/search`
+with `state='IL'` return only IL workers (Champaign/Springfield/Urbana); `state='WI'`
+returns WI. Grep confirms no `workers_500k_v1` index remains paired with `ethereal_workers`.
+Server boots clean. Only `mcp-server/index.ts` was touched.
+
+**Finding — demo-plan P1 was already done.** The "search box that actually filters" item
+in this doc's CURRENT PLAN (P1, from the 2026-04-27 demo audit) is **stale**. The UI now
+posts structured `state`/`role` to the evolved `/intelligence/chat`, which returns real
+`sql_results` (IL Forklift → 167 matches, correct workers; WI → WI). The P1 section below
+describes a bug that no longer exists — don't re-implement it.
+
+**⚠ Uncommitted WIP left untouched (pre-existing, not from this session).**
+`mcp-server/search.html` + `proof.html` (edited 2026-05-30; proof.html restructured into
+chapters) and a `biometric_endpoint.rs` comment cleanup remain uncommitted in the working
+tree. Also untracked and newer than this doc: `RECOVERY_PLAN_2026-05-05.md`,
+`mcp-server/UX_FIX_PLAN.md`, `docs/scope-sentinel/`. Someone should reconcile these — they
+imply planning context this file doesn't yet capture.
+
+---
+
+## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard)
+
+After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`.
+
+| Commit | What | Verified |
+|---|---|---|
+| `76cb5ac` | `POST /biometric/subject/{id}/consent` — Gate 2 backend. Flips status NeverCollected→Given, computes retention_until = given_at + 18mo (per retention schedule v1 §4), captures consent_version_hash + collection_method (closed set: electronic_signature/paper/click_acceptance) + operator + evidence path to audit row. State machine: 409 already_given, 409 post-withdrawal-requires-erase, 403 subject_inactive. | 12 unit tests; live POST returns 401/400/404 on guards |
+| `7f0f500` | Candidate intake UI at `/biometric/intake?candidate_id=XXX` (mcp-server :3700, exposed via nginx at devop.live/lakehouse/biometric/intake). 4-screen flow: operator auth → consent template render + click-accept → photo capture (file or webcam getUserMedia) → confirmation showing audit hmac. SHA-256 of rendered consent block becomes consent_version_hash. sessionStorage-backed token (clears on tab close), neo-brutalist style matching onboard.html. | 22631-byte HTML, mcp-server route returns 200 |
+| `68d226c` | Three things in one wave: <br>(a) `POST /biometric/subject/{id}/withdraw` — BIPA right of withdrawal. Flips Given→Withdrawn, accelerates retention_until from 18mo→30d (per consent template v1 §2 SLA), audit kind=biometric_consent_withdrawal. State machine 409s on NeverCollected/Pending (nothing_to_withdraw), Withdrawn (already_withdrawn), Expired (already_expired). 12 unit tests. <br>(b) Withdraw UI at `/biometric/withdraw` — 3-screen operator flow (token+name auth → reason+evidence form → confirmation showing 30-day clock + verify curl recipe). <br>(c) `lakehouse-retention-sweep.{service,timer}` systemd units in `ops/systemd/`. Daily 03:00 UTC, Persistent=true, install.sh updated to handle paired timer+oneshot service. <br>Plus operator_of_record bug fix in intake UI (was hardcoded `'intake_ui_operator'`). | 46/46 biometric_endpoint + 71/71 catalogd lib tests; manual sweep run: 100 subjects, 0 overdue, exit 0 |
+| **(current HEAD post-this-wave)** | Stats endpoint `GET /biometric/stats` (legal-tier auth, returns subject counts by status + photo count + oldest active retention + last 20 state-change events with anonymizable trace_ids) + ops dashboard at `/biometric/dashboard` (single-page, polls /stats every 15s, table + status tiles, XSS-safe DOM construction not innerHTML). Plus consent_versions allowlist: `BiometricEndpointState.allowed_consent_versions: Option<Arc<HashSet<String>>>`, loaded from `/etc/lakehouse/consent_versions.json` (`LH_CONSENT_VERSIONS_FILE` override), missing-file = permissive (v1 compat), present + populated = strict mode (refuses unknown hashes with 400 consent_version_unknown). Plus `scripts/staffing/subject_timeline.sh` — pretty-prints any subject's full BIPA lifecycle from /audit/subject/{id} (manifest + on-disk photo + chronological audit chain + verification status). | 5 new allowlist unit tests + 4 stats tests; live demo on WORKER-100 ran end-to-end (consent → photo → withdraw, chain verified=true, chain_root=a47563ff…) |
+
+### Live demo evidence (WORKER-100, 2026-05-05 20:12 UTC)
+The full lifecycle was exercised against the live gateway as a verification artifact. The audit chain on WORKER-100 now contains 3 rows:
+
+```
+20:12:33.054  BIOMETRIC_CONSENT_GRANT       result=given       hmac=9c6f4153341e97d2…  trace=live-demo-2026-05-05
+20:12:47.957  BIOMETRIC_COLLECTION          result=success     hmac=856be6173c88277c…  trace=live-demo-2026-05-05
+20:15:27.298  BIOMETRIC_CONSENT_WITHDRAWAL  result=withdrawn   hmac=a47563ff937d50de…  trace=live-demo-2026-05-05
+```
+
+`chain_verified=true`, chain_root = a47563ff937d50de43b09a0c903cff954233836c219a928ee8ca2aa6792272dd. Photo file at `data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg` (180 bytes — a minimal real JFIF JPEG), retention_until=2026-06-04 (= 30 days from withdrawal). Retention sweep will flag this subject on or after that date; operator runs `/biometric/subject/WORKER-100/erase` to destroy.
+
+To re-verify: `./scripts/staffing/subject_timeline.sh WORKER-100`.
+
+### Endpoint matrix (v1 BIPA lifecycle complete)
+
+| Event | Endpoint | Method | Auth | Status flip | retention_until |
+|---|---|---|---|---|---|
+| consent given | `/biometric/subject/{id}/consent` | POST | legal | NeverCollected/Pending → Given | now + 18mo |
+| photo collected | `/biometric/subject/{id}/photo` | POST | legal + consent gate | (no change) | (no change) |
+| consent withdrawn | `/biometric/subject/{id}/withdraw` | POST | legal | Given → Withdrawn | now + 30d |
+| destruction | `/biometric/subject/{id}/erase` | POST | legal | (manifest cleared) | n/a |
+| audit read | `/audit/subject/{id}` | GET | legal | (read-only) | (read-only) |
+| ops aggregates | `/biometric/stats` | GET | legal | (read-only) | (read-only) |
+
+UIs:
+- `/biometric/intake?candidate_id=X` — operator-driven consent + photo
+- `/biometric/withdraw` — operator-driven withdrawal recording
+- `/biometric/dashboard` — read-only ops aggregate, auto-refresh
+
+CLI tools:
+- `scripts/staffing/verify_biometric_erasure.sh <id>` — post-erasure verification
+- `scripts/staffing/biometric_destruction_report.sh --month YYYY-MM` — anonymized monthly report
+- `scripts/staffing/subject_timeline.sh <id>` — full lifecycle pretty-print (NEW 2026-05-05)
+- `scripts/staffing/bundle_counsel_packet.sh` — counsel review tarball
+- `scripts/staffing/attest_pre_identityd_biometric_state.sh` — defense attestation generator
+
+### What's blocking production cutover NOW
+**Counsel calendar.** Engineering substrate is done end-to-end: every state transition has a defensible endpoint, every endpoint has tests + live verification, every UI is reachable, retention sweep is scheduled, allowlist hardening is wired. The remaining work is signature/review:
+1. Counsel review of consent template v1 (revised for Option C — classifications deferred)
+2. Counsel review of retention schedule v1 (revised for Option C)
+3. Counsel review of destruction runbook
+4. Counsel + J signatures on §2 attestation
+5. Once counsel signs the consent template, populate `/etc/lakehouse/consent_versions.json` with the signed hash to flip the gateway from permissive to strict mode
+
+Counsel-review packet at `reports/counsel/counsel_packet_2026-05-05.tar.gz` (regenerable via `bundle_counsel_packet.sh` to pick up the latest doc state).
+
+---
+
+## WHAT LANDED 2026-05-05 (morning wave — doc reconciliation + Gate 3b decision + counsel packet)
+
+This was a **doc-only wave**, not code. Background: J asked for an audit of the BIPA/biometric documentation before production cutover. Audit found moderate fragmentation between docs and shipped code (post-`identityd` collapse, post-Gate-3a-ship, pre-Gate-3b-decision). Closed it in one pass.
+
+| Item | What changed | Status |
+|---|---|---|
+| **Gate 3b — DECIDED: Option C (defer classifications)** | `BiometricCollection.classifications` stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status flipped from "draft / awaits product" to "DECIDED 2026-05-05". | Locked |
+| **Endpoint-path drift** | `PHASE_1_6_BIPA_GATES.md` (3 spots), `BIPA_DESTRUCTION_RUNBOOK.md` (2 spots), `biometric_retention_schedule_v1.md` (1 spot) updated from legacy `/v1/identity/subjects/*` (proposed under separate identityd daemon, never shipped) to actual `/biometric/subject/*` (catalogd-local, shipped `848a458`). Schema block in `PHASE_1_6_BIPA_GATES.md` rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate (not the proposed Postgres `subjects` table). | Reconciled |
+| **Consent template + retention schedule** | Both revised for Option C: removed all "automated facial-classification" / "deepface" language so disclosed scope matches implemented scope. Pending counsel review — they were already eng-staged with ⚖ markers. | Eng-staged for counsel |
+| **`scripts/staffing/verify_biometric_erasure.sh`** (NEW) | Operator-side verification of an erasure event. Curls `/audit/subject/{id}` with legal-tier token, checks: manifest.biometric_collection null, uploads dir empty, last audit row is `biometric_erasure`/`full_erasure` with `erased`/`success`, chain_verified=true. Writes a hashed report to `reports/biometric/`. | Smoke-tested live |
+| **`scripts/staffing/biometric_destruction_report.sh`** (NEW) | Monthly destruction-event aggregation. Anonymizes candidate IDs (sha256-12 prefix), counts by scope + trigger, flags anomalies. Smoke-test on May 2026 data found 1 historical `biometric_erasure`/`consent_withdrawal` event (test fixture). | Smoke-tested live |
+| **`docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`** (NEW) | Captures the rotation procedure operationalized after the 2026-05-05 `/tmp` wipe incident. Covers: when to rotate, pre-rotation snapshot, atomic-swap procedure, post-rotation verification (incl. expected pre-rotation chain tamper-detect under new key), recovery from lost key, ⚖ counsel notes. | Authored |
+| **`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`** + `bundle_counsel_packet.sh` (NEW) | Cover note bundling all eng-staged BIPA docs for counsel review with per-doc questions, sign-off checklist, recommended review sequence. Bundler script tarballs the 8 referenced files + emits a SHA-256 manifest. Tarball ready for transmission: `reports/counsel/counsel_packet_2026-05-05.tar.gz`. | Bundled, ready to send |
+
+### Eng follow-up that this wave surfaced
+- **Double-upload file leak — DIAGNOSED + FIXED** (2026-05-05 same wave). `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo file. Investigation showed:
+  - The file was 13 bytes of test fixture (`ff d8 ff d9 + ASCII "TESTBYTES"`), byte-identical to the unit-test fixture at `biometric_endpoint.rs:841`. NO PII, NO biometric content, NO synthetic-face content. Came from manual integration testing on 2026-05-03.
+  - Audit log timeline showed two consecutive uploads (09:54, 10:04) followed by one erasure (10:22). The erasure unlinked only the SECOND file (which the manifest pointed at by then); the first file was orphaned because the second upload had silently overwritten `manifest.data_path`.
+  - **Real bug found**: the upload handler did NOT refuse a second upload to a subject with `biometric_collection.is_some()`. Patched `process_upload` to return HTTP 409 + `error: "biometric_already_collected"` when a re-upload is attempted; operator must explicitly POST `/biometric/subject/{id}/erase` first.
+  - Stranded test file removed (`rm` of the 13-byte fixture).
+  - New unit test `second_upload_without_erase_returns_409` asserts the 409 + that the first photo's data_path remains unchanged + that the first file remains untouched on disk.
+  - Existing `repeated_uploads_grow_the_chain` replaced with `upload_erase_upload_grows_the_chain_cleanly` (covers the legitimate re-collection cycle: upload → erase → upload, chain grows to 3 rows).
+  - Existing `content_type_with_parameters_accepted` test updated to use two distinct subjects (it had used one subject for two uploads to test content-type parsing — now would 409).
+  - **22 biometric_endpoint tests + 59 catalogd lib tests all green** post-patch (was 21+58 pre-patch).
+  - Production posture: gateway binary needs rebuild (`cargo build --release`) + `systemctl restart lakehouse.service` to pick up the new 409 behavior in live traffic.
+- **Pre-rotation chain tamper-detect (expected, not a bug).** WORKER-{1..5} had pre-2026-05-05 audit chains under the prior `LH_SUBJECT_AUDIT_KEY`. Under the new key (post-`/tmp` wipe rotation), those chains correctly tamper-detect. The rotation runbook §4.4 documents this as expected; a §2.2 pre-rotation snapshot is what would prove they were intact pre-rotation if defensibility ever needs it.
+
+### What's blocking production cutover NOW (after this wave)
+- **Counsel calendar:** the four sign-off items in `COUNSEL_REVIEW_PACKET_2026-05-05.md` (retention schedule, consent template, destruction runbook, pre-identityd attestation). The packet tarball is ready; ⚖ counsel is the bottleneck.
+- **Nothing else.** Engineering is no longer the long pole.
+
+### Phase 1.6 BIPA gates — status table (this is the final post-Option-C state)
+
+| # | Gate | Status |
+|---|---|---|
+| 1 | Public retention schedule | **eng-staged**, revised for Option C, ready for counsel |
+| 2 | Informed consent template | **eng-staged**, revised for Option C, ready for counsel |
+| 3a | Photo upload endpoint | **DONE** (shipped `f1fa6e4`, 11 unit tests, live verified) |
+| 3b | Deepface classification | **DECIDED 2026-05-05: Option C (defer)** |
+| 4 | Name → ethnicity inference removal | **DONE** (shipped, 4/4 mcp-server tests pass) |
+| 5 | Destruction runbook + erasure endpoint | **eng-DONE** (`848a458`, 21 tests). Runbook scripts (verify + report) shipped 2026-05-05. Counsel review pending. |
+| §2 | Pre-identityd attestation | **eng-DONE** (3/3 evidence checks). Awaits J + counsel signature. |
+| §3 | Employee training | **deferred** (consolidated into runbook §7 acknowledgment for current operator population) |
+
+---
+
+## WHAT LANDED 2026-05-03 (16 commits this wave — local-first audit substrate + Phase 1.6 BIPA gates)
+
+The dominant work today: **`docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8 SHIPPED end-to-end** + **5 of 7 Phase 1.6 BIPA pre-launch gates** + **6th cross-runtime parity probe**. Wave was structured as eight ship-then-scrum cycles — every wave caught real bugs, every fix wave landed within the same session.
+
+| Commit | What | Verified |
+|---|---|---|
+| `d259909` | catalogd: Step 1 — `SubjectManifest` type + Registry CRUD | 17 catalogd subject tests PASS |
+| `d16131b` | catalogd: Step 2 — `SubjectAuditWriter` HMAC-SHA256 chain + per-subject Mutex + canonical-JSON via BTreeMap | tamper-detection + concurrent-append race tests PASS |
+| `bce6dfd` | catalogd: Step 3 — `bin/backfill_subjects` (BIPA-defensible defaults: vertical=unknown, consent=pending_backfill_review, retention=4yr) | 100 subjects loaded into live catalog |
+| `fef1efd` | gateway: Step 4 — wire SubjectAuditWriter into `/v1/chat` tool dispatch + `audit_subject_hits_in` (inline, not spawn) | tool calls log accessor.kind="gateway_lookup" |
+| `cd8c59a` | gateway: Step 5 — `AuditingWorkerLookup` decorator wraps validator's WorkerLookup; spawns audit on every find() | live `/v1/validate` produces audit rows |
+| `e38f357` | subjects Steps 1-4 fixes from cross-lineage scrum (concurrency race, schema-evolution HMAC drift, hardcoded "success" classifier) | 41 tests green |
+| `15cfd76` | catalogd + gateway: Step 6 — `/audit/subject/{id}` legal-tier endpoint with constant-time-eq token check + tampering detection | live curl returns chain_verified=true |
+| `2a4b316` | subjects 2nd scrum fix wave (token min 16→32, chain_root from full chain via `chain_tip()`, rebuild collision warn, tightened result-state heuristic) | 17 catalogd + 6 gateway tests PASS |
+| `8fc6238` | catalogd: Step 7 — `bin/retention_sweep` (BIPA-aware on biometric clock, idempotent across daily runs, no auto-mutation) | 8 sweep tests PASS, live verified at `--as-of 2031-06-01` flagging 100/100 expired |
+| `2413c96` | catalogd: Step 8 — `bin/parity_subject_audit` (Rust side of cross-runtime parity probe) | known-answer + verify modes match Go byte-for-byte |
+| `2222227` | parity helper hardening (panic-noise → die() helper, abs path stripped from doc) from scrum | parity probe still 6/6 |
+| `4708717` | **Phase 1.6 BIPA wave** — Gate 4 absence test (4/4 with bypass coverage), §2 attestation script (3/3 evidence checks), Gate 1/2/5 doc scaffolds with ⚖ COUNSEL markers | 4/4 mcp-server Bun tests, 3/3 evidence on live data |
+| `c7aa607` | Phase 1.6 scrum fixes — schema fingerprint hashes name+type+nullable, Gate 4 catches object-literal + class-field bypasses, pyarrow dep gate, item 7 deferral rationale | 4/4 + 3/3 still pass |
+| `f1fa6e4` | **Phase 1.6 Gate 3a** — `crates/catalogd/src/biometric_endpoint.rs`: `POST /biometric/subject/{id}/photo` with consent gate, quarantined storage (mode 0700/0600), audit chain link, `BiometricCollection` field on SubjectManifest | 11 unit tests PASS, live roundtrip 200 |
+| `3708e6a` | Gate 3a scrum fixes — transactional rollback on audit failure (BIPA convergent BLOCK), Content-Type parameter handling, relative data_path, ts+uuid filename, dead code removed | 11 tests + cross-runtime parity 6/6 |
+| `7e0112b` | retention_sweep: stray indent fix on biometric_collection field | sweep tests still 8/8 |
+| `848a458` | **Phase 1.6 Gate 5** — `POST /biometric/subject/{id}/erase` per BIPA destruction runbook. Two scopes (biometric_only / full); audit row appended BEFORE photo unlink so the chain has legal proof of intent even if file delete fails; manifest rolled back on audit failure. Trigger taxonomy: retention_expiry / consent_withdrawal / rtbf / court_order. | 21 unit tests (10 erasure-specific) PASS |
+| `8ec43e0` | **Phase 1.6 Gate 3b** — deepface integration design doc (Option A subprocess / Option B ONNX-in-Rust / **Option C defer**). Recommends C: BIPA-safest, classifications field stays None, all load-bearing surfaces (consent + audit + retention + erasure) ship without it. Forces "do we actually need classifications" to be answered by product, not spec inertia. | doc-only |
+
+**Cross-runtime parity (post-this-wave):** 6 probes, 38/38 byte-identical assertions —
+`validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8) + subject_audit(6/6)`.
+Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify.
+
+**Three runtime-divergence classes caught + fixed by the parity probe authoring loop** (cataloged because they recur):
+1. Go `omitempty` on string fields strips empty values that Rust serde always emits → broken HMAC
+2. `time.RFC3339Nano` truncates trailing-zero nanoseconds where chrono `AutoSi` keeps 9 digits → broken HMAC
+3. Go `json.Marshal` HTML-escapes `<>&` where serde keeps literal → broken HMAC on any field with those chars
+
+All three have regression-locked tests; structural impossibility going forward.
+
+**Phase 1.6 BIPA pre-launch gates — status table:**
+
+| Item | Status | Evidence |
+|---|---|---|
+| Gate 1 — public retention schedule | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_retention_schedule_v1.md` |
+| Gate 2 — informed consent template | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_consent_template_v1.md` |
+| Gate 3a — photo-upload endpoint | **DONE** | 11 unit tests + live `POST /biometric/subject/{id}/photo` |
+| Gate 3b — deepface classification | **design doc shipped** (`8ec43e0`) — recommends Option C (defer); awaits product confirmation | `docs/PHASE_1_6_BIPA_GATES.md` Gate 3b section |
+| Gate 4 — name→ethnicity removal | **DONE** | `mcp-server/phase_1_6_gate_4.test.ts` 4/4 with bypass coverage |
+| Gate 5 — destruction runbook + erasure endpoint | **eng-DONE** (`848a458`); ⚖ counsel review of runbook still pending | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` + `POST /biometric/subject/{id}/erase` (21 tests) |
+| §2 cryptographic attestation | eng-DONE, signature pending | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (SHA-256 evidence hash, 3/3 checks pass on live data) |
+| §3 employee training | deferred | conditional on operator population size |
+
+**Calendar bottleneck:** counsel review of items 1/2/5-runbook/§2 attestation. Engineering long pole is Gate 3b (deepface) — design doc landed (`8ec43e0`); needs product confirmation that classifications are required before engineering starts. Recommendation in doc is Option C (defer) on BIPA-safety grounds.
+
+**Operational state:**
+- `LH_SUBJECT_AUDIT_KEY=/etc/lakehouse/subject_audit.key` (32-byte HMAC signing key, mode 0400) loaded into systemd unit. **Moved off /tmp 2026-05-05** — /tmp wipes on reboot, which on May 5 disabled `/audit` + `/biometric` endpoints (gateway fails-closed at `crates/gateway/src/main.rs:459` if signing key is absent). Persistent path is per spec line 112.
+- `LH_LEGAL_AUDIT_TOKEN_FILE=/etc/lakehouse/legal_audit.token` (44-char legal-tier token, mode 0400) loaded into systemd unit
+- **Key rotation 2026-05-05:** prior key was lost when /tmp wiped on reboot. New key generated at canonical path. The 5 pre-rotation audit chains for `WORKER-{1..5}` (backfill data with `consent=pending_backfill_review`) will tamper-detect under the new key — expected and correct behavior on key rotation, not a bug. New chain entries from 2026-05-05 forward verify cleanly.
+- `data/_catalog/subjects/` holds 100 backfilled `WORKER-N.json` manifests + per-subject `WORKER-N.audit.jsonl` HMAC chains
+- `data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext>` quarantined photo storage (mode 0700 dir / 0600 file). 2 photos uploaded for WORKER-2 during live verify.
+- `/audit/subject/{id}` mounted on gateway with chain_verified=true on every probe
+- `/biometric/subject/{id}/photo` mounted on gateway, refuses 403 without `consent.biometric.status="given"`
+
+---
+
+## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits — Lance gauntlet + cross-runtime parity wave)
+
+| Commit | What | Verified |
+|---|---|---|
+| `5d30b3d` | lance: auto-build doc_id btree in `lance_migrate` handler | doc-fetch ~5ms (was ~100ms full scan) on scale_test_10m |
+| `044650a` | lance-bench: same scalar build post-IVF (matches gateway) | cargo check clean |
+| `7594725` | lance: 4-pack — `sanitize_lance_err` + 7 unit tests + 9-probe smoke + 10M re-bench | smoke 9/9 PASS, tests 7/7 PASS |
+| `98b6647` | gateway: `IterateResponse.trace_id` echoed; session_log_path enabled | parity probes see one unified JSONL |
+| `57bde63` | gateway: trace-id propagation + coordinator session JSONL (Rust parity with Go wave) | session_log_parity 4/4 |
+| `ba928b1` | aibridge: drop Python sidecar from hot path; AiClient → direct Ollama | aibridge tests 32/32 PASS, /ai/embed live 768d |
+| `654797a` | gateway: pub `extract_json` + `parity_extract_json` bin | extract_json_parity 12/12 |
+| `c5654d4` | docs: pointer to `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` | — |
+| `150cc3b` | aibridge: LRU embed cache, 236× RPS warm (78ms → 129us p50) | load test |
+| `9eed982` | mcp-server: /_go/* pass-through for G5 cutover slice | — |
+| `6e34ef7` | gitignore: stop tracking 100+ runtime ephemera (data/_*, lance, logs, node_modules) | untracked dropped 100+ → 0 |
+| `41b0a99` | chore: add 33 real items that were sitting untracked (scripts, scenarios, kimi reports, dev UIs) | clean working tree |
+
+**Cross-runtime parity (post-this-wave):** 32/32 across 5 probes — `validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8)`. Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify.
+
+**Lance backend (was untested 5 days ago, now gauntlet-ready):**
+- `cargo test -p vectord-lance --release` → 7/7 PASS
+- `./scripts/lance_smoke.sh` → 9/9 PASS against live gateway
+- `reports/lance_10m_rebench_2026-05-02.md` — search warm ~20ms / cold ~46ms median, doc-fetch ~5ms post-btree
+
+---
+
+## VERIFIED WORKING RIGHT NOW
+
+### The client demo (Staffing Co-Pilot)
+
+**Public URL:** `https://devop.live/lakehouse/` — 200, "Staffing Co-Pilot" (159 KB SPA, leaflet maps, dark theme).
+**Local URL:** `http://localhost:3700/` — same page, served by `mcp-server/index.ts` (PID 1271, started 09:48 CDT today).
+
+**The staffers console** (the one the client was thoroughly impressed with):
+- `https://devop.live/lakehouse/console` — 200, "Lakehouse — What Your Staffing System Would Do" (26 KB)
+- Pulls project index via `/api/catalog/datasets` (36 datasets) + playbook memory via `/api/vectors/playbook_memory/stats` (4,701 entries with embeddings, real ops like *"fill: Maintenance Tech x2 in Milwaukee, WI"*)
+
+Client-visible flow that works end-to-end on the public URL:
+
+| Endpoint | Sample output |
+|---|---|
+| `GET /api/catalog/datasets` | 36 datasets indexed: timesheets 1M, call_log 800K, workers_500k 500K, email_log 500K, workers_100k 100K, candidates 100K, placements 50K, job_orders 15K, successful_playbooks_live 2,077 |
+| `GET /api/vectors/playbook_memory/stats` | 4,701 fill operations with embeddings |
+| `GET /system/summary` | 36 datasets, 2.98M rows, 60 indexes, 500K workers loaded, 1K candidates |
+| `POST /intelligence/staffing_forecast` | 744 Production Workers needed in 30d, 11,281 bench (4,687 reliable), coverage 1,444%, risk=ok. Same for Electrician (need 32, bench 2,440) and Maintenance Tech (need 17, bench 5,004). |
+| `POST /intelligence/permit_contracts` | permit `3442956` $500K → 3 Production Workers, 886-candidate pool, 95% fill, $36K gross. 5 more Chicago permits with 8 workers each, same pool, 95% fill, $96K each. |
+| `POST /intelligence/market` | major Chicago permits ranked: $730M O'Hare, $615M 307 N Michigan, $580M casino, $445M Loop transit (real geo coords). |
+| `POST /intelligence/permit_entities` | architects + contractors from permit contacts (e.g. "KACPRZYNSKI, ANDY", "SLS ELECTRICAL SERVICE"). |
+| `POST /intelligence/activity` + `/intelligence/arch_signals` + `/intelligence/chat` | all 200 |
+
+The demo tells the story: *"upcoming Chicago contracts → workers needed → coverage from the bench → architects/contractors involved → revenue and margin."* That's the "live data + anticipating contracts + complete workflow" pitch — working as of right now.
+
+### Backend, verified live this session
+
+| Surface | State |
+|---|---|
+| Gateway `:3100` | up, 4 providers configured, `/v1/health` 200 with 500K workers loaded |
+| MCP server `:3700` (Co-Pilot demo) | up, all `/intelligence/*` endpoints respond |
+| VCP UI `:3950` | started this session, `/data/*` 200, real numbers |
+| Observer `:3800` | ring full (2,000/2,000) — older events evicted, query Langfuse for 24h-ago state |
+| Sidecar `:3200` | up |
+| Langfuse `:3001` | recording, `gw:/log` + `v1.chat:openrouter` traces visible |
+| LLM Team UI `:5000` | up, only `extract` mode registered |
+| OpenCode fleet | **40 models reachable through one `sk-*` key** (verified live `GET https://opencode.ai/zen/v1/models`) |
+
+OpenCode catalog (live):
+- Claude: opus-4-7, opus-4-6, opus-4-5, opus-4-1, sonnet-4-6, sonnet-4-5, sonnet-4, haiku-4-5
+- GPT-5: 5.5-pro, 5.5, 5.4-pro, 5.4, 5.4-mini, 5.4-nano, 5.3-codex-spark, 5.3-codex, 5.2, 5.2-codex, 5.1-codex-max, 5.1-codex, 5.1-codex-mini, 5.1, 5-codex, 5-nano, 5
+- Gemini: 3.1-pro, 3-flash
+- GLM: 5.1, 5
+- Minimax: m2.7, m2.5
+- Kimi: k2.6, k2.5
+- Qwen: 3.6-plus, 3.5-plus
+- Other: BIG-PKL (was a typo-prone name in the catalog, model id starts with "big-pkl-something")
+- Free tier: minimax-m2.5-free, hy3-preview-free, ling-2.6-flash-free, trinity-large-preview-free
+
+### The substrate (frozen — do not re-architect)
+
+- Distillation v1.0.0 at tag `e7636f2` — **145/145 bun tests pass, 22/22 acceptance, 16/16 audit-full**
+- Output: `data/_kb/distilled_{facts,procedures,config_hints}.jsonl` + `data/vectors/distilled_{factual,procedural,config_hint}_v20260423102847.parquet`
+- Auditor cross-lineage: Kimi K2.6 ↔ Haiku 4.5 alternation, Opus auto-promote on diffs >100k chars, **per-PR cap=3 with auto-reset on new head SHA**
+- Pathway memory: 88 traces, 11/11 successful replays (probation gate crossed)
+- Mode runner: 5 native modes; `codereview_isolation` is default; composed-corpus auto-downgrade verified Apr 26 (composed lost 5/5 vs isolation, p=0.031)
+
+### Matrix indexer
+
+30+ live corpora including:
+- 5 versions of `workers_500k_v1..v9` (50K embedded chunks each)
+- 11 batched 2K-row shards `w500k_b3..b17`
+- `chicago_permits_v1` (3,420), `resumes_100k_v2` (100K candidates), `ethereal_workers_v1` (10K)
+- `lakehouse_arch_v1` (2,119), `lakehouse_symbols_v1` (2,470), `lakehouse_answers_v1` (1,269), `scrum_findings_v1` (1,260)
+- `kb_team_runs_v1` (12,693) + `kb_team_runs_agent` (4,407) — LLM-team play history embedded
+- `distilled_factual_v20260423102507` (8) — distillation output
+
+### Code health
+
+- `cargo check --workspace` → **0 warnings, 0 errors**
+- `bun test auditor + tests/distillation` → **145/145 pass**
+- `ui/server.ts` + `auditor.ts` bundle clean
+
+---
+
+## DO NOT RELITIGATE
+
+- **PR #11 is merged into `origin/main` as `ed57eda`** — do not "still need to merge PR #11."
+- **Distillation tag `distillation-v1.0.0` at `e7636f2` is FROZEN** — do not re-architect schemas, scorer rules, audit fixtures.
+- **Kimi forensic HOLD verdict (2026-04-27) was 2/8 false + 6/8 latent** — do not re-debate, see `reports/kimi/audit-last-week-full.md`.
+- **`candidates_safe` `vertical` column bug** — fixed at catalog metadata layer in commit `c3c9c21`. Do not "discover" it again.
+- **Decisions A/B/C/D from `synthetic-data-gap-report.md`** — all four scripts shipped today (`d56f08e`, `940737d`, `c3c9c21`). Do not "ask J for approval."
+- **`workers_500k.phone` type fixup** — already string. The fixup script is idempotent; running it is a no-op.
+- **`client_workerskjkk` typo dataset** — was breaking every SQL query (catalog had it registered, file didn't exist). Removed via `DELETE /catalog/datasets/by-name/client_workerskjkk` this session. Do not re-add. Adding a startup gate that errors on unrecognized parquet names is the long-term fix per now.md Step 2C.
+- **Python sidecar dropped from hot path 2026-05-02 (`ba928b1`)** — AiClient calls Ollama directly. Do not "wire python embedding back in." `lab_ui.py` + `pipeline_lab.py` keep running as dev-only UIs (not on the runtime path).
+- **Lance backend gauntlet (2026-05-02)** — sanitizer over all 5 routes, 7 unit tests, 9-probe smoke, 10M re-bench. The `doc_id` btree auto-builds inside `lance_migrate` AND `lance-bench`. Do not "discover" the missing scalar index again or the leaked filesystem paths in error bodies.
+- **Cross-runtime parity = 32/32** across 5 probes in `golangLAKEHOUSE/scripts/cutover/parity/`. Do not "build a parity probe for X" without checking — validator, extract_json, session_log, materializer, and embed are all already covered.
+- **Decisions tracker is `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`** — single living source of truth for cross-runtime decisions. As of 2026-05-02 it has 0 `_open_` code work items; only 2 strategic items left (Lance vs Parquet+HNSW-with-spilling, Go-vs-Rust primary cutover).
+- **PRD line 70 is load-bearing — "Everything runs locally, no cloud APIs."** Yesterday's PR #13 violated this by routing customer hot-path inference to opencode/openrouter/ollama_cloud. **REVERTED 2026-05-03 (`d054c0b`).** The customer hot path (modes.toml staffing_inference, doc_drift_check; execution_loop overseer escalation) is now local Ollama (qwen3.5:latest). Cloud providers stay configured in providers.toml for **explicit dev-tool opt-in only** (scrum, auditor, bot/propose). Do NOT re-add cloud models to the hot path defaults — the customer demo runs on local + free.
+- **`/v1/usage` shows by_provider=ollama only** for any customer-shape request. If you see by_provider including kimi/openrouter/opencode/ollama_cloud for normal /v1/iterate or /v1/respond traffic, something has regressed. Verify with: `curl -sf http://127.0.0.1:3100/v1/usage | jq .by_provider`.
+- **`./scrum` is a TOOL, not architecture.** Lives at repo root. J runs it manually. Outputs findings to `data/_kb/scrum_findings.jsonl` (the KB). Findings inform development; they do NOT auto-fold into PRDs/design docs/code. If you find yourself proposing to "wire scrum into [some pipeline]" — stop. It's J's tool.
+- **Test code in main is ACTIVELY being cleaned out.** 2026-05-03 commits `6aafd41` + `f4ebd22` removed 12 orphaned dev experiments (~2900 LOC) from `tests/real-world/` and `scripts/`. If you find more zero-reference experimental files, surface them — don't auto-delete unless the pattern is clearly a one-time-experiment-with-zero-consumers like the ones already removed.
+
+---
+
+## FIXES MADE THIS SESSION (2026-04-27 evening)
+
+1. **`crates/gateway/src/v1/iterate.rs:93`** — `state` → `_state` (cleared the one cargo warning).
+2. **`lakehouse-ui.service` (Dioxus)** — disabled. Was failing 7,242 times against a missing `target/dx/ui/debug/web/public` build dir. `systemctl stop && disable`.
+3. **VCP UI on `:3950`** — started `bun run ui/server.ts` (PID 1162212, log `/tmp/lakehouse_ui.log`). `/data/*` endpoints now 200 with real data.
+4. **`client_workerskjkk` catalog entry** — `DELETE /catalog/datasets/by-name/client_workerskjkk` removed the dead manifest. **This was the actual root cause** of `/system/summary` reporting `workers_500k_rows: 0` and the demo showing zero bench. Every SQL query was failing schema inference on the missing file before reaching its target table. Fixed → `workers_500k_rows: 500000`, `candidates_rows: 1000`, demo coverage flipped from "critical 0%" to actual percentages on devop.live/lakehouse.
+
+## FIXES MADE THIS SESSION (2026-04-28 early — face pool)
+
+5. **Synthetic StyleGAN face pool — 1000 faces, gender+race+age tagged.** `scripts/staffing/fetch_face_pool.py` fetches from thispersondoesnotexist.com; `scripts/staffing/tag_face_pool.py --min-age 22` runs deepface and excludes minors. `data/headshots/manifest.jsonl` now has gender (494 men / 458 women), race (caucasian 662 · east_asian 128 · hispanic 86 · middle_eastern 59 · black 14 · south_asian 3), age, and 48 minor exclusions. Server pool = 952 servable faces.
+6. **`mcp-server/index.ts:1308` `/headshots/:key` route** — gender×race×age intersection bucketing with graceful fallback (gender-only → all). Same key always returns same face; different keys spread evenly.
+7. **`/headshots/_thumbs/` pre-resized 384×384 webp** (60× smaller: 587KB → ~11KB). Without this, 40-card grids overran Chrome's parallel-connection budget and ~75% of tiles never finished decoding. Generated via parallel ffmpeg (`xargs -P 8`); `.gitignore`d.
+8. **`mcp-server/search.html` + `console.html`** — dropped `img.loading='lazy'`. With 11KB thumbs, eager load is cheap (~500KB for 50 cards) and avoids the off-screen race that lazy decode produced.
+9. **ComfyUI on-demand uniqueness — `serve_imagegen.py:32`** added `seed` to `_cache_key()` (was caching by prompt only — 3 different worker seeds collapsed to 1 cached image). Verified: seed=839185194/195/196 → 3 distinct md5s.
+10. **`mcp-server/index.ts:1234` `/headshots/generate/:key`** — ComfyUI hot-path that derives a deterministic-per-worker seed via djb2-style hash; cold ~1.5s, cached ~1ms. Worker prompt format: `professional corporate headshot portrait of a {age}-year-old {race} {gender}, {role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`. Cache at `data/headshots_gen/` (gitignored, regeneratable).
+11. **Confidence-default name resolution** in `search.html` — `genderFor()` and `guessEthnicityFromFirstName()` lookup tables (FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN). Xavier → man+hispanic, Aisha → woman+black, etc. Every worker resolves to a face-pool bucket.
+
+End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift+operators+IL` → 21/21 cards loaded, 0 broken, all 384×384 webp thumbs.
+
+---
+
+## ⚠ PRODUCTION-READY BLOCKER (2026-05-03)
+
+**Audit-trail capability is the gate to client signature.** Smoke + parity tests prove the surface compiles; they do NOT prove an audit response can be produced for a specific person. Staffing client won't sign without defensible discrimination-claim response capability.
+
+**Authoritative document:** `docs/AUDIT_TRAIL_PRD.md` — drafted 2026-05-03. Defines worked example (John Martinez at Warehouse B), the per-decision audit row schema, the surface map of where decisions happen today, current-state-vs-target gap table, and 9-phase implementation sequence.
+
+**Phase 1 (discovery walk) requires NO J approval — it's read-only.** Phases 2+ have explicit open questions in §10 of the PRD that need J's call before they can start.
+
+Until phase 9 exit criterion is met, **do not claim "production-ready" on customer-facing surfaces.** Internal substrate (Lance, sidecar drop, parity probes) is solid; subject-of-record audit story is not.
+
+---
+
+## OPEN — but not blocking the demo
+
+| Item | What | When to act |
+|---|---|---|
+| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
+| Open PRs #6, #7, #10 | All closed 2026-05-02 — superseded / empty / stale. PR #12 merged 2026-05-03 (`a5d9070`). PR #13 merged 2026-05-03 (`feb638e`). | Done. |
+| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
+| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
+| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
+
+---
+
+## RUNTIME CHEATSHEET
+
+```bash
+# Verify the demo (public + local both work)
+curl -sS https://devop.live/lakehouse/                   # Co-Pilot HTML
+curl -sS https://devop.live/lakehouse/console            # staffers console
+curl -sS -X POST https://devop.live/lakehouse/intelligence/staffing_forecast \
+  -d '{}' -H 'content-type: application/json' \
+  | jq '.forecast[] | {role, demand_workers, bench_total, coverage_pct, risk}'
+
+# Restart sequence (after Rust changes)
+sudo systemctl restart lakehouse.service                 # gateway :3100
+sudo systemctl restart lakehouse-auditor                 # auditor daemon
+sudo systemctl restart lakehouse-observer                # observer :3800
+# UI bun on :3950 is NOT systemd-managed (lakehouse-ui.service is disabled).
+# Restart manually:  kill <pid>; nohup bun run ui/server.ts > /tmp/lakehouse_ui.log 2>&1 &
+
+# Health checks
+curl -sS http://localhost:3100/v1/health | jq            # workers_count, providers
+curl -sS http://localhost:3100/vectors/pathway/stats | jq
+curl -sS http://localhost:3100/v1/usage | jq             # since-restart cost
+curl -sS http://localhost:3700/system/summary | jq       # dataset counts
+```
+
+---
+
+## VISION — what we're actually building (not what's done)
+
+J's framing for the legacy staffing company:
+
+- Pull live data, anticipate contracts based on Chicago permits → real architect/contractor associations, headcount, time period, money, scope.
+- Hybrid + memory index → search large corpora cheaply.
+- Email comes in → verify against contract; SMS comes in → alert when index changes.
+- Real-time.
+- Invent metrics nobody else has using the hybrid index.
+- Next stage: workers download an app → geolocation clock-in → automatic responsiveness measurement, no user effort, with incentives for using it.
+- Find people getting certificates (passive cert tracking).
+- Pull union data → bring contracts that work for **employees**, not just employers.
+- All metrics visible, nothing hidden, value-aligned with what each side actually needs.
+
+If a future session is shaving away from this vision toward "fix the cutover" or "land Phase X," the vision wins. Phases are scaffolding for the vision, not the goal.
+
+---
+
+## CURRENT PLAN — fix the demo for the legacy staffing client
+
+Built from playwright audit of the live demo (2026-04-27 evening). Each item ends in something the client can SEE, not internal cleanups.
+
+**Demo state is anchored by git tag `demo-2026-04-27`** (commit `ed57eda`, the merge of PR #11). To restore code state: `git checkout demo-2026-04-27`. To restore runtime state: `DELETE /catalog/datasets/by-name/client_workerskjkk` (catalog hot-fix is not in git).
+
+### P1 — Search box that actually filters (highest visible impact)
+
+**Problem:** typing in `#sq` and pressing Enter fires `POST /intelligence/chat` with body `{"message":"<query>"}`. The state (`#sst`) and role (`#srl`) selects are ignored — never sent in the body. So every search returns a generic chat completion, never a SQL+vector hybrid filter against `workers_500k`. That is the "cached/generic response" the client sees.
+
+**Fix:** in `mcp-server/search.html`, change the search-submit handler to call the real worker search endpoint with `{query, state, role, top_k}`. The MCP `search_workers` tool surface already exists; route the form there. Render returned worker rows in the existing card grid.
+
+**Done when:** typing "forklift" + state IL + role "Forklift Operator" returns ≤ top_k IL Forklift Operators, and changing state to WI returns different workers.
+
+### P2 — Contractor-name click → `/contractor` profile page
+
+**Problem:** clicking a contractor name in any rendered card stays on `/lakehouse/`. URL doesn't change.
+
+**Fix:** wrap contractor names in `<a href="/contractor?name=<encoded>">`. The page `mcp-server/contractor.html` (14.8 KB, "Contractor Profile · Staffing Co-Pilot") already exists at `/contractor` and the data endpoint `/intelligence/contractor_profile` already returns rich data.
+
+**Then check contractor.html actually shows:** full history of every record the database has on that contractor + heat map of locations underneath + relevant info (per J 2026-04-27). If the page is incomplete, finish it. Otherwise just wire the link.
+
+**Done when:** clicking "KACPRZYNSKI, ANDY" opens a profile with: every Chicago permit they're contact_1 or contact_2 on, a leaflet map with markers for each address, and any matched workers from prior placements at their sites.
+
+### P3 — Substrate signal at the bottom shows the right numbers
+
+**Problem:** J reports the bottom panel says "playbook memory empty, 80 traces 0 replies." Reality from the live endpoints: `/api/vectors/playbook_memory/stats` = 4,701 entries with embeddings; `/vectors/pathway/stats` = 88 traces, 11/11 replays.
+
+**Fix:** find the renderer in search.html that builds the substrate signal panel; verify it's hitting the right endpoints and reading the right keys; fix shape mismatches.
+
+**Done when:** bottom panel shows real numbers (4,701 playbooks, 88 traces, 11/11 replays) and references at least one specific recent operation from the playbook stats sample.
+
+### P4 — Top nav reflects today's architecture
+
+**Problem:** Walkthrough/Architecture/Spec/Onboard/Alerts/Workspaces tabs all return 200 but content is from old architecture. Doesn't mention: gateway scratchpad, memory indexer, ranker, mode runner, OpenCode 40-model fleet, distillation substrate, auditor cross-lineage.
+
+**Fix:** rewrite `mcp-server/proof.html` (or add a single new page "What's running" that replaces Architecture+Spec) to describe what's actually shipped as of `demo-2026-04-27`. Keep one architecture page, drop redundancy. Either complete or hide Onboard/Alerts/Workspaces — J's call which.
+
+**Done when:** the architecture page tells a non-technical reader, in 2 minutes, what each piece does in coordinator-relatable terms ("intern that read every email", not "3-stage adversarial inference pipeline").
+
+### P5 — Caching for the project-index build_signal (J flagged unfinished)
+
+**Problem:** "we never finished our caching for project index build signal it's not pulling new information." Need to find what `build_signal` refers to. Likely a scrum/auditor signal that should rebuild the `lakehouse_arch_v1` corpus on commit but isn't wired to.
+
+**Fix:** identify the build-signal pipeline (likely in `auditor/` or `crates/vectord/`), wire its emit to a corpus rebuild, verify by making a test commit and watching the new chunk appear in `/vectors/indexes` for `lakehouse_arch_v1`.
+
+**Done when:** committing a new file to `crates/` causes `lakehouse_arch_v1` chunk_count to increase within N minutes.
+
+### P0 — Anchor the demo state (DONE)
+
+Tagged `ed57eda` as `demo-2026-04-27`. Future sessions: `git checkout demo-2026-04-27` to land in this exact code state.
+
+---
+
+## EXECUTION ORDER
+
+1. **P1 first** — biggest visible bug, ~30-60 min
+2. **P2 next** — contractor click is the second-biggest "doesn't work" the client sees, ~20 min if profile is mostly done
+3. **P3** — small fix, big "looks alive" win
+4. **P4** — biggest scope; might split across sessions
+5. **P5** — feature work, only after the visible bugs are fixed
+
+Each item commits independently with the format `demo: P<n> — <one-line>` so the commit log doubles as a progress journal. After each merge to main, re-tag `demo-latest` to point at the new HEAD.
+
+Stop here and let J pick which item to start with. Do not silently extend scope.

auditor/audit.ts

@@ -23,6 +23,7 @@ import { runStaticCheck } from "./checks/static.ts";
 import { runDynamicCheck } from "./checks/dynamic.ts";
 import { runInferenceCheck } from "./checks/inference.ts";
 import { runKbCheck } from "./checks/kb_query.ts";
+import { runKimiArchitectCheck } from "./checks/kimi_architect.ts";
 
 const VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/verdicts";
 // Playbook for audit findings — one row per block/warn finding from a
@@ -67,6 +68,29 @@ export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<
     ...kbFindings,
   ];
 
+  // Kimi-architect second-pass review. Off by default; enabled with
+  // LH_AUDITOR_KIMI=1. Sequential (not in the parallel block above)
+  // because it consumes the prior findings as context — Kimi sees what
+  // deepseek already flagged and is asked "what did everyone miss?"
+  // Failure-isolated by design: any error returns a single info-level
+  // skip finding so the existing audit pipeline never blocks on Kimi.
+  if (process.env.LH_AUDITOR_KIMI === "1") {
+    try {
+      const kimiFindings = await runKimiArchitectCheck(diff, allFindings, {
+        pr_number: pr.number,
+        head_sha: pr.head_sha,
+      });
+      allFindings.push(...kimiFindings);
+    } catch (e) {
+      allFindings.push({
+        check: "kimi_architect",
+        severity: "info",
+        summary: `kimi_architect outer error — ${(e as Error).message.slice(0, 160)}`,
+        evidence: [(e as Error).stack?.slice(0, 360) ?? ""],
+      });
+    }
+  }
+
   const duration_ms = Date.now() - t0;
   const metrics = {
     audit_duration_ms: duration_ms,
@@ -184,7 +208,7 @@ function formatReviewBody(v: Verdict): string {
   lines.push("");
 
   // Per-check sections, only if the check produced findings.
-  const checkOrder = ["static", "dynamic", "inference", "kb_query"] as const;
+  const checkOrder = ["static", "dynamic", "inference", "kb_query", "kimi_architect"] as const;
   for (const check of checkOrder) {
     const fs = byCheck[check] ?? [];
     if (fs.length === 0) continue;
@@ -217,6 +241,6 @@ function formatReviewBody(v: Verdict): string {
   return lines.join("\n");
 }
 
-function stubFinding(check: "dynamic" | "inference", why: string): Finding[] {
+function stubFinding(check: "dynamic" | "inference" | "kimi_architect", why: string): Finding[] {
   return [{ check, severity: "info", summary: `${check} check skipped — ${why}`, evidence: [why] }];
 }

auditor/checks/inference.ts

@@ -18,36 +18,42 @@ import { readFile, mkdir, appendFile } from "node:fs/promises";
 import { extractFacts } from "../fact_extractor.ts";
 
 const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
-const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "gpt-oss:120b";
-// Tie-breaker for claims where the N=3 consensus produces a 1-1-1
-// split (genuinely borderline). Different architecture from the
-// primary reviewer (gpt-oss) so the tie-break isn't correlated with
-// the original disagreement. qwen3-coder:480b is a newer coding
-// specialist at 480B params, well-suited to PR-diff claim verification
-// and distinct in training lineage from gpt-oss.
-const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "qwen3-coder:480b";
+// Rebuild 2026-04-26: route claim verification through /v1/mode/execute
+// (task_class=pr_audit) so we get pathway memory + lakehouse_answers_v1
+// + JSON-shaped framing molded into ONE prompt. The hand-rolled
+// systemMsg/userMsg path was reinventing the mode runner badly.
+//
+// 2026-04-27 update: original default kimi-k2:1t hit a sustained
+// upstream outage on Ollama Cloud (consistent 500 ISE across hours of
+// retries — verified with trivial 8-token probes). Swapped default to
+// deepseek-v3.1:671b which is proven working end-to-end through the
+// pr_audit mode runner during Phase 5 distillation acceptance testing.
+//
+// 2026-09-02 update: Ollama Cloud subscription lapsed — every
+// ollama_cloud model 402s ("requires a subscription"). J's call: sweep
+// the auditor to OpenRouter. Same deepseek lineage via OpenRouter;
+// tie-breaker moves to grok-4.3 because OpenRouter 404s grok-4.1-fast
+// as deprecated. Both probed live through the gateway before landing.
+// Vendor lineages stay distinct so consensus + tie-break won't
+// fail-correlate.
+const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "openrouter/deepseek/deepseek-chat-v3.1";
+const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "openrouter/x-ai/grok-4.3";
 const N_CONSENSUS = Number(process.env.LH_AUDITOR_CONSENSUS_N ?? 3);
 const AUDIT_DISCREPANCIES_JSONL = "/home/profit/lakehouse/data/_kb/audit_discrepancies.jsonl";
-// 40KB comfortably fits gpt-oss:120b's context. PR #1 (~39KB) was
-// previously truncated at 15KB causing the reviewer to miss later
-// files (gitea.ts, policy.ts) and flag "no Gitea client present" as a
-// block finding when the file was simply outside the truncation window.
-//
-// Above this threshold we curate via tree-split rather than truncate,
-// following the scrum_master pattern: shard the diff, summarize each
-// shard against the claim-verification task, merge into a compact
-// scratchpad, then ask the cloud to verify claims against the
-// scratchpad. This gives the cloud full-PR fidelity without bursting
-// its context window (observed failure mode: empty response or
-// unparseable output when prompt exceeds model's comfortable range).
+// 40KB comfortably fits the consensus models' context windows
+// (deepseek-v3.1 64K, gpt-oss-120b 128K). When the raw PR diff
+// exceeds this, we truncate and signal it via curationNote — the
+// pr_audit mode runner's matrix retrieval (lakehouse_answers_v1 +
+// arch + symbols) supplies the cross-PR context that tree-split
+// used to synthesize from scratch. Tree-split itself was retired
+// 2026-04-27 (see commit deleting treeSplitDiff/callCloud/SHARD_*).
 const MAX_DIFF_CHARS = 40000;
-// Tree-split kicks in above this. 30KB is below MAX_DIFF_CHARS so we
-// curate BEFORE truncation would happen — never lose signal to a hard
-// cut. Shard size is chosen so ~10 shards cover PR #8-size diffs in a
-// reasonable round-trip budget.
-const CURATION_THRESHOLD = 30000;
-const DIFF_SHARD_SIZE = 4500;
 const CALL_TIMEOUT_MS = 120_000;
+// Mode runner can take longer than a raw /v1/chat call because it does
+// pathway-fingerprint lookup + matrix retrieval + relevance filter
+// before the LLM call. Budget extra time so we don't trip on a slow
+// answers-corpus search.
+const MODE_RUNNER_TIMEOUT_MS = 240_000;
 const REPO_ROOT = "/home/profit/lakehouse";
 
 export interface InferenceContext {
@@ -86,26 +92,23 @@ export async function runInferenceCheck(
     }];
   }
 
-  // Diff source for the cloud prompt — either the raw diff (small
-  // enough to fit), or a tree-split scratchpad (curation layer). We
-  // prefer curation to truncation: truncation silently drops files
-  // past the window; curation summarizes them so the cloud still sees
-  // what changed, just densified.
-  let diffForPrompt: string;
-  let curationNote = "";
-  if (diff.length > CURATION_THRESHOLD) {
-    const ts = await treeSplitDiff(diff, verifiable);
-    diffForPrompt = ts.scratchpad;
-    curationNote = ` (curated: ${diff.length} chars → ${ts.shards} shards → scratchpad ${ts.scratchpad.length} chars)`;
-  } else {
-    diffForPrompt = diff;
-  }
-  // Belt-and-suspenders truncation — even a tree-split scratchpad
-  // shouldn't exceed MAX_DIFF_CHARS in practice, but guard anyway so
-  // pathological inputs can't burst the prompt.
-  const truncated = diffForPrompt.length > MAX_DIFF_CHARS
-    ? diffForPrompt.slice(0, MAX_DIFF_CHARS) + `\n...[${diffForPrompt.length - MAX_DIFF_CHARS} more chars truncated]`
-    : diffForPrompt;
+  // 2026-04-27 architecture simplification: dropped the tree-split
+  // scratchpad layer. Rationale: the mode runner's pr_audit pipeline
+  // pulls from lakehouse_answers_v1 (gold-standard prior audits) +
+  // lakehouse_arch_v1 + lakehouse_symbols_v1 via matrix retrieval. That
+  // corpus IS the cross-PR context the tree-split was synthesizing
+  // from scratch on every audit run. With the distillation substrate
+  // shipped (commits 27b1d27..1b433a9), per-shard fact extraction is
+  // redundant — and gpt-oss:120b at 168 calls/audit was the dominant
+  // cost. Now: truncate diff to MAX_DIFF_CHARS, hand straight to the
+  // mode runner, let retrieval supply context. ONE strong-model call
+  // per consensus rep × N=3 reps = 3 calls total per audit.
+  const truncated = diff.length > MAX_DIFF_CHARS
+    ? diff.slice(0, MAX_DIFF_CHARS) + `\n...[${diff.length - MAX_DIFF_CHARS} more chars truncated — the pr_audit mode runner has matrix retrieval against lakehouse_answers_v1 + arch + symbols for cross-PR context]`
+    : diff;
+  const curationNote = diff.length > MAX_DIFF_CHARS
+    ? ` (truncated ${diff.length}→${MAX_DIFF_CHARS} chars; matrix retrieval supplies cross-PR context)`
+    : "";
 
   // Build the reviewer prompt in the same shape as run_codereview's
   // review stage (llm_team_ui.py:10950), adapted for claim verification:
@@ -114,79 +117,20 @@ export async function runInferenceCheck(
   //   "Review: bugs/security/perf/style/edge. Provide corrected code."
   // We add: claim list upfront + ask for structured JSON verdict.
   //
-  // When the diff was curated (tree-split scratchpad), we add an
-  // explicit anti-false-positive instruction: the scratchpad is a
-  // distillation, not the full source, so absence-from-scratchpad is
-  // NOT evidence of absence-from-diff. Mirrors the fix we made in
-  // scrum_master's review prompt for the same class of error.
+  // Curation flag is now just a truncation flag — when the diff was
+  // cut, tell the reviewer it didn't see the full picture so it doesn't
+  // confidently mark a claim NOT BACKED based on absence in the
+  // (potentially incomplete) input.
   const isCurated = curationNote.length > 0;
-  const curationGuard = isCurated
-    ? [
-        "",
-        "CRITICAL: the 'Diff' below is a curated multi-shard scratchpad,",
-        "NOT the full raw diff. The scratchpad distills each shard down",
-        "to facts useful for claim verification and drops the rest.",
-        "DO NOT flag a function/field/feature as 'missing' or 'not",
-        "implemented' based solely on its absence from the scratchpad —",
-        "absence in a distillation is NOT evidence of absence in the",
-        "actual diff. Only judge a claim NOT BACKED when the scratchpad",
-        "DIRECTLY contradicts it (e.g. scratchpad shows the function was",
-        "added empty, or shows the claimed code path is a stub).",
-        "Skip the unflagged_gaps section entirely when operating on a",
-        "curated scratchpad — you can't reliably detect gaps from a",
-        "distillation, and false positives there are worse than misses.",
-      ].join("\n")
-    : "";
-  const systemMsg = [
-    "You review pull-request diffs against the author's own ship-claims.",
-    "For each claim, decide: is it backed by actual code in the diff, or is",
-    "it placeholder / aspirational / unwired?",
-    "",
-    "A claim is BACKED when the diff contains a real code path that delivers",
-    "the claimed behavior. A claim is NOT BACKED when:",
-    "  - the claim asserts functionality but the diff only adds types/fields",
-    "    with no consumer",
-    "  - the claim mentions tests but no test function was added",
-    "  - the claim claims integration but the integration point is a stub",
-    "  - the diff contains unimplemented!() / todo!() / TODO comments",
-    "  - the claim says 'works end-to-end' but the diff has no end-to-end test",
-    curationGuard,
-    "",
-    "Respond with strict JSON only. No prose before or after. Shape:",
-    "{",
-    '  "claim_verdicts": [',
-    '    {"claim_idx": 0, "backed": false, "evidence": "short reason"}',
-    "  ],",
-    '  "unflagged_gaps": [',
-    '    {"location": "file:line", "summary": "short description"}',
-    "  ]",
-    "}",
-  ].join("\n");
+  const prNumber = ctx?.pr_number ?? 0;
 
-  const userMsg = [
-    `Ship-claims the author made (numbered 0..N-1):`,
-    verifiable.map((c, i) => `  ${i}. [${c.strength}] "${c.text}" at ${c.location}`).join("\n"),
-    "",
-    `Diff:`,
-    "```",
-    truncated,
-    "```",
-    "",
-    `For each numbered claim above, emit a claim_verdicts entry. For gaps the`,
-    `author DIDN'T claim but that look like placeholder code, emit unflagged_gaps.`,
-    `Strict JSON only, matching the shape described. No prose outside JSON.`,
-  ].join("\n");
-
-  // N=3 consensus — run the primary reviewer in parallel, collect
-  // all three parsed responses, majority-vote per claim. Parallel
-  // (Promise.all) because each call is ~20-30s and they're independent;
-  // wall-clock stays ~same as single call, cost 3x tokens. Empirical
-  // justification: in 3-run determinism tests, 7/8 findings were
-  // stable but 1 flipped across runs — majority vote stabilizes the
-  // flipping class without losing the stable signal.
+  // N=3 consensus — fire the mode runner three times in parallel.
+  // Each /v1/mode/execute call composes pathway memory + answers corpus
+  // + JSON-shaped pr_audit framing internally, so the auditor's only
+  // job here is to vote-aggregate. Wall-clock ~= single call.
   const primaryRuns = await Promise.all(
     Array.from({ length: N_CONSENSUS }, () =>
-      runCloudInference(systemMsg, userMsg, MODEL)),
+      runModeRunnerInference(truncated, verifiable, prNumber, isCurated, MODEL)),
   );
 
   const parsedRuns = primaryRuns.filter(r => r.parsed !== null);
@@ -209,9 +153,19 @@ export async function runInferenceCheck(
   interface Votes { trues: number; falses: number; evidences: string[] }
   const votesByClaim = new Map<number, Votes>();
   const unflaggedByRun: any[][] = [];
-  let totalTokens = 0;
+  // The N=3 consensus calls run via Promise.all — wall-clock is
+  // bounded by the SLOWEST call, not the sum. Pre-2026-04-27 we
+  // summed and reported "Xms total" which double/triple-counted
+  // (Opus self-audit caught it). Use max for accurate wall-clock.
+  let maxLatencyMs = 0;
+  let totalEnrichedChars = 0;
+  let bugFingerprintsSeen = 0;
+  let matrixKeptSeen = 0;
   for (const run of parsedRuns) {
-    totalTokens += run.tokens;
+    maxLatencyMs = Math.max(maxLatencyMs, run.latency_ms ?? 0);
+    totalEnrichedChars += run.enriched_chars ?? 0;
+    bugFingerprintsSeen = Math.max(bugFingerprintsSeen, run.bug_fingerprints ?? 0);
+    matrixKeptSeen = Math.max(matrixKeptSeen, run.matrix_kept ?? 0);
     unflaggedByRun.push(Array.isArray(run.parsed?.unflagged_gaps) ? run.parsed.unflagged_gaps : []);
     for (const v of run.parsed?.claim_verdicts ?? []) {
       const idx = Number(v?.claim_idx);
@@ -233,10 +187,11 @@ export async function runInferenceCheck(
   findings.push({
     check: "inference",
     severity: "info",
-    summary: `cloud review completed (model=${MODEL}, consensus=${parsedRuns.length}/${N_CONSENSUS}, tokens=${totalTokens})${curationNote}`,
+    summary: `pr_audit mode runner completed (model=${MODEL}, consensus=${parsedRuns.length}/${N_CONSENSUS}, ${maxLatencyMs}ms wall-clock)${curationNote}`,
     evidence: [
       `claims voted: ${votesByClaim.size}`,
       `parsed runs: ${parsedRuns.length} / ${N_CONSENSUS}`,
+      `enrichment: ${bugFingerprintsSeen} bug fingerprints, ${matrixKeptSeen} answers-corpus chunks, prompt avg ${Math.round(totalEnrichedChars / Math.max(parsedRuns.length, 1))} chars`,
     ],
   });
 
@@ -266,8 +221,9 @@ export async function runInferenceCheck(
       notBacked = false;
       resolution = "majority_backed";
     } else {
-      // Tie. Run tie-breaker with a different-architecture model.
-      const tb = await runCloudInference(systemMsg, userMsg, TIEBREAKER_MODEL);
+      // Tie. Run tie-breaker with a different-architecture model
+      // through the same mode runner so framing/enrichment match.
+      const tb = await runModeRunnerInference(truncated, verifiable, prNumber, isCurated, TIEBREAKER_MODEL);
       if (tb.parsed) {
         const tv = (tb.parsed.claim_verdicts ?? []).find((v: any) => Number(v?.claim_idx) === idx);
         if (tv?.backed === false) {
@@ -335,9 +291,13 @@ export async function runInferenceCheck(
   // don't exit before extraction lands; the systemd poller has plenty
   // of headroom (90s cycle vs ~15s extraction). A failure inside
   // extractAndPersistFacts is caught + logged but never throws.
+  // Post-2026-04-27: extraction now runs against the truncated diff
+  // (no scratchpad to extract from since tree-split was retired).
+  // Fact extraction is still useful for surfacing entities/symbols
+  // into audit_facts.jsonl even from truncated input.
   if (isCurated && ctx && process.env.LH_AUDITOR_SKIP_EXTRACT !== "1") {
     try {
-      await extractAndPersistFacts(diffForPrompt, ctx);
+      await extractAndPersistFacts(truncated, ctx);
     } catch (e) {
       console.error(`[inference] fact extraction failed: ${(e as Error).message}`);
     }
@@ -394,60 +354,106 @@ export async function runInferenceCheck(
   return findings;
 }
 
-// Single cloud call — the consensus loop calls this N times in
-// parallel. Returns the parsed JSON shape + token usage + any error
-// diagnostic. NEVER throws; the consensus aggregator handles partial
-// failures by dropping non-parsed runs from the vote.
+// Single mode-runner call — consensus + tie-breaker dispatch through
+// here. Returns parsed JSON shape + telemetry from /v1/mode/execute
+// (latency, enrichment metrics) + any error diagnostic. NEVER throws.
+// The consensus aggregator handles partial failures by dropping
+// non-parsed runs from the vote.
 interface CloudRunResult {
   parsed: any | null;
-  tokens: number;
+  latency_ms: number;
+  enriched_chars: number;
+  bug_fingerprints: number;
+  matrix_kept: number;
   error?: string;       // "unreachable" | "non_200" | "unparseable"
   diagnostic?: string;  // first 200 chars for debugging
   model: string;
 }
 
-async function runCloudInference(systemMsg: string, userMsg: string, model: string): Promise<CloudRunResult> {
+async function runModeRunnerInference(
+  diffOrScratchpad: string,
+  claims: Claim[],
+  prNumber: number,
+  isCurated: boolean,
+  model: string,
+): Promise<CloudRunResult> {
+  // user_question carries the claim list + the curation note (if any).
+  // pr_audit's framing (mode.rs FRAMING_PR_AUDIT) holds the JSON shape +
+  // strict-output rules so we don't repeat them here.
+  const claimDigest = claims
+    .map((c, i) => `  ${i}. [${c.strength}] "${c.text}" at ${c.location}`)
+    .join("\n");
+  const curationNote = isCurated
+    ? "\n\nNOTE: the FILE below is a curated multi-shard scratchpad of the diff, not the raw diff itself. Absence in the scratchpad is NOT evidence of absence in the actual diff. Only mark backed=false on direct contradiction (e.g. scratchpad shows the function is empty / a stub). Skip unflagged_gaps entirely when scratchpad is curated."
+    : "";
+  const userQuestion = [
+    "Verify each ship-claim against the diff (or scratchpad).",
+    "",
+    "Ship-claims (numbered 0..N-1):",
+    claimDigest,
+    curationNote,
+    "",
+    "Every claim above must produce exactly one claim_verdicts entry. Output strict JSON only — no prose outside the JSON object.",
+  ].join("\n");
+
   let resp: Response;
   try {
-    resp = await fetch(`${GATEWAY}/v1/chat`, {
+    resp = await fetch(`${GATEWAY}/v1/mode/execute`, {
       method: "POST",
       headers: { "content-type": "application/json" },
       body: JSON.stringify({
-        provider: "ollama_cloud",
-        model,
-        messages: [
-          { role: "system", content: systemMsg },
-          { role: "user", content: userMsg },
-        ],
-        // temp=0 (greedy) + think=true. think=true is required for
-        // gpt-oss:120b — without it the model returns empty content
-        // on large prompts. Variance from the think trace is observed
-        // in practice, which is why we use N=3 consensus, not single-
-        // call determinism.
-        max_tokens: 3000,
-        temperature: 0,
-        think: true,
+        task_class: "pr_audit",
+        file_path: `pr-${prNumber}.diff`,
+        file_content: diffOrScratchpad,
+        user_question: userQuestion,
+        force_model: model,
+        force_temperature: 0,
       }),
-      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
+      signal: AbortSignal.timeout(MODE_RUNNER_TIMEOUT_MS),
     });
   } catch (e) {
-    return { parsed: null, tokens: 0, error: "unreachable", diagnostic: (e as Error).message.slice(0, 200), model };
+    return {
+      parsed: null, latency_ms: 0, enriched_chars: 0, bug_fingerprints: 0, matrix_kept: 0,
+      error: "unreachable", diagnostic: (e as Error).message.slice(0, 200), model,
+    };
   }
   if (!resp.ok) {
-    return { parsed: null, tokens: 0, error: "non_200", diagnostic: `${resp.status}: ${(await resp.text()).slice(0, 160)}`, model };
+    return {
+      parsed: null, latency_ms: 0, enriched_chars: 0, bug_fingerprints: 0, matrix_kept: 0,
+      error: "non_200", diagnostic: `${resp.status}: ${(await resp.text()).slice(0, 160)}`, model,
+    };
   }
   let body: any;
   try { body = await resp.json(); }
-  catch (e) { return { parsed: null, tokens: 0, error: "unparseable", diagnostic: (e as Error).message, model }; }
-  const content: string = body?.choices?.[0]?.message?.content ?? "";
-  const tokens: number = body?.usage?.total_tokens ?? 0;
-  const parsed = extractJson(content);
-  if (!parsed) {
-    return { parsed: null, tokens, error: "unparseable", diagnostic: content.slice(0, 200), model };
+  catch (e) {
+    return {
+      parsed: null, latency_ms: 0, enriched_chars: 0, bug_fingerprints: 0, matrix_kept: 0,
+      error: "unparseable", diagnostic: (e as Error).message, model,
+    };
   }
-  return { parsed, tokens, model };
+  const content: string = typeof body?.response === "string" ? body.response : "";
+  const parsed = extractJson(content);
+  // Number-coerced extractors so a non-numeric upstream value (string,
+  // null, NaN) collapses to 0 instead of poisoning downstream
+  // arithmetic. Caught 2026-04-27 by kimi_architect self-audit —
+  // optional-chaining + ?? only catches null/undefined, not type drift.
+  const num = (v: unknown): number => {
+    const n = typeof v === "number" ? v : Number(v);
+    return Number.isFinite(n) ? n : 0;
+  };
+  return {
+    parsed,
+    latency_ms: num(body?.latency_ms),
+    enriched_chars: num(body?.enriched_prompt_chars),
+    bug_fingerprints: num(body?.sources?.bug_fingerprints_count),
+    matrix_kept: num(body?.sources?.matrix_chunks_kept),
+    error: parsed ? undefined : "unparseable",
+    diagnostic: parsed ? undefined : content.slice(0, 200),
+    model,
+  };
 }
 
+
 async function persistDiscrepancies(ctx: InferenceContext, discrepancies: any[]): Promise<void> {
   await mkdir("/home/profit/lakehouse/data/_kb", { recursive: true });
   const rows = discrepancies.map(d => JSON.stringify({
@@ -490,94 +496,7 @@ async function extractAndPersistFacts(scratchpad: string, ctx: InferenceContext)
   await appendFile(AUDIT_FACTS_JSONL, JSON.stringify(row) + "\n");
 }
 
-// Curation via tree-split — ports the scrum_master pattern into the
-// inference check. Shards the raw diff into DIFF_SHARD_SIZE chunks,
-// summarizes each shard *against the claim-verification task* so the
-// summary preserves exactly what the cloud needs to judge claims
-// (function signatures, struct fields, deletions, new files), drops
-// everything else. Merges into a compact scratchpad.
-//
-// Cost: N cloud calls for the shard summaries + 1 cloud call for the
-// final verification = N+1 calls instead of 1. Mitigation: shards run
-// serially (not parallel) to keep gateway load bounded; summary calls
-// use max_tokens=400 so they're fast (~2s each on gpt-oss:120b).
-//
-// Determinism: each shard summary call uses temp=0 + think=true (same
-// as the top-level inference call), so identical input yields
-// identical scratchpad. The final verification call then sees a
-// stable scratchpad, giving stable verdicts.
-async function treeSplitDiff(
-  fullDiff: string,
-  claims: Claim[],
-): Promise<{ scratchpad: string; shards: number }> {
-  const shards: Array<{ from: number; to: number; text: string }> = [];
-  for (let i = 0; i < fullDiff.length; i += DIFF_SHARD_SIZE) {
-    const end = Math.min(i + DIFF_SHARD_SIZE, fullDiff.length);
-    shards.push({ from: i, to: end, text: fullDiff.slice(i, end) });
-  }
-  // Curate the claim list into a short form the summary prompt can
-  // use to bias extraction toward relevant facts.
-  const claimDigest = claims.map((c, i) =>
-    `${i}. [${c.strength}] "${c.text.slice(0, 100)}"`
-  ).join("\n");
 
-  let scratchpad = "";
-  for (const [si, shard] of shards.entries()) {
-    const prompt = [
-      `You are summarizing shard ${si + 1}/${shards.length} (chars ${shard.from}..${shard.to}) of a PR diff.`,
-      `The downstream task will verify these ship-claims against the full-PR summary. Extract ONLY facts that could confirm or refute these claims:`,
-      "",
-      claimDigest,
-      "",
-      "Extract: new function/method signatures, struct fields, deletions, new files, wiring (function X calls Y), absence-of-implementation markers, TODO comments on added lines.",
-      "Skip: comment-only edits, whitespace, import reordering, unrelated cosmetic changes.",
-      "",
-      "─────── shard diff ───────",
-      shard.text,
-      "─────── end shard ───────",
-      "",
-      "Output: up to 180 words of facts in bullet form. No prose preamble, no claim verdicts (that's for the downstream step).",
-    ].join("\n");
-
-    const r = await callCloud(prompt, 400);
-    if (r.content) {
-      scratchpad += `\n--- shard ${si + 1} (chars ${shard.from}..${shard.to}) ---\n${r.content.trim()}\n`;
-    }
-  }
-  return { scratchpad: scratchpad.trim(), shards: shards.length };
-}
-
-// Minimal cloud caller used only by treeSplitDiff — same gateway +
-// model as the top-level call, but think=false. Shards are small
-// (≤DIFF_SHARD_SIZE ~4500 chars) and the task is pure fact
-// extraction, not reasoning. think=true on the shards introduced
-// variance in reasoning traces that compounded across 23 calls into
-// a non-deterministic scratchpad (observed during curation
-// validation: same-SHA runs produced 5/7/8 final findings).
-// think=false on small prompts is stable — only breaks at the main
-// call's 10K+ prompt size, which keeps think=true.
-async function callCloud(prompt: string, maxTokens: number): Promise<{ content: string }> {
-  try {
-    const r = await fetch(`${GATEWAY}/v1/chat`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        provider: "ollama_cloud",
-        model: MODEL,
-        messages: [{ role: "user", content: prompt }],
-        max_tokens: maxTokens,
-        temperature: 0,
-        think: false,
-      }),
-      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
-    });
-    if (!r.ok) return { content: "" };
-    const j: any = await r.json();
-    return { content: j?.choices?.[0]?.message?.content ?? "" };
-  } catch {
-    return { content: "" };
-  }
-}
 
 // Pull out plausible code-symbol names from a summary string.
 // Matches:

auditor/checks/kimi_architect.ts

@@ -0,0 +1,465 @@
+// Kimi-architect check — second-pass senior architectural review using
+// kimi-for-coding (Kimi K2.6) via /v1/chat provider=kimi.
+//
+// Runs AFTER the deepseek inference check (N=3 consensus) and the
+// static/kb_query checks. Reads their findings as context and asks Kimi
+// "what did everyone else miss?" — complementing the cheap-consensus
+// voting with a sparse senior pass that catches load-bearing issues
+// (compile errors, false telemetry, schema bypasses, etc.) which the
+// voting structure can't see.
+//
+// Why Kimi here and not in the inner inference loop:
+// - Cost: ~3min wall-clock per call vs ~30s for deepseek consensus.
+// - TOS: api.kimi.com is User-Agent-gated (see crates/gateway/src/v1/
+//   kimi.rs); cost-bounded calls only.
+// - Value: experiment 2026-04-27 showed 7/7 grounding rate with full
+//   files vs ~50% on truncated input. Best as a sparse complement, not
+//   a replacement.
+//
+// Failure-isolated: any Kimi error returns a single info-level Finding
+// "kimi_architect skipped — <reason>" so the existing audit pipeline
+// is never blocked by a Kimi outage / TOS revocation / 429.
+//
+// Cost cap: if a kimi_verdicts/<pr>-<sha>.json file exists less than 24h
+// old, return cached findings without calling upstream. New commits
+// produce new SHAs so this is per-head, not per-day.
+//
+// Off by default: caller checks LH_AUDITOR_KIMI=1 before invoking.
+
+import { readFile, writeFile, mkdir, appendFile, stat, realpath } from "node:fs/promises";
+import { existsSync, realpathSync } from "node:fs";
+import { dirname, join, resolve } from "node:path";
+import type { Finding, CheckKind } from "../types.ts";
+
+const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
+const KIMI_VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/kimi_verdicts";
+const KIMI_AUDITS_JSONL = "/home/profit/lakehouse/data/_kb/kimi_audits.jsonl";
+const REPO_ROOT = "/home/profit/lakehouse";
+// Canonicalize at module load — REPO_ROOT itself may be a symlink in
+// some environments (e.g. /home/profit is a bind-mount). Computing
+// once at startup means the per-finding grounding loop can compare
+// realpath(target) against this stable anchor.
+const REPO_ROOT_REAL = (() => {
+  try { return realpathSync(REPO_ROOT); }
+  catch { return REPO_ROOT; }
+})();
+// 15 min budget. Bun's fetch has an intrinsic ~300s limit that our
+// AbortController + setTimeout combo could not override; we use curl
+// via Bun.spawn instead (callKimi below). Curl honors -m for max
+// transfer time without a hard intrinsic ceiling.
+const CALL_TIMEOUT_MS = 900_000;
+const CACHE_TTL_MS = 24 * 60 * 60 * 1000;
+const MAX_DIFF_CHARS = 180_000;
+const MAX_PRIOR_FINDINGS = 50;
+// Default provider/model = openrouter/moonshotai/kimi-k2.6.
+// History: pre-2026-04-27 direct api.kimi.com (User-Agent gating),
+// then ollama_cloud/kimi-k2.6 while the sub was active. 2026-09-02:
+// Ollama Cloud subscription lapsed (402 on every cloud model), J's
+// call was to sweep the auditor to OpenRouter — same Moonshot model,
+// probed live through the gateway. The api.kimi.com path
+// (provider=kimi) remains wired in the gateway as a fallback.
+const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "openrouter";
+const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "moonshotai/kimi-k2.6";
+// Cross-lineage alternation. 2026-04-27 J's call: Opus is too
+// expensive to auto-fire (~$0.30/audit). Kimi K2.6 via Go-sub is
+// effectively free; Haiku 4.5 via Zen is ~$0.04. Alternate between
+// them so we get cross-lineage signal (Moonshot vs Anthropic) on
+// every PR's audit history without burning the budget.
+//
+// Default: Kimi K2.6 on even audits, Haiku 4.5 on odd. Each PR's
+// audits flip between vendors as new SHAs come in.
+//
+// Frontier models (Opus 4.7, GPT-5.5, Gemini 3.1) are NOT in the
+// auto path. Operator hands distilled findings to a frontier model
+// manually when high-leverage decisions need it. Removing Opus from
+// auto-promotion saves ~$1-3/day on the daemon at our cadence.
+//
+// Override the alternation entirely with LH_AUDITOR_KIMI_MODEL
+// (forces one model regardless of audit count); set
+// LH_AUDITOR_KIMI_ALT_MODEL to the alternate.
+// 2026-09-02: opencode lane 401s (CreditsError, balance drained) — alt
+// lane joins the OpenRouter sweep. Same Anthropic lineage, probed live.
+const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "anthropic/claude-haiku-4.5";
+const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "openrouter";
+const FORCE_DEFAULT = process.env.LH_AUDITOR_KIMI_MODEL !== undefined && process.env.LH_AUDITOR_KIMI_MODEL !== "";
+
+function selectModel(diffLen: number, auditIndex: number = 0): { provider: string; model: string; promoted: boolean } {
+  // Operator override — env-pinned model wins.
+  if (FORCE_DEFAULT) {
+    return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
+  }
+  // Alternate Kimi (default, even index) ↔ Haiku (alt, odd index).
+  // diffLen kept in the signature for future "big diff → Haiku
+  // anyway" logic; not used yet so we don't auto-burn on big PRs.
+  void diffLen;
+  if (auditIndex % 2 === 1) {
+    return { provider: ALT_PROVIDER, model: ALT_MODEL, promoted: true };
+  }
+  return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
+}
+// Model-aware max_tokens. Different upstream APIs cap at different
+// limits and reject requests that exceed them:
+//   - Anthropic Opus 4.x: 32K output (with extended-output header)
+//   - Anthropic Haiku 4.5: 8K output
+//   - Kimi K2.6 (reasoning): 128K — needs headroom because
+//     reasoning_content counts against the budget
+//   - Default: 16K, conservative middle ground
+//
+// 2026-04-27 BLOCK from Opus self-audit: the prior single-default of
+// 128K worked silently (Anthropic clamps server-side) but was
+// technically invalid. Per-model caps make it explicit. Override via
+// LH_AUDITOR_KIMI_MAX_TOKENS to force a value (also fixes the empty-
+// env Number("") -> 0 trap by using `||` not `??`).
+const MAX_TOKENS_OVERRIDE = Number(process.env.LH_AUDITOR_KIMI_MAX_TOKENS) || 0;
+function maxTokensFor(model: string): number {
+  if (MAX_TOKENS_OVERRIDE > 0) return MAX_TOKENS_OVERRIDE;
+  if (model.startsWith("claude-opus")) return 32_000;
+  if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet") || model.includes("/claude-haiku") || model.includes("/claude-sonnet")) return 8_192;
+  if (model.startsWith("kimi-") || model.includes("/kimi-")) return 128_000;
+  if (model.startsWith("gpt-5") || model.startsWith("o1") || model.startsWith("o3") || model.startsWith("o4")) return 32_000;
+  return 16_000;
+}
+
+export interface KimiArchitectContext {
+  pr_number: number;
+  head_sha: string;
+}
+
+interface KimiVerdictFile {
+  pr_number: number;
+  head_sha: string;
+  cached_at: string;
+  model: string;
+  latency_ms: number;
+  finish_reason: string;
+  usage: { prompt_tokens: number; completion_tokens: number; total_tokens: number };
+  raw_content: string;
+  findings: Finding[];
+  grounding: { total: number; verified: number; rate: number };
+}
+
+export async function runKimiArchitectCheck(
+  diff: string,
+  priorFindings: Finding[],
+  ctx: KimiArchitectContext,
+): Promise<Finding[]> {
+  const cachePath = join(KIMI_VERDICTS_DIR, `${ctx.pr_number}-${ctx.head_sha.slice(0, 12)}.json`);
+  const outageSentinel = `${cachePath}.outage`;
+  const OUTAGE_TTL_MS = 10 * 60 * 1000;
+
+  // Outage negative-cache — if upstream failed within OUTAGE_TTL_MS,
+  // skip this audit and return immediately. Prevents the daemon from
+  // hammering a downed Kimi/Anthropic upstream every 90s.
+  if (existsSync(outageSentinel)) {
+    try {
+      const s = await stat(outageSentinel);
+      if (Date.now() - s.mtimeMs < OUTAGE_TTL_MS) {
+        const note = JSON.parse(await readFile(outageSentinel, "utf8"));
+        return [skipFinding(`upstream still down (cached ${Math.round((Date.now() - s.mtimeMs) / 1000)}s ago): ${String(note.reason).slice(0, 160)}`)];
+      }
+    } catch { /* malformed sentinel — fall through to fresh call */ }
+  }
+
+  // Cost cap — return cached findings if a verdict for this exact head
+  // SHA was generated within the TTL.
+  const cached = await loadCachedVerdict(cachePath);
+  if (cached) {
+    return cached.findings.length > 0
+      ? cached.findings
+      : [{ check: "kimi_architect" as CheckKind, severity: "info", summary: "kimi_architect cached — 0 findings", evidence: [`cache: ${cachePath}`] }];
+  }
+
+  // Alternate model based on how many audits this PR has had — gives
+  // cross-lineage signal (Kimi/Moonshot ↔ Haiku/Anthropic) on every
+  // PR's audit history. Count is derived from existing kimi_verdicts
+  // files for this PR; cheap O(N_PRs) directory read.
+  let auditIndex = 0;
+  try {
+    const dir = "/home/profit/lakehouse/data/_auditor/kimi_verdicts";
+    if (existsSync(dir)) {
+      const all = require("node:fs").readdirSync(dir) as string[];
+      auditIndex = all.filter((f) => f.startsWith(`${ctx.pr_number}-`)).length;
+    }
+  } catch { /* default 0 — Kimi */ }
+
+  const selected = selectModel(diff.length, auditIndex);
+  let response: { content: string; usage: any; finish_reason: string; latency_ms: number };
+  try {
+    response = await callKimi(buildPrompt(diff, priorFindings, ctx), selected.provider, selected.model);
+  } catch (e) {
+    // Negative-cache for 10 min on outage (caught 2026-04-27 by Opus
+    // self-audit): without this, every audit cycle within the 24h
+    // TTL re-calls upstream while it's still down. Use a sentinel
+    // file with mtime check rather than persisting a verdict so the
+    // happy-path cache reader doesn't have to special-case it.
+    const sentinel = `${cachePath}.outage`;
+    try { await writeFile(sentinel, JSON.stringify({ at: new Date().toISOString(), reason: (e as Error).message.slice(0, 200) })); } catch {}
+    return [skipFinding(`kimi call failed (${selected.model}): ${(e as Error).message.slice(0, 200)}`)];
+  }
+
+  const findings = parseFindings(response.content);
+  const grounding = await computeGrounding(findings);
+
+  const verdict: KimiVerdictFile = {
+    pr_number: ctx.pr_number,
+    head_sha: ctx.head_sha,
+    cached_at: new Date().toISOString(),
+    model: selected.model,
+    latency_ms: response.latency_ms,
+    finish_reason: response.finish_reason,
+    usage: {
+      prompt_tokens: response.usage?.prompt_tokens ?? 0,
+      completion_tokens: response.usage?.completion_tokens ?? 0,
+      total_tokens: response.usage?.total_tokens ?? 0,
+    },
+    raw_content: response.content,
+    findings,
+    grounding,
+  };
+
+  // Cache-poisoning guard (caught 2026-04-27 by Opus self-audit):
+  // when parseFindings returns 0 findings (Kimi rambled, prompt too
+  // big, or the markdown shape changed and our regex missed every
+  // block), persisting the empty verdict short-circuits all future
+  // audits in the 24h TTL window with a useless cached "0 findings"
+  // result. Better to leave no cache and re-call upstream next time.
+  // Always append metrics — observability shouldn't depend on whether
+  // findings parsed.
+  await appendMetrics(verdict);
+  if (findings.length > 0) {
+    await persistVerdict(cachePath, verdict);
+    return findings;
+  }
+  return [{
+    check: "kimi_architect" as CheckKind,
+    severity: "info",
+    summary: `kimi_architect produced 0 ranked findings (${response.finish_reason}, ${verdict.usage.completion_tokens} tokens) — not cached`,
+    evidence: [`raw saved (no cache): see kimi_audits.jsonl ${verdict.cached_at}`],
+  }];
+}
+
+async function loadCachedVerdict(path: string): Promise<KimiVerdictFile | null> {
+  if (!existsSync(path)) return null;
+  try {
+    const s = await stat(path);
+    if (Date.now() - s.mtimeMs > CACHE_TTL_MS) return null;
+    return JSON.parse(await readFile(path, "utf8")) as KimiVerdictFile;
+  } catch { return null; }
+}
+
+function buildPrompt(diff: string, priorFindings: Finding[], ctx: KimiArchitectContext): string {
+  const truncatedDiff = diff.length > MAX_DIFF_CHARS
+    ? diff.slice(0, MAX_DIFF_CHARS) + `\n\n... [truncated; original diff was ${diff.length} chars]`
+    : diff;
+
+  const priorBlock = priorFindings
+    .filter(f => f.severity !== "info")
+    .slice(0, MAX_PRIOR_FINDINGS)
+    .map(f => `- [${f.check}/${f.severity}] ${f.summary}${f.evidence?.[0] ? ` — ${f.evidence[0].slice(0, 160)}` : ""}`)
+    .join("\n");
+
+  return `You are a senior software architect doing a second-pass review on PR #${ctx.pr_number} (head ${ctx.head_sha.slice(0, 12)}). The team's automated auditor (deepseek-v3.1:671b, N=3 consensus) already produced findings. Your job is NOT to repeat what they found — your job is to catch what their voting structure CAN'T see: compile errors, type-system bypasses, false telemetry, silent determinism leaks, schema-bypass anti-patterns, load-bearing assumptions that look fine line-by-line.
+
+GROUNDING RULES (non-negotiable):
+- Cite file:line for EVERY finding. Lines you cite must actually contain what you claim. Confabulating a finding wastes more time than missing one.
+- If the diff is truncated and you can't verify a claim, say "diff-truncated, can't verify" — DO NOT guess.
+- Distinguish architectural concerns (no specific line) from concrete bugs (specific line). Don't dress one as the other.
+
+PRIOR FINDINGS FROM DEEPSEEK CONSENSUS (do not repeat these):
+${priorBlock || "(none)"}
+
+OUTPUT FORMAT (markdown):
+- ## Verdict (one sentence)
+- ## Findings (5-10 items, each formatted EXACTLY as below)
+
+For each finding use this exact shape so a parser can lift them:
+
+### F1: <one-line summary>
+- **Severity:** block | warn | info
+- **File:** path/to/file.ext:LINE
+- **Rationale:** one or two sentences
+
+THE DIFF:
+
+${truncatedDiff}
+`;
+}
+
+async function callKimi(prompt: string, provider: string, model: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
+  const t0 = Date.now();
+  const body = JSON.stringify({
+    provider,
+    model,
+    messages: [{ role: "user", content: prompt }],
+    max_tokens: maxTokensFor(model),
+    temperature: 0.2,
+  });
+  // curl via Bun.spawn — bypasses Bun fetch's ~300s intrinsic ceiling.
+  // -m sets the max transfer time honored end-to-end. Body is piped via
+  // stdin to avoid argv length limits on big audit prompts (~50K+ tokens).
+  const proc = Bun.spawn({
+    cmd: [
+      "curl", "-sS", "-X", "POST",
+      "-m", String(Math.ceil(CALL_TIMEOUT_MS / 1000)),
+      "-H", "content-type: application/json",
+      "--data-binary", "@-",
+      `${GATEWAY}/v1/chat`,
+    ],
+    stdin: "pipe",
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  proc.stdin.write(body);
+  await proc.stdin.end();
+  const [stdout, stderr, exitCode] = await Promise.all([
+    new Response(proc.stdout).text(),
+    new Response(proc.stderr).text(),
+    proc.exited,
+  ]);
+  if (exitCode !== 0) {
+    throw new Error(`curl exit ${exitCode}: ${stderr.slice(0, 300)}`);
+  }
+  let j: any;
+  try { j = JSON.parse(stdout); }
+  catch (e) {
+    throw new Error(`bad response (${stdout.length} bytes): ${stdout.slice(0, 300)}`);
+  }
+  if (j.error || !j.choices) {
+    throw new Error(`gateway error: ${JSON.stringify(j).slice(0, 300)}`);
+  }
+  return {
+    content: j.choices?.[0]?.message?.content ?? "",
+    usage: j.usage ?? {},
+    finish_reason: j.choices?.[0]?.finish_reason ?? "unknown",
+    latency_ms: Date.now() - t0,
+  };
+}
+
+// Parse Kimi's markdown into Finding[]. Format expected (per buildPrompt):
+//   ### F<N>: <summary>
+//   - **Severity:** block | warn | info
+//   - **File:** path:line
+//   - **Rationale:** ...
+function parseFindings(content: string): Finding[] {
+  const findings: Finding[] = [];
+  const blocks = content.split(/^###\s+F\d+:\s*/m).slice(1);
+  for (const block of blocks) {
+    const summary = (block.split("\n")[0] ?? "").trim();
+    if (!summary) continue;
+    const sev = /\*\*Severity:\*\*\s*(block|warn|info)/i.exec(block)?.[1]?.toLowerCase();
+    const fileLine = /\*\*File:\*\*\s*(\S+)/i.exec(block)?.[1] ?? "unknown";
+    const rationale = /\*\*Rationale:\*\*\s*([\s\S]+?)(?=\n###|\n\*\*|$)/i.exec(block)?.[1]?.trim() ?? "";
+    const severity: Finding["severity"] = sev === "block" ? "block" : sev === "warn" ? "warn" : "info";
+    findings.push({
+      check: "kimi_architect" as CheckKind,
+      severity,
+      summary: summary.slice(0, 240),
+      evidence: [fileLine, rationale.slice(0, 360)].filter(Boolean),
+    });
+  }
+  return findings;
+}
+
+// For each finding's cited file:line, grep the actual file to verify
+// the line exists. Returns total + verified counts; per-finding metadata
+// is appended into the evidence array so the reader can see which
+// citations were verified.
+async function computeGrounding(findings: Finding[]): Promise<{ total: number; verified: number; rate: number }> {
+  // readFile (async) instead of readFileSync — caught 2026-04-27 by
+  // Kimi's self-audit. Sync I/O in an async fn blocks the event loop
+  // for every cited file; doesn't matter at 10 findings, would matter
+  // at 100+.
+  const checks = await Promise.all(findings.map(async (f) => {
+    const cite = f.evidence[0] ?? "";
+    const m = /^(\S+?):(\d+)/.exec(cite);
+    if (!m) return false;
+    const [, relpath, lineStr] = m;
+    const line = Number(lineStr);
+    if (!line || !relpath) return false;
+
+    // Path-traversal guard, two-layer (caught 2026-04-27 by Kimi
+    // self-audits on dd77632 then 2d9cb12).
+    //
+    // Layer 1 (lexical): resolve() normalizes `..` segments. Refuse
+    // any path that doesn't anchor under REPO_ROOT.
+    //
+    // Layer 2 (symlink): even if the lexical path is anchored, it
+    // could be a symlink whose target escapes. realpath() resolves
+    // symlinks; compare the real path against REPO_ROOT_REAL.
+    //
+    // Both layers exist because attackers might bypass either alone:
+    // raw `../etc/passwd` triggers layer 1; a planted symlink at
+    // ./safe-looking-name → /etc/passwd triggers layer 2.
+    const abs = resolve(REPO_ROOT, relpath);
+    if (!abs.startsWith(REPO_ROOT + "/") && abs !== REPO_ROOT) {
+      f.evidence.push(`[grounding: path escapes repo root, refusing]`);
+      return false;
+    }
+
+    if (!existsSync(abs)) {
+      f.evidence.push("[grounding: file not found]");
+      return false;
+    }
+    try {
+      // Symlink-resolution check before any read. realpath() throws
+      // if the file doesn't exist; existsSync above shields the
+      // common case but a TOCTOU race could still error here — the
+      // outer catch handles it.
+      const realPath = await realpath(abs);
+      if (!realPath.startsWith(REPO_ROOT_REAL + "/") && realPath !== REPO_ROOT_REAL) {
+        f.evidence.push(`[grounding: symlink target escapes repo root, refusing]`);
+        return false;
+      }
+      const lines = (await readFile(realPath, "utf8")).split("\n");
+      if (line < 1 || line > lines.length) {
+        f.evidence.push(`[grounding: line ${line} > EOF (${lines.length})]`);
+        return false;
+      }
+      f.evidence.push(`[grounding: verified at ${relpath}:${line}]`);
+      return true;
+    } catch (e) {
+      f.evidence.push(`[grounding: read failed: ${(e as Error).message.slice(0, 80)}]`);
+      return false;
+    }
+  }));
+  const verified = checks.filter(Boolean).length;
+  const total = findings.length;
+  return { total, verified, rate: total === 0 ? 0 : verified / total };
+}
+
+async function persistVerdict(path: string, v: KimiVerdictFile): Promise<void> {
+  await mkdir(KIMI_VERDICTS_DIR, { recursive: true });
+  await writeFile(path, JSON.stringify(v, null, 2));
+}
+
+async function appendMetrics(v: KimiVerdictFile): Promise<void> {
+  // dirname() instead of join(path, "..") — caught 2026-04-27 by both
+  // Haiku and Opus self-audits. The "/.." idiom resolves correctly
+  // via Node path normalization but is non-idiomatic + breaks if the
+  // path ever has trailing dots.
+  await mkdir(dirname(KIMI_AUDITS_JSONL), { recursive: true });
+  await appendFile(KIMI_AUDITS_JSONL, JSON.stringify({
+    pr_number: v.pr_number,
+    head_sha: v.head_sha,
+    audited_at: v.cached_at,
+    model: v.model,
+    latency_ms: v.latency_ms,
+    finish_reason: v.finish_reason,
+    prompt_tokens: v.usage.prompt_tokens,
+    completion_tokens: v.usage.completion_tokens,
+    findings_total: v.findings.length,
+    findings_block: v.findings.filter(f => f.severity === "block").length,
+    findings_warn: v.findings.filter(f => f.severity === "warn").length,
+    grounding_verified: v.grounding.verified,
+    grounding_rate: Number(v.grounding.rate.toFixed(3)),
+  }) + "\n");
+}
+
+function skipFinding(why: string): Finding {
+  return {
+    check: "kimi_architect" as CheckKind,
+    severity: "info",
+    summary: `kimi_architect skipped — ${why}`,
+    evidence: [why],
+  };
+}

auditor/checks/static.ts

@@ -54,49 +54,87 @@ export function runStaticCheck(diff: string): Finding[] {
     const isAuditorCheckerFile = path.startsWith("auditor/checks/") ||
                                  path.startsWith("auditor/fixtures/");
 
+    // Track multi-line backtick-template state across the file. Walks
+    // all post-merge lines (context + added, skipping removed lines)
+    // in order and keeps `inMultilineBacktick` flipping on each
+    // unescaped backtick. Pre-2026-04-26 the per-line walk in
+    // isInsideQuotedString missed `todo!()` matches inside docstring
+    // template literals because the opening backtick lived on a
+    // line above the match. Now we OR the file-level state into the
+    // per-line check.
+    let inMultilineBacktick = false;
+
     for (let idx = 0; idx < lines.length; idx++) {
       const line = lines[idx];
-      if (!line.startsWith("+") || line.startsWith("+++")) continue;
-      const added = line.slice(1);
 
-      if (!isAuditorCheckerFile) {
-        for (const { re, why } of BLOCK_PATTERNS) {
-          const m = added.match(re);
-          if (m && typeof m.index === "number") {
-            // Skip if the match sits inside a quoted string literal —
-            // this is how rubric files (tests/real-world/*, prompt
-            // templates) legitimately reference the patterns they
-            // guard against, without actually executing them.
-            if (isInsideQuotedString(added, m.index)) continue;
+      // Diff bookkeeping lines and removed lines don't contribute to
+      // the post-merge file's string state.
+      if (line.startsWith("+++") || line.startsWith("---") ||
+          line.startsWith("@@") || line.startsWith("\\ No newline")) continue;
+      if (line.startsWith("-")) continue;
+
+      const isAdded = line.startsWith("+");
+      // Strip the diff prefix (' ' for context, '+' for added).
+      const body = (isAdded || line.startsWith(" ")) ? line.slice(1) : line;
+
+      // Compute the file-level backtick state ENTERING this line.
+      // The state machine sees pattern matches against the right
+      // context: a line that opens a backtick block has its own
+      // pattern checks evaluated under "inside-backtick" semantics
+      // for the portion AFTER the opening tick. Pre-2026-04-27 the
+      // state was updated AFTER the pattern checks, so the FIRST
+      // pattern on a backtick-opening line slipped through with
+      // stale "outside-backtick" semantics. Caught by Kimi self-audit.
+      const stateAtLineStart = inMultilineBacktick;
+      const stateAtLineEnd = updateBacktickState(body, stateAtLineStart);
+
+      if (isAdded) {
+        const added = body;
+
+        if (!isAuditorCheckerFile) {
+          for (const { re, why } of BLOCK_PATTERNS) {
+            const m = added.match(re);
+            if (m && typeof m.index === "number") {
+              // Skip if EITHER (a) the file was already inside a
+              // multi-line backtick block when this line started, OR
+              // (b) the match sits inside a quoted string literal on
+              // THIS line. The earlier code only checked stateAtLineStart;
+              // now we also check that the match isn't past the
+              // opening backtick of a block that opens on this line.
+              if (stateAtLineStart || isInsideQuotedString(added, m.index)) continue;
+              findings.push({
+                check: "static",
+                severity: "block",
+                summary: `${why} in ${path}`,
+                evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
+              });
+            }
+          }
+        }
+        for (const { re, why } of WARN_COMMENT_PATTERNS) {
+          if (re.test(line)) {
             findings.push({
               check: "static",
-              severity: "block",
+              severity: "warn",
+              summary: `${why} in ${path}`,
+              evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
+            });
+          }
+        }
+        for (const { re, why } of INFO_HARDCODED_PATTERNS) {
+          if (re.test(added)) {
+            findings.push({
+              check: "static",
+              severity: "info",
               summary: `${why} in ${path}`,
               evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
             });
           }
         }
       }
-      for (const { re, why } of WARN_COMMENT_PATTERNS) {
-        if (re.test(line)) {
-          findings.push({
-            check: "static",
-            severity: "warn",
-            summary: `${why} in ${path}`,
-            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
-          });
-        }
-      }
-      for (const { re, why } of INFO_HARDCODED_PATTERNS) {
-        if (re.test(added)) {
-          findings.push({
-            check: "static",
-            severity: "info",
-            summary: `${why} in ${path}`,
-            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
-          });
-        }
-      }
+
+      // Carry the end-of-line state forward to the next iteration.
+      inMultilineBacktick = stateAtLineEnd;
     }
 
     // "Field added but never read" heuristic — catches exactly the
@@ -105,10 +143,20 @@ export function runStaticCheck(diff: string): Finding[] {
     // elsewhere might exist). The point is: if NEITHER this diff nor
     // any other line in the diff reads the field, the PR is shipping
     // state without a consumer.
+    //
+    // Serde exemption: if the field's parent struct derives Serialize
+    // or Deserialize, the read-site is the macro itself — JSON
+    // round-trips consume every public field. Without this exemption
+    // the check produces false positives on every response/request
+    // struct shipped through `/v1/*`.
     const addedLines = lines.filter(l => l.startsWith("+") && !l.startsWith("+++"))
       .map(l => l.slice(1));
-    const newFields = extractNewFields(addedLines);
-    for (const field of newFields) {
+    const newFields = extractNewFieldsWithLine(lines);
+    const seenNames = new Set<string>();
+    for (const { name: field, lineIdx } of newFields) {
+      if (seenNames.has(field)) continue;
+      seenNames.add(field);
+      if (parentStructHasSerdeDerive(lines, lineIdx)) continue;
       const readPattern = new RegExp(`[\\.:]\\s*${escape(field)}\\b|\\b${escape(field)}\\s*:`);
       // The definition line itself matches readPattern — filter it out
       // by requiring at least TWO lines in the diff mention the field
@@ -146,26 +194,105 @@ function splitDiffByFile(diff: string): Map<string, string[]> {
   return out;
 }
 
-// Extract new `pub name: Type,` fields from added lines. Rust syntax.
-// Narrowly-scoped: only matches at the start of a trimmed line,
-// requires `pub ` prefix, ignores `pub fn` / `pub struct` / etc.
-function extractNewFields(addedLines: string[]): string[] {
-  const fields = new Set<string>();
-  for (const line of addedLines) {
-    const t = line.trim();
-    // pub NAME: Type,
+// Extract new `pub name: Type,` fields from the per-file diff lines,
+// keeping each occurrence's line index so the caller can resolve the
+// parent struct. Same narrow rules as before: starts with `pub `,
+// excludes `pub fn` / `pub struct` / etc.
+function extractNewFieldsWithLine(lines: string[]): Array<{ name: string; lineIdx: number }> {
+  const out: Array<{ name: string; lineIdx: number }> = [];
+  for (let i = 0; i < lines.length; i++) {
+    const line = lines[i];
+    if (!line.startsWith("+") || line.startsWith("+++")) continue;
+    const t = line.slice(1).trim();
     const m = t.match(/^pub\s+(?!fn\b|struct\b|enum\b|mod\b|use\b|trait\b|impl\b|const\b|static\b|type\b)(\w+)\s*:/);
-    if (m) fields.add(m[1]);
+    if (m) out.push({ name: m[1], lineIdx: i });
   }
-  return Array.from(fields);
+  return out;
+}
+
+// True if the field at `fieldLineIdx` lives inside a struct whose
+// declaration carries `#[derive(... Serialize|Deserialize ...)]`. We
+// walk backward through the diff (added + context lines both count —
+// a struct declaration unchanged by the PR still appears as context)
+// to find the nearest `pub struct` boundary, then scan a few lines
+// above it for derive attributes. Conservative bounds:
+//   - 80 lines back to find `struct` (struct definitions can grow large)
+//   - 8 lines above the `struct` keyword for attribute lines
+// Stops the struct-search early if we hit a `}` at zero indent
+// (the previous scope) or another `pub struct` (we left ours).
+function parentStructHasSerdeDerive(lines: string[], fieldLineIdx: number): boolean {
+  // Bounds-check fieldLineIdx (caught 2026-04-27 by Kimi self-audit).
+  // Pre-fix: if fieldLineIdx >= lines.length, the loop ran from a
+  // negative implicit upper bound (fieldLineIdx - 80 could be > 0
+  // even when fieldLineIdx is past EOF) and read undefined slots.
+  // Defensive: bail early on out-of-range input.
+  if (fieldLineIdx < 0 || fieldLineIdx >= lines.length) return false;
+
+  let structLineIdx = -1;
+  for (let i = fieldLineIdx - 1; i >= 0 && i >= fieldLineIdx - 80; i--) {
+    const raw = lines[i];
+    if (typeof raw !== "string" || raw.length === 0) continue;
+    const body = stripDiffPrefix(raw);
+    const trimmed = body.trim();
+    if (/^pub\s+struct\s+\w/.test(trimmed)) {
+      structLineIdx = i;
+      break;
+    }
+    // Closing brace at column 0 means the enclosing scope ended above
+    // the field — we're not actually inside a struct.
+    if (body.startsWith("}")) return false;
+  }
+  if (structLineIdx < 0) return false;
+
+  for (let j = structLineIdx - 1; j >= 0 && j >= structLineIdx - 8; j--) {
+    const raw = lines[j];
+    if (typeof raw !== "string") continue;
+    const trimmed = stripDiffPrefix(raw).trim();
+    if (trimmed === "" || trimmed.startsWith("//") || trimmed.startsWith("///")) continue;
+    if (!trimmed.startsWith("#[")) break;
+    if (/derive\s*\([^)]*\b(Serialize|Deserialize)\b/.test(trimmed)) return true;
+  }
+  return false;
+}
+
+// Strip leading +/-/space from a unified-diff line, leaving the raw
+// source line. Handles the case where the line is shorter than 1 char
+// (rare but real for empty-context lines).
+function stripDiffPrefix(line: string): string {
+  if (line.length === 0) return line;
+  const c = line[0];
+  if (c === "+" || c === "-" || c === " ") return line.slice(1);
+  return line;
+}
+
+// Walk a single line and toggle the cross-line backtick state on each
+// unescaped backtick. Single-quote and double-quote runs are line-
+// bounded in JS/TS/Rust by language rules (string literals don't span
+// newlines without explicit `\` continuation), so we only track
+// backticks across lines. Returns the new state for the next line.
+function updateBacktickState(line: string, inBacktick: boolean): boolean {
+  let state = inBacktick;
+  let inDouble = false;
+  let inSingle = false;
+  for (let i = 0; i < line.length; i++) {
+    const c = line[i];
+    const esc = i > 0 && line[i - 1] === "\\";
+    if (esc) continue;
+    // Inside a multi-line backtick template, single/double quotes
+    // don't open new strings — they're literal characters of the
+    // template. Same applies the other way around.
+    if (c === '"' && !inSingle && !state) inDouble = !inDouble;
+    else if (c === "'" && !inDouble && !state) inSingle = !inSingle;
+    else if (c === "`" && !inDouble && !inSingle) state = !state;
+  }
+  return state;
 }
 
 // True if `pos` falls inside a double- or single-quoted string on this
 // line (backtick template literals too). Walks left→right toggling the
-// "in quote" state on each unescaped quote. Good enough for single-
-// line matches; multi-line strings aren't parsed (they're extremely
-// rare in the patterns we're blocking on, and would require a proper
-// tokenizer to handle correctly).
+// "in quote" state on each unescaped quote. Per-line only — the file-
+// level walk in runStaticCheck handles multi-line backtick templates
+// via updateBacktickState.
 function isInsideQuotedString(line: string, pos: number): boolean {
   let inDouble = false, inSingle = false, inBacktick = false;
   for (let i = 0; i < pos; i++) {

auditor/gitea.ts

@@ -9,28 +9,42 @@
 import { readFile } from "node:fs/promises";
 import type { PrSnapshot } from "./types.ts";
 
-const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
+// 2026-09-02: repo home moved to git.warren.help in the VPS
+// consolidation; git.agentview.dev now dead-ends at a Traefik default
+// cert + 404 (auditor had been failing listOpenPrs every cycle since
+// Aug 25). Credentials come from ~/.git-credentials for HOST's domain,
+// falling back to the user:pass embedded in this repo's origin remote.
+// Basic auth works for both a PAT and a password on Gitea.
+const HOST = process.env.GITEA_HOST ?? "https://git.warren.help";
 const OWNER = "profit";
 const REPO = "lakehouse";
 const CRED_FILE = "/home/profit/.git-credentials";
+const GIT_CONFIG = "/home/profit/lakehouse/.git/config";
 
-let cachedPat: string | null = null;
+let cachedAuth: string | null = null;
 
-async function getPat(): Promise<string> {
-  if (cachedPat) return cachedPat;
-  const raw = await readFile(CRED_FILE, "utf8");
-  for (const line of raw.split("\n")) {
-    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
-    if (m) { cachedPat = m[1]; return m[1]; }
+async function getAuthHeader(): Promise<string> {
+  if (cachedAuth) return cachedAuth;
+  const hostDomain = new URL(HOST).host.replace(/\./g, "\\.");
+  const credRe = new RegExp(`^https://([^:]+):([^@]+)@${hostDomain}`);
+  for (const file of [CRED_FILE, GIT_CONFIG]) {
+    let raw: string;
+    try { raw = await readFile(file, "utf8"); } catch { continue; }
+    for (const line of raw.split("\n")) {
+      const m = line.trim().replace(/^url = /, "").match(credRe);
+      if (m) {
+        cachedAuth = `Basic ${Buffer.from(`${decodeURIComponent(m[1])}:${decodeURIComponent(m[2])}`).toString("base64")}`;
+        return cachedAuth;
+      }
+    }
   }
-  throw new Error(`no Gitea PAT in ${CRED_FILE}`);
+  throw new Error(`no Gitea credentials for ${HOST} in ${CRED_FILE} or ${GIT_CONFIG}`);
 }
 
 async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
-  const pat = await getPat();
   const url = `${HOST}/api/v1${path}`;
   const headers = new Headers(init.headers);
-  headers.set("Authorization", `token ${pat}`);
+  headers.set("Authorization", await getAuthHeader());
   if (init.body && !headers.has("content-type")) {
     headers.set("content-type", "application/json");
   }

auditor/index.ts

@@ -24,14 +24,30 @@ const POLL_INTERVAL_MS = 90_000; // 90s — enough budget for audit runs to comp
 const PAUSE_FILE = "/home/profit/lakehouse/auditor.paused";
 const STATE_FILE = "/home/profit/lakehouse/data/_auditor/state.json";
 
+// Per-PR audit cap. Prevents the daemon from running away on a PR
+// when each push surfaces new findings — operator wants to review
+// in batch, not have the daemon burn budget while they're away.
+// Default 3 audits per PR. Override via LH_AUDITOR_MAX_AUDITS_PER_PR.
+// Set to 0 to disable the cap.
+//
+// Reset (after manual review): edit data/_auditor/state.json and
+// set audit_count_per_pr.<N> = 0 (or delete the key). Daemon picks
+// up the change on the next cycle without restart.
+const MAX_AUDITS_PER_PR = Number(process.env.LH_AUDITOR_MAX_AUDITS_PER_PR) || 3;
+
 interface State {
   // Map: PR number → last-audited head SHA. Lets us dedupe audits
   // across restarts (poller can crash/restart without re-auditing
   // all open PRs from scratch).
   last_audited: Record<string, string>;
+  // Map: PR number → number of audits run on that PR since last reset.
+  // Daemon halts auditing a PR once this hits MAX_AUDITS_PER_PR.
+  // Operator clears the entry to resume.
+  audit_count_per_pr: Record<string, number>;
   started_at: string;
   cycles_total: number;
   cycles_skipped_paused: number;
+  cycles_skipped_capped: number;
   audits_run: number;
   last_cycle_at?: string;
 }
@@ -47,17 +63,21 @@ async function loadState(): Promise<State> {
     return {
       last_audited: s.last_audited ?? {},
       started_at: s.started_at ?? new Date().toISOString(),
+      audit_count_per_pr: s.audit_count_per_pr ?? {},
       cycles_total: s.cycles_total ?? 0,
       cycles_skipped_paused: s.cycles_skipped_paused ?? 0,
+      cycles_skipped_capped: s.cycles_skipped_capped ?? 0,
       audits_run: s.audits_run ?? 0,
       last_cycle_at: s.last_cycle_at,
     };
   } catch {
     return {
       last_audited: {},
+      audit_count_per_pr: {},
       started_at: new Date().toISOString(),
       cycles_total: 0,
       cycles_skipped_paused: 0,
+      cycles_skipped_capped: 0,
       audits_run: 0,
     };
   }
@@ -89,12 +109,38 @@ async function runCycle(state: State): Promise<State> {
   console.log(`[auditor] cycle ${state.cycles_total}: ${prs.length} open PR(s)`);
 
   for (const pr of prs) {
-    const last = state.last_audited[String(pr.number)];
+    const prKey = String(pr.number);
+    const last = state.last_audited[prKey];
     if (last === pr.head_sha) {
       console.log(`[auditor]   skip PR #${pr.number} (SHA ${pr.head_sha.slice(0, 8)} already audited)`);
       continue;
     }
-    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)}`);
+    // Per-head-SHA audit cap. Each new push gets MAX_AUDITS_PER_PR
+    // fresh attempts; the counter auto-resets when the head SHA
+    // changes. Operator only intervenes manually if a single SHA
+    // somehow needs MORE than the cap (rare — usually transient
+    // upstream errors clear themselves inside 3 attempts).
+    //
+    // Reset rule: if `last` exists (we've seen this PR before) AND
+    // pr.head_sha != last, that's a new push. Drop the counter.
+    // The dedup branch above already handles same-SHA → skip, so
+    // we only land here when the SHA actually moved.
+    if (last !== undefined && (state.audit_count_per_pr[prKey] ?? 0) > 0) {
+      const prior_count = state.audit_count_per_pr[prKey];
+      console.log(`[auditor]   PR #${pr.number} new head ${pr.head_sha.slice(0, 8)} (prior ${last.slice(0, 8)}, was ${prior_count}/${MAX_AUDITS_PER_PR}) — resetting cap counter`);
+      state.audit_count_per_pr[prKey] = 0;
+    }
+    const auditedSoFar = state.audit_count_per_pr[prKey] ?? 0;
+    if (MAX_AUDITS_PER_PR > 0 && auditedSoFar >= MAX_AUDITS_PER_PR) {
+      // This branch only fires now if the SAME head SHA somehow
+      // burned MAX audits (transient upstream errors retried that
+      // many times). Operator can clear state.audit_count_per_pr.<N>
+      // = 0 to force one more attempt; otherwise wait for next push.
+      console.log(`[auditor]   skip PR #${pr.number} (same head ${pr.head_sha.slice(0, 8)} burned ${auditedSoFar}/${MAX_AUDITS_PER_PR} — push new code or clear state.json audit_count_per_pr.${prKey})`);
+      state.cycles_skipped_capped += 1;
+      continue;
+    }
+    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)} [${auditedSoFar + 1}/${MAX_AUDITS_PER_PR}]`);
     try {
       // Skip dynamic by default: it mutates live playbook state and
       // re-runs on every PR update would pollute quickly. Operator
@@ -106,8 +152,22 @@ async function runCycle(state: State): Promise<State> {
         skip_inference: process.env.LH_AUDITOR_SKIP_INFERENCE === "1",
       });
       console.log(`[auditor]     verdict=${verdict.overall} findings=${verdict.metrics.findings_total} (block=${verdict.metrics.findings_block} warn=${verdict.metrics.findings_warn})`);
-      state.last_audited[String(pr.number)] = pr.head_sha;
+      state.last_audited[prKey] = pr.head_sha;
+      state.audit_count_per_pr[prKey] = auditedSoFar + 1;
       state.audits_run += 1;
+      if (state.audit_count_per_pr[prKey] >= MAX_AUDITS_PER_PR) {
+        console.log(`[auditor]     PR #${pr.number} reached cap (${MAX_AUDITS_PER_PR} audits) — daemon will skip further audits until reset`);
+      }
+      // Persist state immediately after each successful audit so the
+      // increment survives a crash. Pre-2026-04-27 the cycle saved
+      // once at the end (main.ts:140), which lost the count if the
+      // daemon was killed mid-cycle. Fix lifted from kimi_architect's
+      // own audit on this very file. saveState is idempotent + cheap
+      // (one JSON write), so per-audit cost is negligible.
+      try { await saveState(state); }
+      catch (e) {
+        console.error(`[auditor]     saveState mid-cycle failed: ${(e as Error).message} — count held in memory`);
+      }
     } catch (e) {
       console.error(`[auditor]     audit failed: ${(e as Error).message}`);
     }

auditor/schemas/distillation/drift_report.ts

@@ -0,0 +1,85 @@
+// drift_report.ts — comparison of a current run summary vs the
+// previous run summary on disk. Spec calls this "drift detection";
+// concretely it answers: did the pipeline behave the same way as
+// last time, and if not, was the change explained by an input change
+// or did it appear out of nowhere (silent drift)?
+//
+// Severity:
+//   ok    — within 20% on every metric, no hash surprises
+//   warn  — record-count or category swing > 20%, OR new error class
+//   alert — output_hash differs while input_hash is identical
+//           (deterministic violation — same input → different output)
+
+import {
+  ValidationResult, requireString, requireIsoTimestamp,
+} from "./types";
+import type { StageName } from "./stage_receipt";
+
+export const DRIFT_REPORT_SCHEMA_VERSION = 2;
+export const DRIFT_THRESHOLD_PCT = 0.20;
+
+export type DriftSeverity = "ok" | "warn" | "alert";
+
+export interface StageDrift {
+  stage: StageName;
+  delta_records_in: number;       // current - prior
+  delta_records_out: number;
+  delta_accepted: number;
+  delta_quarantined: number;
+  pct_change_out: number | null;  // null when prior had 0 records
+  // null when input_hash isn't materialized into the stage summary —
+  // schema v1 lied and reported `true` here. v2 is honest: callers
+  // that want determinism enforcement must read the full StageReceipt
+  // off disk and compute input_hash equality there.
+  input_hash_match: boolean | null;
+  output_hash_match: boolean;
+  // alert if input_hash matches but output_hash diverges
+  deterministic_violation: boolean;
+  notes: string[];
+}
+
+export interface DriftReport {
+  schema_version: number;
+  run_id: string;
+  prior_run_id: string | null;    // null when no prior run on disk
+  generated_at: string;
+  severity: DriftSeverity;
+  stages: StageDrift[];
+  // Top-level swings the human reader should see immediately.
+  flags: string[];
+}
+
+export function validateDriftReport(input: unknown): ValidationResult<DriftReport> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object"] };
+  }
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== DRIFT_REPORT_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${DRIFT_REPORT_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.run_id, "run_id", errors) && ok;
+  if (r.prior_run_id !== null && typeof r.prior_run_id !== "string") {
+    errors.push("prior_run_id: must be string or null");
+    ok = false;
+  }
+  ok = requireIsoTimestamp(r.generated_at, "generated_at", errors) && ok;
+  if (!["ok", "warn", "alert"].includes(r.severity as string)) {
+    errors.push(`severity: must be ok|warn|alert, got ${JSON.stringify(r.severity)}`);
+    ok = false;
+  }
+  if (!Array.isArray(r.stages)) {
+    errors.push("stages: expected array");
+    ok = false;
+  }
+  if (!Array.isArray(r.flags)) {
+    errors.push("flags: expected array");
+    ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as DriftReport };
+}

auditor/schemas/distillation/evidence_record.test.ts

@@ -0,0 +1,116 @@
+// EvidenceRecord schema tests.
+//
+// Two positive fixtures (one per real-source prototype: distilled_facts
+// + contract_analyses) and three negative fixtures pinning the
+// non-negotiable invariants the spec demands:
+//   - every record must trace to a source (provenance)
+//   - schema_version must match — silent v1/v2 drift is the worst kind
+//   - required identity fields (run_id) cannot be missing
+//
+// Run with: bun test auditor/schemas/distillation/evidence_record.test.ts
+
+import { test, expect } from "bun:test";
+import { readFileSync } from "node:fs";
+import { resolve } from "node:path";
+
+import { validateEvidenceRecord, EVIDENCE_SCHEMA_VERSION } from "./evidence_record";
+
+const FIXTURE_DIR = resolve(import.meta.dir, "fixtures");
+
+function loadFixture(name: string): unknown {
+  return JSON.parse(readFileSync(resolve(FIXTURE_DIR, name), "utf8"));
+}
+
+test("EVIDENCE_SCHEMA_VERSION is 1 — bump deliberately, never silently", () => {
+  expect(EVIDENCE_SCHEMA_VERSION).toBe(1);
+});
+
+test("positive: distilled_fact materialized record validates", () => {
+  const r = validateEvidenceRecord(loadFixture("evidence_positive_distilled_fact.json"));
+  if (!r.valid) console.error("unexpected errors:", r.errors);
+  expect(r.valid).toBe(true);
+  if (r.valid) {
+    expect(r.value.run_id).toBe("cae21289");
+    expect(r.value.model_role).toBe("extractor");
+    expect(r.value.provenance.source_file).toBe("data/_kb/distilled_facts.jsonl");
+  }
+});
+
+test("positive: contract_analysis materialized record validates with retrieval + observer fields", () => {
+  const r = validateEvidenceRecord(loadFixture("evidence_positive_contract_analysis.json"));
+  if (!r.valid) console.error("unexpected errors:", r.errors);
+  expect(r.valid).toBe(true);
+  if (r.valid) {
+    expect(r.value.observer_verdict).toBe("reject");
+    expect(r.value.observer_confidence).toBe(95);
+    expect(r.value.retrieved_context?.matrix_corpora?.length).toBe(4);
+    expect(r.value.failure_markers).toContain("observer_rejected");
+  }
+});
+
+test("negative: missing run_id is rejected with a specific error", () => {
+  const r = validateEvidenceRecord(loadFixture("evidence_negative_no_run_id.json"));
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    expect(r.errors.some(e => e.includes("run_id"))).toBe(true);
+  }
+});
+
+test("negative: schema_version mismatch is rejected (silent v1/v2 drift guard)", () => {
+  const r = validateEvidenceRecord(loadFixture("evidence_negative_bad_schema_version.json"));
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    expect(r.errors.some(e => e.includes("schema_version"))).toBe(true);
+  }
+});
+
+test("negative: bad provenance (non-sha256 sig_hash, non-ISO timestamp) is rejected", () => {
+  const r = validateEvidenceRecord(loadFixture("evidence_negative_bad_provenance.json"));
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    // Must catch BOTH the sig_hash AND the recorded_at — comprehensive
+    // error reporting is part of the contract.
+    expect(r.errors.some(e => e.includes("sig_hash"))).toBe(true);
+    expect(r.errors.some(e => e.includes("recorded_at"))).toBe(true);
+  }
+});
+
+test("negative: non-object input is rejected with clear error", () => {
+  const r = validateEvidenceRecord("not an object");
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    expect(r.errors[0]).toContain("expected object");
+  }
+});
+
+test("negative: human_override with invalid decision is rejected", () => {
+  const fixture = loadFixture("evidence_positive_distilled_fact.json") as Record<string, unknown>;
+  fixture.human_override = {
+    overrider: "test-user",
+    decision: "maybe",  // invalid — must be accept|reject|needs_review
+    reason: "test",
+    overridden_at: "2026-04-26T22:30:00.000Z",
+  };
+  const r = validateEvidenceRecord(fixture);
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    expect(r.errors.some(e => e.includes("human_override.decision"))).toBe(true);
+  }
+});
+
+test("positive: human_override = null is allowed (explicitly no override)", () => {
+  const fixture = loadFixture("evidence_positive_distilled_fact.json") as Record<string, unknown>;
+  fixture.human_override = null;
+  const r = validateEvidenceRecord(fixture);
+  expect(r.valid).toBe(true);
+});
+
+test("negative: observer_confidence outside [0, 100] is rejected", () => {
+  const fixture = loadFixture("evidence_positive_contract_analysis.json") as Record<string, unknown>;
+  fixture.observer_confidence = 150;
+  const r = validateEvidenceRecord(fixture);
+  expect(r.valid).toBe(false);
+  if (!r.valid) {
+    expect(r.errors.some(e => e.includes("observer_confidence"))).toBe(true);
+  }
+});

auditor/schemas/distillation/evidence_record.ts

@@ -0,0 +1,202 @@
+// EvidenceRecord — the unified per-execution-trace record that the
+// Evidence View emits and the Success Scorer reads.
+//
+// Derived from now.md spec + reconciliation of two existing prototypes:
+//   - distilled_facts.jsonl / distilled_procedures.jsonl (LLM-extracted
+//     text with run_id + sig_hash + extractor + verifier + embedding)
+//   - contract_analyses.jsonl (observer integration + retrieval
+//     telemetry + cost + duration)
+//
+// Required fields are the ones every record MUST have for traceability:
+// run_id, task_id, timestamp, schema_version, provenance. Everything
+// else is typed-but-optional because no single source has all of them
+// — the Evidence View materializes them by JOINing across streams when
+// the source data is present.
+//
+// schema_version starts at 1 and gets bumped on breaking changes.
+// Validators MUST check schema_version and refuse unknown values so a
+// future v2 reader doesn't silently accept v1 records (or vice versa).
+
+import {
+  ValidationResult, Provenance,
+  requireString, requireNumber, requireIsoTimestamp, requireProvenance, requireStringArray,
+} from "./types";
+
+export const EVIDENCE_SCHEMA_VERSION = 1;
+
+export type ModelRole =
+  | "executor"      // produced the answer (e.g. scrum reviewer, mode runner LLM call)
+  | "reviewer"      // judged an executor output (e.g. observer, hand-review)
+  | "extractor"     // pulled structured data from text (e.g. fact_extractor)
+  | "verifier"      // confirmed/rejected an extracted claim (verifier in distilled_*)
+  | "categorizer"   // assigned a category (categorizer in distilled_*)
+  | "tiebreaker"    // resolved a consensus split
+  | "applier"       // landed code (scrum_applier)
+  | "embedder"      // produced embeddings
+  | "other";
+
+export interface EvidenceRecord {
+  // ── Identity ──
+  // run_id ties this record to a specific execution. Sources use it
+  // inconsistently (some stream-level, some per-call). The Evidence
+  // View canonicalizes to per-call; if the source is stream-level,
+  // synthesize as `${stream_run_id}:${row_index}`.
+  run_id: string;
+
+  // task_id groups records by logical task (e.g. one PR = one task_id
+  // across multiple per-call runs). Defaults to run_id when no group
+  // exists — never null.
+  task_id: string;
+
+  // ISO 8601 of when the EXECUTION happened, not when this record was
+  // materialized. Use the source row's timestamp; provenance carries
+  // the materialization time separately.
+  timestamp: string;
+
+  schema_version: number;
+
+  // ── Provenance ── (required — no record without source linkage)
+  provenance: Provenance;
+
+  // ── Model attribution (optional) ──
+  model_name?: string;          // e.g. "kimi-k2:1t", "gpt-oss:120b"
+  model_provider?: string;      // e.g. "ollama_cloud", "openrouter", "ollama"
+  model_role?: ModelRole;
+
+  // ── Content hashes (optional) ──
+  // sha256 of the full input prompt and full output content. Pre-
+  // computed so the Evidence Index can dedup across re-runs of the
+  // same prompt without re-hashing.
+  input_hash?: string;
+  output_hash?: string;
+
+  // ── Repo + execution context ──
+  source_files?: string[];      // files the run touched/read
+  commands_run?: string[];      // shell commands or tool calls fired
+  retrieved_context?: {         // what the model saw via retrieval
+    matrix_corpora?: string[];
+    matrix_hits?: number;
+    matrix_chunks_kept?: number;
+    matrix_chunks_dropped?: number;
+    pathway_fingerprints_seen?: number;
+  };
+
+  // ── Observer + scratchpad ──
+  observer_notes?: string[];    // observer.review() free-form notes
+  observer_verdict?: "accept" | "reject" | "cycle" | string;
+  observer_confidence?: number; // 0-100
+  scratchpad_summary?: string;  // tree-split scratchpad text or hash ref
+
+  // ── Outcome markers ──
+  // Both arrays exist because a run can have multiple succeeded gates
+  // AND multiple failed gates simultaneously. Empty arrays are valid;
+  // missing arrays are also valid (means "no evidence either way").
+  success_markers?: string[];   // e.g. "cargo_green", "tests_passed", "anchor_grounded"
+  failure_markers?: string[];   // e.g. "warning_count_up", "rationale_mismatch", "consensus_split"
+
+  // ── Validation telemetry ──
+  validation_results?: {
+    grounded_fraction?: number; // mode_compare grounding %
+    schema_valid?: boolean;
+    pathway_replay_succeeded?: boolean;
+    [key: string]: unknown;
+  };
+
+  // ── Human-in-loop ──
+  human_override?: {
+    overrider: string;          // user identifier
+    decision: "accept" | "reject" | "needs_review";
+    reason: string;
+    overridden_at: string;      // ISO 8601
+  } | null;
+
+  // ── Performance ──
+  cost_usd?: number;
+  latency_ms?: number;
+  prompt_tokens?: number;
+  completion_tokens?: number;
+
+  // ── Free-form text content (the actual run output) ──
+  // Optional because some sources are pure metadata (auto_apply.jsonl)
+  // and have no text payload. Present for distilled_*, contract_analyses,
+  // mode_experiments, scrum_reviews etc.
+  text?: string;
+
+  // ── Domain-specific metadata bucket ──
+  // Source-specific fields that don't earn a top-level slot. e.g.
+  // contract_analyses rows carry `contractor` here; mode_experiments
+  // could carry `corpus_set`. Typed scalar values only — keep this
+  // small or it becomes a junk drawer. Added 2026-04-27 (Kimi audit
+  // flagged `(ev as any).contractor` schema bypass at export_sft.ts:126).
+  metadata?: Record<string, string | number | boolean>;
+}
+
+export function validateEvidenceRecord(input: unknown): ValidationResult<EvidenceRecord> {
+  const errors: string[] = [];
+
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object, got " + (input === null ? "null" : typeof input)] };
+  }
+  const r = input as Record<string, unknown>;
+
+  // Required
+  let ok = true;
+  ok = requireString(r.run_id, "run_id", errors) && ok;
+  ok = requireString(r.task_id, "task_id", errors) && ok;
+  ok = requireIsoTimestamp(r.timestamp, "timestamp", errors) && ok;
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  if (r.schema_version !== EVIDENCE_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${EVIDENCE_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+
+  // Optional but typed-when-present
+  if (r.model_role !== undefined) {
+    const valid: ModelRole[] = ["executor", "reviewer", "extractor", "verifier", "categorizer", "tiebreaker", "applier", "embedder", "other"];
+    if (!valid.includes(r.model_role as ModelRole)) {
+      errors.push(`model_role: must be one of ${valid.join("|")}, got ${JSON.stringify(r.model_role)}`);
+      ok = false;
+    }
+  }
+  if (r.input_hash !== undefined && !/^[0-9a-f]{64}$/.test(String(r.input_hash))) {
+    errors.push("input_hash: must be hex sha256 when present");
+    ok = false;
+  }
+  if (r.output_hash !== undefined && !/^[0-9a-f]{64}$/.test(String(r.output_hash))) {
+    errors.push("output_hash: must be hex sha256 when present");
+    ok = false;
+  }
+  if (r.source_files !== undefined && !requireStringArray(r.source_files, "source_files", errors)) ok = false;
+  if (r.commands_run !== undefined && !requireStringArray(r.commands_run, "commands_run", errors)) ok = false;
+  if (r.success_markers !== undefined && !requireStringArray(r.success_markers, "success_markers", errors)) ok = false;
+  if (r.failure_markers !== undefined && !requireStringArray(r.failure_markers, "failure_markers", errors)) ok = false;
+  if (r.observer_notes !== undefined && !requireStringArray(r.observer_notes, "observer_notes", errors)) ok = false;
+
+  if (r.observer_confidence !== undefined) {
+    if (!requireNumber(r.observer_confidence, "observer_confidence", errors)) ok = false;
+    else if ((r.observer_confidence as number) < 0 || (r.observer_confidence as number) > 100) {
+      errors.push("observer_confidence: must be in [0, 100]");
+      ok = false;
+    }
+  }
+
+  if (r.human_override !== undefined && r.human_override !== null) {
+    const ho = r.human_override as Record<string, unknown>;
+    if (typeof ho !== "object") {
+      errors.push("human_override: expected object or null");
+      ok = false;
+    } else {
+      ok = requireString(ho.overrider, "human_override.overrider", errors) && ok;
+      ok = requireString(ho.reason, "human_override.reason", errors) && ok;
+      ok = requireIsoTimestamp(ho.overridden_at, "human_override.overridden_at", errors) && ok;
+      if (!["accept", "reject", "needs_review"].includes(ho.decision as string)) {
+        errors.push(`human_override.decision: must be accept|reject|needs_review`);
+        ok = false;
+      }
+    }
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as EvidenceRecord };
+}

auditor/schemas/distillation/fixtures/evidence_negative_bad_provenance.json

@@ -0,0 +1,11 @@
+{
+  "run_id": "cae21289",
+  "task_id": "team_runs:637",
+  "timestamp": "2026-04-23T09:54:40.729599Z",
+  "schema_version": 1,
+  "provenance": {
+    "source_file": "data/_kb/distilled_facts.jsonl",
+    "sig_hash": "not-a-real-sha256",
+    "recorded_at": "yesterday"
+  }
+}

auditor/schemas/distillation/fixtures/evidence_negative_bad_schema_version.json

@@ -0,0 +1,11 @@
+{
+  "run_id": "cae21289",
+  "task_id": "team_runs:637",
+  "timestamp": "2026-04-23T09:54:40.729599Z",
+  "schema_version": 99,
+  "provenance": {
+    "source_file": "data/_kb/distilled_facts.jsonl",
+    "sig_hash": "21a809e2dc43dfae0000000000000000000000000000000000000000deadbeef",
+    "recorded_at": "2026-04-26T22:30:00.000Z"
+  }
+}

auditor/schemas/distillation/fixtures/evidence_negative_no_run_id.json

@@ -0,0 +1,11 @@
+{
+  "task_id": "team_runs:637",
+  "timestamp": "2026-04-23T09:54:40.729599Z",
+  "schema_version": 1,
+  "provenance": {
+    "source_file": "data/_kb/distilled_facts.jsonl",
+    "sig_hash": "21a809e2dc43dfae0000000000000000000000000000000000000000deadbeef",
+    "recorded_at": "2026-04-26T22:30:00.000Z"
+  },
+  "text": "missing run_id should fail validation"
+}

auditor/schemas/distillation/fixtures/evidence_positive_contract_analysis.json

@@ -0,0 +1,27 @@
+{
+  "run_id": "contract_analysis:101078392:1777250758717",
+  "task_id": "permit:101078392",
+  "timestamp": "2026-04-25T23:45:58.717Z",
+  "schema_version": 1,
+  "provenance": {
+    "source_file": "data/_kb/contract_analyses.jsonl",
+    "line_offset": 0,
+    "sig_hash": "f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1",
+    "recorded_at": "2026-04-26T22:30:00.000Z"
+  },
+  "model_name": "kimi-k2:1t",
+  "model_role": "executor",
+  "model_provider": "ollama_cloud",
+  "retrieved_context": {
+    "matrix_corpora": ["entity_brief_v1", "chicago_permits_v1", "distilled_procedural_v20260423102847", "sec_tickers_v1"],
+    "matrix_hits": 10
+  },
+  "observer_notes": ["contractor history shows 0 prior fills in Chicago downtown zone"],
+  "observer_verdict": "reject",
+  "observer_confidence": 95,
+  "success_markers": ["matrix_hits_above_threshold"],
+  "failure_markers": ["observer_rejected"],
+  "cost_usd": 0.0002,
+  "latency_ms": 25419,
+  "text": "Permit 101078392 contractor ANTHONY FIORE — analysis: insufficient prior performance signal; recommend escalation."
+}

auditor/schemas/distillation/fixtures/evidence_positive_distilled_fact.json

@@ -0,0 +1,19 @@
+{
+  "run_id": "cae21289",
+  "task_id": "team_runs:637",
+  "timestamp": "2026-04-23T09:54:40.729599Z",
+  "schema_version": 1,
+  "provenance": {
+    "source_file": "data/_kb/distilled_facts.jsonl",
+    "line_offset": 0,
+    "sig_hash": "21a809e2dc43dfae0000000000000000000000000000000000000000deadbeef",
+    "recorded_at": "2026-04-26T22:30:00.000Z"
+  },
+  "model_name": "qwen2.5:latest",
+  "model_role": "extractor",
+  "model_provider": "ollama",
+  "text": "Convergence refers to the system stabilizing into a state of high performance with low variance across iterations.",
+  "validation_results": {
+    "schema_valid": true
+  }
+}

auditor/schemas/distillation/model_ledger.ts

@@ -0,0 +1,56 @@
+// ModelLedgerEntry — aggregate per-task-type-per-model performance.
+// Built by aggregating mode_experiments.jsonl + model_trust.jsonl.
+// Updated rather than appended — one row per (model_name, task_type)
+// representing latest aggregates.
+import {
+  ValidationResult, requireString, requireNumber, requireIsoTimestamp, requireStringArray,
+} from "./types";
+
+export const MODEL_LEDGER_SCHEMA_VERSION = 1;
+
+export interface ModelLedgerEntry {
+  schema_version: number;
+  model_name: string;
+  model_provider: string;
+  task_type: string;
+  success_rate: number;          // [0, 1]
+  failure_modes: string[];       // top failure mode tags
+  best_partner_model?: string;   // pairs well with X (consensus / tie-break)
+  escalation_role?: string;      // when this model gets escalated TO (or FROM)
+  cost_usd_p50?: number;
+  latency_ms_p50?: number;
+  latency_ms_p95?: number;
+  context_window?: number;
+  sample_count: number;
+  last_updated: string;          // ISO 8601
+  notes?: string;
+}
+
+export function validateModelLedgerEntry(input: unknown): ValidationResult<ModelLedgerEntry> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== MODEL_LEDGER_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${MODEL_LEDGER_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.model_name, "model_name", errors) && ok;
+  ok = requireString(r.model_provider, "model_provider", errors) && ok;
+  ok = requireString(r.task_type, "task_type", errors) && ok;
+  ok = requireIsoTimestamp(r.last_updated, "last_updated", errors) && ok;
+  ok = requireStringArray(r.failure_modes, "failure_modes", errors) && ok;
+
+  if (!requireNumber(r.success_rate, "success_rate", errors)) ok = false;
+  else if ((r.success_rate as number) < 0 || (r.success_rate as number) > 1) {
+    errors.push("success_rate: must be in [0, 1]"); ok = false;
+  }
+  if (!requireNumber(r.sample_count, "sample_count", errors)) ok = false;
+  else if ((r.sample_count as number) < 1 || !Number.isInteger(r.sample_count)) {
+    errors.push("sample_count: must be positive integer (no aggregate from zero samples)"); ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as ModelLedgerEntry };
+}

auditor/schemas/distillation/playbook.ts

@@ -0,0 +1,68 @@
+// Playbook — procedural knowledge extracted from accepted/partially-
+// accepted runs. Different from pathway_memory's bug_fingerprints (which
+// are pattern-detectors) — playbooks describe HOW to handle a task type.
+import {
+  ValidationResult, requireString, requireIsoTimestamp, requireProvenance, requireStringArray,
+} from "./types";
+
+export const PLAYBOOK_SCHEMA_VERSION = 1;
+
+export interface Playbook {
+  schema_version: number;
+  playbook_id: string;
+  task_type: string;                // e.g. "scrum_review", "pr_audit", "staffing.fill"
+  problem_pattern: string;          // when does this playbook apply?
+  useful_context: string[];         // what to retrieve before running
+  model_routing_path: string[];     // ordered model attempts that worked
+  commands_worked: string[];
+  commands_failed: string[];
+  validation_steps: string[];
+  repo_files_touched: string[];
+  recovery_strategy: string;        // what to do when the path fails
+  known_failure_modes: string[];
+  escalation_threshold: string;     // when to switch to a stronger model
+  acceptance_criteria: string[];    // how to know it succeeded
+  source_run_ids: string[];         // FK to EvidenceRecord.run_id (provenance — every playbook traces to source)
+  created_at: string;
+  provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
+}
+
+export function validatePlaybook(input: unknown): ValidationResult<Playbook> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== PLAYBOOK_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${PLAYBOOK_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.playbook_id, "playbook_id", errors) && ok;
+  ok = requireString(r.task_type, "task_type", errors) && ok;
+  ok = requireString(r.problem_pattern, "problem_pattern", errors) && ok;
+  ok = requireString(r.recovery_strategy, "recovery_strategy", errors) && ok;
+  ok = requireString(r.escalation_threshold, "escalation_threshold", errors) && ok;
+  ok = requireIsoTimestamp(r.created_at, "created_at", errors) && ok;
+  ok = requireStringArray(r.useful_context, "useful_context", errors) && ok;
+  ok = requireStringArray(r.model_routing_path, "model_routing_path", errors) && ok;
+  ok = requireStringArray(r.commands_worked, "commands_worked", errors) && ok;
+  ok = requireStringArray(r.commands_failed, "commands_failed", errors) && ok;
+  ok = requireStringArray(r.validation_steps, "validation_steps", errors) && ok;
+  ok = requireStringArray(r.repo_files_touched, "repo_files_touched", errors) && ok;
+  ok = requireStringArray(r.known_failure_modes, "known_failure_modes", errors) && ok;
+  ok = requireStringArray(r.acceptance_criteria, "acceptance_criteria", errors) && ok;
+  ok = requireStringArray(r.source_run_ids, "source_run_ids", errors) && ok;
+
+  if (Array.isArray(r.source_run_ids) && r.source_run_ids.length === 0) {
+    errors.push("source_run_ids: must be non-empty — every playbook traces to source evidence (spec non-negotiable)");
+    ok = false;
+  }
+  if (Array.isArray(r.acceptance_criteria) && r.acceptance_criteria.length === 0) {
+    errors.push("acceptance_criteria: must be non-empty — every playbook needs success criteria (spec non-negotiable)");
+    ok = false;
+  }
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as Playbook };
+}

auditor/schemas/distillation/preference_sample.ts

@@ -0,0 +1,60 @@
+// PreferenceSample — entry in exports/preference/chosen_rejected.jsonl.
+// Source: real disagreements (audit_discrepancies, scrum ladder retries).
+// Validator pins: chosen != rejected, both source_run_ids present, reason
+// is non-empty. No synthesized preferences.
+import {
+  ValidationResult, requireString, requireIsoTimestamp, requireProvenance,
+} from "./types";
+
+export const PREFERENCE_SAMPLE_SCHEMA_VERSION = 1;
+
+export interface PreferenceSample {
+  schema_version: number;
+  id: string;
+  prompt: string;
+  chosen: string;
+  rejected: string;
+  reason: string;                  // why chosen > rejected — must be non-empty
+  chosen_run_id: string;
+  rejected_run_id: string;
+  created_at: string;
+  provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
+}
+
+export function validatePreferenceSample(input: unknown): ValidationResult<PreferenceSample> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== PREFERENCE_SAMPLE_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${PREFERENCE_SAMPLE_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.id, "id", errors) && ok;
+  ok = requireString(r.prompt, "prompt", errors) && ok;
+  ok = requireString(r.chosen, "chosen", errors) && ok;
+  ok = requireString(r.rejected, "rejected", errors) && ok;
+  ok = requireString(r.reason, "reason", errors) && ok;
+  ok = requireString(r.chosen_run_id, "chosen_run_id", errors) && ok;
+  ok = requireString(r.rejected_run_id, "rejected_run_id", errors) && ok;
+  ok = requireIsoTimestamp(r.created_at, "created_at", errors) && ok;
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  // Self-pairing guard.
+  if (r.chosen === r.rejected && typeof r.chosen === "string") {
+    errors.push("chosen and rejected must differ — preference data needs a real disagreement");
+    ok = false;
+  }
+  if (r.chosen_run_id === r.rejected_run_id && typeof r.chosen_run_id === "string") {
+    errors.push("chosen_run_id and rejected_run_id must differ — same run can't disagree with itself");
+    ok = false;
+  }
+  if (typeof r.reason === "string" && (r.reason as string).trim().length === 0) {
+    errors.push("reason: must be non-whitespace (every preference needs WHY chosen > rejected)");
+    ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as PreferenceSample };
+}

auditor/schemas/distillation/rag_sample.ts

@@ -0,0 +1,72 @@
+// RagSample — entry in exports/rag/playbooks.jsonl. Spec shape exactly,
+// plus provenance + success_score (so the index can re-rank by quality).
+import {
+  ValidationResult, requireString, requireNumber, requireIsoTimestamp, requireProvenance, requireStringArray,
+} from "./types";
+
+export const RAG_SAMPLE_SCHEMA_VERSION = 1;
+
+// Allowed source_category values. RAG accepts accepted/partial freely;
+// needs_human_review is opt-in (must be tagged so consumers can filter
+// it out for SFT).
+export const RAG_ALLOWED_CATEGORIES = ["accepted", "partially_accepted", "needs_human_review"] as const;
+export type RagSourceCategory = (typeof RAG_ALLOWED_CATEGORIES)[number];
+
+export interface RagSample {
+  schema_version: number;
+  id: string;
+  title: string;
+  content: string;
+  tags: string[];
+  source_run_id: string;
+  // Snapshot of the score the source carried at export time. Lets a
+  // consumer see "this was partial" without re-reading scored-runs.
+  success_score: RagSourceCategory;
+  // Same value as success_score by spec (now.md asks for both fields).
+  // Kept distinct so future schemas can diverge them (e.g. an
+  // "is_review_material" flag) without breaking old consumers.
+  source_category: RagSourceCategory;
+  embedding_text: string;                            // the text to embed (often == content but can be shorter)
+  created_at: string;
+  provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
+}
+
+export function validateRagSample(input: unknown): ValidationResult<RagSample> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== RAG_SAMPLE_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${RAG_SAMPLE_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.id, "id", errors) && ok;
+  ok = requireString(r.title, "title", errors) && ok;
+  ok = requireString(r.content, "content", errors) && ok;
+  ok = requireString(r.embedding_text, "embedding_text", errors) && ok;
+  ok = requireString(r.source_run_id, "source_run_id", errors) && ok;
+  ok = requireIsoTimestamp(r.created_at, "created_at", errors) && ok;
+  ok = requireStringArray(r.tags, "tags", errors) && ok;
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  if (!RAG_ALLOWED_CATEGORIES.includes(r.success_score as RagSourceCategory)) {
+    errors.push(`success_score: must be one of ${RAG_ALLOWED_CATEGORIES.join("|")} (rejected never enters RAG)`);
+    ok = false;
+  }
+  if (!RAG_ALLOWED_CATEGORIES.includes(r.source_category as RagSourceCategory)) {
+    errors.push(`source_category: must be one of ${RAG_ALLOWED_CATEGORIES.join("|")}`);
+    ok = false;
+  }
+  if (r.success_score !== r.source_category) {
+    errors.push("success_score and source_category must match (mirrored fields per spec)");
+    ok = false;
+  }
+  if (typeof r.content === "string" && (r.content as string).trim().length === 0) {
+    errors.push("content: must be non-whitespace");
+    ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as RagSample };
+}

auditor/schemas/distillation/realdata.test.ts

@@ -0,0 +1,286 @@
+// Real-data validation test — proves the EvidenceRecord schema fits
+// what we ALREADY produce, with the minimum transformation each source
+// stream requires. Doubles as the stale-extraction probe: if
+// distilled_facts.jsonl rows can't materialize, we know that stream
+// has rotted and Phase 2 sources from elsewhere.
+//
+// Strategy:
+//   1. Read first N rows from each source jsonl (skip if missing)
+//   2. Apply minimal transformer: add schema_version + provenance,
+//      synthesize run_id/task_id when source doesn't carry them
+//   3. Validate each materialized record
+//   4. Tally pass/fail per source + collect failure reasons
+//
+// This file is allowed to skip when source files don't exist (fresh
+// clone), so it acts as both a CI guard and a real-environment probe.
+
+import { test, expect } from "bun:test";
+import { existsSync, readFileSync } from "node:fs";
+import { resolve } from "node:path";
+
+import {
+  validateEvidenceRecord, EVIDENCE_SCHEMA_VERSION, EvidenceRecord, ModelRole,
+} from "./evidence_record";
+
+const ROOT = "/home/profit/lakehouse";
+const SAMPLE_PER_SOURCE = 10;
+
+interface SourceProbe {
+  source_file: string;
+  transform: (row: any, lineNo: number) => Partial<EvidenceRecord> | null;
+}
+
+// Canonical 64-char synthetic sha256 for tests where the source row
+// lacks one. Pretends the materializer would compute it via
+// canonicalSha256(orderedKeys(row)) at Phase 2 time. We use a fixed
+// value here to keep the test deterministic; real materialization
+// re-hashes per row.
+const PLACEHOLDER_SHA = "0000000000000000000000000000000000000000000000000000000000000000";
+const RECORDED = "2026-04-26T22:30:00.000Z";
+
+function provFor(source_file: string, lineNo: number, sigHashRaw?: string): EvidenceRecord["provenance"] {
+  // Pad shorter hashes (distilled_* uses 16-char) to 64 — mimics
+  // canonical recompute.
+  const sig = sigHashRaw && /^[0-9a-f]+$/.test(sigHashRaw)
+    ? sigHashRaw.padEnd(64, "0").slice(0, 64)
+    : PLACEHOLDER_SHA;
+  return {
+    source_file: source_file.replace(`${ROOT}/`, ""),
+    line_offset: lineNo,
+    sig_hash: sig,
+    recorded_at: RECORDED,
+  };
+}
+
+const PROBES: SourceProbe[] = [
+  {
+    source_file: `${ROOT}/data/_kb/distilled_facts.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: String(row.run_id ?? `distilled_facts:${lineNo}`),
+      task_id: String(row.source_label ?? `distilled_facts:${lineNo}`),
+      timestamp: row.created_at,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/distilled_facts.jsonl`, lineNo, row.sig_hash),
+      model_name: row.extractor,
+      model_role: "extractor" as ModelRole,
+      model_provider: "ollama",
+      text: row.text,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/distilled_procedures.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: String(row.run_id ?? `distilled_procedures:${lineNo}`),
+      task_id: String(row.source_label ?? `distilled_procedures:${lineNo}`),
+      timestamp: row.created_at,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/distilled_procedures.jsonl`, lineNo, row.sig_hash),
+      model_name: row.extractor,
+      model_role: "extractor" as ModelRole,
+      model_provider: "ollama",
+      text: row.text,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/contract_analyses.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: `contract_analysis:${row.permit_id}:${new Date(row.ts).getTime()}`,
+      task_id: `permit:${row.permit_id}`,
+      timestamp: row.ts,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/contract_analyses.jsonl`, lineNo),
+      model_role: "executor" as ModelRole,
+      retrieved_context: {
+        matrix_corpora: Object.keys(row.matrix_corpora ?? {}),
+        matrix_hits: row.matrix_hits,
+      },
+      observer_notes: row.observer_notes ? [row.observer_notes].flat() : undefined,
+      observer_verdict: row.observer_verdict,
+      observer_confidence: row.observer_conf,
+      success_markers: row.ok ? ["matrix_hits_above_threshold"] : undefined,
+      failure_markers: !row.ok || row.observer_verdict === "reject" ? ["observer_rejected"] : undefined,
+      cost_usd: typeof row.cost === "number" ? row.cost / 1_000_000 : undefined,
+      latency_ms: row.duration_ms,
+      text: row.analysis,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/mode_experiments.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: `mode_exec:${new Date(row.ts).getTime()}:${row.file_path ?? "?"}`,
+      task_id: row.task_class,
+      timestamp: row.ts,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/mode_experiments.jsonl`, lineNo),
+      model_name: row.model,
+      model_role: "executor" as ModelRole,
+      model_provider: row.model?.includes("/") ? "openrouter" : "ollama_cloud",
+      retrieved_context: {
+        matrix_corpora: row.sources?.matrix_corpus,
+        matrix_chunks_kept: row.sources?.matrix_chunks_kept,
+        matrix_chunks_dropped: row.sources?.matrix_chunks_dropped,
+        pathway_fingerprints_seen: row.sources?.bug_fingerprints_count,
+      },
+      latency_ms: row.latency_ms,
+      text: row.response,
+      source_files: row.file_path ? [row.file_path] : undefined,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/scrum_reviews.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: `scrum:${new Date(row.reviewed_at).getTime()}:${row.file}`,
+      task_id: `scrum_review:${row.file}`,
+      timestamp: row.reviewed_at,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/scrum_reviews.jsonl`, lineNo),
+      model_name: row.accepted_model,
+      model_role: "executor" as ModelRole,
+      source_files: [row.file],
+      success_markers: row.accepted_on_attempt ? [`accepted_on_attempt_${row.accepted_on_attempt}`] : undefined,
+      text: row.suggestions_preview,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/observer_escalations.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: `obs_esc:${new Date(row.ts).getTime()}:${row.sig_hash}`,
+      task_id: `observer_escalation:${row.cluster_endpoint ?? "?"}`,
+      timestamp: row.ts,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/observer_escalations.jsonl`, lineNo, row.sig_hash),
+      model_role: "reviewer" as ModelRole,
+      prompt_tokens: row.prompt_tokens,
+      completion_tokens: row.completion_tokens,
+      text: row.analysis,
+    }),
+  },
+  {
+    source_file: `${ROOT}/data/_kb/audit_facts.jsonl`,
+    transform: (row: any, lineNo: number) => ({
+      run_id: `audit_facts:${row.head_sha}:${lineNo}`,
+      task_id: `pr:${row.pr_number}`,
+      timestamp: row.extracted_at,
+      schema_version: EVIDENCE_SCHEMA_VERSION,
+      provenance: provFor(`${ROOT}/data/_kb/audit_facts.jsonl`, lineNo),
+      model_name: row.extractor,
+      model_role: "extractor" as ModelRole,
+      // facts/entities/relationships go into text as a JSON dump for now;
+      // structured handling lives in Phase 2 where we map to specific
+      // EvidenceRecord substructures.
+      text: JSON.stringify({
+        facts: row.facts?.length ?? 0,
+        entities: row.entities?.length ?? 0,
+        relationships: row.relationships?.length ?? 0,
+      }),
+    }),
+  },
+];
+
+interface ProbeResult {
+  source_file: string;
+  rows_attempted: number;
+  rows_present: boolean;
+  passed: number;
+  failed: number;
+  failure_reasons: string[];   // unique error strings, top 5
+}
+
+const RESULTS: ProbeResult[] = [];
+
+for (const probe of PROBES) {
+  const sourceLabel = probe.source_file.replace(`${ROOT}/`, "");
+
+  test(`real-data: ${sourceLabel}`, () => {
+    const result: ProbeResult = {
+      source_file: sourceLabel,
+      rows_attempted: 0,
+      rows_present: false,
+      passed: 0,
+      failed: 0,
+      failure_reasons: [],
+    };
+
+    if (!existsSync(probe.source_file)) {
+      RESULTS.push(result);
+      // Skip silently — fresh clones won't have these files
+      return;
+    }
+
+    result.rows_present = true;
+    const lines = readFileSync(probe.source_file, "utf8").split("\n").filter(Boolean).slice(0, SAMPLE_PER_SOURCE);
+    const reasons = new Set<string>();
+
+    for (let i = 0; i < lines.length; i++) {
+      result.rows_attempted++;
+      let row: unknown;
+      try { row = JSON.parse(lines[i]); }
+      catch { continue; }
+
+      const transformed = probe.transform(row, i);
+      if (!transformed) continue;
+
+      const v = validateEvidenceRecord(transformed);
+      if (v.valid) result.passed++;
+      else {
+        result.failed++;
+        for (const e of v.errors) reasons.add(e);
+      }
+    }
+    result.failure_reasons = Array.from(reasons).slice(0, 5);
+    RESULTS.push(result);
+
+    // Test passes as long as we attempted something and got a result.
+    // Per-source pass/fail counts are reported in the markdown writeup.
+    expect(result.rows_attempted).toBeGreaterThanOrEqual(0);
+  });
+}
+
+test("real-data: emit markdown report", () => {
+  const md: string[] = [];
+  md.push("# Real-data validation report");
+  md.push("");
+  md.push("Schema = EvidenceRecord v" + EVIDENCE_SCHEMA_VERSION + ". Sample = first " + SAMPLE_PER_SOURCE + " rows per source.");
+  md.push("");
+  md.push("| Source | Present | Rows | Pass | Fail | Pass% |");
+  md.push("|---|---|---|---|---|---|");
+  for (const r of RESULTS) {
+    const pct = r.rows_attempted > 0 ? Math.round(100 * r.passed / r.rows_attempted) + "%" : "—";
+    md.push(`| ${r.source_file} | ${r.rows_present ? "✓" : "—"} | ${r.rows_attempted} | ${r.passed} | ${r.failed} | ${pct} |`);
+  }
+  md.push("");
+  let hasFailures = false;
+  for (const r of RESULTS) {
+    if (r.failed > 0) {
+      hasFailures = true;
+      md.push(`## Failures in ${r.source_file}`);
+      for (const reason of r.failure_reasons) md.push(`- \`${reason}\``);
+      md.push("");
+    }
+  }
+  if (!hasFailures) {
+    md.push("**No failures across all probed sources.** Every materialized record validates against EvidenceRecord v1.");
+    md.push("");
+  }
+  // Stale extraction probe: explicit pass/fail
+  const distilledFacts = RESULTS.find(r => r.source_file.endsWith("distilled_facts.jsonl"));
+  const distilledProc = RESULTS.find(r => r.source_file.endsWith("distilled_procedures.jsonl"));
+  md.push("## Stale-extraction probe");
+  md.push("");
+  if (distilledFacts && distilledFacts.rows_present && distilledFacts.passed > 0) {
+    md.push(`- **distilled_facts.jsonl:** ${distilledFacts.passed}/${distilledFacts.rows_attempted} materialize cleanly. Stream is alive at the schema level.`);
+  } else if (distilledFacts && !distilledFacts.rows_present) {
+    md.push(`- **distilled_facts.jsonl:** missing — stale or never produced. Phase 2 sources from live streams instead.`);
+  } else {
+    md.push(`- **distilled_facts.jsonl:** present but materialization failures; treat as suspect, prefer mode_experiments + scrum_reviews.`);
+  }
+  if (distilledProc && distilledProc.rows_present && distilledProc.passed > 0) {
+    md.push(`- **distilled_procedures.jsonl:** ${distilledProc.passed}/${distilledProc.rows_attempted} materialize cleanly.`);
+  }
+  md.push("");
+
+  // Write the markdown to a stable path and stdout
+  const out = md.join("\n");
+  Bun.write(`${ROOT}/data/_kb/realdata_validation_report.md`, out);
+  console.log("\n" + out);
+});

auditor/schemas/distillation/receipt.ts

@@ -0,0 +1,111 @@
+// Receipt — per-pipeline-stage record with everything needed to
+// reproduce the run. Spec non-negotiable: substantive receipts, not
+// "ran successfully". Every field below has a deterministic source so
+// the receipt schema validator catches "I forgot to fill it in" the
+// same way it catches type errors.
+import {
+  ValidationResult, requireString, requireNumber, requireIsoTimestamp,
+} from "./types";
+
+export const RECEIPT_SCHEMA_VERSION = 1;
+
+export interface FileReference {
+  path: string;     // relative to repo root
+  sha256: string;   // hex
+  bytes?: number;   // optional but recommended
+}
+
+export interface Receipt {
+  schema_version: number;
+  command: string;             // shell-line or script identifier
+  git_sha: string;             // 40-char hex (full SHA1)
+  git_branch?: string;
+  git_dirty?: boolean;         // true if working tree had uncommitted changes
+  started_at: string;          // ISO 8601
+  ended_at: string;            // ISO 8601
+  duration_ms: number;
+  input_files: FileReference[];
+  output_files: FileReference[];
+  record_counts: {
+    in: number;
+    out: number;
+    [key: string]: number;     // per-stage extras (filtered, dropped, etc.)
+  };
+  validation_pass: boolean;    // explicit — never inferred
+  errors: string[];
+  warnings: string[];
+}
+
+function validateFileRef(v: unknown, field: string, errors: string[]): boolean {
+  if (typeof v !== "object" || v === null) {
+    errors.push(`${field}: expected object`);
+    return false;
+  }
+  const f = v as Record<string, unknown>;
+  let ok = true;
+  ok = requireString(f.path, `${field}.path`, errors) && ok;
+  if (typeof f.sha256 !== "string" || !/^[0-9a-f]{64}$/.test(f.sha256)) {
+    errors.push(`${field}.sha256: must be hex sha256`);
+    ok = false;
+  }
+  if (f.bytes !== undefined && typeof f.bytes !== "number") {
+    errors.push(`${field}.bytes: expected number when present`);
+    ok = false;
+  }
+  return ok;
+}
+
+export function validateReceipt(input: unknown): ValidationResult<Receipt> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object"] };
+  }
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== RECEIPT_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${RECEIPT_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.command, "command", errors) && ok;
+  if (typeof r.git_sha !== "string" || !/^[0-9a-f]{40}$/.test(r.git_sha as string)) {
+    errors.push("git_sha: must be 40-char hex");
+    ok = false;
+  }
+  ok = requireIsoTimestamp(r.started_at, "started_at", errors) && ok;
+  ok = requireIsoTimestamp(r.ended_at, "ended_at", errors) && ok;
+  ok = requireNumber(r.duration_ms, "duration_ms", errors) && ok;
+  if (typeof r.validation_pass !== "boolean") {
+    errors.push("validation_pass: must be boolean (explicit, never inferred)");
+    ok = false;
+  }
+  if (!Array.isArray(r.input_files)) {
+    errors.push("input_files: expected array");
+    ok = false;
+  } else {
+    for (let i = 0; i < r.input_files.length; i++) {
+      if (!validateFileRef(r.input_files[i], `input_files[${i}]`, errors)) ok = false;
+    }
+  }
+  if (!Array.isArray(r.output_files)) {
+    errors.push("output_files: expected array");
+    ok = false;
+  } else {
+    for (let i = 0; i < r.output_files.length; i++) {
+      if (!validateFileRef(r.output_files[i], `output_files[${i}]`, errors)) ok = false;
+    }
+  }
+  if (typeof r.record_counts !== "object" || r.record_counts === null) {
+    errors.push("record_counts: expected object");
+    ok = false;
+  } else {
+    const rc = r.record_counts as Record<string, unknown>;
+    if (typeof rc.in !== "number") { errors.push("record_counts.in: expected number"); ok = false; }
+    if (typeof rc.out !== "number") { errors.push("record_counts.out: expected number"); ok = false; }
+  }
+  if (!Array.isArray(r.errors)) { errors.push("errors: expected array"); ok = false; }
+  if (!Array.isArray(r.warnings)) { errors.push("warnings: expected array"); ok = false; }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as Receipt };
+}

auditor/schemas/distillation/run_summary.ts

@@ -0,0 +1,90 @@
+// run_summary.ts — aggregates StageReceipt rows for one run_id.
+// Spec field set: total records processed, total accepted/rejected/
+// quarantined, dataset sizes, validation status, overall hash of run.
+
+import {
+  ValidationResult, requireString, requireNumber, requireIsoTimestamp, requireSha256,
+} from "./types";
+import type { StageName } from "./stage_receipt";
+
+export const RUN_SUMMARY_SCHEMA_VERSION = 1;
+
+export interface RunStageSummary {
+  stage: StageName;
+  records_in: number;
+  records_out: number;
+  accepted: number;
+  rejected: number;
+  quarantined: number;
+  skipped: number;
+  passed: boolean;
+  duration_ms: number;
+  output_hash: string;
+}
+
+export interface RunSummary {
+  schema_version: number;
+  run_id: string;
+  started_at: string;        // earliest stage timestamp
+  ended_at: string;          // latest stage timestamp + duration
+  git_commit: string;
+  stages: RunStageSummary[];
+  // Aggregates across stages
+  total_records_in: number;
+  total_records_out: number;
+  total_accepted: number;
+  total_rejected: number;
+  total_quarantined: number;
+  total_skipped: number;
+  // Dataset sizes — final outputs of each export stage
+  rag_records: number;
+  sft_records: number;
+  preference_pairs: number;
+  // Pipeline-wide pass = AND of every stage validation.passed
+  overall_passed: boolean;
+  // Run-wide hash: sha256 over each stage's output hash, sorted by stage name.
+  // Detects ANY change in any stage output across runs.
+  run_hash: string;
+  total_duration_ms: number;
+}
+
+export function validateRunSummary(input: unknown): ValidationResult<RunSummary> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object"] };
+  }
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== RUN_SUMMARY_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${RUN_SUMMARY_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.run_id, "run_id", errors) && ok;
+  ok = requireIsoTimestamp(r.started_at, "started_at", errors) && ok;
+  ok = requireIsoTimestamp(r.ended_at, "ended_at", errors) && ok;
+  if (typeof r.git_commit !== "string" || !/^[0-9a-f]{40}$/.test(r.git_commit as string)) {
+    errors.push("git_commit: must be 40-char hex");
+    ok = false;
+  }
+  if (typeof r.overall_passed !== "boolean") {
+    errors.push("overall_passed: must be boolean");
+    ok = false;
+  }
+  ok = requireSha256(r.run_hash, "run_hash", errors) && ok;
+  for (const k of ["total_records_in", "total_records_out", "total_accepted", "total_rejected",
+                   "total_quarantined", "total_skipped", "rag_records", "sft_records",
+                   "preference_pairs", "total_duration_ms"]) {
+    if (typeof (r as any)[k] !== "number") {
+      errors.push(`${k}: expected number`);
+      ok = false;
+    }
+  }
+  if (!Array.isArray(r.stages)) {
+    errors.push("stages: expected array");
+    ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as RunSummary };
+}

auditor/schemas/distillation/schemas.test.ts

@@ -0,0 +1,367 @@
+// Combined schema tests for ScoredRun, Receipt, Playbook,
+// ScratchpadSummary, ModelLedgerEntry, RagSample, SftSample,
+// PreferenceSample. EvidenceRecord lives in its own file because it's
+// the foundational schema and warrants the JSON-fixture round-trip
+// pattern; the rest use inline fixture makers since they're simpler.
+//
+// Each schema: 1 positive fixture + 4-5 negative cases pinning the
+// non-negotiable invariants from now.md.
+//
+// Run: bun test auditor/schemas/distillation/schemas.test.ts
+
+import { test, expect } from "bun:test";
+
+import { validateScoredRun, SCORED_RUN_SCHEMA_VERSION } from "./scored_run";
+import { validateReceipt, RECEIPT_SCHEMA_VERSION } from "./receipt";
+import { validatePlaybook, PLAYBOOK_SCHEMA_VERSION } from "./playbook";
+import { validateScratchpadSummary, SCRATCHPAD_SCHEMA_VERSION } from "./scratchpad_summary";
+import { validateModelLedgerEntry, MODEL_LEDGER_SCHEMA_VERSION } from "./model_ledger";
+import { validateRagSample, RAG_SAMPLE_SCHEMA_VERSION } from "./rag_sample";
+import { validateSftSample, SFT_SAMPLE_SCHEMA_VERSION } from "./sft_sample";
+import { validatePreferenceSample, PREFERENCE_SAMPLE_SCHEMA_VERSION } from "./preference_sample";
+
+const NOW = "2026-04-26T22:30:00.000Z";
+const SHA = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
+const GIT_SHA = "f753e11157eef753e11157eef753e11157eef753";
+
+const PROVENANCE = {
+  source_file: "data/_kb/scored_runs.jsonl",
+  line_offset: 0,
+  sig_hash: SHA,
+  recorded_at: NOW,
+};
+
+// ─── ScoredRun ───────────────────────────────────────────────────────
+
+const SCORED_RUN_OK = {
+  schema_version: SCORED_RUN_SCHEMA_VERSION,
+  evidence_run_id: "run-abc",
+  evidence_task_id: "task-abc",
+  category: "accepted",
+  reasons: ["cargo_green=true", "anchor_grounding=0.95"],
+  scored_at: NOW,
+  scorer_version: "v1.0.0",
+  sub_scores: { cargo_green: true, anchor_grounding: 0.95 },
+  provenance: PROVENANCE,
+};
+
+test("ScoredRun: positive validates", () => {
+  const r = validateScoredRun(SCORED_RUN_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("ScoredRun: empty reasons rejected (every score needs a reason)", () => {
+  const r = validateScoredRun({ ...SCORED_RUN_OK, reasons: [] });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("reasons"))).toBe(true);
+});
+
+test("ScoredRun: invalid category rejected", () => {
+  const r = validateScoredRun({ ...SCORED_RUN_OK, category: "maybe_ok" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("category"))).toBe(true);
+});
+
+test("ScoredRun: anchor_grounding > 1 rejected (must be in [0, 1])", () => {
+  const r = validateScoredRun({ ...SCORED_RUN_OK, sub_scores: { ...SCORED_RUN_OK.sub_scores, anchor_grounding: 1.5 } });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("anchor_grounding"))).toBe(true);
+});
+
+// ─── Receipt ─────────────────────────────────────────────────────────
+
+const RECEIPT_OK = {
+  schema_version: RECEIPT_SCHEMA_VERSION,
+  command: "bun run scripts/build_evidence_index.ts",
+  git_sha: GIT_SHA,
+  git_branch: "scrum/auto-apply-19814",
+  git_dirty: false,
+  started_at: NOW,
+  ended_at: NOW,
+  duration_ms: 1234,
+  input_files: [{ path: "data/_kb/scrum_reviews.jsonl", sha256: SHA, bytes: 448000 }],
+  output_files: [{ path: "data/evidence/2026/04/26/run.jsonl", sha256: SHA }],
+  record_counts: { in: 100, out: 95, filtered: 5 },
+  validation_pass: true,
+  errors: [],
+  warnings: [],
+};
+
+test("Receipt: positive validates", () => {
+  const r = validateReceipt(RECEIPT_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("Receipt: bad git_sha rejected (must be 40-char hex)", () => {
+  const r = validateReceipt({ ...RECEIPT_OK, git_sha: "abc123" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("git_sha"))).toBe(true);
+});
+
+test("Receipt: validation_pass must be boolean (never inferred)", () => {
+  const r = validateReceipt({ ...RECEIPT_OK, validation_pass: "yes" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("validation_pass"))).toBe(true);
+});
+
+test("Receipt: file refs without proper sha256 rejected", () => {
+  const r = validateReceipt({ ...RECEIPT_OK, output_files: [{ path: "x", sha256: "short" }] });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("sha256"))).toBe(true);
+});
+
+// ─── Playbook ────────────────────────────────────────────────────────
+
+const PLAYBOOK_OK = {
+  schema_version: PLAYBOOK_SCHEMA_VERSION,
+  playbook_id: "pb-scrum-review-001",
+  task_type: "scrum_review",
+  problem_pattern: "Cargo workspace warning escalation after applier patch",
+  useful_context: ["pathway memory bug fingerprints for the file area"],
+  model_routing_path: ["x-ai/grok-4.1-fast"],
+  commands_worked: ["cargo check --workspace"],
+  commands_failed: [],
+  validation_steps: ["warning count must not increase"],
+  repo_files_touched: ["crates/queryd/src/service.rs"],
+  recovery_strategy: "git checkout -- file when cargo red",
+  known_failure_modes: ["unused import noise"],
+  escalation_threshold: "use kimi-k2:1t when isolation mode rejects 2 attempts",
+  acceptance_criteria: ["cargo green", "warning count stable", "rationale-diff aligned"],
+  source_run_ids: ["run-xyz", "run-abc"],
+  created_at: NOW,
+  provenance: PROVENANCE,
+};
+
+test("Playbook: positive validates", () => {
+  const r = validatePlaybook(PLAYBOOK_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("Playbook: empty source_run_ids rejected (every playbook traces to source — spec)", () => {
+  const r = validatePlaybook({ ...PLAYBOOK_OK, source_run_ids: [] });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("source_run_ids"))).toBe(true);
+});
+
+test("Playbook: empty acceptance_criteria rejected (every playbook needs success criteria — spec)", () => {
+  const r = validatePlaybook({ ...PLAYBOOK_OK, acceptance_criteria: [] });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("acceptance_criteria"))).toBe(true);
+});
+
+// ─── ScratchpadSummary ───────────────────────────────────────────────
+
+const SCRATCHPAD_OK = {
+  schema_version: SCRATCHPAD_SCHEMA_VERSION,
+  run_id: "run-abc",
+  current_objective: "verify pr_audit mode end-to-end",
+  completed_steps: ["restart gateway"],
+  failed_steps: ["cloud chat returned 500"],
+  pending_steps: ["swap default model"],
+  important_paths: ["auditor/checks/inference.ts"],
+  decisions: ["defer kimi-k2 swap until upstream returns"],
+  unresolved_questions: ["does deepseek match kimi quality?"],
+  validation_status: "partial",
+  next_command: "bun run auditor/audit_one.ts 11",
+  source_scratchpad_hash: SHA,
+  summarized_at: NOW,
+  provenance: PROVENANCE,
+};
+
+test("ScratchpadSummary: positive validates", () => {
+  const r = validateScratchpadSummary(SCRATCHPAD_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("ScratchpadSummary: invalid validation_status rejected", () => {
+  const r = validateScratchpadSummary({ ...SCRATCHPAD_OK, validation_status: "tbd" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("validation_status"))).toBe(true);
+});
+
+test("ScratchpadSummary: short scratchpad_hash rejected", () => {
+  const r = validateScratchpadSummary({ ...SCRATCHPAD_OK, source_scratchpad_hash: "short" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("source_scratchpad_hash"))).toBe(true);
+});
+
+// ─── ModelLedgerEntry ────────────────────────────────────────────────
+
+const LEDGER_OK = {
+  schema_version: MODEL_LEDGER_SCHEMA_VERSION,
+  model_name: "kimi-k2:1t",
+  model_provider: "ollama_cloud",
+  task_type: "pr_audit",
+  success_rate: 0.85,
+  failure_modes: ["upstream_500", "context_truncation"],
+  best_partner_model: "x-ai/grok-4.1-fast",
+  escalation_role: "primary",
+  cost_usd_p50: 0.0002,
+  latency_ms_p50: 50000,
+  latency_ms_p95: 90000,
+  context_window: 200000,
+  sample_count: 47,
+  last_updated: NOW,
+};
+
+test("ModelLedgerEntry: positive validates", () => {
+  const r = validateModelLedgerEntry(LEDGER_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("ModelLedgerEntry: success_rate > 1 rejected", () => {
+  const r = validateModelLedgerEntry({ ...LEDGER_OK, success_rate: 1.5 });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("success_rate"))).toBe(true);
+});
+
+test("ModelLedgerEntry: zero sample_count rejected (no aggregate from zero)", () => {
+  const r = validateModelLedgerEntry({ ...LEDGER_OK, sample_count: 0 });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("sample_count"))).toBe(true);
+});
+
+// ─── RagSample ───────────────────────────────────────────────────────
+
+const RAG_OK = {
+  schema_version: RAG_SAMPLE_SCHEMA_VERSION,
+  id: "rag-pb-001",
+  title: "Scrum applier rationale-diff alignment",
+  content: "When the applier emits a patch with rationale claiming X but the diff shows Y, the rationale-token alignment gate catches it...",
+  tags: ["scrum_review", "applier"],
+  source_run_id: "run-xyz",
+  success_score: "accepted",
+  source_category: "accepted",
+  embedding_text: "applier rationale-diff alignment guard scrum",
+  created_at: NOW,
+  provenance: PROVENANCE,
+};
+
+test("RagSample: positive validates", () => {
+  const r = validateRagSample(RAG_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("RagSample: success_score=rejected forbidden (RAG never takes rejected)", () => {
+  const r = validateRagSample({ ...RAG_OK, success_score: "rejected", source_category: "rejected" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("success_score"))).toBe(true);
+});
+
+test("RagSample: success_score and source_category must match", () => {
+  const r = validateRagSample({ ...RAG_OK, success_score: "accepted", source_category: "partially_accepted" });
+  expect(r.valid).toBe(false);
+});
+
+test("RagSample: whitespace-only content rejected", () => {
+  const r = validateRagSample({ ...RAG_OK, content: "   \n  " });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("content"))).toBe(true);
+});
+
+// ─── SftSample (the strict one) ──────────────────────────────────────
+
+const SFT_OK = {
+  schema_version: SFT_SAMPLE_SCHEMA_VERSION,
+  id: "sft-pr11-001",
+  instruction: "Audit this PR diff against ship-claims.",
+  context: "claims: 3 strong, 2 moderate",
+  response: "{\"claim_verdicts\": [...]}",
+  source_run_id: "run-pr11",
+  quality_score: "accepted",
+  created_at: NOW,
+  provenance: PROVENANCE,
+};
+
+test("SftSample: positive validates", () => {
+  const r = validateSftSample(SFT_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("SftSample: quality_score=partially_accepted ACCEPTED (--include-partial path)", () => {
+  // Phase 4 update: partial allowed at schema layer; CLI gate decides.
+  const r = validateSftSample({ ...SFT_OK, quality_score: "partially_accepted" });
+  expect(r.valid).toBe(true);
+});
+
+test("SftSample: quality_score=rejected REJECTED (spec non-negotiable, no leak)", () => {
+  const r = validateSftSample({ ...SFT_OK, quality_score: "rejected" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("quality_score"))).toBe(true);
+});
+
+test("SftSample: quality_score=needs_human_review REJECTED (no leak)", () => {
+  const r = validateSftSample({ ...SFT_OK, quality_score: "needs_human_review" });
+  expect(r.valid).toBe(false);
+});
+
+test("SftSample: missing context rejected (must be string, even if empty)", () => {
+  const fixture: Record<string, unknown> = { ...SFT_OK };
+  delete fixture.context;
+  const r = validateSftSample(fixture);
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("context"))).toBe(true);
+});
+
+test("SftSample: empty-string context allowed", () => {
+  const r = validateSftSample({ ...SFT_OK, context: "" });
+  expect(r.valid).toBe(true);
+});
+
+test("SftSample: empty response rejected (no empty pairs)", () => {
+  const r = validateSftSample({ ...SFT_OK, response: "" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("response"))).toBe(true);
+});
+
+test("SftSample: whitespace-only instruction rejected", () => {
+  const r = validateSftSample({ ...SFT_OK, instruction: "  \t\n " });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("instruction"))).toBe(true);
+});
+
+// ─── PreferenceSample ────────────────────────────────────────────────
+
+const PREF_OK = {
+  schema_version: PREFERENCE_SAMPLE_SCHEMA_VERSION,
+  id: "pref-task-x-001",
+  prompt: "Verify claim: 'all 3 services running on matrix-test'",
+  chosen: "{\"backed\": true, \"evidence\": \"systemctl status confirms 3 active\"}",
+  rejected: "{\"backed\": true, \"evidence\": \"the README says so\"}",
+  reason: "chosen cites runtime evidence, rejected cites doc claim only",
+  chosen_run_id: "run-A",
+  rejected_run_id: "run-B",
+  created_at: NOW,
+  provenance: PROVENANCE,
+};
+
+test("PreferenceSample: positive validates", () => {
+  const r = validatePreferenceSample(PREF_OK);
+  if (!r.valid) console.error(r.errors);
+  expect(r.valid).toBe(true);
+});
+
+test("PreferenceSample: chosen == rejected rejected (no self-pairing)", () => {
+  const r = validatePreferenceSample({ ...PREF_OK, chosen: "x", rejected: "x" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("chosen and rejected"))).toBe(true);
+});
+
+test("PreferenceSample: chosen_run_id == rejected_run_id rejected (no self-disagreement)", () => {
+  const r = validatePreferenceSample({ ...PREF_OK, chosen_run_id: "run-A", rejected_run_id: "run-A" });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("chosen_run_id"))).toBe(true);
+});
+
+test("PreferenceSample: empty reason rejected (every preference needs WHY)", () => {
+  const r = validatePreferenceSample({ ...PREF_OK, reason: "  " });
+  expect(r.valid).toBe(false);
+  if (!r.valid) expect(r.errors.some(e => e.includes("reason"))).toBe(true);
+});

auditor/schemas/distillation/scored_run.ts

@@ -0,0 +1,86 @@
+// ScoredRun — output of the deterministic Success Scorer (Phase 3).
+// Spec mandates 4 categories with explicit reasons; we add scorer
+// versioning so a future scorer change is detectable in historical data.
+import {
+  ValidationResult, requireString, requireIsoTimestamp, requireProvenance, requireStringArray, requireNumber,
+} from "./types";
+
+export const SCORED_RUN_SCHEMA_VERSION = 1;
+export const SCORE_CATEGORIES = ["accepted", "partially_accepted", "rejected", "needs_human_review"] as const;
+export type ScoreCategory = (typeof SCORE_CATEGORIES)[number];
+
+export interface ScoredRun {
+  schema_version: number;
+  evidence_run_id: string;     // FK to EvidenceRecord.run_id
+  evidence_task_id: string;    // FK to EvidenceRecord.task_id
+  category: ScoreCategory;
+  reasons: string[];           // human-readable, e.g. ["cargo_green=true", "anchor_grounding<0.7"]
+  scored_at: string;           // ISO 8601
+  scorer_version: string;      // e.g. "v1.0.0" — bumped on scorer code change
+  // Sub-scores that the scorer collapsed into the category. Persisted
+  // so a downstream UI can show "why" without re-running the scorer.
+  sub_scores?: {
+    cargo_green?: boolean;
+    anchor_grounding?: number;
+    schema_valid?: boolean;
+    pathway_replay_succeeded?: boolean;
+    observer_verdict?: "accept" | "reject" | "cycle";
+    [key: string]: unknown;
+  };
+  provenance: {
+    source_file: string;
+    line_offset?: number;
+    sig_hash: string;
+    recorded_at: string;
+  };
+}
+
+export function validateScoredRun(input: unknown): ValidationResult<ScoredRun> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object"] };
+  }
+  const r = input as Record<string, unknown>;
+  let ok = true;
+  if (r.schema_version !== SCORED_RUN_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${SCORED_RUN_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.evidence_run_id, "evidence_run_id", errors) && ok;
+  ok = requireString(r.evidence_task_id, "evidence_task_id", errors) && ok;
+  ok = requireIsoTimestamp(r.scored_at, "scored_at", errors) && ok;
+  ok = requireString(r.scorer_version, "scorer_version", errors) && ok;
+  ok = requireStringArray(r.reasons, "reasons", errors) && ok;
+  if (Array.isArray(r.reasons) && r.reasons.length === 0) {
+    errors.push("reasons: must be non-empty (every score must have at least one reason)");
+    ok = false;
+  }
+  if (!SCORE_CATEGORIES.includes(r.category as ScoreCategory)) {
+    errors.push(`category: must be one of ${SCORE_CATEGORIES.join("|")}, got ${JSON.stringify(r.category)}`);
+    ok = false;
+  }
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  if (r.sub_scores !== undefined) {
+    if (typeof r.sub_scores !== "object" || r.sub_scores === null) {
+      errors.push("sub_scores: expected object when present");
+      ok = false;
+    } else {
+      const ss = r.sub_scores as Record<string, unknown>;
+      if (ss.anchor_grounding !== undefined) {
+        if (!requireNumber(ss.anchor_grounding, "sub_scores.anchor_grounding", errors)) ok = false;
+        else if ((ss.anchor_grounding as number) < 0 || (ss.anchor_grounding as number) > 1) {
+          errors.push("sub_scores.anchor_grounding: must be in [0, 1]");
+          ok = false;
+        }
+      }
+      if (ss.observer_verdict !== undefined && !["accept", "reject", "cycle"].includes(ss.observer_verdict as string)) {
+        errors.push("sub_scores.observer_verdict: must be accept|reject|cycle");
+        ok = false;
+      }
+    }
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as ScoredRun };
+}

auditor/schemas/distillation/scratchpad_summary.ts

@@ -0,0 +1,65 @@
+// ScratchpadSummary — structured normalization of a tree-split or
+// long-running scratchpad. Distinct from EvidenceRecord because a
+// scratchpad accumulates across many calls; this schema captures the
+// state at a checkpoint moment.
+import {
+  ValidationResult, requireString, requireIsoTimestamp, requireProvenance, requireStringArray,
+} from "./types";
+
+export const SCRATCHPAD_SCHEMA_VERSION = 1;
+
+export interface ScratchpadSummary {
+  schema_version: number;
+  run_id: string;
+  current_objective: string;
+  completed_steps: string[];
+  failed_steps: string[];
+  pending_steps: string[];
+  important_paths: string[];        // file paths the scratchpad references
+  decisions: string[];              // architectural/scope decisions made
+  unresolved_questions: string[];
+  validation_status: "pass" | "fail" | "partial" | "pending";
+  next_command?: string;            // recommendation for next action
+  source_scratchpad_hash: string;   // sha256 of the full source scratchpad text — diff detection
+  summarized_at: string;            // ISO 8601
+  provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
+}
+
+const STATUS = ["pass", "fail", "partial", "pending"];
+
+export function validateScratchpadSummary(input: unknown): ValidationResult<ScratchpadSummary> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== SCRATCHPAD_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${SCRATCHPAD_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.run_id, "run_id", errors) && ok;
+  ok = requireString(r.current_objective, "current_objective", errors) && ok;
+  ok = requireIsoTimestamp(r.summarized_at, "summarized_at", errors) && ok;
+  if (typeof r.source_scratchpad_hash !== "string" || !/^[0-9a-f]{64}$/.test(r.source_scratchpad_hash as string)) {
+    errors.push("source_scratchpad_hash: must be hex sha256");
+    ok = false;
+  }
+  ok = requireStringArray(r.completed_steps, "completed_steps", errors) && ok;
+  ok = requireStringArray(r.failed_steps, "failed_steps", errors) && ok;
+  ok = requireStringArray(r.pending_steps, "pending_steps", errors) && ok;
+  ok = requireStringArray(r.important_paths, "important_paths", errors) && ok;
+  ok = requireStringArray(r.decisions, "decisions", errors) && ok;
+  ok = requireStringArray(r.unresolved_questions, "unresolved_questions", errors) && ok;
+  if (!STATUS.includes(r.validation_status as string)) {
+    errors.push(`validation_status: must be one of ${STATUS.join("|")}`);
+    ok = false;
+  }
+  if (r.next_command !== undefined && typeof r.next_command !== "string") {
+    errors.push("next_command: expected string when present");
+    ok = false;
+  }
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as ScratchpadSummary };
+}

auditor/schemas/distillation/sft_sample.ts

@@ -0,0 +1,69 @@
+// SftSample — entry in exports/sft/instruction_response.jsonl. Spec
+// non-negotiable: ONLY accepted runs, never partial/rejected/needs_human.
+// Validator enforces that invariant — exporters can't bypass.
+import {
+  ValidationResult, requireString, requireIsoTimestamp, requireProvenance, requireNumber,
+} from "./types";
+
+export const SFT_SAMPLE_SCHEMA_VERSION = 1;
+
+// SFT default: only `accepted` ships. With --include-partial CLI flag,
+// `partially_accepted` becomes legal. `rejected` and `needs_human_review`
+// NEVER ship to SFT — that's the contamination firewall.
+export const SFT_QUALITY_SCORES = ["accepted", "partially_accepted"] as const;
+export type SftQualityScore = (typeof SFT_QUALITY_SCORES)[number];
+
+export interface SftSample {
+  schema_version: number;
+  id: string;
+  instruction: string;             // the prompt / user message
+  context: string;                 // retrieved context that was visible (empty string allowed; null/undefined not)
+  response: string;                // the model output that was accepted
+  source_run_id: string;
+  quality_score: SftQualityScore;
+  created_at: string;
+  provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
+}
+
+export function validateSftSample(input: unknown): ValidationResult<SftSample> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== SFT_SAMPLE_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${SFT_SAMPLE_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.id, "id", errors) && ok;
+  ok = requireString(r.instruction, "instruction", errors) && ok;
+  ok = requireString(r.response, "response", errors) && ok;
+  ok = requireString(r.source_run_id, "source_run_id", errors) && ok;
+  ok = requireIsoTimestamp(r.created_at, "created_at", errors) && ok;
+  ok = requireProvenance(r.provenance, "provenance", errors) && ok;
+
+  // Empty pair guard.
+  if (typeof r.instruction === "string" && (r.instruction as string).trim().length === 0) {
+    errors.push("instruction: must be non-whitespace (no empty pairs)");
+    ok = false;
+  }
+  if (typeof r.response === "string" && (r.response as string).trim().length === 0) {
+    errors.push("response: must be non-whitespace (no empty pairs)");
+    ok = false;
+  }
+  // Context is required-string but empty is allowed (some SFT samples
+  // are pure instruction→response with no retrieval context).
+  if (typeof r.context !== "string") {
+    errors.push("context: expected string (use empty string for no-context samples)");
+    ok = false;
+  }
+  // The non-negotiable: SFT samples MUST have quality_score in
+  // SFT_QUALITY_SCORES. Anything else is a leak.
+  if (!SFT_QUALITY_SCORES.includes(r.quality_score as SftQualityScore)) {
+    errors.push(`quality_score: must be one of ${SFT_QUALITY_SCORES.join("|")} (no rejected/needs_human leak into SFT — spec non-negotiable). Got ${JSON.stringify(r.quality_score)}`);
+    ok = false;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as SftSample };
+}

auditor/schemas/distillation/stage_receipt.ts

@@ -0,0 +1,190 @@
+// stage_receipt.ts — forensic-grade per-stage receipt.
+//
+// Distinct from auditor/schemas/distillation/receipt.ts (Phase 1):
+// - Phase 1 Receipt is per-script invocation, format inherited from
+//   the early auditor wiring
+// - StageReceipt (THIS file) matches the now.md Phase 5 spec exactly
+//   and is the canonical artifact for pipeline observability
+//
+// Every pipeline stage (collect, score, export-rag, export-sft,
+// export-preference, future extract-playbooks/index) emits ONE
+// StageReceipt per run. Receipts are joined by `run_id` (shared
+// across all stages of a single `run-all` invocation) so a future
+// query can aggregate across the whole pipeline.
+
+import {
+  ValidationResult, requireString, requireNumber, requireIsoTimestamp, requireSha256,
+  requireStringArray,
+} from "./types";
+
+export const STAGE_RECEIPT_SCHEMA_VERSION = 1;
+
+export const STAGE_NAMES = [
+  "collect",          // build_evidence_index — materialize source jsonls → EvidenceRecord
+  "score",            // score_runs — EvidenceRecord → ScoredRun
+  "export-rag",       // exports/rag/playbooks.jsonl
+  "export-sft",       // exports/sft/instruction_response.jsonl
+  "export-preference",// exports/preference/chosen_rejected.jsonl
+  // Reserved for future stages — accept them in the schema so a stage
+  // can be added without bumping schema_version.
+  "extract-playbooks",
+  "index",
+] as const;
+export type StageName = (typeof STAGE_NAMES)[number];
+
+export interface StageFileRef {
+  path: string;          // relative to repo root
+  sha256: string;        // 64-char hex
+  bytes?: number;
+  record_count?: number; // line count for jsonl, when meaningful
+}
+
+export interface StageIO {
+  files: StageFileRef[];
+  record_count: number;
+  hash: string;          // 64-char hex — aggregate over all file hashes (sorted)
+}
+
+export interface StageStats {
+  accepted: number;       // rows that ended up in the stage's output
+  rejected: number;       // explicit category=rejected (Score), invalid pairs (Preference), etc.
+  quarantined: number;    // routed to exports/quarantine/* with structured reason
+  skipped: number;        // parse failures, schema violations at write time
+}
+
+export interface StageValidation {
+  passed: boolean;        // explicit boolean — never inferred (spec non-negotiable)
+  errors: string[];
+  warnings: string[];
+}
+
+export interface StageReceipt {
+  schema_version: number;
+  run_id: string;         // shared across all stages of one pipeline run
+  stage: StageName;
+  timestamp: string;      // ISO 8601 — stage start
+  git_commit: string;     // 40-char hex
+  inputs: StageIO;
+  outputs: StageIO;
+  stats: StageStats;
+  validation: StageValidation;
+  duration_ms: number;
+}
+
+function validateStageIO(v: unknown, field: string, errors: string[]): boolean {
+  if (typeof v !== "object" || v === null) {
+    errors.push(`${field}: expected object`);
+    return false;
+  }
+  const io = v as Record<string, unknown>;
+  let ok = true;
+  if (!Array.isArray(io.files)) {
+    errors.push(`${field}.files: expected array`);
+    ok = false;
+  } else {
+    for (let i = 0; i < io.files.length; i++) {
+      const f = io.files[i] as Record<string, unknown>;
+      if (typeof f !== "object" || f === null) {
+        errors.push(`${field}.files[${i}]: expected object`);
+        ok = false;
+        continue;
+      }
+      ok = requireString(f.path, `${field}.files[${i}].path`, errors) && ok;
+      ok = requireSha256(f.sha256, `${field}.files[${i}].sha256`, errors) && ok;
+      if (f.bytes !== undefined && typeof f.bytes !== "number") {
+        errors.push(`${field}.files[${i}].bytes: expected number when present`);
+        ok = false;
+      }
+      if (f.record_count !== undefined && typeof f.record_count !== "number") {
+        errors.push(`${field}.files[${i}].record_count: expected number when present`);
+        ok = false;
+      }
+    }
+  }
+  ok = requireNumber(io.record_count, `${field}.record_count`, errors) && ok;
+  ok = requireSha256(io.hash, `${field}.hash`, errors) && ok;
+  return ok;
+}
+
+export function validateStageReceipt(input: unknown): ValidationResult<StageReceipt> {
+  const errors: string[] = [];
+  if (typeof input !== "object" || input === null) {
+    return { valid: false, errors: ["expected object"] };
+  }
+  const r = input as Record<string, unknown>;
+  let ok = true;
+
+  if (r.schema_version !== STAGE_RECEIPT_SCHEMA_VERSION) {
+    errors.push(`schema_version: expected ${STAGE_RECEIPT_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
+    ok = false;
+  }
+  ok = requireString(r.run_id, "run_id", errors) && ok;
+  if (typeof r.run_id === "string" && r.run_id.length < 8) {
+    errors.push("run_id: too short — expect uuid-like");
+    ok = false;
+  }
+  if (typeof r.stage !== "string" || !STAGE_NAMES.includes(r.stage as StageName)) {
+    errors.push(`stage: must be one of ${STAGE_NAMES.join("|")}`);
+    ok = false;
+  }
+  ok = requireIsoTimestamp(r.timestamp, "timestamp", errors) && ok;
+  if (typeof r.git_commit !== "string" || !/^[0-9a-f]{40}$/.test(r.git_commit as string)) {
+    errors.push("git_commit: must be 40-char hex");
+    ok = false;
+  }
+  if (typeof r.duration_ms !== "number") {
+    errors.push("duration_ms: expected number");
+    ok = false;
+  }
+  if (typeof r.inputs !== "object" || r.inputs === null) {
+    errors.push("inputs: expected object");
+    ok = false;
+  } else {
+    ok = validateStageIO(r.inputs, "inputs", errors) && ok;
+  }
+  if (typeof r.outputs !== "object" || r.outputs === null) {
+    errors.push("outputs: expected object");
+    ok = false;
+  } else {
+    ok = validateStageIO(r.outputs, "outputs", errors) && ok;
+  }
+  if (typeof r.stats !== "object" || r.stats === null) {
+    errors.push("stats: expected object");
+    ok = false;
+  } else {
+    const s = r.stats as Record<string, unknown>;
+    for (const k of ["accepted", "rejected", "quarantined", "skipped"]) {
+      if (typeof s[k] !== "number") { errors.push(`stats.${k}: expected number`); ok = false; }
+    }
+  }
+  if (typeof r.validation !== "object" || r.validation === null) {
+    errors.push("validation: expected object");
+    ok = false;
+  } else {
+    const v = r.validation as Record<string, unknown>;
+    if (typeof v.passed !== "boolean") {
+      errors.push("validation.passed: must be boolean (explicit, never inferred)");
+      ok = false;
+    }
+    if (!Array.isArray(v.errors)) { errors.push("validation.errors: expected array"); ok = false; }
+    if (!Array.isArray(v.warnings)) { errors.push("validation.warnings: expected array"); ok = false; }
+    if (Array.isArray(v.errors)) ok = requireStringArray(v.errors, "validation.errors", errors) && ok;
+    if (Array.isArray(v.warnings)) ok = requireStringArray(v.warnings, "validation.warnings", errors) && ok;
+  }
+
+  if (!ok) return { valid: false, errors };
+  return { valid: true, value: r as unknown as StageReceipt };
+}
+
+// Compute the canonical aggregate hash over a list of file refs.
+// Sorted by path so order-of-iteration doesn't drift the hash.
+// Each entry contributes "<path>|<sha256>|<record_count>" so two
+// files with identical content but different paths produce distinct
+// digests (real difference = real hash difference).
+export async function aggregateIoHash(files: StageFileRef[]): Promise<string> {
+  const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path));
+  const parts = sorted.map(f => `${f.path}|${f.sha256}|${f.record_count ?? 0}`);
+  const h = new Bun.CryptoHasher("sha256");
+  h.update(parts.join("\n"));
+  return h.digest("hex");
+}

auditor/schemas/distillation/types.ts

@@ -0,0 +1,141 @@
+// Shared types for distillation schemas. Hand-rolled validators (no Zod
+// dependency) — bun:test runs them; runtime cost is one tiny function
+// per record. Pattern: each schema exports `validate(x): ValidationResult`
+// returning `{valid: true, value}` or `{valid: false, errors}`.
+//
+// Why hand-rolled: the auditor + scrum + observer pipelines emit JSONL
+// rows in shapes that already work; we want to ENFORCE those shapes
+// without adding a 100KB dependency or rewriting producers. The
+// validators codify what we already produce.
+//
+// Naming: schemas live as nouns (`EvidenceRecord`), validators as
+// `validate<Noun>`. Each schema file exports both the type and the
+// validator.
+
+export interface Provenance {
+  // Path to the JSONL or other source where this row came from. Always
+  // relative to /home/profit/lakehouse so receipts are reproducible
+  // across deploys with the same repo layout.
+  source_file: string;
+
+  // Optional byte offset / line number into the source file. Lets a
+  // future "open the source row" UI jump directly to the line. Some
+  // sources (single-row JSON files like _playbook_lessons/*.json) don't
+  // need this.
+  line_offset?: number;
+
+  // SHA-256 of the canonical JSON of the source row (sorted keys, no
+  // whitespace). This is the dedup key — running distillation twice on
+  // the same source produces identical sig_hash, so duplicates are
+  // detectable without full row comparison.
+  sig_hash: string;
+
+  // ISO 8601 of when this provenance link was recorded — usually the
+  // moment the unified Evidence Index ran. Distinct from the source
+  // row's own timestamp, which lives on the EvidenceRecord itself.
+  recorded_at: string;
+}
+
+// Returned by every schema validator. The shape is `{valid: true, value}`
+// for success (so callers can use `value` with the right type narrowed)
+// or `{valid: false, errors}` for failure (so callers can surface
+// every error at once, not just the first).
+export type ValidationResult<T> =
+  | { valid: true; value: T }
+  | { valid: false; errors: string[] };
+
+// Standard helpers used by every schema. Centralized so naming +
+// error message format stay consistent across schemas.
+
+export function requireString(v: unknown, field: string, errors: string[]): v is string {
+  if (typeof v !== "string") {
+    errors.push(`${field}: expected string, got ${typeof v}`);
+    return false;
+  }
+  if (v.length === 0) {
+    errors.push(`${field}: must be non-empty`);
+    return false;
+  }
+  return true;
+}
+
+export function requireNumber(v: unknown, field: string, errors: string[]): v is number {
+  if (typeof v !== "number" || !Number.isFinite(v)) {
+    errors.push(`${field}: expected finite number, got ${typeof v}`);
+    return false;
+  }
+  return true;
+}
+
+export function requireIsoTimestamp(v: unknown, field: string, errors: string[]): v is string {
+  if (!requireString(v, field, errors)) return false;
+  // Permissive ISO 8601: YYYY-MM-DDTHH:MM:SS(.fraction)?(Z|±HH:MM)?
+  const re = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?$/;
+  if (!re.test(v as string)) {
+    errors.push(`${field}: not a valid ISO 8601 timestamp: ${(v as string).slice(0, 60)}`);
+    return false;
+  }
+  return true;
+}
+
+export function requireSha256(v: unknown, field: string, errors: string[]): v is string {
+  if (!requireString(v, field, errors)) return false;
+  if (!/^[0-9a-f]{64}$/.test(v as string)) {
+    errors.push(`${field}: not a valid hex sha256: ${(v as string).slice(0, 80)}`);
+    return false;
+  }
+  return true;
+}
+
+export function requireProvenance(v: unknown, field: string, errors: string[]): v is Provenance {
+  if (typeof v !== "object" || v === null) {
+    errors.push(`${field}: expected object, got ${v === null ? "null" : typeof v}`);
+    return false;
+  }
+  const p = v as Record<string, unknown>;
+  let ok = true;
+  ok = requireString(p.source_file, `${field}.source_file`, errors) && ok;
+  ok = requireSha256(p.sig_hash, `${field}.sig_hash`, errors) && ok;
+  ok = requireIsoTimestamp(p.recorded_at, `${field}.recorded_at`, errors) && ok;
+  if (p.line_offset !== undefined && typeof p.line_offset !== "number") {
+    errors.push(`${field}.line_offset: expected number when present`);
+    ok = false;
+  }
+  return ok;
+}
+
+export function requireStringArray(v: unknown, field: string, errors: string[]): v is string[] {
+  if (!Array.isArray(v)) {
+    errors.push(`${field}: expected array, got ${typeof v}`);
+    return false;
+  }
+  for (let i = 0; i < v.length; i++) {
+    if (typeof v[i] !== "string") {
+      errors.push(`${field}[${i}]: expected string, got ${typeof v[i]}`);
+      return false;
+    }
+  }
+  return true;
+}
+
+// Compute the canonical sha256 used for sig_hash. Sorts keys so the
+// hash is stable regardless of producer's serialization order. Uses
+// Bun.CryptoHasher (sync, fast) rather than node:crypto — matches the
+// rest of the auditor.
+export async function canonicalSha256(obj: unknown): Promise<string> {
+  const ordered = orderKeys(obj);
+  const json = JSON.stringify(ordered);
+  const hasher = new Bun.CryptoHasher("sha256");
+  hasher.update(json);
+  return hasher.digest("hex");
+}
+
+function orderKeys(v: unknown): unknown {
+  if (v === null || typeof v !== "object") return v;
+  if (Array.isArray(v)) return v.map(orderKeys);
+  const out: Record<string, unknown> = {};
+  for (const k of Object.keys(v as object).sort()) {
+    out[k] = orderKeys((v as Record<string, unknown>)[k]);
+  }
+  return out;
+}

auditor/types.ts

@@ -2,7 +2,7 @@
 // if something can't be verified from a check, it goes into `evidence`
 // so the verdict is inspectable, not a black box.
 
-export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
+export type CheckKind = "static" | "dynamic" | "inference" | "kb_query" | "kimi_architect";
 
 export type Severity = "info" | "warn" | "block";
 

bot/propose.ts

@@ -13,10 +13,17 @@ import { readFile } from "node:fs/promises";
 import { createHash } from "node:crypto";
 import type { Gap, Proposal } from "./types.ts";
 
-const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
+// Phase 44 migration (2026-04-27): bot/propose.ts now flows through
+// the gateway's /v1/chat instead of hitting the sidecar's /generate
+// directly. /v1/usage tracks the call, Langfuse traces it, observer
+// sees it. Gateway owns the routing.
+//
+// 2026-04-28: gpt-oss:120b → deepseek-v3.2 via Ollama Pro. Newer
+// DeepSeek revision, faster, still on the same OLLAMA_CLOUD_KEY.
+const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
 const REPO_ROOT = "/home/profit/lakehouse";
 const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
-const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
+const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "deepseek-v3.2";
 const MAX_TOKENS = 6000;
 
 export async function findGaps(): Promise<Gap[]> {
@@ -72,13 +79,16 @@ export async function generateProposal(gap: Gap, historySummary: string = ""): P
   sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
   const userPrompt = sections.join("\n");
 
-  const r = await fetch(`${SIDECAR_URL}/generate`, {
+  const r = await fetch(`${GATEWAY_URL}/v1/chat`, {
     method: "POST",
     headers: { "content-type": "application/json" },
     body: JSON.stringify({
       model: CLOUD_MODEL,
-      system: SYSTEM_PROMPT,
-      prompt: userPrompt,
+      provider: "ollama_cloud",
+      messages: [
+        { role: "system", content: SYSTEM_PROMPT },
+        { role: "user", content: userPrompt },
+      ],
       temperature: 0.2,
       max_tokens: MAX_TOKENS,
       think: false,
@@ -86,10 +96,10 @@ export async function generateProposal(gap: Gap, historySummary: string = ""): P
     signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
   });
   if (!r.ok) {
-    throw new Error(`sidecar ${r.status}: ${await r.text()}`);
+    throw new Error(`gateway /v1/chat ${r.status}: ${await r.text()}`);
   }
   const j = await r.json() as any;
-  const raw: string = j.text ?? j.response ?? "";
+  const raw: string = j?.choices?.[0]?.message?.content ?? "";
   const usage = j.usage ?? {};
   const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
 

config/modes.toml

@@ -0,0 +1,89 @@
+# Mode router config — task_class → mode mapping
+#
+# `preferred_mode` is the first choice for a task class; `fallback_modes`
+# get tried in order if the preferred one isn't available (LLM Team can
+# return Unknown mode for some, OR the matrix has stronger signal for a
+# fallback). `default_model` seeds the mode runner's model field if the
+# caller doesn't override.
+#
+# Modes are dispatched against LLM Team UI (localhost:5000/api/run) for
+# now; future Rust-native runners will short-circuit before the proxy.
+# See crates/gateway/src/v1/mode.rs for the dispatch path.
+
+[[task_class]]
+name = "scrum_review"
+# 2026-04-26 pass5 variance test (5 reps × 4 conditions, grok-4.1-fast,
+# pathway_memory.rs): composed corpus LOST 5/5 vs isolation (Δ −1.8
+# grounded findings, p=0.031). See docs/MODE_RUNNER_TUNING_PLAN.md.
+# Default is now isolation — bug fingerprints + adversarial framing +
+# file content carries strong models without matrix noise. The
+# `codereview_lakehouse` matrix path remains available via force_mode
+# (auto-downgrades to isolation on strong models — see the
+# is_strong_model gate in crates/gateway/src/v1/mode.rs).
+preferred_mode = "codereview_isolation"
+fallback_modes = ["codereview_lakehouse", "codereview", "consensus", "ladder"]
+default_model = "qwen3-coder:480b"
+# Corpora kept defined so experimental modes (codereview_matrix_only,
+# pass2/pass5 sweeps) and weak-model rescue rungs can still pull them.
+# scrum_findings_v1 is built but EXCLUDED — bake-off showed 24% OOB
+# line citations from cross-file drift, only safe with same-file gating.
+matrix_corpus = ["lakehouse_arch_v1", "lakehouse_symbols_v1"]
+
+[[task_class]]
+name = "contract_analysis"
+preferred_mode = "deep_analysis"
+fallback_modes = ["research", "extract"]
+default_model = "kimi-k2:1t"
+matrix_corpus = "chicago_permits_v1"
+
+[[task_class]]
+name = "staffing_inference"
+# Staffing-domain native enrichment runner — Pass 4 (2026-04-26).
+# Same composer architecture as codereview_lakehouse but with staffing
+# framing + workers corpus.
+preferred_mode = "staffing_inference_lakehouse"
+fallback_modes = ["ladder", "consensus", "pipeline"]
+# 2026-05-03: REVERTED to local. PRD line 70 — everything runs locally,
+# no cloud APIs on the customer hot path. Cloud models stay available
+# in providers.toml for explicit dev-tool opt-in (scrum, auditor).
+default_model = "qwen3.5:latest"
+matrix_corpus = "workers_500k_v8"
+
+[[task_class]]
+name = "fact_extract"
+preferred_mode = "extract"
+fallback_modes = ["distill"]
+default_model = "qwen2.5"
+matrix_corpus = "kb_team_runs_v1"
+
+[[task_class]]
+name = "doc_drift_check"
+preferred_mode = "drift"
+fallback_modes = ["validator"]
+# 2026-05-03: REVERTED to local per PRD line 70.
+default_model = "qwen3.5:latest"
+matrix_corpus = "distilled_factual_v20260423095819"
+
+[[task_class]]
+name = "pr_audit"
+# Auditor's claim-vs-diff verification mode (2026-04-26 rebuild).
+# Replaces the auditor's hand-rolled inference check with the mode-runner
+# composer: pathway memory (PR-level patterns) + lakehouse_answers_v1
+# corpus (prior accepted reviews + observer escalations) + adversarial
+# JSON-shaped framing. Default model is paid Ollama Cloud kimi-k2:1t for
+# strong claim-grounding; tie-breaker via auditor-side env override.
+preferred_mode = "pr_audit"
+fallback_modes = ["consensus", "ladder"]
+# kimi-k2:1t broken upstream 2026-04-27 (Ollama Cloud 500 ISE, multi-hour
+# sustained outage verified by repeated probes). deepseek-v3.1:671b is
+# the drop-in substitute — proven working end-to-end through pr_audit
+# during Phase 5 distillation acceptance testing.
+default_model = "deepseek-v3.1:671b"
+matrix_corpus = "lakehouse_answers_v1"
+
+# Fallback when task_class isn't in the table — useful for ad-hoc calls
+# during development that don't yet have a mapped mode.
+[default]
+preferred_mode = "pipeline"
+fallback_modes = ["consensus", "ladder"]
+default_model = "qwen3.5:latest"

config/providers.toml

@@ -15,22 +15,29 @@
 
 [[provider]]
 name = "ollama"
-base_url = "http://localhost:3200"
+base_url = "http://localhost:11434"
 auth = "none"
 default_model = "qwen3.5:latest"
-# Hot-path local inference. No bearer needed — Python sidecar on
-# localhost handles the Ollama API. Model names are bare
-# (e.g. "qwen3.5:latest", not "ollama/qwen3.5:latest").
+# Hot-path local inference. No bearer needed — direct to Ollama as of
+# 2026-05-02 (Python sidecar's pass-through wrapper retired). Model
+# names are bare (e.g. "qwen3.5:latest", not "ollama/qwen3.5:latest").
 
 [[provider]]
 name = "ollama_cloud"
 base_url = "https://ollama.com"
 auth = "bearer"
 auth_env = "OLLAMA_CLOUD_KEY"
-default_model = "gpt-oss:120b"
-# Cloud-tier Ollama. Key resolved from OLLAMA_CLOUD_KEY env at gateway
-# boot. Model-prefix routing: "cloud/<model>" auto-routes here
-# (see gateway::v1::resolve_provider).
+default_model = "deepseek-v3.2"
+# Cloud-tier Ollama (Pro plan as of 2026-04-28). Key resolved from
+# OLLAMA_CLOUD_KEY at gateway boot; Pro tier upgraded the account so
+# rate limits + model access widen without a key change. Model-prefix
+# routing: "cloud/<model>" auto-routes here. 39-model fleet now
+# includes deepseek-v3.2, deepseek-v4-{flash,pro}, gemini-3-flash-
+# preview, glm-{5,5.1}, kimi-k2.6, qwen3-coder-next.
+# 2026-04-28: default upgraded gpt-oss:120b → deepseek-v3.2 (newest
+# DeepSeek revision). NOTE: kimi-k2:1t is upstream-broken (HTTP 500
+# on Ollama Pro probe 2026-04-28) — do not route to it. Use kimi-k2.6
+# instead, which is what staffing_inference points at.
 
 [[provider]]
 name = "openrouter"
@@ -38,13 +45,50 @@ base_url = "https://openrouter.ai/api/v1"
 auth = "bearer"
 auth_env = "OPENROUTER_API_KEY"
 auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"]
-default_model = "openai/gpt-oss-120b:free"
+default_model = "x-ai/grok-4.1-fast"
 # Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax,
 # Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs
 # resolve_openrouter_key() — env first, then fallback files.
 # Model-prefix routing: "openrouter/<vendor>/<model>" auto-routes here,
 # prefix stripped before upstream call.
 
+[[provider]]
+name = "opencode"
+base_url = "https://opencode.ai/zen/v1"
+# Unified endpoint — covers BOTH Zen (pay-per-token Anthropic/OpenAI/
+# Gemini frontier) AND Go (flat-sub Kimi/GLM/DeepSeek/Qwen/Minimax).
+# Upstream bills per-model: Zen models hit Zen balance, Go models hit
+# Go subscription cap. /zen/go/v1 is the Go-only sub-path (rejects
+# Zen models), kept for reference but not used by this provider.
+auth = "bearer"
+auth_env = "OPENCODE_API_KEY"
+default_model = "claude-opus-4-7"
+# OpenCode (Zen + GO unified endpoint). One sk-* key reaches Claude
+# Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+# Qwen, plus 4 free-tier models. OpenAI-compatible Chat Completions
+# at /v1/chat/completions. Model-prefix routing: "opencode/<name>"
+# auto-routes here, prefix stripped before upstream call.
+# Key file: /etc/lakehouse/opencode.env (loaded via systemd EnvironmentFile).
+# Model catalog: curl -H "Authorization: Bearer ..." https://opencode.ai/zen/v1/models
+# Note: /zen/go/v1 is the GO-only sub-path (Kimi/GLM/DeepSeek tier);
+# /zen/v1 covers everything including Anthropic (which /zen/go/v1 rejects).
+
+[[provider]]
+name = "kimi"
+base_url = "https://api.kimi.com/coding/v1"
+auth = "bearer"
+auth_env = "KIMI_API_KEY"
+default_model = "kimi-for-coding"
+# Direct Kimi For Coding provider. `api.kimi.com` is a SEPARATE account
+# system from `api.moonshot.ai` and `api.moonshot.cn` — keys are NOT
+# interchangeable. Used as a fallback when Ollama Cloud's kimi-k2.6 is
+# unavailable and OpenRouter's `moonshotai/kimi-k2.6` is rate-limited.
+# (Was `kimi-k2:1t` here pre-2026-05-03 — that model is upstream-broken
+# and removed from operator guidance.)
+# Model id: `kimi-for-coding` (kimi-k2.6 underneath).
+# Key file: /etc/lakehouse/kimi.env (loaded via systemd EnvironmentFile).
+# Model-prefix routing: "kimi/<model>" auto-routes here, prefix stripped.
+
 # Planned (Phase 40 long-horizon — adapters not yet shipped):
 #
 # [[provider]]

crates/aibridge/Cargo.toml

@@ -12,3 +12,4 @@ serde_json = { workspace = true }
 tracing = { workspace = true }
 reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
 async-trait = "0.1"
+lru = "0.12"

crates/aibridge/src/client.rs

@@ -1,12 +1,74 @@
+use lru::LruCache;
 use reqwest::Client;
 use serde::{Deserialize, Serialize};
+use std::num::NonZeroUsize;
+use std::sync::Mutex;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
 use std::time::Duration;
 
-/// HTTP client for the Python AI sidecar.
+/// HTTP client for Ollama (post-2026-05-02 — sidecar dropped).
+///
+/// `base_url` was historically the Python sidecar at `:3200`, which
+/// pass-through-proxied to Ollama at `:11434`. The sidecar added zero
+/// logic on the hot path (embed.py + generate.py + rerank.py +
+/// admin.py = ~120 LOC of pure Ollama wrappers), so this client now
+/// talks to Ollama directly and the sidecar process can be retired.
+///
+/// What stayed Python: `lab_ui.py` + `pipeline_lab.py` (~888 LOC of
+/// dev-mode Streamlit-shape UIs) — those aren't on the runtime hot
+/// path and continue running for prompt experimentation.
+///
+/// `generate()` has two transport modes:
+/// - When `gateway_url` is None (default), posts directly to Ollama's
+///   `${base_url}/api/generate`.
+/// - When `gateway_url` is `Some(url)`, posts to `${url}/v1/chat`
+///   with `provider="ollama"` so the call appears in `/v1/usage` and
+///   Langfuse traces.
+///
+/// `embed()`, `rerank()`, and admin methods always go direct to
+/// Ollama — no `/v1` equivalent for those surfaces yet.
+///
+/// Phase 44 part 2 (2026-04-27): the gateway URL is wired in by
+/// callers that want observability (vectord modules); it's left
+/// unset by callers that ARE the gateway internals (avoids self-loops
+/// + redundant hops).
+/// Per-text embed cache key. We key on (model, text) so different
+/// model selections produce distinct cache lines — a query embedded
+/// under nomic-embed-text-v2-moe must NOT collide with the same
+/// query under nomic-embed-text v1.
+#[derive(Eq, PartialEq, Hash, Clone)]
+struct EmbedCacheKey {
+    model: String,
+    text: String,
+}
+
+/// Default LRU cache size — 4096 entries × ~6KB per 768-d f64
+/// vector ≈ 24MB. Sized for typical staffing-domain repetition
+/// (coordinator workflows have query repetition rates around 70-90%
+/// per session). Tunable via [aibridge].embed_cache_size in the
+/// config; 0 disables the cache entirely.
+const DEFAULT_EMBED_CACHE_SIZE: usize = 4096;
+
 #[derive(Clone)]
 pub struct AiClient {
     client: Client,
     base_url: String,
+    gateway_url: Option<String>,
+    /// Closes the 63× perf gap with Go side. Mirrors the shape of
+    /// Go's internal/embed/cached.go::CachedProvider — same
+    /// (model, text) → vector caching, same nil-disable semantics.
+    /// None = caching disabled (cache_size=0); Some = bounded LRU.
+    embed_cache: Option<Arc<Mutex<LruCache<EmbedCacheKey, Vec<f64>>>>>,
+    /// Hit / miss counters for /admin observability + load-test
+    /// validation. Atomic so Clone'd AiClients share the same counts.
+    embed_cache_hits: Arc<AtomicU64>,
+    embed_cache_misses: Arc<AtomicU64>,
+    /// Pinned at construction time so the EmbedResponse can carry
+    /// dimension consistently even when every text was a cache hit
+    /// (no fresh upstream call to learn the dim from). Set on first
+    /// successful Ollama embed; checked on every cache hit.
+    cached_dim: Arc<AtomicU64>,
 }
 
 // -- Request/Response types --
@@ -79,68 +141,386 @@ pub struct RerankResponse {
 
 impl AiClient {
     pub fn new(base_url: &str) -> Self {
+        Self::with_embed_cache(base_url, DEFAULT_EMBED_CACHE_SIZE)
+    }
+
+    /// Constructs an AiClient with an explicit embed-cache size.
+    /// Pass 0 to disable the cache entirely (matches Go-side
+    /// CachedProvider's nil-cache semantics).
+    pub fn with_embed_cache(base_url: &str, cache_size: usize) -> Self {
         let client = Client::builder()
             .timeout(Duration::from_secs(120))
             .build()
             .expect("failed to build HTTP client");
+        let embed_cache = if cache_size > 0 {
+            // SAFETY: cache_size > 0 just verified, NonZeroUsize::new
+            // returns Some.
+            let cap = NonZeroUsize::new(cache_size).expect("cache_size > 0");
+            Some(Arc::new(Mutex::new(LruCache::new(cap))))
+        } else {
+            None
+        };
         Self {
             client,
             base_url: base_url.trim_end_matches('/').to_string(),
+            gateway_url: None,
+            embed_cache,
+            embed_cache_hits: Arc::new(AtomicU64::new(0)),
+            embed_cache_misses: Arc::new(AtomicU64::new(0)),
+            cached_dim: Arc::new(AtomicU64::new(0)),
         }
     }
 
+    /// Cache hit/miss/size snapshot. Useful for /admin endpoints +
+    /// load-test validation ("did the cache fire as expected?").
+    pub fn embed_cache_stats(&self) -> (u64, u64, usize) {
+        let hits = self.embed_cache_hits.load(Ordering::Relaxed);
+        let misses = self.embed_cache_misses.load(Ordering::Relaxed);
+        let len = self
+            .embed_cache
+            .as_ref()
+            .map(|c| c.lock().map(|g| g.len()).unwrap_or(0))
+            .unwrap_or(0);
+        (hits, misses, len)
+    }
+
+    /// Same as `new`, but every `generate()` is routed through
+    /// `${gateway_url}/v1/chat` (provider=ollama) for observability.
+    /// Use this for callers OUTSIDE the gateway. Inside the gateway
+    /// itself, prefer `new()` — calling /v1/chat from /v1/chat works
+    /// (no infinite loop, ollama_arm doesn't use AiClient) but adds
+    /// a wasted localhost hop.
+    pub fn new_with_gateway(base_url: &str, gateway_url: &str) -> Self {
+        let mut c = Self::new(base_url);
+        c.gateway_url = Some(gateway_url.trim_end_matches('/').to_string());
+        c
+    }
+
+    /// Reachability + version check. Hits Ollama's `/api/version`,
+    /// returns a sidecar-shaped envelope so callers reading
+    /// `.status` / `.ollama_url` don't break across the
+    /// pre-/post-2026-05-02 cutover.
     pub async fn health(&self) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .get(format!("{}/health", self.base_url))
+            .get(format!("{}/api/version", self.base_url))
             .send()
             .await
-            .map_err(|e| format!("sidecar unreachable: {e}"))?;
-        resp.json().await.map_err(|e| format!("invalid response: {e}"))
+            .map_err(|e| format!("ollama unreachable: {e}"))?;
+        let body: serde_json::Value = resp.json().await
+            .map_err(|e| format!("invalid response: {e}"))?;
+        Ok(serde_json::json!({
+            "status": "ok",
+            "ollama_url": &self.base_url,
+            "ollama_version": body.get("version"),
+        }))
     }
 
+    /// Embed with per-text LRU caching. Mirrors Go-side
+    /// CachedProvider behavior: cache key is (model, text);
+    /// cache-hit texts skip the sidecar; cache-miss texts batch
+    /// into a single sidecar call; results are interleaved in the
+    /// caller's input order.
+    ///
+    /// Closes ~95% of the load-test perf gap vs Go side (loadgen
+    /// 2026-05-01: Rust 128 RPS → with cache ≥ 7000 RPS expected
+    /// for warm-cache workloads). Cold-cache behavior unchanged
+    /// (every text is a miss → single sidecar call, identical to
+    /// pre-cache).
     pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> {
-        let resp = self.client
-            .post(format!("{}/embed", self.base_url))
-            .json(&req)
-            .send()
-            .await
-            .map_err(|e| format!("embed request failed: {e}"))?;
+        let model_key = req.model.clone().unwrap_or_default();
 
-        if !resp.status().is_success() {
-            let text = resp.text().await.unwrap_or_default();
-            return Err(format!("embed error ({}): {text}", text.len()));
+        // Fast path: cache disabled → original behavior.
+        let Some(cache) = self.embed_cache.as_ref() else {
+            return self.embed_uncached(&req).await;
+        };
+        if req.texts.is_empty() {
+            return self.embed_uncached(&req).await;
         }
-        resp.json().await.map_err(|e| format!("embed parse error: {e}"))
+
+        // First pass: check cache for each text. Track which positions
+        // need a sidecar fetch.
+        let mut embeddings: Vec<Option<Vec<f64>>> = vec![None; req.texts.len()];
+        let mut miss_indices: Vec<usize> = Vec::new();
+        let mut miss_texts: Vec<String> = Vec::new();
+        {
+            let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
+            for (i, text) in req.texts.iter().enumerate() {
+                let key = EmbedCacheKey { model: model_key.clone(), text: text.clone() };
+                if let Some(vec) = guard.get(&key) {
+                    embeddings[i] = Some(vec.clone());
+                    self.embed_cache_hits.fetch_add(1, Ordering::Relaxed);
+                } else {
+                    miss_indices.push(i);
+                    miss_texts.push(text.clone());
+                    self.embed_cache_misses.fetch_add(1, Ordering::Relaxed);
+                }
+            }
+        }
+
+        // All hit? Return immediately. Use cached_dim to populate
+        // the response dimension (no sidecar to ask).
+        if miss_indices.is_empty() {
+            let dim = self.cached_dim.load(Ordering::Relaxed) as usize;
+            let dim = if dim == 0 { embeddings[0].as_ref().map(|v| v.len()).unwrap_or(0) } else { dim };
+            return Ok(EmbedResponse {
+                embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
+                model: req.model.unwrap_or_else(|| "nomic-embed-text".to_string()),
+                dimensions: dim,
+            });
+        }
+
+        // Second pass: fetch the misses in one sidecar call.
+        let miss_req = EmbedRequest { texts: miss_texts.clone(), model: req.model.clone() };
+        let resp = self.embed_uncached(&miss_req).await?;
+        if resp.embeddings.len() != miss_texts.len() {
+            return Err(format!(
+                "embed cache: sidecar returned {} embeddings for {} texts",
+                resp.embeddings.len(),
+                miss_texts.len()
+            ));
+        }
+
+        // Pin cached_dim on first successful response.
+        if resp.dimensions > 0 {
+            self.cached_dim.store(resp.dimensions as u64, Ordering::Relaxed);
+        }
+
+        // Insert misses into cache + fill response slots.
+        {
+            let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
+            for (j, idx) in miss_indices.iter().enumerate() {
+                let key = EmbedCacheKey {
+                    model: model_key.clone(),
+                    text: miss_texts[j].clone(),
+                };
+                let vec = resp.embeddings[j].clone();
+                guard.put(key, vec.clone());
+                embeddings[*idx] = Some(vec);
+            }
+        }
+
+        Ok(EmbedResponse {
+            embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
+            model: resp.model,
+            dimensions: resp.dimensions,
+        })
+    }
+
+    /// Direct Ollama call — used internally by embed() for cache-miss
+    /// batches and as the transparent fallback when the cache is
+    /// disabled. Loops per-text against `${base_url}/api/embed`,
+    /// matching the sidecar's pre-2026-05-02 behavior. Ollama 0.4+
+    /// supports batch input but per-text keeps compatibility broader
+    /// + lets cache-miss-only batches share the loop with cold runs.
+    async fn embed_uncached(&self, req: &EmbedRequest) -> Result<EmbedResponse, String> {
+        let model = req.model.clone().unwrap_or_else(|| "nomic-embed-text".to_string());
+        let mut embeddings: Vec<Vec<f64>> = Vec::with_capacity(req.texts.len());
+
+        for text in &req.texts {
+            let resp = self.client
+                .post(format!("{}/api/embed", self.base_url))
+                .json(&serde_json::json!({
+                    "model": &model,
+                    "input": text,
+                }))
+                .send()
+                .await
+                .map_err(|e| format!("embed request failed: {e}"))?;
+
+            if !resp.status().is_success() {
+                let body = resp.text().await.unwrap_or_default();
+                return Err(format!("ollama embed error: {body}"));
+            }
+            // Ollama returns {"embeddings": [[...]], "model": "...", ...}.
+            // The outer `embeddings` is always a list; for a scalar input
+            // we get a single inner vector.
+            let parsed: serde_json::Value = resp.json().await
+                .map_err(|e| format!("embed parse error: {e}"))?;
+            let arr = parsed.get("embeddings")
+                .and_then(|v| v.as_array())
+                .ok_or_else(|| format!("ollama embed: missing 'embeddings' field in {parsed}"))?;
+            if arr.is_empty() {
+                return Err("ollama embed: empty embeddings array".to_string());
+            }
+            let first = arr[0].as_array()
+                .ok_or_else(|| "ollama embed: embeddings[0] not an array".to_string())?;
+            let vec: Vec<f64> = first.iter()
+                .filter_map(|n| n.as_f64())
+                .collect();
+            if vec.is_empty() {
+                return Err("ollama embed: numeric coercion produced empty vector".to_string());
+            }
+            embeddings.push(vec);
+        }
+
+        let dimensions = embeddings.first().map(|v| v.len()).unwrap_or(0);
+        Ok(EmbedResponse {
+            embeddings,
+            model,
+            dimensions,
+        })
     }
 
     pub async fn generate(&self, req: GenerateRequest) -> Result<GenerateResponse, String> {
+        if let Some(gw) = self.gateway_url.as_deref() {
+            return self.generate_via_gateway(gw, req).await;
+        }
+        // Direct Ollama path. Used by gateway internals (so the ollama
+        // provider can call upstream without a self-loop through
+        // /v1/chat) and by any consumer that wants raw transport
+        // without /v1/usage accounting.
+        let model = req.model.clone().unwrap_or_else(|| "qwen3.5:latest".to_string());
+        let mut body = serde_json::json!({
+            "model": &model,
+            "prompt": &req.prompt,
+            "stream": false,
+        });
+        let mut options = serde_json::Map::new();
+        if let Some(t) = req.temperature {
+            options.insert("temperature".to_string(), serde_json::json!(t));
+        }
+        if let Some(mt) = req.max_tokens {
+            options.insert("num_predict".to_string(), serde_json::json!(mt));
+        }
+        if !options.is_empty() {
+            body["options"] = serde_json::Value::Object(options);
+        }
+        if let Some(sys) = &req.system {
+            body["system"] = serde_json::json!(sys);
+        }
+        if let Some(th) = req.think {
+            body["think"] = serde_json::json!(th);
+        }
+
         let resp = self.client
-            .post(format!("{}/generate", self.base_url))
-            .json(&req)
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&body)
             .send()
             .await
             .map_err(|e| format!("generate request failed: {e}"))?;
 
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("generate error: {text}"));
+            return Err(format!("ollama generate error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("generate parse error: {e}"))
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("generate parse error: {e}"))?;
+
+        Ok(GenerateResponse {
+            text: parsed.get("response").and_then(|v| v.as_str()).unwrap_or("").to_string(),
+            model,
+            tokens_evaluated: parsed.get("prompt_eval_count").and_then(|v| v.as_u64()),
+            tokens_generated: parsed.get("eval_count").and_then(|v| v.as_u64()),
+        })
     }
 
-    pub async fn rerank(&self, req: RerankRequest) -> Result<RerankResponse, String> {
+    /// Phase 44 part 2: route generate() through the gateway's
+    /// /v1/chat with provider="ollama" so the call lands in
+    /// /v1/usage + Langfuse. Translates between the sidecar
+    /// GenerateRequest/Response shape and the OpenAI-compat
+    /// chat shape on the wire.
+    async fn generate_via_gateway(&self, gateway_url: &str, req: GenerateRequest) -> Result<GenerateResponse, String> {
+        let mut messages = Vec::with_capacity(2);
+        if let Some(sys) = &req.system {
+            messages.push(serde_json::json!({"role": "system", "content": sys}));
+        }
+        messages.push(serde_json::json!({"role": "user", "content": req.prompt}));
+        let mut body = serde_json::json!({
+            "messages": messages,
+            "provider": "ollama",
+        });
+        if let Some(m) = &req.model { body["model"] = serde_json::json!(m); }
+        if let Some(t) = req.temperature { body["temperature"] = serde_json::json!(t); }
+        if let Some(mt) = req.max_tokens { body["max_tokens"] = serde_json::json!(mt); }
+        if let Some(th) = req.think { body["think"] = serde_json::json!(th); }
+
         let resp = self.client
-            .post(format!("{}/rerank", self.base_url))
-            .json(&req)
+            .post(format!("{}/v1/chat", gateway_url))
+            .json(&body)
             .send()
             .await
-            .map_err(|e| format!("rerank request failed: {e}"))?;
-
+            .map_err(|e| format!("/v1/chat request failed: {e}"))?;
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("rerank error: {text}"));
+            return Err(format!("/v1/chat error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("rerank parse error: {e}"))
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("/v1/chat parse error: {e}"))?;
+
+        let text = parsed
+            .pointer("/choices/0/message/content")
+            .and_then(|v| v.as_str())
+            .unwrap_or("")
+            .to_string();
+        let model = parsed.get("model")
+            .and_then(|v| v.as_str())
+            .unwrap_or_else(|| req.model.as_deref().unwrap_or(""))
+            .to_string();
+        let prompt_tokens = parsed.pointer("/usage/prompt_tokens").and_then(|v| v.as_u64());
+        let completion_tokens = parsed.pointer("/usage/completion_tokens").and_then(|v| v.as_u64());
+
+        Ok(GenerateResponse {
+            text,
+            model,
+            tokens_evaluated: prompt_tokens,
+            tokens_generated: completion_tokens,
+        })
+    }
+
+    /// Cross-encoder reranking via Ollama generate. Asks the model to
+    /// rate each document's relevance to the query 0-10, then sorts
+    /// descending. Mirrors the sidecar's pre-2026-05-02 algorithm
+    /// exactly so callers see the same scores.
+    pub async fn rerank(&self, req: RerankRequest) -> Result<RerankResponse, String> {
+        let model = req.model.clone().unwrap_or_else(|| "qwen3.5:latest".to_string());
+        let mut scored: Vec<ScoredDocument> = Vec::with_capacity(req.documents.len());
+
+        for (i, doc) in req.documents.iter().enumerate() {
+            let prompt = format!(
+                "Rate the relevance of the following document to the query on a scale of 0 to 10. \
+                 Respond with ONLY a number.\n\n\
+                 Query: {}\n\n\
+                 Document: {}\n\n\
+                 Score:",
+                req.query, doc,
+            );
+            let resp = self.client
+                .post(format!("{}/api/generate", self.base_url))
+                .json(&serde_json::json!({
+                    "model": &model,
+                    "prompt": prompt,
+                    "stream": false,
+                    "options": {"temperature": 0.0, "num_predict": 8},
+                }))
+                .send()
+                .await
+                .map_err(|e| format!("rerank request failed: {e}"))?;
+
+            if !resp.status().is_success() {
+                let body = resp.text().await.unwrap_or_default();
+                return Err(format!("ollama rerank error: {body}"));
+            }
+            let parsed: serde_json::Value = resp.json().await
+                .map_err(|e| format!("rerank parse error: {e}"))?;
+            let text = parsed.get("response").and_then(|v| v.as_str()).unwrap_or("").trim();
+            // Parse the leading number; tolerate "7", "7.5", "7 — strong match".
+            let score = text.split_whitespace().next()
+                .and_then(|t| t.parse::<f64>().ok())
+                .unwrap_or(0.0)
+                .clamp(0.0, 10.0);
+
+            scored.push(ScoredDocument {
+                index: i,
+                text: doc.clone(),
+                score,
+            });
+        }
+
+        scored.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
+        if let Some(k) = req.top_k {
+            scored.truncate(k);
+        }
+        Ok(RerankResponse { results: scored, model })
     }
 
     /// Force Ollama to unload the named model from VRAM (keep_alive=0).
@@ -149,40 +529,116 @@ impl AiClient {
     /// profile's model can linger in VRAM next to the new one.
     pub async fn unload_model(&self, model: &str) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .post(format!("{}/admin/unload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&serde_json::json!({
+                "model": model,
+                "prompt": "",
+                "keep_alive": 0,
+                "stream": false,
+            }))
             .send().await
             .map_err(|e| format!("unload request failed: {e}"))?;
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("unload error: {text}"));
+            return Err(format!("ollama unload error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("unload parse error: {e}"))
+        // Ollama returns 200 with the empty-prompt response shape.
+        // Fold into the legacy {"unloaded": "<model>"} envelope so
+        // callers' parsing doesn't break.
+        Ok(serde_json::json!({ "unloaded": model }))
     }
 
     /// Ask Ollama to load the named model into VRAM proactively. Makes
     /// the first real request after profile activation fast (no cold-load
-    /// latency).
+    /// latency). Empty prompts confuse some models, so we send a single
+    /// space + cap num_predict=1 (matches the sidecar's prior behavior).
     pub async fn preload_model(&self, model: &str) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .post(format!("{}/admin/preload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&serde_json::json!({
+                "model": model,
+                "prompt": " ",
+                "keep_alive": "5m",
+                "stream": false,
+                "options": {"num_predict": 1},
+            }))
             .send().await
             .map_err(|e| format!("preload request failed: {e}"))?;
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("preload error: {text}"));
+            return Err(format!("ollama preload error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("preload parse error: {e}"))
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("preload parse error: {e}"))?;
+        Ok(serde_json::json!({
+            "preloaded": model,
+            "load_duration_ns": parsed.get("load_duration"),
+            "total_duration_ns": parsed.get("total_duration"),
+        }))
     }
 
-    /// GPU + loaded-model snapshot from the sidecar. Combines nvidia-smi
-    /// output (if available) with Ollama's /api/ps.
+    /// GPU + loaded-model snapshot. Combines nvidia-smi output (when
+    /// available) with Ollama's /api/ps. Same shape as the prior
+    /// sidecar /admin/vram endpoint so callers don't need updating.
     pub async fn vram_snapshot(&self) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .get(format!("{}/admin/vram", self.base_url))
+            .get(format!("{}/api/ps", self.base_url))
             .send().await
-            .map_err(|e| format!("vram request failed: {e}"))?;
-        resp.json().await.map_err(|e| format!("vram parse error: {e}"))
+            .map_err(|e| format!("ollama ps request failed: {e}"))?;
+        let loaded: Vec<serde_json::Value> = if resp.status().is_success() {
+            let parsed: serde_json::Value = resp.json().await.unwrap_or(serde_json::Value::Null);
+            parsed.get("models")
+                .and_then(|v| v.as_array())
+                .map(|arr| arr.iter().map(|m| serde_json::json!({
+                    "name": m.get("name"),
+                    "size_vram_mib": m.get("size_vram").and_then(|v| v.as_u64()).map(|n| n / (1024 * 1024)),
+                    "expires_at": m.get("expires_at"),
+                })).collect())
+                .unwrap_or_default()
+        } else {
+            Vec::new()
+        };
+
+        let gpu = nvidia_smi_snapshot();
+
+        Ok(serde_json::json!({
+            "gpu": gpu,
+            "ollama_loaded": loaded,
+        }))
     }
 }
+
+/// One-shot nvidia-smi poll. Returns Null if the tool isn't on PATH
+/// or the call fails. Mirrors the sidecar's `_nvidia_smi_snapshot`
+/// shape exactly so callers reading vram_snapshot don't break.
+fn nvidia_smi_snapshot() -> serde_json::Value {
+    use std::process::Command;
+    let out = Command::new("nvidia-smi")
+        .args([
+            "--query-gpu=memory.used,memory.total,utilization.gpu,name",
+            "--format=csv,noheader,nounits",
+        ])
+        .output();
+    let stdout = match out {
+        Ok(o) if o.status.success() => o.stdout,
+        _ => return serde_json::Value::Null,
+    };
+    let line = String::from_utf8_lossy(&stdout);
+    let line = line.trim();
+    if line.is_empty() {
+        return serde_json::Value::Null;
+    }
+    let parts: Vec<&str> = line.split(',').map(|s| s.trim()).collect();
+    if parts.len() < 4 {
+        return serde_json::Value::Null;
+    }
+    let used = parts[0].parse::<u64>().unwrap_or(0);
+    let total = parts[1].parse::<u64>().unwrap_or(0);
+    let util = parts[2].parse::<u64>().unwrap_or(0);
+    serde_json::json!({
+        "name": parts[3],
+        "used_mib": used,
+        "total_mib": total,
+        "utilization_pct": util,
+    })
+}

crates/aibridge/src/continuation.rs

@@ -195,8 +195,11 @@ pub async fn generate_continuable<G: TextGenerator>(
         let req = make_request(opts, prompt.to_string(), current_max);
         let resp = generator.generate_text(req).await?;
         calls += 1;
-        prompt_tokens = prompt_tokens.saturating_add(resp.tokens_evaluated.unwrap_or(0) as u32);
-        completion_tokens = completion_tokens.saturating_add(resp.tokens_generated.unwrap_or(0) as u32);
+        // u32::try_from saturates at u32::MAX instead of silently
+        // truncating bits when tokens_evaluated/_generated comes back
+        // as a u64 > 4 billion. Caught 2026-04-27 by Opus self-audit.
+        prompt_tokens = prompt_tokens.saturating_add(u32::try_from(resp.tokens_evaluated.unwrap_or(0)).unwrap_or(u32::MAX));
+        completion_tokens = completion_tokens.saturating_add(u32::try_from(resp.tokens_generated.unwrap_or(0)).unwrap_or(u32::MAX));
         if !resp.text.trim().is_empty() {
             combined = resp.text;
             break;
@@ -227,8 +230,8 @@ pub async fn generate_continuable<G: TextGenerator>(
         let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
         let resp = generator.generate_text(req).await?;
         calls += 1;
-        prompt_tokens = prompt_tokens.saturating_add(resp.tokens_evaluated.unwrap_or(0) as u32);
-        completion_tokens = completion_tokens.saturating_add(resp.tokens_generated.unwrap_or(0) as u32);
+        prompt_tokens = prompt_tokens.saturating_add(u32::try_from(resp.tokens_evaluated.unwrap_or(0)).unwrap_or(u32::MAX));
+        completion_tokens = completion_tokens.saturating_add(u32::try_from(resp.tokens_generated.unwrap_or(0)).unwrap_or(u32::MAX));
         combined.push_str(&resp.text);
         continuations += 1;
     }

crates/catalogd/Cargo.toml

@@ -16,5 +16,8 @@ chrono = { workspace = true }
 uuid = { workspace = true }
 object_store = { workspace = true }
 arrow = { workspace = true }
+parquet = { workspace = true }
+sha2 = { workspace = true }
+hmac = { workspace = true }
 proto = { path = "../proto" }
 tonic = { workspace = true }

crates/catalogd/src/audit_endpoint.rs

@@ -0,0 +1,505 @@
+//! `/audit/subject/{id}` HTTP endpoint — legal-tier subject audit response.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §4 + §6.
+//!
+//! Responds to: `GET /subject/{candidate_id}?from=ISO&to=ISO`
+//! Auth: requires `X-Lakehouse-Legal-Token` header matching the value in
+//! the legal-token file (see LegalAuditAuth). Constant-time compare.
+//!
+//! Response shape (subject_audit_response.v1):
+//!   - manifest: full SubjectManifest from catalogd
+//!   - audit_log.rows: SubjectAuditRow entries within [from, to]
+//!   - audit_log.chain_verified: bool (HMAC chain verifies end-to-end)
+//!   - completeness_attestation: text the lawyer can quote
+//!
+//! V1 deliberately does NOT include:
+//!   - Ed25519 signature on the response — chain verification IS the
+//!     v1 attestation; signing is future hardening (spec §10 v2 reserve)
+//!   - Live row projection from the underlying datasets — caller can
+//!     fetch via /query/sql against the safe_views named in the manifest;
+//!     keeping projection out of this endpoint avoids depending on queryd
+//!   - External anchor / S3 Object Lock — local HMAC chain is the v1
+//!     integrity surface
+//!
+//! These are documented as future hardening, not silent gaps.
+
+use crate::registry::Registry;
+use crate::subject_audit::SubjectAuditWriter;
+use axum::{
+    extract::{Path, Query, State},
+    http::{HeaderMap, StatusCode},
+    response::IntoResponse,
+    routing::get,
+    Json, Router,
+};
+use serde::{Deserialize, Serialize};
+use shared::types::{SubjectAuditRow, SubjectManifest};
+use std::sync::Arc;
+
+const LEGAL_TOKEN_HEADER: &str = "x-lakehouse-legal-token";
+const RESPONSE_SCHEMA: &str = "subject_audit_response.v1";
+
+/// State for the audit endpoint router. Held outside Registry because
+/// the audit writer + legal token aren't part of catalogd's data model
+/// — they're sidecar concerns specific to this endpoint.
+#[derive(Clone)]
+pub struct AuditEndpointState {
+    pub registry: Registry,
+    pub writer: Arc<SubjectAuditWriter>,
+    /// Legal-tier token. Constant-time compared against the header on
+    /// every request. None = endpoint returns 503 (audit unavailable).
+    pub legal_token: Option<Arc<String>>,
+}
+
+impl AuditEndpointState {
+    /// Construct from existing pieces. `legal_token_path` is read at
+    /// construction and trimmed of trailing whitespace. Missing/empty
+    /// = endpoint will 503 every request with a clear reason.
+    pub async fn new(
+        registry: Registry,
+        writer: Arc<SubjectAuditWriter>,
+        legal_token_path: &std::path::Path,
+    ) -> Self {
+        let legal_token = match tokio::fs::read_to_string(legal_token_path).await {
+            Ok(s) => {
+                let trimmed = s.trim().to_string();
+                if trimmed.is_empty() {
+                    tracing::warn!(
+                        "audit endpoint: legal token file at {} is empty — endpoint will 503",
+                        legal_token_path.display()
+                    );
+                    None
+                } else if trimmed.len() < 32 {
+                    // 2026-05-03 kimi scrum BLOCK at audit_endpoint.rs:59-60:
+                    // 16-byte minimum was too permissive for a legal-tier
+                    // secret. Aligned to the SubjectAuditWriter signing-key
+                    // minimum (32 bytes — HMAC-SHA256 best practice).
+                    tracing::warn!(
+                        "audit endpoint: legal token at {} is {} chars (minimum 32) — endpoint will 503",
+                        legal_token_path.display(), trimmed.len()
+                    );
+                    None
+                } else {
+                    tracing::info!("audit endpoint: legal token loaded from {}", legal_token_path.display());
+                    Some(Arc::new(trimmed))
+                }
+            }
+            Err(e) => {
+                tracing::warn!(
+                    "audit endpoint: legal token unreadable from {} ({e}) — endpoint will 503",
+                    legal_token_path.display()
+                );
+                None
+            }
+        };
+        Self { registry, writer, legal_token }
+    }
+}
+
+pub fn router(state: AuditEndpointState) -> Router {
+    Router::new()
+        .route("/subject/{candidate_id}", get(audit_subject))
+        .route("/health", get(audit_health))
+        .with_state(state)
+}
+
+async fn audit_health(State(state): State<AuditEndpointState>) -> impl IntoResponse {
+    if state.legal_token.is_some() {
+        (StatusCode::OK, "audit endpoint ready").into_response()
+    } else {
+        (StatusCode::SERVICE_UNAVAILABLE, "audit endpoint disabled (legal token missing)").into_response()
+    }
+}
+
+#[derive(Deserialize)]
+struct AuditQuery {
+    /// ISO-8601 start of audit window. None = unbounded back.
+    from: Option<chrono::DateTime<chrono::Utc>>,
+    /// ISO-8601 end of audit window. None = unbounded forward.
+    to: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Serialize)]
+struct AuditWindowEcho {
+    from: Option<chrono::DateTime<chrono::Utc>>,
+    to: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Serialize)]
+struct AuditLogSection {
+    rows_in_window: usize,
+    rows: Vec<SubjectAuditRow>,
+    chain_root: String,
+    chain_verified: bool,
+    chain_rows_total: usize,
+    chain_verification_error: Option<String>,
+}
+
+#[derive(Serialize)]
+struct AuditResponse {
+    schema: &'static str,
+    candidate_id: String,
+    generated_at: chrono::DateTime<chrono::Utc>,
+    generated_by: &'static str,
+    request_window: AuditWindowEcho,
+    manifest: SubjectManifest,
+    audit_log: AuditLogSection,
+    /// Datasets the manifest says contain rows for this subject — caller
+    /// can issue follow-up queries against the named safe_views to get
+    /// the actual row projection. Keeping projection out of this endpoint
+    /// avoids depending on queryd from catalogd.
+    datasets_referenced: Vec<DatasetRefEcho>,
+    safe_views_available: Vec<String>,
+    completeness_attestation: String,
+}
+
+#[derive(Serialize)]
+struct DatasetRefEcho {
+    name: String,
+    key_column: String,
+    key_value: String,
+}
+
+/// Constant-time string comparison to avoid timing leaks on the legal
+/// token check. `subtle` would be the standard but introducing a dep
+/// for one comparison is overkill — manual byte-eq with an unconditional
+/// fold over both byte slices.
+fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
+    if a.len() != b.len() {
+        return false;
+    }
+    let mut diff: u8 = 0;
+    for (x, y) in a.iter().zip(b.iter()) {
+        diff |= x ^ y;
+    }
+    diff == 0
+}
+
+/// Extract + verify the legal-tier token from the request. Returns
+/// Err((status, body)) on any failure (missing header, missing token
+/// configured, mismatch). Does NOT log token values.
+fn require_legal_auth(
+    state: &AuditEndpointState,
+    headers: &HeaderMap,
+) -> Result<(), (StatusCode, &'static str)> {
+    let configured = state.legal_token.as_ref().ok_or((
+        StatusCode::SERVICE_UNAVAILABLE,
+        "audit endpoint disabled — no legal token configured on this gateway",
+    ))?;
+    let provided = headers.get(LEGAL_TOKEN_HEADER).ok_or((
+        StatusCode::UNAUTHORIZED,
+        "missing X-Lakehouse-Legal-Token header",
+    ))?;
+    let provided = provided.to_str().map_err(|_| (
+        StatusCode::UNAUTHORIZED,
+        "X-Lakehouse-Legal-Token contains non-ASCII characters",
+    ))?;
+    if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) {
+        return Err((StatusCode::UNAUTHORIZED, "X-Lakehouse-Legal-Token mismatch"));
+    }
+    Ok(())
+}
+
+async fn audit_subject(
+    State(state): State<AuditEndpointState>,
+    Path(candidate_id): Path<String>,
+    Query(q): Query<AuditQuery>,
+    headers: HeaderMap,
+) -> impl IntoResponse {
+    if let Err((status, msg)) = require_legal_auth(&state, &headers) {
+        return (status, msg).into_response();
+    }
+    if candidate_id.is_empty() {
+        return (StatusCode::BAD_REQUEST, "candidate_id is empty").into_response();
+    }
+
+    // 1. Load the subject manifest.
+    let manifest = match state.registry.get_subject(&candidate_id).await {
+        Some(m) => m,
+        None => return (
+            StatusCode::NOT_FOUND,
+            format!("subject manifest not found for candidate_id={candidate_id}"),
+        ).into_response(),
+    };
+
+    // 2. Read audit rows in window.
+    let rows = match state.writer.read_rows_in_range(&candidate_id, q.from, q.to).await {
+        Ok(r) => r,
+        Err(e) => return (
+            StatusCode::INTERNAL_SERVER_ERROR,
+            format!("audit log read failed: {e}"),
+        ).into_response(),
+    };
+
+    // 3. Verify chain end-to-end (full log, not just the windowed slice;
+    //    chain integrity is global, not per-window). chain_root MUST
+    //    come from the actual log tip — taking it from the windowed
+    //    `rows` would lie when `to` is set or the window catches no
+    //    rows. (2026-05-03 opus scrum WARN audit_endpoint.rs:206.)
+    let (chain_verified, chain_rows_total, chain_root, chain_err) =
+        match state.writer.verify_chain(&candidate_id).await {
+            Ok(n) => {
+                let tip = state.writer.chain_tip(&candidate_id).await
+                    .unwrap_or_else(|| "GENESIS".into());
+                (true, n, tip, None)
+            }
+            Err(e) => (false, 0, String::new(), Some(e)),
+        };
+
+    // 4. Tampering detection: if the manifest claims a non-trivial
+    //    chain root but the log is empty (chain_rows_total == 0), the
+    //    log was deleted between writes. Flag it. (2026-05-03 opus
+    //    scrum WARN subject_audit.rs:347 + audit_endpoint coverage.)
+    let manifest_root = manifest.audit_log_chain_root.clone();
+    let log_was_tampered = chain_verified
+        && chain_rows_total == 0
+        && !manifest_root.is_empty()
+        && manifest_root != "GENESIS";
+    let (chain_verified, chain_err) = if log_was_tampered {
+        (false, Some(format!(
+            "manifest expects chain root '{}' but the audit log has 0 rows; log was deleted or wiped",
+            manifest_root,
+        )))
+    } else {
+        (chain_verified, chain_err)
+    };
+
+    let datasets_referenced = manifest.datasets.iter()
+        .map(|d| DatasetRefEcho {
+            name: d.name.clone(),
+            key_column: d.key_column.clone(),
+            key_value: d.key_value.clone(),
+        })
+        .collect();
+    let safe_views_available = manifest.safe_views.clone();
+
+    let attestation = format!(
+        "All audit rows for candidate_id={} within [{}, {}] are included. \
+         HMAC chain verified end-to-end: {} rows total, integrity={}.",
+        candidate_id,
+        q.from.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
+        q.to.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
+        chain_rows_total,
+        if chain_verified { "verified" } else { "BROKEN" },
+    );
+
+    let resp = AuditResponse {
+        schema: RESPONSE_SCHEMA,
+        candidate_id: candidate_id.clone(),
+        generated_at: chrono::Utc::now(),
+        generated_by: "catalogd",
+        request_window: AuditWindowEcho { from: q.from, to: q.to },
+        manifest,
+        audit_log: AuditLogSection {
+            rows_in_window: rows.len(),
+            rows,
+            chain_root,
+            chain_verified,
+            chain_rows_total,
+            chain_verification_error: chain_err,
+        },
+        datasets_referenced,
+        safe_views_available,
+        completeness_attestation: attestation,
+    };
+
+    // Log the legal-tier read so it's discoverable in operations review.
+    // Token value is NEVER logged. Only the fact of the read.
+    tracing::info!(
+        candidate_id = %candidate_id,
+        rows_in_window = resp.audit_log.rows_in_window,
+        chain_verified = resp.audit_log.chain_verified,
+        "legal-tier audit response served"
+    );
+
+    (StatusCode::OK, Json(resp)).into_response()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::subject_audit::SubjectAuditWriter;
+    use object_store::memory::InMemory;
+    use shared::types::{
+        AuditAccessor, BiometricConsent, BiometricConsentStatus, ConsentStatus,
+        GeneralPiiConsent, SubjectAuditRow, SubjectConsent, SubjectRetention,
+        SubjectStatus, SubjectVertical,
+    };
+
+    fn fixture_manifest(candidate_id: &str) -> SubjectManifest {
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: candidate_id.into(),
+            created_at: chrono::Utc::now(),
+            updated_at: chrono::Utc::now(),
+            status: SubjectStatus::Active,
+            vertical: SubjectVertical::Unknown,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::PendingBackfillReview,
+                    version: String::new(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: BiometricConsentStatus::NeverCollected,
+                    retention_until: None,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
+                policy: "test".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+                biometric_collection: None,
+        }
+    }
+
+    fn fixture_audit_row(candidate_id: &str) -> SubjectAuditRow {
+        SubjectAuditRow {
+            schema: "subject_audit.v1".into(),
+            ts: chrono::Utc::now(),
+            candidate_id: candidate_id.into(),
+            accessor: AuditAccessor {
+                kind: "test".into(),
+                daemon: "test".into(),
+                purpose: "test".into(),
+                trace_id: String::new(),
+            },
+            fields_accessed: vec!["name".into()],
+            result: "success".into(),
+            prev_chain_hash: String::new(),
+            row_hmac: String::new(),
+        }
+    }
+
+    async fn fixture_state_with_token(token: &str) -> AuditEndpointState {
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store, (0u8..32).collect(),
+        ));
+        AuditEndpointState {
+            registry,
+            writer,
+            legal_token: Some(Arc::new(token.into())),
+        }
+    }
+
+    #[test]
+    fn constant_time_eq_basic() {
+        assert!(constant_time_eq(b"abc", b"abc"));
+        assert!(!constant_time_eq(b"abc", b"abd"));
+        assert!(!constant_time_eq(b"abc", b"abcd"));
+        assert!(constant_time_eq(b"", b""));
+    }
+
+    #[tokio::test]
+    async fn missing_legal_token_returns_503() {
+        let state = AuditEndpointState {
+            registry: Registry::new(Arc::new(InMemory::new())),
+            writer: Arc::new(SubjectAuditWriter::with_inline_key(
+                Arc::new(InMemory::new()), (0u8..32).collect(),
+            )),
+            legal_token: None,
+        };
+        let headers = HeaderMap::new();
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::SERVICE_UNAVAILABLE, _))));
+    }
+
+    #[tokio::test]
+    async fn missing_header_returns_401() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        let headers = HeaderMap::new();
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
+    }
+
+    #[tokio::test]
+    async fn wrong_token_returns_401() {
+        let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
+        let mut headers = HeaderMap::new();
+        headers.insert(LEGAL_TOKEN_HEADER, "wrong".parse().unwrap());
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
+    }
+
+    #[tokio::test]
+    async fn correct_token_passes_auth() {
+        let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
+        let mut headers = HeaderMap::new();
+        headers.insert(LEGAL_TOKEN_HEADER, "correct-token-must-be-long-enough".parse().unwrap());
+        let r = require_legal_auth(&state, &headers);
+        assert!(r.is_ok());
+    }
+
+    #[tokio::test]
+    async fn audit_response_assembly_full_path() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        // Plant a manifest + 3 audit rows.
+        state.registry.put_subject(fixture_manifest("CAND-AUDIT-1")).await.unwrap();
+        for _ in 0..3 {
+            state.writer.append(fixture_audit_row("CAND-AUDIT-1")).await.unwrap();
+        }
+        // Read rows in unbounded window.
+        let rows = state.writer
+            .read_rows_in_range("CAND-AUDIT-1", None, None)
+            .await
+            .unwrap();
+        assert_eq!(rows.len(), 3);
+        // Verify chain integrity.
+        let n = state.writer.verify_chain("CAND-AUDIT-1").await.unwrap();
+        assert_eq!(n, 3);
+    }
+
+    #[tokio::test]
+    async fn audit_response_window_filters_rows() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        state.registry.put_subject(fixture_manifest("CAND-WIN-1")).await.unwrap();
+        // Append 5 rows.
+        let mut row_times = Vec::new();
+        for _ in 0..5 {
+            let row = fixture_audit_row("CAND-WIN-1");
+            row_times.push(row.ts);
+            state.writer.append(row).await.unwrap();
+            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
+        }
+        // Window covering only the middle 3 rows.
+        let from = row_times[1];
+        let to = row_times[3];
+        let rows = state.writer
+            .read_rows_in_range("CAND-WIN-1", Some(from), Some(to))
+            .await
+            .unwrap();
+        assert!(rows.len() >= 3, "expected ≥3 rows in window; got {}", rows.len());
+        assert!(rows.len() <= 3, "expected ≤3 rows in window; got {}", rows.len());
+    }
+
+    #[tokio::test]
+    async fn empty_token_file_results_in_disabled_endpoint() {
+        let tmp = std::env::temp_dir().join(format!("legal_empty_{}.tok", std::process::id()));
+        tokio::fs::write(&tmp, "").await.unwrap();
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
+        let state = AuditEndpointState::new(registry, writer, &tmp).await;
+        assert!(state.legal_token.is_none());
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+
+    #[tokio::test]
+    async fn short_token_file_rejected_at_load() {
+        let tmp = std::env::temp_dir().join(format!("legal_short_{}.tok", std::process::id()));
+        tokio::fs::write(&tmp, "tooshort").await.unwrap();
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
+        let state = AuditEndpointState::new(registry, writer, &tmp).await;
+        assert!(state.legal_token.is_none());
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+}

crates/catalogd/src/bin/backfill_subjects.rs

@@ -0,0 +1,341 @@
+//! Backfill subject manifests from a parquet source.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3.
+//!
+//! For each row in the source parquet, creates a SubjectManifest with
+//! the BIPA-defensible defaults from the spec:
+//!   - vertical = unknown                (HIPAA fail-closed routing)
+//!   - consent.general_pii = pending_backfill_review  (NOT inferred_existing)
+//!   - consent.biometric  = never_collected           (no biometric backfill)
+//!   - retention.general_pii_until = now + 4 years    (default policy)
+//!
+//! Idempotent: if a manifest already exists for a candidate_id, the
+//! backfill skips it. Re-runs are safe.
+//!
+//! Conservative defaults: --limit 1000 by default. Pass --all to do
+//! the full source. Pass --concurrency to bound parallel writes.
+//!
+//! Usage examples:
+//!   cargo run --bin backfill_subjects -- \
+//!     --source data/datasets/workers_500k.parquet \
+//!     --dataset workers_500k \
+//!     --key-column worker_id \
+//!     --candidate-id-prefix WORKER- \
+//!     --limit 100 \
+//!     --dry-run
+//!
+//!   cargo run --bin backfill_subjects -- \
+//!     --source data/datasets/workers_500k.parquet \
+//!     --dataset workers_500k \
+//!     --key-column worker_id \
+//!     --candidate-id-prefix WORKER- \
+//!     --safe-view workers_safe \
+//!     --all
+//!
+//! On the live host the binary reads from / writes to ./data so it
+//! integrates with the running catalogd's persistence path.
+
+use catalogd::registry::Registry;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use shared::types::{
+    BiometricConsent, BiometricConsentStatus, ConsentStatus, GeneralPiiConsent,
+    SubjectConsent, SubjectDatasetRef, SubjectManifest, SubjectRetention,
+    SubjectStatus, SubjectVertical,
+};
+use std::fs::File;
+use std::path::PathBuf;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use storaged::backend;
+
+const DEFAULT_LIMIT: usize = 1000;
+const DEFAULT_CONCURRENCY: usize = 32;
+const DEFAULT_RETENTION_YEARS: i64 = 4;
+
+#[derive(Debug)]
+struct Args {
+    source: PathBuf,
+    dataset_name: String,
+    key_column: String,
+    candidate_id_prefix: String,
+    safe_view: Option<String>,
+    storage_root: PathBuf,
+    limit: Option<usize>,
+    concurrency: usize,
+    dry_run: bool,
+}
+
+fn parse_args() -> Result<Args, String> {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut source: Option<PathBuf> = None;
+    let mut dataset_name: Option<String> = None;
+    let mut key_column: Option<String> = None;
+    let mut candidate_id_prefix: String = String::new();
+    let mut safe_view: Option<String> = None;
+    let mut storage_root: PathBuf = PathBuf::from("./data");
+    let mut limit: Option<usize> = Some(DEFAULT_LIMIT);
+    let mut concurrency: usize = DEFAULT_CONCURRENCY;
+    let mut dry_run = false;
+
+    let mut i = 1;
+    while i < argv.len() {
+        let arg = &argv[i];
+        let next = || -> Result<String, String> {
+            argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
+        };
+        match arg.as_str() {
+            "--source" => { source = Some(PathBuf::from(next()?)); i += 2; }
+            "--dataset" => { dataset_name = Some(next()?); i += 2; }
+            "--key-column" => { key_column = Some(next()?); i += 2; }
+            "--candidate-id-prefix" => { candidate_id_prefix = next()?; i += 2; }
+            "--safe-view" => { safe_view = Some(next()?); i += 2; }
+            "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
+            "--limit" => { limit = Some(next()?.parse().map_err(|e| format!("--limit: {e}"))?); i += 2; }
+            "--all" => { limit = None; i += 1; }
+            "--concurrency" => { concurrency = next()?.parse().map_err(|e| format!("--concurrency: {e}"))?; i += 2; }
+            "--dry-run" => { dry_run = true; i += 1; }
+            "-h" | "--help" => { print_help(); std::process::exit(0); }
+            other => return Err(format!("unknown arg: {other}")),
+        }
+    }
+    Ok(Args {
+        source: source.ok_or("missing --source")?,
+        dataset_name: dataset_name.ok_or("missing --dataset")?,
+        key_column: key_column.ok_or("missing --key-column")?,
+        candidate_id_prefix,
+        safe_view,
+        storage_root,
+        limit,
+        concurrency,
+        dry_run,
+    })
+}
+
+fn print_help() {
+    eprintln!("backfill_subjects — populate subject manifests from a parquet source\n");
+    eprintln!("Usage: backfill_subjects [flags]\n");
+    eprintln!("Required:");
+    eprintln!("  --source PATH                parquet file containing subject identifiers");
+    eprintln!("  --dataset NAME               catalogd dataset name (must already exist)");
+    eprintln!("  --key-column COL             column in the parquet that holds subject identifiers\n");
+    eprintln!("Optional:");
+    eprintln!("  --candidate-id-prefix STR    prefix added to each row's value to form candidate_id");
+    eprintln!("                                 (e.g. 'WORKER-' on a worker_id int64 column)");
+    eprintln!("  --safe-view NAME             name of an AiView that safely projects this subject's data");
+    eprintln!("  --storage-root PATH          local-filesystem catalog root (default: ./data)");
+    eprintln!("  --limit N                    process at most N rows (default: {DEFAULT_LIMIT})");
+    eprintln!("  --all                        process every row in the source");
+    eprintln!("  --concurrency N              max concurrent writes (default: {DEFAULT_CONCURRENCY})");
+    eprintln!("  --dry-run                    parse + count + sample, don't write anything");
+}
+
+#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
+async fn main() {
+    let args = match parse_args() {
+        Ok(a) => a,
+        Err(e) => {
+            eprintln!("error: {e}");
+            print_help();
+            std::process::exit(2);
+        }
+    };
+
+    if let Err(e) = run(args).await {
+        eprintln!("backfill failed: {e}");
+        std::process::exit(1);
+    }
+}
+
+async fn run(args: Args) -> Result<(), String> {
+    eprintln!("[backfill] source:  {}", args.source.display());
+    eprintln!("[backfill] dataset: {}", args.dataset_name);
+    eprintln!("[backfill] key:     {} (prefix: {:?})", args.key_column, args.candidate_id_prefix);
+    eprintln!("[backfill] limit:   {}", args.limit.map(|n| n.to_string()).unwrap_or("all".into()));
+    eprintln!("[backfill] dry-run: {}", args.dry_run);
+
+    // Stand up catalogd over the same local FS the running daemon uses.
+    let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
+    let reg = Arc::new(Registry::new(store));
+    reg.rebuild().await?;
+
+    if reg.get_by_name(&args.dataset_name).await.is_none() {
+        return Err(format!(
+            "dataset '{}' not found in catalog at {}; backfill needs the dataset registered first",
+            args.dataset_name, args.storage_root.display(),
+        ));
+    }
+    eprintln!("[backfill] dataset '{}' exists in catalog ✓", args.dataset_name);
+
+    // Open parquet reader.
+    let file = File::open(&args.source).map_err(|e| format!("open {}: {e}", args.source.display()))?;
+    let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(|e| e.to_string())?;
+    let total_rows = builder.metadata().file_metadata().num_rows() as usize;
+    eprintln!("[backfill] parquet rows: {total_rows}");
+
+    // Project only the key column for IO efficiency.
+    let schema = builder.schema().clone();
+    let col_idx = schema.index_of(&args.key_column)
+        .map_err(|_| format!("column '{}' not found; parquet has: {:?}",
+            args.key_column,
+            schema.fields().iter().map(|f| f.name()).collect::<Vec<_>>(),
+        ))?;
+    let projection = parquet::arrow::ProjectionMask::roots(builder.parquet_schema(), [col_idx]);
+    let mut reader = builder.with_projection(projection).build().map_err(|e| e.to_string())?;
+
+    // Collect candidate IDs (limited).
+    let cap = args.limit.unwrap_or(total_rows);
+    let mut candidate_ids: Vec<String> = Vec::with_capacity(cap.min(total_rows));
+    let mut rows_seen = 0usize;
+    'outer: while let Some(batch_result) = reader.next() {
+        let batch = batch_result.map_err(|e| format!("batch: {e}"))?;
+        let column = batch.column(0); // single projected column
+        for row in 0..batch.num_rows() {
+            if let Some(limit) = args.limit {
+                if candidate_ids.len() >= limit { break 'outer; }
+            }
+            rows_seen += 1;
+            let raw = format_cell_value(column, row)?;
+            candidate_ids.push(format!("{}{}", args.candidate_id_prefix, raw));
+        }
+    }
+    eprintln!("[backfill] collected {} candidate_ids ({} rows scanned)", candidate_ids.len(), rows_seen);
+
+    // Sample for the operator.
+    eprintln!("[backfill] sample (first 5):");
+    for cid in candidate_ids.iter().take(5) {
+        eprintln!("  - {cid}");
+    }
+
+    if args.dry_run {
+        eprintln!("[backfill] DRY-RUN — no writes. Re-run without --dry-run to commit.");
+        return Ok(());
+    }
+
+    // Write manifests with bounded concurrency.
+    let now = chrono::Utc::now();
+    let retention_until = now + chrono::Duration::days(365 * DEFAULT_RETENTION_YEARS);
+    let safe_views: Vec<String> = args.safe_view.into_iter().collect();
+    let dataset_ref = SubjectDatasetRef {
+        name: args.dataset_name.clone(),
+        key_column: args.key_column.clone(),
+        key_value: String::new(), // filled per-subject below
+    };
+
+    let attempted = Arc::new(AtomicUsize::new(0));
+    let inserted = Arc::new(AtomicUsize::new(0));
+    let skipped = Arc::new(AtomicUsize::new(0));
+    let failed = Arc::new(AtomicUsize::new(0));
+    let progress_every = ((candidate_ids.len() / 20).max(1)).min(5_000);
+
+    // Bounded concurrency via Semaphore.
+    let sem = Arc::new(tokio::sync::Semaphore::new(args.concurrency));
+    let mut handles = Vec::with_capacity(candidate_ids.len());
+    for (idx, cid) in candidate_ids.into_iter().enumerate() {
+        let permit = sem.clone().acquire_owned().await.map_err(|e| e.to_string())?;
+        let reg = reg.clone();
+        let mut ds_ref = dataset_ref.clone();
+        // strip_prefix gives one-shot semantics; trim_start_matches strips
+        // repeated occurrences (a "WORKER-WORKER-1" id would lose both
+        // prefixes). 2026-05-03 opus scrum WARN at backfill_subjects.rs:189.
+        ds_ref.key_value = cid.strip_prefix(&args.candidate_id_prefix)
+            .unwrap_or(&cid).to_string();
+        let safe_views = safe_views.clone();
+        let attempted = attempted.clone();
+        let inserted = inserted.clone();
+        let skipped = skipped.clone();
+        let failed = failed.clone();
+
+        let handle = tokio::spawn(async move {
+            let _permit = permit;
+            attempted.fetch_add(1, Ordering::Relaxed);
+            // Idempotency: skip if subject already exists.
+            if reg.get_subject(&cid).await.is_some() {
+                skipped.fetch_add(1, Ordering::Relaxed);
+                return;
+            }
+            let manifest = SubjectManifest {
+                schema: "subject_manifest.v1".into(),
+                candidate_id: cid.clone(),
+                created_at: now,
+                updated_at: now,
+                status: SubjectStatus::Active,
+                vertical: SubjectVertical::Unknown,
+                consent: SubjectConsent {
+                    general_pii: GeneralPiiConsent {
+                        status: ConsentStatus::PendingBackfillReview,
+                        version: String::new(),
+                        given_at: None,
+                        withdrawn_at: None,
+                    },
+                    biometric: BiometricConsent {
+                        status: BiometricConsentStatus::NeverCollected,
+                        retention_until: None,
+                    },
+                },
+                retention: SubjectRetention {
+                    general_pii_until: retention_until,
+                    policy: "4_year_default".into(),
+                },
+                datasets: vec![ds_ref],
+                safe_views,
+                audit_log_path: String::new(),
+                audit_log_chain_root: String::new(),
+                biometric_collection: None,
+            };
+            match reg.put_subject(manifest).await {
+                Ok(_) => { inserted.fetch_add(1, Ordering::Relaxed); }
+                Err(e) => {
+                    failed.fetch_add(1, Ordering::Relaxed);
+                    eprintln!("  ✗ {cid}: {e}");
+                }
+            }
+            let cur = attempted.load(Ordering::Relaxed);
+            if cur % progress_every == 0 {
+                eprintln!("[backfill] progress: {} attempted ({} inserted, {} skipped, {} failed)",
+                    cur, inserted.load(Ordering::Relaxed),
+                    skipped.load(Ordering::Relaxed),
+                    failed.load(Ordering::Relaxed));
+            }
+            let _ = idx;
+        });
+        handles.push(handle);
+    }
+    for h in handles {
+        let _ = h.await;
+    }
+
+    eprintln!();
+    eprintln!("──────────────────────────────────────────────────────────────────");
+    eprintln!("[backfill] done");
+    eprintln!("[backfill]   attempted: {}", attempted.load(Ordering::Relaxed));
+    eprintln!("[backfill]   inserted:  {}", inserted.load(Ordering::Relaxed));
+    eprintln!("[backfill]   skipped:   {} (already existed — idempotent re-run)", skipped.load(Ordering::Relaxed));
+    eprintln!("[backfill]   failed:    {}", failed.load(Ordering::Relaxed));
+    eprintln!("[backfill] subjects in catalog now: {}", reg.subjects_count().await);
+    Ok(())
+}
+
+/// Format a single cell value as a string for use in the candidate_id.
+/// Supports the integer + string types most likely to be subject keys.
+fn format_cell_value(array: &Arc<dyn arrow::array::Array>, row: usize) -> Result<String, String> {
+    use arrow::array::{Int32Array, Int64Array, StringArray, UInt32Array, UInt64Array};
+    use arrow::datatypes::DataType;
+    if array.is_null(row) {
+        return Err(format!("row {row}: key column is null — cannot form candidate_id"));
+    }
+    let dt = array.data_type();
+    let s = match dt {
+        DataType::Int64 => array.as_any().downcast_ref::<Int64Array>()
+            .ok_or("int64 downcast")?.value(row).to_string(),
+        DataType::Int32 => array.as_any().downcast_ref::<Int32Array>()
+            .ok_or("int32 downcast")?.value(row).to_string(),
+        DataType::UInt64 => array.as_any().downcast_ref::<UInt64Array>()
+            .ok_or("uint64 downcast")?.value(row).to_string(),
+        DataType::UInt32 => array.as_any().downcast_ref::<UInt32Array>()
+            .ok_or("uint32 downcast")?.value(row).to_string(),
+        DataType::Utf8 => array.as_any().downcast_ref::<StringArray>()
+            .ok_or("utf8 downcast")?.value(row).to_string(),
+        other => return Err(format!("unsupported key column type: {other:?}")),
+    };
+    Ok(s)
+}

crates/catalogd/src/bin/parity_subject_audit.rs

@@ -0,0 +1,235 @@
+//! Cross-runtime parity helper for subject-audit chain.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.
+//!
+//! This binary is consumed by the Go-repo's
+//! scripts/cutover/parity/subject_audit_parity.sh.
+//! Go counterpart: scripts/cutover/parity/subject_audit_helper/main.go
+//! (in the Go repo). Both helpers are kept in lockstep by the parity
+//! script's byte-equality assertions — change one, change both.
+//!
+//! Both helpers MUST produce byte-identical output for the same inputs.
+//! Divergence here is a parity break — a SubjectManifest written by Rust
+//! that Go cannot verify, or vice versa.
+//!
+//! Two modes:
+//!
+//!   --known-answer
+//!       Print the canonical bytes + HMAC of a hardcoded fixture row.
+//!       The Go helper must produce IDENTICAL bytes + IDENTICAL hash.
+//!       Hardcoded fixture matches Go test TestKnownAnswerVector.
+//!
+//!   --verify <audit_log_path> --key <key_path>
+//!       Parse the JSONL audit log, replay the HMAC chain. Print
+//!       JSON: {"count": N, "tip": "<hash|GENESIS>", "verified": bool, "error": "<msg|null>"}.
+//!
+//! Output format: ONE JSON object per stdout, terminating newline. The
+//! parity script diffs Rust stdout vs Go stdout via `diff -q`.
+
+use catalogd::subject_audit::SubjectAuditWriter;
+use hmac::{Hmac, Mac};
+use object_store::ObjectStore;
+use object_store::memory::InMemory;
+use serde::Serialize;
+use sha2::Sha256;
+use shared::types::{AuditAccessor, SubjectAuditRow};
+use std::collections::BTreeMap;
+use std::path::PathBuf;
+use std::sync::Arc;
+
+type HmacSha256 = Hmac<Sha256>;
+
+const GENESIS: &str = "GENESIS";
+
+/// Same canonical-JSON algorithm as crates/catalogd/src/subject_audit.rs.
+/// Reproduced here so the helper does not depend on internal-only items
+/// in that file. If the algorithm there changes, change it here AND in
+/// the Go helper in lockstep.
+fn canonical_json(v: &serde_json::Value) -> Vec<u8> {
+    fn rewrite(v: &serde_json::Value) -> serde_json::Value {
+        match v {
+            serde_json::Value::Object(map) => {
+                let sorted: BTreeMap<String, serde_json::Value> = map
+                    .iter()
+                    .map(|(k, v)| (k.clone(), rewrite(v)))
+                    .collect();
+                serde_json::Value::Object(sorted.into_iter().collect())
+            }
+            serde_json::Value::Array(arr) => {
+                serde_json::Value::Array(arr.iter().map(rewrite).collect())
+            }
+            other => other.clone(),
+        }
+    }
+    serde_json::to_vec(&rewrite(v)).expect("canonical-json serialize")
+}
+
+fn canonical_row_bytes(row: &SubjectAuditRow) -> Vec<u8> {
+    let mut v: serde_json::Value = serde_json::to_value(row).expect("row to value");
+    if let Some(obj) = v.as_object_mut() {
+        obj.remove("row_hmac");
+    }
+    canonical_json(&v)
+}
+
+fn compute_hmac(key: &[u8], prev: &str, canonical: &[u8]) -> String {
+    let mut mac = <HmacSha256 as Mac>::new_from_slice(key).expect("HMAC accepts any key length");
+    mac.update(prev.as_bytes());
+    mac.update(canonical);
+    let result = mac.finalize().into_bytes();
+    const HEX: &[u8; 16] = b"0123456789abcdef";
+    let mut s = String::with_capacity(64);
+    for byte in result {
+        s.push(HEX[(byte >> 4) as usize] as char);
+        s.push(HEX[(byte & 0x0f) as usize] as char);
+    }
+    s
+}
+
+fn deterministic_key() -> Vec<u8> {
+    (0u8..32).collect()
+}
+
+/// Print to stderr + exit(2). Never returns. Used in place of `.expect()`
+/// on argv / file reads so failures produce a clean usage error rather
+/// than a Rust panic backtrace that the parity script would mistake
+/// for a parity break.
+fn die(msg: &str) -> ! {
+    eprintln!("error: {msg}");
+    std::process::exit(2);
+}
+
+#[derive(Serialize)]
+struct KnownAnswerOut {
+    mode: &'static str,
+    canonical: String,
+    hmac: String,
+    canonical_bytes_len: usize,
+}
+
+#[derive(Serialize)]
+struct VerifyOut {
+    mode: &'static str,
+    count: usize,
+    tip: String,
+    verified: bool,
+    error: Option<String>,
+}
+
+fn known_answer() {
+    let row = SubjectAuditRow {
+        schema: "subject_audit.v1".into(),
+        ts: chrono::DateTime::parse_from_rfc3339("2026-05-03T12:00:00Z")
+            .unwrap()
+            .with_timezone(&chrono::Utc),
+        candidate_id: "WORKER-FIXED".into(),
+        accessor: AuditAccessor {
+            kind: "gateway_lookup".into(),
+            daemon: "gateway".into(),
+            purpose: "parity_test".into(),
+            trace_id: "trace-fixed".into(),
+        },
+        fields_accessed: vec!["name".into()],
+        result: "success".into(),
+        prev_chain_hash: GENESIS.into(),
+        row_hmac: String::new(),
+    };
+    let canon = canonical_row_bytes(&row);
+    let hmac = compute_hmac(&deterministic_key(), GENESIS, &canon);
+    let out = KnownAnswerOut {
+        mode: "known_answer",
+        canonical: String::from_utf8(canon.clone()).expect("canonical is utf-8"),
+        hmac,
+        canonical_bytes_len: canon.len(),
+    };
+    println!("{}", serde_json::to_string(&out).unwrap());
+}
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut mode_known_answer = false;
+    let mut audit_path: Option<PathBuf> = None;
+    let mut key_path: Option<PathBuf> = None;
+
+    // Argv parsing fails via stderr+exit, never via panic. The parity
+    // script captures stdout for byte-comparison; a Rust panic backtrace
+    // would land in stdout (the script merges 2>&1) and read as a parity
+    // break instead of a usage error. (Caught 2026-05-03 opus scrum WARN.)
+    let mut i = 1;
+    while i < argv.len() {
+        match argv[i].as_str() {
+            "--known-answer" => {
+                mode_known_answer = true;
+                i += 1;
+            }
+            "--verify" => {
+                let p = argv.get(i + 1).unwrap_or_else(|| die("--verify needs a path"));
+                audit_path = Some(PathBuf::from(p));
+                i += 2;
+            }
+            "--key" => {
+                let p = argv.get(i + 1).unwrap_or_else(|| die("--key needs a path"));
+                key_path = Some(PathBuf::from(p));
+                i += 2;
+            }
+            "-h" | "--help" => {
+                eprintln!("parity_subject_audit --known-answer");
+                eprintln!("parity_subject_audit --verify <audit_log_path> --key <key_path>");
+                std::process::exit(0);
+            }
+            other => {
+                eprintln!("unknown arg: {other}");
+                std::process::exit(2);
+            }
+        }
+    }
+
+    if mode_known_answer {
+        known_answer();
+        return;
+    }
+
+    let audit_path = audit_path.unwrap_or_else(|| die("need --known-answer OR --verify <path> --key <path>"));
+    let key_path = key_path.unwrap_or_else(|| die("--verify also needs --key"));
+
+    let key = std::fs::read(&key_path).unwrap_or_else(|e| die(&format!("read key file: {e}")));
+    let candidate_id = audit_path
+        .file_name()
+        .and_then(|s| s.to_str())
+        .and_then(|s| s.strip_suffix(".audit.jsonl"))
+        .unwrap_or_else(|| die("audit log path must end with <candidate_id>.audit.jsonl"))
+        .to_string();
+
+    // Stand up an in-memory object store, seed it with the audit log
+    // bytes at the canonical key, then ask SubjectAuditWriter to verify.
+    // This way we exercise the SAME verify_chain function the production
+    // gateway calls — not a re-implementation that might drift.
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let bytes = std::fs::read(&audit_path).expect("read audit log");
+    let log_key = format!("_catalog/subjects/{}.audit.jsonl", candidate_id);
+    storaged::ops::put(&store, &log_key, bytes::Bytes::from(bytes))
+        .await
+        .expect("seed object store");
+
+    let writer = SubjectAuditWriter::with_inline_key(store, key);
+    let result = writer.verify_chain(&candidate_id).await;
+    let tip = writer.chain_tip(&candidate_id).await.unwrap_or(GENESIS.into());
+    let out = match result {
+        Ok(count) => VerifyOut {
+            mode: "verify",
+            count,
+            tip,
+            verified: true,
+            error: None,
+        },
+        Err(e) => VerifyOut {
+            mode: "verify",
+            count: 0,
+            tip: GENESIS.into(),
+            verified: false,
+            error: Some(e),
+        },
+    };
+    println!("{}", serde_json::to_string(&out).unwrap());
+}

crates/catalogd/src/bin/retention_sweep.rs

@@ -0,0 +1,376 @@
+//! Daily retention sweep for subject manifests.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7:
+//!   "Subjects whose retention.general_pii_until < now AND status != erased
+//!    get marked for review (don't auto-delete; legal needs to approve)."
+//!
+//! Per shared::types::BiometricConsent doc-comment (BIPA requirement):
+//!   "max 3 years from last interaction. Implementation MUST enforce
+//!    daily expiration sweep against this field."
+//!
+//! Therefore this binary checks BOTH retention clocks and reports any
+//! subject whose general-pii window OR biometric window has expired
+//! while the subject is not already erased / already-flagged.
+//!
+//! It does NOT mutate subject manifests. It produces a JSONL review
+//! queue at:
+//!   data/_catalog/subjects/_retention_sweep_<YYYY-MM-DD>.jsonl
+//!
+//! Operators / legal review the queue and act:
+//!   - extend retention (legal contract update),
+//!   - flip status to retention_expired,
+//!   - schedule erasure.
+//!
+//! Usage:
+//!   cargo run --bin retention_sweep                      # dry-run, stderr only
+//!   cargo run --bin retention_sweep -- --apply           # also write report file
+//!   cargo run --bin retention_sweep -- --as-of 2030-01-01T00:00:00Z
+//!   cargo run --bin retention_sweep -- --storage-root /var/lib/lakehouse/data --apply
+
+use catalogd::registry::Registry;
+use chrono::{DateTime, Utc};
+use serde::Serialize;
+use shared::types::{BiometricConsentStatus, SubjectManifest, SubjectStatus};
+use std::path::PathBuf;
+use std::sync::Arc;
+use storaged::backend;
+
+#[derive(Debug)]
+struct Args {
+    storage_root: PathBuf,
+    apply: bool,
+    as_of: Option<DateTime<Utc>>,
+}
+
+fn parse_args() -> Result<Args, String> {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut storage_root = PathBuf::from("./data");
+    let mut apply = false;
+    let mut as_of: Option<DateTime<Utc>> = None;
+
+    let mut i = 1;
+    while i < argv.len() {
+        let arg = &argv[i];
+        let next = || -> Result<String, String> {
+            argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
+        };
+        match arg.as_str() {
+            "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
+            "--apply" => { apply = true; i += 1; }
+            "--as-of" => {
+                let raw = next()?;
+                as_of = Some(raw.parse().map_err(|e| format!("--as-of (need RFC3339): {e}"))?);
+                i += 2;
+            }
+            "-h" | "--help" => { print_help(); std::process::exit(0); }
+            other => return Err(format!("unknown arg: {other}")),
+        }
+    }
+    Ok(Args { storage_root, apply, as_of })
+}
+
+fn print_help() {
+    eprintln!("retention_sweep — daily retention check on subject manifests");
+    eprintln!();
+    eprintln!("Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7.");
+    eprintln!("Reports overdue subjects to a JSONL review queue. Does NOT");
+    eprintln!("auto-delete. Legal must approve every action.");
+    eprintln!();
+    eprintln!("Flags:");
+    eprintln!("  --storage-root PATH   local-filesystem catalog root (default: ./data)");
+    eprintln!("  --apply               write report to data/_catalog/subjects/_retention_sweep_<date>.jsonl");
+    eprintln!("  --as-of RFC3339       use this timestamp as 'now' (test/forecast helper)");
+}
+
+#[derive(Debug, PartialEq, Clone, Copy)]
+enum OverdueReason {
+    GeneralPii,
+    Biometric,
+    Both,
+}
+
+impl OverdueReason {
+    fn as_str(self) -> &'static str {
+        match self {
+            OverdueReason::GeneralPii => "general_pii_expired",
+            OverdueReason::Biometric => "biometric_expired",
+            OverdueReason::Both => "both_expired",
+        }
+    }
+}
+
+#[derive(Debug, Serialize)]
+struct OverdueRow<'a> {
+    candidate_id: &'a str,
+    status: &'a str,
+    general_pii_until: DateTime<Utc>,
+    days_overdue_general: i64,
+    biometric_status: &'a str,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    biometric_retention_until: Option<DateTime<Utc>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    days_overdue_biometric: Option<i64>,
+    reason: &'a str,
+    swept_at: DateTime<Utc>,
+}
+
+/// Skipped statuses (idempotent across daily runs):
+///   - Erased: subject's data is already gone; nothing to review.
+///   - RetentionExpired: subject is already flagged; do not re-fire.
+///
+/// Actionable statuses: PendingConsent, Active, Withdrawn — each can
+/// still hold PII rows on disk past the retention window.
+fn is_overdue(subject: &SubjectManifest, now: DateTime<Utc>) -> Option<OverdueReason> {
+    if matches!(subject.status, SubjectStatus::Erased | SubjectStatus::RetentionExpired) {
+        return None;
+    }
+    let general_overdue = subject.retention.general_pii_until < now;
+    let biometric_overdue = subject
+        .consent
+        .biometric
+        .retention_until
+        .map(|t| t < now)
+        .unwrap_or(false);
+    match (general_overdue, biometric_overdue) {
+        (true, true) => Some(OverdueReason::Both),
+        (true, false) => Some(OverdueReason::GeneralPii),
+        (false, true) => Some(OverdueReason::Biometric),
+        (false, false) => None,
+    }
+}
+
+fn status_str(s: &SubjectStatus) -> &'static str {
+    match s {
+        SubjectStatus::PendingConsent => "pending_consent",
+        SubjectStatus::Active => "active",
+        SubjectStatus::Withdrawn => "withdrawn",
+        SubjectStatus::RetentionExpired => "retention_expired",
+        SubjectStatus::Erased => "erased",
+    }
+}
+
+fn biometric_status_str(s: &BiometricConsentStatus) -> &'static str {
+    match s {
+        BiometricConsentStatus::NeverCollected => "never_collected",
+        BiometricConsentStatus::Pending => "pending",
+        BiometricConsentStatus::Given => "given",
+        BiometricConsentStatus::Withdrawn => "withdrawn",
+        BiometricConsentStatus::Expired => "expired",
+    }
+}
+
+#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
+async fn main() {
+    let args = match parse_args() {
+        Ok(a) => a,
+        Err(e) => {
+            eprintln!("error: {e}");
+            print_help();
+            std::process::exit(2);
+        }
+    };
+    if let Err(e) = run(args).await {
+        eprintln!("retention_sweep failed: {e}");
+        std::process::exit(1);
+    }
+}
+
+async fn run(args: Args) -> Result<(), String> {
+    let now = args.as_of.unwrap_or_else(Utc::now);
+    eprintln!("[retention_sweep] storage-root: {}", args.storage_root.display());
+    eprintln!("[retention_sweep] now (sweep clock): {}", now.to_rfc3339());
+    eprintln!("[retention_sweep] apply: {}", args.apply);
+
+    let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
+    let reg = Arc::new(Registry::new(store));
+    reg.rebuild().await?;
+
+    let subjects = reg.list_subjects().await;
+    eprintln!("[retention_sweep] subjects loaded: {}", subjects.len());
+
+    let mut overdue_rows: Vec<OverdueRow> = Vec::new();
+    let mut by_reason: std::collections::BTreeMap<&'static str, usize> = Default::default();
+
+    for s in &subjects {
+        if let Some(reason) = is_overdue(s, now) {
+            let reason_str: &'static str = reason.as_str();
+            *by_reason.entry(reason_str).or_insert(0) += 1;
+            overdue_rows.push(OverdueRow {
+                candidate_id: &s.candidate_id,
+                status: status_str(&s.status),
+                general_pii_until: s.retention.general_pii_until,
+                days_overdue_general: (now - s.retention.general_pii_until).num_days(),
+                biometric_status: biometric_status_str(&s.consent.biometric.status),
+                biometric_retention_until: s.consent.biometric.retention_until,
+                days_overdue_biometric: s
+                    .consent
+                    .biometric
+                    .retention_until
+                    .map(|t| (now - t).num_days()),
+                reason: reason_str,
+                swept_at: now,
+            });
+        }
+    }
+
+    eprintln!("[retention_sweep] overdue subjects: {}", overdue_rows.len());
+    for (reason, n) in &by_reason {
+        eprintln!("[retention_sweep]   {reason}: {n}");
+    }
+
+    if overdue_rows.is_empty() {
+        eprintln!("[retention_sweep] nothing to do; all subjects within retention window.");
+        return Ok(());
+    }
+
+    eprintln!("[retention_sweep] sample (first 5):");
+    for row in overdue_rows.iter().take(5) {
+        eprintln!(
+            "  - {} status={} reason={} general_overdue_days={}",
+            row.candidate_id, row.status, row.reason, row.days_overdue_general
+        );
+    }
+
+    if !args.apply {
+        eprintln!();
+        eprintln!("[retention_sweep] DRY-RUN: no file written. Re-run with --apply to persist.");
+        return Ok(());
+    }
+
+    let date_stamp = now.format("%Y-%m-%d");
+    let report_path = args
+        .storage_root
+        .join("_catalog")
+        .join("subjects")
+        .join(format!("_retention_sweep_{date_stamp}.jsonl"));
+    if let Some(parent) = report_path.parent() {
+        std::fs::create_dir_all(parent).map_err(|e| format!("mkdir {parent:?}: {e}"))?;
+    }
+    let mut buf = String::new();
+    for row in &overdue_rows {
+        let line = serde_json::to_string(row).map_err(|e| format!("serialize row: {e}"))?;
+        buf.push_str(&line);
+        buf.push('\n');
+    }
+    std::fs::write(&report_path, &buf).map_err(|e| format!("write {report_path:?}: {e}"))?;
+    eprintln!("[retention_sweep] report written: {}", report_path.display());
+    eprintln!("[retention_sweep] action required by legal/operator. no manifests were mutated.");
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use chrono::Duration;
+    use shared::types::{
+        BiometricConsent, ConsentStatus, GeneralPiiConsent, SubjectConsent, SubjectRetention,
+        SubjectVertical,
+    };
+
+    fn fixture(
+        id: &str,
+        status: SubjectStatus,
+        gen_until: DateTime<Utc>,
+        bio_until: Option<DateTime<Utc>>,
+    ) -> SubjectManifest {
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: id.into(),
+            created_at: Utc::now(),
+            updated_at: Utc::now(),
+            status,
+            vertical: SubjectVertical::General,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::Given,
+                    version: "v1".into(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: if bio_until.is_some() {
+                        BiometricConsentStatus::Given
+                    } else {
+                        BiometricConsentStatus::NeverCollected
+                    },
+                    retention_until: bio_until,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: gen_until,
+                policy: "test".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+            biometric_collection: None,
+        }
+    }
+
+    #[test]
+    fn active_subject_inside_window_not_overdue() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Active, now + Duration::days(30), None);
+        assert!(is_overdue(&s, now).is_none());
+    }
+
+    #[test]
+    fn active_subject_past_general_window_flags_general() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Active, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+
+    #[test]
+    fn erased_subject_never_re_flags_even_if_far_past() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Erased, now - Duration::days(1000), None);
+        assert!(is_overdue(&s, now).is_none(), "erased subjects must not re-fire");
+    }
+
+    #[test]
+    fn already_retention_expired_is_idempotent() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::RetentionExpired, now - Duration::days(1), None);
+        assert!(is_overdue(&s, now).is_none(), "already-flagged subjects must not re-fire");
+    }
+
+    #[test]
+    fn biometric_overdue_alone_triggers_bipa_path() {
+        let now = Utc::now();
+        let s = fixture(
+            "a",
+            SubjectStatus::Active,
+            now + Duration::days(30),
+            Some(now - Duration::days(1)),
+        );
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::Biometric));
+    }
+
+    #[test]
+    fn both_clocks_overdue_reports_both() {
+        let now = Utc::now();
+        let s = fixture(
+            "a",
+            SubjectStatus::Active,
+            now - Duration::days(2),
+            Some(now - Duration::days(1)),
+        );
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::Both));
+    }
+
+    #[test]
+    fn pending_consent_subject_can_still_expire() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::PendingConsent, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+
+    #[test]
+    fn withdrawn_subject_still_subject_to_retention() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Withdrawn, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+}

crates/catalogd/src/lib.rs

@@ -2,3 +2,6 @@ pub mod registry;
 pub mod service;
 pub mod grpc;
 pub mod tombstones;
+pub mod subject_audit;
+pub mod audit_endpoint;
+pub mod biometric_endpoint;

crates/catalogd/src/registry.rs

@@ -1,6 +1,6 @@
 use shared::types::{
     AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile,
-    ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone,
+    ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, SubjectManifest, Tombstone,
 };
 
 use crate::tombstones::TombstoneStore;
@@ -66,16 +66,20 @@ pub struct MetadataUpdate {
 
 const MANIFEST_PREFIX: &str = "_catalog/manifests";
 const VIEW_PREFIX: &str = "_catalog/views";
+const SUBJECT_PREFIX: &str = "_catalog/subjects";
 const PROFILE_PREFIX: &str = "_catalog/profiles";
 
 /// In-memory dataset registry backed by manifest persistence in object storage.
 /// Also tracks AiViews (Phase D) — safe projections over base datasets.
 /// And tombstones (Phase E) — soft-delete markers applied at query time.
+/// And subject manifests (compliance/access metadata layer) — see
+/// docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
 #[derive(Clone)]
 pub struct Registry {
     datasets: Arc<RwLock<HashMap<DatasetId, DatasetManifest>>>,
     views: Arc<RwLock<HashMap<String, AiView>>>,
     profiles: Arc<RwLock<HashMap<String, ModelProfile>>>,
+    subjects: Arc<RwLock<HashMap<String, SubjectManifest>>>,
     tombstones: TombstoneStore,
     store: Arc<dyn ObjectStore>,
 }
@@ -86,6 +90,7 @@ impl Registry {
             datasets: Arc::new(RwLock::new(HashMap::new())),
             views: Arc::new(RwLock::new(HashMap::new())),
             profiles: Arc::new(RwLock::new(HashMap::new())),
+            subjects: Arc::new(RwLock::new(HashMap::new())),
             tombstones: TombstoneStore::new(store.clone()),
             store,
         }
@@ -179,6 +184,47 @@ impl Registry {
             tracing::info!("catalog: {} model profiles loaded", profiles.len());
         }
 
+        // Subject manifests (compliance/access metadata layer).
+        let subject_keys = ops::list(&self.store, Some(SUBJECT_PREFIX)).await.unwrap_or_default();
+        let mut subjects = self.subjects.write().await;
+        subjects.clear();
+        for key in &subject_keys {
+            if !key.ends_with(".json") { continue; }
+            let data = match ops::get(&self.store, key).await {
+                Ok(d) => d,
+                Err(e) => {
+                    tracing::warn!("subject '{key}': read failed: {e}");
+                    continue;
+                }
+            };
+            match serde_json::from_slice::<SubjectManifest>(&data) {
+                // Match put_subject: in-memory key is the SANITIZED form
+                // so get_subject lookups match the storage key shape.
+                Ok(subj) => {
+                    let map_key = sanitize_view_name(&subj.candidate_id);
+                    // Rebuild collision detection — symmetric with the
+                    // put_subject collision guard. If pre-existing storage
+                    // somehow contains two manifests whose raw ids
+                    // sanitize to the same key, log loudly so operators
+                    // know one was dropped instead of silently losing it.
+                    // (2026-05-03 opus scrum WARN registry.rs:201.)
+                    if let Some(existing) = subjects.get(&map_key) {
+                        if existing.candidate_id != subj.candidate_id {
+                            tracing::warn!(
+                                "subject rebuild collision on sanitized key '{map_key}': raw '{}' was loaded earlier; raw '{}' is being silently overwritten in memory. Run dedupe + delete the duplicate manifest file at storage.",
+                                existing.candidate_id, subj.candidate_id,
+                            );
+                        }
+                    }
+                    subjects.insert(map_key, subj);
+                }
+                Err(e) => tracing::warn!("subject '{key}': parse failed: {e}"),
+            }
+        }
+        if !subjects.is_empty() {
+            tracing::info!("catalog: {} subject manifests loaded", subjects.len());
+        }
+
         Ok(count)
     }
 
@@ -686,6 +732,94 @@ impl Registry {
         Ok(())
     }
 
+    // --- Subject manifests (compliance/access metadata layer) ---
+    // Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
+    // Mirrors the put_view / get_view / list_views / delete_view shape.
+    // The audit log per subject is owned separately by SubjectAuditWriter
+    // (Step 2) — this layer just persists the manifest itself.
+
+    /// Create or replace a subject manifest. Validates that referenced
+    /// datasets exist in the catalog (fail-fast so dangling references
+    /// don't accumulate). Persists to `_catalog/subjects/<id>.json`.
+    ///
+    /// **Storage-key collision guard** (2026-05-03 opus scrum WARN
+    /// at registry.rs:735): the storage key is `sanitize_view_name(id)`
+    /// — `/`, `:`, etc. become `_`. A naive in-memory keying by raw
+    /// candidate_id would let two distinct ids (`CAND/1` + `CAND_1`)
+    /// collide on disk while appearing distinct in memory; the second
+    /// put silently overwrites the first, and `rebuild` only loads one.
+    /// Both the in-memory map AND the storage key now use the sanitized
+    /// form. Caller's raw candidate_id is preserved on the manifest for
+    /// audit-trail integrity (the manifest body has the original).
+    /// We surface a clear error if two raw ids would collide on the
+    /// sanitized key.
+    pub async fn put_subject(&self, mut subj: SubjectManifest) -> Result<SubjectManifest, String> {
+        if subj.candidate_id.is_empty() {
+            return Err("subject candidate_id is empty".into());
+        }
+        for ds in &subj.datasets {
+            if self.get_by_name(&ds.name).await.is_none() {
+                return Err(format!(
+                    "subject '{}' references dataset '{}' which is not in the catalog",
+                    subj.candidate_id, ds.name,
+                ));
+            }
+        }
+        let now = chrono::Utc::now();
+        if subj.created_at.timestamp() == 0 {
+            subj.created_at = now;
+        }
+        subj.updated_at = now;
+
+        let map_key = sanitize_view_name(&subj.candidate_id);
+        let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
+
+        // Collision guard: if the sanitized key already exists for a
+        // DIFFERENT raw candidate_id, refuse the put. Better to fail
+        // loudly than silently overwrite an existing subject.
+        {
+            let subjects = self.subjects.read().await;
+            if let Some(existing) = subjects.get(&map_key) {
+                if existing.candidate_id != subj.candidate_id {
+                    return Err(format!(
+                        "subject id '{}' collides with existing subject '{}' on sanitized storage key '{}'",
+                        subj.candidate_id, existing.candidate_id, map_key,
+                    ));
+                }
+            }
+        }
+
+        let json = serde_json::to_vec_pretty(&subj).map_err(|e| e.to_string())?;
+        ops::put(&self.store, &key, json.into()).await?;
+
+        let mut subjects = self.subjects.write().await;
+        subjects.insert(map_key, subj.clone());
+        tracing::debug!("subject manifest persisted: {}", subj.candidate_id);
+        Ok(subj)
+    }
+
+    pub async fn get_subject(&self, candidate_id: &str) -> Option<SubjectManifest> {
+        self.subjects.read().await.get(&sanitize_view_name(candidate_id)).cloned()
+    }
+
+    pub async fn list_subjects(&self) -> Vec<SubjectManifest> {
+        self.subjects.read().await.values().cloned().collect()
+    }
+
+    pub async fn delete_subject(&self, candidate_id: &str) -> Result<(), String> {
+        let map_key = sanitize_view_name(candidate_id);
+        let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
+        ops::delete(&self.store, &key).await?;
+        self.subjects.write().await.remove(&map_key);
+        Ok(())
+    }
+
+    /// Subject count — convenience for the daily retention sweep + audit
+    /// dashboards that don't need the full list.
+    pub async fn subjects_count(&self) -> usize {
+        self.subjects.read().await.len()
+    }
+
     /// Remove a dataset from the catalog by name. Deletes the manifest
     /// from both the in-memory registry and object storage.
     ///
@@ -1027,4 +1161,130 @@ mod tests {
         assert!(reg.get(&loser_a).await.is_none());
         assert!(reg.get(&loser_b).await.is_none());
     }
+
+    // --- Subject manifests (Step 1 of subject_manifests_on_catalogd) ---
+
+    fn fixture_subject(id: &str) -> shared::types::SubjectManifest {
+        use shared::types::{
+            BiometricConsent, BiometricConsentStatus, ConsentStatus,
+            GeneralPiiConsent, SubjectConsent, SubjectManifest, SubjectRetention,
+            SubjectStatus, SubjectVertical,
+        };
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: id.to_string(),
+            created_at: chrono::Utc::now(),
+            updated_at: chrono::Utc::now(),
+            status: SubjectStatus::Active,
+            vertical: SubjectVertical::Unknown,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::PendingBackfillReview,
+                    version: String::new(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: BiometricConsentStatus::NeverCollected,
+                    retention_until: None,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
+                policy: "4_year_default".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+                biometric_collection: None,
+        }
+    }
+
+    #[tokio::test]
+    async fn put_subject_with_no_dataset_refs_succeeds() {
+        let reg = fixture();
+        let s = fixture_subject("CAND-000001");
+        let stored = reg.put_subject(s).await.unwrap();
+        assert_eq!(stored.candidate_id, "CAND-000001");
+        assert_eq!(reg.subjects_count().await, 1);
+    }
+
+    #[tokio::test]
+    async fn put_subject_rejects_dangling_dataset_ref() {
+        use shared::types::SubjectDatasetRef;
+        let reg = fixture();
+        let mut s = fixture_subject("CAND-000002");
+        s.datasets.push(SubjectDatasetRef {
+            name: "no_such_dataset".into(),
+            key_column: "candidate_id".into(),
+            key_value: "CAND-000002".into(),
+        });
+        let err = reg.put_subject(s).await.unwrap_err();
+        assert!(err.contains("not in the catalog"), "got: {err}");
+    }
+
+    #[tokio::test]
+    async fn put_subject_with_valid_dataset_ref_succeeds() {
+        use shared::types::SubjectDatasetRef;
+        let reg = fixture();
+        // Register a dataset first so the subject's reference resolves.
+        reg.register("workers".into(), fp("w1"), vec![obj("w.parquet")]).await.unwrap();
+
+        let mut s = fixture_subject("CAND-000003");
+        s.datasets.push(SubjectDatasetRef {
+            name: "workers".into(),
+            key_column: "candidate_id".into(),
+            key_value: "CAND-000003".into(),
+        });
+        let stored = reg.put_subject(s).await.unwrap();
+        assert_eq!(stored.datasets.len(), 1);
+        assert_eq!(stored.datasets[0].name, "workers");
+    }
+
+    #[tokio::test]
+    async fn subject_round_trips_through_object_store() {
+        let store = Arc::new(object_store::memory::InMemory::new());
+        let reg = Registry::new(store.clone());
+        reg.put_subject(fixture_subject("CAND-RT-001")).await.unwrap();
+
+        // Open a fresh registry over the same store and load → confirms
+        // persistence + load_all path actually round-trips the manifest.
+        let reg2 = Registry::new(store);
+        reg2.rebuild().await.unwrap();
+        let loaded = reg2.get_subject("CAND-RT-001").await;
+        assert!(loaded.is_some(), "subject not loaded from store");
+        assert_eq!(loaded.unwrap().status, shared::types::SubjectStatus::Active);
+    }
+
+    #[tokio::test]
+    async fn put_subject_rejects_sanitize_collision() {
+        // 2026-05-03 opus scrum WARN at registry.rs:735 regression:
+        // CAND/1 and CAND_1 sanitize to the same storage key. Without
+        // the collision guard, the second put silently overwrites the
+        // first on disk; with it, the second put fails loudly.
+        let reg = fixture();
+        let s1 = fixture_subject("CAND/1");
+        reg.put_subject(s1).await.unwrap();
+        let s2 = fixture_subject("CAND_1"); // sanitizes to same key
+        let err = reg.put_subject(s2).await.unwrap_err();
+        assert!(err.contains("collides"), "got: {err}");
+        assert_eq!(reg.subjects_count().await, 1, "first subject must survive");
+    }
+
+    #[tokio::test]
+    async fn delete_subject_removes_in_memory_and_persistence() {
+        let store = Arc::new(object_store::memory::InMemory::new());
+        let reg = Registry::new(store.clone());
+        reg.put_subject(fixture_subject("CAND-DEL-001")).await.unwrap();
+        assert_eq!(reg.subjects_count().await, 1);
+
+        reg.delete_subject("CAND-DEL-001").await.unwrap();
+        assert_eq!(reg.subjects_count().await, 0);
+
+        // Fresh registry over same store — confirm it's also gone from persistence.
+        let reg2 = Registry::new(store);
+        reg2.rebuild().await.unwrap();
+        assert_eq!(reg2.subjects_count().await, 0);
+    }
 }

crates/catalogd/src/subject_audit.rs

@@ -0,0 +1,587 @@
+//! Subject audit log writer with HMAC chain.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §3.
+//!
+//! Per-subject append-only JSONL at `_catalog/subjects/<id>.audit.jsonl`.
+//! Each row's `row_hmac` is HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac).
+//! The chain root is mirrored to the SubjectManifest.audit_log_chain_root
+//! field after each append, so verifiers can find the latest valid root
+//! from the manifest alone.
+//!
+//! Failures are non-blocking — audit-write errors are logged but do NOT
+//! propagate to the caller (per spec §3.2: "better to leak a row than
+//! block legitimate operations"). Operators MUST monitor the error count.
+//!
+//! Key loading: from a sealed file at the path the writer is constructed
+//! with. Tests use `with_inline_key` to inject a deterministic key.
+
+use bytes::Bytes;
+use hmac::{Hmac, Mac};
+use object_store::ObjectStore;
+use sha2::Sha256;
+use shared::types::SubjectAuditRow;
+use std::collections::HashMap;
+use std::sync::Arc;
+use storaged::ops;
+use tokio::sync::Mutex;
+
+const SUBJECT_PREFIX: &str = "_catalog/subjects";
+pub const GENESIS_HASH: &str = "GENESIS";
+
+type HmacSha256 = Hmac<Sha256>;
+
+/// Render a JSON value canonically: object keys sorted alphabetically,
+/// no insignificant whitespace, no trailing newline. Recursive — nested
+/// objects also sort their keys. Arrays preserve order (semantically
+/// significant). Numbers, strings, booleans, nulls pass through.
+///
+/// This is intentionally narrower than RFC 8785 (no number normalization,
+/// no UTF-8 NFC re-encoding). For our use — hashing structurally simple
+/// audit-row JSON — alphabetical key order is the only canonicalization
+/// that matters: it makes the byte sequence stable across struct field
+/// reordering and additive schema evolution.
+fn canonical_json(v: &serde_json::Value) -> Result<Vec<u8>, String> {
+    use std::collections::BTreeMap;
+    fn rewrite(v: &serde_json::Value) -> serde_json::Value {
+        match v {
+            serde_json::Value::Object(map) => {
+                let sorted: BTreeMap<String, serde_json::Value> = map
+                    .iter()
+                    .map(|(k, v)| (k.clone(), rewrite(v)))
+                    .collect();
+                serde_json::Value::Object(sorted.into_iter().collect())
+            }
+            serde_json::Value::Array(arr) => {
+                serde_json::Value::Array(arr.iter().map(rewrite).collect())
+            }
+            other => other.clone(),
+        }
+    }
+    let canonical = rewrite(v);
+    serde_json::to_vec(&canonical).map_err(|e| e.to_string())
+}
+
+/// Per-subject audit log writer. Holds the signing key, a small
+/// in-memory cache of latest chain hash per subject, and per-subject
+/// mutexes that serialize concurrent appends to the same subject's log.
+///
+/// **No Debug impl deliberately** — auto-deriving Debug on this struct
+/// would risk leaking the signing key into log lines. Tests that need
+/// `Result::unwrap_err` on a function returning `Result<Self, _>` must
+/// match on the result instead of using `.unwrap_err()`.
+///
+/// Concurrency: appends to DIFFERENT subjects run in parallel; appends
+/// to the SAME subject serialize via a per-candidate Mutex. Without
+/// per-subject locking the read-modify-write append_line race
+/// (concurrent reads of the same `prev_hash` + concurrent writes
+/// overwriting each other) silently corrupts the chain. Caught by
+/// 2026-05-03 opus scrum BLOCK on subject_audit.rs:172.
+#[derive(Clone)]
+pub struct SubjectAuditWriter {
+    store: Arc<dyn ObjectStore>,
+    /// HMAC signing key. Loaded once at construction. Never logged.
+    /// `Vec<u8>` not `String` — keys are bytes, not text.
+    key: Arc<Vec<u8>>,
+    /// In-memory cache of the latest chain hash per candidate_id.
+    /// Loaded lazily from the audit JSONL on first append per subject.
+    latest_hash: Arc<Mutex<HashMap<String, String>>>,
+    /// Per-subject mutexes. Append acquires the subject's lock before
+    /// the read-modify-write to serialize concurrent appends to the
+    /// same subject. Different subjects don't contend.
+    subject_locks: Arc<Mutex<HashMap<String, Arc<Mutex<()>>>>>,
+}
+
+impl SubjectAuditWriter {
+    /// Construct a writer that loads its signing key from a file at
+    /// `key_path`. The file MUST be mode 0400 owner-only on production
+    /// hosts (operator's responsibility — not enforced here because the
+    /// permission check would be a footgun in tests/dev).
+    pub async fn from_key_file(
+        store: Arc<dyn ObjectStore>,
+        key_path: &std::path::Path,
+    ) -> Result<Self, String> {
+        let key = tokio::fs::read(key_path)
+            .await
+            .map_err(|e| format!("read signing key from {}: {e}", key_path.display()))?;
+        if key.len() < 32 {
+            return Err(format!(
+                "signing key is {} bytes; recommend ≥32 bytes for HMAC-SHA256",
+                key.len()
+            ));
+        }
+        Ok(Self {
+            store,
+            key: Arc::new(key),
+            latest_hash: Arc::new(Mutex::new(HashMap::new())),
+            subject_locks: Arc::new(Mutex::new(HashMap::new())),
+        })
+    }
+
+    /// Construct a writer with an inline key. Intended for tests + bring-up.
+    /// Production should use `from_key_file`.
+    pub fn with_inline_key(store: Arc<dyn ObjectStore>, key: Vec<u8>) -> Self {
+        Self {
+            store,
+            key: Arc::new(key),
+            latest_hash: Arc::new(Mutex::new(HashMap::new())),
+            subject_locks: Arc::new(Mutex::new(HashMap::new())),
+        }
+    }
+
+    fn audit_key(candidate_id: &str) -> String {
+        let safe: String = candidate_id
+            .chars()
+            .map(|c| {
+                if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' {
+                    c
+                } else {
+                    '_'
+                }
+            })
+            .collect();
+        format!("{SUBJECT_PREFIX}/{}.audit.jsonl", safe)
+    }
+
+    /// Compute HMAC-SHA256(key, prev_hash_bytes || canonical_row_bytes).
+    /// Returns the HMAC as lowercase hex. Uses byte-level hex formatting
+    /// in a single pass (avoids per-byte format! allocations).
+    fn compute_hmac(&self, prev_hash: &str, canonical_row: &[u8]) -> String {
+        let mut mac = <HmacSha256 as Mac>::new_from_slice(&self.key)
+            .expect("HMAC accepts any key length");
+        mac.update(prev_hash.as_bytes());
+        mac.update(canonical_row);
+        let result = mac.finalize().into_bytes();
+        const HEX: &[u8; 16] = b"0123456789abcdef";
+        let mut s = String::with_capacity(64);
+        for byte in result {
+            s.push(HEX[(byte >> 4) as usize] as char);
+            s.push(HEX[(byte & 0x0f) as usize] as char);
+        }
+        s
+    }
+
+    /// Render an audit row as canonical JSON: keys sorted alphabetically,
+    /// no insignificant whitespace, row_hmac field excluded. This is the
+    /// byte sequence the HMAC is computed over.
+    ///
+    /// **Why a manual canonicalizer instead of `serde_json::to_vec`:**
+    /// serde_json serializes struct fields in DECLARATION order. If the
+    /// SubjectAuditRow struct ever gains/loses/reorders fields (schema
+    /// evolution), the bytes change even though the LOGICAL content is
+    /// the same. A verifier compiled against the new struct shape
+    /// re-serializes a stored row and gets DIFFERENT bytes than the
+    /// original write — chain breaks silently. Caught by 2026-05-03
+    /// opus scrum BLOCK on subject_audit.rs:108.
+    ///
+    /// Canonical-JSON (RFC 8785-ish, simplified for our scalar types)
+    /// fixes this by using alphabetical key order + minimal whitespace,
+    /// stable across struct evolution.
+    fn canonical_row_bytes(row: &SubjectAuditRow) -> Result<Vec<u8>, String> {
+        // Project to a generic Value, drop row_hmac, then re-serialize
+        // through a BTreeMap so keys come out alphabetical.
+        let mut v: serde_json::Value = serde_json::to_value(row)
+            .map_err(|e| format!("audit row to value: {e}"))?;
+        if let Some(obj) = v.as_object_mut() {
+            obj.remove("row_hmac");
+        }
+        canonical_json(&v).map_err(|e| format!("canonicalize: {e}"))
+    }
+
+    /// Get-or-create per-subject lock. Held across the read-modify-write
+    /// in append() so concurrent appends to the same subject serialize.
+    async fn lock_for(&self, candidate_id: &str) -> Arc<Mutex<()>> {
+        let mut locks = self.subject_locks.lock().await;
+        locks
+            .entry(candidate_id.to_string())
+            .or_insert_with(|| Arc::new(Mutex::new(())))
+            .clone()
+    }
+
+    /// Append one audit row for a subject. Computes the HMAC chain link,
+    /// writes to the per-subject JSONL, returns the new chain root (which
+    /// the caller MAY mirror to SubjectManifest.audit_log_chain_root).
+    ///
+    /// **Concurrency**: appends to the SAME subject serialize via a
+    /// per-subject Mutex. Appends to DIFFERENT subjects run in parallel.
+    /// Without the per-subject lock, the read-modify-write in
+    /// append_line races and silently corrupts the chain. (2026-05-03
+    /// opus scrum BLOCK on subject_audit.rs:172.)
+    ///
+    /// On error: logs and returns Err. Caller decides whether to propagate
+    /// (per spec §3.2 the gateway tool registry SHOULD log + drop).
+    pub async fn append(&self, mut row: SubjectAuditRow) -> Result<String, String> {
+        let cid = row.candidate_id.clone();
+        if cid.is_empty() {
+            return Err("audit row candidate_id is empty".into());
+        }
+
+        // Acquire the per-subject lock for the duration of this append.
+        let subject_lock = self.lock_for(&cid).await;
+        let _guard = subject_lock.lock().await;
+
+        // 1. Find prev_chain_hash. Cache hot path; cold path scans
+        //    the existing JSONL tail-style to find the last row.
+        let prev = {
+            let cache = self.latest_hash.lock().await;
+            cache.get(&cid).cloned()
+        };
+        let prev_hash = match prev {
+            Some(h) => h,
+            None => self.scan_latest_hash(&cid).await,
+        };
+        row.prev_chain_hash = prev_hash.clone();
+        row.row_hmac = String::new(); // Always cleared before HMAC computation.
+
+        // 2. Canonical-JSON the row (sorted keys, no whitespace, no
+        //    row_hmac), then HMAC. Stable across struct evolution.
+        let canon = Self::canonical_row_bytes(&row)?;
+        let new_hmac = self.compute_hmac(&prev_hash, &canon);
+        row.row_hmac = new_hmac.clone();
+
+        // 3. Append the row to the JSONL. The per-subject lock above
+        //    serializes the read-modify-write inside append_line.
+        let line = serde_json::to_vec(&row)
+            .map_err(|e| format!("serialize audit row: {e}"))?;
+        self.append_line(&cid, &line).await?;
+
+        // 4. Update cache.
+        self.latest_hash.lock().await.insert(cid, new_hmac.clone());
+
+        Ok(new_hmac)
+    }
+
+    /// Scan the audit JSONL for a subject and return the last row's
+    /// row_hmac. GENESIS if no log exists or it's empty.
+    /// Object-store-aware: works across LocalFileSystem + InMemory + S3.
+    async fn scan_latest_hash(&self, candidate_id: &str) -> String {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return GENESIS_HASH.to_string(), // file doesn't exist
+        };
+        // Find the last newline-delimited line that parses as a row.
+        let text = match std::str::from_utf8(&bytes) {
+            Ok(t) => t,
+            Err(_) => return GENESIS_HASH.to_string(),
+        };
+        for line in text.lines().rev() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            if let Ok(parsed) = serde_json::from_str::<SubjectAuditRow>(line) {
+                return parsed.row_hmac;
+            }
+        }
+        GENESIS_HASH.to_string()
+    }
+
+    /// Append a single newline-terminated line to the per-subject log.
+    /// Reads the existing content (may be empty), appends, writes back.
+    /// Object stores don't have native append; this is the standard
+    /// "read-modify-write the tail file" pattern.
+    async fn append_line(&self, candidate_id: &str, line: &[u8]) -> Result<(), String> {
+        let key = Self::audit_key(candidate_id);
+        let existing = ops::get(&self.store, &key).await.unwrap_or_default();
+        let mut buf = Vec::with_capacity(existing.len() + line.len() + 1);
+        buf.extend_from_slice(&existing);
+        if !buf.is_empty() && !buf.ends_with(b"\n") {
+            buf.push(b'\n');
+        }
+        buf.extend_from_slice(line);
+        buf.push(b'\n');
+        ops::put(&self.store, &key, Bytes::from(buf)).await
+    }
+
+    /// Read all audit rows for a subject, optionally filtered by time
+    /// window (inclusive bounds). Returns rows in append order. Used by
+    /// the /audit/subject/{id} endpoint to assemble the legal-tier
+    /// response. Window bounds are optional — `from=None`+`to=None`
+    /// returns the full log.
+    pub async fn read_rows_in_range(
+        &self,
+        candidate_id: &str,
+        from: Option<chrono::DateTime<chrono::Utc>>,
+        to: Option<chrono::DateTime<chrono::Utc>>,
+    ) -> Result<Vec<SubjectAuditRow>, String> {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return Ok(Vec::new()), // no audit log = empty result
+        };
+        let text = std::str::from_utf8(&bytes)
+            .map_err(|e| format!("audit log not utf-8: {e}"))?;
+        let mut out = Vec::new();
+        for line in text.lines() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            let row: SubjectAuditRow = match serde_json::from_str(line) {
+                Ok(r) => r,
+                Err(_) => continue, // skip unparseable lines (defensive)
+            };
+            if let Some(from) = from {
+                if row.ts < from { continue; }
+            }
+            if let Some(to) = to {
+                if row.ts > to { continue; }
+            }
+            out.push(row);
+        }
+        Ok(out)
+    }
+
+    /// Return the latest row_hmac in this subject's audit log, or None
+    /// if the log doesn't exist / has no rows. Cache-aware: hits the
+    /// in-memory cache when the writer has appended at least once for
+    /// this subject in the current process, otherwise scans the log
+    /// tail (single object_store read).
+    ///
+    /// Used by /audit/subject/{id} to echo the actual chain tip in the
+    /// response. Taking the tip from a windowed slice of rows would
+    /// lie when the window doesn't cover the latest row.
+    pub async fn chain_tip(&self, candidate_id: &str) -> Option<String> {
+        // Cache hit?
+        if let Some(h) = self.latest_hash.lock().await.get(candidate_id).cloned() {
+            return Some(h);
+        }
+        // Cold path — scan the log tail. Returns GENESIS for empty/missing
+        // logs; we coerce that to None so callers can distinguish "no audit
+        // activity yet" from "real chain tip exists".
+        let tip = self.scan_latest_hash(candidate_id).await;
+        if tip == GENESIS_HASH { None } else { Some(tip) }
+    }
+
+    /// Verify the full HMAC chain for a subject. Returns Ok(rows_verified)
+    /// or Err with the first chain break encountered.
+    ///
+    /// Special case: if no audit log exists (the subject has had no PII
+    /// access yet), returns Ok(0) — an empty log is trivially valid.
+    /// Callers that want to detect "log was deleted" should compare
+    /// against SubjectManifest.audit_log_chain_root: if the manifest's
+    /// chain root is non-empty/non-GENESIS but verify_chain returns 0,
+    /// the file was tampered.
+    pub async fn verify_chain(&self, candidate_id: &str) -> Result<usize, String> {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return Ok(0), // no log = 0 rows = trivially valid
+        };
+        let text = std::str::from_utf8(&bytes)
+            .map_err(|e| format!("audit log not utf-8: {e}"))?;
+        let mut prev = GENESIS_HASH.to_string();
+        let mut count = 0usize;
+        for (lineno, line) in text.lines().enumerate() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            let mut row: SubjectAuditRow = serde_json::from_str(line)
+                .map_err(|e| format!("line {} parse failed: {e}", lineno + 1))?;
+            if row.prev_chain_hash != prev {
+                return Err(format!(
+                    "chain break at line {}: prev_chain_hash={} expected={}",
+                    lineno + 1, row.prev_chain_hash, prev,
+                ));
+            }
+            let claimed = std::mem::take(&mut row.row_hmac);
+            let canon = Self::canonical_row_bytes(&row)
+                .map_err(|e| format!("canonicalize line {}: {e}", lineno + 1))?;
+            let recomputed = self.compute_hmac(&prev, &canon);
+            if recomputed != claimed {
+                return Err(format!(
+                    "hmac mismatch at line {}: stored={} recomputed={}",
+                    lineno + 1, claimed, recomputed,
+                ));
+            }
+            prev = claimed;
+            count += 1;
+        }
+        Ok(count)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use object_store::memory::InMemory;
+    use shared::types::AuditAccessor;
+
+    fn fixture_writer() -> SubjectAuditWriter {
+        SubjectAuditWriter::with_inline_key(
+            Arc::new(InMemory::new()),
+            // Deterministic 32-byte key for test reproducibility.
+            (0u8..32).collect(),
+        )
+    }
+
+    fn fixture_row(candidate_id: &str, fields: &[&str]) -> SubjectAuditRow {
+        SubjectAuditRow {
+            schema: "subject_audit.v1".into(),
+            ts: chrono::Utc::now(),
+            candidate_id: candidate_id.into(),
+            accessor: AuditAccessor {
+                kind: "gateway_lookup".into(),
+                daemon: "gateway".into(),
+                purpose: "fill_validation".into(),
+                trace_id: String::new(),
+            },
+            fields_accessed: fields.iter().map(|s| s.to_string()).collect(),
+            result: "success".into(),
+            prev_chain_hash: String::new(), // filled in by writer
+            row_hmac: String::new(),         // filled in by writer
+        }
+    }
+
+    #[tokio::test]
+    async fn first_append_uses_genesis_prev_hash() {
+        let w = fixture_writer();
+        let new_root = w.append(fixture_row("CAND-1", &["name"])).await.unwrap();
+        assert!(!new_root.is_empty());
+        assert_eq!(new_root.len(), 64); // SHA-256 hex
+        // After first append, the only row should reference GENESIS as prev.
+        let bytes = ops::get(&w.store, &SubjectAuditWriter::audit_key("CAND-1"))
+            .await.unwrap();
+        let text = std::str::from_utf8(&bytes).unwrap();
+        let row: SubjectAuditRow = serde_json::from_str(text.trim()).unwrap();
+        assert_eq!(row.prev_chain_hash, "GENESIS");
+        assert_eq!(row.row_hmac, new_root);
+    }
+
+    #[tokio::test]
+    async fn chain_links_each_append() {
+        let w = fixture_writer();
+        let h1 = w.append(fixture_row("CAND-2", &["name"])).await.unwrap();
+        let h2 = w.append(fixture_row("CAND-2", &["phone"])).await.unwrap();
+        let h3 = w.append(fixture_row("CAND-2", &["email"])).await.unwrap();
+        assert_ne!(h1, h2);
+        assert_ne!(h2, h3);
+        let count = w.verify_chain("CAND-2").await.unwrap();
+        assert_eq!(count, 3);
+    }
+
+    #[tokio::test]
+    async fn separate_subjects_have_independent_chains() {
+        let w = fixture_writer();
+        let _ = w.append(fixture_row("CAND-A", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-B", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-A", &["phone"])).await.unwrap();
+        assert_eq!(w.verify_chain("CAND-A").await.unwrap(), 2);
+        assert_eq!(w.verify_chain("CAND-B").await.unwrap(), 1);
+    }
+
+    #[tokio::test]
+    async fn tamper_detected_on_verify() {
+        let w = fixture_writer();
+        let _ = w.append(fixture_row("CAND-T", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-T", &["phone"])).await.unwrap();
+        // Mutate the file to flip a `result` field on the first row.
+        let key = SubjectAuditWriter::audit_key("CAND-T");
+        let bytes = ops::get(&w.store, &key).await.unwrap();
+        let mut text = std::str::from_utf8(&bytes).unwrap().to_string();
+        text = text.replacen("\"success\"", "\"denied\"", 1);
+        ops::put(&w.store, &key, text.into_bytes().into()).await.unwrap();
+        let err = w.verify_chain("CAND-T").await.unwrap_err();
+        assert!(err.contains("hmac mismatch"), "expected hmac mismatch, got: {err}");
+    }
+
+    #[tokio::test]
+    async fn cold_writer_picks_up_existing_chain() {
+        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        let key: Vec<u8> = (0u8..32).collect();
+        let w1 = SubjectAuditWriter::with_inline_key(store.clone(), key.clone());
+        let h1 = w1.append(fixture_row("CAND-COLD", &["name"])).await.unwrap();
+        // Drop w1's cache, simulate process restart.
+        drop(w1);
+        let w2 = SubjectAuditWriter::with_inline_key(store.clone(), key);
+        let h2 = w2.append(fixture_row("CAND-COLD", &["phone"])).await.unwrap();
+        assert_ne!(h1, h2);
+        assert_eq!(w2.verify_chain("CAND-COLD").await.unwrap(), 2);
+    }
+
+    #[tokio::test]
+    async fn empty_candidate_id_rejected() {
+        let w = fixture_writer();
+        let row = fixture_row("", &["name"]);
+        let err = w.append(row).await.unwrap_err();
+        assert!(err.contains("empty"), "got: {err}");
+    }
+
+    #[tokio::test]
+    async fn concurrent_appends_to_same_subject_serialize() {
+        // 2026-05-03 opus scrum BLOCK regression: without per-subject
+        // locking, concurrent appends raced and silently lost rows.
+        // This test spawns 50 parallel appends and asserts ALL 50
+        // land with an intact chain.
+        let w = Arc::new(fixture_writer());
+        let mut handles = Vec::new();
+        for i in 0..50 {
+            let w = w.clone();
+            handles.push(tokio::spawn(async move {
+                w.append(fixture_row("CAND-RACE", &[&format!("f{i}")])).await
+            }));
+        }
+        for h in handles {
+            h.await.unwrap().unwrap();
+        }
+        let count = w.verify_chain("CAND-RACE").await.unwrap();
+        assert_eq!(count, 50, "expected 50 rows; chain was corrupted");
+    }
+
+    #[tokio::test]
+    async fn concurrent_appends_to_different_subjects_run_parallel() {
+        let w = Arc::new(fixture_writer());
+        let mut handles = Vec::new();
+        for i in 0..20 {
+            let w = w.clone();
+            handles.push(tokio::spawn(async move {
+                w.append(fixture_row(&format!("CAND-PARA-{i}"), &["f"])).await
+            }));
+        }
+        for h in handles {
+            h.await.unwrap().unwrap();
+        }
+        // Each of the 20 subjects has exactly 1 row.
+        for i in 0..20 {
+            assert_eq!(w.verify_chain(&format!("CAND-PARA-{i}")).await.unwrap(), 1);
+        }
+    }
+
+    #[test]
+    fn canonical_json_sorts_keys_alphabetically() {
+        let v = serde_json::json!({"z": 1, "a": 2, "m": {"y": 1, "b": 2}});
+        let bytes = canonical_json(&v).unwrap();
+        let s = std::str::from_utf8(&bytes).unwrap();
+        // Keys should appear in alphabetical order at every nesting level.
+        let a_pos = s.find("\"a\"").unwrap();
+        let m_pos = s.find("\"m\"").unwrap();
+        let z_pos = s.find("\"z\"").unwrap();
+        assert!(a_pos < m_pos);
+        assert!(m_pos < z_pos);
+        // Nested object's keys also sorted.
+        let b_pos = s.find("\"b\"").unwrap();
+        let y_pos = s.find("\"y\"").unwrap();
+        assert!(b_pos < y_pos);
+    }
+
+    #[test]
+    fn canonical_json_arrays_preserve_order() {
+        let v = serde_json::json!({"k": ["c", "a", "b"]});
+        let bytes = canonical_json(&v).unwrap();
+        let s = std::str::from_utf8(&bytes).unwrap();
+        assert!(s.contains("\"c\",\"a\",\"b\""), "got: {s}");
+    }
+
+    #[tokio::test]
+    async fn key_too_short_rejected_via_file() {
+        // Write a 16-byte key file (under the 32-byte minimum).
+        let tmp = std::env::temp_dir().join(format!("sa_key_short_{}.bin", std::process::id()));
+        tokio::fs::write(&tmp, vec![0u8; 16]).await.unwrap();
+        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        // `unwrap_err` would require Debug on SubjectAuditWriter, which
+        // we deliberately don't impl (key-leak risk). Match instead.
+        match SubjectAuditWriter::from_key_file(store, &tmp).await {
+            Err(e) => assert!(e.contains("≥32 bytes"), "got: {e}"),
+            Ok(_) => panic!("expected too-short-key rejection"),
+        }
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+}

crates/gateway/Cargo.toml

@@ -13,6 +13,7 @@ ingestd = { path = "../ingestd" }
 vectord = { path = "../vectord" }
 journald = { path = "../journald" }
 truth = { path = "../truth" }
+validator = { path = "../validator" }
 tokio = { workspace = true }
 axum = { workspace = true }
 serde = { workspace = true }
@@ -30,3 +31,4 @@ tracing-opentelemetry = { workspace = true }
 arrow = { workspace = true }
 chrono = { workspace = true }
 reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
+toml = { workspace = true }

crates/gateway/src/bin/parity_extract_json.rs

@@ -0,0 +1,37 @@
+//! Cross-runtime parity helper for `extract_json`.
+//!
+//! Reads a single model-output string from stdin, runs the Rust
+//! extract_json, prints `{"matched": bool, "value": <object|null>}`
+//! to stdout as JSON. Exit 0 on success, exit 1 on internal error.
+//!
+//! The Go counterpart lives at
+//! `golangLAKEHOUSE/internal/validator/iterate.go::ExtractJSON`. The
+//! parity probe at
+//! `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`
+//! feeds the same fixtures through both and diffs the outputs.
+//!
+//! Usage:
+//!   echo '<raw model output>' | parity_extract_json
+//!   parity_extract_json <<< '...'
+
+use std::io::Read;
+
+fn main() {
+    let mut buf = String::new();
+    if let Err(e) = std::io::stdin().read_to_string(&mut buf) {
+        eprintln!("read stdin: {e}");
+        std::process::exit(1);
+    }
+    let result = gateway::v1::iterate::extract_json(&buf);
+    let body = serde_json::json!({
+        "matched": result.is_some(),
+        "value": result.unwrap_or(serde_json::Value::Null),
+    });
+    match serde_json::to_string(&body) {
+        Ok(s) => println!("{s}"),
+        Err(e) => {
+            eprintln!("serialize result: {e}");
+            std::process::exit(1);
+        }
+    }
+}

crates/gateway/src/bin/parity_session_log.rs

@@ -0,0 +1,71 @@
+//! Cross-runtime parity helper for `SessionRecord` JSON shape.
+//!
+//! Reads a fixture JSON on stdin, builds a `SessionRecord`, emits
+//! one JSONL row on stdout. Used by
+//! `golangLAKEHOUSE/scripts/cutover/parity/session_log_parity.sh`
+//! to verify the Rust gateway's session log shape stays byte-equal
+//! to the Go-side validatord's `validator.SessionRecord` (commit
+//! 1a3a82a in golangLAKEHOUSE).
+
+use gateway::v1::session_log::{SessionAttemptRecord, SessionRecord, SESSION_RECORD_SCHEMA};
+use serde::Deserialize;
+use std::io::Read;
+
+#[derive(Deserialize)]
+struct FixtureInput {
+    session_id: String,
+    kind: String,
+    model: String,
+    provider: String,
+    prompt: String,
+    iterations: u32,
+    max_iterations: u32,
+    final_verdict: String,
+    attempts: Vec<SessionAttemptRecord>,
+    #[serde(default)]
+    artifact: Option<serde_json::Value>,
+    #[serde(default)]
+    grounded_in_roster: Option<bool>,
+    duration_ms: u64,
+}
+
+fn main() {
+    let mut buf = String::new();
+    if let Err(e) = std::io::stdin().read_to_string(&mut buf) {
+        eprintln!("read stdin: {e}");
+        std::process::exit(1);
+    }
+    let input: FixtureInput = match serde_json::from_str(&buf) {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!("parse stdin: {e}");
+            std::process::exit(1);
+        }
+    };
+    let rec = SessionRecord {
+        schema: SESSION_RECORD_SCHEMA.to_string(),
+        session_id: input.session_id,
+        // Pinned timestamp so both runtimes' rows compare byte-equal
+        // when the test wrapper normalizes on `daemon` only.
+        timestamp: "2026-01-01T00:00:00+00:00".to_string(),
+        daemon: "gateway".to_string(),
+        kind: input.kind,
+        model: input.model,
+        provider: input.provider,
+        prompt: input.prompt,
+        iterations: input.iterations,
+        max_iterations: input.max_iterations,
+        final_verdict: input.final_verdict,
+        attempts: input.attempts,
+        artifact: input.artifact,
+        grounded_in_roster: input.grounded_in_roster,
+        duration_ms: input.duration_ms,
+    };
+    match serde_json::to_string(&rec) {
+        Ok(s) => println!("{s}"),
+        Err(e) => {
+            eprintln!("marshal: {e}");
+            std::process::exit(1);
+        }
+    }
+}

crates/gateway/src/execution_loop/mod.rs

@@ -212,6 +212,13 @@ impl ExecutionLoop {
             if let Action::ToolCall { tool, args, .. } = &exec_action {
                 match self.dispatch_tool(tool, args).await {
                     Ok(result) => {
+                        // Subject-audit hook (per Phase 1.6 SUBJECT_MANIFESTS spec
+                        // §3.2): every tool result that returns rows referencing a
+                        // subject must produce an audit row. Failures are logged
+                        // but never block the tool result — the spec is explicit:
+                        // "better to leak a row than block legitimate operations."
+                        self.audit_subject_hits_in(&result, tool).await;
+
                         let trimmed = trim_result(&result);
                         self.append(LogEntry::new(
                             turn, "executor", &executor_model, "tool_result", trimmed,
@@ -341,6 +348,68 @@ impl ExecutionLoop {
         self.log.push(e);
     }
 
+    /// Subject-audit hook: scan a tool result JSON for subject identifiers,
+    /// fire one audit row per (subject, fields) pair via the writer in
+    /// V1State.subject_audit. No-op if writer is None (audit disabled).
+    ///
+    /// Conventions (matches the backfill binary in catalogd/src/bin/
+    /// backfill_subjects.rs):
+    /// - "candidate_id":"X"   → audit candidate_id="X"
+    /// - "worker_id":42       → audit candidate_id="WORKER-42"
+    ///
+    /// Walks the result JSON recursively. For each object encountered
+    /// that contains a candidate_id or worker_id field, fires one audit
+    /// row capturing all the OTHER scalar field names as fields_accessed
+    /// (so the audit log records "this access surfaced name + phone for
+    /// candidate X" — sufficient for compliance review).
+    ///
+    /// Failures are logged but NEVER propagated. Per spec §3.2: an audit
+    /// gap is preferable to blocking the tool response.
+    async fn audit_subject_hits_in(&self, result: &serde_json::Value, tool: &str) {
+        let Some(audit) = self.state.subject_audit.clone() else {
+            return; // audit disabled — no-op
+        };
+        // Collect (candidate_id, fields_accessed) pairs from the JSON.
+        let mut hits: Vec<(String, Vec<String>)> = Vec::new();
+        collect_subject_hits(result, &mut hits);
+        if hits.is_empty() {
+            return;
+        }
+        // Determine result-state from the payload shape, NOT from the
+        // transport status. A tool can return Ok(Value) where the body
+        // itself is `{"error":"..."}` — hardcoding "success" here would
+        // mislead compliance review (opus WARN at execution_loop:401).
+        let result_state = audit_result_state(result);
+
+        // Sequential await — NOT tokio::spawn. The previous spawn-per-row
+        // shape fanned out concurrent writes that raced inside the
+        // per-subject append. The audit writer now serializes per-subject
+        // internally, but spawning would still flood those locks under
+        // burst load with no observable latency win. Inline await is
+        // simpler AND correct. (2026-05-03 opus scrum BLOCKs on
+        // execution_loop:391 + subject_audit:172.)
+        for (cid, fields) in hits {
+            let row = shared::types::SubjectAuditRow {
+                schema: "subject_audit.v1".into(),
+                ts: chrono::Utc::now(),
+                candidate_id: cid.clone(),
+                accessor: shared::types::AuditAccessor {
+                    kind: "gateway_lookup".into(),
+                    daemon: "gateway".into(),
+                    purpose: tool.to_string(),
+                    trace_id: String::new(), // TODO: thread X-Lakehouse-Trace-Id through
+                },
+                fields_accessed: fields,
+                result: result_state.clone(),
+                prev_chain_hash: String::new(),
+                row_hmac: String::new(),
+            };
+            if let Err(e) = audit.append(row).await {
+                tracing::warn!("subject audit write failed for {cid}: {e}");
+            }
+        }
+    }
+
     /// Dispatch: model name prefix → provider.
     /// Local path uses Phase 21 `generate_continuable` (auto-continuation,
     /// retry on empty thinking-model response). Cloud path hits
@@ -378,19 +447,21 @@ impl ExecutionLoop {
                 attempts = attempt + 1;
                 let req = ChatRequest {
                     model: model.to_string(),
-                    messages: vec![Message { role: "user".into(), content: prompt.to_string() }],
+                    messages: vec![Message::new_text("user", prompt.to_string())],
                     temperature: Some(temperature),
                     max_tokens: None,
                     stream: Some(false),
                     think: Some(think),
                     provider: Some("ollama_cloud".into()),
+                    tools: None,
+                    tool_choice: None,
                 };
                 let resp = crate::v1::ollama_cloud::chat(key, &req).await
                     .map_err(|e| format!("ollama_cloud: {e}"))?;
                 tokens_p = tokens_p.saturating_add(resp.usage.prompt_tokens);
                 tokens_c = tokens_c.saturating_add(resp.usage.completion_tokens);
-                let t = resp.choices.into_iter().next()
-                    .map(|c| c.message.content).unwrap_or_default();
+                let t: String = resp.choices.into_iter().next()
+                    .map(|c| c.message.text()).unwrap_or_default();
                 if !t.trim().is_empty() {
                     text = t;
                     break;
@@ -428,7 +499,7 @@ impl ExecutionLoop {
             lf.emit_chat(ChatTrace {
                 provider: provider.to_string(),
                 model: model.to_string(),
-                input: vec![Message { role: "user".into(), content: prompt.to_string() }],
+                input: vec![Message::new_text("user", prompt.to_string())],
                 output: text.clone(),
                 prompt_tokens,
                 completion_tokens,
@@ -438,6 +509,10 @@ impl ExecutionLoop {
                 start_time: start_time.to_rfc3339(),
                 end_time: end_time.to_rfc3339(),
                 latency_ms: elapsed_ms,
+                // Internal execution-loop traffic is its own top-level
+                // trace per call. If a future caller threads a parent
+                // trace into self.state, lift this to Some(parent_id).
+                parent_trace_id: None,
             });
         }
 
@@ -582,10 +657,10 @@ impl ExecutionLoop {
     /// Phase 20 step (8) — T3 overseer escalation.
     ///
     /// When the local executor/reviewer loop can't self-correct, call
-    /// the cloud overseer (`gpt-oss:120b` via Ollama Cloud) with (a)
-    /// the KB context — recent outcomes + prior corrections for this
-    /// sig_hash + task_class, across every profile that has run it —
-    /// and (b) the recent log tail. Its output is appended as a
+    /// the cloud overseer (`claude-opus-4-7` via OpenCode Zen) with
+    /// (a) the KB context — recent outcomes + prior corrections for
+    /// this sig_hash + task_class, across every profile that has run
+    /// it — and (b) the recent log tail. Its output is appended as a
     /// `system` role turn so the next executor generation sees it,
     /// AND written to `data/_kb/overseer_corrections.jsonl` so every
     /// future profile activation reads from the same learning pool.
@@ -593,67 +668,72 @@ impl ExecutionLoop {
     /// This is the "pipe to the overviewer" piece from 2026-04-23 —
     /// the overseer is now a first-class KB consumer AND producer, not
     /// a one-shot correction oracle.
+    ///
+    /// 2026-04-28: routed through OpenCode (Zen tier) for Claude Opus
+    /// 4.7. Frontier reasoning matters here because the overseer fires
+    /// only after local self-correction has failed twice — by that
+    /// point we need the strongest reasoning available, not the
+    /// cheapest token. Frequency is low so the Zen pay-per-token cost
+    /// stays bounded.
     async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> {
-        let Some(cloud_key) = self.state.ollama_cloud_key.clone() else {
-            return Err("OLLAMA_CLOUD_KEY not configured — skipping escalation".into());
-        };
-
+        // 2026-05-03: REVERTED to local-only per PRD line 70. Cloud
+        // overseer (opencode/claude-opus-4-7) was a recent addition that
+        // moved a hot-path call OFF the local Ollama runtime onto a paid
+        // cloud provider. Reverted to local Ollama (qwen3.5:latest).
+        // Cloud overseer can be re-enabled by setting LH_OVERSEER_CLOUD=1
+        // for development; production stays local.
         let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await;
         let prompt = build_overseer_prompt(&self.req, &kb, &self.log, reason);
 
         let started = std::time::Instant::now();
         let start_time = chrono::Utc::now();
         let chat_req = crate::v1::ChatRequest {
-            model: "gpt-oss:120b".to_string(),
-            messages: vec![crate::v1::Message {
-                role: "user".into(),
-                content: prompt.clone(),
-            }],
+            model: "qwen3.5:latest".to_string(),
+            messages: vec![crate::v1::Message::new_text("user", prompt.clone())],
             temperature: Some(0.1),
             max_tokens: None,
             stream: Some(false),
-            think: Some(true),    // overseer KEEPS thinking (Phase 20 rule)
-            provider: Some("ollama_cloud".into()),
+            think: Some(false),
+            provider: Some("ollama".into()),
+            tools: None,
+            tool_choice: None,
         };
-        let resp = crate::v1::ollama_cloud::chat(&cloud_key, &chat_req).await
-            .map_err(|e| format!("ollama_cloud: {e}"))?;
+        let resp = crate::v1::ollama::chat(&self.state.ai_client, &chat_req).await
+            .map_err(|e| format!("ollama overseer: {e}"))?;
         let latency_ms = started.elapsed().as_millis() as u64;
         let end_time = chrono::Utc::now();
-        let correction_text = resp.choices.into_iter().next()
-            .map(|c| c.message.content).unwrap_or_default();
+        let correction_text: String = resp.choices.into_iter().next()
+            .map(|c| c.message.text()).unwrap_or_default();
 
-        // Stamp per-task stats — cloud call counts against the same
-        // usage counter so `/v1/usage` shows cloud token spend too.
         self.stats.requests = self.stats.requests.saturating_add(1);
         self.stats.prompt_tokens = self.stats.prompt_tokens.saturating_add(resp.usage.prompt_tokens as u64);
         self.stats.completion_tokens = self.stats.completion_tokens.saturating_add(resp.usage.completion_tokens as u64);
         self.stats.total_tokens = self.stats.total_tokens.saturating_add(resp.usage.total_tokens as u64);
         self.stats.latency_ms = self.stats.latency_ms.saturating_add(latency_ms);
 
-        // Langfuse trace for the overseer call (same pipe that feeds
-        // the observer/KB, so this correction's cost lands in the KB
-        // too — closing the loop).
+        // Langfuse trace for the overseer call (local-only now).
         if let Some(lf) = &self.state.langfuse {
             use crate::v1::langfuse_trace::ChatTrace;
             lf.emit_chat(ChatTrace {
-                provider: "ollama_cloud".into(),
-                model: "gpt-oss:120b".into(),
-                input: vec![crate::v1::Message { role: "user".into(), content: prompt.clone() }],
+                provider: "ollama".into(),
+                model: "qwen3.5:latest".into(),
+                input: vec![crate::v1::Message::new_text("user", prompt.clone())],
                 output: correction_text.clone(),
                 prompt_tokens: resp.usage.prompt_tokens,
                 completion_tokens: resp.usage.completion_tokens,
                 temperature: Some(0.1),
                 max_tokens: None,
-                think: Some(true),
+                think: Some(false),
                 start_time: start_time.to_rfc3339(),
                 end_time: end_time.to_rfc3339(),
                 latency_ms,
+                parent_trace_id: None,
             });
         }
 
         // Append to the transcript so the next executor turn sees it.
         self.append(LogEntry::new(
-            turn, "system", "gpt-oss:120b", "overseer_correction",
+            turn, "system", "qwen3.5:latest", "overseer_correction",
             serde_json::json!({
                 "reason": reason,
                 "correction": correction_text,
@@ -675,7 +755,7 @@ impl ExecutionLoop {
             "task_class": self.req.task_class,
             "operation": self.req.operation,
             "reason": reason,
-            "model": "gpt-oss:120b",
+            "model": "qwen3.5:latest",
             "correction": correction_text,
             "applied_at_turn": turn,
             "kb_context_used": kb,
@@ -977,6 +1057,119 @@ async fn append_outcomes_row_at(
 
 /// PORT FROM orchestrator.ts:306-311. Cap `rows` at 20 entries and
 /// annotate the truncation so the executor sees it on the next turn
+/// Inspect a tool result JSON payload and return the appropriate
+/// audit `result` field per spec §3.2 enumeration:
+/// "success" | "denied" | "not_found" | "error".
+///
+/// Heuristic on the top-level object:
+/// - body has `"error"` key (truthy)             → "error"
+/// - body has `"denied"` key (truthy)            → "denied"
+/// - body has `"not_found"` key (truthy)         → "not_found"
+/// - body has `"status":"denied"`/etc            → that status
+/// - otherwise                                    → "success"
+///
+/// Conservative — only flags explicit signals, defaults to success.
+/// Keeps audit log honest about ambiguous payloads (an error inside
+/// rows[*] doesn't downgrade the whole call to "error" — that's a
+/// row-level concern, not a result-state concern).
+pub(crate) fn audit_result_state(v: &serde_json::Value) -> String {
+    if let Some(obj) = v.as_object() {
+        // Tighter than before (2026-05-03 opus scrum INFO at
+        // execution_loop:1085): only flag error/denied/not_found when
+        // the payload doesn't ALSO have a rows/data/results sibling.
+        // A successful tool whose response includes an "error" field
+        // describing one row's problem (out of many) shouldn't taint
+        // the whole call as "error" — that's a row-level concern, not
+        // a result-state concern.
+        let has_data_sibling = obj.contains_key("rows")
+            || obj.contains_key("data")
+            || obj.contains_key("results");
+        if !has_data_sibling {
+            if obj.get("error").map(is_truthy).unwrap_or(false) {
+                return "error".into();
+            }
+            if obj.get("denied").map(is_truthy).unwrap_or(false) {
+                return "denied".into();
+            }
+            if obj.get("not_found").map(is_truthy).unwrap_or(false) {
+                return "not_found".into();
+            }
+        }
+        // status is always authoritative — if a tool explicitly says
+        // status=denied that's not a row-level concern.
+        if let Some(s) = obj.get("status").and_then(|v| v.as_str()) {
+            match s {
+                "denied" | "not_found" | "error" => return s.into(),
+                _ => {}
+            }
+        }
+    }
+    "success".into()
+}
+
+fn is_truthy(v: &serde_json::Value) -> bool {
+    match v {
+        serde_json::Value::Null => false,
+        serde_json::Value::Bool(b) => *b,
+        serde_json::Value::String(s) => !s.is_empty(),
+        serde_json::Value::Array(a) => !a.is_empty(),
+        serde_json::Value::Object(o) => !o.is_empty(),
+        serde_json::Value::Number(n) => n.as_f64().map(|f| f != 0.0).unwrap_or(false),
+    }
+}
+
+/// Walk a tool-result JSON value recursively. For each object that
+/// contains a candidate_id (string) or worker_id (int/string), append
+/// (canonical_candidate_id, fields_in_object) to `out`. Conventions
+/// match the backfill binary: bare candidate_id stays as-is; worker_id
+/// gets prefixed with "WORKER-".
+///
+/// fields_in_object captures the OTHER scalar field names found in the
+/// same object — so the audit row records "this access surfaced
+/// {name, phone} for candidate X". The id field itself is NOT included
+/// in fields_accessed (it's the subject identifier, not PII data).
+pub(crate) fn collect_subject_hits(
+    v: &serde_json::Value,
+    out: &mut Vec<(String, Vec<String>)>,
+) {
+    match v {
+        serde_json::Value::Object(map) => {
+            // Detect a subject identifier in this object's direct fields.
+            let mut subject_id: Option<String> = None;
+            if let Some(cid) = map.get("candidate_id").and_then(|v| v.as_str()) {
+                if !cid.is_empty() { subject_id = Some(cid.to_string()); }
+            } else if let Some(wid) = map.get("worker_id") {
+                if let Some(s) = wid.as_str() {
+                    if !s.is_empty() { subject_id = Some(format!("WORKER-{s}")); }
+                } else if let Some(n) = wid.as_i64() {
+                    subject_id = Some(format!("WORKER-{n}"));
+                } else if let Some(n) = wid.as_u64() {
+                    subject_id = Some(format!("WORKER-{n}"));
+                }
+            }
+            if let Some(cid) = subject_id {
+                let fields: Vec<String> = map
+                    .keys()
+                    .filter(|k| *k != "candidate_id" && *k != "worker_id")
+                    .cloned()
+                    .collect();
+                out.push((cid, fields));
+            }
+            // Recurse — nested objects may carry additional subjects
+            // (e.g. join results with both candidate + recruiter rows).
+            for v in map.values() {
+                collect_subject_hits(v, out);
+            }
+        }
+        serde_json::Value::Array(arr) => {
+            for v in arr {
+                collect_subject_hits(v, out);
+            }
+        }
+        _ => {}
+    }
+}
+
 /// prompt — prevents a 1000-row hybrid_search result from wiping the
 /// context budget on a single tool call.
 fn trim_result(r: &serde_json::Value) -> serde_json::Value {
@@ -1855,4 +2048,123 @@ mod tests {
         let p = build_reviewer_prompt(&req, &log);
         assert!(!p.contains("HARD RULE"));
     }
+
+    // --- Subject-audit hook tests ---
+
+    #[test]
+    fn collect_subject_hits_finds_candidate_id_strings() {
+        let v = serde_json::json!({
+            "rows": [
+                {"candidate_id": "CAND-001", "name": "x", "phone": "y"},
+                {"candidate_id": "CAND-002", "name": "z"},
+            ]
+        });
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 2);
+        assert_eq!(hits[0].0, "CAND-001");
+        // Field order from JSON object is preserved in serde_json::Map.
+        assert!(hits[0].1.contains(&"name".to_string()));
+        assert!(hits[0].1.contains(&"phone".to_string()));
+        assert!(!hits[0].1.contains(&"candidate_id".to_string()));
+        assert_eq!(hits[1].0, "CAND-002");
+    }
+
+    #[test]
+    fn collect_subject_hits_prefixes_worker_id_int() {
+        let v = serde_json::json!({"rows": [{"worker_id": 42, "name": "Alice"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 1);
+        assert_eq!(hits[0].0, "WORKER-42");
+        assert_eq!(hits[0].1, vec!["name".to_string()]);
+    }
+
+    #[test]
+    fn collect_subject_hits_handles_worker_id_string() {
+        let v = serde_json::json!({"worker_id": "42", "name": "Bob"});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 1);
+        assert_eq!(hits[0].0, "WORKER-42");
+    }
+
+    #[test]
+    fn collect_subject_hits_recurses_through_nested_objects() {
+        let v = serde_json::json!({
+            "outer": {
+                "candidate_id": "CAND-A",
+                "linked": {
+                    "worker_id": 7,
+                    "role": "welder"
+                }
+            }
+        });
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 2);
+        let ids: Vec<&str> = hits.iter().map(|(c, _)| c.as_str()).collect();
+        assert!(ids.contains(&"CAND-A"));
+        assert!(ids.contains(&"WORKER-7"));
+    }
+
+    #[test]
+    fn collect_subject_hits_ignores_objects_without_id_fields() {
+        let v = serde_json::json!({"rows": [{"name": "x"}, {"role": "y"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 0);
+    }
+
+    #[test]
+    fn collect_subject_hits_skips_empty_id_strings() {
+        let v = serde_json::json!({"rows": [{"candidate_id": "", "name": "x"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 0);
+    }
+
+    #[test]
+    fn audit_result_state_default_is_success() {
+        let v = serde_json::json!({"rows": [{"candidate_id": "X"}]});
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_detects_error_payload() {
+        let v = serde_json::json!({"error": "not found"});
+        assert_eq!(audit_result_state(&v), "error");
+    }
+
+    #[test]
+    fn audit_result_state_detects_status_field() {
+        let v = serde_json::json!({"status": "denied", "rows": []});
+        assert_eq!(audit_result_state(&v), "denied");
+    }
+
+    #[test]
+    fn audit_result_state_ignores_falsy_signals() {
+        // Empty/null/false should not trigger error state.
+        let v = serde_json::json!({"error": null, "denied": false, "rows": [{"candidate_id":"X"}]});
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_does_not_classify_error_when_data_sibling_present() {
+        // 2026-05-03 opus scrum INFO at execution_loop:1085 regression:
+        // a tool that returns rows BUT also has an "error" field describing
+        // one bad row shouldn't taint the whole call as "error".
+        let v = serde_json::json!({
+            "rows": [{"candidate_id": "OK"}, {"candidate_id": "BAD"}],
+            "error": "1 row had a malformed certification"
+        });
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_status_is_authoritative_even_with_data() {
+        // Explicit status="denied" wins over data-sibling rule.
+        let v = serde_json::json!({"status": "denied", "rows": []});
+        assert_eq!(audit_result_state(&v), "denied");
+    }
 }

crates/gateway/src/lib.rs

@@ -0,0 +1,19 @@
+//! Library facade for the gateway crate so sub-binaries (e.g.
+//! `parity_extract_json`) can reuse the same modules the gateway
+//! binary uses.
+//!
+//! Added 2026-05-02 to support the cross-runtime parity probe at
+//! `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`.
+//! `extract_json` is the load-bearing public surface for that probe.
+//!
+//! main.rs still uses local `mod foo;` declarations independently —
+//! adding this file is purely additive (the binary's module tree is
+//! unchanged).
+
+pub mod access;
+pub mod access_service;
+pub mod auth;
+pub mod execution_loop;
+pub mod observability;
+pub mod tools;
+pub mod v1;

crates/gateway/src/main.rs

@@ -95,8 +95,35 @@ async fn main() {
         tracing::warn!("workspace rebuild: {e}");
     }
 
-    // AI sidecar client
-    let ai_client = aibridge::client::AiClient::new(&config.sidecar.url);
+    // AI sidecar clients — Phase 44 part 3 (2026-04-27).
+    //
+    // Two flavors of the same client:
+    // - `ai_client_direct` posts directly to ${sidecar}/generate. Used
+    //   inside the gateway by V1State + the legacy /ai proxy. These
+    //   call sites are themselves the implementation of /v1/chat
+    //   (or its sidecar shim), so routing them through /v1/chat
+    //   would self-loop.
+    // - `ai_client_observable` posts via ${gateway}/v1/chat with
+    //   provider="ollama". Used by vectord modules (autotune agent,
+    //   /vectors service) so their LLM calls land in /v1/usage and
+    //   Langfuse traces. Adds one localhost HTTP hop per call (~ms);
+    //   accepted for the observability gain.
+    //
+    // The gateway can call its own /v1/chat over localhost during
+    // boot's transient period because we don't fire any LLM calls
+    // until the listener is up — the observable client is just
+    // configured here, not exercised.
+    let ai_client_direct = aibridge::client::AiClient::new(&config.sidecar.url);
+    let gateway_self_url = format!("http://{}:{}", config.gateway.host, config.gateway.port);
+    let ai_client_observable = aibridge::client::AiClient::new_with_gateway(
+        &config.sidecar.url,
+        &gateway_self_url,
+    );
+    // Backwards-compat alias for the (many) existing references in this file.
+    // Defaults to direct so the existing wiring (V1State, /ai proxy)
+    // keeps its non-self-loop transport. New vectord wiring below
+    // explicitly uses ai_client_observable.
+    let ai_client = ai_client_direct.clone();
 
     // Vector service components — built before the router because both the
     // /vectors service AND ingestd need the agent handle to enqueue triggers.
@@ -134,7 +161,9 @@ async fn main() {
         agent_cfg,
         vectord::agent::AgentDeps {
             store: store.clone(),
-            ai_client: ai_client.clone(),
+            // Observable: autotune agent's LLM calls go through
+            // /v1/chat for /v1/usage + Langfuse visibility.
+            ai_client: ai_client_observable.clone(),
             catalog: registry.clone(),
             index_registry: index_reg.clone(),
             hnsw_store: hnsw.clone(),
@@ -189,7 +218,9 @@ async fn main() {
         }))
         .nest("/vectors", vectord::service::router(vectord::service::VectorState {
             store: store.clone(),
-            ai_client: ai_client.clone(),
+            // Observable: /vectors service's LLM calls (RAG, summary,
+            // playbook synthesis, etc.) flow through /v1/chat.
+            ai_client: ai_client_observable.clone(),
             job_tracker: vectord::jobs::JobTracker::new(),
             index_registry: index_reg.clone(),
             hnsw_store: hnsw,
@@ -224,6 +255,33 @@ async fn main() {
         // surface over the existing aibridge → Ollama path. Future
         // phases add provider adapters (39), routing engine (40),
         // session state (41), etc. All without changing this mount.
+        ;
+
+    // Build subject-audit writer BEFORE the V1State literal so the
+    // validate_workers wrapper can reference it. None = audit disabled
+    // (key file missing or unreadable); the gateway still serves PII
+    // paths but produces no audit trail per spec §3.2.
+    let subject_audit_writer: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>> = {
+        let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY")
+            .unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into());
+        match catalogd::subject_audit::SubjectAuditWriter::from_key_file(
+            store.clone(), std::path::Path::new(&key_path)
+        ).await {
+            Ok(w) => {
+                tracing::info!("v1: subject audit log enabled (key: {})", key_path);
+                Some(std::sync::Arc::new(w))
+            }
+            Err(e) => {
+                tracing::warn!(
+                    "v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)",
+                    e
+                );
+                None
+            }
+        }
+    };
+
+    app = app
         .nest("/v1", v1::router(v1::V1State {
             ai_client: ai_client.clone(),
             usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
@@ -271,6 +329,63 @@ async fn main() {
                 }
                 k
             },
+            kimi_key: {
+                // Direct Kimi For Coding (api.kimi.com) — bypasses the
+                // broken-upstream kimi-k2:1t and OpenRouter rate caps.
+                // Key from /etc/lakehouse/kimi.env (KIMI_API_KEY=sk-kimi-…).
+                let k = v1::kimi::resolve_kimi_key();
+                if k.is_some() {
+                    tracing::info!("v1: Kimi key loaded — /v1/chat provider=kimi enabled (model=kimi-for-coding)");
+                } else {
+                    tracing::debug!("v1: no Kimi key — provider=kimi will 503");
+                }
+                k
+            },
+            opencode_key: {
+                // OpenCode GO multi-vendor gateway — Claude Opus 4.7,
+                // GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+                // Qwen + free-tier. Key from /etc/lakehouse/opencode.env.
+                let k = v1::opencode::resolve_opencode_key();
+                if k.is_some() {
+                    tracing::info!("v1: OpenCode key loaded — /v1/chat provider=opencode enabled (40 models)");
+                } else {
+                    tracing::debug!("v1: no OpenCode key — provider=opencode will 503");
+                }
+                k
+            },
+            validate_workers: {
+                // Load workers_500k.parquet snapshot for /v1/validate.
+                // Path overridable via LH_WORKERS_PARQUET env. Missing
+                // file is non-fatal — validators run schema/PII checks
+                // unaffected; only worker-existence checks fail clean.
+                let path_str = std::env::var("LH_WORKERS_PARQUET")
+                    .unwrap_or_else(|_| "/home/profit/lakehouse/data/datasets/workers_500k.parquet".into());
+                let path = std::path::Path::new(&path_str);
+                let raw: std::sync::Arc<dyn validator::WorkerLookup> = if path.exists() {
+                    match validator::staffing::parquet_lookup::load_workers_parquet(path) {
+                        Ok(lookup) => {
+                            tracing::info!("v1: workers parquet loaded from {} — /v1/validate worker-existence checks enabled", path_str);
+                            lookup
+                        }
+                        Err(e) => {
+                            tracing::warn!("v1: workers parquet at {} unreadable ({e}) — /v1/validate worker-existence checks will fail Consistency", path_str);
+                            std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
+                        }
+                    }
+                } else {
+                    tracing::warn!("v1: workers parquet at {} not found — /v1/validate worker-existence checks will fail Consistency", path_str);
+                    std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
+                };
+                // Wrap with subject-audit decorator. When subject_audit_writer
+                // is None, the wrapper is a transparent passthrough — no
+                // audit, no overhead. When Some, every find() call fires an
+                // audit row via the per-subject HMAC chain. (Step 5 of
+                // SUBJECT_MANIFESTS_ON_CATALOGD spec.)
+                std::sync::Arc::new(v1::auditing_worker_lookup::AuditingWorkerLookup::new(
+                    raw,
+                    subject_audit_writer.clone(),
+                ))
+            },
             // Phase 40 early deliverable — Langfuse trace emitter.
             // Defaults match mcp-server/tracing.ts conventions so
             // gateway traces land in the same staffing project.
@@ -283,8 +398,75 @@ async fn main() {
                 }
                 c
             },
+            // Coordinator session JSONL — one row per /v1/iterate
+            // session for offline DuckDB analysis. Cross-runtime
+            // parity with Go-side validatord (commit 1a3a82a).
+            session_log: {
+                let path = &config.gateway.session_log_path;
+                let s = v1::session_log::SessionLogger::from_path(path);
+                if s.is_some() {
+                    tracing::info!(
+                        "v1: session log enabled — coordinator sessions written to {}",
+                        path
+                    );
+                } else {
+                    tracing::info!("v1: session log disabled (set [gateway].session_log_path to enable)");
+                }
+                s
+            },
+            // subject_audit was constructed BEFORE the V1State literal
+            // (see `subject_audit_writer` above) so validate_workers
+            // could wrap with it. Reuse the same Arc here.
+            subject_audit: subject_audit_writer.clone(),
         }));
 
+    // Step 6 of SUBJECT_MANIFESTS_ON_CATALOGD spec — /audit/subject/{id}
+    // legal-tier endpoint. Only mounted when BOTH the audit writer AND
+    // the legal token file are present. Disabled-by-default keeps the
+    // audit surface from accidentally serving without proper credentials
+    // in dev / bring-up environments.
+    if let Some(writer) = subject_audit_writer.clone() {
+        let legal_token_path = std::env::var("LH_LEGAL_AUDIT_TOKEN_FILE")
+            .unwrap_or_else(|_| "/etc/lakehouse/legal_audit.token".into());
+        let audit_state = catalogd::audit_endpoint::AuditEndpointState::new(
+            registry.clone(),
+            writer.clone(),
+            std::path::Path::new(&legal_token_path),
+        ).await;
+        app = app.nest("/audit", catalogd::audit_endpoint::router(audit_state));
+        tracing::info!("audit endpoint mounted at /audit (legal token: {})", legal_token_path);
+
+        // Phase 1.6 Gate 3 — biometric photo upload endpoint. Same
+        // legal-token gate as /audit (writes to legal-tier storage are
+        // privileged). Storage root defaults to ./data/biometric/uploads
+        // unless overridden via LH_BIOMETRIC_STORAGE_ROOT (operators on
+        // multi-host deployments will want a dedicated mount point).
+        let biometric_storage_root: std::path::PathBuf = std::env::var("LH_BIOMETRIC_STORAGE_ROOT")
+            .map(std::path::PathBuf::from)
+            .unwrap_or_else(|_| std::path::PathBuf::from("./data/biometric/uploads"));
+        // consent_versions allowlist — counsel-tier deployment safety.
+        // Defaults to /etc/lakehouse/consent_versions.json; override
+        // via LH_CONSENT_VERSIONS_FILE. Missing file → permissive
+        // mode (any non-empty hash accepted, the v1 behavior).
+        let consent_versions_path: std::path::PathBuf = std::env::var("LH_CONSENT_VERSIONS_FILE")
+            .map(std::path::PathBuf::from)
+            .unwrap_or_else(|_| std::path::PathBuf::from("/etc/lakehouse/consent_versions.json"));
+        let biometric_state = catalogd::biometric_endpoint::BiometricEndpointState::new(
+            registry.clone(),
+            writer,
+            std::path::Path::new(&legal_token_path),
+            biometric_storage_root.clone(),
+        ).await
+            .with_consent_versions(&consent_versions_path).await;
+        app = app.nest("/biometric", catalogd::biometric_endpoint::router(biometric_state));
+        tracing::info!(
+            "biometric endpoint mounted at /biometric (storage_root: {}, legal token: {}, consent_versions file: {})",
+            biometric_storage_root.display(), legal_token_path, consent_versions_path.display()
+        );
+    } else {
+        tracing::warn!("/audit + /biometric endpoints NOT mounted — subject_audit writer is None (no signing key)");
+    }
+
     // Auth middleware (if enabled) — P5-001 fix 2026-04-23:
     // previously only inserted the ApiKey as an extension and never layered
     // the middleware, so auth.enabled=true enforced nothing. Now wraps the

crates/gateway/src/v1/auditing_worker_lookup.rs

@@ -0,0 +1,194 @@
+//! Validator WorkerLookup decorator that fires per-find audit rows.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5.
+//!
+//! Wraps an inner `dyn WorkerLookup`. Every `find(candidate_id)` call
+//! produces one `SubjectAuditRow` via the bundled `SubjectAuditWriter`,
+//! tagged with `purpose="validator_worker_lookup"`. Audit-write failures
+//! are logged only — never propagated to the caller (per spec §3.2:
+//! audit-side gaps are preferable to blocking validation).
+//!
+//! **Why a decorator and not a trait extension**: `WorkerLookup` is a
+//! sync trait by design (validator's contract is "in-memory snapshot,
+//! no per-call I/O"). Audit writes are async. The decorator bridges
+//! them by spawning the audit append onto the current Tokio runtime
+//! handle. Per-subject sequencing inside the writer ensures the chain
+//! stays intact even with spawned writes (regression-tested at
+//! catalogd::subject_audit::tests::concurrent_appends_to_same_subject_serialize).
+
+use catalogd::subject_audit::SubjectAuditWriter;
+use shared::types::{AuditAccessor, SubjectAuditRow};
+use std::sync::Arc;
+use validator::{WorkerLookup, WorkerRecord};
+
+pub struct AuditingWorkerLookup {
+    inner: Arc<dyn WorkerLookup>,
+    /// None = audit disabled. find() still runs; just no audit row.
+    audit: Option<Arc<SubjectAuditWriter>>,
+    /// Cached Tokio handle so the sync find() path can spawn audit
+    /// writes onto the right runtime. None at construction = audit
+    /// is no-op (no runtime context to dispatch into).
+    runtime_handle: Option<tokio::runtime::Handle>,
+}
+
+impl AuditingWorkerLookup {
+    /// Wrap an inner lookup with optional audit. If `audit` is None,
+    /// behavior is identical to the inner — no overhead, no audit.
+    /// MUST be constructed inside a Tokio runtime context (uses
+    /// `Handle::try_current` to capture the handle).
+    pub fn new(inner: Arc<dyn WorkerLookup>, audit: Option<Arc<SubjectAuditWriter>>) -> Self {
+        let runtime_handle = if audit.is_some() {
+            tokio::runtime::Handle::try_current().ok()
+        } else {
+            None
+        };
+        if audit.is_some() && runtime_handle.is_none() {
+            tracing::warn!(
+                "AuditingWorkerLookup: audit enabled but no Tokio runtime — audit writes disabled"
+            );
+        }
+        Self { inner, audit, runtime_handle }
+    }
+}
+
+impl WorkerLookup for AuditingWorkerLookup {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
+        let result = self.inner.find(candidate_id);
+        // Fire audit AFTER the find so the result-state field reflects
+        // the actual outcome ("success" if found, "not_found" if not).
+        if let (Some(audit), Some(handle)) = (self.audit.clone(), self.runtime_handle.clone()) {
+            let cid = candidate_id.to_string();
+            let result_state = if result.is_some() { "success" } else { "not_found" };
+            // Single field: "exists" — validator only consumes existence
+            // + role/city/state from the WorkerRecord. We don't enumerate
+            // every WorkerRecord field as fields_accessed because that
+            // would imply they were USED by the caller; the validator
+            // only proves existence + then reads policy fields downstream.
+            let row = SubjectAuditRow {
+                schema: "subject_audit.v1".into(),
+                ts: chrono::Utc::now(),
+                candidate_id: cid.clone(),
+                accessor: AuditAccessor {
+                    kind: "validator_lookup".into(),
+                    daemon: "gateway".into(),
+                    purpose: "validator_worker_lookup".into(),
+                    trace_id: String::new(),
+                },
+                fields_accessed: vec!["exists".into()],
+                result: result_state.into(),
+                prev_chain_hash: String::new(),
+                row_hmac: String::new(),
+            };
+            // Spawn fire-and-forget. Chain integrity is preserved by the
+            // writer's per-subject Mutex (regression-tested), so spawning
+            // is safe. Audit-write errors are logged only.
+            handle.spawn(async move {
+                if let Err(e) = audit.append(row).await {
+                    tracing::warn!("validator audit write failed for {cid}: {e}");
+                }
+            });
+        }
+        result
+    }
+
+    fn len(&self) -> usize {
+        self.inner.len()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use catalogd::subject_audit::SubjectAuditWriter;
+    use object_store::memory::InMemory;
+    use validator::InMemoryWorkerLookup;
+
+    fn fixture_inner_with(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn fixture_record(candidate_id: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: candidate_id.into(),
+            name: "Test".into(),
+            status: "active".into(),
+            city: Some("Chicago".into()),
+            state: Some("IL".into()),
+            role: Some("Welder".into()),
+            blacklisted_clients: vec![],
+        }
+    }
+
+    #[tokio::test]
+    async fn find_existing_subject_writes_success_audit_row() {
+        let inner = fixture_inner_with(vec![fixture_record("CAND-EXIST")]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        let r = lookup.find("CAND-EXIST");
+        assert!(r.is_some());
+        // Audit is spawned — give it a tick to land.
+        tokio::task::yield_now().await;
+        tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+        let count = writer.verify_chain("CAND-EXIST").await.unwrap();
+        assert_eq!(count, 1);
+    }
+
+    #[tokio::test]
+    async fn find_missing_subject_writes_not_found_audit_row() {
+        let inner = fixture_inner_with(vec![]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        let r = lookup.find("CAND-PHANTOM");
+        assert!(r.is_none());
+        tokio::task::yield_now().await;
+        tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+        // verify_chain returns the row count for that subject (1 row,
+        // result="not_found"). We don't assert on result text here —
+        // that's covered by the hand-rolled JSON inspection below.
+        assert_eq!(writer.verify_chain("CAND-PHANTOM").await.unwrap(), 1);
+    }
+
+    #[tokio::test]
+    async fn audit_disabled_means_no_writes_no_overhead() {
+        let inner = fixture_inner_with(vec![fixture_record("CAND-A")]);
+        let lookup = AuditingWorkerLookup::new(inner, None);
+        // Just confirm find() still works with no audit — no panic, no
+        // runtime-handle requirement.
+        assert!(lookup.find("CAND-A").is_some());
+        assert!(lookup.find("CAND-MISSING").is_none());
+        assert_eq!(lookup.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn many_finds_to_same_subject_produce_intact_chain() {
+        // Same regression as catalogd's concurrent test, but exercised
+        // through the validator wrapper path. 30 sequential finds on
+        // the SAME subject should produce an intact 30-row chain.
+        let inner = fixture_inner_with(vec![fixture_record("CAND-CHAIN")]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        for _ in 0..30 {
+            let _ = lookup.find("CAND-CHAIN");
+        }
+        // Wait for all spawns to finish — the runtime is currently
+        // single-threaded test runtime, so a few yields drains.
+        for _ in 0..50 {
+            tokio::task::yield_now().await;
+        }
+        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
+        let count = writer.verify_chain("CAND-CHAIN").await.unwrap();
+        assert_eq!(count, 30, "chain corrupted under sequential spawn load");
+    }
+}

crates/gateway/src/v1/claude.rs

@@ -46,12 +46,12 @@ pub async fn chat(
     let mut msgs: Vec<AnMessage> = Vec::new();
     for m in &req.messages {
         if m.role == "system" {
-            system_parts.push(m.content.clone());
+            system_parts.push(m.text());
         } else {
             // Anthropic expects strictly "user" or "assistant"; anything
             // else we normalize to "user".
             let role = if m.role == "assistant" { "assistant" } else { "user" };
-            msgs.push(AnMessage { role: role.to_string(), content: m.content.clone() });
+            msgs.push(AnMessage { role: role.to_string(), content: m.text() });
         }
     }
     let system = if system_parts.is_empty() {
@@ -99,7 +99,7 @@ pub async fn chat(
         .unwrap_or_default();
 
     let prompt_tokens = parsed.usage.as_ref().map(|u| u.input_tokens).unwrap_or_else(|| {
-        let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
         ((chars + 3) / 4) as u32
     });
     let completion_tokens = parsed.usage.as_ref().map(|u| u.output_tokens).unwrap_or_else(|| {
@@ -123,7 +123,7 @@ pub async fn chat(
         model,
         choices: vec![Choice {
             index: 0,
-            message: Message { role: "assistant".into(), content: text },
+            message: Message::new_text("assistant", text),
             finish_reason: parsed.stop_reason.unwrap_or_else(|| "stop".into()),
         }],
         usage: UsageBlock {

crates/gateway/src/v1/gemini.rs

@@ -52,7 +52,7 @@ pub async fn chat(
         };
         contents.push(GmContent {
             role: role.to_string(),
-            parts: vec![GmPart { text: m.content.clone() }],
+            parts: vec![GmPart { text: m.text() }],
         });
     }
 
@@ -98,7 +98,7 @@ pub async fn chat(
     let prompt_tokens = parsed.usage_metadata.as_ref()
         .map(|u| u.prompt_token_count)
         .unwrap_or_else(|| {
-            let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
+            let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
             ((chars + 3) / 4) as u32
         });
     let completion_tokens = parsed.usage_metadata.as_ref()
@@ -122,7 +122,7 @@ pub async fn chat(
         model,
         choices: vec![Choice {
             index: 0,
-            message: Message { role: "assistant".into(), content: text },
+            message: Message::new_text("assistant", text),
             finish_reason: candidate.finish_reason.unwrap_or_else(|| "stop".into()),
         }],
         usage: UsageBlock {

crates/gateway/src/v1/iterate.rs

@@ -0,0 +1,543 @@
+//! /v1/iterate — the Phase 43 PRD's "generate → validate → correct → retry" loop.
+//!
+//! Closes the "0→85% with iteration" thesis structurally. A caller
+//! posts a prompt + artifact kind + validation context; the gateway:
+//!   1. Generates a JSON artifact via /v1/chat (any provider/model)
+//!   2. Extracts the JSON object from the model output
+//!   3. Validates via /v1/validate (FillValidator / EmailValidator /
+//!      PlaybookValidator with the shared WorkerLookup)
+//!   4. On ValidationError, appends the error to the prompt and
+//!      retries up to `max_iterations` (default 3)
+//!   5. Returns the accepted artifact + Report on success, OR the
+//!      attempt history + final error on max-iter exhaustion
+//!
+//! Internal calls go via HTTP loopback to localhost:gateway_port so
+//! the same /v1/usage tracking and Langfuse traces apply. A small
+//! latency cost (~1-3ms per loopback hop) for clean separation of
+//! concerns and observability.
+//!
+//! 2026-04-27 Phase 43 v3 part 3: this endpoint makes the iteration
+//! loop a first-class lakehouse capability rather than a per-caller
+//! re-implementation. Staffing executors, agent loops, and future
+//! validators all reach the same code path.
+
+use axum::{extract::State, http::{HeaderMap, StatusCode}, response::IntoResponse, Json};
+use serde::{Deserialize, Serialize};
+
+const DEFAULT_MAX_ITERATIONS: u32 = 3;
+const LOOPBACK_TIMEOUT_SECS: u64 = 240;
+
+/// Header name used to propagate a Langfuse parent trace id across
+/// daemon boundaries. Matches Go's `shared.TraceIDHeader` constant
+/// byte-for-byte (commit d6d2fdf in golangLAKEHOUSE) — same wire
+/// format means a Go caller can hit Rust's /v1/iterate (or vice
+/// versa) and the resulting Langfuse trees nest correctly.
+pub const TRACE_ID_HEADER: &str = "x-lakehouse-trace-id";
+
+#[derive(Deserialize)]
+pub struct IterateRequest {
+    /// "fill" | "email" | "playbook" — picks which validator runs.
+    pub kind: String,
+    /// The prompt to seed generation. Validation errors from prior
+    /// attempts are appended on retry.
+    pub prompt: String,
+    /// Provider/model passed through to /v1/chat. e.g. "ollama_cloud"
+    /// + "kimi-k2.6", or "opencode" + "claude-haiku-4-5".
+    pub provider: String,
+    pub model: String,
+    /// Optional system prompt — sent to /v1/chat as the system message.
+    #[serde(default)]
+    pub system: Option<String>,
+    /// Validation context (target_count, city, state, role, client_id
+    /// for fills; candidate_id for emails). Forwarded to /v1/validate.
+    #[serde(default)]
+    pub context: Option<serde_json::Value>,
+    /// Cap on iteration count. Defaults to 3 per the Phase 43 PRD.
+    #[serde(default)]
+    pub max_iterations: Option<u32>,
+    /// Forwarded to /v1/chat. Defaults to 0.2 if unset.
+    #[serde(default)]
+    pub temperature: Option<f64>,
+    /// Forwarded to /v1/chat. Defaults to 4096 if unset.
+    #[serde(default)]
+    pub max_tokens: Option<u32>,
+}
+
+#[derive(Serialize)]
+pub struct IterateAttempt {
+    pub iteration: u32,
+    pub raw: String,
+    pub status: AttemptStatus,
+}
+
+#[derive(Serialize)]
+#[serde(tag = "kind", rename_all = "snake_case")]
+pub enum AttemptStatus {
+    /// Model output didn't contain extractable JSON.
+    NoJson,
+    /// JSON extracted but failed validation; carries the error.
+    ValidationFailed { error: serde_json::Value },
+    /// Validation passed (last attempt's terminal status).
+    Accepted,
+}
+
+#[derive(Serialize)]
+pub struct IterateResponse {
+    pub artifact: serde_json::Value,
+    pub validation: serde_json::Value,
+    pub iterations: u32,
+    pub history: Vec<IterateAttempt>,
+    /// Echoes the resolved trace id (caller-forwarded header, body
+    /// field, langfuse-middleware mint, or local fallback). Operators
+    /// pivot from this id straight into Langfuse + the
+    /// coordinator_sessions.jsonl join key. Cross-runtime parity with
+    /// Go's `validator.IterateResponse` (commit 6847bbc in
+    /// golangLAKEHOUSE).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub trace_id: Option<String>,
+}
+
+#[derive(Serialize)]
+pub struct IterateFailure {
+    pub error: String,
+    pub iterations: u32,
+    pub history: Vec<IterateAttempt>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub trace_id: Option<String>,
+}
+
+pub async fn iterate(
+    State(state): State<super::V1State>,
+    headers: HeaderMap,
+    Json(req): Json<IterateRequest>,
+) -> impl IntoResponse {
+    let max_iter = req.max_iterations.unwrap_or(DEFAULT_MAX_ITERATIONS).max(1);
+    let temperature = req.temperature.unwrap_or(0.2);
+    let max_tokens = req.max_tokens.unwrap_or(4096);
+    let mut history: Vec<IterateAttempt> = Vec::with_capacity(max_iter as usize);
+    let mut attempt_records: Vec<super::session_log::SessionAttemptRecord> = Vec::with_capacity(max_iter as usize);
+    let mut current_prompt = req.prompt.clone();
+
+    // Resolve the parent Langfuse trace id. Caller-forwarded header
+    // wins (cross-daemon tree linkage); otherwise mint a fresh id so
+    // the iterate session is its own tree. Same shape as the Go-side
+    // validatord trace propagation.
+    let trace_id: String = headers
+        .get(TRACE_ID_HEADER)
+        .and_then(|v| v.to_str().ok())
+        .filter(|s| !s.is_empty())
+        .map(|s| s.to_string())
+        .unwrap_or_else(new_trace_id);
+
+    let client = match reqwest::Client::builder()
+        .timeout(std::time::Duration::from_secs(LOOPBACK_TIMEOUT_SECS))
+        .build() {
+        Ok(c) => c,
+        Err(e) => {
+            // Even infrastructure failures get a session row so a
+            // missing /v1/iterate event never silently disappears
+            // from the longitudinal log.
+            write_infra_error(&state, &req, &trace_id, max_iter, 0, format!("client build: {e}")).await;
+            return (StatusCode::INTERNAL_SERVER_ERROR, format!("client build: {e}")).into_response();
+        }
+    };
+    // Self-loopback to the gateway port. Carries gateway internal
+    // calls through /v1/chat + /v1/validate so /v1/usage tracks them.
+    let gateway = "http://127.0.0.1:3100";
+    let t0 = std::time::Instant::now();
+
+    for iteration in 0..max_iter {
+        let attempt_started = chrono::Utc::now();
+        // ── Generate ──
+        let mut messages = Vec::with_capacity(2);
+        if let Some(sys) = &req.system {
+            messages.push(serde_json::json!({"role": "system", "content": sys}));
+        }
+        messages.push(serde_json::json!({"role": "user", "content": current_prompt}));
+        let chat_body = serde_json::json!({
+            "messages": messages,
+            "provider": req.provider,
+            "model": req.model,
+            "temperature": temperature,
+            "max_tokens": max_tokens,
+        });
+        let raw = match call_chat(&client, gateway, &chat_body, &trace_id).await {
+            Ok(r) => r,
+            Err(e) => {
+                write_infra_error(&state, &req, &trace_id, max_iter, t0.elapsed().as_millis() as u64, format!("/v1/chat hop failed at iter {iteration}: {e}")).await;
+                return (StatusCode::BAD_GATEWAY, format!("/v1/chat hop failed at iter {iteration}: {e}")).into_response();
+            }
+        };
+
+        // ── Extract JSON ──
+        let artifact = match extract_json(&raw) {
+            Some(a) => a,
+            None => {
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "no_json", None,
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::NoJson,
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "no_json".to_string(),
+                    error: None,
+                    span_id,
+                });
+                current_prompt = format!(
+                    "{}\n\nYour previous attempt did not contain a JSON object. Reply with ONLY a valid JSON object matching the requested artifact shape.",
+                    req.prompt,
+                );
+                continue;
+            }
+        };
+
+        // ── Validate ──
+        let validate_body = serde_json::json!({
+            "kind": req.kind,
+            "artifact": artifact,
+            "context": req.context.clone().unwrap_or(serde_json::Value::Null),
+        });
+        match call_validate(&client, gateway, &validate_body, &trace_id).await {
+            Ok(report) => {
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "accepted", None,
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::Accepted,
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "accepted".to_string(),
+                    error: None,
+                    span_id,
+                });
+                let duration_ms = t0.elapsed().as_millis() as u64;
+                let grounded = grounded_in_roster(&state, &req.kind, &artifact);
+                write_session_accepted(&state, &req, &trace_id, iteration + 1, max_iter, attempt_records, &artifact, grounded, duration_ms).await;
+                return (StatusCode::OK, Json(IterateResponse {
+                    artifact,
+                    validation: report,
+                    iterations: iteration + 1,
+                    history,
+                    trace_id: Some(trace_id.clone()),
+                })).into_response();
+            }
+            Err(err) => {
+                let err_summary = err.to_string();
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "validation_failed",
+                    Some(err_summary.clone()),
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::ValidationFailed {
+                        error: serde_json::to_value(&err_summary).unwrap_or(serde_json::Value::Null),
+                    },
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "validation_failed".to_string(),
+                    error: Some(err_summary.clone()),
+                    span_id,
+                });
+                // Append validation feedback to prompt for next iter.
+                // The model sees concrete failure mode + retries with
+                // corrective context. This is the "observer correction"
+                // in Phase 43 PRD shape, simplified — the validator
+                // itself IS the observer for now.
+                current_prompt = format!(
+                    "{}\n\nPrior attempt failed validation:\n{}\n\nFix the specific issue above and respond with a corrected JSON object.",
+                    req.prompt, err_summary,
+                );
+                continue;
+            }
+        }
+    }
+
+    let duration_ms = t0.elapsed().as_millis() as u64;
+    write_session_failure(&state, &req, &trace_id, max_iter, max_iter, attempt_records, duration_ms).await;
+    (StatusCode::UNPROCESSABLE_ENTITY, Json(IterateFailure {
+        error: format!("max iterations reached ({max_iter}) without passing validation"),
+        iterations: max_iter,
+        history,
+        trace_id: Some(trace_id.clone()),
+    })).into_response()
+}
+
+// ─── Helpers — Langfuse spans + session log + roster check ─────────
+
+fn emit_attempt_span(
+    state: &super::V1State,
+    trace_id: &str,
+    iteration: u32,
+    req: &IterateRequest,
+    prompt: &str,
+    raw: &str,
+    verdict: &str,
+    error: Option<String>,
+    started: chrono::DateTime<chrono::Utc>,
+    ended: chrono::DateTime<chrono::Utc>,
+) -> Option<String> {
+    let lf = state.langfuse.as_ref()?;
+    Some(lf.emit_attempt_span(super::langfuse_trace::AttemptSpan {
+        trace_id: trace_id.to_string(),
+        iteration,
+        model: req.model.clone(),
+        provider: req.provider.clone(),
+        prompt: prompt.to_string(),
+        raw: raw.to_string(),
+        verdict: verdict.to_string(),
+        error,
+        start_time: started.to_rfc3339(),
+        end_time: ended.to_rfc3339(),
+    }))
+}
+
+/// Verify every fill artifact's candidate IDs exist in the roster.
+/// Returns Some(true)/Some(false) on the fill kind, None otherwise
+/// (other kinds don't have worker IDs to ground). Same semantics as
+/// Go's `handlers.rosterCheckFor("fill")`.
+fn grounded_in_roster(
+    state: &super::V1State,
+    kind: &str,
+    artifact: &serde_json::Value,
+) -> Option<bool> {
+    if kind != "fill" {
+        return None;
+    }
+    let fills = artifact.get("fills").and_then(|v| v.as_array())?;
+    for f in fills {
+        let id = match f.get("candidate_id").and_then(|v| v.as_str()) {
+            Some(s) if !s.is_empty() => s,
+            _ => return Some(false),
+        };
+        if state.validate_workers.find(id).is_none() {
+            return Some(false);
+        }
+    }
+    Some(true)
+}
+
+async fn write_session_accepted(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    artifact: &serde_json::Value,
+    grounded: Option<bool>,
+    duration_ms: u64,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let rec = build_session_record(req, trace_id, "accepted", iterations, max_iter, attempts, Some(artifact.clone()), grounded, duration_ms);
+    logger.append(rec).await;
+}
+
+async fn write_session_failure(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    duration_ms: u64,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let rec = build_session_record(req, trace_id, "max_iter_exhausted", iterations, max_iter, attempts, None, None, duration_ms);
+    logger.append(rec).await;
+}
+
+async fn write_infra_error(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    max_iter: u32,
+    duration_ms: u64,
+    error: String,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let attempts = vec![super::session_log::SessionAttemptRecord {
+        iteration: 0,
+        verdict_kind: "infra_error".to_string(),
+        error: Some(error),
+        span_id: None,
+    }];
+    let rec = build_session_record(req, trace_id, "infra_error", 0, max_iter, attempts, None, None, duration_ms);
+    logger.append(rec).await;
+}
+
+fn build_session_record(
+    req: &IterateRequest,
+    trace_id: &str,
+    final_verdict: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    artifact: Option<serde_json::Value>,
+    grounded: Option<bool>,
+    duration_ms: u64,
+) -> super::session_log::SessionRecord {
+    super::session_log::SessionRecord {
+        schema: super::session_log::SESSION_RECORD_SCHEMA.to_string(),
+        session_id: trace_id.to_string(),
+        timestamp: chrono::Utc::now().to_rfc3339(),
+        daemon: "gateway".to_string(),
+        kind: req.kind.clone(),
+        model: req.model.clone(),
+        provider: req.provider.clone(),
+        prompt: super::session_log::truncate(&req.prompt, 4000),
+        iterations,
+        max_iterations: max_iter,
+        final_verdict: final_verdict.to_string(),
+        attempts,
+        artifact,
+        grounded_in_roster: grounded,
+        duration_ms,
+    }
+}
+
+/// Generate a fresh trace id when no parent was forwarded. Same
+/// time-ordered hex shape Langfuse already accepts elsewhere in this
+/// crate (see `langfuse_trace::uuid_v7_like`).
+fn new_trace_id() -> String {
+    let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
+    let rand = std::time::SystemTime::now()
+        .duration_since(std::time::UNIX_EPOCH)
+        .map(|d| d.subsec_nanos())
+        .unwrap_or(0);
+    format!("{:016x}-{:08x}", ts, rand)
+}
+
+async fn call_chat(client: &reqwest::Client, gateway: &str, body: &serde_json::Value, trace_id: &str) -> Result<String, String> {
+    let mut req = client.post(format!("{gateway}/v1/chat")).json(body);
+    if !trace_id.is_empty() {
+        req = req.header(TRACE_ID_HEADER, trace_id);
+    }
+    let resp = req.send().await.map_err(|e| format!("chat hop: {e}"))?;
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_default();
+        return Err(format!("chat {}: {}", status, body.chars().take(300).collect::<String>()));
+    }
+    let parsed: serde_json::Value = resp.json().await.map_err(|e| format!("chat parse: {e}"))?;
+    Ok(parsed.pointer("/choices/0/message/content")
+        .and_then(|v| v.as_str())
+        .unwrap_or("")
+        .to_string())
+}
+
+async fn call_validate(client: &reqwest::Client, gateway: &str, body: &serde_json::Value, trace_id: &str) -> Result<serde_json::Value, String> {
+    let mut req = client.post(format!("{gateway}/v1/validate")).json(body);
+    if !trace_id.is_empty() {
+        req = req.header(TRACE_ID_HEADER, trace_id);
+    }
+    let resp = req.send().await.map_err(|e| format!("validate hop: {e}"))?;
+    let status = resp.status();
+    let parsed: serde_json::Value = resp.json().await.map_err(|e| format!("validate parse: {e}"))?;
+    if status.is_success() {
+        Ok(parsed)
+    } else {
+        // The /v1/validate endpoint returns a ValidationError JSON
+        // on 422; surface its structure verbatim so the prompt-
+        // appending step gets specific failure detail.
+        Err(serde_json::to_string(&parsed).unwrap_or_else(|_| format!("validation {} (unparseable body)", status)))
+    }
+}
+
+/// Extract the first JSON object from a model's output. Handles
+/// fenced code blocks (```json ... ```), bare braces, and stray
+/// prose around the JSON. Returns None on no extractable object.
+///
+/// Made `pub` 2026-05-02 to support the cross-runtime parity probe
+/// at `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`.
+/// The Go counterpart lives at `internal/validator/iterate.go::ExtractJSON`;
+/// when either runtime's algorithm changes the parity probe surfaces
+/// the divergence.
+pub fn extract_json(raw: &str) -> Option<serde_json::Value> {
+    // Try fenced first.
+    let candidates: Vec<String> = {
+        let mut out = vec![];
+        let mut s = raw;
+        while let Some(start) = s.find("```") {
+            let after = &s[start + 3..];
+            // Skip optional language tag (json, etc.)
+            let body_start = after.find('\n').map(|n| n + 1).unwrap_or(0);
+            let body = &after[body_start..];
+            if let Some(end) = body.find("```") {
+                out.push(body[..end].trim().to_string());
+                s = &body[end + 3..];
+            } else { break; }
+        }
+        out
+    };
+    for c in &candidates {
+        if let Ok(v) = serde_json::from_str::<serde_json::Value>(c) {
+            if v.is_object() { return Some(v); }
+        }
+    }
+    // Fall back to outermost {...} balance.
+    let bytes = raw.as_bytes();
+    let mut depth = 0i32;
+    let mut start: Option<usize> = None;
+    for (i, &b) in bytes.iter().enumerate() {
+        match b {
+            b'{' => { if start.is_none() { start = Some(i); } depth += 1; }
+            b'}' => {
+                depth -= 1;
+                if depth == 0 {
+                    if let Some(s) = start {
+                        let slice = &raw[s..=i];
+                        if let Ok(v) = serde_json::from_str::<serde_json::Value>(slice) {
+                            if v.is_object() { return Some(v); }
+                        }
+                        start = None;
+                    }
+                }
+            }
+            _ => {}
+        }
+    }
+    None
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn extract_json_from_fenced_block() {
+        let raw = "Here's my answer:\n```json\n{\"fills\": [{\"candidate_id\": \"W-1\"}]}\n```\nDone.";
+        let v = extract_json(raw).unwrap();
+        assert!(v.get("fills").is_some());
+    }
+
+    #[test]
+    fn extract_json_from_bare_braces() {
+        let raw = "Here you go: {\"fills\": [{\"candidate_id\": \"W-2\"}]}";
+        let v = extract_json(raw).unwrap();
+        assert!(v.get("fills").is_some());
+    }
+
+    #[test]
+    fn extract_json_returns_none_on_no_object() {
+        assert!(extract_json("just prose, no json").is_none());
+    }
+
+    #[test]
+    fn extract_json_picks_first_balanced() {
+        let raw = "{\"a\":1} then {\"b\":2}";
+        let v = extract_json(raw).unwrap();
+        assert_eq!(v.get("a").and_then(|v| v.as_i64()), Some(1));
+    }
+}

crates/gateway/src/v1/kimi.rs

@@ -0,0 +1,227 @@
+//! Kimi For Coding adapter — direct provider for `kimi-for-coding`
+//! (kimi-k2.6 underneath). Used when Ollama Cloud's `kimi-k2:1t` is
+//! returning sustained 5xx (broken upstream) and OpenRouter's
+//! `moonshotai/kimi-k2.6` is rate-limited.
+//!
+//! Endpoint per `kimi.com/code/docs` and `moonshotai.github.io/kimi-cli`:
+//!   base_url:  https://api.kimi.com/coding/v1
+//!   model id:  kimi-for-coding
+//!   auth:      Bearer sk-kimi-…
+//!   protocol:  OpenAI Chat Completions compatible
+//!
+//! IMPORTANT: `api.kimi.com` is a separate account system from
+//! `api.moonshot.ai` and `api.moonshot.cn`. Keys are NOT interchangeable.
+//! This adapter is for `sk-kimi-*` keys provisioned via the Kimi
+//! membership console only.
+//!
+//! Key sourcing priority:
+//!   1. Env var `KIMI_API_KEY` (loaded from /etc/lakehouse/kimi.env via
+//!      systemd EnvironmentFile=)
+//!   2. /etc/lakehouse/kimi.env directly (rescue path if env not loaded)
+//!
+//! First hit wins. Resolved once at gateway startup, stored on
+//! `V1State.kimi_key`.
+
+use std::time::Duration;
+use serde::{Deserialize, Serialize};
+
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+const KIMI_BASE_URL: &str = "https://api.kimi.com/coding/v1";
+// Default 600s — kimi-for-coding is a reasoning model; on large
+// code-audit prompts (~50KB+ input + 8K output) it routinely needs
+// 3-8 min to think + emit. Override with KIMI_TIMEOUT_SECS env var.
+const KIMI_TIMEOUT_SECS_DEFAULT: u64 = 600;
+
+fn kimi_timeout_secs() -> u64 {
+    std::env::var("KIMI_TIMEOUT_SECS")
+        .ok()
+        .and_then(|s| s.trim().parse::<u64>().ok())
+        .filter(|&n| n > 0)
+        .unwrap_or(KIMI_TIMEOUT_SECS_DEFAULT)
+}
+
+pub fn resolve_kimi_key() -> Option<String> {
+    if let Ok(k) = std::env::var("KIMI_API_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    if let Ok(raw) = std::fs::read_to_string("/etc/lakehouse/kimi.env") {
+        for line in raw.lines() {
+            if let Some(rest) = line.strip_prefix("KIMI_API_KEY=") {
+                let k = rest.trim().trim_matches('"').trim_matches('\'');
+                if !k.is_empty() { return Some(k.to_string()); }
+            }
+        }
+    }
+    None
+}
+
+pub async fn chat(
+    key: &str,
+    req: &ChatRequest,
+) -> Result<ChatResponse, String> {
+    // Strip the "kimi/" namespace prefix if the caller used it so the
+    // upstream API sees the bare model id (e.g. "kimi-for-coding").
+    let model = req.model.strip_prefix("kimi/").unwrap_or(&req.model).to_string();
+
+    // Flatten content to a plain String. api.kimi.com is text-only on
+    // the coding endpoint; the OpenAI multimodal array shape
+    // ([{type:"text",text:"..."},{type:"image_url",...}]) returns 400.
+    // Message::text() concats text-parts and drops non-text. Caught
+    // 2026-04-27 by Kimi's self-audit (kimi.rs:137 — content as raw
+    // serde_json::Value risked upstream rejection).
+    let body = KimiChatBody {
+        model: model.clone(),
+        messages: req.messages.iter().map(|m| KimiMessage {
+            role: m.role.clone(),
+            content: serde_json::Value::String(m.text()),
+        }).collect(),
+        max_tokens: req.max_tokens.unwrap_or(800),
+        temperature: req.temperature.unwrap_or(0.3),
+        stream: false,
+    };
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(kimi_timeout_secs()))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+
+    let t0 = std::time::Instant::now();
+    let resp = client
+        .post(format!("{}/chat/completions", KIMI_BASE_URL))
+        .bearer_auth(key)
+        // api.kimi.com gates this endpoint by User-Agent — only sanctioned
+        // coding agents (Claude Code, Kimi CLI, Roo Code, Kilo Code) get
+        // through. Generic clients receive 403 access_terminated_error.
+        // J accepted the TOS risk on 2026-04-27; revisit if Moonshot
+        // tightens enforcement.
+        .header("User-Agent", "claude-code/1.0.0")
+        .json(&body)
+        .send()
+        .await
+        .map_err(|e| format!("api.kimi.com unreachable: {e}"))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("api.kimi.com {}: {}", status, body));
+    }
+
+    let parsed: KimiChatResponse = resp.json().await
+        .map_err(|e| format!("invalid kimi response: {e}"))?;
+
+    let latency_ms = t0.elapsed().as_millis();
+    let choice = parsed.choices.into_iter().next()
+        .ok_or_else(|| "kimi returned no choices".to_string())?;
+    let text = choice.message.content;
+
+    let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
+        ((chars + 3) / 4) as u32
+    });
+    let completion_tokens = parsed.usage.as_ref().map(|u| u.completion_tokens).unwrap_or_else(|| {
+        ((text.chars().count() + 3) / 4) as u32
+    });
+
+    tracing::info!(
+        target: "v1.chat",
+        provider = "kimi",
+        model = %model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "kimi chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model,
+        choices: vec![Choice {
+            index: 0,
+            message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() },
+            finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+// -- Kimi wire shapes (OpenAI-compatible) --
+
+#[derive(Serialize)]
+struct KimiChatBody {
+    model: String,
+    messages: Vec<KimiMessage>,
+    max_tokens: u32,
+    temperature: f64,
+    stream: bool,
+}
+
+#[derive(Serialize)]
+struct KimiMessage { role: String, content: serde_json::Value }
+
+#[derive(Deserialize)]
+struct KimiChatResponse {
+    choices: Vec<KimiChoice>,
+    #[serde(default)]
+    usage: Option<KimiUsage>,
+}
+
+#[derive(Deserialize)]
+struct KimiChoice {
+    message: KimiMessageResp,
+    #[serde(default)]
+    finish_reason: Option<String>,
+}
+
+#[derive(Deserialize)]
+struct KimiMessageResp { content: String }
+
+#[derive(Deserialize)]
+struct KimiUsage { prompt_tokens: u32, completion_tokens: u32 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_kimi_key_does_not_panic() {
+        let _ = resolve_kimi_key();
+    }
+
+    #[test]
+    fn chat_body_serializes_to_openai_shape() {
+        let body = KimiChatBody {
+            model: "kimi-for-coding".into(),
+            messages: vec![
+                KimiMessage { role: "user".into(), content: "review this".into() },
+            ],
+            max_tokens: 800,
+            temperature: 0.3,
+            stream: false,
+        };
+        let json = serde_json::to_string(&body).unwrap();
+        assert!(json.contains("\"model\":\"kimi-for-coding\""));
+        assert!(json.contains("\"messages\""));
+        assert!(json.contains("\"max_tokens\":800"));
+        assert!(json.contains("\"stream\":false"));
+    }
+
+    #[test]
+    fn model_prefix_strip() {
+        let cases = [
+            ("kimi/kimi-for-coding", "kimi-for-coding"),
+            ("kimi-for-coding", "kimi-for-coding"),
+            ("kimi/kimi-k2.6", "kimi-k2.6"),
+        ];
+        for (input, expected) in cases {
+            let out = input.strip_prefix("kimi/").unwrap_or(input);
+            assert_eq!(out, expected, "{input} should become {expected}");
+        }
+    }
+}

crates/gateway/src/v1/langfuse_trace.rs

@@ -76,63 +76,54 @@ impl LangfuseClient {
         });
     }
 
-    async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
-        let trace_id = uuid_v7_like();
-        let gen_id = uuid_v7_like();
-        let trace_ts = ev.start_time.clone();
+    /// Fire-and-forget per-iteration span emit. Returns the generated
+    /// span id synchronously so the caller can stamp it on
+    /// `IterateAttempt.span_id` before the network round-trip resolves.
+    /// Mirrors Go's `validator.Tracer` callback shape.
+    pub fn emit_attempt_span(&self, sp: AttemptSpan) -> String {
+        let span_id = uuid_v7_like();
+        let span_id_for_caller = span_id.clone();
+        let this = self.clone();
+        tokio::spawn(async move {
+            if let Err(e) = this.emit_attempt_span_inner(span_id, sp).await {
+                tracing::warn!(target: "v1.langfuse", "iterate span drop: {e}");
+            }
+        });
+        span_id_for_caller
+    }
 
+    async fn emit_attempt_span_inner(&self, span_id: String, sp: AttemptSpan) -> Result<(), String> {
+        let level = if sp.verdict == "accepted" { "DEFAULT" } else { "WARNING" };
         let batch = IngestionBatch {
-            batch: vec![
-                IngestionEvent {
-                    id: uuid_v7_like(),
-                    timestamp: trace_ts.clone(),
-                    kind: "trace-create",
-                    body: serde_json::json!({
-                        "id": trace_id,
-                        "name": format!("v1.chat:{}", ev.provider),
-                        "input": serde_json::json!({
-                            "model": ev.model,
-                            "messages": ev.input,
-                        }),
-                        "metadata": serde_json::json!({
-                            "provider": ev.provider,
-                            "think": ev.think,
-                        }),
+            batch: vec![IngestionEvent {
+                id: uuid_v7_like(),
+                timestamp: sp.end_time.clone(),
+                kind: "span-create",
+                body: serde_json::json!({
+                    "id": span_id,
+                    "traceId": sp.trace_id,
+                    "name": format!("iterate.attempt[{}]", sp.iteration),
+                    "input": serde_json::json!({
+                        "iteration": sp.iteration,
+                        "model": sp.model,
+                        "provider": sp.provider,
+                        "prompt": truncate(&sp.prompt, 4000),
                     }),
-                },
-                IngestionEvent {
-                    id: uuid_v7_like(),
-                    timestamp: ev.end_time.clone(),
-                    kind: "generation-create",
-                    body: serde_json::json!({
-                        "id": gen_id,
-                        "traceId": trace_id,
-                        "name": "chat",
-                        "model": ev.model,
-                        "modelParameters": serde_json::json!({
-                            "temperature": ev.temperature,
-                            "max_tokens": ev.max_tokens,
-                            "think": ev.think,
-                        }),
-                        "input": ev.input,
-                        "output": ev.output,
-                        "usage": serde_json::json!({
-                            "input": ev.prompt_tokens,
-                            "output": ev.completion_tokens,
-                            "total": ev.prompt_tokens + ev.completion_tokens,
-                            "unit": "TOKENS",
-                        }),
-                        "startTime": ev.start_time,
-                        "endTime": ev.end_time,
-                        "metadata": serde_json::json!({
-                            "provider": ev.provider,
-                            "latency_ms": ev.latency_ms,
-                        }),
+                    "output": serde_json::json!({
+                        "verdict": sp.verdict,
+                        "error": sp.error,
+                        "raw": truncate(&sp.raw, 4000),
                     }),
-                },
-            ],
+                    "level": level,
+                    "startTime": sp.start_time,
+                    "endTime": sp.end_time,
+                }),
+            }],
         };
+        self.post_batch(batch).await
+    }
 
+    async fn post_batch(&self, batch: IngestionBatch) -> Result<(), String> {
         let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
         let resp = self.inner.http
             .post(url)
@@ -146,6 +137,81 @@ impl LangfuseClient {
         }
         Ok(())
     }
+
+    async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
+        // When the caller forwarded a parent trace id (via the
+        // X-Lakehouse-Trace-Id header → V1State plumbing), attach the
+        // generation as a child of that trace. Without a parent we
+        // mint a new top-level trace per call (Phase 40 default).
+        let trace_id = ev.parent_trace_id.clone().unwrap_or_else(uuid_v7_like);
+        let nested = ev.parent_trace_id.is_some();
+        let gen_id = uuid_v7_like();
+        let trace_ts = ev.start_time.clone();
+
+        let mut events = Vec::with_capacity(2);
+        if !nested {
+            // Only mint a fresh trace-create when we don't have a parent.
+            // Reusing a parent trace id without re-creating it is the
+            // contract that lets validatord's iterate-session show up
+            // as one tree in Langfuse.
+            events.push(IngestionEvent {
+                id: uuid_v7_like(),
+                timestamp: trace_ts.clone(),
+                kind: "trace-create",
+                body: serde_json::json!({
+                    "id": trace_id,
+                    "name": format!("v1.chat:{}", ev.provider),
+                    "input": serde_json::json!({
+                        "model": ev.model,
+                        "messages": ev.input,
+                    }),
+                    "metadata": serde_json::json!({
+                        "provider": ev.provider,
+                        "think": ev.think,
+                    }),
+                }),
+            });
+        }
+        events.push(IngestionEvent {
+            id: uuid_v7_like(),
+            timestamp: ev.end_time.clone(),
+            kind: "generation-create",
+            body: serde_json::json!({
+                "id": gen_id,
+                "traceId": trace_id,
+                "name": "chat",
+                "model": ev.model,
+                "modelParameters": serde_json::json!({
+                    "temperature": ev.temperature,
+                    "max_tokens": ev.max_tokens,
+                    "think": ev.think,
+                }),
+                "input": ev.input,
+                "output": ev.output,
+                "usage": serde_json::json!({
+                    "input": ev.prompt_tokens,
+                    "output": ev.completion_tokens,
+                    "total": ev.prompt_tokens + ev.completion_tokens,
+                    "unit": "TOKENS",
+                }),
+                "startTime": ev.start_time,
+                "endTime": ev.end_time,
+                "metadata": serde_json::json!({
+                    "provider": ev.provider,
+                    "latency_ms": ev.latency_ms,
+                }),
+            }),
+        });
+
+        self.post_batch(IngestionBatch { batch: events }).await
+    }
+}
+
+/// Truncate a string to at most `n` chars (NOT bytes). Matches the Go
+/// `trim` helper used in session log + attempt-span emission so an
+/// operator reading two cross-runtime traces sees the same boundary.
+fn truncate(s: &str, n: usize) -> String {
+    s.chars().take(n).collect()
 }
 
 /// Everything the v1.chat handler collects for one completed call.
@@ -162,6 +228,32 @@ pub struct ChatTrace {
     pub start_time: String,
     pub end_time: String,
     pub latency_ms: u64,
+    /// When set, attach this chat trace as a child of the named
+    /// Langfuse trace instead of starting a new top-level trace. Used
+    /// by `/v1/iterate` to nest its inner /v1/chat hops under the
+    /// iterate-session trace so a multi-call session shows in
+    /// Langfuse as ONE trace tree, not N+1 disconnected traces.
+    /// Matches the Go-side `X-Lakehouse-Trace-Id` propagation
+    /// (commit d6d2fdf in golangLAKEHOUSE).
+    pub parent_trace_id: Option<String>,
+}
+
+/// One iteration attempt inside `/v1/iterate`'s loop. Becomes one
+/// span on the parent trace when emitted via `emit_attempt_span`.
+/// Matches Go's `validator.AttemptSpan` shape so the cross-runtime
+/// observability surface is consistent.
+pub struct AttemptSpan {
+    pub trace_id: String,
+    pub iteration: u32,
+    pub model: String,
+    pub provider: String,
+    pub prompt: String,
+    pub raw: String,
+    /// Verdict kind: "no_json" | "validation_failed" | "accepted"
+    pub verdict: String,
+    pub error: Option<String>,
+    pub start_time: String,
+    pub end_time: String,
 }
 
 #[derive(Serialize)]

crates/gateway/src/v1/mod.rs

@@ -16,7 +16,14 @@ pub mod ollama_cloud;
 pub mod openrouter;
 pub mod gemini;
 pub mod claude;
+pub mod kimi;
+pub mod opencode;
+pub mod validate;
+pub mod iterate;
+pub mod auditing_worker_lookup;
 pub mod langfuse_trace;
+pub mod session_log;
+pub mod mode;
 pub mod respond;
 pub mod truth;
 
@@ -52,10 +59,45 @@ pub struct V1State {
     /// `claude::resolve_claude_key()`. None = provider="claude" calls
     /// 503. Phase 40 deliverable.
     pub claude_key: Option<String>,
+    /// Kimi For Coding (api.kimi.com) bearer token — direct provider
+    /// for `kimi-for-coding`. Used when Ollama Cloud's `kimi-k2:1t` is
+    /// upstream-broken. Loaded at startup via `kimi::resolve_kimi_key()`
+    /// from `KIMI_API_KEY` env or `/etc/lakehouse/kimi.env`. None =
+    /// provider="kimi" calls 503.
+    pub kimi_key: Option<String>,
+    /// OpenCode GO (opencode.ai) bearer token — multi-vendor curated
+    /// gateway. One sk-* key reaches Claude Opus 4.7, GPT-5.5-pro,
+    /// Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM, Qwen + free-tier.
+    /// Loaded at startup via `opencode::resolve_opencode_key()` from
+    /// `OPENCODE_API_KEY` env or `/etc/lakehouse/opencode.env`. None =
+    /// provider="opencode" calls 503.
+    pub opencode_key: Option<String>,
+    /// Shared WorkerLookup loaded once at startup from
+    /// workers_500k.parquet (path: LH_WORKERS_PARQUET env, default
+    /// data/datasets/workers_500k.parquet). Used by /v1/validate to
+    /// run FillValidator/EmailValidator with worker-existence checks.
+    /// Falls back to an empty InMemoryWorkerLookup if the file is
+    /// missing — validators still run schema/PII checks but every
+    /// worker-existence check fails (Consistency error), which is
+    /// the correct behavior when the roster isn't configured.
+    pub validate_workers: std::sync::Arc<dyn validator::WorkerLookup>,
     /// Phase 40 early deliverable — Langfuse client. None = tracing
     /// disabled (keys missing or container unreachable). Traces are
     /// fire-and-forget: never block the response path.
     pub langfuse: Option<langfuse_trace::LangfuseClient>,
+    /// Coordinator session JSONL writer (path from
+    /// `[gateway].session_log_path`). One row per `/v1/iterate`
+    /// session for offline DuckDB analysis. None = disabled.
+    /// Cross-runtime parity with the Go-side `validatord`
+    /// `[validatord].session_log_path` (commit 1a3a82a in
+    /// golangLAKEHOUSE).
+    pub session_log: Option<session_log::SessionLogger>,
+    /// Per-subject audit log writer (HMAC-chained JSONL). Loaded at
+    /// startup from /etc/lakehouse/subject_audit.key if the file is
+    /// present and ≥32 bytes. None = audit disabled with a warning
+    /// at startup; PII access is still served but produces no audit
+    /// trail. See docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
+    pub subject_audit: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>>,
 }
 
 #[derive(Default, Clone, Serialize)]
@@ -79,19 +121,83 @@ pub struct ProviderUsage {
 pub fn router(state: V1State) -> Router {
     Router::new()
         .route("/chat", post(chat))
+        // Canonical OpenAI path alias — lets any client built on the
+        // openai SDK (pi-ai, langchain-js, etc.) treat the gateway as
+        // a drop-in middleware via OPENAI_BASE_URL=http://gw/v1 alone.
+        // Same handler as /chat; same OpenAI-compatible request shape.
+        .route("/chat/completions", post(chat))
         .route("/respond", post(respond::respond))
         .route("/usage", get(usage))
         .route("/sessions", get(sessions))
         .route("/context", get(truth::context))
+        .route("/mode", post(mode::route))
+        .route("/mode/list", get(mode::list))
+        .route("/mode/execute", post(mode::execute))
+        .route("/validate", post(validate::validate))
+        .route("/iterate", post(iterate::iterate))
+        .route("/health", get(health))
         .with_state(state)
 }
 
 // -- Shared types (OpenAI-compatible) --
 
-#[derive(Serialize, Deserialize, Clone, Debug)]
+/// OpenAI-compatible message. `content` accepts either a plain string or
+/// an array of content parts (the modern multimodal shape:
+/// `[{type:"text", text:"..."}, {type:"image_url", ...}]`). We store as
+/// `serde_json::Value` to preserve client shape on forward; downstream
+/// providers can take it verbatim. `Message::text()` flattens for
+/// places that need a plain string (Ollama prompt assembly, char
+/// counts, the assistant's own response synthesis).
+#[derive(Serialize, Deserialize, Clone, Debug, Default)]
 pub struct Message {
     pub role: String,
-    pub content: String,
+    /// `Value::Null` is legal: an assistant turn that only makes tool calls.
+    #[serde(default)]
+    pub content: serde_json::Value,
+    /// OpenAI tool-calling fields (2026-09-19). Passed through verbatim on the OpenRouter lane;
+    /// omitted from JSON when unset, so plain-text traffic is byte-identical to before. Before
+    /// this, serde silently DROPPED them — tool-using agents routed through :3100 never saw their
+    /// tools and fabricated tool output (autobun KB finding infra.lakehouse_gateway.tools).
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub tool_calls: Option<serde_json::Value>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub tool_call_id: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+}
+
+impl Message {
+    /// Construct a plain text message — the common shape for callers
+    /// that don't need multimodal content. Wraps the body in
+    /// `serde_json::Value::String` so downstream serializers see the
+    /// canonical OpenAI shape.
+    pub fn new_text(role: impl Into<String>, body: impl Into<String>) -> Self {
+        Self {
+            role: role.into(),
+            content: serde_json::Value::String(body.into()),
+            ..Default::default()
+        }
+    }
+    /// Flatten content to a plain string. Strings pass through; content-
+    /// part arrays concatenate the `text` fields with newlines and skip
+    /// non-text parts (images etc.) — Phase 38/39 callers are text-only,
+    /// real multimodal forwarding is queued.
+    pub fn text(&self) -> String {
+        match &self.content {
+            serde_json::Value::String(s) => s.clone(),
+            serde_json::Value::Array(parts) => {
+                let mut out = String::new();
+                for p in parts {
+                    if let Some(t) = p.get("text").and_then(|v| v.as_str()) {
+                        if !out.is_empty() { out.push('\n'); }
+                        out.push_str(t);
+                    }
+                }
+                out
+            }
+            other => other.to_string(),
+        }
+    }
 }
 
 #[derive(Deserialize, Debug, Clone)]
@@ -102,9 +208,8 @@ pub struct ChatRequest {
     pub temperature: Option<f64>,
     #[serde(default)]
     pub max_tokens: Option<u32>,
-    /// Accepted for shape-compat but ignored in the thin slice —
-    /// Phase 38 returns non-streaming even when the client asked for it.
-    /// Phase 39+ wires real streaming.
+    /// Real SSE passthrough on the openrouter lane (2026-09-20); other lanes reject
+    /// stream=true with a 400 rather than silently returning a non-streamed body.
     #[serde(default)]
     pub stream: Option<bool>,
     /// Non-OpenAI extension. Passes through to the provider's thinking
@@ -122,6 +227,21 @@ pub struct ChatRequest {
     /// from the model name; Phase 38/39 requires explicit selection.
     #[serde(default)]
     pub provider: Option<String>,
+    /// OpenAI tool definitions + choice (2026-09-19). Forwarded on the OpenRouter lane only;
+    /// any other provider REJECTS them with 400 instead of silently stripping them.
+    #[serde(default)]
+    pub tools: Option<serde_json::Value>,
+    #[serde(default)]
+    pub tool_choice: Option<serde_json::Value>,
+}
+
+impl ChatRequest {
+    /// True if the request uses tool calling anywhere (definitions or tool-call history).
+    pub fn uses_tools(&self) -> bool {
+        self.tools.is_some()
+            || self.tool_choice.is_some()
+            || self.messages.iter().any(|m| m.role == "tool" || m.tool_calls.is_some() || m.tool_call_id.is_some())
+    }
 }
 
 #[derive(Serialize)]
@@ -175,6 +295,41 @@ fn resolve_provider(req: &ChatRequest) -> (String, String) {
     if let Some(rest) = req.model.strip_prefix("claude/") {
         return ("claude".to_string(), rest.to_string());
     }
+    if let Some(rest) = req.model.strip_prefix("kimi/") {
+        return ("kimi".to_string(), rest.to_string());
+    }
+    if let Some(rest) = req.model.strip_prefix("opencode/") {
+        return ("opencode".to_string(), rest.to_string());
+    }
+    // Bare `vendor/model` shape (e.g. `x-ai/grok-4.1-fast`,
+    // `moonshotai/kimi-k2`, `openai/gpt-oss-120b:free`) → OpenRouter.
+    // This makes the gateway a drop-in OpenAI-compatible middleware:
+    // clients using the official `openai` SDK only set OPENAI_BASE_URL
+    // + a model name and get correct upstream routing without needing
+    // our custom `provider` field. Ollama models in J's stack use
+    // `model:tag` form with NO slash (`qwen3.5:latest`, `kimi-k2:1t`),
+    // so a slash here unambiguously means "namespaced provider/model".
+    if req.model.contains('/') {
+        return ("openrouter".to_string(), req.model.clone());
+    }
+    // Vendor-bare model names (no slash, no colon) — `gpt-4o-mini`,
+    // `claude-3-5-sonnet-20241022`, etc. Tools like pi-ai validate
+    // models against an OpenAI-style catalog (no namespace prefix),
+    // so they send the bare name. Map to OpenRouter's namespaced form
+    // by inferring the vendor from the leading token. Falls through to
+    // ollama if no pattern matches — preserves existing behavior.
+    if !req.model.contains(':') && !req.model.contains('/') {
+        let m = req.model.as_str();
+        if m.starts_with("gpt-") || m.starts_with("o1-") || m.starts_with("o3-") || m.starts_with("o4-") || m == "o1" || m == "o3" || m == "o4-mini" {
+            return ("openrouter".to_string(), format!("openai/{}", m));
+        }
+        if m.starts_with("claude-") {
+            return ("openrouter".to_string(), format!("anthropic/{}", m));
+        }
+        if m.starts_with("grok-") {
+            return ("openrouter".to_string(), format!("x-ai/{}", m));
+        }
+    }
     ("ollama".to_string(), req.model.clone())
 }
 
@@ -191,9 +346,47 @@ mod resolve_provider_tests {
             stream: None,
             think: None,
             provider: provider.map(|s| s.to_string()),
+            tools: None,
+            tool_choice: None,
         }
     }
 
+    /// Streaming is a per-lane capability, and the routing decision must be visible in a test:
+    /// before 2026-09-20 the gateway accepted stream=true on every lane and returned a single
+    /// JSON body, so a client waiting for SSE frames hung until its own timeout.
+    #[test]
+    fn streaming_is_openrouter_only() {
+        let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash");
+        r.stream = Some(true);
+        assert_eq!(resolve_provider(&r).0, "openrouter", "openrouter lane streams");
+
+        let mut local = mk_req(None, "qwen3.5:latest");
+        local.stream = Some(true);
+        assert_eq!(
+            resolve_provider(&local).0,
+            "ollama",
+            "a local model resolves to ollama, where the handler rejects stream=true with a 400 \
+             instead of silently returning a non-streamed body"
+        );
+    }
+
+    #[test]
+    fn uses_tools_detects_definitions_and_history() {
+        let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash");
+        assert!(!r.uses_tools());
+        r.tools = Some(serde_json::json!([]));
+        assert!(r.uses_tools());
+        let mut r2 = mk_req(None, "qwen3.5:latest");
+        r2.messages.push(Message { role: "tool".into(), content: "ok".into(), tool_call_id: Some("c1".into()), ..Default::default() });
+        assert!(r2.uses_tools());
+    }
+
+    #[test]
+    fn plain_message_serialization_unchanged() {
+        let m = Message::new_text("user", "hi");
+        assert_eq!(serde_json::to_string(&m).unwrap(), r#"{"role":"user","content":"hi"}"#);
+    }
+
     #[test]
     fn explicit_provider_wins() {
         let r = mk_req(Some("openrouter"), "qwen3.5:latest");
@@ -238,24 +431,39 @@ mod resolve_provider_tests {
         let r = mk_req(None, "claude/claude-3-5-sonnet-latest");
         assert_eq!(resolve_provider(&r), ("claude".into(), "claude-3-5-sonnet-latest".into()));
     }
+
+    #[test]
+    fn kimi_prefix_infers_and_strips() {
+        let r = mk_req(None, "kimi/kimi-for-coding");
+        assert_eq!(resolve_provider(&r), ("kimi".into(), "kimi-for-coding".into()));
+    }
 }
 
 async fn chat(
     State(state): State<V1State>,
+    headers: axum::http::HeaderMap,
     Json(req): Json<ChatRequest>,
-) -> Result<Json<ChatResponse>, (StatusCode, String)> {
+) -> Result<axum::response::Response, (StatusCode, String)> {
+    use axum::response::IntoResponse as _;
     if req.messages.is_empty() {
         return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
     }
-    if req.stream.unwrap_or(false) {
-        tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
-    }
-
     // Provider resolution: explicit `req.provider` wins; otherwise
     // infer from a model-name prefix. Phase 39 PRD gate example:
     // `model: "openrouter/openai/gpt-4o-mini"` → provider "openrouter",
     // adapter gets the stripped "openai/gpt-4o-mini".
     let (provider, effective_model) = resolve_provider(&req);
+    // Tool calling is only wired on the OpenRouter lane. Refuse loudly elsewhere — silently
+    // dropping tools made agents fabricate tool results (2026-09-19).
+    if req.uses_tools() && !matches!(provider.as_str(), "openrouter" | "openrouter_free") {
+        return Err((
+            StatusCode::BAD_REQUEST,
+            format!(
+                "tools/tool_calls are only supported on the openrouter lane via this gateway \
+                 (resolved provider: {provider}); use an OpenRouter model id (e.g. openrouter/<vendor>/<model>)"
+            ),
+        ));
+    }
     let start_time = chrono::Utc::now();
     let start_instant = std::time::Instant::now();
 
@@ -271,6 +479,45 @@ async fn chat(
             std::borrow::Cow::Owned(cloned)
         };
 
+    // STREAMING (2026-09-20). Phase 38 accepted `stream: true` and returned a single JSON body;
+    // a client waiting for SSE frames hangs and then reports "no response". Forward the upstream
+    // SSE verbatim on the lane that supports it, and say so plainly on the lanes that do not
+    // instead of pretending. Streamed calls skip usage metering + Langfuse (see chat_stream).
+    if req.stream.unwrap_or(false) {
+        if !matches!(provider.as_str(), "openrouter" | "openrouter_free") {
+            return Err((
+                StatusCode::BAD_REQUEST,
+                format!(
+                    "stream=true is only supported on the openrouter lane via this gateway \
+                     (resolved provider: {provider}); resend with stream=false or use an \
+                     OpenRouter model id"
+                ),
+            ));
+        }
+        let key = state.openrouter_key.as_deref().ok_or((
+            StatusCode::SERVICE_UNAVAILABLE,
+            "OPENROUTER_API_KEY not configured".to_string(),
+        ))?;
+        let upstream = openrouter::chat_stream(key, &*req_for_adapter)
+            .await
+            .map_err(|e| (StatusCode::BAD_GATEWAY, format!("openrouter stream: {e}")))?;
+        tracing::info!(
+            target: "v1.chat",
+            provider = %provider,
+            model = %effective_model,
+            "streaming passthrough — usage metering and Langfuse are skipped for this call",
+        );
+        let body = axum::body::Body::from_stream(upstream.bytes_stream());
+        return Ok((
+            [
+                (axum::http::header::CONTENT_TYPE, "text/event-stream"),
+                (axum::http::header::CACHE_CONTROL, "no-cache"),
+            ],
+            body,
+        )
+            .into_response());
+    }
+
     let (resp, used_provider) = match provider.as_str() {
         "ollama" | "local" | "" => {
             let r = ollama::chat(&state.ai_client, &*req_for_adapter)
@@ -325,10 +572,37 @@ async fn chat(
                 .map_err(|e| (StatusCode::BAD_GATEWAY, format!("claude: {e}")))?;
             (r, "claude".to_string())
         }
+        "kimi" => {
+            // Direct Kimi For Coding provider — bypasses Ollama Cloud's
+            // upstream-broken kimi-k2:1t and OpenRouter's rate-limited
+            // moonshotai/kimi-k2.6. Uses sk-kimi-* keys from the Kimi
+            // membership console.
+            let key = state.kimi_key.as_deref().ok_or((
+                StatusCode::SERVICE_UNAVAILABLE,
+                "KIMI_API_KEY not configured".to_string(),
+            ))?;
+            let r = kimi::chat(key, &*req_for_adapter)
+                .await
+                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("kimi: {e}")))?;
+            (r, "kimi".to_string())
+        }
+        "opencode" => {
+            // OpenCode GO multi-vendor gateway — Claude Opus 4.7,
+            // GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+            // Qwen, free-tier. OpenAI-compat at opencode.ai/zen/go/v1.
+            let key = state.opencode_key.as_deref().ok_or((
+                StatusCode::SERVICE_UNAVAILABLE,
+                "OPENCODE_API_KEY not configured".to_string(),
+            ))?;
+            let r = opencode::chat(key, &*req_for_adapter)
+                .await
+                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("opencode: {e}")))?;
+            (r, "opencode".to_string())
+        }
         other => {
             return Err((
                 StatusCode::BAD_REQUEST,
-                format!("unknown provider '{other}' — supported: ollama, ollama_cloud, openrouter, gemini, claude"),
+                format!("unknown provider '{other}' — supported: ollama, ollama_cloud, openrouter, gemini, claude, kimi, opencode"),
             ));
         }
     };
@@ -342,8 +616,19 @@ async fn chat(
     // untouched.
     if let Some(lf) = &state.langfuse {
         let output = resp.choices.first()
-            .map(|c| c.message.content.clone())
+            .map(|c| c.message.text())
             .unwrap_or_default();
+        // Cross-runtime trace linkage. When a caller (validatord on
+        // Go side, /v1/iterate on Rust side) forwards a parent trace
+        // id via X-Lakehouse-Trace-Id, attach this generation to that
+        // trace so the iterate session and its inner chat hops show
+        // up as ONE trace tree in Langfuse. Header name matches the
+        // Go-side `shared.TraceIDHeader` constant byte-for-byte.
+        let parent_trace_id = headers
+            .get(crate::v1::iterate::TRACE_ID_HEADER)
+            .and_then(|v| v.to_str().ok())
+            .map(|s| s.to_string())
+            .filter(|s| !s.is_empty());
         lf.emit_chat(langfuse_trace::ChatTrace {
             provider: used_provider.clone(),
             model: resp.model.clone(),
@@ -357,6 +642,47 @@ async fn chat(
             start_time: start_time.to_rfc3339(),
             end_time: end_time.to_rfc3339(),
             latency_ms,
+            parent_trace_id,
+        });
+    }
+
+    // Phase 40 part 2 — fire-and-forget /event to observer at :3800.
+    // Same ring-buffer that scrum + scenario events land in, so any
+    // tool-routed-through-our-gateway (Pi, Archon, openai SDK clients)
+    // shows up alongside scrum_master events for KB consolidation +
+    // pathway-memory + bug-fingerprint compounding. Best-effort:
+    // observer being down doesn't block the chat response.
+    {
+        let provider = used_provider.clone();
+        let model = resp.model.clone();
+        let prompt_tokens = resp.usage.prompt_tokens;
+        let completion_tokens = resp.usage.completion_tokens;
+        let success = true;
+        tokio::spawn(async move {
+            let body = serde_json::json!({
+                "endpoint": "/v1/chat",
+                "source": "v1.chat",
+                "event_kind": "chat_completion",
+                "input_summary": format!(
+                    "{} {} prompt={}t",
+                    provider, model, prompt_tokens
+                ),
+                "output_summary": format!(
+                    "completion={}t {}ms",
+                    completion_tokens, latency_ms
+                ),
+                "success": success,
+                "duration_ms": latency_ms,
+            });
+            let client = reqwest::Client::builder()
+                .timeout(std::time::Duration::from_secs(2))
+                .build()
+                .unwrap_or_else(|_| reqwest::Client::new());
+            let _ = client
+                .post("http://localhost:3800/event")
+                .json(&body)
+                .send()
+                .await;
         });
     }
 
@@ -375,7 +701,7 @@ async fn chat(
         provider_usage.total_tokens += resp.usage.total_tokens as u64;
     }
 
-    Ok(Json(resp))
+    Ok(Json(resp).into_response())
 }
 
 async fn usage(State(state): State<V1State>) -> impl IntoResponse {
@@ -383,6 +709,43 @@ async fn usage(State(state): State<V1State>) -> impl IntoResponse {
     Json(snapshot)
 }
 
+/// Production operational health endpoint.
+///
+/// `/v1/health` reports per-subsystem status as a JSON object so an
+/// operator (or the lakehouse-auditor service, or a load balancer)
+/// can verify the gateway is fully booted, has its provider keys
+/// loaded, the worker roster is hot, and Langfuse is reachable.
+/// Returns 200 always — fields are observed-state, not pass/fail
+/// gates. A monitoring tool should evaluate the booleans + counts
+/// against its own thresholds.
+async fn health(State(state): State<V1State>) -> impl IntoResponse {
+    // Honest worker count via WorkerLookup::len. Production switchover
+    // verification: after swapping workers_500k.parquet → real Chicago
+    // data and restarting, this number should match the row count of
+    // the new file. 0 means the file was missing / unreadable / had a
+    // schema mismatch and the gateway booted with the empty fallback.
+    let workers_count = state.validate_workers.len();
+    let providers_configured = serde_json::json!({
+        "ollama_cloud": state.ollama_cloud_key.is_some(),
+        "openrouter": state.openrouter_key.is_some(),
+        "kimi": state.kimi_key.is_some(),
+        "opencode": state.opencode_key.is_some(),
+        "gemini": state.gemini_key.is_some(),
+        "claude": state.claude_key.is_some(),
+    });
+    let langfuse_configured = state.langfuse.is_some();
+    let usage_snapshot = state.usage.read().await.clone();
+    Json(serde_json::json!({
+        "status": "ok",
+        "workers_count": workers_count,
+        "workers_loaded": workers_count > 0,
+        "providers_configured": providers_configured,
+        "langfuse_configured": langfuse_configured,
+        "usage_total_requests": usage_snapshot.requests,
+        "usage_by_provider": usage_snapshot.by_provider.keys().collect::<Vec<_>>(),
+    }))
+}
+
 // Phase 38 is stateless — no session persistence yet. Return an empty
 // list in OpenAI-ish shape so clients that probe this endpoint don't
 // 404. Real session state lands in Phase 41 with the profile-system
@@ -414,7 +777,7 @@ mod tests {
         assert_eq!(r.model, "qwen3.5:latest");
         assert_eq!(r.messages.len(), 2);
         assert_eq!(r.messages[0].role, "system");
-        assert_eq!(r.messages[1].content, "Hi");
+        assert_eq!(r.messages[1].text(), "Hi");
         assert_eq!(r.temperature, Some(0.2));
         assert_eq!(r.max_tokens, Some(100));
     }

crates/gateway/src/v1/ollama.rs

@@ -60,10 +60,7 @@ pub async fn chat(client: &AiClient, req: &ChatRequest) -> Result<ChatResponse,
         model: resp.model,
         choices: vec![Choice {
             index: 0,
-            message: Message {
-                role: "assistant".into(),
-                content: resp.text,
-            },
+            message: Message::new_text("assistant", resp.text),
             finish_reason: "stop".into(),
         }],
         usage: UsageBlock {
@@ -89,13 +86,14 @@ fn flatten_messages(messages: &[Message]) -> (String, String) {
     let mut system = String::new();
     let mut prompt = String::new();
     for m in messages {
+        let body = m.text();
         if m.role == "system" {
             if !system.is_empty() { system.push('\n'); }
-            system.push_str(&m.content);
+            system.push_str(&body);
         } else {
             prompt.push_str(&m.role);
             prompt.push_str(": ");
-            prompt.push_str(&m.content);
+            prompt.push_str(&body);
             prompt.push_str("\n\n");
         }
     }
@@ -104,7 +102,7 @@ fn flatten_messages(messages: &[Message]) -> (String, String) {
 }
 
 fn estimate_prompt_tokens(messages: &[Message]) -> u32 {
-    let chars: usize = messages.iter().map(|m| m.content.chars().count()).sum();
+    let chars: usize = messages.iter().map(|m| m.text().chars().count()).sum();
     ((chars + 3) / 4) as u32
 }
 
@@ -115,10 +113,10 @@ mod tests {
     #[test]
     fn flatten_separates_system_from_turns() {
         let msgs = vec![
-            Message { role: "system".into(), content: "Rules here.".into() },
-            Message { role: "user".into(), content: "Q1".into() },
-            Message { role: "assistant".into(), content: "A1".into() },
-            Message { role: "user".into(), content: "Q2".into() },
+            Message { role: "system".into(), content: "Rules here.".into(), ..Default::default() },
+            Message { role: "user".into(), content: "Q1".into(), ..Default::default() },
+            Message { role: "assistant".into(), content: "A1".into(), ..Default::default() },
+            Message { role: "user".into(), content: "Q2".into(), ..Default::default() },
         ];
         let (system, prompt) = flatten_messages(&msgs);
         assert_eq!(system, "Rules here.");
@@ -131,9 +129,9 @@ mod tests {
     #[test]
     fn flatten_concatenates_multiple_system_messages() {
         let msgs = vec![
-            Message { role: "system".into(), content: "First.".into() },
-            Message { role: "system".into(), content: "Second.".into() },
-            Message { role: "user".into(), content: "Hi".into() },
+            Message { role: "system".into(), content: "First.".into(), ..Default::default() },
+            Message { role: "system".into(), content: "Second.".into(), ..Default::default() },
+            Message { role: "user".into(), content: "Hi".into(), ..Default::default() },
         ];
         let (system, _) = flatten_messages(&msgs);
         assert_eq!(system, "First.\nSecond.");
@@ -141,7 +139,7 @@ mod tests {
 
     #[test]
     fn flatten_with_no_system_returns_empty_system() {
-        let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
+        let msgs = vec![Message { role: "user".into(), content: "hi".into(), ..Default::default() }];
         let (system, prompt) = flatten_messages(&msgs);
         assert!(system.is_empty());
         assert!(prompt.contains("user: hi"));
@@ -149,11 +147,11 @@ mod tests {
 
     #[test]
     fn estimate_tokens_chars_div_4_ceiling() {
-        let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
+        let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into(), ..Default::default() }];
         // 8 chars / 4 = 2, with ceiling → 2
         assert_eq!(estimate_prompt_tokens(&msgs), 2);
 
-        let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
+        let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into(), ..Default::default() }];
         // 9 chars → (9+3)/4 = 3 (ceiling)
         assert_eq!(estimate_prompt_tokens(&msgs2), 3);
     }

crates/gateway/src/v1/ollama_cloud.rs

@@ -88,7 +88,7 @@ pub async fn chat(
     let text = parsed.response.unwrap_or_default();
 
     let prompt_tokens = parsed.prompt_eval_count.unwrap_or_else(|| {
-        let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
         ((chars + 3) / 4) as u32
     });
     let completion_tokens = parsed.eval_count.unwrap_or_else(|| {
@@ -112,7 +112,7 @@ pub async fn chat(
         model: parsed.model.unwrap_or_else(|| req.model.clone()),
         choices: vec![Choice {
             index: 0,
-            message: Message { role: "assistant".into(), content: text },
+            message: Message::new_text("assistant", text),
             finish_reason: "stop".into(),
         }],
         usage: UsageBlock {

crates/gateway/src/v1/opencode.rs

@@ -0,0 +1,228 @@
+//! OpenCode GO adapter — multi-vendor curated gateway via opencode.ai/zen/go.
+//!
+//! One sk-* key reaches Claude Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro,
+//! Kimi K2.6, DeepSeek, GLM, Qwen, plus 4 free-tier models.
+//! OpenAI-compatible Chat Completions; auth via Bearer.
+//!
+//! Why a separate adapter (vs reusing openrouter.rs):
+//! - Different account, different key, different base_url
+//! - No HTTP-Referer / X-Title headers (those are OpenRouter-specific)
+//! - Future-proof for any opencode-only request shaping
+//!
+//! Key sourcing priority:
+//!   1. Env var `OPENCODE_API_KEY` (loaded from /etc/lakehouse/opencode.env
+//!      via systemd EnvironmentFile=)
+//!   2. /etc/lakehouse/opencode.env directly (rescue path if env missing)
+//!
+//! Resolved once at gateway startup, stored on `V1State.opencode_key`.
+//! Model-prefix routing: "opencode/<model>" auto-routes here, prefix
+//! stripped before upstream call.
+
+use std::time::Duration;
+use serde::{Deserialize, Serialize};
+
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+// /zen/v1 is the unified OpenCode endpoint that covers BOTH the
+// Zen pay-per-token tier (Claude/GPT/Gemini frontier) AND the Go
+// subscription tier (Kimi/GLM/DeepSeek/Qwen/Minimax/mimo). When the
+// caller has both, opencode bills per-model: Zen models charge Zen
+// balance, Go models charge against the Go subscription cap.
+//
+// /zen/go/v1 exists as a Go-only sub-path (rejects Zen models with
+// "Model not supported"); we use the unified /zen/v1 since the same
+// key works for both with correct billing routing upstream.
+const OPENCODE_BASE_URL: &str = "https://opencode.ai/zen/v1";
+// 600s default — opencode upstream models include reasoning-heavy
+// variants (Claude Opus, Kimi K2.6, GLM-5.1) that legitimately take
+// 3-5 min on big audit prompts. Override via OPENCODE_TIMEOUT_SECS.
+const OPENCODE_TIMEOUT_SECS_DEFAULT: u64 = 600;
+
+fn opencode_timeout_secs() -> u64 {
+    std::env::var("OPENCODE_TIMEOUT_SECS")
+        .ok()
+        .and_then(|s| s.trim().parse::<u64>().ok())
+        .filter(|&n| n > 0)
+        .unwrap_or(OPENCODE_TIMEOUT_SECS_DEFAULT)
+}
+
+pub fn resolve_opencode_key() -> Option<String> {
+    if let Ok(k) = std::env::var("OPENCODE_API_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    if let Ok(raw) = std::fs::read_to_string("/etc/lakehouse/opencode.env") {
+        for line in raw.lines() {
+            if let Some(rest) = line.strip_prefix("OPENCODE_API_KEY=") {
+                let k = rest.trim().trim_matches('"').trim_matches('\'');
+                if !k.is_empty() { return Some(k.to_string()); }
+            }
+        }
+    }
+    None
+}
+
+pub async fn chat(
+    key: &str,
+    req: &ChatRequest,
+) -> Result<ChatResponse, String> {
+    // Strip the "opencode/" namespace prefix so the upstream sees the
+    // bare model id (e.g. "claude-opus-4-7", "kimi-k2.6").
+    let model = req.model.strip_prefix("opencode/").unwrap_or(&req.model).to_string();
+
+    // Anthropic models on opencode reject `temperature` with a 400
+    // "temperature is deprecated for this model" error. Strip the
+    // field for claude-* and the new gpt-5.x reasoning lineages
+    // (Anthropic/OpenAI's reasoning models all moved away from temp).
+    // Other models keep the caller's value or default to 0.3.
+    let drop_temp = model.starts_with("claude-")
+        || model.starts_with("gpt-5")
+        || model.starts_with("o1")
+        || model.starts_with("o3")
+        || model.starts_with("o4");
+    let body = OCChatBody {
+        model: model.clone(),
+        messages: req.messages.iter().map(|m| OCMessage {
+            role: m.role.clone(),
+            content: m.content.clone(),
+        }).collect(),
+        // filter(|&n| n > 0) catches Some(0) — same trap that bit the
+        // Kimi adapter when callers passed empty-env-parsed-to-0.
+        max_tokens: req.max_tokens.filter(|&n| n > 0).unwrap_or(800),
+        temperature: if drop_temp { None } else { Some(req.temperature.unwrap_or(0.3)) },
+        stream: false,
+    };
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(opencode_timeout_secs()))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+
+    let t0 = std::time::Instant::now();
+    let resp = client
+        .post(format!("{}/chat/completions", OPENCODE_BASE_URL))
+        .bearer_auth(key)
+        .json(&body)
+        .send()
+        .await
+        .map_err(|e| format!("opencode.ai unreachable: {e}"))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("opencode.ai {}: {}", status, body));
+    }
+
+    let parsed: OCChatResponse = resp.json().await
+        .map_err(|e| format!("invalid opencode response: {e}"))?;
+
+    let latency_ms = t0.elapsed().as_millis();
+    let choice = parsed.choices.into_iter().next()
+        .ok_or_else(|| "opencode returned no choices".to_string())?;
+    let text = choice.message.content;
+
+    let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
+        ((chars + 3) / 4) as u32
+    });
+    let completion_tokens = parsed.usage.as_ref().map(|u| u.completion_tokens).unwrap_or_else(|| {
+        ((text.chars().count() + 3) / 4) as u32
+    });
+
+    tracing::info!(
+        target: "v1.chat",
+        provider = "opencode",
+        model = %model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "opencode chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model,
+        choices: vec![Choice {
+            index: 0,
+            message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() },
+            finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+// -- OpenCode wire shapes (OpenAI-compatible) --
+
+#[derive(Serialize)]
+struct OCChatBody {
+    model: String,
+    messages: Vec<OCMessage>,
+    max_tokens: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    temperature: Option<f64>,
+    stream: bool,
+}
+
+#[derive(Serialize)]
+struct OCMessage { role: String, content: serde_json::Value }
+
+#[derive(Deserialize)]
+struct OCChatResponse {
+    choices: Vec<OCChoice>,
+    #[serde(default)]
+    usage: Option<OCUsage>,
+}
+
+#[derive(Deserialize)]
+struct OCChoice {
+    message: OCMessageResp,
+    #[serde(default)]
+    finish_reason: Option<String>,
+}
+
+#[derive(Deserialize)]
+struct OCMessageResp { content: String }
+
+#[derive(Deserialize)]
+struct OCUsage { prompt_tokens: u32, completion_tokens: u32 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_opencode_key_does_not_panic() {
+        let _ = resolve_opencode_key();
+    }
+
+    #[test]
+    fn model_prefix_strip() {
+        let cases = [
+            ("opencode/claude-opus-4-7", "claude-opus-4-7"),
+            ("opencode/kimi-k2.6", "kimi-k2.6"),
+            ("claude-opus-4-7", "claude-opus-4-7"),
+        ];
+        for (input, expected) in cases {
+            let out = input.strip_prefix("opencode/").unwrap_or(input);
+            assert_eq!(out, expected);
+        }
+    }
+
+    #[test]
+    fn max_tokens_filters_zero() {
+        // The trap: empty env -> Number("") -> 0 -> Some(0). Adapter
+        // must not pass 0 upstream; should fall to 800.
+        let some_zero: Option<u32> = Some(0);
+        let result = some_zero.filter(|&n| n > 0).unwrap_or(800);
+        assert_eq!(result, 800);
+        let some_real: Option<u32> = Some(4096);
+        assert_eq!(some_real.filter(|&n| n > 0).unwrap_or(800), 4096);
+        let none_val: Option<u32> = None;
+        assert_eq!(none_val.filter(|&n| n > 0).unwrap_or(800), 800);
+    }
+}

crates/gateway/src/v1/openrouter.rs

@@ -20,7 +20,19 @@ use serde::{Deserialize, Serialize};
 use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
 
 const OR_BASE_URL: &str = "https://openrouter.ai/api/v1";
-const OR_TIMEOUT_SECS: u64 = 180;
+// Default request ceiling. 180s starves long-context calls (kimi_architect's
+// ~180K-char second pass died at exactly this wall on 2026-09-02, surfacing
+// as a misleading "error decoding response body") — override per host with
+// OR_TIMEOUT_SECS. The auditor budgets up to 900s client-side.
+const OR_TIMEOUT_SECS_DEFAULT: u64 = 180;
+
+fn or_timeout_secs() -> u64 {
+    std::env::var("OR_TIMEOUT_SECS")
+        .ok()
+        .and_then(|v| v.trim().parse::<u64>().ok())
+        .filter(|&s| s > 0)
+        .unwrap_or(OR_TIMEOUT_SECS_DEFAULT)
+}
 
 pub fn resolve_openrouter_key() -> Option<String> {
     if let Ok(k) = std::env::var("OPENROUTER_API_KEY") {
@@ -49,6 +61,52 @@ pub fn resolve_openrouter_key() -> Option<String> {
     None
 }
 
+/// Streaming passthrough (2026-09-20). The gateway used to accept `stream: true` and return a
+/// single non-streamed JSON body — a client waiting for SSE frames hangs until its own timeout
+/// and reports "no response" (this is what broke openclaw's gateway lane). We now forward the
+/// upstream SSE byte-for-byte.
+///
+/// The caller streams the body straight to its client, so nothing here parses the chunks:
+/// usage metering and Langfuse tracing are SKIPPED for streamed calls, and the handler logs
+/// that. Metering streamed responses needs the final usage chunk and belongs in its own change.
+pub async fn chat_stream(key: &str, req: &ChatRequest) -> Result<reqwest::Response, String> {
+    let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string();
+    let body = ORChatBody {
+        model,
+        messages: req.messages.iter().map(|m| ORMessage {
+            role: m.role.clone(),
+            content: m.content.clone(),
+            tool_calls: m.tool_calls.clone(),
+            tool_call_id: m.tool_call_id.clone(),
+            name: m.name.clone(),
+        }).collect(),
+        tools: req.tools.clone(),
+        tool_choice: req.tool_choice.clone(),
+        max_tokens: Some(req.max_tokens.unwrap_or(800)),
+        temperature: req.temperature.unwrap_or(0.3),
+        stream: true,
+    };
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(or_timeout_secs()))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+    let resp = client
+        .post(format!("{}/chat/completions", OR_BASE_URL))
+        .bearer_auth(key)
+        .header("HTTP-Referer", "https://vcp.devop.live")
+        .header("X-Title", "Lakehouse Scrum")
+        .json(&serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?)
+        .send()
+        .await
+        .map_err(|e| format!("openrouter.ai unreachable: {e}"))?;
+    if !resp.status().is_success() {
+        let code = resp.status();
+        let text = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("openrouter {code}: {}", text.chars().take(300).collect::<String>()));
+    }
+    Ok(resp)
+}
+
 pub async fn chat(
     key: &str,
     req: &ChatRequest,
@@ -57,35 +115,86 @@ pub async fn chat(
     // form so OpenRouter sees the raw model id (e.g. "openai/gpt-oss-120b:free").
     let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string();
 
-    let body = ORChatBody {
+    let mut body = ORChatBody {
         model: model.clone(),
+        // Pass content through verbatim — preserves OpenAI's multimodal
+        // content-parts shape (`[{type:"text",text:"..."}, ...]`) so the
+        // upstream provider sees exactly what the client sent.
         messages: req.messages.iter().map(|m| ORMessage {
             role: m.role.clone(),
             content: m.content.clone(),
+            tool_calls: m.tool_calls.clone(),
+            tool_call_id: m.tool_call_id.clone(),
+            name: m.name.clone(),
         }).collect(),
-        max_tokens: req.max_tokens.unwrap_or(800),
+        tools: req.tools.clone(),
+        tool_choice: req.tool_choice.clone(),
+        max_tokens: Some(req.max_tokens.unwrap_or(800)),
         temperature: req.temperature.unwrap_or(0.3),
         stream: false,
     };
 
     let client = reqwest::Client::builder()
-        .timeout(Duration::from_secs(OR_TIMEOUT_SECS))
+        .timeout(Duration::from_secs(or_timeout_secs()))
         .build()
         .map_err(|e| format!("build client: {e}"))?;
 
     let t0 = std::time::Instant::now();
-    let resp = client
-        .post(format!("{}/chat/completions", OR_BASE_URL))
-        .bearer_auth(key)
-        // OpenRouter recommends Referer + Title for attribution; absent
-        // headers do not fail the call but help us see our traffic in
-        // their dashboard.
-        .header("HTTP-Referer", "https://vcp.devop.live")
-        .header("X-Title", "Lakehouse Scrum")
-        .json(&body)
-        .send()
-        .await
-        .map_err(|e| format!("openrouter.ai unreachable: {e}"))?;
+    let send = |client: reqwest::Client, key: String, body_json: serde_json::Value| async move {
+        client
+            .post(format!("{}/chat/completions", OR_BASE_URL))
+            .bearer_auth(key)
+            // OpenRouter recommends Referer + Title for attribution; absent
+            // headers do not fail the call but help us see our traffic in
+            // their dashboard.
+            .header("HTTP-Referer", "https://vcp.devop.live")
+            .header("X-Title", "Lakehouse Scrum")
+            .json(&body_json)
+            .send()
+            .await
+            .map_err(|e| format!("openrouter.ai unreachable: {e}"))
+    };
+    let mut resp = send(
+        client.clone(),
+        key.to_string(),
+        serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
+    )
+    .await?;
+
+    // OpenRouter's affordability precheck reserves max_tokens × price against
+    // the account balance. Measured 2026-08-28: stripping the cap makes it
+    // WORSE (the precheck then prices the model's own maximum, e.g. 65536) —
+    // so a low-balance 402 surfaces verbatim and the fix is funding the
+    // account the key belongs to, not request-shape games. One clamp retry:
+    // when the error names an affordable budget ≥256 tokens, retry once at
+    // that budget so small answers still get through on a thin balance.
+    if resp.status() == reqwest::StatusCode::PAYMENT_REQUIRED && body.max_tokens.is_some() {
+        let err_body = resp.text().await.unwrap_or_else(|_| "?".into());
+        let afford = err_body
+            .split("can only afford ")
+            .nth(1)
+            .and_then(|s| s.split(|c: char| !c.is_ascii_digit()).next())
+            .and_then(|n| n.parse::<u32>().ok());
+        match afford {
+            Some(n) if n >= 256 => {
+                tracing::warn!(
+                    target: "v1.chat",
+                    provider = "openrouter",
+                    model = %model,
+                    afford = n,
+                    "402 affordability precheck — clamping max_tokens to the stated budget",
+                );
+                body.max_tokens = Some(n);
+                resp = send(
+                    client.clone(),
+                    key.to_string(),
+                    serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
+                )
+                .await?;
+            }
+            _ => return Err(format!("openrouter.ai 402 Payment Required: {}", err_body)),
+        }
+    }
 
     let status = resp.status();
     if !status.is_success() {
@@ -99,10 +208,27 @@ pub async fn chat(
     let latency_ms = t0.elapsed().as_millis();
     let choice = parsed.choices.into_iter().next()
         .ok_or_else(|| "openrouter returned no choices".to_string())?;
-    let text = choice.message.content;
+    let tool_calls = choice.message.tool_calls.clone();
+    let reasoning_only = choice.message.content.is_none()
+        && choice.message.reasoning.is_some()
+        && tool_calls.is_none();
+    let text = choice.message.content.unwrap_or_default();
+    if reasoning_only {
+        // The model spent its whole budget thinking. Return well-formed empty
+        // content (callers' schema validation + retry budgets handle it
+        // deterministically) rather than a decode error that reads as a
+        // gateway fault — and never pass thinking-text off as an answer.
+        tracing::warn!(
+            target: "v1.chat",
+            provider = "openrouter",
+            model = %model,
+            finish = %choice.finish_reason.as_deref().unwrap_or("?"),
+            "reasoning-only response (content null) — raise max_tokens or use a non-reasoning model",
+        );
+    }
 
     let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
-        let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
         ((chars + 3) / 4) as u32
     });
     let completion_tokens = parsed.usage.as_ref().map(|u| u.completion_tokens).unwrap_or_else(|| {
@@ -126,7 +252,17 @@ pub async fn chat(
         model,
         choices: vec![Choice {
             index: 0,
-            message: Message { role: "assistant".into(), content: text },
+            message: Message {
+                role: "assistant".into(),
+                // OpenAI shape: a tool-call-only turn carries `content: null`.
+                content: if tool_calls.is_some() && text.is_empty() {
+                    serde_json::Value::Null
+                } else {
+                    serde_json::Value::String(text)
+                },
+                tool_calls,
+                ..Default::default()
+            },
             finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
         }],
         usage: UsageBlock {
@@ -143,13 +279,27 @@ pub async fn chat(
 struct ORChatBody {
     model: String,
     messages: Vec<ORMessage>,
-    max_tokens: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    max_tokens: Option<u32>,
     temperature: f64,
     stream: bool,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    tools: Option<serde_json::Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    tool_choice: Option<serde_json::Value>,
 }
 
 #[derive(Serialize)]
-struct ORMessage { role: String, content: String }
+struct ORMessage {
+    role: String,
+    content: serde_json::Value,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    tool_calls: Option<serde_json::Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    tool_call_id: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    name: Option<String>,
+}
 
 #[derive(Deserialize)]
 struct ORChatResponse {
@@ -166,7 +316,21 @@ struct ORChoice {
 }
 
 #[derive(Deserialize)]
-struct ORMessageResp { content: String }
+struct ORMessageResp {
+    // Reasoning models (GLM-5.x, deepseek-v4-pro) return `content: null` plus a
+    // `reasoning` field when they exhaust max_tokens mid-thought. A bare String
+    // here made serde reject the whole body ("invalid openrouter response",
+    // 2026-08-28) — which took down every reasoning model on the lane,
+    // including phase2's configured glm-5.2. Content stays the only text we
+    // ever return; reasoning is observability, never an answer.
+    #[serde(default)]
+    content: Option<String>,
+    #[serde(default)]
+    reasoning: Option<String>,
+    /// OpenAI tool calls requested by the model (passed back to the client verbatim).
+    #[serde(default)]
+    tool_calls: Option<serde_json::Value>,
+}
 
 #[derive(Deserialize)]
 struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
@@ -175,6 +339,21 @@ struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
 mod tests {
     use super::*;
 
+    #[test]
+    fn reasoning_model_null_content_parses() {
+        // GLM-5.x / deepseek-v4-pro shape: content null + reasoning present
+        // (the 2026-08-28 decode failure). Must parse, not error.
+        let body = r#"{"choices":[{"message":{"role":"assistant","content":null,"reasoning":"thinking…"},"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":10}}"#;
+        let parsed: ORChatResponse = serde_json::from_str(body).expect("null content must parse");
+        let choice = &parsed.choices[0];
+        assert!(choice.message.content.is_none());
+        assert_eq!(choice.message.reasoning.as_deref(), Some("thinking…"));
+        // And the normal shape still parses with reasoning absent.
+        let normal = r#"{"choices":[{"message":{"role":"assistant","content":"OK"},"finish_reason":"stop"}]}"#;
+        let parsed: ORChatResponse = serde_json::from_str(normal).expect("normal shape");
+        assert_eq!(parsed.choices[0].message.content.as_deref(), Some("OK"));
+    }
+
     #[test]
     fn resolve_openrouter_key_does_not_panic() {
         // Smoke test — all three sources may or may not be set depending
@@ -187,17 +366,57 @@ mod tests {
         let body = ORChatBody {
             model: "openai/gpt-oss-120b:free".into(),
             messages: vec![
-                ORMessage { role: "user".into(), content: "review this".into() },
+                ORMessage { role: "user".into(), content: "review this".into(),
+                            tool_calls: None, tool_call_id: None, name: None },
             ],
-            max_tokens: 800,
+            max_tokens: Some(800),
             temperature: 0.3,
             stream: false,
+            tools: None,
+            tool_choice: None,
         };
         let json = serde_json::to_string(&body).unwrap();
         assert!(json.contains("\"model\":\"openai/gpt-oss-120b:free\""));
         assert!(json.contains("\"messages\""));
         assert!(json.contains("\"max_tokens\":800"));
         assert!(json.contains("\"stream\":false"));
+        // no tool fields on plain traffic (byte-compat with pre-2026-09-19 bodies)
+        assert!(!json.contains("tools") && !json.contains("tool_call"));
+    }
+
+    #[test]
+    fn chat_body_forwards_tools_and_tool_messages() {
+        let tools = serde_json::json!([{"type":"function","function":{"name":"exec","parameters":{"type":"object"}}}]);
+        let body = ORChatBody {
+            model: "z-ai/glm-5.3-flash".into(),
+            messages: vec![
+                ORMessage { role: "assistant".into(), content: serde_json::Value::Null,
+                            tool_calls: Some(serde_json::json!([{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}])),
+                            tool_call_id: None, name: None },
+                ORMessage { role: "tool".into(), content: "ok".into(), tool_calls: None,
+                            tool_call_id: Some("c1".into()), name: Some("exec".into()) },
+            ],
+            max_tokens: Some(100),
+            temperature: 0.0,
+            stream: false,
+            tools: Some(tools),
+            tool_choice: Some(serde_json::json!("auto")),
+        };
+        let json = serde_json::to_string(&body).unwrap();
+        assert!(json.contains("\"tools\":[{"));
+        assert!(json.contains("\"tool_choice\":\"auto\""));
+        assert!(json.contains("\"tool_call_id\":\"c1\""));
+        assert!(json.contains("\"tool_calls\":[{"));
+    }
+
+    #[test]
+    fn response_with_tool_calls_parses() {
+        let raw = r#"{"choices":[{"message":{"content":null,"tool_calls":[{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}]},"finish_reason":"tool_calls"}]}"#;
+        let parsed: ORChatResponse = serde_json::from_str(raw).unwrap();
+        let c = &parsed.choices[0];
+        assert!(c.message.content.is_none());
+        assert!(c.message.tool_calls.is_some());
+        assert_eq!(c.finish_reason.as_deref(), Some("tool_calls"));
     }
 
     #[test]

crates/gateway/src/v1/session_log.rs

@@ -0,0 +1,235 @@
+//! Coordinator session JSONL writer — Rust parity with the Go-side
+//! `internal/validator/session_log.go` (commit 1a3a82a in
+//! golangLAKEHOUSE). Same schema, same field names, same producer
+//! semantics, so a unified longitudinal log can pull from either
+//! runtime via DuckDB.
+//!
+//! Schema: `session.iterate.v1`. One row per `/v1/iterate` session.
+//! Append-only. Best-effort posture: errors warn and the iterate
+//! response always ships.
+//!
+//! See `golangLAKEHOUSE/docs/SESSION_LOG.md` for the full schema
+//! reference + DuckDB query examples. This module produces rows
+//! with `daemon: "gateway"`; the Go side produces `daemon:
+//! "validatord"`. Operators who want a unified stream can point both
+//! to the same path (the OS write-append is atomic for the row sizes
+//! we produce) or query both files together via duckdb's `read_json`
+//! glob support.
+
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+pub const SESSION_RECORD_SCHEMA: &str = "session.iterate.v1";
+
+/// One row in coordinator_sessions.jsonl. Field names are the on-wire
+/// names — must stay byte-equal to the Go side's
+/// `validator.SessionRecord` (proven by the cross-runtime parity
+/// probe at golangLAKEHOUSE/scripts/cutover/parity/).
+// Deserialize is supported so the parity helper binary can round-trip
+// fixture inputs through serde without hand-rolling a parser. Production
+// emit path uses Serialize only; SessionRecord rows are written by the
+// gateway and consumed by DuckDB / external tooling, never re-read by us.
+#[derive(Serialize, Deserialize)]
+pub struct SessionRecord {
+    pub schema: String,
+    pub session_id: String,
+    pub timestamp: String,
+    pub daemon: String,
+    pub kind: String,
+    pub model: String,
+    pub provider: String,
+    pub prompt: String,
+    pub iterations: u32,
+    pub max_iterations: u32,
+    pub final_verdict: String, // "accepted" | "max_iter_exhausted" | "infra_error"
+    pub attempts: Vec<SessionAttemptRecord>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub artifact: Option<serde_json::Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub grounded_in_roster: Option<bool>,
+    pub duration_ms: u64,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct SessionAttemptRecord {
+    pub iteration: u32,
+    pub verdict_kind: String, // "no_json" | "validation_failed" | "accepted" | "infra_error"
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub error: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub span_id: Option<String>,
+}
+
+/// Append-only writer. Cloneable handle — internal state is Arc'd so
+/// V1State can keep its own clone and per-request clones are cheap.
+#[derive(Clone)]
+pub struct SessionLogger {
+    inner: Arc<Inner>,
+}
+
+struct Inner {
+    path: String,
+    /// tokio::Mutex (not std) because we hold it across the async
+    /// fs write. Contention is low (one row per /v1/iterate session).
+    mu: Mutex<()>,
+}
+
+impl SessionLogger {
+    /// Construct a logger writing to `path`. Empty path → None
+    /// (skip the wiring in the iterate handler entirely).
+    pub fn from_path(path: &str) -> Option<Self> {
+        if path.is_empty() {
+            return None;
+        }
+        Some(Self {
+            inner: Arc::new(Inner {
+                path: path.to_string(),
+                mu: Mutex::new(()),
+            }),
+        })
+    }
+
+    /// Append one record. Best-effort: failures land in `tracing::warn!`
+    /// and the caller sees Ok(()) — observability is a witness, never
+    /// a gate. Returns Err only on impossible cases the type system
+    /// can't rule out (here: serde_json::to_string failing on a
+    /// well-formed struct, which shouldn't happen).
+    pub async fn append(&self, rec: SessionRecord) {
+        let body = match serde_json::to_string(&rec) {
+            Ok(s) => s,
+            Err(e) => {
+                tracing::warn!(target: "v1.session_log", "marshal: {e}");
+                return;
+            }
+        };
+        let _guard = self.inner.mu.lock().await;
+        if let Err(e) = self.write(&body).await {
+            tracing::warn!(target: "v1.session_log", "write {}: {e}", self.inner.path);
+        }
+    }
+
+    async fn write(&self, body: &str) -> std::io::Result<()> {
+        use tokio::fs::OpenOptions;
+        use tokio::io::AsyncWriteExt;
+
+        // Lazy mkdir on first write so a not-yet-mounted volume at
+        // startup doesn't kill the daemon.
+        if let Some(parent) = std::path::Path::new(&self.inner.path).parent() {
+            if !parent.as_os_str().is_empty() {
+                tokio::fs::create_dir_all(parent).await?;
+            }
+        }
+        let mut f = OpenOptions::new()
+            .append(true)
+            .create(true)
+            .open(&self.inner.path)
+            .await?;
+        f.write_all(body.as_bytes()).await?;
+        f.write_all(b"\n").await?;
+        Ok(())
+    }
+}
+
+/// Best-effort UTF-8 char truncation. Matches Go's `trim` helper so
+/// rows produced by either runtime cap fields at the same boundaries.
+pub fn truncate(s: &str, n: usize) -> String {
+    s.chars().take(n).collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::PathBuf;
+    use tokio::fs;
+
+    fn fixture_record(session_id: &str) -> SessionRecord {
+        SessionRecord {
+            schema: SESSION_RECORD_SCHEMA.to_string(),
+            session_id: session_id.to_string(),
+            timestamp: "2026-05-02T08:00:00Z".to_string(),
+            daemon: "gateway".to_string(),
+            kind: "fill".to_string(),
+            model: "qwen3.5:latest".to_string(),
+            provider: "ollama".to_string(),
+            prompt: "produce a fill artifact".to_string(),
+            iterations: 1,
+            max_iterations: 3,
+            final_verdict: "accepted".to_string(),
+            attempts: vec![SessionAttemptRecord {
+                iteration: 0,
+                verdict_kind: "accepted".to_string(),
+                error: None,
+                span_id: Some("span-0".to_string()),
+            }],
+            artifact: Some(serde_json::json!({"fills":[{"candidate_id":"W-1"}]})),
+            grounded_in_roster: Some(true),
+            duration_ms: 50,
+        }
+    }
+
+    #[tokio::test]
+    async fn from_path_empty_returns_none() {
+        assert!(SessionLogger::from_path("").is_none());
+    }
+
+    #[tokio::test]
+    async fn append_writes_jsonl_row_with_schema_field() {
+        let dir = tempdir();
+        let path = dir.join("sessions.jsonl");
+        let path_str = path.to_string_lossy().to_string();
+        let logger = SessionLogger::from_path(&path_str).unwrap();
+        logger.append(fixture_record("trace-a")).await;
+
+        let body = fs::read_to_string(&path).await.unwrap();
+        assert!(body.contains("\"schema\":\"session.iterate.v1\""));
+        assert!(body.contains("\"session_id\":\"trace-a\""));
+        assert!(body.contains("\"grounded_in_roster\":true"));
+        assert!(body.ends_with('\n'));
+    }
+
+    #[tokio::test]
+    async fn append_concurrent_safe() {
+        let dir = tempdir();
+        let path = dir.join("sessions.jsonl");
+        let path_str = path.to_string_lossy().to_string();
+        let logger = SessionLogger::from_path(&path_str).unwrap();
+
+        let n = 32;
+        let mut handles = Vec::with_capacity(n);
+        for i in 0..n {
+            let l = logger.clone();
+            handles.push(tokio::spawn(async move {
+                l.append(fixture_record(&format!("trace-{i}"))).await;
+            }));
+        }
+        for h in handles {
+            h.await.unwrap();
+        }
+
+        let body = fs::read_to_string(&path).await.unwrap();
+        let lines: Vec<_> = body.lines().filter(|l| !l.is_empty()).collect();
+        assert_eq!(lines.len(), n, "expected {n} rows, got {}", lines.len());
+        // Every row must round-trip through serde — a torn write
+        // would surface as a parse error.
+        for line in lines {
+            let _: serde_json::Value = serde_json::from_str(line).expect("valid json per row");
+        }
+    }
+
+    fn tempdir() -> PathBuf {
+        // Per-test unique path so prior runs don't pollute the next.
+        // The static counter increments across the whole test binary,
+        // so back-to-back tests in the same module get distinct dirs.
+        static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
+        let n = COUNTER.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+        let p = std::env::temp_dir().join(format!(
+            "session_log_test_{}_{}_{}",
+            std::process::id(),
+            chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0),
+            n,
+        ));
+        std::fs::create_dir_all(&p).unwrap();
+        p
+    }
+}

crates/gateway/src/v1/validate.rs

@@ -0,0 +1,82 @@
+//! /v1/validate — gateway-side artifact validation endpoint.
+//!
+//! Phase 43 v3 part 2: makes the validator crate network-callable.
+//! Any caller (scrum loop, test harness, future agent) can POST a
+//! generated artifact and get back a Report (success) or
+//! ValidationError (failure with structured field/reason).
+//!
+//! Request shape:
+//!   POST /v1/validate
+//!   {
+//!     "kind": "fill" | "email" | "playbook",
+//!     "artifact": { ... },
+//!     "context": { ... }   // optional — folded into artifact._context
+//!   }
+//!
+//! Response on success: 200 + Report JSON
+//! Response on failure: 422 + ValidationError JSON
+//! Response on bad request: 400 + plain-text error
+//!
+//! The shared WorkerLookup is loaded once at gateway startup from
+//! workers_500k.parquet (path configurable via LH_WORKERS_PARQUET
+//! env, defaults to data/datasets/workers_500k.parquet). Falls back
+//! to an empty InMemoryWorkerLookup if the file is missing — the
+//! validators will still run schema/length/PII checks but worker-
+//! existence checks will all fail (Consistency error), which is the
+//! correct behavior when the roster isn't configured.
+
+use axum::{extract::State, http::StatusCode, response::IntoResponse, Json};
+use serde::Deserialize;
+use validator::{
+    Artifact, Validator, ValidationError,
+    staffing::{
+        fill::FillValidator,
+        email::EmailValidator,
+        playbook::PlaybookValidator,
+    },
+};
+
+#[derive(Deserialize)]
+pub struct ValidateRequest {
+    /// `"fill" | "email" | "playbook"` — picks which validator runs.
+    pub kind: String,
+    /// The artifact JSON (free-form; shape depends on `kind`).
+    pub artifact: serde_json::Value,
+    /// Optional context bag — merged into `artifact._context` so the
+    /// validator can read fields like `target_count`, `city`,
+    /// `client_id`, `candidate_id` without callers having to embed
+    /// `_context` in the artifact themselves.
+    #[serde(default)]
+    pub context: Option<serde_json::Value>,
+}
+
+pub async fn validate(
+    State(state): State<super::V1State>,
+    Json(req): Json<ValidateRequest>,
+) -> impl IntoResponse {
+    // Merge context into artifact under `_context` so validators can
+    // pull contract metadata uniformly.
+    let mut artifact_value = req.artifact;
+    if let Some(ctx) = req.context {
+        if let Some(obj) = artifact_value.as_object_mut() {
+            obj.insert("_context".to_string(), ctx);
+        }
+    }
+
+    // Dispatch.
+    let workers = state.validate_workers.clone();
+    let result: Result<validator::Report, ValidationError> = match req.kind.as_str() {
+        "fill" => FillValidator::new(workers).validate(&Artifact::FillProposal(artifact_value)),
+        "email" => EmailValidator::new(workers).validate(&Artifact::EmailDraft(artifact_value)),
+        "playbook" => PlaybookValidator.validate(&Artifact::Playbook(artifact_value)),
+        other => return (
+            StatusCode::BAD_REQUEST,
+            format!("unknown kind '{other}' — expected fill | email | playbook"),
+        ).into_response(),
+    };
+
+    match result {
+        Ok(report) => (StatusCode::OK, Json(report)).into_response(),
+        Err(e) => (StatusCode::UNPROCESSABLE_ENTITY, Json(e)).into_response(),
+    }
+}

crates/lance-bench/src/main.rs

@@ -456,6 +456,26 @@ async fn build_lance_vector_index(path: &str, _dims: usize) -> Result<()> {
         .await
         .context("create_index")?;
 
+    // Also build the scalar btree on doc_id. This bench's
+    // measure_random_access_lance uses take(row_position) which doesn't
+    // need the btree, but the dataset this bench writes is also queried
+    // downstream by /vectors/lance/doc/<name>/<doc_id> (the production
+    // lookup path) — without this index that path falls back to a full
+    // table scan. Cheap to build (~1.2s on 10M rows) and matches the
+    // gateway's lance_migrate handler behavior so bench-produced datasets
+    // are immediately production-shape.
+    use lance_index::scalar::ScalarIndexParams;
+    dataset
+        .create_index(
+            &["doc_id"],
+            IndexType::Scalar,
+            Some("doc_id_btree".into()),
+            &ScalarIndexParams::default(),
+            true,
+        )
+        .await
+        .context("create_index doc_id btree")?;
+
     Ok(())
 }
 

crates/shared/src/config.rs

@@ -62,6 +62,15 @@ pub struct GatewayConfig {
     pub host: String,
     #[serde(default = "default_gateway_port")]
     pub port: u16,
+    /// Coordinator session JSONL output path. One row per
+    /// `/v1/iterate` session, schema=`session.iterate.v1`. Empty =
+    /// disabled. Cross-runtime parity with the Go side's
+    /// `[validatord].session_log_path` (added 2026-05-02). Default
+    /// empty so existing deployments aren't perturbed; production
+    /// sets `/var/lib/lakehouse/gateway/sessions.jsonl`. See
+    /// `golangLAKEHOUSE/docs/SESSION_LOG.md` for query examples.
+    #[serde(default)]
+    pub session_log_path: String,
 }
 
 #[derive(Debug, Clone, Deserialize)]
@@ -149,7 +158,13 @@ fn default_gateway_port() -> u16 { 3100 }
 fn default_storage_root() -> String { "./data".to_string() }
 fn default_profile_root() -> String { "./data/_profiles".to_string() }
 fn default_manifest_prefix() -> String { "_catalog/manifests".to_string() }
-fn default_sidecar_url() -> String { "http://localhost:3200".to_string() }
+// Post-2026-05-02: AiClient talks directly to Ollama; the Python
+// sidecar's hot-path role was retired. The config field name
+// `[sidecar].url` is preserved for migration compatibility (operators
+// with existing TOMLs don't need to rename anything), but the value
+// now points at Ollama. Lab UI / pipeline_lab Python remains as a
+// dev-only tool; not on this URL.
+fn default_sidecar_url() -> String { "http://localhost:11434".to_string() }
 fn default_embed_model() -> String { "nomic-embed-text".to_string() }
 fn default_gen_model() -> String { "qwen2.5".to_string() }
 fn default_rerank_model() -> String { "qwen2.5".to_string() }
@@ -184,7 +199,11 @@ impl Config {
 impl Default for Config {
     fn default() -> Self {
         Self {
-            gateway: GatewayConfig { host: default_host(), port: default_gateway_port() },
+            gateway: GatewayConfig {
+                host: default_host(),
+                port: default_gateway_port(),
+                session_log_path: String::new(),
+            },
             storage: StorageConfig {
                 root: default_storage_root(),
                 profile_root: default_profile_root(),

crates/shared/src/types.rs

@@ -364,3 +364,228 @@ pub enum Redaction {
         keep_suffix: usize,
     },
 }
+
+// ─── Subject manifests (compliance/access metadata layer) ────────────
+// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
+//
+// A SubjectManifest is a per-person record threading existing catalogd
+// primitives (datasets, views, tombstones) together by subject token.
+// Stored at _catalog/subjects/<candidate_id>.json. NOT a separate
+// daemon — extends catalogd's existing manifest pattern.
+//
+// The audit log per subject (HMAC-chained JSONL at
+// _catalog/subjects/<candidate_id>.audit.jsonl) is owned by a separate
+// SubjectAuditWriter (Step 2) — not modeled here.
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum SubjectStatus {
+    PendingConsent,
+    Active,
+    Withdrawn,
+    RetentionExpired,
+    Erased,
+}
+
+/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed:
+/// the gateway treats Unknown as healthcare-equivalent until reclassified
+/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1).
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum SubjectVertical {
+    Unknown,
+    General,
+    Healthcare,
+    Finance,
+    Other,
+}
+
+impl Default for SubjectVertical {
+    fn default() -> Self { SubjectVertical::Unknown }
+}
+
+/// State machine for general-PII consent. BIPA-specific biometric
+/// consent is tracked separately on `BiometricConsent`.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum ConsentStatus {
+    /// Backfilled from existing data sources before consent was tracked.
+    /// SHOULD route as if no consent exists until reviewed.
+    PendingBackfillReview,
+    /// New subject; awaiting consent UX.
+    PendingFirstContact,
+    Given,
+    Withdrawn,
+    Expired,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum BiometricConsentStatus {
+    /// Default. No biometric data exists for this subject.
+    NeverCollected,
+    Pending,
+    Given,
+    Withdrawn,
+    Expired,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct GeneralPiiConsent {
+    pub status: ConsentStatus,
+    /// Reference to the consent template version (matches a row in the
+    /// future consent_versions table). Empty when status is pending_*.
+    #[serde(default)]
+    pub version: String,
+    #[serde(default)]
+    pub given_at: Option<chrono::DateTime<chrono::Utc>>,
+    #[serde(default)]
+    pub withdrawn_at: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BiometricConsent {
+    pub status: BiometricConsentStatus,
+    /// BIPA: max 3 years from last interaction. Implementation MUST
+    /// enforce daily expiration sweep against this field.
+    #[serde(default)]
+    pub retention_until: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectConsent {
+    pub general_pii: GeneralPiiConsent,
+    pub biometric: BiometricConsent,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectRetention {
+    pub general_pii_until: chrono::DateTime<chrono::Utc>,
+    /// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026").
+    #[serde(default)]
+    pub policy: String,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectDatasetRef {
+    /// catalogd dataset name (must resolve to an existing manifest).
+    pub name: String,
+    /// Column in that dataset that holds the subject's identifier.
+    pub key_column: String,
+    /// Specific identifier value (typically same as candidate_id; differs
+    /// only when a dataset uses a foreign-key with a different format).
+    pub key_value: String,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectManifest {
+    /// Schema version tag — always "subject_manifest.v1" for v1.
+    #[serde(default = "default_subject_manifest_schema")]
+    pub schema: String,
+    /// Subject identifier (the canonical token, e.g. "CAND-000001").
+    pub candidate_id: String,
+    pub created_at: chrono::DateTime<chrono::Utc>,
+    pub updated_at: chrono::DateTime<chrono::Utc>,
+    pub status: SubjectStatus,
+    #[serde(default)]
+    pub vertical: SubjectVertical,
+    pub consent: SubjectConsent,
+    pub retention: SubjectRetention,
+    /// Datasets that contain rows for this subject. Used by audit-response
+    /// to know what to project. Each entry MUST reference an existing
+    /// catalogd dataset manifest (validated on put_subject).
+    pub datasets: Vec<SubjectDatasetRef>,
+    /// Names of existing AiViews that safely project this subject's data.
+    /// Used by service-tier readers (no PII exposure).
+    #[serde(default)]
+    pub safe_views: Vec<String>,
+    /// Path (relative to repo root) of the per-subject audit JSONL.
+    /// Owned by SubjectAuditWriter (Step 2). Empty until first write.
+    #[serde(default)]
+    pub audit_log_path: String,
+    /// SHA-256 of the most recent HMAC chain checkpoint of the audit log.
+    /// Empty until first audit row written.
+    #[serde(default)]
+    pub audit_log_chain_root: String,
+    /// Biometric collection metadata. None until the first biometric
+    /// upload via Phase 1.6 Gate 3 photo endpoint. Cleared on erasure.
+    /// `serde(default)` so existing manifests parse without re-emit.
+    #[serde(default)]
+    pub biometric_collection: Option<BiometricCollection>,
+}
+
+/// Per-subject biometric collection record. Written by the Gate 3
+/// photo-upload endpoint when a candidate's photo is accepted under
+/// `consent.biometric.status == "given"`.
+///
+/// Reset to None by the BIPA erasure flow (destruction runbook).
+/// Preserved across schema evolutions via the BTreeMap-canonicalized
+/// HMAC chain — old audit rows verify regardless of new fields.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct BiometricCollection {
+    /// Quarantined filesystem path of the original photo bytes,
+    /// relative to the configured biometric storage root. NEVER under
+    /// `data/headshots/` (which is reserved for synthetic StyleGAN
+    /// faces). Format: `data/biometric/uploads/<sanitized_id>/<ts>.<ext>`.
+    pub data_path: String,
+    /// SHA-256 of the original photo bytes. Used for integrity check
+    /// (file-on-disk matches what was uploaded), NOT for
+    /// re-derivation of the original from the hash.
+    pub template_hash: String,
+    pub collected_at: chrono::DateTime<chrono::Utc>,
+    /// SHA-256 of the consent template version that was in force when
+    /// this photo was uploaded. Anchors the consent record to a
+    /// specific document version (per Phase 1.6 Gate 2 §6).
+    #[serde(default)]
+    pub consent_version_hash: String,
+    /// deepface classifications, when the offline classifier has run.
+    /// None until classifier integration ships (Gate 3b).
+    #[serde(default)]
+    pub classifications: Option<serde_json::Value>,
+}
+
+fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() }
+
+/// Audit accessor — who/why touched a subject's PII.
+/// Persisted in subject_audit.v1 rows; never logged in plaintext outside
+/// the per-subject audit file.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct AuditAccessor {
+    /// One of: "gateway_lookup", "audit_response", "legal_request",
+    /// "system_resolve". Caller-defined; SubjectAuditWriter does not
+    /// validate the kind string.
+    pub kind: String,
+    /// Daemon name (e.g. "gateway", "validator", "catalogd").
+    pub daemon: String,
+    /// Purpose token (e.g. "fill_validation", "audit_subject_response").
+    pub purpose: String,
+    /// X-Lakehouse-Trace-Id for cross-daemon trace correlation. Empty
+    /// when the request didn't carry one.
+    #[serde(default)]
+    pub trace_id: String,
+}
+
+/// One audit row in a subject's per-subject audit JSONL.
+/// HMAC chain: row_hmac = HMAC-SHA256(key, prev_chain_hash || canon(row - row_hmac))
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectAuditRow {
+    /// Always "subject_audit.v1" for v1.
+    #[serde(default = "default_subject_audit_schema")]
+    pub schema: String,
+    pub ts: chrono::DateTime<chrono::Utc>,
+    pub candidate_id: String,
+    pub accessor: AuditAccessor,
+    /// Field names accessed (e.g. ["name"], ["name", "phone"]).
+    pub fields_accessed: Vec<String>,
+    /// "success" | "denied" | "not_found" | "error".
+    pub result: String,
+    /// Hash of the previous row in this subject's audit log. "GENESIS"
+    /// for the first row.
+    pub prev_chain_hash: String,
+    /// HMAC-SHA256 over (prev_chain_hash || canonical_json(row - row_hmac)).
+    /// Computed by SubjectAuditWriter; callers leave empty on input.
+    #[serde(default)]
+    pub row_hmac: String,
+}
+
+fn default_subject_audit_schema() -> String { "subject_audit.v1".into() }

crates/validator/Cargo.toml

@@ -9,3 +9,7 @@ serde_json = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true }
 tracing = { workspace = true }
+# Parquet loader for ParquetWorkerLookup (Phase 43 v3 — production
+# WorkerLookup backed by workers_500k.parquet snapshot).
+arrow = { workspace = true }
+parquet = { workspace = true }

crates/validator/src/lib.rs

@@ -93,3 +93,89 @@ pub trait Validator: Send + Sync {
     /// Human-readable name for logs + Langfuse traces.
     fn name(&self) -> &'static str;
 }
+
+// ─── Worker lookup (Phase 43 v2) ────────────────────────────────────────
+//
+// Validators that cross-check artifacts against the worker roster
+// (FillValidator, EmailValidator) take an `Arc<dyn WorkerLookup>` at
+// construction. Keeping the trait sync + in-memory mirrors the
+// lakehouse pattern of "load truth into memory, validate against
+// snapshot, refresh periodically" rather than per-call DB hits.
+//
+// Production impl: wrap a parquet snapshot loaded from
+// `data/datasets/workers_500k.parquet` (or its safe view counterpart
+// once Track A.B lands). Tests use `InMemoryWorkerLookup`.
+
+/// One worker row from the staffing roster — the fields validators
+/// actually read. Anything not on this struct (resume_text, scores,
+/// communications) is intentionally hidden from the validator path.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct WorkerRecord {
+    pub candidate_id: String,
+    pub name: String,
+    /// Free-form. Validators check for `"active"` (any other value
+    /// fails the status check). Common values from existing data:
+    /// "active", "inactive", "placed", "blacklisted".
+    pub status: String,
+    pub city: Option<String>,
+    pub state: Option<String>,
+    pub role: Option<String>,
+    /// Client ids this worker has been blacklisted from. Populated
+    /// from joining a blacklist table; empty when not provided.
+    #[serde(default)]
+    pub blacklisted_clients: Vec<String>,
+}
+
+/// Worker lookup contract. Sync by design — implementations should
+/// hold an in-memory snapshot, not perform per-call I/O.
+pub trait WorkerLookup: Send + Sync {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord>;
+    /// Number of workers in the snapshot. Default 0 for impls that
+    /// genuinely don't know (e.g. a future SQL-backed lookup that
+    /// counts on demand). InMemoryWorkerLookup overrides with the
+    /// HashMap size; ParquetWorkerLookup constructs an
+    /// InMemoryWorkerLookup so it inherits the override. Used by
+    /// /v1/health to report data-load status during production
+    /// switchover (the Chicago dataset replaces synthetic test data;
+    /// the health endpoint is how operators verify the new file
+    /// loaded correctly without restart-and-pray).
+    fn len(&self) -> usize { 0 }
+}
+
+/// HashMap-backed lookup. Used by validator unit tests + as a
+/// reasonable bootstrap impl for production once the parquet loader
+/// fills it on startup.
+pub struct InMemoryWorkerLookup {
+    rows: std::collections::HashMap<String, WorkerRecord>,
+}
+
+impl InMemoryWorkerLookup {
+    pub fn new() -> Self {
+        Self { rows: Default::default() }
+    }
+    pub fn from_records(records: Vec<WorkerRecord>) -> Self {
+        let mut rows = std::collections::HashMap::with_capacity(records.len());
+        for r in records {
+            rows.insert(r.candidate_id.clone(), r);
+        }
+        Self { rows }
+    }
+    pub fn insert(&mut self, record: WorkerRecord) {
+        self.rows.insert(record.candidate_id.clone(), record);
+    }
+    pub fn len(&self) -> usize { self.rows.len() }
+    pub fn is_empty(&self) -> bool { self.rows.is_empty() }
+}
+
+impl Default for InMemoryWorkerLookup {
+    fn default() -> Self { Self::new() }
+}
+
+impl WorkerLookup for InMemoryWorkerLookup {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
+        self.rows.get(candidate_id).cloned()
+    }
+    fn len(&self) -> usize {
+        self.rows.len()
+    }
+}

crates/validator/src/staffing/email.rs

@@ -1,4 +1,4 @@
-//! Email/SMS draft validator.
+//! Email/SMS draft validator (Phase 43 v2 — real PII + name checks).
 //!
 //! PRD checks:
 //!   - Schema (TO/BODY fields present)
@@ -6,15 +6,31 @@
 //!   - PII absence (no SSN / salary leaked into outgoing text)
 //!   - Worker-name consistency (name in message matches worker record)
 //!
-//! Scaffold implements schema + length. PII regex (SSN pattern,
-//! salary-number pattern) lives in `shared::pii::strip_pii` — plug in
-//! a follow-up when the validator caller knows the worker record to
-//! cross-check name consistency.
+//! Like FillValidator, EmailValidator takes `Arc<dyn WorkerLookup>` at
+//! construction. The contract metadata (which worker the message is
+//! about) travels under `_context.candidate_id` in the JSON payload.
+//! When `_context.candidate_id` is present and resolves, the validator
+//! cross-checks that the worker's name appears verbatim in the body.
+//!
+//! PII detection is std-only (no regex dep) — a hand-rolled scan
+//! covers the patterns we actually care about: SSN (NNN-NN-NNNN),
+//! salary statements ("salary" / "compensation" near a $ amount).
 
-use crate::{Artifact, Report, Validator, ValidationError};
+use crate::{
+    Artifact, Report, Validator, ValidationError, WorkerLookup,
+};
+use std::sync::Arc;
 use std::time::Instant;
 
-pub struct EmailValidator;
+pub struct EmailValidator {
+    workers: Arc<dyn WorkerLookup>,
+}
+
+impl EmailValidator {
+    pub fn new(workers: Arc<dyn WorkerLookup>) -> Self {
+        Self { workers }
+    }
+}
 
 const SMS_MAX_CHARS: usize = 160;
 const EMAIL_SUBJECT_MAX_CHARS: usize = 78;
@@ -32,7 +48,7 @@ impl Validator for EmailValidator {
             }),
         };
 
-        let to = value.get("to").and_then(|v| v.as_str()).ok_or(
+        let _to = value.get("to").and_then(|v| v.as_str()).ok_or(
             ValidationError::Schema {
                 field: "to".into(),
                 reason: "missing or not a string".into(),
@@ -63,54 +79,292 @@ impl Validator for EmailValidator {
             }
         }
 
-        let _ = to; // touched for future name-consistency check
-        // TODO(phase-43 v2): PII scan + worker-name consistency.
+        // ── PII scan on body + subject combined ──
+        let scanned = format!(
+            "{} {}",
+            value.get("subject").and_then(|v| v.as_str()).unwrap_or(""),
+            body
+        );
+        if contains_ssn_pattern(&scanned) {
+            return Err(ValidationError::Policy {
+                reason: "body contains an SSN-shaped sequence (NNN-NN-NNNN); strip before send".into(),
+            });
+        }
+        if contains_salary_disclosure(&scanned) {
+            return Err(ValidationError::Policy {
+                reason: "body discloses salary/compensation amount; staffing PII rule says strip before send".into(),
+            });
+        }
+
+        // ── Worker-name consistency ──
+        let candidate_id = value.get("_context")
+            .and_then(|c| c.get("candidate_id"))
+            .and_then(|v| v.as_str());
+        let mut findings: Vec<crate::Finding> = vec![];
+        if let Some(cid) = candidate_id {
+            match self.workers.find(cid) {
+                Some(worker) => {
+                    // Body should mention the worker's name (or at least
+                    // their first name) — drafts that address a different
+                    // person than the contracted worker are a recurring
+                    // class of LLM mistake.
+                    let first = worker.name.split_whitespace().next().unwrap_or(&worker.name);
+                    let body_lower = body.to_lowercase();
+                    let first_lower = first.to_lowercase();
+                    if !first_lower.is_empty() && !body_lower.contains(&first_lower) {
+                        findings.push(crate::Finding {
+                            field: "body".into(),
+                            severity: crate::Severity::Warning,
+                            message: format!(
+                                "body doesn't mention worker first name {first:?} (candidate_id {cid:?})"
+                            ),
+                        });
+                    }
+                    // Also detect *another* worker's name appearing in
+                    // place of the contracted one — outright wrong-target.
+                    // We can only check this when we have a different
+                    // expected name; skip if the body is generic enough.
+                }
+                None => {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "_context.candidate_id {cid:?} not found in worker roster"
+                        ),
+                    });
+                }
+            }
+        }
 
         Ok(Report {
-            findings: vec![],
+            findings,
             elapsed_ms: started.elapsed().as_millis() as u64,
         })
     }
 }
 
+// ─── PII scanners (std-only) ────────────────────────────────────────────
+
+/// Detects an SSN-shaped sequence: 3 digits, dash, 2 digits, dash, 4 digits.
+/// Walks the byte buffer; rejects sequences that are part of a longer run
+/// of digits (so phone-area-code-like NNN-NNN-NNNN isn't flagged). Tight
+/// false-positive surface: it's specifically the NNN-NN-NNNN shape.
+fn contains_ssn_pattern(s: &str) -> bool {
+    let bytes = s.as_bytes();
+    if bytes.len() < 11 { return false; }
+    for i in 0..=bytes.len().saturating_sub(11) {
+        let win = &bytes[i..i + 11];
+        let shape = win.iter().enumerate().all(|(j, &b)| match j {
+            0 | 1 | 2 | 4 | 5 | 7 | 8 | 9 | 10 => b.is_ascii_digit(),
+            3 | 6 => b == b'-',
+            _ => unreachable!(),
+        });
+        if !shape { continue; }
+        // Reject if the byte BEFORE this window is a digit or `-` —
+        // we're inside a longer numeric run, probably not an SSN.
+        if i > 0 {
+            let prev = bytes[i - 1];
+            if prev.is_ascii_digit() || prev == b'-' { continue; }
+        }
+        // Reject if the byte AFTER is a digit or `-` (same reason).
+        if i + 11 < bytes.len() {
+            let next = bytes[i + 11];
+            if next.is_ascii_digit() || next == b'-' { continue; }
+        }
+        return true;
+    }
+    false
+}
+
+/// Detects salary/compensation disclosure: the keywords "salary",
+/// "compensation", "pay rate", "bill rate", "hourly rate" appearing
+/// within ~40 chars of a `$` followed by digits. Coarse on purpose —
+/// it's better to false-positive on a legit phrase like "discuss your
+/// hourly rate of $30/hr" than to miss it.
+fn contains_salary_disclosure(s: &str) -> bool {
+    let lower = s.to_lowercase();
+    const KEYWORDS: &[&str] = &[
+        "salary", "compensation", "pay rate", "bill rate", "hourly rate",
+    ];
+    let mut keyword_positions: Vec<usize> = vec![];
+    for kw in KEYWORDS {
+        let mut start = 0;
+        while let Some(found) = lower[start..].find(kw) {
+            let abs = start + found;
+            keyword_positions.push(abs);
+            start = abs + kw.len();
+        }
+    }
+    if keyword_positions.is_empty() { return false; }
+
+    // Find every `$NNN+` in the text.
+    let bytes = lower.as_bytes();
+    let mut dollar_positions: Vec<usize> = vec![];
+    for (i, &b) in bytes.iter().enumerate() {
+        if b == b'$' && i + 1 < bytes.len() && bytes[i + 1].is_ascii_digit() {
+            dollar_positions.push(i);
+        }
+    }
+    if dollar_positions.is_empty() { return false; }
+
+    // Any (keyword, $) pair within 40 chars triggers the policy rule.
+    for &kp in &keyword_positions {
+        for &dp in &dollar_positions {
+            if kp.abs_diff(dp) <= 40 {
+                return true;
+            }
+        }
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::{InMemoryWorkerLookup, WorkerRecord};
+    use serde_json::json;
+
+    fn lookup(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn worker(id: &str, name: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: id.into(),
+            name: name.into(),
+            status: "active".into(),
+            city: None, state: None, role: None,
+            blacklisted_clients: vec![],
+        }
+    }
 
     #[test]
     fn long_sms_fails_completeness() {
+        let v = EmailValidator::new(lookup(vec![]));
         let body = "x".repeat(200);
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
-            "to": "+15555550123",
-            "body": body,
-            "kind": "sms"
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "+15555550123", "body": body, "kind": "sms"
         })));
         assert!(matches!(r, Err(ValidationError::Completeness { .. })));
     }
 
     #[test]
     fn long_email_subject_fails_completeness() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
-            "to": "a@b.com",
-            "body": "hi",
-            "subject": "x".repeat(100)
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "a@b.com", "body": "hi", "subject": "x".repeat(100)
         })));
         assert!(matches!(r, Err(ValidationError::Completeness { .. })));
     }
 
     #[test]
     fn missing_to_fails_schema() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({"body": "hi"})));
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({"body": "hi"})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field == "to"));
     }
 
     #[test]
     fn well_formed_email_passes() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
             "to": "hiring@example.com",
             "subject": "Interview: Friday 10am",
             "body": "Hi Jane — confirming interview Friday 10am."
         })));
         assert!(r.is_ok(), "well-formed email should pass: {:?}", r);
     }
+
+    #[test]
+    fn ssn_in_body_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane — your file shows 123-45-6789 on record."
+        })));
+        match r {
+            Err(ValidationError::Policy { reason }) => assert!(reason.contains("SSN")),
+            other => panic!("expected Policy SSN error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn ssn_in_subject_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "subject": "Re: ID 123-45-6789",
+            "body": "details inside"
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn phone_number_does_not_trigger_ssn_false_positive() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Call me at 555-123-4567 to confirm."
+        })));
+        assert!(r.is_ok(), "phone NNN-NNN-NNNN should NOT match SSN NNN-NN-NNNN: {:?}", r);
+    }
+
+    #[test]
+    fn salary_disclosure_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Confirming your hourly rate of $32.50 per hour."
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn discussing_dollars_without_salary_keyword_passes() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "The $20 parking pass is at the front desk."
+        })));
+        assert!(r.is_ok(), "non-salary $ should pass: {:?}", r);
+    }
+
+    #[test]
+    fn unknown_candidate_id_fails_consistency() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane",
+            "_context": {"candidate_id": "W-FAKE"}
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("not found")),
+            other => panic!("expected Consistency, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn missing_first_name_in_body_is_warning() {
+        let v = EmailValidator::new(lookup(vec![worker("W-1", "Jane Doe")]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi there — confirming your interview Friday.",
+            "_context": {"candidate_id": "W-1"}
+        })));
+        let report = r.expect("missing name should be warning, not error");
+        assert_eq!(report.findings.len(), 1);
+        assert_eq!(report.findings[0].severity, crate::Severity::Warning);
+        assert!(report.findings[0].message.to_lowercase().contains("first name"));
+    }
+
+    #[test]
+    fn matching_first_name_passes_clean() {
+        let v = EmailValidator::new(lookup(vec![worker("W-1", "Jane Doe")]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane — confirming your interview Friday.",
+            "_context": {"candidate_id": "W-1"}
+        })));
+        let report = r.expect("matching name should pass");
+        assert!(report.findings.is_empty(), "expected no findings, got {:?}", report.findings);
+    }
 }

crates/validator/src/staffing/fill.rs

@@ -1,22 +1,67 @@
-//! Fill-proposal validator.
+//! Fill-proposal validator (Phase 43 v2 — real consistency checks).
 //!
 //! PRD checks:
-//!   - Schema compliance (propose_done shape matches
-//!     `{fills: [{candidate_id, name}]}`)
+//!   - Schema compliance (propose_done shape: `{fills: [{candidate_id, name}]}`)
 //!   - Completeness (endorsed count == target_count)
-//!   - Worker existence (every candidate_id present in workers_500k)
-//!   - Status check (active, not_on_client_blacklist)
+//!   - Worker existence (every candidate_id present in workers roster)
+//!   - Status check (worker.status == "active")
+//!   - Client blacklist (worker NOT in client.blacklisted_clients)
 //!   - Geo/role match (worker city/state/role matches contract)
 //!
-//! Today this is a scaffold — schema check is real (it's cheap); the
-//! worker-existence / status / geo checks need a catalog lookup and
-//! land in a follow-up when the catalog query helper is wired into
-//! this crate.
+//! The contract metadata (target_count, city, state, role, client_id)
+//! travels alongside the JSON payload under a `_context` key:
+//! `{"_context": {"target_count": 2, "city": "Toledo", "state": "OH",
+//!   "role": "Welder", "client_id": "CLI-00099"}, "fills": [...]}`.
+//! This keeps the Validator trait signature stable while letting the
+//! validator cross-check fills against contract truth.
+//!
+//! Worker-existence + status + geo + blacklist all share a single
+//! lookup trait (`WorkerLookup`) so the validator stays decoupled
+//! from queryd / parquet / catalogd transport details.
 
-use crate::{Artifact, Report, Validator, ValidationError};
+use crate::{
+    Artifact, Report, Validator, ValidationError, WorkerLookup, WorkerRecord,
+};
+use std::sync::Arc;
 use std::time::Instant;
 
-pub struct FillValidator;
+pub struct FillValidator {
+    workers: Arc<dyn WorkerLookup>,
+}
+
+impl FillValidator {
+    pub fn new(workers: Arc<dyn WorkerLookup>) -> Self {
+        Self { workers }
+    }
+}
+
+#[derive(Debug, Default)]
+struct FillContext {
+    target_count: Option<usize>,
+    city: Option<String>,
+    state: Option<String>,
+    role: Option<String>,
+    client_id: Option<String>,
+}
+
+fn extract_context(value: &serde_json::Value) -> FillContext {
+    let ctx_obj = value.get("_context").and_then(|c| c.as_object());
+    let ctx = match ctx_obj {
+        Some(o) => o,
+        None => return FillContext::default(),
+    };
+    FillContext {
+        target_count: ctx.get("target_count").and_then(|v| v.as_u64()).map(|n| n as usize),
+        city: ctx.get("city").and_then(|v| v.as_str()).map(String::from),
+        state: ctx.get("state").and_then(|v| v.as_str()).map(String::from),
+        role: ctx.get("role").and_then(|v| v.as_str()).map(String::from),
+        client_id: ctx.get("client_id").and_then(|v| v.as_str()).map(String::from),
+    }
+}
+
+fn eq_ci(a: &str, b: &str) -> bool {
+    a.trim().eq_ignore_ascii_case(b.trim())
+}
 
 impl Validator for FillValidator {
     fn name(&self) -> &'static str { "staffing.fill" }
@@ -31,9 +76,7 @@ impl Validator for FillValidator {
             }),
         };
 
-        // Schema check — the only real validation shipped in this
-        // scaffold. Catches the common "model emitted prose instead of
-        // JSON" failure mode before the consistency checks even run.
+        // ── Schema check ──
         let fills = value.get("fills").and_then(|f| f.as_array()).ok_or(
             ValidationError::Schema {
                 field: "fills".into(),
@@ -55,12 +98,116 @@ impl Validator for FillValidator {
             }
         }
 
-        // TODO(phase-43 v2): worker-existence / status / geo checks.
-        // Need a catalog query handle injected into FillValidator's
-        // constructor — out of scope for the scaffold.
+        let ctx = extract_context(value);
+
+        // ── Completeness: count match ──
+        if let Some(target) = ctx.target_count {
+            if fills.len() != target {
+                return Err(ValidationError::Completeness {
+                    reason: format!(
+                        "endorsed count {} != target_count {target}",
+                        fills.len()
+                    ),
+                });
+            }
+        }
+
+        // ── Cross-roster checks ──
+        let mut findings: Vec<crate::Finding> = vec![];
+        let mut seen_ids = std::collections::HashSet::new();
+        for (i, fill) in fills.iter().enumerate() {
+            let candidate_id = fill.get("candidate_id").and_then(|v| v.as_str()).unwrap_or("");
+            let proposed_name = fill.get("name").and_then(|v| v.as_str()).unwrap_or("");
+
+            // Duplicate-ID guard inside one fill.
+            if !seen_ids.insert(candidate_id.to_string()) {
+                return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "duplicate candidate_id {candidate_id:?} appears multiple times in fills"
+                    ),
+                });
+            }
+
+            // Worker existence — the gate that catches phantom IDs the
+            // model fabricates. This is the load-bearing check for
+            // the 0→85% pattern.
+            let worker: WorkerRecord = match self.workers.find(candidate_id) {
+                Some(w) => w,
+                None => return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "fills[{i}].candidate_id {candidate_id:?} does not exist in worker roster"
+                    ),
+                }),
+            };
+
+            // Status — only "active" workers can be endorsed.
+            if !eq_ci(&worker.status, "active") {
+                return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "fills[{i}] worker {candidate_id:?} has status {:?}, expected \"active\"",
+                        worker.status
+                    ),
+                });
+            }
+
+            // Client blacklist.
+            if let Some(client) = ctx.client_id.as_deref() {
+                if worker.blacklisted_clients.iter().any(|b| eq_ci(b, client)) {
+                    return Err(ValidationError::Policy {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} blacklisted for client {client:?}"
+                        ),
+                    });
+                }
+            }
+
+            // Geo / role match — warn-level when missing context, hard
+            // fail on mismatch with explicit contract values.
+            if let (Some(want_city), Some(have_city)) = (ctx.city.as_deref(), worker.city.as_deref()) {
+                if !eq_ci(want_city, have_city) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} city {have_city:?} doesn't match contract city {want_city:?}"
+                        ),
+                    });
+                }
+            }
+            if let (Some(want_state), Some(have_state)) = (ctx.state.as_deref(), worker.state.as_deref()) {
+                if !eq_ci(want_state, have_state) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} state {have_state:?} doesn't match contract state {want_state:?}"
+                        ),
+                    });
+                }
+            }
+            if let (Some(want_role), Some(have_role)) = (ctx.role.as_deref(), worker.role.as_deref()) {
+                if !eq_ci(want_role, have_role) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} role {have_role:?} doesn't match contract role {want_role:?}"
+                        ),
+                    });
+                }
+            }
+
+            // Name-mismatch is a warning, not an error — recruiters
+            // sometimes send updated names through the proposal layer
+            // before the roster is updated.
+            if !proposed_name.is_empty() && !eq_ci(proposed_name, &worker.name) {
+                findings.push(crate::Finding {
+                    field: format!("fills[{i}].name"),
+                    severity: crate::Severity::Warning,
+                    message: format!(
+                        "proposed name {proposed_name:?} differs from roster name {:?} for {candidate_id:?}",
+                        worker.name
+                    ),
+                });
+            }
+        }
 
         Ok(Report {
-            findings: vec![],
+            findings,
             elapsed_ms: started.elapsed().as_millis() as u64,
         })
     }
@@ -69,35 +216,168 @@ impl Validator for FillValidator {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::InMemoryWorkerLookup;
+    use serde_json::json;
+
+    fn lookup(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn worker(id: &str, name: &str, status: &str, city: &str, state: &str, role: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: id.into(),
+            name: name.into(),
+            status: status.into(),
+            city: Some(city.into()),
+            state: Some(state.into()),
+            role: Some(role.into()),
+            blacklisted_clients: vec![],
+        }
+    }
 
     #[test]
     fn wrong_artifact_type_fails_schema() {
-        let r = FillValidator.validate(&Artifact::EmailDraft(serde_json::json!({})));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({})));
         assert!(matches!(r, Err(ValidationError::Schema { .. })));
     }
 
     #[test]
     fn missing_fills_array_fails_schema() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({})));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::FillProposal(json!({})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field == "fills"));
     }
 
     #[test]
     fn fill_without_candidate_id_fails() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({
-            "fills": [{"name": "Jane"}]
-        })));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::FillProposal(json!({"fills": [{"name": "Jane"}]})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field.contains("candidate_id")));
     }
 
     #[test]
-    fn well_formed_proposal_passes_schema() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({
+    fn well_formed_proposal_with_real_workers_passes() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane Doe", "active", "Toledo", "OH", "Welder"),
+            worker("W-2", "John Smith", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
             "fills": [
-                {"candidate_id": "W-123", "name": "Jane Doe"},
-                {"candidate_id": "W-456", "name": "John Smith"}
+                {"candidate_id": "W-1", "name": "Jane Doe"},
+                {"candidate_id": "W-2", "name": "John Smith"}
             ]
         })));
-        assert!(r.is_ok(), "well-formed proposal should pass schema: {:?}", r);
+        assert!(r.is_ok(), "expected pass, got {:?}", r);
+    }
+
+    #[test]
+    fn phantom_candidate_id_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Toledo", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-FAKE-99999", "name": "Imaginary"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("does not exist")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn inactive_worker_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "inactive", "Toledo", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("inactive")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn wrong_city_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Cincinnati", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.to_lowercase().contains("city")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn wrong_role_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Toledo", "OH", "Driver")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.to_lowercase().contains("role")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn count_mismatch_fails_completeness() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        assert!(matches!(r, Err(ValidationError::Completeness { .. })));
+    }
+
+    #[test]
+    fn duplicate_candidate_id_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [
+                {"candidate_id": "W-1", "name": "Jane"},
+                {"candidate_id": "W-1", "name": "Jane"}
+            ]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("duplicate")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn blacklisted_worker_fails_policy() {
+        let mut w = worker("W-1", "Jane", "active", "Toledo", "OH", "Welder");
+        w.blacklisted_clients = vec!["CLI-00099".into()];
+        let v = FillValidator::new(lookup(vec![w]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder", "client_id": "CLI-00099"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn name_mismatch_is_warning_not_error() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane Doe", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Janet Doe"}]
+        })));
+        let report = r.expect("name mismatch should be warning, not error");
+        assert_eq!(report.findings.len(), 1);
+        assert_eq!(report.findings[0].severity, crate::Severity::Warning);
+        assert!(report.findings[0].message.contains("differs from roster"));
     }
 }

crates/validator/src/staffing/mod.rs

@@ -6,3 +6,4 @@
 pub mod fill;
 pub mod email;
 pub mod playbook;
+pub mod parquet_lookup;

crates/validator/src/staffing/parquet_lookup.rs

@@ -0,0 +1,165 @@
+//! Production WorkerLookup backed by a workers_500k.parquet snapshot.
+//!
+//! Loads the full roster into memory at startup (one-shot). 500K rows
+//! at ~150 bytes per WorkerRecord ≈ 75 MB resident — fine for any
+//! production lakehouse process. Refresh is intentionally
+//! caller-driven (call `from_parquet` again to rebuild) rather than
+//! automatic — operators decide when staffing data has changed enough
+//! to justify the few-second reload.
+//!
+//! Schema mapping (workers_500k.parquet → WorkerRecord):
+//!   worker_id (int64)     → candidate_id = "W-{id}"
+//!   name (string)         → name
+//!   role (string)         → role
+//!   city (string)         → city
+//!   state (string)        → state
+//!   availability (double) → status: "active" if >0 else "inactive"
+//!
+//! No status column on workers_500k, so we derive from availability —
+//! the floor convention used elsewhere in the lakehouse staffing
+//! pipeline. Workers with availability=0.0 are treated as inactive
+//! (vacation, suspended, etc.). Once the Track-A.B `_safe` view ships
+//! with proper `status`, switch this loader to read it directly.
+//!
+//! Blacklist join is not done here — caller is expected to populate
+//! `blacklisted_clients` from a separate source (Phase 43 PRD says
+//! `client_blacklist` table; not yet defined). Default empty.
+
+use crate::{InMemoryWorkerLookup, WorkerLookup, WorkerRecord};
+use parquet::file::reader::{FileReader, SerializedFileReader};
+use parquet::record::Field;
+use std::fs::File;
+use std::path::Path;
+use std::sync::Arc;
+
+#[derive(Debug, thiserror::Error)]
+pub enum LookupLoadError {
+    #[error("opening parquet at {path}: {source}")]
+    Open { path: String, #[source] source: std::io::Error },
+    #[error("parsing parquet at {path}: {source}")]
+    Parse { path: String, #[source] source: parquet::errors::ParquetError },
+    #[error("missing required column {column}")]
+    MissingColumn { column: String },
+    #[error("row {row}: {reason}")]
+    BadRow { row: usize, reason: String },
+}
+
+/// Build an `InMemoryWorkerLookup` from a workers_500k-shaped parquet
+/// file. Returned as `Arc<dyn WorkerLookup>` to drop into validator
+/// constructors.
+pub fn load_workers_parquet(path: &Path) -> Result<Arc<dyn WorkerLookup>, LookupLoadError> {
+    let file = File::open(path).map_err(|e| LookupLoadError::Open {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+    let reader = SerializedFileReader::new(file).map_err(|e| LookupLoadError::Parse {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+
+    // Validate schema covers what we need before iterating rows.
+    let schema = reader.metadata().file_metadata().schema();
+    let column_names: Vec<&str> = schema.get_fields().iter().map(|f| f.name()).collect();
+    for required in &["worker_id", "name", "role", "city", "state", "availability"] {
+        if !column_names.contains(required) {
+            return Err(LookupLoadError::MissingColumn { column: (*required).to_string() });
+        }
+    }
+
+    let row_iter = reader.get_row_iter(None).map_err(|e| LookupLoadError::Parse {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+
+    let mut records: Vec<WorkerRecord> = Vec::with_capacity(reader.metadata().file_metadata().num_rows() as usize);
+    let mut row_idx = 0usize;
+    for row_result in row_iter {
+        let row = row_result.map_err(|e| LookupLoadError::Parse {
+            path: path.display().to_string(),
+            source: e,
+        })?;
+        let mut worker_id: Option<i64> = None;
+        let mut name: Option<String> = None;
+        let mut role: Option<String> = None;
+        let mut city: Option<String> = None;
+        let mut state: Option<String> = None;
+        let mut availability: f64 = 0.0;
+        for (col_name, field) in row.get_column_iter() {
+            match (col_name.as_str(), field) {
+                ("worker_id", Field::Long(v)) => worker_id = Some(*v),
+                ("worker_id", Field::Int(v)) => worker_id = Some(*v as i64),
+                ("name", Field::Str(v)) => name = Some(v.clone()),
+                ("role", Field::Str(v)) => role = Some(v.clone()),
+                ("city", Field::Str(v)) => city = Some(v.clone()),
+                ("state", Field::Str(v)) => state = Some(v.clone()),
+                ("availability", Field::Double(v)) => availability = *v,
+                ("availability", Field::Float(v)) => availability = *v as f64,
+                _ => { /* extra columns ignored */ }
+            }
+        }
+        let id = worker_id.ok_or_else(|| LookupLoadError::BadRow {
+            row: row_idx,
+            reason: "worker_id missing or non-integer".into(),
+        })?;
+        let nm = name.ok_or_else(|| LookupLoadError::BadRow {
+            row: row_idx,
+            reason: "name missing".into(),
+        })?;
+        records.push(WorkerRecord {
+            candidate_id: format!("W-{id}"),
+            name: nm,
+            // status derived from availability (workers_500k has no
+            // status column). 0.0 → inactive, >0.0 → active.
+            status: if availability > 0.0 { "active".into() } else { "inactive".into() },
+            city,
+            state,
+            role,
+            blacklisted_clients: vec![],
+        });
+        row_idx += 1;
+    }
+
+    tracing::info!(
+        target: "validator.parquet_lookup",
+        rows = records.len(),
+        path = %path.display(),
+        "loaded workers parquet snapshot"
+    );
+
+    Ok(Arc::new(InMemoryWorkerLookup::from_records(records)))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::PathBuf;
+
+    /// Smoke test against the live workers_500k.parquet on disk.
+    /// Skipped automatically if the file isn't present (CI / sparse
+    /// checkouts) so the test suite stays portable.
+    #[test]
+    fn load_real_workers_500k() {
+        let path = PathBuf::from("/home/profit/lakehouse/data/datasets/workers_500k.parquet");
+        if !path.exists() {
+            eprintln!("skip: {} not present", path.display());
+            return;
+        }
+        let lookup = load_workers_parquet(&path).expect("load");
+        // Basic shape: at least one worker resolves and has the
+        // expected fields populated.
+        let probe = lookup.find("W-1");
+        assert!(probe.is_some(), "W-1 should exist in 500K-row parquet");
+        let w = probe.unwrap();
+        assert!(!w.name.is_empty(), "name should be populated");
+        assert!(w.status == "active" || w.status == "inactive");
+        assert!(w.role.is_some());
+        assert!(w.city.is_some());
+        assert!(w.state.is_some());
+    }
+
+    #[test]
+    fn missing_file_returns_error() {
+        let r = load_workers_parquet(Path::new("/nonexistent.parquet"));
+        assert!(matches!(r, Err(LookupLoadError::Open { .. })));
+    }
+}

crates/vectord-lance/src/lib.rs

@@ -603,3 +603,210 @@ fn row_from_batch(batch: &RecordBatch, row: usize) -> Result<Row, String> {
 
     Ok(Row { doc_id, chunk_text, vector: v, source, chunk_idx })
 }
+
+// =================== Tests ===================
+//
+// All tests run against a temp directory — never the production
+// data/lance/ tree. Lance reads/writes are async + filesystem-bound,
+// so we use #[tokio::test]. Each test uses a unique per-pid + per-
+// nanosecond temp dir so concurrent runs don't collide and a re-run
+// of a single test doesn't see prior state.
+//
+// Surfaced 2026-05-02 audit: vectord-lance had ZERO tests despite
+// being on the live HTTP path. These are the load-bearing locks for
+// the public API contract.
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn temp_path(label: &str) -> String {
+        // Per-process atomic counter — guarantees uniqueness regardless
+        // of clock resolution or test scheduling. Combined with pid, the
+        // result is unique within and across processes for any practical
+        // test workload. Nanosecond timestamps were not enough on their
+        // own: opus WARN at lib.rs:622 from the 2026-05-02 scrum noted
+        // that under tokio scheduling, multiple tests in the same cargo
+        // process can hit the same nanos bucket.
+        use std::sync::atomic::{AtomicU64, Ordering};
+        static COUNTER: AtomicU64 = AtomicU64::new(0);
+        let seq = COUNTER.fetch_add(1, Ordering::Relaxed);
+        let pid = std::process::id();
+        let nanos = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .map(|d| d.subsec_nanos())
+            .unwrap_or(0);
+        std::env::temp_dir()
+            .join(format!("vlance_test_{label}_{pid}_{nanos}_{seq}"))
+            .to_string_lossy()
+            .to_string()
+    }
+
+    /// Build a minimal in-memory Parquet file matching vectord's
+    /// binary-blob schema. Used as input to migrate_from_parquet_bytes.
+    fn synth_parquet_bytes(n_rows: usize, dims: usize) -> Vec<u8> {
+        use parquet::arrow::ArrowWriter;
+        use std::io::Cursor;
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("source", DataType::Utf8, true),
+            Field::new("doc_id", DataType::Utf8, false),
+            Field::new("chunk_idx", DataType::Int32, true),
+            Field::new("chunk_text", DataType::Utf8, true),
+            Field::new("vector", DataType::Binary, false),
+        ]));
+
+        let sources: Vec<Option<&str>> = (0..n_rows).map(|_| Some("test")).collect();
+        let doc_ids: Vec<String> = (0..n_rows).map(|i| format!("DOC-{i:04}")).collect();
+        let chunk_idxs: Vec<Option<i32>> = (0..n_rows).map(|i| Some(i as i32)).collect();
+        let chunk_texts: Vec<String> = (0..n_rows).map(|i| format!("synth chunk {i}")).collect();
+        let vectors: Vec<Vec<u8>> = (0..n_rows).map(|i| {
+            let v: Vec<f32> = (0..dims).map(|j| (i * dims + j) as f32 * 0.01).collect();
+            let mut bytes = Vec::with_capacity(dims * 4);
+            for f in v { bytes.extend_from_slice(&f.to_le_bytes()); }
+            bytes
+        }).collect();
+
+        let batch = RecordBatch::try_new(schema.clone(), vec![
+            Arc::new(StringArray::from(sources)),
+            Arc::new(StringArray::from(doc_ids)),
+            Arc::new(Int32Array::from(chunk_idxs)),
+            Arc::new(StringArray::from(chunk_texts)),
+            Arc::new(BinaryArray::from(vectors.iter().map(|v| v.as_slice()).collect::<Vec<_>>())),
+        ]).expect("synth parquet batch");
+
+        let mut buf = Cursor::new(Vec::new());
+        let mut writer = ArrowWriter::try_new(&mut buf, schema, None).expect("arrow writer");
+        writer.write(&batch).expect("write batch");
+        writer.close().expect("close writer");
+        buf.into_inner()
+    }
+
+    #[tokio::test]
+    async fn fresh_store_reports_no_state() {
+        let path = temp_path("fresh");
+        let store = LanceVectorStore::new(path.clone());
+        assert_eq!(store.path(), path);
+        assert_eq!(store.count().await.unwrap_or(0), 0);
+        assert!(!store.has_vector_index().await.unwrap_or(true));
+    }
+
+    #[tokio::test]
+    async fn migrate_then_count_and_fetch() {
+        let path = temp_path("migrate_fetch");
+        let store = LanceVectorStore::new(path.clone());
+        let bytes = synth_parquet_bytes(8, 4);
+
+        let stats = store.migrate_from_parquet_bytes(&bytes).await.expect("migrate");
+        assert_eq!(stats.rows_written, 8);
+        assert_eq!(stats.dimensions, 4);
+        assert!(stats.disk_bytes > 0, "lance dataset should occupy disk");
+
+        assert_eq!(store.count().await.unwrap(), 8);
+
+        let row = store.get_by_doc_id("DOC-0003").await
+            .expect("get_by_doc_id Ok").expect("DOC-0003 exists");
+        assert_eq!(row.doc_id, "DOC-0003");
+        assert_eq!(row.chunk_text, "synth chunk 3");
+        assert_eq!(row.vector.len(), 4);
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Load-bearing contract: get_by_doc_id distinguishes "dataset
+    /// missing" (Err) from "id missing" (Ok(None)) so the HTTP
+    /// handler can return 404 without inspecting error strings.
+    #[tokio::test]
+    async fn get_by_doc_id_missing_returns_none() {
+        let path = temp_path("missing_id");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let row = store.get_by_doc_id("DOC-NEVER-EXISTS").await.expect("Ok");
+        assert!(row.is_none(), "missing id must return Ok(None), not Err");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Verifies the load-bearing structural-difference claim of
+    /// ADR-019: Lance appends without rewriting the whole file. Row
+    /// count grows; new rows are fetchable by their doc_ids.
+    #[tokio::test]
+    async fn append_grows_count_and_new_rows_fetchable() {
+        let path = temp_path("append");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+        assert_eq!(store.count().await.unwrap(), 4);
+
+        let stats = store.append(
+            Some("appended".into()),
+            vec!["NEW-A".into(), "NEW-B".into()],
+            vec![0, 0],
+            vec!["new chunk a".into(), "new chunk b".into()],
+            vec![vec![0.1, 0.2, 0.3, 0.4], vec![0.5, 0.6, 0.7, 0.8]],
+        ).await.expect("append");
+
+        assert_eq!(stats.rows_appended, 2);
+        assert_eq!(store.count().await.unwrap(), 6);
+
+        let new_a = store.get_by_doc_id("NEW-A").await.unwrap().expect("NEW-A");
+        assert_eq!(new_a.chunk_text, "new chunk a");
+        assert_eq!(new_a.source.as_deref(), Some("appended"));
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Without this guard a dim-mismatch row would land on disk and
+    /// silently break search at query time.
+    #[tokio::test]
+    async fn append_dim_mismatch_errors() {
+        let path = temp_path("dim_mismatch");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let err = store.append(
+            None, vec!["X".into(), "Y".into()], vec![0, 0],
+            vec!["a".into(), "b".into()],
+            vec![vec![1.0, 2.0, 3.0, 4.0], vec![1.0, 2.0]],
+        ).await;
+        assert!(err.is_err(), "dim mismatch must error");
+        let msg = err.unwrap_err();
+        assert!(msg.contains("dim") || msg.contains("expected"),
+            "error must mention the dimension problem; got: {msg}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Search round-trip: query the exact vector for one row, top-1
+    /// must be that row. Verifies the search path works on small
+    /// datasets where IVF training would normally be skipped.
+    #[tokio::test]
+    async fn search_returns_nearest() {
+        let path = temp_path("search");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(8, 4)).await.expect("migrate");
+
+        let target: Vec<f32> = (0..4).map(|j| (5 * 4 + j) as f32 * 0.01).collect();
+        let hits = store.search(&target, 3, None, None).await.expect("search");
+        assert!(!hits.is_empty(), "search must return at least 1 hit");
+        assert_eq!(hits[0].doc_id, "DOC-0005",
+            "exact-vector match should be top-1; got {hits:?}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// stats() summarizes the dataset state in one call. Locks the
+    /// field shape so downstream consumers don't break on a rename.
+    #[tokio::test]
+    async fn stats_reports_post_migrate_state() {
+        let path = temp_path("stats");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(5, 4)).await.expect("migrate");
+
+        let s = store.stats().await.expect("stats");
+        assert_eq!(s.rows, 5);
+        assert!(s.disk_bytes > 0);
+        assert!(!s.has_vector_index, "no vector index built yet");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+}

crates/vectord/src/pathway_memory.rs

@@ -373,6 +373,11 @@ pub struct PathwayMemory {
 #[derive(Debug, Serialize)]
 pub struct HotSwapCandidate {
     pub pathway_id: String,
+    /// trace_uid of the SPECIFIC trace this hot-swap recommendation
+    /// came from. Lets a caller call /pathway/retire with single-trace
+    /// precision when observer rejects the result — the audit-consensus
+    /// → retire wire (HANDOVER §queued, ADR-021).
+    pub trace_uid: String,
     pub similarity: f32,
     pub replay_count: u32,
     pub success_rate: f32,
@@ -754,6 +759,7 @@ impl PathwayMemory {
         let accepted = p.ladder_attempts.iter().find(|a| a.accepted)?;
         Some(HotSwapCandidate {
             pathway_id: p.pathway_id.clone(),
+            trace_uid: p.trace_uid.clone(),
             similarity,
             replay_count: p.replay_count,
             success_rate: p.success_rate(),
@@ -919,7 +925,7 @@ mod tests {
             reject_reason: None,
         }];
         let mut trace = PathwayTrace {
-            pathway_id,
+            pathway_id: pathway_id.clone(),
             task_class: "scrum_review".into(),
             file_path: format!("crates/{id_tag}/src/x.rs"),
             signal_class: Some("CONVERGING".into()),
@@ -948,6 +954,14 @@ mod tests {
             replay_count: replays,
             replays_succeeded: succ,
             retired: false,
+            // Versioning fields added by Mem0 wave (commit 6ac7f61) — defaults
+            // mirror "this trace is the live head with no parent/successor".
+            trace_uid: format!("test-{pathway_id}"),
+            version: 1,
+            parent_trace_uid: None,
+            superseded_at: None,
+            superseded_by_trace_uid: None,
+            retirement_reason: None,
         };
         trace.pathway_vec = build_pathway_vec(&trace);
         trace

crates/vectord/src/rag.rs

@@ -163,7 +163,11 @@ pub async fn query(
     // production caller of the Phase 21 primitives — see audit finding
     // "Phase 21 Rust primitives are wired but not CALLED by any
     // production surface" from 2026-04-21.
-    let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
+    // 2026-04-30 model bump: qwen2.5:latest → qwen3.5:latest to match
+    // the small-model-pipeline local-tier default. Same JSON-clean
+    // property, more capacity. think=Some(false) preserved — RAG hot
+    // path doesn't need reasoning traces; direct answers only.
+    let mut cont_opts = ContinuableOpts::new("qwen3.5:latest");
     cont_opts.max_tokens = Some(512);
     cont_opts.temperature = Some(0.2);
     cont_opts.shape = ResponseShape::Text;
@@ -176,7 +180,7 @@ pub async fn query(
         // echoes whatever Ollama loaded). Use the configured tier model
         // for now; if RAG needs to report the actual resolved model,
         // the runner can add a post-call ps probe later.
-        model: "qwen2.5:latest".to_string(),
+        model: "qwen3.5:latest".to_string(),
         sources: results,
         tokens_generated: None,
     })

crates/vectord/src/service.rs

@@ -146,6 +146,11 @@ pub fn router(state: VectorState) -> Router {
         // Phase 45 slice 3 — doc drift detection + human re-admission.
         .route("/playbook_memory/doc_drift/check/{id}", post(check_doc_drift))
         .route("/playbook_memory/doc_drift/resolve/{id}", post(resolve_doc_drift))
+        // Phase 45 closure (2026-04-27) — batch scan across all active
+        // playbooks. Operator runs this on a schedule (cron or manual);
+        // each newly-detected drift writes a row to
+        // data/_kb/doc_drift_corrections.jsonl for downstream review.
+        .route("/playbook_memory/doc_drift/scan", post(scan_doc_drift))
         // Pathway memory — consensus-designed sidecar (2026-04-24).
         // scrum_master_pipeline POSTs /pathway/insert at the end of each
         // review, calls /pathway/query before running the ladder for a
@@ -1850,10 +1855,10 @@ async fn lance_migrate(
         .map_err(|e| (StatusCode::NOT_FOUND, format!("read parquet: {e}")))?;
 
     let lance_store = state.lance.store_for_new(&index_name, &bucket).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let stats = lance_store.migrate_from_parquet_bytes(&bytes).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     tracing::info!(
         "lance migrate '{}': {} rows, {}d, {} bytes on disk, {:.2}s",
@@ -1861,11 +1866,40 @@ async fn lance_migrate(
         stats.disk_bytes, stats.duration_secs,
     );
 
+    // Auto-build the doc_id btree. The scalar index is what makes
+    // get_doc_by_id O(log n) instead of a full table scan; ADR-019
+    // calls this out as the load-bearing feature for hybrid lookup.
+    // Verified 2026-05-02: skipping this on a 10M-row dataset turns
+    // ~5ms doc-fetch into ~100ms (full scan over 35GB). Cheap to
+    // build (~1.2s on 10M, +269MB on disk) and only runs once per
+    // dataset since `has_scalar_index` short-circuits subsequent calls.
+    let scalar_stats = if !lance_store.has_scalar_index("doc_id").await.unwrap_or(false) {
+        match lance_store.build_scalar_index("doc_id").await {
+            Ok(s) => {
+                tracing::info!(
+                    "lance migrate '{}': doc_id btree built in {:.2}s (+{} bytes)",
+                    index_name, s.build_time_secs, s.disk_bytes_added,
+                );
+                Some(s)
+            }
+            Err(e) => {
+                // Don't fail the whole migrate over a missing btree —
+                // the dataset is still queryable, just slowly. Log it
+                // so it's debuggable.
+                tracing::warn!("lance migrate '{}': doc_id btree build failed (will fall back to scan): {e}", index_name);
+                None
+            }
+        }
+    } else {
+        None
+    };
+
     Ok::<_, (StatusCode, String)>(Json(serde_json::json!({
         "index_name": index_name,
         "bucket": bucket,
         "lance_path": lance_store.path(),
         "stats": stats,
+        "scalar_index": scalar_stats,
     })))
 }
 
@@ -1883,6 +1917,300 @@ fn default_partitions() -> u32 { 316 }   // ≈√100K — sane for the referenc
 fn default_bits() -> u32 { 8 }
 fn default_subvectors() -> u32 { 48 }    // 768/48 = 16 dims per subvector
 
+/// Sanitize a Lance backend error before returning it to the HTTP
+/// caller. Two responsibilities:
+///
+/// 1. Map "dataset not found" patterns to HTTP 404 instead of 500.
+///    A missing index isn't an internal failure — it's a resource
+///    lookup miss, and the response code should reflect that.
+/// 2. Strip server-side filesystem paths and Rust crate registry
+///    paths (`/root/.cargo/registry/src/index.crates.io-...`) from
+///    the message body. An attacker probing the surface shouldn't
+///    learn the server's directory layout or our exact dep versions.
+///
+/// Surfaced 2026-05-02 by the Lance backend audit: missing-index
+/// search returned 500 + leaked the lakehouse data path AND the
+/// .cargo/registry path with crate versions.
+fn sanitize_lance_err(err: String, index_name: &str) -> (StatusCode, String) {
+    // 404 detection — narrowed across two 2026-05-02→03 scrum waves.
+    // First wave (opus WARN service.rs:1908): the original `lower.contains
+    // ("not found")` was too broad — caught "column not found" /
+    // "field not found in schema" which are real 500s. Second wave (opus
+    // WARN service.rs:1949): the looser `mentions_path_missing` branch I
+    // added would 404 on a registry-file error like "/root/.cargo/.../x.rs:
+    // no such file or directory" because it triggers without dataset
+    // context. Drop the standalone path-missing branch; require dataset
+    // context AND a missing-shape phrase. Lance's actual error format
+    // ("Dataset at path X was not found") satisfies this.
+    let lower = err.to_lowercase();
+    let mentions_dataset = lower.contains("dataset");
+    let lance_dataset_missing = mentions_dataset && (
+        lower.contains("not found") || lower.contains("does not exist")
+    );
+    // Excluded shapes — these contain "not found" but are real 500s.
+    let column_or_field = lower.contains("column not found")
+        || lower.contains("field not found")
+        || lower.contains("schema not found");
+    let is_not_found = lance_dataset_missing && !column_or_field;
+    if is_not_found {
+        return (StatusCode::NOT_FOUND, format!("lance dataset not found: {index_name}"));
+    }
+
+    // Path redaction — replace path-shaped substrings with [REDACTED]
+    // rather than truncating, per opus BLOCK at service.rs:1914 from the
+    // 2026-05-02 scrum. The previous `err.split("/home/").next()` returned
+    // Some("") when the error string STARTED with "/home/", erasing the
+    // entire message and falling back to a generic "lance backend error"
+    // that lost all real error context. Replacing keeps the structural
+    // error (the "what failed") while stripping the location.
+    let cleaned = redact_paths(&err)
+        .trim_end_matches([',', ' ', '\n', '\t'])
+        .to_string();
+    let msg = if cleaned.is_empty() {
+        format!("lance backend error on {index_name}")
+    } else {
+        cleaned
+    };
+    (StatusCode::INTERNAL_SERVER_ERROR, msg)
+}
+
+/// Replace absolute-path substrings (under known leak-prone roots) with
+/// "[REDACTED]". Walks the input once, identifying path-shaped runs that
+/// start with one of the configured prefixes and continue until a
+/// path-terminating character (whitespace, quote, comma, paren, EOL).
+///
+/// Linear time, no regex dep. Catches multi-occurrence cases that
+/// `String::split(p).next()` lost. The path-redaction surface intentionally
+/// includes /var, /tmp, /etc, /usr, /opt in addition to /home and
+/// /root/.cargo because Lance/Arrow errors surface system paths in
+/// addition to project paths.
+fn redact_paths(s: &str) -> String {
+    // Two prefix sets:
+    // - ABSOLUTE: paths starting with '/' (always safe to redact)
+    // - RELATIVE: same path bodies but without leading '/' (Lance occasionally
+    //   strips the leading slash when echoing dataset paths back, observed
+    //   live 2026-05-02 — "Dataset at path home/profit/lakehouse/data/lance/x
+    //   was not found"). Match these only when preceded by a non-alpha char
+    //   (start of string, space, colon, etc.) so we don't redact innocent
+    //   tokens like "homecoming" or "etcetera".
+    const ABSOLUTE: &[&str] = &[
+        "/root/.cargo", "/home", "/var", "/tmp", "/etc", "/usr", "/opt",
+    ];
+    const RELATIVE: &[&str] = &[
+        "root/.cargo", "home/", "var/", "tmp/", "etc/", "usr/", "opt/",
+    ];
+    fn is_path_term(b: u8) -> bool {
+        matches!(b, b' ' | b'\t' | b'\n' | b'\r' | b'"' | b'\'' | b',' | b')' | b']' | b'}')
+    }
+    fn is_word_boundary_before(bytes: &[u8], i: usize) -> bool {
+        // True if byte at i-1 is non-alphanumeric (so this position starts
+        // a fresh token). True at start-of-input.
+        if i == 0 { return true; }
+        let b = bytes[i - 1];
+        !(b.is_ascii_alphanumeric() || b == b'_' || b == b'.' || b == b'-')
+    }
+    // Walk by byte index but slice the original &str when emitting, never
+    // cast bytes to char (that would corrupt multi-byte UTF-8 — opus WARN
+    // at service.rs:2018 from the 2026-05-03 re-scrum). Path prefixes are
+    // pure ASCII so byte-level matching is sound; what matters is that
+    // we emit non-matched stretches as &str slices, not byte-by-byte.
+    let bytes = s.as_bytes();
+    let mut out = String::with_capacity(s.len());
+    let mut i = 0;
+    let mut copy_start = 0usize;  // start of an in-progress unmatched run
+    while i < bytes.len() {
+        let mut matched_len: Option<usize> = None;
+        // Try absolute prefixes first (always allowed).
+        for p in ABSOLUTE {
+            let pb = p.as_bytes();
+            if i + pb.len() <= bytes.len() && &bytes[i..i + pb.len()] == pb {
+                let after = i + pb.len();
+                if after == bytes.len() || bytes[after] == b'/' || is_path_term(bytes[after]) {
+                    matched_len = Some(pb.len());
+                    break;
+                }
+            }
+        }
+        // Then relative prefixes — only at word boundaries.
+        if matched_len.is_none() && is_word_boundary_before(bytes, i) {
+            for p in RELATIVE {
+                let pb = p.as_bytes();
+                if i + pb.len() <= bytes.len() && &bytes[i..i + pb.len()] == pb {
+                    matched_len = Some(pb.len());
+                    break;
+                }
+            }
+        }
+        if let Some(prefix_len) = matched_len {
+            // Flush any pending unmatched run as a UTF-8-safe slice.
+            if copy_start < i {
+                out.push_str(&s[copy_start..i]);
+            }
+            out.push_str("[REDACTED]");
+            // Skip past the prefix and the path body (until terminator).
+            let mut j = i + prefix_len;
+            while j < bytes.len() && !is_path_term(bytes[j]) {
+                j += 1;
+            }
+            i = j;
+            copy_start = i;
+        } else {
+            // Advance one CHAR (not one byte) so multi-byte UTF-8 sequences
+            // stay intact in the eventual slice. Look up the next char
+            // boundary using the public API.
+            i += utf8_char_len(bytes, i);
+        }
+    }
+    if copy_start < bytes.len() {
+        out.push_str(&s[copy_start..]);
+    }
+    out
+}
+
+/// Length in bytes of the UTF-8 character starting at byte `i`. Bytes are
+/// guaranteed to be a valid UTF-8 sequence start (callers ensure that).
+fn utf8_char_len(bytes: &[u8], i: usize) -> usize {
+    let b = bytes[i];
+    if b < 0x80 { 1 }
+    else if b < 0xC0 { 1 }  // continuation byte — defensive, shouldn't start here
+    else if b < 0xE0 { 2 }
+    else if b < 0xF0 { 3 }
+    else { 4 }
+}
+
+#[cfg(test)]
+mod sanitize_tests {
+    use super::*;
+
+    #[test]
+    fn redact_path_at_offset_zero() {
+        // Regression: opus BLOCK 2026-05-02. Old impl returned Some("")
+        // when err started with "/home/", erasing the whole message.
+        let out = redact_paths("/home/profit/lakehouse/data/lance not a directory");
+        assert_eq!(out, "[REDACTED] not a directory");
+    }
+
+    #[test]
+    fn redact_keeps_pre_and_post_text() {
+        let out = redact_paths("failed to open /home/profit/lakehouse/data/x for read: ENOENT");
+        assert_eq!(out, "failed to open [REDACTED] for read: ENOENT");
+    }
+
+    #[test]
+    fn redact_multiple_paths() {
+        let out = redact_paths("at /root/.cargo/registry/src/index.crates.io-foo/lance-table-4.0.0/src/io/commit.rs:364:26 from /home/profit/lakehouse");
+        assert!(!out.contains("/root/.cargo"));
+        assert!(!out.contains("/home/"));
+        assert!(out.contains("[REDACTED]"));
+    }
+
+    #[test]
+    fn redact_preserves_quote_terminator() {
+        let out = redact_paths("{\"path\":\"/home/profit/x\",\"err\":\"bad\"}");
+        assert_eq!(out, "{\"path\":\"[REDACTED]\",\"err\":\"bad\"}");
+    }
+
+    #[test]
+    fn is_not_found_narrow_dataset_only() {
+        // Regression: opus WARN 2026-05-02. Old impl 404'd on any "not
+        // found" — including legitimate column/field-not-found 500s.
+        let (status, _) = sanitize_lance_err(
+            "column not found: vector".into(), "test_idx",
+        );
+        assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR);
+
+        let (status, _) = sanitize_lance_err(
+            "dataset not found at /home/profit/lakehouse/data/lance/missing".into(), "test_idx",
+        );
+        assert_eq!(status, StatusCode::NOT_FOUND);
+    }
+
+    #[test]
+    fn redact_does_not_match_prefix_substring() {
+        // /etcetera should NOT trigger /etc redaction.
+        let out = redact_paths("etcetera and /etcd");
+        assert_eq!(out, "etcetera and /etcd");
+    }
+
+    #[test]
+    fn redact_relative_paths_lance_emits() {
+        // 2026-05-02: live missing-index probe surfaced Lance error of the
+        // form "Dataset at path home/profit/lakehouse/data/lance/x was not
+        // found" — leading slash stripped. Need to redact the relative form
+        // when preceded by a word boundary.
+        let out = redact_paths("Dataset at path home/profit/lakehouse/data/lance/x was not found");
+        assert!(!out.contains("home/profit"), "should redact: {out}");
+        assert!(out.contains("Dataset at path"));
+        assert!(out.contains("was not found"));
+    }
+
+    #[test]
+    fn redact_does_not_eat_innocent_prefix_words() {
+        // "homecoming" must NOT trigger "home/" redaction. "Etcetera" must
+        // NOT trigger "etc/" redaction. The word-boundary guard handles this.
+        let out = redact_paths("homecoming etcetera vary tmpfile");
+        assert_eq!(out, "homecoming etcetera vary tmpfile");
+    }
+
+    #[test]
+    fn is_not_found_lance_actual_phrasing() {
+        // Lance's actual error format observed live: "Dataset at path X was
+        // not found: Not found: ...". Must 404, not 500.
+        let (status, _) = sanitize_lance_err(
+            "Dataset at path home/profit/lakehouse/data/lance/x was not found".into(),
+            "x",
+        );
+        assert_eq!(status, StatusCode::NOT_FOUND);
+    }
+
+    #[test]
+    fn is_not_found_excludes_column_field_schema() {
+        // Real 500s with the "not found" phrase that aren't dataset-missing.
+        for err in [
+            "column not found: vector",
+            "field not found in schema: doc_id",
+            "schema not found for dataset xyz",
+        ] {
+            let (status, _) = sanitize_lance_err(err.into(), "test_idx");
+            assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR, "{err}");
+        }
+    }
+
+    #[test]
+    fn is_not_found_does_not_match_unrelated_path_missing() {
+        // Regression: opus WARN at service.rs:1949 from the 2026-05-03
+        // re-scrum. A registry-file error from inside a Lance internal
+        // module should NOT be coerced to 404 just because it contains
+        // "no such file or directory" — it's a real 500.
+        let (status, _) = sanitize_lance_err(
+            "/root/.cargo/registry/src/index.crates.io-foo/lance-table-4.0.0/src/io/commit.rs: no such file or directory".into(),
+            "test_idx",
+        );
+        assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR);
+        // (And the path is still redacted in the message.)
+        let (_, msg) = sanitize_lance_err(
+            "/root/.cargo/registry/src/lance-foo/x.rs: no such file or directory".into(),
+            "test_idx",
+        );
+        assert!(!msg.contains("/root/.cargo"), "path leak: {msg}");
+    }
+
+    #[test]
+    fn redact_preserves_multibyte_utf8() {
+        // Regression: opus WARN at service.rs:2018 from the 2026-05-03
+        // re-scrum. Old impl did `out.push(bytes[i] as char)` which
+        // corrupted multi-byte UTF-8 (e.g. a path containing user-supplied
+        // names with non-ASCII characters) into Latin-1 mojibake.
+        let input = "Failed to open /home/profit/工作/data — café not found";
+        let out = redact_paths(input);
+        // The path is redacted...
+        assert!(!out.contains("/home/profit"), "path leak: {out}");
+        // ...AND the multi-byte characters elsewhere are preserved verbatim.
+        assert!(out.contains("café"), "lost UTF-8: {out}");
+        assert!(out.contains("not found"), "lost trailing context: {out}");
+    }
+}
+
 /// Build the IVF_PQ index on the Lance dataset.
 async fn lance_build_index(
     State(state): State<VectorState>,
@@ -1890,10 +2218,10 @@ async fn lance_build_index(
     Json(req): Json<LanceIndexRequest>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     match lance_store.build_index(req.num_partitions, req.num_bits, req.num_sub_vectors).await {
         Ok(stats) => Ok(Json(stats)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -1942,13 +2270,13 @@ async fn lance_search(
     let qv: Vec<f32> = embed_resp.embeddings[0].iter().map(|&x| x as f32).collect();
 
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let t0 = std::time::Instant::now();
     let nprobes = req.nprobes.or(Some(LANCE_DEFAULT_NPROBES));
     let refine = req.refine_factor.or(Some(LANCE_DEFAULT_REFINE_FACTOR));
     let hits = lance_store.search(&qv, req.top_k, nprobes, refine).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     Ok(Json(serde_json::json!({
         "index_name": index_name,
@@ -1966,7 +2294,7 @@ async fn lance_get_doc(
     Path((index_name, doc_id)): Path<(String, String)>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     let t0 = std::time::Instant::now();
     match lance_store.get_by_doc_id(&doc_id).await {
         Ok(Some(row)) => Ok(Json(serde_json::json!({
@@ -1976,7 +2304,7 @@ async fn lance_get_doc(
             "row": row,
         }))),
         Ok(None) => Err((StatusCode::NOT_FOUND, format!("doc_id not found: {doc_id}"))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -2008,7 +2336,7 @@ async fn lance_append(
         return Err((StatusCode::BAD_REQUEST, "rows array is empty".into()));
     }
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let mut doc_ids = Vec::with_capacity(req.rows.len());
     let mut chunk_idxs = Vec::with_capacity(req.rows.len());
@@ -2539,6 +2867,119 @@ async fn check_doc_drift(
     })))
 }
 
+/// Phase 45 closure (2026-04-27) — POST /playbook_memory/doc_drift/scan
+///
+/// Iterates all active playbooks (non-retired, has doc_refs), runs
+/// drift check against context7 for each, flags drifted entries via
+/// PlaybookMemory::flag_doc_drift, and appends a row to
+/// data/_kb/doc_drift_corrections.jsonl for each drift detected.
+///
+/// Returns aggregate stats so an operator can see at-a-glance how
+/// many playbooks drifted and which tools moved.
+///
+/// Honors entries already flagged: they're counted in `already_flagged`
+/// (no double-flag, no duplicate corrections.jsonl row).
+async fn scan_doc_drift(
+    State(state): State<VectorState>,
+) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
+    use crate::doc_drift::{check_all_refs, DriftCheckerConfig, DriftOutcome};
+
+    let entries = state.playbook_memory.snapshot().await;
+    let now = chrono::Utc::now().to_rfc3339();
+    let cfg = DriftCheckerConfig::default();
+
+    let mut scanned = 0usize;
+    let mut newly_flagged = 0usize;
+    let mut already_flagged = 0usize;
+    let mut skipped_no_refs = 0usize;
+    let mut skipped_retired = 0usize;
+    let mut tool_counts: std::collections::HashMap<String, usize> = Default::default();
+    let mut corrections_rows: Vec<String> = vec![];
+
+    for e in entries.iter() {
+        if e.retired_at.is_some() { skipped_retired += 1; continue; }
+        if e.doc_refs.is_empty() { skipped_no_refs += 1; continue; }
+        if e.doc_drift_flagged_at.is_some() && e.doc_drift_reviewed_at.is_none() {
+            already_flagged += 1;
+            continue;
+        }
+        scanned += 1;
+        let results = check_all_refs(&cfg, &e.doc_refs).await;
+        let drifted_tools: Vec<&str> = results.iter()
+            .filter(|r| matches!(r.outcome, DriftOutcome::Drifted { .. }))
+            .map(|r| r.tool.as_str())
+            .collect();
+        if drifted_tools.is_empty() { continue; }
+
+        // Flag the entry.
+        let flagged = state.playbook_memory.flag_doc_drift(&e.playbook_id).await
+            .unwrap_or(false);
+        if flagged { newly_flagged += 1; }
+        for t in &drifted_tools {
+            *tool_counts.entry(t.to_string()).or_insert(0) += 1;
+        }
+
+        // Build corrections.jsonl row — one per drifted playbook with
+        // the tool list inline. Downstream consumers (overview model,
+        // operator dashboard) read this to decide reviews + revisions.
+        let row = serde_json::json!({
+            "playbook_id": e.playbook_id,
+            "scanned_at": now,
+            "drifted_tools": drifted_tools,
+            "per_tool": results.iter().map(|r| {
+                let (drifted, current, src) = match &r.outcome {
+                    DriftOutcome::Drifted { current_snippet_hash, source_url } =>
+                        (true, Some(current_snippet_hash.clone()), source_url.clone()),
+                    _ => (false, None, None),
+                };
+                serde_json::json!({
+                    "tool": r.tool, "version_seen": r.version_seen,
+                    "drifted": drifted, "current_snippet_hash": current, "source_url": src,
+                })
+            }).collect::<Vec<_>>(),
+            "recommended_action": "review-and-resolve",
+        });
+        corrections_rows.push(row.to_string());
+    }
+
+    // Persist corrections.jsonl row(s) for the operator/overview model.
+    if !corrections_rows.is_empty() {
+        let path = std::path::PathBuf::from("/home/profit/lakehouse/data/_kb/doc_drift_corrections.jsonl");
+        if let Some(parent) = path.parent() {
+            if let Err(e) = tokio::fs::create_dir_all(parent).await {
+                tracing::warn!(target: "vectord.doc_drift", "create_dir_all {parent:?}: {e}");
+            }
+        }
+        let body = corrections_rows.join("\n") + "\n";
+        if let Err(e) = tokio::fs::OpenOptions::new()
+            .create(true).append(true).open(&path).await
+        {
+            tracing::warn!(target: "vectord.doc_drift", "open {path:?}: {e}");
+        } else {
+            use tokio::io::AsyncWriteExt;
+            match tokio::fs::OpenOptions::new().create(true).append(true).open(&path).await {
+                Ok(mut f) => {
+                    if let Err(e) = f.write_all(body.as_bytes()).await {
+                        tracing::warn!(target: "vectord.doc_drift", "append {path:?}: {e}");
+                    }
+                }
+                Err(e) => tracing::warn!(target: "vectord.doc_drift", "reopen {path:?}: {e}"),
+            }
+        }
+    }
+
+    Ok(Json(serde_json::json!({
+        "scanned_at": now,
+        "scanned": scanned,
+        "newly_flagged": newly_flagged,
+        "already_flagged": already_flagged,
+        "skipped_retired": skipped_retired,
+        "skipped_no_refs": skipped_no_refs,
+        "drifted_by_tool": tool_counts,
+        "corrections_written": corrections_rows.len(),
+    })))
+}
+
 /// Phase 45 slice 3 — POST /playbook_memory/doc_drift/resolve/{id}
 ///
 /// Human-in-the-loop re-admission. Stamps `doc_drift_reviewed_at`.

data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json

@@ -11,15 +11,51 @@
     }
   ],
   "created_at": "2026-04-20T11:07:57.308050648Z",
-  "updated_at": "2026-04-22T03:28:28.343843823Z",
+  "updated_at": "2026-04-28T01:28:31.280305207Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "timestamp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "operation",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "approach",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "result",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "context",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null,
+  "row_count": 2077,
   "last_embedded_at": null,
   "embedding_stale_since": null,
   "embedding_refresh_policy": null

data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json

@@ -1,117 +0,0 @@
-{
-  "id": "564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7",
-  "name": "client_workerskjkk",
-  "schema_fingerprint": "cdfe85348885ddf329e5e6e9bf0e2c75c92d1a86fdb0fd3875ed46e3f93c4d82",
-  "objects": [
-    {
-      "bucket": "primary",
-      "key": "datasets/client_workerskjkk.parquet",
-      "size_bytes": 32201,
-      "created_at": "2026-04-21T00:49:04.623625149Z"
-    }
-  ],
-  "created_at": "2026-04-21T00:49:04.623626738Z",
-  "updated_at": "2026-04-21T00:49:04.623901788Z",
-  "description": "",
-  "owner": "",
-  "sensitivity": "pii",
-  "columns": [
-    {
-      "name": "worker_id",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "name",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "role",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "city",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "state",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "email",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "phone",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "skills",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "certifications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "availability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "reliability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "archetype",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    }
-  ],
-  "lineage": {
-    "source_system": "csv",
-    "source_file": "staffing_roster_sample.csv",
-    "ingest_job": "ingest-1776732544623",
-    "ingest_timestamp": "2026-04-21T00:49:04.623625149Z",
-    "parent_datasets": []
-  },
-  "freshness": null,
-  "tags": [],
-  "row_count": 180,
-  "last_embedded_at": null,
-  "embedding_stale_since": null,
-  "embedding_refresh_policy": null
-}

data/_catalog/views/candidates_safe.json

@@ -0,0 +1,24 @@
+{
+  "name": "candidates_safe",
+  "base_dataset": "candidates",
+  "columns": [
+    "candidate_id",
+    "first_name",
+    "city",
+    "state",
+    "skills",
+    "years_experience",
+    "status"
+  ],
+  "row_filter": "status != 'blocked'",
+  "column_redactions": {
+    "candidate_id": {
+      "kind": "mask",
+      "keep_prefix": 3,
+      "keep_suffix": 2
+    }
+  },
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "PII-free candidate projection — drops last_name, email, phone, hourly_rate_usd. candidate_id masked (keep first 3, last 2). Visible to recruiter / mode-runner agents."
+}

data/_catalog/views/jobs_safe.json

@@ -0,0 +1,26 @@
+{
+  "name": "jobs_safe",
+  "base_dataset": "job_orders",
+  "columns": [
+    "job_order_id",
+    "client_id",
+    "title",
+    "vertical",
+    "status",
+    "city",
+    "state",
+    "zip",
+    "bill_rate",
+    "pay_rate"
+  ],
+  "column_redactions": {
+    "client_id": {
+      "kind": "mask",
+      "keep_prefix": 3,
+      "keep_suffix": 2
+    }
+  },
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "Job-order projection with client_id masked. Drops description (often quotes client names verbatim, no text-scrubber available). bill_rate / pay_rate kept — commercial info, not PII per staffing PRD."
+}

data/_catalog/views/workers_safe.json

@@ -0,0 +1,22 @@
+{
+  "name": "workers_safe",
+  "base_dataset": "workers_500k",
+  "columns": [
+    "worker_id",
+    "role",
+    "city",
+    "state",
+    "skills",
+    "certifications",
+    "archetype",
+    "reliability",
+    "responsiveness",
+    "engagement",
+    "compliance",
+    "availability"
+  ],
+  "column_redactions": {},
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "PII-free worker projection — drops name, email, phone, zip, communications, resume_text. resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale. Skills + certifications + scores carry the matching signal for staffing inference. Source for workers_500k_v9 vector corpus rebuild."
+}

docs/ADR-019-vector-storage.md

@@ -70,10 +70,33 @@ But the written rule missed something. It assumed Lance's value would show up as
 
 ### What we keep watching (but don't act on yet)
 
-- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.
+- ~~**Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.~~ **Done 2026-05-02** — see "Follow-up: 10M re-bench" below.
 - **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system.
 - **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot.
 
+---
+
+## Follow-up: 10M re-bench (2026-05-02)
+
+The 10M re-bench above ran. Numbers from `data/lance/scale_test_10m` (33 GB, 10M × 768d, IVF_PQ live, post-doc_id-btree-build). Full report: `reports/lance_10m_rebench_2026-05-02.md`.
+
+| Op | 100K (this ADR) | 10M (re-bench) | Notes |
+|---|---:|---:|---|
+| Search (cold) | 2229μs | ~46ms median | 21× slower at 100× scale → reasonable for IVF_PQ |
+| Search (warm) | (not measured) | ~20ms p50 | Stable across 5 trials |
+| Doc fetch by id | 311μs | ~5ms | Structural ADR-019 win confirmed once `doc_id` btree is built |
+| Index method | lance_ivf_pq | lance_ivf_pq | response tag confirms |
+
+**The HNSW comparison at 10M doesn't exist** — at 10M × 768d × 4 bytes = ~30 GB just for vectors, doubled for the graph. HNSW doesn't fit on a single 128 GB box at this scale. So the original "Lance pulls ahead at 10M" framing is true by elimination: Lance is the only contender that operationally exists at 10M. The strategic question is reframed as "Lance vs Parquet+HNSW-with-spilling," deferred until we have a workload where the Parquet path is the bottleneck (currently tracked in `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` decisions tracker).
+
+**Bug surfaced and fixed during re-bench:** Initial doc-fetch was ~100ms (full table scan). Root cause: the `doc_id` scalar btree was never built — `lance-bench` writes datasets by bypassing `IndexMeta`, and the activation-time auto-build only runs for IndexMeta-registered indexes. Fixed at two layers (commits `5d30b3d` + `044650a`):
+- `lance_migrate` HTTP handler auto-builds the btree inline (~1.2s on 10M, +269MB on disk)
+- `lance-bench` binary builds the btree post-IVF for parity with the gateway path
+
+**Gauntlet added 2026-05-02:** the lance crates had zero tests + no smoke when audited. Now have 7 unit tests in `crates/vectord-lance` + 12 sanitize tests in `crates/vectord` + 10-probe `scripts/lance_smoke.sh` + sanitized error boundary across all 5 routes. See commits `7bb66f0`, `ac7c996`, `e9d17f7` (sanitizer iterations driven by cross-lineage scrum).
+
+**Status:** ADR-019's hybrid architecture (Parquet+HNSW primary, Lance secondary) is now empirically validated up to 10M. The "watch and re-bench" item is closed.
+
 ## Why this isn't moving the goalposts
 
 The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile.

docs/ARCHITECTURE_COMPARISON.md

@@ -0,0 +1,46 @@
+# Lakehouse: Rust vs Go architecture comparison
+
+> **Source of truth lives in the golangLAKEHOUSE repo:**
+> [`/home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`](file:///home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md)
+>
+> J's living document — pulled from there into this repo's docs as
+> a pointer so the comparison is reachable from either side.
+
+## Why the source lives in golangLAKEHOUSE
+
+The Go rewrite was the trigger for the comparison. The doc updates as
+J ships fixes on either side, and most of the open backlog items
+(materializer port, replay port, validators network surface) land in
+the Go repo. Keeping the source there means PR auditing on Go
+commits also catches doc drift.
+
+## When to update from this side
+
+If a fix lands in the Rust repo that changes a comparison value
+(e.g. embed cache change, sidecar drop, new validator), update both:
+
+1. The source at `/home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`
+2. The change log section at the bottom of the same file
+
+This file is a pointer — **do not put authoritative content here.**
+Drift between two copies wastes the discipline.
+
+## Quick links
+
+- **Decisions tracker** — section near the top of the source file.
+  Lists actioned items + open backlog with LOC estimates.
+- **Performance numbers** — Python dependency section. Updated each
+  time a load test is rerun.
+- **Distillation porting status** — table of phase-by-phase port
+  state across runtimes.
+- **Recommendation** — current working hypothesis on Go-primary vs
+  Rust-primary. Subject to change as fixes ship.
+
+## Last known state
+
+- **2026-05-01**: Rust embed cache shipped (`150cc3b`), 236× RPS gain.
+- **2026-05-01**: Go validator port shipped (`b03521a`), production
+  safety net now on Go side.
+- **Open**: Drop Rust Python sidecar (~200 LOC, universal-win).
+- **Open**: Port Rust materializer to Go (~500-800 LOC, unblocks
+  Go-only end-to-end pipeline).

docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md

@@ -0,0 +1,177 @@
+# Phase 1.5 — BIPA Schema Audit + outcomes.jsonl Content Sample
+
+**Status:** Complete — 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md)
+
+> **Scope.** Two follow-up walks defined in `AUDIT_PHASE_1_DISCOVERY.md` §10/C4 + §10/single-reviewer (gemini "culture fit"): (1) BIPA-specific photo/video/biometric column + code path inventory, (2) actual content sample of `data/_kb/outcomes.jsonl` to confirm whether PII or discrimination-proxy reasoning lands there. Read-only walk; no code changes.
+
+---
+
+## §1 — BIPA walk: photo/video/biometric surface
+
+### 1A — The face pipeline
+
+The Rust legacy ships a synthetic face pool with deepface-extracted demographic classifications. **Even though the FACES are synthetic, the LOGIC is BIPA-relevant** — when real photos arrive (J 2026-05-03: "we will" have them), the same pipeline processes them.
+
+| File:line | What | BIPA relevance |
+|---|---|---|
+| `scripts/staffing/fetch_face_pool.py` | Fetches 1000 synthetic faces from `thispersondoesnotexist.com` (StyleGAN). | Synthetic — no biometric collection at this stage. |
+| `scripts/staffing/tag_face_pool.py:18-31` | Runs `deepface` on each face to extract `gender` (man / woman) + `race` (asian / black / hispanic / indian / middle_eastern / white) + `age`. Writes to `data/headshots/manifest.jsonl`. | **BIPA biometric processing.** Deepface extracts demographic classifications from face geometry. The output IS biometric information per 740 ILCS 14/10. |
+| `data/headshots/manifest.jsonl` | 1000 rows, each `{id, file, gender, race, age}`. Sampled rows confirm shape: e.g. `{id: 0, file: "face_0000.jpg", gender: "woman", race: "east_asian", age: 33}`. | The persisted classifications. **For real photos, this file becomes a regulated biometric database.** |
+| `mcp-server/index.ts:1308` (`/headshots/:key` route) | Serves face matched by gender × race × age intersection bucketing for a candidate. | Routing logic predicated on biometric-derived demographic attributes. |
+| `mcp-server/index.ts:1367` (`/headshots/generate/:key`) | ComfyUI on-demand portrait generation. | Custom-generated image keyed to candidate. |
+| `mcp-server/index.ts:1408` (ComfyUI prompt template) | `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic` | **Protected attributes (age + race + gender) embedded directly into the model prompt.** Even with synthetic output, this is system-level encoding of protected-attribute features into AI workflow. |
+
+### 1B — Name-based ethnicity inference (inferred-attribute risk)
+
+`mcp-server/search.html:3375-3432` defines lookup tables that classify candidates by demographic attributes derived from their NAMES (not photos):
+
+| Constant | What |
+|---|---|
+| `FEMALE_NAMES` (~200 entries) + `MALE_NAMES` | Gender inference from first name |
+| `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` | Ethnicity inference from first name |
+| `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` | Ethnicity inference from surname (more diagnostic than first name per the source comment) |
+
+**This is hard-coded inferred-attribute classification.** Per CLAUDE.md, the rationale is "Confidence-default name resolution... so a forklift operator looks like a forklift operator" — i.e., assigning representative photos. But the code path is:
+
+`candidate name → genderFor() / guessEthnicityFromFirstName() → face-pool bucket selection`
+
+A discrimination plaintiff's lawyer would frame this as: "the system EXPLICITLY classifies candidates by inferred ethnicity and uses that classification in its workflow." Whether the downstream use is "just for the photo display" or "ranks candidates" doesn't matter for BIPA-style strict-liability framing — it matters for Title VII, where intent / disparate-impact tests apply.
+
+### 1C — Data directories holding face content
+
+| Path | Count today | Purpose | gitignored? |
+|---|---:|---|---|
+| `data/headshots/` (face_*.jpg + manifest.jsonl) | 2500 entries | Synthetic face pool + biometric classifications | partial — manifest.jsonl is tracked, face_*.jpg are gitignored |
+| `data/headshots_role_pool/` | 7 entries | Role-specific representative faces | gitignored |
+| `data/headshots_gen/` | 4 entries | ComfyUI on-demand portrait cache | gitignored |
+| `data/icons_pool/` | 2 entries | Certification + role icons | gitignored |
+| `data/face_test/` | 1 entry | Likely deepface dev/test scratch | gitignored |
+| `data/_imagecache/` | (not enumerated) | Image cache | gitignored |
+
+The `manifest.jsonl` file is **tracked in git** today and contains the demographic-classification table. For synthetic faces this is fine; **for real candidate faces, this would be a biometric database in version control** — a serious BIPA + general-data-protection failure.
+
+### 1D — Biometric-info contract risk when real photos arrive
+
+When the system transitions from synthetic to real candidate photos (J: "we will"), the BIPA exposure becomes immediate:
+
+1. **No written consent flow exists.** BIPA §15(b) requires "informed written consent" before biometric collection. The headshot upload path (wherever it gets built) needs a consent gate.
+
+2. **No public retention schedule exists.** BIPA §15(a) requires a written, publicly-available retention schedule. No such document found.
+
+3. **No deletion procedure exists.** BIPA §15(a) requires destruction "when the initial purpose for collecting or obtaining such identifiers or information has been satisfied or within 3 years of the individual's last interaction with the private entity, whichever occurs first." No such procedure exists.
+
+4. **No employee training documented.** BIPA litigation routinely cites lack of employee training on biometric handling as evidence of negligence.
+
+5. **The deepface library is itself a biometric processor.** Using it triggers the BIPA disclosure obligation regardless of whether the system stores the underlying face image — extracting "race" and "gender" from a face IS biometric processing under modern interpretations.
+
+6. **Per-violation damages.** Negligent violations: $1,000 per violation per person. Intentional/reckless: $5,000 per violation per person. The IL Supreme Court has held each separate scan/extraction can constitute a separate violation (Cothron v. White Castle, 2023). Class actions are routine; recent settlements:
+   - Facebook (Patel v. Facebook, 2020): $650M
+   - TikTok (In re TikTok BIPA, 2022): $92M
+   - Six Flags (Rosenbach v. Six Flags, 2019): set the "no actual harm needed" precedent
+   - Smaller staffing companies have settled for $1M-$50M ranges.
+
+### 1E — What this means for the architecture
+
+Even with synthetic data, the architecture has BIPA-shaped surface that is one upload path away from being live. **Recommended pre-real-photo gates:**
+
+1. **Disable deepface tagging on real-photo path** until consent + retention schedule are in place. Synthetic faces can keep using deepface; real faces cannot.
+2. **Quarantine `data/headshots/manifest.jsonl` for real photos.** Real-photo classifications must NOT land here. Either move to a dedicated `data/biometric/` tree with separate retention or — preferable — don't extract demographic features from real photos at all.
+3. **Replace the name → ethnicity lookup tables with a non-classifying default.** The current logic exists to pick "representative" photos. For real photos, the candidate's own photo IS representative — no inference needed. The lookup tables become dead code.
+4. **Upload path needs consent gate.** Whatever endpoint accepts real photos must check a stored consent timestamp + version before storing the file.
+5. **Retention schedule documented + auto-enforced.** Crontab / ingestd background sweep that deletes biometric data per the schedule.
+
+These five gates are NOT in `AUDIT_TRAIL_PRD.md` §8 phase plan. **Recommend adding as Phase 1.6 (BIPA pre-launch gates)** ahead of any real-photo intake.
+
+---
+
+## §2 — outcomes.jsonl content sample
+
+### 2A — File state
+
+| Property | Value |
+|---|---|
+| Rows | 101 |
+| Path | `data/_kb/outcomes.jsonl` |
+| Rows with `name` field in fills | **39** |
+| Rows with `first_name` field | 0 (the field is `name`, not `first_name` — names are pre-concatenated in this sink) |
+| Sample names in fills (synthetic but representative) | `Alex Rivera`, `Carmen I. Garcia`, `Jacob N. Patel`, `Jamal Z. Jones`, `James Park`, `Maria Chen`, `Mei Y. Cooper`, `Peter E. Carter`, `Sam Torres` |
+
+### 2B — Confirmed PII shapes in the file today
+
+- `fills[*].name` — full candidate name (synthetic but real shape)
+- `operation` — natural-language fill request (e.g. "fill: Welder x2 in Toledo, OH"). Currently doesn't carry candidate names in the operation string itself, BUT the `error` field can — model reasoning that names specific candidates lands here
+- `error` — model self-correction text and validator failure messages. Sampled rows contain phrases like "executor parse failure on turn 8: no JSON object in Executor response: ..." which is technical content. Real-data scenarios where the model reasons about specific candidates would put names here.
+
+### 2C — Discrimination-proxy phrase audit (gemini scrum flag §10/single-reviewer)
+
+Searched for common LLM-generated discrimination proxy phrases. **0 hits in current synthetic-data sample**:
+
+- "culture fit" — 0
+- "communication" — 0
+- "team chemistry" — 0
+- "personality" — 0
+- "soft skills" — 0
+- "work ethic" — 0
+- "professional attitude" — 0
+
+**Interpretation:** the current synthetic data doesn't generate these phrases. However:
+- The file is structurally capable of holding them (it persists model `error` text)
+- When real models (kimi-k2.6, deepseek-v3.2 per PR #13) reason about real candidates, they WILL generate phrases like these — these are common in HR/staffing LLM output
+- Once a phrase like "rejected as not a culture fit" lands in outcomes.jsonl, it's append-only persistence. Cryptographic erasure is the only RTBF path.
+
+**Action:** add a content-redaction pass to the outcomes.jsonl writer BEFORE production data lands. Strip discrimination-proxy phrases at write time. (This is not a substitute for the model not GENERATING them — it's a defense-in-depth layer.)
+
+### 2D — Implications for the audit trail
+
+The outcomes.jsonl sample confirms what `AUDIT_PHASE_1_DISCOVERY.md` §1D predicted: **39 of 101 rows persist candidate names in the `fills[*].name` field today**. The shape is real even if the data is synthetic.
+
+For the Phase 2 identity service design:
+
+1. **The training-safe export pipeline (per J 2026-05-03 answer 11) MUST address outcomes.jsonl directly.** This file is the most-likely candidate for RAG re-indexing or fine-tuning corpus, and it carries names.
+
+2. **Crypto-erasure target list** must include `data/_kb/outcomes.jsonl`, `data/_kb/overseer_corrections.jsonl`, `/tmp/lakehouse-validator/sessions.jsonl`, plus pathway memory state, plus Langfuse storage.
+
+3. **Subject_id top-level promotion (per AUDIT_PHASE_1_DISCOVERY §10/C5)** can replace `fills[*].name` as the queryable field. The name itself can move into a `[REDACTED]` token until legal access dereferences via the identity service.
+
+---
+
+## §3 — Synthesis: net new findings for Phase 2 design
+
+| Finding | Source | Phase 2 design implication |
+|---|---|---|
+| **Phase 1.6 (NEW) — BIPA pre-launch gates needed before real photos accepted** | §1E above | Phase 2 identity service design must include biometric-info table with consent + retention metadata schema. |
+| **Name-based ethnicity inference is a separate Title VII / disparate-impact risk** | §1B above | Phase 2 must include a "no protected-attribute inference at routing time" architectural rule. The lookup tables should be deprecated as part of the identity-service migration. |
+| **deepface library use itself triggers BIPA when real photos arrive** | §1D point 5 | Phase 2 design must specify "biometric processing requires consent token presence" as a gateway boundary check. |
+| **outcomes.jsonl persists names today (39/101 rows)** | §2A above | Phase 2 needs subject_id top-level promotion + name field as `[REDACTED-{token}]` reference (token resolves via identity service for authorized callers). |
+| **No discrimination-proxy phrases in current synthetic data, but the sink is shape-ready for them** | §2C above | Phase 2 should specify a redaction-on-write pass for outcomes.jsonl + sessions.jsonl writers. |
+| **manifest.jsonl is tracked in git today** | §1C above | If real photos arrive without changing the manifest path, biometric data lands in version control — must be addressed as part of Phase 1.6 gates. |
+
+---
+
+## §4 — Open questions surfaced by Phase 1.5
+
+1. **Where will real photos enter the system?** No upload endpoint walked. Need to know the entry point before designing the consent gate.
+2. **Will the deepface tagging path be preserved for real photos, or stripped?** Architectural decision — affects whether biometric extraction is in scope at all.
+3. **What's the staffing client's consent UX?** Does the candidate sign consent during onboarding, separately for biometrics, etc.? Affects the consent-token design.
+4. **How long do BIPA records need to persist?** BIPA's "3 years from last interaction" is a maximum, not a target — staffing-co's contract may set a shorter floor (e.g., "delete on placement+1 year"). Need staffing-co policy.
+5. **Is there a Designated Privacy Officer / DPO?** Per CCPA/GDPR best practice + BIPA litigation hygiene, this role should exist before real-PII intake. Out of scope for code; flag for J.
+
+---
+
+## §5 — Recommended Phase 1.6 BIPA pre-launch gates (NEW)
+
+Lands as new phase BETWEEN Phase 1.5 (this doc) and Phase 2 (identity service design). Read-only/design work; one code commit (the consent gate). Estimated 4-8 hours.
+
+1. **Write `docs/BIPA_COMPLIANCE_POLICY.md`** — public retention schedule, written consent template, deletion procedure, employee training acknowledgment. Counsel must review before publication.
+2. **Add consent gate to whatever endpoint accepts real photos** (not yet built — design with the gate from day 1, don't bolt on after).
+3. **Quarantine real-photo classifications to `data/biometric/`** — separate path, separate retention, separate access control. Real-photo `manifest.jsonl` does NOT land in `data/headshots/`.
+4. **Deprecate name → ethnicity lookup tables in `mcp-server/search.html`** (commit removing those constants + the call sites). Replace with "use candidate's actual photo" path or non-inferred default.
+5. **Add a unit test that asserts `data/headshots/manifest.jsonl` row count matches the SYNTHETIC face count** — fails if real-photo classifications leak into the synthetic manifest.
+
+These five fit between Phase 1.5 and Phase 2 because (a) they're MORE URGENT than Phase 2 (BIPA exposure starts the moment a real photo touches deepface), (b) they're independent of the identity service design, (c) they need counsel sign-off before they can ship anyway.
+
+---
+
+## Change log
+
+- 2026-05-03 — Phase 1.5 BIPA + outcomes content audit complete. New finding: synthetic data + deepface + name-ethnicity inference creates BIPA-shaped surface that is one upload path away from being live. Recommend Phase 1.6 (BIPA pre-launch gates) before Phase 2 design starts.

docs/AUDIT_PHASE_1_DISCOVERY.md

@@ -0,0 +1,295 @@
+# Phase 1 Discovery — Subject + PII Surface Map
+
+**Status:** Draft — 2026-05-03 · **Drafted by:** working session 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md)
+
+> **Purpose.** Read-only walk of both runtimes. Fills in the "UNKNOWN" cells in `AUDIT_TRAIL_PRD.md` §3 surface map and §7 current-state-vs-target table with file:line evidence. **No code changes.** Output: a complete picture of where subject identifiers + PII flow today, where they leak, and what an audit response could and could not produce right now.
+
+---
+
+## TL;DR — what the walk found
+
+1. **A defense layer EXISTS but is BYPASSED.** Two PII-masked views (`candidates_safe`, `workers_safe`) live in `data/_catalog/views/` and correctly drop name/email/phone. **No production tool query uses them.** The MCP tool registry's `search_candidates` and `get_candidate` SQL templates query the raw `candidates` / `workers_500k` tables and return full PII to the LLM context. Worth considering: the views are functioning policy artifacts that nobody routes through.
+
+2. **PII flows freely through the LLM substrate.** In a single fill scenario, candidate names + emails + phones traverse: SQL → tool_result → LogEntry → execution_loop log → `/v1/respond` HTTP response → Langfuse trace input + output → `data/_kb/outcomes.jsonl` (fills with name) → `data/_kb/overseer_corrections.jsonl` (operation + correction text). At least 7 distinct persistence/transmission paths per scenario.
+
+3. **`candidate_id` is a stable token but not an isolated one.** It's a column on the same `workers_500k.parquet` that holds name/email/phone — there is no separate identity service. Joining candidate_id back to PII is one DataFusion query away. The audit-trail PRD's §5 identity-service intent is not yet built; the architectural separation does not exist.
+
+4. **No `/audit/subject/{id}` endpoint exists.** Reconstructing "every decision about person X" today requires manual cross-correlation across 4+ JSONL files + Langfuse + pathway memory + observer events. There is no canonical query path; the audit response cannot be produced in any reasonable time today.
+
+5. **Go side mirrors the Rust pattern.** Go validator's `rosterRow` carries Name; Go SessionRecord carries `Prompt` (truncated to 4000 chars) which contains the natural-language operation including candidate names. Cross-runtime parity in the PII-leak too.
+
+6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today.
+
+7. **The matrix-indexer fingerprint is subject-agnostic; trace bodies are UNVERIFIED.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers, which is structurally defensive. **However:** trace bodies (`reducer_summary`, `final_verdict`) are written from execution-loop output and are highly likely to leak PII. Per §8.1 these are unverified. Treat the matrix indexer as a SUSPECTED PII SINK until sampled — do NOT rely on "matrix can't drive discrimination" framing. (Walked back from earlier draft per cross-lineage scrum §10/C1: that claim was over-stated in 3/3 reviews.)
+
+---
+
+## §1 — PII flow paths in the Rust stack (file:line evidence)
+
+### 1A — MCP tools query raw tables (PII enters the LLM context)
+
+| File:line | What | PII columns returned |
+|---|---|---|
+| `crates/gateway/src/tools/registry.rs:96` | `search_candidates` SQL template | `first_name, last_name, phone, email, city, state` |
+| `crates/gateway/src/tools/registry.rs:107` | `get_candidate` SQL template | `SELECT *` — returns ALL columns including hourly_rate_usd, address, anything in the table |
+| `crates/gateway/src/tools/registry.rs:129` | `top_recruiters` SQL | recruiter names (employee PII, not candidate but still PII) |
+| `crates/gateway/src/tools/registry.rs:141` | `engaged_unplaced_candidates` SQL | `c.candidate_id, c.first_name, c.last_name, c.phone, c.vertical` |
+| `mcp-server/index.ts:1722, 2173, 2269` | address concatenation (`street_number + street_direction + street_name`) | full street addresses surfaced to MCP UI clients |
+
+**Architectural gap:** the MCP tool registry has no "view-only" mode. Adding `_safe` view routing would be one `sql_template` rewrite per tool. **Today: every fill scenario invokes search_candidates → LLM sees full PII for every candidate in the result set.**
+
+### 1B — LLM tool results captured in execution log
+
+| File:line | What |
+|---|---|
+| `crates/gateway/src/execution_loop/mod.rs:30-39` | `LogEntry` struct: `{turn, role, model, kind, content: Value, at}`. `content` is the raw tool_result body (carries PII when kind=`tool_result`). |
+| `crates/gateway/src/execution_loop/mod.rs:217` | `self.append(LogEntry::new(..., "tool_result", trimmed))` — every tool result appended to `self.log` |
+| `crates/gateway/src/execution_loop/mod.rs:1319` | per-render cap: `kind == "tool_result"` gets 1200 char cap (vs 200 for other kinds). Trimming is for token economy, **not PII redaction** — names/emails fit in 1200 chars. |
+| `crates/gateway/src/execution_loop/mod.rs:30-50` | `Fill` struct (in same file): `{candidate_id: String, name: String, reason: Option<String>}`. The `name` field is load-bearing per the PRD-defined FillProposal artifact contract. |
+
+### 1C — Execution log returned to HTTP caller
+
+| File:line | What |
+|---|---|
+| `crates/gateway/src/v1/respond.rs:111-119` | `RespondResponse { status, artifact, log: outcome.into_log(), iterations, error }` — full log array returned in JSON response body to whoever called `POST /v1/respond` |
+| `crates/gateway/src/execution_loop/mod.rs:82-93` | `RespondOutcome::{Ok,Failed,Blocked}` all carry `log: Vec<LogEntry>` — no path drops the log on its way to the caller |
+
+**Audit implication:** any HTTP client calling `/v1/respond` today receives full PII in the response. Authentication on `/v1/respond` is the only access control. Authorization is binary (either you can call it or you can't); there is no row-level filtering of which candidates you're allowed to see in the response.
+
+### 1D — Persisted JSONL sinks on disk
+
+| File:line | Path written | PII shape |
+|---|---|---|
+| `crates/gateway/src/execution_loop/mod.rs:508` | `data/_kb/outcomes.jsonl` (append) | `{operation, fills: [{candidate_id, name, reason?}], error, ...}` — observed live: `operation` carries natural-language fill request including role + city + state; `fills` carries the full Fill struct with name. |
+| `crates/gateway/src/execution_loop/mod.rs:701` | `data/_kb/overseer_corrections.jsonl` (append) | `{operation, correction, sig_hash, ...}` — `correction` is the overseer's free-text guidance which often references specific candidates by name (e.g. "the executor picked Emily Garcia who is at fill capacity, try Maria Rodriguez instead") |
+| `mcp-server/observer.ts:138-139` | `data/_observer/ops.jsonl` (append) | observer event log; PII content depends on what the operation context included. **Empty on this box right now** (file doesn't exist), but the writer is wired. |
+| `mcp-server/observer.ts:161` | `data/_kb/observer_escalations.jsonl` (append) | sampled live: technical analysis fields (no PII observed in current rows), but `analysis` field is free-form LLM output and could include names depending on the escalation trigger. |
+| `crates/gateway/src/v1/session_log.rs` (writer) → `/tmp/lakehouse-validator/sessions.jsonl` | per `lakehouse.toml` `[gateway].session_log_path` | `SessionRecord { ..., prompt, attempts: [{... raw}], artifact }` — **`prompt` carries the operation text (PII), `raw` carries each model attempt's raw text (PII when model reasoned about specific candidates), `artifact` carries the final FillProposal `{candidate_id, name}` shape on success.** |
+
+### 1E — Langfuse external persistence
+
+| File:line | What gets sent |
+|---|---|
+| `crates/gateway/src/v1/langfuse_trace.rs:166` | `body.input.messages = ev.input` — full message array (system + user + assistant + tool messages). Tool messages contain raw PII for tool_result entries. |
+| `crates/gateway/src/v1/langfuse_trace.rs:191` | `body.output = ev.output` — full model generation. When the model emits a Fill or reasons about candidates, names appear here. |
+
+Langfuse runs at `:3001` per memory, with credentials in `/etc/lakehouse/langfuse.env`. The Langfuse storage tier holds these traces. **PII leaves the lakehouse infrastructure boundary at this point** — Langfuse's storage layer (Postgres + ClickHouse, default config) holds it.
+
+### 1F — Pathway memory
+
+| File:line | What |
+|---|---|
+| `crates/vectord/src/pathway_memory.rs:169` | `PathwayTrace` struct definition |
+| `data/_pathway_memory/state.json` | Persisted state — **need read** to confirm whether real names appear in traces. |
+
+Pathway traces are keyed by `pathway_id = SHA256(task_class + file_prefix + signal_class)` — none of which are subject identifiers. **The fingerprint is structurally subject-agnostic.** Whether the trace BODY (kb_chunks, observer_signals, reducer_summary, final_verdict) leaks PII depends on what those fields contain at write time. From the struct shape: `reducer_summary` and `final_verdict` are strings written from execution loop output — **highly likely to leak PII** when summarizing fill outcomes. Need to confirm against live state.json.
+
+---
+
+## §2 — Defense layer that exists but is bypassed
+
+`data/_catalog/views/candidates_safe.json`:
+- Drops: `last_name, email, phone, hourly_rate_usd`
+- Masks: `candidate_id` (keep first 3, last 2 chars — `CAND-...01` shape)
+- Row filter: `status != 'blocked'`
+- Visibility intent (per `description`): "Visible to recruiter / mode-runner agents."
+
+`data/_catalog/views/workers_safe.json`:
+- Drops: `name, email, phone, zip, communications, resume_text`
+- Reason given: "resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale"
+- Source for the rebuilt `workers_500k_v9` vector corpus
+
+**The `_safe` views are the right policy artifact.** The ARCHITECTURAL gap is at the SQL-template layer in `crates/gateway/src/tools/registry.rs` — the LLM-facing tools query the RAW tables, not the SAFE views. Three out of three candidate-touching tool templates (search_candidates, get_candidate, engaged_unplaced_candidates) bypass the safe view.
+
+There is no enforcement preventing this. There is no test that asserts "tool SQL must reference only `_safe` tables." There is no warning logged when raw tables are queried via tool surface.
+
+**Quick fix shape (NOT a phase 1 deliverable, just noting the change shape):** rewrite the tool sql_templates to `FROM candidates_safe` (or `FROM workers_safe`); add a build-time check that `crates/gateway/src/tools/registry.rs` only references `*_safe` tables; add a runtime gate in queryd that refuses LLM-attributed queries on raw tables. These would land in Phase 4 (subject tagging across substrates) or possibly Phase 1.5 (defense-layer enforcement) per the AUDIT_TRAIL_PRD.
+
+---
+
+## §3 — Identity / candidate_id provenance
+
+| Question | Finding |
+|---|---|
+| Is `candidate_id` a stable token? | Yes — observed format `CAND-NNNNNN` (e.g. `CAND-000001`). Stable across the table. Used as join key in the schema (per `crates/ui/src/main.rs:160` cross-table join hints). |
+| Is the candidate_id ↔ PII mapping in a separate service? | **No.** Both live in `data/datasets/workers_500k.parquet`. A SQL `SELECT name, email FROM candidates WHERE candidate_id = '...'` resolves the mapping in one query. |
+| Is the mapping itself audited? | **No.** No log records "who looked up the PII for which candidate when." This is the identity-service gap from PRD §5. |
+| Does anything ever generate a *different* token (UUID, opaque hash) instead of using candidate_id directly? | **No** — every tool, every validator, every persistence sink uses candidate_id as-is. |
+
+**Production-ready implication:** the staffing client's lawyer asks "show me the access log for candidate X's PII" → we cannot produce one. Every access happens via SQL against `workers_500k`; no row-level access log exists.
+
+---
+
+## §4 — Go-side parity with the leak surface
+
+| Rust file:line | Equivalent Go file:line | Same shape? |
+|---|---|---|
+| `crates/validator/src/lib.rs` (FillProposal carries `{candidate_id, name}`) | `internal/validator/fill.go` + `internal/validator/lookup_jsonl.go:23-31` (`rosterRow` carries `CandidateID, Name, Status, City, State, Role, BlacklistedClients`) | Yes — same PII shape |
+| `crates/gateway/src/v1/session_log.rs` SessionRecord with prompt/artifact | `internal/validator/session_log.go` SessionRecord with `Prompt string`, `Attempts[].Raw string`, `Artifact map[string]any` | Yes — same shape |
+| `outcomes.jsonl` / `overseer_corrections.jsonl` writers | Not present on Go side (Go validatord doesn't currently run the execution_loop with tool dispatch — Phase 4 of Go rewrite is what wires that) | **Asymmetric** — Go side writes less but only because feature parity isn't done |
+| MCP tools registry | Not on Go side yet (mcp-server is still Bun) | **Bun is the surface** for tool dispatch in both runtimes today |
+
+**Cross-runtime audit implication:** even though our 5 parity probes (validator/extract_json/session_log/materializer/embed) all pass 32/32, **none of them assert PII handling**. A new `pii_parity.sh` probe would feed identical PII-tagged input through both runtimes and assert identical redaction behavior. As of today, neither runtime *does* redaction at the substrate level, so the probe would just confirm "both leak identically."
+
+---
+
+## §5 — Mapping back to AUDIT_TRAIL_PRD §3 surface map
+
+Updated cells with file:line evidence:
+
+| Decision happens at | Currently logged where | Audit-completeness gap (revised) |
+|---|---|---|
+| Ingestion (candidate added to pool) | `data/datasets/workers_*.parquet` rows; ingestd writes via `crates/ingestd/`; no per-subject "added at" event journal found | **GAP** — no subject-tagged ingest event. When + who + how a candidate entered is not auditable per-subject. |
+| Embedding creation | `crates/aibridge/src/client.rs` (post-sidecar-drop direct path); LRU cache `crates/aibridge/src/cache.rs` (commit `150cc3b`); cached entries are keyed by `(model, text)` — **the text is the cached body, which for candidate embeddings IS PII (candidate name + role + skills appear in source text per workers_500k_v9 build SQL).** Cache itself is in-memory, not persisted, but text-as-key means PII is in process memory. | **MAJOR GAP** — no per-embedding audit row. Cache key contains PII. No subject tagging means we cannot answer "what embedding was generated for candidate X." |
+| Search inclusion | Tool result entries in execution log + Langfuse + outcomes.jsonl `fills` field | Partial — fills are logged, but you have to grep across outcomes.jsonl + Langfuse for any mention of a candidate_id. No "all search results that included X" canonical view. |
+| Search rank | Result set in chat traces (Langfuse), not indexed by candidate | Partial — Langfuse trace has the result set; no inversion (candidate → ranks-received). |
+| Fill recommendation | `outcomes.jsonl` `fills` array with name + candidate_id | Partial — present, but mixed with PII in the natural-language `operation` field. |
+| Validation outcome | `sessions.jsonl` per `[gateway].session_log_path` config; per-attempt `verdict_kind`, `error` | Partial — works per-session, not per-subject. To find "all validations that touched candidate X" you'd need to grep the JSONL by candidate_id. |
+| Iterate retry escalations | Same sessions.jsonl `attempts[]` array | Same as above |
+| Observer signals | `data/_observer/ops.jsonl` (file not present today on this box, writer is wired) | UNKNOWN content shape until the writer fires for a fill scenario — needs verification |
+| Matrix-indexer compounding | `data/_pathway_memory/state.json` — fingerprinted by code, NOT by subject | **NO subject leak in fingerprint structure** (good for audit defensibility). Trace bodies (`reducer_summary`, `final_verdict`) MAY carry PII — needs sampling to confirm. |
+
+---
+
+## §6 — Mapping back to AUDIT_TRAIL_PRD §7 current-state-vs-target gap table
+
+| Capability | Phase 1 finding | Status |
+|---|---|---|
+| candidate_id as canonical token | Stable `CAND-NNNNNN` format. Same column lives in same parquet as PII. | Token exists; isolation does not. |
+| Identity service | Doesn't exist. PII + candidate_id co-located in `workers_500k.parquet`. | Real gap — needs new service. |
+| `/audit/subject/{id}` endpoint | Doesn't exist. No audit route in `crates/gateway/src/v1/mod.rs` route listing. | Real gap — needs new endpoint. |
+| Subject-tagged embeddings | LRU cache `(model, text)` keys; text contains PII; no audit row per embed. | Real gap. |
+| Subject-tagged search results | Langfuse trace contains result set; not subject-indexed. | Real gap (queryability, not capture). |
+| Subject-tagged validation outcomes | Yes — sessions.jsonl carries candidate_id (in the attempt's `raw` field) but not as a queryable top-level field. | Partial — needs subject_id top-level promotion. |
+| Subject-tagged matrix indexer entries | Pathway fingerprints are subject-agnostic by design. Trace bodies may leak. | Decision needed (PRD open question 7) — keep code-only OR risk PII surface. |
+| Protected-attribute filter at decision time | **Not enforced anywhere.** SQL templates return whatever columns they SELECT; no protected-attribute removal at the gateway boundary. The `candidates` table schema in the demo SQL includes age (years_experience is a proxy), and the call_log/email_log tables likely contain free-text correspondence. | Real gap — major. Requires both schema audit + boundary enforcement. |
+| Retention policy | None enforced. Append-only files grow indefinitely. | Real gap. |
+| Right to be forgotten | Not implementable today on append-only logs without cryptographic erasure. | Per PRD §6 design, real engineering needed. |
+| Cross-runtime parity | 5 algorithm probes pass; 0 audit/PII probes exist. | Probe set needs extension. |
+
+---
+
+## §7 — Worked example: John Martinez audit trail TODAY (negative result)
+
+If John Martinez (candidate_id `CAND-042195`) requests an audit:
+
+1. **Find his candidate_id.** Manual SQL: `SELECT candidate_id FROM candidates WHERE first_name='John' AND last_name='Martinez'` — returns 1+ rows. Already a leak: someone with SQL access can correlate name → candidate_id ad-hoc.
+
+2. **Find every fill scenario that included him.** `grep "CAND-042195" data/_kb/outcomes.jsonl` — returns rows where his fill was included. But the row's natural-language `operation` field may NOT contain his candidate_id (it's the fill request, e.g. "fill: Welder x2 in Toledo, OH"), so we'd miss scenarios where he was a *candidate* but not a *fill*. To find those, we'd need to grep Langfuse traces (off-box) or the per-tool result content in execution logs (which aren't persisted as separate JSONL — they live in the HTTP response that already left the building).
+
+3. **Find every validation that touched him.** Grep `/tmp/lakehouse-validator/sessions.jsonl` for `CAND-042195` — would catch FillValidator phantom-ID rejections AND successful fills, but the candidate_id would appear in either the prompt or the raw attempt text, not as a top-level queryable field.
+
+4. **Find every embedding generated for him.** Cache is in-memory; nothing persisted. Cannot answer.
+
+5. **Find every search result that ranked him.** Off-box in Langfuse. Untrieable in lakehouse without a separate Langfuse query pipeline (which doesn't exist).
+
+6. **Find pathway memory traces involving him.** Pathway fingerprints don't carry his ID. The `reducer_summary` strings might mention him (need to grep state.json) but the fingerprint search wouldn't surface them.
+
+7. **Show what protected attributes were exposed to the model.** No record of input_features per decision — the LLM saw whatever the SQL returned, and we have no per-decision input_features audit row.
+
+8. **Format the output for legal.** Even if we collected all the above, there's no signing, no integrity hash, no schema, no template.
+
+**Estimated time to produce a complete-and-defensible response today: not possible.** Estimated time to produce an INCOMPLETE response by cobbling JSONLs + Langfuse exports: 2-5 hours per request, manual, error-prone, and the response would over-share (other candidates appearing in the same fill scenarios) AND under-share (embedding events, search rankings, pathway traces missed).
+
+This is the Phase 1 result the AUDIT_TRAIL_PRD predicted: today's substrate is not production-ready for a discrimination-defense audit response.
+
+---
+
+## §8 — What this discovery DID NOT cover
+
+Phase 1 was scoped to file:line evidence + sampling of live JSONL state. The following deserve their own subsequent walks before phase 2+ design:
+
+1. **Live sample of `data/_pathway_memory/state.json`** — confirm whether `reducer_summary` / `final_verdict` strings actually leak PII or stay generic. Read 3-5 traces and grep for names from `workers_500k`.
+2. **Live sample of Langfuse traces** — confirm input message PII for a real fill scenario from past 7 days. Use Langfuse `:3001` query API.
+3. **observerd event content** — the writer is wired but `data/_observer/ops.jsonl` doesn't exist on this box. Trigger a fill scenario and inspect the resulting event.
+4. **Bun mcp-server tool dispatch** — does the Bun server log tool calls anywhere? `mcp-server/index.ts` is 2900+ lines; partial walk only here.
+5. **bot/propose.ts** — the bot proposal flow likely touches candidates; not walked in this pass.
+6. **`crates/journald` mutation log** — designed for row-level mutations per ADR-012; haven't confirmed whether candidate-table mutations land here with PII.
+7. **Go side observerd + chatd PII surface** — the Go cmd/* binaries likely have analogous logging; confirmed validator parity but didn't walk observer/chat logging.
+8. **Process memory + crash dumps** — if the gateway dumps core, what PII is in it? Out of scope for code walk; comes up in security audit.
+9. **Operator runbook** — who has access to logs/, /tmp/lakehouse-validator/, MinIO buckets, Langfuse Postgres? Out of scope for code walk; comes up in operational security review.
+
+These are listed so the next phase doesn't accidentally re-walk what's done OR skip what wasn't covered.
+
+---
+
+## §9 — Recommended next moves (not commitments)
+
+Per AUDIT_TRAIL_PRD §8, Phase 2 is the identity service design doc. Before that doc gets written, the cheapest high-value moves discovered here:
+
+1. **Defense-layer enforcement (1-2 hours work).** Rewrite the 3 tool SQL templates in `crates/gateway/src/tools/registry.rs` to use `_safe` views. Add a unit test that asserts no `crates/gateway/src/tools/registry.rs` template references `FROM candidates ` or `FROM workers_500k ` (only `*_safe`). This is one commit. It prevents the most-trafficked PII leak path TODAY without waiting on the identity service. **Cost:** the LLM sees masked candidate_ids (`CAN...95` instead of `CAND-042195`); some downstream tools (validator existence check) would need a "resolve to full ID" path that goes through the identity service — but that's exactly the architectural shape PRD §5 wants anyway.
+
+2. **Sample state.json + Langfuse before phase 2 design.** §8.1 + §8.2 above. ~30 minutes. Either confirms or refutes the "matrix indexer is subject-clean" finding from §1F.
+
+3. **Document the Bun mcp-server tool surface.** §8.4. The Bun layer is a major PII transit point not fully covered here.
+
+4. **Identify whether protected attributes (age proxies, photo features, zip-code → race correlations) are currently in any tool-returned column.** Schema-level audit of `candidates` + `workers_500k`. ~1 hour. Might surface that some "neutral" columns are actually protected-attribute proxies.
+
+These four moves give the phase-2 design doc strong evidence to lean on. None are commitments — J's call on what to do next.
+
+---
+
+## §10 — Cross-lineage scrum review of this discovery doc (2026-05-03)
+
+After the discovery walk, this document was reviewed by three independent model lineages via `/v1/chat` against the Go gateway (post-PR-#13 model fleet): **opus** (`opencode/claude-opus-4-7`), **kimi** (`ollama_cloud/kimi-k2.6`), **gemini** (`ollama_cloud/gemini-3-flash-preview`). Custom prompt: senior security architect reviewing a discovery report. (DeepSeek-v3.2 timed out; not included.)
+
+Verbatim reviews saved at `/tmp/audit_scrum/{opus,kimi,gemini}_review.md`. Convergent findings (≥2 reviewers) are treated as high signal per `feedback_cross_lineage_review.md`.
+
+### Convergent findings — must address before phase 2 design
+
+**C1. §1F matrix-indexer claim is OVER-CLAIMED (3/3 reviewers).** The TL;DR #7 line "good for audit defensibility (matrix index can't drive discrimination)" overstates structural subject-agnosticism as behavioral fairness. Per opus: "until §8.1 is executed, the correct framing is 'fingerprint structurally subject-agnostic; body content unverified — treat as suspected PII sink.'" Per kimi: dangerous reasoning — if `reducer_summary` says "candidates named Emily Garcia were rejected for fill capacity," the matrix learns proxy variables and future similar names get downranked = temporal discrimination risk. Per gemini: "you cannot claim audit defensibility until you prove the *content* of the matrix indexer doesn't contain PII." **Action:** walk back the §1F TL;DR claim; reframe as "fingerprint structure is defensive; body content unverified — treat as suspected PII sink until §8.1 confirms."
+
+**C2. §1E (Langfuse) is the MOST DANGEROUS leak — fix FIRST (3/3 reviewers).** Opus: "boundary-crossing leak that makes a regulator's eyes light up." Kimi: "Article 44 GDPR transfer if SaaS-hosted; CPRA 'sale/sharing' question; subprocessor notification failure." Gemini: "un-certifiable for SOC2 Type II" + "unauthorized data transfer to third-party storage tier." All three would do Langfuse redaction/sampling BEFORE the §9.1 view-routing fix. **Action:** revise §9 priority order — Langfuse boundary first, view-routing second.
+
+**C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills (3/3 reviewers).** Opus: "you need not just 'what did we do for candidate X' but 'what was the selection rate for protected class Y vs comparator' — that requires capturing protected attributes WITH outcomes." Kimi: "EEOC Uniform Guidelines on Employee Selection Procedures (1978) — matrix index that learns from historical fill outcomes IS a selection procedure under the Guidelines." Gemini: "The system doesn't log the *entire* candidate pool for a specific search — only the fills (§5). To defend a lawsuit, you must show the stats for everyone who *could* have been picked, not just the person who was." **Action:** Phase 1 didn't capture this load-bearing requirement. PRD §1 worked example needs expanding: the audit response must include the comparator pool + adverse-impact statistics, not just the subject's decision row.
+
+**C4. BIPA + biometric exposure UNDERSTATED (3/3 reviewers).** Already in `AUDIT_TRAIL_PRD.md` §10.5 jurisdictional checklist but NOT translated into Phase 1 findings actionables. If `workers_500k` columns include photo paths, video interview references, or anything that could yield biometric inference (per gemini: "even descriptors that could be reconstructed into biometric templates"), BIPA's $1k-$5k per-violation regime applies BEFORE the GDPR analysis matters. **Action:** add to §8 (what discovery did NOT cover): explicit photo/video/biometric column audit of `workers_500k` schema.
+
+**C5. `candidate_id` must be PROMOTED to top-level field in all JSONL sinks (3/3 reviewers).** Opus + kimi + gemini converge: grepping natural-language strings (operation, raw, prompt, reducer_summary) for candidate_id is not a defensible audit strategy. Even if subject_id appears in those strings TODAY, it appears co-mingled with other candidate names, model reasoning, etc. — making subject filtering unsafe. **Action:** add to PRD §7 target column "subject_id top-level promotion" — change session_log writer + outcomes.jsonl writer + observer event writer to ALL include `subject_id` (or `subject_ids[]`) as a first-class top-level field.
+
+### Single-reviewer findings — verified, worth incorporating
+
+**OPUS unique:**
+- **8th PII path missing: LLM provider egress.** PR #13 routes models through opencode + ollama_cloud + openrouter — opencode and openrouter are external services. Cross-border data transfer under GDPR Ch. V; third-party processor relationship requiring DPA under Art. 28. Phase 1 did not enumerate this path. **Action:** add §1G "LLM provider egress" to §1.
+- **GDPR Art. 22 / EU AI Act right to explanation.** Audit must capture the model's REASONING CHAIN, not just decision output. Phase 1 §7 worked example doesn't include this. **Action:** add to subject-audit response shape (PRD §2).
+- **Special-category data under GDPR Art. 9.** resume_text + call_log + email_log routinely contain health (accommodation requests), union, religion. Higher legal bar — Art. 9(2) explicit consent required. Phase 1 §6 mentions these tables exist but doesn't flag the special-category exposure. **Action:** call out in §6.
+- **DPIA / ROPA / DPA inventory.** None of these documents referenced. Some may exist outside code; Phase 1 should at minimum note their absence as Phase 1.5 input.
+
+**KIMI unique:**
+- **Sequential `CAND-NNNNNN` IDs enable enumeration attacks.** Predictable IDs let an attacker scan the candidate space. Security finding Phase 1 didn't flag.
+- **Langfuse retention config unaudited.** Default trace retention is 30 days in some versions, indefinite in others. Directly impacts RTBF analysis. **Action:** check live Langfuse config.
+- **CCPA "de-identified in place" may be faster than crypto-shred.** Replacing PII with `REDACTED-{hash}` while preserving log structure may satisfy CPRA's de-identified exception. Worth considering vs. crypto-shred.
+- **Bun MCP server is the cross-runtime bridge — likely COMMON-MODE failure.** Phase 1 framed Go side as "mirrors Rust pattern" implying independent failure; it's likely shared infrastructure failure. **Action:** add to §4 — common-mode reframe.
+- **Cryptographic integrity for the audit trail itself.** Merkle trees / signed logs / chain-of-custody under FRE 901. Opposing expert can challenge admissibility without this. **Action:** add to PRD §2 audit response shape — integrity-signed.
+- **HIPAA Business Associate Agreement scoping not done.** If any candidate is healthcare-vertical, BAA analysis required.
+- **`get_candidate` SELECT * has 40+ load-bearing columns.** §9.1 "1-2 hours" estimate for view rewrite is irresponsible without scoping downstream consumers. **Action:** revise §9.1 estimate; flag dependency analysis as prerequisite.
+
+**GEMINI unique:**
+- **Data residency — JSONL on US box.** If any candidate is EU resident, GDPR violation without SCC/DPF. Phase 1 didn't ask whether IL+IN target market includes EU residents (probably not, but staffing-co clients sometimes have international placements).
+- **"Culture fit" reasoning strings as discrimination proxies.** Common LLM-generated phrases ("not a culture fit," "communication concerns," "team chemistry") often correlate with protected-attribute discrimination. Phase 1 didn't audit the actual reasoning text in `outcomes.jsonl` for these phrases.
+- **Comparator-pool snapshot for every fill.** Need to capture WHO COULD HAVE BEEN PICKED, not just who was. **Action:** PRD §2 audit response shape needs `comparator_pool` field per decision.
+
+### Revised §9 — recommended next moves (reordered by scrum convergence)
+
+1. **(NEW PRIORITY 1, was P3) Sample state.json + Langfuse content** — confirm/refute the matrix-indexer subject-clean claim and quantify Langfuse PII exposure. Cheapest move that resolves the over-claim AND informs the boundary-leak fix.
+2. **(NEW PRIORITY 2, was P1) Defense-layer enforcement at SQL template level** — rewrite tool registry to use `_safe` views. **Estimate revised UPWARD per kimi**: scope `get_candidate` SELECT * downstream consumers first; estimate 4-8 hours including the existence-check resolution path through the (not-yet-built) identity service. Stop-gap until then: add LLM-attribution flag to queryd, refuse `FROM candidates `/`FROM workers_500k ` queries that originate from tool dispatch.
+3. **(NEW) Langfuse boundary audit + redaction** — sample retention config, check DPA status, scope a redaction/sampling layer that strips PII from message arrays before the Langfuse POST. This is the boundary-crossing leak — fix BEFORE chasing internal sinks.
+4. **(NEW) Subject_id top-level promotion to all JSONL writers** — single architectural change spanning Rust + Go session_log + observerd event writer + outcomes/corrections appenders. Makes subject-correlation queries defensible (no more grepping natural language strings).
+5. **(was P4) Schema-audit for protected-attribute proxies** — extend to include "culture fit"-shaped reasoning text in outcomes.jsonl + the comparator-pool retention requirement.
+6. **(NEW) BIPA-specific audit of workers_500k schema** — explicit photo/video/biometric column inventory before any production deployment in IL.
+7. **(NEW) Operational discovery** — DPIA, ROPA, DPA inventory, SCC for cross-border, Langfuse retention config. Out-of-code-walk; needs J + counsel input.
+
+### What I'm walking back
+
+§1F TL;DR claim "matrix indexer is good for audit defensibility" — per all 3 reviewers, this is over-claimed without §8.1 verification. The correct frame is "fingerprint structure is subject-agnostic by design; trace body content unverified — treat as suspected PII sink until sampled."
+
+§9 ordering — view-routing was P1; per all 3 reviewers, Langfuse boundary should be P3 in front of it. View-routing is the source-side fix; Langfuse is the boundary-crossing fix; both matter, do them in BOUNDARY-FIRST order.
+
+§9.1 effort estimate — kimi's "irresponsible without dependency scoping" critique is right. Revised UP to 4-8 hours.
+
+---
+
+## Change log
+
+- 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`.
+- 2026-05-03 — §10 added: cross-lineage scrum review of the discovery doc (opus + kimi + gemini). 5 convergent findings (matrix-indexer over-claim, Langfuse first-priority, comparator pool gap, BIPA understated, subject_id top-level promotion). Plus single-reviewer high-value catches. §9 reordered.

docs/AUDIT_TRAIL_PRD.md

@@ -0,0 +1,374 @@
+# PRD: Production-Ready Audit Trail
+
+> **⚠ OVER-SCOPED — 9-phase plan needs to shrink for local-only deployment.**
+>
+> 2026-05-03 evening: J reframed the system as local-only per PRD line 70. The 9-phase plan in §8 was sized for SaaS-tier infrastructure with cloud HSM, separate identity daemon, dual-control JWT, etc. For a single-box local deployment, audit trail can be a few hundred LOC of local writers + a signed local file, not a 17-20 day phase plan.
+>
+> **What stays valid:**
+> - The legal use case (worked example: John Martinez at Warehouse B requests audit) — this is the real problem
+> - The §10.5 jurisdictional surface (IL BIPA, IN, federal) — counsel reads this
+> - The §3 surface map: where decisions get made today (file:line evidence — see `AUDIT_PHASE_1_DISCOVERY.md`)
+> - Phase 1.6 BIPA pre-launch gates — those still apply when real photos arrive
+>
+> **What's over-scoped:**
+> - The 9-phase implementation plan (§8) — should compress to 3-4 phases for local-only
+> - The identity service design (`IDENTITY_SERVICE_DESIGN.md`) — see that doc's deprecation header
+>
+> Do NOT execute the §8 phase plan as-written. When J greenlights, draft a v2 plan sized for local single-box.
+
+**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
+
+> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation.
+
+---
+
+## 1. The legal use case (worked example)
+
+**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2.
+
+**What we must produce.** A response that proves either:
+- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR
+- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI."
+
+**What we must NOT produce.**
+- Other subjects' data (response leaks if even one other candidate's name appears)
+- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped)
+- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence
+
+**The defensibility chain.** The audit shows:
+1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into
+2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank
+3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned
+4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections)
+5. **Outcome decisions** — final fills, rejections, hand-offs
+
+For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale.
+
+---
+
+## 2. The subject audit response — output format
+
+```
+GET /audit/subject/{candidate_id}?from=D1&to=D2
+→ JSON or signed PDF (legal preference TBD)
+```
+
+**Header section:**
+- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash
+- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls.
+
+**Per-decision row schema** (shape, not exhaustive):
+```json
+{
+  "ts": "ISO-8601 UTC",
+  "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal",
+  "daemon": "gateway | validatord | observerd | matrixd | ingestd",
+  "model": "kimi-k2.6 | deepseek-v3.2 | ...",
+  "provider": "ollama_cloud | opencode | openrouter",
+  "input_features": { /* what the model SAW — sanitized per §4 */ },
+  "output": { /* what the model decided */ },
+  "rationale": "model's natural-language explanation, or rule-based justification",
+  "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree",
+  "session_id": "iterate session that produced this row"
+}
+```
+
+**Footer section:**
+- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy"
+- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited)
+
+---
+
+## 3. Surface map — where decisions happen
+
+| Decision happens at | Currently logged where | Audit-completeness gap |
+|---|---|---|
+| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk |
+| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding |
+| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation |
+| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial |
+| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified |
+| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today |
+| Iterate retry escalations | Session JSONL `attempts[]` array | OK |
+| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk |
+| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject |
+
+**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly.
+
+---
+
+## 4. PII handling rules
+
+**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces.
+
+**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it.
+
+**Protected-attribute exclusion at decision time:** the model NEVER receives:
+- Race, ethnicity, national origin
+- Sex, gender, marital status, pregnancy
+- Age, date of birth (allowed: years of experience)
+- Religion
+- Disability, genetic information
+- Veteran status (unless legally relevant for the role)
+- Sexual orientation, gender identity
+
+If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input.
+
+**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls.
+
+**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows.
+
+---
+
+## 5. Identity service — candidate_id ↔ PII mapping
+
+**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change.
+
+**Target state:**
+- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping
+- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id`
+- Identity reads require a separate auth credential held by legal-authorized operators
+- Every identity read is itself audited (log who accessed PII for which candidate when)
+- Identity service runs as its own daemon, port-isolated from the gateway
+- Cross-runtime: same identity service backs both Rust and Go
+
+**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary.
+
+---
+
+## 6. Retention policy
+
+**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA.
+
+**Target state (proposed):**
+- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s)
+- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract.
+- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed.
+- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable.
+
+**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off.
+
+---
+
+## 7. Current state vs target state
+
+| Capability | Today | Production-ready target | Gap |
+|---|---|---|---|
+| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service |
+| Identity service | none | separate daemon, audited reads | Real — build new |
+| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new |
+| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument |
+| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk |
+| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial |
+| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major |
+| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk |
+| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build |
+| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build |
+| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set |
+
+---
+
+## 8. Implementation phases (proposed sequence)
+
+Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds.
+
+### Phase 1 — Discovery walk (read-only, ~3-4 hours)
+Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references.
+
+**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells.
+
+### Phase 2 — Identity service design (design doc, ~2 hours)
+Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.**
+
+**Exit:** J approves the design.
+
+### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours)
+Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong.
+
+**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued.
+
+### Phase 4 — Subject tagging across substrates
+Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror.
+
+**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end).
+
+### Phase 5 — Identity service build
+Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly.
+
+**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject.
+
+### Phase 6 — Protected-attribute boundary enforcement
+Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing.
+
+**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping.
+
+### Phase 7 — Retention + right-to-be-forgotten
+Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint.
+
+**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows.
+
+### Phase 8 — Legal-format export + signing
+Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow.
+
+**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies.
+
+### Phase 9 — End-to-end discrimination defense rehearsal
+Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them.
+
+**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness.
+
+---
+
+## 9. Cross-runtime requirement
+
+**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5.
+
+The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version).
+
+---
+
+## 10. Open questions blocking phase 2 (resolved 2026-05-03)
+
+**These were the original Phase 1 open questions. J answered the load-bearing 5 in conversation 2026-05-03; answers folded in below. Remaining items 1, 4, 6 still need J's call before Phase 2 design ships.**
+
+1. **Identity service: separate daemon vs in-process?** _Recommended: separate. **Status: pending J confirmation.**_
+2. ~~**Retention period N years?**~~ — Out of scope for now; will be set at deployment time per client contract. Default to 4-year hot retention until set.
+3. ✅ **Photos/video in scope?** **YES** (J 2026-05-03). BIPA (740 ILCS 14) applies in full. Per-violation $1k-$5k statutory damages, written consent + retention schedule mandatory. **This becomes a Phase 1.5 priority** — see §10.5/§13 for revised phase ordering.
+4. **JSON or signed PDF for legal export?** _Recommend signed JSON with PDF rendering option. **Status: pending.**_
+5. ✅ **RTBF under append-only:** Cryptographic erasure approach approved in principle (J 2026-05-03 implicit via approval of plan).
+6. **Audit endpoint auth model:** _Recommend legal-only credential separate from admin token. **Status: pending.**_
+7. ✅ **Matrix indexer subject-awareness:** Per scrum review (`AUDIT_PHASE_1_DISCOVERY.md` §10/C1), matrix-indexer is suspected PII sink (trace bodies unverified). Action: sample state.json before deciding (a) keep code-only + add PII-redact-on-write to trace bodies, OR (b) remove subject-summary text from trace bodies entirely. Decision deferred until §8.1 sampling completes.
+
+### Newly answered 2026-05-03 (J)
+
+8. ✅ **Langfuse hosting model:** **Self-hosted.** Removes the GDPR Art. 44 cross-border-transfer concern that 3/3 scrum reviewers flagged. Langfuse retention config + Postgres/ClickHouse access controls still need to be audited as part of Phase 1.5 — but the boundary stays inside J's infrastructure, which is materially better than SaaS Langfuse.
+
+9. ✅ **EU candidates in scope:** **Not currently — may need placeholder later.** Design choice: build the identity-service interface to be EU-compatible (DPIA-shaped fields, lawful-basis tracking, SCC-ready transfer mechanism slots) but DO NOT gate Phase 2 on EU compliance. Phase 2 ships IL+IN-shaped; EU additions are a follow-up phase.
+
+10. ✅ **Healthcare vertical / HIPAA:** **Same framework — yes.** Healthcare staffing IS in scope. PHI in `resume_text`, `communications`, and `call_log` is realistic. **Implications:**
+    - Business Associate Agreement (BAA) required with any third-party model provider that processes content from healthcare-vertical staffing requests
+    - opencode + ollama_cloud + openrouter (per PR #13 routing) are external — BAAs needed OR healthcare requests must route to local-only models (Ollama on-box)
+    - PHI redaction at the gateway boundary becomes mandatory before the model call leaves the box, OR the model call must stay on-box for healthcare requests
+    - Vertical detection at the gateway boundary becomes a Phase 2 requirement
+
+11. ✅ **Training / RAG re-runs may use historical outcomes:** **Yes — design as if it WILL.** Implications:
+    - `outcomes.jsonl` and `overseer_corrections.jsonl` cannot remain raw-PII forever — anything that lands in a training corpus or RAG re-index becomes ungeneratable to delete (PII in model weights)
+    - Phase 2 design must include a "training-safe export" pipeline that strips PII from outcomes before feeding to any training/RAG path
+    - Crypto-erasure of historical outcomes becomes load-bearing — if a candidate exercises RTBF and their data already trained a model, we must be able to evidence "the source was destroyed; the model retains it indistinguishably from synthetic patterns"
+
+### Effect on the §8 phase plan
+
+The user-confirmed answers shift priorities. Revised ordering (incorporating scrum-driven priority changes from `AUDIT_PHASE_1_DISCOVERY.md` §10):
+
+- **Phase 1.5 (NEW)** — BIPA-specific photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample. Lands BEFORE Phase 2 design starts.
+- **Phase 2 (identity service design)** — now must include EU-placeholder fields, vertical-detection (healthcare flag), training-safe export interface, BIPA consent + retention metadata
+- **Phase 3** (audit endpoint skeleton) — unchanged
+- **Phase 4** (subject tagging) — must include healthcare-vertical routing decision at gateway boundary
+- **Phase 5** (identity service build) — must include BIPA-compliant biometric metadata table
+- **Phase 6** (protected-attribute boundary) — must include PHI redaction for healthcare-vertical requests
+- **Phase 7** (retention + RTBF) — must include training-safe export evidence chain
+- **Phase 8** (legal export) — unchanged
+- **Phase 9** (rehearsal) — must include both EEOC discrimination scenario AND BIPA biometric scenario AND healthcare PHI breach scenario
+
+---
+
+## 10.5 Jurisdictional surface (IL + IN)
+
+> **⚠ Not legal advice.** This is a research-grade checklist for J to take into a conversation with actual employment + privacy counsel. The system is targeting **Chicago (Illinois)** and **Indiana** placements per 2026-05-03 conversation. Counsel needs to verify what currently applies, what's pending, and whether case law has moved any of these in 2026. **Verify with counsel before claiming compliance with any item below.**
+
+### Federal layer (always applies)
+
+| Statute / framework | Relevance to this system |
+|---|---|
+| Title VII (Civil Rights Act) | Bans discrimination on race, color, religion, sex, national origin in hiring. Discrimination claim defense is the worked example in §1. |
+| ADEA (Age Discrimination in Employment) | Bans age-based discrimination for workers 40+. DOB must be excluded from features per §4. |
+| ADA (Americans with Disabilities Act) | Bans disability discrimination + requires reasonable accommodation. Disability-inferring features (gait, photo features, medical history) need exclusion. |
+| EEOC enforcement | Receives complaints, issues right-to-sue. Audit response per §2 is what defends in EEOC investigation. |
+| OFCCP | Applies if our staffing client serves federal contractors. Adds affirmative-action recordkeeping on top of EEOC. |
+| FCRA (Fair Credit Reporting Act) | Triggers if background checks are performed. Pre-adverse-action notice + dispute process needed. |
+| Section 1981 | Race-based contract discrimination — staffing is contract relationship. |
+
+### Illinois-specific (Chicago jurisdiction)
+
+| Statute | What | What we need |
+|---|---|---|
+| **BIPA** (Biometric Information Privacy Act, 740 ILCS 14) | Bans collection of biometric identifiers (face geometry, fingerprints, voiceprints) without informed written consent + retention schedule. Penalties: $1,000-$5,000 per violation per person. **Class actions are common and aggressive.** | If we use candidate photos for any feature (face match, headshot rendering, photo-derived attributes), BIPA almost certainly applies. The headshot pool we generate (per CLAUDE.md commit `5d93a71` area) needs careful review — synthetic faces are probably OK; real candidate photos are NOT without explicit BIPA-compliant consent. **Counsel must review.** |
+| **Illinois AI Video Interview Act** (820 ILCS 42) | If AI analyzes recorded video interviews, employer must disclose AI use, obtain consent, provide explanation of how AI works, and limit who can review the video. | If we ever ingest video, this applies. Currently we don't, but worth flagging to counsel as a "what if we add this in 12 months" boundary. |
+| **Illinois Human Rights Act** (775 ILCS 5) | Broader than federal Title VII — adds protected classes including arrest record, military status, marital status, order of protection, citizenship status (in some cases), unfavorable military discharge. | Protected attribute exclusion list in §4 needs expanding to cover IL-specific classes. |
+| **Personal Information Protection Act** (PIPA, 815 ILCS 530) | Breach notification — must notify Illinois residents whose unencrypted PII was breached. | If identity service or workers parquet is breached, notification clock starts. Need incident response runbook. |
+| **Illinois Day and Temporary Labor Services Act** (820 ILCS 175) | Specific to staffing/temporary services industry. Includes equal-pay-for-equal-work + record-keeping requirements + worker notification. | Highly relevant — applies directly to staffing-company clients. Audit retention may interact with these recordkeeping requirements. |
+| **Workplace Transparency Act** | Restrictions on non-disclosure agreements re: harassment/discrimination | Tangential but worth noting. |
+| **City of Chicago Human Rights Ordinance** (Title 6 Chicago Municipal Code) | Adds protected classes beyond IHRA (source of income, parental status, military discharge status, credit history). | Chicago-specific protected attributes list. |
+| **Cook County Human Rights Ordinance** | Similar additions county-wide. | Chicago is in Cook County so this stacks. |
+| **Possible: AI hiring transparency** | Several states/cities have proposed/passed laws modeled on NYC Local Law 144 (annual bias audit + candidate notification). I do not know whether IL or Chicago has such a law on the books as of 2026-01 cutoff. | **Counsel must check current state.** If it exists, we need annual bias audit reports (which IS what this PRD is building toward, but the report format may have specific requirements). |
+
+### Indiana-specific
+
+| Statute | What | What we need |
+|---|---|---|
+| **Indiana Data Breach Disclosure** (IC 24-4.9) | Breach notification within "without unreasonable delay" | Same incident response runbook as IL PIPA. |
+| **Indiana Civil Rights Law** (IC 22-9) | State-level employment discrimination | Largely tracks federal Title VII, fewer expansions than IL. |
+| **Indiana Genetic Information Privacy Act** | Bans use of genetic info in employment | Already in §4 protected list. |
+| **General observation** | Indiana is generally less aggressive than Illinois on AI/employment regulation as of cutoff. | The IL bar is higher — if we satisfy IL, IN typically follows. **Counsel must confirm this isn't backwards.** |
+
+### Cross-cutting (security frameworks for SaaS sales)
+
+These aren't laws but are commonly required by enterprise customers (including staffing clients) before sale.
+
+| Framework | What | Relevance |
+|---|---|---|
+| **SOC 2 Type II** | Auditor attestation of operating effectiveness over 6-12 months across Trust Service Criteria (Security, Availability, Processing Integrity, Confidentiality, Privacy). | The Privacy criterion overlaps heavily with this PRD. Privacy + Security are the two load-bearing TSCs. Effort to first Type II report: 6-9 months. Type I (point-in-time) is faster (weeks) but enterprise buyers usually want Type II. |
+| **SOC 3** | Public-facing summary of SOC 2 (no detailed control descriptions). | Nice-to-have for marketing but the staffing client will want SOC 2 Type II report under NDA. |
+| **HIPAA** | Healthcare data protection. | Triggers ONLY if staffing places workers into healthcare roles where they handle PHI. Currently not in scope per CLAUDE.md. **Confirm scoping with J.** |
+| **PCI DSS** | Payment card data | Not currently in scope. |
+| **ISO 27001** | International information security management | Alternative to SOC 2; more common in EU. Probably unnecessary for IL/IN-only deployments. |
+
+### What this means for phase ordering
+
+The 9-phase plan in §8 is technically correct but may need re-ordering once counsel weighs in:
+
+- **BIPA risk on photos** is so high and so aggressive that if we use real candidate photos *anywhere*, that may need to be the FIRST thing we resolve — before the audit-trail work starts. Class-action exposure is enormous.
+- **SOC 2 Type II prep** runs in parallel with this work, not after. If the staffing client says "show us your SOC 2 report" we need to have started the engagement weeks/months before.
+- **Day and Temporary Labor Services Act** may impose recordkeeping that interacts with our retention SLA (§6) — counsel may say "no, retention has to be N years for THIS reason, not your defaulted 4."
+
+### Open questions for counsel (one ask)
+
+1. Does the staffing client have an existing SOC 2 report we leverage, or do we need our own?
+2. Are we using any real candidate photos? If yes, is BIPA consent in place?
+3. Does Illinois have an AI hiring transparency law on the books in 2026? If yes, what does the bias audit report need to look like?
+4. What's the IL Day and Temporary Labor Services Act recordkeeping retention period? Does it interact with our 4-year proposed SLA?
+5. Are background checks performed? If yes, do we need FCRA pre-adverse-action workflow integration?
+6. Any healthcare placements? (HIPAA scoping)
+7. Is the staffing client a federal contractor? (OFCCP scoping)
+
+Counsel's answers shape whether the §8 phase plan ships as-is or needs reordering.
+
+---
+
+## 11. What this PRD is NOT
+
+- Not a contract with the staffing client. That document needs lawyers and signs after this is built.
+- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC/BIPA/etc — passing actual certification is its own project.
+- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made.
+- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires.
+- **Not legal advice.** The §10.5 jurisdictional surface is a research-grade checklist, NOT counsel's analysis. Verify everything with actual employment + privacy counsel licensed in IL + IN before claiming compliance with anything in this document.
+
+---
+
+## 12. Appendix — terms
+
+- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`.
+- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.).
+- **Audit row** — one record in the audit response per decision, with the schema in §2.
+- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes.
+- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list.
+- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation).
+- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth.
+- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it.
+- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs.
+
+---
+
+## Change log
+
+- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker.

docs/DECISIONS.md

@@ -104,3 +104,8 @@
 **Date:** 2026-04-24
 **Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec<SemanticFlag>` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec<TypeHint>` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec<BugFingerprint>` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002.
 **Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon.
+
+## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly
+**Date:** 2026-05-02
+**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call.
+**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running.

docs/IDENTITY_SERVICE_DESIGN.md

@@ -0,0 +1,584 @@
+# Identity Service — Phase 2 Design (v2 — post-scrum revisions)
+
+> **⚠ OVER-SCOPED FOR LOCAL-ONLY DEPLOYMENT — needs simpler rewrite before implementation.**
+>
+> 2026-05-03 evening: J reframed the system as local-only per PRD line 70 ("Everything runs locally — no cloud APIs"). This document was drafted assuming SaaS-tier infrastructure (HashiCorp Vault, AWS KMS, S3 Object Lock, dual-control JWT split-secret ceremony, mTLS CA, separate Postgres database). For J's local-only single-box deployment serving IL+IN staffing, the audit trail can be MUCH smaller: local SQLite or Postgres, local key file, local HMAC chain to an append-only JSONL.
+>
+> The discovery findings in `AUDIT_PHASE_1_DISCOVERY.md` and `AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md` remain valid (PII flow paths, BIPA exposure, etc.). The PROBLEM is real. This DOC's solution shape is wrong for the deployment.
+>
+> Do NOT implement this document as-written. When J greenlights audit-trail work, draft a v3 that's local-only sized (~3-5 days, not 17-20).
+>
+> See `STATE_OF_PLAY.md` "PRD line 70 is load-bearing" entry for the binding direction.
+
+**Status:** Draft v2 — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
+**Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md)
+
+> **v2 history.** v1 (2026-05-03 morning) was scrummed across opus + kimi + gemini lineages. 3/3 reviewers converged on 6 critical issues — wrong migration order, master key + legal token "security theater," BIPA `inferred_existing` prima facie violation, healthcare default in wrong direction, Merkle chain without external anchor. v2 incorporates those changes. Full scrum reviews preserved at `/tmp/identity_scrum/{opus,kimi,gemini}_review.md`. Diff between v1 and v2 captured in §11 change log.
+
+> **Why this exists.** Phase 1 + 1.5 confirmed today's substrate has no separation between candidate_id and PII. Both live in `workers_500k.parquet`. No per-access audit, no consent gate, no retention enforcement. This document specifies the new identity service that holds the candidate_id ↔ PII mapping, gates every PII read, audits every access, and serves as the single legal-attestable boundary between PII and the rest of the system.
+
+> **Prerequisite:** **Phase 1.6 (BIPA pre-launch gates) MUST ship before any identityd backfill begins.** Per kimi+gemini scrum, backfilling with `consent_status='inferred_existing'` is a BIPA §15 prima facie violation. Phase 1.6 establishes the consent template + retention schedule + employee training that turns backfill from "inferred" into "pending_backfill_review with documented escalation path."
+
+---
+
+## 1. Scope and non-goals
+
+### In scope
+- Single source of truth for `candidate_id ↔ PII` mapping
+- Per-PII-access audit log (who/what/when/why)
+- Consent + retention metadata (BIPA + IL Day and Temporary Labor Services Act + healthcare PHI)
+- Legal-tier access — short-lived JWT with **split-secret dual-control issuance** (NOT a static file token, per all 3 reviewers)
+- Healthcare-vertical routing with **fail-closed default** (`unknown` treated as healthcare until classified, per opus+gemini)
+- EU-compatible interface — fields exist, enforcement is per-subject (no system-wide flag flip needed)
+- Hardware-backed master key — **HashiCorp Vault Transit OR AWS KMS** for v1 (NOT a sealed file, per all 3 reviewers; sealed-file v1 was "obfuscation in any defensible sense" — kimi)
+- Signed-JSON audit response with PDF render path (PDF deferred; JSON ships first per opus)
+- External-anchor for audit-log integrity (S3 Object Lock + signed timestamps; NOT just Postgres Merkle, per all 3 reviewers)
+- mTLS or Unix Domain Socket between gateway ↔ identityd (per kimi — "port-isolated is theater without authenticated channel")
+- Field-level authorization on `/subjects/{id}` (per-purpose field allowlists, per kimi)
+- Synchronous cache invalidation hook for RTBF (per opus — even if Phase 7 is the full RTBF build)
+
+### Out of scope (for this phase)
+- The `/audit/subject/{id}` endpoint itself (Phase 3)
+- Full subject-tagging across other substrates (Phase 4)
+- Right-to-be-forgotten implementation full-flow (Phase 7)
+- Training-safe export (deferred from Phase 2 per opus — "not on critical path for audit-trail defense")
+- BIPA pre-launch gates content (Phase 1.6 — separate doc)
+
+### Architecturally simplified from v1 (per scrum)
+- **Per-row encryption keys** deferred to Phase 7. v2 uses a single Data Encryption Key (DEK) wrapped under HSM-backed Key Encryption Key (KEK), with per-subject **ciphertext-deletion** as the v2 erasure mechanism. This matches kimi's recommendation: "A single data-encryption key with HSM-backed rotation, plus per-subject deletion of ciphertext, achieves equivalent practical erasure with far less complexity." Cryptographic-erasure via per-subject keys becomes a Phase 7 enhancement.
+- **EU placeholder fields** kept (per J 2026-05-03), but enforcement code is genuinely no-op until first eu_resident=true subject — no code paths run, no schema migrations needed when EU comes online (just per-subject field population).
+- **Merkle chaining** narrowed (per gemini): full Merkle tree only for legal-tier and consent/erasure events. Standard `gateway_lookup` events get a simpler signed-HMAC chain. Both anchored externally.
+
+---
+
+## 2. Architectural shape
+
+### 2.1 — Process model + transport
+
+Per J 2026-05-03 + kimi scrum: separate daemon, mTLS-mandatory transport. Single Go implementation, both runtimes call it.
+
+| Property | Value |
+|---|---|
+| Name | `identityd` |
+| Port | `:3225` (single port, not dual; per kimi simplification — runtime-agnostic routing) |
+| Implementation | Go |
+| Storage | Postgres in **isolated database** (per J answer, confirmed). Schema-level isolation; no shared schemas with Langfuse or other lakehouse storage. |
+| Transport gateway↔identityd | **mTLS mandatory.** Self-signed CA managed by identityd at startup; gateway clients have their own client certs. Plain HTTP is rejected at the listener. |
+| Encryption-at-rest | **Single DEK wrapped under KEK in HashiCorp Vault Transit** (recommended) OR AWS KMS. Master key NEVER on disk. v1 of identityd refuses to start if KEK is unreachable. |
+| Audit-log signing | **Separate Ed25519 keypair from the master encryption key.** Signing key in Vault Transit (signing backend) OR a separate sealed-secret file with strict rotation procedure. |
+| External anchor for audit chain | **S3 Object Lock** (compliance mode, 7-year retention) holding periodic Merkle root commitments. Daemon writes hourly + on legal-tier event. |
+| Backup | Postgres standard backup; KEK backup separate (different storage, different access). Crypto-erasure model only works if these are not co-located. |
+| Rate limiting | Per-token QPS + daily-volume caps. Default legal-token: 100 lookups/day + 10K daily volume; alerting at 50%. Configurable per-token. |
+| Token revocation | `token_revocations` table checked on every legal-tier auth. Revocation propagates within 60s (TTL on cache). |
+
+### 2.2 — Schema (Postgres DDL, v2)
+
+```sql
+-- Subject record. PII columns are ciphertext under the DEK.
+CREATE TABLE subjects (
+    candidate_id    TEXT PRIMARY KEY,           -- UUID v7, NOT sequential (per kimi enumeration concern)
+    -- Encrypted PII fields. AES-256-GCM under DEK. NULL = not collected.
+    name_ct         BYTEA,
+    email_ct        BYTEA,
+    phone_ct        BYTEA,
+    address_ct      BYTEA,
+    ssn_ct          BYTEA,
+    dob_ct          BYTEA,
+    -- DEK version this subject was encrypted under. KEK rotates; DEK rotates per
+    -- KEK rotation. New DEK version = subjects re-encrypted in background sweep.
+    dek_version     INT NOT NULL,
+    -- Lawful basis + consent metadata
+    consent_status  TEXT NOT NULL CHECK (consent_status IN (
+        'pending_backfill_review',  -- backfill default; no biometric/PHI use until reviewed
+        'pending_first_contact',     -- new subject, awaiting consent UX
+        'given',                     -- explicit consent recorded
+        'withdrawn',                 -- subject revoked
+        'expired'                    -- consent timeout
+    )),
+    consent_version           TEXT,             -- references published consent template version
+    consent_given_at          TIMESTAMPTZ,
+    consent_withdrawn_at      TIMESTAMPTZ,
+    -- BIPA-specific fields (Phase 1.5 §1E + Phase 1.6 prerequisite)
+    biometric_consent_status  TEXT NOT NULL DEFAULT 'never_collected' CHECK (biometric_consent_status IN (
+        'never_collected', 'pending', 'given', 'withdrawn', 'expired'
+    )),
+    biometric_retention_until TIMESTAMPTZ,      -- BIPA: max 3 years from last interaction
+    -- Vertical detection — drives healthcare PHI routing.
+    -- DEFAULT 'unknown' (per opus+gemini scrum) — fail-closed routing treats
+    -- unknown as healthcare-equivalent until reclassified.
+    vertical                  TEXT NOT NULL DEFAULT 'unknown' CHECK (vertical IN (
+        'unknown', 'general', 'healthcare', 'finance', 'other'
+    )),
+    -- Per-vertical retention period. Drives the daily erasure sweep.
+    -- (Per opus: this was missing in v1. Required for BIPA's 3-year-from-
+    -- last-interaction rule which differs from generic retention.)
+    retention_period_days     INT NOT NULL,
+    -- EU-placeholder fields. Enforcement is per-subject; nothing runs
+    -- until a row has eu_resident=true AND lawful_basis IS NOT NULL.
+    eu_resident       BOOLEAN NOT NULL DEFAULT FALSE,
+    lawful_basis      TEXT,                     -- GDPR Art. 6 basis when eu_resident=true
+    transfer_mechanism TEXT,                    -- SCC, DPF, BCR — populated when EU comes online
+    -- Standard audit columns
+    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
+    last_interaction  TIMESTAMPTZ,              -- drives retention sweep
+    -- Crypto-erasure state (v2: ciphertext deletion, NOT key destruction)
+    erased_at         TIMESTAMPTZ,
+    erasure_reason    TEXT
+);
+
+-- Append-only access log. EVERY PII read writes a row.
+-- Field-level: fields_accessed records WHICH fields the caller resolved.
+-- purpose enforced against an allowlist per-purpose-token.
+CREATE TABLE pii_access_log (
+    access_id       BIGSERIAL PRIMARY KEY,
+    candidate_id    TEXT NOT NULL,
+    accessed_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
+    accessor_kind   TEXT NOT NULL,              -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve'
+    accessor_id     TEXT NOT NULL,              -- daemon name + caller token-hash; never raw token
+    purpose_token   TEXT NOT NULL,              -- opaque token; resolved through purpose_definitions table
+    fields_accessed TEXT[] NOT NULL,
+    request_trace_id TEXT,
+    chain_kind      TEXT NOT NULL CHECK (chain_kind IN ('hmac', 'merkle')),  -- HMAC for standard, Merkle for legal/consent/erasure
+    integrity_hash  TEXT NOT NULL,
+    -- Per opus security finding: this row's existence is itself sensitive
+    -- ("candidate X is under legal review" leaks via purpose). Read access
+    -- to this table requires legal-tier auth or scoped per-subject auth.
+    is_legal_tier_event BOOLEAN NOT NULL DEFAULT FALSE
+);
+
+-- Purpose definitions — enforces field-level authorization (per kimi).
+-- A given purpose_token can only request fields in its allowlist.
+CREATE TABLE purpose_definitions (
+    purpose_token   TEXT PRIMARY KEY,           -- e.g. 'fill_validation', 'audit_subject_response'
+    description     TEXT NOT NULL,
+    allowed_fields  TEXT[] NOT NULL,            -- e.g. ARRAY['name'] for fill_validation
+    auth_tier       TEXT NOT NULL CHECK (auth_tier IN ('service', 'legal')),
+    rate_limit_qps  INT,
+    daily_volume_cap INT
+);
+
+-- Token revocation list. Checked on every auth, cached 60s.
+CREATE TABLE token_revocations (
+    token_hash      TEXT PRIMARY KEY,
+    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+    revoked_by      TEXT NOT NULL,
+    reason          TEXT
+);
+
+-- Consent template versioning. Hash references, not embedded text per row.
+CREATE TABLE consent_versions (
+    version         TEXT PRIMARY KEY,
+    effective_at    TIMESTAMPTZ NOT NULL,
+    superseded_at   TIMESTAMPTZ,
+    template_hash   TEXT NOT NULL,              -- SHA256 of the canonical template
+    template_path   TEXT NOT NULL               -- where the canonical text lives (e.g. data/_consent/v3-2026-04-15.md)
+);
+
+-- External anchor checkpoints. Periodically committed to S3 Object Lock.
+CREATE TABLE anchor_commits (
+    commit_id       BIGSERIAL PRIMARY KEY,
+    chain_kind      TEXT NOT NULL,              -- 'hmac' or 'merkle'
+    checkpoint_at   TIMESTAMPTZ NOT NULL,
+    last_access_id  BIGINT NOT NULL,            -- max access_id covered by this checkpoint
+    root_hash       TEXT NOT NULL,
+    s3_object_uri   TEXT NOT NULL,              -- s3://anchor-bucket/identityd/...
+    s3_lock_until   TIMESTAMPTZ NOT NULL        -- compliance-mode retention end
+);
+```
+
+### 2.3 — HTTP surface (v2)
+
+All under `/v1/identity/`. mTLS + token required on every endpoint.
+
+| Method + Path | Purpose | Auth tier | Notes |
+|---|---|---|---|
+| `POST /v1/identity/subjects` | Create new subject. Body: PII fields + `purpose_token`. Returns: candidate_id (UUID v7). | service | Validates purpose_token allowlist matches submitted fields |
+| `GET /v1/identity/subjects/{candidate_id}?purpose={token}&fields={list}` | Resolve PII for candidate. | service | **Field-level enforcement**: returned fields ⊆ purpose_token's allowed_fields. Mismatch → 403. Logs access row. |
+| `GET /v1/identity/subjects/{candidate_id}/vertical` | **Minimal-disclosure** vertical lookup (per kimi HIPAA minimum-necessary). Returns only `{vertical, consent_status}`. | service | Used by gateway routing. Cheaper access-log row. |
+| `GET /v1/identity/subjects/{candidate_id}/full` | Complete subject record + audit summary. | **legal** | Short-lived JWT only. Logged with `is_legal_tier_event=true`. Triggers real-time notification to designated counsel + J. |
+| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn with version. | service | |
+| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure. Idempotent. | **legal** | Synchronously triggers cache invalidation hooks (per opus). |
+| `GET /v1/identity/access_log/{candidate_id}` | Per-subject access log for audit response. | **legal** | |
+| `POST /v1/identity/auth/legal_token` | Issue short-lived JWT (max 24h). Requires **dual-control attestation**: J's signed nonce + counsel's signed nonce, both verified against pre-registered public keys. | dual-control | Per all 3 reviewers — replaces v1's static file approach |
+| `GET /v1/identity/health` | Liveness | none | Returns only `{status: "ok"}` — no version, no schema info |
+| `(no public training-safe-export endpoint in v2)` | — | — | Deferred per opus — not on critical path |
+
+### 2.4 — Auth model (v2 — split-secret + short-lived)
+
+**Service-tier auth** (gateway → identityd for routine PII resolution):
+- mTLS client cert per gateway (Rust :3100 + Go :4110 each get their own cert)
+- Bearer token in `Authorization` header (long-lived — 90d), rotated per ops runbook
+
+**Legal-tier auth** (per-request, short-lived JWT, dual-control issuance):
+- J holds private key A
+- Designated outside counsel holds private key B
+- `POST /v1/identity/auth/legal_token` requires:
+  - HTTP body: `{purpose, ttl_seconds, requested_fields, signature_a, signature_b, nonce_a, nonce_b}`
+  - identityd verifies both signatures against pre-registered A and B public keys
+  - On success: emits a JWT signed by identityd, valid for `ttl_seconds` (max 24h), scoped to `purpose`+`fields`
+- The JWT:
+  - Cannot be issued without BOTH A and B signatures (per gemini "split-secret startup ceremony" pattern, applied per-token)
+  - Carries scope (purpose + fields) inside its claims; identityd enforces at use time
+  - Per-token rate limit + daily cap recorded on issuance
+  - Revocation: token_hash added to `token_revocations`; identityd refuses on next check (60s cache TTL)
+
+**Why this matters:**
+- v1's "static file mode 0400" was security theater per all 3 reviewers
+- A leaked legal token v1 = unbounded exfiltration window until rotation
+- v2's leaked JWT = bounded by TTL (max 24h) AND revocable in <60s
+- Single-actor compromise (J's machine OR counsel's machine, not both) cannot mint legal tokens
+
+---
+
+## 3. Integration with the rest of the substrate
+
+### 3.1 — Gateway changes
+
+When gateway needs PII for a fill scenario:
+
+1. Gateway has only `candidate_id` (post-Phase-1.5 view-routing fix)
+2. Gateway calls `GET /v1/identity/subjects/{candidate_id}/vertical` first if it might be healthcare-vertical-sensitive routing
+3. Gateway calls `GET /v1/identity/subjects/{candidate_id}?purpose=fill_validation&fields=name`
+4. identityd validates: purpose_token=`fill_validation` allows `[name]` only; if request asks for `[name,ssn]` → 403
+5. identityd writes `pii_access_log` row, decrypts requested fields, returns
+6. Gateway uses fields, then **explicitly zeros memory** (kimi's drop-and-overwrite pattern) before returning HTTP response
+7. **No PII caching** in gateway memory beyond request lifetime
+
+**LRU embed cache** (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Phase 4 task to re-key as `(model, candidate_id, field_subset_hash)`. **Phase 2 must add the synchronous cache-purge hook** so RTBF in Phase 2 invalidates Phase 4's cache when it ships (opus finding).
+
+### 3.2 — Cross-runtime: same identityd, both gateways call it
+
+Both gateways call identityd over mTLS. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates identical PII request through both gateways produces identical access-log rows (modulo daemon-name field).
+
+### 3.3 — JSONL writer changes (subject_id top-level promotion)
+
+Per `AUDIT_PHASE_1_DISCOVERY` §10/C5:
+- All JSONL sinks (outcomes, sessions, overseer_corrections, observerd ops) gain top-level `subject_ids: [...]` field
+- `fills[*].name` replaced with `name_ref: "[REDACTED-{candidate_id}]"`
+- Authorized callers dereference via identityd
+
+### 3.4 — Langfuse boundary redaction (defense in depth, per opus TOCTOU finding)
+
+v1 design: per-request map of `subject_id → resolved_PII`, replace before Langfuse POST.
+
+**v2 adds** (per opus): outbound regex/NER pass on the Langfuse payload as defense-in-depth. The model can hallucinate names not in the resolved-PII map. The outbound NER pass catches them.
+
+```
+Gateway constructs Langfuse payload
+  → Pass 1: per-request resolved-PII map replacement
+  → Pass 2: NER scan (regex for SSN/phone shapes; named-entity model for names)
+  → Either pass detecting unredacted PII → drop the trace OR replace with [POSSIBLE-PII-DETECTED]
+  → POST to Langfuse
+```
+
+### 3.5 — Healthcare vertical routing — fail-closed default
+
+**Per opus + gemini scrum: Default is `unknown` → treated as healthcare for routing purposes.**
+
+When gateway receives a request involving any candidate:
+1. Query `/vertical` for the candidate
+2. If `vertical IN ('healthcare', 'unknown')` → route to **on-box Ollama only** (no opencode/openrouter/ollama_cloud egress)
+3. If `vertical='general'` (or other non-healthcare) → cloud routing OK
+4. If identityd unreachable → fail closed (refuse the request, return 503)
+
+**Reclassification path:** subjects can be moved from `unknown` → `general` via explicit operator action (after manual review). Subjects flagged `healthcare` stay healthcare unless explicitly downgraded.
+
+**Auto-escalation** (per kimi): if a subject's resume_text or call_log content is updated with healthcare-pattern matches (RN, BSN, hospital, MD, physician, etc.), vertical auto-escalates to `healthcare`. Never auto-de-escalates.
+
+### 3.6 — Cache invalidation hook (Phase 2 must ship even though full RTBF is Phase 7)
+
+When `POST /v1/identity/subjects/{id}/erase` fires:
+1. Mark `subjects.erased_at` and zero out ciphertext columns
+2. Write `pii_access_log` row with `purpose_token='retention_expired'` or `'rtbf_request'`
+3. **Synchronously** publish a `subject_erased` event to a pub/sub channel (Redis or Postgres LISTEN/NOTIFY)
+4. Gateway subscribes; on event, purges any in-flight cache entries for that candidate_id
+5. Eventual-consistency window: <5s between erase-call and cache-flush
+
+Without this hook, RTBF in Phase 7 would be a lie because the gateway's LRU embed cache could still hold the subject's data.
+
+---
+
+## 4. Audit response shape (Phase 3 preview, v2 — adds model-version snapshot per gemini)
+
+```json
+{
+  "schema": "audit.subject.v1",
+  "subject_token": "01926f2e-7c1b-7000-...",     // UUID v7, NOT CAND-NNNNNN
+  "request_window": { "from": "2026-01-01", "to": "2026-05-03" },
+  "generated_at": "2026-05-03T12:00:00Z",
+  "generated_by": "identityd@hostname",
+  "merkle_root": "sha256:...",                    // root hash of legal/consent/erasure events
+  "external_anchor": "s3://anchor-bucket/identityd/2026-05-03T12-00.json",  // S3 Object Lock URI
+  "signature": "ed25519:...",                     // separate signing key from encryption KEK
+  "consent": { ... },
+  "decisions": [
+    {
+      "ts": "2026-04-22T09:15:23Z",
+      "decision_kind": "fill_recommendation",
+      "daemon": "gateway",
+      "model": "kimi-k2.6",
+      "model_version_hash": "sha256:...",         // NEW per gemini — proves what model existed AT decision time
+      "provider": "ollama_cloud",
+      "trace_id": "trace-abc",
+      "session_id": "session-xyz",
+      "input_features": { ... },                  // sanitized; no protected attributes; no inferred-attribute proxies
+      "output": "...",
+      "rationale": "...",
+      "comparator_pool_size": 47,
+      "comparator_appendix_ref": "see comparator_appendix.A"
+    }
+  ],
+  "comparator_appendix": {
+    "A": {
+      "scope": "fill scenarios in window matching role X, geo Y",
+      "total_pool_size": 47,
+      "selection_rate_by_protected_class": {
+        // Aggregated; NO other subjects' identifiers leak
+        "race": { "white": 0.33, "black": 0.31, ... },
+        "gender": { "man": 0.34, "woman": 0.30, ... }
+      },
+      "four_fifths_test": "passed"  // or "concern: rate ratio 0.71"
+    }
+  },
+  "access_log": [...],
+  "footer": {
+    "completeness_attestation": "all decisions about subject_token in window per retention policy v2 are included",
+    "merkle_proof": "...",                        // proof this audit's root is in S3 anchor
+    "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0"
+  }
+}
+```
+
+---
+
+## 5. Migration path (v2 — REORDERED per all 3 reviewers)
+
+v1's order was wrong. New order, with explicit prerequisites:
+
+| Step | Action | Prerequisite |
+|---|---|---|
+| **0** | **Phase 1.6 BIPA pre-launch gates SHIPPED** — consent template published, retention schedule public, deletion procedure documented, employee training acknowledged. | (separate work — see PHASE_1_6_BIPA_GATES.md when written) |
+| **1** | Stand up identityd with **synthetic test subjects only**. KEK in vault, mTLS live, all endpoints serve. | Vault/KMS available; mTLS CA bootstrapped |
+| **2** | **Audit-parity probe** (`audit_parity.sh`) green on synthetic data. Cross-runtime equivalence verified. | Step 1 |
+| **3** | Add gateway feature flag `LH_USE_IDENTITY_SERVICE`. **Shadow-write only** — gateway writes new subjects to identityd AND continues SQL path. No reads from identityd yet. | Step 2 |
+| **4** | Run shadow-write for ≥1 week. Validate access logs, encryption correctness, write-path performance under real traffic. | Step 3 |
+| **5** | Backfill from `workers_500k.parquet`. **`consent_status='pending_backfill_review'`** for ALL existing rows. **`vertical='unknown'`** default (NOT 'general'). **`biometric_consent_status='never_collected'`** — backfill does NOT include any biometric data. | Step 0 (BIPA gates) + Step 4 |
+| **6** | Human-review queue for vertical reclassification. Subjects move from `unknown` to `general` only after explicit review. Healthcare-pattern matches auto-escalate to `healthcare` (never auto-downgrade). | Step 5 |
+| **7** | **Shadow-read** — gateway reads from BOTH SQL and identityd, compares, logs divergences, returns SQL result. Run ≥1 week. | Step 5 |
+| **8** | Feature-flag cutover — gateway reads from identityd, falls back to SQL on error, alerts on every fallback. | Step 7 |
+| **9** | Quarantine PII columns in workers_500k.parquet — only after **cryptographic attestation of identityd completeness** (Merkle proof: source row hash = identityd row hash for every candidate_id). Move PII columns to a different bucket; the candidate_id-only projection becomes operational. | Step 8 |
+
+**Key v2 changes from v1:**
+- New Step 0 (BIPA gates prerequisite) — backfill cannot start without this
+- Steps 3-4 (shadow-write before backfill) — production validates the WRITE path before data lands
+- Step 5 backfill consent: `pending_backfill_review` not `inferred_existing` (BIPA defense)
+- Step 5 vertical default: `unknown` not `general` (HIPAA fail-closed)
+- New Step 6 (human review) — vertical classification via explicit operator action
+- Step 7 added (shadow-read after backfill) — catches encoding/normalization bugs before cutover
+- Step 9 (quarantine) requires cryptographic attestation of completeness
+
+Each step is its own commit, its own gate, its own rollback path.
+
+---
+
+## 6. Cross-runtime parity probe (NEW, per scrum §6 plus opus 'oracle test' addition)
+
+`audit_parity.sh` ships in Phase 5. Asserts:
+
+1. Same PII fetch through Rust gateway and Go gateway produces identical identityd access-log rows (modulo daemon name)
+2. Crypto-erasure of test subject through Rust gateway is honored when Go gateway tries to fetch
+3. Healthcare-vertical routing decision identical across both runtimes
+4. **Oracle test** (per kimi SOC2 CC4.1 finding): probe with known-good inputs against expected outputs. A bug present in BOTH implementations must not pass the parity probe.
+5. Discrimination-proxy phrase redaction triggers as expected on adversarial test cases
+
+---
+
+## 7. What this design intentionally does NOT solve
+
+- Protected-attribute exclusion at decision time (Phase 6)
+- Pathway memory trace body redaction (Phase 4)
+- Retroactive Langfuse history scrub (separate runbook)
+- GDPR Art. 22 right to explanation full implementation (Phase 8)
+- Multi-region data residency (single-region US-Midwest by default)
+- Training-safe export (deferred — not on critical path per opus)
+
+---
+
+## 8. Open questions — RESOLVED 2026-05-03
+
+J confirmed all 6 v1 recommendations + scrum-driven changes:
+
+1. ✅ **Master key:** HashiCorp Vault Transit (recommended) OR AWS KMS. NOT a sealed file. v1's sealed-file recommendation is rejected per all 3 reviewers ("obfuscation in any defensible sense").
+2. ✅ **Postgres isolation:** identityd's own database, isolated schema.
+3. ✅ **Vertical backfill:** `'unknown'` default with fail-closed routing. NOT `'general'` per opus+gemini scrum.
+4. ✅ **Legal token:** Split-secret dual-control issuance (J + counsel both sign). Short-lived JWT, max 24h, revocable in <60s.
+5. ✅ **Crypto-erasure sweep:** Daily 03:00 UTC. (Note: v2 erasure mechanism is ciphertext deletion, not key destruction. Per-row keys deferred to Phase 7.)
+6. ✅ **EU enforcement:** Per-subject. Schema fields exist; nothing runs until first eu_resident=true subject.
+
+### Newly resolved per scrum
+
+7. ✅ **Migration order:** REORDERED per §5 above. New Step 0 (BIPA prerequisite). Shadow-write before backfill. Shadow-read before cutover.
+8. ✅ **Audit-log external anchor:** S3 Object Lock with compliance-mode 7-year retention. Hourly + on-event commits.
+9. ✅ **Audit-log signing key:** Separate Ed25519 keypair from KEK. Vault Transit signing backend OR sealed-secret with strict rotation runbook.
+10. ✅ **mTLS gateway↔identityd:** Mandatory. Self-signed CA managed by identityd at startup.
+11. ✅ **Per-row encryption keys:** Deferred to Phase 7. v2 uses single DEK with HSM-wrapped KEK + ciphertext deletion for erasure.
+12. ✅ **Field-level authorization:** purpose_definitions table enforces per-purpose field allowlists.
+13. ✅ **Synchronous cache invalidation:** pub/sub event on erase; gateway subscribes.
+14. ✅ **Outbound NER pass for Langfuse:** Defense-in-depth in addition to symbol-table replacement.
+15. ✅ **Model version hash in audit response:** Captured per decision row.
+16. ✅ **PDF render:** Deferred from Phase 2; JSON ships first.
+
+---
+
+## 9. Estimated implementation cost (revised v2)
+
+| Sub-phase | Effort | Notes |
+|---|---|---|
+| 2A — Postgres schema + Vault/KMS integration | 1.5-2 days | Includes mTLS CA bootstrap + signing-key separation |
+| 2B — identityd HTTP surface (Go) | 2-3 days | All endpoints, auth, dual-control JWT issuance, rate limiting, revocation |
+| 2C — Backfill ETL (BIPA-compliant — pending_backfill_review) | 1 day | Plus Merkle attestation of completeness |
+| 2D — Gateway integration (Rust + Go, mTLS, shadow-write phase) | 2-3 days | Per-tool migration, parity probe |
+| 2E — JSONL writer changes (subject_id top-level promotion) | 1 day | All sinks |
+| 2F — Langfuse redaction (symbol-table + NER) | 2 days | Two passes + drop-on-detect |
+| 2G — Healthcare-vertical routing (fail-closed) | 0.5 day | Plus auto-escalation pattern matcher |
+| 2H — Cache invalidation pub/sub hook | 0.5 day | Critical for Phase 7 RTBF |
+| 2I — External anchor (S3 Object Lock) | 1 day | Hourly + on-event commits |
+| 2J — Cross-runtime parity probe with oracle tests | 1 day | New probe |
+| **Total** | **~12-15 working days** | (v1 estimated 8-10; v2 added BIPA gates dependency, mTLS, dual-control JWT, NER pass, anchor, etc.) |
+
+Bigger than v1. Worth it — every addition was a 3/3-reviewer convergent finding.
+
+---
+
+## 10. The four "would not build" blockers from scrum (all addressed in v2)
+
+| # | v1 issue (3/3 reviewers) | v2 resolution |
+|---|---|---|
+| 1 | Migration order (backfill before validation) | §5 reordered; Step 0 BIPA gates prereq added |
+| 2 | Master key on disk + legal token static file | §2.1 Vault/KMS for KEK; §2.4 split-secret short-lived JWT |
+| 3 | `inferred_existing` BIPA prima facie violation | §5 Step 5 uses `pending_backfill_review`; Step 0 gates required first |
+| 4 | Healthcare default `general` (HIPAA exposure window) | §3.5 fail-closed `unknown`-as-healthcare; §5 Step 5 backfill default `unknown` |
+
+All three reviewers said "I would not build v1 as written." All four blockers are resolved in v2. Re-scrum recommended before implementation starts.
+
+---
+
+## 11. Change log (v1 → v2)
+
+| Section | v1 | v2 |
+|---|---|---|
+| Master key storage | Sealed file `/etc/lakehouse/identityd_master.key` | HashiCorp Vault Transit / AWS KMS |
+| Legal token | Static file mode 0400 | Split-secret short-lived JWT (max 24h), dual-control issuance |
+| Encryption keys | Per-row keys with master wrapping | Single DEK + HSM-wrapped KEK + ciphertext deletion (per-row deferred to Phase 7) |
+| Healthcare default | `vertical='general'` backfill | `vertical='unknown'` backfill, fail-closed routing |
+| Migration §5 order | 1: stand up, 2: backfill, 3: feature flag, 4: cutover, 5: quarantine | 0: BIPA gates, 1: stand up, 2: parity probe, 3: shadow-write, 4: shadow-write soak, 5: backfill (pending_backfill_review), 6: human vertical review, 7: shadow-read, 8: cutover, 9: quarantine with cryptographic attestation |
+| Audit-log integrity | Postgres Merkle chain only | Postgres Merkle for legal/consent/erasure + HMAC for standard + S3 Object Lock external anchor |
+| Audit-log signing key | Same as encryption key | Separate Ed25519 keypair; Vault signing backend |
+| Gateway↔identityd transport | Plain HTTP (implied) | mTLS mandatory |
+| Field authorization | Per-endpoint only | Per-purpose-token field allowlists |
+| Cache invalidation | Phase 4 / Phase 7 | Phase 2 ships pub/sub hook |
+| Langfuse redaction | Symbol-table replacement only | Symbol-table + outbound NER pass + drop-on-detect |
+| Audit response | No model version | model_version_hash per decision |
+| PDF render | Phase 2 | Deferred — JSON ships first |
+| Training-safe export | Phase 2 | Deferred — not critical path |
+| Effort estimate | 8-10 days | 12-15 days |
+
+---
+
+## 12. v3 amendments (post-second-pass scrum, 2026-05-03 evening)
+
+v2 was re-scrummed across opus + kimi + gemini. **All 3 verdict: BUILD-WITH-CHANGES.** All 4 v1 blockers verified RESOLVED. New v2 findings are tractable design fixes (not re-architecture). Folded in below. Reviews preserved at `/tmp/identity_scrum_v2/{opus,kimi,gemini}_review.md`.
+
+### Convergent v2 findings (≥2 reviewers) — must fix before implementation
+
+**v3-A1 — mTLS CA root must NOT live in identityd** (opus + gemini converge). v2 said "self-signed CA managed by identityd at startup" — if identityd is both the CA and authenticated party, compromise of identityd compromises the whole trust fabric. **v3 change:** mTLS CA root lives in **Vault PKI** (or, if Vault PKI is unavailable, an offline root with identityd as a properly-issued intermediate). identityd never has root-CA private key access.
+
+**v3-A2 — Dual-control public key registry must be tamper-evident** (opus + gemini converge). v2 said "pre-registered public keys" but didn't say where. If they live in identityd's Postgres, DB-write compromise defeats dual-control. **v3 change:** J + counsel public keys live in **Vault KV with separate access policies** (or a signed config file at `/etc/lakehouse/identityd_dual_control.yaml` with its own dual-control rotation procedure). Public-key changes themselves require dual-control attestation. Plus: `nonce_a`/`nonce_b` get **server-issued challenges** (identityd issues a fresh nonce per request; clients sign that nonce; replay-protection via 5-min nonce cache).
+
+### Single-reviewer v2 findings — also folded into v3
+
+**v3-B1 (opus): Step 8 fallback-to-SQL needs explicit time bound.** Otherwise becomes permanent dual-read debt. v3: fallback active for max 14 days post-cutover; alert on every fallback; auto-disable at 14 days regardless of fallback rate.
+
+**v3-B2 (opus): NER drop-on-detect needs alerting.** Silent observability gap if model regression starts hallucinating PII. v3: drop counter exposed as `identityd_ner_drops_total` Prometheus metric; alert on drop_rate >0.1% over rolling 1h.
+
+**v3-B3 (opus): legal-tier notification transport must be specified.** v2 said "real-time notification to designated counsel + J" without saying how. v3: notification transport is **signed Slack webhook OR signed email** (configurable per deployment); message body never contains candidate_id or PII; only `{event_kind, timestamp, accessor_kind, integrity_hash}`. Notification failure does NOT block the legal-tier action — failure is logged for follow-up but token issuance proceeds.
+
+**v3-B4 (opus): Step 6 human review queue needs SLA + volume estimate.** With 500k backfilled rows defaulting to `unknown`, all healthcare-adjacent traffic routes to on-box Ollama until reclassified. v3: budget assumption — auto-pattern-matching pre-classifies ~80% to confidently-non-healthcare; remaining 20% (~100k) stays `unknown` pending review at ~500/day operator throughput = ~7 months of queue. **Operational decision needed:** either staff the queue, accept the on-box-Ollama-only routing for ~7 months, or relax pattern matching at the cost of higher false-negative rate. Flag for J.
+
+**v3-B5 (opus): Memory zeroing in Go is non-trivial.** v2 §3.1 step 6 says "explicitly zeros memory" — Go GC-managed strings are immutable, not zeroable. v3 implementation note: PII handling code path uses `[]byte` throughout; convert to string only at the JSON serialization boundary; explicit `runtime.GC()` after request completes is theater (Go won't actually zero the slice). **Acceptance:** "best-effort zero" — overwrite the `[]byte` slice contents post-use. Document this is best-effort, not cryptographic-grade scrubbing. (Rust-side: use the `zeroize` crate which IS cryptographic-grade.)
+
+**v3-B6 (kimi): Service-tier purpose_definitions needs versioning + emergency revocation.** A misconfigured purpose with overly-broad `allowed_fields` is standing exfiltration authorization. v3 schema additions:
+
+```sql
+CREATE TABLE purpose_versions (
+    purpose_token   TEXT NOT NULL,
+    version         INT NOT NULL,
+    effective_at    TIMESTAMPTZ NOT NULL,
+    superseded_at   TIMESTAMPTZ,
+    PRIMARY KEY (purpose_token, version)
+);
+
+CREATE TABLE purpose_revocations (
+    purpose_token   TEXT PRIMARY KEY,
+    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+    revoked_by      TEXT NOT NULL,
+    reason          TEXT
+);
+```
+
+Auth path checks `purpose_revocations` on every call (not cached, OR cached <5s). Revocation is <60s without code deploy.
+
+**v3-B7 (kimi): Cache invalidation needs erasure-generation atomicity.** v2 §3.6 pub/sub is best-effort. If Redis drops the message or gateway crashes between erase + purge, cache holds stale PII. v3 change:
+- `subjects` table gains `erasure_generation INT NOT NULL DEFAULT 0`. Increments on every erase.
+- Identityd PII responses include current generation in headers
+- Gateway cache entries tagged with the generation they were filled at
+- Cache hits with `cached_generation < current_generation` are rejected, force re-fetch from identityd
+- Eventually-consistent within ONE request hop, not eventually-consistent over time
+
+**v3-B8 (kimi): Dual-control issuance needs cooling-off period to prevent bypass culture.** v3: 15-minute mandatory delay between nonce submission and token issuance, with notification to BOTH parties at delay-start AND issuance. Bypass requires both J and counsel to sign an "emergency bypass" attestation that's logged separately + escalated to a board-level reviewer.
+
+**v3-B9 (kimi): NER must have calibrated false-negative rate.** v3: maintain a synthetic adversarial test set (50 hallucinated-PII examples + 50 legitimate non-PII). NER must achieve recall ≥99.5% on the synthetic set in CI; production drop-rate >0.1% triggers re-calibration.
+
+**v3-B10 (kimi): S3 Object Lock root credentials need separate AWS account.** Same ops team holding Vault admin + AWS root = limited independence. v3: Object Lock bucket lives in **separate AWS account** with **write-only IAM role** issued to identityd. Root credentials for that account held by named external party (could be outside counsel) with quarterly attestation.
+
+**v3-B11 (kimi): biometric_consent_status='never_collected' may not satisfy BIPA infrastructure-as-notice argument.** Plaintiffs may argue presence of biometric schema fields constitutes constructive notice of intent. v3: Phase 1.6 BIPA policy doc must explicitly attest "no biometric data exists in the pre-identityd substrate" with cryptographic evidence (hash of pre-identityd schema). Counsel must bless this attestation.
+
+**v3-B12 (gemini): Backup retention window vs ciphertext-deletion erasure.** Phase 2 uses ciphertext deletion + single-DEK; if Postgres backups retain pre-erasure ciphertext + the DEK still exists, backup is recoverable. v3 erasure runbook: includes maximum backup retention period (e.g., 30 days). RTBF status reported to subject as "erased now; residual in backups expires by {date}." For full crypto-erasure-on-demand, Phase 7 per-row keys is the path.
+
+### Net v3 effort delta
+
+v2 estimate: 12-15 days. v3 amendments add:
+- mTLS CA via Vault PKI integration: +0.5 day
+- Public-key registry in Vault KV: +0.5 day
+- Server-issued nonces for dual-control: +0.25 day
+- Step 8 fallback time bound: +0.1 day
+- NER drop-rate metric + alert: +0.25 day
+- Legal-tier notification transport: +0.5 day
+- purpose_versions + purpose_revocations: +0.5 day
+- erasure_generation cache atomicity: +0.5 day
+- 15-min cooling-off: +0.25 day
+- NER calibrated test set: +0.5 day
+- Separate AWS account for Object Lock: +0.5 day
+- BIPA infrastructure-as-notice attestation: +0.25 day (mostly doc work)
+- Backup-retention erasure runbook: +0.25 day
+
+**Total v3 delta: ~5 days. Revised estimate: 17-20 days.**
+
+The cost is real. The reason it's worth paying is what those 5 days buy: a design that 3 independent senior security architects (across 3 model lineages) all said they would build. v1 said "do not build." v3 says "build."
+
+### What's v3-deferred vs v3-must-have
+
+**v3 must-have (block implementation):** v3-A1, v3-A2, v3-B1, v3-B6, v3-B7, v3-B11.
+
+**v3 should-have (ship in Phase 2 if calendar allows; otherwise Phase 5):** v3-B2, v3-B3, v3-B4 (operational decision), v3-B5 (best-effort), v3-B8, v3-B9, v3-B10, v3-B12.
+
+If schedule pressure forces a cut, the should-have items can ship in Phase 5 (identity service build completion). The must-haves cannot — they're integral to the security boundary.
+
+---
+
+## Change log
+
+- 2026-05-03 — v1 initial draft.
+- 2026-05-03 — v2 post-first-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum recommended.
+- 2026-05-03 — v3 post-second-scrum: 3/3 BUILD-WITH-CHANGES. v1 blockers verified resolved. 12 new v2 findings folded into v3 amendments (§12). Re-scrum NOT required for v3 — diminishing returns; the must-have items are concrete fixes with clear acceptance criteria.

docs/MATRIX_AGENT_HANDOVER.md

@@ -0,0 +1,103 @@
+# Matrix Agent Validated — Handover Notes
+
+**Date created:** 2026-04-25
+**Status:** active checkpoint with rolling fixes
+
+This doc explains the relationship between the lakehouse repo (this one)
+and the standalone `matrix-agent-validated` checkpoint that was carved
+out on 2026-04-25 to give a fresh Claude Code session a clean entry
+point on a fresh box.
+
+## The two repos
+
+| Repo | Purpose | Branch |
+|---|---|---|
+| `https://git.agentview.dev/profit/lakehouse` (this) | Full project history, ~12 months, 15 crates, all phases. Source of truth for active development. | `scrum/auto-apply-19814` (PR #11) |
+| `https://git.agentview.dev/profit/matrix-agent-validated` | Standalone checkpoint of the matrix-driven agent loop + Mem0-versioned pathway_memory. Single deploy unit. | `main` |
+
+The checkpoint repo is **not a fork** — it's a deliberate snapshot. It
+carries the bare minimum needed to run the agent loop on a fresh box
+with no lakehouse history baggage. Bug fixes that apply to both should
+land in lakehouse first, then cherry-pick (or be re-snapshotted) to
+matrix-agent-validated.
+
+## The Ansible playbook
+
+`/home/profit/handover/ansible/` on the source box (192.168.1.176)
+provisions a fresh Debian 13 box end-to-end:
+
+```
+swap → system → postgres → minio → repo → test_data → build → services → handover_note
+```
+
+The `handover_note` role renders `HANDOVER.md` on the destination from
+`roles/handover_note/templates/HANDOVER.md.j2` — this is THE entry-point
+file a fresh Claude Code session should read first when landing on a
+deployed box. If it is missing, re-run `ansible-playbook -i inventory.ini
+playbook.yml --tags handover_note`.
+
+## Destinations — what is real, what is test
+
+### `matrix-test` — REAL destination
+- Incus container on the source machine (192.168.1.176).
+- Reachable on `10.111.129.50` (incusbr0 tunnel, used by Ansible inventory),
+  `192.168.2.112`, `192.168.1.32`.
+- SSH: `profit:profit13`, sudo via password, key at `/root/.ssh/ironclaw`.
+- Install dir: `/home/profit/matrix-agent-validated/`.
+- All four services running: gateway/sidecar/observer/postgres.
+- Pathway memory: 82+ traces (don't wipe `data/_pathway_memory/state.json`).
+- Treat any work landing here as canonical for the checkpoint.
+
+### `192.168.1.145` — TEST VENV ONLY, NOT A DEPLOY TARGET
+- Separate VPS, hostname `ironclaw` — that's the preinstalled
+  community-scripts daemon, **NOT** what this project is.
+- The Ansible playbook ran here once on 2026-04-25 ~20:02 to verify
+  the repo would boot on a fresh box. Got partway through the `system`
+  role (swap + apt + ollama + bun + adduser) and stopped.
+- **No matrix-agent-validated dir exists here.** No lakehouse services.
+  No HANDOVER.md.
+- **Do not push fixes here. Do not treat it as production.** If you
+  find yourself debugging .145, you're in the wrong place — pivot to
+  matrix-test.
+
+## What landed today (2026-04-25 evening)
+
+1. **Observer health-probe bug fixed** (`mcp-server/observer.ts:645`) —
+   `r.json()` on the gateway's `text/plain "lakehouse ok"` response was
+   throwing, causing the observer service to crash-loop every 5s.
+   Fix: `r.ok ? r.text() : null`. Sealed in pathway_memory as
+   `TypeConfusion:fetch-health-json`. Committed both repos.
+
+2. **HANDOVER.md rendered on matrix-test** — the `handover_note` role
+   was previously missed; one re-run rendered the doc + backed up
+   `state.json` for safety.
+
+3. **Relevance filter shipped** — `mcp-server/relevance.ts` (heuristic
+   scorer) + observer `/relevance` endpoint + `scrum_master_pipeline.ts`
+   wiring (env opt-out via `LH_RELEVANCE_FILTER=0`, threshold via
+   `LH_RELEVANCE_THRESHOLD`). 9 unit tests green. Drops adjacency-pollution
+   chunks from matrix retrieval before the reviewer LLM sees them.
+
+## Still queued (from the rendered HANDOVER.md on matrix-test)
+
+- Audit-consensus → retire wire (auto-retire poisoned pathways when
+  observer rejects)
+- Mode router (port LLM Team patterns from source `/root/llm_team_ui.py`)
+- Pgvector backend (third option alongside Parquet + Lance, RAG/training
+  focused)
+- Local-model scratchpad daemon (continuous summarizer for active context)
+
+## Commands a fresh session should run first
+
+```bash
+# On the source box (192.168.1.176)
+git -C /home/profit/lakehouse log --oneline -10                # what's recent
+cat /home/profit/lakehouse/docs/MATRIX_AGENT_HANDOVER.md       # this file
+cat /home/profit/lakehouse/docs/SCRUM_MASTER_SPEC.md           # scrum loop spec
+
+# On matrix-test (real destination)
+ssh -i /root/.ssh/ironclaw profit@10.111.129.50 \
+  'cat /home/profit/matrix-agent-validated/HANDOVER.md'         # cold-start guide
+ssh -i /root/.ssh/ironclaw profit@10.111.129.50 \
+  'curl -s http://localhost:3100/health'                        # smoke test
+```

docs/MODE_RUNNER_TUNING_PLAN.md

@@ -0,0 +1,114 @@
+# Mode Runner Tuning Plan
+
+**Date:** 2026-04-26
+**Branch:** `scrum/auto-apply-19814` (PR #11)
+**Status:** Pass 5 variance test complete; conclusions locked. Implementation in progress.
+
+A fresh Claude session reading this + the pass5 row range in `data/_kb/mode_experiments.jsonl` should be able to continue the work without re-running anything.
+
+---
+
+## What we set out to do
+
+J's directive 2026-04-26 evening: "Mode runner experiment + corpus tightening."
+
+Symptom in memory before the session: scrum_review's matrix corpus was kept-rate 0/2 across every call — silent failure. Question: should we tighten the corpus, build new ones, or change retrieval?
+
+## What we built
+
+Three new corpora indexed under `/vectors/index`:
+
+| Corpus | Builder | Docs | Chunks | Source |
+|---|---|---|---|---|
+| `lakehouse_arch_v1` | `scripts/build_lakehouse_corpus.ts` | 93 | 2119 | DECISIONS.md ADRs + standalone ADRs + PHASES.md + PRD.md + CONTROL_PLANE_PRD.md + SCRUM_MASTER_SPEC.md |
+| `scrum_findings_v1` | `scripts/build_scrum_findings_corpus.ts` | 168 | 1260 | Past `scrum_reviews.jsonl` rows |
+| `lakehouse_symbols_v1` | `scripts/build_symbols_corpus.ts` | 656 | 2470 | Regex-extracted `pub fn|struct|enum|trait` + `///` docs from `crates/**/*.rs` |
+
+Multi-corpus support added to the mode runner:
+- `crates/gateway/src/v1/mode.rs` — `matrix_corpus` is now `Vec<String>` (string OR array in modes.toml/JSON via `deserialize_string_or_vec`)
+- Top-K retrieved from each corpus, merged by score, top 8 globally before relevance filter
+- Each chunk tagged with `corpus` for telemetry
+- Prompt assembly prefers `doc_id` over `source` so reviewer sees `[adr:009]` not `[lakehouse_arch]`
+
+Validation infra:
+- `scripts/mode_pass5_variance_paid.ts` — N reps × M conditions on one file, paid model
+- `scripts/mode_pass5_summarize.ts` — mean ± stddev + head-to-head wins/losses with parser handling 3 finding-table shapes (numbered, path-with-line, path-with-symbol)
+- `scripts/mode_compare.ts` — extended grouping key to `mode|corpus` (sorted+joined when multiple corpora) so multi-corpus sweeps don't last-write-wins-clobber
+
+## What we learned
+
+### Single-rep bake-off (free-tier `openai/gpt-oss-120b:free`, 3 files)
+
+Confirmed `lakehouse_arch_v1` adds +1.7 grounded findings/file vs isolation, 100% groundedness, −20s latency. **But:** matrix slightly *hurts* on small files (273-line `delta.rs`: lakehouse 7 vs isolation 9) and unlocks +9 findings on the large file (1355-line `pathway_memory.rs`).
+
+`scrum_findings_v1` produced 24% out-of-bounds line citations from cross-file line-number drift — **dangerous, excluded from defaults**. Only safe with same-file gating (TBD if needed).
+
+### Single-rep bake-off (paid `x-ai/grok-4.1-fast`, 3 files × 4 conditions)
+
+Picture *flips* on a strong model. Composed corpus −1.4 grounded vs isolation. Symbols-alone slightly negative. Arch-alone negative. Suggested kitchen-sinking enrichment denigrates results when the model is good enough to handle the file directly.
+
+### Pass 5 variance test (paid grok-4.1-fast, 5 reps × 4 conditions on `pathway_memory.rs`)
+
+| Condition | n | mean grounded ± σ | range | H2H vs isolation | Δ mean |
+|---|---|---|---|---|---|
+| **isolation** | 5 | 6.2 ± 1.3 | [5–8] | baseline | — |
+| arch_only | 5 | 5.2 ± 0.8 | [4–6] | 0W–3L–2T | −1.0 |
+| symbols_only | 5 | 6.4 ± 1.5 | [4–8] | 3W–2L–0T | +0.2 |
+| **composed (A+C)** | 5 | 4.4 ± 1.1 | [3–6] | **0W–5L–0T** | **−1.8** |
+
+**Composed loses 5/5 head-to-head against isolation on this file with this model.** Probability under random noise = 1/2⁵ = 3.1%. Statistically significant.
+
+Data window: rows in `data/_kb/mode_experiments.jsonl` where `ts > "2026-04-26T21:50:03Z"` and `file_path == "crates/vectord/src/pathway_memory.rs"`. Re-aggregate any time with `bun run scripts/mode_pass5_summarize.ts --since 2026-04-26T21:50:03Z`.
+
+## Decisions taken
+
+1. **Composed-corpus default is reverted.** `scrum_review.preferred_mode` switches from `codereview_lakehouse` → `codereview_isolation`. Matrix corpora stay defined in modes.toml but only fire when a caller explicitly forces `codereview_lakehouse` or one of the matrix-only experimental modes.
+
+2. **Model-aware enrichment downgrade (α) is wired** in `crates/gateway/src/v1/mode.rs::execute`. When a caller resolves a "strong" model AND the resolved mode is `codereview_lakehouse`, the runner downgrades to `codereview_isolation` flag-set automatically. Strong patterns: `x-ai/grok-*`, `anthropic/*`, `openai/gpt-4*`, `openai/gpt-5*`, `deepseek/deepseek-v4*`, `moonshotai/kimi-k2*`, `google/gemini-2.5*`. Override via `LH_FORCE_FULL_ENRICHMENT=1` for diagnostic runs.
+
+3. **`scrum_findings_v1` stays excluded from defaults** until same-file gating lands. Built and indexed; do not point any task class at it without that gate.
+
+## Open follow-ups (not landed in this batch)
+
+- **Same-file gating for `scrum_findings_v1`** — restrict retrieval to chunks where `file_path == focus_file` so cross-file line-number drift can't happen. Then it becomes a per-file "what was found before" signal.
+- **Variance test on small files** — pass 5 was 1 file (the largest, where matrix-hurt was sharpest). Confirm direction holds on 273-line / 333-line files. ~15 min × 2 files = ~30 min.
+- **Verify weak-model gain holds with α** — the bake-off showed matrix helps free-tier `gpt-oss-120b:free` on the large file. After α is wired, re-run on a free-tier model to confirm full enrichment still fires for it. ~5 min.
+- **Higher-signal matrix (β fork)** — if we ever want matrix back as a default, it can't be whole-ADR/whole-section chunks. Better: only retrieve chunks where the focus file's defined symbols appear. Tighter signal, fewer chunks. Postponed.
+
+## Reference data + tools
+
+- **Mode-runner code:** `crates/gateway/src/v1/mode.rs`
+- **Mode config:** `config/modes.toml`
+- **Per-call experiment log:** `data/_kb/mode_experiments.jsonl`
+- **Sweep harnesses:**
+  - `scripts/mode_experiment.ts` — files × modes × 1 rep (default model: `x-ai/grok-4.1-fast`)
+  - `scripts/mode_pass2_corpus_sweep.ts` — corpus × threshold sweep
+  - `scripts/mode_pass3_variance.ts` — temp × reps on one mode
+  - `scripts/mode_pass5_variance_paid.ts` — N reps × M conditions on one file
+- **Aggregators:**
+  - `scripts/mode_compare.ts` — full per-mode comparison with grounding check
+  - `scripts/mode_pass5_summarize.ts` — variance + head-to-head, robust to 3 table shapes
+- **Corpus builders (re-runnable when source docs / scrum_reviews / source code change):**
+  - `scripts/build_lakehouse_corpus.ts`
+  - `scripts/build_scrum_findings_corpus.ts`
+  - `scripts/build_symbols_corpus.ts`
+
+## Re-entry recipe (fresh session)
+
+```bash
+cd /home/profit/lakehouse
+git log --oneline scrum/auto-apply-19814 -10               # what's recent
+cat docs/MODE_RUNNER_TUNING_PLAN.md                        # this file
+bun run scripts/mode_pass5_summarize.ts --since 2026-04-26T21:50:03Z  # locked result
+curl -s http://localhost:3100/v1/mode/list | jq '.task_classes.scrum_review'  # current config
+```
+
+If you want to reproduce the bake-off:
+
+```bash
+# Strong model variance test (~17 min):
+bun run scripts/mode_pass5_variance_paid.ts
+
+# Weak-model regression (~10 min):
+LH_MODEL=openai/gpt-oss-120b:free LH_REPS=3 bun run scripts/mode_pass5_variance_paid.ts
+```

docs/PHASES.md

@@ -349,6 +349,24 @@
   - Per-type endpoints: `/profiles/retrieval`, `/profiles/memory`, `/profiles/observer`
   - `profile_type` field on ModelProfile
   - Guard fix: automated scrumaudit.py finds real issues
+- [x] **Phase 42: Truth Layer** (2026-04-27 closure verified)
+  - `crates/truth/{lib,staffing,devops,loader}.rs`
+  - Staffing rules populated; devops scaffold by design
+  - `/v1/context` serves task_classes + rules; 37 tests green
+- [x] **Phase 43: Validation Pipeline** (2026-04-27)
+  - `crates/validator/` real validators + WorkerLookup + ParquetWorkerLookup
+  - 500K-row workers_500k.parquet loaded at gateway boot
+  - `POST /v1/validate` + `POST /v1/iterate` (the 0→85% loop)
+  - 33 validator tests green
+- [x] **Phase 44: Caller Migration** (2026-04-27)
+  - TS callers + aibridge::AiClient::new_with_gateway opt-in
+  - Vectord routed through /v1/chat for autotune + RAG
+  - scripts/check_phase44_callers.sh CI guard
+- [x] **Phase 45: Doc-Drift Detection** (2026-04-27)
+  - DocRef + doc_drift module + context7 bridge
+  - /doc_drift/check + /scan + /resolve endpoints
+  - data/_kb/doc_drift_corrections.jsonl writes
+  - boost exclusion of unreviewed drift-flagged entries
 - [ ] Fine-tuned domain models (Phase 25+)
 - [ ] Multi-node query distribution (only if ceilings bite)
 

docs/PHASE_1_6_BIPA_GATES.md

@@ -0,0 +1,334 @@
+# Phase 1.6 — BIPA Pre-Launch Gates
+
+**Status:** Draft — 2026-05-03 · **Owner:** J + outside counsel · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md), [`IDENTITY_SERVICE_DESIGN.md`](IDENTITY_SERVICE_DESIGN.md)
+
+> **Why this exists.** `IDENTITY_SERVICE_DESIGN.md` v3 §5 Step 0 names Phase 1.6 as a HARD PREREQUISITE: identityd backfill cannot start until Phase 1.6 ships. This doc specifies what Phase 1.6 contains.
+>
+> **Scope.** BIPA (740 ILCS 14) compliance gates that must be in place BEFORE the system accepts a single real candidate photo. Synthetic-data face pool can keep operating; real-photo intake CANNOT begin without these gates.
+>
+> **Authority.** This is an engineering scaffold. Sections marked `⚖ COUNSEL` need outside counsel to author the actual legally-binding text. Engineering ships the procedural gates; counsel writes the words.
+
+---
+
+## 1. The five BIPA pre-launch gates
+
+Each gate is a deliverable that must ship before real-photo intake. None is optional. Order shown is the recommended ship sequence.
+
+### Gate 1 — Public retention schedule (BIPA §15(a))
+
+**Required:** A publicly-available, written retention schedule for biometric identifiers and information.
+
+**What ships:**
+- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file
+- Linked from public privacy policy at the deployment URL
+- Specifies:
+  - Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`)
+  - Purpose of collection (identity matching for staffing operations)
+  - Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin)
+  - Destruction procedure: per Gate 5 below
+- Versioned (this is v1; future updates supersede with a new version)
+
+**⚖ COUNSEL** — write the actual schedule. Engineering provides the operational facts; counsel writes the binding language.
+
+**Engineering acceptance:** the file is committed, the public URL renders it, and identityd's `consent_versions` table references it by hash.
+
+---
+
+### Gate 2 — Informed written consent (BIPA §15(b))
+
+**Required:** Informed, written consent BEFORE any biometric collection occurs.
+
+**What ships:**
+- `docs/policies/consent/biometric_consent_template_v1.md` — public consent template
+- Versioned, hashed, referenced from identityd's `consent_versions` table
+- Must disclose, per BIPA §15(b)(1)-(3):
+  1. That biometric identifiers/information will be collected
+  2. The specific purpose for collection (and the length of term — references Gate 1)
+  3. Receipt of a written release authorizing collection
+- Consent flow at intake:
+  - Candidate sees the disclosure on a UI surface (web form / paper / digital signature)
+  - Candidate provides explicit affirmative action (signature, click-acceptance with timestamp, etc.)
+  - Identityd records `biometric_consent_status='given'` with `consent_version` reference + `consent_given_at` timestamp
+  - **Without identityd recording 'given', no biometric data flows through deepface.**
+
+**⚖ COUNSEL** — write the consent template. Recommended content (engineering view):
+- Clear language (not just legal boilerplate)
+- Specific to facial-classification (not generic biometrics)
+- Includes withdrawal procedure
+- Includes data-subject rights enumeration
+
+**Engineering acceptance:** consent gate is enforced in code at the photo-upload endpoint; identityd refuses biometric writes when `biometric_consent_status != 'given'`; pre-existing synthetic-face pool is exempt (no consent needed because no real subject).
+
+---
+
+### Gate 3 — Photo-upload endpoint with consent enforcement
+
+**Required:** Code-level enforcement that real-photo intake checks consent before processing.
+
+**What ships:**
+
+An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior:
+
+1. Caller authenticates with service-tier token
+2. Endpoint queries identityd for `subjects.biometric_consent_status`
+3. If status ≠ `'given'` → HTTP 403 with reason `"BIPA consent required before biometric processing"`
+4. If status = `'given'`:
+   a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`)
+   b. deepface tagging runs against the photo
+   c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1.
+   d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule
+   e. `pii_access_log` row written with `purpose_token='biometric_collection'`
+5. Response: `{candidate_id, retention_until, consent_version}`
+
+**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):**
+
+The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/<id>.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option<JSON>`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type.
+
+```rust
+// crates/catalogd/src/subject_manifest.rs (paraphrased)
+pub struct BiometricCollection {
+    pub data_path: String,                // quarantined path
+    pub template_hash: String,            // SHA-256 of original bytes (integrity, NOT re-derivation)
+    pub collected_at: DateTime<Utc>,
+    pub classifications: Option<Value>,   // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md)
+}
+```
+
+**Engineering acceptance:**
+- Endpoint refuses uploads when consent missing (verified by integration test)
+- deepface output never lands in the synthetic-face manifest (`data/headshots/manifest.jsonl`)
+- Real-photo classifications are isolated to identityd `subjects` table — never flow to JSONL sinks
+- The `/headshots/:key` route in mcp-server REMAINS synthetic-only — does NOT serve real candidate photos to LLMs without an explicit allowance (proposed: real photos served only to authenticated staffer UI, never to model context)
+
+---
+
+### Gate 4 — Deprecate name → ethnicity inference
+
+**Required:** The hard-coded `NAMES_HISPANIC` / `SURNAMES_*` lookup tables in `mcp-server/search.html:3375-3432` (per Phase 1.5 §1B walk) get removed.
+
+**What ships:**
+- A code commit that removes:
+  - `FEMALE_NAMES`, `MALE_NAMES` constants
+  - `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` constants
+  - `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` constants
+  - The `genderFor()` and `guessEthnicityFromFirstName()` functions
+  - All call sites that consumed these (face-pool bucket selection)
+- Replacement strategy:
+  - For SYNTHETIC face pool routing: deterministic hash of candidate_id selects a face bucket, no demographic inference
+  - For REAL candidate photos: the candidate's actual photo IS the representation; no inference needed
+
+**Why this is BIPA + Title VII risk separately:** name-based ethnicity classification is BOTH a discriminatory feature engineering practice (Title VII) AND, when combined with photo-based attribute extraction, a "biometric information derived from a biometric identifier" pattern (BIPA broad reading). Removing the lookup tables forecloses both arguments.
+
+**Engineering acceptance:**
+- Lookup tables removed from search.html
+- Unit test asserts no protected-attribute inference functions exist in search.html or any mcp-server module
+- Face-pool routing for synthetic faces uses candidate_id hash exclusively
+- Phase 1.5 §1B finding closed
+
+---
+
+### Gate 5 — Documented destruction procedure
+
+**Required:** A written procedure for biometric data destruction at retention expiry OR consent withdrawal OR right-to-be-forgotten request.
+
+**What ships:**
+- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing
+- Specifies:
+  - Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request
+  - Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly
+  - Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads/<id>/*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails
+  - Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed
+  - Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction)
+  - Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request
+
+**⚖ COUNSEL** — review the runbook for legal sufficiency. Engineering writes the procedure; counsel attests that the procedure satisfies BIPA §15(a) destruction requirements.
+
+**Engineering acceptance:**
+- Runbook committed
+- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full)
+- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock)
+- Erasure events are logged with cryptographic attestation
+
+---
+
+## 2. Cryptographic attestation: no biometric data exists pre-identityd
+
+**Per `IDENTITY_SERVICE_DESIGN` v3-B11.** Plaintiffs may argue that the EXISTENCE of biometric schema fields constitutes constructive notice of intent to collect biometric data — therefore consent should have preceded the schema. The defense: prove that no biometric data was actually collected from real candidates before identityd + the consent gate.
+
+**What ships:**
+- A one-shot script `scripts/staffing/attest_pre_identityd_biometric_state.sh` that:
+  - Queries `data/datasets/workers_500k.parquet` schema and confirms NO column named `photo`, `biometric_*`, `face_*`, `image_*` exists
+  - Greps `data/_kb/*.jsonl` and `data/_pathway_memory/state.json` for any base64-encoded image bytes (deepface output, photo blobs)
+  - Verifies `data/headshots/manifest.jsonl` rows ≤ synthetic face pool size
+  - Hashes the schema + summary; commits the hash to S3 Object Lock (per identity service v3 anchor pattern)
+- Attestation document `docs/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-XX.md` signed by J + outside counsel
+
+**This is a one-time defense artifact.** It establishes the baseline: "as of this date, no biometric data was collected from real candidates."
+
+---
+
+## 3. Employee training acknowledgment (general BIPA hygiene)
+
+**Required:** People with access to biometric data acknowledge BIPA-handling training.
+
+**What ships:**
+- `docs/policies/BIPA_HANDLING_TRAINING_v1.md` — training material covering:
+  - What constitutes biometric identifiers / information
+  - The consent + retention procedures
+  - Destruction obligations
+  - Reporting suspected exposure
+- Acknowledgment record per individual (initially: J + counsel + named operators)
+- Annual refresh
+
+**⚖ COUNSEL** — write training content. Engineering doesn't author legal-compliance training.
+
+---
+
+## 4. Phase 1.6 exit criteria (gates Phase 2 backfill)
+
+All 5 gates must be DONE before identityd backfill begins. Status as
+of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code:
+
+| # | Gate | Engineering | Counsel | Status |
+|---|---|---|---|---|
+| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** |
+| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** |
+| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option<JSON> = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** |
+| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** |
+| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** |
+
+PLUS:
+
+| # | Item | Engineering | Counsel | Status |
+|---|---|---|---|---|
+| 6 | Cryptographic attestation pre-identityd | DONE — `scripts/staffing/attest_pre_identityd_biometric_state.sh` + `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (3/3 evidence checks pass; signature lines pending) | pending signature | **eng-DONE, signature-pending** |
+| 7 | Employee training material | scaffold deferred — Gate 5 runbook §7 acknowledgment may serve as substrate | pending | **deferred** |
+
+**Blocking set for Phase 2 backfill:** items **1, 2, 3, 4, 5, 6** must
+all be DONE. Item 7 (employee training) is reduced from blocking to
+"deferred" because the Gate 5 destruction runbook §7 already requires
+operator acknowledgment before legal-tier credentials are issued —
+that acknowledgment is procedurally equivalent to the training-record
+requirement when the operator population is small (J + 1-2 named
+operators). If the operator population grows beyond that, item 7
+re-promotes to blocking and a separate training program must be authored.
+
+⚖ COUNSEL — confirm whether item 7 deferral is acceptable for the
+expected operator population size, or restore it to the blocking set.
+
+**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel
+review of the engineering scaffolds. Gate 3 substrate is fully
+shipped; Gate 3b deepface classification was DECIDED on 2026-05-05
+as Option C (defer) — `BiometricCollection.classifications` stays
+`None` in v1, consent + retention docs revised to match this
+narrower scope. If a future product requirement surfaces a real
+need for classifications, the substrate is forward-compatible
+(`Option<JSON>`) and either Option A (~1 day) or Option B (~5 days)
+of the design doc can be picked up then under a v2 consent template.
+
+---
+
+## 5. Effort estimate
+
+| Gate | Engineering effort | Legal effort |
+|---|---|---|
+| Gate 1 (retention schedule) | 0.5 day | counsel-dependent (typically 1-2 weeks for review) |
+| Gate 2 (consent template) | 0.5 day | counsel-dependent (typically 2-4 weeks for review and consent UX design) |
+| Gate 3 (photo-upload endpoint) | 1-2 days | review of endpoint behavior |
+| Gate 4 (deprecate name-ethnicity inference) | 0.5 day | none (engineering-only fix) |
+| Gate 5 (destruction runbook) | 1 day | counsel sign-off |
+| §2 cryptographic attestation | 0.5 day | counsel + J signature |
+| §3 employee training | 0.25 day (admin) | counsel-authored content |
+| **Total engineering** | **~4-5 days** | — |
+| **Total counsel** | — | **~3-6 weeks calendar** (review cycles) |
+
+**The calendar bottleneck is counsel, not engineering.** Engineering can stage all 5 gates ready-to-ship in a week. Counsel sign-off + consent UX rollout is the longer pole.
+
+---
+
+## 6. Open questions for J + counsel
+
+1. **Photo-upload UX:** is there an existing intake form / staffer console where photo upload would happen? Or is this new UI work?
+2. **Consent collection mechanism:** electronic signature service (DocuSign, Adobe Sign), in-app click-acceptance, paper form? Each has different evidentiary weight in litigation.
+3. **Operator list with biometric access:** who, today, would be on the named-operators list for §3 training?
+4. **Counsel for sign-off:** named outside counsel — same or different from the dual-control legal-token party in identity service?
+5. **Public privacy policy URL:** does one exist? If yes, where; if no, that's a separate Gate-1.5 deliverable.
+
+---
+
+## 6.5. Post-signature deployment runbook
+
+When counsel returns the countersigned consent template + retention
+schedule, the engineering side of "flip from permissive to strict
+mode" is one command:
+
+```bash
+# 1. Counsel commits their signature to §7 of the consent template
+#    markdown (or J commits the signed PDF + updates §7 with counsel's
+#    name + date). The markdown is the BINDING TEXT — the PDF is just
+#    a rendering of it.
+
+# 2. Hash the canonical signed text + seed the gateway allowlist.
+./scripts/staffing/seed_consent_version.sh \
+    docs/policies/consent/biometric_consent_template_v1.md \
+    --label "v1 signed YYYY-MM-DD by [counsel name]"
+
+# The script:
+#   - computes SHA-256 of the markdown (binding text)
+#   - atomically writes /etc/lakehouse/consent_versions.json with
+#     the new hash + per-seed audit metadata (timestamp, label,
+#     source path)
+#   - restarts lakehouse.service so the gateway re-reads the
+#     allowlist
+#   - probes /biometric/health for clean restart
+
+# 3. Verify strict mode is rejecting unknown hashes:
+TOKEN=$(cat /etc/lakehouse/legal_audit.token)
+curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \
+    -H "X-Lakehouse-Legal-Token: $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"consent_version_hash":"deadbeefdeadbeef000000000000000000000000000000000000000000000000","consent_collection_method":"electronic_signature","operator_of_record":"strict_mode_probe"}'
+# Expect: HTTP 400 + {"error":"consent_version_unknown", ...}
+```
+
+After this, the gateway is in counsel-tier strict mode:
+
+- Any consent grant POST whose `consent_version_hash` doesn't match
+  a known signed template is refused at intake
+- Operator typos (mistyped hash) become loud failures, not silent
+  bad records
+- Future template revisions (v2, v3, ...) require counsel re-sign
+  AND a new `seed_consent_version.sh` run before being accepted —
+  the v1 hash stays in the allowlist for already-collected subjects'
+  audit-trail compatibility
+
+### Pre-counsel demo seed
+
+For deployments that want to exercise strict mode BEFORE counsel
+signs, the same script works against the eng-staged template:
+
+```bash
+./scripts/staffing/seed_consent_version.sh \
+    docs/policies/consent/biometric_consent_template_v1.md \
+    --label "eng-staged demo seed (NOT counsel-signed)"
+```
+
+The hash entry should be replaced (rotate the demo hash out, add
+the counsel-signed hash) when counsel completes review. The
+allowlist's `_meta.seeded_at[]` array preserves the seed history.
+
+---
+
+## 7. What this PRD is NOT
+
+- Not legal advice. The `⚖ COUNSEL` markers exist because the binding text needs lawyers, not engineers.
+- Not a substitute for a DPIA / PIA. Phase 1.6 satisfies BIPA-specific gates; a Data Protection Impact Assessment is broader and may be required separately.
+- Not a SOC2 Type II deliverable. SOC2 is a parallel work stream.
+- Not the only gate before production. The full 9-phase audit-trail program continues; Phase 1.6 specifically unblocks Phase 2 (identity service implementation).
+
+---
+
+## Change log
+
+- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup.
+- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill.

docs/PHASE_AUDIT_GUIDE.md

@@ -0,0 +1,107 @@
+# Phase Audit Guidance for Claude Code
+
+## Purpose
+This document provides the proper workflow for auditing completed phases in the Lakehouse project.
+
+## ⚠️ Important: Do NOT Skip Steps
+Each phase requires BOTH:
+1. PRD spec verification (check code exists)
+2. Full SCRUM execution (6 commands)
+
+## Proper Phase Audit Workflow
+
+### Step 1: Read PRD Specification
+For each phase, read the PRD to understand what's supposed to ship:
+```bash
+# Read from docs/PRD.md or docs/PHASES.md
+cat docs/PHASES.md | grep -A20 "Phase N:"
+```
+
+### Step 2: Verify Code Exists
+Check that each deliverable from the PRD spec has corresponding code:
+```bash
+# Example - check for specific implementations
+grep -r "function_name" crates/*/src/
+ls crates/*/src/*.rs
+```
+
+### Step 3: Run Full SCRUM (6 Commands)
+In order, execute ALL of these for the phase's crates:
+
+```bash
+# 1. Build
+cargo build -p <crate-name>
+
+# 2. Test  
+cargo test -p <crate-name>
+
+# 3. Clippy (if installed)
+cargo clippy -p <crate-name> -- -D warnings
+
+# 4. Format check
+cargo fmt -p <crate-name> -- --check
+
+# 5. Cargo check
+cargo check -p <crate-name>
+
+# 6. Doc check
+cargo doc -p <crate-name> --no-deps
+```
+
+### Step 4: Fix Issues
+If any SCRUM command fails:
+- Fix the code
+- Re-run the failing command
+- Re-run ALL 6 commands to verify
+
+### Step 5: Update Phase Documentation
+Only mark as ✅ after ALL 6 SCRUM commands pass:
+```markdown
+## Phase N: [Name] ✅
+- [x] spec item 1
+- [x] spec item 2
+  - SCRUM: build ✅ test ✅ clippy ✅ fmt ✅ check ✅ doc ✅
+```
+
+## Current Phase Status
+
+| Phase | Status | Notes |
+|-------|--------|-------|
+| 0 | ✅ | Bootstrap complete |
+| 1 | ✅ | Storage + Catalog |
+| 2 | ✅ | Query Engine |
+| 3 | ✅ | AI Integration |
+| 4 | ✅ | Frontend |
+| 5 | ✅ | Hardening |
+| 6-42 | ✅ | See docs/PHASES.md |
+
+## Notes from Previous Session
+
+- Clippy and rustfmt are NOT installed on this system
+- Run `rustup component add clippy rustfmt` to install
+- Some crates have 0 unit tests (expected for service crates)
+- 28 warnings remain in unused code paths (ui/vectord)
+
+## Key Files
+
+- `docs/PHASES.md` - Phase tracker with checkboxes
+- `docs/PRD.md` - Full product requirements
+- `docs/CONTROL_PLANE_PRD.md` - Phases 38+ specifications
+- `crates/*/` - All crate implementations
+
+## Quick Reference
+
+```bash
+# Full workspace SCRUM
+cargo build --workspace
+cargo test --workspace
+# (clippy if installed)
+cargo fmt -- --check
+cargo check --workspace
+cargo doc --no-deps
+
+# Per-crate
+cargo build -p <crate>
+cargo test -p <crate>
+cargo check -p <crate>
+```