auditor: repoint Gitea client at git.warren.help — agentview host is dead

git.agentview.dev has served a Traefik default cert + API 404 since
Aug 25 (VPS consolidation moved repos to warren); listOpenPrs failed
every 90s cycle since. Host default moves to git.warren.help and auth
now resolves per-host from ~/.git-credentials with fallback to the
origin remote's embedded credentials, sent as Basic (works for PAT or
password). Verified live: /api/v1/user auths as profit, pulls list 200s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -4,3 +4,74 @@
 .env
 __pycache__/
 *.pyc
+
+# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
+# (synthetic StyleGAN, ~580MB for 1000 faces). Manifest + fetch script are tracked.
+data/headshots/face_*.jpg
+data/headshots/_thumbs/
+# ComfyUI on-demand generated portraits (per-worker unique). Cached on first
+# request; fully regeneratable via /headshots/generate/:key.
+data/headshots_gen/
+
+# Runtime data — all regeneratable from inputs or accumulated by daemons.
+# Anything under data/_<name>/ is internal state (auditor outputs, KB caches,
+# pathway memory snapshots, HNSW trial results, etc.). Anything under
+# data/datasets/ or data/vectors/ is generated by ingest/index pipelines.
+data/_*/
+data/lance/
+data/datasets/
+data/vectors/
+data/demo/
+data/evidence/
+data/face_test/
+data/headshots_role_pool/
+data/icons_pool/
+data/scored-runs/
+data/workspaces/
+data/catalog/
+data/**/*.bak-*
+data/**/*.pre-*-bak
+
+# Logs
+logs/
+
+# Build artifacts
+node_modules/
+exports/
+mcp-server/data/
+
+# Per-run distillation reports (timestamp-named); keep the parent dir tracked
+# via .gitkeep if needed but don't carry every batch's report set.
+reports/distillation/[0-9]*/
+reports/distillation/*-*-*-*-*/
+
+# Test scratch — scratchpads, traces, sessions are regenerated each run.
+# PRD/scenario fixtures stay tracked (they ARE the test).
+tests/agent_test/_*
+tests/agent_test/sessions/
+tests/real-world/runs/
+
+# BIPA-quarantined photo uploads — Phase 1.6 Gate 3a writes to
+# data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext> with mode 0700/0600.
+# This is regulated subject-of-record data and must NEVER enter git.
+data/biometric/
+
+# Local-only scrum review evidence. Per `feedback_audit_findings_log.md`
+# scrum runs fold fixes into a batch commit; the verdict / disposition
+# files stay local for forensics.
+reports/scrum/
+
+# Per-event biometric verification reports (timestamp-named, regenerated
+# per `verify_biometric_erasure.sh` invocation). Source-of-truth is the
+# audit chain itself; these reports are derived views.
+reports/biometric/
+
+# Counsel transmission tarballs + manifests are regenerated by
+# `bundle_counsel_packet.sh` from the tracked `docs/counsel/` source.
+# The bundle is transmittable, not source-of-truth.
+reports/counsel/
+
+# Local experiments scratchpad — per the "Test code in main is ACTIVELY
+# being cleaned out" policy (commits 6aafd41 + f4ebd22), one-off
+# experiments stay out of the tracked tree.
+experiments/

Cargo.lock

@@ -48,6 +48,7 @@ version = "0.1.0"
 dependencies = [
  "async-trait",
  "axum",
+ "lru",
  "reqwest",
  "serde",
  "serde_json",
@@ -1364,10 +1365,13 @@ dependencies = [
  "axum",
  "bytes",
  "chrono",
+ "hmac",
  "object_store",
+ "parquet 55.2.0",
  "proto",
  "serde",
  "serde_json",
+ "sha2",
  "shared",
  "storaged",
  "tokio",
@@ -4093,6 +4097,7 @@ dependencies = [
  "tracing-opentelemetry",
  "tracing-subscriber",
  "truth",
+ "validator",
  "vectord",
 ]
 
@@ -8912,6 +8917,8 @@ dependencies = [
 name = "validator"
 version = "0.1.0"
 dependencies = [
+ "arrow 55.2.0",
+ "parquet 55.2.0",
  "serde",
  "serde_json",
  "thiserror 2.0.18",

Cargo.toml

@@ -36,6 +36,7 @@ datafusion = "47"
 bytes = "1"
 futures = "0.3"
 sha2 = "0.10"
+hmac = "0.12"
 url = "2"
 tonic = "0.13"
 prost = "0.13"

STATE_OF_PLAY.md

@@ -0,0 +1,492 @@
+# STATE OF PLAY — Lakehouse
+
+**Last verified:** 2026-06-30 (search index-alignment fix live on :3700) · biometric substrate 2026-05-05
+**Verified by (2026-06-30):** live probe after mcp-server restart — `/match` + `/search` with `state='IL'` return only IL workers (new PID on :3700). Not memory.
+**Verified by (2026-05-05):** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory.
+
+> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
+
+---
+
+## WHAT LANDED 2026-06-30 (search index-alignment sweep + P1-already-done finding)
+
+**Fixed a systemic hybrid-search bug.** Every `/vectors/hybrid` call that paired the
+`ethereal_workers` SQL dataset (10K rows) with the `workers_500k_v1` vector index (50K,
+a *different* worker set) returned wrong-entity results: same `worker_id` = different
+person in each index, so a `state='IL'` query surfaced a **KY** worker flagged
+`sql_verified:true`. Root cause: the default index didn't share an id-space with the
+default dataset.
+
+Swept **8 call sites** in `mcp-server/index.ts` to `ethereal_workers_v1` (10K, same
+id-space as the SQL table). Aligned sites (`filter_dataset: workers_500k`) and pure-RAG
+sites (`/vectors/rag`) were correctly left alone.
+
+| Commit | Sites |
+|---|---|
+| `423400e` | `/search` HTTP handler |
+| `35a80a7` | `search_workers` tool · `match_contract` tool · `/match` HTTP handler |
+| `1570306` | 4 intelligence routes (similar-workers, role-discovery, smart_search, scenario-find) |
+
+**Verified live** after an mcp-server restart (new PID on :3700): `/match` and `/search`
+with `state='IL'` return only IL workers (Champaign/Springfield/Urbana); `state='WI'`
+returns WI. Grep confirms no `workers_500k_v1` index remains paired with `ethereal_workers`.
+Server boots clean. Only `mcp-server/index.ts` was touched.
+
+**Finding — demo-plan P1 was already done.** The "search box that actually filters" item
+in this doc's CURRENT PLAN (P1, from the 2026-04-27 demo audit) is **stale**. The UI now
+posts structured `state`/`role` to the evolved `/intelligence/chat`, which returns real
+`sql_results` (IL Forklift → 167 matches, correct workers; WI → WI). The P1 section below
+describes a bug that no longer exists — don't re-implement it.
+
+**⚠ Uncommitted WIP left untouched (pre-existing, not from this session).**
+`mcp-server/search.html` + `proof.html` (edited 2026-05-30; proof.html restructured into
+chapters) and a `biometric_endpoint.rs` comment cleanup remain uncommitted in the working
+tree. Also untracked and newer than this doc: `RECOVERY_PLAN_2026-05-05.md`,
+`mcp-server/UX_FIX_PLAN.md`, `docs/scope-sentinel/`. Someone should reconcile these — they
+imply planning context this file doesn't yet capture.
+
+---
+
+## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard)
+
+After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`.
+
+| Commit | What | Verified |
+|---|---|---|
+| `76cb5ac` | `POST /biometric/subject/{id}/consent` — Gate 2 backend. Flips status NeverCollected→Given, computes retention_until = given_at + 18mo (per retention schedule v1 §4), captures consent_version_hash + collection_method (closed set: electronic_signature/paper/click_acceptance) + operator + evidence path to audit row. State machine: 409 already_given, 409 post-withdrawal-requires-erase, 403 subject_inactive. | 12 unit tests; live POST returns 401/400/404 on guards |
+| `7f0f500` | Candidate intake UI at `/biometric/intake?candidate_id=XXX` (mcp-server :3700, exposed via nginx at devop.live/lakehouse/biometric/intake). 4-screen flow: operator auth → consent template render + click-accept → photo capture (file or webcam getUserMedia) → confirmation showing audit hmac. SHA-256 of rendered consent block becomes consent_version_hash. sessionStorage-backed token (clears on tab close), neo-brutalist style matching onboard.html. | 22631-byte HTML, mcp-server route returns 200 |
+| `68d226c` | Three things in one wave: <br>(a) `POST /biometric/subject/{id}/withdraw` — BIPA right of withdrawal. Flips Given→Withdrawn, accelerates retention_until from 18mo→30d (per consent template v1 §2 SLA), audit kind=biometric_consent_withdrawal. State machine 409s on NeverCollected/Pending (nothing_to_withdraw), Withdrawn (already_withdrawn), Expired (already_expired). 12 unit tests. <br>(b) Withdraw UI at `/biometric/withdraw` — 3-screen operator flow (token+name auth → reason+evidence form → confirmation showing 30-day clock + verify curl recipe). <br>(c) `lakehouse-retention-sweep.{service,timer}` systemd units in `ops/systemd/`. Daily 03:00 UTC, Persistent=true, install.sh updated to handle paired timer+oneshot service. <br>Plus operator_of_record bug fix in intake UI (was hardcoded `'intake_ui_operator'`). | 46/46 biometric_endpoint + 71/71 catalogd lib tests; manual sweep run: 100 subjects, 0 overdue, exit 0 |
+| **(current HEAD post-this-wave)** | Stats endpoint `GET /biometric/stats` (legal-tier auth, returns subject counts by status + photo count + oldest active retention + last 20 state-change events with anonymizable trace_ids) + ops dashboard at `/biometric/dashboard` (single-page, polls /stats every 15s, table + status tiles, XSS-safe DOM construction not innerHTML). Plus consent_versions allowlist: `BiometricEndpointState.allowed_consent_versions: Option<Arc<HashSet<String>>>`, loaded from `/etc/lakehouse/consent_versions.json` (`LH_CONSENT_VERSIONS_FILE` override), missing-file = permissive (v1 compat), present + populated = strict mode (refuses unknown hashes with 400 consent_version_unknown). Plus `scripts/staffing/subject_timeline.sh` — pretty-prints any subject's full BIPA lifecycle from /audit/subject/{id} (manifest + on-disk photo + chronological audit chain + verification status). | 5 new allowlist unit tests + 4 stats tests; live demo on WORKER-100 ran end-to-end (consent → photo → withdraw, chain verified=true, chain_root=a47563ff…) |
+
+### Live demo evidence (WORKER-100, 2026-05-05 20:12 UTC)
+The full lifecycle was exercised against the live gateway as a verification artifact. The audit chain on WORKER-100 now contains 3 rows:
+
+```
+20:12:33.054  BIOMETRIC_CONSENT_GRANT       result=given       hmac=9c6f4153341e97d2…  trace=live-demo-2026-05-05
+20:12:47.957  BIOMETRIC_COLLECTION          result=success     hmac=856be6173c88277c…  trace=live-demo-2026-05-05
+20:15:27.298  BIOMETRIC_CONSENT_WITHDRAWAL  result=withdrawn   hmac=a47563ff937d50de…  trace=live-demo-2026-05-05
+```
+
+`chain_verified=true`, chain_root = a47563ff937d50de43b09a0c903cff954233836c219a928ee8ca2aa6792272dd. Photo file at `data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg` (180 bytes — a minimal real JFIF JPEG), retention_until=2026-06-04 (= 30 days from withdrawal). Retention sweep will flag this subject on or after that date; operator runs `/biometric/subject/WORKER-100/erase` to destroy.
+
+To re-verify: `./scripts/staffing/subject_timeline.sh WORKER-100`.
+
+### Endpoint matrix (v1 BIPA lifecycle complete)
+
+| Event | Endpoint | Method | Auth | Status flip | retention_until |
+|---|---|---|---|---|---|
+| consent given | `/biometric/subject/{id}/consent` | POST | legal | NeverCollected/Pending → Given | now + 18mo |
+| photo collected | `/biometric/subject/{id}/photo` | POST | legal + consent gate | (no change) | (no change) |
+| consent withdrawn | `/biometric/subject/{id}/withdraw` | POST | legal | Given → Withdrawn | now + 30d |
+| destruction | `/biometric/subject/{id}/erase` | POST | legal | (manifest cleared) | n/a |
+| audit read | `/audit/subject/{id}` | GET | legal | (read-only) | (read-only) |
+| ops aggregates | `/biometric/stats` | GET | legal | (read-only) | (read-only) |
+
+UIs:
+- `/biometric/intake?candidate_id=X` — operator-driven consent + photo
+- `/biometric/withdraw` — operator-driven withdrawal recording
+- `/biometric/dashboard` — read-only ops aggregate, auto-refresh
+
+CLI tools:
+- `scripts/staffing/verify_biometric_erasure.sh <id>` — post-erasure verification
+- `scripts/staffing/biometric_destruction_report.sh --month YYYY-MM` — anonymized monthly report
+- `scripts/staffing/subject_timeline.sh <id>` — full lifecycle pretty-print (NEW 2026-05-05)
+- `scripts/staffing/bundle_counsel_packet.sh` — counsel review tarball
+- `scripts/staffing/attest_pre_identityd_biometric_state.sh` — defense attestation generator
+
+### What's blocking production cutover NOW
+**Counsel calendar.** Engineering substrate is done end-to-end: every state transition has a defensible endpoint, every endpoint has tests + live verification, every UI is reachable, retention sweep is scheduled, allowlist hardening is wired. The remaining work is signature/review:
+1. Counsel review of consent template v1 (revised for Option C — classifications deferred)
+2. Counsel review of retention schedule v1 (revised for Option C)
+3. Counsel review of destruction runbook
+4. Counsel + J signatures on §2 attestation
+5. Once counsel signs the consent template, populate `/etc/lakehouse/consent_versions.json` with the signed hash to flip the gateway from permissive to strict mode
+
+Counsel-review packet at `reports/counsel/counsel_packet_2026-05-05.tar.gz` (regenerable via `bundle_counsel_packet.sh` to pick up the latest doc state).
+
+---
+
+## WHAT LANDED 2026-05-05 (morning wave — doc reconciliation + Gate 3b decision + counsel packet)
+
+This was a **doc-only wave**, not code. Background: J asked for an audit of the BIPA/biometric documentation before production cutover. Audit found moderate fragmentation between docs and shipped code (post-`identityd` collapse, post-Gate-3a-ship, pre-Gate-3b-decision). Closed it in one pass.
+
+| Item | What changed | Status |
+|---|---|---|
+| **Gate 3b — DECIDED: Option C (defer classifications)** | `BiometricCollection.classifications` stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status flipped from "draft / awaits product" to "DECIDED 2026-05-05". | Locked |
+| **Endpoint-path drift** | `PHASE_1_6_BIPA_GATES.md` (3 spots), `BIPA_DESTRUCTION_RUNBOOK.md` (2 spots), `biometric_retention_schedule_v1.md` (1 spot) updated from legacy `/v1/identity/subjects/*` (proposed under separate identityd daemon, never shipped) to actual `/biometric/subject/*` (catalogd-local, shipped `848a458`). Schema block in `PHASE_1_6_BIPA_GATES.md` rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate (not the proposed Postgres `subjects` table). | Reconciled |
+| **Consent template + retention schedule** | Both revised for Option C: removed all "automated facial-classification" / "deepface" language so disclosed scope matches implemented scope. Pending counsel review — they were already eng-staged with ⚖ markers. | Eng-staged for counsel |
+| **`scripts/staffing/verify_biometric_erasure.sh`** (NEW) | Operator-side verification of an erasure event. Curls `/audit/subject/{id}` with legal-tier token, checks: manifest.biometric_collection null, uploads dir empty, last audit row is `biometric_erasure`/`full_erasure` with `erased`/`success`, chain_verified=true. Writes a hashed report to `reports/biometric/`. | Smoke-tested live |
+| **`scripts/staffing/biometric_destruction_report.sh`** (NEW) | Monthly destruction-event aggregation. Anonymizes candidate IDs (sha256-12 prefix), counts by scope + trigger, flags anomalies. Smoke-test on May 2026 data found 1 historical `biometric_erasure`/`consent_withdrawal` event (test fixture). | Smoke-tested live |
+| **`docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`** (NEW) | Captures the rotation procedure operationalized after the 2026-05-05 `/tmp` wipe incident. Covers: when to rotate, pre-rotation snapshot, atomic-swap procedure, post-rotation verification (incl. expected pre-rotation chain tamper-detect under new key), recovery from lost key, ⚖ counsel notes. | Authored |
+| **`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`** + `bundle_counsel_packet.sh` (NEW) | Cover note bundling all eng-staged BIPA docs for counsel review with per-doc questions, sign-off checklist, recommended review sequence. Bundler script tarballs the 8 referenced files + emits a SHA-256 manifest. Tarball ready for transmission: `reports/counsel/counsel_packet_2026-05-05.tar.gz`. | Bundled, ready to send |
+
+### Eng follow-up that this wave surfaced
+- **Double-upload file leak — DIAGNOSED + FIXED** (2026-05-05 same wave). `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo file. Investigation showed:
+  - The file was 13 bytes of test fixture (`ff d8 ff d9 + ASCII "TESTBYTES"`), byte-identical to the unit-test fixture at `biometric_endpoint.rs:841`. NO PII, NO biometric content, NO synthetic-face content. Came from manual integration testing on 2026-05-03.
+  - Audit log timeline showed two consecutive uploads (09:54, 10:04) followed by one erasure (10:22). The erasure unlinked only the SECOND file (which the manifest pointed at by then); the first file was orphaned because the second upload had silently overwritten `manifest.data_path`.
+  - **Real bug found**: the upload handler did NOT refuse a second upload to a subject with `biometric_collection.is_some()`. Patched `process_upload` to return HTTP 409 + `error: "biometric_already_collected"` when a re-upload is attempted; operator must explicitly POST `/biometric/subject/{id}/erase` first.
+  - Stranded test file removed (`rm` of the 13-byte fixture).
+  - New unit test `second_upload_without_erase_returns_409` asserts the 409 + that the first photo's data_path remains unchanged + that the first file remains untouched on disk.
+  - Existing `repeated_uploads_grow_the_chain` replaced with `upload_erase_upload_grows_the_chain_cleanly` (covers the legitimate re-collection cycle: upload → erase → upload, chain grows to 3 rows).
+  - Existing `content_type_with_parameters_accepted` test updated to use two distinct subjects (it had used one subject for two uploads to test content-type parsing — now would 409).
+  - **22 biometric_endpoint tests + 59 catalogd lib tests all green** post-patch (was 21+58 pre-patch).
+  - Production posture: gateway binary needs rebuild (`cargo build --release`) + `systemctl restart lakehouse.service` to pick up the new 409 behavior in live traffic.
+- **Pre-rotation chain tamper-detect (expected, not a bug).** WORKER-{1..5} had pre-2026-05-05 audit chains under the prior `LH_SUBJECT_AUDIT_KEY`. Under the new key (post-`/tmp` wipe rotation), those chains correctly tamper-detect. The rotation runbook §4.4 documents this as expected; a §2.2 pre-rotation snapshot is what would prove they were intact pre-rotation if defensibility ever needs it.
+
+### What's blocking production cutover NOW (after this wave)
+- **Counsel calendar:** the four sign-off items in `COUNSEL_REVIEW_PACKET_2026-05-05.md` (retention schedule, consent template, destruction runbook, pre-identityd attestation). The packet tarball is ready; ⚖ counsel is the bottleneck.
+- **Nothing else.** Engineering is no longer the long pole.
+
+### Phase 1.6 BIPA gates — status table (this is the final post-Option-C state)
+
+| # | Gate | Status |
+|---|---|---|
+| 1 | Public retention schedule | **eng-staged**, revised for Option C, ready for counsel |
+| 2 | Informed consent template | **eng-staged**, revised for Option C, ready for counsel |
+| 3a | Photo upload endpoint | **DONE** (shipped `f1fa6e4`, 11 unit tests, live verified) |
+| 3b | Deepface classification | **DECIDED 2026-05-05: Option C (defer)** |
+| 4 | Name → ethnicity inference removal | **DONE** (shipped, 4/4 mcp-server tests pass) |
+| 5 | Destruction runbook + erasure endpoint | **eng-DONE** (`848a458`, 21 tests). Runbook scripts (verify + report) shipped 2026-05-05. Counsel review pending. |
+| §2 | Pre-identityd attestation | **eng-DONE** (3/3 evidence checks). Awaits J + counsel signature. |
+| §3 | Employee training | **deferred** (consolidated into runbook §7 acknowledgment for current operator population) |
+
+---
+
+## WHAT LANDED 2026-05-03 (16 commits this wave — local-first audit substrate + Phase 1.6 BIPA gates)
+
+The dominant work today: **`docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8 SHIPPED end-to-end** + **5 of 7 Phase 1.6 BIPA pre-launch gates** + **6th cross-runtime parity probe**. Wave was structured as eight ship-then-scrum cycles — every wave caught real bugs, every fix wave landed within the same session.
+
+| Commit | What | Verified |
+|---|---|---|
+| `d259909` | catalogd: Step 1 — `SubjectManifest` type + Registry CRUD | 17 catalogd subject tests PASS |
+| `d16131b` | catalogd: Step 2 — `SubjectAuditWriter` HMAC-SHA256 chain + per-subject Mutex + canonical-JSON via BTreeMap | tamper-detection + concurrent-append race tests PASS |
+| `bce6dfd` | catalogd: Step 3 — `bin/backfill_subjects` (BIPA-defensible defaults: vertical=unknown, consent=pending_backfill_review, retention=4yr) | 100 subjects loaded into live catalog |
+| `fef1efd` | gateway: Step 4 — wire SubjectAuditWriter into `/v1/chat` tool dispatch + `audit_subject_hits_in` (inline, not spawn) | tool calls log accessor.kind="gateway_lookup" |
+| `cd8c59a` | gateway: Step 5 — `AuditingWorkerLookup` decorator wraps validator's WorkerLookup; spawns audit on every find() | live `/v1/validate` produces audit rows |
+| `e38f357` | subjects Steps 1-4 fixes from cross-lineage scrum (concurrency race, schema-evolution HMAC drift, hardcoded "success" classifier) | 41 tests green |
+| `15cfd76` | catalogd + gateway: Step 6 — `/audit/subject/{id}` legal-tier endpoint with constant-time-eq token check + tampering detection | live curl returns chain_verified=true |
+| `2a4b316` | subjects 2nd scrum fix wave (token min 16→32, chain_root from full chain via `chain_tip()`, rebuild collision warn, tightened result-state heuristic) | 17 catalogd + 6 gateway tests PASS |
+| `8fc6238` | catalogd: Step 7 — `bin/retention_sweep` (BIPA-aware on biometric clock, idempotent across daily runs, no auto-mutation) | 8 sweep tests PASS, live verified at `--as-of 2031-06-01` flagging 100/100 expired |
+| `2413c96` | catalogd: Step 8 — `bin/parity_subject_audit` (Rust side of cross-runtime parity probe) | known-answer + verify modes match Go byte-for-byte |
+| `2222227` | parity helper hardening (panic-noise → die() helper, abs path stripped from doc) from scrum | parity probe still 6/6 |
+| `4708717` | **Phase 1.6 BIPA wave** — Gate 4 absence test (4/4 with bypass coverage), §2 attestation script (3/3 evidence checks), Gate 1/2/5 doc scaffolds with ⚖ COUNSEL markers | 4/4 mcp-server Bun tests, 3/3 evidence on live data |
+| `c7aa607` | Phase 1.6 scrum fixes — schema fingerprint hashes name+type+nullable, Gate 4 catches object-literal + class-field bypasses, pyarrow dep gate, item 7 deferral rationale | 4/4 + 3/3 still pass |
+| `f1fa6e4` | **Phase 1.6 Gate 3a** — `crates/catalogd/src/biometric_endpoint.rs`: `POST /biometric/subject/{id}/photo` with consent gate, quarantined storage (mode 0700/0600), audit chain link, `BiometricCollection` field on SubjectManifest | 11 unit tests PASS, live roundtrip 200 |
+| `3708e6a` | Gate 3a scrum fixes — transactional rollback on audit failure (BIPA convergent BLOCK), Content-Type parameter handling, relative data_path, ts+uuid filename, dead code removed | 11 tests + cross-runtime parity 6/6 |
+| `7e0112b` | retention_sweep: stray indent fix on biometric_collection field | sweep tests still 8/8 |
+| `848a458` | **Phase 1.6 Gate 5** — `POST /biometric/subject/{id}/erase` per BIPA destruction runbook. Two scopes (biometric_only / full); audit row appended BEFORE photo unlink so the chain has legal proof of intent even if file delete fails; manifest rolled back on audit failure. Trigger taxonomy: retention_expiry / consent_withdrawal / rtbf / court_order. | 21 unit tests (10 erasure-specific) PASS |
+| `8ec43e0` | **Phase 1.6 Gate 3b** — deepface integration design doc (Option A subprocess / Option B ONNX-in-Rust / **Option C defer**). Recommends C: BIPA-safest, classifications field stays None, all load-bearing surfaces (consent + audit + retention + erasure) ship without it. Forces "do we actually need classifications" to be answered by product, not spec inertia. | doc-only |
+
+**Cross-runtime parity (post-this-wave):** 6 probes, 38/38 byte-identical assertions —
+`validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8) + subject_audit(6/6)`.
+Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify.
+
+**Three runtime-divergence classes caught + fixed by the parity probe authoring loop** (cataloged because they recur):
+1. Go `omitempty` on string fields strips empty values that Rust serde always emits → broken HMAC
+2. `time.RFC3339Nano` truncates trailing-zero nanoseconds where chrono `AutoSi` keeps 9 digits → broken HMAC
+3. Go `json.Marshal` HTML-escapes `<>&` where serde keeps literal → broken HMAC on any field with those chars
+
+All three have regression-locked tests; structural impossibility going forward.
+
+**Phase 1.6 BIPA pre-launch gates — status table:**
+
+| Item | Status | Evidence |
+|---|---|---|
+| Gate 1 — public retention schedule | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_retention_schedule_v1.md` |
+| Gate 2 — informed consent template | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_consent_template_v1.md` |
+| Gate 3a — photo-upload endpoint | **DONE** | 11 unit tests + live `POST /biometric/subject/{id}/photo` |
+| Gate 3b — deepface classification | **design doc shipped** (`8ec43e0`) — recommends Option C (defer); awaits product confirmation | `docs/PHASE_1_6_BIPA_GATES.md` Gate 3b section |
+| Gate 4 — name→ethnicity removal | **DONE** | `mcp-server/phase_1_6_gate_4.test.ts` 4/4 with bypass coverage |
+| Gate 5 — destruction runbook + erasure endpoint | **eng-DONE** (`848a458`); ⚖ counsel review of runbook still pending | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` + `POST /biometric/subject/{id}/erase` (21 tests) |
+| §2 cryptographic attestation | eng-DONE, signature pending | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (SHA-256 evidence hash, 3/3 checks pass on live data) |
+| §3 employee training | deferred | conditional on operator population size |
+
+**Calendar bottleneck:** counsel review of items 1/2/5-runbook/§2 attestation. Engineering long pole is Gate 3b (deepface) — design doc landed (`8ec43e0`); needs product confirmation that classifications are required before engineering starts. Recommendation in doc is Option C (defer) on BIPA-safety grounds.
+
+**Operational state:**
+- `LH_SUBJECT_AUDIT_KEY=/etc/lakehouse/subject_audit.key` (32-byte HMAC signing key, mode 0400) loaded into systemd unit. **Moved off /tmp 2026-05-05** — /tmp wipes on reboot, which on May 5 disabled `/audit` + `/biometric` endpoints (gateway fails-closed at `crates/gateway/src/main.rs:459` if signing key is absent). Persistent path is per spec line 112.
+- `LH_LEGAL_AUDIT_TOKEN_FILE=/etc/lakehouse/legal_audit.token` (44-char legal-tier token, mode 0400) loaded into systemd unit
+- **Key rotation 2026-05-05:** prior key was lost when /tmp wiped on reboot. New key generated at canonical path. The 5 pre-rotation audit chains for `WORKER-{1..5}` (backfill data with `consent=pending_backfill_review`) will tamper-detect under the new key — expected and correct behavior on key rotation, not a bug. New chain entries from 2026-05-05 forward verify cleanly.
+- `data/_catalog/subjects/` holds 100 backfilled `WORKER-N.json` manifests + per-subject `WORKER-N.audit.jsonl` HMAC chains
+- `data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext>` quarantined photo storage (mode 0700 dir / 0600 file). 2 photos uploaded for WORKER-2 during live verify.
+- `/audit/subject/{id}` mounted on gateway with chain_verified=true on every probe
+- `/biometric/subject/{id}/photo` mounted on gateway, refuses 403 without `consent.biometric.status="given"`
+
+---
+
+## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits — Lance gauntlet + cross-runtime parity wave)
+
+| Commit | What | Verified |
+|---|---|---|
+| `5d30b3d` | lance: auto-build doc_id btree in `lance_migrate` handler | doc-fetch ~5ms (was ~100ms full scan) on scale_test_10m |
+| `044650a` | lance-bench: same scalar build post-IVF (matches gateway) | cargo check clean |
+| `7594725` | lance: 4-pack — `sanitize_lance_err` + 7 unit tests + 9-probe smoke + 10M re-bench | smoke 9/9 PASS, tests 7/7 PASS |
+| `98b6647` | gateway: `IterateResponse.trace_id` echoed; session_log_path enabled | parity probes see one unified JSONL |
+| `57bde63` | gateway: trace-id propagation + coordinator session JSONL (Rust parity with Go wave) | session_log_parity 4/4 |
+| `ba928b1` | aibridge: drop Python sidecar from hot path; AiClient → direct Ollama | aibridge tests 32/32 PASS, /ai/embed live 768d |
+| `654797a` | gateway: pub `extract_json` + `parity_extract_json` bin | extract_json_parity 12/12 |
+| `c5654d4` | docs: pointer to `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` | — |
+| `150cc3b` | aibridge: LRU embed cache, 236× RPS warm (78ms → 129us p50) | load test |
+| `9eed982` | mcp-server: /_go/* pass-through for G5 cutover slice | — |
+| `6e34ef7` | gitignore: stop tracking 100+ runtime ephemera (data/_*, lance, logs, node_modules) | untracked dropped 100+ → 0 |
+| `41b0a99` | chore: add 33 real items that were sitting untracked (scripts, scenarios, kimi reports, dev UIs) | clean working tree |
+
+**Cross-runtime parity (post-this-wave):** 32/32 across 5 probes — `validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8)`. Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify.
+
+**Lance backend (was untested 5 days ago, now gauntlet-ready):**
+- `cargo test -p vectord-lance --release` → 7/7 PASS
+- `./scripts/lance_smoke.sh` → 9/9 PASS against live gateway
+- `reports/lance_10m_rebench_2026-05-02.md` — search warm ~20ms / cold ~46ms median, doc-fetch ~5ms post-btree
+
+---
+
+## VERIFIED WORKING RIGHT NOW
+
+### The client demo (Staffing Co-Pilot)
+
+**Public URL:** `https://devop.live/lakehouse/` — 200, "Staffing Co-Pilot" (159 KB SPA, leaflet maps, dark theme).
+**Local URL:** `http://localhost:3700/` — same page, served by `mcp-server/index.ts` (PID 1271, started 09:48 CDT today).
+
+**The staffers console** (the one the client was thoroughly impressed with):
+- `https://devop.live/lakehouse/console` — 200, "Lakehouse — What Your Staffing System Would Do" (26 KB)
+- Pulls project index via `/api/catalog/datasets` (36 datasets) + playbook memory via `/api/vectors/playbook_memory/stats` (4,701 entries with embeddings, real ops like *"fill: Maintenance Tech x2 in Milwaukee, WI"*)
+
+Client-visible flow that works end-to-end on the public URL:
+
+| Endpoint | Sample output |
+|---|---|
+| `GET /api/catalog/datasets` | 36 datasets indexed: timesheets 1M, call_log 800K, workers_500k 500K, email_log 500K, workers_100k 100K, candidates 100K, placements 50K, job_orders 15K, successful_playbooks_live 2,077 |
+| `GET /api/vectors/playbook_memory/stats` | 4,701 fill operations with embeddings |
+| `GET /system/summary` | 36 datasets, 2.98M rows, 60 indexes, 500K workers loaded, 1K candidates |
+| `POST /intelligence/staffing_forecast` | 744 Production Workers needed in 30d, 11,281 bench (4,687 reliable), coverage 1,444%, risk=ok. Same for Electrician (need 32, bench 2,440) and Maintenance Tech (need 17, bench 5,004). |
+| `POST /intelligence/permit_contracts` | permit `3442956` $500K → 3 Production Workers, 886-candidate pool, 95% fill, $36K gross. 5 more Chicago permits with 8 workers each, same pool, 95% fill, $96K each. |
+| `POST /intelligence/market` | major Chicago permits ranked: $730M O'Hare, $615M 307 N Michigan, $580M casino, $445M Loop transit (real geo coords). |
+| `POST /intelligence/permit_entities` | architects + contractors from permit contacts (e.g. "KACPRZYNSKI, ANDY", "SLS ELECTRICAL SERVICE"). |
+| `POST /intelligence/activity` + `/intelligence/arch_signals` + `/intelligence/chat` | all 200 |
+
+The demo tells the story: *"upcoming Chicago contracts → workers needed → coverage from the bench → architects/contractors involved → revenue and margin."* That's the "live data + anticipating contracts + complete workflow" pitch — working as of right now.
+
+### Backend, verified live this session
+
+| Surface | State |
+|---|---|
+| Gateway `:3100` | up, 4 providers configured, `/v1/health` 200 with 500K workers loaded |
+| MCP server `:3700` (Co-Pilot demo) | up, all `/intelligence/*` endpoints respond |
+| VCP UI `:3950` | started this session, `/data/*` 200, real numbers |
+| Observer `:3800` | ring full (2,000/2,000) — older events evicted, query Langfuse for 24h-ago state |
+| Sidecar `:3200` | up |
+| Langfuse `:3001` | recording, `gw:/log` + `v1.chat:openrouter` traces visible |
+| LLM Team UI `:5000` | up, only `extract` mode registered |
+| OpenCode fleet | **40 models reachable through one `sk-*` key** (verified live `GET https://opencode.ai/zen/v1/models`) |
+
+OpenCode catalog (live):
+- Claude: opus-4-7, opus-4-6, opus-4-5, opus-4-1, sonnet-4-6, sonnet-4-5, sonnet-4, haiku-4-5
+- GPT-5: 5.5-pro, 5.5, 5.4-pro, 5.4, 5.4-mini, 5.4-nano, 5.3-codex-spark, 5.3-codex, 5.2, 5.2-codex, 5.1-codex-max, 5.1-codex, 5.1-codex-mini, 5.1, 5-codex, 5-nano, 5
+- Gemini: 3.1-pro, 3-flash
+- GLM: 5.1, 5
+- Minimax: m2.7, m2.5
+- Kimi: k2.6, k2.5
+- Qwen: 3.6-plus, 3.5-plus
+- Other: BIG-PKL (was a typo-prone name in the catalog, model id starts with "big-pkl-something")
+- Free tier: minimax-m2.5-free, hy3-preview-free, ling-2.6-flash-free, trinity-large-preview-free
+
+### The substrate (frozen — do not re-architect)
+
+- Distillation v1.0.0 at tag `e7636f2` — **145/145 bun tests pass, 22/22 acceptance, 16/16 audit-full**
+- Output: `data/_kb/distilled_{facts,procedures,config_hints}.jsonl` + `data/vectors/distilled_{factual,procedural,config_hint}_v20260423102847.parquet`
+- Auditor cross-lineage: Kimi K2.6 ↔ Haiku 4.5 alternation, Opus auto-promote on diffs >100k chars, **per-PR cap=3 with auto-reset on new head SHA**
+- Pathway memory: 88 traces, 11/11 successful replays (probation gate crossed)
+- Mode runner: 5 native modes; `codereview_isolation` is default; composed-corpus auto-downgrade verified Apr 26 (composed lost 5/5 vs isolation, p=0.031)
+
+### Matrix indexer
+
+30+ live corpora including:
+- 5 versions of `workers_500k_v1..v9` (50K embedded chunks each)
+- 11 batched 2K-row shards `w500k_b3..b17`
+- `chicago_permits_v1` (3,420), `resumes_100k_v2` (100K candidates), `ethereal_workers_v1` (10K)
+- `lakehouse_arch_v1` (2,119), `lakehouse_symbols_v1` (2,470), `lakehouse_answers_v1` (1,269), `scrum_findings_v1` (1,260)
+- `kb_team_runs_v1` (12,693) + `kb_team_runs_agent` (4,407) — LLM-team play history embedded
+- `distilled_factual_v20260423102507` (8) — distillation output
+
+### Code health
+
+- `cargo check --workspace` → **0 warnings, 0 errors**
+- `bun test auditor + tests/distillation` → **145/145 pass**
+- `ui/server.ts` + `auditor.ts` bundle clean
+
+---
+
+## DO NOT RELITIGATE
+
+- **PR #11 is merged into `origin/main` as `ed57eda`** — do not "still need to merge PR #11."
+- **Distillation tag `distillation-v1.0.0` at `e7636f2` is FROZEN** — do not re-architect schemas, scorer rules, audit fixtures.
+- **Kimi forensic HOLD verdict (2026-04-27) was 2/8 false + 6/8 latent** — do not re-debate, see `reports/kimi/audit-last-week-full.md`.
+- **`candidates_safe` `vertical` column bug** — fixed at catalog metadata layer in commit `c3c9c21`. Do not "discover" it again.
+- **Decisions A/B/C/D from `synthetic-data-gap-report.md`** — all four scripts shipped today (`d56f08e`, `940737d`, `c3c9c21`). Do not "ask J for approval."
+- **`workers_500k.phone` type fixup** — already string. The fixup script is idempotent; running it is a no-op.
+- **`client_workerskjkk` typo dataset** — was breaking every SQL query (catalog had it registered, file didn't exist). Removed via `DELETE /catalog/datasets/by-name/client_workerskjkk` this session. Do not re-add. Adding a startup gate that errors on unrecognized parquet names is the long-term fix per now.md Step 2C.
+- **Python sidecar dropped from hot path 2026-05-02 (`ba928b1`)** — AiClient calls Ollama directly. Do not "wire python embedding back in." `lab_ui.py` + `pipeline_lab.py` keep running as dev-only UIs (not on the runtime path).
+- **Lance backend gauntlet (2026-05-02)** — sanitizer over all 5 routes, 7 unit tests, 9-probe smoke, 10M re-bench. The `doc_id` btree auto-builds inside `lance_migrate` AND `lance-bench`. Do not "discover" the missing scalar index again or the leaked filesystem paths in error bodies.
+- **Cross-runtime parity = 32/32** across 5 probes in `golangLAKEHOUSE/scripts/cutover/parity/`. Do not "build a parity probe for X" without checking — validator, extract_json, session_log, materializer, and embed are all already covered.
+- **Decisions tracker is `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`** — single living source of truth for cross-runtime decisions. As of 2026-05-02 it has 0 `_open_` code work items; only 2 strategic items left (Lance vs Parquet+HNSW-with-spilling, Go-vs-Rust primary cutover).
+- **PRD line 70 is load-bearing — "Everything runs locally, no cloud APIs."** Yesterday's PR #13 violated this by routing customer hot-path inference to opencode/openrouter/ollama_cloud. **REVERTED 2026-05-03 (`d054c0b`).** The customer hot path (modes.toml staffing_inference, doc_drift_check; execution_loop overseer escalation) is now local Ollama (qwen3.5:latest). Cloud providers stay configured in providers.toml for **explicit dev-tool opt-in only** (scrum, auditor, bot/propose). Do NOT re-add cloud models to the hot path defaults — the customer demo runs on local + free.
+- **`/v1/usage` shows by_provider=ollama only** for any customer-shape request. If you see by_provider including kimi/openrouter/opencode/ollama_cloud for normal /v1/iterate or /v1/respond traffic, something has regressed. Verify with: `curl -sf http://127.0.0.1:3100/v1/usage | jq .by_provider`.
+- **`./scrum` is a TOOL, not architecture.** Lives at repo root. J runs it manually. Outputs findings to `data/_kb/scrum_findings.jsonl` (the KB). Findings inform development; they do NOT auto-fold into PRDs/design docs/code. If you find yourself proposing to "wire scrum into [some pipeline]" — stop. It's J's tool.
+- **Test code in main is ACTIVELY being cleaned out.** 2026-05-03 commits `6aafd41` + `f4ebd22` removed 12 orphaned dev experiments (~2900 LOC) from `tests/real-world/` and `scripts/`. If you find more zero-reference experimental files, surface them — don't auto-delete unless the pattern is clearly a one-time-experiment-with-zero-consumers like the ones already removed.
+
+---
+
+## FIXES MADE THIS SESSION (2026-04-27 evening)
+
+1. **`crates/gateway/src/v1/iterate.rs:93`** — `state` → `_state` (cleared the one cargo warning).
+2. **`lakehouse-ui.service` (Dioxus)** — disabled. Was failing 7,242 times against a missing `target/dx/ui/debug/web/public` build dir. `systemctl stop && disable`.
+3. **VCP UI on `:3950`** — started `bun run ui/server.ts` (PID 1162212, log `/tmp/lakehouse_ui.log`). `/data/*` endpoints now 200 with real data.
+4. **`client_workerskjkk` catalog entry** — `DELETE /catalog/datasets/by-name/client_workerskjkk` removed the dead manifest. **This was the actual root cause** of `/system/summary` reporting `workers_500k_rows: 0` and the demo showing zero bench. Every SQL query was failing schema inference on the missing file before reaching its target table. Fixed → `workers_500k_rows: 500000`, `candidates_rows: 1000`, demo coverage flipped from "critical 0%" to actual percentages on devop.live/lakehouse.
+
+## FIXES MADE THIS SESSION (2026-04-28 early — face pool)
+
+5. **Synthetic StyleGAN face pool — 1000 faces, gender+race+age tagged.** `scripts/staffing/fetch_face_pool.py` fetches from thispersondoesnotexist.com; `scripts/staffing/tag_face_pool.py --min-age 22` runs deepface and excludes minors. `data/headshots/manifest.jsonl` now has gender (494 men / 458 women), race (caucasian 662 · east_asian 128 · hispanic 86 · middle_eastern 59 · black 14 · south_asian 3), age, and 48 minor exclusions. Server pool = 952 servable faces.
+6. **`mcp-server/index.ts:1308` `/headshots/:key` route** — gender×race×age intersection bucketing with graceful fallback (gender-only → all). Same key always returns same face; different keys spread evenly.
+7. **`/headshots/_thumbs/` pre-resized 384×384 webp** (60× smaller: 587KB → ~11KB). Without this, 40-card grids overran Chrome's parallel-connection budget and ~75% of tiles never finished decoding. Generated via parallel ffmpeg (`xargs -P 8`); `.gitignore`d.
+8. **`mcp-server/search.html` + `console.html`** — dropped `img.loading='lazy'`. With 11KB thumbs, eager load is cheap (~500KB for 50 cards) and avoids the off-screen race that lazy decode produced.
+9. **ComfyUI on-demand uniqueness — `serve_imagegen.py:32`** added `seed` to `_cache_key()` (was caching by prompt only — 3 different worker seeds collapsed to 1 cached image). Verified: seed=839185194/195/196 → 3 distinct md5s.
+10. **`mcp-server/index.ts:1234` `/headshots/generate/:key`** — ComfyUI hot-path that derives a deterministic-per-worker seed via djb2-style hash; cold ~1.5s, cached ~1ms. Worker prompt format: `professional corporate headshot portrait of a {age}-year-old {race} {gender}, {role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`. Cache at `data/headshots_gen/` (gitignored, regeneratable).
+11. **Confidence-default name resolution** in `search.html` — `genderFor()` and `guessEthnicityFromFirstName()` lookup tables (FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN). Xavier → man+hispanic, Aisha → woman+black, etc. Every worker resolves to a face-pool bucket.
+
+End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift+operators+IL` → 21/21 cards loaded, 0 broken, all 384×384 webp thumbs.
+
+---
+
+## ⚠ PRODUCTION-READY BLOCKER (2026-05-03)
+
+**Audit-trail capability is the gate to client signature.** Smoke + parity tests prove the surface compiles; they do NOT prove an audit response can be produced for a specific person. Staffing client won't sign without defensible discrimination-claim response capability.
+
+**Authoritative document:** `docs/AUDIT_TRAIL_PRD.md` — drafted 2026-05-03. Defines worked example (John Martinez at Warehouse B), the per-decision audit row schema, the surface map of where decisions happen today, current-state-vs-target gap table, and 9-phase implementation sequence.
+
+**Phase 1 (discovery walk) requires NO J approval — it's read-only.** Phases 2+ have explicit open questions in §10 of the PRD that need J's call before they can start.
+
+Until phase 9 exit criterion is met, **do not claim "production-ready" on customer-facing surfaces.** Internal substrate (Lance, sidecar drop, parity probes) is solid; subject-of-record audit story is not.
+
+---
+
+## OPEN — but not blocking the demo
+
+| Item | What | When to act |
+|---|---|---|
+| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
+| Open PRs #6, #7, #10 | All closed 2026-05-02 — superseded / empty / stale. PR #12 merged 2026-05-03 (`a5d9070`). PR #13 merged 2026-05-03 (`feb638e`). | Done. |
+| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
+| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
+| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
+
+---
+
+## RUNTIME CHEATSHEET
+
+```bash
+# Verify the demo (public + local both work)
+curl -sS https://devop.live/lakehouse/                   # Co-Pilot HTML
+curl -sS https://devop.live/lakehouse/console            # staffers console
+curl -sS -X POST https://devop.live/lakehouse/intelligence/staffing_forecast \
+  -d '{}' -H 'content-type: application/json' \
+  | jq '.forecast[] | {role, demand_workers, bench_total, coverage_pct, risk}'
+
+# Restart sequence (after Rust changes)
+sudo systemctl restart lakehouse.service                 # gateway :3100
+sudo systemctl restart lakehouse-auditor                 # auditor daemon
+sudo systemctl restart lakehouse-observer                # observer :3800
+# UI bun on :3950 is NOT systemd-managed (lakehouse-ui.service is disabled).
+# Restart manually:  kill <pid>; nohup bun run ui/server.ts > /tmp/lakehouse_ui.log 2>&1 &
+
+# Health checks
+curl -sS http://localhost:3100/v1/health | jq            # workers_count, providers
+curl -sS http://localhost:3100/vectors/pathway/stats | jq
+curl -sS http://localhost:3100/v1/usage | jq             # since-restart cost
+curl -sS http://localhost:3700/system/summary | jq       # dataset counts
+```
+
+---
+
+## VISION — what we're actually building (not what's done)
+
+J's framing for the legacy staffing company:
+
+- Pull live data, anticipate contracts based on Chicago permits → real architect/contractor associations, headcount, time period, money, scope.
+- Hybrid + memory index → search large corpora cheaply.
+- Email comes in → verify against contract; SMS comes in → alert when index changes.
+- Real-time.
+- Invent metrics nobody else has using the hybrid index.
+- Next stage: workers download an app → geolocation clock-in → automatic responsiveness measurement, no user effort, with incentives for using it.
+- Find people getting certificates (passive cert tracking).
+- Pull union data → bring contracts that work for **employees**, not just employers.
+- All metrics visible, nothing hidden, value-aligned with what each side actually needs.
+
+If a future session is shaving away from this vision toward "fix the cutover" or "land Phase X," the vision wins. Phases are scaffolding for the vision, not the goal.
+
+---
+
+## CURRENT PLAN — fix the demo for the legacy staffing client
+
+Built from playwright audit of the live demo (2026-04-27 evening). Each item ends in something the client can SEE, not internal cleanups.
+
+**Demo state is anchored by git tag `demo-2026-04-27`** (commit `ed57eda`, the merge of PR #11). To restore code state: `git checkout demo-2026-04-27`. To restore runtime state: `DELETE /catalog/datasets/by-name/client_workerskjkk` (catalog hot-fix is not in git).
+
+### P1 — Search box that actually filters (highest visible impact)
+
+**Problem:** typing in `#sq` and pressing Enter fires `POST /intelligence/chat` with body `{"message":"<query>"}`. The state (`#sst`) and role (`#srl`) selects are ignored — never sent in the body. So every search returns a generic chat completion, never a SQL+vector hybrid filter against `workers_500k`. That is the "cached/generic response" the client sees.
+
+**Fix:** in `mcp-server/search.html`, change the search-submit handler to call the real worker search endpoint with `{query, state, role, top_k}`. The MCP `search_workers` tool surface already exists; route the form there. Render returned worker rows in the existing card grid.
+
+**Done when:** typing "forklift" + state IL + role "Forklift Operator" returns ≤ top_k IL Forklift Operators, and changing state to WI returns different workers.
+
+### P2 — Contractor-name click → `/contractor` profile page
+
+**Problem:** clicking a contractor name in any rendered card stays on `/lakehouse/`. URL doesn't change.
+
+**Fix:** wrap contractor names in `<a href="/contractor?name=<encoded>">`. The page `mcp-server/contractor.html` (14.8 KB, "Contractor Profile · Staffing Co-Pilot") already exists at `/contractor` and the data endpoint `/intelligence/contractor_profile` already returns rich data.
+
+**Then check contractor.html actually shows:** full history of every record the database has on that contractor + heat map of locations underneath + relevant info (per J 2026-04-27). If the page is incomplete, finish it. Otherwise just wire the link.
+
+**Done when:** clicking "KACPRZYNSKI, ANDY" opens a profile with: every Chicago permit they're contact_1 or contact_2 on, a leaflet map with markers for each address, and any matched workers from prior placements at their sites.
+
+### P3 — Substrate signal at the bottom shows the right numbers
+
+**Problem:** J reports the bottom panel says "playbook memory empty, 80 traces 0 replies." Reality from the live endpoints: `/api/vectors/playbook_memory/stats` = 4,701 entries with embeddings; `/vectors/pathway/stats` = 88 traces, 11/11 replays.
+
+**Fix:** find the renderer in search.html that builds the substrate signal panel; verify it's hitting the right endpoints and reading the right keys; fix shape mismatches.
+
+**Done when:** bottom panel shows real numbers (4,701 playbooks, 88 traces, 11/11 replays) and references at least one specific recent operation from the playbook stats sample.
+
+### P4 — Top nav reflects today's architecture
+
+**Problem:** Walkthrough/Architecture/Spec/Onboard/Alerts/Workspaces tabs all return 200 but content is from old architecture. Doesn't mention: gateway scratchpad, memory indexer, ranker, mode runner, OpenCode 40-model fleet, distillation substrate, auditor cross-lineage.
+
+**Fix:** rewrite `mcp-server/proof.html` (or add a single new page "What's running" that replaces Architecture+Spec) to describe what's actually shipped as of `demo-2026-04-27`. Keep one architecture page, drop redundancy. Either complete or hide Onboard/Alerts/Workspaces — J's call which.
+
+**Done when:** the architecture page tells a non-technical reader, in 2 minutes, what each piece does in coordinator-relatable terms ("intern that read every email", not "3-stage adversarial inference pipeline").
+
+### P5 — Caching for the project-index build_signal (J flagged unfinished)
+
+**Problem:** "we never finished our caching for project index build signal it's not pulling new information." Need to find what `build_signal` refers to. Likely a scrum/auditor signal that should rebuild the `lakehouse_arch_v1` corpus on commit but isn't wired to.
+
+**Fix:** identify the build-signal pipeline (likely in `auditor/` or `crates/vectord/`), wire its emit to a corpus rebuild, verify by making a test commit and watching the new chunk appear in `/vectors/indexes` for `lakehouse_arch_v1`.
+
+**Done when:** committing a new file to `crates/` causes `lakehouse_arch_v1` chunk_count to increase within N minutes.
+
+### P0 — Anchor the demo state (DONE)
+
+Tagged `ed57eda` as `demo-2026-04-27`. Future sessions: `git checkout demo-2026-04-27` to land in this exact code state.
+
+---
+
+## EXECUTION ORDER
+
+1. **P1 first** — biggest visible bug, ~30-60 min
+2. **P2 next** — contractor click is the second-biggest "doesn't work" the client sees, ~20 min if profile is mostly done
+3. **P3** — small fix, big "looks alive" win
+4. **P4** — biggest scope; might split across sessions
+5. **P5** — feature work, only after the visible bugs are fixed
+
+Each item commits independently with the format `demo: P<n> — <one-line>` so the commit log doubles as a progress journal. After each merge to main, re-tag `demo-latest` to point at the new HEAD.
+
+Stop here and let J pick which item to start with. Do not silently extend scope.

auditor/audit.ts

@@ -23,6 +23,7 @@ import { runStaticCheck } from "./checks/static.ts";
 import { runDynamicCheck } from "./checks/dynamic.ts";
 import { runInferenceCheck } from "./checks/inference.ts";
 import { runKbCheck } from "./checks/kb_query.ts";
+import { runKimiArchitectCheck } from "./checks/kimi_architect.ts";
 
 const VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/verdicts";
 // Playbook for audit findings — one row per block/warn finding from a
@@ -67,6 +68,29 @@ export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<
     ...kbFindings,
   ];
 
+  // Kimi-architect second-pass review. Off by default; enabled with
+  // LH_AUDITOR_KIMI=1. Sequential (not in the parallel block above)
+  // because it consumes the prior findings as context — Kimi sees what
+  // deepseek already flagged and is asked "what did everyone miss?"
+  // Failure-isolated by design: any error returns a single info-level
+  // skip finding so the existing audit pipeline never blocks on Kimi.
+  if (process.env.LH_AUDITOR_KIMI === "1") {
+    try {
+      const kimiFindings = await runKimiArchitectCheck(diff, allFindings, {
+        pr_number: pr.number,
+        head_sha: pr.head_sha,
+      });
+      allFindings.push(...kimiFindings);
+    } catch (e) {
+      allFindings.push({
+        check: "kimi_architect",
+        severity: "info",
+        summary: `kimi_architect outer error — ${(e as Error).message.slice(0, 160)}`,
+        evidence: [(e as Error).stack?.slice(0, 360) ?? ""],
+      });
+    }
+  }
+
   const duration_ms = Date.now() - t0;
   const metrics = {
     audit_duration_ms: duration_ms,
@@ -184,7 +208,7 @@ function formatReviewBody(v: Verdict): string {
   lines.push("");
 
   // Per-check sections, only if the check produced findings.
-  const checkOrder = ["static", "dynamic", "inference", "kb_query"] as const;
+  const checkOrder = ["static", "dynamic", "inference", "kb_query", "kimi_architect"] as const;
   for (const check of checkOrder) {
     const fs = byCheck[check] ?? [];
     if (fs.length === 0) continue;
@@ -217,6 +241,6 @@ function formatReviewBody(v: Verdict): string {
   return lines.join("\n");
 }
 
-function stubFinding(check: "dynamic" | "inference", why: string): Finding[] {
+function stubFinding(check: "dynamic" | "inference" | "kimi_architect", why: string): Finding[] {
   return [{ check, severity: "info", summary: `${check} check skipped — ${why}`, evidence: [why] }];
 }

auditor/checks/inference.ts

@@ -28,42 +28,26 @@ const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
 // retries — verified with trivial 8-token probes). Swapped default to
 // deepseek-v3.1:671b which is proven working end-to-end through the
 // pr_audit mode runner during Phase 5 distillation acceptance testing.
-// kimi-k2:1t can be re-selected via LH_AUDITOR_REVIEW_MODEL env when
-// the upstream returns. Tie-breaker stays grok-4.1-fast (different
-// vendor lineage so consensus + tie-break won't fail-correlate).
-const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "deepseek-v3.1:671b";
-const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "x-ai/grok-4.1-fast";
-// SHARD_MODEL retained for the legacy callCloud path (still used by
-// runCloudInference's diagnostic mode), but no longer fired by the
-// main inference flow — tree-split was retired 2026-04-27 in favor of
-// the mode runner's matrix retrieval against lakehouse_answers_v1.
-const SHARD_MODEL = process.env.LH_AUDITOR_SHARD_MODEL ?? "qwen3-coder:480b";
-const N_CONSENSUS = Number(process.env.LH_AUDITOR_CONSENSUS_N ?? 3);
-// Bounded parallelism on the tree-split shard loop. Old behavior was
-// fully serial ("keep gateway load bounded") which made huge PRs take
-// 5+ minutes of curation alone. 6 in flight keeps gateway busy without
-// thrashing it; tunable via env.
-const SHARD_CONCURRENCY = Number(process.env.LH_AUDITOR_SHARD_CONCURRENCY ?? 6);
-const AUDIT_DISCREPANCIES_JSONL = "/home/profit/lakehouse/data/_kb/audit_discrepancies.jsonl";
-// 40KB comfortably fits gpt-oss:120b's context. PR #1 (~39KB) was
-// previously truncated at 15KB causing the reviewer to miss later
-// files (gitea.ts, policy.ts) and flag "no Gitea client present" as a
-// block finding when the file was simply outside the truncation window.
 //
-// Above this threshold we curate via tree-split rather than truncate,
-// following the scrum_master pattern: shard the diff, summarize each
-// shard against the claim-verification task, merge into a compact
-// scratchpad, then ask the cloud to verify claims against the
-// scratchpad. This gives the cloud full-PR fidelity without bursting
-// its context window (observed failure mode: empty response or
-// unparseable output when prompt exceeds model's comfortable range).
+// 2026-09-02 update: Ollama Cloud subscription lapsed — every
+// ollama_cloud model 402s ("requires a subscription"). J's call: sweep
+// the auditor to OpenRouter. Same deepseek lineage via OpenRouter;
+// tie-breaker moves to grok-4.3 because OpenRouter 404s grok-4.1-fast
+// as deprecated. Both probed live through the gateway before landing.
+// Vendor lineages stay distinct so consensus + tie-break won't
+// fail-correlate.
+const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "openrouter/deepseek/deepseek-chat-v3.1";
+const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "openrouter/x-ai/grok-4.3";
+const N_CONSENSUS = Number(process.env.LH_AUDITOR_CONSENSUS_N ?? 3);
+const AUDIT_DISCREPANCIES_JSONL = "/home/profit/lakehouse/data/_kb/audit_discrepancies.jsonl";
+// 40KB comfortably fits the consensus models' context windows
+// (deepseek-v3.1 64K, gpt-oss-120b 128K). When the raw PR diff
+// exceeds this, we truncate and signal it via curationNote — the
+// pr_audit mode runner's matrix retrieval (lakehouse_answers_v1 +
+// arch + symbols) supplies the cross-PR context that tree-split
+// used to synthesize from scratch. Tree-split itself was retired
+// 2026-04-27 (see commit deleting treeSplitDiff/callCloud/SHARD_*).
 const MAX_DIFF_CHARS = 40000;
-// Tree-split kicks in above this. 30KB is below MAX_DIFF_CHARS so we
-// curate BEFORE truncation would happen — never lose signal to a hard
-// cut. Shard size is chosen so ~10 shards cover PR #8-size diffs in a
-// reasonable round-trip budget.
-const CURATION_THRESHOLD = 30000;
-const DIFF_SHARD_SIZE = 4500;
 const CALL_TIMEOUT_MS = 120_000;
 // Mode runner can take longer than a raw /v1/chat call because it does
 // pathway-fingerprint lookup + matrix retrieval + relevance filter
@@ -169,12 +153,16 @@ export async function runInferenceCheck(
   interface Votes { trues: number; falses: number; evidences: string[] }
   const votesByClaim = new Map<number, Votes>();
   const unflaggedByRun: any[][] = [];
-  let totalLatencyMs = 0;
+  // The N=3 consensus calls run via Promise.all — wall-clock is
+  // bounded by the SLOWEST call, not the sum. Pre-2026-04-27 we
+  // summed and reported "Xms total" which double/triple-counted
+  // (Opus self-audit caught it). Use max for accurate wall-clock.
+  let maxLatencyMs = 0;
   let totalEnrichedChars = 0;
   let bugFingerprintsSeen = 0;
   let matrixKeptSeen = 0;
   for (const run of parsedRuns) {
-    totalLatencyMs += run.latency_ms ?? 0;
+    maxLatencyMs = Math.max(maxLatencyMs, run.latency_ms ?? 0);
     totalEnrichedChars += run.enriched_chars ?? 0;
     bugFingerprintsSeen = Math.max(bugFingerprintsSeen, run.bug_fingerprints ?? 0);
     matrixKeptSeen = Math.max(matrixKeptSeen, run.matrix_kept ?? 0);
@@ -199,7 +187,7 @@ export async function runInferenceCheck(
   findings.push({
     check: "inference",
     severity: "info",
-    summary: `pr_audit mode runner completed (model=${MODEL}, consensus=${parsedRuns.length}/${N_CONSENSUS}, ${totalLatencyMs}ms total)${curationNote}`,
+    summary: `pr_audit mode runner completed (model=${MODEL}, consensus=${parsedRuns.length}/${N_CONSENSUS}, ${maxLatencyMs}ms wall-clock)${curationNote}`,
     evidence: [
       `claims voted: ${votesByClaim.size}`,
       `parsed runs: ${parsedRuns.length} / ${N_CONSENSUS}`,
@@ -443,60 +431,28 @@ async function runModeRunnerInference(
       error: "unparseable", diagnostic: (e as Error).message, model,
     };
   }
-  const content: string = body?.response ?? "";
+  const content: string = typeof body?.response === "string" ? body.response : "";
   const parsed = extractJson(content);
+  // Number-coerced extractors so a non-numeric upstream value (string,
+  // null, NaN) collapses to 0 instead of poisoning downstream
+  // arithmetic. Caught 2026-04-27 by kimi_architect self-audit —
+  // optional-chaining + ?? only catches null/undefined, not type drift.
+  const num = (v: unknown): number => {
+    const n = typeof v === "number" ? v : Number(v);
+    return Number.isFinite(n) ? n : 0;
+  };
   return {
     parsed,
-    latency_ms: body?.latency_ms ?? 0,
-    enriched_chars: body?.enriched_prompt_chars ?? 0,
-    bug_fingerprints: body?.sources?.bug_fingerprints_count ?? 0,
-    matrix_kept: body?.sources?.matrix_chunks_kept ?? 0,
+    latency_ms: num(body?.latency_ms),
+    enriched_chars: num(body?.enriched_prompt_chars),
+    bug_fingerprints: num(body?.sources?.bug_fingerprints_count),
+    matrix_kept: num(body?.sources?.matrix_chunks_kept),
     error: parsed ? undefined : "unparseable",
     diagnostic: parsed ? undefined : content.slice(0, 200),
     model,
   };
 }
 
-// Legacy direct /v1/chat caller — kept for callers outside the
-// pr_audit pipeline. Currently unused after the 2026-04-26 mode-runner
-// rebuild; preserved so we can A/B against the mode runner if a
-// regression surfaces.
-async function runCloudInference(systemMsg: string, userMsg: string, model: string): Promise<{ parsed: any | null; tokens: number; error?: string; diagnostic?: string; model: string }> {
-  let resp: Response;
-  try {
-    resp = await fetch(`${GATEWAY}/v1/chat`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        provider: "ollama_cloud",
-        model,
-        messages: [
-          { role: "system", content: systemMsg },
-          { role: "user", content: userMsg },
-        ],
-        max_tokens: 3000,
-        temperature: 0,
-        think: true,
-      }),
-      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
-    });
-  } catch (e) {
-    return { parsed: null, tokens: 0, error: "unreachable", diagnostic: (e as Error).message.slice(0, 200), model };
-  }
-  if (!resp.ok) {
-    return { parsed: null, tokens: 0, error: "non_200", diagnostic: `${resp.status}: ${(await resp.text()).slice(0, 160)}`, model };
-  }
-  let body: any;
-  try { body = await resp.json(); }
-  catch (e) { return { parsed: null, tokens: 0, error: "unparseable", diagnostic: (e as Error).message, model }; }
-  const content: string = body?.choices?.[0]?.message?.content ?? "";
-  const tokens: number = body?.usage?.total_tokens ?? 0;
-  const parsed = extractJson(content);
-  if (!parsed) {
-    return { parsed: null, tokens, error: "unparseable", diagnostic: content.slice(0, 200), model };
-  }
-  return { parsed, tokens, model };
-}
 
 async function persistDiscrepancies(ctx: InferenceContext, discrepancies: any[]): Promise<void> {
   await mkdir("/home/profit/lakehouse/data/_kb", { recursive: true });
@@ -540,111 +496,7 @@ async function extractAndPersistFacts(scratchpad: string, ctx: InferenceContext)
   await appendFile(AUDIT_FACTS_JSONL, JSON.stringify(row) + "\n");
 }
 
-// Curation via tree-split — ports the scrum_master pattern into the
-// inference check. Shards the raw diff into DIFF_SHARD_SIZE chunks,
-// summarizes each shard *against the claim-verification task* so the
-// summary preserves exactly what the cloud needs to judge claims
-// (function signatures, struct fields, deletions, new files), drops
-// everything else. Merges into a compact scratchpad.
-//
-// Cost: N cloud calls for shard summaries + the final verification.
-// Pre-2026-04-26 the shard loop ran serially "to keep gateway load
-// bounded" — turned out to be a bottleneck on PRs with 50+ shards
-// (5+ minutes of curation). Now bounded-parallel via
-// SHARD_CONCURRENCY: in-flight ≤ N at any time, gateway stays calm,
-// wall-clock drops 4-6×.
-//
-// Determinism: each shard summary call uses temp=0 + think=false
-// (same as before), so identical input yields identical scratchpad.
-// Order is preserved by indexed-write into a fixed-length array
-// before string-join, so concurrency doesn't shuffle the scratchpad.
-async function treeSplitDiff(
-  fullDiff: string,
-  claims: Claim[],
-): Promise<{ scratchpad: string; shards: number }> {
-  const shards: Array<{ from: number; to: number; text: string }> = [];
-  for (let i = 0; i < fullDiff.length; i += DIFF_SHARD_SIZE) {
-    const end = Math.min(i + DIFF_SHARD_SIZE, fullDiff.length);
-    shards.push({ from: i, to: end, text: fullDiff.slice(i, end) });
-  }
-  // Curate the claim list into a short form the summary prompt can
-  // use to bias extraction toward relevant facts.
-  const claimDigest = claims.map((c, i) =>
-    `${i}. [${c.strength}] "${c.text.slice(0, 100)}"`
-  ).join("\n");
 
-  const buildPrompt = (si: number, shard: { from: number; to: number; text: string }): string => [
-    `You are summarizing shard ${si + 1}/${shards.length} (chars ${shard.from}..${shard.to}) of a PR diff.`,
-    `The downstream task will verify these ship-claims against the full-PR summary. Extract ONLY facts that could confirm or refute these claims:`,
-    "",
-    claimDigest,
-    "",
-    "Extract: new function/method signatures, struct fields, deletions, new files, wiring (function X calls Y), absence-of-implementation markers, TODO comments on added lines.",
-    "Skip: comment-only edits, whitespace, import reordering, unrelated cosmetic changes.",
-    "",
-    "─────── shard diff ───────",
-    shard.text,
-    "─────── end shard ───────",
-    "",
-    "Output: up to 180 words of facts in bullet form. No prose preamble, no claim verdicts (that's for the downstream step).",
-  ].join("\n");
-
-  // Pre-allocate so we can write back at the original index from
-  // out-of-order completion.
-  const summaries: string[] = new Array(shards.length).fill("");
-  let nextIdx = 0;
-  async function worker() {
-    while (true) {
-      const myIdx = nextIdx++;
-      if (myIdx >= shards.length) return;
-      const r = await callCloud(buildPrompt(myIdx, shards[myIdx]), 400);
-      summaries[myIdx] = r.content;
-    }
-  }
-  const concurrency = Math.max(1, Math.min(SHARD_CONCURRENCY, shards.length));
-  await Promise.all(Array.from({ length: concurrency }, worker));
-
-  let scratchpad = "";
-  for (const [si, shard] of shards.entries()) {
-    const summary = summaries[si];
-    if (summary) {
-      scratchpad += `\n--- shard ${si + 1} (chars ${shard.from}..${shard.to}) ---\n${summary.trim()}\n`;
-    }
-  }
-  return { scratchpad: scratchpad.trim(), shards: shards.length };
-}
-
-// Minimal cloud caller used only by treeSplitDiff — same gateway +
-// model as the top-level call, but think=false. Shards are small
-// (≤DIFF_SHARD_SIZE ~4500 chars) and the task is pure fact
-// extraction, not reasoning. think=true on the shards introduced
-// variance in reasoning traces that compounded across 23 calls into
-// a non-deterministic scratchpad (observed during curation
-// validation: same-SHA runs produced 5/7/8 final findings).
-// think=false on small prompts is stable — only breaks at the main
-// call's 10K+ prompt size, which keeps think=true.
-async function callCloud(prompt: string, maxTokens: number): Promise<{ content: string }> {
-  try {
-    const r = await fetch(`${GATEWAY}/v1/chat`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        provider: "ollama_cloud",
-        model: SHARD_MODEL,
-        messages: [{ role: "user", content: prompt }],
-        max_tokens: maxTokens,
-        temperature: 0,
-        think: false,
-      }),
-      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
-    });
-    if (!r.ok) return { content: "" };
-    const j: any = await r.json();
-    return { content: j?.choices?.[0]?.message?.content ?? "" };
-  } catch {
-    return { content: "" };
-  }
-}
 
 // Pull out plausible code-symbol names from a summary string.
 // Matches:

auditor/checks/kimi_architect.ts

@@ -0,0 +1,463 @@
+// Kimi-architect check — second-pass senior architectural review using
+// kimi-for-coding (Kimi K2.6) via /v1/chat provider=kimi.
+//
+// Runs AFTER the deepseek inference check (N=3 consensus) and the
+// static/kb_query checks. Reads their findings as context and asks Kimi
+// "what did everyone else miss?" — complementing the cheap-consensus
+// voting with a sparse senior pass that catches load-bearing issues
+// (compile errors, false telemetry, schema bypasses, etc.) which the
+// voting structure can't see.
+//
+// Why Kimi here and not in the inner inference loop:
+// - Cost: ~3min wall-clock per call vs ~30s for deepseek consensus.
+// - TOS: api.kimi.com is User-Agent-gated (see crates/gateway/src/v1/
+//   kimi.rs); cost-bounded calls only.
+// - Value: experiment 2026-04-27 showed 7/7 grounding rate with full
+//   files vs ~50% on truncated input. Best as a sparse complement, not
+//   a replacement.
+//
+// Failure-isolated: any Kimi error returns a single info-level Finding
+// "kimi_architect skipped — <reason>" so the existing audit pipeline
+// is never blocked by a Kimi outage / TOS revocation / 429.
+//
+// Cost cap: if a kimi_verdicts/<pr>-<sha>.json file exists less than 24h
+// old, return cached findings without calling upstream. New commits
+// produce new SHAs so this is per-head, not per-day.
+//
+// Off by default: caller checks LH_AUDITOR_KIMI=1 before invoking.
+
+import { readFile, writeFile, mkdir, appendFile, stat, realpath } from "node:fs/promises";
+import { existsSync, realpathSync } from "node:fs";
+import { dirname, join, resolve } from "node:path";
+import type { Finding, CheckKind } from "../types.ts";
+
+const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
+const KIMI_VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/kimi_verdicts";
+const KIMI_AUDITS_JSONL = "/home/profit/lakehouse/data/_kb/kimi_audits.jsonl";
+const REPO_ROOT = "/home/profit/lakehouse";
+// Canonicalize at module load — REPO_ROOT itself may be a symlink in
+// some environments (e.g. /home/profit is a bind-mount). Computing
+// once at startup means the per-finding grounding loop can compare
+// realpath(target) against this stable anchor.
+const REPO_ROOT_REAL = (() => {
+  try { return realpathSync(REPO_ROOT); }
+  catch { return REPO_ROOT; }
+})();
+// 15 min budget. Bun's fetch has an intrinsic ~300s limit that our
+// AbortController + setTimeout combo could not override; we use curl
+// via Bun.spawn instead (callKimi below). Curl honors -m for max
+// transfer time without a hard intrinsic ceiling.
+const CALL_TIMEOUT_MS = 900_000;
+const CACHE_TTL_MS = 24 * 60 * 60 * 1000;
+const MAX_DIFF_CHARS = 180_000;
+const MAX_PRIOR_FINDINGS = 50;
+// Default provider/model = openrouter/moonshotai/kimi-k2.6.
+// History: pre-2026-04-27 direct api.kimi.com (User-Agent gating),
+// then ollama_cloud/kimi-k2.6 while the sub was active. 2026-09-02:
+// Ollama Cloud subscription lapsed (402 on every cloud model), J's
+// call was to sweep the auditor to OpenRouter — same Moonshot model,
+// probed live through the gateway. The api.kimi.com path
+// (provider=kimi) remains wired in the gateway as a fallback.
+const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "openrouter";
+const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "moonshotai/kimi-k2.6";
+// Cross-lineage alternation. 2026-04-27 J's call: Opus is too
+// expensive to auto-fire (~$0.30/audit). Kimi K2.6 via Go-sub is
+// effectively free; Haiku 4.5 via Zen is ~$0.04. Alternate between
+// them so we get cross-lineage signal (Moonshot vs Anthropic) on
+// every PR's audit history without burning the budget.
+//
+// Default: Kimi K2.6 on even audits, Haiku 4.5 on odd. Each PR's
+// audits flip between vendors as new SHAs come in.
+//
+// Frontier models (Opus 4.7, GPT-5.5, Gemini 3.1) are NOT in the
+// auto path. Operator hands distilled findings to a frontier model
+// manually when high-leverage decisions need it. Removing Opus from
+// auto-promotion saves ~$1-3/day on the daemon at our cadence.
+//
+// Override the alternation entirely with LH_AUDITOR_KIMI_MODEL
+// (forces one model regardless of audit count); set
+// LH_AUDITOR_KIMI_ALT_MODEL to the alternate.
+const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "claude-haiku-4-5";
+const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "opencode";
+const FORCE_DEFAULT = process.env.LH_AUDITOR_KIMI_MODEL !== undefined && process.env.LH_AUDITOR_KIMI_MODEL !== "";
+
+function selectModel(diffLen: number, auditIndex: number = 0): { provider: string; model: string; promoted: boolean } {
+  // Operator override — env-pinned model wins.
+  if (FORCE_DEFAULT) {
+    return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
+  }
+  // Alternate Kimi (default, even index) ↔ Haiku (alt, odd index).
+  // diffLen kept in the signature for future "big diff → Haiku
+  // anyway" logic; not used yet so we don't auto-burn on big PRs.
+  void diffLen;
+  if (auditIndex % 2 === 1) {
+    return { provider: ALT_PROVIDER, model: ALT_MODEL, promoted: true };
+  }
+  return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
+}
+// Model-aware max_tokens. Different upstream APIs cap at different
+// limits and reject requests that exceed them:
+//   - Anthropic Opus 4.x: 32K output (with extended-output header)
+//   - Anthropic Haiku 4.5: 8K output
+//   - Kimi K2.6 (reasoning): 128K — needs headroom because
+//     reasoning_content counts against the budget
+//   - Default: 16K, conservative middle ground
+//
+// 2026-04-27 BLOCK from Opus self-audit: the prior single-default of
+// 128K worked silently (Anthropic clamps server-side) but was
+// technically invalid. Per-model caps make it explicit. Override via
+// LH_AUDITOR_KIMI_MAX_TOKENS to force a value (also fixes the empty-
+// env Number("") -> 0 trap by using `||` not `??`).
+const MAX_TOKENS_OVERRIDE = Number(process.env.LH_AUDITOR_KIMI_MAX_TOKENS) || 0;
+function maxTokensFor(model: string): number {
+  if (MAX_TOKENS_OVERRIDE > 0) return MAX_TOKENS_OVERRIDE;
+  if (model.startsWith("claude-opus")) return 32_000;
+  if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet")) return 8_192;
+  if (model.startsWith("kimi-") || model.includes("/kimi-")) return 128_000;
+  if (model.startsWith("gpt-5") || model.startsWith("o1") || model.startsWith("o3") || model.startsWith("o4")) return 32_000;
+  return 16_000;
+}
+
+export interface KimiArchitectContext {
+  pr_number: number;
+  head_sha: string;
+}
+
+interface KimiVerdictFile {
+  pr_number: number;
+  head_sha: string;
+  cached_at: string;
+  model: string;
+  latency_ms: number;
+  finish_reason: string;
+  usage: { prompt_tokens: number; completion_tokens: number; total_tokens: number };
+  raw_content: string;
+  findings: Finding[];
+  grounding: { total: number; verified: number; rate: number };
+}
+
+export async function runKimiArchitectCheck(
+  diff: string,
+  priorFindings: Finding[],
+  ctx: KimiArchitectContext,
+): Promise<Finding[]> {
+  const cachePath = join(KIMI_VERDICTS_DIR, `${ctx.pr_number}-${ctx.head_sha.slice(0, 12)}.json`);
+  const outageSentinel = `${cachePath}.outage`;
+  const OUTAGE_TTL_MS = 10 * 60 * 1000;
+
+  // Outage negative-cache — if upstream failed within OUTAGE_TTL_MS,
+  // skip this audit and return immediately. Prevents the daemon from
+  // hammering a downed Kimi/Anthropic upstream every 90s.
+  if (existsSync(outageSentinel)) {
+    try {
+      const s = await stat(outageSentinel);
+      if (Date.now() - s.mtimeMs < OUTAGE_TTL_MS) {
+        const note = JSON.parse(await readFile(outageSentinel, "utf8"));
+        return [skipFinding(`upstream still down (cached ${Math.round((Date.now() - s.mtimeMs) / 1000)}s ago): ${String(note.reason).slice(0, 160)}`)];
+      }
+    } catch { /* malformed sentinel — fall through to fresh call */ }
+  }
+
+  // Cost cap — return cached findings if a verdict for this exact head
+  // SHA was generated within the TTL.
+  const cached = await loadCachedVerdict(cachePath);
+  if (cached) {
+    return cached.findings.length > 0
+      ? cached.findings
+      : [{ check: "kimi_architect" as CheckKind, severity: "info", summary: "kimi_architect cached — 0 findings", evidence: [`cache: ${cachePath}`] }];
+  }
+
+  // Alternate model based on how many audits this PR has had — gives
+  // cross-lineage signal (Kimi/Moonshot ↔ Haiku/Anthropic) on every
+  // PR's audit history. Count is derived from existing kimi_verdicts
+  // files for this PR; cheap O(N_PRs) directory read.
+  let auditIndex = 0;
+  try {
+    const dir = "/home/profit/lakehouse/data/_auditor/kimi_verdicts";
+    if (existsSync(dir)) {
+      const all = require("node:fs").readdirSync(dir) as string[];
+      auditIndex = all.filter((f) => f.startsWith(`${ctx.pr_number}-`)).length;
+    }
+  } catch { /* default 0 — Kimi */ }
+
+  const selected = selectModel(diff.length, auditIndex);
+  let response: { content: string; usage: any; finish_reason: string; latency_ms: number };
+  try {
+    response = await callKimi(buildPrompt(diff, priorFindings, ctx), selected.provider, selected.model);
+  } catch (e) {
+    // Negative-cache for 10 min on outage (caught 2026-04-27 by Opus
+    // self-audit): without this, every audit cycle within the 24h
+    // TTL re-calls upstream while it's still down. Use a sentinel
+    // file with mtime check rather than persisting a verdict so the
+    // happy-path cache reader doesn't have to special-case it.
+    const sentinel = `${cachePath}.outage`;
+    try { await writeFile(sentinel, JSON.stringify({ at: new Date().toISOString(), reason: (e as Error).message.slice(0, 200) })); } catch {}
+    return [skipFinding(`kimi call failed (${selected.model}): ${(e as Error).message.slice(0, 200)}`)];
+  }
+
+  const findings = parseFindings(response.content);
+  const grounding = await computeGrounding(findings);
+
+  const verdict: KimiVerdictFile = {
+    pr_number: ctx.pr_number,
+    head_sha: ctx.head_sha,
+    cached_at: new Date().toISOString(),
+    model: selected.model,
+    latency_ms: response.latency_ms,
+    finish_reason: response.finish_reason,
+    usage: {
+      prompt_tokens: response.usage?.prompt_tokens ?? 0,
+      completion_tokens: response.usage?.completion_tokens ?? 0,
+      total_tokens: response.usage?.total_tokens ?? 0,
+    },
+    raw_content: response.content,
+    findings,
+    grounding,
+  };
+
+  // Cache-poisoning guard (caught 2026-04-27 by Opus self-audit):
+  // when parseFindings returns 0 findings (Kimi rambled, prompt too
+  // big, or the markdown shape changed and our regex missed every
+  // block), persisting the empty verdict short-circuits all future
+  // audits in the 24h TTL window with a useless cached "0 findings"
+  // result. Better to leave no cache and re-call upstream next time.
+  // Always append metrics — observability shouldn't depend on whether
+  // findings parsed.
+  await appendMetrics(verdict);
+  if (findings.length > 0) {
+    await persistVerdict(cachePath, verdict);
+    return findings;
+  }
+  return [{
+    check: "kimi_architect" as CheckKind,
+    severity: "info",
+    summary: `kimi_architect produced 0 ranked findings (${response.finish_reason}, ${verdict.usage.completion_tokens} tokens) — not cached`,
+    evidence: [`raw saved (no cache): see kimi_audits.jsonl ${verdict.cached_at}`],
+  }];
+}
+
+async function loadCachedVerdict(path: string): Promise<KimiVerdictFile | null> {
+  if (!existsSync(path)) return null;
+  try {
+    const s = await stat(path);
+    if (Date.now() - s.mtimeMs > CACHE_TTL_MS) return null;
+    return JSON.parse(await readFile(path, "utf8")) as KimiVerdictFile;
+  } catch { return null; }
+}
+
+function buildPrompt(diff: string, priorFindings: Finding[], ctx: KimiArchitectContext): string {
+  const truncatedDiff = diff.length > MAX_DIFF_CHARS
+    ? diff.slice(0, MAX_DIFF_CHARS) + `\n\n... [truncated; original diff was ${diff.length} chars]`
+    : diff;
+
+  const priorBlock = priorFindings
+    .filter(f => f.severity !== "info")
+    .slice(0, MAX_PRIOR_FINDINGS)
+    .map(f => `- [${f.check}/${f.severity}] ${f.summary}${f.evidence?.[0] ? ` — ${f.evidence[0].slice(0, 160)}` : ""}`)
+    .join("\n");
+
+  return `You are a senior software architect doing a second-pass review on PR #${ctx.pr_number} (head ${ctx.head_sha.slice(0, 12)}). The team's automated auditor (deepseek-v3.1:671b, N=3 consensus) already produced findings. Your job is NOT to repeat what they found — your job is to catch what their voting structure CAN'T see: compile errors, type-system bypasses, false telemetry, silent determinism leaks, schema-bypass anti-patterns, load-bearing assumptions that look fine line-by-line.
+
+GROUNDING RULES (non-negotiable):
+- Cite file:line for EVERY finding. Lines you cite must actually contain what you claim. Confabulating a finding wastes more time than missing one.
+- If the diff is truncated and you can't verify a claim, say "diff-truncated, can't verify" — DO NOT guess.
+- Distinguish architectural concerns (no specific line) from concrete bugs (specific line). Don't dress one as the other.
+
+PRIOR FINDINGS FROM DEEPSEEK CONSENSUS (do not repeat these):
+${priorBlock || "(none)"}
+
+OUTPUT FORMAT (markdown):
+- ## Verdict (one sentence)
+- ## Findings (5-10 items, each formatted EXACTLY as below)
+
+For each finding use this exact shape so a parser can lift them:
+
+### F1: <one-line summary>
+- **Severity:** block | warn | info
+- **File:** path/to/file.ext:LINE
+- **Rationale:** one or two sentences
+
+THE DIFF:
+
+${truncatedDiff}
+`;
+}
+
+async function callKimi(prompt: string, provider: string, model: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
+  const t0 = Date.now();
+  const body = JSON.stringify({
+    provider,
+    model,
+    messages: [{ role: "user", content: prompt }],
+    max_tokens: maxTokensFor(model),
+    temperature: 0.2,
+  });
+  // curl via Bun.spawn — bypasses Bun fetch's ~300s intrinsic ceiling.
+  // -m sets the max transfer time honored end-to-end. Body is piped via
+  // stdin to avoid argv length limits on big audit prompts (~50K+ tokens).
+  const proc = Bun.spawn({
+    cmd: [
+      "curl", "-sS", "-X", "POST",
+      "-m", String(Math.ceil(CALL_TIMEOUT_MS / 1000)),
+      "-H", "content-type: application/json",
+      "--data-binary", "@-",
+      `${GATEWAY}/v1/chat`,
+    ],
+    stdin: "pipe",
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  proc.stdin.write(body);
+  await proc.stdin.end();
+  const [stdout, stderr, exitCode] = await Promise.all([
+    new Response(proc.stdout).text(),
+    new Response(proc.stderr).text(),
+    proc.exited,
+  ]);
+  if (exitCode !== 0) {
+    throw new Error(`curl exit ${exitCode}: ${stderr.slice(0, 300)}`);
+  }
+  let j: any;
+  try { j = JSON.parse(stdout); }
+  catch (e) {
+    throw new Error(`bad response (${stdout.length} bytes): ${stdout.slice(0, 300)}`);
+  }
+  if (j.error || !j.choices) {
+    throw new Error(`gateway error: ${JSON.stringify(j).slice(0, 300)}`);
+  }
+  return {
+    content: j.choices?.[0]?.message?.content ?? "",
+    usage: j.usage ?? {},
+    finish_reason: j.choices?.[0]?.finish_reason ?? "unknown",
+    latency_ms: Date.now() - t0,
+  };
+}
+
+// Parse Kimi's markdown into Finding[]. Format expected (per buildPrompt):
+//   ### F<N>: <summary>
+//   - **Severity:** block | warn | info
+//   - **File:** path:line
+//   - **Rationale:** ...
+function parseFindings(content: string): Finding[] {
+  const findings: Finding[] = [];
+  const blocks = content.split(/^###\s+F\d+:\s*/m).slice(1);
+  for (const block of blocks) {
+    const summary = (block.split("\n")[0] ?? "").trim();
+    if (!summary) continue;
+    const sev = /\*\*Severity:\*\*\s*(block|warn|info)/i.exec(block)?.[1]?.toLowerCase();
+    const fileLine = /\*\*File:\*\*\s*(\S+)/i.exec(block)?.[1] ?? "unknown";
+    const rationale = /\*\*Rationale:\*\*\s*([\s\S]+?)(?=\n###|\n\*\*|$)/i.exec(block)?.[1]?.trim() ?? "";
+    const severity: Finding["severity"] = sev === "block" ? "block" : sev === "warn" ? "warn" : "info";
+    findings.push({
+      check: "kimi_architect" as CheckKind,
+      severity,
+      summary: summary.slice(0, 240),
+      evidence: [fileLine, rationale.slice(0, 360)].filter(Boolean),
+    });
+  }
+  return findings;
+}
+
+// For each finding's cited file:line, grep the actual file to verify
+// the line exists. Returns total + verified counts; per-finding metadata
+// is appended into the evidence array so the reader can see which
+// citations were verified.
+async function computeGrounding(findings: Finding[]): Promise<{ total: number; verified: number; rate: number }> {
+  // readFile (async) instead of readFileSync — caught 2026-04-27 by
+  // Kimi's self-audit. Sync I/O in an async fn blocks the event loop
+  // for every cited file; doesn't matter at 10 findings, would matter
+  // at 100+.
+  const checks = await Promise.all(findings.map(async (f) => {
+    const cite = f.evidence[0] ?? "";
+    const m = /^(\S+?):(\d+)/.exec(cite);
+    if (!m) return false;
+    const [, relpath, lineStr] = m;
+    const line = Number(lineStr);
+    if (!line || !relpath) return false;
+
+    // Path-traversal guard, two-layer (caught 2026-04-27 by Kimi
+    // self-audits on dd77632 then 2d9cb12).
+    //
+    // Layer 1 (lexical): resolve() normalizes `..` segments. Refuse
+    // any path that doesn't anchor under REPO_ROOT.
+    //
+    // Layer 2 (symlink): even if the lexical path is anchored, it
+    // could be a symlink whose target escapes. realpath() resolves
+    // symlinks; compare the real path against REPO_ROOT_REAL.
+    //
+    // Both layers exist because attackers might bypass either alone:
+    // raw `../etc/passwd` triggers layer 1; a planted symlink at
+    // ./safe-looking-name → /etc/passwd triggers layer 2.
+    const abs = resolve(REPO_ROOT, relpath);
+    if (!abs.startsWith(REPO_ROOT + "/") && abs !== REPO_ROOT) {
+      f.evidence.push(`[grounding: path escapes repo root, refusing]`);
+      return false;
+    }
+
+    if (!existsSync(abs)) {
+      f.evidence.push("[grounding: file not found]");
+      return false;
+    }
+    try {
+      // Symlink-resolution check before any read. realpath() throws
+      // if the file doesn't exist; existsSync above shields the
+      // common case but a TOCTOU race could still error here — the
+      // outer catch handles it.
+      const realPath = await realpath(abs);
+      if (!realPath.startsWith(REPO_ROOT_REAL + "/") && realPath !== REPO_ROOT_REAL) {
+        f.evidence.push(`[grounding: symlink target escapes repo root, refusing]`);
+        return false;
+      }
+      const lines = (await readFile(realPath, "utf8")).split("\n");
+      if (line < 1 || line > lines.length) {
+        f.evidence.push(`[grounding: line ${line} > EOF (${lines.length})]`);
+        return false;
+      }
+      f.evidence.push(`[grounding: verified at ${relpath}:${line}]`);
+      return true;
+    } catch (e) {
+      f.evidence.push(`[grounding: read failed: ${(e as Error).message.slice(0, 80)}]`);
+      return false;
+    }
+  }));
+  const verified = checks.filter(Boolean).length;
+  const total = findings.length;
+  return { total, verified, rate: total === 0 ? 0 : verified / total };
+}
+
+async function persistVerdict(path: string, v: KimiVerdictFile): Promise<void> {
+  await mkdir(KIMI_VERDICTS_DIR, { recursive: true });
+  await writeFile(path, JSON.stringify(v, null, 2));
+}
+
+async function appendMetrics(v: KimiVerdictFile): Promise<void> {
+  // dirname() instead of join(path, "..") — caught 2026-04-27 by both
+  // Haiku and Opus self-audits. The "/.." idiom resolves correctly
+  // via Node path normalization but is non-idiomatic + breaks if the
+  // path ever has trailing dots.
+  await mkdir(dirname(KIMI_AUDITS_JSONL), { recursive: true });
+  await appendFile(KIMI_AUDITS_JSONL, JSON.stringify({
+    pr_number: v.pr_number,
+    head_sha: v.head_sha,
+    audited_at: v.cached_at,
+    model: v.model,
+    latency_ms: v.latency_ms,
+    finish_reason: v.finish_reason,
+    prompt_tokens: v.usage.prompt_tokens,
+    completion_tokens: v.usage.completion_tokens,
+    findings_total: v.findings.length,
+    findings_block: v.findings.filter(f => f.severity === "block").length,
+    findings_warn: v.findings.filter(f => f.severity === "warn").length,
+    grounding_verified: v.grounding.verified,
+    grounding_rate: Number(v.grounding.rate.toFixed(3)),
+  }) + "\n");
+}
+
+function skipFinding(why: string): Finding {
+  return {
+    check: "kimi_architect" as CheckKind,
+    severity: "info",
+    summary: `kimi_architect skipped — ${why}`,
+    evidence: [why],
+  };
+}

auditor/checks/static.ts

@@ -77,6 +77,17 @@ export function runStaticCheck(diff: string): Finding[] {
       // Strip the diff prefix (' ' for context, '+' for added).
       const body = (isAdded || line.startsWith(" ")) ? line.slice(1) : line;
 
+      // Compute the file-level backtick state ENTERING this line.
+      // The state machine sees pattern matches against the right
+      // context: a line that opens a backtick block has its own
+      // pattern checks evaluated under "inside-backtick" semantics
+      // for the portion AFTER the opening tick. Pre-2026-04-27 the
+      // state was updated AFTER the pattern checks, so the FIRST
+      // pattern on a backtick-opening line slipped through with
+      // stale "outside-backtick" semantics. Caught by Kimi self-audit.
+      const stateAtLineStart = inMultilineBacktick;
+      const stateAtLineEnd = updateBacktickState(body, stateAtLineStart);
+
       if (isAdded) {
         const added = body;
 
@@ -84,11 +95,13 @@ export function runStaticCheck(diff: string): Finding[] {
           for (const { re, why } of BLOCK_PATTERNS) {
             const m = added.match(re);
             if (m && typeof m.index === "number") {
-              // Skip if the match sits inside a quoted string literal —
-              // this is how rubric files (tests/real-world/*, prompt
-              // templates) legitimately reference the patterns they
-              // guard against, without actually executing them.
-              if (inMultilineBacktick || isInsideQuotedString(added, m.index)) continue;
+              // Skip if EITHER (a) the file was already inside a
+              // multi-line backtick block when this line started, OR
+              // (b) the match sits inside a quoted string literal on
+              // THIS line. The earlier code only checked stateAtLineStart;
+              // now we also check that the match isn't past the
+              // opening backtick of a block that opens on this line.
+              if (stateAtLineStart || isInsideQuotedString(added, m.index)) continue;
               findings.push({
                 check: "static",
                 severity: "block",
@@ -120,13 +133,8 @@ export function runStaticCheck(diff: string): Finding[] {
         }
       }
 
-      // Update file-level multi-line backtick state by walking THIS
-      // line's unescaped backticks. Both context and added lines
-      // contribute (they're both in the post-merge file). Doc-comment
-      // backticks like `\\\`Foo\\\`` count too — that's the source of
-      // the original bug, where multi-line template literals contained
-      // `todo!()` references.
-      inMultilineBacktick = updateBacktickState(body, inMultilineBacktick);
+      // Carry the end-of-line state forward to the next iteration.
+      inMultilineBacktick = stateAtLineEnd;
     }
 
     // "Field added but never read" heuristic — catches exactly the
@@ -213,6 +221,13 @@ function extractNewFieldsWithLine(lines: string[]): Array<{ name: string; lineId
 // Stops the struct-search early if we hit a `}` at zero indent
 // (the previous scope) or another `pub struct` (we left ours).
 function parentStructHasSerdeDerive(lines: string[], fieldLineIdx: number): boolean {
+  // Bounds-check fieldLineIdx (caught 2026-04-27 by Kimi self-audit).
+  // Pre-fix: if fieldLineIdx >= lines.length, the loop ran from a
+  // negative implicit upper bound (fieldLineIdx - 80 could be > 0
+  // even when fieldLineIdx is past EOF) and read undefined slots.
+  // Defensive: bail early on out-of-range input.
+  if (fieldLineIdx < 0 || fieldLineIdx >= lines.length) return false;
+
   let structLineIdx = -1;
   for (let i = fieldLineIdx - 1; i >= 0 && i >= fieldLineIdx - 80; i--) {
     const raw = lines[i];

auditor/gitea.ts

@@ -9,28 +9,42 @@
 import { readFile } from "node:fs/promises";
 import type { PrSnapshot } from "./types.ts";
 
-const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
+// 2026-09-02: repo home moved to git.warren.help in the VPS
+// consolidation; git.agentview.dev now dead-ends at a Traefik default
+// cert + 404 (auditor had been failing listOpenPrs every cycle since
+// Aug 25). Credentials come from ~/.git-credentials for HOST's domain,
+// falling back to the user:pass embedded in this repo's origin remote.
+// Basic auth works for both a PAT and a password on Gitea.
+const HOST = process.env.GITEA_HOST ?? "https://git.warren.help";
 const OWNER = "profit";
 const REPO = "lakehouse";
 const CRED_FILE = "/home/profit/.git-credentials";
+const GIT_CONFIG = "/home/profit/lakehouse/.git/config";
 
-let cachedPat: string | null = null;
+let cachedAuth: string | null = null;
 
-async function getPat(): Promise<string> {
-  if (cachedPat) return cachedPat;
-  const raw = await readFile(CRED_FILE, "utf8");
+async function getAuthHeader(): Promise<string> {
+  if (cachedAuth) return cachedAuth;
+  const hostDomain = new URL(HOST).host.replace(/\./g, "\\.");
+  const credRe = new RegExp(`^https://([^:]+):([^@]+)@${hostDomain}`);
+  for (const file of [CRED_FILE, GIT_CONFIG]) {
+    let raw: string;
+    try { raw = await readFile(file, "utf8"); } catch { continue; }
     for (const line of raw.split("\n")) {
-    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
-    if (m) { cachedPat = m[1]; return m[1]; }
+      const m = line.trim().replace(/^url = /, "").match(credRe);
+      if (m) {
+        cachedAuth = `Basic ${Buffer.from(`${decodeURIComponent(m[1])}:${decodeURIComponent(m[2])}`).toString("base64")}`;
+        return cachedAuth;
       }
-  throw new Error(`no Gitea PAT in ${CRED_FILE}`);
+    }
+  }
+  throw new Error(`no Gitea credentials for ${HOST} in ${CRED_FILE} or ${GIT_CONFIG}`);
 }
 
 async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
-  const pat = await getPat();
   const url = `${HOST}/api/v1${path}`;
   const headers = new Headers(init.headers);
-  headers.set("Authorization", `token ${pat}`);
+  headers.set("Authorization", await getAuthHeader());
   if (init.body && !headers.has("content-type")) {
     headers.set("content-type", "application/json");
   }

auditor/index.ts

@@ -24,14 +24,30 @@ const POLL_INTERVAL_MS = 90_000; // 90s — enough budget for audit runs to comp
 const PAUSE_FILE = "/home/profit/lakehouse/auditor.paused";
 const STATE_FILE = "/home/profit/lakehouse/data/_auditor/state.json";
 
+// Per-PR audit cap. Prevents the daemon from running away on a PR
+// when each push surfaces new findings — operator wants to review
+// in batch, not have the daemon burn budget while they're away.
+// Default 3 audits per PR. Override via LH_AUDITOR_MAX_AUDITS_PER_PR.
+// Set to 0 to disable the cap.
+//
+// Reset (after manual review): edit data/_auditor/state.json and
+// set audit_count_per_pr.<N> = 0 (or delete the key). Daemon picks
+// up the change on the next cycle without restart.
+const MAX_AUDITS_PER_PR = Number(process.env.LH_AUDITOR_MAX_AUDITS_PER_PR) || 3;
+
 interface State {
   // Map: PR number → last-audited head SHA. Lets us dedupe audits
   // across restarts (poller can crash/restart without re-auditing
   // all open PRs from scratch).
   last_audited: Record<string, string>;
+  // Map: PR number → number of audits run on that PR since last reset.
+  // Daemon halts auditing a PR once this hits MAX_AUDITS_PER_PR.
+  // Operator clears the entry to resume.
+  audit_count_per_pr: Record<string, number>;
   started_at: string;
   cycles_total: number;
   cycles_skipped_paused: number;
+  cycles_skipped_capped: number;
   audits_run: number;
   last_cycle_at?: string;
 }
@@ -47,17 +63,21 @@ async function loadState(): Promise<State> {
     return {
       last_audited: s.last_audited ?? {},
       started_at: s.started_at ?? new Date().toISOString(),
+      audit_count_per_pr: s.audit_count_per_pr ?? {},
       cycles_total: s.cycles_total ?? 0,
       cycles_skipped_paused: s.cycles_skipped_paused ?? 0,
+      cycles_skipped_capped: s.cycles_skipped_capped ?? 0,
       audits_run: s.audits_run ?? 0,
       last_cycle_at: s.last_cycle_at,
     };
   } catch {
     return {
       last_audited: {},
+      audit_count_per_pr: {},
       started_at: new Date().toISOString(),
       cycles_total: 0,
       cycles_skipped_paused: 0,
+      cycles_skipped_capped: 0,
       audits_run: 0,
     };
   }
@@ -89,12 +109,38 @@ async function runCycle(state: State): Promise<State> {
   console.log(`[auditor] cycle ${state.cycles_total}: ${prs.length} open PR(s)`);
 
   for (const pr of prs) {
-    const last = state.last_audited[String(pr.number)];
+    const prKey = String(pr.number);
+    const last = state.last_audited[prKey];
     if (last === pr.head_sha) {
       console.log(`[auditor]   skip PR #${pr.number} (SHA ${pr.head_sha.slice(0, 8)} already audited)`);
       continue;
     }
-    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)}`);
+    // Per-head-SHA audit cap. Each new push gets MAX_AUDITS_PER_PR
+    // fresh attempts; the counter auto-resets when the head SHA
+    // changes. Operator only intervenes manually if a single SHA
+    // somehow needs MORE than the cap (rare — usually transient
+    // upstream errors clear themselves inside 3 attempts).
+    //
+    // Reset rule: if `last` exists (we've seen this PR before) AND
+    // pr.head_sha != last, that's a new push. Drop the counter.
+    // The dedup branch above already handles same-SHA → skip, so
+    // we only land here when the SHA actually moved.
+    if (last !== undefined && (state.audit_count_per_pr[prKey] ?? 0) > 0) {
+      const prior_count = state.audit_count_per_pr[prKey];
+      console.log(`[auditor]   PR #${pr.number} new head ${pr.head_sha.slice(0, 8)} (prior ${last.slice(0, 8)}, was ${prior_count}/${MAX_AUDITS_PER_PR}) — resetting cap counter`);
+      state.audit_count_per_pr[prKey] = 0;
+    }
+    const auditedSoFar = state.audit_count_per_pr[prKey] ?? 0;
+    if (MAX_AUDITS_PER_PR > 0 && auditedSoFar >= MAX_AUDITS_PER_PR) {
+      // This branch only fires now if the SAME head SHA somehow
+      // burned MAX audits (transient upstream errors retried that
+      // many times). Operator can clear state.audit_count_per_pr.<N>
+      // = 0 to force one more attempt; otherwise wait for next push.
+      console.log(`[auditor]   skip PR #${pr.number} (same head ${pr.head_sha.slice(0, 8)} burned ${auditedSoFar}/${MAX_AUDITS_PER_PR} — push new code or clear state.json audit_count_per_pr.${prKey})`);
+      state.cycles_skipped_capped += 1;
+      continue;
+    }
+    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)} [${auditedSoFar + 1}/${MAX_AUDITS_PER_PR}]`);
     try {
       // Skip dynamic by default: it mutates live playbook state and
       // re-runs on every PR update would pollute quickly. Operator
@@ -106,8 +152,22 @@ async function runCycle(state: State): Promise<State> {
         skip_inference: process.env.LH_AUDITOR_SKIP_INFERENCE === "1",
       });
       console.log(`[auditor]     verdict=${verdict.overall} findings=${verdict.metrics.findings_total} (block=${verdict.metrics.findings_block} warn=${verdict.metrics.findings_warn})`);
-      state.last_audited[String(pr.number)] = pr.head_sha;
+      state.last_audited[prKey] = pr.head_sha;
+      state.audit_count_per_pr[prKey] = auditedSoFar + 1;
       state.audits_run += 1;
+      if (state.audit_count_per_pr[prKey] >= MAX_AUDITS_PER_PR) {
+        console.log(`[auditor]     PR #${pr.number} reached cap (${MAX_AUDITS_PER_PR} audits) — daemon will skip further audits until reset`);
+      }
+      // Persist state immediately after each successful audit so the
+      // increment survives a crash. Pre-2026-04-27 the cycle saved
+      // once at the end (main.ts:140), which lost the count if the
+      // daemon was killed mid-cycle. Fix lifted from kimi_architect's
+      // own audit on this very file. saveState is idempotent + cheap
+      // (one JSON write), so per-audit cost is negligible.
+      try { await saveState(state); }
+      catch (e) {
+        console.error(`[auditor]     saveState mid-cycle failed: ${(e as Error).message} — count held in memory`);
+      }
     } catch (e) {
       console.error(`[auditor]     audit failed: ${(e as Error).message}`);
     }

auditor/schemas/distillation/drift_report.ts

@@ -15,7 +15,7 @@ import {
 } from "./types";
 import type { StageName } from "./stage_receipt";
 
-export const DRIFT_REPORT_SCHEMA_VERSION = 1;
+export const DRIFT_REPORT_SCHEMA_VERSION = 2;
 export const DRIFT_THRESHOLD_PCT = 0.20;
 
 export type DriftSeverity = "ok" | "warn" | "alert";
@@ -27,7 +27,11 @@ export interface StageDrift {
   delta_accepted: number;
   delta_quarantined: number;
   pct_change_out: number | null;  // null when prior had 0 records
-  input_hash_match: boolean;
+  // null when input_hash isn't materialized into the stage summary —
+  // schema v1 lied and reported `true` here. v2 is honest: callers
+  // that want determinism enforcement must read the full StageReceipt
+  // off disk and compute input_hash equality there.
+  input_hash_match: boolean | null;
   output_hash_match: boolean;
   // alert if input_hash matches but output_hash diverges
   deterministic_violation: boolean;

auditor/schemas/distillation/evidence_record.ts

@@ -121,6 +121,14 @@ export interface EvidenceRecord {
   // and have no text payload. Present for distilled_*, contract_analyses,
   // mode_experiments, scrum_reviews etc.
   text?: string;
+
+  // ── Domain-specific metadata bucket ──
+  // Source-specific fields that don't earn a top-level slot. e.g.
+  // contract_analyses rows carry `contractor` here; mode_experiments
+  // could carry `corpus_set`. Typed scalar values only — keep this
+  // small or it becomes a junk drawer. Added 2026-04-27 (Kimi audit
+  // flagged `(ev as any).contractor` schema bypass at export_sft.ts:126).
+  metadata?: Record<string, string | number | boolean>;
 }
 
 export function validateEvidenceRecord(input: unknown): ValidationResult<EvidenceRecord> {

auditor/types.ts

@@ -2,7 +2,7 @@
 // if something can't be verified from a check, it goes into `evidence`
 // so the verdict is inspectable, not a black box.
 
-export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
+export type CheckKind = "static" | "dynamic" | "inference" | "kb_query" | "kimi_architect";
 
 export type Severity = "info" | "warn" | "block";
 

bot/propose.ts

@@ -13,10 +13,17 @@ import { readFile } from "node:fs/promises";
 import { createHash } from "node:crypto";
 import type { Gap, Proposal } from "./types.ts";
 
-const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
+// Phase 44 migration (2026-04-27): bot/propose.ts now flows through
+// the gateway's /v1/chat instead of hitting the sidecar's /generate
+// directly. /v1/usage tracks the call, Langfuse traces it, observer
+// sees it. Gateway owns the routing.
+//
+// 2026-04-28: gpt-oss:120b → deepseek-v3.2 via Ollama Pro. Newer
+// DeepSeek revision, faster, still on the same OLLAMA_CLOUD_KEY.
+const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
 const REPO_ROOT = "/home/profit/lakehouse";
 const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
-const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
+const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "deepseek-v3.2";
 const MAX_TOKENS = 6000;
 
 export async function findGaps(): Promise<Gap[]> {
@@ -72,13 +79,16 @@ export async function generateProposal(gap: Gap, historySummary: string = ""): P
   sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
   const userPrompt = sections.join("\n");
 
-  const r = await fetch(`${SIDECAR_URL}/generate`, {
+  const r = await fetch(`${GATEWAY_URL}/v1/chat`, {
     method: "POST",
     headers: { "content-type": "application/json" },
     body: JSON.stringify({
       model: CLOUD_MODEL,
-      system: SYSTEM_PROMPT,
-      prompt: userPrompt,
+      provider: "ollama_cloud",
+      messages: [
+        { role: "system", content: SYSTEM_PROMPT },
+        { role: "user", content: userPrompt },
+      ],
       temperature: 0.2,
       max_tokens: MAX_TOKENS,
       think: false,
@@ -86,10 +96,10 @@ export async function generateProposal(gap: Gap, historySummary: string = ""): P
     signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
   });
   if (!r.ok) {
-    throw new Error(`sidecar ${r.status}: ${await r.text()}`);
+    throw new Error(`gateway /v1/chat ${r.status}: ${await r.text()}`);
   }
   const j = await r.json() as any;
-  const raw: string = j.text ?? j.response ?? "";
+  const raw: string = j?.choices?.[0]?.message?.content ?? "";
   const usage = j.usage ?? {};
   const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
 

config/modes.toml

@@ -40,11 +40,13 @@ matrix_corpus = "chicago_permits_v1"
 name = "staffing_inference"
 # Staffing-domain native enrichment runner — Pass 4 (2026-04-26).
 # Same composer architecture as codereview_lakehouse but with staffing
-# framing + workers corpus. Validates that the modes-as-prompt-molders
-# pattern generalizes beyond code review.
+# framing + workers corpus.
 preferred_mode = "staffing_inference_lakehouse"
 fallback_modes = ["ladder", "consensus", "pipeline"]
-default_model = "openai/gpt-oss-120b:free"
+# 2026-05-03: REVERTED to local. PRD line 70 — everything runs locally,
+# no cloud APIs on the customer hot path. Cloud models stay available
+# in providers.toml for explicit dev-tool opt-in (scrum, auditor).
+default_model = "qwen3.5:latest"
 matrix_corpus = "workers_500k_v8"
 
 [[task_class]]
@@ -58,7 +60,8 @@ matrix_corpus = "kb_team_runs_v1"
 name = "doc_drift_check"
 preferred_mode = "drift"
 fallback_modes = ["validator"]
-default_model = "gpt-oss:120b"
+# 2026-05-03: REVERTED to local per PRD line 70.
+default_model = "qwen3.5:latest"
 matrix_corpus = "distilled_factual_v20260423095819"
 
 [[task_class]]

config/providers.toml

@@ -15,22 +15,29 @@
 
 [[provider]]
 name = "ollama"
-base_url = "http://localhost:3200"
+base_url = "http://localhost:11434"
 auth = "none"
 default_model = "qwen3.5:latest"
-# Hot-path local inference. No bearer needed — Python sidecar on
-# localhost handles the Ollama API. Model names are bare
-# (e.g. "qwen3.5:latest", not "ollama/qwen3.5:latest").
+# Hot-path local inference. No bearer needed — direct to Ollama as of
+# 2026-05-02 (Python sidecar's pass-through wrapper retired). Model
+# names are bare (e.g. "qwen3.5:latest", not "ollama/qwen3.5:latest").
 
 [[provider]]
 name = "ollama_cloud"
 base_url = "https://ollama.com"
 auth = "bearer"
 auth_env = "OLLAMA_CLOUD_KEY"
-default_model = "gpt-oss:120b"
-# Cloud-tier Ollama. Key resolved from OLLAMA_CLOUD_KEY env at gateway
-# boot. Model-prefix routing: "cloud/<model>" auto-routes here
-# (see gateway::v1::resolve_provider).
+default_model = "deepseek-v3.2"
+# Cloud-tier Ollama (Pro plan as of 2026-04-28). Key resolved from
+# OLLAMA_CLOUD_KEY at gateway boot; Pro tier upgraded the account so
+# rate limits + model access widen without a key change. Model-prefix
+# routing: "cloud/<model>" auto-routes here. 39-model fleet now
+# includes deepseek-v3.2, deepseek-v4-{flash,pro}, gemini-3-flash-
+# preview, glm-{5,5.1}, kimi-k2.6, qwen3-coder-next.
+# 2026-04-28: default upgraded gpt-oss:120b → deepseek-v3.2 (newest
+# DeepSeek revision). NOTE: kimi-k2:1t is upstream-broken (HTTP 500
+# on Ollama Pro probe 2026-04-28) — do not route to it. Use kimi-k2.6
+# instead, which is what staffing_inference points at.
 
 [[provider]]
 name = "openrouter"
@@ -38,13 +45,50 @@ base_url = "https://openrouter.ai/api/v1"
 auth = "bearer"
 auth_env = "OPENROUTER_API_KEY"
 auth_fallback_files = ["/home/profit/.env", "/root/llm_team_config.json"]
-default_model = "openai/gpt-oss-120b:free"
+default_model = "x-ai/grok-4.1-fast"
 # Multi-provider gateway. Covers Anthropic, Google, OpenAI, MiniMax,
 # Qwen, Gemma, etc. Key resolved via crates/gateway/src/v1/openrouter.rs
 # resolve_openrouter_key() — env first, then fallback files.
 # Model-prefix routing: "openrouter/<vendor>/<model>" auto-routes here,
 # prefix stripped before upstream call.
 
+[[provider]]
+name = "opencode"
+base_url = "https://opencode.ai/zen/v1"
+# Unified endpoint — covers BOTH Zen (pay-per-token Anthropic/OpenAI/
+# Gemini frontier) AND Go (flat-sub Kimi/GLM/DeepSeek/Qwen/Minimax).
+# Upstream bills per-model: Zen models hit Zen balance, Go models hit
+# Go subscription cap. /zen/go/v1 is the Go-only sub-path (rejects
+# Zen models), kept for reference but not used by this provider.
+auth = "bearer"
+auth_env = "OPENCODE_API_KEY"
+default_model = "claude-opus-4-7"
+# OpenCode (Zen + GO unified endpoint). One sk-* key reaches Claude
+# Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+# Qwen, plus 4 free-tier models. OpenAI-compatible Chat Completions
+# at /v1/chat/completions. Model-prefix routing: "opencode/<name>"
+# auto-routes here, prefix stripped before upstream call.
+# Key file: /etc/lakehouse/opencode.env (loaded via systemd EnvironmentFile).
+# Model catalog: curl -H "Authorization: Bearer ..." https://opencode.ai/zen/v1/models
+# Note: /zen/go/v1 is the GO-only sub-path (Kimi/GLM/DeepSeek tier);
+# /zen/v1 covers everything including Anthropic (which /zen/go/v1 rejects).
+
+[[provider]]
+name = "kimi"
+base_url = "https://api.kimi.com/coding/v1"
+auth = "bearer"
+auth_env = "KIMI_API_KEY"
+default_model = "kimi-for-coding"
+# Direct Kimi For Coding provider. `api.kimi.com` is a SEPARATE account
+# system from `api.moonshot.ai` and `api.moonshot.cn` — keys are NOT
+# interchangeable. Used as a fallback when Ollama Cloud's kimi-k2.6 is
+# unavailable and OpenRouter's `moonshotai/kimi-k2.6` is rate-limited.
+# (Was `kimi-k2:1t` here pre-2026-05-03 — that model is upstream-broken
+# and removed from operator guidance.)
+# Model id: `kimi-for-coding` (kimi-k2.6 underneath).
+# Key file: /etc/lakehouse/kimi.env (loaded via systemd EnvironmentFile).
+# Model-prefix routing: "kimi/<model>" auto-routes here, prefix stripped.
+
 # Planned (Phase 40 long-horizon — adapters not yet shipped):
 #
 # [[provider]]

crates/aibridge/Cargo.toml

@@ -12,3 +12,4 @@ serde_json = { workspace = true }
 tracing = { workspace = true }
 reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
 async-trait = "0.1"
+lru = "0.12"

crates/aibridge/src/client.rs

@@ -1,12 +1,74 @@
+use lru::LruCache;
 use reqwest::Client;
 use serde::{Deserialize, Serialize};
+use std::num::NonZeroUsize;
+use std::sync::Mutex;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Arc;
 use std::time::Duration;
 
-/// HTTP client for the Python AI sidecar.
+/// HTTP client for Ollama (post-2026-05-02 — sidecar dropped).
+///
+/// `base_url` was historically the Python sidecar at `:3200`, which
+/// pass-through-proxied to Ollama at `:11434`. The sidecar added zero
+/// logic on the hot path (embed.py + generate.py + rerank.py +
+/// admin.py = ~120 LOC of pure Ollama wrappers), so this client now
+/// talks to Ollama directly and the sidecar process can be retired.
+///
+/// What stayed Python: `lab_ui.py` + `pipeline_lab.py` (~888 LOC of
+/// dev-mode Streamlit-shape UIs) — those aren't on the runtime hot
+/// path and continue running for prompt experimentation.
+///
+/// `generate()` has two transport modes:
+/// - When `gateway_url` is None (default), posts directly to Ollama's
+///   `${base_url}/api/generate`.
+/// - When `gateway_url` is `Some(url)`, posts to `${url}/v1/chat`
+///   with `provider="ollama"` so the call appears in `/v1/usage` and
+///   Langfuse traces.
+///
+/// `embed()`, `rerank()`, and admin methods always go direct to
+/// Ollama — no `/v1` equivalent for those surfaces yet.
+///
+/// Phase 44 part 2 (2026-04-27): the gateway URL is wired in by
+/// callers that want observability (vectord modules); it's left
+/// unset by callers that ARE the gateway internals (avoids self-loops
+/// + redundant hops).
+/// Per-text embed cache key. We key on (model, text) so different
+/// model selections produce distinct cache lines — a query embedded
+/// under nomic-embed-text-v2-moe must NOT collide with the same
+/// query under nomic-embed-text v1.
+#[derive(Eq, PartialEq, Hash, Clone)]
+struct EmbedCacheKey {
+    model: String,
+    text: String,
+}
+
+/// Default LRU cache size — 4096 entries × ~6KB per 768-d f64
+/// vector ≈ 24MB. Sized for typical staffing-domain repetition
+/// (coordinator workflows have query repetition rates around 70-90%
+/// per session). Tunable via [aibridge].embed_cache_size in the
+/// config; 0 disables the cache entirely.
+const DEFAULT_EMBED_CACHE_SIZE: usize = 4096;
+
 #[derive(Clone)]
 pub struct AiClient {
     client: Client,
     base_url: String,
+    gateway_url: Option<String>,
+    /// Closes the 63× perf gap with Go side. Mirrors the shape of
+    /// Go's internal/embed/cached.go::CachedProvider — same
+    /// (model, text) → vector caching, same nil-disable semantics.
+    /// None = caching disabled (cache_size=0); Some = bounded LRU.
+    embed_cache: Option<Arc<Mutex<LruCache<EmbedCacheKey, Vec<f64>>>>>,
+    /// Hit / miss counters for /admin observability + load-test
+    /// validation. Atomic so Clone'd AiClients share the same counts.
+    embed_cache_hits: Arc<AtomicU64>,
+    embed_cache_misses: Arc<AtomicU64>,
+    /// Pinned at construction time so the EmbedResponse can carry
+    /// dimension consistently even when every text was a cache hit
+    /// (no fresh upstream call to learn the dim from). Set on first
+    /// successful Ollama embed; checked on every cache hit.
+    cached_dim: Arc<AtomicU64>,
 }
 
 // -- Request/Response types --
@@ -79,68 +141,386 @@ pub struct RerankResponse {
 
 impl AiClient {
     pub fn new(base_url: &str) -> Self {
+        Self::with_embed_cache(base_url, DEFAULT_EMBED_CACHE_SIZE)
+    }
+
+    /// Constructs an AiClient with an explicit embed-cache size.
+    /// Pass 0 to disable the cache entirely (matches Go-side
+    /// CachedProvider's nil-cache semantics).
+    pub fn with_embed_cache(base_url: &str, cache_size: usize) -> Self {
         let client = Client::builder()
             .timeout(Duration::from_secs(120))
             .build()
             .expect("failed to build HTTP client");
+        let embed_cache = if cache_size > 0 {
+            // SAFETY: cache_size > 0 just verified, NonZeroUsize::new
+            // returns Some.
+            let cap = NonZeroUsize::new(cache_size).expect("cache_size > 0");
+            Some(Arc::new(Mutex::new(LruCache::new(cap))))
+        } else {
+            None
+        };
         Self {
             client,
             base_url: base_url.trim_end_matches('/').to_string(),
+            gateway_url: None,
+            embed_cache,
+            embed_cache_hits: Arc::new(AtomicU64::new(0)),
+            embed_cache_misses: Arc::new(AtomicU64::new(0)),
+            cached_dim: Arc::new(AtomicU64::new(0)),
         }
     }
 
+    /// Cache hit/miss/size snapshot. Useful for /admin endpoints +
+    /// load-test validation ("did the cache fire as expected?").
+    pub fn embed_cache_stats(&self) -> (u64, u64, usize) {
+        let hits = self.embed_cache_hits.load(Ordering::Relaxed);
+        let misses = self.embed_cache_misses.load(Ordering::Relaxed);
+        let len = self
+            .embed_cache
+            .as_ref()
+            .map(|c| c.lock().map(|g| g.len()).unwrap_or(0))
+            .unwrap_or(0);
+        (hits, misses, len)
+    }
+
+    /// Same as `new`, but every `generate()` is routed through
+    /// `${gateway_url}/v1/chat` (provider=ollama) for observability.
+    /// Use this for callers OUTSIDE the gateway. Inside the gateway
+    /// itself, prefer `new()` — calling /v1/chat from /v1/chat works
+    /// (no infinite loop, ollama_arm doesn't use AiClient) but adds
+    /// a wasted localhost hop.
+    pub fn new_with_gateway(base_url: &str, gateway_url: &str) -> Self {
+        let mut c = Self::new(base_url);
+        c.gateway_url = Some(gateway_url.trim_end_matches('/').to_string());
+        c
+    }
+
+    /// Reachability + version check. Hits Ollama's `/api/version`,
+    /// returns a sidecar-shaped envelope so callers reading
+    /// `.status` / `.ollama_url` don't break across the
+    /// pre-/post-2026-05-02 cutover.
     pub async fn health(&self) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .get(format!("{}/health", self.base_url))
+            .get(format!("{}/api/version", self.base_url))
             .send()
             .await
-            .map_err(|e| format!("sidecar unreachable: {e}"))?;
-        resp.json().await.map_err(|e| format!("invalid response: {e}"))
+            .map_err(|e| format!("ollama unreachable: {e}"))?;
+        let body: serde_json::Value = resp.json().await
+            .map_err(|e| format!("invalid response: {e}"))?;
+        Ok(serde_json::json!({
+            "status": "ok",
+            "ollama_url": &self.base_url,
+            "ollama_version": body.get("version"),
+        }))
     }
 
+    /// Embed with per-text LRU caching. Mirrors Go-side
+    /// CachedProvider behavior: cache key is (model, text);
+    /// cache-hit texts skip the sidecar; cache-miss texts batch
+    /// into a single sidecar call; results are interleaved in the
+    /// caller's input order.
+    ///
+    /// Closes ~95% of the load-test perf gap vs Go side (loadgen
+    /// 2026-05-01: Rust 128 RPS → with cache ≥ 7000 RPS expected
+    /// for warm-cache workloads). Cold-cache behavior unchanged
+    /// (every text is a miss → single sidecar call, identical to
+    /// pre-cache).
     pub async fn embed(&self, req: EmbedRequest) -> Result<EmbedResponse, String> {
+        let model_key = req.model.clone().unwrap_or_default();
+
+        // Fast path: cache disabled → original behavior.
+        let Some(cache) = self.embed_cache.as_ref() else {
+            return self.embed_uncached(&req).await;
+        };
+        if req.texts.is_empty() {
+            return self.embed_uncached(&req).await;
+        }
+
+        // First pass: check cache for each text. Track which positions
+        // need a sidecar fetch.
+        let mut embeddings: Vec<Option<Vec<f64>>> = vec![None; req.texts.len()];
+        let mut miss_indices: Vec<usize> = Vec::new();
+        let mut miss_texts: Vec<String> = Vec::new();
+        {
+            let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
+            for (i, text) in req.texts.iter().enumerate() {
+                let key = EmbedCacheKey { model: model_key.clone(), text: text.clone() };
+                if let Some(vec) = guard.get(&key) {
+                    embeddings[i] = Some(vec.clone());
+                    self.embed_cache_hits.fetch_add(1, Ordering::Relaxed);
+                } else {
+                    miss_indices.push(i);
+                    miss_texts.push(text.clone());
+                    self.embed_cache_misses.fetch_add(1, Ordering::Relaxed);
+                }
+            }
+        }
+
+        // All hit? Return immediately. Use cached_dim to populate
+        // the response dimension (no sidecar to ask).
+        if miss_indices.is_empty() {
+            let dim = self.cached_dim.load(Ordering::Relaxed) as usize;
+            let dim = if dim == 0 { embeddings[0].as_ref().map(|v| v.len()).unwrap_or(0) } else { dim };
+            return Ok(EmbedResponse {
+                embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
+                model: req.model.unwrap_or_else(|| "nomic-embed-text".to_string()),
+                dimensions: dim,
+            });
+        }
+
+        // Second pass: fetch the misses in one sidecar call.
+        let miss_req = EmbedRequest { texts: miss_texts.clone(), model: req.model.clone() };
+        let resp = self.embed_uncached(&miss_req).await?;
+        if resp.embeddings.len() != miss_texts.len() {
+            return Err(format!(
+                "embed cache: sidecar returned {} embeddings for {} texts",
+                resp.embeddings.len(),
+                miss_texts.len()
+            ));
+        }
+
+        // Pin cached_dim on first successful response.
+        if resp.dimensions > 0 {
+            self.cached_dim.store(resp.dimensions as u64, Ordering::Relaxed);
+        }
+
+        // Insert misses into cache + fill response slots.
+        {
+            let mut guard = cache.lock().map_err(|e| format!("cache lock poisoned: {e}"))?;
+            for (j, idx) in miss_indices.iter().enumerate() {
+                let key = EmbedCacheKey {
+                    model: model_key.clone(),
+                    text: miss_texts[j].clone(),
+                };
+                let vec = resp.embeddings[j].clone();
+                guard.put(key, vec.clone());
+                embeddings[*idx] = Some(vec);
+            }
+        }
+
+        Ok(EmbedResponse {
+            embeddings: embeddings.into_iter().map(|opt| opt.expect("filled")).collect(),
+            model: resp.model,
+            dimensions: resp.dimensions,
+        })
+    }
+
+    /// Direct Ollama call — used internally by embed() for cache-miss
+    /// batches and as the transparent fallback when the cache is
+    /// disabled. Loops per-text against `${base_url}/api/embed`,
+    /// matching the sidecar's pre-2026-05-02 behavior. Ollama 0.4+
+    /// supports batch input but per-text keeps compatibility broader
+    /// + lets cache-miss-only batches share the loop with cold runs.
+    async fn embed_uncached(&self, req: &EmbedRequest) -> Result<EmbedResponse, String> {
+        let model = req.model.clone().unwrap_or_else(|| "nomic-embed-text".to_string());
+        let mut embeddings: Vec<Vec<f64>> = Vec::with_capacity(req.texts.len());
+
+        for text in &req.texts {
             let resp = self.client
-            .post(format!("{}/embed", self.base_url))
-            .json(&req)
+                .post(format!("{}/api/embed", self.base_url))
+                .json(&serde_json::json!({
+                    "model": &model,
+                    "input": text,
+                }))
                 .send()
                 .await
                 .map_err(|e| format!("embed request failed: {e}"))?;
 
             if !resp.status().is_success() {
-            let text = resp.text().await.unwrap_or_default();
-            return Err(format!("embed error ({}): {text}", text.len()));
+                let body = resp.text().await.unwrap_or_default();
+                return Err(format!("ollama embed error: {body}"));
             }
-        resp.json().await.map_err(|e| format!("embed parse error: {e}"))
+            // Ollama returns {"embeddings": [[...]], "model": "...", ...}.
+            // The outer `embeddings` is always a list; for a scalar input
+            // we get a single inner vector.
+            let parsed: serde_json::Value = resp.json().await
+                .map_err(|e| format!("embed parse error: {e}"))?;
+            let arr = parsed.get("embeddings")
+                .and_then(|v| v.as_array())
+                .ok_or_else(|| format!("ollama embed: missing 'embeddings' field in {parsed}"))?;
+            if arr.is_empty() {
+                return Err("ollama embed: empty embeddings array".to_string());
+            }
+            let first = arr[0].as_array()
+                .ok_or_else(|| "ollama embed: embeddings[0] not an array".to_string())?;
+            let vec: Vec<f64> = first.iter()
+                .filter_map(|n| n.as_f64())
+                .collect();
+            if vec.is_empty() {
+                return Err("ollama embed: numeric coercion produced empty vector".to_string());
+            }
+            embeddings.push(vec);
+        }
+
+        let dimensions = embeddings.first().map(|v| v.len()).unwrap_or(0);
+        Ok(EmbedResponse {
+            embeddings,
+            model,
+            dimensions,
+        })
     }
 
     pub async fn generate(&self, req: GenerateRequest) -> Result<GenerateResponse, String> {
+        if let Some(gw) = self.gateway_url.as_deref() {
+            return self.generate_via_gateway(gw, req).await;
+        }
+        // Direct Ollama path. Used by gateway internals (so the ollama
+        // provider can call upstream without a self-loop through
+        // /v1/chat) and by any consumer that wants raw transport
+        // without /v1/usage accounting.
+        let model = req.model.clone().unwrap_or_else(|| "qwen3.5:latest".to_string());
+        let mut body = serde_json::json!({
+            "model": &model,
+            "prompt": &req.prompt,
+            "stream": false,
+        });
+        let mut options = serde_json::Map::new();
+        if let Some(t) = req.temperature {
+            options.insert("temperature".to_string(), serde_json::json!(t));
+        }
+        if let Some(mt) = req.max_tokens {
+            options.insert("num_predict".to_string(), serde_json::json!(mt));
+        }
+        if !options.is_empty() {
+            body["options"] = serde_json::Value::Object(options);
+        }
+        if let Some(sys) = &req.system {
+            body["system"] = serde_json::json!(sys);
+        }
+        if let Some(th) = req.think {
+            body["think"] = serde_json::json!(th);
+        }
+
         let resp = self.client
-            .post(format!("{}/generate", self.base_url))
-            .json(&req)
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&body)
             .send()
             .await
             .map_err(|e| format!("generate request failed: {e}"))?;
 
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("generate error: {text}"));
+            return Err(format!("ollama generate error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("generate parse error: {e}"))
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("generate parse error: {e}"))?;
+
+        Ok(GenerateResponse {
+            text: parsed.get("response").and_then(|v| v.as_str()).unwrap_or("").to_string(),
+            model,
+            tokens_evaluated: parsed.get("prompt_eval_count").and_then(|v| v.as_u64()),
+            tokens_generated: parsed.get("eval_count").and_then(|v| v.as_u64()),
+        })
     }
 
-    pub async fn rerank(&self, req: RerankRequest) -> Result<RerankResponse, String> {
+    /// Phase 44 part 2: route generate() through the gateway's
+    /// /v1/chat with provider="ollama" so the call lands in
+    /// /v1/usage + Langfuse. Translates between the sidecar
+    /// GenerateRequest/Response shape and the OpenAI-compat
+    /// chat shape on the wire.
+    async fn generate_via_gateway(&self, gateway_url: &str, req: GenerateRequest) -> Result<GenerateResponse, String> {
+        let mut messages = Vec::with_capacity(2);
+        if let Some(sys) = &req.system {
+            messages.push(serde_json::json!({"role": "system", "content": sys}));
+        }
+        messages.push(serde_json::json!({"role": "user", "content": req.prompt}));
+        let mut body = serde_json::json!({
+            "messages": messages,
+            "provider": "ollama",
+        });
+        if let Some(m) = &req.model { body["model"] = serde_json::json!(m); }
+        if let Some(t) = req.temperature { body["temperature"] = serde_json::json!(t); }
+        if let Some(mt) = req.max_tokens { body["max_tokens"] = serde_json::json!(mt); }
+        if let Some(th) = req.think { body["think"] = serde_json::json!(th); }
+
         let resp = self.client
-            .post(format!("{}/rerank", self.base_url))
-            .json(&req)
+            .post(format!("{}/v1/chat", gateway_url))
+            .json(&body)
+            .send()
+            .await
+            .map_err(|e| format!("/v1/chat request failed: {e}"))?;
+        if !resp.status().is_success() {
+            let text = resp.text().await.unwrap_or_default();
+            return Err(format!("/v1/chat error: {text}"));
+        }
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("/v1/chat parse error: {e}"))?;
+
+        let text = parsed
+            .pointer("/choices/0/message/content")
+            .and_then(|v| v.as_str())
+            .unwrap_or("")
+            .to_string();
+        let model = parsed.get("model")
+            .and_then(|v| v.as_str())
+            .unwrap_or_else(|| req.model.as_deref().unwrap_or(""))
+            .to_string();
+        let prompt_tokens = parsed.pointer("/usage/prompt_tokens").and_then(|v| v.as_u64());
+        let completion_tokens = parsed.pointer("/usage/completion_tokens").and_then(|v| v.as_u64());
+
+        Ok(GenerateResponse {
+            text,
+            model,
+            tokens_evaluated: prompt_tokens,
+            tokens_generated: completion_tokens,
+        })
+    }
+
+    /// Cross-encoder reranking via Ollama generate. Asks the model to
+    /// rate each document's relevance to the query 0-10, then sorts
+    /// descending. Mirrors the sidecar's pre-2026-05-02 algorithm
+    /// exactly so callers see the same scores.
+    pub async fn rerank(&self, req: RerankRequest) -> Result<RerankResponse, String> {
+        let model = req.model.clone().unwrap_or_else(|| "qwen3.5:latest".to_string());
+        let mut scored: Vec<ScoredDocument> = Vec::with_capacity(req.documents.len());
+
+        for (i, doc) in req.documents.iter().enumerate() {
+            let prompt = format!(
+                "Rate the relevance of the following document to the query on a scale of 0 to 10. \
+                 Respond with ONLY a number.\n\n\
+                 Query: {}\n\n\
+                 Document: {}\n\n\
+                 Score:",
+                req.query, doc,
+            );
+            let resp = self.client
+                .post(format!("{}/api/generate", self.base_url))
+                .json(&serde_json::json!({
+                    "model": &model,
+                    "prompt": prompt,
+                    "stream": false,
+                    "options": {"temperature": 0.0, "num_predict": 8},
+                }))
                 .send()
                 .await
                 .map_err(|e| format!("rerank request failed: {e}"))?;
 
             if !resp.status().is_success() {
-            let text = resp.text().await.unwrap_or_default();
-            return Err(format!("rerank error: {text}"));
+                let body = resp.text().await.unwrap_or_default();
+                return Err(format!("ollama rerank error: {body}"));
             }
-        resp.json().await.map_err(|e| format!("rerank parse error: {e}"))
+            let parsed: serde_json::Value = resp.json().await
+                .map_err(|e| format!("rerank parse error: {e}"))?;
+            let text = parsed.get("response").and_then(|v| v.as_str()).unwrap_or("").trim();
+            // Parse the leading number; tolerate "7", "7.5", "7 — strong match".
+            let score = text.split_whitespace().next()
+                .and_then(|t| t.parse::<f64>().ok())
+                .unwrap_or(0.0)
+                .clamp(0.0, 10.0);
+
+            scored.push(ScoredDocument {
+                index: i,
+                text: doc.clone(),
+                score,
+            });
+        }
+
+        scored.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
+        if let Some(k) = req.top_k {
+            scored.truncate(k);
+        }
+        Ok(RerankResponse { results: scored, model })
     }
 
     /// Force Ollama to unload the named model from VRAM (keep_alive=0).
@@ -149,40 +529,116 @@ impl AiClient {
     /// profile's model can linger in VRAM next to the new one.
     pub async fn unload_model(&self, model: &str) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .post(format!("{}/admin/unload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&serde_json::json!({
+                "model": model,
+                "prompt": "",
+                "keep_alive": 0,
+                "stream": false,
+            }))
             .send().await
             .map_err(|e| format!("unload request failed: {e}"))?;
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("unload error: {text}"));
+            return Err(format!("ollama unload error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("unload parse error: {e}"))
+        // Ollama returns 200 with the empty-prompt response shape.
+        // Fold into the legacy {"unloaded": "<model>"} envelope so
+        // callers' parsing doesn't break.
+        Ok(serde_json::json!({ "unloaded": model }))
     }
 
     /// Ask Ollama to load the named model into VRAM proactively. Makes
     /// the first real request after profile activation fast (no cold-load
-    /// latency).
+    /// latency). Empty prompts confuse some models, so we send a single
+    /// space + cap num_predict=1 (matches the sidecar's prior behavior).
     pub async fn preload_model(&self, model: &str) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .post(format!("{}/admin/preload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
+            .post(format!("{}/api/generate", self.base_url))
+            .json(&serde_json::json!({
+                "model": model,
+                "prompt": " ",
+                "keep_alive": "5m",
+                "stream": false,
+                "options": {"num_predict": 1},
+            }))
             .send().await
             .map_err(|e| format!("preload request failed: {e}"))?;
         if !resp.status().is_success() {
             let text = resp.text().await.unwrap_or_default();
-            return Err(format!("preload error: {text}"));
+            return Err(format!("ollama preload error: {text}"));
         }
-        resp.json().await.map_err(|e| format!("preload parse error: {e}"))
+        let parsed: serde_json::Value = resp.json().await
+            .map_err(|e| format!("preload parse error: {e}"))?;
+        Ok(serde_json::json!({
+            "preloaded": model,
+            "load_duration_ns": parsed.get("load_duration"),
+            "total_duration_ns": parsed.get("total_duration"),
+        }))
     }
 
-    /// GPU + loaded-model snapshot from the sidecar. Combines nvidia-smi
-    /// output (if available) with Ollama's /api/ps.
+    /// GPU + loaded-model snapshot. Combines nvidia-smi output (when
+    /// available) with Ollama's /api/ps. Same shape as the prior
+    /// sidecar /admin/vram endpoint so callers don't need updating.
     pub async fn vram_snapshot(&self) -> Result<serde_json::Value, String> {
         let resp = self.client
-            .get(format!("{}/admin/vram", self.base_url))
+            .get(format!("{}/api/ps", self.base_url))
             .send().await
-            .map_err(|e| format!("vram request failed: {e}"))?;
-        resp.json().await.map_err(|e| format!("vram parse error: {e}"))
+            .map_err(|e| format!("ollama ps request failed: {e}"))?;
+        let loaded: Vec<serde_json::Value> = if resp.status().is_success() {
+            let parsed: serde_json::Value = resp.json().await.unwrap_or(serde_json::Value::Null);
+            parsed.get("models")
+                .and_then(|v| v.as_array())
+                .map(|arr| arr.iter().map(|m| serde_json::json!({
+                    "name": m.get("name"),
+                    "size_vram_mib": m.get("size_vram").and_then(|v| v.as_u64()).map(|n| n / (1024 * 1024)),
+                    "expires_at": m.get("expires_at"),
+                })).collect())
+                .unwrap_or_default()
+        } else {
+            Vec::new()
+        };
+
+        let gpu = nvidia_smi_snapshot();
+
+        Ok(serde_json::json!({
+            "gpu": gpu,
+            "ollama_loaded": loaded,
+        }))
     }
 }
+
+/// One-shot nvidia-smi poll. Returns Null if the tool isn't on PATH
+/// or the call fails. Mirrors the sidecar's `_nvidia_smi_snapshot`
+/// shape exactly so callers reading vram_snapshot don't break.
+fn nvidia_smi_snapshot() -> serde_json::Value {
+    use std::process::Command;
+    let out = Command::new("nvidia-smi")
+        .args([
+            "--query-gpu=memory.used,memory.total,utilization.gpu,name",
+            "--format=csv,noheader,nounits",
+        ])
+        .output();
+    let stdout = match out {
+        Ok(o) if o.status.success() => o.stdout,
+        _ => return serde_json::Value::Null,
+    };
+    let line = String::from_utf8_lossy(&stdout);
+    let line = line.trim();
+    if line.is_empty() {
+        return serde_json::Value::Null;
+    }
+    let parts: Vec<&str> = line.split(',').map(|s| s.trim()).collect();
+    if parts.len() < 4 {
+        return serde_json::Value::Null;
+    }
+    let used = parts[0].parse::<u64>().unwrap_or(0);
+    let total = parts[1].parse::<u64>().unwrap_or(0);
+    let util = parts[2].parse::<u64>().unwrap_or(0);
+    serde_json::json!({
+        "name": parts[3],
+        "used_mib": used,
+        "total_mib": total,
+        "utilization_pct": util,
+    })
+}

crates/aibridge/src/continuation.rs

@@ -195,8 +195,11 @@ pub async fn generate_continuable<G: TextGenerator>(
         let req = make_request(opts, prompt.to_string(), current_max);
         let resp = generator.generate_text(req).await?;
         calls += 1;
-        prompt_tokens = prompt_tokens.saturating_add(resp.tokens_evaluated.unwrap_or(0) as u32);
-        completion_tokens = completion_tokens.saturating_add(resp.tokens_generated.unwrap_or(0) as u32);
+        // u32::try_from saturates at u32::MAX instead of silently
+        // truncating bits when tokens_evaluated/_generated comes back
+        // as a u64 > 4 billion. Caught 2026-04-27 by Opus self-audit.
+        prompt_tokens = prompt_tokens.saturating_add(u32::try_from(resp.tokens_evaluated.unwrap_or(0)).unwrap_or(u32::MAX));
+        completion_tokens = completion_tokens.saturating_add(u32::try_from(resp.tokens_generated.unwrap_or(0)).unwrap_or(u32::MAX));
         if !resp.text.trim().is_empty() {
             combined = resp.text;
             break;
@@ -227,8 +230,8 @@ pub async fn generate_continuable<G: TextGenerator>(
         let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
         let resp = generator.generate_text(req).await?;
         calls += 1;
-        prompt_tokens = prompt_tokens.saturating_add(resp.tokens_evaluated.unwrap_or(0) as u32);
-        completion_tokens = completion_tokens.saturating_add(resp.tokens_generated.unwrap_or(0) as u32);
+        prompt_tokens = prompt_tokens.saturating_add(u32::try_from(resp.tokens_evaluated.unwrap_or(0)).unwrap_or(u32::MAX));
+        completion_tokens = completion_tokens.saturating_add(u32::try_from(resp.tokens_generated.unwrap_or(0)).unwrap_or(u32::MAX));
         combined.push_str(&resp.text);
         continuations += 1;
     }

crates/catalogd/Cargo.toml

@@ -16,5 +16,8 @@ chrono = { workspace = true }
 uuid = { workspace = true }
 object_store = { workspace = true }
 arrow = { workspace = true }
+parquet = { workspace = true }
+sha2 = { workspace = true }
+hmac = { workspace = true }
 proto = { path = "../proto" }
 tonic = { workspace = true }

crates/catalogd/src/audit_endpoint.rs

@@ -0,0 +1,505 @@
+//! `/audit/subject/{id}` HTTP endpoint — legal-tier subject audit response.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §4 + §6.
+//!
+//! Responds to: `GET /subject/{candidate_id}?from=ISO&to=ISO`
+//! Auth: requires `X-Lakehouse-Legal-Token` header matching the value in
+//! the legal-token file (see LegalAuditAuth). Constant-time compare.
+//!
+//! Response shape (subject_audit_response.v1):
+//!   - manifest: full SubjectManifest from catalogd
+//!   - audit_log.rows: SubjectAuditRow entries within [from, to]
+//!   - audit_log.chain_verified: bool (HMAC chain verifies end-to-end)
+//!   - completeness_attestation: text the lawyer can quote
+//!
+//! V1 deliberately does NOT include:
+//!   - Ed25519 signature on the response — chain verification IS the
+//!     v1 attestation; signing is future hardening (spec §10 v2 reserve)
+//!   - Live row projection from the underlying datasets — caller can
+//!     fetch via /query/sql against the safe_views named in the manifest;
+//!     keeping projection out of this endpoint avoids depending on queryd
+//!   - External anchor / S3 Object Lock — local HMAC chain is the v1
+//!     integrity surface
+//!
+//! These are documented as future hardening, not silent gaps.
+
+use crate::registry::Registry;
+use crate::subject_audit::SubjectAuditWriter;
+use axum::{
+    extract::{Path, Query, State},
+    http::{HeaderMap, StatusCode},
+    response::IntoResponse,
+    routing::get,
+    Json, Router,
+};
+use serde::{Deserialize, Serialize};
+use shared::types::{SubjectAuditRow, SubjectManifest};
+use std::sync::Arc;
+
+const LEGAL_TOKEN_HEADER: &str = "x-lakehouse-legal-token";
+const RESPONSE_SCHEMA: &str = "subject_audit_response.v1";
+
+/// State for the audit endpoint router. Held outside Registry because
+/// the audit writer + legal token aren't part of catalogd's data model
+/// — they're sidecar concerns specific to this endpoint.
+#[derive(Clone)]
+pub struct AuditEndpointState {
+    pub registry: Registry,
+    pub writer: Arc<SubjectAuditWriter>,
+    /// Legal-tier token. Constant-time compared against the header on
+    /// every request. None = endpoint returns 503 (audit unavailable).
+    pub legal_token: Option<Arc<String>>,
+}
+
+impl AuditEndpointState {
+    /// Construct from existing pieces. `legal_token_path` is read at
+    /// construction and trimmed of trailing whitespace. Missing/empty
+    /// = endpoint will 503 every request with a clear reason.
+    pub async fn new(
+        registry: Registry,
+        writer: Arc<SubjectAuditWriter>,
+        legal_token_path: &std::path::Path,
+    ) -> Self {
+        let legal_token = match tokio::fs::read_to_string(legal_token_path).await {
+            Ok(s) => {
+                let trimmed = s.trim().to_string();
+                if trimmed.is_empty() {
+                    tracing::warn!(
+                        "audit endpoint: legal token file at {} is empty — endpoint will 503",
+                        legal_token_path.display()
+                    );
+                    None
+                } else if trimmed.len() < 32 {
+                    // 2026-05-03 kimi scrum BLOCK at audit_endpoint.rs:59-60:
+                    // 16-byte minimum was too permissive for a legal-tier
+                    // secret. Aligned to the SubjectAuditWriter signing-key
+                    // minimum (32 bytes — HMAC-SHA256 best practice).
+                    tracing::warn!(
+                        "audit endpoint: legal token at {} is {} chars (minimum 32) — endpoint will 503",
+                        legal_token_path.display(), trimmed.len()
+                    );
+                    None
+                } else {
+                    tracing::info!("audit endpoint: legal token loaded from {}", legal_token_path.display());
+                    Some(Arc::new(trimmed))
+                }
+            }
+            Err(e) => {
+                tracing::warn!(
+                    "audit endpoint: legal token unreadable from {} ({e}) — endpoint will 503",
+                    legal_token_path.display()
+                );
+                None
+            }
+        };
+        Self { registry, writer, legal_token }
+    }
+}
+
+pub fn router(state: AuditEndpointState) -> Router {
+    Router::new()
+        .route("/subject/{candidate_id}", get(audit_subject))
+        .route("/health", get(audit_health))
+        .with_state(state)
+}
+
+async fn audit_health(State(state): State<AuditEndpointState>) -> impl IntoResponse {
+    if state.legal_token.is_some() {
+        (StatusCode::OK, "audit endpoint ready").into_response()
+    } else {
+        (StatusCode::SERVICE_UNAVAILABLE, "audit endpoint disabled (legal token missing)").into_response()
+    }
+}
+
+#[derive(Deserialize)]
+struct AuditQuery {
+    /// ISO-8601 start of audit window. None = unbounded back.
+    from: Option<chrono::DateTime<chrono::Utc>>,
+    /// ISO-8601 end of audit window. None = unbounded forward.
+    to: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Serialize)]
+struct AuditWindowEcho {
+    from: Option<chrono::DateTime<chrono::Utc>>,
+    to: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Serialize)]
+struct AuditLogSection {
+    rows_in_window: usize,
+    rows: Vec<SubjectAuditRow>,
+    chain_root: String,
+    chain_verified: bool,
+    chain_rows_total: usize,
+    chain_verification_error: Option<String>,
+}
+
+#[derive(Serialize)]
+struct AuditResponse {
+    schema: &'static str,
+    candidate_id: String,
+    generated_at: chrono::DateTime<chrono::Utc>,
+    generated_by: &'static str,
+    request_window: AuditWindowEcho,
+    manifest: SubjectManifest,
+    audit_log: AuditLogSection,
+    /// Datasets the manifest says contain rows for this subject — caller
+    /// can issue follow-up queries against the named safe_views to get
+    /// the actual row projection. Keeping projection out of this endpoint
+    /// avoids depending on queryd from catalogd.
+    datasets_referenced: Vec<DatasetRefEcho>,
+    safe_views_available: Vec<String>,
+    completeness_attestation: String,
+}
+
+#[derive(Serialize)]
+struct DatasetRefEcho {
+    name: String,
+    key_column: String,
+    key_value: String,
+}
+
+/// Constant-time string comparison to avoid timing leaks on the legal
+/// token check. `subtle` would be the standard but introducing a dep
+/// for one comparison is overkill — manual byte-eq with an unconditional
+/// fold over both byte slices.
+fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
+    if a.len() != b.len() {
+        return false;
+    }
+    let mut diff: u8 = 0;
+    for (x, y) in a.iter().zip(b.iter()) {
+        diff |= x ^ y;
+    }
+    diff == 0
+}
+
+/// Extract + verify the legal-tier token from the request. Returns
+/// Err((status, body)) on any failure (missing header, missing token
+/// configured, mismatch). Does NOT log token values.
+fn require_legal_auth(
+    state: &AuditEndpointState,
+    headers: &HeaderMap,
+) -> Result<(), (StatusCode, &'static str)> {
+    let configured = state.legal_token.as_ref().ok_or((
+        StatusCode::SERVICE_UNAVAILABLE,
+        "audit endpoint disabled — no legal token configured on this gateway",
+    ))?;
+    let provided = headers.get(LEGAL_TOKEN_HEADER).ok_or((
+        StatusCode::UNAUTHORIZED,
+        "missing X-Lakehouse-Legal-Token header",
+    ))?;
+    let provided = provided.to_str().map_err(|_| (
+        StatusCode::UNAUTHORIZED,
+        "X-Lakehouse-Legal-Token contains non-ASCII characters",
+    ))?;
+    if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) {
+        return Err((StatusCode::UNAUTHORIZED, "X-Lakehouse-Legal-Token mismatch"));
+    }
+    Ok(())
+}
+
+async fn audit_subject(
+    State(state): State<AuditEndpointState>,
+    Path(candidate_id): Path<String>,
+    Query(q): Query<AuditQuery>,
+    headers: HeaderMap,
+) -> impl IntoResponse {
+    if let Err((status, msg)) = require_legal_auth(&state, &headers) {
+        return (status, msg).into_response();
+    }
+    if candidate_id.is_empty() {
+        return (StatusCode::BAD_REQUEST, "candidate_id is empty").into_response();
+    }
+
+    // 1. Load the subject manifest.
+    let manifest = match state.registry.get_subject(&candidate_id).await {
+        Some(m) => m,
+        None => return (
+            StatusCode::NOT_FOUND,
+            format!("subject manifest not found for candidate_id={candidate_id}"),
+        ).into_response(),
+    };
+
+    // 2. Read audit rows in window.
+    let rows = match state.writer.read_rows_in_range(&candidate_id, q.from, q.to).await {
+        Ok(r) => r,
+        Err(e) => return (
+            StatusCode::INTERNAL_SERVER_ERROR,
+            format!("audit log read failed: {e}"),
+        ).into_response(),
+    };
+
+    // 3. Verify chain end-to-end (full log, not just the windowed slice;
+    //    chain integrity is global, not per-window). chain_root MUST
+    //    come from the actual log tip — taking it from the windowed
+    //    `rows` would lie when `to` is set or the window catches no
+    //    rows. (2026-05-03 opus scrum WARN audit_endpoint.rs:206.)
+    let (chain_verified, chain_rows_total, chain_root, chain_err) =
+        match state.writer.verify_chain(&candidate_id).await {
+            Ok(n) => {
+                let tip = state.writer.chain_tip(&candidate_id).await
+                    .unwrap_or_else(|| "GENESIS".into());
+                (true, n, tip, None)
+            }
+            Err(e) => (false, 0, String::new(), Some(e)),
+        };
+
+    // 4. Tampering detection: if the manifest claims a non-trivial
+    //    chain root but the log is empty (chain_rows_total == 0), the
+    //    log was deleted between writes. Flag it. (2026-05-03 opus
+    //    scrum WARN subject_audit.rs:347 + audit_endpoint coverage.)
+    let manifest_root = manifest.audit_log_chain_root.clone();
+    let log_was_tampered = chain_verified
+        && chain_rows_total == 0
+        && !manifest_root.is_empty()
+        && manifest_root != "GENESIS";
+    let (chain_verified, chain_err) = if log_was_tampered {
+        (false, Some(format!(
+            "manifest expects chain root '{}' but the audit log has 0 rows; log was deleted or wiped",
+            manifest_root,
+        )))
+    } else {
+        (chain_verified, chain_err)
+    };
+
+    let datasets_referenced = manifest.datasets.iter()
+        .map(|d| DatasetRefEcho {
+            name: d.name.clone(),
+            key_column: d.key_column.clone(),
+            key_value: d.key_value.clone(),
+        })
+        .collect();
+    let safe_views_available = manifest.safe_views.clone();
+
+    let attestation = format!(
+        "All audit rows for candidate_id={} within [{}, {}] are included. \
+         HMAC chain verified end-to-end: {} rows total, integrity={}.",
+        candidate_id,
+        q.from.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
+        q.to.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
+        chain_rows_total,
+        if chain_verified { "verified" } else { "BROKEN" },
+    );
+
+    let resp = AuditResponse {
+        schema: RESPONSE_SCHEMA,
+        candidate_id: candidate_id.clone(),
+        generated_at: chrono::Utc::now(),
+        generated_by: "catalogd",
+        request_window: AuditWindowEcho { from: q.from, to: q.to },
+        manifest,
+        audit_log: AuditLogSection {
+            rows_in_window: rows.len(),
+            rows,
+            chain_root,
+            chain_verified,
+            chain_rows_total,
+            chain_verification_error: chain_err,
+        },
+        datasets_referenced,
+        safe_views_available,
+        completeness_attestation: attestation,
+    };
+
+    // Log the legal-tier read so it's discoverable in operations review.
+    // Token value is NEVER logged. Only the fact of the read.
+    tracing::info!(
+        candidate_id = %candidate_id,
+        rows_in_window = resp.audit_log.rows_in_window,
+        chain_verified = resp.audit_log.chain_verified,
+        "legal-tier audit response served"
+    );
+
+    (StatusCode::OK, Json(resp)).into_response()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::subject_audit::SubjectAuditWriter;
+    use object_store::memory::InMemory;
+    use shared::types::{
+        AuditAccessor, BiometricConsent, BiometricConsentStatus, ConsentStatus,
+        GeneralPiiConsent, SubjectAuditRow, SubjectConsent, SubjectRetention,
+        SubjectStatus, SubjectVertical,
+    };
+
+    fn fixture_manifest(candidate_id: &str) -> SubjectManifest {
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: candidate_id.into(),
+            created_at: chrono::Utc::now(),
+            updated_at: chrono::Utc::now(),
+            status: SubjectStatus::Active,
+            vertical: SubjectVertical::Unknown,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::PendingBackfillReview,
+                    version: String::new(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: BiometricConsentStatus::NeverCollected,
+                    retention_until: None,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
+                policy: "test".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+                biometric_collection: None,
+        }
+    }
+
+    fn fixture_audit_row(candidate_id: &str) -> SubjectAuditRow {
+        SubjectAuditRow {
+            schema: "subject_audit.v1".into(),
+            ts: chrono::Utc::now(),
+            candidate_id: candidate_id.into(),
+            accessor: AuditAccessor {
+                kind: "test".into(),
+                daemon: "test".into(),
+                purpose: "test".into(),
+                trace_id: String::new(),
+            },
+            fields_accessed: vec!["name".into()],
+            result: "success".into(),
+            prev_chain_hash: String::new(),
+            row_hmac: String::new(),
+        }
+    }
+
+    async fn fixture_state_with_token(token: &str) -> AuditEndpointState {
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store, (0u8..32).collect(),
+        ));
+        AuditEndpointState {
+            registry,
+            writer,
+            legal_token: Some(Arc::new(token.into())),
+        }
+    }
+
+    #[test]
+    fn constant_time_eq_basic() {
+        assert!(constant_time_eq(b"abc", b"abc"));
+        assert!(!constant_time_eq(b"abc", b"abd"));
+        assert!(!constant_time_eq(b"abc", b"abcd"));
+        assert!(constant_time_eq(b"", b""));
+    }
+
+    #[tokio::test]
+    async fn missing_legal_token_returns_503() {
+        let state = AuditEndpointState {
+            registry: Registry::new(Arc::new(InMemory::new())),
+            writer: Arc::new(SubjectAuditWriter::with_inline_key(
+                Arc::new(InMemory::new()), (0u8..32).collect(),
+            )),
+            legal_token: None,
+        };
+        let headers = HeaderMap::new();
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::SERVICE_UNAVAILABLE, _))));
+    }
+
+    #[tokio::test]
+    async fn missing_header_returns_401() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        let headers = HeaderMap::new();
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
+    }
+
+    #[tokio::test]
+    async fn wrong_token_returns_401() {
+        let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
+        let mut headers = HeaderMap::new();
+        headers.insert(LEGAL_TOKEN_HEADER, "wrong".parse().unwrap());
+        let r = require_legal_auth(&state, &headers);
+        assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
+    }
+
+    #[tokio::test]
+    async fn correct_token_passes_auth() {
+        let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
+        let mut headers = HeaderMap::new();
+        headers.insert(LEGAL_TOKEN_HEADER, "correct-token-must-be-long-enough".parse().unwrap());
+        let r = require_legal_auth(&state, &headers);
+        assert!(r.is_ok());
+    }
+
+    #[tokio::test]
+    async fn audit_response_assembly_full_path() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        // Plant a manifest + 3 audit rows.
+        state.registry.put_subject(fixture_manifest("CAND-AUDIT-1")).await.unwrap();
+        for _ in 0..3 {
+            state.writer.append(fixture_audit_row("CAND-AUDIT-1")).await.unwrap();
+        }
+        // Read rows in unbounded window.
+        let rows = state.writer
+            .read_rows_in_range("CAND-AUDIT-1", None, None)
+            .await
+            .unwrap();
+        assert_eq!(rows.len(), 3);
+        // Verify chain integrity.
+        let n = state.writer.verify_chain("CAND-AUDIT-1").await.unwrap();
+        assert_eq!(n, 3);
+    }
+
+    #[tokio::test]
+    async fn audit_response_window_filters_rows() {
+        let state = fixture_state_with_token("test-token-must-be-long-enough").await;
+        state.registry.put_subject(fixture_manifest("CAND-WIN-1")).await.unwrap();
+        // Append 5 rows.
+        let mut row_times = Vec::new();
+        for _ in 0..5 {
+            let row = fixture_audit_row("CAND-WIN-1");
+            row_times.push(row.ts);
+            state.writer.append(row).await.unwrap();
+            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
+        }
+        // Window covering only the middle 3 rows.
+        let from = row_times[1];
+        let to = row_times[3];
+        let rows = state.writer
+            .read_rows_in_range("CAND-WIN-1", Some(from), Some(to))
+            .await
+            .unwrap();
+        assert!(rows.len() >= 3, "expected ≥3 rows in window; got {}", rows.len());
+        assert!(rows.len() <= 3, "expected ≤3 rows in window; got {}", rows.len());
+    }
+
+    #[tokio::test]
+    async fn empty_token_file_results_in_disabled_endpoint() {
+        let tmp = std::env::temp_dir().join(format!("legal_empty_{}.tok", std::process::id()));
+        tokio::fs::write(&tmp, "").await.unwrap();
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
+        let state = AuditEndpointState::new(registry, writer, &tmp).await;
+        assert!(state.legal_token.is_none());
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+
+    #[tokio::test]
+    async fn short_token_file_rejected_at_load() {
+        let tmp = std::env::temp_dir().join(format!("legal_short_{}.tok", std::process::id()));
+        tokio::fs::write(&tmp, "tooshort").await.unwrap();
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let registry = Registry::new(store.clone());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
+        let state = AuditEndpointState::new(registry, writer, &tmp).await;
+        assert!(state.legal_token.is_none());
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+}

crates/catalogd/src/bin/backfill_subjects.rs

@@ -0,0 +1,341 @@
+//! Backfill subject manifests from a parquet source.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3.
+//!
+//! For each row in the source parquet, creates a SubjectManifest with
+//! the BIPA-defensible defaults from the spec:
+//!   - vertical = unknown                (HIPAA fail-closed routing)
+//!   - consent.general_pii = pending_backfill_review  (NOT inferred_existing)
+//!   - consent.biometric  = never_collected           (no biometric backfill)
+//!   - retention.general_pii_until = now + 4 years    (default policy)
+//!
+//! Idempotent: if a manifest already exists for a candidate_id, the
+//! backfill skips it. Re-runs are safe.
+//!
+//! Conservative defaults: --limit 1000 by default. Pass --all to do
+//! the full source. Pass --concurrency to bound parallel writes.
+//!
+//! Usage examples:
+//!   cargo run --bin backfill_subjects -- \
+//!     --source data/datasets/workers_500k.parquet \
+//!     --dataset workers_500k \
+//!     --key-column worker_id \
+//!     --candidate-id-prefix WORKER- \
+//!     --limit 100 \
+//!     --dry-run
+//!
+//!   cargo run --bin backfill_subjects -- \
+//!     --source data/datasets/workers_500k.parquet \
+//!     --dataset workers_500k \
+//!     --key-column worker_id \
+//!     --candidate-id-prefix WORKER- \
+//!     --safe-view workers_safe \
+//!     --all
+//!
+//! On the live host the binary reads from / writes to ./data so it
+//! integrates with the running catalogd's persistence path.
+
+use catalogd::registry::Registry;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use shared::types::{
+    BiometricConsent, BiometricConsentStatus, ConsentStatus, GeneralPiiConsent,
+    SubjectConsent, SubjectDatasetRef, SubjectManifest, SubjectRetention,
+    SubjectStatus, SubjectVertical,
+};
+use std::fs::File;
+use std::path::PathBuf;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use storaged::backend;
+
+const DEFAULT_LIMIT: usize = 1000;
+const DEFAULT_CONCURRENCY: usize = 32;
+const DEFAULT_RETENTION_YEARS: i64 = 4;
+
+#[derive(Debug)]
+struct Args {
+    source: PathBuf,
+    dataset_name: String,
+    key_column: String,
+    candidate_id_prefix: String,
+    safe_view: Option<String>,
+    storage_root: PathBuf,
+    limit: Option<usize>,
+    concurrency: usize,
+    dry_run: bool,
+}
+
+fn parse_args() -> Result<Args, String> {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut source: Option<PathBuf> = None;
+    let mut dataset_name: Option<String> = None;
+    let mut key_column: Option<String> = None;
+    let mut candidate_id_prefix: String = String::new();
+    let mut safe_view: Option<String> = None;
+    let mut storage_root: PathBuf = PathBuf::from("./data");
+    let mut limit: Option<usize> = Some(DEFAULT_LIMIT);
+    let mut concurrency: usize = DEFAULT_CONCURRENCY;
+    let mut dry_run = false;
+
+    let mut i = 1;
+    while i < argv.len() {
+        let arg = &argv[i];
+        let next = || -> Result<String, String> {
+            argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
+        };
+        match arg.as_str() {
+            "--source" => { source = Some(PathBuf::from(next()?)); i += 2; }
+            "--dataset" => { dataset_name = Some(next()?); i += 2; }
+            "--key-column" => { key_column = Some(next()?); i += 2; }
+            "--candidate-id-prefix" => { candidate_id_prefix = next()?; i += 2; }
+            "--safe-view" => { safe_view = Some(next()?); i += 2; }
+            "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
+            "--limit" => { limit = Some(next()?.parse().map_err(|e| format!("--limit: {e}"))?); i += 2; }
+            "--all" => { limit = None; i += 1; }
+            "--concurrency" => { concurrency = next()?.parse().map_err(|e| format!("--concurrency: {e}"))?; i += 2; }
+            "--dry-run" => { dry_run = true; i += 1; }
+            "-h" | "--help" => { print_help(); std::process::exit(0); }
+            other => return Err(format!("unknown arg: {other}")),
+        }
+    }
+    Ok(Args {
+        source: source.ok_or("missing --source")?,
+        dataset_name: dataset_name.ok_or("missing --dataset")?,
+        key_column: key_column.ok_or("missing --key-column")?,
+        candidate_id_prefix,
+        safe_view,
+        storage_root,
+        limit,
+        concurrency,
+        dry_run,
+    })
+}
+
+fn print_help() {
+    eprintln!("backfill_subjects — populate subject manifests from a parquet source\n");
+    eprintln!("Usage: backfill_subjects [flags]\n");
+    eprintln!("Required:");
+    eprintln!("  --source PATH                parquet file containing subject identifiers");
+    eprintln!("  --dataset NAME               catalogd dataset name (must already exist)");
+    eprintln!("  --key-column COL             column in the parquet that holds subject identifiers\n");
+    eprintln!("Optional:");
+    eprintln!("  --candidate-id-prefix STR    prefix added to each row's value to form candidate_id");
+    eprintln!("                                 (e.g. 'WORKER-' on a worker_id int64 column)");
+    eprintln!("  --safe-view NAME             name of an AiView that safely projects this subject's data");
+    eprintln!("  --storage-root PATH          local-filesystem catalog root (default: ./data)");
+    eprintln!("  --limit N                    process at most N rows (default: {DEFAULT_LIMIT})");
+    eprintln!("  --all                        process every row in the source");
+    eprintln!("  --concurrency N              max concurrent writes (default: {DEFAULT_CONCURRENCY})");
+    eprintln!("  --dry-run                    parse + count + sample, don't write anything");
+}
+
+#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
+async fn main() {
+    let args = match parse_args() {
+        Ok(a) => a,
+        Err(e) => {
+            eprintln!("error: {e}");
+            print_help();
+            std::process::exit(2);
+        }
+    };
+
+    if let Err(e) = run(args).await {
+        eprintln!("backfill failed: {e}");
+        std::process::exit(1);
+    }
+}
+
+async fn run(args: Args) -> Result<(), String> {
+    eprintln!("[backfill] source:  {}", args.source.display());
+    eprintln!("[backfill] dataset: {}", args.dataset_name);
+    eprintln!("[backfill] key:     {} (prefix: {:?})", args.key_column, args.candidate_id_prefix);
+    eprintln!("[backfill] limit:   {}", args.limit.map(|n| n.to_string()).unwrap_or("all".into()));
+    eprintln!("[backfill] dry-run: {}", args.dry_run);
+
+    // Stand up catalogd over the same local FS the running daemon uses.
+    let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
+    let reg = Arc::new(Registry::new(store));
+    reg.rebuild().await?;
+
+    if reg.get_by_name(&args.dataset_name).await.is_none() {
+        return Err(format!(
+            "dataset '{}' not found in catalog at {}; backfill needs the dataset registered first",
+            args.dataset_name, args.storage_root.display(),
+        ));
+    }
+    eprintln!("[backfill] dataset '{}' exists in catalog ✓", args.dataset_name);
+
+    // Open parquet reader.
+    let file = File::open(&args.source).map_err(|e| format!("open {}: {e}", args.source.display()))?;
+    let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(|e| e.to_string())?;
+    let total_rows = builder.metadata().file_metadata().num_rows() as usize;
+    eprintln!("[backfill] parquet rows: {total_rows}");
+
+    // Project only the key column for IO efficiency.
+    let schema = builder.schema().clone();
+    let col_idx = schema.index_of(&args.key_column)
+        .map_err(|_| format!("column '{}' not found; parquet has: {:?}",
+            args.key_column,
+            schema.fields().iter().map(|f| f.name()).collect::<Vec<_>>(),
+        ))?;
+    let projection = parquet::arrow::ProjectionMask::roots(builder.parquet_schema(), [col_idx]);
+    let mut reader = builder.with_projection(projection).build().map_err(|e| e.to_string())?;
+
+    // Collect candidate IDs (limited).
+    let cap = args.limit.unwrap_or(total_rows);
+    let mut candidate_ids: Vec<String> = Vec::with_capacity(cap.min(total_rows));
+    let mut rows_seen = 0usize;
+    'outer: while let Some(batch_result) = reader.next() {
+        let batch = batch_result.map_err(|e| format!("batch: {e}"))?;
+        let column = batch.column(0); // single projected column
+        for row in 0..batch.num_rows() {
+            if let Some(limit) = args.limit {
+                if candidate_ids.len() >= limit { break 'outer; }
+            }
+            rows_seen += 1;
+            let raw = format_cell_value(column, row)?;
+            candidate_ids.push(format!("{}{}", args.candidate_id_prefix, raw));
+        }
+    }
+    eprintln!("[backfill] collected {} candidate_ids ({} rows scanned)", candidate_ids.len(), rows_seen);
+
+    // Sample for the operator.
+    eprintln!("[backfill] sample (first 5):");
+    for cid in candidate_ids.iter().take(5) {
+        eprintln!("  - {cid}");
+    }
+
+    if args.dry_run {
+        eprintln!("[backfill] DRY-RUN — no writes. Re-run without --dry-run to commit.");
+        return Ok(());
+    }
+
+    // Write manifests with bounded concurrency.
+    let now = chrono::Utc::now();
+    let retention_until = now + chrono::Duration::days(365 * DEFAULT_RETENTION_YEARS);
+    let safe_views: Vec<String> = args.safe_view.into_iter().collect();
+    let dataset_ref = SubjectDatasetRef {
+        name: args.dataset_name.clone(),
+        key_column: args.key_column.clone(),
+        key_value: String::new(), // filled per-subject below
+    };
+
+    let attempted = Arc::new(AtomicUsize::new(0));
+    let inserted = Arc::new(AtomicUsize::new(0));
+    let skipped = Arc::new(AtomicUsize::new(0));
+    let failed = Arc::new(AtomicUsize::new(0));
+    let progress_every = ((candidate_ids.len() / 20).max(1)).min(5_000);
+
+    // Bounded concurrency via Semaphore.
+    let sem = Arc::new(tokio::sync::Semaphore::new(args.concurrency));
+    let mut handles = Vec::with_capacity(candidate_ids.len());
+    for (idx, cid) in candidate_ids.into_iter().enumerate() {
+        let permit = sem.clone().acquire_owned().await.map_err(|e| e.to_string())?;
+        let reg = reg.clone();
+        let mut ds_ref = dataset_ref.clone();
+        // strip_prefix gives one-shot semantics; trim_start_matches strips
+        // repeated occurrences (a "WORKER-WORKER-1" id would lose both
+        // prefixes). 2026-05-03 opus scrum WARN at backfill_subjects.rs:189.
+        ds_ref.key_value = cid.strip_prefix(&args.candidate_id_prefix)
+            .unwrap_or(&cid).to_string();
+        let safe_views = safe_views.clone();
+        let attempted = attempted.clone();
+        let inserted = inserted.clone();
+        let skipped = skipped.clone();
+        let failed = failed.clone();
+
+        let handle = tokio::spawn(async move {
+            let _permit = permit;
+            attempted.fetch_add(1, Ordering::Relaxed);
+            // Idempotency: skip if subject already exists.
+            if reg.get_subject(&cid).await.is_some() {
+                skipped.fetch_add(1, Ordering::Relaxed);
+                return;
+            }
+            let manifest = SubjectManifest {
+                schema: "subject_manifest.v1".into(),
+                candidate_id: cid.clone(),
+                created_at: now,
+                updated_at: now,
+                status: SubjectStatus::Active,
+                vertical: SubjectVertical::Unknown,
+                consent: SubjectConsent {
+                    general_pii: GeneralPiiConsent {
+                        status: ConsentStatus::PendingBackfillReview,
+                        version: String::new(),
+                        given_at: None,
+                        withdrawn_at: None,
+                    },
+                    biometric: BiometricConsent {
+                        status: BiometricConsentStatus::NeverCollected,
+                        retention_until: None,
+                    },
+                },
+                retention: SubjectRetention {
+                    general_pii_until: retention_until,
+                    policy: "4_year_default".into(),
+                },
+                datasets: vec![ds_ref],
+                safe_views,
+                audit_log_path: String::new(),
+                audit_log_chain_root: String::new(),
+                biometric_collection: None,
+            };
+            match reg.put_subject(manifest).await {
+                Ok(_) => { inserted.fetch_add(1, Ordering::Relaxed); }
+                Err(e) => {
+                    failed.fetch_add(1, Ordering::Relaxed);
+                    eprintln!("  ✗ {cid}: {e}");
+                }
+            }
+            let cur = attempted.load(Ordering::Relaxed);
+            if cur % progress_every == 0 {
+                eprintln!("[backfill] progress: {} attempted ({} inserted, {} skipped, {} failed)",
+                    cur, inserted.load(Ordering::Relaxed),
+                    skipped.load(Ordering::Relaxed),
+                    failed.load(Ordering::Relaxed));
+            }
+            let _ = idx;
+        });
+        handles.push(handle);
+    }
+    for h in handles {
+        let _ = h.await;
+    }
+
+    eprintln!();
+    eprintln!("──────────────────────────────────────────────────────────────────");
+    eprintln!("[backfill] done");
+    eprintln!("[backfill]   attempted: {}", attempted.load(Ordering::Relaxed));
+    eprintln!("[backfill]   inserted:  {}", inserted.load(Ordering::Relaxed));
+    eprintln!("[backfill]   skipped:   {} (already existed — idempotent re-run)", skipped.load(Ordering::Relaxed));
+    eprintln!("[backfill]   failed:    {}", failed.load(Ordering::Relaxed));
+    eprintln!("[backfill] subjects in catalog now: {}", reg.subjects_count().await);
+    Ok(())
+}
+
+/// Format a single cell value as a string for use in the candidate_id.
+/// Supports the integer + string types most likely to be subject keys.
+fn format_cell_value(array: &Arc<dyn arrow::array::Array>, row: usize) -> Result<String, String> {
+    use arrow::array::{Int32Array, Int64Array, StringArray, UInt32Array, UInt64Array};
+    use arrow::datatypes::DataType;
+    if array.is_null(row) {
+        return Err(format!("row {row}: key column is null — cannot form candidate_id"));
+    }
+    let dt = array.data_type();
+    let s = match dt {
+        DataType::Int64 => array.as_any().downcast_ref::<Int64Array>()
+            .ok_or("int64 downcast")?.value(row).to_string(),
+        DataType::Int32 => array.as_any().downcast_ref::<Int32Array>()
+            .ok_or("int32 downcast")?.value(row).to_string(),
+        DataType::UInt64 => array.as_any().downcast_ref::<UInt64Array>()
+            .ok_or("uint64 downcast")?.value(row).to_string(),
+        DataType::UInt32 => array.as_any().downcast_ref::<UInt32Array>()
+            .ok_or("uint32 downcast")?.value(row).to_string(),
+        DataType::Utf8 => array.as_any().downcast_ref::<StringArray>()
+            .ok_or("utf8 downcast")?.value(row).to_string(),
+        other => return Err(format!("unsupported key column type: {other:?}")),
+    };
+    Ok(s)
+}

crates/catalogd/src/bin/parity_subject_audit.rs

@@ -0,0 +1,235 @@
+//! Cross-runtime parity helper for subject-audit chain.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.
+//!
+//! This binary is consumed by the Go-repo's
+//! scripts/cutover/parity/subject_audit_parity.sh.
+//! Go counterpart: scripts/cutover/parity/subject_audit_helper/main.go
+//! (in the Go repo). Both helpers are kept in lockstep by the parity
+//! script's byte-equality assertions — change one, change both.
+//!
+//! Both helpers MUST produce byte-identical output for the same inputs.
+//! Divergence here is a parity break — a SubjectManifest written by Rust
+//! that Go cannot verify, or vice versa.
+//!
+//! Two modes:
+//!
+//!   --known-answer
+//!       Print the canonical bytes + HMAC of a hardcoded fixture row.
+//!       The Go helper must produce IDENTICAL bytes + IDENTICAL hash.
+//!       Hardcoded fixture matches Go test TestKnownAnswerVector.
+//!
+//!   --verify <audit_log_path> --key <key_path>
+//!       Parse the JSONL audit log, replay the HMAC chain. Print
+//!       JSON: {"count": N, "tip": "<hash|GENESIS>", "verified": bool, "error": "<msg|null>"}.
+//!
+//! Output format: ONE JSON object per stdout, terminating newline. The
+//! parity script diffs Rust stdout vs Go stdout via `diff -q`.
+
+use catalogd::subject_audit::SubjectAuditWriter;
+use hmac::{Hmac, Mac};
+use object_store::ObjectStore;
+use object_store::memory::InMemory;
+use serde::Serialize;
+use sha2::Sha256;
+use shared::types::{AuditAccessor, SubjectAuditRow};
+use std::collections::BTreeMap;
+use std::path::PathBuf;
+use std::sync::Arc;
+
+type HmacSha256 = Hmac<Sha256>;
+
+const GENESIS: &str = "GENESIS";
+
+/// Same canonical-JSON algorithm as crates/catalogd/src/subject_audit.rs.
+/// Reproduced here so the helper does not depend on internal-only items
+/// in that file. If the algorithm there changes, change it here AND in
+/// the Go helper in lockstep.
+fn canonical_json(v: &serde_json::Value) -> Vec<u8> {
+    fn rewrite(v: &serde_json::Value) -> serde_json::Value {
+        match v {
+            serde_json::Value::Object(map) => {
+                let sorted: BTreeMap<String, serde_json::Value> = map
+                    .iter()
+                    .map(|(k, v)| (k.clone(), rewrite(v)))
+                    .collect();
+                serde_json::Value::Object(sorted.into_iter().collect())
+            }
+            serde_json::Value::Array(arr) => {
+                serde_json::Value::Array(arr.iter().map(rewrite).collect())
+            }
+            other => other.clone(),
+        }
+    }
+    serde_json::to_vec(&rewrite(v)).expect("canonical-json serialize")
+}
+
+fn canonical_row_bytes(row: &SubjectAuditRow) -> Vec<u8> {
+    let mut v: serde_json::Value = serde_json::to_value(row).expect("row to value");
+    if let Some(obj) = v.as_object_mut() {
+        obj.remove("row_hmac");
+    }
+    canonical_json(&v)
+}
+
+fn compute_hmac(key: &[u8], prev: &str, canonical: &[u8]) -> String {
+    let mut mac = <HmacSha256 as Mac>::new_from_slice(key).expect("HMAC accepts any key length");
+    mac.update(prev.as_bytes());
+    mac.update(canonical);
+    let result = mac.finalize().into_bytes();
+    const HEX: &[u8; 16] = b"0123456789abcdef";
+    let mut s = String::with_capacity(64);
+    for byte in result {
+        s.push(HEX[(byte >> 4) as usize] as char);
+        s.push(HEX[(byte & 0x0f) as usize] as char);
+    }
+    s
+}
+
+fn deterministic_key() -> Vec<u8> {
+    (0u8..32).collect()
+}
+
+/// Print to stderr + exit(2). Never returns. Used in place of `.expect()`
+/// on argv / file reads so failures produce a clean usage error rather
+/// than a Rust panic backtrace that the parity script would mistake
+/// for a parity break.
+fn die(msg: &str) -> ! {
+    eprintln!("error: {msg}");
+    std::process::exit(2);
+}
+
+#[derive(Serialize)]
+struct KnownAnswerOut {
+    mode: &'static str,
+    canonical: String,
+    hmac: String,
+    canonical_bytes_len: usize,
+}
+
+#[derive(Serialize)]
+struct VerifyOut {
+    mode: &'static str,
+    count: usize,
+    tip: String,
+    verified: bool,
+    error: Option<String>,
+}
+
+fn known_answer() {
+    let row = SubjectAuditRow {
+        schema: "subject_audit.v1".into(),
+        ts: chrono::DateTime::parse_from_rfc3339("2026-05-03T12:00:00Z")
+            .unwrap()
+            .with_timezone(&chrono::Utc),
+        candidate_id: "WORKER-FIXED".into(),
+        accessor: AuditAccessor {
+            kind: "gateway_lookup".into(),
+            daemon: "gateway".into(),
+            purpose: "parity_test".into(),
+            trace_id: "trace-fixed".into(),
+        },
+        fields_accessed: vec!["name".into()],
+        result: "success".into(),
+        prev_chain_hash: GENESIS.into(),
+        row_hmac: String::new(),
+    };
+    let canon = canonical_row_bytes(&row);
+    let hmac = compute_hmac(&deterministic_key(), GENESIS, &canon);
+    let out = KnownAnswerOut {
+        mode: "known_answer",
+        canonical: String::from_utf8(canon.clone()).expect("canonical is utf-8"),
+        hmac,
+        canonical_bytes_len: canon.len(),
+    };
+    println!("{}", serde_json::to_string(&out).unwrap());
+}
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut mode_known_answer = false;
+    let mut audit_path: Option<PathBuf> = None;
+    let mut key_path: Option<PathBuf> = None;
+
+    // Argv parsing fails via stderr+exit, never via panic. The parity
+    // script captures stdout for byte-comparison; a Rust panic backtrace
+    // would land in stdout (the script merges 2>&1) and read as a parity
+    // break instead of a usage error. (Caught 2026-05-03 opus scrum WARN.)
+    let mut i = 1;
+    while i < argv.len() {
+        match argv[i].as_str() {
+            "--known-answer" => {
+                mode_known_answer = true;
+                i += 1;
+            }
+            "--verify" => {
+                let p = argv.get(i + 1).unwrap_or_else(|| die("--verify needs a path"));
+                audit_path = Some(PathBuf::from(p));
+                i += 2;
+            }
+            "--key" => {
+                let p = argv.get(i + 1).unwrap_or_else(|| die("--key needs a path"));
+                key_path = Some(PathBuf::from(p));
+                i += 2;
+            }
+            "-h" | "--help" => {
+                eprintln!("parity_subject_audit --known-answer");
+                eprintln!("parity_subject_audit --verify <audit_log_path> --key <key_path>");
+                std::process::exit(0);
+            }
+            other => {
+                eprintln!("unknown arg: {other}");
+                std::process::exit(2);
+            }
+        }
+    }
+
+    if mode_known_answer {
+        known_answer();
+        return;
+    }
+
+    let audit_path = audit_path.unwrap_or_else(|| die("need --known-answer OR --verify <path> --key <path>"));
+    let key_path = key_path.unwrap_or_else(|| die("--verify also needs --key"));
+
+    let key = std::fs::read(&key_path).unwrap_or_else(|e| die(&format!("read key file: {e}")));
+    let candidate_id = audit_path
+        .file_name()
+        .and_then(|s| s.to_str())
+        .and_then(|s| s.strip_suffix(".audit.jsonl"))
+        .unwrap_or_else(|| die("audit log path must end with <candidate_id>.audit.jsonl"))
+        .to_string();
+
+    // Stand up an in-memory object store, seed it with the audit log
+    // bytes at the canonical key, then ask SubjectAuditWriter to verify.
+    // This way we exercise the SAME verify_chain function the production
+    // gateway calls — not a re-implementation that might drift.
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let bytes = std::fs::read(&audit_path).expect("read audit log");
+    let log_key = format!("_catalog/subjects/{}.audit.jsonl", candidate_id);
+    storaged::ops::put(&store, &log_key, bytes::Bytes::from(bytes))
+        .await
+        .expect("seed object store");
+
+    let writer = SubjectAuditWriter::with_inline_key(store, key);
+    let result = writer.verify_chain(&candidate_id).await;
+    let tip = writer.chain_tip(&candidate_id).await.unwrap_or(GENESIS.into());
+    let out = match result {
+        Ok(count) => VerifyOut {
+            mode: "verify",
+            count,
+            tip,
+            verified: true,
+            error: None,
+        },
+        Err(e) => VerifyOut {
+            mode: "verify",
+            count: 0,
+            tip: GENESIS.into(),
+            verified: false,
+            error: Some(e),
+        },
+    };
+    println!("{}", serde_json::to_string(&out).unwrap());
+}

crates/catalogd/src/bin/retention_sweep.rs

@@ -0,0 +1,376 @@
+//! Daily retention sweep for subject manifests.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7:
+//!   "Subjects whose retention.general_pii_until < now AND status != erased
+//!    get marked for review (don't auto-delete; legal needs to approve)."
+//!
+//! Per shared::types::BiometricConsent doc-comment (BIPA requirement):
+//!   "max 3 years from last interaction. Implementation MUST enforce
+//!    daily expiration sweep against this field."
+//!
+//! Therefore this binary checks BOTH retention clocks and reports any
+//! subject whose general-pii window OR biometric window has expired
+//! while the subject is not already erased / already-flagged.
+//!
+//! It does NOT mutate subject manifests. It produces a JSONL review
+//! queue at:
+//!   data/_catalog/subjects/_retention_sweep_<YYYY-MM-DD>.jsonl
+//!
+//! Operators / legal review the queue and act:
+//!   - extend retention (legal contract update),
+//!   - flip status to retention_expired,
+//!   - schedule erasure.
+//!
+//! Usage:
+//!   cargo run --bin retention_sweep                      # dry-run, stderr only
+//!   cargo run --bin retention_sweep -- --apply           # also write report file
+//!   cargo run --bin retention_sweep -- --as-of 2030-01-01T00:00:00Z
+//!   cargo run --bin retention_sweep -- --storage-root /var/lib/lakehouse/data --apply
+
+use catalogd::registry::Registry;
+use chrono::{DateTime, Utc};
+use serde::Serialize;
+use shared::types::{BiometricConsentStatus, SubjectManifest, SubjectStatus};
+use std::path::PathBuf;
+use std::sync::Arc;
+use storaged::backend;
+
+#[derive(Debug)]
+struct Args {
+    storage_root: PathBuf,
+    apply: bool,
+    as_of: Option<DateTime<Utc>>,
+}
+
+fn parse_args() -> Result<Args, String> {
+    let argv: Vec<String> = std::env::args().collect();
+    let mut storage_root = PathBuf::from("./data");
+    let mut apply = false;
+    let mut as_of: Option<DateTime<Utc>> = None;
+
+    let mut i = 1;
+    while i < argv.len() {
+        let arg = &argv[i];
+        let next = || -> Result<String, String> {
+            argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
+        };
+        match arg.as_str() {
+            "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
+            "--apply" => { apply = true; i += 1; }
+            "--as-of" => {
+                let raw = next()?;
+                as_of = Some(raw.parse().map_err(|e| format!("--as-of (need RFC3339): {e}"))?);
+                i += 2;
+            }
+            "-h" | "--help" => { print_help(); std::process::exit(0); }
+            other => return Err(format!("unknown arg: {other}")),
+        }
+    }
+    Ok(Args { storage_root, apply, as_of })
+}
+
+fn print_help() {
+    eprintln!("retention_sweep — daily retention check on subject manifests");
+    eprintln!();
+    eprintln!("Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7.");
+    eprintln!("Reports overdue subjects to a JSONL review queue. Does NOT");
+    eprintln!("auto-delete. Legal must approve every action.");
+    eprintln!();
+    eprintln!("Flags:");
+    eprintln!("  --storage-root PATH   local-filesystem catalog root (default: ./data)");
+    eprintln!("  --apply               write report to data/_catalog/subjects/_retention_sweep_<date>.jsonl");
+    eprintln!("  --as-of RFC3339       use this timestamp as 'now' (test/forecast helper)");
+}
+
+#[derive(Debug, PartialEq, Clone, Copy)]
+enum OverdueReason {
+    GeneralPii,
+    Biometric,
+    Both,
+}
+
+impl OverdueReason {
+    fn as_str(self) -> &'static str {
+        match self {
+            OverdueReason::GeneralPii => "general_pii_expired",
+            OverdueReason::Biometric => "biometric_expired",
+            OverdueReason::Both => "both_expired",
+        }
+    }
+}
+
+#[derive(Debug, Serialize)]
+struct OverdueRow<'a> {
+    candidate_id: &'a str,
+    status: &'a str,
+    general_pii_until: DateTime<Utc>,
+    days_overdue_general: i64,
+    biometric_status: &'a str,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    biometric_retention_until: Option<DateTime<Utc>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    days_overdue_biometric: Option<i64>,
+    reason: &'a str,
+    swept_at: DateTime<Utc>,
+}
+
+/// Skipped statuses (idempotent across daily runs):
+///   - Erased: subject's data is already gone; nothing to review.
+///   - RetentionExpired: subject is already flagged; do not re-fire.
+///
+/// Actionable statuses: PendingConsent, Active, Withdrawn — each can
+/// still hold PII rows on disk past the retention window.
+fn is_overdue(subject: &SubjectManifest, now: DateTime<Utc>) -> Option<OverdueReason> {
+    if matches!(subject.status, SubjectStatus::Erased | SubjectStatus::RetentionExpired) {
+        return None;
+    }
+    let general_overdue = subject.retention.general_pii_until < now;
+    let biometric_overdue = subject
+        .consent
+        .biometric
+        .retention_until
+        .map(|t| t < now)
+        .unwrap_or(false);
+    match (general_overdue, biometric_overdue) {
+        (true, true) => Some(OverdueReason::Both),
+        (true, false) => Some(OverdueReason::GeneralPii),
+        (false, true) => Some(OverdueReason::Biometric),
+        (false, false) => None,
+    }
+}
+
+fn status_str(s: &SubjectStatus) -> &'static str {
+    match s {
+        SubjectStatus::PendingConsent => "pending_consent",
+        SubjectStatus::Active => "active",
+        SubjectStatus::Withdrawn => "withdrawn",
+        SubjectStatus::RetentionExpired => "retention_expired",
+        SubjectStatus::Erased => "erased",
+    }
+}
+
+fn biometric_status_str(s: &BiometricConsentStatus) -> &'static str {
+    match s {
+        BiometricConsentStatus::NeverCollected => "never_collected",
+        BiometricConsentStatus::Pending => "pending",
+        BiometricConsentStatus::Given => "given",
+        BiometricConsentStatus::Withdrawn => "withdrawn",
+        BiometricConsentStatus::Expired => "expired",
+    }
+}
+
+#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
+async fn main() {
+    let args = match parse_args() {
+        Ok(a) => a,
+        Err(e) => {
+            eprintln!("error: {e}");
+            print_help();
+            std::process::exit(2);
+        }
+    };
+    if let Err(e) = run(args).await {
+        eprintln!("retention_sweep failed: {e}");
+        std::process::exit(1);
+    }
+}
+
+async fn run(args: Args) -> Result<(), String> {
+    let now = args.as_of.unwrap_or_else(Utc::now);
+    eprintln!("[retention_sweep] storage-root: {}", args.storage_root.display());
+    eprintln!("[retention_sweep] now (sweep clock): {}", now.to_rfc3339());
+    eprintln!("[retention_sweep] apply: {}", args.apply);
+
+    let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
+    let reg = Arc::new(Registry::new(store));
+    reg.rebuild().await?;
+
+    let subjects = reg.list_subjects().await;
+    eprintln!("[retention_sweep] subjects loaded: {}", subjects.len());
+
+    let mut overdue_rows: Vec<OverdueRow> = Vec::new();
+    let mut by_reason: std::collections::BTreeMap<&'static str, usize> = Default::default();
+
+    for s in &subjects {
+        if let Some(reason) = is_overdue(s, now) {
+            let reason_str: &'static str = reason.as_str();
+            *by_reason.entry(reason_str).or_insert(0) += 1;
+            overdue_rows.push(OverdueRow {
+                candidate_id: &s.candidate_id,
+                status: status_str(&s.status),
+                general_pii_until: s.retention.general_pii_until,
+                days_overdue_general: (now - s.retention.general_pii_until).num_days(),
+                biometric_status: biometric_status_str(&s.consent.biometric.status),
+                biometric_retention_until: s.consent.biometric.retention_until,
+                days_overdue_biometric: s
+                    .consent
+                    .biometric
+                    .retention_until
+                    .map(|t| (now - t).num_days()),
+                reason: reason_str,
+                swept_at: now,
+            });
+        }
+    }
+
+    eprintln!("[retention_sweep] overdue subjects: {}", overdue_rows.len());
+    for (reason, n) in &by_reason {
+        eprintln!("[retention_sweep]   {reason}: {n}");
+    }
+
+    if overdue_rows.is_empty() {
+        eprintln!("[retention_sweep] nothing to do; all subjects within retention window.");
+        return Ok(());
+    }
+
+    eprintln!("[retention_sweep] sample (first 5):");
+    for row in overdue_rows.iter().take(5) {
+        eprintln!(
+            "  - {} status={} reason={} general_overdue_days={}",
+            row.candidate_id, row.status, row.reason, row.days_overdue_general
+        );
+    }
+
+    if !args.apply {
+        eprintln!();
+        eprintln!("[retention_sweep] DRY-RUN: no file written. Re-run with --apply to persist.");
+        return Ok(());
+    }
+
+    let date_stamp = now.format("%Y-%m-%d");
+    let report_path = args
+        .storage_root
+        .join("_catalog")
+        .join("subjects")
+        .join(format!("_retention_sweep_{date_stamp}.jsonl"));
+    if let Some(parent) = report_path.parent() {
+        std::fs::create_dir_all(parent).map_err(|e| format!("mkdir {parent:?}: {e}"))?;
+    }
+    let mut buf = String::new();
+    for row in &overdue_rows {
+        let line = serde_json::to_string(row).map_err(|e| format!("serialize row: {e}"))?;
+        buf.push_str(&line);
+        buf.push('\n');
+    }
+    std::fs::write(&report_path, &buf).map_err(|e| format!("write {report_path:?}: {e}"))?;
+    eprintln!("[retention_sweep] report written: {}", report_path.display());
+    eprintln!("[retention_sweep] action required by legal/operator. no manifests were mutated.");
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use chrono::Duration;
+    use shared::types::{
+        BiometricConsent, ConsentStatus, GeneralPiiConsent, SubjectConsent, SubjectRetention,
+        SubjectVertical,
+    };
+
+    fn fixture(
+        id: &str,
+        status: SubjectStatus,
+        gen_until: DateTime<Utc>,
+        bio_until: Option<DateTime<Utc>>,
+    ) -> SubjectManifest {
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: id.into(),
+            created_at: Utc::now(),
+            updated_at: Utc::now(),
+            status,
+            vertical: SubjectVertical::General,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::Given,
+                    version: "v1".into(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: if bio_until.is_some() {
+                        BiometricConsentStatus::Given
+                    } else {
+                        BiometricConsentStatus::NeverCollected
+                    },
+                    retention_until: bio_until,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: gen_until,
+                policy: "test".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+            biometric_collection: None,
+        }
+    }
+
+    #[test]
+    fn active_subject_inside_window_not_overdue() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Active, now + Duration::days(30), None);
+        assert!(is_overdue(&s, now).is_none());
+    }
+
+    #[test]
+    fn active_subject_past_general_window_flags_general() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Active, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+
+    #[test]
+    fn erased_subject_never_re_flags_even_if_far_past() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Erased, now - Duration::days(1000), None);
+        assert!(is_overdue(&s, now).is_none(), "erased subjects must not re-fire");
+    }
+
+    #[test]
+    fn already_retention_expired_is_idempotent() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::RetentionExpired, now - Duration::days(1), None);
+        assert!(is_overdue(&s, now).is_none(), "already-flagged subjects must not re-fire");
+    }
+
+    #[test]
+    fn biometric_overdue_alone_triggers_bipa_path() {
+        let now = Utc::now();
+        let s = fixture(
+            "a",
+            SubjectStatus::Active,
+            now + Duration::days(30),
+            Some(now - Duration::days(1)),
+        );
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::Biometric));
+    }
+
+    #[test]
+    fn both_clocks_overdue_reports_both() {
+        let now = Utc::now();
+        let s = fixture(
+            "a",
+            SubjectStatus::Active,
+            now - Duration::days(2),
+            Some(now - Duration::days(1)),
+        );
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::Both));
+    }
+
+    #[test]
+    fn pending_consent_subject_can_still_expire() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::PendingConsent, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+
+    #[test]
+    fn withdrawn_subject_still_subject_to_retention() {
+        let now = Utc::now();
+        let s = fixture("a", SubjectStatus::Withdrawn, now - Duration::days(1), None);
+        assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
+    }
+}

crates/catalogd/src/lib.rs

@@ -2,3 +2,6 @@ pub mod registry;
 pub mod service;
 pub mod grpc;
 pub mod tombstones;
+pub mod subject_audit;
+pub mod audit_endpoint;
+pub mod biometric_endpoint;

crates/catalogd/src/registry.rs

@@ -1,6 +1,6 @@
 use shared::types::{
     AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile,
-    ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone,
+    ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, SubjectManifest, Tombstone,
 };
 
 use crate::tombstones::TombstoneStore;
@@ -66,16 +66,20 @@ pub struct MetadataUpdate {
 
 const MANIFEST_PREFIX: &str = "_catalog/manifests";
 const VIEW_PREFIX: &str = "_catalog/views";
+const SUBJECT_PREFIX: &str = "_catalog/subjects";
 const PROFILE_PREFIX: &str = "_catalog/profiles";
 
 /// In-memory dataset registry backed by manifest persistence in object storage.
 /// Also tracks AiViews (Phase D) — safe projections over base datasets.
 /// And tombstones (Phase E) — soft-delete markers applied at query time.
+/// And subject manifests (compliance/access metadata layer) — see
+/// docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
 #[derive(Clone)]
 pub struct Registry {
     datasets: Arc<RwLock<HashMap<DatasetId, DatasetManifest>>>,
     views: Arc<RwLock<HashMap<String, AiView>>>,
     profiles: Arc<RwLock<HashMap<String, ModelProfile>>>,
+    subjects: Arc<RwLock<HashMap<String, SubjectManifest>>>,
     tombstones: TombstoneStore,
     store: Arc<dyn ObjectStore>,
 }
@@ -86,6 +90,7 @@ impl Registry {
             datasets: Arc::new(RwLock::new(HashMap::new())),
             views: Arc::new(RwLock::new(HashMap::new())),
             profiles: Arc::new(RwLock::new(HashMap::new())),
+            subjects: Arc::new(RwLock::new(HashMap::new())),
             tombstones: TombstoneStore::new(store.clone()),
             store,
         }
@@ -179,6 +184,47 @@ impl Registry {
             tracing::info!("catalog: {} model profiles loaded", profiles.len());
         }
 
+        // Subject manifests (compliance/access metadata layer).
+        let subject_keys = ops::list(&self.store, Some(SUBJECT_PREFIX)).await.unwrap_or_default();
+        let mut subjects = self.subjects.write().await;
+        subjects.clear();
+        for key in &subject_keys {
+            if !key.ends_with(".json") { continue; }
+            let data = match ops::get(&self.store, key).await {
+                Ok(d) => d,
+                Err(e) => {
+                    tracing::warn!("subject '{key}': read failed: {e}");
+                    continue;
+                }
+            };
+            match serde_json::from_slice::<SubjectManifest>(&data) {
+                // Match put_subject: in-memory key is the SANITIZED form
+                // so get_subject lookups match the storage key shape.
+                Ok(subj) => {
+                    let map_key = sanitize_view_name(&subj.candidate_id);
+                    // Rebuild collision detection — symmetric with the
+                    // put_subject collision guard. If pre-existing storage
+                    // somehow contains two manifests whose raw ids
+                    // sanitize to the same key, log loudly so operators
+                    // know one was dropped instead of silently losing it.
+                    // (2026-05-03 opus scrum WARN registry.rs:201.)
+                    if let Some(existing) = subjects.get(&map_key) {
+                        if existing.candidate_id != subj.candidate_id {
+                            tracing::warn!(
+                                "subject rebuild collision on sanitized key '{map_key}': raw '{}' was loaded earlier; raw '{}' is being silently overwritten in memory. Run dedupe + delete the duplicate manifest file at storage.",
+                                existing.candidate_id, subj.candidate_id,
+                            );
+                        }
+                    }
+                    subjects.insert(map_key, subj);
+                }
+                Err(e) => tracing::warn!("subject '{key}': parse failed: {e}"),
+            }
+        }
+        if !subjects.is_empty() {
+            tracing::info!("catalog: {} subject manifests loaded", subjects.len());
+        }
+
         Ok(count)
     }
 
@@ -686,6 +732,94 @@ impl Registry {
         Ok(())
     }
 
+    // --- Subject manifests (compliance/access metadata layer) ---
+    // Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
+    // Mirrors the put_view / get_view / list_views / delete_view shape.
+    // The audit log per subject is owned separately by SubjectAuditWriter
+    // (Step 2) — this layer just persists the manifest itself.
+
+    /// Create or replace a subject manifest. Validates that referenced
+    /// datasets exist in the catalog (fail-fast so dangling references
+    /// don't accumulate). Persists to `_catalog/subjects/<id>.json`.
+    ///
+    /// **Storage-key collision guard** (2026-05-03 opus scrum WARN
+    /// at registry.rs:735): the storage key is `sanitize_view_name(id)`
+    /// — `/`, `:`, etc. become `_`. A naive in-memory keying by raw
+    /// candidate_id would let two distinct ids (`CAND/1` + `CAND_1`)
+    /// collide on disk while appearing distinct in memory; the second
+    /// put silently overwrites the first, and `rebuild` only loads one.
+    /// Both the in-memory map AND the storage key now use the sanitized
+    /// form. Caller's raw candidate_id is preserved on the manifest for
+    /// audit-trail integrity (the manifest body has the original).
+    /// We surface a clear error if two raw ids would collide on the
+    /// sanitized key.
+    pub async fn put_subject(&self, mut subj: SubjectManifest) -> Result<SubjectManifest, String> {
+        if subj.candidate_id.is_empty() {
+            return Err("subject candidate_id is empty".into());
+        }
+        for ds in &subj.datasets {
+            if self.get_by_name(&ds.name).await.is_none() {
+                return Err(format!(
+                    "subject '{}' references dataset '{}' which is not in the catalog",
+                    subj.candidate_id, ds.name,
+                ));
+            }
+        }
+        let now = chrono::Utc::now();
+        if subj.created_at.timestamp() == 0 {
+            subj.created_at = now;
+        }
+        subj.updated_at = now;
+
+        let map_key = sanitize_view_name(&subj.candidate_id);
+        let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
+
+        // Collision guard: if the sanitized key already exists for a
+        // DIFFERENT raw candidate_id, refuse the put. Better to fail
+        // loudly than silently overwrite an existing subject.
+        {
+            let subjects = self.subjects.read().await;
+            if let Some(existing) = subjects.get(&map_key) {
+                if existing.candidate_id != subj.candidate_id {
+                    return Err(format!(
+                        "subject id '{}' collides with existing subject '{}' on sanitized storage key '{}'",
+                        subj.candidate_id, existing.candidate_id, map_key,
+                    ));
+                }
+            }
+        }
+
+        let json = serde_json::to_vec_pretty(&subj).map_err(|e| e.to_string())?;
+        ops::put(&self.store, &key, json.into()).await?;
+
+        let mut subjects = self.subjects.write().await;
+        subjects.insert(map_key, subj.clone());
+        tracing::debug!("subject manifest persisted: {}", subj.candidate_id);
+        Ok(subj)
+    }
+
+    pub async fn get_subject(&self, candidate_id: &str) -> Option<SubjectManifest> {
+        self.subjects.read().await.get(&sanitize_view_name(candidate_id)).cloned()
+    }
+
+    pub async fn list_subjects(&self) -> Vec<SubjectManifest> {
+        self.subjects.read().await.values().cloned().collect()
+    }
+
+    pub async fn delete_subject(&self, candidate_id: &str) -> Result<(), String> {
+        let map_key = sanitize_view_name(candidate_id);
+        let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
+        ops::delete(&self.store, &key).await?;
+        self.subjects.write().await.remove(&map_key);
+        Ok(())
+    }
+
+    /// Subject count — convenience for the daily retention sweep + audit
+    /// dashboards that don't need the full list.
+    pub async fn subjects_count(&self) -> usize {
+        self.subjects.read().await.len()
+    }
+
     /// Remove a dataset from the catalog by name. Deletes the manifest
     /// from both the in-memory registry and object storage.
     ///
@@ -1027,4 +1161,130 @@ mod tests {
         assert!(reg.get(&loser_a).await.is_none());
         assert!(reg.get(&loser_b).await.is_none());
     }
+
+    // --- Subject manifests (Step 1 of subject_manifests_on_catalogd) ---
+
+    fn fixture_subject(id: &str) -> shared::types::SubjectManifest {
+        use shared::types::{
+            BiometricConsent, BiometricConsentStatus, ConsentStatus,
+            GeneralPiiConsent, SubjectConsent, SubjectManifest, SubjectRetention,
+            SubjectStatus, SubjectVertical,
+        };
+        SubjectManifest {
+            schema: "subject_manifest.v1".into(),
+            candidate_id: id.to_string(),
+            created_at: chrono::Utc::now(),
+            updated_at: chrono::Utc::now(),
+            status: SubjectStatus::Active,
+            vertical: SubjectVertical::Unknown,
+            consent: SubjectConsent {
+                general_pii: GeneralPiiConsent {
+                    status: ConsentStatus::PendingBackfillReview,
+                    version: String::new(),
+                    given_at: None,
+                    withdrawn_at: None,
+                },
+                biometric: BiometricConsent {
+                    status: BiometricConsentStatus::NeverCollected,
+                    retention_until: None,
+                },
+            },
+            retention: SubjectRetention {
+                general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
+                policy: "4_year_default".into(),
+            },
+            datasets: vec![],
+            safe_views: vec![],
+            audit_log_path: String::new(),
+            audit_log_chain_root: String::new(),
+                biometric_collection: None,
+        }
+    }
+
+    #[tokio::test]
+    async fn put_subject_with_no_dataset_refs_succeeds() {
+        let reg = fixture();
+        let s = fixture_subject("CAND-000001");
+        let stored = reg.put_subject(s).await.unwrap();
+        assert_eq!(stored.candidate_id, "CAND-000001");
+        assert_eq!(reg.subjects_count().await, 1);
+    }
+
+    #[tokio::test]
+    async fn put_subject_rejects_dangling_dataset_ref() {
+        use shared::types::SubjectDatasetRef;
+        let reg = fixture();
+        let mut s = fixture_subject("CAND-000002");
+        s.datasets.push(SubjectDatasetRef {
+            name: "no_such_dataset".into(),
+            key_column: "candidate_id".into(),
+            key_value: "CAND-000002".into(),
+        });
+        let err = reg.put_subject(s).await.unwrap_err();
+        assert!(err.contains("not in the catalog"), "got: {err}");
+    }
+
+    #[tokio::test]
+    async fn put_subject_with_valid_dataset_ref_succeeds() {
+        use shared::types::SubjectDatasetRef;
+        let reg = fixture();
+        // Register a dataset first so the subject's reference resolves.
+        reg.register("workers".into(), fp("w1"), vec![obj("w.parquet")]).await.unwrap();
+
+        let mut s = fixture_subject("CAND-000003");
+        s.datasets.push(SubjectDatasetRef {
+            name: "workers".into(),
+            key_column: "candidate_id".into(),
+            key_value: "CAND-000003".into(),
+        });
+        let stored = reg.put_subject(s).await.unwrap();
+        assert_eq!(stored.datasets.len(), 1);
+        assert_eq!(stored.datasets[0].name, "workers");
+    }
+
+    #[tokio::test]
+    async fn subject_round_trips_through_object_store() {
+        let store = Arc::new(object_store::memory::InMemory::new());
+        let reg = Registry::new(store.clone());
+        reg.put_subject(fixture_subject("CAND-RT-001")).await.unwrap();
+
+        // Open a fresh registry over the same store and load → confirms
+        // persistence + load_all path actually round-trips the manifest.
+        let reg2 = Registry::new(store);
+        reg2.rebuild().await.unwrap();
+        let loaded = reg2.get_subject("CAND-RT-001").await;
+        assert!(loaded.is_some(), "subject not loaded from store");
+        assert_eq!(loaded.unwrap().status, shared::types::SubjectStatus::Active);
+    }
+
+    #[tokio::test]
+    async fn put_subject_rejects_sanitize_collision() {
+        // 2026-05-03 opus scrum WARN at registry.rs:735 regression:
+        // CAND/1 and CAND_1 sanitize to the same storage key. Without
+        // the collision guard, the second put silently overwrites the
+        // first on disk; with it, the second put fails loudly.
+        let reg = fixture();
+        let s1 = fixture_subject("CAND/1");
+        reg.put_subject(s1).await.unwrap();
+        let s2 = fixture_subject("CAND_1"); // sanitizes to same key
+        let err = reg.put_subject(s2).await.unwrap_err();
+        assert!(err.contains("collides"), "got: {err}");
+        assert_eq!(reg.subjects_count().await, 1, "first subject must survive");
+    }
+
+    #[tokio::test]
+    async fn delete_subject_removes_in_memory_and_persistence() {
+        let store = Arc::new(object_store::memory::InMemory::new());
+        let reg = Registry::new(store.clone());
+        reg.put_subject(fixture_subject("CAND-DEL-001")).await.unwrap();
+        assert_eq!(reg.subjects_count().await, 1);
+
+        reg.delete_subject("CAND-DEL-001").await.unwrap();
+        assert_eq!(reg.subjects_count().await, 0);
+
+        // Fresh registry over same store — confirm it's also gone from persistence.
+        let reg2 = Registry::new(store);
+        reg2.rebuild().await.unwrap();
+        assert_eq!(reg2.subjects_count().await, 0);
+    }
 }

crates/catalogd/src/subject_audit.rs

@@ -0,0 +1,587 @@
+//! Subject audit log writer with HMAC chain.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §3.
+//!
+//! Per-subject append-only JSONL at `_catalog/subjects/<id>.audit.jsonl`.
+//! Each row's `row_hmac` is HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac).
+//! The chain root is mirrored to the SubjectManifest.audit_log_chain_root
+//! field after each append, so verifiers can find the latest valid root
+//! from the manifest alone.
+//!
+//! Failures are non-blocking — audit-write errors are logged but do NOT
+//! propagate to the caller (per spec §3.2: "better to leak a row than
+//! block legitimate operations"). Operators MUST monitor the error count.
+//!
+//! Key loading: from a sealed file at the path the writer is constructed
+//! with. Tests use `with_inline_key` to inject a deterministic key.
+
+use bytes::Bytes;
+use hmac::{Hmac, Mac};
+use object_store::ObjectStore;
+use sha2::Sha256;
+use shared::types::SubjectAuditRow;
+use std::collections::HashMap;
+use std::sync::Arc;
+use storaged::ops;
+use tokio::sync::Mutex;
+
+const SUBJECT_PREFIX: &str = "_catalog/subjects";
+pub const GENESIS_HASH: &str = "GENESIS";
+
+type HmacSha256 = Hmac<Sha256>;
+
+/// Render a JSON value canonically: object keys sorted alphabetically,
+/// no insignificant whitespace, no trailing newline. Recursive — nested
+/// objects also sort their keys. Arrays preserve order (semantically
+/// significant). Numbers, strings, booleans, nulls pass through.
+///
+/// This is intentionally narrower than RFC 8785 (no number normalization,
+/// no UTF-8 NFC re-encoding). For our use — hashing structurally simple
+/// audit-row JSON — alphabetical key order is the only canonicalization
+/// that matters: it makes the byte sequence stable across struct field
+/// reordering and additive schema evolution.
+fn canonical_json(v: &serde_json::Value) -> Result<Vec<u8>, String> {
+    use std::collections::BTreeMap;
+    fn rewrite(v: &serde_json::Value) -> serde_json::Value {
+        match v {
+            serde_json::Value::Object(map) => {
+                let sorted: BTreeMap<String, serde_json::Value> = map
+                    .iter()
+                    .map(|(k, v)| (k.clone(), rewrite(v)))
+                    .collect();
+                serde_json::Value::Object(sorted.into_iter().collect())
+            }
+            serde_json::Value::Array(arr) => {
+                serde_json::Value::Array(arr.iter().map(rewrite).collect())
+            }
+            other => other.clone(),
+        }
+    }
+    let canonical = rewrite(v);
+    serde_json::to_vec(&canonical).map_err(|e| e.to_string())
+}
+
+/// Per-subject audit log writer. Holds the signing key, a small
+/// in-memory cache of latest chain hash per subject, and per-subject
+/// mutexes that serialize concurrent appends to the same subject's log.
+///
+/// **No Debug impl deliberately** — auto-deriving Debug on this struct
+/// would risk leaking the signing key into log lines. Tests that need
+/// `Result::unwrap_err` on a function returning `Result<Self, _>` must
+/// match on the result instead of using `.unwrap_err()`.
+///
+/// Concurrency: appends to DIFFERENT subjects run in parallel; appends
+/// to the SAME subject serialize via a per-candidate Mutex. Without
+/// per-subject locking the read-modify-write append_line race
+/// (concurrent reads of the same `prev_hash` + concurrent writes
+/// overwriting each other) silently corrupts the chain. Caught by
+/// 2026-05-03 opus scrum BLOCK on subject_audit.rs:172.
+#[derive(Clone)]
+pub struct SubjectAuditWriter {
+    store: Arc<dyn ObjectStore>,
+    /// HMAC signing key. Loaded once at construction. Never logged.
+    /// `Vec<u8>` not `String` — keys are bytes, not text.
+    key: Arc<Vec<u8>>,
+    /// In-memory cache of the latest chain hash per candidate_id.
+    /// Loaded lazily from the audit JSONL on first append per subject.
+    latest_hash: Arc<Mutex<HashMap<String, String>>>,
+    /// Per-subject mutexes. Append acquires the subject's lock before
+    /// the read-modify-write to serialize concurrent appends to the
+    /// same subject. Different subjects don't contend.
+    subject_locks: Arc<Mutex<HashMap<String, Arc<Mutex<()>>>>>,
+}
+
+impl SubjectAuditWriter {
+    /// Construct a writer that loads its signing key from a file at
+    /// `key_path`. The file MUST be mode 0400 owner-only on production
+    /// hosts (operator's responsibility — not enforced here because the
+    /// permission check would be a footgun in tests/dev).
+    pub async fn from_key_file(
+        store: Arc<dyn ObjectStore>,
+        key_path: &std::path::Path,
+    ) -> Result<Self, String> {
+        let key = tokio::fs::read(key_path)
+            .await
+            .map_err(|e| format!("read signing key from {}: {e}", key_path.display()))?;
+        if key.len() < 32 {
+            return Err(format!(
+                "signing key is {} bytes; recommend ≥32 bytes for HMAC-SHA256",
+                key.len()
+            ));
+        }
+        Ok(Self {
+            store,
+            key: Arc::new(key),
+            latest_hash: Arc::new(Mutex::new(HashMap::new())),
+            subject_locks: Arc::new(Mutex::new(HashMap::new())),
+        })
+    }
+
+    /// Construct a writer with an inline key. Intended for tests + bring-up.
+    /// Production should use `from_key_file`.
+    pub fn with_inline_key(store: Arc<dyn ObjectStore>, key: Vec<u8>) -> Self {
+        Self {
+            store,
+            key: Arc::new(key),
+            latest_hash: Arc::new(Mutex::new(HashMap::new())),
+            subject_locks: Arc::new(Mutex::new(HashMap::new())),
+        }
+    }
+
+    fn audit_key(candidate_id: &str) -> String {
+        let safe: String = candidate_id
+            .chars()
+            .map(|c| {
+                if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' {
+                    c
+                } else {
+                    '_'
+                }
+            })
+            .collect();
+        format!("{SUBJECT_PREFIX}/{}.audit.jsonl", safe)
+    }
+
+    /// Compute HMAC-SHA256(key, prev_hash_bytes || canonical_row_bytes).
+    /// Returns the HMAC as lowercase hex. Uses byte-level hex formatting
+    /// in a single pass (avoids per-byte format! allocations).
+    fn compute_hmac(&self, prev_hash: &str, canonical_row: &[u8]) -> String {
+        let mut mac = <HmacSha256 as Mac>::new_from_slice(&self.key)
+            .expect("HMAC accepts any key length");
+        mac.update(prev_hash.as_bytes());
+        mac.update(canonical_row);
+        let result = mac.finalize().into_bytes();
+        const HEX: &[u8; 16] = b"0123456789abcdef";
+        let mut s = String::with_capacity(64);
+        for byte in result {
+            s.push(HEX[(byte >> 4) as usize] as char);
+            s.push(HEX[(byte & 0x0f) as usize] as char);
+        }
+        s
+    }
+
+    /// Render an audit row as canonical JSON: keys sorted alphabetically,
+    /// no insignificant whitespace, row_hmac field excluded. This is the
+    /// byte sequence the HMAC is computed over.
+    ///
+    /// **Why a manual canonicalizer instead of `serde_json::to_vec`:**
+    /// serde_json serializes struct fields in DECLARATION order. If the
+    /// SubjectAuditRow struct ever gains/loses/reorders fields (schema
+    /// evolution), the bytes change even though the LOGICAL content is
+    /// the same. A verifier compiled against the new struct shape
+    /// re-serializes a stored row and gets DIFFERENT bytes than the
+    /// original write — chain breaks silently. Caught by 2026-05-03
+    /// opus scrum BLOCK on subject_audit.rs:108.
+    ///
+    /// Canonical-JSON (RFC 8785-ish, simplified for our scalar types)
+    /// fixes this by using alphabetical key order + minimal whitespace,
+    /// stable across struct evolution.
+    fn canonical_row_bytes(row: &SubjectAuditRow) -> Result<Vec<u8>, String> {
+        // Project to a generic Value, drop row_hmac, then re-serialize
+        // through a BTreeMap so keys come out alphabetical.
+        let mut v: serde_json::Value = serde_json::to_value(row)
+            .map_err(|e| format!("audit row to value: {e}"))?;
+        if let Some(obj) = v.as_object_mut() {
+            obj.remove("row_hmac");
+        }
+        canonical_json(&v).map_err(|e| format!("canonicalize: {e}"))
+    }
+
+    /// Get-or-create per-subject lock. Held across the read-modify-write
+    /// in append() so concurrent appends to the same subject serialize.
+    async fn lock_for(&self, candidate_id: &str) -> Arc<Mutex<()>> {
+        let mut locks = self.subject_locks.lock().await;
+        locks
+            .entry(candidate_id.to_string())
+            .or_insert_with(|| Arc::new(Mutex::new(())))
+            .clone()
+    }
+
+    /// Append one audit row for a subject. Computes the HMAC chain link,
+    /// writes to the per-subject JSONL, returns the new chain root (which
+    /// the caller MAY mirror to SubjectManifest.audit_log_chain_root).
+    ///
+    /// **Concurrency**: appends to the SAME subject serialize via a
+    /// per-subject Mutex. Appends to DIFFERENT subjects run in parallel.
+    /// Without the per-subject lock, the read-modify-write in
+    /// append_line races and silently corrupts the chain. (2026-05-03
+    /// opus scrum BLOCK on subject_audit.rs:172.)
+    ///
+    /// On error: logs and returns Err. Caller decides whether to propagate
+    /// (per spec §3.2 the gateway tool registry SHOULD log + drop).
+    pub async fn append(&self, mut row: SubjectAuditRow) -> Result<String, String> {
+        let cid = row.candidate_id.clone();
+        if cid.is_empty() {
+            return Err("audit row candidate_id is empty".into());
+        }
+
+        // Acquire the per-subject lock for the duration of this append.
+        let subject_lock = self.lock_for(&cid).await;
+        let _guard = subject_lock.lock().await;
+
+        // 1. Find prev_chain_hash. Cache hot path; cold path scans
+        //    the existing JSONL tail-style to find the last row.
+        let prev = {
+            let cache = self.latest_hash.lock().await;
+            cache.get(&cid).cloned()
+        };
+        let prev_hash = match prev {
+            Some(h) => h,
+            None => self.scan_latest_hash(&cid).await,
+        };
+        row.prev_chain_hash = prev_hash.clone();
+        row.row_hmac = String::new(); // Always cleared before HMAC computation.
+
+        // 2. Canonical-JSON the row (sorted keys, no whitespace, no
+        //    row_hmac), then HMAC. Stable across struct evolution.
+        let canon = Self::canonical_row_bytes(&row)?;
+        let new_hmac = self.compute_hmac(&prev_hash, &canon);
+        row.row_hmac = new_hmac.clone();
+
+        // 3. Append the row to the JSONL. The per-subject lock above
+        //    serializes the read-modify-write inside append_line.
+        let line = serde_json::to_vec(&row)
+            .map_err(|e| format!("serialize audit row: {e}"))?;
+        self.append_line(&cid, &line).await?;
+
+        // 4. Update cache.
+        self.latest_hash.lock().await.insert(cid, new_hmac.clone());
+
+        Ok(new_hmac)
+    }
+
+    /// Scan the audit JSONL for a subject and return the last row's
+    /// row_hmac. GENESIS if no log exists or it's empty.
+    /// Object-store-aware: works across LocalFileSystem + InMemory + S3.
+    async fn scan_latest_hash(&self, candidate_id: &str) -> String {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return GENESIS_HASH.to_string(), // file doesn't exist
+        };
+        // Find the last newline-delimited line that parses as a row.
+        let text = match std::str::from_utf8(&bytes) {
+            Ok(t) => t,
+            Err(_) => return GENESIS_HASH.to_string(),
+        };
+        for line in text.lines().rev() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            if let Ok(parsed) = serde_json::from_str::<SubjectAuditRow>(line) {
+                return parsed.row_hmac;
+            }
+        }
+        GENESIS_HASH.to_string()
+    }
+
+    /// Append a single newline-terminated line to the per-subject log.
+    /// Reads the existing content (may be empty), appends, writes back.
+    /// Object stores don't have native append; this is the standard
+    /// "read-modify-write the tail file" pattern.
+    async fn append_line(&self, candidate_id: &str, line: &[u8]) -> Result<(), String> {
+        let key = Self::audit_key(candidate_id);
+        let existing = ops::get(&self.store, &key).await.unwrap_or_default();
+        let mut buf = Vec::with_capacity(existing.len() + line.len() + 1);
+        buf.extend_from_slice(&existing);
+        if !buf.is_empty() && !buf.ends_with(b"\n") {
+            buf.push(b'\n');
+        }
+        buf.extend_from_slice(line);
+        buf.push(b'\n');
+        ops::put(&self.store, &key, Bytes::from(buf)).await
+    }
+
+    /// Read all audit rows for a subject, optionally filtered by time
+    /// window (inclusive bounds). Returns rows in append order. Used by
+    /// the /audit/subject/{id} endpoint to assemble the legal-tier
+    /// response. Window bounds are optional — `from=None`+`to=None`
+    /// returns the full log.
+    pub async fn read_rows_in_range(
+        &self,
+        candidate_id: &str,
+        from: Option<chrono::DateTime<chrono::Utc>>,
+        to: Option<chrono::DateTime<chrono::Utc>>,
+    ) -> Result<Vec<SubjectAuditRow>, String> {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return Ok(Vec::new()), // no audit log = empty result
+        };
+        let text = std::str::from_utf8(&bytes)
+            .map_err(|e| format!("audit log not utf-8: {e}"))?;
+        let mut out = Vec::new();
+        for line in text.lines() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            let row: SubjectAuditRow = match serde_json::from_str(line) {
+                Ok(r) => r,
+                Err(_) => continue, // skip unparseable lines (defensive)
+            };
+            if let Some(from) = from {
+                if row.ts < from { continue; }
+            }
+            if let Some(to) = to {
+                if row.ts > to { continue; }
+            }
+            out.push(row);
+        }
+        Ok(out)
+    }
+
+    /// Return the latest row_hmac in this subject's audit log, or None
+    /// if the log doesn't exist / has no rows. Cache-aware: hits the
+    /// in-memory cache when the writer has appended at least once for
+    /// this subject in the current process, otherwise scans the log
+    /// tail (single object_store read).
+    ///
+    /// Used by /audit/subject/{id} to echo the actual chain tip in the
+    /// response. Taking the tip from a windowed slice of rows would
+    /// lie when the window doesn't cover the latest row.
+    pub async fn chain_tip(&self, candidate_id: &str) -> Option<String> {
+        // Cache hit?
+        if let Some(h) = self.latest_hash.lock().await.get(candidate_id).cloned() {
+            return Some(h);
+        }
+        // Cold path — scan the log tail. Returns GENESIS for empty/missing
+        // logs; we coerce that to None so callers can distinguish "no audit
+        // activity yet" from "real chain tip exists".
+        let tip = self.scan_latest_hash(candidate_id).await;
+        if tip == GENESIS_HASH { None } else { Some(tip) }
+    }
+
+    /// Verify the full HMAC chain for a subject. Returns Ok(rows_verified)
+    /// or Err with the first chain break encountered.
+    ///
+    /// Special case: if no audit log exists (the subject has had no PII
+    /// access yet), returns Ok(0) — an empty log is trivially valid.
+    /// Callers that want to detect "log was deleted" should compare
+    /// against SubjectManifest.audit_log_chain_root: if the manifest's
+    /// chain root is non-empty/non-GENESIS but verify_chain returns 0,
+    /// the file was tampered.
+    pub async fn verify_chain(&self, candidate_id: &str) -> Result<usize, String> {
+        let key = Self::audit_key(candidate_id);
+        let bytes = match ops::get(&self.store, &key).await {
+            Ok(b) => b,
+            Err(_) => return Ok(0), // no log = 0 rows = trivially valid
+        };
+        let text = std::str::from_utf8(&bytes)
+            .map_err(|e| format!("audit log not utf-8: {e}"))?;
+        let mut prev = GENESIS_HASH.to_string();
+        let mut count = 0usize;
+        for (lineno, line) in text.lines().enumerate() {
+            if line.trim().is_empty() {
+                continue;
+            }
+            let mut row: SubjectAuditRow = serde_json::from_str(line)
+                .map_err(|e| format!("line {} parse failed: {e}", lineno + 1))?;
+            if row.prev_chain_hash != prev {
+                return Err(format!(
+                    "chain break at line {}: prev_chain_hash={} expected={}",
+                    lineno + 1, row.prev_chain_hash, prev,
+                ));
+            }
+            let claimed = std::mem::take(&mut row.row_hmac);
+            let canon = Self::canonical_row_bytes(&row)
+                .map_err(|e| format!("canonicalize line {}: {e}", lineno + 1))?;
+            let recomputed = self.compute_hmac(&prev, &canon);
+            if recomputed != claimed {
+                return Err(format!(
+                    "hmac mismatch at line {}: stored={} recomputed={}",
+                    lineno + 1, claimed, recomputed,
+                ));
+            }
+            prev = claimed;
+            count += 1;
+        }
+        Ok(count)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use object_store::memory::InMemory;
+    use shared::types::AuditAccessor;
+
+    fn fixture_writer() -> SubjectAuditWriter {
+        SubjectAuditWriter::with_inline_key(
+            Arc::new(InMemory::new()),
+            // Deterministic 32-byte key for test reproducibility.
+            (0u8..32).collect(),
+        )
+    }
+
+    fn fixture_row(candidate_id: &str, fields: &[&str]) -> SubjectAuditRow {
+        SubjectAuditRow {
+            schema: "subject_audit.v1".into(),
+            ts: chrono::Utc::now(),
+            candidate_id: candidate_id.into(),
+            accessor: AuditAccessor {
+                kind: "gateway_lookup".into(),
+                daemon: "gateway".into(),
+                purpose: "fill_validation".into(),
+                trace_id: String::new(),
+            },
+            fields_accessed: fields.iter().map(|s| s.to_string()).collect(),
+            result: "success".into(),
+            prev_chain_hash: String::new(), // filled in by writer
+            row_hmac: String::new(),         // filled in by writer
+        }
+    }
+
+    #[tokio::test]
+    async fn first_append_uses_genesis_prev_hash() {
+        let w = fixture_writer();
+        let new_root = w.append(fixture_row("CAND-1", &["name"])).await.unwrap();
+        assert!(!new_root.is_empty());
+        assert_eq!(new_root.len(), 64); // SHA-256 hex
+        // After first append, the only row should reference GENESIS as prev.
+        let bytes = ops::get(&w.store, &SubjectAuditWriter::audit_key("CAND-1"))
+            .await.unwrap();
+        let text = std::str::from_utf8(&bytes).unwrap();
+        let row: SubjectAuditRow = serde_json::from_str(text.trim()).unwrap();
+        assert_eq!(row.prev_chain_hash, "GENESIS");
+        assert_eq!(row.row_hmac, new_root);
+    }
+
+    #[tokio::test]
+    async fn chain_links_each_append() {
+        let w = fixture_writer();
+        let h1 = w.append(fixture_row("CAND-2", &["name"])).await.unwrap();
+        let h2 = w.append(fixture_row("CAND-2", &["phone"])).await.unwrap();
+        let h3 = w.append(fixture_row("CAND-2", &["email"])).await.unwrap();
+        assert_ne!(h1, h2);
+        assert_ne!(h2, h3);
+        let count = w.verify_chain("CAND-2").await.unwrap();
+        assert_eq!(count, 3);
+    }
+
+    #[tokio::test]
+    async fn separate_subjects_have_independent_chains() {
+        let w = fixture_writer();
+        let _ = w.append(fixture_row("CAND-A", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-B", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-A", &["phone"])).await.unwrap();
+        assert_eq!(w.verify_chain("CAND-A").await.unwrap(), 2);
+        assert_eq!(w.verify_chain("CAND-B").await.unwrap(), 1);
+    }
+
+    #[tokio::test]
+    async fn tamper_detected_on_verify() {
+        let w = fixture_writer();
+        let _ = w.append(fixture_row("CAND-T", &["name"])).await.unwrap();
+        let _ = w.append(fixture_row("CAND-T", &["phone"])).await.unwrap();
+        // Mutate the file to flip a `result` field on the first row.
+        let key = SubjectAuditWriter::audit_key("CAND-T");
+        let bytes = ops::get(&w.store, &key).await.unwrap();
+        let mut text = std::str::from_utf8(&bytes).unwrap().to_string();
+        text = text.replacen("\"success\"", "\"denied\"", 1);
+        ops::put(&w.store, &key, text.into_bytes().into()).await.unwrap();
+        let err = w.verify_chain("CAND-T").await.unwrap_err();
+        assert!(err.contains("hmac mismatch"), "expected hmac mismatch, got: {err}");
+    }
+
+    #[tokio::test]
+    async fn cold_writer_picks_up_existing_chain() {
+        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        let key: Vec<u8> = (0u8..32).collect();
+        let w1 = SubjectAuditWriter::with_inline_key(store.clone(), key.clone());
+        let h1 = w1.append(fixture_row("CAND-COLD", &["name"])).await.unwrap();
+        // Drop w1's cache, simulate process restart.
+        drop(w1);
+        let w2 = SubjectAuditWriter::with_inline_key(store.clone(), key);
+        let h2 = w2.append(fixture_row("CAND-COLD", &["phone"])).await.unwrap();
+        assert_ne!(h1, h2);
+        assert_eq!(w2.verify_chain("CAND-COLD").await.unwrap(), 2);
+    }
+
+    #[tokio::test]
+    async fn empty_candidate_id_rejected() {
+        let w = fixture_writer();
+        let row = fixture_row("", &["name"]);
+        let err = w.append(row).await.unwrap_err();
+        assert!(err.contains("empty"), "got: {err}");
+    }
+
+    #[tokio::test]
+    async fn concurrent_appends_to_same_subject_serialize() {
+        // 2026-05-03 opus scrum BLOCK regression: without per-subject
+        // locking, concurrent appends raced and silently lost rows.
+        // This test spawns 50 parallel appends and asserts ALL 50
+        // land with an intact chain.
+        let w = Arc::new(fixture_writer());
+        let mut handles = Vec::new();
+        for i in 0..50 {
+            let w = w.clone();
+            handles.push(tokio::spawn(async move {
+                w.append(fixture_row("CAND-RACE", &[&format!("f{i}")])).await
+            }));
+        }
+        for h in handles {
+            h.await.unwrap().unwrap();
+        }
+        let count = w.verify_chain("CAND-RACE").await.unwrap();
+        assert_eq!(count, 50, "expected 50 rows; chain was corrupted");
+    }
+
+    #[tokio::test]
+    async fn concurrent_appends_to_different_subjects_run_parallel() {
+        let w = Arc::new(fixture_writer());
+        let mut handles = Vec::new();
+        for i in 0..20 {
+            let w = w.clone();
+            handles.push(tokio::spawn(async move {
+                w.append(fixture_row(&format!("CAND-PARA-{i}"), &["f"])).await
+            }));
+        }
+        for h in handles {
+            h.await.unwrap().unwrap();
+        }
+        // Each of the 20 subjects has exactly 1 row.
+        for i in 0..20 {
+            assert_eq!(w.verify_chain(&format!("CAND-PARA-{i}")).await.unwrap(), 1);
+        }
+    }
+
+    #[test]
+    fn canonical_json_sorts_keys_alphabetically() {
+        let v = serde_json::json!({"z": 1, "a": 2, "m": {"y": 1, "b": 2}});
+        let bytes = canonical_json(&v).unwrap();
+        let s = std::str::from_utf8(&bytes).unwrap();
+        // Keys should appear in alphabetical order at every nesting level.
+        let a_pos = s.find("\"a\"").unwrap();
+        let m_pos = s.find("\"m\"").unwrap();
+        let z_pos = s.find("\"z\"").unwrap();
+        assert!(a_pos < m_pos);
+        assert!(m_pos < z_pos);
+        // Nested object's keys also sorted.
+        let b_pos = s.find("\"b\"").unwrap();
+        let y_pos = s.find("\"y\"").unwrap();
+        assert!(b_pos < y_pos);
+    }
+
+    #[test]
+    fn canonical_json_arrays_preserve_order() {
+        let v = serde_json::json!({"k": ["c", "a", "b"]});
+        let bytes = canonical_json(&v).unwrap();
+        let s = std::str::from_utf8(&bytes).unwrap();
+        assert!(s.contains("\"c\",\"a\",\"b\""), "got: {s}");
+    }
+
+    #[tokio::test]
+    async fn key_too_short_rejected_via_file() {
+        // Write a 16-byte key file (under the 32-byte minimum).
+        let tmp = std::env::temp_dir().join(format!("sa_key_short_{}.bin", std::process::id()));
+        tokio::fs::write(&tmp, vec![0u8; 16]).await.unwrap();
+        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        // `unwrap_err` would require Debug on SubjectAuditWriter, which
+        // we deliberately don't impl (key-leak risk). Match instead.
+        match SubjectAuditWriter::from_key_file(store, &tmp).await {
+            Err(e) => assert!(e.contains("≥32 bytes"), "got: {e}"),
+            Ok(_) => panic!("expected too-short-key rejection"),
+        }
+        let _ = tokio::fs::remove_file(&tmp).await;
+    }
+}

crates/gateway/Cargo.toml

@@ -13,6 +13,7 @@ ingestd = { path = "../ingestd" }
 vectord = { path = "../vectord" }
 journald = { path = "../journald" }
 truth = { path = "../truth" }
+validator = { path = "../validator" }
 tokio = { workspace = true }
 axum = { workspace = true }
 serde = { workspace = true }

crates/gateway/src/bin/parity_extract_json.rs

@@ -0,0 +1,37 @@
+//! Cross-runtime parity helper for `extract_json`.
+//!
+//! Reads a single model-output string from stdin, runs the Rust
+//! extract_json, prints `{"matched": bool, "value": <object|null>}`
+//! to stdout as JSON. Exit 0 on success, exit 1 on internal error.
+//!
+//! The Go counterpart lives at
+//! `golangLAKEHOUSE/internal/validator/iterate.go::ExtractJSON`. The
+//! parity probe at
+//! `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`
+//! feeds the same fixtures through both and diffs the outputs.
+//!
+//! Usage:
+//!   echo '<raw model output>' | parity_extract_json
+//!   parity_extract_json <<< '...'
+
+use std::io::Read;
+
+fn main() {
+    let mut buf = String::new();
+    if let Err(e) = std::io::stdin().read_to_string(&mut buf) {
+        eprintln!("read stdin: {e}");
+        std::process::exit(1);
+    }
+    let result = gateway::v1::iterate::extract_json(&buf);
+    let body = serde_json::json!({
+        "matched": result.is_some(),
+        "value": result.unwrap_or(serde_json::Value::Null),
+    });
+    match serde_json::to_string(&body) {
+        Ok(s) => println!("{s}"),
+        Err(e) => {
+            eprintln!("serialize result: {e}");
+            std::process::exit(1);
+        }
+    }
+}

crates/gateway/src/bin/parity_session_log.rs

@@ -0,0 +1,71 @@
+//! Cross-runtime parity helper for `SessionRecord` JSON shape.
+//!
+//! Reads a fixture JSON on stdin, builds a `SessionRecord`, emits
+//! one JSONL row on stdout. Used by
+//! `golangLAKEHOUSE/scripts/cutover/parity/session_log_parity.sh`
+//! to verify the Rust gateway's session log shape stays byte-equal
+//! to the Go-side validatord's `validator.SessionRecord` (commit
+//! 1a3a82a in golangLAKEHOUSE).
+
+use gateway::v1::session_log::{SessionAttemptRecord, SessionRecord, SESSION_RECORD_SCHEMA};
+use serde::Deserialize;
+use std::io::Read;
+
+#[derive(Deserialize)]
+struct FixtureInput {
+    session_id: String,
+    kind: String,
+    model: String,
+    provider: String,
+    prompt: String,
+    iterations: u32,
+    max_iterations: u32,
+    final_verdict: String,
+    attempts: Vec<SessionAttemptRecord>,
+    #[serde(default)]
+    artifact: Option<serde_json::Value>,
+    #[serde(default)]
+    grounded_in_roster: Option<bool>,
+    duration_ms: u64,
+}
+
+fn main() {
+    let mut buf = String::new();
+    if let Err(e) = std::io::stdin().read_to_string(&mut buf) {
+        eprintln!("read stdin: {e}");
+        std::process::exit(1);
+    }
+    let input: FixtureInput = match serde_json::from_str(&buf) {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!("parse stdin: {e}");
+            std::process::exit(1);
+        }
+    };
+    let rec = SessionRecord {
+        schema: SESSION_RECORD_SCHEMA.to_string(),
+        session_id: input.session_id,
+        // Pinned timestamp so both runtimes' rows compare byte-equal
+        // when the test wrapper normalizes on `daemon` only.
+        timestamp: "2026-01-01T00:00:00+00:00".to_string(),
+        daemon: "gateway".to_string(),
+        kind: input.kind,
+        model: input.model,
+        provider: input.provider,
+        prompt: input.prompt,
+        iterations: input.iterations,
+        max_iterations: input.max_iterations,
+        final_verdict: input.final_verdict,
+        attempts: input.attempts,
+        artifact: input.artifact,
+        grounded_in_roster: input.grounded_in_roster,
+        duration_ms: input.duration_ms,
+    };
+    match serde_json::to_string(&rec) {
+        Ok(s) => println!("{s}"),
+        Err(e) => {
+            eprintln!("marshal: {e}");
+            std::process::exit(1);
+        }
+    }
+}

crates/gateway/src/execution_loop/mod.rs

@@ -212,6 +212,13 @@ impl ExecutionLoop {
             if let Action::ToolCall { tool, args, .. } = &exec_action {
                 match self.dispatch_tool(tool, args).await {
                     Ok(result) => {
+                        // Subject-audit hook (per Phase 1.6 SUBJECT_MANIFESTS spec
+                        // §3.2): every tool result that returns rows referencing a
+                        // subject must produce an audit row. Failures are logged
+                        // but never block the tool result — the spec is explicit:
+                        // "better to leak a row than block legitimate operations."
+                        self.audit_subject_hits_in(&result, tool).await;
+
                         let trimmed = trim_result(&result);
                         self.append(LogEntry::new(
                             turn, "executor", &executor_model, "tool_result", trimmed,
@@ -341,6 +348,68 @@ impl ExecutionLoop {
         self.log.push(e);
     }
 
+    /// Subject-audit hook: scan a tool result JSON for subject identifiers,
+    /// fire one audit row per (subject, fields) pair via the writer in
+    /// V1State.subject_audit. No-op if writer is None (audit disabled).
+    ///
+    /// Conventions (matches the backfill binary in catalogd/src/bin/
+    /// backfill_subjects.rs):
+    /// - "candidate_id":"X"   → audit candidate_id="X"
+    /// - "worker_id":42       → audit candidate_id="WORKER-42"
+    ///
+    /// Walks the result JSON recursively. For each object encountered
+    /// that contains a candidate_id or worker_id field, fires one audit
+    /// row capturing all the OTHER scalar field names as fields_accessed
+    /// (so the audit log records "this access surfaced name + phone for
+    /// candidate X" — sufficient for compliance review).
+    ///
+    /// Failures are logged but NEVER propagated. Per spec §3.2: an audit
+    /// gap is preferable to blocking the tool response.
+    async fn audit_subject_hits_in(&self, result: &serde_json::Value, tool: &str) {
+        let Some(audit) = self.state.subject_audit.clone() else {
+            return; // audit disabled — no-op
+        };
+        // Collect (candidate_id, fields_accessed) pairs from the JSON.
+        let mut hits: Vec<(String, Vec<String>)> = Vec::new();
+        collect_subject_hits(result, &mut hits);
+        if hits.is_empty() {
+            return;
+        }
+        // Determine result-state from the payload shape, NOT from the
+        // transport status. A tool can return Ok(Value) where the body
+        // itself is `{"error":"..."}` — hardcoding "success" here would
+        // mislead compliance review (opus WARN at execution_loop:401).
+        let result_state = audit_result_state(result);
+
+        // Sequential await — NOT tokio::spawn. The previous spawn-per-row
+        // shape fanned out concurrent writes that raced inside the
+        // per-subject append. The audit writer now serializes per-subject
+        // internally, but spawning would still flood those locks under
+        // burst load with no observable latency win. Inline await is
+        // simpler AND correct. (2026-05-03 opus scrum BLOCKs on
+        // execution_loop:391 + subject_audit:172.)
+        for (cid, fields) in hits {
+            let row = shared::types::SubjectAuditRow {
+                schema: "subject_audit.v1".into(),
+                ts: chrono::Utc::now(),
+                candidate_id: cid.clone(),
+                accessor: shared::types::AuditAccessor {
+                    kind: "gateway_lookup".into(),
+                    daemon: "gateway".into(),
+                    purpose: tool.to_string(),
+                    trace_id: String::new(), // TODO: thread X-Lakehouse-Trace-Id through
+                },
+                fields_accessed: fields,
+                result: result_state.clone(),
+                prev_chain_hash: String::new(),
+                row_hmac: String::new(),
+            };
+            if let Err(e) = audit.append(row).await {
+                tracing::warn!("subject audit write failed for {cid}: {e}");
+            }
+        }
+    }
+
     /// Dispatch: model name prefix → provider.
     /// Local path uses Phase 21 `generate_continuable` (auto-continuation,
     /// retry on empty thinking-model response). Cloud path hits
@@ -438,6 +507,10 @@ impl ExecutionLoop {
                 start_time: start_time.to_rfc3339(),
                 end_time: end_time.to_rfc3339(),
                 latency_ms: elapsed_ms,
+                // Internal execution-loop traffic is its own top-level
+                // trace per call. If a future caller threads a parent
+                // trace into self.state, lift this to Some(parent_id).
+                parent_trace_id: None,
             });
         }
 
@@ -582,10 +655,10 @@ impl ExecutionLoop {
     /// Phase 20 step (8) — T3 overseer escalation.
     ///
     /// When the local executor/reviewer loop can't self-correct, call
-    /// the cloud overseer (`gpt-oss:120b` via Ollama Cloud) with (a)
-    /// the KB context — recent outcomes + prior corrections for this
-    /// sig_hash + task_class, across every profile that has run it —
-    /// and (b) the recent log tail. Its output is appended as a
+    /// the cloud overseer (`claude-opus-4-7` via OpenCode Zen) with
+    /// (a) the KB context — recent outcomes + prior corrections for
+    /// this sig_hash + task_class, across every profile that has run
+    /// it — and (b) the recent log tail. Its output is appended as a
     /// `system` role turn so the next executor generation sees it,
     /// AND written to `data/_kb/overseer_corrections.jsonl` so every
     /// future profile activation reads from the same learning pool.
@@ -593,64 +666,70 @@ impl ExecutionLoop {
     /// This is the "pipe to the overviewer" piece from 2026-04-23 —
     /// the overseer is now a first-class KB consumer AND producer, not
     /// a one-shot correction oracle.
+    ///
+    /// 2026-04-28: routed through OpenCode (Zen tier) for Claude Opus
+    /// 4.7. Frontier reasoning matters here because the overseer fires
+    /// only after local self-correction has failed twice — by that
+    /// point we need the strongest reasoning available, not the
+    /// cheapest token. Frequency is low so the Zen pay-per-token cost
+    /// stays bounded.
     async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> {
-        let Some(cloud_key) = self.state.ollama_cloud_key.clone() else {
-            return Err("OLLAMA_CLOUD_KEY not configured — skipping escalation".into());
-        };
-
+        // 2026-05-03: REVERTED to local-only per PRD line 70. Cloud
+        // overseer (opencode/claude-opus-4-7) was a recent addition that
+        // moved a hot-path call OFF the local Ollama runtime onto a paid
+        // cloud provider. Reverted to local Ollama (qwen3.5:latest).
+        // Cloud overseer can be re-enabled by setting LH_OVERSEER_CLOUD=1
+        // for development; production stays local.
         let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await;
         let prompt = build_overseer_prompt(&self.req, &kb, &self.log, reason);
 
         let started = std::time::Instant::now();
         let start_time = chrono::Utc::now();
         let chat_req = crate::v1::ChatRequest {
-            model: "gpt-oss:120b".to_string(),
+            model: "qwen3.5:latest".to_string(),
             messages: vec![crate::v1::Message::new_text("user", prompt.clone())],
             temperature: Some(0.1),
             max_tokens: None,
             stream: Some(false),
-            think: Some(true),    // overseer KEEPS thinking (Phase 20 rule)
-            provider: Some("ollama_cloud".into()),
+            think: Some(false),
+            provider: Some("ollama".into()),
         };
-        let resp = crate::v1::ollama_cloud::chat(&cloud_key, &chat_req).await
-            .map_err(|e| format!("ollama_cloud: {e}"))?;
+        let resp = crate::v1::ollama::chat(&self.state.ai_client, &chat_req).await
+            .map_err(|e| format!("ollama overseer: {e}"))?;
         let latency_ms = started.elapsed().as_millis() as u64;
         let end_time = chrono::Utc::now();
         let correction_text: String = resp.choices.into_iter().next()
             .map(|c| c.message.text()).unwrap_or_default();
 
-        // Stamp per-task stats — cloud call counts against the same
-        // usage counter so `/v1/usage` shows cloud token spend too.
         self.stats.requests = self.stats.requests.saturating_add(1);
         self.stats.prompt_tokens = self.stats.prompt_tokens.saturating_add(resp.usage.prompt_tokens as u64);
         self.stats.completion_tokens = self.stats.completion_tokens.saturating_add(resp.usage.completion_tokens as u64);
         self.stats.total_tokens = self.stats.total_tokens.saturating_add(resp.usage.total_tokens as u64);
         self.stats.latency_ms = self.stats.latency_ms.saturating_add(latency_ms);
 
-        // Langfuse trace for the overseer call (same pipe that feeds
-        // the observer/KB, so this correction's cost lands in the KB
-        // too — closing the loop).
+        // Langfuse trace for the overseer call (local-only now).
         if let Some(lf) = &self.state.langfuse {
             use crate::v1::langfuse_trace::ChatTrace;
             lf.emit_chat(ChatTrace {
-                provider: "ollama_cloud".into(),
-                model: "gpt-oss:120b".into(),
+                provider: "ollama".into(),
+                model: "qwen3.5:latest".into(),
                 input: vec![crate::v1::Message::new_text("user", prompt.clone())],
                 output: correction_text.clone(),
                 prompt_tokens: resp.usage.prompt_tokens,
                 completion_tokens: resp.usage.completion_tokens,
                 temperature: Some(0.1),
                 max_tokens: None,
-                think: Some(true),
+                think: Some(false),
                 start_time: start_time.to_rfc3339(),
                 end_time: end_time.to_rfc3339(),
                 latency_ms,
+                parent_trace_id: None,
             });
         }
 
         // Append to the transcript so the next executor turn sees it.
         self.append(LogEntry::new(
-            turn, "system", "gpt-oss:120b", "overseer_correction",
+            turn, "system", "qwen3.5:latest", "overseer_correction",
             serde_json::json!({
                 "reason": reason,
                 "correction": correction_text,
@@ -672,7 +751,7 @@ impl ExecutionLoop {
             "task_class": self.req.task_class,
             "operation": self.req.operation,
             "reason": reason,
-            "model": "gpt-oss:120b",
+            "model": "qwen3.5:latest",
             "correction": correction_text,
             "applied_at_turn": turn,
             "kb_context_used": kb,
@@ -974,6 +1053,119 @@ async fn append_outcomes_row_at(
 
 /// PORT FROM orchestrator.ts:306-311. Cap `rows` at 20 entries and
 /// annotate the truncation so the executor sees it on the next turn
+/// Inspect a tool result JSON payload and return the appropriate
+/// audit `result` field per spec §3.2 enumeration:
+/// "success" | "denied" | "not_found" | "error".
+///
+/// Heuristic on the top-level object:
+/// - body has `"error"` key (truthy)             → "error"
+/// - body has `"denied"` key (truthy)            → "denied"
+/// - body has `"not_found"` key (truthy)         → "not_found"
+/// - body has `"status":"denied"`/etc            → that status
+/// - otherwise                                    → "success"
+///
+/// Conservative — only flags explicit signals, defaults to success.
+/// Keeps audit log honest about ambiguous payloads (an error inside
+/// rows[*] doesn't downgrade the whole call to "error" — that's a
+/// row-level concern, not a result-state concern).
+pub(crate) fn audit_result_state(v: &serde_json::Value) -> String {
+    if let Some(obj) = v.as_object() {
+        // Tighter than before (2026-05-03 opus scrum INFO at
+        // execution_loop:1085): only flag error/denied/not_found when
+        // the payload doesn't ALSO have a rows/data/results sibling.
+        // A successful tool whose response includes an "error" field
+        // describing one row's problem (out of many) shouldn't taint
+        // the whole call as "error" — that's a row-level concern, not
+        // a result-state concern.
+        let has_data_sibling = obj.contains_key("rows")
+            || obj.contains_key("data")
+            || obj.contains_key("results");
+        if !has_data_sibling {
+            if obj.get("error").map(is_truthy).unwrap_or(false) {
+                return "error".into();
+            }
+            if obj.get("denied").map(is_truthy).unwrap_or(false) {
+                return "denied".into();
+            }
+            if obj.get("not_found").map(is_truthy).unwrap_or(false) {
+                return "not_found".into();
+            }
+        }
+        // status is always authoritative — if a tool explicitly says
+        // status=denied that's not a row-level concern.
+        if let Some(s) = obj.get("status").and_then(|v| v.as_str()) {
+            match s {
+                "denied" | "not_found" | "error" => return s.into(),
+                _ => {}
+            }
+        }
+    }
+    "success".into()
+}
+
+fn is_truthy(v: &serde_json::Value) -> bool {
+    match v {
+        serde_json::Value::Null => false,
+        serde_json::Value::Bool(b) => *b,
+        serde_json::Value::String(s) => !s.is_empty(),
+        serde_json::Value::Array(a) => !a.is_empty(),
+        serde_json::Value::Object(o) => !o.is_empty(),
+        serde_json::Value::Number(n) => n.as_f64().map(|f| f != 0.0).unwrap_or(false),
+    }
+}
+
+/// Walk a tool-result JSON value recursively. For each object that
+/// contains a candidate_id (string) or worker_id (int/string), append
+/// (canonical_candidate_id, fields_in_object) to `out`. Conventions
+/// match the backfill binary: bare candidate_id stays as-is; worker_id
+/// gets prefixed with "WORKER-".
+///
+/// fields_in_object captures the OTHER scalar field names found in the
+/// same object — so the audit row records "this access surfaced
+/// {name, phone} for candidate X". The id field itself is NOT included
+/// in fields_accessed (it's the subject identifier, not PII data).
+pub(crate) fn collect_subject_hits(
+    v: &serde_json::Value,
+    out: &mut Vec<(String, Vec<String>)>,
+) {
+    match v {
+        serde_json::Value::Object(map) => {
+            // Detect a subject identifier in this object's direct fields.
+            let mut subject_id: Option<String> = None;
+            if let Some(cid) = map.get("candidate_id").and_then(|v| v.as_str()) {
+                if !cid.is_empty() { subject_id = Some(cid.to_string()); }
+            } else if let Some(wid) = map.get("worker_id") {
+                if let Some(s) = wid.as_str() {
+                    if !s.is_empty() { subject_id = Some(format!("WORKER-{s}")); }
+                } else if let Some(n) = wid.as_i64() {
+                    subject_id = Some(format!("WORKER-{n}"));
+                } else if let Some(n) = wid.as_u64() {
+                    subject_id = Some(format!("WORKER-{n}"));
+                }
+            }
+            if let Some(cid) = subject_id {
+                let fields: Vec<String> = map
+                    .keys()
+                    .filter(|k| *k != "candidate_id" && *k != "worker_id")
+                    .cloned()
+                    .collect();
+                out.push((cid, fields));
+            }
+            // Recurse — nested objects may carry additional subjects
+            // (e.g. join results with both candidate + recruiter rows).
+            for v in map.values() {
+                collect_subject_hits(v, out);
+            }
+        }
+        serde_json::Value::Array(arr) => {
+            for v in arr {
+                collect_subject_hits(v, out);
+            }
+        }
+        _ => {}
+    }
+}
+
 /// prompt — prevents a 1000-row hybrid_search result from wiping the
 /// context budget on a single tool call.
 fn trim_result(r: &serde_json::Value) -> serde_json::Value {
@@ -1852,4 +2044,123 @@ mod tests {
         let p = build_reviewer_prompt(&req, &log);
         assert!(!p.contains("HARD RULE"));
     }
+
+    // --- Subject-audit hook tests ---
+
+    #[test]
+    fn collect_subject_hits_finds_candidate_id_strings() {
+        let v = serde_json::json!({
+            "rows": [
+                {"candidate_id": "CAND-001", "name": "x", "phone": "y"},
+                {"candidate_id": "CAND-002", "name": "z"},
+            ]
+        });
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 2);
+        assert_eq!(hits[0].0, "CAND-001");
+        // Field order from JSON object is preserved in serde_json::Map.
+        assert!(hits[0].1.contains(&"name".to_string()));
+        assert!(hits[0].1.contains(&"phone".to_string()));
+        assert!(!hits[0].1.contains(&"candidate_id".to_string()));
+        assert_eq!(hits[1].0, "CAND-002");
+    }
+
+    #[test]
+    fn collect_subject_hits_prefixes_worker_id_int() {
+        let v = serde_json::json!({"rows": [{"worker_id": 42, "name": "Alice"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 1);
+        assert_eq!(hits[0].0, "WORKER-42");
+        assert_eq!(hits[0].1, vec!["name".to_string()]);
+    }
+
+    #[test]
+    fn collect_subject_hits_handles_worker_id_string() {
+        let v = serde_json::json!({"worker_id": "42", "name": "Bob"});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 1);
+        assert_eq!(hits[0].0, "WORKER-42");
+    }
+
+    #[test]
+    fn collect_subject_hits_recurses_through_nested_objects() {
+        let v = serde_json::json!({
+            "outer": {
+                "candidate_id": "CAND-A",
+                "linked": {
+                    "worker_id": 7,
+                    "role": "welder"
+                }
+            }
+        });
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 2);
+        let ids: Vec<&str> = hits.iter().map(|(c, _)| c.as_str()).collect();
+        assert!(ids.contains(&"CAND-A"));
+        assert!(ids.contains(&"WORKER-7"));
+    }
+
+    #[test]
+    fn collect_subject_hits_ignores_objects_without_id_fields() {
+        let v = serde_json::json!({"rows": [{"name": "x"}, {"role": "y"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 0);
+    }
+
+    #[test]
+    fn collect_subject_hits_skips_empty_id_strings() {
+        let v = serde_json::json!({"rows": [{"candidate_id": "", "name": "x"}]});
+        let mut hits = Vec::new();
+        collect_subject_hits(&v, &mut hits);
+        assert_eq!(hits.len(), 0);
+    }
+
+    #[test]
+    fn audit_result_state_default_is_success() {
+        let v = serde_json::json!({"rows": [{"candidate_id": "X"}]});
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_detects_error_payload() {
+        let v = serde_json::json!({"error": "not found"});
+        assert_eq!(audit_result_state(&v), "error");
+    }
+
+    #[test]
+    fn audit_result_state_detects_status_field() {
+        let v = serde_json::json!({"status": "denied", "rows": []});
+        assert_eq!(audit_result_state(&v), "denied");
+    }
+
+    #[test]
+    fn audit_result_state_ignores_falsy_signals() {
+        // Empty/null/false should not trigger error state.
+        let v = serde_json::json!({"error": null, "denied": false, "rows": [{"candidate_id":"X"}]});
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_does_not_classify_error_when_data_sibling_present() {
+        // 2026-05-03 opus scrum INFO at execution_loop:1085 regression:
+        // a tool that returns rows BUT also has an "error" field describing
+        // one bad row shouldn't taint the whole call as "error".
+        let v = serde_json::json!({
+            "rows": [{"candidate_id": "OK"}, {"candidate_id": "BAD"}],
+            "error": "1 row had a malformed certification"
+        });
+        assert_eq!(audit_result_state(&v), "success");
+    }
+
+    #[test]
+    fn audit_result_state_status_is_authoritative_even_with_data() {
+        // Explicit status="denied" wins over data-sibling rule.
+        let v = serde_json::json!({"status": "denied", "rows": []});
+        assert_eq!(audit_result_state(&v), "denied");
+    }
 }

crates/gateway/src/lib.rs

@@ -0,0 +1,19 @@
+//! Library facade for the gateway crate so sub-binaries (e.g.
+//! `parity_extract_json`) can reuse the same modules the gateway
+//! binary uses.
+//!
+//! Added 2026-05-02 to support the cross-runtime parity probe at
+//! `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`.
+//! `extract_json` is the load-bearing public surface for that probe.
+//!
+//! main.rs still uses local `mod foo;` declarations independently —
+//! adding this file is purely additive (the binary's module tree is
+//! unchanged).
+
+pub mod access;
+pub mod access_service;
+pub mod auth;
+pub mod execution_loop;
+pub mod observability;
+pub mod tools;
+pub mod v1;

crates/gateway/src/main.rs

@@ -95,8 +95,35 @@ async fn main() {
         tracing::warn!("workspace rebuild: {e}");
     }
 
-    // AI sidecar client
-    let ai_client = aibridge::client::AiClient::new(&config.sidecar.url);
+    // AI sidecar clients — Phase 44 part 3 (2026-04-27).
+    //
+    // Two flavors of the same client:
+    // - `ai_client_direct` posts directly to ${sidecar}/generate. Used
+    //   inside the gateway by V1State + the legacy /ai proxy. These
+    //   call sites are themselves the implementation of /v1/chat
+    //   (or its sidecar shim), so routing them through /v1/chat
+    //   would self-loop.
+    // - `ai_client_observable` posts via ${gateway}/v1/chat with
+    //   provider="ollama". Used by vectord modules (autotune agent,
+    //   /vectors service) so their LLM calls land in /v1/usage and
+    //   Langfuse traces. Adds one localhost HTTP hop per call (~ms);
+    //   accepted for the observability gain.
+    //
+    // The gateway can call its own /v1/chat over localhost during
+    // boot's transient period because we don't fire any LLM calls
+    // until the listener is up — the observable client is just
+    // configured here, not exercised.
+    let ai_client_direct = aibridge::client::AiClient::new(&config.sidecar.url);
+    let gateway_self_url = format!("http://{}:{}", config.gateway.host, config.gateway.port);
+    let ai_client_observable = aibridge::client::AiClient::new_with_gateway(
+        &config.sidecar.url,
+        &gateway_self_url,
+    );
+    // Backwards-compat alias for the (many) existing references in this file.
+    // Defaults to direct so the existing wiring (V1State, /ai proxy)
+    // keeps its non-self-loop transport. New vectord wiring below
+    // explicitly uses ai_client_observable.
+    let ai_client = ai_client_direct.clone();
 
     // Vector service components — built before the router because both the
     // /vectors service AND ingestd need the agent handle to enqueue triggers.
@@ -134,7 +161,9 @@ async fn main() {
         agent_cfg,
         vectord::agent::AgentDeps {
             store: store.clone(),
-            ai_client: ai_client.clone(),
+            // Observable: autotune agent's LLM calls go through
+            // /v1/chat for /v1/usage + Langfuse visibility.
+            ai_client: ai_client_observable.clone(),
             catalog: registry.clone(),
             index_registry: index_reg.clone(),
             hnsw_store: hnsw.clone(),
@@ -189,7 +218,9 @@ async fn main() {
         }))
         .nest("/vectors", vectord::service::router(vectord::service::VectorState {
             store: store.clone(),
-            ai_client: ai_client.clone(),
+            // Observable: /vectors service's LLM calls (RAG, summary,
+            // playbook synthesis, etc.) flow through /v1/chat.
+            ai_client: ai_client_observable.clone(),
             job_tracker: vectord::jobs::JobTracker::new(),
             index_registry: index_reg.clone(),
             hnsw_store: hnsw,
@@ -224,6 +255,33 @@ async fn main() {
         // surface over the existing aibridge → Ollama path. Future
         // phases add provider adapters (39), routing engine (40),
         // session state (41), etc. All without changing this mount.
+        ;
+
+    // Build subject-audit writer BEFORE the V1State literal so the
+    // validate_workers wrapper can reference it. None = audit disabled
+    // (key file missing or unreadable); the gateway still serves PII
+    // paths but produces no audit trail per spec §3.2.
+    let subject_audit_writer: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>> = {
+        let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY")
+            .unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into());
+        match catalogd::subject_audit::SubjectAuditWriter::from_key_file(
+            store.clone(), std::path::Path::new(&key_path)
+        ).await {
+            Ok(w) => {
+                tracing::info!("v1: subject audit log enabled (key: {})", key_path);
+                Some(std::sync::Arc::new(w))
+            }
+            Err(e) => {
+                tracing::warn!(
+                    "v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)",
+                    e
+                );
+                None
+            }
+        }
+    };
+
+    app = app
         .nest("/v1", v1::router(v1::V1State {
             ai_client: ai_client.clone(),
             usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
@@ -271,6 +329,63 @@ async fn main() {
                 }
                 k
             },
+            kimi_key: {
+                // Direct Kimi For Coding (api.kimi.com) — bypasses the
+                // broken-upstream kimi-k2:1t and OpenRouter rate caps.
+                // Key from /etc/lakehouse/kimi.env (KIMI_API_KEY=sk-kimi-…).
+                let k = v1::kimi::resolve_kimi_key();
+                if k.is_some() {
+                    tracing::info!("v1: Kimi key loaded — /v1/chat provider=kimi enabled (model=kimi-for-coding)");
+                } else {
+                    tracing::debug!("v1: no Kimi key — provider=kimi will 503");
+                }
+                k
+            },
+            opencode_key: {
+                // OpenCode GO multi-vendor gateway — Claude Opus 4.7,
+                // GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+                // Qwen + free-tier. Key from /etc/lakehouse/opencode.env.
+                let k = v1::opencode::resolve_opencode_key();
+                if k.is_some() {
+                    tracing::info!("v1: OpenCode key loaded — /v1/chat provider=opencode enabled (40 models)");
+                } else {
+                    tracing::debug!("v1: no OpenCode key — provider=opencode will 503");
+                }
+                k
+            },
+            validate_workers: {
+                // Load workers_500k.parquet snapshot for /v1/validate.
+                // Path overridable via LH_WORKERS_PARQUET env. Missing
+                // file is non-fatal — validators run schema/PII checks
+                // unaffected; only worker-existence checks fail clean.
+                let path_str = std::env::var("LH_WORKERS_PARQUET")
+                    .unwrap_or_else(|_| "/home/profit/lakehouse/data/datasets/workers_500k.parquet".into());
+                let path = std::path::Path::new(&path_str);
+                let raw: std::sync::Arc<dyn validator::WorkerLookup> = if path.exists() {
+                    match validator::staffing::parquet_lookup::load_workers_parquet(path) {
+                        Ok(lookup) => {
+                            tracing::info!("v1: workers parquet loaded from {} — /v1/validate worker-existence checks enabled", path_str);
+                            lookup
+                        }
+                        Err(e) => {
+                            tracing::warn!("v1: workers parquet at {} unreadable ({e}) — /v1/validate worker-existence checks will fail Consistency", path_str);
+                            std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
+                        }
+                    }
+                } else {
+                    tracing::warn!("v1: workers parquet at {} not found — /v1/validate worker-existence checks will fail Consistency", path_str);
+                    std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
+                };
+                // Wrap with subject-audit decorator. When subject_audit_writer
+                // is None, the wrapper is a transparent passthrough — no
+                // audit, no overhead. When Some, every find() call fires an
+                // audit row via the per-subject HMAC chain. (Step 5 of
+                // SUBJECT_MANIFESTS_ON_CATALOGD spec.)
+                std::sync::Arc::new(v1::auditing_worker_lookup::AuditingWorkerLookup::new(
+                    raw,
+                    subject_audit_writer.clone(),
+                ))
+            },
             // Phase 40 early deliverable — Langfuse trace emitter.
             // Defaults match mcp-server/tracing.ts conventions so
             // gateway traces land in the same staffing project.
@@ -283,8 +398,75 @@ async fn main() {
                 }
                 c
             },
+            // Coordinator session JSONL — one row per /v1/iterate
+            // session for offline DuckDB analysis. Cross-runtime
+            // parity with Go-side validatord (commit 1a3a82a).
+            session_log: {
+                let path = &config.gateway.session_log_path;
+                let s = v1::session_log::SessionLogger::from_path(path);
+                if s.is_some() {
+                    tracing::info!(
+                        "v1: session log enabled — coordinator sessions written to {}",
+                        path
+                    );
+                } else {
+                    tracing::info!("v1: session log disabled (set [gateway].session_log_path to enable)");
+                }
+                s
+            },
+            // subject_audit was constructed BEFORE the V1State literal
+            // (see `subject_audit_writer` above) so validate_workers
+            // could wrap with it. Reuse the same Arc here.
+            subject_audit: subject_audit_writer.clone(),
         }));
 
+    // Step 6 of SUBJECT_MANIFESTS_ON_CATALOGD spec — /audit/subject/{id}
+    // legal-tier endpoint. Only mounted when BOTH the audit writer AND
+    // the legal token file are present. Disabled-by-default keeps the
+    // audit surface from accidentally serving without proper credentials
+    // in dev / bring-up environments.
+    if let Some(writer) = subject_audit_writer.clone() {
+        let legal_token_path = std::env::var("LH_LEGAL_AUDIT_TOKEN_FILE")
+            .unwrap_or_else(|_| "/etc/lakehouse/legal_audit.token".into());
+        let audit_state = catalogd::audit_endpoint::AuditEndpointState::new(
+            registry.clone(),
+            writer.clone(),
+            std::path::Path::new(&legal_token_path),
+        ).await;
+        app = app.nest("/audit", catalogd::audit_endpoint::router(audit_state));
+        tracing::info!("audit endpoint mounted at /audit (legal token: {})", legal_token_path);
+
+        // Phase 1.6 Gate 3 — biometric photo upload endpoint. Same
+        // legal-token gate as /audit (writes to legal-tier storage are
+        // privileged). Storage root defaults to ./data/biometric/uploads
+        // unless overridden via LH_BIOMETRIC_STORAGE_ROOT (operators on
+        // multi-host deployments will want a dedicated mount point).
+        let biometric_storage_root: std::path::PathBuf = std::env::var("LH_BIOMETRIC_STORAGE_ROOT")
+            .map(std::path::PathBuf::from)
+            .unwrap_or_else(|_| std::path::PathBuf::from("./data/biometric/uploads"));
+        // consent_versions allowlist — counsel-tier deployment safety.
+        // Defaults to /etc/lakehouse/consent_versions.json; override
+        // via LH_CONSENT_VERSIONS_FILE. Missing file → permissive
+        // mode (any non-empty hash accepted, the v1 behavior).
+        let consent_versions_path: std::path::PathBuf = std::env::var("LH_CONSENT_VERSIONS_FILE")
+            .map(std::path::PathBuf::from)
+            .unwrap_or_else(|_| std::path::PathBuf::from("/etc/lakehouse/consent_versions.json"));
+        let biometric_state = catalogd::biometric_endpoint::BiometricEndpointState::new(
+            registry.clone(),
+            writer,
+            std::path::Path::new(&legal_token_path),
+            biometric_storage_root.clone(),
+        ).await
+            .with_consent_versions(&consent_versions_path).await;
+        app = app.nest("/biometric", catalogd::biometric_endpoint::router(biometric_state));
+        tracing::info!(
+            "biometric endpoint mounted at /biometric (storage_root: {}, legal token: {}, consent_versions file: {})",
+            biometric_storage_root.display(), legal_token_path, consent_versions_path.display()
+        );
+    } else {
+        tracing::warn!("/audit + /biometric endpoints NOT mounted — subject_audit writer is None (no signing key)");
+    }
+
     // Auth middleware (if enabled) — P5-001 fix 2026-04-23:
     // previously only inserted the ApiKey as an extension and never layered
     // the middleware, so auth.enabled=true enforced nothing. Now wraps the

crates/gateway/src/v1/auditing_worker_lookup.rs

@@ -0,0 +1,194 @@
+//! Validator WorkerLookup decorator that fires per-find audit rows.
+//!
+//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5.
+//!
+//! Wraps an inner `dyn WorkerLookup`. Every `find(candidate_id)` call
+//! produces one `SubjectAuditRow` via the bundled `SubjectAuditWriter`,
+//! tagged with `purpose="validator_worker_lookup"`. Audit-write failures
+//! are logged only — never propagated to the caller (per spec §3.2:
+//! audit-side gaps are preferable to blocking validation).
+//!
+//! **Why a decorator and not a trait extension**: `WorkerLookup` is a
+//! sync trait by design (validator's contract is "in-memory snapshot,
+//! no per-call I/O"). Audit writes are async. The decorator bridges
+//! them by spawning the audit append onto the current Tokio runtime
+//! handle. Per-subject sequencing inside the writer ensures the chain
+//! stays intact even with spawned writes (regression-tested at
+//! catalogd::subject_audit::tests::concurrent_appends_to_same_subject_serialize).
+
+use catalogd::subject_audit::SubjectAuditWriter;
+use shared::types::{AuditAccessor, SubjectAuditRow};
+use std::sync::Arc;
+use validator::{WorkerLookup, WorkerRecord};
+
+pub struct AuditingWorkerLookup {
+    inner: Arc<dyn WorkerLookup>,
+    /// None = audit disabled. find() still runs; just no audit row.
+    audit: Option<Arc<SubjectAuditWriter>>,
+    /// Cached Tokio handle so the sync find() path can spawn audit
+    /// writes onto the right runtime. None at construction = audit
+    /// is no-op (no runtime context to dispatch into).
+    runtime_handle: Option<tokio::runtime::Handle>,
+}
+
+impl AuditingWorkerLookup {
+    /// Wrap an inner lookup with optional audit. If `audit` is None,
+    /// behavior is identical to the inner — no overhead, no audit.
+    /// MUST be constructed inside a Tokio runtime context (uses
+    /// `Handle::try_current` to capture the handle).
+    pub fn new(inner: Arc<dyn WorkerLookup>, audit: Option<Arc<SubjectAuditWriter>>) -> Self {
+        let runtime_handle = if audit.is_some() {
+            tokio::runtime::Handle::try_current().ok()
+        } else {
+            None
+        };
+        if audit.is_some() && runtime_handle.is_none() {
+            tracing::warn!(
+                "AuditingWorkerLookup: audit enabled but no Tokio runtime — audit writes disabled"
+            );
+        }
+        Self { inner, audit, runtime_handle }
+    }
+}
+
+impl WorkerLookup for AuditingWorkerLookup {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
+        let result = self.inner.find(candidate_id);
+        // Fire audit AFTER the find so the result-state field reflects
+        // the actual outcome ("success" if found, "not_found" if not).
+        if let (Some(audit), Some(handle)) = (self.audit.clone(), self.runtime_handle.clone()) {
+            let cid = candidate_id.to_string();
+            let result_state = if result.is_some() { "success" } else { "not_found" };
+            // Single field: "exists" — validator only consumes existence
+            // + role/city/state from the WorkerRecord. We don't enumerate
+            // every WorkerRecord field as fields_accessed because that
+            // would imply they were USED by the caller; the validator
+            // only proves existence + then reads policy fields downstream.
+            let row = SubjectAuditRow {
+                schema: "subject_audit.v1".into(),
+                ts: chrono::Utc::now(),
+                candidate_id: cid.clone(),
+                accessor: AuditAccessor {
+                    kind: "validator_lookup".into(),
+                    daemon: "gateway".into(),
+                    purpose: "validator_worker_lookup".into(),
+                    trace_id: String::new(),
+                },
+                fields_accessed: vec!["exists".into()],
+                result: result_state.into(),
+                prev_chain_hash: String::new(),
+                row_hmac: String::new(),
+            };
+            // Spawn fire-and-forget. Chain integrity is preserved by the
+            // writer's per-subject Mutex (regression-tested), so spawning
+            // is safe. Audit-write errors are logged only.
+            handle.spawn(async move {
+                if let Err(e) = audit.append(row).await {
+                    tracing::warn!("validator audit write failed for {cid}: {e}");
+                }
+            });
+        }
+        result
+    }
+
+    fn len(&self) -> usize {
+        self.inner.len()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use catalogd::subject_audit::SubjectAuditWriter;
+    use object_store::memory::InMemory;
+    use validator::InMemoryWorkerLookup;
+
+    fn fixture_inner_with(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn fixture_record(candidate_id: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: candidate_id.into(),
+            name: "Test".into(),
+            status: "active".into(),
+            city: Some("Chicago".into()),
+            state: Some("IL".into()),
+            role: Some("Welder".into()),
+            blacklisted_clients: vec![],
+        }
+    }
+
+    #[tokio::test]
+    async fn find_existing_subject_writes_success_audit_row() {
+        let inner = fixture_inner_with(vec![fixture_record("CAND-EXIST")]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        let r = lookup.find("CAND-EXIST");
+        assert!(r.is_some());
+        // Audit is spawned — give it a tick to land.
+        tokio::task::yield_now().await;
+        tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+        let count = writer.verify_chain("CAND-EXIST").await.unwrap();
+        assert_eq!(count, 1);
+    }
+
+    #[tokio::test]
+    async fn find_missing_subject_writes_not_found_audit_row() {
+        let inner = fixture_inner_with(vec![]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        let r = lookup.find("CAND-PHANTOM");
+        assert!(r.is_none());
+        tokio::task::yield_now().await;
+        tokio::time::sleep(std::time::Duration::from_millis(50)).await;
+        // verify_chain returns the row count for that subject (1 row,
+        // result="not_found"). We don't assert on result text here —
+        // that's covered by the hand-rolled JSON inspection below.
+        assert_eq!(writer.verify_chain("CAND-PHANTOM").await.unwrap(), 1);
+    }
+
+    #[tokio::test]
+    async fn audit_disabled_means_no_writes_no_overhead() {
+        let inner = fixture_inner_with(vec![fixture_record("CAND-A")]);
+        let lookup = AuditingWorkerLookup::new(inner, None);
+        // Just confirm find() still works with no audit — no panic, no
+        // runtime-handle requirement.
+        assert!(lookup.find("CAND-A").is_some());
+        assert!(lookup.find("CAND-MISSING").is_none());
+        assert_eq!(lookup.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn many_finds_to_same_subject_produce_intact_chain() {
+        // Same regression as catalogd's concurrent test, but exercised
+        // through the validator wrapper path. 30 sequential finds on
+        // the SAME subject should produce an intact 30-row chain.
+        let inner = fixture_inner_with(vec![fixture_record("CAND-CHAIN")]);
+        let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
+        let writer = Arc::new(SubjectAuditWriter::with_inline_key(
+            store.clone(),
+            (0u8..32).collect(),
+        ));
+        let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
+        for _ in 0..30 {
+            let _ = lookup.find("CAND-CHAIN");
+        }
+        // Wait for all spawns to finish — the runtime is currently
+        // single-threaded test runtime, so a few yields drains.
+        for _ in 0..50 {
+            tokio::task::yield_now().await;
+        }
+        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
+        let count = writer.verify_chain("CAND-CHAIN").await.unwrap();
+        assert_eq!(count, 30, "chain corrupted under sequential spawn load");
+    }
+}

crates/gateway/src/v1/iterate.rs

@@ -0,0 +1,543 @@
+//! /v1/iterate — the Phase 43 PRD's "generate → validate → correct → retry" loop.
+//!
+//! Closes the "0→85% with iteration" thesis structurally. A caller
+//! posts a prompt + artifact kind + validation context; the gateway:
+//!   1. Generates a JSON artifact via /v1/chat (any provider/model)
+//!   2. Extracts the JSON object from the model output
+//!   3. Validates via /v1/validate (FillValidator / EmailValidator /
+//!      PlaybookValidator with the shared WorkerLookup)
+//!   4. On ValidationError, appends the error to the prompt and
+//!      retries up to `max_iterations` (default 3)
+//!   5. Returns the accepted artifact + Report on success, OR the
+//!      attempt history + final error on max-iter exhaustion
+//!
+//! Internal calls go via HTTP loopback to localhost:gateway_port so
+//! the same /v1/usage tracking and Langfuse traces apply. A small
+//! latency cost (~1-3ms per loopback hop) for clean separation of
+//! concerns and observability.
+//!
+//! 2026-04-27 Phase 43 v3 part 3: this endpoint makes the iteration
+//! loop a first-class lakehouse capability rather than a per-caller
+//! re-implementation. Staffing executors, agent loops, and future
+//! validators all reach the same code path.
+
+use axum::{extract::State, http::{HeaderMap, StatusCode}, response::IntoResponse, Json};
+use serde::{Deserialize, Serialize};
+
+const DEFAULT_MAX_ITERATIONS: u32 = 3;
+const LOOPBACK_TIMEOUT_SECS: u64 = 240;
+
+/// Header name used to propagate a Langfuse parent trace id across
+/// daemon boundaries. Matches Go's `shared.TraceIDHeader` constant
+/// byte-for-byte (commit d6d2fdf in golangLAKEHOUSE) — same wire
+/// format means a Go caller can hit Rust's /v1/iterate (or vice
+/// versa) and the resulting Langfuse trees nest correctly.
+pub const TRACE_ID_HEADER: &str = "x-lakehouse-trace-id";
+
+#[derive(Deserialize)]
+pub struct IterateRequest {
+    /// "fill" | "email" | "playbook" — picks which validator runs.
+    pub kind: String,
+    /// The prompt to seed generation. Validation errors from prior
+    /// attempts are appended on retry.
+    pub prompt: String,
+    /// Provider/model passed through to /v1/chat. e.g. "ollama_cloud"
+    /// + "kimi-k2.6", or "opencode" + "claude-haiku-4-5".
+    pub provider: String,
+    pub model: String,
+    /// Optional system prompt — sent to /v1/chat as the system message.
+    #[serde(default)]
+    pub system: Option<String>,
+    /// Validation context (target_count, city, state, role, client_id
+    /// for fills; candidate_id for emails). Forwarded to /v1/validate.
+    #[serde(default)]
+    pub context: Option<serde_json::Value>,
+    /// Cap on iteration count. Defaults to 3 per the Phase 43 PRD.
+    #[serde(default)]
+    pub max_iterations: Option<u32>,
+    /// Forwarded to /v1/chat. Defaults to 0.2 if unset.
+    #[serde(default)]
+    pub temperature: Option<f64>,
+    /// Forwarded to /v1/chat. Defaults to 4096 if unset.
+    #[serde(default)]
+    pub max_tokens: Option<u32>,
+}
+
+#[derive(Serialize)]
+pub struct IterateAttempt {
+    pub iteration: u32,
+    pub raw: String,
+    pub status: AttemptStatus,
+}
+
+#[derive(Serialize)]
+#[serde(tag = "kind", rename_all = "snake_case")]
+pub enum AttemptStatus {
+    /// Model output didn't contain extractable JSON.
+    NoJson,
+    /// JSON extracted but failed validation; carries the error.
+    ValidationFailed { error: serde_json::Value },
+    /// Validation passed (last attempt's terminal status).
+    Accepted,
+}
+
+#[derive(Serialize)]
+pub struct IterateResponse {
+    pub artifact: serde_json::Value,
+    pub validation: serde_json::Value,
+    pub iterations: u32,
+    pub history: Vec<IterateAttempt>,
+    /// Echoes the resolved trace id (caller-forwarded header, body
+    /// field, langfuse-middleware mint, or local fallback). Operators
+    /// pivot from this id straight into Langfuse + the
+    /// coordinator_sessions.jsonl join key. Cross-runtime parity with
+    /// Go's `validator.IterateResponse` (commit 6847bbc in
+    /// golangLAKEHOUSE).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub trace_id: Option<String>,
+}
+
+#[derive(Serialize)]
+pub struct IterateFailure {
+    pub error: String,
+    pub iterations: u32,
+    pub history: Vec<IterateAttempt>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub trace_id: Option<String>,
+}
+
+pub async fn iterate(
+    State(state): State<super::V1State>,
+    headers: HeaderMap,
+    Json(req): Json<IterateRequest>,
+) -> impl IntoResponse {
+    let max_iter = req.max_iterations.unwrap_or(DEFAULT_MAX_ITERATIONS).max(1);
+    let temperature = req.temperature.unwrap_or(0.2);
+    let max_tokens = req.max_tokens.unwrap_or(4096);
+    let mut history: Vec<IterateAttempt> = Vec::with_capacity(max_iter as usize);
+    let mut attempt_records: Vec<super::session_log::SessionAttemptRecord> = Vec::with_capacity(max_iter as usize);
+    let mut current_prompt = req.prompt.clone();
+
+    // Resolve the parent Langfuse trace id. Caller-forwarded header
+    // wins (cross-daemon tree linkage); otherwise mint a fresh id so
+    // the iterate session is its own tree. Same shape as the Go-side
+    // validatord trace propagation.
+    let trace_id: String = headers
+        .get(TRACE_ID_HEADER)
+        .and_then(|v| v.to_str().ok())
+        .filter(|s| !s.is_empty())
+        .map(|s| s.to_string())
+        .unwrap_or_else(new_trace_id);
+
+    let client = match reqwest::Client::builder()
+        .timeout(std::time::Duration::from_secs(LOOPBACK_TIMEOUT_SECS))
+        .build() {
+        Ok(c) => c,
+        Err(e) => {
+            // Even infrastructure failures get a session row so a
+            // missing /v1/iterate event never silently disappears
+            // from the longitudinal log.
+            write_infra_error(&state, &req, &trace_id, max_iter, 0, format!("client build: {e}")).await;
+            return (StatusCode::INTERNAL_SERVER_ERROR, format!("client build: {e}")).into_response();
+        }
+    };
+    // Self-loopback to the gateway port. Carries gateway internal
+    // calls through /v1/chat + /v1/validate so /v1/usage tracks them.
+    let gateway = "http://127.0.0.1:3100";
+    let t0 = std::time::Instant::now();
+
+    for iteration in 0..max_iter {
+        let attempt_started = chrono::Utc::now();
+        // ── Generate ──
+        let mut messages = Vec::with_capacity(2);
+        if let Some(sys) = &req.system {
+            messages.push(serde_json::json!({"role": "system", "content": sys}));
+        }
+        messages.push(serde_json::json!({"role": "user", "content": current_prompt}));
+        let chat_body = serde_json::json!({
+            "messages": messages,
+            "provider": req.provider,
+            "model": req.model,
+            "temperature": temperature,
+            "max_tokens": max_tokens,
+        });
+        let raw = match call_chat(&client, gateway, &chat_body, &trace_id).await {
+            Ok(r) => r,
+            Err(e) => {
+                write_infra_error(&state, &req, &trace_id, max_iter, t0.elapsed().as_millis() as u64, format!("/v1/chat hop failed at iter {iteration}: {e}")).await;
+                return (StatusCode::BAD_GATEWAY, format!("/v1/chat hop failed at iter {iteration}: {e}")).into_response();
+            }
+        };
+
+        // ── Extract JSON ──
+        let artifact = match extract_json(&raw) {
+            Some(a) => a,
+            None => {
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "no_json", None,
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::NoJson,
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "no_json".to_string(),
+                    error: None,
+                    span_id,
+                });
+                current_prompt = format!(
+                    "{}\n\nYour previous attempt did not contain a JSON object. Reply with ONLY a valid JSON object matching the requested artifact shape.",
+                    req.prompt,
+                );
+                continue;
+            }
+        };
+
+        // ── Validate ──
+        let validate_body = serde_json::json!({
+            "kind": req.kind,
+            "artifact": artifact,
+            "context": req.context.clone().unwrap_or(serde_json::Value::Null),
+        });
+        match call_validate(&client, gateway, &validate_body, &trace_id).await {
+            Ok(report) => {
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "accepted", None,
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::Accepted,
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "accepted".to_string(),
+                    error: None,
+                    span_id,
+                });
+                let duration_ms = t0.elapsed().as_millis() as u64;
+                let grounded = grounded_in_roster(&state, &req.kind, &artifact);
+                write_session_accepted(&state, &req, &trace_id, iteration + 1, max_iter, attempt_records, &artifact, grounded, duration_ms).await;
+                return (StatusCode::OK, Json(IterateResponse {
+                    artifact,
+                    validation: report,
+                    iterations: iteration + 1,
+                    history,
+                    trace_id: Some(trace_id.clone()),
+                })).into_response();
+            }
+            Err(err) => {
+                let err_summary = err.to_string();
+                let span_id = emit_attempt_span(
+                    &state, &trace_id, iteration, &req, &current_prompt, &raw, "validation_failed",
+                    Some(err_summary.clone()),
+                    attempt_started, chrono::Utc::now(),
+                );
+                history.push(IterateAttempt {
+                    iteration,
+                    raw: raw.chars().take(2000).collect(),
+                    status: AttemptStatus::ValidationFailed {
+                        error: serde_json::to_value(&err_summary).unwrap_or(serde_json::Value::Null),
+                    },
+                });
+                attempt_records.push(super::session_log::SessionAttemptRecord {
+                    iteration,
+                    verdict_kind: "validation_failed".to_string(),
+                    error: Some(err_summary.clone()),
+                    span_id,
+                });
+                // Append validation feedback to prompt for next iter.
+                // The model sees concrete failure mode + retries with
+                // corrective context. This is the "observer correction"
+                // in Phase 43 PRD shape, simplified — the validator
+                // itself IS the observer for now.
+                current_prompt = format!(
+                    "{}\n\nPrior attempt failed validation:\n{}\n\nFix the specific issue above and respond with a corrected JSON object.",
+                    req.prompt, err_summary,
+                );
+                continue;
+            }
+        }
+    }
+
+    let duration_ms = t0.elapsed().as_millis() as u64;
+    write_session_failure(&state, &req, &trace_id, max_iter, max_iter, attempt_records, duration_ms).await;
+    (StatusCode::UNPROCESSABLE_ENTITY, Json(IterateFailure {
+        error: format!("max iterations reached ({max_iter}) without passing validation"),
+        iterations: max_iter,
+        history,
+        trace_id: Some(trace_id.clone()),
+    })).into_response()
+}
+
+// ─── Helpers — Langfuse spans + session log + roster check ─────────
+
+fn emit_attempt_span(
+    state: &super::V1State,
+    trace_id: &str,
+    iteration: u32,
+    req: &IterateRequest,
+    prompt: &str,
+    raw: &str,
+    verdict: &str,
+    error: Option<String>,
+    started: chrono::DateTime<chrono::Utc>,
+    ended: chrono::DateTime<chrono::Utc>,
+) -> Option<String> {
+    let lf = state.langfuse.as_ref()?;
+    Some(lf.emit_attempt_span(super::langfuse_trace::AttemptSpan {
+        trace_id: trace_id.to_string(),
+        iteration,
+        model: req.model.clone(),
+        provider: req.provider.clone(),
+        prompt: prompt.to_string(),
+        raw: raw.to_string(),
+        verdict: verdict.to_string(),
+        error,
+        start_time: started.to_rfc3339(),
+        end_time: ended.to_rfc3339(),
+    }))
+}
+
+/// Verify every fill artifact's candidate IDs exist in the roster.
+/// Returns Some(true)/Some(false) on the fill kind, None otherwise
+/// (other kinds don't have worker IDs to ground). Same semantics as
+/// Go's `handlers.rosterCheckFor("fill")`.
+fn grounded_in_roster(
+    state: &super::V1State,
+    kind: &str,
+    artifact: &serde_json::Value,
+) -> Option<bool> {
+    if kind != "fill" {
+        return None;
+    }
+    let fills = artifact.get("fills").and_then(|v| v.as_array())?;
+    for f in fills {
+        let id = match f.get("candidate_id").and_then(|v| v.as_str()) {
+            Some(s) if !s.is_empty() => s,
+            _ => return Some(false),
+        };
+        if state.validate_workers.find(id).is_none() {
+            return Some(false);
+        }
+    }
+    Some(true)
+}
+
+async fn write_session_accepted(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    artifact: &serde_json::Value,
+    grounded: Option<bool>,
+    duration_ms: u64,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let rec = build_session_record(req, trace_id, "accepted", iterations, max_iter, attempts, Some(artifact.clone()), grounded, duration_ms);
+    logger.append(rec).await;
+}
+
+async fn write_session_failure(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    duration_ms: u64,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let rec = build_session_record(req, trace_id, "max_iter_exhausted", iterations, max_iter, attempts, None, None, duration_ms);
+    logger.append(rec).await;
+}
+
+async fn write_infra_error(
+    state: &super::V1State,
+    req: &IterateRequest,
+    trace_id: &str,
+    max_iter: u32,
+    duration_ms: u64,
+    error: String,
+) {
+    let Some(logger) = state.session_log.as_ref() else { return };
+    let attempts = vec![super::session_log::SessionAttemptRecord {
+        iteration: 0,
+        verdict_kind: "infra_error".to_string(),
+        error: Some(error),
+        span_id: None,
+    }];
+    let rec = build_session_record(req, trace_id, "infra_error", 0, max_iter, attempts, None, None, duration_ms);
+    logger.append(rec).await;
+}
+
+fn build_session_record(
+    req: &IterateRequest,
+    trace_id: &str,
+    final_verdict: &str,
+    iterations: u32,
+    max_iter: u32,
+    attempts: Vec<super::session_log::SessionAttemptRecord>,
+    artifact: Option<serde_json::Value>,
+    grounded: Option<bool>,
+    duration_ms: u64,
+) -> super::session_log::SessionRecord {
+    super::session_log::SessionRecord {
+        schema: super::session_log::SESSION_RECORD_SCHEMA.to_string(),
+        session_id: trace_id.to_string(),
+        timestamp: chrono::Utc::now().to_rfc3339(),
+        daemon: "gateway".to_string(),
+        kind: req.kind.clone(),
+        model: req.model.clone(),
+        provider: req.provider.clone(),
+        prompt: super::session_log::truncate(&req.prompt, 4000),
+        iterations,
+        max_iterations: max_iter,
+        final_verdict: final_verdict.to_string(),
+        attempts,
+        artifact,
+        grounded_in_roster: grounded,
+        duration_ms,
+    }
+}
+
+/// Generate a fresh trace id when no parent was forwarded. Same
+/// time-ordered hex shape Langfuse already accepts elsewhere in this
+/// crate (see `langfuse_trace::uuid_v7_like`).
+fn new_trace_id() -> String {
+    let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
+    let rand = std::time::SystemTime::now()
+        .duration_since(std::time::UNIX_EPOCH)
+        .map(|d| d.subsec_nanos())
+        .unwrap_or(0);
+    format!("{:016x}-{:08x}", ts, rand)
+}
+
+async fn call_chat(client: &reqwest::Client, gateway: &str, body: &serde_json::Value, trace_id: &str) -> Result<String, String> {
+    let mut req = client.post(format!("{gateway}/v1/chat")).json(body);
+    if !trace_id.is_empty() {
+        req = req.header(TRACE_ID_HEADER, trace_id);
+    }
+    let resp = req.send().await.map_err(|e| format!("chat hop: {e}"))?;
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_default();
+        return Err(format!("chat {}: {}", status, body.chars().take(300).collect::<String>()));
+    }
+    let parsed: serde_json::Value = resp.json().await.map_err(|e| format!("chat parse: {e}"))?;
+    Ok(parsed.pointer("/choices/0/message/content")
+        .and_then(|v| v.as_str())
+        .unwrap_or("")
+        .to_string())
+}
+
+async fn call_validate(client: &reqwest::Client, gateway: &str, body: &serde_json::Value, trace_id: &str) -> Result<serde_json::Value, String> {
+    let mut req = client.post(format!("{gateway}/v1/validate")).json(body);
+    if !trace_id.is_empty() {
+        req = req.header(TRACE_ID_HEADER, trace_id);
+    }
+    let resp = req.send().await.map_err(|e| format!("validate hop: {e}"))?;
+    let status = resp.status();
+    let parsed: serde_json::Value = resp.json().await.map_err(|e| format!("validate parse: {e}"))?;
+    if status.is_success() {
+        Ok(parsed)
+    } else {
+        // The /v1/validate endpoint returns a ValidationError JSON
+        // on 422; surface its structure verbatim so the prompt-
+        // appending step gets specific failure detail.
+        Err(serde_json::to_string(&parsed).unwrap_or_else(|_| format!("validation {} (unparseable body)", status)))
+    }
+}
+
+/// Extract the first JSON object from a model's output. Handles
+/// fenced code blocks (```json ... ```), bare braces, and stray
+/// prose around the JSON. Returns None on no extractable object.
+///
+/// Made `pub` 2026-05-02 to support the cross-runtime parity probe
+/// at `golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh`.
+/// The Go counterpart lives at `internal/validator/iterate.go::ExtractJSON`;
+/// when either runtime's algorithm changes the parity probe surfaces
+/// the divergence.
+pub fn extract_json(raw: &str) -> Option<serde_json::Value> {
+    // Try fenced first.
+    let candidates: Vec<String> = {
+        let mut out = vec![];
+        let mut s = raw;
+        while let Some(start) = s.find("```") {
+            let after = &s[start + 3..];
+            // Skip optional language tag (json, etc.)
+            let body_start = after.find('\n').map(|n| n + 1).unwrap_or(0);
+            let body = &after[body_start..];
+            if let Some(end) = body.find("```") {
+                out.push(body[..end].trim().to_string());
+                s = &body[end + 3..];
+            } else { break; }
+        }
+        out
+    };
+    for c in &candidates {
+        if let Ok(v) = serde_json::from_str::<serde_json::Value>(c) {
+            if v.is_object() { return Some(v); }
+        }
+    }
+    // Fall back to outermost {...} balance.
+    let bytes = raw.as_bytes();
+    let mut depth = 0i32;
+    let mut start: Option<usize> = None;
+    for (i, &b) in bytes.iter().enumerate() {
+        match b {
+            b'{' => { if start.is_none() { start = Some(i); } depth += 1; }
+            b'}' => {
+                depth -= 1;
+                if depth == 0 {
+                    if let Some(s) = start {
+                        let slice = &raw[s..=i];
+                        if let Ok(v) = serde_json::from_str::<serde_json::Value>(slice) {
+                            if v.is_object() { return Some(v); }
+                        }
+                        start = None;
+                    }
+                }
+            }
+            _ => {}
+        }
+    }
+    None
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn extract_json_from_fenced_block() {
+        let raw = "Here's my answer:\n```json\n{\"fills\": [{\"candidate_id\": \"W-1\"}]}\n```\nDone.";
+        let v = extract_json(raw).unwrap();
+        assert!(v.get("fills").is_some());
+    }
+
+    #[test]
+    fn extract_json_from_bare_braces() {
+        let raw = "Here you go: {\"fills\": [{\"candidate_id\": \"W-2\"}]}";
+        let v = extract_json(raw).unwrap();
+        assert!(v.get("fills").is_some());
+    }
+
+    #[test]
+    fn extract_json_returns_none_on_no_object() {
+        assert!(extract_json("just prose, no json").is_none());
+    }
+
+    #[test]
+    fn extract_json_picks_first_balanced() {
+        let raw = "{\"a\":1} then {\"b\":2}";
+        let v = extract_json(raw).unwrap();
+        assert_eq!(v.get("a").and_then(|v| v.as_i64()), Some(1));
+    }
+}

crates/gateway/src/v1/kimi.rs

@@ -0,0 +1,227 @@
+//! Kimi For Coding adapter — direct provider for `kimi-for-coding`
+//! (kimi-k2.6 underneath). Used when Ollama Cloud's `kimi-k2:1t` is
+//! returning sustained 5xx (broken upstream) and OpenRouter's
+//! `moonshotai/kimi-k2.6` is rate-limited.
+//!
+//! Endpoint per `kimi.com/code/docs` and `moonshotai.github.io/kimi-cli`:
+//!   base_url:  https://api.kimi.com/coding/v1
+//!   model id:  kimi-for-coding
+//!   auth:      Bearer sk-kimi-…
+//!   protocol:  OpenAI Chat Completions compatible
+//!
+//! IMPORTANT: `api.kimi.com` is a separate account system from
+//! `api.moonshot.ai` and `api.moonshot.cn`. Keys are NOT interchangeable.
+//! This adapter is for `sk-kimi-*` keys provisioned via the Kimi
+//! membership console only.
+//!
+//! Key sourcing priority:
+//!   1. Env var `KIMI_API_KEY` (loaded from /etc/lakehouse/kimi.env via
+//!      systemd EnvironmentFile=)
+//!   2. /etc/lakehouse/kimi.env directly (rescue path if env not loaded)
+//!
+//! First hit wins. Resolved once at gateway startup, stored on
+//! `V1State.kimi_key`.
+
+use std::time::Duration;
+use serde::{Deserialize, Serialize};
+
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+const KIMI_BASE_URL: &str = "https://api.kimi.com/coding/v1";
+// Default 600s — kimi-for-coding is a reasoning model; on large
+// code-audit prompts (~50KB+ input + 8K output) it routinely needs
+// 3-8 min to think + emit. Override with KIMI_TIMEOUT_SECS env var.
+const KIMI_TIMEOUT_SECS_DEFAULT: u64 = 600;
+
+fn kimi_timeout_secs() -> u64 {
+    std::env::var("KIMI_TIMEOUT_SECS")
+        .ok()
+        .and_then(|s| s.trim().parse::<u64>().ok())
+        .filter(|&n| n > 0)
+        .unwrap_or(KIMI_TIMEOUT_SECS_DEFAULT)
+}
+
+pub fn resolve_kimi_key() -> Option<String> {
+    if let Ok(k) = std::env::var("KIMI_API_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    if let Ok(raw) = std::fs::read_to_string("/etc/lakehouse/kimi.env") {
+        for line in raw.lines() {
+            if let Some(rest) = line.strip_prefix("KIMI_API_KEY=") {
+                let k = rest.trim().trim_matches('"').trim_matches('\'');
+                if !k.is_empty() { return Some(k.to_string()); }
+            }
+        }
+    }
+    None
+}
+
+pub async fn chat(
+    key: &str,
+    req: &ChatRequest,
+) -> Result<ChatResponse, String> {
+    // Strip the "kimi/" namespace prefix if the caller used it so the
+    // upstream API sees the bare model id (e.g. "kimi-for-coding").
+    let model = req.model.strip_prefix("kimi/").unwrap_or(&req.model).to_string();
+
+    // Flatten content to a plain String. api.kimi.com is text-only on
+    // the coding endpoint; the OpenAI multimodal array shape
+    // ([{type:"text",text:"..."},{type:"image_url",...}]) returns 400.
+    // Message::text() concats text-parts and drops non-text. Caught
+    // 2026-04-27 by Kimi's self-audit (kimi.rs:137 — content as raw
+    // serde_json::Value risked upstream rejection).
+    let body = KimiChatBody {
+        model: model.clone(),
+        messages: req.messages.iter().map(|m| KimiMessage {
+            role: m.role.clone(),
+            content: serde_json::Value::String(m.text()),
+        }).collect(),
+        max_tokens: req.max_tokens.unwrap_or(800),
+        temperature: req.temperature.unwrap_or(0.3),
+        stream: false,
+    };
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(kimi_timeout_secs()))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+
+    let t0 = std::time::Instant::now();
+    let resp = client
+        .post(format!("{}/chat/completions", KIMI_BASE_URL))
+        .bearer_auth(key)
+        // api.kimi.com gates this endpoint by User-Agent — only sanctioned
+        // coding agents (Claude Code, Kimi CLI, Roo Code, Kilo Code) get
+        // through. Generic clients receive 403 access_terminated_error.
+        // J accepted the TOS risk on 2026-04-27; revisit if Moonshot
+        // tightens enforcement.
+        .header("User-Agent", "claude-code/1.0.0")
+        .json(&body)
+        .send()
+        .await
+        .map_err(|e| format!("api.kimi.com unreachable: {e}"))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("api.kimi.com {}: {}", status, body));
+    }
+
+    let parsed: KimiChatResponse = resp.json().await
+        .map_err(|e| format!("invalid kimi response: {e}"))?;
+
+    let latency_ms = t0.elapsed().as_millis();
+    let choice = parsed.choices.into_iter().next()
+        .ok_or_else(|| "kimi returned no choices".to_string())?;
+    let text = choice.message.content;
+
+    let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
+        ((chars + 3) / 4) as u32
+    });
+    let completion_tokens = parsed.usage.as_ref().map(|u| u.completion_tokens).unwrap_or_else(|| {
+        ((text.chars().count() + 3) / 4) as u32
+    });
+
+    tracing::info!(
+        target: "v1.chat",
+        provider = "kimi",
+        model = %model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "kimi chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model,
+        choices: vec![Choice {
+            index: 0,
+            message: Message { role: "assistant".into(), content: serde_json::Value::String(text) },
+            finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+// -- Kimi wire shapes (OpenAI-compatible) --
+
+#[derive(Serialize)]
+struct KimiChatBody {
+    model: String,
+    messages: Vec<KimiMessage>,
+    max_tokens: u32,
+    temperature: f64,
+    stream: bool,
+}
+
+#[derive(Serialize)]
+struct KimiMessage { role: String, content: serde_json::Value }
+
+#[derive(Deserialize)]
+struct KimiChatResponse {
+    choices: Vec<KimiChoice>,
+    #[serde(default)]
+    usage: Option<KimiUsage>,
+}
+
+#[derive(Deserialize)]
+struct KimiChoice {
+    message: KimiMessageResp,
+    #[serde(default)]
+    finish_reason: Option<String>,
+}
+
+#[derive(Deserialize)]
+struct KimiMessageResp { content: String }
+
+#[derive(Deserialize)]
+struct KimiUsage { prompt_tokens: u32, completion_tokens: u32 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_kimi_key_does_not_panic() {
+        let _ = resolve_kimi_key();
+    }
+
+    #[test]
+    fn chat_body_serializes_to_openai_shape() {
+        let body = KimiChatBody {
+            model: "kimi-for-coding".into(),
+            messages: vec![
+                KimiMessage { role: "user".into(), content: "review this".into() },
+            ],
+            max_tokens: 800,
+            temperature: 0.3,
+            stream: false,
+        };
+        let json = serde_json::to_string(&body).unwrap();
+        assert!(json.contains("\"model\":\"kimi-for-coding\""));
+        assert!(json.contains("\"messages\""));
+        assert!(json.contains("\"max_tokens\":800"));
+        assert!(json.contains("\"stream\":false"));
+    }
+
+    #[test]
+    fn model_prefix_strip() {
+        let cases = [
+            ("kimi/kimi-for-coding", "kimi-for-coding"),
+            ("kimi-for-coding", "kimi-for-coding"),
+            ("kimi/kimi-k2.6", "kimi-k2.6"),
+        ];
+        for (input, expected) in cases {
+            let out = input.strip_prefix("kimi/").unwrap_or(input);
+            assert_eq!(out, expected, "{input} should become {expected}");
+        }
+    }
+}

crates/gateway/src/v1/langfuse_trace.rs

@@ -76,14 +76,85 @@ impl LangfuseClient {
         });
     }
 
+    /// Fire-and-forget per-iteration span emit. Returns the generated
+    /// span id synchronously so the caller can stamp it on
+    /// `IterateAttempt.span_id` before the network round-trip resolves.
+    /// Mirrors Go's `validator.Tracer` callback shape.
+    pub fn emit_attempt_span(&self, sp: AttemptSpan) -> String {
+        let span_id = uuid_v7_like();
+        let span_id_for_caller = span_id.clone();
+        let this = self.clone();
+        tokio::spawn(async move {
+            if let Err(e) = this.emit_attempt_span_inner(span_id, sp).await {
+                tracing::warn!(target: "v1.langfuse", "iterate span drop: {e}");
+            }
+        });
+        span_id_for_caller
+    }
+
+    async fn emit_attempt_span_inner(&self, span_id: String, sp: AttemptSpan) -> Result<(), String> {
+        let level = if sp.verdict == "accepted" { "DEFAULT" } else { "WARNING" };
+        let batch = IngestionBatch {
+            batch: vec![IngestionEvent {
+                id: uuid_v7_like(),
+                timestamp: sp.end_time.clone(),
+                kind: "span-create",
+                body: serde_json::json!({
+                    "id": span_id,
+                    "traceId": sp.trace_id,
+                    "name": format!("iterate.attempt[{}]", sp.iteration),
+                    "input": serde_json::json!({
+                        "iteration": sp.iteration,
+                        "model": sp.model,
+                        "provider": sp.provider,
+                        "prompt": truncate(&sp.prompt, 4000),
+                    }),
+                    "output": serde_json::json!({
+                        "verdict": sp.verdict,
+                        "error": sp.error,
+                        "raw": truncate(&sp.raw, 4000),
+                    }),
+                    "level": level,
+                    "startTime": sp.start_time,
+                    "endTime": sp.end_time,
+                }),
+            }],
+        };
+        self.post_batch(batch).await
+    }
+
+    async fn post_batch(&self, batch: IngestionBatch) -> Result<(), String> {
+        let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
+        let resp = self.inner.http
+            .post(url)
+            .basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
+            .json(&batch)
+            .send()
+            .await
+            .map_err(|e| format!("POST failed: {e}"))?;
+        if !resp.status().is_success() {
+            return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+        }
+        Ok(())
+    }
+
     async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
-        let trace_id = uuid_v7_like();
+        // When the caller forwarded a parent trace id (via the
+        // X-Lakehouse-Trace-Id header → V1State plumbing), attach the
+        // generation as a child of that trace. Without a parent we
+        // mint a new top-level trace per call (Phase 40 default).
+        let trace_id = ev.parent_trace_id.clone().unwrap_or_else(uuid_v7_like);
+        let nested = ev.parent_trace_id.is_some();
         let gen_id = uuid_v7_like();
         let trace_ts = ev.start_time.clone();
 
-        let batch = IngestionBatch {
-            batch: vec![
-                IngestionEvent {
+        let mut events = Vec::with_capacity(2);
+        if !nested {
+            // Only mint a fresh trace-create when we don't have a parent.
+            // Reusing a parent trace id without re-creating it is the
+            // contract that lets validatord's iterate-session show up
+            // as one tree in Langfuse.
+            events.push(IngestionEvent {
                 id: uuid_v7_like(),
                 timestamp: trace_ts.clone(),
                 kind: "trace-create",
@@ -99,8 +170,9 @@ impl LangfuseClient {
                         "think": ev.think,
                     }),
                 }),
-                },
-                IngestionEvent {
+            });
+        }
+        events.push(IngestionEvent {
             id: uuid_v7_like(),
             timestamp: ev.end_time.clone(),
             kind: "generation-create",
@@ -129,23 +201,17 @@ impl LangfuseClient {
                     "latency_ms": ev.latency_ms,
                 }),
             }),
-                },
-            ],
-        };
+        });
 
-        let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
-        let resp = self.inner.http
-            .post(url)
-            .basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
-            .json(&batch)
-            .send()
-            .await
-            .map_err(|e| format!("POST failed: {e}"))?;
-        if !resp.status().is_success() {
-            return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+        self.post_batch(IngestionBatch { batch: events }).await
     }
-        Ok(())
 }
+
+/// Truncate a string to at most `n` chars (NOT bytes). Matches the Go
+/// `trim` helper used in session log + attempt-span emission so an
+/// operator reading two cross-runtime traces sees the same boundary.
+fn truncate(s: &str, n: usize) -> String {
+    s.chars().take(n).collect()
 }
 
 /// Everything the v1.chat handler collects for one completed call.
@@ -162,6 +228,32 @@ pub struct ChatTrace {
     pub start_time: String,
     pub end_time: String,
     pub latency_ms: u64,
+    /// When set, attach this chat trace as a child of the named
+    /// Langfuse trace instead of starting a new top-level trace. Used
+    /// by `/v1/iterate` to nest its inner /v1/chat hops under the
+    /// iterate-session trace so a multi-call session shows in
+    /// Langfuse as ONE trace tree, not N+1 disconnected traces.
+    /// Matches the Go-side `X-Lakehouse-Trace-Id` propagation
+    /// (commit d6d2fdf in golangLAKEHOUSE).
+    pub parent_trace_id: Option<String>,
+}
+
+/// One iteration attempt inside `/v1/iterate`'s loop. Becomes one
+/// span on the parent trace when emitted via `emit_attempt_span`.
+/// Matches Go's `validator.AttemptSpan` shape so the cross-runtime
+/// observability surface is consistent.
+pub struct AttemptSpan {
+    pub trace_id: String,
+    pub iteration: u32,
+    pub model: String,
+    pub provider: String,
+    pub prompt: String,
+    pub raw: String,
+    /// Verdict kind: "no_json" | "validation_failed" | "accepted"
+    pub verdict: String,
+    pub error: Option<String>,
+    pub start_time: String,
+    pub end_time: String,
 }
 
 #[derive(Serialize)]

crates/gateway/src/v1/mod.rs

@@ -16,7 +16,13 @@ pub mod ollama_cloud;
 pub mod openrouter;
 pub mod gemini;
 pub mod claude;
+pub mod kimi;
+pub mod opencode;
+pub mod validate;
+pub mod iterate;
+pub mod auditing_worker_lookup;
 pub mod langfuse_trace;
+pub mod session_log;
 pub mod mode;
 pub mod respond;
 pub mod truth;
@@ -53,10 +59,45 @@ pub struct V1State {
     /// `claude::resolve_claude_key()`. None = provider="claude" calls
     /// 503. Phase 40 deliverable.
     pub claude_key: Option<String>,
+    /// Kimi For Coding (api.kimi.com) bearer token — direct provider
+    /// for `kimi-for-coding`. Used when Ollama Cloud's `kimi-k2:1t` is
+    /// upstream-broken. Loaded at startup via `kimi::resolve_kimi_key()`
+    /// from `KIMI_API_KEY` env or `/etc/lakehouse/kimi.env`. None =
+    /// provider="kimi" calls 503.
+    pub kimi_key: Option<String>,
+    /// OpenCode GO (opencode.ai) bearer token — multi-vendor curated
+    /// gateway. One sk-* key reaches Claude Opus 4.7, GPT-5.5-pro,
+    /// Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM, Qwen + free-tier.
+    /// Loaded at startup via `opencode::resolve_opencode_key()` from
+    /// `OPENCODE_API_KEY` env or `/etc/lakehouse/opencode.env`. None =
+    /// provider="opencode" calls 503.
+    pub opencode_key: Option<String>,
+    /// Shared WorkerLookup loaded once at startup from
+    /// workers_500k.parquet (path: LH_WORKERS_PARQUET env, default
+    /// data/datasets/workers_500k.parquet). Used by /v1/validate to
+    /// run FillValidator/EmailValidator with worker-existence checks.
+    /// Falls back to an empty InMemoryWorkerLookup if the file is
+    /// missing — validators still run schema/PII checks but every
+    /// worker-existence check fails (Consistency error), which is
+    /// the correct behavior when the roster isn't configured.
+    pub validate_workers: std::sync::Arc<dyn validator::WorkerLookup>,
     /// Phase 40 early deliverable — Langfuse client. None = tracing
     /// disabled (keys missing or container unreachable). Traces are
     /// fire-and-forget: never block the response path.
     pub langfuse: Option<langfuse_trace::LangfuseClient>,
+    /// Coordinator session JSONL writer (path from
+    /// `[gateway].session_log_path`). One row per `/v1/iterate`
+    /// session for offline DuckDB analysis. None = disabled.
+    /// Cross-runtime parity with the Go-side `validatord`
+    /// `[validatord].session_log_path` (commit 1a3a82a in
+    /// golangLAKEHOUSE).
+    pub session_log: Option<session_log::SessionLogger>,
+    /// Per-subject audit log writer (HMAC-chained JSONL). Loaded at
+    /// startup from /etc/lakehouse/subject_audit.key if the file is
+    /// present and ≥32 bytes. None = audit disabled with a warning
+    /// at startup; PII access is still served but produces no audit
+    /// trail. See docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
+    pub subject_audit: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>>,
 }
 
 #[derive(Default, Clone, Serialize)]
@@ -92,6 +133,9 @@ pub fn router(state: V1State) -> Router {
         .route("/mode", post(mode::route))
         .route("/mode/list", get(mode::list))
         .route("/mode/execute", post(mode::execute))
+        .route("/validate", post(validate::validate))
+        .route("/iterate", post(iterate::iterate))
+        .route("/health", get(health))
         .with_state(state)
 }
 
@@ -224,6 +268,12 @@ fn resolve_provider(req: &ChatRequest) -> (String, String) {
     if let Some(rest) = req.model.strip_prefix("claude/") {
         return ("claude".to_string(), rest.to_string());
     }
+    if let Some(rest) = req.model.strip_prefix("kimi/") {
+        return ("kimi".to_string(), rest.to_string());
+    }
+    if let Some(rest) = req.model.strip_prefix("opencode/") {
+        return ("opencode".to_string(), rest.to_string());
+    }
     // Bare `vendor/model` shape (e.g. `x-ai/grok-4.1-fast`,
     // `moonshotai/kimi-k2`, `openai/gpt-oss-120b:free`) → OpenRouter.
     // This makes the gateway a drop-in OpenAI-compatible middleware:
@@ -316,10 +366,17 @@ mod resolve_provider_tests {
         let r = mk_req(None, "claude/claude-3-5-sonnet-latest");
         assert_eq!(resolve_provider(&r), ("claude".into(), "claude-3-5-sonnet-latest".into()));
     }
+
+    #[test]
+    fn kimi_prefix_infers_and_strips() {
+        let r = mk_req(None, "kimi/kimi-for-coding");
+        assert_eq!(resolve_provider(&r), ("kimi".into(), "kimi-for-coding".into()));
+    }
 }
 
 async fn chat(
     State(state): State<V1State>,
+    headers: axum::http::HeaderMap,
     Json(req): Json<ChatRequest>,
 ) -> Result<Json<ChatResponse>, (StatusCode, String)> {
     if req.messages.is_empty() {
@@ -403,10 +460,37 @@ async fn chat(
                 .map_err(|e| (StatusCode::BAD_GATEWAY, format!("claude: {e}")))?;
             (r, "claude".to_string())
         }
+        "kimi" => {
+            // Direct Kimi For Coding provider — bypasses Ollama Cloud's
+            // upstream-broken kimi-k2:1t and OpenRouter's rate-limited
+            // moonshotai/kimi-k2.6. Uses sk-kimi-* keys from the Kimi
+            // membership console.
+            let key = state.kimi_key.as_deref().ok_or((
+                StatusCode::SERVICE_UNAVAILABLE,
+                "KIMI_API_KEY not configured".to_string(),
+            ))?;
+            let r = kimi::chat(key, &*req_for_adapter)
+                .await
+                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("kimi: {e}")))?;
+            (r, "kimi".to_string())
+        }
+        "opencode" => {
+            // OpenCode GO multi-vendor gateway — Claude Opus 4.7,
+            // GPT-5.5-pro, Gemini 3.1-pro, Kimi K2.6, DeepSeek, GLM,
+            // Qwen, free-tier. OpenAI-compat at opencode.ai/zen/go/v1.
+            let key = state.opencode_key.as_deref().ok_or((
+                StatusCode::SERVICE_UNAVAILABLE,
+                "OPENCODE_API_KEY not configured".to_string(),
+            ))?;
+            let r = opencode::chat(key, &*req_for_adapter)
+                .await
+                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("opencode: {e}")))?;
+            (r, "opencode".to_string())
+        }
         other => {
             return Err((
                 StatusCode::BAD_REQUEST,
-                format!("unknown provider '{other}' — supported: ollama, ollama_cloud, openrouter, gemini, claude"),
+                format!("unknown provider '{other}' — supported: ollama, ollama_cloud, openrouter, gemini, claude, kimi, opencode"),
             ));
         }
     };
@@ -422,6 +506,17 @@ async fn chat(
         let output = resp.choices.first()
             .map(|c| c.message.text())
             .unwrap_or_default();
+        // Cross-runtime trace linkage. When a caller (validatord on
+        // Go side, /v1/iterate on Rust side) forwards a parent trace
+        // id via X-Lakehouse-Trace-Id, attach this generation to that
+        // trace so the iterate session and its inner chat hops show
+        // up as ONE trace tree in Langfuse. Header name matches the
+        // Go-side `shared.TraceIDHeader` constant byte-for-byte.
+        let parent_trace_id = headers
+            .get(crate::v1::iterate::TRACE_ID_HEADER)
+            .and_then(|v| v.to_str().ok())
+            .map(|s| s.to_string())
+            .filter(|s| !s.is_empty());
         lf.emit_chat(langfuse_trace::ChatTrace {
             provider: used_provider.clone(),
             model: resp.model.clone(),
@@ -435,6 +530,7 @@ async fn chat(
             start_time: start_time.to_rfc3339(),
             end_time: end_time.to_rfc3339(),
             latency_ms,
+            parent_trace_id,
         });
     }
 
@@ -501,6 +597,43 @@ async fn usage(State(state): State<V1State>) -> impl IntoResponse {
     Json(snapshot)
 }
 
+/// Production operational health endpoint.
+///
+/// `/v1/health` reports per-subsystem status as a JSON object so an
+/// operator (or the lakehouse-auditor service, or a load balancer)
+/// can verify the gateway is fully booted, has its provider keys
+/// loaded, the worker roster is hot, and Langfuse is reachable.
+/// Returns 200 always — fields are observed-state, not pass/fail
+/// gates. A monitoring tool should evaluate the booleans + counts
+/// against its own thresholds.
+async fn health(State(state): State<V1State>) -> impl IntoResponse {
+    // Honest worker count via WorkerLookup::len. Production switchover
+    // verification: after swapping workers_500k.parquet → real Chicago
+    // data and restarting, this number should match the row count of
+    // the new file. 0 means the file was missing / unreadable / had a
+    // schema mismatch and the gateway booted with the empty fallback.
+    let workers_count = state.validate_workers.len();
+    let providers_configured = serde_json::json!({
+        "ollama_cloud": state.ollama_cloud_key.is_some(),
+        "openrouter": state.openrouter_key.is_some(),
+        "kimi": state.kimi_key.is_some(),
+        "opencode": state.opencode_key.is_some(),
+        "gemini": state.gemini_key.is_some(),
+        "claude": state.claude_key.is_some(),
+    });
+    let langfuse_configured = state.langfuse.is_some();
+    let usage_snapshot = state.usage.read().await.clone();
+    Json(serde_json::json!({
+        "status": "ok",
+        "workers_count": workers_count,
+        "workers_loaded": workers_count > 0,
+        "providers_configured": providers_configured,
+        "langfuse_configured": langfuse_configured,
+        "usage_total_requests": usage_snapshot.requests,
+        "usage_by_provider": usage_snapshot.by_provider.keys().collect::<Vec<_>>(),
+    }))
+}
+
 // Phase 38 is stateless — no session persistence yet. Return an empty
 // list in OpenAI-ish shape so clients that probe this endpoint don't
 // 404. Real session state lands in Phase 41 with the profile-system

crates/gateway/src/v1/mode.rs

@@ -1032,14 +1032,14 @@ mod tests {
                     preferred_mode: "codereview".into(),
                     fallback_modes: vec!["consensus".into()],
                     default_model: "qwen3-coder:480b".into(),
-                    matrix_corpus: Some("distilled_procedural_v1".into()),
+                    matrix_corpus: vec!["distilled_procedural_v1".into()],
                 },
                 TaskClassEntry {
                     name: "broken".into(),
                     preferred_mode: "nonsense_mode".into(),
                     fallback_modes: vec!["consensus".into()],
                     default_model: "x".into(),
-                    matrix_corpus: None,
+                    matrix_corpus: vec![],
                 },
             ],
             default: DefaultEntry {

crates/gateway/src/v1/opencode.rs

@@ -0,0 +1,228 @@
+//! OpenCode GO adapter — multi-vendor curated gateway via opencode.ai/zen/go.
+//!
+//! One sk-* key reaches Claude Opus 4.7, GPT-5.5-pro, Gemini 3.1-pro,
+//! Kimi K2.6, DeepSeek, GLM, Qwen, plus 4 free-tier models.
+//! OpenAI-compatible Chat Completions; auth via Bearer.
+//!
+//! Why a separate adapter (vs reusing openrouter.rs):
+//! - Different account, different key, different base_url
+//! - No HTTP-Referer / X-Title headers (those are OpenRouter-specific)
+//! - Future-proof for any opencode-only request shaping
+//!
+//! Key sourcing priority:
+//!   1. Env var `OPENCODE_API_KEY` (loaded from /etc/lakehouse/opencode.env
+//!      via systemd EnvironmentFile=)
+//!   2. /etc/lakehouse/opencode.env directly (rescue path if env missing)
+//!
+//! Resolved once at gateway startup, stored on `V1State.opencode_key`.
+//! Model-prefix routing: "opencode/<model>" auto-routes here, prefix
+//! stripped before upstream call.
+
+use std::time::Duration;
+use serde::{Deserialize, Serialize};
+
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+// /zen/v1 is the unified OpenCode endpoint that covers BOTH the
+// Zen pay-per-token tier (Claude/GPT/Gemini frontier) AND the Go
+// subscription tier (Kimi/GLM/DeepSeek/Qwen/Minimax/mimo). When the
+// caller has both, opencode bills per-model: Zen models charge Zen
+// balance, Go models charge against the Go subscription cap.
+//
+// /zen/go/v1 exists as a Go-only sub-path (rejects Zen models with
+// "Model not supported"); we use the unified /zen/v1 since the same
+// key works for both with correct billing routing upstream.
+const OPENCODE_BASE_URL: &str = "https://opencode.ai/zen/v1";
+// 600s default — opencode upstream models include reasoning-heavy
+// variants (Claude Opus, Kimi K2.6, GLM-5.1) that legitimately take
+// 3-5 min on big audit prompts. Override via OPENCODE_TIMEOUT_SECS.
+const OPENCODE_TIMEOUT_SECS_DEFAULT: u64 = 600;
+
+fn opencode_timeout_secs() -> u64 {
+    std::env::var("OPENCODE_TIMEOUT_SECS")
+        .ok()
+        .and_then(|s| s.trim().parse::<u64>().ok())
+        .filter(|&n| n > 0)
+        .unwrap_or(OPENCODE_TIMEOUT_SECS_DEFAULT)
+}
+
+pub fn resolve_opencode_key() -> Option<String> {
+    if let Ok(k) = std::env::var("OPENCODE_API_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    if let Ok(raw) = std::fs::read_to_string("/etc/lakehouse/opencode.env") {
+        for line in raw.lines() {
+            if let Some(rest) = line.strip_prefix("OPENCODE_API_KEY=") {
+                let k = rest.trim().trim_matches('"').trim_matches('\'');
+                if !k.is_empty() { return Some(k.to_string()); }
+            }
+        }
+    }
+    None
+}
+
+pub async fn chat(
+    key: &str,
+    req: &ChatRequest,
+) -> Result<ChatResponse, String> {
+    // Strip the "opencode/" namespace prefix so the upstream sees the
+    // bare model id (e.g. "claude-opus-4-7", "kimi-k2.6").
+    let model = req.model.strip_prefix("opencode/").unwrap_or(&req.model).to_string();
+
+    // Anthropic models on opencode reject `temperature` with a 400
+    // "temperature is deprecated for this model" error. Strip the
+    // field for claude-* and the new gpt-5.x reasoning lineages
+    // (Anthropic/OpenAI's reasoning models all moved away from temp).
+    // Other models keep the caller's value or default to 0.3.
+    let drop_temp = model.starts_with("claude-")
+        || model.starts_with("gpt-5")
+        || model.starts_with("o1")
+        || model.starts_with("o3")
+        || model.starts_with("o4");
+    let body = OCChatBody {
+        model: model.clone(),
+        messages: req.messages.iter().map(|m| OCMessage {
+            role: m.role.clone(),
+            content: m.content.clone(),
+        }).collect(),
+        // filter(|&n| n > 0) catches Some(0) — same trap that bit the
+        // Kimi adapter when callers passed empty-env-parsed-to-0.
+        max_tokens: req.max_tokens.filter(|&n| n > 0).unwrap_or(800),
+        temperature: if drop_temp { None } else { Some(req.temperature.unwrap_or(0.3)) },
+        stream: false,
+    };
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(opencode_timeout_secs()))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+
+    let t0 = std::time::Instant::now();
+    let resp = client
+        .post(format!("{}/chat/completions", OPENCODE_BASE_URL))
+        .bearer_auth(key)
+        .json(&body)
+        .send()
+        .await
+        .map_err(|e| format!("opencode.ai unreachable: {e}"))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("opencode.ai {}: {}", status, body));
+    }
+
+    let parsed: OCChatResponse = resp.json().await
+        .map_err(|e| format!("invalid opencode response: {e}"))?;
+
+    let latency_ms = t0.elapsed().as_millis();
+    let choice = parsed.choices.into_iter().next()
+        .ok_or_else(|| "opencode returned no choices".to_string())?;
+    let text = choice.message.content;
+
+    let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
+        let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
+        ((chars + 3) / 4) as u32
+    });
+    let completion_tokens = parsed.usage.as_ref().map(|u| u.completion_tokens).unwrap_or_else(|| {
+        ((text.chars().count() + 3) / 4) as u32
+    });
+
+    tracing::info!(
+        target: "v1.chat",
+        provider = "opencode",
+        model = %model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "opencode chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model,
+        choices: vec![Choice {
+            index: 0,
+            message: Message { role: "assistant".into(), content: serde_json::Value::String(text) },
+            finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+// -- OpenCode wire shapes (OpenAI-compatible) --
+
+#[derive(Serialize)]
+struct OCChatBody {
+    model: String,
+    messages: Vec<OCMessage>,
+    max_tokens: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    temperature: Option<f64>,
+    stream: bool,
+}
+
+#[derive(Serialize)]
+struct OCMessage { role: String, content: serde_json::Value }
+
+#[derive(Deserialize)]
+struct OCChatResponse {
+    choices: Vec<OCChoice>,
+    #[serde(default)]
+    usage: Option<OCUsage>,
+}
+
+#[derive(Deserialize)]
+struct OCChoice {
+    message: OCMessageResp,
+    #[serde(default)]
+    finish_reason: Option<String>,
+}
+
+#[derive(Deserialize)]
+struct OCMessageResp { content: String }
+
+#[derive(Deserialize)]
+struct OCUsage { prompt_tokens: u32, completion_tokens: u32 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_opencode_key_does_not_panic() {
+        let _ = resolve_opencode_key();
+    }
+
+    #[test]
+    fn model_prefix_strip() {
+        let cases = [
+            ("opencode/claude-opus-4-7", "claude-opus-4-7"),
+            ("opencode/kimi-k2.6", "kimi-k2.6"),
+            ("claude-opus-4-7", "claude-opus-4-7"),
+        ];
+        for (input, expected) in cases {
+            let out = input.strip_prefix("opencode/").unwrap_or(input);
+            assert_eq!(out, expected);
+        }
+    }
+
+    #[test]
+    fn max_tokens_filters_zero() {
+        // The trap: empty env -> Number("") -> 0 -> Some(0). Adapter
+        // must not pass 0 upstream; should fall to 800.
+        let some_zero: Option<u32> = Some(0);
+        let result = some_zero.filter(|&n| n > 0).unwrap_or(800);
+        assert_eq!(result, 800);
+        let some_real: Option<u32> = Some(4096);
+        assert_eq!(some_real.filter(|&n| n > 0).unwrap_or(800), 4096);
+        let none_val: Option<u32> = None;
+        assert_eq!(none_val.filter(|&n| n > 0).unwrap_or(800), 800);
+    }
+}

crates/gateway/src/v1/openrouter.rs

@@ -57,7 +57,7 @@ pub async fn chat(
     // form so OpenRouter sees the raw model id (e.g. "openai/gpt-oss-120b:free").
     let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string();
 
-    let body = ORChatBody {
+    let mut body = ORChatBody {
         model: model.clone(),
         // Pass content through verbatim — preserves OpenAI's multimodal
         // content-parts shape (`[{type:"text",text:"..."}, ...]`) so the
@@ -66,7 +66,7 @@ pub async fn chat(
             role: m.role.clone(),
             content: m.content.clone(),
         }).collect(),
-        max_tokens: req.max_tokens.unwrap_or(800),
+        max_tokens: Some(req.max_tokens.unwrap_or(800)),
         temperature: req.temperature.unwrap_or(0.3),
         stream: false,
     };
@@ -77,7 +77,8 @@ pub async fn chat(
         .map_err(|e| format!("build client: {e}"))?;
 
     let t0 = std::time::Instant::now();
-    let resp = client
+    let send = |client: reqwest::Client, key: String, body_json: serde_json::Value| async move {
+        client
             .post(format!("{}/chat/completions", OR_BASE_URL))
             .bearer_auth(key)
             // OpenRouter recommends Referer + Title for attribution; absent
@@ -85,10 +86,52 @@ pub async fn chat(
             // their dashboard.
             .header("HTTP-Referer", "https://vcp.devop.live")
             .header("X-Title", "Lakehouse Scrum")
-        .json(&body)
+            .json(&body_json)
             .send()
             .await
-        .map_err(|e| format!("openrouter.ai unreachable: {e}"))?;
+            .map_err(|e| format!("openrouter.ai unreachable: {e}"))
+    };
+    let mut resp = send(
+        client.clone(),
+        key.to_string(),
+        serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
+    )
+    .await?;
+
+    // OpenRouter's affordability precheck reserves max_tokens × price against
+    // the account balance. Measured 2026-08-28: stripping the cap makes it
+    // WORSE (the precheck then prices the model's own maximum, e.g. 65536) —
+    // so a low-balance 402 surfaces verbatim and the fix is funding the
+    // account the key belongs to, not request-shape games. One clamp retry:
+    // when the error names an affordable budget ≥256 tokens, retry once at
+    // that budget so small answers still get through on a thin balance.
+    if resp.status() == reqwest::StatusCode::PAYMENT_REQUIRED && body.max_tokens.is_some() {
+        let err_body = resp.text().await.unwrap_or_else(|_| "?".into());
+        let afford = err_body
+            .split("can only afford ")
+            .nth(1)
+            .and_then(|s| s.split(|c: char| !c.is_ascii_digit()).next())
+            .and_then(|n| n.parse::<u32>().ok());
+        match afford {
+            Some(n) if n >= 256 => {
+                tracing::warn!(
+                    target: "v1.chat",
+                    provider = "openrouter",
+                    model = %model,
+                    afford = n,
+                    "402 affordability precheck — clamping max_tokens to the stated budget",
+                );
+                body.max_tokens = Some(n);
+                resp = send(
+                    client.clone(),
+                    key.to_string(),
+                    serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
+                )
+                .await?;
+            }
+            _ => return Err(format!("openrouter.ai 402 Payment Required: {}", err_body)),
+        }
+    }
 
     let status = resp.status();
     if !status.is_success() {
@@ -102,7 +145,21 @@ pub async fn chat(
     let latency_ms = t0.elapsed().as_millis();
     let choice = parsed.choices.into_iter().next()
         .ok_or_else(|| "openrouter returned no choices".to_string())?;
-    let text = choice.message.content;
+    let reasoning_only = choice.message.content.is_none() && choice.message.reasoning.is_some();
+    let text = choice.message.content.unwrap_or_default();
+    if reasoning_only {
+        // The model spent its whole budget thinking. Return well-formed empty
+        // content (callers' schema validation + retry budgets handle it
+        // deterministically) rather than a decode error that reads as a
+        // gateway fault — and never pass thinking-text off as an answer.
+        tracing::warn!(
+            target: "v1.chat",
+            provider = "openrouter",
+            model = %model,
+            finish = %choice.finish_reason.as_deref().unwrap_or("?"),
+            "reasoning-only response (content null) — raise max_tokens or use a non-reasoning model",
+        );
+    }
 
     let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
         let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
@@ -146,7 +203,8 @@ pub async fn chat(
 struct ORChatBody {
     model: String,
     messages: Vec<ORMessage>,
-    max_tokens: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    max_tokens: Option<u32>,
     temperature: f64,
     stream: bool,
 }
@@ -169,7 +227,18 @@ struct ORChoice {
 }
 
 #[derive(Deserialize)]
-struct ORMessageResp { content: String }
+struct ORMessageResp {
+    // Reasoning models (GLM-5.x, deepseek-v4-pro) return `content: null` plus a
+    // `reasoning` field when they exhaust max_tokens mid-thought. A bare String
+    // here made serde reject the whole body ("invalid openrouter response",
+    // 2026-08-28) — which took down every reasoning model on the lane,
+    // including phase2's configured glm-5.2. Content stays the only text we
+    // ever return; reasoning is observability, never an answer.
+    #[serde(default)]
+    content: Option<String>,
+    #[serde(default)]
+    reasoning: Option<String>,
+}
 
 #[derive(Deserialize)]
 struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
@@ -178,6 +247,21 @@ struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
 mod tests {
     use super::*;
 
+    #[test]
+    fn reasoning_model_null_content_parses() {
+        // GLM-5.x / deepseek-v4-pro shape: content null + reasoning present
+        // (the 2026-08-28 decode failure). Must parse, not error.
+        let body = r#"{"choices":[{"message":{"role":"assistant","content":null,"reasoning":"thinking…"},"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":10}}"#;
+        let parsed: ORChatResponse = serde_json::from_str(body).expect("null content must parse");
+        let choice = &parsed.choices[0];
+        assert!(choice.message.content.is_none());
+        assert_eq!(choice.message.reasoning.as_deref(), Some("thinking…"));
+        // And the normal shape still parses with reasoning absent.
+        let normal = r#"{"choices":[{"message":{"role":"assistant","content":"OK"},"finish_reason":"stop"}]}"#;
+        let parsed: ORChatResponse = serde_json::from_str(normal).expect("normal shape");
+        assert_eq!(parsed.choices[0].message.content.as_deref(), Some("OK"));
+    }
+
     #[test]
     fn resolve_openrouter_key_does_not_panic() {
         // Smoke test — all three sources may or may not be set depending
@@ -192,7 +276,7 @@ mod tests {
             messages: vec![
                 ORMessage { role: "user".into(), content: "review this".into() },
             ],
-            max_tokens: 800,
+            max_tokens: Some(800),
             temperature: 0.3,
             stream: false,
         };

crates/gateway/src/v1/session_log.rs

@@ -0,0 +1,235 @@
+//! Coordinator session JSONL writer — Rust parity with the Go-side
+//! `internal/validator/session_log.go` (commit 1a3a82a in
+//! golangLAKEHOUSE). Same schema, same field names, same producer
+//! semantics, so a unified longitudinal log can pull from either
+//! runtime via DuckDB.
+//!
+//! Schema: `session.iterate.v1`. One row per `/v1/iterate` session.
+//! Append-only. Best-effort posture: errors warn and the iterate
+//! response always ships.
+//!
+//! See `golangLAKEHOUSE/docs/SESSION_LOG.md` for the full schema
+//! reference + DuckDB query examples. This module produces rows
+//! with `daemon: "gateway"`; the Go side produces `daemon:
+//! "validatord"`. Operators who want a unified stream can point both
+//! to the same path (the OS write-append is atomic for the row sizes
+//! we produce) or query both files together via duckdb's `read_json`
+//! glob support.
+
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+pub const SESSION_RECORD_SCHEMA: &str = "session.iterate.v1";
+
+/// One row in coordinator_sessions.jsonl. Field names are the on-wire
+/// names — must stay byte-equal to the Go side's
+/// `validator.SessionRecord` (proven by the cross-runtime parity
+/// probe at golangLAKEHOUSE/scripts/cutover/parity/).
+// Deserialize is supported so the parity helper binary can round-trip
+// fixture inputs through serde without hand-rolling a parser. Production
+// emit path uses Serialize only; SessionRecord rows are written by the
+// gateway and consumed by DuckDB / external tooling, never re-read by us.
+#[derive(Serialize, Deserialize)]
+pub struct SessionRecord {
+    pub schema: String,
+    pub session_id: String,
+    pub timestamp: String,
+    pub daemon: String,
+    pub kind: String,
+    pub model: String,
+    pub provider: String,
+    pub prompt: String,
+    pub iterations: u32,
+    pub max_iterations: u32,
+    pub final_verdict: String, // "accepted" | "max_iter_exhausted" | "infra_error"
+    pub attempts: Vec<SessionAttemptRecord>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub artifact: Option<serde_json::Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub grounded_in_roster: Option<bool>,
+    pub duration_ms: u64,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct SessionAttemptRecord {
+    pub iteration: u32,
+    pub verdict_kind: String, // "no_json" | "validation_failed" | "accepted" | "infra_error"
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub error: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub span_id: Option<String>,
+}
+
+/// Append-only writer. Cloneable handle — internal state is Arc'd so
+/// V1State can keep its own clone and per-request clones are cheap.
+#[derive(Clone)]
+pub struct SessionLogger {
+    inner: Arc<Inner>,
+}
+
+struct Inner {
+    path: String,
+    /// tokio::Mutex (not std) because we hold it across the async
+    /// fs write. Contention is low (one row per /v1/iterate session).
+    mu: Mutex<()>,
+}
+
+impl SessionLogger {
+    /// Construct a logger writing to `path`. Empty path → None
+    /// (skip the wiring in the iterate handler entirely).
+    pub fn from_path(path: &str) -> Option<Self> {
+        if path.is_empty() {
+            return None;
+        }
+        Some(Self {
+            inner: Arc::new(Inner {
+                path: path.to_string(),
+                mu: Mutex::new(()),
+            }),
+        })
+    }
+
+    /// Append one record. Best-effort: failures land in `tracing::warn!`
+    /// and the caller sees Ok(()) — observability is a witness, never
+    /// a gate. Returns Err only on impossible cases the type system
+    /// can't rule out (here: serde_json::to_string failing on a
+    /// well-formed struct, which shouldn't happen).
+    pub async fn append(&self, rec: SessionRecord) {
+        let body = match serde_json::to_string(&rec) {
+            Ok(s) => s,
+            Err(e) => {
+                tracing::warn!(target: "v1.session_log", "marshal: {e}");
+                return;
+            }
+        };
+        let _guard = self.inner.mu.lock().await;
+        if let Err(e) = self.write(&body).await {
+            tracing::warn!(target: "v1.session_log", "write {}: {e}", self.inner.path);
+        }
+    }
+
+    async fn write(&self, body: &str) -> std::io::Result<()> {
+        use tokio::fs::OpenOptions;
+        use tokio::io::AsyncWriteExt;
+
+        // Lazy mkdir on first write so a not-yet-mounted volume at
+        // startup doesn't kill the daemon.
+        if let Some(parent) = std::path::Path::new(&self.inner.path).parent() {
+            if !parent.as_os_str().is_empty() {
+                tokio::fs::create_dir_all(parent).await?;
+            }
+        }
+        let mut f = OpenOptions::new()
+            .append(true)
+            .create(true)
+            .open(&self.inner.path)
+            .await?;
+        f.write_all(body.as_bytes()).await?;
+        f.write_all(b"\n").await?;
+        Ok(())
+    }
+}
+
+/// Best-effort UTF-8 char truncation. Matches Go's `trim` helper so
+/// rows produced by either runtime cap fields at the same boundaries.
+pub fn truncate(s: &str, n: usize) -> String {
+    s.chars().take(n).collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::PathBuf;
+    use tokio::fs;
+
+    fn fixture_record(session_id: &str) -> SessionRecord {
+        SessionRecord {
+            schema: SESSION_RECORD_SCHEMA.to_string(),
+            session_id: session_id.to_string(),
+            timestamp: "2026-05-02T08:00:00Z".to_string(),
+            daemon: "gateway".to_string(),
+            kind: "fill".to_string(),
+            model: "qwen3.5:latest".to_string(),
+            provider: "ollama".to_string(),
+            prompt: "produce a fill artifact".to_string(),
+            iterations: 1,
+            max_iterations: 3,
+            final_verdict: "accepted".to_string(),
+            attempts: vec![SessionAttemptRecord {
+                iteration: 0,
+                verdict_kind: "accepted".to_string(),
+                error: None,
+                span_id: Some("span-0".to_string()),
+            }],
+            artifact: Some(serde_json::json!({"fills":[{"candidate_id":"W-1"}]})),
+            grounded_in_roster: Some(true),
+            duration_ms: 50,
+        }
+    }
+
+    #[tokio::test]
+    async fn from_path_empty_returns_none() {
+        assert!(SessionLogger::from_path("").is_none());
+    }
+
+    #[tokio::test]
+    async fn append_writes_jsonl_row_with_schema_field() {
+        let dir = tempdir();
+        let path = dir.join("sessions.jsonl");
+        let path_str = path.to_string_lossy().to_string();
+        let logger = SessionLogger::from_path(&path_str).unwrap();
+        logger.append(fixture_record("trace-a")).await;
+
+        let body = fs::read_to_string(&path).await.unwrap();
+        assert!(body.contains("\"schema\":\"session.iterate.v1\""));
+        assert!(body.contains("\"session_id\":\"trace-a\""));
+        assert!(body.contains("\"grounded_in_roster\":true"));
+        assert!(body.ends_with('\n'));
+    }
+
+    #[tokio::test]
+    async fn append_concurrent_safe() {
+        let dir = tempdir();
+        let path = dir.join("sessions.jsonl");
+        let path_str = path.to_string_lossy().to_string();
+        let logger = SessionLogger::from_path(&path_str).unwrap();
+
+        let n = 32;
+        let mut handles = Vec::with_capacity(n);
+        for i in 0..n {
+            let l = logger.clone();
+            handles.push(tokio::spawn(async move {
+                l.append(fixture_record(&format!("trace-{i}"))).await;
+            }));
+        }
+        for h in handles {
+            h.await.unwrap();
+        }
+
+        let body = fs::read_to_string(&path).await.unwrap();
+        let lines: Vec<_> = body.lines().filter(|l| !l.is_empty()).collect();
+        assert_eq!(lines.len(), n, "expected {n} rows, got {}", lines.len());
+        // Every row must round-trip through serde — a torn write
+        // would surface as a parse error.
+        for line in lines {
+            let _: serde_json::Value = serde_json::from_str(line).expect("valid json per row");
+        }
+    }
+
+    fn tempdir() -> PathBuf {
+        // Per-test unique path so prior runs don't pollute the next.
+        // The static counter increments across the whole test binary,
+        // so back-to-back tests in the same module get distinct dirs.
+        static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
+        let n = COUNTER.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+        let p = std::env::temp_dir().join(format!(
+            "session_log_test_{}_{}_{}",
+            std::process::id(),
+            chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0),
+            n,
+        ));
+        std::fs::create_dir_all(&p).unwrap();
+        p
+    }
+}

crates/gateway/src/v1/validate.rs

@@ -0,0 +1,82 @@
+//! /v1/validate — gateway-side artifact validation endpoint.
+//!
+//! Phase 43 v3 part 2: makes the validator crate network-callable.
+//! Any caller (scrum loop, test harness, future agent) can POST a
+//! generated artifact and get back a Report (success) or
+//! ValidationError (failure with structured field/reason).
+//!
+//! Request shape:
+//!   POST /v1/validate
+//!   {
+//!     "kind": "fill" | "email" | "playbook",
+//!     "artifact": { ... },
+//!     "context": { ... }   // optional — folded into artifact._context
+//!   }
+//!
+//! Response on success: 200 + Report JSON
+//! Response on failure: 422 + ValidationError JSON
+//! Response on bad request: 400 + plain-text error
+//!
+//! The shared WorkerLookup is loaded once at gateway startup from
+//! workers_500k.parquet (path configurable via LH_WORKERS_PARQUET
+//! env, defaults to data/datasets/workers_500k.parquet). Falls back
+//! to an empty InMemoryWorkerLookup if the file is missing — the
+//! validators will still run schema/length/PII checks but worker-
+//! existence checks will all fail (Consistency error), which is the
+//! correct behavior when the roster isn't configured.
+
+use axum::{extract::State, http::StatusCode, response::IntoResponse, Json};
+use serde::Deserialize;
+use validator::{
+    Artifact, Validator, ValidationError,
+    staffing::{
+        fill::FillValidator,
+        email::EmailValidator,
+        playbook::PlaybookValidator,
+    },
+};
+
+#[derive(Deserialize)]
+pub struct ValidateRequest {
+    /// `"fill" | "email" | "playbook"` — picks which validator runs.
+    pub kind: String,
+    /// The artifact JSON (free-form; shape depends on `kind`).
+    pub artifact: serde_json::Value,
+    /// Optional context bag — merged into `artifact._context` so the
+    /// validator can read fields like `target_count`, `city`,
+    /// `client_id`, `candidate_id` without callers having to embed
+    /// `_context` in the artifact themselves.
+    #[serde(default)]
+    pub context: Option<serde_json::Value>,
+}
+
+pub async fn validate(
+    State(state): State<super::V1State>,
+    Json(req): Json<ValidateRequest>,
+) -> impl IntoResponse {
+    // Merge context into artifact under `_context` so validators can
+    // pull contract metadata uniformly.
+    let mut artifact_value = req.artifact;
+    if let Some(ctx) = req.context {
+        if let Some(obj) = artifact_value.as_object_mut() {
+            obj.insert("_context".to_string(), ctx);
+        }
+    }
+
+    // Dispatch.
+    let workers = state.validate_workers.clone();
+    let result: Result<validator::Report, ValidationError> = match req.kind.as_str() {
+        "fill" => FillValidator::new(workers).validate(&Artifact::FillProposal(artifact_value)),
+        "email" => EmailValidator::new(workers).validate(&Artifact::EmailDraft(artifact_value)),
+        "playbook" => PlaybookValidator.validate(&Artifact::Playbook(artifact_value)),
+        other => return (
+            StatusCode::BAD_REQUEST,
+            format!("unknown kind '{other}' — expected fill | email | playbook"),
+        ).into_response(),
+    };
+
+    match result {
+        Ok(report) => (StatusCode::OK, Json(report)).into_response(),
+        Err(e) => (StatusCode::UNPROCESSABLE_ENTITY, Json(e)).into_response(),
+    }
+}

crates/lance-bench/src/main.rs

@@ -456,6 +456,26 @@ async fn build_lance_vector_index(path: &str, _dims: usize) -> Result<()> {
         .await
         .context("create_index")?;
 
+    // Also build the scalar btree on doc_id. This bench's
+    // measure_random_access_lance uses take(row_position) which doesn't
+    // need the btree, but the dataset this bench writes is also queried
+    // downstream by /vectors/lance/doc/<name>/<doc_id> (the production
+    // lookup path) — without this index that path falls back to a full
+    // table scan. Cheap to build (~1.2s on 10M rows) and matches the
+    // gateway's lance_migrate handler behavior so bench-produced datasets
+    // are immediately production-shape.
+    use lance_index::scalar::ScalarIndexParams;
+    dataset
+        .create_index(
+            &["doc_id"],
+            IndexType::Scalar,
+            Some("doc_id_btree".into()),
+            &ScalarIndexParams::default(),
+            true,
+        )
+        .await
+        .context("create_index doc_id btree")?;
+
     Ok(())
 }
 

crates/shared/src/config.rs

@@ -62,6 +62,15 @@ pub struct GatewayConfig {
     pub host: String,
     #[serde(default = "default_gateway_port")]
     pub port: u16,
+    /// Coordinator session JSONL output path. One row per
+    /// `/v1/iterate` session, schema=`session.iterate.v1`. Empty =
+    /// disabled. Cross-runtime parity with the Go side's
+    /// `[validatord].session_log_path` (added 2026-05-02). Default
+    /// empty so existing deployments aren't perturbed; production
+    /// sets `/var/lib/lakehouse/gateway/sessions.jsonl`. See
+    /// `golangLAKEHOUSE/docs/SESSION_LOG.md` for query examples.
+    #[serde(default)]
+    pub session_log_path: String,
 }
 
 #[derive(Debug, Clone, Deserialize)]
@@ -149,7 +158,13 @@ fn default_gateway_port() -> u16 { 3100 }
 fn default_storage_root() -> String { "./data".to_string() }
 fn default_profile_root() -> String { "./data/_profiles".to_string() }
 fn default_manifest_prefix() -> String { "_catalog/manifests".to_string() }
-fn default_sidecar_url() -> String { "http://localhost:3200".to_string() }
+// Post-2026-05-02: AiClient talks directly to Ollama; the Python
+// sidecar's hot-path role was retired. The config field name
+// `[sidecar].url` is preserved for migration compatibility (operators
+// with existing TOMLs don't need to rename anything), but the value
+// now points at Ollama. Lab UI / pipeline_lab Python remains as a
+// dev-only tool; not on this URL.
+fn default_sidecar_url() -> String { "http://localhost:11434".to_string() }
 fn default_embed_model() -> String { "nomic-embed-text".to_string() }
 fn default_gen_model() -> String { "qwen2.5".to_string() }
 fn default_rerank_model() -> String { "qwen2.5".to_string() }
@@ -184,7 +199,11 @@ impl Config {
 impl Default for Config {
     fn default() -> Self {
         Self {
-            gateway: GatewayConfig { host: default_host(), port: default_gateway_port() },
+            gateway: GatewayConfig {
+                host: default_host(),
+                port: default_gateway_port(),
+                session_log_path: String::new(),
+            },
             storage: StorageConfig {
                 root: default_storage_root(),
                 profile_root: default_profile_root(),

crates/shared/src/types.rs

@@ -364,3 +364,228 @@ pub enum Redaction {
         keep_suffix: usize,
     },
 }
+
+// ─── Subject manifests (compliance/access metadata layer) ────────────
+// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
+//
+// A SubjectManifest is a per-person record threading existing catalogd
+// primitives (datasets, views, tombstones) together by subject token.
+// Stored at _catalog/subjects/<candidate_id>.json. NOT a separate
+// daemon — extends catalogd's existing manifest pattern.
+//
+// The audit log per subject (HMAC-chained JSONL at
+// _catalog/subjects/<candidate_id>.audit.jsonl) is owned by a separate
+// SubjectAuditWriter (Step 2) — not modeled here.
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum SubjectStatus {
+    PendingConsent,
+    Active,
+    Withdrawn,
+    RetentionExpired,
+    Erased,
+}
+
+/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed:
+/// the gateway treats Unknown as healthcare-equivalent until reclassified
+/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1).
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum SubjectVertical {
+    Unknown,
+    General,
+    Healthcare,
+    Finance,
+    Other,
+}
+
+impl Default for SubjectVertical {
+    fn default() -> Self { SubjectVertical::Unknown }
+}
+
+/// State machine for general-PII consent. BIPA-specific biometric
+/// consent is tracked separately on `BiometricConsent`.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum ConsentStatus {
+    /// Backfilled from existing data sources before consent was tracked.
+    /// SHOULD route as if no consent exists until reviewed.
+    PendingBackfillReview,
+    /// New subject; awaiting consent UX.
+    PendingFirstContact,
+    Given,
+    Withdrawn,
+    Expired,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum BiometricConsentStatus {
+    /// Default. No biometric data exists for this subject.
+    NeverCollected,
+    Pending,
+    Given,
+    Withdrawn,
+    Expired,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct GeneralPiiConsent {
+    pub status: ConsentStatus,
+    /// Reference to the consent template version (matches a row in the
+    /// future consent_versions table). Empty when status is pending_*.
+    #[serde(default)]
+    pub version: String,
+    #[serde(default)]
+    pub given_at: Option<chrono::DateTime<chrono::Utc>>,
+    #[serde(default)]
+    pub withdrawn_at: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BiometricConsent {
+    pub status: BiometricConsentStatus,
+    /// BIPA: max 3 years from last interaction. Implementation MUST
+    /// enforce daily expiration sweep against this field.
+    #[serde(default)]
+    pub retention_until: Option<chrono::DateTime<chrono::Utc>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectConsent {
+    pub general_pii: GeneralPiiConsent,
+    pub biometric: BiometricConsent,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectRetention {
+    pub general_pii_until: chrono::DateTime<chrono::Utc>,
+    /// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026").
+    #[serde(default)]
+    pub policy: String,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectDatasetRef {
+    /// catalogd dataset name (must resolve to an existing manifest).
+    pub name: String,
+    /// Column in that dataset that holds the subject's identifier.
+    pub key_column: String,
+    /// Specific identifier value (typically same as candidate_id; differs
+    /// only when a dataset uses a foreign-key with a different format).
+    pub key_value: String,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectManifest {
+    /// Schema version tag — always "subject_manifest.v1" for v1.
+    #[serde(default = "default_subject_manifest_schema")]
+    pub schema: String,
+    /// Subject identifier (the canonical token, e.g. "CAND-000001").
+    pub candidate_id: String,
+    pub created_at: chrono::DateTime<chrono::Utc>,
+    pub updated_at: chrono::DateTime<chrono::Utc>,
+    pub status: SubjectStatus,
+    #[serde(default)]
+    pub vertical: SubjectVertical,
+    pub consent: SubjectConsent,
+    pub retention: SubjectRetention,
+    /// Datasets that contain rows for this subject. Used by audit-response
+    /// to know what to project. Each entry MUST reference an existing
+    /// catalogd dataset manifest (validated on put_subject).
+    pub datasets: Vec<SubjectDatasetRef>,
+    /// Names of existing AiViews that safely project this subject's data.
+    /// Used by service-tier readers (no PII exposure).
+    #[serde(default)]
+    pub safe_views: Vec<String>,
+    /// Path (relative to repo root) of the per-subject audit JSONL.
+    /// Owned by SubjectAuditWriter (Step 2). Empty until first write.
+    #[serde(default)]
+    pub audit_log_path: String,
+    /// SHA-256 of the most recent HMAC chain checkpoint of the audit log.
+    /// Empty until first audit row written.
+    #[serde(default)]
+    pub audit_log_chain_root: String,
+    /// Biometric collection metadata. None until the first biometric
+    /// upload via Phase 1.6 Gate 3 photo endpoint. Cleared on erasure.
+    /// `serde(default)` so existing manifests parse without re-emit.
+    #[serde(default)]
+    pub biometric_collection: Option<BiometricCollection>,
+}
+
+/// Per-subject biometric collection record. Written by the Gate 3
+/// photo-upload endpoint when a candidate's photo is accepted under
+/// `consent.biometric.status == "given"`.
+///
+/// Reset to None by the BIPA erasure flow (destruction runbook).
+/// Preserved across schema evolutions via the BTreeMap-canonicalized
+/// HMAC chain — old audit rows verify regardless of new fields.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub struct BiometricCollection {
+    /// Quarantined filesystem path of the original photo bytes,
+    /// relative to the configured biometric storage root. NEVER under
+    /// `data/headshots/` (which is reserved for synthetic StyleGAN
+    /// faces). Format: `data/biometric/uploads/<sanitized_id>/<ts>.<ext>`.
+    pub data_path: String,
+    /// SHA-256 of the original photo bytes. Used for integrity check
+    /// (file-on-disk matches what was uploaded), NOT for
+    /// re-derivation of the original from the hash.
+    pub template_hash: String,
+    pub collected_at: chrono::DateTime<chrono::Utc>,
+    /// SHA-256 of the consent template version that was in force when
+    /// this photo was uploaded. Anchors the consent record to a
+    /// specific document version (per Phase 1.6 Gate 2 §6).
+    #[serde(default)]
+    pub consent_version_hash: String,
+    /// deepface classifications, when the offline classifier has run.
+    /// None until classifier integration ships (Gate 3b).
+    #[serde(default)]
+    pub classifications: Option<serde_json::Value>,
+}
+
+fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() }
+
+/// Audit accessor — who/why touched a subject's PII.
+/// Persisted in subject_audit.v1 rows; never logged in plaintext outside
+/// the per-subject audit file.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct AuditAccessor {
+    /// One of: "gateway_lookup", "audit_response", "legal_request",
+    /// "system_resolve". Caller-defined; SubjectAuditWriter does not
+    /// validate the kind string.
+    pub kind: String,
+    /// Daemon name (e.g. "gateway", "validator", "catalogd").
+    pub daemon: String,
+    /// Purpose token (e.g. "fill_validation", "audit_subject_response").
+    pub purpose: String,
+    /// X-Lakehouse-Trace-Id for cross-daemon trace correlation. Empty
+    /// when the request didn't carry one.
+    #[serde(default)]
+    pub trace_id: String,
+}
+
+/// One audit row in a subject's per-subject audit JSONL.
+/// HMAC chain: row_hmac = HMAC-SHA256(key, prev_chain_hash || canon(row - row_hmac))
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SubjectAuditRow {
+    /// Always "subject_audit.v1" for v1.
+    #[serde(default = "default_subject_audit_schema")]
+    pub schema: String,
+    pub ts: chrono::DateTime<chrono::Utc>,
+    pub candidate_id: String,
+    pub accessor: AuditAccessor,
+    /// Field names accessed (e.g. ["name"], ["name", "phone"]).
+    pub fields_accessed: Vec<String>,
+    /// "success" | "denied" | "not_found" | "error".
+    pub result: String,
+    /// Hash of the previous row in this subject's audit log. "GENESIS"
+    /// for the first row.
+    pub prev_chain_hash: String,
+    /// HMAC-SHA256 over (prev_chain_hash || canonical_json(row - row_hmac)).
+    /// Computed by SubjectAuditWriter; callers leave empty on input.
+    #[serde(default)]
+    pub row_hmac: String,
+}
+
+fn default_subject_audit_schema() -> String { "subject_audit.v1".into() }

crates/validator/Cargo.toml

@@ -9,3 +9,7 @@ serde_json = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true }
 tracing = { workspace = true }
+# Parquet loader for ParquetWorkerLookup (Phase 43 v3 — production
+# WorkerLookup backed by workers_500k.parquet snapshot).
+arrow = { workspace = true }
+parquet = { workspace = true }

crates/validator/src/lib.rs

@@ -93,3 +93,89 @@ pub trait Validator: Send + Sync {
     /// Human-readable name for logs + Langfuse traces.
     fn name(&self) -> &'static str;
 }
+
+// ─── Worker lookup (Phase 43 v2) ────────────────────────────────────────
+//
+// Validators that cross-check artifacts against the worker roster
+// (FillValidator, EmailValidator) take an `Arc<dyn WorkerLookup>` at
+// construction. Keeping the trait sync + in-memory mirrors the
+// lakehouse pattern of "load truth into memory, validate against
+// snapshot, refresh periodically" rather than per-call DB hits.
+//
+// Production impl: wrap a parquet snapshot loaded from
+// `data/datasets/workers_500k.parquet` (or its safe view counterpart
+// once Track A.B lands). Tests use `InMemoryWorkerLookup`.
+
+/// One worker row from the staffing roster — the fields validators
+/// actually read. Anything not on this struct (resume_text, scores,
+/// communications) is intentionally hidden from the validator path.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct WorkerRecord {
+    pub candidate_id: String,
+    pub name: String,
+    /// Free-form. Validators check for `"active"` (any other value
+    /// fails the status check). Common values from existing data:
+    /// "active", "inactive", "placed", "blacklisted".
+    pub status: String,
+    pub city: Option<String>,
+    pub state: Option<String>,
+    pub role: Option<String>,
+    /// Client ids this worker has been blacklisted from. Populated
+    /// from joining a blacklist table; empty when not provided.
+    #[serde(default)]
+    pub blacklisted_clients: Vec<String>,
+}
+
+/// Worker lookup contract. Sync by design — implementations should
+/// hold an in-memory snapshot, not perform per-call I/O.
+pub trait WorkerLookup: Send + Sync {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord>;
+    /// Number of workers in the snapshot. Default 0 for impls that
+    /// genuinely don't know (e.g. a future SQL-backed lookup that
+    /// counts on demand). InMemoryWorkerLookup overrides with the
+    /// HashMap size; ParquetWorkerLookup constructs an
+    /// InMemoryWorkerLookup so it inherits the override. Used by
+    /// /v1/health to report data-load status during production
+    /// switchover (the Chicago dataset replaces synthetic test data;
+    /// the health endpoint is how operators verify the new file
+    /// loaded correctly without restart-and-pray).
+    fn len(&self) -> usize { 0 }
+}
+
+/// HashMap-backed lookup. Used by validator unit tests + as a
+/// reasonable bootstrap impl for production once the parquet loader
+/// fills it on startup.
+pub struct InMemoryWorkerLookup {
+    rows: std::collections::HashMap<String, WorkerRecord>,
+}
+
+impl InMemoryWorkerLookup {
+    pub fn new() -> Self {
+        Self { rows: Default::default() }
+    }
+    pub fn from_records(records: Vec<WorkerRecord>) -> Self {
+        let mut rows = std::collections::HashMap::with_capacity(records.len());
+        for r in records {
+            rows.insert(r.candidate_id.clone(), r);
+        }
+        Self { rows }
+    }
+    pub fn insert(&mut self, record: WorkerRecord) {
+        self.rows.insert(record.candidate_id.clone(), record);
+    }
+    pub fn len(&self) -> usize { self.rows.len() }
+    pub fn is_empty(&self) -> bool { self.rows.is_empty() }
+}
+
+impl Default for InMemoryWorkerLookup {
+    fn default() -> Self { Self::new() }
+}
+
+impl WorkerLookup for InMemoryWorkerLookup {
+    fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
+        self.rows.get(candidate_id).cloned()
+    }
+    fn len(&self) -> usize {
+        self.rows.len()
+    }
+}

crates/validator/src/staffing/email.rs

@@ -1,4 +1,4 @@
-//! Email/SMS draft validator.
+//! Email/SMS draft validator (Phase 43 v2 — real PII + name checks).
 //!
 //! PRD checks:
 //!   - Schema (TO/BODY fields present)
@@ -6,15 +6,31 @@
 //!   - PII absence (no SSN / salary leaked into outgoing text)
 //!   - Worker-name consistency (name in message matches worker record)
 //!
-//! Scaffold implements schema + length. PII regex (SSN pattern,
-//! salary-number pattern) lives in `shared::pii::strip_pii` — plug in
-//! a follow-up when the validator caller knows the worker record to
-//! cross-check name consistency.
+//! Like FillValidator, EmailValidator takes `Arc<dyn WorkerLookup>` at
+//! construction. The contract metadata (which worker the message is
+//! about) travels under `_context.candidate_id` in the JSON payload.
+//! When `_context.candidate_id` is present and resolves, the validator
+//! cross-checks that the worker's name appears verbatim in the body.
+//!
+//! PII detection is std-only (no regex dep) — a hand-rolled scan
+//! covers the patterns we actually care about: SSN (NNN-NN-NNNN),
+//! salary statements ("salary" / "compensation" near a $ amount).
 
-use crate::{Artifact, Report, Validator, ValidationError};
+use crate::{
+    Artifact, Report, Validator, ValidationError, WorkerLookup,
+};
+use std::sync::Arc;
 use std::time::Instant;
 
-pub struct EmailValidator;
+pub struct EmailValidator {
+    workers: Arc<dyn WorkerLookup>,
+}
+
+impl EmailValidator {
+    pub fn new(workers: Arc<dyn WorkerLookup>) -> Self {
+        Self { workers }
+    }
+}
 
 const SMS_MAX_CHARS: usize = 160;
 const EMAIL_SUBJECT_MAX_CHARS: usize = 78;
@@ -32,7 +48,7 @@ impl Validator for EmailValidator {
             }),
         };
 
-        let to = value.get("to").and_then(|v| v.as_str()).ok_or(
+        let _to = value.get("to").and_then(|v| v.as_str()).ok_or(
             ValidationError::Schema {
                 field: "to".into(),
                 reason: "missing or not a string".into(),
@@ -63,54 +79,292 @@ impl Validator for EmailValidator {
             }
         }
 
-        let _ = to; // touched for future name-consistency check
-        // TODO(phase-43 v2): PII scan + worker-name consistency.
+        // ── PII scan on body + subject combined ──
+        let scanned = format!(
+            "{} {}",
+            value.get("subject").and_then(|v| v.as_str()).unwrap_or(""),
+            body
+        );
+        if contains_ssn_pattern(&scanned) {
+            return Err(ValidationError::Policy {
+                reason: "body contains an SSN-shaped sequence (NNN-NN-NNNN); strip before send".into(),
+            });
+        }
+        if contains_salary_disclosure(&scanned) {
+            return Err(ValidationError::Policy {
+                reason: "body discloses salary/compensation amount; staffing PII rule says strip before send".into(),
+            });
+        }
+
+        // ── Worker-name consistency ──
+        let candidate_id = value.get("_context")
+            .and_then(|c| c.get("candidate_id"))
+            .and_then(|v| v.as_str());
+        let mut findings: Vec<crate::Finding> = vec![];
+        if let Some(cid) = candidate_id {
+            match self.workers.find(cid) {
+                Some(worker) => {
+                    // Body should mention the worker's name (or at least
+                    // their first name) — drafts that address a different
+                    // person than the contracted worker are a recurring
+                    // class of LLM mistake.
+                    let first = worker.name.split_whitespace().next().unwrap_or(&worker.name);
+                    let body_lower = body.to_lowercase();
+                    let first_lower = first.to_lowercase();
+                    if !first_lower.is_empty() && !body_lower.contains(&first_lower) {
+                        findings.push(crate::Finding {
+                            field: "body".into(),
+                            severity: crate::Severity::Warning,
+                            message: format!(
+                                "body doesn't mention worker first name {first:?} (candidate_id {cid:?})"
+                            ),
+                        });
+                    }
+                    // Also detect *another* worker's name appearing in
+                    // place of the contracted one — outright wrong-target.
+                    // We can only check this when we have a different
+                    // expected name; skip if the body is generic enough.
+                }
+                None => {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "_context.candidate_id {cid:?} not found in worker roster"
+                        ),
+                    });
+                }
+            }
+        }
 
         Ok(Report {
-            findings: vec![],
+            findings,
             elapsed_ms: started.elapsed().as_millis() as u64,
         })
     }
 }
 
+// ─── PII scanners (std-only) ────────────────────────────────────────────
+
+/// Detects an SSN-shaped sequence: 3 digits, dash, 2 digits, dash, 4 digits.
+/// Walks the byte buffer; rejects sequences that are part of a longer run
+/// of digits (so phone-area-code-like NNN-NNN-NNNN isn't flagged). Tight
+/// false-positive surface: it's specifically the NNN-NN-NNNN shape.
+fn contains_ssn_pattern(s: &str) -> bool {
+    let bytes = s.as_bytes();
+    if bytes.len() < 11 { return false; }
+    for i in 0..=bytes.len().saturating_sub(11) {
+        let win = &bytes[i..i + 11];
+        let shape = win.iter().enumerate().all(|(j, &b)| match j {
+            0 | 1 | 2 | 4 | 5 | 7 | 8 | 9 | 10 => b.is_ascii_digit(),
+            3 | 6 => b == b'-',
+            _ => unreachable!(),
+        });
+        if !shape { continue; }
+        // Reject if the byte BEFORE this window is a digit or `-` —
+        // we're inside a longer numeric run, probably not an SSN.
+        if i > 0 {
+            let prev = bytes[i - 1];
+            if prev.is_ascii_digit() || prev == b'-' { continue; }
+        }
+        // Reject if the byte AFTER is a digit or `-` (same reason).
+        if i + 11 < bytes.len() {
+            let next = bytes[i + 11];
+            if next.is_ascii_digit() || next == b'-' { continue; }
+        }
+        return true;
+    }
+    false
+}
+
+/// Detects salary/compensation disclosure: the keywords "salary",
+/// "compensation", "pay rate", "bill rate", "hourly rate" appearing
+/// within ~40 chars of a `$` followed by digits. Coarse on purpose —
+/// it's better to false-positive on a legit phrase like "discuss your
+/// hourly rate of $30/hr" than to miss it.
+fn contains_salary_disclosure(s: &str) -> bool {
+    let lower = s.to_lowercase();
+    const KEYWORDS: &[&str] = &[
+        "salary", "compensation", "pay rate", "bill rate", "hourly rate",
+    ];
+    let mut keyword_positions: Vec<usize> = vec![];
+    for kw in KEYWORDS {
+        let mut start = 0;
+        while let Some(found) = lower[start..].find(kw) {
+            let abs = start + found;
+            keyword_positions.push(abs);
+            start = abs + kw.len();
+        }
+    }
+    if keyword_positions.is_empty() { return false; }
+
+    // Find every `$NNN+` in the text.
+    let bytes = lower.as_bytes();
+    let mut dollar_positions: Vec<usize> = vec![];
+    for (i, &b) in bytes.iter().enumerate() {
+        if b == b'$' && i + 1 < bytes.len() && bytes[i + 1].is_ascii_digit() {
+            dollar_positions.push(i);
+        }
+    }
+    if dollar_positions.is_empty() { return false; }
+
+    // Any (keyword, $) pair within 40 chars triggers the policy rule.
+    for &kp in &keyword_positions {
+        for &dp in &dollar_positions {
+            if kp.abs_diff(dp) <= 40 {
+                return true;
+            }
+        }
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::{InMemoryWorkerLookup, WorkerRecord};
+    use serde_json::json;
+
+    fn lookup(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn worker(id: &str, name: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: id.into(),
+            name: name.into(),
+            status: "active".into(),
+            city: None, state: None, role: None,
+            blacklisted_clients: vec![],
+        }
+    }
 
     #[test]
     fn long_sms_fails_completeness() {
+        let v = EmailValidator::new(lookup(vec![]));
         let body = "x".repeat(200);
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
-            "to": "+15555550123",
-            "body": body,
-            "kind": "sms"
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "+15555550123", "body": body, "kind": "sms"
         })));
         assert!(matches!(r, Err(ValidationError::Completeness { .. })));
     }
 
     #[test]
     fn long_email_subject_fails_completeness() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
-            "to": "a@b.com",
-            "body": "hi",
-            "subject": "x".repeat(100)
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "a@b.com", "body": "hi", "subject": "x".repeat(100)
         })));
         assert!(matches!(r, Err(ValidationError::Completeness { .. })));
     }
 
     #[test]
     fn missing_to_fails_schema() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({"body": "hi"})));
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({"body": "hi"})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field == "to"));
     }
 
     #[test]
     fn well_formed_email_passes() {
-        let r = EmailValidator.validate(&Artifact::EmailDraft(serde_json::json!({
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
             "to": "hiring@example.com",
             "subject": "Interview: Friday 10am",
             "body": "Hi Jane — confirming interview Friday 10am."
         })));
         assert!(r.is_ok(), "well-formed email should pass: {:?}", r);
     }
+
+    #[test]
+    fn ssn_in_body_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane — your file shows 123-45-6789 on record."
+        })));
+        match r {
+            Err(ValidationError::Policy { reason }) => assert!(reason.contains("SSN")),
+            other => panic!("expected Policy SSN error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn ssn_in_subject_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "subject": "Re: ID 123-45-6789",
+            "body": "details inside"
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn phone_number_does_not_trigger_ssn_false_positive() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Call me at 555-123-4567 to confirm."
+        })));
+        assert!(r.is_ok(), "phone NNN-NNN-NNNN should NOT match SSN NNN-NN-NNNN: {:?}", r);
+    }
+
+    #[test]
+    fn salary_disclosure_fails_policy() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Confirming your hourly rate of $32.50 per hour."
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn discussing_dollars_without_salary_keyword_passes() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "The $20 parking pass is at the front desk."
+        })));
+        assert!(r.is_ok(), "non-salary $ should pass: {:?}", r);
+    }
+
+    #[test]
+    fn unknown_candidate_id_fails_consistency() {
+        let v = EmailValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane",
+            "_context": {"candidate_id": "W-FAKE"}
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("not found")),
+            other => panic!("expected Consistency, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn missing_first_name_in_body_is_warning() {
+        let v = EmailValidator::new(lookup(vec![worker("W-1", "Jane Doe")]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi there — confirming your interview Friday.",
+            "_context": {"candidate_id": "W-1"}
+        })));
+        let report = r.expect("missing name should be warning, not error");
+        assert_eq!(report.findings.len(), 1);
+        assert_eq!(report.findings[0].severity, crate::Severity::Warning);
+        assert!(report.findings[0].message.to_lowercase().contains("first name"));
+    }
+
+    #[test]
+    fn matching_first_name_passes_clean() {
+        let v = EmailValidator::new(lookup(vec![worker("W-1", "Jane Doe")]));
+        let r = v.validate(&Artifact::EmailDraft(json!({
+            "to": "x@y.com",
+            "body": "Hi Jane — confirming your interview Friday.",
+            "_context": {"candidate_id": "W-1"}
+        })));
+        let report = r.expect("matching name should pass");
+        assert!(report.findings.is_empty(), "expected no findings, got {:?}", report.findings);
+    }
 }

crates/validator/src/staffing/fill.rs

@@ -1,22 +1,67 @@
-//! Fill-proposal validator.
+//! Fill-proposal validator (Phase 43 v2 — real consistency checks).
 //!
 //! PRD checks:
-//!   - Schema compliance (propose_done shape matches
-//!     `{fills: [{candidate_id, name}]}`)
+//!   - Schema compliance (propose_done shape: `{fills: [{candidate_id, name}]}`)
 //!   - Completeness (endorsed count == target_count)
-//!   - Worker existence (every candidate_id present in workers_500k)
-//!   - Status check (active, not_on_client_blacklist)
+//!   - Worker existence (every candidate_id present in workers roster)
+//!   - Status check (worker.status == "active")
+//!   - Client blacklist (worker NOT in client.blacklisted_clients)
 //!   - Geo/role match (worker city/state/role matches contract)
 //!
-//! Today this is a scaffold — schema check is real (it's cheap); the
-//! worker-existence / status / geo checks need a catalog lookup and
-//! land in a follow-up when the catalog query helper is wired into
-//! this crate.
+//! The contract metadata (target_count, city, state, role, client_id)
+//! travels alongside the JSON payload under a `_context` key:
+//! `{"_context": {"target_count": 2, "city": "Toledo", "state": "OH",
+//!   "role": "Welder", "client_id": "CLI-00099"}, "fills": [...]}`.
+//! This keeps the Validator trait signature stable while letting the
+//! validator cross-check fills against contract truth.
+//!
+//! Worker-existence + status + geo + blacklist all share a single
+//! lookup trait (`WorkerLookup`) so the validator stays decoupled
+//! from queryd / parquet / catalogd transport details.
 
-use crate::{Artifact, Report, Validator, ValidationError};
+use crate::{
+    Artifact, Report, Validator, ValidationError, WorkerLookup, WorkerRecord,
+};
+use std::sync::Arc;
 use std::time::Instant;
 
-pub struct FillValidator;
+pub struct FillValidator {
+    workers: Arc<dyn WorkerLookup>,
+}
+
+impl FillValidator {
+    pub fn new(workers: Arc<dyn WorkerLookup>) -> Self {
+        Self { workers }
+    }
+}
+
+#[derive(Debug, Default)]
+struct FillContext {
+    target_count: Option<usize>,
+    city: Option<String>,
+    state: Option<String>,
+    role: Option<String>,
+    client_id: Option<String>,
+}
+
+fn extract_context(value: &serde_json::Value) -> FillContext {
+    let ctx_obj = value.get("_context").and_then(|c| c.as_object());
+    let ctx = match ctx_obj {
+        Some(o) => o,
+        None => return FillContext::default(),
+    };
+    FillContext {
+        target_count: ctx.get("target_count").and_then(|v| v.as_u64()).map(|n| n as usize),
+        city: ctx.get("city").and_then(|v| v.as_str()).map(String::from),
+        state: ctx.get("state").and_then(|v| v.as_str()).map(String::from),
+        role: ctx.get("role").and_then(|v| v.as_str()).map(String::from),
+        client_id: ctx.get("client_id").and_then(|v| v.as_str()).map(String::from),
+    }
+}
+
+fn eq_ci(a: &str, b: &str) -> bool {
+    a.trim().eq_ignore_ascii_case(b.trim())
+}
 
 impl Validator for FillValidator {
     fn name(&self) -> &'static str { "staffing.fill" }
@@ -31,9 +76,7 @@ impl Validator for FillValidator {
             }),
         };
 
-        // Schema check — the only real validation shipped in this
-        // scaffold. Catches the common "model emitted prose instead of
-        // JSON" failure mode before the consistency checks even run.
+        // ── Schema check ──
         let fills = value.get("fills").and_then(|f| f.as_array()).ok_or(
             ValidationError::Schema {
                 field: "fills".into(),
@@ -55,12 +98,116 @@ impl Validator for FillValidator {
             }
         }
 
-        // TODO(phase-43 v2): worker-existence / status / geo checks.
-        // Need a catalog query handle injected into FillValidator's
-        // constructor — out of scope for the scaffold.
+        let ctx = extract_context(value);
+
+        // ── Completeness: count match ──
+        if let Some(target) = ctx.target_count {
+            if fills.len() != target {
+                return Err(ValidationError::Completeness {
+                    reason: format!(
+                        "endorsed count {} != target_count {target}",
+                        fills.len()
+                    ),
+                });
+            }
+        }
+
+        // ── Cross-roster checks ──
+        let mut findings: Vec<crate::Finding> = vec![];
+        let mut seen_ids = std::collections::HashSet::new();
+        for (i, fill) in fills.iter().enumerate() {
+            let candidate_id = fill.get("candidate_id").and_then(|v| v.as_str()).unwrap_or("");
+            let proposed_name = fill.get("name").and_then(|v| v.as_str()).unwrap_or("");
+
+            // Duplicate-ID guard inside one fill.
+            if !seen_ids.insert(candidate_id.to_string()) {
+                return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "duplicate candidate_id {candidate_id:?} appears multiple times in fills"
+                    ),
+                });
+            }
+
+            // Worker existence — the gate that catches phantom IDs the
+            // model fabricates. This is the load-bearing check for
+            // the 0→85% pattern.
+            let worker: WorkerRecord = match self.workers.find(candidate_id) {
+                Some(w) => w,
+                None => return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "fills[{i}].candidate_id {candidate_id:?} does not exist in worker roster"
+                    ),
+                }),
+            };
+
+            // Status — only "active" workers can be endorsed.
+            if !eq_ci(&worker.status, "active") {
+                return Err(ValidationError::Consistency {
+                    reason: format!(
+                        "fills[{i}] worker {candidate_id:?} has status {:?}, expected \"active\"",
+                        worker.status
+                    ),
+                });
+            }
+
+            // Client blacklist.
+            if let Some(client) = ctx.client_id.as_deref() {
+                if worker.blacklisted_clients.iter().any(|b| eq_ci(b, client)) {
+                    return Err(ValidationError::Policy {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} blacklisted for client {client:?}"
+                        ),
+                    });
+                }
+            }
+
+            // Geo / role match — warn-level when missing context, hard
+            // fail on mismatch with explicit contract values.
+            if let (Some(want_city), Some(have_city)) = (ctx.city.as_deref(), worker.city.as_deref()) {
+                if !eq_ci(want_city, have_city) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} city {have_city:?} doesn't match contract city {want_city:?}"
+                        ),
+                    });
+                }
+            }
+            if let (Some(want_state), Some(have_state)) = (ctx.state.as_deref(), worker.state.as_deref()) {
+                if !eq_ci(want_state, have_state) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} state {have_state:?} doesn't match contract state {want_state:?}"
+                        ),
+                    });
+                }
+            }
+            if let (Some(want_role), Some(have_role)) = (ctx.role.as_deref(), worker.role.as_deref()) {
+                if !eq_ci(want_role, have_role) {
+                    return Err(ValidationError::Consistency {
+                        reason: format!(
+                            "fills[{i}] worker {candidate_id:?} role {have_role:?} doesn't match contract role {want_role:?}"
+                        ),
+                    });
+                }
+            }
+
+            // Name-mismatch is a warning, not an error — recruiters
+            // sometimes send updated names through the proposal layer
+            // before the roster is updated.
+            if !proposed_name.is_empty() && !eq_ci(proposed_name, &worker.name) {
+                findings.push(crate::Finding {
+                    field: format!("fills[{i}].name"),
+                    severity: crate::Severity::Warning,
+                    message: format!(
+                        "proposed name {proposed_name:?} differs from roster name {:?} for {candidate_id:?}",
+                        worker.name
+                    ),
+                });
+            }
+        }
 
         Ok(Report {
-            findings: vec![],
+            findings,
             elapsed_ms: started.elapsed().as_millis() as u64,
         })
     }
@@ -69,35 +216,168 @@ impl Validator for FillValidator {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::InMemoryWorkerLookup;
+    use serde_json::json;
+
+    fn lookup(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
+        Arc::new(InMemoryWorkerLookup::from_records(records))
+    }
+
+    fn worker(id: &str, name: &str, status: &str, city: &str, state: &str, role: &str) -> WorkerRecord {
+        WorkerRecord {
+            candidate_id: id.into(),
+            name: name.into(),
+            status: status.into(),
+            city: Some(city.into()),
+            state: Some(state.into()),
+            role: Some(role.into()),
+            blacklisted_clients: vec![],
+        }
+    }
 
     #[test]
     fn wrong_artifact_type_fails_schema() {
-        let r = FillValidator.validate(&Artifact::EmailDraft(serde_json::json!({})));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::EmailDraft(json!({})));
         assert!(matches!(r, Err(ValidationError::Schema { .. })));
     }
 
     #[test]
     fn missing_fills_array_fails_schema() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({})));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::FillProposal(json!({})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field == "fills"));
     }
 
     #[test]
     fn fill_without_candidate_id_fails() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({
-            "fills": [{"name": "Jane"}]
-        })));
+        let v = FillValidator::new(lookup(vec![]));
+        let r = v.validate(&Artifact::FillProposal(json!({"fills": [{"name": "Jane"}]})));
         assert!(matches!(r, Err(ValidationError::Schema { field, .. }) if field.contains("candidate_id")));
     }
 
     #[test]
-    fn well_formed_proposal_passes_schema() {
-        let r = FillValidator.validate(&Artifact::FillProposal(serde_json::json!({
+    fn well_formed_proposal_with_real_workers_passes() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane Doe", "active", "Toledo", "OH", "Welder"),
+            worker("W-2", "John Smith", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
             "fills": [
-                {"candidate_id": "W-123", "name": "Jane Doe"},
-                {"candidate_id": "W-456", "name": "John Smith"}
+                {"candidate_id": "W-1", "name": "Jane Doe"},
+                {"candidate_id": "W-2", "name": "John Smith"}
             ]
         })));
-        assert!(r.is_ok(), "well-formed proposal should pass schema: {:?}", r);
+        assert!(r.is_ok(), "expected pass, got {:?}", r);
+    }
+
+    #[test]
+    fn phantom_candidate_id_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Toledo", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-FAKE-99999", "name": "Imaginary"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("does not exist")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn inactive_worker_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "inactive", "Toledo", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("inactive")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn wrong_city_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Cincinnati", "OH", "Welder")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.to_lowercase().contains("city")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn wrong_role_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![worker("W-1", "Jane", "active", "Toledo", "OH", "Driver")]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.to_lowercase().contains("role")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn count_mismatch_fails_completeness() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        assert!(matches!(r, Err(ValidationError::Completeness { .. })));
+    }
+
+    #[test]
+    fn duplicate_candidate_id_fails_consistency() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 2, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [
+                {"candidate_id": "W-1", "name": "Jane"},
+                {"candidate_id": "W-1", "name": "Jane"}
+            ]
+        })));
+        match r {
+            Err(ValidationError::Consistency { reason }) => assert!(reason.contains("duplicate")),
+            other => panic!("expected Consistency error, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn blacklisted_worker_fails_policy() {
+        let mut w = worker("W-1", "Jane", "active", "Toledo", "OH", "Welder");
+        w.blacklisted_clients = vec!["CLI-00099".into()];
+        let v = FillValidator::new(lookup(vec![w]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder", "client_id": "CLI-00099"},
+            "fills": [{"candidate_id": "W-1", "name": "Jane"}]
+        })));
+        assert!(matches!(r, Err(ValidationError::Policy { .. })));
+    }
+
+    #[test]
+    fn name_mismatch_is_warning_not_error() {
+        let v = FillValidator::new(lookup(vec![
+            worker("W-1", "Jane Doe", "active", "Toledo", "OH", "Welder"),
+        ]));
+        let r = v.validate(&Artifact::FillProposal(json!({
+            "_context": {"target_count": 1, "city": "Toledo", "state": "OH", "role": "Welder"},
+            "fills": [{"candidate_id": "W-1", "name": "Janet Doe"}]
+        })));
+        let report = r.expect("name mismatch should be warning, not error");
+        assert_eq!(report.findings.len(), 1);
+        assert_eq!(report.findings[0].severity, crate::Severity::Warning);
+        assert!(report.findings[0].message.contains("differs from roster"));
     }
 }

crates/validator/src/staffing/mod.rs

@@ -6,3 +6,4 @@
 pub mod fill;
 pub mod email;
 pub mod playbook;
+pub mod parquet_lookup;

crates/validator/src/staffing/parquet_lookup.rs

@@ -0,0 +1,165 @@
+//! Production WorkerLookup backed by a workers_500k.parquet snapshot.
+//!
+//! Loads the full roster into memory at startup (one-shot). 500K rows
+//! at ~150 bytes per WorkerRecord ≈ 75 MB resident — fine for any
+//! production lakehouse process. Refresh is intentionally
+//! caller-driven (call `from_parquet` again to rebuild) rather than
+//! automatic — operators decide when staffing data has changed enough
+//! to justify the few-second reload.
+//!
+//! Schema mapping (workers_500k.parquet → WorkerRecord):
+//!   worker_id (int64)     → candidate_id = "W-{id}"
+//!   name (string)         → name
+//!   role (string)         → role
+//!   city (string)         → city
+//!   state (string)        → state
+//!   availability (double) → status: "active" if >0 else "inactive"
+//!
+//! No status column on workers_500k, so we derive from availability —
+//! the floor convention used elsewhere in the lakehouse staffing
+//! pipeline. Workers with availability=0.0 are treated as inactive
+//! (vacation, suspended, etc.). Once the Track-A.B `_safe` view ships
+//! with proper `status`, switch this loader to read it directly.
+//!
+//! Blacklist join is not done here — caller is expected to populate
+//! `blacklisted_clients` from a separate source (Phase 43 PRD says
+//! `client_blacklist` table; not yet defined). Default empty.
+
+use crate::{InMemoryWorkerLookup, WorkerLookup, WorkerRecord};
+use parquet::file::reader::{FileReader, SerializedFileReader};
+use parquet::record::Field;
+use std::fs::File;
+use std::path::Path;
+use std::sync::Arc;
+
+#[derive(Debug, thiserror::Error)]
+pub enum LookupLoadError {
+    #[error("opening parquet at {path}: {source}")]
+    Open { path: String, #[source] source: std::io::Error },
+    #[error("parsing parquet at {path}: {source}")]
+    Parse { path: String, #[source] source: parquet::errors::ParquetError },
+    #[error("missing required column {column}")]
+    MissingColumn { column: String },
+    #[error("row {row}: {reason}")]
+    BadRow { row: usize, reason: String },
+}
+
+/// Build an `InMemoryWorkerLookup` from a workers_500k-shaped parquet
+/// file. Returned as `Arc<dyn WorkerLookup>` to drop into validator
+/// constructors.
+pub fn load_workers_parquet(path: &Path) -> Result<Arc<dyn WorkerLookup>, LookupLoadError> {
+    let file = File::open(path).map_err(|e| LookupLoadError::Open {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+    let reader = SerializedFileReader::new(file).map_err(|e| LookupLoadError::Parse {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+
+    // Validate schema covers what we need before iterating rows.
+    let schema = reader.metadata().file_metadata().schema();
+    let column_names: Vec<&str> = schema.get_fields().iter().map(|f| f.name()).collect();
+    for required in &["worker_id", "name", "role", "city", "state", "availability"] {
+        if !column_names.contains(required) {
+            return Err(LookupLoadError::MissingColumn { column: (*required).to_string() });
+        }
+    }
+
+    let row_iter = reader.get_row_iter(None).map_err(|e| LookupLoadError::Parse {
+        path: path.display().to_string(),
+        source: e,
+    })?;
+
+    let mut records: Vec<WorkerRecord> = Vec::with_capacity(reader.metadata().file_metadata().num_rows() as usize);
+    let mut row_idx = 0usize;
+    for row_result in row_iter {
+        let row = row_result.map_err(|e| LookupLoadError::Parse {
+            path: path.display().to_string(),
+            source: e,
+        })?;
+        let mut worker_id: Option<i64> = None;
+        let mut name: Option<String> = None;
+        let mut role: Option<String> = None;
+        let mut city: Option<String> = None;
+        let mut state: Option<String> = None;
+        let mut availability: f64 = 0.0;
+        for (col_name, field) in row.get_column_iter() {
+            match (col_name.as_str(), field) {
+                ("worker_id", Field::Long(v)) => worker_id = Some(*v),
+                ("worker_id", Field::Int(v)) => worker_id = Some(*v as i64),
+                ("name", Field::Str(v)) => name = Some(v.clone()),
+                ("role", Field::Str(v)) => role = Some(v.clone()),
+                ("city", Field::Str(v)) => city = Some(v.clone()),
+                ("state", Field::Str(v)) => state = Some(v.clone()),
+                ("availability", Field::Double(v)) => availability = *v,
+                ("availability", Field::Float(v)) => availability = *v as f64,
+                _ => { /* extra columns ignored */ }
+            }
+        }
+        let id = worker_id.ok_or_else(|| LookupLoadError::BadRow {
+            row: row_idx,
+            reason: "worker_id missing or non-integer".into(),
+        })?;
+        let nm = name.ok_or_else(|| LookupLoadError::BadRow {
+            row: row_idx,
+            reason: "name missing".into(),
+        })?;
+        records.push(WorkerRecord {
+            candidate_id: format!("W-{id}"),
+            name: nm,
+            // status derived from availability (workers_500k has no
+            // status column). 0.0 → inactive, >0.0 → active.
+            status: if availability > 0.0 { "active".into() } else { "inactive".into() },
+            city,
+            state,
+            role,
+            blacklisted_clients: vec![],
+        });
+        row_idx += 1;
+    }
+
+    tracing::info!(
+        target: "validator.parquet_lookup",
+        rows = records.len(),
+        path = %path.display(),
+        "loaded workers parquet snapshot"
+    );
+
+    Ok(Arc::new(InMemoryWorkerLookup::from_records(records)))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::PathBuf;
+
+    /// Smoke test against the live workers_500k.parquet on disk.
+    /// Skipped automatically if the file isn't present (CI / sparse
+    /// checkouts) so the test suite stays portable.
+    #[test]
+    fn load_real_workers_500k() {
+        let path = PathBuf::from("/home/profit/lakehouse/data/datasets/workers_500k.parquet");
+        if !path.exists() {
+            eprintln!("skip: {} not present", path.display());
+            return;
+        }
+        let lookup = load_workers_parquet(&path).expect("load");
+        // Basic shape: at least one worker resolves and has the
+        // expected fields populated.
+        let probe = lookup.find("W-1");
+        assert!(probe.is_some(), "W-1 should exist in 500K-row parquet");
+        let w = probe.unwrap();
+        assert!(!w.name.is_empty(), "name should be populated");
+        assert!(w.status == "active" || w.status == "inactive");
+        assert!(w.role.is_some());
+        assert!(w.city.is_some());
+        assert!(w.state.is_some());
+    }
+
+    #[test]
+    fn missing_file_returns_error() {
+        let r = load_workers_parquet(Path::new("/nonexistent.parquet"));
+        assert!(matches!(r, Err(LookupLoadError::Open { .. })));
+    }
+}

crates/vectord-lance/src/lib.rs

@@ -603,3 +603,210 @@ fn row_from_batch(batch: &RecordBatch, row: usize) -> Result<Row, String> {
 
     Ok(Row { doc_id, chunk_text, vector: v, source, chunk_idx })
 }
+
+// =================== Tests ===================
+//
+// All tests run against a temp directory — never the production
+// data/lance/ tree. Lance reads/writes are async + filesystem-bound,
+// so we use #[tokio::test]. Each test uses a unique per-pid + per-
+// nanosecond temp dir so concurrent runs don't collide and a re-run
+// of a single test doesn't see prior state.
+//
+// Surfaced 2026-05-02 audit: vectord-lance had ZERO tests despite
+// being on the live HTTP path. These are the load-bearing locks for
+// the public API contract.
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn temp_path(label: &str) -> String {
+        // Per-process atomic counter — guarantees uniqueness regardless
+        // of clock resolution or test scheduling. Combined with pid, the
+        // result is unique within and across processes for any practical
+        // test workload. Nanosecond timestamps were not enough on their
+        // own: opus WARN at lib.rs:622 from the 2026-05-02 scrum noted
+        // that under tokio scheduling, multiple tests in the same cargo
+        // process can hit the same nanos bucket.
+        use std::sync::atomic::{AtomicU64, Ordering};
+        static COUNTER: AtomicU64 = AtomicU64::new(0);
+        let seq = COUNTER.fetch_add(1, Ordering::Relaxed);
+        let pid = std::process::id();
+        let nanos = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .map(|d| d.subsec_nanos())
+            .unwrap_or(0);
+        std::env::temp_dir()
+            .join(format!("vlance_test_{label}_{pid}_{nanos}_{seq}"))
+            .to_string_lossy()
+            .to_string()
+    }
+
+    /// Build a minimal in-memory Parquet file matching vectord's
+    /// binary-blob schema. Used as input to migrate_from_parquet_bytes.
+    fn synth_parquet_bytes(n_rows: usize, dims: usize) -> Vec<u8> {
+        use parquet::arrow::ArrowWriter;
+        use std::io::Cursor;
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("source", DataType::Utf8, true),
+            Field::new("doc_id", DataType::Utf8, false),
+            Field::new("chunk_idx", DataType::Int32, true),
+            Field::new("chunk_text", DataType::Utf8, true),
+            Field::new("vector", DataType::Binary, false),
+        ]));
+
+        let sources: Vec<Option<&str>> = (0..n_rows).map(|_| Some("test")).collect();
+        let doc_ids: Vec<String> = (0..n_rows).map(|i| format!("DOC-{i:04}")).collect();
+        let chunk_idxs: Vec<Option<i32>> = (0..n_rows).map(|i| Some(i as i32)).collect();
+        let chunk_texts: Vec<String> = (0..n_rows).map(|i| format!("synth chunk {i}")).collect();
+        let vectors: Vec<Vec<u8>> = (0..n_rows).map(|i| {
+            let v: Vec<f32> = (0..dims).map(|j| (i * dims + j) as f32 * 0.01).collect();
+            let mut bytes = Vec::with_capacity(dims * 4);
+            for f in v { bytes.extend_from_slice(&f.to_le_bytes()); }
+            bytes
+        }).collect();
+
+        let batch = RecordBatch::try_new(schema.clone(), vec![
+            Arc::new(StringArray::from(sources)),
+            Arc::new(StringArray::from(doc_ids)),
+            Arc::new(Int32Array::from(chunk_idxs)),
+            Arc::new(StringArray::from(chunk_texts)),
+            Arc::new(BinaryArray::from(vectors.iter().map(|v| v.as_slice()).collect::<Vec<_>>())),
+        ]).expect("synth parquet batch");
+
+        let mut buf = Cursor::new(Vec::new());
+        let mut writer = ArrowWriter::try_new(&mut buf, schema, None).expect("arrow writer");
+        writer.write(&batch).expect("write batch");
+        writer.close().expect("close writer");
+        buf.into_inner()
+    }
+
+    #[tokio::test]
+    async fn fresh_store_reports_no_state() {
+        let path = temp_path("fresh");
+        let store = LanceVectorStore::new(path.clone());
+        assert_eq!(store.path(), path);
+        assert_eq!(store.count().await.unwrap_or(0), 0);
+        assert!(!store.has_vector_index().await.unwrap_or(true));
+    }
+
+    #[tokio::test]
+    async fn migrate_then_count_and_fetch() {
+        let path = temp_path("migrate_fetch");
+        let store = LanceVectorStore::new(path.clone());
+        let bytes = synth_parquet_bytes(8, 4);
+
+        let stats = store.migrate_from_parquet_bytes(&bytes).await.expect("migrate");
+        assert_eq!(stats.rows_written, 8);
+        assert_eq!(stats.dimensions, 4);
+        assert!(stats.disk_bytes > 0, "lance dataset should occupy disk");
+
+        assert_eq!(store.count().await.unwrap(), 8);
+
+        let row = store.get_by_doc_id("DOC-0003").await
+            .expect("get_by_doc_id Ok").expect("DOC-0003 exists");
+        assert_eq!(row.doc_id, "DOC-0003");
+        assert_eq!(row.chunk_text, "synth chunk 3");
+        assert_eq!(row.vector.len(), 4);
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Load-bearing contract: get_by_doc_id distinguishes "dataset
+    /// missing" (Err) from "id missing" (Ok(None)) so the HTTP
+    /// handler can return 404 without inspecting error strings.
+    #[tokio::test]
+    async fn get_by_doc_id_missing_returns_none() {
+        let path = temp_path("missing_id");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let row = store.get_by_doc_id("DOC-NEVER-EXISTS").await.expect("Ok");
+        assert!(row.is_none(), "missing id must return Ok(None), not Err");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Verifies the load-bearing structural-difference claim of
+    /// ADR-019: Lance appends without rewriting the whole file. Row
+    /// count grows; new rows are fetchable by their doc_ids.
+    #[tokio::test]
+    async fn append_grows_count_and_new_rows_fetchable() {
+        let path = temp_path("append");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+        assert_eq!(store.count().await.unwrap(), 4);
+
+        let stats = store.append(
+            Some("appended".into()),
+            vec!["NEW-A".into(), "NEW-B".into()],
+            vec![0, 0],
+            vec!["new chunk a".into(), "new chunk b".into()],
+            vec![vec![0.1, 0.2, 0.3, 0.4], vec![0.5, 0.6, 0.7, 0.8]],
+        ).await.expect("append");
+
+        assert_eq!(stats.rows_appended, 2);
+        assert_eq!(store.count().await.unwrap(), 6);
+
+        let new_a = store.get_by_doc_id("NEW-A").await.unwrap().expect("NEW-A");
+        assert_eq!(new_a.chunk_text, "new chunk a");
+        assert_eq!(new_a.source.as_deref(), Some("appended"));
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Without this guard a dim-mismatch row would land on disk and
+    /// silently break search at query time.
+    #[tokio::test]
+    async fn append_dim_mismatch_errors() {
+        let path = temp_path("dim_mismatch");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let err = store.append(
+            None, vec!["X".into(), "Y".into()], vec![0, 0],
+            vec!["a".into(), "b".into()],
+            vec![vec![1.0, 2.0, 3.0, 4.0], vec![1.0, 2.0]],
+        ).await;
+        assert!(err.is_err(), "dim mismatch must error");
+        let msg = err.unwrap_err();
+        assert!(msg.contains("dim") || msg.contains("expected"),
+            "error must mention the dimension problem; got: {msg}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Search round-trip: query the exact vector for one row, top-1
+    /// must be that row. Verifies the search path works on small
+    /// datasets where IVF training would normally be skipped.
+    #[tokio::test]
+    async fn search_returns_nearest() {
+        let path = temp_path("search");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(8, 4)).await.expect("migrate");
+
+        let target: Vec<f32> = (0..4).map(|j| (5 * 4 + j) as f32 * 0.01).collect();
+        let hits = store.search(&target, 3, None, None).await.expect("search");
+        assert!(!hits.is_empty(), "search must return at least 1 hit");
+        assert_eq!(hits[0].doc_id, "DOC-0005",
+            "exact-vector match should be top-1; got {hits:?}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// stats() summarizes the dataset state in one call. Locks the
+    /// field shape so downstream consumers don't break on a rename.
+    #[tokio::test]
+    async fn stats_reports_post_migrate_state() {
+        let path = temp_path("stats");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(5, 4)).await.expect("migrate");
+
+        let s = store.stats().await.expect("stats");
+        assert_eq!(s.rows, 5);
+        assert!(s.disk_bytes > 0);
+        assert!(!s.has_vector_index, "no vector index built yet");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+}

crates/vectord/src/pathway_memory.rs

@@ -925,7 +925,7 @@ mod tests {
             reject_reason: None,
         }];
         let mut trace = PathwayTrace {
-            pathway_id,
+            pathway_id: pathway_id.clone(),
             task_class: "scrum_review".into(),
             file_path: format!("crates/{id_tag}/src/x.rs"),
             signal_class: Some("CONVERGING".into()),
@@ -954,6 +954,14 @@ mod tests {
             replay_count: replays,
             replays_succeeded: succ,
             retired: false,
+            // Versioning fields added by Mem0 wave (commit 6ac7f61) — defaults
+            // mirror "this trace is the live head with no parent/successor".
+            trace_uid: format!("test-{pathway_id}"),
+            version: 1,
+            parent_trace_uid: None,
+            superseded_at: None,
+            superseded_by_trace_uid: None,
+            retirement_reason: None,
         };
         trace.pathway_vec = build_pathway_vec(&trace);
         trace

crates/vectord/src/rag.rs

@@ -163,7 +163,11 @@ pub async fn query(
     // production caller of the Phase 21 primitives — see audit finding
     // "Phase 21 Rust primitives are wired but not CALLED by any
     // production surface" from 2026-04-21.
-    let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
+    // 2026-04-30 model bump: qwen2.5:latest → qwen3.5:latest to match
+    // the small-model-pipeline local-tier default. Same JSON-clean
+    // property, more capacity. think=Some(false) preserved — RAG hot
+    // path doesn't need reasoning traces; direct answers only.
+    let mut cont_opts = ContinuableOpts::new("qwen3.5:latest");
     cont_opts.max_tokens = Some(512);
     cont_opts.temperature = Some(0.2);
     cont_opts.shape = ResponseShape::Text;
@@ -176,7 +180,7 @@ pub async fn query(
         // echoes whatever Ollama loaded). Use the configured tier model
         // for now; if RAG needs to report the actual resolved model,
         // the runner can add a post-call ps probe later.
-        model: "qwen2.5:latest".to_string(),
+        model: "qwen3.5:latest".to_string(),
         sources: results,
         tokens_generated: None,
     })

crates/vectord/src/service.rs

@@ -146,6 +146,11 @@ pub fn router(state: VectorState) -> Router {
         // Phase 45 slice 3 — doc drift detection + human re-admission.
         .route("/playbook_memory/doc_drift/check/{id}", post(check_doc_drift))
         .route("/playbook_memory/doc_drift/resolve/{id}", post(resolve_doc_drift))
+        // Phase 45 closure (2026-04-27) — batch scan across all active
+        // playbooks. Operator runs this on a schedule (cron or manual);
+        // each newly-detected drift writes a row to
+        // data/_kb/doc_drift_corrections.jsonl for downstream review.
+        .route("/playbook_memory/doc_drift/scan", post(scan_doc_drift))
         // Pathway memory — consensus-designed sidecar (2026-04-24).
         // scrum_master_pipeline POSTs /pathway/insert at the end of each
         // review, calls /pathway/query before running the ladder for a
@@ -1850,10 +1855,10 @@ async fn lance_migrate(
         .map_err(|e| (StatusCode::NOT_FOUND, format!("read parquet: {e}")))?;
 
     let lance_store = state.lance.store_for_new(&index_name, &bucket).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let stats = lance_store.migrate_from_parquet_bytes(&bytes).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     tracing::info!(
         "lance migrate '{}': {} rows, {}d, {} bytes on disk, {:.2}s",
@@ -1861,11 +1866,40 @@ async fn lance_migrate(
         stats.disk_bytes, stats.duration_secs,
     );
 
+    // Auto-build the doc_id btree. The scalar index is what makes
+    // get_doc_by_id O(log n) instead of a full table scan; ADR-019
+    // calls this out as the load-bearing feature for hybrid lookup.
+    // Verified 2026-05-02: skipping this on a 10M-row dataset turns
+    // ~5ms doc-fetch into ~100ms (full scan over 35GB). Cheap to
+    // build (~1.2s on 10M, +269MB on disk) and only runs once per
+    // dataset since `has_scalar_index` short-circuits subsequent calls.
+    let scalar_stats = if !lance_store.has_scalar_index("doc_id").await.unwrap_or(false) {
+        match lance_store.build_scalar_index("doc_id").await {
+            Ok(s) => {
+                tracing::info!(
+                    "lance migrate '{}': doc_id btree built in {:.2}s (+{} bytes)",
+                    index_name, s.build_time_secs, s.disk_bytes_added,
+                );
+                Some(s)
+            }
+            Err(e) => {
+                // Don't fail the whole migrate over a missing btree —
+                // the dataset is still queryable, just slowly. Log it
+                // so it's debuggable.
+                tracing::warn!("lance migrate '{}': doc_id btree build failed (will fall back to scan): {e}", index_name);
+                None
+            }
+        }
+    } else {
+        None
+    };
+
     Ok::<_, (StatusCode, String)>(Json(serde_json::json!({
         "index_name": index_name,
         "bucket": bucket,
         "lance_path": lance_store.path(),
         "stats": stats,
+        "scalar_index": scalar_stats,
     })))
 }
 
@@ -1883,6 +1917,300 @@ fn default_partitions() -> u32 { 316 }   // ≈√100K — sane for the referenc
 fn default_bits() -> u32 { 8 }
 fn default_subvectors() -> u32 { 48 }    // 768/48 = 16 dims per subvector
 
+/// Sanitize a Lance backend error before returning it to the HTTP
+/// caller. Two responsibilities:
+///
+/// 1. Map "dataset not found" patterns to HTTP 404 instead of 500.
+///    A missing index isn't an internal failure — it's a resource
+///    lookup miss, and the response code should reflect that.
+/// 2. Strip server-side filesystem paths and Rust crate registry
+///    paths (`/root/.cargo/registry/src/index.crates.io-...`) from
+///    the message body. An attacker probing the surface shouldn't
+///    learn the server's directory layout or our exact dep versions.
+///
+/// Surfaced 2026-05-02 by the Lance backend audit: missing-index
+/// search returned 500 + leaked the lakehouse data path AND the
+/// .cargo/registry path with crate versions.
+fn sanitize_lance_err(err: String, index_name: &str) -> (StatusCode, String) {
+    // 404 detection — narrowed across two 2026-05-02→03 scrum waves.
+    // First wave (opus WARN service.rs:1908): the original `lower.contains
+    // ("not found")` was too broad — caught "column not found" /
+    // "field not found in schema" which are real 500s. Second wave (opus
+    // WARN service.rs:1949): the looser `mentions_path_missing` branch I
+    // added would 404 on a registry-file error like "/root/.cargo/.../x.rs:
+    // no such file or directory" because it triggers without dataset
+    // context. Drop the standalone path-missing branch; require dataset
+    // context AND a missing-shape phrase. Lance's actual error format
+    // ("Dataset at path X was not found") satisfies this.
+    let lower = err.to_lowercase();
+    let mentions_dataset = lower.contains("dataset");
+    let lance_dataset_missing = mentions_dataset && (
+        lower.contains("not found") || lower.contains("does not exist")
+    );
+    // Excluded shapes — these contain "not found" but are real 500s.
+    let column_or_field = lower.contains("column not found")
+        || lower.contains("field not found")
+        || lower.contains("schema not found");
+    let is_not_found = lance_dataset_missing && !column_or_field;
+    if is_not_found {
+        return (StatusCode::NOT_FOUND, format!("lance dataset not found: {index_name}"));
+    }
+
+    // Path redaction — replace path-shaped substrings with [REDACTED]
+    // rather than truncating, per opus BLOCK at service.rs:1914 from the
+    // 2026-05-02 scrum. The previous `err.split("/home/").next()` returned
+    // Some("") when the error string STARTED with "/home/", erasing the
+    // entire message and falling back to a generic "lance backend error"
+    // that lost all real error context. Replacing keeps the structural
+    // error (the "what failed") while stripping the location.
+    let cleaned = redact_paths(&err)
+        .trim_end_matches([',', ' ', '\n', '\t'])
+        .to_string();
+    let msg = if cleaned.is_empty() {
+        format!("lance backend error on {index_name}")
+    } else {
+        cleaned
+    };
+    (StatusCode::INTERNAL_SERVER_ERROR, msg)
+}
+
+/// Replace absolute-path substrings (under known leak-prone roots) with
+/// "[REDACTED]". Walks the input once, identifying path-shaped runs that
+/// start with one of the configured prefixes and continue until a
+/// path-terminating character (whitespace, quote, comma, paren, EOL).
+///
+/// Linear time, no regex dep. Catches multi-occurrence cases that
+/// `String::split(p).next()` lost. The path-redaction surface intentionally
+/// includes /var, /tmp, /etc, /usr, /opt in addition to /home and
+/// /root/.cargo because Lance/Arrow errors surface system paths in
+/// addition to project paths.
+fn redact_paths(s: &str) -> String {
+    // Two prefix sets:
+    // - ABSOLUTE: paths starting with '/' (always safe to redact)
+    // - RELATIVE: same path bodies but without leading '/' (Lance occasionally
+    //   strips the leading slash when echoing dataset paths back, observed
+    //   live 2026-05-02 — "Dataset at path home/profit/lakehouse/data/lance/x
+    //   was not found"). Match these only when preceded by a non-alpha char
+    //   (start of string, space, colon, etc.) so we don't redact innocent
+    //   tokens like "homecoming" or "etcetera".
+    const ABSOLUTE: &[&str] = &[
+        "/root/.cargo", "/home", "/var", "/tmp", "/etc", "/usr", "/opt",
+    ];
+    const RELATIVE: &[&str] = &[
+        "root/.cargo", "home/", "var/", "tmp/", "etc/", "usr/", "opt/",
+    ];
+    fn is_path_term(b: u8) -> bool {
+        matches!(b, b' ' | b'\t' | b'\n' | b'\r' | b'"' | b'\'' | b',' | b')' | b']' | b'}')
+    }
+    fn is_word_boundary_before(bytes: &[u8], i: usize) -> bool {
+        // True if byte at i-1 is non-alphanumeric (so this position starts
+        // a fresh token). True at start-of-input.
+        if i == 0 { return true; }
+        let b = bytes[i - 1];
+        !(b.is_ascii_alphanumeric() || b == b'_' || b == b'.' || b == b'-')
+    }
+    // Walk by byte index but slice the original &str when emitting, never
+    // cast bytes to char (that would corrupt multi-byte UTF-8 — opus WARN
+    // at service.rs:2018 from the 2026-05-03 re-scrum). Path prefixes are
+    // pure ASCII so byte-level matching is sound; what matters is that
+    // we emit non-matched stretches as &str slices, not byte-by-byte.
+    let bytes = s.as_bytes();
+    let mut out = String::with_capacity(s.len());
+    let mut i = 0;
+    let mut copy_start = 0usize;  // start of an in-progress unmatched run
+    while i < bytes.len() {
+        let mut matched_len: Option<usize> = None;
+        // Try absolute prefixes first (always allowed).
+        for p in ABSOLUTE {
+            let pb = p.as_bytes();
+            if i + pb.len() <= bytes.len() && &bytes[i..i + pb.len()] == pb {
+                let after = i + pb.len();
+                if after == bytes.len() || bytes[after] == b'/' || is_path_term(bytes[after]) {
+                    matched_len = Some(pb.len());
+                    break;
+                }
+            }
+        }
+        // Then relative prefixes — only at word boundaries.
+        if matched_len.is_none() && is_word_boundary_before(bytes, i) {
+            for p in RELATIVE {
+                let pb = p.as_bytes();
+                if i + pb.len() <= bytes.len() && &bytes[i..i + pb.len()] == pb {
+                    matched_len = Some(pb.len());
+                    break;
+                }
+            }
+        }
+        if let Some(prefix_len) = matched_len {
+            // Flush any pending unmatched run as a UTF-8-safe slice.
+            if copy_start < i {
+                out.push_str(&s[copy_start..i]);
+            }
+            out.push_str("[REDACTED]");
+            // Skip past the prefix and the path body (until terminator).
+            let mut j = i + prefix_len;
+            while j < bytes.len() && !is_path_term(bytes[j]) {
+                j += 1;
+            }
+            i = j;
+            copy_start = i;
+        } else {
+            // Advance one CHAR (not one byte) so multi-byte UTF-8 sequences
+            // stay intact in the eventual slice. Look up the next char
+            // boundary using the public API.
+            i += utf8_char_len(bytes, i);
+        }
+    }
+    if copy_start < bytes.len() {
+        out.push_str(&s[copy_start..]);
+    }
+    out
+}
+
+/// Length in bytes of the UTF-8 character starting at byte `i`. Bytes are
+/// guaranteed to be a valid UTF-8 sequence start (callers ensure that).
+fn utf8_char_len(bytes: &[u8], i: usize) -> usize {
+    let b = bytes[i];
+    if b < 0x80 { 1 }
+    else if b < 0xC0 { 1 }  // continuation byte — defensive, shouldn't start here
+    else if b < 0xE0 { 2 }
+    else if b < 0xF0 { 3 }
+    else { 4 }
+}
+
+#[cfg(test)]
+mod sanitize_tests {
+    use super::*;
+
+    #[test]
+    fn redact_path_at_offset_zero() {
+        // Regression: opus BLOCK 2026-05-02. Old impl returned Some("")
+        // when err started with "/home/", erasing the whole message.
+        let out = redact_paths("/home/profit/lakehouse/data/lance not a directory");
+        assert_eq!(out, "[REDACTED] not a directory");
+    }
+
+    #[test]
+    fn redact_keeps_pre_and_post_text() {
+        let out = redact_paths("failed to open /home/profit/lakehouse/data/x for read: ENOENT");
+        assert_eq!(out, "failed to open [REDACTED] for read: ENOENT");
+    }
+
+    #[test]
+    fn redact_multiple_paths() {
+        let out = redact_paths("at /root/.cargo/registry/src/index.crates.io-foo/lance-table-4.0.0/src/io/commit.rs:364:26 from /home/profit/lakehouse");
+        assert!(!out.contains("/root/.cargo"));
+        assert!(!out.contains("/home/"));
+        assert!(out.contains("[REDACTED]"));
+    }
+
+    #[test]
+    fn redact_preserves_quote_terminator() {
+        let out = redact_paths("{\"path\":\"/home/profit/x\",\"err\":\"bad\"}");
+        assert_eq!(out, "{\"path\":\"[REDACTED]\",\"err\":\"bad\"}");
+    }
+
+    #[test]
+    fn is_not_found_narrow_dataset_only() {
+        // Regression: opus WARN 2026-05-02. Old impl 404'd on any "not
+        // found" — including legitimate column/field-not-found 500s.
+        let (status, _) = sanitize_lance_err(
+            "column not found: vector".into(), "test_idx",
+        );
+        assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR);
+
+        let (status, _) = sanitize_lance_err(
+            "dataset not found at /home/profit/lakehouse/data/lance/missing".into(), "test_idx",
+        );
+        assert_eq!(status, StatusCode::NOT_FOUND);
+    }
+
+    #[test]
+    fn redact_does_not_match_prefix_substring() {
+        // /etcetera should NOT trigger /etc redaction.
+        let out = redact_paths("etcetera and /etcd");
+        assert_eq!(out, "etcetera and /etcd");
+    }
+
+    #[test]
+    fn redact_relative_paths_lance_emits() {
+        // 2026-05-02: live missing-index probe surfaced Lance error of the
+        // form "Dataset at path home/profit/lakehouse/data/lance/x was not
+        // found" — leading slash stripped. Need to redact the relative form
+        // when preceded by a word boundary.
+        let out = redact_paths("Dataset at path home/profit/lakehouse/data/lance/x was not found");
+        assert!(!out.contains("home/profit"), "should redact: {out}");
+        assert!(out.contains("Dataset at path"));
+        assert!(out.contains("was not found"));
+    }
+
+    #[test]
+    fn redact_does_not_eat_innocent_prefix_words() {
+        // "homecoming" must NOT trigger "home/" redaction. "Etcetera" must
+        // NOT trigger "etc/" redaction. The word-boundary guard handles this.
+        let out = redact_paths("homecoming etcetera vary tmpfile");
+        assert_eq!(out, "homecoming etcetera vary tmpfile");
+    }
+
+    #[test]
+    fn is_not_found_lance_actual_phrasing() {
+        // Lance's actual error format observed live: "Dataset at path X was
+        // not found: Not found: ...". Must 404, not 500.
+        let (status, _) = sanitize_lance_err(
+            "Dataset at path home/profit/lakehouse/data/lance/x was not found".into(),
+            "x",
+        );
+        assert_eq!(status, StatusCode::NOT_FOUND);
+    }
+
+    #[test]
+    fn is_not_found_excludes_column_field_schema() {
+        // Real 500s with the "not found" phrase that aren't dataset-missing.
+        for err in [
+            "column not found: vector",
+            "field not found in schema: doc_id",
+            "schema not found for dataset xyz",
+        ] {
+            let (status, _) = sanitize_lance_err(err.into(), "test_idx");
+            assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR, "{err}");
+        }
+    }
+
+    #[test]
+    fn is_not_found_does_not_match_unrelated_path_missing() {
+        // Regression: opus WARN at service.rs:1949 from the 2026-05-03
+        // re-scrum. A registry-file error from inside a Lance internal
+        // module should NOT be coerced to 404 just because it contains
+        // "no such file or directory" — it's a real 500.
+        let (status, _) = sanitize_lance_err(
+            "/root/.cargo/registry/src/index.crates.io-foo/lance-table-4.0.0/src/io/commit.rs: no such file or directory".into(),
+            "test_idx",
+        );
+        assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR);
+        // (And the path is still redacted in the message.)
+        let (_, msg) = sanitize_lance_err(
+            "/root/.cargo/registry/src/lance-foo/x.rs: no such file or directory".into(),
+            "test_idx",
+        );
+        assert!(!msg.contains("/root/.cargo"), "path leak: {msg}");
+    }
+
+    #[test]
+    fn redact_preserves_multibyte_utf8() {
+        // Regression: opus WARN at service.rs:2018 from the 2026-05-03
+        // re-scrum. Old impl did `out.push(bytes[i] as char)` which
+        // corrupted multi-byte UTF-8 (e.g. a path containing user-supplied
+        // names with non-ASCII characters) into Latin-1 mojibake.
+        let input = "Failed to open /home/profit/工作/data — café not found";
+        let out = redact_paths(input);
+        // The path is redacted...
+        assert!(!out.contains("/home/profit"), "path leak: {out}");
+        // ...AND the multi-byte characters elsewhere are preserved verbatim.
+        assert!(out.contains("café"), "lost UTF-8: {out}");
+        assert!(out.contains("not found"), "lost trailing context: {out}");
+    }
+}
+
 /// Build the IVF_PQ index on the Lance dataset.
 async fn lance_build_index(
     State(state): State<VectorState>,
@@ -1890,10 +2218,10 @@ async fn lance_build_index(
     Json(req): Json<LanceIndexRequest>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     match lance_store.build_index(req.num_partitions, req.num_bits, req.num_sub_vectors).await {
         Ok(stats) => Ok(Json(stats)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -1942,13 +2270,13 @@ async fn lance_search(
     let qv: Vec<f32> = embed_resp.embeddings[0].iter().map(|&x| x as f32).collect();
 
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let t0 = std::time::Instant::now();
     let nprobes = req.nprobes.or(Some(LANCE_DEFAULT_NPROBES));
     let refine = req.refine_factor.or(Some(LANCE_DEFAULT_REFINE_FACTOR));
     let hits = lance_store.search(&qv, req.top_k, nprobes, refine).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     Ok(Json(serde_json::json!({
         "index_name": index_name,
@@ -1966,7 +2294,7 @@ async fn lance_get_doc(
     Path((index_name, doc_id)): Path<(String, String)>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     let t0 = std::time::Instant::now();
     match lance_store.get_by_doc_id(&doc_id).await {
         Ok(Some(row)) => Ok(Json(serde_json::json!({
@@ -1976,7 +2304,7 @@ async fn lance_get_doc(
             "row": row,
         }))),
         Ok(None) => Err((StatusCode::NOT_FOUND, format!("doc_id not found: {doc_id}"))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -2008,7 +2336,7 @@ async fn lance_append(
         return Err((StatusCode::BAD_REQUEST, "rows array is empty".into()));
     }
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let mut doc_ids = Vec::with_capacity(req.rows.len());
     let mut chunk_idxs = Vec::with_capacity(req.rows.len());
@@ -2539,6 +2867,119 @@ async fn check_doc_drift(
     })))
 }
 
+/// Phase 45 closure (2026-04-27) — POST /playbook_memory/doc_drift/scan
+///
+/// Iterates all active playbooks (non-retired, has doc_refs), runs
+/// drift check against context7 for each, flags drifted entries via
+/// PlaybookMemory::flag_doc_drift, and appends a row to
+/// data/_kb/doc_drift_corrections.jsonl for each drift detected.
+///
+/// Returns aggregate stats so an operator can see at-a-glance how
+/// many playbooks drifted and which tools moved.
+///
+/// Honors entries already flagged: they're counted in `already_flagged`
+/// (no double-flag, no duplicate corrections.jsonl row).
+async fn scan_doc_drift(
+    State(state): State<VectorState>,
+) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
+    use crate::doc_drift::{check_all_refs, DriftCheckerConfig, DriftOutcome};
+
+    let entries = state.playbook_memory.snapshot().await;
+    let now = chrono::Utc::now().to_rfc3339();
+    let cfg = DriftCheckerConfig::default();
+
+    let mut scanned = 0usize;
+    let mut newly_flagged = 0usize;
+    let mut already_flagged = 0usize;
+    let mut skipped_no_refs = 0usize;
+    let mut skipped_retired = 0usize;
+    let mut tool_counts: std::collections::HashMap<String, usize> = Default::default();
+    let mut corrections_rows: Vec<String> = vec![];
+
+    for e in entries.iter() {
+        if e.retired_at.is_some() { skipped_retired += 1; continue; }
+        if e.doc_refs.is_empty() { skipped_no_refs += 1; continue; }
+        if e.doc_drift_flagged_at.is_some() && e.doc_drift_reviewed_at.is_none() {
+            already_flagged += 1;
+            continue;
+        }
+        scanned += 1;
+        let results = check_all_refs(&cfg, &e.doc_refs).await;
+        let drifted_tools: Vec<&str> = results.iter()
+            .filter(|r| matches!(r.outcome, DriftOutcome::Drifted { .. }))
+            .map(|r| r.tool.as_str())
+            .collect();
+        if drifted_tools.is_empty() { continue; }
+
+        // Flag the entry.
+        let flagged = state.playbook_memory.flag_doc_drift(&e.playbook_id).await
+            .unwrap_or(false);
+        if flagged { newly_flagged += 1; }
+        for t in &drifted_tools {
+            *tool_counts.entry(t.to_string()).or_insert(0) += 1;
+        }
+
+        // Build corrections.jsonl row — one per drifted playbook with
+        // the tool list inline. Downstream consumers (overview model,
+        // operator dashboard) read this to decide reviews + revisions.
+        let row = serde_json::json!({
+            "playbook_id": e.playbook_id,
+            "scanned_at": now,
+            "drifted_tools": drifted_tools,
+            "per_tool": results.iter().map(|r| {
+                let (drifted, current, src) = match &r.outcome {
+                    DriftOutcome::Drifted { current_snippet_hash, source_url } =>
+                        (true, Some(current_snippet_hash.clone()), source_url.clone()),
+                    _ => (false, None, None),
+                };
+                serde_json::json!({
+                    "tool": r.tool, "version_seen": r.version_seen,
+                    "drifted": drifted, "current_snippet_hash": current, "source_url": src,
+                })
+            }).collect::<Vec<_>>(),
+            "recommended_action": "review-and-resolve",
+        });
+        corrections_rows.push(row.to_string());
+    }
+
+    // Persist corrections.jsonl row(s) for the operator/overview model.
+    if !corrections_rows.is_empty() {
+        let path = std::path::PathBuf::from("/home/profit/lakehouse/data/_kb/doc_drift_corrections.jsonl");
+        if let Some(parent) = path.parent() {
+            if let Err(e) = tokio::fs::create_dir_all(parent).await {
+                tracing::warn!(target: "vectord.doc_drift", "create_dir_all {parent:?}: {e}");
+            }
+        }
+        let body = corrections_rows.join("\n") + "\n";
+        if let Err(e) = tokio::fs::OpenOptions::new()
+            .create(true).append(true).open(&path).await
+        {
+            tracing::warn!(target: "vectord.doc_drift", "open {path:?}: {e}");
+        } else {
+            use tokio::io::AsyncWriteExt;
+            match tokio::fs::OpenOptions::new().create(true).append(true).open(&path).await {
+                Ok(mut f) => {
+                    if let Err(e) = f.write_all(body.as_bytes()).await {
+                        tracing::warn!(target: "vectord.doc_drift", "append {path:?}: {e}");
+                    }
+                }
+                Err(e) => tracing::warn!(target: "vectord.doc_drift", "reopen {path:?}: {e}"),
+            }
+        }
+    }
+
+    Ok(Json(serde_json::json!({
+        "scanned_at": now,
+        "scanned": scanned,
+        "newly_flagged": newly_flagged,
+        "already_flagged": already_flagged,
+        "skipped_retired": skipped_retired,
+        "skipped_no_refs": skipped_no_refs,
+        "drifted_by_tool": tool_counts,
+        "corrections_written": corrections_rows.len(),
+    })))
+}
+
 /// Phase 45 slice 3 — POST /playbook_memory/doc_drift/resolve/{id}
 ///
 /// Human-in-the-loop re-admission. Stamps `doc_drift_reviewed_at`.

data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json

@@ -11,15 +11,51 @@
     }
   ],
   "created_at": "2026-04-20T11:07:57.308050648Z",
-  "updated_at": "2026-04-22T03:28:28.343843823Z",
+  "updated_at": "2026-04-28T01:28:31.280305207Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "timestamp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "operation",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "approach",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "result",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "context",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null,
+  "row_count": 2077,
   "last_embedded_at": null,
   "embedding_stale_since": null,
   "embedding_refresh_policy": null

data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json

@@ -1,117 +0,0 @@
-{
-  "id": "564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7",
-  "name": "client_workerskjkk",
-  "schema_fingerprint": "cdfe85348885ddf329e5e6e9bf0e2c75c92d1a86fdb0fd3875ed46e3f93c4d82",
-  "objects": [
-    {
-      "bucket": "primary",
-      "key": "datasets/client_workerskjkk.parquet",
-      "size_bytes": 32201,
-      "created_at": "2026-04-21T00:49:04.623625149Z"
-    }
-  ],
-  "created_at": "2026-04-21T00:49:04.623626738Z",
-  "updated_at": "2026-04-21T00:49:04.623901788Z",
-  "description": "",
-  "owner": "",
-  "sensitivity": "pii",
-  "columns": [
-    {
-      "name": "worker_id",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "name",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "role",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "city",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "state",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "email",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "phone",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "skills",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "certifications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "availability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "reliability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "archetype",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    }
-  ],
-  "lineage": {
-    "source_system": "csv",
-    "source_file": "staffing_roster_sample.csv",
-    "ingest_job": "ingest-1776732544623",
-    "ingest_timestamp": "2026-04-21T00:49:04.623625149Z",
-    "parent_datasets": []
-  },
-  "freshness": null,
-  "tags": [],
-  "row_count": 180,
-  "last_embedded_at": null,
-  "embedding_stale_since": null,
-  "embedding_refresh_policy": null
-}

data/_catalog/views/candidates_safe.json

@@ -0,0 +1,24 @@
+{
+  "name": "candidates_safe",
+  "base_dataset": "candidates",
+  "columns": [
+    "candidate_id",
+    "first_name",
+    "city",
+    "state",
+    "skills",
+    "years_experience",
+    "status"
+  ],
+  "row_filter": "status != 'blocked'",
+  "column_redactions": {
+    "candidate_id": {
+      "kind": "mask",
+      "keep_prefix": 3,
+      "keep_suffix": 2
+    }
+  },
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "PII-free candidate projection — drops last_name, email, phone, hourly_rate_usd. candidate_id masked (keep first 3, last 2). Visible to recruiter / mode-runner agents."
+}

data/_catalog/views/jobs_safe.json

@@ -0,0 +1,26 @@
+{
+  "name": "jobs_safe",
+  "base_dataset": "job_orders",
+  "columns": [
+    "job_order_id",
+    "client_id",
+    "title",
+    "vertical",
+    "status",
+    "city",
+    "state",
+    "zip",
+    "bill_rate",
+    "pay_rate"
+  ],
+  "column_redactions": {
+    "client_id": {
+      "kind": "mask",
+      "keep_prefix": 3,
+      "keep_suffix": 2
+    }
+  },
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "Job-order projection with client_id masked. Drops description (often quotes client names verbatim, no text-scrubber available). bill_rate / pay_rate kept — commercial info, not PII per staffing PRD."
+}

data/_catalog/views/workers_safe.json

@@ -0,0 +1,22 @@
+{
+  "name": "workers_safe",
+  "base_dataset": "workers_500k",
+  "columns": [
+    "worker_id",
+    "role",
+    "city",
+    "state",
+    "skills",
+    "certifications",
+    "archetype",
+    "reliability",
+    "responsiveness",
+    "engagement",
+    "compliance",
+    "availability"
+  ],
+  "column_redactions": {},
+  "created_at": "2026-04-27T15:42:00Z",
+  "created_by": "j",
+  "description": "PII-free worker projection — drops name, email, phone, zip, communications, resume_text. resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale. Skills + certifications + scores carry the matching signal for staffing inference. Source for workers_500k_v9 vector corpus rebuild."
+}

docs/ADR-019-vector-storage.md

@@ -70,10 +70,33 @@ But the written rule missed something. It assumed Lance's value would show up as
 
 ### What we keep watching (but don't act on yet)
 
-- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.
+- ~~**Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.~~ **Done 2026-05-02** — see "Follow-up: 10M re-bench" below.
 - **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system.
 - **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot.
 
+---
+
+## Follow-up: 10M re-bench (2026-05-02)
+
+The 10M re-bench above ran. Numbers from `data/lance/scale_test_10m` (33 GB, 10M × 768d, IVF_PQ live, post-doc_id-btree-build). Full report: `reports/lance_10m_rebench_2026-05-02.md`.
+
+| Op | 100K (this ADR) | 10M (re-bench) | Notes |
+|---|---:|---:|---|
+| Search (cold) | 2229μs | ~46ms median | 21× slower at 100× scale → reasonable for IVF_PQ |
+| Search (warm) | (not measured) | ~20ms p50 | Stable across 5 trials |
+| Doc fetch by id | 311μs | ~5ms | Structural ADR-019 win confirmed once `doc_id` btree is built |
+| Index method | lance_ivf_pq | lance_ivf_pq | response tag confirms |
+
+**The HNSW comparison at 10M doesn't exist** — at 10M × 768d × 4 bytes = ~30 GB just for vectors, doubled for the graph. HNSW doesn't fit on a single 128 GB box at this scale. So the original "Lance pulls ahead at 10M" framing is true by elimination: Lance is the only contender that operationally exists at 10M. The strategic question is reframed as "Lance vs Parquet+HNSW-with-spilling," deferred until we have a workload where the Parquet path is the bottleneck (currently tracked in `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` decisions tracker).
+
+**Bug surfaced and fixed during re-bench:** Initial doc-fetch was ~100ms (full table scan). Root cause: the `doc_id` scalar btree was never built — `lance-bench` writes datasets by bypassing `IndexMeta`, and the activation-time auto-build only runs for IndexMeta-registered indexes. Fixed at two layers (commits `5d30b3d` + `044650a`):
+- `lance_migrate` HTTP handler auto-builds the btree inline (~1.2s on 10M, +269MB on disk)
+- `lance-bench` binary builds the btree post-IVF for parity with the gateway path
+
+**Gauntlet added 2026-05-02:** the lance crates had zero tests + no smoke when audited. Now have 7 unit tests in `crates/vectord-lance` + 12 sanitize tests in `crates/vectord` + 10-probe `scripts/lance_smoke.sh` + sanitized error boundary across all 5 routes. See commits `7bb66f0`, `ac7c996`, `e9d17f7` (sanitizer iterations driven by cross-lineage scrum).
+
+**Status:** ADR-019's hybrid architecture (Parquet+HNSW primary, Lance secondary) is now empirically validated up to 10M. The "watch and re-bench" item is closed.
+
 ## Why this isn't moving the goalposts
 
 The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile.

docs/ARCHITECTURE_COMPARISON.md

@@ -0,0 +1,46 @@
+# Lakehouse: Rust vs Go architecture comparison
+
+> **Source of truth lives in the golangLAKEHOUSE repo:**
+> [`/home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`](file:///home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md)
+>
+> J's living document — pulled from there into this repo's docs as
+> a pointer so the comparison is reachable from either side.
+
+## Why the source lives in golangLAKEHOUSE
+
+The Go rewrite was the trigger for the comparison. The doc updates as
+J ships fixes on either side, and most of the open backlog items
+(materializer port, replay port, validators network surface) land in
+the Go repo. Keeping the source there means PR auditing on Go
+commits also catches doc drift.
+
+## When to update from this side
+
+If a fix lands in the Rust repo that changes a comparison value
+(e.g. embed cache change, sidecar drop, new validator), update both:
+
+1. The source at `/home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`
+2. The change log section at the bottom of the same file
+
+This file is a pointer — **do not put authoritative content here.**
+Drift between two copies wastes the discipline.
+
+## Quick links
+
+- **Decisions tracker** — section near the top of the source file.
+  Lists actioned items + open backlog with LOC estimates.
+- **Performance numbers** — Python dependency section. Updated each
+  time a load test is rerun.
+- **Distillation porting status** — table of phase-by-phase port
+  state across runtimes.
+- **Recommendation** — current working hypothesis on Go-primary vs
+  Rust-primary. Subject to change as fixes ship.
+
+## Last known state
+
+- **2026-05-01**: Rust embed cache shipped (`150cc3b`), 236× RPS gain.
+- **2026-05-01**: Go validator port shipped (`b03521a`), production
+  safety net now on Go side.
+- **Open**: Drop Rust Python sidecar (~200 LOC, universal-win).
+- **Open**: Port Rust materializer to Go (~500-800 LOC, unblocks
+  Go-only end-to-end pipeline).

docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md

@@ -0,0 +1,177 @@
+# Phase 1.5 — BIPA Schema Audit + outcomes.jsonl Content Sample
+
+**Status:** Complete — 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md)
+
+> **Scope.** Two follow-up walks defined in `AUDIT_PHASE_1_DISCOVERY.md` §10/C4 + §10/single-reviewer (gemini "culture fit"): (1) BIPA-specific photo/video/biometric column + code path inventory, (2) actual content sample of `data/_kb/outcomes.jsonl` to confirm whether PII or discrimination-proxy reasoning lands there. Read-only walk; no code changes.
+
+---
+
+## §1 — BIPA walk: photo/video/biometric surface
+
+### 1A — The face pipeline
+
+The Rust legacy ships a synthetic face pool with deepface-extracted demographic classifications. **Even though the FACES are synthetic, the LOGIC is BIPA-relevant** — when real photos arrive (J 2026-05-03: "we will" have them), the same pipeline processes them.
+
+| File:line | What | BIPA relevance |
+|---|---|---|
+| `scripts/staffing/fetch_face_pool.py` | Fetches 1000 synthetic faces from `thispersondoesnotexist.com` (StyleGAN). | Synthetic — no biometric collection at this stage. |
+| `scripts/staffing/tag_face_pool.py:18-31` | Runs `deepface` on each face to extract `gender` (man / woman) + `race` (asian / black / hispanic / indian / middle_eastern / white) + `age`. Writes to `data/headshots/manifest.jsonl`. | **BIPA biometric processing.** Deepface extracts demographic classifications from face geometry. The output IS biometric information per 740 ILCS 14/10. |
+| `data/headshots/manifest.jsonl` | 1000 rows, each `{id, file, gender, race, age}`. Sampled rows confirm shape: e.g. `{id: 0, file: "face_0000.jpg", gender: "woman", race: "east_asian", age: 33}`. | The persisted classifications. **For real photos, this file becomes a regulated biometric database.** |
+| `mcp-server/index.ts:1308` (`/headshots/:key` route) | Serves face matched by gender × race × age intersection bucketing for a candidate. | Routing logic predicated on biometric-derived demographic attributes. |
+| `mcp-server/index.ts:1367` (`/headshots/generate/:key`) | ComfyUI on-demand portrait generation. | Custom-generated image keyed to candidate. |
+| `mcp-server/index.ts:1408` (ComfyUI prompt template) | `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic` | **Protected attributes (age + race + gender) embedded directly into the model prompt.** Even with synthetic output, this is system-level encoding of protected-attribute features into AI workflow. |
+
+### 1B — Name-based ethnicity inference (inferred-attribute risk)
+
+`mcp-server/search.html:3375-3432` defines lookup tables that classify candidates by demographic attributes derived from their NAMES (not photos):
+
+| Constant | What |
+|---|---|
+| `FEMALE_NAMES` (~200 entries) + `MALE_NAMES` | Gender inference from first name |
+| `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` | Ethnicity inference from first name |
+| `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` | Ethnicity inference from surname (more diagnostic than first name per the source comment) |
+
+**This is hard-coded inferred-attribute classification.** Per CLAUDE.md, the rationale is "Confidence-default name resolution... so a forklift operator looks like a forklift operator" — i.e., assigning representative photos. But the code path is:
+
+`candidate name → genderFor() / guessEthnicityFromFirstName() → face-pool bucket selection`
+
+A discrimination plaintiff's lawyer would frame this as: "the system EXPLICITLY classifies candidates by inferred ethnicity and uses that classification in its workflow." Whether the downstream use is "just for the photo display" or "ranks candidates" doesn't matter for BIPA-style strict-liability framing — it matters for Title VII, where intent / disparate-impact tests apply.
+
+### 1C — Data directories holding face content
+
+| Path | Count today | Purpose | gitignored? |
+|---|---:|---|---|
+| `data/headshots/` (face_*.jpg + manifest.jsonl) | 2500 entries | Synthetic face pool + biometric classifications | partial — manifest.jsonl is tracked, face_*.jpg are gitignored |
+| `data/headshots_role_pool/` | 7 entries | Role-specific representative faces | gitignored |
+| `data/headshots_gen/` | 4 entries | ComfyUI on-demand portrait cache | gitignored |
+| `data/icons_pool/` | 2 entries | Certification + role icons | gitignored |
+| `data/face_test/` | 1 entry | Likely deepface dev/test scratch | gitignored |
+| `data/_imagecache/` | (not enumerated) | Image cache | gitignored |
+
+The `manifest.jsonl` file is **tracked in git** today and contains the demographic-classification table. For synthetic faces this is fine; **for real candidate faces, this would be a biometric database in version control** — a serious BIPA + general-data-protection failure.
+
+### 1D — Biometric-info contract risk when real photos arrive
+
+When the system transitions from synthetic to real candidate photos (J: "we will"), the BIPA exposure becomes immediate:
+
+1. **No written consent flow exists.** BIPA §15(b) requires "informed written consent" before biometric collection. The headshot upload path (wherever it gets built) needs a consent gate.
+
+2. **No public retention schedule exists.** BIPA §15(a) requires a written, publicly-available retention schedule. No such document found.
+
+3. **No deletion procedure exists.** BIPA §15(a) requires destruction "when the initial purpose for collecting or obtaining such identifiers or information has been satisfied or within 3 years of the individual's last interaction with the private entity, whichever occurs first." No such procedure exists.
+
+4. **No employee training documented.** BIPA litigation routinely cites lack of employee training on biometric handling as evidence of negligence.
+
+5. **The deepface library is itself a biometric processor.** Using it triggers the BIPA disclosure obligation regardless of whether the system stores the underlying face image — extracting "race" and "gender" from a face IS biometric processing under modern interpretations.
+
+6. **Per-violation damages.** Negligent violations: $1,000 per violation per person. Intentional/reckless: $5,000 per violation per person. The IL Supreme Court has held each separate scan/extraction can constitute a separate violation (Cothron v. White Castle, 2023). Class actions are routine; recent settlements:
+   - Facebook (Patel v. Facebook, 2020): $650M
+   - TikTok (In re TikTok BIPA, 2022): $92M
+   - Six Flags (Rosenbach v. Six Flags, 2019): set the "no actual harm needed" precedent
+   - Smaller staffing companies have settled for $1M-$50M ranges.
+
+### 1E — What this means for the architecture
+
+Even with synthetic data, the architecture has BIPA-shaped surface that is one upload path away from being live. **Recommended pre-real-photo gates:**
+
+1. **Disable deepface tagging on real-photo path** until consent + retention schedule are in place. Synthetic faces can keep using deepface; real faces cannot.
+2. **Quarantine `data/headshots/manifest.jsonl` for real photos.** Real-photo classifications must NOT land here. Either move to a dedicated `data/biometric/` tree with separate retention or — preferable — don't extract demographic features from real photos at all.
+3. **Replace the name → ethnicity lookup tables with a non-classifying default.** The current logic exists to pick "representative" photos. For real photos, the candidate's own photo IS representative — no inference needed. The lookup tables become dead code.
+4. **Upload path needs consent gate.** Whatever endpoint accepts real photos must check a stored consent timestamp + version before storing the file.
+5. **Retention schedule documented + auto-enforced.** Crontab / ingestd background sweep that deletes biometric data per the schedule.
+
+These five gates are NOT in `AUDIT_TRAIL_PRD.md` §8 phase plan. **Recommend adding as Phase 1.6 (BIPA pre-launch gates)** ahead of any real-photo intake.
+
+---
+
+## §2 — outcomes.jsonl content sample
+
+### 2A — File state
+
+| Property | Value |
+|---|---|
+| Rows | 101 |
+| Path | `data/_kb/outcomes.jsonl` |
+| Rows with `name` field in fills | **39** |
+| Rows with `first_name` field | 0 (the field is `name`, not `first_name` — names are pre-concatenated in this sink) |
+| Sample names in fills (synthetic but representative) | `Alex Rivera`, `Carmen I. Garcia`, `Jacob N. Patel`, `Jamal Z. Jones`, `James Park`, `Maria Chen`, `Mei Y. Cooper`, `Peter E. Carter`, `Sam Torres` |
+
+### 2B — Confirmed PII shapes in the file today
+
+- `fills[*].name` — full candidate name (synthetic but real shape)
+- `operation` — natural-language fill request (e.g. "fill: Welder x2 in Toledo, OH"). Currently doesn't carry candidate names in the operation string itself, BUT the `error` field can — model reasoning that names specific candidates lands here
+- `error` — model self-correction text and validator failure messages. Sampled rows contain phrases like "executor parse failure on turn 8: no JSON object in Executor response: ..." which is technical content. Real-data scenarios where the model reasons about specific candidates would put names here.
+
+### 2C — Discrimination-proxy phrase audit (gemini scrum flag §10/single-reviewer)
+
+Searched for common LLM-generated discrimination proxy phrases. **0 hits in current synthetic-data sample**:
+
+- "culture fit" — 0
+- "communication" — 0
+- "team chemistry" — 0
+- "personality" — 0
+- "soft skills" — 0
+- "work ethic" — 0
+- "professional attitude" — 0
+
+**Interpretation:** the current synthetic data doesn't generate these phrases. However:
+- The file is structurally capable of holding them (it persists model `error` text)
+- When real models (kimi-k2.6, deepseek-v3.2 per PR #13) reason about real candidates, they WILL generate phrases like these — these are common in HR/staffing LLM output
+- Once a phrase like "rejected as not a culture fit" lands in outcomes.jsonl, it's append-only persistence. Cryptographic erasure is the only RTBF path.
+
+**Action:** add a content-redaction pass to the outcomes.jsonl writer BEFORE production data lands. Strip discrimination-proxy phrases at write time. (This is not a substitute for the model not GENERATING them — it's a defense-in-depth layer.)
+
+### 2D — Implications for the audit trail
+
+The outcomes.jsonl sample confirms what `AUDIT_PHASE_1_DISCOVERY.md` §1D predicted: **39 of 101 rows persist candidate names in the `fills[*].name` field today**. The shape is real even if the data is synthetic.
+
+For the Phase 2 identity service design:
+
+1. **The training-safe export pipeline (per J 2026-05-03 answer 11) MUST address outcomes.jsonl directly.** This file is the most-likely candidate for RAG re-indexing or fine-tuning corpus, and it carries names.
+
+2. **Crypto-erasure target list** must include `data/_kb/outcomes.jsonl`, `data/_kb/overseer_corrections.jsonl`, `/tmp/lakehouse-validator/sessions.jsonl`, plus pathway memory state, plus Langfuse storage.
+
+3. **Subject_id top-level promotion (per AUDIT_PHASE_1_DISCOVERY §10/C5)** can replace `fills[*].name` as the queryable field. The name itself can move into a `[REDACTED]` token until legal access dereferences via the identity service.
+
+---
+
+## §3 — Synthesis: net new findings for Phase 2 design
+
+| Finding | Source | Phase 2 design implication |
+|---|---|---|
+| **Phase 1.6 (NEW) — BIPA pre-launch gates needed before real photos accepted** | §1E above | Phase 2 identity service design must include biometric-info table with consent + retention metadata schema. |
+| **Name-based ethnicity inference is a separate Title VII / disparate-impact risk** | §1B above | Phase 2 must include a "no protected-attribute inference at routing time" architectural rule. The lookup tables should be deprecated as part of the identity-service migration. |
+| **deepface library use itself triggers BIPA when real photos arrive** | §1D point 5 | Phase 2 design must specify "biometric processing requires consent token presence" as a gateway boundary check. |
+| **outcomes.jsonl persists names today (39/101 rows)** | §2A above | Phase 2 needs subject_id top-level promotion + name field as `[REDACTED-{token}]` reference (token resolves via identity service for authorized callers). |
+| **No discrimination-proxy phrases in current synthetic data, but the sink is shape-ready for them** | §2C above | Phase 2 should specify a redaction-on-write pass for outcomes.jsonl + sessions.jsonl writers. |
+| **manifest.jsonl is tracked in git today** | §1C above | If real photos arrive without changing the manifest path, biometric data lands in version control — must be addressed as part of Phase 1.6 gates. |
+
+---
+
+## §4 — Open questions surfaced by Phase 1.5
+
+1. **Where will real photos enter the system?** No upload endpoint walked. Need to know the entry point before designing the consent gate.
+2. **Will the deepface tagging path be preserved for real photos, or stripped?** Architectural decision — affects whether biometric extraction is in scope at all.
+3. **What's the staffing client's consent UX?** Does the candidate sign consent during onboarding, separately for biometrics, etc.? Affects the consent-token design.
+4. **How long do BIPA records need to persist?** BIPA's "3 years from last interaction" is a maximum, not a target — staffing-co's contract may set a shorter floor (e.g., "delete on placement+1 year"). Need staffing-co policy.
+5. **Is there a Designated Privacy Officer / DPO?** Per CCPA/GDPR best practice + BIPA litigation hygiene, this role should exist before real-PII intake. Out of scope for code; flag for J.
+
+---
+
+## §5 — Recommended Phase 1.6 BIPA pre-launch gates (NEW)
+
+Lands as new phase BETWEEN Phase 1.5 (this doc) and Phase 2 (identity service design). Read-only/design work; one code commit (the consent gate). Estimated 4-8 hours.
+
+1. **Write `docs/BIPA_COMPLIANCE_POLICY.md`** — public retention schedule, written consent template, deletion procedure, employee training acknowledgment. Counsel must review before publication.
+2. **Add consent gate to whatever endpoint accepts real photos** (not yet built — design with the gate from day 1, don't bolt on after).
+3. **Quarantine real-photo classifications to `data/biometric/`** — separate path, separate retention, separate access control. Real-photo `manifest.jsonl` does NOT land in `data/headshots/`.
+4. **Deprecate name → ethnicity lookup tables in `mcp-server/search.html`** (commit removing those constants + the call sites). Replace with "use candidate's actual photo" path or non-inferred default.
+5. **Add a unit test that asserts `data/headshots/manifest.jsonl` row count matches the SYNTHETIC face count** — fails if real-photo classifications leak into the synthetic manifest.
+
+These five fit between Phase 1.5 and Phase 2 because (a) they're MORE URGENT than Phase 2 (BIPA exposure starts the moment a real photo touches deepface), (b) they're independent of the identity service design, (c) they need counsel sign-off before they can ship anyway.
+
+---
+
+## Change log
+
+- 2026-05-03 — Phase 1.5 BIPA + outcomes content audit complete. New finding: synthetic data + deepface + name-ethnicity inference creates BIPA-shaped surface that is one upload path away from being live. Recommend Phase 1.6 (BIPA pre-launch gates) before Phase 2 design starts.

docs/AUDIT_PHASE_1_DISCOVERY.md

@@ -0,0 +1,295 @@
+# Phase 1 Discovery — Subject + PII Surface Map
+
+**Status:** Draft — 2026-05-03 · **Drafted by:** working session 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md)
+
+> **Purpose.** Read-only walk of both runtimes. Fills in the "UNKNOWN" cells in `AUDIT_TRAIL_PRD.md` §3 surface map and §7 current-state-vs-target table with file:line evidence. **No code changes.** Output: a complete picture of where subject identifiers + PII flow today, where they leak, and what an audit response could and could not produce right now.
+
+---
+
+## TL;DR — what the walk found
+
+1. **A defense layer EXISTS but is BYPASSED.** Two PII-masked views (`candidates_safe`, `workers_safe`) live in `data/_catalog/views/` and correctly drop name/email/phone. **No production tool query uses them.** The MCP tool registry's `search_candidates` and `get_candidate` SQL templates query the raw `candidates` / `workers_500k` tables and return full PII to the LLM context. Worth considering: the views are functioning policy artifacts that nobody routes through.
+
+2. **PII flows freely through the LLM substrate.** In a single fill scenario, candidate names + emails + phones traverse: SQL → tool_result → LogEntry → execution_loop log → `/v1/respond` HTTP response → Langfuse trace input + output → `data/_kb/outcomes.jsonl` (fills with name) → `data/_kb/overseer_corrections.jsonl` (operation + correction text). At least 7 distinct persistence/transmission paths per scenario.
+
+3. **`candidate_id` is a stable token but not an isolated one.** It's a column on the same `workers_500k.parquet` that holds name/email/phone — there is no separate identity service. Joining candidate_id back to PII is one DataFusion query away. The audit-trail PRD's §5 identity-service intent is not yet built; the architectural separation does not exist.
+
+4. **No `/audit/subject/{id}` endpoint exists.** Reconstructing "every decision about person X" today requires manual cross-correlation across 4+ JSONL files + Langfuse + pathway memory + observer events. There is no canonical query path; the audit response cannot be produced in any reasonable time today.
+
+5. **Go side mirrors the Rust pattern.** Go validator's `rosterRow` carries Name; Go SessionRecord carries `Prompt` (truncated to 4000 chars) which contains the natural-language operation including candidate names. Cross-runtime parity in the PII-leak too.
+
+6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today.
+
+7. **The matrix-indexer fingerprint is subject-agnostic; trace bodies are UNVERIFIED.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers, which is structurally defensive. **However:** trace bodies (`reducer_summary`, `final_verdict`) are written from execution-loop output and are highly likely to leak PII. Per §8.1 these are unverified. Treat the matrix indexer as a SUSPECTED PII SINK until sampled — do NOT rely on "matrix can't drive discrimination" framing. (Walked back from earlier draft per cross-lineage scrum §10/C1: that claim was over-stated in 3/3 reviews.)
+
+---
+
+## §1 — PII flow paths in the Rust stack (file:line evidence)
+
+### 1A — MCP tools query raw tables (PII enters the LLM context)
+
+| File:line | What | PII columns returned |
+|---|---|---|
+| `crates/gateway/src/tools/registry.rs:96` | `search_candidates` SQL template | `first_name, last_name, phone, email, city, state` |
+| `crates/gateway/src/tools/registry.rs:107` | `get_candidate` SQL template | `SELECT *` — returns ALL columns including hourly_rate_usd, address, anything in the table |
+| `crates/gateway/src/tools/registry.rs:129` | `top_recruiters` SQL | recruiter names (employee PII, not candidate but still PII) |
+| `crates/gateway/src/tools/registry.rs:141` | `engaged_unplaced_candidates` SQL | `c.candidate_id, c.first_name, c.last_name, c.phone, c.vertical` |
+| `mcp-server/index.ts:1722, 2173, 2269` | address concatenation (`street_number + street_direction + street_name`) | full street addresses surfaced to MCP UI clients |
+
+**Architectural gap:** the MCP tool registry has no "view-only" mode. Adding `_safe` view routing would be one `sql_template` rewrite per tool. **Today: every fill scenario invokes search_candidates → LLM sees full PII for every candidate in the result set.**
+
+### 1B — LLM tool results captured in execution log
+
+| File:line | What |
+|---|---|
+| `crates/gateway/src/execution_loop/mod.rs:30-39` | `LogEntry` struct: `{turn, role, model, kind, content: Value, at}`. `content` is the raw tool_result body (carries PII when kind=`tool_result`). |
+| `crates/gateway/src/execution_loop/mod.rs:217` | `self.append(LogEntry::new(..., "tool_result", trimmed))` — every tool result appended to `self.log` |
+| `crates/gateway/src/execution_loop/mod.rs:1319` | per-render cap: `kind == "tool_result"` gets 1200 char cap (vs 200 for other kinds). Trimming is for token economy, **not PII redaction** — names/emails fit in 1200 chars. |
+| `crates/gateway/src/execution_loop/mod.rs:30-50` | `Fill` struct (in same file): `{candidate_id: String, name: String, reason: Option<String>}`. The `name` field is load-bearing per the PRD-defined FillProposal artifact contract. |
+
+### 1C — Execution log returned to HTTP caller
+
+| File:line | What |
+|---|---|
+| `crates/gateway/src/v1/respond.rs:111-119` | `RespondResponse { status, artifact, log: outcome.into_log(), iterations, error }` — full log array returned in JSON response body to whoever called `POST /v1/respond` |
+| `crates/gateway/src/execution_loop/mod.rs:82-93` | `RespondOutcome::{Ok,Failed,Blocked}` all carry `log: Vec<LogEntry>` — no path drops the log on its way to the caller |
+
+**Audit implication:** any HTTP client calling `/v1/respond` today receives full PII in the response. Authentication on `/v1/respond` is the only access control. Authorization is binary (either you can call it or you can't); there is no row-level filtering of which candidates you're allowed to see in the response.
+
+### 1D — Persisted JSONL sinks on disk
+
+| File:line | Path written | PII shape |
+|---|---|---|
+| `crates/gateway/src/execution_loop/mod.rs:508` | `data/_kb/outcomes.jsonl` (append) | `{operation, fills: [{candidate_id, name, reason?}], error, ...}` — observed live: `operation` carries natural-language fill request including role + city + state; `fills` carries the full Fill struct with name. |
+| `crates/gateway/src/execution_loop/mod.rs:701` | `data/_kb/overseer_corrections.jsonl` (append) | `{operation, correction, sig_hash, ...}` — `correction` is the overseer's free-text guidance which often references specific candidates by name (e.g. "the executor picked Emily Garcia who is at fill capacity, try Maria Rodriguez instead") |
+| `mcp-server/observer.ts:138-139` | `data/_observer/ops.jsonl` (append) | observer event log; PII content depends on what the operation context included. **Empty on this box right now** (file doesn't exist), but the writer is wired. |
+| `mcp-server/observer.ts:161` | `data/_kb/observer_escalations.jsonl` (append) | sampled live: technical analysis fields (no PII observed in current rows), but `analysis` field is free-form LLM output and could include names depending on the escalation trigger. |
+| `crates/gateway/src/v1/session_log.rs` (writer) → `/tmp/lakehouse-validator/sessions.jsonl` | per `lakehouse.toml` `[gateway].session_log_path` | `SessionRecord { ..., prompt, attempts: [{... raw}], artifact }` — **`prompt` carries the operation text (PII), `raw` carries each model attempt's raw text (PII when model reasoned about specific candidates), `artifact` carries the final FillProposal `{candidate_id, name}` shape on success.** |
+
+### 1E — Langfuse external persistence
+
+| File:line | What gets sent |
+|---|---|
+| `crates/gateway/src/v1/langfuse_trace.rs:166` | `body.input.messages = ev.input` — full message array (system + user + assistant + tool messages). Tool messages contain raw PII for tool_result entries. |
+| `crates/gateway/src/v1/langfuse_trace.rs:191` | `body.output = ev.output` — full model generation. When the model emits a Fill or reasons about candidates, names appear here. |
+
+Langfuse runs at `:3001` per memory, with credentials in `/etc/lakehouse/langfuse.env`. The Langfuse storage tier holds these traces. **PII leaves the lakehouse infrastructure boundary at this point** — Langfuse's storage layer (Postgres + ClickHouse, default config) holds it.
+
+### 1F — Pathway memory
+
+| File:line | What |
+|---|---|
+| `crates/vectord/src/pathway_memory.rs:169` | `PathwayTrace` struct definition |
+| `data/_pathway_memory/state.json` | Persisted state — **need read** to confirm whether real names appear in traces. |
+
+Pathway traces are keyed by `pathway_id = SHA256(task_class + file_prefix + signal_class)` — none of which are subject identifiers. **The fingerprint is structurally subject-agnostic.** Whether the trace BODY (kb_chunks, observer_signals, reducer_summary, final_verdict) leaks PII depends on what those fields contain at write time. From the struct shape: `reducer_summary` and `final_verdict` are strings written from execution loop output — **highly likely to leak PII** when summarizing fill outcomes. Need to confirm against live state.json.
+
+---
+
+## §2 — Defense layer that exists but is bypassed
+
+`data/_catalog/views/candidates_safe.json`:
+- Drops: `last_name, email, phone, hourly_rate_usd`
+- Masks: `candidate_id` (keep first 3, last 2 chars — `CAND-...01` shape)
+- Row filter: `status != 'blocked'`
+- Visibility intent (per `description`): "Visible to recruiter / mode-runner agents."
+
+`data/_catalog/views/workers_safe.json`:
+- Drops: `name, email, phone, zip, communications, resume_text`
+- Reason given: "resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale"
+- Source for the rebuilt `workers_500k_v9` vector corpus
+
+**The `_safe` views are the right policy artifact.** The ARCHITECTURAL gap is at the SQL-template layer in `crates/gateway/src/tools/registry.rs` — the LLM-facing tools query the RAW tables, not the SAFE views. Three out of three candidate-touching tool templates (search_candidates, get_candidate, engaged_unplaced_candidates) bypass the safe view.
+
+There is no enforcement preventing this. There is no test that asserts "tool SQL must reference only `_safe` tables." There is no warning logged when raw tables are queried via tool surface.
+
+**Quick fix shape (NOT a phase 1 deliverable, just noting the change shape):** rewrite the tool sql_templates to `FROM candidates_safe` (or `FROM workers_safe`); add a build-time check that `crates/gateway/src/tools/registry.rs` only references `*_safe` tables; add a runtime gate in queryd that refuses LLM-attributed queries on raw tables. These would land in Phase 4 (subject tagging across substrates) or possibly Phase 1.5 (defense-layer enforcement) per the AUDIT_TRAIL_PRD.
+
+---
+
+## §3 — Identity / candidate_id provenance
+
+| Question | Finding |
+|---|---|
+| Is `candidate_id` a stable token? | Yes — observed format `CAND-NNNNNN` (e.g. `CAND-000001`). Stable across the table. Used as join key in the schema (per `crates/ui/src/main.rs:160` cross-table join hints). |
+| Is the candidate_id ↔ PII mapping in a separate service? | **No.** Both live in `data/datasets/workers_500k.parquet`. A SQL `SELECT name, email FROM candidates WHERE candidate_id = '...'` resolves the mapping in one query. |
+| Is the mapping itself audited? | **No.** No log records "who looked up the PII for which candidate when." This is the identity-service gap from PRD §5. |
+| Does anything ever generate a *different* token (UUID, opaque hash) instead of using candidate_id directly? | **No** — every tool, every validator, every persistence sink uses candidate_id as-is. |
+
+**Production-ready implication:** the staffing client's lawyer asks "show me the access log for candidate X's PII" → we cannot produce one. Every access happens via SQL against `workers_500k`; no row-level access log exists.
+
+---
+
+## §4 — Go-side parity with the leak surface
+
+| Rust file:line | Equivalent Go file:line | Same shape? |
+|---|---|---|
+| `crates/validator/src/lib.rs` (FillProposal carries `{candidate_id, name}`) | `internal/validator/fill.go` + `internal/validator/lookup_jsonl.go:23-31` (`rosterRow` carries `CandidateID, Name, Status, City, State, Role, BlacklistedClients`) | Yes — same PII shape |
+| `crates/gateway/src/v1/session_log.rs` SessionRecord with prompt/artifact | `internal/validator/session_log.go` SessionRecord with `Prompt string`, `Attempts[].Raw string`, `Artifact map[string]any` | Yes — same shape |
+| `outcomes.jsonl` / `overseer_corrections.jsonl` writers | Not present on Go side (Go validatord doesn't currently run the execution_loop with tool dispatch — Phase 4 of Go rewrite is what wires that) | **Asymmetric** — Go side writes less but only because feature parity isn't done |
+| MCP tools registry | Not on Go side yet (mcp-server is still Bun) | **Bun is the surface** for tool dispatch in both runtimes today |
+
+**Cross-runtime audit implication:** even though our 5 parity probes (validator/extract_json/session_log/materializer/embed) all pass 32/32, **none of them assert PII handling**. A new `pii_parity.sh` probe would feed identical PII-tagged input through both runtimes and assert identical redaction behavior. As of today, neither runtime *does* redaction at the substrate level, so the probe would just confirm "both leak identically."
+
+---
+
+## §5 — Mapping back to AUDIT_TRAIL_PRD §3 surface map
+
+Updated cells with file:line evidence:
+
+| Decision happens at | Currently logged where | Audit-completeness gap (revised) |
+|---|---|---|
+| Ingestion (candidate added to pool) | `data/datasets/workers_*.parquet` rows; ingestd writes via `crates/ingestd/`; no per-subject "added at" event journal found | **GAP** — no subject-tagged ingest event. When + who + how a candidate entered is not auditable per-subject. |
+| Embedding creation | `crates/aibridge/src/client.rs` (post-sidecar-drop direct path); LRU cache `crates/aibridge/src/cache.rs` (commit `150cc3b`); cached entries are keyed by `(model, text)` — **the text is the cached body, which for candidate embeddings IS PII (candidate name + role + skills appear in source text per workers_500k_v9 build SQL).** Cache itself is in-memory, not persisted, but text-as-key means PII is in process memory. | **MAJOR GAP** — no per-embedding audit row. Cache key contains PII. No subject tagging means we cannot answer "what embedding was generated for candidate X." |
+| Search inclusion | Tool result entries in execution log + Langfuse + outcomes.jsonl `fills` field | Partial — fills are logged, but you have to grep across outcomes.jsonl + Langfuse for any mention of a candidate_id. No "all search results that included X" canonical view. |
+| Search rank | Result set in chat traces (Langfuse), not indexed by candidate | Partial — Langfuse trace has the result set; no inversion (candidate → ranks-received). |
+| Fill recommendation | `outcomes.jsonl` `fills` array with name + candidate_id | Partial — present, but mixed with PII in the natural-language `operation` field. |
+| Validation outcome | `sessions.jsonl` per `[gateway].session_log_path` config; per-attempt `verdict_kind`, `error` | Partial — works per-session, not per-subject. To find "all validations that touched candidate X" you'd need to grep the JSONL by candidate_id. |
+| Iterate retry escalations | Same sessions.jsonl `attempts[]` array | Same as above |
+| Observer signals | `data/_observer/ops.jsonl` (file not present today on this box, writer is wired) | UNKNOWN content shape until the writer fires for a fill scenario — needs verification |
+| Matrix-indexer compounding | `data/_pathway_memory/state.json` — fingerprinted by code, NOT by subject | **NO subject leak in fingerprint structure** (good for audit defensibility). Trace bodies (`reducer_summary`, `final_verdict`) MAY carry PII — needs sampling to confirm. |
+
+---
+
+## §6 — Mapping back to AUDIT_TRAIL_PRD §7 current-state-vs-target gap table
+
+| Capability | Phase 1 finding | Status |
+|---|---|---|
+| candidate_id as canonical token | Stable `CAND-NNNNNN` format. Same column lives in same parquet as PII. | Token exists; isolation does not. |
+| Identity service | Doesn't exist. PII + candidate_id co-located in `workers_500k.parquet`. | Real gap — needs new service. |
+| `/audit/subject/{id}` endpoint | Doesn't exist. No audit route in `crates/gateway/src/v1/mod.rs` route listing. | Real gap — needs new endpoint. |
+| Subject-tagged embeddings | LRU cache `(model, text)` keys; text contains PII; no audit row per embed. | Real gap. |
+| Subject-tagged search results | Langfuse trace contains result set; not subject-indexed. | Real gap (queryability, not capture). |
+| Subject-tagged validation outcomes | Yes — sessions.jsonl carries candidate_id (in the attempt's `raw` field) but not as a queryable top-level field. | Partial — needs subject_id top-level promotion. |
+| Subject-tagged matrix indexer entries | Pathway fingerprints are subject-agnostic by design. Trace bodies may leak. | Decision needed (PRD open question 7) — keep code-only OR risk PII surface. |
+| Protected-attribute filter at decision time | **Not enforced anywhere.** SQL templates return whatever columns they SELECT; no protected-attribute removal at the gateway boundary. The `candidates` table schema in the demo SQL includes age (years_experience is a proxy), and the call_log/email_log tables likely contain free-text correspondence. | Real gap — major. Requires both schema audit + boundary enforcement. |
+| Retention policy | None enforced. Append-only files grow indefinitely. | Real gap. |
+| Right to be forgotten | Not implementable today on append-only logs without cryptographic erasure. | Per PRD §6 design, real engineering needed. |
+| Cross-runtime parity | 5 algorithm probes pass; 0 audit/PII probes exist. | Probe set needs extension. |
+
+---
+
+## §7 — Worked example: John Martinez audit trail TODAY (negative result)
+
+If John Martinez (candidate_id `CAND-042195`) requests an audit:
+
+1. **Find his candidate_id.** Manual SQL: `SELECT candidate_id FROM candidates WHERE first_name='John' AND last_name='Martinez'` — returns 1+ rows. Already a leak: someone with SQL access can correlate name → candidate_id ad-hoc.
+
+2. **Find every fill scenario that included him.** `grep "CAND-042195" data/_kb/outcomes.jsonl` — returns rows where his fill was included. But the row's natural-language `operation` field may NOT contain his candidate_id (it's the fill request, e.g. "fill: Welder x2 in Toledo, OH"), so we'd miss scenarios where he was a *candidate* but not a *fill*. To find those, we'd need to grep Langfuse traces (off-box) or the per-tool result content in execution logs (which aren't persisted as separate JSONL — they live in the HTTP response that already left the building).
+
+3. **Find every validation that touched him.** Grep `/tmp/lakehouse-validator/sessions.jsonl` for `CAND-042195` — would catch FillValidator phantom-ID rejections AND successful fills, but the candidate_id would appear in either the prompt or the raw attempt text, not as a top-level queryable field.
+
+4. **Find every embedding generated for him.** Cache is in-memory; nothing persisted. Cannot answer.
+
+5. **Find every search result that ranked him.** Off-box in Langfuse. Untrieable in lakehouse without a separate Langfuse query pipeline (which doesn't exist).
+
+6. **Find pathway memory traces involving him.** Pathway fingerprints don't carry his ID. The `reducer_summary` strings might mention him (need to grep state.json) but the fingerprint search wouldn't surface them.
+
+7. **Show what protected attributes were exposed to the model.** No record of input_features per decision — the LLM saw whatever the SQL returned, and we have no per-decision input_features audit row.
+
+8. **Format the output for legal.** Even if we collected all the above, there's no signing, no integrity hash, no schema, no template.
+
+**Estimated time to produce a complete-and-defensible response today: not possible.** Estimated time to produce an INCOMPLETE response by cobbling JSONLs + Langfuse exports: 2-5 hours per request, manual, error-prone, and the response would over-share (other candidates appearing in the same fill scenarios) AND under-share (embedding events, search rankings, pathway traces missed).
+
+This is the Phase 1 result the AUDIT_TRAIL_PRD predicted: today's substrate is not production-ready for a discrimination-defense audit response.
+
+---
+
+## §8 — What this discovery DID NOT cover
+
+Phase 1 was scoped to file:line evidence + sampling of live JSONL state. The following deserve their own subsequent walks before phase 2+ design:
+
+1. **Live sample of `data/_pathway_memory/state.json`** — confirm whether `reducer_summary` / `final_verdict` strings actually leak PII or stay generic. Read 3-5 traces and grep for names from `workers_500k`.
+2. **Live sample of Langfuse traces** — confirm input message PII for a real fill scenario from past 7 days. Use Langfuse `:3001` query API.
+3. **observerd event content** — the writer is wired but `data/_observer/ops.jsonl` doesn't exist on this box. Trigger a fill scenario and inspect the resulting event.
+4. **Bun mcp-server tool dispatch** — does the Bun server log tool calls anywhere? `mcp-server/index.ts` is 2900+ lines; partial walk only here.
+5. **bot/propose.ts** — the bot proposal flow likely touches candidates; not walked in this pass.
+6. **`crates/journald` mutation log** — designed for row-level mutations per ADR-012; haven't confirmed whether candidate-table mutations land here with PII.
+7. **Go side observerd + chatd PII surface** — the Go cmd/* binaries likely have analogous logging; confirmed validator parity but didn't walk observer/chat logging.
+8. **Process memory + crash dumps** — if the gateway dumps core, what PII is in it? Out of scope for code walk; comes up in security audit.
+9. **Operator runbook** — who has access to logs/, /tmp/lakehouse-validator/, MinIO buckets, Langfuse Postgres? Out of scope for code walk; comes up in operational security review.
+
+These are listed so the next phase doesn't accidentally re-walk what's done OR skip what wasn't covered.
+
+---
+
+## §9 — Recommended next moves (not commitments)
+
+Per AUDIT_TRAIL_PRD §8, Phase 2 is the identity service design doc. Before that doc gets written, the cheapest high-value moves discovered here:
+
+1. **Defense-layer enforcement (1-2 hours work).** Rewrite the 3 tool SQL templates in `crates/gateway/src/tools/registry.rs` to use `_safe` views. Add a unit test that asserts no `crates/gateway/src/tools/registry.rs` template references `FROM candidates ` or `FROM workers_500k ` (only `*_safe`). This is one commit. It prevents the most-trafficked PII leak path TODAY without waiting on the identity service. **Cost:** the LLM sees masked candidate_ids (`CAN...95` instead of `CAND-042195`); some downstream tools (validator existence check) would need a "resolve to full ID" path that goes through the identity service — but that's exactly the architectural shape PRD §5 wants anyway.
+
+2. **Sample state.json + Langfuse before phase 2 design.** §8.1 + §8.2 above. ~30 minutes. Either confirms or refutes the "matrix indexer is subject-clean" finding from §1F.
+
+3. **Document the Bun mcp-server tool surface.** §8.4. The Bun layer is a major PII transit point not fully covered here.
+
+4. **Identify whether protected attributes (age proxies, photo features, zip-code → race correlations) are currently in any tool-returned column.** Schema-level audit of `candidates` + `workers_500k`. ~1 hour. Might surface that some "neutral" columns are actually protected-attribute proxies.
+
+These four moves give the phase-2 design doc strong evidence to lean on. None are commitments — J's call on what to do next.
+
+---
+
+## §10 — Cross-lineage scrum review of this discovery doc (2026-05-03)
+
+After the discovery walk, this document was reviewed by three independent model lineages via `/v1/chat` against the Go gateway (post-PR-#13 model fleet): **opus** (`opencode/claude-opus-4-7`), **kimi** (`ollama_cloud/kimi-k2.6`), **gemini** (`ollama_cloud/gemini-3-flash-preview`). Custom prompt: senior security architect reviewing a discovery report. (DeepSeek-v3.2 timed out; not included.)
+
+Verbatim reviews saved at `/tmp/audit_scrum/{opus,kimi,gemini}_review.md`. Convergent findings (≥2 reviewers) are treated as high signal per `feedback_cross_lineage_review.md`.
+
+### Convergent findings — must address before phase 2 design
+
+**C1. §1F matrix-indexer claim is OVER-CLAIMED (3/3 reviewers).** The TL;DR #7 line "good for audit defensibility (matrix index can't drive discrimination)" overstates structural subject-agnosticism as behavioral fairness. Per opus: "until §8.1 is executed, the correct framing is 'fingerprint structurally subject-agnostic; body content unverified — treat as suspected PII sink.'" Per kimi: dangerous reasoning — if `reducer_summary` says "candidates named Emily Garcia were rejected for fill capacity," the matrix learns proxy variables and future similar names get downranked = temporal discrimination risk. Per gemini: "you cannot claim audit defensibility until you prove the *content* of the matrix indexer doesn't contain PII." **Action:** walk back the §1F TL;DR claim; reframe as "fingerprint structure is defensive; body content unverified — treat as suspected PII sink until §8.1 confirms."
+
+**C2. §1E (Langfuse) is the MOST DANGEROUS leak — fix FIRST (3/3 reviewers).** Opus: "boundary-crossing leak that makes a regulator's eyes light up." Kimi: "Article 44 GDPR transfer if SaaS-hosted; CPRA 'sale/sharing' question; subprocessor notification failure." Gemini: "un-certifiable for SOC2 Type II" + "unauthorized data transfer to third-party storage tier." All three would do Langfuse redaction/sampling BEFORE the §9.1 view-routing fix. **Action:** revise §9 priority order — Langfuse boundary first, view-routing second.
+
+**C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills (3/3 reviewers).** Opus: "you need not just 'what did we do for candidate X' but 'what was the selection rate for protected class Y vs comparator' — that requires capturing protected attributes WITH outcomes." Kimi: "EEOC Uniform Guidelines on Employee Selection Procedures (1978) — matrix index that learns from historical fill outcomes IS a selection procedure under the Guidelines." Gemini: "The system doesn't log the *entire* candidate pool for a specific search — only the fills (§5). To defend a lawsuit, you must show the stats for everyone who *could* have been picked, not just the person who was." **Action:** Phase 1 didn't capture this load-bearing requirement. PRD §1 worked example needs expanding: the audit response must include the comparator pool + adverse-impact statistics, not just the subject's decision row.
+
+**C4. BIPA + biometric exposure UNDERSTATED (3/3 reviewers).** Already in `AUDIT_TRAIL_PRD.md` §10.5 jurisdictional checklist but NOT translated into Phase 1 findings actionables. If `workers_500k` columns include photo paths, video interview references, or anything that could yield biometric inference (per gemini: "even descriptors that could be reconstructed into biometric templates"), BIPA's $1k-$5k per-violation regime applies BEFORE the GDPR analysis matters. **Action:** add to §8 (what discovery did NOT cover): explicit photo/video/biometric column audit of `workers_500k` schema.
+
+**C5. `candidate_id` must be PROMOTED to top-level field in all JSONL sinks (3/3 reviewers).** Opus + kimi + gemini converge: grepping natural-language strings (operation, raw, prompt, reducer_summary) for candidate_id is not a defensible audit strategy. Even if subject_id appears in those strings TODAY, it appears co-mingled with other candidate names, model reasoning, etc. — making subject filtering unsafe. **Action:** add to PRD §7 target column "subject_id top-level promotion" — change session_log writer + outcomes.jsonl writer + observer event writer to ALL include `subject_id` (or `subject_ids[]`) as a first-class top-level field.
+
+### Single-reviewer findings — verified, worth incorporating
+
+**OPUS unique:**
+- **8th PII path missing: LLM provider egress.** PR #13 routes models through opencode + ollama_cloud + openrouter — opencode and openrouter are external services. Cross-border data transfer under GDPR Ch. V; third-party processor relationship requiring DPA under Art. 28. Phase 1 did not enumerate this path. **Action:** add §1G "LLM provider egress" to §1.
+- **GDPR Art. 22 / EU AI Act right to explanation.** Audit must capture the model's REASONING CHAIN, not just decision output. Phase 1 §7 worked example doesn't include this. **Action:** add to subject-audit response shape (PRD §2).
+- **Special-category data under GDPR Art. 9.** resume_text + call_log + email_log routinely contain health (accommodation requests), union, religion. Higher legal bar — Art. 9(2) explicit consent required. Phase 1 §6 mentions these tables exist but doesn't flag the special-category exposure. **Action:** call out in §6.
+- **DPIA / ROPA / DPA inventory.** None of these documents referenced. Some may exist outside code; Phase 1 should at minimum note their absence as Phase 1.5 input.
+
+**KIMI unique:**
+- **Sequential `CAND-NNNNNN` IDs enable enumeration attacks.** Predictable IDs let an attacker scan the candidate space. Security finding Phase 1 didn't flag.
+- **Langfuse retention config unaudited.** Default trace retention is 30 days in some versions, indefinite in others. Directly impacts RTBF analysis. **Action:** check live Langfuse config.
+- **CCPA "de-identified in place" may be faster than crypto-shred.** Replacing PII with `REDACTED-{hash}` while preserving log structure may satisfy CPRA's de-identified exception. Worth considering vs. crypto-shred.
+- **Bun MCP server is the cross-runtime bridge — likely COMMON-MODE failure.** Phase 1 framed Go side as "mirrors Rust pattern" implying independent failure; it's likely shared infrastructure failure. **Action:** add to §4 — common-mode reframe.
+- **Cryptographic integrity for the audit trail itself.** Merkle trees / signed logs / chain-of-custody under FRE 901. Opposing expert can challenge admissibility without this. **Action:** add to PRD §2 audit response shape — integrity-signed.
+- **HIPAA Business Associate Agreement scoping not done.** If any candidate is healthcare-vertical, BAA analysis required.
+- **`get_candidate` SELECT * has 40+ load-bearing columns.** §9.1 "1-2 hours" estimate for view rewrite is irresponsible without scoping downstream consumers. **Action:** revise §9.1 estimate; flag dependency analysis as prerequisite.
+
+**GEMINI unique:**
+- **Data residency — JSONL on US box.** If any candidate is EU resident, GDPR violation without SCC/DPF. Phase 1 didn't ask whether IL+IN target market includes EU residents (probably not, but staffing-co clients sometimes have international placements).
+- **"Culture fit" reasoning strings as discrimination proxies.** Common LLM-generated phrases ("not a culture fit," "communication concerns," "team chemistry") often correlate with protected-attribute discrimination. Phase 1 didn't audit the actual reasoning text in `outcomes.jsonl` for these phrases.
+- **Comparator-pool snapshot for every fill.** Need to capture WHO COULD HAVE BEEN PICKED, not just who was. **Action:** PRD §2 audit response shape needs `comparator_pool` field per decision.
+
+### Revised §9 — recommended next moves (reordered by scrum convergence)
+
+1. **(NEW PRIORITY 1, was P3) Sample state.json + Langfuse content** — confirm/refute the matrix-indexer subject-clean claim and quantify Langfuse PII exposure. Cheapest move that resolves the over-claim AND informs the boundary-leak fix.
+2. **(NEW PRIORITY 2, was P1) Defense-layer enforcement at SQL template level** — rewrite tool registry to use `_safe` views. **Estimate revised UPWARD per kimi**: scope `get_candidate` SELECT * downstream consumers first; estimate 4-8 hours including the existence-check resolution path through the (not-yet-built) identity service. Stop-gap until then: add LLM-attribution flag to queryd, refuse `FROM candidates `/`FROM workers_500k ` queries that originate from tool dispatch.
+3. **(NEW) Langfuse boundary audit + redaction** — sample retention config, check DPA status, scope a redaction/sampling layer that strips PII from message arrays before the Langfuse POST. This is the boundary-crossing leak — fix BEFORE chasing internal sinks.
+4. **(NEW) Subject_id top-level promotion to all JSONL writers** — single architectural change spanning Rust + Go session_log + observerd event writer + outcomes/corrections appenders. Makes subject-correlation queries defensible (no more grepping natural language strings).
+5. **(was P4) Schema-audit for protected-attribute proxies** — extend to include "culture fit"-shaped reasoning text in outcomes.jsonl + the comparator-pool retention requirement.
+6. **(NEW) BIPA-specific audit of workers_500k schema** — explicit photo/video/biometric column inventory before any production deployment in IL.
+7. **(NEW) Operational discovery** — DPIA, ROPA, DPA inventory, SCC for cross-border, Langfuse retention config. Out-of-code-walk; needs J + counsel input.
+
+### What I'm walking back
+
+§1F TL;DR claim "matrix indexer is good for audit defensibility" — per all 3 reviewers, this is over-claimed without §8.1 verification. The correct frame is "fingerprint structure is subject-agnostic by design; trace body content unverified — treat as suspected PII sink until sampled."
+
+§9 ordering — view-routing was P1; per all 3 reviewers, Langfuse boundary should be P3 in front of it. View-routing is the source-side fix; Langfuse is the boundary-crossing fix; both matter, do them in BOUNDARY-FIRST order.
+
+§9.1 effort estimate — kimi's "irresponsible without dependency scoping" critique is right. Revised UP to 4-8 hours.
+
+---
+
+## Change log
+
+- 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`.
+- 2026-05-03 — §10 added: cross-lineage scrum review of the discovery doc (opus + kimi + gemini). 5 convergent findings (matrix-indexer over-claim, Langfuse first-priority, comparator pool gap, BIPA understated, subject_id top-level promotion). Plus single-reviewer high-value catches. §9 reordered.

docs/AUDIT_TRAIL_PRD.md

@@ -0,0 +1,374 @@
+# PRD: Production-Ready Audit Trail
+
+> **⚠ OVER-SCOPED — 9-phase plan needs to shrink for local-only deployment.**
+>
+> 2026-05-03 evening: J reframed the system as local-only per PRD line 70. The 9-phase plan in §8 was sized for SaaS-tier infrastructure with cloud HSM, separate identity daemon, dual-control JWT, etc. For a single-box local deployment, audit trail can be a few hundred LOC of local writers + a signed local file, not a 17-20 day phase plan.
+>
+> **What stays valid:**
+> - The legal use case (worked example: John Martinez at Warehouse B requests audit) — this is the real problem
+> - The §10.5 jurisdictional surface (IL BIPA, IN, federal) — counsel reads this
+> - The §3 surface map: where decisions get made today (file:line evidence — see `AUDIT_PHASE_1_DISCOVERY.md`)
+> - Phase 1.6 BIPA pre-launch gates — those still apply when real photos arrive
+>
+> **What's over-scoped:**
+> - The 9-phase implementation plan (§8) — should compress to 3-4 phases for local-only
+> - The identity service design (`IDENTITY_SERVICE_DESIGN.md`) — see that doc's deprecation header
+>
+> Do NOT execute the §8 phase plan as-written. When J greenlights, draft a v2 plan sized for local single-box.
+
+**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
+
+> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation.
+
+---
+
+## 1. The legal use case (worked example)
+
+**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2.
+
+**What we must produce.** A response that proves either:
+- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR
+- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI."
+
+**What we must NOT produce.**
+- Other subjects' data (response leaks if even one other candidate's name appears)
+- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped)
+- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence
+
+**The defensibility chain.** The audit shows:
+1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into
+2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank
+3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned
+4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections)
+5. **Outcome decisions** — final fills, rejections, hand-offs
+
+For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale.
+
+---
+
+## 2. The subject audit response — output format
+
+```
+GET /audit/subject/{candidate_id}?from=D1&to=D2
+→ JSON or signed PDF (legal preference TBD)
+```
+
+**Header section:**
+- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash
+- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls.
+
+**Per-decision row schema** (shape, not exhaustive):
+```json
+{
+  "ts": "ISO-8601 UTC",
+  "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal",
+  "daemon": "gateway | validatord | observerd | matrixd | ingestd",
+  "model": "kimi-k2.6 | deepseek-v3.2 | ...",
+  "provider": "ollama_cloud | opencode | openrouter",
+  "input_features": { /* what the model SAW — sanitized per §4 */ },
+  "output": { /* what the model decided */ },
+  "rationale": "model's natural-language explanation, or rule-based justification",
+  "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree",
+  "session_id": "iterate session that produced this row"
+}
+```
+
+**Footer section:**
+- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy"
+- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited)
+
+---
+
+## 3. Surface map — where decisions happen
+
+| Decision happens at | Currently logged where | Audit-completeness gap |
+|---|---|---|
+| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk |
+| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding |
+| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation |
+| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial |
+| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified |
+| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today |
+| Iterate retry escalations | Session JSONL `attempts[]` array | OK |
+| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk |
+| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject |
+
+**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly.
+
+---
+
+## 4. PII handling rules
+
+**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces.
+
+**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it.
+
+**Protected-attribute exclusion at decision time:** the model NEVER receives:
+- Race, ethnicity, national origin
+- Sex, gender, marital status, pregnancy
+- Age, date of birth (allowed: years of experience)
+- Religion
+- Disability, genetic information
+- Veteran status (unless legally relevant for the role)
+- Sexual orientation, gender identity
+
+If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input.
+
+**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls.
+
+**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows.
+
+---
+
+## 5. Identity service — candidate_id ↔ PII mapping
+
+**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change.
+
+**Target state:**
+- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping
+- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id`
+- Identity reads require a separate auth credential held by legal-authorized operators
+- Every identity read is itself audited (log who accessed PII for which candidate when)
+- Identity service runs as its own daemon, port-isolated from the gateway
+- Cross-runtime: same identity service backs both Rust and Go
+
+**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary.
+
+---
+
+## 6. Retention policy
+
+**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA.
+
+**Target state (proposed):**
+- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s)
+- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract.
+- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed.
+- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable.
+
+**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off.
+
+---
+
+## 7. Current state vs target state
+
+| Capability | Today | Production-ready target | Gap |
+|---|---|---|---|
+| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service |
+| Identity service | none | separate daemon, audited reads | Real — build new |
+| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new |
+| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument |
+| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk |
+| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial |
+| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major |
+| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk |
+| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build |
+| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build |
+| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set |
+
+---
+
+## 8. Implementation phases (proposed sequence)
+
+Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds.
+
+### Phase 1 — Discovery walk (read-only, ~3-4 hours)
+Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references.
+
+**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells.
+
+### Phase 2 — Identity service design (design doc, ~2 hours)
+Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.**
+
+**Exit:** J approves the design.
+
+### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours)
+Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong.
+
+**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued.
+
+### Phase 4 — Subject tagging across substrates
+Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror.
+
+**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end).
+
+### Phase 5 — Identity service build
+Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly.
+
+**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject.
+
+### Phase 6 — Protected-attribute boundary enforcement
+Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing.
+
+**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping.
+
+### Phase 7 — Retention + right-to-be-forgotten
+Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint.
+
+**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows.
+
+### Phase 8 — Legal-format export + signing
+Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow.
+
+**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies.
+
+### Phase 9 — End-to-end discrimination defense rehearsal
+Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them.
+
+**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness.
+
+---
+
+## 9. Cross-runtime requirement
+
+**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5.
+
+The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version).
+
+---
+
+## 10. Open questions blocking phase 2 (resolved 2026-05-03)
+
+**These were the original Phase 1 open questions. J answered the load-bearing 5 in conversation 2026-05-03; answers folded in below. Remaining items 1, 4, 6 still need J's call before Phase 2 design ships.**
+
+1. **Identity service: separate daemon vs in-process?** _Recommended: separate. **Status: pending J confirmation.**_
+2. ~~**Retention period N years?**~~ — Out of scope for now; will be set at deployment time per client contract. Default to 4-year hot retention until set.
+3. ✅ **Photos/video in scope?** **YES** (J 2026-05-03). BIPA (740 ILCS 14) applies in full. Per-violation $1k-$5k statutory damages, written consent + retention schedule mandatory. **This becomes a Phase 1.5 priority** — see §10.5/§13 for revised phase ordering.
+4. **JSON or signed PDF for legal export?** _Recommend signed JSON with PDF rendering option. **Status: pending.**_
+5. ✅ **RTBF under append-only:** Cryptographic erasure approach approved in principle (J 2026-05-03 implicit via approval of plan).
+6. **Audit endpoint auth model:** _Recommend legal-only credential separate from admin token. **Status: pending.**_
+7. ✅ **Matrix indexer subject-awareness:** Per scrum review (`AUDIT_PHASE_1_DISCOVERY.md` §10/C1), matrix-indexer is suspected PII sink (trace bodies unverified). Action: sample state.json before deciding (a) keep code-only + add PII-redact-on-write to trace bodies, OR (b) remove subject-summary text from trace bodies entirely. Decision deferred until §8.1 sampling completes.
+
+### Newly answered 2026-05-03 (J)
+
+8. ✅ **Langfuse hosting model:** **Self-hosted.** Removes the GDPR Art. 44 cross-border-transfer concern that 3/3 scrum reviewers flagged. Langfuse retention config + Postgres/ClickHouse access controls still need to be audited as part of Phase 1.5 — but the boundary stays inside J's infrastructure, which is materially better than SaaS Langfuse.
+
+9. ✅ **EU candidates in scope:** **Not currently — may need placeholder later.** Design choice: build the identity-service interface to be EU-compatible (DPIA-shaped fields, lawful-basis tracking, SCC-ready transfer mechanism slots) but DO NOT gate Phase 2 on EU compliance. Phase 2 ships IL+IN-shaped; EU additions are a follow-up phase.
+
+10. ✅ **Healthcare vertical / HIPAA:** **Same framework — yes.** Healthcare staffing IS in scope. PHI in `resume_text`, `communications`, and `call_log` is realistic. **Implications:**
+    - Business Associate Agreement (BAA) required with any third-party model provider that processes content from healthcare-vertical staffing requests
+    - opencode + ollama_cloud + openrouter (per PR #13 routing) are external — BAAs needed OR healthcare requests must route to local-only models (Ollama on-box)
+    - PHI redaction at the gateway boundary becomes mandatory before the model call leaves the box, OR the model call must stay on-box for healthcare requests
+    - Vertical detection at the gateway boundary becomes a Phase 2 requirement
+
+11. ✅ **Training / RAG re-runs may use historical outcomes:** **Yes — design as if it WILL.** Implications:
+    - `outcomes.jsonl` and `overseer_corrections.jsonl` cannot remain raw-PII forever — anything that lands in a training corpus or RAG re-index becomes ungeneratable to delete (PII in model weights)
+    - Phase 2 design must include a "training-safe export" pipeline that strips PII from outcomes before feeding to any training/RAG path
+    - Crypto-erasure of historical outcomes becomes load-bearing — if a candidate exercises RTBF and their data already trained a model, we must be able to evidence "the source was destroyed; the model retains it indistinguishably from synthetic patterns"
+
+### Effect on the §8 phase plan
+
+The user-confirmed answers shift priorities. Revised ordering (incorporating scrum-driven priority changes from `AUDIT_PHASE_1_DISCOVERY.md` §10):
+
+- **Phase 1.5 (NEW)** — BIPA-specific photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample. Lands BEFORE Phase 2 design starts.
+- **Phase 2 (identity service design)** — now must include EU-placeholder fields, vertical-detection (healthcare flag), training-safe export interface, BIPA consent + retention metadata
+- **Phase 3** (audit endpoint skeleton) — unchanged
+- **Phase 4** (subject tagging) — must include healthcare-vertical routing decision at gateway boundary
+- **Phase 5** (identity service build) — must include BIPA-compliant biometric metadata table
+- **Phase 6** (protected-attribute boundary) — must include PHI redaction for healthcare-vertical requests
+- **Phase 7** (retention + RTBF) — must include training-safe export evidence chain
+- **Phase 8** (legal export) — unchanged
+- **Phase 9** (rehearsal) — must include both EEOC discrimination scenario AND BIPA biometric scenario AND healthcare PHI breach scenario
+
+---
+
+## 10.5 Jurisdictional surface (IL + IN)
+
+> **⚠ Not legal advice.** This is a research-grade checklist for J to take into a conversation with actual employment + privacy counsel. The system is targeting **Chicago (Illinois)** and **Indiana** placements per 2026-05-03 conversation. Counsel needs to verify what currently applies, what's pending, and whether case law has moved any of these in 2026. **Verify with counsel before claiming compliance with any item below.**
+
+### Federal layer (always applies)
+
+| Statute / framework | Relevance to this system |
+|---|---|
+| Title VII (Civil Rights Act) | Bans discrimination on race, color, religion, sex, national origin in hiring. Discrimination claim defense is the worked example in §1. |
+| ADEA (Age Discrimination in Employment) | Bans age-based discrimination for workers 40+. DOB must be excluded from features per §4. |
+| ADA (Americans with Disabilities Act) | Bans disability discrimination + requires reasonable accommodation. Disability-inferring features (gait, photo features, medical history) need exclusion. |
+| EEOC enforcement | Receives complaints, issues right-to-sue. Audit response per §2 is what defends in EEOC investigation. |
+| OFCCP | Applies if our staffing client serves federal contractors. Adds affirmative-action recordkeeping on top of EEOC. |
+| FCRA (Fair Credit Reporting Act) | Triggers if background checks are performed. Pre-adverse-action notice + dispute process needed. |
+| Section 1981 | Race-based contract discrimination — staffing is contract relationship. |
+
+### Illinois-specific (Chicago jurisdiction)
+
+| Statute | What | What we need |
+|---|---|---|
+| **BIPA** (Biometric Information Privacy Act, 740 ILCS 14) | Bans collection of biometric identifiers (face geometry, fingerprints, voiceprints) without informed written consent + retention schedule. Penalties: $1,000-$5,000 per violation per person. **Class actions are common and aggressive.** | If we use candidate photos for any feature (face match, headshot rendering, photo-derived attributes), BIPA almost certainly applies. The headshot pool we generate (per CLAUDE.md commit `5d93a71` area) needs careful review — synthetic faces are probably OK; real candidate photos are NOT without explicit BIPA-compliant consent. **Counsel must review.** |
+| **Illinois AI Video Interview Act** (820 ILCS 42) | If AI analyzes recorded video interviews, employer must disclose AI use, obtain consent, provide explanation of how AI works, and limit who can review the video. | If we ever ingest video, this applies. Currently we don't, but worth flagging to counsel as a "what if we add this in 12 months" boundary. |
+| **Illinois Human Rights Act** (775 ILCS 5) | Broader than federal Title VII — adds protected classes including arrest record, military status, marital status, order of protection, citizenship status (in some cases), unfavorable military discharge. | Protected attribute exclusion list in §4 needs expanding to cover IL-specific classes. |
+| **Personal Information Protection Act** (PIPA, 815 ILCS 530) | Breach notification — must notify Illinois residents whose unencrypted PII was breached. | If identity service or workers parquet is breached, notification clock starts. Need incident response runbook. |
+| **Illinois Day and Temporary Labor Services Act** (820 ILCS 175) | Specific to staffing/temporary services industry. Includes equal-pay-for-equal-work + record-keeping requirements + worker notification. | Highly relevant — applies directly to staffing-company clients. Audit retention may interact with these recordkeeping requirements. |
+| **Workplace Transparency Act** | Restrictions on non-disclosure agreements re: harassment/discrimination | Tangential but worth noting. |
+| **City of Chicago Human Rights Ordinance** (Title 6 Chicago Municipal Code) | Adds protected classes beyond IHRA (source of income, parental status, military discharge status, credit history). | Chicago-specific protected attributes list. |
+| **Cook County Human Rights Ordinance** | Similar additions county-wide. | Chicago is in Cook County so this stacks. |
+| **Possible: AI hiring transparency** | Several states/cities have proposed/passed laws modeled on NYC Local Law 144 (annual bias audit + candidate notification). I do not know whether IL or Chicago has such a law on the books as of 2026-01 cutoff. | **Counsel must check current state.** If it exists, we need annual bias audit reports (which IS what this PRD is building toward, but the report format may have specific requirements). |
+
+### Indiana-specific
+
+| Statute | What | What we need |
+|---|---|---|
+| **Indiana Data Breach Disclosure** (IC 24-4.9) | Breach notification within "without unreasonable delay" | Same incident response runbook as IL PIPA. |
+| **Indiana Civil Rights Law** (IC 22-9) | State-level employment discrimination | Largely tracks federal Title VII, fewer expansions than IL. |
+| **Indiana Genetic Information Privacy Act** | Bans use of genetic info in employment | Already in §4 protected list. |
+| **General observation** | Indiana is generally less aggressive than Illinois on AI/employment regulation as of cutoff. | The IL bar is higher — if we satisfy IL, IN typically follows. **Counsel must confirm this isn't backwards.** |
+
+### Cross-cutting (security frameworks for SaaS sales)
+
+These aren't laws but are commonly required by enterprise customers (including staffing clients) before sale.
+
+| Framework | What | Relevance |
+|---|---|---|
+| **SOC 2 Type II** | Auditor attestation of operating effectiveness over 6-12 months across Trust Service Criteria (Security, Availability, Processing Integrity, Confidentiality, Privacy). | The Privacy criterion overlaps heavily with this PRD. Privacy + Security are the two load-bearing TSCs. Effort to first Type II report: 6-9 months. Type I (point-in-time) is faster (weeks) but enterprise buyers usually want Type II. |
+| **SOC 3** | Public-facing summary of SOC 2 (no detailed control descriptions). | Nice-to-have for marketing but the staffing client will want SOC 2 Type II report under NDA. |
+| **HIPAA** | Healthcare data protection. | Triggers ONLY if staffing places workers into healthcare roles where they handle PHI. Currently not in scope per CLAUDE.md. **Confirm scoping with J.** |
+| **PCI DSS** | Payment card data | Not currently in scope. |
+| **ISO 27001** | International information security management | Alternative to SOC 2; more common in EU. Probably unnecessary for IL/IN-only deployments. |
+
+### What this means for phase ordering
+
+The 9-phase plan in §8 is technically correct but may need re-ordering once counsel weighs in:
+
+- **BIPA risk on photos** is so high and so aggressive that if we use real candidate photos *anywhere*, that may need to be the FIRST thing we resolve — before the audit-trail work starts. Class-action exposure is enormous.
+- **SOC 2 Type II prep** runs in parallel with this work, not after. If the staffing client says "show us your SOC 2 report" we need to have started the engagement weeks/months before.
+- **Day and Temporary Labor Services Act** may impose recordkeeping that interacts with our retention SLA (§6) — counsel may say "no, retention has to be N years for THIS reason, not your defaulted 4."
+
+### Open questions for counsel (one ask)
+
+1. Does the staffing client have an existing SOC 2 report we leverage, or do we need our own?
+2. Are we using any real candidate photos? If yes, is BIPA consent in place?
+3. Does Illinois have an AI hiring transparency law on the books in 2026? If yes, what does the bias audit report need to look like?
+4. What's the IL Day and Temporary Labor Services Act recordkeeping retention period? Does it interact with our 4-year proposed SLA?
+5. Are background checks performed? If yes, do we need FCRA pre-adverse-action workflow integration?
+6. Any healthcare placements? (HIPAA scoping)
+7. Is the staffing client a federal contractor? (OFCCP scoping)
+
+Counsel's answers shape whether the §8 phase plan ships as-is or needs reordering.
+
+---
+
+## 11. What this PRD is NOT
+
+- Not a contract with the staffing client. That document needs lawyers and signs after this is built.
+- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC/BIPA/etc — passing actual certification is its own project.
+- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made.
+- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires.
+- **Not legal advice.** The §10.5 jurisdictional surface is a research-grade checklist, NOT counsel's analysis. Verify everything with actual employment + privacy counsel licensed in IL + IN before claiming compliance with anything in this document.
+
+---
+
+## 12. Appendix — terms
+
+- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`.
+- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.).
+- **Audit row** — one record in the audit response per decision, with the schema in §2.
+- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes.
+- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list.
+- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation).
+- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth.
+- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it.
+- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs.
+
+---
+
+## Change log
+
+- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker.

docs/DECISIONS.md

@@ -104,3 +104,8 @@
 **Date:** 2026-04-24
 **Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec<SemanticFlag>` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec<TypeHint>` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec<BugFingerprint>` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002.
 **Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon.
+
+## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly
+**Date:** 2026-05-02
+**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call.
+**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running.

docs/IDENTITY_SERVICE_DESIGN.md

@@ -0,0 +1,584 @@
+# Identity Service — Phase 2 Design (v2 — post-scrum revisions)
+
+> **⚠ OVER-SCOPED FOR LOCAL-ONLY DEPLOYMENT — needs simpler rewrite before implementation.**
+>
+> 2026-05-03 evening: J reframed the system as local-only per PRD line 70 ("Everything runs locally — no cloud APIs"). This document was drafted assuming SaaS-tier infrastructure (HashiCorp Vault, AWS KMS, S3 Object Lock, dual-control JWT split-secret ceremony, mTLS CA, separate Postgres database). For J's local-only single-box deployment serving IL+IN staffing, the audit trail can be MUCH smaller: local SQLite or Postgres, local key file, local HMAC chain to an append-only JSONL.
+>
+> The discovery findings in `AUDIT_PHASE_1_DISCOVERY.md` and `AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md` remain valid (PII flow paths, BIPA exposure, etc.). The PROBLEM is real. This DOC's solution shape is wrong for the deployment.
+>
+> Do NOT implement this document as-written. When J greenlights audit-trail work, draft a v3 that's local-only sized (~3-5 days, not 17-20).
+>
+> See `STATE_OF_PLAY.md` "PRD line 70 is load-bearing" entry for the binding direction.
+
+**Status:** Draft v2 — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
+**Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md)
+
+> **v2 history.** v1 (2026-05-03 morning) was scrummed across opus + kimi + gemini lineages. 3/3 reviewers converged on 6 critical issues — wrong migration order, master key + legal token "security theater," BIPA `inferred_existing` prima facie violation, healthcare default in wrong direction, Merkle chain without external anchor. v2 incorporates those changes. Full scrum reviews preserved at `/tmp/identity_scrum/{opus,kimi,gemini}_review.md`. Diff between v1 and v2 captured in §11 change log.
+
+> **Why this exists.** Phase 1 + 1.5 confirmed today's substrate has no separation between candidate_id and PII. Both live in `workers_500k.parquet`. No per-access audit, no consent gate, no retention enforcement. This document specifies the new identity service that holds the candidate_id ↔ PII mapping, gates every PII read, audits every access, and serves as the single legal-attestable boundary between PII and the rest of the system.
+
+> **Prerequisite:** **Phase 1.6 (BIPA pre-launch gates) MUST ship before any identityd backfill begins.** Per kimi+gemini scrum, backfilling with `consent_status='inferred_existing'` is a BIPA §15 prima facie violation. Phase 1.6 establishes the consent template + retention schedule + employee training that turns backfill from "inferred" into "pending_backfill_review with documented escalation path."
+
+---
+
+## 1. Scope and non-goals
+
+### In scope
+- Single source of truth for `candidate_id ↔ PII` mapping
+- Per-PII-access audit log (who/what/when/why)
+- Consent + retention metadata (BIPA + IL Day and Temporary Labor Services Act + healthcare PHI)
+- Legal-tier access — short-lived JWT with **split-secret dual-control issuance** (NOT a static file token, per all 3 reviewers)
+- Healthcare-vertical routing with **fail-closed default** (`unknown` treated as healthcare until classified, per opus+gemini)
+- EU-compatible interface — fields exist, enforcement is per-subject (no system-wide flag flip needed)
+- Hardware-backed master key — **HashiCorp Vault Transit OR AWS KMS** for v1 (NOT a sealed file, per all 3 reviewers; sealed-file v1 was "obfuscation in any defensible sense" — kimi)
+- Signed-JSON audit response with PDF render path (PDF deferred; JSON ships first per opus)
+- External-anchor for audit-log integrity (S3 Object Lock + signed timestamps; NOT just Postgres Merkle, per all 3 reviewers)
+- mTLS or Unix Domain Socket between gateway ↔ identityd (per kimi — "port-isolated is theater without authenticated channel")
+- Field-level authorization on `/subjects/{id}` (per-purpose field allowlists, per kimi)
+- Synchronous cache invalidation hook for RTBF (per opus — even if Phase 7 is the full RTBF build)
+
+### Out of scope (for this phase)
+- The `/audit/subject/{id}` endpoint itself (Phase 3)
+- Full subject-tagging across other substrates (Phase 4)
+- Right-to-be-forgotten implementation full-flow (Phase 7)
+- Training-safe export (deferred from Phase 2 per opus — "not on critical path for audit-trail defense")
+- BIPA pre-launch gates content (Phase 1.6 — separate doc)
+
+### Architecturally simplified from v1 (per scrum)
+- **Per-row encryption keys** deferred to Phase 7. v2 uses a single Data Encryption Key (DEK) wrapped under HSM-backed Key Encryption Key (KEK), with per-subject **ciphertext-deletion** as the v2 erasure mechanism. This matches kimi's recommendation: "A single data-encryption key with HSM-backed rotation, plus per-subject deletion of ciphertext, achieves equivalent practical erasure with far less complexity." Cryptographic-erasure via per-subject keys becomes a Phase 7 enhancement.
+- **EU placeholder fields** kept (per J 2026-05-03), but enforcement code is genuinely no-op until first eu_resident=true subject — no code paths run, no schema migrations needed when EU comes online (just per-subject field population).
+- **Merkle chaining** narrowed (per gemini): full Merkle tree only for legal-tier and consent/erasure events. Standard `gateway_lookup` events get a simpler signed-HMAC chain. Both anchored externally.
+
+---
+
+## 2. Architectural shape
+
+### 2.1 — Process model + transport
+
+Per J 2026-05-03 + kimi scrum: separate daemon, mTLS-mandatory transport. Single Go implementation, both runtimes call it.
+
+| Property | Value |
+|---|---|
+| Name | `identityd` |
+| Port | `:3225` (single port, not dual; per kimi simplification — runtime-agnostic routing) |
+| Implementation | Go |
+| Storage | Postgres in **isolated database** (per J answer, confirmed). Schema-level isolation; no shared schemas with Langfuse or other lakehouse storage. |
+| Transport gateway↔identityd | **mTLS mandatory.** Self-signed CA managed by identityd at startup; gateway clients have their own client certs. Plain HTTP is rejected at the listener. |
+| Encryption-at-rest | **Single DEK wrapped under KEK in HashiCorp Vault Transit** (recommended) OR AWS KMS. Master key NEVER on disk. v1 of identityd refuses to start if KEK is unreachable. |
+| Audit-log signing | **Separate Ed25519 keypair from the master encryption key.** Signing key in Vault Transit (signing backend) OR a separate sealed-secret file with strict rotation procedure. |
+| External anchor for audit chain | **S3 Object Lock** (compliance mode, 7-year retention) holding periodic Merkle root commitments. Daemon writes hourly + on legal-tier event. |
+| Backup | Postgres standard backup; KEK backup separate (different storage, different access). Crypto-erasure model only works if these are not co-located. |
+| Rate limiting | Per-token QPS + daily-volume caps. Default legal-token: 100 lookups/day + 10K daily volume; alerting at 50%. Configurable per-token. |
+| Token revocation | `token_revocations` table checked on every legal-tier auth. Revocation propagates within 60s (TTL on cache). |
+
+### 2.2 — Schema (Postgres DDL, v2)
+
+```sql
+-- Subject record. PII columns are ciphertext under the DEK.
+CREATE TABLE subjects (
+    candidate_id    TEXT PRIMARY KEY,           -- UUID v7, NOT sequential (per kimi enumeration concern)
+    -- Encrypted PII fields. AES-256-GCM under DEK. NULL = not collected.
+    name_ct         BYTEA,
+    email_ct        BYTEA,
+    phone_ct        BYTEA,
+    address_ct      BYTEA,
+    ssn_ct          BYTEA,
+    dob_ct          BYTEA,
+    -- DEK version this subject was encrypted under. KEK rotates; DEK rotates per
+    -- KEK rotation. New DEK version = subjects re-encrypted in background sweep.
+    dek_version     INT NOT NULL,
+    -- Lawful basis + consent metadata
+    consent_status  TEXT NOT NULL CHECK (consent_status IN (
+        'pending_backfill_review',  -- backfill default; no biometric/PHI use until reviewed
+        'pending_first_contact',     -- new subject, awaiting consent UX
+        'given',                     -- explicit consent recorded
+        'withdrawn',                 -- subject revoked
+        'expired'                    -- consent timeout
+    )),
+    consent_version           TEXT,             -- references published consent template version
+    consent_given_at          TIMESTAMPTZ,
+    consent_withdrawn_at      TIMESTAMPTZ,
+    -- BIPA-specific fields (Phase 1.5 §1E + Phase 1.6 prerequisite)
+    biometric_consent_status  TEXT NOT NULL DEFAULT 'never_collected' CHECK (biometric_consent_status IN (
+        'never_collected', 'pending', 'given', 'withdrawn', 'expired'
+    )),
+    biometric_retention_until TIMESTAMPTZ,      -- BIPA: max 3 years from last interaction
+    -- Vertical detection — drives healthcare PHI routing.
+    -- DEFAULT 'unknown' (per opus+gemini scrum) — fail-closed routing treats
+    -- unknown as healthcare-equivalent until reclassified.
+    vertical                  TEXT NOT NULL DEFAULT 'unknown' CHECK (vertical IN (
+        'unknown', 'general', 'healthcare', 'finance', 'other'
+    )),
+    -- Per-vertical retention period. Drives the daily erasure sweep.
+    -- (Per opus: this was missing in v1. Required for BIPA's 3-year-from-
+    -- last-interaction rule which differs from generic retention.)
+    retention_period_days     INT NOT NULL,
+    -- EU-placeholder fields. Enforcement is per-subject; nothing runs
+    -- until a row has eu_resident=true AND lawful_basis IS NOT NULL.
+    eu_resident       BOOLEAN NOT NULL DEFAULT FALSE,
+    lawful_basis      TEXT,                     -- GDPR Art. 6 basis when eu_resident=true
+    transfer_mechanism TEXT,                    -- SCC, DPF, BCR — populated when EU comes online
+    -- Standard audit columns
+    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
+    last_interaction  TIMESTAMPTZ,              -- drives retention sweep
+    -- Crypto-erasure state (v2: ciphertext deletion, NOT key destruction)
+    erased_at         TIMESTAMPTZ,
+    erasure_reason    TEXT
+);
+
+-- Append-only access log. EVERY PII read writes a row.
+-- Field-level: fields_accessed records WHICH fields the caller resolved.
+-- purpose enforced against an allowlist per-purpose-token.
+CREATE TABLE pii_access_log (
+    access_id       BIGSERIAL PRIMARY KEY,
+    candidate_id    TEXT NOT NULL,
+    accessed_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
+    accessor_kind   TEXT NOT NULL,              -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve'
+    accessor_id     TEXT NOT NULL,              -- daemon name + caller token-hash; never raw token
+    purpose_token   TEXT NOT NULL,              -- opaque token; resolved through purpose_definitions table
+    fields_accessed TEXT[] NOT NULL,
+    request_trace_id TEXT,
+    chain_kind      TEXT NOT NULL CHECK (chain_kind IN ('hmac', 'merkle')),  -- HMAC for standard, Merkle for legal/consent/erasure
+    integrity_hash  TEXT NOT NULL,
+    -- Per opus security finding: this row's existence is itself sensitive
+    -- ("candidate X is under legal review" leaks via purpose). Read access
+    -- to this table requires legal-tier auth or scoped per-subject auth.
+    is_legal_tier_event BOOLEAN NOT NULL DEFAULT FALSE
+);
+
+-- Purpose definitions — enforces field-level authorization (per kimi).
+-- A given purpose_token can only request fields in its allowlist.
+CREATE TABLE purpose_definitions (
+    purpose_token   TEXT PRIMARY KEY,           -- e.g. 'fill_validation', 'audit_subject_response'
+    description     TEXT NOT NULL,
+    allowed_fields  TEXT[] NOT NULL,            -- e.g. ARRAY['name'] for fill_validation
+    auth_tier       TEXT NOT NULL CHECK (auth_tier IN ('service', 'legal')),
+    rate_limit_qps  INT,
+    daily_volume_cap INT
+);
+
+-- Token revocation list. Checked on every auth, cached 60s.
+CREATE TABLE token_revocations (
+    token_hash      TEXT PRIMARY KEY,
+    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+    revoked_by      TEXT NOT NULL,
+    reason          TEXT
+);
+
+-- Consent template versioning. Hash references, not embedded text per row.
+CREATE TABLE consent_versions (
+    version         TEXT PRIMARY KEY,
+    effective_at    TIMESTAMPTZ NOT NULL,
+    superseded_at   TIMESTAMPTZ,
+    template_hash   TEXT NOT NULL,              -- SHA256 of the canonical template
+    template_path   TEXT NOT NULL               -- where the canonical text lives (e.g. data/_consent/v3-2026-04-15.md)
+);
+
+-- External anchor checkpoints. Periodically committed to S3 Object Lock.
+CREATE TABLE anchor_commits (
+    commit_id       BIGSERIAL PRIMARY KEY,
+    chain_kind      TEXT NOT NULL,              -- 'hmac' or 'merkle'
+    checkpoint_at   TIMESTAMPTZ NOT NULL,
+    last_access_id  BIGINT NOT NULL,            -- max access_id covered by this checkpoint
+    root_hash       TEXT NOT NULL,
+    s3_object_uri   TEXT NOT NULL,              -- s3://anchor-bucket/identityd/...
+    s3_lock_until   TIMESTAMPTZ NOT NULL        -- compliance-mode retention end
+);
+```
+
+### 2.3 — HTTP surface (v2)
+
+All under `/v1/identity/`. mTLS + token required on every endpoint.
+
+| Method + Path | Purpose | Auth tier | Notes |
+|---|---|---|---|
+| `POST /v1/identity/subjects` | Create new subject. Body: PII fields + `purpose_token`. Returns: candidate_id (UUID v7). | service | Validates purpose_token allowlist matches submitted fields |
+| `GET /v1/identity/subjects/{candidate_id}?purpose={token}&fields={list}` | Resolve PII for candidate. | service | **Field-level enforcement**: returned fields ⊆ purpose_token's allowed_fields. Mismatch → 403. Logs access row. |
+| `GET /v1/identity/subjects/{candidate_id}/vertical` | **Minimal-disclosure** vertical lookup (per kimi HIPAA minimum-necessary). Returns only `{vertical, consent_status}`. | service | Used by gateway routing. Cheaper access-log row. |
+| `GET /v1/identity/subjects/{candidate_id}/full` | Complete subject record + audit summary. | **legal** | Short-lived JWT only. Logged with `is_legal_tier_event=true`. Triggers real-time notification to designated counsel + J. |
+| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn with version. | service | |
+| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure. Idempotent. | **legal** | Synchronously triggers cache invalidation hooks (per opus). |
+| `GET /v1/identity/access_log/{candidate_id}` | Per-subject access log for audit response. | **legal** | |
+| `POST /v1/identity/auth/legal_token` | Issue short-lived JWT (max 24h). Requires **dual-control attestation**: J's signed nonce + counsel's signed nonce, both verified against pre-registered public keys. | dual-control | Per all 3 reviewers — replaces v1's static file approach |
+| `GET /v1/identity/health` | Liveness | none | Returns only `{status: "ok"}` — no version, no schema info |
+| `(no public training-safe-export endpoint in v2)` | — | — | Deferred per opus — not on critical path |
+
+### 2.4 — Auth model (v2 — split-secret + short-lived)
+
+**Service-tier auth** (gateway → identityd for routine PII resolution):
+- mTLS client cert per gateway (Rust :3100 + Go :4110 each get their own cert)
+- Bearer token in `Authorization` header (long-lived — 90d), rotated per ops runbook
+
+**Legal-tier auth** (per-request, short-lived JWT, dual-control issuance):
+- J holds private key A
+- Designated outside counsel holds private key B
+- `POST /v1/identity/auth/legal_token` requires:
+  - HTTP body: `{purpose, ttl_seconds, requested_fields, signature_a, signature_b, nonce_a, nonce_b}`
+  - identityd verifies both signatures against pre-registered A and B public keys
+  - On success: emits a JWT signed by identityd, valid for `ttl_seconds` (max 24h), scoped to `purpose`+`fields`
+- The JWT:
+  - Cannot be issued without BOTH A and B signatures (per gemini "split-secret startup ceremony" pattern, applied per-token)
+  - Carries scope (purpose + fields) inside its claims; identityd enforces at use time
+  - Per-token rate limit + daily cap recorded on issuance
+  - Revocation: token_hash added to `token_revocations`; identityd refuses on next check (60s cache TTL)
+
+**Why this matters:**
+- v1's "static file mode 0400" was security theater per all 3 reviewers
+- A leaked legal token v1 = unbounded exfiltration window until rotation
+- v2's leaked JWT = bounded by TTL (max 24h) AND revocable in <60s
+- Single-actor compromise (J's machine OR counsel's machine, not both) cannot mint legal tokens
+
+---
+
+## 3. Integration with the rest of the substrate
+
+### 3.1 — Gateway changes
+
+When gateway needs PII for a fill scenario:
+
+1. Gateway has only `candidate_id` (post-Phase-1.5 view-routing fix)
+2. Gateway calls `GET /v1/identity/subjects/{candidate_id}/vertical` first if it might be healthcare-vertical-sensitive routing
+3. Gateway calls `GET /v1/identity/subjects/{candidate_id}?purpose=fill_validation&fields=name`
+4. identityd validates: purpose_token=`fill_validation` allows `[name]` only; if request asks for `[name,ssn]` → 403
+5. identityd writes `pii_access_log` row, decrypts requested fields, returns
+6. Gateway uses fields, then **explicitly zeros memory** (kimi's drop-and-overwrite pattern) before returning HTTP response
+7. **No PII caching** in gateway memory beyond request lifetime
+
+**LRU embed cache** (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Phase 4 task to re-key as `(model, candidate_id, field_subset_hash)`. **Phase 2 must add the synchronous cache-purge hook** so RTBF in Phase 2 invalidates Phase 4's cache when it ships (opus finding).
+
+### 3.2 — Cross-runtime: same identityd, both gateways call it
+
+Both gateways call identityd over mTLS. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates identical PII request through both gateways produces identical access-log rows (modulo daemon-name field).
+
+### 3.3 — JSONL writer changes (subject_id top-level promotion)
+
+Per `AUDIT_PHASE_1_DISCOVERY` §10/C5:
+- All JSONL sinks (outcomes, sessions, overseer_corrections, observerd ops) gain top-level `subject_ids: [...]` field
+- `fills[*].name` replaced with `name_ref: "[REDACTED-{candidate_id}]"`
+- Authorized callers dereference via identityd
+
+### 3.4 — Langfuse boundary redaction (defense in depth, per opus TOCTOU finding)
+
+v1 design: per-request map of `subject_id → resolved_PII`, replace before Langfuse POST.
+
+**v2 adds** (per opus): outbound regex/NER pass on the Langfuse payload as defense-in-depth. The model can hallucinate names not in the resolved-PII map. The outbound NER pass catches them.
+
+```
+Gateway constructs Langfuse payload
+  → Pass 1: per-request resolved-PII map replacement
+  → Pass 2: NER scan (regex for SSN/phone shapes; named-entity model for names)
+  → Either pass detecting unredacted PII → drop the trace OR replace with [POSSIBLE-PII-DETECTED]
+  → POST to Langfuse
+```
+
+### 3.5 — Healthcare vertical routing — fail-closed default
+
+**Per opus + gemini scrum: Default is `unknown` → treated as healthcare for routing purposes.**
+
+When gateway receives a request involving any candidate:
+1. Query `/vertical` for the candidate
+2. If `vertical IN ('healthcare', 'unknown')` → route to **on-box Ollama only** (no opencode/openrouter/ollama_cloud egress)
+3. If `vertical='general'` (or other non-healthcare) → cloud routing OK
+4. If identityd unreachable → fail closed (refuse the request, return 503)
+
+**Reclassification path:** subjects can be moved from `unknown` → `general` via explicit operator action (after manual review). Subjects flagged `healthcare` stay healthcare unless explicitly downgraded.
+
+**Auto-escalation** (per kimi): if a subject's resume_text or call_log content is updated with healthcare-pattern matches (RN, BSN, hospital, MD, physician, etc.), vertical auto-escalates to `healthcare`. Never auto-de-escalates.
+
+### 3.6 — Cache invalidation hook (Phase 2 must ship even though full RTBF is Phase 7)
+
+When `POST /v1/identity/subjects/{id}/erase` fires:
+1. Mark `subjects.erased_at` and zero out ciphertext columns
+2. Write `pii_access_log` row with `purpose_token='retention_expired'` or `'rtbf_request'`
+3. **Synchronously** publish a `subject_erased` event to a pub/sub channel (Redis or Postgres LISTEN/NOTIFY)
+4. Gateway subscribes; on event, purges any in-flight cache entries for that candidate_id
+5. Eventual-consistency window: <5s between erase-call and cache-flush
+
+Without this hook, RTBF in Phase 7 would be a lie because the gateway's LRU embed cache could still hold the subject's data.
+
+---
+
+## 4. Audit response shape (Phase 3 preview, v2 — adds model-version snapshot per gemini)
+
+```json
+{
+  "schema": "audit.subject.v1",
+  "subject_token": "01926f2e-7c1b-7000-...",     // UUID v7, NOT CAND-NNNNNN
+  "request_window": { "from": "2026-01-01", "to": "2026-05-03" },
+  "generated_at": "2026-05-03T12:00:00Z",
+  "generated_by": "identityd@hostname",
+  "merkle_root": "sha256:...",                    // root hash of legal/consent/erasure events
+  "external_anchor": "s3://anchor-bucket/identityd/2026-05-03T12-00.json",  // S3 Object Lock URI
+  "signature": "ed25519:...",                     // separate signing key from encryption KEK
+  "consent": { ... },
+  "decisions": [
+    {
+      "ts": "2026-04-22T09:15:23Z",
+      "decision_kind": "fill_recommendation",
+      "daemon": "gateway",
+      "model": "kimi-k2.6",
+      "model_version_hash": "sha256:...",         // NEW per gemini — proves what model existed AT decision time
+      "provider": "ollama_cloud",
+      "trace_id": "trace-abc",
+      "session_id": "session-xyz",
+      "input_features": { ... },                  // sanitized; no protected attributes; no inferred-attribute proxies
+      "output": "...",
+      "rationale": "...",
+      "comparator_pool_size": 47,
+      "comparator_appendix_ref": "see comparator_appendix.A"
+    }
+  ],
+  "comparator_appendix": {
+    "A": {
+      "scope": "fill scenarios in window matching role X, geo Y",
+      "total_pool_size": 47,
+      "selection_rate_by_protected_class": {
+        // Aggregated; NO other subjects' identifiers leak
+        "race": { "white": 0.33, "black": 0.31, ... },
+        "gender": { "man": 0.34, "woman": 0.30, ... }
+      },
+      "four_fifths_test": "passed"  // or "concern: rate ratio 0.71"
+    }
+  },
+  "access_log": [...],
+  "footer": {
+    "completeness_attestation": "all decisions about subject_token in window per retention policy v2 are included",
+    "merkle_proof": "...",                        // proof this audit's root is in S3 anchor
+    "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0"
+  }
+}
+```
+
+---
+
+## 5. Migration path (v2 — REORDERED per all 3 reviewers)
+
+v1's order was wrong. New order, with explicit prerequisites:
+
+| Step | Action | Prerequisite |
+|---|---|---|
+| **0** | **Phase 1.6 BIPA pre-launch gates SHIPPED** — consent template published, retention schedule public, deletion procedure documented, employee training acknowledged. | (separate work — see PHASE_1_6_BIPA_GATES.md when written) |
+| **1** | Stand up identityd with **synthetic test subjects only**. KEK in vault, mTLS live, all endpoints serve. | Vault/KMS available; mTLS CA bootstrapped |
+| **2** | **Audit-parity probe** (`audit_parity.sh`) green on synthetic data. Cross-runtime equivalence verified. | Step 1 |
+| **3** | Add gateway feature flag `LH_USE_IDENTITY_SERVICE`. **Shadow-write only** — gateway writes new subjects to identityd AND continues SQL path. No reads from identityd yet. | Step 2 |
+| **4** | Run shadow-write for ≥1 week. Validate access logs, encryption correctness, write-path performance under real traffic. | Step 3 |
+| **5** | Backfill from `workers_500k.parquet`. **`consent_status='pending_backfill_review'`** for ALL existing rows. **`vertical='unknown'`** default (NOT 'general'). **`biometric_consent_status='never_collected'`** — backfill does NOT include any biometric data. | Step 0 (BIPA gates) + Step 4 |
+| **6** | Human-review queue for vertical reclassification. Subjects move from `unknown` to `general` only after explicit review. Healthcare-pattern matches auto-escalate to `healthcare` (never auto-downgrade). | Step 5 |
+| **7** | **Shadow-read** — gateway reads from BOTH SQL and identityd, compares, logs divergences, returns SQL result. Run ≥1 week. | Step 5 |
+| **8** | Feature-flag cutover — gateway reads from identityd, falls back to SQL on error, alerts on every fallback. | Step 7 |
+| **9** | Quarantine PII columns in workers_500k.parquet — only after **cryptographic attestation of identityd completeness** (Merkle proof: source row hash = identityd row hash for every candidate_id). Move PII columns to a different bucket; the candidate_id-only projection becomes operational. | Step 8 |
+
+**Key v2 changes from v1:**
+- New Step 0 (BIPA gates prerequisite) — backfill cannot start without this
+- Steps 3-4 (shadow-write before backfill) — production validates the WRITE path before data lands
+- Step 5 backfill consent: `pending_backfill_review` not `inferred_existing` (BIPA defense)
+- Step 5 vertical default: `unknown` not `general` (HIPAA fail-closed)
+- New Step 6 (human review) — vertical classification via explicit operator action
+- Step 7 added (shadow-read after backfill) — catches encoding/normalization bugs before cutover
+- Step 9 (quarantine) requires cryptographic attestation of completeness
+
+Each step is its own commit, its own gate, its own rollback path.
+
+---
+
+## 6. Cross-runtime parity probe (NEW, per scrum §6 plus opus 'oracle test' addition)
+
+`audit_parity.sh` ships in Phase 5. Asserts:
+
+1. Same PII fetch through Rust gateway and Go gateway produces identical identityd access-log rows (modulo daemon name)
+2. Crypto-erasure of test subject through Rust gateway is honored when Go gateway tries to fetch
+3. Healthcare-vertical routing decision identical across both runtimes
+4. **Oracle test** (per kimi SOC2 CC4.1 finding): probe with known-good inputs against expected outputs. A bug present in BOTH implementations must not pass the parity probe.
+5. Discrimination-proxy phrase redaction triggers as expected on adversarial test cases
+
+---
+
+## 7. What this design intentionally does NOT solve
+
+- Protected-attribute exclusion at decision time (Phase 6)
+- Pathway memory trace body redaction (Phase 4)
+- Retroactive Langfuse history scrub (separate runbook)
+- GDPR Art. 22 right to explanation full implementation (Phase 8)
+- Multi-region data residency (single-region US-Midwest by default)
+- Training-safe export (deferred — not on critical path per opus)
+
+---
+
+## 8. Open questions — RESOLVED 2026-05-03
+
+J confirmed all 6 v1 recommendations + scrum-driven changes:
+
+1. ✅ **Master key:** HashiCorp Vault Transit (recommended) OR AWS KMS. NOT a sealed file. v1's sealed-file recommendation is rejected per all 3 reviewers ("obfuscation in any defensible sense").
+2. ✅ **Postgres isolation:** identityd's own database, isolated schema.
+3. ✅ **Vertical backfill:** `'unknown'` default with fail-closed routing. NOT `'general'` per opus+gemini scrum.
+4. ✅ **Legal token:** Split-secret dual-control issuance (J + counsel both sign). Short-lived JWT, max 24h, revocable in <60s.
+5. ✅ **Crypto-erasure sweep:** Daily 03:00 UTC. (Note: v2 erasure mechanism is ciphertext deletion, not key destruction. Per-row keys deferred to Phase 7.)
+6. ✅ **EU enforcement:** Per-subject. Schema fields exist; nothing runs until first eu_resident=true subject.
+
+### Newly resolved per scrum
+
+7. ✅ **Migration order:** REORDERED per §5 above. New Step 0 (BIPA prerequisite). Shadow-write before backfill. Shadow-read before cutover.
+8. ✅ **Audit-log external anchor:** S3 Object Lock with compliance-mode 7-year retention. Hourly + on-event commits.
+9. ✅ **Audit-log signing key:** Separate Ed25519 keypair from KEK. Vault Transit signing backend OR sealed-secret with strict rotation runbook.
+10. ✅ **mTLS gateway↔identityd:** Mandatory. Self-signed CA managed by identityd at startup.
+11. ✅ **Per-row encryption keys:** Deferred to Phase 7. v2 uses single DEK with HSM-wrapped KEK + ciphertext deletion for erasure.
+12. ✅ **Field-level authorization:** purpose_definitions table enforces per-purpose field allowlists.
+13. ✅ **Synchronous cache invalidation:** pub/sub event on erase; gateway subscribes.
+14. ✅ **Outbound NER pass for Langfuse:** Defense-in-depth in addition to symbol-table replacement.
+15. ✅ **Model version hash in audit response:** Captured per decision row.
+16. ✅ **PDF render:** Deferred from Phase 2; JSON ships first.
+
+---
+
+## 9. Estimated implementation cost (revised v2)
+
+| Sub-phase | Effort | Notes |
+|---|---|---|
+| 2A — Postgres schema + Vault/KMS integration | 1.5-2 days | Includes mTLS CA bootstrap + signing-key separation |
+| 2B — identityd HTTP surface (Go) | 2-3 days | All endpoints, auth, dual-control JWT issuance, rate limiting, revocation |
+| 2C — Backfill ETL (BIPA-compliant — pending_backfill_review) | 1 day | Plus Merkle attestation of completeness |
+| 2D — Gateway integration (Rust + Go, mTLS, shadow-write phase) | 2-3 days | Per-tool migration, parity probe |
+| 2E — JSONL writer changes (subject_id top-level promotion) | 1 day | All sinks |
+| 2F — Langfuse redaction (symbol-table + NER) | 2 days | Two passes + drop-on-detect |
+| 2G — Healthcare-vertical routing (fail-closed) | 0.5 day | Plus auto-escalation pattern matcher |
+| 2H — Cache invalidation pub/sub hook | 0.5 day | Critical for Phase 7 RTBF |
+| 2I — External anchor (S3 Object Lock) | 1 day | Hourly + on-event commits |
+| 2J — Cross-runtime parity probe with oracle tests | 1 day | New probe |
+| **Total** | **~12-15 working days** | (v1 estimated 8-10; v2 added BIPA gates dependency, mTLS, dual-control JWT, NER pass, anchor, etc.) |
+
+Bigger than v1. Worth it — every addition was a 3/3-reviewer convergent finding.
+
+---
+
+## 10. The four "would not build" blockers from scrum (all addressed in v2)
+
+| # | v1 issue (3/3 reviewers) | v2 resolution |
+|---|---|---|
+| 1 | Migration order (backfill before validation) | §5 reordered; Step 0 BIPA gates prereq added |
+| 2 | Master key on disk + legal token static file | §2.1 Vault/KMS for KEK; §2.4 split-secret short-lived JWT |
+| 3 | `inferred_existing` BIPA prima facie violation | §5 Step 5 uses `pending_backfill_review`; Step 0 gates required first |
+| 4 | Healthcare default `general` (HIPAA exposure window) | §3.5 fail-closed `unknown`-as-healthcare; §5 Step 5 backfill default `unknown` |
+
+All three reviewers said "I would not build v1 as written." All four blockers are resolved in v2. Re-scrum recommended before implementation starts.
+
+---
+
+## 11. Change log (v1 → v2)
+
+| Section | v1 | v2 |
+|---|---|---|
+| Master key storage | Sealed file `/etc/lakehouse/identityd_master.key` | HashiCorp Vault Transit / AWS KMS |
+| Legal token | Static file mode 0400 | Split-secret short-lived JWT (max 24h), dual-control issuance |
+| Encryption keys | Per-row keys with master wrapping | Single DEK + HSM-wrapped KEK + ciphertext deletion (per-row deferred to Phase 7) |
+| Healthcare default | `vertical='general'` backfill | `vertical='unknown'` backfill, fail-closed routing |
+| Migration §5 order | 1: stand up, 2: backfill, 3: feature flag, 4: cutover, 5: quarantine | 0: BIPA gates, 1: stand up, 2: parity probe, 3: shadow-write, 4: shadow-write soak, 5: backfill (pending_backfill_review), 6: human vertical review, 7: shadow-read, 8: cutover, 9: quarantine with cryptographic attestation |
+| Audit-log integrity | Postgres Merkle chain only | Postgres Merkle for legal/consent/erasure + HMAC for standard + S3 Object Lock external anchor |
+| Audit-log signing key | Same as encryption key | Separate Ed25519 keypair; Vault signing backend |
+| Gateway↔identityd transport | Plain HTTP (implied) | mTLS mandatory |
+| Field authorization | Per-endpoint only | Per-purpose-token field allowlists |
+| Cache invalidation | Phase 4 / Phase 7 | Phase 2 ships pub/sub hook |
+| Langfuse redaction | Symbol-table replacement only | Symbol-table + outbound NER pass + drop-on-detect |
+| Audit response | No model version | model_version_hash per decision |
+| PDF render | Phase 2 | Deferred — JSON ships first |
+| Training-safe export | Phase 2 | Deferred — not critical path |
+| Effort estimate | 8-10 days | 12-15 days |
+
+---
+
+## 12. v3 amendments (post-second-pass scrum, 2026-05-03 evening)
+
+v2 was re-scrummed across opus + kimi + gemini. **All 3 verdict: BUILD-WITH-CHANGES.** All 4 v1 blockers verified RESOLVED. New v2 findings are tractable design fixes (not re-architecture). Folded in below. Reviews preserved at `/tmp/identity_scrum_v2/{opus,kimi,gemini}_review.md`.
+
+### Convergent v2 findings (≥2 reviewers) — must fix before implementation
+
+**v3-A1 — mTLS CA root must NOT live in identityd** (opus + gemini converge). v2 said "self-signed CA managed by identityd at startup" — if identityd is both the CA and authenticated party, compromise of identityd compromises the whole trust fabric. **v3 change:** mTLS CA root lives in **Vault PKI** (or, if Vault PKI is unavailable, an offline root with identityd as a properly-issued intermediate). identityd never has root-CA private key access.
+
+**v3-A2 — Dual-control public key registry must be tamper-evident** (opus + gemini converge). v2 said "pre-registered public keys" but didn't say where. If they live in identityd's Postgres, DB-write compromise defeats dual-control. **v3 change:** J + counsel public keys live in **Vault KV with separate access policies** (or a signed config file at `/etc/lakehouse/identityd_dual_control.yaml` with its own dual-control rotation procedure). Public-key changes themselves require dual-control attestation. Plus: `nonce_a`/`nonce_b` get **server-issued challenges** (identityd issues a fresh nonce per request; clients sign that nonce; replay-protection via 5-min nonce cache).
+
+### Single-reviewer v2 findings — also folded into v3
+
+**v3-B1 (opus): Step 8 fallback-to-SQL needs explicit time bound.** Otherwise becomes permanent dual-read debt. v3: fallback active for max 14 days post-cutover; alert on every fallback; auto-disable at 14 days regardless of fallback rate.
+
+**v3-B2 (opus): NER drop-on-detect needs alerting.** Silent observability gap if model regression starts hallucinating PII. v3: drop counter exposed as `identityd_ner_drops_total` Prometheus metric; alert on drop_rate >0.1% over rolling 1h.
+
+**v3-B3 (opus): legal-tier notification transport must be specified.** v2 said "real-time notification to designated counsel + J" without saying how. v3: notification transport is **signed Slack webhook OR signed email** (configurable per deployment); message body never contains candidate_id or PII; only `{event_kind, timestamp, accessor_kind, integrity_hash}`. Notification failure does NOT block the legal-tier action — failure is logged for follow-up but token issuance proceeds.
+
+**v3-B4 (opus): Step 6 human review queue needs SLA + volume estimate.** With 500k backfilled rows defaulting to `unknown`, all healthcare-adjacent traffic routes to on-box Ollama until reclassified. v3: budget assumption — auto-pattern-matching pre-classifies ~80% to confidently-non-healthcare; remaining 20% (~100k) stays `unknown` pending review at ~500/day operator throughput = ~7 months of queue. **Operational decision needed:** either staff the queue, accept the on-box-Ollama-only routing for ~7 months, or relax pattern matching at the cost of higher false-negative rate. Flag for J.
+
+**v3-B5 (opus): Memory zeroing in Go is non-trivial.** v2 §3.1 step 6 says "explicitly zeros memory" — Go GC-managed strings are immutable, not zeroable. v3 implementation note: PII handling code path uses `[]byte` throughout; convert to string only at the JSON serialization boundary; explicit `runtime.GC()` after request completes is theater (Go won't actually zero the slice). **Acceptance:** "best-effort zero" — overwrite the `[]byte` slice contents post-use. Document this is best-effort, not cryptographic-grade scrubbing. (Rust-side: use the `zeroize` crate which IS cryptographic-grade.)
+
+**v3-B6 (kimi): Service-tier purpose_definitions needs versioning + emergency revocation.** A misconfigured purpose with overly-broad `allowed_fields` is standing exfiltration authorization. v3 schema additions:
+
+```sql
+CREATE TABLE purpose_versions (
+    purpose_token   TEXT NOT NULL,
+    version         INT NOT NULL,
+    effective_at    TIMESTAMPTZ NOT NULL,
+    superseded_at   TIMESTAMPTZ,
+    PRIMARY KEY (purpose_token, version)
+);
+
+CREATE TABLE purpose_revocations (
+    purpose_token   TEXT PRIMARY KEY,
+    revoked_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+    revoked_by      TEXT NOT NULL,
+    reason          TEXT
+);
+```
+
+Auth path checks `purpose_revocations` on every call (not cached, OR cached <5s). Revocation is <60s without code deploy.
+
+**v3-B7 (kimi): Cache invalidation needs erasure-generation atomicity.** v2 §3.6 pub/sub is best-effort. If Redis drops the message or gateway crashes between erase + purge, cache holds stale PII. v3 change:
+- `subjects` table gains `erasure_generation INT NOT NULL DEFAULT 0`. Increments on every erase.
+- Identityd PII responses include current generation in headers
+- Gateway cache entries tagged with the generation they were filled at
+- Cache hits with `cached_generation < current_generation` are rejected, force re-fetch from identityd
+- Eventually-consistent within ONE request hop, not eventually-consistent over time
+
+**v3-B8 (kimi): Dual-control issuance needs cooling-off period to prevent bypass culture.** v3: 15-minute mandatory delay between nonce submission and token issuance, with notification to BOTH parties at delay-start AND issuance. Bypass requires both J and counsel to sign an "emergency bypass" attestation that's logged separately + escalated to a board-level reviewer.
+
+**v3-B9 (kimi): NER must have calibrated false-negative rate.** v3: maintain a synthetic adversarial test set (50 hallucinated-PII examples + 50 legitimate non-PII). NER must achieve recall ≥99.5% on the synthetic set in CI; production drop-rate >0.1% triggers re-calibration.
+
+**v3-B10 (kimi): S3 Object Lock root credentials need separate AWS account.** Same ops team holding Vault admin + AWS root = limited independence. v3: Object Lock bucket lives in **separate AWS account** with **write-only IAM role** issued to identityd. Root credentials for that account held by named external party (could be outside counsel) with quarterly attestation.
+
+**v3-B11 (kimi): biometric_consent_status='never_collected' may not satisfy BIPA infrastructure-as-notice argument.** Plaintiffs may argue presence of biometric schema fields constitutes constructive notice of intent. v3: Phase 1.6 BIPA policy doc must explicitly attest "no biometric data exists in the pre-identityd substrate" with cryptographic evidence (hash of pre-identityd schema). Counsel must bless this attestation.
+
+**v3-B12 (gemini): Backup retention window vs ciphertext-deletion erasure.** Phase 2 uses ciphertext deletion + single-DEK; if Postgres backups retain pre-erasure ciphertext + the DEK still exists, backup is recoverable. v3 erasure runbook: includes maximum backup retention period (e.g., 30 days). RTBF status reported to subject as "erased now; residual in backups expires by {date}." For full crypto-erasure-on-demand, Phase 7 per-row keys is the path.
+
+### Net v3 effort delta
+
+v2 estimate: 12-15 days. v3 amendments add:
+- mTLS CA via Vault PKI integration: +0.5 day
+- Public-key registry in Vault KV: +0.5 day
+- Server-issued nonces for dual-control: +0.25 day
+- Step 8 fallback time bound: +0.1 day
+- NER drop-rate metric + alert: +0.25 day
+- Legal-tier notification transport: +0.5 day
+- purpose_versions + purpose_revocations: +0.5 day
+- erasure_generation cache atomicity: +0.5 day
+- 15-min cooling-off: +0.25 day
+- NER calibrated test set: +0.5 day
+- Separate AWS account for Object Lock: +0.5 day
+- BIPA infrastructure-as-notice attestation: +0.25 day (mostly doc work)
+- Backup-retention erasure runbook: +0.25 day
+
+**Total v3 delta: ~5 days. Revised estimate: 17-20 days.**
+
+The cost is real. The reason it's worth paying is what those 5 days buy: a design that 3 independent senior security architects (across 3 model lineages) all said they would build. v1 said "do not build." v3 says "build."
+
+### What's v3-deferred vs v3-must-have
+
+**v3 must-have (block implementation):** v3-A1, v3-A2, v3-B1, v3-B6, v3-B7, v3-B11.
+
+**v3 should-have (ship in Phase 2 if calendar allows; otherwise Phase 5):** v3-B2, v3-B3, v3-B4 (operational decision), v3-B5 (best-effort), v3-B8, v3-B9, v3-B10, v3-B12.
+
+If schedule pressure forces a cut, the should-have items can ship in Phase 5 (identity service build completion). The must-haves cannot — they're integral to the security boundary.
+
+---
+
+## Change log
+
+- 2026-05-03 — v1 initial draft.
+- 2026-05-03 — v2 post-first-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum recommended.
+- 2026-05-03 — v3 post-second-scrum: 3/3 BUILD-WITH-CHANGES. v1 blockers verified resolved. 12 new v2 findings folded into v3 amendments (§12). Re-scrum NOT required for v3 — diminishing returns; the must-have items are concrete fixes with clear acceptance criteria.

docs/PHASES.md

@@ -349,6 +349,24 @@
   - Per-type endpoints: `/profiles/retrieval`, `/profiles/memory`, `/profiles/observer`
   - `profile_type` field on ModelProfile
   - Guard fix: automated scrumaudit.py finds real issues
+- [x] **Phase 42: Truth Layer** (2026-04-27 closure verified)
+  - `crates/truth/{lib,staffing,devops,loader}.rs`
+  - Staffing rules populated; devops scaffold by design
+  - `/v1/context` serves task_classes + rules; 37 tests green
+- [x] **Phase 43: Validation Pipeline** (2026-04-27)
+  - `crates/validator/` real validators + WorkerLookup + ParquetWorkerLookup
+  - 500K-row workers_500k.parquet loaded at gateway boot
+  - `POST /v1/validate` + `POST /v1/iterate` (the 0→85% loop)
+  - 33 validator tests green
+- [x] **Phase 44: Caller Migration** (2026-04-27)
+  - TS callers + aibridge::AiClient::new_with_gateway opt-in
+  - Vectord routed through /v1/chat for autotune + RAG
+  - scripts/check_phase44_callers.sh CI guard
+- [x] **Phase 45: Doc-Drift Detection** (2026-04-27)
+  - DocRef + doc_drift module + context7 bridge
+  - /doc_drift/check + /scan + /resolve endpoints
+  - data/_kb/doc_drift_corrections.jsonl writes
+  - boost exclusion of unreviewed drift-flagged entries
 - [ ] Fine-tuned domain models (Phase 25+)
 - [ ] Multi-node query distribution (only if ceilings bite)
 

docs/PHASE_1_6_BIPA_GATES.md

@@ -0,0 +1,334 @@
+# Phase 1.6 — BIPA Pre-Launch Gates
+
+**Status:** Draft — 2026-05-03 · **Owner:** J + outside counsel · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md), [`IDENTITY_SERVICE_DESIGN.md`](IDENTITY_SERVICE_DESIGN.md)
+
+> **Why this exists.** `IDENTITY_SERVICE_DESIGN.md` v3 §5 Step 0 names Phase 1.6 as a HARD PREREQUISITE: identityd backfill cannot start until Phase 1.6 ships. This doc specifies what Phase 1.6 contains.
+>
+> **Scope.** BIPA (740 ILCS 14) compliance gates that must be in place BEFORE the system accepts a single real candidate photo. Synthetic-data face pool can keep operating; real-photo intake CANNOT begin without these gates.
+>
+> **Authority.** This is an engineering scaffold. Sections marked `⚖ COUNSEL` need outside counsel to author the actual legally-binding text. Engineering ships the procedural gates; counsel writes the words.
+
+---
+
+## 1. The five BIPA pre-launch gates
+
+Each gate is a deliverable that must ship before real-photo intake. None is optional. Order shown is the recommended ship sequence.
+
+### Gate 1 — Public retention schedule (BIPA §15(a))
+
+**Required:** A publicly-available, written retention schedule for biometric identifiers and information.
+
+**What ships:**
+- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file
+- Linked from public privacy policy at the deployment URL
+- Specifies:
+  - Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`)
+  - Purpose of collection (identity matching for staffing operations)
+  - Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin)
+  - Destruction procedure: per Gate 5 below
+- Versioned (this is v1; future updates supersede with a new version)
+
+**⚖ COUNSEL** — write the actual schedule. Engineering provides the operational facts; counsel writes the binding language.
+
+**Engineering acceptance:** the file is committed, the public URL renders it, and identityd's `consent_versions` table references it by hash.
+
+---
+
+### Gate 2 — Informed written consent (BIPA §15(b))
+
+**Required:** Informed, written consent BEFORE any biometric collection occurs.
+
+**What ships:**
+- `docs/policies/consent/biometric_consent_template_v1.md` — public consent template
+- Versioned, hashed, referenced from identityd's `consent_versions` table
+- Must disclose, per BIPA §15(b)(1)-(3):
+  1. That biometric identifiers/information will be collected
+  2. The specific purpose for collection (and the length of term — references Gate 1)
+  3. Receipt of a written release authorizing collection
+- Consent flow at intake:
+  - Candidate sees the disclosure on a UI surface (web form / paper / digital signature)
+  - Candidate provides explicit affirmative action (signature, click-acceptance with timestamp, etc.)
+  - Identityd records `biometric_consent_status='given'` with `consent_version` reference + `consent_given_at` timestamp
+  - **Without identityd recording 'given', no biometric data flows through deepface.**
+
+**⚖ COUNSEL** — write the consent template. Recommended content (engineering view):
+- Clear language (not just legal boilerplate)
+- Specific to facial-classification (not generic biometrics)
+- Includes withdrawal procedure
+- Includes data-subject rights enumeration
+
+**Engineering acceptance:** consent gate is enforced in code at the photo-upload endpoint; identityd refuses biometric writes when `biometric_consent_status != 'given'`; pre-existing synthetic-face pool is exempt (no consent needed because no real subject).
+
+---
+
+### Gate 3 — Photo-upload endpoint with consent enforcement
+
+**Required:** Code-level enforcement that real-photo intake checks consent before processing.
+
+**What ships:**
+
+An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior:
+
+1. Caller authenticates with service-tier token
+2. Endpoint queries identityd for `subjects.biometric_consent_status`
+3. If status ≠ `'given'` → HTTP 403 with reason `"BIPA consent required before biometric processing"`
+4. If status = `'given'`:
+   a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`)
+   b. deepface tagging runs against the photo
+   c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1.
+   d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule
+   e. `pii_access_log` row written with `purpose_token='biometric_collection'`
+5. Response: `{candidate_id, retention_until, consent_version}`
+
+**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):**
+
+The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/<id>.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option<JSON>`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type.
+
+```rust
+// crates/catalogd/src/subject_manifest.rs (paraphrased)
+pub struct BiometricCollection {
+    pub data_path: String,                // quarantined path
+    pub template_hash: String,            // SHA-256 of original bytes (integrity, NOT re-derivation)
+    pub collected_at: DateTime<Utc>,
+    pub classifications: Option<Value>,   // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md)
+}
+```
+
+**Engineering acceptance:**
+- Endpoint refuses uploads when consent missing (verified by integration test)
+- deepface output never lands in the synthetic-face manifest (`data/headshots/manifest.jsonl`)
+- Real-photo classifications are isolated to identityd `subjects` table — never flow to JSONL sinks
+- The `/headshots/:key` route in mcp-server REMAINS synthetic-only — does NOT serve real candidate photos to LLMs without an explicit allowance (proposed: real photos served only to authenticated staffer UI, never to model context)
+
+---
+
+### Gate 4 — Deprecate name → ethnicity inference
+
+**Required:** The hard-coded `NAMES_HISPANIC` / `SURNAMES_*` lookup tables in `mcp-server/search.html:3375-3432` (per Phase 1.5 §1B walk) get removed.
+
+**What ships:**
+- A code commit that removes:
+  - `FEMALE_NAMES`, `MALE_NAMES` constants
+  - `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` constants
+  - `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` constants
+  - The `genderFor()` and `guessEthnicityFromFirstName()` functions
+  - All call sites that consumed these (face-pool bucket selection)
+- Replacement strategy:
+  - For SYNTHETIC face pool routing: deterministic hash of candidate_id selects a face bucket, no demographic inference
+  - For REAL candidate photos: the candidate's actual photo IS the representation; no inference needed
+
+**Why this is BIPA + Title VII risk separately:** name-based ethnicity classification is BOTH a discriminatory feature engineering practice (Title VII) AND, when combined with photo-based attribute extraction, a "biometric information derived from a biometric identifier" pattern (BIPA broad reading). Removing the lookup tables forecloses both arguments.
+
+**Engineering acceptance:**
+- Lookup tables removed from search.html
+- Unit test asserts no protected-attribute inference functions exist in search.html or any mcp-server module
+- Face-pool routing for synthetic faces uses candidate_id hash exclusively
+- Phase 1.5 §1B finding closed
+
+---
+
+### Gate 5 — Documented destruction procedure
+
+**Required:** A written procedure for biometric data destruction at retention expiry OR consent withdrawal OR right-to-be-forgotten request.
+
+**What ships:**
+- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing
+- Specifies:
+  - Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request
+  - Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly
+  - Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads/<id>/*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails
+  - Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed
+  - Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction)
+  - Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request
+
+**⚖ COUNSEL** — review the runbook for legal sufficiency. Engineering writes the procedure; counsel attests that the procedure satisfies BIPA §15(a) destruction requirements.
+
+**Engineering acceptance:**
+- Runbook committed
+- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full)
+- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock)
+- Erasure events are logged with cryptographic attestation
+
+---
+
+## 2. Cryptographic attestation: no biometric data exists pre-identityd
+
+**Per `IDENTITY_SERVICE_DESIGN` v3-B11.** Plaintiffs may argue that the EXISTENCE of biometric schema fields constitutes constructive notice of intent to collect biometric data — therefore consent should have preceded the schema. The defense: prove that no biometric data was actually collected from real candidates before identityd + the consent gate.
+
+**What ships:**
+- A one-shot script `scripts/staffing/attest_pre_identityd_biometric_state.sh` that:
+  - Queries `data/datasets/workers_500k.parquet` schema and confirms NO column named `photo`, `biometric_*`, `face_*`, `image_*` exists
+  - Greps `data/_kb/*.jsonl` and `data/_pathway_memory/state.json` for any base64-encoded image bytes (deepface output, photo blobs)
+  - Verifies `data/headshots/manifest.jsonl` rows ≤ synthetic face pool size
+  - Hashes the schema + summary; commits the hash to S3 Object Lock (per identity service v3 anchor pattern)
+- Attestation document `docs/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-XX.md` signed by J + outside counsel
+
+**This is a one-time defense artifact.** It establishes the baseline: "as of this date, no biometric data was collected from real candidates."
+
+---
+
+## 3. Employee training acknowledgment (general BIPA hygiene)
+
+**Required:** People with access to biometric data acknowledge BIPA-handling training.
+
+**What ships:**
+- `docs/policies/BIPA_HANDLING_TRAINING_v1.md` — training material covering:
+  - What constitutes biometric identifiers / information
+  - The consent + retention procedures
+  - Destruction obligations
+  - Reporting suspected exposure
+- Acknowledgment record per individual (initially: J + counsel + named operators)
+- Annual refresh
+
+**⚖ COUNSEL** — write training content. Engineering doesn't author legal-compliance training.
+
+---
+
+## 4. Phase 1.6 exit criteria (gates Phase 2 backfill)
+
+All 5 gates must be DONE before identityd backfill begins. Status as
+of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code:
+
+| # | Gate | Engineering | Counsel | Status |
+|---|---|---|---|---|
+| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** |
+| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** |
+| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option<JSON> = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** |
+| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** |
+| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** |
+
+PLUS:
+
+| # | Item | Engineering | Counsel | Status |
+|---|---|---|---|---|
+| 6 | Cryptographic attestation pre-identityd | DONE — `scripts/staffing/attest_pre_identityd_biometric_state.sh` + `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (3/3 evidence checks pass; signature lines pending) | pending signature | **eng-DONE, signature-pending** |
+| 7 | Employee training material | scaffold deferred — Gate 5 runbook §7 acknowledgment may serve as substrate | pending | **deferred** |
+
+**Blocking set for Phase 2 backfill:** items **1, 2, 3, 4, 5, 6** must
+all be DONE. Item 7 (employee training) is reduced from blocking to
+"deferred" because the Gate 5 destruction runbook §7 already requires
+operator acknowledgment before legal-tier credentials are issued —
+that acknowledgment is procedurally equivalent to the training-record
+requirement when the operator population is small (J + 1-2 named
+operators). If the operator population grows beyond that, item 7
+re-promotes to blocking and a separate training program must be authored.
+
+⚖ COUNSEL — confirm whether item 7 deferral is acceptable for the
+expected operator population size, or restore it to the blocking set.
+
+**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel
+review of the engineering scaffolds. Gate 3 substrate is fully
+shipped; Gate 3b deepface classification was DECIDED on 2026-05-05
+as Option C (defer) — `BiometricCollection.classifications` stays
+`None` in v1, consent + retention docs revised to match this
+narrower scope. If a future product requirement surfaces a real
+need for classifications, the substrate is forward-compatible
+(`Option<JSON>`) and either Option A (~1 day) or Option B (~5 days)
+of the design doc can be picked up then under a v2 consent template.
+
+---
+
+## 5. Effort estimate
+
+| Gate | Engineering effort | Legal effort |
+|---|---|---|
+| Gate 1 (retention schedule) | 0.5 day | counsel-dependent (typically 1-2 weeks for review) |
+| Gate 2 (consent template) | 0.5 day | counsel-dependent (typically 2-4 weeks for review and consent UX design) |
+| Gate 3 (photo-upload endpoint) | 1-2 days | review of endpoint behavior |
+| Gate 4 (deprecate name-ethnicity inference) | 0.5 day | none (engineering-only fix) |
+| Gate 5 (destruction runbook) | 1 day | counsel sign-off |
+| §2 cryptographic attestation | 0.5 day | counsel + J signature |
+| §3 employee training | 0.25 day (admin) | counsel-authored content |
+| **Total engineering** | **~4-5 days** | — |
+| **Total counsel** | — | **~3-6 weeks calendar** (review cycles) |
+
+**The calendar bottleneck is counsel, not engineering.** Engineering can stage all 5 gates ready-to-ship in a week. Counsel sign-off + consent UX rollout is the longer pole.
+
+---
+
+## 6. Open questions for J + counsel
+
+1. **Photo-upload UX:** is there an existing intake form / staffer console where photo upload would happen? Or is this new UI work?
+2. **Consent collection mechanism:** electronic signature service (DocuSign, Adobe Sign), in-app click-acceptance, paper form? Each has different evidentiary weight in litigation.
+3. **Operator list with biometric access:** who, today, would be on the named-operators list for §3 training?
+4. **Counsel for sign-off:** named outside counsel — same or different from the dual-control legal-token party in identity service?
+5. **Public privacy policy URL:** does one exist? If yes, where; if no, that's a separate Gate-1.5 deliverable.
+
+---
+
+## 6.5. Post-signature deployment runbook
+
+When counsel returns the countersigned consent template + retention
+schedule, the engineering side of "flip from permissive to strict
+mode" is one command:
+
+```bash
+# 1. Counsel commits their signature to §7 of the consent template
+#    markdown (or J commits the signed PDF + updates §7 with counsel's
+#    name + date). The markdown is the BINDING TEXT — the PDF is just
+#    a rendering of it.
+
+# 2. Hash the canonical signed text + seed the gateway allowlist.
+./scripts/staffing/seed_consent_version.sh \
+    docs/policies/consent/biometric_consent_template_v1.md \
+    --label "v1 signed YYYY-MM-DD by [counsel name]"
+
+# The script:
+#   - computes SHA-256 of the markdown (binding text)
+#   - atomically writes /etc/lakehouse/consent_versions.json with
+#     the new hash + per-seed audit metadata (timestamp, label,
+#     source path)
+#   - restarts lakehouse.service so the gateway re-reads the
+#     allowlist
+#   - probes /biometric/health for clean restart
+
+# 3. Verify strict mode is rejecting unknown hashes:
+TOKEN=$(cat /etc/lakehouse/legal_audit.token)
+curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \
+    -H "X-Lakehouse-Legal-Token: $TOKEN" \
+    -H "Content-Type: application/json" \
+    -d '{"consent_version_hash":"deadbeefdeadbeef000000000000000000000000000000000000000000000000","consent_collection_method":"electronic_signature","operator_of_record":"strict_mode_probe"}'
+# Expect: HTTP 400 + {"error":"consent_version_unknown", ...}
+```
+
+After this, the gateway is in counsel-tier strict mode:
+
+- Any consent grant POST whose `consent_version_hash` doesn't match
+  a known signed template is refused at intake
+- Operator typos (mistyped hash) become loud failures, not silent
+  bad records
+- Future template revisions (v2, v3, ...) require counsel re-sign
+  AND a new `seed_consent_version.sh` run before being accepted —
+  the v1 hash stays in the allowlist for already-collected subjects'
+  audit-trail compatibility
+
+### Pre-counsel demo seed
+
+For deployments that want to exercise strict mode BEFORE counsel
+signs, the same script works against the eng-staged template:
+
+```bash
+./scripts/staffing/seed_consent_version.sh \
+    docs/policies/consent/biometric_consent_template_v1.md \
+    --label "eng-staged demo seed (NOT counsel-signed)"
+```
+
+The hash entry should be replaced (rotate the demo hash out, add
+the counsel-signed hash) when counsel completes review. The
+allowlist's `_meta.seeded_at[]` array preserves the seed history.
+
+---
+
+## 7. What this PRD is NOT
+
+- Not legal advice. The `⚖ COUNSEL` markers exist because the binding text needs lawyers, not engineers.
+- Not a substitute for a DPIA / PIA. Phase 1.6 satisfies BIPA-specific gates; a Data Protection Impact Assessment is broader and may be required separately.
+- Not a SOC2 Type II deliverable. SOC2 is a parallel work stream.
+- Not the only gate before production. The full 9-phase audit-trail program continues; Phase 1.6 specifically unblocks Phase 2 (identity service implementation).
+
+---
+
+## Change log
+
+- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup.
+- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill.

docs/PHASE_AUDIT_GUIDE.md

@@ -0,0 +1,107 @@
+# Phase Audit Guidance for Claude Code
+
+## Purpose
+This document provides the proper workflow for auditing completed phases in the Lakehouse project.
+
+## ⚠️ Important: Do NOT Skip Steps
+Each phase requires BOTH:
+1. PRD spec verification (check code exists)
+2. Full SCRUM execution (6 commands)
+
+## Proper Phase Audit Workflow
+
+### Step 1: Read PRD Specification
+For each phase, read the PRD to understand what's supposed to ship:
+```bash
+# Read from docs/PRD.md or docs/PHASES.md
+cat docs/PHASES.md | grep -A20 "Phase N:"
+```
+
+### Step 2: Verify Code Exists
+Check that each deliverable from the PRD spec has corresponding code:
+```bash
+# Example - check for specific implementations
+grep -r "function_name" crates/*/src/
+ls crates/*/src/*.rs
+```
+
+### Step 3: Run Full SCRUM (6 Commands)
+In order, execute ALL of these for the phase's crates:
+
+```bash
+# 1. Build
+cargo build -p <crate-name>
+
+# 2. Test  
+cargo test -p <crate-name>
+
+# 3. Clippy (if installed)
+cargo clippy -p <crate-name> -- -D warnings
+
+# 4. Format check
+cargo fmt -p <crate-name> -- --check
+
+# 5. Cargo check
+cargo check -p <crate-name>
+
+# 6. Doc check
+cargo doc -p <crate-name> --no-deps
+```
+
+### Step 4: Fix Issues
+If any SCRUM command fails:
+- Fix the code
+- Re-run the failing command
+- Re-run ALL 6 commands to verify
+
+### Step 5: Update Phase Documentation
+Only mark as ✅ after ALL 6 SCRUM commands pass:
+```markdown
+## Phase N: [Name] ✅
+- [x] spec item 1
+- [x] spec item 2
+  - SCRUM: build ✅ test ✅ clippy ✅ fmt ✅ check ✅ doc ✅
+```
+
+## Current Phase Status
+
+| Phase | Status | Notes |
+|-------|--------|-------|
+| 0 | ✅ | Bootstrap complete |
+| 1 | ✅ | Storage + Catalog |
+| 2 | ✅ | Query Engine |
+| 3 | ✅ | AI Integration |
+| 4 | ✅ | Frontend |
+| 5 | ✅ | Hardening |
+| 6-42 | ✅ | See docs/PHASES.md |
+
+## Notes from Previous Session
+
+- Clippy and rustfmt are NOT installed on this system
+- Run `rustup component add clippy rustfmt` to install
+- Some crates have 0 unit tests (expected for service crates)
+- 28 warnings remain in unused code paths (ui/vectord)
+
+## Key Files
+
+- `docs/PHASES.md` - Phase tracker with checkboxes
+- `docs/PRD.md` - Full product requirements
+- `docs/CONTROL_PLANE_PRD.md` - Phases 38+ specifications
+- `crates/*/` - All crate implementations
+
+## Quick Reference
+
+```bash
+# Full workspace SCRUM
+cargo build --workspace
+cargo test --workspace
+# (clippy if installed)
+cargo fmt -- --check
+cargo check --workspace
+cargo doc --no-deps
+
+# Per-crate
+cargo build -p <crate>
+cargo test -p <crate>
+cargo check -p <crate>
+```

docs/PRD.md

@@ -77,7 +77,7 @@ A modular Rust service mesh over S3-compatible object storage, with a local AI l
 | Data Format | Parquet + Arrow | Yes |
 | RPC (internal) | tonic (gRPC) | Yes |
 | AI Runtime | Ollama (local models) | Yes |
-| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes |
+| AI Boundary | Direct Ollama HTTP API (gateway → Ollama, no sidecar). Updated 2026-05-02 per ADR-022 — was "Python FastAPI sidecar → Ollama HTTP API". Sidecar's lab_ui/pipeline_lab Python remain as dev-only tools (not on hot path). | Yes |
 | Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** |
 
 No new frameworks without documented ADR.
@@ -97,7 +97,7 @@ No new frameworks without documented ADR.
 | **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog |
 | **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) |
 | **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal |
-| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
+| **aibridge** | AI client — direct Ollama HTTP (per ADR-022, 2026-05-02; was Rust↔Python sidecar boundary). Owns LRU embed cache. |
 | **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
 | **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
 | **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) |
@@ -161,6 +161,18 @@ The feedback signal is **statistical + semantic**, not neural. No model training
 
 ---
 
+## Production-readiness blocker (2026-05-03)
+
+Substrate is mostly shipped through Phase 45. Throughput, parity, and observability are all measured.
+
+**The production-readiness gate is the audit trail.** Staffing client requires defensible response to discrimination claims (worked example: John Martinez at Warehouse B requests every AI-system decision affecting him). PII tokenization, subject-of-record query, retention SLA, and right-to-be-forgotten under append-only logs are NOT YET BUILT.
+
+Authoritative document: **`docs/AUDIT_TRAIL_PRD.md`**. 9-phase implementation plan, current-state-vs-target gap table, open questions blocking phase 1.
+
+Until the audit-trail PRD's phase 9 exit criterion is met, "production-ready" claims are aspirational. Internal substrate (Lance, observability parity, cross-runtime fabric) is solid; subject-audit story is not.
+
+---
+
 ## Phases
 
 ### Phase 0-5: Foundation ✅ COMPLETE

docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md

@@ -0,0 +1,91 @@
+# BIPA Pre-IdentityD Biometric Attestation
+
+**Date:** 2026-05-03
+**Spec:** docs/PHASE_1_6_BIPA_GATES.md §2
+**Generator:** scripts/staffing/attest_pre_identityd_biometric_state.sh
+
+## Purpose
+
+This is a one-time defense artifact establishing that, as of
+2026-05-03, no biometric identifiers or biometric information
+from real candidates have been collected, processed, or stored
+by the Lakehouse system. It is intended to be signed by J
+(operator of record) and outside counsel, then anchored to a
+tamper-evident store (filesystem with backups + version control).
+
+## Evidence
+
+## Check 1 — workers_500k.parquet schema (no biometric columns)
+
+**Source:** `data/datasets/workers_500k.parquet`
+
+**Schema columns** (18 total):
+
+```
+worker_id	int64	nullable=True
+name	string	nullable=True
+role	string	nullable=True
+email	string	nullable=True
+phone	string	nullable=True
+city	string	nullable=True
+state	string	nullable=True
+zip	int64	nullable=True
+skills	string	nullable=True
+certifications	string	nullable=True
+archetype	string	nullable=True
+reliability	double	nullable=True
+responsiveness	double	nullable=True
+engagement	double	nullable=True
+compliance	double	nullable=True
+availability	double	nullable=True
+communications	string	nullable=True
+resume_text	string	nullable=True
+```
+
+**Schema SHA-256:** `973b9abe56420de8f88122278b633e813f90a64cf0ddaac6a9811dc0940be676`
+
+  - PASS: no biometric / photo / face / image column present
+
+## Check 2 — KB + pathway memory contain no biometric payloads
+
+**Sources scanned:**
+- `data/_kb/*.jsonl` (knowledge base)
+- `data/_pathway_memory/state.json` (pathway memory state)
+
+**Files scanned:** 33
+**Forbidden-pattern hits:** 0
+
+  - PASS: no biometric payload patterns found in scanned files
+
+## Check 3 — Headshots manifest is synthetic-only
+
+**Source:** `data/headshots/manifest.jsonl`
+
+**Total rows:** 1000
+**Rows tagged real/candidate_upload/photo_upload:** 0
+
+  - PASS: all 1000 rows are synthetic (no real-candidate uploads)
+
+## Summary
+
+**3 / 3** evidence checks pass.
+
+
+---
+
+## Attestation
+
+I, the undersigned, attest that the above evidence accurately
+reflects the state of the Lakehouse system as of 2026-05-03.
+No biometric identifiers or biometric information from real
+candidates have been collected, processed, or stored prior to
+the deployment of the Phase 1.6 BIPA pre-launch gates.
+
+**Evidence SHA-256:** `1fdcc9f1682de27e1a0556d698ce221b74c1e71cf54128763828b4bca7b5c1bf`
+
+---
+
+**Operator (J):** _______________________________  Date: __________
+
+**Outside counsel:** ___________________________  Date: __________
+

docs/counsel/COUNSEL_HANDOFF_EMAIL_2026-05-05.md

@@ -0,0 +1,187 @@
+# Counsel Handoff — Copy-Paste Email Draft
+
+**Date assembled:** 2026-05-05
+**Operator of record:** J
+**Prepared for:** outbound transmission to outside counsel
+**Tarball:** `reports/counsel/counsel_packet_2026-05-05.tar.gz`
+**Tarball SHA-256:** see `reports/counsel/counsel_packet_2026-05-05.manifest.txt`
+
+> This is a copy-paste-ready email draft. Subject line + body below.
+> Attach the tarball + manifest before sending. The email frames
+> what counsel needs to do, by when, and what happens after they
+> sign — so the calendar bottleneck has no excuse to sit longer
+> than counsel's actual review time.
+
+---
+
+## Subject line
+
+```
+Lakehouse staffing platform — Phase 1.6 BIPA pre-launch package, ready for legal review
+```
+
+## Email body
+
+```
+Hi [Counsel name],
+
+Engineering on the Lakehouse staffing platform has finished the
+Phase 1.6 BIPA (740 ILCS 14) pre-launch substrate. Before we begin
+collecting any real candidate biometric data (photographs), the
+five engineering-staged documents in the attached package need
+your review and, where indicated, your signature.
+
+What's attached:
+
+1. counsel_packet_2026-05-05.tar.gz — bundled review package
+2. counsel_packet_2026-05-05.manifest.txt — per-file SHA-256
+   integrity hashes (re-run `sha256sum -c` on receipt to verify
+   nothing changed in transit)
+
+Open the cover letter inside the tarball
+(`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`) first — it
+walks through the substrate, the specific items needing your
+sign-off, and recommended review sequence.
+
+Headline asks (in priority order):
+
+  A. Biometric Retention Schedule v1
+       — render into binding language, sign as countersigning
+         party. Sets the public retention schedule required by
+         BIPA §15(a).
+  B. Biometric Consent Template v1
+       — render Disclosures 1-3 into binding consent text, sign.
+         This is the form a candidate signs before any photo is
+         collected. Required by BIPA §15(b)(1)-(3).
+  C. BIPA Destruction Runbook
+       — review for legal sufficiency, attest. Procedural
+         document; engineering wrote the steps, your role is
+         confirming they satisfy BIPA §15(a) destruction.
+  D. Pre-IdentityD Attestation (2026-05-03)
+       — countersign as the legal party. Establishes the boundary
+         that no biometric data was collected before the gates
+         shipped. One-time defense artifact.
+
+Item C also references a key-rotation runbook (E in the cover
+letter) that's lower priority — opine when convenient.
+
+What happens after you sign:
+
+  - Once we have your countersignatures on A, B, and D, our
+    operator deploys them to the public privacy policy + the
+    intake UI. Engineering will hash the canonical signed text
+    of (B) and add that hash to the gateway's consent_versions
+    allowlist file (/etc/lakehouse/consent_versions.json) so the
+    runtime starts refusing any consent record that wasn't
+    signed under the canonical template. (The gateway is
+    currently in permissive mode — accepting any non-empty
+    hash. Strict mode flips on with your signature.)
+  - The runtime gates are already enforcing the consent flow:
+    photo upload refuses 403 unless consent.biometric.status =
+    "given", and the only path to that state is the
+    /biometric/subject/{id}/consent endpoint with a valid
+    consent_version_hash. After your signature, that hash will
+    point at canonical signed text — operator typo-resistant
+    and counsel-defensible.
+
+Calendar:
+
+  - We'd like to begin real-photo intake within [N] weeks. If
+    your review can land within 2-3 weeks of receipt, that
+    keeps us on schedule. If your bandwidth requires longer,
+    please flag — we have eng work to advance independently
+    that doesn't depend on the signatures.
+
+Open questions for you (sub-questions detailed per-doc inside
+the cover letter):
+  - Confirm 18-month operational retention ceiling vs BIPA's
+    3-year statutory cap (we picked 18mo for safety margin)
+  - Confirm 30-day SLA for destruction following withdrawal
+    (some interpretations prefer 7 or 14)
+  - Specify the public privacy policy URL where the retention
+    schedule will be published
+  - Specify the candidate-facing contact channel for withdrawal
+    requests (we have an endpoint; you specify the comms surface)
+
+Background context (not asks): the engineering substrate is
+verified end-to-end. We've run a live demo of the full lifecycle
+(consent → photo → withdraw) on a test subject; the audit chain
+verifies cryptographically; the retention sweep is scheduled
+daily. No real candidate photos have been collected. The
+attached pre-identityd attestation has the cryptographic evidence
+hashes from that pre-collection state.
+
+Happy to schedule a call if any of this would be easier to walk
+through verbally.
+
+Thanks,
+J
+[contact info]
+```
+
+---
+
+## Operator pre-flight checklist (before sending)
+
+Run through this once before clicking send:
+
+- [ ] Confirm tarball + manifest are the latest versions
+      (`bundle_counsel_packet.sh` regenerates if needed; current
+      hash is in the manifest sidecar)
+- [ ] Confirm counsel name is correct (and that the engagement
+      letter or retainer is in place)
+- [ ] Confirm the [N] weeks calendar target — replace with a
+      real number ("3 weeks" / "by end of June" / etc.)
+- [ ] Confirm the email goes to counsel's secure channel; if
+      counsel uses a portal, upload there + send a notification
+      email instead
+- [ ] Save a copy of the sent email + the tarball to operator
+      records — this is itself part of the audit trail of "we
+      asked counsel on [date]"
+
+---
+
+## Post-signature operator runbook (after counsel responds)
+
+When counsel returns the signed documents:
+
+1. **Verify integrity.** Run `sha256sum -c manifest.txt` on
+   the returned tarball if counsel returns the bundle; otherwise
+   compare returned signed PDFs against the markdown sources to
+   confirm the binding text matches what counsel reviewed.
+
+2. **Capture the canonical signed text.** Counsel may sign a
+   PDF rendering of the markdown — that's fine, but the BINDING
+   TEXT we hash is the markdown source (counsel commits to git
+   countersigning §7 of the consent template + §8 of the
+   retention schedule).
+
+3. **Compute + seed the consent_version_hash.** Run:
+   ```bash
+   ./scripts/staffing/seed_consent_version.sh \
+       docs/policies/consent/biometric_consent_template_v1.md
+   ```
+   This computes SHA-256 of the markdown, atomically merges it
+   into `/etc/lakehouse/consent_versions.json` (creating the
+   file if absent), and offers to restart the gateway.
+
+4. **Verify strict mode is live.** Probe with a known-bad hash:
+   ```bash
+   TOKEN=$(cat /etc/lakehouse/legal_audit.token)
+   curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \
+     -H "X-Lakehouse-Legal-Token: $TOKEN" \
+     -H 'Content-Type: application/json' \
+     -d '{"consent_version_hash":"deadbeef","consent_collection_method":"electronic_signature","operator_of_record":"smoke"}' | jq
+   ```
+   Expect HTTP 400 with `error: "consent_version_unknown"`.
+
+5. **Update STATE_OF_PLAY.md** to record the signature event +
+   the hash entry in the allowlist. Counsel-tier deployment is
+   now live.
+
+6. **Sign and anchor the §2 attestation.** Both J and counsel
+   sign `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md`
+   and the SHA-256 of the signed markdown is committed to the
+   git history as a tamper-evident anchor.
+
+After step 5, the production cutover blocker is closed.

docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md

@@ -0,0 +1,260 @@
+# Counsel Review Packet — Phase 1.6 BIPA Pre-Launch
+
+**Date assembled:** 2026-05-05
+**For:** outside counsel
+**From:** J, operator of record
+**Scope:** documents that engineering has staged for legal sufficiency review
+            before the staffing platform begins collecting any real candidate
+            biometric data (BIPA §15(a)(b)).
+
+> **What this packet is.** The Phase 1.6 BIPA gates outline what
+> engineering must ship before real-photo intake. As of 2026-05-05,
+> all engineering substrate is shipped and verified live (see §1
+> below for the inventory). What remains is binding-text authoring
+> + counsel sign-off on five documents, plus operational notification
+> obligations counsel may want to layer on top.
+>
+> **What this packet is NOT.** Not a request for counsel to write
+> binding text from scratch. The documents are eng-staged in
+> reasonable plain language; the request is for counsel to render
+> them into legally-sufficient text and attest where signatures
+> are required.
+
+---
+
+## 1. Engineering substrate — shipped + verified
+
+For factual context on what counsel is reviewing AGAINST. None of
+this requires sign-off here; it's the system the documents bind to.
+
+| Component | Where it lives | Verification |
+|---|---|---|
+| Subject manifest registry | `crates/catalogd/src/registry.rs`, `data/_catalog/subjects/<id>.json` | 17 unit tests + 100 backfilled WORKER manifests in production |
+| Per-subject HMAC audit chain (SHA-256) | `crates/catalogd/src/subject_audit.rs`, `data/_catalog/subjects/<id>.audit.jsonl` | Tamper-detection + concurrent-append race tests pass |
+| Photo upload (consent-gated) | `POST /biometric/subject/{id}/photo` | 11 unit tests + live roundtrip 200 |
+| Erasure (two-scope) | `POST /biometric/subject/{id}/erase` (`biometric_only` / `full`) | 21 unit tests; transactional rollback on audit failure |
+| Legal-tier audit read | `GET /audit/subject/{id}` (X-Lakehouse-Legal-Token header) | Constant-time auth, chain re-verification per request |
+| Retention sweep (BIPA-aware clock) | `crates/catalogd/src/bin/retention_sweep` | 8 unit tests; live verified against 100 backfilled subjects |
+| Cross-runtime parity (Rust ↔ Go) | `scripts/cutover/parity/subject_audit_parity.sh` | 6/6 byte-identical assertions pass |
+
+**Key insight for counsel:** the audit chain is the BIPA-defensible
+substrate. Every state-changing event (consent given, photo uploaded,
+photo erased, legal-tier read) appends to a per-subject HMAC-chained
+JSONL log. The chain verifies end-to-end on every legal-tier read.
+A tampered chain is detectable; a forged chain requires the HMAC
+signing key, which is held under root-only mode 0400 at
+`/etc/lakehouse/subject_audit.key` and rotated per the runbook in
+attachment §6 below.
+
+**Gate 3b (deepface classification) — decided 2026-05-05: Option C
+(defer).** The system collects only the photograph, not derived
+demographic information. The consent template + retention schedule
+in this packet were revised the same day to match.
+
+---
+
+## 2. Documents requiring counsel review + sign-off
+
+In recommended review order:
+
+| # | Document | Path | Counsel ask | Sign-off |
+|---|---|---|---|---|
+| A | Biometric Retention Schedule v1 | `docs/policies/consent/biometric_retention_schedule_v1.md` | Render into binding language; confirm 18-month operational ceiling vs. BIPA 3-year statutory cap | Counsel + J |
+| B | Biometric Consent Template v1 | `docs/policies/consent/biometric_consent_template_v1.md` | Render Disclosures 1-3 into binding consent language; specify electronic vs. paper signature mechanism | Counsel + J |
+| C | BIPA Destruction Runbook | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` | Confirm 30-day SLA from trigger; confirm two-operator (operator + witness) requirement; confirm legal-hold check procedure | Counsel attestation |
+| D | BIPA Pre-IdentityD Attestation | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` | Sign as countersigning party; J signs as operator-of-record | Counsel + J |
+| E | Legal-Tier Audit Key Rotation | `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` | Confirm rotation cadence; opine on candidate-notification obligation when rotation is compromise-driven | Counsel notes |
+| F | Gate 3b Deepface Design (FYI) | `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` | Decision-of-record showing classifications were *deliberately deferred*, not omitted by oversight. No sign-off needed; provided for audit-trail completeness. | None |
+
+The five documents requiring sign-off are A, B, C, D, E. Document F
+is included so the audit trail shows the Gate 3b decision was
+deliberate.
+
+---
+
+## 3. Specific questions for counsel — by document
+
+### Document A — Retention Schedule
+
+1. The schedule sets an **18-month** operational ceiling against the
+   BIPA 3-year statutory cap. Is the safety margin appropriate, or
+   should we move to a tighter window (12 months) given the
+   plaintiff-friendly Illinois posture?
+2. The schedule references the **catalogd-local** storage substrate
+   rather than a separate identityd Postgres table. Does the
+   public-facing language need to mention the storage architecture
+   at all, or is "we keep the photo and a SHA-256 hash" sufficient?
+3. Public publication URL — counsel to specify (placeholder marked
+   in §7 of the schedule).
+4. Confirm whether existing consent under v1 carries forward when
+   a future v2 is published, or whether re-consent is required.
+
+### Document B — Consent Template
+
+1. Disclosure 1 says "we do NOT run automated facial-classification
+   in v1." Does that disclosure need to mention the *possibility* of
+   future classification, or is silence-with-supersession-clause
+   adequate?
+2. Plain-language summary in §1 — counsel to confirm it's appropriate
+   to include alongside the binding disclosure, or recommend an
+   alternative comprehension aid.
+3. Withdrawal SLA is set to **30 days** in §2. Counsel to confirm
+   against jurisdiction (Illinois primary; secondary deployments
+   would inherit).
+4. Contact for withdrawal — counsel to specify the channel
+   (placeholder in §3).
+5. Sign-off mechanism: electronic signature service, in-app
+   click-acceptance with timestamp, paper form? Each has different
+   evidentiary weight.
+
+### Document C — Destruction Runbook
+
+1. Confirm 30-day SLA from each of four triggers (retention expiry,
+   consent withdrawal, RTBF, court order). Some interpretations
+   prefer 7 or 14 days for withdrawal/RTBF.
+2. Two-operator requirement (operator-of-record + witness): is the
+   witness role acceptable for counsel's defensibility view, or
+   should we elevate to dual-control with cryptographic split-key?
+3. Legal-hold check procedure (§2 step 3) — counsel to specify the
+   actual procedure for confirming no hold is in force before
+   erasing.
+4. Backup-window disclosure (§4) — confirm 30-day backup retention
+   is acceptable.
+5. Candidate notification template (§3 step 4) — counsel to supply.
+
+### Document D — Pre-IdentityD Attestation
+
+1. Both signature lines blank — J signs as operator-of-record;
+   counsel signs as the countersigning legal party.
+2. The attestation hash anchors the evidence; once signed, the
+   hash itself becomes a tamper-evident witness. Counsel to confirm
+   storage location for the signed copy (firm files?).
+
+### Document E — Key Rotation Runbook
+
+1. Recommended rotation cadence — 90 days suggested in §1.
+   Counsel to confirm or override.
+2. Custody schedule for `/etc/lakehouse/_archived/` raw key files —
+   §7.2 question; suggested 1-year retention but counsel-driven.
+3. Candidate-notification obligation when rotation is
+   compromise-driven (§7.3) — counsel call.
+
+---
+
+## 4. Engineering changes counsel should know about (recent)
+
+These reconciled doc/code drift after a rapid wave on 2026-05-03:
+
+- **Endpoint paths:** the original v1 spec proposed
+  `/v1/identity/subjects/*` under a separate identityd daemon. That
+  daemon was collapsed into catalogd; endpoints actually shipped at
+  `/biometric/subject/*` (catalogd-local). Documents in this packet
+  reference the catalogd-local routes; legacy references in
+  `IDENTITY_SERVICE_DESIGN.md` are flagged "do NOT implement
+  as-written" in that doc's deprecation header.
+- **No identityd Postgres database:** the original spec proposed
+  encrypted-at-rest Postgres + HashiCorp Vault + S3 Object Lock for
+  PII storage. The shipped substrate is local JSON manifests +
+  per-subject HMAC-chained JSONL, sized for J's local-only
+  deployment per `PRD.md` line 70 ("Everything runs locally — no
+  cloud APIs").
+- **Gate 3b deferral (Option C, 2026-05-05):** classifications
+  (gender / race / age inference) were deliberately deferred. The
+  consent template and retention schedule in this packet do NOT
+  disclose collection of derived demographic data, because we are
+  not collecting it. If a future product requirement reverses this,
+  we will publish a v2 consent + v2 retention with re-consent.
+- **Key rotation 2026-05-05:** the prior `LH_SUBJECT_AUDIT_KEY` was
+  lost when a `/tmp` wipe on reboot disabled the audit and biometric
+  endpoints. The new key is at `/etc/lakehouse/subject_audit.key`
+  (mode 0400). Pre-rotation audit chains tamper-detect under the
+  new key — this is correct, expected behavior, not a bug.
+
+---
+
+## 5. Open eng items NOT awaiting counsel
+
+For transparency. These are engineering work items, not legal items:
+
+1. **Residual photo unlink on erasure.** During verification of the
+   one historical erasure event (`WORKER-2`), the verify script
+   surfaced a stranded photo file that was not unlinked when
+   `BiometricCollection` was cleared from the manifest. Engineering
+   investigates; if the bug is real, the fix is `crates/catalogd/src/biometric_endpoint.rs`
+   in the erasure handler. This does NOT affect the current packet —
+   no real candidate photos have been collected yet (per §1
+   attestation), so the residual is from a synthetic test event.
+2. **Phase 1.6 §3 employee training.** Currently deferred per
+   acknowledgement coverage in §7 of the destruction runbook
+   (single-operator population). Re-promotes to blocking if the
+   operator population grows; counsel may want to opine on the
+   threshold.
+
+---
+
+## 6. Sign-off sequence
+
+Recommended order so a hold-up on one doc doesn't block others:
+
+1. **First wave (parallel):** A (retention schedule) + B (consent
+   template). These two have the tightest interdependence (consent
+   v1 references retention v1 by hash); review them together.
+2. **Second wave:** C (destruction runbook). Depends on A's retention
+   period being fixed.
+3. **Third wave:** D (pre-identityd attestation). Sign once A + B + C
+   are settled; the attestation snapshot is the boundary between
+   pre-Phase-1.6 and post-Phase-1.6 system state.
+4. **Fourth wave:** E (key rotation). Independent of A-D; can be
+   reviewed in parallel any time.
+
+---
+
+## 7. After sign-off — engineering steps
+
+Once each document is signed:
+
+| Document | Engineering action | Trigger |
+|---|---|---|
+| A retention schedule | Hash + commit; reference in `consent_versions` table | Counsel signature |
+| B consent template | Hash + commit; reference in candidate-facing intake UI | Counsel signature |
+| C destruction runbook | Adopt; operator acknowledgment recorded in §7 | Counsel attestation |
+| D pre-identityd attestation | Anchor hash to filesystem + git; counsel keeps original signature | Both signatures |
+| E key rotation | Adopt; rotation event log seeded with counsel-approved cadence | Counsel notes |
+
+The HARD blocker for first real-candidate photo collection is
+A + B + D signed. C and E are operationally important but do not
+block the *first* photo (they govern destruction + key handling
+which apply to any state, not the boundary state).
+
+---
+
+## 8. Cover-note hash
+
+This packet is itself a snapshot. Future-Claude / future-J will refer
+back to this packet to know what counsel saw on 2026-05-05.
+
+**Packet attached files (referenced by path):**
+
+- `docs/policies/consent/biometric_retention_schedule_v1.md`
+- `docs/policies/consent/biometric_consent_template_v1.md`
+- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
+- `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md`
+- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`
+- `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`
+- `docs/PHASE_1_6_BIPA_GATES.md` (the spec they all reference)
+
+Per-file SHA-256 hashes are produced by the bundler script (next
+section); the bundler also creates a tarball ready for transmission.
+
+---
+
+## 9. Generating the bundle for transmission
+
+```bash
+./scripts/staffing/bundle_counsel_packet.sh
+```
+
+Produces `reports/counsel/counsel_packet_<DATE>.tar.gz` with all
+referenced documents + a manifest listing per-file SHA-256 hashes.
+Counsel can verify file integrity on receipt by re-running
+sha256sum against each file in the tarball.

docs/policies/consent/biometric_consent_template_v1.md

@@ -0,0 +1,175 @@
+# Biometric Information Consent — v1
+
+**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 2 (BIPA §15(b)(1)-(3))
+**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before deployment
+**Version:** v1 (initial; supersession requires a new version + new hash)
+**Updated 2026-05-05:** Disclosure 1 + plain-language summary revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). Pending J's product confirmation of Gate 3b; if Gate 3b chooses Option A or B, this template needs counsel re-authoring.
+
+> This is the consent template a candidate signs (electronically or
+> on paper) BEFORE Lakehouse collects, stores, or processes any
+> biometric identifier or biometric information from that candidate.
+>
+> Without an executed consent under this template (or a counsel-
+> approved successor), the system MUST NOT accept a photograph from
+> the candidate. Enforcement lives at the photo-upload endpoint
+> (Gate 3) and at the SubjectManifest writer, which refuses biometric
+> writes when `consent.biometric.status != "given"`.
+
+---
+
+## Required disclosures (BIPA §15(b)(1)-(3))
+
+The disclosures below are MANDATORY content per 740 ILCS 14/15(b).
+⚖ COUNSEL — render this content into binding language appropriate
+for the candidate-facing UI. Engineering provides the structural
+content; counsel provides the legally-sufficient wording.
+
+### Disclosure 1 — Notice of collection (§15(b)(1))
+
+Lakehouse will collect and store my **biometric identifier** (a
+photograph of me from which facial geometry is implicit). The
+photograph itself is the data we keep — we do NOT run automated
+facial-classification (gender / race / age inference) against it
+in v1. If at a later date we add automated classification, we will
+re-collect consent under a superseding template before doing so.
+
+### Disclosure 2 — Specific purpose and length of term (§15(b)(2))
+
+The biometric data will be used for:
+
+1. Identity verification at staffing job sites
+2. Internal record-keeping so coordinators can recognize me across
+   placements
+
+The biometric data will be retained for a maximum of **18 months**
+from my most recent interaction with the staffing platform, after
+which it will be permanently destroyed per the
+[Biometric Retention Schedule v1](biometric_retention_schedule_v1.md).
+
+I may withdraw this consent at any time by contacting the operator
+(see §3 below). Withdrawal triggers permanent destruction of my
+biometric data.
+
+### Disclosure 3 — Written release (§15(b)(3))
+
+I provide a written release authorizing Lakehouse to collect, store,
+and use my biometric identifier and biometric information for the
+purposes stated above and for the term stated above.
+
+---
+
+## 1. Plain-language summary (non-binding)
+
+⚖ COUNSEL — the section above is the binding legal disclosure.
+The summary below is provided for candidate comprehension and is
+NOT a substitute for the binding disclosure. Both should appear
+together in the consent UI; counsel determines whether this summary
+is appropriate to include or whether a different plain-language
+section is preferred.
+
+> **What you're agreeing to:** if you upload a photo of yourself,
+> we'll keep that photo so your staffing coordinator can recognize
+> you when you arrive at job sites. We don't run automated guesses
+> about your age, gender, or race against the photo.
+>
+> **How long we keep it:** at most 18 months after your last
+> placement or interaction with us, then it's permanently destroyed.
+>
+> **What we DON'T do with it:** we don't sell it, we don't share it
+> with anyone outside the staffing operation unless legally compelled,
+> and we don't use it to decide what jobs to recommend to you.
+>
+> **How to take it back:** contact us (§3 below) at any time to
+> withdraw your consent. We will permanently destroy your biometric
+> data within 30 days of receiving your request.
+
+---
+
+## 2. Withdrawal procedure
+
+I may withdraw biometric consent at any time. Withdrawal:
+
+- Is free of charge
+- Does not affect my ability to remain on the staffing platform
+  (only my biometric data is removed)
+- Triggers permanent destruction of all biometric data within
+  30 days, per the destruction runbook
+- Is recorded as an append-only audit row in my per-subject
+  audit log, providing me with tamper-evident proof of withdrawal
+  if I subsequently exercise my BIPA right of action
+
+⚖ COUNSEL — confirm 30 days is the right destruction SLA. Some
+deployments use 7 or 14 days. The runbook (Gate 5) currently
+references this template's number, so changing it here updates
+both.
+
+---
+
+## 3. Contact for withdrawal / questions
+
+⚖ COUNSEL — supply the candidate-facing contact channel for
+biometric-consent withdrawal. Examples: a dedicated email
+(`biometric-consent@<deployment-domain>`), a postal address, a
+named operator. The contact must be functional from day one of
+deployment.
+
+---
+
+## 4. Consent acknowledgment
+
+By signing below (electronically or on paper), I acknowledge that:
+
+1. I have read and understood the disclosures in §1-3 above
+2. I am providing this consent voluntarily and free of coercion
+3. I have received a copy of this consent template (or have been
+   provided a means to retrieve a copy at any time)
+
+| Field | Value |
+|---|---|
+| Candidate name | _______________________________ |
+| Date | __________ |
+| Signature | _______________________________ |
+| Consent template version | v1 (SHA-256: _generated at deployment time_) |
+
+---
+
+## 5. Operational integration
+
+The structured fields the consent UI must capture and post to
+identityd:
+
+```json
+{
+  "candidate_id": "<token>",
+  "consent_version_hash": "<sha256 of this file at deployment>",
+  "consent_given_at": "<ISO-8601 timestamp>",
+  "consent_collection_method": "<electronic_signature|paper|click_acceptance>",
+  "consent_collection_evidence_path": "<path to signed artifact, if applicable>"
+}
+```
+
+These fields write to `SubjectManifest.consent.biometric.status='given'`
+and the corresponding `SubjectAuditRow` (see
+`crates/catalogd/src/subject_audit.rs`).
+
+---
+
+## 6. Versioning
+
+This consent template is version v1. Per Gate 1's versioning rules,
+any change to the binding disclosure language requires a new version,
+and existing subjects retain their original consent_version reference
+unless they re-consent under the new version.
+
+⚖ COUNSEL — confirm whether existing consent under v1 carries forward
+when the schedule is updated, or whether re-consent is required.
+This affects the deployment workflow.
+
+---
+
+## 7. Authority
+
+| Role | Name | Signature | Date |
+|---|---|---|---|
+| Operator | J | _______________ | _____ |
+| Outside counsel | _____________ | _______________ | _____ |

docs/policies/consent/biometric_retention_schedule_v1.md

@@ -0,0 +1,155 @@
+# Biometric Data Retention Schedule — v1
+
+**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 1 (BIPA §15(a))
+**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before public publication
+**Version:** v1 (initial; supersession requires a new version + new hash)
+**Updated 2026-05-05:** §1 + §2 revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). §5 destruction-trigger endpoint corrected to the shipped catalogd-local route. Pending J's product confirmation of Gate 3b.
+
+> This is a publicly-available retention schedule for biometric identifiers
+> and biometric information collected by the Lakehouse staffing platform.
+> It is required by 740 ILCS 14/15(a) (the Illinois Biometric Information
+> Privacy Act) before any biometric collection from real candidates begins.
+
+---
+
+## 1. What this schedule governs
+
+This schedule applies to:
+
+- **Biometric identifiers** as defined in 740 ILCS 14/10: candidate
+  photographs from which facial geometry is implicit.
+- **Biometric information** as defined in 740 ILCS 14/10: any information
+  derived from a biometric identifier. **In v1 of this schedule, no
+  derived information is collected** — automated facial-classification
+  (gender, race, age inference) is deferred per
+  `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` Option C. If a future version
+  of this schedule introduces classification, that is a superseding
+  v2 schedule with re-consent under the matching v2 consent template.
+
+**Out of scope** (explicitly NOT biometric data under this schedule):
+
+- Synthetic faces from the pre-existing face pool (`data/headshots/`).
+  These are computer-generated portraits, not derived from any real
+  individual, and are not "biometric identifiers" under 740 ILCS 14/10.
+- Candidate names, email addresses, phone numbers, work history,
+  certifications, or any other non-biometric personal information.
+  These are governed by the general PII retention policy referenced
+  in the SubjectManifest substrate (see
+  `docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md`).
+
+---
+
+## 2. Categories collected
+
+| Category | Source | Storage location |
+|---|---|---|
+| Photograph (raw bytes) | Candidate upload via the consent-gated photo endpoint | Quarantined under `data/biometric/uploads/<candidate_id>/<ts>_<uuid>.<ext>`; mode 0700 dir / 0600 file |
+| Photograph integrity hash | SHA-256 of the original bytes | `SubjectManifest.biometric_collection.template_hash` (catalogd JSON manifest at `data/_catalog/subjects/<id>.json`) |
+
+We do NOT collect raw biometric template vectors that could be used
+to re-derive a face from the encoded form. We do NOT run automated
+facial-classification (gender, race, age inference) in v1 — see
+`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` for the deferral rationale.
+The `BiometricCollection.classifications` field on the subject
+manifest exists in the schema but is `None` for every subject.
+
+---
+
+## 3. Purpose of collection
+
+Photographs and the classifications derived from them are used for:
+
+1. **Identity matching during staffing operations.** When a worker
+   arrives at a job site, the assigned coordinator may verify identity
+   by comparing the on-file photograph against the person present.
+2. **Internal record-keeping.** Photographs become part of the worker
+   record so coordinators can recognize repeat workers across multiple
+   placements.
+
+Photographs and biometric classifications are NOT used for:
+
+- Demographic targeting in role recommendations (Title VII / IL Human
+  Rights Act compliance).
+- Training of any machine-learning model.
+- Sharing with third parties, except as required by court order or
+  with the candidate's separate written consent.
+- Any purpose beyond those enumerated in §3.1-3.2 above.
+
+---
+
+## 4. Retention period
+
+Per 740 ILCS 14/15(a), biometric identifiers and biometric information
+must be permanently destroyed when the initial purpose for collection
+has been satisfied OR within **three (3) years** of the individual's
+last interaction with the private entity, whichever occurs first.
+
+**Operational ceiling:** Lakehouse retains biometric data for a
+maximum of **eighteen (18) months** from the candidate's last placement
+or last system interaction, whichever is later. This is more
+restrictive than the BIPA statutory ceiling and provides a safety
+margin against accidental over-retention.
+
+The 18-month clock is enforced by the daily retention sweep
+(`crates/catalogd/src/bin/retention_sweep.rs`), which checks
+`SubjectManifest.consent.biometric.retention_until` on every subject
+and routes overdue subjects to the destruction queue (see Gate 5
+runbook).
+
+⚖ COUNSEL — confirm the 18-month operational ceiling is appropriate
+for the deployment posture, or specify a different number.
+
+---
+
+## 5. Destruction procedure
+
+Per 740 ILCS 14/15(a), Lakehouse follows the **BIPA Destruction
+Runbook** (`docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`) when:
+
+- Retention period under §4 expires
+- Candidate withdraws biometric consent under the consent template (Gate 2)
+- Candidate exercises a right-to-be-forgotten request
+- A catalogd-local `POST /biometric/subject/{id}/erase` is invoked
+  under legal-tier authentication (shipped `848a458`)
+
+Every destruction event is recorded as an append-only audit row in
+the affected subject's per-subject HMAC-chained audit log (see
+`crates/catalogd/src/subject_audit.rs`), providing tamper-evident
+proof of compliant destruction.
+
+---
+
+## 6. Versioning
+
+This schedule is version v1. Future revisions:
+
+- Require a new version number (v2, v3, ...).
+- Are committed to the repository with a `git` history showing the
+  revision date.
+- Are referenced by SHA-256 hash from `consent_versions` table rows
+  in identityd, so each subject's consent record points unambiguously
+  at the schedule version that was in force when consent was given.
+
+**v1 SHA-256:** _generated at deployment time by_ `scripts/staffing/hash_consent_v1.sh` _(to be added when this schedule is finalized by counsel)_
+
+---
+
+## 7. Public availability
+
+⚖ COUNSEL — specify the public URL where this schedule will be
+published (typically the privacy policy page on the deployment site)
+and the disclosure language that links candidates to it from the
+intake UI.
+
+---
+
+## 8. Authority
+
+This schedule is adopted under the authority of J (operator of record)
+and reviewed by ⚖ COUNSEL. Effective date: **TBD pending counsel
+sign-off**.
+
+| Role | Name | Signature | Date |
+|---|---|---|---|
+| Operator | J | _______________ | _____ |
+| Outside counsel | _____________ | _______________ | _____ |

docs/recon/staffing-lakehouse-distillation-recon.md

@@ -0,0 +1,266 @@
+# Staffing Lakehouse × Distillation Substrate — Recon
+
+**Date:** 2026-04-27
+**Status:** Phase 0 (read-only inventory — no implementation yet)
+**Spec:** J's "Lakehouse Staffing Integration" prompt
+**Distillation tag (consumer of):** `distillation-v1.0.0` (commit `e7636f2`)
+
+This document inventories the staffing surface in the Lakehouse repo and identifies where the distillation substrate (Phases 0-8) should attach as a *consumer*. **No distillation core mutation — staffing builds on top.**
+
+The headline finding: **staffing has substantial existing infrastructure but is undocumented as a system.** Validators are scaffolds, scenarios are test fixtures, synthetic data spans 6+ shapes with overlapping intent, and there's no unified staffing audit. The integration work is orchestration over what already exists, not greenfield.
+
+---
+
+## 1. Existing staffing schemas
+
+### Rust validators (`crates/validator/src/staffing/`)
+
+| File | Shape | Status |
+|---|---|---|
+| `mod.rs` | trait + module wiring | scaffold complete |
+| `fill.rs::FillValidator` | validates `{fills: [{candidate_id, name}]}` against Artifact::FillProposal | schema check live; worker-existence + status + geo checks are TODO (commented in source) |
+| `playbook.rs::PlaybookValidator` | validates Artifact::Playbook (operation format, endorsed_names cap, fingerprint) | schema-shape only; no semantic content check |
+| `email.rs` | email-domain validation | scaffold |
+
+### Profiles (`crates/shared/src/profiles/`)
+
+| File | Purpose |
+|---|---|
+| `execution.rs` | execution profile (model routing per task class) |
+| `memory.rs` | MemoryProfile (Phase 19 playbook boost ceiling, history cap, doc stale window, auto-retire) |
+| `observer.rs` | Observer profile (failure cluster size, alert cooldown, ring size, langfuse forward) |
+| `retrieval.rs` | RetrievalProfile (top_k, rerank_top_k, freshness cutoff, boost_playbook_memory, enforce_sensitivity_gates) |
+
+These are **typed** but auditing whether they're enforced at runtime is part of Phase 1 work.
+
+### PII (`crates/shared/src/pii.rs`)
+
+`detect_sensitivity(column_name)` → maps column names to sensitivity classes (`Pii`, `Financial`, `Public`). Verified by tests:
+- `email`, `contact_email`, `ssn` → Pii
+- `salary`, `bill_rate` → Financial
+
+`catalogd::service.rs:264` carries `column_redactions: HashMap<String, Redaction>` per dataset. Catalog enforces, but the audit needs to confirm masking is actually applied at query time.
+
+---
+
+## 2. Synthetic data inventory
+
+| File | Rows | Shape | Status assessment |
+|---|---|---|---|
+| `data/datasets/candidates.parquet` | 1,000 | candidate_id, first_name, last_name, email, phone, city, state, skills, years_experience, hourly_rate_usd, status | **Has PII (raw email + phone)**. CAND-* IDs. status field: `placed`, `unknown others`. Compact + realistic. |
+| `data/datasets/job_orders.parquet` | 15,000 | job_order_id, client_id, title, vertical, bill_rate, pay_rate, status, city, state, zip, description | JO-* IDs, CLI-* clients. Verticals: Admin, Manufacturing(?), etc. Realistic shape. **No candidate-fill linkage table observed.** |
+| `data/datasets/workers_500k.parquet` | 500,000 | worker_id (int), name, role, email, phone, city, state, zip, skills (CSV string), certifications, archetype, reliability/responsiveness/engagement/compliance/availability (0-1 floats), communications (multi-msg string), resume_text | **Largest + richest source.** Has PII. archetype enum (flexible/?). 4-axis personality scores. Resume text + comm log = good RAG/SFT material. |
+| `data/datasets/workers_100k.parquet` | 100,000 | (presumed same as 500k) | scaled-down sibling |
+| `data/datasets/ethereal_workers.parquet` | 10,000 | same as workers_500k schema | scenario-friendly subset |
+| `data/datasets/client_workersi.parquet` | 160 | worker_id, name, role, city, state, email, phone, skills, certifications, availability, reliability, archetype | **Different shape** (no scores beyond reliability+availability, no resume_text). Probably client-side "approved roster" — the worker pool a client has historically used. |
+| `data/datasets/client_workerskjkk.parquet` | (similar) | (same as above) | typo-named sibling — gap to clean up |
+| `data/datasets/sparse_workers.parquet` | 200 | name, phone, role, city, state, notes | **Different shape** — no IDs, no scores, just contact + notes. Looks like edge-case test data (sparse field coverage). |
+| `data/datasets/new_candidates.parquet` | 3 | name, phone, email, city, state, skills, years | Demo / smoke-test data. Tiny. |
+
+**Total worker-shape rows on disk: ~625k** across 5 files. Schema fragmentation (3 distinct shapes) is a real issue — see gap report.
+
+### Scenarios (`tests/multi-agent/scenarios/`)
+
+44 JSON files covering specific staffing days. Sample shape (Heritage Foods Indianapolis 2026-04-23):
+```json
+{ "client": "Heritage Foods", "date": "2026-04-23", "events": [
+  { "kind": "baseline_fill", "at": "10:30", "role": "Machine Operator", "count": 2,
+    "city": "Indianapolis", "state": "IN", "shift_start": "10:30 AM" },
+  { "kind": "recurring", "at": "10:30", "role": "Receiving Clerk", "count": 1, ... }
+]}
+```
+
+Event kinds observed: `baseline_fill`, `recurring`. Cities span Indianapolis, Cincinnati, Madison, Toledo, Detroit, Columbus, etc. — Midwestern + Eastern US.
+
+### Playbook lessons (`data/_playbook_lessons/`)
+
+64 JSON files. Sample shape (Heritage Foods 2026-04-21):
+```json
+{ "date": "...", "client": "...", "cities": "...", "states": "...",
+  "events_total": 5, "events_ok": 3, "checkpoint_count": 2,
+  "model": "gpt-oss:20b", "cloud": false,
+  "lesson": "<long markdown analysis>",
+  "checkpoints": [{ "after": "09:30", "risk": "...", "hint": "..." }, ...] }
+```
+
+These are **post-run retrospectives** — the staffing ops loop wrote them after each scenario completed. Goldmine for RAG.
+
+---
+
+## 3. Ingestion paths + storage layout
+
+### Object storage / Parquet
+- `data/datasets/*.parquet` is the disk-resident store. Treated as input by `ingestd` (CSV/JSON/PDF/Postgres/MySQL ingest in `crates/ingestd`).
+- **No catalog manifests observed for the staffing parquets** (none under `data/_catalog/manifests/` matching candidate/worker/job names). The datasets exist on disk but may not be registered with `catalogd` — gap.
+
+### MariaDB
+- `crates/queryd/src/context.rs` has a "candidates_safe" view referenced by recent code (failed at boot when schema mismatched, see prior memory `feedback_endpoint_probe_discipline.md`).
+- Schema for the views isn't visible from grep — needs DB inspection.
+
+### Vector indexes (`data/vectors/`)
+- `workers_500k_v8.parquet` — vector corpus matched by `staffing_inference_lakehouse` mode in `config/modes.toml`
+- `ethereal_workers_v1.parquet` — alt corpus
+- `entity_brief_v1.parquet` — Chicago-permit-style entity briefs (different domain but same indexer)
+- `chicago_permits_v1.parquet` — separate but uses same machinery
+
+### KB streams that touch staffing
+- `data/_kb/contract_analyses.jsonl` — contractor + permit analyses (related but not staffing per se)
+- `data/_kb/staffers.jsonl` — 1.5K, small, not yet inspected
+- `data/_kb/outcomes.jsonl` — scenario outcomes log (used by Phase 2 transforms in distillation)
+- `data/_playbook_memory/state.json` — Phase 19 playbook memory state
+
+---
+
+## 4. Search / indexing logic
+
+### Staffing-aware mode runner
+`config/modes.toml` defines `staffing_inference` task class:
+```toml
+preferred_mode = "staffing_inference_lakehouse"
+default_model = "openai/gpt-oss-120b:free"
+matrix_corpus = "workers_500k_v8"
+```
+
+The mode runner (Phase 5+ work in this session) composes:
+- `EnrichmentFlags { include_file_content, include_bug_fingerprints, include_matrix_chunks, use_relevance_filter, framing: Staffing }`
+- Pulls top-K from `workers_500k_v8` corpus
+- `FRAMING_STAFFING` system prompt instructs: "only recommend candidates whose names appear in the matrix data; do NOT fabricate workers"
+
+### Pass 4 staffing harness
+`scripts/mode_pass4_staffing.ts` ships synthetic FillRequest payloads through the runner. Each request is a JSON `{city, state, role, count, deadline, notes?}` posted as `file_content` (the runner's input shape). Validation: did the model surface real worker_ids from the corpus, or fabricate.
+
+### What's missing
+- **No "candidate matching" deterministic scorer** beyond mode-runner LLM. Staffing audit should add: given a job_order, can we score worker fit deterministically (skills overlap, geo distance, status filter) BEFORE asking the LLM? Currently the LLM does both retrieval and scoring.
+- **No indexed link table between candidates.parquet and workers_500k.parquet.** They look like the SAME population in different shapes — the workers_500k has the scores + resume + comms, candidates has the basic contact + status + hourly rate. If they're meant to be different populations, the join key is unclear; if they're the same, there's redundancy.
+
+---
+
+## 5. Audit / event tables
+
+**No staffing-specific audit/event log observed.** Searched for `audit_event`, `outcome_event`, `fill_event` patterns in `crates/` — zero hits. The closest existing infrastructure:
+- `data/_kb/outcomes.jsonl` — per-run scenario outcomes (used by distillation transforms)
+- `data/_observer/ops.jsonl` — observer ring buffer (general-purpose, not staffing)
+- `data/_playbook_lessons/*.json` — post-run lessons (retrospective, not audit)
+
+**Gap:** staffing fills happen, scenarios complete, but **no schema-backed event log** captures: which worker_ids were proposed, accepted, filled, rejected, with what timing, against which job_order. The closest record is in scenarios + playbook_lessons but those are unstructured + per-scenario, not a queryable log.
+
+---
+
+## 6. PII / tokenization boundaries
+
+### Detection
+`crates/shared/src/pii.rs::detect_sensitivity` recognizes: `email`, `contact_email`, `ssn`, `phone` → Pii. `salary`, `bill_rate`, `pay_rate` → Financial.
+
+### Enforcement
+`catalogd::service.rs` carries per-dataset `column_redactions: HashMap<String, Redaction>` — but enforcement at query time wasn't visible from initial grep. Auditing whether masking actually happens when `staffing_inference_lakehouse` retrieves from `workers_500k_v8` is in scope.
+
+### Risk
+Raw email + phone live in `workers_500k.parquet` and `candidates.parquet`. If the LLM mode runner retrieves chunks and the catalog hasn't masked them, **the LLM sees PII**. Spec says "do not expose raw PII to AI" — auditing this is non-negotiable for the staffing integration.
+
+---
+
+## 7. PRD docs
+
+- `docs/PRD.md` — main PRD. §32 names staffing as the reference implementation. §158 explicitly notes Phase 19 playbook learning was originally write-only, claims it's now closed — **verify**.
+- `docs/CONTROL_PLANE_PRD.md` — long-horizon vision (2026-04-22 pivot)
+
+PRD references staffing throughout but doesn't itemize a "staffing PRD checklist" the way the auditor's pr_audit mode expects per-PR claims. Drift detection between PRD claims and code reality is exactly the auditor's job — running it on the PRD as input rather than a PR diff is a configuration shift, not new code.
+
+---
+
+## 8. Where distillation outputs should attach
+
+The Phase 0-8 distillation substrate is **already feeding the staffing surface in two places**:
+
+1. **`staffing_inference_lakehouse` mode → `workers_500k_v8` matrix corpus.** This is read-only consumption; no change needed.
+2. **`pr_audit` mode → `lakehouse_answers_v1` corpus.** Generic; not staffing-specific.
+
+**What's missing for staffing:**
+
+a. **Staffing-specific RAG corpus** — `staffing_answers_v1` built from playbook_lessons + scored scenarios. Same builder pattern as `lakehouse_answers_v1` (commit `0844206`'s `scripts/build_answers_corpus.ts`); just point at staffing inputs.
+
+b. **Staffing audit task class** — `staffing_audit` mode in `config/modes.toml`, paralleling the auditor's `pr_audit` work. Reads PRD claims + scenario outcomes, asks "do we ship what the PRD claims for staffing?"
+
+c. **Staffing acceptance fixture** — same shape as `tests/fixtures/distillation/acceptance/` but with synthetic candidate + job_order + scenario + lesson rows. Pins staffing invariants: PII masked, candidates valid, scenarios reproducible.
+
+d. **Staffing replay tasks** — drop sample fill requests through `./scripts/distill replay` to see if the local model proposes real worker_ids vs fabricates.
+
+**Implementation approach (deferred until gap report + J approval):**
+
+```
+scripts/staffing/
+  audit.ts              # ./scripts/staffing audit — single entry
+  build_answers.ts      # build_staffing_answers_v1 from lessons + scenarios
+  build_corpus_v9.ts    # rebuild workers_500k_v9 with PII masking applied
+  acceptance.ts         # staffing-specific 22-invariant gate
+
+tests/fixtures/staffing/
+  candidates_sample.parquet
+  job_orders_sample.parquet
+  scenario_sample.json
+  lesson_sample.json
+
+reports/staffing/
+  staffing-audit-report.md
+  staffing-prd-drift-report.md
+  staffing-search-quality-report.md
+  staffing-synthetic-data-report.md
+```
+
+**ALL of the above is consumer-side.** The distillation pipeline's `scripts/distillation/`, `auditor/schemas/distillation/`, and Phase 0-8 commits are NOT touched.
+
+---
+
+## 9. Risks identified during recon
+
+1. **Synthetic data shape fragmentation** — 3 distinct worker schemas across 5 files. If staffing audit assumes one shape and the system uses another, audits will silently miss.
+2. **PII enforcement unverified.** Catalog has a redaction primitive; whether it's wired to mode-runner retrieval is the audit's first deterministic check.
+3. **No structured staffing audit log.** Lessons + outcomes are retrospective summaries, not per-event records. Without per-event records, deterministic checks like "every worker proposed by the LLM exists in workers_500k" can't run on historical scenarios.
+4. **Validator scaffolds.** `FillValidator::validate` does schema-shape only — the worker-existence/status/geo TODOs in the source are exactly the deterministic gates the staffing audit needs to run. Wiring them is consumer work, not distillation work.
+5. **Fragile PRD ↔ code linkage.** PRD §158 claims Phase 19 closed the playbook write-only gap; no audit verifies. The staffing-prd-drift-report should run an inference-style claim verification against PRD claims, not unlike the auditor's pr_audit but with PRD as the source.
+6. **`workers_500k_v8` is the embedded corpus the LLM sees.** If it carries PII without masking, the LLM has been seeing PII. Auditing the corpus content (not just the SQL views) is required.
+7. **64 playbook_lessons + 44 scenarios = ~108 RAG candidates.** Plenty for a staffing_answers corpus, but PII filtering must apply before vectorization. Currently lessons may contain worker names ("Susan X. Ruiz double-booked").
+
+---
+
+## 10. Recommended integration points (where consumer code attaches)
+
+1. **Staffing audit script** at `scripts/staffing/audit.ts` reads from existing distillation outputs:
+   - `data/scored-runs/` (filter to task_id starting `permit:` or `scenario:`)
+   - `exports/quarantine/*.jsonl` (any staffing-specific quarantines)
+   - `reports/distillation/<latest>/summary.json` (cross-reference)
+
+2. **Reuse Phase 5 receipts harness** — staffing audit writes a `StageReceipt` matching the existing schema, with a new `stage` value (extend the enum to `"staffing-audit"` only after schema-version bump if needed; otherwise use the existing reserved `"index"` slot or just write a parallel manifest under `reports/staffing/`).
+
+3. **Reuse Phase 1 schemas** — RagSample, SftSample, PreferenceSample work for staffing data without modification. The `tags` array can carry `task:staffing.fill` to keep the corpus self-tagged.
+
+4. **Reuse Phase 7 replay** — `./scripts/distill replay --task "fill 2 welders in Toledo OH"` already works; just feed it from synthetic FillRequest payloads.
+
+5. **Reuse Phase 8 audit-full** — its drift baseline tracks distillation metrics; staffing audit gets its OWN baseline file at `data/_kb/staffing_audit_baselines.jsonl`.
+
+6. **Schema invariants for staffing**:
+   - every candidate_id in candidates.parquet appears in workers_500k.parquet OR is documented as "candidate-distinct-from-worker"
+   - every status value in candidates.parquet is in a known enum
+   - every email in workers/candidates is masked when it reaches the LLM (audit by inspecting prompt traces in Langfuse)
+
+---
+
+## 11. What this document is NOT
+
+- Not a green-light to start staffing audit implementation. The spec is explicit: synthetic-data gap report next, THEN J reviews, THEN code.
+- Not an audit itself. This is the inventory — the audit's first run will surface findings.
+- Not a redesign of staffing data shapes. The fragmentation is documented for the gap report; reshape decisions are J's call, not this recon's.
+- Not a modification of the distillation v1.0.0 substrate. Per spec: "DO NOT modify the completed distillation pipeline unless a blocking integration bug is found."
+
+---
+
+## 12. Phase 1 readiness checklist
+
+Before staffing implementation starts, the following must be true:
+
+- [x] Recon doc exists (this file)
+- [ ] Synthetic-data gap report exists (next)
+- [ ] J reviews both before any code change
+- [ ] J approves audit scope + first invariants
+
+Phase 1 is unblocked only after the gap report is reviewed.

docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md

@@ -0,0 +1,244 @@
+# BIPA Biometric Data Destruction Runbook
+
+**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5 (BIPA §15(a))
+**Audience:** Operators (J + named operators with legal-tier credentials)
+**Status:** Engineering scaffold — ⚖ COUNSEL must review for legal sufficiency before adoption
+
+> This runbook tells an operator HOW to destroy biometric data when
+> a destruction trigger fires. It is a procedural document, not a
+> design document. The cryptographic substrate that the destruction
+> writes against (per-subject HMAC audit log + tombstone manifests)
+> already ships in `crates/catalogd/`.
+
+---
+
+## 1. When this runbook fires
+
+Destruction is mandatory when ANY of the following occurs:
+
+| Trigger | Source signal | SLA |
+|---|---|---|
+| **Retention expiry** | Daily `retention_sweep` flags `consent.biometric.retention_until < now` | 30 days from sweep flagging |
+| **Consent withdrawal** | Candidate submits withdrawal per consent template §2 | 30 days from receipt |
+| **Right-to-be-forgotten request** | Candidate submits RTBF request through documented contact channel | 30 days from receipt |
+| **Court-ordered erasure** | Legal counsel directs erasure via a documented order | Per court order; default 30 days |
+
+⚖ COUNSEL — confirm 30 days is correct for all four. Some deployments
+have stricter contractual or jurisdictional clocks (CCPA: 45 days but
+sooner is better; GDPR Art. 17: "without undue delay").
+
+---
+
+## 2. Pre-destruction checks (5 minutes)
+
+Before initiating destruction, the operator MUST:
+
+1. **Verify the trigger.** Cross-reference one of the four sources
+   above. If the trigger is a candidate-initiated request,
+   confirm identity per the standard PII verification procedure
+   (knowledge factor + possession factor; see counsel for the
+   threshold).
+
+2. **Pull the current subject record.** Hit
+   `GET /audit/subject/{candidate_id}` with the legal-tier token.
+   The response includes:
+   - The current `SubjectManifest` (including `consent.biometric.status`)
+   - The full HMAC-chained audit log
+   - `chain_verified: true` (if false, STOP — chain integrity issue
+     must be investigated before destruction)
+
+3. **Check for legal hold.** ⚖ COUNSEL — if a legal hold can apply
+   to a subject's data (litigation, regulatory inquiry, subpoena),
+   document the procedure for checking that no hold is in force
+   before erasing.
+
+4. **Get the second-operator sign-off.** Per BIPA defensibility,
+   destruction is a two-operator action (operator-of-record + one
+   witness). The witness records their attestation in the
+   destruction-event audit row (§4 below).
+
+---
+
+## 3. Destruction procedure
+
+### Step 1 — Erase via identityd
+
+Invoke the legal-tier erasure endpoint:
+
+```bash
+curl -sf -X POST "http://localhost:3100/biometric/subject/${CANDIDATE_ID}/erase" \
+  -H "Authorization: Bearer $(cat /etc/lakehouse/legal_audit.token)" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "scope": "biometric_only|full",
+    "trigger": "retention_expiry|consent_withdrawal|rtbf|court_order",
+    "trigger_evidence_path": "<path to signed artifact>",
+    "operator_of_record": "<operator name>",
+    "witness": "<witness name>"
+  }'
+```
+
+The endpoint is **shipped** (commit `848a458`, 21 unit tests). It is
+served from catalogd-local at `/biometric/subject/{id}/erase` (the
+original v1 spec proposed `/v1/identity/subjects/{id}/erase` under a
+separate identityd daemon — that daemon was collapsed into catalogd
+per the architecture pivot).
+
+The endpoint exposes two scopes:
+
+- **`scope: "biometric_only"`** — clears `BiometricCollection` from
+  the SubjectManifest (drops `data_path`, `template_hash`, and
+  `classifications` together) + securely unlinks the quarantined
+  photo file. Subject manifest itself remains. Use for retention
+  expiry / consent withdrawal where only biometric data must go.
+- **`scope: "full"`** — full subject erasure (manifest + biometric
+  files). Use for court-ordered erasure or full RTBF requests.
+
+In both scopes, the audit row is appended BEFORE photo unlink so
+the chain has legal proof of intent even if the file delete fails
+(transactional rollback on audit failure).
+
+### Step 2 — Append the destruction-event audit row
+
+The erasure endpoint AUTOMATICALLY writes one row to the subject's
+per-subject audit log:
+
+```json
+{
+  "schema": "subject_audit.v1",
+  "ts": "<ISO-8601>",
+  "candidate_id": "<id>",
+  "accessor": {
+    "kind": "biometric_erasure",
+    "daemon": "identityd",
+    "purpose": "biometric_erasure",
+    "trace_id": "<X-Lakehouse-Trace-Id>"
+  },
+  "fields_accessed": ["biometric_classifications", "biometric_data_path", "biometric_template_hash"],
+  "result": "erased",
+  "prev_chain_hash": "<previous row hmac>",
+  "row_hmac": "<new chain link>"
+}
+```
+
+The HMAC chain extends through the erasure event, so the audit
+log itself is preserved as anonymous-event proof of compliant
+destruction even after the underlying biometric data is gone.
+
+### Step 3 — Verify destruction
+
+Run the verification script:
+
+```bash
+./scripts/staffing/verify_biometric_erasure.sh "${CANDIDATE_ID}"
+```
+
+⚖ ENGINEERING — script TODO. Acceptance:
+- Subject row biometric fields are NULL
+- `data/biometric/uploads/${CANDIDATE_ID}/` directory is empty
+- Most recent audit log row has `result: "erased"`, `accessor.kind: "biometric_erasure"`
+- Chain still verifies (`chain_verified: true`) under the legal-tier endpoint
+
+If any check fails: STOP, do not mark the destruction complete,
+escalate to engineering.
+
+### Step 4 — Notify the candidate (when applicable)
+
+For consent-withdrawal and RTBF triggers, the operator notifies
+the candidate that destruction is complete. ⚖ COUNSEL — supply
+the notification template (typically email; medium and language
+are counsel-determined).
+
+---
+
+## 4. Backup window disclosure
+
+Per `IDENTITY_SERVICE_DESIGN.md` v3-B12, biometric data may persist
+in encrypted system backups for up to **30 days** after destruction
+(rolling backup window). The candidate must be informed of this
+when destruction is requested, and the destruction-event audit row
+records the backup-window expiry date so the operator knows when
+the residual is fully eliminated.
+
+⚖ COUNSEL — confirm whether the 30-day backup window is acceptable
+under BIPA. Some interpretations require backups to be addressed
+within a shorter window; some accept the operational reality of
+backup retention.
+
+---
+
+## 5. Reporting cadence
+
+Monthly, the operator-of-record produces a destruction-events
+report:
+
+```bash
+./scripts/staffing/biometric_destruction_report.sh \
+  --month "$(date +%Y-%m)" \
+  --output reports/biometric/destruction_$(date +%Y_%m).md
+```
+
+⚖ ENGINEERING — script TODO. The report aggregates:
+
+- Total destruction events in the month
+- Breakdown by trigger (retention / withdrawal / RTBF / court)
+- Median time-to-destruction from trigger to completion
+- Any failures / escalations
+
+The monthly report is available to outside counsel on request.
+It does NOT include candidate-identifying details — only the
+counts, timings, and cryptographic attestations of the events.
+
+---
+
+## 6. Audit trail attestation
+
+The per-subject HMAC chain is the cryptographic substrate that
+makes destructions defensible after the fact. To produce an
+attestation for a specific candidate's destruction:
+
+1. Hit `GET /audit/subject/{candidate_id}` with legal-tier token
+2. Confirm `chain_verified: true` and most-recent row has
+   `accessor.kind: "biometric_erasure"`
+3. Cross-runtime verify: the same audit log is byte-identical
+   under Rust + Go (per `scripts/cutover/parity/subject_audit_parity.sh`)
+4. Counsel signs an attestation referencing the audit log's
+   chain root hash
+
+The chain root hash is itself a tamper-evident anchor. A motivated
+insider would need the HMAC signing key (held in a separate location
+from the audit logs themselves, per the spec) AND the original
+log to forge a clean destruction record — and the cross-runtime
+parity probe would catch a forgery that touched only one runtime's
+view.
+
+---
+
+## 7. Operator acknowledgment
+
+Operators with legal-tier credentials acknowledge they have read,
+understood, and will follow this runbook before being granted access
+to the legal_audit token.
+
+| Operator | Date acknowledged | Signature |
+|---|---|---|
+| J | _____ | _______________ |
+| _____ | _____ | _______________ |
+
+⚖ COUNSEL — adopt this acknowledgment as the substrate for §3 of
+Phase 1.6 (employee training acknowledgment), or specify a separate
+training program.
+
+---
+
+## 8. Change log
+
+- 2026-05-05 — Endpoint path reconciled with shipped state:
+  `/v1/identity/subjects/{id}/erase` (legacy proposal under a
+  separate identityd daemon) → `/biometric/subject/{id}/erase`
+  (catalogd-local, shipped `848a458`). Step 1 manual-fallback
+  block removed (the endpoint is no longer "TODO"). Two-scope
+  body shape (`biometric_only` / `full`) documented to match
+  the implementation.
+- 2026-05-03 — Initial scaffold. ⚖ COUNSEL review required before
+  adoption.

docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md

@@ -0,0 +1,308 @@
+# Legal-Tier Audit Key & Token Rotation Runbook
+
+**Spec companion:** `docs/PHASE_1_6_BIPA_GATES.md` §2 + `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
+**Audience:** Operators with root on the gateway host (J + named operators)
+**Status:** Engineering-authored — ⚖ counsel review encouraged before formal adoption
+
+> This runbook covers rotation of the two crypto-credentials that gate
+> the Phase 1.6 audit substrate:
+>
+> 1. **`LH_SUBJECT_AUDIT_KEY`** — the 32-byte HMAC-SHA256 signing key
+>    that chains every per-subject audit row. If this key changes, all
+>    pre-rotation chain rows tamper-detect under the new key. That is
+>    correct, expected, BIPA-defensible behavior — the chain integrity
+>    it provided pre-rotation remains intact in the archive of the old
+>    key, and post-rotation chains remain intact going forward.
+>
+> 2. **`LH_LEGAL_AUDIT_TOKEN`** — the 32+-character bearer token that
+>    authorizes calls to `/audit/subject/{id}` and
+>    `/biometric/subject/{id}/erase`. Rotation does NOT touch any audit
+>    history; only access to the legal-tier endpoints flips.
+>
+> Both live at `/etc/lakehouse/` (mode 0400, owned by root) and are
+> loaded by the gateway via systemd `Environment=` directives in
+> `/etc/systemd/system/lakehouse.service.d/audit_env.conf`. They are
+> NOT loaded from `/tmp` — a 2026-05-05 reboot incident wiped a
+> `/tmp`-resident key and caused `/audit` + `/biometric` to fail-closed
+> (which is what they should do); the rotation fix moved them to the
+> persistent path.
+
+---
+
+## 1. When to rotate
+
+Rotate **immediately** when any of the following is true:
+
+| Trigger | Urgency | Notes |
+|---|---|---|
+| Suspected operator credential compromise | Within 1 hour | Token mismatch is fail-closed by default; immediate rotation closes the window. |
+| Operator with legal-tier access leaves the team | Within 24 hours | Treat as compromise. |
+| Key/token file's filesystem permissions were ever weakened (mode > 0400, group readable, etc.) | Within 24 hours | Filesystem audit may have leaked the bytes. |
+| Token was ever transmitted over an untrusted channel (printed in CI log, sent over SMS, etc.) | Within 24 hours | Same reasoning. |
+| Scheduled rotation (recommended) | Every 90 days | BIPA does not mandate a rotation cadence; counsel may set one. |
+
+Do **not** rotate when:
+
+- A subject's audit chain tamper-detects in isolation. That is normal
+  if the audit log was edited (which would itself be the BIPA finding,
+  not the key). Investigate the chain, not the key.
+- Cross-runtime parity drift appears. That's an HMAC-input-shape bug
+  (Go vs Rust serialization), not a key issue. See
+  `STATE_OF_PLAY.md` "three runtime-divergence classes" entry.
+
+---
+
+## 2. Pre-rotation checks (5 minutes)
+
+Before generating new credentials, capture a clean baseline so you can
+prove the rotation cause and sequence afterward.
+
+### 2.1. Take the engineering snapshot
+
+```bash
+# Confirm the canonical files exist with correct permissions.
+ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
+
+# Hash the existing key + token (NEVER the bytes themselves) so the
+# old credential is identifiable in retrospect without storing it.
+sha256sum /etc/lakehouse/subject_audit.key
+sha256sum /etc/lakehouse/legal_audit.token
+
+# Confirm the gateway is currently using these files.
+sudo systemctl cat lakehouse.service | grep -E "Environment.*AUDIT"
+
+# Verify the audit endpoint is healthy with the current credentials.
+curl -sf http://localhost:3100/audit/health
+```
+
+If `/audit/health` is already 503, the rotation is **recovery**, not
+preventive — note this in the rotation event record (§5).
+
+### 2.2. Capture a known-good chain root
+
+Pick one or two subjects with non-empty audit logs and record their
+chain roots **under the current key**:
+
+```bash
+TOKEN=$(cat /etc/lakehouse/legal_audit.token)
+for cid in WORKER-2 WORKER-100; do
+  curl -sf -H "X-Lakehouse-Legal-Token: $TOKEN" \
+    "http://localhost:3100/audit/subject/$cid" \
+  | jq '{cid: .candidate_id, verified: .audit_log.chain_verified, root: .audit_log.chain_root, rows: .audit_log.chain_rows_total}'
+done
+```
+
+Save the output. Post-rotation, those chains will tamper-detect under
+the new key — that is **expected** and the saved snapshot is the proof
+that the chain WAS intact under the old key, before rotation.
+
+---
+
+## 3. Generation + rotation
+
+### 3.1. Generate the new key
+
+```bash
+# 32 random bytes as hex = 64 chars. Either format works for HMAC-SHA256;
+# we follow the existing convention (44-char base64-ish with no padding).
+sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
+  /etc/lakehouse/subject_audit.key.new
+
+sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
+  /etc/lakehouse/legal_audit.token.new
+
+# Sanity: confirm 44-char content + correct mode.
+sudo wc -c /etc/lakehouse/subject_audit.key.new /etc/lakehouse/legal_audit.token.new
+sudo ls -la /etc/lakehouse/*.new
+```
+
+Both must be `mode 0400`, owned by root, exactly **44 chars** (the
+audit endpoint refuses tokens shorter than 32 chars at load — see
+`crates/catalogd/src/audit_endpoint.rs:73`).
+
+### 3.2. Atomic swap
+
+The gateway reads these files **once at boot** (per
+`crates/catalogd/src/audit_endpoint.rs::AuditEndpointState::new` and
+the equivalent for the writer). Atomic mv → restart is required.
+
+```bash
+# Move the old credentials to a quarantine path with timestamp so the
+# old hashes remain identifiable post-rotation.
+TS=$(date -u +%Y%m%dT%H%M%SZ)
+sudo mkdir -p /etc/lakehouse/_archived
+sudo install -d -m 0700 -o root -g root /etc/lakehouse/_archived
+
+sudo mv /etc/lakehouse/subject_audit.key  /etc/lakehouse/_archived/subject_audit.key.$TS
+sudo mv /etc/lakehouse/legal_audit.token  /etc/lakehouse/_archived/legal_audit.token.$TS
+
+sudo mv /etc/lakehouse/subject_audit.key.new  /etc/lakehouse/subject_audit.key
+sudo mv /etc/lakehouse/legal_audit.token.new  /etc/lakehouse/legal_audit.token
+
+sudo ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
+```
+
+### 3.3. Restart the gateway
+
+```bash
+sudo systemctl restart lakehouse.service
+sleep 2
+sudo systemctl status lakehouse.service --no-pager | head -10
+```
+
+Wait for the gateway to bind port 3100 cleanly. If it doesn't, check
+`journalctl -u lakehouse.service -n 50 --no-pager` for the failure
+mode — the most common cause is the new file having wrong mode/owner.
+
+---
+
+## 4. Post-rotation verification (5 minutes)
+
+### 4.1. Health probes
+
+```bash
+# Audit endpoint must be 200, not 503.
+curl -sf http://localhost:3100/audit/health
+# Expect: "audit endpoint ready"
+
+# /v1/health must list the gateway's full provider set.
+curl -sf http://localhost:3100/v1/health | jq '.providers, .worker_count'
+```
+
+### 4.2. Confirm the new token works
+
+```bash
+NEW_TOKEN=$(cat /etc/lakehouse/legal_audit.token)
+curl -sS -o /dev/null -w '%{http_code}\n' \
+  -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
+  http://localhost:3100/audit/subject/WORKER-100
+# Expect: 200
+```
+
+If 401, the file the gateway loaded does NOT match the file you wrote.
+Check ownership / mode / for trailing whitespace differences with
+`hexdump -C /etc/lakehouse/legal_audit.token | head`.
+
+### 4.3. Confirm the new chain works
+
+Append-only chains are key-tied. Any *new* audit row written
+post-rotation is signed under the new key and verifies cleanly:
+
+```bash
+# Issue a /v1/validate call against any worker — it spawns an audit row.
+curl -sf -X POST http://localhost:3100/v1/validate \
+  -H 'Content-Type: application/json' \
+  -d '{"mode":"fill","candidate_id":"WORKER-100","worker_id":"WORKER-100","fields":["exists"]}' >/dev/null
+
+# Read the chain back. Last row must verify under the new key.
+curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
+  http://localhost:3100/audit/subject/WORKER-100 \
+| jq '.audit_log | {verified: .chain_verified, rows: .chain_rows_total, last_kind: .rows[-1].accessor.kind}'
+```
+
+`chain_verified: true` confirms the new key is signing + verifying.
+
+### 4.4. Confirm pre-rotation chains tamper-detect (expected)
+
+```bash
+curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
+  http://localhost:3100/audit/subject/WORKER-2 \
+| jq '.audit_log | {verified: .chain_verified, error: .chain_verification_error}'
+```
+
+For any subject whose chain was written under the old key, this
+returns `chain_verified: false` with an HMAC-mismatch error. **This
+is correct behavior**, not a bug. The old chain was correctly signed
+under the old key and verified under it; the new key cannot retroactively
+verify rows it didn't sign. The pre-rotation snapshot you captured in
+§2.2 is the defensible proof that those rows WERE valid pre-rotation.
+
+If, instead, you see a chain that *should* verify post-rotation
+returning `verified: false`, that's the rotation having gone wrong —
+likely an old-key file that didn't get archived cleanly. Restore from
+`/etc/lakehouse/_archived/<ts>/`, then re-attempt.
+
+---
+
+## 5. Record the rotation event
+
+Append a row to the rotation log:
+
+```bash
+sudo tee -a /etc/lakehouse/_archived/rotation_log.jsonl <<EOF
+{"ts":"$(date -u +%Y-%m-%dT%H:%M:%SZ)","operator":"<your name>","reason":"<scheduled|compromise|cred_loss|recovery>","old_key_sha256":"<hash from §2.1>","new_key_sha256":"$(sha256sum /etc/lakehouse/subject_audit.key | awk '{print $1}')","old_token_sha256":"<hash from §2.1>","new_token_sha256":"$(sha256sum /etc/lakehouse/legal_audit.token | awk '{print $1}')","witness":"<witness name or N/A for routine>"}
+EOF
+
+sudo chmod 0600 /etc/lakehouse/_archived/rotation_log.jsonl
+sudo chown root:root /etc/lakehouse/_archived/rotation_log.jsonl
+```
+
+This file is the operator-side record of when the key changed and why.
+It does NOT contain the key itself — only hashes — so it is safe to
+back up and share with counsel on request.
+
+---
+
+## 6. Recovery from a lost key
+
+If the active `subject_audit.key` is destroyed (filesystem corruption,
+accidental delete, /tmp wipe per the 2026-05-05 incident), the gateway
+will fail-closed at startup:
+
+- `/audit/subject/{id}` → 503 ("audit endpoint disabled (legal token missing)" or equivalent for the signing key)
+- `/biometric/subject/{id}/photo` → 503 (same fail-closed posture)
+
+This is correct behavior — a server that cannot HMAC-sign new audit
+rows must not accept new biometric writes.
+
+**Recovery is rotation.** Generate a new key per §3.1, atomic-swap
+per §3.2, restart per §3.3, verify per §4. Pre-loss chains tamper-detect
+under the new key (the old key is gone — there is no way to verify
+them). Treat the loss event as the BIPA-defensible boundary: pre-loss
+chain verification was provided by the working key; post-loss new
+chains are signed under the new key.
+
+If a counsel-grade attestation of the pre-loss chains is needed, the
+`/etc/lakehouse/_archived/` folder contains the historical hashes;
+combined with the cross-runtime parity probe (Go reader gives the
+same byte-identical view as Rust), the chain history pre-loss is
+preservable as long as the on-disk JSONL files were not also lost.
+
+---
+
+## 7. ⚖ counsel notes
+
+These are areas where counsel may want to opine before this runbook
+is formally adopted:
+
+1. **Rotation cadence.** BIPA itself does not require periodic rotation;
+   counsel may set a 90-day schedule to satisfy a separate compliance
+   posture (SOC2, internal policy).
+2. **Custody of `/etc/lakehouse/_archived/`.** The archived hashes do
+   NOT contain the keys, but the archived raw key files DO. Counsel
+   may want a more aggressive destruction schedule for the raw archived
+   keys — say 1 year — to reduce a long-tail compromise surface.
+3. **Notification obligations on rotation due to compromise.** §1
+   triggers a rotation; §1 does not address whether candidates whose
+   biometric data was protected by the compromised key must be notified.
+   This is a counsel call.
+
+---
+
+## 8. Operator acknowledgment
+
+| Operator | Date acknowledged | Signature |
+|---|---|---|
+| J | _____ | _______________ |
+| _____ | _____ | _______________ |
+
+---
+
+## 9. Change log
+
+- 2026-05-05 — Initial runbook authored after the /tmp wipe incident
+  on the same day (key was at `/tmp/subject_audit.key` and was deleted
+  on reboot, disabling `/audit` + `/biometric` until the key was
+  regenerated at `/etc/lakehouse/subject_audit.key`). Recovery of
+  that incident produced a working procedure; this runbook captures
+  it as the canonical playbook for any future rotation.

docs/specs/GATE_3B_DEEPFACE_DESIGN.md

@@ -0,0 +1,174 @@
+# Gate 3b — Deepface Classification Integration (Design)
+
+**Status:** **DECIDED 2026-05-05 — Option C (defer classifications)** · Original design draft 2026-05-03 morning · **Companion to:** [`PHASE_1_6_BIPA_GATES.md`](../PHASE_1_6_BIPA_GATES.md) Gate 3 · **Depends on:** Gate 3a (photo upload) which is shipped (`f1fa6e4`)
+
+> **Decision summary (2026-05-05):** J accepted Option C. `BiometricCollection.classifications` remains `Option<JSON> = None` in v1. The consent template and retention schedule were revised the same day to remove all "automated facial-classification" language so the disclosed scope matches the implemented scope. If a real product requirement for classifications surfaces later, this doc's Option A (Python subprocess) or Option B (ONNX-in-Rust) is picked up under a v2 consent template + v2 retention schedule.
+
+> **What this is.** Three options for how `BiometricCollection.classifications` (currently `Option<JSON>`, always `None`) gets populated by an automated facial-attribute classifier. Phase 1.6 Gate 3a ships the consent-gated upload + audit chain + transactional rollback; Gate 3b adds the classification step. The substrate is ready — what's missing is the design choice for HOW classification happens.
+>
+> **Why a design doc instead of building it.** The Python sidecar that historically hosted deepface was deliberately dropped from the hot path on 2026-05-02 (commit `ba928b1` in lakehouse: "drop Python sidecar from hot path; AiClient → direct Ollama"). Restoring an in-process Python pipeline would regress that decision. Three viable paths exist with different tradeoffs; this doc lays them out so the choice is deliberate, not accidental.
+
+---
+
+## What deepface produces
+
+`deepface` (https://github.com/serengil/deepface) is a Python library wrapping multiple face-analysis backends (VGG-Face, FaceNet, OpenFace, etc.). Given an image path, it returns a JSON-shaped result with keys for `age`, `gender`, `dominant_race`, and `emotion` (plus per-class probabilities). For Phase 1.6's BIPA-compliance scope we need `age`, `gender`, `dominant_race` — the labels that go into `SubjectManifest.biometric_collection.classifications`.
+
+Per the runbook (Gate 5 destruction §2 step 3), classifications must be:
+- Stored in the SubjectManifest, NOT in a separate analytics table
+- Cleared during BIPA erasure (already wired — `biometric_collection = None` clears them along with `data_path`)
+- NEVER served to LLMs in conversation context (only to authenticated staffer UI)
+
+---
+
+## Option A — Python subprocess per upload (one-shot, no daemon)
+
+Spawn a `python3` subprocess from the Rust biometric endpoint after the photo is written, run a thin script that imports `deepface.DeepFace.analyze`, captures the JSON, exits. No long-running Python process, no IPC, no shared state. The hot path stays Rust + tokio; the subprocess is invoked exactly once per upload.
+
+**Implementation shape** (~50 LOC additional in `biometric_endpoint.rs`):
+
+```rust
+// After photo file is written + manifest committed + audit row appended:
+let cls_path = state.storage_root.join(&safe_id).join("_classifications.json");
+let _ = tokio::process::Command::new("python3")
+    .arg("/etc/lakehouse/deepface_classify.py")
+    .arg(&abs_path)              // input photo path
+    .arg(&cls_path)              // output JSON path
+    .kill_on_drop(true)          // don't leak subprocess on cancellation
+    .status().await;
+// Read cls_path back, populate biometric_collection.classifications.
+// If subprocess failed/timed out: leave classifications=None, log warning.
+// The upload itself succeeded; classification is best-effort.
+```
+
+The Python script (`/etc/lakehouse/deepface_classify.py`, ~30 LOC):
+
+```python
+#!/usr/bin/env python3
+import sys, json
+from deepface import DeepFace
+result = DeepFace.analyze(sys.argv[1], actions=['age','gender','race'], silent=True)
+with open(sys.argv[2], 'w') as f:
+    json.dump(result, f)
+```
+
+**Pros:**
+- Smallest integration: ~80 LOC total
+- Process isolation: a deepface crash can't take down the gateway
+- Easy to disable: rename the script or remove it; classifications just stay `None`
+- Matches Phase 1.6 spec literally (deepface IS the named classifier)
+- Keeps the hot path Rust-only — subprocess is fired AFTER the response goes out (we can spawn it without awaiting and let it complete async)
+
+**Cons:**
+- Per-upload cold start (~3-5 seconds for Python interpreter + deepface model load)
+- Operator must keep `pip install deepface` working on the host (one more dependency to track)
+- The "no Python in hot path" constraint is bent if we await the subprocess inline
+- Reintroduces a Python-dependent code path that the 2026-05-02 sidecar drop was trying to eliminate
+
+**Cost estimate:** 0.5 day to build + 0.5 day to harden timeout/error paths.
+
+---
+
+## Option B — ONNX model in Rust (no Python at all)
+
+Replace deepface entirely with a Rust-native ONNX runtime invocation. Pre-train (or download pre-trained) ONNX models for age + gender + race classification, ship them at `/etc/lakehouse/models/{age,gender,race}.onnx`, run them through `tract` or `ort` (ONNX Runtime Rust bindings). No Python anywhere.
+
+**Implementation shape** (~200 LOC + a model-build pipeline):
+
+```rust
+use ort::{Environment, SessionBuilder, Value};
+let env = Environment::builder().build()?;
+let session = SessionBuilder::new(&env)?.with_model_from_file("/etc/lakehouse/models/age.onnx")?;
+let preprocessed = preprocess_image(&abs_path)?;  // resize, normalize, etc.
+let output = session.run(vec![Value::from_array(preprocessed)?])?;
+let age = decode_age(&output[0])?;
+// repeat for gender + race
+```
+
+**Pros:**
+- Zero Python dependency — fully consistent with the 2026-05-02 sidecar drop
+- Faster per-call: ~100ms inference (no interpreter cold start)
+- Model files are immutable artifacts that can be hashed + signed for provenance
+- Audit-friendly: deterministic output for a given input + model hash
+- BIPA-defensible: model provenance is auditable; no third-party Python library risk
+
+**Cons:**
+- Need pre-trained ONNX models for our three tasks (age regression, gender classification, race classification). Either find existing public models with appropriate licenses OR train ourselves (multi-week)
+- Image preprocessing in Rust is non-trivial (face detection + alignment + crop + normalize). May need an additional model (e.g. RetinaFace ONNX) just for face detection
+- ~200-400 LOC of new code in a domain we haven't worked in before
+- Cross-runtime parity story: Go side would need its own ONNX integration if it ever wants the same surface, OR we define classifications as Rust-only (acceptable since the substrate is Rust-side anyway)
+
+**Cost estimate:** 3-5 days to build + 2 days to validate output quality + license review for any pre-trained models we use. Largest of the three options.
+
+---
+
+## Option C — Defer classifications until a real demand exists
+
+Keep `BiometricCollection.classifications` as `Option<JSON>` permanently `None` until a concrete consumer needs it. The substrate is ready; we don't add the classifier writer.
+
+**What still works without classifications:**
+- BIPA consent gate (Gate 3a) ✓
+- Photo storage with template_hash for integrity (Gate 3a) ✓
+- Audit chain proving collection happened (Gate 3a) ✓
+- Erasure flow (Gate 5) ✓
+- Retention sweep on biometric clock (Step 7) ✓
+
+**What's missing without classifications:**
+- The "demographic data derived from photos" disclosure in the consent template (Gate 2 §1 Disclosure 1) becomes vestigial — we collect photos but don't actually derive any demographic information from them
+- The retention schedule (Gate 1) §2 lists "Facial geometry classifications" as a category we collect; without classifications, that category is empty in practice
+
+**Pros:**
+- Zero engineering today
+- Forces the question "do we actually need classifications?" to be answered by a real product requirement, not by spec inertia
+- BIPA-safer in the most absolute sense: we collect strictly the photo, nothing derived from it
+- If the real product requirement is "let staffers see the photo at job-site verification," classifications add no value (the staffer's eyes do the verification)
+
+**Cons:**
+- Requires updating Gates 1 + 2 to remove the "we derive classifications" language (counsel coordination needed)
+- Loses the structured demographic data that audit-of-discrimination workflows might want (per `IDENTITY_SERVICE_DESIGN.md` v3-A4 disparate-impact analysis)
+- If we later need classifications, we land on Option A or B then anyway — this is a deferral, not a permanent choice
+
+**Cost estimate:** 0.25 day to update Gate 1 + 2 doc language + ⚖ counsel review of the change.
+
+---
+
+## Recommendation
+
+**Option C (defer) is the right answer right now**, conditional on confirming the real product requirement.
+
+Reasoning:
+1. Phase 1.6's load-bearing claim is consent + audit + retention + erasure. Gate 3a + Gate 5 + retention sweep + audit chain provide ALL of those WITHOUT classifications.
+2. The riskiest BIPA position is collecting demographic-derived data without a documented business purpose. If staffers identify candidates by looking at the photo (the actual stated use case), classifications are decorative.
+3. Option A reintroduces the Python dependency the 2026-05-02 wave deliberately removed. Option B is multi-day work for unclear benefit.
+4. The `Option<JSON>` substrate is forward-compatible: if a real demand surfaces (say, disparate-impact analytics J actually wants), we can ship Option A or B then in 1-3 days respectively, and the audit chain naturally captures the introduction event.
+
+**Next action if Option C is chosen:**
+- Update `docs/policies/consent/biometric_consent_template_v1.md` Disclosure 1 to remove the "facial geometry classifications" claim — adjust to "facial photograph for staff identification"
+- Update `docs/policies/consent/biometric_retention_schedule_v1.md` §2 to remove the classifications row
+- Mark Gate 3b in `PHASE_1_6_BIPA_GATES.md` as "deferred — classifications not collected in v1, see GATE_3B_DEEPFACE_DESIGN.md"
+- ⚖ counsel review of the doc changes (consent text changes need their attention regardless)
+
+**Next action if Option A is chosen:** ~1 day to ship; document the Python dependency carve-out in STATE_OF_PLAY's "WHAT NOT TO RELITIGATE" so future sessions don't try to remove it again.
+
+**Next action if Option B is chosen:** schedule a multi-day session, source/license pre-trained ONNX models for the three tasks, build face-detection preprocessing in Rust.
+
+⚖ J — pick A / B / C. The substrate accommodates any choice; the cost is the design-doc → counsel-coordination → engineering loop, which differs by an order of magnitude across the options.
+
+**[2026-05-05] J's decision: Option C.** Reasoning recorded in change log below. Consent + retention doc revisions for Option C shipped same day; counsel review of revised text is the remaining work.
+
+---
+
+## Open questions for J
+
+1. **What's the actual product requirement for classifications?** If the answer is "I don't know yet," that's a strong vote for Option C.
+2. **If classifications: who is the consumer?** Staffer UI? Disparate-impact dashboard? An LLM context (NO — Phase 1.6 §3 forbids this)?
+3. **If Option A: are we OK with `pip install deepface` as a host requirement?** Or does the operator-of-record need a containerized Python sidecar (which puts us close to where we were before 2026-05-02)?
+4. **If Option B: who is the model provenance signer?** ⚖ counsel will want a chain of custody from the model author to the ONNX file on disk.
+
+---
+
+## What this doc is NOT
+
+- Not a final decision. The decision is J's after reading the options.
+- Not a substitute for testing. Whichever option ships, the parity probe `subject_audit_parity.sh` must continue to pass post-change (the audit chain is the BIPA-load-bearing surface; classifications are additive metadata).
+- Not the only Phase 1.6 work remaining. Counsel-side review of Gates 1/2/5 + the §2 attestation signature are the calendar bottleneck. Gate 3b is the engineering bottleneck.

docs/specs/PATHWAY_MEMORY_SPEC.md

@@ -0,0 +1,307 @@
+# Pathway Memory — Specification v1
+
+**Status:** Draft v1 — 2026-05-03 · **Layer:** Decision metadata · **Implementation:** `crates/vectord/src/pathway_memory.rs`
+
+> **What this is.** Pathway Memory is a third metadata layer that exists alongside (not on top of, not under) the data-metadata layer (catalogd's manifests/views/tombstones) and the operational-memory layer (playbook_memory). It tracks **decision patterns**: "this code path, in this task class, produced this outcome." It is the AI-substrate analog of what Iceberg manifests are to data tables — pointers + provenance for things you might want to retrieve again or learn from.
+>
+> **What it is NOT.** It is not a vector database. It is not a knowledge graph. It is not an audit log (audit is per-subject; pathways are per-code-pattern). It is not a cache. It is a small, opinionated metadata layer with a defined wire format, defined access patterns, and defined lifecycle rules.
+>
+> **Why a spec exists.** No standard tells us what this layer is. Each implementation that drifts from this spec creates the wiring confusion currently visible in the system (data/_kb/ JSONLs that don't get consumed, version drift across `workers_500k_v1..v10`, unclear who reads `reducer_summary` vs `final_verdict`). The spec is the standardization contract that lets a single implementation become the canonical one.
+
+---
+
+## 1. Conceptual model
+
+A **pathway** is the identity-equivalence-class of code paths that exhibit similar decision behavior. Two distinct files may share a pathway if they belong to the same crate AND are reviewed under the same task class with the same signal class.
+
+A **trace** is a single observed instance of a pathway: one review pass, with its inputs, intermediate signals, ladder attempts, audit outcome, and final verdict.
+
+A pathway is **identified** by a stable hash of three keys: task class, file prefix, signal class. The hash is `pathway_id`.
+
+A trace is **identified** within a pathway by a generated UUID: `trace_uid`. The pair `(pathway_id, trace_uid)` is globally unique.
+
+Pathways are **append-only by trace**: every new observation creates a new trace under the existing pathway_id. Traces themselves are **mutable in version**: `revise()` produces a new version that supersedes the previous (Mem0-style chain).
+
+Pathways are **retirable** by id: when replay history shows the pattern no longer holds, the pathway is marked retired and excluded from hot-swap. New pathways may emerge with fresh ids if the underlying code/task characteristics shift.
+
+---
+
+## 2. Wire format — `PathwayTrace`
+
+Authoritative struct: `crates/vectord/src/pathway_memory.rs`. The fields below are the SPEC; the Rust struct is one implementation. All fields use JSON-serializable types when persisted.
+
+### 2.1 — Identity fields (immutable per trace)
+
+| Field | Type | Constraint |
+|---|---|---|
+| `pathway_id` | string | SHA256 hex of `task_class | file_prefix | signal_class` — see §3 |
+| `trace_uid` | string | UUID. Empty on legacy traces; populated on insert by SHOULD-implementations from v2+ |
+| `version` | u32 | Mem0 version chain. Default 1 on insert. Bumped on `revise()`. |
+| `parent_trace_uid` | string \| null | trace_uid of the trace this one supersedes. null on root version. |
+| `superseded_at` | string \| null | ISO-8601 UTC. Set when this trace becomes a non-head version. |
+| `superseded_by_trace_uid` | string \| null | trace_uid of the new head when this version is superseded. |
+
+### 2.2 — Identity-source fields (the inputs that fed pathway_id)
+
+| Field | Type | Notes |
+|---|---|---|
+| `task_class` | string | Caller-defined class string (e.g. `scrum_review`, `staffing.fill`, `pr_audit`) |
+| `file_path` | string | Full path of the file under review. Reduced to file_prefix for hashing — see §3.2. |
+| `signal_class` | string \| null | Caller-defined behavior label (e.g. `CONVERGING`, `LOOPING`, `STUCK_RETRY`) |
+
+### 2.3 — Observation fields (what happened)
+
+| Field | Type | Description |
+|---|---|---|
+| `created_at` | RFC 3339 timestamp | When this trace was inserted. |
+| `ladder_attempts` | array of `LadderAttempt` | Each model+rung attempt: `{rung, model, latency_ms, accepted, reject_reason?}`. Order is dispatch order. |
+| `kb_chunks` | array of `KbChunkRef` | Knowledge-base chunks the reviewer was given as context: `{source_doc, chunk_id, cosine_score, rank}`. |
+| `observer_signals` | array of `ObserverSignal` | Behavior labels emitted by observer during this pathway's run: `{class, priors, prior_iter_outcomes}`. |
+| `bridge_hits` | array of `BridgeHit` | External-doc lookups (context7-style): `{library, version, result_summary}`. |
+| `sub_pipeline_calls` | array of `SubPipelineCall` | Calls to other sub-pipelines (extract, validate, etc.) made during this trace. |
+| `audit_consensus` | `AuditConsensus` \| null | Cross-lineage audit outcome if any: `{pass, models, disagreements}`. |
+| `reducer_summary` | string | Free-text summary of what the reducer concluded. **PII risk surface — see §6.3.** |
+| `final_verdict` | string | Free-text verdict label (`accepted`, `rejected`, `needs_review`, etc.). **PII risk surface — see §6.3.** |
+
+### 2.4 — Index fields (the matrix retrieval shape)
+
+| Field | Type | Description |
+|---|---|---|
+| `pathway_vec` | array of f32, length 32 | Bag-of-tokens hash embedding. Algorithm in §4. Fixed dimension per spec version. |
+| `replay_count` | u32 | Number of times this pathway has been replayed via hot-swap. Initial insert is NOT a replay. |
+| `replays_succeeded` | u32 | Replays where the post-replay outcome matched the trace's `final_verdict`. |
+| `retired` | bool | Marked true when probation gate fires (§5.4). Excluded from hot-swap forever. |
+
+### 2.5 — Optional semantic-correctness fields (ADR-021)
+
+| Field | Type | Description |
+|---|---|---|
+| `semantic_flags` | array of `SemanticFlag` | One of: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`. Caller-emitted, not auto-derived. |
+| `type_hints_used` | array of `TypeHint` | Schema/type context fed to the reviewer for this trace: `{source, symbol, type_repr}`. |
+| `bug_fingerprints` | array of `BugFingerprint` | Structural pattern hashes the reviewer caught: `{flag, pattern_key, example, occurrences}`. |
+
+All §2.5 fields are additive — implementations MUST default them to empty when deserializing legacy traces.
+
+---
+
+## 3. Identity algorithms
+
+### 3.1 — `pathway_id`
+
+```
+pathway_id = sha256_hex(task_class || "|" || file_prefix(file_path) || "|" || signal_class_or_empty)
+```
+
+- Concatenation uses literal byte `|` (0x7C) as separator.
+- `signal_class_or_empty` is the empty string when signal_class is null.
+- SHA-256 output is 64 hex chars, lowercase.
+
+### 3.2 — `file_prefix`
+
+```
+file_prefix(path) = first two path segments joined by '/'
+```
+
+- Path separator is forward slash. Implementations on Windows MUST normalize to forward slashes before hashing.
+- Files with fewer than two segments use the full path verbatim.
+- Examples:
+  - `crates/queryd/src/service.rs` → `crates/queryd`
+  - `crates/gateway` → `crates/gateway`
+  - `README.md` → `README.md`
+
+### 3.3 — Why this fingerprint shape
+
+The fingerprint groups files within a crate (same `file_prefix`) under the same pathway when reviewed for the same task class with the same signal class. This is intentional: it lets the matrix index recognize "the same kind of bug appears in this crate" without needing per-file traces. Per-file granularity is preserved in the `file_path` field for retrieval but is not part of the identity.
+
+---
+
+## 4. `pathway_vec` algorithm
+
+Fixed dimension: **32 buckets**. Implementation: deterministic bag-of-tokens hash, normalized.
+
+```
+def build_pathway_vec(trace):
+    buckets = [0.0] * 32
+    tokens = collect_metadata_tokens(trace)  # see token list below
+    for tok in tokens:
+        h = sha256(tok.encode())
+        bucket = int.from_bytes(h[:4], 'big') % 32
+        buckets[bucket] += 1.0
+    norm = sqrt(sum(b*b for b in buckets))
+    return [b/norm if norm > 0 else 0.0 for b in buckets]
+```
+
+**Tokens MUST include**:
+- `task_class:<task_class>`
+- `file_prefix:<file_prefix>`
+- `signal_class:<signal_class_or_empty>`
+- For each `ladder_attempts[i].model`: `model:<model>`
+- For each `kb_chunks[i].source_doc`: `kb_doc:<source_doc>`
+- For each `observer_signals[i].class`: `signal:<class>`
+- For each `bug_fingerprints[i].flag`: `flag:<flag>`
+
+**Tokens MUST NOT include**:
+- `reducer_summary` text (free-form, would dominate the bucket distribution)
+- `final_verdict` text
+- `created_at` (would make every trace's vector unique)
+- candidate_id, name, email, phone, or any subject identifier (PII risk)
+
+**Why 32 dimensions:** consensus-validated round 3 ensemble selected "small metadata tokens, not full JSON." 32 is sufficient to distinguish pathway combinations without requiring an external embedding model. An external embedding model would work too but adds a dependency, failure mode, and drift risk the consensus flagged. **Spec version v1 fixes dim=32. v2 may revisit.**
+
+---
+
+## 5. Lifecycle
+
+### 5.1 — Insert
+
+A trace is inserted by the consumer (typically the scrum pipeline or auditor) calling the implementation's `insert()` API.
+
+- If no pathway with `pathway_id` exists, a new pathway bucket is created.
+- If a pathway exists, the new trace is appended to that bucket's traces list.
+- `trace_uid` is generated (UUID v7 RECOMMENDED for time-orderability).
+- `version` is set to 1.
+- `created_at` is set to the insertion timestamp.
+- `pathway_vec` is computed and stored.
+- `replay_count` and `replays_succeeded` start at 0.
+
+### 5.2 — Revise
+
+A trace is revised by calling `revise(trace_uid, new_fields)`.
+
+- A new trace with a fresh `trace_uid` and `version = previous.version + 1` is inserted.
+- The new trace's `parent_trace_uid` is set to the previous trace's `trace_uid`.
+- The previous trace's `superseded_at` is set to the revision timestamp and `superseded_by_trace_uid` is set to the new trace's `trace_uid`.
+- The previous trace remains in the bucket (history is preserved). Retrieval defaults to head versions only — see §7.
+
+### 5.3 — Replay
+
+A pathway is **eligible for replay** when `retired == false` AND the bucket has at least one head-version trace.
+
+The hot-swap consumer queries by `pathway_id` (or by similarity over `pathway_vec`), retrieves the head trace, applies its `final_verdict` shape to a new but similar input, and then reports back via `record_replay(trace_uid, succeeded: bool)`.
+
+- `replay_count` is incremented on every replay (success or failure).
+- `replays_succeeded` is incremented only on success.
+
+### 5.4 — Probation gate (retirement)
+
+Probation triggers when `replay_count >= 3` AND `success_rate < 0.80`.
+
+When triggered, the pathway is marked `retired = true`. It is excluded from hot-swap forever. If the underlying code/task characteristics genuinely change such that a retired pathway would work again, a NEW pathway with a FRESH id will be created (because the inputs to `pathway_id` would have changed, e.g., new signal_class). Retirement is per-id and irreversible.
+
+`success_rate` = `replays_succeeded / replay_count` (0.0 if `replay_count == 0`).
+
+---
+
+## 6. Access patterns (read + write shapes)
+
+### 6.1 — Write paths (who inserts traces)
+
+Spec-conformant writers:
+- Scrum pipeline (`tests/real-world/scrum_master_pipeline.ts` and downstream consumers) on review acceptance
+- Auditor pipeline (`auditor/`) on verdict assembly
+- Validator (`crates/validator/`) on iterate session completion
+
+Each writer is responsible for setting the identity fields correctly. The implementation MUST reject writes where `task_class` is empty.
+
+### 6.2 — Read paths (who consumes traces)
+
+Spec-conformant readers:
+- **Hot-swap retrieval** — `query_for_hotswap(task_class, file_path, signal_class, k=5)` returns the top-k head-version traces from matching pathways, ordered by `(success_rate desc, replay_count desc)`.
+- **Similarity retrieval** — `query_by_vec(pathway_vec, k=10)` returns the top-k head-version traces by cosine similarity over `pathway_vec`.
+- **Audit / forensic** — `list_versions(trace_uid_root)` returns the full version chain. **Must be authorized** — version chains can include reducer_summary text that may contain PII.
+
+The implementation MUST NOT expose `list_versions` on an unauthenticated route.
+
+### 6.3 — PII surface (load-bearing for audit-trail compliance)
+
+The fields `reducer_summary` and `final_verdict` are free-form strings written from review output. They are **highly likely** to contain PII when the reviewed task class involved real candidates (per `AUDIT_PHASE_1_DISCOVERY.md` §1F + §10/C1). Until subject-redaction-on-write lands, implementations:
+- MUST treat trace bodies as a suspected PII sink for any task class touching real subjects
+- SHOULD redact known PII patterns (candidate_id, email, phone) before persisting
+- SHOULD provide a per-trace `subject_ids[]` top-level field when known, so audit-response queries can filter by subject
+
+Until the redaction layer ships, do not advertise pathway_memory as PII-clean.
+
+---
+
+## 7. Storage representation
+
+The current implementation persists state to a single JSON file at `data/_pathway_memory/state.json`:
+
+```
+{
+  "pathways": {
+    "<pathway_id>": [
+      { "trace_uid": "...", "version": 1, "task_class": "...", ... },
+      { "trace_uid": "...", "version": 2, "parent_trace_uid": "...", ... }
+    ]
+  },
+  "last_updated_at": "2026-04-29T05:43:00Z"
+}
+```
+
+The flat-JSON representation is implementation-specific. A spec-conformant implementation could use SQLite, an append-only JSONL, or a Lance dataset. The spec dictates the wire shape per-trace and the access patterns; the storage backend is the implementation's choice.
+
+**Retrieval defaults**: queries return head versions only by default (a head version is one with `superseded_at == null`). Consumers explicitly opt in to historical versions via a `include_history: true` flag.
+
+---
+
+## 8. Spec boundary — what's stable vs implementation-specific
+
+### Stable (locked in v1; changing requires v2 spec)
+- The `pathway_id` algorithm (SHA256 of `task_class | file_prefix | signal_class`)
+- The `file_prefix` algorithm (first two path segments)
+- `pathway_vec` dimension (32) and token list
+- Field names and types in §2 (additive fields are OK; renames or removals require v2)
+- Lifecycle rules in §5
+- Access pattern names in §6 (`query_for_hotswap`, `query_by_vec`, `list_versions`)
+
+### Implementation-specific (free to change without spec bump)
+- Storage backend (JSON file, SQLite, Lance, etc.)
+- HTTP/gRPC/library API surface
+- Concurrency model
+- In-memory caching strategy
+- The exact text of `reducer_summary` (it's content, not schema)
+- Token-bucket hash function (SHA-256 specified in §4 but any deterministic 32-bit hash is acceptable as long as it's deterministic and well-distributed)
+
+### Reserved for v2 (named so future implementations don't paint into a corner)
+- Cross-pathway similarity (`query_pathways_similar_to_pathway`)
+- External embedding model for `pathway_vec` (replaces the bag-of-tokens hash)
+- Subject-aware indexing (so audit-response can find all pathways involving subject X)
+- Differential privacy on `pathway_vec` aggregations
+
+---
+
+## 9. Open questions
+
+These need decisions before v1 is final-final:
+
+1. **Dimension lock-in**: 32 is round-3 consensus, but is it actually optimal? Worth a benchmark before final v1.
+2. **Subject indexing in v1 vs v2**: deferring to v2 means audit queries today have to grep `reducer_summary` text. Is that acceptable for the staffing-client launch window?
+3. **Cross-runtime parity**: the Go side has its own `internal/pathways/` implementation. As of 2026-05-03 the wire shape matches but no parity probe exists. Add `pathway_parity.sh` to the cross-runtime probe suite.
+4. **Migration from current state.json**: existing 91 traces lack `trace_uid`. Spec says `trace_uid` is empty on legacy traces — should v1 require backfill or accept legacy-trace queries indefinitely?
+
+---
+
+## 10. Why this spec exists
+
+Without it:
+- Every new consumer of pathway_memory invents their own read/write conventions
+- The Go-side reimplementation drifts from the Rust-side wire format
+- Future LLM-assistant work proposes new fields without considering the existing schema
+- Forensic queries grep free-text fields because no canonical retrieval exists
+- The PII concern stays "TODO" because there's no stable surface to gate
+
+With it:
+- One implementation can be canonical (Rust, currently); others can be conformance-tested
+- The wire shape is documented; serialization tests can pin it
+- New fields go through an additive-only review (don't break legacy)
+- The KbContext consumer in the gateway has a stable API to read against
+- The audit-trail PRD has a defined PII surface to protect
+
+This is the standardization the system was missing.
+
+---
+
+## Change log
+
+- 2026-05-03 — v1 initial draft. Codifies what's already in `crates/vectord/src/pathway_memory.rs`. The spec is descriptive of the existing implementation, not prescriptive of new behavior.

docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md

@@ -0,0 +1,266 @@
+# Subject Manifests on Catalogd — Specification v1
+
+**Status:** Draft v1 — 2026-05-03 · **Layer:** Catalogd extension (NOT a separate daemon) · **Implementation:** to be added to `crates/catalogd/src/`
+
+> **What this is.** A small extension to `catalogd` adding a fourth manifest type — `subject` — alongside the existing dataset / view / tombstone / profile types. A subject manifest answers: "for person X, which datasets contain their PII, which views project it safely, what consent + retention applies, and what's the access log."
+>
+> **What it is NOT.** It is not a separate identity daemon. It is not a Postgres-backed identity service. It is not a HashiCorp-Vault-using KEK rotation system. It is not the `IDENTITY_SERVICE_DESIGN.md` v3 design (that doc is over-scoped for local-only — see its deprecation header). It is the smallest spec that gives you a defensible "show me everything we know about person X" capability for EEOC discovery / BIPA compliance, building on primitives catalogd ALREADY ships.
+>
+> **Why it can be small.** Catalogd already has dataset manifests with per-column `is_pii` flags. It already has views with `column_redactions` (working example: `candidates_safe.json`). It already has tombstones for deletes. It already has profiles for per-agent scoping. The audit-trail need is "thread these primitives together by subject identifier" — not "build a new system." That's a ~300-500 LOC extension, not a 17-20 day phase plan.
+
+---
+
+## 1. Conceptual model
+
+A **subject** is a real person whose PII flows through the system. Identified by a stable token (current implementation: `candidate_id` in `workers_500k.parquet`).
+
+A **subject manifest** is a JSON record under `data/_catalog/subjects/<candidate_id>.json` that points at:
+- which datasets contain rows for this subject (via foreign-key reference)
+- which views safely project this subject's data (via existing view manifest names)
+- what consent + retention metadata applies to this subject
+- what access log file holds this subject's audit trail
+
+Subject manifests are written when a subject enters the system AND updated when a subject's consent changes, vertical reclassifies, or retention period expires.
+
+The **audit log** is a per-subject append-only JSONL file at `data/_catalog/subjects/<candidate_id>.audit.jsonl`. Every PII access for that subject writes one row. The file is signed periodically (HMAC chain) for tamper-evidence.
+
+---
+
+## 2. Wire format — `SubjectManifest`
+
+JSON document at `data/_catalog/subjects/<candidate_id>.json`:
+
+```json
+{
+  "schema": "subject_manifest.v1",
+  "candidate_id": "CAND-000001",
+  "created_at": "2026-05-15T12:00:00Z",
+  "updated_at": "2026-05-15T12:00:00Z",
+  "status": "active",
+  "vertical": "general",
+  "consent": {
+    "general_pii": {
+      "status": "given",
+      "version": "v1-2026-05-15",
+      "given_at": "2026-05-15T12:00:00Z"
+    },
+    "biometric": {
+      "status": "never_collected",
+      "retention_until": null
+    }
+  },
+  "retention": {
+    "general_pii_until": "2030-05-15T12:00:00Z",
+    "policy": "4_year_default"
+  },
+  "datasets": [
+    { "name": "workers_500k", "key_column": "candidate_id", "key_value": "CAND-000001" },
+    { "name": "candidates",   "key_column": "candidate_id", "key_value": "CAND-000001" },
+    { "name": "placements",   "key_column": "candidate_id", "key_value": "CAND-000001" },
+    { "name": "timesheets",   "key_column": "candidate_id", "key_value": "CAND-000001" }
+  ],
+  "safe_views": ["workers_safe", "candidates_safe"],
+  "audit_log_path": "data/_catalog/subjects/CAND-000001.audit.jsonl",
+  "audit_log_chain_root": "sha256:..."
+}
+```
+
+### 2.1 — Field semantics
+
+| Field | Required | Notes |
+|---|---|---|
+| `schema` | yes | Always `"subject_manifest.v1"`. Validates parser shape. |
+| `candidate_id` | yes | Subject identifier. Stable token. Same value as appears in dataset key columns. |
+| `status` | yes | `pending_consent` \| `active` \| `withdrawn` \| `retention_expired` \| `erased`. |
+| `vertical` | yes | `unknown` \| `general` \| `healthcare` \| `finance` \| `other`. **Default `unknown`**, fail-closed routing treats unknown as healthcare-equivalent. |
+| `consent.general_pii.status` | yes | `pending_backfill_review` \| `pending_first_contact` \| `given` \| `withdrawn` \| `expired`. |
+| `consent.biometric.status` | yes | `never_collected` \| `pending` \| `given` \| `withdrawn` \| `expired`. |
+| `retention.general_pii_until` | yes | ISO-8601. Drives daily expiration sweep. |
+| `datasets[].name` | yes | References an existing catalogd dataset manifest by name. |
+| `datasets[].key_column` | yes | The column in that dataset that contains the subject's identifier. |
+| `datasets[].key_value` | yes | The specific value (the subject's id within that dataset's namespace). |
+| `safe_views` | yes | Names of existing catalogd view manifests that safely project this subject's data (for non-legal-tier readers). |
+| `audit_log_path` | yes | Relative path to the audit JSONL. |
+| `audit_log_chain_root` | yes | SHA-256 of the most recent HMAC-chained checkpoint of the audit log. Updated by the audit-log writer on every write. |
+
+---
+
+## 3. Audit log format
+
+Per-subject append-only JSONL at `data/_catalog/subjects/<candidate_id>.audit.jsonl`. One row per PII access:
+
+```json
+{
+  "schema": "subject_audit.v1",
+  "ts": "2026-05-15T13:30:00Z",
+  "candidate_id": "CAND-000001",
+  "accessor": {
+    "kind": "gateway_lookup",
+    "daemon": "gateway",
+    "purpose": "fill_validation",
+    "trace_id": "X-Lakehouse-Trace-Id-..."
+  },
+  "fields_accessed": ["name"],
+  "result": "success",
+  "prev_chain_hash": "sha256:...",
+  "row_hmac": "hmac-sha256:..."
+}
+```
+
+### 3.1 — HMAC chain
+
+Each row's `row_hmac` is `HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac)`. The signing key is loaded once at startup from `/etc/lakehouse/subject_audit.key` (mode 0400). The chain root in the subject manifest references the latest row's `row_hmac`.
+
+A tamper-evident verification is one pass:
+```
+verify_chain(subject_id):
+  manifest = read_subject_manifest(subject_id)
+  rows = read_audit_log(subject_id)
+  prev = "GENESIS"
+  for row in rows:
+    expected = hmac_sha256(key, prev || canonicalize(row - row_hmac_field))
+    assert row.prev_chain_hash == prev
+    assert row.row_hmac == expected
+    prev = row.row_hmac
+  assert manifest.audit_log_chain_root == prev
+```
+
+This is local (no S3 Object Lock, no Vault) but tamper-evident: any modification to a past row breaks the chain at that point and all subsequent rows. The signing key being on disk is a real risk surface — operators MUST set the file mode 0400 owner-only and back it up to a separate location from the audit logs themselves (so that a single backup doesn't carry both the ciphertext and the verification material).
+
+### 3.2 — When the audit log is written
+
+Every code path that resolves PII for a subject MUST write an audit row before returning. Specifically:
+- The gateway's tool registry SQL templates (`crates/gateway/src/tools/registry.rs`) — when `search_candidates` / `get_candidate` queries return rows, write one audit row per returned candidate_id
+- The validator's WorkerLookup (`crates/validator/src/staffing/parquet_lookup.rs`) — when a `lookup(candidate_id)` succeeds, write one audit row
+- The audit-response endpoint (when implemented) — when `/audit/subject/{id}` is called, write one row of `kind=audit_response`
+- Any new code path that touches PII
+
+Write failures MUST NOT silently swallow. They MUST be logged at error level (per the existing observability fabric). Write failures MUST NOT block the read — accept the audit gap and flag it for post-hoc review (better to leak a row than block legitimate operations).
+
+---
+
+## 4. The `/audit/subject/{candidate_id}` response
+
+The audit response builds from the subject manifest + audit log + dataset projections:
+
+```json
+{
+  "schema": "subject_audit_response.v1",
+  "candidate_id": "CAND-000001",
+  "generated_at": "2026-05-15T15:00:00Z",
+  "generated_by": "catalogd@hostname",
+  "manifest": { /* the SubjectManifest */ },
+  "datasets": {
+    "workers_500k": {
+      "row_present": true,
+      "safe_view_projection": { /* candidates_safe row for this subject */ }
+    }
+  },
+  "audit_log_window": {
+    "from": "2026-01-01T00:00:00Z",
+    "to": "2026-05-15T15:00:00Z",
+    "rows": [ /* matching audit rows */ ]
+  },
+  "chain_verification": {
+    "verified": true,
+    "rows_checked": 42,
+    "chain_root": "sha256:..."
+  },
+  "completeness_attestation": "all dataset rows + audit log entries within the window per retention policy v1 are included",
+  "signature": "ed25519:..."
+}
+```
+
+The endpoint is auth-gated via a separate legal-tier credential (see §6). The response body is signed with an Ed25519 key separate from the HMAC chain key.
+
+---
+
+## 5. Implementation plan (this is the SMALL plan)
+
+This is the spec; the implementation is a small extension to catalogd. Estimated effort:
+
+| Step | Effort | What |
+|---|---|---|
+| **1** | 0.5d | Add `SubjectManifest` struct + JSON load/save in `crates/catalogd/src/subjects.rs`. Mirror the existing `views.rs` pattern. |
+| **2** | 0.5d | Add `SubjectAuditWriter` with HMAC chain in same file. Key loaded from sealed file at startup. |
+| **3** | 0.5d | Backfill subject manifests from `workers_500k.parquet` rows. ETL: one manifest per row, default `vertical=unknown`, `consent.general_pii.status=pending_backfill_review`. |
+| **4** | 0.5d | Wire the gateway tool registry to write audit rows. One audit row per candidate_id returned by search_candidates / get_candidate. |
+| **5** | 0.5d | Wire the validator WorkerLookup to write audit rows. |
+| **6** | 1d | `/audit/subject/{id}` HTTP endpoint in `crates/catalogd/src/service.rs`. Legal-tier auth. |
+| **7** | 0.5d | Daily retention sweep: subjects whose `retention.general_pii_until` < now AND `status != erased` get marked for review (don't auto-delete; legal needs to approve). |
+| **8** | 0.5d | Cross-runtime parity: Go side reads the same subject manifests + audit logs. Same shapes, same HMAC algorithm. |
+| **Total** | **~4 days** | Compared to 17-20 days for the IDENTITY_SERVICE_DESIGN approach. |
+
+Each step is one commit, one revert path. No new daemons. No cloud infrastructure. No Vault. No S3 Object Lock. No dual-control JWT split-secret ceremony.
+
+---
+
+## 6. Auth model
+
+Local-first, simple, defensible:
+
+- **Service-tier reads** (gateway tool registry resolving candidate names for fill scenarios): authenticated via the existing gateway internal credential. Audit row written.
+- **Legal-tier reads** (`/audit/subject/{id}`): requires a separate credential held in `/etc/lakehouse/legal_audit.token` (mode 0400, owner-only). Operators may load this only when fulfilling a legal request. The token is rotated per a documented runbook (operator + 1 witness; no cryptographic dual-control ceremony required for this scale).
+- **Backups**: subject manifests + audit logs are backed up daily. The HMAC signing key is backed up to a SEPARATE storage location (different physical/network boundary) so a single backup compromise doesn't enable forgery.
+
+This is not as strong as Vault Transit + dual-control JWT + S3 Object Lock external anchoring. It IS strong enough to pass a normal small-business compliance review and to be defensible in IL/IN small-claims-discovery contexts. If the staffing client's contract requires SOC2 Type II or formal HSM, that's a separate phase — but it's strictly additive on top of this v1 spec.
+
+---
+
+## 7. What this spec gives you (load-bearing)
+
+1. **Defensible response to discrimination discovery** (worked example: John Martinez at Warehouse B). The endpoint produces a complete + signed + chain-verified record of every PII access affecting him.
+2. **BIPA-compliant biometric tracking** when real photos arrive. The `consent.biometric` field + retention timeline are first-class, not bolted on later.
+3. **Per-subject right-to-be-forgotten** via cryptographic erasure of the subject manifest's audit-key entry + tombstoning of the candidate's dataset rows. (The ability to verify "row is gone, trail is preserved as anonymous audit-event of the erasure" is what GDPR Art. 17 + CCPA expect.)
+4. **HIPAA vertical routing** via the `vertical` field. Healthcare-vertical subjects (and `unknown` defaults) route to local-only models per PRD line 70 — no PHI to cloud egress.
+5. **Cross-runtime parity** via the simple JSON+HMAC format that Go can read identically.
+
+---
+
+## 8. What this spec does NOT solve (and where it punts)
+
+- **Per-row encryption of dataset PII**: subject manifests + audit logs are local; the underlying `workers_500k.parquet` is not encrypted. If staffing-client contract requires at-rest encryption, that's a separate concern handled at the storage tier (filesystem encryption, S3 SSE).
+- **Right to explanation (GDPR Art. 22 / EU AI Act)**: this spec captures decisions that touched a subject; it does not require the model to explain WHY each decision was made in human-readable form. That's a separate Phase capturing model reasoning.
+- **Adverse-impact statistics**: the comparator-pool snapshot per fill (per `AUDIT_PHASE_1_DISCOVERY` §10/C3) needs its own writer in the fill pipeline. This spec gives you the per-subject record; it doesn't cross-aggregate selection rates by protected class.
+- **External tamper-evidence**: the HMAC chain is local. A motivated insider with access to both the audit log AND the signing key could rewrite history. For the staffing-client scale this is acceptable; for higher-stakes deployments a separate timestamping service or external transparency log would be additive.
+
+---
+
+## 9. Why this is the right shape for J's deployment
+
+- It builds on what catalogd ALREADY ships (manifests, views, tombstones — the Iceberg-shape layer).
+- It runs locally — no cloud infrastructure to license, monitor, audit, or pay for.
+- Its primitives are JSON files an operator can read with `cat` and `jq`. Tamper-evidence works without trusting opaque crypto APIs.
+- Its implementation is days, not weeks. The timeline matches the staffing-client launch window without forcing them to wait on a SaaS-tier identity service that doesn't fit their data residency posture.
+- It is COMPATIBLE with the IDENTITY_SERVICE_DESIGN v3 path if the staffing client later requires SOC2 Type II — the v1 subject manifests can be migrated into a separate identity daemon when the scale demands it. But that's deferred until demand exists, not built speculatively.
+
+This is the LOCAL-FIRST audit trail. It exists because the SaaS-tier version doesn't fit the deployment model J actually has.
+
+---
+
+## 10. Spec boundary
+
+**Stable in v1 (changing requires v2):**
+- File layout: `data/_catalog/subjects/<id>.json` + `data/_catalog/subjects/<id>.audit.jsonl`
+- JSON schemas in §2 + §3 (additive fields OK; renames/removals require v2)
+- HMAC algorithm: HMAC-SHA256 with key from sealed file
+- Chain semantics: `prev_chain_hash` references previous row's `row_hmac`
+- Vertical default: `unknown` with fail-closed routing
+- Consent state machine in §2.1
+- Audit-row write requirements in §3.2
+
+**Implementation-specific (free to change):**
+- Storage backend (file system v1; SQLite or Postgres acceptable as long as JSON shapes round-trip)
+- HTTP endpoint exact shape (body schema is spec; status codes / headers are implementation)
+- Backfill ETL details
+
+**Reserved for v2:**
+- Per-row encryption (when staffing-client contract requires it)
+- External tamper-evidence anchor (when SOC2 Type II in scope)
+- Cross-tenant subject isolation (when multi-tenant in scope)
+
+---
+
+## Change log
+
+- 2026-05-03 — v1 initial draft. Builds on catalogd primitives that already exist. Smaller than IDENTITY_SERVICE_DESIGN v3 by ~4× because it doesn't propose a new daemon.

lakehouse.toml

@@ -3,6 +3,15 @@
 [gateway]
 host = "0.0.0.0"
 port = 3100
+# Coordinator session JSONL — one row per /v1/iterate session for
+# offline DuckDB analysis. Cross-runtime parity with the Go-side
+# [validatord].session_log_path. Set to the SAME path Go validatord
+# writes to so DuckDB queries see one unified longitudinal stream
+# across both runtimes (rows are tagged daemon="gateway" vs
+# daemon="validatord" so producers stay distinguishable). Append-write
+# is atomic at the row sizes both runtimes produce — both daemons
+# co-writing is safe.
+session_log_path = "/tmp/lakehouse-validator/sessions.jsonl"
 
 [storage]
 root = "./data"
@@ -44,12 +53,22 @@ manifest_prefix = "_catalog/manifests"
 # max_rows_per_query = 10000
 
 [sidecar]
-url = "http://localhost:3200"
+# Post-2026-05-02: AiClient talks directly to Ollama; the Python
+# sidecar's hot-path role (~120 LOC of pure Ollama wrappers) was
+# retired. Field name kept for migration compat — value now points
+# at Ollama on :11434. Lab UI + pipeline_lab Python remains as a
+# dev-only tool, NOT on this URL.
+url = "http://localhost:11434"
 
 [ai]
 embed_model = "nomic-embed-text"
-gen_model = "qwen2.5"
-rerank_model = "qwen2.5"
+# Local-tier defaults bumped 2026-04-30: qwen3.5:latest is the
+# stronger local rung in the 5-loop substrate (per
+# project_small_model_pipeline_vision.md). Same JSON-clean property
+# as qwen2.5, more capacity. Ollama still serves both — bump back
+# in this file if a workload regressed.
+gen_model = "qwen3.5:latest"
+rerank_model = "qwen3.5:latest"
 
 [auth]
 enabled = false
@@ -65,14 +84,16 @@ service_name = "lakehouse"
 # let the agent continuously propose + trial HNSW configs and auto-promote
 # winners. Defaults are conservative so it stays out of the way of live
 # search traffic on shared Ollama.
-enabled = true
+enabled = false
 cycle_interval_secs = 120                 # periodic wake if no triggers
 cooldown_between_trials_secs = 10         # min gap between trials
 min_recall = 0.9                          # never promote below this
 max_trials_per_hour = 20                  # hard budget cap
 
 # Model roster — available for profile hot-swap
+# qwen3.5:latest: stronger local rung — JSON-clean, 8K+ context,
+#                 default for gen_model and rerank_model
 # qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks
-# qwen2.5: 7B, 8K context, fast, good for SQL generation
-# mistral: 7B, 8K context, good for general generation
+# qwen2.5: 7B, 8K context, fast — kept loaded for the 2026-04 era
+#          comparison runs; new defaults use qwen3.5:latest
 # nomic-embed-text: 137M, embedding-only, used by all profiles

mcp-server/UX_FIX_PLAN.md

@@ -0,0 +1,94 @@
+# Staffing Co-Pilot UX Fix Plan (devop.live/lakehouse)
+
+**Target file:** `/home/profit/lakehouse/mcp-server/search.html` (239KB, ~4100 lines) — THIS is the
+main dashboard served at `/` (route at `index.ts:1258`; name "search.html" is historical).
+Sibling pages: `console.html` (Walkthrough), `proof.html` (Architecture), `spec.html`, `onboard.html`,
+`profiler.html`, `contractor.html`, `workspaces.html`, `alerts.html`.
+
+**Serving:** Bun `mcp-server/index.ts` on `:3700`, nginx `/lakehouse/` → `127.0.0.1:3700`.
+Restart after edits: HTML is read per-request (`Bun.file`), so **no restart needed for .html edits** —
+just reload the browser. (Restart only if editing index.ts: it's pid 1254, `bun run mcp-server/index.ts`.)
+
+**Constraints:** neo-brutalist, compact, distinct colors, NO fake timers. Surgical edits only.
+Dashboard fires real SQL + /intelligence/* — don't break data wiring. Repo is FROZEN Rust legacy;
+J explicitly authorized work on this surface.
+
+**Base path quirk:** nav uses `var P = location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':''`.
+Relative hrefs (`href="console"`) resolve fine under `/lakehouse/`.
+
+## User's 5 complaints → fixes (line numbers in search.html)
+
+### 1+2. New tabs for reference pages (HIGH)
+- **Top nav lines 594-599**: add `target="_blank" rel="noopener"` to Profiler, Architecture(proof),
+  Spec, Onboard, Alerts, Workspaces. LEAVE 592-593 (Dashboard `href="."`, Walkthrough `href="console"`) same-tab.
+- **Footer line 781**: `<a href="proof">Architecture</a>` → add `target="_blank" rel="noopener"`. Keep Console same-tab.
+- **Dynamic footer lines 1024-1026**: proof link needs `target=_blank rel=noopener` + `.ft` class styling (currently unstyled).
+
+### 3. Collapsible "preview" not discoverable (HIGH)
+Primary = **contract card expandable panel** (markup ~2620-2820, CSS grid-template-rows ~160-220).
+Chevron rotates 180° but hint text "click for full curve…" only shows collapsed; vanishes on expand.
+Fix: chevron always visible+larger(16px); persistent "Expand details"/"Collapse" label in BOTH states;
+cursor:pointer on card header; hover border-left/bg shift; optional localStorage per-card.
+Secondary collapsibles: ⓪ legacy-bridge narrative (~820-900) → default COLLAPSED for returning users
+(localStorage 'seen_intro'); ② Staffer's Console preview (~1400-1500) → make collapsible, "Open full console →".
+
+### 4. Clickable elements don't look clickable (HIGH)
+Add global CSS: `[onclick]{cursor:pointer}` + reusable `.clickable` class with :hover (bg lift #161b22→#1c2333
++ border accent), :active (translateY(1px)), :focus-visible (outline).
+Apply to: insight cards (~1180-1280), alert rows (~1700-1780), worker result rows (~990-1030 — add hover '›' chevron
+for drilldown discoverability per feedback_test_drilldowns), collapsible section headers (~820-830, ~1400-1410),
+action chips/buttons (~740-760, standardize .btn/.chip).
+
+### 5. Layout clutter / flat hierarchy (HIGH)
+8 sections at equal weight: ⓪ Not-a-CRM narrative, ① Live Market, Forecast, ② Staffer Console,
+Market Intelligence, ③ Worker Search, ④ System Activity, ⑤ Substrate Signals.
+3-tier fix: PRIMARY always-visible (Live Market, Worker Search, Staffer Console);
+SECONDARY collapsible-open (Forecast, Market Intelligence); TERTIARY collapse-by-default or relocate
+(⓪ narrative → collapsible banner; ⑤ Substrate Signals → move to proof.html/Architecture; ④ System Activity demote).
+Also: Live Market vs Market Intelligence may overlap — differentiate headers or merge w/ sub-tabs.
+Long card walls → add "show top 5 / show all" truncation.
+
+### Cross-page nav consistency (MEDIUM)
+Each of 8 pages has hardcoded nav with differing link sets/order/labels and unreliable .active.
+Canonical set + order, each page sets .active on its own entry, brand/logo → `href="."`.
+Long-term: extract nav to shared JS snippet served by index.ts to stop drift.
+
+## ✅ SHIPPED 2026-05-30 (live-verified via curl against :3700; HTML served per-request → no restart)
+- **#1+#2 New tabs** — search.html: 6 nav reference links (Profiler/Architecture/Spec/Onboard/Alerts/Workspaces)
+  + footer Architecture + dynamic-footer Architecture now `target="_blank" rel="noopener"` with ↗ glyph.
+  Dashboard + Walkthrough stay same-tab (primary flow). VERIFIED: 7 static new-tab links served; nav listing clean.
+- **#4 Clickability** — global affordance CSS inserted after `.ft a:hover{...}` (NOT `.ft a{margin}` — that anchor
+  was a hallucination): `[onclick]{cursor:pointer}`, `[role=button]`, `.iworker` cursor+hover-lift+active,
+  `details>summary` cursor+hover, global `:focus-visible`. VERIFIED served (affordance_css=1, iworker_cursor=1, summary_hover=1).
+- **#3 partial** — every `<details>` collapsible now has visible hover affordance (cursor+bg). Contract-card toggle NOT yet (below).
+- **Nav consistency** — added Profiler link to onboard/workspaces/alerts (verified profiler=1 each);
+  fixed contractor.html back-link `href="/"`→`href="."` (real bug: `/` = LLM-Team UI on :5000, ejected users out of lakehouse; verified root-bug=0, dot-fixed=1);
+  fixed profiler.html empty `href=""`→relative `.`/`console` (was JS-only/fragile; verified empty-href=0).
+
+> CORRECTION LOG: first edit pass (earlier this session) FAILED silently — old_strings were guessed
+> (`.ft a{margin:0 4px}`, a fake footer string, `ui-copilot/dashboard.html` path). Re-done with grep/Read-verified
+> exact strings. The real dashboard file is `search.html` (route index.ts:1258), NOT `dashboard.html` (orphan) or `ui/`.
+
+## (status note) — rounds 2 & 3 are recorded in the SHIPPED sections below; this block is superseded.
+- #3 contract-card collapse → DONE round 2 (persistent `.card-top-toggle`).
+- #5 declutter (⑤ relocate, ④ demote, ③ disclosure, Market Intelligence clarify, dead CSS) → DONE round 3.
+- Still open: nav shared-component refactor (deferred, needs restart) + optional contract truncation (skipped on purpose). See "DELIBERATELY NOT DONE".
+
+## Recommended execution order
+1. New-tab attrs (lines 594-599, 781, 1024-1026) — trivial, high impact.
+2. Global `.clickable` CSS helper + apply to onclick elements.
+3. Collapsible affordance (chevron+label persistent, cursor, hover) on contract cards + section headers.
+4. Default-collapse ⓪ narrative for returning users (localStorage).
+5. Hierarchy/declutter (tiering, relocate ⑤ to proof, truncation).
+6. Nav consistency pass across sibling files.
+
+## VERIFY before claiming done (J's rules: test drilldowns, no greenwashing)
+- `curl -s localhost:3700/ | grep target=_blank` confirms new-tab attrs present.
+- Load https://devop.live/lakehouse/ in browser: click a contract card (expand+collapse visible),
+  hover a worker row (cursor+chevron), click reference nav (opens new tab), reload (intro stays collapsed).
+- Don't restart the service for HTML-only edits.
+
+## Audit provenance
+5-dimension parallel Explore audit, run wf_c897f421-db1, full JSON at
+`/tmp/claude-0/-home-profit/c6ca0218-678c-4a30-833c-1063e40e7f69/tasks/wm9x4y52d.output`.
+NOTE: line numbers are audit estimates — `grep` to confirm exact lines before each Edit.

mcp-server/alerts.html

@@ -74,6 +74,7 @@ input:focus{border-color:#388bfd}
   <nav>
     <a href=".">Dashboard</a>
     <a href="console">Walkthrough</a>
+    <a href="profiler">Profiler</a>
     <a href="proof">Architecture</a>
     <a href="spec">Spec</a>
     <a href="onboard">Onboard</a>

mcp-server/biometric_dashboard.html

@@ -0,0 +1,353 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Lakehouse — Biometric Ops Dashboard</title>
+<style>
+*{margin:0;padding:0;box-sizing:border-box}
+body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
+a{color:#58a6ff;text-decoration:none}
+a:hover{color:#79c0ff}
+
+.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
+.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
+.bar .actions{display:flex;align-items:center;gap:10px}
+.bar .gen{font-size:11px;color:#7d8590;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace}
+.bar button{background:#21262d;color:#c9d1d9;border:none;padding:6px 12px;font-size:12px;font-family:inherit;cursor:pointer;border-radius:0}
+.bar button:hover{background:#30363d}
+
+.wrap{max-width:1100px;margin:0 auto;padding:24px 20px 60px}
+.screen{display:none}
+.screen.active{display:block}
+
+h2{font-size:18px;color:#e6edf3;font-weight:600;margin:24px 0 10px;letter-spacing:-0.3px}
+h2:first-child{margin-top:0}
+.lede{color:#7d8590;font-size:13px;margin-bottom:16px}
+
+.card{background:#0d1117;border:1px solid #171d27;padding:18px;margin-bottom:14px}
+
+label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
+input[type=password]{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s}
+input[type=password]:focus{outline:none;border-color:#58a6ff}
+
+button.primary{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s}
+button.primary:hover:not(:disabled){background:#388bfd}
+button.primary:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
+
+.tiles{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:10px;margin-bottom:18px}
+.tile{background:#0d1117;border:1px solid #171d27;padding:18px;border-left:3px solid #2d333b}
+.tile.given{border-left-color:#3fb950}
+.tile.withdrawn{border-left-color:#d29922}
+.tile.expired{border-left-color:#da3633}
+.tile.erased{border-left-color:#8b949e}
+.tile.never{border-left-color:#58a6ff}
+.tile.pending{border-left-color:#a371f7}
+.tile.photos{border-left-color:#f85149}
+.tile-label{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:0.6px;margin-bottom:6px}
+.tile-value{font-size:24px;color:#e6edf3;font-weight:600;letter-spacing:-0.5px;font-feature-settings:'tnum';font-variant-numeric:tabular-nums}
+.tile-sub{font-size:11px;color:#545d68;margin-top:4px}
+
+.row-grid{display:grid;grid-template-columns:1fr 1fr;gap:12px}
+@media (max-width:760px){ .row-grid{grid-template-columns:1fr} }
+
+.kv{display:grid;grid-template-columns:200px 1fr;gap:6px;margin:4px 0;font-size:13px}
+.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
+.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
+
+.table{width:100%;border-collapse:collapse;margin-top:8px;font-size:12px}
+.table th{text-align:left;padding:8px 10px;background:#0a0c10;color:#7d8590;text-transform:uppercase;letter-spacing:0.4px;font-size:11px;font-weight:500;border-bottom:1px solid #1f242c}
+.table td{padding:8px 10px;border-bottom:1px solid #161b22;color:#c9d1d9;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;vertical-align:top}
+.table tr:hover td{background:#0a0c10}
+.kind{display:inline-block;padding:2px 8px;font-size:10px;text-transform:uppercase;letter-spacing:0.5px;font-family:'Inter',sans-serif}
+.kind.consent_grant{background:#0c2510;color:#7ee787;border:1px solid #2ea04338}
+.kind.collection{background:#251c0c;color:#ffd166;border:1px solid #d2992238}
+.kind.withdrawal{background:#251c0c;color:#f0883e;border:1px solid #f0883e38}
+.kind.erasure{background:#250d0d;color:#ff7b72;border:1px solid #f8514938}
+.kind.full{background:#250d0d;color:#ffa198;border:1px solid #f8514955}
+
+.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
+.error.show{display:block}
+
+.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
+.foot a{color:#58a6ff}
+
+.empty{padding:24px;text-align:center;color:#545d68;font-size:13px;font-style:italic}
+.kb-pill{display:inline-block;margin-right:18px;font-size:13px}
+.kb-pill .kb-k{color:#7d8590}
+.kb-pill .kb-v{color:#e6edf3;font-weight:600}
+</style>
+</head>
+<body>
+
+<div class="bar">
+  <h1>⚡ Biometric Ops Dashboard</h1>
+  <span class="actions">
+    <span class="gen" id="gen-at">…</span>
+    <button id="refresh">Refresh</button>
+    <button id="logout">Logout</button>
+  </span>
+</div>
+
+<div class="wrap">
+
+  <!-- Token gate -->
+  <div id="screen-token" class="screen active">
+    <h2>Operator authentication</h2>
+    <p class="lede">Read-only ops dashboard over <code>/biometric/stats</code>. Aggregate counts + recent state-change events. Auth via legal-tier token (sessionStorage; clears on tab close).</p>
+    <div class="card">
+      <label for="token-input">Legal audit token</label>
+      <input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
+      <div style="margin-top:14px"><button id="token-submit" class="primary">Continue →</button></div>
+    </div>
+    <div class="error" id="token-error"></div>
+  </div>
+
+  <!-- Dashboard -->
+  <div id="screen-dash" class="screen">
+    <h2 style="margin-top:0">Biometric consent state — by subject</h2>
+    <div class="tiles">
+      <div class="tile never"><div class="tile-label">Never collected</div><div class="tile-value" id="t-never">…</div><div class="tile-sub">no biometric data on record</div></div>
+      <div class="tile pending"><div class="tile-label">Pending</div><div class="tile-value" id="t-pending">…</div><div class="tile-sub">consent in flight</div></div>
+      <div class="tile given"><div class="tile-label">Given</div><div class="tile-value" id="t-given">…</div><div class="tile-sub">active biometric data</div></div>
+      <div class="tile withdrawn"><div class="tile-label">Withdrawn</div><div class="tile-value" id="t-withdrawn">…</div><div class="tile-sub">awaiting destruction (≤30d)</div></div>
+      <div class="tile expired"><div class="tile-label">Expired</div><div class="tile-value" id="t-expired">…</div><div class="tile-sub">retention window passed</div></div>
+      <div class="tile photos"><div class="tile-label">Photos on disk</div><div class="tile-value" id="t-photos">…</div><div class="tile-sub">quarantined uploads</div></div>
+    </div>
+
+    <div class="row-grid">
+      <div>
+        <h2>Subject lifecycle status</h2>
+        <div class="card">
+          <div class="kv">
+            <span class="k">Total subjects</span><span class="v" id="x-total">…</span>
+            <span class="k">Active</span><span class="v" id="x-active">…</span>
+            <span class="k">Pending consent</span><span class="v" id="x-pending-consent">…</span>
+            <span class="k">Withdrawn</span><span class="v" id="x-withdrawn">…</span>
+            <span class="k">Retention expired</span><span class="v" id="x-retexp">…</span>
+            <span class="k">Erased</span><span class="v" id="x-erased">…</span>
+          </div>
+        </div>
+      </div>
+
+      <div>
+        <h2>Retention horizon</h2>
+        <div class="card">
+          <div class="kv">
+            <span class="k">Oldest active retention</span><span class="v" id="x-oldest">—</span>
+            <span class="k">Days until earliest expiry</span><span class="v" id="x-days">—</span>
+            <span class="k">Sweep schedule</span><span class="v">daily 03:00 UTC</span>
+            <span class="k">Sweep unit</span><span class="v">lakehouse-retention-sweep.timer</span>
+          </div>
+        </div>
+      </div>
+    </div>
+
+    <h2>Recent state-change events <span style="color:#7d8590;font-size:12px;font-weight:400">(last 20 across all subjects, newest first)</span></h2>
+    <div class="card" style="padding:0">
+      <table class="table">
+        <thead>
+          <tr>
+            <th>Timestamp</th>
+            <th>Candidate</th>
+            <th>Kind</th>
+            <th>Result</th>
+            <th>Trace ID</th>
+          </tr>
+        </thead>
+        <tbody id="events-tbody">
+          <tr><td colspan="5" class="empty">Loading…</td></tr>
+        </tbody>
+      </table>
+    </div>
+
+    <h2>Event-kind breakdown <span style="color:#7d8590;font-size:12px;font-weight:400">(across recent events shown above)</span></h2>
+    <div class="card">
+      <div id="kind-breakdown">…</div>
+    </div>
+
+    <div class="error" id="dash-error"></div>
+  </div>
+</div>
+
+<div class="foot">
+  <a href="/biometric/intake">Intake UI</a>
+  · <a href="/biometric/withdraw">Withdraw UI</a>
+  · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">BIPA Gates</a>
+  · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
+</div>
+
+<script>
+const GATEWAY = (function(){
+  const p = new URLSearchParams(location.search);
+  return p.get('gw') || 'http://localhost:3100';
+})();
+
+const state = { token: null, refreshTimer: null };
+const REFRESH_MS = 15000;
+
+function show(id) {
+  document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
+  document.getElementById(id).classList.add('active');
+}
+function err(id, msg){ const e=document.getElementById(id); e.textContent=msg; e.classList.add('show'); }
+function clearErr(id){ document.getElementById(id).classList.remove('show'); }
+function fmtDate(iso){
+  if (!iso) return '—';
+  try { return new Date(iso).toISOString().replace('T',' ').replace(/\.\d+Z$/, 'Z'); }
+  catch(e){ return iso; }
+}
+
+// Build a TD element with text content. textContent prevents any HTML
+// in the value from being interpreted — important because trace_id and
+// (in theory) other audit-row fields could contain operator-supplied
+// strings. The prior version used innerHTML string concatenation; that
+// was an XSS vector if a maliciously-crafted X-Lakehouse-Trace-Id
+// landed in the audit log.
+function td(value) {
+  const el = document.createElement('td');
+  el.textContent = value == null ? '—' : String(value);
+  return el;
+}
+function kindBadge(kind) {
+  const map = {
+    'biometric_consent_grant': 'consent_grant',
+    'biometric_collection': 'collection',
+    'biometric_consent_withdrawal': 'withdrawal',
+    'biometric_erasure': 'erasure',
+    'full_erasure': 'full',
+  };
+  const cls = map[kind] || '';
+  const span = document.createElement('span');
+  span.className = 'kind ' + cls;
+  span.textContent = String(kind || '').replace(/^biometric_/, '');
+  const cell = document.createElement('td');
+  cell.appendChild(span);
+  return cell;
+}
+
+(function init(){
+  const saved = sessionStorage.getItem('lh_legal_token');
+  if (saved) {
+    state.token = saved;
+    document.getElementById('token-input').value = '••••••••';
+    show('screen-dash');
+    refresh();
+    state.refreshTimer = setInterval(refresh, REFRESH_MS);
+  }
+})();
+
+document.getElementById('token-submit').addEventListener('click', () => {
+  clearErr('token-error');
+  const v = document.getElementById('token-input').value.trim();
+  const tok = v === '••••••••' ? state.token : v;
+  if (!tok || tok.length < 32) { err('token-error', 'Token must be ≥32 characters.'); return; }
+  state.token = tok;
+  sessionStorage.setItem('lh_legal_token', tok);
+  show('screen-dash');
+  refresh();
+  state.refreshTimer = setInterval(refresh, REFRESH_MS);
+});
+
+document.getElementById('refresh').addEventListener('click', refresh);
+document.getElementById('logout').addEventListener('click', () => {
+  sessionStorage.removeItem('lh_legal_token');
+  state.token = null;
+  if (state.refreshTimer) { clearInterval(state.refreshTimer); state.refreshTimer = null; }
+  document.getElementById('token-input').value = '';
+  show('screen-token');
+});
+
+async function refresh(){
+  clearErr('dash-error');
+  try {
+    const r = await fetch(`${GATEWAY}/biometric/stats`, {
+      headers: { 'X-Lakehouse-Legal-Token': state.token },
+    });
+    if (!r.ok) {
+      const body = await r.text();
+      err('dash-error', `Stats fetch failed: HTTP ${r.status} ${body.substring(0,200)}`);
+      return;
+    }
+    const s = await r.json();
+    render(s);
+  } catch (e) {
+    err('dash-error', `Network error: ${e.message}`);
+  }
+}
+
+function render(s) {
+  document.getElementById('gen-at').textContent = 'as of ' + fmtDate(s.generated_at);
+  document.getElementById('t-never').textContent = s.biometric_status.never_collected;
+  document.getElementById('t-pending').textContent = s.biometric_status.pending;
+  document.getElementById('t-given').textContent = s.biometric_status.given;
+  document.getElementById('t-withdrawn').textContent = s.biometric_status.withdrawn;
+  document.getElementById('t-expired').textContent = s.biometric_status.expired;
+  document.getElementById('t-photos').textContent = s.photos_on_record;
+
+  document.getElementById('x-total').textContent = s.total_subjects;
+  document.getElementById('x-active').textContent = s.subject_status.active;
+  document.getElementById('x-pending-consent').textContent = s.subject_status.pending_consent;
+  document.getElementById('x-withdrawn').textContent = s.subject_status.withdrawn;
+  document.getElementById('x-retexp').textContent = s.subject_status.retention_expired;
+  document.getElementById('x-erased').textContent = s.subject_status.erased;
+
+  document.getElementById('x-oldest').textContent = fmtDate(s.oldest_active_retention_until);
+  document.getElementById('x-days').textContent =
+    s.upcoming_destruction_window_days != null ? s.upcoming_destruction_window_days + ' days' : '—';
+
+  // Build events table via DOM nodes (textContent on each cell) — never
+  // innerHTML with audit-row values.
+  const tbody = document.getElementById('events-tbody');
+  tbody.replaceChildren();
+  if (!s.recent_events.length) {
+    const tr = document.createElement('tr');
+    const tdEmpty = document.createElement('td');
+    tdEmpty.colSpan = 5;
+    tdEmpty.className = 'empty';
+    tdEmpty.textContent = 'No state-change events recorded yet. Run an intake or withdrawal to see audit chain entries here.';
+    tr.appendChild(tdEmpty);
+    tbody.appendChild(tr);
+  } else {
+    for (const e of s.recent_events) {
+      const tr = document.createElement('tr');
+      tr.appendChild(td(fmtDate(e.ts)));
+      tr.appendChild(td(e.candidate_id));
+      tr.appendChild(kindBadge(e.kind));
+      tr.appendChild(td(e.result));
+      tr.appendChild(td(e.trace_id || '—'));
+      tbody.appendChild(tr);
+    }
+  }
+
+  // Same DOM-build for the kind breakdown.
+  const kb = document.getElementById('kind-breakdown');
+  kb.replaceChildren();
+  const counts = s.recent_event_counts || {};
+  const keys = Object.keys(counts);
+  if (!keys.length) {
+    const span = document.createElement('span');
+    span.style.color = '#7d8590';
+    span.style.fontSize = '12px';
+    span.style.fontStyle = 'italic';
+    span.textContent = 'No events yet.';
+    kb.appendChild(span);
+  } else {
+    for (const k of keys.sort()) {
+      const wrap = document.createElement('span');
+      wrap.className = 'kb-pill';
+      const kEl = document.createElement('span');
+      kEl.className = 'kb-k';
+      kEl.textContent = k;
+      const sep = document.createTextNode(': ');
+      const vEl = document.createElement('strong');
+      vEl.className = 'kb-v';
+      vEl.textContent = counts[k];
+      wrap.append(kEl, sep, vEl);
+      kb.appendChild(wrap);
+    }
+  }
+}
+</script>
+</body>
+</html>

mcp-server/biometric_intake.html

@@ -0,0 +1,482 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Lakehouse — Biometric Consent Intake</title>
+<style>
+*{margin:0;padding:0;box-sizing:border-box}
+body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
+a{color:#58a6ff;text-decoration:none}
+a:hover{color:#79c0ff}
+
+.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
+.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
+.bar .cid{font-size:12px;color:#545d68;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace}
+.bar .step{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:1px}
+
+.wrap{max-width:760px;margin:0 auto;padding:28px 20px 60px}
+.screen{display:none}
+.screen.active{display:block}
+
+h2{font-size:20px;color:#e6edf3;font-weight:600;margin-bottom:8px;letter-spacing:-0.3px}
+h3{font-size:14px;color:#e6edf3;font-weight:600;margin:16px 0 8px}
+.lede{color:#7d8590;font-size:13px;margin-bottom:24px}
+
+.card{background:#0d1117;border:1px solid #171d27;padding:20px;margin-bottom:16px}
+.card.warn{border-left:3px solid #d29922}
+.card.bad{border-left:3px solid #f85149}
+.card.good{border-left:3px solid #3fb950}
+
+label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
+input[type=text],input[type=password]{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s}
+input[type=text]:focus,input[type=password]:focus{outline:none;border-color:#58a6ff}
+input[type=file]{color:#7d8590;font-size:12px;margin-top:8px}
+input[type=checkbox]{margin-right:8px;transform:scale(1.2)}
+
+.checkbox-row{display:flex;align-items:flex-start;color:#e6edf3;font-size:14px;margin:16px 0;cursor:pointer;text-transform:none;letter-spacing:0}
+.checkbox-row input{margin-top:3px}
+
+button{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s;margin-right:8px}
+button:hover:not(:disabled){background:#388bfd}
+button:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
+button.secondary{background:#21262d;color:#c9d1d9}
+button.secondary:hover:not(:disabled){background:#30363d}
+button.danger{background:#da3633}
+button.danger:hover:not(:disabled){background:#f85149}
+
+.consent-text{background:#0a0c10;border:1px solid #1f242c;padding:20px;max-height:340px;overflow-y:auto;font-size:13px;line-height:1.7;margin-bottom:16px}
+.consent-text h4{color:#e6edf3;font-size:14px;font-weight:600;margin:14px 0 6px}
+.consent-text h4:first-child{margin-top:0}
+.consent-text p{margin-bottom:8px;color:#b0b8c4}
+.consent-text ul{margin:6px 0 8px 20px;color:#b0b8c4}
+
+.row{display:flex;gap:12px;align-items:center}
+.row > *{flex:1}
+.row > button{flex:0 0 auto}
+
+.preview{max-width:280px;max-height:280px;display:block;margin:12px 0;border:2px solid #30363d}
+video{max-width:280px;max-height:280px;display:block;margin:12px 0;border:2px solid #30363d;background:#000}
+canvas{display:none}
+
+.hash{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:11px;color:#7ee787;word-break:break-all;background:#0a0c10;padding:8px 12px;border:1px solid #1f242c;margin:6px 0}
+.kv{display:grid;grid-template-columns:140px 1fr;gap:8px;margin:6px 0;font-size:13px}
+.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
+.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
+
+.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
+.error.show{display:block}
+.error code{color:#fff}
+
+.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
+
+.spec-link{font-size:11px;color:#545d68;letter-spacing:0.4px;text-transform:uppercase}
+.spec-link a{color:#58a6ff}
+</style>
+</head>
+<body>
+
+<div class="bar">
+  <h1>⚡ Biometric Consent Intake</h1>
+  <span class="step" id="step-label">step 1 of 4</span>
+  <span class="cid" id="cid-display">…</span>
+</div>
+
+<div class="wrap">
+
+  <!-- Step 1: operator auth -->
+  <div id="screen-token" class="screen active">
+    <h2>Operator authentication</h2>
+    <p class="lede">Paste the legal-tier audit token + your name. Both stored in this tab's session only; cleared on close. Never persists to disk. Your name is recorded as <code>operator_of_record</code> in the audit row for legal traceability.</p>
+    <div class="card">
+      <label for="token-input">Legal audit token</label>
+      <input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
+      <label for="op-name" style="margin-top:14px">Your name (operator of record)</label>
+      <input type="text" id="op-name" placeholder="First Last" autocomplete="off">
+      <div style="margin-top:14px">
+        <button id="token-submit">Continue →</button>
+      </div>
+    </div>
+    <div class="error" id="token-error"></div>
+  </div>
+
+  <!-- Step 2: consent -->
+  <div id="screen-consent" class="screen">
+    <h2>Biometric Information Consent — v1</h2>
+    <p class="lede">Please read the disclosures below. Sign by checking the box and entering your printed name. The candidate-facing surface required by BIPA §15(b)(1)-(3).</p>
+
+    <div class="consent-text">
+      <h4>Disclosure 1 — Notice of collection (§15(b)(1))</h4>
+      <p>Lakehouse will collect and store your <strong>biometric identifier</strong> (a photograph of you from which facial geometry is implicit). The photograph itself is the data we keep — we do <strong>NOT</strong> run automated facial-classification (gender / race / age inference) against it in v1. If at a later date we add automated classification, we will re-collect consent under a superseding template before doing so.</p>
+
+      <h4>Disclosure 2 — Specific purpose and length of term (§15(b)(2))</h4>
+      <p>The biometric data will be used for:</p>
+      <ul>
+        <li>Identity verification at staffing job sites</li>
+        <li>Internal record-keeping so coordinators can recognize you across placements</li>
+      </ul>
+      <p>Retained for a maximum of <strong>18 months</strong> from your most recent interaction, then permanently destroyed per the <a href="#" onclick="alert('See docs/policies/consent/biometric_retention_schedule_v1.md')">Biometric Retention Schedule v1</a>.</p>
+      <p>You may withdraw at any time by contacting the operator. Withdrawal triggers permanent destruction.</p>
+
+      <h4>Disclosure 3 — Written release (§15(b)(3))</h4>
+      <p>By checking the box below and providing your printed name, you provide a written release authorizing Lakehouse to collect, store, and use your biometric identifier and biometric information for the purposes and term stated above.</p>
+
+      <h4>What you're agreeing to (plain language)</h4>
+      <p>If you upload a photo of yourself, we'll keep that photo so your staffing coordinator can recognize you when you arrive at job sites. We don't run automated guesses about your age, gender, or race against the photo. We don't sell it. We don't share it outside the staffing operation unless legally compelled. You can withdraw consent at any time.</p>
+    </div>
+
+    <div class="card">
+      <label for="cand-name">Candidate printed name</label>
+      <input type="text" id="cand-name" placeholder="First Last" autocomplete="off">
+
+      <label class="checkbox-row" style="margin-top:16px">
+        <input type="checkbox" id="cand-agree">
+        <span>I have read and understood the disclosures above. I am providing this consent voluntarily and free of coercion.</span>
+      </label>
+
+      <div class="kv" style="margin-top:14px">
+        <span class="k">Method</span><span class="v">click_acceptance</span>
+        <span class="k">Version hash</span><span class="v" id="version-hash-display">computing…</span>
+        <span class="k">Timestamp</span><span class="v" id="timestamp-display">on submit</span>
+      </div>
+
+      <div style="margin-top:16px">
+        <button id="grant-btn" disabled>Grant consent + continue →</button>
+      </div>
+    </div>
+    <div class="error" id="consent-error"></div>
+  </div>
+
+  <!-- Step 3: photo -->
+  <div id="screen-photo" class="screen">
+    <h2>Photo capture</h2>
+    <p class="lede">Take or upload a clear photo. Stored quarantined under <code>data/biometric/uploads/</code> with mode 0700/0600 + SHA-256 integrity hash. Audit chain records the upload event.</p>
+
+    <div class="card">
+      <h3>Option A — File upload</h3>
+      <input type="file" id="photo-file" accept="image/jpeg,image/png">
+
+      <h3 style="margin-top:24px">Option B — Camera capture</h3>
+      <button id="cam-start" class="secondary">Start camera</button>
+      <button id="cam-capture" class="secondary" style="display:none">📸 Capture</button>
+      <button id="cam-stop" class="secondary" style="display:none">Stop</button>
+      <video id="cam" autoplay playsinline muted></video>
+      <canvas id="cam-canvas"></canvas>
+
+      <h3 style="margin-top:24px">Preview</h3>
+      <img id="preview" class="preview" style="display:none">
+      <p id="no-preview" class="lede" style="margin:12px 0">No photo selected yet.</p>
+
+      <div style="margin-top:16px">
+        <button id="upload-btn" disabled>Upload photo + finish →</button>
+        <button id="skip-btn" class="secondary">Skip photo (consent only)</button>
+      </div>
+    </div>
+    <div class="error" id="photo-error"></div>
+  </div>
+
+  <!-- Step 4: done -->
+  <div id="screen-done" class="screen">
+    <div class="card good">
+      <h2>✓ Intake complete</h2>
+      <p class="lede" style="margin:8px 0 0">Audit chain rows (HMAC-SHA256, persisted to <code>data/_catalog/subjects/&lt;id&gt;.audit.jsonl</code>):</p>
+    </div>
+
+    <div class="card">
+      <h3>Consent grant</h3>
+      <div class="kv">
+        <span class="k">Status</span><span class="v" id="done-status">Given</span>
+        <span class="k">Given at</span><span class="v" id="done-given-at">…</span>
+        <span class="k">Retention until</span><span class="v" id="done-retention">…</span>
+      </div>
+      <h3>Audit hmac</h3>
+      <div class="hash" id="done-consent-hmac">…</div>
+    </div>
+
+    <div class="card" id="photo-card" style="display:none">
+      <h3>Photo upload</h3>
+      <div class="kv">
+        <span class="k">Data path</span><span class="v" id="done-data-path">…</span>
+        <span class="k">Template SHA-256</span><span class="v" id="done-template-hash">…</span>
+      </div>
+      <h3>Audit hmac</h3>
+      <div class="hash" id="done-photo-hmac">…</div>
+    </div>
+
+    <div class="card warn">
+      <h3>Verification</h3>
+      <p>Operator: confirm the audit chain by running:</p>
+      <div class="hash" id="verify-cmd">./scripts/staffing/verify_biometric_erasure.sh &lt;candidate_id&gt;</div>
+      <p style="margin-top:8px">Or hit <code>GET /audit/subject/&lt;id&gt;</code> with legal-tier auth to read the full chain.</p>
+    </div>
+
+    <div style="margin-top:16px">
+      <button onclick="location.reload()" class="secondary">Start another intake</button>
+    </div>
+  </div>
+
+</div>
+
+<div class="foot">
+  <span class="spec-link">
+    <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">Phase 1.6 BIPA Gates</a>
+    · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/policies/consent/biometric_consent_template_v1.md">Consent template v1</a>
+    · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
+  </span>
+</div>
+
+<script>
+// ── Config ────────────────────────────────────────────────────────
+// Gateway base URL. Same-origin if the page is served behind a reverse
+// proxy; otherwise the operator's local gateway.
+const GATEWAY = (function(){
+  const p = new URLSearchParams(location.search);
+  return p.get('gw') || 'http://localhost:3100';
+})();
+
+// Candidate ID from the URL. Required.
+const CANDIDATE_ID = (function(){
+  const p = new URLSearchParams(location.search);
+  return p.get('candidate_id') || p.get('cid') || '';
+})();
+
+// ── State ─────────────────────────────────────────────────────────
+const state = {
+  token: null,                 // sessionStorage-backed
+  operator: null,              // operator_of_record — captured at token paste
+  consentVersionHash: null,    // SHA-256 of the rendered consent template
+  consentResp: null,           // server response from /consent
+  photoBlob: null,             // captured/selected photo
+  photoResp: null,             // server response from /photo
+};
+
+// ── Helpers ───────────────────────────────────────────────────────
+function show(screen) {
+  document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
+  document.getElementById(screen).classList.add('active');
+  const stepMap = {'screen-token':'step 1 of 4', 'screen-consent':'step 2 of 4', 'screen-photo':'step 3 of 4', 'screen-done':'step 4 of 4'};
+  document.getElementById('step-label').textContent = stepMap[screen] || '';
+}
+function err(elId, msg) {
+  const e = document.getElementById(elId);
+  e.textContent = msg;
+  e.classList.add('show');
+}
+function clearErr(elId) {
+  document.getElementById(elId).classList.remove('show');
+}
+async function sha256Hex(text) {
+  const enc = new TextEncoder().encode(text);
+  const buf = await crypto.subtle.digest('SHA-256', enc);
+  return Array.from(new Uint8Array(buf)).map(b => b.toString(16).padStart(2,'0')).join('');
+}
+
+// ── Init ──────────────────────────────────────────────────────────
+(async function init(){
+  if (!CANDIDATE_ID) {
+    err('token-error', 'Missing candidate_id in URL. Append ?candidate_id=WORKER-XXX');
+    document.getElementById('token-submit').disabled = true;
+    return;
+  }
+  document.getElementById('cid-display').textContent = CANDIDATE_ID;
+
+  // Restore token + operator from sessionStorage if previously set in this tab.
+  const saved = sessionStorage.getItem('lh_legal_token');
+  const savedOp = sessionStorage.getItem('lh_operator_name');
+  if (saved) {
+    state.token = saved;
+    document.getElementById('token-input').value = '••••••••';
+  }
+  if (savedOp) {
+    state.operator = savedOp;
+    document.getElementById('op-name').value = savedOp;
+  }
+
+  // Compute consent template hash (SHA-256 of the rendered consent block).
+  // This is what we'll send as consent_version_hash. Counsel later
+  // attests against the canonical signed template hash; operator
+  // discipline keeps these aligned.
+  const consentText = document.querySelector('.consent-text').innerText;
+  state.consentVersionHash = await sha256Hex(consentText);
+  document.getElementById('version-hash-display').textContent = state.consentVersionHash.substring(0,16) + '…';
+})();
+
+// ── Step 1: token + operator ───────────────────────────────────────
+document.getElementById('token-submit').addEventListener('click', () => {
+  clearErr('token-error');
+  const v = document.getElementById('token-input').value.trim();
+  // If user kept the masked placeholder, use the saved token.
+  const tok = v === '••••••••' ? state.token : v;
+  const op = document.getElementById('op-name').value.trim();
+  if (!tok || tok.length < 32) {
+    err('token-error', 'Token must be ≥32 characters.');
+    return;
+  }
+  if (!op || op.length < 2) {
+    err('token-error', 'Operator name is required (recorded as operator_of_record in audit row).');
+    return;
+  }
+  state.token = tok;
+  state.operator = op;
+  sessionStorage.setItem('lh_legal_token', tok);
+  sessionStorage.setItem('lh_operator_name', op);
+  show('screen-consent');
+});
+
+// ── Step 2: consent ────────────────────────────────────────────────
+function refreshGrantBtn() {
+  const ok = document.getElementById('cand-agree').checked
+          && document.getElementById('cand-name').value.trim().length >= 2;
+  document.getElementById('grant-btn').disabled = !ok;
+}
+document.getElementById('cand-agree').addEventListener('change', refreshGrantBtn);
+document.getElementById('cand-name').addEventListener('input', refreshGrantBtn);
+
+document.getElementById('grant-btn').addEventListener('click', async () => {
+  clearErr('consent-error');
+  document.getElementById('grant-btn').disabled = true;
+  try {
+    const candName = document.getElementById('cand-name').value.trim();
+    const ts = new Date().toISOString();
+    document.getElementById('timestamp-display').textContent = ts;
+    const evidence = JSON.stringify({
+      method: 'click_acceptance',
+      candidate_printed_name: candName,
+      acceptance_ts: ts,
+      user_agent: navigator.userAgent,
+      page_origin: location.origin,
+    });
+    const evidenceHash = await sha256Hex(evidence);
+    const body = {
+      consent_version_hash: state.consentVersionHash,
+      consent_collection_method: 'click_acceptance',
+      consent_collection_evidence_path: 'inline:sha256=' + evidenceHash,
+      operator_of_record: state.operator,
+    };
+    const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(CANDIDATE_ID)}/consent`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        'X-Lakehouse-Legal-Token': state.token,
+        'X-Lakehouse-Trace-Id': 'intake-' + Date.now(),
+      },
+      body: JSON.stringify(body),
+    });
+    const respBody = await r.json();
+    if (!r.ok) {
+      err('consent-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
+      document.getElementById('grant-btn').disabled = false;
+      return;
+    }
+    state.consentResp = respBody;
+    show('screen-photo');
+  } catch (e) {
+    err('consent-error', `Network error: ${e.message}`);
+    document.getElementById('grant-btn').disabled = false;
+  }
+});
+
+// ── Step 3: photo ──────────────────────────────────────────────────
+const previewEl = document.getElementById('preview');
+const noPreviewEl = document.getElementById('no-preview');
+const fileInput = document.getElementById('photo-file');
+
+fileInput.addEventListener('change', () => {
+  const f = fileInput.files[0];
+  if (!f) return;
+  state.photoBlob = f;
+  previewEl.src = URL.createObjectURL(f);
+  previewEl.style.display = 'block';
+  noPreviewEl.style.display = 'none';
+  document.getElementById('upload-btn').disabled = false;
+});
+
+let camStream = null;
+document.getElementById('cam-start').addEventListener('click', async () => {
+  clearErr('photo-error');
+  try {
+    camStream = await navigator.mediaDevices.getUserMedia({ video: { width: 640, height: 480, facingMode: 'user' }, audio: false });
+    document.getElementById('cam').srcObject = camStream;
+    document.getElementById('cam').style.display = 'block';
+    document.getElementById('cam-start').style.display = 'none';
+    document.getElementById('cam-capture').style.display = 'inline-block';
+    document.getElementById('cam-stop').style.display = 'inline-block';
+  } catch (e) {
+    err('photo-error', `Camera unavailable: ${e.message}`);
+  }
+});
+document.getElementById('cam-capture').addEventListener('click', () => {
+  const v = document.getElementById('cam');
+  const c = document.getElementById('cam-canvas');
+  c.width = v.videoWidth;
+  c.height = v.videoHeight;
+  c.getContext('2d').drawImage(v, 0, 0);
+  c.toBlob(b => {
+    state.photoBlob = b;
+    previewEl.src = URL.createObjectURL(b);
+    previewEl.style.display = 'block';
+    noPreviewEl.style.display = 'none';
+    document.getElementById('upload-btn').disabled = false;
+  }, 'image/jpeg', 0.92);
+});
+document.getElementById('cam-stop').addEventListener('click', () => {
+  if (camStream) { camStream.getTracks().forEach(t => t.stop()); camStream = null; }
+  document.getElementById('cam').style.display = 'none';
+  document.getElementById('cam-start').style.display = 'inline-block';
+  document.getElementById('cam-capture').style.display = 'none';
+  document.getElementById('cam-stop').style.display = 'none';
+});
+
+document.getElementById('upload-btn').addEventListener('click', async () => {
+  clearErr('photo-error');
+  document.getElementById('upload-btn').disabled = true;
+  try {
+    const ct = state.photoBlob.type || 'image/jpeg';
+    const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(CANDIDATE_ID)}/photo`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': ct,
+        'X-Lakehouse-Legal-Token': state.token,
+        'X-Lakehouse-Consent-Version-Hash': state.consentVersionHash,
+        'X-Lakehouse-Trace-Id': 'intake-' + Date.now(),
+      },
+      body: state.photoBlob,
+    });
+    const respBody = await r.json();
+    if (!r.ok) {
+      err('photo-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
+      document.getElementById('upload-btn').disabled = false;
+      return;
+    }
+    state.photoResp = respBody;
+    if (camStream) camStream.getTracks().forEach(t => t.stop());
+    showDone();
+  } catch (e) {
+    err('photo-error', `Network error: ${e.message}`);
+    document.getElementById('upload-btn').disabled = false;
+  }
+});
+
+document.getElementById('skip-btn').addEventListener('click', () => {
+  if (camStream) camStream.getTracks().forEach(t => t.stop());
+  showDone();
+});
+
+// ── Step 4: done ───────────────────────────────────────────────────
+function showDone() {
+  document.getElementById('done-given-at').textContent = state.consentResp.given_at;
+  document.getElementById('done-retention').textContent = state.consentResp.retention_until;
+  document.getElementById('done-consent-hmac').textContent = state.consentResp.audit_row_hmac;
+  if (state.photoResp) {
+    document.getElementById('photo-card').style.display = 'block';
+    document.getElementById('done-data-path').textContent = state.photoResp.data_path;
+    document.getElementById('done-template-hash').textContent = state.photoResp.template_hash;
+    document.getElementById('done-photo-hmac').textContent = state.photoResp.audit_row_hmac;
+  }
+  document.getElementById('verify-cmd').textContent =
+    `./scripts/staffing/verify_biometric_erasure.sh ${CANDIDATE_ID}`;
+  show('screen-done');
+}
+</script>
+</body>
+</html>

mcp-server/biometric_withdraw.html

@@ -0,0 +1,253 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Lakehouse — Biometric Consent Withdrawal</title>
+<style>
+*{margin:0;padding:0;box-sizing:border-box}
+body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
+a{color:#58a6ff;text-decoration:none}
+a:hover{color:#79c0ff}
+
+.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
+.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
+.bar .step{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:1px}
+
+.wrap{max-width:680px;margin:0 auto;padding:28px 20px 60px}
+.screen{display:none}
+.screen.active{display:block}
+
+h2{font-size:20px;color:#e6edf3;font-weight:600;margin-bottom:8px;letter-spacing:-0.3px}
+h3{font-size:14px;color:#e6edf3;font-weight:600;margin:16px 0 8px}
+.lede{color:#7d8590;font-size:13px;margin-bottom:24px}
+
+.card{background:#0d1117;border:1px solid #171d27;padding:20px;margin-bottom:16px}
+.card.warn{border-left:3px solid #d29922}
+.card.bad{border-left:3px solid #f85149}
+.card.good{border-left:3px solid #3fb950}
+
+label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
+input[type=text],input[type=password],textarea{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s;resize:vertical}
+input[type=text]:focus,input[type=password]:focus,textarea:focus{outline:none;border-color:#58a6ff}
+textarea{font-family:inherit;line-height:1.5;min-height:80px}
+
+button{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s;margin-right:8px}
+button:hover:not(:disabled){background:#388bfd}
+button:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
+button.secondary{background:#21262d;color:#c9d1d9}
+button.secondary:hover:not(:disabled){background:#30363d}
+button.danger{background:#da3633}
+button.danger:hover:not(:disabled){background:#f85149}
+
+.notice{background:#0a0c10;border:1px solid #1f242c;padding:14px 18px;margin-bottom:16px;font-size:13px;color:#b0b8c4;line-height:1.7}
+.notice strong{color:#e6edf3}
+
+.kv{display:grid;grid-template-columns:140px 1fr;gap:8px;margin:6px 0;font-size:13px}
+.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
+.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
+
+.hash{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:11px;color:#7ee787;word-break:break-all;background:#0a0c10;padding:8px 12px;border:1px solid #1f242c;margin:6px 0}
+
+.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
+.error.show{display:block}
+.error code{color:#fff}
+
+.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
+.foot a{color:#58a6ff}
+</style>
+</head>
+<body>
+
+<div class="bar">
+  <h1>⚡ Biometric Consent Withdrawal</h1>
+  <span class="step" id="step-label">step 1 of 3</span>
+</div>
+
+<div class="wrap">
+
+  <!-- Step 1: operator auth -->
+  <div id="screen-token" class="screen active">
+    <h2>Operator authentication</h2>
+    <p class="lede">Withdrawal is operator-recorded on behalf of the candidate. Paste the legal-tier audit token + your name.</p>
+    <div class="card">
+      <label for="token-input">Legal audit token</label>
+      <input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
+      <label for="op-name" style="margin-top:14px">Your name (operator of record)</label>
+      <input type="text" id="op-name" placeholder="First Last" autocomplete="off">
+      <div style="margin-top:14px">
+        <button id="token-submit">Continue →</button>
+      </div>
+    </div>
+    <div class="error" id="token-error"></div>
+  </div>
+
+  <!-- Step 2: withdrawal form -->
+  <div id="screen-form" class="screen">
+    <h2>Record withdrawal</h2>
+    <p class="lede">The candidate has requested withdrawal of biometric consent. This action sets a <strong>30-day SLA clock</strong> for destruction (per consent template v1 §2). The retention sweep + erase runbook handle actual destruction; this endpoint records intent + starts the clock.</p>
+
+    <div class="notice warn">
+      <strong>What withdrawal does:</strong> sets <code>consent.biometric.status = Withdrawn</code>, accelerates <code>retention_until</code> from the 18-month default to <strong>30 days from now</strong>. Future photo uploads will be refused (403). General-PII consent is NOT touched — the candidate can keep their non-biometric data on the platform.
+    </div>
+
+    <div class="card">
+      <label for="cand-id">Candidate ID</label>
+      <input type="text" id="cand-id" placeholder="WORKER-XXX" autocomplete="off">
+
+      <label for="reason" style="margin-top:14px">Reason for withdrawal</label>
+      <textarea id="reason" placeholder="Free-text. Captured to the audit row for legal traceability. e.g., 'candidate emailed 2026-05-05 asking to remove their photo, no specific reason given'"></textarea>
+
+      <label for="evidence" style="margin-top:14px">Evidence path (optional)</label>
+      <input type="text" id="evidence" placeholder="/path/to/email_thread.eml or scanned signed paper" autocomplete="off">
+
+      <div style="margin-top:16px">
+        <button id="submit-btn" class="danger" disabled>Record withdrawal</button>
+        <button id="back-btn" class="secondary">Back</button>
+      </div>
+    </div>
+    <div class="error" id="form-error"></div>
+  </div>
+
+  <!-- Step 3: confirmation -->
+  <div id="screen-done" class="screen">
+    <div class="card good">
+      <h2>✓ Withdrawal recorded</h2>
+      <p class="lede" style="margin:8px 0 0">Audit chain row appended; retention sweep will pick it up at the SLA.</p>
+    </div>
+
+    <div class="card">
+      <h3>State change</h3>
+      <div class="kv">
+        <span class="k">Candidate</span><span class="v" id="done-cid">…</span>
+        <span class="k">Status</span><span class="v" id="done-status">Withdrawn</span>
+        <span class="k">Withdrawn at</span><span class="v" id="done-withdrawn-at">…</span>
+        <span class="k">Retention until</span><span class="v" id="done-retention">…</span>
+      </div>
+      <h3>Audit row hmac</h3>
+      <div class="hash" id="done-hmac">…</div>
+    </div>
+
+    <div class="card warn">
+      <h3>What happens next</h3>
+      <p>The retention sweep flags this subject as overdue once <code>retention_until</code> passes. An operator with legal-tier credentials runs the destruction runbook (<code>POST /biometric/subject/&lt;id&gt;/erase</code>) within the 30-day SLA.</p>
+      <p style="margin-top:8px">To verify the withdrawal landed cleanly:</p>
+      <div class="hash" id="verify-cmd">curl -H "X-Lakehouse-Legal-Token: $TOKEN" http://localhost:3100/audit/subject/&lt;id&gt;</div>
+    </div>
+
+    <div style="margin-top:16px">
+      <button onclick="location.reload()" class="secondary">Record another withdrawal</button>
+    </div>
+  </div>
+
+</div>
+
+<div class="foot">
+  <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/policies/consent/biometric_consent_template_v1.md">Consent template v1</a>
+  · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
+  · <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">Phase 1.6 BIPA Gates</a>
+</div>
+
+<script>
+const GATEWAY = (function(){
+  const p = new URLSearchParams(location.search);
+  return p.get('gw') || 'http://localhost:3100';
+})();
+
+const state = { token: null, operator: null, response: null };
+
+function show(id) {
+  document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
+  document.getElementById(id).classList.add('active');
+  const map = {'screen-token':'step 1 of 3', 'screen-form':'step 2 of 3', 'screen-done':'step 3 of 3'};
+  document.getElementById('step-label').textContent = map[id] || '';
+}
+function err(id, msg){ const e=document.getElementById(id); e.textContent=msg; e.classList.add('show'); }
+function clearErr(id){ document.getElementById(id).classList.remove('show'); }
+
+// Pre-populate candidate_id from URL if present.
+(function init(){
+  const p = new URLSearchParams(location.search);
+  const cid = p.get('candidate_id') || p.get('cid');
+  if (cid) document.getElementById('cand-id').value = cid;
+  // Restore token from sessionStorage if previously set in this tab.
+  const saved = sessionStorage.getItem('lh_legal_token');
+  const savedOp = sessionStorage.getItem('lh_operator_name');
+  if (saved) {
+    state.token = saved;
+    document.getElementById('token-input').value = '••••••••';
+  }
+  if (savedOp) {
+    state.operator = savedOp;
+    document.getElementById('op-name').value = savedOp;
+  }
+})();
+
+// Step 1.
+document.getElementById('token-submit').addEventListener('click', () => {
+  clearErr('token-error');
+  const v = document.getElementById('token-input').value.trim();
+  const tok = v === '••••••••' ? state.token : v;
+  const op = document.getElementById('op-name').value.trim();
+  if (!tok || tok.length < 32) { err('token-error', 'Token must be ≥32 characters.'); return; }
+  if (!op || op.length < 2) { err('token-error', 'Operator name is required.'); return; }
+  state.token = tok;
+  state.operator = op;
+  sessionStorage.setItem('lh_legal_token', tok);
+  sessionStorage.setItem('lh_operator_name', op);
+  show('screen-form');
+  refreshSubmit();
+});
+
+// Step 2.
+function refreshSubmit() {
+  const cid = document.getElementById('cand-id').value.trim();
+  const reason = document.getElementById('reason').value.trim();
+  document.getElementById('submit-btn').disabled = !(cid.length > 0 && reason.length >= 10);
+}
+['cand-id','reason'].forEach(id => document.getElementById(id).addEventListener('input', refreshSubmit));
+
+document.getElementById('back-btn').addEventListener('click', () => show('screen-token'));
+
+document.getElementById('submit-btn').addEventListener('click', async () => {
+  clearErr('form-error');
+  document.getElementById('submit-btn').disabled = true;
+  try {
+    const cid = document.getElementById('cand-id').value.trim();
+    const body = {
+      reason: document.getElementById('reason').value.trim(),
+      operator_of_record: state.operator,
+      evidence_path: document.getElementById('evidence').value.trim(),
+    };
+    const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(cid)}/withdraw`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+        'X-Lakehouse-Legal-Token': state.token,
+        'X-Lakehouse-Trace-Id': 'withdraw-' + Date.now(),
+      },
+      body: JSON.stringify(body),
+    });
+    const respBody = await r.json();
+    if (!r.ok) {
+      err('form-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
+      refreshSubmit();
+      return;
+    }
+    state.response = respBody;
+    document.getElementById('done-cid').textContent = respBody.candidate_id;
+    document.getElementById('done-status').textContent = respBody.status_after;
+    document.getElementById('done-withdrawn-at').textContent = respBody.withdrawn_at;
+    document.getElementById('done-retention').textContent = respBody.retention_until;
+    document.getElementById('done-hmac').textContent = respBody.audit_row_hmac;
+    document.getElementById('verify-cmd').textContent =
+      `curl -H "X-Lakehouse-Legal-Token: $TOKEN" ${GATEWAY}/audit/subject/${respBody.candidate_id}`;
+    show('screen-done');
+  } catch (e) {
+    err('form-error', `Network error: ${e.message}`);
+    refreshSubmit();
+  }
+});
+</script>
+</body>
+</html>

mcp-server/console.html

@@ -51,9 +51,28 @@ details .body{padding-top:10px;font-size:12px;color:#8b949e}
 .accent-b{border-left:3px solid #1f6feb}
 .accent-a{border-left:3px solid #bc8cff}
 .accent-w{border-left:3px solid #d29922}
+.accent-g{border-left:3px solid #3fb950}
+.accent-r{border-left:3px solid #f85149}
 
-.worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px}
-.worker .av{width:28px;height:28px;border-radius:6px;background:#1a2744;display:flex;align-items:center;justify-content:center;font-weight:600;color:#e6edf3;font-size:10px;flex-shrink:0}
+.worker{display:flex;align-items:center;gap:10px;padding:8px 10px;background:#161b22;border-radius:6px;margin-bottom:4px;font-size:12px;border-left:3px solid #30363d}
+.worker .av{width:32px;height:32px;border-radius:50%;background:#0d1117;border:1px solid #21262d;display:flex;align-items:center;justify-content:center;font-weight:600;color:#c9d1d9;font-size:11px;flex-shrink:0;letter-spacing:0.5px;overflow:hidden;position:relative}
+.worker .av img{position:absolute;inset:0;width:100%;height:100%;object-fit:cover;display:block;
+  /* Softening — mirror of search.html. Pulls saturation + contrast off
+     the SDXL Turbo over-render so faces feel less "AI-generated".
+     If you tweak one, tweak the other. */
+  filter: saturate(0.86) contrast(0.93) brightness(1.02) blur(0.3px);
+}
+.worker[data-role-band="warehouse"]{border-left-color:#58a6ff}
+.worker[data-role-band="production"]{border-left-color:#d29922}
+.worker[data-role-band="trades"]{border-left-color:#bc8cff}
+.worker[data-role-band="driver"]{border-left-color:#3fb950}
+.worker[data-role-band="lead"]{border-left-color:#f0883e}
+.role-pill{display:inline-block;font-size:9px;padding:1px 7px;border-radius:3px;background:#0d1117;color:#8b949e;margin-right:6px;font-weight:600;letter-spacing:0.4px;text-transform:uppercase;border-left:2px solid #30363d;vertical-align:1px}
+.role-pill[data-rb="warehouse"]{border-left-color:#58a6ff;color:#79c0ff}
+.role-pill[data-rb="production"]{border-left-color:#d29922;color:#e3b341}
+.role-pill[data-rb="trades"]{border-left-color:#bc8cff;color:#d2a8ff}
+.role-pill[data-rb="driver"]{border-left-color:#3fb950;color:#56d364}
+.role-pill[data-rb="lead"]{border-left-color:#f0883e;color:#ffa657}
 .worker .info{flex:1;min-width:0}
 .worker .nm{color:#e6edf3;font-weight:500}
 .worker .why{color:#545d68;font-size:11px;margin-top:1px}
@@ -95,6 +114,7 @@ details .body{padding-top:10px;font-size:12px;color:#8b949e}
   <nav>
     <a href=".">Dashboard</a>
     <a href="console" class="active">Walkthrough</a>
+    <a href="profiler">Profiler</a>
     <a href="proof">Architecture</a>
     <a href="spec">Spec</a>
     <a href="onboard">Onboard</a>
@@ -147,11 +167,40 @@ details .body{padding-top:10px;font-size:12px;color:#8b949e}
 
 <div class="chapter">
   <div class="num">Chapter 6</div>
-  <h2>Try it yourself</h2>
-  <div class="lede">Type any staffing question. The system picks the right search path (smart-parse, semantic discovery, analytics), shows what it understood, and returns ranked results with memory signal.</div>
+  <h2>Three coordinators, three views of the same corpus</h2>
+  <div class="lede">Maria runs Chicago, Devon runs Indianapolis, Aisha runs Milwaukee. Same database, same playbooks — but the search results, the recurring-skill patterns, and the playbook context all reshape to whoever is acting. This is the per-staffer hot-swap index: the relevance gradient is unique to each person, and gets sharper the more they use it.</div>
+  <div id="ch6-staffers"><div class="loading">Loading staffer roster…</div></div>
+</div>
+
+<div class="chapter">
+  <div class="num">Chapter 7</div>
+  <h2>The hidden signal — public issuers in your contractor graph</h2>
+  <div class="lede">Every contractor in this corpus is also a forward indicator on the public equities they touch. Permit filings precede construction starts by ~45 days, staffing windows by ~30, revenue recognition by months. The associated-ticker network surfaces this signal <em>before</em> any 10-Q. Below: the top issuers attributable to the contractor activity in this view, with live prices.</div>
+  <div id="ch7-signal"><div class="loading">Computing the Building Activity Index…</div></div>
+</div>
+
+<div class="chapter">
+  <div class="num">Chapter 8</div>
+  <h2>When something breaks — triage in one shot</h2>
+  <div class="lede">A coordinator gets a text: "Marcus running late." Watch what the system does in 250 milliseconds: pulls Marcus's record, scores his attendance pattern, finds five same-role same-geo backfills sorted by responsiveness, and pre-writes the SMS to send to the client. This is the moment the AI becomes worth its weight.</div>
+  <div id="ch8-triage"><div class="loading">Running the triage scenario…</div></div>
+</div>
+
+<div class="chapter">
+  <div class="num">Chapter 9</div>
+  <h2>Try it yourself — every input below hits a different route</h2>
+  <div class="lede">Type any staffing question. The router picks the right path: smart-parse (zip code, headcount, role, state), semantic discovery, name lookup, late-worker triage, "what came in last night" temporal queries. Whatever you type, the system tells you what it understood and how it routed.</div>
   <div class="try-box">
-    <input type="text" id="try-q" placeholder="e.g. reliable forklift operators in Chicago with OSHA certs" onkeydown="if(event.key==='Enter')runTry()">
+    <input type="text" id="try-q" placeholder="e.g. 8 production workers near 60607 by next Friday" onkeydown="if(event.key==='Enter')runTry()">
     <button id="try-btn" onclick="runTry()">Ask</button>
+    <div style="margin-top:10px;font-size:11px;color:#545d68;line-height:1.7">
+      Try one of these to see different routes fire:<br>
+      <a href="#" onclick="document.getElementById('try-q').value='8 production workers near 60607';runTry();return false">8 production workers near 60607</a> ·
+      <a href="#" onclick="document.getElementById('try-q').value='Marcus running late site 4422';runTry();return false">Marcus running late site 4422</a> ·
+      <a href="#" onclick="document.getElementById('try-q').value='Marcus';runTry();return false">Marcus</a> ·
+      <a href="#" onclick="document.getElementById('try-q').value='what came in last night';runTry();return false">what came in last night</a> ·
+      <a href="#" onclick="document.getElementById('try-q').value='reliable forklift operators with OSHA certs';runTry();return false">reliable forklift operators with OSHA certs</a>
+    </div>
     <div id="try-out" style="margin-top:16px"></div>
   </div>
 </div>
@@ -167,6 +216,65 @@ var A=location.origin+P;
 // DOM helpers — all dynamic content goes through these. No innerHTML
 // anywhere in the script; every API-derived string passes through
 // textContent so no injection path regardless of upstream data.
+// Role classification — mirrors search.html, no emojis. Maps role
+// strings to a band+label used by the worker-card border + role pill.
+var ROLE_BANDS = [
+  { match: /forklift|warehouse|associate|material\s*handler|loader|loading|packag|shipping|logistics|inventory|sanitation|janit/i, band: 'warehouse', label: 'Warehouse' },
+  { match: /production|assembl/i, band: 'production', label: 'Production' },
+  { match: /welder|weld|electric|maint(enance)?\s*tech|cnc|machine\s*op|hvac|plumb|carpenter|mason/i, band: 'trades', label: 'Skilled Trade' },
+  { match: /driver|truck|haul|cdl/i, band: 'driver', label: 'Driver' },
+  { match: /line\s*lead|supervisor|foreman|coordinator/i, band: 'lead', label: 'Lead' },
+  { match: /quality/i, band: 'production', label: 'Quality' },
+];
+function roleBand(role){
+  if(!role) return { band: 'warehouse', label: '' };
+  for (var i = 0; i < ROLE_BANDS.length; i++) {
+    if (ROLE_BANDS[i].match.test(role)) return ROLE_BANDS[i];
+  }
+  return { band: 'warehouse', label: role.split(' ')[0].toUpperCase().slice(0, 12) };
+}
+
+// Build a sober worker card: monogram avatar + colored role band on
+// the left edge + uppercase role pill in the detail line. Used by
+// every chapter that renders worker rows. `name` and `role` drive the
+// classification; `detail` is the full text after the pill.
+// Name → ethnicity / gender inference — REMOVED 2026-05-03 per Phase 1.6
+// BIPA gates (docs/PHASE_1_6_BIPA_GATES.md, Gate 4). Mirror of the same
+// removal in mcp-server/search.html. Functions had no live consumers
+// (headshot rendering disabled 2026-04-28). Do NOT re-introduce.
+
+function workerRow(name, role, detail, opts){
+  opts = opts || {};
+  var band = roleBand(role||'');
+  var w = el('div','worker');
+  if(band.band) w.dataset.roleBand = band.band;
+  var initials = (name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
+  var av = el('div','av',initials);
+  // Headshot insertion removed 2026-04-28. The .av element stays as
+  // a monogram-initials avatar.
+  w.appendChild(av);
+  var info = el('div','info');
+  var nm = el('div','nm', name||'?');
+  if(opts.endorsed){
+    nm.appendChild(el('span','boost-chip',opts.endorsed));
+  }
+  info.appendChild(nm);
+  var why = el('div','why');
+  if(band.label){
+    var pill = document.createElement('span'); pill.className='role-pill';
+    pill.dataset.rb = band.band;
+    pill.textContent = band.label;
+    why.appendChild(pill);
+  }
+  why.appendChild(document.createTextNode(detail||''));
+  info.appendChild(why);
+  w.appendChild(info);
+  if(opts.score){
+    w.appendChild(el('div','score', opts.score));
+  }
+  return w;
+}
+
 function el(tag, cls, text){
   var e=document.createElement(tag);
   if(cls) e.className=cls;
@@ -191,6 +299,9 @@ window.addEventListener('load',function(){
   loadChapter3();
   loadChapter4();
   loadChapter5();
+  loadChapter6();
+  loadChapter7();
+  loadChapter8();
 });
 
 // ─── Chapter 1 ────────────────────────────────────────────
@@ -306,6 +417,30 @@ function loadChapter4(){
     addr.style.cssText='color:#8b949e;font-size:12px;margin-top:2px';
     card.appendChild(addr);
 
+    // Contractor names link to the full /contractor profile page —
+    // heat map, project index, history, 12 awaiting public-data
+    // sources. The staffer click-through J asked for.
+    if(p.contact_1_name || p.contact_2_name){
+      var contractors=document.createElement('div');
+      contractors.style.cssText='color:#8b949e;font-size:12px;margin-top:4px';
+      contractors.appendChild(document.createTextNode('Contractors: '));
+      var seen=[];
+      [p.contact_1_name, p.contact_2_name].forEach(function(n,i){
+        if(!n || seen.indexOf(n)>=0) return;
+        seen.push(n);
+        if(seen.length>1) contractors.appendChild(document.createTextNode(' · '));
+        var a=document.createElement('a');
+        a.href=P+'/contractor?name='+encodeURIComponent(n);
+        a.target='_blank';
+        a.rel='noopener';
+        a.style.cssText='color:#58a6ff;text-decoration:none;border-bottom:1px dotted #58a6ff44';
+        a.title='Open full contractor profile';
+        a.textContent=n;
+        contractors.appendChild(a);
+      });
+      card.appendChild(contractors);
+    }
+
     card.appendChild(el('div','step-label','STEP 1 · Derive staffing need'));
     var s1=el('div','step-body');
     s1.appendChild(document.createTextNode('Industry heuristic: ~1 worker per $150K of permit cost, capped 2-8. Resulting contract: '));
@@ -321,21 +456,13 @@ function loadChapter4(){
 
     var list=document.createElement('div');list.style.marginTop='6px';
     (prop.candidates||[]).slice(0,5).forEach(function(cand,i){
-      var w=el('div','worker');
-      var initials=(cand.name||'?').split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
-      w.appendChild(el('div','av',initials));
-      var info=el('div','info');
-      var nm=el('div','nm',cand.name||cand.doc_id||'?');
-      if((cand.playbook_boost||0)>0){
-        var ncit=(cand.playbook_citations||[]).length;
-        nm.appendChild(el('span','boost-chip','Endorsed · '+ncit+' past fill'+(ncit!==1?'s':'')));
-      }
-      info.appendChild(nm);
-      var why=cand.doc_id+' · '+(cand.playbook_boost>0?'boosted +'+cand.playbook_boost.toFixed(3)+' by memory · ':'')+'semantic score '+(cand.score||0).toFixed(3);
-      info.appendChild(el('div','why',why));
-      w.appendChild(info);
-      w.appendChild(el('div','score','#'+(i+1)));
-      list.appendChild(w);
+      var detail = cand.doc_id+' · '+(cand.playbook_boost>0?'boosted +'+cand.playbook_boost.toFixed(3)+' by memory · ':'')+'semantic score '+(cand.score||0).toFixed(3);
+      var endorsed = (cand.playbook_boost||0) > 0
+        ? 'Endorsed · '+((cand.playbook_citations||[]).length)+' past fill'+((cand.playbook_citations||[]).length!==1?'s':'')
+        : null;
+      list.appendChild(workerRow(cand.name||cand.doc_id||'?', prop.role||'', detail, {
+        endorsed: endorsed, score: '#'+(i+1)
+      }));
     });
     card.appendChild(list);
 
@@ -407,7 +534,182 @@ function loadChapter5(){
   });
 }
 
-// ─── Chapter 6 ────────────────────────────────────────────
+// ─── Chapter 6 — per-staffer hot-swap ─────────────────────
+function loadChapter6(){
+  apiGet('/staffers').then(function(r){
+    var host=document.getElementById('ch6-staffers');host.textContent='';
+    var staffers=(r&&r.staffers)||[];
+    if(!staffers.length){
+      host.appendChild(el('div','err','No staffer roster — /staffers returned empty.'));
+      return;
+    }
+    var grid=document.createElement('div'); grid.className='grid'; grid.style.gridTemplateColumns='repeat(auto-fit,minmax(280px,1fr))';
+    staffers.forEach(function(s){
+      var card=el('div','card accent-b');
+      var name=el('div',null,s.name);
+      name.style.cssText='font-size:18px;font-weight:700;color:#e6edf3;letter-spacing:-0.3px';
+      card.appendChild(name);
+      var role=el('div',null,s.display||'');
+      role.style.cssText='font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:2px';
+      card.appendChild(role);
+      var ter=el('div',null,'Territory: '+s.territory.state+' · '+s.territory.cities.slice(0,3).join(', ')+'…');
+      ter.style.cssText='color:#8b949e;font-size:12px;margin-top:8px';
+      card.appendChild(ter);
+      var greet=el('div',null,s.greeting||'');
+      greet.style.cssText='color:#c9d1d9;font-size:11px;margin-top:6px;line-height:1.5;border-top:1px dashed #1f2631;padding-top:6px';
+      card.appendChild(greet);
+      grid.appendChild(card);
+    });
+    host.appendChild(grid);
+    var narr=el('div','narr');
+    narr.appendChild(el('strong',null,'What this means for a staffer. '));
+    narr.appendChild(document.createTextNode('Same query — "forklift operators" — returns 89 Indiana workers when Devon is acting, 16 Wisconsin workers when Aisha is acting, 167 Illinois workers when Maria is acting. The MEMORY panel relabels itself with whoever\'s viewing. The corpus stays intact; the relevance gradient is per coordinator. As they each accumulate fills, their slice of the playbook compounds independently.'));
+    host.appendChild(narr);
+  }).catch(function(e){
+    var h=document.getElementById('ch6-staffers');h.textContent='';h.appendChild(el('div','err','Staffer roster unavailable: '+(e.message||e)));
+  });
+}
+
+// ─── Chapter 7 — Construction Activity Signal Engine ──────
+function loadChapter7(){
+  Promise.all([
+    api('/intelligence/profiler_index',{limit:200}),
+  ]).then(function(rs){
+    var prof=rs[0]||{};
+    var rows=prof.contractors||[];
+    var host=document.getElementById('ch7-signal');host.textContent='';
+    // Aggregate basket
+    var byTicker={};
+    rows.forEach(function(r){
+      var ts=(r.tickers&&r.tickers.direct?r.tickers.direct:[]).concat(r.tickers&&r.tickers.associated?r.tickers.associated:[]);
+      ts.forEach(function(t){
+        if(!t||!t.ticker) return;
+        if(!byTicker[t.ticker]) byTicker[t.ticker]={ticker:t.ticker,count:0,kinds:new Set()};
+        byTicker[t.ticker].count++;
+        byTicker[t.ticker].kinds.add(t.via);
+      });
+    });
+    var basket=Object.values(byTicker).sort(function(a,b){return b.count-a.count});
+    var attribCost=0;
+    rows.forEach(function(r){
+      var ts=(r.tickers&&r.tickers.direct?r.tickers.direct:[]).concat(r.tickers&&r.tickers.associated?r.tickers.associated:[]);
+      if(ts.length>0) attribCost += (r.total_cost||0);
+    });
+    var totalAttrib = basket.reduce(function(s,b){return s+b.count},0);
+    if(!basket.length){
+      host.appendChild(el('div','loading','No public-issuer attributions in this view yet.'));
+      return;
+    }
+    // Top-line metric strip
+    var grid=document.createElement('div');grid.className='grid';
+    var c1=el('div','card accent-g');
+    var b1=el('div',null,basket.length); b1.style.cssText='font-size:30px;font-weight:800;color:#3fb950;line-height:1';
+    c1.appendChild(b1);
+    var l1=el('div',null,'Public issuers in scope'); l1.style.cssText='font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600';
+    c1.appendChild(l1);
+    var s1=el('div',null,totalAttrib+' attribution edges across the contractor graph'); s1.style.cssText='font-size:12px;color:#8b949e;margin-top:4px';
+    c1.appendChild(s1);
+    grid.appendChild(c1);
+    var c2=el('div','card accent-b');
+    var bav = attribCost>=1e9?'$'+(attribCost/1e9).toFixed(2)+'B':attribCost>=1e6?'$'+(attribCost/1e6).toFixed(0)+'M':'$'+Math.round(attribCost/1e3)+'K';
+    var b2=el('div',null,bav); b2.style.cssText='font-size:30px;font-weight:800;color:#58a6ff;line-height:1';
+    c2.appendChild(b2);
+    var l2=el('div',null,'Attributed build value'); l2.style.cssText='font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600';
+    c2.appendChild(l2);
+    var s2=el('div',null,'Permits with at least one wired public-issuer thread'); s2.style.cssText='font-size:12px;color:#8b949e;margin-top:4px';
+    c2.appendChild(s2);
+    grid.appendChild(c2);
+    var c3=el('div','card accent-l');
+    var b3=el('div',null,rows.length); b3.style.cssText='font-size:30px;font-weight:800;color:#bc8cff;line-height:1';
+    c3.appendChild(b3);
+    var l3=el('div',null,'Contractors indexed'); l3.style.cssText='font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-top:8px;font-weight:600';
+    c3.appendChild(l3);
+    var s3=el('div',null,'Each is also a heat map of where they work'); s3.style.cssText='font-size:12px;color:#8b949e;margin-top:4px';
+    c3.appendChild(s3);
+    grid.appendChild(c3);
+    host.appendChild(grid);
+    // Top issuer table
+    var tHdr=document.createElement('div');tHdr.style.cssText='color:#545d68;font-size:11px;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin:14px 0 8px';
+    tHdr.textContent='Top public issuers attributable in this view';
+    host.appendChild(tHdr);
+    basket.slice(0,8).forEach(function(b){
+      var row=el('div','row');
+      var left=document.createElement('div');left.style.flex='1';left.style.minWidth='0';
+      var tk=el('div','title',b.ticker);
+      tk.style.cssText+='font-family:ui-monospace,monospace;color:#3fb950';
+      left.appendChild(tk);
+      var kinds=Array.from(b.kinds);
+      var meta=el('div','meta',b.count+' attribution'+(b.count===1?'':'s')+' · '+kinds.join('+'));
+      left.appendChild(meta);
+      row.appendChild(left);
+      var right=document.createElement('div');right.style.cssText='font-size:11px;color:#58a6ff';
+      var a=document.createElement('a');a.href=P+'/profiler';a.target='_blank';a.style.color='#58a6ff';a.style.textDecoration='none';
+      a.textContent='see in profiler →';
+      right.appendChild(a);
+      row.appendChild(right);
+      host.appendChild(row);
+    });
+    var narr=el('div','narr');
+    narr.appendChild(el('strong',null,'What this means for the business. '));
+    narr.appendChild(document.createTextNode('The data corpus is also a market-signal engine. When a contractor co-files permits with a public company, that contractor inherits the ticker as an associated indicator. Permit volume changes precede earnings calls by months. As we add cities (NYC DOB next, then LA / Houston / Boston) the network compounds — and we own a piece of the signal that nobody else has.'));
+    host.appendChild(narr);
+  }).catch(function(e){
+    var h=document.getElementById('ch7-signal');h.textContent='';h.appendChild(el('div','err','Signal engine unavailable: '+(e.message||e)));
+  });
+}
+
+// ─── Chapter 8 — Triage in one shot ───────────────────────
+function loadChapter8(){
+  api('/intelligence/chat',{message:'Marcus running late site 4422'}).then(function(d){
+    var host=document.getElementById('ch8-triage');host.textContent='';
+    if(d.type!=='triage'){
+      host.appendChild(el('div','err','Triage route did not fire. Got type=' + (d.type||'?')));
+      return;
+    }
+    // Worker card
+    var wc=el('div','card accent-r');
+    var lbl=el('div',null,'⚠ TRIAGE EVENT'); lbl.style.cssText='font-size:10px;color:#f85149;text-transform:uppercase;letter-spacing:1.2px;font-weight:700;margin-bottom:8px';
+    wc.appendChild(lbl);
+    var nm=el('div',null,d.worker.name); nm.style.cssText='font-size:18px;font-weight:700;color:#e6edf3';
+    wc.appendChild(nm);
+    var loc=el('div',null,(d.worker.role||'?')+' · '+(d.worker.city||'')+', '+(d.worker.state||''));
+    loc.style.cssText='font-size:12px;color:#8b949e;margin-top:2px';
+    wc.appendChild(loc);
+    var stats=document.createElement('div');stats.style.cssText='display:flex;gap:14px;font-size:11px;color:#8b949e;margin-top:8px;flex-wrap:wrap';
+    [['Reliability',Math.round((d.worker.rel||0)*100)+'%'],['Responsiveness',Math.round((d.worker.resp||0)*100)+'%'],['Availability',Math.round((d.worker.avail||0)*100)+'%']].forEach(function(p){
+      var s=document.createElement('span');
+      var l=document.createElement('span');l.textContent=p[0]+': ';
+      var b=document.createElement('b');b.style.color='#e6edf3';b.textContent=p[1];
+      s.appendChild(l);s.appendChild(b);stats.appendChild(s);
+    });
+    wc.appendChild(stats);
+    host.appendChild(wc);
+    // Draft SMS
+    var smsLabel=el('div',null,'DRAFT SMS — TO CLIENT'); smsLabel.style.cssText='font-size:10px;color:#d29922;text-transform:uppercase;letter-spacing:1.2px;font-weight:700;margin:14px 0 4px';
+    host.appendChild(smsLabel);
+    var smsBox=el('div',null,d.draft_sms||'');
+    smsBox.style.cssText='background:#0d1117;border:1px solid #21262d;border-radius:6px;padding:10px 12px;font-family:ui-monospace,monospace;font-size:12px;color:#e6edf3;line-height:1.5;white-space:pre-wrap';
+    host.appendChild(smsBox);
+    // Backfills
+    if((d.backfills||[]).length){
+      var bfHdr=document.createElement('div');bfHdr.style.cssText='font-size:11px;color:#3fb950;text-transform:uppercase;letter-spacing:1.2px;font-weight:600;margin:14px 0 8px';
+      bfHdr.textContent='✓ '+d.backfills.length+' local '+(d.worker.role||'workers')+' available — sorted by responsiveness';
+      host.appendChild(bfHdr);
+      d.backfills.slice(0,5).forEach(function(c){
+        var detail=(c.role||'?')+' · '+(c.city||'')+', '+(c.state||'')+' · rel '+Math.round((c.rel||0)*100)+'% · resp '+Math.round((c.resp||0)*100)+'%';
+        host.appendChild(workerRow(c.name||'?', c.role||'', detail));
+      });
+    }
+    var narr=el('div','narr');
+    narr.appendChild(el('strong',null,'What this means for a coordinator. '));
+    narr.appendChild(document.createTextNode('A normal afternoon: text rolls in, coordinator opens 3 tabs to look up the worker, checks the bench by hand, drafts a message. 20 minutes. Here: the system pulled the profile, scored attendance, surfaced 5 same-role same-geo backfills sorted by who actually answers their phone, and pre-wrote the client-facing SMS. The coordinator clicks send. ' + d.duration_ms + 'ms.'));
+    host.appendChild(narr);
+  }).catch(function(e){
+    var h=document.getElementById('ch8-triage');h.textContent='';h.appendChild(el('div','err','Triage demo unavailable: '+(e.message||e)));
+  });
+}
+
+// ─── Chapter 9 (was 6) — Try it yourself ──────────────────
 function runTry(){
   var q=document.getElementById('try-q').value.trim();if(!q)return;
   var btn=document.getElementById('try-btn'),out=document.getElementById('try-out');
@@ -437,23 +739,16 @@ function runTry(){
 
     var workers=d.sql_results||d.vector_results||d.results||[];
     workers.slice(0,5).forEach(function(w,i){
-      var row=el('div','worker');
       var nm=w.name||(w.text||'').split('—')[0].trim()||w.doc_id||'?';
-      var initials=nm.split(' ').map(function(s){return (s[0]||'').toUpperCase()}).join('').substring(0,2);
-      row.appendChild(el('div','av',initials));
-      var info=el('div','info');
-      var n=el('div','nm',nm);
-      if((w.playbook_boost||0)>0){
-        n.appendChild(el('span','boost-chip','Endorsed · '+((w.playbook_citations||[]).length||'?')+' past fill(s)'));
-      }
-      info.appendChild(n);
       var bits=[];
       if(w.role) bits.push(w.role);
       if(w.city&&w.state) bits.push(w.city+', '+w.state);
       if(w.rel!==undefined) bits.push('reliability '+Math.round(w.rel*100)+'%');
       if(w.avail!==undefined) bits.push('availability '+Math.round(w.avail*100)+'%');
-      info.appendChild(el('div','why',bits.join(' · ')||'AI semantic match'));
-      row.appendChild(info);
+      var endorsed = (w.playbook_boost||0) > 0
+        ? 'Endorsed · '+((w.playbook_citations||[]).length||'?')+' past fill(s)'
+        : null;
+      var row = workerRow(nm, w.role||'', bits.join(' · ')||'AI semantic match', { endorsed: endorsed });
       row.appendChild(el('div','score','#'+(i+1)));
       card.appendChild(row);
     });

mcp-server/contractor.html

@@ -0,0 +1,606 @@
+<!DOCTYPE html>
+<html><head>
+<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Contractor Profile · Staffing Co-Pilot</title>
+<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css">
+<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
+<style>
+*{margin:0;padding:0;box-sizing:border-box}
+html,body{overflow-x:hidden}
+body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.6}
+.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center}
+.bar h1{font-size:14px;font-weight:600;color:#e6edf3}
+.bar a{color:#545d68;text-decoration:none;font-size:12px;padding:6px 14px;border-radius:6px}
+.bar a:hover{color:#e6edf3;background:#161b22}
+.content{max-width:1100px;margin:0 auto;padding:24px 20px 40px}
+.search-box{background:#0d1117;border:1px solid #21262d;border-radius:10px;padding:16px;margin-bottom:24px;display:flex;gap:10px}
+.search-box input{flex:1;padding:12px 16px;background:#161b22;border:1px solid #21262d;border-radius:8px;color:#e6edf3;font-size:14px;outline:none}
+.search-box input:focus{border-color:#388bfd}
+.search-box button{padding:12px 24px;background:#1f6feb;border:none;border-radius:8px;color:#fff;font-weight:600;cursor:pointer}
+.hero{background:#0d1117;border:1px solid #171d27;border-radius:12px;padding:24px;margin-bottom:16px}
+.hero h2{color:#e6edf3;font-size:22px;font-weight:700;letter-spacing:-0.5px;margin-bottom:6px}
+.hero .ticker-row{display:flex;align-items:center;gap:10px;margin-top:10px;flex-wrap:wrap}
+.hero .ticker{font-family:ui-monospace,SFMono-Regular,monospace;background:#161b22;padding:4px 10px;border-radius:6px;color:#3fb950;border:1px solid #3fb95066;font-weight:600;font-size:12px}
+.hero .meta{font-size:12px;color:#8b949e}
+.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(320px,1fr));gap:14px}
+.card{background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:16px}
+.card h3{font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:1.2px;margin-bottom:10px;font-weight:600}
+.card .big{font-size:24px;font-weight:700;color:#e6edf3;letter-spacing:-0.5px;margin-bottom:4px}
+.card .sub{font-size:11px;color:#8b949e;line-height:1.5}
+.card a{color:#58a6ff;text-decoration:none;font-size:11px}
+.row{display:flex;justify-content:space-between;align-items:baseline;padding:6px 0;border-bottom:1px dashed #1f2631;font-size:11px}
+.row:last-child{border:none}
+.row .l{color:#8b949e}
+.row .v{color:#e6edf3;font-family:ui-monospace,monospace;font-variant-numeric:tabular-nums}
+.chip{display:inline-block;padding:3px 8px;border-radius:9px;font-size:10px;font-weight:600;margin-right:6px;margin-bottom:4px}
+.ld{color:#3d444d;text-align:center;padding:60px;font-size:13px}
+.empty{color:#545d68;font-size:11px;font-style:italic;line-height:1.5}
+.wide{grid-column:1/-1}
+.heatmap{height:380px;border-radius:8px;border:1px solid #1f2631;overflow:hidden;margin-top:10px}
+.heatmap .leaflet-container{background:#0a0d12}
+.timeline{margin-top:10px;display:flex;align-items:flex-end;gap:2px;height:80px;padding:6px 0;border-bottom:1px solid #1f2631}
+.timeline .tbar{flex:1;background:#1f6feb;min-height:2px;border-radius:2px 2px 0 0;position:relative;cursor:help}
+.timeline .tbar:hover{background:#58a6ff}
+.timeline-axis{display:flex;justify-content:space-between;font-size:10px;color:#545d68;padding-top:4px;font-family:ui-monospace,monospace}
+.placeholder-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(280px,1fr));gap:10px;margin-top:14px}
+.ph-card{background:#0a0d12;border:1px dashed #21262d;border-radius:8px;padding:12px 14px;position:relative}
+.ph-card h4{font-size:11px;color:#8b949e;font-weight:600;margin-bottom:4px;display:flex;align-items:center;gap:6px}
+.ph-card h4 .badge{font-size:9px;padding:2px 6px;border-radius:8px;background:#161b22;color:#d29922;border:1px solid #d2992244;font-weight:600;letter-spacing:0.5px;text-transform:uppercase}
+.ph-card .why{font-size:11px;color:#e6edf3;line-height:1.5;margin-bottom:6px}
+.ph-card .would{font-size:10px;color:#545d68;font-family:ui-monospace,monospace;line-height:1.5;border-top:1px dashed #1f2631;padding-top:6px;margin-top:6px}
+.section-label{font-size:10px;color:#545d68;text-transform:uppercase;letter-spacing:1.4px;font-weight:600;margin:24px 0 8px}
+@media(max-width:640px){.bar{padding:0 14px}.content{padding:14px}.hero{padding:16px}.hero h2{font-size:18px}.card{padding:12px}}
+</style>
+</head><body>
+<div class="bar">
+  <h1>Staffing Co-Pilot · Contractor Profile</h1>
+  <a href=".">← Dashboard</a>
+</div>
+<div class="content">
+  <div class="search-box">
+    <input id="q" type="text" placeholder="Type a contractor name (e.g., Turner Construction Company)" onkeydown="if(event.key==='Enter')lookup()">
+    <button onclick="lookup()">Look up</button>
+  </div>
+  <div id="out"><div class="ld">Type a name above to load the full portfolio across every wired data source.</div></div>
+</div>
+<script>
+function $(id){return document.getElementById(id)}
+
+// Path prefix detection — devop.live serves this page under /lakehouse,
+// localhost:3700 serves it at root. URL rewrites must preserve whatever
+// prefix the user reached the page through, otherwise the back-link and
+// browser refresh break.
+var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
+
+// Bootstrap from URL: /contractor?name=Turner+Construction
+window.addEventListener('load', function(){
+  var name = new URLSearchParams(location.search).get('name');
+  if(name){
+    $('q').value = name;
+    lookup();
+  }
+  // Back link respects the prefix too
+  var back=document.querySelector('.bar a');
+  if(back) back.href=P+'/';
+});
+
+function lookup(){
+  var name = $('q').value.trim();
+  if(!name){ $('out').textContent = ''; return; }
+  history.replaceState({}, '', P+'/contractor?name='+encodeURIComponent(name));
+  var out = $('out');
+  out.textContent = '';
+  var ld = document.createElement('div');
+  ld.className = 'ld';
+  ld.textContent = 'Pulling OSHA, SEC, Stooq, Chicago history, USASpending… (~5-10s on cold cache)';
+  out.appendChild(ld);
+  fetch(P+'/intelligence/contractor_profile',{
+    method:'POST',
+    headers:{'Content-Type':'application/json'},
+    body:JSON.stringify({name:name})
+  }).then(function(r){return r.json()}).then(function(d){
+    render(d);
+  }).catch(function(e){
+    out.textContent = '';
+    var err = document.createElement('div');
+    err.className = 'ld';
+    err.style.color = '#f85149';
+    err.textContent = 'profile failed: '+e.message;
+    out.appendChild(err);
+  });
+}
+
+function render(d){
+  var out = $('out');
+  out.textContent = '';
+
+  // ─── Hero — name, ticker, parent ─────────────
+  var hero = document.createElement('div');
+  hero.className = 'hero';
+  var h2 = document.createElement('h2');
+  h2.textContent = d.display_name;
+  hero.appendChild(h2);
+  var sub = document.createElement('div');
+  sub.className = 'meta';
+  sub.textContent = 'Internal ticker: '+(d.ticker||'?')+' · profile generated '+new Date(d.generated_at).toLocaleTimeString();
+  hero.appendChild(sub);
+
+  var trow = document.createElement('div');
+  trow.className = 'ticker-row';
+  // Direct ticker
+  var s = d.stock;
+  if(s && s.status==='ok'){
+    var tk = document.createElement('span');
+    tk.className = 'ticker';
+    tk.textContent = s.ticker;
+    trow.appendChild(tk);
+    var px = document.createElement('span');
+    px.className = 'meta';
+    px.textContent = (s.company_name||'')+(s.exchange?' · '+s.exchange:'')+(s.price?' · $'+s.price.toFixed(2):'');
+    if(s.day_change_pct!=null && !isNaN(s.day_change_pct)){
+      var ch = (s.day_change_pct>=0?'+':'')+s.day_change_pct.toFixed(2)+'%';
+      var chSpan = document.createElement('span');
+      chSpan.style.color = s.day_change_pct>=0?'#3fb950':'#f85149';
+      chSpan.style.marginLeft = '6px';
+      chSpan.textContent = ch;
+      px.appendChild(chSpan);
+    }
+    trow.appendChild(px);
+  } else {
+    var noTk = document.createElement('span');
+    noTk.className = 'meta';
+    noTk.textContent = 'Private — no direct US ticker';
+    trow.appendChild(noTk);
+  }
+  // Parent link
+  var pl = d.parent_link;
+  if(pl && pl.status==='ok'){
+    var arrow = document.createElement('span');
+    arrow.className = 'meta';
+    arrow.style.color = '#545d68';
+    arrow.textContent = ' → parent ';
+    trow.appendChild(arrow);
+    var pTk = document.createElement('span');
+    pTk.className = 'ticker';
+    pTk.style.color = '#d29922';
+    pTk.style.borderColor = '#d2992266';
+    pTk.textContent = pl.parent_ticker || '?';
+    pTk.title = pl.link_source || '';
+    trow.appendChild(pTk);
+    var pName = document.createElement('span');
+    pName.className = 'meta';
+    pName.textContent = pl.parent_name+(pl.parent_exchange?' · '+pl.parent_exchange:'')+(pl.parent_country?' · '+pl.parent_country:'');
+    trow.appendChild(pName);
+  } else if(pl && pl.status==='no_link'){
+    var pp = document.createElement('span');
+    pp.className = 'meta';
+    pp.style.fontStyle = 'italic';
+    pp.textContent = ' · '+(pl.reason||'no public parent identified');
+    trow.appendChild(pp);
+  }
+  hero.appendChild(trow);
+  out.appendChild(hero);
+
+  // ─── Grid of cards ─────────────────────────────
+  var grid = document.createElement('div');
+  grid.className = 'grid';
+
+  // OSHA
+  var oCard = card('OSHA SAFETY HISTORY (NATIONAL)');
+  var osha = d.osha || {};
+  if(osha.status==='ok'){
+    big(oCard, osha.inspection_count + ' inspections', 'most recent '+(osha.most_recent_date||'?'));
+    rowEl(oCard, 'States seen', (osha.states_seen||[]).join(', ') || '?');
+    rowEl(oCard, 'Most recent', osha.most_recent_date||'?');
+    if(osha.recent_inspections && osha.recent_inspections.length){
+      var rep = document.createElement('div');
+      rep.style.marginTop = '8px';
+      rep.style.fontSize = '10px';
+      rep.style.color = '#545d68';
+      rep.textContent = 'Recent inspections:';
+      oCard.appendChild(rep);
+      osha.recent_inspections.slice(0,5).forEach(function(i){
+        var r = document.createElement('div');
+        r.style.fontSize = '10px';
+        r.style.color = '#8b949e';
+        r.style.fontFamily = 'ui-monospace,monospace';
+        r.style.padding = '2px 0';
+        var a = document.createElement('a');
+        a.href = i.detail_url;
+        a.target = '_blank';
+        a.textContent = i.id;
+        r.appendChild(a);
+        r.appendChild(document.createTextNode(' · '+i.date+' · '+i.state+' · '+i.type+' · '+i.scope));
+        oCard.appendChild(r);
+      });
+    }
+  } else if(osha.status==='no_match'){
+    big(oCard, 'No inspections', 'clean record');
+  } else {
+    empty(oCard, 'OSHA fetch error: '+(osha.error||'unknown'));
+  }
+  grid.appendChild(oCard);
+
+  // Chicago history
+  var hCard = card('CHICAGO PERMIT HISTORY (24mo + LIFETIME)');
+  var hist = d.history || {};
+  if(hist.status==='ok'){
+    big(hCard, hist.permits_historical_total+' permits all-time',
+        hist.permits_last_180d+' in last 180d · '+hist.permits_last_24mo+' in 24mo · trend: '+hist.trend);
+    rowEl(hCard, 'Cost (24mo)', hist.total_cost_last_24mo>=1e6 ? '$'+(hist.total_cost_last_24mo/1e6).toFixed(1)+'M' : '$'+Math.round(hist.total_cost_last_24mo/1e3)+'K');
+    if(hist.recent_permits && hist.recent_permits.length){
+      var rh = document.createElement('div');
+      rh.style.marginTop = '8px';
+      rh.style.fontSize = '10px';
+      rh.style.color = '#545d68';
+      rh.textContent = 'Recent Chicago permits:';
+      hCard.appendChild(rh);
+      hist.recent_permits.slice(0,5).forEach(function(p){
+        var r = document.createElement('div');
+        r.style.fontSize = '10px';
+        r.style.color = '#8b949e';
+        r.style.padding = '2px 0';
+        r.textContent = '· '+(p.date||'?')+' · '+p.work_type+' · $'+(p.cost||0).toLocaleString()+' · '+p.address;
+        hCard.appendChild(r);
+      });
+    }
+  } else {
+    empty(hCard, 'Chicago history error');
+  }
+  grid.appendChild(hCard);
+
+  // Federal contracts
+  var fCard = card('FEDERAL CONTRACTS (USASpending.gov)');
+  var fed = d.federal || {};
+  if(fed.status==='ok' && fed.total_awards_count>0){
+    var dollars = fed.total_awards_value>=1e9 ? '$'+(fed.total_awards_value/1e9).toFixed(2)+'B'
+                : fed.total_awards_value>=1e6 ? '$'+(fed.total_awards_value/1e6).toFixed(1)+'M'
+                : '$'+Math.round(fed.total_awards_value/1e3)+'K';
+    big(fCard, dollars, fed.total_awards_count+' awards · most recent '+(fed.most_recent_award_date||'?'));
+    if(fed.top_agencies && fed.top_agencies.length){
+      var ta = document.createElement('div');
+      ta.style.marginTop = '6px';
+      ta.style.fontSize = '10px';
+      ta.style.color = '#545d68';
+      ta.textContent = 'Top awarding agencies:';
+      fCard.appendChild(ta);
+      fed.top_agencies.forEach(function(a){
+        var r = document.createElement('div');
+        r.style.fontSize = '11px';
+        r.style.color = '#8b949e';
+        r.style.padding = '3px 0';
+        var dollars2 = a.value>=1e6 ? '$'+(a.value/1e6).toFixed(1)+'M' : '$'+Math.round(a.value/1e3)+'K';
+        r.textContent = '· '+a.agency+' — '+dollars2;
+        fCard.appendChild(r);
+      });
+    }
+    if(fed.source_url){
+      var lnk = document.createElement('a');
+      lnk.href = fed.source_url;
+      lnk.target = '_blank';
+      lnk.style.display = 'inline-block';
+      lnk.style.marginTop = '8px';
+      lnk.textContent = 'View on usaspending.gov ↗';
+      fCard.appendChild(lnk);
+    }
+  } else if(fed.status==='no_match'){
+    big(fCard, 'No federal contracts', 'on file under this name');
+  } else {
+    empty(fCard, 'usaspending error');
+  }
+  grid.appendChild(fCard);
+
+  // Debarment + NLRB combined
+  var rCard = card('DEBARMENT + LABOR ACTIONS');
+  var deb = d.debarment || {};
+  var nlrb = d.nlrb || {};
+  rowEl(rCard, 'SAM.gov excluded', deb.status==='needs_setup' ? 'awaiting API key' : (deb.sam_excluded?'YES':'no'));
+  rowEl(rCard, 'IDOL debarred', deb.status==='needs_setup' ? 'awaiting scrape' : (deb.idol_debarred?'YES':'no'));
+  rowEl(rCard, 'NLRB cases', nlrb.status==='needs_setup' ? 'awaiting scrape' : (nlrb.total_cases||0));
+  if(deb.status==='needs_setup' || nlrb.status==='needs_setup'){
+    var dn = document.createElement('div');
+    dn.className = 'empty';
+    dn.style.marginTop = '8px';
+    dn.textContent = 'Both sources pending wire-up: '+(deb.reason||nlrb.reason||'');
+    rCard.appendChild(dn);
+  }
+  grid.appendChild(rCard);
+
+  // ILSOS
+  var iCard = card('CORPORATE REGISTRY (Illinois SoS)');
+  var ilsos = d.ilsos || {};
+  if(ilsos.status==='source_unreachable'){
+    rowEl(iCard, 'Status', 'source blocked at our ASN');
+    var en = document.createElement('div');
+    en.className = 'empty';
+    en.style.marginTop = '8px';
+    en.textContent = ilsos.reason||'';
+    iCard.appendChild(en);
+  } else if(ilsos.status==='ok'){
+    rowEl(iCard, 'Entity name', ilsos.entity_name||'?');
+    rowEl(iCard, 'File #', ilsos.file_number||'?');
+    rowEl(iCard, 'Status', ilsos.status_text||'?');
+    rowEl(iCard, 'Formed', ilsos.formation_date||'?');
+    rowEl(iCard, 'Registered agent', ilsos.registered_agent||'?');
+  } else {
+    empty(iCard, 'no ILSOS data');
+  }
+  grid.appendChild(iCard);
+
+  out.appendChild(grid);
+
+  // ─── Project Index summary — the staffer-facing build-signal score ──
+  var pixHeader = document.createElement('div');
+  pixHeader.className = 'section-label';
+  pixHeader.textContent = '◆ Project Index — build-signal score';
+  out.appendChild(pixHeader);
+
+  var pixCard = document.createElement('div');
+  pixCard.className = 'card wide';
+  // Score is a simple weighted blend of the wired signals — designed to
+  // be replaced with a real model once enough placeholders are wired.
+  var hist2 = d.history || {};
+  var pixScore = 0;
+  var pixDrivers = [];
+  if(hist2.permits_last_180d){ pixScore += Math.min(hist2.permits_last_180d * 5, 30); pixDrivers.push(hist2.permits_last_180d+' Chicago permits in 180d (+'+Math.min(hist2.permits_last_180d*5,30)+')'); }
+  if(hist2.trend === 'rising'){ pixScore += 10; pixDrivers.push('permit trend rising (+10)'); }
+  if(d.osha && d.osha.status==='ok' && d.osha.inspection_count>0){ pixScore -= Math.min(d.osha.inspection_count*5, 25); pixDrivers.push(d.osha.inspection_count+' OSHA inspections (-'+Math.min(d.osha.inspection_count*5,25)+')'); }
+  if(d.federal && d.federal.status==='ok' && d.federal.total_awards_count>0){ pixScore += 15; pixDrivers.push('federally-vetted contractor (+15)'); }
+  if(d.debarment && d.debarment.sam_excluded){ pixScore -= 50; pixDrivers.push('SAM.gov excluded (-50)'); }
+  if(d.stock && d.stock.status==='ok'){ pixScore += 5; pixDrivers.push('public ticker (+5)'); }
+  pixScore = Math.max(0, Math.min(100, 50 + pixScore));
+  var pixColor = pixScore >= 70 ? '#3fb950' : pixScore >= 40 ? '#d29922' : '#f85149';
+  var pixHero = document.createElement('div');
+  pixHero.style.cssText = 'display:flex;align-items:baseline;gap:14px;margin-bottom:8px';
+  var pixBig = document.createElement('span');
+  pixBig.style.cssText = 'font-size:42px;font-weight:700;color:'+pixColor+';letter-spacing:-1px';
+  pixBig.textContent = pixScore;
+  pixHero.appendChild(pixBig);
+  var pixLabel = document.createElement('span');
+  pixLabel.style.cssText = 'font-size:12px;color:#8b949e';
+  pixLabel.textContent = pixScore >= 70 ? 'Strong staffing partner — wired signals positive' : pixScore >= 40 ? 'Mixed signals — review drivers below' : 'Caution — wired signals negative';
+  pixHero.appendChild(pixLabel);
+  pixCard.appendChild(pixHero);
+  if(pixDrivers.length){
+    var pixDrv = document.createElement('div');
+    pixDrv.style.cssText = 'font-size:11px;color:#8b949e;line-height:1.7;font-family:ui-monospace,monospace';
+    pixDrv.textContent = pixDrivers.join(' · ');
+    pixCard.appendChild(pixDrv);
+  }
+  var pixFoot = document.createElement('div');
+  pixFoot.style.cssText = 'font-size:10px;color:#545d68;margin-top:8px;font-style:italic;line-height:1.5';
+  pixFoot.textContent = 'Score is a placeholder weighted blend of the 6 wired signals above. Real ML model lands once 12 awaiting sources below ship — that gives the index 18 features instead of 6.';
+  pixCard.appendChild(pixFoot);
+  out.appendChild(pixCard);
+
+  // ─── Heat map — every Chicago permit they're contact_1 or contact_2 on ─
+  var mapHeader = document.createElement('div');
+  mapHeader.className = 'section-label';
+  mapHeader.textContent = '◆ Where they\'ve worked — Chicago permits, last 24 months';
+  out.appendChild(mapHeader);
+  var mapCard = document.createElement('div');
+  mapCard.className = 'card wide';
+  var mapDiv = document.createElement('div');
+  mapDiv.className = 'heatmap';
+  mapDiv.id = 'cmap';
+  mapCard.appendChild(mapDiv);
+  var mapHint = document.createElement('div');
+  mapHint.style.cssText = 'font-size:11px;color:#545d68;margin-top:8px';
+  mapHint.textContent = 'Loading geo from chicago_permits…';
+  mapCard.appendChild(mapHint);
+  out.appendChild(mapCard);
+
+  // Plot the recent_permits embedded in the contractor profile (now
+  // includes lat/lng/permit_id/description per the entity.ts change).
+  // Color by cost: green <$100K, amber $100K-$1M, red ≥$1M.
+  var permits = (hist2.recent_permits||[]).filter(function(p){return p.lat&&p.lng});
+  if(!permits.length){
+    mapHint.textContent = 'No geocoded permits in the contractor history (Socrata may not have lat/lng for these records).';
+  } else {
+    // Construct map only after the div is in the DOM; defer one tick.
+    setTimeout(function(){
+      var map = L.map('cmap', {zoomControl:true, attributionControl:false}).setView([41.88,-87.63], 11);
+      L.tileLayer('https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png',{maxZoom:19}).addTo(map);
+      var bounds = [];
+      var costs = permits.map(function(p){return Number(p.cost)||0});
+      var maxCost = Math.max.apply(null, costs.concat([1]));
+      permits.forEach(function(p){
+        var c = Number(p.cost)||0;
+        var radius = 4 + (c/maxCost)*16;
+        var color = c >= 1000000 ? '#f85149' : c >= 100000 ? '#d29922' : '#3fb950';
+        var marker = L.circleMarker([p.lat,p.lng],{radius:radius, color:color, weight:1, fillOpacity:0.55});
+        // Build popup via DOM (no innerHTML — keeps the XSS hook happy)
+        var pop = document.createElement('div');
+        pop.style.cssText = 'font-family:ui-monospace,monospace;font-size:11px;color:#0a0d12;min-width:160px';
+        var costRow = document.createElement('div');
+        costRow.style.cssText = 'font-weight:700;margin-bottom:4px';
+        costRow.textContent = '$'+c.toLocaleString()+' · '+(p.date||'?');
+        pop.appendChild(costRow);
+        var wt = document.createElement('div');
+        wt.textContent = p.work_type||'?';
+        pop.appendChild(wt);
+        var addr = document.createElement('div');
+        addr.style.color = '#545d68';
+        addr.textContent = p.address||'?';
+        pop.appendChild(addr);
+        if(p.permit_id){
+          var pid = document.createElement('div');
+          pid.style.cssText = 'color:#545d68;margin-top:4px;font-size:10px';
+          pid.textContent = 'permit '+p.permit_id;
+          pop.appendChild(pid);
+        }
+        marker.bindPopup(pop);
+        marker.addTo(map);
+        bounds.push([p.lat, p.lng]);
+      });
+      if(bounds.length>1) map.fitBounds(bounds, {padding:[24,24]});
+      mapHint.textContent = permits.length+' permits plotted · green <$100K, amber $100K-$1M, red ≥$1M · radius: relative cost';
+    }, 50);
+  }
+
+  // ─── History timeline — monthly permit volume + cost trend ─────────
+  if(hist2.recent_permits && hist2.recent_permits.length){
+    var tlHeader = document.createElement('div');
+    tlHeader.className = 'section-label';
+    tlHeader.textContent = '◆ Activity timeline — Chicago permits by month';
+    out.appendChild(tlHeader);
+    var tlCard = document.createElement('div');
+    tlCard.className = 'card wide';
+    // Bucket by year-month
+    var buckets = {};
+    hist2.recent_permits.forEach(function(p){
+      var d = (p.date||'').substring(0,7); // YYYY-MM
+      if(!d) return;
+      buckets[d] = buckets[d] || {count:0, cost:0};
+      buckets[d].count++;
+      buckets[d].cost += Number(p.cost)||0;
+    });
+    var months = Object.keys(buckets).sort();
+    if(months.length){
+      var maxC = Math.max.apply(null, months.map(function(m){return buckets[m].count}));
+      var tl = document.createElement('div'); tl.className='timeline';
+      months.forEach(function(m){
+        var b = buckets[m];
+        var bar = document.createElement('div'); bar.className='tbar';
+        bar.style.height = Math.max(2, Math.round(b.count/maxC*72)) + 'px';
+        bar.title = m+' · '+b.count+' permit'+(b.count===1?'':'s')+' · $'+Math.round(b.cost).toLocaleString();
+        tl.appendChild(bar);
+      });
+      tlCard.appendChild(tl);
+      var ax = document.createElement('div'); ax.className='timeline-axis';
+      var first = document.createElement('span'); first.textContent = months[0];
+      var last = document.createElement('span'); last.textContent = months[months.length-1];
+      ax.appendChild(first); ax.appendChild(last);
+      tlCard.appendChild(ax);
+    }
+    out.appendChild(tlCard);
+  }
+
+  // ─── 12 awaiting-source placeholders ──────────────────────────────
+  // Each one names a real public data source that would feed the
+  // build-signal index, with a one-line "why a staffer cares" framing
+  // and a sample shape of what the panel would show once wired.
+  var phHeader = document.createElement('div');
+  phHeader.className = 'section-label';
+  phHeader.textContent = '◆ 12 awaiting sources — what plugs in next';
+  out.appendChild(phHeader);
+  var phGrid = document.createElement('div');
+  phGrid.className = 'placeholder-grid';
+  PLACEHOLDERS.forEach(function(p){
+    var c = document.createElement('div'); c.className='ph-card';
+    var h = document.createElement('h4');
+    var name = document.createElement('span'); name.textContent = p.name;
+    var badge = document.createElement('span'); badge.className='badge'; badge.textContent='AWAITING';
+    h.appendChild(name); h.appendChild(badge);
+    c.appendChild(h);
+    var why = document.createElement('div'); why.className='why'; why.textContent = p.why;
+    c.appendChild(why);
+    var would = document.createElement('div'); would.className='would';
+    would.textContent = 'Would show: ' + p.would;
+    c.appendChild(would);
+    phGrid.appendChild(c);
+  });
+  out.appendChild(phGrid);
+
+  // Roadmap footer
+  var foot = document.createElement('div');
+  foot.style.marginTop = '20px';
+  foot.style.fontSize = '10px';
+  foot.style.color = '#484f58';
+  foot.style.lineHeight = '1.6';
+  foot.textContent = 'Wired: OSHA Enforcement · SEC EDGAR + Stooq · Chicago Socrata permits (lat/lng) · USASpending.gov · curated parent-ticker map · ILSOS (datacenter ASN blocked). 12 awaiting sources above are real public datasets that would 3× the feature count of the build-signal index — each one labeled with the one-liner the staffer would ask before placing a worker.';
+  out.appendChild(foot);
+}
+
+// Twelve real public data sources, framed in coordinator language.
+// Each is a placeholder; the panel renders them as "AWAITING" with a
+// description of what they'd add once wired. Order is roughly: highest
+// staffing-decision relevance first.
+var PLACEHOLDERS = [
+  {
+    name: 'DOL Wage & Hour (WHD)',
+    why: 'Has this contractor stiffed workers before? WHD posts every back-wage settlement and unpaid-overtime case.',
+    would: 'cases last 24mo · total back wages owed · status by state · most recent settlement date · whether the workers got paid',
+  },
+  {
+    name: 'State Licensure Boards',
+    why: 'Is the contractor legally allowed to do this work today, in this state?',
+    would: 'license # · status (active / expired / suspended) · trade scope · expiration date · disciplinary history',
+  },
+  {
+    name: 'Surety Bond Capacity',
+    why: 'How big a job can this contractor actually take? Bond ceiling = upper bound on what they\'re bonded for.',
+    would: 'bonding company · single-contract ceiling · aggregate cap · current utilization · recent bond denials',
+  },
+  {
+    name: 'EPA ECHO Compliance',
+    why: 'If a worker shows up to a site with hazmat issues, that\'s the staffing company\'s problem too.',
+    would: 'facility-level violations · last enforcement action · pollutants · whether OSHA escalated',
+  },
+  {
+    name: 'DOT/FMCSA Carrier Safety',
+    why: 'For warehouses with on-site driving or carriers we cross-staff: crash rate, driver out-of-service rate, IFTA filings.',
+    would: 'crash rate per million miles · driver OOS % · vehicle OOS % · safety rating · last compliance review',
+  },
+  {
+    name: 'BBB Complaints + Rating',
+    why: 'What do this contractor\'s own employees say happens to them? BBB aggregates complaints from workers and clients.',
+    would: 'rating · complaint count last 36mo · complaint categories (pay, safety, ghosted) · response rate',
+  },
+  {
+    name: 'PACER Civil Suits (Federal)',
+    why: 'Are they being sued for FLSA, discrimination, or wrongful termination? Filings predate enforcement actions.',
+    would: 'open suits · FLSA / Title VII / ADA breakdowns · counterparties · year-over-year filing rate',
+  },
+  {
+    name: 'UCC Lien Filings',
+    why: 'When a contractor stops paying suppliers, mechanics liens hit the public record. Cash-flow distress signal.',
+    would: 'open liens · total face value · filers (suppliers, banks) · last filing · whether resolved',
+  },
+  {
+    name: 'D&B / Credit Bureau',
+    why: 'Will they pay our staffing invoices? D&B PAYDEX score is the standard.',
+    would: 'PAYDEX (1-100) · days-beyond-terms · credit limit recommendation · UCC link · trade payment trend',
+  },
+  {
+    name: 'State UI Employer Claims',
+    why: 'Workforce stability proxy. A spike in unemployment claims at this employer = layoffs or churn we should know about.',
+    would: 'claims filed against this employer last 12mo · approval rate · separation-reason breakdown',
+  },
+  {
+    name: 'MSHA Mine Safety',
+    why: 'For excavation, demolition, materials, aggregate — MSHA owns the citation history.',
+    would: 'citations · S&S violations · most recent fatality / serious injury · pattern-of-violation flag',
+  },
+  {
+    name: 'Registered Apprenticeships (DOL RAPIDS)',
+    why: 'A contractor with active apprenticeship programs has built a workforce pipeline — different staffing partnership story than one without.',
+    would: 'active programs · apprentice count · trades covered · graduation rate · ethnic/gender diversity reported',
+  },
+];
+
+function card(title){
+  var c = document.createElement('div');
+  c.className = 'card';
+  var h = document.createElement('h3');
+  h.textContent = title;
+  c.appendChild(h);
+  return c;
+}
+function big(c, value, sub){
+  var b = document.createElement('div'); b.className='big'; b.textContent=value;
+  var s = document.createElement('div'); s.className='sub'; s.textContent=sub;
+  c.appendChild(b); c.appendChild(s);
+}
+function rowEl(c, label, value){
+  var r = document.createElement('div'); r.className='row';
+  var l = document.createElement('span'); l.className='l'; l.textContent=label;
+  var v = document.createElement('span'); v.className='v'; v.textContent=value||'—';
+  r.appendChild(l); r.appendChild(v); c.appendChild(r);
+}
+function empty(c, msg){
+  var e = document.createElement('div'); e.className='empty'; e.textContent=msg;
+  c.appendChild(e);
+}
+</script>
+</body></html>

mcp-server/icon_recipes.ts

@@ -0,0 +1,123 @@
+// Visual filler iconography rendered through ComfyUI. Distinct from
+// role_scenes.ts (which renders portraits) — these are object/badge
+// style renders that fill dead space on worker cards: cert pills,
+// role-prop chips, hazard indicators, empty-state heroes.
+//
+// Layout on disk:
+//   data/icons_pool/{category}/{slug}.webp
+//
+// Cache invalidation:
+//   ICONS_VERSION mixes into the on-disk filename (slug includes
+//   version). Bump it after editing a recipe so prior renders are
+//   ignored on next view.
+
+export type IconCategory = "cert" | "role_prop" | "status" | "hazard" | "empty";
+
+export interface IconRecipe {
+  slug: string;
+  category: IconCategory;
+  // Text label that appears next to / under the icon. The front-end
+  // already renders this text in cert pills; the icon is supplementary.
+  display: string;
+  // Full diffusion prompt. Style guidance baked in. SDXL Turbo at 8
+  // steps reliably produces clean macro photography, so default to
+  // photographic prop shots over flat-vector illustrations (the model
+  // hallucinates noise into flat-vector geometry at low step counts).
+  prompt: string;
+  // Negative prompt — what NOT to render. Crucial for icons because
+  // SDXL likes to add hands/text/people unprompted.
+  negative?: string;
+}
+
+// Default negative prompt baked into every icon render unless the
+// recipe overrides. Empirically, these terms are the top SDXL Turbo
+// off-style failures.
+export const DEFAULT_NEGATIVE =
+  "people, hands, faces, blurry, low quality, watermark, signature, "
+  + "logos, copyright, distorted text, garbled letters, multiple objects";
+
+// TODO J — review and tune the prompts here. Each one is what diffusion
+// sees verbatim. The visual decision: photographic prop shots (macro
+// photo of an actual badge / placard / sticker) vs flat-icon vector
+// style. Default below is photographic — matches the worker headshot
+// aesthetic. Flip a recipe to flat-vector by replacing "macro photograph"
+// with "flat icon illustration on solid color background, minimal vector".
+//
+// Visual cues that work well in SDXL Turbo at 8 steps:
+//   - "macro photograph", "isolated on plain background", "studio lighting"
+//   - Concrete colors ("orange and black warning diamond") not adjectives
+//   - Avoid: small text in the prompt (model garbles it), specific brand
+//     names (creates fake logos), detailed scene composition
+const CERT_ICONS: IconRecipe[] = [
+  { slug: "osha-10", category: "cert", display: "OSHA-10",
+    prompt: "macro photograph of a circular yellow safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "osha-30", category: "cert", display: "OSHA-30",
+    prompt: "macro photograph of a circular orange safety badge with a black hard hat icon at center, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "first-aid-cpr", category: "cert", display: "First Aid/CPR",
+    prompt: "macro photograph of a small enamel pin badge featuring a bold red cross on a white circular background, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "hazmat", category: "cert", display: "Hazmat",
+    prompt: "macro photograph of a HAZMAT warning placard, bold orange and black diamond shape with a flame icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "forklift", category: "cert", display: "Forklift",
+    prompt: "macro photograph of a yellow industrial forklift safety badge with a forklift silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "reach-truck", category: "cert", display: "Reach Truck",
+    prompt: "macro photograph of a navy blue industrial certification badge with a warehouse reach-truck silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "order-picker", category: "cert", display: "Order Picker",
+    prompt: "macro photograph of a green industrial certification badge with a warehouse order-picker silhouette icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "lockout-tagout", category: "cert", display: "Lockout/Tagout",
+    prompt: "macro photograph of a bright red padlock tag with a danger warning, hanging on a metal industrial valve, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "msds", category: "cert", display: "MSDS",
+    prompt: "macro photograph of a folded chemical safety data sheet booklet with chemical hazard pictograms visible on cover, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "confined-space", category: "cert", display: "Confined Space",
+    prompt: "macro photograph of a yellow confined space warning sign featuring a manhole entry icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "servsafe", category: "cert", display: "ServSafe",
+    prompt: "macro photograph of a dark green food safety certification badge featuring a stylized chef hat icon, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "fire-safety", category: "cert", display: "Fire Safety",
+    prompt: "macro photograph of a red enamel pin badge featuring a flame icon and a fire extinguisher silhouette, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "iso-9001", category: "cert", display: "ISO 9001",
+    prompt: "macro photograph of a deep blue circular quality-management certification seal with embossed metallic ring, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+];
+
+// Role-band visual chips — small icons that go in the role pill area.
+// One per band, optional inline supplement to the existing colored pill.
+const ROLE_PROP_ICONS: IconRecipe[] = [
+  { slug: "warehouse", category: "role_prop", display: "Warehouse",
+    prompt: "macro photograph of a yellow hard hat with a high-visibility safety vest folded behind it, isolated on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "production", category: "role_prop", display: "Production",
+    prompt: "macro photograph of a navy blue work shirt and protective safety glasses on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "trades", category: "role_prop", display: "Trades",
+    prompt: "macro photograph of a leather work glove and a small adjustable wrench on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "driver", category: "role_prop", display: "Driver",
+    prompt: "macro photograph of a navy delivery driver baseball cap and a clipboard manifest on a neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+  { slug: "lead", category: "role_prop", display: "Lead",
+    prompt: "macro photograph of a tablet showing a bar chart and a high-vis vest folded beside it on neutral grey backdrop, photorealistic, sharp focus, studio lighting" },
+];
+
+export const ICONS: Record<string, IconRecipe> = Object.fromEntries(
+  [...CERT_ICONS, ...ROLE_PROP_ICONS].map((r) => [`${r.category}/${r.slug}`, r]),
+);
+
+// v2 — 256×256 canvas, intended to be displayed monochrome via CSS
+// `filter: grayscale(1)`. Smaller canvas, tighter crops, crisper at
+// 14px display size.
+export const ICONS_VERSION = "v2";
+
+// Map a free-form cert string from the data ("First Aid/CPR",
+// "OSHA-10", "Lockout/Tagout") to the canonical slug used here.
+// Returns null if no recipe matches.
+export function certToSlug(cert: string): string | null {
+  const c = (cert || "").trim().toLowerCase().replace(/\s+/g, "-");
+  if (c === "osha-10") return "osha-10";
+  if (c === "osha-30") return "osha-30";
+  if (c.startsWith("first") || c.includes("cpr")) return "first-aid-cpr";
+  if (c === "hazmat" || c.startsWith("hazwoper")) return "hazmat";
+  if (c === "forklift" || c.startsWith("pit")) return "forklift";
+  if (c.startsWith("reach")) return "reach-truck";
+  if (c.startsWith("order")) return "order-picker";
+  if (c.startsWith("lockout") || c.includes("tagout")) return "lockout-tagout";
+  if (c === "msds" || c.startsWith("ghs")) return "msds";
+  if (c.startsWith("confined")) return "confined-space";
+  if (c === "servsafe") return "servsafe";
+  if (c.startsWith("fire")) return "fire-safety";
+  if (c.startsWith("iso")) return "iso-9001";
+  return null;
+}

mcp-server/observer.ts

@@ -146,15 +146,16 @@ async function persistOp(op: ObservedOp) {
 // ─── LLM Team escalation (code_review mode) ───
 //
 // When recent failures on a single sig_hash cross a threshold the
-// local qwen2.5 analysis is probably insufficient. J's 2026-04-24
+// local-model analysis is probably insufficient. J's 2026-04-24
 // direction: "the observer would trigger to give more context" —
 // route failure clusters to LLM Team's specialized code_review mode
 // (via /api/run) so richer structured signal lands in the KB for
 // scrum + auditor + playbook memory to consume next pass.
 //
-// Non-destructive: runs in parallel to the existing qwen2.5 analysis,
-// never replaces it. Writes to data/_kb/observer_escalations.jsonl
-// as a dedicated audit surface.
+// Non-destructive: runs in parallel to the existing local diagnose
+// call (qwen3.5:latest after the 2026-04-30 bump), never replaces
+// it. Writes to data/_kb/observer_escalations.jsonl as a dedicated
+// audit surface.
 
 const LLM_TEAM = process.env.LH_LLM_TEAM_URL ?? "http://localhost:5000";
 const LLM_TEAM_ESCALATIONS = "/home/profit/lakehouse/data/_kb/observer_escalations.jsonl";
@@ -542,7 +543,7 @@ async function analyzeErrors() {
   if (failures.length === 0) return;
 
   // NEW 2026-04-24: escalate recurring sig_hash clusters to LLM Team
-  // code_review mode. Runs in parallel to the local qwen2.5 analysis
+  // code_review mode. Runs in parallel to the local diagnose call
   // below — non-blocking, richer downstream signal for scrum/auditor.
   maybeEscalate(failures).catch(() => {});
 
@@ -550,13 +551,20 @@ async function analyzeErrors() {
     `[${f.endpoint}] ${f.input_summary}: ${f.error}`
   ).join("\n");
 
-  // Ask local model to diagnose
+  // Ask local model to diagnose. Phase 44 migration (2026-04-27):
+  // /v1/chat instead of legacy /ai/generate so /v1/usage tracks the
+  // call + Langfuse traces it. 2026-04-30 model bump: qwen2.5 →
+  // qwen3.5:latest to match the small-model-pipeline local-tier default.
   try {
-    const resp = await fetch(`${LAKEHOUSE}/ai/generate`, {
+    const resp = await fetch(`${LAKEHOUSE}/v1/chat`, {
       method: "POST",
       headers: { "Content-Type": "application/json" },
       body: JSON.stringify({
-        prompt: `You are a system reliability observer. Analyze these recent failures and suggest fixes:
+        model: "qwen3.5:latest",
+        provider: "ollama",
+        messages: [{
+          role: "user",
+          content: `You are a system reliability observer. Analyze these recent failures and suggest fixes:
 
 ${errorSummary}
 
@@ -566,14 +574,15 @@ For each error:
 3. Should this be added to the playbook as a "don't do this"?
 
 Be specific and actionable. Under 200 words.`,
-        model: "qwen2.5",
+        }],
         max_tokens: 400,
         temperature: 0.2,
       }),
     });
-    const analysis = await resp.json();
-    if (analysis.text) {
-      console.error(`[observer] Error analysis:\n${analysis.text}`);
+    const analysis = await resp.json() as any;
+    const analysisText = analysis?.choices?.[0]?.message?.content ?? "";
+    if (analysisText) {
+      console.error(`[observer] Error analysis:\n${analysisText}`);
       // Log the analysis as a playbook entry
       await fetch(`${GATEWAY}/log`, {
         method: "POST",
@@ -581,7 +590,7 @@ Be specific and actionable. Under 200 words.`,
         body: JSON.stringify({
           operation: `error_analysis: ${failures.length} failures`,
           approach: "LLM-analyzed error patterns",
-          result: analysis.text.slice(0, 500),
+          result: analysisText.slice(0, 500),
           context: errorSummary.slice(0, 500),
         }),
       });
@@ -762,7 +771,7 @@ async function tailOverseerCorrections(): Promise<number> {
     try { row = JSON.parse(line); } catch { continue; }
     const op: ObservedOp = {
       timestamp: row.created_at ?? new Date().toISOString(),
-      endpoint: `overseer:${row.model ?? "gpt-oss:120b"}`,
+      endpoint: `overseer:${row.model ?? "qwen3.5:latest"}`,
       input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`,
       // Correction itself is neither success nor failure — it's a
       // mitigation attempt. We mark success=true so analyzeErrors

mcp-server/onboard.html

@@ -91,6 +91,7 @@ table.preview tr:hover td{background:#0d1117}
   <nav>
     <a href=".">Dashboard</a>
     <a href="console">Walkthrough</a>
+    <a href="profiler">Profiler</a>
     <a href="proof">Architecture</a>
     <a href="spec">Spec</a>
     <a href="onboard" class="active">Onboard</a>

mcp-server/phase_1_6_gate_4.test.ts

@@ -0,0 +1,170 @@
+// Phase 1.6 Gate 4 absence test.
+//
+// Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 4 — Engineering acceptance:
+//   "Unit test asserts no protected-attribute inference functions exist
+//    in search.html or any mcp-server module"
+//
+// What this guards: the FEMALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup
+// tables and the genderFor() / guessEthnicityFromFirstName() / etc.
+// inference functions removed 2026-05-03. Re-introduction would re-open
+// (1) Title VII / IL Human Rights Act discriminatory-feature risk and
+// (2) BIPA's broad-reading "biometric information derived from a biometric
+// identifier" pattern when combined with deepface output.
+//
+// Strategy: walk every .html / .ts / .tsx / .js / .mjs file under
+// mcp-server/ and grep-assert that none of them DEFINE the forbidden
+// symbols. We deliberately allow the symbol NAMES to appear inside
+// comments — search.html has a removal note that names them so future
+// readers know what was excised — but we forbid actual definition
+// patterns (var / const / let / function / class member / object literal).
+
+import { test, expect } from "bun:test";
+import { readdirSync, statSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+
+const FORBIDDEN_DATA_TABLES = [
+  // First-name lookup tables
+  "FEMALE_NAMES",
+  "MALE_NAMES",
+  "NAMES_HISPANIC",
+  "NAMES_BLACK",
+  "NAMES_SOUTH_ASIAN",
+  "NAMES_EAST_ASIAN",
+  "NAMES_MIDDLE_EASTERN",
+  // Surname lookup tables
+  "SURNAMES_HISPANIC",
+  "SURNAMES_BLACK",
+  "SURNAMES_SOUTH_ASIAN",
+  "SURNAMES_EAST_ASIAN",
+  "SURNAMES_MIDDLE_EASTERN",
+];
+
+const FORBIDDEN_FUNCTIONS = [
+  "guessGenderFromFirstName",
+  "guessEthnicityFromName",
+  "guessEthnicityFromFirstName",
+  "genderFor",
+];
+
+function* walkSource(dir: string): Generator<string> {
+  for (const entry of readdirSync(dir)) {
+    if (entry === "node_modules" || entry === "dist" || entry.startsWith(".")) continue;
+    const path = join(dir, entry);
+    const stat = statSync(path);
+    if (stat.isDirectory()) {
+      yield* walkSource(path);
+    } else if (/\.(html|ts|tsx|js|mjs)$/.test(entry)) {
+      // Don't grep this test file itself — it lists the forbidden tokens
+      // by name as match targets, not as definitions.
+      if (path.endsWith("phase_1_6_gate_4.test.ts")) continue;
+      yield path;
+    }
+  }
+}
+
+// definitionPatternsFor: returns regexes that match common DEFINITION
+// forms in JS/TS/HTML embedded scripts. A bare reference inside a
+// comment is intentionally NOT matched.
+//
+// 2026-05-03 opus scrum WARN (gate_4_test:60) added object-literal +
+// class-field patterns: a developer wrapping the lookup tables in
+// `const tables = { FEMALE_NAMES: [...] }` or a TypeScript class
+// field `FEMALE_NAMES: string[] = [...]` would have bypassed the
+// original 4 patterns silently.
+function definitionPatternsFor(symbol: string): RegExp[] {
+  return [
+    // var / const / let SYMBOL =
+    new RegExp(`\\b(?:var|const|let)\\s+${symbol}\\b\\s*=`),
+    // function SYMBOL(
+    new RegExp(`\\bfunction\\s+${symbol}\\s*\\(`),
+    // SYMBOL = function(  OR  SYMBOL = (...) =>
+    new RegExp(`\\b${symbol}\\s*=\\s*(?:function\\s*\\(|\\([^)]*\\)\\s*=>|async\\s*(?:\\(|function))`),
+    // class method: SYMBOL(...) {
+    new RegExp(`^\\s*${symbol}\\s*\\([^)]*\\)\\s*\\{`, "m"),
+    // object-literal property assigned to an array OR object value:
+    //   { SYMBOL: [...]  }  or  SYMBOL: {...}  or  SYMBOL: new Set(...)
+    new RegExp(`(?:^|[,{])\\s*${symbol}\\s*:\\s*(?:\\[|\\{|new\\s+(?:Set|Map|Array)\\b)`, "m"),
+    // TypeScript / class field with type annotation. The boundary
+    // before SYMBOL is start-of-line OR `{`/`;`/`}` so single-line
+    // class bodies (`class L { public NAMES_X: string[] = []; }`)
+    // are caught alongside multi-line ones.
+    new RegExp(`(?:^|[{};])\\s*(?:public|private|protected|readonly|static\\s+)*\\s*${symbol}\\s*:\\s*[^=;{]+=\\s*[\\[\\{]`, "m"),
+  ];
+}
+
+function findOffenders(filePath: string, text: string): string[] {
+  const out: string[] = [];
+  for (const sym of [...FORBIDDEN_DATA_TABLES, ...FORBIDDEN_FUNCTIONS]) {
+    for (const pattern of definitionPatternsFor(sym)) {
+      if (pattern.test(text)) {
+        out.push(`${filePath}: definition of ${sym} (matched ${pattern})`);
+      }
+    }
+  }
+  return out;
+}
+
+test("Gate 4: no protected-attribute inference DEFINITIONS in mcp-server", () => {
+  const root = import.meta.dir;
+  const offenders: string[] = [];
+  for (const path of walkSource(root)) {
+    const text = readFileSync(path, "utf8");
+    offenders.push(...findOffenders(path, text));
+  }
+  if (offenders.length > 0) {
+    throw new Error(
+      `Phase 1.6 Gate 4 violation — protected-attribute inference symbols defined in mcp-server:\n` +
+        offenders.map((o) => `  - ${o}`).join("\n"),
+    );
+  }
+  expect(offenders.length).toBe(0);
+});
+
+// Sanity: confirm the test actually walks files (otherwise the absence
+// assertion is vacuously true). If mcp-server ever lost its source
+// tree, this would catch it.
+test("Gate 4: walker actually finds source files to scan", () => {
+  const root = import.meta.dir;
+  let count = 0;
+  for (const _ of walkSource(root)) count++;
+  expect(count).toBeGreaterThan(5); // mcp-server has more than 5 source files
+});
+
+// Defense in depth: the regex itself must catch a synthetic positive.
+// If the definition pattern ever stops matching real code, the absence
+// test would silently pass on actual reintroductions.
+test("Gate 4: regex catches a synthetic positive (defense in depth)", () => {
+  const synthetic =
+    `var NAMES_HISPANIC = ["Maria"];\n` +
+    `function guessEthnicityFromFirstName(name) { return "?"; }\n`;
+  const offenders = findOffenders("synthetic_test_input", synthetic);
+  expect(offenders.length).toBeGreaterThanOrEqual(2);
+  expect(offenders.some((o) => o.includes("NAMES_HISPANIC"))).toBe(true);
+  expect(offenders.some((o) => o.includes("guessEthnicityFromFirstName"))).toBe(true);
+});
+
+// 2026-05-03 opus scrum WARN regression: the bypass forms a developer
+// might use to wrap the lookup tables without tripping the original
+// four patterns. All of these MUST trip a definition regex.
+test("Gate 4: object-literal and class-field bypasses are caught", () => {
+  const bypassForms = [
+    // Inline object-literal property → array
+    `const tables = { FEMALE_NAMES: ["Maria"] };`,
+    // Object property → Set/Map constructor
+    `const lookups = { NAMES_BLACK: new Set(["X"]) };`,
+    // Multi-line object literal
+    `const all = {\n  NAMES_HISPANIC: [...],\n};`,
+    // TypeScript class field with type annotation + initializer
+    `class Lookup {\n  SURNAMES_BLACK: string[] = ["X"];\n}`,
+    // TypeScript public field
+    `class L { public NAMES_EAST_ASIAN: string[] = []; }`,
+  ];
+  for (const synthetic of bypassForms) {
+    const offenders = findOffenders("bypass_synthetic", synthetic);
+    if (offenders.length === 0) {
+      throw new Error(
+        `Gate 4 bypass not caught — pattern would slip past:\n  ${synthetic}`,
+      );
+    }
+  }
+});