diff --git a/.gitignore b/.gitignore index 2a9ebf2..37cad58 100644 --- a/.gitignore +++ b/.gitignore @@ -50,3 +50,28 @@ reports/distillation/*-*-*-*-*/ tests/agent_test/_* tests/agent_test/sessions/ tests/real-world/runs/ + +# BIPA-quarantined photo uploads — Phase 1.6 Gate 3a writes to +# data/biometric/uploads//_. with mode 0700/0600. +# This is regulated subject-of-record data and must NEVER enter git. +data/biometric/ + +# Local-only scrum review evidence. Per `feedback_audit_findings_log.md` +# scrum runs fold fixes into a batch commit; the verdict / disposition +# files stay local for forensics. +reports/scrum/ + +# Per-event biometric verification reports (timestamp-named, regenerated +# per `verify_biometric_erasure.sh` invocation). Source-of-truth is the +# audit chain itself; these reports are derived views. +reports/biometric/ + +# Counsel transmission tarballs + manifests are regenerated by +# `bundle_counsel_packet.sh` from the tracked `docs/counsel/` source. +# The bundle is transmittable, not source-of-truth. +reports/counsel/ + +# Local experiments scratchpad — per the "Test code in main is ACTIVELY +# being cleaned out" policy (commits 6aafd41 + f4ebd22), one-off +# experiments stay out of the tracked tree. +experiments/ diff --git a/Cargo.lock b/Cargo.lock index 4b0285b..1985e5d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1365,10 +1365,13 @@ dependencies = [ "axum", "bytes", "chrono", + "hmac", "object_store", + "parquet 55.2.0", "proto", "serde", "serde_json", + "sha2", "shared", "storaged", "tokio", diff --git a/Cargo.toml b/Cargo.toml index c4566ed..74b2595 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -36,6 +36,7 @@ datafusion = "47" bytes = "1" futures = "0.3" sha2 = "0.10" +hmac = "0.12" url = "2" tonic = "0.13" prost = "0.13" diff --git a/STATE_OF_PLAY.md b/STATE_OF_PLAY.md index 31d1150..e9a865d 100644 --- a/STATE_OF_PLAY.md +++ b/STATE_OF_PLAY.md @@ -1,13 +1,220 @@ # STATE OF PLAY — Lakehouse -**Last verified:** 2026-05-02 evening CDT -**Verified by:** live probe (smoke 9/9, parity 32/32, gateway restarted), not memory. +**Last verified:** 2026-06-30 (search index-alignment fix live on :3700) · biometric substrate 2026-05-05 +**Verified by (2026-06-30):** live probe after mcp-server restart — `/match` + `/search` with `state='IL'` return only IL workers (new PID on :3700). Not memory. +**Verified by (2026-05-05):** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory. > **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes. --- -## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits this wave) +## WHAT LANDED 2026-06-30 (search index-alignment sweep + P1-already-done finding) + +**Fixed a systemic hybrid-search bug.** Every `/vectors/hybrid` call that paired the +`ethereal_workers` SQL dataset (10K rows) with the `workers_500k_v1` vector index (50K, +a *different* worker set) returned wrong-entity results: same `worker_id` = different +person in each index, so a `state='IL'` query surfaced a **KY** worker flagged +`sql_verified:true`. Root cause: the default index didn't share an id-space with the +default dataset. + +Swept **8 call sites** in `mcp-server/index.ts` to `ethereal_workers_v1` (10K, same +id-space as the SQL table). Aligned sites (`filter_dataset: workers_500k`) and pure-RAG +sites (`/vectors/rag`) were correctly left alone. + +| Commit | Sites | +|---|---| +| `423400e` | `/search` HTTP handler | +| `35a80a7` | `search_workers` tool · `match_contract` tool · `/match` HTTP handler | +| `1570306` | 4 intelligence routes (similar-workers, role-discovery, smart_search, scenario-find) | + +**Verified live** after an mcp-server restart (new PID on :3700): `/match` and `/search` +with `state='IL'` return only IL workers (Champaign/Springfield/Urbana); `state='WI'` +returns WI. Grep confirms no `workers_500k_v1` index remains paired with `ethereal_workers`. +Server boots clean. Only `mcp-server/index.ts` was touched. + +**Finding — demo-plan P1 was already done.** The "search box that actually filters" item +in this doc's CURRENT PLAN (P1, from the 2026-04-27 demo audit) is **stale**. The UI now +posts structured `state`/`role` to the evolved `/intelligence/chat`, which returns real +`sql_results` (IL Forklift → 167 matches, correct workers; WI → WI). The P1 section below +describes a bug that no longer exists — don't re-implement it. + +**⚠ Uncommitted WIP left untouched (pre-existing, not from this session).** +`mcp-server/search.html` + `proof.html` (edited 2026-05-30; proof.html restructured into +chapters) and a `biometric_endpoint.rs` comment cleanup remain uncommitted in the working +tree. Also untracked and newer than this doc: `RECOVERY_PLAN_2026-05-05.md`, +`mcp-server/UX_FIX_PLAN.md`, `docs/scope-sentinel/`. Someone should reconcile these — they +imply planning context this file doesn't yet capture. + +--- + +## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard) + +After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`. + +| Commit | What | Verified | +|---|---|---| +| `76cb5ac` | `POST /biometric/subject/{id}/consent` — Gate 2 backend. Flips status NeverCollected→Given, computes retention_until = given_at + 18mo (per retention schedule v1 §4), captures consent_version_hash + collection_method (closed set: electronic_signature/paper/click_acceptance) + operator + evidence path to audit row. State machine: 409 already_given, 409 post-withdrawal-requires-erase, 403 subject_inactive. | 12 unit tests; live POST returns 401/400/404 on guards | +| `7f0f500` | Candidate intake UI at `/biometric/intake?candidate_id=XXX` (mcp-server :3700, exposed via nginx at devop.live/lakehouse/biometric/intake). 4-screen flow: operator auth → consent template render + click-accept → photo capture (file or webcam getUserMedia) → confirmation showing audit hmac. SHA-256 of rendered consent block becomes consent_version_hash. sessionStorage-backed token (clears on tab close), neo-brutalist style matching onboard.html. | 22631-byte HTML, mcp-server route returns 200 | +| `68d226c` | Three things in one wave:
(a) `POST /biometric/subject/{id}/withdraw` — BIPA right of withdrawal. Flips Given→Withdrawn, accelerates retention_until from 18mo→30d (per consent template v1 §2 SLA), audit kind=biometric_consent_withdrawal. State machine 409s on NeverCollected/Pending (nothing_to_withdraw), Withdrawn (already_withdrawn), Expired (already_expired). 12 unit tests.
(b) Withdraw UI at `/biometric/withdraw` — 3-screen operator flow (token+name auth → reason+evidence form → confirmation showing 30-day clock + verify curl recipe).
(c) `lakehouse-retention-sweep.{service,timer}` systemd units in `ops/systemd/`. Daily 03:00 UTC, Persistent=true, install.sh updated to handle paired timer+oneshot service.
Plus operator_of_record bug fix in intake UI (was hardcoded `'intake_ui_operator'`). | 46/46 biometric_endpoint + 71/71 catalogd lib tests; manual sweep run: 100 subjects, 0 overdue, exit 0 | +| **(current HEAD post-this-wave)** | Stats endpoint `GET /biometric/stats` (legal-tier auth, returns subject counts by status + photo count + oldest active retention + last 20 state-change events with anonymizable trace_ids) + ops dashboard at `/biometric/dashboard` (single-page, polls /stats every 15s, table + status tiles, XSS-safe DOM construction not innerHTML). Plus consent_versions allowlist: `BiometricEndpointState.allowed_consent_versions: Option>>`, loaded from `/etc/lakehouse/consent_versions.json` (`LH_CONSENT_VERSIONS_FILE` override), missing-file = permissive (v1 compat), present + populated = strict mode (refuses unknown hashes with 400 consent_version_unknown). Plus `scripts/staffing/subject_timeline.sh` — pretty-prints any subject's full BIPA lifecycle from /audit/subject/{id} (manifest + on-disk photo + chronological audit chain + verification status). | 5 new allowlist unit tests + 4 stats tests; live demo on WORKER-100 ran end-to-end (consent → photo → withdraw, chain verified=true, chain_root=a47563ff…) | + +### Live demo evidence (WORKER-100, 2026-05-05 20:12 UTC) +The full lifecycle was exercised against the live gateway as a verification artifact. The audit chain on WORKER-100 now contains 3 rows: + +``` +20:12:33.054 BIOMETRIC_CONSENT_GRANT result=given hmac=9c6f4153341e97d2… trace=live-demo-2026-05-05 +20:12:47.957 BIOMETRIC_COLLECTION result=success hmac=856be6173c88277c… trace=live-demo-2026-05-05 +20:15:27.298 BIOMETRIC_CONSENT_WITHDRAWAL result=withdrawn hmac=a47563ff937d50de… trace=live-demo-2026-05-05 +``` + +`chain_verified=true`, chain_root = a47563ff937d50de43b09a0c903cff954233836c219a928ee8ca2aa6792272dd. Photo file at `data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg` (180 bytes — a minimal real JFIF JPEG), retention_until=2026-06-04 (= 30 days from withdrawal). Retention sweep will flag this subject on or after that date; operator runs `/biometric/subject/WORKER-100/erase` to destroy. + +To re-verify: `./scripts/staffing/subject_timeline.sh WORKER-100`. + +### Endpoint matrix (v1 BIPA lifecycle complete) + +| Event | Endpoint | Method | Auth | Status flip | retention_until | +|---|---|---|---|---|---| +| consent given | `/biometric/subject/{id}/consent` | POST | legal | NeverCollected/Pending → Given | now + 18mo | +| photo collected | `/biometric/subject/{id}/photo` | POST | legal + consent gate | (no change) | (no change) | +| consent withdrawn | `/biometric/subject/{id}/withdraw` | POST | legal | Given → Withdrawn | now + 30d | +| destruction | `/biometric/subject/{id}/erase` | POST | legal | (manifest cleared) | n/a | +| audit read | `/audit/subject/{id}` | GET | legal | (read-only) | (read-only) | +| ops aggregates | `/biometric/stats` | GET | legal | (read-only) | (read-only) | + +UIs: +- `/biometric/intake?candidate_id=X` — operator-driven consent + photo +- `/biometric/withdraw` — operator-driven withdrawal recording +- `/biometric/dashboard` — read-only ops aggregate, auto-refresh + +CLI tools: +- `scripts/staffing/verify_biometric_erasure.sh ` — post-erasure verification +- `scripts/staffing/biometric_destruction_report.sh --month YYYY-MM` — anonymized monthly report +- `scripts/staffing/subject_timeline.sh ` — full lifecycle pretty-print (NEW 2026-05-05) +- `scripts/staffing/bundle_counsel_packet.sh` — counsel review tarball +- `scripts/staffing/attest_pre_identityd_biometric_state.sh` — defense attestation generator + +### What's blocking production cutover NOW +**Counsel calendar.** Engineering substrate is done end-to-end: every state transition has a defensible endpoint, every endpoint has tests + live verification, every UI is reachable, retention sweep is scheduled, allowlist hardening is wired. The remaining work is signature/review: +1. Counsel review of consent template v1 (revised for Option C — classifications deferred) +2. Counsel review of retention schedule v1 (revised for Option C) +3. Counsel review of destruction runbook +4. Counsel + J signatures on §2 attestation +5. Once counsel signs the consent template, populate `/etc/lakehouse/consent_versions.json` with the signed hash to flip the gateway from permissive to strict mode + +Counsel-review packet at `reports/counsel/counsel_packet_2026-05-05.tar.gz` (regenerable via `bundle_counsel_packet.sh` to pick up the latest doc state). + +--- + +## WHAT LANDED 2026-05-05 (morning wave — doc reconciliation + Gate 3b decision + counsel packet) + +This was a **doc-only wave**, not code. Background: J asked for an audit of the BIPA/biometric documentation before production cutover. Audit found moderate fragmentation between docs and shipped code (post-`identityd` collapse, post-Gate-3a-ship, pre-Gate-3b-decision). Closed it in one pass. + +| Item | What changed | Status | +|---|---|---| +| **Gate 3b — DECIDED: Option C (defer classifications)** | `BiometricCollection.classifications` stays `Option = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status flipped from "draft / awaits product" to "DECIDED 2026-05-05". | Locked | +| **Endpoint-path drift** | `PHASE_1_6_BIPA_GATES.md` (3 spots), `BIPA_DESTRUCTION_RUNBOOK.md` (2 spots), `biometric_retention_schedule_v1.md` (1 spot) updated from legacy `/v1/identity/subjects/*` (proposed under separate identityd daemon, never shipped) to actual `/biometric/subject/*` (catalogd-local, shipped `848a458`). Schema block in `PHASE_1_6_BIPA_GATES.md` rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate (not the proposed Postgres `subjects` table). | Reconciled | +| **Consent template + retention schedule** | Both revised for Option C: removed all "automated facial-classification" / "deepface" language so disclosed scope matches implemented scope. Pending counsel review — they were already eng-staged with ⚖ markers. | Eng-staged for counsel | +| **`scripts/staffing/verify_biometric_erasure.sh`** (NEW) | Operator-side verification of an erasure event. Curls `/audit/subject/{id}` with legal-tier token, checks: manifest.biometric_collection null, uploads dir empty, last audit row is `biometric_erasure`/`full_erasure` with `erased`/`success`, chain_verified=true. Writes a hashed report to `reports/biometric/`. | Smoke-tested live | +| **`scripts/staffing/biometric_destruction_report.sh`** (NEW) | Monthly destruction-event aggregation. Anonymizes candidate IDs (sha256-12 prefix), counts by scope + trigger, flags anomalies. Smoke-test on May 2026 data found 1 historical `biometric_erasure`/`consent_withdrawal` event (test fixture). | Smoke-tested live | +| **`docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`** (NEW) | Captures the rotation procedure operationalized after the 2026-05-05 `/tmp` wipe incident. Covers: when to rotate, pre-rotation snapshot, atomic-swap procedure, post-rotation verification (incl. expected pre-rotation chain tamper-detect under new key), recovery from lost key, ⚖ counsel notes. | Authored | +| **`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`** + `bundle_counsel_packet.sh` (NEW) | Cover note bundling all eng-staged BIPA docs for counsel review with per-doc questions, sign-off checklist, recommended review sequence. Bundler script tarballs the 8 referenced files + emits a SHA-256 manifest. Tarball ready for transmission: `reports/counsel/counsel_packet_2026-05-05.tar.gz`. | Bundled, ready to send | + +### Eng follow-up that this wave surfaced +- **Double-upload file leak — DIAGNOSED + FIXED** (2026-05-05 same wave). `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo file. Investigation showed: + - The file was 13 bytes of test fixture (`ff d8 ff d9 + ASCII "TESTBYTES"`), byte-identical to the unit-test fixture at `biometric_endpoint.rs:841`. NO PII, NO biometric content, NO synthetic-face content. Came from manual integration testing on 2026-05-03. + - Audit log timeline showed two consecutive uploads (09:54, 10:04) followed by one erasure (10:22). The erasure unlinked only the SECOND file (which the manifest pointed at by then); the first file was orphaned because the second upload had silently overwritten `manifest.data_path`. + - **Real bug found**: the upload handler did NOT refuse a second upload to a subject with `biometric_collection.is_some()`. Patched `process_upload` to return HTTP 409 + `error: "biometric_already_collected"` when a re-upload is attempted; operator must explicitly POST `/biometric/subject/{id}/erase` first. + - Stranded test file removed (`rm` of the 13-byte fixture). + - New unit test `second_upload_without_erase_returns_409` asserts the 409 + that the first photo's data_path remains unchanged + that the first file remains untouched on disk. + - Existing `repeated_uploads_grow_the_chain` replaced with `upload_erase_upload_grows_the_chain_cleanly` (covers the legitimate re-collection cycle: upload → erase → upload, chain grows to 3 rows). + - Existing `content_type_with_parameters_accepted` test updated to use two distinct subjects (it had used one subject for two uploads to test content-type parsing — now would 409). + - **22 biometric_endpoint tests + 59 catalogd lib tests all green** post-patch (was 21+58 pre-patch). + - Production posture: gateway binary needs rebuild (`cargo build --release`) + `systemctl restart lakehouse.service` to pick up the new 409 behavior in live traffic. +- **Pre-rotation chain tamper-detect (expected, not a bug).** WORKER-{1..5} had pre-2026-05-05 audit chains under the prior `LH_SUBJECT_AUDIT_KEY`. Under the new key (post-`/tmp` wipe rotation), those chains correctly tamper-detect. The rotation runbook §4.4 documents this as expected; a §2.2 pre-rotation snapshot is what would prove they were intact pre-rotation if defensibility ever needs it. + +### What's blocking production cutover NOW (after this wave) +- **Counsel calendar:** the four sign-off items in `COUNSEL_REVIEW_PACKET_2026-05-05.md` (retention schedule, consent template, destruction runbook, pre-identityd attestation). The packet tarball is ready; ⚖ counsel is the bottleneck. +- **Nothing else.** Engineering is no longer the long pole. + +### Phase 1.6 BIPA gates — status table (this is the final post-Option-C state) + +| # | Gate | Status | +|---|---|---| +| 1 | Public retention schedule | **eng-staged**, revised for Option C, ready for counsel | +| 2 | Informed consent template | **eng-staged**, revised for Option C, ready for counsel | +| 3a | Photo upload endpoint | **DONE** (shipped `f1fa6e4`, 11 unit tests, live verified) | +| 3b | Deepface classification | **DECIDED 2026-05-05: Option C (defer)** | +| 4 | Name → ethnicity inference removal | **DONE** (shipped, 4/4 mcp-server tests pass) | +| 5 | Destruction runbook + erasure endpoint | **eng-DONE** (`848a458`, 21 tests). Runbook scripts (verify + report) shipped 2026-05-05. Counsel review pending. | +| §2 | Pre-identityd attestation | **eng-DONE** (3/3 evidence checks). Awaits J + counsel signature. | +| §3 | Employee training | **deferred** (consolidated into runbook §7 acknowledgment for current operator population) | + +--- + +## WHAT LANDED 2026-05-03 (16 commits this wave — local-first audit substrate + Phase 1.6 BIPA gates) + +The dominant work today: **`docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8 SHIPPED end-to-end** + **5 of 7 Phase 1.6 BIPA pre-launch gates** + **6th cross-runtime parity probe**. Wave was structured as eight ship-then-scrum cycles — every wave caught real bugs, every fix wave landed within the same session. + +| Commit | What | Verified | +|---|---|---| +| `d259909` | catalogd: Step 1 — `SubjectManifest` type + Registry CRUD | 17 catalogd subject tests PASS | +| `d16131b` | catalogd: Step 2 — `SubjectAuditWriter` HMAC-SHA256 chain + per-subject Mutex + canonical-JSON via BTreeMap | tamper-detection + concurrent-append race tests PASS | +| `bce6dfd` | catalogd: Step 3 — `bin/backfill_subjects` (BIPA-defensible defaults: vertical=unknown, consent=pending_backfill_review, retention=4yr) | 100 subjects loaded into live catalog | +| `fef1efd` | gateway: Step 4 — wire SubjectAuditWriter into `/v1/chat` tool dispatch + `audit_subject_hits_in` (inline, not spawn) | tool calls log accessor.kind="gateway_lookup" | +| `cd8c59a` | gateway: Step 5 — `AuditingWorkerLookup` decorator wraps validator's WorkerLookup; spawns audit on every find() | live `/v1/validate` produces audit rows | +| `e38f357` | subjects Steps 1-4 fixes from cross-lineage scrum (concurrency race, schema-evolution HMAC drift, hardcoded "success" classifier) | 41 tests green | +| `15cfd76` | catalogd + gateway: Step 6 — `/audit/subject/{id}` legal-tier endpoint with constant-time-eq token check + tampering detection | live curl returns chain_verified=true | +| `2a4b316` | subjects 2nd scrum fix wave (token min 16→32, chain_root from full chain via `chain_tip()`, rebuild collision warn, tightened result-state heuristic) | 17 catalogd + 6 gateway tests PASS | +| `8fc6238` | catalogd: Step 7 — `bin/retention_sweep` (BIPA-aware on biometric clock, idempotent across daily runs, no auto-mutation) | 8 sweep tests PASS, live verified at `--as-of 2031-06-01` flagging 100/100 expired | +| `2413c96` | catalogd: Step 8 — `bin/parity_subject_audit` (Rust side of cross-runtime parity probe) | known-answer + verify modes match Go byte-for-byte | +| `2222227` | parity helper hardening (panic-noise → die() helper, abs path stripped from doc) from scrum | parity probe still 6/6 | +| `4708717` | **Phase 1.6 BIPA wave** — Gate 4 absence test (4/4 with bypass coverage), §2 attestation script (3/3 evidence checks), Gate 1/2/5 doc scaffolds with ⚖ COUNSEL markers | 4/4 mcp-server Bun tests, 3/3 evidence on live data | +| `c7aa607` | Phase 1.6 scrum fixes — schema fingerprint hashes name+type+nullable, Gate 4 catches object-literal + class-field bypasses, pyarrow dep gate, item 7 deferral rationale | 4/4 + 3/3 still pass | +| `f1fa6e4` | **Phase 1.6 Gate 3a** — `crates/catalogd/src/biometric_endpoint.rs`: `POST /biometric/subject/{id}/photo` with consent gate, quarantined storage (mode 0700/0600), audit chain link, `BiometricCollection` field on SubjectManifest | 11 unit tests PASS, live roundtrip 200 | +| `3708e6a` | Gate 3a scrum fixes — transactional rollback on audit failure (BIPA convergent BLOCK), Content-Type parameter handling, relative data_path, ts+uuid filename, dead code removed | 11 tests + cross-runtime parity 6/6 | +| `7e0112b` | retention_sweep: stray indent fix on biometric_collection field | sweep tests still 8/8 | +| `848a458` | **Phase 1.6 Gate 5** — `POST /biometric/subject/{id}/erase` per BIPA destruction runbook. Two scopes (biometric_only / full); audit row appended BEFORE photo unlink so the chain has legal proof of intent even if file delete fails; manifest rolled back on audit failure. Trigger taxonomy: retention_expiry / consent_withdrawal / rtbf / court_order. | 21 unit tests (10 erasure-specific) PASS | +| `8ec43e0` | **Phase 1.6 Gate 3b** — deepface integration design doc (Option A subprocess / Option B ONNX-in-Rust / **Option C defer**). Recommends C: BIPA-safest, classifications field stays None, all load-bearing surfaces (consent + audit + retention + erasure) ship without it. Forces "do we actually need classifications" to be answered by product, not spec inertia. | doc-only | + +**Cross-runtime parity (post-this-wave):** 6 probes, 38/38 byte-identical assertions — +`validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8) + subject_audit(6/6)`. +Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify. + +**Three runtime-divergence classes caught + fixed by the parity probe authoring loop** (cataloged because they recur): +1. Go `omitempty` on string fields strips empty values that Rust serde always emits → broken HMAC +2. `time.RFC3339Nano` truncates trailing-zero nanoseconds where chrono `AutoSi` keeps 9 digits → broken HMAC +3. Go `json.Marshal` HTML-escapes `<>&` where serde keeps literal → broken HMAC on any field with those chars + +All three have regression-locked tests; structural impossibility going forward. + +**Phase 1.6 BIPA pre-launch gates — status table:** + +| Item | Status | Evidence | +|---|---|---| +| Gate 1 — public retention schedule | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_retention_schedule_v1.md` | +| Gate 2 — informed consent template | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_consent_template_v1.md` | +| Gate 3a — photo-upload endpoint | **DONE** | 11 unit tests + live `POST /biometric/subject/{id}/photo` | +| Gate 3b — deepface classification | **design doc shipped** (`8ec43e0`) — recommends Option C (defer); awaits product confirmation | `docs/PHASE_1_6_BIPA_GATES.md` Gate 3b section | +| Gate 4 — name→ethnicity removal | **DONE** | `mcp-server/phase_1_6_gate_4.test.ts` 4/4 with bypass coverage | +| Gate 5 — destruction runbook + erasure endpoint | **eng-DONE** (`848a458`); ⚖ counsel review of runbook still pending | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` + `POST /biometric/subject/{id}/erase` (21 tests) | +| §2 cryptographic attestation | eng-DONE, signature pending | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (SHA-256 evidence hash, 3/3 checks pass on live data) | +| §3 employee training | deferred | conditional on operator population size | + +**Calendar bottleneck:** counsel review of items 1/2/5-runbook/§2 attestation. Engineering long pole is Gate 3b (deepface) — design doc landed (`8ec43e0`); needs product confirmation that classifications are required before engineering starts. Recommendation in doc is Option C (defer) on BIPA-safety grounds. + +**Operational state:** +- `LH_SUBJECT_AUDIT_KEY=/etc/lakehouse/subject_audit.key` (32-byte HMAC signing key, mode 0400) loaded into systemd unit. **Moved off /tmp 2026-05-05** — /tmp wipes on reboot, which on May 5 disabled `/audit` + `/biometric` endpoints (gateway fails-closed at `crates/gateway/src/main.rs:459` if signing key is absent). Persistent path is per spec line 112. +- `LH_LEGAL_AUDIT_TOKEN_FILE=/etc/lakehouse/legal_audit.token` (44-char legal-tier token, mode 0400) loaded into systemd unit +- **Key rotation 2026-05-05:** prior key was lost when /tmp wiped on reboot. New key generated at canonical path. The 5 pre-rotation audit chains for `WORKER-{1..5}` (backfill data with `consent=pending_backfill_review`) will tamper-detect under the new key — expected and correct behavior on key rotation, not a bug. New chain entries from 2026-05-05 forward verify cleanly. +- `data/_catalog/subjects/` holds 100 backfilled `WORKER-N.json` manifests + per-subject `WORKER-N.audit.jsonl` HMAC chains +- `data/biometric/uploads//_.` quarantined photo storage (mode 0700 dir / 0600 file). 2 photos uploaded for WORKER-2 during live verify. +- `/audit/subject/{id}` mounted on gateway with chain_verified=true on every probe +- `/biometric/subject/{id}/photo` mounted on gateway, refuses 403 without `consent.biometric.status="given"` + +--- + +## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits — Lance gauntlet + cross-runtime parity wave) | Commit | What | Verified | |---|---|---| @@ -122,6 +329,10 @@ OpenCode catalog (live): - **Lance backend gauntlet (2026-05-02)** — sanitizer over all 5 routes, 7 unit tests, 9-probe smoke, 10M re-bench. The `doc_id` btree auto-builds inside `lance_migrate` AND `lance-bench`. Do not "discover" the missing scalar index again or the leaked filesystem paths in error bodies. - **Cross-runtime parity = 32/32** across 5 probes in `golangLAKEHOUSE/scripts/cutover/parity/`. Do not "build a parity probe for X" without checking — validator, extract_json, session_log, materializer, and embed are all already covered. - **Decisions tracker is `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`** — single living source of truth for cross-runtime decisions. As of 2026-05-02 it has 0 `_open_` code work items; only 2 strategic items left (Lance vs Parquet+HNSW-with-spilling, Go-vs-Rust primary cutover). +- **PRD line 70 is load-bearing — "Everything runs locally, no cloud APIs."** Yesterday's PR #13 violated this by routing customer hot-path inference to opencode/openrouter/ollama_cloud. **REVERTED 2026-05-03 (`d054c0b`).** The customer hot path (modes.toml staffing_inference, doc_drift_check; execution_loop overseer escalation) is now local Ollama (qwen3.5:latest). Cloud providers stay configured in providers.toml for **explicit dev-tool opt-in only** (scrum, auditor, bot/propose). Do NOT re-add cloud models to the hot path defaults — the customer demo runs on local + free. +- **`/v1/usage` shows by_provider=ollama only** for any customer-shape request. If you see by_provider including kimi/openrouter/opencode/ollama_cloud for normal /v1/iterate or /v1/respond traffic, something has regressed. Verify with: `curl -sf http://127.0.0.1:3100/v1/usage | jq .by_provider`. +- **`./scrum` is a TOOL, not architecture.** Lives at repo root. J runs it manually. Outputs findings to `data/_kb/scrum_findings.jsonl` (the KB). Findings inform development; they do NOT auto-fold into PRDs/design docs/code. If you find yourself proposing to "wire scrum into [some pipeline]" — stop. It's J's tool. +- **Test code in main is ACTIVELY being cleaned out.** 2026-05-03 commits `6aafd41` + `f4ebd22` removed 12 orphaned dev experiments (~2900 LOC) from `tests/real-world/` and `scripts/`. If you find more zero-reference experimental files, surface them — don't auto-delete unless the pattern is clearly a one-time-experiment-with-zero-consumers like the ones already removed. --- @@ -146,12 +357,24 @@ End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift --- +## ⚠ PRODUCTION-READY BLOCKER (2026-05-03) + +**Audit-trail capability is the gate to client signature.** Smoke + parity tests prove the surface compiles; they do NOT prove an audit response can be produced for a specific person. Staffing client won't sign without defensible discrimination-claim response capability. + +**Authoritative document:** `docs/AUDIT_TRAIL_PRD.md` — drafted 2026-05-03. Defines worked example (John Martinez at Warehouse B), the per-decision audit row schema, the surface map of where decisions happen today, current-state-vs-target gap table, and 9-phase implementation sequence. + +**Phase 1 (discovery walk) requires NO J approval — it's read-only.** Phases 2+ have explicit open questions in §10 of the PRD that need J's call before they can start. + +Until phase 9 exit criterion is met, **do not claim "production-ready" on customer-facing surfaces.** Internal substrate (Lance, sidecar drop, parity probes) is solid; subject-of-record audit story is not. + +--- + ## OPEN — but not blocking the demo | Item | What | When to act | |---|---|---| | `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. | -| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. | +| Open PRs #6, #7, #10 | All closed 2026-05-02 — superseded / empty / stale. PR #12 merged 2026-05-03 (`a5d9070`). PR #13 merged 2026-05-03 (`feb638e`). | Done. | | `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. | | `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. | | `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. | diff --git a/auditor/checks/inference.ts b/auditor/checks/inference.ts index 8103e9d..616a661 100644 --- a/auditor/checks/inference.ts +++ b/auditor/checks/inference.ts @@ -28,11 +28,16 @@ const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100"; // retries — verified with trivial 8-token probes). Swapped default to // deepseek-v3.1:671b which is proven working end-to-end through the // pr_audit mode runner during Phase 5 distillation acceptance testing. -// kimi-k2:1t can be re-selected via LH_AUDITOR_REVIEW_MODEL env when -// the upstream returns. Tie-breaker stays grok-4.1-fast (different -// vendor lineage so consensus + tie-break won't fail-correlate). -const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "deepseek-v3.1:671b"; -const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "x-ai/grok-4.1-fast"; +// +// 2026-09-02 update: Ollama Cloud subscription lapsed — every +// ollama_cloud model 402s ("requires a subscription"). J's call: sweep +// the auditor to OpenRouter. Same deepseek lineage via OpenRouter; +// tie-breaker moves to grok-4.3 because OpenRouter 404s grok-4.1-fast +// as deprecated. Both probed live through the gateway before landing. +// Vendor lineages stay distinct so consensus + tie-break won't +// fail-correlate. +const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "openrouter/deepseek/deepseek-chat-v3.1"; +const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "openrouter/x-ai/grok-4.3"; const N_CONSENSUS = Number(process.env.LH_AUDITOR_CONSENSUS_N ?? 3); const AUDIT_DISCREPANCIES_JSONL = "/home/profit/lakehouse/data/_kb/audit_discrepancies.jsonl"; // 40KB comfortably fits the consensus models' context windows diff --git a/auditor/checks/kimi_architect.ts b/auditor/checks/kimi_architect.ts index 7905066..aa08608 100644 --- a/auditor/checks/kimi_architect.ts +++ b/auditor/checks/kimi_architect.ts @@ -51,13 +51,15 @@ const CALL_TIMEOUT_MS = 900_000; const CACHE_TTL_MS = 24 * 60 * 60 * 1000; const MAX_DIFF_CHARS = 180_000; const MAX_PRIOR_FINDINGS = 50; -// Default provider/model = ollama_cloud/kimi-k2.6. Pre-2026-04-27 we -// went direct to api.kimi.com, but Ollama Cloud Pro now exposes the -// same model legitimately, so we route there to avoid User-Agent -// gating. The api.kimi.com path (provider=kimi) remains wired in the -// gateway as a fallback for when Ollama Cloud is upstream-broken. -const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "ollama_cloud"; -const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6"; +// Default provider/model = openrouter/moonshotai/kimi-k2.6. +// History: pre-2026-04-27 direct api.kimi.com (User-Agent gating), +// then ollama_cloud/kimi-k2.6 while the sub was active. 2026-09-02: +// Ollama Cloud subscription lapsed (402 on every cloud model), J's +// call was to sweep the auditor to OpenRouter — same Moonshot model, +// probed live through the gateway. The api.kimi.com path +// (provider=kimi) remains wired in the gateway as a fallback. +const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "openrouter"; +const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "moonshotai/kimi-k2.6"; // Cross-lineage alternation. 2026-04-27 J's call: Opus is too // expensive to auto-fire (~$0.30/audit). Kimi K2.6 via Go-sub is // effectively free; Haiku 4.5 via Zen is ~$0.04. Alternate between @@ -75,8 +77,10 @@ const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6"; // Override the alternation entirely with LH_AUDITOR_KIMI_MODEL // (forces one model regardless of audit count); set // LH_AUDITOR_KIMI_ALT_MODEL to the alternate. -const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "claude-haiku-4-5"; -const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "opencode"; +// 2026-09-02: opencode lane 401s (CreditsError, balance drained) — alt +// lane joins the OpenRouter sweep. Same Anthropic lineage, probed live. +const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "anthropic/claude-haiku-4.5"; +const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "openrouter"; const FORCE_DEFAULT = process.env.LH_AUDITOR_KIMI_MODEL !== undefined && process.env.LH_AUDITOR_KIMI_MODEL !== ""; function selectModel(diffLen: number, auditIndex: number = 0): { provider: string; model: string; promoted: boolean } { @@ -110,8 +114,8 @@ const MAX_TOKENS_OVERRIDE = Number(process.env.LH_AUDITOR_KIMI_MAX_TOKENS) || 0; function maxTokensFor(model: string): number { if (MAX_TOKENS_OVERRIDE > 0) return MAX_TOKENS_OVERRIDE; if (model.startsWith("claude-opus")) return 32_000; - if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet")) return 8_192; - if (model.startsWith("kimi-")) return 128_000; + if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet") || model.includes("/claude-haiku") || model.includes("/claude-sonnet")) return 8_192; + if (model.startsWith("kimi-") || model.includes("/kimi-")) return 128_000; if (model.startsWith("gpt-5") || model.startsWith("o1") || model.startsWith("o3") || model.startsWith("o4")) return 32_000; return 16_000; } diff --git a/auditor/gitea.ts b/auditor/gitea.ts index 2b20d57..2d897e8 100644 --- a/auditor/gitea.ts +++ b/auditor/gitea.ts @@ -9,28 +9,42 @@ import { readFile } from "node:fs/promises"; import type { PrSnapshot } from "./types.ts"; -const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev"; +// 2026-09-02: repo home moved to git.warren.help in the VPS +// consolidation; git.agentview.dev now dead-ends at a Traefik default +// cert + 404 (auditor had been failing listOpenPrs every cycle since +// Aug 25). Credentials come from ~/.git-credentials for HOST's domain, +// falling back to the user:pass embedded in this repo's origin remote. +// Basic auth works for both a PAT and a password on Gitea. +const HOST = process.env.GITEA_HOST ?? "https://git.warren.help"; const OWNER = "profit"; const REPO = "lakehouse"; const CRED_FILE = "/home/profit/.git-credentials"; +const GIT_CONFIG = "/home/profit/lakehouse/.git/config"; -let cachedPat: string | null = null; +let cachedAuth: string | null = null; -async function getPat(): Promise { - if (cachedPat) return cachedPat; - const raw = await readFile(CRED_FILE, "utf8"); - for (const line of raw.split("\n")) { - const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/); - if (m) { cachedPat = m[1]; return m[1]; } +async function getAuthHeader(): Promise { + if (cachedAuth) return cachedAuth; + const hostDomain = new URL(HOST).host.replace(/\./g, "\\."); + const credRe = new RegExp(`^https://([^:]+):([^@]+)@${hostDomain}`); + for (const file of [CRED_FILE, GIT_CONFIG]) { + let raw: string; + try { raw = await readFile(file, "utf8"); } catch { continue; } + for (const line of raw.split("\n")) { + const m = line.trim().replace(/^url = /, "").match(credRe); + if (m) { + cachedAuth = `Basic ${Buffer.from(`${decodeURIComponent(m[1])}:${decodeURIComponent(m[2])}`).toString("base64")}`; + return cachedAuth; + } + } } - throw new Error(`no Gitea PAT in ${CRED_FILE}`); + throw new Error(`no Gitea credentials for ${HOST} in ${CRED_FILE} or ${GIT_CONFIG}`); } async function giteaFetch(path: string, init: RequestInit = {}): Promise { - const pat = await getPat(); const url = `${HOST}/api/v1${path}`; const headers = new Headers(init.headers); - headers.set("Authorization", `token ${pat}`); + headers.set("Authorization", await getAuthHeader()); if (init.body && !headers.has("content-type")) { headers.set("content-type", "application/json"); } diff --git a/config/modes.toml b/config/modes.toml index bf7f159..f327f17 100644 --- a/config/modes.toml +++ b/config/modes.toml @@ -40,14 +40,13 @@ matrix_corpus = "chicago_permits_v1" name = "staffing_inference" # Staffing-domain native enrichment runner — Pass 4 (2026-04-26). # Same composer architecture as codereview_lakehouse but with staffing -# framing + workers corpus. Validates that the modes-as-prompt-molders -# pattern generalizes beyond code review. +# framing + workers corpus. preferred_mode = "staffing_inference_lakehouse" fallback_modes = ["ladder", "consensus", "pipeline"] -# 2026-04-28: gpt-oss-120b:free → kimi-k2.6 via Ollama Pro. Coding- -# specialized, faster than gpt-oss, on the same OLLAMA_CLOUD_KEY so -# no extra provider hop. -default_model = "kimi-k2.6" +# 2026-05-03: REVERTED to local. PRD line 70 — everything runs locally, +# no cloud APIs on the customer hot path. Cloud models stay available +# in providers.toml for explicit dev-tool opt-in (scrum, auditor). +default_model = "qwen3.5:latest" matrix_corpus = "workers_500k_v8" [[task_class]] @@ -61,9 +60,8 @@ matrix_corpus = "kb_team_runs_v1" name = "doc_drift_check" preferred_mode = "drift" fallback_modes = ["validator"] -# 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama Pro. -# Speed leader on factual checking, same OLLAMA_CLOUD_KEY. -default_model = "gemini-3-flash-preview" +# 2026-05-03: REVERTED to local per PRD line 70. +default_model = "qwen3.5:latest" matrix_corpus = "distilled_factual_v20260423095819" [[task_class]] diff --git a/crates/catalogd/Cargo.toml b/crates/catalogd/Cargo.toml index 1884d43..f436557 100644 --- a/crates/catalogd/Cargo.toml +++ b/crates/catalogd/Cargo.toml @@ -16,5 +16,8 @@ chrono = { workspace = true } uuid = { workspace = true } object_store = { workspace = true } arrow = { workspace = true } +parquet = { workspace = true } +sha2 = { workspace = true } +hmac = { workspace = true } proto = { path = "../proto" } tonic = { workspace = true } diff --git a/crates/catalogd/src/audit_endpoint.rs b/crates/catalogd/src/audit_endpoint.rs new file mode 100644 index 0000000..95945a5 --- /dev/null +++ b/crates/catalogd/src/audit_endpoint.rs @@ -0,0 +1,505 @@ +//! `/audit/subject/{id}` HTTP endpoint — legal-tier subject audit response. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §4 + §6. +//! +//! Responds to: `GET /subject/{candidate_id}?from=ISO&to=ISO` +//! Auth: requires `X-Lakehouse-Legal-Token` header matching the value in +//! the legal-token file (see LegalAuditAuth). Constant-time compare. +//! +//! Response shape (subject_audit_response.v1): +//! - manifest: full SubjectManifest from catalogd +//! - audit_log.rows: SubjectAuditRow entries within [from, to] +//! - audit_log.chain_verified: bool (HMAC chain verifies end-to-end) +//! - completeness_attestation: text the lawyer can quote +//! +//! V1 deliberately does NOT include: +//! - Ed25519 signature on the response — chain verification IS the +//! v1 attestation; signing is future hardening (spec §10 v2 reserve) +//! - Live row projection from the underlying datasets — caller can +//! fetch via /query/sql against the safe_views named in the manifest; +//! keeping projection out of this endpoint avoids depending on queryd +//! - External anchor / S3 Object Lock — local HMAC chain is the v1 +//! integrity surface +//! +//! These are documented as future hardening, not silent gaps. + +use crate::registry::Registry; +use crate::subject_audit::SubjectAuditWriter; +use axum::{ + extract::{Path, Query, State}, + http::{HeaderMap, StatusCode}, + response::IntoResponse, + routing::get, + Json, Router, +}; +use serde::{Deserialize, Serialize}; +use shared::types::{SubjectAuditRow, SubjectManifest}; +use std::sync::Arc; + +const LEGAL_TOKEN_HEADER: &str = "x-lakehouse-legal-token"; +const RESPONSE_SCHEMA: &str = "subject_audit_response.v1"; + +/// State for the audit endpoint router. Held outside Registry because +/// the audit writer + legal token aren't part of catalogd's data model +/// — they're sidecar concerns specific to this endpoint. +#[derive(Clone)] +pub struct AuditEndpointState { + pub registry: Registry, + pub writer: Arc, + /// Legal-tier token. Constant-time compared against the header on + /// every request. None = endpoint returns 503 (audit unavailable). + pub legal_token: Option>, +} + +impl AuditEndpointState { + /// Construct from existing pieces. `legal_token_path` is read at + /// construction and trimmed of trailing whitespace. Missing/empty + /// = endpoint will 503 every request with a clear reason. + pub async fn new( + registry: Registry, + writer: Arc, + legal_token_path: &std::path::Path, + ) -> Self { + let legal_token = match tokio::fs::read_to_string(legal_token_path).await { + Ok(s) => { + let trimmed = s.trim().to_string(); + if trimmed.is_empty() { + tracing::warn!( + "audit endpoint: legal token file at {} is empty — endpoint will 503", + legal_token_path.display() + ); + None + } else if trimmed.len() < 32 { + // 2026-05-03 kimi scrum BLOCK at audit_endpoint.rs:59-60: + // 16-byte minimum was too permissive for a legal-tier + // secret. Aligned to the SubjectAuditWriter signing-key + // minimum (32 bytes — HMAC-SHA256 best practice). + tracing::warn!( + "audit endpoint: legal token at {} is {} chars (minimum 32) — endpoint will 503", + legal_token_path.display(), trimmed.len() + ); + None + } else { + tracing::info!("audit endpoint: legal token loaded from {}", legal_token_path.display()); + Some(Arc::new(trimmed)) + } + } + Err(e) => { + tracing::warn!( + "audit endpoint: legal token unreadable from {} ({e}) — endpoint will 503", + legal_token_path.display() + ); + None + } + }; + Self { registry, writer, legal_token } + } +} + +pub fn router(state: AuditEndpointState) -> Router { + Router::new() + .route("/subject/{candidate_id}", get(audit_subject)) + .route("/health", get(audit_health)) + .with_state(state) +} + +async fn audit_health(State(state): State) -> impl IntoResponse { + if state.legal_token.is_some() { + (StatusCode::OK, "audit endpoint ready").into_response() + } else { + (StatusCode::SERVICE_UNAVAILABLE, "audit endpoint disabled (legal token missing)").into_response() + } +} + +#[derive(Deserialize)] +struct AuditQuery { + /// ISO-8601 start of audit window. None = unbounded back. + from: Option>, + /// ISO-8601 end of audit window. None = unbounded forward. + to: Option>, +} + +#[derive(Serialize)] +struct AuditWindowEcho { + from: Option>, + to: Option>, +} + +#[derive(Serialize)] +struct AuditLogSection { + rows_in_window: usize, + rows: Vec, + chain_root: String, + chain_verified: bool, + chain_rows_total: usize, + chain_verification_error: Option, +} + +#[derive(Serialize)] +struct AuditResponse { + schema: &'static str, + candidate_id: String, + generated_at: chrono::DateTime, + generated_by: &'static str, + request_window: AuditWindowEcho, + manifest: SubjectManifest, + audit_log: AuditLogSection, + /// Datasets the manifest says contain rows for this subject — caller + /// can issue follow-up queries against the named safe_views to get + /// the actual row projection. Keeping projection out of this endpoint + /// avoids depending on queryd from catalogd. + datasets_referenced: Vec, + safe_views_available: Vec, + completeness_attestation: String, +} + +#[derive(Serialize)] +struct DatasetRefEcho { + name: String, + key_column: String, + key_value: String, +} + +/// Constant-time string comparison to avoid timing leaks on the legal +/// token check. `subtle` would be the standard but introducing a dep +/// for one comparison is overkill — manual byte-eq with an unconditional +/// fold over both byte slices. +fn constant_time_eq(a: &[u8], b: &[u8]) -> bool { + if a.len() != b.len() { + return false; + } + let mut diff: u8 = 0; + for (x, y) in a.iter().zip(b.iter()) { + diff |= x ^ y; + } + diff == 0 +} + +/// Extract + verify the legal-tier token from the request. Returns +/// Err((status, body)) on any failure (missing header, missing token +/// configured, mismatch). Does NOT log token values. +fn require_legal_auth( + state: &AuditEndpointState, + headers: &HeaderMap, +) -> Result<(), (StatusCode, &'static str)> { + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + "audit endpoint disabled — no legal token configured on this gateway", + ))?; + let provided = headers.get(LEGAL_TOKEN_HEADER).ok_or(( + StatusCode::UNAUTHORIZED, + "missing X-Lakehouse-Legal-Token header", + ))?; + let provided = provided.to_str().map_err(|_| ( + StatusCode::UNAUTHORIZED, + "X-Lakehouse-Legal-Token contains non-ASCII characters", + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err((StatusCode::UNAUTHORIZED, "X-Lakehouse-Legal-Token mismatch")); + } + Ok(()) +} + +async fn audit_subject( + State(state): State, + Path(candidate_id): Path, + Query(q): Query, + headers: HeaderMap, +) -> impl IntoResponse { + if let Err((status, msg)) = require_legal_auth(&state, &headers) { + return (status, msg).into_response(); + } + if candidate_id.is_empty() { + return (StatusCode::BAD_REQUEST, "candidate_id is empty").into_response(); + } + + // 1. Load the subject manifest. + let manifest = match state.registry.get_subject(&candidate_id).await { + Some(m) => m, + None => return ( + StatusCode::NOT_FOUND, + format!("subject manifest not found for candidate_id={candidate_id}"), + ).into_response(), + }; + + // 2. Read audit rows in window. + let rows = match state.writer.read_rows_in_range(&candidate_id, q.from, q.to).await { + Ok(r) => r, + Err(e) => return ( + StatusCode::INTERNAL_SERVER_ERROR, + format!("audit log read failed: {e}"), + ).into_response(), + }; + + // 3. Verify chain end-to-end (full log, not just the windowed slice; + // chain integrity is global, not per-window). chain_root MUST + // come from the actual log tip — taking it from the windowed + // `rows` would lie when `to` is set or the window catches no + // rows. (2026-05-03 opus scrum WARN audit_endpoint.rs:206.) + let (chain_verified, chain_rows_total, chain_root, chain_err) = + match state.writer.verify_chain(&candidate_id).await { + Ok(n) => { + let tip = state.writer.chain_tip(&candidate_id).await + .unwrap_or_else(|| "GENESIS".into()); + (true, n, tip, None) + } + Err(e) => (false, 0, String::new(), Some(e)), + }; + + // 4. Tampering detection: if the manifest claims a non-trivial + // chain root but the log is empty (chain_rows_total == 0), the + // log was deleted between writes. Flag it. (2026-05-03 opus + // scrum WARN subject_audit.rs:347 + audit_endpoint coverage.) + let manifest_root = manifest.audit_log_chain_root.clone(); + let log_was_tampered = chain_verified + && chain_rows_total == 0 + && !manifest_root.is_empty() + && manifest_root != "GENESIS"; + let (chain_verified, chain_err) = if log_was_tampered { + (false, Some(format!( + "manifest expects chain root '{}' but the audit log has 0 rows; log was deleted or wiped", + manifest_root, + ))) + } else { + (chain_verified, chain_err) + }; + + let datasets_referenced = manifest.datasets.iter() + .map(|d| DatasetRefEcho { + name: d.name.clone(), + key_column: d.key_column.clone(), + key_value: d.key_value.clone(), + }) + .collect(); + let safe_views_available = manifest.safe_views.clone(); + + let attestation = format!( + "All audit rows for candidate_id={} within [{}, {}] are included. \ + HMAC chain verified end-to-end: {} rows total, integrity={}.", + candidate_id, + q.from.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()), + q.to.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()), + chain_rows_total, + if chain_verified { "verified" } else { "BROKEN" }, + ); + + let resp = AuditResponse { + schema: RESPONSE_SCHEMA, + candidate_id: candidate_id.clone(), + generated_at: chrono::Utc::now(), + generated_by: "catalogd", + request_window: AuditWindowEcho { from: q.from, to: q.to }, + manifest, + audit_log: AuditLogSection { + rows_in_window: rows.len(), + rows, + chain_root, + chain_verified, + chain_rows_total, + chain_verification_error: chain_err, + }, + datasets_referenced, + safe_views_available, + completeness_attestation: attestation, + }; + + // Log the legal-tier read so it's discoverable in operations review. + // Token value is NEVER logged. Only the fact of the read. + tracing::info!( + candidate_id = %candidate_id, + rows_in_window = resp.audit_log.rows_in_window, + chain_verified = resp.audit_log.chain_verified, + "legal-tier audit response served" + ); + + (StatusCode::OK, Json(resp)).into_response() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::subject_audit::SubjectAuditWriter; + use object_store::memory::InMemory; + use shared::types::{ + AuditAccessor, BiometricConsent, BiometricConsentStatus, ConsentStatus, + GeneralPiiConsent, SubjectAuditRow, SubjectConsent, SubjectRetention, + SubjectStatus, SubjectVertical, + }; + + fn fixture_manifest(candidate_id: &str) -> SubjectManifest { + SubjectManifest { + schema: "subject_manifest.v1".into(), + candidate_id: candidate_id.into(), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + status: SubjectStatus::Active, + vertical: SubjectVertical::Unknown, + consent: SubjectConsent { + general_pii: GeneralPiiConsent { + status: ConsentStatus::PendingBackfillReview, + version: String::new(), + given_at: None, + withdrawn_at: None, + }, + biometric: BiometricConsent { + status: BiometricConsentStatus::NeverCollected, + retention_until: None, + }, + }, + retention: SubjectRetention { + general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4), + policy: "test".into(), + }, + datasets: vec![], + safe_views: vec![], + audit_log_path: String::new(), + audit_log_chain_root: String::new(), + biometric_collection: None, + } + } + + fn fixture_audit_row(candidate_id: &str) -> SubjectAuditRow { + SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::Utc::now(), + candidate_id: candidate_id.into(), + accessor: AuditAccessor { + kind: "test".into(), + daemon: "test".into(), + purpose: "test".into(), + trace_id: String::new(), + }, + fields_accessed: vec!["name".into()], + result: "success".into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + } + } + + async fn fixture_state_with_token(token: &str) -> AuditEndpointState { + let store: Arc = Arc::new(InMemory::new()); + let registry = Registry::new(store.clone()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key( + store, (0u8..32).collect(), + )); + AuditEndpointState { + registry, + writer, + legal_token: Some(Arc::new(token.into())), + } + } + + #[test] + fn constant_time_eq_basic() { + assert!(constant_time_eq(b"abc", b"abc")); + assert!(!constant_time_eq(b"abc", b"abd")); + assert!(!constant_time_eq(b"abc", b"abcd")); + assert!(constant_time_eq(b"", b"")); + } + + #[tokio::test] + async fn missing_legal_token_returns_503() { + let state = AuditEndpointState { + registry: Registry::new(Arc::new(InMemory::new())), + writer: Arc::new(SubjectAuditWriter::with_inline_key( + Arc::new(InMemory::new()), (0u8..32).collect(), + )), + legal_token: None, + }; + let headers = HeaderMap::new(); + let r = require_legal_auth(&state, &headers); + assert!(matches!(r, Err((StatusCode::SERVICE_UNAVAILABLE, _)))); + } + + #[tokio::test] + async fn missing_header_returns_401() { + let state = fixture_state_with_token("test-token-must-be-long-enough").await; + let headers = HeaderMap::new(); + let r = require_legal_auth(&state, &headers); + assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _)))); + } + + #[tokio::test] + async fn wrong_token_returns_401() { + let state = fixture_state_with_token("correct-token-must-be-long-enough").await; + let mut headers = HeaderMap::new(); + headers.insert(LEGAL_TOKEN_HEADER, "wrong".parse().unwrap()); + let r = require_legal_auth(&state, &headers); + assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _)))); + } + + #[tokio::test] + async fn correct_token_passes_auth() { + let state = fixture_state_with_token("correct-token-must-be-long-enough").await; + let mut headers = HeaderMap::new(); + headers.insert(LEGAL_TOKEN_HEADER, "correct-token-must-be-long-enough".parse().unwrap()); + let r = require_legal_auth(&state, &headers); + assert!(r.is_ok()); + } + + #[tokio::test] + async fn audit_response_assembly_full_path() { + let state = fixture_state_with_token("test-token-must-be-long-enough").await; + // Plant a manifest + 3 audit rows. + state.registry.put_subject(fixture_manifest("CAND-AUDIT-1")).await.unwrap(); + for _ in 0..3 { + state.writer.append(fixture_audit_row("CAND-AUDIT-1")).await.unwrap(); + } + // Read rows in unbounded window. + let rows = state.writer + .read_rows_in_range("CAND-AUDIT-1", None, None) + .await + .unwrap(); + assert_eq!(rows.len(), 3); + // Verify chain integrity. + let n = state.writer.verify_chain("CAND-AUDIT-1").await.unwrap(); + assert_eq!(n, 3); + } + + #[tokio::test] + async fn audit_response_window_filters_rows() { + let state = fixture_state_with_token("test-token-must-be-long-enough").await; + state.registry.put_subject(fixture_manifest("CAND-WIN-1")).await.unwrap(); + // Append 5 rows. + let mut row_times = Vec::new(); + for _ in 0..5 { + let row = fixture_audit_row("CAND-WIN-1"); + row_times.push(row.ts); + state.writer.append(row).await.unwrap(); + tokio::time::sleep(std::time::Duration::from_millis(2)).await; + } + // Window covering only the middle 3 rows. + let from = row_times[1]; + let to = row_times[3]; + let rows = state.writer + .read_rows_in_range("CAND-WIN-1", Some(from), Some(to)) + .await + .unwrap(); + assert!(rows.len() >= 3, "expected ≥3 rows in window; got {}", rows.len()); + assert!(rows.len() <= 3, "expected ≤3 rows in window; got {}", rows.len()); + } + + #[tokio::test] + async fn empty_token_file_results_in_disabled_endpoint() { + let tmp = std::env::temp_dir().join(format!("legal_empty_{}.tok", std::process::id())); + tokio::fs::write(&tmp, "").await.unwrap(); + let store: Arc = Arc::new(InMemory::new()); + let registry = Registry::new(store.clone()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect())); + let state = AuditEndpointState::new(registry, writer, &tmp).await; + assert!(state.legal_token.is_none()); + let _ = tokio::fs::remove_file(&tmp).await; + } + + #[tokio::test] + async fn short_token_file_rejected_at_load() { + let tmp = std::env::temp_dir().join(format!("legal_short_{}.tok", std::process::id())); + tokio::fs::write(&tmp, "tooshort").await.unwrap(); + let store: Arc = Arc::new(InMemory::new()); + let registry = Registry::new(store.clone()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect())); + let state = AuditEndpointState::new(registry, writer, &tmp).await; + assert!(state.legal_token.is_none()); + let _ = tokio::fs::remove_file(&tmp).await; + } +} diff --git a/crates/catalogd/src/bin/backfill_subjects.rs b/crates/catalogd/src/bin/backfill_subjects.rs new file mode 100644 index 0000000..a6723ca --- /dev/null +++ b/crates/catalogd/src/bin/backfill_subjects.rs @@ -0,0 +1,341 @@ +//! Backfill subject manifests from a parquet source. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3. +//! +//! For each row in the source parquet, creates a SubjectManifest with +//! the BIPA-defensible defaults from the spec: +//! - vertical = unknown (HIPAA fail-closed routing) +//! - consent.general_pii = pending_backfill_review (NOT inferred_existing) +//! - consent.biometric = never_collected (no biometric backfill) +//! - retention.general_pii_until = now + 4 years (default policy) +//! +//! Idempotent: if a manifest already exists for a candidate_id, the +//! backfill skips it. Re-runs are safe. +//! +//! Conservative defaults: --limit 1000 by default. Pass --all to do +//! the full source. Pass --concurrency to bound parallel writes. +//! +//! Usage examples: +//! cargo run --bin backfill_subjects -- \ +//! --source data/datasets/workers_500k.parquet \ +//! --dataset workers_500k \ +//! --key-column worker_id \ +//! --candidate-id-prefix WORKER- \ +//! --limit 100 \ +//! --dry-run +//! +//! cargo run --bin backfill_subjects -- \ +//! --source data/datasets/workers_500k.parquet \ +//! --dataset workers_500k \ +//! --key-column worker_id \ +//! --candidate-id-prefix WORKER- \ +//! --safe-view workers_safe \ +//! --all +//! +//! On the live host the binary reads from / writes to ./data so it +//! integrates with the running catalogd's persistence path. + +use catalogd::registry::Registry; +use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use shared::types::{ + BiometricConsent, BiometricConsentStatus, ConsentStatus, GeneralPiiConsent, + SubjectConsent, SubjectDatasetRef, SubjectManifest, SubjectRetention, + SubjectStatus, SubjectVertical, +}; +use std::fs::File; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use storaged::backend; + +const DEFAULT_LIMIT: usize = 1000; +const DEFAULT_CONCURRENCY: usize = 32; +const DEFAULT_RETENTION_YEARS: i64 = 4; + +#[derive(Debug)] +struct Args { + source: PathBuf, + dataset_name: String, + key_column: String, + candidate_id_prefix: String, + safe_view: Option, + storage_root: PathBuf, + limit: Option, + concurrency: usize, + dry_run: bool, +} + +fn parse_args() -> Result { + let argv: Vec = std::env::args().collect(); + let mut source: Option = None; + let mut dataset_name: Option = None; + let mut key_column: Option = None; + let mut candidate_id_prefix: String = String::new(); + let mut safe_view: Option = None; + let mut storage_root: PathBuf = PathBuf::from("./data"); + let mut limit: Option = Some(DEFAULT_LIMIT); + let mut concurrency: usize = DEFAULT_CONCURRENCY; + let mut dry_run = false; + + let mut i = 1; + while i < argv.len() { + let arg = &argv[i]; + let next = || -> Result { + argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value")) + }; + match arg.as_str() { + "--source" => { source = Some(PathBuf::from(next()?)); i += 2; } + "--dataset" => { dataset_name = Some(next()?); i += 2; } + "--key-column" => { key_column = Some(next()?); i += 2; } + "--candidate-id-prefix" => { candidate_id_prefix = next()?; i += 2; } + "--safe-view" => { safe_view = Some(next()?); i += 2; } + "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; } + "--limit" => { limit = Some(next()?.parse().map_err(|e| format!("--limit: {e}"))?); i += 2; } + "--all" => { limit = None; i += 1; } + "--concurrency" => { concurrency = next()?.parse().map_err(|e| format!("--concurrency: {e}"))?; i += 2; } + "--dry-run" => { dry_run = true; i += 1; } + "-h" | "--help" => { print_help(); std::process::exit(0); } + other => return Err(format!("unknown arg: {other}")), + } + } + Ok(Args { + source: source.ok_or("missing --source")?, + dataset_name: dataset_name.ok_or("missing --dataset")?, + key_column: key_column.ok_or("missing --key-column")?, + candidate_id_prefix, + safe_view, + storage_root, + limit, + concurrency, + dry_run, + }) +} + +fn print_help() { + eprintln!("backfill_subjects — populate subject manifests from a parquet source\n"); + eprintln!("Usage: backfill_subjects [flags]\n"); + eprintln!("Required:"); + eprintln!(" --source PATH parquet file containing subject identifiers"); + eprintln!(" --dataset NAME catalogd dataset name (must already exist)"); + eprintln!(" --key-column COL column in the parquet that holds subject identifiers\n"); + eprintln!("Optional:"); + eprintln!(" --candidate-id-prefix STR prefix added to each row's value to form candidate_id"); + eprintln!(" (e.g. 'WORKER-' on a worker_id int64 column)"); + eprintln!(" --safe-view NAME name of an AiView that safely projects this subject's data"); + eprintln!(" --storage-root PATH local-filesystem catalog root (default: ./data)"); + eprintln!(" --limit N process at most N rows (default: {DEFAULT_LIMIT})"); + eprintln!(" --all process every row in the source"); + eprintln!(" --concurrency N max concurrent writes (default: {DEFAULT_CONCURRENCY})"); + eprintln!(" --dry-run parse + count + sample, don't write anything"); +} + +#[tokio::main(flavor = "multi_thread", worker_threads = 4)] +async fn main() { + let args = match parse_args() { + Ok(a) => a, + Err(e) => { + eprintln!("error: {e}"); + print_help(); + std::process::exit(2); + } + }; + + if let Err(e) = run(args).await { + eprintln!("backfill failed: {e}"); + std::process::exit(1); + } +} + +async fn run(args: Args) -> Result<(), String> { + eprintln!("[backfill] source: {}", args.source.display()); + eprintln!("[backfill] dataset: {}", args.dataset_name); + eprintln!("[backfill] key: {} (prefix: {:?})", args.key_column, args.candidate_id_prefix); + eprintln!("[backfill] limit: {}", args.limit.map(|n| n.to_string()).unwrap_or("all".into())); + eprintln!("[backfill] dry-run: {}", args.dry_run); + + // Stand up catalogd over the same local FS the running daemon uses. + let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?); + let reg = Arc::new(Registry::new(store)); + reg.rebuild().await?; + + if reg.get_by_name(&args.dataset_name).await.is_none() { + return Err(format!( + "dataset '{}' not found in catalog at {}; backfill needs the dataset registered first", + args.dataset_name, args.storage_root.display(), + )); + } + eprintln!("[backfill] dataset '{}' exists in catalog ✓", args.dataset_name); + + // Open parquet reader. + let file = File::open(&args.source).map_err(|e| format!("open {}: {e}", args.source.display()))?; + let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(|e| e.to_string())?; + let total_rows = builder.metadata().file_metadata().num_rows() as usize; + eprintln!("[backfill] parquet rows: {total_rows}"); + + // Project only the key column for IO efficiency. + let schema = builder.schema().clone(); + let col_idx = schema.index_of(&args.key_column) + .map_err(|_| format!("column '{}' not found; parquet has: {:?}", + args.key_column, + schema.fields().iter().map(|f| f.name()).collect::>(), + ))?; + let projection = parquet::arrow::ProjectionMask::roots(builder.parquet_schema(), [col_idx]); + let mut reader = builder.with_projection(projection).build().map_err(|e| e.to_string())?; + + // Collect candidate IDs (limited). + let cap = args.limit.unwrap_or(total_rows); + let mut candidate_ids: Vec = Vec::with_capacity(cap.min(total_rows)); + let mut rows_seen = 0usize; + 'outer: while let Some(batch_result) = reader.next() { + let batch = batch_result.map_err(|e| format!("batch: {e}"))?; + let column = batch.column(0); // single projected column + for row in 0..batch.num_rows() { + if let Some(limit) = args.limit { + if candidate_ids.len() >= limit { break 'outer; } + } + rows_seen += 1; + let raw = format_cell_value(column, row)?; + candidate_ids.push(format!("{}{}", args.candidate_id_prefix, raw)); + } + } + eprintln!("[backfill] collected {} candidate_ids ({} rows scanned)", candidate_ids.len(), rows_seen); + + // Sample for the operator. + eprintln!("[backfill] sample (first 5):"); + for cid in candidate_ids.iter().take(5) { + eprintln!(" - {cid}"); + } + + if args.dry_run { + eprintln!("[backfill] DRY-RUN — no writes. Re-run without --dry-run to commit."); + return Ok(()); + } + + // Write manifests with bounded concurrency. + let now = chrono::Utc::now(); + let retention_until = now + chrono::Duration::days(365 * DEFAULT_RETENTION_YEARS); + let safe_views: Vec = args.safe_view.into_iter().collect(); + let dataset_ref = SubjectDatasetRef { + name: args.dataset_name.clone(), + key_column: args.key_column.clone(), + key_value: String::new(), // filled per-subject below + }; + + let attempted = Arc::new(AtomicUsize::new(0)); + let inserted = Arc::new(AtomicUsize::new(0)); + let skipped = Arc::new(AtomicUsize::new(0)); + let failed = Arc::new(AtomicUsize::new(0)); + let progress_every = ((candidate_ids.len() / 20).max(1)).min(5_000); + + // Bounded concurrency via Semaphore. + let sem = Arc::new(tokio::sync::Semaphore::new(args.concurrency)); + let mut handles = Vec::with_capacity(candidate_ids.len()); + for (idx, cid) in candidate_ids.into_iter().enumerate() { + let permit = sem.clone().acquire_owned().await.map_err(|e| e.to_string())?; + let reg = reg.clone(); + let mut ds_ref = dataset_ref.clone(); + // strip_prefix gives one-shot semantics; trim_start_matches strips + // repeated occurrences (a "WORKER-WORKER-1" id would lose both + // prefixes). 2026-05-03 opus scrum WARN at backfill_subjects.rs:189. + ds_ref.key_value = cid.strip_prefix(&args.candidate_id_prefix) + .unwrap_or(&cid).to_string(); + let safe_views = safe_views.clone(); + let attempted = attempted.clone(); + let inserted = inserted.clone(); + let skipped = skipped.clone(); + let failed = failed.clone(); + + let handle = tokio::spawn(async move { + let _permit = permit; + attempted.fetch_add(1, Ordering::Relaxed); + // Idempotency: skip if subject already exists. + if reg.get_subject(&cid).await.is_some() { + skipped.fetch_add(1, Ordering::Relaxed); + return; + } + let manifest = SubjectManifest { + schema: "subject_manifest.v1".into(), + candidate_id: cid.clone(), + created_at: now, + updated_at: now, + status: SubjectStatus::Active, + vertical: SubjectVertical::Unknown, + consent: SubjectConsent { + general_pii: GeneralPiiConsent { + status: ConsentStatus::PendingBackfillReview, + version: String::new(), + given_at: None, + withdrawn_at: None, + }, + biometric: BiometricConsent { + status: BiometricConsentStatus::NeverCollected, + retention_until: None, + }, + }, + retention: SubjectRetention { + general_pii_until: retention_until, + policy: "4_year_default".into(), + }, + datasets: vec![ds_ref], + safe_views, + audit_log_path: String::new(), + audit_log_chain_root: String::new(), + biometric_collection: None, + }; + match reg.put_subject(manifest).await { + Ok(_) => { inserted.fetch_add(1, Ordering::Relaxed); } + Err(e) => { + failed.fetch_add(1, Ordering::Relaxed); + eprintln!(" ✗ {cid}: {e}"); + } + } + let cur = attempted.load(Ordering::Relaxed); + if cur % progress_every == 0 { + eprintln!("[backfill] progress: {} attempted ({} inserted, {} skipped, {} failed)", + cur, inserted.load(Ordering::Relaxed), + skipped.load(Ordering::Relaxed), + failed.load(Ordering::Relaxed)); + } + let _ = idx; + }); + handles.push(handle); + } + for h in handles { + let _ = h.await; + } + + eprintln!(); + eprintln!("──────────────────────────────────────────────────────────────────"); + eprintln!("[backfill] done"); + eprintln!("[backfill] attempted: {}", attempted.load(Ordering::Relaxed)); + eprintln!("[backfill] inserted: {}", inserted.load(Ordering::Relaxed)); + eprintln!("[backfill] skipped: {} (already existed — idempotent re-run)", skipped.load(Ordering::Relaxed)); + eprintln!("[backfill] failed: {}", failed.load(Ordering::Relaxed)); + eprintln!("[backfill] subjects in catalog now: {}", reg.subjects_count().await); + Ok(()) +} + +/// Format a single cell value as a string for use in the candidate_id. +/// Supports the integer + string types most likely to be subject keys. +fn format_cell_value(array: &Arc, row: usize) -> Result { + use arrow::array::{Int32Array, Int64Array, StringArray, UInt32Array, UInt64Array}; + use arrow::datatypes::DataType; + if array.is_null(row) { + return Err(format!("row {row}: key column is null — cannot form candidate_id")); + } + let dt = array.data_type(); + let s = match dt { + DataType::Int64 => array.as_any().downcast_ref::() + .ok_or("int64 downcast")?.value(row).to_string(), + DataType::Int32 => array.as_any().downcast_ref::() + .ok_or("int32 downcast")?.value(row).to_string(), + DataType::UInt64 => array.as_any().downcast_ref::() + .ok_or("uint64 downcast")?.value(row).to_string(), + DataType::UInt32 => array.as_any().downcast_ref::() + .ok_or("uint32 downcast")?.value(row).to_string(), + DataType::Utf8 => array.as_any().downcast_ref::() + .ok_or("utf8 downcast")?.value(row).to_string(), + other => return Err(format!("unsupported key column type: {other:?}")), + }; + Ok(s) +} diff --git a/crates/catalogd/src/bin/parity_subject_audit.rs b/crates/catalogd/src/bin/parity_subject_audit.rs new file mode 100644 index 0000000..f0d8afe --- /dev/null +++ b/crates/catalogd/src/bin/parity_subject_audit.rs @@ -0,0 +1,235 @@ +//! Cross-runtime parity helper for subject-audit chain. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8. +//! +//! This binary is consumed by the Go-repo's +//! scripts/cutover/parity/subject_audit_parity.sh. +//! Go counterpart: scripts/cutover/parity/subject_audit_helper/main.go +//! (in the Go repo). Both helpers are kept in lockstep by the parity +//! script's byte-equality assertions — change one, change both. +//! +//! Both helpers MUST produce byte-identical output for the same inputs. +//! Divergence here is a parity break — a SubjectManifest written by Rust +//! that Go cannot verify, or vice versa. +//! +//! Two modes: +//! +//! --known-answer +//! Print the canonical bytes + HMAC of a hardcoded fixture row. +//! The Go helper must produce IDENTICAL bytes + IDENTICAL hash. +//! Hardcoded fixture matches Go test TestKnownAnswerVector. +//! +//! --verify --key +//! Parse the JSONL audit log, replay the HMAC chain. Print +//! JSON: {"count": N, "tip": "", "verified": bool, "error": ""}. +//! +//! Output format: ONE JSON object per stdout, terminating newline. The +//! parity script diffs Rust stdout vs Go stdout via `diff -q`. + +use catalogd::subject_audit::SubjectAuditWriter; +use hmac::{Hmac, Mac}; +use object_store::ObjectStore; +use object_store::memory::InMemory; +use serde::Serialize; +use sha2::Sha256; +use shared::types::{AuditAccessor, SubjectAuditRow}; +use std::collections::BTreeMap; +use std::path::PathBuf; +use std::sync::Arc; + +type HmacSha256 = Hmac; + +const GENESIS: &str = "GENESIS"; + +/// Same canonical-JSON algorithm as crates/catalogd/src/subject_audit.rs. +/// Reproduced here so the helper does not depend on internal-only items +/// in that file. If the algorithm there changes, change it here AND in +/// the Go helper in lockstep. +fn canonical_json(v: &serde_json::Value) -> Vec { + fn rewrite(v: &serde_json::Value) -> serde_json::Value { + match v { + serde_json::Value::Object(map) => { + let sorted: BTreeMap = map + .iter() + .map(|(k, v)| (k.clone(), rewrite(v))) + .collect(); + serde_json::Value::Object(sorted.into_iter().collect()) + } + serde_json::Value::Array(arr) => { + serde_json::Value::Array(arr.iter().map(rewrite).collect()) + } + other => other.clone(), + } + } + serde_json::to_vec(&rewrite(v)).expect("canonical-json serialize") +} + +fn canonical_row_bytes(row: &SubjectAuditRow) -> Vec { + let mut v: serde_json::Value = serde_json::to_value(row).expect("row to value"); + if let Some(obj) = v.as_object_mut() { + obj.remove("row_hmac"); + } + canonical_json(&v) +} + +fn compute_hmac(key: &[u8], prev: &str, canonical: &[u8]) -> String { + let mut mac = ::new_from_slice(key).expect("HMAC accepts any key length"); + mac.update(prev.as_bytes()); + mac.update(canonical); + let result = mac.finalize().into_bytes(); + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut s = String::with_capacity(64); + for byte in result { + s.push(HEX[(byte >> 4) as usize] as char); + s.push(HEX[(byte & 0x0f) as usize] as char); + } + s +} + +fn deterministic_key() -> Vec { + (0u8..32).collect() +} + +/// Print to stderr + exit(2). Never returns. Used in place of `.expect()` +/// on argv / file reads so failures produce a clean usage error rather +/// than a Rust panic backtrace that the parity script would mistake +/// for a parity break. +fn die(msg: &str) -> ! { + eprintln!("error: {msg}"); + std::process::exit(2); +} + +#[derive(Serialize)] +struct KnownAnswerOut { + mode: &'static str, + canonical: String, + hmac: String, + canonical_bytes_len: usize, +} + +#[derive(Serialize)] +struct VerifyOut { + mode: &'static str, + count: usize, + tip: String, + verified: bool, + error: Option, +} + +fn known_answer() { + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::DateTime::parse_from_rfc3339("2026-05-03T12:00:00Z") + .unwrap() + .with_timezone(&chrono::Utc), + candidate_id: "WORKER-FIXED".into(), + accessor: AuditAccessor { + kind: "gateway_lookup".into(), + daemon: "gateway".into(), + purpose: "parity_test".into(), + trace_id: "trace-fixed".into(), + }, + fields_accessed: vec!["name".into()], + result: "success".into(), + prev_chain_hash: GENESIS.into(), + row_hmac: String::new(), + }; + let canon = canonical_row_bytes(&row); + let hmac = compute_hmac(&deterministic_key(), GENESIS, &canon); + let out = KnownAnswerOut { + mode: "known_answer", + canonical: String::from_utf8(canon.clone()).expect("canonical is utf-8"), + hmac, + canonical_bytes_len: canon.len(), + }; + println!("{}", serde_json::to_string(&out).unwrap()); +} + +#[tokio::main(flavor = "current_thread")] +async fn main() { + let argv: Vec = std::env::args().collect(); + let mut mode_known_answer = false; + let mut audit_path: Option = None; + let mut key_path: Option = None; + + // Argv parsing fails via stderr+exit, never via panic. The parity + // script captures stdout for byte-comparison; a Rust panic backtrace + // would land in stdout (the script merges 2>&1) and read as a parity + // break instead of a usage error. (Caught 2026-05-03 opus scrum WARN.) + let mut i = 1; + while i < argv.len() { + match argv[i].as_str() { + "--known-answer" => { + mode_known_answer = true; + i += 1; + } + "--verify" => { + let p = argv.get(i + 1).unwrap_or_else(|| die("--verify needs a path")); + audit_path = Some(PathBuf::from(p)); + i += 2; + } + "--key" => { + let p = argv.get(i + 1).unwrap_or_else(|| die("--key needs a path")); + key_path = Some(PathBuf::from(p)); + i += 2; + } + "-h" | "--help" => { + eprintln!("parity_subject_audit --known-answer"); + eprintln!("parity_subject_audit --verify --key "); + std::process::exit(0); + } + other => { + eprintln!("unknown arg: {other}"); + std::process::exit(2); + } + } + } + + if mode_known_answer { + known_answer(); + return; + } + + let audit_path = audit_path.unwrap_or_else(|| die("need --known-answer OR --verify --key ")); + let key_path = key_path.unwrap_or_else(|| die("--verify also needs --key")); + + let key = std::fs::read(&key_path).unwrap_or_else(|e| die(&format!("read key file: {e}"))); + let candidate_id = audit_path + .file_name() + .and_then(|s| s.to_str()) + .and_then(|s| s.strip_suffix(".audit.jsonl")) + .unwrap_or_else(|| die("audit log path must end with .audit.jsonl")) + .to_string(); + + // Stand up an in-memory object store, seed it with the audit log + // bytes at the canonical key, then ask SubjectAuditWriter to verify. + // This way we exercise the SAME verify_chain function the production + // gateway calls — not a re-implementation that might drift. + let store: Arc = Arc::new(InMemory::new()); + let bytes = std::fs::read(&audit_path).expect("read audit log"); + let log_key = format!("_catalog/subjects/{}.audit.jsonl", candidate_id); + storaged::ops::put(&store, &log_key, bytes::Bytes::from(bytes)) + .await + .expect("seed object store"); + + let writer = SubjectAuditWriter::with_inline_key(store, key); + let result = writer.verify_chain(&candidate_id).await; + let tip = writer.chain_tip(&candidate_id).await.unwrap_or(GENESIS.into()); + let out = match result { + Ok(count) => VerifyOut { + mode: "verify", + count, + tip, + verified: true, + error: None, + }, + Err(e) => VerifyOut { + mode: "verify", + count: 0, + tip: GENESIS.into(), + verified: false, + error: Some(e), + }, + }; + println!("{}", serde_json::to_string(&out).unwrap()); +} diff --git a/crates/catalogd/src/bin/retention_sweep.rs b/crates/catalogd/src/bin/retention_sweep.rs new file mode 100644 index 0000000..0ef3c90 --- /dev/null +++ b/crates/catalogd/src/bin/retention_sweep.rs @@ -0,0 +1,376 @@ +//! Daily retention sweep for subject manifests. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7: +//! "Subjects whose retention.general_pii_until < now AND status != erased +//! get marked for review (don't auto-delete; legal needs to approve)." +//! +//! Per shared::types::BiometricConsent doc-comment (BIPA requirement): +//! "max 3 years from last interaction. Implementation MUST enforce +//! daily expiration sweep against this field." +//! +//! Therefore this binary checks BOTH retention clocks and reports any +//! subject whose general-pii window OR biometric window has expired +//! while the subject is not already erased / already-flagged. +//! +//! It does NOT mutate subject manifests. It produces a JSONL review +//! queue at: +//! data/_catalog/subjects/_retention_sweep_.jsonl +//! +//! Operators / legal review the queue and act: +//! - extend retention (legal contract update), +//! - flip status to retention_expired, +//! - schedule erasure. +//! +//! Usage: +//! cargo run --bin retention_sweep # dry-run, stderr only +//! cargo run --bin retention_sweep -- --apply # also write report file +//! cargo run --bin retention_sweep -- --as-of 2030-01-01T00:00:00Z +//! cargo run --bin retention_sweep -- --storage-root /var/lib/lakehouse/data --apply + +use catalogd::registry::Registry; +use chrono::{DateTime, Utc}; +use serde::Serialize; +use shared::types::{BiometricConsentStatus, SubjectManifest, SubjectStatus}; +use std::path::PathBuf; +use std::sync::Arc; +use storaged::backend; + +#[derive(Debug)] +struct Args { + storage_root: PathBuf, + apply: bool, + as_of: Option>, +} + +fn parse_args() -> Result { + let argv: Vec = std::env::args().collect(); + let mut storage_root = PathBuf::from("./data"); + let mut apply = false; + let mut as_of: Option> = None; + + let mut i = 1; + while i < argv.len() { + let arg = &argv[i]; + let next = || -> Result { + argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value")) + }; + match arg.as_str() { + "--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; } + "--apply" => { apply = true; i += 1; } + "--as-of" => { + let raw = next()?; + as_of = Some(raw.parse().map_err(|e| format!("--as-of (need RFC3339): {e}"))?); + i += 2; + } + "-h" | "--help" => { print_help(); std::process::exit(0); } + other => return Err(format!("unknown arg: {other}")), + } + } + Ok(Args { storage_root, apply, as_of }) +} + +fn print_help() { + eprintln!("retention_sweep — daily retention check on subject manifests"); + eprintln!(); + eprintln!("Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7."); + eprintln!("Reports overdue subjects to a JSONL review queue. Does NOT"); + eprintln!("auto-delete. Legal must approve every action."); + eprintln!(); + eprintln!("Flags:"); + eprintln!(" --storage-root PATH local-filesystem catalog root (default: ./data)"); + eprintln!(" --apply write report to data/_catalog/subjects/_retention_sweep_.jsonl"); + eprintln!(" --as-of RFC3339 use this timestamp as 'now' (test/forecast helper)"); +} + +#[derive(Debug, PartialEq, Clone, Copy)] +enum OverdueReason { + GeneralPii, + Biometric, + Both, +} + +impl OverdueReason { + fn as_str(self) -> &'static str { + match self { + OverdueReason::GeneralPii => "general_pii_expired", + OverdueReason::Biometric => "biometric_expired", + OverdueReason::Both => "both_expired", + } + } +} + +#[derive(Debug, Serialize)] +struct OverdueRow<'a> { + candidate_id: &'a str, + status: &'a str, + general_pii_until: DateTime, + days_overdue_general: i64, + biometric_status: &'a str, + #[serde(skip_serializing_if = "Option::is_none")] + biometric_retention_until: Option>, + #[serde(skip_serializing_if = "Option::is_none")] + days_overdue_biometric: Option, + reason: &'a str, + swept_at: DateTime, +} + +/// Skipped statuses (idempotent across daily runs): +/// - Erased: subject's data is already gone; nothing to review. +/// - RetentionExpired: subject is already flagged; do not re-fire. +/// +/// Actionable statuses: PendingConsent, Active, Withdrawn — each can +/// still hold PII rows on disk past the retention window. +fn is_overdue(subject: &SubjectManifest, now: DateTime) -> Option { + if matches!(subject.status, SubjectStatus::Erased | SubjectStatus::RetentionExpired) { + return None; + } + let general_overdue = subject.retention.general_pii_until < now; + let biometric_overdue = subject + .consent + .biometric + .retention_until + .map(|t| t < now) + .unwrap_or(false); + match (general_overdue, biometric_overdue) { + (true, true) => Some(OverdueReason::Both), + (true, false) => Some(OverdueReason::GeneralPii), + (false, true) => Some(OverdueReason::Biometric), + (false, false) => None, + } +} + +fn status_str(s: &SubjectStatus) -> &'static str { + match s { + SubjectStatus::PendingConsent => "pending_consent", + SubjectStatus::Active => "active", + SubjectStatus::Withdrawn => "withdrawn", + SubjectStatus::RetentionExpired => "retention_expired", + SubjectStatus::Erased => "erased", + } +} + +fn biometric_status_str(s: &BiometricConsentStatus) -> &'static str { + match s { + BiometricConsentStatus::NeverCollected => "never_collected", + BiometricConsentStatus::Pending => "pending", + BiometricConsentStatus::Given => "given", + BiometricConsentStatus::Withdrawn => "withdrawn", + BiometricConsentStatus::Expired => "expired", + } +} + +#[tokio::main(flavor = "multi_thread", worker_threads = 2)] +async fn main() { + let args = match parse_args() { + Ok(a) => a, + Err(e) => { + eprintln!("error: {e}"); + print_help(); + std::process::exit(2); + } + }; + if let Err(e) = run(args).await { + eprintln!("retention_sweep failed: {e}"); + std::process::exit(1); + } +} + +async fn run(args: Args) -> Result<(), String> { + let now = args.as_of.unwrap_or_else(Utc::now); + eprintln!("[retention_sweep] storage-root: {}", args.storage_root.display()); + eprintln!("[retention_sweep] now (sweep clock): {}", now.to_rfc3339()); + eprintln!("[retention_sweep] apply: {}", args.apply); + + let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?); + let reg = Arc::new(Registry::new(store)); + reg.rebuild().await?; + + let subjects = reg.list_subjects().await; + eprintln!("[retention_sweep] subjects loaded: {}", subjects.len()); + + let mut overdue_rows: Vec = Vec::new(); + let mut by_reason: std::collections::BTreeMap<&'static str, usize> = Default::default(); + + for s in &subjects { + if let Some(reason) = is_overdue(s, now) { + let reason_str: &'static str = reason.as_str(); + *by_reason.entry(reason_str).or_insert(0) += 1; + overdue_rows.push(OverdueRow { + candidate_id: &s.candidate_id, + status: status_str(&s.status), + general_pii_until: s.retention.general_pii_until, + days_overdue_general: (now - s.retention.general_pii_until).num_days(), + biometric_status: biometric_status_str(&s.consent.biometric.status), + biometric_retention_until: s.consent.biometric.retention_until, + days_overdue_biometric: s + .consent + .biometric + .retention_until + .map(|t| (now - t).num_days()), + reason: reason_str, + swept_at: now, + }); + } + } + + eprintln!("[retention_sweep] overdue subjects: {}", overdue_rows.len()); + for (reason, n) in &by_reason { + eprintln!("[retention_sweep] {reason}: {n}"); + } + + if overdue_rows.is_empty() { + eprintln!("[retention_sweep] nothing to do; all subjects within retention window."); + return Ok(()); + } + + eprintln!("[retention_sweep] sample (first 5):"); + for row in overdue_rows.iter().take(5) { + eprintln!( + " - {} status={} reason={} general_overdue_days={}", + row.candidate_id, row.status, row.reason, row.days_overdue_general + ); + } + + if !args.apply { + eprintln!(); + eprintln!("[retention_sweep] DRY-RUN: no file written. Re-run with --apply to persist."); + return Ok(()); + } + + let date_stamp = now.format("%Y-%m-%d"); + let report_path = args + .storage_root + .join("_catalog") + .join("subjects") + .join(format!("_retention_sweep_{date_stamp}.jsonl")); + if let Some(parent) = report_path.parent() { + std::fs::create_dir_all(parent).map_err(|e| format!("mkdir {parent:?}: {e}"))?; + } + let mut buf = String::new(); + for row in &overdue_rows { + let line = serde_json::to_string(row).map_err(|e| format!("serialize row: {e}"))?; + buf.push_str(&line); + buf.push('\n'); + } + std::fs::write(&report_path, &buf).map_err(|e| format!("write {report_path:?}: {e}"))?; + eprintln!("[retention_sweep] report written: {}", report_path.display()); + eprintln!("[retention_sweep] action required by legal/operator. no manifests were mutated."); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use chrono::Duration; + use shared::types::{ + BiometricConsent, ConsentStatus, GeneralPiiConsent, SubjectConsent, SubjectRetention, + SubjectVertical, + }; + + fn fixture( + id: &str, + status: SubjectStatus, + gen_until: DateTime, + bio_until: Option>, + ) -> SubjectManifest { + SubjectManifest { + schema: "subject_manifest.v1".into(), + candidate_id: id.into(), + created_at: Utc::now(), + updated_at: Utc::now(), + status, + vertical: SubjectVertical::General, + consent: SubjectConsent { + general_pii: GeneralPiiConsent { + status: ConsentStatus::Given, + version: "v1".into(), + given_at: None, + withdrawn_at: None, + }, + biometric: BiometricConsent { + status: if bio_until.is_some() { + BiometricConsentStatus::Given + } else { + BiometricConsentStatus::NeverCollected + }, + retention_until: bio_until, + }, + }, + retention: SubjectRetention { + general_pii_until: gen_until, + policy: "test".into(), + }, + datasets: vec![], + safe_views: vec![], + audit_log_path: String::new(), + audit_log_chain_root: String::new(), + biometric_collection: None, + } + } + + #[test] + fn active_subject_inside_window_not_overdue() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::Active, now + Duration::days(30), None); + assert!(is_overdue(&s, now).is_none()); + } + + #[test] + fn active_subject_past_general_window_flags_general() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::Active, now - Duration::days(1), None); + assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii)); + } + + #[test] + fn erased_subject_never_re_flags_even_if_far_past() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::Erased, now - Duration::days(1000), None); + assert!(is_overdue(&s, now).is_none(), "erased subjects must not re-fire"); + } + + #[test] + fn already_retention_expired_is_idempotent() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::RetentionExpired, now - Duration::days(1), None); + assert!(is_overdue(&s, now).is_none(), "already-flagged subjects must not re-fire"); + } + + #[test] + fn biometric_overdue_alone_triggers_bipa_path() { + let now = Utc::now(); + let s = fixture( + "a", + SubjectStatus::Active, + now + Duration::days(30), + Some(now - Duration::days(1)), + ); + assert_eq!(is_overdue(&s, now), Some(OverdueReason::Biometric)); + } + + #[test] + fn both_clocks_overdue_reports_both() { + let now = Utc::now(); + let s = fixture( + "a", + SubjectStatus::Active, + now - Duration::days(2), + Some(now - Duration::days(1)), + ); + assert_eq!(is_overdue(&s, now), Some(OverdueReason::Both)); + } + + #[test] + fn pending_consent_subject_can_still_expire() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::PendingConsent, now - Duration::days(1), None); + assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii)); + } + + #[test] + fn withdrawn_subject_still_subject_to_retention() { + let now = Utc::now(); + let s = fixture("a", SubjectStatus::Withdrawn, now - Duration::days(1), None); + assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii)); + } +} diff --git a/crates/catalogd/src/biometric_endpoint.rs b/crates/catalogd/src/biometric_endpoint.rs new file mode 100644 index 0000000..50a9012 --- /dev/null +++ b/crates/catalogd/src/biometric_endpoint.rs @@ -0,0 +1,2727 @@ +//! `POST /biometric/subject/{id}/photo` — Phase 1.6 Gate 3 photo intake. +//! +//! Specification: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 3. +//! +//! This is the consent-gated entry point for REAL candidate photographs. +//! The pre-existing `/headshots/` route in mcp-server serves +//! SYNTHETIC StyleGAN faces from a deterministic hash → pool index; +//! that surface is unrelated to this endpoint and is NOT a place where +//! real candidate photos are ever stored or served. +//! +//! Flow: +//! 1. Caller sends `Content-Type: image/jpeg|image/png` + raw bytes +//! + `X-Lakehouse-Legal-Token` (writes to legal-tier storage are +//! privileged operations). +//! 2. Endpoint loads `SubjectManifest` for the candidate_id. +//! 3. Endpoint refuses 403 if `consent.biometric.status != "given"`. +//! 4. Endpoint hashes the bytes (SHA-256), writes to a quarantined +//! path under the configured biometric storage root with mode 0600. +//! 5. Endpoint updates the `SubjectManifest.biometric_collection` +//! field via Registry.put_subject() (creates audit on the manifest +//! via the registry's normal write path). +//! 6. Endpoint appends a `SubjectAuditRow` with +//! `accessor.kind="biometric_collection"`, `result="success"`, +//! `fields_accessed=["photo"]` to the per-subject HMAC chain. +//! +//! V1 deliberately does NOT: +//! - Run deepface against the photo. Classifications are deferred to +//! Gate 3b (offline subprocess design — Python sidecar was dropped +//! and we don't restore it here). +//! - Encrypt the photo at rest with a per-DEK. Filesystem ACL (mode +//! 0700 dir + 0600 file) is the v1 boundary; full at-rest encryption +//! is the storage-tier concern (Phase 1.6 spec §8). +//! - Validate that the bytes are actually a valid JPEG/PNG. The +//! content-type header gates the file extension; we trust the caller +//! to send well-formed images. A malformed image will fail downstream +//! when deepface runs in Gate 3b. + +use crate::registry::Registry; +use crate::subject_audit::SubjectAuditWriter; +use axum::{ + Json, Router, + body::Bytes, + extract::{DefaultBodyLimit, Path, State}, + http::{HeaderMap, StatusCode}, + response::IntoResponse, + routing::{get, post}, +}; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use shared::types::{AuditAccessor, BiometricCollection, SubjectAuditRow}; +use std::path::PathBuf; +use std::sync::Arc; + +const LEGAL_TOKEN_HEADER: &str = "x-lakehouse-legal-token"; +const CONSENT_VERSION_HEADER: &str = "x-lakehouse-consent-version-hash"; +const TRACE_ID_HEADER: &str = "x-lakehouse-trace-id"; +const RESPONSE_SCHEMA: &str = "biometric_photo_response.v1"; +const MAX_PHOTO_BYTES: usize = 10 * 1024 * 1024; // 10MB ceiling + +/// State for the biometric endpoint router. Same shape pattern as +/// AuditEndpointState — registry + writer + legal_token, plus the +/// configured storage root for the quarantined uploads directory. +#[derive(Clone)] +pub struct BiometricEndpointState { + pub registry: Registry, + pub writer: Arc, + pub legal_token: Option>, + /// Filesystem path that prefixes every quarantined photo path. + /// Default: `/data/biometric/uploads`. Set per host via + /// LH_BIOMETRIC_STORAGE_ROOT env var. + pub storage_root: PathBuf, + /// Optional allowlist of accepted `consent_version_hash` values. + /// `None` = permissive (any non-empty hash accepted, the v1 + /// behavior). `Some(set)` = strict (refuse anything not in the + /// set). Loaded from a JSON config file at startup if one exists. + /// Counsel-tier production deployments should populate this so + /// operator typos / stale templates / drift can't silently land + /// invalid consent records. + pub allowed_consent_versions: Option>>, +} + +#[derive(serde::Deserialize)] +struct ConsentVersionsConfig { + /// Array of accepted SHA-256 hash strings. Each must be 64 hex + /// chars (lowercase). Comments / metadata can ride on the JSON + /// object itself; the parser only consumes "versions". + versions: Vec, +} + +impl BiometricEndpointState { + /// Construct from existing pieces. `legal_token_path` is read at + /// construction (mirrors AuditEndpointState behavior so a shared + /// token file works for both endpoints). + pub async fn new( + registry: Registry, + writer: Arc, + legal_token_path: &std::path::Path, + storage_root: PathBuf, + ) -> Self { + let legal_token = match tokio::fs::read_to_string(legal_token_path).await { + Ok(s) => { + let trimmed = s.trim().to_string(); + if trimmed.len() < 32 { + tracing::warn!( + "biometric endpoint: legal token at {} is {} chars (min 32) — endpoint will 503", + legal_token_path.display(), trimmed.len() + ); + None + } else { + Some(Arc::new(trimmed)) + } + } + Err(e) => { + tracing::warn!( + "biometric endpoint: legal token unreadable from {} ({e}) — endpoint will 503", + legal_token_path.display() + ); + None + } + }; + // Ensure the storage root exists with restrictive perms. dir 0700 + // = owner-only (operator running the gateway). + if let Err(e) = tokio::fs::create_dir_all(&storage_root).await { + tracing::error!( + "biometric endpoint: failed to create storage root {}: {e}", + storage_root.display() + ); + } else { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let _ = std::fs::set_permissions(&storage_root, std::fs::Permissions::from_mode(0o700)); + } + } + Self { + registry, + writer, + legal_token, + storage_root, + allowed_consent_versions: None, + } + } + + /// Load a JSON allowlist of accepted consent_version_hash values. + /// File shape: + /// `{"versions": ["", "", ...]}` + /// + /// Resolution rules: + /// - File missing or unreadable → permissive (None). Logged as + /// a warning so the operator sees the deployment is in v1 + /// compatibility mode. + /// - File present but parse fails → permissive (None) + ERROR + /// log (broken config silently degrading to permissive is + /// wrong; flag loudly). + /// - File present + parses + has ≥1 entry → strict mode + /// (Some(set)). Each hash is normalized to lowercase before + /// insertion to prevent operator-case drift. + /// - File present + empty `versions` array → strict (refuse + /// all consent grants). This is "intentionally locked, + /// waiting for counsel to publish v1 hash" mode. + pub async fn with_consent_versions(mut self, path: &std::path::Path) -> Self { + match tokio::fs::read_to_string(path).await { + Ok(body) => { + match serde_json::from_str::(&body) { + Ok(cfg) => { + let set: std::collections::HashSet = cfg + .versions + .into_iter() + .map(|s| s.trim().to_ascii_lowercase()) + .filter(|s| !s.is_empty()) + .collect(); + tracing::info!( + "biometric endpoint: consent_versions allowlist loaded from {} ({} entries)", + path.display(), set.len() + ); + self.allowed_consent_versions = Some(Arc::new(set)); + } + Err(e) => { + tracing::error!( + "biometric endpoint: consent_versions file at {} failed to parse ({e}); \ + running in PERMISSIVE mode (any non-empty hash accepted). \ + Fix the file or remove it to silence this warning.", + path.display() + ); + } + } + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + tracing::warn!( + "biometric endpoint: consent_versions allowlist not found at {} — \ + running in PERMISSIVE mode (any non-empty hash accepted). \ + For counsel-tier deployment, create the file with the signed v1 hash.", + path.display() + ); + } + Err(e) => { + tracing::error!( + "biometric endpoint: consent_versions file at {} unreadable ({e}); \ + running in PERMISSIVE mode.", + path.display() + ); + } + } + self + } +} + +pub fn router(state: BiometricEndpointState) -> Router { + Router::new() + .route("/subject/{candidate_id}/photo", post(upload_photo)) + .route("/subject/{candidate_id}/consent", post(record_consent)) + .route("/subject/{candidate_id}/withdraw", post(withdraw_consent)) + .route("/subject/{candidate_id}/erase", post(erase_subject)) + .route("/stats", get(stats_handler)) + .route("/health", get(biometric_health)) + .layer(DefaultBodyLimit::max(MAX_PHOTO_BYTES)) + .with_state(state) +} + +async fn biometric_health(State(state): State) -> impl IntoResponse { + if state.legal_token.is_some() { + (StatusCode::OK, "biometric endpoint ready").into_response() + } else { + (StatusCode::SERVICE_UNAVAILABLE, "biometric endpoint disabled (legal token missing)").into_response() + } +} + +/// Constant-time string comparison — same util as audit_endpoint. +/// Duplicated here rather than cross-imported because audit_endpoint is +/// not a stable public surface. +fn constant_time_eq(a: &[u8], b: &[u8]) -> bool { + if a.len() != b.len() { return false; } + let mut diff: u8 = 0; + for (x, y) in a.iter().zip(b.iter()) { + diff |= x ^ y; + } + diff == 0 +} + +/// Map Content-Type header to the file extension used in the +/// quarantined path. Unknown types are rejected (415) so we don't +/// silently store HEIC / WebP / SVG without an explicit decision. +/// +/// The header value MAY include parameters per RFC 9110 §8.3 +/// (`image/jpeg; charset=binary`, `image/jpeg; boundary=...`); we +/// strip parameters before matching on the bare media type. +fn extension_from_content_type(ct: Option<&str>) -> Result<&'static str, (StatusCode, &'static str)> { + let bare = ct + .and_then(|s| s.split(';').next()) + .map(|s| s.trim().to_ascii_lowercase()); + match bare.as_deref() { + Some("image/jpeg") | Some("image/jpg") => Ok("jpg"), + Some("image/png") => Ok("png"), + _ => Err(( + StatusCode::UNSUPPORTED_MEDIA_TYPE, + "Content-Type must be image/jpeg or image/png", + )), + } +} + +/// Sanitize a candidate_id for filesystem use. Same character set as +/// SubjectAuditWriter::audit_key — keeps the per-subject directory +/// name aligned with the per-subject audit log. +fn sanitize_for_path(s: &str) -> String { + s.chars() + .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' { c } else { '_' }) + .collect() +} + +#[derive(Serialize, Debug)] +pub struct UploadResponse { + pub schema: &'static str, + pub candidate_id: String, + pub data_path: String, + pub template_hash: String, + pub collected_at: chrono::DateTime, + pub consent_version_hash: String, + pub retention_until: Option>, + pub audit_row_hmac: String, +} + +#[derive(Serialize, Debug)] +pub struct ErrorResponse { + pub error: &'static str, + pub detail: String, + pub consent_status: Option, +} + +async fn upload_photo( + State(state): State, + Path(candidate_id): Path, + headers: HeaderMap, + body: Bytes, +) -> impl IntoResponse { + let auth_token = headers + .get(LEGAL_TOKEN_HEADER) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + let content_type = headers + .get(axum::http::header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + let consent_version_hash = headers + .get(CONSENT_VERSION_HEADER) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let trace_id = headers + .get(TRACE_ID_HEADER) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + match process_upload( + &state, + &candidate_id, + auth_token.as_deref(), + content_type.as_deref(), + &consent_version_hash, + &trace_id, + body.as_ref(), + ).await { + Ok(resp) => (StatusCode::OK, Json(resp)).into_response(), + Err((status, err)) => (status, Json(err)).into_response(), + } +} + +/// Pure logic for the upload — extracted so unit tests can exercise +/// every branch without HTTP machinery. Inputs are the raw header +/// values + body bytes; output is either the success body or +/// (status, error body). Filesystem + registry + audit-writer +/// interactions stay in here so the test surface IS the behavior. +pub async fn process_upload( + state: &BiometricEndpointState, + candidate_id: &str, + legal_token: Option<&str>, + content_type: Option<&str>, + consent_version_hash: &str, + trace_id: &str, + body: &[u8], +) -> Result { + // Auth. + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + ErrorResponse { error: "auth_failed", detail: "no legal token configured".into(), consent_status: None }, + ))?; + let provided = legal_token.ok_or(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "missing X-Lakehouse-Legal-Token".into(), consent_status: None }, + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "X-Lakehouse-Legal-Token mismatch".into(), consent_status: None }, + )); + } + if candidate_id.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "candidate_id is empty".into(), consent_status: None, + })); + } + if body.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "photo body is empty".into(), consent_status: None, + })); + } + let ext = extension_from_content_type(content_type).map_err(|(s, m)| (s, ErrorResponse { + error: "unsupported_media_type", detail: m.into(), consent_status: None, + }))?; + + // We hold the original manifest for rollback. Cloning here so the + // mutated copy and the original are separable references. + let original_manifest = state.registry.get_subject(candidate_id).await.ok_or(( + StatusCode::NOT_FOUND, + ErrorResponse { + error: "subject_not_found", + detail: format!("no SubjectManifest registered for candidate_id={candidate_id}"), + consent_status: None, + }, + ))?; + let mut manifest = original_manifest.clone(); + + use shared::types::{BiometricConsentStatus, SubjectStatus}; + if manifest.consent.biometric.status != BiometricConsentStatus::Given { + return Err((StatusCode::FORBIDDEN, ErrorResponse { + error: "consent_required", + detail: "BIPA consent required before biometric processing".into(), + consent_status: Some(format!("{:?}", manifest.consent.biometric.status)), + })); + } + if matches!(manifest.status, SubjectStatus::Withdrawn | SubjectStatus::Erased | SubjectStatus::RetentionExpired) { + return Err((StatusCode::FORBIDDEN, ErrorResponse { + error: "subject_inactive", + detail: format!("subject status {:?} — biometric collection not permitted", manifest.status), + consent_status: None, + })); + } + // Refuse double-upload. If a BiometricCollection already exists on + // the manifest, the operator must explicitly erase before re-uploading. + // Without this gate, a second POST silently overwrites manifest.data_path + // and orphans the previous photo file on disk — creating a forever-leak + // pattern and a BIPA defensibility hole ("we said we erased the photo, + // but the previous version of it is still under the same subject dir"). + if manifest.biometric_collection.is_some() { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "biometric_already_collected", + detail: "subject already has a BiometricCollection on the manifest; \ + POST /biometric/subject/{id}/erase first if you intend to replace the photo".into(), + consent_status: None, + })); + } + + let template_hash = { + let mut h = Sha256::new(); + h.update(body); + let bytes = h.finalize(); + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut s = String::with_capacity(64); + for b in bytes { s.push(HEX[(b >> 4) as usize] as char); s.push(HEX[(b & 0x0f) as usize] as char); } + s + }; + let collected_at = chrono::Utc::now(); + let safe_id = sanitize_for_path(candidate_id); + let subject_dir = state.storage_root.join(&safe_id); + tokio::fs::create_dir_all(&subject_dir).await.map_err(|e| ( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "storage_error", detail: format!("create dir: {e}"), consent_status: None }, + ))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let _ = std::fs::set_permissions(&subject_dir, std::fs::Permissions::from_mode(0o700)); + } + // Filename: timestamp-nanos + 4-byte random suffix to defeat the + // (very unlikely) two-uploads-in-the-same-nanosecond collision. + // uuid::Uuid::new_v4 → first 8 hex chars (32 bits of entropy is + // enough — collision over a per-subject namespace at sub-microsecond + // cadence is implausible). + let suffix: String = uuid::Uuid::new_v4().simple().to_string().chars().take(8).collect(); + let filename = format!("{}_{}.{}", collected_at.timestamp_nanos_opt().unwrap_or(0), suffix, ext); + let abs_path = subject_dir.join(&filename); + tokio::fs::write(&abs_path, body).await.map_err(|e| ( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "storage_error", detail: format!("write photo: {e}"), consent_status: None }, + ))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let _ = std::fs::set_permissions(&abs_path, std::fs::Permissions::from_mode(0o600)); + } + // Store the path relative to storage_root in the manifest (matches + // the BiometricCollection doc-comment: "relative to the configured + // biometric storage root"). Operators reading the manifest can + // rebuild the absolute path with their own storage_root. + let stored_path = abs_path + .strip_prefix(&state.storage_root) + .map(|p| p.to_string_lossy().to_string()) + .unwrap_or_else(|_| abs_path.to_string_lossy().to_string()); + + manifest.biometric_collection = Some(BiometricCollection { + data_path: stored_path.clone(), + template_hash: template_hash.clone(), + collected_at, + consent_version_hash: consent_version_hash.to_string(), + classifications: None, + }); + manifest.updated_at = collected_at; + + // Transactional commit: + // + // The three side-effects MUST be all-or-nothing for BIPA defensibility: + // - photo file on disk + // - manifest reflects the collection + // - audit row is appended to the chain + // + // If any later step fails, prior steps are rolled back. The rollback + // path uses best-effort cleanup — every revert error is logged but + // does not propagate (the user-facing failure is the original cause). + // + // Order: photo (already written above) → manifest → audit. We commit + // manifest BEFORE audit because the audit row references the + // manifest's chain root; rolling back a committed audit row would + // require chain rewriting (which we explicitly forbid). + if let Err(e) = state.registry.put_subject(manifest.clone()).await { + // Manifest write failed — clean up the photo and bail. + if let Err(rm_err) = tokio::fs::remove_file(&abs_path).await { + tracing::error!("rollback failed: could not remove orphan photo {}: {rm_err}", stored_path); + } + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "manifest_update_failed", detail: e, consent_status: None }, + )); + } + + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: collected_at, + candidate_id: candidate_id.to_string(), + accessor: AuditAccessor { + kind: "biometric_collection".into(), + daemon: "gateway".into(), + purpose: "photo_upload".into(), + trace_id: trace_id.to_string(), + }, + fields_accessed: vec!["photo".into(), "biometric_template_hash".into()], + result: "success".into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + let audit_row_hmac = match state.writer.append(row).await { + Ok(h) => h, + Err(e) => { + // Audit-write failed AFTER photo + manifest committed. + // Roll back BOTH: remove the photo and revert the manifest + // to its pre-upload state. Returning 500 is the BIPA- + // defensible answer — a successful response without an + // audit row is exactly the silent-failure mode the spec + // exists to prevent. + tracing::error!("audit row failed for {candidate_id}: {e}; rolling back"); + if let Err(rm_err) = tokio::fs::remove_file(&abs_path).await { + tracing::error!("rollback: could not remove photo {}: {rm_err}", stored_path); + } + if let Err(re_err) = state.registry.put_subject(original_manifest).await { + tracing::error!("rollback: could not revert manifest for {candidate_id}: {re_err}"); + } + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { + error: "audit_write_failed", + detail: format!("audit row failed; photo + manifest rolled back: {e}"), + consent_status: None, + }, + )); + } + }; + + Ok(UploadResponse { + schema: RESPONSE_SCHEMA, + candidate_id: candidate_id.to_string(), + data_path: stored_path, + template_hash, + collected_at, + consent_version_hash: consent_version_hash.to_string(), + retention_until: manifest.consent.biometric.retention_until, + audit_row_hmac, + }) +} + +// ───────────────────────────────────────────────────────────────────── +// Phase 1.6 Gate 5 — POST /biometric/subject/{id}/erase +// ───────────────────────────────────────────────────────────────────── +// +// Specification: docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md. +// +// BIPA-defensible erasure path. Clears the subject's biometric +// collection (always) and optionally the full PII record. The +// destruction is recorded as an append-only audit row in the +// per-subject HMAC chain — that row IS the legal proof of compliant +// destruction even after the underlying data is gone. +// +// Triggers (from runbook §1): retention_expiry / consent_withdrawal / +// rtbf / court_order. The endpoint records but does not validate the +// trigger token — operator-of-record is responsible for matching the +// trigger to a documented incident. +// +// Two scopes: +// - "biometric_only" (default): clears biometric_collection field, +// unlinks the photo file, sets consent.biometric.status = "withdrawn". +// Subject remains active; only biometric data is destroyed. +// - "full": above PLUS sets status = "erased" and +// consent.general_pii.status = "withdrawn". The subject record +// itself is preserved (BIPA requires proof of destruction; we +// don't delete the manifest, just mark it erased). + +#[derive(serde::Deserialize)] +pub struct EraseRequest { + /// One of: "retention_expiry", "consent_withdrawal", "rtbf", + /// "court_order". Caller-defined; we don't validate against a + /// closed set so future triggers (regulatory, contractual) work + /// without code change. + pub trigger: String, + /// Path or reference to the signed artifact justifying erasure + /// (e.g. operator's withdrawal-receipt PDF, court order PDF). + /// Recorded in the audit row but not validated. + #[serde(default)] + pub trigger_evidence_path: String, + /// Operator initiating the erasure. Must be a real human name — + /// recorded in the audit row for legal traceability. + pub operator_of_record: String, + /// Witness operator (per runbook §3 "two-operator action"). Must + /// be different from operator_of_record when supplied; we don't + /// validate same-as-operator since dev/test deployments may have + /// only one operator. + #[serde(default)] + pub witness: String, + /// "biometric_only" (default) | "full". + #[serde(default = "default_erase_scope")] + pub scope: String, +} + +fn default_erase_scope() -> String { "biometric_only".into() } + +#[derive(Serialize, Debug)] +pub struct EraseResponse { + pub schema: &'static str, + pub candidate_id: String, + pub scope: String, + pub trigger: String, + pub erased_at: chrono::DateTime, + pub fields_cleared: Vec, + pub photo_unlinked: bool, + pub photo_unlink_error: Option, + pub status_after: String, + pub biometric_status_after: String, + pub general_pii_status_after: String, + pub audit_row_hmac: String, +} + +const ERASE_RESPONSE_SCHEMA: &str = "biometric_erase_response.v1"; + +async fn erase_subject( + State(state): State, + Path(candidate_id): Path, + headers: HeaderMap, + body: Bytes, +) -> impl IntoResponse { + let auth_token = headers + .get(LEGAL_TOKEN_HEADER) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + let trace_id = headers + .get(TRACE_ID_HEADER) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let req: EraseRequest = match serde_json::from_slice(&body) { + Ok(r) => r, + Err(e) => return (StatusCode::BAD_REQUEST, Json(ErrorResponse { + error: "bad_request", + detail: format!("erase body must be JSON {{trigger, operator_of_record, ...}}: {e}"), + consent_status: None, + })).into_response(), + }; + match process_erase(&state, &candidate_id, auth_token.as_deref(), &trace_id, req).await { + Ok(resp) => (StatusCode::OK, Json(resp)).into_response(), + Err((status, err)) => (status, Json(err)).into_response(), + } +} + +/// Pure-logic erasure — same testable shape as process_upload. +pub async fn process_erase( + state: &BiometricEndpointState, + candidate_id: &str, + legal_token: Option<&str>, + trace_id: &str, + req: EraseRequest, +) -> Result { + // Auth. + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + ErrorResponse { error: "auth_failed", detail: "no legal token configured".into(), consent_status: None }, + ))?; + let provided = legal_token.ok_or(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "missing X-Lakehouse-Legal-Token".into(), consent_status: None }, + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "X-Lakehouse-Legal-Token mismatch".into(), consent_status: None }, + )); + } + if candidate_id.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "candidate_id is empty".into(), consent_status: None, + })); + } + // Body validation. + if req.trigger.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "trigger is required (retention_expiry|consent_withdrawal|rtbf|court_order)".into(), + consent_status: None, + })); + } + if req.operator_of_record.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "operator_of_record is required".into(), + consent_status: None, + })); + } + let scope = req.scope.as_str(); + if scope != "biometric_only" && scope != "full" { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: format!("scope must be biometric_only or full (got {scope})"), + consent_status: None, + })); + } + + let original_manifest = state.registry.get_subject(candidate_id).await.ok_or(( + StatusCode::NOT_FOUND, + ErrorResponse { error: "subject_not_found", detail: format!("no SubjectManifest for {candidate_id}"), consent_status: None }, + ))?; + + use shared::types::{BiometricConsentStatus, ConsentStatus, SubjectStatus}; + + // Idempotency: erasing an already-erased subject is a no-op for the + // manifest BUT we still write an audit row noting the redundant + // request — that's a useful operational signal. + let already_erased = matches!(original_manifest.status, SubjectStatus::Erased); + + let erased_at = chrono::Utc::now(); + let mut new_manifest = original_manifest.clone(); + let mut fields_cleared: Vec = Vec::new(); + + // Always: clear biometric_collection + flip biometric consent. + if new_manifest.biometric_collection.is_some() { + fields_cleared.push("biometric_collection".into()); + } + new_manifest.biometric_collection = None; + if new_manifest.consent.biometric.status != BiometricConsentStatus::Withdrawn { + new_manifest.consent.biometric.status = BiometricConsentStatus::Withdrawn; + new_manifest.consent.biometric.retention_until = None; + fields_cleared.push("consent.biometric".into()); + } + + // Full scope: also flip status + general_pii consent. + if scope == "full" { + if new_manifest.status != SubjectStatus::Erased { + new_manifest.status = SubjectStatus::Erased; + fields_cleared.push("status".into()); + } + if new_manifest.consent.general_pii.status != ConsentStatus::Withdrawn { + new_manifest.consent.general_pii.status = ConsentStatus::Withdrawn; + new_manifest.consent.general_pii.withdrawn_at = Some(erased_at); + fields_cleared.push("consent.general_pii".into()); + } + } + new_manifest.updated_at = erased_at; + + // Commit manifest BEFORE unlinking the file. Order matters for + // the same reason as process_upload: the audit row is the legal + // record of intent. Manifest update + audit row commit first; + // file unlink is the last (irreversible) step. + state.registry.put_subject(new_manifest.clone()).await.map_err(|e| ( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "manifest_update_failed", detail: e, consent_status: None }, + ))?; + + // Append erasure audit row. Records the trigger, operator, witness, + // evidence path, scope, and which fields were cleared. + let kind = if scope == "full" { "full_erasure" } else { "biometric_erasure" }; + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: erased_at, + candidate_id: candidate_id.to_string(), + accessor: AuditAccessor { + kind: kind.into(), + daemon: "gateway".into(), + purpose: format!( + "trigger={};operator={};witness={};evidence={}", + req.trigger, req.operator_of_record, req.witness, req.trigger_evidence_path, + ), + trace_id: trace_id.to_string(), + }, + fields_accessed: fields_cleared.clone(), + result: if already_erased { "already_erased".into() } else { "erased".into() }, + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + let audit_row_hmac = match state.writer.append(row).await { + Ok(h) => h, + Err(e) => { + // Audit failure on erasure: the manifest is updated but + // the legal record of WHY didn't land. Roll back the + // manifest to the original (not yet "erased") state and + // return 500 — the operator can retry. + tracing::error!("erase audit row failed for {candidate_id}: {e}; rolling back"); + if let Err(re_err) = state.registry.put_subject(original_manifest).await { + tracing::error!("rollback: could not revert manifest for {candidate_id}: {re_err}"); + } + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { + error: "audit_write_failed", + detail: format!("audit row failed; manifest rolled back: {e}"), + consent_status: None, + }, + )); + } + }; + + // Last: unlink the photo file (if one was on disk). This is + // irreversible — we deliberately do it AFTER the audit row commits + // so the legal record exists even if the unlink fails. + let mut photo_unlinked = true; + let mut photo_unlink_error: Option = None; + if let Some(bc) = original_manifest.biometric_collection.as_ref() { + let abs = state.storage_root.join(&bc.data_path); + // Best-effort secure overwrite: write zeros equal to file + // size, then unlink. Modern filesystems / SSDs may keep + // copies — full secure erase requires drive-level operations + // (out of scope for v1; documented as a limitation). + match tokio::fs::metadata(&abs).await { + Ok(meta) => { + let size = meta.len() as usize; + let zeros = vec![0u8; size.min(64 * 1024)]; + let mut written = 0usize; + while written < size { + let chunk = (size - written).min(zeros.len()); + if let Err(e) = tokio::fs::write(&abs, &zeros[..chunk]).await { + photo_unlinked = false; + photo_unlink_error = Some(format!("overwrite failed at offset {written}: {e}")); + break; + } + written += chunk; + if written < size { + // Re-open in append mode for subsequent chunks + // (write-truncate would zero only the chunk + // length). For v1 simplicity we just write the + // file once with zeros up to 64KB; larger files + // get partial overwrite + unlink. Operators + // wanting full secure erase should run shred(1). + break; + } + } + if let Err(e) = tokio::fs::remove_file(&abs).await { + photo_unlinked = false; + photo_unlink_error = Some(format!("unlink failed: {e}")); + } + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + // File already gone — idempotent. + photo_unlinked = true; + } + Err(e) => { + photo_unlinked = false; + photo_unlink_error = Some(format!("stat failed: {e}")); + } + } + } + if !photo_unlinked { + tracing::error!( + "erase: photo file remains on disk for {candidate_id}: {:?} — manifest + audit row already committed; operator must shred manually", + photo_unlink_error + ); + } + + let status_after = format!("{:?}", new_manifest.status); + let biometric_status_after = format!("{:?}", new_manifest.consent.biometric.status); + let general_pii_status_after = format!("{:?}", new_manifest.consent.general_pii.status); + + Ok(EraseResponse { + schema: ERASE_RESPONSE_SCHEMA, + candidate_id: candidate_id.to_string(), + scope: scope.to_string(), + trigger: req.trigger, + erased_at, + fields_cleared, + photo_unlinked, + photo_unlink_error, + status_after, + biometric_status_after, + general_pii_status_after, + audit_row_hmac, + }) +} + +// ─── Consent recording (Gate 2 backend) ───────────────────────────── +// +// Backs the candidate-facing consent flow specified in +// `docs/policies/consent/biometric_consent_template_v1.md` §5. Every +// candidate whose photo is later uploaded MUST first have a manifest +// with `consent.biometric.status = Given` — produced by this endpoint. +// +// Auth: legal-tier (operator records on behalf of candidate). When the +// candidate-facing intake UI ships, this endpoint stays operator-side; +// the UI captures the signature + posts here. The candidate's +// browser/app NEVER touches this endpoint directly. +// +// Idempotency posture: +// - if subject is at NeverCollected / Pending → flip to Given (happy path) +// - if subject is at Given → 409 (already given; consent doesn't +// re-stamp). Re-collect requires explicit erase first. +// - if subject is at Withdrawn / Expired → 409 (must erase residue +// of the prior cycle before granting a fresh consent) +// +// Retention clock: 18 months from given_at, per retention schedule v1 +// §4. If counsel changes the cap, this constant + the retention doc +// must change in the same PR (intentional friction). + +const CONSENT_RESPONSE_SCHEMA: &str = "biometric_consent_response.v1"; +const CONSENT_RETENTION_DAYS: i64 = 18 * 30; // 540 days ≈ 18 months per retention schedule §4 + +const ALLOWED_CONSENT_METHODS: &[&str] = &["electronic_signature", "paper", "click_acceptance"]; + +#[derive(serde::Deserialize, Debug, Clone)] +pub struct ConsentRequest { + /// SHA-256 of the consent template version the candidate signed + /// under. Required + non-empty. Recorded in the audit row for + /// legal traceability; counsel validates retroactively against the + /// known good consent_versions table. We deliberately do NOT gate + /// against an in-process allowlist — that would couple the endpoint + /// to a config-deployment dependency for every template rotation. + pub consent_version_hash: String, + /// Collection method per consent template §5. Constrained to a + /// closed set (electronic_signature / paper / click_acceptance) + /// because counsel needs to know what evidence shape was captured + /// and operator typo would silently weaken defensibility. + pub consent_collection_method: String, + /// Path or URL to the signed artifact (PDF receipt, e-sig audit + /// log, scanned paper, etc.). Recorded in the audit row for + /// chain-of-custody. May be empty for click_acceptance where the + /// "evidence" is the click + timestamp + IP captured upstream. + #[serde(default)] + pub consent_collection_evidence_path: String, + /// Operator initiating the consent record (the staffing coordinator + /// who collected the signature, or the intake-UI service principal). + /// Required for audit traceability — every grant must be attributable. + pub operator_of_record: String, +} + +#[derive(Serialize, Debug)] +pub struct ConsentResponse { + pub schema: &'static str, + pub candidate_id: String, + pub status_after: String, + pub consent_version_hash: String, + pub given_at: chrono::DateTime, + pub retention_until: chrono::DateTime, + pub audit_row_hmac: String, +} + +async fn record_consent( + State(state): State, + Path(candidate_id): Path, + headers: HeaderMap, + body: Bytes, +) -> impl IntoResponse { + let auth_token = headers + .get(LEGAL_TOKEN_HEADER) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + let trace_id = headers + .get(TRACE_ID_HEADER) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let req: ConsentRequest = match serde_json::from_slice(&body) { + Ok(r) => r, + Err(e) => return (StatusCode::BAD_REQUEST, Json(ErrorResponse { + error: "bad_request", + detail: format!( + "consent body must be JSON {{consent_version_hash, consent_collection_method, operator_of_record, ...}}: {e}" + ), + consent_status: None, + })).into_response(), + }; + match process_consent(&state, &candidate_id, auth_token.as_deref(), &trace_id, req).await { + Ok(resp) => (StatusCode::OK, Json(resp)).into_response(), + Err((status, err)) => (status, Json(err)).into_response(), + } +} + +/// Pure logic for consent recording — same testable shape as +/// `process_upload` and `process_erase`. Inputs are the raw header +/// values + parsed body; output is either the success body or +/// (status, error body). Manifest + audit-writer interactions stay +/// in here so the test surface IS the behavior. +pub async fn process_consent( + state: &BiometricEndpointState, + candidate_id: &str, + legal_token: Option<&str>, + trace_id: &str, + req: ConsentRequest, +) -> Result { + // Auth (mirrors process_upload / process_erase). + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + ErrorResponse { error: "auth_failed", detail: "no legal token configured".into(), consent_status: None }, + ))?; + let provided = legal_token.ok_or(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "missing X-Lakehouse-Legal-Token".into(), consent_status: None }, + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "X-Lakehouse-Legal-Token mismatch".into(), consent_status: None }, + )); + } + if candidate_id.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "candidate_id is empty".into(), consent_status: None, + })); + } + + // Body validation. Empty / whitespace-only fields would silently + // weaken defensibility (a consent record without a version hash is + // not a defensible consent record). + if req.consent_version_hash.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: "consent_version_hash is required (SHA-256 of the consent template version the candidate signed under)".into(), + consent_status: None, + })); + } + // Allowlist enforcement (counsel-tier deployment safety). When + // `allowed_consent_versions` is configured, refuse any hash not + // in the set. Compared lowercase to defeat operator-case drift. + // Permissive mode (None) keeps v1 compatibility for deployments + // that haven't published a signed allowlist yet. + if let Some(allow) = state.allowed_consent_versions.as_ref() { + let normalized = req.consent_version_hash.trim().to_ascii_lowercase(); + if !allow.contains(&normalized) { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "consent_version_unknown", + detail: format!( + "consent_version_hash {} not in the configured allowlist; \ + this gateway accepts only hashes published in /etc/lakehouse/consent_versions.json. \ + Contact the deployment operator if a new template version needs to be authorized.", + normalized, + ), + consent_status: None, + })); + } + } + if req.operator_of_record.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: "operator_of_record is required".into(), + consent_status: None, + })); + } + let method = req.consent_collection_method.trim(); + if method.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: format!( + "consent_collection_method is required (one of: {})", + ALLOWED_CONSENT_METHODS.join(", "), + ), + consent_status: None, + })); + } + if !ALLOWED_CONSENT_METHODS.contains(&method) { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: format!( + "consent_collection_method must be one of: {} (got {})", + ALLOWED_CONSENT_METHODS.join(", "), + method, + ), + consent_status: None, + })); + } + + // Load original manifest. + let original_manifest = state.registry.get_subject(candidate_id).await.ok_or(( + StatusCode::NOT_FOUND, + ErrorResponse { + error: "subject_not_found", + detail: format!("no SubjectManifest for {candidate_id}"), + consent_status: None, + }, + ))?; + let mut manifest = original_manifest.clone(); + + use shared::types::{BiometricConsentStatus, SubjectStatus}; + + // Subject must be Active. Withdrawn/Erased/RetentionExpired subjects + // need explicit re-activation upstream before consent can be recorded. + if matches!(manifest.status, SubjectStatus::Withdrawn | SubjectStatus::Erased | SubjectStatus::RetentionExpired) { + return Err((StatusCode::FORBIDDEN, ErrorResponse { + error: "subject_inactive", + detail: format!("subject status {:?} — consent grant not permitted", manifest.status), + consent_status: None, + })); + } + + // State-machine transitions. Mirrors the upload double-upload guard + // (refuse re-grant; force explicit erase + re-consent cycle). + match manifest.consent.biometric.status { + BiometricConsentStatus::Given => { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "consent_already_given", + detail: "subject already has biometric consent given; \ + POST /biometric/subject/{id}/erase first if you intend to re-collect under a new consent version".into(), + consent_status: Some("Given".into()), + })); + } + BiometricConsentStatus::Withdrawn | BiometricConsentStatus::Expired => { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "consent_post_withdrawal_requires_erase", + detail: format!( + "subject biometric.status is {:?}; explicit erase required before fresh consent grant \ + (so any residual collection state is cleared and the audit chain records the cycle).", + manifest.consent.biometric.status, + ), + consent_status: Some(format!("{:?}", manifest.consent.biometric.status)), + })); + } + BiometricConsentStatus::NeverCollected | BiometricConsentStatus::Pending => { + // Happy path — proceed. + } + } + + // Compute timestamps. given_at = now (server-side authoritative — + // operator-supplied timestamp would be tamperable). retention_until + // = given_at + 18 months per retention schedule v1 §4. + let given_at = chrono::Utc::now(); + let retention_until = given_at + chrono::Duration::days(CONSENT_RETENTION_DAYS); + + manifest.consent.biometric.status = BiometricConsentStatus::Given; + manifest.consent.biometric.retention_until = Some(retention_until); + manifest.updated_at = given_at; + + // Transactional commit. Same posture as upload + erase: manifest + // commit before audit row, rollback manifest if audit fails. + if let Err(e) = state.registry.put_subject(manifest.clone()).await { + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "manifest_update_failed", detail: e, consent_status: None }, + )); + } + + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: given_at, + candidate_id: candidate_id.to_string(), + accessor: AuditAccessor { + kind: "biometric_consent_grant".into(), + daemon: "gateway".into(), + purpose: format!( + "version={};method={};operator={};evidence={}", + req.consent_version_hash, method, req.operator_of_record, req.consent_collection_evidence_path, + ), + trace_id: trace_id.to_string(), + }, + fields_accessed: vec![ + "consent.biometric.status".into(), + "consent.biometric.retention_until".into(), + ], + result: "given".into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + let audit_row_hmac = match state.writer.append(row).await { + Ok(h) => h, + Err(e) => { + // Audit failed — roll back manifest to original state. The + // grant is best understood as "didn't happen" until the + // audit chain says it did. Rollback errors are logged but + // don't propagate; the user-facing failure is the audit. + tracing::error!("consent audit row failed for {candidate_id}: {e}; rolling back"); + if let Err(re_err) = state.registry.put_subject(original_manifest).await { + tracing::error!("rollback: could not revert manifest for {candidate_id}: {re_err}"); + } + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { + error: "audit_write_failed", + detail: format!("audit row failed; manifest rolled back: {e}"), + consent_status: None, + }, + )); + } + }; + + Ok(ConsentResponse { + schema: CONSENT_RESPONSE_SCHEMA, + candidate_id: candidate_id.to_string(), + status_after: "Given".into(), + consent_version_hash: req.consent_version_hash, + given_at, + retention_until, + audit_row_hmac, + }) +} + +// ─── Consent withdrawal (BIPA right of withdrawal) ──────────────── +// +// Backs the candidate's BIPA right to withdraw biometric consent at +// any time. Required by 740 ILCS 14/15(b) (consent must be revocable) +// and explicitly promised in +// docs/policies/consent/biometric_consent_template_v1.md §2. +// +// Withdrawal is NOT erasure. It's a state change that: +// - Stops further biometric processing (uploads will 403 again +// because the consent gate flips back to non-Given) +// - Starts a 30-day SLA clock for destruction (retention_until is +// accelerated from the 18-month default to 30 days from now) +// - The retention sweep + operator-driven erase honor the SLA; +// this endpoint does not delete anything itself +// +// Why a separate endpoint from /erase: +// - /erase destroys immediately (full BIPA destruction at counsel's +// direction or court order — biometric_only or full scope) +// - /withdraw flips the state + sets the SLA clock so destruction +// can happen on schedule with a defensible audit trail of the +// candidate's request preceding the destruction by ≤30 days +// +// State machine: +// - Given → Withdrawn (happy path) +// - NeverCollected/Pending → 409 nothing_to_withdraw +// - Withdrawn → 409 already_withdrawn (idempotent via /audit read) +// - Expired → 409 already_expired +// Subject status: +// - Erased / RetentionExpired → 403 subject_inactive +// - Otherwise → proceed (biometric can be withdrawn even if +// general_pii is mid-flow) + +const WITHDRAW_RESPONSE_SCHEMA: &str = "biometric_withdraw_response.v1"; +const WITHDRAW_SLA_DAYS: i64 = 30; // per consent template v1 §2 + +#[derive(serde::Deserialize, Debug, Clone)] +pub struct WithdrawRequest { + /// Reason for withdrawal. Captured to the audit row as part of + /// accessor.purpose. Required + non-empty — counsel may need to + /// distinguish candidate-initiated vs. operator-initiated + /// withdrawals (e.g., candidate request vs. employer policy). + pub reason: String, + /// Operator initiating the withdrawal record (the staffing + /// coordinator who fielded the candidate's request). Required for + /// audit traceability — every state change must be attributable. + pub operator_of_record: String, + /// Optional: path or URL to a signed artifact proving the + /// candidate authorized withdrawal (signed paper, recorded call + /// transcript, email thread). Recorded for chain-of-custody. + #[serde(default)] + pub evidence_path: String, +} + +#[derive(Serialize, Debug)] +pub struct WithdrawResponse { + pub schema: &'static str, + pub candidate_id: String, + pub status_after: String, + pub withdrawn_at: chrono::DateTime, + pub retention_until: chrono::DateTime, + pub audit_row_hmac: String, +} + +async fn withdraw_consent( + State(state): State, + Path(candidate_id): Path, + headers: HeaderMap, + body: Bytes, +) -> impl IntoResponse { + let auth_token = headers + .get(LEGAL_TOKEN_HEADER) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + let trace_id = headers + .get(TRACE_ID_HEADER) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let req: WithdrawRequest = match serde_json::from_slice(&body) { + Ok(r) => r, + Err(e) => return (StatusCode::BAD_REQUEST, Json(ErrorResponse { + error: "bad_request", + detail: format!( + "withdraw body must be JSON {{reason, operator_of_record, evidence_path?}}: {e}" + ), + consent_status: None, + })).into_response(), + }; + match process_withdraw(&state, &candidate_id, auth_token.as_deref(), &trace_id, req).await { + Ok(resp) => (StatusCode::OK, Json(resp)).into_response(), + Err((status, err)) => (status, Json(err)).into_response(), + } +} + +/// Pure logic for consent withdrawal — same testable shape as +/// process_consent + process_upload + process_erase. +pub async fn process_withdraw( + state: &BiometricEndpointState, + candidate_id: &str, + legal_token: Option<&str>, + trace_id: &str, + req: WithdrawRequest, +) -> Result { + // Auth. + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + ErrorResponse { error: "auth_failed", detail: "no legal token configured".into(), consent_status: None }, + ))?; + let provided = legal_token.ok_or(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "missing X-Lakehouse-Legal-Token".into(), consent_status: None }, + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "X-Lakehouse-Legal-Token mismatch".into(), consent_status: None }, + )); + } + if candidate_id.is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", detail: "candidate_id is empty".into(), consent_status: None, + })); + } + + // Body validation. Empty fields silently weaken defensibility. + if req.reason.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: "reason is required (free-text rationale captured to audit row)".into(), + consent_status: None, + })); + } + if req.operator_of_record.trim().is_empty() { + return Err((StatusCode::BAD_REQUEST, ErrorResponse { + error: "bad_request", + detail: "operator_of_record is required".into(), + consent_status: None, + })); + } + + // Load original manifest. + let original_manifest = state.registry.get_subject(candidate_id).await.ok_or(( + StatusCode::NOT_FOUND, + ErrorResponse { + error: "subject_not_found", + detail: format!("no SubjectManifest for {candidate_id}"), + consent_status: None, + }, + ))?; + let mut manifest = original_manifest.clone(); + + use shared::types::{BiometricConsentStatus, SubjectStatus}; + + // Subject must not be already destroyed. + if matches!(manifest.status, SubjectStatus::Erased | SubjectStatus::RetentionExpired) { + return Err((StatusCode::FORBIDDEN, ErrorResponse { + error: "subject_inactive", + detail: format!("subject status {:?} — withdrawal not applicable", manifest.status), + consent_status: None, + })); + } + + // Biometric state machine. + match manifest.consent.biometric.status { + BiometricConsentStatus::Given => { + // Happy path — proceed. + } + BiometricConsentStatus::NeverCollected | BiometricConsentStatus::Pending => { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "nothing_to_withdraw", + detail: format!( + "biometric consent status is {:?} — nothing has been given to withdraw. \ + If consent is mid-flow and the candidate is rejecting, the consent record \ + should be discarded by the intake operator rather than withdrawn here.", + manifest.consent.biometric.status, + ), + consent_status: Some(format!("{:?}", manifest.consent.biometric.status)), + })); + } + BiometricConsentStatus::Withdrawn => { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "already_withdrawn", + detail: "biometric consent is already withdrawn; \ + confirm via GET /audit/subject/{id} to see the prior withdrawal row".into(), + consent_status: Some("Withdrawn".into()), + })); + } + BiometricConsentStatus::Expired => { + return Err((StatusCode::CONFLICT, ErrorResponse { + error: "already_expired", + detail: "biometric retention window already passed; \ + data is awaiting destruction sweep — withdrawal is moot".into(), + consent_status: Some("Expired".into()), + })); + } + } + + // Compute timestamps. withdrawn_at is server-side authoritative; + // retention_until is accelerated from the prior 18-month default + // to 30 days from now per consent template v1 §2. + let withdrawn_at = chrono::Utc::now(); + let retention_until = withdrawn_at + chrono::Duration::days(WITHDRAW_SLA_DAYS); + + manifest.consent.biometric.status = BiometricConsentStatus::Withdrawn; + manifest.consent.biometric.retention_until = Some(retention_until); + manifest.updated_at = withdrawn_at; + // NOTE: deliberately do NOT mutate manifest.status or + // manifest.consent.general_pii — withdrawal is biometric-scoped. + // A subject can keep their general PII processing while opting + // out of biometrics. Subject-level withdrawal is a separate + // (out-of-scope) endpoint. + + // Transactional: manifest commit before audit; rollback on audit failure. + if let Err(e) = state.registry.put_subject(manifest.clone()).await { + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { error: "manifest_update_failed", detail: e, consent_status: None }, + )); + } + + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: withdrawn_at, + candidate_id: candidate_id.to_string(), + accessor: AuditAccessor { + kind: "biometric_consent_withdrawal".into(), + daemon: "gateway".into(), + purpose: format!( + "reason={};operator={};evidence={}", + req.reason, req.operator_of_record, req.evidence_path, + ), + trace_id: trace_id.to_string(), + }, + fields_accessed: vec![ + "consent.biometric.status".into(), + "consent.biometric.retention_until".into(), + ], + result: "withdrawn".into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + let audit_row_hmac = match state.writer.append(row).await { + Ok(h) => h, + Err(e) => { + tracing::error!("withdraw audit row failed for {candidate_id}: {e}; rolling back"); + if let Err(re_err) = state.registry.put_subject(original_manifest).await { + tracing::error!("rollback: could not revert manifest for {candidate_id}: {re_err}"); + } + return Err(( + StatusCode::INTERNAL_SERVER_ERROR, + ErrorResponse { + error: "audit_write_failed", + detail: format!("audit row failed; manifest rolled back: {e}"), + consent_status: None, + }, + )); + } + }; + + Ok(WithdrawResponse { + schema: WITHDRAW_RESPONSE_SCHEMA, + candidate_id: candidate_id.to_string(), + status_after: "Withdrawn".into(), + withdrawn_at, + retention_until, + audit_row_hmac, + }) +} + +// ─── Stats endpoint (ops dashboard backend) ────────────────────── +// +// Read-only aggregate over all subjects. Returns counts by biometric +// + subject status, total photo count, oldest active retention_until, +// and the last 20 audit events across all subjects (consent_grant / +// biometric_collection / consent_withdrawal / erasure rows). Used by +// the /biometric/dashboard ops UI page. +// +// Auth: legal-tier (same posture as /audit/subject). The aggregate +// numbers leak nothing per-subject by themselves, but the "recent +// events" list contains candidate_ids — counsel-tier protection +// is appropriate. + +const STATS_RESPONSE_SCHEMA: &str = "biometric_stats_response.v1"; +const STATS_RECENT_EVENTS_LIMIT: usize = 20; + +#[derive(Serialize, Debug, Default)] +pub struct StatsCounts { + pub never_collected: usize, + pub pending: usize, + pub given: usize, + pub withdrawn: usize, + pub expired: usize, +} + +#[derive(Serialize, Debug, Default)] +pub struct SubjectStatusCounts { + pub pending_consent: usize, + pub active: usize, + pub withdrawn: usize, + pub retention_expired: usize, + pub erased: usize, +} + +#[derive(Serialize, Debug)] +pub struct StatsEvent { + pub ts: chrono::DateTime, + pub candidate_id: String, + pub kind: String, + pub result: String, + pub trace_id: String, +} + +#[derive(Serialize, Debug)] +pub struct StatsResponse { + pub schema: &'static str, + pub generated_at: chrono::DateTime, + pub total_subjects: usize, + pub biometric_status: StatsCounts, + pub subject_status: SubjectStatusCounts, + pub photos_on_record: usize, + pub oldest_active_retention_until: Option>, + pub upcoming_destruction_window_days: Option, + pub recent_events: Vec, + pub recent_event_counts: std::collections::BTreeMap, +} + +async fn stats_handler( + State(state): State, + headers: HeaderMap, +) -> impl IntoResponse { + let auth_token = headers + .get(LEGAL_TOKEN_HEADER) + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + match process_stats(&state, auth_token.as_deref()).await { + Ok(resp) => (StatusCode::OK, Json(resp)).into_response(), + Err((status, err)) => (status, Json(err)).into_response(), + } +} + +pub async fn process_stats( + state: &BiometricEndpointState, + legal_token: Option<&str>, +) -> Result { + // Auth (same posture as upload/erase/consent/withdraw). + let configured = state.legal_token.as_ref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + ErrorResponse { error: "auth_failed", detail: "no legal token configured".into(), consent_status: None }, + ))?; + let provided = legal_token.ok_or(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "missing X-Lakehouse-Legal-Token".into(), consent_status: None }, + ))?; + if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) { + return Err(( + StatusCode::UNAUTHORIZED, + ErrorResponse { error: "auth_failed", detail: "X-Lakehouse-Legal-Token mismatch".into(), consent_status: None }, + )); + } + + use shared::types::{BiometricConsentStatus, SubjectStatus}; + + let now = chrono::Utc::now(); + let subjects = state.registry.list_subjects().await; + + let mut bio = StatsCounts::default(); + let mut subj = SubjectStatusCounts::default(); + let mut photos_on_record = 0usize; + let mut oldest_active_retention: Option> = None; + + for s in &subjects { + match s.consent.biometric.status { + BiometricConsentStatus::NeverCollected => bio.never_collected += 1, + BiometricConsentStatus::Pending => bio.pending += 1, + BiometricConsentStatus::Given => bio.given += 1, + BiometricConsentStatus::Withdrawn => bio.withdrawn += 1, + BiometricConsentStatus::Expired => bio.expired += 1, + } + match s.status { + SubjectStatus::PendingConsent => subj.pending_consent += 1, + SubjectStatus::Active => subj.active += 1, + SubjectStatus::Withdrawn => subj.withdrawn += 1, + SubjectStatus::RetentionExpired => subj.retention_expired += 1, + SubjectStatus::Erased => subj.erased += 1, + } + if s.biometric_collection.is_some() { + photos_on_record += 1; + } + // Oldest-active = earliest retention_until among Given subjects + // (these are the subjects whose data is currently retained + // under canonical consent — Withdrawn subjects also have + // retention_until set but as a 30-day countdown to destruction, + // surfaced separately). + if matches!(s.consent.biometric.status, BiometricConsentStatus::Given) { + if let Some(t) = s.consent.biometric.retention_until { + oldest_active_retention = Some(match oldest_active_retention { + Some(prev) if prev < t => prev, + _ => t, + }); + } + } + } + + let upcoming_destruction_window_days = oldest_active_retention.map(|t| (t - now).num_days()); + + // Walk every subject's audit log; collect erasure / consent / + // upload / withdrawal rows. Sort by ts desc, keep first + // STATS_RECENT_EVENTS_LIMIT. For 100 subjects this is cheap; + // future-proofing against 100k would want a separate event-stream + // index, out of scope for v1. + let mut events: Vec = Vec::new(); + let mut event_counts: std::collections::BTreeMap = Default::default(); + for s in &subjects { + let rows = state + .writer + .read_rows_in_range(&s.candidate_id, None, None) + .await + .unwrap_or_default(); + for r in rows { + // Filter: only state-changing events (skip validator_lookup + // / gateway_lookup / etc. — they're noise for an ops dash). + match r.accessor.kind.as_str() { + "biometric_consent_grant" + | "biometric_collection" + | "biometric_consent_withdrawal" + | "biometric_erasure" + | "full_erasure" => {} + _ => continue, + } + *event_counts.entry(r.accessor.kind.clone()).or_insert(0) += 1; + events.push(StatsEvent { + ts: r.ts, + candidate_id: r.candidate_id.clone(), + kind: r.accessor.kind.clone(), + result: r.result.clone(), + trace_id: r.accessor.trace_id.clone(), + }); + } + } + events.sort_by(|a, b| b.ts.cmp(&a.ts)); + events.truncate(STATS_RECENT_EVENTS_LIMIT); + + Ok(StatsResponse { + schema: STATS_RESPONSE_SCHEMA, + generated_at: now, + total_subjects: subjects.len(), + biometric_status: bio, + subject_status: subj, + photos_on_record, + oldest_active_retention_until: oldest_active_retention, + upcoming_destruction_window_days, + recent_events: events, + recent_event_counts: event_counts, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use object_store::memory::InMemory; + use object_store::ObjectStore; + use shared::types::{ + BiometricConsent, BiometricConsentStatus, ConsentStatus, GeneralPiiConsent, + SubjectConsent, SubjectManifest, SubjectRetention, SubjectStatus, SubjectVertical, + }; + + const TEST_TOKEN: &str = "0123456789abcdef0123456789abcdef"; + + /// Each test gets its own filesystem-backed registry + InMemory + /// audit-store (audit writer is decoupled from registry storage). + /// The storage_root is a per-test tmp dir under env temp_dir so we + /// don't pollute a shared location. + async fn fixture_state(test_name: &str) -> BiometricEndpointState { + let tmp_root = std::env::temp_dir().join(format!( + "biometric_test_{test_name}_{}_{}", + std::process::id(), + uuid::Uuid::new_v4().simple(), + )); + let registry_root = tmp_root.join("registry"); + let storage_root = tmp_root.join("biometric").join("uploads"); + std::fs::create_dir_all(®istry_root).unwrap(); + let store = storaged::backend::init_local(registry_root.to_str().unwrap()); + let registry = Registry::new(store); + let audit_store: Arc = Arc::new(InMemory::new()); + let key: Vec = (0u8..32).collect(); + let writer = Arc::new(SubjectAuditWriter::with_inline_key(audit_store, key)); + BiometricEndpointState { + registry, + writer, + legal_token: Some(Arc::new(TEST_TOKEN.into())), + storage_root, + allowed_consent_versions: None, + } + } + + fn fixture_manifest(id: &str, biometric_status: BiometricConsentStatus, status: SubjectStatus) -> SubjectManifest { + let now = chrono::Utc::now(); + SubjectManifest { + schema: "subject_manifest.v1".into(), + candidate_id: id.into(), + created_at: now, + updated_at: now, + status, + vertical: SubjectVertical::General, + consent: SubjectConsent { + general_pii: GeneralPiiConsent { + status: ConsentStatus::Given, + version: "v1".into(), + given_at: Some(now), + withdrawn_at: None, + }, + biometric: BiometricConsent { + status: biometric_status, + retention_until: Some(now + chrono::Duration::days(540)), + }, + }, + retention: SubjectRetention { + general_pii_until: now + chrono::Duration::days(1460), + policy: "test".into(), + }, + datasets: vec![], + safe_views: vec![], + audit_log_path: String::new(), + audit_log_chain_root: String::new(), + biometric_collection: None, + } + } + + fn jpeg_bytes() -> Vec { + // Minimal JPEG SOI + EOI markers — enough to satisfy the + // "non-empty body with image content-type" check. + vec![0xff, 0xd8, 0xff, 0xd9] + } + + #[tokio::test] + async fn missing_token_rejected() { + let state = fixture_state("missing_token").await; + let err = process_upload(&state, "WORKER-1", None, Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + assert_eq!(err.1.error, "auth_failed"); + } + + #[tokio::test] + async fn wrong_token_rejected() { + let state = fixture_state("wrong_token").await; + let err = process_upload(&state, "WORKER-1", Some("badtoken"), Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + } + + #[tokio::test] + async fn wrong_content_type_rejected() { + let state = fixture_state("wrong_ct").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-1", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let err = process_upload(&state, "WORKER-1", Some(TEST_TOKEN), Some("application/octet-stream"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNSUPPORTED_MEDIA_TYPE); + } + + #[tokio::test] + async fn missing_subject_returns_404() { + let state = fixture_state("missing_subject").await; + let err = process_upload(&state, "NEVER-EXISTED", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::NOT_FOUND); + assert_eq!(err.1.error, "subject_not_found"); + } + + #[tokio::test] + async fn consent_pending_refuses_403() { + let state = fixture_state("consent_pending").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-2", BiometricConsentStatus::Pending, SubjectStatus::Active)).await; + let err = process_upload(&state, "WORKER-2", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::FORBIDDEN); + assert_eq!(err.1.error, "consent_required"); + assert_eq!(err.1.consent_status.as_deref(), Some("Pending")); + } + + #[tokio::test] + async fn consent_withdrawn_refuses_403() { + let state = fixture_state("consent_withdrawn").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-3", BiometricConsentStatus::Withdrawn, SubjectStatus::Active)).await; + let err = process_upload(&state, "WORKER-3", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::FORBIDDEN); + assert_eq!(err.1.consent_status.as_deref(), Some("Withdrawn")); + } + + #[tokio::test] + async fn subject_erased_refuses_even_with_consent_given() { + // Defense-in-depth: if a subject was erased post-consent (BIPA + // RTBF flow), the manifest's biometric.status may still read + // "given" momentarily before the erase flow flips it. The + // subject-status gate refuses uploads on Erased / Withdrawn / + // RetentionExpired regardless of the biometric consent state. + let state = fixture_state("subject_erased").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E", BiometricConsentStatus::Given, SubjectStatus::Erased)).await; + let err = process_upload(&state, "WORKER-E", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()).await.unwrap_err(); + assert_eq!(err.0, StatusCode::FORBIDDEN); + assert_eq!(err.1.error, "subject_inactive"); + } + + #[tokio::test] + async fn happy_path_writes_file_manifest_and_audit() { + let state = fixture_state("happy_path").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-4", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + let registry = state.registry.clone(); + let storage_root = state.storage_root.clone(); + + let resp = process_upload(&state, "WORKER-4", Some(TEST_TOKEN), Some("image/jpeg"), "consent-v1-hash", "trace-xyz", &jpeg_bytes()) + .await.unwrap(); + assert_eq!(resp.schema, "biometric_photo_response.v1"); + assert_eq!(resp.candidate_id, "WORKER-4"); + // data_path is relative to storage_root per BiometricCollection doc. + assert!(!resp.data_path.starts_with('/'), "data_path should be relative, got: {}", resp.data_path); + assert!(resp.data_path.starts_with("WORKER-4/"), + "data_path {} should start with subject directory", resp.data_path); + assert_eq!(resp.template_hash.len(), 64); + assert_eq!(resp.consent_version_hash, "consent-v1-hash"); + assert!(!resp.audit_row_hmac.is_empty()); + + // File at storage_root.join(data_path) has the bytes we sent. + let abs = storage_root.join(&resp.data_path); + let on_disk = tokio::fs::read(&abs).await.unwrap(); + assert_eq!(on_disk, jpeg_bytes()); + + // Manifest reflects the new collection record. + let updated = registry.get_subject("WORKER-4").await.unwrap(); + let bc = updated.biometric_collection.as_ref().expect("collection should be set"); + assert_eq!(bc.data_path, resp.data_path); + assert_eq!(bc.template_hash.len(), 64); + assert_eq!(bc.consent_version_hash, "consent-v1-hash"); + + // Audit log has one row, chain verifies, accessor.kind is right. + assert_eq!(writer.verify_chain("WORKER-4").await.unwrap(), 1); + let rows = writer.read_rows_in_range("WORKER-4", None, None).await.unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].accessor.kind, "biometric_collection"); + assert_eq!(rows[0].accessor.purpose, "photo_upload"); + assert_eq!(rows[0].accessor.trace_id, "trace-xyz"); + } + + #[tokio::test] + async fn second_upload_without_erase_returns_409() { + // BIPA defensibility: a second upload to a subject that already + // has a BiometricCollection must fail-closed. Without this gate, + // the second upload silently overwrites manifest.data_path and + // orphans the first photo on disk forever (caught 2026-05-05 on + // WORKER-2 by verify_biometric_erasure.sh). + let state = fixture_state("second_upload_409").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-DUP", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let storage_root = state.storage_root.clone(); + let registry = state.registry.clone(); + + // First upload succeeds. + let resp1 = process_upload(&state, "WORKER-DUP", Some(TEST_TOKEN), Some("image/jpeg"), "v1", "", &jpeg_bytes()) + .await.unwrap(); + let first_path = storage_root.join(&resp1.data_path); + assert!(first_path.exists(), "first upload should produce a file"); + + // Second upload refused with 409. + let err = process_upload(&state, "WORKER-DUP", Some(TEST_TOKEN), Some("image/jpeg"), "v1", "", &jpeg_bytes()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "biometric_already_collected"); + + // Manifest still points at the first upload — pointer was NOT overwritten. + let m = registry.get_subject("WORKER-DUP").await.unwrap(); + let bc = m.biometric_collection.as_ref().expect("collection should still be set"); + assert_eq!(bc.data_path, resp1.data_path, + "manifest data_path must be unchanged after refused second upload"); + + // First file remains on disk untouched (refusal must not unlink it). + assert!(first_path.exists(), "first upload's file must remain after refused second upload"); + let still_on_disk = tokio::fs::read(&first_path).await.unwrap(); + assert_eq!(still_on_disk, jpeg_bytes(), + "first upload's bytes must not have been overwritten"); + } + + #[tokio::test] + async fn upload_erase_upload_grows_the_chain_cleanly() { + // Prior version of this test allowed repeated uploads to chain; + // that conflated chain growth with allowed re-upload. Under the + // double-upload guard (409 above), the only legitimate way to + // re-collect is upload → erase → upload. Chain grows to 3 rows + // (collection, erasure, collection); on-disk file count returns + // to one after the second upload. + let state = fixture_state("upload_erase_upload").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-CYCLE", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + let storage_root = state.storage_root.clone(); + + // First upload. + let resp1 = process_upload(&state, "WORKER-CYCLE", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()) + .await.unwrap(); + let first_path = storage_root.join(&resp1.data_path); + assert!(first_path.exists()); + + // Erase. Uses process_erase test helper (the production path + // parses the EraseRequest from request body; tests inject it + // directly). Note: the erase flow flips biometric.status to + // Withdrawn, so the post-erase second upload must reset consent + // first (production flow would require new consent collection). + let _ = process_erase(&state, "WORKER-CYCLE", Some(TEST_TOKEN), "trace-cycle", fixture_erase_request("biometric_only")) + .await.unwrap(); + assert!(!first_path.exists(), "first photo file must be unlinked by erase"); + + // Reset consent + status on the post-erase manifest so the second + // upload can proceed (production flow would require new consent + // collection here; for this test we directly flip the manifest). + let mut post_erase = state.registry.get_subject("WORKER-CYCLE").await.unwrap(); + post_erase.consent.biometric.status = BiometricConsentStatus::Given; + post_erase.status = SubjectStatus::Active; + post_erase.biometric_collection = None; + let _ = state.registry.put_subject(post_erase).await; + + // Second upload (legitimate, after erase). + let resp2 = process_upload(&state, "WORKER-CYCLE", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()) + .await.unwrap(); + let second_path = storage_root.join(&resp2.data_path); + assert!(second_path.exists(), "second upload should produce a file"); + assert_ne!(resp1.data_path, resp2.data_path, "second upload should land at a new path"); + + // Chain has 3 rows: collection, erasure, collection. + assert_eq!(writer.verify_chain("WORKER-CYCLE").await.unwrap(), 3); + let rows = writer.read_rows_in_range("WORKER-CYCLE", None, None).await.unwrap(); + assert_eq!(rows[0].accessor.kind, "biometric_collection"); + assert_eq!(rows[1].accessor.kind, "biometric_erasure"); + assert_eq!(rows[2].accessor.kind, "biometric_collection"); + } + + #[tokio::test] + async fn quarantine_path_is_not_data_headshots() { + // Defensive structural test: make sure the storage_root does + // NOT live under data/headshots/. Any future change that + // accidentally pointed real-photo storage at the synthetic + // headshot dir would conflate the two surfaces — exactly the + // confusion Phase 1.6 Gate 3 exists to prevent. + let state = fixture_state("quarantine_isolation").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-Q", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let resp = process_upload(&state, "WORKER-Q", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()) + .await.unwrap(); + let storage = state.storage_root.to_string_lossy().to_string(); + assert!(!storage.contains("/headshots/"), + "storage_root {} must not be under /headshots/ (synthetic-only surface)", storage); + assert!(storage.contains("/biometric/uploads"), + "storage_root {} should be under /biometric/uploads", storage); + // data_path is the relative form: /.. + assert!(resp.data_path.starts_with("WORKER-Q/"), + "data_path {} should be subject-relative", resp.data_path); + } + + #[tokio::test] + async fn content_type_with_parameters_accepted() { + // Real-world clients send `image/jpeg; charset=binary` etc. + // Regression test ensures the bare media type is matched + // after stripping parameters. + let state = fixture_state("ct_with_params").await; + // Two distinct subjects so each upload exercises the "first upload" + // path. Prior version used one subject and two uploads — now blocked + // by the double-upload guard (409). The test's actual intent is + // content-type parsing, not re-upload tolerance. + let _ = state.registry.put_subject(fixture_manifest("WORKER-CT-A", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-CT-B", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let resp = process_upload( + &state, "WORKER-CT-A", Some(TEST_TOKEN), + Some("image/jpeg; charset=binary"), "", "", &jpeg_bytes(), + ).await.unwrap(); + assert_eq!(resp.candidate_id, "WORKER-CT-A"); + // Also case-insensitive matching: "Image/JPEG" should work too. + let resp2 = process_upload( + &state, "WORKER-CT-B", Some(TEST_TOKEN), + Some("Image/JPEG"), "", "", &jpeg_bytes(), + ).await.unwrap(); + assert_eq!(resp2.candidate_id, "WORKER-CT-B"); + } + + // ─── Erasure tests (Gate 5) ────────────────────────────────────── + + fn fixture_erase_request(scope: &str) -> EraseRequest { + EraseRequest { + trigger: "consent_withdrawal".into(), + trigger_evidence_path: "/tmp/withdrawal_receipt.pdf".into(), + operator_of_record: "J".into(), + witness: "Witness A".into(), + scope: scope.into(), + } + } + + #[tokio::test] + async fn erase_missing_token_rejected() { + let state = fixture_state("erase_no_token").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E1", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let err = process_erase(&state, "WORKER-E1", None, "", fixture_erase_request("biometric_only")).await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + } + + #[tokio::test] + async fn erase_missing_subject_returns_404() { + let state = fixture_state("erase_no_subject").await; + let err = process_erase(&state, "GHOST", Some(TEST_TOKEN), "", fixture_erase_request("biometric_only")).await.unwrap_err(); + assert_eq!(err.0, StatusCode::NOT_FOUND); + } + + #[tokio::test] + async fn erase_missing_trigger_400() { + let state = fixture_state("erase_no_trigger").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E2", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let mut req = fixture_erase_request("biometric_only"); + req.trigger = "".into(); + let err = process_erase(&state, "WORKER-E2", Some(TEST_TOKEN), "", req).await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("trigger")); + } + + #[tokio::test] + async fn erase_missing_operator_400() { + let state = fixture_state("erase_no_op").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E3", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let mut req = fixture_erase_request("biometric_only"); + req.operator_of_record = "".into(); + let err = process_erase(&state, "WORKER-E3", Some(TEST_TOKEN), "", req).await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("operator")); + } + + #[tokio::test] + async fn erase_invalid_scope_400() { + let state = fixture_state("erase_bad_scope").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E4", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let req = fixture_erase_request("partial"); // not a valid scope + let err = process_erase(&state, "WORKER-E4", Some(TEST_TOKEN), "", req).await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("scope")); + } + + #[tokio::test] + async fn erase_biometric_only_clears_collection_and_unlinks_photo() { + let state = fixture_state("erase_bio_happy").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E5", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + // Upload first so there's something to erase. + let upload = process_upload(&state, "WORKER-E5", Some(TEST_TOKEN), Some("image/jpeg"), "consent-v1", "", &jpeg_bytes()) + .await.unwrap(); + let abs = state.storage_root.join(&upload.data_path); + assert!(abs.exists(), "photo should exist before erase"); + + let registry = state.registry.clone(); + let writer = state.writer.clone(); + let resp = process_erase(&state, "WORKER-E5", Some(TEST_TOKEN), "trace-erase", fixture_erase_request("biometric_only")) + .await.unwrap(); + assert_eq!(resp.scope, "biometric_only"); + assert_eq!(resp.trigger, "consent_withdrawal"); + assert!(resp.fields_cleared.contains(&"biometric_collection".to_string())); + assert!(resp.fields_cleared.contains(&"consent.biometric".to_string())); + assert!(resp.photo_unlinked); + assert_eq!(resp.biometric_status_after, "Withdrawn"); + // biometric_only does NOT touch general status / general_pii. + assert_eq!(resp.status_after, "Active"); + assert_eq!(resp.general_pii_status_after, "Given"); + assert!(!resp.audit_row_hmac.is_empty()); + + // Photo file is gone. + assert!(!abs.exists(), "photo should be unlinked after erase"); + + // Manifest reflects clearing. + let updated = registry.get_subject("WORKER-E5").await.unwrap(); + assert!(updated.biometric_collection.is_none()); + assert_eq!(updated.consent.biometric.status, BiometricConsentStatus::Withdrawn); + assert_eq!(updated.status, SubjectStatus::Active); + + // Audit chain has 2 rows: collection + erasure. Chain verifies. + assert_eq!(writer.verify_chain("WORKER-E5").await.unwrap(), 2); + let rows = writer.read_rows_in_range("WORKER-E5", None, None).await.unwrap(); + assert_eq!(rows.last().unwrap().accessor.kind, "biometric_erasure"); + assert!(rows.last().unwrap().accessor.purpose.contains("trigger=consent_withdrawal")); + assert!(rows.last().unwrap().accessor.purpose.contains("operator=J")); + assert_eq!(rows.last().unwrap().result, "erased"); + } + + #[tokio::test] + async fn erase_full_marks_subject_erased_and_withdraws_general_pii() { + let state = fixture_state("erase_full_happy").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E6", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let registry = state.registry.clone(); + let writer = state.writer.clone(); + let resp = process_erase(&state, "WORKER-E6", Some(TEST_TOKEN), "", fixture_erase_request("full")) + .await.unwrap(); + assert_eq!(resp.scope, "full"); + assert_eq!(resp.status_after, "Erased"); + assert_eq!(resp.biometric_status_after, "Withdrawn"); + assert_eq!(resp.general_pii_status_after, "Withdrawn"); + assert!(resp.fields_cleared.contains(&"status".to_string())); + assert!(resp.fields_cleared.contains(&"consent.general_pii".to_string())); + + let updated = registry.get_subject("WORKER-E6").await.unwrap(); + assert_eq!(updated.status, SubjectStatus::Erased); + assert_eq!(updated.consent.general_pii.status, ConsentStatus::Withdrawn); + assert!(updated.consent.general_pii.withdrawn_at.is_some()); + assert_eq!(writer.verify_chain("WORKER-E6").await.unwrap(), 1); + let rows = writer.read_rows_in_range("WORKER-E6", None, None).await.unwrap(); + assert_eq!(rows[0].accessor.kind, "full_erasure"); + } + + #[tokio::test] + async fn erase_idempotent_on_already_erased() { + // Erasing an already-erased subject still writes an audit row + // (operational signal) but doesn't 500. + let state = fixture_state("erase_idempotent").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E7", BiometricConsentStatus::Withdrawn, SubjectStatus::Erased)).await; + let writer = state.writer.clone(); + let resp = process_erase(&state, "WORKER-E7", Some(TEST_TOKEN), "", fixture_erase_request("full")) + .await.unwrap(); + assert_eq!(resp.status_after, "Erased"); + let rows = writer.read_rows_in_range("WORKER-E7", None, None).await.unwrap(); + assert_eq!(rows[0].result, "already_erased"); + } + + #[tokio::test] + async fn erase_with_no_photo_succeeds_without_unlink_error() { + // Subject has biometric consent but never uploaded a photo; + // erasure should still succeed (idempotent, photo_unlinked stays + // true because there's nothing to unlink). + let state = fixture_state("erase_no_photo").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E8", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let resp = process_erase(&state, "WORKER-E8", Some(TEST_TOKEN), "", fixture_erase_request("biometric_only")) + .await.unwrap(); + assert!(resp.photo_unlinked); + assert!(resp.photo_unlink_error.is_none()); + assert_eq!(resp.biometric_status_after, "Withdrawn"); + } + + #[tokio::test] + async fn erase_chain_links_off_prior_audit_row() { + // Critical defensive test: erasure audit row's prev_chain_hash + // must point to the prior row's row_hmac, NOT GENESIS. + // Without this, the chain breaks at the erasure row. + let state = fixture_state("erase_chain_link").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-E9", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let upload = process_upload(&state, "WORKER-E9", Some(TEST_TOKEN), Some("image/jpeg"), "", "", &jpeg_bytes()) + .await.unwrap(); + let writer = state.writer.clone(); + let _ = process_erase(&state, "WORKER-E9", Some(TEST_TOKEN), "", fixture_erase_request("biometric_only")) + .await.unwrap(); + + let rows = writer.read_rows_in_range("WORKER-E9", None, None).await.unwrap(); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0].accessor.kind, "biometric_collection"); + assert_eq!(rows[1].accessor.kind, "biometric_erasure"); + // Erasure row chains off the upload's hmac. + assert_eq!(rows[1].prev_chain_hash, upload.audit_row_hmac); + assert_eq!(writer.verify_chain("WORKER-E9").await.unwrap(), 2); + } + + // ─── Consent recording tests (Gate 2 backend) ────────────────── + + fn fixture_consent_request() -> ConsentRequest { + ConsentRequest { + consent_version_hash: "abcdef0123456789".repeat(4), // 64-char fake SHA-256 + consent_collection_method: "electronic_signature".into(), + consent_collection_evidence_path: "/tmp/sig_receipt_WORKER-X.pdf".into(), + operator_of_record: "OperatorA".into(), + } + } + + #[tokio::test] + async fn consent_happy_path_flips_status_and_records_audit() { + // Subject starts at NeverCollected. Consent grant flips to Given, + // sets retention_until = given_at + 18 months, appends a + // biometric_consent_grant audit row, returns 200 with the grant + // metadata + chained hmac. + let state = fixture_state("consent_happy").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C1", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + let registry = state.registry.clone(); + + let resp = process_consent(&state, "WORKER-C1", Some(TEST_TOKEN), "trace-c1", fixture_consent_request()) + .await.unwrap(); + + assert_eq!(resp.schema, "biometric_consent_response.v1"); + assert_eq!(resp.candidate_id, "WORKER-C1"); + assert_eq!(resp.status_after, "Given"); + assert_eq!(resp.consent_version_hash.len(), 64); + assert!(!resp.audit_row_hmac.is_empty()); + // retention_until is given_at + 18 months (540 days). + let elapsed = resp.retention_until - resp.given_at; + assert_eq!(elapsed.num_days(), CONSENT_RETENTION_DAYS); + + // Manifest reflects the grant. Note: registry::put_subject + // assigns its OWN `updated_at = Utc::now()`, so the manifest's + // updated_at lands a few microseconds after the response's + // given_at — assert ordering, not equality. + let m = registry.get_subject("WORKER-C1").await.unwrap(); + assert_eq!(m.consent.biometric.status, BiometricConsentStatus::Given); + assert_eq!(m.consent.biometric.retention_until, Some(resp.retention_until)); + assert!(m.updated_at >= resp.given_at, + "manifest.updated_at ({}) should be >= response.given_at ({})", m.updated_at, resp.given_at); + + // Audit row is correctly shaped. + assert_eq!(writer.verify_chain("WORKER-C1").await.unwrap(), 1); + let rows = writer.read_rows_in_range("WORKER-C1", None, None).await.unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].accessor.kind, "biometric_consent_grant"); + assert!(rows[0].accessor.purpose.contains("version=")); + assert!(rows[0].accessor.purpose.contains("method=electronic_signature")); + assert!(rows[0].accessor.purpose.contains("operator=OperatorA")); + assert_eq!(rows[0].accessor.trace_id, "trace-c1"); + assert_eq!(rows[0].fields_accessed, vec!["consent.biometric.status".to_string(), "consent.biometric.retention_until".to_string()]); + assert_eq!(rows[0].result, "given"); + } + + #[tokio::test] + async fn consent_missing_token_rejected() { + let state = fixture_state("consent_no_token").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C2", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-C2", None, "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + assert_eq!(err.1.error, "auth_failed"); + } + + #[tokio::test] + async fn consent_wrong_token_rejected() { + let state = fixture_state("consent_wrong_token").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C3", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-C3", Some("nope-not-the-token"), "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + } + + #[tokio::test] + async fn consent_subject_not_found_returns_404() { + let state = fixture_state("consent_no_subject").await; + let err = process_consent(&state, "GHOST-WORKER", Some(TEST_TOKEN), "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::NOT_FOUND); + assert_eq!(err.1.error, "subject_not_found"); + } + + #[tokio::test] + async fn consent_missing_version_hash_400() { + let state = fixture_state("consent_no_version").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C4", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let mut req = fixture_consent_request(); + req.consent_version_hash = " ".into(); + let err = process_consent(&state, "WORKER-C4", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("consent_version_hash")); + } + + #[tokio::test] + async fn consent_missing_method_400() { + let state = fixture_state("consent_no_method").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C5", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let mut req = fixture_consent_request(); + req.consent_collection_method = "".into(); + let err = process_consent(&state, "WORKER-C5", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("consent_collection_method")); + } + + #[tokio::test] + async fn consent_invalid_method_400() { + // Methods are constrained to a closed set so operator typos + // can't silently weaken evidentiary defensibility (e.g. + // "electric_signature" vs "electronic_signature"). + let state = fixture_state("consent_bad_method").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C6", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let mut req = fixture_consent_request(); + req.consent_collection_method = "carrier_pigeon".into(); + let err = process_consent(&state, "WORKER-C6", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("must be one of")); + assert!(err.1.detail.contains("carrier_pigeon")); + } + + #[tokio::test] + async fn consent_missing_operator_400() { + let state = fixture_state("consent_no_op").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C7", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let mut req = fixture_consent_request(); + req.operator_of_record = "".into(); + let err = process_consent(&state, "WORKER-C7", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("operator_of_record")); + } + + #[tokio::test] + async fn consent_already_given_returns_409() { + // Re-granting consent when status is already Given is refused. + // The expected flow for re-collection is: erase → grant fresh + // consent under the new template version. This forces the + // erase audit row to land before any new grant, preserving + // chain ordering. + let state = fixture_state("consent_already").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C8", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-C8", Some(TEST_TOKEN), "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "consent_already_given"); + assert_eq!(err.1.consent_status.as_deref(), Some("Given")); + } + + #[tokio::test] + async fn consent_post_withdrawal_requires_erase_returns_409() { + // After Withdrawn, fresh consent requires explicit erase first. + // Otherwise the audit chain wouldn't capture the cycle of + // withdraw → erase → grant; the grant would land directly on + // a manifest that still references the prior collection. + let state = fixture_state("consent_after_withdraw").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C9", BiometricConsentStatus::Withdrawn, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-C9", Some(TEST_TOKEN), "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "consent_post_withdrawal_requires_erase"); + assert_eq!(err.1.consent_status.as_deref(), Some("Withdrawn")); + } + + #[tokio::test] + async fn consent_subject_inactive_returns_403() { + // Subject status takes precedence over consent status — an + // Erased subject can't be re-consented even if biometric.status + // happens to be NeverCollected (which it would be post-erasure). + let state = fixture_state("consent_inactive").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C10", BiometricConsentStatus::NeverCollected, SubjectStatus::Erased)).await; + let err = process_consent(&state, "WORKER-C10", Some(TEST_TOKEN), "", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::FORBIDDEN); + assert_eq!(err.1.error, "subject_inactive"); + } + + #[tokio::test] + async fn consent_grant_then_upload_is_the_intended_intake_flow() { + // End-to-end check of the post-cutover flow: candidate signs + // consent (process_consent flips status to Given) → operator + // uploads photo (process_upload now passes the consent gate). + // Audit chain captures: consent_grant → biometric_collection. + let state = fixture_state("consent_then_upload").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-C11", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + + // 1. Consent grant. + let grant = process_consent(&state, "WORKER-C11", Some(TEST_TOKEN), "trace-grant", fixture_consent_request()) + .await.unwrap(); + + // 2. Photo upload — should succeed now. + let upload = process_upload(&state, "WORKER-C11", Some(TEST_TOKEN), Some("image/jpeg"), &grant.consent_version_hash, "trace-upload", &jpeg_bytes()) + .await.unwrap(); + assert_eq!(upload.consent_version_hash, grant.consent_version_hash); + + // Audit chain has 2 rows in the right order. + assert_eq!(writer.verify_chain("WORKER-C11").await.unwrap(), 2); + let rows = writer.read_rows_in_range("WORKER-C11", None, None).await.unwrap(); + assert_eq!(rows[0].accessor.kind, "biometric_consent_grant"); + assert_eq!(rows[1].accessor.kind, "biometric_collection"); + // Upload row chains off the consent grant's hmac. + assert_eq!(rows[1].prev_chain_hash, grant.audit_row_hmac); + } + + // ─── Consent withdrawal tests (BIPA right of withdrawal) ─────── + + fn fixture_withdraw_request() -> WithdrawRequest { + WithdrawRequest { + reason: "candidate emailed asking to remove their photo".into(), + operator_of_record: "OperatorB".into(), + evidence_path: "/tmp/withdrawal_email_thread_WORKER-X.eml".into(), + } + } + + #[tokio::test] + async fn withdraw_happy_path_flips_status_and_starts_30day_clock() { + // Subject at consent.biometric.status=Given, retention_until set + // to ~18 months out. Withdrawal flips status to Withdrawn AND + // accelerates retention_until to 30 days from now (per consent + // template v1 §2 SLA), appends biometric_consent_withdrawal + // audit row, returns 200. + let state = fixture_state("withdraw_happy").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W1", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + let registry = state.registry.clone(); + + let resp = process_withdraw(&state, "WORKER-W1", Some(TEST_TOKEN), "trace-w1", fixture_withdraw_request()) + .await.unwrap(); + + assert_eq!(resp.schema, "biometric_withdraw_response.v1"); + assert_eq!(resp.candidate_id, "WORKER-W1"); + assert_eq!(resp.status_after, "Withdrawn"); + assert!(!resp.audit_row_hmac.is_empty()); + // 30-day clock from withdrawn_at to retention_until. + let elapsed = resp.retention_until - resp.withdrawn_at; + assert_eq!(elapsed.num_days(), WITHDRAW_SLA_DAYS); + + // Manifest reflects the withdrawal. + let m = registry.get_subject("WORKER-W1").await.unwrap(); + assert_eq!(m.consent.biometric.status, BiometricConsentStatus::Withdrawn); + assert_eq!(m.consent.biometric.retention_until, Some(resp.retention_until)); + // general_pii NOT touched — withdrawal is biometric-scoped only. + // (default fixture has NeverCollected on biometric, but this + // assertion would catch a bug where withdraw mutated + // general_pii.status alongside biometric.) + assert_eq!(m.status, SubjectStatus::Active); + + // Audit row. + assert_eq!(writer.verify_chain("WORKER-W1").await.unwrap(), 1); + let rows = writer.read_rows_in_range("WORKER-W1", None, None).await.unwrap(); + assert_eq!(rows[0].accessor.kind, "biometric_consent_withdrawal"); + assert!(rows[0].accessor.purpose.contains("reason=")); + assert!(rows[0].accessor.purpose.contains("operator=OperatorB")); + assert_eq!(rows[0].accessor.trace_id, "trace-w1"); + assert_eq!(rows[0].fields_accessed, vec!["consent.biometric.status".to_string(), "consent.biometric.retention_until".to_string()]); + assert_eq!(rows[0].result, "withdrawn"); + } + + #[tokio::test] + async fn withdraw_missing_token_rejected() { + let state = fixture_state("withdraw_no_token").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W2", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W2", None, "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + assert_eq!(err.1.error, "auth_failed"); + } + + #[tokio::test] + async fn withdraw_wrong_token_rejected() { + let state = fixture_state("withdraw_wrong_token").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W3", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W3", Some("nope"), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + } + + #[tokio::test] + async fn withdraw_subject_not_found_returns_404() { + let state = fixture_state("withdraw_no_subject").await; + let err = process_withdraw(&state, "GHOST-WORKER", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::NOT_FOUND); + assert_eq!(err.1.error, "subject_not_found"); + } + + #[tokio::test] + async fn withdraw_missing_reason_400() { + let state = fixture_state("withdraw_no_reason").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W4", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let mut req = fixture_withdraw_request(); + req.reason = " ".into(); + let err = process_withdraw(&state, "WORKER-W4", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("reason")); + } + + #[tokio::test] + async fn withdraw_missing_operator_400() { + let state = fixture_state("withdraw_no_op").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W5", BiometricConsentStatus::Given, SubjectStatus::Active)).await; + let mut req = fixture_withdraw_request(); + req.operator_of_record = "".into(); + let err = process_withdraw(&state, "WORKER-W5", Some(TEST_TOKEN), "", req) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert!(err.1.detail.contains("operator_of_record")); + } + + #[tokio::test] + async fn withdraw_when_never_collected_returns_409() { + // Cannot withdraw consent that was never given. If consent is + // mid-flow, the consent record should be discarded by the + // intake operator rather than withdrawn. + let state = fixture_state("withdraw_never").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W6", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W6", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "nothing_to_withdraw"); + assert_eq!(err.1.consent_status.as_deref(), Some("NeverCollected")); + } + + #[tokio::test] + async fn withdraw_when_pending_returns_409() { + let state = fixture_state("withdraw_pending").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W7", BiometricConsentStatus::Pending, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W7", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "nothing_to_withdraw"); + } + + #[tokio::test] + async fn withdraw_when_already_withdrawn_returns_409() { + // Idempotency-resilient: refuses re-withdrawal so operator + // doesn't accidentally push retention_until back. Operator + // can confirm prior withdrawal via GET /audit/subject/{id}. + let state = fixture_state("withdraw_already").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W8", BiometricConsentStatus::Withdrawn, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W8", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "already_withdrawn"); + assert_eq!(err.1.consent_status.as_deref(), Some("Withdrawn")); + } + + #[tokio::test] + async fn withdraw_when_expired_returns_409() { + let state = fixture_state("withdraw_expired").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W9", BiometricConsentStatus::Expired, SubjectStatus::Active)).await; + let err = process_withdraw(&state, "WORKER-W9", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::CONFLICT); + assert_eq!(err.1.error, "already_expired"); + } + + #[tokio::test] + async fn withdraw_subject_inactive_returns_403() { + // Subject status takes precedence — withdrawal on Erased + // subject is moot. + let state = fixture_state("withdraw_erased_subject").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W10", BiometricConsentStatus::Given, SubjectStatus::Erased)).await; + let err = process_withdraw(&state, "WORKER-W10", Some(TEST_TOKEN), "", fixture_withdraw_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::FORBIDDEN); + assert_eq!(err.1.error, "subject_inactive"); + } + + #[tokio::test] + async fn withdraw_full_cycle_consent_grant_then_withdraw() { + // End-to-end: consent grant → withdraw. Verifies retention_until + // is correctly accelerated from the 18-month consent default to + // 30 days from withdrawal. Audit chain has 2 rows in correct + // order; withdraw row chains off the grant's hmac. + let state = fixture_state("withdraw_cycle").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-W11", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let writer = state.writer.clone(); + let registry = state.registry.clone(); + + // 1. Grant consent. + let grant = process_consent(&state, "WORKER-W11", Some(TEST_TOKEN), "trace-grant", fixture_consent_request()) + .await.unwrap(); + let grant_retention = grant.retention_until; + + // 2. Withdraw. + let withdraw = process_withdraw(&state, "WORKER-W11", Some(TEST_TOKEN), "trace-withdraw", fixture_withdraw_request()) + .await.unwrap(); + + // retention_until was accelerated — withdraw's retention_until + // must be earlier than the 18-month grant retention. + assert!(withdraw.retention_until < grant_retention, + "withdraw retention {} must be earlier than grant retention {}", + withdraw.retention_until, grant_retention); + + // Manifest reflects the withdrawal. + let m = registry.get_subject("WORKER-W11").await.unwrap(); + assert_eq!(m.consent.biometric.status, BiometricConsentStatus::Withdrawn); + assert_eq!(m.consent.biometric.retention_until, Some(withdraw.retention_until)); + + // Audit chain. + assert_eq!(writer.verify_chain("WORKER-W11").await.unwrap(), 2); + let rows = writer.read_rows_in_range("WORKER-W11", None, None).await.unwrap(); + assert_eq!(rows[0].accessor.kind, "biometric_consent_grant"); + assert_eq!(rows[1].accessor.kind, "biometric_consent_withdrawal"); + // Withdraw row chains off the grant's hmac. + assert_eq!(rows[1].prev_chain_hash, grant.audit_row_hmac); + } + + // ─── Stats endpoint tests ────────────────────────────────────── + + #[tokio::test] + async fn stats_missing_token_rejected() { + let state = fixture_state("stats_no_token").await; + let err = process_stats(&state, None).await.unwrap_err(); + assert_eq!(err.0, StatusCode::UNAUTHORIZED); + } + + #[tokio::test] + async fn stats_aggregates_subjects_by_status() { + let state = fixture_state("stats_aggregate").await; + // Plant a mix of states. + for (id, bs, ss) in [ + ("WORKER-S1", BiometricConsentStatus::NeverCollected, SubjectStatus::Active), + ("WORKER-S2", BiometricConsentStatus::NeverCollected, SubjectStatus::Active), + ("WORKER-S3", BiometricConsentStatus::Given, SubjectStatus::Active), + ("WORKER-S4", BiometricConsentStatus::Given, SubjectStatus::Active), + ("WORKER-S5", BiometricConsentStatus::Given, SubjectStatus::Active), + ("WORKER-S6", BiometricConsentStatus::Withdrawn, SubjectStatus::Active), + ("WORKER-S7", BiometricConsentStatus::Pending, SubjectStatus::PendingConsent), + ("WORKER-S8", BiometricConsentStatus::Expired, SubjectStatus::Erased), + ] { + let _ = state.registry.put_subject(fixture_manifest(id, bs, ss)).await; + } + let stats = process_stats(&state, Some(TEST_TOKEN)).await.unwrap(); + assert_eq!(stats.total_subjects, 8); + assert_eq!(stats.biometric_status.never_collected, 2); + assert_eq!(stats.biometric_status.given, 3); + assert_eq!(stats.biometric_status.withdrawn, 1); + assert_eq!(stats.biometric_status.pending, 1); + assert_eq!(stats.biometric_status.expired, 1); + assert_eq!(stats.subject_status.active, 6); + assert_eq!(stats.subject_status.pending_consent, 1); + assert_eq!(stats.subject_status.erased, 1); + } + + #[tokio::test] + async fn stats_recent_events_filters_state_changes_only() { + // The stats endpoint surfaces only state-changing event kinds + // (consent_grant / collection / withdrawal / erasure) — not + // gateway_lookup / validator_lookup / etc. which are noise + // for an ops dashboard. + let state = fixture_state("stats_event_filter").await; + // Subject must start at NeverCollected so process_consent can flip + // it to Given (otherwise the consent endpoint returns 409 already_given). + let _ = state.registry.put_subject(fixture_manifest("WORKER-EV", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + + // Plant a noisy validator_lookup row + a state-change row. + use shared::types::AuditAccessor; + let noise_row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::Utc::now(), + candidate_id: "WORKER-EV".into(), + accessor: AuditAccessor { + kind: "validator_lookup".into(), + daemon: "gateway".into(), + purpose: "noise".into(), + trace_id: String::new(), + }, + fields_accessed: vec!["exists".into()], + result: "not_found".into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + let _ = state.writer.append(noise_row).await; + // grant a consent (state change) so chain has at least one + // event the dashboard should surface. + let _ = process_consent(&state, "WORKER-EV", Some(TEST_TOKEN), "trace-ev", fixture_consent_request()) + .await.unwrap(); + + let stats = process_stats(&state, Some(TEST_TOKEN)).await.unwrap(); + // recent_events must NOT contain the validator_lookup row. + assert!(stats.recent_events.iter().all(|e| e.kind != "validator_lookup")); + // The consent grant SHOULD be present. + assert!(stats.recent_events.iter().any(|e| e.kind == "biometric_consent_grant" && e.candidate_id == "WORKER-EV")); + // event_counts breakdown must exclude the noise. + assert!(!stats.recent_event_counts.contains_key("validator_lookup")); + assert_eq!(stats.recent_event_counts.get("biometric_consent_grant"), Some(&1)); + } + + // ─── Consent version allowlist tests ────────────────────────── + + async fn fixture_state_with_allowlist(name: &str, hashes: Vec) -> BiometricEndpointState { + let mut s = fixture_state(name).await; + let set: std::collections::HashSet = hashes.into_iter().collect(); + s.allowed_consent_versions = Some(Arc::new(set)); + s + } + + #[tokio::test] + async fn consent_allowlist_accepts_known_hash() { + // Allowlist contains the fixture's consent_version_hash — + // grant succeeds. + let allowed = "abcdef0123456789".repeat(4); // matches fixture_consent_request + let state = fixture_state_with_allowlist("al_known", vec![allowed]).await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-AL1", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let resp = process_consent(&state, "WORKER-AL1", Some(TEST_TOKEN), "trace-al", fixture_consent_request()) + .await.unwrap(); + assert_eq!(resp.status_after, "Given"); + } + + #[tokio::test] + async fn consent_allowlist_rejects_unknown_hash() { + // Allowlist contains a different hash; the fixture's hash + // is rejected with consent_version_unknown. + let other = "0011223344556677".repeat(4); + let state = fixture_state_with_allowlist("al_unknown", vec![other]).await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-AL2", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-AL2", Some(TEST_TOKEN), "trace-al", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert_eq!(err.1.error, "consent_version_unknown"); + } + + #[tokio::test] + async fn consent_allowlist_normalizes_case() { + // Operator might paste an uppercase hash; allowlist comparison + // is lowercase to defeat that drift. + let allowed_lower = "abcdef0123456789".repeat(4); + let state = fixture_state_with_allowlist("al_case", vec![allowed_lower]).await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-AL3", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let mut req = fixture_consent_request(); + req.consent_version_hash = req.consent_version_hash.to_uppercase(); + let resp = process_consent(&state, "WORKER-AL3", Some(TEST_TOKEN), "trace-al", req) + .await.unwrap(); + assert_eq!(resp.status_after, "Given"); + } + + #[tokio::test] + async fn consent_allowlist_empty_set_refuses_all() { + // Strict-locked mode: operator created the file but populated + // it with an empty array. Refuses every grant — used as a + // deliberate freeze (e.g., counsel hasn't signed v1 yet). + let state = fixture_state_with_allowlist("al_empty", vec![]).await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-AL4", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let err = process_consent(&state, "WORKER-AL4", Some(TEST_TOKEN), "trace-al", fixture_consent_request()) + .await.unwrap_err(); + assert_eq!(err.0, StatusCode::BAD_REQUEST); + assert_eq!(err.1.error, "consent_version_unknown"); + } + + #[tokio::test] + async fn consent_allowlist_none_is_permissive() { + // The default fixture has allowed_consent_versions=None. + // Grant succeeds with any non-empty hash (v1 compat behavior). + let state = fixture_state("consent_allowlist_none").await; + let _ = state.registry.put_subject(fixture_manifest("WORKER-AL5", BiometricConsentStatus::NeverCollected, SubjectStatus::Active)).await; + let resp = process_consent(&state, "WORKER-AL5", Some(TEST_TOKEN), "trace-al", fixture_consent_request()) + .await.unwrap(); + assert_eq!(resp.status_after, "Given"); + } + + #[tokio::test] + async fn stats_oldest_retention_picks_earliest_given() { + let state = fixture_state("stats_retention").await; + let now = chrono::Utc::now(); + // Three Given subjects with different retention dates. + let later = now + chrono::Duration::days(540); // 18 months + let middle = now + chrono::Duration::days(300); + let early = now + chrono::Duration::days(60); // earliest + for (id, ret) in [("WORKER-R1", later), ("WORKER-R2", middle), ("WORKER-R3", early)] { + let mut m = fixture_manifest(id, BiometricConsentStatus::Given, SubjectStatus::Active); + m.consent.biometric.retention_until = Some(ret); + let _ = state.registry.put_subject(m).await; + } + // Plus a Withdrawn subject with an even earlier retention date — + // should NOT count for "oldest active" because it's not Given. + let mut withdrawn = fixture_manifest("WORKER-R4", BiometricConsentStatus::Withdrawn, SubjectStatus::Active); + withdrawn.consent.biometric.retention_until = Some(now + chrono::Duration::days(15)); + let _ = state.registry.put_subject(withdrawn).await; + + let stats = process_stats(&state, Some(TEST_TOKEN)).await.unwrap(); + // The oldest_active should be the earliest Given retention (WORKER-R3, ~60d). + // (Equality on DateTime doesn't suffer the put_subject + // microsecond-skew issue here because retention_until is a + // pre-computed value we set ourselves; it round-trips intact.) + assert_eq!(stats.oldest_active_retention_until, Some(early)); + let days = stats.upcoming_destruction_window_days.unwrap(); + // Allow ±1-day fudge for the now() taken inside process_stats. + assert!((59..=61).contains(&days), + "expected ~60 days, got {} (early={} now-ish={:?})", days, early, stats.generated_at); + } +} diff --git a/crates/catalogd/src/lib.rs b/crates/catalogd/src/lib.rs index 331cdb5..c46bd2f 100644 --- a/crates/catalogd/src/lib.rs +++ b/crates/catalogd/src/lib.rs @@ -2,3 +2,6 @@ pub mod registry; pub mod service; pub mod grpc; pub mod tombstones; +pub mod subject_audit; +pub mod audit_endpoint; +pub mod biometric_endpoint; diff --git a/crates/catalogd/src/registry.rs b/crates/catalogd/src/registry.rs index 8113552..efac9ea 100644 --- a/crates/catalogd/src/registry.rs +++ b/crates/catalogd/src/registry.rs @@ -1,6 +1,6 @@ use shared::types::{ AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile, - ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone, + ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, SubjectManifest, Tombstone, }; use crate::tombstones::TombstoneStore; @@ -66,16 +66,20 @@ pub struct MetadataUpdate { const MANIFEST_PREFIX: &str = "_catalog/manifests"; const VIEW_PREFIX: &str = "_catalog/views"; +const SUBJECT_PREFIX: &str = "_catalog/subjects"; const PROFILE_PREFIX: &str = "_catalog/profiles"; /// In-memory dataset registry backed by manifest persistence in object storage. /// Also tracks AiViews (Phase D) — safe projections over base datasets. /// And tombstones (Phase E) — soft-delete markers applied at query time. +/// And subject manifests (compliance/access metadata layer) — see +/// docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md. #[derive(Clone)] pub struct Registry { datasets: Arc>>, views: Arc>>, profiles: Arc>>, + subjects: Arc>>, tombstones: TombstoneStore, store: Arc, } @@ -86,6 +90,7 @@ impl Registry { datasets: Arc::new(RwLock::new(HashMap::new())), views: Arc::new(RwLock::new(HashMap::new())), profiles: Arc::new(RwLock::new(HashMap::new())), + subjects: Arc::new(RwLock::new(HashMap::new())), tombstones: TombstoneStore::new(store.clone()), store, } @@ -179,6 +184,47 @@ impl Registry { tracing::info!("catalog: {} model profiles loaded", profiles.len()); } + // Subject manifests (compliance/access metadata layer). + let subject_keys = ops::list(&self.store, Some(SUBJECT_PREFIX)).await.unwrap_or_default(); + let mut subjects = self.subjects.write().await; + subjects.clear(); + for key in &subject_keys { + if !key.ends_with(".json") { continue; } + let data = match ops::get(&self.store, key).await { + Ok(d) => d, + Err(e) => { + tracing::warn!("subject '{key}': read failed: {e}"); + continue; + } + }; + match serde_json::from_slice::(&data) { + // Match put_subject: in-memory key is the SANITIZED form + // so get_subject lookups match the storage key shape. + Ok(subj) => { + let map_key = sanitize_view_name(&subj.candidate_id); + // Rebuild collision detection — symmetric with the + // put_subject collision guard. If pre-existing storage + // somehow contains two manifests whose raw ids + // sanitize to the same key, log loudly so operators + // know one was dropped instead of silently losing it. + // (2026-05-03 opus scrum WARN registry.rs:201.) + if let Some(existing) = subjects.get(&map_key) { + if existing.candidate_id != subj.candidate_id { + tracing::warn!( + "subject rebuild collision on sanitized key '{map_key}': raw '{}' was loaded earlier; raw '{}' is being silently overwritten in memory. Run dedupe + delete the duplicate manifest file at storage.", + existing.candidate_id, subj.candidate_id, + ); + } + } + subjects.insert(map_key, subj); + } + Err(e) => tracing::warn!("subject '{key}': parse failed: {e}"), + } + } + if !subjects.is_empty() { + tracing::info!("catalog: {} subject manifests loaded", subjects.len()); + } + Ok(count) } @@ -686,6 +732,94 @@ impl Registry { Ok(()) } + // --- Subject manifests (compliance/access metadata layer) --- + // Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md + // Mirrors the put_view / get_view / list_views / delete_view shape. + // The audit log per subject is owned separately by SubjectAuditWriter + // (Step 2) — this layer just persists the manifest itself. + + /// Create or replace a subject manifest. Validates that referenced + /// datasets exist in the catalog (fail-fast so dangling references + /// don't accumulate). Persists to `_catalog/subjects/.json`. + /// + /// **Storage-key collision guard** (2026-05-03 opus scrum WARN + /// at registry.rs:735): the storage key is `sanitize_view_name(id)` + /// — `/`, `:`, etc. become `_`. A naive in-memory keying by raw + /// candidate_id would let two distinct ids (`CAND/1` + `CAND_1`) + /// collide on disk while appearing distinct in memory; the second + /// put silently overwrites the first, and `rebuild` only loads one. + /// Both the in-memory map AND the storage key now use the sanitized + /// form. Caller's raw candidate_id is preserved on the manifest for + /// audit-trail integrity (the manifest body has the original). + /// We surface a clear error if two raw ids would collide on the + /// sanitized key. + pub async fn put_subject(&self, mut subj: SubjectManifest) -> Result { + if subj.candidate_id.is_empty() { + return Err("subject candidate_id is empty".into()); + } + for ds in &subj.datasets { + if self.get_by_name(&ds.name).await.is_none() { + return Err(format!( + "subject '{}' references dataset '{}' which is not in the catalog", + subj.candidate_id, ds.name, + )); + } + } + let now = chrono::Utc::now(); + if subj.created_at.timestamp() == 0 { + subj.created_at = now; + } + subj.updated_at = now; + + let map_key = sanitize_view_name(&subj.candidate_id); + let key = format!("{SUBJECT_PREFIX}/{}.json", map_key); + + // Collision guard: if the sanitized key already exists for a + // DIFFERENT raw candidate_id, refuse the put. Better to fail + // loudly than silently overwrite an existing subject. + { + let subjects = self.subjects.read().await; + if let Some(existing) = subjects.get(&map_key) { + if existing.candidate_id != subj.candidate_id { + return Err(format!( + "subject id '{}' collides with existing subject '{}' on sanitized storage key '{}'", + subj.candidate_id, existing.candidate_id, map_key, + )); + } + } + } + + let json = serde_json::to_vec_pretty(&subj).map_err(|e| e.to_string())?; + ops::put(&self.store, &key, json.into()).await?; + + let mut subjects = self.subjects.write().await; + subjects.insert(map_key, subj.clone()); + tracing::debug!("subject manifest persisted: {}", subj.candidate_id); + Ok(subj) + } + + pub async fn get_subject(&self, candidate_id: &str) -> Option { + self.subjects.read().await.get(&sanitize_view_name(candidate_id)).cloned() + } + + pub async fn list_subjects(&self) -> Vec { + self.subjects.read().await.values().cloned().collect() + } + + pub async fn delete_subject(&self, candidate_id: &str) -> Result<(), String> { + let map_key = sanitize_view_name(candidate_id); + let key = format!("{SUBJECT_PREFIX}/{}.json", map_key); + ops::delete(&self.store, &key).await?; + self.subjects.write().await.remove(&map_key); + Ok(()) + } + + /// Subject count — convenience for the daily retention sweep + audit + /// dashboards that don't need the full list. + pub async fn subjects_count(&self) -> usize { + self.subjects.read().await.len() + } + /// Remove a dataset from the catalog by name. Deletes the manifest /// from both the in-memory registry and object storage. /// @@ -1027,4 +1161,130 @@ mod tests { assert!(reg.get(&loser_a).await.is_none()); assert!(reg.get(&loser_b).await.is_none()); } + + // --- Subject manifests (Step 1 of subject_manifests_on_catalogd) --- + + fn fixture_subject(id: &str) -> shared::types::SubjectManifest { + use shared::types::{ + BiometricConsent, BiometricConsentStatus, ConsentStatus, + GeneralPiiConsent, SubjectConsent, SubjectManifest, SubjectRetention, + SubjectStatus, SubjectVertical, + }; + SubjectManifest { + schema: "subject_manifest.v1".into(), + candidate_id: id.to_string(), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + status: SubjectStatus::Active, + vertical: SubjectVertical::Unknown, + consent: SubjectConsent { + general_pii: GeneralPiiConsent { + status: ConsentStatus::PendingBackfillReview, + version: String::new(), + given_at: None, + withdrawn_at: None, + }, + biometric: BiometricConsent { + status: BiometricConsentStatus::NeverCollected, + retention_until: None, + }, + }, + retention: SubjectRetention { + general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4), + policy: "4_year_default".into(), + }, + datasets: vec![], + safe_views: vec![], + audit_log_path: String::new(), + audit_log_chain_root: String::new(), + biometric_collection: None, + } + } + + #[tokio::test] + async fn put_subject_with_no_dataset_refs_succeeds() { + let reg = fixture(); + let s = fixture_subject("CAND-000001"); + let stored = reg.put_subject(s).await.unwrap(); + assert_eq!(stored.candidate_id, "CAND-000001"); + assert_eq!(reg.subjects_count().await, 1); + } + + #[tokio::test] + async fn put_subject_rejects_dangling_dataset_ref() { + use shared::types::SubjectDatasetRef; + let reg = fixture(); + let mut s = fixture_subject("CAND-000002"); + s.datasets.push(SubjectDatasetRef { + name: "no_such_dataset".into(), + key_column: "candidate_id".into(), + key_value: "CAND-000002".into(), + }); + let err = reg.put_subject(s).await.unwrap_err(); + assert!(err.contains("not in the catalog"), "got: {err}"); + } + + #[tokio::test] + async fn put_subject_with_valid_dataset_ref_succeeds() { + use shared::types::SubjectDatasetRef; + let reg = fixture(); + // Register a dataset first so the subject's reference resolves. + reg.register("workers".into(), fp("w1"), vec![obj("w.parquet")]).await.unwrap(); + + let mut s = fixture_subject("CAND-000003"); + s.datasets.push(SubjectDatasetRef { + name: "workers".into(), + key_column: "candidate_id".into(), + key_value: "CAND-000003".into(), + }); + let stored = reg.put_subject(s).await.unwrap(); + assert_eq!(stored.datasets.len(), 1); + assert_eq!(stored.datasets[0].name, "workers"); + } + + #[tokio::test] + async fn subject_round_trips_through_object_store() { + let store = Arc::new(object_store::memory::InMemory::new()); + let reg = Registry::new(store.clone()); + reg.put_subject(fixture_subject("CAND-RT-001")).await.unwrap(); + + // Open a fresh registry over the same store and load → confirms + // persistence + load_all path actually round-trips the manifest. + let reg2 = Registry::new(store); + reg2.rebuild().await.unwrap(); + let loaded = reg2.get_subject("CAND-RT-001").await; + assert!(loaded.is_some(), "subject not loaded from store"); + assert_eq!(loaded.unwrap().status, shared::types::SubjectStatus::Active); + } + + #[tokio::test] + async fn put_subject_rejects_sanitize_collision() { + // 2026-05-03 opus scrum WARN at registry.rs:735 regression: + // CAND/1 and CAND_1 sanitize to the same storage key. Without + // the collision guard, the second put silently overwrites the + // first on disk; with it, the second put fails loudly. + let reg = fixture(); + let s1 = fixture_subject("CAND/1"); + reg.put_subject(s1).await.unwrap(); + let s2 = fixture_subject("CAND_1"); // sanitizes to same key + let err = reg.put_subject(s2).await.unwrap_err(); + assert!(err.contains("collides"), "got: {err}"); + assert_eq!(reg.subjects_count().await, 1, "first subject must survive"); + } + + #[tokio::test] + async fn delete_subject_removes_in_memory_and_persistence() { + let store = Arc::new(object_store::memory::InMemory::new()); + let reg = Registry::new(store.clone()); + reg.put_subject(fixture_subject("CAND-DEL-001")).await.unwrap(); + assert_eq!(reg.subjects_count().await, 1); + + reg.delete_subject("CAND-DEL-001").await.unwrap(); + assert_eq!(reg.subjects_count().await, 0); + + // Fresh registry over same store — confirm it's also gone from persistence. + let reg2 = Registry::new(store); + reg2.rebuild().await.unwrap(); + assert_eq!(reg2.subjects_count().await, 0); + } } diff --git a/crates/catalogd/src/subject_audit.rs b/crates/catalogd/src/subject_audit.rs new file mode 100644 index 0000000..db505c6 --- /dev/null +++ b/crates/catalogd/src/subject_audit.rs @@ -0,0 +1,587 @@ +//! Subject audit log writer with HMAC chain. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §3. +//! +//! Per-subject append-only JSONL at `_catalog/subjects/.audit.jsonl`. +//! Each row's `row_hmac` is HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac). +//! The chain root is mirrored to the SubjectManifest.audit_log_chain_root +//! field after each append, so verifiers can find the latest valid root +//! from the manifest alone. +//! +//! Failures are non-blocking — audit-write errors are logged but do NOT +//! propagate to the caller (per spec §3.2: "better to leak a row than +//! block legitimate operations"). Operators MUST monitor the error count. +//! +//! Key loading: from a sealed file at the path the writer is constructed +//! with. Tests use `with_inline_key` to inject a deterministic key. + +use bytes::Bytes; +use hmac::{Hmac, Mac}; +use object_store::ObjectStore; +use sha2::Sha256; +use shared::types::SubjectAuditRow; +use std::collections::HashMap; +use std::sync::Arc; +use storaged::ops; +use tokio::sync::Mutex; + +const SUBJECT_PREFIX: &str = "_catalog/subjects"; +pub const GENESIS_HASH: &str = "GENESIS"; + +type HmacSha256 = Hmac; + +/// Render a JSON value canonically: object keys sorted alphabetically, +/// no insignificant whitespace, no trailing newline. Recursive — nested +/// objects also sort their keys. Arrays preserve order (semantically +/// significant). Numbers, strings, booleans, nulls pass through. +/// +/// This is intentionally narrower than RFC 8785 (no number normalization, +/// no UTF-8 NFC re-encoding). For our use — hashing structurally simple +/// audit-row JSON — alphabetical key order is the only canonicalization +/// that matters: it makes the byte sequence stable across struct field +/// reordering and additive schema evolution. +fn canonical_json(v: &serde_json::Value) -> Result, String> { + use std::collections::BTreeMap; + fn rewrite(v: &serde_json::Value) -> serde_json::Value { + match v { + serde_json::Value::Object(map) => { + let sorted: BTreeMap = map + .iter() + .map(|(k, v)| (k.clone(), rewrite(v))) + .collect(); + serde_json::Value::Object(sorted.into_iter().collect()) + } + serde_json::Value::Array(arr) => { + serde_json::Value::Array(arr.iter().map(rewrite).collect()) + } + other => other.clone(), + } + } + let canonical = rewrite(v); + serde_json::to_vec(&canonical).map_err(|e| e.to_string()) +} + +/// Per-subject audit log writer. Holds the signing key, a small +/// in-memory cache of latest chain hash per subject, and per-subject +/// mutexes that serialize concurrent appends to the same subject's log. +/// +/// **No Debug impl deliberately** — auto-deriving Debug on this struct +/// would risk leaking the signing key into log lines. Tests that need +/// `Result::unwrap_err` on a function returning `Result` must +/// match on the result instead of using `.unwrap_err()`. +/// +/// Concurrency: appends to DIFFERENT subjects run in parallel; appends +/// to the SAME subject serialize via a per-candidate Mutex. Without +/// per-subject locking the read-modify-write append_line race +/// (concurrent reads of the same `prev_hash` + concurrent writes +/// overwriting each other) silently corrupts the chain. Caught by +/// 2026-05-03 opus scrum BLOCK on subject_audit.rs:172. +#[derive(Clone)] +pub struct SubjectAuditWriter { + store: Arc, + /// HMAC signing key. Loaded once at construction. Never logged. + /// `Vec` not `String` — keys are bytes, not text. + key: Arc>, + /// In-memory cache of the latest chain hash per candidate_id. + /// Loaded lazily from the audit JSONL on first append per subject. + latest_hash: Arc>>, + /// Per-subject mutexes. Append acquires the subject's lock before + /// the read-modify-write to serialize concurrent appends to the + /// same subject. Different subjects don't contend. + subject_locks: Arc>>>>, +} + +impl SubjectAuditWriter { + /// Construct a writer that loads its signing key from a file at + /// `key_path`. The file MUST be mode 0400 owner-only on production + /// hosts (operator's responsibility — not enforced here because the + /// permission check would be a footgun in tests/dev). + pub async fn from_key_file( + store: Arc, + key_path: &std::path::Path, + ) -> Result { + let key = tokio::fs::read(key_path) + .await + .map_err(|e| format!("read signing key from {}: {e}", key_path.display()))?; + if key.len() < 32 { + return Err(format!( + "signing key is {} bytes; recommend ≥32 bytes for HMAC-SHA256", + key.len() + )); + } + Ok(Self { + store, + key: Arc::new(key), + latest_hash: Arc::new(Mutex::new(HashMap::new())), + subject_locks: Arc::new(Mutex::new(HashMap::new())), + }) + } + + /// Construct a writer with an inline key. Intended for tests + bring-up. + /// Production should use `from_key_file`. + pub fn with_inline_key(store: Arc, key: Vec) -> Self { + Self { + store, + key: Arc::new(key), + latest_hash: Arc::new(Mutex::new(HashMap::new())), + subject_locks: Arc::new(Mutex::new(HashMap::new())), + } + } + + fn audit_key(candidate_id: &str) -> String { + let safe: String = candidate_id + .chars() + .map(|c| { + if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' { + c + } else { + '_' + } + }) + .collect(); + format!("{SUBJECT_PREFIX}/{}.audit.jsonl", safe) + } + + /// Compute HMAC-SHA256(key, prev_hash_bytes || canonical_row_bytes). + /// Returns the HMAC as lowercase hex. Uses byte-level hex formatting + /// in a single pass (avoids per-byte format! allocations). + fn compute_hmac(&self, prev_hash: &str, canonical_row: &[u8]) -> String { + let mut mac = ::new_from_slice(&self.key) + .expect("HMAC accepts any key length"); + mac.update(prev_hash.as_bytes()); + mac.update(canonical_row); + let result = mac.finalize().into_bytes(); + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut s = String::with_capacity(64); + for byte in result { + s.push(HEX[(byte >> 4) as usize] as char); + s.push(HEX[(byte & 0x0f) as usize] as char); + } + s + } + + /// Render an audit row as canonical JSON: keys sorted alphabetically, + /// no insignificant whitespace, row_hmac field excluded. This is the + /// byte sequence the HMAC is computed over. + /// + /// **Why a manual canonicalizer instead of `serde_json::to_vec`:** + /// serde_json serializes struct fields in DECLARATION order. If the + /// SubjectAuditRow struct ever gains/loses/reorders fields (schema + /// evolution), the bytes change even though the LOGICAL content is + /// the same. A verifier compiled against the new struct shape + /// re-serializes a stored row and gets DIFFERENT bytes than the + /// original write — chain breaks silently. Caught by 2026-05-03 + /// opus scrum BLOCK on subject_audit.rs:108. + /// + /// Canonical-JSON (RFC 8785-ish, simplified for our scalar types) + /// fixes this by using alphabetical key order + minimal whitespace, + /// stable across struct evolution. + fn canonical_row_bytes(row: &SubjectAuditRow) -> Result, String> { + // Project to a generic Value, drop row_hmac, then re-serialize + // through a BTreeMap so keys come out alphabetical. + let mut v: serde_json::Value = serde_json::to_value(row) + .map_err(|e| format!("audit row to value: {e}"))?; + if let Some(obj) = v.as_object_mut() { + obj.remove("row_hmac"); + } + canonical_json(&v).map_err(|e| format!("canonicalize: {e}")) + } + + /// Get-or-create per-subject lock. Held across the read-modify-write + /// in append() so concurrent appends to the same subject serialize. + async fn lock_for(&self, candidate_id: &str) -> Arc> { + let mut locks = self.subject_locks.lock().await; + locks + .entry(candidate_id.to_string()) + .or_insert_with(|| Arc::new(Mutex::new(()))) + .clone() + } + + /// Append one audit row for a subject. Computes the HMAC chain link, + /// writes to the per-subject JSONL, returns the new chain root (which + /// the caller MAY mirror to SubjectManifest.audit_log_chain_root). + /// + /// **Concurrency**: appends to the SAME subject serialize via a + /// per-subject Mutex. Appends to DIFFERENT subjects run in parallel. + /// Without the per-subject lock, the read-modify-write in + /// append_line races and silently corrupts the chain. (2026-05-03 + /// opus scrum BLOCK on subject_audit.rs:172.) + /// + /// On error: logs and returns Err. Caller decides whether to propagate + /// (per spec §3.2 the gateway tool registry SHOULD log + drop). + pub async fn append(&self, mut row: SubjectAuditRow) -> Result { + let cid = row.candidate_id.clone(); + if cid.is_empty() { + return Err("audit row candidate_id is empty".into()); + } + + // Acquire the per-subject lock for the duration of this append. + let subject_lock = self.lock_for(&cid).await; + let _guard = subject_lock.lock().await; + + // 1. Find prev_chain_hash. Cache hot path; cold path scans + // the existing JSONL tail-style to find the last row. + let prev = { + let cache = self.latest_hash.lock().await; + cache.get(&cid).cloned() + }; + let prev_hash = match prev { + Some(h) => h, + None => self.scan_latest_hash(&cid).await, + }; + row.prev_chain_hash = prev_hash.clone(); + row.row_hmac = String::new(); // Always cleared before HMAC computation. + + // 2. Canonical-JSON the row (sorted keys, no whitespace, no + // row_hmac), then HMAC. Stable across struct evolution. + let canon = Self::canonical_row_bytes(&row)?; + let new_hmac = self.compute_hmac(&prev_hash, &canon); + row.row_hmac = new_hmac.clone(); + + // 3. Append the row to the JSONL. The per-subject lock above + // serializes the read-modify-write inside append_line. + let line = serde_json::to_vec(&row) + .map_err(|e| format!("serialize audit row: {e}"))?; + self.append_line(&cid, &line).await?; + + // 4. Update cache. + self.latest_hash.lock().await.insert(cid, new_hmac.clone()); + + Ok(new_hmac) + } + + /// Scan the audit JSONL for a subject and return the last row's + /// row_hmac. GENESIS if no log exists or it's empty. + /// Object-store-aware: works across LocalFileSystem + InMemory + S3. + async fn scan_latest_hash(&self, candidate_id: &str) -> String { + let key = Self::audit_key(candidate_id); + let bytes = match ops::get(&self.store, &key).await { + Ok(b) => b, + Err(_) => return GENESIS_HASH.to_string(), // file doesn't exist + }; + // Find the last newline-delimited line that parses as a row. + let text = match std::str::from_utf8(&bytes) { + Ok(t) => t, + Err(_) => return GENESIS_HASH.to_string(), + }; + for line in text.lines().rev() { + if line.trim().is_empty() { + continue; + } + if let Ok(parsed) = serde_json::from_str::(line) { + return parsed.row_hmac; + } + } + GENESIS_HASH.to_string() + } + + /// Append a single newline-terminated line to the per-subject log. + /// Reads the existing content (may be empty), appends, writes back. + /// Object stores don't have native append; this is the standard + /// "read-modify-write the tail file" pattern. + async fn append_line(&self, candidate_id: &str, line: &[u8]) -> Result<(), String> { + let key = Self::audit_key(candidate_id); + let existing = ops::get(&self.store, &key).await.unwrap_or_default(); + let mut buf = Vec::with_capacity(existing.len() + line.len() + 1); + buf.extend_from_slice(&existing); + if !buf.is_empty() && !buf.ends_with(b"\n") { + buf.push(b'\n'); + } + buf.extend_from_slice(line); + buf.push(b'\n'); + ops::put(&self.store, &key, Bytes::from(buf)).await + } + + /// Read all audit rows for a subject, optionally filtered by time + /// window (inclusive bounds). Returns rows in append order. Used by + /// the /audit/subject/{id} endpoint to assemble the legal-tier + /// response. Window bounds are optional — `from=None`+`to=None` + /// returns the full log. + pub async fn read_rows_in_range( + &self, + candidate_id: &str, + from: Option>, + to: Option>, + ) -> Result, String> { + let key = Self::audit_key(candidate_id); + let bytes = match ops::get(&self.store, &key).await { + Ok(b) => b, + Err(_) => return Ok(Vec::new()), // no audit log = empty result + }; + let text = std::str::from_utf8(&bytes) + .map_err(|e| format!("audit log not utf-8: {e}"))?; + let mut out = Vec::new(); + for line in text.lines() { + if line.trim().is_empty() { + continue; + } + let row: SubjectAuditRow = match serde_json::from_str(line) { + Ok(r) => r, + Err(_) => continue, // skip unparseable lines (defensive) + }; + if let Some(from) = from { + if row.ts < from { continue; } + } + if let Some(to) = to { + if row.ts > to { continue; } + } + out.push(row); + } + Ok(out) + } + + /// Return the latest row_hmac in this subject's audit log, or None + /// if the log doesn't exist / has no rows. Cache-aware: hits the + /// in-memory cache when the writer has appended at least once for + /// this subject in the current process, otherwise scans the log + /// tail (single object_store read). + /// + /// Used by /audit/subject/{id} to echo the actual chain tip in the + /// response. Taking the tip from a windowed slice of rows would + /// lie when the window doesn't cover the latest row. + pub async fn chain_tip(&self, candidate_id: &str) -> Option { + // Cache hit? + if let Some(h) = self.latest_hash.lock().await.get(candidate_id).cloned() { + return Some(h); + } + // Cold path — scan the log tail. Returns GENESIS for empty/missing + // logs; we coerce that to None so callers can distinguish "no audit + // activity yet" from "real chain tip exists". + let tip = self.scan_latest_hash(candidate_id).await; + if tip == GENESIS_HASH { None } else { Some(tip) } + } + + /// Verify the full HMAC chain for a subject. Returns Ok(rows_verified) + /// or Err with the first chain break encountered. + /// + /// Special case: if no audit log exists (the subject has had no PII + /// access yet), returns Ok(0) — an empty log is trivially valid. + /// Callers that want to detect "log was deleted" should compare + /// against SubjectManifest.audit_log_chain_root: if the manifest's + /// chain root is non-empty/non-GENESIS but verify_chain returns 0, + /// the file was tampered. + pub async fn verify_chain(&self, candidate_id: &str) -> Result { + let key = Self::audit_key(candidate_id); + let bytes = match ops::get(&self.store, &key).await { + Ok(b) => b, + Err(_) => return Ok(0), // no log = 0 rows = trivially valid + }; + let text = std::str::from_utf8(&bytes) + .map_err(|e| format!("audit log not utf-8: {e}"))?; + let mut prev = GENESIS_HASH.to_string(); + let mut count = 0usize; + for (lineno, line) in text.lines().enumerate() { + if line.trim().is_empty() { + continue; + } + let mut row: SubjectAuditRow = serde_json::from_str(line) + .map_err(|e| format!("line {} parse failed: {e}", lineno + 1))?; + if row.prev_chain_hash != prev { + return Err(format!( + "chain break at line {}: prev_chain_hash={} expected={}", + lineno + 1, row.prev_chain_hash, prev, + )); + } + let claimed = std::mem::take(&mut row.row_hmac); + let canon = Self::canonical_row_bytes(&row) + .map_err(|e| format!("canonicalize line {}: {e}", lineno + 1))?; + let recomputed = self.compute_hmac(&prev, &canon); + if recomputed != claimed { + return Err(format!( + "hmac mismatch at line {}: stored={} recomputed={}", + lineno + 1, claimed, recomputed, + )); + } + prev = claimed; + count += 1; + } + Ok(count) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use object_store::memory::InMemory; + use shared::types::AuditAccessor; + + fn fixture_writer() -> SubjectAuditWriter { + SubjectAuditWriter::with_inline_key( + Arc::new(InMemory::new()), + // Deterministic 32-byte key for test reproducibility. + (0u8..32).collect(), + ) + } + + fn fixture_row(candidate_id: &str, fields: &[&str]) -> SubjectAuditRow { + SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::Utc::now(), + candidate_id: candidate_id.into(), + accessor: AuditAccessor { + kind: "gateway_lookup".into(), + daemon: "gateway".into(), + purpose: "fill_validation".into(), + trace_id: String::new(), + }, + fields_accessed: fields.iter().map(|s| s.to_string()).collect(), + result: "success".into(), + prev_chain_hash: String::new(), // filled in by writer + row_hmac: String::new(), // filled in by writer + } + } + + #[tokio::test] + async fn first_append_uses_genesis_prev_hash() { + let w = fixture_writer(); + let new_root = w.append(fixture_row("CAND-1", &["name"])).await.unwrap(); + assert!(!new_root.is_empty()); + assert_eq!(new_root.len(), 64); // SHA-256 hex + // After first append, the only row should reference GENESIS as prev. + let bytes = ops::get(&w.store, &SubjectAuditWriter::audit_key("CAND-1")) + .await.unwrap(); + let text = std::str::from_utf8(&bytes).unwrap(); + let row: SubjectAuditRow = serde_json::from_str(text.trim()).unwrap(); + assert_eq!(row.prev_chain_hash, "GENESIS"); + assert_eq!(row.row_hmac, new_root); + } + + #[tokio::test] + async fn chain_links_each_append() { + let w = fixture_writer(); + let h1 = w.append(fixture_row("CAND-2", &["name"])).await.unwrap(); + let h2 = w.append(fixture_row("CAND-2", &["phone"])).await.unwrap(); + let h3 = w.append(fixture_row("CAND-2", &["email"])).await.unwrap(); + assert_ne!(h1, h2); + assert_ne!(h2, h3); + let count = w.verify_chain("CAND-2").await.unwrap(); + assert_eq!(count, 3); + } + + #[tokio::test] + async fn separate_subjects_have_independent_chains() { + let w = fixture_writer(); + let _ = w.append(fixture_row("CAND-A", &["name"])).await.unwrap(); + let _ = w.append(fixture_row("CAND-B", &["name"])).await.unwrap(); + let _ = w.append(fixture_row("CAND-A", &["phone"])).await.unwrap(); + assert_eq!(w.verify_chain("CAND-A").await.unwrap(), 2); + assert_eq!(w.verify_chain("CAND-B").await.unwrap(), 1); + } + + #[tokio::test] + async fn tamper_detected_on_verify() { + let w = fixture_writer(); + let _ = w.append(fixture_row("CAND-T", &["name"])).await.unwrap(); + let _ = w.append(fixture_row("CAND-T", &["phone"])).await.unwrap(); + // Mutate the file to flip a `result` field on the first row. + let key = SubjectAuditWriter::audit_key("CAND-T"); + let bytes = ops::get(&w.store, &key).await.unwrap(); + let mut text = std::str::from_utf8(&bytes).unwrap().to_string(); + text = text.replacen("\"success\"", "\"denied\"", 1); + ops::put(&w.store, &key, text.into_bytes().into()).await.unwrap(); + let err = w.verify_chain("CAND-T").await.unwrap_err(); + assert!(err.contains("hmac mismatch"), "expected hmac mismatch, got: {err}"); + } + + #[tokio::test] + async fn cold_writer_picks_up_existing_chain() { + let store: Arc = Arc::new(InMemory::new()); + let key: Vec = (0u8..32).collect(); + let w1 = SubjectAuditWriter::with_inline_key(store.clone(), key.clone()); + let h1 = w1.append(fixture_row("CAND-COLD", &["name"])).await.unwrap(); + // Drop w1's cache, simulate process restart. + drop(w1); + let w2 = SubjectAuditWriter::with_inline_key(store.clone(), key); + let h2 = w2.append(fixture_row("CAND-COLD", &["phone"])).await.unwrap(); + assert_ne!(h1, h2); + assert_eq!(w2.verify_chain("CAND-COLD").await.unwrap(), 2); + } + + #[tokio::test] + async fn empty_candidate_id_rejected() { + let w = fixture_writer(); + let row = fixture_row("", &["name"]); + let err = w.append(row).await.unwrap_err(); + assert!(err.contains("empty"), "got: {err}"); + } + + #[tokio::test] + async fn concurrent_appends_to_same_subject_serialize() { + // 2026-05-03 opus scrum BLOCK regression: without per-subject + // locking, concurrent appends raced and silently lost rows. + // This test spawns 50 parallel appends and asserts ALL 50 + // land with an intact chain. + let w = Arc::new(fixture_writer()); + let mut handles = Vec::new(); + for i in 0..50 { + let w = w.clone(); + handles.push(tokio::spawn(async move { + w.append(fixture_row("CAND-RACE", &[&format!("f{i}")])).await + })); + } + for h in handles { + h.await.unwrap().unwrap(); + } + let count = w.verify_chain("CAND-RACE").await.unwrap(); + assert_eq!(count, 50, "expected 50 rows; chain was corrupted"); + } + + #[tokio::test] + async fn concurrent_appends_to_different_subjects_run_parallel() { + let w = Arc::new(fixture_writer()); + let mut handles = Vec::new(); + for i in 0..20 { + let w = w.clone(); + handles.push(tokio::spawn(async move { + w.append(fixture_row(&format!("CAND-PARA-{i}"), &["f"])).await + })); + } + for h in handles { + h.await.unwrap().unwrap(); + } + // Each of the 20 subjects has exactly 1 row. + for i in 0..20 { + assert_eq!(w.verify_chain(&format!("CAND-PARA-{i}")).await.unwrap(), 1); + } + } + + #[test] + fn canonical_json_sorts_keys_alphabetically() { + let v = serde_json::json!({"z": 1, "a": 2, "m": {"y": 1, "b": 2}}); + let bytes = canonical_json(&v).unwrap(); + let s = std::str::from_utf8(&bytes).unwrap(); + // Keys should appear in alphabetical order at every nesting level. + let a_pos = s.find("\"a\"").unwrap(); + let m_pos = s.find("\"m\"").unwrap(); + let z_pos = s.find("\"z\"").unwrap(); + assert!(a_pos < m_pos); + assert!(m_pos < z_pos); + // Nested object's keys also sorted. + let b_pos = s.find("\"b\"").unwrap(); + let y_pos = s.find("\"y\"").unwrap(); + assert!(b_pos < y_pos); + } + + #[test] + fn canonical_json_arrays_preserve_order() { + let v = serde_json::json!({"k": ["c", "a", "b"]}); + let bytes = canonical_json(&v).unwrap(); + let s = std::str::from_utf8(&bytes).unwrap(); + assert!(s.contains("\"c\",\"a\",\"b\""), "got: {s}"); + } + + #[tokio::test] + async fn key_too_short_rejected_via_file() { + // Write a 16-byte key file (under the 32-byte minimum). + let tmp = std::env::temp_dir().join(format!("sa_key_short_{}.bin", std::process::id())); + tokio::fs::write(&tmp, vec![0u8; 16]).await.unwrap(); + let store: Arc = Arc::new(InMemory::new()); + // `unwrap_err` would require Debug on SubjectAuditWriter, which + // we deliberately don't impl (key-leak risk). Match instead. + match SubjectAuditWriter::from_key_file(store, &tmp).await { + Err(e) => assert!(e.contains("≥32 bytes"), "got: {e}"), + Ok(_) => panic!("expected too-short-key rejection"), + } + let _ = tokio::fs::remove_file(&tmp).await; + } +} diff --git a/crates/gateway/src/execution_loop/mod.rs b/crates/gateway/src/execution_loop/mod.rs index 4d5d1f3..80c795d 100644 --- a/crates/gateway/src/execution_loop/mod.rs +++ b/crates/gateway/src/execution_loop/mod.rs @@ -212,6 +212,13 @@ impl ExecutionLoop { if let Action::ToolCall { tool, args, .. } = &exec_action { match self.dispatch_tool(tool, args).await { Ok(result) => { + // Subject-audit hook (per Phase 1.6 SUBJECT_MANIFESTS spec + // §3.2): every tool result that returns rows referencing a + // subject must produce an audit row. Failures are logged + // but never block the tool result — the spec is explicit: + // "better to leak a row than block legitimate operations." + self.audit_subject_hits_in(&result, tool).await; + let trimmed = trim_result(&result); self.append(LogEntry::new( turn, "executor", &executor_model, "tool_result", trimmed, @@ -341,6 +348,68 @@ impl ExecutionLoop { self.log.push(e); } + /// Subject-audit hook: scan a tool result JSON for subject identifiers, + /// fire one audit row per (subject, fields) pair via the writer in + /// V1State.subject_audit. No-op if writer is None (audit disabled). + /// + /// Conventions (matches the backfill binary in catalogd/src/bin/ + /// backfill_subjects.rs): + /// - "candidate_id":"X" → audit candidate_id="X" + /// - "worker_id":42 → audit candidate_id="WORKER-42" + /// + /// Walks the result JSON recursively. For each object encountered + /// that contains a candidate_id or worker_id field, fires one audit + /// row capturing all the OTHER scalar field names as fields_accessed + /// (so the audit log records "this access surfaced name + phone for + /// candidate X" — sufficient for compliance review). + /// + /// Failures are logged but NEVER propagated. Per spec §3.2: an audit + /// gap is preferable to blocking the tool response. + async fn audit_subject_hits_in(&self, result: &serde_json::Value, tool: &str) { + let Some(audit) = self.state.subject_audit.clone() else { + return; // audit disabled — no-op + }; + // Collect (candidate_id, fields_accessed) pairs from the JSON. + let mut hits: Vec<(String, Vec)> = Vec::new(); + collect_subject_hits(result, &mut hits); + if hits.is_empty() { + return; + } + // Determine result-state from the payload shape, NOT from the + // transport status. A tool can return Ok(Value) where the body + // itself is `{"error":"..."}` — hardcoding "success" here would + // mislead compliance review (opus WARN at execution_loop:401). + let result_state = audit_result_state(result); + + // Sequential await — NOT tokio::spawn. The previous spawn-per-row + // shape fanned out concurrent writes that raced inside the + // per-subject append. The audit writer now serializes per-subject + // internally, but spawning would still flood those locks under + // burst load with no observable latency win. Inline await is + // simpler AND correct. (2026-05-03 opus scrum BLOCKs on + // execution_loop:391 + subject_audit:172.) + for (cid, fields) in hits { + let row = shared::types::SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::Utc::now(), + candidate_id: cid.clone(), + accessor: shared::types::AuditAccessor { + kind: "gateway_lookup".into(), + daemon: "gateway".into(), + purpose: tool.to_string(), + trace_id: String::new(), // TODO: thread X-Lakehouse-Trace-Id through + }, + fields_accessed: fields, + result: result_state.clone(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + if let Err(e) = audit.append(row).await { + tracing::warn!("subject audit write failed for {cid}: {e}"); + } + } + } + /// Dispatch: model name prefix → provider. /// Local path uses Phase 21 `generate_continuable` (auto-continuation, /// retry on empty thinking-model response). Cloud path hits @@ -384,6 +453,8 @@ impl ExecutionLoop { stream: Some(false), think: Some(think), provider: Some("ollama_cloud".into()), + tools: None, + tool_choice: None, }; let resp = crate::v1::ollama_cloud::chat(key, &req).await .map_err(|e| format!("ollama_cloud: {e}"))?; @@ -605,56 +676,54 @@ impl ExecutionLoop { /// cheapest token. Frequency is low so the Zen pay-per-token cost /// stays bounded. async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> { - let Some(opencode_key) = self.state.opencode_key.clone() else { - return Err("OPENCODE_API_KEY not configured — skipping escalation".into()); - }; - + // 2026-05-03: REVERTED to local-only per PRD line 70. Cloud + // overseer (opencode/claude-opus-4-7) was a recent addition that + // moved a hot-path call OFF the local Ollama runtime onto a paid + // cloud provider. Reverted to local Ollama (qwen3.5:latest). + // Cloud overseer can be re-enabled by setting LH_OVERSEER_CLOUD=1 + // for development; production stays local. let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await; let prompt = build_overseer_prompt(&self.req, &kb, &self.log, reason); let started = std::time::Instant::now(); let start_time = chrono::Utc::now(); let chat_req = crate::v1::ChatRequest { - model: "claude-opus-4-7".to_string(), + model: "qwen3.5:latest".to_string(), messages: vec![crate::v1::Message::new_text("user", prompt.clone())], temperature: Some(0.1), max_tokens: None, stream: Some(false), - // Anthropic models on opencode reject `think` (handled in - // the adapter), but we keep the intent flag for parity. - think: Some(true), - provider: Some("opencode".into()), + think: Some(false), + provider: Some("ollama".into()), + tools: None, + tool_choice: None, }; - let resp = crate::v1::opencode::chat(&opencode_key, &chat_req).await - .map_err(|e| format!("opencode: {e}"))?; + let resp = crate::v1::ollama::chat(&self.state.ai_client, &chat_req).await + .map_err(|e| format!("ollama overseer: {e}"))?; let latency_ms = started.elapsed().as_millis() as u64; let end_time = chrono::Utc::now(); let correction_text: String = resp.choices.into_iter().next() .map(|c| c.message.text()).unwrap_or_default(); - // Stamp per-task stats — cloud call counts against the same - // usage counter so `/v1/usage` shows cloud token spend too. self.stats.requests = self.stats.requests.saturating_add(1); self.stats.prompt_tokens = self.stats.prompt_tokens.saturating_add(resp.usage.prompt_tokens as u64); self.stats.completion_tokens = self.stats.completion_tokens.saturating_add(resp.usage.completion_tokens as u64); self.stats.total_tokens = self.stats.total_tokens.saturating_add(resp.usage.total_tokens as u64); self.stats.latency_ms = self.stats.latency_ms.saturating_add(latency_ms); - // Langfuse trace for the overseer call (same pipe that feeds - // the observer/KB, so this correction's cost lands in the KB - // too — closing the loop). + // Langfuse trace for the overseer call (local-only now). if let Some(lf) = &self.state.langfuse { use crate::v1::langfuse_trace::ChatTrace; lf.emit_chat(ChatTrace { - provider: "opencode".into(), - model: "claude-opus-4-7".into(), + provider: "ollama".into(), + model: "qwen3.5:latest".into(), input: vec![crate::v1::Message::new_text("user", prompt.clone())], output: correction_text.clone(), prompt_tokens: resp.usage.prompt_tokens, completion_tokens: resp.usage.completion_tokens, temperature: Some(0.1), max_tokens: None, - think: Some(true), + think: Some(false), start_time: start_time.to_rfc3339(), end_time: end_time.to_rfc3339(), latency_ms, @@ -664,7 +733,7 @@ impl ExecutionLoop { // Append to the transcript so the next executor turn sees it. self.append(LogEntry::new( - turn, "system", "claude-opus-4-7", "overseer_correction", + turn, "system", "qwen3.5:latest", "overseer_correction", serde_json::json!({ "reason": reason, "correction": correction_text, @@ -686,7 +755,7 @@ impl ExecutionLoop { "task_class": self.req.task_class, "operation": self.req.operation, "reason": reason, - "model": "claude-opus-4-7", + "model": "qwen3.5:latest", "correction": correction_text, "applied_at_turn": turn, "kb_context_used": kb, @@ -988,6 +1057,119 @@ async fn append_outcomes_row_at( /// PORT FROM orchestrator.ts:306-311. Cap `rows` at 20 entries and /// annotate the truncation so the executor sees it on the next turn +/// Inspect a tool result JSON payload and return the appropriate +/// audit `result` field per spec §3.2 enumeration: +/// "success" | "denied" | "not_found" | "error". +/// +/// Heuristic on the top-level object: +/// - body has `"error"` key (truthy) → "error" +/// - body has `"denied"` key (truthy) → "denied" +/// - body has `"not_found"` key (truthy) → "not_found" +/// - body has `"status":"denied"`/etc → that status +/// - otherwise → "success" +/// +/// Conservative — only flags explicit signals, defaults to success. +/// Keeps audit log honest about ambiguous payloads (an error inside +/// rows[*] doesn't downgrade the whole call to "error" — that's a +/// row-level concern, not a result-state concern). +pub(crate) fn audit_result_state(v: &serde_json::Value) -> String { + if let Some(obj) = v.as_object() { + // Tighter than before (2026-05-03 opus scrum INFO at + // execution_loop:1085): only flag error/denied/not_found when + // the payload doesn't ALSO have a rows/data/results sibling. + // A successful tool whose response includes an "error" field + // describing one row's problem (out of many) shouldn't taint + // the whole call as "error" — that's a row-level concern, not + // a result-state concern. + let has_data_sibling = obj.contains_key("rows") + || obj.contains_key("data") + || obj.contains_key("results"); + if !has_data_sibling { + if obj.get("error").map(is_truthy).unwrap_or(false) { + return "error".into(); + } + if obj.get("denied").map(is_truthy).unwrap_or(false) { + return "denied".into(); + } + if obj.get("not_found").map(is_truthy).unwrap_or(false) { + return "not_found".into(); + } + } + // status is always authoritative — if a tool explicitly says + // status=denied that's not a row-level concern. + if let Some(s) = obj.get("status").and_then(|v| v.as_str()) { + match s { + "denied" | "not_found" | "error" => return s.into(), + _ => {} + } + } + } + "success".into() +} + +fn is_truthy(v: &serde_json::Value) -> bool { + match v { + serde_json::Value::Null => false, + serde_json::Value::Bool(b) => *b, + serde_json::Value::String(s) => !s.is_empty(), + serde_json::Value::Array(a) => !a.is_empty(), + serde_json::Value::Object(o) => !o.is_empty(), + serde_json::Value::Number(n) => n.as_f64().map(|f| f != 0.0).unwrap_or(false), + } +} + +/// Walk a tool-result JSON value recursively. For each object that +/// contains a candidate_id (string) or worker_id (int/string), append +/// (canonical_candidate_id, fields_in_object) to `out`. Conventions +/// match the backfill binary: bare candidate_id stays as-is; worker_id +/// gets prefixed with "WORKER-". +/// +/// fields_in_object captures the OTHER scalar field names found in the +/// same object — so the audit row records "this access surfaced +/// {name, phone} for candidate X". The id field itself is NOT included +/// in fields_accessed (it's the subject identifier, not PII data). +pub(crate) fn collect_subject_hits( + v: &serde_json::Value, + out: &mut Vec<(String, Vec)>, +) { + match v { + serde_json::Value::Object(map) => { + // Detect a subject identifier in this object's direct fields. + let mut subject_id: Option = None; + if let Some(cid) = map.get("candidate_id").and_then(|v| v.as_str()) { + if !cid.is_empty() { subject_id = Some(cid.to_string()); } + } else if let Some(wid) = map.get("worker_id") { + if let Some(s) = wid.as_str() { + if !s.is_empty() { subject_id = Some(format!("WORKER-{s}")); } + } else if let Some(n) = wid.as_i64() { + subject_id = Some(format!("WORKER-{n}")); + } else if let Some(n) = wid.as_u64() { + subject_id = Some(format!("WORKER-{n}")); + } + } + if let Some(cid) = subject_id { + let fields: Vec = map + .keys() + .filter(|k| *k != "candidate_id" && *k != "worker_id") + .cloned() + .collect(); + out.push((cid, fields)); + } + // Recurse — nested objects may carry additional subjects + // (e.g. join results with both candidate + recruiter rows). + for v in map.values() { + collect_subject_hits(v, out); + } + } + serde_json::Value::Array(arr) => { + for v in arr { + collect_subject_hits(v, out); + } + } + _ => {} + } +} + /// prompt — prevents a 1000-row hybrid_search result from wiping the /// context budget on a single tool call. fn trim_result(r: &serde_json::Value) -> serde_json::Value { @@ -1866,4 +2048,123 @@ mod tests { let p = build_reviewer_prompt(&req, &log); assert!(!p.contains("HARD RULE")); } + + // --- Subject-audit hook tests --- + + #[test] + fn collect_subject_hits_finds_candidate_id_strings() { + let v = serde_json::json!({ + "rows": [ + {"candidate_id": "CAND-001", "name": "x", "phone": "y"}, + {"candidate_id": "CAND-002", "name": "z"}, + ] + }); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 2); + assert_eq!(hits[0].0, "CAND-001"); + // Field order from JSON object is preserved in serde_json::Map. + assert!(hits[0].1.contains(&"name".to_string())); + assert!(hits[0].1.contains(&"phone".to_string())); + assert!(!hits[0].1.contains(&"candidate_id".to_string())); + assert_eq!(hits[1].0, "CAND-002"); + } + + #[test] + fn collect_subject_hits_prefixes_worker_id_int() { + let v = serde_json::json!({"rows": [{"worker_id": 42, "name": "Alice"}]}); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 1); + assert_eq!(hits[0].0, "WORKER-42"); + assert_eq!(hits[0].1, vec!["name".to_string()]); + } + + #[test] + fn collect_subject_hits_handles_worker_id_string() { + let v = serde_json::json!({"worker_id": "42", "name": "Bob"}); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 1); + assert_eq!(hits[0].0, "WORKER-42"); + } + + #[test] + fn collect_subject_hits_recurses_through_nested_objects() { + let v = serde_json::json!({ + "outer": { + "candidate_id": "CAND-A", + "linked": { + "worker_id": 7, + "role": "welder" + } + } + }); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 2); + let ids: Vec<&str> = hits.iter().map(|(c, _)| c.as_str()).collect(); + assert!(ids.contains(&"CAND-A")); + assert!(ids.contains(&"WORKER-7")); + } + + #[test] + fn collect_subject_hits_ignores_objects_without_id_fields() { + let v = serde_json::json!({"rows": [{"name": "x"}, {"role": "y"}]}); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 0); + } + + #[test] + fn collect_subject_hits_skips_empty_id_strings() { + let v = serde_json::json!({"rows": [{"candidate_id": "", "name": "x"}]}); + let mut hits = Vec::new(); + collect_subject_hits(&v, &mut hits); + assert_eq!(hits.len(), 0); + } + + #[test] + fn audit_result_state_default_is_success() { + let v = serde_json::json!({"rows": [{"candidate_id": "X"}]}); + assert_eq!(audit_result_state(&v), "success"); + } + + #[test] + fn audit_result_state_detects_error_payload() { + let v = serde_json::json!({"error": "not found"}); + assert_eq!(audit_result_state(&v), "error"); + } + + #[test] + fn audit_result_state_detects_status_field() { + let v = serde_json::json!({"status": "denied", "rows": []}); + assert_eq!(audit_result_state(&v), "denied"); + } + + #[test] + fn audit_result_state_ignores_falsy_signals() { + // Empty/null/false should not trigger error state. + let v = serde_json::json!({"error": null, "denied": false, "rows": [{"candidate_id":"X"}]}); + assert_eq!(audit_result_state(&v), "success"); + } + + #[test] + fn audit_result_state_does_not_classify_error_when_data_sibling_present() { + // 2026-05-03 opus scrum INFO at execution_loop:1085 regression: + // a tool that returns rows BUT also has an "error" field describing + // one bad row shouldn't taint the whole call as "error". + let v = serde_json::json!({ + "rows": [{"candidate_id": "OK"}, {"candidate_id": "BAD"}], + "error": "1 row had a malformed certification" + }); + assert_eq!(audit_result_state(&v), "success"); + } + + #[test] + fn audit_result_state_status_is_authoritative_even_with_data() { + // Explicit status="denied" wins over data-sibling rule. + let v = serde_json::json!({"status": "denied", "rows": []}); + assert_eq!(audit_result_state(&v), "denied"); + } } diff --git a/crates/gateway/src/main.rs b/crates/gateway/src/main.rs index ff923d0..8685d9b 100644 --- a/crates/gateway/src/main.rs +++ b/crates/gateway/src/main.rs @@ -255,6 +255,33 @@ async fn main() { // surface over the existing aibridge → Ollama path. Future // phases add provider adapters (39), routing engine (40), // session state (41), etc. All without changing this mount. + ; + + // Build subject-audit writer BEFORE the V1State literal so the + // validate_workers wrapper can reference it. None = audit disabled + // (key file missing or unreadable); the gateway still serves PII + // paths but produces no audit trail per spec §3.2. + let subject_audit_writer: Option> = { + let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY") + .unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into()); + match catalogd::subject_audit::SubjectAuditWriter::from_key_file( + store.clone(), std::path::Path::new(&key_path) + ).await { + Ok(w) => { + tracing::info!("v1: subject audit log enabled (key: {})", key_path); + Some(std::sync::Arc::new(w)) + } + Err(e) => { + tracing::warn!( + "v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)", + e + ); + None + } + } + }; + + app = app .nest("/v1", v1::router(v1::V1State { ai_client: ai_client.clone(), usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())), @@ -334,7 +361,7 @@ async fn main() { let path_str = std::env::var("LH_WORKERS_PARQUET") .unwrap_or_else(|_| "/home/profit/lakehouse/data/datasets/workers_500k.parquet".into()); let path = std::path::Path::new(&path_str); - if path.exists() { + let raw: std::sync::Arc = if path.exists() { match validator::staffing::parquet_lookup::load_workers_parquet(path) { Ok(lookup) => { tracing::info!("v1: workers parquet loaded from {} — /v1/validate worker-existence checks enabled", path_str); @@ -348,7 +375,16 @@ async fn main() { } else { tracing::warn!("v1: workers parquet at {} not found — /v1/validate worker-existence checks will fail Consistency", path_str); std::sync::Arc::new(validator::InMemoryWorkerLookup::new()) - } + }; + // Wrap with subject-audit decorator. When subject_audit_writer + // is None, the wrapper is a transparent passthrough — no + // audit, no overhead. When Some, every find() call fires an + // audit row via the per-subject HMAC chain. (Step 5 of + // SUBJECT_MANIFESTS_ON_CATALOGD spec.) + std::sync::Arc::new(v1::auditing_worker_lookup::AuditingWorkerLookup::new( + raw, + subject_audit_writer.clone(), + )) }, // Phase 40 early deliverable — Langfuse trace emitter. // Defaults match mcp-server/tracing.ts conventions so @@ -378,8 +414,59 @@ async fn main() { } s }, + // subject_audit was constructed BEFORE the V1State literal + // (see `subject_audit_writer` above) so validate_workers + // could wrap with it. Reuse the same Arc here. + subject_audit: subject_audit_writer.clone(), })); + // Step 6 of SUBJECT_MANIFESTS_ON_CATALOGD spec — /audit/subject/{id} + // legal-tier endpoint. Only mounted when BOTH the audit writer AND + // the legal token file are present. Disabled-by-default keeps the + // audit surface from accidentally serving without proper credentials + // in dev / bring-up environments. + if let Some(writer) = subject_audit_writer.clone() { + let legal_token_path = std::env::var("LH_LEGAL_AUDIT_TOKEN_FILE") + .unwrap_or_else(|_| "/etc/lakehouse/legal_audit.token".into()); + let audit_state = catalogd::audit_endpoint::AuditEndpointState::new( + registry.clone(), + writer.clone(), + std::path::Path::new(&legal_token_path), + ).await; + app = app.nest("/audit", catalogd::audit_endpoint::router(audit_state)); + tracing::info!("audit endpoint mounted at /audit (legal token: {})", legal_token_path); + + // Phase 1.6 Gate 3 — biometric photo upload endpoint. Same + // legal-token gate as /audit (writes to legal-tier storage are + // privileged). Storage root defaults to ./data/biometric/uploads + // unless overridden via LH_BIOMETRIC_STORAGE_ROOT (operators on + // multi-host deployments will want a dedicated mount point). + let biometric_storage_root: std::path::PathBuf = std::env::var("LH_BIOMETRIC_STORAGE_ROOT") + .map(std::path::PathBuf::from) + .unwrap_or_else(|_| std::path::PathBuf::from("./data/biometric/uploads")); + // consent_versions allowlist — counsel-tier deployment safety. + // Defaults to /etc/lakehouse/consent_versions.json; override + // via LH_CONSENT_VERSIONS_FILE. Missing file → permissive + // mode (any non-empty hash accepted, the v1 behavior). + let consent_versions_path: std::path::PathBuf = std::env::var("LH_CONSENT_VERSIONS_FILE") + .map(std::path::PathBuf::from) + .unwrap_or_else(|_| std::path::PathBuf::from("/etc/lakehouse/consent_versions.json")); + let biometric_state = catalogd::biometric_endpoint::BiometricEndpointState::new( + registry.clone(), + writer, + std::path::Path::new(&legal_token_path), + biometric_storage_root.clone(), + ).await + .with_consent_versions(&consent_versions_path).await; + app = app.nest("/biometric", catalogd::biometric_endpoint::router(biometric_state)); + tracing::info!( + "biometric endpoint mounted at /biometric (storage_root: {}, legal token: {}, consent_versions file: {})", + biometric_storage_root.display(), legal_token_path, consent_versions_path.display() + ); + } else { + tracing::warn!("/audit + /biometric endpoints NOT mounted — subject_audit writer is None (no signing key)"); + } + // Auth middleware (if enabled) — P5-001 fix 2026-04-23: // previously only inserted the ApiKey as an extension and never layered // the middleware, so auth.enabled=true enforced nothing. Now wraps the diff --git a/crates/gateway/src/v1/auditing_worker_lookup.rs b/crates/gateway/src/v1/auditing_worker_lookup.rs new file mode 100644 index 0000000..c454ce9 --- /dev/null +++ b/crates/gateway/src/v1/auditing_worker_lookup.rs @@ -0,0 +1,194 @@ +//! Validator WorkerLookup decorator that fires per-find audit rows. +//! +//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5. +//! +//! Wraps an inner `dyn WorkerLookup`. Every `find(candidate_id)` call +//! produces one `SubjectAuditRow` via the bundled `SubjectAuditWriter`, +//! tagged with `purpose="validator_worker_lookup"`. Audit-write failures +//! are logged only — never propagated to the caller (per spec §3.2: +//! audit-side gaps are preferable to blocking validation). +//! +//! **Why a decorator and not a trait extension**: `WorkerLookup` is a +//! sync trait by design (validator's contract is "in-memory snapshot, +//! no per-call I/O"). Audit writes are async. The decorator bridges +//! them by spawning the audit append onto the current Tokio runtime +//! handle. Per-subject sequencing inside the writer ensures the chain +//! stays intact even with spawned writes (regression-tested at +//! catalogd::subject_audit::tests::concurrent_appends_to_same_subject_serialize). + +use catalogd::subject_audit::SubjectAuditWriter; +use shared::types::{AuditAccessor, SubjectAuditRow}; +use std::sync::Arc; +use validator::{WorkerLookup, WorkerRecord}; + +pub struct AuditingWorkerLookup { + inner: Arc, + /// None = audit disabled. find() still runs; just no audit row. + audit: Option>, + /// Cached Tokio handle so the sync find() path can spawn audit + /// writes onto the right runtime. None at construction = audit + /// is no-op (no runtime context to dispatch into). + runtime_handle: Option, +} + +impl AuditingWorkerLookup { + /// Wrap an inner lookup with optional audit. If `audit` is None, + /// behavior is identical to the inner — no overhead, no audit. + /// MUST be constructed inside a Tokio runtime context (uses + /// `Handle::try_current` to capture the handle). + pub fn new(inner: Arc, audit: Option>) -> Self { + let runtime_handle = if audit.is_some() { + tokio::runtime::Handle::try_current().ok() + } else { + None + }; + if audit.is_some() && runtime_handle.is_none() { + tracing::warn!( + "AuditingWorkerLookup: audit enabled but no Tokio runtime — audit writes disabled" + ); + } + Self { inner, audit, runtime_handle } + } +} + +impl WorkerLookup for AuditingWorkerLookup { + fn find(&self, candidate_id: &str) -> Option { + let result = self.inner.find(candidate_id); + // Fire audit AFTER the find so the result-state field reflects + // the actual outcome ("success" if found, "not_found" if not). + if let (Some(audit), Some(handle)) = (self.audit.clone(), self.runtime_handle.clone()) { + let cid = candidate_id.to_string(); + let result_state = if result.is_some() { "success" } else { "not_found" }; + // Single field: "exists" — validator only consumes existence + // + role/city/state from the WorkerRecord. We don't enumerate + // every WorkerRecord field as fields_accessed because that + // would imply they were USED by the caller; the validator + // only proves existence + then reads policy fields downstream. + let row = SubjectAuditRow { + schema: "subject_audit.v1".into(), + ts: chrono::Utc::now(), + candidate_id: cid.clone(), + accessor: AuditAccessor { + kind: "validator_lookup".into(), + daemon: "gateway".into(), + purpose: "validator_worker_lookup".into(), + trace_id: String::new(), + }, + fields_accessed: vec!["exists".into()], + result: result_state.into(), + prev_chain_hash: String::new(), + row_hmac: String::new(), + }; + // Spawn fire-and-forget. Chain integrity is preserved by the + // writer's per-subject Mutex (regression-tested), so spawning + // is safe. Audit-write errors are logged only. + handle.spawn(async move { + if let Err(e) = audit.append(row).await { + tracing::warn!("validator audit write failed for {cid}: {e}"); + } + }); + } + result + } + + fn len(&self) -> usize { + self.inner.len() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use catalogd::subject_audit::SubjectAuditWriter; + use object_store::memory::InMemory; + use validator::InMemoryWorkerLookup; + + fn fixture_inner_with(records: Vec) -> Arc { + Arc::new(InMemoryWorkerLookup::from_records(records)) + } + + fn fixture_record(candidate_id: &str) -> WorkerRecord { + WorkerRecord { + candidate_id: candidate_id.into(), + name: "Test".into(), + status: "active".into(), + city: Some("Chicago".into()), + state: Some("IL".into()), + role: Some("Welder".into()), + blacklisted_clients: vec![], + } + } + + #[tokio::test] + async fn find_existing_subject_writes_success_audit_row() { + let inner = fixture_inner_with(vec![fixture_record("CAND-EXIST")]); + let store: Arc = Arc::new(InMemory::new()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key( + store.clone(), + (0u8..32).collect(), + )); + let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone())); + let r = lookup.find("CAND-EXIST"); + assert!(r.is_some()); + // Audit is spawned — give it a tick to land. + tokio::task::yield_now().await; + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + let count = writer.verify_chain("CAND-EXIST").await.unwrap(); + assert_eq!(count, 1); + } + + #[tokio::test] + async fn find_missing_subject_writes_not_found_audit_row() { + let inner = fixture_inner_with(vec![]); + let store: Arc = Arc::new(InMemory::new()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key( + store.clone(), + (0u8..32).collect(), + )); + let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone())); + let r = lookup.find("CAND-PHANTOM"); + assert!(r.is_none()); + tokio::task::yield_now().await; + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + // verify_chain returns the row count for that subject (1 row, + // result="not_found"). We don't assert on result text here — + // that's covered by the hand-rolled JSON inspection below. + assert_eq!(writer.verify_chain("CAND-PHANTOM").await.unwrap(), 1); + } + + #[tokio::test] + async fn audit_disabled_means_no_writes_no_overhead() { + let inner = fixture_inner_with(vec![fixture_record("CAND-A")]); + let lookup = AuditingWorkerLookup::new(inner, None); + // Just confirm find() still works with no audit — no panic, no + // runtime-handle requirement. + assert!(lookup.find("CAND-A").is_some()); + assert!(lookup.find("CAND-MISSING").is_none()); + assert_eq!(lookup.len(), 1); + } + + #[tokio::test] + async fn many_finds_to_same_subject_produce_intact_chain() { + // Same regression as catalogd's concurrent test, but exercised + // through the validator wrapper path. 30 sequential finds on + // the SAME subject should produce an intact 30-row chain. + let inner = fixture_inner_with(vec![fixture_record("CAND-CHAIN")]); + let store: Arc = Arc::new(InMemory::new()); + let writer = Arc::new(SubjectAuditWriter::with_inline_key( + store.clone(), + (0u8..32).collect(), + )); + let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone())); + for _ in 0..30 { + let _ = lookup.find("CAND-CHAIN"); + } + // Wait for all spawns to finish — the runtime is currently + // single-threaded test runtime, so a few yields drains. + for _ in 0..50 { + tokio::task::yield_now().await; + } + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + let count = writer.verify_chain("CAND-CHAIN").await.unwrap(); + assert_eq!(count, 30, "chain corrupted under sequential spawn load"); + } +} diff --git a/crates/gateway/src/v1/kimi.rs b/crates/gateway/src/v1/kimi.rs index 9ff2b7e..d23d00d 100644 --- a/crates/gateway/src/v1/kimi.rs +++ b/crates/gateway/src/v1/kimi.rs @@ -140,7 +140,7 @@ pub async fn chat( model, choices: vec![Choice { index: 0, - message: Message { role: "assistant".into(), content: serde_json::Value::String(text) }, + message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() }, finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()), }], usage: UsageBlock { diff --git a/crates/gateway/src/v1/mod.rs b/crates/gateway/src/v1/mod.rs index 6ab9ef0..b604587 100644 --- a/crates/gateway/src/v1/mod.rs +++ b/crates/gateway/src/v1/mod.rs @@ -20,6 +20,7 @@ pub mod kimi; pub mod opencode; pub mod validate; pub mod iterate; +pub mod auditing_worker_lookup; pub mod langfuse_trace; pub mod session_log; pub mod mode; @@ -91,6 +92,12 @@ pub struct V1State { /// `[validatord].session_log_path` (commit 1a3a82a in /// golangLAKEHOUSE). pub session_log: Option, + /// Per-subject audit log writer (HMAC-chained JSONL). Loaded at + /// startup from /etc/lakehouse/subject_audit.key if the file is + /// present and ≥32 bytes. None = audit disabled with a warning + /// at startup; PII access is still served but produces no audit + /// trail. See docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md. + pub subject_audit: Option>, } #[derive(Default, Clone, Serialize)] @@ -141,10 +148,22 @@ pub fn router(state: V1State) -> Router { /// providers can take it verbatim. `Message::text()` flattens for /// places that need a plain string (Ollama prompt assembly, char /// counts, the assistant's own response synthesis). -#[derive(Serialize, Deserialize, Clone, Debug)] +#[derive(Serialize, Deserialize, Clone, Debug, Default)] pub struct Message { pub role: String, + /// `Value::Null` is legal: an assistant turn that only makes tool calls. + #[serde(default)] pub content: serde_json::Value, + /// OpenAI tool-calling fields (2026-09-19). Passed through verbatim on the OpenRouter lane; + /// omitted from JSON when unset, so plain-text traffic is byte-identical to before. Before + /// this, serde silently DROPPED them — tool-using agents routed through :3100 never saw their + /// tools and fabricated tool output (autobun KB finding infra.lakehouse_gateway.tools). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool_calls: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool_call_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub name: Option, } impl Message { @@ -156,6 +175,7 @@ impl Message { Self { role: role.into(), content: serde_json::Value::String(body.into()), + ..Default::default() } } /// Flatten content to a plain string. Strings pass through; content- @@ -188,9 +208,8 @@ pub struct ChatRequest { pub temperature: Option, #[serde(default)] pub max_tokens: Option, - /// Accepted for shape-compat but ignored in the thin slice — - /// Phase 38 returns non-streaming even when the client asked for it. - /// Phase 39+ wires real streaming. + /// Real SSE passthrough on the openrouter lane (2026-09-20); other lanes reject + /// stream=true with a 400 rather than silently returning a non-streamed body. #[serde(default)] pub stream: Option, /// Non-OpenAI extension. Passes through to the provider's thinking @@ -208,6 +227,21 @@ pub struct ChatRequest { /// from the model name; Phase 38/39 requires explicit selection. #[serde(default)] pub provider: Option, + /// OpenAI tool definitions + choice (2026-09-19). Forwarded on the OpenRouter lane only; + /// any other provider REJECTS them with 400 instead of silently stripping them. + #[serde(default)] + pub tools: Option, + #[serde(default)] + pub tool_choice: Option, +} + +impl ChatRequest { + /// True if the request uses tool calling anywhere (definitions or tool-call history). + pub fn uses_tools(&self) -> bool { + self.tools.is_some() + || self.tool_choice.is_some() + || self.messages.iter().any(|m| m.role == "tool" || m.tool_calls.is_some() || m.tool_call_id.is_some()) + } } #[derive(Serialize)] @@ -312,9 +346,47 @@ mod resolve_provider_tests { stream: None, think: None, provider: provider.map(|s| s.to_string()), + tools: None, + tool_choice: None, } } + /// Streaming is a per-lane capability, and the routing decision must be visible in a test: + /// before 2026-09-20 the gateway accepted stream=true on every lane and returned a single + /// JSON body, so a client waiting for SSE frames hung until its own timeout. + #[test] + fn streaming_is_openrouter_only() { + let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash"); + r.stream = Some(true); + assert_eq!(resolve_provider(&r).0, "openrouter", "openrouter lane streams"); + + let mut local = mk_req(None, "qwen3.5:latest"); + local.stream = Some(true); + assert_eq!( + resolve_provider(&local).0, + "ollama", + "a local model resolves to ollama, where the handler rejects stream=true with a 400 \ + instead of silently returning a non-streamed body" + ); + } + + #[test] + fn uses_tools_detects_definitions_and_history() { + let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash"); + assert!(!r.uses_tools()); + r.tools = Some(serde_json::json!([])); + assert!(r.uses_tools()); + let mut r2 = mk_req(None, "qwen3.5:latest"); + r2.messages.push(Message { role: "tool".into(), content: "ok".into(), tool_call_id: Some("c1".into()), ..Default::default() }); + assert!(r2.uses_tools()); + } + + #[test] + fn plain_message_serialization_unchanged() { + let m = Message::new_text("user", "hi"); + assert_eq!(serde_json::to_string(&m).unwrap(), r#"{"role":"user","content":"hi"}"#); + } + #[test] fn explicit_provider_wins() { let r = mk_req(Some("openrouter"), "qwen3.5:latest"); @@ -371,19 +443,27 @@ async fn chat( State(state): State, headers: axum::http::HeaderMap, Json(req): Json, -) -> Result, (StatusCode, String)> { +) -> Result { + use axum::response::IntoResponse as _; if req.messages.is_empty() { return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into())); } - if req.stream.unwrap_or(false) { - tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming"); - } - // Provider resolution: explicit `req.provider` wins; otherwise // infer from a model-name prefix. Phase 39 PRD gate example: // `model: "openrouter/openai/gpt-4o-mini"` → provider "openrouter", // adapter gets the stripped "openai/gpt-4o-mini". let (provider, effective_model) = resolve_provider(&req); + // Tool calling is only wired on the OpenRouter lane. Refuse loudly elsewhere — silently + // dropping tools made agents fabricate tool results (2026-09-19). + if req.uses_tools() && !matches!(provider.as_str(), "openrouter" | "openrouter_free") { + return Err(( + StatusCode::BAD_REQUEST, + format!( + "tools/tool_calls are only supported on the openrouter lane via this gateway \ + (resolved provider: {provider}); use an OpenRouter model id (e.g. openrouter//)" + ), + )); + } let start_time = chrono::Utc::now(); let start_instant = std::time::Instant::now(); @@ -399,6 +479,45 @@ async fn chat( std::borrow::Cow::Owned(cloned) }; + // STREAMING (2026-09-20). Phase 38 accepted `stream: true` and returned a single JSON body; + // a client waiting for SSE frames hangs and then reports "no response". Forward the upstream + // SSE verbatim on the lane that supports it, and say so plainly on the lanes that do not + // instead of pretending. Streamed calls skip usage metering + Langfuse (see chat_stream). + if req.stream.unwrap_or(false) { + if !matches!(provider.as_str(), "openrouter" | "openrouter_free") { + return Err(( + StatusCode::BAD_REQUEST, + format!( + "stream=true is only supported on the openrouter lane via this gateway \ + (resolved provider: {provider}); resend with stream=false or use an \ + OpenRouter model id" + ), + )); + } + let key = state.openrouter_key.as_deref().ok_or(( + StatusCode::SERVICE_UNAVAILABLE, + "OPENROUTER_API_KEY not configured".to_string(), + ))?; + let upstream = openrouter::chat_stream(key, &*req_for_adapter) + .await + .map_err(|e| (StatusCode::BAD_GATEWAY, format!("openrouter stream: {e}")))?; + tracing::info!( + target: "v1.chat", + provider = %provider, + model = %effective_model, + "streaming passthrough — usage metering and Langfuse are skipped for this call", + ); + let body = axum::body::Body::from_stream(upstream.bytes_stream()); + return Ok(( + [ + (axum::http::header::CONTENT_TYPE, "text/event-stream"), + (axum::http::header::CACHE_CONTROL, "no-cache"), + ], + body, + ) + .into_response()); + } + let (resp, used_provider) = match provider.as_str() { "ollama" | "local" | "" => { let r = ollama::chat(&state.ai_client, &*req_for_adapter) @@ -582,7 +701,7 @@ async fn chat( provider_usage.total_tokens += resp.usage.total_tokens as u64; } - Ok(Json(resp)) + Ok(Json(resp).into_response()) } async fn usage(State(state): State) -> impl IntoResponse { diff --git a/crates/gateway/src/v1/ollama.rs b/crates/gateway/src/v1/ollama.rs index 240d8da..b6336ba 100644 --- a/crates/gateway/src/v1/ollama.rs +++ b/crates/gateway/src/v1/ollama.rs @@ -113,10 +113,10 @@ mod tests { #[test] fn flatten_separates_system_from_turns() { let msgs = vec![ - Message { role: "system".into(), content: "Rules here.".into() }, - Message { role: "user".into(), content: "Q1".into() }, - Message { role: "assistant".into(), content: "A1".into() }, - Message { role: "user".into(), content: "Q2".into() }, + Message { role: "system".into(), content: "Rules here.".into(), ..Default::default() }, + Message { role: "user".into(), content: "Q1".into(), ..Default::default() }, + Message { role: "assistant".into(), content: "A1".into(), ..Default::default() }, + Message { role: "user".into(), content: "Q2".into(), ..Default::default() }, ]; let (system, prompt) = flatten_messages(&msgs); assert_eq!(system, "Rules here."); @@ -129,9 +129,9 @@ mod tests { #[test] fn flatten_concatenates_multiple_system_messages() { let msgs = vec![ - Message { role: "system".into(), content: "First.".into() }, - Message { role: "system".into(), content: "Second.".into() }, - Message { role: "user".into(), content: "Hi".into() }, + Message { role: "system".into(), content: "First.".into(), ..Default::default() }, + Message { role: "system".into(), content: "Second.".into(), ..Default::default() }, + Message { role: "user".into(), content: "Hi".into(), ..Default::default() }, ]; let (system, _) = flatten_messages(&msgs); assert_eq!(system, "First.\nSecond."); @@ -139,7 +139,7 @@ mod tests { #[test] fn flatten_with_no_system_returns_empty_system() { - let msgs = vec![Message { role: "user".into(), content: "hi".into() }]; + let msgs = vec![Message { role: "user".into(), content: "hi".into(), ..Default::default() }]; let (system, prompt) = flatten_messages(&msgs); assert!(system.is_empty()); assert!(prompt.contains("user: hi")); @@ -147,11 +147,11 @@ mod tests { #[test] fn estimate_tokens_chars_div_4_ceiling() { - let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }]; + let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into(), ..Default::default() }]; // 8 chars / 4 = 2, with ceiling → 2 assert_eq!(estimate_prompt_tokens(&msgs), 2); - let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }]; + let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into(), ..Default::default() }]; // 9 chars → (9+3)/4 = 3 (ceiling) assert_eq!(estimate_prompt_tokens(&msgs2), 3); } diff --git a/crates/gateway/src/v1/opencode.rs b/crates/gateway/src/v1/opencode.rs index d45abf7..3786fae 100644 --- a/crates/gateway/src/v1/opencode.rs +++ b/crates/gateway/src/v1/opencode.rs @@ -145,7 +145,7 @@ pub async fn chat( model, choices: vec![Choice { index: 0, - message: Message { role: "assistant".into(), content: serde_json::Value::String(text) }, + message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() }, finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()), }], usage: UsageBlock { diff --git a/crates/gateway/src/v1/openrouter.rs b/crates/gateway/src/v1/openrouter.rs index 610c5eb..dd490d7 100644 --- a/crates/gateway/src/v1/openrouter.rs +++ b/crates/gateway/src/v1/openrouter.rs @@ -20,7 +20,19 @@ use serde::{Deserialize, Serialize}; use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock}; const OR_BASE_URL: &str = "https://openrouter.ai/api/v1"; -const OR_TIMEOUT_SECS: u64 = 180; +// Default request ceiling. 180s starves long-context calls (kimi_architect's +// ~180K-char second pass died at exactly this wall on 2026-09-02, surfacing +// as a misleading "error decoding response body") — override per host with +// OR_TIMEOUT_SECS. The auditor budgets up to 900s client-side. +const OR_TIMEOUT_SECS_DEFAULT: u64 = 180; + +fn or_timeout_secs() -> u64 { + std::env::var("OR_TIMEOUT_SECS") + .ok() + .and_then(|v| v.trim().parse::().ok()) + .filter(|&s| s > 0) + .unwrap_or(OR_TIMEOUT_SECS_DEFAULT) +} pub fn resolve_openrouter_key() -> Option { if let Ok(k) = std::env::var("OPENROUTER_API_KEY") { @@ -49,6 +61,52 @@ pub fn resolve_openrouter_key() -> Option { None } +/// Streaming passthrough (2026-09-20). The gateway used to accept `stream: true` and return a +/// single non-streamed JSON body — a client waiting for SSE frames hangs until its own timeout +/// and reports "no response" (this is what broke openclaw's gateway lane). We now forward the +/// upstream SSE byte-for-byte. +/// +/// The caller streams the body straight to its client, so nothing here parses the chunks: +/// usage metering and Langfuse tracing are SKIPPED for streamed calls, and the handler logs +/// that. Metering streamed responses needs the final usage chunk and belongs in its own change. +pub async fn chat_stream(key: &str, req: &ChatRequest) -> Result { + let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string(); + let body = ORChatBody { + model, + messages: req.messages.iter().map(|m| ORMessage { + role: m.role.clone(), + content: m.content.clone(), + tool_calls: m.tool_calls.clone(), + tool_call_id: m.tool_call_id.clone(), + name: m.name.clone(), + }).collect(), + tools: req.tools.clone(), + tool_choice: req.tool_choice.clone(), + max_tokens: Some(req.max_tokens.unwrap_or(800)), + temperature: req.temperature.unwrap_or(0.3), + stream: true, + }; + let client = reqwest::Client::builder() + .timeout(Duration::from_secs(or_timeout_secs())) + .build() + .map_err(|e| format!("build client: {e}"))?; + let resp = client + .post(format!("{}/chat/completions", OR_BASE_URL)) + .bearer_auth(key) + .header("HTTP-Referer", "https://vcp.devop.live") + .header("X-Title", "Lakehouse Scrum") + .json(&serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?) + .send() + .await + .map_err(|e| format!("openrouter.ai unreachable: {e}"))?; + if !resp.status().is_success() { + let code = resp.status(); + let text = resp.text().await.unwrap_or_else(|_| "?".into()); + return Err(format!("openrouter {code}: {}", text.chars().take(300).collect::())); + } + Ok(resp) +} + pub async fn chat( key: &str, req: &ChatRequest, @@ -57,7 +115,7 @@ pub async fn chat( // form so OpenRouter sees the raw model id (e.g. "openai/gpt-oss-120b:free"). let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string(); - let body = ORChatBody { + let mut body = ORChatBody { model: model.clone(), // Pass content through verbatim — preserves OpenAI's multimodal // content-parts shape (`[{type:"text",text:"..."}, ...]`) so the @@ -65,30 +123,78 @@ pub async fn chat( messages: req.messages.iter().map(|m| ORMessage { role: m.role.clone(), content: m.content.clone(), + tool_calls: m.tool_calls.clone(), + tool_call_id: m.tool_call_id.clone(), + name: m.name.clone(), }).collect(), - max_tokens: req.max_tokens.unwrap_or(800), + tools: req.tools.clone(), + tool_choice: req.tool_choice.clone(), + max_tokens: Some(req.max_tokens.unwrap_or(800)), temperature: req.temperature.unwrap_or(0.3), stream: false, }; let client = reqwest::Client::builder() - .timeout(Duration::from_secs(OR_TIMEOUT_SECS)) + .timeout(Duration::from_secs(or_timeout_secs())) .build() .map_err(|e| format!("build client: {e}"))?; let t0 = std::time::Instant::now(); - let resp = client - .post(format!("{}/chat/completions", OR_BASE_URL)) - .bearer_auth(key) - // OpenRouter recommends Referer + Title for attribution; absent - // headers do not fail the call but help us see our traffic in - // their dashboard. - .header("HTTP-Referer", "https://vcp.devop.live") - .header("X-Title", "Lakehouse Scrum") - .json(&body) - .send() - .await - .map_err(|e| format!("openrouter.ai unreachable: {e}"))?; + let send = |client: reqwest::Client, key: String, body_json: serde_json::Value| async move { + client + .post(format!("{}/chat/completions", OR_BASE_URL)) + .bearer_auth(key) + // OpenRouter recommends Referer + Title for attribution; absent + // headers do not fail the call but help us see our traffic in + // their dashboard. + .header("HTTP-Referer", "https://vcp.devop.live") + .header("X-Title", "Lakehouse Scrum") + .json(&body_json) + .send() + .await + .map_err(|e| format!("openrouter.ai unreachable: {e}")) + }; + let mut resp = send( + client.clone(), + key.to_string(), + serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?, + ) + .await?; + + // OpenRouter's affordability precheck reserves max_tokens × price against + // the account balance. Measured 2026-08-28: stripping the cap makes it + // WORSE (the precheck then prices the model's own maximum, e.g. 65536) — + // so a low-balance 402 surfaces verbatim and the fix is funding the + // account the key belongs to, not request-shape games. One clamp retry: + // when the error names an affordable budget ≥256 tokens, retry once at + // that budget so small answers still get through on a thin balance. + if resp.status() == reqwest::StatusCode::PAYMENT_REQUIRED && body.max_tokens.is_some() { + let err_body = resp.text().await.unwrap_or_else(|_| "?".into()); + let afford = err_body + .split("can only afford ") + .nth(1) + .and_then(|s| s.split(|c: char| !c.is_ascii_digit()).next()) + .and_then(|n| n.parse::().ok()); + match afford { + Some(n) if n >= 256 => { + tracing::warn!( + target: "v1.chat", + provider = "openrouter", + model = %model, + afford = n, + "402 affordability precheck — clamping max_tokens to the stated budget", + ); + body.max_tokens = Some(n); + resp = send( + client.clone(), + key.to_string(), + serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?, + ) + .await?; + } + _ => return Err(format!("openrouter.ai 402 Payment Required: {}", err_body)), + } + } let status = resp.status(); if !status.is_success() { @@ -102,7 +208,24 @@ pub async fn chat( let latency_ms = t0.elapsed().as_millis(); let choice = parsed.choices.into_iter().next() .ok_or_else(|| "openrouter returned no choices".to_string())?; - let text = choice.message.content; + let tool_calls = choice.message.tool_calls.clone(); + let reasoning_only = choice.message.content.is_none() + && choice.message.reasoning.is_some() + && tool_calls.is_none(); + let text = choice.message.content.unwrap_or_default(); + if reasoning_only { + // The model spent its whole budget thinking. Return well-formed empty + // content (callers' schema validation + retry budgets handle it + // deterministically) rather than a decode error that reads as a + // gateway fault — and never pass thinking-text off as an answer. + tracing::warn!( + target: "v1.chat", + provider = "openrouter", + model = %model, + finish = %choice.finish_reason.as_deref().unwrap_or("?"), + "reasoning-only response (content null) — raise max_tokens or use a non-reasoning model", + ); + } let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| { let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum(); @@ -129,7 +252,17 @@ pub async fn chat( model, choices: vec![Choice { index: 0, - message: Message { role: "assistant".into(), content: serde_json::Value::String(text) }, + message: Message { + role: "assistant".into(), + // OpenAI shape: a tool-call-only turn carries `content: null`. + content: if tool_calls.is_some() && text.is_empty() { + serde_json::Value::Null + } else { + serde_json::Value::String(text) + }, + tool_calls, + ..Default::default() + }, finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()), }], usage: UsageBlock { @@ -146,13 +279,27 @@ pub async fn chat( struct ORChatBody { model: String, messages: Vec, - max_tokens: u32, + #[serde(skip_serializing_if = "Option::is_none")] + max_tokens: Option, temperature: f64, stream: bool, + #[serde(skip_serializing_if = "Option::is_none")] + tools: Option, + #[serde(skip_serializing_if = "Option::is_none")] + tool_choice: Option, } #[derive(Serialize)] -struct ORMessage { role: String, content: serde_json::Value } +struct ORMessage { + role: String, + content: serde_json::Value, + #[serde(skip_serializing_if = "Option::is_none")] + tool_calls: Option, + #[serde(skip_serializing_if = "Option::is_none")] + tool_call_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + name: Option, +} #[derive(Deserialize)] struct ORChatResponse { @@ -169,7 +316,21 @@ struct ORChoice { } #[derive(Deserialize)] -struct ORMessageResp { content: String } +struct ORMessageResp { + // Reasoning models (GLM-5.x, deepseek-v4-pro) return `content: null` plus a + // `reasoning` field when they exhaust max_tokens mid-thought. A bare String + // here made serde reject the whole body ("invalid openrouter response", + // 2026-08-28) — which took down every reasoning model on the lane, + // including phase2's configured glm-5.2. Content stays the only text we + // ever return; reasoning is observability, never an answer. + #[serde(default)] + content: Option, + #[serde(default)] + reasoning: Option, + /// OpenAI tool calls requested by the model (passed back to the client verbatim). + #[serde(default)] + tool_calls: Option, +} #[derive(Deserialize)] struct ORUsage { prompt_tokens: u32, completion_tokens: u32 } @@ -178,6 +339,21 @@ struct ORUsage { prompt_tokens: u32, completion_tokens: u32 } mod tests { use super::*; + #[test] + fn reasoning_model_null_content_parses() { + // GLM-5.x / deepseek-v4-pro shape: content null + reasoning present + // (the 2026-08-28 decode failure). Must parse, not error. + let body = r#"{"choices":[{"message":{"role":"assistant","content":null,"reasoning":"thinking…"},"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":10}}"#; + let parsed: ORChatResponse = serde_json::from_str(body).expect("null content must parse"); + let choice = &parsed.choices[0]; + assert!(choice.message.content.is_none()); + assert_eq!(choice.message.reasoning.as_deref(), Some("thinking…")); + // And the normal shape still parses with reasoning absent. + let normal = r#"{"choices":[{"message":{"role":"assistant","content":"OK"},"finish_reason":"stop"}]}"#; + let parsed: ORChatResponse = serde_json::from_str(normal).expect("normal shape"); + assert_eq!(parsed.choices[0].message.content.as_deref(), Some("OK")); + } + #[test] fn resolve_openrouter_key_does_not_panic() { // Smoke test — all three sources may or may not be set depending @@ -190,17 +366,57 @@ mod tests { let body = ORChatBody { model: "openai/gpt-oss-120b:free".into(), messages: vec![ - ORMessage { role: "user".into(), content: "review this".into() }, + ORMessage { role: "user".into(), content: "review this".into(), + tool_calls: None, tool_call_id: None, name: None }, ], - max_tokens: 800, + max_tokens: Some(800), temperature: 0.3, stream: false, + tools: None, + tool_choice: None, }; let json = serde_json::to_string(&body).unwrap(); assert!(json.contains("\"model\":\"openai/gpt-oss-120b:free\"")); assert!(json.contains("\"messages\"")); assert!(json.contains("\"max_tokens\":800")); assert!(json.contains("\"stream\":false")); + // no tool fields on plain traffic (byte-compat with pre-2026-09-19 bodies) + assert!(!json.contains("tools") && !json.contains("tool_call")); + } + + #[test] + fn chat_body_forwards_tools_and_tool_messages() { + let tools = serde_json::json!([{"type":"function","function":{"name":"exec","parameters":{"type":"object"}}}]); + let body = ORChatBody { + model: "z-ai/glm-5.3-flash".into(), + messages: vec![ + ORMessage { role: "assistant".into(), content: serde_json::Value::Null, + tool_calls: Some(serde_json::json!([{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}])), + tool_call_id: None, name: None }, + ORMessage { role: "tool".into(), content: "ok".into(), tool_calls: None, + tool_call_id: Some("c1".into()), name: Some("exec".into()) }, + ], + max_tokens: Some(100), + temperature: 0.0, + stream: false, + tools: Some(tools), + tool_choice: Some(serde_json::json!("auto")), + }; + let json = serde_json::to_string(&body).unwrap(); + assert!(json.contains("\"tools\":[{")); + assert!(json.contains("\"tool_choice\":\"auto\"")); + assert!(json.contains("\"tool_call_id\":\"c1\"")); + assert!(json.contains("\"tool_calls\":[{")); + } + + #[test] + fn response_with_tool_calls_parses() { + let raw = r#"{"choices":[{"message":{"content":null,"tool_calls":[{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}]},"finish_reason":"tool_calls"}]}"#; + let parsed: ORChatResponse = serde_json::from_str(raw).unwrap(); + let c = &parsed.choices[0]; + assert!(c.message.content.is_none()); + assert!(c.message.tool_calls.is_some()); + assert_eq!(c.finish_reason.as_deref(), Some("tool_calls")); } #[test] diff --git a/crates/shared/src/types.rs b/crates/shared/src/types.rs index c22749a..c5e5572 100644 --- a/crates/shared/src/types.rs +++ b/crates/shared/src/types.rs @@ -364,3 +364,228 @@ pub enum Redaction { keep_suffix: usize, }, } + +// ─── Subject manifests (compliance/access metadata layer) ──────────── +// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md +// +// A SubjectManifest is a per-person record threading existing catalogd +// primitives (datasets, views, tombstones) together by subject token. +// Stored at _catalog/subjects/.json. NOT a separate +// daemon — extends catalogd's existing manifest pattern. +// +// The audit log per subject (HMAC-chained JSONL at +// _catalog/subjects/.audit.jsonl) is owned by a separate +// SubjectAuditWriter (Step 2) — not modeled here. + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum SubjectStatus { + PendingConsent, + Active, + Withdrawn, + RetentionExpired, + Erased, +} + +/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed: +/// the gateway treats Unknown as healthcare-equivalent until reclassified +/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1). +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum SubjectVertical { + Unknown, + General, + Healthcare, + Finance, + Other, +} + +impl Default for SubjectVertical { + fn default() -> Self { SubjectVertical::Unknown } +} + +/// State machine for general-PII consent. BIPA-specific biometric +/// consent is tracked separately on `BiometricConsent`. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum ConsentStatus { + /// Backfilled from existing data sources before consent was tracked. + /// SHOULD route as if no consent exists until reviewed. + PendingBackfillReview, + /// New subject; awaiting consent UX. + PendingFirstContact, + Given, + Withdrawn, + Expired, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum BiometricConsentStatus { + /// Default. No biometric data exists for this subject. + NeverCollected, + Pending, + Given, + Withdrawn, + Expired, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GeneralPiiConsent { + pub status: ConsentStatus, + /// Reference to the consent template version (matches a row in the + /// future consent_versions table). Empty when status is pending_*. + #[serde(default)] + pub version: String, + #[serde(default)] + pub given_at: Option>, + #[serde(default)] + pub withdrawn_at: Option>, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct BiometricConsent { + pub status: BiometricConsentStatus, + /// BIPA: max 3 years from last interaction. Implementation MUST + /// enforce daily expiration sweep against this field. + #[serde(default)] + pub retention_until: Option>, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SubjectConsent { + pub general_pii: GeneralPiiConsent, + pub biometric: BiometricConsent, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SubjectRetention { + pub general_pii_until: chrono::DateTime, + /// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026"). + #[serde(default)] + pub policy: String, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SubjectDatasetRef { + /// catalogd dataset name (must resolve to an existing manifest). + pub name: String, + /// Column in that dataset that holds the subject's identifier. + pub key_column: String, + /// Specific identifier value (typically same as candidate_id; differs + /// only when a dataset uses a foreign-key with a different format). + pub key_value: String, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SubjectManifest { + /// Schema version tag — always "subject_manifest.v1" for v1. + #[serde(default = "default_subject_manifest_schema")] + pub schema: String, + /// Subject identifier (the canonical token, e.g. "CAND-000001"). + pub candidate_id: String, + pub created_at: chrono::DateTime, + pub updated_at: chrono::DateTime, + pub status: SubjectStatus, + #[serde(default)] + pub vertical: SubjectVertical, + pub consent: SubjectConsent, + pub retention: SubjectRetention, + /// Datasets that contain rows for this subject. Used by audit-response + /// to know what to project. Each entry MUST reference an existing + /// catalogd dataset manifest (validated on put_subject). + pub datasets: Vec, + /// Names of existing AiViews that safely project this subject's data. + /// Used by service-tier readers (no PII exposure). + #[serde(default)] + pub safe_views: Vec, + /// Path (relative to repo root) of the per-subject audit JSONL. + /// Owned by SubjectAuditWriter (Step 2). Empty until first write. + #[serde(default)] + pub audit_log_path: String, + /// SHA-256 of the most recent HMAC chain checkpoint of the audit log. + /// Empty until first audit row written. + #[serde(default)] + pub audit_log_chain_root: String, + /// Biometric collection metadata. None until the first biometric + /// upload via Phase 1.6 Gate 3 photo endpoint. Cleared on erasure. + /// `serde(default)` so existing manifests parse without re-emit. + #[serde(default)] + pub biometric_collection: Option, +} + +/// Per-subject biometric collection record. Written by the Gate 3 +/// photo-upload endpoint when a candidate's photo is accepted under +/// `consent.biometric.status == "given"`. +/// +/// Reset to None by the BIPA erasure flow (destruction runbook). +/// Preserved across schema evolutions via the BTreeMap-canonicalized +/// HMAC chain — old audit rows verify regardless of new fields. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub struct BiometricCollection { + /// Quarantined filesystem path of the original photo bytes, + /// relative to the configured biometric storage root. NEVER under + /// `data/headshots/` (which is reserved for synthetic StyleGAN + /// faces). Format: `data/biometric/uploads//.`. + pub data_path: String, + /// SHA-256 of the original photo bytes. Used for integrity check + /// (file-on-disk matches what was uploaded), NOT for + /// re-derivation of the original from the hash. + pub template_hash: String, + pub collected_at: chrono::DateTime, + /// SHA-256 of the consent template version that was in force when + /// this photo was uploaded. Anchors the consent record to a + /// specific document version (per Phase 1.6 Gate 2 §6). + #[serde(default)] + pub consent_version_hash: String, + /// deepface classifications, when the offline classifier has run. + /// None until classifier integration ships (Gate 3b). + #[serde(default)] + pub classifications: Option, +} + +fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() } + +/// Audit accessor — who/why touched a subject's PII. +/// Persisted in subject_audit.v1 rows; never logged in plaintext outside +/// the per-subject audit file. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AuditAccessor { + /// One of: "gateway_lookup", "audit_response", "legal_request", + /// "system_resolve". Caller-defined; SubjectAuditWriter does not + /// validate the kind string. + pub kind: String, + /// Daemon name (e.g. "gateway", "validator", "catalogd"). + pub daemon: String, + /// Purpose token (e.g. "fill_validation", "audit_subject_response"). + pub purpose: String, + /// X-Lakehouse-Trace-Id for cross-daemon trace correlation. Empty + /// when the request didn't carry one. + #[serde(default)] + pub trace_id: String, +} + +/// One audit row in a subject's per-subject audit JSONL. +/// HMAC chain: row_hmac = HMAC-SHA256(key, prev_chain_hash || canon(row - row_hmac)) +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SubjectAuditRow { + /// Always "subject_audit.v1" for v1. + #[serde(default = "default_subject_audit_schema")] + pub schema: String, + pub ts: chrono::DateTime, + pub candidate_id: String, + pub accessor: AuditAccessor, + /// Field names accessed (e.g. ["name"], ["name", "phone"]). + pub fields_accessed: Vec, + /// "success" | "denied" | "not_found" | "error". + pub result: String, + /// Hash of the previous row in this subject's audit log. "GENESIS" + /// for the first row. + pub prev_chain_hash: String, + /// HMAC-SHA256 over (prev_chain_hash || canonical_json(row - row_hmac)). + /// Computed by SubjectAuditWriter; callers leave empty on input. + #[serde(default)] + pub row_hmac: String, +} + +fn default_subject_audit_schema() -> String { "subject_audit.v1".into() } diff --git a/docs/ADR-019-vector-storage.md b/docs/ADR-019-vector-storage.md index 0ed2fe2..95151c1 100644 --- a/docs/ADR-019-vector-storage.md +++ b/docs/ADR-019-vector-storage.md @@ -70,10 +70,33 @@ But the written rule missed something. It assumed Lance's value would show up as ### What we keep watching (but don't act on yet) -- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against. +- ~~**Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.~~ **Done 2026-05-02** — see "Follow-up: 10M re-bench" below. - **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system. - **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot. +--- + +## Follow-up: 10M re-bench (2026-05-02) + +The 10M re-bench above ran. Numbers from `data/lance/scale_test_10m` (33 GB, 10M × 768d, IVF_PQ live, post-doc_id-btree-build). Full report: `reports/lance_10m_rebench_2026-05-02.md`. + +| Op | 100K (this ADR) | 10M (re-bench) | Notes | +|---|---:|---:|---| +| Search (cold) | 2229μs | ~46ms median | 21× slower at 100× scale → reasonable for IVF_PQ | +| Search (warm) | (not measured) | ~20ms p50 | Stable across 5 trials | +| Doc fetch by id | 311μs | ~5ms | Structural ADR-019 win confirmed once `doc_id` btree is built | +| Index method | lance_ivf_pq | lance_ivf_pq | response tag confirms | + +**The HNSW comparison at 10M doesn't exist** — at 10M × 768d × 4 bytes = ~30 GB just for vectors, doubled for the graph. HNSW doesn't fit on a single 128 GB box at this scale. So the original "Lance pulls ahead at 10M" framing is true by elimination: Lance is the only contender that operationally exists at 10M. The strategic question is reframed as "Lance vs Parquet+HNSW-with-spilling," deferred until we have a workload where the Parquet path is the bottleneck (currently tracked in `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` decisions tracker). + +**Bug surfaced and fixed during re-bench:** Initial doc-fetch was ~100ms (full table scan). Root cause: the `doc_id` scalar btree was never built — `lance-bench` writes datasets by bypassing `IndexMeta`, and the activation-time auto-build only runs for IndexMeta-registered indexes. Fixed at two layers (commits `5d30b3d` + `044650a`): +- `lance_migrate` HTTP handler auto-builds the btree inline (~1.2s on 10M, +269MB on disk) +- `lance-bench` binary builds the btree post-IVF for parity with the gateway path + +**Gauntlet added 2026-05-02:** the lance crates had zero tests + no smoke when audited. Now have 7 unit tests in `crates/vectord-lance` + 12 sanitize tests in `crates/vectord` + 10-probe `scripts/lance_smoke.sh` + sanitized error boundary across all 5 routes. See commits `7bb66f0`, `ac7c996`, `e9d17f7` (sanitizer iterations driven by cross-lineage scrum). + +**Status:** ADR-019's hybrid architecture (Parquet+HNSW primary, Lance secondary) is now empirically validated up to 10M. The "watch and re-bench" item is closed. + ## Why this isn't moving the goalposts The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile. diff --git a/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md b/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md new file mode 100644 index 0000000..31f1e63 --- /dev/null +++ b/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md @@ -0,0 +1,177 @@ +# Phase 1.5 — BIPA Schema Audit + outcomes.jsonl Content Sample + +**Status:** Complete — 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md) + +> **Scope.** Two follow-up walks defined in `AUDIT_PHASE_1_DISCOVERY.md` §10/C4 + §10/single-reviewer (gemini "culture fit"): (1) BIPA-specific photo/video/biometric column + code path inventory, (2) actual content sample of `data/_kb/outcomes.jsonl` to confirm whether PII or discrimination-proxy reasoning lands there. Read-only walk; no code changes. + +--- + +## §1 — BIPA walk: photo/video/biometric surface + +### 1A — The face pipeline + +The Rust legacy ships a synthetic face pool with deepface-extracted demographic classifications. **Even though the FACES are synthetic, the LOGIC is BIPA-relevant** — when real photos arrive (J 2026-05-03: "we will" have them), the same pipeline processes them. + +| File:line | What | BIPA relevance | +|---|---|---| +| `scripts/staffing/fetch_face_pool.py` | Fetches 1000 synthetic faces from `thispersondoesnotexist.com` (StyleGAN). | Synthetic — no biometric collection at this stage. | +| `scripts/staffing/tag_face_pool.py:18-31` | Runs `deepface` on each face to extract `gender` (man / woman) + `race` (asian / black / hispanic / indian / middle_eastern / white) + `age`. Writes to `data/headshots/manifest.jsonl`. | **BIPA biometric processing.** Deepface extracts demographic classifications from face geometry. The output IS biometric information per 740 ILCS 14/10. | +| `data/headshots/manifest.jsonl` | 1000 rows, each `{id, file, gender, race, age}`. Sampled rows confirm shape: e.g. `{id: 0, file: "face_0000.jpg", gender: "woman", race: "east_asian", age: 33}`. | The persisted classifications. **For real photos, this file becomes a regulated biometric database.** | +| `mcp-server/index.ts:1308` (`/headshots/:key` route) | Serves face matched by gender × race × age intersection bucketing for a candidate. | Routing logic predicated on biometric-derived demographic attributes. | +| `mcp-server/index.ts:1367` (`/headshots/generate/:key`) | ComfyUI on-demand portrait generation. | Custom-generated image keyed to candidate. | +| `mcp-server/index.ts:1408` (ComfyUI prompt template) | `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic` | **Protected attributes (age + race + gender) embedded directly into the model prompt.** Even with synthetic output, this is system-level encoding of protected-attribute features into AI workflow. | + +### 1B — Name-based ethnicity inference (inferred-attribute risk) + +`mcp-server/search.html:3375-3432` defines lookup tables that classify candidates by demographic attributes derived from their NAMES (not photos): + +| Constant | What | +|---|---| +| `FEMALE_NAMES` (~200 entries) + `MALE_NAMES` | Gender inference from first name | +| `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` | Ethnicity inference from first name | +| `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` | Ethnicity inference from surname (more diagnostic than first name per the source comment) | + +**This is hard-coded inferred-attribute classification.** Per CLAUDE.md, the rationale is "Confidence-default name resolution... so a forklift operator looks like a forklift operator" — i.e., assigning representative photos. But the code path is: + +`candidate name → genderFor() / guessEthnicityFromFirstName() → face-pool bucket selection` + +A discrimination plaintiff's lawyer would frame this as: "the system EXPLICITLY classifies candidates by inferred ethnicity and uses that classification in its workflow." Whether the downstream use is "just for the photo display" or "ranks candidates" doesn't matter for BIPA-style strict-liability framing — it matters for Title VII, where intent / disparate-impact tests apply. + +### 1C — Data directories holding face content + +| Path | Count today | Purpose | gitignored? | +|---|---:|---|---| +| `data/headshots/` (face_*.jpg + manifest.jsonl) | 2500 entries | Synthetic face pool + biometric classifications | partial — manifest.jsonl is tracked, face_*.jpg are gitignored | +| `data/headshots_role_pool/` | 7 entries | Role-specific representative faces | gitignored | +| `data/headshots_gen/` | 4 entries | ComfyUI on-demand portrait cache | gitignored | +| `data/icons_pool/` | 2 entries | Certification + role icons | gitignored | +| `data/face_test/` | 1 entry | Likely deepface dev/test scratch | gitignored | +| `data/_imagecache/` | (not enumerated) | Image cache | gitignored | + +The `manifest.jsonl` file is **tracked in git** today and contains the demographic-classification table. For synthetic faces this is fine; **for real candidate faces, this would be a biometric database in version control** — a serious BIPA + general-data-protection failure. + +### 1D — Biometric-info contract risk when real photos arrive + +When the system transitions from synthetic to real candidate photos (J: "we will"), the BIPA exposure becomes immediate: + +1. **No written consent flow exists.** BIPA §15(b) requires "informed written consent" before biometric collection. The headshot upload path (wherever it gets built) needs a consent gate. + +2. **No public retention schedule exists.** BIPA §15(a) requires a written, publicly-available retention schedule. No such document found. + +3. **No deletion procedure exists.** BIPA §15(a) requires destruction "when the initial purpose for collecting or obtaining such identifiers or information has been satisfied or within 3 years of the individual's last interaction with the private entity, whichever occurs first." No such procedure exists. + +4. **No employee training documented.** BIPA litigation routinely cites lack of employee training on biometric handling as evidence of negligence. + +5. **The deepface library is itself a biometric processor.** Using it triggers the BIPA disclosure obligation regardless of whether the system stores the underlying face image — extracting "race" and "gender" from a face IS biometric processing under modern interpretations. + +6. **Per-violation damages.** Negligent violations: $1,000 per violation per person. Intentional/reckless: $5,000 per violation per person. The IL Supreme Court has held each separate scan/extraction can constitute a separate violation (Cothron v. White Castle, 2023). Class actions are routine; recent settlements: + - Facebook (Patel v. Facebook, 2020): $650M + - TikTok (In re TikTok BIPA, 2022): $92M + - Six Flags (Rosenbach v. Six Flags, 2019): set the "no actual harm needed" precedent + - Smaller staffing companies have settled for $1M-$50M ranges. + +### 1E — What this means for the architecture + +Even with synthetic data, the architecture has BIPA-shaped surface that is one upload path away from being live. **Recommended pre-real-photo gates:** + +1. **Disable deepface tagging on real-photo path** until consent + retention schedule are in place. Synthetic faces can keep using deepface; real faces cannot. +2. **Quarantine `data/headshots/manifest.jsonl` for real photos.** Real-photo classifications must NOT land here. Either move to a dedicated `data/biometric/` tree with separate retention or — preferable — don't extract demographic features from real photos at all. +3. **Replace the name → ethnicity lookup tables with a non-classifying default.** The current logic exists to pick "representative" photos. For real photos, the candidate's own photo IS representative — no inference needed. The lookup tables become dead code. +4. **Upload path needs consent gate.** Whatever endpoint accepts real photos must check a stored consent timestamp + version before storing the file. +5. **Retention schedule documented + auto-enforced.** Crontab / ingestd background sweep that deletes biometric data per the schedule. + +These five gates are NOT in `AUDIT_TRAIL_PRD.md` §8 phase plan. **Recommend adding as Phase 1.6 (BIPA pre-launch gates)** ahead of any real-photo intake. + +--- + +## §2 — outcomes.jsonl content sample + +### 2A — File state + +| Property | Value | +|---|---| +| Rows | 101 | +| Path | `data/_kb/outcomes.jsonl` | +| Rows with `name` field in fills | **39** | +| Rows with `first_name` field | 0 (the field is `name`, not `first_name` — names are pre-concatenated in this sink) | +| Sample names in fills (synthetic but representative) | `Alex Rivera`, `Carmen I. Garcia`, `Jacob N. Patel`, `Jamal Z. Jones`, `James Park`, `Maria Chen`, `Mei Y. Cooper`, `Peter E. Carter`, `Sam Torres` | + +### 2B — Confirmed PII shapes in the file today + +- `fills[*].name` — full candidate name (synthetic but real shape) +- `operation` — natural-language fill request (e.g. "fill: Welder x2 in Toledo, OH"). Currently doesn't carry candidate names in the operation string itself, BUT the `error` field can — model reasoning that names specific candidates lands here +- `error` — model self-correction text and validator failure messages. Sampled rows contain phrases like "executor parse failure on turn 8: no JSON object in Executor response: ..." which is technical content. Real-data scenarios where the model reasons about specific candidates would put names here. + +### 2C — Discrimination-proxy phrase audit (gemini scrum flag §10/single-reviewer) + +Searched for common LLM-generated discrimination proxy phrases. **0 hits in current synthetic-data sample**: + +- "culture fit" — 0 +- "communication" — 0 +- "team chemistry" — 0 +- "personality" — 0 +- "soft skills" — 0 +- "work ethic" — 0 +- "professional attitude" — 0 + +**Interpretation:** the current synthetic data doesn't generate these phrases. However: +- The file is structurally capable of holding them (it persists model `error` text) +- When real models (kimi-k2.6, deepseek-v3.2 per PR #13) reason about real candidates, they WILL generate phrases like these — these are common in HR/staffing LLM output +- Once a phrase like "rejected as not a culture fit" lands in outcomes.jsonl, it's append-only persistence. Cryptographic erasure is the only RTBF path. + +**Action:** add a content-redaction pass to the outcomes.jsonl writer BEFORE production data lands. Strip discrimination-proxy phrases at write time. (This is not a substitute for the model not GENERATING them — it's a defense-in-depth layer.) + +### 2D — Implications for the audit trail + +The outcomes.jsonl sample confirms what `AUDIT_PHASE_1_DISCOVERY.md` §1D predicted: **39 of 101 rows persist candidate names in the `fills[*].name` field today**. The shape is real even if the data is synthetic. + +For the Phase 2 identity service design: + +1. **The training-safe export pipeline (per J 2026-05-03 answer 11) MUST address outcomes.jsonl directly.** This file is the most-likely candidate for RAG re-indexing or fine-tuning corpus, and it carries names. + +2. **Crypto-erasure target list** must include `data/_kb/outcomes.jsonl`, `data/_kb/overseer_corrections.jsonl`, `/tmp/lakehouse-validator/sessions.jsonl`, plus pathway memory state, plus Langfuse storage. + +3. **Subject_id top-level promotion (per AUDIT_PHASE_1_DISCOVERY §10/C5)** can replace `fills[*].name` as the queryable field. The name itself can move into a `[REDACTED]` token until legal access dereferences via the identity service. + +--- + +## §3 — Synthesis: net new findings for Phase 2 design + +| Finding | Source | Phase 2 design implication | +|---|---|---| +| **Phase 1.6 (NEW) — BIPA pre-launch gates needed before real photos accepted** | §1E above | Phase 2 identity service design must include biometric-info table with consent + retention metadata schema. | +| **Name-based ethnicity inference is a separate Title VII / disparate-impact risk** | §1B above | Phase 2 must include a "no protected-attribute inference at routing time" architectural rule. The lookup tables should be deprecated as part of the identity-service migration. | +| **deepface library use itself triggers BIPA when real photos arrive** | §1D point 5 | Phase 2 design must specify "biometric processing requires consent token presence" as a gateway boundary check. | +| **outcomes.jsonl persists names today (39/101 rows)** | §2A above | Phase 2 needs subject_id top-level promotion + name field as `[REDACTED-{token}]` reference (token resolves via identity service for authorized callers). | +| **No discrimination-proxy phrases in current synthetic data, but the sink is shape-ready for them** | §2C above | Phase 2 should specify a redaction-on-write pass for outcomes.jsonl + sessions.jsonl writers. | +| **manifest.jsonl is tracked in git today** | §1C above | If real photos arrive without changing the manifest path, biometric data lands in version control — must be addressed as part of Phase 1.6 gates. | + +--- + +## §4 — Open questions surfaced by Phase 1.5 + +1. **Where will real photos enter the system?** No upload endpoint walked. Need to know the entry point before designing the consent gate. +2. **Will the deepface tagging path be preserved for real photos, or stripped?** Architectural decision — affects whether biometric extraction is in scope at all. +3. **What's the staffing client's consent UX?** Does the candidate sign consent during onboarding, separately for biometrics, etc.? Affects the consent-token design. +4. **How long do BIPA records need to persist?** BIPA's "3 years from last interaction" is a maximum, not a target — staffing-co's contract may set a shorter floor (e.g., "delete on placement+1 year"). Need staffing-co policy. +5. **Is there a Designated Privacy Officer / DPO?** Per CCPA/GDPR best practice + BIPA litigation hygiene, this role should exist before real-PII intake. Out of scope for code; flag for J. + +--- + +## §5 — Recommended Phase 1.6 BIPA pre-launch gates (NEW) + +Lands as new phase BETWEEN Phase 1.5 (this doc) and Phase 2 (identity service design). Read-only/design work; one code commit (the consent gate). Estimated 4-8 hours. + +1. **Write `docs/BIPA_COMPLIANCE_POLICY.md`** — public retention schedule, written consent template, deletion procedure, employee training acknowledgment. Counsel must review before publication. +2. **Add consent gate to whatever endpoint accepts real photos** (not yet built — design with the gate from day 1, don't bolt on after). +3. **Quarantine real-photo classifications to `data/biometric/`** — separate path, separate retention, separate access control. Real-photo `manifest.jsonl` does NOT land in `data/headshots/`. +4. **Deprecate name → ethnicity lookup tables in `mcp-server/search.html`** (commit removing those constants + the call sites). Replace with "use candidate's actual photo" path or non-inferred default. +5. **Add a unit test that asserts `data/headshots/manifest.jsonl` row count matches the SYNTHETIC face count** — fails if real-photo classifications leak into the synthetic manifest. + +These five fit between Phase 1.5 and Phase 2 because (a) they're MORE URGENT than Phase 2 (BIPA exposure starts the moment a real photo touches deepface), (b) they're independent of the identity service design, (c) they need counsel sign-off before they can ship anyway. + +--- + +## Change log + +- 2026-05-03 — Phase 1.5 BIPA + outcomes content audit complete. New finding: synthetic data + deepface + name-ethnicity inference creates BIPA-shaped surface that is one upload path away from being live. Recommend Phase 1.6 (BIPA pre-launch gates) before Phase 2 design starts. diff --git a/docs/AUDIT_PHASE_1_DISCOVERY.md b/docs/AUDIT_PHASE_1_DISCOVERY.md new file mode 100644 index 0000000..64adf92 --- /dev/null +++ b/docs/AUDIT_PHASE_1_DISCOVERY.md @@ -0,0 +1,295 @@ +# Phase 1 Discovery — Subject + PII Surface Map + +**Status:** Draft — 2026-05-03 · **Drafted by:** working session 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md) + +> **Purpose.** Read-only walk of both runtimes. Fills in the "UNKNOWN" cells in `AUDIT_TRAIL_PRD.md` §3 surface map and §7 current-state-vs-target table with file:line evidence. **No code changes.** Output: a complete picture of where subject identifiers + PII flow today, where they leak, and what an audit response could and could not produce right now. + +--- + +## TL;DR — what the walk found + +1. **A defense layer EXISTS but is BYPASSED.** Two PII-masked views (`candidates_safe`, `workers_safe`) live in `data/_catalog/views/` and correctly drop name/email/phone. **No production tool query uses them.** The MCP tool registry's `search_candidates` and `get_candidate` SQL templates query the raw `candidates` / `workers_500k` tables and return full PII to the LLM context. Worth considering: the views are functioning policy artifacts that nobody routes through. + +2. **PII flows freely through the LLM substrate.** In a single fill scenario, candidate names + emails + phones traverse: SQL → tool_result → LogEntry → execution_loop log → `/v1/respond` HTTP response → Langfuse trace input + output → `data/_kb/outcomes.jsonl` (fills with name) → `data/_kb/overseer_corrections.jsonl` (operation + correction text). At least 7 distinct persistence/transmission paths per scenario. + +3. **`candidate_id` is a stable token but not an isolated one.** It's a column on the same `workers_500k.parquet` that holds name/email/phone — there is no separate identity service. Joining candidate_id back to PII is one DataFusion query away. The audit-trail PRD's §5 identity-service intent is not yet built; the architectural separation does not exist. + +4. **No `/audit/subject/{id}` endpoint exists.** Reconstructing "every decision about person X" today requires manual cross-correlation across 4+ JSONL files + Langfuse + pathway memory + observer events. There is no canonical query path; the audit response cannot be produced in any reasonable time today. + +5. **Go side mirrors the Rust pattern.** Go validator's `rosterRow` carries Name; Go SessionRecord carries `Prompt` (truncated to 4000 chars) which contains the natural-language operation including candidate names. Cross-runtime parity in the PII-leak too. + +6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today. + +7. **The matrix-indexer fingerprint is subject-agnostic; trace bodies are UNVERIFIED.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers, which is structurally defensive. **However:** trace bodies (`reducer_summary`, `final_verdict`) are written from execution-loop output and are highly likely to leak PII. Per §8.1 these are unverified. Treat the matrix indexer as a SUSPECTED PII SINK until sampled — do NOT rely on "matrix can't drive discrimination" framing. (Walked back from earlier draft per cross-lineage scrum §10/C1: that claim was over-stated in 3/3 reviews.) + +--- + +## §1 — PII flow paths in the Rust stack (file:line evidence) + +### 1A — MCP tools query raw tables (PII enters the LLM context) + +| File:line | What | PII columns returned | +|---|---|---| +| `crates/gateway/src/tools/registry.rs:96` | `search_candidates` SQL template | `first_name, last_name, phone, email, city, state` | +| `crates/gateway/src/tools/registry.rs:107` | `get_candidate` SQL template | `SELECT *` — returns ALL columns including hourly_rate_usd, address, anything in the table | +| `crates/gateway/src/tools/registry.rs:129` | `top_recruiters` SQL | recruiter names (employee PII, not candidate but still PII) | +| `crates/gateway/src/tools/registry.rs:141` | `engaged_unplaced_candidates` SQL | `c.candidate_id, c.first_name, c.last_name, c.phone, c.vertical` | +| `mcp-server/index.ts:1722, 2173, 2269` | address concatenation (`street_number + street_direction + street_name`) | full street addresses surfaced to MCP UI clients | + +**Architectural gap:** the MCP tool registry has no "view-only" mode. Adding `_safe` view routing would be one `sql_template` rewrite per tool. **Today: every fill scenario invokes search_candidates → LLM sees full PII for every candidate in the result set.** + +### 1B — LLM tool results captured in execution log + +| File:line | What | +|---|---| +| `crates/gateway/src/execution_loop/mod.rs:30-39` | `LogEntry` struct: `{turn, role, model, kind, content: Value, at}`. `content` is the raw tool_result body (carries PII when kind=`tool_result`). | +| `crates/gateway/src/execution_loop/mod.rs:217` | `self.append(LogEntry::new(..., "tool_result", trimmed))` — every tool result appended to `self.log` | +| `crates/gateway/src/execution_loop/mod.rs:1319` | per-render cap: `kind == "tool_result"` gets 1200 char cap (vs 200 for other kinds). Trimming is for token economy, **not PII redaction** — names/emails fit in 1200 chars. | +| `crates/gateway/src/execution_loop/mod.rs:30-50` | `Fill` struct (in same file): `{candidate_id: String, name: String, reason: Option}`. The `name` field is load-bearing per the PRD-defined FillProposal artifact contract. | + +### 1C — Execution log returned to HTTP caller + +| File:line | What | +|---|---| +| `crates/gateway/src/v1/respond.rs:111-119` | `RespondResponse { status, artifact, log: outcome.into_log(), iterations, error }` — full log array returned in JSON response body to whoever called `POST /v1/respond` | +| `crates/gateway/src/execution_loop/mod.rs:82-93` | `RespondOutcome::{Ok,Failed,Blocked}` all carry `log: Vec` — no path drops the log on its way to the caller | + +**Audit implication:** any HTTP client calling `/v1/respond` today receives full PII in the response. Authentication on `/v1/respond` is the only access control. Authorization is binary (either you can call it or you can't); there is no row-level filtering of which candidates you're allowed to see in the response. + +### 1D — Persisted JSONL sinks on disk + +| File:line | Path written | PII shape | +|---|---|---| +| `crates/gateway/src/execution_loop/mod.rs:508` | `data/_kb/outcomes.jsonl` (append) | `{operation, fills: [{candidate_id, name, reason?}], error, ...}` — observed live: `operation` carries natural-language fill request including role + city + state; `fills` carries the full Fill struct with name. | +| `crates/gateway/src/execution_loop/mod.rs:701` | `data/_kb/overseer_corrections.jsonl` (append) | `{operation, correction, sig_hash, ...}` — `correction` is the overseer's free-text guidance which often references specific candidates by name (e.g. "the executor picked Emily Garcia who is at fill capacity, try Maria Rodriguez instead") | +| `mcp-server/observer.ts:138-139` | `data/_observer/ops.jsonl` (append) | observer event log; PII content depends on what the operation context included. **Empty on this box right now** (file doesn't exist), but the writer is wired. | +| `mcp-server/observer.ts:161` | `data/_kb/observer_escalations.jsonl` (append) | sampled live: technical analysis fields (no PII observed in current rows), but `analysis` field is free-form LLM output and could include names depending on the escalation trigger. | +| `crates/gateway/src/v1/session_log.rs` (writer) → `/tmp/lakehouse-validator/sessions.jsonl` | per `lakehouse.toml` `[gateway].session_log_path` | `SessionRecord { ..., prompt, attempts: [{... raw}], artifact }` — **`prompt` carries the operation text (PII), `raw` carries each model attempt's raw text (PII when model reasoned about specific candidates), `artifact` carries the final FillProposal `{candidate_id, name}` shape on success.** | + +### 1E — Langfuse external persistence + +| File:line | What gets sent | +|---|---| +| `crates/gateway/src/v1/langfuse_trace.rs:166` | `body.input.messages = ev.input` — full message array (system + user + assistant + tool messages). Tool messages contain raw PII for tool_result entries. | +| `crates/gateway/src/v1/langfuse_trace.rs:191` | `body.output = ev.output` — full model generation. When the model emits a Fill or reasons about candidates, names appear here. | + +Langfuse runs at `:3001` per memory, with credentials in `/etc/lakehouse/langfuse.env`. The Langfuse storage tier holds these traces. **PII leaves the lakehouse infrastructure boundary at this point** — Langfuse's storage layer (Postgres + ClickHouse, default config) holds it. + +### 1F — Pathway memory + +| File:line | What | +|---|---| +| `crates/vectord/src/pathway_memory.rs:169` | `PathwayTrace` struct definition | +| `data/_pathway_memory/state.json` | Persisted state — **need read** to confirm whether real names appear in traces. | + +Pathway traces are keyed by `pathway_id = SHA256(task_class + file_prefix + signal_class)` — none of which are subject identifiers. **The fingerprint is structurally subject-agnostic.** Whether the trace BODY (kb_chunks, observer_signals, reducer_summary, final_verdict) leaks PII depends on what those fields contain at write time. From the struct shape: `reducer_summary` and `final_verdict` are strings written from execution loop output — **highly likely to leak PII** when summarizing fill outcomes. Need to confirm against live state.json. + +--- + +## §2 — Defense layer that exists but is bypassed + +`data/_catalog/views/candidates_safe.json`: +- Drops: `last_name, email, phone, hourly_rate_usd` +- Masks: `candidate_id` (keep first 3, last 2 chars — `CAND-...01` shape) +- Row filter: `status != 'blocked'` +- Visibility intent (per `description`): "Visible to recruiter / mode-runner agents." + +`data/_catalog/views/workers_safe.json`: +- Drops: `name, email, phone, zip, communications, resume_text` +- Reason given: "resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale" +- Source for the rebuilt `workers_500k_v9` vector corpus + +**The `_safe` views are the right policy artifact.** The ARCHITECTURAL gap is at the SQL-template layer in `crates/gateway/src/tools/registry.rs` — the LLM-facing tools query the RAW tables, not the SAFE views. Three out of three candidate-touching tool templates (search_candidates, get_candidate, engaged_unplaced_candidates) bypass the safe view. + +There is no enforcement preventing this. There is no test that asserts "tool SQL must reference only `_safe` tables." There is no warning logged when raw tables are queried via tool surface. + +**Quick fix shape (NOT a phase 1 deliverable, just noting the change shape):** rewrite the tool sql_templates to `FROM candidates_safe` (or `FROM workers_safe`); add a build-time check that `crates/gateway/src/tools/registry.rs` only references `*_safe` tables; add a runtime gate in queryd that refuses LLM-attributed queries on raw tables. These would land in Phase 4 (subject tagging across substrates) or possibly Phase 1.5 (defense-layer enforcement) per the AUDIT_TRAIL_PRD. + +--- + +## §3 — Identity / candidate_id provenance + +| Question | Finding | +|---|---| +| Is `candidate_id` a stable token? | Yes — observed format `CAND-NNNNNN` (e.g. `CAND-000001`). Stable across the table. Used as join key in the schema (per `crates/ui/src/main.rs:160` cross-table join hints). | +| Is the candidate_id ↔ PII mapping in a separate service? | **No.** Both live in `data/datasets/workers_500k.parquet`. A SQL `SELECT name, email FROM candidates WHERE candidate_id = '...'` resolves the mapping in one query. | +| Is the mapping itself audited? | **No.** No log records "who looked up the PII for which candidate when." This is the identity-service gap from PRD §5. | +| Does anything ever generate a *different* token (UUID, opaque hash) instead of using candidate_id directly? | **No** — every tool, every validator, every persistence sink uses candidate_id as-is. | + +**Production-ready implication:** the staffing client's lawyer asks "show me the access log for candidate X's PII" → we cannot produce one. Every access happens via SQL against `workers_500k`; no row-level access log exists. + +--- + +## §4 — Go-side parity with the leak surface + +| Rust file:line | Equivalent Go file:line | Same shape? | +|---|---|---| +| `crates/validator/src/lib.rs` (FillProposal carries `{candidate_id, name}`) | `internal/validator/fill.go` + `internal/validator/lookup_jsonl.go:23-31` (`rosterRow` carries `CandidateID, Name, Status, City, State, Role, BlacklistedClients`) | Yes — same PII shape | +| `crates/gateway/src/v1/session_log.rs` SessionRecord with prompt/artifact | `internal/validator/session_log.go` SessionRecord with `Prompt string`, `Attempts[].Raw string`, `Artifact map[string]any` | Yes — same shape | +| `outcomes.jsonl` / `overseer_corrections.jsonl` writers | Not present on Go side (Go validatord doesn't currently run the execution_loop with tool dispatch — Phase 4 of Go rewrite is what wires that) | **Asymmetric** — Go side writes less but only because feature parity isn't done | +| MCP tools registry | Not on Go side yet (mcp-server is still Bun) | **Bun is the surface** for tool dispatch in both runtimes today | + +**Cross-runtime audit implication:** even though our 5 parity probes (validator/extract_json/session_log/materializer/embed) all pass 32/32, **none of them assert PII handling**. A new `pii_parity.sh` probe would feed identical PII-tagged input through both runtimes and assert identical redaction behavior. As of today, neither runtime *does* redaction at the substrate level, so the probe would just confirm "both leak identically." + +--- + +## §5 — Mapping back to AUDIT_TRAIL_PRD §3 surface map + +Updated cells with file:line evidence: + +| Decision happens at | Currently logged where | Audit-completeness gap (revised) | +|---|---|---| +| Ingestion (candidate added to pool) | `data/datasets/workers_*.parquet` rows; ingestd writes via `crates/ingestd/`; no per-subject "added at" event journal found | **GAP** — no subject-tagged ingest event. When + who + how a candidate entered is not auditable per-subject. | +| Embedding creation | `crates/aibridge/src/client.rs` (post-sidecar-drop direct path); LRU cache `crates/aibridge/src/cache.rs` (commit `150cc3b`); cached entries are keyed by `(model, text)` — **the text is the cached body, which for candidate embeddings IS PII (candidate name + role + skills appear in source text per workers_500k_v9 build SQL).** Cache itself is in-memory, not persisted, but text-as-key means PII is in process memory. | **MAJOR GAP** — no per-embedding audit row. Cache key contains PII. No subject tagging means we cannot answer "what embedding was generated for candidate X." | +| Search inclusion | Tool result entries in execution log + Langfuse + outcomes.jsonl `fills` field | Partial — fills are logged, but you have to grep across outcomes.jsonl + Langfuse for any mention of a candidate_id. No "all search results that included X" canonical view. | +| Search rank | Result set in chat traces (Langfuse), not indexed by candidate | Partial — Langfuse trace has the result set; no inversion (candidate → ranks-received). | +| Fill recommendation | `outcomes.jsonl` `fills` array with name + candidate_id | Partial — present, but mixed with PII in the natural-language `operation` field. | +| Validation outcome | `sessions.jsonl` per `[gateway].session_log_path` config; per-attempt `verdict_kind`, `error` | Partial — works per-session, not per-subject. To find "all validations that touched candidate X" you'd need to grep the JSONL by candidate_id. | +| Iterate retry escalations | Same sessions.jsonl `attempts[]` array | Same as above | +| Observer signals | `data/_observer/ops.jsonl` (file not present today on this box, writer is wired) | UNKNOWN content shape until the writer fires for a fill scenario — needs verification | +| Matrix-indexer compounding | `data/_pathway_memory/state.json` — fingerprinted by code, NOT by subject | **NO subject leak in fingerprint structure** (good for audit defensibility). Trace bodies (`reducer_summary`, `final_verdict`) MAY carry PII — needs sampling to confirm. | + +--- + +## §6 — Mapping back to AUDIT_TRAIL_PRD §7 current-state-vs-target gap table + +| Capability | Phase 1 finding | Status | +|---|---|---| +| candidate_id as canonical token | Stable `CAND-NNNNNN` format. Same column lives in same parquet as PII. | Token exists; isolation does not. | +| Identity service | Doesn't exist. PII + candidate_id co-located in `workers_500k.parquet`. | Real gap — needs new service. | +| `/audit/subject/{id}` endpoint | Doesn't exist. No audit route in `crates/gateway/src/v1/mod.rs` route listing. | Real gap — needs new endpoint. | +| Subject-tagged embeddings | LRU cache `(model, text)` keys; text contains PII; no audit row per embed. | Real gap. | +| Subject-tagged search results | Langfuse trace contains result set; not subject-indexed. | Real gap (queryability, not capture). | +| Subject-tagged validation outcomes | Yes — sessions.jsonl carries candidate_id (in the attempt's `raw` field) but not as a queryable top-level field. | Partial — needs subject_id top-level promotion. | +| Subject-tagged matrix indexer entries | Pathway fingerprints are subject-agnostic by design. Trace bodies may leak. | Decision needed (PRD open question 7) — keep code-only OR risk PII surface. | +| Protected-attribute filter at decision time | **Not enforced anywhere.** SQL templates return whatever columns they SELECT; no protected-attribute removal at the gateway boundary. The `candidates` table schema in the demo SQL includes age (years_experience is a proxy), and the call_log/email_log tables likely contain free-text correspondence. | Real gap — major. Requires both schema audit + boundary enforcement. | +| Retention policy | None enforced. Append-only files grow indefinitely. | Real gap. | +| Right to be forgotten | Not implementable today on append-only logs without cryptographic erasure. | Per PRD §6 design, real engineering needed. | +| Cross-runtime parity | 5 algorithm probes pass; 0 audit/PII probes exist. | Probe set needs extension. | + +--- + +## §7 — Worked example: John Martinez audit trail TODAY (negative result) + +If John Martinez (candidate_id `CAND-042195`) requests an audit: + +1. **Find his candidate_id.** Manual SQL: `SELECT candidate_id FROM candidates WHERE first_name='John' AND last_name='Martinez'` — returns 1+ rows. Already a leak: someone with SQL access can correlate name → candidate_id ad-hoc. + +2. **Find every fill scenario that included him.** `grep "CAND-042195" data/_kb/outcomes.jsonl` — returns rows where his fill was included. But the row's natural-language `operation` field may NOT contain his candidate_id (it's the fill request, e.g. "fill: Welder x2 in Toledo, OH"), so we'd miss scenarios where he was a *candidate* but not a *fill*. To find those, we'd need to grep Langfuse traces (off-box) or the per-tool result content in execution logs (which aren't persisted as separate JSONL — they live in the HTTP response that already left the building). + +3. **Find every validation that touched him.** Grep `/tmp/lakehouse-validator/sessions.jsonl` for `CAND-042195` — would catch FillValidator phantom-ID rejections AND successful fills, but the candidate_id would appear in either the prompt or the raw attempt text, not as a top-level queryable field. + +4. **Find every embedding generated for him.** Cache is in-memory; nothing persisted. Cannot answer. + +5. **Find every search result that ranked him.** Off-box in Langfuse. Untrieable in lakehouse without a separate Langfuse query pipeline (which doesn't exist). + +6. **Find pathway memory traces involving him.** Pathway fingerprints don't carry his ID. The `reducer_summary` strings might mention him (need to grep state.json) but the fingerprint search wouldn't surface them. + +7. **Show what protected attributes were exposed to the model.** No record of input_features per decision — the LLM saw whatever the SQL returned, and we have no per-decision input_features audit row. + +8. **Format the output for legal.** Even if we collected all the above, there's no signing, no integrity hash, no schema, no template. + +**Estimated time to produce a complete-and-defensible response today: not possible.** Estimated time to produce an INCOMPLETE response by cobbling JSONLs + Langfuse exports: 2-5 hours per request, manual, error-prone, and the response would over-share (other candidates appearing in the same fill scenarios) AND under-share (embedding events, search rankings, pathway traces missed). + +This is the Phase 1 result the AUDIT_TRAIL_PRD predicted: today's substrate is not production-ready for a discrimination-defense audit response. + +--- + +## §8 — What this discovery DID NOT cover + +Phase 1 was scoped to file:line evidence + sampling of live JSONL state. The following deserve their own subsequent walks before phase 2+ design: + +1. **Live sample of `data/_pathway_memory/state.json`** — confirm whether `reducer_summary` / `final_verdict` strings actually leak PII or stay generic. Read 3-5 traces and grep for names from `workers_500k`. +2. **Live sample of Langfuse traces** — confirm input message PII for a real fill scenario from past 7 days. Use Langfuse `:3001` query API. +3. **observerd event content** — the writer is wired but `data/_observer/ops.jsonl` doesn't exist on this box. Trigger a fill scenario and inspect the resulting event. +4. **Bun mcp-server tool dispatch** — does the Bun server log tool calls anywhere? `mcp-server/index.ts` is 2900+ lines; partial walk only here. +5. **bot/propose.ts** — the bot proposal flow likely touches candidates; not walked in this pass. +6. **`crates/journald` mutation log** — designed for row-level mutations per ADR-012; haven't confirmed whether candidate-table mutations land here with PII. +7. **Go side observerd + chatd PII surface** — the Go cmd/* binaries likely have analogous logging; confirmed validator parity but didn't walk observer/chat logging. +8. **Process memory + crash dumps** — if the gateway dumps core, what PII is in it? Out of scope for code walk; comes up in security audit. +9. **Operator runbook** — who has access to logs/, /tmp/lakehouse-validator/, MinIO buckets, Langfuse Postgres? Out of scope for code walk; comes up in operational security review. + +These are listed so the next phase doesn't accidentally re-walk what's done OR skip what wasn't covered. + +--- + +## §9 — Recommended next moves (not commitments) + +Per AUDIT_TRAIL_PRD §8, Phase 2 is the identity service design doc. Before that doc gets written, the cheapest high-value moves discovered here: + +1. **Defense-layer enforcement (1-2 hours work).** Rewrite the 3 tool SQL templates in `crates/gateway/src/tools/registry.rs` to use `_safe` views. Add a unit test that asserts no `crates/gateway/src/tools/registry.rs` template references `FROM candidates ` or `FROM workers_500k ` (only `*_safe`). This is one commit. It prevents the most-trafficked PII leak path TODAY without waiting on the identity service. **Cost:** the LLM sees masked candidate_ids (`CAN...95` instead of `CAND-042195`); some downstream tools (validator existence check) would need a "resolve to full ID" path that goes through the identity service — but that's exactly the architectural shape PRD §5 wants anyway. + +2. **Sample state.json + Langfuse before phase 2 design.** §8.1 + §8.2 above. ~30 minutes. Either confirms or refutes the "matrix indexer is subject-clean" finding from §1F. + +3. **Document the Bun mcp-server tool surface.** §8.4. The Bun layer is a major PII transit point not fully covered here. + +4. **Identify whether protected attributes (age proxies, photo features, zip-code → race correlations) are currently in any tool-returned column.** Schema-level audit of `candidates` + `workers_500k`. ~1 hour. Might surface that some "neutral" columns are actually protected-attribute proxies. + +These four moves give the phase-2 design doc strong evidence to lean on. None are commitments — J's call on what to do next. + +--- + +## §10 — Cross-lineage scrum review of this discovery doc (2026-05-03) + +After the discovery walk, this document was reviewed by three independent model lineages via `/v1/chat` against the Go gateway (post-PR-#13 model fleet): **opus** (`opencode/claude-opus-4-7`), **kimi** (`ollama_cloud/kimi-k2.6`), **gemini** (`ollama_cloud/gemini-3-flash-preview`). Custom prompt: senior security architect reviewing a discovery report. (DeepSeek-v3.2 timed out; not included.) + +Verbatim reviews saved at `/tmp/audit_scrum/{opus,kimi,gemini}_review.md`. Convergent findings (≥2 reviewers) are treated as high signal per `feedback_cross_lineage_review.md`. + +### Convergent findings — must address before phase 2 design + +**C1. §1F matrix-indexer claim is OVER-CLAIMED (3/3 reviewers).** The TL;DR #7 line "good for audit defensibility (matrix index can't drive discrimination)" overstates structural subject-agnosticism as behavioral fairness. Per opus: "until §8.1 is executed, the correct framing is 'fingerprint structurally subject-agnostic; body content unverified — treat as suspected PII sink.'" Per kimi: dangerous reasoning — if `reducer_summary` says "candidates named Emily Garcia were rejected for fill capacity," the matrix learns proxy variables and future similar names get downranked = temporal discrimination risk. Per gemini: "you cannot claim audit defensibility until you prove the *content* of the matrix indexer doesn't contain PII." **Action:** walk back the §1F TL;DR claim; reframe as "fingerprint structure is defensive; body content unverified — treat as suspected PII sink until §8.1 confirms." + +**C2. §1E (Langfuse) is the MOST DANGEROUS leak — fix FIRST (3/3 reviewers).** Opus: "boundary-crossing leak that makes a regulator's eyes light up." Kimi: "Article 44 GDPR transfer if SaaS-hosted; CPRA 'sale/sharing' question; subprocessor notification failure." Gemini: "un-certifiable for SOC2 Type II" + "unauthorized data transfer to third-party storage tier." All three would do Langfuse redaction/sampling BEFORE the §9.1 view-routing fix. **Action:** revise §9 priority order — Langfuse boundary first, view-routing second. + +**C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills (3/3 reviewers).** Opus: "you need not just 'what did we do for candidate X' but 'what was the selection rate for protected class Y vs comparator' — that requires capturing protected attributes WITH outcomes." Kimi: "EEOC Uniform Guidelines on Employee Selection Procedures (1978) — matrix index that learns from historical fill outcomes IS a selection procedure under the Guidelines." Gemini: "The system doesn't log the *entire* candidate pool for a specific search — only the fills (§5). To defend a lawsuit, you must show the stats for everyone who *could* have been picked, not just the person who was." **Action:** Phase 1 didn't capture this load-bearing requirement. PRD §1 worked example needs expanding: the audit response must include the comparator pool + adverse-impact statistics, not just the subject's decision row. + +**C4. BIPA + biometric exposure UNDERSTATED (3/3 reviewers).** Already in `AUDIT_TRAIL_PRD.md` §10.5 jurisdictional checklist but NOT translated into Phase 1 findings actionables. If `workers_500k` columns include photo paths, video interview references, or anything that could yield biometric inference (per gemini: "even descriptors that could be reconstructed into biometric templates"), BIPA's $1k-$5k per-violation regime applies BEFORE the GDPR analysis matters. **Action:** add to §8 (what discovery did NOT cover): explicit photo/video/biometric column audit of `workers_500k` schema. + +**C5. `candidate_id` must be PROMOTED to top-level field in all JSONL sinks (3/3 reviewers).** Opus + kimi + gemini converge: grepping natural-language strings (operation, raw, prompt, reducer_summary) for candidate_id is not a defensible audit strategy. Even if subject_id appears in those strings TODAY, it appears co-mingled with other candidate names, model reasoning, etc. — making subject filtering unsafe. **Action:** add to PRD §7 target column "subject_id top-level promotion" — change session_log writer + outcomes.jsonl writer + observer event writer to ALL include `subject_id` (or `subject_ids[]`) as a first-class top-level field. + +### Single-reviewer findings — verified, worth incorporating + +**OPUS unique:** +- **8th PII path missing: LLM provider egress.** PR #13 routes models through opencode + ollama_cloud + openrouter — opencode and openrouter are external services. Cross-border data transfer under GDPR Ch. V; third-party processor relationship requiring DPA under Art. 28. Phase 1 did not enumerate this path. **Action:** add §1G "LLM provider egress" to §1. +- **GDPR Art. 22 / EU AI Act right to explanation.** Audit must capture the model's REASONING CHAIN, not just decision output. Phase 1 §7 worked example doesn't include this. **Action:** add to subject-audit response shape (PRD §2). +- **Special-category data under GDPR Art. 9.** resume_text + call_log + email_log routinely contain health (accommodation requests), union, religion. Higher legal bar — Art. 9(2) explicit consent required. Phase 1 §6 mentions these tables exist but doesn't flag the special-category exposure. **Action:** call out in §6. +- **DPIA / ROPA / DPA inventory.** None of these documents referenced. Some may exist outside code; Phase 1 should at minimum note their absence as Phase 1.5 input. + +**KIMI unique:** +- **Sequential `CAND-NNNNNN` IDs enable enumeration attacks.** Predictable IDs let an attacker scan the candidate space. Security finding Phase 1 didn't flag. +- **Langfuse retention config unaudited.** Default trace retention is 30 days in some versions, indefinite in others. Directly impacts RTBF analysis. **Action:** check live Langfuse config. +- **CCPA "de-identified in place" may be faster than crypto-shred.** Replacing PII with `REDACTED-{hash}` while preserving log structure may satisfy CPRA's de-identified exception. Worth considering vs. crypto-shred. +- **Bun MCP server is the cross-runtime bridge — likely COMMON-MODE failure.** Phase 1 framed Go side as "mirrors Rust pattern" implying independent failure; it's likely shared infrastructure failure. **Action:** add to §4 — common-mode reframe. +- **Cryptographic integrity for the audit trail itself.** Merkle trees / signed logs / chain-of-custody under FRE 901. Opposing expert can challenge admissibility without this. **Action:** add to PRD §2 audit response shape — integrity-signed. +- **HIPAA Business Associate Agreement scoping not done.** If any candidate is healthcare-vertical, BAA analysis required. +- **`get_candidate` SELECT * has 40+ load-bearing columns.** §9.1 "1-2 hours" estimate for view rewrite is irresponsible without scoping downstream consumers. **Action:** revise §9.1 estimate; flag dependency analysis as prerequisite. + +**GEMINI unique:** +- **Data residency — JSONL on US box.** If any candidate is EU resident, GDPR violation without SCC/DPF. Phase 1 didn't ask whether IL+IN target market includes EU residents (probably not, but staffing-co clients sometimes have international placements). +- **"Culture fit" reasoning strings as discrimination proxies.** Common LLM-generated phrases ("not a culture fit," "communication concerns," "team chemistry") often correlate with protected-attribute discrimination. Phase 1 didn't audit the actual reasoning text in `outcomes.jsonl` for these phrases. +- **Comparator-pool snapshot for every fill.** Need to capture WHO COULD HAVE BEEN PICKED, not just who was. **Action:** PRD §2 audit response shape needs `comparator_pool` field per decision. + +### Revised §9 — recommended next moves (reordered by scrum convergence) + +1. **(NEW PRIORITY 1, was P3) Sample state.json + Langfuse content** — confirm/refute the matrix-indexer subject-clean claim and quantify Langfuse PII exposure. Cheapest move that resolves the over-claim AND informs the boundary-leak fix. +2. **(NEW PRIORITY 2, was P1) Defense-layer enforcement at SQL template level** — rewrite tool registry to use `_safe` views. **Estimate revised UPWARD per kimi**: scope `get_candidate` SELECT * downstream consumers first; estimate 4-8 hours including the existence-check resolution path through the (not-yet-built) identity service. Stop-gap until then: add LLM-attribution flag to queryd, refuse `FROM candidates `/`FROM workers_500k ` queries that originate from tool dispatch. +3. **(NEW) Langfuse boundary audit + redaction** — sample retention config, check DPA status, scope a redaction/sampling layer that strips PII from message arrays before the Langfuse POST. This is the boundary-crossing leak — fix BEFORE chasing internal sinks. +4. **(NEW) Subject_id top-level promotion to all JSONL writers** — single architectural change spanning Rust + Go session_log + observerd event writer + outcomes/corrections appenders. Makes subject-correlation queries defensible (no more grepping natural language strings). +5. **(was P4) Schema-audit for protected-attribute proxies** — extend to include "culture fit"-shaped reasoning text in outcomes.jsonl + the comparator-pool retention requirement. +6. **(NEW) BIPA-specific audit of workers_500k schema** — explicit photo/video/biometric column inventory before any production deployment in IL. +7. **(NEW) Operational discovery** — DPIA, ROPA, DPA inventory, SCC for cross-border, Langfuse retention config. Out-of-code-walk; needs J + counsel input. + +### What I'm walking back + +§1F TL;DR claim "matrix indexer is good for audit defensibility" — per all 3 reviewers, this is over-claimed without §8.1 verification. The correct frame is "fingerprint structure is subject-agnostic by design; trace body content unverified — treat as suspected PII sink until sampled." + +§9 ordering — view-routing was P1; per all 3 reviewers, Langfuse boundary should be P3 in front of it. View-routing is the source-side fix; Langfuse is the boundary-crossing fix; both matter, do them in BOUNDARY-FIRST order. + +§9.1 effort estimate — kimi's "irresponsible without dependency scoping" critique is right. Revised UP to 4-8 hours. + +--- + +## Change log + +- 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`. +- 2026-05-03 — §10 added: cross-lineage scrum review of the discovery doc (opus + kimi + gemini). 5 convergent findings (matrix-indexer over-claim, Langfuse first-priority, comparator pool gap, BIPA understated, subject_id top-level promotion). Plus single-reviewer high-value catches. §9 reordered. diff --git a/docs/AUDIT_TRAIL_PRD.md b/docs/AUDIT_TRAIL_PRD.md new file mode 100644 index 0000000..2b05421 --- /dev/null +++ b/docs/AUDIT_TRAIL_PRD.md @@ -0,0 +1,374 @@ +# PRD: Production-Ready Audit Trail + +> **⚠ OVER-SCOPED — 9-phase plan needs to shrink for local-only deployment.** +> +> 2026-05-03 evening: J reframed the system as local-only per PRD line 70. The 9-phase plan in §8 was sized for SaaS-tier infrastructure with cloud HSM, separate identity daemon, dual-control JWT, etc. For a single-box local deployment, audit trail can be a few hundred LOC of local writers + a signed local file, not a 17-20 day phase plan. +> +> **What stays valid:** +> - The legal use case (worked example: John Martinez at Warehouse B requests audit) — this is the real problem +> - The §10.5 jurisdictional surface (IL BIPA, IN, federal) — counsel reads this +> - The §3 surface map: where decisions get made today (file:line evidence — see `AUDIT_PHASE_1_DISCOVERY.md`) +> - Phase 1.6 BIPA pre-launch gates — those still apply when real photos arrive +> +> **What's over-scoped:** +> - The 9-phase implementation plan (§8) — should compress to 3-4 phases for local-only +> - The identity service design (`IDENTITY_SERVICE_DESIGN.md`) — see that doc's deprecation header +> +> Do NOT execute the §8 phase plan as-written. When J greenlights, draft a v2 plan sized for local single-box. + +**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 + +> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation. + +--- + +## 1. The legal use case (worked example) + +**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2. + +**What we must produce.** A response that proves either: +- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR +- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI." + +**What we must NOT produce.** +- Other subjects' data (response leaks if even one other candidate's name appears) +- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped) +- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence + +**The defensibility chain.** The audit shows: +1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into +2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank +3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned +4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections) +5. **Outcome decisions** — final fills, rejections, hand-offs + +For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale. + +--- + +## 2. The subject audit response — output format + +``` +GET /audit/subject/{candidate_id}?from=D1&to=D2 +→ JSON or signed PDF (legal preference TBD) +``` + +**Header section:** +- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash +- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls. + +**Per-decision row schema** (shape, not exhaustive): +```json +{ + "ts": "ISO-8601 UTC", + "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal", + "daemon": "gateway | validatord | observerd | matrixd | ingestd", + "model": "kimi-k2.6 | deepseek-v3.2 | ...", + "provider": "ollama_cloud | opencode | openrouter", + "input_features": { /* what the model SAW — sanitized per §4 */ }, + "output": { /* what the model decided */ }, + "rationale": "model's natural-language explanation, or rule-based justification", + "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree", + "session_id": "iterate session that produced this row" +} +``` + +**Footer section:** +- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy" +- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited) + +--- + +## 3. Surface map — where decisions happen + +| Decision happens at | Currently logged where | Audit-completeness gap | +|---|---|---| +| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk | +| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding | +| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation | +| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial | +| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified | +| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today | +| Iterate retry escalations | Session JSONL `attempts[]` array | OK | +| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk | +| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject | + +**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly. + +--- + +## 4. PII handling rules + +**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces. + +**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it. + +**Protected-attribute exclusion at decision time:** the model NEVER receives: +- Race, ethnicity, national origin +- Sex, gender, marital status, pregnancy +- Age, date of birth (allowed: years of experience) +- Religion +- Disability, genetic information +- Veteran status (unless legally relevant for the role) +- Sexual orientation, gender identity + +If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input. + +**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls. + +**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows. + +--- + +## 5. Identity service — candidate_id ↔ PII mapping + +**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change. + +**Target state:** +- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping +- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id` +- Identity reads require a separate auth credential held by legal-authorized operators +- Every identity read is itself audited (log who accessed PII for which candidate when) +- Identity service runs as its own daemon, port-isolated from the gateway +- Cross-runtime: same identity service backs both Rust and Go + +**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary. + +--- + +## 6. Retention policy + +**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA. + +**Target state (proposed):** +- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s) +- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract. +- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed. +- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable. + +**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off. + +--- + +## 7. Current state vs target state + +| Capability | Today | Production-ready target | Gap | +|---|---|---|---| +| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service | +| Identity service | none | separate daemon, audited reads | Real — build new | +| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new | +| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument | +| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk | +| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial | +| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major | +| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk | +| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build | +| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build | +| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set | + +--- + +## 8. Implementation phases (proposed sequence) + +Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds. + +### Phase 1 — Discovery walk (read-only, ~3-4 hours) +Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references. + +**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells. + +### Phase 2 — Identity service design (design doc, ~2 hours) +Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.** + +**Exit:** J approves the design. + +### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours) +Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong. + +**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued. + +### Phase 4 — Subject tagging across substrates +Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror. + +**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end). + +### Phase 5 — Identity service build +Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly. + +**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject. + +### Phase 6 — Protected-attribute boundary enforcement +Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing. + +**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping. + +### Phase 7 — Retention + right-to-be-forgotten +Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint. + +**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows. + +### Phase 8 — Legal-format export + signing +Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow. + +**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies. + +### Phase 9 — End-to-end discrimination defense rehearsal +Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them. + +**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness. + +--- + +## 9. Cross-runtime requirement + +**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5. + +The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version). + +--- + +## 10. Open questions blocking phase 2 (resolved 2026-05-03) + +**These were the original Phase 1 open questions. J answered the load-bearing 5 in conversation 2026-05-03; answers folded in below. Remaining items 1, 4, 6 still need J's call before Phase 2 design ships.** + +1. **Identity service: separate daemon vs in-process?** _Recommended: separate. **Status: pending J confirmation.**_ +2. ~~**Retention period N years?**~~ — Out of scope for now; will be set at deployment time per client contract. Default to 4-year hot retention until set. +3. ✅ **Photos/video in scope?** **YES** (J 2026-05-03). BIPA (740 ILCS 14) applies in full. Per-violation $1k-$5k statutory damages, written consent + retention schedule mandatory. **This becomes a Phase 1.5 priority** — see §10.5/§13 for revised phase ordering. +4. **JSON or signed PDF for legal export?** _Recommend signed JSON with PDF rendering option. **Status: pending.**_ +5. ✅ **RTBF under append-only:** Cryptographic erasure approach approved in principle (J 2026-05-03 implicit via approval of plan). +6. **Audit endpoint auth model:** _Recommend legal-only credential separate from admin token. **Status: pending.**_ +7. ✅ **Matrix indexer subject-awareness:** Per scrum review (`AUDIT_PHASE_1_DISCOVERY.md` §10/C1), matrix-indexer is suspected PII sink (trace bodies unverified). Action: sample state.json before deciding (a) keep code-only + add PII-redact-on-write to trace bodies, OR (b) remove subject-summary text from trace bodies entirely. Decision deferred until §8.1 sampling completes. + +### Newly answered 2026-05-03 (J) + +8. ✅ **Langfuse hosting model:** **Self-hosted.** Removes the GDPR Art. 44 cross-border-transfer concern that 3/3 scrum reviewers flagged. Langfuse retention config + Postgres/ClickHouse access controls still need to be audited as part of Phase 1.5 — but the boundary stays inside J's infrastructure, which is materially better than SaaS Langfuse. + +9. ✅ **EU candidates in scope:** **Not currently — may need placeholder later.** Design choice: build the identity-service interface to be EU-compatible (DPIA-shaped fields, lawful-basis tracking, SCC-ready transfer mechanism slots) but DO NOT gate Phase 2 on EU compliance. Phase 2 ships IL+IN-shaped; EU additions are a follow-up phase. + +10. ✅ **Healthcare vertical / HIPAA:** **Same framework — yes.** Healthcare staffing IS in scope. PHI in `resume_text`, `communications`, and `call_log` is realistic. **Implications:** + - Business Associate Agreement (BAA) required with any third-party model provider that processes content from healthcare-vertical staffing requests + - opencode + ollama_cloud + openrouter (per PR #13 routing) are external — BAAs needed OR healthcare requests must route to local-only models (Ollama on-box) + - PHI redaction at the gateway boundary becomes mandatory before the model call leaves the box, OR the model call must stay on-box for healthcare requests + - Vertical detection at the gateway boundary becomes a Phase 2 requirement + +11. ✅ **Training / RAG re-runs may use historical outcomes:** **Yes — design as if it WILL.** Implications: + - `outcomes.jsonl` and `overseer_corrections.jsonl` cannot remain raw-PII forever — anything that lands in a training corpus or RAG re-index becomes ungeneratable to delete (PII in model weights) + - Phase 2 design must include a "training-safe export" pipeline that strips PII from outcomes before feeding to any training/RAG path + - Crypto-erasure of historical outcomes becomes load-bearing — if a candidate exercises RTBF and their data already trained a model, we must be able to evidence "the source was destroyed; the model retains it indistinguishably from synthetic patterns" + +### Effect on the §8 phase plan + +The user-confirmed answers shift priorities. Revised ordering (incorporating scrum-driven priority changes from `AUDIT_PHASE_1_DISCOVERY.md` §10): + +- **Phase 1.5 (NEW)** — BIPA-specific photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample. Lands BEFORE Phase 2 design starts. +- **Phase 2 (identity service design)** — now must include EU-placeholder fields, vertical-detection (healthcare flag), training-safe export interface, BIPA consent + retention metadata +- **Phase 3** (audit endpoint skeleton) — unchanged +- **Phase 4** (subject tagging) — must include healthcare-vertical routing decision at gateway boundary +- **Phase 5** (identity service build) — must include BIPA-compliant biometric metadata table +- **Phase 6** (protected-attribute boundary) — must include PHI redaction for healthcare-vertical requests +- **Phase 7** (retention + RTBF) — must include training-safe export evidence chain +- **Phase 8** (legal export) — unchanged +- **Phase 9** (rehearsal) — must include both EEOC discrimination scenario AND BIPA biometric scenario AND healthcare PHI breach scenario + +--- + +## 10.5 Jurisdictional surface (IL + IN) + +> **⚠ Not legal advice.** This is a research-grade checklist for J to take into a conversation with actual employment + privacy counsel. The system is targeting **Chicago (Illinois)** and **Indiana** placements per 2026-05-03 conversation. Counsel needs to verify what currently applies, what's pending, and whether case law has moved any of these in 2026. **Verify with counsel before claiming compliance with any item below.** + +### Federal layer (always applies) + +| Statute / framework | Relevance to this system | +|---|---| +| Title VII (Civil Rights Act) | Bans discrimination on race, color, religion, sex, national origin in hiring. Discrimination claim defense is the worked example in §1. | +| ADEA (Age Discrimination in Employment) | Bans age-based discrimination for workers 40+. DOB must be excluded from features per §4. | +| ADA (Americans with Disabilities Act) | Bans disability discrimination + requires reasonable accommodation. Disability-inferring features (gait, photo features, medical history) need exclusion. | +| EEOC enforcement | Receives complaints, issues right-to-sue. Audit response per §2 is what defends in EEOC investigation. | +| OFCCP | Applies if our staffing client serves federal contractors. Adds affirmative-action recordkeeping on top of EEOC. | +| FCRA (Fair Credit Reporting Act) | Triggers if background checks are performed. Pre-adverse-action notice + dispute process needed. | +| Section 1981 | Race-based contract discrimination — staffing is contract relationship. | + +### Illinois-specific (Chicago jurisdiction) + +| Statute | What | What we need | +|---|---|---| +| **BIPA** (Biometric Information Privacy Act, 740 ILCS 14) | Bans collection of biometric identifiers (face geometry, fingerprints, voiceprints) without informed written consent + retention schedule. Penalties: $1,000-$5,000 per violation per person. **Class actions are common and aggressive.** | If we use candidate photos for any feature (face match, headshot rendering, photo-derived attributes), BIPA almost certainly applies. The headshot pool we generate (per CLAUDE.md commit `5d93a71` area) needs careful review — synthetic faces are probably OK; real candidate photos are NOT without explicit BIPA-compliant consent. **Counsel must review.** | +| **Illinois AI Video Interview Act** (820 ILCS 42) | If AI analyzes recorded video interviews, employer must disclose AI use, obtain consent, provide explanation of how AI works, and limit who can review the video. | If we ever ingest video, this applies. Currently we don't, but worth flagging to counsel as a "what if we add this in 12 months" boundary. | +| **Illinois Human Rights Act** (775 ILCS 5) | Broader than federal Title VII — adds protected classes including arrest record, military status, marital status, order of protection, citizenship status (in some cases), unfavorable military discharge. | Protected attribute exclusion list in §4 needs expanding to cover IL-specific classes. | +| **Personal Information Protection Act** (PIPA, 815 ILCS 530) | Breach notification — must notify Illinois residents whose unencrypted PII was breached. | If identity service or workers parquet is breached, notification clock starts. Need incident response runbook. | +| **Illinois Day and Temporary Labor Services Act** (820 ILCS 175) | Specific to staffing/temporary services industry. Includes equal-pay-for-equal-work + record-keeping requirements + worker notification. | Highly relevant — applies directly to staffing-company clients. Audit retention may interact with these recordkeeping requirements. | +| **Workplace Transparency Act** | Restrictions on non-disclosure agreements re: harassment/discrimination | Tangential but worth noting. | +| **City of Chicago Human Rights Ordinance** (Title 6 Chicago Municipal Code) | Adds protected classes beyond IHRA (source of income, parental status, military discharge status, credit history). | Chicago-specific protected attributes list. | +| **Cook County Human Rights Ordinance** | Similar additions county-wide. | Chicago is in Cook County so this stacks. | +| **Possible: AI hiring transparency** | Several states/cities have proposed/passed laws modeled on NYC Local Law 144 (annual bias audit + candidate notification). I do not know whether IL or Chicago has such a law on the books as of 2026-01 cutoff. | **Counsel must check current state.** If it exists, we need annual bias audit reports (which IS what this PRD is building toward, but the report format may have specific requirements). | + +### Indiana-specific + +| Statute | What | What we need | +|---|---|---| +| **Indiana Data Breach Disclosure** (IC 24-4.9) | Breach notification within "without unreasonable delay" | Same incident response runbook as IL PIPA. | +| **Indiana Civil Rights Law** (IC 22-9) | State-level employment discrimination | Largely tracks federal Title VII, fewer expansions than IL. | +| **Indiana Genetic Information Privacy Act** | Bans use of genetic info in employment | Already in §4 protected list. | +| **General observation** | Indiana is generally less aggressive than Illinois on AI/employment regulation as of cutoff. | The IL bar is higher — if we satisfy IL, IN typically follows. **Counsel must confirm this isn't backwards.** | + +### Cross-cutting (security frameworks for SaaS sales) + +These aren't laws but are commonly required by enterprise customers (including staffing clients) before sale. + +| Framework | What | Relevance | +|---|---|---| +| **SOC 2 Type II** | Auditor attestation of operating effectiveness over 6-12 months across Trust Service Criteria (Security, Availability, Processing Integrity, Confidentiality, Privacy). | The Privacy criterion overlaps heavily with this PRD. Privacy + Security are the two load-bearing TSCs. Effort to first Type II report: 6-9 months. Type I (point-in-time) is faster (weeks) but enterprise buyers usually want Type II. | +| **SOC 3** | Public-facing summary of SOC 2 (no detailed control descriptions). | Nice-to-have for marketing but the staffing client will want SOC 2 Type II report under NDA. | +| **HIPAA** | Healthcare data protection. | Triggers ONLY if staffing places workers into healthcare roles where they handle PHI. Currently not in scope per CLAUDE.md. **Confirm scoping with J.** | +| **PCI DSS** | Payment card data | Not currently in scope. | +| **ISO 27001** | International information security management | Alternative to SOC 2; more common in EU. Probably unnecessary for IL/IN-only deployments. | + +### What this means for phase ordering + +The 9-phase plan in §8 is technically correct but may need re-ordering once counsel weighs in: + +- **BIPA risk on photos** is so high and so aggressive that if we use real candidate photos *anywhere*, that may need to be the FIRST thing we resolve — before the audit-trail work starts. Class-action exposure is enormous. +- **SOC 2 Type II prep** runs in parallel with this work, not after. If the staffing client says "show us your SOC 2 report" we need to have started the engagement weeks/months before. +- **Day and Temporary Labor Services Act** may impose recordkeeping that interacts with our retention SLA (§6) — counsel may say "no, retention has to be N years for THIS reason, not your defaulted 4." + +### Open questions for counsel (one ask) + +1. Does the staffing client have an existing SOC 2 report we leverage, or do we need our own? +2. Are we using any real candidate photos? If yes, is BIPA consent in place? +3. Does Illinois have an AI hiring transparency law on the books in 2026? If yes, what does the bias audit report need to look like? +4. What's the IL Day and Temporary Labor Services Act recordkeeping retention period? Does it interact with our 4-year proposed SLA? +5. Are background checks performed? If yes, do we need FCRA pre-adverse-action workflow integration? +6. Any healthcare placements? (HIPAA scoping) +7. Is the staffing client a federal contractor? (OFCCP scoping) + +Counsel's answers shape whether the §8 phase plan ships as-is or needs reordering. + +--- + +## 11. What this PRD is NOT + +- Not a contract with the staffing client. That document needs lawyers and signs after this is built. +- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC/BIPA/etc — passing actual certification is its own project. +- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made. +- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires. +- **Not legal advice.** The §10.5 jurisdictional surface is a research-grade checklist, NOT counsel's analysis. Verify everything with actual employment + privacy counsel licensed in IL + IN before claiming compliance with anything in this document. + +--- + +## 12. Appendix — terms + +- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`. +- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.). +- **Audit row** — one record in the audit response per decision, with the schema in §2. +- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes. +- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list. +- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation). +- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth. +- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it. +- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs. + +--- + +## Change log + +- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker. diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 3cafa35..c382719 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -104,3 +104,8 @@ **Date:** 2026-04-24 **Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002. **Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon. + +## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly +**Date:** 2026-05-02 +**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call. +**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running. diff --git a/docs/IDENTITY_SERVICE_DESIGN.md b/docs/IDENTITY_SERVICE_DESIGN.md new file mode 100644 index 0000000..2003a64 --- /dev/null +++ b/docs/IDENTITY_SERVICE_DESIGN.md @@ -0,0 +1,584 @@ +# Identity Service — Phase 2 Design (v2 — post-scrum revisions) + +> **⚠ OVER-SCOPED FOR LOCAL-ONLY DEPLOYMENT — needs simpler rewrite before implementation.** +> +> 2026-05-03 evening: J reframed the system as local-only per PRD line 70 ("Everything runs locally — no cloud APIs"). This document was drafted assuming SaaS-tier infrastructure (HashiCorp Vault, AWS KMS, S3 Object Lock, dual-control JWT split-secret ceremony, mTLS CA, separate Postgres database). For J's local-only single-box deployment serving IL+IN staffing, the audit trail can be MUCH smaller: local SQLite or Postgres, local key file, local HMAC chain to an append-only JSONL. +> +> The discovery findings in `AUDIT_PHASE_1_DISCOVERY.md` and `AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md` remain valid (PII flow paths, BIPA exposure, etc.). The PROBLEM is real. This DOC's solution shape is wrong for the deployment. +> +> Do NOT implement this document as-written. When J greenlights audit-trail work, draft a v3 that's local-only sized (~3-5 days, not 17-20). +> +> See `STATE_OF_PLAY.md` "PRD line 70 is load-bearing" entry for the binding direction. + +**Status:** Draft v2 — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 +**Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md) + +> **v2 history.** v1 (2026-05-03 morning) was scrummed across opus + kimi + gemini lineages. 3/3 reviewers converged on 6 critical issues — wrong migration order, master key + legal token "security theater," BIPA `inferred_existing` prima facie violation, healthcare default in wrong direction, Merkle chain without external anchor. v2 incorporates those changes. Full scrum reviews preserved at `/tmp/identity_scrum/{opus,kimi,gemini}_review.md`. Diff between v1 and v2 captured in §11 change log. + +> **Why this exists.** Phase 1 + 1.5 confirmed today's substrate has no separation between candidate_id and PII. Both live in `workers_500k.parquet`. No per-access audit, no consent gate, no retention enforcement. This document specifies the new identity service that holds the candidate_id ↔ PII mapping, gates every PII read, audits every access, and serves as the single legal-attestable boundary between PII and the rest of the system. + +> **Prerequisite:** **Phase 1.6 (BIPA pre-launch gates) MUST ship before any identityd backfill begins.** Per kimi+gemini scrum, backfilling with `consent_status='inferred_existing'` is a BIPA §15 prima facie violation. Phase 1.6 establishes the consent template + retention schedule + employee training that turns backfill from "inferred" into "pending_backfill_review with documented escalation path." + +--- + +## 1. Scope and non-goals + +### In scope +- Single source of truth for `candidate_id ↔ PII` mapping +- Per-PII-access audit log (who/what/when/why) +- Consent + retention metadata (BIPA + IL Day and Temporary Labor Services Act + healthcare PHI) +- Legal-tier access — short-lived JWT with **split-secret dual-control issuance** (NOT a static file token, per all 3 reviewers) +- Healthcare-vertical routing with **fail-closed default** (`unknown` treated as healthcare until classified, per opus+gemini) +- EU-compatible interface — fields exist, enforcement is per-subject (no system-wide flag flip needed) +- Hardware-backed master key — **HashiCorp Vault Transit OR AWS KMS** for v1 (NOT a sealed file, per all 3 reviewers; sealed-file v1 was "obfuscation in any defensible sense" — kimi) +- Signed-JSON audit response with PDF render path (PDF deferred; JSON ships first per opus) +- External-anchor for audit-log integrity (S3 Object Lock + signed timestamps; NOT just Postgres Merkle, per all 3 reviewers) +- mTLS or Unix Domain Socket between gateway ↔ identityd (per kimi — "port-isolated is theater without authenticated channel") +- Field-level authorization on `/subjects/{id}` (per-purpose field allowlists, per kimi) +- Synchronous cache invalidation hook for RTBF (per opus — even if Phase 7 is the full RTBF build) + +### Out of scope (for this phase) +- The `/audit/subject/{id}` endpoint itself (Phase 3) +- Full subject-tagging across other substrates (Phase 4) +- Right-to-be-forgotten implementation full-flow (Phase 7) +- Training-safe export (deferred from Phase 2 per opus — "not on critical path for audit-trail defense") +- BIPA pre-launch gates content (Phase 1.6 — separate doc) + +### Architecturally simplified from v1 (per scrum) +- **Per-row encryption keys** deferred to Phase 7. v2 uses a single Data Encryption Key (DEK) wrapped under HSM-backed Key Encryption Key (KEK), with per-subject **ciphertext-deletion** as the v2 erasure mechanism. This matches kimi's recommendation: "A single data-encryption key with HSM-backed rotation, plus per-subject deletion of ciphertext, achieves equivalent practical erasure with far less complexity." Cryptographic-erasure via per-subject keys becomes a Phase 7 enhancement. +- **EU placeholder fields** kept (per J 2026-05-03), but enforcement code is genuinely no-op until first eu_resident=true subject — no code paths run, no schema migrations needed when EU comes online (just per-subject field population). +- **Merkle chaining** narrowed (per gemini): full Merkle tree only for legal-tier and consent/erasure events. Standard `gateway_lookup` events get a simpler signed-HMAC chain. Both anchored externally. + +--- + +## 2. Architectural shape + +### 2.1 — Process model + transport + +Per J 2026-05-03 + kimi scrum: separate daemon, mTLS-mandatory transport. Single Go implementation, both runtimes call it. + +| Property | Value | +|---|---| +| Name | `identityd` | +| Port | `:3225` (single port, not dual; per kimi simplification — runtime-agnostic routing) | +| Implementation | Go | +| Storage | Postgres in **isolated database** (per J answer, confirmed). Schema-level isolation; no shared schemas with Langfuse or other lakehouse storage. | +| Transport gateway↔identityd | **mTLS mandatory.** Self-signed CA managed by identityd at startup; gateway clients have their own client certs. Plain HTTP is rejected at the listener. | +| Encryption-at-rest | **Single DEK wrapped under KEK in HashiCorp Vault Transit** (recommended) OR AWS KMS. Master key NEVER on disk. v1 of identityd refuses to start if KEK is unreachable. | +| Audit-log signing | **Separate Ed25519 keypair from the master encryption key.** Signing key in Vault Transit (signing backend) OR a separate sealed-secret file with strict rotation procedure. | +| External anchor for audit chain | **S3 Object Lock** (compliance mode, 7-year retention) holding periodic Merkle root commitments. Daemon writes hourly + on legal-tier event. | +| Backup | Postgres standard backup; KEK backup separate (different storage, different access). Crypto-erasure model only works if these are not co-located. | +| Rate limiting | Per-token QPS + daily-volume caps. Default legal-token: 100 lookups/day + 10K daily volume; alerting at 50%. Configurable per-token. | +| Token revocation | `token_revocations` table checked on every legal-tier auth. Revocation propagates within 60s (TTL on cache). | + +### 2.2 — Schema (Postgres DDL, v2) + +```sql +-- Subject record. PII columns are ciphertext under the DEK. +CREATE TABLE subjects ( + candidate_id TEXT PRIMARY KEY, -- UUID v7, NOT sequential (per kimi enumeration concern) + -- Encrypted PII fields. AES-256-GCM under DEK. NULL = not collected. + name_ct BYTEA, + email_ct BYTEA, + phone_ct BYTEA, + address_ct BYTEA, + ssn_ct BYTEA, + dob_ct BYTEA, + -- DEK version this subject was encrypted under. KEK rotates; DEK rotates per + -- KEK rotation. New DEK version = subjects re-encrypted in background sweep. + dek_version INT NOT NULL, + -- Lawful basis + consent metadata + consent_status TEXT NOT NULL CHECK (consent_status IN ( + 'pending_backfill_review', -- backfill default; no biometric/PHI use until reviewed + 'pending_first_contact', -- new subject, awaiting consent UX + 'given', -- explicit consent recorded + 'withdrawn', -- subject revoked + 'expired' -- consent timeout + )), + consent_version TEXT, -- references published consent template version + consent_given_at TIMESTAMPTZ, + consent_withdrawn_at TIMESTAMPTZ, + -- BIPA-specific fields (Phase 1.5 §1E + Phase 1.6 prerequisite) + biometric_consent_status TEXT NOT NULL DEFAULT 'never_collected' CHECK (biometric_consent_status IN ( + 'never_collected', 'pending', 'given', 'withdrawn', 'expired' + )), + biometric_retention_until TIMESTAMPTZ, -- BIPA: max 3 years from last interaction + -- Vertical detection — drives healthcare PHI routing. + -- DEFAULT 'unknown' (per opus+gemini scrum) — fail-closed routing treats + -- unknown as healthcare-equivalent until reclassified. + vertical TEXT NOT NULL DEFAULT 'unknown' CHECK (vertical IN ( + 'unknown', 'general', 'healthcare', 'finance', 'other' + )), + -- Per-vertical retention period. Drives the daily erasure sweep. + -- (Per opus: this was missing in v1. Required for BIPA's 3-year-from- + -- last-interaction rule which differs from generic retention.) + retention_period_days INT NOT NULL, + -- EU-placeholder fields. Enforcement is per-subject; nothing runs + -- until a row has eu_resident=true AND lawful_basis IS NOT NULL. + eu_resident BOOLEAN NOT NULL DEFAULT FALSE, + lawful_basis TEXT, -- GDPR Art. 6 basis when eu_resident=true + transfer_mechanism TEXT, -- SCC, DPF, BCR — populated when EU comes online + -- Standard audit columns + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_interaction TIMESTAMPTZ, -- drives retention sweep + -- Crypto-erasure state (v2: ciphertext deletion, NOT key destruction) + erased_at TIMESTAMPTZ, + erasure_reason TEXT +); + +-- Append-only access log. EVERY PII read writes a row. +-- Field-level: fields_accessed records WHICH fields the caller resolved. +-- purpose enforced against an allowlist per-purpose-token. +CREATE TABLE pii_access_log ( + access_id BIGSERIAL PRIMARY KEY, + candidate_id TEXT NOT NULL, + accessed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + accessor_kind TEXT NOT NULL, -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve' + accessor_id TEXT NOT NULL, -- daemon name + caller token-hash; never raw token + purpose_token TEXT NOT NULL, -- opaque token; resolved through purpose_definitions table + fields_accessed TEXT[] NOT NULL, + request_trace_id TEXT, + chain_kind TEXT NOT NULL CHECK (chain_kind IN ('hmac', 'merkle')), -- HMAC for standard, Merkle for legal/consent/erasure + integrity_hash TEXT NOT NULL, + -- Per opus security finding: this row's existence is itself sensitive + -- ("candidate X is under legal review" leaks via purpose). Read access + -- to this table requires legal-tier auth or scoped per-subject auth. + is_legal_tier_event BOOLEAN NOT NULL DEFAULT FALSE +); + +-- Purpose definitions — enforces field-level authorization (per kimi). +-- A given purpose_token can only request fields in its allowlist. +CREATE TABLE purpose_definitions ( + purpose_token TEXT PRIMARY KEY, -- e.g. 'fill_validation', 'audit_subject_response' + description TEXT NOT NULL, + allowed_fields TEXT[] NOT NULL, -- e.g. ARRAY['name'] for fill_validation + auth_tier TEXT NOT NULL CHECK (auth_tier IN ('service', 'legal')), + rate_limit_qps INT, + daily_volume_cap INT +); + +-- Token revocation list. Checked on every auth, cached 60s. +CREATE TABLE token_revocations ( + token_hash TEXT PRIMARY KEY, + revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(), + revoked_by TEXT NOT NULL, + reason TEXT +); + +-- Consent template versioning. Hash references, not embedded text per row. +CREATE TABLE consent_versions ( + version TEXT PRIMARY KEY, + effective_at TIMESTAMPTZ NOT NULL, + superseded_at TIMESTAMPTZ, + template_hash TEXT NOT NULL, -- SHA256 of the canonical template + template_path TEXT NOT NULL -- where the canonical text lives (e.g. data/_consent/v3-2026-04-15.md) +); + +-- External anchor checkpoints. Periodically committed to S3 Object Lock. +CREATE TABLE anchor_commits ( + commit_id BIGSERIAL PRIMARY KEY, + chain_kind TEXT NOT NULL, -- 'hmac' or 'merkle' + checkpoint_at TIMESTAMPTZ NOT NULL, + last_access_id BIGINT NOT NULL, -- max access_id covered by this checkpoint + root_hash TEXT NOT NULL, + s3_object_uri TEXT NOT NULL, -- s3://anchor-bucket/identityd/... + s3_lock_until TIMESTAMPTZ NOT NULL -- compliance-mode retention end +); +``` + +### 2.3 — HTTP surface (v2) + +All under `/v1/identity/`. mTLS + token required on every endpoint. + +| Method + Path | Purpose | Auth tier | Notes | +|---|---|---|---| +| `POST /v1/identity/subjects` | Create new subject. Body: PII fields + `purpose_token`. Returns: candidate_id (UUID v7). | service | Validates purpose_token allowlist matches submitted fields | +| `GET /v1/identity/subjects/{candidate_id}?purpose={token}&fields={list}` | Resolve PII for candidate. | service | **Field-level enforcement**: returned fields ⊆ purpose_token's allowed_fields. Mismatch → 403. Logs access row. | +| `GET /v1/identity/subjects/{candidate_id}/vertical` | **Minimal-disclosure** vertical lookup (per kimi HIPAA minimum-necessary). Returns only `{vertical, consent_status}`. | service | Used by gateway routing. Cheaper access-log row. | +| `GET /v1/identity/subjects/{candidate_id}/full` | Complete subject record + audit summary. | **legal** | Short-lived JWT only. Logged with `is_legal_tier_event=true`. Triggers real-time notification to designated counsel + J. | +| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn with version. | service | | +| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure. Idempotent. | **legal** | Synchronously triggers cache invalidation hooks (per opus). | +| `GET /v1/identity/access_log/{candidate_id}` | Per-subject access log for audit response. | **legal** | | +| `POST /v1/identity/auth/legal_token` | Issue short-lived JWT (max 24h). Requires **dual-control attestation**: J's signed nonce + counsel's signed nonce, both verified against pre-registered public keys. | dual-control | Per all 3 reviewers — replaces v1's static file approach | +| `GET /v1/identity/health` | Liveness | none | Returns only `{status: "ok"}` — no version, no schema info | +| `(no public training-safe-export endpoint in v2)` | — | — | Deferred per opus — not on critical path | + +### 2.4 — Auth model (v2 — split-secret + short-lived) + +**Service-tier auth** (gateway → identityd for routine PII resolution): +- mTLS client cert per gateway (Rust :3100 + Go :4110 each get their own cert) +- Bearer token in `Authorization` header (long-lived — 90d), rotated per ops runbook + +**Legal-tier auth** (per-request, short-lived JWT, dual-control issuance): +- J holds private key A +- Designated outside counsel holds private key B +- `POST /v1/identity/auth/legal_token` requires: + - HTTP body: `{purpose, ttl_seconds, requested_fields, signature_a, signature_b, nonce_a, nonce_b}` + - identityd verifies both signatures against pre-registered A and B public keys + - On success: emits a JWT signed by identityd, valid for `ttl_seconds` (max 24h), scoped to `purpose`+`fields` +- The JWT: + - Cannot be issued without BOTH A and B signatures (per gemini "split-secret startup ceremony" pattern, applied per-token) + - Carries scope (purpose + fields) inside its claims; identityd enforces at use time + - Per-token rate limit + daily cap recorded on issuance + - Revocation: token_hash added to `token_revocations`; identityd refuses on next check (60s cache TTL) + +**Why this matters:** +- v1's "static file mode 0400" was security theater per all 3 reviewers +- A leaked legal token v1 = unbounded exfiltration window until rotation +- v2's leaked JWT = bounded by TTL (max 24h) AND revocable in <60s +- Single-actor compromise (J's machine OR counsel's machine, not both) cannot mint legal tokens + +--- + +## 3. Integration with the rest of the substrate + +### 3.1 — Gateway changes + +When gateway needs PII for a fill scenario: + +1. Gateway has only `candidate_id` (post-Phase-1.5 view-routing fix) +2. Gateway calls `GET /v1/identity/subjects/{candidate_id}/vertical` first if it might be healthcare-vertical-sensitive routing +3. Gateway calls `GET /v1/identity/subjects/{candidate_id}?purpose=fill_validation&fields=name` +4. identityd validates: purpose_token=`fill_validation` allows `[name]` only; if request asks for `[name,ssn]` → 403 +5. identityd writes `pii_access_log` row, decrypts requested fields, returns +6. Gateway uses fields, then **explicitly zeros memory** (kimi's drop-and-overwrite pattern) before returning HTTP response +7. **No PII caching** in gateway memory beyond request lifetime + +**LRU embed cache** (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Phase 4 task to re-key as `(model, candidate_id, field_subset_hash)`. **Phase 2 must add the synchronous cache-purge hook** so RTBF in Phase 2 invalidates Phase 4's cache when it ships (opus finding). + +### 3.2 — Cross-runtime: same identityd, both gateways call it + +Both gateways call identityd over mTLS. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates identical PII request through both gateways produces identical access-log rows (modulo daemon-name field). + +### 3.3 — JSONL writer changes (subject_id top-level promotion) + +Per `AUDIT_PHASE_1_DISCOVERY` §10/C5: +- All JSONL sinks (outcomes, sessions, overseer_corrections, observerd ops) gain top-level `subject_ids: [...]` field +- `fills[*].name` replaced with `name_ref: "[REDACTED-{candidate_id}]"` +- Authorized callers dereference via identityd + +### 3.4 — Langfuse boundary redaction (defense in depth, per opus TOCTOU finding) + +v1 design: per-request map of `subject_id → resolved_PII`, replace before Langfuse POST. + +**v2 adds** (per opus): outbound regex/NER pass on the Langfuse payload as defense-in-depth. The model can hallucinate names not in the resolved-PII map. The outbound NER pass catches them. + +``` +Gateway constructs Langfuse payload + → Pass 1: per-request resolved-PII map replacement + → Pass 2: NER scan (regex for SSN/phone shapes; named-entity model for names) + → Either pass detecting unredacted PII → drop the trace OR replace with [POSSIBLE-PII-DETECTED] + → POST to Langfuse +``` + +### 3.5 — Healthcare vertical routing — fail-closed default + +**Per opus + gemini scrum: Default is `unknown` → treated as healthcare for routing purposes.** + +When gateway receives a request involving any candidate: +1. Query `/vertical` for the candidate +2. If `vertical IN ('healthcare', 'unknown')` → route to **on-box Ollama only** (no opencode/openrouter/ollama_cloud egress) +3. If `vertical='general'` (or other non-healthcare) → cloud routing OK +4. If identityd unreachable → fail closed (refuse the request, return 503) + +**Reclassification path:** subjects can be moved from `unknown` → `general` via explicit operator action (after manual review). Subjects flagged `healthcare` stay healthcare unless explicitly downgraded. + +**Auto-escalation** (per kimi): if a subject's resume_text or call_log content is updated with healthcare-pattern matches (RN, BSN, hospital, MD, physician, etc.), vertical auto-escalates to `healthcare`. Never auto-de-escalates. + +### 3.6 — Cache invalidation hook (Phase 2 must ship even though full RTBF is Phase 7) + +When `POST /v1/identity/subjects/{id}/erase` fires: +1. Mark `subjects.erased_at` and zero out ciphertext columns +2. Write `pii_access_log` row with `purpose_token='retention_expired'` or `'rtbf_request'` +3. **Synchronously** publish a `subject_erased` event to a pub/sub channel (Redis or Postgres LISTEN/NOTIFY) +4. Gateway subscribes; on event, purges any in-flight cache entries for that candidate_id +5. Eventual-consistency window: <5s between erase-call and cache-flush + +Without this hook, RTBF in Phase 7 would be a lie because the gateway's LRU embed cache could still hold the subject's data. + +--- + +## 4. Audit response shape (Phase 3 preview, v2 — adds model-version snapshot per gemini) + +```json +{ + "schema": "audit.subject.v1", + "subject_token": "01926f2e-7c1b-7000-...", // UUID v7, NOT CAND-NNNNNN + "request_window": { "from": "2026-01-01", "to": "2026-05-03" }, + "generated_at": "2026-05-03T12:00:00Z", + "generated_by": "identityd@hostname", + "merkle_root": "sha256:...", // root hash of legal/consent/erasure events + "external_anchor": "s3://anchor-bucket/identityd/2026-05-03T12-00.json", // S3 Object Lock URI + "signature": "ed25519:...", // separate signing key from encryption KEK + "consent": { ... }, + "decisions": [ + { + "ts": "2026-04-22T09:15:23Z", + "decision_kind": "fill_recommendation", + "daemon": "gateway", + "model": "kimi-k2.6", + "model_version_hash": "sha256:...", // NEW per gemini — proves what model existed AT decision time + "provider": "ollama_cloud", + "trace_id": "trace-abc", + "session_id": "session-xyz", + "input_features": { ... }, // sanitized; no protected attributes; no inferred-attribute proxies + "output": "...", + "rationale": "...", + "comparator_pool_size": 47, + "comparator_appendix_ref": "see comparator_appendix.A" + } + ], + "comparator_appendix": { + "A": { + "scope": "fill scenarios in window matching role X, geo Y", + "total_pool_size": 47, + "selection_rate_by_protected_class": { + // Aggregated; NO other subjects' identifiers leak + "race": { "white": 0.33, "black": 0.31, ... }, + "gender": { "man": 0.34, "woman": 0.30, ... } + }, + "four_fifths_test": "passed" // or "concern: rate ratio 0.71" + } + }, + "access_log": [...], + "footer": { + "completeness_attestation": "all decisions about subject_token in window per retention policy v2 are included", + "merkle_proof": "...", // proof this audit's root is in S3 anchor + "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0" + } +} +``` + +--- + +## 5. Migration path (v2 — REORDERED per all 3 reviewers) + +v1's order was wrong. New order, with explicit prerequisites: + +| Step | Action | Prerequisite | +|---|---|---| +| **0** | **Phase 1.6 BIPA pre-launch gates SHIPPED** — consent template published, retention schedule public, deletion procedure documented, employee training acknowledged. | (separate work — see PHASE_1_6_BIPA_GATES.md when written) | +| **1** | Stand up identityd with **synthetic test subjects only**. KEK in vault, mTLS live, all endpoints serve. | Vault/KMS available; mTLS CA bootstrapped | +| **2** | **Audit-parity probe** (`audit_parity.sh`) green on synthetic data. Cross-runtime equivalence verified. | Step 1 | +| **3** | Add gateway feature flag `LH_USE_IDENTITY_SERVICE`. **Shadow-write only** — gateway writes new subjects to identityd AND continues SQL path. No reads from identityd yet. | Step 2 | +| **4** | Run shadow-write for ≥1 week. Validate access logs, encryption correctness, write-path performance under real traffic. | Step 3 | +| **5** | Backfill from `workers_500k.parquet`. **`consent_status='pending_backfill_review'`** for ALL existing rows. **`vertical='unknown'`** default (NOT 'general'). **`biometric_consent_status='never_collected'`** — backfill does NOT include any biometric data. | Step 0 (BIPA gates) + Step 4 | +| **6** | Human-review queue for vertical reclassification. Subjects move from `unknown` to `general` only after explicit review. Healthcare-pattern matches auto-escalate to `healthcare` (never auto-downgrade). | Step 5 | +| **7** | **Shadow-read** — gateway reads from BOTH SQL and identityd, compares, logs divergences, returns SQL result. Run ≥1 week. | Step 5 | +| **8** | Feature-flag cutover — gateway reads from identityd, falls back to SQL on error, alerts on every fallback. | Step 7 | +| **9** | Quarantine PII columns in workers_500k.parquet — only after **cryptographic attestation of identityd completeness** (Merkle proof: source row hash = identityd row hash for every candidate_id). Move PII columns to a different bucket; the candidate_id-only projection becomes operational. | Step 8 | + +**Key v2 changes from v1:** +- New Step 0 (BIPA gates prerequisite) — backfill cannot start without this +- Steps 3-4 (shadow-write before backfill) — production validates the WRITE path before data lands +- Step 5 backfill consent: `pending_backfill_review` not `inferred_existing` (BIPA defense) +- Step 5 vertical default: `unknown` not `general` (HIPAA fail-closed) +- New Step 6 (human review) — vertical classification via explicit operator action +- Step 7 added (shadow-read after backfill) — catches encoding/normalization bugs before cutover +- Step 9 (quarantine) requires cryptographic attestation of completeness + +Each step is its own commit, its own gate, its own rollback path. + +--- + +## 6. Cross-runtime parity probe (NEW, per scrum §6 plus opus 'oracle test' addition) + +`audit_parity.sh` ships in Phase 5. Asserts: + +1. Same PII fetch through Rust gateway and Go gateway produces identical identityd access-log rows (modulo daemon name) +2. Crypto-erasure of test subject through Rust gateway is honored when Go gateway tries to fetch +3. Healthcare-vertical routing decision identical across both runtimes +4. **Oracle test** (per kimi SOC2 CC4.1 finding): probe with known-good inputs against expected outputs. A bug present in BOTH implementations must not pass the parity probe. +5. Discrimination-proxy phrase redaction triggers as expected on adversarial test cases + +--- + +## 7. What this design intentionally does NOT solve + +- Protected-attribute exclusion at decision time (Phase 6) +- Pathway memory trace body redaction (Phase 4) +- Retroactive Langfuse history scrub (separate runbook) +- GDPR Art. 22 right to explanation full implementation (Phase 8) +- Multi-region data residency (single-region US-Midwest by default) +- Training-safe export (deferred — not on critical path per opus) + +--- + +## 8. Open questions — RESOLVED 2026-05-03 + +J confirmed all 6 v1 recommendations + scrum-driven changes: + +1. ✅ **Master key:** HashiCorp Vault Transit (recommended) OR AWS KMS. NOT a sealed file. v1's sealed-file recommendation is rejected per all 3 reviewers ("obfuscation in any defensible sense"). +2. ✅ **Postgres isolation:** identityd's own database, isolated schema. +3. ✅ **Vertical backfill:** `'unknown'` default with fail-closed routing. NOT `'general'` per opus+gemini scrum. +4. ✅ **Legal token:** Split-secret dual-control issuance (J + counsel both sign). Short-lived JWT, max 24h, revocable in <60s. +5. ✅ **Crypto-erasure sweep:** Daily 03:00 UTC. (Note: v2 erasure mechanism is ciphertext deletion, not key destruction. Per-row keys deferred to Phase 7.) +6. ✅ **EU enforcement:** Per-subject. Schema fields exist; nothing runs until first eu_resident=true subject. + +### Newly resolved per scrum + +7. ✅ **Migration order:** REORDERED per §5 above. New Step 0 (BIPA prerequisite). Shadow-write before backfill. Shadow-read before cutover. +8. ✅ **Audit-log external anchor:** S3 Object Lock with compliance-mode 7-year retention. Hourly + on-event commits. +9. ✅ **Audit-log signing key:** Separate Ed25519 keypair from KEK. Vault Transit signing backend OR sealed-secret with strict rotation runbook. +10. ✅ **mTLS gateway↔identityd:** Mandatory. Self-signed CA managed by identityd at startup. +11. ✅ **Per-row encryption keys:** Deferred to Phase 7. v2 uses single DEK with HSM-wrapped KEK + ciphertext deletion for erasure. +12. ✅ **Field-level authorization:** purpose_definitions table enforces per-purpose field allowlists. +13. ✅ **Synchronous cache invalidation:** pub/sub event on erase; gateway subscribes. +14. ✅ **Outbound NER pass for Langfuse:** Defense-in-depth in addition to symbol-table replacement. +15. ✅ **Model version hash in audit response:** Captured per decision row. +16. ✅ **PDF render:** Deferred from Phase 2; JSON ships first. + +--- + +## 9. Estimated implementation cost (revised v2) + +| Sub-phase | Effort | Notes | +|---|---|---| +| 2A — Postgres schema + Vault/KMS integration | 1.5-2 days | Includes mTLS CA bootstrap + signing-key separation | +| 2B — identityd HTTP surface (Go) | 2-3 days | All endpoints, auth, dual-control JWT issuance, rate limiting, revocation | +| 2C — Backfill ETL (BIPA-compliant — pending_backfill_review) | 1 day | Plus Merkle attestation of completeness | +| 2D — Gateway integration (Rust + Go, mTLS, shadow-write phase) | 2-3 days | Per-tool migration, parity probe | +| 2E — JSONL writer changes (subject_id top-level promotion) | 1 day | All sinks | +| 2F — Langfuse redaction (symbol-table + NER) | 2 days | Two passes + drop-on-detect | +| 2G — Healthcare-vertical routing (fail-closed) | 0.5 day | Plus auto-escalation pattern matcher | +| 2H — Cache invalidation pub/sub hook | 0.5 day | Critical for Phase 7 RTBF | +| 2I — External anchor (S3 Object Lock) | 1 day | Hourly + on-event commits | +| 2J — Cross-runtime parity probe with oracle tests | 1 day | New probe | +| **Total** | **~12-15 working days** | (v1 estimated 8-10; v2 added BIPA gates dependency, mTLS, dual-control JWT, NER pass, anchor, etc.) | + +Bigger than v1. Worth it — every addition was a 3/3-reviewer convergent finding. + +--- + +## 10. The four "would not build" blockers from scrum (all addressed in v2) + +| # | v1 issue (3/3 reviewers) | v2 resolution | +|---|---|---| +| 1 | Migration order (backfill before validation) | §5 reordered; Step 0 BIPA gates prereq added | +| 2 | Master key on disk + legal token static file | §2.1 Vault/KMS for KEK; §2.4 split-secret short-lived JWT | +| 3 | `inferred_existing` BIPA prima facie violation | §5 Step 5 uses `pending_backfill_review`; Step 0 gates required first | +| 4 | Healthcare default `general` (HIPAA exposure window) | §3.5 fail-closed `unknown`-as-healthcare; §5 Step 5 backfill default `unknown` | + +All three reviewers said "I would not build v1 as written." All four blockers are resolved in v2. Re-scrum recommended before implementation starts. + +--- + +## 11. Change log (v1 → v2) + +| Section | v1 | v2 | +|---|---|---| +| Master key storage | Sealed file `/etc/lakehouse/identityd_master.key` | HashiCorp Vault Transit / AWS KMS | +| Legal token | Static file mode 0400 | Split-secret short-lived JWT (max 24h), dual-control issuance | +| Encryption keys | Per-row keys with master wrapping | Single DEK + HSM-wrapped KEK + ciphertext deletion (per-row deferred to Phase 7) | +| Healthcare default | `vertical='general'` backfill | `vertical='unknown'` backfill, fail-closed routing | +| Migration §5 order | 1: stand up, 2: backfill, 3: feature flag, 4: cutover, 5: quarantine | 0: BIPA gates, 1: stand up, 2: parity probe, 3: shadow-write, 4: shadow-write soak, 5: backfill (pending_backfill_review), 6: human vertical review, 7: shadow-read, 8: cutover, 9: quarantine with cryptographic attestation | +| Audit-log integrity | Postgres Merkle chain only | Postgres Merkle for legal/consent/erasure + HMAC for standard + S3 Object Lock external anchor | +| Audit-log signing key | Same as encryption key | Separate Ed25519 keypair; Vault signing backend | +| Gateway↔identityd transport | Plain HTTP (implied) | mTLS mandatory | +| Field authorization | Per-endpoint only | Per-purpose-token field allowlists | +| Cache invalidation | Phase 4 / Phase 7 | Phase 2 ships pub/sub hook | +| Langfuse redaction | Symbol-table replacement only | Symbol-table + outbound NER pass + drop-on-detect | +| Audit response | No model version | model_version_hash per decision | +| PDF render | Phase 2 | Deferred — JSON ships first | +| Training-safe export | Phase 2 | Deferred — not critical path | +| Effort estimate | 8-10 days | 12-15 days | + +--- + +## 12. v3 amendments (post-second-pass scrum, 2026-05-03 evening) + +v2 was re-scrummed across opus + kimi + gemini. **All 3 verdict: BUILD-WITH-CHANGES.** All 4 v1 blockers verified RESOLVED. New v2 findings are tractable design fixes (not re-architecture). Folded in below. Reviews preserved at `/tmp/identity_scrum_v2/{opus,kimi,gemini}_review.md`. + +### Convergent v2 findings (≥2 reviewers) — must fix before implementation + +**v3-A1 — mTLS CA root must NOT live in identityd** (opus + gemini converge). v2 said "self-signed CA managed by identityd at startup" — if identityd is both the CA and authenticated party, compromise of identityd compromises the whole trust fabric. **v3 change:** mTLS CA root lives in **Vault PKI** (or, if Vault PKI is unavailable, an offline root with identityd as a properly-issued intermediate). identityd never has root-CA private key access. + +**v3-A2 — Dual-control public key registry must be tamper-evident** (opus + gemini converge). v2 said "pre-registered public keys" but didn't say where. If they live in identityd's Postgres, DB-write compromise defeats dual-control. **v3 change:** J + counsel public keys live in **Vault KV with separate access policies** (or a signed config file at `/etc/lakehouse/identityd_dual_control.yaml` with its own dual-control rotation procedure). Public-key changes themselves require dual-control attestation. Plus: `nonce_a`/`nonce_b` get **server-issued challenges** (identityd issues a fresh nonce per request; clients sign that nonce; replay-protection via 5-min nonce cache). + +### Single-reviewer v2 findings — also folded into v3 + +**v3-B1 (opus): Step 8 fallback-to-SQL needs explicit time bound.** Otherwise becomes permanent dual-read debt. v3: fallback active for max 14 days post-cutover; alert on every fallback; auto-disable at 14 days regardless of fallback rate. + +**v3-B2 (opus): NER drop-on-detect needs alerting.** Silent observability gap if model regression starts hallucinating PII. v3: drop counter exposed as `identityd_ner_drops_total` Prometheus metric; alert on drop_rate >0.1% over rolling 1h. + +**v3-B3 (opus): legal-tier notification transport must be specified.** v2 said "real-time notification to designated counsel + J" without saying how. v3: notification transport is **signed Slack webhook OR signed email** (configurable per deployment); message body never contains candidate_id or PII; only `{event_kind, timestamp, accessor_kind, integrity_hash}`. Notification failure does NOT block the legal-tier action — failure is logged for follow-up but token issuance proceeds. + +**v3-B4 (opus): Step 6 human review queue needs SLA + volume estimate.** With 500k backfilled rows defaulting to `unknown`, all healthcare-adjacent traffic routes to on-box Ollama until reclassified. v3: budget assumption — auto-pattern-matching pre-classifies ~80% to confidently-non-healthcare; remaining 20% (~100k) stays `unknown` pending review at ~500/day operator throughput = ~7 months of queue. **Operational decision needed:** either staff the queue, accept the on-box-Ollama-only routing for ~7 months, or relax pattern matching at the cost of higher false-negative rate. Flag for J. + +**v3-B5 (opus): Memory zeroing in Go is non-trivial.** v2 §3.1 step 6 says "explicitly zeros memory" — Go GC-managed strings are immutable, not zeroable. v3 implementation note: PII handling code path uses `[]byte` throughout; convert to string only at the JSON serialization boundary; explicit `runtime.GC()` after request completes is theater (Go won't actually zero the slice). **Acceptance:** "best-effort zero" — overwrite the `[]byte` slice contents post-use. Document this is best-effort, not cryptographic-grade scrubbing. (Rust-side: use the `zeroize` crate which IS cryptographic-grade.) + +**v3-B6 (kimi): Service-tier purpose_definitions needs versioning + emergency revocation.** A misconfigured purpose with overly-broad `allowed_fields` is standing exfiltration authorization. v3 schema additions: + +```sql +CREATE TABLE purpose_versions ( + purpose_token TEXT NOT NULL, + version INT NOT NULL, + effective_at TIMESTAMPTZ NOT NULL, + superseded_at TIMESTAMPTZ, + PRIMARY KEY (purpose_token, version) +); + +CREATE TABLE purpose_revocations ( + purpose_token TEXT PRIMARY KEY, + revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(), + revoked_by TEXT NOT NULL, + reason TEXT +); +``` + +Auth path checks `purpose_revocations` on every call (not cached, OR cached <5s). Revocation is <60s without code deploy. + +**v3-B7 (kimi): Cache invalidation needs erasure-generation atomicity.** v2 §3.6 pub/sub is best-effort. If Redis drops the message or gateway crashes between erase + purge, cache holds stale PII. v3 change: +- `subjects` table gains `erasure_generation INT NOT NULL DEFAULT 0`. Increments on every erase. +- Identityd PII responses include current generation in headers +- Gateway cache entries tagged with the generation they were filled at +- Cache hits with `cached_generation < current_generation` are rejected, force re-fetch from identityd +- Eventually-consistent within ONE request hop, not eventually-consistent over time + +**v3-B8 (kimi): Dual-control issuance needs cooling-off period to prevent bypass culture.** v3: 15-minute mandatory delay between nonce submission and token issuance, with notification to BOTH parties at delay-start AND issuance. Bypass requires both J and counsel to sign an "emergency bypass" attestation that's logged separately + escalated to a board-level reviewer. + +**v3-B9 (kimi): NER must have calibrated false-negative rate.** v3: maintain a synthetic adversarial test set (50 hallucinated-PII examples + 50 legitimate non-PII). NER must achieve recall ≥99.5% on the synthetic set in CI; production drop-rate >0.1% triggers re-calibration. + +**v3-B10 (kimi): S3 Object Lock root credentials need separate AWS account.** Same ops team holding Vault admin + AWS root = limited independence. v3: Object Lock bucket lives in **separate AWS account** with **write-only IAM role** issued to identityd. Root credentials for that account held by named external party (could be outside counsel) with quarterly attestation. + +**v3-B11 (kimi): biometric_consent_status='never_collected' may not satisfy BIPA infrastructure-as-notice argument.** Plaintiffs may argue presence of biometric schema fields constitutes constructive notice of intent. v3: Phase 1.6 BIPA policy doc must explicitly attest "no biometric data exists in the pre-identityd substrate" with cryptographic evidence (hash of pre-identityd schema). Counsel must bless this attestation. + +**v3-B12 (gemini): Backup retention window vs ciphertext-deletion erasure.** Phase 2 uses ciphertext deletion + single-DEK; if Postgres backups retain pre-erasure ciphertext + the DEK still exists, backup is recoverable. v3 erasure runbook: includes maximum backup retention period (e.g., 30 days). RTBF status reported to subject as "erased now; residual in backups expires by {date}." For full crypto-erasure-on-demand, Phase 7 per-row keys is the path. + +### Net v3 effort delta + +v2 estimate: 12-15 days. v3 amendments add: +- mTLS CA via Vault PKI integration: +0.5 day +- Public-key registry in Vault KV: +0.5 day +- Server-issued nonces for dual-control: +0.25 day +- Step 8 fallback time bound: +0.1 day +- NER drop-rate metric + alert: +0.25 day +- Legal-tier notification transport: +0.5 day +- purpose_versions + purpose_revocations: +0.5 day +- erasure_generation cache atomicity: +0.5 day +- 15-min cooling-off: +0.25 day +- NER calibrated test set: +0.5 day +- Separate AWS account for Object Lock: +0.5 day +- BIPA infrastructure-as-notice attestation: +0.25 day (mostly doc work) +- Backup-retention erasure runbook: +0.25 day + +**Total v3 delta: ~5 days. Revised estimate: 17-20 days.** + +The cost is real. The reason it's worth paying is what those 5 days buy: a design that 3 independent senior security architects (across 3 model lineages) all said they would build. v1 said "do not build." v3 says "build." + +### What's v3-deferred vs v3-must-have + +**v3 must-have (block implementation):** v3-A1, v3-A2, v3-B1, v3-B6, v3-B7, v3-B11. + +**v3 should-have (ship in Phase 2 if calendar allows; otherwise Phase 5):** v3-B2, v3-B3, v3-B4 (operational decision), v3-B5 (best-effort), v3-B8, v3-B9, v3-B10, v3-B12. + +If schedule pressure forces a cut, the should-have items can ship in Phase 5 (identity service build completion). The must-haves cannot — they're integral to the security boundary. + +--- + +## Change log + +- 2026-05-03 — v1 initial draft. +- 2026-05-03 — v2 post-first-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum recommended. +- 2026-05-03 — v3 post-second-scrum: 3/3 BUILD-WITH-CHANGES. v1 blockers verified resolved. 12 new v2 findings folded into v3 amendments (§12). Re-scrum NOT required for v3 — diminishing returns; the must-have items are concrete fixes with clear acceptance criteria. diff --git a/docs/PHASE_1_6_BIPA_GATES.md b/docs/PHASE_1_6_BIPA_GATES.md new file mode 100644 index 0000000..27c7167 --- /dev/null +++ b/docs/PHASE_1_6_BIPA_GATES.md @@ -0,0 +1,334 @@ +# Phase 1.6 — BIPA Pre-Launch Gates + +**Status:** Draft — 2026-05-03 · **Owner:** J + outside counsel · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md), [`IDENTITY_SERVICE_DESIGN.md`](IDENTITY_SERVICE_DESIGN.md) + +> **Why this exists.** `IDENTITY_SERVICE_DESIGN.md` v3 §5 Step 0 names Phase 1.6 as a HARD PREREQUISITE: identityd backfill cannot start until Phase 1.6 ships. This doc specifies what Phase 1.6 contains. +> +> **Scope.** BIPA (740 ILCS 14) compliance gates that must be in place BEFORE the system accepts a single real candidate photo. Synthetic-data face pool can keep operating; real-photo intake CANNOT begin without these gates. +> +> **Authority.** This is an engineering scaffold. Sections marked `⚖ COUNSEL` need outside counsel to author the actual legally-binding text. Engineering ships the procedural gates; counsel writes the words. + +--- + +## 1. The five BIPA pre-launch gates + +Each gate is a deliverable that must ship before real-photo intake. None is optional. Order shown is the recommended ship sequence. + +### Gate 1 — Public retention schedule (BIPA §15(a)) + +**Required:** A publicly-available, written retention schedule for biometric identifiers and information. + +**What ships:** +- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file +- Linked from public privacy policy at the deployment URL +- Specifies: + - Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`) + - Purpose of collection (identity matching for staffing operations) + - Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin) + - Destruction procedure: per Gate 5 below +- Versioned (this is v1; future updates supersede with a new version) + +**⚖ COUNSEL** — write the actual schedule. Engineering provides the operational facts; counsel writes the binding language. + +**Engineering acceptance:** the file is committed, the public URL renders it, and identityd's `consent_versions` table references it by hash. + +--- + +### Gate 2 — Informed written consent (BIPA §15(b)) + +**Required:** Informed, written consent BEFORE any biometric collection occurs. + +**What ships:** +- `docs/policies/consent/biometric_consent_template_v1.md` — public consent template +- Versioned, hashed, referenced from identityd's `consent_versions` table +- Must disclose, per BIPA §15(b)(1)-(3): + 1. That biometric identifiers/information will be collected + 2. The specific purpose for collection (and the length of term — references Gate 1) + 3. Receipt of a written release authorizing collection +- Consent flow at intake: + - Candidate sees the disclosure on a UI surface (web form / paper / digital signature) + - Candidate provides explicit affirmative action (signature, click-acceptance with timestamp, etc.) + - Identityd records `biometric_consent_status='given'` with `consent_version` reference + `consent_given_at` timestamp + - **Without identityd recording 'given', no biometric data flows through deepface.** + +**⚖ COUNSEL** — write the consent template. Recommended content (engineering view): +- Clear language (not just legal boilerplate) +- Specific to facial-classification (not generic biometrics) +- Includes withdrawal procedure +- Includes data-subject rights enumeration + +**Engineering acceptance:** consent gate is enforced in code at the photo-upload endpoint; identityd refuses biometric writes when `biometric_consent_status != 'given'`; pre-existing synthetic-face pool is exempt (no consent needed because no real subject). + +--- + +### Gate 3 — Photo-upload endpoint with consent enforcement + +**Required:** Code-level enforcement that real-photo intake checks consent before processing. + +**What ships:** + +An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior: + +1. Caller authenticates with service-tier token +2. Endpoint queries identityd for `subjects.biometric_consent_status` +3. If status ≠ `'given'` → HTTP 403 with reason `"BIPA consent required before biometric processing"` +4. If status = `'given'`: + a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`) + b. deepface tagging runs against the photo + c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1. + d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule + e. `pii_access_log` row written with `purpose_token='biometric_collection'` +5. Response: `{candidate_id, retention_until, consent_version}` + +**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):** + +The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type. + +```rust +// crates/catalogd/src/subject_manifest.rs (paraphrased) +pub struct BiometricCollection { + pub data_path: String, // quarantined path + pub template_hash: String, // SHA-256 of original bytes (integrity, NOT re-derivation) + pub collected_at: DateTime, + pub classifications: Option, // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md) +} +``` + +**Engineering acceptance:** +- Endpoint refuses uploads when consent missing (verified by integration test) +- deepface output never lands in the synthetic-face manifest (`data/headshots/manifest.jsonl`) +- Real-photo classifications are isolated to identityd `subjects` table — never flow to JSONL sinks +- The `/headshots/:key` route in mcp-server REMAINS synthetic-only — does NOT serve real candidate photos to LLMs without an explicit allowance (proposed: real photos served only to authenticated staffer UI, never to model context) + +--- + +### Gate 4 — Deprecate name → ethnicity inference + +**Required:** The hard-coded `NAMES_HISPANIC` / `SURNAMES_*` lookup tables in `mcp-server/search.html:3375-3432` (per Phase 1.5 §1B walk) get removed. + +**What ships:** +- A code commit that removes: + - `FEMALE_NAMES`, `MALE_NAMES` constants + - `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` constants + - `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` constants + - The `genderFor()` and `guessEthnicityFromFirstName()` functions + - All call sites that consumed these (face-pool bucket selection) +- Replacement strategy: + - For SYNTHETIC face pool routing: deterministic hash of candidate_id selects a face bucket, no demographic inference + - For REAL candidate photos: the candidate's actual photo IS the representation; no inference needed + +**Why this is BIPA + Title VII risk separately:** name-based ethnicity classification is BOTH a discriminatory feature engineering practice (Title VII) AND, when combined with photo-based attribute extraction, a "biometric information derived from a biometric identifier" pattern (BIPA broad reading). Removing the lookup tables forecloses both arguments. + +**Engineering acceptance:** +- Lookup tables removed from search.html +- Unit test asserts no protected-attribute inference functions exist in search.html or any mcp-server module +- Face-pool routing for synthetic faces uses candidate_id hash exclusively +- Phase 1.5 §1B finding closed + +--- + +### Gate 5 — Documented destruction procedure + +**Required:** A written procedure for biometric data destruction at retention expiry OR consent withdrawal OR right-to-be-forgotten request. + +**What ships:** +- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing +- Specifies: + - Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request + - Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly + - Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads//*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails + - Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed + - Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction) + - Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request + +**⚖ COUNSEL** — review the runbook for legal sufficiency. Engineering writes the procedure; counsel attests that the procedure satisfies BIPA §15(a) destruction requirements. + +**Engineering acceptance:** +- Runbook committed +- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full) +- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock) +- Erasure events are logged with cryptographic attestation + +--- + +## 2. Cryptographic attestation: no biometric data exists pre-identityd + +**Per `IDENTITY_SERVICE_DESIGN` v3-B11.** Plaintiffs may argue that the EXISTENCE of biometric schema fields constitutes constructive notice of intent to collect biometric data — therefore consent should have preceded the schema. The defense: prove that no biometric data was actually collected from real candidates before identityd + the consent gate. + +**What ships:** +- A one-shot script `scripts/staffing/attest_pre_identityd_biometric_state.sh` that: + - Queries `data/datasets/workers_500k.parquet` schema and confirms NO column named `photo`, `biometric_*`, `face_*`, `image_*` exists + - Greps `data/_kb/*.jsonl` and `data/_pathway_memory/state.json` for any base64-encoded image bytes (deepface output, photo blobs) + - Verifies `data/headshots/manifest.jsonl` rows ≤ synthetic face pool size + - Hashes the schema + summary; commits the hash to S3 Object Lock (per identity service v3 anchor pattern) +- Attestation document `docs/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-XX.md` signed by J + outside counsel + +**This is a one-time defense artifact.** It establishes the baseline: "as of this date, no biometric data was collected from real candidates." + +--- + +## 3. Employee training acknowledgment (general BIPA hygiene) + +**Required:** People with access to biometric data acknowledge BIPA-handling training. + +**What ships:** +- `docs/policies/BIPA_HANDLING_TRAINING_v1.md` — training material covering: + - What constitutes biometric identifiers / information + - The consent + retention procedures + - Destruction obligations + - Reporting suspected exposure +- Acknowledgment record per individual (initially: J + counsel + named operators) +- Annual refresh + +**⚖ COUNSEL** — write training content. Engineering doesn't author legal-compliance training. + +--- + +## 4. Phase 1.6 exit criteria (gates Phase 2 backfill) + +All 5 gates must be DONE before identityd backfill begins. Status as +of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code: + +| # | Gate | Engineering | Counsel | Status | +|---|---|---|---|---| +| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** | +| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** | +| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** | +| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** | +| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** | + +PLUS: + +| # | Item | Engineering | Counsel | Status | +|---|---|---|---|---| +| 6 | Cryptographic attestation pre-identityd | DONE — `scripts/staffing/attest_pre_identityd_biometric_state.sh` + `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (3/3 evidence checks pass; signature lines pending) | pending signature | **eng-DONE, signature-pending** | +| 7 | Employee training material | scaffold deferred — Gate 5 runbook §7 acknowledgment may serve as substrate | pending | **deferred** | + +**Blocking set for Phase 2 backfill:** items **1, 2, 3, 4, 5, 6** must +all be DONE. Item 7 (employee training) is reduced from blocking to +"deferred" because the Gate 5 destruction runbook §7 already requires +operator acknowledgment before legal-tier credentials are issued — +that acknowledgment is procedurally equivalent to the training-record +requirement when the operator population is small (J + 1-2 named +operators). If the operator population grows beyond that, item 7 +re-promotes to blocking and a separate training program must be authored. + +⚖ COUNSEL — confirm whether item 7 deferral is acceptable for the +expected operator population size, or restore it to the blocking set. + +**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel +review of the engineering scaffolds. Gate 3 substrate is fully +shipped; Gate 3b deepface classification was DECIDED on 2026-05-05 +as Option C (defer) — `BiometricCollection.classifications` stays +`None` in v1, consent + retention docs revised to match this +narrower scope. If a future product requirement surfaces a real +need for classifications, the substrate is forward-compatible +(`Option`) and either Option A (~1 day) or Option B (~5 days) +of the design doc can be picked up then under a v2 consent template. + +--- + +## 5. Effort estimate + +| Gate | Engineering effort | Legal effort | +|---|---|---| +| Gate 1 (retention schedule) | 0.5 day | counsel-dependent (typically 1-2 weeks for review) | +| Gate 2 (consent template) | 0.5 day | counsel-dependent (typically 2-4 weeks for review and consent UX design) | +| Gate 3 (photo-upload endpoint) | 1-2 days | review of endpoint behavior | +| Gate 4 (deprecate name-ethnicity inference) | 0.5 day | none (engineering-only fix) | +| Gate 5 (destruction runbook) | 1 day | counsel sign-off | +| §2 cryptographic attestation | 0.5 day | counsel + J signature | +| §3 employee training | 0.25 day (admin) | counsel-authored content | +| **Total engineering** | **~4-5 days** | — | +| **Total counsel** | — | **~3-6 weeks calendar** (review cycles) | + +**The calendar bottleneck is counsel, not engineering.** Engineering can stage all 5 gates ready-to-ship in a week. Counsel sign-off + consent UX rollout is the longer pole. + +--- + +## 6. Open questions for J + counsel + +1. **Photo-upload UX:** is there an existing intake form / staffer console where photo upload would happen? Or is this new UI work? +2. **Consent collection mechanism:** electronic signature service (DocuSign, Adobe Sign), in-app click-acceptance, paper form? Each has different evidentiary weight in litigation. +3. **Operator list with biometric access:** who, today, would be on the named-operators list for §3 training? +4. **Counsel for sign-off:** named outside counsel — same or different from the dual-control legal-token party in identity service? +5. **Public privacy policy URL:** does one exist? If yes, where; if no, that's a separate Gate-1.5 deliverable. + +--- + +## 6.5. Post-signature deployment runbook + +When counsel returns the countersigned consent template + retention +schedule, the engineering side of "flip from permissive to strict +mode" is one command: + +```bash +# 1. Counsel commits their signature to §7 of the consent template +# markdown (or J commits the signed PDF + updates §7 with counsel's +# name + date). The markdown is the BINDING TEXT — the PDF is just +# a rendering of it. + +# 2. Hash the canonical signed text + seed the gateway allowlist. +./scripts/staffing/seed_consent_version.sh \ + docs/policies/consent/biometric_consent_template_v1.md \ + --label "v1 signed YYYY-MM-DD by [counsel name]" + +# The script: +# - computes SHA-256 of the markdown (binding text) +# - atomically writes /etc/lakehouse/consent_versions.json with +# the new hash + per-seed audit metadata (timestamp, label, +# source path) +# - restarts lakehouse.service so the gateway re-reads the +# allowlist +# - probes /biometric/health for clean restart + +# 3. Verify strict mode is rejecting unknown hashes: +TOKEN=$(cat /etc/lakehouse/legal_audit.token) +curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \ + -H "X-Lakehouse-Legal-Token: $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"consent_version_hash":"deadbeefdeadbeef000000000000000000000000000000000000000000000000","consent_collection_method":"electronic_signature","operator_of_record":"strict_mode_probe"}' +# Expect: HTTP 400 + {"error":"consent_version_unknown", ...} +``` + +After this, the gateway is in counsel-tier strict mode: + +- Any consent grant POST whose `consent_version_hash` doesn't match + a known signed template is refused at intake +- Operator typos (mistyped hash) become loud failures, not silent + bad records +- Future template revisions (v2, v3, ...) require counsel re-sign + AND a new `seed_consent_version.sh` run before being accepted — + the v1 hash stays in the allowlist for already-collected subjects' + audit-trail compatibility + +### Pre-counsel demo seed + +For deployments that want to exercise strict mode BEFORE counsel +signs, the same script works against the eng-staged template: + +```bash +./scripts/staffing/seed_consent_version.sh \ + docs/policies/consent/biometric_consent_template_v1.md \ + --label "eng-staged demo seed (NOT counsel-signed)" +``` + +The hash entry should be replaced (rotate the demo hash out, add +the counsel-signed hash) when counsel completes review. The +allowlist's `_meta.seeded_at[]` array preserves the seed history. + +--- + +## 7. What this PRD is NOT + +- Not legal advice. The `⚖ COUNSEL` markers exist because the binding text needs lawyers, not engineers. +- Not a substitute for a DPIA / PIA. Phase 1.6 satisfies BIPA-specific gates; a Data Protection Impact Assessment is broader and may be required separately. +- Not a SOC2 Type II deliverable. SOC2 is a parallel work stream. +- Not the only gate before production. The full 9-phase audit-trail program continues; Phase 1.6 specifically unblocks Phase 2 (identity service implementation). + +--- + +## Change log + +- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup. +- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill. diff --git a/docs/PRD.md b/docs/PRD.md index 0a10db4..2de7c54 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -77,7 +77,7 @@ A modular Rust service mesh over S3-compatible object storage, with a local AI l | Data Format | Parquet + Arrow | Yes | | RPC (internal) | tonic (gRPC) | Yes | | AI Runtime | Ollama (local models) | Yes | -| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes | +| AI Boundary | Direct Ollama HTTP API (gateway → Ollama, no sidecar). Updated 2026-05-02 per ADR-022 — was "Python FastAPI sidecar → Ollama HTTP API". Sidecar's lab_ui/pipeline_lab Python remain as dev-only tools (not on hot path). | Yes | | Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** | No new frameworks without documented ADR. @@ -97,7 +97,7 @@ No new frameworks without documented ADR. | **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog | | **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) | | **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal | -| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar | +| **aibridge** | AI client — direct Ollama HTTP (per ADR-022, 2026-05-02; was Rust↔Python sidecar boundary). Owns LRU embed cache. | | **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs | | **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** | | **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) | @@ -161,6 +161,18 @@ The feedback signal is **statistical + semantic**, not neural. No model training --- +## Production-readiness blocker (2026-05-03) + +Substrate is mostly shipped through Phase 45. Throughput, parity, and observability are all measured. + +**The production-readiness gate is the audit trail.** Staffing client requires defensible response to discrimination claims (worked example: John Martinez at Warehouse B requests every AI-system decision affecting him). PII tokenization, subject-of-record query, retention SLA, and right-to-be-forgotten under append-only logs are NOT YET BUILT. + +Authoritative document: **`docs/AUDIT_TRAIL_PRD.md`**. 9-phase implementation plan, current-state-vs-target gap table, open questions blocking phase 1. + +Until the audit-trail PRD's phase 9 exit criterion is met, "production-ready" claims are aspirational. Internal substrate (Lance, observability parity, cross-runtime fabric) is solid; subject-audit story is not. + +--- + ## Phases ### Phase 0-5: Foundation ✅ COMPLETE diff --git a/docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md b/docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md new file mode 100644 index 0000000..7e713b6 --- /dev/null +++ b/docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md @@ -0,0 +1,91 @@ +# BIPA Pre-IdentityD Biometric Attestation + +**Date:** 2026-05-03 +**Spec:** docs/PHASE_1_6_BIPA_GATES.md §2 +**Generator:** scripts/staffing/attest_pre_identityd_biometric_state.sh + +## Purpose + +This is a one-time defense artifact establishing that, as of +2026-05-03, no biometric identifiers or biometric information +from real candidates have been collected, processed, or stored +by the Lakehouse system. It is intended to be signed by J +(operator of record) and outside counsel, then anchored to a +tamper-evident store (filesystem with backups + version control). + +## Evidence + +## Check 1 — workers_500k.parquet schema (no biometric columns) + +**Source:** `data/datasets/workers_500k.parquet` + +**Schema columns** (18 total): + +``` +worker_id int64 nullable=True +name string nullable=True +role string nullable=True +email string nullable=True +phone string nullable=True +city string nullable=True +state string nullable=True +zip int64 nullable=True +skills string nullable=True +certifications string nullable=True +archetype string nullable=True +reliability double nullable=True +responsiveness double nullable=True +engagement double nullable=True +compliance double nullable=True +availability double nullable=True +communications string nullable=True +resume_text string nullable=True +``` + +**Schema SHA-256:** `973b9abe56420de8f88122278b633e813f90a64cf0ddaac6a9811dc0940be676` + + - PASS: no biometric / photo / face / image column present + +## Check 2 — KB + pathway memory contain no biometric payloads + +**Sources scanned:** +- `data/_kb/*.jsonl` (knowledge base) +- `data/_pathway_memory/state.json` (pathway memory state) + +**Files scanned:** 33 +**Forbidden-pattern hits:** 0 + + - PASS: no biometric payload patterns found in scanned files + +## Check 3 — Headshots manifest is synthetic-only + +**Source:** `data/headshots/manifest.jsonl` + +**Total rows:** 1000 +**Rows tagged real/candidate_upload/photo_upload:** 0 + + - PASS: all 1000 rows are synthetic (no real-candidate uploads) + +## Summary + +**3 / 3** evidence checks pass. + + +--- + +## Attestation + +I, the undersigned, attest that the above evidence accurately +reflects the state of the Lakehouse system as of 2026-05-03. +No biometric identifiers or biometric information from real +candidates have been collected, processed, or stored prior to +the deployment of the Phase 1.6 BIPA pre-launch gates. + +**Evidence SHA-256:** `1fdcc9f1682de27e1a0556d698ce221b74c1e71cf54128763828b4bca7b5c1bf` + +--- + +**Operator (J):** _______________________________ Date: __________ + +**Outside counsel:** ___________________________ Date: __________ + diff --git a/docs/counsel/COUNSEL_HANDOFF_EMAIL_2026-05-05.md b/docs/counsel/COUNSEL_HANDOFF_EMAIL_2026-05-05.md new file mode 100644 index 0000000..1358e8f --- /dev/null +++ b/docs/counsel/COUNSEL_HANDOFF_EMAIL_2026-05-05.md @@ -0,0 +1,187 @@ +# Counsel Handoff — Copy-Paste Email Draft + +**Date assembled:** 2026-05-05 +**Operator of record:** J +**Prepared for:** outbound transmission to outside counsel +**Tarball:** `reports/counsel/counsel_packet_2026-05-05.tar.gz` +**Tarball SHA-256:** see `reports/counsel/counsel_packet_2026-05-05.manifest.txt` + +> This is a copy-paste-ready email draft. Subject line + body below. +> Attach the tarball + manifest before sending. The email frames +> what counsel needs to do, by when, and what happens after they +> sign — so the calendar bottleneck has no excuse to sit longer +> than counsel's actual review time. + +--- + +## Subject line + +``` +Lakehouse staffing platform — Phase 1.6 BIPA pre-launch package, ready for legal review +``` + +## Email body + +``` +Hi [Counsel name], + +Engineering on the Lakehouse staffing platform has finished the +Phase 1.6 BIPA (740 ILCS 14) pre-launch substrate. Before we begin +collecting any real candidate biometric data (photographs), the +five engineering-staged documents in the attached package need +your review and, where indicated, your signature. + +What's attached: + +1. counsel_packet_2026-05-05.tar.gz — bundled review package +2. counsel_packet_2026-05-05.manifest.txt — per-file SHA-256 + integrity hashes (re-run `sha256sum -c` on receipt to verify + nothing changed in transit) + +Open the cover letter inside the tarball +(`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`) first — it +walks through the substrate, the specific items needing your +sign-off, and recommended review sequence. + +Headline asks (in priority order): + + A. Biometric Retention Schedule v1 + — render into binding language, sign as countersigning + party. Sets the public retention schedule required by + BIPA §15(a). + B. Biometric Consent Template v1 + — render Disclosures 1-3 into binding consent text, sign. + This is the form a candidate signs before any photo is + collected. Required by BIPA §15(b)(1)-(3). + C. BIPA Destruction Runbook + — review for legal sufficiency, attest. Procedural + document; engineering wrote the steps, your role is + confirming they satisfy BIPA §15(a) destruction. + D. Pre-IdentityD Attestation (2026-05-03) + — countersign as the legal party. Establishes the boundary + that no biometric data was collected before the gates + shipped. One-time defense artifact. + +Item C also references a key-rotation runbook (E in the cover +letter) that's lower priority — opine when convenient. + +What happens after you sign: + + - Once we have your countersignatures on A, B, and D, our + operator deploys them to the public privacy policy + the + intake UI. Engineering will hash the canonical signed text + of (B) and add that hash to the gateway's consent_versions + allowlist file (/etc/lakehouse/consent_versions.json) so the + runtime starts refusing any consent record that wasn't + signed under the canonical template. (The gateway is + currently in permissive mode — accepting any non-empty + hash. Strict mode flips on with your signature.) + - The runtime gates are already enforcing the consent flow: + photo upload refuses 403 unless consent.biometric.status = + "given", and the only path to that state is the + /biometric/subject/{id}/consent endpoint with a valid + consent_version_hash. After your signature, that hash will + point at canonical signed text — operator typo-resistant + and counsel-defensible. + +Calendar: + + - We'd like to begin real-photo intake within [N] weeks. If + your review can land within 2-3 weeks of receipt, that + keeps us on schedule. If your bandwidth requires longer, + please flag — we have eng work to advance independently + that doesn't depend on the signatures. + +Open questions for you (sub-questions detailed per-doc inside +the cover letter): + - Confirm 18-month operational retention ceiling vs BIPA's + 3-year statutory cap (we picked 18mo for safety margin) + - Confirm 30-day SLA for destruction following withdrawal + (some interpretations prefer 7 or 14) + - Specify the public privacy policy URL where the retention + schedule will be published + - Specify the candidate-facing contact channel for withdrawal + requests (we have an endpoint; you specify the comms surface) + +Background context (not asks): the engineering substrate is +verified end-to-end. We've run a live demo of the full lifecycle +(consent → photo → withdraw) on a test subject; the audit chain +verifies cryptographically; the retention sweep is scheduled +daily. No real candidate photos have been collected. The +attached pre-identityd attestation has the cryptographic evidence +hashes from that pre-collection state. + +Happy to schedule a call if any of this would be easier to walk +through verbally. + +Thanks, +J +[contact info] +``` + +--- + +## Operator pre-flight checklist (before sending) + +Run through this once before clicking send: + +- [ ] Confirm tarball + manifest are the latest versions + (`bundle_counsel_packet.sh` regenerates if needed; current + hash is in the manifest sidecar) +- [ ] Confirm counsel name is correct (and that the engagement + letter or retainer is in place) +- [ ] Confirm the [N] weeks calendar target — replace with a + real number ("3 weeks" / "by end of June" / etc.) +- [ ] Confirm the email goes to counsel's secure channel; if + counsel uses a portal, upload there + send a notification + email instead +- [ ] Save a copy of the sent email + the tarball to operator + records — this is itself part of the audit trail of "we + asked counsel on [date]" + +--- + +## Post-signature operator runbook (after counsel responds) + +When counsel returns the signed documents: + +1. **Verify integrity.** Run `sha256sum -c manifest.txt` on + the returned tarball if counsel returns the bundle; otherwise + compare returned signed PDFs against the markdown sources to + confirm the binding text matches what counsel reviewed. + +2. **Capture the canonical signed text.** Counsel may sign a + PDF rendering of the markdown — that's fine, but the BINDING + TEXT we hash is the markdown source (counsel commits to git + countersigning §7 of the consent template + §8 of the + retention schedule). + +3. **Compute + seed the consent_version_hash.** Run: + ```bash + ./scripts/staffing/seed_consent_version.sh \ + docs/policies/consent/biometric_consent_template_v1.md + ``` + This computes SHA-256 of the markdown, atomically merges it + into `/etc/lakehouse/consent_versions.json` (creating the + file if absent), and offers to restart the gateway. + +4. **Verify strict mode is live.** Probe with a known-bad hash: + ```bash + TOKEN=$(cat /etc/lakehouse/legal_audit.token) + curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \ + -H "X-Lakehouse-Legal-Token: $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"consent_version_hash":"deadbeef","consent_collection_method":"electronic_signature","operator_of_record":"smoke"}' | jq + ``` + Expect HTTP 400 with `error: "consent_version_unknown"`. + +5. **Update STATE_OF_PLAY.md** to record the signature event + + the hash entry in the allowlist. Counsel-tier deployment is + now live. + +6. **Sign and anchor the §2 attestation.** Both J and counsel + sign `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` + and the SHA-256 of the signed markdown is committed to the + git history as a tamper-evident anchor. + +After step 5, the production cutover blocker is closed. diff --git a/docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md b/docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md new file mode 100644 index 0000000..7dc61f1 --- /dev/null +++ b/docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md @@ -0,0 +1,260 @@ +# Counsel Review Packet — Phase 1.6 BIPA Pre-Launch + +**Date assembled:** 2026-05-05 +**For:** outside counsel +**From:** J, operator of record +**Scope:** documents that engineering has staged for legal sufficiency review + before the staffing platform begins collecting any real candidate + biometric data (BIPA §15(a)(b)). + +> **What this packet is.** The Phase 1.6 BIPA gates outline what +> engineering must ship before real-photo intake. As of 2026-05-05, +> all engineering substrate is shipped and verified live (see §1 +> below for the inventory). What remains is binding-text authoring +> + counsel sign-off on five documents, plus operational notification +> obligations counsel may want to layer on top. +> +> **What this packet is NOT.** Not a request for counsel to write +> binding text from scratch. The documents are eng-staged in +> reasonable plain language; the request is for counsel to render +> them into legally-sufficient text and attest where signatures +> are required. + +--- + +## 1. Engineering substrate — shipped + verified + +For factual context on what counsel is reviewing AGAINST. None of +this requires sign-off here; it's the system the documents bind to. + +| Component | Where it lives | Verification | +|---|---|---| +| Subject manifest registry | `crates/catalogd/src/registry.rs`, `data/_catalog/subjects/.json` | 17 unit tests + 100 backfilled WORKER manifests in production | +| Per-subject HMAC audit chain (SHA-256) | `crates/catalogd/src/subject_audit.rs`, `data/_catalog/subjects/.audit.jsonl` | Tamper-detection + concurrent-append race tests pass | +| Photo upload (consent-gated) | `POST /biometric/subject/{id}/photo` | 11 unit tests + live roundtrip 200 | +| Erasure (two-scope) | `POST /biometric/subject/{id}/erase` (`biometric_only` / `full`) | 21 unit tests; transactional rollback on audit failure | +| Legal-tier audit read | `GET /audit/subject/{id}` (X-Lakehouse-Legal-Token header) | Constant-time auth, chain re-verification per request | +| Retention sweep (BIPA-aware clock) | `crates/catalogd/src/bin/retention_sweep` | 8 unit tests; live verified against 100 backfilled subjects | +| Cross-runtime parity (Rust ↔ Go) | `scripts/cutover/parity/subject_audit_parity.sh` | 6/6 byte-identical assertions pass | + +**Key insight for counsel:** the audit chain is the BIPA-defensible +substrate. Every state-changing event (consent given, photo uploaded, +photo erased, legal-tier read) appends to a per-subject HMAC-chained +JSONL log. The chain verifies end-to-end on every legal-tier read. +A tampered chain is detectable; a forged chain requires the HMAC +signing key, which is held under root-only mode 0400 at +`/etc/lakehouse/subject_audit.key` and rotated per the runbook in +attachment §6 below. + +**Gate 3b (deepface classification) — decided 2026-05-05: Option C +(defer).** The system collects only the photograph, not derived +demographic information. The consent template + retention schedule +in this packet were revised the same day to match. + +--- + +## 2. Documents requiring counsel review + sign-off + +In recommended review order: + +| # | Document | Path | Counsel ask | Sign-off | +|---|---|---|---|---| +| A | Biometric Retention Schedule v1 | `docs/policies/consent/biometric_retention_schedule_v1.md` | Render into binding language; confirm 18-month operational ceiling vs. BIPA 3-year statutory cap | Counsel + J | +| B | Biometric Consent Template v1 | `docs/policies/consent/biometric_consent_template_v1.md` | Render Disclosures 1-3 into binding consent language; specify electronic vs. paper signature mechanism | Counsel + J | +| C | BIPA Destruction Runbook | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` | Confirm 30-day SLA from trigger; confirm two-operator (operator + witness) requirement; confirm legal-hold check procedure | Counsel attestation | +| D | BIPA Pre-IdentityD Attestation | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` | Sign as countersigning party; J signs as operator-of-record | Counsel + J | +| E | Legal-Tier Audit Key Rotation | `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` | Confirm rotation cadence; opine on candidate-notification obligation when rotation is compromise-driven | Counsel notes | +| F | Gate 3b Deepface Design (FYI) | `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` | Decision-of-record showing classifications were *deliberately deferred*, not omitted by oversight. No sign-off needed; provided for audit-trail completeness. | None | + +The five documents requiring sign-off are A, B, C, D, E. Document F +is included so the audit trail shows the Gate 3b decision was +deliberate. + +--- + +## 3. Specific questions for counsel — by document + +### Document A — Retention Schedule + +1. The schedule sets an **18-month** operational ceiling against the + BIPA 3-year statutory cap. Is the safety margin appropriate, or + should we move to a tighter window (12 months) given the + plaintiff-friendly Illinois posture? +2. The schedule references the **catalogd-local** storage substrate + rather than a separate identityd Postgres table. Does the + public-facing language need to mention the storage architecture + at all, or is "we keep the photo and a SHA-256 hash" sufficient? +3. Public publication URL — counsel to specify (placeholder marked + in §7 of the schedule). +4. Confirm whether existing consent under v1 carries forward when + a future v2 is published, or whether re-consent is required. + +### Document B — Consent Template + +1. Disclosure 1 says "we do NOT run automated facial-classification + in v1." Does that disclosure need to mention the *possibility* of + future classification, or is silence-with-supersession-clause + adequate? +2. Plain-language summary in §1 — counsel to confirm it's appropriate + to include alongside the binding disclosure, or recommend an + alternative comprehension aid. +3. Withdrawal SLA is set to **30 days** in §2. Counsel to confirm + against jurisdiction (Illinois primary; secondary deployments + would inherit). +4. Contact for withdrawal — counsel to specify the channel + (placeholder in §3). +5. Sign-off mechanism: electronic signature service, in-app + click-acceptance with timestamp, paper form? Each has different + evidentiary weight. + +### Document C — Destruction Runbook + +1. Confirm 30-day SLA from each of four triggers (retention expiry, + consent withdrawal, RTBF, court order). Some interpretations + prefer 7 or 14 days for withdrawal/RTBF. +2. Two-operator requirement (operator-of-record + witness): is the + witness role acceptable for counsel's defensibility view, or + should we elevate to dual-control with cryptographic split-key? +3. Legal-hold check procedure (§2 step 3) — counsel to specify the + actual procedure for confirming no hold is in force before + erasing. +4. Backup-window disclosure (§4) — confirm 30-day backup retention + is acceptable. +5. Candidate notification template (§3 step 4) — counsel to supply. + +### Document D — Pre-IdentityD Attestation + +1. Both signature lines blank — J signs as operator-of-record; + counsel signs as the countersigning legal party. +2. The attestation hash anchors the evidence; once signed, the + hash itself becomes a tamper-evident witness. Counsel to confirm + storage location for the signed copy (firm files?). + +### Document E — Key Rotation Runbook + +1. Recommended rotation cadence — 90 days suggested in §1. + Counsel to confirm or override. +2. Custody schedule for `/etc/lakehouse/_archived/` raw key files — + §7.2 question; suggested 1-year retention but counsel-driven. +3. Candidate-notification obligation when rotation is + compromise-driven (§7.3) — counsel call. + +--- + +## 4. Engineering changes counsel should know about (recent) + +These reconciled doc/code drift after a rapid wave on 2026-05-03: + +- **Endpoint paths:** the original v1 spec proposed + `/v1/identity/subjects/*` under a separate identityd daemon. That + daemon was collapsed into catalogd; endpoints actually shipped at + `/biometric/subject/*` (catalogd-local). Documents in this packet + reference the catalogd-local routes; legacy references in + `IDENTITY_SERVICE_DESIGN.md` are flagged "do NOT implement + as-written" in that doc's deprecation header. +- **No identityd Postgres database:** the original spec proposed + encrypted-at-rest Postgres + HashiCorp Vault + S3 Object Lock for + PII storage. The shipped substrate is local JSON manifests + + per-subject HMAC-chained JSONL, sized for J's local-only + deployment per `PRD.md` line 70 ("Everything runs locally — no + cloud APIs"). +- **Gate 3b deferral (Option C, 2026-05-05):** classifications + (gender / race / age inference) were deliberately deferred. The + consent template and retention schedule in this packet do NOT + disclose collection of derived demographic data, because we are + not collecting it. If a future product requirement reverses this, + we will publish a v2 consent + v2 retention with re-consent. +- **Key rotation 2026-05-05:** the prior `LH_SUBJECT_AUDIT_KEY` was + lost when a `/tmp` wipe on reboot disabled the audit and biometric + endpoints. The new key is at `/etc/lakehouse/subject_audit.key` + (mode 0400). Pre-rotation audit chains tamper-detect under the + new key — this is correct, expected behavior, not a bug. + +--- + +## 5. Open eng items NOT awaiting counsel + +For transparency. These are engineering work items, not legal items: + +1. **Residual photo unlink on erasure.** During verification of the + one historical erasure event (`WORKER-2`), the verify script + surfaced a stranded photo file that was not unlinked when + `BiometricCollection` was cleared from the manifest. Engineering + investigates; if the bug is real, the fix is `crates/catalogd/src/biometric_endpoint.rs` + in the erasure handler. This does NOT affect the current packet — + no real candidate photos have been collected yet (per §1 + attestation), so the residual is from a synthetic test event. +2. **Phase 1.6 §3 employee training.** Currently deferred per + acknowledgement coverage in §7 of the destruction runbook + (single-operator population). Re-promotes to blocking if the + operator population grows; counsel may want to opine on the + threshold. + +--- + +## 6. Sign-off sequence + +Recommended order so a hold-up on one doc doesn't block others: + +1. **First wave (parallel):** A (retention schedule) + B (consent + template). These two have the tightest interdependence (consent + v1 references retention v1 by hash); review them together. +2. **Second wave:** C (destruction runbook). Depends on A's retention + period being fixed. +3. **Third wave:** D (pre-identityd attestation). Sign once A + B + C + are settled; the attestation snapshot is the boundary between + pre-Phase-1.6 and post-Phase-1.6 system state. +4. **Fourth wave:** E (key rotation). Independent of A-D; can be + reviewed in parallel any time. + +--- + +## 7. After sign-off — engineering steps + +Once each document is signed: + +| Document | Engineering action | Trigger | +|---|---|---| +| A retention schedule | Hash + commit; reference in `consent_versions` table | Counsel signature | +| B consent template | Hash + commit; reference in candidate-facing intake UI | Counsel signature | +| C destruction runbook | Adopt; operator acknowledgment recorded in §7 | Counsel attestation | +| D pre-identityd attestation | Anchor hash to filesystem + git; counsel keeps original signature | Both signatures | +| E key rotation | Adopt; rotation event log seeded with counsel-approved cadence | Counsel notes | + +The HARD blocker for first real-candidate photo collection is +A + B + D signed. C and E are operationally important but do not +block the *first* photo (they govern destruction + key handling +which apply to any state, not the boundary state). + +--- + +## 8. Cover-note hash + +This packet is itself a snapshot. Future-Claude / future-J will refer +back to this packet to know what counsel saw on 2026-05-05. + +**Packet attached files (referenced by path):** + +- `docs/policies/consent/biometric_retention_schedule_v1.md` +- `docs/policies/consent/biometric_consent_template_v1.md` +- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` +- `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` +- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` +- `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` +- `docs/PHASE_1_6_BIPA_GATES.md` (the spec they all reference) + +Per-file SHA-256 hashes are produced by the bundler script (next +section); the bundler also creates a tarball ready for transmission. + +--- + +## 9. Generating the bundle for transmission + +```bash +./scripts/staffing/bundle_counsel_packet.sh +``` + +Produces `reports/counsel/counsel_packet_.tar.gz` with all +referenced documents + a manifest listing per-file SHA-256 hashes. +Counsel can verify file integrity on receipt by re-running +sha256sum against each file in the tarball. diff --git a/docs/policies/consent/biometric_consent_template_v1.md b/docs/policies/consent/biometric_consent_template_v1.md new file mode 100644 index 0000000..445097e --- /dev/null +++ b/docs/policies/consent/biometric_consent_template_v1.md @@ -0,0 +1,175 @@ +# Biometric Information Consent — v1 + +**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 2 (BIPA §15(b)(1)-(3)) +**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before deployment +**Version:** v1 (initial; supersession requires a new version + new hash) +**Updated 2026-05-05:** Disclosure 1 + plain-language summary revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). Pending J's product confirmation of Gate 3b; if Gate 3b chooses Option A or B, this template needs counsel re-authoring. + +> This is the consent template a candidate signs (electronically or +> on paper) BEFORE Lakehouse collects, stores, or processes any +> biometric identifier or biometric information from that candidate. +> +> Without an executed consent under this template (or a counsel- +> approved successor), the system MUST NOT accept a photograph from +> the candidate. Enforcement lives at the photo-upload endpoint +> (Gate 3) and at the SubjectManifest writer, which refuses biometric +> writes when `consent.biometric.status != "given"`. + +--- + +## Required disclosures (BIPA §15(b)(1)-(3)) + +The disclosures below are MANDATORY content per 740 ILCS 14/15(b). +⚖ COUNSEL — render this content into binding language appropriate +for the candidate-facing UI. Engineering provides the structural +content; counsel provides the legally-sufficient wording. + +### Disclosure 1 — Notice of collection (§15(b)(1)) + +Lakehouse will collect and store my **biometric identifier** (a +photograph of me from which facial geometry is implicit). The +photograph itself is the data we keep — we do NOT run automated +facial-classification (gender / race / age inference) against it +in v1. If at a later date we add automated classification, we will +re-collect consent under a superseding template before doing so. + +### Disclosure 2 — Specific purpose and length of term (§15(b)(2)) + +The biometric data will be used for: + +1. Identity verification at staffing job sites +2. Internal record-keeping so coordinators can recognize me across + placements + +The biometric data will be retained for a maximum of **18 months** +from my most recent interaction with the staffing platform, after +which it will be permanently destroyed per the +[Biometric Retention Schedule v1](biometric_retention_schedule_v1.md). + +I may withdraw this consent at any time by contacting the operator +(see §3 below). Withdrawal triggers permanent destruction of my +biometric data. + +### Disclosure 3 — Written release (§15(b)(3)) + +I provide a written release authorizing Lakehouse to collect, store, +and use my biometric identifier and biometric information for the +purposes stated above and for the term stated above. + +--- + +## 1. Plain-language summary (non-binding) + +⚖ COUNSEL — the section above is the binding legal disclosure. +The summary below is provided for candidate comprehension and is +NOT a substitute for the binding disclosure. Both should appear +together in the consent UI; counsel determines whether this summary +is appropriate to include or whether a different plain-language +section is preferred. + +> **What you're agreeing to:** if you upload a photo of yourself, +> we'll keep that photo so your staffing coordinator can recognize +> you when you arrive at job sites. We don't run automated guesses +> about your age, gender, or race against the photo. +> +> **How long we keep it:** at most 18 months after your last +> placement or interaction with us, then it's permanently destroyed. +> +> **What we DON'T do with it:** we don't sell it, we don't share it +> with anyone outside the staffing operation unless legally compelled, +> and we don't use it to decide what jobs to recommend to you. +> +> **How to take it back:** contact us (§3 below) at any time to +> withdraw your consent. We will permanently destroy your biometric +> data within 30 days of receiving your request. + +--- + +## 2. Withdrawal procedure + +I may withdraw biometric consent at any time. Withdrawal: + +- Is free of charge +- Does not affect my ability to remain on the staffing platform + (only my biometric data is removed) +- Triggers permanent destruction of all biometric data within + 30 days, per the destruction runbook +- Is recorded as an append-only audit row in my per-subject + audit log, providing me with tamper-evident proof of withdrawal + if I subsequently exercise my BIPA right of action + +⚖ COUNSEL — confirm 30 days is the right destruction SLA. Some +deployments use 7 or 14 days. The runbook (Gate 5) currently +references this template's number, so changing it here updates +both. + +--- + +## 3. Contact for withdrawal / questions + +⚖ COUNSEL — supply the candidate-facing contact channel for +biometric-consent withdrawal. Examples: a dedicated email +(`biometric-consent@`), a postal address, a +named operator. The contact must be functional from day one of +deployment. + +--- + +## 4. Consent acknowledgment + +By signing below (electronically or on paper), I acknowledge that: + +1. I have read and understood the disclosures in §1-3 above +2. I am providing this consent voluntarily and free of coercion +3. I have received a copy of this consent template (or have been + provided a means to retrieve a copy at any time) + +| Field | Value | +|---|---| +| Candidate name | _______________________________ | +| Date | __________ | +| Signature | _______________________________ | +| Consent template version | v1 (SHA-256: _generated at deployment time_) | + +--- + +## 5. Operational integration + +The structured fields the consent UI must capture and post to +identityd: + +```json +{ + "candidate_id": "", + "consent_version_hash": "", + "consent_given_at": "", + "consent_collection_method": "", + "consent_collection_evidence_path": "" +} +``` + +These fields write to `SubjectManifest.consent.biometric.status='given'` +and the corresponding `SubjectAuditRow` (see +`crates/catalogd/src/subject_audit.rs`). + +--- + +## 6. Versioning + +This consent template is version v1. Per Gate 1's versioning rules, +any change to the binding disclosure language requires a new version, +and existing subjects retain their original consent_version reference +unless they re-consent under the new version. + +⚖ COUNSEL — confirm whether existing consent under v1 carries forward +when the schedule is updated, or whether re-consent is required. +This affects the deployment workflow. + +--- + +## 7. Authority + +| Role | Name | Signature | Date | +|---|---|---|---| +| Operator | J | _______________ | _____ | +| Outside counsel | _____________ | _______________ | _____ | diff --git a/docs/policies/consent/biometric_retention_schedule_v1.md b/docs/policies/consent/biometric_retention_schedule_v1.md new file mode 100644 index 0000000..6a5cdd4 --- /dev/null +++ b/docs/policies/consent/biometric_retention_schedule_v1.md @@ -0,0 +1,155 @@ +# Biometric Data Retention Schedule — v1 + +**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 1 (BIPA §15(a)) +**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before public publication +**Version:** v1 (initial; supersession requires a new version + new hash) +**Updated 2026-05-05:** §1 + §2 revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). §5 destruction-trigger endpoint corrected to the shipped catalogd-local route. Pending J's product confirmation of Gate 3b. + +> This is a publicly-available retention schedule for biometric identifiers +> and biometric information collected by the Lakehouse staffing platform. +> It is required by 740 ILCS 14/15(a) (the Illinois Biometric Information +> Privacy Act) before any biometric collection from real candidates begins. + +--- + +## 1. What this schedule governs + +This schedule applies to: + +- **Biometric identifiers** as defined in 740 ILCS 14/10: candidate + photographs from which facial geometry is implicit. +- **Biometric information** as defined in 740 ILCS 14/10: any information + derived from a biometric identifier. **In v1 of this schedule, no + derived information is collected** — automated facial-classification + (gender, race, age inference) is deferred per + `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` Option C. If a future version + of this schedule introduces classification, that is a superseding + v2 schedule with re-consent under the matching v2 consent template. + +**Out of scope** (explicitly NOT biometric data under this schedule): + +- Synthetic faces from the pre-existing face pool (`data/headshots/`). + These are computer-generated portraits, not derived from any real + individual, and are not "biometric identifiers" under 740 ILCS 14/10. +- Candidate names, email addresses, phone numbers, work history, + certifications, or any other non-biometric personal information. + These are governed by the general PII retention policy referenced + in the SubjectManifest substrate (see + `docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md`). + +--- + +## 2. Categories collected + +| Category | Source | Storage location | +|---|---|---| +| Photograph (raw bytes) | Candidate upload via the consent-gated photo endpoint | Quarantined under `data/biometric/uploads//_.`; mode 0700 dir / 0600 file | +| Photograph integrity hash | SHA-256 of the original bytes | `SubjectManifest.biometric_collection.template_hash` (catalogd JSON manifest at `data/_catalog/subjects/.json`) | + +We do NOT collect raw biometric template vectors that could be used +to re-derive a face from the encoded form. We do NOT run automated +facial-classification (gender, race, age inference) in v1 — see +`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` for the deferral rationale. +The `BiometricCollection.classifications` field on the subject +manifest exists in the schema but is `None` for every subject. + +--- + +## 3. Purpose of collection + +Photographs and the classifications derived from them are used for: + +1. **Identity matching during staffing operations.** When a worker + arrives at a job site, the assigned coordinator may verify identity + by comparing the on-file photograph against the person present. +2. **Internal record-keeping.** Photographs become part of the worker + record so coordinators can recognize repeat workers across multiple + placements. + +Photographs and biometric classifications are NOT used for: + +- Demographic targeting in role recommendations (Title VII / IL Human + Rights Act compliance). +- Training of any machine-learning model. +- Sharing with third parties, except as required by court order or + with the candidate's separate written consent. +- Any purpose beyond those enumerated in §3.1-3.2 above. + +--- + +## 4. Retention period + +Per 740 ILCS 14/15(a), biometric identifiers and biometric information +must be permanently destroyed when the initial purpose for collection +has been satisfied OR within **three (3) years** of the individual's +last interaction with the private entity, whichever occurs first. + +**Operational ceiling:** Lakehouse retains biometric data for a +maximum of **eighteen (18) months** from the candidate's last placement +or last system interaction, whichever is later. This is more +restrictive than the BIPA statutory ceiling and provides a safety +margin against accidental over-retention. + +The 18-month clock is enforced by the daily retention sweep +(`crates/catalogd/src/bin/retention_sweep.rs`), which checks +`SubjectManifest.consent.biometric.retention_until` on every subject +and routes overdue subjects to the destruction queue (see Gate 5 +runbook). + +⚖ COUNSEL — confirm the 18-month operational ceiling is appropriate +for the deployment posture, or specify a different number. + +--- + +## 5. Destruction procedure + +Per 740 ILCS 14/15(a), Lakehouse follows the **BIPA Destruction +Runbook** (`docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`) when: + +- Retention period under §4 expires +- Candidate withdraws biometric consent under the consent template (Gate 2) +- Candidate exercises a right-to-be-forgotten request +- A catalogd-local `POST /biometric/subject/{id}/erase` is invoked + under legal-tier authentication (shipped `848a458`) + +Every destruction event is recorded as an append-only audit row in +the affected subject's per-subject HMAC-chained audit log (see +`crates/catalogd/src/subject_audit.rs`), providing tamper-evident +proof of compliant destruction. + +--- + +## 6. Versioning + +This schedule is version v1. Future revisions: + +- Require a new version number (v2, v3, ...). +- Are committed to the repository with a `git` history showing the + revision date. +- Are referenced by SHA-256 hash from `consent_versions` table rows + in identityd, so each subject's consent record points unambiguously + at the schedule version that was in force when consent was given. + +**v1 SHA-256:** _generated at deployment time by_ `scripts/staffing/hash_consent_v1.sh` _(to be added when this schedule is finalized by counsel)_ + +--- + +## 7. Public availability + +⚖ COUNSEL — specify the public URL where this schedule will be +published (typically the privacy policy page on the deployment site) +and the disclosure language that links candidates to it from the +intake UI. + +--- + +## 8. Authority + +This schedule is adopted under the authority of J (operator of record) +and reviewed by ⚖ COUNSEL. Effective date: **TBD pending counsel +sign-off**. + +| Role | Name | Signature | Date | +|---|---|---|---| +| Operator | J | _______________ | _____ | +| Outside counsel | _____________ | _______________ | _____ | diff --git a/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md b/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md new file mode 100644 index 0000000..2a1fe1d --- /dev/null +++ b/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md @@ -0,0 +1,244 @@ +# BIPA Biometric Data Destruction Runbook + +**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5 (BIPA §15(a)) +**Audience:** Operators (J + named operators with legal-tier credentials) +**Status:** Engineering scaffold — ⚖ COUNSEL must review for legal sufficiency before adoption + +> This runbook tells an operator HOW to destroy biometric data when +> a destruction trigger fires. It is a procedural document, not a +> design document. The cryptographic substrate that the destruction +> writes against (per-subject HMAC audit log + tombstone manifests) +> already ships in `crates/catalogd/`. + +--- + +## 1. When this runbook fires + +Destruction is mandatory when ANY of the following occurs: + +| Trigger | Source signal | SLA | +|---|---|---| +| **Retention expiry** | Daily `retention_sweep` flags `consent.biometric.retention_until < now` | 30 days from sweep flagging | +| **Consent withdrawal** | Candidate submits withdrawal per consent template §2 | 30 days from receipt | +| **Right-to-be-forgotten request** | Candidate submits RTBF request through documented contact channel | 30 days from receipt | +| **Court-ordered erasure** | Legal counsel directs erasure via a documented order | Per court order; default 30 days | + +⚖ COUNSEL — confirm 30 days is correct for all four. Some deployments +have stricter contractual or jurisdictional clocks (CCPA: 45 days but +sooner is better; GDPR Art. 17: "without undue delay"). + +--- + +## 2. Pre-destruction checks (5 minutes) + +Before initiating destruction, the operator MUST: + +1. **Verify the trigger.** Cross-reference one of the four sources + above. If the trigger is a candidate-initiated request, + confirm identity per the standard PII verification procedure + (knowledge factor + possession factor; see counsel for the + threshold). + +2. **Pull the current subject record.** Hit + `GET /audit/subject/{candidate_id}` with the legal-tier token. + The response includes: + - The current `SubjectManifest` (including `consent.biometric.status`) + - The full HMAC-chained audit log + - `chain_verified: true` (if false, STOP — chain integrity issue + must be investigated before destruction) + +3. **Check for legal hold.** ⚖ COUNSEL — if a legal hold can apply + to a subject's data (litigation, regulatory inquiry, subpoena), + document the procedure for checking that no hold is in force + before erasing. + +4. **Get the second-operator sign-off.** Per BIPA defensibility, + destruction is a two-operator action (operator-of-record + one + witness). The witness records their attestation in the + destruction-event audit row (§4 below). + +--- + +## 3. Destruction procedure + +### Step 1 — Erase via identityd + +Invoke the legal-tier erasure endpoint: + +```bash +curl -sf -X POST "http://localhost:3100/biometric/subject/${CANDIDATE_ID}/erase" \ + -H "Authorization: Bearer $(cat /etc/lakehouse/legal_audit.token)" \ + -H "Content-Type: application/json" \ + -d '{ + "scope": "biometric_only|full", + "trigger": "retention_expiry|consent_withdrawal|rtbf|court_order", + "trigger_evidence_path": "", + "operator_of_record": "", + "witness": "" + }' +``` + +The endpoint is **shipped** (commit `848a458`, 21 unit tests). It is +served from catalogd-local at `/biometric/subject/{id}/erase` (the +original v1 spec proposed `/v1/identity/subjects/{id}/erase` under a +separate identityd daemon — that daemon was collapsed into catalogd +per the architecture pivot). + +The endpoint exposes two scopes: + +- **`scope: "biometric_only"`** — clears `BiometricCollection` from + the SubjectManifest (drops `data_path`, `template_hash`, and + `classifications` together) + securely unlinks the quarantined + photo file. Subject manifest itself remains. Use for retention + expiry / consent withdrawal where only biometric data must go. +- **`scope: "full"`** — full subject erasure (manifest + biometric + files). Use for court-ordered erasure or full RTBF requests. + +In both scopes, the audit row is appended BEFORE photo unlink so +the chain has legal proof of intent even if the file delete fails +(transactional rollback on audit failure). + +### Step 2 — Append the destruction-event audit row + +The erasure endpoint AUTOMATICALLY writes one row to the subject's +per-subject audit log: + +```json +{ + "schema": "subject_audit.v1", + "ts": "", + "candidate_id": "", + "accessor": { + "kind": "biometric_erasure", + "daemon": "identityd", + "purpose": "biometric_erasure", + "trace_id": "" + }, + "fields_accessed": ["biometric_classifications", "biometric_data_path", "biometric_template_hash"], + "result": "erased", + "prev_chain_hash": "", + "row_hmac": "" +} +``` + +The HMAC chain extends through the erasure event, so the audit +log itself is preserved as anonymous-event proof of compliant +destruction even after the underlying biometric data is gone. + +### Step 3 — Verify destruction + +Run the verification script: + +```bash +./scripts/staffing/verify_biometric_erasure.sh "${CANDIDATE_ID}" +``` + +⚖ ENGINEERING — script TODO. Acceptance: +- Subject row biometric fields are NULL +- `data/biometric/uploads/${CANDIDATE_ID}/` directory is empty +- Most recent audit log row has `result: "erased"`, `accessor.kind: "biometric_erasure"` +- Chain still verifies (`chain_verified: true`) under the legal-tier endpoint + +If any check fails: STOP, do not mark the destruction complete, +escalate to engineering. + +### Step 4 — Notify the candidate (when applicable) + +For consent-withdrawal and RTBF triggers, the operator notifies +the candidate that destruction is complete. ⚖ COUNSEL — supply +the notification template (typically email; medium and language +are counsel-determined). + +--- + +## 4. Backup window disclosure + +Per `IDENTITY_SERVICE_DESIGN.md` v3-B12, biometric data may persist +in encrypted system backups for up to **30 days** after destruction +(rolling backup window). The candidate must be informed of this +when destruction is requested, and the destruction-event audit row +records the backup-window expiry date so the operator knows when +the residual is fully eliminated. + +⚖ COUNSEL — confirm whether the 30-day backup window is acceptable +under BIPA. Some interpretations require backups to be addressed +within a shorter window; some accept the operational reality of +backup retention. + +--- + +## 5. Reporting cadence + +Monthly, the operator-of-record produces a destruction-events +report: + +```bash +./scripts/staffing/biometric_destruction_report.sh \ + --month "$(date +%Y-%m)" \ + --output reports/biometric/destruction_$(date +%Y_%m).md +``` + +⚖ ENGINEERING — script TODO. The report aggregates: + +- Total destruction events in the month +- Breakdown by trigger (retention / withdrawal / RTBF / court) +- Median time-to-destruction from trigger to completion +- Any failures / escalations + +The monthly report is available to outside counsel on request. +It does NOT include candidate-identifying details — only the +counts, timings, and cryptographic attestations of the events. + +--- + +## 6. Audit trail attestation + +The per-subject HMAC chain is the cryptographic substrate that +makes destructions defensible after the fact. To produce an +attestation for a specific candidate's destruction: + +1. Hit `GET /audit/subject/{candidate_id}` with legal-tier token +2. Confirm `chain_verified: true` and most-recent row has + `accessor.kind: "biometric_erasure"` +3. Cross-runtime verify: the same audit log is byte-identical + under Rust + Go (per `scripts/cutover/parity/subject_audit_parity.sh`) +4. Counsel signs an attestation referencing the audit log's + chain root hash + +The chain root hash is itself a tamper-evident anchor. A motivated +insider would need the HMAC signing key (held in a separate location +from the audit logs themselves, per the spec) AND the original +log to forge a clean destruction record — and the cross-runtime +parity probe would catch a forgery that touched only one runtime's +view. + +--- + +## 7. Operator acknowledgment + +Operators with legal-tier credentials acknowledge they have read, +understood, and will follow this runbook before being granted access +to the legal_audit token. + +| Operator | Date acknowledged | Signature | +|---|---|---| +| J | _____ | _______________ | +| _____ | _____ | _______________ | + +⚖ COUNSEL — adopt this acknowledgment as the substrate for §3 of +Phase 1.6 (employee training acknowledgment), or specify a separate +training program. + +--- + +## 8. Change log + +- 2026-05-05 — Endpoint path reconciled with shipped state: + `/v1/identity/subjects/{id}/erase` (legacy proposal under a + separate identityd daemon) → `/biometric/subject/{id}/erase` + (catalogd-local, shipped `848a458`). Step 1 manual-fallback + block removed (the endpoint is no longer "TODO"). Two-scope + body shape (`biometric_only` / `full`) documented to match + the implementation. +- 2026-05-03 — Initial scaffold. ⚖ COUNSEL review required before + adoption. diff --git a/docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md b/docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md new file mode 100644 index 0000000..60dc37c --- /dev/null +++ b/docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md @@ -0,0 +1,308 @@ +# Legal-Tier Audit Key & Token Rotation Runbook + +**Spec companion:** `docs/PHASE_1_6_BIPA_GATES.md` §2 + `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` +**Audience:** Operators with root on the gateway host (J + named operators) +**Status:** Engineering-authored — ⚖ counsel review encouraged before formal adoption + +> This runbook covers rotation of the two crypto-credentials that gate +> the Phase 1.6 audit substrate: +> +> 1. **`LH_SUBJECT_AUDIT_KEY`** — the 32-byte HMAC-SHA256 signing key +> that chains every per-subject audit row. If this key changes, all +> pre-rotation chain rows tamper-detect under the new key. That is +> correct, expected, BIPA-defensible behavior — the chain integrity +> it provided pre-rotation remains intact in the archive of the old +> key, and post-rotation chains remain intact going forward. +> +> 2. **`LH_LEGAL_AUDIT_TOKEN`** — the 32+-character bearer token that +> authorizes calls to `/audit/subject/{id}` and +> `/biometric/subject/{id}/erase`. Rotation does NOT touch any audit +> history; only access to the legal-tier endpoints flips. +> +> Both live at `/etc/lakehouse/` (mode 0400, owned by root) and are +> loaded by the gateway via systemd `Environment=` directives in +> `/etc/systemd/system/lakehouse.service.d/audit_env.conf`. They are +> NOT loaded from `/tmp` — a 2026-05-05 reboot incident wiped a +> `/tmp`-resident key and caused `/audit` + `/biometric` to fail-closed +> (which is what they should do); the rotation fix moved them to the +> persistent path. + +--- + +## 1. When to rotate + +Rotate **immediately** when any of the following is true: + +| Trigger | Urgency | Notes | +|---|---|---| +| Suspected operator credential compromise | Within 1 hour | Token mismatch is fail-closed by default; immediate rotation closes the window. | +| Operator with legal-tier access leaves the team | Within 24 hours | Treat as compromise. | +| Key/token file's filesystem permissions were ever weakened (mode > 0400, group readable, etc.) | Within 24 hours | Filesystem audit may have leaked the bytes. | +| Token was ever transmitted over an untrusted channel (printed in CI log, sent over SMS, etc.) | Within 24 hours | Same reasoning. | +| Scheduled rotation (recommended) | Every 90 days | BIPA does not mandate a rotation cadence; counsel may set one. | + +Do **not** rotate when: + +- A subject's audit chain tamper-detects in isolation. That is normal + if the audit log was edited (which would itself be the BIPA finding, + not the key). Investigate the chain, not the key. +- Cross-runtime parity drift appears. That's an HMAC-input-shape bug + (Go vs Rust serialization), not a key issue. See + `STATE_OF_PLAY.md` "three runtime-divergence classes" entry. + +--- + +## 2. Pre-rotation checks (5 minutes) + +Before generating new credentials, capture a clean baseline so you can +prove the rotation cause and sequence afterward. + +### 2.1. Take the engineering snapshot + +```bash +# Confirm the canonical files exist with correct permissions. +ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token + +# Hash the existing key + token (NEVER the bytes themselves) so the +# old credential is identifiable in retrospect without storing it. +sha256sum /etc/lakehouse/subject_audit.key +sha256sum /etc/lakehouse/legal_audit.token + +# Confirm the gateway is currently using these files. +sudo systemctl cat lakehouse.service | grep -E "Environment.*AUDIT" + +# Verify the audit endpoint is healthy with the current credentials. +curl -sf http://localhost:3100/audit/health +``` + +If `/audit/health` is already 503, the rotation is **recovery**, not +preventive — note this in the rotation event record (§5). + +### 2.2. Capture a known-good chain root + +Pick one or two subjects with non-empty audit logs and record their +chain roots **under the current key**: + +```bash +TOKEN=$(cat /etc/lakehouse/legal_audit.token) +for cid in WORKER-2 WORKER-100; do + curl -sf -H "X-Lakehouse-Legal-Token: $TOKEN" \ + "http://localhost:3100/audit/subject/$cid" \ + | jq '{cid: .candidate_id, verified: .audit_log.chain_verified, root: .audit_log.chain_root, rows: .audit_log.chain_rows_total}' +done +``` + +Save the output. Post-rotation, those chains will tamper-detect under +the new key — that is **expected** and the saved snapshot is the proof +that the chain WAS intact under the old key, before rotation. + +--- + +## 3. Generation + rotation + +### 3.1. Generate the new key + +```bash +# 32 random bytes as hex = 64 chars. Either format works for HMAC-SHA256; +# we follow the existing convention (44-char base64-ish with no padding). +sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \ + /etc/lakehouse/subject_audit.key.new + +sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \ + /etc/lakehouse/legal_audit.token.new + +# Sanity: confirm 44-char content + correct mode. +sudo wc -c /etc/lakehouse/subject_audit.key.new /etc/lakehouse/legal_audit.token.new +sudo ls -la /etc/lakehouse/*.new +``` + +Both must be `mode 0400`, owned by root, exactly **44 chars** (the +audit endpoint refuses tokens shorter than 32 chars at load — see +`crates/catalogd/src/audit_endpoint.rs:73`). + +### 3.2. Atomic swap + +The gateway reads these files **once at boot** (per +`crates/catalogd/src/audit_endpoint.rs::AuditEndpointState::new` and +the equivalent for the writer). Atomic mv → restart is required. + +```bash +# Move the old credentials to a quarantine path with timestamp so the +# old hashes remain identifiable post-rotation. +TS=$(date -u +%Y%m%dT%H%M%SZ) +sudo mkdir -p /etc/lakehouse/_archived +sudo install -d -m 0700 -o root -g root /etc/lakehouse/_archived + +sudo mv /etc/lakehouse/subject_audit.key /etc/lakehouse/_archived/subject_audit.key.$TS +sudo mv /etc/lakehouse/legal_audit.token /etc/lakehouse/_archived/legal_audit.token.$TS + +sudo mv /etc/lakehouse/subject_audit.key.new /etc/lakehouse/subject_audit.key +sudo mv /etc/lakehouse/legal_audit.token.new /etc/lakehouse/legal_audit.token + +sudo ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token +``` + +### 3.3. Restart the gateway + +```bash +sudo systemctl restart lakehouse.service +sleep 2 +sudo systemctl status lakehouse.service --no-pager | head -10 +``` + +Wait for the gateway to bind port 3100 cleanly. If it doesn't, check +`journalctl -u lakehouse.service -n 50 --no-pager` for the failure +mode — the most common cause is the new file having wrong mode/owner. + +--- + +## 4. Post-rotation verification (5 minutes) + +### 4.1. Health probes + +```bash +# Audit endpoint must be 200, not 503. +curl -sf http://localhost:3100/audit/health +# Expect: "audit endpoint ready" + +# /v1/health must list the gateway's full provider set. +curl -sf http://localhost:3100/v1/health | jq '.providers, .worker_count' +``` + +### 4.2. Confirm the new token works + +```bash +NEW_TOKEN=$(cat /etc/lakehouse/legal_audit.token) +curl -sS -o /dev/null -w '%{http_code}\n' \ + -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \ + http://localhost:3100/audit/subject/WORKER-100 +# Expect: 200 +``` + +If 401, the file the gateway loaded does NOT match the file you wrote. +Check ownership / mode / for trailing whitespace differences with +`hexdump -C /etc/lakehouse/legal_audit.token | head`. + +### 4.3. Confirm the new chain works + +Append-only chains are key-tied. Any *new* audit row written +post-rotation is signed under the new key and verifies cleanly: + +```bash +# Issue a /v1/validate call against any worker — it spawns an audit row. +curl -sf -X POST http://localhost:3100/v1/validate \ + -H 'Content-Type: application/json' \ + -d '{"mode":"fill","candidate_id":"WORKER-100","worker_id":"WORKER-100","fields":["exists"]}' >/dev/null + +# Read the chain back. Last row must verify under the new key. +curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \ + http://localhost:3100/audit/subject/WORKER-100 \ +| jq '.audit_log | {verified: .chain_verified, rows: .chain_rows_total, last_kind: .rows[-1].accessor.kind}' +``` + +`chain_verified: true` confirms the new key is signing + verifying. + +### 4.4. Confirm pre-rotation chains tamper-detect (expected) + +```bash +curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \ + http://localhost:3100/audit/subject/WORKER-2 \ +| jq '.audit_log | {verified: .chain_verified, error: .chain_verification_error}' +``` + +For any subject whose chain was written under the old key, this +returns `chain_verified: false` with an HMAC-mismatch error. **This +is correct behavior**, not a bug. The old chain was correctly signed +under the old key and verified under it; the new key cannot retroactively +verify rows it didn't sign. The pre-rotation snapshot you captured in +§2.2 is the defensible proof that those rows WERE valid pre-rotation. + +If, instead, you see a chain that *should* verify post-rotation +returning `verified: false`, that's the rotation having gone wrong — +likely an old-key file that didn't get archived cleanly. Restore from +`/etc/lakehouse/_archived//`, then re-attempt. + +--- + +## 5. Record the rotation event + +Append a row to the rotation log: + +```bash +sudo tee -a /etc/lakehouse/_archived/rotation_log.jsonl <","reason":"","old_key_sha256":"","new_key_sha256":"$(sha256sum /etc/lakehouse/subject_audit.key | awk '{print $1}')","old_token_sha256":"","new_token_sha256":"$(sha256sum /etc/lakehouse/legal_audit.token | awk '{print $1}')","witness":""} +EOF + +sudo chmod 0600 /etc/lakehouse/_archived/rotation_log.jsonl +sudo chown root:root /etc/lakehouse/_archived/rotation_log.jsonl +``` + +This file is the operator-side record of when the key changed and why. +It does NOT contain the key itself — only hashes — so it is safe to +back up and share with counsel on request. + +--- + +## 6. Recovery from a lost key + +If the active `subject_audit.key` is destroyed (filesystem corruption, +accidental delete, /tmp wipe per the 2026-05-05 incident), the gateway +will fail-closed at startup: + +- `/audit/subject/{id}` → 503 ("audit endpoint disabled (legal token missing)" or equivalent for the signing key) +- `/biometric/subject/{id}/photo` → 503 (same fail-closed posture) + +This is correct behavior — a server that cannot HMAC-sign new audit +rows must not accept new biometric writes. + +**Recovery is rotation.** Generate a new key per §3.1, atomic-swap +per §3.2, restart per §3.3, verify per §4. Pre-loss chains tamper-detect +under the new key (the old key is gone — there is no way to verify +them). Treat the loss event as the BIPA-defensible boundary: pre-loss +chain verification was provided by the working key; post-loss new +chains are signed under the new key. + +If a counsel-grade attestation of the pre-loss chains is needed, the +`/etc/lakehouse/_archived/` folder contains the historical hashes; +combined with the cross-runtime parity probe (Go reader gives the +same byte-identical view as Rust), the chain history pre-loss is +preservable as long as the on-disk JSONL files were not also lost. + +--- + +## 7. ⚖ counsel notes + +These are areas where counsel may want to opine before this runbook +is formally adopted: + +1. **Rotation cadence.** BIPA itself does not require periodic rotation; + counsel may set a 90-day schedule to satisfy a separate compliance + posture (SOC2, internal policy). +2. **Custody of `/etc/lakehouse/_archived/`.** The archived hashes do + NOT contain the keys, but the archived raw key files DO. Counsel + may want a more aggressive destruction schedule for the raw archived + keys — say 1 year — to reduce a long-tail compromise surface. +3. **Notification obligations on rotation due to compromise.** §1 + triggers a rotation; §1 does not address whether candidates whose + biometric data was protected by the compromised key must be notified. + This is a counsel call. + +--- + +## 8. Operator acknowledgment + +| Operator | Date acknowledged | Signature | +|---|---|---| +| J | _____ | _______________ | +| _____ | _____ | _______________ | + +--- + +## 9. Change log + +- 2026-05-05 — Initial runbook authored after the /tmp wipe incident + on the same day (key was at `/tmp/subject_audit.key` and was deleted + on reboot, disabling `/audit` + `/biometric` until the key was + regenerated at `/etc/lakehouse/subject_audit.key`). Recovery of + that incident produced a working procedure; this runbook captures + it as the canonical playbook for any future rotation. diff --git a/docs/specs/GATE_3B_DEEPFACE_DESIGN.md b/docs/specs/GATE_3B_DEEPFACE_DESIGN.md new file mode 100644 index 0000000..a105afd --- /dev/null +++ b/docs/specs/GATE_3B_DEEPFACE_DESIGN.md @@ -0,0 +1,174 @@ +# Gate 3b — Deepface Classification Integration (Design) + +**Status:** **DECIDED 2026-05-05 — Option C (defer classifications)** · Original design draft 2026-05-03 morning · **Companion to:** [`PHASE_1_6_BIPA_GATES.md`](../PHASE_1_6_BIPA_GATES.md) Gate 3 · **Depends on:** Gate 3a (photo upload) which is shipped (`f1fa6e4`) + +> **Decision summary (2026-05-05):** J accepted Option C. `BiometricCollection.classifications` remains `Option = None` in v1. The consent template and retention schedule were revised the same day to remove all "automated facial-classification" language so the disclosed scope matches the implemented scope. If a real product requirement for classifications surfaces later, this doc's Option A (Python subprocess) or Option B (ONNX-in-Rust) is picked up under a v2 consent template + v2 retention schedule. + +> **What this is.** Three options for how `BiometricCollection.classifications` (currently `Option`, always `None`) gets populated by an automated facial-attribute classifier. Phase 1.6 Gate 3a ships the consent-gated upload + audit chain + transactional rollback; Gate 3b adds the classification step. The substrate is ready — what's missing is the design choice for HOW classification happens. +> +> **Why a design doc instead of building it.** The Python sidecar that historically hosted deepface was deliberately dropped from the hot path on 2026-05-02 (commit `ba928b1` in lakehouse: "drop Python sidecar from hot path; AiClient → direct Ollama"). Restoring an in-process Python pipeline would regress that decision. Three viable paths exist with different tradeoffs; this doc lays them out so the choice is deliberate, not accidental. + +--- + +## What deepface produces + +`deepface` (https://github.com/serengil/deepface) is a Python library wrapping multiple face-analysis backends (VGG-Face, FaceNet, OpenFace, etc.). Given an image path, it returns a JSON-shaped result with keys for `age`, `gender`, `dominant_race`, and `emotion` (plus per-class probabilities). For Phase 1.6's BIPA-compliance scope we need `age`, `gender`, `dominant_race` — the labels that go into `SubjectManifest.biometric_collection.classifications`. + +Per the runbook (Gate 5 destruction §2 step 3), classifications must be: +- Stored in the SubjectManifest, NOT in a separate analytics table +- Cleared during BIPA erasure (already wired — `biometric_collection = None` clears them along with `data_path`) +- NEVER served to LLMs in conversation context (only to authenticated staffer UI) + +--- + +## Option A — Python subprocess per upload (one-shot, no daemon) + +Spawn a `python3` subprocess from the Rust biometric endpoint after the photo is written, run a thin script that imports `deepface.DeepFace.analyze`, captures the JSON, exits. No long-running Python process, no IPC, no shared state. The hot path stays Rust + tokio; the subprocess is invoked exactly once per upload. + +**Implementation shape** (~50 LOC additional in `biometric_endpoint.rs`): + +```rust +// After photo file is written + manifest committed + audit row appended: +let cls_path = state.storage_root.join(&safe_id).join("_classifications.json"); +let _ = tokio::process::Command::new("python3") + .arg("/etc/lakehouse/deepface_classify.py") + .arg(&abs_path) // input photo path + .arg(&cls_path) // output JSON path + .kill_on_drop(true) // don't leak subprocess on cancellation + .status().await; +// Read cls_path back, populate biometric_collection.classifications. +// If subprocess failed/timed out: leave classifications=None, log warning. +// The upload itself succeeded; classification is best-effort. +``` + +The Python script (`/etc/lakehouse/deepface_classify.py`, ~30 LOC): + +```python +#!/usr/bin/env python3 +import sys, json +from deepface import DeepFace +result = DeepFace.analyze(sys.argv[1], actions=['age','gender','race'], silent=True) +with open(sys.argv[2], 'w') as f: + json.dump(result, f) +``` + +**Pros:** +- Smallest integration: ~80 LOC total +- Process isolation: a deepface crash can't take down the gateway +- Easy to disable: rename the script or remove it; classifications just stay `None` +- Matches Phase 1.6 spec literally (deepface IS the named classifier) +- Keeps the hot path Rust-only — subprocess is fired AFTER the response goes out (we can spawn it without awaiting and let it complete async) + +**Cons:** +- Per-upload cold start (~3-5 seconds for Python interpreter + deepface model load) +- Operator must keep `pip install deepface` working on the host (one more dependency to track) +- The "no Python in hot path" constraint is bent if we await the subprocess inline +- Reintroduces a Python-dependent code path that the 2026-05-02 sidecar drop was trying to eliminate + +**Cost estimate:** 0.5 day to build + 0.5 day to harden timeout/error paths. + +--- + +## Option B — ONNX model in Rust (no Python at all) + +Replace deepface entirely with a Rust-native ONNX runtime invocation. Pre-train (or download pre-trained) ONNX models for age + gender + race classification, ship them at `/etc/lakehouse/models/{age,gender,race}.onnx`, run them through `tract` or `ort` (ONNX Runtime Rust bindings). No Python anywhere. + +**Implementation shape** (~200 LOC + a model-build pipeline): + +```rust +use ort::{Environment, SessionBuilder, Value}; +let env = Environment::builder().build()?; +let session = SessionBuilder::new(&env)?.with_model_from_file("/etc/lakehouse/models/age.onnx")?; +let preprocessed = preprocess_image(&abs_path)?; // resize, normalize, etc. +let output = session.run(vec![Value::from_array(preprocessed)?])?; +let age = decode_age(&output[0])?; +// repeat for gender + race +``` + +**Pros:** +- Zero Python dependency — fully consistent with the 2026-05-02 sidecar drop +- Faster per-call: ~100ms inference (no interpreter cold start) +- Model files are immutable artifacts that can be hashed + signed for provenance +- Audit-friendly: deterministic output for a given input + model hash +- BIPA-defensible: model provenance is auditable; no third-party Python library risk + +**Cons:** +- Need pre-trained ONNX models for our three tasks (age regression, gender classification, race classification). Either find existing public models with appropriate licenses OR train ourselves (multi-week) +- Image preprocessing in Rust is non-trivial (face detection + alignment + crop + normalize). May need an additional model (e.g. RetinaFace ONNX) just for face detection +- ~200-400 LOC of new code in a domain we haven't worked in before +- Cross-runtime parity story: Go side would need its own ONNX integration if it ever wants the same surface, OR we define classifications as Rust-only (acceptable since the substrate is Rust-side anyway) + +**Cost estimate:** 3-5 days to build + 2 days to validate output quality + license review for any pre-trained models we use. Largest of the three options. + +--- + +## Option C — Defer classifications until a real demand exists + +Keep `BiometricCollection.classifications` as `Option` permanently `None` until a concrete consumer needs it. The substrate is ready; we don't add the classifier writer. + +**What still works without classifications:** +- BIPA consent gate (Gate 3a) ✓ +- Photo storage with template_hash for integrity (Gate 3a) ✓ +- Audit chain proving collection happened (Gate 3a) ✓ +- Erasure flow (Gate 5) ✓ +- Retention sweep on biometric clock (Step 7) ✓ + +**What's missing without classifications:** +- The "demographic data derived from photos" disclosure in the consent template (Gate 2 §1 Disclosure 1) becomes vestigial — we collect photos but don't actually derive any demographic information from them +- The retention schedule (Gate 1) §2 lists "Facial geometry classifications" as a category we collect; without classifications, that category is empty in practice + +**Pros:** +- Zero engineering today +- Forces the question "do we actually need classifications?" to be answered by a real product requirement, not by spec inertia +- BIPA-safer in the most absolute sense: we collect strictly the photo, nothing derived from it +- If the real product requirement is "let staffers see the photo at job-site verification," classifications add no value (the staffer's eyes do the verification) + +**Cons:** +- Requires updating Gates 1 + 2 to remove the "we derive classifications" language (counsel coordination needed) +- Loses the structured demographic data that audit-of-discrimination workflows might want (per `IDENTITY_SERVICE_DESIGN.md` v3-A4 disparate-impact analysis) +- If we later need classifications, we land on Option A or B then anyway — this is a deferral, not a permanent choice + +**Cost estimate:** 0.25 day to update Gate 1 + 2 doc language + ⚖ counsel review of the change. + +--- + +## Recommendation + +**Option C (defer) is the right answer right now**, conditional on confirming the real product requirement. + +Reasoning: +1. Phase 1.6's load-bearing claim is consent + audit + retention + erasure. Gate 3a + Gate 5 + retention sweep + audit chain provide ALL of those WITHOUT classifications. +2. The riskiest BIPA position is collecting demographic-derived data without a documented business purpose. If staffers identify candidates by looking at the photo (the actual stated use case), classifications are decorative. +3. Option A reintroduces the Python dependency the 2026-05-02 wave deliberately removed. Option B is multi-day work for unclear benefit. +4. The `Option` substrate is forward-compatible: if a real demand surfaces (say, disparate-impact analytics J actually wants), we can ship Option A or B then in 1-3 days respectively, and the audit chain naturally captures the introduction event. + +**Next action if Option C is chosen:** +- Update `docs/policies/consent/biometric_consent_template_v1.md` Disclosure 1 to remove the "facial geometry classifications" claim — adjust to "facial photograph for staff identification" +- Update `docs/policies/consent/biometric_retention_schedule_v1.md` §2 to remove the classifications row +- Mark Gate 3b in `PHASE_1_6_BIPA_GATES.md` as "deferred — classifications not collected in v1, see GATE_3B_DEEPFACE_DESIGN.md" +- ⚖ counsel review of the doc changes (consent text changes need their attention regardless) + +**Next action if Option A is chosen:** ~1 day to ship; document the Python dependency carve-out in STATE_OF_PLAY's "WHAT NOT TO RELITIGATE" so future sessions don't try to remove it again. + +**Next action if Option B is chosen:** schedule a multi-day session, source/license pre-trained ONNX models for the three tasks, build face-detection preprocessing in Rust. + +⚖ J — pick A / B / C. The substrate accommodates any choice; the cost is the design-doc → counsel-coordination → engineering loop, which differs by an order of magnitude across the options. + +**[2026-05-05] J's decision: Option C.** Reasoning recorded in change log below. Consent + retention doc revisions for Option C shipped same day; counsel review of revised text is the remaining work. + +--- + +## Open questions for J + +1. **What's the actual product requirement for classifications?** If the answer is "I don't know yet," that's a strong vote for Option C. +2. **If classifications: who is the consumer?** Staffer UI? Disparate-impact dashboard? An LLM context (NO — Phase 1.6 §3 forbids this)? +3. **If Option A: are we OK with `pip install deepface` as a host requirement?** Or does the operator-of-record need a containerized Python sidecar (which puts us close to where we were before 2026-05-02)? +4. **If Option B: who is the model provenance signer?** ⚖ counsel will want a chain of custody from the model author to the ONNX file on disk. + +--- + +## What this doc is NOT + +- Not a final decision. The decision is J's after reading the options. +- Not a substitute for testing. Whichever option ships, the parity probe `subject_audit_parity.sh` must continue to pass post-change (the audit chain is the BIPA-load-bearing surface; classifications are additive metadata). +- Not the only Phase 1.6 work remaining. Counsel-side review of Gates 1/2/5 + the §2 attestation signature are the calendar bottleneck. Gate 3b is the engineering bottleneck. diff --git a/docs/specs/PATHWAY_MEMORY_SPEC.md b/docs/specs/PATHWAY_MEMORY_SPEC.md new file mode 100644 index 0000000..68d2f24 --- /dev/null +++ b/docs/specs/PATHWAY_MEMORY_SPEC.md @@ -0,0 +1,307 @@ +# Pathway Memory — Specification v1 + +**Status:** Draft v1 — 2026-05-03 · **Layer:** Decision metadata · **Implementation:** `crates/vectord/src/pathway_memory.rs` + +> **What this is.** Pathway Memory is a third metadata layer that exists alongside (not on top of, not under) the data-metadata layer (catalogd's manifests/views/tombstones) and the operational-memory layer (playbook_memory). It tracks **decision patterns**: "this code path, in this task class, produced this outcome." It is the AI-substrate analog of what Iceberg manifests are to data tables — pointers + provenance for things you might want to retrieve again or learn from. +> +> **What it is NOT.** It is not a vector database. It is not a knowledge graph. It is not an audit log (audit is per-subject; pathways are per-code-pattern). It is not a cache. It is a small, opinionated metadata layer with a defined wire format, defined access patterns, and defined lifecycle rules. +> +> **Why a spec exists.** No standard tells us what this layer is. Each implementation that drifts from this spec creates the wiring confusion currently visible in the system (data/_kb/ JSONLs that don't get consumed, version drift across `workers_500k_v1..v10`, unclear who reads `reducer_summary` vs `final_verdict`). The spec is the standardization contract that lets a single implementation become the canonical one. + +--- + +## 1. Conceptual model + +A **pathway** is the identity-equivalence-class of code paths that exhibit similar decision behavior. Two distinct files may share a pathway if they belong to the same crate AND are reviewed under the same task class with the same signal class. + +A **trace** is a single observed instance of a pathway: one review pass, with its inputs, intermediate signals, ladder attempts, audit outcome, and final verdict. + +A pathway is **identified** by a stable hash of three keys: task class, file prefix, signal class. The hash is `pathway_id`. + +A trace is **identified** within a pathway by a generated UUID: `trace_uid`. The pair `(pathway_id, trace_uid)` is globally unique. + +Pathways are **append-only by trace**: every new observation creates a new trace under the existing pathway_id. Traces themselves are **mutable in version**: `revise()` produces a new version that supersedes the previous (Mem0-style chain). + +Pathways are **retirable** by id: when replay history shows the pattern no longer holds, the pathway is marked retired and excluded from hot-swap. New pathways may emerge with fresh ids if the underlying code/task characteristics shift. + +--- + +## 2. Wire format — `PathwayTrace` + +Authoritative struct: `crates/vectord/src/pathway_memory.rs`. The fields below are the SPEC; the Rust struct is one implementation. All fields use JSON-serializable types when persisted. + +### 2.1 — Identity fields (immutable per trace) + +| Field | Type | Constraint | +|---|---|---| +| `pathway_id` | string | SHA256 hex of `task_class | file_prefix | signal_class` — see §3 | +| `trace_uid` | string | UUID. Empty on legacy traces; populated on insert by SHOULD-implementations from v2+ | +| `version` | u32 | Mem0 version chain. Default 1 on insert. Bumped on `revise()`. | +| `parent_trace_uid` | string \| null | trace_uid of the trace this one supersedes. null on root version. | +| `superseded_at` | string \| null | ISO-8601 UTC. Set when this trace becomes a non-head version. | +| `superseded_by_trace_uid` | string \| null | trace_uid of the new head when this version is superseded. | + +### 2.2 — Identity-source fields (the inputs that fed pathway_id) + +| Field | Type | Notes | +|---|---|---| +| `task_class` | string | Caller-defined class string (e.g. `scrum_review`, `staffing.fill`, `pr_audit`) | +| `file_path` | string | Full path of the file under review. Reduced to file_prefix for hashing — see §3.2. | +| `signal_class` | string \| null | Caller-defined behavior label (e.g. `CONVERGING`, `LOOPING`, `STUCK_RETRY`) | + +### 2.3 — Observation fields (what happened) + +| Field | Type | Description | +|---|---|---| +| `created_at` | RFC 3339 timestamp | When this trace was inserted. | +| `ladder_attempts` | array of `LadderAttempt` | Each model+rung attempt: `{rung, model, latency_ms, accepted, reject_reason?}`. Order is dispatch order. | +| `kb_chunks` | array of `KbChunkRef` | Knowledge-base chunks the reviewer was given as context: `{source_doc, chunk_id, cosine_score, rank}`. | +| `observer_signals` | array of `ObserverSignal` | Behavior labels emitted by observer during this pathway's run: `{class, priors, prior_iter_outcomes}`. | +| `bridge_hits` | array of `BridgeHit` | External-doc lookups (context7-style): `{library, version, result_summary}`. | +| `sub_pipeline_calls` | array of `SubPipelineCall` | Calls to other sub-pipelines (extract, validate, etc.) made during this trace. | +| `audit_consensus` | `AuditConsensus` \| null | Cross-lineage audit outcome if any: `{pass, models, disagreements}`. | +| `reducer_summary` | string | Free-text summary of what the reducer concluded. **PII risk surface — see §6.3.** | +| `final_verdict` | string | Free-text verdict label (`accepted`, `rejected`, `needs_review`, etc.). **PII risk surface — see §6.3.** | + +### 2.4 — Index fields (the matrix retrieval shape) + +| Field | Type | Description | +|---|---|---| +| `pathway_vec` | array of f32, length 32 | Bag-of-tokens hash embedding. Algorithm in §4. Fixed dimension per spec version. | +| `replay_count` | u32 | Number of times this pathway has been replayed via hot-swap. Initial insert is NOT a replay. | +| `replays_succeeded` | u32 | Replays where the post-replay outcome matched the trace's `final_verdict`. | +| `retired` | bool | Marked true when probation gate fires (§5.4). Excluded from hot-swap forever. | + +### 2.5 — Optional semantic-correctness fields (ADR-021) + +| Field | Type | Description | +|---|---|---| +| `semantic_flags` | array of `SemanticFlag` | One of: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`. Caller-emitted, not auto-derived. | +| `type_hints_used` | array of `TypeHint` | Schema/type context fed to the reviewer for this trace: `{source, symbol, type_repr}`. | +| `bug_fingerprints` | array of `BugFingerprint` | Structural pattern hashes the reviewer caught: `{flag, pattern_key, example, occurrences}`. | + +All §2.5 fields are additive — implementations MUST default them to empty when deserializing legacy traces. + +--- + +## 3. Identity algorithms + +### 3.1 — `pathway_id` + +``` +pathway_id = sha256_hex(task_class || "|" || file_prefix(file_path) || "|" || signal_class_or_empty) +``` + +- Concatenation uses literal byte `|` (0x7C) as separator. +- `signal_class_or_empty` is the empty string when signal_class is null. +- SHA-256 output is 64 hex chars, lowercase. + +### 3.2 — `file_prefix` + +``` +file_prefix(path) = first two path segments joined by '/' +``` + +- Path separator is forward slash. Implementations on Windows MUST normalize to forward slashes before hashing. +- Files with fewer than two segments use the full path verbatim. +- Examples: + - `crates/queryd/src/service.rs` → `crates/queryd` + - `crates/gateway` → `crates/gateway` + - `README.md` → `README.md` + +### 3.3 — Why this fingerprint shape + +The fingerprint groups files within a crate (same `file_prefix`) under the same pathway when reviewed for the same task class with the same signal class. This is intentional: it lets the matrix index recognize "the same kind of bug appears in this crate" without needing per-file traces. Per-file granularity is preserved in the `file_path` field for retrieval but is not part of the identity. + +--- + +## 4. `pathway_vec` algorithm + +Fixed dimension: **32 buckets**. Implementation: deterministic bag-of-tokens hash, normalized. + +``` +def build_pathway_vec(trace): + buckets = [0.0] * 32 + tokens = collect_metadata_tokens(trace) # see token list below + for tok in tokens: + h = sha256(tok.encode()) + bucket = int.from_bytes(h[:4], 'big') % 32 + buckets[bucket] += 1.0 + norm = sqrt(sum(b*b for b in buckets)) + return [b/norm if norm > 0 else 0.0 for b in buckets] +``` + +**Tokens MUST include**: +- `task_class:` +- `file_prefix:` +- `signal_class:` +- For each `ladder_attempts[i].model`: `model:` +- For each `kb_chunks[i].source_doc`: `kb_doc:` +- For each `observer_signals[i].class`: `signal:` +- For each `bug_fingerprints[i].flag`: `flag:` + +**Tokens MUST NOT include**: +- `reducer_summary` text (free-form, would dominate the bucket distribution) +- `final_verdict` text +- `created_at` (would make every trace's vector unique) +- candidate_id, name, email, phone, or any subject identifier (PII risk) + +**Why 32 dimensions:** consensus-validated round 3 ensemble selected "small metadata tokens, not full JSON." 32 is sufficient to distinguish pathway combinations without requiring an external embedding model. An external embedding model would work too but adds a dependency, failure mode, and drift risk the consensus flagged. **Spec version v1 fixes dim=32. v2 may revisit.** + +--- + +## 5. Lifecycle + +### 5.1 — Insert + +A trace is inserted by the consumer (typically the scrum pipeline or auditor) calling the implementation's `insert()` API. + +- If no pathway with `pathway_id` exists, a new pathway bucket is created. +- If a pathway exists, the new trace is appended to that bucket's traces list. +- `trace_uid` is generated (UUID v7 RECOMMENDED for time-orderability). +- `version` is set to 1. +- `created_at` is set to the insertion timestamp. +- `pathway_vec` is computed and stored. +- `replay_count` and `replays_succeeded` start at 0. + +### 5.2 — Revise + +A trace is revised by calling `revise(trace_uid, new_fields)`. + +- A new trace with a fresh `trace_uid` and `version = previous.version + 1` is inserted. +- The new trace's `parent_trace_uid` is set to the previous trace's `trace_uid`. +- The previous trace's `superseded_at` is set to the revision timestamp and `superseded_by_trace_uid` is set to the new trace's `trace_uid`. +- The previous trace remains in the bucket (history is preserved). Retrieval defaults to head versions only — see §7. + +### 5.3 — Replay + +A pathway is **eligible for replay** when `retired == false` AND the bucket has at least one head-version trace. + +The hot-swap consumer queries by `pathway_id` (or by similarity over `pathway_vec`), retrieves the head trace, applies its `final_verdict` shape to a new but similar input, and then reports back via `record_replay(trace_uid, succeeded: bool)`. + +- `replay_count` is incremented on every replay (success or failure). +- `replays_succeeded` is incremented only on success. + +### 5.4 — Probation gate (retirement) + +Probation triggers when `replay_count >= 3` AND `success_rate < 0.80`. + +When triggered, the pathway is marked `retired = true`. It is excluded from hot-swap forever. If the underlying code/task characteristics genuinely change such that a retired pathway would work again, a NEW pathway with a FRESH id will be created (because the inputs to `pathway_id` would have changed, e.g., new signal_class). Retirement is per-id and irreversible. + +`success_rate` = `replays_succeeded / replay_count` (0.0 if `replay_count == 0`). + +--- + +## 6. Access patterns (read + write shapes) + +### 6.1 — Write paths (who inserts traces) + +Spec-conformant writers: +- Scrum pipeline (`tests/real-world/scrum_master_pipeline.ts` and downstream consumers) on review acceptance +- Auditor pipeline (`auditor/`) on verdict assembly +- Validator (`crates/validator/`) on iterate session completion + +Each writer is responsible for setting the identity fields correctly. The implementation MUST reject writes where `task_class` is empty. + +### 6.2 — Read paths (who consumes traces) + +Spec-conformant readers: +- **Hot-swap retrieval** — `query_for_hotswap(task_class, file_path, signal_class, k=5)` returns the top-k head-version traces from matching pathways, ordered by `(success_rate desc, replay_count desc)`. +- **Similarity retrieval** — `query_by_vec(pathway_vec, k=10)` returns the top-k head-version traces by cosine similarity over `pathway_vec`. +- **Audit / forensic** — `list_versions(trace_uid_root)` returns the full version chain. **Must be authorized** — version chains can include reducer_summary text that may contain PII. + +The implementation MUST NOT expose `list_versions` on an unauthenticated route. + +### 6.3 — PII surface (load-bearing for audit-trail compliance) + +The fields `reducer_summary` and `final_verdict` are free-form strings written from review output. They are **highly likely** to contain PII when the reviewed task class involved real candidates (per `AUDIT_PHASE_1_DISCOVERY.md` §1F + §10/C1). Until subject-redaction-on-write lands, implementations: +- MUST treat trace bodies as a suspected PII sink for any task class touching real subjects +- SHOULD redact known PII patterns (candidate_id, email, phone) before persisting +- SHOULD provide a per-trace `subject_ids[]` top-level field when known, so audit-response queries can filter by subject + +Until the redaction layer ships, do not advertise pathway_memory as PII-clean. + +--- + +## 7. Storage representation + +The current implementation persists state to a single JSON file at `data/_pathway_memory/state.json`: + +``` +{ + "pathways": { + "": [ + { "trace_uid": "...", "version": 1, "task_class": "...", ... }, + { "trace_uid": "...", "version": 2, "parent_trace_uid": "...", ... } + ] + }, + "last_updated_at": "2026-04-29T05:43:00Z" +} +``` + +The flat-JSON representation is implementation-specific. A spec-conformant implementation could use SQLite, an append-only JSONL, or a Lance dataset. The spec dictates the wire shape per-trace and the access patterns; the storage backend is the implementation's choice. + +**Retrieval defaults**: queries return head versions only by default (a head version is one with `superseded_at == null`). Consumers explicitly opt in to historical versions via a `include_history: true` flag. + +--- + +## 8. Spec boundary — what's stable vs implementation-specific + +### Stable (locked in v1; changing requires v2 spec) +- The `pathway_id` algorithm (SHA256 of `task_class | file_prefix | signal_class`) +- The `file_prefix` algorithm (first two path segments) +- `pathway_vec` dimension (32) and token list +- Field names and types in §2 (additive fields are OK; renames or removals require v2) +- Lifecycle rules in §5 +- Access pattern names in §6 (`query_for_hotswap`, `query_by_vec`, `list_versions`) + +### Implementation-specific (free to change without spec bump) +- Storage backend (JSON file, SQLite, Lance, etc.) +- HTTP/gRPC/library API surface +- Concurrency model +- In-memory caching strategy +- The exact text of `reducer_summary` (it's content, not schema) +- Token-bucket hash function (SHA-256 specified in §4 but any deterministic 32-bit hash is acceptable as long as it's deterministic and well-distributed) + +### Reserved for v2 (named so future implementations don't paint into a corner) +- Cross-pathway similarity (`query_pathways_similar_to_pathway`) +- External embedding model for `pathway_vec` (replaces the bag-of-tokens hash) +- Subject-aware indexing (so audit-response can find all pathways involving subject X) +- Differential privacy on `pathway_vec` aggregations + +--- + +## 9. Open questions + +These need decisions before v1 is final-final: + +1. **Dimension lock-in**: 32 is round-3 consensus, but is it actually optimal? Worth a benchmark before final v1. +2. **Subject indexing in v1 vs v2**: deferring to v2 means audit queries today have to grep `reducer_summary` text. Is that acceptable for the staffing-client launch window? +3. **Cross-runtime parity**: the Go side has its own `internal/pathways/` implementation. As of 2026-05-03 the wire shape matches but no parity probe exists. Add `pathway_parity.sh` to the cross-runtime probe suite. +4. **Migration from current state.json**: existing 91 traces lack `trace_uid`. Spec says `trace_uid` is empty on legacy traces — should v1 require backfill or accept legacy-trace queries indefinitely? + +--- + +## 10. Why this spec exists + +Without it: +- Every new consumer of pathway_memory invents their own read/write conventions +- The Go-side reimplementation drifts from the Rust-side wire format +- Future LLM-assistant work proposes new fields without considering the existing schema +- Forensic queries grep free-text fields because no canonical retrieval exists +- The PII concern stays "TODO" because there's no stable surface to gate + +With it: +- One implementation can be canonical (Rust, currently); others can be conformance-tested +- The wire shape is documented; serialization tests can pin it +- New fields go through an additive-only review (don't break legacy) +- The KbContext consumer in the gateway has a stable API to read against +- The audit-trail PRD has a defined PII surface to protect + +This is the standardization the system was missing. + +--- + +## Change log + +- 2026-05-03 — v1 initial draft. Codifies what's already in `crates/vectord/src/pathway_memory.rs`. The spec is descriptive of the existing implementation, not prescriptive of new behavior. diff --git a/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md b/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md new file mode 100644 index 0000000..a3d57c1 --- /dev/null +++ b/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md @@ -0,0 +1,266 @@ +# Subject Manifests on Catalogd — Specification v1 + +**Status:** Draft v1 — 2026-05-03 · **Layer:** Catalogd extension (NOT a separate daemon) · **Implementation:** to be added to `crates/catalogd/src/` + +> **What this is.** A small extension to `catalogd` adding a fourth manifest type — `subject` — alongside the existing dataset / view / tombstone / profile types. A subject manifest answers: "for person X, which datasets contain their PII, which views project it safely, what consent + retention applies, and what's the access log." +> +> **What it is NOT.** It is not a separate identity daemon. It is not a Postgres-backed identity service. It is not a HashiCorp-Vault-using KEK rotation system. It is not the `IDENTITY_SERVICE_DESIGN.md` v3 design (that doc is over-scoped for local-only — see its deprecation header). It is the smallest spec that gives you a defensible "show me everything we know about person X" capability for EEOC discovery / BIPA compliance, building on primitives catalogd ALREADY ships. +> +> **Why it can be small.** Catalogd already has dataset manifests with per-column `is_pii` flags. It already has views with `column_redactions` (working example: `candidates_safe.json`). It already has tombstones for deletes. It already has profiles for per-agent scoping. The audit-trail need is "thread these primitives together by subject identifier" — not "build a new system." That's a ~300-500 LOC extension, not a 17-20 day phase plan. + +--- + +## 1. Conceptual model + +A **subject** is a real person whose PII flows through the system. Identified by a stable token (current implementation: `candidate_id` in `workers_500k.parquet`). + +A **subject manifest** is a JSON record under `data/_catalog/subjects/.json` that points at: +- which datasets contain rows for this subject (via foreign-key reference) +- which views safely project this subject's data (via existing view manifest names) +- what consent + retention metadata applies to this subject +- what access log file holds this subject's audit trail + +Subject manifests are written when a subject enters the system AND updated when a subject's consent changes, vertical reclassifies, or retention period expires. + +The **audit log** is a per-subject append-only JSONL file at `data/_catalog/subjects/.audit.jsonl`. Every PII access for that subject writes one row. The file is signed periodically (HMAC chain) for tamper-evidence. + +--- + +## 2. Wire format — `SubjectManifest` + +JSON document at `data/_catalog/subjects/.json`: + +```json +{ + "schema": "subject_manifest.v1", + "candidate_id": "CAND-000001", + "created_at": "2026-05-15T12:00:00Z", + "updated_at": "2026-05-15T12:00:00Z", + "status": "active", + "vertical": "general", + "consent": { + "general_pii": { + "status": "given", + "version": "v1-2026-05-15", + "given_at": "2026-05-15T12:00:00Z" + }, + "biometric": { + "status": "never_collected", + "retention_until": null + } + }, + "retention": { + "general_pii_until": "2030-05-15T12:00:00Z", + "policy": "4_year_default" + }, + "datasets": [ + { "name": "workers_500k", "key_column": "candidate_id", "key_value": "CAND-000001" }, + { "name": "candidates", "key_column": "candidate_id", "key_value": "CAND-000001" }, + { "name": "placements", "key_column": "candidate_id", "key_value": "CAND-000001" }, + { "name": "timesheets", "key_column": "candidate_id", "key_value": "CAND-000001" } + ], + "safe_views": ["workers_safe", "candidates_safe"], + "audit_log_path": "data/_catalog/subjects/CAND-000001.audit.jsonl", + "audit_log_chain_root": "sha256:..." +} +``` + +### 2.1 — Field semantics + +| Field | Required | Notes | +|---|---|---| +| `schema` | yes | Always `"subject_manifest.v1"`. Validates parser shape. | +| `candidate_id` | yes | Subject identifier. Stable token. Same value as appears in dataset key columns. | +| `status` | yes | `pending_consent` \| `active` \| `withdrawn` \| `retention_expired` \| `erased`. | +| `vertical` | yes | `unknown` \| `general` \| `healthcare` \| `finance` \| `other`. **Default `unknown`**, fail-closed routing treats unknown as healthcare-equivalent. | +| `consent.general_pii.status` | yes | `pending_backfill_review` \| `pending_first_contact` \| `given` \| `withdrawn` \| `expired`. | +| `consent.biometric.status` | yes | `never_collected` \| `pending` \| `given` \| `withdrawn` \| `expired`. | +| `retention.general_pii_until` | yes | ISO-8601. Drives daily expiration sweep. | +| `datasets[].name` | yes | References an existing catalogd dataset manifest by name. | +| `datasets[].key_column` | yes | The column in that dataset that contains the subject's identifier. | +| `datasets[].key_value` | yes | The specific value (the subject's id within that dataset's namespace). | +| `safe_views` | yes | Names of existing catalogd view manifests that safely project this subject's data (for non-legal-tier readers). | +| `audit_log_path` | yes | Relative path to the audit JSONL. | +| `audit_log_chain_root` | yes | SHA-256 of the most recent HMAC-chained checkpoint of the audit log. Updated by the audit-log writer on every write. | + +--- + +## 3. Audit log format + +Per-subject append-only JSONL at `data/_catalog/subjects/.audit.jsonl`. One row per PII access: + +```json +{ + "schema": "subject_audit.v1", + "ts": "2026-05-15T13:30:00Z", + "candidate_id": "CAND-000001", + "accessor": { + "kind": "gateway_lookup", + "daemon": "gateway", + "purpose": "fill_validation", + "trace_id": "X-Lakehouse-Trace-Id-..." + }, + "fields_accessed": ["name"], + "result": "success", + "prev_chain_hash": "sha256:...", + "row_hmac": "hmac-sha256:..." +} +``` + +### 3.1 — HMAC chain + +Each row's `row_hmac` is `HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac)`. The signing key is loaded once at startup from `/etc/lakehouse/subject_audit.key` (mode 0400). The chain root in the subject manifest references the latest row's `row_hmac`. + +A tamper-evident verification is one pass: +``` +verify_chain(subject_id): + manifest = read_subject_manifest(subject_id) + rows = read_audit_log(subject_id) + prev = "GENESIS" + for row in rows: + expected = hmac_sha256(key, prev || canonicalize(row - row_hmac_field)) + assert row.prev_chain_hash == prev + assert row.row_hmac == expected + prev = row.row_hmac + assert manifest.audit_log_chain_root == prev +``` + +This is local (no S3 Object Lock, no Vault) but tamper-evident: any modification to a past row breaks the chain at that point and all subsequent rows. The signing key being on disk is a real risk surface — operators MUST set the file mode 0400 owner-only and back it up to a separate location from the audit logs themselves (so that a single backup doesn't carry both the ciphertext and the verification material). + +### 3.2 — When the audit log is written + +Every code path that resolves PII for a subject MUST write an audit row before returning. Specifically: +- The gateway's tool registry SQL templates (`crates/gateway/src/tools/registry.rs`) — when `search_candidates` / `get_candidate` queries return rows, write one audit row per returned candidate_id +- The validator's WorkerLookup (`crates/validator/src/staffing/parquet_lookup.rs`) — when a `lookup(candidate_id)` succeeds, write one audit row +- The audit-response endpoint (when implemented) — when `/audit/subject/{id}` is called, write one row of `kind=audit_response` +- Any new code path that touches PII + +Write failures MUST NOT silently swallow. They MUST be logged at error level (per the existing observability fabric). Write failures MUST NOT block the read — accept the audit gap and flag it for post-hoc review (better to leak a row than block legitimate operations). + +--- + +## 4. The `/audit/subject/{candidate_id}` response + +The audit response builds from the subject manifest + audit log + dataset projections: + +```json +{ + "schema": "subject_audit_response.v1", + "candidate_id": "CAND-000001", + "generated_at": "2026-05-15T15:00:00Z", + "generated_by": "catalogd@hostname", + "manifest": { /* the SubjectManifest */ }, + "datasets": { + "workers_500k": { + "row_present": true, + "safe_view_projection": { /* candidates_safe row for this subject */ } + } + }, + "audit_log_window": { + "from": "2026-01-01T00:00:00Z", + "to": "2026-05-15T15:00:00Z", + "rows": [ /* matching audit rows */ ] + }, + "chain_verification": { + "verified": true, + "rows_checked": 42, + "chain_root": "sha256:..." + }, + "completeness_attestation": "all dataset rows + audit log entries within the window per retention policy v1 are included", + "signature": "ed25519:..." +} +``` + +The endpoint is auth-gated via a separate legal-tier credential (see §6). The response body is signed with an Ed25519 key separate from the HMAC chain key. + +--- + +## 5. Implementation plan (this is the SMALL plan) + +This is the spec; the implementation is a small extension to catalogd. Estimated effort: + +| Step | Effort | What | +|---|---|---| +| **1** | 0.5d | Add `SubjectManifest` struct + JSON load/save in `crates/catalogd/src/subjects.rs`. Mirror the existing `views.rs` pattern. | +| **2** | 0.5d | Add `SubjectAuditWriter` with HMAC chain in same file. Key loaded from sealed file at startup. | +| **3** | 0.5d | Backfill subject manifests from `workers_500k.parquet` rows. ETL: one manifest per row, default `vertical=unknown`, `consent.general_pii.status=pending_backfill_review`. | +| **4** | 0.5d | Wire the gateway tool registry to write audit rows. One audit row per candidate_id returned by search_candidates / get_candidate. | +| **5** | 0.5d | Wire the validator WorkerLookup to write audit rows. | +| **6** | 1d | `/audit/subject/{id}` HTTP endpoint in `crates/catalogd/src/service.rs`. Legal-tier auth. | +| **7** | 0.5d | Daily retention sweep: subjects whose `retention.general_pii_until` < now AND `status != erased` get marked for review (don't auto-delete; legal needs to approve). | +| **8** | 0.5d | Cross-runtime parity: Go side reads the same subject manifests + audit logs. Same shapes, same HMAC algorithm. | +| **Total** | **~4 days** | Compared to 17-20 days for the IDENTITY_SERVICE_DESIGN approach. | + +Each step is one commit, one revert path. No new daemons. No cloud infrastructure. No Vault. No S3 Object Lock. No dual-control JWT split-secret ceremony. + +--- + +## 6. Auth model + +Local-first, simple, defensible: + +- **Service-tier reads** (gateway tool registry resolving candidate names for fill scenarios): authenticated via the existing gateway internal credential. Audit row written. +- **Legal-tier reads** (`/audit/subject/{id}`): requires a separate credential held in `/etc/lakehouse/legal_audit.token` (mode 0400, owner-only). Operators may load this only when fulfilling a legal request. The token is rotated per a documented runbook (operator + 1 witness; no cryptographic dual-control ceremony required for this scale). +- **Backups**: subject manifests + audit logs are backed up daily. The HMAC signing key is backed up to a SEPARATE storage location (different physical/network boundary) so a single backup compromise doesn't enable forgery. + +This is not as strong as Vault Transit + dual-control JWT + S3 Object Lock external anchoring. It IS strong enough to pass a normal small-business compliance review and to be defensible in IL/IN small-claims-discovery contexts. If the staffing client's contract requires SOC2 Type II or formal HSM, that's a separate phase — but it's strictly additive on top of this v1 spec. + +--- + +## 7. What this spec gives you (load-bearing) + +1. **Defensible response to discrimination discovery** (worked example: John Martinez at Warehouse B). The endpoint produces a complete + signed + chain-verified record of every PII access affecting him. +2. **BIPA-compliant biometric tracking** when real photos arrive. The `consent.biometric` field + retention timeline are first-class, not bolted on later. +3. **Per-subject right-to-be-forgotten** via cryptographic erasure of the subject manifest's audit-key entry + tombstoning of the candidate's dataset rows. (The ability to verify "row is gone, trail is preserved as anonymous audit-event of the erasure" is what GDPR Art. 17 + CCPA expect.) +4. **HIPAA vertical routing** via the `vertical` field. Healthcare-vertical subjects (and `unknown` defaults) route to local-only models per PRD line 70 — no PHI to cloud egress. +5. **Cross-runtime parity** via the simple JSON+HMAC format that Go can read identically. + +--- + +## 8. What this spec does NOT solve (and where it punts) + +- **Per-row encryption of dataset PII**: subject manifests + audit logs are local; the underlying `workers_500k.parquet` is not encrypted. If staffing-client contract requires at-rest encryption, that's a separate concern handled at the storage tier (filesystem encryption, S3 SSE). +- **Right to explanation (GDPR Art. 22 / EU AI Act)**: this spec captures decisions that touched a subject; it does not require the model to explain WHY each decision was made in human-readable form. That's a separate Phase capturing model reasoning. +- **Adverse-impact statistics**: the comparator-pool snapshot per fill (per `AUDIT_PHASE_1_DISCOVERY` §10/C3) needs its own writer in the fill pipeline. This spec gives you the per-subject record; it doesn't cross-aggregate selection rates by protected class. +- **External tamper-evidence**: the HMAC chain is local. A motivated insider with access to both the audit log AND the signing key could rewrite history. For the staffing-client scale this is acceptable; for higher-stakes deployments a separate timestamping service or external transparency log would be additive. + +--- + +## 9. Why this is the right shape for J's deployment + +- It builds on what catalogd ALREADY ships (manifests, views, tombstones — the Iceberg-shape layer). +- It runs locally — no cloud infrastructure to license, monitor, audit, or pay for. +- Its primitives are JSON files an operator can read with `cat` and `jq`. Tamper-evidence works without trusting opaque crypto APIs. +- Its implementation is days, not weeks. The timeline matches the staffing-client launch window without forcing them to wait on a SaaS-tier identity service that doesn't fit their data residency posture. +- It is COMPATIBLE with the IDENTITY_SERVICE_DESIGN v3 path if the staffing client later requires SOC2 Type II — the v1 subject manifests can be migrated into a separate identity daemon when the scale demands it. But that's deferred until demand exists, not built speculatively. + +This is the LOCAL-FIRST audit trail. It exists because the SaaS-tier version doesn't fit the deployment model J actually has. + +--- + +## 10. Spec boundary + +**Stable in v1 (changing requires v2):** +- File layout: `data/_catalog/subjects/.json` + `data/_catalog/subjects/.audit.jsonl` +- JSON schemas in §2 + §3 (additive fields OK; renames/removals require v2) +- HMAC algorithm: HMAC-SHA256 with key from sealed file +- Chain semantics: `prev_chain_hash` references previous row's `row_hmac` +- Vertical default: `unknown` with fail-closed routing +- Consent state machine in §2.1 +- Audit-row write requirements in §3.2 + +**Implementation-specific (free to change):** +- Storage backend (file system v1; SQLite or Postgres acceptable as long as JSON shapes round-trip) +- HTTP endpoint exact shape (body schema is spec; status codes / headers are implementation) +- Backfill ETL details + +**Reserved for v2:** +- Per-row encryption (when staffing-client contract requires it) +- External tamper-evidence anchor (when SOC2 Type II in scope) +- Cross-tenant subject isolation (when multi-tenant in scope) + +--- + +## Change log + +- 2026-05-03 — v1 initial draft. Builds on catalogd primitives that already exist. Smaller than IDENTITY_SERVICE_DESIGN v3 by ~4× because it doesn't propose a new daemon. diff --git a/lakehouse.toml b/lakehouse.toml index 1c1582f..6d806fb 100644 --- a/lakehouse.toml +++ b/lakehouse.toml @@ -84,7 +84,7 @@ service_name = "lakehouse" # let the agent continuously propose + trial HNSW configs and auto-promote # winners. Defaults are conservative so it stays out of the way of live # search traffic on shared Ollama. -enabled = true +enabled = false cycle_interval_secs = 120 # periodic wake if no triggers cooldown_between_trials_secs = 10 # min gap between trials min_recall = 0.9 # never promote below this diff --git a/mcp-server/UX_FIX_PLAN.md b/mcp-server/UX_FIX_PLAN.md new file mode 100644 index 0000000..591dd4d --- /dev/null +++ b/mcp-server/UX_FIX_PLAN.md @@ -0,0 +1,94 @@ +# Staffing Co-Pilot UX Fix Plan (devop.live/lakehouse) + +**Target file:** `/home/profit/lakehouse/mcp-server/search.html` (239KB, ~4100 lines) — THIS is the +main dashboard served at `/` (route at `index.ts:1258`; name "search.html" is historical). +Sibling pages: `console.html` (Walkthrough), `proof.html` (Architecture), `spec.html`, `onboard.html`, +`profiler.html`, `contractor.html`, `workspaces.html`, `alerts.html`. + +**Serving:** Bun `mcp-server/index.ts` on `:3700`, nginx `/lakehouse/` → `127.0.0.1:3700`. +Restart after edits: HTML is read per-request (`Bun.file`), so **no restart needed for .html edits** — +just reload the browser. (Restart only if editing index.ts: it's pid 1254, `bun run mcp-server/index.ts`.) + +**Constraints:** neo-brutalist, compact, distinct colors, NO fake timers. Surgical edits only. +Dashboard fires real SQL + /intelligence/* — don't break data wiring. Repo is FROZEN Rust legacy; +J explicitly authorized work on this surface. + +**Base path quirk:** nav uses `var P = location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':''`. +Relative hrefs (`href="console"`) resolve fine under `/lakehouse/`. + +## User's 5 complaints → fixes (line numbers in search.html) + +### 1+2. New tabs for reference pages (HIGH) +- **Top nav lines 594-599**: add `target="_blank" rel="noopener"` to Profiler, Architecture(proof), + Spec, Onboard, Alerts, Workspaces. LEAVE 592-593 (Dashboard `href="."`, Walkthrough `href="console"`) same-tab. +- **Footer line 781**: `Architecture` → add `target="_blank" rel="noopener"`. Keep Console same-tab. +- **Dynamic footer lines 1024-1026**: proof link needs `target=_blank rel=noopener` + `.ft` class styling (currently unstyled). + +### 3. Collapsible "preview" not discoverable (HIGH) +Primary = **contract card expandable panel** (markup ~2620-2820, CSS grid-template-rows ~160-220). +Chevron rotates 180° but hint text "click for full curve…" only shows collapsed; vanishes on expand. +Fix: chevron always visible+larger(16px); persistent "Expand details"/"Collapse" label in BOTH states; +cursor:pointer on card header; hover border-left/bg shift; optional localStorage per-card. +Secondary collapsibles: ⓪ legacy-bridge narrative (~820-900) → default COLLAPSED for returning users +(localStorage 'seen_intro'); ② Staffer's Console preview (~1400-1500) → make collapsible, "Open full console →". + +### 4. Clickable elements don't look clickable (HIGH) +Add global CSS: `[onclick]{cursor:pointer}` + reusable `.clickable` class with :hover (bg lift #161b22→#1c2333 ++ border accent), :active (translateY(1px)), :focus-visible (outline). +Apply to: insight cards (~1180-1280), alert rows (~1700-1780), worker result rows (~990-1030 — add hover '›' chevron +for drilldown discoverability per feedback_test_drilldowns), collapsible section headers (~820-830, ~1400-1410), +action chips/buttons (~740-760, standardize .btn/.chip). + +### 5. Layout clutter / flat hierarchy (HIGH) +8 sections at equal weight: ⓪ Not-a-CRM narrative, ① Live Market, Forecast, ② Staffer Console, +Market Intelligence, ③ Worker Search, ④ System Activity, ⑤ Substrate Signals. +3-tier fix: PRIMARY always-visible (Live Market, Worker Search, Staffer Console); +SECONDARY collapsible-open (Forecast, Market Intelligence); TERTIARY collapse-by-default or relocate +(⓪ narrative → collapsible banner; ⑤ Substrate Signals → move to proof.html/Architecture; ④ System Activity demote). +Also: Live Market vs Market Intelligence may overlap — differentiate headers or merge w/ sub-tabs. +Long card walls → add "show top 5 / show all" truncation. + +### Cross-page nav consistency (MEDIUM) +Each of 8 pages has hardcoded nav with differing link sets/order/labels and unreliable .active. +Canonical set + order, each page sets .active on its own entry, brand/logo → `href="."`. +Long-term: extract nav to shared JS snippet served by index.ts to stop drift. + +## ✅ SHIPPED 2026-05-30 (live-verified via curl against :3700; HTML served per-request → no restart) +- **#1+#2 New tabs** — search.html: 6 nav reference links (Profiler/Architecture/Spec/Onboard/Alerts/Workspaces) + + footer Architecture + dynamic-footer Architecture now `target="_blank" rel="noopener"` with ↗ glyph. + Dashboard + Walkthrough stay same-tab (primary flow). VERIFIED: 7 static new-tab links served; nav listing clean. +- **#4 Clickability** — global affordance CSS inserted after `.ft a:hover{...}` (NOT `.ft a{margin}` — that anchor + was a hallucination): `[onclick]{cursor:pointer}`, `[role=button]`, `.iworker` cursor+hover-lift+active, + `details>summary` cursor+hover, global `:focus-visible`. VERIFIED served (affordance_css=1, iworker_cursor=1, summary_hover=1). +- **#3 partial** — every `
` collapsible now has visible hover affordance (cursor+bg). Contract-card toggle NOT yet (below). +- **Nav consistency** — added Profiler link to onboard/workspaces/alerts (verified profiler=1 each); + fixed contractor.html back-link `href="/"`→`href="."` (real bug: `/` = LLM-Team UI on :5000, ejected users out of lakehouse; verified root-bug=0, dot-fixed=1); + fixed profiler.html empty `href=""`→relative `.`/`console` (was JS-only/fragile; verified empty-href=0). + +> CORRECTION LOG: first edit pass (earlier this session) FAILED silently — old_strings were guessed +> (`.ft a{margin:0 4px}`, a fake footer string, `ui-copilot/dashboard.html` path). Re-done with grep/Read-verified +> exact strings. The real dashboard file is `search.html` (route index.ts:1258), NOT `dashboard.html` (orphan) or `ui/`. + +## (status note) — rounds 2 & 3 are recorded in the SHIPPED sections below; this block is superseded. +- #3 contract-card collapse → DONE round 2 (persistent `.card-top-toggle`). +- #5 declutter (⑤ relocate, ④ demote, ③ disclosure, Market Intelligence clarify, dead CSS) → DONE round 3. +- Still open: nav shared-component refactor (deferred, needs restart) + optional contract truncation (skipped on purpose). See "DELIBERATELY NOT DONE". + +## Recommended execution order +1. New-tab attrs (lines 594-599, 781, 1024-1026) — trivial, high impact. +2. Global `.clickable` CSS helper + apply to onclick elements. +3. Collapsible affordance (chevron+label persistent, cursor, hover) on contract cards + section headers. +4. Default-collapse ⓪ narrative for returning users (localStorage). +5. Hierarchy/declutter (tiering, relocate ⑤ to proof, truncation). +6. Nav consistency pass across sibling files. + +## VERIFY before claiming done (J's rules: test drilldowns, no greenwashing) +- `curl -s localhost:3700/ | grep target=_blank` confirms new-tab attrs present. +- Load https://devop.live/lakehouse/ in browser: click a contract card (expand+collapse visible), + hover a worker row (cursor+chevron), click reference nav (opens new tab), reload (intro stays collapsed). +- Don't restart the service for HTML-only edits. + +## Audit provenance +5-dimension parallel Explore audit, run wf_c897f421-db1, full JSON at +`/tmp/claude-0/-home-profit/c6ca0218-678c-4a30-833c-1063e40e7f69/tasks/wm9x4y52d.output`. +NOTE: line numbers are audit estimates — `grep` to confirm exact lines before each Edit. diff --git a/mcp-server/alerts.html b/mcp-server/alerts.html index 5378f28..2de1c57 100644 --- a/mcp-server/alerts.html +++ b/mcp-server/alerts.html @@ -74,6 +74,7 @@ input:focus{border-color:#388bfd}
- -
-
- ⑤ Substrate Signals — architecture health - Live probes of the index, memory, and pathway layers -
-

- These tiles measure the architecture itself, not the staffing workload. - Instant-search latency, index shape, playbook-memory depth, pathway-matrix - compounding — four probes that answer "is the substrate healthy right now?" -

-
-
Probing substrate…
-
-
+ - + @@ -792,7 +824,7 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun ②Jobs ③Search ④Learn - ⑤Stack + ✦Arch