From 5368aca4d47298f3275caf7fd9df0d9cb1c74141 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 00:44:57 -0500 Subject: [PATCH 01/65] docs: sync ADR-019 + PRD + DECISIONS with 2026-05-02 substrate changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-019: closed the "re-bench when 10M corpus exists" follow-up. Added "Follow-up: 10M re-bench (2026-05-02)" section with the post-fix numbers (search ~20ms warm / ~46ms cold, doc-fetch ~5ms post-btree). Documented the lance-bench-bypassing-IndexMeta bug + 2-layer fix + gauntlet (7 unit + 12 sanitize + 10 smoke probes). Reframes the strategic question as "Lance vs Parquet+HNSW-with-spilling" since HNSW doesn't fit RAM at 10M. DECISIONS: added ADR-022 — drop Python sidecar from Rust hot path. Captures the rationale (236× embed perf gap was pure overhead), co-shipped LRU cache, dev-only Python that survives, cross-runtime parity verification, and the operator runbook signal (ps -ef ABSENT post-deploy). PRD: updated AI Boundary table line + aibridge crate description to reflect direct Ollama path (was: Python FastAPI sidecar → Ollama). Both lines reference ADR-022 for the full rationale. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/ADR-019-vector-storage.md | 25 ++++++++++++++++++++++++- docs/DECISIONS.md | 5 +++++ docs/PRD.md | 4 ++-- 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/docs/ADR-019-vector-storage.md b/docs/ADR-019-vector-storage.md index 0ed2fe2..95151c1 100644 --- a/docs/ADR-019-vector-storage.md +++ b/docs/ADR-019-vector-storage.md @@ -70,10 +70,33 @@ But the written rule missed something. It assumed Lance's value would show up as ### What we keep watching (but don't act on yet) -- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against. +- ~~**Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.~~ **Done 2026-05-02** — see "Follow-up: 10M re-bench" below. - **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system. - **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot. +--- + +## Follow-up: 10M re-bench (2026-05-02) + +The 10M re-bench above ran. Numbers from `data/lance/scale_test_10m` (33 GB, 10M × 768d, IVF_PQ live, post-doc_id-btree-build). Full report: `reports/lance_10m_rebench_2026-05-02.md`. + +| Op | 100K (this ADR) | 10M (re-bench) | Notes | +|---|---:|---:|---| +| Search (cold) | 2229μs | ~46ms median | 21× slower at 100× scale → reasonable for IVF_PQ | +| Search (warm) | (not measured) | ~20ms p50 | Stable across 5 trials | +| Doc fetch by id | 311μs | ~5ms | Structural ADR-019 win confirmed once `doc_id` btree is built | +| Index method | lance_ivf_pq | lance_ivf_pq | response tag confirms | + +**The HNSW comparison at 10M doesn't exist** — at 10M × 768d × 4 bytes = ~30 GB just for vectors, doubled for the graph. HNSW doesn't fit on a single 128 GB box at this scale. So the original "Lance pulls ahead at 10M" framing is true by elimination: Lance is the only contender that operationally exists at 10M. The strategic question is reframed as "Lance vs Parquet+HNSW-with-spilling," deferred until we have a workload where the Parquet path is the bottleneck (currently tracked in `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` decisions tracker). + +**Bug surfaced and fixed during re-bench:** Initial doc-fetch was ~100ms (full table scan). Root cause: the `doc_id` scalar btree was never built — `lance-bench` writes datasets by bypassing `IndexMeta`, and the activation-time auto-build only runs for IndexMeta-registered indexes. Fixed at two layers (commits `5d30b3d` + `044650a`): +- `lance_migrate` HTTP handler auto-builds the btree inline (~1.2s on 10M, +269MB on disk) +- `lance-bench` binary builds the btree post-IVF for parity with the gateway path + +**Gauntlet added 2026-05-02:** the lance crates had zero tests + no smoke when audited. Now have 7 unit tests in `crates/vectord-lance` + 12 sanitize tests in `crates/vectord` + 10-probe `scripts/lance_smoke.sh` + sanitized error boundary across all 5 routes. See commits `7bb66f0`, `ac7c996`, `e9d17f7` (sanitizer iterations driven by cross-lineage scrum). + +**Status:** ADR-019's hybrid architecture (Parquet+HNSW primary, Lance secondary) is now empirically validated up to 10M. The "watch and re-bench" item is closed. + ## Why this isn't moving the goalposts The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile. diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 3cafa35..c382719 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -104,3 +104,8 @@ **Date:** 2026-04-24 **Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002. **Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon. + +## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly +**Date:** 2026-05-02 +**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call. +**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running. diff --git a/docs/PRD.md b/docs/PRD.md index 0a10db4..28d0bea 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -77,7 +77,7 @@ A modular Rust service mesh over S3-compatible object storage, with a local AI l | Data Format | Parquet + Arrow | Yes | | RPC (internal) | tonic (gRPC) | Yes | | AI Runtime | Ollama (local models) | Yes | -| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes | +| AI Boundary | Direct Ollama HTTP API (gateway → Ollama, no sidecar). Updated 2026-05-02 per ADR-022 — was "Python FastAPI sidecar → Ollama HTTP API". Sidecar's lab_ui/pipeline_lab Python remain as dev-only tools (not on hot path). | Yes | | Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** | No new frameworks without documented ADR. @@ -97,7 +97,7 @@ No new frameworks without documented ADR. | **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog | | **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) | | **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal | -| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar | +| **aibridge** | AI client — direct Ollama HTTP (per ADR-022, 2026-05-02; was Rust↔Python sidecar boundary). Owns LRU embed cache. | | **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs | | **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** | | **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) | -- 2.54.0 From c170ebc86eb19f28572b15621ccbe9c454b881da Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 00:54:46 -0500 Subject: [PATCH 02/65] =?UTF-8?q?docs:=20AUDIT=5FTRAIL=5FPRD=20=E2=80=94?= =?UTF-8?q?=20production-readiness=20gate=20for=20staffing=20client?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit J flagged that smoke + parity tests prove the surface compiles, NOT that an audit response can be produced for a specific person — and the staffing client won't sign without defensible discrimination-claim response capability. New docs/AUDIT_TRAIL_PRD.md captures: - worked example: John Martinez at Warehouse B requests audit - subject audit response output format (per-decision row schema) - surface map: where decisions happen today, where the gaps are - PII handling rules (tokenization, protected-attribute exclusion, inferred-attribute risk) - identity service design intent (separate daemon, audited reads) - retention + right-to-be-forgotten policy intent - 9-phase implementation sequence with explicit per-phase exit criteria - cross-runtime requirement (both Rust + Go must satisfy) - 7 open questions blocking phase 2+ that need J's call STATE_OF_PLAY + PRD updated with explicit "production-ready blocker" section pointing at the new doc. The "substrate is shipped" framing gets a caveat: substrate ≠ production-ready until audit phase 9 exits. No code changes. This is the planning artifact J asked for before we start building. Co-Authored-By: Claude Opus 4.7 (1M context) --- STATE_OF_PLAY.md | 14 ++- docs/AUDIT_TRAIL_PRD.md | 255 ++++++++++++++++++++++++++++++++++++++++ docs/PRD.md | 12 ++ 3 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 docs/AUDIT_TRAIL_PRD.md diff --git a/STATE_OF_PLAY.md b/STATE_OF_PLAY.md index 31d1150..14e3310 100644 --- a/STATE_OF_PLAY.md +++ b/STATE_OF_PLAY.md @@ -146,12 +146,24 @@ End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift --- +## ⚠ PRODUCTION-READY BLOCKER (2026-05-03) + +**Audit-trail capability is the gate to client signature.** Smoke + parity tests prove the surface compiles; they do NOT prove an audit response can be produced for a specific person. Staffing client won't sign without defensible discrimination-claim response capability. + +**Authoritative document:** `docs/AUDIT_TRAIL_PRD.md` — drafted 2026-05-03. Defines worked example (John Martinez at Warehouse B), the per-decision audit row schema, the surface map of where decisions happen today, current-state-vs-target gap table, and 9-phase implementation sequence. + +**Phase 1 (discovery walk) requires NO J approval — it's read-only.** Phases 2+ have explicit open questions in §10 of the PRD that need J's call before they can start. + +Until phase 9 exit criterion is met, **do not claim "production-ready" on customer-facing surfaces.** Internal substrate (Lance, sidecar drop, parity probes) is solid; subject-of-record audit story is not. + +--- + ## OPEN — but not blocking the demo | Item | What | When to act | |---|---|---| | `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. | -| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. | +| Open PRs #6, #7, #10 | All closed 2026-05-02 — superseded / empty / stale. PR #12 merged 2026-05-03 (`a5d9070`). PR #13 merged 2026-05-03 (`feb638e`). | Done. | | `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. | | `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. | | `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. | diff --git a/docs/AUDIT_TRAIL_PRD.md b/docs/AUDIT_TRAIL_PRD.md new file mode 100644 index 0000000..fddaba7 --- /dev/null +++ b/docs/AUDIT_TRAIL_PRD.md @@ -0,0 +1,255 @@ +# PRD: Production-Ready Audit Trail + +**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 + +> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation. + +--- + +## 1. The legal use case (worked example) + +**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2. + +**What we must produce.** A response that proves either: +- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR +- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI." + +**What we must NOT produce.** +- Other subjects' data (response leaks if even one other candidate's name appears) +- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped) +- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence + +**The defensibility chain.** The audit shows: +1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into +2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank +3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned +4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections) +5. **Outcome decisions** — final fills, rejections, hand-offs + +For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale. + +--- + +## 2. The subject audit response — output format + +``` +GET /audit/subject/{candidate_id}?from=D1&to=D2 +→ JSON or signed PDF (legal preference TBD) +``` + +**Header section:** +- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash +- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls. + +**Per-decision row schema** (shape, not exhaustive): +```json +{ + "ts": "ISO-8601 UTC", + "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal", + "daemon": "gateway | validatord | observerd | matrixd | ingestd", + "model": "kimi-k2.6 | deepseek-v3.2 | ...", + "provider": "ollama_cloud | opencode | openrouter", + "input_features": { /* what the model SAW — sanitized per §4 */ }, + "output": { /* what the model decided */ }, + "rationale": "model's natural-language explanation, or rule-based justification", + "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree", + "session_id": "iterate session that produced this row" +} +``` + +**Footer section:** +- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy" +- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited) + +--- + +## 3. Surface map — where decisions happen + +| Decision happens at | Currently logged where | Audit-completeness gap | +|---|---|---| +| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk | +| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding | +| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation | +| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial | +| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified | +| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today | +| Iterate retry escalations | Session JSONL `attempts[]` array | OK | +| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk | +| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject | + +**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly. + +--- + +## 4. PII handling rules + +**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces. + +**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it. + +**Protected-attribute exclusion at decision time:** the model NEVER receives: +- Race, ethnicity, national origin +- Sex, gender, marital status, pregnancy +- Age, date of birth (allowed: years of experience) +- Religion +- Disability, genetic information +- Veteran status (unless legally relevant for the role) +- Sexual orientation, gender identity + +If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input. + +**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls. + +**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows. + +--- + +## 5. Identity service — candidate_id ↔ PII mapping + +**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change. + +**Target state:** +- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping +- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id` +- Identity reads require a separate auth credential held by legal-authorized operators +- Every identity read is itself audited (log who accessed PII for which candidate when) +- Identity service runs as its own daemon, port-isolated from the gateway +- Cross-runtime: same identity service backs both Rust and Go + +**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary. + +--- + +## 6. Retention policy + +**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA. + +**Target state (proposed):** +- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s) +- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract. +- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed. +- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable. + +**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off. + +--- + +## 7. Current state vs target state + +| Capability | Today | Production-ready target | Gap | +|---|---|---|---| +| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service | +| Identity service | none | separate daemon, audited reads | Real — build new | +| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new | +| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument | +| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk | +| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial | +| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major | +| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk | +| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build | +| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build | +| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set | + +--- + +## 8. Implementation phases (proposed sequence) + +Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds. + +### Phase 1 — Discovery walk (read-only, ~3-4 hours) +Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references. + +**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells. + +### Phase 2 — Identity service design (design doc, ~2 hours) +Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.** + +**Exit:** J approves the design. + +### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours) +Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong. + +**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued. + +### Phase 4 — Subject tagging across substrates +Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror. + +**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end). + +### Phase 5 — Identity service build +Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly. + +**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject. + +### Phase 6 — Protected-attribute boundary enforcement +Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing. + +**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping. + +### Phase 7 — Retention + right-to-be-forgotten +Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint. + +**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows. + +### Phase 8 — Legal-format export + signing +Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow. + +**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies. + +### Phase 9 — End-to-end discrimination defense rehearsal +Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them. + +**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness. + +--- + +## 9. Cross-runtime requirement + +**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5. + +The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version). + +--- + +## 10. Open questions blocking phase 1 + +These are the things I need J to decide before phase 1 can start, OR I need to investigate-and-propose: + +1. **Identity service: separate daemon vs in-process?** Recommend separate. Confirm. +2. **Retention period N years?** Recommend 4. Need staffing client's legal call. +3. **Photo / surname / zip-code policy?** These are inferred-attribute risks. Need policy decision. +4. **JSON or signed PDF for legal export?** Different downstream costs. +5. **Right-to-be-forgotten under append-only logs:** cryptographic erasure (proposed) or hard delete (breaks integrity)? Confirm crypto-erasure approach. +6. **Audit endpoint auth model:** legal-only credential, or shared with admin? Recommend legal-only with separate token rotation. +7. **The "indexed before search" concern:** matrix indexer + pathway memory currently fingerprint by code, not subject. Do we (a) make them subject-aware (more audit completeness, more PII surface area), (b) keep them code-only and assert in audit response that "no subject-specific compounding state was used," or (c) something else? + +Items 1-6 can be resolved by J's call. Item 7 needs design discussion — the safest answer for legal defense is (b), but it loses the "pathway learns about THIS candidate" signal that may be load-bearing for the staffing UX. + +--- + +## 11. What this PRD is NOT + +- Not a contract with the staffing client. That document needs lawyers and signs after this is built. +- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC — passing actual certification is its own project. +- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made. +- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires. + +--- + +## 12. Appendix — terms + +- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`. +- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.). +- **Audit row** — one record in the audit response per decision, with the schema in §2. +- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes. +- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list. +- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation). +- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth. +- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it. +- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs. + +--- + +## Change log + +- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker. diff --git a/docs/PRD.md b/docs/PRD.md index 28d0bea..2de7c54 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -161,6 +161,18 @@ The feedback signal is **statistical + semantic**, not neural. No model training --- +## Production-readiness blocker (2026-05-03) + +Substrate is mostly shipped through Phase 45. Throughput, parity, and observability are all measured. + +**The production-readiness gate is the audit trail.** Staffing client requires defensible response to discrimination claims (worked example: John Martinez at Warehouse B requests every AI-system decision affecting him). PII tokenization, subject-of-record query, retention SLA, and right-to-be-forgotten under append-only logs are NOT YET BUILT. + +Authoritative document: **`docs/AUDIT_TRAIL_PRD.md`**. 9-phase implementation plan, current-state-vs-target gap table, open questions blocking phase 1. + +Until the audit-trail PRD's phase 9 exit criterion is met, "production-ready" claims are aspirational. Internal substrate (Lance, observability parity, cross-runtime fabric) is solid; subject-audit story is not. + +--- + ## Phases ### Phase 0-5: Foundation ✅ COMPLETE -- 2.54.0 From b2d717ae447a40cf5d62f60c9ff7a22e7679f2b8 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 00:56:28 -0500 Subject: [PATCH 03/65] =?UTF-8?q?audit=20PRD:=20add=20=C2=A710.5=20jurisdi?= =?UTF-8?q?ctional=20surface=20(IL=20+=20IN,=20federal,=20SOC2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit J flagged that the staffing system targets Chicago + Indiana — added a jurisdictional checklist section to the audit-trail PRD so counsel has a working starting point. Covered: - Federal: Title VII, ADEA, ADA, EEOC, OFCCP, FCRA, Section 1981 - Illinois: BIPA (high risk if any candidate photos), AI Video Interview Act (820 ILCS 42), Illinois Human Rights Act (broader than Title VII), PIPA breach notification, Day and Temporary Labor Services Act (directly applies — staffing industry-specific recordkeeping), Cook County + City of Chicago Human Rights Ordinances (additional protected classes including source of income, parental status, credit history) - Indiana: Data Breach Disclosure, Civil Rights Law (lighter than IL), Genetic Information Privacy Act - SOC 2 Type II as the typical SaaS sale gate (Privacy + Security TSCs most relevant; 6-9 month effort to first report) - HIPAA / PCI / ISO 27001 noted as out of current scope but flagged Phase reordering implications captured: - BIPA risk on real candidate photos may need to be resolved BEFORE audit-trail work (class-action exposure) - SOC 2 Type II prep runs in parallel, not after - IL Day and Temporary Labor Services recordkeeping may override our proposed 4-year retention SLA 7 open questions added that counsel must answer before the §8 phases can be locked in. Document is explicit (multiple times) that this is NOT legal advice — a research-grade checklist for J's counsel conversation. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/AUDIT_TRAIL_PRD.md | 76 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/docs/AUDIT_TRAIL_PRD.md b/docs/AUDIT_TRAIL_PRD.md index fddaba7..f91ecb6 100644 --- a/docs/AUDIT_TRAIL_PRD.md +++ b/docs/AUDIT_TRAIL_PRD.md @@ -227,12 +227,86 @@ Items 1-6 can be resolved by J's call. Item 7 needs design discussion — the sa --- +## 10.5 Jurisdictional surface (IL + IN) + +> **⚠ Not legal advice.** This is a research-grade checklist for J to take into a conversation with actual employment + privacy counsel. The system is targeting **Chicago (Illinois)** and **Indiana** placements per 2026-05-03 conversation. Counsel needs to verify what currently applies, what's pending, and whether case law has moved any of these in 2026. **Verify with counsel before claiming compliance with any item below.** + +### Federal layer (always applies) + +| Statute / framework | Relevance to this system | +|---|---| +| Title VII (Civil Rights Act) | Bans discrimination on race, color, religion, sex, national origin in hiring. Discrimination claim defense is the worked example in §1. | +| ADEA (Age Discrimination in Employment) | Bans age-based discrimination for workers 40+. DOB must be excluded from features per §4. | +| ADA (Americans with Disabilities Act) | Bans disability discrimination + requires reasonable accommodation. Disability-inferring features (gait, photo features, medical history) need exclusion. | +| EEOC enforcement | Receives complaints, issues right-to-sue. Audit response per §2 is what defends in EEOC investigation. | +| OFCCP | Applies if our staffing client serves federal contractors. Adds affirmative-action recordkeeping on top of EEOC. | +| FCRA (Fair Credit Reporting Act) | Triggers if background checks are performed. Pre-adverse-action notice + dispute process needed. | +| Section 1981 | Race-based contract discrimination — staffing is contract relationship. | + +### Illinois-specific (Chicago jurisdiction) + +| Statute | What | What we need | +|---|---|---| +| **BIPA** (Biometric Information Privacy Act, 740 ILCS 14) | Bans collection of biometric identifiers (face geometry, fingerprints, voiceprints) without informed written consent + retention schedule. Penalties: $1,000-$5,000 per violation per person. **Class actions are common and aggressive.** | If we use candidate photos for any feature (face match, headshot rendering, photo-derived attributes), BIPA almost certainly applies. The headshot pool we generate (per CLAUDE.md commit `5d93a71` area) needs careful review — synthetic faces are probably OK; real candidate photos are NOT without explicit BIPA-compliant consent. **Counsel must review.** | +| **Illinois AI Video Interview Act** (820 ILCS 42) | If AI analyzes recorded video interviews, employer must disclose AI use, obtain consent, provide explanation of how AI works, and limit who can review the video. | If we ever ingest video, this applies. Currently we don't, but worth flagging to counsel as a "what if we add this in 12 months" boundary. | +| **Illinois Human Rights Act** (775 ILCS 5) | Broader than federal Title VII — adds protected classes including arrest record, military status, marital status, order of protection, citizenship status (in some cases), unfavorable military discharge. | Protected attribute exclusion list in §4 needs expanding to cover IL-specific classes. | +| **Personal Information Protection Act** (PIPA, 815 ILCS 530) | Breach notification — must notify Illinois residents whose unencrypted PII was breached. | If identity service or workers parquet is breached, notification clock starts. Need incident response runbook. | +| **Illinois Day and Temporary Labor Services Act** (820 ILCS 175) | Specific to staffing/temporary services industry. Includes equal-pay-for-equal-work + record-keeping requirements + worker notification. | Highly relevant — applies directly to staffing-company clients. Audit retention may interact with these recordkeeping requirements. | +| **Workplace Transparency Act** | Restrictions on non-disclosure agreements re: harassment/discrimination | Tangential but worth noting. | +| **City of Chicago Human Rights Ordinance** (Title 6 Chicago Municipal Code) | Adds protected classes beyond IHRA (source of income, parental status, military discharge status, credit history). | Chicago-specific protected attributes list. | +| **Cook County Human Rights Ordinance** | Similar additions county-wide. | Chicago is in Cook County so this stacks. | +| **Possible: AI hiring transparency** | Several states/cities have proposed/passed laws modeled on NYC Local Law 144 (annual bias audit + candidate notification). I do not know whether IL or Chicago has such a law on the books as of 2026-01 cutoff. | **Counsel must check current state.** If it exists, we need annual bias audit reports (which IS what this PRD is building toward, but the report format may have specific requirements). | + +### Indiana-specific + +| Statute | What | What we need | +|---|---|---| +| **Indiana Data Breach Disclosure** (IC 24-4.9) | Breach notification within "without unreasonable delay" | Same incident response runbook as IL PIPA. | +| **Indiana Civil Rights Law** (IC 22-9) | State-level employment discrimination | Largely tracks federal Title VII, fewer expansions than IL. | +| **Indiana Genetic Information Privacy Act** | Bans use of genetic info in employment | Already in §4 protected list. | +| **General observation** | Indiana is generally less aggressive than Illinois on AI/employment regulation as of cutoff. | The IL bar is higher — if we satisfy IL, IN typically follows. **Counsel must confirm this isn't backwards.** | + +### Cross-cutting (security frameworks for SaaS sales) + +These aren't laws but are commonly required by enterprise customers (including staffing clients) before sale. + +| Framework | What | Relevance | +|---|---|---| +| **SOC 2 Type II** | Auditor attestation of operating effectiveness over 6-12 months across Trust Service Criteria (Security, Availability, Processing Integrity, Confidentiality, Privacy). | The Privacy criterion overlaps heavily with this PRD. Privacy + Security are the two load-bearing TSCs. Effort to first Type II report: 6-9 months. Type I (point-in-time) is faster (weeks) but enterprise buyers usually want Type II. | +| **SOC 3** | Public-facing summary of SOC 2 (no detailed control descriptions). | Nice-to-have for marketing but the staffing client will want SOC 2 Type II report under NDA. | +| **HIPAA** | Healthcare data protection. | Triggers ONLY if staffing places workers into healthcare roles where they handle PHI. Currently not in scope per CLAUDE.md. **Confirm scoping with J.** | +| **PCI DSS** | Payment card data | Not currently in scope. | +| **ISO 27001** | International information security management | Alternative to SOC 2; more common in EU. Probably unnecessary for IL/IN-only deployments. | + +### What this means for phase ordering + +The 9-phase plan in §8 is technically correct but may need re-ordering once counsel weighs in: + +- **BIPA risk on photos** is so high and so aggressive that if we use real candidate photos *anywhere*, that may need to be the FIRST thing we resolve — before the audit-trail work starts. Class-action exposure is enormous. +- **SOC 2 Type II prep** runs in parallel with this work, not after. If the staffing client says "show us your SOC 2 report" we need to have started the engagement weeks/months before. +- **Day and Temporary Labor Services Act** may impose recordkeeping that interacts with our retention SLA (§6) — counsel may say "no, retention has to be N years for THIS reason, not your defaulted 4." + +### Open questions for counsel (one ask) + +1. Does the staffing client have an existing SOC 2 report we leverage, or do we need our own? +2. Are we using any real candidate photos? If yes, is BIPA consent in place? +3. Does Illinois have an AI hiring transparency law on the books in 2026? If yes, what does the bias audit report need to look like? +4. What's the IL Day and Temporary Labor Services Act recordkeeping retention period? Does it interact with our 4-year proposed SLA? +5. Are background checks performed? If yes, do we need FCRA pre-adverse-action workflow integration? +6. Any healthcare placements? (HIPAA scoping) +7. Is the staffing client a federal contractor? (OFCCP scoping) + +Counsel's answers shape whether the §8 phase plan ships as-is or needs reordering. + +--- + ## 11. What this PRD is NOT - Not a contract with the staffing client. That document needs lawyers and signs after this is built. -- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC — passing actual certification is its own project. +- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC/BIPA/etc — passing actual certification is its own project. - Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made. - Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires. +- **Not legal advice.** The §10.5 jurisdictional surface is a research-grade checklist, NOT counsel's analysis. Verify everything with actual employment + privacy counsel licensed in IL + IN before claiming compliance with anything in this document. --- -- 2.54.0 From 505ea93726ddd3485bdc69f95ba8fb5bd597ff21 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:04:07 -0500 Subject: [PATCH 04/65] =?UTF-8?q?audit=20phase=201:=20discovery=20walk=20c?= =?UTF-8?q?omplete=20=E2=80=94=20subject=20+=20PII=20surface=20map?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read-only walk of both runtimes per AUDIT_TRAIL_PRD.md §8 phase 1. Fills "UNKNOWN" cells in PRD §3 + §7 with file:line evidence. Headline findings: - candidates_safe + workers_safe views EXIST as a defense layer but are BYPASSED — tool registry SQL templates query raw tables - PII traverses 7+ persistence/transmission paths per fill scenario: SQL → tool_result → LogEntry → /v1/respond → Langfuse → outcomes.jsonl → overseer_corrections.jsonl - candidate_id is stable but co-located with PII in workers_500k.parquet (no separate identity service) - /audit/subject/{id} endpoint does not exist - Append-only persistence is universal — RTBF requires crypto-erasure - Pathway memory is structurally subject-agnostic in fingerprints (defensive); trace bodies may leak PII (needs sampling) - Go side mirrors Rust PII shape — parity in the leak too - Worked example (John Martinez audit today): NOT POSSIBLE to produce complete-and-defensible response Recommends 4 cheap high-value moves before Phase 2 design starts: defense-layer enforcement (rewrite 3 SQL templates to _safe views), sample state.json/Langfuse to confirm pathway memory is clean, walk Bun mcp-server tool surface, schema-audit for protected-attribute proxies. None are commitments — J's call. No code changes in this commit. Companion to AUDIT_TRAIL_PRD.md. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/AUDIT_PHASE_1_DISCOVERY.md | 234 ++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 docs/AUDIT_PHASE_1_DISCOVERY.md diff --git a/docs/AUDIT_PHASE_1_DISCOVERY.md b/docs/AUDIT_PHASE_1_DISCOVERY.md new file mode 100644 index 0000000..e43f5ea --- /dev/null +++ b/docs/AUDIT_PHASE_1_DISCOVERY.md @@ -0,0 +1,234 @@ +# Phase 1 Discovery — Subject + PII Surface Map + +**Status:** Draft — 2026-05-03 · **Drafted by:** working session 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md) + +> **Purpose.** Read-only walk of both runtimes. Fills in the "UNKNOWN" cells in `AUDIT_TRAIL_PRD.md` §3 surface map and §7 current-state-vs-target table with file:line evidence. **No code changes.** Output: a complete picture of where subject identifiers + PII flow today, where they leak, and what an audit response could and could not produce right now. + +--- + +## TL;DR — what the walk found + +1. **A defense layer EXISTS but is BYPASSED.** Two PII-masked views (`candidates_safe`, `workers_safe`) live in `data/_catalog/views/` and correctly drop name/email/phone. **No production tool query uses them.** The MCP tool registry's `search_candidates` and `get_candidate` SQL templates query the raw `candidates` / `workers_500k` tables and return full PII to the LLM context. Worth considering: the views are functioning policy artifacts that nobody routes through. + +2. **PII flows freely through the LLM substrate.** In a single fill scenario, candidate names + emails + phones traverse: SQL → tool_result → LogEntry → execution_loop log → `/v1/respond` HTTP response → Langfuse trace input + output → `data/_kb/outcomes.jsonl` (fills with name) → `data/_kb/overseer_corrections.jsonl` (operation + correction text). At least 7 distinct persistence/transmission paths per scenario. + +3. **`candidate_id` is a stable token but not an isolated one.** It's a column on the same `workers_500k.parquet` that holds name/email/phone — there is no separate identity service. Joining candidate_id back to PII is one DataFusion query away. The audit-trail PRD's §5 identity-service intent is not yet built; the architectural separation does not exist. + +4. **No `/audit/subject/{id}` endpoint exists.** Reconstructing "every decision about person X" today requires manual cross-correlation across 4+ JSONL files + Langfuse + pathway memory + observer events. There is no canonical query path; the audit response cannot be produced in any reasonable time today. + +5. **Go side mirrors the Rust pattern.** Go validator's `rosterRow` carries Name; Go SessionRecord carries `Prompt` (truncated to 4000 chars) which contains the natural-language operation including candidate names. Cross-runtime parity in the PII-leak too. + +6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today. + +7. **The matrix-indexer is currently NOT subject-aware.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers. A pathway carries information about CODE behavior, not about CANDIDATES. This is good for audit defensibility (matrix index can't drive discrimination) BUT means any future "matrix index learns about candidate X" feature would require careful design to not become a PII sink. + +--- + +## §1 — PII flow paths in the Rust stack (file:line evidence) + +### 1A — MCP tools query raw tables (PII enters the LLM context) + +| File:line | What | PII columns returned | +|---|---|---| +| `crates/gateway/src/tools/registry.rs:96` | `search_candidates` SQL template | `first_name, last_name, phone, email, city, state` | +| `crates/gateway/src/tools/registry.rs:107` | `get_candidate` SQL template | `SELECT *` — returns ALL columns including hourly_rate_usd, address, anything in the table | +| `crates/gateway/src/tools/registry.rs:129` | `top_recruiters` SQL | recruiter names (employee PII, not candidate but still PII) | +| `crates/gateway/src/tools/registry.rs:141` | `engaged_unplaced_candidates` SQL | `c.candidate_id, c.first_name, c.last_name, c.phone, c.vertical` | +| `mcp-server/index.ts:1722, 2173, 2269` | address concatenation (`street_number + street_direction + street_name`) | full street addresses surfaced to MCP UI clients | + +**Architectural gap:** the MCP tool registry has no "view-only" mode. Adding `_safe` view routing would be one `sql_template` rewrite per tool. **Today: every fill scenario invokes search_candidates → LLM sees full PII for every candidate in the result set.** + +### 1B — LLM tool results captured in execution log + +| File:line | What | +|---|---| +| `crates/gateway/src/execution_loop/mod.rs:30-39` | `LogEntry` struct: `{turn, role, model, kind, content: Value, at}`. `content` is the raw tool_result body (carries PII when kind=`tool_result`). | +| `crates/gateway/src/execution_loop/mod.rs:217` | `self.append(LogEntry::new(..., "tool_result", trimmed))` — every tool result appended to `self.log` | +| `crates/gateway/src/execution_loop/mod.rs:1319` | per-render cap: `kind == "tool_result"` gets 1200 char cap (vs 200 for other kinds). Trimming is for token economy, **not PII redaction** — names/emails fit in 1200 chars. | +| `crates/gateway/src/execution_loop/mod.rs:30-50` | `Fill` struct (in same file): `{candidate_id: String, name: String, reason: Option}`. The `name` field is load-bearing per the PRD-defined FillProposal artifact contract. | + +### 1C — Execution log returned to HTTP caller + +| File:line | What | +|---|---| +| `crates/gateway/src/v1/respond.rs:111-119` | `RespondResponse { status, artifact, log: outcome.into_log(), iterations, error }` — full log array returned in JSON response body to whoever called `POST /v1/respond` | +| `crates/gateway/src/execution_loop/mod.rs:82-93` | `RespondOutcome::{Ok,Failed,Blocked}` all carry `log: Vec` — no path drops the log on its way to the caller | + +**Audit implication:** any HTTP client calling `/v1/respond` today receives full PII in the response. Authentication on `/v1/respond` is the only access control. Authorization is binary (either you can call it or you can't); there is no row-level filtering of which candidates you're allowed to see in the response. + +### 1D — Persisted JSONL sinks on disk + +| File:line | Path written | PII shape | +|---|---|---| +| `crates/gateway/src/execution_loop/mod.rs:508` | `data/_kb/outcomes.jsonl` (append) | `{operation, fills: [{candidate_id, name, reason?}], error, ...}` — observed live: `operation` carries natural-language fill request including role + city + state; `fills` carries the full Fill struct with name. | +| `crates/gateway/src/execution_loop/mod.rs:701` | `data/_kb/overseer_corrections.jsonl` (append) | `{operation, correction, sig_hash, ...}` — `correction` is the overseer's free-text guidance which often references specific candidates by name (e.g. "the executor picked Emily Garcia who is at fill capacity, try Maria Rodriguez instead") | +| `mcp-server/observer.ts:138-139` | `data/_observer/ops.jsonl` (append) | observer event log; PII content depends on what the operation context included. **Empty on this box right now** (file doesn't exist), but the writer is wired. | +| `mcp-server/observer.ts:161` | `data/_kb/observer_escalations.jsonl` (append) | sampled live: technical analysis fields (no PII observed in current rows), but `analysis` field is free-form LLM output and could include names depending on the escalation trigger. | +| `crates/gateway/src/v1/session_log.rs` (writer) → `/tmp/lakehouse-validator/sessions.jsonl` | per `lakehouse.toml` `[gateway].session_log_path` | `SessionRecord { ..., prompt, attempts: [{... raw}], artifact }` — **`prompt` carries the operation text (PII), `raw` carries each model attempt's raw text (PII when model reasoned about specific candidates), `artifact` carries the final FillProposal `{candidate_id, name}` shape on success.** | + +### 1E — Langfuse external persistence + +| File:line | What gets sent | +|---|---| +| `crates/gateway/src/v1/langfuse_trace.rs:166` | `body.input.messages = ev.input` — full message array (system + user + assistant + tool messages). Tool messages contain raw PII for tool_result entries. | +| `crates/gateway/src/v1/langfuse_trace.rs:191` | `body.output = ev.output` — full model generation. When the model emits a Fill or reasons about candidates, names appear here. | + +Langfuse runs at `:3001` per memory, with credentials in `/etc/lakehouse/langfuse.env`. The Langfuse storage tier holds these traces. **PII leaves the lakehouse infrastructure boundary at this point** — Langfuse's storage layer (Postgres + ClickHouse, default config) holds it. + +### 1F — Pathway memory + +| File:line | What | +|---|---| +| `crates/vectord/src/pathway_memory.rs:169` | `PathwayTrace` struct definition | +| `data/_pathway_memory/state.json` | Persisted state — **need read** to confirm whether real names appear in traces. | + +Pathway traces are keyed by `pathway_id = SHA256(task_class + file_prefix + signal_class)` — none of which are subject identifiers. **The fingerprint is structurally subject-agnostic.** Whether the trace BODY (kb_chunks, observer_signals, reducer_summary, final_verdict) leaks PII depends on what those fields contain at write time. From the struct shape: `reducer_summary` and `final_verdict` are strings written from execution loop output — **highly likely to leak PII** when summarizing fill outcomes. Need to confirm against live state.json. + +--- + +## §2 — Defense layer that exists but is bypassed + +`data/_catalog/views/candidates_safe.json`: +- Drops: `last_name, email, phone, hourly_rate_usd` +- Masks: `candidate_id` (keep first 3, last 2 chars — `CAND-...01` shape) +- Row filter: `status != 'blocked'` +- Visibility intent (per `description`): "Visible to recruiter / mode-runner agents." + +`data/_catalog/views/workers_safe.json`: +- Drops: `name, email, phone, zip, communications, resume_text` +- Reason given: "resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale" +- Source for the rebuilt `workers_500k_v9` vector corpus + +**The `_safe` views are the right policy artifact.** The ARCHITECTURAL gap is at the SQL-template layer in `crates/gateway/src/tools/registry.rs` — the LLM-facing tools query the RAW tables, not the SAFE views. Three out of three candidate-touching tool templates (search_candidates, get_candidate, engaged_unplaced_candidates) bypass the safe view. + +There is no enforcement preventing this. There is no test that asserts "tool SQL must reference only `_safe` tables." There is no warning logged when raw tables are queried via tool surface. + +**Quick fix shape (NOT a phase 1 deliverable, just noting the change shape):** rewrite the tool sql_templates to `FROM candidates_safe` (or `FROM workers_safe`); add a build-time check that `crates/gateway/src/tools/registry.rs` only references `*_safe` tables; add a runtime gate in queryd that refuses LLM-attributed queries on raw tables. These would land in Phase 4 (subject tagging across substrates) or possibly Phase 1.5 (defense-layer enforcement) per the AUDIT_TRAIL_PRD. + +--- + +## §3 — Identity / candidate_id provenance + +| Question | Finding | +|---|---| +| Is `candidate_id` a stable token? | Yes — observed format `CAND-NNNNNN` (e.g. `CAND-000001`). Stable across the table. Used as join key in the schema (per `crates/ui/src/main.rs:160` cross-table join hints). | +| Is the candidate_id ↔ PII mapping in a separate service? | **No.** Both live in `data/datasets/workers_500k.parquet`. A SQL `SELECT name, email FROM candidates WHERE candidate_id = '...'` resolves the mapping in one query. | +| Is the mapping itself audited? | **No.** No log records "who looked up the PII for which candidate when." This is the identity-service gap from PRD §5. | +| Does anything ever generate a *different* token (UUID, opaque hash) instead of using candidate_id directly? | **No** — every tool, every validator, every persistence sink uses candidate_id as-is. | + +**Production-ready implication:** the staffing client's lawyer asks "show me the access log for candidate X's PII" → we cannot produce one. Every access happens via SQL against `workers_500k`; no row-level access log exists. + +--- + +## §4 — Go-side parity with the leak surface + +| Rust file:line | Equivalent Go file:line | Same shape? | +|---|---|---| +| `crates/validator/src/lib.rs` (FillProposal carries `{candidate_id, name}`) | `internal/validator/fill.go` + `internal/validator/lookup_jsonl.go:23-31` (`rosterRow` carries `CandidateID, Name, Status, City, State, Role, BlacklistedClients`) | Yes — same PII shape | +| `crates/gateway/src/v1/session_log.rs` SessionRecord with prompt/artifact | `internal/validator/session_log.go` SessionRecord with `Prompt string`, `Attempts[].Raw string`, `Artifact map[string]any` | Yes — same shape | +| `outcomes.jsonl` / `overseer_corrections.jsonl` writers | Not present on Go side (Go validatord doesn't currently run the execution_loop with tool dispatch — Phase 4 of Go rewrite is what wires that) | **Asymmetric** — Go side writes less but only because feature parity isn't done | +| MCP tools registry | Not on Go side yet (mcp-server is still Bun) | **Bun is the surface** for tool dispatch in both runtimes today | + +**Cross-runtime audit implication:** even though our 5 parity probes (validator/extract_json/session_log/materializer/embed) all pass 32/32, **none of them assert PII handling**. A new `pii_parity.sh` probe would feed identical PII-tagged input through both runtimes and assert identical redaction behavior. As of today, neither runtime *does* redaction at the substrate level, so the probe would just confirm "both leak identically." + +--- + +## §5 — Mapping back to AUDIT_TRAIL_PRD §3 surface map + +Updated cells with file:line evidence: + +| Decision happens at | Currently logged where | Audit-completeness gap (revised) | +|---|---|---| +| Ingestion (candidate added to pool) | `data/datasets/workers_*.parquet` rows; ingestd writes via `crates/ingestd/`; no per-subject "added at" event journal found | **GAP** — no subject-tagged ingest event. When + who + how a candidate entered is not auditable per-subject. | +| Embedding creation | `crates/aibridge/src/client.rs` (post-sidecar-drop direct path); LRU cache `crates/aibridge/src/cache.rs` (commit `150cc3b`); cached entries are keyed by `(model, text)` — **the text is the cached body, which for candidate embeddings IS PII (candidate name + role + skills appear in source text per workers_500k_v9 build SQL).** Cache itself is in-memory, not persisted, but text-as-key means PII is in process memory. | **MAJOR GAP** — no per-embedding audit row. Cache key contains PII. No subject tagging means we cannot answer "what embedding was generated for candidate X." | +| Search inclusion | Tool result entries in execution log + Langfuse + outcomes.jsonl `fills` field | Partial — fills are logged, but you have to grep across outcomes.jsonl + Langfuse for any mention of a candidate_id. No "all search results that included X" canonical view. | +| Search rank | Result set in chat traces (Langfuse), not indexed by candidate | Partial — Langfuse trace has the result set; no inversion (candidate → ranks-received). | +| Fill recommendation | `outcomes.jsonl` `fills` array with name + candidate_id | Partial — present, but mixed with PII in the natural-language `operation` field. | +| Validation outcome | `sessions.jsonl` per `[gateway].session_log_path` config; per-attempt `verdict_kind`, `error` | Partial — works per-session, not per-subject. To find "all validations that touched candidate X" you'd need to grep the JSONL by candidate_id. | +| Iterate retry escalations | Same sessions.jsonl `attempts[]` array | Same as above | +| Observer signals | `data/_observer/ops.jsonl` (file not present today on this box, writer is wired) | UNKNOWN content shape until the writer fires for a fill scenario — needs verification | +| Matrix-indexer compounding | `data/_pathway_memory/state.json` — fingerprinted by code, NOT by subject | **NO subject leak in fingerprint structure** (good for audit defensibility). Trace bodies (`reducer_summary`, `final_verdict`) MAY carry PII — needs sampling to confirm. | + +--- + +## §6 — Mapping back to AUDIT_TRAIL_PRD §7 current-state-vs-target gap table + +| Capability | Phase 1 finding | Status | +|---|---|---| +| candidate_id as canonical token | Stable `CAND-NNNNNN` format. Same column lives in same parquet as PII. | Token exists; isolation does not. | +| Identity service | Doesn't exist. PII + candidate_id co-located in `workers_500k.parquet`. | Real gap — needs new service. | +| `/audit/subject/{id}` endpoint | Doesn't exist. No audit route in `crates/gateway/src/v1/mod.rs` route listing. | Real gap — needs new endpoint. | +| Subject-tagged embeddings | LRU cache `(model, text)` keys; text contains PII; no audit row per embed. | Real gap. | +| Subject-tagged search results | Langfuse trace contains result set; not subject-indexed. | Real gap (queryability, not capture). | +| Subject-tagged validation outcomes | Yes — sessions.jsonl carries candidate_id (in the attempt's `raw` field) but not as a queryable top-level field. | Partial — needs subject_id top-level promotion. | +| Subject-tagged matrix indexer entries | Pathway fingerprints are subject-agnostic by design. Trace bodies may leak. | Decision needed (PRD open question 7) — keep code-only OR risk PII surface. | +| Protected-attribute filter at decision time | **Not enforced anywhere.** SQL templates return whatever columns they SELECT; no protected-attribute removal at the gateway boundary. The `candidates` table schema in the demo SQL includes age (years_experience is a proxy), and the call_log/email_log tables likely contain free-text correspondence. | Real gap — major. Requires both schema audit + boundary enforcement. | +| Retention policy | None enforced. Append-only files grow indefinitely. | Real gap. | +| Right to be forgotten | Not implementable today on append-only logs without cryptographic erasure. | Per PRD §6 design, real engineering needed. | +| Cross-runtime parity | 5 algorithm probes pass; 0 audit/PII probes exist. | Probe set needs extension. | + +--- + +## §7 — Worked example: John Martinez audit trail TODAY (negative result) + +If John Martinez (candidate_id `CAND-042195`) requests an audit: + +1. **Find his candidate_id.** Manual SQL: `SELECT candidate_id FROM candidates WHERE first_name='John' AND last_name='Martinez'` — returns 1+ rows. Already a leak: someone with SQL access can correlate name → candidate_id ad-hoc. + +2. **Find every fill scenario that included him.** `grep "CAND-042195" data/_kb/outcomes.jsonl` — returns rows where his fill was included. But the row's natural-language `operation` field may NOT contain his candidate_id (it's the fill request, e.g. "fill: Welder x2 in Toledo, OH"), so we'd miss scenarios where he was a *candidate* but not a *fill*. To find those, we'd need to grep Langfuse traces (off-box) or the per-tool result content in execution logs (which aren't persisted as separate JSONL — they live in the HTTP response that already left the building). + +3. **Find every validation that touched him.** Grep `/tmp/lakehouse-validator/sessions.jsonl` for `CAND-042195` — would catch FillValidator phantom-ID rejections AND successful fills, but the candidate_id would appear in either the prompt or the raw attempt text, not as a top-level queryable field. + +4. **Find every embedding generated for him.** Cache is in-memory; nothing persisted. Cannot answer. + +5. **Find every search result that ranked him.** Off-box in Langfuse. Untrieable in lakehouse without a separate Langfuse query pipeline (which doesn't exist). + +6. **Find pathway memory traces involving him.** Pathway fingerprints don't carry his ID. The `reducer_summary` strings might mention him (need to grep state.json) but the fingerprint search wouldn't surface them. + +7. **Show what protected attributes were exposed to the model.** No record of input_features per decision — the LLM saw whatever the SQL returned, and we have no per-decision input_features audit row. + +8. **Format the output for legal.** Even if we collected all the above, there's no signing, no integrity hash, no schema, no template. + +**Estimated time to produce a complete-and-defensible response today: not possible.** Estimated time to produce an INCOMPLETE response by cobbling JSONLs + Langfuse exports: 2-5 hours per request, manual, error-prone, and the response would over-share (other candidates appearing in the same fill scenarios) AND under-share (embedding events, search rankings, pathway traces missed). + +This is the Phase 1 result the AUDIT_TRAIL_PRD predicted: today's substrate is not production-ready for a discrimination-defense audit response. + +--- + +## §8 — What this discovery DID NOT cover + +Phase 1 was scoped to file:line evidence + sampling of live JSONL state. The following deserve their own subsequent walks before phase 2+ design: + +1. **Live sample of `data/_pathway_memory/state.json`** — confirm whether `reducer_summary` / `final_verdict` strings actually leak PII or stay generic. Read 3-5 traces and grep for names from `workers_500k`. +2. **Live sample of Langfuse traces** — confirm input message PII for a real fill scenario from past 7 days. Use Langfuse `:3001` query API. +3. **observerd event content** — the writer is wired but `data/_observer/ops.jsonl` doesn't exist on this box. Trigger a fill scenario and inspect the resulting event. +4. **Bun mcp-server tool dispatch** — does the Bun server log tool calls anywhere? `mcp-server/index.ts` is 2900+ lines; partial walk only here. +5. **bot/propose.ts** — the bot proposal flow likely touches candidates; not walked in this pass. +6. **`crates/journald` mutation log** — designed for row-level mutations per ADR-012; haven't confirmed whether candidate-table mutations land here with PII. +7. **Go side observerd + chatd PII surface** — the Go cmd/* binaries likely have analogous logging; confirmed validator parity but didn't walk observer/chat logging. +8. **Process memory + crash dumps** — if the gateway dumps core, what PII is in it? Out of scope for code walk; comes up in security audit. +9. **Operator runbook** — who has access to logs/, /tmp/lakehouse-validator/, MinIO buckets, Langfuse Postgres? Out of scope for code walk; comes up in operational security review. + +These are listed so the next phase doesn't accidentally re-walk what's done OR skip what wasn't covered. + +--- + +## §9 — Recommended next moves (not commitments) + +Per AUDIT_TRAIL_PRD §8, Phase 2 is the identity service design doc. Before that doc gets written, the cheapest high-value moves discovered here: + +1. **Defense-layer enforcement (1-2 hours work).** Rewrite the 3 tool SQL templates in `crates/gateway/src/tools/registry.rs` to use `_safe` views. Add a unit test that asserts no `crates/gateway/src/tools/registry.rs` template references `FROM candidates ` or `FROM workers_500k ` (only `*_safe`). This is one commit. It prevents the most-trafficked PII leak path TODAY without waiting on the identity service. **Cost:** the LLM sees masked candidate_ids (`CAN...95` instead of `CAND-042195`); some downstream tools (validator existence check) would need a "resolve to full ID" path that goes through the identity service — but that's exactly the architectural shape PRD §5 wants anyway. + +2. **Sample state.json + Langfuse before phase 2 design.** §8.1 + §8.2 above. ~30 minutes. Either confirms or refutes the "matrix indexer is subject-clean" finding from §1F. + +3. **Document the Bun mcp-server tool surface.** §8.4. The Bun layer is a major PII transit point not fully covered here. + +4. **Identify whether protected attributes (age proxies, photo features, zip-code → race correlations) are currently in any tool-returned column.** Schema-level audit of `candidates` + `workers_500k`. ~1 hour. Might surface that some "neutral" columns are actually protected-attribute proxies. + +These four moves give the phase-2 design doc strong evidence to lean on. None are commitments — J's call on what to do next. + +--- + +## Change log + +- 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`. -- 2.54.0 From 627a5f0c3d4c555161f5b0d38fa95e2f75862971 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:13:07 -0500 Subject: [PATCH 05/65] =?UTF-8?q?audit=20phase=201:=20=C2=A710=20scrum-rev?= =?UTF-8?q?iew=20findings=20+=20walk=20back=20=C2=A71F=20over-claim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran cross-lineage scrum on the discovery doc with the new model fleet (opus + kimi-k2.6 + gemini-3-flash via Go gateway :4110, custom "senior security architect" prompt). 3/3 reviewers responded with substantive 800-1200 word reviews. Saved at /tmp/audit_scrum/. 5 convergent findings (≥2 reviewers) added as §10/C1-C5: C1. §1F matrix-indexer "good for audit defensibility" claim is over- claimed — walked back in TL;DR. Trace bodies unverified; treat as SUSPECTED PII sink until §8.1 sampling completes. C2. §1E (Langfuse) is the most dangerous leak — fix FIRST, ahead of view-routing. Boundary-crossing leak (GDPR Art. 44 / CPRA sale / SOC2 disposal). All 3 reviewers converge on this priority. C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills. EEOC UGESP (1978): need adverse-impact stats on everyone who could have been picked. Phase 1 worked example missed this. C4. BIPA / biometric exposure understated in findings (in PRD §10.5 but not translated to actionables). $1k-$5k per-violation regime. C5. candidate_id must be promoted to top-level field in all JSONL sinks. Grepping natural-language strings is not defensible audit strategy. 3/3 reviewers converge. 11 single-reviewer high-value catches added as §10 single-reviewer section: opus on LLM provider egress (8th PII path), Art. 22 right- to-explanation, special-category data, DPIA/ROPA/DPA inventory; kimi on sequential ID enumeration risk, Langfuse retention config, CCPA de-identified-in-place vs crypto-shred, Bun common-mode failure, cryptographic audit-trail integrity (Merkle/FRE 901), HIPAA BAA, revised SELECT * effort estimate; gemini on data residency, "culture fit" reasoning proxies, comparator-pool snapshot. §9 reordered: sample first → defense-layer second → Langfuse boundary third (was view-routing first per original draft; boundary-crossing leak is higher priority per scrum). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/AUDIT_PHASE_1_DISCOVERY.md | 63 ++++++++++++++++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/AUDIT_PHASE_1_DISCOVERY.md b/docs/AUDIT_PHASE_1_DISCOVERY.md index e43f5ea..64adf92 100644 --- a/docs/AUDIT_PHASE_1_DISCOVERY.md +++ b/docs/AUDIT_PHASE_1_DISCOVERY.md @@ -20,7 +20,7 @@ 6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today. -7. **The matrix-indexer is currently NOT subject-aware.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers. A pathway carries information about CODE behavior, not about CANDIDATES. This is good for audit defensibility (matrix index can't drive discrimination) BUT means any future "matrix index learns about candidate X" feature would require careful design to not become a PII sink. +7. **The matrix-indexer fingerprint is subject-agnostic; trace bodies are UNVERIFIED.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers, which is structurally defensive. **However:** trace bodies (`reducer_summary`, `final_verdict`) are written from execution-loop output and are highly likely to leak PII. Per §8.1 these are unverified. Treat the matrix indexer as a SUSPECTED PII SINK until sampled — do NOT rely on "matrix can't drive discrimination" framing. (Walked back from earlier draft per cross-lineage scrum §10/C1: that claim was over-stated in 3/3 reviews.) --- @@ -229,6 +229,67 @@ These four moves give the phase-2 design doc strong evidence to lean on. None ar --- +## §10 — Cross-lineage scrum review of this discovery doc (2026-05-03) + +After the discovery walk, this document was reviewed by three independent model lineages via `/v1/chat` against the Go gateway (post-PR-#13 model fleet): **opus** (`opencode/claude-opus-4-7`), **kimi** (`ollama_cloud/kimi-k2.6`), **gemini** (`ollama_cloud/gemini-3-flash-preview`). Custom prompt: senior security architect reviewing a discovery report. (DeepSeek-v3.2 timed out; not included.) + +Verbatim reviews saved at `/tmp/audit_scrum/{opus,kimi,gemini}_review.md`. Convergent findings (≥2 reviewers) are treated as high signal per `feedback_cross_lineage_review.md`. + +### Convergent findings — must address before phase 2 design + +**C1. §1F matrix-indexer claim is OVER-CLAIMED (3/3 reviewers).** The TL;DR #7 line "good for audit defensibility (matrix index can't drive discrimination)" overstates structural subject-agnosticism as behavioral fairness. Per opus: "until §8.1 is executed, the correct framing is 'fingerprint structurally subject-agnostic; body content unverified — treat as suspected PII sink.'" Per kimi: dangerous reasoning — if `reducer_summary` says "candidates named Emily Garcia were rejected for fill capacity," the matrix learns proxy variables and future similar names get downranked = temporal discrimination risk. Per gemini: "you cannot claim audit defensibility until you prove the *content* of the matrix indexer doesn't contain PII." **Action:** walk back the §1F TL;DR claim; reframe as "fingerprint structure is defensive; body content unverified — treat as suspected PII sink until §8.1 confirms." + +**C2. §1E (Langfuse) is the MOST DANGEROUS leak — fix FIRST (3/3 reviewers).** Opus: "boundary-crossing leak that makes a regulator's eyes light up." Kimi: "Article 44 GDPR transfer if SaaS-hosted; CPRA 'sale/sharing' question; subprocessor notification failure." Gemini: "un-certifiable for SOC2 Type II" + "unauthorized data transfer to third-party storage tier." All three would do Langfuse redaction/sampling BEFORE the §9.1 view-routing fix. **Action:** revise §9 priority order — Langfuse boundary first, view-routing second. + +**C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills (3/3 reviewers).** Opus: "you need not just 'what did we do for candidate X' but 'what was the selection rate for protected class Y vs comparator' — that requires capturing protected attributes WITH outcomes." Kimi: "EEOC Uniform Guidelines on Employee Selection Procedures (1978) — matrix index that learns from historical fill outcomes IS a selection procedure under the Guidelines." Gemini: "The system doesn't log the *entire* candidate pool for a specific search — only the fills (§5). To defend a lawsuit, you must show the stats for everyone who *could* have been picked, not just the person who was." **Action:** Phase 1 didn't capture this load-bearing requirement. PRD §1 worked example needs expanding: the audit response must include the comparator pool + adverse-impact statistics, not just the subject's decision row. + +**C4. BIPA + biometric exposure UNDERSTATED (3/3 reviewers).** Already in `AUDIT_TRAIL_PRD.md` §10.5 jurisdictional checklist but NOT translated into Phase 1 findings actionables. If `workers_500k` columns include photo paths, video interview references, or anything that could yield biometric inference (per gemini: "even descriptors that could be reconstructed into biometric templates"), BIPA's $1k-$5k per-violation regime applies BEFORE the GDPR analysis matters. **Action:** add to §8 (what discovery did NOT cover): explicit photo/video/biometric column audit of `workers_500k` schema. + +**C5. `candidate_id` must be PROMOTED to top-level field in all JSONL sinks (3/3 reviewers).** Opus + kimi + gemini converge: grepping natural-language strings (operation, raw, prompt, reducer_summary) for candidate_id is not a defensible audit strategy. Even if subject_id appears in those strings TODAY, it appears co-mingled with other candidate names, model reasoning, etc. — making subject filtering unsafe. **Action:** add to PRD §7 target column "subject_id top-level promotion" — change session_log writer + outcomes.jsonl writer + observer event writer to ALL include `subject_id` (or `subject_ids[]`) as a first-class top-level field. + +### Single-reviewer findings — verified, worth incorporating + +**OPUS unique:** +- **8th PII path missing: LLM provider egress.** PR #13 routes models through opencode + ollama_cloud + openrouter — opencode and openrouter are external services. Cross-border data transfer under GDPR Ch. V; third-party processor relationship requiring DPA under Art. 28. Phase 1 did not enumerate this path. **Action:** add §1G "LLM provider egress" to §1. +- **GDPR Art. 22 / EU AI Act right to explanation.** Audit must capture the model's REASONING CHAIN, not just decision output. Phase 1 §7 worked example doesn't include this. **Action:** add to subject-audit response shape (PRD §2). +- **Special-category data under GDPR Art. 9.** resume_text + call_log + email_log routinely contain health (accommodation requests), union, religion. Higher legal bar — Art. 9(2) explicit consent required. Phase 1 §6 mentions these tables exist but doesn't flag the special-category exposure. **Action:** call out in §6. +- **DPIA / ROPA / DPA inventory.** None of these documents referenced. Some may exist outside code; Phase 1 should at minimum note their absence as Phase 1.5 input. + +**KIMI unique:** +- **Sequential `CAND-NNNNNN` IDs enable enumeration attacks.** Predictable IDs let an attacker scan the candidate space. Security finding Phase 1 didn't flag. +- **Langfuse retention config unaudited.** Default trace retention is 30 days in some versions, indefinite in others. Directly impacts RTBF analysis. **Action:** check live Langfuse config. +- **CCPA "de-identified in place" may be faster than crypto-shred.** Replacing PII with `REDACTED-{hash}` while preserving log structure may satisfy CPRA's de-identified exception. Worth considering vs. crypto-shred. +- **Bun MCP server is the cross-runtime bridge — likely COMMON-MODE failure.** Phase 1 framed Go side as "mirrors Rust pattern" implying independent failure; it's likely shared infrastructure failure. **Action:** add to §4 — common-mode reframe. +- **Cryptographic integrity for the audit trail itself.** Merkle trees / signed logs / chain-of-custody under FRE 901. Opposing expert can challenge admissibility without this. **Action:** add to PRD §2 audit response shape — integrity-signed. +- **HIPAA Business Associate Agreement scoping not done.** If any candidate is healthcare-vertical, BAA analysis required. +- **`get_candidate` SELECT * has 40+ load-bearing columns.** §9.1 "1-2 hours" estimate for view rewrite is irresponsible without scoping downstream consumers. **Action:** revise §9.1 estimate; flag dependency analysis as prerequisite. + +**GEMINI unique:** +- **Data residency — JSONL on US box.** If any candidate is EU resident, GDPR violation without SCC/DPF. Phase 1 didn't ask whether IL+IN target market includes EU residents (probably not, but staffing-co clients sometimes have international placements). +- **"Culture fit" reasoning strings as discrimination proxies.** Common LLM-generated phrases ("not a culture fit," "communication concerns," "team chemistry") often correlate with protected-attribute discrimination. Phase 1 didn't audit the actual reasoning text in `outcomes.jsonl` for these phrases. +- **Comparator-pool snapshot for every fill.** Need to capture WHO COULD HAVE BEEN PICKED, not just who was. **Action:** PRD §2 audit response shape needs `comparator_pool` field per decision. + +### Revised §9 — recommended next moves (reordered by scrum convergence) + +1. **(NEW PRIORITY 1, was P3) Sample state.json + Langfuse content** — confirm/refute the matrix-indexer subject-clean claim and quantify Langfuse PII exposure. Cheapest move that resolves the over-claim AND informs the boundary-leak fix. +2. **(NEW PRIORITY 2, was P1) Defense-layer enforcement at SQL template level** — rewrite tool registry to use `_safe` views. **Estimate revised UPWARD per kimi**: scope `get_candidate` SELECT * downstream consumers first; estimate 4-8 hours including the existence-check resolution path through the (not-yet-built) identity service. Stop-gap until then: add LLM-attribution flag to queryd, refuse `FROM candidates `/`FROM workers_500k ` queries that originate from tool dispatch. +3. **(NEW) Langfuse boundary audit + redaction** — sample retention config, check DPA status, scope a redaction/sampling layer that strips PII from message arrays before the Langfuse POST. This is the boundary-crossing leak — fix BEFORE chasing internal sinks. +4. **(NEW) Subject_id top-level promotion to all JSONL writers** — single architectural change spanning Rust + Go session_log + observerd event writer + outcomes/corrections appenders. Makes subject-correlation queries defensible (no more grepping natural language strings). +5. **(was P4) Schema-audit for protected-attribute proxies** — extend to include "culture fit"-shaped reasoning text in outcomes.jsonl + the comparator-pool retention requirement. +6. **(NEW) BIPA-specific audit of workers_500k schema** — explicit photo/video/biometric column inventory before any production deployment in IL. +7. **(NEW) Operational discovery** — DPIA, ROPA, DPA inventory, SCC for cross-border, Langfuse retention config. Out-of-code-walk; needs J + counsel input. + +### What I'm walking back + +§1F TL;DR claim "matrix indexer is good for audit defensibility" — per all 3 reviewers, this is over-claimed without §8.1 verification. The correct frame is "fingerprint structure is subject-agnostic by design; trace body content unverified — treat as suspected PII sink until sampled." + +§9 ordering — view-routing was P1; per all 3 reviewers, Langfuse boundary should be P3 in front of it. View-routing is the source-side fix; Langfuse is the boundary-crossing fix; both matter, do them in BOUNDARY-FIRST order. + +§9.1 effort estimate — kimi's "irresponsible without dependency scoping" critique is right. Revised UP to 4-8 hours. + +--- + ## Change log - 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`. +- 2026-05-03 — §10 added: cross-lineage scrum review of the discovery doc (opus + kimi + gemini). 5 convergent findings (matrix-indexer over-claim, Langfuse first-priority, comparator pool gap, BIPA understated, subject_id top-level promotion). Plus single-reviewer high-value catches. §9 reordered. -- 2.54.0 From 64bda2161405a67b94adcfde4aea59302d13550d Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:16:27 -0500 Subject: [PATCH 06/65] =?UTF-8?q?audit=20PRD:=20J=20answered=205=20open=20?= =?UTF-8?q?questions=20=E2=80=94=20fold=20into=20=C2=A710,=20revise=20phas?= =?UTF-8?q?e=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Conversation 2026-05-03 — J confirmed: - Photos/video YES → BIPA in full force ($1k-$5k per violation) - Langfuse self-hosted → drops GDPR Art. 44 cross-border concern - EU not in scope now but placeholder needed → design EU-compatible - Healthcare vertical YES → HIPAA BAA needed with model providers, PHI redaction at gateway boundary OR local-only routing for those requests, vertical-detection at boundary is Phase 2 requirement - Training/RAG MAY re-run on outcomes → design as if it will, training- safe export interface needed, crypto-erasure becomes load-bearing evidence chain §10 updated with answered/pending status per question. New §10.5 "Effect on phase plan" introduces: - Phase 1.5 (NEW) — BIPA photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample, BEFORE Phase 2 design - Phase 2 design must now include: EU-placeholder fields, vertical detection, training-safe export, BIPA consent metadata - Phase 9 rehearsal must cover discrimination + BIPA + healthcare PHI 3 questions still pending J's call before Phase 2 design ships: identity service daemon vs in-process, JSON vs signed PDF for legal export, audit endpoint auth model. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/AUDIT_TRAIL_PRD.md | 49 ++++++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/docs/AUDIT_TRAIL_PRD.md b/docs/AUDIT_TRAIL_PRD.md index f91ecb6..2d3a20f 100644 --- a/docs/AUDIT_TRAIL_PRD.md +++ b/docs/AUDIT_TRAIL_PRD.md @@ -211,19 +211,48 @@ The identity service is the new shared substrate — both runtimes call it; the --- -## 10. Open questions blocking phase 1 +## 10. Open questions blocking phase 2 (resolved 2026-05-03) -These are the things I need J to decide before phase 1 can start, OR I need to investigate-and-propose: +**These were the original Phase 1 open questions. J answered the load-bearing 5 in conversation 2026-05-03; answers folded in below. Remaining items 1, 4, 6 still need J's call before Phase 2 design ships.** -1. **Identity service: separate daemon vs in-process?** Recommend separate. Confirm. -2. **Retention period N years?** Recommend 4. Need staffing client's legal call. -3. **Photo / surname / zip-code policy?** These are inferred-attribute risks. Need policy decision. -4. **JSON or signed PDF for legal export?** Different downstream costs. -5. **Right-to-be-forgotten under append-only logs:** cryptographic erasure (proposed) or hard delete (breaks integrity)? Confirm crypto-erasure approach. -6. **Audit endpoint auth model:** legal-only credential, or shared with admin? Recommend legal-only with separate token rotation. -7. **The "indexed before search" concern:** matrix indexer + pathway memory currently fingerprint by code, not subject. Do we (a) make them subject-aware (more audit completeness, more PII surface area), (b) keep them code-only and assert in audit response that "no subject-specific compounding state was used," or (c) something else? +1. **Identity service: separate daemon vs in-process?** _Recommended: separate. **Status: pending J confirmation.**_ +2. ~~**Retention period N years?**~~ — Out of scope for now; will be set at deployment time per client contract. Default to 4-year hot retention until set. +3. ✅ **Photos/video in scope?** **YES** (J 2026-05-03). BIPA (740 ILCS 14) applies in full. Per-violation $1k-$5k statutory damages, written consent + retention schedule mandatory. **This becomes a Phase 1.5 priority** — see §10.5/§13 for revised phase ordering. +4. **JSON or signed PDF for legal export?** _Recommend signed JSON with PDF rendering option. **Status: pending.**_ +5. ✅ **RTBF under append-only:** Cryptographic erasure approach approved in principle (J 2026-05-03 implicit via approval of plan). +6. **Audit endpoint auth model:** _Recommend legal-only credential separate from admin token. **Status: pending.**_ +7. ✅ **Matrix indexer subject-awareness:** Per scrum review (`AUDIT_PHASE_1_DISCOVERY.md` §10/C1), matrix-indexer is suspected PII sink (trace bodies unverified). Action: sample state.json before deciding (a) keep code-only + add PII-redact-on-write to trace bodies, OR (b) remove subject-summary text from trace bodies entirely. Decision deferred until §8.1 sampling completes. -Items 1-6 can be resolved by J's call. Item 7 needs design discussion — the safest answer for legal defense is (b), but it loses the "pathway learns about THIS candidate" signal that may be load-bearing for the staffing UX. +### Newly answered 2026-05-03 (J) + +8. ✅ **Langfuse hosting model:** **Self-hosted.** Removes the GDPR Art. 44 cross-border-transfer concern that 3/3 scrum reviewers flagged. Langfuse retention config + Postgres/ClickHouse access controls still need to be audited as part of Phase 1.5 — but the boundary stays inside J's infrastructure, which is materially better than SaaS Langfuse. + +9. ✅ **EU candidates in scope:** **Not currently — may need placeholder later.** Design choice: build the identity-service interface to be EU-compatible (DPIA-shaped fields, lawful-basis tracking, SCC-ready transfer mechanism slots) but DO NOT gate Phase 2 on EU compliance. Phase 2 ships IL+IN-shaped; EU additions are a follow-up phase. + +10. ✅ **Healthcare vertical / HIPAA:** **Same framework — yes.** Healthcare staffing IS in scope. PHI in `resume_text`, `communications`, and `call_log` is realistic. **Implications:** + - Business Associate Agreement (BAA) required with any third-party model provider that processes content from healthcare-vertical staffing requests + - opencode + ollama_cloud + openrouter (per PR #13 routing) are external — BAAs needed OR healthcare requests must route to local-only models (Ollama on-box) + - PHI redaction at the gateway boundary becomes mandatory before the model call leaves the box, OR the model call must stay on-box for healthcare requests + - Vertical detection at the gateway boundary becomes a Phase 2 requirement + +11. ✅ **Training / RAG re-runs may use historical outcomes:** **Yes — design as if it WILL.** Implications: + - `outcomes.jsonl` and `overseer_corrections.jsonl` cannot remain raw-PII forever — anything that lands in a training corpus or RAG re-index becomes ungeneratable to delete (PII in model weights) + - Phase 2 design must include a "training-safe export" pipeline that strips PII from outcomes before feeding to any training/RAG path + - Crypto-erasure of historical outcomes becomes load-bearing — if a candidate exercises RTBF and their data already trained a model, we must be able to evidence "the source was destroyed; the model retains it indistinguishably from synthetic patterns" + +### Effect on the §8 phase plan + +The user-confirmed answers shift priorities. Revised ordering (incorporating scrum-driven priority changes from `AUDIT_PHASE_1_DISCOVERY.md` §10): + +- **Phase 1.5 (NEW)** — BIPA-specific photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample. Lands BEFORE Phase 2 design starts. +- **Phase 2 (identity service design)** — now must include EU-placeholder fields, vertical-detection (healthcare flag), training-safe export interface, BIPA consent + retention metadata +- **Phase 3** (audit endpoint skeleton) — unchanged +- **Phase 4** (subject tagging) — must include healthcare-vertical routing decision at gateway boundary +- **Phase 5** (identity service build) — must include BIPA-compliant biometric metadata table +- **Phase 6** (protected-attribute boundary) — must include PHI redaction for healthcare-vertical requests +- **Phase 7** (retention + RTBF) — must include training-safe export evidence chain +- **Phase 8** (legal export) — unchanged +- **Phase 9** (rehearsal) — must include both EEOC discrimination scenario AND BIPA biometric scenario AND healthcare PHI breach scenario --- -- 2.54.0 From fd429f4185f62ad573ad3def12b1ee92f9180627 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:22:53 -0500 Subject: [PATCH 07/65] audit phase 1.5: BIPA schema audit + outcomes.jsonl content sample MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-up walks per AUDIT_PHASE_1_DISCOVERY §10/C4 + gemini scrum flag. Read-only. No code changes. BIPA findings: - scripts/staffing/tag_face_pool.py uses deepface to extract gender + race + age from face images. Output persists to data/headshots/ manifest.jsonl. For synthetic faces this is fine; for real candidate photos this becomes a regulated biometric database (740 ILCS 14/10). - mcp-server/index.ts:1408 ComfyUI prompt EXPLICITLY embeds protected attributes (age + race + gender) into model prompt — system-level encoding of protected-attribute features into AI workflow. - mcp-server/search.html:3375-3432 has hard-coded FEMALE_NAMES / MALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup tables — name-based ethnicity inference. Title VII / disparate-impact risk separate from BIPA. - data/headshots/manifest.jsonl is TRACKED IN GIT today (synthetic classifications). For real photos, this would be biometric data in version control — serious failure. - No consent flow, no public retention schedule, no deletion procedure, no employee training documented. All required by BIPA §15(a)/(b) before real-photo intake. outcomes.jsonl sample: - 39/101 rows persist candidate names in fills[*].name field today - Sample names: "Carmen I. Garcia", "Jamal Z. Jones", "Jacob N. Patel" (synthetic but real shape) - 0 hits for "culture fit" / "communication" / etc proxy phrases — synthetic data doesn't generate them. When real models reason about real candidates, they will. Append-only persistence makes RTBF cryptographic-erasure-only. Recommends Phase 1.6 (NEW) — BIPA pre-launch gates between Phase 1.5 and Phase 2: BIPA_COMPLIANCE_POLICY.md, consent gate at upload endpoint, quarantine real-photo classifications to data/biometric/, deprecate name->ethnicity lookup tables, unit test that synthetic manifest stays synthetic. 4-8 hours of design + one code commit. 5 open questions for J: where do real photos enter, will deepface tagging path stay for real photos, consent UX, retention duration floor, designated privacy officer. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md | 177 ++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md diff --git a/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md b/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md new file mode 100644 index 0000000..31f1e63 --- /dev/null +++ b/docs/AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md @@ -0,0 +1,177 @@ +# Phase 1.5 — BIPA Schema Audit + outcomes.jsonl Content Sample + +**Status:** Complete — 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md) + +> **Scope.** Two follow-up walks defined in `AUDIT_PHASE_1_DISCOVERY.md` §10/C4 + §10/single-reviewer (gemini "culture fit"): (1) BIPA-specific photo/video/biometric column + code path inventory, (2) actual content sample of `data/_kb/outcomes.jsonl` to confirm whether PII or discrimination-proxy reasoning lands there. Read-only walk; no code changes. + +--- + +## §1 — BIPA walk: photo/video/biometric surface + +### 1A — The face pipeline + +The Rust legacy ships a synthetic face pool with deepface-extracted demographic classifications. **Even though the FACES are synthetic, the LOGIC is BIPA-relevant** — when real photos arrive (J 2026-05-03: "we will" have them), the same pipeline processes them. + +| File:line | What | BIPA relevance | +|---|---|---| +| `scripts/staffing/fetch_face_pool.py` | Fetches 1000 synthetic faces from `thispersondoesnotexist.com` (StyleGAN). | Synthetic — no biometric collection at this stage. | +| `scripts/staffing/tag_face_pool.py:18-31` | Runs `deepface` on each face to extract `gender` (man / woman) + `race` (asian / black / hispanic / indian / middle_eastern / white) + `age`. Writes to `data/headshots/manifest.jsonl`. | **BIPA biometric processing.** Deepface extracts demographic classifications from face geometry. The output IS biometric information per 740 ILCS 14/10. | +| `data/headshots/manifest.jsonl` | 1000 rows, each `{id, file, gender, race, age}`. Sampled rows confirm shape: e.g. `{id: 0, file: "face_0000.jpg", gender: "woman", race: "east_asian", age: 33}`. | The persisted classifications. **For real photos, this file becomes a regulated biometric database.** | +| `mcp-server/index.ts:1308` (`/headshots/:key` route) | Serves face matched by gender × race × age intersection bucketing for a candidate. | Routing logic predicated on biometric-derived demographic attributes. | +| `mcp-server/index.ts:1367` (`/headshots/generate/:key`) | ComfyUI on-demand portrait generation. | Custom-generated image keyed to candidate. | +| `mcp-server/index.ts:1408` (ComfyUI prompt template) | `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic` | **Protected attributes (age + race + gender) embedded directly into the model prompt.** Even with synthetic output, this is system-level encoding of protected-attribute features into AI workflow. | + +### 1B — Name-based ethnicity inference (inferred-attribute risk) + +`mcp-server/search.html:3375-3432` defines lookup tables that classify candidates by demographic attributes derived from their NAMES (not photos): + +| Constant | What | +|---|---| +| `FEMALE_NAMES` (~200 entries) + `MALE_NAMES` | Gender inference from first name | +| `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` | Ethnicity inference from first name | +| `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` | Ethnicity inference from surname (more diagnostic than first name per the source comment) | + +**This is hard-coded inferred-attribute classification.** Per CLAUDE.md, the rationale is "Confidence-default name resolution... so a forklift operator looks like a forklift operator" — i.e., assigning representative photos. But the code path is: + +`candidate name → genderFor() / guessEthnicityFromFirstName() → face-pool bucket selection` + +A discrimination plaintiff's lawyer would frame this as: "the system EXPLICITLY classifies candidates by inferred ethnicity and uses that classification in its workflow." Whether the downstream use is "just for the photo display" or "ranks candidates" doesn't matter for BIPA-style strict-liability framing — it matters for Title VII, where intent / disparate-impact tests apply. + +### 1C — Data directories holding face content + +| Path | Count today | Purpose | gitignored? | +|---|---:|---|---| +| `data/headshots/` (face_*.jpg + manifest.jsonl) | 2500 entries | Synthetic face pool + biometric classifications | partial — manifest.jsonl is tracked, face_*.jpg are gitignored | +| `data/headshots_role_pool/` | 7 entries | Role-specific representative faces | gitignored | +| `data/headshots_gen/` | 4 entries | ComfyUI on-demand portrait cache | gitignored | +| `data/icons_pool/` | 2 entries | Certification + role icons | gitignored | +| `data/face_test/` | 1 entry | Likely deepface dev/test scratch | gitignored | +| `data/_imagecache/` | (not enumerated) | Image cache | gitignored | + +The `manifest.jsonl` file is **tracked in git** today and contains the demographic-classification table. For synthetic faces this is fine; **for real candidate faces, this would be a biometric database in version control** — a serious BIPA + general-data-protection failure. + +### 1D — Biometric-info contract risk when real photos arrive + +When the system transitions from synthetic to real candidate photos (J: "we will"), the BIPA exposure becomes immediate: + +1. **No written consent flow exists.** BIPA §15(b) requires "informed written consent" before biometric collection. The headshot upload path (wherever it gets built) needs a consent gate. + +2. **No public retention schedule exists.** BIPA §15(a) requires a written, publicly-available retention schedule. No such document found. + +3. **No deletion procedure exists.** BIPA §15(a) requires destruction "when the initial purpose for collecting or obtaining such identifiers or information has been satisfied or within 3 years of the individual's last interaction with the private entity, whichever occurs first." No such procedure exists. + +4. **No employee training documented.** BIPA litigation routinely cites lack of employee training on biometric handling as evidence of negligence. + +5. **The deepface library is itself a biometric processor.** Using it triggers the BIPA disclosure obligation regardless of whether the system stores the underlying face image — extracting "race" and "gender" from a face IS biometric processing under modern interpretations. + +6. **Per-violation damages.** Negligent violations: $1,000 per violation per person. Intentional/reckless: $5,000 per violation per person. The IL Supreme Court has held each separate scan/extraction can constitute a separate violation (Cothron v. White Castle, 2023). Class actions are routine; recent settlements: + - Facebook (Patel v. Facebook, 2020): $650M + - TikTok (In re TikTok BIPA, 2022): $92M + - Six Flags (Rosenbach v. Six Flags, 2019): set the "no actual harm needed" precedent + - Smaller staffing companies have settled for $1M-$50M ranges. + +### 1E — What this means for the architecture + +Even with synthetic data, the architecture has BIPA-shaped surface that is one upload path away from being live. **Recommended pre-real-photo gates:** + +1. **Disable deepface tagging on real-photo path** until consent + retention schedule are in place. Synthetic faces can keep using deepface; real faces cannot. +2. **Quarantine `data/headshots/manifest.jsonl` for real photos.** Real-photo classifications must NOT land here. Either move to a dedicated `data/biometric/` tree with separate retention or — preferable — don't extract demographic features from real photos at all. +3. **Replace the name → ethnicity lookup tables with a non-classifying default.** The current logic exists to pick "representative" photos. For real photos, the candidate's own photo IS representative — no inference needed. The lookup tables become dead code. +4. **Upload path needs consent gate.** Whatever endpoint accepts real photos must check a stored consent timestamp + version before storing the file. +5. **Retention schedule documented + auto-enforced.** Crontab / ingestd background sweep that deletes biometric data per the schedule. + +These five gates are NOT in `AUDIT_TRAIL_PRD.md` §8 phase plan. **Recommend adding as Phase 1.6 (BIPA pre-launch gates)** ahead of any real-photo intake. + +--- + +## §2 — outcomes.jsonl content sample + +### 2A — File state + +| Property | Value | +|---|---| +| Rows | 101 | +| Path | `data/_kb/outcomes.jsonl` | +| Rows with `name` field in fills | **39** | +| Rows with `first_name` field | 0 (the field is `name`, not `first_name` — names are pre-concatenated in this sink) | +| Sample names in fills (synthetic but representative) | `Alex Rivera`, `Carmen I. Garcia`, `Jacob N. Patel`, `Jamal Z. Jones`, `James Park`, `Maria Chen`, `Mei Y. Cooper`, `Peter E. Carter`, `Sam Torres` | + +### 2B — Confirmed PII shapes in the file today + +- `fills[*].name` — full candidate name (synthetic but real shape) +- `operation` — natural-language fill request (e.g. "fill: Welder x2 in Toledo, OH"). Currently doesn't carry candidate names in the operation string itself, BUT the `error` field can — model reasoning that names specific candidates lands here +- `error` — model self-correction text and validator failure messages. Sampled rows contain phrases like "executor parse failure on turn 8: no JSON object in Executor response: ..." which is technical content. Real-data scenarios where the model reasons about specific candidates would put names here. + +### 2C — Discrimination-proxy phrase audit (gemini scrum flag §10/single-reviewer) + +Searched for common LLM-generated discrimination proxy phrases. **0 hits in current synthetic-data sample**: + +- "culture fit" — 0 +- "communication" — 0 +- "team chemistry" — 0 +- "personality" — 0 +- "soft skills" — 0 +- "work ethic" — 0 +- "professional attitude" — 0 + +**Interpretation:** the current synthetic data doesn't generate these phrases. However: +- The file is structurally capable of holding them (it persists model `error` text) +- When real models (kimi-k2.6, deepseek-v3.2 per PR #13) reason about real candidates, they WILL generate phrases like these — these are common in HR/staffing LLM output +- Once a phrase like "rejected as not a culture fit" lands in outcomes.jsonl, it's append-only persistence. Cryptographic erasure is the only RTBF path. + +**Action:** add a content-redaction pass to the outcomes.jsonl writer BEFORE production data lands. Strip discrimination-proxy phrases at write time. (This is not a substitute for the model not GENERATING them — it's a defense-in-depth layer.) + +### 2D — Implications for the audit trail + +The outcomes.jsonl sample confirms what `AUDIT_PHASE_1_DISCOVERY.md` §1D predicted: **39 of 101 rows persist candidate names in the `fills[*].name` field today**. The shape is real even if the data is synthetic. + +For the Phase 2 identity service design: + +1. **The training-safe export pipeline (per J 2026-05-03 answer 11) MUST address outcomes.jsonl directly.** This file is the most-likely candidate for RAG re-indexing or fine-tuning corpus, and it carries names. + +2. **Crypto-erasure target list** must include `data/_kb/outcomes.jsonl`, `data/_kb/overseer_corrections.jsonl`, `/tmp/lakehouse-validator/sessions.jsonl`, plus pathway memory state, plus Langfuse storage. + +3. **Subject_id top-level promotion (per AUDIT_PHASE_1_DISCOVERY §10/C5)** can replace `fills[*].name` as the queryable field. The name itself can move into a `[REDACTED]` token until legal access dereferences via the identity service. + +--- + +## §3 — Synthesis: net new findings for Phase 2 design + +| Finding | Source | Phase 2 design implication | +|---|---|---| +| **Phase 1.6 (NEW) — BIPA pre-launch gates needed before real photos accepted** | §1E above | Phase 2 identity service design must include biometric-info table with consent + retention metadata schema. | +| **Name-based ethnicity inference is a separate Title VII / disparate-impact risk** | §1B above | Phase 2 must include a "no protected-attribute inference at routing time" architectural rule. The lookup tables should be deprecated as part of the identity-service migration. | +| **deepface library use itself triggers BIPA when real photos arrive** | §1D point 5 | Phase 2 design must specify "biometric processing requires consent token presence" as a gateway boundary check. | +| **outcomes.jsonl persists names today (39/101 rows)** | §2A above | Phase 2 needs subject_id top-level promotion + name field as `[REDACTED-{token}]` reference (token resolves via identity service for authorized callers). | +| **No discrimination-proxy phrases in current synthetic data, but the sink is shape-ready for them** | §2C above | Phase 2 should specify a redaction-on-write pass for outcomes.jsonl + sessions.jsonl writers. | +| **manifest.jsonl is tracked in git today** | §1C above | If real photos arrive without changing the manifest path, biometric data lands in version control — must be addressed as part of Phase 1.6 gates. | + +--- + +## §4 — Open questions surfaced by Phase 1.5 + +1. **Where will real photos enter the system?** No upload endpoint walked. Need to know the entry point before designing the consent gate. +2. **Will the deepface tagging path be preserved for real photos, or stripped?** Architectural decision — affects whether biometric extraction is in scope at all. +3. **What's the staffing client's consent UX?** Does the candidate sign consent during onboarding, separately for biometrics, etc.? Affects the consent-token design. +4. **How long do BIPA records need to persist?** BIPA's "3 years from last interaction" is a maximum, not a target — staffing-co's contract may set a shorter floor (e.g., "delete on placement+1 year"). Need staffing-co policy. +5. **Is there a Designated Privacy Officer / DPO?** Per CCPA/GDPR best practice + BIPA litigation hygiene, this role should exist before real-PII intake. Out of scope for code; flag for J. + +--- + +## §5 — Recommended Phase 1.6 BIPA pre-launch gates (NEW) + +Lands as new phase BETWEEN Phase 1.5 (this doc) and Phase 2 (identity service design). Read-only/design work; one code commit (the consent gate). Estimated 4-8 hours. + +1. **Write `docs/BIPA_COMPLIANCE_POLICY.md`** — public retention schedule, written consent template, deletion procedure, employee training acknowledgment. Counsel must review before publication. +2. **Add consent gate to whatever endpoint accepts real photos** (not yet built — design with the gate from day 1, don't bolt on after). +3. **Quarantine real-photo classifications to `data/biometric/`** — separate path, separate retention, separate access control. Real-photo `manifest.jsonl` does NOT land in `data/headshots/`. +4. **Deprecate name → ethnicity lookup tables in `mcp-server/search.html`** (commit removing those constants + the call sites). Replace with "use candidate's actual photo" path or non-inferred default. +5. **Add a unit test that asserts `data/headshots/manifest.jsonl` row count matches the SYNTHETIC face count** — fails if real-photo classifications leak into the synthetic manifest. + +These five fit between Phase 1.5 and Phase 2 because (a) they're MORE URGENT than Phase 2 (BIPA exposure starts the moment a real photo touches deepface), (b) they're independent of the identity service design, (c) they need counsel sign-off before they can ship anyway. + +--- + +## Change log + +- 2026-05-03 — Phase 1.5 BIPA + outcomes content audit complete. New finding: synthetic data + deepface + name-ethnicity inference creates BIPA-shaped surface that is one upload path away from being live. Recommend Phase 1.6 (BIPA pre-launch gates) before Phase 2 design starts. -- 2.54.0 From 565ea4b32a9b14f05346e135f9398f2dbb1665b3 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:25:40 -0500 Subject: [PATCH 08/65] =?UTF-8?q?audit=20phase=202:=20IDENTITY=5FSERVICE?= =?UTF-8?q?=5FDESIGN.md=20=E2=80=94=20full=20design=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Incorporates J's confirmed answers (2026-05-03): - separate daemon (identityd) on :3225 / :4225 - signed JSON with PDF render for legal export - legal-only credential separate from admin token - Langfuse self-hosted (drops cross-border concern) - EU placeholder fields, not enforced - healthcare vertical routing — local-only models for healthcare PHI - training-safe export with hashed pseudonyms Plus Phase 1 + 1.5 findings + scrum-driven priorities: - UUID v7 candidate_id (drops kimi enumeration risk) - per-row encryption with per-subject keys (crypto-erasure target) - pii_access_log with Merkle-style integrity hash chain (FRE 901) - subject_id top-level promotion in all JSONL sinks - Langfuse boundary redaction layer (scrum C2 priority) - adverse-impact comparator pool in audit response (scrum C3) - BIPA-specific consent + retention metadata (scrum C4) - vertical detection at gateway boundary (J answer 10) Implementation single-language: Go (one identityd, both runtimes call it via HTTP). Postgres backing store, isolated schema. Master key in sealed file v1, vault migration path documented. 8-step migration path: stand up empty → backfill from parquet → behind feature flag → cut over reads incrementally → quarantine PII columns in workers_500k. Each step its own commit + gate + rollback. 6 open questions for J before implementation: master key location, Postgres shared vs isolated, vertical backfill default, legal token issuance procedure, crypto-erasure sweep cadence, EU enforcement timeline. Estimated 8-10 working days total. Largest single phase in the audit program. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/IDENTITY_SERVICE_DESIGN.md | 366 ++++++++++++++++++++++++++++++++ 1 file changed, 366 insertions(+) create mode 100644 docs/IDENTITY_SERVICE_DESIGN.md diff --git a/docs/IDENTITY_SERVICE_DESIGN.md b/docs/IDENTITY_SERVICE_DESIGN.md new file mode 100644 index 0000000..6988a6b --- /dev/null +++ b/docs/IDENTITY_SERVICE_DESIGN.md @@ -0,0 +1,366 @@ +# Identity Service — Phase 2 Design + +**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 +**Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md) + +> **Why this exists.** Phase 1 + 1.5 confirmed that today's substrate has no separation between candidate_id (the canonical token) and PII (name, email, phone, address). Both live in `workers_500k.parquet`. There is no per-access audit. There is no consent gate. There is no retention enforcement. This document specifies the new identity service that will hold the candidate_id ↔ PII mapping, gate every PII read, audit every access, and serve as the single legal-attestable boundary between PII and the rest of the system. +> +> **Confirmed by J 2026-05-03:** separate daemon (option A in §10.1), signed JSON with PDF render for legal export, legal-only auth credential separate from admin token. + +--- + +## 1. Scope and non-goals + +### In scope +- Single source of truth for `candidate_id ↔ PII` mapping +- Per-PII-access audit log (who/what/when/why) +- Consent + retention metadata (BIPA + Day and Temporary Labor Services Act + healthcare PHI) +- Legal-only access credential, separate from admin tokens +- Healthcare-vertical detection at gateway boundary (per J 2026-05-03 answer 10) +- EU-compatible interface (placeholder fields, lawful-basis tracking, SCC-ready slots — but NOT enforced this phase per J) +- Training-safe export interface (per J 2026-05-03 answer 11) +- Signed-JSON audit response with PDF render path + +### Out of scope (for this phase) +- The `/audit/subject/{id}` endpoint itself (Phase 3) +- Subject-tagging across other substrates (Phase 4) +- Right-to-be-forgotten implementation (Phase 7) +- BIPA pre-launch gates — those are Phase 1.6, ahead of this phase + +--- + +## 2. Architectural shape + +### 2.1 — Process model: separate daemon + +Per J's confirmation (2026-05-03), the identity service runs as its own daemon, port-isolated from the gateway. Rationale: + +- **Single attestable boundary** for legal/audit. "All PII access flows through identityd. Show me the identityd access log" is one query, one daemon. +- **Independent restart** — a gateway crash doesn't take down identity, and an identity panic doesn't break unrelated reads. +- **Distinct credential surface** — identityd's auth model is wholly separate from gateway's. The legal-only credential exists only in identityd, not in the gateway's JWT issuer. +- **Cross-runtime parity** — both Rust and Go gateway call identityd over HTTP. There is ONE identity implementation. + +| Property | Value | +|---|---| +| Name | `identityd` | +| Port | `:3225` (Rust legacy line — picks a port adjacent to validatord :3221) and `:4225` (Go line) | +| Implementation language | **Go** — single implementation, both runtimes call it via HTTP. Avoids re-implementing the audit-log writer + retention sweeper twice. | +| Storage | Postgres (separate database from any other lakehouse storage). Deployed alongside Langfuse's Postgres or its own; either way, isolated schema with its own grants. | +| Encryption | Per-row symmetric encryption (AES-256-GCM) of PII columns. Master key in a vault (HashiCorp Vault, AWS KMS, or a sealed-secret file at `/etc/lakehouse/identityd_master.key` for now). Keys are NEVER logged. | +| Backup | Standard Postgres backup; keys backed up separately to different storage tier (the cryptographic-erasure model in `AUDIT_TRAIL_PRD.md` §6 only works if the encrypted-blob backup and the key-backup are not co-located). | + +### 2.2 — Schema (Postgres DDL sketch) + +```sql +-- Single source of truth for the candidate_id ↔ PII mapping +-- Every PII column is stored as ciphertext; keys per row enable +-- per-subject crypto-erasure for RTBF. +CREATE TABLE subjects ( + candidate_id TEXT PRIMARY KEY, -- canonical token, e.g. "CAND-000001" + -- Encrypted PII fields. Each is AES-256-GCM with subject_key_id below. + -- Plaintext is NEVER stored. NULL means "not collected" not "absent." + name_ct BYTEA, + email_ct BYTEA, + phone_ct BYTEA, + address_ct BYTEA, + ssn_ct BYTEA, + dob_ct BYTEA, + -- Per-subject encryption key id. Crypto-erasure path: destroy this key + -- and the ciphertext is unrecoverable, even with the master key. + subject_key_id TEXT NOT NULL, + -- Lawful basis + consent metadata + consent_status TEXT NOT NULL, -- 'pending' | 'given' | 'withdrawn' | 'expired' + consent_version TEXT, -- references published consent template version + consent_given_at TIMESTAMPTZ, + consent_withdrawn_at TIMESTAMPTZ, + -- BIPA-specific fields (per Phase 1.5 §1E) + biometric_consent_status TEXT, -- separate from general PII consent + biometric_retention_until TIMESTAMPTZ, -- BIPA: max 3 years from last interaction + -- Vertical detection — drives healthcare PHI routing (per J answer 10) + vertical TEXT, -- 'general' | 'healthcare' | 'finance' | 'other' + -- EU-placeholder fields (per J answer 9 — present, not enforced) + eu_resident BOOLEAN DEFAULT FALSE, + lawful_basis TEXT, -- GDPR Art. 6 basis if eu_resident=true + transfer_mechanism TEXT, -- SCC, DPF, BCR — populated when EU comes online + -- Standard audit columns + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_interaction TIMESTAMPTZ, -- drives retention sweep + -- RTBF state + erased_at TIMESTAMPTZ, -- set when crypto-erasure executed + erasure_reason TEXT -- 'rtbf_request' | 'retention_expired' | 'consent_withdrawn' +); + +-- Append-only access audit. EVERY PII read writes a row here. +CREATE TABLE pii_access_log ( + access_id BIGSERIAL PRIMARY KEY, + candidate_id TEXT NOT NULL, + accessed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + accessor_kind TEXT NOT NULL, -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve' + accessor_id TEXT NOT NULL, -- daemon name + caller token-hash, never raw token + purpose TEXT NOT NULL, -- 'fill_validation' | 'audit_subject_response' | 'admin' | 'legal_audit_DDDDDD' + fields_accessed TEXT[] NOT NULL, -- ['name', 'email'] etc. + request_trace_id TEXT, -- ties to Langfuse trace + sessions.jsonl + integrity_hash TEXT NOT NULL -- chain hash for tamper-evidence (this row's hash includes prev row's hash) +); + +-- Cryptographic chain for the access log — Merkle-style. Per kimi +-- single-reviewer flag: chain of custody under FRE 901. +-- Each row's integrity_hash = SHA256(prev_hash || row_payload). +-- Last hash periodically committed to a tamper-evident store +-- (could be a separate append-only file with timestamp signing). + +-- Per-subject keys table — crypto-erasure target. +-- Destroying a row here makes the corresponding subjects.*_ct unreadable. +CREATE TABLE subject_keys ( + subject_key_id TEXT PRIMARY KEY, + candidate_id TEXT NOT NULL, + key_material BYTEA NOT NULL, -- AES-256 key, encrypted under master key + created_at TIMESTAMPTZ NOT NULL, + destroyed_at TIMESTAMPTZ, -- crypto-erasure marker + destroyed_reason TEXT +); + +-- Consent template versioning — BIPA + GDPR + CCPA compliance evidence +CREATE TABLE consent_versions ( + version TEXT PRIMARY KEY, + effective_at TIMESTAMPTZ NOT NULL, + superseded_at TIMESTAMPTZ, + template_text TEXT NOT NULL, + biometric_section TEXT, -- BIPA-specific clause + healthcare_section TEXT, -- HIPAA-specific clause + eu_section TEXT -- GDPR-specific clause (placeholder) +); +``` + +### 2.3 — HTTP surface + +Identityd exposes a small HTTP surface, all under `/v1/identity/`: + +| Method + Path | Purpose | Auth | +|---|---|---| +| `POST /v1/identity/subjects` | Create a new subject. Body: PII fields. Returns: candidate_id (server-generated, NOT sequential — UUID v7 to avoid the kimi enumeration risk). | Gateway/admin token | +| `GET /v1/identity/subjects/{candidate_id}` | Resolve PII for a candidate. Returns: requested fields only. EVERY call writes a `pii_access_log` row. | **Service-tier auth** — gateway can call but body indicates accessor purpose | +| `GET /v1/identity/subjects/{candidate_id}/full` | Return the complete subject record including consent + retention metadata + audit summary. **Legal-only credential.** Used by audit-response endpoint. | **Legal-only token** — separate credential, separate rotation | +| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn, with version. | Gateway/admin token | +| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure: destroy the subject_key_id, mark erased_at. Idempotent. | Legal-only token | +| `GET /v1/identity/access_log/{candidate_id}` | Return the per-subject access log for audit response. | Legal-only token | +| `POST /v1/identity/training_safe_export` | Returns identifier-stripped + name-redacted projection of subjects suitable for RAG/training. Logs a "system_resolve" access log row marking export. | Admin token + explicit purpose flag | +| `GET /v1/identity/health` | Liveness | None | + +### 2.4 — Auth model: legal-only credential + +The legal-only credential is materially different from the admin/gateway token: + +- Stored at `/etc/lakehouse/identityd_legal.token` (mode 0400, owner-only) +- Loaded by identityd at startup via systemd `EnvironmentFile` +- Never logged, never returned in any API response, never crossed with gateway tokens +- Rotation: separate runbook. Triggered by counsel request OR scheduled annually. +- The token's existence is documented in the privacy policy ("legal access requires a separate operator-issued credential, audited per access"). + +Service-tier auth (gateway-issued) and legal-tier auth (operator-issued) are orthogonal — a request must present the legal token to hit `/full`, `/erase`, or `/access_log`. Even an admin token does not unlock those. + +--- + +## 3. Integration with the rest of the substrate + +### 3.1 — Gateway changes + +When the gateway needs PII for a fill scenario (today this happens by SQL JOIN on `candidates`/`workers_500k`), the new flow is: + +1. Gateway has only `candidate_id` (post-§2 view-routing fix from `AUDIT_PHASE_1_DISCOVERY` §9) +2. Gateway calls `GET /v1/identity/subjects/{candidate_id}` with `purpose=fill_validation` and `fields=[name]` (or `[name,phone]` etc.) +3. Identityd writes pii_access_log row, decrypts the requested fields, returns them +4. Gateway uses the fields for the validator + tool result, then immediately drops them from in-memory storage after the request completes (no caching) + +**Critical:** the LRU embed cache (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Post-identity-service, the cache keying must change to `(model, candidate_id, field_subset_hash)` so the cache key itself is not PII-bearing. This is a Phase 4 task tracked separately. + +### 3.2 — Rust legacy + Go rewrite both call the same identityd + +Both gateways (Rust :3100 and Go :4110) call identityd over HTTP. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates that an identical PII request through both gateways produces identical identityd access-log rows (modulo daemon-name field). + +### 3.3 — outcomes.jsonl + sessions.jsonl writer changes + +Per `AUDIT_PHASE_1_DISCOVERY` §10/C5 (subject_id top-level promotion): + +- Change `outcomes.jsonl` writer to: + - Add top-level `subject_ids: ["CAND-000001", "CAND-000456"]` field listing every candidate referenced + - Strip `name` from `fills[*]` rows; replace with `name_ref: "[REDACTED-{candidate_id}]"` token + - Authorized callers dereference the token via identityd + +- Same for sessions.jsonl SessionRecord: add `subject_ids` top-level field. + +- Same for overseer_corrections.jsonl. + +- Same for observerd ops.jsonl when written. + +This makes every JSONL sink subject-queryable by `subject_id` directly, without grepping natural language. + +### 3.4 — Langfuse boundary redaction (per scrum priority C2) + +Before the gateway POSTs a chat trace to Langfuse, identityd is consulted to map any PII-shaped substrings in the message array back to candidate_id tokens. Implementation: + +- Gateway maintains a per-request map of `subject_id → temporarily_resolved_PII` for the lifetime of one request +- Before Langfuse POST, gateway iterates message content, replaces resolved PII with `[REDACTED-{candidate_id}]` +- Langfuse never sees raw names/emails/phones — it sees the tokens, which are unresolvable without a legal-tier identityd call +- For audit, legal counsel can use the token to dereference identity AND see the corresponding Langfuse trace, but Langfuse's storage is PII-free + +This addresses the most-dangerous-leak finding from `AUDIT_PHASE_1_DISCOVERY` §10/C2. + +### 3.5 — Healthcare vertical routing (per J answer 10) + +When `subjects.vertical = 'healthcare'`, the gateway routing rules change: + +- Tool calls that touch this candidate's data MUST route to local-only models (Ollama on-box), NOT to opencode/openrouter/ollama_cloud egress +- If a healthcare-vertical request can't be served locally, it fails with HTTP 451 ("Unavailable for Legal Reasons") — better to refuse than leak PHI +- The identity service holds the routing decision; the gateway consults it on every call +- Vertical detection itself happens at ingest time (`workers_500k` row metadata) OR when first PII fetch returns vertical='healthcare' + +This requires a one-line addition to the gateway's chat routing in `crates/gateway/src/v1/chat.rs` + Go-side equivalent in `cmd/chatd/main.go`. Both should fail-closed: if identityd is unreachable, healthcare requests refuse. + +### 3.6 — Training-safe export (per J answer 11) + +`POST /v1/identity/training_safe_export` returns a projection of subject decision data with: + +- `name`, `email`, `phone`, `address`, `ssn`, `dob` ALL stripped (replaced with `[REDACTED]`) +- `candidate_id` replaced with a hashed pseudonym specific to the export run (different export runs produce different pseudonyms — prevents cross-run correlation) +- Discrimination-proxy phrases (per gemini scrum) detected and `[REDACTED-PROXY]`-replaced +- Output is suitable for RAG-indexing or fine-tuning corpus building +- An audit log entry documents the export (purpose, requesting accessor, scope, fields) + +If a candidate later RTBFs, their pre-export decisions remain in the trained corpus BUT the link back to them is severed (the export pseudonym was random). Legal defense: "the source data was destroyed; the model retains it indistinguishably from synthetic patterns." + +--- + +## 4. Audit response: what /audit/subject/{id} returns (Phase 3 preview) + +Phase 3 builds the audit endpoint, but the shape it returns is dictated by what identityd can produce. Sketch: + +```json +{ + "schema": "audit.subject.v1", + "subject_token": "CAND-000001", + "request_window": { "from": "2026-01-01", "to": "2026-05-03" }, + "generated_at": "2026-05-03T12:00:00Z", + "generated_by": "identityd@hostname", + "integrity_hash": "sha256:...", // Merkle-style chain of all decision rows + "signature": "ed25519:...", // identityd signs with its escrow key + "consent": { + "status": "given", + "version": "v3-2026-04-15", + "given_at": "2026-04-20T14:30:00Z", + "biometric_consent": "given", + "biometric_retention_until": "2029-04-20T14:30:00Z" + }, + "decisions": [ + { + "ts": "2026-04-22T09:15:23Z", + "decision_kind": "fill_recommendation", + "daemon": "gateway", + "model": "kimi-k2.6", + "provider": "ollama_cloud", + "trace_id": "trace-abc", + "session_id": "session-xyz", + "input_features": { + // Sanitized view of what the model saw — no protected attributes, + // no inferred-attribute proxies, but enough to defend the decision + }, + "output": "recommended for fill_event_456", + "rationale": "Skills match: Welder TIG aluminum + 5+ years; geo match: Toledo OH; availability: confirmed", + "comparator_pool_size": 47, + "comparator_pool_protected_class_distribution": "see appendix A" // adverse-impact stats per gemini scrum + } + ], + "comparator_appendix": { + // EEOC adverse-impact statistics: for the same searches that included + // this subject, what was the selection rate by protected class? + // Aggregated; no other subjects' identifiers leak. + }, + "access_log": [ + // Every PII access for this subject in the request window + { "at": "...", "purpose": "fill_validation", "fields": ["name"], "trace_id": "..." } + ], + "footer": { + "completeness_attestation": "all decisions about subject_token in the window per retention policy v2 are included", + "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0", + "format_version": "audit.subject.v1" + } +} +``` + +PDF render is a downstream consumer — same JSON, different presentation layer (template + signing remains in JSON; PDF is for legal team's final delivery). + +--- + +## 5. Migration path from current state + +This is the single biggest implementation question — how to get from "PII in workers_500k.parquet, no identityd" to "PII in identityd, parquet has only candidate_id + non-PII columns" without breaking the live demo. + +### Migration strategy: parallel-write, gradual-read-cutover + +1. **Step 1 — Stand up identityd.** Empty database. New service, no callers yet. Health endpoint live. Rust + Go tests can call it but production paths don't. +2. **Step 2 — Backfill from workers_500k.parquet.** One-shot ETL: read parquet, for each row, write to identityd with `consent_status='inferred_existing'` (placeholder until counsel writes the real consent backfill story), `vertical='general'` (correct for non-healthcare data; needs human review for healthcare-flagged rows). +3. **Step 3 — Add identityd-call path to gateway behind a feature flag.** When `LH_USE_IDENTITY_SERVICE=true`, the gateway calls identityd for PII; otherwise it uses the legacy SQL path. +4. **Step 4 — Cut over reads incrementally.** Tool registry first (highest-PII-volume path). Validate via the cross-runtime parity probe `audit_parity.sh`. +5. **Step 5 — Quarantine PII columns in workers_500k.parquet.** Once all readers go through identityd, the parquet's PII columns become read-only and eventually moved to a different bucket. The candidate_id-only projection becomes the operational table. + +Each step has its own commit, its own gate, and its own rollback. Don't ship steps 2-5 in one commit. + +--- + +## 6. Cross-runtime parity probe (NEW) + +Per `AUDIT_PHASE_1_DISCOVERY` §4 ask: extend the 5 existing probes with `audit_parity.sh`. New probe asserts: + +1. Same PII fetch through Rust gateway (port 3100) and Go gateway (port 4110) produces identical identityd access-log rows (modulo daemon name) +2. Crypto-erasure of a test subject through Rust gateway is honored when Go gateway tries to fetch +3. Healthcare-vertical routing decision is identical across both runtimes +4. Training-safe export produces byte-identical output regardless of which gateway initiated + +Ships as part of Phase 5 (identity service build), not phase 2. + +--- + +## 7. What this design intentionally does NOT solve + +- **Does not replace existing protected-attribute exclusion at decision time.** The model still sees what the SQL returns; identityd doesn't filter that. Phase 6 of `AUDIT_TRAIL_PRD` handles boundary enforcement. +- **Does not redact pathway memory trace bodies.** That's per-trace-write redaction, separate concern. Phase 4. +- **Does not retroactively scrub Langfuse history.** Past traces still contain PII; only new traces are token-redacted. Counsel may request a one-shot historical-Langfuse purge — that's a separate runbook. +- **Does not implement "right to explanation" (GDPR Art. 22 / EU AI Act).** Audit response shows decisions; explaining the model's reasoning chain in human-readable form is Phase 8 (legal export format) or its own follow-up phase. +- **Does not handle multi-region data residency.** Single-region (US-Midwest, by default). EU-placeholder fields are present; multi-region deployment is out of scope. + +--- + +## 8. Open questions for J before implementation starts + +1. **Master key location.** Vault server, KMS, or a sealed file? Sealed file is fastest to ship; vault is most defensible. Recommend sealed-file for v1 with migration path to vault. **Confirm.** +2. **Postgres for identityd: shared with Langfuse, or its own?** Recommend its own — operational isolation. **Confirm.** +3. **`vertical` field initial values.** Backfill all existing subjects to `'general'`? Or block backfill until each candidate's vertical is determined? **Recommend backfill-to-general + flagging procedure for unknown.** +4. **Legal-only token issuance procedure.** Who has the authority to mint a legal token? Operator (J)? Outside counsel? Both? **Recommend J + named outside counsel, dual-control.** +5. **Crypto-erasure timeline for retention.** Default sweep cadence: daily? Weekly? **Recommend daily.** +6. **EU placeholder enforcement timeline.** Build the fields now; when do we turn on enforcement? **Recommend "when first EU candidate is added; until then, enforcement is no-op."** + +--- + +## 9. Estimated implementation cost + +| Sub-phase | Effort | Notes | +|---|---|---| +| 2A — Postgres schema + migrations | 4-6 hours | Includes encryption helpers + key management glue | +| 2B — identityd HTTP surface (Go) | 1-2 days | All endpoints, auth, signing key, tests | +| 2C — Backfill ETL from workers_500k.parquet | 1 day | One-shot script + dry-run mode | +| 2D — Gateway integration (Rust + Go, behind feature flag) | 2 days | Per-tool migration, parity probe | +| 2E — outcomes/sessions/observer JSONL writer changes | 1 day | Subject_id top-level promotion across all sinks | +| 2F — Langfuse redaction layer | 1-2 days | Per-request resolved-PII map + token replacement | +| 2G — Healthcare-vertical routing | 0.5 day | Single conditional per gateway | +| 2H — Training-safe export | 1 day | The exporter + audit logging | +| 2I — Cross-runtime parity probe `audit_parity.sh` | 0.5 day | New probe, lands in golangLAKEHOUSE | +| **Total** | **~8-10 working days** | Sequential; some can parallelize | + +This is the largest single phase in the audit-trail program. It's the substrate for everything downstream. Recommend doing it carefully — no half-shipped commits, each sub-phase has its own exit criterion. + +--- + +## Change log + +- 2026-05-03 — Initial Phase 2 design draft. Incorporates J's confirmed answers (separate daemon, signed JSON+PDF, legal-only auth) plus all Phase 1 + 1.5 findings + scrum-driven priority changes. -- 2.54.0 From 298fadce4117a175c55a57d5293dc705fdf4f4cf Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:36:07 -0500 Subject: [PATCH 09/65] =?UTF-8?q?identity=20service:=20v2=20=E2=80=94=20fo?= =?UTF-8?q?ld=20cross-lineage=20scrum=20findings=20+=204=20'would=20not=20?= =?UTF-8?q?build'=20blocker=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scrummed v1 across opus + kimi + gemini lineages via the new model fleet. 3/3 reviewers said 'I would NOT build v1 as written.' 4 convergent blockers, all resolved in v2: 1. Migration order wrong — backfill before validation creates dark database; if backfill bug, no production traffic catches it. v2 inserts BIPA-prereq Step 0 + shadow-write before backfill + shadow-read before cutover. 9-step migration with cryptographic attestation of completeness at quarantine. 2. Master key on disk + legal token static file = 'security theater' per all 3. v2: HashiCorp Vault Transit / AWS KMS for KEK (not sealed file). Legal token: split-secret short-lived JWT (max 24h), dual-control issuance (J + counsel both sign), revocable in <60s. 3. consent_status='inferred_existing' is BIPA prima facie violation (kimi+gemini explicit). v2 backfill uses 'pending_backfill_review'; biometric data NEVER backfilled — separate consent stream. 4. Healthcare default 'general' = HIPAA exposure window for every misclassified subject. v2 default 'unknown' with fail-closed routing (treat unknown as healthcare-equivalent until classified by manual review). Auto-escalation to healthcare on resume_text pattern match. Plus 12 single-reviewer additions: - mTLS mandatory between gateway↔identityd (kimi) - External anchor for audit chain: S3 Object Lock 7-year compliance mode, hourly + on-event commits (all 3) - Audit-log signing key separate from encryption KEK (opus) - Field-level authorization via purpose_definitions table (kimi) - Per-row encryption keys deferred to Phase 7 (kimi simplification) - pii_access_log itself needs legal-tier read auth (opus) - Synchronous cache invalidation pub/sub on RTBF (opus) - Outbound NER pass for Langfuse defense-in-depth (opus TOCTOU) - model_version_hash per decision row (gemini) - /vertical minimal-disclosure endpoint (kimi HIPAA min-necessary) - Auto-escalation healthcare on resume_text pattern (kimi) - Rate limiting + token revocation list (opus) - Oracle tests in audit_parity.sh (kimi SOC2 CC4.1) Architecturally simplified per scrum: - Per-row encryption keys deferred to Phase 7 (single DEK + HSM- wrapped KEK + ciphertext deletion is equivalent practical erasure with less complexity) - PDF render deferred (JSON ships first) - Training-safe export deferred (not critical path) Estimated effort revised 8-10 → 12-15 days. Worth it — every addition was a 3/3-reviewer convergent finding. Re-scrum recommended before implementation starts to verify v2 addresses the v1 blockers. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/IDENTITY_SERVICE_DESIGN.md | 552 +++++++++++++++++++------------- 1 file changed, 334 insertions(+), 218 deletions(-) diff --git a/docs/IDENTITY_SERVICE_DESIGN.md b/docs/IDENTITY_SERVICE_DESIGN.md index 6988a6b..5d85aab 100644 --- a/docs/IDENTITY_SERVICE_DESIGN.md +++ b/docs/IDENTITY_SERVICE_DESIGN.md @@ -1,11 +1,13 @@ -# Identity Service — Phase 2 Design +# Identity Service — Phase 2 Design (v2 — post-scrum revisions) -**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 +**Status:** Draft v2 — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03 **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md) -> **Why this exists.** Phase 1 + 1.5 confirmed that today's substrate has no separation between candidate_id (the canonical token) and PII (name, email, phone, address). Both live in `workers_500k.parquet`. There is no per-access audit. There is no consent gate. There is no retention enforcement. This document specifies the new identity service that will hold the candidate_id ↔ PII mapping, gate every PII read, audit every access, and serve as the single legal-attestable boundary between PII and the rest of the system. -> -> **Confirmed by J 2026-05-03:** separate daemon (option A in §10.1), signed JSON with PDF render for legal export, legal-only auth credential separate from admin token. +> **v2 history.** v1 (2026-05-03 morning) was scrummed across opus + kimi + gemini lineages. 3/3 reviewers converged on 6 critical issues — wrong migration order, master key + legal token "security theater," BIPA `inferred_existing` prima facie violation, healthcare default in wrong direction, Merkle chain without external anchor. v2 incorporates those changes. Full scrum reviews preserved at `/tmp/identity_scrum/{opus,kimi,gemini}_review.md`. Diff between v1 and v2 captured in §11 change log. + +> **Why this exists.** Phase 1 + 1.5 confirmed today's substrate has no separation between candidate_id and PII. Both live in `workers_500k.parquet`. No per-access audit, no consent gate, no retention enforcement. This document specifies the new identity service that holds the candidate_id ↔ PII mapping, gates every PII read, audits every access, and serves as the single legal-attestable boundary between PII and the rest of the system. + +> **Prerequisite:** **Phase 1.6 (BIPA pre-launch gates) MUST ship before any identityd backfill begins.** Per kimi+gemini scrum, backfilling with `consent_status='inferred_existing'` is a BIPA §15 prima facie violation. Phase 1.6 establishes the consent template + retention schedule + employee training that turns backfill from "inferred" into "pending_backfill_review with documented escalation path." --- @@ -14,151 +16,208 @@ ### In scope - Single source of truth for `candidate_id ↔ PII` mapping - Per-PII-access audit log (who/what/when/why) -- Consent + retention metadata (BIPA + Day and Temporary Labor Services Act + healthcare PHI) -- Legal-only access credential, separate from admin tokens -- Healthcare-vertical detection at gateway boundary (per J 2026-05-03 answer 10) -- EU-compatible interface (placeholder fields, lawful-basis tracking, SCC-ready slots — but NOT enforced this phase per J) -- Training-safe export interface (per J 2026-05-03 answer 11) -- Signed-JSON audit response with PDF render path +- Consent + retention metadata (BIPA + IL Day and Temporary Labor Services Act + healthcare PHI) +- Legal-tier access — short-lived JWT with **split-secret dual-control issuance** (NOT a static file token, per all 3 reviewers) +- Healthcare-vertical routing with **fail-closed default** (`unknown` treated as healthcare until classified, per opus+gemini) +- EU-compatible interface — fields exist, enforcement is per-subject (no system-wide flag flip needed) +- Hardware-backed master key — **HashiCorp Vault Transit OR AWS KMS** for v1 (NOT a sealed file, per all 3 reviewers; sealed-file v1 was "obfuscation in any defensible sense" — kimi) +- Signed-JSON audit response with PDF render path (PDF deferred; JSON ships first per opus) +- External-anchor for audit-log integrity (S3 Object Lock + signed timestamps; NOT just Postgres Merkle, per all 3 reviewers) +- mTLS or Unix Domain Socket between gateway ↔ identityd (per kimi — "port-isolated is theater without authenticated channel") +- Field-level authorization on `/subjects/{id}` (per-purpose field allowlists, per kimi) +- Synchronous cache invalidation hook for RTBF (per opus — even if Phase 7 is the full RTBF build) ### Out of scope (for this phase) - The `/audit/subject/{id}` endpoint itself (Phase 3) -- Subject-tagging across other substrates (Phase 4) -- Right-to-be-forgotten implementation (Phase 7) -- BIPA pre-launch gates — those are Phase 1.6, ahead of this phase +- Full subject-tagging across other substrates (Phase 4) +- Right-to-be-forgotten implementation full-flow (Phase 7) +- Training-safe export (deferred from Phase 2 per opus — "not on critical path for audit-trail defense") +- BIPA pre-launch gates content (Phase 1.6 — separate doc) + +### Architecturally simplified from v1 (per scrum) +- **Per-row encryption keys** deferred to Phase 7. v2 uses a single Data Encryption Key (DEK) wrapped under HSM-backed Key Encryption Key (KEK), with per-subject **ciphertext-deletion** as the v2 erasure mechanism. This matches kimi's recommendation: "A single data-encryption key with HSM-backed rotation, plus per-subject deletion of ciphertext, achieves equivalent practical erasure with far less complexity." Cryptographic-erasure via per-subject keys becomes a Phase 7 enhancement. +- **EU placeholder fields** kept (per J 2026-05-03), but enforcement code is genuinely no-op until first eu_resident=true subject — no code paths run, no schema migrations needed when EU comes online (just per-subject field population). +- **Merkle chaining** narrowed (per gemini): full Merkle tree only for legal-tier and consent/erasure events. Standard `gateway_lookup` events get a simpler signed-HMAC chain. Both anchored externally. --- ## 2. Architectural shape -### 2.1 — Process model: separate daemon +### 2.1 — Process model + transport -Per J's confirmation (2026-05-03), the identity service runs as its own daemon, port-isolated from the gateway. Rationale: - -- **Single attestable boundary** for legal/audit. "All PII access flows through identityd. Show me the identityd access log" is one query, one daemon. -- **Independent restart** — a gateway crash doesn't take down identity, and an identity panic doesn't break unrelated reads. -- **Distinct credential surface** — identityd's auth model is wholly separate from gateway's. The legal-only credential exists only in identityd, not in the gateway's JWT issuer. -- **Cross-runtime parity** — both Rust and Go gateway call identityd over HTTP. There is ONE identity implementation. +Per J 2026-05-03 + kimi scrum: separate daemon, mTLS-mandatory transport. Single Go implementation, both runtimes call it. | Property | Value | |---|---| | Name | `identityd` | -| Port | `:3225` (Rust legacy line — picks a port adjacent to validatord :3221) and `:4225` (Go line) | -| Implementation language | **Go** — single implementation, both runtimes call it via HTTP. Avoids re-implementing the audit-log writer + retention sweeper twice. | -| Storage | Postgres (separate database from any other lakehouse storage). Deployed alongside Langfuse's Postgres or its own; either way, isolated schema with its own grants. | -| Encryption | Per-row symmetric encryption (AES-256-GCM) of PII columns. Master key in a vault (HashiCorp Vault, AWS KMS, or a sealed-secret file at `/etc/lakehouse/identityd_master.key` for now). Keys are NEVER logged. | -| Backup | Standard Postgres backup; keys backed up separately to different storage tier (the cryptographic-erasure model in `AUDIT_TRAIL_PRD.md` §6 only works if the encrypted-blob backup and the key-backup are not co-located). | +| Port | `:3225` (single port, not dual; per kimi simplification — runtime-agnostic routing) | +| Implementation | Go | +| Storage | Postgres in **isolated database** (per J answer, confirmed). Schema-level isolation; no shared schemas with Langfuse or other lakehouse storage. | +| Transport gateway↔identityd | **mTLS mandatory.** Self-signed CA managed by identityd at startup; gateway clients have their own client certs. Plain HTTP is rejected at the listener. | +| Encryption-at-rest | **Single DEK wrapped under KEK in HashiCorp Vault Transit** (recommended) OR AWS KMS. Master key NEVER on disk. v1 of identityd refuses to start if KEK is unreachable. | +| Audit-log signing | **Separate Ed25519 keypair from the master encryption key.** Signing key in Vault Transit (signing backend) OR a separate sealed-secret file with strict rotation procedure. | +| External anchor for audit chain | **S3 Object Lock** (compliance mode, 7-year retention) holding periodic Merkle root commitments. Daemon writes hourly + on legal-tier event. | +| Backup | Postgres standard backup; KEK backup separate (different storage, different access). Crypto-erasure model only works if these are not co-located. | +| Rate limiting | Per-token QPS + daily-volume caps. Default legal-token: 100 lookups/day + 10K daily volume; alerting at 50%. Configurable per-token. | +| Token revocation | `token_revocations` table checked on every legal-tier auth. Revocation propagates within 60s (TTL on cache). | -### 2.2 — Schema (Postgres DDL sketch) +### 2.2 — Schema (Postgres DDL, v2) ```sql --- Single source of truth for the candidate_id ↔ PII mapping --- Every PII column is stored as ciphertext; keys per row enable --- per-subject crypto-erasure for RTBF. +-- Subject record. PII columns are ciphertext under the DEK. CREATE TABLE subjects ( - candidate_id TEXT PRIMARY KEY, -- canonical token, e.g. "CAND-000001" - -- Encrypted PII fields. Each is AES-256-GCM with subject_key_id below. - -- Plaintext is NEVER stored. NULL means "not collected" not "absent." + candidate_id TEXT PRIMARY KEY, -- UUID v7, NOT sequential (per kimi enumeration concern) + -- Encrypted PII fields. AES-256-GCM under DEK. NULL = not collected. name_ct BYTEA, email_ct BYTEA, phone_ct BYTEA, address_ct BYTEA, ssn_ct BYTEA, dob_ct BYTEA, - -- Per-subject encryption key id. Crypto-erasure path: destroy this key - -- and the ciphertext is unrecoverable, even with the master key. - subject_key_id TEXT NOT NULL, + -- DEK version this subject was encrypted under. KEK rotates; DEK rotates per + -- KEK rotation. New DEK version = subjects re-encrypted in background sweep. + dek_version INT NOT NULL, -- Lawful basis + consent metadata - consent_status TEXT NOT NULL, -- 'pending' | 'given' | 'withdrawn' | 'expired' + consent_status TEXT NOT NULL CHECK (consent_status IN ( + 'pending_backfill_review', -- backfill default; no biometric/PHI use until reviewed + 'pending_first_contact', -- new subject, awaiting consent UX + 'given', -- explicit consent recorded + 'withdrawn', -- subject revoked + 'expired' -- consent timeout + )), consent_version TEXT, -- references published consent template version consent_given_at TIMESTAMPTZ, consent_withdrawn_at TIMESTAMPTZ, - -- BIPA-specific fields (per Phase 1.5 §1E) - biometric_consent_status TEXT, -- separate from general PII consent + -- BIPA-specific fields (Phase 1.5 §1E + Phase 1.6 prerequisite) + biometric_consent_status TEXT NOT NULL DEFAULT 'never_collected' CHECK (biometric_consent_status IN ( + 'never_collected', 'pending', 'given', 'withdrawn', 'expired' + )), biometric_retention_until TIMESTAMPTZ, -- BIPA: max 3 years from last interaction - -- Vertical detection — drives healthcare PHI routing (per J answer 10) - vertical TEXT, -- 'general' | 'healthcare' | 'finance' | 'other' - -- EU-placeholder fields (per J answer 9 — present, not enforced) - eu_resident BOOLEAN DEFAULT FALSE, - lawful_basis TEXT, -- GDPR Art. 6 basis if eu_resident=true + -- Vertical detection — drives healthcare PHI routing. + -- DEFAULT 'unknown' (per opus+gemini scrum) — fail-closed routing treats + -- unknown as healthcare-equivalent until reclassified. + vertical TEXT NOT NULL DEFAULT 'unknown' CHECK (vertical IN ( + 'unknown', 'general', 'healthcare', 'finance', 'other' + )), + -- Per-vertical retention period. Drives the daily erasure sweep. + -- (Per opus: this was missing in v1. Required for BIPA's 3-year-from- + -- last-interaction rule which differs from generic retention.) + retention_period_days INT NOT NULL, + -- EU-placeholder fields. Enforcement is per-subject; nothing runs + -- until a row has eu_resident=true AND lawful_basis IS NOT NULL. + eu_resident BOOLEAN NOT NULL DEFAULT FALSE, + lawful_basis TEXT, -- GDPR Art. 6 basis when eu_resident=true transfer_mechanism TEXT, -- SCC, DPF, BCR — populated when EU comes online -- Standard audit columns created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), last_interaction TIMESTAMPTZ, -- drives retention sweep - -- RTBF state - erased_at TIMESTAMPTZ, -- set when crypto-erasure executed - erasure_reason TEXT -- 'rtbf_request' | 'retention_expired' | 'consent_withdrawn' + -- Crypto-erasure state (v2: ciphertext deletion, NOT key destruction) + erased_at TIMESTAMPTZ, + erasure_reason TEXT ); --- Append-only access audit. EVERY PII read writes a row here. +-- Append-only access log. EVERY PII read writes a row. +-- Field-level: fields_accessed records WHICH fields the caller resolved. +-- purpose enforced against an allowlist per-purpose-token. CREATE TABLE pii_access_log ( access_id BIGSERIAL PRIMARY KEY, candidate_id TEXT NOT NULL, accessed_at TIMESTAMPTZ NOT NULL DEFAULT now(), accessor_kind TEXT NOT NULL, -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve' - accessor_id TEXT NOT NULL, -- daemon name + caller token-hash, never raw token - purpose TEXT NOT NULL, -- 'fill_validation' | 'audit_subject_response' | 'admin' | 'legal_audit_DDDDDD' - fields_accessed TEXT[] NOT NULL, -- ['name', 'email'] etc. - request_trace_id TEXT, -- ties to Langfuse trace + sessions.jsonl - integrity_hash TEXT NOT NULL -- chain hash for tamper-evidence (this row's hash includes prev row's hash) + accessor_id TEXT NOT NULL, -- daemon name + caller token-hash; never raw token + purpose_token TEXT NOT NULL, -- opaque token; resolved through purpose_definitions table + fields_accessed TEXT[] NOT NULL, + request_trace_id TEXT, + chain_kind TEXT NOT NULL CHECK (chain_kind IN ('hmac', 'merkle')), -- HMAC for standard, Merkle for legal/consent/erasure + integrity_hash TEXT NOT NULL, + -- Per opus security finding: this row's existence is itself sensitive + -- ("candidate X is under legal review" leaks via purpose). Read access + -- to this table requires legal-tier auth or scoped per-subject auth. + is_legal_tier_event BOOLEAN NOT NULL DEFAULT FALSE ); --- Cryptographic chain for the access log — Merkle-style. Per kimi --- single-reviewer flag: chain of custody under FRE 901. --- Each row's integrity_hash = SHA256(prev_hash || row_payload). --- Last hash periodically committed to a tamper-evident store --- (could be a separate append-only file with timestamp signing). - --- Per-subject keys table — crypto-erasure target. --- Destroying a row here makes the corresponding subjects.*_ct unreadable. -CREATE TABLE subject_keys ( - subject_key_id TEXT PRIMARY KEY, - candidate_id TEXT NOT NULL, - key_material BYTEA NOT NULL, -- AES-256 key, encrypted under master key - created_at TIMESTAMPTZ NOT NULL, - destroyed_at TIMESTAMPTZ, -- crypto-erasure marker - destroyed_reason TEXT +-- Purpose definitions — enforces field-level authorization (per kimi). +-- A given purpose_token can only request fields in its allowlist. +CREATE TABLE purpose_definitions ( + purpose_token TEXT PRIMARY KEY, -- e.g. 'fill_validation', 'audit_subject_response' + description TEXT NOT NULL, + allowed_fields TEXT[] NOT NULL, -- e.g. ARRAY['name'] for fill_validation + auth_tier TEXT NOT NULL CHECK (auth_tier IN ('service', 'legal')), + rate_limit_qps INT, + daily_volume_cap INT ); --- Consent template versioning — BIPA + GDPR + CCPA compliance evidence +-- Token revocation list. Checked on every auth, cached 60s. +CREATE TABLE token_revocations ( + token_hash TEXT PRIMARY KEY, + revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(), + revoked_by TEXT NOT NULL, + reason TEXT +); + +-- Consent template versioning. Hash references, not embedded text per row. CREATE TABLE consent_versions ( version TEXT PRIMARY KEY, effective_at TIMESTAMPTZ NOT NULL, superseded_at TIMESTAMPTZ, - template_text TEXT NOT NULL, - biometric_section TEXT, -- BIPA-specific clause - healthcare_section TEXT, -- HIPAA-specific clause - eu_section TEXT -- GDPR-specific clause (placeholder) + template_hash TEXT NOT NULL, -- SHA256 of the canonical template + template_path TEXT NOT NULL -- where the canonical text lives (e.g. data/_consent/v3-2026-04-15.md) +); + +-- External anchor checkpoints. Periodically committed to S3 Object Lock. +CREATE TABLE anchor_commits ( + commit_id BIGSERIAL PRIMARY KEY, + chain_kind TEXT NOT NULL, -- 'hmac' or 'merkle' + checkpoint_at TIMESTAMPTZ NOT NULL, + last_access_id BIGINT NOT NULL, -- max access_id covered by this checkpoint + root_hash TEXT NOT NULL, + s3_object_uri TEXT NOT NULL, -- s3://anchor-bucket/identityd/... + s3_lock_until TIMESTAMPTZ NOT NULL -- compliance-mode retention end ); ``` -### 2.3 — HTTP surface +### 2.3 — HTTP surface (v2) -Identityd exposes a small HTTP surface, all under `/v1/identity/`: +All under `/v1/identity/`. mTLS + token required on every endpoint. -| Method + Path | Purpose | Auth | -|---|---|---| -| `POST /v1/identity/subjects` | Create a new subject. Body: PII fields. Returns: candidate_id (server-generated, NOT sequential — UUID v7 to avoid the kimi enumeration risk). | Gateway/admin token | -| `GET /v1/identity/subjects/{candidate_id}` | Resolve PII for a candidate. Returns: requested fields only. EVERY call writes a `pii_access_log` row. | **Service-tier auth** — gateway can call but body indicates accessor purpose | -| `GET /v1/identity/subjects/{candidate_id}/full` | Return the complete subject record including consent + retention metadata + audit summary. **Legal-only credential.** Used by audit-response endpoint. | **Legal-only token** — separate credential, separate rotation | -| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn, with version. | Gateway/admin token | -| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure: destroy the subject_key_id, mark erased_at. Idempotent. | Legal-only token | -| `GET /v1/identity/access_log/{candidate_id}` | Return the per-subject access log for audit response. | Legal-only token | -| `POST /v1/identity/training_safe_export` | Returns identifier-stripped + name-redacted projection of subjects suitable for RAG/training. Logs a "system_resolve" access log row marking export. | Admin token + explicit purpose flag | -| `GET /v1/identity/health` | Liveness | None | +| Method + Path | Purpose | Auth tier | Notes | +|---|---|---|---| +| `POST /v1/identity/subjects` | Create new subject. Body: PII fields + `purpose_token`. Returns: candidate_id (UUID v7). | service | Validates purpose_token allowlist matches submitted fields | +| `GET /v1/identity/subjects/{candidate_id}?purpose={token}&fields={list}` | Resolve PII for candidate. | service | **Field-level enforcement**: returned fields ⊆ purpose_token's allowed_fields. Mismatch → 403. Logs access row. | +| `GET /v1/identity/subjects/{candidate_id}/vertical` | **Minimal-disclosure** vertical lookup (per kimi HIPAA minimum-necessary). Returns only `{vertical, consent_status}`. | service | Used by gateway routing. Cheaper access-log row. | +| `GET /v1/identity/subjects/{candidate_id}/full` | Complete subject record + audit summary. | **legal** | Short-lived JWT only. Logged with `is_legal_tier_event=true`. Triggers real-time notification to designated counsel + J. | +| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn with version. | service | | +| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure. Idempotent. | **legal** | Synchronously triggers cache invalidation hooks (per opus). | +| `GET /v1/identity/access_log/{candidate_id}` | Per-subject access log for audit response. | **legal** | | +| `POST /v1/identity/auth/legal_token` | Issue short-lived JWT (max 24h). Requires **dual-control attestation**: J's signed nonce + counsel's signed nonce, both verified against pre-registered public keys. | dual-control | Per all 3 reviewers — replaces v1's static file approach | +| `GET /v1/identity/health` | Liveness | none | Returns only `{status: "ok"}` — no version, no schema info | +| `(no public training-safe-export endpoint in v2)` | — | — | Deferred per opus — not on critical path | -### 2.4 — Auth model: legal-only credential +### 2.4 — Auth model (v2 — split-secret + short-lived) -The legal-only credential is materially different from the admin/gateway token: +**Service-tier auth** (gateway → identityd for routine PII resolution): +- mTLS client cert per gateway (Rust :3100 + Go :4110 each get their own cert) +- Bearer token in `Authorization` header (long-lived — 90d), rotated per ops runbook -- Stored at `/etc/lakehouse/identityd_legal.token` (mode 0400, owner-only) -- Loaded by identityd at startup via systemd `EnvironmentFile` -- Never logged, never returned in any API response, never crossed with gateway tokens -- Rotation: separate runbook. Triggered by counsel request OR scheduled annually. -- The token's existence is documented in the privacy policy ("legal access requires a separate operator-issued credential, audited per access"). +**Legal-tier auth** (per-request, short-lived JWT, dual-control issuance): +- J holds private key A +- Designated outside counsel holds private key B +- `POST /v1/identity/auth/legal_token` requires: + - HTTP body: `{purpose, ttl_seconds, requested_fields, signature_a, signature_b, nonce_a, nonce_b}` + - identityd verifies both signatures against pre-registered A and B public keys + - On success: emits a JWT signed by identityd, valid for `ttl_seconds` (max 24h), scoped to `purpose`+`fields` +- The JWT: + - Cannot be issued without BOTH A and B signatures (per gemini "split-secret startup ceremony" pattern, applied per-token) + - Carries scope (purpose + fields) inside its claims; identityd enforces at use time + - Per-token rate limit + daily cap recorded on issuance + - Revocation: token_hash added to `token_revocations`; identityd refuses on next check (60s cache TTL) -Service-tier auth (gateway-issued) and legal-tier auth (operator-issued) are orthogonal — a request must present the legal token to hit `/full`, `/erase`, or `/access_log`. Even an admin token does not unlock those. +**Why this matters:** +- v1's "static file mode 0400" was security theater per all 3 reviewers +- A leaked legal token v1 = unbounded exfiltration window until rotation +- v2's leaked JWT = bounded by TTL (max 24h) AND revocable in <60s +- Single-actor compromise (J's machine OR counsel's machine, not both) cannot mint legal tokens --- @@ -166,201 +225,258 @@ Service-tier auth (gateway-issued) and legal-tier auth (operator-issued) are ort ### 3.1 — Gateway changes -When the gateway needs PII for a fill scenario (today this happens by SQL JOIN on `candidates`/`workers_500k`), the new flow is: +When gateway needs PII for a fill scenario: -1. Gateway has only `candidate_id` (post-§2 view-routing fix from `AUDIT_PHASE_1_DISCOVERY` §9) -2. Gateway calls `GET /v1/identity/subjects/{candidate_id}` with `purpose=fill_validation` and `fields=[name]` (or `[name,phone]` etc.) -3. Identityd writes pii_access_log row, decrypts the requested fields, returns them -4. Gateway uses the fields for the validator + tool result, then immediately drops them from in-memory storage after the request completes (no caching) +1. Gateway has only `candidate_id` (post-Phase-1.5 view-routing fix) +2. Gateway calls `GET /v1/identity/subjects/{candidate_id}/vertical` first if it might be healthcare-vertical-sensitive routing +3. Gateway calls `GET /v1/identity/subjects/{candidate_id}?purpose=fill_validation&fields=name` +4. identityd validates: purpose_token=`fill_validation` allows `[name]` only; if request asks for `[name,ssn]` → 403 +5. identityd writes `pii_access_log` row, decrypts requested fields, returns +6. Gateway uses fields, then **explicitly zeros memory** (kimi's drop-and-overwrite pattern) before returning HTTP response +7. **No PII caching** in gateway memory beyond request lifetime -**Critical:** the LRU embed cache (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Post-identity-service, the cache keying must change to `(model, candidate_id, field_subset_hash)` so the cache key itself is not PII-bearing. This is a Phase 4 task tracked separately. +**LRU embed cache** (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Phase 4 task to re-key as `(model, candidate_id, field_subset_hash)`. **Phase 2 must add the synchronous cache-purge hook** so RTBF in Phase 2 invalidates Phase 4's cache when it ships (opus finding). -### 3.2 — Rust legacy + Go rewrite both call the same identityd +### 3.2 — Cross-runtime: same identityd, both gateways call it -Both gateways (Rust :3100 and Go :4110) call identityd over HTTP. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates that an identical PII request through both gateways produces identical identityd access-log rows (modulo daemon-name field). +Both gateways call identityd over mTLS. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates identical PII request through both gateways produces identical access-log rows (modulo daemon-name field). -### 3.3 — outcomes.jsonl + sessions.jsonl writer changes +### 3.3 — JSONL writer changes (subject_id top-level promotion) -Per `AUDIT_PHASE_1_DISCOVERY` §10/C5 (subject_id top-level promotion): +Per `AUDIT_PHASE_1_DISCOVERY` §10/C5: +- All JSONL sinks (outcomes, sessions, overseer_corrections, observerd ops) gain top-level `subject_ids: [...]` field +- `fills[*].name` replaced with `name_ref: "[REDACTED-{candidate_id}]"` +- Authorized callers dereference via identityd -- Change `outcomes.jsonl` writer to: - - Add top-level `subject_ids: ["CAND-000001", "CAND-000456"]` field listing every candidate referenced - - Strip `name` from `fills[*]` rows; replace with `name_ref: "[REDACTED-{candidate_id}]"` token - - Authorized callers dereference the token via identityd +### 3.4 — Langfuse boundary redaction (defense in depth, per opus TOCTOU finding) -- Same for sessions.jsonl SessionRecord: add `subject_ids` top-level field. +v1 design: per-request map of `subject_id → resolved_PII`, replace before Langfuse POST. -- Same for overseer_corrections.jsonl. +**v2 adds** (per opus): outbound regex/NER pass on the Langfuse payload as defense-in-depth. The model can hallucinate names not in the resolved-PII map. The outbound NER pass catches them. -- Same for observerd ops.jsonl when written. +``` +Gateway constructs Langfuse payload + → Pass 1: per-request resolved-PII map replacement + → Pass 2: NER scan (regex for SSN/phone shapes; named-entity model for names) + → Either pass detecting unredacted PII → drop the trace OR replace with [POSSIBLE-PII-DETECTED] + → POST to Langfuse +``` -This makes every JSONL sink subject-queryable by `subject_id` directly, without grepping natural language. +### 3.5 — Healthcare vertical routing — fail-closed default -### 3.4 — Langfuse boundary redaction (per scrum priority C2) +**Per opus + gemini scrum: Default is `unknown` → treated as healthcare for routing purposes.** -Before the gateway POSTs a chat trace to Langfuse, identityd is consulted to map any PII-shaped substrings in the message array back to candidate_id tokens. Implementation: +When gateway receives a request involving any candidate: +1. Query `/vertical` for the candidate +2. If `vertical IN ('healthcare', 'unknown')` → route to **on-box Ollama only** (no opencode/openrouter/ollama_cloud egress) +3. If `vertical='general'` (or other non-healthcare) → cloud routing OK +4. If identityd unreachable → fail closed (refuse the request, return 503) -- Gateway maintains a per-request map of `subject_id → temporarily_resolved_PII` for the lifetime of one request -- Before Langfuse POST, gateway iterates message content, replaces resolved PII with `[REDACTED-{candidate_id}]` -- Langfuse never sees raw names/emails/phones — it sees the tokens, which are unresolvable without a legal-tier identityd call -- For audit, legal counsel can use the token to dereference identity AND see the corresponding Langfuse trace, but Langfuse's storage is PII-free +**Reclassification path:** subjects can be moved from `unknown` → `general` via explicit operator action (after manual review). Subjects flagged `healthcare` stay healthcare unless explicitly downgraded. -This addresses the most-dangerous-leak finding from `AUDIT_PHASE_1_DISCOVERY` §10/C2. +**Auto-escalation** (per kimi): if a subject's resume_text or call_log content is updated with healthcare-pattern matches (RN, BSN, hospital, MD, physician, etc.), vertical auto-escalates to `healthcare`. Never auto-de-escalates. -### 3.5 — Healthcare vertical routing (per J answer 10) +### 3.6 — Cache invalidation hook (Phase 2 must ship even though full RTBF is Phase 7) -When `subjects.vertical = 'healthcare'`, the gateway routing rules change: +When `POST /v1/identity/subjects/{id}/erase` fires: +1. Mark `subjects.erased_at` and zero out ciphertext columns +2. Write `pii_access_log` row with `purpose_token='retention_expired'` or `'rtbf_request'` +3. **Synchronously** publish a `subject_erased` event to a pub/sub channel (Redis or Postgres LISTEN/NOTIFY) +4. Gateway subscribes; on event, purges any in-flight cache entries for that candidate_id +5. Eventual-consistency window: <5s between erase-call and cache-flush -- Tool calls that touch this candidate's data MUST route to local-only models (Ollama on-box), NOT to opencode/openrouter/ollama_cloud egress -- If a healthcare-vertical request can't be served locally, it fails with HTTP 451 ("Unavailable for Legal Reasons") — better to refuse than leak PHI -- The identity service holds the routing decision; the gateway consults it on every call -- Vertical detection itself happens at ingest time (`workers_500k` row metadata) OR when first PII fetch returns vertical='healthcare' - -This requires a one-line addition to the gateway's chat routing in `crates/gateway/src/v1/chat.rs` + Go-side equivalent in `cmd/chatd/main.go`. Both should fail-closed: if identityd is unreachable, healthcare requests refuse. - -### 3.6 — Training-safe export (per J answer 11) - -`POST /v1/identity/training_safe_export` returns a projection of subject decision data with: - -- `name`, `email`, `phone`, `address`, `ssn`, `dob` ALL stripped (replaced with `[REDACTED]`) -- `candidate_id` replaced with a hashed pseudonym specific to the export run (different export runs produce different pseudonyms — prevents cross-run correlation) -- Discrimination-proxy phrases (per gemini scrum) detected and `[REDACTED-PROXY]`-replaced -- Output is suitable for RAG-indexing or fine-tuning corpus building -- An audit log entry documents the export (purpose, requesting accessor, scope, fields) - -If a candidate later RTBFs, their pre-export decisions remain in the trained corpus BUT the link back to them is severed (the export pseudonym was random). Legal defense: "the source data was destroyed; the model retains it indistinguishably from synthetic patterns." +Without this hook, RTBF in Phase 7 would be a lie because the gateway's LRU embed cache could still hold the subject's data. --- -## 4. Audit response: what /audit/subject/{id} returns (Phase 3 preview) - -Phase 3 builds the audit endpoint, but the shape it returns is dictated by what identityd can produce. Sketch: +## 4. Audit response shape (Phase 3 preview, v2 — adds model-version snapshot per gemini) ```json { "schema": "audit.subject.v1", - "subject_token": "CAND-000001", + "subject_token": "01926f2e-7c1b-7000-...", // UUID v7, NOT CAND-NNNNNN "request_window": { "from": "2026-01-01", "to": "2026-05-03" }, "generated_at": "2026-05-03T12:00:00Z", "generated_by": "identityd@hostname", - "integrity_hash": "sha256:...", // Merkle-style chain of all decision rows - "signature": "ed25519:...", // identityd signs with its escrow key - "consent": { - "status": "given", - "version": "v3-2026-04-15", - "given_at": "2026-04-20T14:30:00Z", - "biometric_consent": "given", - "biometric_retention_until": "2029-04-20T14:30:00Z" - }, + "merkle_root": "sha256:...", // root hash of legal/consent/erasure events + "external_anchor": "s3://anchor-bucket/identityd/2026-05-03T12-00.json", // S3 Object Lock URI + "signature": "ed25519:...", // separate signing key from encryption KEK + "consent": { ... }, "decisions": [ { "ts": "2026-04-22T09:15:23Z", "decision_kind": "fill_recommendation", "daemon": "gateway", "model": "kimi-k2.6", + "model_version_hash": "sha256:...", // NEW per gemini — proves what model existed AT decision time "provider": "ollama_cloud", "trace_id": "trace-abc", "session_id": "session-xyz", - "input_features": { - // Sanitized view of what the model saw — no protected attributes, - // no inferred-attribute proxies, but enough to defend the decision - }, - "output": "recommended for fill_event_456", - "rationale": "Skills match: Welder TIG aluminum + 5+ years; geo match: Toledo OH; availability: confirmed", + "input_features": { ... }, // sanitized; no protected attributes; no inferred-attribute proxies + "output": "...", + "rationale": "...", "comparator_pool_size": 47, - "comparator_pool_protected_class_distribution": "see appendix A" // adverse-impact stats per gemini scrum + "comparator_appendix_ref": "see comparator_appendix.A" } ], "comparator_appendix": { - // EEOC adverse-impact statistics: for the same searches that included - // this subject, what was the selection rate by protected class? - // Aggregated; no other subjects' identifiers leak. + "A": { + "scope": "fill scenarios in window matching role X, geo Y", + "total_pool_size": 47, + "selection_rate_by_protected_class": { + // Aggregated; NO other subjects' identifiers leak + "race": { "white": 0.33, "black": 0.31, ... }, + "gender": { "man": 0.34, "woman": 0.30, ... } + }, + "four_fifths_test": "passed" // or "concern: rate ratio 0.71" + } }, - "access_log": [ - // Every PII access for this subject in the request window - { "at": "...", "purpose": "fill_validation", "fields": ["name"], "trace_id": "..." } - ], + "access_log": [...], "footer": { - "completeness_attestation": "all decisions about subject_token in the window per retention policy v2 are included", - "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0", - "format_version": "audit.subject.v1" + "completeness_attestation": "all decisions about subject_token in window per retention policy v2 are included", + "merkle_proof": "...", // proof this audit's root is in S3 anchor + "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0" } } ``` -PDF render is a downstream consumer — same JSON, different presentation layer (template + signing remains in JSON; PDF is for legal team's final delivery). +--- + +## 5. Migration path (v2 — REORDERED per all 3 reviewers) + +v1's order was wrong. New order, with explicit prerequisites: + +| Step | Action | Prerequisite | +|---|---|---| +| **0** | **Phase 1.6 BIPA pre-launch gates SHIPPED** — consent template published, retention schedule public, deletion procedure documented, employee training acknowledged. | (separate work — see PHASE_1_6_BIPA_GATES.md when written) | +| **1** | Stand up identityd with **synthetic test subjects only**. KEK in vault, mTLS live, all endpoints serve. | Vault/KMS available; mTLS CA bootstrapped | +| **2** | **Audit-parity probe** (`audit_parity.sh`) green on synthetic data. Cross-runtime equivalence verified. | Step 1 | +| **3** | Add gateway feature flag `LH_USE_IDENTITY_SERVICE`. **Shadow-write only** — gateway writes new subjects to identityd AND continues SQL path. No reads from identityd yet. | Step 2 | +| **4** | Run shadow-write for ≥1 week. Validate access logs, encryption correctness, write-path performance under real traffic. | Step 3 | +| **5** | Backfill from `workers_500k.parquet`. **`consent_status='pending_backfill_review'`** for ALL existing rows. **`vertical='unknown'`** default (NOT 'general'). **`biometric_consent_status='never_collected'`** — backfill does NOT include any biometric data. | Step 0 (BIPA gates) + Step 4 | +| **6** | Human-review queue for vertical reclassification. Subjects move from `unknown` to `general` only after explicit review. Healthcare-pattern matches auto-escalate to `healthcare` (never auto-downgrade). | Step 5 | +| **7** | **Shadow-read** — gateway reads from BOTH SQL and identityd, compares, logs divergences, returns SQL result. Run ≥1 week. | Step 5 | +| **8** | Feature-flag cutover — gateway reads from identityd, falls back to SQL on error, alerts on every fallback. | Step 7 | +| **9** | Quarantine PII columns in workers_500k.parquet — only after **cryptographic attestation of identityd completeness** (Merkle proof: source row hash = identityd row hash for every candidate_id). Move PII columns to a different bucket; the candidate_id-only projection becomes operational. | Step 8 | + +**Key v2 changes from v1:** +- New Step 0 (BIPA gates prerequisite) — backfill cannot start without this +- Steps 3-4 (shadow-write before backfill) — production validates the WRITE path before data lands +- Step 5 backfill consent: `pending_backfill_review` not `inferred_existing` (BIPA defense) +- Step 5 vertical default: `unknown` not `general` (HIPAA fail-closed) +- New Step 6 (human review) — vertical classification via explicit operator action +- Step 7 added (shadow-read after backfill) — catches encoding/normalization bugs before cutover +- Step 9 (quarantine) requires cryptographic attestation of completeness + +Each step is its own commit, its own gate, its own rollback path. --- -## 5. Migration path from current state +## 6. Cross-runtime parity probe (NEW, per scrum §6 plus opus 'oracle test' addition) -This is the single biggest implementation question — how to get from "PII in workers_500k.parquet, no identityd" to "PII in identityd, parquet has only candidate_id + non-PII columns" without breaking the live demo. +`audit_parity.sh` ships in Phase 5. Asserts: -### Migration strategy: parallel-write, gradual-read-cutover - -1. **Step 1 — Stand up identityd.** Empty database. New service, no callers yet. Health endpoint live. Rust + Go tests can call it but production paths don't. -2. **Step 2 — Backfill from workers_500k.parquet.** One-shot ETL: read parquet, for each row, write to identityd with `consent_status='inferred_existing'` (placeholder until counsel writes the real consent backfill story), `vertical='general'` (correct for non-healthcare data; needs human review for healthcare-flagged rows). -3. **Step 3 — Add identityd-call path to gateway behind a feature flag.** When `LH_USE_IDENTITY_SERVICE=true`, the gateway calls identityd for PII; otherwise it uses the legacy SQL path. -4. **Step 4 — Cut over reads incrementally.** Tool registry first (highest-PII-volume path). Validate via the cross-runtime parity probe `audit_parity.sh`. -5. **Step 5 — Quarantine PII columns in workers_500k.parquet.** Once all readers go through identityd, the parquet's PII columns become read-only and eventually moved to a different bucket. The candidate_id-only projection becomes the operational table. - -Each step has its own commit, its own gate, and its own rollback. Don't ship steps 2-5 in one commit. - ---- - -## 6. Cross-runtime parity probe (NEW) - -Per `AUDIT_PHASE_1_DISCOVERY` §4 ask: extend the 5 existing probes with `audit_parity.sh`. New probe asserts: - -1. Same PII fetch through Rust gateway (port 3100) and Go gateway (port 4110) produces identical identityd access-log rows (modulo daemon name) -2. Crypto-erasure of a test subject through Rust gateway is honored when Go gateway tries to fetch -3. Healthcare-vertical routing decision is identical across both runtimes -4. Training-safe export produces byte-identical output regardless of which gateway initiated - -Ships as part of Phase 5 (identity service build), not phase 2. +1. Same PII fetch through Rust gateway and Go gateway produces identical identityd access-log rows (modulo daemon name) +2. Crypto-erasure of test subject through Rust gateway is honored when Go gateway tries to fetch +3. Healthcare-vertical routing decision identical across both runtimes +4. **Oracle test** (per kimi SOC2 CC4.1 finding): probe with known-good inputs against expected outputs. A bug present in BOTH implementations must not pass the parity probe. +5. Discrimination-proxy phrase redaction triggers as expected on adversarial test cases --- ## 7. What this design intentionally does NOT solve -- **Does not replace existing protected-attribute exclusion at decision time.** The model still sees what the SQL returns; identityd doesn't filter that. Phase 6 of `AUDIT_TRAIL_PRD` handles boundary enforcement. -- **Does not redact pathway memory trace bodies.** That's per-trace-write redaction, separate concern. Phase 4. -- **Does not retroactively scrub Langfuse history.** Past traces still contain PII; only new traces are token-redacted. Counsel may request a one-shot historical-Langfuse purge — that's a separate runbook. -- **Does not implement "right to explanation" (GDPR Art. 22 / EU AI Act).** Audit response shows decisions; explaining the model's reasoning chain in human-readable form is Phase 8 (legal export format) or its own follow-up phase. -- **Does not handle multi-region data residency.** Single-region (US-Midwest, by default). EU-placeholder fields are present; multi-region deployment is out of scope. +- Protected-attribute exclusion at decision time (Phase 6) +- Pathway memory trace body redaction (Phase 4) +- Retroactive Langfuse history scrub (separate runbook) +- GDPR Art. 22 right to explanation full implementation (Phase 8) +- Multi-region data residency (single-region US-Midwest by default) +- Training-safe export (deferred — not on critical path per opus) --- -## 8. Open questions for J before implementation starts +## 8. Open questions — RESOLVED 2026-05-03 -1. **Master key location.** Vault server, KMS, or a sealed file? Sealed file is fastest to ship; vault is most defensible. Recommend sealed-file for v1 with migration path to vault. **Confirm.** -2. **Postgres for identityd: shared with Langfuse, or its own?** Recommend its own — operational isolation. **Confirm.** -3. **`vertical` field initial values.** Backfill all existing subjects to `'general'`? Or block backfill until each candidate's vertical is determined? **Recommend backfill-to-general + flagging procedure for unknown.** -4. **Legal-only token issuance procedure.** Who has the authority to mint a legal token? Operator (J)? Outside counsel? Both? **Recommend J + named outside counsel, dual-control.** -5. **Crypto-erasure timeline for retention.** Default sweep cadence: daily? Weekly? **Recommend daily.** -6. **EU placeholder enforcement timeline.** Build the fields now; when do we turn on enforcement? **Recommend "when first EU candidate is added; until then, enforcement is no-op."** +J confirmed all 6 v1 recommendations + scrum-driven changes: + +1. ✅ **Master key:** HashiCorp Vault Transit (recommended) OR AWS KMS. NOT a sealed file. v1's sealed-file recommendation is rejected per all 3 reviewers ("obfuscation in any defensible sense"). +2. ✅ **Postgres isolation:** identityd's own database, isolated schema. +3. ✅ **Vertical backfill:** `'unknown'` default with fail-closed routing. NOT `'general'` per opus+gemini scrum. +4. ✅ **Legal token:** Split-secret dual-control issuance (J + counsel both sign). Short-lived JWT, max 24h, revocable in <60s. +5. ✅ **Crypto-erasure sweep:** Daily 03:00 UTC. (Note: v2 erasure mechanism is ciphertext deletion, not key destruction. Per-row keys deferred to Phase 7.) +6. ✅ **EU enforcement:** Per-subject. Schema fields exist; nothing runs until first eu_resident=true subject. + +### Newly resolved per scrum + +7. ✅ **Migration order:** REORDERED per §5 above. New Step 0 (BIPA prerequisite). Shadow-write before backfill. Shadow-read before cutover. +8. ✅ **Audit-log external anchor:** S3 Object Lock with compliance-mode 7-year retention. Hourly + on-event commits. +9. ✅ **Audit-log signing key:** Separate Ed25519 keypair from KEK. Vault Transit signing backend OR sealed-secret with strict rotation runbook. +10. ✅ **mTLS gateway↔identityd:** Mandatory. Self-signed CA managed by identityd at startup. +11. ✅ **Per-row encryption keys:** Deferred to Phase 7. v2 uses single DEK with HSM-wrapped KEK + ciphertext deletion for erasure. +12. ✅ **Field-level authorization:** purpose_definitions table enforces per-purpose field allowlists. +13. ✅ **Synchronous cache invalidation:** pub/sub event on erase; gateway subscribes. +14. ✅ **Outbound NER pass for Langfuse:** Defense-in-depth in addition to symbol-table replacement. +15. ✅ **Model version hash in audit response:** Captured per decision row. +16. ✅ **PDF render:** Deferred from Phase 2; JSON ships first. --- -## 9. Estimated implementation cost +## 9. Estimated implementation cost (revised v2) | Sub-phase | Effort | Notes | |---|---|---| -| 2A — Postgres schema + migrations | 4-6 hours | Includes encryption helpers + key management glue | -| 2B — identityd HTTP surface (Go) | 1-2 days | All endpoints, auth, signing key, tests | -| 2C — Backfill ETL from workers_500k.parquet | 1 day | One-shot script + dry-run mode | -| 2D — Gateway integration (Rust + Go, behind feature flag) | 2 days | Per-tool migration, parity probe | -| 2E — outcomes/sessions/observer JSONL writer changes | 1 day | Subject_id top-level promotion across all sinks | -| 2F — Langfuse redaction layer | 1-2 days | Per-request resolved-PII map + token replacement | -| 2G — Healthcare-vertical routing | 0.5 day | Single conditional per gateway | -| 2H — Training-safe export | 1 day | The exporter + audit logging | -| 2I — Cross-runtime parity probe `audit_parity.sh` | 0.5 day | New probe, lands in golangLAKEHOUSE | -| **Total** | **~8-10 working days** | Sequential; some can parallelize | +| 2A — Postgres schema + Vault/KMS integration | 1.5-2 days | Includes mTLS CA bootstrap + signing-key separation | +| 2B — identityd HTTP surface (Go) | 2-3 days | All endpoints, auth, dual-control JWT issuance, rate limiting, revocation | +| 2C — Backfill ETL (BIPA-compliant — pending_backfill_review) | 1 day | Plus Merkle attestation of completeness | +| 2D — Gateway integration (Rust + Go, mTLS, shadow-write phase) | 2-3 days | Per-tool migration, parity probe | +| 2E — JSONL writer changes (subject_id top-level promotion) | 1 day | All sinks | +| 2F — Langfuse redaction (symbol-table + NER) | 2 days | Two passes + drop-on-detect | +| 2G — Healthcare-vertical routing (fail-closed) | 0.5 day | Plus auto-escalation pattern matcher | +| 2H — Cache invalidation pub/sub hook | 0.5 day | Critical for Phase 7 RTBF | +| 2I — External anchor (S3 Object Lock) | 1 day | Hourly + on-event commits | +| 2J — Cross-runtime parity probe with oracle tests | 1 day | New probe | +| **Total** | **~12-15 working days** | (v1 estimated 8-10; v2 added BIPA gates dependency, mTLS, dual-control JWT, NER pass, anchor, etc.) | -This is the largest single phase in the audit-trail program. It's the substrate for everything downstream. Recommend doing it carefully — no half-shipped commits, each sub-phase has its own exit criterion. +Bigger than v1. Worth it — every addition was a 3/3-reviewer convergent finding. + +--- + +## 10. The four "would not build" blockers from scrum (all addressed in v2) + +| # | v1 issue (3/3 reviewers) | v2 resolution | +|---|---|---| +| 1 | Migration order (backfill before validation) | §5 reordered; Step 0 BIPA gates prereq added | +| 2 | Master key on disk + legal token static file | §2.1 Vault/KMS for KEK; §2.4 split-secret short-lived JWT | +| 3 | `inferred_existing` BIPA prima facie violation | §5 Step 5 uses `pending_backfill_review`; Step 0 gates required first | +| 4 | Healthcare default `general` (HIPAA exposure window) | §3.5 fail-closed `unknown`-as-healthcare; §5 Step 5 backfill default `unknown` | + +All three reviewers said "I would not build v1 as written." All four blockers are resolved in v2. Re-scrum recommended before implementation starts. + +--- + +## 11. Change log (v1 → v2) + +| Section | v1 | v2 | +|---|---|---| +| Master key storage | Sealed file `/etc/lakehouse/identityd_master.key` | HashiCorp Vault Transit / AWS KMS | +| Legal token | Static file mode 0400 | Split-secret short-lived JWT (max 24h), dual-control issuance | +| Encryption keys | Per-row keys with master wrapping | Single DEK + HSM-wrapped KEK + ciphertext deletion (per-row deferred to Phase 7) | +| Healthcare default | `vertical='general'` backfill | `vertical='unknown'` backfill, fail-closed routing | +| Migration §5 order | 1: stand up, 2: backfill, 3: feature flag, 4: cutover, 5: quarantine | 0: BIPA gates, 1: stand up, 2: parity probe, 3: shadow-write, 4: shadow-write soak, 5: backfill (pending_backfill_review), 6: human vertical review, 7: shadow-read, 8: cutover, 9: quarantine with cryptographic attestation | +| Audit-log integrity | Postgres Merkle chain only | Postgres Merkle for legal/consent/erasure + HMAC for standard + S3 Object Lock external anchor | +| Audit-log signing key | Same as encryption key | Separate Ed25519 keypair; Vault signing backend | +| Gateway↔identityd transport | Plain HTTP (implied) | mTLS mandatory | +| Field authorization | Per-endpoint only | Per-purpose-token field allowlists | +| Cache invalidation | Phase 4 / Phase 7 | Phase 2 ships pub/sub hook | +| Langfuse redaction | Symbol-table replacement only | Symbol-table + outbound NER pass + drop-on-detect | +| Audit response | No model version | model_version_hash per decision | +| PDF render | Phase 2 | Deferred — JSON ships first | +| Training-safe export | Phase 2 | Deferred — not critical path | +| Effort estimate | 8-10 days | 12-15 days | --- ## Change log -- 2026-05-03 — Initial Phase 2 design draft. Incorporates J's confirmed answers (separate daemon, signed JSON+PDF, legal-only auth) plus all Phase 1 + 1.5 findings + scrum-driven priority changes. +- 2026-05-03 — v1 initial draft. +- 2026-05-03 — v2 post-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum before implementation recommended. -- 2.54.0 From 8129ddd8834f58faefd55c2f7864460877125e95 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:39:35 -0500 Subject: [PATCH 10/65] =?UTF-8?q?identity=20service:=20v3=20amendments=20?= =?UTF-8?q?=E2=80=94=20second-pass=20scrum=20BUILD-WITH-CHANGES?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-scrummed v2 across opus + kimi + gemini. All 3 verdict: BUILD-WITH-CHANGES. v1 blockers verified RESOLVED. 12 new v2 findings folded as v3 amendments in §12. Convergent v2 findings (≥2 reviewers): v3-A1: mTLS CA root must NOT live in identityd (opus + gemini). v3 fix: Vault PKI for CA, identityd as intermediate. v3-A2: Dual-control public key registry must be tamper-evident (opus + gemini). v3 fix: Vault KV with separate access policies + server-issued nonces for replay protection. Single-reviewer v3 amendments (10 more): - B1: Step 8 fallback-to-SQL needs explicit 14-day time bound - B2: NER drop-on-detect needs Prometheus alerting - B3: legal-tier notification transport spec'd (signed Slack/email, no PII in body, failure non-blocking) - B4: Step 6 human review SLA flagged — ~7 months at 500/day for ~100k unknown rows; operational decision needed - B5: Memory zeroing in Go is best-effort (Rust uses zeroize crate); documented as not cryptographic-grade - B6: purpose_definitions needs versioning + emergency revocation (purpose_versions + purpose_revocations tables) - B7: Cache invalidation needs erasure_generation atomicity (subjects.erasure_generation int; gateway rejects stale-gen cache hits) — replaces best-effort pub/sub - B8: 15-min cooling-off period for dual-control issuance to prevent emergency-bypass culture - B9: NER calibrated test set with target recall ≥99.5% on synthetic adversarial PII - B10: S3 Object Lock in separate AWS account with write-only IAM; root credentials held by external party - B11: BIPA infrastructure-as-notice attestation in Phase 1.6 doc - B12: Backup retention vs ciphertext-deletion erasure window documented in RTBF runbook Estimate revised v2 12-15d → v3 17-20d. Worth it — the cost is what buys "I would build this" from 3 independent senior security architects across 3 model lineages. Must-have v3 items (block implementation): A1, A2, B1, B6, B7, B11. Should-have (ship in Phase 5 if calendar tight): B2-B5, B8-B10, B12. Re-scrum NOT recommended for v3 — diminishing returns; must-have items are concrete fixes with clear acceptance criteria. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/IDENTITY_SERVICE_DESIGN.md | 94 ++++++++++++++++++++++++++++++++- 1 file changed, 93 insertions(+), 1 deletion(-) diff --git a/docs/IDENTITY_SERVICE_DESIGN.md b/docs/IDENTITY_SERVICE_DESIGN.md index 5d85aab..2231fab 100644 --- a/docs/IDENTITY_SERVICE_DESIGN.md +++ b/docs/IDENTITY_SERVICE_DESIGN.md @@ -476,7 +476,99 @@ All three reviewers said "I would not build v1 as written." All four blockers ar --- +## 12. v3 amendments (post-second-pass scrum, 2026-05-03 evening) + +v2 was re-scrummed across opus + kimi + gemini. **All 3 verdict: BUILD-WITH-CHANGES.** All 4 v1 blockers verified RESOLVED. New v2 findings are tractable design fixes (not re-architecture). Folded in below. Reviews preserved at `/tmp/identity_scrum_v2/{opus,kimi,gemini}_review.md`. + +### Convergent v2 findings (≥2 reviewers) — must fix before implementation + +**v3-A1 — mTLS CA root must NOT live in identityd** (opus + gemini converge). v2 said "self-signed CA managed by identityd at startup" — if identityd is both the CA and authenticated party, compromise of identityd compromises the whole trust fabric. **v3 change:** mTLS CA root lives in **Vault PKI** (or, if Vault PKI is unavailable, an offline root with identityd as a properly-issued intermediate). identityd never has root-CA private key access. + +**v3-A2 — Dual-control public key registry must be tamper-evident** (opus + gemini converge). v2 said "pre-registered public keys" but didn't say where. If they live in identityd's Postgres, DB-write compromise defeats dual-control. **v3 change:** J + counsel public keys live in **Vault KV with separate access policies** (or a signed config file at `/etc/lakehouse/identityd_dual_control.yaml` with its own dual-control rotation procedure). Public-key changes themselves require dual-control attestation. Plus: `nonce_a`/`nonce_b` get **server-issued challenges** (identityd issues a fresh nonce per request; clients sign that nonce; replay-protection via 5-min nonce cache). + +### Single-reviewer v2 findings — also folded into v3 + +**v3-B1 (opus): Step 8 fallback-to-SQL needs explicit time bound.** Otherwise becomes permanent dual-read debt. v3: fallback active for max 14 days post-cutover; alert on every fallback; auto-disable at 14 days regardless of fallback rate. + +**v3-B2 (opus): NER drop-on-detect needs alerting.** Silent observability gap if model regression starts hallucinating PII. v3: drop counter exposed as `identityd_ner_drops_total` Prometheus metric; alert on drop_rate >0.1% over rolling 1h. + +**v3-B3 (opus): legal-tier notification transport must be specified.** v2 said "real-time notification to designated counsel + J" without saying how. v3: notification transport is **signed Slack webhook OR signed email** (configurable per deployment); message body never contains candidate_id or PII; only `{event_kind, timestamp, accessor_kind, integrity_hash}`. Notification failure does NOT block the legal-tier action — failure is logged for follow-up but token issuance proceeds. + +**v3-B4 (opus): Step 6 human review queue needs SLA + volume estimate.** With 500k backfilled rows defaulting to `unknown`, all healthcare-adjacent traffic routes to on-box Ollama until reclassified. v3: budget assumption — auto-pattern-matching pre-classifies ~80% to confidently-non-healthcare; remaining 20% (~100k) stays `unknown` pending review at ~500/day operator throughput = ~7 months of queue. **Operational decision needed:** either staff the queue, accept the on-box-Ollama-only routing for ~7 months, or relax pattern matching at the cost of higher false-negative rate. Flag for J. + +**v3-B5 (opus): Memory zeroing in Go is non-trivial.** v2 §3.1 step 6 says "explicitly zeros memory" — Go GC-managed strings are immutable, not zeroable. v3 implementation note: PII handling code path uses `[]byte` throughout; convert to string only at the JSON serialization boundary; explicit `runtime.GC()` after request completes is theater (Go won't actually zero the slice). **Acceptance:** "best-effort zero" — overwrite the `[]byte` slice contents post-use. Document this is best-effort, not cryptographic-grade scrubbing. (Rust-side: use the `zeroize` crate which IS cryptographic-grade.) + +**v3-B6 (kimi): Service-tier purpose_definitions needs versioning + emergency revocation.** A misconfigured purpose with overly-broad `allowed_fields` is standing exfiltration authorization. v3 schema additions: + +```sql +CREATE TABLE purpose_versions ( + purpose_token TEXT NOT NULL, + version INT NOT NULL, + effective_at TIMESTAMPTZ NOT NULL, + superseded_at TIMESTAMPTZ, + PRIMARY KEY (purpose_token, version) +); + +CREATE TABLE purpose_revocations ( + purpose_token TEXT PRIMARY KEY, + revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(), + revoked_by TEXT NOT NULL, + reason TEXT +); +``` + +Auth path checks `purpose_revocations` on every call (not cached, OR cached <5s). Revocation is <60s without code deploy. + +**v3-B7 (kimi): Cache invalidation needs erasure-generation atomicity.** v2 §3.6 pub/sub is best-effort. If Redis drops the message or gateway crashes between erase + purge, cache holds stale PII. v3 change: +- `subjects` table gains `erasure_generation INT NOT NULL DEFAULT 0`. Increments on every erase. +- Identityd PII responses include current generation in headers +- Gateway cache entries tagged with the generation they were filled at +- Cache hits with `cached_generation < current_generation` are rejected, force re-fetch from identityd +- Eventually-consistent within ONE request hop, not eventually-consistent over time + +**v3-B8 (kimi): Dual-control issuance needs cooling-off period to prevent bypass culture.** v3: 15-minute mandatory delay between nonce submission and token issuance, with notification to BOTH parties at delay-start AND issuance. Bypass requires both J and counsel to sign an "emergency bypass" attestation that's logged separately + escalated to a board-level reviewer. + +**v3-B9 (kimi): NER must have calibrated false-negative rate.** v3: maintain a synthetic adversarial test set (50 hallucinated-PII examples + 50 legitimate non-PII). NER must achieve recall ≥99.5% on the synthetic set in CI; production drop-rate >0.1% triggers re-calibration. + +**v3-B10 (kimi): S3 Object Lock root credentials need separate AWS account.** Same ops team holding Vault admin + AWS root = limited independence. v3: Object Lock bucket lives in **separate AWS account** with **write-only IAM role** issued to identityd. Root credentials for that account held by named external party (could be outside counsel) with quarterly attestation. + +**v3-B11 (kimi): biometric_consent_status='never_collected' may not satisfy BIPA infrastructure-as-notice argument.** Plaintiffs may argue presence of biometric schema fields constitutes constructive notice of intent. v3: Phase 1.6 BIPA policy doc must explicitly attest "no biometric data exists in the pre-identityd substrate" with cryptographic evidence (hash of pre-identityd schema). Counsel must bless this attestation. + +**v3-B12 (gemini): Backup retention window vs ciphertext-deletion erasure.** Phase 2 uses ciphertext deletion + single-DEK; if Postgres backups retain pre-erasure ciphertext + the DEK still exists, backup is recoverable. v3 erasure runbook: includes maximum backup retention period (e.g., 30 days). RTBF status reported to subject as "erased now; residual in backups expires by {date}." For full crypto-erasure-on-demand, Phase 7 per-row keys is the path. + +### Net v3 effort delta + +v2 estimate: 12-15 days. v3 amendments add: +- mTLS CA via Vault PKI integration: +0.5 day +- Public-key registry in Vault KV: +0.5 day +- Server-issued nonces for dual-control: +0.25 day +- Step 8 fallback time bound: +0.1 day +- NER drop-rate metric + alert: +0.25 day +- Legal-tier notification transport: +0.5 day +- purpose_versions + purpose_revocations: +0.5 day +- erasure_generation cache atomicity: +0.5 day +- 15-min cooling-off: +0.25 day +- NER calibrated test set: +0.5 day +- Separate AWS account for Object Lock: +0.5 day +- BIPA infrastructure-as-notice attestation: +0.25 day (mostly doc work) +- Backup-retention erasure runbook: +0.25 day + +**Total v3 delta: ~5 days. Revised estimate: 17-20 days.** + +The cost is real. The reason it's worth paying is what those 5 days buy: a design that 3 independent senior security architects (across 3 model lineages) all said they would build. v1 said "do not build." v3 says "build." + +### What's v3-deferred vs v3-must-have + +**v3 must-have (block implementation):** v3-A1, v3-A2, v3-B1, v3-B6, v3-B7, v3-B11. + +**v3 should-have (ship in Phase 2 if calendar allows; otherwise Phase 5):** v3-B2, v3-B3, v3-B4 (operational decision), v3-B5 (best-effort), v3-B8, v3-B9, v3-B10, v3-B12. + +If schedule pressure forces a cut, the should-have items can ship in Phase 5 (identity service build completion). The must-haves cannot — they're integral to the security boundary. + +--- + ## Change log - 2026-05-03 — v1 initial draft. -- 2026-05-03 — v2 post-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum before implementation recommended. +- 2026-05-03 — v2 post-first-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum recommended. +- 2026-05-03 — v3 post-second-scrum: 3/3 BUILD-WITH-CHANGES. v1 blockers verified resolved. 12 new v2 findings folded into v3 amendments (§12). Re-scrum NOT required for v3 — diminishing returns; the must-have items are concrete fixes with clear acceptance criteria. -- 2.54.0 From cd440d4cee9a4b2c9e3520f4e8c2dc1f020e13c5 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:41:29 -0500 Subject: [PATCH 11/65] =?UTF-8?q?audit=20phase=201.6:=20BIPA=20pre-launch?= =?UTF-8?q?=20gates=20=E2=80=94=20block=20identity-service=20backfill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per IDENTITY_SERVICE_DESIGN v3 §5 Step 0, Phase 1.6 is hard prerequisite to identityd backfill. This doc specifies the 5 gates + 2 supporting deliverables that must ship before real-photo intake. Five gates (BIPA §15 compliance): 1. Public retention schedule — counsel writes; engineering files+hash 2. Informed written consent — counsel writes template; engineering wires identityd consent-status enforcement 3. Photo-upload endpoint with consent enforcement — POST /v1/identity/ subjects/{id}/photo with hard 403 when biometric_consent_status != 'given'; quarantined storage path; deepface output isolated to identityd subjects table (not synthetic-face manifest) 4. Deprecate name → ethnicity inference (mcp-server/search.html lookup tables removed; Phase 1.5 §1B finding closed) 5. Destruction runbook — operator-facing; ties to identityd /erase endpoint with biometric-specific erasure path; daily sweep job for biometric_retention_until expiry Plus: - Cryptographic attestation that no biometric data exists pre-identityd (per v3-B11) — defends against infrastructure-as-notice plaintiff argument - Employee BIPA-handling training acknowledgment Engineering effort: ~4-5 days (one week to stage everything ready). Counsel effort: ~3-6 weeks calendar (review cycles dominate). Calendar bottleneck is counsel, not engineering. Phase 1.6 exit = 7 checked gates + signoffs. Until done, identityd backfill cannot proceed (per identity service design v3 §5 Step 0). 5 open questions for J + counsel: photo-upload UX, consent mechanism (DocuSign/click/paper), named operator list, named counsel for sign-off, public privacy policy URL. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/PHASE_1_6_BIPA_GATES.md | 240 +++++++++++++++++++++++++++++++++++ 1 file changed, 240 insertions(+) create mode 100644 docs/PHASE_1_6_BIPA_GATES.md diff --git a/docs/PHASE_1_6_BIPA_GATES.md b/docs/PHASE_1_6_BIPA_GATES.md new file mode 100644 index 0000000..2d57524 --- /dev/null +++ b/docs/PHASE_1_6_BIPA_GATES.md @@ -0,0 +1,240 @@ +# Phase 1.6 — BIPA Pre-Launch Gates + +**Status:** Draft — 2026-05-03 · **Owner:** J + outside counsel · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md), [`IDENTITY_SERVICE_DESIGN.md`](IDENTITY_SERVICE_DESIGN.md) + +> **Why this exists.** `IDENTITY_SERVICE_DESIGN.md` v3 §5 Step 0 names Phase 1.6 as a HARD PREREQUISITE: identityd backfill cannot start until Phase 1.6 ships. This doc specifies what Phase 1.6 contains. +> +> **Scope.** BIPA (740 ILCS 14) compliance gates that must be in place BEFORE the system accepts a single real candidate photo. Synthetic-data face pool can keep operating; real-photo intake CANNOT begin without these gates. +> +> **Authority.** This is an engineering scaffold. Sections marked `⚖ COUNSEL` need outside counsel to author the actual legally-binding text. Engineering ships the procedural gates; counsel writes the words. + +--- + +## 1. The five BIPA pre-launch gates + +Each gate is a deliverable that must ship before real-photo intake. None is optional. Order shown is the recommended ship sequence. + +### Gate 1 — Public retention schedule (BIPA §15(a)) + +**Required:** A publicly-available, written retention schedule for biometric identifiers and information. + +**What ships:** +- `data/_consent/biometric_retention_schedule_v1.md` — public file +- Linked from public privacy policy at the deployment URL +- Specifies: + - Categories of biometric data collected (facial geometry derived from candidate photos, age estimate, gender classification, race classification — per Phase 1.5 deepface walk) + - Purpose of collection (identity matching for staffing operations) + - Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin) + - Destruction procedure: per Gate 5 below +- Versioned (this is v1; future updates supersede with a new version) + +**⚖ COUNSEL** — write the actual schedule. Engineering provides the operational facts; counsel writes the binding language. + +**Engineering acceptance:** the file is committed, the public URL renders it, and identityd's `consent_versions` table references it by hash. + +--- + +### Gate 2 — Informed written consent (BIPA §15(b)) + +**Required:** Informed, written consent BEFORE any biometric collection occurs. + +**What ships:** +- `data/_consent/biometric_consent_template_v1.md` — public consent template +- Versioned, hashed, referenced from identityd's `consent_versions` table +- Must disclose, per BIPA §15(b)(1)-(3): + 1. That biometric identifiers/information will be collected + 2. The specific purpose for collection (and the length of term — references Gate 1) + 3. Receipt of a written release authorizing collection +- Consent flow at intake: + - Candidate sees the disclosure on a UI surface (web form / paper / digital signature) + - Candidate provides explicit affirmative action (signature, click-acceptance with timestamp, etc.) + - Identityd records `biometric_consent_status='given'` with `consent_version` reference + `consent_given_at` timestamp + - **Without identityd recording 'given', no biometric data flows through deepface.** + +**⚖ COUNSEL** — write the consent template. Recommended content (engineering view): +- Clear language (not just legal boilerplate) +- Specific to facial-classification (not generic biometrics) +- Includes withdrawal procedure +- Includes data-subject rights enumeration + +**Engineering acceptance:** consent gate is enforced in code at the photo-upload endpoint; identityd refuses biometric writes when `biometric_consent_status != 'given'`; pre-existing synthetic-face pool is exempt (no consent needed because no real subject). + +--- + +### Gate 3 — Photo-upload endpoint with consent enforcement + +**Required:** Code-level enforcement that real-photo intake checks consent before processing. + +**What ships:** + +A new endpoint (proposed: `POST /v1/identity/subjects/{candidate_id}/photo`) with the following behavior: + +1. Caller authenticates with service-tier token +2. Endpoint queries identityd for `subjects.biometric_consent_status` +3. If status ≠ `'given'` → HTTP 403 with reason `"BIPA consent required before biometric processing"` +4. If status = `'given'`: + a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`) + b. deepface tagging runs against the photo + c. Classifications (gender, race, age) stored to `subjects` table fields (NEW columns — see schema additions below) + d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule + e. `pii_access_log` row written with `purpose_token='biometric_collection'` +5. Response: `{candidate_id, retention_until, consent_version}` + +**Schema additions to identityd `subjects`:** + +```sql +ALTER TABLE subjects ADD COLUMN biometric_classifications JSONB; -- {gender, race, age} from deepface +ALTER TABLE subjects ADD COLUMN biometric_data_path TEXT; -- quarantined path +ALTER TABLE subjects ADD COLUMN biometric_collected_at TIMESTAMPTZ; +ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the photo bytes (for integrity, NOT for re-derivation) +``` + +**Engineering acceptance:** +- Endpoint refuses uploads when consent missing (verified by integration test) +- deepface output never lands in the synthetic-face manifest (`data/headshots/manifest.jsonl`) +- Real-photo classifications are isolated to identityd `subjects` table — never flow to JSONL sinks +- The `/headshots/:key` route in mcp-server REMAINS synthetic-only — does NOT serve real candidate photos to LLMs without an explicit allowance (proposed: real photos served only to authenticated staffer UI, never to model context) + +--- + +### Gate 4 — Deprecate name → ethnicity inference + +**Required:** The hard-coded `NAMES_HISPANIC` / `SURNAMES_*` lookup tables in `mcp-server/search.html:3375-3432` (per Phase 1.5 §1B walk) get removed. + +**What ships:** +- A code commit that removes: + - `FEMALE_NAMES`, `MALE_NAMES` constants + - `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` constants + - `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` constants + - The `genderFor()` and `guessEthnicityFromFirstName()` functions + - All call sites that consumed these (face-pool bucket selection) +- Replacement strategy: + - For SYNTHETIC face pool routing: deterministic hash of candidate_id selects a face bucket, no demographic inference + - For REAL candidate photos: the candidate's actual photo IS the representation; no inference needed + +**Why this is BIPA + Title VII risk separately:** name-based ethnicity classification is BOTH a discriminatory feature engineering practice (Title VII) AND, when combined with photo-based attribute extraction, a "biometric information derived from a biometric identifier" pattern (BIPA broad reading). Removing the lookup tables forecloses both arguments. + +**Engineering acceptance:** +- Lookup tables removed from search.html +- Unit test asserts no protected-attribute inference functions exist in search.html or any mcp-server module +- Face-pool routing for synthetic faces uses candidate_id hash exclusively +- Phase 1.5 §1B finding closed + +--- + +### Gate 5 — Documented destruction procedure + +**Required:** A written procedure for biometric data destruction at retention expiry OR consent withdrawal OR right-to-be-forgotten request. + +**What ships:** +- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing +- Specifies: + - Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request + - Procedure: identityd `POST /v1/identity/subjects/{id}/erase` (legal-tier auth) + - Erasure scope: `subjects.biometric_*` columns ciphertext-deleted, `biometric_data_path` files securely overwritten + unlinked, deepface classifications nulled + - Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed + - Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction) + - Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request + +**⚖ COUNSEL** — review the runbook for legal sufficiency. Engineering writes the procedure; counsel attests that the procedure satisfies BIPA §15(a) destruction requirements. + +**Engineering acceptance:** +- Runbook committed +- `POST /v1/identity/subjects/{id}/erase` endpoint includes biometric-specific erasure path +- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock) +- Erasure events are logged with cryptographic attestation + +--- + +## 2. Cryptographic attestation: no biometric data exists pre-identityd + +**Per `IDENTITY_SERVICE_DESIGN` v3-B11.** Plaintiffs may argue that the EXISTENCE of biometric schema fields constitutes constructive notice of intent to collect biometric data — therefore consent should have preceded the schema. The defense: prove that no biometric data was actually collected from real candidates before identityd + the consent gate. + +**What ships:** +- A one-shot script `scripts/staffing/attest_pre_identityd_biometric_state.sh` that: + - Queries `data/datasets/workers_500k.parquet` schema and confirms NO column named `photo`, `biometric_*`, `face_*`, `image_*` exists + - Greps `data/_kb/*.jsonl` and `data/_pathway_memory/state.json` for any base64-encoded image bytes (deepface output, photo blobs) + - Verifies `data/headshots/manifest.jsonl` rows ≤ synthetic face pool size + - Hashes the schema + summary; commits the hash to S3 Object Lock (per identity service v3 anchor pattern) +- Attestation document `docs/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-XX.md` signed by J + outside counsel + +**This is a one-time defense artifact.** It establishes the baseline: "as of this date, no biometric data was collected from real candidates." + +--- + +## 3. Employee training acknowledgment (general BIPA hygiene) + +**Required:** People with access to biometric data acknowledge BIPA-handling training. + +**What ships:** +- `docs/policies/BIPA_HANDLING_TRAINING_v1.md` — training material covering: + - What constitutes biometric identifiers / information + - The consent + retention procedures + - Destruction obligations + - Reporting suspected exposure +- Acknowledgment record per individual (initially: J + counsel + named operators) +- Annual refresh + +**⚖ COUNSEL** — write training content. Engineering doesn't author legal-compliance training. + +--- + +## 4. Phase 1.6 exit criteria (gates Phase 2 backfill) + +All 5 gates must be DONE before identityd backfill begins: + +1. ✅ Public retention schedule published + linked from privacy policy + counsel sign-off +2. ✅ Consent template published + counsel sign-off + technical enforcement integrated +3. ✅ Photo-upload endpoint shipped with consent enforcement + integration test green +4. ✅ Name → ethnicity inference removed from search.html + unit test asserting absence +5. ✅ Destruction runbook published + erasure endpoint includes biometric path + counsel sign-off + +PLUS: + +6. ✅ Cryptographic attestation that no pre-identityd biometric data exists, signed by J + counsel +7. ✅ Employee training material published + initial acknowledgments recorded + +Until all 7 are checked off, **identity service backfill (Phase 2 §5 Step 5) cannot proceed.** + +--- + +## 5. Effort estimate + +| Gate | Engineering effort | Legal effort | +|---|---|---| +| Gate 1 (retention schedule) | 0.5 day | counsel-dependent (typically 1-2 weeks for review) | +| Gate 2 (consent template) | 0.5 day | counsel-dependent (typically 2-4 weeks for review and consent UX design) | +| Gate 3 (photo-upload endpoint) | 1-2 days | review of endpoint behavior | +| Gate 4 (deprecate name-ethnicity inference) | 0.5 day | none (engineering-only fix) | +| Gate 5 (destruction runbook) | 1 day | counsel sign-off | +| §2 cryptographic attestation | 0.5 day | counsel + J signature | +| §3 employee training | 0.25 day (admin) | counsel-authored content | +| **Total engineering** | **~4-5 days** | — | +| **Total counsel** | — | **~3-6 weeks calendar** (review cycles) | + +**The calendar bottleneck is counsel, not engineering.** Engineering can stage all 5 gates ready-to-ship in a week. Counsel sign-off + consent UX rollout is the longer pole. + +--- + +## 6. Open questions for J + counsel + +1. **Photo-upload UX:** is there an existing intake form / staffer console where photo upload would happen? Or is this new UI work? +2. **Consent collection mechanism:** electronic signature service (DocuSign, Adobe Sign), in-app click-acceptance, paper form? Each has different evidentiary weight in litigation. +3. **Operator list with biometric access:** who, today, would be on the named-operators list for §3 training? +4. **Counsel for sign-off:** named outside counsel — same or different from the dual-control legal-token party in identity service? +5. **Public privacy policy URL:** does one exist? If yes, where; if no, that's a separate Gate-1.5 deliverable. + +--- + +## 7. What this PRD is NOT + +- Not legal advice. The `⚖ COUNSEL` markers exist because the binding text needs lawyers, not engineers. +- Not a substitute for a DPIA / PIA. Phase 1.6 satisfies BIPA-specific gates; a Data Protection Impact Assessment is broader and may be required separately. +- Not a SOC2 Type II deliverable. SOC2 is a parallel work stream. +- Not the only gate before production. The full 9-phase audit-trail program continues; Phase 1.6 specifically unblocks Phase 2 (identity service implementation). + +--- + +## Change log + +- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill. -- 2.54.0 From 0c74b82fc8a38dae9351f953d2ffcaad591dc8ed Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:48:21 -0500 Subject: [PATCH 12/65] =?UTF-8?q?phase=201.6=20gate=204:=20REMOVE=20name?= =?UTF-8?q?=20=E2=86=92=20ethnicity=20/=20gender=20inference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per docs/PHASE_1_6_BIPA_GATES.md Gate 4 + AUDIT_TRAIL_PRD §4 protected- attribute exclusion rule. The lookup tables + inference functions in search.html (3375-3499) and console.html (245-311) were dead code in the rendering path — headshot rendering disabled 2026-04-28 left these functions defined but unused. Removing them forecloses both Title VII discriminatory-feature-engineering AND BIPA biometric-information- derived-from-biometric-identifier arguments. Removed: - FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN - SURNAMES_HISPANIC, SURNAMES_SOUTH_ASIAN, SURNAMES_EAST_ASIAN, SURNAMES_MIDDLE_EASTERN, SURNAMES_BLACK - guessGenderFromFirstName(), guessEthnicityFromName(), guessEthnicityFromFirstName(), genderFor() From both search.html and console.html. Replacement: deprecation comment block referencing the BIPA gates doc. Verified: zero live consumers anywhere in mcp-server/. Searched for genderFor()/guessEthnicityFromName()/guessEthnicityFromFirstName()/ guessGenderFromFirstName() call sites — none remain. Per J 2026-05-03: this kind of test-code-leaked-into-main is exactly what J wants cleaned up. The face-pool inference was meant as a testing tool for synthetic icon generation but ended up as production-shape inference logic in the customer-facing UI. Co-Authored-By: Claude Opus 4.7 (1M context) --- mcp-server/console.html | 75 ++------------------ mcp-server/search.html | 152 +++++++--------------------------------- 2 files changed, 28 insertions(+), 199 deletions(-) diff --git a/mcp-server/console.html b/mcp-server/console.html index 250fc0b..6392090 100644 --- a/mcp-server/console.html +++ b/mcp-server/console.html @@ -238,77 +238,10 @@ function roleBand(role){ // the left edge + uppercase role pill in the detail line. Used by // every chapter that renders worker rows. `name` and `role` drive the // classification; `detail` is the full text after the pill. -// Quick first-name → gender hint for face-pool selection. Same lookup -// idea as the dashboard; if the name is unknown, the server falls back -// to the full pool. Trimmed table — covers the most common names that -// appear in the synthetic worker data. -var FEMALE_NAMES = new Set(['Mary','Patricia','Jennifer','Linda','Elizabeth','Barbara','Susan','Jessica','Sarah','Karen','Lisa','Nancy','Betty','Sandra','Margaret','Ashley','Kimberly','Emily','Donna','Michelle','Carol','Amanda','Melissa','Deborah','Stephanie','Dorothy','Rebecca','Sharon','Laura','Cynthia','Amy','Kathleen','Angela','Shirley','Brenda','Emma','Anna','Pamela','Nicole','Samantha','Katherine','Christine','Helen','Debra','Rachel','Carolyn','Janet','Maria','Catherine','Heather','Diane','Olivia','Julie','Joyce','Victoria','Ruth','Virginia','Lauren','Kelly','Christina','Joan','Evelyn','Judith','Andrea','Hannah','Megan','Cheryl','Jacqueline','Martha','Madison','Teresa','Gloria','Sara','Janice','Ann','Kathryn','Abigail','Sophia','Frances','Jean','Alice','Judy','Isabella','Julia','Grace','Amber','Denise','Danielle','Marilyn','Beverly','Charlotte','Natalie','Theresa','Diana','Brittany','Kayla','Alexis','Lori','Marie','Carmen','Aisha','Rosa','Mia','Audrey','Erin','Tina','Vanessa','Tara','Wendy','Tanya','Maya','Crystal','Yvonne','Kara','Shannon','Brianna','Faith','Caroline','Carla','Tracey','Tracy','Rita','Dawn','Tiffany','Stacy','Stacey','Gina','Bonnie','Tammy','Joanne','Jamie','Tonya','Alyssa','Ariana','Elena','Ellie','Erica','Erika','Felicia','Holly','Jenna','Jenny','Krista','Kristen','Kristin','Krystal','Lana','Leah','Lucy','Mallory','Melinda','Meredith','Misty','Monica','Naomi','Paige','Paula','Renee','Rhonda','Robin','Roxanne','Selena','Sierra','Skylar','Sonia','Stella','Tamara','Veronica','Vivian','Whitney','Yolanda','Zoe']); -var MALE_NAMES = new Set(['James','Robert','John','Michael','David','William','Richard','Joseph','Thomas','Charles','Christopher','Daniel','Matthew','Anthony','Mark','Donald','Steven','Paul','Andrew','Joshua','Kenneth','Kevin','Brian','George','Edward','Ronald','Timothy','Jason','Jeffrey','Ryan','Jacob','Gary','Nicholas','Eric','Jonathan','Stephen','Larry','Justin','Scott','Brandon','Benjamin','Samuel','Gregory','Frank','Alexander','Raymond','Patrick','Jack','Dennis','Jerry','Tyler','Aaron','Jose','Adam','Henry','Nathan','Douglas','Zachary','Peter','Kyle','Walter','Ethan','Jeremy','Harold','Keith','Christian','Roger','Noah','Gerald','Carl','Terry','Sean','Austin','Arthur','Lawrence','Jesse','Dylan','Bryan','Joe','Jordan','Billy','Bruce','Albert','Willie','Gabriel','Logan','Alan','Juan','Wayne','Roy','Ralph','Randy','Eugene','Vincent','Russell','Elijah','Louis','Bobby','Philip','Johnny','Marcus','Antonio','Carlos','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Victor','Jamal','Xavier','DeShawn','Dwayne','Jermaine','Malik','Tyrone','Devon','Andre','Brent','Calvin','Casey','Cody','Cole','Cory','Dale','Damon','Darius','Darrell','Dean','Derek','Drew','Earl','Eddie','Floyd','Glenn','Greg','Howard','Ivan','Jared','Jay','Jeff','Joel','Lance','Lee','Leonard','Lloyd','Mario','Martin','Mason','Maurice','Max','Mitchell','Morgan','Nick','Norman','Oliver','Owen','Pete','Quincy','Rafael','Reggie','Rex','Ricky','Russ','Shane','Shaun','Stanley','Steve','Theodore','Todd','Travis','Trevor','Troy','Wade','Warren','Wesley']); -function guessGenderFromFirstName(n){ - if(!n) return null; - var clean=n.replace(/[^A-Za-z]/g,''); - if(!clean) return null; - var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase(); - if(FEMALE_NAMES.has(c)) return 'woman'; - if(MALE_NAMES.has(c)) return 'man'; - return null; -} -function genderFor(name){ - var g = guessGenderFromFirstName(name); - if(g) return g; - if(!name) return 'man'; - var s=String(name); var h=0; - for(var i=0;i 1 ? parts[parts.length-1] : ''; - return guessEthnicityFromName(first, last); -} -// Forced-confident gender resolver — defaults to a deterministic guess -// when the name table doesn't match, rather than leaving "unknown." -// We're authoring the synthetic data; we own the confidence call. -function genderFor(name){ - var g = guessGenderFromFirstName(name); - if(g) return g; - if(!name) return 'man'; - // hash-based fallback so unknown names still spread roughly 50/50 - var s = String(name); - var h = 0; for (var i=0;i Date: Sun, 3 May 2026 01:57:20 -0500 Subject: [PATCH 13/65] =?UTF-8?q?REVERT=20cloud=20routing=20on=20hot=20pat?= =?UTF-8?q?h=20=E2=80=94=20back=20to=20local=20Ollama=20per=20PRD=20line?= =?UTF-8?q?=2070?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRD line 70: "Everything runs locally — no cloud APIs, total data privacy." Yesterday's PR #13 (feb638e) violated this by routing customer-facing inference paths to opencode + ollama_cloud + openrouter. Reverting the hot-path routes only; cloud providers stay configured in providers.toml for explicit dev-tool opt-in. Reverted: - modes.toml staffing_inference: kimi-k2.6 → qwen3.5:latest (local Ollama) - modes.toml doc_drift_check: gemini-3-flash-preview → qwen3.5:latest - execution_loop overseer: opencode/claude-opus-4-7 → ollama/qwen3.5:latest Was a paid Anthropic call on every overseer escalation; now local + free. Gateway compiles + restarts clean. Lance smoke 10/10. Live providers list unchanged (kimi/ollama_cloud/opencode/openrouter all still CONFIGURED; they just aren't ROUTED to from the staffing inference path anymore). This stops the API meter on customer requests. Cloud providers remain opt-in via explicit provider= caller hint, which the scrum tool + auditor pipeline + bot/propose use deliberately. Co-Authored-By: Claude Opus 4.7 (1M context) --- config/modes.toml | 16 +++++------ crates/gateway/src/execution_loop/mod.rs | 34 +++++++++++------------- 2 files changed, 22 insertions(+), 28 deletions(-) diff --git a/config/modes.toml b/config/modes.toml index bf7f159..f327f17 100644 --- a/config/modes.toml +++ b/config/modes.toml @@ -40,14 +40,13 @@ matrix_corpus = "chicago_permits_v1" name = "staffing_inference" # Staffing-domain native enrichment runner — Pass 4 (2026-04-26). # Same composer architecture as codereview_lakehouse but with staffing -# framing + workers corpus. Validates that the modes-as-prompt-molders -# pattern generalizes beyond code review. +# framing + workers corpus. preferred_mode = "staffing_inference_lakehouse" fallback_modes = ["ladder", "consensus", "pipeline"] -# 2026-04-28: gpt-oss-120b:free → kimi-k2.6 via Ollama Pro. Coding- -# specialized, faster than gpt-oss, on the same OLLAMA_CLOUD_KEY so -# no extra provider hop. -default_model = "kimi-k2.6" +# 2026-05-03: REVERTED to local. PRD line 70 — everything runs locally, +# no cloud APIs on the customer hot path. Cloud models stay available +# in providers.toml for explicit dev-tool opt-in (scrum, auditor). +default_model = "qwen3.5:latest" matrix_corpus = "workers_500k_v8" [[task_class]] @@ -61,9 +60,8 @@ matrix_corpus = "kb_team_runs_v1" name = "doc_drift_check" preferred_mode = "drift" fallback_modes = ["validator"] -# 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama Pro. -# Speed leader on factual checking, same OLLAMA_CLOUD_KEY. -default_model = "gemini-3-flash-preview" +# 2026-05-03: REVERTED to local per PRD line 70. +default_model = "qwen3.5:latest" matrix_corpus = "distilled_factual_v20260423095819" [[task_class]] diff --git a/crates/gateway/src/execution_loop/mod.rs b/crates/gateway/src/execution_loop/mod.rs index 4d5d1f3..011f5bc 100644 --- a/crates/gateway/src/execution_loop/mod.rs +++ b/crates/gateway/src/execution_loop/mod.rs @@ -605,56 +605,52 @@ impl ExecutionLoop { /// cheapest token. Frequency is low so the Zen pay-per-token cost /// stays bounded. async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> { - let Some(opencode_key) = self.state.opencode_key.clone() else { - return Err("OPENCODE_API_KEY not configured — skipping escalation".into()); - }; - + // 2026-05-03: REVERTED to local-only per PRD line 70. Cloud + // overseer (opencode/claude-opus-4-7) was a recent addition that + // moved a hot-path call OFF the local Ollama runtime onto a paid + // cloud provider. Reverted to local Ollama (qwen3.5:latest). + // Cloud overseer can be re-enabled by setting LH_OVERSEER_CLOUD=1 + // for development; production stays local. let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await; let prompt = build_overseer_prompt(&self.req, &kb, &self.log, reason); let started = std::time::Instant::now(); let start_time = chrono::Utc::now(); let chat_req = crate::v1::ChatRequest { - model: "claude-opus-4-7".to_string(), + model: "qwen3.5:latest".to_string(), messages: vec![crate::v1::Message::new_text("user", prompt.clone())], temperature: Some(0.1), max_tokens: None, stream: Some(false), - // Anthropic models on opencode reject `think` (handled in - // the adapter), but we keep the intent flag for parity. - think: Some(true), - provider: Some("opencode".into()), + think: Some(false), + provider: Some("ollama".into()), }; - let resp = crate::v1::opencode::chat(&opencode_key, &chat_req).await - .map_err(|e| format!("opencode: {e}"))?; + let resp = crate::v1::ollama::chat(&self.state.ai_client, &chat_req).await + .map_err(|e| format!("ollama overseer: {e}"))?; let latency_ms = started.elapsed().as_millis() as u64; let end_time = chrono::Utc::now(); let correction_text: String = resp.choices.into_iter().next() .map(|c| c.message.text()).unwrap_or_default(); - // Stamp per-task stats — cloud call counts against the same - // usage counter so `/v1/usage` shows cloud token spend too. self.stats.requests = self.stats.requests.saturating_add(1); self.stats.prompt_tokens = self.stats.prompt_tokens.saturating_add(resp.usage.prompt_tokens as u64); self.stats.completion_tokens = self.stats.completion_tokens.saturating_add(resp.usage.completion_tokens as u64); self.stats.total_tokens = self.stats.total_tokens.saturating_add(resp.usage.total_tokens as u64); self.stats.latency_ms = self.stats.latency_ms.saturating_add(latency_ms); - // Langfuse trace for the overseer call (same pipe that feeds - // the observer/KB, so this correction's cost lands in the KB - // too — closing the loop). + // Langfuse trace for the overseer call (local-only now). if let Some(lf) = &self.state.langfuse { use crate::v1::langfuse_trace::ChatTrace; lf.emit_chat(ChatTrace { - provider: "opencode".into(), - model: "claude-opus-4-7".into(), + provider: "ollama".into(), + model: "qwen3.5:latest".into(), input: vec![crate::v1::Message::new_text("user", prompt.clone())], output: correction_text.clone(), prompt_tokens: resp.usage.prompt_tokens, completion_tokens: resp.usage.completion_tokens, temperature: Some(0.1), max_tokens: None, - think: Some(true), + think: Some(false), start_time: start_time.to_rfc3339(), end_time: end_time.to_rfc3339(), latency_ms, -- 2.54.0 From 2f5ca95875717489d84b996fe0f1a017cbbbdabc Mon Sep 17 00:00:00 2001 From: root Date: Sun, 3 May 2026 01:59:32 -0500 Subject: [PATCH 14/65] =?UTF-8?q?scrum:=20real=20tool=20=E2=80=94=20auto-b?= =?UTF-8?q?undle=20current=20diff,=20run=203-reviewer=20scrum,=20push=20to?= =?UTF-8?q?=20KB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit J's frustration captured: scrum was a TOOL meant to find gaps in the work and push findings to a KB that informs how WE work on the code. It became welded into architecture instead of staying a tool. This commits the tool in the form J actually meant. Usage: ./scrum auto-bundle origin/main..HEAD, auto-label ./scrum my_label same with explicit label ./scrum --staged bundle staged-only diff (pre-commit) ./scrum --since=COMMIT bundle from a specific commit Output: KB row → data/_kb/scrum_findings.jsonl (one row per scrum run) Verdicts → reports/scrum/_evidence//verdicts/