Compare commits

..
65 Commits
Author SHA1 Message Date
ClawandClaude Opus 5 cbc24a4b19 feat(gateway/v1): real SSE streaming on the openrouter lane; honest 400 elsewhere
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "pretending to have streamed. Verified live: chat.completion.chunk frames from"
The gateway accepted `stream: true` and returned a single JSON body — the code
said 'Phase 39+ wires real streaming' and it never happened. A streaming client
waits for SSE frames that never arrive and reports 'no response': that is what
broke openclaw's gateway lane today and cost six orchestrator rounds before the
cause was found.

Now: openrouter/openrouter_free forward the upstream SSE byte-for-byte; every
other lane rejects stream=true with a 400 that names the lane, instead of
pretending to have streamed. Verified live: chat.completion.chunk frames from
:3100, and the non-streaming path unchanged. Streamed calls skip usage metering
and Langfuse (stated in the code and logged per call) — metering needs the final
usage chunk and belongs in its own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-20 07:33:47 -05:00
ClawandClaude Opus 5 dd74a1ef74 fix(gateway/v1): pass OpenAI tool calling through on the openrouter lane; reject tools loudly elsewhere
lakehouse/auditor 11 blocking issues: cloud: claim not backed — "Verified live after deploy: tool_calls returned (finish_reason tool_calls); plain request"
/v1/chat/completions deserialized into a struct without tools/tool_choice and messages without
tool_calls/tool_call_id, so serde silently DROPPED them. Tool-using agents routed through :3100
(openclaw, 2026-09-18) never saw their tools and fabricated tool output as text.

- ChatRequest: tools + tool_choice (serde default). Message: tool_calls, tool_call_id, name
  (skip_serializing_if none -> plain-text JSON is byte-identical; test pins it).
- openrouter lane forwards them and returns the model's tool_calls (content null on a
  tool-call-only turn, finish_reason passed through).
- every other provider: HTTP 400 when a request uses tools, instead of silent stripping.

Verified live after deploy: tool_calls returned (finish_reason tool_calls); plain request
unchanged; ollama + tools -> 400; multi-turn tool result round-trip answered from the tool.
cargo test -p gateway --lib: 111 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 09:08:29 -05:00
rootandClaude Fable 5 9d5107fb4a auditor: alt lane (Haiku) off drained opencode onto OpenRouter
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "password). Verified live: /api/v1/user auths as profit, pulls list 200s."
PR #14 audit 2 flipped to the alt lane and hit opencode 401
CreditsError (balance drained). Alt defaults move to
openrouter/anthropic/claude-haiku-4.5 (probed live through the
gateway); maxTokensFor learns vendor-prefixed claude ids.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:40:13 -05:00
rootandClaude Fable 5 8bdceda883 gateway: OR_TIMEOUT_SECS env-overridable — 180s wall starved kimi_architect
lakehouse/auditor 8 blocking issues: cloud: claim not backed — "password). Verified live: /api/v1/user auths as profit, pulls list 200s."
The auditor's kimi_architect second pass (~180K-char prompt via
openrouter/moonshotai/kimi-k2.6) died at exactly the hardcoded 180s
client timeout on PR #14's drill audit, surfacing as 'error decoding
response body'. Ceiling now reads OR_TIMEOUT_SECS (default unchanged);
host drop-in sets 900s to match the auditor's own call budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:31:22 -05:00
rootandClaude Fable 5 5972cd2fbc auditor: repoint Gitea client at git.warren.help — agentview host is dead
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "password). Verified live: /api/v1/user auths as profit, pulls list 200s."
git.agentview.dev has served a Traefik default cert + API 404 since
Aug 25 (VPS consolidation moved repos to warren); listOpenPrs failed
every 90s cycle since. Host default moves to git.warren.help and auth
now resolves per-host from ~/.git-credentials with fallback to the
origin remote's embedded credentials, sent as Basic (works for PAT or
password). Verified live: /api/v1/user auths as profit, pulls list 200s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 04:06:54 -05:00
rootandClaude Fable 5 0661c2cb83 auditor: sweep model lanes off lapsed Ollama Cloud onto OpenRouter
Ollama Cloud sub lapsed 2026-09-02 — every ollama_cloud model 402s.
Consensus lane -> openrouter/deepseek/deepseek-chat-v3.1 (same lineage),
tie-breaker -> openrouter/x-ai/grok-4.3 (4.1-fast deprecated upstream),
kimi_architect -> openrouter/moonshotai/kimi-k2.6 via provider default
so the Kimi<->Haiku alternation stays intact. maxTokensFor learns the
vendor-prefixed kimi id. All three lanes + pr_audit mode runner probed
live through the gateway before landing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 04:05:13 -05:00
rootandClaude Fable 5 c97f52de82 gateway/v1: 402 affordability clamp-retry on the openrouter lane
Stripping max_tokens on a 402 made things worse (the precheck then
prices the model's own 65k maximum). Instead: parse the 'can only
afford N' budget from the error and retry once clamped to N when
N >= 256 — small answers survive a thin balance; anything less
surfaces the 402 verbatim, because the real fix is funding the
account the key belongs to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-28 17:12:53 -05:00
rootandClaude Fable 5 6de15ad5f5 gateway/v1: tolerate reasoning-model responses on the openrouter lane
Reasoning models (GLM-5.x, deepseek-v4-pro) return content:null plus a
reasoning field when max_tokens exhausts mid-thought; ORMessageResp's bare
String made serde reject the whole body ('invalid openrouter response'),
taking down every reasoning model on the lane — including phase2's
configured z-ai/glm-5.2. content/reasoning are now Options; content stays
the only text returned (empty when null, so callers' schema validation +
retry budgets handle it deterministically); reasoning-only responses log a
warn with finish_reason. Unit test pins both shapes.

Found during the 2026-08-28 VIX founder-lane dogfood.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-28 17:07:12 -05:00
Claw 8dd0d46c3c build: pin rustc 1.94.0 — the workspace only builds in a narrow band
Established empirically in a clean Debian 13 container (2026-07-28), not inferred
from the dev host:

  1.85.0  (apt install rustc, Debian trixie)  -> FAILS, aws-sdk-* need >= 1.91.1
  1.97.1  (rustup default stable, today)      -> FAILS, ethnum E0512 transmute
  1.94.0                                      -> builds clean, workspace, 2m39s

Both obvious reviewer paths break: the distro compiler is too old and current
rustup stable is too new. Before this pin the build reproduced on our host only
because the toolchain had been restored from the previous machine — reproducible
by accident rather than by construction.

Verified the pin works: set the shell default to the BROKEN stable, and the repo
still resolved 1.94.0 and built.

Minimal native prerequisites, also tested in the clean container:
  protobuf-compiler (lance-encoding/lance-file build scripts need protoc),
  pkg-config, libssl-dev.
NOT required despite being installed on the dev host: cmake, clang, libclang-dev.
2026-07-28 12:33:53 -05:00
ClawandClaude Opus 4.8 ba3024c3af config: disable HNSW auto-tuning agent
Flips [hnsw auto-tune] enabled true->false to keep the background propose/trial/promote
loop off live search traffic on shared Ollama. Revert to re-enable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:55:33 -05:00
ClawandClaude Opus 4.8 0d37a94ebf chore: drop stale scrum-WARN annotations in biometric_endpoint (comments only)
Removes dated 'Caught 2026-05-0X ... WARN' inline comments; no code change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:55:33 -05:00
ClawandClaude Opus 4.8 7365a4c23e ui: staffing co-pilot UX fixes — new-tab reference nav + discoverable collapsibles
Reconciles month-old working-tree UX work (served per-request, live on the demo since
2026-05-30) into git. Implements UX_FIX_PLAN.md's 5 user complaints: reference pages
(Profiler/Architecture/Spec/Onboard/Alerts/Workspaces) open in new tabs; contract-card
collapsible shows a persistent chevron + expand/collapse label in both states; proof.html
gains a substrate-signals chapter. Not authored this session — captured as-is (working
state verified: pages serve 200).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:55:33 -05:00
ClawandClaude Opus 4.8 7dddb98f4b docs: STATE_OF_PLAY — record search index-alignment fix + P1-already-done
- New 2026-06-30 wave: the ethereal_workers↔workers_500k_v1 misalignment fix (8 sites,
  commits 423400e/35a80a7/1570306), verified live after mcp-server restart
- Flag that demo-plan P1 (search-box filter) is already done — the P1 section is stale
- Note the untouched 2026-05-30 uncommitted WIP + newer untracked planning docs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:50:31 -05:00
ClawandClaude Opus 4.8 15703063ea fix: align 4 more intelligence routes to ethereal_workers_v1
Same index↔dataset misalignment swept across the intelligence handler — four more
/vectors/hybrid calls that paired workers_500k_v1 with filter_dataset ethereal_workers:
- similar-workers route (~2418)
- semantic role-discovery route (~2459)
- smart_search route (~2869)
- scenario worker-find (~3531)

Verified: no workers_500k_v1 index remains paired with the ethereal_workers dataset;
the 9 remaining refs are correctly aligned (workers_500k dataset) or pure RAG. Server
boots clean. Takes effect on next mcp-server restart.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:47:16 -05:00
ClawandClaude Opus 4.8 35a80a7306 fix: align search_workers + match_contract + /match to ethereal_workers_v1
Same index↔dataset misalignment as the /search fix (423400e), in three more paths that
pair the ethereal_workers SQL dataset with the workers_500k_v1 vector index:
- search_workers MCP tool
- match_contract MCP tool
- /match HTTP handler

All now default to ethereal_workers_v1 (same 10K id-space as the SQL table). Verified:
/match role=Forklift Operator state=IL returns only IL forklift operators (sql_matches 94).
rag_question / /ask left as-is — pure RAG, no SQL join, so no id-space to misalign.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:44:44 -05:00
ClawandClaude Opus 4.8 423400e843 fix: /search default index aligned to ethereal_workers_v1
The /search hybrid endpoint defaulted to index workers_500k_v1 (50K chunks, a
different worker set) while defaulting filter_dataset to ethereal_workers (10K SQL
rows). Same worker_id → different person in each, so the SQL state filter and the
vector rerank referred to different people: state='IL' surfaced a KY worker flagged
sql_verified:true. Aligned the default index to ethereal_workers_v1 (10K chunks, same
id-space as the SQL table). Verified: state='IL' now returns only IL workers.

Takes effect on next mcp-server restart. Sibling paths /match and the search_workers
tool have the same workers_500k_v1↔ethereal_workers misalignment (not touched here).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:42:14 -05:00
rootandClaude Opus 4.8 ab1f12296d profiler: multi-source quote aggregation with confidence scoring
Replaces the single-source quote feed with a 3-source consensus system,
the differentiator for the index: pull each ticker from independent
feeds, take the median, and score confidence by how many corroborate it.
Disagreement becomes a signal instead of a silently-wrong number.

Context: Stooq retired its free q/l/ CSV quote API (404s server-side;
q/d/l behind a JS challenge), which broke the profiler basket. The
interim Yahoo-only fix worked but had a hidden wrong-baseline bug —
Yahoo's chartPreviousClose over a 5d window references the pre-window
close, so day-change read +11.65% for CAT when the real figure is
+1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class
of quiet error, which a single source can never self-report.

- quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA;
  Yahoo prev-close now derived from the daily candle, not the buggy
  meta field), median consensus, confidence = corroborating sources / 3,
  outlier flagging, and a bun:sqlite hourly time-series cache.
- quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} —
  hourly snapshot into the cache (real-time not required per product
  call). Runs as root to match the mcp-server that co-owns the db;
  busy_timeout serializes the two writers.
- /intelligence/ticker_quotes — serves cache-first (fast), aggregates
  cold tickers on-demand once and caches them. New /quote_ingest admin
  trigger.
- profiler.html — per-card confidence badge (green/amber/red) + a
  per-source breakdown tooltip; "3-source consensus" labeling.
- entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief
  (kept fetchStooqQuote dead-but-reversible).

Verified live on devop.live/lakehouse/profiler: liquid names read
100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only)
honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly
timer armed. Cache db (data/_quote_cache/) is gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 01:28:28 -05:00
rootandClaude Opus 4.7 fcd53168a0 phase 1.6: counsel handoff turnkey + seed_consent_version.sh + strict mode live
The remaining production blocker is counsel-calendar bottleneck
(review + sign-off). Engineering can't make counsel move faster,
but it CAN reduce the round-trip overhead:

(1) docs/counsel/COUNSEL_HANDOFF_EMAIL_2026-05-05.md — copy-paste
    email body J can send to outside counsel. Subject line + body
    + tarball attachment instructions + headline asks (A/B/C/D
    in priority order) + post-signature operator runbook. The
    pre-flight checklist + post-signature workflow turn what
    would have been "I'll figure out the email" into "click send."

(2) scripts/staffing/seed_consent_version.sh — turnkey
    post-signature deployment. Takes the path to a (presumably
    counsel-signed) consent template markdown, computes SHA-256,
    atomically merges into /etc/lakehouse/consent_versions.json
    (creating the file if absent, with per-seed audit metadata
    in _meta.seeded_at[]), restarts lakehouse.service, probes
    /biometric/health post-restart. Idempotent: re-running with
    the same hash is a no-op for the versions array but still
    appends a [reseed] entry to the audit metadata.
    Verified live against the eng-staged template — strict mode
    flipped clean, /biometric/health 200 post-restart.

(3) docs/PHASE_1_6_BIPA_GATES.md §6.5 — post-signature deployment
    runbook embedded in the gates doc. Three steps: counsel signs
    + commits → seed_consent_version.sh → strict-mode probe.
    Plus a "pre-counsel demo seed" subsection documenting how to
    exercise strict mode BEFORE counsel signs (using the
    eng-staged template hash) so the deployment workflow is
    proven before the legal critical path closes.

Strict mode flipped live — verified post-restart:
- /etc/lakehouse/consent_versions.json populated with the
  eng-staged template hash:
  8b09591a8dc15f59197affac48909ce943d575eee01705b42303acf3b32f5c56
- POST /biometric/subject/WORKER-1/consent with deadbeef hash:
  HTTP 400 + error="consent_version_unknown"
- POST with the known eng-staged hash: passes version check
  (then 404 subject_not_found on a ghost candidate, proving
  the gate is hash-aware not auth-broken)

The hash currently seeded is the ENG-STAGED template
(pre-counsel-signature). When counsel returns the signed text,
operator runs `seed_consent_version.sh` again with the
counsel-signed markdown — the new hash gets appended; the demo
hash stays in for backwards-compat with any consent records
collected during the pre-counsel demo period (none, today).

Production blocker is now genuinely just counsel calendar:
1. J transmits reports/counsel/counsel_packet_2026-05-05.tar.gz
   per the handoff email
2. Counsel reviews + signs (their billable time)
3. Counsel returns signed text → operator runs seed script
4. Strict mode flips to canonical hash → cutover complete

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 15:32:16 -05:00
rootandClaude Opus 4.7 87b034f5f9 phase 1.6: ops dashboard + consent_versions allowlist + subject timeline tool
Closes the afternoon's "all four" wave (per J's request to do all the
items in one pass instead of pick-one-of-options):

(1) Live demo on WORKER-100 — full lifecycle exercised end-to-end
    against the running gateway. 3 audit rows landed in correct
    order (consent_grant → biometric_collection →
    consent_withdrawal), chain_verified=true, photo on disk at
    data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg
    (180 bytes JFIF). retention_until=2026-06-04 (30d from
    withdrawal per consent template v1 §2).

(2) GET /biometric/stats — read-only aggregate over all subjects.
    Returns counts by biometric.status + subject.status, photo
    count, oldest_active_retention_until, and the last 20
    state-change events (consent_grant / collection / withdrawal /
    erasure — validator_lookup and other noise filtered out).
    Walks per-subject audit logs via the existing writer; cheap
    for 100 subjects, would want an event-stream index at 100k.
    Legal-tier auth (same posture as /audit). 4 unit tests.

(3) /biometric/dashboard mcp-server frontend. Auto-refreshes
    /biometric/stats every 15s, neo-brutalist tile layout for
    the per-status counts + retention horizon block + recent
    events table with kind badges + event-kind breakdown pills.
    sessionStorage-backed token; logout button clears state.
    DOM-built throughout (textContent + createElement) — never
    innerHTML on audit-row values, since trace_id et al. could
    in theory carry operator-supplied strings.

(4) consent_versions allowlist. BiometricEndpointState gains
    `allowed_consent_versions: Option<Arc<HashSet<String>>>`,
    loaded at startup from /etc/lakehouse/consent_versions.json
    (override via LH_CONSENT_VERSIONS_FILE). process_consent
    refuses unknown hashes with HTTP 400 consent_version_unknown
    when configured. Resolution semantics:
      - Missing file → permissive (v1 compat, warn-log)
      - Parse error → permissive (error-log; broken config
        silently going strict would be worse)
      - Empty array → strict, refuse all (deliberate freeze
        mode for "counsel hasn't signed v1 yet")
      - Populated → strict, lowercase-normalized comparison
    5 unit tests (known/unknown/case/empty/none-permissive).
    Example template at ops/consent_versions.example.json with
    a counsel-tier deployment note.

(5) scripts/staffing/subject_timeline.sh — operator one-shot
    pretty-print of any subject's full BIPA lifecycle. Curls
    /audit/subject/{id} with legal token; renders manifest
    summary + on-disk photo state + chronological audit chain
    with kind badges + chain verification status. Smoke-tested
    on WORKER-100 (3 rows verified).

(6) STATE_OF_PLAY.md refresh. New section "afternoon wave"
    captures all four commits (76cb5ac, 7f0f500, 68d226c, this
    one) + the live demo evidence + the v1 endpoint matrix +
    UI/CLI inventory + the production-cutover blocking set
    (counsel calendar only — eng substrate is done).

Verified live post-restart:
- /audit/health + /biometric/health both 200
- /biometric/stats returns 100 subjects, 2 withdrawn (WORKER-2 from
  earlier scrum + WORKER-100 from today's demo), 1 photo on record,
  6 recent state-change events
- /biometric/intake + /biometric/withdraw + /biometric/dashboard
  all 200 on mcp-server :3700
- subject_timeline.sh on WORKER-100: chain_verified=true,
  chain_root=a47563ff937d50de…
- 88/88 catalogd lib tests + 55/55 biometric_endpoint tests green

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 15:27:52 -05:00
rootandClaude Opus 4.7 68d226c314 phase 1.6: BIPA withdrawal endpoint + UI + retention sweep timer
Closes the four production gaps that were live after the consent
endpoint shipped (76cb5ac):

(1) Withdrawal endpoint POST /biometric/subject/{id}/withdraw
    backs the BIPA right of withdrawal that consent template v1 §2
    explicitly promises. Without it, the only way to honor a
    candidate's withdrawal request was the heavier /erase, which
    destroys immediately rather than starting the 30-day SLA clock
    that the consent template commits to. Side-effects:
      - manifest.consent.biometric.status: Given → Withdrawn
      - manifest.consent.biometric.retention_until: 18mo → 30d
      - audit row kind=biometric_consent_withdrawal, captures
        reason + operator_of_record + evidence_path
      - DOES NOT touch general_pii or subject.status — biometric
        is independently revocable
    State machine: Given→Withdrawn (happy), NeverCollected/Pending→
    409 nothing_to_withdraw, Withdrawn→409 already_withdrawn (won't
    advance the destruction clock), Expired→409 already_expired,
    subject Erased/RetentionExpired→403 subject_inactive.
    12 new unit tests covering happy path + all guards + a full
    grant→withdraw cycle that asserts retention_until is correctly
    accelerated and the audit chain has 2 rows in correct order.

(2) Withdraw UI at /biometric/withdraw (mcp-server-served HTML).
    3-screen flow: operator auth (token + name in sessionStorage),
    withdrawal form (candidate_id + free-text reason ≥10 chars +
    optional evidence path), confirmation showing the audit row
    HMAC + the 30-day retention_until clock + a curl recipe for
    /audit/subject/{id} verification. Same neo-brutalist styling
    as biometric_intake.html. Mounted at
    http://localhost:3700/biometric/withdraw and externally at
    https://devop.live/lakehouse/biometric/withdraw.

(3) Retention sweep systemd timer. crates/catalogd/bin/retention_sweep
    binary already existed; this commit schedules it. Daily 03:00 UTC,
    Persistent=true so a missed boot triggers on next start. Service
    runs as oneshot with --apply (writes a date-stamped JSONL to
    data/_catalog/subjects/_retention_sweep_<date>.jsonl ONLY when
    overdue subjects exist, per the binary's existing semantics).
    install.sh updated to handle .timer + paired .service correctly:
    enables the timer, skips direct start of the oneshot service
    (the timer pulls it in). One-shot manual test confirmed clean:
    100 subjects scanned, 0 overdue (all backfill subjects within
    their 4-year general retention window).

(4) operator_of_record bug fix in intake UI. Previously the page
    hardcoded the literal string 'intake_ui_operator' as the
    operator_of_record sent to /consent — meaning every audit row
    captured the same useless placeholder, defeating the whole
    point of operator traceability. Fixed by adding an operator
    name field to the token-paste step (sessionStorage-backed),
    passed through to consent + photo POSTs as the actual operator.

Verified live post-restart:
- gateway /audit/health + /biometric/health both 200
- mcp-server /biometric/intake + /biometric/withdraw both 200
- Live withdraw probes: 401 (no token), 400 (empty body), 404
  (ghost subject), 409 nothing_to_withdraw on WORKER-1 (which
  is NeverCollected per backfill default) — all expected
- Binary strings contain: process_withdraw, withdraw_consent,
  biometric_consent_withdrawal, biometric_withdraw_response.v1,
  nothing_to_withdraw, already_withdrawn, already_expired,
  /subject/{candidate_id}/withdraw route
- systemd: lakehouse-retention-sweep.timer active+enabled,
  next fire Tue 2026-05-05 22:00 CDT (= 03:00 UTC May 6)
- Manual one-shot of retention sweep service: exit 0/SUCCESS,
  100 subjects loaded, 0 overdue

83/83 catalogd lib tests + 46/46 biometric_endpoint tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 15:09:32 -05:00
rootandClaude Opus 4.7 7f0f500050 phase 1.6: candidate intake UI — operator-driven consent + photo capture
Adds the Gate 2 frontend that operators use to capture candidate
biometric consent + photo at intake. Without this, the
/biometric/subject/{id}/consent endpoint shipped in 76cb5ac has
no caller — operators would have to build curl scripts by hand.

Surfaces:
- mcp-server serves /biometric/intake as static HTML (mounted at
  http://localhost:3700/biometric/intake; externally accessible via
  nginx at https://devop.live/lakehouse/biometric/intake)
- URL must include ?candidate_id=WORKER-XXX

Page flow (4 screens):
- Step 1 — operator authentication. Pastes legal-tier audit token
  into a password field; stored in sessionStorage (cleared on tab
  close), never localStorage, never cookies.
- Step 2 — consent. Renders the v1 consent template text inline
  (Disclosures 1/2/3 + plain-language summary, all matching
  docs/policies/consent/biometric_consent_template_v1.md as of
  this commit). Captures candidate printed name + checkbox accept;
  computes SHA-256 of the rendered consent block as
  consent_version_hash; computes SHA-256 of the click-acceptance
  evidence (method + name + ts + user_agent + page_origin) and
  records it as inline:sha256=<hash> evidence path. POSTs to
  gateway /biometric/subject/{id}/consent.
- Step 3 — photo. Two paths: file upload OR getUserMedia camera
  capture. Preview before submit. Skip-photo button for
  consent-only intake (e.g. consent collected before equipment
  available). POSTs to gateway /biometric/subject/{id}/photo.
- Step 4 — confirmation. Displays the audit row HMACs from both
  endpoint responses + the verify_biometric_erasure.sh command
  the operator can run for chain attestation.

Design choices:
- No framework, no build step. Single self-contained HTML file
  ~22KB. Matches the existing mcp-server precedent
  (onboard.html, console.html, etc.).
- Neo-brutalist dark style matching mcp-server's other pages
  (tracked dark surfaces, monospace technical metadata, sharp
  borders). Consistent with j's UI preferences.
- Server-authoritative timestamps. The page sends its own UA +
  click ts as evidence, but the canonical given_at on the
  manifest comes from the gateway's Utc::now() (per
  process_consent). Page displays whatever the gateway returns.
- Gateway URL configurable via ?gw= query param; defaults to
  http://localhost:3100 for the same-box workstation pattern.
  External-access deployment requires a separate nginx route
  for the gateway (out of scope for v1 to avoid touching the
  devop.live nginx config).

Verified live:
- GET http://localhost:3700/biometric/intake?candidate_id=WORKER-100
  returns 200 + 22KB HTML body
- GET https://devop.live/lakehouse/biometric/intake?candidate_id=WORKER-100
  returns 200 (nginx /lakehouse/ route proxies to :3700)
- Gateway CORS preflight for POST /biometric/subject/{id}/consent
  from origin http://localhost:3700: 200 with allow-methods/headers/origin: *

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 14:12:43 -05:00
rootandClaude Opus 4.7 76cb5acb03 phase 1.6: add consent-record endpoint POST /biometric/subject/{id}/consent (Gate 2 backend)
Backs the candidate-facing consent flow specified in
docs/policies/consent/biometric_consent_template_v1.md §5. Without
this endpoint, no production code path could flip
consent.biometric.status from NeverCollected/Pending to Given —
meaning every photo upload would fail-closed at the consent gate
(403 consent_required) forever, even after counsel signs the
template. This unblocks the post-cutover intake flow.

Endpoint shape:
- POST /biometric/subject/{candidate_id}/consent
- Auth: legal-tier (X-Lakehouse-Legal-Token header)
- Body (JSON): {consent_version_hash, consent_collection_method,
  consent_collection_evidence_path?, operator_of_record}
- consent_collection_method constrained to closed set:
  electronic_signature / paper / click_acceptance — operator
  typo would silently weaken evidentiary defensibility
- consent_version_hash recorded but not gated against allowlist
  at runtime (avoids config-deployment dependency for v2 template
  rotation; counsel validates retroactively against
  consent_versions table)

State-machine semantics (mirrors upload's double-upload guard):
- NeverCollected / Pending → Given (happy path)
- Given → 409 consent_already_given (re-collect requires explicit
  erase + fresh grant under new template version)
- Withdrawn / Expired → 409 consent_post_withdrawal_requires_erase
  (explicit erase preserves audit-chain ordering of the cycle)
- Subject status Withdrawn / Erased / RetentionExpired → 403

Server-side authoritative timestamps:
- given_at = Utc::now() (operator-supplied would be tamperable)
- retention_until = given_at + 540 days (18 months per retention
  schedule v1 §4). If counsel changes the cap, schedule doc + code
  constant CONSENT_RETENTION_DAYS change in the same PR.

Audit row:
- accessor.kind = "biometric_consent_grant"
- accessor.purpose = "version=<hash>;method=<m>;operator=<op>;evidence=<path>"
- fields_accessed = ["consent.biometric.status",
                     "consent.biometric.retention_until"]
- result = "given"
- Transactional: manifest commit before audit append; rollback
  manifest if audit fails.

Tests added (12 new):
- consent_happy_path_flips_status_and_records_audit (full happy
  path + audit row inspection + retention_until math)
- consent_missing_token_rejected
- consent_wrong_token_rejected
- consent_subject_not_found_returns_404
- consent_missing_version_hash_400
- consent_missing_method_400
- consent_invalid_method_400
- consent_missing_operator_400
- consent_already_given_returns_409
- consent_post_withdrawal_requires_erase_returns_409
- consent_subject_inactive_returns_403
- consent_grant_then_upload_is_the_intended_intake_flow
  (end-to-end: grant → upload → verify chain has 2 rows
  consent_grant→biometric_collection in correct order, upload
  row chains off the grant's hmac)

71/71 catalogd lib tests + gateway crate compile clean.

Live verification post-restart:
- /audit/health + /biometric/health both 200
- Live POST returns 400 on missing/malformed body, 404 on ghost
  subject — auth + body parse + subject lookup ordering verified
- strings(target/release/gateway) contains: record_consent symbol,
  biometric_consent_response.v1, biometric_consent_grant,
  consent_already_given, consent_post_withdrawal_requires_erase,
  electronic_signature, /subject/{candidate_id}/consent route

Production posture: gateway running with the new endpoint live.
The candidate-facing consent UI is NOT yet built; that is a
separate session. This endpoint is the backend the UI will call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 13:21:12 -05:00
rootandClaude Opus 4.7 b2c34b80b3 phase 1.6: lock Gate 3b = C, reconcile docs to shipped state, fix double-upload file leak
Four threads landing together — all driven by the audit J asked for before
production cutover.

(1) Gate 3b DECIDED: Option C (defer classifications). `BiometricCollection.classifications`
    stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status
    flipped from "draft / awaits product" to DECIDED. Consent template + retention
    schedule revised to remove all "automated facial-classification" / "deepface"
    language so disclosed scope matches implemented scope.

(2) Endpoint-path drift reconciled across 3 docs. `PHASE_1_6_BIPA_GATES.md`,
    `BIPA_DESTRUCTION_RUNBOOK.md`, and `biometric_retention_schedule_v1.md` had
    references to legacy `/v1/identity/subjects/*` paths (proposed under a separate
    identityd daemon, never shipped) — corrected to actual shipped routes
    `/biometric/subject/*` (catalogd-local). Schema block in PHASE_1_6_BIPA_GATES
    rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate
    (not the proposed Postgres `subjects` table).

(3) New operational artifacts:
    - `scripts/staffing/verify_biometric_erasure.sh` — checks 4 things post-erasure
      (manifest cleared, uploads dir empty, audit row matches, chain verified).
      Smoke-tested live against WORKER-2.
    - `scripts/staffing/biometric_destruction_report.sh` — monthly anonymized
      destruction-event aggregation. Smoke-tested clean.
    - `scripts/staffing/bundle_counsel_packet.sh` — tarballs the counsel-review
      packet with per-file SHA-256 manifest.
    - `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` — formal rotation procedure
      operationalized after the 2026-05-05 /tmp wipe incident.
    - `docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md` — cover note bundling
      all eng-staged BIPA docs for counsel review with per-doc questions, sign-off
      checklist, recommended review sequence.

(4) Double-upload file leak fixed in `crates/catalogd/src/biometric_endpoint.rs`.
    `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo
    file. Investigation showed the file was 13-byte test-fixture bytes (zero PII,
    no biometric content); audit timeline showed two consecutive uploads followed
    by one erasure — the second upload had silently overwritten manifest.data_path,
    orphaning the first file. Patched `process_upload` to refuse a second upload
    with HTTP 409 + `error: "biometric_already_collected"` when
    `biometric_collection.is_some()` on the manifest. Operator must explicitly
    POST `/biometric/subject/{id}/erase` first.

    Tests: new `second_upload_without_erase_returns_409` (asserts 409 + manifest
    pointer unchanged + first file untouched on disk). Replaced
    `repeated_uploads_grow_the_chain` with `upload_erase_upload_grows_the_chain_cleanly`
    (covers the legitimate re-collection cycle: chain grows to 3 rows). Updated
    `content_type_with_parameters_accepted` to use 2 distinct subjects (was
    using 1 subject with 2 uploads to test ct parsing — would now 409).

    22/22 biometric_endpoint tests + 59/59 catalogd lib tests green post-patch.

Production posture: gateway needs `cargo build --release -p gateway` +
`systemctl restart lakehouse.service` to pick up the new 409 in live traffic.

Counsel calendar is now the only remaining blocker for first real-photo intake.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 06:19:40 -05:00
rootandClaude Opus 4.7 03e8a91d97 STATE_OF_PLAY: 2026-05-05 — audit endpoint recovery + anchor refresh
Reset gateway audit substrate after /tmp wipe disabled it on reboot:
- LH_SUBJECT_AUDIT_KEY moved /tmp/lakehouse_audit/ → /etc/lakehouse/
  (canonical persistent path per spec line 112; /tmp wipes on reboot
  and silently disabled /audit + /biometric endpoints)
- Fresh 32B HMAC + 44-char legal token at /etc/lakehouse/, mode 0400
- Systemd drop-in updated; gateway restarted; both endpoints 200
- Pre-rotation chains for WORKER-{1..5} (backfill data) will now
  tamper-detect under the new key — expected and correct on rotation

Anchor wave-table backfilled with 3 commits that landed after the
last STATE_OF_PLAY refresh on 2026-05-03 evening:
- 7e0112b: retention_sweep stray indent fix
- 848a458: Phase 1.6 Gate 5 erasure endpoint POST /biometric/.../erase
- 8ec43e0: Phase 1.6 Gate 3b deepface integration design doc

Phase 1.6 status table: Gate 5 → eng-DONE; Gate 3b → design-doc-shipped
(recommends Option C defer). Calendar bottleneck text updated.

.gitignore extended for runtime ephemera that surfaced this session:
- data/biometric/ (BIPA-quarantined photos, regulated data)
- reports/scrum/ (local-only review forensics per feedback_audit_findings_log.md)
- experiments/ (per "experiments stay out of tracked tree" policy)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:30:53 -05:00
rootandClaude Opus 4.7 8ec43e0721 phase 1.6 Gate 3b: deepface integration design doc (3 options + recommendation)
Per docs/PHASE_1_6_BIPA_GATES.md Gate 3b. Three viable paths for
populating BiometricCollection.classifications, sized + tradeoff'd:

  Option A — Python subprocess per upload (no daemon)
    ~80 LOC, 0.5-1 day. Smallest integration. Reintroduces a Python
    dependency the 2026-05-02 sidecar drop deliberately removed.

  Option B — ONNX models in Rust (no Python at all)
    ~200-400 LOC + model-build pipeline, 5-7 days. Fully consistent
    with sidecar drop. Need pre-trained models with appropriate
    licenses (or train ourselves, multi-week). Adds face detection
    preprocessing in Rust.

  Option C — Defer; classifications field stays None
    0.25 day. BIPA-safest position; substrate is forward-compatible.
    Forces the question "do we actually need classifications?" to be
    answered by a real product requirement, not by spec inertia.

Recommendation: **Option C (defer)**, conditional on confirming the
product requirement. Reasoning:
- All BIPA-load-bearing surfaces (consent + audit + retention +
  erasure) ship without classifications
- Riskiest BIPA position is collecting demographic-derived data
  without a documented business purpose
- Substrate accommodates A or B later in 1-3 days if real demand
  surfaces

Open questions for J at the bottom of the doc — pick A/B/C is the
gating decision before any engineering happens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 05:25:45 -05:00
rootandClaude Opus 4.7 848a4583da phase 1.6 Gate 5: erasure endpoint POST /biometric/subject/{id}/erase
Per docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md. BIPA-defensible
erasure: clears biometric collection (and optionally full PII record),
unlinks the photo file, records the destruction in the per-subject
HMAC chain. The audit row is the legal proof of compliant destruction
even after the underlying data is gone.

Two scopes:
  biometric_only (default): clears biometric_collection field, unlinks
    the photo, sets consent.biometric.status = withdrawn. Subject
    remains active.
  full: above PLUS sets status = erased and consent.general_pii.status
    = withdrawn. Manifest preserved (proof of destruction); subject
    record is logically erased.

Triggers (recorded but not validated against a closed set):
  retention_expiry | consent_withdrawal | rtbf | court_order

Body shape:
  {
    "trigger": "<token>",
    "trigger_evidence_path": "<optional path>",
    "operator_of_record": "<name>",
    "witness": "<name>",
    "scope": "biometric_only|full"
  }

Response (biometric_erase_response.v1):
  candidate_id, scope, trigger, erased_at, fields_cleared,
  photo_unlinked, photo_unlink_error, status_after,
  biometric_status_after, general_pii_status_after, audit_row_hmac

Order matters for BIPA defensibility:
  1. Snapshot original manifest (rollback target)
  2. Update manifest (logical erasure)
  3. Append audit row (LEGAL proof of intent + scope + operator)
  4. Best-effort secure overwrite + unlink photo file (irreversible last)

If audit append fails, manifest is rolled back to original state and
500 returned — the alternative (manifest erased without legal record)
is exactly the silent-failure mode the spec exists to prevent.

If photo unlink fails AFTER audit commits, the response carries
photo_unlinked=false + the error string; operator must manually shred.
Tracing logs the inconsistency loudly.

Tests: 21 unit tests now pass (10 erasure-specific):
  - missing token / missing subject / 404
  - missing trigger / missing operator / invalid scope (400)
  - biometric_only happy path (file unlinked, fields cleared, audit kind=biometric_erasure)
  - full scope (status=Erased, general_pii withdrawn, audit kind=full_erasure)
  - idempotent on already-erased (audit row records "already_erased" result)
  - no-photo case (photo_unlinked=true with no unlink error)
  - chain links off prior audit row's row_hmac (NOT GENESIS)

Live verification (post-restart):
  - POST /biometric/subject/WORKER-2/erase with consent_withdrawal trigger
    → 200 with all expected fields_cleared + photo_unlinked=true
  - Manifest reflects: biometric_collection=null, consent.biometric.status=withdrawn
  - GET /audit/subject/WORKER-2: chain_verified=true, 4 rows total,
    latest kind=biometric_erasure with operator + trigger in purpose field
  - Cross-runtime parity probe: 6/6 byte-identical post-change

Known follow-up (separate bug): photo upload endpoint overwrites
biometric_collection without handling a prior file's data_path —
multiple uploads for the same candidate orphan earlier files. The
erasure endpoint correctly unlinks what the manifest knows about;
operator must shred orphans manually until the upload endpoint
either rejects re-upload (preferred) or maintains a list.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 05:23:54 -05:00
root 7e0112beb7 retention_sweep: fix stray indent on biometric_collection field
Cosmetic — the field was added correctly during the BiometricCollection
substrate landing (commit f1fa6e4) but a batch sed left it misindented.
Tests still pass (8/8); only the formatting was off.
2026-05-03 05:16:49 -05:00
rootandClaude Opus 4.7 47a26fdaa8 STATE_OF_PLAY: 2026-05-03 evening wave (subject-manifest substrate + Phase 1.6 BIPA)
Documents the 13-commit wave end-to-end:
  - SUBJECT_MANIFESTS_ON_CATALOGD spec Steps 1-8 SHIPPED
  - 5/7 Phase 1.6 BIPA gates engineering-complete or eng-staged
  - 6th cross-runtime parity probe (subject_audit, 6/6 byte-identical)

Status table for Phase 1.6 with evidence pointers per item. Operational
state captures the LH_SUBJECT_AUDIT_KEY + LH_LEGAL_AUDIT_TOKEN_FILE
systemd configuration so next session knows what's in place.

Cataloged the three runtime-divergence classes the parity probe loop
caught + structurally killed (omitempty stripping, time.RFC3339Nano
truncation, json.Marshal HTML-escape). Future Go↔Rust work can
reference these patterns instead of rediscovering them.

Calendar bottleneck is now counsel review of items 1/2/5/6 — engineering
has staged everything it can without legal sign-off. Engineering long
pole is Gate 3b (deepface), deferred for design conversation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 05:11:52 -05:00
rootandClaude Opus 4.7 3708e6abf1 biometric endpoint: scrum-driven hardening
Per 2026-05-03 phase_1_6_gate_3a scrum (10 findings, 0 convergent
location-wise but opus + kimi flagged the same audit-failure issue).

Convergent + load-bearing fix:
  Audit-write failure was silently swallowed (returned 200 with empty
  hmac) after photo + manifest persisted. For BIPA defensibility this
  is wrong — a successful response without an audit row is exactly
  the silent-failure mode the spec exists to prevent. Now: full
  transactional rollback. If audit append fails after photo + manifest
  commit, we remove the photo AND revert the manifest to its
  pre-upload state, then return 500 with error="audit_write_failed".

Other real fixes:

  Orphan-file leak (opus WARN): if put_subject fails AFTER the photo
  is written, the file would orphan on disk with no manifest pointer.
  Now removes the photo on manifest-update failure, before returning 500.

  Content-Type parameter handling (opus WARN): real-world clients send
  `image/jpeg; charset=binary` etc. Parser now strips parameters per
  RFC 9110 §8.3 and matches case-insensitively. New regression test
  content_type_with_parameters_accepted exercises both.

  data_path doc/code mismatch (opus WARN): doc said "relative to the
  configured biometric storage root" but code stored absolute path.
  Now stores relative — operators reading the manifest reconstruct
  the absolute path with their own storage_root, manifests are
  portable across deployments. Tests updated.

  Timestamp-nanosecond collision (kimi WARN): added 8-char uuid
  suffix to filename. Sub-microsecond cadence collision was implausible
  but defense-in-depth is cheap.

  Dead code (opus + kimi INFO): removed unused require_legal_auth
  function (process_upload reimplements the auth check inline)
  and the `let _ = ConsentStatus::Given;` no-op type-shape reference.

Skipped (acceptable in v1):
  - qwen BLOCK on image format validation: spec explicitly says "we
    trust the caller; malformed images fail downstream when deepface
    runs in Gate 3b". Documented in the file's module doc-comment.
  - qwen WARN on directory create-then-chmod race: brief window
    between create_dir_all and set_permissions. Mitigation would
    require libc-level umask manipulation; accepted as v1 scope.
  - qwen INFO on constant_time_eq duplication: comment explains the
    cross-import boundary; acceptable short-term per the reviewer.

Tests: 11 unit tests pass (added content_type_with_parameters_accepted).
Live verification post-restart:
  - Content-Type with `; charset=binary` accepted ✓
  - data_path returned as relative `WORKER-2/<ts>_<uuid>.jpg` ✓
  - Chain verified end-to-end (3 rows: validator + 2 biometric) ✓
  - Cross-runtime parity probe still 6/6 byte-identical ✓

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 05:05:12 -05:00
rootandClaude Opus 4.7 f1fa6e4e61 phase 1.6 Gate 3a: photo upload endpoint with consent gate
Per docs/PHASE_1_6_BIPA_GATES.md §1 Gate 3 (consent-gate substrate).
Deepface classification (Gate 3b) deferred to its own session — needs
Python subprocess design conversation after the 2026-05-02 sidecar drop.

What ships:

  shared/types.rs:
    - new BiometricCollection sub-struct: data_path, template_hash,
      collected_at, consent_version_hash, classifications (Option<JSON>)
    - SubjectManifest gains biometric_collection: Option<BiometricCollection>
      with #[serde(default)] so existing on-disk manifests parse and
      re-emit without drift

  catalogd/biometric_endpoint.rs (NEW, ~600 LOC):
    POST /subject/{candidate_id}/photo
      - Auth: X-Lakehouse-Legal-Token, constant-time-eq compared against
        same legal token file as /audit. Same 32-byte minimum.
      - Content-Type: must be image/jpeg or image/png (415 otherwise)
      - Body: raw image bytes, max 10MB
      - 401: missing or wrong token
      - 404: subject not registered
      - 403: consent.biometric.status != "given" (returns current status)
      - 403: subject status in {Withdrawn, Erased, RetentionExpired}
      - 200: writes photo to data/biometric/uploads/<sanitized_id>/<ts>.<ext>
        with mode 0700 dir + 0600 file, updates SubjectManifest with
        BiometricCollection record, appends audit row
        (kind="biometric_collection", purpose="photo_upload"), returns
        UploadResponse with template_hash + audit_row_hmac.

    Logic split: pure async fn process_upload() takes the headers-as-args
    so unit tests exercise every branch without HTTP machinery; the
    axum handler is just glue. 10 tests covering all 4 reject paths +
    happy path + repeated uploads chaining + structural assertion that
    the quarantine path is NOT under data/headshots/ (synthetic faces).

  gateway/main.rs:
    Mounts /biometric on the same condition as /audit — only when the
    SubjectAuditWriter is present AND the legal token loads. Storage
    root configurable via LH_BIOMETRIC_STORAGE_ROOT (default
    ./data/biometric/uploads).

Live verification on the running gateway (post-restart):
  - GET  /biometric/health          → "biometric endpoint ready"
  - POST without token              → 401 auth_failed
  - POST with token, no consent     → 403 consent_required (status=NeverCollected)
  - Flipped WORKER-2 to consent=given, POST → 200 with hash + path
  - File at data/biometric/uploads/WORKER-2/<ts>.jpg, mode 0600
  - Manifest biometric_collection field reflects the upload
  - Audit row chain links cleanly off the prior validator_lookup row
  - GET /audit/subject/WORKER-2 returns chain_verified=true, 2 rows
  - Cross-runtime parity probe still 6/6 byte-identical post-change

Phase 1.6 status table updated: Gate 3a DONE, Gate 3b (deepface)
deferred. Calendar bottleneck remains counsel review of items 1/2/5/6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:55:32 -05:00
rootandClaude Opus 4.7 c7aa607ae4 phase 1.6 BIPA: scrum-driven fixes
Per 2026-05-03 phase_1_6_bipa_gates scrum (13 findings, 0 convergent).
1 BLOCK verified false positive, 4 real fixes shipped:

False positive (verified):
- opus BLOCK on attest:55 — claimed `set -uo pipefail` without `-e`
  makes the post-python3 `if [ $? -ne 0 ]` check unreachable. Verified
  WRONG: `X=$(false); echo $?` prints 1. Bash propagates command-
  substitution exit through $? on the assignment line. The check IS
  the python3 exit gate. Inline comment added to the script noting
  the false positive so future scrums don't re-flag.

Real fixes:
1. opus WARN attestation:18 — schema fingerprint hashed names ONLY,
   missing column-type changes. A column repurposed to hold base64
   photo bytes under its existing name would pass undetected. Now
   hashes "name<TAB>type<TAB>nullable=bool" per row. Re-run produced
   evidence SHA-256 1fdcc9f1... (vs old 230fffeb..., reflecting the
   broader fingerprint scope).

2. opus WARN gate_4_test:60 — definition regex didn't catch
   object-literal property forms (`const t = { FEMALE_NAMES: [...] }`)
   or TypeScript class fields (`class L { public NAMES_X: string[] = [] }`).
   Added two new patterns + a regression test
   (Gate 4: object-literal and class-field bypasses are caught) that
   exercises 5 bypass forms. 4/4 tests green; 1 minor regex tweak
   needed mid-fix to handle single-line class bodies.

3. kimi WARN python3-reliance — script assumed pyarrow installed and
   would emit a stack trace into the attestation if not. Added
   `python3 -c "import pyarrow"` gate at top with clean install
   instructions on failure.

4. opus INFO PHASE_1_6:200 — item 7 (training) silently dropped from
   blocking set with bare "deferred" rationale. Now explicitly states
   the deferral is conditional on small operator population (J + 1-2
   named ops); item 7 re-promotes to blocking if population grows.
   ⚖ COUNSEL marker added.

Skipped (acceptable as ⚖ COUNSEL placeholders by design):
- kimi WARN consent template:30-day-SLA (counsel decides number)
- kimi WARN consent template:email-placeholder (counsel supplies)
- kimi WARN parquet absence (env override exists; redeployment-aware)
- kimi INFO runbook manual-erasure (marked TODO when /erase ships)
- qwen INFO doc path/status nits (already addressed by file moves)

Tests: 4/4 Gate 4 absence test (incl. new bypass-coverage), 3/3
attestation evidence checks pass on live data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:43:17 -05:00
rootandClaude Opus 4.7 4708717f6b phase 1.6 BIPA gates — engineering wave (4 of 7 staged)
Per docs/PHASE_1_6_BIPA_GATES.md. Status table now reflects:

  DONE (engineering-only, no counsel dependency):
  - Gate 4: name→ethnicity inference removed from mcp-server.
    Removal note in search.html:3372 + new Bun absence test
    (mcp-server/phase_1_6_gate_4.test.ts) with 3 assertions:
    walker actually scans files, regex catches synthetic positives,
    no offending DEFINITION patterns in any .html/.ts/.js source.
    3/3 pass.

  ENG-DONE, signature pending:
  - §2 attestation: scripts/staffing/attest_pre_identityd_biometric_state.sh
    runs three checks against the live state:
      1. workers_500k.parquet schema has no biometric/photo/face/image col
      2. data/_kb/*.jsonl + pathway state contain no base64 image magic
         bytes (JPEG /9j/, PNG iVBOR), no data:image/* MIME prefixes,
         no field-name patterns ("photo", "biometric", "deepface_*")
      3. data/headshots/manifest.jsonl is entirely synthetic-tagged
    3/3 evidence checks pass on the live data dir. Generates a
    signed-by-operator+counsel attestation document committed at
    docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md
    with SHA-256 of the evidence summary so post-signature tampering
    is detectable.

  ENG-STAGED, awaiting counsel review:
  - Gate 1 retention schedule scaffold at
    docs/policies/consent/biometric_retention_schedule_v1.md (BIPA
    §15(a)). Engineering facts (categories, 18-month operational
    ceiling vs 3-year statutory cap, destruction procedure pointer
    to Gate 5 runbook) plus ⚖ COUNSEL markers for the binding text.
  - Gate 2 consent template scaffold at
    docs/policies/consent/biometric_consent_template_v1.md (BIPA
    §15(b)(1)-(3)). Required disclosures + plain-language summary +
    withdrawal procedure + the structured fields the consent UI must
    post to identityd.
  - Gate 5 destruction runbook at docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md.
    Triggers, pre-destruction checks (incl. chain-verified gate via
    /audit/subject/{id}), procedure (legal-tier endpoint), automatic
    audit row append (subject_audit.v1 with kind=biometric_erasure),
    backup-window disclosure, monthly reporting cadence, audit-trail
    attestation procedure cross-referencing the cross-runtime parity
    probe.

  BLOCKED on engineering design:
  - Gate 3 photo-upload endpoint. Requires identityd photo intake
    design + deepface integration scope. Deferred to its own session.

  DEFERRED:
  - §3 employee training material. Gate 5 runbook §7 may serve as
    substrate; counsel decides whether a separate program is needed.

Calendar bottleneck is now counsel review. Engineering can stage no
further deliverables until either (a) Gate 3's design conversation
happens or (b) counsel completes review of items 1/2/5/6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:38:49 -05:00
rootandClaude Opus 4.7 2222227c16 catalogd parity helper: scrum-driven hardening
Per 2026-05-03 step_7_8_retention_and_parity scrum (opus). 5 findings,
0 convergent — but two real fixes shipped:

1. WARN parity_subject_audit.rs:argv — replace .expect() panics with
   stderr+exit(2). The parity script captures stdout for byte-compare;
   a Rust panic backtrace lands in stdout (script merges 2>&1) and
   reads as a parity break instead of a usage error. Added die() helper
   that mirrors the Go side's error-exit pattern.

2. INFO parity_subject_audit.rs:5 — doc comment hardcoded the absolute
   path /home/profit/golangLAKEHOUSE/... Replaced with repo-relative
   reference.

INFO findings on retention_sweep argv style + --as-of report path
overwrite were noted but not actioned (style only / acceptable for
the forecast use case).

The major scrum-surfaced bug (Go json.Marshal HTML-escaping <>& while
serde_json keeps them literal) is fixed on the Go side in parallel
commit. Rust side here is correct as-is — serde_json::to_vec doesn't
HTML-escape by default, so no change needed in canonical_json.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:29:38 -05:00
rootandClaude Opus 4.7 2413c96817 catalogd: Step 8 — parity_subject_audit binary (Rust side)
Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.

Cross-runtime parity helper consumed by:
  golangLAKEHOUSE/scripts/cutover/parity/subject_audit_parity.sh

Two modes:

  --known-answer
    Print canonical-JSON + HMAC for a hardcoded fixture row. The Go
    helper at golangLAKEHOUSE/scripts/cutover/parity/subject_audit_helper/
    must produce byte-identical output. Catches algorithm drift
    (canonical-JSON sort order, HMAC algorithm, hex encoding).

  --verify <audit_log_path> --key <key_path>
    Replay the chain on a real production audit log via the live
    SubjectAuditWriter::verify_chain (no re-implementation; the actual
    production verification path). Output: one JSON line with mode,
    count, tip, verified, error.

The helper exercises the SAME verify_chain path the gateway calls, so
algorithm changes in subject_audit.rs automatically flow into the
parity probe.

Live-verified against 5 production audit logs in data/_catalog/subjects;
all 6 parity assertions pass after fixing two real cross-runtime drifts
on the Go side (omitempty trace_id stripping field; time.RFC3339Nano
stripping trailing zero in nanoseconds — both caught by this probe).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:16:50 -05:00
rootandClaude Opus 4.7 8fc6238dea catalogd: Step 7 — daily retention sweep binary
Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7:
  "Subjects whose retention.general_pii_until < now AND status != erased
   get marked for review (don't auto-delete; legal needs to approve)."

Per shared::types::BiometricConsent doc-comment (BIPA requirement on
biometric data, max 3 years from last interaction):
  "Implementation MUST enforce daily expiration sweep against this field."

Therefore the sweep checks BOTH retention clocks. Reports overdue
subjects to data/_catalog/subjects/_retention_sweep_<YYYY-MM-DD>.jsonl.

Idempotent: subjects already in {Erased, RetentionExpired} are skipped
so daily runs do not append duplicate rows.

Does NOT mutate subject manifests. Legal/operator owns the action
(extend, flip status, schedule erasure).

CLI:
  retention_sweep                       # dry-run (default), stderr only
  retention_sweep --apply               # also write JSONL report
  retention_sweep --as-of <RFC3339>     # alternate clock for forecast/test
  retention_sweep --storage-root <dir>  # default ./data

Tests: 8 unit tests on is_overdue covering all 5 SubjectStatus values,
both clocks, BIPA-only path, and idempotency on already-flagged
subjects.

Live verification (100 subjects in ./data/_catalog/subjects):
  - now (2026-05-03): 0 overdue (correct — 4-year retention)
  - --as-of 2031-06-01: 100 overdue, 394 days past, jsonl report shape
    verified with biometric fields correctly omitted via
    serde skip_serializing_if when subject has no biometric clock.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:05:03 -05:00
rootandClaude Opus 4.7 2a4b316a15 subjects: 2nd scrum fix wave (token min, chain_tip, tampering, rebuild collision warn)
Second cross-lineage scrum on Steps 5+6 returned 13 distinct findings, 0 convergent.
Three BLOCK-class claims verified as false positives (cache IS written, per-subject
Mutex IS in place, spawn IS safe under writer's lock). Five real fixes shipped:

1. audit_endpoint: legal token min length 16->32 (HMAC-SHA256 best practice, kimi)
2. subject_audit: new chain_tip() returns last hash from full log; audit_endpoint
   now reports chain_root from full chain instead of windowed slice (opus)
3. registry: rebuild loader now warns on sanitize collision (symmetric with
   put_subject's collision guard - opus)
4. audit_endpoint: tampering detection - if manifest expects non-empty chain_root
   but log returns 0 rows, flag chain_verified=false with explicit message (opus)
5. execution_loop::audit_result_state: tightened heuristic - error/denied/not_found
   only classify when no rows/data/results sibling (opus INFO)

Tests: 17 catalogd subject + 6 gateway audit_result_state, all green.
New: audit_result_state_does_not_classify_error_when_data_sibling_present,
     audit_result_state_status_is_authoritative_even_with_data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 04:00:42 -05:00
rootandClaude Opus 4.7 15cfd76c04 catalogd + gateway: Step 6 — /audit/subject/{id} legal-tier HTTP endpoint
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 6
+ §4 (response shape) + §6 (auth model). The defense-against-EEOC-
discovery surface is live: legal counsel hits one URL with one token,
gets back a signed-by-HMAC-chain audit response naming every PII access
for a subject in a time window.

New module: crates/catalogd/src/audit_endpoint.rs (~340 LOC)
  - AuditEndpointState { registry, writer, legal_token }
  - router() exposes:
      GET /subject/{candidate_id}?from=ISO&to=ISO  (full audit response)
      GET /health                                    (liveness + token check)
  - require_legal_auth() — constant-time-eq compare against the
    X-Lakehouse-Legal-Token header. Avoids timing leaks on the token
    check without pulling in `subtle` for one comparison.
  - Token loaded from /etc/lakehouse/legal_audit.token (env-overridable
    via LH_LEGAL_AUDIT_TOKEN_FILE). Empty file or <16 chars = endpoint
    serves 503 with a clear reason. Token value NEVER logged.
  - Response schema: subject_audit_response.v1 with manifest +
    audit_log (rows + chain verification) + datasets_referenced +
    safe_views_available + completeness_attestation.

New helper on SubjectAuditWriter:
  - read_rows_in_range(candidate_id, from, to) — returns rows in window,
    used by the endpoint to assemble the response without re-reading
    the entire chain.
  - verify_chain() now returns Ok(0) when the audit log file doesn't
    exist (empty = trivially valid). Prevents legitimate "no PII access
    yet for this subject" from showing as integrity=BROKEN in the
    audit response. Caller can detect "log was deleted" via comparison
    to SubjectManifest.audit_log_chain_root (when that mirror lands).

main.rs:
  - Audit endpoint mounted at /audit ONLY when both subject_audit
    writer AND legal token are present. Disabled-by-default keeps the
    surface from accidentally serving in dev/bring-up environments
    without proper credentials.

Tests (9/9 passing):
  - constant_time_eq (correctness on equal/diff/empty/length-mismatch)
  - missing_legal_token_returns_503
  - missing_header_returns_401
  - wrong_token_returns_401
  - correct_token_passes_auth
  - audit_response_assembly_full_path (manifest + 3 rows + chain verify)
  - audit_response_window_filters_rows (time-bounded window)
  - empty_token_file_results_in_disabled_endpoint
  - short_token_file_rejected_at_load (<16 char min)

LIVE end-to-end verification:
  1. Plant signing key + legal token in /tmp/lakehouse_audit/
  2. Restart gateway with LH_SUBJECT_AUDIT_KEY + LH_LEGAL_AUDIT_TOKEN_FILE
     pointing at the test files
  3. /audit/health → 200 "audit endpoint ready"
  4. /audit/subject/WORKER-1 (no token) → 401 "missing X-Lakehouse-Legal-Token"
  5. /audit/subject/WORKER-1 (wrong token) → 401 "X-Lakehouse-Legal-Token mismatch"
  6. /audit/subject/WORKER-1 (correct token) → 200 + full manifest + 0 rows
     + chain_verified=true (empty log path)
  7. POST /v1/validate with candidate_id=WORKER-1 → triggers WorkerLookup.find()
     via the AuditingWorkerLookup wrapper from Step 5
  8. data/_catalog/subjects/WORKER-1.audit.jsonl now exists with 1 row
     (accessor.purpose=validator_worker_lookup, result=not_found,
     prev_chain_hash=GENESIS, valid HMAC)
  9. /audit/subject/WORKER-1 (correct token) → 200 + manifest + 1 row +
     chain_verified=true + chain_rows_total=1 + completeness attestation

The full audit-trail loop (PII access → audit row → chain → audit response)
works end-to-end on the live gateway.

NOT in this commit (future steps):
  - Step 7: Daily retention sweep
  - Step 8: Cross-runtime parity (Go side reads the same shapes)
  - Mirror chain root to SubjectManifest.audit_log_chain_root after
    each append (so tampering detection can use the manifest's
    cached root as ground truth)
  - Live row projection from datasets (currently caller follows up
    via /query/sql against the safe_views named in the response)
  - Ed25519 signature on the response (chain verification IS the v1
    attestation; signing is future hardening per spec §10)

cargo build --release clean. cargo test -p catalogd audit_endpoint
9/9 PASS. Live verification successful.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:52:04 -05:00
rootandClaude Opus 4.7 cd8c59a53d gateway: Step 5 — wire SubjectAuditWriter into validator WorkerLookup
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5.
Every WorkerLookup.find() call from the validator path now produces
one audit row in the per-subject HMAC-chained JSONL. Failures are
non-blocking — validator continues whether audit succeeds or fails.

Approach: decorator pattern. WorkerLookup is a sync trait by design
(validator's contract is "in-memory snapshot, no per-call I/O") and
audit writes are async, so we can't expand the trait. Instead, a
new AuditingWorkerLookup wraps the inner lookup, captures a
tokio::runtime::Handle at construction, and spawns audit writes from
sync find() onto that handle. The chain stays intact under spawn fan-
out because the writer's per-subject Mutex (shipped in the previous
scrum-fix commit) serializes same-subject appends regardless of how
the spawn calls arrive.

Files changed:

  crates/gateway/src/v1/auditing_worker_lookup.rs (NEW, 175 LOC):
    - AuditingWorkerLookup<inner: dyn WorkerLookup, audit: Option<Arc<Writer>>>
    - new() captures Tokio Handle if audit is Some
    - find() runs inner lookup, then spawns audit append with:
        accessor.kind = "validator_lookup"
        accessor.purpose = "validator_worker_lookup"
        fields_accessed = ["exists"] (validator only proves existence
                          of a subject; downstream code reads policy
                          fields separately and would have its own
                          audit if those become PII)
        result = "success" if found, "not_found" otherwise
    - Audit-disabled path (audit: None) is a transparent passthrough
      — zero overhead, no panic, no runtime requirement.

  crates/gateway/src/v1/mod.rs:
    + pub mod auditing_worker_lookup;

  crates/gateway/src/main.rs:
    - Hoisted subject_audit_writer construction OUT of the V1State
      literal (declaration-order constraint: validate_workers needs
      access to the writer). The hoisted Arc is then reused for the
      V1State.subject_audit field.
    - validate_workers now wraps the raw lookup with
      AuditingWorkerLookup::new(raw, subject_audit_writer.clone())

Tests (4/4 passing):
  - find_existing_subject_writes_success_audit_row
  - find_missing_subject_writes_not_found_audit_row (phantom-id case)
  - audit_disabled_means_no_writes_no_overhead (None pathway)
  - many_finds_to_same_subject_produce_intact_chain (30 sequential
    spawns on the same subject — chain verifies all 30, regression
    against the race we fixed in catalogd subject_audit)

Also catches the iterate.rs:324 phantom-ID check transparently —
that codepath calls state.validate_workers.find(...) which now goes
through the wrapper, so every phantom-id rejection logs an audit row
for free.

NOT in this commit (future steps):
  - Step 6: /audit/subject/{id} HTTP endpoint
  - Step 7: Daily retention sweep
  - Threading X-Lakehouse-Trace-Id from request through to audit row
    (currently audit row's accessor.trace_id is empty)

cargo build --release clean. cargo test -p gateway auditing_worker_lookup
4/4 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:43:40 -05:00
rootandClaude Opus 4.7 e38f3573ff subject manifests Steps 1-4 — fix scrum-flagged BLOCKs and WARNs
2026-05-03 cross-lineage scrum on the subjects_steps_1_to_4 wave
returned 14 distinct findings, 0 convergent. opus verdict was HOLD
with 3 BLOCKs around the audit-chain integrity. All real. Fixed:

──────────────────────────────────────────────────────────────────
BLOCK 1 — opus subject_audit.rs:172 + execution_loop.rs:391
  Concurrency race: append_line is read-modify-write; the gateway
  hook used tokio::spawn fan-out → two concurrent appends to the
  same subject both read the same prev_hash, both compute their
  HMAC from the same prev, second write silently overwrites first
  → row lost AND chain broken.

  Fix:
  - SubjectAuditWriter gains per-subject Mutex map. append() acquires
    the subject's lock for the duration of the read-modify-write.
    Different subjects still parallelize.
  - Gateway hook switches from tokio::spawn to inline await. Per-row
    cost is ~1ms (one object_store put); inline is correct AND cheap.
  - New regression test: 50 concurrent appends to the same subject,
    asserts all 50 land with intact chain.

BLOCK 2 — opus subject_audit.rs:108
  Non-deterministic canonicalization: serde_json serializes struct
  fields in declaration order. Schema evolution (adding/reordering
  fields) silently changes the bytes verify_chain hashes → chain
  breaks even when nothing was actually tampered with.

  Fix:
  - New canonical_json() free fn — recursive value rewrite to sort
    object keys alphabetically (BTreeMap projection), arrays preserve
    order, scalars pass through. Stable across struct evolution.
  - Both append() and verify_chain() now compute HMAC over canonical
    bytes, not declaration-order bytes.
  - New regression tests: alphabetical-key + array-order-preserved.

WARN — opus execution_loop:401
  Audit row's `result` was hardcoded to "success" for every Ok(result)
  including payloads like {"error":"not found"}. Misleads compliance.

  Fix:
  - New audit_result_state() free fn that inspects the payload
    top-level for error/denied/not_found/status signals (per spec
    §3.2 enum). Defaults to "success" only when no error signal.
  - 4 new tests covering each enum case + falsy-signals defense.

WARN — opus registry.rs:735
  Storage-key collision: sanitize_view_name(id) is the disk key,
  but the in-memory HashMap was keyed by raw candidate_id. Two
  distinct ids that sanitize to the same key (e.g. "CAND/1" and
  "CAND_1") would collide on disk while appearing distinct in
  memory; second put silently overwrites first; rebuild loads only
  one.

  Fix:
  - put_subject() / get_subject() / delete_subject() / rebuild()
    all key the in-memory HashMap by sanitize_view_name(id), matching
    the storage key shape.
  - Collision guard: put_subject() refuses (with clear error) when
    the sanitized key matches an EXISTING subject with a DIFFERENT
    raw candidate_id.
  - New regression test: put("CAND/1") then put("CAND_1") errors
    + first subject survives.

WARN — opus backfill_subjects.rs:189
  trim_start_matches strips REPEATED prefixes; the spec wanted
  one-shot semantics. Edge case unlikely in practice but real.

  Fix:
  - Switched to strip_prefix(&prefix).unwrap_or(&cid). One-shot.

INFO — opus subject_audit.rs:131
  Per-byte format!("{:02x}", b) allocates each iteration. Hot path
  on every append.

  Fix:
  - Replaced with const HEX lookup table + push() into preallocated
    String. Same output bytes, no per-byte allocation.

──────────────────────────────────────────────────────────────────
Test summary post-fix:
  catalogd subject_audit: 11/11 PASS (added 4 new — concurrency
                          race regression, parallel-different-subjects,
                          canonical-key sort, canonical-array order)
  catalogd registry subject: 6/6 PASS (added 1 new — collision guard)
  gateway execution_loop subject: 10/10 PASS (added 4 new —
                          audit_result_state enum coverage)

  All 27 subject-related tests green. cargo build --release clean.

The convergent-zero scrum result was misleading on its face — opus
caught real BLOCKs that kimi/qwen missed. Per
feedback_cross_lineage_review.md: opus is the load-bearing reviewer;
single-opus BLOCKs warrant manual verification, which here confirmed
all three were correct.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:37:45 -05:00
rootandClaude Opus 4.7 fef1efd2ac gateway: Step 4 — wire SubjectAuditWriter into tool dispatch
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 4.
Every tool result that returns rows referencing a subject now produces
audit rows in the per-subject HMAC-chained JSONL. Failures non-blocking.

Changes:

v1/mod.rs:
  + V1State.subject_audit: Option<Arc<SubjectAuditWriter>>
    (None when key file missing — audit becomes a no-op with warning,
    PII paths still serve)

main.rs:
  + Construct SubjectAuditWriter at startup from
    LH_SUBJECT_AUDIT_KEY env or /etc/lakehouse/subject_audit.key.
    Missing/short key = log warning + leave None (gateway boots, audit
    disabled). Same store as the rest of catalogd.

execution_loop/mod.rs:
  + audit_subject_hits_in() — called after every successful tool
    dispatch. Walks the result JSON, finds candidate_id / worker_id
    fields, fires one SubjectAuditRow per (subject, fields) pair.
    Tokio::spawn so audit latency never adds to tool path.
  + collect_subject_hits() — free fn, recursive JSON walker. Handles:
      "candidate_id":"X"  → audit candidate_id="X"
      "worker_id":42      → audit candidate_id="WORKER-42" (matches
                              backfill convention)
      "worker_id":"42"    → audit candidate_id="WORKER-42" (string form)
    Other fields in the same object become fields_accessed (so audit
    row records "this access surfaced name + phone for candidate X").
    Ignores objects without id fields. Skips empty id strings. Recurses
    through nested objects + arrays.

Tests (6/6 passing — gateway::collect_subject_hits_*):
  - finds_candidate_id_strings (basic case + fields_accessed extraction)
  - prefixes_worker_id_int (int → WORKER-N)
  - handles_worker_id_string (string → WORKER-N)
  - recurses_through_nested_objects (joins / mixed payloads)
  - ignores_objects_without_id_fields (no false positives)
  - skips_empty_id_strings (defensive)

Per spec §3.2: failures are logged, never propagated. Better to leak
an audit row than block a tool response. Operators monitor warning
volume to detect audit-write regressions.

NOT in this commit (future steps):
  - Step 5: Wire validator WorkerLookup similarly (each candidate_id
    resolved by FillValidator gets an audit row)
  - Step 6: /audit/subject/{id} HTTP endpoint
  - Step 7: Daily retention sweep
  - Mirror chain root to SubjectManifest.audit_log_chain_root after
    each append (currently the chain is verifiable via verify_chain()
    even without the manifest mirror; the mirror is an optimization)
  - Thread X-Lakehouse-Trace-Id from request through to audit row

cargo build --release clean. cargo test -p gateway collect_subject_hits
6/6 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:29:24 -05:00
rootandClaude Opus 4.7 bce6dfd1ee catalogd: Step 3 — backfill_subjects binary (BIPA-defensible defaults)
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3.
Reads a parquet source, creates one SubjectManifest per row with the
spec-defined safe defaults, persists via Registry::put_subject().

Defaults baked in (per spec §2 + §5 Step 5):
  - vertical = unknown                     (HIPAA fail-closed)
  - consent.general_pii = pending_backfill_review  (NOT inferred_existing — BIPA defense)
  - consent.biometric  = never_collected   (no biometric data backfilled)
  - retention.general_pii_until = now + 4 years
  - retention.policy = "4_year_default"

Conservative ergonomics:
  - --limit 1000 by default. --all to do the full source.
  - --dry-run for parse + count + sample without writes.
  - --concurrency 32 (bounded via tokio::sync::Semaphore).
  - Idempotent: skips subjects that already exist in catalog.
  - Progress reports every ~5% (or 5K rows, whichever smaller).

Live verification on workers_500k.parquet:
  --limit 100 dry-run:  parsed 100 rows, sampled WORKER-1..5, 0 writes ✓
  --limit 100 commit:   100 inserted, 0 failed, 100 files in
                        data/_catalog/subjects/ ✓
  --limit 100 re-run:   0 inserted, 100 skipped (idempotent) ✓

Sample manifest (data/_catalog/subjects/WORKER-1.json):
  {
    "schema": "subject_manifest.v1",
    "candidate_id": "WORKER-1",
    "status": "active",
    "vertical": "unknown",
    "consent": {
      "general_pii": {"status": "pending_backfill_review", ...},
      "biometric":   {"status": "never_collected",         ...}
    },
    "retention": {"general_pii_until": "2030-05-02T...", "policy": "4_year_default"},
    "datasets": [{"name": "workers_500k", "key_column": "worker_id", "key_value": "1"}]
  }

NOT in this commit (future steps):
  - Step 4: Wire gateway tool registry to write audit rows on every
    candidate_id returned (uses SubjectAuditWriter from Step 2)
  - Step 5: Wire validator WorkerLookup similarly
  - Step 6: /audit/subject/{id} HTTP endpoint
  - Step 7: Daily retention sweep
  - Backfill the full 500K (operator decision: --all when ready;
    note: 500K JSON files in one dir will slow startup load — may
    want SQLite/single-file backend before that scale)

Operator note: backfill is run-once. To extend to candidates table,
re-run with --dataset candidates --key-column candidate_id (no prefix
since candidate_id is already the canonical token there).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:22:54 -05:00
rootandClaude Opus 4.7 d16131bcab catalogd: Step 2 — SubjectAuditWriter with HMAC chain
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 2.
Per-subject append-only audit JSONL with HMAC-SHA256 chain. Local-first
— no Vault, no external anchor (those are v2 if SOC2 Type II becomes
contract-required; v1 deliberately stays small).

shared/types.rs additions:
- AuditAccessor — kind, daemon, purpose, trace_id
- SubjectAuditRow — schema/ts/candidate_id/accessor/fields_accessed/
  result/prev_chain_hash/row_hmac

crates/catalogd/src/subject_audit.rs (NEW):
- SubjectAuditWriter — holds signing key + per-subject latest-hash cache
- from_key_file() — loads key from sealed file, requires ≥32 bytes
- with_inline_key() — for tests + bring-up
- append() — computes HMAC chain link, persists JSONL row, returns new
  chain root (caller mirrors to SubjectManifest.audit_log_chain_root)
- verify_chain() — full re-verification of a subject's audit log,
  catches both prev_hash drift AND row-level HMAC tampering
- scan_latest_hash() — cold-start path, finds prev_hash from JSONL tail
- append_line() — read-modify-write pattern (object stores have no
  native append; same shape as the rest of catalogd's persistence)

Crypto: HMAC-SHA256 via the standard `hmac` crate (added to workspace
+ catalogd deps; not implementing crypto by hand). Output is lowercase
hex matching the rest of the codebase's SHA-256 conventions.

Security choices:
- NO Debug impl on SubjectAuditWriter — auto-deriving Debug would risk
  leaking the signing key into log lines. Tests work around this by
  matching on Result instead of using .unwrap_err().
- Key min length 32 bytes (HMAC-SHA256 block size guidance).
- Failures are NOT swallowed — Result returned, caller decides whether
  to log + continue (per spec §3.2 the gateway tool registry SHOULD
  log + continue rather than block reads).

Tests (7/7 passing):
- first_append_uses_genesis_prev_hash
- chain_links_each_append (3-row chain verifies)
- separate_subjects_have_independent_chains (per-subject isolation)
- tamper_detected_on_verify (mutation in middle of chain breaks verify)
- cold_writer_picks_up_existing_chain (process restart preserves chain)
- empty_candidate_id_rejected
- key_too_short_rejected_via_file

NOT in this commit (future steps):
- Step 3: Backfill ETL from workers_500k.parquet (next per J)
- Step 4: Wire gateway tool registry to call append() on every
  candidate_id returned by search_candidates / get_candidate
- Step 5: Wire validator WorkerLookup similarly
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
- Mirroring chain root to SubjectManifest.audit_log_chain_root
  (separate concern; do at the call site)

cargo check --workspace clean. cargo test -p catalogd subject_audit
7/7 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:19:18 -05:00
rootandClaude Opus 4.7 d25990982c catalogd: Step 1 — SubjectManifest type + Registry CRUD
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 1.
Mirrors the existing AiView put/get/list/delete pattern. NOT a separate
daemon, NOT new infrastructure — extends catalogd's manifest layer with
a fourth manifest type (subject) alongside dataset/view/tombstone/profile.

shared/types.rs additions:
- SubjectManifest (the wire format from spec §2)
- SubjectStatus enum: pending_consent | active | withdrawn |
  retention_expired | erased
- SubjectVertical enum: unknown | general | healthcare | finance | other
  (default = Unknown for fail-closed routing per spec §2.1)
- ConsentStatus enum: pending_backfill_review | pending_first_contact |
  given | withdrawn | expired
- BiometricConsentStatus enum: never_collected | pending | given |
  withdrawn | expired
- GeneralPiiConsent + BiometricConsent + SubjectConsent
- SubjectRetention (general_pii_until + policy)
- SubjectDatasetRef (name + key_column + key_value pointing at existing
  catalogd dataset manifests)

catalogd/registry.rs additions:
- subjects: Arc<RwLock<HashMap<String, SubjectManifest>>> field on Registry
- put_subject() — validates dataset refs, persists to
  _catalog/subjects/<id>.json, updates in-memory cache
- get_subject() / list_subjects() / delete_subject() / subjects_count()
- rebuild() now loads subject manifests at startup alongside views +
  profiles + tombstones

Tests (5/5 passing):
- put_subject_with_no_dataset_refs_succeeds
- put_subject_rejects_dangling_dataset_ref (validation works)
- put_subject_with_valid_dataset_ref_succeeds
- subject_round_trips_through_object_store (persistence works)
- delete_subject_removes_in_memory_and_persistence

NOT in this commit (future steps):
- Step 2: SubjectAuditWriter with HMAC chain
- Step 3: Backfill ETL from workers_500k.parquet
- Steps 4-5: Wire gateway tool registry + validator to write audit rows
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep

cargo check --workspace clean. cargo test -p catalogd subject 5/5 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:13:08 -05:00
rootandClaude Opus 4.7 ed1fcd3c26 specs: pathway_memory v1 + subject_manifests_on_catalogd v1
Two specifications addressing the framing J asked for after reading
the llms3.com blog: standardize what we have so future work doesn't
drift, and apply the local-first thesis to the audit problem instead
of the over-scoped SaaS-tier identity service.

PATHWAY_MEMORY_SPEC.md (~400 lines):
  Documents the existing crates/vectord/src/pathway_memory.rs as a
  spec — the third metadata layer alongside catalogd's data metadata
  and playbook_memory's operational memory. Defines:
    - PathwayTrace wire format
    - pathway_id = SHA256(task_class | file_prefix | signal_class)
    - file_prefix algorithm (first 2 path segments)
    - pathway_vec: 32-bucket bag-of-tokens hash, fixed dim per spec
    - Lifecycle: insert → revise → replay → probation gate retire
    - Mem0 versioning (trace_uid + parent_trace_uid + version chain)
    - Access patterns: query_for_hotswap / query_by_vec / list_versions
    - PII risk surface (reducer_summary + final_verdict)
    - Spec boundary: stable in v1 vs implementation-specific
  No new architecture. Descriptive, not prescriptive.

SUBJECT_MANIFESTS_ON_CATALOGD.md (~400 lines):
  The local-first audit-trail spec. Adds a fourth manifest type to
  catalogd alongside dataset/view/tombstone/profile. NOT a separate
  identity daemon. NOT Vault/KMS/dual-control JWT. Builds on
  primitives catalogd already ships:
    - SubjectManifest at data/_catalog/subjects/<id>.json
    - Per-subject HMAC-chained audit JSONL
    - Daily retention sweep using existing tombstone primitives
    - Vertical-aware routing (healthcare → local-only)
    - Legal-tier credential separate from gateway internal auth
  ~4 days estimated implementation effort vs 17-20 days for the
  IDENTITY_SERVICE_DESIGN approach. Same defensibility for the
  staffing-client launch window. Strictly additive to compatibility
  with the v3 design if SOC2 Type II becomes a contract requirement.

These are SPECS — what the system already does (pathway) and what's
the smallest local-first thing that addresses the audit need
(subject manifests). Not 9-phase plans. Not new daemons.

The pathway spec is descriptive: writing down what exists so the
next person doesn't reinvent it. The subject-manifests spec is
prescriptive: J greenlights, implementation is days not weeks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:07:38 -05:00
rootandClaude Opus 4.7 991db7be1a scrum: wrapper resilience + Python tz deprecation fix
Two small fixes after testing the tool end-to-end:

1. scrum_review.sh's tally-aggregation step occasionally exits non-zero
   even when the per-reviewer markdown verdicts are all written
   correctly. Wrapper was bailing on that exit code and dropping the
   KB write. Now the wrapper:
   - Treats scrum_review.sh's exit code as advisory
   - If a tally markdown exists, uses it
   - If only per-reviewer markdowns exist, auto-rebuilds a minimal
     tally listing them (so KB row still gets written)
   - Only bails if NO per-reviewer verdicts at all

2. Python `datetime.utcnow()` deprecated. Switched to
   `datetime.now(datetime.UTC).isoformat().replace("+00:00", "Z")`
   for the same Z-suffix shape callers expect.

Verified: ./scrum --since=HEAD~1 now writes KB row even when
scrum_review.sh's tally step has issues. KB rows: 2 (and growing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:44:45 -05:00
rootandClaude Opus 4.7 dbcd05c5c5 audit docs: deprecation headers — over-scoped for local-only deployment
Today's PRD-line-70 reframe (everything runs locally) means the audit-trail
docs I drafted earlier this session are over-engineered for J's actual
deployment model. They were sized for SaaS-tier infra (Vault/KMS/S3
Object Lock/dual-control JWT/separate Postgres) — appropriate for a
multi-tenant cloud service, wrong for a single-box local install.

Adding clear deprecation headers so future sessions don't read these
as authoritative and propose another 17-20 day plan involving cloud
infrastructure that would re-violate PRD line 70.

What STAYS valid (preserved in headers):
- The legal use case (John Martinez worked example)
- The IL/IN jurisdictional surface (counsel checklist)
- The Phase 1 + 1.5 discovery findings (PII flow paths file:line)
- Phase 1.6 BIPA gates (when real photos arrive)

What's OVER-SCOPED (flagged in headers):
- The 9-phase implementation plan
- The identity service design (Vault/KMS/dual-control)

Future v2 of these docs needs to be sized for local single-box: a few
hundred LOC of local writers + signed local audit file, not 17-20 days
of distributed-systems design.

No code changes. Just doc-level guardrails for future scope drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:42:05 -05:00
rootandClaude Opus 4.7 5f40b7a312 STATE_OF_PLAY: lock in today's reverts as DO NOT RELITIGATE
Five new entries to prevent today's cleanup from being undone by future
sessions or future PRs that don't read the full context:

- PRD line 70 load-bearing — local-only on customer hot path. PR #13's
  cloud-routing defaults reverted (d054c0b). Cloud is opt-in dev-only.
- /v1/usage by_provider=ollama is the canary. Anything else for
  customer-shape traffic = regression.
- ./scrum is a TOOL, not architecture. Outputs to data/_kb/
  scrum_findings.jsonl. Findings inform dev, do NOT auto-fold into
  design docs.
- Test code in main is actively being cleaned. Today: 12 files / ~2900
  LOC removed (commits 6aafd41 + f4ebd22). Surface more candidates,
  don't auto-delete unless clearly orphaned.

The intent: future me (or future Claude session) reads STATE_OF_PLAY
on cold-start, sees these entries, and doesn't re-make the same
mistakes that drifted scope today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:07:57 -05:00
rootandClaude Opus 4.7 f4ebd2278b remove 7 more orphaned experimental scripts from scripts/
Continuing the test-code-in-main cleanup. These are sequential mode-runner
experiment passes (2/3/4/5) that completed and whose findings were captured
in pathway_memory + the matrix index — the scripts themselves are dead
weight. Plus two one-off probe scripts.

Removed (all 0 refs in production code or automation):
- mode_pass2_corpus_sweep.ts     — 2026-04 corpus sweep experiment
- mode_pass3_variance.ts         — variance measurement run
- mode_pass4_staffing.ts         — staffing-domain pass
- mode_pass5_summarize.ts        — summarization variance
- mode_pass5_variance_paid.ts    — paid-model variance
- overnight_proof.sh             — overnight stress probe (output in logs/)
- ab_t3_test.sh                  — T3 overseer A/B test (output captured in KB)

Verified: 0 references in package.json / justfile / Makefile / any active
.ts/.rs/.sh file. Two mentions remain in docs/recon and docs/MODE_RUNNER_
TUNING_PLAN — those are historical design-doc references, not consumers.

KEPT in scripts/ (have live consumers OR are runtime tools):
- mode_experiment.ts (14 refs), mode_compare.ts (7 refs)
- lance_smoke.sh, build_*_corpus.ts, staffing_demo.py, lance_tune.py,
  generate_demo.py, generate_workers.py, copilot.py, kb_measure.py,
  kb_staffer_report.py, analyze_chicago_contracts.ts, dump_raw_corpus.sh,
  check_phase44_callers.sh, autonomous_agent.py, build_answers_corpus.ts,
  build_lakehouse_corpus.ts, build_scrum_findings_corpus.ts,
  build_symbols_corpus.ts, e2e_pipeline_check.sh, scale_test.py,
  scale_10m_test.sh, run_staffer_demo.sh, stress_test.py

Build clean. If any of these are needed back: git show HEAD~1 -- scripts/<file>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:06:29 -05:00
rootandClaude Opus 4.7 6aafd41785 remove 5 orphaned dev experiments from tests/real-world/
Per J: "all of our test code ended up in the main." These are 5 one-time
dev experiments that were never wired into any automation and have zero
live consumers in the production code path. Deleting them.

Removed (1418 LOC total):
- enrich_prd_pipeline.ts      (528 LOC) — Phase 21 architecture stress test
- nine_consecutive_audits.ts  (185 LOC) — empirical study of audit compounding
- hard_task_escalation.ts     (267 LOC) — escalation-ladder test (refs retired
                                            cloud models qpt-oss:20b/120b)
- autonomous_loop.ts          (214 LOC) — wrapper experiment around scrum_master
- consensus_reducer_design.ts (224 LOC) — N=3 design consultation; output JSON
                                            referenced from pathway_memory.rs
                                            comment but the script itself has
                                            no consumer

Verified: 0 references in package.json / justfile / Makefile / any
production .ts/.rs/.sh file. The single mention from pathway_memory.rs
is a //! doc comment referencing the JSON output (data/_kb/
consensus_reducer_design_*.json), not the script. Build clean post-delete.

KEPT:
- scrum_master_pipeline.ts — referenced from observer.ts, vectord, scripts
- scrum_applier.ts          — referenced from auditor schemas

If you need any of these back, they're in git history. cherry-pick or
git show HEAD~1 -- tests/real-world/<file>.ts will recover the source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:05:24 -05:00
rootandClaude Opus 4.7 bb5a3b3f5e execution_loop: align overseer log/KB strings with reverted local route
Yesterday's revert (d054c0b) changed the API CALL from cloud to local
but missed the LogEntry + KB row that record what model fired. Result:
honest API call to qwen3.5:latest, dishonest log/KB rows saying
"claude-opus-4-7". That's a real audit-trail integrity issue — the
record didn't match reality.

Fixed:
- LogEntry "system" role label (line 663)
- KB row's "model" field (line 685)

Both now correctly show "qwen3.5:latest". Build + restart + smoke
10/10 green. Gateway healthy.

Side note: the only remaining "claude-opus-4-7" mentions in this file
are now in COMMENTS describing the v1 cloud route + the revert
rationale — those are documentation, not log fields. Safe to keep.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:03:06 -05:00
rootandClaude Opus 4.7 a44ccde845 observer: overseer fallback label → qwen3.5:latest (matches reverted route)
Mirror of yesterday's execution_loop overseer revert (commit d054c0b).
The observer logs an "overseer:<model>" endpoint string for analysis;
when row.model is missing it falls back to a hardcoded label. PR #13
set that fallback to "claude-opus-4-7" — but the route now goes to
local Ollama qwen3.5:latest, so the label was wrong.

Trivial one-line fix, no behavior change. Just keeps observer's
endpoint string honest when older rows from the cloud-routing window
get re-analyzed.

End-to-end verification of the local hot path (post-revert):
  BEFORE  /v1/usage by_provider: []
  AFTER   /v1/usage by_provider: [{"k":"ollama","v":2}]
  → /v1/iterate fired 2 chat calls, both to local ollama
  → ZERO cloud requests (no kimi/openrouter/opencode/ollama_cloud)
  → API meter on cloud providers stays at 0 for customer requests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 02:01:18 -05:00
rootandClaude Opus 4.7 2f5ca95875 scrum: real tool — auto-bundle current diff, run 3-reviewer scrum, push to KB
J's frustration captured: scrum was a TOOL meant to find gaps in the work
and push findings to a KB that informs how WE work on the code. It became
welded into architecture instead of staying a tool. This commits the tool
in the form J actually meant.

Usage:
  ./scrum                       auto-bundle origin/main..HEAD, auto-label
  ./scrum my_label              same with explicit label
  ./scrum --staged              bundle staged-only diff (pre-commit)
  ./scrum --since=COMMIT        bundle from a specific commit

Output:
  KB row    → data/_kb/scrum_findings.jsonl  (one row per scrum run)
  Verdicts  → reports/scrum/_evidence/<date>/verdicts/<label>_*.md

The KB row carries: timestamp, label, diff size, findings count,
convergent count, branch, head SHA, tally excerpt, paths to full
verdicts. Queryable via jq or DuckDB.

Cloud models (opus + kimi-k2 + qwen3-coder) are used here BY DESIGN
— 3-lineage cross-review needs distinct training corpora. This is dev
tooling, not the runtime hot path. PRD line 70 (no cloud APIs) applies
to customer requests, not to J's dev tools.

Tested live: ./scrum --since=HEAD~1 revert_only → 7 findings, 1 convergent
(INFO), all 3 reviewers VERDICT: ship for the revert commit.

KB row written:
  {"label":"revert_only","findings_total":7,"findings_convergent":1,
   "diff_bytes":5806,"branch":"demo/post-pr11-polish-2026-04-28",...}

What scrum does NOT do (intentionally):
- It does NOT auto-fold findings into architecture / design docs
- It does NOT block any commit/push
- It does NOT mutate any code
- It does NOT run as part of normal customer requests

It reports. J reads. J decides what to do with the findings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:59:32 -05:00
rootandClaude Opus 4.7 d054c0b8b1 REVERT cloud routing on hot path — back to local Ollama per PRD line 70
PRD line 70: "Everything runs locally — no cloud APIs, total data privacy."
Yesterday's PR #13 (feb638e) violated this by routing customer-facing
inference paths to opencode + ollama_cloud + openrouter. Reverting the
hot-path routes only; cloud providers stay configured in providers.toml
for explicit dev-tool opt-in.

Reverted:
- modes.toml staffing_inference: kimi-k2.6 → qwen3.5:latest (local Ollama)
- modes.toml doc_drift_check: gemini-3-flash-preview → qwen3.5:latest
- execution_loop overseer: opencode/claude-opus-4-7 → ollama/qwen3.5:latest
  Was a paid Anthropic call on every overseer escalation; now local + free.

Gateway compiles + restarts clean. Lance smoke 10/10. Live providers list
unchanged (kimi/ollama_cloud/opencode/openrouter all still CONFIGURED;
they just aren't ROUTED to from the staffing inference path anymore).

This stops the API meter on customer requests. Cloud providers remain
opt-in via explicit provider= caller hint, which the scrum tool +
auditor pipeline + bot/propose use deliberately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:57:20 -05:00
rootandClaude Opus 4.7 0c74b82fc8 phase 1.6 gate 4: REMOVE name → ethnicity / gender inference
Per docs/PHASE_1_6_BIPA_GATES.md Gate 4 + AUDIT_TRAIL_PRD §4 protected-
attribute exclusion rule. The lookup tables + inference functions in
search.html (3375-3499) and console.html (245-311) were dead code in
the rendering path — headshot rendering disabled 2026-04-28 left these
functions defined but unused. Removing them forecloses both Title VII
discriminatory-feature-engineering AND BIPA biometric-information-
derived-from-biometric-identifier arguments.

Removed:
- FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK,
  NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN
- SURNAMES_HISPANIC, SURNAMES_SOUTH_ASIAN, SURNAMES_EAST_ASIAN,
  SURNAMES_MIDDLE_EASTERN, SURNAMES_BLACK
- guessGenderFromFirstName(), guessEthnicityFromName(),
  guessEthnicityFromFirstName(), genderFor()

From both search.html and console.html. Replacement: deprecation
comment block referencing the BIPA gates doc.

Verified: zero live consumers anywhere in mcp-server/. Searched for
genderFor()/guessEthnicityFromName()/guessEthnicityFromFirstName()/
guessGenderFromFirstName() call sites — none remain.

Per J 2026-05-03: this kind of test-code-leaked-into-main is exactly
what J wants cleaned up. The face-pool inference was meant as a
testing tool for synthetic icon generation but ended up as
production-shape inference logic in the customer-facing UI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:48:21 -05:00
rootandClaude Opus 4.7 cd440d4cee audit phase 1.6: BIPA pre-launch gates — block identity-service backfill
Per IDENTITY_SERVICE_DESIGN v3 §5 Step 0, Phase 1.6 is hard
prerequisite to identityd backfill. This doc specifies the 5 gates +
2 supporting deliverables that must ship before real-photo intake.

Five gates (BIPA §15 compliance):
1. Public retention schedule — counsel writes; engineering files+hash
2. Informed written consent — counsel writes template; engineering
   wires identityd consent-status enforcement
3. Photo-upload endpoint with consent enforcement — POST /v1/identity/
   subjects/{id}/photo with hard 403 when biometric_consent_status
   != 'given'; quarantined storage path; deepface output isolated
   to identityd subjects table (not synthetic-face manifest)
4. Deprecate name → ethnicity inference (mcp-server/search.html
   lookup tables removed; Phase 1.5 §1B finding closed)
5. Destruction runbook — operator-facing; ties to identityd
   /erase endpoint with biometric-specific erasure path; daily
   sweep job for biometric_retention_until expiry

Plus:
- Cryptographic attestation that no biometric data exists
  pre-identityd (per v3-B11) — defends against
  infrastructure-as-notice plaintiff argument
- Employee BIPA-handling training acknowledgment

Engineering effort: ~4-5 days (one week to stage everything ready).
Counsel effort: ~3-6 weeks calendar (review cycles dominate).
Calendar bottleneck is counsel, not engineering.

Phase 1.6 exit = 7 checked gates + signoffs. Until done, identityd
backfill cannot proceed (per identity service design v3 §5 Step 0).

5 open questions for J + counsel: photo-upload UX, consent
mechanism (DocuSign/click/paper), named operator list, named
counsel for sign-off, public privacy policy URL.

No code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:41:29 -05:00
rootandClaude Opus 4.7 8129ddd883 identity service: v3 amendments — second-pass scrum BUILD-WITH-CHANGES
Re-scrummed v2 across opus + kimi + gemini. All 3 verdict:
BUILD-WITH-CHANGES. v1 blockers verified RESOLVED. 12 new v2
findings folded as v3 amendments in §12.

Convergent v2 findings (≥2 reviewers):
v3-A1: mTLS CA root must NOT live in identityd (opus + gemini).
       v3 fix: Vault PKI for CA, identityd as intermediate.
v3-A2: Dual-control public key registry must be tamper-evident
       (opus + gemini). v3 fix: Vault KV with separate access
       policies + server-issued nonces for replay protection.

Single-reviewer v3 amendments (10 more):
- B1: Step 8 fallback-to-SQL needs explicit 14-day time bound
- B2: NER drop-on-detect needs Prometheus alerting
- B3: legal-tier notification transport spec'd (signed Slack/email,
      no PII in body, failure non-blocking)
- B4: Step 6 human review SLA flagged — ~7 months at 500/day for
      ~100k unknown rows; operational decision needed
- B5: Memory zeroing in Go is best-effort (Rust uses zeroize crate);
      documented as not cryptographic-grade
- B6: purpose_definitions needs versioning + emergency revocation
      (purpose_versions + purpose_revocations tables)
- B7: Cache invalidation needs erasure_generation atomicity
      (subjects.erasure_generation int; gateway rejects stale-gen
      cache hits) — replaces best-effort pub/sub
- B8: 15-min cooling-off period for dual-control issuance to
      prevent emergency-bypass culture
- B9: NER calibrated test set with target recall ≥99.5% on
      synthetic adversarial PII
- B10: S3 Object Lock in separate AWS account with write-only IAM;
       root credentials held by external party
- B11: BIPA infrastructure-as-notice attestation in Phase 1.6 doc
- B12: Backup retention vs ciphertext-deletion erasure window
       documented in RTBF runbook

Estimate revised v2 12-15d → v3 17-20d. Worth it — the cost is what
buys "I would build this" from 3 independent senior security
architects across 3 model lineages.

Must-have v3 items (block implementation): A1, A2, B1, B6, B7, B11.
Should-have (ship in Phase 5 if calendar tight): B2-B5, B8-B10, B12.

Re-scrum NOT recommended for v3 — diminishing returns; must-have
items are concrete fixes with clear acceptance criteria.

No code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:39:35 -05:00
rootandClaude Opus 4.7 298fadce41 identity service: v2 — fold cross-lineage scrum findings + 4 'would not build' blocker fixes
Scrummed v1 across opus + kimi + gemini lineages via the new model
fleet. 3/3 reviewers said 'I would NOT build v1 as written.' 4
convergent blockers, all resolved in v2:

1. Migration order wrong — backfill before validation creates dark
   database; if backfill bug, no production traffic catches it.
   v2 inserts BIPA-prereq Step 0 + shadow-write before backfill +
   shadow-read before cutover. 9-step migration with cryptographic
   attestation of completeness at quarantine.

2. Master key on disk + legal token static file = 'security theater'
   per all 3. v2: HashiCorp Vault Transit / AWS KMS for KEK (not
   sealed file). Legal token: split-secret short-lived JWT (max 24h),
   dual-control issuance (J + counsel both sign), revocable in <60s.

3. consent_status='inferred_existing' is BIPA prima facie violation
   (kimi+gemini explicit). v2 backfill uses 'pending_backfill_review';
   biometric data NEVER backfilled — separate consent stream.

4. Healthcare default 'general' = HIPAA exposure window for every
   misclassified subject. v2 default 'unknown' with fail-closed
   routing (treat unknown as healthcare-equivalent until classified
   by manual review). Auto-escalation to healthcare on resume_text
   pattern match.

Plus 12 single-reviewer additions:
- mTLS mandatory between gateway↔identityd (kimi)
- External anchor for audit chain: S3 Object Lock 7-year compliance
  mode, hourly + on-event commits (all 3)
- Audit-log signing key separate from encryption KEK (opus)
- Field-level authorization via purpose_definitions table (kimi)
- Per-row encryption keys deferred to Phase 7 (kimi simplification)
- pii_access_log itself needs legal-tier read auth (opus)
- Synchronous cache invalidation pub/sub on RTBF (opus)
- Outbound NER pass for Langfuse defense-in-depth (opus TOCTOU)
- model_version_hash per decision row (gemini)
- /vertical minimal-disclosure endpoint (kimi HIPAA min-necessary)
- Auto-escalation healthcare on resume_text pattern (kimi)
- Rate limiting + token revocation list (opus)
- Oracle tests in audit_parity.sh (kimi SOC2 CC4.1)

Architecturally simplified per scrum:
- Per-row encryption keys deferred to Phase 7 (single DEK + HSM-
  wrapped KEK + ciphertext deletion is equivalent practical erasure
  with less complexity)
- PDF render deferred (JSON ships first)
- Training-safe export deferred (not critical path)

Estimated effort revised 8-10 → 12-15 days. Worth it — every
addition was a 3/3-reviewer convergent finding.

Re-scrum recommended before implementation starts to verify v2
addresses the v1 blockers.

No code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:36:07 -05:00
rootandClaude Opus 4.7 565ea4b32a audit phase 2: IDENTITY_SERVICE_DESIGN.md — full design doc
Incorporates J's confirmed answers (2026-05-03):
- separate daemon (identityd) on :3225 / :4225
- signed JSON with PDF render for legal export
- legal-only credential separate from admin token
- Langfuse self-hosted (drops cross-border concern)
- EU placeholder fields, not enforced
- healthcare vertical routing — local-only models for healthcare PHI
- training-safe export with hashed pseudonyms

Plus Phase 1 + 1.5 findings + scrum-driven priorities:
- UUID v7 candidate_id (drops kimi enumeration risk)
- per-row encryption with per-subject keys (crypto-erasure target)
- pii_access_log with Merkle-style integrity hash chain (FRE 901)
- subject_id top-level promotion in all JSONL sinks
- Langfuse boundary redaction layer (scrum C2 priority)
- adverse-impact comparator pool in audit response (scrum C3)
- BIPA-specific consent + retention metadata (scrum C4)
- vertical detection at gateway boundary (J answer 10)

Implementation single-language: Go (one identityd, both runtimes call
it via HTTP). Postgres backing store, isolated schema. Master key in
sealed file v1, vault migration path documented.

8-step migration path: stand up empty → backfill from parquet → behind
feature flag → cut over reads incrementally → quarantine PII columns
in workers_500k. Each step its own commit + gate + rollback.

6 open questions for J before implementation: master key location,
Postgres shared vs isolated, vertical backfill default, legal token
issuance procedure, crypto-erasure sweep cadence, EU enforcement
timeline.

Estimated 8-10 working days total. Largest single phase in the audit
program.

No code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:25:40 -05:00
rootandClaude Opus 4.7 fd429f4185 audit phase 1.5: BIPA schema audit + outcomes.jsonl content sample
Two follow-up walks per AUDIT_PHASE_1_DISCOVERY §10/C4 + gemini scrum
flag. Read-only. No code changes.

BIPA findings:
- scripts/staffing/tag_face_pool.py uses deepface to extract gender +
  race + age from face images. Output persists to data/headshots/
  manifest.jsonl. For synthetic faces this is fine; for real candidate
  photos this becomes a regulated biometric database (740 ILCS 14/10).
- mcp-server/index.ts:1408 ComfyUI prompt EXPLICITLY embeds protected
  attributes (age + race + gender) into model prompt — system-level
  encoding of protected-attribute features into AI workflow.
- mcp-server/search.html:3375-3432 has hard-coded FEMALE_NAMES /
  MALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup tables — name-based
  ethnicity inference. Title VII / disparate-impact risk separate
  from BIPA.
- data/headshots/manifest.jsonl is TRACKED IN GIT today (synthetic
  classifications). For real photos, this would be biometric data
  in version control — serious failure.
- No consent flow, no public retention schedule, no deletion
  procedure, no employee training documented. All required by BIPA
  §15(a)/(b) before real-photo intake.

outcomes.jsonl sample:
- 39/101 rows persist candidate names in fills[*].name field today
- Sample names: "Carmen I. Garcia", "Jamal Z. Jones", "Jacob N. Patel"
  (synthetic but real shape)
- 0 hits for "culture fit" / "communication" / etc proxy phrases —
  synthetic data doesn't generate them. When real models reason about
  real candidates, they will. Append-only persistence makes RTBF
  cryptographic-erasure-only.

Recommends Phase 1.6 (NEW) — BIPA pre-launch gates between Phase 1.5
and Phase 2: BIPA_COMPLIANCE_POLICY.md, consent gate at upload
endpoint, quarantine real-photo classifications to data/biometric/,
deprecate name->ethnicity lookup tables, unit test that synthetic
manifest stays synthetic. 4-8 hours of design + one code commit.

5 open questions for J: where do real photos enter, will deepface
tagging path stay for real photos, consent UX, retention duration
floor, designated privacy officer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:22:53 -05:00
rootandClaude Opus 4.7 64bda21614 audit PRD: J answered 5 open questions — fold into §10, revise phase plan
Conversation 2026-05-03 — J confirmed:
- Photos/video YES → BIPA in full force ($1k-$5k per violation)
- Langfuse self-hosted → drops GDPR Art. 44 cross-border concern
- EU not in scope now but placeholder needed → design EU-compatible
- Healthcare vertical YES → HIPAA BAA needed with model providers,
  PHI redaction at gateway boundary OR local-only routing for those
  requests, vertical-detection at boundary is Phase 2 requirement
- Training/RAG MAY re-run on outcomes → design as if it will, training-
  safe export interface needed, crypto-erasure becomes load-bearing
  evidence chain

§10 updated with answered/pending status per question. New §10.5
"Effect on phase plan" introduces:
- Phase 1.5 (NEW) — BIPA photo/video schema audit + Langfuse boundary
  scoping + outcomes.jsonl content sample, BEFORE Phase 2 design
- Phase 2 design must now include: EU-placeholder fields, vertical
  detection, training-safe export, BIPA consent metadata
- Phase 9 rehearsal must cover discrimination + BIPA + healthcare PHI

3 questions still pending J's call before Phase 2 design ships:
identity service daemon vs in-process, JSON vs signed PDF for legal
export, audit endpoint auth model.

No code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:16:27 -05:00
rootandClaude Opus 4.7 627a5f0c3d audit phase 1: §10 scrum-review findings + walk back §1F over-claim
Ran cross-lineage scrum on the discovery doc with the new model fleet
(opus + kimi-k2.6 + gemini-3-flash via Go gateway :4110, custom
"senior security architect" prompt). 3/3 reviewers responded with
substantive 800-1200 word reviews. Saved at /tmp/audit_scrum/.

5 convergent findings (≥2 reviewers) added as §10/C1-C5:

C1. §1F matrix-indexer "good for audit defensibility" claim is over-
    claimed — walked back in TL;DR. Trace bodies unverified; treat as
    SUSPECTED PII sink until §8.1 sampling completes.

C2. §1E (Langfuse) is the most dangerous leak — fix FIRST, ahead of
    view-routing. Boundary-crossing leak (GDPR Art. 44 / CPRA sale /
    SOC2 disposal). All 3 reviewers converge on this priority.

C3. Discrimination defense requires the FULL CANDIDATE POOL, not just
    fills. EEOC UGESP (1978): need adverse-impact stats on everyone
    who could have been picked. Phase 1 worked example missed this.

C4. BIPA / biometric exposure understated in findings (in PRD §10.5
    but not translated to actionables). $1k-$5k per-violation regime.

C5. candidate_id must be promoted to top-level field in all JSONL
    sinks. Grepping natural-language strings is not defensible audit
    strategy. 3/3 reviewers converge.

11 single-reviewer high-value catches added as §10 single-reviewer
section: opus on LLM provider egress (8th PII path), Art. 22 right-
to-explanation, special-category data, DPIA/ROPA/DPA inventory; kimi
on sequential ID enumeration risk, Langfuse retention config, CCPA
de-identified-in-place vs crypto-shred, Bun common-mode failure,
cryptographic audit-trail integrity (Merkle/FRE 901), HIPAA BAA,
revised SELECT * effort estimate; gemini on data residency, "culture
fit" reasoning proxies, comparator-pool snapshot.

§9 reordered: sample first → defense-layer second → Langfuse
boundary third (was view-routing first per original draft;
boundary-crossing leak is higher priority per scrum).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:13:07 -05:00
rootandClaude Opus 4.7 505ea93726 audit phase 1: discovery walk complete — subject + PII surface map
Read-only walk of both runtimes per AUDIT_TRAIL_PRD.md §8 phase 1.
Fills "UNKNOWN" cells in PRD §3 + §7 with file:line evidence.

Headline findings:
- candidates_safe + workers_safe views EXIST as a defense layer but
  are BYPASSED — tool registry SQL templates query raw tables
- PII traverses 7+ persistence/transmission paths per fill scenario:
  SQL → tool_result → LogEntry → /v1/respond → Langfuse → outcomes.jsonl
  → overseer_corrections.jsonl
- candidate_id is stable but co-located with PII in workers_500k.parquet
  (no separate identity service)
- /audit/subject/{id} endpoint does not exist
- Append-only persistence is universal — RTBF requires crypto-erasure
- Pathway memory is structurally subject-agnostic in fingerprints
  (defensive); trace bodies may leak PII (needs sampling)
- Go side mirrors Rust PII shape — parity in the leak too
- Worked example (John Martinez audit today): NOT POSSIBLE to produce
  complete-and-defensible response

Recommends 4 cheap high-value moves before Phase 2 design starts:
defense-layer enforcement (rewrite 3 SQL templates to _safe views),
sample state.json/Langfuse to confirm pathway memory is clean, walk
Bun mcp-server tool surface, schema-audit for protected-attribute
proxies. None are commitments — J's call.

No code changes in this commit. Companion to AUDIT_TRAIL_PRD.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 01:04:07 -05:00
rootandClaude Opus 4.7 b2d717ae44 audit PRD: add §10.5 jurisdictional surface (IL + IN, federal, SOC2)
J flagged that the staffing system targets Chicago + Indiana — added a
jurisdictional checklist section to the audit-trail PRD so counsel has
a working starting point.

Covered:
- Federal: Title VII, ADEA, ADA, EEOC, OFCCP, FCRA, Section 1981
- Illinois: BIPA (high risk if any candidate photos), AI Video Interview
  Act (820 ILCS 42), Illinois Human Rights Act (broader than Title VII),
  PIPA breach notification, Day and Temporary Labor Services Act
  (directly applies — staffing industry-specific recordkeeping), Cook
  County + City of Chicago Human Rights Ordinances (additional protected
  classes including source of income, parental status, credit history)
- Indiana: Data Breach Disclosure, Civil Rights Law (lighter than IL),
  Genetic Information Privacy Act
- SOC 2 Type II as the typical SaaS sale gate (Privacy + Security TSCs
  most relevant; 6-9 month effort to first report)
- HIPAA / PCI / ISO 27001 noted as out of current scope but flagged

Phase reordering implications captured:
- BIPA risk on real candidate photos may need to be resolved BEFORE
  audit-trail work (class-action exposure)
- SOC 2 Type II prep runs in parallel, not after
- IL Day and Temporary Labor Services recordkeeping may override our
  proposed 4-year retention SLA

7 open questions added that counsel must answer before the §8 phases
can be locked in. Document is explicit (multiple times) that this is
NOT legal advice — a research-grade checklist for J's counsel
conversation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:56:28 -05:00
rootandClaude Opus 4.7 c170ebc86e docs: AUDIT_TRAIL_PRD — production-readiness gate for staffing client
J flagged that smoke + parity tests prove the surface compiles, NOT
that an audit response can be produced for a specific person — and the
staffing client won't sign without defensible discrimination-claim
response capability.

New docs/AUDIT_TRAIL_PRD.md captures:
- worked example: John Martinez at Warehouse B requests audit
- subject audit response output format (per-decision row schema)
- surface map: where decisions happen today, where the gaps are
- PII handling rules (tokenization, protected-attribute exclusion,
  inferred-attribute risk)
- identity service design intent (separate daemon, audited reads)
- retention + right-to-be-forgotten policy intent
- 9-phase implementation sequence with explicit per-phase exit criteria
- cross-runtime requirement (both Rust + Go must satisfy)
- 7 open questions blocking phase 2+ that need J's call

STATE_OF_PLAY + PRD updated with explicit "production-ready blocker"
section pointing at the new doc. The "substrate is shipped" framing
gets a caveat: substrate ≠ production-ready until audit phase 9 exits.

No code changes. This is the planning artifact J asked for before we
start building.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:54:46 -05:00
rootandClaude Opus 4.7 5368aca4d4 docs: sync ADR-019 + PRD + DECISIONS with 2026-05-02 substrate changes
ADR-019: closed the "re-bench when 10M corpus exists" follow-up. Added
"Follow-up: 10M re-bench (2026-05-02)" section with the post-fix numbers
(search ~20ms warm / ~46ms cold, doc-fetch ~5ms post-btree). Documented
the lance-bench-bypassing-IndexMeta bug + 2-layer fix + gauntlet
(7 unit + 12 sanitize + 10 smoke probes). Reframes the strategic
question as "Lance vs Parquet+HNSW-with-spilling" since HNSW doesn't
fit RAM at 10M.

DECISIONS: added ADR-022 — drop Python sidecar from Rust hot path.
Captures the rationale (236× embed perf gap was pure overhead),
co-shipped LRU cache, dev-only Python that survives, cross-runtime
parity verification, and the operator runbook signal (ps -ef ABSENT
post-deploy).

PRD: updated AI Boundary table line + aibridge crate description to
reflect direct Ollama path (was: Python FastAPI sidecar → Ollama).
Both lines reference ADR-022 for the full rationale.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:44:57 -05:00
89 changed files with 14130 additions and 3032 deletions
+25
View File
@@ -50,3 +50,28 @@ reports/distillation/*-*-*-*-*/
tests/agent_test/_*
tests/agent_test/sessions/
tests/real-world/runs/
# BIPA-quarantined photo uploads — Phase 1.6 Gate 3a writes to
# data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext> with mode 0700/0600.
# This is regulated subject-of-record data and must NEVER enter git.
data/biometric/
# Local-only scrum review evidence. Per `feedback_audit_findings_log.md`
# scrum runs fold fixes into a batch commit; the verdict / disposition
# files stay local for forensics.
reports/scrum/
# Per-event biometric verification reports (timestamp-named, regenerated
# per `verify_biometric_erasure.sh` invocation). Source-of-truth is the
# audit chain itself; these reports are derived views.
reports/biometric/
# Counsel transmission tarballs + manifests are regenerated by
# `bundle_counsel_packet.sh` from the tracked `docs/counsel/` source.
# The bundle is transmittable, not source-of-truth.
reports/counsel/
# Local experiments scratchpad — per the "Test code in main is ACTIVELY
# being cleaned out" policy (commits 6aafd41 + f4ebd22), one-off
# experiments stay out of the tracked tree.
experiments/
Generated
+3
View File
@@ -1365,10 +1365,13 @@ dependencies = [
"axum",
"bytes",
"chrono",
"hmac",
"object_store",
"parquet 55.2.0",
"proto",
"serde",
"serde_json",
"sha2",
"shared",
"storaged",
"tokio",
+1
View File
@@ -36,6 +36,7 @@ datafusion = "47"
bytes = "1"
futures = "0.3"
sha2 = "0.10"
hmac = "0.12"
url = "2"
tonic = "0.13"
prost = "0.13"
+227 -4
View File
@@ -1,13 +1,220 @@
# STATE OF PLAY — Lakehouse
**Last verified:** 2026-05-02 evening CDT
**Verified by:** live probe (smoke 9/9, parity 32/32, gateway restarted), not memory.
**Last verified:** 2026-06-30 (search index-alignment fix live on :3700) · biometric substrate 2026-05-05
**Verified by (2026-06-30):** live probe after mcp-server restart — `/match` + `/search` with `state='IL'` return only IL workers (new PID on :3700). Not memory.
**Verified by (2026-05-05):** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory.
> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
---
## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits this wave)
## WHAT LANDED 2026-06-30 (search index-alignment sweep + P1-already-done finding)
**Fixed a systemic hybrid-search bug.** Every `/vectors/hybrid` call that paired the
`ethereal_workers` SQL dataset (10K rows) with the `workers_500k_v1` vector index (50K,
a *different* worker set) returned wrong-entity results: same `worker_id` = different
person in each index, so a `state='IL'` query surfaced a **KY** worker flagged
`sql_verified:true`. Root cause: the default index didn't share an id-space with the
default dataset.
Swept **8 call sites** in `mcp-server/index.ts` to `ethereal_workers_v1` (10K, same
id-space as the SQL table). Aligned sites (`filter_dataset: workers_500k`) and pure-RAG
sites (`/vectors/rag`) were correctly left alone.
| Commit | Sites |
|---|---|
| `423400e` | `/search` HTTP handler |
| `35a80a7` | `search_workers` tool · `match_contract` tool · `/match` HTTP handler |
| `1570306` | 4 intelligence routes (similar-workers, role-discovery, smart_search, scenario-find) |
**Verified live** after an mcp-server restart (new PID on :3700): `/match` and `/search`
with `state='IL'` return only IL workers (Champaign/Springfield/Urbana); `state='WI'`
returns WI. Grep confirms no `workers_500k_v1` index remains paired with `ethereal_workers`.
Server boots clean. Only `mcp-server/index.ts` was touched.
**Finding — demo-plan P1 was already done.** The "search box that actually filters" item
in this doc's CURRENT PLAN (P1, from the 2026-04-27 demo audit) is **stale**. The UI now
posts structured `state`/`role` to the evolved `/intelligence/chat`, which returns real
`sql_results` (IL Forklift → 167 matches, correct workers; WI → WI). The P1 section below
describes a bug that no longer exists — don't re-implement it.
**⚠ Uncommitted WIP left untouched (pre-existing, not from this session).**
`mcp-server/search.html` + `proof.html` (edited 2026-05-30; proof.html restructured into
chapters) and a `biometric_endpoint.rs` comment cleanup remain uncommitted in the working
tree. Also untracked and newer than this doc: `RECOVERY_PLAN_2026-05-05.md`,
`mcp-server/UX_FIX_PLAN.md`, `docs/scope-sentinel/`. Someone should reconcile these — they
imply planning context this file doesn't yet capture.
---
## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard)
After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`.
| Commit | What | Verified |
|---|---|---|
| `76cb5ac` | `POST /biometric/subject/{id}/consent` — Gate 2 backend. Flips status NeverCollected→Given, computes retention_until = given_at + 18mo (per retention schedule v1 §4), captures consent_version_hash + collection_method (closed set: electronic_signature/paper/click_acceptance) + operator + evidence path to audit row. State machine: 409 already_given, 409 post-withdrawal-requires-erase, 403 subject_inactive. | 12 unit tests; live POST returns 401/400/404 on guards |
| `7f0f500` | Candidate intake UI at `/biometric/intake?candidate_id=XXX` (mcp-server :3700, exposed via nginx at devop.live/lakehouse/biometric/intake). 4-screen flow: operator auth → consent template render + click-accept → photo capture (file or webcam getUserMedia) → confirmation showing audit hmac. SHA-256 of rendered consent block becomes consent_version_hash. sessionStorage-backed token (clears on tab close), neo-brutalist style matching onboard.html. | 22631-byte HTML, mcp-server route returns 200 |
| `68d226c` | Three things in one wave: <br>(a) `POST /biometric/subject/{id}/withdraw` — BIPA right of withdrawal. Flips Given→Withdrawn, accelerates retention_until from 18mo→30d (per consent template v1 §2 SLA), audit kind=biometric_consent_withdrawal. State machine 409s on NeverCollected/Pending (nothing_to_withdraw), Withdrawn (already_withdrawn), Expired (already_expired). 12 unit tests. <br>(b) Withdraw UI at `/biometric/withdraw` — 3-screen operator flow (token+name auth → reason+evidence form → confirmation showing 30-day clock + verify curl recipe). <br>(c) `lakehouse-retention-sweep.{service,timer}` systemd units in `ops/systemd/`. Daily 03:00 UTC, Persistent=true, install.sh updated to handle paired timer+oneshot service. <br>Plus operator_of_record bug fix in intake UI (was hardcoded `'intake_ui_operator'`). | 46/46 biometric_endpoint + 71/71 catalogd lib tests; manual sweep run: 100 subjects, 0 overdue, exit 0 |
| **(current HEAD post-this-wave)** | Stats endpoint `GET /biometric/stats` (legal-tier auth, returns subject counts by status + photo count + oldest active retention + last 20 state-change events with anonymizable trace_ids) + ops dashboard at `/biometric/dashboard` (single-page, polls /stats every 15s, table + status tiles, XSS-safe DOM construction not innerHTML). Plus consent_versions allowlist: `BiometricEndpointState.allowed_consent_versions: Option<Arc<HashSet<String>>>`, loaded from `/etc/lakehouse/consent_versions.json` (`LH_CONSENT_VERSIONS_FILE` override), missing-file = permissive (v1 compat), present + populated = strict mode (refuses unknown hashes with 400 consent_version_unknown). Plus `scripts/staffing/subject_timeline.sh` — pretty-prints any subject's full BIPA lifecycle from /audit/subject/{id} (manifest + on-disk photo + chronological audit chain + verification status). | 5 new allowlist unit tests + 4 stats tests; live demo on WORKER-100 ran end-to-end (consent → photo → withdraw, chain verified=true, chain_root=a47563ff…) |
### Live demo evidence (WORKER-100, 2026-05-05 20:12 UTC)
The full lifecycle was exercised against the live gateway as a verification artifact. The audit chain on WORKER-100 now contains 3 rows:
```
20:12:33.054 BIOMETRIC_CONSENT_GRANT result=given hmac=9c6f4153341e97d2… trace=live-demo-2026-05-05
20:12:47.957 BIOMETRIC_COLLECTION result=success hmac=856be6173c88277c… trace=live-demo-2026-05-05
20:15:27.298 BIOMETRIC_CONSENT_WITHDRAWAL result=withdrawn hmac=a47563ff937d50de… trace=live-demo-2026-05-05
```
`chain_verified=true`, chain_root = a47563ff937d50de43b09a0c903cff954233836c219a928ee8ca2aa6792272dd. Photo file at `data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg` (180 bytes — a minimal real JFIF JPEG), retention_until=2026-06-04 (= 30 days from withdrawal). Retention sweep will flag this subject on or after that date; operator runs `/biometric/subject/WORKER-100/erase` to destroy.
To re-verify: `./scripts/staffing/subject_timeline.sh WORKER-100`.
### Endpoint matrix (v1 BIPA lifecycle complete)
| Event | Endpoint | Method | Auth | Status flip | retention_until |
|---|---|---|---|---|---|
| consent given | `/biometric/subject/{id}/consent` | POST | legal | NeverCollected/Pending → Given | now + 18mo |
| photo collected | `/biometric/subject/{id}/photo` | POST | legal + consent gate | (no change) | (no change) |
| consent withdrawn | `/biometric/subject/{id}/withdraw` | POST | legal | Given → Withdrawn | now + 30d |
| destruction | `/biometric/subject/{id}/erase` | POST | legal | (manifest cleared) | n/a |
| audit read | `/audit/subject/{id}` | GET | legal | (read-only) | (read-only) |
| ops aggregates | `/biometric/stats` | GET | legal | (read-only) | (read-only) |
UIs:
- `/biometric/intake?candidate_id=X` — operator-driven consent + photo
- `/biometric/withdraw` — operator-driven withdrawal recording
- `/biometric/dashboard` — read-only ops aggregate, auto-refresh
CLI tools:
- `scripts/staffing/verify_biometric_erasure.sh <id>` — post-erasure verification
- `scripts/staffing/biometric_destruction_report.sh --month YYYY-MM` — anonymized monthly report
- `scripts/staffing/subject_timeline.sh <id>` — full lifecycle pretty-print (NEW 2026-05-05)
- `scripts/staffing/bundle_counsel_packet.sh` — counsel review tarball
- `scripts/staffing/attest_pre_identityd_biometric_state.sh` — defense attestation generator
### What's blocking production cutover NOW
**Counsel calendar.** Engineering substrate is done end-to-end: every state transition has a defensible endpoint, every endpoint has tests + live verification, every UI is reachable, retention sweep is scheduled, allowlist hardening is wired. The remaining work is signature/review:
1. Counsel review of consent template v1 (revised for Option C — classifications deferred)
2. Counsel review of retention schedule v1 (revised for Option C)
3. Counsel review of destruction runbook
4. Counsel + J signatures on §2 attestation
5. Once counsel signs the consent template, populate `/etc/lakehouse/consent_versions.json` with the signed hash to flip the gateway from permissive to strict mode
Counsel-review packet at `reports/counsel/counsel_packet_2026-05-05.tar.gz` (regenerable via `bundle_counsel_packet.sh` to pick up the latest doc state).
---
## WHAT LANDED 2026-05-05 (morning wave — doc reconciliation + Gate 3b decision + counsel packet)
This was a **doc-only wave**, not code. Background: J asked for an audit of the BIPA/biometric documentation before production cutover. Audit found moderate fragmentation between docs and shipped code (post-`identityd` collapse, post-Gate-3a-ship, pre-Gate-3b-decision). Closed it in one pass.
| Item | What changed | Status |
|---|---|---|
| **Gate 3b — DECIDED: Option C (defer classifications)** | `BiometricCollection.classifications` stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status flipped from "draft / awaits product" to "DECIDED 2026-05-05". | Locked |
| **Endpoint-path drift** | `PHASE_1_6_BIPA_GATES.md` (3 spots), `BIPA_DESTRUCTION_RUNBOOK.md` (2 spots), `biometric_retention_schedule_v1.md` (1 spot) updated from legacy `/v1/identity/subjects/*` (proposed under separate identityd daemon, never shipped) to actual `/biometric/subject/*` (catalogd-local, shipped `848a458`). Schema block in `PHASE_1_6_BIPA_GATES.md` rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate (not the proposed Postgres `subjects` table). | Reconciled |
| **Consent template + retention schedule** | Both revised for Option C: removed all "automated facial-classification" / "deepface" language so disclosed scope matches implemented scope. Pending counsel review — they were already eng-staged with ⚖ markers. | Eng-staged for counsel |
| **`scripts/staffing/verify_biometric_erasure.sh`** (NEW) | Operator-side verification of an erasure event. Curls `/audit/subject/{id}` with legal-tier token, checks: manifest.biometric_collection null, uploads dir empty, last audit row is `biometric_erasure`/`full_erasure` with `erased`/`success`, chain_verified=true. Writes a hashed report to `reports/biometric/`. | Smoke-tested live |
| **`scripts/staffing/biometric_destruction_report.sh`** (NEW) | Monthly destruction-event aggregation. Anonymizes candidate IDs (sha256-12 prefix), counts by scope + trigger, flags anomalies. Smoke-test on May 2026 data found 1 historical `biometric_erasure`/`consent_withdrawal` event (test fixture). | Smoke-tested live |
| **`docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`** (NEW) | Captures the rotation procedure operationalized after the 2026-05-05 `/tmp` wipe incident. Covers: when to rotate, pre-rotation snapshot, atomic-swap procedure, post-rotation verification (incl. expected pre-rotation chain tamper-detect under new key), recovery from lost key, ⚖ counsel notes. | Authored |
| **`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`** + `bundle_counsel_packet.sh` (NEW) | Cover note bundling all eng-staged BIPA docs for counsel review with per-doc questions, sign-off checklist, recommended review sequence. Bundler script tarballs the 8 referenced files + emits a SHA-256 manifest. Tarball ready for transmission: `reports/counsel/counsel_packet_2026-05-05.tar.gz`. | Bundled, ready to send |
### Eng follow-up that this wave surfaced
- **Double-upload file leak — DIAGNOSED + FIXED** (2026-05-05 same wave). `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo file. Investigation showed:
- The file was 13 bytes of test fixture (`ff d8 ff d9 + ASCII "TESTBYTES"`), byte-identical to the unit-test fixture at `biometric_endpoint.rs:841`. NO PII, NO biometric content, NO synthetic-face content. Came from manual integration testing on 2026-05-03.
- Audit log timeline showed two consecutive uploads (09:54, 10:04) followed by one erasure (10:22). The erasure unlinked only the SECOND file (which the manifest pointed at by then); the first file was orphaned because the second upload had silently overwritten `manifest.data_path`.
- **Real bug found**: the upload handler did NOT refuse a second upload to a subject with `biometric_collection.is_some()`. Patched `process_upload` to return HTTP 409 + `error: "biometric_already_collected"` when a re-upload is attempted; operator must explicitly POST `/biometric/subject/{id}/erase` first.
- Stranded test file removed (`rm` of the 13-byte fixture).
- New unit test `second_upload_without_erase_returns_409` asserts the 409 + that the first photo's data_path remains unchanged + that the first file remains untouched on disk.
- Existing `repeated_uploads_grow_the_chain` replaced with `upload_erase_upload_grows_the_chain_cleanly` (covers the legitimate re-collection cycle: upload → erase → upload, chain grows to 3 rows).
- Existing `content_type_with_parameters_accepted` test updated to use two distinct subjects (it had used one subject for two uploads to test content-type parsing — now would 409).
- **22 biometric_endpoint tests + 59 catalogd lib tests all green** post-patch (was 21+58 pre-patch).
- Production posture: gateway binary needs rebuild (`cargo build --release`) + `systemctl restart lakehouse.service` to pick up the new 409 behavior in live traffic.
- **Pre-rotation chain tamper-detect (expected, not a bug).** WORKER-{1..5} had pre-2026-05-05 audit chains under the prior `LH_SUBJECT_AUDIT_KEY`. Under the new key (post-`/tmp` wipe rotation), those chains correctly tamper-detect. The rotation runbook §4.4 documents this as expected; a §2.2 pre-rotation snapshot is what would prove they were intact pre-rotation if defensibility ever needs it.
### What's blocking production cutover NOW (after this wave)
- **Counsel calendar:** the four sign-off items in `COUNSEL_REVIEW_PACKET_2026-05-05.md` (retention schedule, consent template, destruction runbook, pre-identityd attestation). The packet tarball is ready; ⚖ counsel is the bottleneck.
- **Nothing else.** Engineering is no longer the long pole.
### Phase 1.6 BIPA gates — status table (this is the final post-Option-C state)
| # | Gate | Status |
|---|---|---|
| 1 | Public retention schedule | **eng-staged**, revised for Option C, ready for counsel |
| 2 | Informed consent template | **eng-staged**, revised for Option C, ready for counsel |
| 3a | Photo upload endpoint | **DONE** (shipped `f1fa6e4`, 11 unit tests, live verified) |
| 3b | Deepface classification | **DECIDED 2026-05-05: Option C (defer)** |
| 4 | Name → ethnicity inference removal | **DONE** (shipped, 4/4 mcp-server tests pass) |
| 5 | Destruction runbook + erasure endpoint | **eng-DONE** (`848a458`, 21 tests). Runbook scripts (verify + report) shipped 2026-05-05. Counsel review pending. |
| §2 | Pre-identityd attestation | **eng-DONE** (3/3 evidence checks). Awaits J + counsel signature. |
| §3 | Employee training | **deferred** (consolidated into runbook §7 acknowledgment for current operator population) |
---
## WHAT LANDED 2026-05-03 (16 commits this wave — local-first audit substrate + Phase 1.6 BIPA gates)
The dominant work today: **`docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8 SHIPPED end-to-end** + **5 of 7 Phase 1.6 BIPA pre-launch gates** + **6th cross-runtime parity probe**. Wave was structured as eight ship-then-scrum cycles — every wave caught real bugs, every fix wave landed within the same session.
| Commit | What | Verified |
|---|---|---|
| `d259909` | catalogd: Step 1 — `SubjectManifest` type + Registry CRUD | 17 catalogd subject tests PASS |
| `d16131b` | catalogd: Step 2 — `SubjectAuditWriter` HMAC-SHA256 chain + per-subject Mutex + canonical-JSON via BTreeMap | tamper-detection + concurrent-append race tests PASS |
| `bce6dfd` | catalogd: Step 3 — `bin/backfill_subjects` (BIPA-defensible defaults: vertical=unknown, consent=pending_backfill_review, retention=4yr) | 100 subjects loaded into live catalog |
| `fef1efd` | gateway: Step 4 — wire SubjectAuditWriter into `/v1/chat` tool dispatch + `audit_subject_hits_in` (inline, not spawn) | tool calls log accessor.kind="gateway_lookup" |
| `cd8c59a` | gateway: Step 5 — `AuditingWorkerLookup` decorator wraps validator's WorkerLookup; spawns audit on every find() | live `/v1/validate` produces audit rows |
| `e38f357` | subjects Steps 1-4 fixes from cross-lineage scrum (concurrency race, schema-evolution HMAC drift, hardcoded "success" classifier) | 41 tests green |
| `15cfd76` | catalogd + gateway: Step 6 — `/audit/subject/{id}` legal-tier endpoint with constant-time-eq token check + tampering detection | live curl returns chain_verified=true |
| `2a4b316` | subjects 2nd scrum fix wave (token min 16→32, chain_root from full chain via `chain_tip()`, rebuild collision warn, tightened result-state heuristic) | 17 catalogd + 6 gateway tests PASS |
| `8fc6238` | catalogd: Step 7 — `bin/retention_sweep` (BIPA-aware on biometric clock, idempotent across daily runs, no auto-mutation) | 8 sweep tests PASS, live verified at `--as-of 2031-06-01` flagging 100/100 expired |
| `2413c96` | catalogd: Step 8 — `bin/parity_subject_audit` (Rust side of cross-runtime parity probe) | known-answer + verify modes match Go byte-for-byte |
| `2222227` | parity helper hardening (panic-noise → die() helper, abs path stripped from doc) from scrum | parity probe still 6/6 |
| `4708717` | **Phase 1.6 BIPA wave** — Gate 4 absence test (4/4 with bypass coverage), §2 attestation script (3/3 evidence checks), Gate 1/2/5 doc scaffolds with ⚖ COUNSEL markers | 4/4 mcp-server Bun tests, 3/3 evidence on live data |
| `c7aa607` | Phase 1.6 scrum fixes — schema fingerprint hashes name+type+nullable, Gate 4 catches object-literal + class-field bypasses, pyarrow dep gate, item 7 deferral rationale | 4/4 + 3/3 still pass |
| `f1fa6e4` | **Phase 1.6 Gate 3a** — `crates/catalogd/src/biometric_endpoint.rs`: `POST /biometric/subject/{id}/photo` with consent gate, quarantined storage (mode 0700/0600), audit chain link, `BiometricCollection` field on SubjectManifest | 11 unit tests PASS, live roundtrip 200 |
| `3708e6a` | Gate 3a scrum fixes — transactional rollback on audit failure (BIPA convergent BLOCK), Content-Type parameter handling, relative data_path, ts+uuid filename, dead code removed | 11 tests + cross-runtime parity 6/6 |
| `7e0112b` | retention_sweep: stray indent fix on biometric_collection field | sweep tests still 8/8 |
| `848a458` | **Phase 1.6 Gate 5** — `POST /biometric/subject/{id}/erase` per BIPA destruction runbook. Two scopes (biometric_only / full); audit row appended BEFORE photo unlink so the chain has legal proof of intent even if file delete fails; manifest rolled back on audit failure. Trigger taxonomy: retention_expiry / consent_withdrawal / rtbf / court_order. | 21 unit tests (10 erasure-specific) PASS |
| `8ec43e0` | **Phase 1.6 Gate 3b** — deepface integration design doc (Option A subprocess / Option B ONNX-in-Rust / **Option C defer**). Recommends C: BIPA-safest, classifications field stays None, all load-bearing surfaces (consent + audit + retention + erasure) ship without it. Forces "do we actually need classifications" to be answered by product, not spec inertia. | doc-only |
**Cross-runtime parity (post-this-wave):** 6 probes, 38/38 byte-identical assertions —
`validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8) + subject_audit(6/6)`.
Run `cd /home/profit/golangLAKEHOUSE && for p in scripts/cutover/parity/*.sh; do bash "$p"; done` to re-verify.
**Three runtime-divergence classes caught + fixed by the parity probe authoring loop** (cataloged because they recur):
1. Go `omitempty` on string fields strips empty values that Rust serde always emits → broken HMAC
2. `time.RFC3339Nano` truncates trailing-zero nanoseconds where chrono `AutoSi` keeps 9 digits → broken HMAC
3. Go `json.Marshal` HTML-escapes `<>&` where serde keeps literal → broken HMAC on any field with those chars
All three have regression-locked tests; structural impossibility going forward.
**Phase 1.6 BIPA pre-launch gates — status table:**
| Item | Status | Evidence |
|---|---|---|
| Gate 1 — public retention schedule | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_retention_schedule_v1.md` |
| Gate 2 — informed consent template | eng-staged, ⚖ counsel pending | `docs/policies/consent/biometric_consent_template_v1.md` |
| Gate 3a — photo-upload endpoint | **DONE** | 11 unit tests + live `POST /biometric/subject/{id}/photo` |
| Gate 3b — deepface classification | **design doc shipped** (`8ec43e0`) — recommends Option C (defer); awaits product confirmation | `docs/PHASE_1_6_BIPA_GATES.md` Gate 3b section |
| Gate 4 — name→ethnicity removal | **DONE** | `mcp-server/phase_1_6_gate_4.test.ts` 4/4 with bypass coverage |
| Gate 5 — destruction runbook + erasure endpoint | **eng-DONE** (`848a458`); ⚖ counsel review of runbook still pending | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` + `POST /biometric/subject/{id}/erase` (21 tests) |
| §2 cryptographic attestation | eng-DONE, signature pending | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (SHA-256 evidence hash, 3/3 checks pass on live data) |
| §3 employee training | deferred | conditional on operator population size |
**Calendar bottleneck:** counsel review of items 1/2/5-runbook/§2 attestation. Engineering long pole is Gate 3b (deepface) — design doc landed (`8ec43e0`); needs product confirmation that classifications are required before engineering starts. Recommendation in doc is Option C (defer) on BIPA-safety grounds.
**Operational state:**
- `LH_SUBJECT_AUDIT_KEY=/etc/lakehouse/subject_audit.key` (32-byte HMAC signing key, mode 0400) loaded into systemd unit. **Moved off /tmp 2026-05-05** — /tmp wipes on reboot, which on May 5 disabled `/audit` + `/biometric` endpoints (gateway fails-closed at `crates/gateway/src/main.rs:459` if signing key is absent). Persistent path is per spec line 112.
- `LH_LEGAL_AUDIT_TOKEN_FILE=/etc/lakehouse/legal_audit.token` (44-char legal-tier token, mode 0400) loaded into systemd unit
- **Key rotation 2026-05-05:** prior key was lost when /tmp wiped on reboot. New key generated at canonical path. The 5 pre-rotation audit chains for `WORKER-{1..5}` (backfill data with `consent=pending_backfill_review`) will tamper-detect under the new key — expected and correct behavior on key rotation, not a bug. New chain entries from 2026-05-05 forward verify cleanly.
- `data/_catalog/subjects/` holds 100 backfilled `WORKER-N.json` manifests + per-subject `WORKER-N.audit.jsonl` HMAC chains
- `data/biometric/uploads/<safe_id>/<ts>_<uuid>.<ext>` quarantined photo storage (mode 0700 dir / 0600 file). 2 photos uploaded for WORKER-2 during live verify.
- `/audit/subject/{id}` mounted on gateway with chain_verified=true on every probe
- `/biometric/subject/{id}/photo` mounted on gateway, refuses 403 without `consent.biometric.status="given"`
---
## WHAT LANDED 2026-05-01 → 2026-05-02 (10 commits — Lance gauntlet + cross-runtime parity wave)
| Commit | What | Verified |
|---|---|---|
@@ -122,6 +329,10 @@ OpenCode catalog (live):
- **Lance backend gauntlet (2026-05-02)** — sanitizer over all 5 routes, 7 unit tests, 9-probe smoke, 10M re-bench. The `doc_id` btree auto-builds inside `lance_migrate` AND `lance-bench`. Do not "discover" the missing scalar index again or the leaked filesystem paths in error bodies.
- **Cross-runtime parity = 32/32** across 5 probes in `golangLAKEHOUSE/scripts/cutover/parity/`. Do not "build a parity probe for X" without checking — validator, extract_json, session_log, materializer, and embed are all already covered.
- **Decisions tracker is `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md`** — single living source of truth for cross-runtime decisions. As of 2026-05-02 it has 0 `_open_` code work items; only 2 strategic items left (Lance vs Parquet+HNSW-with-spilling, Go-vs-Rust primary cutover).
- **PRD line 70 is load-bearing — "Everything runs locally, no cloud APIs."** Yesterday's PR #13 violated this by routing customer hot-path inference to opencode/openrouter/ollama_cloud. **REVERTED 2026-05-03 (`d054c0b`).** The customer hot path (modes.toml staffing_inference, doc_drift_check; execution_loop overseer escalation) is now local Ollama (qwen3.5:latest). Cloud providers stay configured in providers.toml for **explicit dev-tool opt-in only** (scrum, auditor, bot/propose). Do NOT re-add cloud models to the hot path defaults — the customer demo runs on local + free.
- **`/v1/usage` shows by_provider=ollama only** for any customer-shape request. If you see by_provider including kimi/openrouter/opencode/ollama_cloud for normal /v1/iterate or /v1/respond traffic, something has regressed. Verify with: `curl -sf http://127.0.0.1:3100/v1/usage | jq .by_provider`.
- **`./scrum` is a TOOL, not architecture.** Lives at repo root. J runs it manually. Outputs findings to `data/_kb/scrum_findings.jsonl` (the KB). Findings inform development; they do NOT auto-fold into PRDs/design docs/code. If you find yourself proposing to "wire scrum into [some pipeline]" — stop. It's J's tool.
- **Test code in main is ACTIVELY being cleaned out.** 2026-05-03 commits `6aafd41` + `f4ebd22` removed 12 orphaned dev experiments (~2900 LOC) from `tests/real-world/` and `scripts/`. If you find more zero-reference experimental files, surface them — don't auto-delete unless the pattern is clearly a one-time-experiment-with-zero-consumers like the ones already removed.
---
@@ -146,12 +357,24 @@ End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift
---
## ⚠ PRODUCTION-READY BLOCKER (2026-05-03)
**Audit-trail capability is the gate to client signature.** Smoke + parity tests prove the surface compiles; they do NOT prove an audit response can be produced for a specific person. Staffing client won't sign without defensible discrimination-claim response capability.
**Authoritative document:** `docs/AUDIT_TRAIL_PRD.md` — drafted 2026-05-03. Defines worked example (John Martinez at Warehouse B), the per-decision audit row schema, the surface map of where decisions happen today, current-state-vs-target gap table, and 9-phase implementation sequence.
**Phase 1 (discovery walk) requires NO J approval — it's read-only.** Phases 2+ have explicit open questions in §10 of the PRD that need J's call before they can start.
Until phase 9 exit criterion is met, **do not claim "production-ready" on customer-facing surfaces.** Internal substrate (Lance, sidecar drop, parity probes) is solid; subject-of-record audit story is not.
---
## OPEN — but not blocking the demo
| Item | What | When to act |
|---|---|---|
| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. |
| Open PRs #6, #7, #10 | All closed 2026-05-02 — superseded / empty / stale. PR #12 merged 2026-05-03 (`a5d9070`). PR #13 merged 2026-05-03 (`feb638e`). | Done. |
| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
+10 -5
View File
@@ -28,11 +28,16 @@ const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
// retries — verified with trivial 8-token probes). Swapped default to
// deepseek-v3.1:671b which is proven working end-to-end through the
// pr_audit mode runner during Phase 5 distillation acceptance testing.
// kimi-k2:1t can be re-selected via LH_AUDITOR_REVIEW_MODEL env when
// the upstream returns. Tie-breaker stays grok-4.1-fast (different
// vendor lineage so consensus + tie-break won't fail-correlate).
const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "deepseek-v3.1:671b";
const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "x-ai/grok-4.1-fast";
//
// 2026-09-02 update: Ollama Cloud subscription lapsed — every
// ollama_cloud model 402s ("requires a subscription"). J's call: sweep
// the auditor to OpenRouter. Same deepseek lineage via OpenRouter;
// tie-breaker moves to grok-4.3 because OpenRouter 404s grok-4.1-fast
// as deprecated. Both probed live through the gateway before landing.
// Vendor lineages stay distinct so consensus + tie-break won't
// fail-correlate.
const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "openrouter/deepseek/deepseek-chat-v3.1";
const TIEBREAKER_MODEL = process.env.LH_AUDITOR_TIEBREAKER_MODEL ?? "openrouter/x-ai/grok-4.3";
const N_CONSENSUS = Number(process.env.LH_AUDITOR_CONSENSUS_N ?? 3);
const AUDIT_DISCREPANCIES_JSONL = "/home/profit/lakehouse/data/_kb/audit_discrepancies.jsonl";
// 40KB comfortably fits the consensus models' context windows
+15 -11
View File
@@ -51,13 +51,15 @@ const CALL_TIMEOUT_MS = 900_000;
const CACHE_TTL_MS = 24 * 60 * 60 * 1000;
const MAX_DIFF_CHARS = 180_000;
const MAX_PRIOR_FINDINGS = 50;
// Default provider/model = ollama_cloud/kimi-k2.6. Pre-2026-04-27 we
// went direct to api.kimi.com, but Ollama Cloud Pro now exposes the
// same model legitimately, so we route there to avoid User-Agent
// gating. The api.kimi.com path (provider=kimi) remains wired in the
// gateway as a fallback for when Ollama Cloud is upstream-broken.
const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "ollama_cloud";
const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6";
// Default provider/model = openrouter/moonshotai/kimi-k2.6.
// History: pre-2026-04-27 direct api.kimi.com (User-Agent gating),
// then ollama_cloud/kimi-k2.6 while the sub was active. 2026-09-02:
// Ollama Cloud subscription lapsed (402 on every cloud model), J's
// call was to sweep the auditor to OpenRouter — same Moonshot model,
// probed live through the gateway. The api.kimi.com path
// (provider=kimi) remains wired in the gateway as a fallback.
const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "openrouter";
const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "moonshotai/kimi-k2.6";
// Cross-lineage alternation. 2026-04-27 J's call: Opus is too
// expensive to auto-fire (~$0.30/audit). Kimi K2.6 via Go-sub is
// effectively free; Haiku 4.5 via Zen is ~$0.04. Alternate between
@@ -75,8 +77,10 @@ const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6";
// Override the alternation entirely with LH_AUDITOR_KIMI_MODEL
// (forces one model regardless of audit count); set
// LH_AUDITOR_KIMI_ALT_MODEL to the alternate.
const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "claude-haiku-4-5";
const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "opencode";
// 2026-09-02: opencode lane 401s (CreditsError, balance drained) — alt
// lane joins the OpenRouter sweep. Same Anthropic lineage, probed live.
const ALT_MODEL = process.env.LH_AUDITOR_KIMI_ALT_MODEL ?? "anthropic/claude-haiku-4.5";
const ALT_PROVIDER = process.env.LH_AUDITOR_KIMI_ALT_PROVIDER ?? "openrouter";
const FORCE_DEFAULT = process.env.LH_AUDITOR_KIMI_MODEL !== undefined && process.env.LH_AUDITOR_KIMI_MODEL !== "";
function selectModel(diffLen: number, auditIndex: number = 0): { provider: string; model: string; promoted: boolean } {
@@ -110,8 +114,8 @@ const MAX_TOKENS_OVERRIDE = Number(process.env.LH_AUDITOR_KIMI_MAX_TOKENS) || 0;
function maxTokensFor(model: string): number {
if (MAX_TOKENS_OVERRIDE > 0) return MAX_TOKENS_OVERRIDE;
if (model.startsWith("claude-opus")) return 32_000;
if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet")) return 8_192;
if (model.startsWith("kimi-")) return 128_000;
if (model.startsWith("claude-haiku") || model.startsWith("claude-sonnet") || model.includes("/claude-haiku") || model.includes("/claude-sonnet")) return 8_192;
if (model.startsWith("kimi-") || model.includes("/kimi-")) return 128_000;
if (model.startsWith("gpt-5") || model.startsWith("o1") || model.startsWith("o3") || model.startsWith("o4")) return 32_000;
return 16_000;
}
+25 -11
View File
@@ -9,28 +9,42 @@
import { readFile } from "node:fs/promises";
import type { PrSnapshot } from "./types.ts";
const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
// 2026-09-02: repo home moved to git.warren.help in the VPS
// consolidation; git.agentview.dev now dead-ends at a Traefik default
// cert + 404 (auditor had been failing listOpenPrs every cycle since
// Aug 25). Credentials come from ~/.git-credentials for HOST's domain,
// falling back to the user:pass embedded in this repo's origin remote.
// Basic auth works for both a PAT and a password on Gitea.
const HOST = process.env.GITEA_HOST ?? "https://git.warren.help";
const OWNER = "profit";
const REPO = "lakehouse";
const CRED_FILE = "/home/profit/.git-credentials";
const GIT_CONFIG = "/home/profit/lakehouse/.git/config";
let cachedPat: string | null = null;
let cachedAuth: string | null = null;
async function getPat(): Promise<string> {
if (cachedPat) return cachedPat;
const raw = await readFile(CRED_FILE, "utf8");
for (const line of raw.split("\n")) {
const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
if (m) { cachedPat = m[1]; return m[1]; }
async function getAuthHeader(): Promise<string> {
if (cachedAuth) return cachedAuth;
const hostDomain = new URL(HOST).host.replace(/\./g, "\\.");
const credRe = new RegExp(`^https://([^:]+):([^@]+)@${hostDomain}`);
for (const file of [CRED_FILE, GIT_CONFIG]) {
let raw: string;
try { raw = await readFile(file, "utf8"); } catch { continue; }
for (const line of raw.split("\n")) {
const m = line.trim().replace(/^url = /, "").match(credRe);
if (m) {
cachedAuth = `Basic ${Buffer.from(`${decodeURIComponent(m[1])}:${decodeURIComponent(m[2])}`).toString("base64")}`;
return cachedAuth;
}
}
}
throw new Error(`no Gitea PAT in ${CRED_FILE}`);
throw new Error(`no Gitea credentials for ${HOST} in ${CRED_FILE} or ${GIT_CONFIG}`);
}
async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
const pat = await getPat();
const url = `${HOST}/api/v1${path}`;
const headers = new Headers(init.headers);
headers.set("Authorization", `token ${pat}`);
headers.set("Authorization", await getAuthHeader());
if (init.body && !headers.has("content-type")) {
headers.set("content-type", "application/json");
}
+7 -9
View File
@@ -40,14 +40,13 @@ matrix_corpus = "chicago_permits_v1"
name = "staffing_inference"
# Staffing-domain native enrichment runner — Pass 4 (2026-04-26).
# Same composer architecture as codereview_lakehouse but with staffing
# framing + workers corpus. Validates that the modes-as-prompt-molders
# pattern generalizes beyond code review.
# framing + workers corpus.
preferred_mode = "staffing_inference_lakehouse"
fallback_modes = ["ladder", "consensus", "pipeline"]
# 2026-04-28: gpt-oss-120b:free → kimi-k2.6 via Ollama Pro. Coding-
# specialized, faster than gpt-oss, on the same OLLAMA_CLOUD_KEY so
# no extra provider hop.
default_model = "kimi-k2.6"
# 2026-05-03: REVERTED to local. PRD line 70 — everything runs locally,
# no cloud APIs on the customer hot path. Cloud models stay available
# in providers.toml for explicit dev-tool opt-in (scrum, auditor).
default_model = "qwen3.5:latest"
matrix_corpus = "workers_500k_v8"
[[task_class]]
@@ -61,9 +60,8 @@ matrix_corpus = "kb_team_runs_v1"
name = "doc_drift_check"
preferred_mode = "drift"
fallback_modes = ["validator"]
# 2026-04-28: gpt-oss:120b → gemini-3-flash-preview via Ollama Pro.
# Speed leader on factual checking, same OLLAMA_CLOUD_KEY.
default_model = "gemini-3-flash-preview"
# 2026-05-03: REVERTED to local per PRD line 70.
default_model = "qwen3.5:latest"
matrix_corpus = "distilled_factual_v20260423095819"
[[task_class]]
+3
View File
@@ -16,5 +16,8 @@ chrono = { workspace = true }
uuid = { workspace = true }
object_store = { workspace = true }
arrow = { workspace = true }
parquet = { workspace = true }
sha2 = { workspace = true }
hmac = { workspace = true }
proto = { path = "../proto" }
tonic = { workspace = true }
+505
View File
@@ -0,0 +1,505 @@
//! `/audit/subject/{id}` HTTP endpoint — legal-tier subject audit response.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §4 + §6.
//!
//! Responds to: `GET /subject/{candidate_id}?from=ISO&to=ISO`
//! Auth: requires `X-Lakehouse-Legal-Token` header matching the value in
//! the legal-token file (see LegalAuditAuth). Constant-time compare.
//!
//! Response shape (subject_audit_response.v1):
//! - manifest: full SubjectManifest from catalogd
//! - audit_log.rows: SubjectAuditRow entries within [from, to]
//! - audit_log.chain_verified: bool (HMAC chain verifies end-to-end)
//! - completeness_attestation: text the lawyer can quote
//!
//! V1 deliberately does NOT include:
//! - Ed25519 signature on the response — chain verification IS the
//! v1 attestation; signing is future hardening (spec §10 v2 reserve)
//! - Live row projection from the underlying datasets — caller can
//! fetch via /query/sql against the safe_views named in the manifest;
//! keeping projection out of this endpoint avoids depending on queryd
//! - External anchor / S3 Object Lock — local HMAC chain is the v1
//! integrity surface
//!
//! These are documented as future hardening, not silent gaps.
use crate::registry::Registry;
use crate::subject_audit::SubjectAuditWriter;
use axum::{
extract::{Path, Query, State},
http::{HeaderMap, StatusCode},
response::IntoResponse,
routing::get,
Json, Router,
};
use serde::{Deserialize, Serialize};
use shared::types::{SubjectAuditRow, SubjectManifest};
use std::sync::Arc;
const LEGAL_TOKEN_HEADER: &str = "x-lakehouse-legal-token";
const RESPONSE_SCHEMA: &str = "subject_audit_response.v1";
/// State for the audit endpoint router. Held outside Registry because
/// the audit writer + legal token aren't part of catalogd's data model
/// — they're sidecar concerns specific to this endpoint.
#[derive(Clone)]
pub struct AuditEndpointState {
pub registry: Registry,
pub writer: Arc<SubjectAuditWriter>,
/// Legal-tier token. Constant-time compared against the header on
/// every request. None = endpoint returns 503 (audit unavailable).
pub legal_token: Option<Arc<String>>,
}
impl AuditEndpointState {
/// Construct from existing pieces. `legal_token_path` is read at
/// construction and trimmed of trailing whitespace. Missing/empty
/// = endpoint will 503 every request with a clear reason.
pub async fn new(
registry: Registry,
writer: Arc<SubjectAuditWriter>,
legal_token_path: &std::path::Path,
) -> Self {
let legal_token = match tokio::fs::read_to_string(legal_token_path).await {
Ok(s) => {
let trimmed = s.trim().to_string();
if trimmed.is_empty() {
tracing::warn!(
"audit endpoint: legal token file at {} is empty — endpoint will 503",
legal_token_path.display()
);
None
} else if trimmed.len() < 32 {
// 2026-05-03 kimi scrum BLOCK at audit_endpoint.rs:59-60:
// 16-byte minimum was too permissive for a legal-tier
// secret. Aligned to the SubjectAuditWriter signing-key
// minimum (32 bytes — HMAC-SHA256 best practice).
tracing::warn!(
"audit endpoint: legal token at {} is {} chars (minimum 32) — endpoint will 503",
legal_token_path.display(), trimmed.len()
);
None
} else {
tracing::info!("audit endpoint: legal token loaded from {}", legal_token_path.display());
Some(Arc::new(trimmed))
}
}
Err(e) => {
tracing::warn!(
"audit endpoint: legal token unreadable from {} ({e}) — endpoint will 503",
legal_token_path.display()
);
None
}
};
Self { registry, writer, legal_token }
}
}
pub fn router(state: AuditEndpointState) -> Router {
Router::new()
.route("/subject/{candidate_id}", get(audit_subject))
.route("/health", get(audit_health))
.with_state(state)
}
async fn audit_health(State(state): State<AuditEndpointState>) -> impl IntoResponse {
if state.legal_token.is_some() {
(StatusCode::OK, "audit endpoint ready").into_response()
} else {
(StatusCode::SERVICE_UNAVAILABLE, "audit endpoint disabled (legal token missing)").into_response()
}
}
#[derive(Deserialize)]
struct AuditQuery {
/// ISO-8601 start of audit window. None = unbounded back.
from: Option<chrono::DateTime<chrono::Utc>>,
/// ISO-8601 end of audit window. None = unbounded forward.
to: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Serialize)]
struct AuditWindowEcho {
from: Option<chrono::DateTime<chrono::Utc>>,
to: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Serialize)]
struct AuditLogSection {
rows_in_window: usize,
rows: Vec<SubjectAuditRow>,
chain_root: String,
chain_verified: bool,
chain_rows_total: usize,
chain_verification_error: Option<String>,
}
#[derive(Serialize)]
struct AuditResponse {
schema: &'static str,
candidate_id: String,
generated_at: chrono::DateTime<chrono::Utc>,
generated_by: &'static str,
request_window: AuditWindowEcho,
manifest: SubjectManifest,
audit_log: AuditLogSection,
/// Datasets the manifest says contain rows for this subject — caller
/// can issue follow-up queries against the named safe_views to get
/// the actual row projection. Keeping projection out of this endpoint
/// avoids depending on queryd from catalogd.
datasets_referenced: Vec<DatasetRefEcho>,
safe_views_available: Vec<String>,
completeness_attestation: String,
}
#[derive(Serialize)]
struct DatasetRefEcho {
name: String,
key_column: String,
key_value: String,
}
/// Constant-time string comparison to avoid timing leaks on the legal
/// token check. `subtle` would be the standard but introducing a dep
/// for one comparison is overkill — manual byte-eq with an unconditional
/// fold over both byte slices.
fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
if a.len() != b.len() {
return false;
}
let mut diff: u8 = 0;
for (x, y) in a.iter().zip(b.iter()) {
diff |= x ^ y;
}
diff == 0
}
/// Extract + verify the legal-tier token from the request. Returns
/// Err((status, body)) on any failure (missing header, missing token
/// configured, mismatch). Does NOT log token values.
fn require_legal_auth(
state: &AuditEndpointState,
headers: &HeaderMap,
) -> Result<(), (StatusCode, &'static str)> {
let configured = state.legal_token.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"audit endpoint disabled — no legal token configured on this gateway",
))?;
let provided = headers.get(LEGAL_TOKEN_HEADER).ok_or((
StatusCode::UNAUTHORIZED,
"missing X-Lakehouse-Legal-Token header",
))?;
let provided = provided.to_str().map_err(|_| (
StatusCode::UNAUTHORIZED,
"X-Lakehouse-Legal-Token contains non-ASCII characters",
))?;
if !constant_time_eq(provided.as_bytes(), configured.as_bytes()) {
return Err((StatusCode::UNAUTHORIZED, "X-Lakehouse-Legal-Token mismatch"));
}
Ok(())
}
async fn audit_subject(
State(state): State<AuditEndpointState>,
Path(candidate_id): Path<String>,
Query(q): Query<AuditQuery>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err((status, msg)) = require_legal_auth(&state, &headers) {
return (status, msg).into_response();
}
if candidate_id.is_empty() {
return (StatusCode::BAD_REQUEST, "candidate_id is empty").into_response();
}
// 1. Load the subject manifest.
let manifest = match state.registry.get_subject(&candidate_id).await {
Some(m) => m,
None => return (
StatusCode::NOT_FOUND,
format!("subject manifest not found for candidate_id={candidate_id}"),
).into_response(),
};
// 2. Read audit rows in window.
let rows = match state.writer.read_rows_in_range(&candidate_id, q.from, q.to).await {
Ok(r) => r,
Err(e) => return (
StatusCode::INTERNAL_SERVER_ERROR,
format!("audit log read failed: {e}"),
).into_response(),
};
// 3. Verify chain end-to-end (full log, not just the windowed slice;
// chain integrity is global, not per-window). chain_root MUST
// come from the actual log tip — taking it from the windowed
// `rows` would lie when `to` is set or the window catches no
// rows. (2026-05-03 opus scrum WARN audit_endpoint.rs:206.)
let (chain_verified, chain_rows_total, chain_root, chain_err) =
match state.writer.verify_chain(&candidate_id).await {
Ok(n) => {
let tip = state.writer.chain_tip(&candidate_id).await
.unwrap_or_else(|| "GENESIS".into());
(true, n, tip, None)
}
Err(e) => (false, 0, String::new(), Some(e)),
};
// 4. Tampering detection: if the manifest claims a non-trivial
// chain root but the log is empty (chain_rows_total == 0), the
// log was deleted between writes. Flag it. (2026-05-03 opus
// scrum WARN subject_audit.rs:347 + audit_endpoint coverage.)
let manifest_root = manifest.audit_log_chain_root.clone();
let log_was_tampered = chain_verified
&& chain_rows_total == 0
&& !manifest_root.is_empty()
&& manifest_root != "GENESIS";
let (chain_verified, chain_err) = if log_was_tampered {
(false, Some(format!(
"manifest expects chain root '{}' but the audit log has 0 rows; log was deleted or wiped",
manifest_root,
)))
} else {
(chain_verified, chain_err)
};
let datasets_referenced = manifest.datasets.iter()
.map(|d| DatasetRefEcho {
name: d.name.clone(),
key_column: d.key_column.clone(),
key_value: d.key_value.clone(),
})
.collect();
let safe_views_available = manifest.safe_views.clone();
let attestation = format!(
"All audit rows for candidate_id={} within [{}, {}] are included. \
HMAC chain verified end-to-end: {} rows total, integrity={}.",
candidate_id,
q.from.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
q.to.map(|d| d.to_rfc3339()).unwrap_or_else(|| "unbounded".into()),
chain_rows_total,
if chain_verified { "verified" } else { "BROKEN" },
);
let resp = AuditResponse {
schema: RESPONSE_SCHEMA,
candidate_id: candidate_id.clone(),
generated_at: chrono::Utc::now(),
generated_by: "catalogd",
request_window: AuditWindowEcho { from: q.from, to: q.to },
manifest,
audit_log: AuditLogSection {
rows_in_window: rows.len(),
rows,
chain_root,
chain_verified,
chain_rows_total,
chain_verification_error: chain_err,
},
datasets_referenced,
safe_views_available,
completeness_attestation: attestation,
};
// Log the legal-tier read so it's discoverable in operations review.
// Token value is NEVER logged. Only the fact of the read.
tracing::info!(
candidate_id = %candidate_id,
rows_in_window = resp.audit_log.rows_in_window,
chain_verified = resp.audit_log.chain_verified,
"legal-tier audit response served"
);
(StatusCode::OK, Json(resp)).into_response()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::subject_audit::SubjectAuditWriter;
use object_store::memory::InMemory;
use shared::types::{
AuditAccessor, BiometricConsent, BiometricConsentStatus, ConsentStatus,
GeneralPiiConsent, SubjectAuditRow, SubjectConsent, SubjectRetention,
SubjectStatus, SubjectVertical,
};
fn fixture_manifest(candidate_id: &str) -> SubjectManifest {
SubjectManifest {
schema: "subject_manifest.v1".into(),
candidate_id: candidate_id.into(),
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
status: SubjectStatus::Active,
vertical: SubjectVertical::Unknown,
consent: SubjectConsent {
general_pii: GeneralPiiConsent {
status: ConsentStatus::PendingBackfillReview,
version: String::new(),
given_at: None,
withdrawn_at: None,
},
biometric: BiometricConsent {
status: BiometricConsentStatus::NeverCollected,
retention_until: None,
},
},
retention: SubjectRetention {
general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
policy: "test".into(),
},
datasets: vec![],
safe_views: vec![],
audit_log_path: String::new(),
audit_log_chain_root: String::new(),
biometric_collection: None,
}
}
fn fixture_audit_row(candidate_id: &str) -> SubjectAuditRow {
SubjectAuditRow {
schema: "subject_audit.v1".into(),
ts: chrono::Utc::now(),
candidate_id: candidate_id.into(),
accessor: AuditAccessor {
kind: "test".into(),
daemon: "test".into(),
purpose: "test".into(),
trace_id: String::new(),
},
fields_accessed: vec!["name".into()],
result: "success".into(),
prev_chain_hash: String::new(),
row_hmac: String::new(),
}
}
async fn fixture_state_with_token(token: &str) -> AuditEndpointState {
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let registry = Registry::new(store.clone());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(
store, (0u8..32).collect(),
));
AuditEndpointState {
registry,
writer,
legal_token: Some(Arc::new(token.into())),
}
}
#[test]
fn constant_time_eq_basic() {
assert!(constant_time_eq(b"abc", b"abc"));
assert!(!constant_time_eq(b"abc", b"abd"));
assert!(!constant_time_eq(b"abc", b"abcd"));
assert!(constant_time_eq(b"", b""));
}
#[tokio::test]
async fn missing_legal_token_returns_503() {
let state = AuditEndpointState {
registry: Registry::new(Arc::new(InMemory::new())),
writer: Arc::new(SubjectAuditWriter::with_inline_key(
Arc::new(InMemory::new()), (0u8..32).collect(),
)),
legal_token: None,
};
let headers = HeaderMap::new();
let r = require_legal_auth(&state, &headers);
assert!(matches!(r, Err((StatusCode::SERVICE_UNAVAILABLE, _))));
}
#[tokio::test]
async fn missing_header_returns_401() {
let state = fixture_state_with_token("test-token-must-be-long-enough").await;
let headers = HeaderMap::new();
let r = require_legal_auth(&state, &headers);
assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
}
#[tokio::test]
async fn wrong_token_returns_401() {
let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
let mut headers = HeaderMap::new();
headers.insert(LEGAL_TOKEN_HEADER, "wrong".parse().unwrap());
let r = require_legal_auth(&state, &headers);
assert!(matches!(r, Err((StatusCode::UNAUTHORIZED, _))));
}
#[tokio::test]
async fn correct_token_passes_auth() {
let state = fixture_state_with_token("correct-token-must-be-long-enough").await;
let mut headers = HeaderMap::new();
headers.insert(LEGAL_TOKEN_HEADER, "correct-token-must-be-long-enough".parse().unwrap());
let r = require_legal_auth(&state, &headers);
assert!(r.is_ok());
}
#[tokio::test]
async fn audit_response_assembly_full_path() {
let state = fixture_state_with_token("test-token-must-be-long-enough").await;
// Plant a manifest + 3 audit rows.
state.registry.put_subject(fixture_manifest("CAND-AUDIT-1")).await.unwrap();
for _ in 0..3 {
state.writer.append(fixture_audit_row("CAND-AUDIT-1")).await.unwrap();
}
// Read rows in unbounded window.
let rows = state.writer
.read_rows_in_range("CAND-AUDIT-1", None, None)
.await
.unwrap();
assert_eq!(rows.len(), 3);
// Verify chain integrity.
let n = state.writer.verify_chain("CAND-AUDIT-1").await.unwrap();
assert_eq!(n, 3);
}
#[tokio::test]
async fn audit_response_window_filters_rows() {
let state = fixture_state_with_token("test-token-must-be-long-enough").await;
state.registry.put_subject(fixture_manifest("CAND-WIN-1")).await.unwrap();
// Append 5 rows.
let mut row_times = Vec::new();
for _ in 0..5 {
let row = fixture_audit_row("CAND-WIN-1");
row_times.push(row.ts);
state.writer.append(row).await.unwrap();
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
// Window covering only the middle 3 rows.
let from = row_times[1];
let to = row_times[3];
let rows = state.writer
.read_rows_in_range("CAND-WIN-1", Some(from), Some(to))
.await
.unwrap();
assert!(rows.len() >= 3, "expected ≥3 rows in window; got {}", rows.len());
assert!(rows.len() <= 3, "expected ≤3 rows in window; got {}", rows.len());
}
#[tokio::test]
async fn empty_token_file_results_in_disabled_endpoint() {
let tmp = std::env::temp_dir().join(format!("legal_empty_{}.tok", std::process::id()));
tokio::fs::write(&tmp, "").await.unwrap();
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let registry = Registry::new(store.clone());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
let state = AuditEndpointState::new(registry, writer, &tmp).await;
assert!(state.legal_token.is_none());
let _ = tokio::fs::remove_file(&tmp).await;
}
#[tokio::test]
async fn short_token_file_rejected_at_load() {
let tmp = std::env::temp_dir().join(format!("legal_short_{}.tok", std::process::id()));
tokio::fs::write(&tmp, "tooshort").await.unwrap();
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let registry = Registry::new(store.clone());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(store, (0u8..32).collect()));
let state = AuditEndpointState::new(registry, writer, &tmp).await;
assert!(state.legal_token.is_none());
let _ = tokio::fs::remove_file(&tmp).await;
}
}
@@ -0,0 +1,341 @@
//! Backfill subject manifests from a parquet source.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3.
//!
//! For each row in the source parquet, creates a SubjectManifest with
//! the BIPA-defensible defaults from the spec:
//! - vertical = unknown (HIPAA fail-closed routing)
//! - consent.general_pii = pending_backfill_review (NOT inferred_existing)
//! - consent.biometric = never_collected (no biometric backfill)
//! - retention.general_pii_until = now + 4 years (default policy)
//!
//! Idempotent: if a manifest already exists for a candidate_id, the
//! backfill skips it. Re-runs are safe.
//!
//! Conservative defaults: --limit 1000 by default. Pass --all to do
//! the full source. Pass --concurrency to bound parallel writes.
//!
//! Usage examples:
//! cargo run --bin backfill_subjects -- \
//! --source data/datasets/workers_500k.parquet \
//! --dataset workers_500k \
//! --key-column worker_id \
//! --candidate-id-prefix WORKER- \
//! --limit 100 \
//! --dry-run
//!
//! cargo run --bin backfill_subjects -- \
//! --source data/datasets/workers_500k.parquet \
//! --dataset workers_500k \
//! --key-column worker_id \
//! --candidate-id-prefix WORKER- \
//! --safe-view workers_safe \
//! --all
//!
//! On the live host the binary reads from / writes to ./data so it
//! integrates with the running catalogd's persistence path.
use catalogd::registry::Registry;
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
use shared::types::{
BiometricConsent, BiometricConsentStatus, ConsentStatus, GeneralPiiConsent,
SubjectConsent, SubjectDatasetRef, SubjectManifest, SubjectRetention,
SubjectStatus, SubjectVertical,
};
use std::fs::File;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use storaged::backend;
const DEFAULT_LIMIT: usize = 1000;
const DEFAULT_CONCURRENCY: usize = 32;
const DEFAULT_RETENTION_YEARS: i64 = 4;
#[derive(Debug)]
struct Args {
source: PathBuf,
dataset_name: String,
key_column: String,
candidate_id_prefix: String,
safe_view: Option<String>,
storage_root: PathBuf,
limit: Option<usize>,
concurrency: usize,
dry_run: bool,
}
fn parse_args() -> Result<Args, String> {
let argv: Vec<String> = std::env::args().collect();
let mut source: Option<PathBuf> = None;
let mut dataset_name: Option<String> = None;
let mut key_column: Option<String> = None;
let mut candidate_id_prefix: String = String::new();
let mut safe_view: Option<String> = None;
let mut storage_root: PathBuf = PathBuf::from("./data");
let mut limit: Option<usize> = Some(DEFAULT_LIMIT);
let mut concurrency: usize = DEFAULT_CONCURRENCY;
let mut dry_run = false;
let mut i = 1;
while i < argv.len() {
let arg = &argv[i];
let next = || -> Result<String, String> {
argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
};
match arg.as_str() {
"--source" => { source = Some(PathBuf::from(next()?)); i += 2; }
"--dataset" => { dataset_name = Some(next()?); i += 2; }
"--key-column" => { key_column = Some(next()?); i += 2; }
"--candidate-id-prefix" => { candidate_id_prefix = next()?; i += 2; }
"--safe-view" => { safe_view = Some(next()?); i += 2; }
"--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
"--limit" => { limit = Some(next()?.parse().map_err(|e| format!("--limit: {e}"))?); i += 2; }
"--all" => { limit = None; i += 1; }
"--concurrency" => { concurrency = next()?.parse().map_err(|e| format!("--concurrency: {e}"))?; i += 2; }
"--dry-run" => { dry_run = true; i += 1; }
"-h" | "--help" => { print_help(); std::process::exit(0); }
other => return Err(format!("unknown arg: {other}")),
}
}
Ok(Args {
source: source.ok_or("missing --source")?,
dataset_name: dataset_name.ok_or("missing --dataset")?,
key_column: key_column.ok_or("missing --key-column")?,
candidate_id_prefix,
safe_view,
storage_root,
limit,
concurrency,
dry_run,
})
}
fn print_help() {
eprintln!("backfill_subjects — populate subject manifests from a parquet source\n");
eprintln!("Usage: backfill_subjects [flags]\n");
eprintln!("Required:");
eprintln!(" --source PATH parquet file containing subject identifiers");
eprintln!(" --dataset NAME catalogd dataset name (must already exist)");
eprintln!(" --key-column COL column in the parquet that holds subject identifiers\n");
eprintln!("Optional:");
eprintln!(" --candidate-id-prefix STR prefix added to each row's value to form candidate_id");
eprintln!(" (e.g. 'WORKER-' on a worker_id int64 column)");
eprintln!(" --safe-view NAME name of an AiView that safely projects this subject's data");
eprintln!(" --storage-root PATH local-filesystem catalog root (default: ./data)");
eprintln!(" --limit N process at most N rows (default: {DEFAULT_LIMIT})");
eprintln!(" --all process every row in the source");
eprintln!(" --concurrency N max concurrent writes (default: {DEFAULT_CONCURRENCY})");
eprintln!(" --dry-run parse + count + sample, don't write anything");
}
#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() {
let args = match parse_args() {
Ok(a) => a,
Err(e) => {
eprintln!("error: {e}");
print_help();
std::process::exit(2);
}
};
if let Err(e) = run(args).await {
eprintln!("backfill failed: {e}");
std::process::exit(1);
}
}
async fn run(args: Args) -> Result<(), String> {
eprintln!("[backfill] source: {}", args.source.display());
eprintln!("[backfill] dataset: {}", args.dataset_name);
eprintln!("[backfill] key: {} (prefix: {:?})", args.key_column, args.candidate_id_prefix);
eprintln!("[backfill] limit: {}", args.limit.map(|n| n.to_string()).unwrap_or("all".into()));
eprintln!("[backfill] dry-run: {}", args.dry_run);
// Stand up catalogd over the same local FS the running daemon uses.
let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
let reg = Arc::new(Registry::new(store));
reg.rebuild().await?;
if reg.get_by_name(&args.dataset_name).await.is_none() {
return Err(format!(
"dataset '{}' not found in catalog at {}; backfill needs the dataset registered first",
args.dataset_name, args.storage_root.display(),
));
}
eprintln!("[backfill] dataset '{}' exists in catalog ✓", args.dataset_name);
// Open parquet reader.
let file = File::open(&args.source).map_err(|e| format!("open {}: {e}", args.source.display()))?;
let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(|e| e.to_string())?;
let total_rows = builder.metadata().file_metadata().num_rows() as usize;
eprintln!("[backfill] parquet rows: {total_rows}");
// Project only the key column for IO efficiency.
let schema = builder.schema().clone();
let col_idx = schema.index_of(&args.key_column)
.map_err(|_| format!("column '{}' not found; parquet has: {:?}",
args.key_column,
schema.fields().iter().map(|f| f.name()).collect::<Vec<_>>(),
))?;
let projection = parquet::arrow::ProjectionMask::roots(builder.parquet_schema(), [col_idx]);
let mut reader = builder.with_projection(projection).build().map_err(|e| e.to_string())?;
// Collect candidate IDs (limited).
let cap = args.limit.unwrap_or(total_rows);
let mut candidate_ids: Vec<String> = Vec::with_capacity(cap.min(total_rows));
let mut rows_seen = 0usize;
'outer: while let Some(batch_result) = reader.next() {
let batch = batch_result.map_err(|e| format!("batch: {e}"))?;
let column = batch.column(0); // single projected column
for row in 0..batch.num_rows() {
if let Some(limit) = args.limit {
if candidate_ids.len() >= limit { break 'outer; }
}
rows_seen += 1;
let raw = format_cell_value(column, row)?;
candidate_ids.push(format!("{}{}", args.candidate_id_prefix, raw));
}
}
eprintln!("[backfill] collected {} candidate_ids ({} rows scanned)", candidate_ids.len(), rows_seen);
// Sample for the operator.
eprintln!("[backfill] sample (first 5):");
for cid in candidate_ids.iter().take(5) {
eprintln!(" - {cid}");
}
if args.dry_run {
eprintln!("[backfill] DRY-RUN — no writes. Re-run without --dry-run to commit.");
return Ok(());
}
// Write manifests with bounded concurrency.
let now = chrono::Utc::now();
let retention_until = now + chrono::Duration::days(365 * DEFAULT_RETENTION_YEARS);
let safe_views: Vec<String> = args.safe_view.into_iter().collect();
let dataset_ref = SubjectDatasetRef {
name: args.dataset_name.clone(),
key_column: args.key_column.clone(),
key_value: String::new(), // filled per-subject below
};
let attempted = Arc::new(AtomicUsize::new(0));
let inserted = Arc::new(AtomicUsize::new(0));
let skipped = Arc::new(AtomicUsize::new(0));
let failed = Arc::new(AtomicUsize::new(0));
let progress_every = ((candidate_ids.len() / 20).max(1)).min(5_000);
// Bounded concurrency via Semaphore.
let sem = Arc::new(tokio::sync::Semaphore::new(args.concurrency));
let mut handles = Vec::with_capacity(candidate_ids.len());
for (idx, cid) in candidate_ids.into_iter().enumerate() {
let permit = sem.clone().acquire_owned().await.map_err(|e| e.to_string())?;
let reg = reg.clone();
let mut ds_ref = dataset_ref.clone();
// strip_prefix gives one-shot semantics; trim_start_matches strips
// repeated occurrences (a "WORKER-WORKER-1" id would lose both
// prefixes). 2026-05-03 opus scrum WARN at backfill_subjects.rs:189.
ds_ref.key_value = cid.strip_prefix(&args.candidate_id_prefix)
.unwrap_or(&cid).to_string();
let safe_views = safe_views.clone();
let attempted = attempted.clone();
let inserted = inserted.clone();
let skipped = skipped.clone();
let failed = failed.clone();
let handle = tokio::spawn(async move {
let _permit = permit;
attempted.fetch_add(1, Ordering::Relaxed);
// Idempotency: skip if subject already exists.
if reg.get_subject(&cid).await.is_some() {
skipped.fetch_add(1, Ordering::Relaxed);
return;
}
let manifest = SubjectManifest {
schema: "subject_manifest.v1".into(),
candidate_id: cid.clone(),
created_at: now,
updated_at: now,
status: SubjectStatus::Active,
vertical: SubjectVertical::Unknown,
consent: SubjectConsent {
general_pii: GeneralPiiConsent {
status: ConsentStatus::PendingBackfillReview,
version: String::new(),
given_at: None,
withdrawn_at: None,
},
biometric: BiometricConsent {
status: BiometricConsentStatus::NeverCollected,
retention_until: None,
},
},
retention: SubjectRetention {
general_pii_until: retention_until,
policy: "4_year_default".into(),
},
datasets: vec![ds_ref],
safe_views,
audit_log_path: String::new(),
audit_log_chain_root: String::new(),
biometric_collection: None,
};
match reg.put_subject(manifest).await {
Ok(_) => { inserted.fetch_add(1, Ordering::Relaxed); }
Err(e) => {
failed.fetch_add(1, Ordering::Relaxed);
eprintln!(" ✗ {cid}: {e}");
}
}
let cur = attempted.load(Ordering::Relaxed);
if cur % progress_every == 0 {
eprintln!("[backfill] progress: {} attempted ({} inserted, {} skipped, {} failed)",
cur, inserted.load(Ordering::Relaxed),
skipped.load(Ordering::Relaxed),
failed.load(Ordering::Relaxed));
}
let _ = idx;
});
handles.push(handle);
}
for h in handles {
let _ = h.await;
}
eprintln!();
eprintln!("──────────────────────────────────────────────────────────────────");
eprintln!("[backfill] done");
eprintln!("[backfill] attempted: {}", attempted.load(Ordering::Relaxed));
eprintln!("[backfill] inserted: {}", inserted.load(Ordering::Relaxed));
eprintln!("[backfill] skipped: {} (already existed — idempotent re-run)", skipped.load(Ordering::Relaxed));
eprintln!("[backfill] failed: {}", failed.load(Ordering::Relaxed));
eprintln!("[backfill] subjects in catalog now: {}", reg.subjects_count().await);
Ok(())
}
/// Format a single cell value as a string for use in the candidate_id.
/// Supports the integer + string types most likely to be subject keys.
fn format_cell_value(array: &Arc<dyn arrow::array::Array>, row: usize) -> Result<String, String> {
use arrow::array::{Int32Array, Int64Array, StringArray, UInt32Array, UInt64Array};
use arrow::datatypes::DataType;
if array.is_null(row) {
return Err(format!("row {row}: key column is null — cannot form candidate_id"));
}
let dt = array.data_type();
let s = match dt {
DataType::Int64 => array.as_any().downcast_ref::<Int64Array>()
.ok_or("int64 downcast")?.value(row).to_string(),
DataType::Int32 => array.as_any().downcast_ref::<Int32Array>()
.ok_or("int32 downcast")?.value(row).to_string(),
DataType::UInt64 => array.as_any().downcast_ref::<UInt64Array>()
.ok_or("uint64 downcast")?.value(row).to_string(),
DataType::UInt32 => array.as_any().downcast_ref::<UInt32Array>()
.ok_or("uint32 downcast")?.value(row).to_string(),
DataType::Utf8 => array.as_any().downcast_ref::<StringArray>()
.ok_or("utf8 downcast")?.value(row).to_string(),
other => return Err(format!("unsupported key column type: {other:?}")),
};
Ok(s)
}
@@ -0,0 +1,235 @@
//! Cross-runtime parity helper for subject-audit chain.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.
//!
//! This binary is consumed by the Go-repo's
//! scripts/cutover/parity/subject_audit_parity.sh.
//! Go counterpart: scripts/cutover/parity/subject_audit_helper/main.go
//! (in the Go repo). Both helpers are kept in lockstep by the parity
//! script's byte-equality assertions — change one, change both.
//!
//! Both helpers MUST produce byte-identical output for the same inputs.
//! Divergence here is a parity break — a SubjectManifest written by Rust
//! that Go cannot verify, or vice versa.
//!
//! Two modes:
//!
//! --known-answer
//! Print the canonical bytes + HMAC of a hardcoded fixture row.
//! The Go helper must produce IDENTICAL bytes + IDENTICAL hash.
//! Hardcoded fixture matches Go test TestKnownAnswerVector.
//!
//! --verify <audit_log_path> --key <key_path>
//! Parse the JSONL audit log, replay the HMAC chain. Print
//! JSON: {"count": N, "tip": "<hash|GENESIS>", "verified": bool, "error": "<msg|null>"}.
//!
//! Output format: ONE JSON object per stdout, terminating newline. The
//! parity script diffs Rust stdout vs Go stdout via `diff -q`.
use catalogd::subject_audit::SubjectAuditWriter;
use hmac::{Hmac, Mac};
use object_store::ObjectStore;
use object_store::memory::InMemory;
use serde::Serialize;
use sha2::Sha256;
use shared::types::{AuditAccessor, SubjectAuditRow};
use std::collections::BTreeMap;
use std::path::PathBuf;
use std::sync::Arc;
type HmacSha256 = Hmac<Sha256>;
const GENESIS: &str = "GENESIS";
/// Same canonical-JSON algorithm as crates/catalogd/src/subject_audit.rs.
/// Reproduced here so the helper does not depend on internal-only items
/// in that file. If the algorithm there changes, change it here AND in
/// the Go helper in lockstep.
fn canonical_json(v: &serde_json::Value) -> Vec<u8> {
fn rewrite(v: &serde_json::Value) -> serde_json::Value {
match v {
serde_json::Value::Object(map) => {
let sorted: BTreeMap<String, serde_json::Value> = map
.iter()
.map(|(k, v)| (k.clone(), rewrite(v)))
.collect();
serde_json::Value::Object(sorted.into_iter().collect())
}
serde_json::Value::Array(arr) => {
serde_json::Value::Array(arr.iter().map(rewrite).collect())
}
other => other.clone(),
}
}
serde_json::to_vec(&rewrite(v)).expect("canonical-json serialize")
}
fn canonical_row_bytes(row: &SubjectAuditRow) -> Vec<u8> {
let mut v: serde_json::Value = serde_json::to_value(row).expect("row to value");
if let Some(obj) = v.as_object_mut() {
obj.remove("row_hmac");
}
canonical_json(&v)
}
fn compute_hmac(key: &[u8], prev: &str, canonical: &[u8]) -> String {
let mut mac = <HmacSha256 as Mac>::new_from_slice(key).expect("HMAC accepts any key length");
mac.update(prev.as_bytes());
mac.update(canonical);
let result = mac.finalize().into_bytes();
const HEX: &[u8; 16] = b"0123456789abcdef";
let mut s = String::with_capacity(64);
for byte in result {
s.push(HEX[(byte >> 4) as usize] as char);
s.push(HEX[(byte & 0x0f) as usize] as char);
}
s
}
fn deterministic_key() -> Vec<u8> {
(0u8..32).collect()
}
/// Print to stderr + exit(2). Never returns. Used in place of `.expect()`
/// on argv / file reads so failures produce a clean usage error rather
/// than a Rust panic backtrace that the parity script would mistake
/// for a parity break.
fn die(msg: &str) -> ! {
eprintln!("error: {msg}");
std::process::exit(2);
}
#[derive(Serialize)]
struct KnownAnswerOut {
mode: &'static str,
canonical: String,
hmac: String,
canonical_bytes_len: usize,
}
#[derive(Serialize)]
struct VerifyOut {
mode: &'static str,
count: usize,
tip: String,
verified: bool,
error: Option<String>,
}
fn known_answer() {
let row = SubjectAuditRow {
schema: "subject_audit.v1".into(),
ts: chrono::DateTime::parse_from_rfc3339("2026-05-03T12:00:00Z")
.unwrap()
.with_timezone(&chrono::Utc),
candidate_id: "WORKER-FIXED".into(),
accessor: AuditAccessor {
kind: "gateway_lookup".into(),
daemon: "gateway".into(),
purpose: "parity_test".into(),
trace_id: "trace-fixed".into(),
},
fields_accessed: vec!["name".into()],
result: "success".into(),
prev_chain_hash: GENESIS.into(),
row_hmac: String::new(),
};
let canon = canonical_row_bytes(&row);
let hmac = compute_hmac(&deterministic_key(), GENESIS, &canon);
let out = KnownAnswerOut {
mode: "known_answer",
canonical: String::from_utf8(canon.clone()).expect("canonical is utf-8"),
hmac,
canonical_bytes_len: canon.len(),
};
println!("{}", serde_json::to_string(&out).unwrap());
}
#[tokio::main(flavor = "current_thread")]
async fn main() {
let argv: Vec<String> = std::env::args().collect();
let mut mode_known_answer = false;
let mut audit_path: Option<PathBuf> = None;
let mut key_path: Option<PathBuf> = None;
// Argv parsing fails via stderr+exit, never via panic. The parity
// script captures stdout for byte-comparison; a Rust panic backtrace
// would land in stdout (the script merges 2>&1) and read as a parity
// break instead of a usage error. (Caught 2026-05-03 opus scrum WARN.)
let mut i = 1;
while i < argv.len() {
match argv[i].as_str() {
"--known-answer" => {
mode_known_answer = true;
i += 1;
}
"--verify" => {
let p = argv.get(i + 1).unwrap_or_else(|| die("--verify needs a path"));
audit_path = Some(PathBuf::from(p));
i += 2;
}
"--key" => {
let p = argv.get(i + 1).unwrap_or_else(|| die("--key needs a path"));
key_path = Some(PathBuf::from(p));
i += 2;
}
"-h" | "--help" => {
eprintln!("parity_subject_audit --known-answer");
eprintln!("parity_subject_audit --verify <audit_log_path> --key <key_path>");
std::process::exit(0);
}
other => {
eprintln!("unknown arg: {other}");
std::process::exit(2);
}
}
}
if mode_known_answer {
known_answer();
return;
}
let audit_path = audit_path.unwrap_or_else(|| die("need --known-answer OR --verify <path> --key <path>"));
let key_path = key_path.unwrap_or_else(|| die("--verify also needs --key"));
let key = std::fs::read(&key_path).unwrap_or_else(|e| die(&format!("read key file: {e}")));
let candidate_id = audit_path
.file_name()
.and_then(|s| s.to_str())
.and_then(|s| s.strip_suffix(".audit.jsonl"))
.unwrap_or_else(|| die("audit log path must end with <candidate_id>.audit.jsonl"))
.to_string();
// Stand up an in-memory object store, seed it with the audit log
// bytes at the canonical key, then ask SubjectAuditWriter to verify.
// This way we exercise the SAME verify_chain function the production
// gateway calls — not a re-implementation that might drift.
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
let bytes = std::fs::read(&audit_path).expect("read audit log");
let log_key = format!("_catalog/subjects/{}.audit.jsonl", candidate_id);
storaged::ops::put(&store, &log_key, bytes::Bytes::from(bytes))
.await
.expect("seed object store");
let writer = SubjectAuditWriter::with_inline_key(store, key);
let result = writer.verify_chain(&candidate_id).await;
let tip = writer.chain_tip(&candidate_id).await.unwrap_or(GENESIS.into());
let out = match result {
Ok(count) => VerifyOut {
mode: "verify",
count,
tip,
verified: true,
error: None,
},
Err(e) => VerifyOut {
mode: "verify",
count: 0,
tip: GENESIS.into(),
verified: false,
error: Some(e),
},
};
println!("{}", serde_json::to_string(&out).unwrap());
}
+376
View File
@@ -0,0 +1,376 @@
//! Daily retention sweep for subject manifests.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7:
//! "Subjects whose retention.general_pii_until < now AND status != erased
//! get marked for review (don't auto-delete; legal needs to approve)."
//!
//! Per shared::types::BiometricConsent doc-comment (BIPA requirement):
//! "max 3 years from last interaction. Implementation MUST enforce
//! daily expiration sweep against this field."
//!
//! Therefore this binary checks BOTH retention clocks and reports any
//! subject whose general-pii window OR biometric window has expired
//! while the subject is not already erased / already-flagged.
//!
//! It does NOT mutate subject manifests. It produces a JSONL review
//! queue at:
//! data/_catalog/subjects/_retention_sweep_<YYYY-MM-DD>.jsonl
//!
//! Operators / legal review the queue and act:
//! - extend retention (legal contract update),
//! - flip status to retention_expired,
//! - schedule erasure.
//!
//! Usage:
//! cargo run --bin retention_sweep # dry-run, stderr only
//! cargo run --bin retention_sweep -- --apply # also write report file
//! cargo run --bin retention_sweep -- --as-of 2030-01-01T00:00:00Z
//! cargo run --bin retention_sweep -- --storage-root /var/lib/lakehouse/data --apply
use catalogd::registry::Registry;
use chrono::{DateTime, Utc};
use serde::Serialize;
use shared::types::{BiometricConsentStatus, SubjectManifest, SubjectStatus};
use std::path::PathBuf;
use std::sync::Arc;
use storaged::backend;
#[derive(Debug)]
struct Args {
storage_root: PathBuf,
apply: bool,
as_of: Option<DateTime<Utc>>,
}
fn parse_args() -> Result<Args, String> {
let argv: Vec<String> = std::env::args().collect();
let mut storage_root = PathBuf::from("./data");
let mut apply = false;
let mut as_of: Option<DateTime<Utc>> = None;
let mut i = 1;
while i < argv.len() {
let arg = &argv[i];
let next = || -> Result<String, String> {
argv.get(i + 1).cloned().ok_or_else(|| format!("{arg} needs a value"))
};
match arg.as_str() {
"--storage-root" => { storage_root = PathBuf::from(next()?); i += 2; }
"--apply" => { apply = true; i += 1; }
"--as-of" => {
let raw = next()?;
as_of = Some(raw.parse().map_err(|e| format!("--as-of (need RFC3339): {e}"))?);
i += 2;
}
"-h" | "--help" => { print_help(); std::process::exit(0); }
other => return Err(format!("unknown arg: {other}")),
}
}
Ok(Args { storage_root, apply, as_of })
}
fn print_help() {
eprintln!("retention_sweep — daily retention check on subject manifests");
eprintln!();
eprintln!("Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7.");
eprintln!("Reports overdue subjects to a JSONL review queue. Does NOT");
eprintln!("auto-delete. Legal must approve every action.");
eprintln!();
eprintln!("Flags:");
eprintln!(" --storage-root PATH local-filesystem catalog root (default: ./data)");
eprintln!(" --apply write report to data/_catalog/subjects/_retention_sweep_<date>.jsonl");
eprintln!(" --as-of RFC3339 use this timestamp as 'now' (test/forecast helper)");
}
#[derive(Debug, PartialEq, Clone, Copy)]
enum OverdueReason {
GeneralPii,
Biometric,
Both,
}
impl OverdueReason {
fn as_str(self) -> &'static str {
match self {
OverdueReason::GeneralPii => "general_pii_expired",
OverdueReason::Biometric => "biometric_expired",
OverdueReason::Both => "both_expired",
}
}
}
#[derive(Debug, Serialize)]
struct OverdueRow<'a> {
candidate_id: &'a str,
status: &'a str,
general_pii_until: DateTime<Utc>,
days_overdue_general: i64,
biometric_status: &'a str,
#[serde(skip_serializing_if = "Option::is_none")]
biometric_retention_until: Option<DateTime<Utc>>,
#[serde(skip_serializing_if = "Option::is_none")]
days_overdue_biometric: Option<i64>,
reason: &'a str,
swept_at: DateTime<Utc>,
}
/// Skipped statuses (idempotent across daily runs):
/// - Erased: subject's data is already gone; nothing to review.
/// - RetentionExpired: subject is already flagged; do not re-fire.
///
/// Actionable statuses: PendingConsent, Active, Withdrawn — each can
/// still hold PII rows on disk past the retention window.
fn is_overdue(subject: &SubjectManifest, now: DateTime<Utc>) -> Option<OverdueReason> {
if matches!(subject.status, SubjectStatus::Erased | SubjectStatus::RetentionExpired) {
return None;
}
let general_overdue = subject.retention.general_pii_until < now;
let biometric_overdue = subject
.consent
.biometric
.retention_until
.map(|t| t < now)
.unwrap_or(false);
match (general_overdue, biometric_overdue) {
(true, true) => Some(OverdueReason::Both),
(true, false) => Some(OverdueReason::GeneralPii),
(false, true) => Some(OverdueReason::Biometric),
(false, false) => None,
}
}
fn status_str(s: &SubjectStatus) -> &'static str {
match s {
SubjectStatus::PendingConsent => "pending_consent",
SubjectStatus::Active => "active",
SubjectStatus::Withdrawn => "withdrawn",
SubjectStatus::RetentionExpired => "retention_expired",
SubjectStatus::Erased => "erased",
}
}
fn biometric_status_str(s: &BiometricConsentStatus) -> &'static str {
match s {
BiometricConsentStatus::NeverCollected => "never_collected",
BiometricConsentStatus::Pending => "pending",
BiometricConsentStatus::Given => "given",
BiometricConsentStatus::Withdrawn => "withdrawn",
BiometricConsentStatus::Expired => "expired",
}
}
#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
let args = match parse_args() {
Ok(a) => a,
Err(e) => {
eprintln!("error: {e}");
print_help();
std::process::exit(2);
}
};
if let Err(e) = run(args).await {
eprintln!("retention_sweep failed: {e}");
std::process::exit(1);
}
}
async fn run(args: Args) -> Result<(), String> {
let now = args.as_of.unwrap_or_else(Utc::now);
eprintln!("[retention_sweep] storage-root: {}", args.storage_root.display());
eprintln!("[retention_sweep] now (sweep clock): {}", now.to_rfc3339());
eprintln!("[retention_sweep] apply: {}", args.apply);
let store = backend::init_local(args.storage_root.to_str().ok_or("storage-root not utf-8")?);
let reg = Arc::new(Registry::new(store));
reg.rebuild().await?;
let subjects = reg.list_subjects().await;
eprintln!("[retention_sweep] subjects loaded: {}", subjects.len());
let mut overdue_rows: Vec<OverdueRow> = Vec::new();
let mut by_reason: std::collections::BTreeMap<&'static str, usize> = Default::default();
for s in &subjects {
if let Some(reason) = is_overdue(s, now) {
let reason_str: &'static str = reason.as_str();
*by_reason.entry(reason_str).or_insert(0) += 1;
overdue_rows.push(OverdueRow {
candidate_id: &s.candidate_id,
status: status_str(&s.status),
general_pii_until: s.retention.general_pii_until,
days_overdue_general: (now - s.retention.general_pii_until).num_days(),
biometric_status: biometric_status_str(&s.consent.biometric.status),
biometric_retention_until: s.consent.biometric.retention_until,
days_overdue_biometric: s
.consent
.biometric
.retention_until
.map(|t| (now - t).num_days()),
reason: reason_str,
swept_at: now,
});
}
}
eprintln!("[retention_sweep] overdue subjects: {}", overdue_rows.len());
for (reason, n) in &by_reason {
eprintln!("[retention_sweep] {reason}: {n}");
}
if overdue_rows.is_empty() {
eprintln!("[retention_sweep] nothing to do; all subjects within retention window.");
return Ok(());
}
eprintln!("[retention_sweep] sample (first 5):");
for row in overdue_rows.iter().take(5) {
eprintln!(
" - {} status={} reason={} general_overdue_days={}",
row.candidate_id, row.status, row.reason, row.days_overdue_general
);
}
if !args.apply {
eprintln!();
eprintln!("[retention_sweep] DRY-RUN: no file written. Re-run with --apply to persist.");
return Ok(());
}
let date_stamp = now.format("%Y-%m-%d");
let report_path = args
.storage_root
.join("_catalog")
.join("subjects")
.join(format!("_retention_sweep_{date_stamp}.jsonl"));
if let Some(parent) = report_path.parent() {
std::fs::create_dir_all(parent).map_err(|e| format!("mkdir {parent:?}: {e}"))?;
}
let mut buf = String::new();
for row in &overdue_rows {
let line = serde_json::to_string(row).map_err(|e| format!("serialize row: {e}"))?;
buf.push_str(&line);
buf.push('\n');
}
std::fs::write(&report_path, &buf).map_err(|e| format!("write {report_path:?}: {e}"))?;
eprintln!("[retention_sweep] report written: {}", report_path.display());
eprintln!("[retention_sweep] action required by legal/operator. no manifests were mutated.");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::Duration;
use shared::types::{
BiometricConsent, ConsentStatus, GeneralPiiConsent, SubjectConsent, SubjectRetention,
SubjectVertical,
};
fn fixture(
id: &str,
status: SubjectStatus,
gen_until: DateTime<Utc>,
bio_until: Option<DateTime<Utc>>,
) -> SubjectManifest {
SubjectManifest {
schema: "subject_manifest.v1".into(),
candidate_id: id.into(),
created_at: Utc::now(),
updated_at: Utc::now(),
status,
vertical: SubjectVertical::General,
consent: SubjectConsent {
general_pii: GeneralPiiConsent {
status: ConsentStatus::Given,
version: "v1".into(),
given_at: None,
withdrawn_at: None,
},
biometric: BiometricConsent {
status: if bio_until.is_some() {
BiometricConsentStatus::Given
} else {
BiometricConsentStatus::NeverCollected
},
retention_until: bio_until,
},
},
retention: SubjectRetention {
general_pii_until: gen_until,
policy: "test".into(),
},
datasets: vec![],
safe_views: vec![],
audit_log_path: String::new(),
audit_log_chain_root: String::new(),
biometric_collection: None,
}
}
#[test]
fn active_subject_inside_window_not_overdue() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::Active, now + Duration::days(30), None);
assert!(is_overdue(&s, now).is_none());
}
#[test]
fn active_subject_past_general_window_flags_general() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::Active, now - Duration::days(1), None);
assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
}
#[test]
fn erased_subject_never_re_flags_even_if_far_past() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::Erased, now - Duration::days(1000), None);
assert!(is_overdue(&s, now).is_none(), "erased subjects must not re-fire");
}
#[test]
fn already_retention_expired_is_idempotent() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::RetentionExpired, now - Duration::days(1), None);
assert!(is_overdue(&s, now).is_none(), "already-flagged subjects must not re-fire");
}
#[test]
fn biometric_overdue_alone_triggers_bipa_path() {
let now = Utc::now();
let s = fixture(
"a",
SubjectStatus::Active,
now + Duration::days(30),
Some(now - Duration::days(1)),
);
assert_eq!(is_overdue(&s, now), Some(OverdueReason::Biometric));
}
#[test]
fn both_clocks_overdue_reports_both() {
let now = Utc::now();
let s = fixture(
"a",
SubjectStatus::Active,
now - Duration::days(2),
Some(now - Duration::days(1)),
);
assert_eq!(is_overdue(&s, now), Some(OverdueReason::Both));
}
#[test]
fn pending_consent_subject_can_still_expire() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::PendingConsent, now - Duration::days(1), None);
assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
}
#[test]
fn withdrawn_subject_still_subject_to_retention() {
let now = Utc::now();
let s = fixture("a", SubjectStatus::Withdrawn, now - Duration::days(1), None);
assert_eq!(is_overdue(&s, now), Some(OverdueReason::GeneralPii));
}
}
File diff suppressed because it is too large Load Diff
+3
View File
@@ -2,3 +2,6 @@ pub mod registry;
pub mod service;
pub mod grpc;
pub mod tombstones;
pub mod subject_audit;
pub mod audit_endpoint;
pub mod biometric_endpoint;
+261 -1
View File
@@ -1,6 +1,6 @@
use shared::types::{
AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile,
ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone,
ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, SubjectManifest, Tombstone,
};
use crate::tombstones::TombstoneStore;
@@ -66,16 +66,20 @@ pub struct MetadataUpdate {
const MANIFEST_PREFIX: &str = "_catalog/manifests";
const VIEW_PREFIX: &str = "_catalog/views";
const SUBJECT_PREFIX: &str = "_catalog/subjects";
const PROFILE_PREFIX: &str = "_catalog/profiles";
/// In-memory dataset registry backed by manifest persistence in object storage.
/// Also tracks AiViews (Phase D) — safe projections over base datasets.
/// And tombstones (Phase E) — soft-delete markers applied at query time.
/// And subject manifests (compliance/access metadata layer) — see
/// docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
#[derive(Clone)]
pub struct Registry {
datasets: Arc<RwLock<HashMap<DatasetId, DatasetManifest>>>,
views: Arc<RwLock<HashMap<String, AiView>>>,
profiles: Arc<RwLock<HashMap<String, ModelProfile>>>,
subjects: Arc<RwLock<HashMap<String, SubjectManifest>>>,
tombstones: TombstoneStore,
store: Arc<dyn ObjectStore>,
}
@@ -86,6 +90,7 @@ impl Registry {
datasets: Arc::new(RwLock::new(HashMap::new())),
views: Arc::new(RwLock::new(HashMap::new())),
profiles: Arc::new(RwLock::new(HashMap::new())),
subjects: Arc::new(RwLock::new(HashMap::new())),
tombstones: TombstoneStore::new(store.clone()),
store,
}
@@ -179,6 +184,47 @@ impl Registry {
tracing::info!("catalog: {} model profiles loaded", profiles.len());
}
// Subject manifests (compliance/access metadata layer).
let subject_keys = ops::list(&self.store, Some(SUBJECT_PREFIX)).await.unwrap_or_default();
let mut subjects = self.subjects.write().await;
subjects.clear();
for key in &subject_keys {
if !key.ends_with(".json") { continue; }
let data = match ops::get(&self.store, key).await {
Ok(d) => d,
Err(e) => {
tracing::warn!("subject '{key}': read failed: {e}");
continue;
}
};
match serde_json::from_slice::<SubjectManifest>(&data) {
// Match put_subject: in-memory key is the SANITIZED form
// so get_subject lookups match the storage key shape.
Ok(subj) => {
let map_key = sanitize_view_name(&subj.candidate_id);
// Rebuild collision detection — symmetric with the
// put_subject collision guard. If pre-existing storage
// somehow contains two manifests whose raw ids
// sanitize to the same key, log loudly so operators
// know one was dropped instead of silently losing it.
// (2026-05-03 opus scrum WARN registry.rs:201.)
if let Some(existing) = subjects.get(&map_key) {
if existing.candidate_id != subj.candidate_id {
tracing::warn!(
"subject rebuild collision on sanitized key '{map_key}': raw '{}' was loaded earlier; raw '{}' is being silently overwritten in memory. Run dedupe + delete the duplicate manifest file at storage.",
existing.candidate_id, subj.candidate_id,
);
}
}
subjects.insert(map_key, subj);
}
Err(e) => tracing::warn!("subject '{key}': parse failed: {e}"),
}
}
if !subjects.is_empty() {
tracing::info!("catalog: {} subject manifests loaded", subjects.len());
}
Ok(count)
}
@@ -686,6 +732,94 @@ impl Registry {
Ok(())
}
// --- Subject manifests (compliance/access metadata layer) ---
// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
// Mirrors the put_view / get_view / list_views / delete_view shape.
// The audit log per subject is owned separately by SubjectAuditWriter
// (Step 2) — this layer just persists the manifest itself.
/// Create or replace a subject manifest. Validates that referenced
/// datasets exist in the catalog (fail-fast so dangling references
/// don't accumulate). Persists to `_catalog/subjects/<id>.json`.
///
/// **Storage-key collision guard** (2026-05-03 opus scrum WARN
/// at registry.rs:735): the storage key is `sanitize_view_name(id)`
/// — `/`, `:`, etc. become `_`. A naive in-memory keying by raw
/// candidate_id would let two distinct ids (`CAND/1` + `CAND_1`)
/// collide on disk while appearing distinct in memory; the second
/// put silently overwrites the first, and `rebuild` only loads one.
/// Both the in-memory map AND the storage key now use the sanitized
/// form. Caller's raw candidate_id is preserved on the manifest for
/// audit-trail integrity (the manifest body has the original).
/// We surface a clear error if two raw ids would collide on the
/// sanitized key.
pub async fn put_subject(&self, mut subj: SubjectManifest) -> Result<SubjectManifest, String> {
if subj.candidate_id.is_empty() {
return Err("subject candidate_id is empty".into());
}
for ds in &subj.datasets {
if self.get_by_name(&ds.name).await.is_none() {
return Err(format!(
"subject '{}' references dataset '{}' which is not in the catalog",
subj.candidate_id, ds.name,
));
}
}
let now = chrono::Utc::now();
if subj.created_at.timestamp() == 0 {
subj.created_at = now;
}
subj.updated_at = now;
let map_key = sanitize_view_name(&subj.candidate_id);
let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
// Collision guard: if the sanitized key already exists for a
// DIFFERENT raw candidate_id, refuse the put. Better to fail
// loudly than silently overwrite an existing subject.
{
let subjects = self.subjects.read().await;
if let Some(existing) = subjects.get(&map_key) {
if existing.candidate_id != subj.candidate_id {
return Err(format!(
"subject id '{}' collides with existing subject '{}' on sanitized storage key '{}'",
subj.candidate_id, existing.candidate_id, map_key,
));
}
}
}
let json = serde_json::to_vec_pretty(&subj).map_err(|e| e.to_string())?;
ops::put(&self.store, &key, json.into()).await?;
let mut subjects = self.subjects.write().await;
subjects.insert(map_key, subj.clone());
tracing::debug!("subject manifest persisted: {}", subj.candidate_id);
Ok(subj)
}
pub async fn get_subject(&self, candidate_id: &str) -> Option<SubjectManifest> {
self.subjects.read().await.get(&sanitize_view_name(candidate_id)).cloned()
}
pub async fn list_subjects(&self) -> Vec<SubjectManifest> {
self.subjects.read().await.values().cloned().collect()
}
pub async fn delete_subject(&self, candidate_id: &str) -> Result<(), String> {
let map_key = sanitize_view_name(candidate_id);
let key = format!("{SUBJECT_PREFIX}/{}.json", map_key);
ops::delete(&self.store, &key).await?;
self.subjects.write().await.remove(&map_key);
Ok(())
}
/// Subject count — convenience for the daily retention sweep + audit
/// dashboards that don't need the full list.
pub async fn subjects_count(&self) -> usize {
self.subjects.read().await.len()
}
/// Remove a dataset from the catalog by name. Deletes the manifest
/// from both the in-memory registry and object storage.
///
@@ -1027,4 +1161,130 @@ mod tests {
assert!(reg.get(&loser_a).await.is_none());
assert!(reg.get(&loser_b).await.is_none());
}
// --- Subject manifests (Step 1 of subject_manifests_on_catalogd) ---
fn fixture_subject(id: &str) -> shared::types::SubjectManifest {
use shared::types::{
BiometricConsent, BiometricConsentStatus, ConsentStatus,
GeneralPiiConsent, SubjectConsent, SubjectManifest, SubjectRetention,
SubjectStatus, SubjectVertical,
};
SubjectManifest {
schema: "subject_manifest.v1".into(),
candidate_id: id.to_string(),
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
status: SubjectStatus::Active,
vertical: SubjectVertical::Unknown,
consent: SubjectConsent {
general_pii: GeneralPiiConsent {
status: ConsentStatus::PendingBackfillReview,
version: String::new(),
given_at: None,
withdrawn_at: None,
},
biometric: BiometricConsent {
status: BiometricConsentStatus::NeverCollected,
retention_until: None,
},
},
retention: SubjectRetention {
general_pii_until: chrono::Utc::now() + chrono::Duration::days(365 * 4),
policy: "4_year_default".into(),
},
datasets: vec![],
safe_views: vec![],
audit_log_path: String::new(),
audit_log_chain_root: String::new(),
biometric_collection: None,
}
}
#[tokio::test]
async fn put_subject_with_no_dataset_refs_succeeds() {
let reg = fixture();
let s = fixture_subject("CAND-000001");
let stored = reg.put_subject(s).await.unwrap();
assert_eq!(stored.candidate_id, "CAND-000001");
assert_eq!(reg.subjects_count().await, 1);
}
#[tokio::test]
async fn put_subject_rejects_dangling_dataset_ref() {
use shared::types::SubjectDatasetRef;
let reg = fixture();
let mut s = fixture_subject("CAND-000002");
s.datasets.push(SubjectDatasetRef {
name: "no_such_dataset".into(),
key_column: "candidate_id".into(),
key_value: "CAND-000002".into(),
});
let err = reg.put_subject(s).await.unwrap_err();
assert!(err.contains("not in the catalog"), "got: {err}");
}
#[tokio::test]
async fn put_subject_with_valid_dataset_ref_succeeds() {
use shared::types::SubjectDatasetRef;
let reg = fixture();
// Register a dataset first so the subject's reference resolves.
reg.register("workers".into(), fp("w1"), vec![obj("w.parquet")]).await.unwrap();
let mut s = fixture_subject("CAND-000003");
s.datasets.push(SubjectDatasetRef {
name: "workers".into(),
key_column: "candidate_id".into(),
key_value: "CAND-000003".into(),
});
let stored = reg.put_subject(s).await.unwrap();
assert_eq!(stored.datasets.len(), 1);
assert_eq!(stored.datasets[0].name, "workers");
}
#[tokio::test]
async fn subject_round_trips_through_object_store() {
let store = Arc::new(object_store::memory::InMemory::new());
let reg = Registry::new(store.clone());
reg.put_subject(fixture_subject("CAND-RT-001")).await.unwrap();
// Open a fresh registry over the same store and load → confirms
// persistence + load_all path actually round-trips the manifest.
let reg2 = Registry::new(store);
reg2.rebuild().await.unwrap();
let loaded = reg2.get_subject("CAND-RT-001").await;
assert!(loaded.is_some(), "subject not loaded from store");
assert_eq!(loaded.unwrap().status, shared::types::SubjectStatus::Active);
}
#[tokio::test]
async fn put_subject_rejects_sanitize_collision() {
// 2026-05-03 opus scrum WARN at registry.rs:735 regression:
// CAND/1 and CAND_1 sanitize to the same storage key. Without
// the collision guard, the second put silently overwrites the
// first on disk; with it, the second put fails loudly.
let reg = fixture();
let s1 = fixture_subject("CAND/1");
reg.put_subject(s1).await.unwrap();
let s2 = fixture_subject("CAND_1"); // sanitizes to same key
let err = reg.put_subject(s2).await.unwrap_err();
assert!(err.contains("collides"), "got: {err}");
assert_eq!(reg.subjects_count().await, 1, "first subject must survive");
}
#[tokio::test]
async fn delete_subject_removes_in_memory_and_persistence() {
let store = Arc::new(object_store::memory::InMemory::new());
let reg = Registry::new(store.clone());
reg.put_subject(fixture_subject("CAND-DEL-001")).await.unwrap();
assert_eq!(reg.subjects_count().await, 1);
reg.delete_subject("CAND-DEL-001").await.unwrap();
assert_eq!(reg.subjects_count().await, 0);
// Fresh registry over same store — confirm it's also gone from persistence.
let reg2 = Registry::new(store);
reg2.rebuild().await.unwrap();
assert_eq!(reg2.subjects_count().await, 0);
}
}
+587
View File
@@ -0,0 +1,587 @@
//! Subject audit log writer with HMAC chain.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §3.
//!
//! Per-subject append-only JSONL at `_catalog/subjects/<id>.audit.jsonl`.
//! Each row's `row_hmac` is HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac).
//! The chain root is mirrored to the SubjectManifest.audit_log_chain_root
//! field after each append, so verifiers can find the latest valid root
//! from the manifest alone.
//!
//! Failures are non-blocking — audit-write errors are logged but do NOT
//! propagate to the caller (per spec §3.2: "better to leak a row than
//! block legitimate operations"). Operators MUST monitor the error count.
//!
//! Key loading: from a sealed file at the path the writer is constructed
//! with. Tests use `with_inline_key` to inject a deterministic key.
use bytes::Bytes;
use hmac::{Hmac, Mac};
use object_store::ObjectStore;
use sha2::Sha256;
use shared::types::SubjectAuditRow;
use std::collections::HashMap;
use std::sync::Arc;
use storaged::ops;
use tokio::sync::Mutex;
const SUBJECT_PREFIX: &str = "_catalog/subjects";
pub const GENESIS_HASH: &str = "GENESIS";
type HmacSha256 = Hmac<Sha256>;
/// Render a JSON value canonically: object keys sorted alphabetically,
/// no insignificant whitespace, no trailing newline. Recursive — nested
/// objects also sort their keys. Arrays preserve order (semantically
/// significant). Numbers, strings, booleans, nulls pass through.
///
/// This is intentionally narrower than RFC 8785 (no number normalization,
/// no UTF-8 NFC re-encoding). For our use — hashing structurally simple
/// audit-row JSON — alphabetical key order is the only canonicalization
/// that matters: it makes the byte sequence stable across struct field
/// reordering and additive schema evolution.
fn canonical_json(v: &serde_json::Value) -> Result<Vec<u8>, String> {
use std::collections::BTreeMap;
fn rewrite(v: &serde_json::Value) -> serde_json::Value {
match v {
serde_json::Value::Object(map) => {
let sorted: BTreeMap<String, serde_json::Value> = map
.iter()
.map(|(k, v)| (k.clone(), rewrite(v)))
.collect();
serde_json::Value::Object(sorted.into_iter().collect())
}
serde_json::Value::Array(arr) => {
serde_json::Value::Array(arr.iter().map(rewrite).collect())
}
other => other.clone(),
}
}
let canonical = rewrite(v);
serde_json::to_vec(&canonical).map_err(|e| e.to_string())
}
/// Per-subject audit log writer. Holds the signing key, a small
/// in-memory cache of latest chain hash per subject, and per-subject
/// mutexes that serialize concurrent appends to the same subject's log.
///
/// **No Debug impl deliberately** — auto-deriving Debug on this struct
/// would risk leaking the signing key into log lines. Tests that need
/// `Result::unwrap_err` on a function returning `Result<Self, _>` must
/// match on the result instead of using `.unwrap_err()`.
///
/// Concurrency: appends to DIFFERENT subjects run in parallel; appends
/// to the SAME subject serialize via a per-candidate Mutex. Without
/// per-subject locking the read-modify-write append_line race
/// (concurrent reads of the same `prev_hash` + concurrent writes
/// overwriting each other) silently corrupts the chain. Caught by
/// 2026-05-03 opus scrum BLOCK on subject_audit.rs:172.
#[derive(Clone)]
pub struct SubjectAuditWriter {
store: Arc<dyn ObjectStore>,
/// HMAC signing key. Loaded once at construction. Never logged.
/// `Vec<u8>` not `String` — keys are bytes, not text.
key: Arc<Vec<u8>>,
/// In-memory cache of the latest chain hash per candidate_id.
/// Loaded lazily from the audit JSONL on first append per subject.
latest_hash: Arc<Mutex<HashMap<String, String>>>,
/// Per-subject mutexes. Append acquires the subject's lock before
/// the read-modify-write to serialize concurrent appends to the
/// same subject. Different subjects don't contend.
subject_locks: Arc<Mutex<HashMap<String, Arc<Mutex<()>>>>>,
}
impl SubjectAuditWriter {
/// Construct a writer that loads its signing key from a file at
/// `key_path`. The file MUST be mode 0400 owner-only on production
/// hosts (operator's responsibility — not enforced here because the
/// permission check would be a footgun in tests/dev).
pub async fn from_key_file(
store: Arc<dyn ObjectStore>,
key_path: &std::path::Path,
) -> Result<Self, String> {
let key = tokio::fs::read(key_path)
.await
.map_err(|e| format!("read signing key from {}: {e}", key_path.display()))?;
if key.len() < 32 {
return Err(format!(
"signing key is {} bytes; recommend ≥32 bytes for HMAC-SHA256",
key.len()
));
}
Ok(Self {
store,
key: Arc::new(key),
latest_hash: Arc::new(Mutex::new(HashMap::new())),
subject_locks: Arc::new(Mutex::new(HashMap::new())),
})
}
/// Construct a writer with an inline key. Intended for tests + bring-up.
/// Production should use `from_key_file`.
pub fn with_inline_key(store: Arc<dyn ObjectStore>, key: Vec<u8>) -> Self {
Self {
store,
key: Arc::new(key),
latest_hash: Arc::new(Mutex::new(HashMap::new())),
subject_locks: Arc::new(Mutex::new(HashMap::new())),
}
}
fn audit_key(candidate_id: &str) -> String {
let safe: String = candidate_id
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' {
c
} else {
'_'
}
})
.collect();
format!("{SUBJECT_PREFIX}/{}.audit.jsonl", safe)
}
/// Compute HMAC-SHA256(key, prev_hash_bytes || canonical_row_bytes).
/// Returns the HMAC as lowercase hex. Uses byte-level hex formatting
/// in a single pass (avoids per-byte format! allocations).
fn compute_hmac(&self, prev_hash: &str, canonical_row: &[u8]) -> String {
let mut mac = <HmacSha256 as Mac>::new_from_slice(&self.key)
.expect("HMAC accepts any key length");
mac.update(prev_hash.as_bytes());
mac.update(canonical_row);
let result = mac.finalize().into_bytes();
const HEX: &[u8; 16] = b"0123456789abcdef";
let mut s = String::with_capacity(64);
for byte in result {
s.push(HEX[(byte >> 4) as usize] as char);
s.push(HEX[(byte & 0x0f) as usize] as char);
}
s
}
/// Render an audit row as canonical JSON: keys sorted alphabetically,
/// no insignificant whitespace, row_hmac field excluded. This is the
/// byte sequence the HMAC is computed over.
///
/// **Why a manual canonicalizer instead of `serde_json::to_vec`:**
/// serde_json serializes struct fields in DECLARATION order. If the
/// SubjectAuditRow struct ever gains/loses/reorders fields (schema
/// evolution), the bytes change even though the LOGICAL content is
/// the same. A verifier compiled against the new struct shape
/// re-serializes a stored row and gets DIFFERENT bytes than the
/// original write — chain breaks silently. Caught by 2026-05-03
/// opus scrum BLOCK on subject_audit.rs:108.
///
/// Canonical-JSON (RFC 8785-ish, simplified for our scalar types)
/// fixes this by using alphabetical key order + minimal whitespace,
/// stable across struct evolution.
fn canonical_row_bytes(row: &SubjectAuditRow) -> Result<Vec<u8>, String> {
// Project to a generic Value, drop row_hmac, then re-serialize
// through a BTreeMap so keys come out alphabetical.
let mut v: serde_json::Value = serde_json::to_value(row)
.map_err(|e| format!("audit row to value: {e}"))?;
if let Some(obj) = v.as_object_mut() {
obj.remove("row_hmac");
}
canonical_json(&v).map_err(|e| format!("canonicalize: {e}"))
}
/// Get-or-create per-subject lock. Held across the read-modify-write
/// in append() so concurrent appends to the same subject serialize.
async fn lock_for(&self, candidate_id: &str) -> Arc<Mutex<()>> {
let mut locks = self.subject_locks.lock().await;
locks
.entry(candidate_id.to_string())
.or_insert_with(|| Arc::new(Mutex::new(())))
.clone()
}
/// Append one audit row for a subject. Computes the HMAC chain link,
/// writes to the per-subject JSONL, returns the new chain root (which
/// the caller MAY mirror to SubjectManifest.audit_log_chain_root).
///
/// **Concurrency**: appends to the SAME subject serialize via a
/// per-subject Mutex. Appends to DIFFERENT subjects run in parallel.
/// Without the per-subject lock, the read-modify-write in
/// append_line races and silently corrupts the chain. (2026-05-03
/// opus scrum BLOCK on subject_audit.rs:172.)
///
/// On error: logs and returns Err. Caller decides whether to propagate
/// (per spec §3.2 the gateway tool registry SHOULD log + drop).
pub async fn append(&self, mut row: SubjectAuditRow) -> Result<String, String> {
let cid = row.candidate_id.clone();
if cid.is_empty() {
return Err("audit row candidate_id is empty".into());
}
// Acquire the per-subject lock for the duration of this append.
let subject_lock = self.lock_for(&cid).await;
let _guard = subject_lock.lock().await;
// 1. Find prev_chain_hash. Cache hot path; cold path scans
// the existing JSONL tail-style to find the last row.
let prev = {
let cache = self.latest_hash.lock().await;
cache.get(&cid).cloned()
};
let prev_hash = match prev {
Some(h) => h,
None => self.scan_latest_hash(&cid).await,
};
row.prev_chain_hash = prev_hash.clone();
row.row_hmac = String::new(); // Always cleared before HMAC computation.
// 2. Canonical-JSON the row (sorted keys, no whitespace, no
// row_hmac), then HMAC. Stable across struct evolution.
let canon = Self::canonical_row_bytes(&row)?;
let new_hmac = self.compute_hmac(&prev_hash, &canon);
row.row_hmac = new_hmac.clone();
// 3. Append the row to the JSONL. The per-subject lock above
// serializes the read-modify-write inside append_line.
let line = serde_json::to_vec(&row)
.map_err(|e| format!("serialize audit row: {e}"))?;
self.append_line(&cid, &line).await?;
// 4. Update cache.
self.latest_hash.lock().await.insert(cid, new_hmac.clone());
Ok(new_hmac)
}
/// Scan the audit JSONL for a subject and return the last row's
/// row_hmac. GENESIS if no log exists or it's empty.
/// Object-store-aware: works across LocalFileSystem + InMemory + S3.
async fn scan_latest_hash(&self, candidate_id: &str) -> String {
let key = Self::audit_key(candidate_id);
let bytes = match ops::get(&self.store, &key).await {
Ok(b) => b,
Err(_) => return GENESIS_HASH.to_string(), // file doesn't exist
};
// Find the last newline-delimited line that parses as a row.
let text = match std::str::from_utf8(&bytes) {
Ok(t) => t,
Err(_) => return GENESIS_HASH.to_string(),
};
for line in text.lines().rev() {
if line.trim().is_empty() {
continue;
}
if let Ok(parsed) = serde_json::from_str::<SubjectAuditRow>(line) {
return parsed.row_hmac;
}
}
GENESIS_HASH.to_string()
}
/// Append a single newline-terminated line to the per-subject log.
/// Reads the existing content (may be empty), appends, writes back.
/// Object stores don't have native append; this is the standard
/// "read-modify-write the tail file" pattern.
async fn append_line(&self, candidate_id: &str, line: &[u8]) -> Result<(), String> {
let key = Self::audit_key(candidate_id);
let existing = ops::get(&self.store, &key).await.unwrap_or_default();
let mut buf = Vec::with_capacity(existing.len() + line.len() + 1);
buf.extend_from_slice(&existing);
if !buf.is_empty() && !buf.ends_with(b"\n") {
buf.push(b'\n');
}
buf.extend_from_slice(line);
buf.push(b'\n');
ops::put(&self.store, &key, Bytes::from(buf)).await
}
/// Read all audit rows for a subject, optionally filtered by time
/// window (inclusive bounds). Returns rows in append order. Used by
/// the /audit/subject/{id} endpoint to assemble the legal-tier
/// response. Window bounds are optional — `from=None`+`to=None`
/// returns the full log.
pub async fn read_rows_in_range(
&self,
candidate_id: &str,
from: Option<chrono::DateTime<chrono::Utc>>,
to: Option<chrono::DateTime<chrono::Utc>>,
) -> Result<Vec<SubjectAuditRow>, String> {
let key = Self::audit_key(candidate_id);
let bytes = match ops::get(&self.store, &key).await {
Ok(b) => b,
Err(_) => return Ok(Vec::new()), // no audit log = empty result
};
let text = std::str::from_utf8(&bytes)
.map_err(|e| format!("audit log not utf-8: {e}"))?;
let mut out = Vec::new();
for line in text.lines() {
if line.trim().is_empty() {
continue;
}
let row: SubjectAuditRow = match serde_json::from_str(line) {
Ok(r) => r,
Err(_) => continue, // skip unparseable lines (defensive)
};
if let Some(from) = from {
if row.ts < from { continue; }
}
if let Some(to) = to {
if row.ts > to { continue; }
}
out.push(row);
}
Ok(out)
}
/// Return the latest row_hmac in this subject's audit log, or None
/// if the log doesn't exist / has no rows. Cache-aware: hits the
/// in-memory cache when the writer has appended at least once for
/// this subject in the current process, otherwise scans the log
/// tail (single object_store read).
///
/// Used by /audit/subject/{id} to echo the actual chain tip in the
/// response. Taking the tip from a windowed slice of rows would
/// lie when the window doesn't cover the latest row.
pub async fn chain_tip(&self, candidate_id: &str) -> Option<String> {
// Cache hit?
if let Some(h) = self.latest_hash.lock().await.get(candidate_id).cloned() {
return Some(h);
}
// Cold path — scan the log tail. Returns GENESIS for empty/missing
// logs; we coerce that to None so callers can distinguish "no audit
// activity yet" from "real chain tip exists".
let tip = self.scan_latest_hash(candidate_id).await;
if tip == GENESIS_HASH { None } else { Some(tip) }
}
/// Verify the full HMAC chain for a subject. Returns Ok(rows_verified)
/// or Err with the first chain break encountered.
///
/// Special case: if no audit log exists (the subject has had no PII
/// access yet), returns Ok(0) — an empty log is trivially valid.
/// Callers that want to detect "log was deleted" should compare
/// against SubjectManifest.audit_log_chain_root: if the manifest's
/// chain root is non-empty/non-GENESIS but verify_chain returns 0,
/// the file was tampered.
pub async fn verify_chain(&self, candidate_id: &str) -> Result<usize, String> {
let key = Self::audit_key(candidate_id);
let bytes = match ops::get(&self.store, &key).await {
Ok(b) => b,
Err(_) => return Ok(0), // no log = 0 rows = trivially valid
};
let text = std::str::from_utf8(&bytes)
.map_err(|e| format!("audit log not utf-8: {e}"))?;
let mut prev = GENESIS_HASH.to_string();
let mut count = 0usize;
for (lineno, line) in text.lines().enumerate() {
if line.trim().is_empty() {
continue;
}
let mut row: SubjectAuditRow = serde_json::from_str(line)
.map_err(|e| format!("line {} parse failed: {e}", lineno + 1))?;
if row.prev_chain_hash != prev {
return Err(format!(
"chain break at line {}: prev_chain_hash={} expected={}",
lineno + 1, row.prev_chain_hash, prev,
));
}
let claimed = std::mem::take(&mut row.row_hmac);
let canon = Self::canonical_row_bytes(&row)
.map_err(|e| format!("canonicalize line {}: {e}", lineno + 1))?;
let recomputed = self.compute_hmac(&prev, &canon);
if recomputed != claimed {
return Err(format!(
"hmac mismatch at line {}: stored={} recomputed={}",
lineno + 1, claimed, recomputed,
));
}
prev = claimed;
count += 1;
}
Ok(count)
}
}
#[cfg(test)]
mod tests {
use super::*;
use object_store::memory::InMemory;
use shared::types::AuditAccessor;
fn fixture_writer() -> SubjectAuditWriter {
SubjectAuditWriter::with_inline_key(
Arc::new(InMemory::new()),
// Deterministic 32-byte key for test reproducibility.
(0u8..32).collect(),
)
}
fn fixture_row(candidate_id: &str, fields: &[&str]) -> SubjectAuditRow {
SubjectAuditRow {
schema: "subject_audit.v1".into(),
ts: chrono::Utc::now(),
candidate_id: candidate_id.into(),
accessor: AuditAccessor {
kind: "gateway_lookup".into(),
daemon: "gateway".into(),
purpose: "fill_validation".into(),
trace_id: String::new(),
},
fields_accessed: fields.iter().map(|s| s.to_string()).collect(),
result: "success".into(),
prev_chain_hash: String::new(), // filled in by writer
row_hmac: String::new(), // filled in by writer
}
}
#[tokio::test]
async fn first_append_uses_genesis_prev_hash() {
let w = fixture_writer();
let new_root = w.append(fixture_row("CAND-1", &["name"])).await.unwrap();
assert!(!new_root.is_empty());
assert_eq!(new_root.len(), 64); // SHA-256 hex
// After first append, the only row should reference GENESIS as prev.
let bytes = ops::get(&w.store, &SubjectAuditWriter::audit_key("CAND-1"))
.await.unwrap();
let text = std::str::from_utf8(&bytes).unwrap();
let row: SubjectAuditRow = serde_json::from_str(text.trim()).unwrap();
assert_eq!(row.prev_chain_hash, "GENESIS");
assert_eq!(row.row_hmac, new_root);
}
#[tokio::test]
async fn chain_links_each_append() {
let w = fixture_writer();
let h1 = w.append(fixture_row("CAND-2", &["name"])).await.unwrap();
let h2 = w.append(fixture_row("CAND-2", &["phone"])).await.unwrap();
let h3 = w.append(fixture_row("CAND-2", &["email"])).await.unwrap();
assert_ne!(h1, h2);
assert_ne!(h2, h3);
let count = w.verify_chain("CAND-2").await.unwrap();
assert_eq!(count, 3);
}
#[tokio::test]
async fn separate_subjects_have_independent_chains() {
let w = fixture_writer();
let _ = w.append(fixture_row("CAND-A", &["name"])).await.unwrap();
let _ = w.append(fixture_row("CAND-B", &["name"])).await.unwrap();
let _ = w.append(fixture_row("CAND-A", &["phone"])).await.unwrap();
assert_eq!(w.verify_chain("CAND-A").await.unwrap(), 2);
assert_eq!(w.verify_chain("CAND-B").await.unwrap(), 1);
}
#[tokio::test]
async fn tamper_detected_on_verify() {
let w = fixture_writer();
let _ = w.append(fixture_row("CAND-T", &["name"])).await.unwrap();
let _ = w.append(fixture_row("CAND-T", &["phone"])).await.unwrap();
// Mutate the file to flip a `result` field on the first row.
let key = SubjectAuditWriter::audit_key("CAND-T");
let bytes = ops::get(&w.store, &key).await.unwrap();
let mut text = std::str::from_utf8(&bytes).unwrap().to_string();
text = text.replacen("\"success\"", "\"denied\"", 1);
ops::put(&w.store, &key, text.into_bytes().into()).await.unwrap();
let err = w.verify_chain("CAND-T").await.unwrap_err();
assert!(err.contains("hmac mismatch"), "expected hmac mismatch, got: {err}");
}
#[tokio::test]
async fn cold_writer_picks_up_existing_chain() {
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
let key: Vec<u8> = (0u8..32).collect();
let w1 = SubjectAuditWriter::with_inline_key(store.clone(), key.clone());
let h1 = w1.append(fixture_row("CAND-COLD", &["name"])).await.unwrap();
// Drop w1's cache, simulate process restart.
drop(w1);
let w2 = SubjectAuditWriter::with_inline_key(store.clone(), key);
let h2 = w2.append(fixture_row("CAND-COLD", &["phone"])).await.unwrap();
assert_ne!(h1, h2);
assert_eq!(w2.verify_chain("CAND-COLD").await.unwrap(), 2);
}
#[tokio::test]
async fn empty_candidate_id_rejected() {
let w = fixture_writer();
let row = fixture_row("", &["name"]);
let err = w.append(row).await.unwrap_err();
assert!(err.contains("empty"), "got: {err}");
}
#[tokio::test]
async fn concurrent_appends_to_same_subject_serialize() {
// 2026-05-03 opus scrum BLOCK regression: without per-subject
// locking, concurrent appends raced and silently lost rows.
// This test spawns 50 parallel appends and asserts ALL 50
// land with an intact chain.
let w = Arc::new(fixture_writer());
let mut handles = Vec::new();
for i in 0..50 {
let w = w.clone();
handles.push(tokio::spawn(async move {
w.append(fixture_row("CAND-RACE", &[&format!("f{i}")])).await
}));
}
for h in handles {
h.await.unwrap().unwrap();
}
let count = w.verify_chain("CAND-RACE").await.unwrap();
assert_eq!(count, 50, "expected 50 rows; chain was corrupted");
}
#[tokio::test]
async fn concurrent_appends_to_different_subjects_run_parallel() {
let w = Arc::new(fixture_writer());
let mut handles = Vec::new();
for i in 0..20 {
let w = w.clone();
handles.push(tokio::spawn(async move {
w.append(fixture_row(&format!("CAND-PARA-{i}"), &["f"])).await
}));
}
for h in handles {
h.await.unwrap().unwrap();
}
// Each of the 20 subjects has exactly 1 row.
for i in 0..20 {
assert_eq!(w.verify_chain(&format!("CAND-PARA-{i}")).await.unwrap(), 1);
}
}
#[test]
fn canonical_json_sorts_keys_alphabetically() {
let v = serde_json::json!({"z": 1, "a": 2, "m": {"y": 1, "b": 2}});
let bytes = canonical_json(&v).unwrap();
let s = std::str::from_utf8(&bytes).unwrap();
// Keys should appear in alphabetical order at every nesting level.
let a_pos = s.find("\"a\"").unwrap();
let m_pos = s.find("\"m\"").unwrap();
let z_pos = s.find("\"z\"").unwrap();
assert!(a_pos < m_pos);
assert!(m_pos < z_pos);
// Nested object's keys also sorted.
let b_pos = s.find("\"b\"").unwrap();
let y_pos = s.find("\"y\"").unwrap();
assert!(b_pos < y_pos);
}
#[test]
fn canonical_json_arrays_preserve_order() {
let v = serde_json::json!({"k": ["c", "a", "b"]});
let bytes = canonical_json(&v).unwrap();
let s = std::str::from_utf8(&bytes).unwrap();
assert!(s.contains("\"c\",\"a\",\"b\""), "got: {s}");
}
#[tokio::test]
async fn key_too_short_rejected_via_file() {
// Write a 16-byte key file (under the 32-byte minimum).
let tmp = std::env::temp_dir().join(format!("sa_key_short_{}.bin", std::process::id()));
tokio::fs::write(&tmp, vec![0u8; 16]).await.unwrap();
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
// `unwrap_err` would require Debug on SubjectAuditWriter, which
// we deliberately don't impl (key-leak risk). Match instead.
match SubjectAuditWriter::from_key_file(store, &tmp).await {
Err(e) => assert!(e.contains("≥32 bytes"), "got: {e}"),
Ok(_) => panic!("expected too-short-key rejection"),
}
let _ = tokio::fs::remove_file(&tmp).await;
}
}
+322 -21
View File
@@ -212,6 +212,13 @@ impl ExecutionLoop {
if let Action::ToolCall { tool, args, .. } = &exec_action {
match self.dispatch_tool(tool, args).await {
Ok(result) => {
// Subject-audit hook (per Phase 1.6 SUBJECT_MANIFESTS spec
// §3.2): every tool result that returns rows referencing a
// subject must produce an audit row. Failures are logged
// but never block the tool result — the spec is explicit:
// "better to leak a row than block legitimate operations."
self.audit_subject_hits_in(&result, tool).await;
let trimmed = trim_result(&result);
self.append(LogEntry::new(
turn, "executor", &executor_model, "tool_result", trimmed,
@@ -341,6 +348,68 @@ impl ExecutionLoop {
self.log.push(e);
}
/// Subject-audit hook: scan a tool result JSON for subject identifiers,
/// fire one audit row per (subject, fields) pair via the writer in
/// V1State.subject_audit. No-op if writer is None (audit disabled).
///
/// Conventions (matches the backfill binary in catalogd/src/bin/
/// backfill_subjects.rs):
/// - "candidate_id":"X" → audit candidate_id="X"
/// - "worker_id":42 → audit candidate_id="WORKER-42"
///
/// Walks the result JSON recursively. For each object encountered
/// that contains a candidate_id or worker_id field, fires one audit
/// row capturing all the OTHER scalar field names as fields_accessed
/// (so the audit log records "this access surfaced name + phone for
/// candidate X" — sufficient for compliance review).
///
/// Failures are logged but NEVER propagated. Per spec §3.2: an audit
/// gap is preferable to blocking the tool response.
async fn audit_subject_hits_in(&self, result: &serde_json::Value, tool: &str) {
let Some(audit) = self.state.subject_audit.clone() else {
return; // audit disabled — no-op
};
// Collect (candidate_id, fields_accessed) pairs from the JSON.
let mut hits: Vec<(String, Vec<String>)> = Vec::new();
collect_subject_hits(result, &mut hits);
if hits.is_empty() {
return;
}
// Determine result-state from the payload shape, NOT from the
// transport status. A tool can return Ok(Value) where the body
// itself is `{"error":"..."}` — hardcoding "success" here would
// mislead compliance review (opus WARN at execution_loop:401).
let result_state = audit_result_state(result);
// Sequential await — NOT tokio::spawn. The previous spawn-per-row
// shape fanned out concurrent writes that raced inside the
// per-subject append. The audit writer now serializes per-subject
// internally, but spawning would still flood those locks under
// burst load with no observable latency win. Inline await is
// simpler AND correct. (2026-05-03 opus scrum BLOCKs on
// execution_loop:391 + subject_audit:172.)
for (cid, fields) in hits {
let row = shared::types::SubjectAuditRow {
schema: "subject_audit.v1".into(),
ts: chrono::Utc::now(),
candidate_id: cid.clone(),
accessor: shared::types::AuditAccessor {
kind: "gateway_lookup".into(),
daemon: "gateway".into(),
purpose: tool.to_string(),
trace_id: String::new(), // TODO: thread X-Lakehouse-Trace-Id through
},
fields_accessed: fields,
result: result_state.clone(),
prev_chain_hash: String::new(),
row_hmac: String::new(),
};
if let Err(e) = audit.append(row).await {
tracing::warn!("subject audit write failed for {cid}: {e}");
}
}
}
/// Dispatch: model name prefix → provider.
/// Local path uses Phase 21 `generate_continuable` (auto-continuation,
/// retry on empty thinking-model response). Cloud path hits
@@ -384,6 +453,8 @@ impl ExecutionLoop {
stream: Some(false),
think: Some(think),
provider: Some("ollama_cloud".into()),
tools: None,
tool_choice: None,
};
let resp = crate::v1::ollama_cloud::chat(key, &req).await
.map_err(|e| format!("ollama_cloud: {e}"))?;
@@ -605,56 +676,54 @@ impl ExecutionLoop {
/// cheapest token. Frequency is low so the Zen pay-per-token cost
/// stays bounded.
async fn escalate_to_overseer(&mut self, turn: u32, reason: &str) -> Result<(), String> {
let Some(opencode_key) = self.state.opencode_key.clone() else {
return Err("OPENCODE_API_KEY not configured — skipping escalation".into());
};
// 2026-05-03: REVERTED to local-only per PRD line 70. Cloud
// overseer (opencode/claude-opus-4-7) was a recent addition that
// moved a hot-path call OFF the local Ollama runtime onto a paid
// cloud provider. Reverted to local Ollama (qwen3.5:latest).
// Cloud overseer can be re-enabled by setting LH_OVERSEER_CLOUD=1
// for development; production stays local.
let kb = KbContext::load_for(&sig_hash(&self.req), &self.req.task_class).await;
let prompt = build_overseer_prompt(&self.req, &kb, &self.log, reason);
let started = std::time::Instant::now();
let start_time = chrono::Utc::now();
let chat_req = crate::v1::ChatRequest {
model: "claude-opus-4-7".to_string(),
model: "qwen3.5:latest".to_string(),
messages: vec![crate::v1::Message::new_text("user", prompt.clone())],
temperature: Some(0.1),
max_tokens: None,
stream: Some(false),
// Anthropic models on opencode reject `think` (handled in
// the adapter), but we keep the intent flag for parity.
think: Some(true),
provider: Some("opencode".into()),
think: Some(false),
provider: Some("ollama".into()),
tools: None,
tool_choice: None,
};
let resp = crate::v1::opencode::chat(&opencode_key, &chat_req).await
.map_err(|e| format!("opencode: {e}"))?;
let resp = crate::v1::ollama::chat(&self.state.ai_client, &chat_req).await
.map_err(|e| format!("ollama overseer: {e}"))?;
let latency_ms = started.elapsed().as_millis() as u64;
let end_time = chrono::Utc::now();
let correction_text: String = resp.choices.into_iter().next()
.map(|c| c.message.text()).unwrap_or_default();
// Stamp per-task stats — cloud call counts against the same
// usage counter so `/v1/usage` shows cloud token spend too.
self.stats.requests = self.stats.requests.saturating_add(1);
self.stats.prompt_tokens = self.stats.prompt_tokens.saturating_add(resp.usage.prompt_tokens as u64);
self.stats.completion_tokens = self.stats.completion_tokens.saturating_add(resp.usage.completion_tokens as u64);
self.stats.total_tokens = self.stats.total_tokens.saturating_add(resp.usage.total_tokens as u64);
self.stats.latency_ms = self.stats.latency_ms.saturating_add(latency_ms);
// Langfuse trace for the overseer call (same pipe that feeds
// the observer/KB, so this correction's cost lands in the KB
// too — closing the loop).
// Langfuse trace for the overseer call (local-only now).
if let Some(lf) = &self.state.langfuse {
use crate::v1::langfuse_trace::ChatTrace;
lf.emit_chat(ChatTrace {
provider: "opencode".into(),
model: "claude-opus-4-7".into(),
provider: "ollama".into(),
model: "qwen3.5:latest".into(),
input: vec![crate::v1::Message::new_text("user", prompt.clone())],
output: correction_text.clone(),
prompt_tokens: resp.usage.prompt_tokens,
completion_tokens: resp.usage.completion_tokens,
temperature: Some(0.1),
max_tokens: None,
think: Some(true),
think: Some(false),
start_time: start_time.to_rfc3339(),
end_time: end_time.to_rfc3339(),
latency_ms,
@@ -664,7 +733,7 @@ impl ExecutionLoop {
// Append to the transcript so the next executor turn sees it.
self.append(LogEntry::new(
turn, "system", "claude-opus-4-7", "overseer_correction",
turn, "system", "qwen3.5:latest", "overseer_correction",
serde_json::json!({
"reason": reason,
"correction": correction_text,
@@ -686,7 +755,7 @@ impl ExecutionLoop {
"task_class": self.req.task_class,
"operation": self.req.operation,
"reason": reason,
"model": "claude-opus-4-7",
"model": "qwen3.5:latest",
"correction": correction_text,
"applied_at_turn": turn,
"kb_context_used": kb,
@@ -988,6 +1057,119 @@ async fn append_outcomes_row_at(
/// PORT FROM orchestrator.ts:306-311. Cap `rows` at 20 entries and
/// annotate the truncation so the executor sees it on the next turn
/// Inspect a tool result JSON payload and return the appropriate
/// audit `result` field per spec §3.2 enumeration:
/// "success" | "denied" | "not_found" | "error".
///
/// Heuristic on the top-level object:
/// - body has `"error"` key (truthy) → "error"
/// - body has `"denied"` key (truthy) → "denied"
/// - body has `"not_found"` key (truthy) → "not_found"
/// - body has `"status":"denied"`/etc → that status
/// - otherwise → "success"
///
/// Conservative — only flags explicit signals, defaults to success.
/// Keeps audit log honest about ambiguous payloads (an error inside
/// rows[*] doesn't downgrade the whole call to "error" — that's a
/// row-level concern, not a result-state concern).
pub(crate) fn audit_result_state(v: &serde_json::Value) -> String {
if let Some(obj) = v.as_object() {
// Tighter than before (2026-05-03 opus scrum INFO at
// execution_loop:1085): only flag error/denied/not_found when
// the payload doesn't ALSO have a rows/data/results sibling.
// A successful tool whose response includes an "error" field
// describing one row's problem (out of many) shouldn't taint
// the whole call as "error" — that's a row-level concern, not
// a result-state concern.
let has_data_sibling = obj.contains_key("rows")
|| obj.contains_key("data")
|| obj.contains_key("results");
if !has_data_sibling {
if obj.get("error").map(is_truthy).unwrap_or(false) {
return "error".into();
}
if obj.get("denied").map(is_truthy).unwrap_or(false) {
return "denied".into();
}
if obj.get("not_found").map(is_truthy).unwrap_or(false) {
return "not_found".into();
}
}
// status is always authoritative — if a tool explicitly says
// status=denied that's not a row-level concern.
if let Some(s) = obj.get("status").and_then(|v| v.as_str()) {
match s {
"denied" | "not_found" | "error" => return s.into(),
_ => {}
}
}
}
"success".into()
}
fn is_truthy(v: &serde_json::Value) -> bool {
match v {
serde_json::Value::Null => false,
serde_json::Value::Bool(b) => *b,
serde_json::Value::String(s) => !s.is_empty(),
serde_json::Value::Array(a) => !a.is_empty(),
serde_json::Value::Object(o) => !o.is_empty(),
serde_json::Value::Number(n) => n.as_f64().map(|f| f != 0.0).unwrap_or(false),
}
}
/// Walk a tool-result JSON value recursively. For each object that
/// contains a candidate_id (string) or worker_id (int/string), append
/// (canonical_candidate_id, fields_in_object) to `out`. Conventions
/// match the backfill binary: bare candidate_id stays as-is; worker_id
/// gets prefixed with "WORKER-".
///
/// fields_in_object captures the OTHER scalar field names found in the
/// same object — so the audit row records "this access surfaced
/// {name, phone} for candidate X". The id field itself is NOT included
/// in fields_accessed (it's the subject identifier, not PII data).
pub(crate) fn collect_subject_hits(
v: &serde_json::Value,
out: &mut Vec<(String, Vec<String>)>,
) {
match v {
serde_json::Value::Object(map) => {
// Detect a subject identifier in this object's direct fields.
let mut subject_id: Option<String> = None;
if let Some(cid) = map.get("candidate_id").and_then(|v| v.as_str()) {
if !cid.is_empty() { subject_id = Some(cid.to_string()); }
} else if let Some(wid) = map.get("worker_id") {
if let Some(s) = wid.as_str() {
if !s.is_empty() { subject_id = Some(format!("WORKER-{s}")); }
} else if let Some(n) = wid.as_i64() {
subject_id = Some(format!("WORKER-{n}"));
} else if let Some(n) = wid.as_u64() {
subject_id = Some(format!("WORKER-{n}"));
}
}
if let Some(cid) = subject_id {
let fields: Vec<String> = map
.keys()
.filter(|k| *k != "candidate_id" && *k != "worker_id")
.cloned()
.collect();
out.push((cid, fields));
}
// Recurse — nested objects may carry additional subjects
// (e.g. join results with both candidate + recruiter rows).
for v in map.values() {
collect_subject_hits(v, out);
}
}
serde_json::Value::Array(arr) => {
for v in arr {
collect_subject_hits(v, out);
}
}
_ => {}
}
}
/// prompt — prevents a 1000-row hybrid_search result from wiping the
/// context budget on a single tool call.
fn trim_result(r: &serde_json::Value) -> serde_json::Value {
@@ -1866,4 +2048,123 @@ mod tests {
let p = build_reviewer_prompt(&req, &log);
assert!(!p.contains("HARD RULE"));
}
// --- Subject-audit hook tests ---
#[test]
fn collect_subject_hits_finds_candidate_id_strings() {
let v = serde_json::json!({
"rows": [
{"candidate_id": "CAND-001", "name": "x", "phone": "y"},
{"candidate_id": "CAND-002", "name": "z"},
]
});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 2);
assert_eq!(hits[0].0, "CAND-001");
// Field order from JSON object is preserved in serde_json::Map.
assert!(hits[0].1.contains(&"name".to_string()));
assert!(hits[0].1.contains(&"phone".to_string()));
assert!(!hits[0].1.contains(&"candidate_id".to_string()));
assert_eq!(hits[1].0, "CAND-002");
}
#[test]
fn collect_subject_hits_prefixes_worker_id_int() {
let v = serde_json::json!({"rows": [{"worker_id": 42, "name": "Alice"}]});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].0, "WORKER-42");
assert_eq!(hits[0].1, vec!["name".to_string()]);
}
#[test]
fn collect_subject_hits_handles_worker_id_string() {
let v = serde_json::json!({"worker_id": "42", "name": "Bob"});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].0, "WORKER-42");
}
#[test]
fn collect_subject_hits_recurses_through_nested_objects() {
let v = serde_json::json!({
"outer": {
"candidate_id": "CAND-A",
"linked": {
"worker_id": 7,
"role": "welder"
}
}
});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 2);
let ids: Vec<&str> = hits.iter().map(|(c, _)| c.as_str()).collect();
assert!(ids.contains(&"CAND-A"));
assert!(ids.contains(&"WORKER-7"));
}
#[test]
fn collect_subject_hits_ignores_objects_without_id_fields() {
let v = serde_json::json!({"rows": [{"name": "x"}, {"role": "y"}]});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 0);
}
#[test]
fn collect_subject_hits_skips_empty_id_strings() {
let v = serde_json::json!({"rows": [{"candidate_id": "", "name": "x"}]});
let mut hits = Vec::new();
collect_subject_hits(&v, &mut hits);
assert_eq!(hits.len(), 0);
}
#[test]
fn audit_result_state_default_is_success() {
let v = serde_json::json!({"rows": [{"candidate_id": "X"}]});
assert_eq!(audit_result_state(&v), "success");
}
#[test]
fn audit_result_state_detects_error_payload() {
let v = serde_json::json!({"error": "not found"});
assert_eq!(audit_result_state(&v), "error");
}
#[test]
fn audit_result_state_detects_status_field() {
let v = serde_json::json!({"status": "denied", "rows": []});
assert_eq!(audit_result_state(&v), "denied");
}
#[test]
fn audit_result_state_ignores_falsy_signals() {
// Empty/null/false should not trigger error state.
let v = serde_json::json!({"error": null, "denied": false, "rows": [{"candidate_id":"X"}]});
assert_eq!(audit_result_state(&v), "success");
}
#[test]
fn audit_result_state_does_not_classify_error_when_data_sibling_present() {
// 2026-05-03 opus scrum INFO at execution_loop:1085 regression:
// a tool that returns rows BUT also has an "error" field describing
// one bad row shouldn't taint the whole call as "error".
let v = serde_json::json!({
"rows": [{"candidate_id": "OK"}, {"candidate_id": "BAD"}],
"error": "1 row had a malformed certification"
});
assert_eq!(audit_result_state(&v), "success");
}
#[test]
fn audit_result_state_status_is_authoritative_even_with_data() {
// Explicit status="denied" wins over data-sibling rule.
let v = serde_json::json!({"status": "denied", "rows": []});
assert_eq!(audit_result_state(&v), "denied");
}
}
+89 -2
View File
@@ -255,6 +255,33 @@ async fn main() {
// surface over the existing aibridge → Ollama path. Future
// phases add provider adapters (39), routing engine (40),
// session state (41), etc. All without changing this mount.
;
// Build subject-audit writer BEFORE the V1State literal so the
// validate_workers wrapper can reference it. None = audit disabled
// (key file missing or unreadable); the gateway still serves PII
// paths but produces no audit trail per spec §3.2.
let subject_audit_writer: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>> = {
let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY")
.unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into());
match catalogd::subject_audit::SubjectAuditWriter::from_key_file(
store.clone(), std::path::Path::new(&key_path)
).await {
Ok(w) => {
tracing::info!("v1: subject audit log enabled (key: {})", key_path);
Some(std::sync::Arc::new(w))
}
Err(e) => {
tracing::warn!(
"v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)",
e
);
None
}
}
};
app = app
.nest("/v1", v1::router(v1::V1State {
ai_client: ai_client.clone(),
usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
@@ -334,7 +361,7 @@ async fn main() {
let path_str = std::env::var("LH_WORKERS_PARQUET")
.unwrap_or_else(|_| "/home/profit/lakehouse/data/datasets/workers_500k.parquet".into());
let path = std::path::Path::new(&path_str);
if path.exists() {
let raw: std::sync::Arc<dyn validator::WorkerLookup> = if path.exists() {
match validator::staffing::parquet_lookup::load_workers_parquet(path) {
Ok(lookup) => {
tracing::info!("v1: workers parquet loaded from {} — /v1/validate worker-existence checks enabled", path_str);
@@ -348,7 +375,16 @@ async fn main() {
} else {
tracing::warn!("v1: workers parquet at {} not found — /v1/validate worker-existence checks will fail Consistency", path_str);
std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
}
};
// Wrap with subject-audit decorator. When subject_audit_writer
// is None, the wrapper is a transparent passthrough — no
// audit, no overhead. When Some, every find() call fires an
// audit row via the per-subject HMAC chain. (Step 5 of
// SUBJECT_MANIFESTS_ON_CATALOGD spec.)
std::sync::Arc::new(v1::auditing_worker_lookup::AuditingWorkerLookup::new(
raw,
subject_audit_writer.clone(),
))
},
// Phase 40 early deliverable — Langfuse trace emitter.
// Defaults match mcp-server/tracing.ts conventions so
@@ -378,8 +414,59 @@ async fn main() {
}
s
},
// subject_audit was constructed BEFORE the V1State literal
// (see `subject_audit_writer` above) so validate_workers
// could wrap with it. Reuse the same Arc here.
subject_audit: subject_audit_writer.clone(),
}));
// Step 6 of SUBJECT_MANIFESTS_ON_CATALOGD spec — /audit/subject/{id}
// legal-tier endpoint. Only mounted when BOTH the audit writer AND
// the legal token file are present. Disabled-by-default keeps the
// audit surface from accidentally serving without proper credentials
// in dev / bring-up environments.
if let Some(writer) = subject_audit_writer.clone() {
let legal_token_path = std::env::var("LH_LEGAL_AUDIT_TOKEN_FILE")
.unwrap_or_else(|_| "/etc/lakehouse/legal_audit.token".into());
let audit_state = catalogd::audit_endpoint::AuditEndpointState::new(
registry.clone(),
writer.clone(),
std::path::Path::new(&legal_token_path),
).await;
app = app.nest("/audit", catalogd::audit_endpoint::router(audit_state));
tracing::info!("audit endpoint mounted at /audit (legal token: {})", legal_token_path);
// Phase 1.6 Gate 3 — biometric photo upload endpoint. Same
// legal-token gate as /audit (writes to legal-tier storage are
// privileged). Storage root defaults to ./data/biometric/uploads
// unless overridden via LH_BIOMETRIC_STORAGE_ROOT (operators on
// multi-host deployments will want a dedicated mount point).
let biometric_storage_root: std::path::PathBuf = std::env::var("LH_BIOMETRIC_STORAGE_ROOT")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| std::path::PathBuf::from("./data/biometric/uploads"));
// consent_versions allowlist — counsel-tier deployment safety.
// Defaults to /etc/lakehouse/consent_versions.json; override
// via LH_CONSENT_VERSIONS_FILE. Missing file → permissive
// mode (any non-empty hash accepted, the v1 behavior).
let consent_versions_path: std::path::PathBuf = std::env::var("LH_CONSENT_VERSIONS_FILE")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| std::path::PathBuf::from("/etc/lakehouse/consent_versions.json"));
let biometric_state = catalogd::biometric_endpoint::BiometricEndpointState::new(
registry.clone(),
writer,
std::path::Path::new(&legal_token_path),
biometric_storage_root.clone(),
).await
.with_consent_versions(&consent_versions_path).await;
app = app.nest("/biometric", catalogd::biometric_endpoint::router(biometric_state));
tracing::info!(
"biometric endpoint mounted at /biometric (storage_root: {}, legal token: {}, consent_versions file: {})",
biometric_storage_root.display(), legal_token_path, consent_versions_path.display()
);
} else {
tracing::warn!("/audit + /biometric endpoints NOT mounted — subject_audit writer is None (no signing key)");
}
// Auth middleware (if enabled) — P5-001 fix 2026-04-23:
// previously only inserted the ApiKey as an extension and never layered
// the middleware, so auth.enabled=true enforced nothing. Now wraps the
@@ -0,0 +1,194 @@
//! Validator WorkerLookup decorator that fires per-find audit rows.
//!
//! Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5.
//!
//! Wraps an inner `dyn WorkerLookup`. Every `find(candidate_id)` call
//! produces one `SubjectAuditRow` via the bundled `SubjectAuditWriter`,
//! tagged with `purpose="validator_worker_lookup"`. Audit-write failures
//! are logged only — never propagated to the caller (per spec §3.2:
//! audit-side gaps are preferable to blocking validation).
//!
//! **Why a decorator and not a trait extension**: `WorkerLookup` is a
//! sync trait by design (validator's contract is "in-memory snapshot,
//! no per-call I/O"). Audit writes are async. The decorator bridges
//! them by spawning the audit append onto the current Tokio runtime
//! handle. Per-subject sequencing inside the writer ensures the chain
//! stays intact even with spawned writes (regression-tested at
//! catalogd::subject_audit::tests::concurrent_appends_to_same_subject_serialize).
use catalogd::subject_audit::SubjectAuditWriter;
use shared::types::{AuditAccessor, SubjectAuditRow};
use std::sync::Arc;
use validator::{WorkerLookup, WorkerRecord};
pub struct AuditingWorkerLookup {
inner: Arc<dyn WorkerLookup>,
/// None = audit disabled. find() still runs; just no audit row.
audit: Option<Arc<SubjectAuditWriter>>,
/// Cached Tokio handle so the sync find() path can spawn audit
/// writes onto the right runtime. None at construction = audit
/// is no-op (no runtime context to dispatch into).
runtime_handle: Option<tokio::runtime::Handle>,
}
impl AuditingWorkerLookup {
/// Wrap an inner lookup with optional audit. If `audit` is None,
/// behavior is identical to the inner — no overhead, no audit.
/// MUST be constructed inside a Tokio runtime context (uses
/// `Handle::try_current` to capture the handle).
pub fn new(inner: Arc<dyn WorkerLookup>, audit: Option<Arc<SubjectAuditWriter>>) -> Self {
let runtime_handle = if audit.is_some() {
tokio::runtime::Handle::try_current().ok()
} else {
None
};
if audit.is_some() && runtime_handle.is_none() {
tracing::warn!(
"AuditingWorkerLookup: audit enabled but no Tokio runtime — audit writes disabled"
);
}
Self { inner, audit, runtime_handle }
}
}
impl WorkerLookup for AuditingWorkerLookup {
fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
let result = self.inner.find(candidate_id);
// Fire audit AFTER the find so the result-state field reflects
// the actual outcome ("success" if found, "not_found" if not).
if let (Some(audit), Some(handle)) = (self.audit.clone(), self.runtime_handle.clone()) {
let cid = candidate_id.to_string();
let result_state = if result.is_some() { "success" } else { "not_found" };
// Single field: "exists" — validator only consumes existence
// + role/city/state from the WorkerRecord. We don't enumerate
// every WorkerRecord field as fields_accessed because that
// would imply they were USED by the caller; the validator
// only proves existence + then reads policy fields downstream.
let row = SubjectAuditRow {
schema: "subject_audit.v1".into(),
ts: chrono::Utc::now(),
candidate_id: cid.clone(),
accessor: AuditAccessor {
kind: "validator_lookup".into(),
daemon: "gateway".into(),
purpose: "validator_worker_lookup".into(),
trace_id: String::new(),
},
fields_accessed: vec!["exists".into()],
result: result_state.into(),
prev_chain_hash: String::new(),
row_hmac: String::new(),
};
// Spawn fire-and-forget. Chain integrity is preserved by the
// writer's per-subject Mutex (regression-tested), so spawning
// is safe. Audit-write errors are logged only.
handle.spawn(async move {
if let Err(e) = audit.append(row).await {
tracing::warn!("validator audit write failed for {cid}: {e}");
}
});
}
result
}
fn len(&self) -> usize {
self.inner.len()
}
}
#[cfg(test)]
mod tests {
use super::*;
use catalogd::subject_audit::SubjectAuditWriter;
use object_store::memory::InMemory;
use validator::InMemoryWorkerLookup;
fn fixture_inner_with(records: Vec<WorkerRecord>) -> Arc<dyn WorkerLookup> {
Arc::new(InMemoryWorkerLookup::from_records(records))
}
fn fixture_record(candidate_id: &str) -> WorkerRecord {
WorkerRecord {
candidate_id: candidate_id.into(),
name: "Test".into(),
status: "active".into(),
city: Some("Chicago".into()),
state: Some("IL".into()),
role: Some("Welder".into()),
blacklisted_clients: vec![],
}
}
#[tokio::test]
async fn find_existing_subject_writes_success_audit_row() {
let inner = fixture_inner_with(vec![fixture_record("CAND-EXIST")]);
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(
store.clone(),
(0u8..32).collect(),
));
let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
let r = lookup.find("CAND-EXIST");
assert!(r.is_some());
// Audit is spawned — give it a tick to land.
tokio::task::yield_now().await;
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
let count = writer.verify_chain("CAND-EXIST").await.unwrap();
assert_eq!(count, 1);
}
#[tokio::test]
async fn find_missing_subject_writes_not_found_audit_row() {
let inner = fixture_inner_with(vec![]);
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(
store.clone(),
(0u8..32).collect(),
));
let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
let r = lookup.find("CAND-PHANTOM");
assert!(r.is_none());
tokio::task::yield_now().await;
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
// verify_chain returns the row count for that subject (1 row,
// result="not_found"). We don't assert on result text here —
// that's covered by the hand-rolled JSON inspection below.
assert_eq!(writer.verify_chain("CAND-PHANTOM").await.unwrap(), 1);
}
#[tokio::test]
async fn audit_disabled_means_no_writes_no_overhead() {
let inner = fixture_inner_with(vec![fixture_record("CAND-A")]);
let lookup = AuditingWorkerLookup::new(inner, None);
// Just confirm find() still works with no audit — no panic, no
// runtime-handle requirement.
assert!(lookup.find("CAND-A").is_some());
assert!(lookup.find("CAND-MISSING").is_none());
assert_eq!(lookup.len(), 1);
}
#[tokio::test]
async fn many_finds_to_same_subject_produce_intact_chain() {
// Same regression as catalogd's concurrent test, but exercised
// through the validator wrapper path. 30 sequential finds on
// the SAME subject should produce an intact 30-row chain.
let inner = fixture_inner_with(vec![fixture_record("CAND-CHAIN")]);
let store: Arc<dyn object_store::ObjectStore> = Arc::new(InMemory::new());
let writer = Arc::new(SubjectAuditWriter::with_inline_key(
store.clone(),
(0u8..32).collect(),
));
let lookup = AuditingWorkerLookup::new(inner, Some(writer.clone()));
for _ in 0..30 {
let _ = lookup.find("CAND-CHAIN");
}
// Wait for all spawns to finish — the runtime is currently
// single-threaded test runtime, so a few yields drains.
for _ in 0..50 {
tokio::task::yield_now().await;
}
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let count = writer.verify_chain("CAND-CHAIN").await.unwrap();
assert_eq!(count, 30, "chain corrupted under sequential spawn load");
}
}
+1 -1
View File
@@ -140,7 +140,7 @@ pub async fn chat(
model,
choices: vec![Choice {
index: 0,
message: Message { role: "assistant".into(), content: serde_json::Value::String(text) },
message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() },
finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
}],
usage: UsageBlock {
+129 -10
View File
@@ -20,6 +20,7 @@ pub mod kimi;
pub mod opencode;
pub mod validate;
pub mod iterate;
pub mod auditing_worker_lookup;
pub mod langfuse_trace;
pub mod session_log;
pub mod mode;
@@ -91,6 +92,12 @@ pub struct V1State {
/// `[validatord].session_log_path` (commit 1a3a82a in
/// golangLAKEHOUSE).
pub session_log: Option<session_log::SessionLogger>,
/// Per-subject audit log writer (HMAC-chained JSONL). Loaded at
/// startup from /etc/lakehouse/subject_audit.key if the file is
/// present and ≥32 bytes. None = audit disabled with a warning
/// at startup; PII access is still served but produces no audit
/// trail. See docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
pub subject_audit: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>>,
}
#[derive(Default, Clone, Serialize)]
@@ -141,10 +148,22 @@ pub fn router(state: V1State) -> Router {
/// providers can take it verbatim. `Message::text()` flattens for
/// places that need a plain string (Ollama prompt assembly, char
/// counts, the assistant's own response synthesis).
#[derive(Serialize, Deserialize, Clone, Debug)]
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
pub struct Message {
pub role: String,
/// `Value::Null` is legal: an assistant turn that only makes tool calls.
#[serde(default)]
pub content: serde_json::Value,
/// OpenAI tool-calling fields (2026-09-19). Passed through verbatim on the OpenRouter lane;
/// omitted from JSON when unset, so plain-text traffic is byte-identical to before. Before
/// this, serde silently DROPPED them — tool-using agents routed through :3100 never saw their
/// tools and fabricated tool output (autobun KB finding infra.lakehouse_gateway.tools).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tool_calls: Option<serde_json::Value>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tool_call_id: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
}
impl Message {
@@ -156,6 +175,7 @@ impl Message {
Self {
role: role.into(),
content: serde_json::Value::String(body.into()),
..Default::default()
}
}
/// Flatten content to a plain string. Strings pass through; content-
@@ -188,9 +208,8 @@ pub struct ChatRequest {
pub temperature: Option<f64>,
#[serde(default)]
pub max_tokens: Option<u32>,
/// Accepted for shape-compat but ignored in the thin slice —
/// Phase 38 returns non-streaming even when the client asked for it.
/// Phase 39+ wires real streaming.
/// Real SSE passthrough on the openrouter lane (2026-09-20); other lanes reject
/// stream=true with a 400 rather than silently returning a non-streamed body.
#[serde(default)]
pub stream: Option<bool>,
/// Non-OpenAI extension. Passes through to the provider's thinking
@@ -208,6 +227,21 @@ pub struct ChatRequest {
/// from the model name; Phase 38/39 requires explicit selection.
#[serde(default)]
pub provider: Option<String>,
/// OpenAI tool definitions + choice (2026-09-19). Forwarded on the OpenRouter lane only;
/// any other provider REJECTS them with 400 instead of silently stripping them.
#[serde(default)]
pub tools: Option<serde_json::Value>,
#[serde(default)]
pub tool_choice: Option<serde_json::Value>,
}
impl ChatRequest {
/// True if the request uses tool calling anywhere (definitions or tool-call history).
pub fn uses_tools(&self) -> bool {
self.tools.is_some()
|| self.tool_choice.is_some()
|| self.messages.iter().any(|m| m.role == "tool" || m.tool_calls.is_some() || m.tool_call_id.is_some())
}
}
#[derive(Serialize)]
@@ -312,9 +346,47 @@ mod resolve_provider_tests {
stream: None,
think: None,
provider: provider.map(|s| s.to_string()),
tools: None,
tool_choice: None,
}
}
/// Streaming is a per-lane capability, and the routing decision must be visible in a test:
/// before 2026-09-20 the gateway accepted stream=true on every lane and returned a single
/// JSON body, so a client waiting for SSE frames hung until its own timeout.
#[test]
fn streaming_is_openrouter_only() {
let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash");
r.stream = Some(true);
assert_eq!(resolve_provider(&r).0, "openrouter", "openrouter lane streams");
let mut local = mk_req(None, "qwen3.5:latest");
local.stream = Some(true);
assert_eq!(
resolve_provider(&local).0,
"ollama",
"a local model resolves to ollama, where the handler rejects stream=true with a 400 \
instead of silently returning a non-streamed body"
);
}
#[test]
fn uses_tools_detects_definitions_and_history() {
let mut r = mk_req(None, "openrouter/z-ai/glm-5.3-flash");
assert!(!r.uses_tools());
r.tools = Some(serde_json::json!([]));
assert!(r.uses_tools());
let mut r2 = mk_req(None, "qwen3.5:latest");
r2.messages.push(Message { role: "tool".into(), content: "ok".into(), tool_call_id: Some("c1".into()), ..Default::default() });
assert!(r2.uses_tools());
}
#[test]
fn plain_message_serialization_unchanged() {
let m = Message::new_text("user", "hi");
assert_eq!(serde_json::to_string(&m).unwrap(), r#"{"role":"user","content":"hi"}"#);
}
#[test]
fn explicit_provider_wins() {
let r = mk_req(Some("openrouter"), "qwen3.5:latest");
@@ -371,19 +443,27 @@ async fn chat(
State(state): State<V1State>,
headers: axum::http::HeaderMap,
Json(req): Json<ChatRequest>,
) -> Result<Json<ChatResponse>, (StatusCode, String)> {
) -> Result<axum::response::Response, (StatusCode, String)> {
use axum::response::IntoResponse as _;
if req.messages.is_empty() {
return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
}
if req.stream.unwrap_or(false) {
tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
}
// Provider resolution: explicit `req.provider` wins; otherwise
// infer from a model-name prefix. Phase 39 PRD gate example:
// `model: "openrouter/openai/gpt-4o-mini"` → provider "openrouter",
// adapter gets the stripped "openai/gpt-4o-mini".
let (provider, effective_model) = resolve_provider(&req);
// Tool calling is only wired on the OpenRouter lane. Refuse loudly elsewhere — silently
// dropping tools made agents fabricate tool results (2026-09-19).
if req.uses_tools() && !matches!(provider.as_str(), "openrouter" | "openrouter_free") {
return Err((
StatusCode::BAD_REQUEST,
format!(
"tools/tool_calls are only supported on the openrouter lane via this gateway \
(resolved provider: {provider}); use an OpenRouter model id (e.g. openrouter/<vendor>/<model>)"
),
));
}
let start_time = chrono::Utc::now();
let start_instant = std::time::Instant::now();
@@ -399,6 +479,45 @@ async fn chat(
std::borrow::Cow::Owned(cloned)
};
// STREAMING (2026-09-20). Phase 38 accepted `stream: true` and returned a single JSON body;
// a client waiting for SSE frames hangs and then reports "no response". Forward the upstream
// SSE verbatim on the lane that supports it, and say so plainly on the lanes that do not
// instead of pretending. Streamed calls skip usage metering + Langfuse (see chat_stream).
if req.stream.unwrap_or(false) {
if !matches!(provider.as_str(), "openrouter" | "openrouter_free") {
return Err((
StatusCode::BAD_REQUEST,
format!(
"stream=true is only supported on the openrouter lane via this gateway \
(resolved provider: {provider}); resend with stream=false or use an \
OpenRouter model id"
),
));
}
let key = state.openrouter_key.as_deref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"OPENROUTER_API_KEY not configured".to_string(),
))?;
let upstream = openrouter::chat_stream(key, &*req_for_adapter)
.await
.map_err(|e| (StatusCode::BAD_GATEWAY, format!("openrouter stream: {e}")))?;
tracing::info!(
target: "v1.chat",
provider = %provider,
model = %effective_model,
"streaming passthrough — usage metering and Langfuse are skipped for this call",
);
let body = axum::body::Body::from_stream(upstream.bytes_stream());
return Ok((
[
(axum::http::header::CONTENT_TYPE, "text/event-stream"),
(axum::http::header::CACHE_CONTROL, "no-cache"),
],
body,
)
.into_response());
}
let (resp, used_provider) = match provider.as_str() {
"ollama" | "local" | "" => {
let r = ollama::chat(&state.ai_client, &*req_for_adapter)
@@ -582,7 +701,7 @@ async fn chat(
provider_usage.total_tokens += resp.usage.total_tokens as u64;
}
Ok(Json(resp))
Ok(Json(resp).into_response())
}
async fn usage(State(state): State<V1State>) -> impl IntoResponse {
+10 -10
View File
@@ -113,10 +113,10 @@ mod tests {
#[test]
fn flatten_separates_system_from_turns() {
let msgs = vec![
Message { role: "system".into(), content: "Rules here.".into() },
Message { role: "user".into(), content: "Q1".into() },
Message { role: "assistant".into(), content: "A1".into() },
Message { role: "user".into(), content: "Q2".into() },
Message { role: "system".into(), content: "Rules here.".into(), ..Default::default() },
Message { role: "user".into(), content: "Q1".into(), ..Default::default() },
Message { role: "assistant".into(), content: "A1".into(), ..Default::default() },
Message { role: "user".into(), content: "Q2".into(), ..Default::default() },
];
let (system, prompt) = flatten_messages(&msgs);
assert_eq!(system, "Rules here.");
@@ -129,9 +129,9 @@ mod tests {
#[test]
fn flatten_concatenates_multiple_system_messages() {
let msgs = vec![
Message { role: "system".into(), content: "First.".into() },
Message { role: "system".into(), content: "Second.".into() },
Message { role: "user".into(), content: "Hi".into() },
Message { role: "system".into(), content: "First.".into(), ..Default::default() },
Message { role: "system".into(), content: "Second.".into(), ..Default::default() },
Message { role: "user".into(), content: "Hi".into(), ..Default::default() },
];
let (system, _) = flatten_messages(&msgs);
assert_eq!(system, "First.\nSecond.");
@@ -139,7 +139,7 @@ mod tests {
#[test]
fn flatten_with_no_system_returns_empty_system() {
let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
let msgs = vec![Message { role: "user".into(), content: "hi".into(), ..Default::default() }];
let (system, prompt) = flatten_messages(&msgs);
assert!(system.is_empty());
assert!(prompt.contains("user: hi"));
@@ -147,11 +147,11 @@ mod tests {
#[test]
fn estimate_tokens_chars_div_4_ceiling() {
let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into(), ..Default::default() }];
// 8 chars / 4 = 2, with ceiling → 2
assert_eq!(estimate_prompt_tokens(&msgs), 2);
let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into(), ..Default::default() }];
// 9 chars → (9+3)/4 = 3 (ceiling)
assert_eq!(estimate_prompt_tokens(&msgs2), 3);
}
+1 -1
View File
@@ -145,7 +145,7 @@ pub async fn chat(
model,
choices: vec![Choice {
index: 0,
message: Message { role: "assistant".into(), content: serde_json::Value::String(text) },
message: Message { role: "assistant".into(), content: serde_json::Value::String(text), ..Default::default() },
finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
}],
usage: UsageBlock {
+239 -23
View File
@@ -20,7 +20,19 @@ use serde::{Deserialize, Serialize};
use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
const OR_BASE_URL: &str = "https://openrouter.ai/api/v1";
const OR_TIMEOUT_SECS: u64 = 180;
// Default request ceiling. 180s starves long-context calls (kimi_architect's
// ~180K-char second pass died at exactly this wall on 2026-09-02, surfacing
// as a misleading "error decoding response body") — override per host with
// OR_TIMEOUT_SECS. The auditor budgets up to 900s client-side.
const OR_TIMEOUT_SECS_DEFAULT: u64 = 180;
fn or_timeout_secs() -> u64 {
std::env::var("OR_TIMEOUT_SECS")
.ok()
.and_then(|v| v.trim().parse::<u64>().ok())
.filter(|&s| s > 0)
.unwrap_or(OR_TIMEOUT_SECS_DEFAULT)
}
pub fn resolve_openrouter_key() -> Option<String> {
if let Ok(k) = std::env::var("OPENROUTER_API_KEY") {
@@ -49,6 +61,52 @@ pub fn resolve_openrouter_key() -> Option<String> {
None
}
/// Streaming passthrough (2026-09-20). The gateway used to accept `stream: true` and return a
/// single non-streamed JSON body — a client waiting for SSE frames hangs until its own timeout
/// and reports "no response" (this is what broke openclaw's gateway lane). We now forward the
/// upstream SSE byte-for-byte.
///
/// The caller streams the body straight to its client, so nothing here parses the chunks:
/// usage metering and Langfuse tracing are SKIPPED for streamed calls, and the handler logs
/// that. Metering streamed responses needs the final usage chunk and belongs in its own change.
pub async fn chat_stream(key: &str, req: &ChatRequest) -> Result<reqwest::Response, String> {
let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string();
let body = ORChatBody {
model,
messages: req.messages.iter().map(|m| ORMessage {
role: m.role.clone(),
content: m.content.clone(),
tool_calls: m.tool_calls.clone(),
tool_call_id: m.tool_call_id.clone(),
name: m.name.clone(),
}).collect(),
tools: req.tools.clone(),
tool_choice: req.tool_choice.clone(),
max_tokens: Some(req.max_tokens.unwrap_or(800)),
temperature: req.temperature.unwrap_or(0.3),
stream: true,
};
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(or_timeout_secs()))
.build()
.map_err(|e| format!("build client: {e}"))?;
let resp = client
.post(format!("{}/chat/completions", OR_BASE_URL))
.bearer_auth(key)
.header("HTTP-Referer", "https://vcp.devop.live")
.header("X-Title", "Lakehouse Scrum")
.json(&serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?)
.send()
.await
.map_err(|e| format!("openrouter.ai unreachable: {e}"))?;
if !resp.status().is_success() {
let code = resp.status();
let text = resp.text().await.unwrap_or_else(|_| "?".into());
return Err(format!("openrouter {code}: {}", text.chars().take(300).collect::<String>()));
}
Ok(resp)
}
pub async fn chat(
key: &str,
req: &ChatRequest,
@@ -57,7 +115,7 @@ pub async fn chat(
// form so OpenRouter sees the raw model id (e.g. "openai/gpt-oss-120b:free").
let model = req.model.strip_prefix("openrouter/").unwrap_or(&req.model).to_string();
let body = ORChatBody {
let mut body = ORChatBody {
model: model.clone(),
// Pass content through verbatim — preserves OpenAI's multimodal
// content-parts shape (`[{type:"text",text:"..."}, ...]`) so the
@@ -65,30 +123,78 @@ pub async fn chat(
messages: req.messages.iter().map(|m| ORMessage {
role: m.role.clone(),
content: m.content.clone(),
tool_calls: m.tool_calls.clone(),
tool_call_id: m.tool_call_id.clone(),
name: m.name.clone(),
}).collect(),
max_tokens: req.max_tokens.unwrap_or(800),
tools: req.tools.clone(),
tool_choice: req.tool_choice.clone(),
max_tokens: Some(req.max_tokens.unwrap_or(800)),
temperature: req.temperature.unwrap_or(0.3),
stream: false,
};
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(OR_TIMEOUT_SECS))
.timeout(Duration::from_secs(or_timeout_secs()))
.build()
.map_err(|e| format!("build client: {e}"))?;
let t0 = std::time::Instant::now();
let resp = client
.post(format!("{}/chat/completions", OR_BASE_URL))
.bearer_auth(key)
// OpenRouter recommends Referer + Title for attribution; absent
// headers do not fail the call but help us see our traffic in
// their dashboard.
.header("HTTP-Referer", "https://vcp.devop.live")
.header("X-Title", "Lakehouse Scrum")
.json(&body)
.send()
.await
.map_err(|e| format!("openrouter.ai unreachable: {e}"))?;
let send = |client: reqwest::Client, key: String, body_json: serde_json::Value| async move {
client
.post(format!("{}/chat/completions", OR_BASE_URL))
.bearer_auth(key)
// OpenRouter recommends Referer + Title for attribution; absent
// headers do not fail the call but help us see our traffic in
// their dashboard.
.header("HTTP-Referer", "https://vcp.devop.live")
.header("X-Title", "Lakehouse Scrum")
.json(&body_json)
.send()
.await
.map_err(|e| format!("openrouter.ai unreachable: {e}"))
};
let mut resp = send(
client.clone(),
key.to_string(),
serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
)
.await?;
// OpenRouter's affordability precheck reserves max_tokens × price against
// the account balance. Measured 2026-08-28: stripping the cap makes it
// WORSE (the precheck then prices the model's own maximum, e.g. 65536) —
// so a low-balance 402 surfaces verbatim and the fix is funding the
// account the key belongs to, not request-shape games. One clamp retry:
// when the error names an affordable budget ≥256 tokens, retry once at
// that budget so small answers still get through on a thin balance.
if resp.status() == reqwest::StatusCode::PAYMENT_REQUIRED && body.max_tokens.is_some() {
let err_body = resp.text().await.unwrap_or_else(|_| "?".into());
let afford = err_body
.split("can only afford ")
.nth(1)
.and_then(|s| s.split(|c: char| !c.is_ascii_digit()).next())
.and_then(|n| n.parse::<u32>().ok());
match afford {
Some(n) if n >= 256 => {
tracing::warn!(
target: "v1.chat",
provider = "openrouter",
model = %model,
afford = n,
"402 affordability precheck — clamping max_tokens to the stated budget",
);
body.max_tokens = Some(n);
resp = send(
client.clone(),
key.to_string(),
serde_json::to_value(&body).map_err(|e| format!("serialize body: {e}"))?,
)
.await?;
}
_ => return Err(format!("openrouter.ai 402 Payment Required: {}", err_body)),
}
}
let status = resp.status();
if !status.is_success() {
@@ -102,7 +208,24 @@ pub async fn chat(
let latency_ms = t0.elapsed().as_millis();
let choice = parsed.choices.into_iter().next()
.ok_or_else(|| "openrouter returned no choices".to_string())?;
let text = choice.message.content;
let tool_calls = choice.message.tool_calls.clone();
let reasoning_only = choice.message.content.is_none()
&& choice.message.reasoning.is_some()
&& tool_calls.is_none();
let text = choice.message.content.unwrap_or_default();
if reasoning_only {
// The model spent its whole budget thinking. Return well-formed empty
// content (callers' schema validation + retry budgets handle it
// deterministically) rather than a decode error that reads as a
// gateway fault — and never pass thinking-text off as an answer.
tracing::warn!(
target: "v1.chat",
provider = "openrouter",
model = %model,
finish = %choice.finish_reason.as_deref().unwrap_or("?"),
"reasoning-only response (content null) — raise max_tokens or use a non-reasoning model",
);
}
let prompt_tokens = parsed.usage.as_ref().map(|u| u.prompt_tokens).unwrap_or_else(|| {
let chars: usize = req.messages.iter().map(|m| m.text().chars().count()).sum();
@@ -129,7 +252,17 @@ pub async fn chat(
model,
choices: vec![Choice {
index: 0,
message: Message { role: "assistant".into(), content: serde_json::Value::String(text) },
message: Message {
role: "assistant".into(),
// OpenAI shape: a tool-call-only turn carries `content: null`.
content: if tool_calls.is_some() && text.is_empty() {
serde_json::Value::Null
} else {
serde_json::Value::String(text)
},
tool_calls,
..Default::default()
},
finish_reason: choice.finish_reason.unwrap_or_else(|| "stop".into()),
}],
usage: UsageBlock {
@@ -146,13 +279,27 @@ pub async fn chat(
struct ORChatBody {
model: String,
messages: Vec<ORMessage>,
max_tokens: u32,
#[serde(skip_serializing_if = "Option::is_none")]
max_tokens: Option<u32>,
temperature: f64,
stream: bool,
#[serde(skip_serializing_if = "Option::is_none")]
tools: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
tool_choice: Option<serde_json::Value>,
}
#[derive(Serialize)]
struct ORMessage { role: String, content: serde_json::Value }
struct ORMessage {
role: String,
content: serde_json::Value,
#[serde(skip_serializing_if = "Option::is_none")]
tool_calls: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
tool_call_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
name: Option<String>,
}
#[derive(Deserialize)]
struct ORChatResponse {
@@ -169,7 +316,21 @@ struct ORChoice {
}
#[derive(Deserialize)]
struct ORMessageResp { content: String }
struct ORMessageResp {
// Reasoning models (GLM-5.x, deepseek-v4-pro) return `content: null` plus a
// `reasoning` field when they exhaust max_tokens mid-thought. A bare String
// here made serde reject the whole body ("invalid openrouter response",
// 2026-08-28) — which took down every reasoning model on the lane,
// including phase2's configured glm-5.2. Content stays the only text we
// ever return; reasoning is observability, never an answer.
#[serde(default)]
content: Option<String>,
#[serde(default)]
reasoning: Option<String>,
/// OpenAI tool calls requested by the model (passed back to the client verbatim).
#[serde(default)]
tool_calls: Option<serde_json::Value>,
}
#[derive(Deserialize)]
struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
@@ -178,6 +339,21 @@ struct ORUsage { prompt_tokens: u32, completion_tokens: u32 }
mod tests {
use super::*;
#[test]
fn reasoning_model_null_content_parses() {
// GLM-5.x / deepseek-v4-pro shape: content null + reasoning present
// (the 2026-08-28 decode failure). Must parse, not error.
let body = r#"{"choices":[{"message":{"role":"assistant","content":null,"reasoning":"thinking…"},"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":10}}"#;
let parsed: ORChatResponse = serde_json::from_str(body).expect("null content must parse");
let choice = &parsed.choices[0];
assert!(choice.message.content.is_none());
assert_eq!(choice.message.reasoning.as_deref(), Some("thinking…"));
// And the normal shape still parses with reasoning absent.
let normal = r#"{"choices":[{"message":{"role":"assistant","content":"OK"},"finish_reason":"stop"}]}"#;
let parsed: ORChatResponse = serde_json::from_str(normal).expect("normal shape");
assert_eq!(parsed.choices[0].message.content.as_deref(), Some("OK"));
}
#[test]
fn resolve_openrouter_key_does_not_panic() {
// Smoke test — all three sources may or may not be set depending
@@ -190,17 +366,57 @@ mod tests {
let body = ORChatBody {
model: "openai/gpt-oss-120b:free".into(),
messages: vec![
ORMessage { role: "user".into(), content: "review this".into() },
ORMessage { role: "user".into(), content: "review this".into(),
tool_calls: None, tool_call_id: None, name: None },
],
max_tokens: 800,
max_tokens: Some(800),
temperature: 0.3,
stream: false,
tools: None,
tool_choice: None,
};
let json = serde_json::to_string(&body).unwrap();
assert!(json.contains("\"model\":\"openai/gpt-oss-120b:free\""));
assert!(json.contains("\"messages\""));
assert!(json.contains("\"max_tokens\":800"));
assert!(json.contains("\"stream\":false"));
// no tool fields on plain traffic (byte-compat with pre-2026-09-19 bodies)
assert!(!json.contains("tools") && !json.contains("tool_call"));
}
#[test]
fn chat_body_forwards_tools_and_tool_messages() {
let tools = serde_json::json!([{"type":"function","function":{"name":"exec","parameters":{"type":"object"}}}]);
let body = ORChatBody {
model: "z-ai/glm-5.3-flash".into(),
messages: vec![
ORMessage { role: "assistant".into(), content: serde_json::Value::Null,
tool_calls: Some(serde_json::json!([{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}])),
tool_call_id: None, name: None },
ORMessage { role: "tool".into(), content: "ok".into(), tool_calls: None,
tool_call_id: Some("c1".into()), name: Some("exec".into()) },
],
max_tokens: Some(100),
temperature: 0.0,
stream: false,
tools: Some(tools),
tool_choice: Some(serde_json::json!("auto")),
};
let json = serde_json::to_string(&body).unwrap();
assert!(json.contains("\"tools\":[{"));
assert!(json.contains("\"tool_choice\":\"auto\""));
assert!(json.contains("\"tool_call_id\":\"c1\""));
assert!(json.contains("\"tool_calls\":[{"));
}
#[test]
fn response_with_tool_calls_parses() {
let raw = r#"{"choices":[{"message":{"content":null,"tool_calls":[{"id":"c1","type":"function","function":{"name":"exec","arguments":"{}"}}]},"finish_reason":"tool_calls"}]}"#;
let parsed: ORChatResponse = serde_json::from_str(raw).unwrap();
let c = &parsed.choices[0];
assert!(c.message.content.is_none());
assert!(c.message.tool_calls.is_some());
assert_eq!(c.finish_reason.as_deref(), Some("tool_calls"));
}
#[test]
+225
View File
@@ -364,3 +364,228 @@ pub enum Redaction {
keep_suffix: usize,
},
}
// ─── Subject manifests (compliance/access metadata layer) ────────────
// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
//
// A SubjectManifest is a per-person record threading existing catalogd
// primitives (datasets, views, tombstones) together by subject token.
// Stored at _catalog/subjects/<candidate_id>.json. NOT a separate
// daemon — extends catalogd's existing manifest pattern.
//
// The audit log per subject (HMAC-chained JSONL at
// _catalog/subjects/<candidate_id>.audit.jsonl) is owned by a separate
// SubjectAuditWriter (Step 2) — not modeled here.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum SubjectStatus {
PendingConsent,
Active,
Withdrawn,
RetentionExpired,
Erased,
}
/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed:
/// the gateway treats Unknown as healthcare-equivalent until reclassified
/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1).
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum SubjectVertical {
Unknown,
General,
Healthcare,
Finance,
Other,
}
impl Default for SubjectVertical {
fn default() -> Self { SubjectVertical::Unknown }
}
/// State machine for general-PII consent. BIPA-specific biometric
/// consent is tracked separately on `BiometricConsent`.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum ConsentStatus {
/// Backfilled from existing data sources before consent was tracked.
/// SHOULD route as if no consent exists until reviewed.
PendingBackfillReview,
/// New subject; awaiting consent UX.
PendingFirstContact,
Given,
Withdrawn,
Expired,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum BiometricConsentStatus {
/// Default. No biometric data exists for this subject.
NeverCollected,
Pending,
Given,
Withdrawn,
Expired,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GeneralPiiConsent {
pub status: ConsentStatus,
/// Reference to the consent template version (matches a row in the
/// future consent_versions table). Empty when status is pending_*.
#[serde(default)]
pub version: String,
#[serde(default)]
pub given_at: Option<chrono::DateTime<chrono::Utc>>,
#[serde(default)]
pub withdrawn_at: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BiometricConsent {
pub status: BiometricConsentStatus,
/// BIPA: max 3 years from last interaction. Implementation MUST
/// enforce daily expiration sweep against this field.
#[serde(default)]
pub retention_until: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectConsent {
pub general_pii: GeneralPiiConsent,
pub biometric: BiometricConsent,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectRetention {
pub general_pii_until: chrono::DateTime<chrono::Utc>,
/// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026").
#[serde(default)]
pub policy: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectDatasetRef {
/// catalogd dataset name (must resolve to an existing manifest).
pub name: String,
/// Column in that dataset that holds the subject's identifier.
pub key_column: String,
/// Specific identifier value (typically same as candidate_id; differs
/// only when a dataset uses a foreign-key with a different format).
pub key_value: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectManifest {
/// Schema version tag — always "subject_manifest.v1" for v1.
#[serde(default = "default_subject_manifest_schema")]
pub schema: String,
/// Subject identifier (the canonical token, e.g. "CAND-000001").
pub candidate_id: String,
pub created_at: chrono::DateTime<chrono::Utc>,
pub updated_at: chrono::DateTime<chrono::Utc>,
pub status: SubjectStatus,
#[serde(default)]
pub vertical: SubjectVertical,
pub consent: SubjectConsent,
pub retention: SubjectRetention,
/// Datasets that contain rows for this subject. Used by audit-response
/// to know what to project. Each entry MUST reference an existing
/// catalogd dataset manifest (validated on put_subject).
pub datasets: Vec<SubjectDatasetRef>,
/// Names of existing AiViews that safely project this subject's data.
/// Used by service-tier readers (no PII exposure).
#[serde(default)]
pub safe_views: Vec<String>,
/// Path (relative to repo root) of the per-subject audit JSONL.
/// Owned by SubjectAuditWriter (Step 2). Empty until first write.
#[serde(default)]
pub audit_log_path: String,
/// SHA-256 of the most recent HMAC chain checkpoint of the audit log.
/// Empty until first audit row written.
#[serde(default)]
pub audit_log_chain_root: String,
/// Biometric collection metadata. None until the first biometric
/// upload via Phase 1.6 Gate 3 photo endpoint. Cleared on erasure.
/// `serde(default)` so existing manifests parse without re-emit.
#[serde(default)]
pub biometric_collection: Option<BiometricCollection>,
}
/// Per-subject biometric collection record. Written by the Gate 3
/// photo-upload endpoint when a candidate's photo is accepted under
/// `consent.biometric.status == "given"`.
///
/// Reset to None by the BIPA erasure flow (destruction runbook).
/// Preserved across schema evolutions via the BTreeMap-canonicalized
/// HMAC chain — old audit rows verify regardless of new fields.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct BiometricCollection {
/// Quarantined filesystem path of the original photo bytes,
/// relative to the configured biometric storage root. NEVER under
/// `data/headshots/` (which is reserved for synthetic StyleGAN
/// faces). Format: `data/biometric/uploads/<sanitized_id>/<ts>.<ext>`.
pub data_path: String,
/// SHA-256 of the original photo bytes. Used for integrity check
/// (file-on-disk matches what was uploaded), NOT for
/// re-derivation of the original from the hash.
pub template_hash: String,
pub collected_at: chrono::DateTime<chrono::Utc>,
/// SHA-256 of the consent template version that was in force when
/// this photo was uploaded. Anchors the consent record to a
/// specific document version (per Phase 1.6 Gate 2 §6).
#[serde(default)]
pub consent_version_hash: String,
/// deepface classifications, when the offline classifier has run.
/// None until classifier integration ships (Gate 3b).
#[serde(default)]
pub classifications: Option<serde_json::Value>,
}
fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() }
/// Audit accessor — who/why touched a subject's PII.
/// Persisted in subject_audit.v1 rows; never logged in plaintext outside
/// the per-subject audit file.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AuditAccessor {
/// One of: "gateway_lookup", "audit_response", "legal_request",
/// "system_resolve". Caller-defined; SubjectAuditWriter does not
/// validate the kind string.
pub kind: String,
/// Daemon name (e.g. "gateway", "validator", "catalogd").
pub daemon: String,
/// Purpose token (e.g. "fill_validation", "audit_subject_response").
pub purpose: String,
/// X-Lakehouse-Trace-Id for cross-daemon trace correlation. Empty
/// when the request didn't carry one.
#[serde(default)]
pub trace_id: String,
}
/// One audit row in a subject's per-subject audit JSONL.
/// HMAC chain: row_hmac = HMAC-SHA256(key, prev_chain_hash || canon(row - row_hmac))
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectAuditRow {
/// Always "subject_audit.v1" for v1.
#[serde(default = "default_subject_audit_schema")]
pub schema: String,
pub ts: chrono::DateTime<chrono::Utc>,
pub candidate_id: String,
pub accessor: AuditAccessor,
/// Field names accessed (e.g. ["name"], ["name", "phone"]).
pub fields_accessed: Vec<String>,
/// "success" | "denied" | "not_found" | "error".
pub result: String,
/// Hash of the previous row in this subject's audit log. "GENESIS"
/// for the first row.
pub prev_chain_hash: String,
/// HMAC-SHA256 over (prev_chain_hash || canonical_json(row - row_hmac)).
/// Computed by SubjectAuditWriter; callers leave empty on input.
#[serde(default)]
pub row_hmac: String,
}
fn default_subject_audit_schema() -> String { "subject_audit.v1".into() }
+24 -1
View File
@@ -70,10 +70,33 @@ But the written rule missed something. It assumed Lance's value would show up as
### What we keep watching (but don't act on yet)
- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.
- ~~**Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.~~ **Done 2026-05-02** — see "Follow-up: 10M re-bench" below.
- **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system.
- **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot.
---
## Follow-up: 10M re-bench (2026-05-02)
The 10M re-bench above ran. Numbers from `data/lance/scale_test_10m` (33 GB, 10M × 768d, IVF_PQ live, post-doc_id-btree-build). Full report: `reports/lance_10m_rebench_2026-05-02.md`.
| Op | 100K (this ADR) | 10M (re-bench) | Notes |
|---|---:|---:|---|
| Search (cold) | 2229μs | ~46ms median | 21× slower at 100× scale → reasonable for IVF_PQ |
| Search (warm) | (not measured) | ~20ms p50 | Stable across 5 trials |
| Doc fetch by id | 311μs | ~5ms | Structural ADR-019 win confirmed once `doc_id` btree is built |
| Index method | lance_ivf_pq | lance_ivf_pq | response tag confirms |
**The HNSW comparison at 10M doesn't exist** — at 10M × 768d × 4 bytes = ~30 GB just for vectors, doubled for the graph. HNSW doesn't fit on a single 128 GB box at this scale. So the original "Lance pulls ahead at 10M" framing is true by elimination: Lance is the only contender that operationally exists at 10M. The strategic question is reframed as "Lance vs Parquet+HNSW-with-spilling," deferred until we have a workload where the Parquet path is the bottleneck (currently tracked in `golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md` decisions tracker).
**Bug surfaced and fixed during re-bench:** Initial doc-fetch was ~100ms (full table scan). Root cause: the `doc_id` scalar btree was never built — `lance-bench` writes datasets by bypassing `IndexMeta`, and the activation-time auto-build only runs for IndexMeta-registered indexes. Fixed at two layers (commits `5d30b3d` + `044650a`):
- `lance_migrate` HTTP handler auto-builds the btree inline (~1.2s on 10M, +269MB on disk)
- `lance-bench` binary builds the btree post-IVF for parity with the gateway path
**Gauntlet added 2026-05-02:** the lance crates had zero tests + no smoke when audited. Now have 7 unit tests in `crates/vectord-lance` + 12 sanitize tests in `crates/vectord` + 10-probe `scripts/lance_smoke.sh` + sanitized error boundary across all 5 routes. See commits `7bb66f0`, `ac7c996`, `e9d17f7` (sanitizer iterations driven by cross-lineage scrum).
**Status:** ADR-019's hybrid architecture (Parquet+HNSW primary, Lance secondary) is now empirically validated up to 10M. The "watch and re-bench" item is closed.
## Why this isn't moving the goalposts
The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile.
+177
View File
@@ -0,0 +1,177 @@
# Phase 1.5 — BIPA Schema Audit + outcomes.jsonl Content Sample
**Status:** Complete — 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md)
> **Scope.** Two follow-up walks defined in `AUDIT_PHASE_1_DISCOVERY.md` §10/C4 + §10/single-reviewer (gemini "culture fit"): (1) BIPA-specific photo/video/biometric column + code path inventory, (2) actual content sample of `data/_kb/outcomes.jsonl` to confirm whether PII or discrimination-proxy reasoning lands there. Read-only walk; no code changes.
---
## §1 — BIPA walk: photo/video/biometric surface
### 1A — The face pipeline
The Rust legacy ships a synthetic face pool with deepface-extracted demographic classifications. **Even though the FACES are synthetic, the LOGIC is BIPA-relevant** — when real photos arrive (J 2026-05-03: "we will" have them), the same pipeline processes them.
| File:line | What | BIPA relevance |
|---|---|---|
| `scripts/staffing/fetch_face_pool.py` | Fetches 1000 synthetic faces from `thispersondoesnotexist.com` (StyleGAN). | Synthetic — no biometric collection at this stage. |
| `scripts/staffing/tag_face_pool.py:18-31` | Runs `deepface` on each face to extract `gender` (man / woman) + `race` (asian / black / hispanic / indian / middle_eastern / white) + `age`. Writes to `data/headshots/manifest.jsonl`. | **BIPA biometric processing.** Deepface extracts demographic classifications from face geometry. The output IS biometric information per 740 ILCS 14/10. |
| `data/headshots/manifest.jsonl` | 1000 rows, each `{id, file, gender, race, age}`. Sampled rows confirm shape: e.g. `{id: 0, file: "face_0000.jpg", gender: "woman", race: "east_asian", age: 33}`. | The persisted classifications. **For real photos, this file becomes a regulated biometric database.** |
| `mcp-server/index.ts:1308` (`/headshots/:key` route) | Serves face matched by gender × race × age intersection bucketing for a candidate. | Routing logic predicated on biometric-derived demographic attributes. |
| `mcp-server/index.ts:1367` (`/headshots/generate/:key`) | ComfyUI on-demand portrait generation. | Custom-generated image keyed to candidate. |
| `mcp-server/index.ts:1408` (ComfyUI prompt template) | `professional headshot portrait of a ${age}-year-old ${raceText} ${genderText} ${role}, ${scene}, neutral confident expression, sharp focus, photorealistic` | **Protected attributes (age + race + gender) embedded directly into the model prompt.** Even with synthetic output, this is system-level encoding of protected-attribute features into AI workflow. |
### 1B — Name-based ethnicity inference (inferred-attribute risk)
`mcp-server/search.html:3375-3432` defines lookup tables that classify candidates by demographic attributes derived from their NAMES (not photos):
| Constant | What |
|---|---|
| `FEMALE_NAMES` (~200 entries) + `MALE_NAMES` | Gender inference from first name |
| `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` | Ethnicity inference from first name |
| `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` | Ethnicity inference from surname (more diagnostic than first name per the source comment) |
**This is hard-coded inferred-attribute classification.** Per CLAUDE.md, the rationale is "Confidence-default name resolution... so a forklift operator looks like a forklift operator" — i.e., assigning representative photos. But the code path is:
`candidate name → genderFor() / guessEthnicityFromFirstName() → face-pool bucket selection`
A discrimination plaintiff's lawyer would frame this as: "the system EXPLICITLY classifies candidates by inferred ethnicity and uses that classification in its workflow." Whether the downstream use is "just for the photo display" or "ranks candidates" doesn't matter for BIPA-style strict-liability framing — it matters for Title VII, where intent / disparate-impact tests apply.
### 1C — Data directories holding face content
| Path | Count today | Purpose | gitignored? |
|---|---:|---|---|
| `data/headshots/` (face_*.jpg + manifest.jsonl) | 2500 entries | Synthetic face pool + biometric classifications | partial — manifest.jsonl is tracked, face_*.jpg are gitignored |
| `data/headshots_role_pool/` | 7 entries | Role-specific representative faces | gitignored |
| `data/headshots_gen/` | 4 entries | ComfyUI on-demand portrait cache | gitignored |
| `data/icons_pool/` | 2 entries | Certification + role icons | gitignored |
| `data/face_test/` | 1 entry | Likely deepface dev/test scratch | gitignored |
| `data/_imagecache/` | (not enumerated) | Image cache | gitignored |
The `manifest.jsonl` file is **tracked in git** today and contains the demographic-classification table. For synthetic faces this is fine; **for real candidate faces, this would be a biometric database in version control** — a serious BIPA + general-data-protection failure.
### 1D — Biometric-info contract risk when real photos arrive
When the system transitions from synthetic to real candidate photos (J: "we will"), the BIPA exposure becomes immediate:
1. **No written consent flow exists.** BIPA §15(b) requires "informed written consent" before biometric collection. The headshot upload path (wherever it gets built) needs a consent gate.
2. **No public retention schedule exists.** BIPA §15(a) requires a written, publicly-available retention schedule. No such document found.
3. **No deletion procedure exists.** BIPA §15(a) requires destruction "when the initial purpose for collecting or obtaining such identifiers or information has been satisfied or within 3 years of the individual's last interaction with the private entity, whichever occurs first." No such procedure exists.
4. **No employee training documented.** BIPA litigation routinely cites lack of employee training on biometric handling as evidence of negligence.
5. **The deepface library is itself a biometric processor.** Using it triggers the BIPA disclosure obligation regardless of whether the system stores the underlying face image — extracting "race" and "gender" from a face IS biometric processing under modern interpretations.
6. **Per-violation damages.** Negligent violations: $1,000 per violation per person. Intentional/reckless: $5,000 per violation per person. The IL Supreme Court has held each separate scan/extraction can constitute a separate violation (Cothron v. White Castle, 2023). Class actions are routine; recent settlements:
- Facebook (Patel v. Facebook, 2020): $650M
- TikTok (In re TikTok BIPA, 2022): $92M
- Six Flags (Rosenbach v. Six Flags, 2019): set the "no actual harm needed" precedent
- Smaller staffing companies have settled for $1M-$50M ranges.
### 1E — What this means for the architecture
Even with synthetic data, the architecture has BIPA-shaped surface that is one upload path away from being live. **Recommended pre-real-photo gates:**
1. **Disable deepface tagging on real-photo path** until consent + retention schedule are in place. Synthetic faces can keep using deepface; real faces cannot.
2. **Quarantine `data/headshots/manifest.jsonl` for real photos.** Real-photo classifications must NOT land here. Either move to a dedicated `data/biometric/` tree with separate retention or — preferable — don't extract demographic features from real photos at all.
3. **Replace the name → ethnicity lookup tables with a non-classifying default.** The current logic exists to pick "representative" photos. For real photos, the candidate's own photo IS representative — no inference needed. The lookup tables become dead code.
4. **Upload path needs consent gate.** Whatever endpoint accepts real photos must check a stored consent timestamp + version before storing the file.
5. **Retention schedule documented + auto-enforced.** Crontab / ingestd background sweep that deletes biometric data per the schedule.
These five gates are NOT in `AUDIT_TRAIL_PRD.md` §8 phase plan. **Recommend adding as Phase 1.6 (BIPA pre-launch gates)** ahead of any real-photo intake.
---
## §2 — outcomes.jsonl content sample
### 2A — File state
| Property | Value |
|---|---|
| Rows | 101 |
| Path | `data/_kb/outcomes.jsonl` |
| Rows with `name` field in fills | **39** |
| Rows with `first_name` field | 0 (the field is `name`, not `first_name` — names are pre-concatenated in this sink) |
| Sample names in fills (synthetic but representative) | `Alex Rivera`, `Carmen I. Garcia`, `Jacob N. Patel`, `Jamal Z. Jones`, `James Park`, `Maria Chen`, `Mei Y. Cooper`, `Peter E. Carter`, `Sam Torres` |
### 2B — Confirmed PII shapes in the file today
- `fills[*].name` — full candidate name (synthetic but real shape)
- `operation` — natural-language fill request (e.g. "fill: Welder x2 in Toledo, OH"). Currently doesn't carry candidate names in the operation string itself, BUT the `error` field can — model reasoning that names specific candidates lands here
- `error` — model self-correction text and validator failure messages. Sampled rows contain phrases like "executor parse failure on turn 8: no JSON object in Executor response: ..." which is technical content. Real-data scenarios where the model reasons about specific candidates would put names here.
### 2C — Discrimination-proxy phrase audit (gemini scrum flag §10/single-reviewer)
Searched for common LLM-generated discrimination proxy phrases. **0 hits in current synthetic-data sample**:
- "culture fit" — 0
- "communication" — 0
- "team chemistry" — 0
- "personality" — 0
- "soft skills" — 0
- "work ethic" — 0
- "professional attitude" — 0
**Interpretation:** the current synthetic data doesn't generate these phrases. However:
- The file is structurally capable of holding them (it persists model `error` text)
- When real models (kimi-k2.6, deepseek-v3.2 per PR #13) reason about real candidates, they WILL generate phrases like these — these are common in HR/staffing LLM output
- Once a phrase like "rejected as not a culture fit" lands in outcomes.jsonl, it's append-only persistence. Cryptographic erasure is the only RTBF path.
**Action:** add a content-redaction pass to the outcomes.jsonl writer BEFORE production data lands. Strip discrimination-proxy phrases at write time. (This is not a substitute for the model not GENERATING them — it's a defense-in-depth layer.)
### 2D — Implications for the audit trail
The outcomes.jsonl sample confirms what `AUDIT_PHASE_1_DISCOVERY.md` §1D predicted: **39 of 101 rows persist candidate names in the `fills[*].name` field today**. The shape is real even if the data is synthetic.
For the Phase 2 identity service design:
1. **The training-safe export pipeline (per J 2026-05-03 answer 11) MUST address outcomes.jsonl directly.** This file is the most-likely candidate for RAG re-indexing or fine-tuning corpus, and it carries names.
2. **Crypto-erasure target list** must include `data/_kb/outcomes.jsonl`, `data/_kb/overseer_corrections.jsonl`, `/tmp/lakehouse-validator/sessions.jsonl`, plus pathway memory state, plus Langfuse storage.
3. **Subject_id top-level promotion (per AUDIT_PHASE_1_DISCOVERY §10/C5)** can replace `fills[*].name` as the queryable field. The name itself can move into a `[REDACTED]` token until legal access dereferences via the identity service.
---
## §3 — Synthesis: net new findings for Phase 2 design
| Finding | Source | Phase 2 design implication |
|---|---|---|
| **Phase 1.6 (NEW) — BIPA pre-launch gates needed before real photos accepted** | §1E above | Phase 2 identity service design must include biometric-info table with consent + retention metadata schema. |
| **Name-based ethnicity inference is a separate Title VII / disparate-impact risk** | §1B above | Phase 2 must include a "no protected-attribute inference at routing time" architectural rule. The lookup tables should be deprecated as part of the identity-service migration. |
| **deepface library use itself triggers BIPA when real photos arrive** | §1D point 5 | Phase 2 design must specify "biometric processing requires consent token presence" as a gateway boundary check. |
| **outcomes.jsonl persists names today (39/101 rows)** | §2A above | Phase 2 needs subject_id top-level promotion + name field as `[REDACTED-{token}]` reference (token resolves via identity service for authorized callers). |
| **No discrimination-proxy phrases in current synthetic data, but the sink is shape-ready for them** | §2C above | Phase 2 should specify a redaction-on-write pass for outcomes.jsonl + sessions.jsonl writers. |
| **manifest.jsonl is tracked in git today** | §1C above | If real photos arrive without changing the manifest path, biometric data lands in version control — must be addressed as part of Phase 1.6 gates. |
---
## §4 — Open questions surfaced by Phase 1.5
1. **Where will real photos enter the system?** No upload endpoint walked. Need to know the entry point before designing the consent gate.
2. **Will the deepface tagging path be preserved for real photos, or stripped?** Architectural decision — affects whether biometric extraction is in scope at all.
3. **What's the staffing client's consent UX?** Does the candidate sign consent during onboarding, separately for biometrics, etc.? Affects the consent-token design.
4. **How long do BIPA records need to persist?** BIPA's "3 years from last interaction" is a maximum, not a target — staffing-co's contract may set a shorter floor (e.g., "delete on placement+1 year"). Need staffing-co policy.
5. **Is there a Designated Privacy Officer / DPO?** Per CCPA/GDPR best practice + BIPA litigation hygiene, this role should exist before real-PII intake. Out of scope for code; flag for J.
---
## §5 — Recommended Phase 1.6 BIPA pre-launch gates (NEW)
Lands as new phase BETWEEN Phase 1.5 (this doc) and Phase 2 (identity service design). Read-only/design work; one code commit (the consent gate). Estimated 4-8 hours.
1. **Write `docs/BIPA_COMPLIANCE_POLICY.md`** — public retention schedule, written consent template, deletion procedure, employee training acknowledgment. Counsel must review before publication.
2. **Add consent gate to whatever endpoint accepts real photos** (not yet built — design with the gate from day 1, don't bolt on after).
3. **Quarantine real-photo classifications to `data/biometric/`** — separate path, separate retention, separate access control. Real-photo `manifest.jsonl` does NOT land in `data/headshots/`.
4. **Deprecate name → ethnicity lookup tables in `mcp-server/search.html`** (commit removing those constants + the call sites). Replace with "use candidate's actual photo" path or non-inferred default.
5. **Add a unit test that asserts `data/headshots/manifest.jsonl` row count matches the SYNTHETIC face count** — fails if real-photo classifications leak into the synthetic manifest.
These five fit between Phase 1.5 and Phase 2 because (a) they're MORE URGENT than Phase 2 (BIPA exposure starts the moment a real photo touches deepface), (b) they're independent of the identity service design, (c) they need counsel sign-off before they can ship anyway.
---
## Change log
- 2026-05-03 — Phase 1.5 BIPA + outcomes content audit complete. New finding: synthetic data + deepface + name-ethnicity inference creates BIPA-shaped surface that is one upload path away from being live. Recommend Phase 1.6 (BIPA pre-launch gates) before Phase 2 design starts.
+295
View File
@@ -0,0 +1,295 @@
# Phase 1 Discovery — Subject + PII Surface Map
**Status:** Draft — 2026-05-03 · **Drafted by:** working session 2026-05-03 · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md)
> **Purpose.** Read-only walk of both runtimes. Fills in the "UNKNOWN" cells in `AUDIT_TRAIL_PRD.md` §3 surface map and §7 current-state-vs-target table with file:line evidence. **No code changes.** Output: a complete picture of where subject identifiers + PII flow today, where they leak, and what an audit response could and could not produce right now.
---
## TL;DR — what the walk found
1. **A defense layer EXISTS but is BYPASSED.** Two PII-masked views (`candidates_safe`, `workers_safe`) live in `data/_catalog/views/` and correctly drop name/email/phone. **No production tool query uses them.** The MCP tool registry's `search_candidates` and `get_candidate` SQL templates query the raw `candidates` / `workers_500k` tables and return full PII to the LLM context. Worth considering: the views are functioning policy artifacts that nobody routes through.
2. **PII flows freely through the LLM substrate.** In a single fill scenario, candidate names + emails + phones traverse: SQL → tool_result → LogEntry → execution_loop log → `/v1/respond` HTTP response → Langfuse trace input + output → `data/_kb/outcomes.jsonl` (fills with name) → `data/_kb/overseer_corrections.jsonl` (operation + correction text). At least 7 distinct persistence/transmission paths per scenario.
3. **`candidate_id` is a stable token but not an isolated one.** It's a column on the same `workers_500k.parquet` that holds name/email/phone — there is no separate identity service. Joining candidate_id back to PII is one DataFusion query away. The audit-trail PRD's §5 identity-service intent is not yet built; the architectural separation does not exist.
4. **No `/audit/subject/{id}` endpoint exists.** Reconstructing "every decision about person X" today requires manual cross-correlation across 4+ JSONL files + Langfuse + pathway memory + observer events. There is no canonical query path; the audit response cannot be produced in any reasonable time today.
5. **Go side mirrors the Rust pattern.** Go validator's `rosterRow` carries Name; Go SessionRecord carries `Prompt` (truncated to 4000 chars) which contains the natural-language operation including candidate names. Cross-runtime parity in the PII-leak too.
6. **Append-only persistence is universal.** `outcomes.jsonl`, `overseer_corrections.jsonl`, `pathway_memory/state.json`, `sessions.jsonl`, Langfuse — all append-only. Right-to-be-forgotten under the current architecture requires the cryptographic-erasure approach from PRD §6 because *no* hot data store supports per-subject deletion today.
7. **The matrix-indexer fingerprint is subject-agnostic; trace bodies are UNVERIFIED.** `pathway_memory::PathwayTrace` fingerprints are keyed by `task_class + file_prefix + signal_class` — none of those are subject identifiers, which is structurally defensive. **However:** trace bodies (`reducer_summary`, `final_verdict`) are written from execution-loop output and are highly likely to leak PII. Per §8.1 these are unverified. Treat the matrix indexer as a SUSPECTED PII SINK until sampled — do NOT rely on "matrix can't drive discrimination" framing. (Walked back from earlier draft per cross-lineage scrum §10/C1: that claim was over-stated in 3/3 reviews.)
---
## §1 — PII flow paths in the Rust stack (file:line evidence)
### 1A — MCP tools query raw tables (PII enters the LLM context)
| File:line | What | PII columns returned |
|---|---|---|
| `crates/gateway/src/tools/registry.rs:96` | `search_candidates` SQL template | `first_name, last_name, phone, email, city, state` |
| `crates/gateway/src/tools/registry.rs:107` | `get_candidate` SQL template | `SELECT *` — returns ALL columns including hourly_rate_usd, address, anything in the table |
| `crates/gateway/src/tools/registry.rs:129` | `top_recruiters` SQL | recruiter names (employee PII, not candidate but still PII) |
| `crates/gateway/src/tools/registry.rs:141` | `engaged_unplaced_candidates` SQL | `c.candidate_id, c.first_name, c.last_name, c.phone, c.vertical` |
| `mcp-server/index.ts:1722, 2173, 2269` | address concatenation (`street_number + street_direction + street_name`) | full street addresses surfaced to MCP UI clients |
**Architectural gap:** the MCP tool registry has no "view-only" mode. Adding `_safe` view routing would be one `sql_template` rewrite per tool. **Today: every fill scenario invokes search_candidates → LLM sees full PII for every candidate in the result set.**
### 1B — LLM tool results captured in execution log
| File:line | What |
|---|---|
| `crates/gateway/src/execution_loop/mod.rs:30-39` | `LogEntry` struct: `{turn, role, model, kind, content: Value, at}`. `content` is the raw tool_result body (carries PII when kind=`tool_result`). |
| `crates/gateway/src/execution_loop/mod.rs:217` | `self.append(LogEntry::new(..., "tool_result", trimmed))` — every tool result appended to `self.log` |
| `crates/gateway/src/execution_loop/mod.rs:1319` | per-render cap: `kind == "tool_result"` gets 1200 char cap (vs 200 for other kinds). Trimming is for token economy, **not PII redaction** — names/emails fit in 1200 chars. |
| `crates/gateway/src/execution_loop/mod.rs:30-50` | `Fill` struct (in same file): `{candidate_id: String, name: String, reason: Option<String>}`. The `name` field is load-bearing per the PRD-defined FillProposal artifact contract. |
### 1C — Execution log returned to HTTP caller
| File:line | What |
|---|---|
| `crates/gateway/src/v1/respond.rs:111-119` | `RespondResponse { status, artifact, log: outcome.into_log(), iterations, error }` — full log array returned in JSON response body to whoever called `POST /v1/respond` |
| `crates/gateway/src/execution_loop/mod.rs:82-93` | `RespondOutcome::{Ok,Failed,Blocked}` all carry `log: Vec<LogEntry>` — no path drops the log on its way to the caller |
**Audit implication:** any HTTP client calling `/v1/respond` today receives full PII in the response. Authentication on `/v1/respond` is the only access control. Authorization is binary (either you can call it or you can't); there is no row-level filtering of which candidates you're allowed to see in the response.
### 1D — Persisted JSONL sinks on disk
| File:line | Path written | PII shape |
|---|---|---|
| `crates/gateway/src/execution_loop/mod.rs:508` | `data/_kb/outcomes.jsonl` (append) | `{operation, fills: [{candidate_id, name, reason?}], error, ...}` — observed live: `operation` carries natural-language fill request including role + city + state; `fills` carries the full Fill struct with name. |
| `crates/gateway/src/execution_loop/mod.rs:701` | `data/_kb/overseer_corrections.jsonl` (append) | `{operation, correction, sig_hash, ...}` — `correction` is the overseer's free-text guidance which often references specific candidates by name (e.g. "the executor picked Emily Garcia who is at fill capacity, try Maria Rodriguez instead") |
| `mcp-server/observer.ts:138-139` | `data/_observer/ops.jsonl` (append) | observer event log; PII content depends on what the operation context included. **Empty on this box right now** (file doesn't exist), but the writer is wired. |
| `mcp-server/observer.ts:161` | `data/_kb/observer_escalations.jsonl` (append) | sampled live: technical analysis fields (no PII observed in current rows), but `analysis` field is free-form LLM output and could include names depending on the escalation trigger. |
| `crates/gateway/src/v1/session_log.rs` (writer) → `/tmp/lakehouse-validator/sessions.jsonl` | per `lakehouse.toml` `[gateway].session_log_path` | `SessionRecord { ..., prompt, attempts: [{... raw}], artifact }` — **`prompt` carries the operation text (PII), `raw` carries each model attempt's raw text (PII when model reasoned about specific candidates), `artifact` carries the final FillProposal `{candidate_id, name}` shape on success.** |
### 1E — Langfuse external persistence
| File:line | What gets sent |
|---|---|
| `crates/gateway/src/v1/langfuse_trace.rs:166` | `body.input.messages = ev.input` — full message array (system + user + assistant + tool messages). Tool messages contain raw PII for tool_result entries. |
| `crates/gateway/src/v1/langfuse_trace.rs:191` | `body.output = ev.output` — full model generation. When the model emits a Fill or reasons about candidates, names appear here. |
Langfuse runs at `:3001` per memory, with credentials in `/etc/lakehouse/langfuse.env`. The Langfuse storage tier holds these traces. **PII leaves the lakehouse infrastructure boundary at this point** — Langfuse's storage layer (Postgres + ClickHouse, default config) holds it.
### 1F — Pathway memory
| File:line | What |
|---|---|
| `crates/vectord/src/pathway_memory.rs:169` | `PathwayTrace` struct definition |
| `data/_pathway_memory/state.json` | Persisted state — **need read** to confirm whether real names appear in traces. |
Pathway traces are keyed by `pathway_id = SHA256(task_class + file_prefix + signal_class)` — none of which are subject identifiers. **The fingerprint is structurally subject-agnostic.** Whether the trace BODY (kb_chunks, observer_signals, reducer_summary, final_verdict) leaks PII depends on what those fields contain at write time. From the struct shape: `reducer_summary` and `final_verdict` are strings written from execution loop output — **highly likely to leak PII** when summarizing fill outcomes. Need to confirm against live state.json.
---
## §2 — Defense layer that exists but is bypassed
`data/_catalog/views/candidates_safe.json`:
- Drops: `last_name, email, phone, hourly_rate_usd`
- Masks: `candidate_id` (keep first 3, last 2 chars — `CAND-...01` shape)
- Row filter: `status != 'blocked'`
- Visibility intent (per `description`): "Visible to recruiter / mode-runner agents."
`data/_catalog/views/workers_safe.json`:
- Drops: `name, email, phone, zip, communications, resume_text`
- Reason given: "resume_text + communications carry verbatim PII (full names) and there's no in-view text scrubber, so they're dropped wholesale"
- Source for the rebuilt `workers_500k_v9` vector corpus
**The `_safe` views are the right policy artifact.** The ARCHITECTURAL gap is at the SQL-template layer in `crates/gateway/src/tools/registry.rs` — the LLM-facing tools query the RAW tables, not the SAFE views. Three out of three candidate-touching tool templates (search_candidates, get_candidate, engaged_unplaced_candidates) bypass the safe view.
There is no enforcement preventing this. There is no test that asserts "tool SQL must reference only `_safe` tables." There is no warning logged when raw tables are queried via tool surface.
**Quick fix shape (NOT a phase 1 deliverable, just noting the change shape):** rewrite the tool sql_templates to `FROM candidates_safe` (or `FROM workers_safe`); add a build-time check that `crates/gateway/src/tools/registry.rs` only references `*_safe` tables; add a runtime gate in queryd that refuses LLM-attributed queries on raw tables. These would land in Phase 4 (subject tagging across substrates) or possibly Phase 1.5 (defense-layer enforcement) per the AUDIT_TRAIL_PRD.
---
## §3 — Identity / candidate_id provenance
| Question | Finding |
|---|---|
| Is `candidate_id` a stable token? | Yes — observed format `CAND-NNNNNN` (e.g. `CAND-000001`). Stable across the table. Used as join key in the schema (per `crates/ui/src/main.rs:160` cross-table join hints). |
| Is the candidate_id ↔ PII mapping in a separate service? | **No.** Both live in `data/datasets/workers_500k.parquet`. A SQL `SELECT name, email FROM candidates WHERE candidate_id = '...'` resolves the mapping in one query. |
| Is the mapping itself audited? | **No.** No log records "who looked up the PII for which candidate when." This is the identity-service gap from PRD §5. |
| Does anything ever generate a *different* token (UUID, opaque hash) instead of using candidate_id directly? | **No** — every tool, every validator, every persistence sink uses candidate_id as-is. |
**Production-ready implication:** the staffing client's lawyer asks "show me the access log for candidate X's PII" → we cannot produce one. Every access happens via SQL against `workers_500k`; no row-level access log exists.
---
## §4 — Go-side parity with the leak surface
| Rust file:line | Equivalent Go file:line | Same shape? |
|---|---|---|
| `crates/validator/src/lib.rs` (FillProposal carries `{candidate_id, name}`) | `internal/validator/fill.go` + `internal/validator/lookup_jsonl.go:23-31` (`rosterRow` carries `CandidateID, Name, Status, City, State, Role, BlacklistedClients`) | Yes — same PII shape |
| `crates/gateway/src/v1/session_log.rs` SessionRecord with prompt/artifact | `internal/validator/session_log.go` SessionRecord with `Prompt string`, `Attempts[].Raw string`, `Artifact map[string]any` | Yes — same shape |
| `outcomes.jsonl` / `overseer_corrections.jsonl` writers | Not present on Go side (Go validatord doesn't currently run the execution_loop with tool dispatch — Phase 4 of Go rewrite is what wires that) | **Asymmetric** — Go side writes less but only because feature parity isn't done |
| MCP tools registry | Not on Go side yet (mcp-server is still Bun) | **Bun is the surface** for tool dispatch in both runtimes today |
**Cross-runtime audit implication:** even though our 5 parity probes (validator/extract_json/session_log/materializer/embed) all pass 32/32, **none of them assert PII handling**. A new `pii_parity.sh` probe would feed identical PII-tagged input through both runtimes and assert identical redaction behavior. As of today, neither runtime *does* redaction at the substrate level, so the probe would just confirm "both leak identically."
---
## §5 — Mapping back to AUDIT_TRAIL_PRD §3 surface map
Updated cells with file:line evidence:
| Decision happens at | Currently logged where | Audit-completeness gap (revised) |
|---|---|---|
| Ingestion (candidate added to pool) | `data/datasets/workers_*.parquet` rows; ingestd writes via `crates/ingestd/`; no per-subject "added at" event journal found | **GAP** — no subject-tagged ingest event. When + who + how a candidate entered is not auditable per-subject. |
| Embedding creation | `crates/aibridge/src/client.rs` (post-sidecar-drop direct path); LRU cache `crates/aibridge/src/cache.rs` (commit `150cc3b`); cached entries are keyed by `(model, text)` — **the text is the cached body, which for candidate embeddings IS PII (candidate name + role + skills appear in source text per workers_500k_v9 build SQL).** Cache itself is in-memory, not persisted, but text-as-key means PII is in process memory. | **MAJOR GAP** — no per-embedding audit row. Cache key contains PII. No subject tagging means we cannot answer "what embedding was generated for candidate X." |
| Search inclusion | Tool result entries in execution log + Langfuse + outcomes.jsonl `fills` field | Partial — fills are logged, but you have to grep across outcomes.jsonl + Langfuse for any mention of a candidate_id. No "all search results that included X" canonical view. |
| Search rank | Result set in chat traces (Langfuse), not indexed by candidate | Partial — Langfuse trace has the result set; no inversion (candidate → ranks-received). |
| Fill recommendation | `outcomes.jsonl` `fills` array with name + candidate_id | Partial — present, but mixed with PII in the natural-language `operation` field. |
| Validation outcome | `sessions.jsonl` per `[gateway].session_log_path` config; per-attempt `verdict_kind`, `error` | Partial — works per-session, not per-subject. To find "all validations that touched candidate X" you'd need to grep the JSONL by candidate_id. |
| Iterate retry escalations | Same sessions.jsonl `attempts[]` array | Same as above |
| Observer signals | `data/_observer/ops.jsonl` (file not present today on this box, writer is wired) | UNKNOWN content shape until the writer fires for a fill scenario — needs verification |
| Matrix-indexer compounding | `data/_pathway_memory/state.json` — fingerprinted by code, NOT by subject | **NO subject leak in fingerprint structure** (good for audit defensibility). Trace bodies (`reducer_summary`, `final_verdict`) MAY carry PII — needs sampling to confirm. |
---
## §6 — Mapping back to AUDIT_TRAIL_PRD §7 current-state-vs-target gap table
| Capability | Phase 1 finding | Status |
|---|---|---|
| candidate_id as canonical token | Stable `CAND-NNNNNN` format. Same column lives in same parquet as PII. | Token exists; isolation does not. |
| Identity service | Doesn't exist. PII + candidate_id co-located in `workers_500k.parquet`. | Real gap — needs new service. |
| `/audit/subject/{id}` endpoint | Doesn't exist. No audit route in `crates/gateway/src/v1/mod.rs` route listing. | Real gap — needs new endpoint. |
| Subject-tagged embeddings | LRU cache `(model, text)` keys; text contains PII; no audit row per embed. | Real gap. |
| Subject-tagged search results | Langfuse trace contains result set; not subject-indexed. | Real gap (queryability, not capture). |
| Subject-tagged validation outcomes | Yes — sessions.jsonl carries candidate_id (in the attempt's `raw` field) but not as a queryable top-level field. | Partial — needs subject_id top-level promotion. |
| Subject-tagged matrix indexer entries | Pathway fingerprints are subject-agnostic by design. Trace bodies may leak. | Decision needed (PRD open question 7) — keep code-only OR risk PII surface. |
| Protected-attribute filter at decision time | **Not enforced anywhere.** SQL templates return whatever columns they SELECT; no protected-attribute removal at the gateway boundary. The `candidates` table schema in the demo SQL includes age (years_experience is a proxy), and the call_log/email_log tables likely contain free-text correspondence. | Real gap — major. Requires both schema audit + boundary enforcement. |
| Retention policy | None enforced. Append-only files grow indefinitely. | Real gap. |
| Right to be forgotten | Not implementable today on append-only logs without cryptographic erasure. | Per PRD §6 design, real engineering needed. |
| Cross-runtime parity | 5 algorithm probes pass; 0 audit/PII probes exist. | Probe set needs extension. |
---
## §7 — Worked example: John Martinez audit trail TODAY (negative result)
If John Martinez (candidate_id `CAND-042195`) requests an audit:
1. **Find his candidate_id.** Manual SQL: `SELECT candidate_id FROM candidates WHERE first_name='John' AND last_name='Martinez'` — returns 1+ rows. Already a leak: someone with SQL access can correlate name → candidate_id ad-hoc.
2. **Find every fill scenario that included him.** `grep "CAND-042195" data/_kb/outcomes.jsonl` — returns rows where his fill was included. But the row's natural-language `operation` field may NOT contain his candidate_id (it's the fill request, e.g. "fill: Welder x2 in Toledo, OH"), so we'd miss scenarios where he was a *candidate* but not a *fill*. To find those, we'd need to grep Langfuse traces (off-box) or the per-tool result content in execution logs (which aren't persisted as separate JSONL — they live in the HTTP response that already left the building).
3. **Find every validation that touched him.** Grep `/tmp/lakehouse-validator/sessions.jsonl` for `CAND-042195` — would catch FillValidator phantom-ID rejections AND successful fills, but the candidate_id would appear in either the prompt or the raw attempt text, not as a top-level queryable field.
4. **Find every embedding generated for him.** Cache is in-memory; nothing persisted. Cannot answer.
5. **Find every search result that ranked him.** Off-box in Langfuse. Untrieable in lakehouse without a separate Langfuse query pipeline (which doesn't exist).
6. **Find pathway memory traces involving him.** Pathway fingerprints don't carry his ID. The `reducer_summary` strings might mention him (need to grep state.json) but the fingerprint search wouldn't surface them.
7. **Show what protected attributes were exposed to the model.** No record of input_features per decision — the LLM saw whatever the SQL returned, and we have no per-decision input_features audit row.
8. **Format the output for legal.** Even if we collected all the above, there's no signing, no integrity hash, no schema, no template.
**Estimated time to produce a complete-and-defensible response today: not possible.** Estimated time to produce an INCOMPLETE response by cobbling JSONLs + Langfuse exports: 2-5 hours per request, manual, error-prone, and the response would over-share (other candidates appearing in the same fill scenarios) AND under-share (embedding events, search rankings, pathway traces missed).
This is the Phase 1 result the AUDIT_TRAIL_PRD predicted: today's substrate is not production-ready for a discrimination-defense audit response.
---
## §8 — What this discovery DID NOT cover
Phase 1 was scoped to file:line evidence + sampling of live JSONL state. The following deserve their own subsequent walks before phase 2+ design:
1. **Live sample of `data/_pathway_memory/state.json`** — confirm whether `reducer_summary` / `final_verdict` strings actually leak PII or stay generic. Read 3-5 traces and grep for names from `workers_500k`.
2. **Live sample of Langfuse traces** — confirm input message PII for a real fill scenario from past 7 days. Use Langfuse `:3001` query API.
3. **observerd event content** — the writer is wired but `data/_observer/ops.jsonl` doesn't exist on this box. Trigger a fill scenario and inspect the resulting event.
4. **Bun mcp-server tool dispatch** — does the Bun server log tool calls anywhere? `mcp-server/index.ts` is 2900+ lines; partial walk only here.
5. **bot/propose.ts** — the bot proposal flow likely touches candidates; not walked in this pass.
6. **`crates/journald` mutation log** — designed for row-level mutations per ADR-012; haven't confirmed whether candidate-table mutations land here with PII.
7. **Go side observerd + chatd PII surface** — the Go cmd/* binaries likely have analogous logging; confirmed validator parity but didn't walk observer/chat logging.
8. **Process memory + crash dumps** — if the gateway dumps core, what PII is in it? Out of scope for code walk; comes up in security audit.
9. **Operator runbook** — who has access to logs/, /tmp/lakehouse-validator/, MinIO buckets, Langfuse Postgres? Out of scope for code walk; comes up in operational security review.
These are listed so the next phase doesn't accidentally re-walk what's done OR skip what wasn't covered.
---
## §9 — Recommended next moves (not commitments)
Per AUDIT_TRAIL_PRD §8, Phase 2 is the identity service design doc. Before that doc gets written, the cheapest high-value moves discovered here:
1. **Defense-layer enforcement (1-2 hours work).** Rewrite the 3 tool SQL templates in `crates/gateway/src/tools/registry.rs` to use `_safe` views. Add a unit test that asserts no `crates/gateway/src/tools/registry.rs` template references `FROM candidates ` or `FROM workers_500k ` (only `*_safe`). This is one commit. It prevents the most-trafficked PII leak path TODAY without waiting on the identity service. **Cost:** the LLM sees masked candidate_ids (`CAN...95` instead of `CAND-042195`); some downstream tools (validator existence check) would need a "resolve to full ID" path that goes through the identity service — but that's exactly the architectural shape PRD §5 wants anyway.
2. **Sample state.json + Langfuse before phase 2 design.** §8.1 + §8.2 above. ~30 minutes. Either confirms or refutes the "matrix indexer is subject-clean" finding from §1F.
3. **Document the Bun mcp-server tool surface.** §8.4. The Bun layer is a major PII transit point not fully covered here.
4. **Identify whether protected attributes (age proxies, photo features, zip-code → race correlations) are currently in any tool-returned column.** Schema-level audit of `candidates` + `workers_500k`. ~1 hour. Might surface that some "neutral" columns are actually protected-attribute proxies.
These four moves give the phase-2 design doc strong evidence to lean on. None are commitments — J's call on what to do next.
---
## §10 — Cross-lineage scrum review of this discovery doc (2026-05-03)
After the discovery walk, this document was reviewed by three independent model lineages via `/v1/chat` against the Go gateway (post-PR-#13 model fleet): **opus** (`opencode/claude-opus-4-7`), **kimi** (`ollama_cloud/kimi-k2.6`), **gemini** (`ollama_cloud/gemini-3-flash-preview`). Custom prompt: senior security architect reviewing a discovery report. (DeepSeek-v3.2 timed out; not included.)
Verbatim reviews saved at `/tmp/audit_scrum/{opus,kimi,gemini}_review.md`. Convergent findings (≥2 reviewers) are treated as high signal per `feedback_cross_lineage_review.md`.
### Convergent findings — must address before phase 2 design
**C1. §1F matrix-indexer claim is OVER-CLAIMED (3/3 reviewers).** The TL;DR #7 line "good for audit defensibility (matrix index can't drive discrimination)" overstates structural subject-agnosticism as behavioral fairness. Per opus: "until §8.1 is executed, the correct framing is 'fingerprint structurally subject-agnostic; body content unverified — treat as suspected PII sink.'" Per kimi: dangerous reasoning — if `reducer_summary` says "candidates named Emily Garcia were rejected for fill capacity," the matrix learns proxy variables and future similar names get downranked = temporal discrimination risk. Per gemini: "you cannot claim audit defensibility until you prove the *content* of the matrix indexer doesn't contain PII." **Action:** walk back the §1F TL;DR claim; reframe as "fingerprint structure is defensive; body content unverified — treat as suspected PII sink until §8.1 confirms."
**C2. §1E (Langfuse) is the MOST DANGEROUS leak — fix FIRST (3/3 reviewers).** Opus: "boundary-crossing leak that makes a regulator's eyes light up." Kimi: "Article 44 GDPR transfer if SaaS-hosted; CPRA 'sale/sharing' question; subprocessor notification failure." Gemini: "un-certifiable for SOC2 Type II" + "unauthorized data transfer to third-party storage tier." All three would do Langfuse redaction/sampling BEFORE the §9.1 view-routing fix. **Action:** revise §9 priority order — Langfuse boundary first, view-routing second.
**C3. Discrimination defense requires the FULL CANDIDATE POOL, not just fills (3/3 reviewers).** Opus: "you need not just 'what did we do for candidate X' but 'what was the selection rate for protected class Y vs comparator' — that requires capturing protected attributes WITH outcomes." Kimi: "EEOC Uniform Guidelines on Employee Selection Procedures (1978) — matrix index that learns from historical fill outcomes IS a selection procedure under the Guidelines." Gemini: "The system doesn't log the *entire* candidate pool for a specific search — only the fills (§5). To defend a lawsuit, you must show the stats for everyone who *could* have been picked, not just the person who was." **Action:** Phase 1 didn't capture this load-bearing requirement. PRD §1 worked example needs expanding: the audit response must include the comparator pool + adverse-impact statistics, not just the subject's decision row.
**C4. BIPA + biometric exposure UNDERSTATED (3/3 reviewers).** Already in `AUDIT_TRAIL_PRD.md` §10.5 jurisdictional checklist but NOT translated into Phase 1 findings actionables. If `workers_500k` columns include photo paths, video interview references, or anything that could yield biometric inference (per gemini: "even descriptors that could be reconstructed into biometric templates"), BIPA's $1k-$5k per-violation regime applies BEFORE the GDPR analysis matters. **Action:** add to §8 (what discovery did NOT cover): explicit photo/video/biometric column audit of `workers_500k` schema.
**C5. `candidate_id` must be PROMOTED to top-level field in all JSONL sinks (3/3 reviewers).** Opus + kimi + gemini converge: grepping natural-language strings (operation, raw, prompt, reducer_summary) for candidate_id is not a defensible audit strategy. Even if subject_id appears in those strings TODAY, it appears co-mingled with other candidate names, model reasoning, etc. — making subject filtering unsafe. **Action:** add to PRD §7 target column "subject_id top-level promotion" — change session_log writer + outcomes.jsonl writer + observer event writer to ALL include `subject_id` (or `subject_ids[]`) as a first-class top-level field.
### Single-reviewer findings — verified, worth incorporating
**OPUS unique:**
- **8th PII path missing: LLM provider egress.** PR #13 routes models through opencode + ollama_cloud + openrouter — opencode and openrouter are external services. Cross-border data transfer under GDPR Ch. V; third-party processor relationship requiring DPA under Art. 28. Phase 1 did not enumerate this path. **Action:** add §1G "LLM provider egress" to §1.
- **GDPR Art. 22 / EU AI Act right to explanation.** Audit must capture the model's REASONING CHAIN, not just decision output. Phase 1 §7 worked example doesn't include this. **Action:** add to subject-audit response shape (PRD §2).
- **Special-category data under GDPR Art. 9.** resume_text + call_log + email_log routinely contain health (accommodation requests), union, religion. Higher legal bar — Art. 9(2) explicit consent required. Phase 1 §6 mentions these tables exist but doesn't flag the special-category exposure. **Action:** call out in §6.
- **DPIA / ROPA / DPA inventory.** None of these documents referenced. Some may exist outside code; Phase 1 should at minimum note their absence as Phase 1.5 input.
**KIMI unique:**
- **Sequential `CAND-NNNNNN` IDs enable enumeration attacks.** Predictable IDs let an attacker scan the candidate space. Security finding Phase 1 didn't flag.
- **Langfuse retention config unaudited.** Default trace retention is 30 days in some versions, indefinite in others. Directly impacts RTBF analysis. **Action:** check live Langfuse config.
- **CCPA "de-identified in place" may be faster than crypto-shred.** Replacing PII with `REDACTED-{hash}` while preserving log structure may satisfy CPRA's de-identified exception. Worth considering vs. crypto-shred.
- **Bun MCP server is the cross-runtime bridge — likely COMMON-MODE failure.** Phase 1 framed Go side as "mirrors Rust pattern" implying independent failure; it's likely shared infrastructure failure. **Action:** add to §4 — common-mode reframe.
- **Cryptographic integrity for the audit trail itself.** Merkle trees / signed logs / chain-of-custody under FRE 901. Opposing expert can challenge admissibility without this. **Action:** add to PRD §2 audit response shape — integrity-signed.
- **HIPAA Business Associate Agreement scoping not done.** If any candidate is healthcare-vertical, BAA analysis required.
- **`get_candidate` SELECT * has 40+ load-bearing columns.** §9.1 "1-2 hours" estimate for view rewrite is irresponsible without scoping downstream consumers. **Action:** revise §9.1 estimate; flag dependency analysis as prerequisite.
**GEMINI unique:**
- **Data residency — JSONL on US box.** If any candidate is EU resident, GDPR violation without SCC/DPF. Phase 1 didn't ask whether IL+IN target market includes EU residents (probably not, but staffing-co clients sometimes have international placements).
- **"Culture fit" reasoning strings as discrimination proxies.** Common LLM-generated phrases ("not a culture fit," "communication concerns," "team chemistry") often correlate with protected-attribute discrimination. Phase 1 didn't audit the actual reasoning text in `outcomes.jsonl` for these phrases.
- **Comparator-pool snapshot for every fill.** Need to capture WHO COULD HAVE BEEN PICKED, not just who was. **Action:** PRD §2 audit response shape needs `comparator_pool` field per decision.
### Revised §9 — recommended next moves (reordered by scrum convergence)
1. **(NEW PRIORITY 1, was P3) Sample state.json + Langfuse content** — confirm/refute the matrix-indexer subject-clean claim and quantify Langfuse PII exposure. Cheapest move that resolves the over-claim AND informs the boundary-leak fix.
2. **(NEW PRIORITY 2, was P1) Defense-layer enforcement at SQL template level** — rewrite tool registry to use `_safe` views. **Estimate revised UPWARD per kimi**: scope `get_candidate` SELECT * downstream consumers first; estimate 4-8 hours including the existence-check resolution path through the (not-yet-built) identity service. Stop-gap until then: add LLM-attribution flag to queryd, refuse `FROM candidates `/`FROM workers_500k ` queries that originate from tool dispatch.
3. **(NEW) Langfuse boundary audit + redaction** — sample retention config, check DPA status, scope a redaction/sampling layer that strips PII from message arrays before the Langfuse POST. This is the boundary-crossing leak — fix BEFORE chasing internal sinks.
4. **(NEW) Subject_id top-level promotion to all JSONL writers** — single architectural change spanning Rust + Go session_log + observerd event writer + outcomes/corrections appenders. Makes subject-correlation queries defensible (no more grepping natural language strings).
5. **(was P4) Schema-audit for protected-attribute proxies** — extend to include "culture fit"-shaped reasoning text in outcomes.jsonl + the comparator-pool retention requirement.
6. **(NEW) BIPA-specific audit of workers_500k schema** — explicit photo/video/biometric column inventory before any production deployment in IL.
7. **(NEW) Operational discovery** — DPIA, ROPA, DPA inventory, SCC for cross-border, Langfuse retention config. Out-of-code-walk; needs J + counsel input.
### What I'm walking back
§1F TL;DR claim "matrix indexer is good for audit defensibility" — per all 3 reviewers, this is over-claimed without §8.1 verification. The correct frame is "fingerprint structure is subject-agnostic by design; trace body content unverified — treat as suspected PII sink until sampled."
§9 ordering — view-routing was P1; per all 3 reviewers, Langfuse boundary should be P3 in front of it. View-routing is the source-side fix; Langfuse is the boundary-crossing fix; both matter, do them in BOUNDARY-FIRST order.
§9.1 effort estimate — kimi's "irresponsible without dependency scoping" critique is right. Revised UP to 4-8 hours.
---
## Change log
- 2026-05-03 — Phase 1 discovery walk complete. Findings cited above with file:line references. No code changes. Companion to `AUDIT_TRAIL_PRD.md`.
- 2026-05-03 — §10 added: cross-lineage scrum review of the discovery doc (opus + kimi + gemini). 5 convergent findings (matrix-indexer over-claim, Langfuse first-priority, comparator pool gap, BIPA understated, subject_id top-level promotion). Plus single-reviewer high-value catches. §9 reordered.
+374
View File
@@ -0,0 +1,374 @@
# PRD: Production-Ready Audit Trail
> **⚠ OVER-SCOPED — 9-phase plan needs to shrink for local-only deployment.**
>
> 2026-05-03 evening: J reframed the system as local-only per PRD line 70. The 9-phase plan in §8 was sized for SaaS-tier infrastructure with cloud HSM, separate identity daemon, dual-control JWT, etc. For a single-box local deployment, audit trail can be a few hundred LOC of local writers + a signed local file, not a 17-20 day phase plan.
>
> **What stays valid:**
> - The legal use case (worked example: John Martinez at Warehouse B requests audit) — this is the real problem
> - The §10.5 jurisdictional surface (IL BIPA, IN, federal) — counsel reads this
> - The §3 surface map: where decisions get made today (file:line evidence — see `AUDIT_PHASE_1_DISCOVERY.md`)
> - Phase 1.6 BIPA pre-launch gates — those still apply when real photos arrive
>
> **What's over-scoped:**
> - The 9-phase implementation plan (§8) — should compress to 3-4 phases for local-only
> - The identity service design (`IDENTITY_SERVICE_DESIGN.md`) — see that doc's deprecation header
>
> Do NOT execute the §8 phase plan as-written. When J greenlights, draft a v2 plan sized for local single-box.
**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation.
---
## 1. The legal use case (worked example)
**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2.
**What we must produce.** A response that proves either:
- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR
- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI."
**What we must NOT produce.**
- Other subjects' data (response leaks if even one other candidate's name appears)
- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped)
- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence
**The defensibility chain.** The audit shows:
1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into
2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank
3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned
4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections)
5. **Outcome decisions** — final fills, rejections, hand-offs
For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale.
---
## 2. The subject audit response — output format
```
GET /audit/subject/{candidate_id}?from=D1&to=D2
→ JSON or signed PDF (legal preference TBD)
```
**Header section:**
- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash
- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls.
**Per-decision row schema** (shape, not exhaustive):
```json
{
"ts": "ISO-8601 UTC",
"decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal",
"daemon": "gateway | validatord | observerd | matrixd | ingestd",
"model": "kimi-k2.6 | deepseek-v3.2 | ...",
"provider": "ollama_cloud | opencode | openrouter",
"input_features": { /* what the model SAW — sanitized per §4 */ },
"output": { /* what the model decided */ },
"rationale": "model's natural-language explanation, or rule-based justification",
"trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree",
"session_id": "iterate session that produced this row"
}
```
**Footer section:**
- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy"
- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited)
---
## 3. Surface map — where decisions happen
| Decision happens at | Currently logged where | Audit-completeness gap |
|---|---|---|
| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk |
| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding |
| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation |
| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial |
| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified |
| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today |
| Iterate retry escalations | Session JSONL `attempts[]` array | OK |
| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk |
| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject |
**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly.
---
## 4. PII handling rules
**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces.
**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it.
**Protected-attribute exclusion at decision time:** the model NEVER receives:
- Race, ethnicity, national origin
- Sex, gender, marital status, pregnancy
- Age, date of birth (allowed: years of experience)
- Religion
- Disability, genetic information
- Veteran status (unless legally relevant for the role)
- Sexual orientation, gender identity
If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input.
**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls.
**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows.
---
## 5. Identity service — candidate_id ↔ PII mapping
**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change.
**Target state:**
- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping
- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id`
- Identity reads require a separate auth credential held by legal-authorized operators
- Every identity read is itself audited (log who accessed PII for which candidate when)
- Identity service runs as its own daemon, port-isolated from the gateway
- Cross-runtime: same identity service backs both Rust and Go
**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary.
---
## 6. Retention policy
**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA.
**Target state (proposed):**
- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s)
- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract.
- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed.
- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable.
**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off.
---
## 7. Current state vs target state
| Capability | Today | Production-ready target | Gap |
|---|---|---|---|
| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service |
| Identity service | none | separate daemon, audited reads | Real — build new |
| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new |
| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument |
| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk |
| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial |
| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major |
| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk |
| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build |
| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build |
| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set |
---
## 8. Implementation phases (proposed sequence)
Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds.
### Phase 1 — Discovery walk (read-only, ~3-4 hours)
Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references.
**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells.
### Phase 2 — Identity service design (design doc, ~2 hours)
Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.**
**Exit:** J approves the design.
### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours)
Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong.
**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued.
### Phase 4 — Subject tagging across substrates
Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror.
**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end).
### Phase 5 — Identity service build
Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly.
**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject.
### Phase 6 — Protected-attribute boundary enforcement
Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing.
**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping.
### Phase 7 — Retention + right-to-be-forgotten
Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint.
**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows.
### Phase 8 — Legal-format export + signing
Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow.
**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies.
### Phase 9 — End-to-end discrimination defense rehearsal
Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them.
**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness.
---
## 9. Cross-runtime requirement
**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5.
The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version).
---
## 10. Open questions blocking phase 2 (resolved 2026-05-03)
**These were the original Phase 1 open questions. J answered the load-bearing 5 in conversation 2026-05-03; answers folded in below. Remaining items 1, 4, 6 still need J's call before Phase 2 design ships.**
1. **Identity service: separate daemon vs in-process?** _Recommended: separate. **Status: pending J confirmation.**_
2. ~~**Retention period N years?**~~ — Out of scope for now; will be set at deployment time per client contract. Default to 4-year hot retention until set.
3. ✅ **Photos/video in scope?** **YES** (J 2026-05-03). BIPA (740 ILCS 14) applies in full. Per-violation $1k-$5k statutory damages, written consent + retention schedule mandatory. **This becomes a Phase 1.5 priority** — see §10.5/§13 for revised phase ordering.
4. **JSON or signed PDF for legal export?** _Recommend signed JSON with PDF rendering option. **Status: pending.**_
5. ✅ **RTBF under append-only:** Cryptographic erasure approach approved in principle (J 2026-05-03 implicit via approval of plan).
6. **Audit endpoint auth model:** _Recommend legal-only credential separate from admin token. **Status: pending.**_
7. ✅ **Matrix indexer subject-awareness:** Per scrum review (`AUDIT_PHASE_1_DISCOVERY.md` §10/C1), matrix-indexer is suspected PII sink (trace bodies unverified). Action: sample state.json before deciding (a) keep code-only + add PII-redact-on-write to trace bodies, OR (b) remove subject-summary text from trace bodies entirely. Decision deferred until §8.1 sampling completes.
### Newly answered 2026-05-03 (J)
8. ✅ **Langfuse hosting model:** **Self-hosted.** Removes the GDPR Art. 44 cross-border-transfer concern that 3/3 scrum reviewers flagged. Langfuse retention config + Postgres/ClickHouse access controls still need to be audited as part of Phase 1.5 — but the boundary stays inside J's infrastructure, which is materially better than SaaS Langfuse.
9. ✅ **EU candidates in scope:** **Not currently — may need placeholder later.** Design choice: build the identity-service interface to be EU-compatible (DPIA-shaped fields, lawful-basis tracking, SCC-ready transfer mechanism slots) but DO NOT gate Phase 2 on EU compliance. Phase 2 ships IL+IN-shaped; EU additions are a follow-up phase.
10. ✅ **Healthcare vertical / HIPAA:** **Same framework — yes.** Healthcare staffing IS in scope. PHI in `resume_text`, `communications`, and `call_log` is realistic. **Implications:**
- Business Associate Agreement (BAA) required with any third-party model provider that processes content from healthcare-vertical staffing requests
- opencode + ollama_cloud + openrouter (per PR #13 routing) are external — BAAs needed OR healthcare requests must route to local-only models (Ollama on-box)
- PHI redaction at the gateway boundary becomes mandatory before the model call leaves the box, OR the model call must stay on-box for healthcare requests
- Vertical detection at the gateway boundary becomes a Phase 2 requirement
11. ✅ **Training / RAG re-runs may use historical outcomes:** **Yes — design as if it WILL.** Implications:
- `outcomes.jsonl` and `overseer_corrections.jsonl` cannot remain raw-PII forever — anything that lands in a training corpus or RAG re-index becomes ungeneratable to delete (PII in model weights)
- Phase 2 design must include a "training-safe export" pipeline that strips PII from outcomes before feeding to any training/RAG path
- Crypto-erasure of historical outcomes becomes load-bearing — if a candidate exercises RTBF and their data already trained a model, we must be able to evidence "the source was destroyed; the model retains it indistinguishably from synthetic patterns"
### Effect on the §8 phase plan
The user-confirmed answers shift priorities. Revised ordering (incorporating scrum-driven priority changes from `AUDIT_PHASE_1_DISCOVERY.md` §10):
- **Phase 1.5 (NEW)** — BIPA-specific photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample. Lands BEFORE Phase 2 design starts.
- **Phase 2 (identity service design)** — now must include EU-placeholder fields, vertical-detection (healthcare flag), training-safe export interface, BIPA consent + retention metadata
- **Phase 3** (audit endpoint skeleton) — unchanged
- **Phase 4** (subject tagging) — must include healthcare-vertical routing decision at gateway boundary
- **Phase 5** (identity service build) — must include BIPA-compliant biometric metadata table
- **Phase 6** (protected-attribute boundary) — must include PHI redaction for healthcare-vertical requests
- **Phase 7** (retention + RTBF) — must include training-safe export evidence chain
- **Phase 8** (legal export) — unchanged
- **Phase 9** (rehearsal) — must include both EEOC discrimination scenario AND BIPA biometric scenario AND healthcare PHI breach scenario
---
## 10.5 Jurisdictional surface (IL + IN)
> **⚠ Not legal advice.** This is a research-grade checklist for J to take into a conversation with actual employment + privacy counsel. The system is targeting **Chicago (Illinois)** and **Indiana** placements per 2026-05-03 conversation. Counsel needs to verify what currently applies, what's pending, and whether case law has moved any of these in 2026. **Verify with counsel before claiming compliance with any item below.**
### Federal layer (always applies)
| Statute / framework | Relevance to this system |
|---|---|
| Title VII (Civil Rights Act) | Bans discrimination on race, color, religion, sex, national origin in hiring. Discrimination claim defense is the worked example in §1. |
| ADEA (Age Discrimination in Employment) | Bans age-based discrimination for workers 40+. DOB must be excluded from features per §4. |
| ADA (Americans with Disabilities Act) | Bans disability discrimination + requires reasonable accommodation. Disability-inferring features (gait, photo features, medical history) need exclusion. |
| EEOC enforcement | Receives complaints, issues right-to-sue. Audit response per §2 is what defends in EEOC investigation. |
| OFCCP | Applies if our staffing client serves federal contractors. Adds affirmative-action recordkeeping on top of EEOC. |
| FCRA (Fair Credit Reporting Act) | Triggers if background checks are performed. Pre-adverse-action notice + dispute process needed. |
| Section 1981 | Race-based contract discrimination — staffing is contract relationship. |
### Illinois-specific (Chicago jurisdiction)
| Statute | What | What we need |
|---|---|---|
| **BIPA** (Biometric Information Privacy Act, 740 ILCS 14) | Bans collection of biometric identifiers (face geometry, fingerprints, voiceprints) without informed written consent + retention schedule. Penalties: $1,000-$5,000 per violation per person. **Class actions are common and aggressive.** | If we use candidate photos for any feature (face match, headshot rendering, photo-derived attributes), BIPA almost certainly applies. The headshot pool we generate (per CLAUDE.md commit `5d93a71` area) needs careful review — synthetic faces are probably OK; real candidate photos are NOT without explicit BIPA-compliant consent. **Counsel must review.** |
| **Illinois AI Video Interview Act** (820 ILCS 42) | If AI analyzes recorded video interviews, employer must disclose AI use, obtain consent, provide explanation of how AI works, and limit who can review the video. | If we ever ingest video, this applies. Currently we don't, but worth flagging to counsel as a "what if we add this in 12 months" boundary. |
| **Illinois Human Rights Act** (775 ILCS 5) | Broader than federal Title VII — adds protected classes including arrest record, military status, marital status, order of protection, citizenship status (in some cases), unfavorable military discharge. | Protected attribute exclusion list in §4 needs expanding to cover IL-specific classes. |
| **Personal Information Protection Act** (PIPA, 815 ILCS 530) | Breach notification — must notify Illinois residents whose unencrypted PII was breached. | If identity service or workers parquet is breached, notification clock starts. Need incident response runbook. |
| **Illinois Day and Temporary Labor Services Act** (820 ILCS 175) | Specific to staffing/temporary services industry. Includes equal-pay-for-equal-work + record-keeping requirements + worker notification. | Highly relevant — applies directly to staffing-company clients. Audit retention may interact with these recordkeeping requirements. |
| **Workplace Transparency Act** | Restrictions on non-disclosure agreements re: harassment/discrimination | Tangential but worth noting. |
| **City of Chicago Human Rights Ordinance** (Title 6 Chicago Municipal Code) | Adds protected classes beyond IHRA (source of income, parental status, military discharge status, credit history). | Chicago-specific protected attributes list. |
| **Cook County Human Rights Ordinance** | Similar additions county-wide. | Chicago is in Cook County so this stacks. |
| **Possible: AI hiring transparency** | Several states/cities have proposed/passed laws modeled on NYC Local Law 144 (annual bias audit + candidate notification). I do not know whether IL or Chicago has such a law on the books as of 2026-01 cutoff. | **Counsel must check current state.** If it exists, we need annual bias audit reports (which IS what this PRD is building toward, but the report format may have specific requirements). |
### Indiana-specific
| Statute | What | What we need |
|---|---|---|
| **Indiana Data Breach Disclosure** (IC 24-4.9) | Breach notification within "without unreasonable delay" | Same incident response runbook as IL PIPA. |
| **Indiana Civil Rights Law** (IC 22-9) | State-level employment discrimination | Largely tracks federal Title VII, fewer expansions than IL. |
| **Indiana Genetic Information Privacy Act** | Bans use of genetic info in employment | Already in §4 protected list. |
| **General observation** | Indiana is generally less aggressive than Illinois on AI/employment regulation as of cutoff. | The IL bar is higher — if we satisfy IL, IN typically follows. **Counsel must confirm this isn't backwards.** |
### Cross-cutting (security frameworks for SaaS sales)
These aren't laws but are commonly required by enterprise customers (including staffing clients) before sale.
| Framework | What | Relevance |
|---|---|---|
| **SOC 2 Type II** | Auditor attestation of operating effectiveness over 6-12 months across Trust Service Criteria (Security, Availability, Processing Integrity, Confidentiality, Privacy). | The Privacy criterion overlaps heavily with this PRD. Privacy + Security are the two load-bearing TSCs. Effort to first Type II report: 6-9 months. Type I (point-in-time) is faster (weeks) but enterprise buyers usually want Type II. |
| **SOC 3** | Public-facing summary of SOC 2 (no detailed control descriptions). | Nice-to-have for marketing but the staffing client will want SOC 2 Type II report under NDA. |
| **HIPAA** | Healthcare data protection. | Triggers ONLY if staffing places workers into healthcare roles where they handle PHI. Currently not in scope per CLAUDE.md. **Confirm scoping with J.** |
| **PCI DSS** | Payment card data | Not currently in scope. |
| **ISO 27001** | International information security management | Alternative to SOC 2; more common in EU. Probably unnecessary for IL/IN-only deployments. |
### What this means for phase ordering
The 9-phase plan in §8 is technically correct but may need re-ordering once counsel weighs in:
- **BIPA risk on photos** is so high and so aggressive that if we use real candidate photos *anywhere*, that may need to be the FIRST thing we resolve — before the audit-trail work starts. Class-action exposure is enormous.
- **SOC 2 Type II prep** runs in parallel with this work, not after. If the staffing client says "show us your SOC 2 report" we need to have started the engagement weeks/months before.
- **Day and Temporary Labor Services Act** may impose recordkeeping that interacts with our retention SLA (§6) — counsel may say "no, retention has to be N years for THIS reason, not your defaulted 4."
### Open questions for counsel (one ask)
1. Does the staffing client have an existing SOC 2 report we leverage, or do we need our own?
2. Are we using any real candidate photos? If yes, is BIPA consent in place?
3. Does Illinois have an AI hiring transparency law on the books in 2026? If yes, what does the bias audit report need to look like?
4. What's the IL Day and Temporary Labor Services Act recordkeeping retention period? Does it interact with our 4-year proposed SLA?
5. Are background checks performed? If yes, do we need FCRA pre-adverse-action workflow integration?
6. Any healthcare placements? (HIPAA scoping)
7. Is the staffing client a federal contractor? (OFCCP scoping)
Counsel's answers shape whether the §8 phase plan ships as-is or needs reordering.
---
## 11. What this PRD is NOT
- Not a contract with the staffing client. That document needs lawyers and signs after this is built.
- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC/BIPA/etc — passing actual certification is its own project.
- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made.
- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires.
- **Not legal advice.** The §10.5 jurisdictional surface is a research-grade checklist, NOT counsel's analysis. Verify everything with actual employment + privacy counsel licensed in IL + IN before claiming compliance with anything in this document.
---
## 12. Appendix — terms
- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`.
- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.).
- **Audit row** — one record in the audit response per decision, with the schema in §2.
- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes.
- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list.
- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation).
- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth.
- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it.
- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs.
---
## Change log
- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker.
+5
View File
@@ -104,3 +104,8 @@
**Date:** 2026-04-24
**Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec<SemanticFlag>` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec<TypeHint>` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec<BugFingerprint>` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002.
**Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon.
## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly
**Date:** 2026-05-02
**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call.
**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running.
+584
View File
@@ -0,0 +1,584 @@
# Identity Service — Phase 2 Design (v2 — post-scrum revisions)
> **⚠ OVER-SCOPED FOR LOCAL-ONLY DEPLOYMENT — needs simpler rewrite before implementation.**
>
> 2026-05-03 evening: J reframed the system as local-only per PRD line 70 ("Everything runs locally — no cloud APIs"). This document was drafted assuming SaaS-tier infrastructure (HashiCorp Vault, AWS KMS, S3 Object Lock, dual-control JWT split-secret ceremony, mTLS CA, separate Postgres database). For J's local-only single-box deployment serving IL+IN staffing, the audit trail can be MUCH smaller: local SQLite or Postgres, local key file, local HMAC chain to an append-only JSONL.
>
> The discovery findings in `AUDIT_PHASE_1_DISCOVERY.md` and `AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md` remain valid (PII flow paths, BIPA exposure, etc.). The PROBLEM is real. This DOC's solution shape is wrong for the deployment.
>
> Do NOT implement this document as-written. When J greenlights audit-trail work, draft a v3 that's local-only sized (~3-5 days, not 17-20).
>
> See `STATE_OF_PLAY.md` "PRD line 70 is load-bearing" entry for the binding direction.
**Status:** Draft v2 — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03
**Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_DISCOVERY.md`](AUDIT_PHASE_1_DISCOVERY.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md)
> **v2 history.** v1 (2026-05-03 morning) was scrummed across opus + kimi + gemini lineages. 3/3 reviewers converged on 6 critical issues — wrong migration order, master key + legal token "security theater," BIPA `inferred_existing` prima facie violation, healthcare default in wrong direction, Merkle chain without external anchor. v2 incorporates those changes. Full scrum reviews preserved at `/tmp/identity_scrum/{opus,kimi,gemini}_review.md`. Diff between v1 and v2 captured in §11 change log.
> **Why this exists.** Phase 1 + 1.5 confirmed today's substrate has no separation between candidate_id and PII. Both live in `workers_500k.parquet`. No per-access audit, no consent gate, no retention enforcement. This document specifies the new identity service that holds the candidate_id ↔ PII mapping, gates every PII read, audits every access, and serves as the single legal-attestable boundary between PII and the rest of the system.
> **Prerequisite:** **Phase 1.6 (BIPA pre-launch gates) MUST ship before any identityd backfill begins.** Per kimi+gemini scrum, backfilling with `consent_status='inferred_existing'` is a BIPA §15 prima facie violation. Phase 1.6 establishes the consent template + retention schedule + employee training that turns backfill from "inferred" into "pending_backfill_review with documented escalation path."
---
## 1. Scope and non-goals
### In scope
- Single source of truth for `candidate_id ↔ PII` mapping
- Per-PII-access audit log (who/what/when/why)
- Consent + retention metadata (BIPA + IL Day and Temporary Labor Services Act + healthcare PHI)
- Legal-tier access — short-lived JWT with **split-secret dual-control issuance** (NOT a static file token, per all 3 reviewers)
- Healthcare-vertical routing with **fail-closed default** (`unknown` treated as healthcare until classified, per opus+gemini)
- EU-compatible interface — fields exist, enforcement is per-subject (no system-wide flag flip needed)
- Hardware-backed master key — **HashiCorp Vault Transit OR AWS KMS** for v1 (NOT a sealed file, per all 3 reviewers; sealed-file v1 was "obfuscation in any defensible sense" — kimi)
- Signed-JSON audit response with PDF render path (PDF deferred; JSON ships first per opus)
- External-anchor for audit-log integrity (S3 Object Lock + signed timestamps; NOT just Postgres Merkle, per all 3 reviewers)
- mTLS or Unix Domain Socket between gateway ↔ identityd (per kimi — "port-isolated is theater without authenticated channel")
- Field-level authorization on `/subjects/{id}` (per-purpose field allowlists, per kimi)
- Synchronous cache invalidation hook for RTBF (per opus — even if Phase 7 is the full RTBF build)
### Out of scope (for this phase)
- The `/audit/subject/{id}` endpoint itself (Phase 3)
- Full subject-tagging across other substrates (Phase 4)
- Right-to-be-forgotten implementation full-flow (Phase 7)
- Training-safe export (deferred from Phase 2 per opus — "not on critical path for audit-trail defense")
- BIPA pre-launch gates content (Phase 1.6 — separate doc)
### Architecturally simplified from v1 (per scrum)
- **Per-row encryption keys** deferred to Phase 7. v2 uses a single Data Encryption Key (DEK) wrapped under HSM-backed Key Encryption Key (KEK), with per-subject **ciphertext-deletion** as the v2 erasure mechanism. This matches kimi's recommendation: "A single data-encryption key with HSM-backed rotation, plus per-subject deletion of ciphertext, achieves equivalent practical erasure with far less complexity." Cryptographic-erasure via per-subject keys becomes a Phase 7 enhancement.
- **EU placeholder fields** kept (per J 2026-05-03), but enforcement code is genuinely no-op until first eu_resident=true subject — no code paths run, no schema migrations needed when EU comes online (just per-subject field population).
- **Merkle chaining** narrowed (per gemini): full Merkle tree only for legal-tier and consent/erasure events. Standard `gateway_lookup` events get a simpler signed-HMAC chain. Both anchored externally.
---
## 2. Architectural shape
### 2.1 — Process model + transport
Per J 2026-05-03 + kimi scrum: separate daemon, mTLS-mandatory transport. Single Go implementation, both runtimes call it.
| Property | Value |
|---|---|
| Name | `identityd` |
| Port | `:3225` (single port, not dual; per kimi simplification — runtime-agnostic routing) |
| Implementation | Go |
| Storage | Postgres in **isolated database** (per J answer, confirmed). Schema-level isolation; no shared schemas with Langfuse or other lakehouse storage. |
| Transport gateway↔identityd | **mTLS mandatory.** Self-signed CA managed by identityd at startup; gateway clients have their own client certs. Plain HTTP is rejected at the listener. |
| Encryption-at-rest | **Single DEK wrapped under KEK in HashiCorp Vault Transit** (recommended) OR AWS KMS. Master key NEVER on disk. v1 of identityd refuses to start if KEK is unreachable. |
| Audit-log signing | **Separate Ed25519 keypair from the master encryption key.** Signing key in Vault Transit (signing backend) OR a separate sealed-secret file with strict rotation procedure. |
| External anchor for audit chain | **S3 Object Lock** (compliance mode, 7-year retention) holding periodic Merkle root commitments. Daemon writes hourly + on legal-tier event. |
| Backup | Postgres standard backup; KEK backup separate (different storage, different access). Crypto-erasure model only works if these are not co-located. |
| Rate limiting | Per-token QPS + daily-volume caps. Default legal-token: 100 lookups/day + 10K daily volume; alerting at 50%. Configurable per-token. |
| Token revocation | `token_revocations` table checked on every legal-tier auth. Revocation propagates within 60s (TTL on cache). |
### 2.2 — Schema (Postgres DDL, v2)
```sql
-- Subject record. PII columns are ciphertext under the DEK.
CREATE TABLE subjects (
candidate_id TEXT PRIMARY KEY, -- UUID v7, NOT sequential (per kimi enumeration concern)
-- Encrypted PII fields. AES-256-GCM under DEK. NULL = not collected.
name_ct BYTEA,
email_ct BYTEA,
phone_ct BYTEA,
address_ct BYTEA,
ssn_ct BYTEA,
dob_ct BYTEA,
-- DEK version this subject was encrypted under. KEK rotates; DEK rotates per
-- KEK rotation. New DEK version = subjects re-encrypted in background sweep.
dek_version INT NOT NULL,
-- Lawful basis + consent metadata
consent_status TEXT NOT NULL CHECK (consent_status IN (
'pending_backfill_review', -- backfill default; no biometric/PHI use until reviewed
'pending_first_contact', -- new subject, awaiting consent UX
'given', -- explicit consent recorded
'withdrawn', -- subject revoked
'expired' -- consent timeout
)),
consent_version TEXT, -- references published consent template version
consent_given_at TIMESTAMPTZ,
consent_withdrawn_at TIMESTAMPTZ,
-- BIPA-specific fields (Phase 1.5 §1E + Phase 1.6 prerequisite)
biometric_consent_status TEXT NOT NULL DEFAULT 'never_collected' CHECK (biometric_consent_status IN (
'never_collected', 'pending', 'given', 'withdrawn', 'expired'
)),
biometric_retention_until TIMESTAMPTZ, -- BIPA: max 3 years from last interaction
-- Vertical detection — drives healthcare PHI routing.
-- DEFAULT 'unknown' (per opus+gemini scrum) — fail-closed routing treats
-- unknown as healthcare-equivalent until reclassified.
vertical TEXT NOT NULL DEFAULT 'unknown' CHECK (vertical IN (
'unknown', 'general', 'healthcare', 'finance', 'other'
)),
-- Per-vertical retention period. Drives the daily erasure sweep.
-- (Per opus: this was missing in v1. Required for BIPA's 3-year-from-
-- last-interaction rule which differs from generic retention.)
retention_period_days INT NOT NULL,
-- EU-placeholder fields. Enforcement is per-subject; nothing runs
-- until a row has eu_resident=true AND lawful_basis IS NOT NULL.
eu_resident BOOLEAN NOT NULL DEFAULT FALSE,
lawful_basis TEXT, -- GDPR Art. 6 basis when eu_resident=true
transfer_mechanism TEXT, -- SCC, DPF, BCR — populated when EU comes online
-- Standard audit columns
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_interaction TIMESTAMPTZ, -- drives retention sweep
-- Crypto-erasure state (v2: ciphertext deletion, NOT key destruction)
erased_at TIMESTAMPTZ,
erasure_reason TEXT
);
-- Append-only access log. EVERY PII read writes a row.
-- Field-level: fields_accessed records WHICH fields the caller resolved.
-- purpose enforced against an allowlist per-purpose-token.
CREATE TABLE pii_access_log (
access_id BIGSERIAL PRIMARY KEY,
candidate_id TEXT NOT NULL,
accessed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
accessor_kind TEXT NOT NULL, -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve'
accessor_id TEXT NOT NULL, -- daemon name + caller token-hash; never raw token
purpose_token TEXT NOT NULL, -- opaque token; resolved through purpose_definitions table
fields_accessed TEXT[] NOT NULL,
request_trace_id TEXT,
chain_kind TEXT NOT NULL CHECK (chain_kind IN ('hmac', 'merkle')), -- HMAC for standard, Merkle for legal/consent/erasure
integrity_hash TEXT NOT NULL,
-- Per opus security finding: this row's existence is itself sensitive
-- ("candidate X is under legal review" leaks via purpose). Read access
-- to this table requires legal-tier auth or scoped per-subject auth.
is_legal_tier_event BOOLEAN NOT NULL DEFAULT FALSE
);
-- Purpose definitions — enforces field-level authorization (per kimi).
-- A given purpose_token can only request fields in its allowlist.
CREATE TABLE purpose_definitions (
purpose_token TEXT PRIMARY KEY, -- e.g. 'fill_validation', 'audit_subject_response'
description TEXT NOT NULL,
allowed_fields TEXT[] NOT NULL, -- e.g. ARRAY['name'] for fill_validation
auth_tier TEXT NOT NULL CHECK (auth_tier IN ('service', 'legal')),
rate_limit_qps INT,
daily_volume_cap INT
);
-- Token revocation list. Checked on every auth, cached 60s.
CREATE TABLE token_revocations (
token_hash TEXT PRIMARY KEY,
revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
revoked_by TEXT NOT NULL,
reason TEXT
);
-- Consent template versioning. Hash references, not embedded text per row.
CREATE TABLE consent_versions (
version TEXT PRIMARY KEY,
effective_at TIMESTAMPTZ NOT NULL,
superseded_at TIMESTAMPTZ,
template_hash TEXT NOT NULL, -- SHA256 of the canonical template
template_path TEXT NOT NULL -- where the canonical text lives (e.g. data/_consent/v3-2026-04-15.md)
);
-- External anchor checkpoints. Periodically committed to S3 Object Lock.
CREATE TABLE anchor_commits (
commit_id BIGSERIAL PRIMARY KEY,
chain_kind TEXT NOT NULL, -- 'hmac' or 'merkle'
checkpoint_at TIMESTAMPTZ NOT NULL,
last_access_id BIGINT NOT NULL, -- max access_id covered by this checkpoint
root_hash TEXT NOT NULL,
s3_object_uri TEXT NOT NULL, -- s3://anchor-bucket/identityd/...
s3_lock_until TIMESTAMPTZ NOT NULL -- compliance-mode retention end
);
```
### 2.3 — HTTP surface (v2)
All under `/v1/identity/`. mTLS + token required on every endpoint.
| Method + Path | Purpose | Auth tier | Notes |
|---|---|---|---|
| `POST /v1/identity/subjects` | Create new subject. Body: PII fields + `purpose_token`. Returns: candidate_id (UUID v7). | service | Validates purpose_token allowlist matches submitted fields |
| `GET /v1/identity/subjects/{candidate_id}?purpose={token}&fields={list}` | Resolve PII for candidate. | service | **Field-level enforcement**: returned fields ⊆ purpose_token's allowed_fields. Mismatch → 403. Logs access row. |
| `GET /v1/identity/subjects/{candidate_id}/vertical` | **Minimal-disclosure** vertical lookup (per kimi HIPAA minimum-necessary). Returns only `{vertical, consent_status}`. | service | Used by gateway routing. Cheaper access-log row. |
| `GET /v1/identity/subjects/{candidate_id}/full` | Complete subject record + audit summary. | **legal** | Short-lived JWT only. Logged with `is_legal_tier_event=true`. Triggers real-time notification to designated counsel + J. |
| `POST /v1/identity/subjects/{candidate_id}/consent` | Record consent given/withdrawn with version. | service | |
| `POST /v1/identity/subjects/{candidate_id}/erase` | Crypto-erasure. Idempotent. | **legal** | Synchronously triggers cache invalidation hooks (per opus). |
| `GET /v1/identity/access_log/{candidate_id}` | Per-subject access log for audit response. | **legal** | |
| `POST /v1/identity/auth/legal_token` | Issue short-lived JWT (max 24h). Requires **dual-control attestation**: J's signed nonce + counsel's signed nonce, both verified against pre-registered public keys. | dual-control | Per all 3 reviewers — replaces v1's static file approach |
| `GET /v1/identity/health` | Liveness | none | Returns only `{status: "ok"}` — no version, no schema info |
| `(no public training-safe-export endpoint in v2)` | — | — | Deferred per opus — not on critical path |
### 2.4 — Auth model (v2 — split-secret + short-lived)
**Service-tier auth** (gateway → identityd for routine PII resolution):
- mTLS client cert per gateway (Rust :3100 + Go :4110 each get their own cert)
- Bearer token in `Authorization` header (long-lived — 90d), rotated per ops runbook
**Legal-tier auth** (per-request, short-lived JWT, dual-control issuance):
- J holds private key A
- Designated outside counsel holds private key B
- `POST /v1/identity/auth/legal_token` requires:
- HTTP body: `{purpose, ttl_seconds, requested_fields, signature_a, signature_b, nonce_a, nonce_b}`
- identityd verifies both signatures against pre-registered A and B public keys
- On success: emits a JWT signed by identityd, valid for `ttl_seconds` (max 24h), scoped to `purpose`+`fields`
- The JWT:
- Cannot be issued without BOTH A and B signatures (per gemini "split-secret startup ceremony" pattern, applied per-token)
- Carries scope (purpose + fields) inside its claims; identityd enforces at use time
- Per-token rate limit + daily cap recorded on issuance
- Revocation: token_hash added to `token_revocations`; identityd refuses on next check (60s cache TTL)
**Why this matters:**
- v1's "static file mode 0400" was security theater per all 3 reviewers
- A leaked legal token v1 = unbounded exfiltration window until rotation
- v2's leaked JWT = bounded by TTL (max 24h) AND revocable in <60s
- Single-actor compromise (J's machine OR counsel's machine, not both) cannot mint legal tokens
---
## 3. Integration with the rest of the substrate
### 3.1 — Gateway changes
When gateway needs PII for a fill scenario:
1. Gateway has only `candidate_id` (post-Phase-1.5 view-routing fix)
2. Gateway calls `GET /v1/identity/subjects/{candidate_id}/vertical` first if it might be healthcare-vertical-sensitive routing
3. Gateway calls `GET /v1/identity/subjects/{candidate_id}?purpose=fill_validation&fields=name`
4. identityd validates: purpose_token=`fill_validation` allows `[name]` only; if request asks for `[name,ssn]` → 403
5. identityd writes `pii_access_log` row, decrypts requested fields, returns
6. Gateway uses fields, then **explicitly zeros memory** (kimi's drop-and-overwrite pattern) before returning HTTP response
7. **No PII caching** in gateway memory beyond request lifetime
**LRU embed cache** (commit `150cc3b`) currently keys by `(model, text)` where text contains PII. Phase 4 task to re-key as `(model, candidate_id, field_subset_hash)`. **Phase 2 must add the synchronous cache-purge hook** so RTBF in Phase 2 invalidates Phase 4's cache when it ships (opus finding).
### 3.2 — Cross-runtime: same identityd, both gateways call it
Both gateways call identityd over mTLS. Same endpoints, same auth model. New cross-runtime parity probe `audit_parity.sh` validates identical PII request through both gateways produces identical access-log rows (modulo daemon-name field).
### 3.3 — JSONL writer changes (subject_id top-level promotion)
Per `AUDIT_PHASE_1_DISCOVERY` §10/C5:
- All JSONL sinks (outcomes, sessions, overseer_corrections, observerd ops) gain top-level `subject_ids: [...]` field
- `fills[*].name` replaced with `name_ref: "[REDACTED-{candidate_id}]"`
- Authorized callers dereference via identityd
### 3.4 — Langfuse boundary redaction (defense in depth, per opus TOCTOU finding)
v1 design: per-request map of `subject_id → resolved_PII`, replace before Langfuse POST.
**v2 adds** (per opus): outbound regex/NER pass on the Langfuse payload as defense-in-depth. The model can hallucinate names not in the resolved-PII map. The outbound NER pass catches them.
```
Gateway constructs Langfuse payload
→ Pass 1: per-request resolved-PII map replacement
→ Pass 2: NER scan (regex for SSN/phone shapes; named-entity model for names)
→ Either pass detecting unredacted PII → drop the trace OR replace with [POSSIBLE-PII-DETECTED]
→ POST to Langfuse
```
### 3.5 — Healthcare vertical routing — fail-closed default
**Per opus + gemini scrum: Default is `unknown` → treated as healthcare for routing purposes.**
When gateway receives a request involving any candidate:
1. Query `/vertical` for the candidate
2. If `vertical IN ('healthcare', 'unknown')` → route to **on-box Ollama only** (no opencode/openrouter/ollama_cloud egress)
3. If `vertical='general'` (or other non-healthcare) → cloud routing OK
4. If identityd unreachable → fail closed (refuse the request, return 503)
**Reclassification path:** subjects can be moved from `unknown` → `general` via explicit operator action (after manual review). Subjects flagged `healthcare` stay healthcare unless explicitly downgraded.
**Auto-escalation** (per kimi): if a subject's resume_text or call_log content is updated with healthcare-pattern matches (RN, BSN, hospital, MD, physician, etc.), vertical auto-escalates to `healthcare`. Never auto-de-escalates.
### 3.6 — Cache invalidation hook (Phase 2 must ship even though full RTBF is Phase 7)
When `POST /v1/identity/subjects/{id}/erase` fires:
1. Mark `subjects.erased_at` and zero out ciphertext columns
2. Write `pii_access_log` row with `purpose_token='retention_expired'` or `'rtbf_request'`
3. **Synchronously** publish a `subject_erased` event to a pub/sub channel (Redis or Postgres LISTEN/NOTIFY)
4. Gateway subscribes; on event, purges any in-flight cache entries for that candidate_id
5. Eventual-consistency window: <5s between erase-call and cache-flush
Without this hook, RTBF in Phase 7 would be a lie because the gateway's LRU embed cache could still hold the subject's data.
---
## 4. Audit response shape (Phase 3 preview, v2 — adds model-version snapshot per gemini)
```json
{
"schema": "audit.subject.v1",
"subject_token": "01926f2e-7c1b-7000-...", // UUID v7, NOT CAND-NNNNNN
"request_window": { "from": "2026-01-01", "to": "2026-05-03" },
"generated_at": "2026-05-03T12:00:00Z",
"generated_by": "identityd@hostname",
"merkle_root": "sha256:...", // root hash of legal/consent/erasure events
"external_anchor": "s3://anchor-bucket/identityd/2026-05-03T12-00.json", // S3 Object Lock URI
"signature": "ed25519:...", // separate signing key from encryption KEK
"consent": { ... },
"decisions": [
{
"ts": "2026-04-22T09:15:23Z",
"decision_kind": "fill_recommendation",
"daemon": "gateway",
"model": "kimi-k2.6",
"model_version_hash": "sha256:...", // NEW per gemini — proves what model existed AT decision time
"provider": "ollama_cloud",
"trace_id": "trace-abc",
"session_id": "session-xyz",
"input_features": { ... }, // sanitized; no protected attributes; no inferred-attribute proxies
"output": "...",
"rationale": "...",
"comparator_pool_size": 47,
"comparator_appendix_ref": "see comparator_appendix.A"
}
],
"comparator_appendix": {
"A": {
"scope": "fill scenarios in window matching role X, geo Y",
"total_pool_size": 47,
"selection_rate_by_protected_class": {
// Aggregated; NO other subjects' identifiers leak
"race": { "white": 0.33, "black": 0.31, ... },
"gender": { "man": 0.34, "woman": 0.30, ... }
},
"four_fifths_test": "passed" // or "concern: rate ratio 0.71"
}
},
"access_log": [...],
"footer": {
"completeness_attestation": "all decisions about subject_token in window per retention policy v2 are included",
"merkle_proof": "...", // proof this audit's root is in S3 anchor
"what_was_excluded": "decisions older than 4 years (retention expired) — count: 0"
}
}
```
---
## 5. Migration path (v2 — REORDERED per all 3 reviewers)
v1's order was wrong. New order, with explicit prerequisites:
| Step | Action | Prerequisite |
|---|---|---|
| **0** | **Phase 1.6 BIPA pre-launch gates SHIPPED** — consent template published, retention schedule public, deletion procedure documented, employee training acknowledged. | (separate work — see PHASE_1_6_BIPA_GATES.md when written) |
| **1** | Stand up identityd with **synthetic test subjects only**. KEK in vault, mTLS live, all endpoints serve. | Vault/KMS available; mTLS CA bootstrapped |
| **2** | **Audit-parity probe** (`audit_parity.sh`) green on synthetic data. Cross-runtime equivalence verified. | Step 1 |
| **3** | Add gateway feature flag `LH_USE_IDENTITY_SERVICE`. **Shadow-write only** — gateway writes new subjects to identityd AND continues SQL path. No reads from identityd yet. | Step 2 |
| **4** | Run shadow-write for ≥1 week. Validate access logs, encryption correctness, write-path performance under real traffic. | Step 3 |
| **5** | Backfill from `workers_500k.parquet`. **`consent_status='pending_backfill_review'`** for ALL existing rows. **`vertical='unknown'`** default (NOT 'general'). **`biometric_consent_status='never_collected'`** — backfill does NOT include any biometric data. | Step 0 (BIPA gates) + Step 4 |
| **6** | Human-review queue for vertical reclassification. Subjects move from `unknown` to `general` only after explicit review. Healthcare-pattern matches auto-escalate to `healthcare` (never auto-downgrade). | Step 5 |
| **7** | **Shadow-read** — gateway reads from BOTH SQL and identityd, compares, logs divergences, returns SQL result. Run ≥1 week. | Step 5 |
| **8** | Feature-flag cutover — gateway reads from identityd, falls back to SQL on error, alerts on every fallback. | Step 7 |
| **9** | Quarantine PII columns in workers_500k.parquet — only after **cryptographic attestation of identityd completeness** (Merkle proof: source row hash = identityd row hash for every candidate_id). Move PII columns to a different bucket; the candidate_id-only projection becomes operational. | Step 8 |
**Key v2 changes from v1:**
- New Step 0 (BIPA gates prerequisite) — backfill cannot start without this
- Steps 3-4 (shadow-write before backfill) — production validates the WRITE path before data lands
- Step 5 backfill consent: `pending_backfill_review` not `inferred_existing` (BIPA defense)
- Step 5 vertical default: `unknown` not `general` (HIPAA fail-closed)
- New Step 6 (human review) — vertical classification via explicit operator action
- Step 7 added (shadow-read after backfill) — catches encoding/normalization bugs before cutover
- Step 9 (quarantine) requires cryptographic attestation of completeness
Each step is its own commit, its own gate, its own rollback path.
---
## 6. Cross-runtime parity probe (NEW, per scrum §6 plus opus 'oracle test' addition)
`audit_parity.sh` ships in Phase 5. Asserts:
1. Same PII fetch through Rust gateway and Go gateway produces identical identityd access-log rows (modulo daemon name)
2. Crypto-erasure of test subject through Rust gateway is honored when Go gateway tries to fetch
3. Healthcare-vertical routing decision identical across both runtimes
4. **Oracle test** (per kimi SOC2 CC4.1 finding): probe with known-good inputs against expected outputs. A bug present in BOTH implementations must not pass the parity probe.
5. Discrimination-proxy phrase redaction triggers as expected on adversarial test cases
---
## 7. What this design intentionally does NOT solve
- Protected-attribute exclusion at decision time (Phase 6)
- Pathway memory trace body redaction (Phase 4)
- Retroactive Langfuse history scrub (separate runbook)
- GDPR Art. 22 right to explanation full implementation (Phase 8)
- Multi-region data residency (single-region US-Midwest by default)
- Training-safe export (deferred — not on critical path per opus)
---
## 8. Open questions — RESOLVED 2026-05-03
J confirmed all 6 v1 recommendations + scrum-driven changes:
1. ✅ **Master key:** HashiCorp Vault Transit (recommended) OR AWS KMS. NOT a sealed file. v1's sealed-file recommendation is rejected per all 3 reviewers ("obfuscation in any defensible sense").
2. ✅ **Postgres isolation:** identityd's own database, isolated schema.
3. ✅ **Vertical backfill:** `'unknown'` default with fail-closed routing. NOT `'general'` per opus+gemini scrum.
4. ✅ **Legal token:** Split-secret dual-control issuance (J + counsel both sign). Short-lived JWT, max 24h, revocable in <60s.
5. ✅ **Crypto-erasure sweep:** Daily 03:00 UTC. (Note: v2 erasure mechanism is ciphertext deletion, not key destruction. Per-row keys deferred to Phase 7.)
6. ✅ **EU enforcement:** Per-subject. Schema fields exist; nothing runs until first eu_resident=true subject.
### Newly resolved per scrum
7. ✅ **Migration order:** REORDERED per §5 above. New Step 0 (BIPA prerequisite). Shadow-write before backfill. Shadow-read before cutover.
8. ✅ **Audit-log external anchor:** S3 Object Lock with compliance-mode 7-year retention. Hourly + on-event commits.
9. ✅ **Audit-log signing key:** Separate Ed25519 keypair from KEK. Vault Transit signing backend OR sealed-secret with strict rotation runbook.
10. ✅ **mTLS gateway↔identityd:** Mandatory. Self-signed CA managed by identityd at startup.
11. ✅ **Per-row encryption keys:** Deferred to Phase 7. v2 uses single DEK with HSM-wrapped KEK + ciphertext deletion for erasure.
12. ✅ **Field-level authorization:** purpose_definitions table enforces per-purpose field allowlists.
13. ✅ **Synchronous cache invalidation:** pub/sub event on erase; gateway subscribes.
14. ✅ **Outbound NER pass for Langfuse:** Defense-in-depth in addition to symbol-table replacement.
15. ✅ **Model version hash in audit response:** Captured per decision row.
16. ✅ **PDF render:** Deferred from Phase 2; JSON ships first.
---
## 9. Estimated implementation cost (revised v2)
| Sub-phase | Effort | Notes |
|---|---|---|
| 2A — Postgres schema + Vault/KMS integration | 1.5-2 days | Includes mTLS CA bootstrap + signing-key separation |
| 2B — identityd HTTP surface (Go) | 2-3 days | All endpoints, auth, dual-control JWT issuance, rate limiting, revocation |
| 2C — Backfill ETL (BIPA-compliant — pending_backfill_review) | 1 day | Plus Merkle attestation of completeness |
| 2D — Gateway integration (Rust + Go, mTLS, shadow-write phase) | 2-3 days | Per-tool migration, parity probe |
| 2E — JSONL writer changes (subject_id top-level promotion) | 1 day | All sinks |
| 2F — Langfuse redaction (symbol-table + NER) | 2 days | Two passes + drop-on-detect |
| 2G — Healthcare-vertical routing (fail-closed) | 0.5 day | Plus auto-escalation pattern matcher |
| 2H — Cache invalidation pub/sub hook | 0.5 day | Critical for Phase 7 RTBF |
| 2I — External anchor (S3 Object Lock) | 1 day | Hourly + on-event commits |
| 2J — Cross-runtime parity probe with oracle tests | 1 day | New probe |
| **Total** | **~12-15 working days** | (v1 estimated 8-10; v2 added BIPA gates dependency, mTLS, dual-control JWT, NER pass, anchor, etc.) |
Bigger than v1. Worth it — every addition was a 3/3-reviewer convergent finding.
---
## 10. The four "would not build" blockers from scrum (all addressed in v2)
| # | v1 issue (3/3 reviewers) | v2 resolution |
|---|---|---|
| 1 | Migration order (backfill before validation) | §5 reordered; Step 0 BIPA gates prereq added |
| 2 | Master key on disk + legal token static file | §2.1 Vault/KMS for KEK; §2.4 split-secret short-lived JWT |
| 3 | `inferred_existing` BIPA prima facie violation | §5 Step 5 uses `pending_backfill_review`; Step 0 gates required first |
| 4 | Healthcare default `general` (HIPAA exposure window) | §3.5 fail-closed `unknown`-as-healthcare; §5 Step 5 backfill default `unknown` |
All three reviewers said "I would not build v1 as written." All four blockers are resolved in v2. Re-scrum recommended before implementation starts.
---
## 11. Change log (v1 → v2)
| Section | v1 | v2 |
|---|---|---|
| Master key storage | Sealed file `/etc/lakehouse/identityd_master.key` | HashiCorp Vault Transit / AWS KMS |
| Legal token | Static file mode 0400 | Split-secret short-lived JWT (max 24h), dual-control issuance |
| Encryption keys | Per-row keys with master wrapping | Single DEK + HSM-wrapped KEK + ciphertext deletion (per-row deferred to Phase 7) |
| Healthcare default | `vertical='general'` backfill | `vertical='unknown'` backfill, fail-closed routing |
| Migration §5 order | 1: stand up, 2: backfill, 3: feature flag, 4: cutover, 5: quarantine | 0: BIPA gates, 1: stand up, 2: parity probe, 3: shadow-write, 4: shadow-write soak, 5: backfill (pending_backfill_review), 6: human vertical review, 7: shadow-read, 8: cutover, 9: quarantine with cryptographic attestation |
| Audit-log integrity | Postgres Merkle chain only | Postgres Merkle for legal/consent/erasure + HMAC for standard + S3 Object Lock external anchor |
| Audit-log signing key | Same as encryption key | Separate Ed25519 keypair; Vault signing backend |
| Gateway↔identityd transport | Plain HTTP (implied) | mTLS mandatory |
| Field authorization | Per-endpoint only | Per-purpose-token field allowlists |
| Cache invalidation | Phase 4 / Phase 7 | Phase 2 ships pub/sub hook |
| Langfuse redaction | Symbol-table replacement only | Symbol-table + outbound NER pass + drop-on-detect |
| Audit response | No model version | model_version_hash per decision |
| PDF render | Phase 2 | Deferred — JSON ships first |
| Training-safe export | Phase 2 | Deferred — not critical path |
| Effort estimate | 8-10 days | 12-15 days |
---
## 12. v3 amendments (post-second-pass scrum, 2026-05-03 evening)
v2 was re-scrummed across opus + kimi + gemini. **All 3 verdict: BUILD-WITH-CHANGES.** All 4 v1 blockers verified RESOLVED. New v2 findings are tractable design fixes (not re-architecture). Folded in below. Reviews preserved at `/tmp/identity_scrum_v2/{opus,kimi,gemini}_review.md`.
### Convergent v2 findings (≥2 reviewers) — must fix before implementation
**v3-A1 — mTLS CA root must NOT live in identityd** (opus + gemini converge). v2 said "self-signed CA managed by identityd at startup" — if identityd is both the CA and authenticated party, compromise of identityd compromises the whole trust fabric. **v3 change:** mTLS CA root lives in **Vault PKI** (or, if Vault PKI is unavailable, an offline root with identityd as a properly-issued intermediate). identityd never has root-CA private key access.
**v3-A2 — Dual-control public key registry must be tamper-evident** (opus + gemini converge). v2 said "pre-registered public keys" but didn't say where. If they live in identityd's Postgres, DB-write compromise defeats dual-control. **v3 change:** J + counsel public keys live in **Vault KV with separate access policies** (or a signed config file at `/etc/lakehouse/identityd_dual_control.yaml` with its own dual-control rotation procedure). Public-key changes themselves require dual-control attestation. Plus: `nonce_a`/`nonce_b` get **server-issued challenges** (identityd issues a fresh nonce per request; clients sign that nonce; replay-protection via 5-min nonce cache).
### Single-reviewer v2 findings — also folded into v3
**v3-B1 (opus): Step 8 fallback-to-SQL needs explicit time bound.** Otherwise becomes permanent dual-read debt. v3: fallback active for max 14 days post-cutover; alert on every fallback; auto-disable at 14 days regardless of fallback rate.
**v3-B2 (opus): NER drop-on-detect needs alerting.** Silent observability gap if model regression starts hallucinating PII. v3: drop counter exposed as `identityd_ner_drops_total` Prometheus metric; alert on drop_rate >0.1% over rolling 1h.
**v3-B3 (opus): legal-tier notification transport must be specified.** v2 said "real-time notification to designated counsel + J" without saying how. v3: notification transport is **signed Slack webhook OR signed email** (configurable per deployment); message body never contains candidate_id or PII; only `{event_kind, timestamp, accessor_kind, integrity_hash}`. Notification failure does NOT block the legal-tier action — failure is logged for follow-up but token issuance proceeds.
**v3-B4 (opus): Step 6 human review queue needs SLA + volume estimate.** With 500k backfilled rows defaulting to `unknown`, all healthcare-adjacent traffic routes to on-box Ollama until reclassified. v3: budget assumption — auto-pattern-matching pre-classifies ~80% to confidently-non-healthcare; remaining 20% (~100k) stays `unknown` pending review at ~500/day operator throughput = ~7 months of queue. **Operational decision needed:** either staff the queue, accept the on-box-Ollama-only routing for ~7 months, or relax pattern matching at the cost of higher false-negative rate. Flag for J.
**v3-B5 (opus): Memory zeroing in Go is non-trivial.** v2 §3.1 step 6 says "explicitly zeros memory" — Go GC-managed strings are immutable, not zeroable. v3 implementation note: PII handling code path uses `[]byte` throughout; convert to string only at the JSON serialization boundary; explicit `runtime.GC()` after request completes is theater (Go won't actually zero the slice). **Acceptance:** "best-effort zero" — overwrite the `[]byte` slice contents post-use. Document this is best-effort, not cryptographic-grade scrubbing. (Rust-side: use the `zeroize` crate which IS cryptographic-grade.)
**v3-B6 (kimi): Service-tier purpose_definitions needs versioning + emergency revocation.** A misconfigured purpose with overly-broad `allowed_fields` is standing exfiltration authorization. v3 schema additions:
```sql
CREATE TABLE purpose_versions (
purpose_token TEXT NOT NULL,
version INT NOT NULL,
effective_at TIMESTAMPTZ NOT NULL,
superseded_at TIMESTAMPTZ,
PRIMARY KEY (purpose_token, version)
);
CREATE TABLE purpose_revocations (
purpose_token TEXT PRIMARY KEY,
revoked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
revoked_by TEXT NOT NULL,
reason TEXT
);
```
Auth path checks `purpose_revocations` on every call (not cached, OR cached <5s). Revocation is <60s without code deploy.
**v3-B7 (kimi): Cache invalidation needs erasure-generation atomicity.** v2 §3.6 pub/sub is best-effort. If Redis drops the message or gateway crashes between erase + purge, cache holds stale PII. v3 change:
- `subjects` table gains `erasure_generation INT NOT NULL DEFAULT 0`. Increments on every erase.
- Identityd PII responses include current generation in headers
- Gateway cache entries tagged with the generation they were filled at
- Cache hits with `cached_generation < current_generation` are rejected, force re-fetch from identityd
- Eventually-consistent within ONE request hop, not eventually-consistent over time
**v3-B8 (kimi): Dual-control issuance needs cooling-off period to prevent bypass culture.** v3: 15-minute mandatory delay between nonce submission and token issuance, with notification to BOTH parties at delay-start AND issuance. Bypass requires both J and counsel to sign an "emergency bypass" attestation that's logged separately + escalated to a board-level reviewer.
**v3-B9 (kimi): NER must have calibrated false-negative rate.** v3: maintain a synthetic adversarial test set (50 hallucinated-PII examples + 50 legitimate non-PII). NER must achieve recall ≥99.5% on the synthetic set in CI; production drop-rate >0.1% triggers re-calibration.
**v3-B10 (kimi): S3 Object Lock root credentials need separate AWS account.** Same ops team holding Vault admin + AWS root = limited independence. v3: Object Lock bucket lives in **separate AWS account** with **write-only IAM role** issued to identityd. Root credentials for that account held by named external party (could be outside counsel) with quarterly attestation.
**v3-B11 (kimi): biometric_consent_status='never_collected' may not satisfy BIPA infrastructure-as-notice argument.** Plaintiffs may argue presence of biometric schema fields constitutes constructive notice of intent. v3: Phase 1.6 BIPA policy doc must explicitly attest "no biometric data exists in the pre-identityd substrate" with cryptographic evidence (hash of pre-identityd schema). Counsel must bless this attestation.
**v3-B12 (gemini): Backup retention window vs ciphertext-deletion erasure.** Phase 2 uses ciphertext deletion + single-DEK; if Postgres backups retain pre-erasure ciphertext + the DEK still exists, backup is recoverable. v3 erasure runbook: includes maximum backup retention period (e.g., 30 days). RTBF status reported to subject as "erased now; residual in backups expires by {date}." For full crypto-erasure-on-demand, Phase 7 per-row keys is the path.
### Net v3 effort delta
v2 estimate: 12-15 days. v3 amendments add:
- mTLS CA via Vault PKI integration: +0.5 day
- Public-key registry in Vault KV: +0.5 day
- Server-issued nonces for dual-control: +0.25 day
- Step 8 fallback time bound: +0.1 day
- NER drop-rate metric + alert: +0.25 day
- Legal-tier notification transport: +0.5 day
- purpose_versions + purpose_revocations: +0.5 day
- erasure_generation cache atomicity: +0.5 day
- 15-min cooling-off: +0.25 day
- NER calibrated test set: +0.5 day
- Separate AWS account for Object Lock: +0.5 day
- BIPA infrastructure-as-notice attestation: +0.25 day (mostly doc work)
- Backup-retention erasure runbook: +0.25 day
**Total v3 delta: ~5 days. Revised estimate: 17-20 days.**
The cost is real. The reason it's worth paying is what those 5 days buy: a design that 3 independent senior security architects (across 3 model lineages) all said they would build. v1 said "do not build." v3 says "build."
### What's v3-deferred vs v3-must-have
**v3 must-have (block implementation):** v3-A1, v3-A2, v3-B1, v3-B6, v3-B7, v3-B11.
**v3 should-have (ship in Phase 2 if calendar allows; otherwise Phase 5):** v3-B2, v3-B3, v3-B4 (operational decision), v3-B5 (best-effort), v3-B8, v3-B9, v3-B10, v3-B12.
If schedule pressure forces a cut, the should-have items can ship in Phase 5 (identity service build completion). The must-haves cannot — they're integral to the security boundary.
---
## Change log
- 2026-05-03 — v1 initial draft.
- 2026-05-03 — v2 post-first-scrum: 3/3 reviewer convergent findings folded in. 4 "would not build" blockers all resolved. Re-scrum recommended.
- 2026-05-03 — v3 post-second-scrum: 3/3 BUILD-WITH-CHANGES. v1 blockers verified resolved. 12 new v2 findings folded into v3 amendments (§12). Re-scrum NOT required for v3 — diminishing returns; the must-have items are concrete fixes with clear acceptance criteria.
+334
View File
@@ -0,0 +1,334 @@
# Phase 1.6 — BIPA Pre-Launch Gates
**Status:** Draft — 2026-05-03 · **Owner:** J + outside counsel · **Companion to:** [`AUDIT_TRAIL_PRD.md`](AUDIT_TRAIL_PRD.md), [`AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md`](AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md), [`IDENTITY_SERVICE_DESIGN.md`](IDENTITY_SERVICE_DESIGN.md)
> **Why this exists.** `IDENTITY_SERVICE_DESIGN.md` v3 §5 Step 0 names Phase 1.6 as a HARD PREREQUISITE: identityd backfill cannot start until Phase 1.6 ships. This doc specifies what Phase 1.6 contains.
>
> **Scope.** BIPA (740 ILCS 14) compliance gates that must be in place BEFORE the system accepts a single real candidate photo. Synthetic-data face pool can keep operating; real-photo intake CANNOT begin without these gates.
>
> **Authority.** This is an engineering scaffold. Sections marked `⚖ COUNSEL` need outside counsel to author the actual legally-binding text. Engineering ships the procedural gates; counsel writes the words.
---
## 1. The five BIPA pre-launch gates
Each gate is a deliverable that must ship before real-photo intake. None is optional. Order shown is the recommended ship sequence.
### Gate 1 — Public retention schedule (BIPA §15(a))
**Required:** A publicly-available, written retention schedule for biometric identifiers and information.
**What ships:**
- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file
- Linked from public privacy policy at the deployment URL
- Specifies:
- Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`)
- Purpose of collection (identity matching for staffing operations)
- Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin)
- Destruction procedure: per Gate 5 below
- Versioned (this is v1; future updates supersede with a new version)
**⚖ COUNSEL** — write the actual schedule. Engineering provides the operational facts; counsel writes the binding language.
**Engineering acceptance:** the file is committed, the public URL renders it, and identityd's `consent_versions` table references it by hash.
---
### Gate 2 — Informed written consent (BIPA §15(b))
**Required:** Informed, written consent BEFORE any biometric collection occurs.
**What ships:**
- `docs/policies/consent/biometric_consent_template_v1.md` — public consent template
- Versioned, hashed, referenced from identityd's `consent_versions` table
- Must disclose, per BIPA §15(b)(1)-(3):
1. That biometric identifiers/information will be collected
2. The specific purpose for collection (and the length of term — references Gate 1)
3. Receipt of a written release authorizing collection
- Consent flow at intake:
- Candidate sees the disclosure on a UI surface (web form / paper / digital signature)
- Candidate provides explicit affirmative action (signature, click-acceptance with timestamp, etc.)
- Identityd records `biometric_consent_status='given'` with `consent_version` reference + `consent_given_at` timestamp
- **Without identityd recording 'given', no biometric data flows through deepface.**
**⚖ COUNSEL** — write the consent template. Recommended content (engineering view):
- Clear language (not just legal boilerplate)
- Specific to facial-classification (not generic biometrics)
- Includes withdrawal procedure
- Includes data-subject rights enumeration
**Engineering acceptance:** consent gate is enforced in code at the photo-upload endpoint; identityd refuses biometric writes when `biometric_consent_status != 'given'`; pre-existing synthetic-face pool is exempt (no consent needed because no real subject).
---
### Gate 3 — Photo-upload endpoint with consent enforcement
**Required:** Code-level enforcement that real-photo intake checks consent before processing.
**What ships:**
An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior:
1. Caller authenticates with service-tier token
2. Endpoint queries identityd for `subjects.biometric_consent_status`
3. If status ≠ `'given'` → HTTP 403 with reason `"BIPA consent required before biometric processing"`
4. If status = `'given'`:
a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`)
b. deepface tagging runs against the photo
c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1.
d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule
e. `pii_access_log` row written with `purpose_token='biometric_collection'`
5. Response: `{candidate_id, retention_until, consent_version}`
**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):**
The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/<id>.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option<JSON>`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type.
```rust
// crates/catalogd/src/subject_manifest.rs (paraphrased)
pub struct BiometricCollection {
pub data_path: String, // quarantined path
pub template_hash: String, // SHA-256 of original bytes (integrity, NOT re-derivation)
pub collected_at: DateTime<Utc>,
pub classifications: Option<Value>, // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md)
}
```
**Engineering acceptance:**
- Endpoint refuses uploads when consent missing (verified by integration test)
- deepface output never lands in the synthetic-face manifest (`data/headshots/manifest.jsonl`)
- Real-photo classifications are isolated to identityd `subjects` table — never flow to JSONL sinks
- The `/headshots/:key` route in mcp-server REMAINS synthetic-only — does NOT serve real candidate photos to LLMs without an explicit allowance (proposed: real photos served only to authenticated staffer UI, never to model context)
---
### Gate 4 — Deprecate name → ethnicity inference
**Required:** The hard-coded `NAMES_HISPANIC` / `SURNAMES_*` lookup tables in `mcp-server/search.html:3375-3432` (per Phase 1.5 §1B walk) get removed.
**What ships:**
- A code commit that removes:
- `FEMALE_NAMES`, `MALE_NAMES` constants
- `NAMES_HISPANIC`, `NAMES_BLACK`, `NAMES_SOUTH_ASIAN`, `NAMES_EAST_ASIAN`, `NAMES_MIDDLE_EASTERN` constants
- `SURNAMES_HISPANIC`, `SURNAMES_SOUTH_ASIAN`, `SURNAMES_EAST_ASIAN`, `SURNAMES_MIDDLE_EASTERN`, `SURNAMES_BLACK` constants
- The `genderFor()` and `guessEthnicityFromFirstName()` functions
- All call sites that consumed these (face-pool bucket selection)
- Replacement strategy:
- For SYNTHETIC face pool routing: deterministic hash of candidate_id selects a face bucket, no demographic inference
- For REAL candidate photos: the candidate's actual photo IS the representation; no inference needed
**Why this is BIPA + Title VII risk separately:** name-based ethnicity classification is BOTH a discriminatory feature engineering practice (Title VII) AND, when combined with photo-based attribute extraction, a "biometric information derived from a biometric identifier" pattern (BIPA broad reading). Removing the lookup tables forecloses both arguments.
**Engineering acceptance:**
- Lookup tables removed from search.html
- Unit test asserts no protected-attribute inference functions exist in search.html or any mcp-server module
- Face-pool routing for synthetic faces uses candidate_id hash exclusively
- Phase 1.5 §1B finding closed
---
### Gate 5 — Documented destruction procedure
**Required:** A written procedure for biometric data destruction at retention expiry OR consent withdrawal OR right-to-be-forgotten request.
**What ships:**
- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing
- Specifies:
- Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request
- Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly
- Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads/<id>/*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails
- Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed
- Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction)
- Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request
**⚖ COUNSEL** — review the runbook for legal sufficiency. Engineering writes the procedure; counsel attests that the procedure satisfies BIPA §15(a) destruction requirements.
**Engineering acceptance:**
- Runbook committed
- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full)
- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock)
- Erasure events are logged with cryptographic attestation
---
## 2. Cryptographic attestation: no biometric data exists pre-identityd
**Per `IDENTITY_SERVICE_DESIGN` v3-B11.** Plaintiffs may argue that the EXISTENCE of biometric schema fields constitutes constructive notice of intent to collect biometric data — therefore consent should have preceded the schema. The defense: prove that no biometric data was actually collected from real candidates before identityd + the consent gate.
**What ships:**
- A one-shot script `scripts/staffing/attest_pre_identityd_biometric_state.sh` that:
- Queries `data/datasets/workers_500k.parquet` schema and confirms NO column named `photo`, `biometric_*`, `face_*`, `image_*` exists
- Greps `data/_kb/*.jsonl` and `data/_pathway_memory/state.json` for any base64-encoded image bytes (deepface output, photo blobs)
- Verifies `data/headshots/manifest.jsonl` rows ≤ synthetic face pool size
- Hashes the schema + summary; commits the hash to S3 Object Lock (per identity service v3 anchor pattern)
- Attestation document `docs/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-XX.md` signed by J + outside counsel
**This is a one-time defense artifact.** It establishes the baseline: "as of this date, no biometric data was collected from real candidates."
---
## 3. Employee training acknowledgment (general BIPA hygiene)
**Required:** People with access to biometric data acknowledge BIPA-handling training.
**What ships:**
- `docs/policies/BIPA_HANDLING_TRAINING_v1.md` — training material covering:
- What constitutes biometric identifiers / information
- The consent + retention procedures
- Destruction obligations
- Reporting suspected exposure
- Acknowledgment record per individual (initially: J + counsel + named operators)
- Annual refresh
**⚖ COUNSEL** — write training content. Engineering doesn't author legal-compliance training.
---
## 4. Phase 1.6 exit criteria (gates Phase 2 backfill)
All 5 gates must be DONE before identityd backfill begins. Status as
of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code:
| # | Gate | Engineering | Counsel | Status |
|---|---|---|---|---|
| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** |
| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** |
| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option<JSON> = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** |
| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** |
| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** |
PLUS:
| # | Item | Engineering | Counsel | Status |
|---|---|---|---|---|
| 6 | Cryptographic attestation pre-identityd | DONE — `scripts/staffing/attest_pre_identityd_biometric_state.sh` + `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` (3/3 evidence checks pass; signature lines pending) | pending signature | **eng-DONE, signature-pending** |
| 7 | Employee training material | scaffold deferred — Gate 5 runbook §7 acknowledgment may serve as substrate | pending | **deferred** |
**Blocking set for Phase 2 backfill:** items **1, 2, 3, 4, 5, 6** must
all be DONE. Item 7 (employee training) is reduced from blocking to
"deferred" because the Gate 5 destruction runbook §7 already requires
operator acknowledgment before legal-tier credentials are issued —
that acknowledgment is procedurally equivalent to the training-record
requirement when the operator population is small (J + 1-2 named
operators). If the operator population grows beyond that, item 7
re-promotes to blocking and a separate training program must be authored.
⚖ COUNSEL — confirm whether item 7 deferral is acceptable for the
expected operator population size, or restore it to the blocking set.
**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel
review of the engineering scaffolds. Gate 3 substrate is fully
shipped; Gate 3b deepface classification was DECIDED on 2026-05-05
as Option C (defer) — `BiometricCollection.classifications` stays
`None` in v1, consent + retention docs revised to match this
narrower scope. If a future product requirement surfaces a real
need for classifications, the substrate is forward-compatible
(`Option<JSON>`) and either Option A (~1 day) or Option B (~5 days)
of the design doc can be picked up then under a v2 consent template.
---
## 5. Effort estimate
| Gate | Engineering effort | Legal effort |
|---|---|---|
| Gate 1 (retention schedule) | 0.5 day | counsel-dependent (typically 1-2 weeks for review) |
| Gate 2 (consent template) | 0.5 day | counsel-dependent (typically 2-4 weeks for review and consent UX design) |
| Gate 3 (photo-upload endpoint) | 1-2 days | review of endpoint behavior |
| Gate 4 (deprecate name-ethnicity inference) | 0.5 day | none (engineering-only fix) |
| Gate 5 (destruction runbook) | 1 day | counsel sign-off |
| §2 cryptographic attestation | 0.5 day | counsel + J signature |
| §3 employee training | 0.25 day (admin) | counsel-authored content |
| **Total engineering** | **~4-5 days** | — |
| **Total counsel** | — | **~3-6 weeks calendar** (review cycles) |
**The calendar bottleneck is counsel, not engineering.** Engineering can stage all 5 gates ready-to-ship in a week. Counsel sign-off + consent UX rollout is the longer pole.
---
## 6. Open questions for J + counsel
1. **Photo-upload UX:** is there an existing intake form / staffer console where photo upload would happen? Or is this new UI work?
2. **Consent collection mechanism:** electronic signature service (DocuSign, Adobe Sign), in-app click-acceptance, paper form? Each has different evidentiary weight in litigation.
3. **Operator list with biometric access:** who, today, would be on the named-operators list for §3 training?
4. **Counsel for sign-off:** named outside counsel — same or different from the dual-control legal-token party in identity service?
5. **Public privacy policy URL:** does one exist? If yes, where; if no, that's a separate Gate-1.5 deliverable.
---
## 6.5. Post-signature deployment runbook
When counsel returns the countersigned consent template + retention
schedule, the engineering side of "flip from permissive to strict
mode" is one command:
```bash
# 1. Counsel commits their signature to §7 of the consent template
# markdown (or J commits the signed PDF + updates §7 with counsel's
# name + date). The markdown is the BINDING TEXT — the PDF is just
# a rendering of it.
# 2. Hash the canonical signed text + seed the gateway allowlist.
./scripts/staffing/seed_consent_version.sh \
docs/policies/consent/biometric_consent_template_v1.md \
--label "v1 signed YYYY-MM-DD by [counsel name]"
# The script:
# - computes SHA-256 of the markdown (binding text)
# - atomically writes /etc/lakehouse/consent_versions.json with
# the new hash + per-seed audit metadata (timestamp, label,
# source path)
# - restarts lakehouse.service so the gateway re-reads the
# allowlist
# - probes /biometric/health for clean restart
# 3. Verify strict mode is rejecting unknown hashes:
TOKEN=$(cat /etc/lakehouse/legal_audit.token)
curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \
-H "X-Lakehouse-Legal-Token: $TOKEN" \
-H "Content-Type: application/json" \
-d '{"consent_version_hash":"deadbeefdeadbeef000000000000000000000000000000000000000000000000","consent_collection_method":"electronic_signature","operator_of_record":"strict_mode_probe"}'
# Expect: HTTP 400 + {"error":"consent_version_unknown", ...}
```
After this, the gateway is in counsel-tier strict mode:
- Any consent grant POST whose `consent_version_hash` doesn't match
a known signed template is refused at intake
- Operator typos (mistyped hash) become loud failures, not silent
bad records
- Future template revisions (v2, v3, ...) require counsel re-sign
AND a new `seed_consent_version.sh` run before being accepted —
the v1 hash stays in the allowlist for already-collected subjects'
audit-trail compatibility
### Pre-counsel demo seed
For deployments that want to exercise strict mode BEFORE counsel
signs, the same script works against the eng-staged template:
```bash
./scripts/staffing/seed_consent_version.sh \
docs/policies/consent/biometric_consent_template_v1.md \
--label "eng-staged demo seed (NOT counsel-signed)"
```
The hash entry should be replaced (rotate the demo hash out, add
the counsel-signed hash) when counsel completes review. The
allowlist's `_meta.seeded_at[]` array preserves the seed history.
---
## 7. What this PRD is NOT
- Not legal advice. The `⚖ COUNSEL` markers exist because the binding text needs lawyers, not engineers.
- Not a substitute for a DPIA / PIA. Phase 1.6 satisfies BIPA-specific gates; a Data Protection Impact Assessment is broader and may be required separately.
- Not a SOC2 Type II deliverable. SOC2 is a parallel work stream.
- Not the only gate before production. The full 9-phase audit-trail program continues; Phase 1.6 specifically unblocks Phase 2 (identity service implementation).
---
## Change log
- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup.
- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill.
+14 -2
View File
@@ -77,7 +77,7 @@ A modular Rust service mesh over S3-compatible object storage, with a local AI l
| Data Format | Parquet + Arrow | Yes |
| RPC (internal) | tonic (gRPC) | Yes |
| AI Runtime | Ollama (local models) | Yes |
| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes |
| AI Boundary | Direct Ollama HTTP API (gateway → Ollama, no sidecar). Updated 2026-05-02 per ADR-022 — was "Python FastAPI sidecar → Ollama HTTP API". Sidecar's lab_ui/pipeline_lab Python remain as dev-only tools (not on hot path). | Yes |
| Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** |
No new frameworks without documented ADR.
@@ -97,7 +97,7 @@ No new frameworks without documented ADR.
| **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog |
| **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) |
| **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal |
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
| **aibridge** | AI client — direct Ollama HTTP (per ADR-022, 2026-05-02; was Rust↔Python sidecar boundary). Owns LRU embed cache. |
| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
| **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) |
@@ -161,6 +161,18 @@ The feedback signal is **statistical + semantic**, not neural. No model training
---
## Production-readiness blocker (2026-05-03)
Substrate is mostly shipped through Phase 45. Throughput, parity, and observability are all measured.
**The production-readiness gate is the audit trail.** Staffing client requires defensible response to discrimination claims (worked example: John Martinez at Warehouse B requests every AI-system decision affecting him). PII tokenization, subject-of-record query, retention SLA, and right-to-be-forgotten under append-only logs are NOT YET BUILT.
Authoritative document: **`docs/AUDIT_TRAIL_PRD.md`**. 9-phase implementation plan, current-state-vs-target gap table, open questions blocking phase 1.
Until the audit-trail PRD's phase 9 exit criterion is met, "production-ready" claims are aspirational. Internal substrate (Lance, observability parity, cross-runtime fabric) is solid; subject-audit story is not.
---
## Phases
### Phase 0-5: Foundation ✅ COMPLETE
@@ -0,0 +1,91 @@
# BIPA Pre-IdentityD Biometric Attestation
**Date:** 2026-05-03
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §2
**Generator:** scripts/staffing/attest_pre_identityd_biometric_state.sh
## Purpose
This is a one-time defense artifact establishing that, as of
2026-05-03, no biometric identifiers or biometric information
from real candidates have been collected, processed, or stored
by the Lakehouse system. It is intended to be signed by J
(operator of record) and outside counsel, then anchored to a
tamper-evident store (filesystem with backups + version control).
## Evidence
## Check 1 — workers_500k.parquet schema (no biometric columns)
**Source:** `data/datasets/workers_500k.parquet`
**Schema columns** (18 total):
```
worker_id int64 nullable=True
name string nullable=True
role string nullable=True
email string nullable=True
phone string nullable=True
city string nullable=True
state string nullable=True
zip int64 nullable=True
skills string nullable=True
certifications string nullable=True
archetype string nullable=True
reliability double nullable=True
responsiveness double nullable=True
engagement double nullable=True
compliance double nullable=True
availability double nullable=True
communications string nullable=True
resume_text string nullable=True
```
**Schema SHA-256:** `973b9abe56420de8f88122278b633e813f90a64cf0ddaac6a9811dc0940be676`
- PASS: no biometric / photo / face / image column present
## Check 2 — KB + pathway memory contain no biometric payloads
**Sources scanned:**
- `data/_kb/*.jsonl` (knowledge base)
- `data/_pathway_memory/state.json` (pathway memory state)
**Files scanned:** 33
**Forbidden-pattern hits:** 0
- PASS: no biometric payload patterns found in scanned files
## Check 3 — Headshots manifest is synthetic-only
**Source:** `data/headshots/manifest.jsonl`
**Total rows:** 1000
**Rows tagged real/candidate_upload/photo_upload:** 0
- PASS: all 1000 rows are synthetic (no real-candidate uploads)
## Summary
**3 / 3** evidence checks pass.
---
## Attestation
I, the undersigned, attest that the above evidence accurately
reflects the state of the Lakehouse system as of 2026-05-03.
No biometric identifiers or biometric information from real
candidates have been collected, processed, or stored prior to
the deployment of the Phase 1.6 BIPA pre-launch gates.
**Evidence SHA-256:** `1fdcc9f1682de27e1a0556d698ce221b74c1e71cf54128763828b4bca7b5c1bf`
---
**Operator (J):** _______________________________ Date: __________
**Outside counsel:** ___________________________ Date: __________
@@ -0,0 +1,187 @@
# Counsel Handoff — Copy-Paste Email Draft
**Date assembled:** 2026-05-05
**Operator of record:** J
**Prepared for:** outbound transmission to outside counsel
**Tarball:** `reports/counsel/counsel_packet_2026-05-05.tar.gz`
**Tarball SHA-256:** see `reports/counsel/counsel_packet_2026-05-05.manifest.txt`
> This is a copy-paste-ready email draft. Subject line + body below.
> Attach the tarball + manifest before sending. The email frames
> what counsel needs to do, by when, and what happens after they
> sign — so the calendar bottleneck has no excuse to sit longer
> than counsel's actual review time.
---
## Subject line
```
Lakehouse staffing platform — Phase 1.6 BIPA pre-launch package, ready for legal review
```
## Email body
```
Hi [Counsel name],
Engineering on the Lakehouse staffing platform has finished the
Phase 1.6 BIPA (740 ILCS 14) pre-launch substrate. Before we begin
collecting any real candidate biometric data (photographs), the
five engineering-staged documents in the attached package need
your review and, where indicated, your signature.
What's attached:
1. counsel_packet_2026-05-05.tar.gz — bundled review package
2. counsel_packet_2026-05-05.manifest.txt — per-file SHA-256
integrity hashes (re-run `sha256sum -c` on receipt to verify
nothing changed in transit)
Open the cover letter inside the tarball
(`docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md`) first — it
walks through the substrate, the specific items needing your
sign-off, and recommended review sequence.
Headline asks (in priority order):
A. Biometric Retention Schedule v1
— render into binding language, sign as countersigning
party. Sets the public retention schedule required by
BIPA §15(a).
B. Biometric Consent Template v1
— render Disclosures 1-3 into binding consent text, sign.
This is the form a candidate signs before any photo is
collected. Required by BIPA §15(b)(1)-(3).
C. BIPA Destruction Runbook
— review for legal sufficiency, attest. Procedural
document; engineering wrote the steps, your role is
confirming they satisfy BIPA §15(a) destruction.
D. Pre-IdentityD Attestation (2026-05-03)
— countersign as the legal party. Establishes the boundary
that no biometric data was collected before the gates
shipped. One-time defense artifact.
Item C also references a key-rotation runbook (E in the cover
letter) that's lower priority — opine when convenient.
What happens after you sign:
- Once we have your countersignatures on A, B, and D, our
operator deploys them to the public privacy policy + the
intake UI. Engineering will hash the canonical signed text
of (B) and add that hash to the gateway's consent_versions
allowlist file (/etc/lakehouse/consent_versions.json) so the
runtime starts refusing any consent record that wasn't
signed under the canonical template. (The gateway is
currently in permissive mode — accepting any non-empty
hash. Strict mode flips on with your signature.)
- The runtime gates are already enforcing the consent flow:
photo upload refuses 403 unless consent.biometric.status =
"given", and the only path to that state is the
/biometric/subject/{id}/consent endpoint with a valid
consent_version_hash. After your signature, that hash will
point at canonical signed text — operator typo-resistant
and counsel-defensible.
Calendar:
- We'd like to begin real-photo intake within [N] weeks. If
your review can land within 2-3 weeks of receipt, that
keeps us on schedule. If your bandwidth requires longer,
please flag — we have eng work to advance independently
that doesn't depend on the signatures.
Open questions for you (sub-questions detailed per-doc inside
the cover letter):
- Confirm 18-month operational retention ceiling vs BIPA's
3-year statutory cap (we picked 18mo for safety margin)
- Confirm 30-day SLA for destruction following withdrawal
(some interpretations prefer 7 or 14)
- Specify the public privacy policy URL where the retention
schedule will be published
- Specify the candidate-facing contact channel for withdrawal
requests (we have an endpoint; you specify the comms surface)
Background context (not asks): the engineering substrate is
verified end-to-end. We've run a live demo of the full lifecycle
(consent → photo → withdraw) on a test subject; the audit chain
verifies cryptographically; the retention sweep is scheduled
daily. No real candidate photos have been collected. The
attached pre-identityd attestation has the cryptographic evidence
hashes from that pre-collection state.
Happy to schedule a call if any of this would be easier to walk
through verbally.
Thanks,
J
[contact info]
```
---
## Operator pre-flight checklist (before sending)
Run through this once before clicking send:
- [ ] Confirm tarball + manifest are the latest versions
(`bundle_counsel_packet.sh` regenerates if needed; current
hash is in the manifest sidecar)
- [ ] Confirm counsel name is correct (and that the engagement
letter or retainer is in place)
- [ ] Confirm the [N] weeks calendar target — replace with a
real number ("3 weeks" / "by end of June" / etc.)
- [ ] Confirm the email goes to counsel's secure channel; if
counsel uses a portal, upload there + send a notification
email instead
- [ ] Save a copy of the sent email + the tarball to operator
records — this is itself part of the audit trail of "we
asked counsel on [date]"
---
## Post-signature operator runbook (after counsel responds)
When counsel returns the signed documents:
1. **Verify integrity.** Run `sha256sum -c manifest.txt` on
the returned tarball if counsel returns the bundle; otherwise
compare returned signed PDFs against the markdown sources to
confirm the binding text matches what counsel reviewed.
2. **Capture the canonical signed text.** Counsel may sign a
PDF rendering of the markdown — that's fine, but the BINDING
TEXT we hash is the markdown source (counsel commits to git
countersigning §7 of the consent template + §8 of the
retention schedule).
3. **Compute + seed the consent_version_hash.** Run:
```bash
./scripts/staffing/seed_consent_version.sh \
docs/policies/consent/biometric_consent_template_v1.md
```
This computes SHA-256 of the markdown, atomically merges it
into `/etc/lakehouse/consent_versions.json` (creating the
file if absent), and offers to restart the gateway.
4. **Verify strict mode is live.** Probe with a known-bad hash:
```bash
TOKEN=$(cat /etc/lakehouse/legal_audit.token)
curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \
-H "X-Lakehouse-Legal-Token: $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"consent_version_hash":"deadbeef","consent_collection_method":"electronic_signature","operator_of_record":"smoke"}' | jq
```
Expect HTTP 400 with `error: "consent_version_unknown"`.
5. **Update STATE_OF_PLAY.md** to record the signature event +
the hash entry in the allowlist. Counsel-tier deployment is
now live.
6. **Sign and anchor the §2 attestation.** Both J and counsel
sign `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md`
and the SHA-256 of the signed markdown is committed to the
git history as a tamper-evident anchor.
After step 5, the production cutover blocker is closed.
@@ -0,0 +1,260 @@
# Counsel Review Packet — Phase 1.6 BIPA Pre-Launch
**Date assembled:** 2026-05-05
**For:** outside counsel
**From:** J, operator of record
**Scope:** documents that engineering has staged for legal sufficiency review
before the staffing platform begins collecting any real candidate
biometric data (BIPA §15(a)(b)).
> **What this packet is.** The Phase 1.6 BIPA gates outline what
> engineering must ship before real-photo intake. As of 2026-05-05,
> all engineering substrate is shipped and verified live (see §1
> below for the inventory). What remains is binding-text authoring
> + counsel sign-off on five documents, plus operational notification
> obligations counsel may want to layer on top.
>
> **What this packet is NOT.** Not a request for counsel to write
> binding text from scratch. The documents are eng-staged in
> reasonable plain language; the request is for counsel to render
> them into legally-sufficient text and attest where signatures
> are required.
---
## 1. Engineering substrate — shipped + verified
For factual context on what counsel is reviewing AGAINST. None of
this requires sign-off here; it's the system the documents bind to.
| Component | Where it lives | Verification |
|---|---|---|
| Subject manifest registry | `crates/catalogd/src/registry.rs`, `data/_catalog/subjects/<id>.json` | 17 unit tests + 100 backfilled WORKER manifests in production |
| Per-subject HMAC audit chain (SHA-256) | `crates/catalogd/src/subject_audit.rs`, `data/_catalog/subjects/<id>.audit.jsonl` | Tamper-detection + concurrent-append race tests pass |
| Photo upload (consent-gated) | `POST /biometric/subject/{id}/photo` | 11 unit tests + live roundtrip 200 |
| Erasure (two-scope) | `POST /biometric/subject/{id}/erase` (`biometric_only` / `full`) | 21 unit tests; transactional rollback on audit failure |
| Legal-tier audit read | `GET /audit/subject/{id}` (X-Lakehouse-Legal-Token header) | Constant-time auth, chain re-verification per request |
| Retention sweep (BIPA-aware clock) | `crates/catalogd/src/bin/retention_sweep` | 8 unit tests; live verified against 100 backfilled subjects |
| Cross-runtime parity (Rust ↔ Go) | `scripts/cutover/parity/subject_audit_parity.sh` | 6/6 byte-identical assertions pass |
**Key insight for counsel:** the audit chain is the BIPA-defensible
substrate. Every state-changing event (consent given, photo uploaded,
photo erased, legal-tier read) appends to a per-subject HMAC-chained
JSONL log. The chain verifies end-to-end on every legal-tier read.
A tampered chain is detectable; a forged chain requires the HMAC
signing key, which is held under root-only mode 0400 at
`/etc/lakehouse/subject_audit.key` and rotated per the runbook in
attachment §6 below.
**Gate 3b (deepface classification) — decided 2026-05-05: Option C
(defer).** The system collects only the photograph, not derived
demographic information. The consent template + retention schedule
in this packet were revised the same day to match.
---
## 2. Documents requiring counsel review + sign-off
In recommended review order:
| # | Document | Path | Counsel ask | Sign-off |
|---|---|---|---|---|
| A | Biometric Retention Schedule v1 | `docs/policies/consent/biometric_retention_schedule_v1.md` | Render into binding language; confirm 18-month operational ceiling vs. BIPA 3-year statutory cap | Counsel + J |
| B | Biometric Consent Template v1 | `docs/policies/consent/biometric_consent_template_v1.md` | Render Disclosures 1-3 into binding consent language; specify electronic vs. paper signature mechanism | Counsel + J |
| C | BIPA Destruction Runbook | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` | Confirm 30-day SLA from trigger; confirm two-operator (operator + witness) requirement; confirm legal-hold check procedure | Counsel attestation |
| D | BIPA Pre-IdentityD Attestation | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` | Sign as countersigning party; J signs as operator-of-record | Counsel + J |
| E | Legal-Tier Audit Key Rotation | `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` | Confirm rotation cadence; opine on candidate-notification obligation when rotation is compromise-driven | Counsel notes |
| F | Gate 3b Deepface Design (FYI) | `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` | Decision-of-record showing classifications were *deliberately deferred*, not omitted by oversight. No sign-off needed; provided for audit-trail completeness. | None |
The five documents requiring sign-off are A, B, C, D, E. Document F
is included so the audit trail shows the Gate 3b decision was
deliberate.
---
## 3. Specific questions for counsel — by document
### Document A — Retention Schedule
1. The schedule sets an **18-month** operational ceiling against the
BIPA 3-year statutory cap. Is the safety margin appropriate, or
should we move to a tighter window (12 months) given the
plaintiff-friendly Illinois posture?
2. The schedule references the **catalogd-local** storage substrate
rather than a separate identityd Postgres table. Does the
public-facing language need to mention the storage architecture
at all, or is "we keep the photo and a SHA-256 hash" sufficient?
3. Public publication URL — counsel to specify (placeholder marked
in §7 of the schedule).
4. Confirm whether existing consent under v1 carries forward when
a future v2 is published, or whether re-consent is required.
### Document B — Consent Template
1. Disclosure 1 says "we do NOT run automated facial-classification
in v1." Does that disclosure need to mention the *possibility* of
future classification, or is silence-with-supersession-clause
adequate?
2. Plain-language summary in §1 — counsel to confirm it's appropriate
to include alongside the binding disclosure, or recommend an
alternative comprehension aid.
3. Withdrawal SLA is set to **30 days** in §2. Counsel to confirm
against jurisdiction (Illinois primary; secondary deployments
would inherit).
4. Contact for withdrawal — counsel to specify the channel
(placeholder in §3).
5. Sign-off mechanism: electronic signature service, in-app
click-acceptance with timestamp, paper form? Each has different
evidentiary weight.
### Document C — Destruction Runbook
1. Confirm 30-day SLA from each of four triggers (retention expiry,
consent withdrawal, RTBF, court order). Some interpretations
prefer 7 or 14 days for withdrawal/RTBF.
2. Two-operator requirement (operator-of-record + witness): is the
witness role acceptable for counsel's defensibility view, or
should we elevate to dual-control with cryptographic split-key?
3. Legal-hold check procedure (§2 step 3) — counsel to specify the
actual procedure for confirming no hold is in force before
erasing.
4. Backup-window disclosure (§4) — confirm 30-day backup retention
is acceptable.
5. Candidate notification template (§3 step 4) — counsel to supply.
### Document D — Pre-IdentityD Attestation
1. Both signature lines blank — J signs as operator-of-record;
counsel signs as the countersigning legal party.
2. The attestation hash anchors the evidence; once signed, the
hash itself becomes a tamper-evident witness. Counsel to confirm
storage location for the signed copy (firm files?).
### Document E — Key Rotation Runbook
1. Recommended rotation cadence — 90 days suggested in §1.
Counsel to confirm or override.
2. Custody schedule for `/etc/lakehouse/_archived/` raw key files —
§7.2 question; suggested 1-year retention but counsel-driven.
3. Candidate-notification obligation when rotation is
compromise-driven (§7.3) — counsel call.
---
## 4. Engineering changes counsel should know about (recent)
These reconciled doc/code drift after a rapid wave on 2026-05-03:
- **Endpoint paths:** the original v1 spec proposed
`/v1/identity/subjects/*` under a separate identityd daemon. That
daemon was collapsed into catalogd; endpoints actually shipped at
`/biometric/subject/*` (catalogd-local). Documents in this packet
reference the catalogd-local routes; legacy references in
`IDENTITY_SERVICE_DESIGN.md` are flagged "do NOT implement
as-written" in that doc's deprecation header.
- **No identityd Postgres database:** the original spec proposed
encrypted-at-rest Postgres + HashiCorp Vault + S3 Object Lock for
PII storage. The shipped substrate is local JSON manifests +
per-subject HMAC-chained JSONL, sized for J's local-only
deployment per `PRD.md` line 70 ("Everything runs locally — no
cloud APIs").
- **Gate 3b deferral (Option C, 2026-05-05):** classifications
(gender / race / age inference) were deliberately deferred. The
consent template and retention schedule in this packet do NOT
disclose collection of derived demographic data, because we are
not collecting it. If a future product requirement reverses this,
we will publish a v2 consent + v2 retention with re-consent.
- **Key rotation 2026-05-05:** the prior `LH_SUBJECT_AUDIT_KEY` was
lost when a `/tmp` wipe on reboot disabled the audit and biometric
endpoints. The new key is at `/etc/lakehouse/subject_audit.key`
(mode 0400). Pre-rotation audit chains tamper-detect under the
new key — this is correct, expected behavior, not a bug.
---
## 5. Open eng items NOT awaiting counsel
For transparency. These are engineering work items, not legal items:
1. **Residual photo unlink on erasure.** During verification of the
one historical erasure event (`WORKER-2`), the verify script
surfaced a stranded photo file that was not unlinked when
`BiometricCollection` was cleared from the manifest. Engineering
investigates; if the bug is real, the fix is `crates/catalogd/src/biometric_endpoint.rs`
in the erasure handler. This does NOT affect the current packet —
no real candidate photos have been collected yet (per §1
attestation), so the residual is from a synthetic test event.
2. **Phase 1.6 §3 employee training.** Currently deferred per
acknowledgement coverage in §7 of the destruction runbook
(single-operator population). Re-promotes to blocking if the
operator population grows; counsel may want to opine on the
threshold.
---
## 6. Sign-off sequence
Recommended order so a hold-up on one doc doesn't block others:
1. **First wave (parallel):** A (retention schedule) + B (consent
template). These two have the tightest interdependence (consent
v1 references retention v1 by hash); review them together.
2. **Second wave:** C (destruction runbook). Depends on A's retention
period being fixed.
3. **Third wave:** D (pre-identityd attestation). Sign once A + B + C
are settled; the attestation snapshot is the boundary between
pre-Phase-1.6 and post-Phase-1.6 system state.
4. **Fourth wave:** E (key rotation). Independent of A-D; can be
reviewed in parallel any time.
---
## 7. After sign-off — engineering steps
Once each document is signed:
| Document | Engineering action | Trigger |
|---|---|---|
| A retention schedule | Hash + commit; reference in `consent_versions` table | Counsel signature |
| B consent template | Hash + commit; reference in candidate-facing intake UI | Counsel signature |
| C destruction runbook | Adopt; operator acknowledgment recorded in §7 | Counsel attestation |
| D pre-identityd attestation | Anchor hash to filesystem + git; counsel keeps original signature | Both signatures |
| E key rotation | Adopt; rotation event log seeded with counsel-approved cadence | Counsel notes |
The HARD blocker for first real-candidate photo collection is
A + B + D signed. C and E are operationally important but do not
block the *first* photo (they govern destruction + key handling
which apply to any state, not the boundary state).
---
## 8. Cover-note hash
This packet is itself a snapshot. Future-Claude / future-J will refer
back to this packet to know what counsel saw on 2026-05-05.
**Packet attached files (referenced by path):**
- `docs/policies/consent/biometric_retention_schedule_v1.md`
- `docs/policies/consent/biometric_consent_template_v1.md`
- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
- `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md`
- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`
- `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`
- `docs/PHASE_1_6_BIPA_GATES.md` (the spec they all reference)
Per-file SHA-256 hashes are produced by the bundler script (next
section); the bundler also creates a tarball ready for transmission.
---
## 9. Generating the bundle for transmission
```bash
./scripts/staffing/bundle_counsel_packet.sh
```
Produces `reports/counsel/counsel_packet_<DATE>.tar.gz` with all
referenced documents + a manifest listing per-file SHA-256 hashes.
Counsel can verify file integrity on receipt by re-running
sha256sum against each file in the tarball.
@@ -0,0 +1,175 @@
# Biometric Information Consent — v1
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 2 (BIPA §15(b)(1)-(3))
**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before deployment
**Version:** v1 (initial; supersession requires a new version + new hash)
**Updated 2026-05-05:** Disclosure 1 + plain-language summary revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). Pending J's product confirmation of Gate 3b; if Gate 3b chooses Option A or B, this template needs counsel re-authoring.
> This is the consent template a candidate signs (electronically or
> on paper) BEFORE Lakehouse collects, stores, or processes any
> biometric identifier or biometric information from that candidate.
>
> Without an executed consent under this template (or a counsel-
> approved successor), the system MUST NOT accept a photograph from
> the candidate. Enforcement lives at the photo-upload endpoint
> (Gate 3) and at the SubjectManifest writer, which refuses biometric
> writes when `consent.biometric.status != "given"`.
---
## Required disclosures (BIPA §15(b)(1)-(3))
The disclosures below are MANDATORY content per 740 ILCS 14/15(b).
⚖ COUNSEL — render this content into binding language appropriate
for the candidate-facing UI. Engineering provides the structural
content; counsel provides the legally-sufficient wording.
### Disclosure 1 — Notice of collection (§15(b)(1))
Lakehouse will collect and store my **biometric identifier** (a
photograph of me from which facial geometry is implicit). The
photograph itself is the data we keep — we do NOT run automated
facial-classification (gender / race / age inference) against it
in v1. If at a later date we add automated classification, we will
re-collect consent under a superseding template before doing so.
### Disclosure 2 — Specific purpose and length of term (§15(b)(2))
The biometric data will be used for:
1. Identity verification at staffing job sites
2. Internal record-keeping so coordinators can recognize me across
placements
The biometric data will be retained for a maximum of **18 months**
from my most recent interaction with the staffing platform, after
which it will be permanently destroyed per the
[Biometric Retention Schedule v1](biometric_retention_schedule_v1.md).
I may withdraw this consent at any time by contacting the operator
(see §3 below). Withdrawal triggers permanent destruction of my
biometric data.
### Disclosure 3 — Written release (§15(b)(3))
I provide a written release authorizing Lakehouse to collect, store,
and use my biometric identifier and biometric information for the
purposes stated above and for the term stated above.
---
## 1. Plain-language summary (non-binding)
⚖ COUNSEL — the section above is the binding legal disclosure.
The summary below is provided for candidate comprehension and is
NOT a substitute for the binding disclosure. Both should appear
together in the consent UI; counsel determines whether this summary
is appropriate to include or whether a different plain-language
section is preferred.
> **What you're agreeing to:** if you upload a photo of yourself,
> we'll keep that photo so your staffing coordinator can recognize
> you when you arrive at job sites. We don't run automated guesses
> about your age, gender, or race against the photo.
>
> **How long we keep it:** at most 18 months after your last
> placement or interaction with us, then it's permanently destroyed.
>
> **What we DON'T do with it:** we don't sell it, we don't share it
> with anyone outside the staffing operation unless legally compelled,
> and we don't use it to decide what jobs to recommend to you.
>
> **How to take it back:** contact us (§3 below) at any time to
> withdraw your consent. We will permanently destroy your biometric
> data within 30 days of receiving your request.
---
## 2. Withdrawal procedure
I may withdraw biometric consent at any time. Withdrawal:
- Is free of charge
- Does not affect my ability to remain on the staffing platform
(only my biometric data is removed)
- Triggers permanent destruction of all biometric data within
30 days, per the destruction runbook
- Is recorded as an append-only audit row in my per-subject
audit log, providing me with tamper-evident proof of withdrawal
if I subsequently exercise my BIPA right of action
⚖ COUNSEL — confirm 30 days is the right destruction SLA. Some
deployments use 7 or 14 days. The runbook (Gate 5) currently
references this template's number, so changing it here updates
both.
---
## 3. Contact for withdrawal / questions
⚖ COUNSEL — supply the candidate-facing contact channel for
biometric-consent withdrawal. Examples: a dedicated email
(`biometric-consent@<deployment-domain>`), a postal address, a
named operator. The contact must be functional from day one of
deployment.
---
## 4. Consent acknowledgment
By signing below (electronically or on paper), I acknowledge that:
1. I have read and understood the disclosures in §1-3 above
2. I am providing this consent voluntarily and free of coercion
3. I have received a copy of this consent template (or have been
provided a means to retrieve a copy at any time)
| Field | Value |
|---|---|
| Candidate name | _______________________________ |
| Date | __________ |
| Signature | _______________________________ |
| Consent template version | v1 (SHA-256: _generated at deployment time_) |
---
## 5. Operational integration
The structured fields the consent UI must capture and post to
identityd:
```json
{
"candidate_id": "<token>",
"consent_version_hash": "<sha256 of this file at deployment>",
"consent_given_at": "<ISO-8601 timestamp>",
"consent_collection_method": "<electronic_signature|paper|click_acceptance>",
"consent_collection_evidence_path": "<path to signed artifact, if applicable>"
}
```
These fields write to `SubjectManifest.consent.biometric.status='given'`
and the corresponding `SubjectAuditRow` (see
`crates/catalogd/src/subject_audit.rs`).
---
## 6. Versioning
This consent template is version v1. Per Gate 1's versioning rules,
any change to the binding disclosure language requires a new version,
and existing subjects retain their original consent_version reference
unless they re-consent under the new version.
⚖ COUNSEL — confirm whether existing consent under v1 carries forward
when the schedule is updated, or whether re-consent is required.
This affects the deployment workflow.
---
## 7. Authority
| Role | Name | Signature | Date |
|---|---|---|---|
| Operator | J | _______________ | _____ |
| Outside counsel | _____________ | _______________ | _____ |
@@ -0,0 +1,155 @@
# Biometric Data Retention Schedule — v1
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 1 (BIPA §15(a))
**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before public publication
**Version:** v1 (initial; supersession requires a new version + new hash)
**Updated 2026-05-05:** §1 + §2 revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). §5 destruction-trigger endpoint corrected to the shipped catalogd-local route. Pending J's product confirmation of Gate 3b.
> This is a publicly-available retention schedule for biometric identifiers
> and biometric information collected by the Lakehouse staffing platform.
> It is required by 740 ILCS 14/15(a) (the Illinois Biometric Information
> Privacy Act) before any biometric collection from real candidates begins.
---
## 1. What this schedule governs
This schedule applies to:
- **Biometric identifiers** as defined in 740 ILCS 14/10: candidate
photographs from which facial geometry is implicit.
- **Biometric information** as defined in 740 ILCS 14/10: any information
derived from a biometric identifier. **In v1 of this schedule, no
derived information is collected** — automated facial-classification
(gender, race, age inference) is deferred per
`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` Option C. If a future version
of this schedule introduces classification, that is a superseding
v2 schedule with re-consent under the matching v2 consent template.
**Out of scope** (explicitly NOT biometric data under this schedule):
- Synthetic faces from the pre-existing face pool (`data/headshots/`).
These are computer-generated portraits, not derived from any real
individual, and are not "biometric identifiers" under 740 ILCS 14/10.
- Candidate names, email addresses, phone numbers, work history,
certifications, or any other non-biometric personal information.
These are governed by the general PII retention policy referenced
in the SubjectManifest substrate (see
`docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md`).
---
## 2. Categories collected
| Category | Source | Storage location |
|---|---|---|
| Photograph (raw bytes) | Candidate upload via the consent-gated photo endpoint | Quarantined under `data/biometric/uploads/<candidate_id>/<ts>_<uuid>.<ext>`; mode 0700 dir / 0600 file |
| Photograph integrity hash | SHA-256 of the original bytes | `SubjectManifest.biometric_collection.template_hash` (catalogd JSON manifest at `data/_catalog/subjects/<id>.json`) |
We do NOT collect raw biometric template vectors that could be used
to re-derive a face from the encoded form. We do NOT run automated
facial-classification (gender, race, age inference) in v1 — see
`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` for the deferral rationale.
The `BiometricCollection.classifications` field on the subject
manifest exists in the schema but is `None` for every subject.
---
## 3. Purpose of collection
Photographs and the classifications derived from them are used for:
1. **Identity matching during staffing operations.** When a worker
arrives at a job site, the assigned coordinator may verify identity
by comparing the on-file photograph against the person present.
2. **Internal record-keeping.** Photographs become part of the worker
record so coordinators can recognize repeat workers across multiple
placements.
Photographs and biometric classifications are NOT used for:
- Demographic targeting in role recommendations (Title VII / IL Human
Rights Act compliance).
- Training of any machine-learning model.
- Sharing with third parties, except as required by court order or
with the candidate's separate written consent.
- Any purpose beyond those enumerated in §3.1-3.2 above.
---
## 4. Retention period
Per 740 ILCS 14/15(a), biometric identifiers and biometric information
must be permanently destroyed when the initial purpose for collection
has been satisfied OR within **three (3) years** of the individual's
last interaction with the private entity, whichever occurs first.
**Operational ceiling:** Lakehouse retains biometric data for a
maximum of **eighteen (18) months** from the candidate's last placement
or last system interaction, whichever is later. This is more
restrictive than the BIPA statutory ceiling and provides a safety
margin against accidental over-retention.
The 18-month clock is enforced by the daily retention sweep
(`crates/catalogd/src/bin/retention_sweep.rs`), which checks
`SubjectManifest.consent.biometric.retention_until` on every subject
and routes overdue subjects to the destruction queue (see Gate 5
runbook).
⚖ COUNSEL — confirm the 18-month operational ceiling is appropriate
for the deployment posture, or specify a different number.
---
## 5. Destruction procedure
Per 740 ILCS 14/15(a), Lakehouse follows the **BIPA Destruction
Runbook** (`docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`) when:
- Retention period under §4 expires
- Candidate withdraws biometric consent under the consent template (Gate 2)
- Candidate exercises a right-to-be-forgotten request
- A catalogd-local `POST /biometric/subject/{id}/erase` is invoked
under legal-tier authentication (shipped `848a458`)
Every destruction event is recorded as an append-only audit row in
the affected subject's per-subject HMAC-chained audit log (see
`crates/catalogd/src/subject_audit.rs`), providing tamper-evident
proof of compliant destruction.
---
## 6. Versioning
This schedule is version v1. Future revisions:
- Require a new version number (v2, v3, ...).
- Are committed to the repository with a `git` history showing the
revision date.
- Are referenced by SHA-256 hash from `consent_versions` table rows
in identityd, so each subject's consent record points unambiguously
at the schedule version that was in force when consent was given.
**v1 SHA-256:** _generated at deployment time by_ `scripts/staffing/hash_consent_v1.sh` _(to be added when this schedule is finalized by counsel)_
---
## 7. Public availability
⚖ COUNSEL — specify the public URL where this schedule will be
published (typically the privacy policy page on the deployment site)
and the disclosure language that links candidates to it from the
intake UI.
---
## 8. Authority
This schedule is adopted under the authority of J (operator of record)
and reviewed by ⚖ COUNSEL. Effective date: **TBD pending counsel
sign-off**.
| Role | Name | Signature | Date |
|---|---|---|---|
| Operator | J | _______________ | _____ |
| Outside counsel | _____________ | _______________ | _____ |
+244
View File
@@ -0,0 +1,244 @@
# BIPA Biometric Data Destruction Runbook
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5 (BIPA §15(a))
**Audience:** Operators (J + named operators with legal-tier credentials)
**Status:** Engineering scaffold — ⚖ COUNSEL must review for legal sufficiency before adoption
> This runbook tells an operator HOW to destroy biometric data when
> a destruction trigger fires. It is a procedural document, not a
> design document. The cryptographic substrate that the destruction
> writes against (per-subject HMAC audit log + tombstone manifests)
> already ships in `crates/catalogd/`.
---
## 1. When this runbook fires
Destruction is mandatory when ANY of the following occurs:
| Trigger | Source signal | SLA |
|---|---|---|
| **Retention expiry** | Daily `retention_sweep` flags `consent.biometric.retention_until < now` | 30 days from sweep flagging |
| **Consent withdrawal** | Candidate submits withdrawal per consent template §2 | 30 days from receipt |
| **Right-to-be-forgotten request** | Candidate submits RTBF request through documented contact channel | 30 days from receipt |
| **Court-ordered erasure** | Legal counsel directs erasure via a documented order | Per court order; default 30 days |
⚖ COUNSEL — confirm 30 days is correct for all four. Some deployments
have stricter contractual or jurisdictional clocks (CCPA: 45 days but
sooner is better; GDPR Art. 17: "without undue delay").
---
## 2. Pre-destruction checks (5 minutes)
Before initiating destruction, the operator MUST:
1. **Verify the trigger.** Cross-reference one of the four sources
above. If the trigger is a candidate-initiated request,
confirm identity per the standard PII verification procedure
(knowledge factor + possession factor; see counsel for the
threshold).
2. **Pull the current subject record.** Hit
`GET /audit/subject/{candidate_id}` with the legal-tier token.
The response includes:
- The current `SubjectManifest` (including `consent.biometric.status`)
- The full HMAC-chained audit log
- `chain_verified: true` (if false, STOP — chain integrity issue
must be investigated before destruction)
3. **Check for legal hold.** ⚖ COUNSEL — if a legal hold can apply
to a subject's data (litigation, regulatory inquiry, subpoena),
document the procedure for checking that no hold is in force
before erasing.
4. **Get the second-operator sign-off.** Per BIPA defensibility,
destruction is a two-operator action (operator-of-record + one
witness). The witness records their attestation in the
destruction-event audit row (§4 below).
---
## 3. Destruction procedure
### Step 1 — Erase via identityd
Invoke the legal-tier erasure endpoint:
```bash
curl -sf -X POST "http://localhost:3100/biometric/subject/${CANDIDATE_ID}/erase" \
-H "Authorization: Bearer $(cat /etc/lakehouse/legal_audit.token)" \
-H "Content-Type: application/json" \
-d '{
"scope": "biometric_only|full",
"trigger": "retention_expiry|consent_withdrawal|rtbf|court_order",
"trigger_evidence_path": "<path to signed artifact>",
"operator_of_record": "<operator name>",
"witness": "<witness name>"
}'
```
The endpoint is **shipped** (commit `848a458`, 21 unit tests). It is
served from catalogd-local at `/biometric/subject/{id}/erase` (the
original v1 spec proposed `/v1/identity/subjects/{id}/erase` under a
separate identityd daemon — that daemon was collapsed into catalogd
per the architecture pivot).
The endpoint exposes two scopes:
- **`scope: "biometric_only"`** — clears `BiometricCollection` from
the SubjectManifest (drops `data_path`, `template_hash`, and
`classifications` together) + securely unlinks the quarantined
photo file. Subject manifest itself remains. Use for retention
expiry / consent withdrawal where only biometric data must go.
- **`scope: "full"`** — full subject erasure (manifest + biometric
files). Use for court-ordered erasure or full RTBF requests.
In both scopes, the audit row is appended BEFORE photo unlink so
the chain has legal proof of intent even if the file delete fails
(transactional rollback on audit failure).
### Step 2 — Append the destruction-event audit row
The erasure endpoint AUTOMATICALLY writes one row to the subject's
per-subject audit log:
```json
{
"schema": "subject_audit.v1",
"ts": "<ISO-8601>",
"candidate_id": "<id>",
"accessor": {
"kind": "biometric_erasure",
"daemon": "identityd",
"purpose": "biometric_erasure",
"trace_id": "<X-Lakehouse-Trace-Id>"
},
"fields_accessed": ["biometric_classifications", "biometric_data_path", "biometric_template_hash"],
"result": "erased",
"prev_chain_hash": "<previous row hmac>",
"row_hmac": "<new chain link>"
}
```
The HMAC chain extends through the erasure event, so the audit
log itself is preserved as anonymous-event proof of compliant
destruction even after the underlying biometric data is gone.
### Step 3 — Verify destruction
Run the verification script:
```bash
./scripts/staffing/verify_biometric_erasure.sh "${CANDIDATE_ID}"
```
⚖ ENGINEERING — script TODO. Acceptance:
- Subject row biometric fields are NULL
- `data/biometric/uploads/${CANDIDATE_ID}/` directory is empty
- Most recent audit log row has `result: "erased"`, `accessor.kind: "biometric_erasure"`
- Chain still verifies (`chain_verified: true`) under the legal-tier endpoint
If any check fails: STOP, do not mark the destruction complete,
escalate to engineering.
### Step 4 — Notify the candidate (when applicable)
For consent-withdrawal and RTBF triggers, the operator notifies
the candidate that destruction is complete. ⚖ COUNSEL — supply
the notification template (typically email; medium and language
are counsel-determined).
---
## 4. Backup window disclosure
Per `IDENTITY_SERVICE_DESIGN.md` v3-B12, biometric data may persist
in encrypted system backups for up to **30 days** after destruction
(rolling backup window). The candidate must be informed of this
when destruction is requested, and the destruction-event audit row
records the backup-window expiry date so the operator knows when
the residual is fully eliminated.
⚖ COUNSEL — confirm whether the 30-day backup window is acceptable
under BIPA. Some interpretations require backups to be addressed
within a shorter window; some accept the operational reality of
backup retention.
---
## 5. Reporting cadence
Monthly, the operator-of-record produces a destruction-events
report:
```bash
./scripts/staffing/biometric_destruction_report.sh \
--month "$(date +%Y-%m)" \
--output reports/biometric/destruction_$(date +%Y_%m).md
```
⚖ ENGINEERING — script TODO. The report aggregates:
- Total destruction events in the month
- Breakdown by trigger (retention / withdrawal / RTBF / court)
- Median time-to-destruction from trigger to completion
- Any failures / escalations
The monthly report is available to outside counsel on request.
It does NOT include candidate-identifying details — only the
counts, timings, and cryptographic attestations of the events.
---
## 6. Audit trail attestation
The per-subject HMAC chain is the cryptographic substrate that
makes destructions defensible after the fact. To produce an
attestation for a specific candidate's destruction:
1. Hit `GET /audit/subject/{candidate_id}` with legal-tier token
2. Confirm `chain_verified: true` and most-recent row has
`accessor.kind: "biometric_erasure"`
3. Cross-runtime verify: the same audit log is byte-identical
under Rust + Go (per `scripts/cutover/parity/subject_audit_parity.sh`)
4. Counsel signs an attestation referencing the audit log's
chain root hash
The chain root hash is itself a tamper-evident anchor. A motivated
insider would need the HMAC signing key (held in a separate location
from the audit logs themselves, per the spec) AND the original
log to forge a clean destruction record — and the cross-runtime
parity probe would catch a forgery that touched only one runtime's
view.
---
## 7. Operator acknowledgment
Operators with legal-tier credentials acknowledge they have read,
understood, and will follow this runbook before being granted access
to the legal_audit token.
| Operator | Date acknowledged | Signature |
|---|---|---|
| J | _____ | _______________ |
| _____ | _____ | _______________ |
⚖ COUNSEL — adopt this acknowledgment as the substrate for §3 of
Phase 1.6 (employee training acknowledgment), or specify a separate
training program.
---
## 8. Change log
- 2026-05-05 — Endpoint path reconciled with shipped state:
`/v1/identity/subjects/{id}/erase` (legacy proposal under a
separate identityd daemon) → `/biometric/subject/{id}/erase`
(catalogd-local, shipped `848a458`). Step 1 manual-fallback
block removed (the endpoint is no longer "TODO"). Two-scope
body shape (`biometric_only` / `full`) documented to match
the implementation.
- 2026-05-03 — Initial scaffold. ⚖ COUNSEL review required before
adoption.
+308
View File
@@ -0,0 +1,308 @@
# Legal-Tier Audit Key & Token Rotation Runbook
**Spec companion:** `docs/PHASE_1_6_BIPA_GATES.md` §2 + `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
**Audience:** Operators with root on the gateway host (J + named operators)
**Status:** Engineering-authored — ⚖ counsel review encouraged before formal adoption
> This runbook covers rotation of the two crypto-credentials that gate
> the Phase 1.6 audit substrate:
>
> 1. **`LH_SUBJECT_AUDIT_KEY`** — the 32-byte HMAC-SHA256 signing key
> that chains every per-subject audit row. If this key changes, all
> pre-rotation chain rows tamper-detect under the new key. That is
> correct, expected, BIPA-defensible behavior — the chain integrity
> it provided pre-rotation remains intact in the archive of the old
> key, and post-rotation chains remain intact going forward.
>
> 2. **`LH_LEGAL_AUDIT_TOKEN`** — the 32+-character bearer token that
> authorizes calls to `/audit/subject/{id}` and
> `/biometric/subject/{id}/erase`. Rotation does NOT touch any audit
> history; only access to the legal-tier endpoints flips.
>
> Both live at `/etc/lakehouse/` (mode 0400, owned by root) and are
> loaded by the gateway via systemd `Environment=` directives in
> `/etc/systemd/system/lakehouse.service.d/audit_env.conf`. They are
> NOT loaded from `/tmp` — a 2026-05-05 reboot incident wiped a
> `/tmp`-resident key and caused `/audit` + `/biometric` to fail-closed
> (which is what they should do); the rotation fix moved them to the
> persistent path.
---
## 1. When to rotate
Rotate **immediately** when any of the following is true:
| Trigger | Urgency | Notes |
|---|---|---|
| Suspected operator credential compromise | Within 1 hour | Token mismatch is fail-closed by default; immediate rotation closes the window. |
| Operator with legal-tier access leaves the team | Within 24 hours | Treat as compromise. |
| Key/token file's filesystem permissions were ever weakened (mode > 0400, group readable, etc.) | Within 24 hours | Filesystem audit may have leaked the bytes. |
| Token was ever transmitted over an untrusted channel (printed in CI log, sent over SMS, etc.) | Within 24 hours | Same reasoning. |
| Scheduled rotation (recommended) | Every 90 days | BIPA does not mandate a rotation cadence; counsel may set one. |
Do **not** rotate when:
- A subject's audit chain tamper-detects in isolation. That is normal
if the audit log was edited (which would itself be the BIPA finding,
not the key). Investigate the chain, not the key.
- Cross-runtime parity drift appears. That's an HMAC-input-shape bug
(Go vs Rust serialization), not a key issue. See
`STATE_OF_PLAY.md` "three runtime-divergence classes" entry.
---
## 2. Pre-rotation checks (5 minutes)
Before generating new credentials, capture a clean baseline so you can
prove the rotation cause and sequence afterward.
### 2.1. Take the engineering snapshot
```bash
# Confirm the canonical files exist with correct permissions.
ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
# Hash the existing key + token (NEVER the bytes themselves) so the
# old credential is identifiable in retrospect without storing it.
sha256sum /etc/lakehouse/subject_audit.key
sha256sum /etc/lakehouse/legal_audit.token
# Confirm the gateway is currently using these files.
sudo systemctl cat lakehouse.service | grep -E "Environment.*AUDIT"
# Verify the audit endpoint is healthy with the current credentials.
curl -sf http://localhost:3100/audit/health
```
If `/audit/health` is already 503, the rotation is **recovery**, not
preventive — note this in the rotation event record (§5).
### 2.2. Capture a known-good chain root
Pick one or two subjects with non-empty audit logs and record their
chain roots **under the current key**:
```bash
TOKEN=$(cat /etc/lakehouse/legal_audit.token)
for cid in WORKER-2 WORKER-100; do
curl -sf -H "X-Lakehouse-Legal-Token: $TOKEN" \
"http://localhost:3100/audit/subject/$cid" \
| jq '{cid: .candidate_id, verified: .audit_log.chain_verified, root: .audit_log.chain_root, rows: .audit_log.chain_rows_total}'
done
```
Save the output. Post-rotation, those chains will tamper-detect under
the new key — that is **expected** and the saved snapshot is the proof
that the chain WAS intact under the old key, before rotation.
---
## 3. Generation + rotation
### 3.1. Generate the new key
```bash
# 32 random bytes as hex = 64 chars. Either format works for HMAC-SHA256;
# we follow the existing convention (44-char base64-ish with no padding).
sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
/etc/lakehouse/subject_audit.key.new
sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
/etc/lakehouse/legal_audit.token.new
# Sanity: confirm 44-char content + correct mode.
sudo wc -c /etc/lakehouse/subject_audit.key.new /etc/lakehouse/legal_audit.token.new
sudo ls -la /etc/lakehouse/*.new
```
Both must be `mode 0400`, owned by root, exactly **44 chars** (the
audit endpoint refuses tokens shorter than 32 chars at load — see
`crates/catalogd/src/audit_endpoint.rs:73`).
### 3.2. Atomic swap
The gateway reads these files **once at boot** (per
`crates/catalogd/src/audit_endpoint.rs::AuditEndpointState::new` and
the equivalent for the writer). Atomic mv → restart is required.
```bash
# Move the old credentials to a quarantine path with timestamp so the
# old hashes remain identifiable post-rotation.
TS=$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p /etc/lakehouse/_archived
sudo install -d -m 0700 -o root -g root /etc/lakehouse/_archived
sudo mv /etc/lakehouse/subject_audit.key /etc/lakehouse/_archived/subject_audit.key.$TS
sudo mv /etc/lakehouse/legal_audit.token /etc/lakehouse/_archived/legal_audit.token.$TS
sudo mv /etc/lakehouse/subject_audit.key.new /etc/lakehouse/subject_audit.key
sudo mv /etc/lakehouse/legal_audit.token.new /etc/lakehouse/legal_audit.token
sudo ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
```
### 3.3. Restart the gateway
```bash
sudo systemctl restart lakehouse.service
sleep 2
sudo systemctl status lakehouse.service --no-pager | head -10
```
Wait for the gateway to bind port 3100 cleanly. If it doesn't, check
`journalctl -u lakehouse.service -n 50 --no-pager` for the failure
mode — the most common cause is the new file having wrong mode/owner.
---
## 4. Post-rotation verification (5 minutes)
### 4.1. Health probes
```bash
# Audit endpoint must be 200, not 503.
curl -sf http://localhost:3100/audit/health
# Expect: "audit endpoint ready"
# /v1/health must list the gateway's full provider set.
curl -sf http://localhost:3100/v1/health | jq '.providers, .worker_count'
```
### 4.2. Confirm the new token works
```bash
NEW_TOKEN=$(cat /etc/lakehouse/legal_audit.token)
curl -sS -o /dev/null -w '%{http_code}\n' \
-H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-100
# Expect: 200
```
If 401, the file the gateway loaded does NOT match the file you wrote.
Check ownership / mode / for trailing whitespace differences with
`hexdump -C /etc/lakehouse/legal_audit.token | head`.
### 4.3. Confirm the new chain works
Append-only chains are key-tied. Any *new* audit row written
post-rotation is signed under the new key and verifies cleanly:
```bash
# Issue a /v1/validate call against any worker — it spawns an audit row.
curl -sf -X POST http://localhost:3100/v1/validate \
-H 'Content-Type: application/json' \
-d '{"mode":"fill","candidate_id":"WORKER-100","worker_id":"WORKER-100","fields":["exists"]}' >/dev/null
# Read the chain back. Last row must verify under the new key.
curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-100 \
| jq '.audit_log | {verified: .chain_verified, rows: .chain_rows_total, last_kind: .rows[-1].accessor.kind}'
```
`chain_verified: true` confirms the new key is signing + verifying.
### 4.4. Confirm pre-rotation chains tamper-detect (expected)
```bash
curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-2 \
| jq '.audit_log | {verified: .chain_verified, error: .chain_verification_error}'
```
For any subject whose chain was written under the old key, this
returns `chain_verified: false` with an HMAC-mismatch error. **This
is correct behavior**, not a bug. The old chain was correctly signed
under the old key and verified under it; the new key cannot retroactively
verify rows it didn't sign. The pre-rotation snapshot you captured in
§2.2 is the defensible proof that those rows WERE valid pre-rotation.
If, instead, you see a chain that *should* verify post-rotation
returning `verified: false`, that's the rotation having gone wrong —
likely an old-key file that didn't get archived cleanly. Restore from
`/etc/lakehouse/_archived/<ts>/`, then re-attempt.
---
## 5. Record the rotation event
Append a row to the rotation log:
```bash
sudo tee -a /etc/lakehouse/_archived/rotation_log.jsonl <<EOF
{"ts":"$(date -u +%Y-%m-%dT%H:%M:%SZ)","operator":"<your name>","reason":"<scheduled|compromise|cred_loss|recovery>","old_key_sha256":"<hash from §2.1>","new_key_sha256":"$(sha256sum /etc/lakehouse/subject_audit.key | awk '{print $1}')","old_token_sha256":"<hash from §2.1>","new_token_sha256":"$(sha256sum /etc/lakehouse/legal_audit.token | awk '{print $1}')","witness":"<witness name or N/A for routine>"}
EOF
sudo chmod 0600 /etc/lakehouse/_archived/rotation_log.jsonl
sudo chown root:root /etc/lakehouse/_archived/rotation_log.jsonl
```
This file is the operator-side record of when the key changed and why.
It does NOT contain the key itself — only hashes — so it is safe to
back up and share with counsel on request.
---
## 6. Recovery from a lost key
If the active `subject_audit.key` is destroyed (filesystem corruption,
accidental delete, /tmp wipe per the 2026-05-05 incident), the gateway
will fail-closed at startup:
- `/audit/subject/{id}` → 503 ("audit endpoint disabled (legal token missing)" or equivalent for the signing key)
- `/biometric/subject/{id}/photo` → 503 (same fail-closed posture)
This is correct behavior — a server that cannot HMAC-sign new audit
rows must not accept new biometric writes.
**Recovery is rotation.** Generate a new key per §3.1, atomic-swap
per §3.2, restart per §3.3, verify per §4. Pre-loss chains tamper-detect
under the new key (the old key is gone — there is no way to verify
them). Treat the loss event as the BIPA-defensible boundary: pre-loss
chain verification was provided by the working key; post-loss new
chains are signed under the new key.
If a counsel-grade attestation of the pre-loss chains is needed, the
`/etc/lakehouse/_archived/` folder contains the historical hashes;
combined with the cross-runtime parity probe (Go reader gives the
same byte-identical view as Rust), the chain history pre-loss is
preservable as long as the on-disk JSONL files were not also lost.
---
## 7. ⚖ counsel notes
These are areas where counsel may want to opine before this runbook
is formally adopted:
1. **Rotation cadence.** BIPA itself does not require periodic rotation;
counsel may set a 90-day schedule to satisfy a separate compliance
posture (SOC2, internal policy).
2. **Custody of `/etc/lakehouse/_archived/`.** The archived hashes do
NOT contain the keys, but the archived raw key files DO. Counsel
may want a more aggressive destruction schedule for the raw archived
keys — say 1 year — to reduce a long-tail compromise surface.
3. **Notification obligations on rotation due to compromise.** §1
triggers a rotation; §1 does not address whether candidates whose
biometric data was protected by the compromised key must be notified.
This is a counsel call.
---
## 8. Operator acknowledgment
| Operator | Date acknowledged | Signature |
|---|---|---|
| J | _____ | _______________ |
| _____ | _____ | _______________ |
---
## 9. Change log
- 2026-05-05 — Initial runbook authored after the /tmp wipe incident
on the same day (key was at `/tmp/subject_audit.key` and was deleted
on reboot, disabling `/audit` + `/biometric` until the key was
regenerated at `/etc/lakehouse/subject_audit.key`). Recovery of
that incident produced a working procedure; this runbook captures
it as the canonical playbook for any future rotation.
+174
View File
@@ -0,0 +1,174 @@
# Gate 3b — Deepface Classification Integration (Design)
**Status:** **DECIDED 2026-05-05 — Option C (defer classifications)** · Original design draft 2026-05-03 morning · **Companion to:** [`PHASE_1_6_BIPA_GATES.md`](../PHASE_1_6_BIPA_GATES.md) Gate 3 · **Depends on:** Gate 3a (photo upload) which is shipped (`f1fa6e4`)
> **Decision summary (2026-05-05):** J accepted Option C. `BiometricCollection.classifications` remains `Option<JSON> = None` in v1. The consent template and retention schedule were revised the same day to remove all "automated facial-classification" language so the disclosed scope matches the implemented scope. If a real product requirement for classifications surfaces later, this doc's Option A (Python subprocess) or Option B (ONNX-in-Rust) is picked up under a v2 consent template + v2 retention schedule.
> **What this is.** Three options for how `BiometricCollection.classifications` (currently `Option<JSON>`, always `None`) gets populated by an automated facial-attribute classifier. Phase 1.6 Gate 3a ships the consent-gated upload + audit chain + transactional rollback; Gate 3b adds the classification step. The substrate is ready — what's missing is the design choice for HOW classification happens.
>
> **Why a design doc instead of building it.** The Python sidecar that historically hosted deepface was deliberately dropped from the hot path on 2026-05-02 (commit `ba928b1` in lakehouse: "drop Python sidecar from hot path; AiClient → direct Ollama"). Restoring an in-process Python pipeline would regress that decision. Three viable paths exist with different tradeoffs; this doc lays them out so the choice is deliberate, not accidental.
---
## What deepface produces
`deepface` (https://github.com/serengil/deepface) is a Python library wrapping multiple face-analysis backends (VGG-Face, FaceNet, OpenFace, etc.). Given an image path, it returns a JSON-shaped result with keys for `age`, `gender`, `dominant_race`, and `emotion` (plus per-class probabilities). For Phase 1.6's BIPA-compliance scope we need `age`, `gender`, `dominant_race` — the labels that go into `SubjectManifest.biometric_collection.classifications`.
Per the runbook (Gate 5 destruction §2 step 3), classifications must be:
- Stored in the SubjectManifest, NOT in a separate analytics table
- Cleared during BIPA erasure (already wired — `biometric_collection = None` clears them along with `data_path`)
- NEVER served to LLMs in conversation context (only to authenticated staffer UI)
---
## Option A — Python subprocess per upload (one-shot, no daemon)
Spawn a `python3` subprocess from the Rust biometric endpoint after the photo is written, run a thin script that imports `deepface.DeepFace.analyze`, captures the JSON, exits. No long-running Python process, no IPC, no shared state. The hot path stays Rust + tokio; the subprocess is invoked exactly once per upload.
**Implementation shape** (~50 LOC additional in `biometric_endpoint.rs`):
```rust
// After photo file is written + manifest committed + audit row appended:
let cls_path = state.storage_root.join(&safe_id).join("_classifications.json");
let _ = tokio::process::Command::new("python3")
.arg("/etc/lakehouse/deepface_classify.py")
.arg(&abs_path) // input photo path
.arg(&cls_path) // output JSON path
.kill_on_drop(true) // don't leak subprocess on cancellation
.status().await;
// Read cls_path back, populate biometric_collection.classifications.
// If subprocess failed/timed out: leave classifications=None, log warning.
// The upload itself succeeded; classification is best-effort.
```
The Python script (`/etc/lakehouse/deepface_classify.py`, ~30 LOC):
```python
#!/usr/bin/env python3
import sys, json
from deepface import DeepFace
result = DeepFace.analyze(sys.argv[1], actions=['age','gender','race'], silent=True)
with open(sys.argv[2], 'w') as f:
json.dump(result, f)
```
**Pros:**
- Smallest integration: ~80 LOC total
- Process isolation: a deepface crash can't take down the gateway
- Easy to disable: rename the script or remove it; classifications just stay `None`
- Matches Phase 1.6 spec literally (deepface IS the named classifier)
- Keeps the hot path Rust-only — subprocess is fired AFTER the response goes out (we can spawn it without awaiting and let it complete async)
**Cons:**
- Per-upload cold start (~3-5 seconds for Python interpreter + deepface model load)
- Operator must keep `pip install deepface` working on the host (one more dependency to track)
- The "no Python in hot path" constraint is bent if we await the subprocess inline
- Reintroduces a Python-dependent code path that the 2026-05-02 sidecar drop was trying to eliminate
**Cost estimate:** 0.5 day to build + 0.5 day to harden timeout/error paths.
---
## Option B — ONNX model in Rust (no Python at all)
Replace deepface entirely with a Rust-native ONNX runtime invocation. Pre-train (or download pre-trained) ONNX models for age + gender + race classification, ship them at `/etc/lakehouse/models/{age,gender,race}.onnx`, run them through `tract` or `ort` (ONNX Runtime Rust bindings). No Python anywhere.
**Implementation shape** (~200 LOC + a model-build pipeline):
```rust
use ort::{Environment, SessionBuilder, Value};
let env = Environment::builder().build()?;
let session = SessionBuilder::new(&env)?.with_model_from_file("/etc/lakehouse/models/age.onnx")?;
let preprocessed = preprocess_image(&abs_path)?; // resize, normalize, etc.
let output = session.run(vec![Value::from_array(preprocessed)?])?;
let age = decode_age(&output[0])?;
// repeat for gender + race
```
**Pros:**
- Zero Python dependency — fully consistent with the 2026-05-02 sidecar drop
- Faster per-call: ~100ms inference (no interpreter cold start)
- Model files are immutable artifacts that can be hashed + signed for provenance
- Audit-friendly: deterministic output for a given input + model hash
- BIPA-defensible: model provenance is auditable; no third-party Python library risk
**Cons:**
- Need pre-trained ONNX models for our three tasks (age regression, gender classification, race classification). Either find existing public models with appropriate licenses OR train ourselves (multi-week)
- Image preprocessing in Rust is non-trivial (face detection + alignment + crop + normalize). May need an additional model (e.g. RetinaFace ONNX) just for face detection
- ~200-400 LOC of new code in a domain we haven't worked in before
- Cross-runtime parity story: Go side would need its own ONNX integration if it ever wants the same surface, OR we define classifications as Rust-only (acceptable since the substrate is Rust-side anyway)
**Cost estimate:** 3-5 days to build + 2 days to validate output quality + license review for any pre-trained models we use. Largest of the three options.
---
## Option C — Defer classifications until a real demand exists
Keep `BiometricCollection.classifications` as `Option<JSON>` permanently `None` until a concrete consumer needs it. The substrate is ready; we don't add the classifier writer.
**What still works without classifications:**
- BIPA consent gate (Gate 3a) ✓
- Photo storage with template_hash for integrity (Gate 3a) ✓
- Audit chain proving collection happened (Gate 3a) ✓
- Erasure flow (Gate 5) ✓
- Retention sweep on biometric clock (Step 7) ✓
**What's missing without classifications:**
- The "demographic data derived from photos" disclosure in the consent template (Gate 2 §1 Disclosure 1) becomes vestigial — we collect photos but don't actually derive any demographic information from them
- The retention schedule (Gate 1) §2 lists "Facial geometry classifications" as a category we collect; without classifications, that category is empty in practice
**Pros:**
- Zero engineering today
- Forces the question "do we actually need classifications?" to be answered by a real product requirement, not by spec inertia
- BIPA-safer in the most absolute sense: we collect strictly the photo, nothing derived from it
- If the real product requirement is "let staffers see the photo at job-site verification," classifications add no value (the staffer's eyes do the verification)
**Cons:**
- Requires updating Gates 1 + 2 to remove the "we derive classifications" language (counsel coordination needed)
- Loses the structured demographic data that audit-of-discrimination workflows might want (per `IDENTITY_SERVICE_DESIGN.md` v3-A4 disparate-impact analysis)
- If we later need classifications, we land on Option A or B then anyway — this is a deferral, not a permanent choice
**Cost estimate:** 0.25 day to update Gate 1 + 2 doc language + ⚖ counsel review of the change.
---
## Recommendation
**Option C (defer) is the right answer right now**, conditional on confirming the real product requirement.
Reasoning:
1. Phase 1.6's load-bearing claim is consent + audit + retention + erasure. Gate 3a + Gate 5 + retention sweep + audit chain provide ALL of those WITHOUT classifications.
2. The riskiest BIPA position is collecting demographic-derived data without a documented business purpose. If staffers identify candidates by looking at the photo (the actual stated use case), classifications are decorative.
3. Option A reintroduces the Python dependency the 2026-05-02 wave deliberately removed. Option B is multi-day work for unclear benefit.
4. The `Option<JSON>` substrate is forward-compatible: if a real demand surfaces (say, disparate-impact analytics J actually wants), we can ship Option A or B then in 1-3 days respectively, and the audit chain naturally captures the introduction event.
**Next action if Option C is chosen:**
- Update `docs/policies/consent/biometric_consent_template_v1.md` Disclosure 1 to remove the "facial geometry classifications" claim — adjust to "facial photograph for staff identification"
- Update `docs/policies/consent/biometric_retention_schedule_v1.md` §2 to remove the classifications row
- Mark Gate 3b in `PHASE_1_6_BIPA_GATES.md` as "deferred — classifications not collected in v1, see GATE_3B_DEEPFACE_DESIGN.md"
- ⚖ counsel review of the doc changes (consent text changes need their attention regardless)
**Next action if Option A is chosen:** ~1 day to ship; document the Python dependency carve-out in STATE_OF_PLAY's "WHAT NOT TO RELITIGATE" so future sessions don't try to remove it again.
**Next action if Option B is chosen:** schedule a multi-day session, source/license pre-trained ONNX models for the three tasks, build face-detection preprocessing in Rust.
⚖ J — pick A / B / C. The substrate accommodates any choice; the cost is the design-doc → counsel-coordination → engineering loop, which differs by an order of magnitude across the options.
**[2026-05-05] J's decision: Option C.** Reasoning recorded in change log below. Consent + retention doc revisions for Option C shipped same day; counsel review of revised text is the remaining work.
---
## Open questions for J
1. **What's the actual product requirement for classifications?** If the answer is "I don't know yet," that's a strong vote for Option C.
2. **If classifications: who is the consumer?** Staffer UI? Disparate-impact dashboard? An LLM context (NO — Phase 1.6 §3 forbids this)?
3. **If Option A: are we OK with `pip install deepface` as a host requirement?** Or does the operator-of-record need a containerized Python sidecar (which puts us close to where we were before 2026-05-02)?
4. **If Option B: who is the model provenance signer?** ⚖ counsel will want a chain of custody from the model author to the ONNX file on disk.
---
## What this doc is NOT
- Not a final decision. The decision is J's after reading the options.
- Not a substitute for testing. Whichever option ships, the parity probe `subject_audit_parity.sh` must continue to pass post-change (the audit chain is the BIPA-load-bearing surface; classifications are additive metadata).
- Not the only Phase 1.6 work remaining. Counsel-side review of Gates 1/2/5 + the §2 attestation signature are the calendar bottleneck. Gate 3b is the engineering bottleneck.
+307
View File
@@ -0,0 +1,307 @@
# Pathway Memory — Specification v1
**Status:** Draft v1 — 2026-05-03 · **Layer:** Decision metadata · **Implementation:** `crates/vectord/src/pathway_memory.rs`
> **What this is.** Pathway Memory is a third metadata layer that exists alongside (not on top of, not under) the data-metadata layer (catalogd's manifests/views/tombstones) and the operational-memory layer (playbook_memory). It tracks **decision patterns**: "this code path, in this task class, produced this outcome." It is the AI-substrate analog of what Iceberg manifests are to data tables — pointers + provenance for things you might want to retrieve again or learn from.
>
> **What it is NOT.** It is not a vector database. It is not a knowledge graph. It is not an audit log (audit is per-subject; pathways are per-code-pattern). It is not a cache. It is a small, opinionated metadata layer with a defined wire format, defined access patterns, and defined lifecycle rules.
>
> **Why a spec exists.** No standard tells us what this layer is. Each implementation that drifts from this spec creates the wiring confusion currently visible in the system (data/_kb/ JSONLs that don't get consumed, version drift across `workers_500k_v1..v10`, unclear who reads `reducer_summary` vs `final_verdict`). The spec is the standardization contract that lets a single implementation become the canonical one.
---
## 1. Conceptual model
A **pathway** is the identity-equivalence-class of code paths that exhibit similar decision behavior. Two distinct files may share a pathway if they belong to the same crate AND are reviewed under the same task class with the same signal class.
A **trace** is a single observed instance of a pathway: one review pass, with its inputs, intermediate signals, ladder attempts, audit outcome, and final verdict.
A pathway is **identified** by a stable hash of three keys: task class, file prefix, signal class. The hash is `pathway_id`.
A trace is **identified** within a pathway by a generated UUID: `trace_uid`. The pair `(pathway_id, trace_uid)` is globally unique.
Pathways are **append-only by trace**: every new observation creates a new trace under the existing pathway_id. Traces themselves are **mutable in version**: `revise()` produces a new version that supersedes the previous (Mem0-style chain).
Pathways are **retirable** by id: when replay history shows the pattern no longer holds, the pathway is marked retired and excluded from hot-swap. New pathways may emerge with fresh ids if the underlying code/task characteristics shift.
---
## 2. Wire format — `PathwayTrace`
Authoritative struct: `crates/vectord/src/pathway_memory.rs`. The fields below are the SPEC; the Rust struct is one implementation. All fields use JSON-serializable types when persisted.
### 2.1 — Identity fields (immutable per trace)
| Field | Type | Constraint |
|---|---|---|
| `pathway_id` | string | SHA256 hex of `task_class | file_prefix | signal_class` — see §3 |
| `trace_uid` | string | UUID. Empty on legacy traces; populated on insert by SHOULD-implementations from v2+ |
| `version` | u32 | Mem0 version chain. Default 1 on insert. Bumped on `revise()`. |
| `parent_trace_uid` | string \| null | trace_uid of the trace this one supersedes. null on root version. |
| `superseded_at` | string \| null | ISO-8601 UTC. Set when this trace becomes a non-head version. |
| `superseded_by_trace_uid` | string \| null | trace_uid of the new head when this version is superseded. |
### 2.2 — Identity-source fields (the inputs that fed pathway_id)
| Field | Type | Notes |
|---|---|---|
| `task_class` | string | Caller-defined class string (e.g. `scrum_review`, `staffing.fill`, `pr_audit`) |
| `file_path` | string | Full path of the file under review. Reduced to file_prefix for hashing — see §3.2. |
| `signal_class` | string \| null | Caller-defined behavior label (e.g. `CONVERGING`, `LOOPING`, `STUCK_RETRY`) |
### 2.3 — Observation fields (what happened)
| Field | Type | Description |
|---|---|---|
| `created_at` | RFC 3339 timestamp | When this trace was inserted. |
| `ladder_attempts` | array of `LadderAttempt` | Each model+rung attempt: `{rung, model, latency_ms, accepted, reject_reason?}`. Order is dispatch order. |
| `kb_chunks` | array of `KbChunkRef` | Knowledge-base chunks the reviewer was given as context: `{source_doc, chunk_id, cosine_score, rank}`. |
| `observer_signals` | array of `ObserverSignal` | Behavior labels emitted by observer during this pathway's run: `{class, priors, prior_iter_outcomes}`. |
| `bridge_hits` | array of `BridgeHit` | External-doc lookups (context7-style): `{library, version, result_summary}`. |
| `sub_pipeline_calls` | array of `SubPipelineCall` | Calls to other sub-pipelines (extract, validate, etc.) made during this trace. |
| `audit_consensus` | `AuditConsensus` \| null | Cross-lineage audit outcome if any: `{pass, models, disagreements}`. |
| `reducer_summary` | string | Free-text summary of what the reducer concluded. **PII risk surface — see §6.3.** |
| `final_verdict` | string | Free-text verdict label (`accepted`, `rejected`, `needs_review`, etc.). **PII risk surface — see §6.3.** |
### 2.4 — Index fields (the matrix retrieval shape)
| Field | Type | Description |
|---|---|---|
| `pathway_vec` | array of f32, length 32 | Bag-of-tokens hash embedding. Algorithm in §4. Fixed dimension per spec version. |
| `replay_count` | u32 | Number of times this pathway has been replayed via hot-swap. Initial insert is NOT a replay. |
| `replays_succeeded` | u32 | Replays where the post-replay outcome matched the trace's `final_verdict`. |
| `retired` | bool | Marked true when probation gate fires (§5.4). Excluded from hot-swap forever. |
### 2.5 — Optional semantic-correctness fields (ADR-021)
| Field | Type | Description |
|---|---|---|
| `semantic_flags` | array of `SemanticFlag` | One of: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`. Caller-emitted, not auto-derived. |
| `type_hints_used` | array of `TypeHint` | Schema/type context fed to the reviewer for this trace: `{source, symbol, type_repr}`. |
| `bug_fingerprints` | array of `BugFingerprint` | Structural pattern hashes the reviewer caught: `{flag, pattern_key, example, occurrences}`. |
All §2.5 fields are additive — implementations MUST default them to empty when deserializing legacy traces.
---
## 3. Identity algorithms
### 3.1 — `pathway_id`
```
pathway_id = sha256_hex(task_class || "|" || file_prefix(file_path) || "|" || signal_class_or_empty)
```
- Concatenation uses literal byte `|` (0x7C) as separator.
- `signal_class_or_empty` is the empty string when signal_class is null.
- SHA-256 output is 64 hex chars, lowercase.
### 3.2 — `file_prefix`
```
file_prefix(path) = first two path segments joined by '/'
```
- Path separator is forward slash. Implementations on Windows MUST normalize to forward slashes before hashing.
- Files with fewer than two segments use the full path verbatim.
- Examples:
- `crates/queryd/src/service.rs` → `crates/queryd`
- `crates/gateway` → `crates/gateway`
- `README.md` → `README.md`
### 3.3 — Why this fingerprint shape
The fingerprint groups files within a crate (same `file_prefix`) under the same pathway when reviewed for the same task class with the same signal class. This is intentional: it lets the matrix index recognize "the same kind of bug appears in this crate" without needing per-file traces. Per-file granularity is preserved in the `file_path` field for retrieval but is not part of the identity.
---
## 4. `pathway_vec` algorithm
Fixed dimension: **32 buckets**. Implementation: deterministic bag-of-tokens hash, normalized.
```
def build_pathway_vec(trace):
buckets = [0.0] * 32
tokens = collect_metadata_tokens(trace) # see token list below
for tok in tokens:
h = sha256(tok.encode())
bucket = int.from_bytes(h[:4], 'big') % 32
buckets[bucket] += 1.0
norm = sqrt(sum(b*b for b in buckets))
return [b/norm if norm > 0 else 0.0 for b in buckets]
```
**Tokens MUST include**:
- `task_class:<task_class>`
- `file_prefix:<file_prefix>`
- `signal_class:<signal_class_or_empty>`
- For each `ladder_attempts[i].model`: `model:<model>`
- For each `kb_chunks[i].source_doc`: `kb_doc:<source_doc>`
- For each `observer_signals[i].class`: `signal:<class>`
- For each `bug_fingerprints[i].flag`: `flag:<flag>`
**Tokens MUST NOT include**:
- `reducer_summary` text (free-form, would dominate the bucket distribution)
- `final_verdict` text
- `created_at` (would make every trace's vector unique)
- candidate_id, name, email, phone, or any subject identifier (PII risk)
**Why 32 dimensions:** consensus-validated round 3 ensemble selected "small metadata tokens, not full JSON." 32 is sufficient to distinguish pathway combinations without requiring an external embedding model. An external embedding model would work too but adds a dependency, failure mode, and drift risk the consensus flagged. **Spec version v1 fixes dim=32. v2 may revisit.**
---
## 5. Lifecycle
### 5.1 — Insert
A trace is inserted by the consumer (typically the scrum pipeline or auditor) calling the implementation's `insert()` API.
- If no pathway with `pathway_id` exists, a new pathway bucket is created.
- If a pathway exists, the new trace is appended to that bucket's traces list.
- `trace_uid` is generated (UUID v7 RECOMMENDED for time-orderability).
- `version` is set to 1.
- `created_at` is set to the insertion timestamp.
- `pathway_vec` is computed and stored.
- `replay_count` and `replays_succeeded` start at 0.
### 5.2 — Revise
A trace is revised by calling `revise(trace_uid, new_fields)`.
- A new trace with a fresh `trace_uid` and `version = previous.version + 1` is inserted.
- The new trace's `parent_trace_uid` is set to the previous trace's `trace_uid`.
- The previous trace's `superseded_at` is set to the revision timestamp and `superseded_by_trace_uid` is set to the new trace's `trace_uid`.
- The previous trace remains in the bucket (history is preserved). Retrieval defaults to head versions only — see §7.
### 5.3 — Replay
A pathway is **eligible for replay** when `retired == false` AND the bucket has at least one head-version trace.
The hot-swap consumer queries by `pathway_id` (or by similarity over `pathway_vec`), retrieves the head trace, applies its `final_verdict` shape to a new but similar input, and then reports back via `record_replay(trace_uid, succeeded: bool)`.
- `replay_count` is incremented on every replay (success or failure).
- `replays_succeeded` is incremented only on success.
### 5.4 — Probation gate (retirement)
Probation triggers when `replay_count >= 3` AND `success_rate < 0.80`.
When triggered, the pathway is marked `retired = true`. It is excluded from hot-swap forever. If the underlying code/task characteristics genuinely change such that a retired pathway would work again, a NEW pathway with a FRESH id will be created (because the inputs to `pathway_id` would have changed, e.g., new signal_class). Retirement is per-id and irreversible.
`success_rate` = `replays_succeeded / replay_count` (0.0 if `replay_count == 0`).
---
## 6. Access patterns (read + write shapes)
### 6.1 — Write paths (who inserts traces)
Spec-conformant writers:
- Scrum pipeline (`tests/real-world/scrum_master_pipeline.ts` and downstream consumers) on review acceptance
- Auditor pipeline (`auditor/`) on verdict assembly
- Validator (`crates/validator/`) on iterate session completion
Each writer is responsible for setting the identity fields correctly. The implementation MUST reject writes where `task_class` is empty.
### 6.2 — Read paths (who consumes traces)
Spec-conformant readers:
- **Hot-swap retrieval** — `query_for_hotswap(task_class, file_path, signal_class, k=5)` returns the top-k head-version traces from matching pathways, ordered by `(success_rate desc, replay_count desc)`.
- **Similarity retrieval** — `query_by_vec(pathway_vec, k=10)` returns the top-k head-version traces by cosine similarity over `pathway_vec`.
- **Audit / forensic** — `list_versions(trace_uid_root)` returns the full version chain. **Must be authorized** — version chains can include reducer_summary text that may contain PII.
The implementation MUST NOT expose `list_versions` on an unauthenticated route.
### 6.3 — PII surface (load-bearing for audit-trail compliance)
The fields `reducer_summary` and `final_verdict` are free-form strings written from review output. They are **highly likely** to contain PII when the reviewed task class involved real candidates (per `AUDIT_PHASE_1_DISCOVERY.md` §1F + §10/C1). Until subject-redaction-on-write lands, implementations:
- MUST treat trace bodies as a suspected PII sink for any task class touching real subjects
- SHOULD redact known PII patterns (candidate_id, email, phone) before persisting
- SHOULD provide a per-trace `subject_ids[]` top-level field when known, so audit-response queries can filter by subject
Until the redaction layer ships, do not advertise pathway_memory as PII-clean.
---
## 7. Storage representation
The current implementation persists state to a single JSON file at `data/_pathway_memory/state.json`:
```
{
"pathways": {
"<pathway_id>": [
{ "trace_uid": "...", "version": 1, "task_class": "...", ... },
{ "trace_uid": "...", "version": 2, "parent_trace_uid": "...", ... }
]
},
"last_updated_at": "2026-04-29T05:43:00Z"
}
```
The flat-JSON representation is implementation-specific. A spec-conformant implementation could use SQLite, an append-only JSONL, or a Lance dataset. The spec dictates the wire shape per-trace and the access patterns; the storage backend is the implementation's choice.
**Retrieval defaults**: queries return head versions only by default (a head version is one with `superseded_at == null`). Consumers explicitly opt in to historical versions via a `include_history: true` flag.
---
## 8. Spec boundary — what's stable vs implementation-specific
### Stable (locked in v1; changing requires v2 spec)
- The `pathway_id` algorithm (SHA256 of `task_class | file_prefix | signal_class`)
- The `file_prefix` algorithm (first two path segments)
- `pathway_vec` dimension (32) and token list
- Field names and types in §2 (additive fields are OK; renames or removals require v2)
- Lifecycle rules in §5
- Access pattern names in §6 (`query_for_hotswap`, `query_by_vec`, `list_versions`)
### Implementation-specific (free to change without spec bump)
- Storage backend (JSON file, SQLite, Lance, etc.)
- HTTP/gRPC/library API surface
- Concurrency model
- In-memory caching strategy
- The exact text of `reducer_summary` (it's content, not schema)
- Token-bucket hash function (SHA-256 specified in §4 but any deterministic 32-bit hash is acceptable as long as it's deterministic and well-distributed)
### Reserved for v2 (named so future implementations don't paint into a corner)
- Cross-pathway similarity (`query_pathways_similar_to_pathway`)
- External embedding model for `pathway_vec` (replaces the bag-of-tokens hash)
- Subject-aware indexing (so audit-response can find all pathways involving subject X)
- Differential privacy on `pathway_vec` aggregations
---
## 9. Open questions
These need decisions before v1 is final-final:
1. **Dimension lock-in**: 32 is round-3 consensus, but is it actually optimal? Worth a benchmark before final v1.
2. **Subject indexing in v1 vs v2**: deferring to v2 means audit queries today have to grep `reducer_summary` text. Is that acceptable for the staffing-client launch window?
3. **Cross-runtime parity**: the Go side has its own `internal/pathways/` implementation. As of 2026-05-03 the wire shape matches but no parity probe exists. Add `pathway_parity.sh` to the cross-runtime probe suite.
4. **Migration from current state.json**: existing 91 traces lack `trace_uid`. Spec says `trace_uid` is empty on legacy traces — should v1 require backfill or accept legacy-trace queries indefinitely?
---
## 10. Why this spec exists
Without it:
- Every new consumer of pathway_memory invents their own read/write conventions
- The Go-side reimplementation drifts from the Rust-side wire format
- Future LLM-assistant work proposes new fields without considering the existing schema
- Forensic queries grep free-text fields because no canonical retrieval exists
- The PII concern stays "TODO" because there's no stable surface to gate
With it:
- One implementation can be canonical (Rust, currently); others can be conformance-tested
- The wire shape is documented; serialization tests can pin it
- New fields go through an additive-only review (don't break legacy)
- The KbContext consumer in the gateway has a stable API to read against
- The audit-trail PRD has a defined PII surface to protect
This is the standardization the system was missing.
---
## Change log
- 2026-05-03 — v1 initial draft. Codifies what's already in `crates/vectord/src/pathway_memory.rs`. The spec is descriptive of the existing implementation, not prescriptive of new behavior.
+266
View File
@@ -0,0 +1,266 @@
# Subject Manifests on Catalogd — Specification v1
**Status:** Draft v1 — 2026-05-03 · **Layer:** Catalogd extension (NOT a separate daemon) · **Implementation:** to be added to `crates/catalogd/src/`
> **What this is.** A small extension to `catalogd` adding a fourth manifest type — `subject` — alongside the existing dataset / view / tombstone / profile types. A subject manifest answers: "for person X, which datasets contain their PII, which views project it safely, what consent + retention applies, and what's the access log."
>
> **What it is NOT.** It is not a separate identity daemon. It is not a Postgres-backed identity service. It is not a HashiCorp-Vault-using KEK rotation system. It is not the `IDENTITY_SERVICE_DESIGN.md` v3 design (that doc is over-scoped for local-only — see its deprecation header). It is the smallest spec that gives you a defensible "show me everything we know about person X" capability for EEOC discovery / BIPA compliance, building on primitives catalogd ALREADY ships.
>
> **Why it can be small.** Catalogd already has dataset manifests with per-column `is_pii` flags. It already has views with `column_redactions` (working example: `candidates_safe.json`). It already has tombstones for deletes. It already has profiles for per-agent scoping. The audit-trail need is "thread these primitives together by subject identifier" — not "build a new system." That's a ~300-500 LOC extension, not a 17-20 day phase plan.
---
## 1. Conceptual model
A **subject** is a real person whose PII flows through the system. Identified by a stable token (current implementation: `candidate_id` in `workers_500k.parquet`).
A **subject manifest** is a JSON record under `data/_catalog/subjects/<candidate_id>.json` that points at:
- which datasets contain rows for this subject (via foreign-key reference)
- which views safely project this subject's data (via existing view manifest names)
- what consent + retention metadata applies to this subject
- what access log file holds this subject's audit trail
Subject manifests are written when a subject enters the system AND updated when a subject's consent changes, vertical reclassifies, or retention period expires.
The **audit log** is a per-subject append-only JSONL file at `data/_catalog/subjects/<candidate_id>.audit.jsonl`. Every PII access for that subject writes one row. The file is signed periodically (HMAC chain) for tamper-evidence.
---
## 2. Wire format — `SubjectManifest`
JSON document at `data/_catalog/subjects/<candidate_id>.json`:
```json
{
"schema": "subject_manifest.v1",
"candidate_id": "CAND-000001",
"created_at": "2026-05-15T12:00:00Z",
"updated_at": "2026-05-15T12:00:00Z",
"status": "active",
"vertical": "general",
"consent": {
"general_pii": {
"status": "given",
"version": "v1-2026-05-15",
"given_at": "2026-05-15T12:00:00Z"
},
"biometric": {
"status": "never_collected",
"retention_until": null
}
},
"retention": {
"general_pii_until": "2030-05-15T12:00:00Z",
"policy": "4_year_default"
},
"datasets": [
{ "name": "workers_500k", "key_column": "candidate_id", "key_value": "CAND-000001" },
{ "name": "candidates", "key_column": "candidate_id", "key_value": "CAND-000001" },
{ "name": "placements", "key_column": "candidate_id", "key_value": "CAND-000001" },
{ "name": "timesheets", "key_column": "candidate_id", "key_value": "CAND-000001" }
],
"safe_views": ["workers_safe", "candidates_safe"],
"audit_log_path": "data/_catalog/subjects/CAND-000001.audit.jsonl",
"audit_log_chain_root": "sha256:..."
}
```
### 2.1 — Field semantics
| Field | Required | Notes |
|---|---|---|
| `schema` | yes | Always `"subject_manifest.v1"`. Validates parser shape. |
| `candidate_id` | yes | Subject identifier. Stable token. Same value as appears in dataset key columns. |
| `status` | yes | `pending_consent` \| `active` \| `withdrawn` \| `retention_expired` \| `erased`. |
| `vertical` | yes | `unknown` \| `general` \| `healthcare` \| `finance` \| `other`. **Default `unknown`**, fail-closed routing treats unknown as healthcare-equivalent. |
| `consent.general_pii.status` | yes | `pending_backfill_review` \| `pending_first_contact` \| `given` \| `withdrawn` \| `expired`. |
| `consent.biometric.status` | yes | `never_collected` \| `pending` \| `given` \| `withdrawn` \| `expired`. |
| `retention.general_pii_until` | yes | ISO-8601. Drives daily expiration sweep. |
| `datasets[].name` | yes | References an existing catalogd dataset manifest by name. |
| `datasets[].key_column` | yes | The column in that dataset that contains the subject's identifier. |
| `datasets[].key_value` | yes | The specific value (the subject's id within that dataset's namespace). |
| `safe_views` | yes | Names of existing catalogd view manifests that safely project this subject's data (for non-legal-tier readers). |
| `audit_log_path` | yes | Relative path to the audit JSONL. |
| `audit_log_chain_root` | yes | SHA-256 of the most recent HMAC-chained checkpoint of the audit log. Updated by the audit-log writer on every write. |
---
## 3. Audit log format
Per-subject append-only JSONL at `data/_catalog/subjects/<candidate_id>.audit.jsonl`. One row per PII access:
```json
{
"schema": "subject_audit.v1",
"ts": "2026-05-15T13:30:00Z",
"candidate_id": "CAND-000001",
"accessor": {
"kind": "gateway_lookup",
"daemon": "gateway",
"purpose": "fill_validation",
"trace_id": "X-Lakehouse-Trace-Id-..."
},
"fields_accessed": ["name"],
"result": "success",
"prev_chain_hash": "sha256:...",
"row_hmac": "hmac-sha256:..."
}
```
### 3.1 — HMAC chain
Each row's `row_hmac` is `HMAC-SHA256(key, prev_chain_hash || canonical_json_of_row_minus_hmac)`. The signing key is loaded once at startup from `/etc/lakehouse/subject_audit.key` (mode 0400). The chain root in the subject manifest references the latest row's `row_hmac`.
A tamper-evident verification is one pass:
```
verify_chain(subject_id):
manifest = read_subject_manifest(subject_id)
rows = read_audit_log(subject_id)
prev = "GENESIS"
for row in rows:
expected = hmac_sha256(key, prev || canonicalize(row - row_hmac_field))
assert row.prev_chain_hash == prev
assert row.row_hmac == expected
prev = row.row_hmac
assert manifest.audit_log_chain_root == prev
```
This is local (no S3 Object Lock, no Vault) but tamper-evident: any modification to a past row breaks the chain at that point and all subsequent rows. The signing key being on disk is a real risk surface — operators MUST set the file mode 0400 owner-only and back it up to a separate location from the audit logs themselves (so that a single backup doesn't carry both the ciphertext and the verification material).
### 3.2 — When the audit log is written
Every code path that resolves PII for a subject MUST write an audit row before returning. Specifically:
- The gateway's tool registry SQL templates (`crates/gateway/src/tools/registry.rs`) — when `search_candidates` / `get_candidate` queries return rows, write one audit row per returned candidate_id
- The validator's WorkerLookup (`crates/validator/src/staffing/parquet_lookup.rs`) — when a `lookup(candidate_id)` succeeds, write one audit row
- The audit-response endpoint (when implemented) — when `/audit/subject/{id}` is called, write one row of `kind=audit_response`
- Any new code path that touches PII
Write failures MUST NOT silently swallow. They MUST be logged at error level (per the existing observability fabric). Write failures MUST NOT block the read — accept the audit gap and flag it for post-hoc review (better to leak a row than block legitimate operations).
---
## 4. The `/audit/subject/{candidate_id}` response
The audit response builds from the subject manifest + audit log + dataset projections:
```json
{
"schema": "subject_audit_response.v1",
"candidate_id": "CAND-000001",
"generated_at": "2026-05-15T15:00:00Z",
"generated_by": "catalogd@hostname",
"manifest": { /* the SubjectManifest */ },
"datasets": {
"workers_500k": {
"row_present": true,
"safe_view_projection": { /* candidates_safe row for this subject */ }
}
},
"audit_log_window": {
"from": "2026-01-01T00:00:00Z",
"to": "2026-05-15T15:00:00Z",
"rows": [ /* matching audit rows */ ]
},
"chain_verification": {
"verified": true,
"rows_checked": 42,
"chain_root": "sha256:..."
},
"completeness_attestation": "all dataset rows + audit log entries within the window per retention policy v1 are included",
"signature": "ed25519:..."
}
```
The endpoint is auth-gated via a separate legal-tier credential (see §6). The response body is signed with an Ed25519 key separate from the HMAC chain key.
---
## 5. Implementation plan (this is the SMALL plan)
This is the spec; the implementation is a small extension to catalogd. Estimated effort:
| Step | Effort | What |
|---|---|---|
| **1** | 0.5d | Add `SubjectManifest` struct + JSON load/save in `crates/catalogd/src/subjects.rs`. Mirror the existing `views.rs` pattern. |
| **2** | 0.5d | Add `SubjectAuditWriter` with HMAC chain in same file. Key loaded from sealed file at startup. |
| **3** | 0.5d | Backfill subject manifests from `workers_500k.parquet` rows. ETL: one manifest per row, default `vertical=unknown`, `consent.general_pii.status=pending_backfill_review`. |
| **4** | 0.5d | Wire the gateway tool registry to write audit rows. One audit row per candidate_id returned by search_candidates / get_candidate. |
| **5** | 0.5d | Wire the validator WorkerLookup to write audit rows. |
| **6** | 1d | `/audit/subject/{id}` HTTP endpoint in `crates/catalogd/src/service.rs`. Legal-tier auth. |
| **7** | 0.5d | Daily retention sweep: subjects whose `retention.general_pii_until` < now AND `status != erased` get marked for review (don't auto-delete; legal needs to approve). |
| **8** | 0.5d | Cross-runtime parity: Go side reads the same subject manifests + audit logs. Same shapes, same HMAC algorithm. |
| **Total** | **~4 days** | Compared to 17-20 days for the IDENTITY_SERVICE_DESIGN approach. |
Each step is one commit, one revert path. No new daemons. No cloud infrastructure. No Vault. No S3 Object Lock. No dual-control JWT split-secret ceremony.
---
## 6. Auth model
Local-first, simple, defensible:
- **Service-tier reads** (gateway tool registry resolving candidate names for fill scenarios): authenticated via the existing gateway internal credential. Audit row written.
- **Legal-tier reads** (`/audit/subject/{id}`): requires a separate credential held in `/etc/lakehouse/legal_audit.token` (mode 0400, owner-only). Operators may load this only when fulfilling a legal request. The token is rotated per a documented runbook (operator + 1 witness; no cryptographic dual-control ceremony required for this scale).
- **Backups**: subject manifests + audit logs are backed up daily. The HMAC signing key is backed up to a SEPARATE storage location (different physical/network boundary) so a single backup compromise doesn't enable forgery.
This is not as strong as Vault Transit + dual-control JWT + S3 Object Lock external anchoring. It IS strong enough to pass a normal small-business compliance review and to be defensible in IL/IN small-claims-discovery contexts. If the staffing client's contract requires SOC2 Type II or formal HSM, that's a separate phase — but it's strictly additive on top of this v1 spec.
---
## 7. What this spec gives you (load-bearing)
1. **Defensible response to discrimination discovery** (worked example: John Martinez at Warehouse B). The endpoint produces a complete + signed + chain-verified record of every PII access affecting him.
2. **BIPA-compliant biometric tracking** when real photos arrive. The `consent.biometric` field + retention timeline are first-class, not bolted on later.
3. **Per-subject right-to-be-forgotten** via cryptographic erasure of the subject manifest's audit-key entry + tombstoning of the candidate's dataset rows. (The ability to verify "row is gone, trail is preserved as anonymous audit-event of the erasure" is what GDPR Art. 17 + CCPA expect.)
4. **HIPAA vertical routing** via the `vertical` field. Healthcare-vertical subjects (and `unknown` defaults) route to local-only models per PRD line 70 — no PHI to cloud egress.
5. **Cross-runtime parity** via the simple JSON+HMAC format that Go can read identically.
---
## 8. What this spec does NOT solve (and where it punts)
- **Per-row encryption of dataset PII**: subject manifests + audit logs are local; the underlying `workers_500k.parquet` is not encrypted. If staffing-client contract requires at-rest encryption, that's a separate concern handled at the storage tier (filesystem encryption, S3 SSE).
- **Right to explanation (GDPR Art. 22 / EU AI Act)**: this spec captures decisions that touched a subject; it does not require the model to explain WHY each decision was made in human-readable form. That's a separate Phase capturing model reasoning.
- **Adverse-impact statistics**: the comparator-pool snapshot per fill (per `AUDIT_PHASE_1_DISCOVERY` §10/C3) needs its own writer in the fill pipeline. This spec gives you the per-subject record; it doesn't cross-aggregate selection rates by protected class.
- **External tamper-evidence**: the HMAC chain is local. A motivated insider with access to both the audit log AND the signing key could rewrite history. For the staffing-client scale this is acceptable; for higher-stakes deployments a separate timestamping service or external transparency log would be additive.
---
## 9. Why this is the right shape for J's deployment
- It builds on what catalogd ALREADY ships (manifests, views, tombstones — the Iceberg-shape layer).
- It runs locally — no cloud infrastructure to license, monitor, audit, or pay for.
- Its primitives are JSON files an operator can read with `cat` and `jq`. Tamper-evidence works without trusting opaque crypto APIs.
- Its implementation is days, not weeks. The timeline matches the staffing-client launch window without forcing them to wait on a SaaS-tier identity service that doesn't fit their data residency posture.
- It is COMPATIBLE with the IDENTITY_SERVICE_DESIGN v3 path if the staffing client later requires SOC2 Type II — the v1 subject manifests can be migrated into a separate identity daemon when the scale demands it. But that's deferred until demand exists, not built speculatively.
This is the LOCAL-FIRST audit trail. It exists because the SaaS-tier version doesn't fit the deployment model J actually has.
---
## 10. Spec boundary
**Stable in v1 (changing requires v2):**
- File layout: `data/_catalog/subjects/<id>.json` + `data/_catalog/subjects/<id>.audit.jsonl`
- JSON schemas in §2 + §3 (additive fields OK; renames/removals require v2)
- HMAC algorithm: HMAC-SHA256 with key from sealed file
- Chain semantics: `prev_chain_hash` references previous row's `row_hmac`
- Vertical default: `unknown` with fail-closed routing
- Consent state machine in §2.1
- Audit-row write requirements in §3.2
**Implementation-specific (free to change):**
- Storage backend (file system v1; SQLite or Postgres acceptable as long as JSON shapes round-trip)
- HTTP endpoint exact shape (body schema is spec; status codes / headers are implementation)
- Backfill ETL details
**Reserved for v2:**
- Per-row encryption (when staffing-client contract requires it)
- External tamper-evidence anchor (when SOC2 Type II in scope)
- Cross-tenant subject isolation (when multi-tenant in scope)
---
## Change log
- 2026-05-03 — v1 initial draft. Builds on catalogd primitives that already exist. Smaller than IDENTITY_SERVICE_DESIGN v3 by ~4× because it doesn't propose a new daemon.
+1 -1
View File
@@ -84,7 +84,7 @@ service_name = "lakehouse"
# let the agent continuously propose + trial HNSW configs and auto-promote
# winners. Defaults are conservative so it stays out of the way of live
# search traffic on shared Ollama.
enabled = true
enabled = false
cycle_interval_secs = 120 # periodic wake if no triggers
cooldown_between_trials_secs = 10 # min gap between trials
min_recall = 0.9 # never promote below this
+94
View File
@@ -0,0 +1,94 @@
# Staffing Co-Pilot UX Fix Plan (devop.live/lakehouse)
**Target file:** `/home/profit/lakehouse/mcp-server/search.html` (239KB, ~4100 lines) — THIS is the
main dashboard served at `/` (route at `index.ts:1258`; name "search.html" is historical).
Sibling pages: `console.html` (Walkthrough), `proof.html` (Architecture), `spec.html`, `onboard.html`,
`profiler.html`, `contractor.html`, `workspaces.html`, `alerts.html`.
**Serving:** Bun `mcp-server/index.ts` on `:3700`, nginx `/lakehouse/` → `127.0.0.1:3700`.
Restart after edits: HTML is read per-request (`Bun.file`), so **no restart needed for .html edits** —
just reload the browser. (Restart only if editing index.ts: it's pid 1254, `bun run mcp-server/index.ts`.)
**Constraints:** neo-brutalist, compact, distinct colors, NO fake timers. Surgical edits only.
Dashboard fires real SQL + /intelligence/* — don't break data wiring. Repo is FROZEN Rust legacy;
J explicitly authorized work on this surface.
**Base path quirk:** nav uses `var P = location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':''`.
Relative hrefs (`href="console"`) resolve fine under `/lakehouse/`.
## User's 5 complaints → fixes (line numbers in search.html)
### 1+2. New tabs for reference pages (HIGH)
- **Top nav lines 594-599**: add `target="_blank" rel="noopener"` to Profiler, Architecture(proof),
Spec, Onboard, Alerts, Workspaces. LEAVE 592-593 (Dashboard `href="."`, Walkthrough `href="console"`) same-tab.
- **Footer line 781**: `<a href="proof">Architecture</a>` → add `target="_blank" rel="noopener"`. Keep Console same-tab.
- **Dynamic footer lines 1024-1026**: proof link needs `target=_blank rel=noopener` + `.ft` class styling (currently unstyled).
### 3. Collapsible "preview" not discoverable (HIGH)
Primary = **contract card expandable panel** (markup ~2620-2820, CSS grid-template-rows ~160-220).
Chevron rotates 180° but hint text "click for full curve…" only shows collapsed; vanishes on expand.
Fix: chevron always visible+larger(16px); persistent "Expand details"/"Collapse" label in BOTH states;
cursor:pointer on card header; hover border-left/bg shift; optional localStorage per-card.
Secondary collapsibles: ⓪ legacy-bridge narrative (~820-900) → default COLLAPSED for returning users
(localStorage 'seen_intro'); ② Staffer's Console preview (~1400-1500) → make collapsible, "Open full console →".
### 4. Clickable elements don't look clickable (HIGH)
Add global CSS: `[onclick]{cursor:pointer}` + reusable `.clickable` class with :hover (bg lift #161b22→#1c2333
+ border accent), :active (translateY(1px)), :focus-visible (outline).
Apply to: insight cards (~1180-1280), alert rows (~1700-1780), worker result rows (~990-1030 — add hover '›' chevron
for drilldown discoverability per feedback_test_drilldowns), collapsible section headers (~820-830, ~1400-1410),
action chips/buttons (~740-760, standardize .btn/.chip).
### 5. Layout clutter / flat hierarchy (HIGH)
8 sections at equal weight: ⓪ Not-a-CRM narrative, ① Live Market, Forecast, ② Staffer Console,
Market Intelligence, ③ Worker Search, ④ System Activity, ⑤ Substrate Signals.
3-tier fix: PRIMARY always-visible (Live Market, Worker Search, Staffer Console);
SECONDARY collapsible-open (Forecast, Market Intelligence); TERTIARY collapse-by-default or relocate
(⓪ narrative → collapsible banner; ⑤ Substrate Signals → move to proof.html/Architecture; ④ System Activity demote).
Also: Live Market vs Market Intelligence may overlap — differentiate headers or merge w/ sub-tabs.
Long card walls → add "show top 5 / show all" truncation.
### Cross-page nav consistency (MEDIUM)
Each of 8 pages has hardcoded nav with differing link sets/order/labels and unreliable .active.
Canonical set + order, each page sets .active on its own entry, brand/logo → `href="."`.
Long-term: extract nav to shared JS snippet served by index.ts to stop drift.
## ✅ SHIPPED 2026-05-30 (live-verified via curl against :3700; HTML served per-request → no restart)
- **#1+#2 New tabs** — search.html: 6 nav reference links (Profiler/Architecture/Spec/Onboard/Alerts/Workspaces)
+ footer Architecture + dynamic-footer Architecture now `target="_blank" rel="noopener"` with ↗ glyph.
Dashboard + Walkthrough stay same-tab (primary flow). VERIFIED: 7 static new-tab links served; nav listing clean.
- **#4 Clickability** — global affordance CSS inserted after `.ft a:hover{...}` (NOT `.ft a{margin}` — that anchor
was a hallucination): `[onclick]{cursor:pointer}`, `[role=button]`, `.iworker` cursor+hover-lift+active,
`details>summary` cursor+hover, global `:focus-visible`. VERIFIED served (affordance_css=1, iworker_cursor=1, summary_hover=1).
- **#3 partial** — every `<details>` collapsible now has visible hover affordance (cursor+bg). Contract-card toggle NOT yet (below).
- **Nav consistency** — added Profiler link to onboard/workspaces/alerts (verified profiler=1 each);
fixed contractor.html back-link `href="/"`→`href="."` (real bug: `/` = LLM-Team UI on :5000, ejected users out of lakehouse; verified root-bug=0, dot-fixed=1);
fixed profiler.html empty `href=""`→relative `.`/`console` (was JS-only/fragile; verified empty-href=0).
> CORRECTION LOG: first edit pass (earlier this session) FAILED silently — old_strings were guessed
> (`.ft a{margin:0 4px}`, a fake footer string, `ui-copilot/dashboard.html` path). Re-done with grep/Read-verified
> exact strings. The real dashboard file is `search.html` (route index.ts:1258), NOT `dashboard.html` (orphan) or `ui/`.
## (status note) — rounds 2 & 3 are recorded in the SHIPPED sections below; this block is superseded.
- #3 contract-card collapse → DONE round 2 (persistent `.card-top-toggle`).
- #5 declutter (⑤ relocate, ④ demote, ③ disclosure, Market Intelligence clarify, dead CSS) → DONE round 3.
- Still open: nav shared-component refactor (deferred, needs restart) + optional contract truncation (skipped on purpose). See "DELIBERATELY NOT DONE".
## Recommended execution order
1. New-tab attrs (lines 594-599, 781, 1024-1026) — trivial, high impact.
2. Global `.clickable` CSS helper + apply to onclick elements.
3. Collapsible affordance (chevron+label persistent, cursor, hover) on contract cards + section headers.
4. Default-collapse ⓪ narrative for returning users (localStorage).
5. Hierarchy/declutter (tiering, relocate ⑤ to proof, truncation).
6. Nav consistency pass across sibling files.
## VERIFY before claiming done (J's rules: test drilldowns, no greenwashing)
- `curl -s localhost:3700/ | grep target=_blank` confirms new-tab attrs present.
- Load https://devop.live/lakehouse/ in browser: click a contract card (expand+collapse visible),
hover a worker row (cursor+chevron), click reference nav (opens new tab), reload (intro stays collapsed).
- Don't restart the service for HTML-only edits.
## Audit provenance
5-dimension parallel Explore audit, run wf_c897f421-db1, full JSON at
`/tmp/claude-0/-home-profit/c6ca0218-678c-4a30-833c-1063e40e7f69/tasks/wm9x4y52d.output`.
NOTE: line numbers are audit estimates — `grep` to confirm exact lines before each Edit.
+1
View File
@@ -74,6 +74,7 @@ input:focus{border-color:#388bfd}
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="profiler">Profiler</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
+353
View File
@@ -0,0 +1,353 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Biometric Ops Dashboard</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar .actions{display:flex;align-items:center;gap:10px}
.bar .gen{font-size:11px;color:#7d8590;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace}
.bar button{background:#21262d;color:#c9d1d9;border:none;padding:6px 12px;font-size:12px;font-family:inherit;cursor:pointer;border-radius:0}
.bar button:hover{background:#30363d}
.wrap{max-width:1100px;margin:0 auto;padding:24px 20px 60px}
.screen{display:none}
.screen.active{display:block}
h2{font-size:18px;color:#e6edf3;font-weight:600;margin:24px 0 10px;letter-spacing:-0.3px}
h2:first-child{margin-top:0}
.lede{color:#7d8590;font-size:13px;margin-bottom:16px}
.card{background:#0d1117;border:1px solid #171d27;padding:18px;margin-bottom:14px}
label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
input[type=password]{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s}
input[type=password]:focus{outline:none;border-color:#58a6ff}
button.primary{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s}
button.primary:hover:not(:disabled){background:#388bfd}
button.primary:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
.tiles{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:10px;margin-bottom:18px}
.tile{background:#0d1117;border:1px solid #171d27;padding:18px;border-left:3px solid #2d333b}
.tile.given{border-left-color:#3fb950}
.tile.withdrawn{border-left-color:#d29922}
.tile.expired{border-left-color:#da3633}
.tile.erased{border-left-color:#8b949e}
.tile.never{border-left-color:#58a6ff}
.tile.pending{border-left-color:#a371f7}
.tile.photos{border-left-color:#f85149}
.tile-label{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:0.6px;margin-bottom:6px}
.tile-value{font-size:24px;color:#e6edf3;font-weight:600;letter-spacing:-0.5px;font-feature-settings:'tnum';font-variant-numeric:tabular-nums}
.tile-sub{font-size:11px;color:#545d68;margin-top:4px}
.row-grid{display:grid;grid-template-columns:1fr 1fr;gap:12px}
@media (max-width:760px){ .row-grid{grid-template-columns:1fr} }
.kv{display:grid;grid-template-columns:200px 1fr;gap:6px;margin:4px 0;font-size:13px}
.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
.table{width:100%;border-collapse:collapse;margin-top:8px;font-size:12px}
.table th{text-align:left;padding:8px 10px;background:#0a0c10;color:#7d8590;text-transform:uppercase;letter-spacing:0.4px;font-size:11px;font-weight:500;border-bottom:1px solid #1f242c}
.table td{padding:8px 10px;border-bottom:1px solid #161b22;color:#c9d1d9;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;vertical-align:top}
.table tr:hover td{background:#0a0c10}
.kind{display:inline-block;padding:2px 8px;font-size:10px;text-transform:uppercase;letter-spacing:0.5px;font-family:'Inter',sans-serif}
.kind.consent_grant{background:#0c2510;color:#7ee787;border:1px solid #2ea04338}
.kind.collection{background:#251c0c;color:#ffd166;border:1px solid #d2992238}
.kind.withdrawal{background:#251c0c;color:#f0883e;border:1px solid #f0883e38}
.kind.erasure{background:#250d0d;color:#ff7b72;border:1px solid #f8514938}
.kind.full{background:#250d0d;color:#ffa198;border:1px solid #f8514955}
.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
.error.show{display:block}
.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
.foot a{color:#58a6ff}
.empty{padding:24px;text-align:center;color:#545d68;font-size:13px;font-style:italic}
.kb-pill{display:inline-block;margin-right:18px;font-size:13px}
.kb-pill .kb-k{color:#7d8590}
.kb-pill .kb-v{color:#e6edf3;font-weight:600}
</style>
</head>
<body>
<div class="bar">
<h1>⚡ Biometric Ops Dashboard</h1>
<span class="actions">
<span class="gen" id="gen-at">…</span>
<button id="refresh">Refresh</button>
<button id="logout">Logout</button>
</span>
</div>
<div class="wrap">
<!-- Token gate -->
<div id="screen-token" class="screen active">
<h2>Operator authentication</h2>
<p class="lede">Read-only ops dashboard over <code>/biometric/stats</code>. Aggregate counts + recent state-change events. Auth via legal-tier token (sessionStorage; clears on tab close).</p>
<div class="card">
<label for="token-input">Legal audit token</label>
<input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
<div style="margin-top:14px"><button id="token-submit" class="primary">Continue →</button></div>
</div>
<div class="error" id="token-error"></div>
</div>
<!-- Dashboard -->
<div id="screen-dash" class="screen">
<h2 style="margin-top:0">Biometric consent state — by subject</h2>
<div class="tiles">
<div class="tile never"><div class="tile-label">Never collected</div><div class="tile-value" id="t-never">…</div><div class="tile-sub">no biometric data on record</div></div>
<div class="tile pending"><div class="tile-label">Pending</div><div class="tile-value" id="t-pending">…</div><div class="tile-sub">consent in flight</div></div>
<div class="tile given"><div class="tile-label">Given</div><div class="tile-value" id="t-given">…</div><div class="tile-sub">active biometric data</div></div>
<div class="tile withdrawn"><div class="tile-label">Withdrawn</div><div class="tile-value" id="t-withdrawn">…</div><div class="tile-sub">awaiting destruction (≤30d)</div></div>
<div class="tile expired"><div class="tile-label">Expired</div><div class="tile-value" id="t-expired">…</div><div class="tile-sub">retention window passed</div></div>
<div class="tile photos"><div class="tile-label">Photos on disk</div><div class="tile-value" id="t-photos">…</div><div class="tile-sub">quarantined uploads</div></div>
</div>
<div class="row-grid">
<div>
<h2>Subject lifecycle status</h2>
<div class="card">
<div class="kv">
<span class="k">Total subjects</span><span class="v" id="x-total">…</span>
<span class="k">Active</span><span class="v" id="x-active">…</span>
<span class="k">Pending consent</span><span class="v" id="x-pending-consent">…</span>
<span class="k">Withdrawn</span><span class="v" id="x-withdrawn">…</span>
<span class="k">Retention expired</span><span class="v" id="x-retexp">…</span>
<span class="k">Erased</span><span class="v" id="x-erased">…</span>
</div>
</div>
</div>
<div>
<h2>Retention horizon</h2>
<div class="card">
<div class="kv">
<span class="k">Oldest active retention</span><span class="v" id="x-oldest">—</span>
<span class="k">Days until earliest expiry</span><span class="v" id="x-days">—</span>
<span class="k">Sweep schedule</span><span class="v">daily 03:00 UTC</span>
<span class="k">Sweep unit</span><span class="v">lakehouse-retention-sweep.timer</span>
</div>
</div>
</div>
</div>
<h2>Recent state-change events <span style="color:#7d8590;font-size:12px;font-weight:400">(last 20 across all subjects, newest first)</span></h2>
<div class="card" style="padding:0">
<table class="table">
<thead>
<tr>
<th>Timestamp</th>
<th>Candidate</th>
<th>Kind</th>
<th>Result</th>
<th>Trace ID</th>
</tr>
</thead>
<tbody id="events-tbody">
<tr><td colspan="5" class="empty">Loading…</td></tr>
</tbody>
</table>
</div>
<h2>Event-kind breakdown <span style="color:#7d8590;font-size:12px;font-weight:400">(across recent events shown above)</span></h2>
<div class="card">
<div id="kind-breakdown">…</div>
</div>
<div class="error" id="dash-error"></div>
</div>
</div>
<div class="foot">
<a href="/biometric/intake">Intake UI</a>
· <a href="/biometric/withdraw">Withdraw UI</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">BIPA Gates</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
</div>
<script>
const GATEWAY = (function(){
const p = new URLSearchParams(location.search);
return p.get('gw') || 'http://localhost:3100';
})();
const state = { token: null, refreshTimer: null };
const REFRESH_MS = 15000;
function show(id) {
document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
document.getElementById(id).classList.add('active');
}
function err(id, msg){ const e=document.getElementById(id); e.textContent=msg; e.classList.add('show'); }
function clearErr(id){ document.getElementById(id).classList.remove('show'); }
function fmtDate(iso){
if (!iso) return '—';
try { return new Date(iso).toISOString().replace('T',' ').replace(/\.\d+Z$/, 'Z'); }
catch(e){ return iso; }
}
// Build a TD element with text content. textContent prevents any HTML
// in the value from being interpreted — important because trace_id and
// (in theory) other audit-row fields could contain operator-supplied
// strings. The prior version used innerHTML string concatenation; that
// was an XSS vector if a maliciously-crafted X-Lakehouse-Trace-Id
// landed in the audit log.
function td(value) {
const el = document.createElement('td');
el.textContent = value == null ? '—' : String(value);
return el;
}
function kindBadge(kind) {
const map = {
'biometric_consent_grant': 'consent_grant',
'biometric_collection': 'collection',
'biometric_consent_withdrawal': 'withdrawal',
'biometric_erasure': 'erasure',
'full_erasure': 'full',
};
const cls = map[kind] || '';
const span = document.createElement('span');
span.className = 'kind ' + cls;
span.textContent = String(kind || '').replace(/^biometric_/, '');
const cell = document.createElement('td');
cell.appendChild(span);
return cell;
}
(function init(){
const saved = sessionStorage.getItem('lh_legal_token');
if (saved) {
state.token = saved;
document.getElementById('token-input').value = '••••••••';
show('screen-dash');
refresh();
state.refreshTimer = setInterval(refresh, REFRESH_MS);
}
})();
document.getElementById('token-submit').addEventListener('click', () => {
clearErr('token-error');
const v = document.getElementById('token-input').value.trim();
const tok = v === '••••••••' ? state.token : v;
if (!tok || tok.length < 32) { err('token-error', 'Token must be ≥32 characters.'); return; }
state.token = tok;
sessionStorage.setItem('lh_legal_token', tok);
show('screen-dash');
refresh();
state.refreshTimer = setInterval(refresh, REFRESH_MS);
});
document.getElementById('refresh').addEventListener('click', refresh);
document.getElementById('logout').addEventListener('click', () => {
sessionStorage.removeItem('lh_legal_token');
state.token = null;
if (state.refreshTimer) { clearInterval(state.refreshTimer); state.refreshTimer = null; }
document.getElementById('token-input').value = '';
show('screen-token');
});
async function refresh(){
clearErr('dash-error');
try {
const r = await fetch(`${GATEWAY}/biometric/stats`, {
headers: { 'X-Lakehouse-Legal-Token': state.token },
});
if (!r.ok) {
const body = await r.text();
err('dash-error', `Stats fetch failed: HTTP ${r.status} ${body.substring(0,200)}`);
return;
}
const s = await r.json();
render(s);
} catch (e) {
err('dash-error', `Network error: ${e.message}`);
}
}
function render(s) {
document.getElementById('gen-at').textContent = 'as of ' + fmtDate(s.generated_at);
document.getElementById('t-never').textContent = s.biometric_status.never_collected;
document.getElementById('t-pending').textContent = s.biometric_status.pending;
document.getElementById('t-given').textContent = s.biometric_status.given;
document.getElementById('t-withdrawn').textContent = s.biometric_status.withdrawn;
document.getElementById('t-expired').textContent = s.biometric_status.expired;
document.getElementById('t-photos').textContent = s.photos_on_record;
document.getElementById('x-total').textContent = s.total_subjects;
document.getElementById('x-active').textContent = s.subject_status.active;
document.getElementById('x-pending-consent').textContent = s.subject_status.pending_consent;
document.getElementById('x-withdrawn').textContent = s.subject_status.withdrawn;
document.getElementById('x-retexp').textContent = s.subject_status.retention_expired;
document.getElementById('x-erased').textContent = s.subject_status.erased;
document.getElementById('x-oldest').textContent = fmtDate(s.oldest_active_retention_until);
document.getElementById('x-days').textContent =
s.upcoming_destruction_window_days != null ? s.upcoming_destruction_window_days + ' days' : '—';
// Build events table via DOM nodes (textContent on each cell) — never
// innerHTML with audit-row values.
const tbody = document.getElementById('events-tbody');
tbody.replaceChildren();
if (!s.recent_events.length) {
const tr = document.createElement('tr');
const tdEmpty = document.createElement('td');
tdEmpty.colSpan = 5;
tdEmpty.className = 'empty';
tdEmpty.textContent = 'No state-change events recorded yet. Run an intake or withdrawal to see audit chain entries here.';
tr.appendChild(tdEmpty);
tbody.appendChild(tr);
} else {
for (const e of s.recent_events) {
const tr = document.createElement('tr');
tr.appendChild(td(fmtDate(e.ts)));
tr.appendChild(td(e.candidate_id));
tr.appendChild(kindBadge(e.kind));
tr.appendChild(td(e.result));
tr.appendChild(td(e.trace_id || '—'));
tbody.appendChild(tr);
}
}
// Same DOM-build for the kind breakdown.
const kb = document.getElementById('kind-breakdown');
kb.replaceChildren();
const counts = s.recent_event_counts || {};
const keys = Object.keys(counts);
if (!keys.length) {
const span = document.createElement('span');
span.style.color = '#7d8590';
span.style.fontSize = '12px';
span.style.fontStyle = 'italic';
span.textContent = 'No events yet.';
kb.appendChild(span);
} else {
for (const k of keys.sort()) {
const wrap = document.createElement('span');
wrap.className = 'kb-pill';
const kEl = document.createElement('span');
kEl.className = 'kb-k';
kEl.textContent = k;
const sep = document.createTextNode(': ');
const vEl = document.createElement('strong');
vEl.className = 'kb-v';
vEl.textContent = counts[k];
wrap.append(kEl, sep, vEl);
kb.appendChild(wrap);
}
}
}
</script>
</body>
</html>
+482
View File
@@ -0,0 +1,482 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Biometric Consent Intake</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar .cid{font-size:12px;color:#545d68;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace}
.bar .step{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:1px}
.wrap{max-width:760px;margin:0 auto;padding:28px 20px 60px}
.screen{display:none}
.screen.active{display:block}
h2{font-size:20px;color:#e6edf3;font-weight:600;margin-bottom:8px;letter-spacing:-0.3px}
h3{font-size:14px;color:#e6edf3;font-weight:600;margin:16px 0 8px}
.lede{color:#7d8590;font-size:13px;margin-bottom:24px}
.card{background:#0d1117;border:1px solid #171d27;padding:20px;margin-bottom:16px}
.card.warn{border-left:3px solid #d29922}
.card.bad{border-left:3px solid #f85149}
.card.good{border-left:3px solid #3fb950}
label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
input[type=text],input[type=password]{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s}
input[type=text]:focus,input[type=password]:focus{outline:none;border-color:#58a6ff}
input[type=file]{color:#7d8590;font-size:12px;margin-top:8px}
input[type=checkbox]{margin-right:8px;transform:scale(1.2)}
.checkbox-row{display:flex;align-items:flex-start;color:#e6edf3;font-size:14px;margin:16px 0;cursor:pointer;text-transform:none;letter-spacing:0}
.checkbox-row input{margin-top:3px}
button{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s;margin-right:8px}
button:hover:not(:disabled){background:#388bfd}
button:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
button.secondary{background:#21262d;color:#c9d1d9}
button.secondary:hover:not(:disabled){background:#30363d}
button.danger{background:#da3633}
button.danger:hover:not(:disabled){background:#f85149}
.consent-text{background:#0a0c10;border:1px solid #1f242c;padding:20px;max-height:340px;overflow-y:auto;font-size:13px;line-height:1.7;margin-bottom:16px}
.consent-text h4{color:#e6edf3;font-size:14px;font-weight:600;margin:14px 0 6px}
.consent-text h4:first-child{margin-top:0}
.consent-text p{margin-bottom:8px;color:#b0b8c4}
.consent-text ul{margin:6px 0 8px 20px;color:#b0b8c4}
.row{display:flex;gap:12px;align-items:center}
.row > *{flex:1}
.row > button{flex:0 0 auto}
.preview{max-width:280px;max-height:280px;display:block;margin:12px 0;border:2px solid #30363d}
video{max-width:280px;max-height:280px;display:block;margin:12px 0;border:2px solid #30363d;background:#000}
canvas{display:none}
.hash{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:11px;color:#7ee787;word-break:break-all;background:#0a0c10;padding:8px 12px;border:1px solid #1f242c;margin:6px 0}
.kv{display:grid;grid-template-columns:140px 1fr;gap:8px;margin:6px 0;font-size:13px}
.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
.error.show{display:block}
.error code{color:#fff}
.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
.spec-link{font-size:11px;color:#545d68;letter-spacing:0.4px;text-transform:uppercase}
.spec-link a{color:#58a6ff}
</style>
</head>
<body>
<div class="bar">
<h1>⚡ Biometric Consent Intake</h1>
<span class="step" id="step-label">step 1 of 4</span>
<span class="cid" id="cid-display">…</span>
</div>
<div class="wrap">
<!-- Step 1: operator auth -->
<div id="screen-token" class="screen active">
<h2>Operator authentication</h2>
<p class="lede">Paste the legal-tier audit token + your name. Both stored in this tab's session only; cleared on close. Never persists to disk. Your name is recorded as <code>operator_of_record</code> in the audit row for legal traceability.</p>
<div class="card">
<label for="token-input">Legal audit token</label>
<input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
<label for="op-name" style="margin-top:14px">Your name (operator of record)</label>
<input type="text" id="op-name" placeholder="First Last" autocomplete="off">
<div style="margin-top:14px">
<button id="token-submit">Continue →</button>
</div>
</div>
<div class="error" id="token-error"></div>
</div>
<!-- Step 2: consent -->
<div id="screen-consent" class="screen">
<h2>Biometric Information Consent — v1</h2>
<p class="lede">Please read the disclosures below. Sign by checking the box and entering your printed name. The candidate-facing surface required by BIPA §15(b)(1)-(3).</p>
<div class="consent-text">
<h4>Disclosure 1 — Notice of collection (§15(b)(1))</h4>
<p>Lakehouse will collect and store your <strong>biometric identifier</strong> (a photograph of you from which facial geometry is implicit). The photograph itself is the data we keep — we do <strong>NOT</strong> run automated facial-classification (gender / race / age inference) against it in v1. If at a later date we add automated classification, we will re-collect consent under a superseding template before doing so.</p>
<h4>Disclosure 2 — Specific purpose and length of term (§15(b)(2))</h4>
<p>The biometric data will be used for:</p>
<ul>
<li>Identity verification at staffing job sites</li>
<li>Internal record-keeping so coordinators can recognize you across placements</li>
</ul>
<p>Retained for a maximum of <strong>18 months</strong> from your most recent interaction, then permanently destroyed per the <a href="#" onclick="alert('See docs/policies/consent/biometric_retention_schedule_v1.md')">Biometric Retention Schedule v1</a>.</p>
<p>You may withdraw at any time by contacting the operator. Withdrawal triggers permanent destruction.</p>
<h4>Disclosure 3 — Written release (§15(b)(3))</h4>
<p>By checking the box below and providing your printed name, you provide a written release authorizing Lakehouse to collect, store, and use your biometric identifier and biometric information for the purposes and term stated above.</p>
<h4>What you're agreeing to (plain language)</h4>
<p>If you upload a photo of yourself, we'll keep that photo so your staffing coordinator can recognize you when you arrive at job sites. We don't run automated guesses about your age, gender, or race against the photo. We don't sell it. We don't share it outside the staffing operation unless legally compelled. You can withdraw consent at any time.</p>
</div>
<div class="card">
<label for="cand-name">Candidate printed name</label>
<input type="text" id="cand-name" placeholder="First Last" autocomplete="off">
<label class="checkbox-row" style="margin-top:16px">
<input type="checkbox" id="cand-agree">
<span>I have read and understood the disclosures above. I am providing this consent voluntarily and free of coercion.</span>
</label>
<div class="kv" style="margin-top:14px">
<span class="k">Method</span><span class="v">click_acceptance</span>
<span class="k">Version hash</span><span class="v" id="version-hash-display">computing…</span>
<span class="k">Timestamp</span><span class="v" id="timestamp-display">on submit</span>
</div>
<div style="margin-top:16px">
<button id="grant-btn" disabled>Grant consent + continue →</button>
</div>
</div>
<div class="error" id="consent-error"></div>
</div>
<!-- Step 3: photo -->
<div id="screen-photo" class="screen">
<h2>Photo capture</h2>
<p class="lede">Take or upload a clear photo. Stored quarantined under <code>data/biometric/uploads/</code> with mode 0700/0600 + SHA-256 integrity hash. Audit chain records the upload event.</p>
<div class="card">
<h3>Option A — File upload</h3>
<input type="file" id="photo-file" accept="image/jpeg,image/png">
<h3 style="margin-top:24px">Option B — Camera capture</h3>
<button id="cam-start" class="secondary">Start camera</button>
<button id="cam-capture" class="secondary" style="display:none">📸 Capture</button>
<button id="cam-stop" class="secondary" style="display:none">Stop</button>
<video id="cam" autoplay playsinline muted></video>
<canvas id="cam-canvas"></canvas>
<h3 style="margin-top:24px">Preview</h3>
<img id="preview" class="preview" style="display:none">
<p id="no-preview" class="lede" style="margin:12px 0">No photo selected yet.</p>
<div style="margin-top:16px">
<button id="upload-btn" disabled>Upload photo + finish →</button>
<button id="skip-btn" class="secondary">Skip photo (consent only)</button>
</div>
</div>
<div class="error" id="photo-error"></div>
</div>
<!-- Step 4: done -->
<div id="screen-done" class="screen">
<div class="card good">
<h2>✓ Intake complete</h2>
<p class="lede" style="margin:8px 0 0">Audit chain rows (HMAC-SHA256, persisted to <code>data/_catalog/subjects/&lt;id&gt;.audit.jsonl</code>):</p>
</div>
<div class="card">
<h3>Consent grant</h3>
<div class="kv">
<span class="k">Status</span><span class="v" id="done-status">Given</span>
<span class="k">Given at</span><span class="v" id="done-given-at">…</span>
<span class="k">Retention until</span><span class="v" id="done-retention">…</span>
</div>
<h3>Audit hmac</h3>
<div class="hash" id="done-consent-hmac">…</div>
</div>
<div class="card" id="photo-card" style="display:none">
<h3>Photo upload</h3>
<div class="kv">
<span class="k">Data path</span><span class="v" id="done-data-path">…</span>
<span class="k">Template SHA-256</span><span class="v" id="done-template-hash">…</span>
</div>
<h3>Audit hmac</h3>
<div class="hash" id="done-photo-hmac">…</div>
</div>
<div class="card warn">
<h3>Verification</h3>
<p>Operator: confirm the audit chain by running:</p>
<div class="hash" id="verify-cmd">./scripts/staffing/verify_biometric_erasure.sh &lt;candidate_id&gt;</div>
<p style="margin-top:8px">Or hit <code>GET /audit/subject/&lt;id&gt;</code> with legal-tier auth to read the full chain.</p>
</div>
<div style="margin-top:16px">
<button onclick="location.reload()" class="secondary">Start another intake</button>
</div>
</div>
</div>
<div class="foot">
<span class="spec-link">
<a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">Phase 1.6 BIPA Gates</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/policies/consent/biometric_consent_template_v1.md">Consent template v1</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
</span>
</div>
<script>
// ── Config ────────────────────────────────────────────────────────
// Gateway base URL. Same-origin if the page is served behind a reverse
// proxy; otherwise the operator's local gateway.
const GATEWAY = (function(){
const p = new URLSearchParams(location.search);
return p.get('gw') || 'http://localhost:3100';
})();
// Candidate ID from the URL. Required.
const CANDIDATE_ID = (function(){
const p = new URLSearchParams(location.search);
return p.get('candidate_id') || p.get('cid') || '';
})();
// ── State ─────────────────────────────────────────────────────────
const state = {
token: null, // sessionStorage-backed
operator: null, // operator_of_record — captured at token paste
consentVersionHash: null, // SHA-256 of the rendered consent template
consentResp: null, // server response from /consent
photoBlob: null, // captured/selected photo
photoResp: null, // server response from /photo
};
// ── Helpers ───────────────────────────────────────────────────────
function show(screen) {
document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
document.getElementById(screen).classList.add('active');
const stepMap = {'screen-token':'step 1 of 4', 'screen-consent':'step 2 of 4', 'screen-photo':'step 3 of 4', 'screen-done':'step 4 of 4'};
document.getElementById('step-label').textContent = stepMap[screen] || '';
}
function err(elId, msg) {
const e = document.getElementById(elId);
e.textContent = msg;
e.classList.add('show');
}
function clearErr(elId) {
document.getElementById(elId).classList.remove('show');
}
async function sha256Hex(text) {
const enc = new TextEncoder().encode(text);
const buf = await crypto.subtle.digest('SHA-256', enc);
return Array.from(new Uint8Array(buf)).map(b => b.toString(16).padStart(2,'0')).join('');
}
// ── Init ──────────────────────────────────────────────────────────
(async function init(){
if (!CANDIDATE_ID) {
err('token-error', 'Missing candidate_id in URL. Append ?candidate_id=WORKER-XXX');
document.getElementById('token-submit').disabled = true;
return;
}
document.getElementById('cid-display').textContent = CANDIDATE_ID;
// Restore token + operator from sessionStorage if previously set in this tab.
const saved = sessionStorage.getItem('lh_legal_token');
const savedOp = sessionStorage.getItem('lh_operator_name');
if (saved) {
state.token = saved;
document.getElementById('token-input').value = '••••••••';
}
if (savedOp) {
state.operator = savedOp;
document.getElementById('op-name').value = savedOp;
}
// Compute consent template hash (SHA-256 of the rendered consent block).
// This is what we'll send as consent_version_hash. Counsel later
// attests against the canonical signed template hash; operator
// discipline keeps these aligned.
const consentText = document.querySelector('.consent-text').innerText;
state.consentVersionHash = await sha256Hex(consentText);
document.getElementById('version-hash-display').textContent = state.consentVersionHash.substring(0,16) + '…';
})();
// ── Step 1: token + operator ───────────────────────────────────────
document.getElementById('token-submit').addEventListener('click', () => {
clearErr('token-error');
const v = document.getElementById('token-input').value.trim();
// If user kept the masked placeholder, use the saved token.
const tok = v === '••••••••' ? state.token : v;
const op = document.getElementById('op-name').value.trim();
if (!tok || tok.length < 32) {
err('token-error', 'Token must be ≥32 characters.');
return;
}
if (!op || op.length < 2) {
err('token-error', 'Operator name is required (recorded as operator_of_record in audit row).');
return;
}
state.token = tok;
state.operator = op;
sessionStorage.setItem('lh_legal_token', tok);
sessionStorage.setItem('lh_operator_name', op);
show('screen-consent');
});
// ── Step 2: consent ────────────────────────────────────────────────
function refreshGrantBtn() {
const ok = document.getElementById('cand-agree').checked
&& document.getElementById('cand-name').value.trim().length >= 2;
document.getElementById('grant-btn').disabled = !ok;
}
document.getElementById('cand-agree').addEventListener('change', refreshGrantBtn);
document.getElementById('cand-name').addEventListener('input', refreshGrantBtn);
document.getElementById('grant-btn').addEventListener('click', async () => {
clearErr('consent-error');
document.getElementById('grant-btn').disabled = true;
try {
const candName = document.getElementById('cand-name').value.trim();
const ts = new Date().toISOString();
document.getElementById('timestamp-display').textContent = ts;
const evidence = JSON.stringify({
method: 'click_acceptance',
candidate_printed_name: candName,
acceptance_ts: ts,
user_agent: navigator.userAgent,
page_origin: location.origin,
});
const evidenceHash = await sha256Hex(evidence);
const body = {
consent_version_hash: state.consentVersionHash,
consent_collection_method: 'click_acceptance',
consent_collection_evidence_path: 'inline:sha256=' + evidenceHash,
operator_of_record: state.operator,
};
const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(CANDIDATE_ID)}/consent`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Lakehouse-Legal-Token': state.token,
'X-Lakehouse-Trace-Id': 'intake-' + Date.now(),
},
body: JSON.stringify(body),
});
const respBody = await r.json();
if (!r.ok) {
err('consent-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
document.getElementById('grant-btn').disabled = false;
return;
}
state.consentResp = respBody;
show('screen-photo');
} catch (e) {
err('consent-error', `Network error: ${e.message}`);
document.getElementById('grant-btn').disabled = false;
}
});
// ── Step 3: photo ──────────────────────────────────────────────────
const previewEl = document.getElementById('preview');
const noPreviewEl = document.getElementById('no-preview');
const fileInput = document.getElementById('photo-file');
fileInput.addEventListener('change', () => {
const f = fileInput.files[0];
if (!f) return;
state.photoBlob = f;
previewEl.src = URL.createObjectURL(f);
previewEl.style.display = 'block';
noPreviewEl.style.display = 'none';
document.getElementById('upload-btn').disabled = false;
});
let camStream = null;
document.getElementById('cam-start').addEventListener('click', async () => {
clearErr('photo-error');
try {
camStream = await navigator.mediaDevices.getUserMedia({ video: { width: 640, height: 480, facingMode: 'user' }, audio: false });
document.getElementById('cam').srcObject = camStream;
document.getElementById('cam').style.display = 'block';
document.getElementById('cam-start').style.display = 'none';
document.getElementById('cam-capture').style.display = 'inline-block';
document.getElementById('cam-stop').style.display = 'inline-block';
} catch (e) {
err('photo-error', `Camera unavailable: ${e.message}`);
}
});
document.getElementById('cam-capture').addEventListener('click', () => {
const v = document.getElementById('cam');
const c = document.getElementById('cam-canvas');
c.width = v.videoWidth;
c.height = v.videoHeight;
c.getContext('2d').drawImage(v, 0, 0);
c.toBlob(b => {
state.photoBlob = b;
previewEl.src = URL.createObjectURL(b);
previewEl.style.display = 'block';
noPreviewEl.style.display = 'none';
document.getElementById('upload-btn').disabled = false;
}, 'image/jpeg', 0.92);
});
document.getElementById('cam-stop').addEventListener('click', () => {
if (camStream) { camStream.getTracks().forEach(t => t.stop()); camStream = null; }
document.getElementById('cam').style.display = 'none';
document.getElementById('cam-start').style.display = 'inline-block';
document.getElementById('cam-capture').style.display = 'none';
document.getElementById('cam-stop').style.display = 'none';
});
document.getElementById('upload-btn').addEventListener('click', async () => {
clearErr('photo-error');
document.getElementById('upload-btn').disabled = true;
try {
const ct = state.photoBlob.type || 'image/jpeg';
const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(CANDIDATE_ID)}/photo`, {
method: 'POST',
headers: {
'Content-Type': ct,
'X-Lakehouse-Legal-Token': state.token,
'X-Lakehouse-Consent-Version-Hash': state.consentVersionHash,
'X-Lakehouse-Trace-Id': 'intake-' + Date.now(),
},
body: state.photoBlob,
});
const respBody = await r.json();
if (!r.ok) {
err('photo-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
document.getElementById('upload-btn').disabled = false;
return;
}
state.photoResp = respBody;
if (camStream) camStream.getTracks().forEach(t => t.stop());
showDone();
} catch (e) {
err('photo-error', `Network error: ${e.message}`);
document.getElementById('upload-btn').disabled = false;
}
});
document.getElementById('skip-btn').addEventListener('click', () => {
if (camStream) camStream.getTracks().forEach(t => t.stop());
showDone();
});
// ── Step 4: done ───────────────────────────────────────────────────
function showDone() {
document.getElementById('done-given-at').textContent = state.consentResp.given_at;
document.getElementById('done-retention').textContent = state.consentResp.retention_until;
document.getElementById('done-consent-hmac').textContent = state.consentResp.audit_row_hmac;
if (state.photoResp) {
document.getElementById('photo-card').style.display = 'block';
document.getElementById('done-data-path').textContent = state.photoResp.data_path;
document.getElementById('done-template-hash').textContent = state.photoResp.template_hash;
document.getElementById('done-photo-hmac').textContent = state.photoResp.audit_row_hmac;
}
document.getElementById('verify-cmd').textContent =
`./scripts/staffing/verify_biometric_erasure.sh ${CANDIDATE_ID}`;
show('screen-done');
}
</script>
</body>
</html>
+253
View File
@@ -0,0 +1,253 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Lakehouse — Biometric Consent Withdrawal</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;color:#b0b8c4;font-size:14px;line-height:1.55;-webkit-font-smoothing:antialiased}
a{color:#58a6ff;text-decoration:none}
a:hover{color:#79c0ff}
.bar{background:#0d1117;padding:0 24px;height:56px;border-bottom:1px solid #171d27;display:flex;justify-content:space-between;align-items:center;position:sticky;top:0;z-index:10}
.bar h1{font-size:14px;font-weight:600;color:#e6edf3;letter-spacing:-0.2px}
.bar .step{font-size:11px;color:#7d8590;text-transform:uppercase;letter-spacing:1px}
.wrap{max-width:680px;margin:0 auto;padding:28px 20px 60px}
.screen{display:none}
.screen.active{display:block}
h2{font-size:20px;color:#e6edf3;font-weight:600;margin-bottom:8px;letter-spacing:-0.3px}
h3{font-size:14px;color:#e6edf3;font-weight:600;margin:16px 0 8px}
.lede{color:#7d8590;font-size:13px;margin-bottom:24px}
.card{background:#0d1117;border:1px solid #171d27;padding:20px;margin-bottom:16px}
.card.warn{border-left:3px solid #d29922}
.card.bad{border-left:3px solid #f85149}
.card.good{border-left:3px solid #3fb950}
label{display:block;margin-bottom:12px;color:#7d8590;font-size:12px;text-transform:uppercase;letter-spacing:0.5px}
input[type=text],input[type=password],textarea{width:100%;background:#0d1117;border:1px solid #2d333b;color:#e6edf3;padding:10px 12px;font-family:inherit;font-size:14px;border-radius:0;margin-top:6px;transition:border-color .15s;resize:vertical}
input[type=text]:focus,input[type=password]:focus,textarea:focus{outline:none;border-color:#58a6ff}
textarea{font-family:inherit;line-height:1.5;min-height:80px}
button{background:#1f6feb;color:#fff;border:none;padding:10px 20px;font-family:inherit;font-size:14px;font-weight:500;cursor:pointer;border-radius:0;transition:background .15s;margin-right:8px}
button:hover:not(:disabled){background:#388bfd}
button:disabled{background:#21262d;color:#545d68;cursor:not-allowed}
button.secondary{background:#21262d;color:#c9d1d9}
button.secondary:hover:not(:disabled){background:#30363d}
button.danger{background:#da3633}
button.danger:hover:not(:disabled){background:#f85149}
.notice{background:#0a0c10;border:1px solid #1f242c;padding:14px 18px;margin-bottom:16px;font-size:13px;color:#b0b8c4;line-height:1.7}
.notice strong{color:#e6edf3}
.kv{display:grid;grid-template-columns:140px 1fr;gap:8px;margin:6px 0;font-size:13px}
.kv .k{color:#7d8590;text-transform:uppercase;font-size:11px;letter-spacing:0.5px;padding-top:2px}
.kv .v{color:#e6edf3;font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:12px}
.hash{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:11px;color:#7ee787;word-break:break-all;background:#0a0c10;padding:8px 12px;border:1px solid #1f242c;margin:6px 0}
.error{background:#3a0f0f;border:1px solid #f85149;color:#ffa198;padding:12px 16px;margin:12px 0;font-size:13px;display:none}
.error.show{display:block}
.error code{color:#fff}
.foot{margin-top:32px;padding-top:16px;border-top:1px solid #171d27;font-size:11px;color:#545d68;text-transform:uppercase;letter-spacing:0.5px}
.foot a{color:#58a6ff}
</style>
</head>
<body>
<div class="bar">
<h1>⚡ Biometric Consent Withdrawal</h1>
<span class="step" id="step-label">step 1 of 3</span>
</div>
<div class="wrap">
<!-- Step 1: operator auth -->
<div id="screen-token" class="screen active">
<h2>Operator authentication</h2>
<p class="lede">Withdrawal is operator-recorded on behalf of the candidate. Paste the legal-tier audit token + your name.</p>
<div class="card">
<label for="token-input">Legal audit token</label>
<input type="password" id="token-input" placeholder="44-char token from /etc/lakehouse/legal_audit.token" autocomplete="off">
<label for="op-name" style="margin-top:14px">Your name (operator of record)</label>
<input type="text" id="op-name" placeholder="First Last" autocomplete="off">
<div style="margin-top:14px">
<button id="token-submit">Continue →</button>
</div>
</div>
<div class="error" id="token-error"></div>
</div>
<!-- Step 2: withdrawal form -->
<div id="screen-form" class="screen">
<h2>Record withdrawal</h2>
<p class="lede">The candidate has requested withdrawal of biometric consent. This action sets a <strong>30-day SLA clock</strong> for destruction (per consent template v1 §2). The retention sweep + erase runbook handle actual destruction; this endpoint records intent + starts the clock.</p>
<div class="notice warn">
<strong>What withdrawal does:</strong> sets <code>consent.biometric.status = Withdrawn</code>, accelerates <code>retention_until</code> from the 18-month default to <strong>30 days from now</strong>. Future photo uploads will be refused (403). General-PII consent is NOT touched — the candidate can keep their non-biometric data on the platform.
</div>
<div class="card">
<label for="cand-id">Candidate ID</label>
<input type="text" id="cand-id" placeholder="WORKER-XXX" autocomplete="off">
<label for="reason" style="margin-top:14px">Reason for withdrawal</label>
<textarea id="reason" placeholder="Free-text. Captured to the audit row for legal traceability. e.g., 'candidate emailed 2026-05-05 asking to remove their photo, no specific reason given'"></textarea>
<label for="evidence" style="margin-top:14px">Evidence path (optional)</label>
<input type="text" id="evidence" placeholder="/path/to/email_thread.eml or scanned signed paper" autocomplete="off">
<div style="margin-top:16px">
<button id="submit-btn" class="danger" disabled>Record withdrawal</button>
<button id="back-btn" class="secondary">Back</button>
</div>
</div>
<div class="error" id="form-error"></div>
</div>
<!-- Step 3: confirmation -->
<div id="screen-done" class="screen">
<div class="card good">
<h2>✓ Withdrawal recorded</h2>
<p class="lede" style="margin:8px 0 0">Audit chain row appended; retention sweep will pick it up at the SLA.</p>
</div>
<div class="card">
<h3>State change</h3>
<div class="kv">
<span class="k">Candidate</span><span class="v" id="done-cid">…</span>
<span class="k">Status</span><span class="v" id="done-status">Withdrawn</span>
<span class="k">Withdrawn at</span><span class="v" id="done-withdrawn-at">…</span>
<span class="k">Retention until</span><span class="v" id="done-retention">…</span>
</div>
<h3>Audit row hmac</h3>
<div class="hash" id="done-hmac">…</div>
</div>
<div class="card warn">
<h3>What happens next</h3>
<p>The retention sweep flags this subject as overdue once <code>retention_until</code> passes. An operator with legal-tier credentials runs the destruction runbook (<code>POST /biometric/subject/&lt;id&gt;/erase</code>) within the 30-day SLA.</p>
<p style="margin-top:8px">To verify the withdrawal landed cleanly:</p>
<div class="hash" id="verify-cmd">curl -H "X-Lakehouse-Legal-Token: $TOKEN" http://localhost:3100/audit/subject/&lt;id&gt;</div>
</div>
<div style="margin-top:16px">
<button onclick="location.reload()" class="secondary">Record another withdrawal</button>
</div>
</div>
</div>
<div class="foot">
<a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/policies/consent/biometric_consent_template_v1.md">Consent template v1</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md">Destruction runbook</a>
· <a href="https://git.agentview.dev/profit/lakehouse/src/branch/main/docs/PHASE_1_6_BIPA_GATES.md">Phase 1.6 BIPA Gates</a>
</div>
<script>
const GATEWAY = (function(){
const p = new URLSearchParams(location.search);
return p.get('gw') || 'http://localhost:3100';
})();
const state = { token: null, operator: null, response: null };
function show(id) {
document.querySelectorAll('.screen').forEach(s => s.classList.remove('active'));
document.getElementById(id).classList.add('active');
const map = {'screen-token':'step 1 of 3', 'screen-form':'step 2 of 3', 'screen-done':'step 3 of 3'};
document.getElementById('step-label').textContent = map[id] || '';
}
function err(id, msg){ const e=document.getElementById(id); e.textContent=msg; e.classList.add('show'); }
function clearErr(id){ document.getElementById(id).classList.remove('show'); }
// Pre-populate candidate_id from URL if present.
(function init(){
const p = new URLSearchParams(location.search);
const cid = p.get('candidate_id') || p.get('cid');
if (cid) document.getElementById('cand-id').value = cid;
// Restore token from sessionStorage if previously set in this tab.
const saved = sessionStorage.getItem('lh_legal_token');
const savedOp = sessionStorage.getItem('lh_operator_name');
if (saved) {
state.token = saved;
document.getElementById('token-input').value = '••••••••';
}
if (savedOp) {
state.operator = savedOp;
document.getElementById('op-name').value = savedOp;
}
})();
// Step 1.
document.getElementById('token-submit').addEventListener('click', () => {
clearErr('token-error');
const v = document.getElementById('token-input').value.trim();
const tok = v === '••••••••' ? state.token : v;
const op = document.getElementById('op-name').value.trim();
if (!tok || tok.length < 32) { err('token-error', 'Token must be ≥32 characters.'); return; }
if (!op || op.length < 2) { err('token-error', 'Operator name is required.'); return; }
state.token = tok;
state.operator = op;
sessionStorage.setItem('lh_legal_token', tok);
sessionStorage.setItem('lh_operator_name', op);
show('screen-form');
refreshSubmit();
});
// Step 2.
function refreshSubmit() {
const cid = document.getElementById('cand-id').value.trim();
const reason = document.getElementById('reason').value.trim();
document.getElementById('submit-btn').disabled = !(cid.length > 0 && reason.length >= 10);
}
['cand-id','reason'].forEach(id => document.getElementById(id).addEventListener('input', refreshSubmit));
document.getElementById('back-btn').addEventListener('click', () => show('screen-token'));
document.getElementById('submit-btn').addEventListener('click', async () => {
clearErr('form-error');
document.getElementById('submit-btn').disabled = true;
try {
const cid = document.getElementById('cand-id').value.trim();
const body = {
reason: document.getElementById('reason').value.trim(),
operator_of_record: state.operator,
evidence_path: document.getElementById('evidence').value.trim(),
};
const r = await fetch(`${GATEWAY}/biometric/subject/${encodeURIComponent(cid)}/withdraw`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Lakehouse-Legal-Token': state.token,
'X-Lakehouse-Trace-Id': 'withdraw-' + Date.now(),
},
body: JSON.stringify(body),
});
const respBody = await r.json();
if (!r.ok) {
err('form-error', `${r.status} ${respBody.error || 'unknown'} — ${respBody.detail || ''}`);
refreshSubmit();
return;
}
state.response = respBody;
document.getElementById('done-cid').textContent = respBody.candidate_id;
document.getElementById('done-status').textContent = respBody.status_after;
document.getElementById('done-withdrawn-at').textContent = respBody.withdrawn_at;
document.getElementById('done-retention').textContent = respBody.retention_until;
document.getElementById('done-hmac').textContent = respBody.audit_row_hmac;
document.getElementById('verify-cmd').textContent =
`curl -H "X-Lakehouse-Legal-Token: $TOKEN" ${GATEWAY}/audit/subject/${respBody.candidate_id}`;
show('screen-done');
} catch (e) {
err('form-error', `Network error: ${e.message}`);
refreshSubmit();
}
});
</script>
</body>
</html>
+4 -71
View File
@@ -238,77 +238,10 @@ function roleBand(role){
// the left edge + uppercase role pill in the detail line. Used by
// every chapter that renders worker rows. `name` and `role` drive the
// classification; `detail` is the full text after the pill.
// Quick first-name → gender hint for face-pool selection. Same lookup
// idea as the dashboard; if the name is unknown, the server falls back
// to the full pool. Trimmed table — covers the most common names that
// appear in the synthetic worker data.
var FEMALE_NAMES = new Set(['Mary','Patricia','Jennifer','Linda','Elizabeth','Barbara','Susan','Jessica','Sarah','Karen','Lisa','Nancy','Betty','Sandra','Margaret','Ashley','Kimberly','Emily','Donna','Michelle','Carol','Amanda','Melissa','Deborah','Stephanie','Dorothy','Rebecca','Sharon','Laura','Cynthia','Amy','Kathleen','Angela','Shirley','Brenda','Emma','Anna','Pamela','Nicole','Samantha','Katherine','Christine','Helen','Debra','Rachel','Carolyn','Janet','Maria','Catherine','Heather','Diane','Olivia','Julie','Joyce','Victoria','Ruth','Virginia','Lauren','Kelly','Christina','Joan','Evelyn','Judith','Andrea','Hannah','Megan','Cheryl','Jacqueline','Martha','Madison','Teresa','Gloria','Sara','Janice','Ann','Kathryn','Abigail','Sophia','Frances','Jean','Alice','Judy','Isabella','Julia','Grace','Amber','Denise','Danielle','Marilyn','Beverly','Charlotte','Natalie','Theresa','Diana','Brittany','Kayla','Alexis','Lori','Marie','Carmen','Aisha','Rosa','Mia','Audrey','Erin','Tina','Vanessa','Tara','Wendy','Tanya','Maya','Crystal','Yvonne','Kara','Shannon','Brianna','Faith','Caroline','Carla','Tracey','Tracy','Rita','Dawn','Tiffany','Stacy','Stacey','Gina','Bonnie','Tammy','Joanne','Jamie','Tonya','Alyssa','Ariana','Elena','Ellie','Erica','Erika','Felicia','Holly','Jenna','Jenny','Krista','Kristen','Kristin','Krystal','Lana','Leah','Lucy','Mallory','Melinda','Meredith','Misty','Monica','Naomi','Paige','Paula','Renee','Rhonda','Robin','Roxanne','Selena','Sierra','Skylar','Sonia','Stella','Tamara','Veronica','Vivian','Whitney','Yolanda','Zoe']);
var MALE_NAMES = new Set(['James','Robert','John','Michael','David','William','Richard','Joseph','Thomas','Charles','Christopher','Daniel','Matthew','Anthony','Mark','Donald','Steven','Paul','Andrew','Joshua','Kenneth','Kevin','Brian','George','Edward','Ronald','Timothy','Jason','Jeffrey','Ryan','Jacob','Gary','Nicholas','Eric','Jonathan','Stephen','Larry','Justin','Scott','Brandon','Benjamin','Samuel','Gregory','Frank','Alexander','Raymond','Patrick','Jack','Dennis','Jerry','Tyler','Aaron','Jose','Adam','Henry','Nathan','Douglas','Zachary','Peter','Kyle','Walter','Ethan','Jeremy','Harold','Keith','Christian','Roger','Noah','Gerald','Carl','Terry','Sean','Austin','Arthur','Lawrence','Jesse','Dylan','Bryan','Joe','Jordan','Billy','Bruce','Albert','Willie','Gabriel','Logan','Alan','Juan','Wayne','Roy','Ralph','Randy','Eugene','Vincent','Russell','Elijah','Louis','Bobby','Philip','Johnny','Marcus','Antonio','Carlos','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Victor','Jamal','Xavier','DeShawn','Dwayne','Jermaine','Malik','Tyrone','Devon','Andre','Brent','Calvin','Casey','Cody','Cole','Cory','Dale','Damon','Darius','Darrell','Dean','Derek','Drew','Earl','Eddie','Floyd','Glenn','Greg','Howard','Ivan','Jared','Jay','Jeff','Joel','Lance','Lee','Leonard','Lloyd','Mario','Martin','Mason','Maurice','Max','Mitchell','Morgan','Nick','Norman','Oliver','Owen','Pete','Quincy','Rafael','Reggie','Rex','Ricky','Russ','Shane','Shaun','Stanley','Steve','Theodore','Todd','Travis','Trevor','Troy','Wade','Warren','Wesley']);
function guessGenderFromFirstName(n){
if(!n) return null;
var clean=n.replace(/[^A-Za-z]/g,'');
if(!clean) return null;
var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase();
if(FEMALE_NAMES.has(c)) return 'woman';
if(MALE_NAMES.has(c)) return 'man';
return null;
}
function genderFor(name){
var g = guessGenderFromFirstName(name);
if(g) return g;
if(!name) return 'man';
var s=String(name); var h=0;
for(var i=0;i<s.length;i++) h=(h*31+s.charCodeAt(i))|0;
return (Math.abs(h)&1)?'man':'woman';
}
// Confident first-name → ethnicity. Synthetic data — we own the call.
var NAMES_SOUTH_ASIAN_C=new Set(['Raj','Anil','Rohan','Vikram','Arjun','Sanjay','Ravi','Krishna','Pradeep','Sunil','Amit','Deepak','Ashok','Manoj','Rahul','Vijay','Suresh','Naveen','Anand','Nikhil','Aditya','Karan','Rajesh','Priya','Anjali','Neha','Kavya','Pooja','Divya','Meera','Lakshmi','Rani','Asha','Saanvi','Aanya','Aaradhya','Shreya','Riya','Tanvi','Ishita','Aarav','Ishaan','Shivani']);
var NAMES_EAST_ASIAN_C=new Set(['Wei','Mei','Yi','Jin','Chen','Lin','Liu','Wang','Zhang','Yang','Wu','Zhao','Sun','Hiroshi','Yuki','Akira','Kenji','Sakura','Aiko','Haruto','Sora','Hyun','Eun','Yoon','Kai','Long','Hong','Xiu','Lan','Hua','Hao','Tao','Bao','Cheng','Feng','Jian','Dong','Bin','Min','Lei','Hui','Yu','Xin','Ying','Zhen','Yuan','Yan']);
var NAMES_HISPANIC_C=new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon']);
var NAMES_BLACK_C=new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maurice','Tyrell','Kwame','Khalil','Terrell','Cedric','Nia','Zuri','Jada','Ebony','Dominique']);
var NAMES_MIDDLE_EASTERN_C=new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Sami','Wael','Zaid','Amira','Iman','Lina','Mona','Noor','Rana','Soha','Zara']);
// Surname → ethnicity. Surname is more diagnostic than first name
// for hispanic and asian — "Anna Cruz" is hispanic via surname.
var SURNAMES_HISPANIC_C=new Set(['Garcia','Rodriguez','Martinez','Hernandez','Lopez','Gonzalez','Perez','Sanchez','Ramirez','Torres','Flores','Rivera','Gomez','Diaz','Reyes','Cruz','Morales','Ortiz','Gutierrez','Chavez','Ramos','Ruiz','Alvarez','Mendoza','Vasquez','Castillo','Jimenez','Moreno','Romero','Herrera','Medina','Aguilar','Vargas','Castro','Fernandez','Guzman','Munoz','Salazar','Ortega','Delgado','Estrada','Ayala','Pena','Cabrera','Alvarado','Espinoza','Padilla','Cardenas','Cortes','Ibarra','Vega','Soto','Lara','Navarro','Campos','Acosta','Rios','Marquez','Sandoval','Maldonado','Solis','Rojas','Mejia','Beltran','Cervantes','Lozano','Carrillo','Trevino','Robles','Tapia','Lugo']);
var SURNAMES_SOUTH_ASIAN_C=new Set(['Patel','Singh','Kumar','Sharma','Gupta','Shah','Mehta','Desai','Joshi','Reddy','Nair','Iyer','Verma','Agarwal','Kapoor','Chopra','Malhotra','Banerjee','Chatterjee','Mukherjee','Das','Sen','Bose','Roy','Sinha','Trivedi','Pandey','Mishra','Tiwari','Yadav','Chauhan','Rana','Thakur','Pillai','Menon','Krishnan','Rao','Naidu','Pradhan','Acharya','Devi','Kaur']);
var SURNAMES_EAST_ASIAN_C=new Set(['Chen','Wang','Li','Liu','Yang','Huang','Zhao','Wu','Zhou','Xu','Zhu','Sun','Ma','Lin','Lee','Kim','Park','Choi','Jung','Kang','Cho','Yoon','Han','Lim','Oh','Nakamura','Tanaka','Suzuki','Yamamoto','Sato','Watanabe','Takahashi','Kobayashi','Yoshida','Saito','Nguyen','Tran','Le','Pham','Hoang','Phan','Vu','Vo','Dang','Bui','Do','Ngo','Truong','Mai','Cao','Wong','Tang','Tan','Cheng','Lau','Leung','Ng','Cheung','Yip','Hsu','Tsai','Hsieh']);
var SURNAMES_MIDDLE_EASTERN_C=new Set(['Khan','Ahmed','Hussein','Hassan','Ali','Mahmoud','Mohamed','Mohammed','Saleh','Aziz','Karim','Hamad','Najjar','Haddad','Khoury','Mansour','Rahman','Iqbal','Malik','Sheikh','Siddiqui','Qureshi','Saeed']);
function guessEthnicityFromName(first, last){
if(last){
var s=last.replace(/[^A-Za-z]/g,'');
if(s){
var sc=s[0].toUpperCase()+s.slice(1).toLowerCase();
if(SURNAMES_HISPANIC_C.has(sc)) return 'hispanic';
if(SURNAMES_MIDDLE_EASTERN_C.has(sc)) return 'middle_eastern';
if(SURNAMES_SOUTH_ASIAN_C.has(sc)) return 'south_asian';
if(SURNAMES_EAST_ASIAN_C.has(sc)) return 'east_asian';
}
}
if(first){
var clean=first.replace(/[^A-Za-z]/g,'');
if(clean){
var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase();
if(NAMES_MIDDLE_EASTERN_C.has(c)) return 'middle_eastern';
if(NAMES_BLACK_C.has(c)) return 'black';
if(NAMES_HISPANIC_C.has(c)) return 'hispanic';
if(NAMES_SOUTH_ASIAN_C.has(c)) return 'south_asian';
if(NAMES_EAST_ASIAN_C.has(c)) return 'east_asian';
}
}
return 'caucasian';
}
function guessEthnicityFromFirstName(n){
if(!n) return 'caucasian';
var clean=n.replace(/[^A-Za-z]/g,''); if(!clean) return 'caucasian';
var c=clean[0].toUpperCase()+clean.slice(1).toLowerCase();
if(NAMES_MIDDLE_EASTERN_C.has(c)) return 'middle_eastern';
if(NAMES_BLACK_C.has(c)) return 'black';
if(NAMES_HISPANIC_C.has(c)) return 'hispanic';
if(NAMES_SOUTH_ASIAN_C.has(c)) return 'south_asian';
if(NAMES_EAST_ASIAN_C.has(c)) return 'east_asian';
return 'caucasian';
}
// Name → ethnicity / gender inference — REMOVED 2026-05-03 per Phase 1.6
// BIPA gates (docs/PHASE_1_6_BIPA_GATES.md, Gate 4). Mirror of the same
// removal in mcp-server/search.html. Functions had no live consumers
// (headshot rendering disabled 2026-04-28). Do NOT re-introduce.
function workerRow(name, role, detail, opts){
opts = opts || {};
+1 -1
View File
@@ -54,7 +54,7 @@ body{font-family:'Inter',-apple-system,system-ui,sans-serif;background:#090c10;c
</head><body>
<div class="bar">
<h1>Staffing Co-Pilot · Contractor Profile</h1>
<a href="/">← Dashboard</a>
<a href=".">← Dashboard</a>
</div>
<div class="content">
<div class="search-box">
+57 -4
View File
@@ -1140,6 +1140,56 @@ export async function fetchStooqQuote(ticker: string): Promise<{
}
}
// Yahoo Finance v8 chart API — the replacement quote source (2026-06-18).
// Stooq retired its free `q/l/` light-quote CSV endpoint (now 404s for
// non-browser clients; the historical `q/d/l/` CSV sits behind a JS
// browser-verification challenge), so fetchStooqQuote returns null for
// every symbol and the profiler basket renders no quotes. The v8 chart
// endpoint is free, key-less, single-symbol JSON and exposes everything
// the basket needs in `.chart.result[0].meta`: regularMarketPrice (last
// price), chartPreviousClose (for the conventional day-change), plus
// day high/low/volume. Same single-symbol shape as Stooq, so the
// per-ticker fan-out in /intelligence/ticker_quotes is unchanged.
//
// fetchStooqQuote is kept above (dead upstream, not deleted) so the
// swap is reversible if a keyed provider replaces this later.
export async function fetchYahooQuote(ticker: string): Promise<{
price?: number;
prev_close?: number;
price_date?: string;
open?: number;
high?: number;
low?: number;
volume?: number;
} | null> {
try {
const res = await fetch(
`https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(ticker.toUpperCase())}?range=5d&interval=1d`,
{ headers: { "User-Agent": UA }, signal: AbortSignal.timeout(8000) },
);
if (!res.ok) return null;
const j: any = await res.json();
const r = j?.chart?.result?.[0];
const m = r?.meta;
if (!m || typeof m.regularMarketPrice !== "number") return null;
// Last daily candle for the open (meta has no regularMarketOpen).
const q = r?.indicators?.quote?.[0] ?? {};
const lastOpen = Array.isArray(q.open) ? q.open[q.open.length - 1] : undefined;
const t = typeof m.regularMarketTime === "number" ? m.regularMarketTime : null;
return {
price: m.regularMarketPrice,
prev_close: typeof m.chartPreviousClose === "number" ? m.chartPreviousClose : undefined,
price_date: t ? new Date(t * 1000).toISOString().slice(0, 10) : undefined,
open: typeof lastOpen === "number" ? lastOpen : undefined,
high: typeof m.regularMarketDayHigh === "number" ? m.regularMarketDayHigh : undefined,
low: typeof m.regularMarketDayLow === "number" ? m.regularMarketDayLow : undefined,
volume: typeof m.regularMarketVolume === "number" ? m.regularMarketVolume : undefined,
};
} catch {
return null;
}
}
// Cheap name-to-ticker lookup for batch use (e.g. profiler index over
// 200 contractors). Hits the in-memory SEC tickers index and the
// KNOWN_PARENT_MAP. No SEC profile fetch, no Stooq fetch. Returns
@@ -1198,14 +1248,17 @@ export async function fetchTickerBrief(name: string): Promise<TickerBrief> {
}
const [profile, quote] = await Promise.all([
fetchSecProfile(hit.cik_str),
fetchStooqQuote(hit.ticker),
fetchYahooQuote(hit.ticker), // Stooq CSV retired 2026-06-18; see fetchYahooQuote
]);
const cap_proxy =
quote?.price && quote?.volume ? quote.price * quote.volume : undefined;
// Day change vs previous close (conventional); fall back to vs-open.
const day_change_pct =
quote?.price && quote?.open && quote.open > 0
? ((quote.price - quote.open) / quote.open) * 100
: undefined;
quote?.price && quote?.prev_close && quote.prev_close > 0
? ((quote.price - quote.prev_close) / quote.prev_close) * 100
: quote?.price && quote?.open && quote.open > 0
? ((quote.price - quote.open) / quote.open) * 100
: undefined;
const brief: TickerBrief = {
source: "sec+stooq",
fetched_at: now,
+90 -38
View File
@@ -151,7 +151,9 @@ server.tool(
},
async ({ question, sql_filter, dataset, id_column, top_k }) => {
const body: any = {
question, index_name: "workers_500k_v1", filter_dataset: dataset, id_column, top_k, generate: true,
// Index must share an id-space with filter_dataset (default ethereal_workers, 10K)
// → ethereal_workers_v1. workers_500k_v1 (50K, different workers) misaligned.
question, index_name: "ethereal_workers_v1", filter_dataset: dataset, id_column, top_k, generate: true,
use_playbook_memory: true,
};
if (sql_filter) body.sql_filter = sql_filter;
@@ -185,7 +187,8 @@ server.tool(
if (city) filter += ` AND city = '${city}'`;
const r = await api("POST", "/vectors/hybrid", {
question: `Find the best ${role} workers with relevant skills and certifications`,
index_name: "workers_500k_v1", sql_filter: filter,
// ethereal_workers_v1 aligns with filter_dataset ethereal_workers (same 10K ids).
index_name: "ethereal_workers_v1", sql_filter: filter,
filter_dataset: "ethereal_workers", id_column: "worker_id",
top_k: headcount * 2, generate: false,
use_playbook_memory: true,
@@ -519,7 +522,12 @@ async function main() {
}
}
const hybridRes = await api("POST", "/vectors/hybrid", {
question: b.question, index_name: b.index || "workers_500k_v1",
// Default index MUST share an id-space with the default filter_dataset.
// ethereal_workers (10K SQL rows) aligns with ethereal_workers_v1 (10K
// chunks). The old default workers_500k_v1 (50K chunks, different worker set)
// returned wrong-state workers flagged sql_verified:true — e.g. state='IL'
// surfaced a KY worker, because worker_id 1768 is a different person in each.
question: b.question, index_name: b.index || "ethereal_workers_v1",
sql_filter: filter, filter_dataset: b.dataset || "ethereal_workers",
id_column: b.id_column || "worker_id", top_k: b.top_k || 5, generate: b.generate !== false,
use_playbook_memory: b.use_playbook_memory !== false,
@@ -551,7 +559,8 @@ async function main() {
if (b.city) filter += ` AND city = '${b.city}'`;
return ok(await api("POST", "/vectors/hybrid", {
question: `Best ${b.role} workers with relevant skills`,
index_name: b.index || "workers_500k_v1", sql_filter: filter,
// Align default index with the ethereal_workers dataset (see /search fix).
index_name: b.index || "ethereal_workers_v1", sql_filter: filter,
filter_dataset: b.dataset || "ethereal_workers",
id_column: "worker_id", top_k: (b.headcount || 5) * 2, generate: false,
use_playbook_memory: true,
@@ -767,6 +776,37 @@ async function main() {
});
}
// Biometric intake — Phase 1.6 Gate 2 frontend. Operator-driven
// candidate consent + photo capture flow. POSTs to gateway's
// /biometric/subject/{id}/consent + /photo. URL must include
// ?candidate_id=WORKER-XXX. Operator's legal-tier audit token
// is captured into sessionStorage (cleared on tab close).
if (url.pathname === "/biometric/intake") {
return new Response(Bun.file(import.meta.dir + "/biometric_intake.html"), {
headers: { ...cors, "Content-Type": "text/html" },
});
}
// Biometric withdrawal — BIPA right of withdrawal frontend.
// Operator records candidate's withdrawal request. POSTs to
// gateway's /biometric/subject/{id}/withdraw. Sets a 30-day
// SLA clock for destruction. Optional ?candidate_id=WORKER-XXX
// pre-populates the form.
if (url.pathname === "/biometric/withdraw") {
return new Response(Bun.file(import.meta.dir + "/biometric_withdraw.html"), {
headers: { ...cors, "Content-Type": "text/html" },
});
}
// Biometric ops dashboard — read-only aggregate over
// /biometric/stats. Counts by status + recent state-change
// events. Auto-refreshes every 15s. Operator-tier auth.
if (url.pathname === "/biometric/dashboard") {
return new Response(Bun.file(import.meta.dir + "/biometric_dashboard.html"), {
headers: { ...cors, "Content-Type": "text/html" },
});
}
// Workspaces — per-contract state (Phase 8.5). UI layer over the
// gateway's /workspaces/* routes: list, create, detail, handoff,
// save-search, shortlist, log-activity. All persisted on the
@@ -905,45 +945,57 @@ async function main() {
}
}
// Batch ticker quotes — used by the profiler page's scrolling
// ticker basket. Stooq's HTTP CSV API is single-symbol per call,
// so this fans out N tickers in parallel and returns a flat
// map. Non-US tickers (HOC.DE, SKA-B.ST, LLC.AX) won't have a
// Stooq.us entry; we surface that as null so the UI can render
// them with a "—" placeholder.
// Batch ticker quotes — the profiler basket. Multi-source
// consensus (Yahoo + Nasdaq + CNBC), cache-first from the local
// SQLite time-series; cold tickers are aggregated live once and
// cached. Each quote carries price + day_change_pct (median across
// sources), a confidence score (corroborating sources / 3), the
// per-source breakdown, and a stale flag. Dotted/foreign symbols
// (HOC.DE, SKA-B.ST) are skipped → null → "—" placeholder.
// See quotes.ts; refreshed hourly by quote_ingest.ts.
if (url.pathname === "/intelligence/ticker_quotes" && req.method === "POST") {
const start = Date.now();
const b = await json();
const tickers: string[] = Array.isArray(b.tickers) ? b.tickers.map((t: any) => String(t).toUpperCase()).filter(Boolean) : [];
if (!tickers.length) return ok({ quotes: {}, duration_ms: 0 });
const { fetchStooqQuote } = await import("./entity.js");
const dedup = Array.from(new Set(tickers)).slice(0, 50);
const results = await Promise.all(dedup.map(async (t) => {
// Skip non-US suffixes — Stooq.us won't have them
if (t.includes(".")) return [t, null] as const;
try {
const q = await fetchStooqQuote(t);
if (!q || !q.price) return [t, null] as const;
const change_pct = q.open && q.open > 0 ? ((q.price - q.open) / q.open) * 100 : null;
return [t, {
ticker: t,
price: q.price,
price_date: q.price_date,
open: q.open,
high: q.high,
low: q.low,
day_change_pct: change_pct,
stooq_url: `https://stooq.com/q/?s=${t.toLowerCase()}.us`,
}] as const;
} catch {
return [t, null] as const;
}
}));
const { serveQuotes } = await import("./quotes.ts");
const dedup = Array.from(new Set(tickers)).slice(0, 50).filter((t) => !t.includes("."));
const served = await serveQuotes(dedup);
const quotes: Record<string, any> = {};
for (const [t, q] of results) quotes[t] = q;
for (const t of dedup) {
const a = served[t];
quotes[t] = a
? {
ticker: t,
price: a.price,
day_change_pct: a.day_change_pct,
price_date: a.price_date,
confidence: a.confidence,
source_count: a.source_count,
spread_pp: a.spread_pp,
sources: a.sources,
stale: a.stale,
as_of: a.ts,
source_url: `https://finance.yahoo.com/quote/${t}`,
}
: null;
}
return ok({ quotes, count: dedup.length, duration_ms: Date.now() - start });
}
// Manual quote-ingest trigger (admin / timer). POST {tickers?:[]}
// — with a list, ingests those; without, refreshes every tracked
// ticker. The systemd timer hits this hourly; also callable by
// hand. Returns the write summary.
if (url.pathname === "/intelligence/quote_ingest" && req.method === "POST") {
const start = Date.now();
const b = await json().catch(() => ({}));
const list = Array.isArray(b?.tickers) ? b.tickers.map((t: any) => String(t).toUpperCase()) : undefined;
const { ingestQuotes } = await import("./quotes.ts");
const r = await ingestQuotes(list);
return ok({ ...r, duration_ms: Date.now() - start });
}
// Profiler index — directory of every contractor that has filed
// a Chicago permit recently, ranked by permit count + total
// cost. Each name in the response links to the full /contractor
@@ -2363,7 +2415,7 @@ async function main() {
queries.push(`Vector: Semantic similarity on ${worker.name}'s full profile → ${stateFilter}`);
const searchR = await api("POST", "/vectors/hybrid", {
question: worker.resume_text || `${worker.role} in ${worker.city} with skills ${worker.skills}`,
index_name: "workers_500k_v1",
index_name: "ethereal_workers_v1", // align with ethereal_workers dataset
sql_filter: stateFilter + ` AND CAST(reliability AS DOUBLE) >= 0.7`,
filter_dataset: "ethereal_workers", id_column: "worker_id", top_k: 5, generate: false,
});
@@ -2404,7 +2456,7 @@ async function main() {
queries.push(`Vector: Semantic search for "${roleDesc}" — no exact role match needed`);
const searchR = await api("POST", "/vectors/hybrid", {
question: `Worker experienced in ${roleDesc}, relevant skills and certifications`,
index_name: "workers_500k_v1", sql_filter: "CAST(reliability AS DOUBLE) >= 0.75",
index_name: "ethereal_workers_v1", sql_filter: "CAST(reliability AS DOUBLE) >= 0.75", // align w/ dataset
filter_dataset: "ethereal_workers", id_column: "worker_id", top_k: 8, generate: false,
});
return ok({ type: "discovery", summary: `${(searchR.sources||[]).length} workers found through semantic skill matching for: "${roleDesc}"`,
@@ -2814,7 +2866,7 @@ async function main() {
const [searchR, directR, patternR] = await Promise.all([
api("POST", "/vectors/hybrid", {
question: q, index_name: "workers_500k_v1", sql_filter: filterStr,
question: q, index_name: "ethereal_workers_v1", sql_filter: filterStr, // align w/ dataset
filter_dataset: "ethereal_workers", id_column: "worker_id",
// Honor the parsed headcount (capped at 25 to keep the
// vector rerank from re-scoring more rows than render).
@@ -3476,7 +3528,7 @@ async function runWeekSimulation() {
const filt = `role = '${role}' AND state = '${state}' AND reliability >= ${minRel}`;
const r = await api("POST", "/vectors/hybrid", {
question: `Find ${role} workers in ${city}, ${state} for ${scenario.situation}`,
index_name: "workers_500k_v1",
index_name: "ethereal_workers_v1", // align with ethereal_workers dataset
sql_filter: filt,
filter_dataset: "ethereal_workers",
id_column: "worker_id",
+1 -1
View File
@@ -771,7 +771,7 @@ async function tailOverseerCorrections(): Promise<number> {
try { row = JSON.parse(line); } catch { continue; }
const op: ObservedOp = {
timestamp: row.created_at ?? new Date().toISOString(),
endpoint: `overseer:${row.model ?? "claude-opus-4-7"}`,
endpoint: `overseer:${row.model ?? "qwen3.5:latest"}`,
input_summary: `${row.task_class ?? "?"}: ${row.reason ?? "escalation"}`,
// Correction itself is neither success nor failure — it's a
// mitigation attempt. We mark success=true so analyzeErrors
+1
View File
@@ -91,6 +91,7 @@ table.preview tr:hover td{background:#0d1117}
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="profiler">Profiler</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard" class="active">Onboard</a>
+170
View File
@@ -0,0 +1,170 @@
// Phase 1.6 Gate 4 absence test.
//
// Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 4 — Engineering acceptance:
// "Unit test asserts no protected-attribute inference functions exist
// in search.html or any mcp-server module"
//
// What this guards: the FEMALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup
// tables and the genderFor() / guessEthnicityFromFirstName() / etc.
// inference functions removed 2026-05-03. Re-introduction would re-open
// (1) Title VII / IL Human Rights Act discriminatory-feature risk and
// (2) BIPA's broad-reading "biometric information derived from a biometric
// identifier" pattern when combined with deepface output.
//
// Strategy: walk every .html / .ts / .tsx / .js / .mjs file under
// mcp-server/ and grep-assert that none of them DEFINE the forbidden
// symbols. We deliberately allow the symbol NAMES to appear inside
// comments — search.html has a removal note that names them so future
// readers know what was excised — but we forbid actual definition
// patterns (var / const / let / function / class member / object literal).
import { test, expect } from "bun:test";
import { readdirSync, statSync, readFileSync } from "node:fs";
import { join } from "node:path";
const FORBIDDEN_DATA_TABLES = [
// First-name lookup tables
"FEMALE_NAMES",
"MALE_NAMES",
"NAMES_HISPANIC",
"NAMES_BLACK",
"NAMES_SOUTH_ASIAN",
"NAMES_EAST_ASIAN",
"NAMES_MIDDLE_EASTERN",
// Surname lookup tables
"SURNAMES_HISPANIC",
"SURNAMES_BLACK",
"SURNAMES_SOUTH_ASIAN",
"SURNAMES_EAST_ASIAN",
"SURNAMES_MIDDLE_EASTERN",
];
const FORBIDDEN_FUNCTIONS = [
"guessGenderFromFirstName",
"guessEthnicityFromName",
"guessEthnicityFromFirstName",
"genderFor",
];
function* walkSource(dir: string): Generator<string> {
for (const entry of readdirSync(dir)) {
if (entry === "node_modules" || entry === "dist" || entry.startsWith(".")) continue;
const path = join(dir, entry);
const stat = statSync(path);
if (stat.isDirectory()) {
yield* walkSource(path);
} else if (/\.(html|ts|tsx|js|mjs)$/.test(entry)) {
// Don't grep this test file itself — it lists the forbidden tokens
// by name as match targets, not as definitions.
if (path.endsWith("phase_1_6_gate_4.test.ts")) continue;
yield path;
}
}
}
// definitionPatternsFor: returns regexes that match common DEFINITION
// forms in JS/TS/HTML embedded scripts. A bare reference inside a
// comment is intentionally NOT matched.
//
// 2026-05-03 opus scrum WARN (gate_4_test:60) added object-literal +
// class-field patterns: a developer wrapping the lookup tables in
// `const tables = { FEMALE_NAMES: [...] }` or a TypeScript class
// field `FEMALE_NAMES: string[] = [...]` would have bypassed the
// original 4 patterns silently.
function definitionPatternsFor(symbol: string): RegExp[] {
return [
// var / const / let SYMBOL =
new RegExp(`\\b(?:var|const|let)\\s+${symbol}\\b\\s*=`),
// function SYMBOL(
new RegExp(`\\bfunction\\s+${symbol}\\s*\\(`),
// SYMBOL = function( OR SYMBOL = (...) =>
new RegExp(`\\b${symbol}\\s*=\\s*(?:function\\s*\\(|\\([^)]*\\)\\s*=>|async\\s*(?:\\(|function))`),
// class method: SYMBOL(...) {
new RegExp(`^\\s*${symbol}\\s*\\([^)]*\\)\\s*\\{`, "m"),
// object-literal property assigned to an array OR object value:
// { SYMBOL: [...] } or SYMBOL: {...} or SYMBOL: new Set(...)
new RegExp(`(?:^|[,{])\\s*${symbol}\\s*:\\s*(?:\\[|\\{|new\\s+(?:Set|Map|Array)\\b)`, "m"),
// TypeScript / class field with type annotation. The boundary
// before SYMBOL is start-of-line OR `{`/`;`/`}` so single-line
// class bodies (`class L { public NAMES_X: string[] = []; }`)
// are caught alongside multi-line ones.
new RegExp(`(?:^|[{};])\\s*(?:public|private|protected|readonly|static\\s+)*\\s*${symbol}\\s*:\\s*[^=;{]+=\\s*[\\[\\{]`, "m"),
];
}
function findOffenders(filePath: string, text: string): string[] {
const out: string[] = [];
for (const sym of [...FORBIDDEN_DATA_TABLES, ...FORBIDDEN_FUNCTIONS]) {
for (const pattern of definitionPatternsFor(sym)) {
if (pattern.test(text)) {
out.push(`${filePath}: definition of ${sym} (matched ${pattern})`);
}
}
}
return out;
}
test("Gate 4: no protected-attribute inference DEFINITIONS in mcp-server", () => {
const root = import.meta.dir;
const offenders: string[] = [];
for (const path of walkSource(root)) {
const text = readFileSync(path, "utf8");
offenders.push(...findOffenders(path, text));
}
if (offenders.length > 0) {
throw new Error(
`Phase 1.6 Gate 4 violation — protected-attribute inference symbols defined in mcp-server:\n` +
offenders.map((o) => ` - ${o}`).join("\n"),
);
}
expect(offenders.length).toBe(0);
});
// Sanity: confirm the test actually walks files (otherwise the absence
// assertion is vacuously true). If mcp-server ever lost its source
// tree, this would catch it.
test("Gate 4: walker actually finds source files to scan", () => {
const root = import.meta.dir;
let count = 0;
for (const _ of walkSource(root)) count++;
expect(count).toBeGreaterThan(5); // mcp-server has more than 5 source files
});
// Defense in depth: the regex itself must catch a synthetic positive.
// If the definition pattern ever stops matching real code, the absence
// test would silently pass on actual reintroductions.
test("Gate 4: regex catches a synthetic positive (defense in depth)", () => {
const synthetic =
`var NAMES_HISPANIC = ["Maria"];\n` +
`function guessEthnicityFromFirstName(name) { return "?"; }\n`;
const offenders = findOffenders("synthetic_test_input", synthetic);
expect(offenders.length).toBeGreaterThanOrEqual(2);
expect(offenders.some((o) => o.includes("NAMES_HISPANIC"))).toBe(true);
expect(offenders.some((o) => o.includes("guessEthnicityFromFirstName"))).toBe(true);
});
// 2026-05-03 opus scrum WARN regression: the bypass forms a developer
// might use to wrap the lookup tables without tripping the original
// four patterns. All of these MUST trip a definition regex.
test("Gate 4: object-literal and class-field bypasses are caught", () => {
const bypassForms = [
// Inline object-literal property → array
`const tables = { FEMALE_NAMES: ["Maria"] };`,
// Object property → Set/Map constructor
`const lookups = { NAMES_BLACK: new Set(["X"]) };`,
// Multi-line object literal
`const all = {\n NAMES_HISPANIC: [...],\n};`,
// TypeScript class field with type annotation + initializer
`class Lookup {\n SURNAMES_BLACK: string[] = ["X"];\n}`,
// TypeScript public field
`class L { public NAMES_EAST_ASIAN: string[] = []; }`,
];
for (const synthetic of bypassForms) {
const offenders = findOffenders("bypass_synthetic", synthetic);
if (offenders.length === 0) {
throw new Error(
`Gate 4 bypass not caught — pattern would slip past:\n ${synthetic}`,
);
}
}
});
+24 -3
View File
@@ -87,6 +87,11 @@ td.role .pill{display:inline-block;padding:2px 7px;border-radius:9px;font-size:9
.bk-card .ch.up{color:#3fb950}
.bk-card .ch.down{color:#f85149}
.bk-card .ch.flat{color:#545d68}
.bk-card .cf{font-family:ui-monospace,monospace;font-size:9px;margin-top:3px;font-weight:700;letter-spacing:0.3px;display:inline-block;padding:1px 5px;border-radius:4px}
.bk-card .cf.full{background:#0d2818;border:1px solid #2ea04366;color:#3fb950}
.bk-card .cf.part{background:#1a1410;border:1px solid #d2992266;color:#d29922}
.bk-card .cf.thin{background:#2d1316;border:1px solid #f8514966;color:#f85149}
.bk-card .cf.stale{opacity:0.6;border-style:dashed}
.bk-card .meta{font-size:9px;color:#545d68;margin-top:5px;text-transform:uppercase;letter-spacing:0.6px}
.bk-card .kind-bar{position:absolute;left:0;top:0;bottom:0;width:3px;border-radius:8px 0 0 8px}
.bk-card .kind-bar.exact,.bk-card .kind-bar.direct{background:#3fb950}
@@ -108,8 +113,8 @@ td.role .pill{display:inline-block;padding:2px 7px;border-radius:9px;font-size:9
<div class="bar">
<h1>Staffing Co-Pilot · Profiler Index</h1>
<nav>
<a href="" id="back-dashboard">← Dashboard</a>
<a href="" id="back-console">Console</a>
<a href="." id="back-dashboard">← Dashboard</a>
<a href="console" id="back-console">Console</a>
</nav>
</div>
<div class="content">
@@ -303,6 +308,7 @@ function buildBasket(){
var tk=document.createElement('div'); tk.className='tk'; tk.textContent=b.ticker; card.appendChild(tk);
var px=document.createElement('div'); px.className='px'; px.textContent='—'; card.appendChild(px);
var ch=document.createElement('div'); ch.className='ch flat'; ch.textContent=' '; card.appendChild(ch);
var cf=document.createElement('div'); cf.className='cf'; cf.textContent=''; card.appendChild(cf);
var meta=document.createElement('div'); meta.className='meta';
meta.textContent=b.count+' attribution'+(b.count===1?'':'s')+' · '+b.kinds.join('+');
card.appendChild(meta);
@@ -328,7 +334,7 @@ function buildBasket(){
}).then(function(r){return r.json()}).then(function(qd){
var quotes=qd.quotes||{};
lastQuotes = quotes;
document.getElementById('bk-meta').textContent='quotes via Stooq · '+(qd.duration_ms||0)+'ms';
document.getElementById('bk-meta').textContent='quotes via 3-source consensus (Yahoo·Nasdaq·CNBC) · '+(qd.duration_ms||0)+'ms';
Array.prototype.forEach.call(track.children, function(card){
var t=card.dataset.ticker; var q=quotes[t];
if(!q || !q.price) return;
@@ -338,6 +344,21 @@ function buildBasket(){
if(q.day_change_pct==null){ ch.textContent='close '+(q.price_date||''); ch.className='ch flat'; }
else if(q.day_change_pct>=0){ ch.textContent='+'+q.day_change_pct.toFixed(2)+'%'; ch.className='ch up'; }
else { ch.textContent=q.day_change_pct.toFixed(2)+'%'; ch.className='ch down'; }
// Multi-source confidence badge: corroborating sources / 3. Green
// at full agreement, amber when one source is missing/divergent,
// red when thin. The whole differentiator made visible per-ticker.
var cf=card.querySelector('.cf');
if(cf && q.confidence!=null){
var cls = q.confidence>=100 ? 'cf full' : q.confidence>=67 ? 'cf part' : 'cf thin';
if(q.stale) cls += ' stale';
cf.className = cls;
cf.textContent = (q.source_count||0)+'/3 ✓ '+q.confidence+'%'+(q.stale?' · stale':'');
// Tooltip: per-source breakdown so a staffer can audit the number.
var lines=(q.sources||[]).map(function(s){
return ' '+s.source+': '+(s.day_change_pct>=0?'+':'')+s.day_change_pct.toFixed(2)+'%'+(s.outlier?' ⚠ outlier':'');
}).join('\n');
card.title=(q.ticker||t)+' — '+q.confidence+'% confidence ('+(q.source_count||0)+'/3 sources, spread '+(q.spread_pp!=null?q.spread_pp.toFixed(2):'?')+'pp)\n'+lines;
}
});
updateThesisMetrics();
}).catch(function(){
+41
View File
@@ -478,6 +478,15 @@ bun test auditor/schemas/distillation/ tests/distillation/
</div>
</div>
<div class="chapter">
<div class="num">Chapter 11</div>
<h2>Substrate signals — live architecture health</h2>
<div class="lede">Four live probes of the architecture itself, not the staffing workload: instant-search latency, index shape, playbook-memory depth, and pathway-matrix compounding. These answer "is the substrate healthy right now?" — moved here from the operator dashboard (2026-05-30) where they competed with the day's work.</div>
<div class="grid" id="arch-signals" style="grid-template-columns:repeat(auto-fit,minmax(200px,1fr));gap:10px">
<div class="loading" style="grid-column:1/-1">Probing substrate…</div>
</div>
</div>
</div>
<div class="footer">Lakehouse · Architecture page regenerates tests on every load · <a href="console">walkthrough</a> · <a href=".">dashboard</a></div>
@@ -501,8 +510,40 @@ window.addEventListener('load',function(){
loadLiveSections();
loadPathwayLive();
loadSignalLive();
loadArchSignals();
});
// ⑤ Substrate Signals — moved from the operator dashboard (search.html)
// 2026-05-30 so the dashboard stays focused on staffing work and the
// architecture-health probes live with the rest of the architecture page.
// Uses apiPost (the /intelligence/arch_signals endpoint accepts GET or POST).
function loadArchSignals(){
var el=document.getElementById('arch-signals');
if(!el) return;
apiPost('/intelligence/arch_signals',{}).then(function(s){
el.textContent='';
function tile(label, big, sub, accent){
var t=document.createElement('div');
t.style.cssText='background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:14px 16px;border-left:3px solid '+(accent||'#58a6ff');
var l=document.createElement('div');l.style.cssText='font-size:10px;text-transform:uppercase;letter-spacing:1px;color:#545d68;margin-bottom:6px';l.textContent=label;
var b=document.createElement('div');b.style.cssText='font-size:20px;font-weight:600;color:#e6edf3;line-height:1.1';b.textContent=big;
var u=document.createElement('div');u.style.cssText='font-size:10px;color:#8b949e;margin-top:6px;line-height:1.4';u.textContent=sub;
t.appendChild(l);t.appendChild(b);t.appendChild(u);
return t;
}
var idx=s.index||{};
var pbm=s.playbook_memory||{};
var pwm=s.pathway_memory||{};
var latencyColor=s.instant_search_probe_ms<100?'#2ea043':s.instant_search_probe_ms<500?'#d29922':'#f85149';
el.appendChild(tile('Instant Search',(s.instant_search_probe_ms||'?')+'ms','Live /vectors/hybrid probe · 500K-row index · '+(idx.chunk_count||0).toLocaleString()+' chunks',latencyColor));
el.appendChild(tile('Index',(idx.dimensions||768)+'d · '+(idx.model||'?'),(idx.source||'?')+' → '+(idx.name||'?')+' · '+(idx.backend||'parquet'),'#58a6ff'));
el.appendChild(tile('Playbook Memory',(pbm.entries||0).toLocaleString()+' entries',pbm.entries>0?'Meta-index active · boosts candidates from past fills':'Empty · POST /vectors/playbook_memory/rebuild to populate',pbm.entries>0?'#2ea043':'#d29922'));
el.appendChild(tile('Pathway Matrix',(pwm.total_pathways||0)+' traces',pwm.retired+' retired · '+pwm.total_replays+' replays · ADR-021 compounding','#58a6ff'));
}).catch(function(e){
el.textContent='substrate signals unavailable: '+e.message;
});
}
// Pathway memory live counters in Chapter 4 — small inline spans.
function loadPathwayLive(){
fetch(A+'/api/vectors/pathway/stats').then(function(r){return r.json()}).then(function(p){
+31
View File
@@ -0,0 +1,31 @@
// Hourly quote ingest — CLI entry for the systemd timer (and manual runs).
//
// bun run quote_ingest.ts AAPL MSFT CAT # ingest a specific list
// bun run quote_ingest.ts # refresh every tracked ticker
//
// Writes multi-source consensus snapshots into the local SQLite cache
// (data/_quote_cache/quotes.db). See quotes.ts for the aggregation logic.
import { ingestQuotes, trackedTickers } from "./quotes.ts";
const args = process.argv.slice(2).map((s) => s.toUpperCase());
const t0 = Date.now();
const tracked = trackedTickers();
const target = args.length ? args : tracked;
if (!target.length) {
console.log(
"[quote_ingest] nothing to ingest — cache is empty and no tickers passed.\n" +
" Seed it by passing symbols (e.g. `bun run quote_ingest.ts AAPL CAT HD`),\n" +
" or just hit the profiler once so the serving path warms the cache.",
);
process.exit(0);
}
console.log(
`[quote_ingest] ${args.length ? "explicit list" : "refreshing tracked universe"}: ${target.length} tickers`,
);
const r = await ingestQuotes(target);
console.log(
`[quote_ingest] done in ${Date.now() - t0}ms — ${r.written}/${r.requested} written` +
(r.failed.length ? ` · no source answered for: ${r.failed.join(", ")}` : ""),
);
+347
View File
@@ -0,0 +1,347 @@
// Multi-source quote aggregation + local hourly cache (2026-06-18).
//
// Why this exists: a single quote source can be confidently wrong and
// give you no way to know (Stooq died silently; Yahoo's chartPreviousClose
// over a 5d window references the pre-window close, so a Yahoo-only day
// change reads +1.5% when the real figure is -1.1%). The fix isn't a
// "better" single source — it's CONSENSUS. Pull the same ticker from
// several independent feeds, take the median, and score confidence by how
// many of them corroborate it. Disagreement becomes a signal, not a
// hidden bug. This is the differentiator: the index's numbers come with a
// provenance + confidence trail, not a single opaque vendor call.
//
// Real-time is not a requirement here (per product call): an hourly
// ingest job writes aggregates into a local SQLite time-series, and the
// serving path reads the cache. Cold tickers are aggregated on-demand and
// cached so the page never blocks on a missing symbol.
//
// Sources (all keyless from this host, verified 2026-06-18):
// - Yahoo Finance v8 chart (query1.finance.yahoo.com)
// - Nasdaq quote API (api.nasdaq.com)
// - CNBC quote webservice (quote.cnbc.com)
import { Database } from "bun:sqlite";
// Consumer finance endpoints (Nasdaq/CNBC) gate on a browser-shaped UA —
// the custom courtesy UA gets intermittently dropped. Low-volume hourly
// polling of public no-auth JSON, so a standard browser UA is the right
// call for reliability.
const UA =
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36";
// A source counts as "corroborating" the consensus if its day-change is
// within this many percentage points of the median. Legit feeds differ by
// timing/rounding (<0.1pp); a stale or wrong-baseline feed lands far
// outside and is flagged as an outlier.
const TOL_PP = 0.2;
// Confidence is scored against this many target sources, so a single
// source can never read as "fully confident" — corroboration needs peers.
const TARGET_SOURCES = 3;
// Serving marks a cached aggregate "stale" past this age (the hourly job
// should keep everything fresher than this); still served, just flagged.
export const STALE_AFTER_MS = 90 * 60 * 1000;
const DB_PATH = `${import.meta.dir}/../data/_quote_cache/quotes.db`;
export type SourceQuote = {
source: string;
price: number;
day_change_pct: number;
price_date?: string;
};
export type AggregateSource = SourceQuote & { outlier: boolean };
export type AggregateQuote = {
ticker: string;
price: number; // median price across corroborating sources
day_change_pct: number; // median day change across sources
confidence: number; // 0-100 — corroborating sources / TARGET_SOURCES
source_count: number;
spread_pp: number; // max-min day change across sources (dispersion)
sources: AggregateSource[];
price_date?: string;
ts: number; // unix ms when aggregated
};
// ── per-source fetchers ─────────────────────────────────────────────
// Each returns a normalized SourceQuote or null (never throws). A null
// just means "this feed didn't answer for this symbol" — the aggregator
// works with whatever subset responds.
const num = (s: any): number | null => {
if (typeof s === "number") return Number.isFinite(s) ? s : null;
if (typeof s !== "string") return null;
const n = parseFloat(s.replace(/[$,%\s]/g, ""));
return Number.isFinite(n) ? n : null;
};
async function fromYahoo(ticker: string): Promise<SourceQuote | null> {
try {
const res = await fetch(
`https://query1.finance.yahoo.com/v8/finance/chart/${encodeURIComponent(ticker.toUpperCase())}?range=5d&interval=1d`,
{ headers: { "User-Agent": UA }, signal: AbortSignal.timeout(8000) },
);
if (!res.ok) return null;
const j: any = await res.json();
const r = j?.chart?.result?.[0];
const price = r?.meta?.regularMarketPrice;
if (typeof price !== "number") return null;
// Prev close = the daily close BEFORE the latest candle. meta's
// chartPreviousClose is the pre-window close over range=5d (wrong
// baseline), so derive from the candle array instead.
const closes: number[] = (r?.indicators?.quote?.[0]?.close ?? []).filter(
(c: any) => typeof c === "number",
);
const prev = closes.length >= 2 ? closes[closes.length - 2] : undefined;
if (!prev || prev <= 0) return null;
const t = typeof r?.meta?.regularMarketTime === "number" ? r.meta.regularMarketTime : null;
return {
source: "yahoo",
price,
day_change_pct: ((price - prev) / prev) * 100,
price_date: t ? new Date(t * 1000).toISOString().slice(0, 10) : undefined,
};
} catch {
return null;
}
}
async function fromNasdaq(ticker: string): Promise<SourceQuote | null> {
try {
const res = await fetch(
`https://api.nasdaq.com/api/quote/${encodeURIComponent(ticker.toUpperCase())}/info?assetclass=stocks`,
{ headers: { "User-Agent": UA, Accept: "application/json" }, signal: AbortSignal.timeout(8000) },
);
if (!res.ok) return null;
const j: any = await res.json();
const p = j?.data?.primaryData;
const price = num(p?.lastSalePrice);
const pct = num(p?.percentageChange);
if (price == null || pct == null) return null;
return { source: "nasdaq", price, day_change_pct: pct };
} catch {
return null;
}
}
async function fromCnbc(ticker: string): Promise<SourceQuote | null> {
try {
const res = await fetch(
`https://quote.cnbc.com/quote-html-webservice/restQuote/symbolType/symbol?symbols=${encodeURIComponent(ticker.toUpperCase())}&requestMethod=itv&output=json`,
{ headers: { "User-Agent": UA }, signal: AbortSignal.timeout(8000) },
);
if (!res.ok) return null;
const j: any = await res.json();
const q = j?.FormattedQuoteResult?.FormattedQuote?.[0];
const price = num(q?.last);
const pct = num(q?.change_pct);
if (price == null || pct == null) return null;
return { source: "cnbc", price, day_change_pct: pct, price_date: q?.last_time };
} catch {
return null;
}
}
const FETCHERS = [fromYahoo, fromNasdaq, fromCnbc];
// ── aggregation ─────────────────────────────────────────────────────
function median(xs: number[]): number {
const s = [...xs].sort((a, b) => a - b);
const m = Math.floor(s.length / 2);
return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;
}
// Fetch every source in parallel and fuse into one consensus quote.
// Returns null only if NO source answered.
export async function aggregateTicker(ticker: string): Promise<AggregateQuote | null> {
const got = (await Promise.all(FETCHERS.map((f) => f(ticker)))).filter(
(q): q is SourceQuote => q != null,
);
if (!got.length) return null;
const changes = got.map((q) => q.day_change_pct);
const consensusChange = median(changes);
const consensusPrice = median(got.map((q) => q.price));
const spread = Math.max(...changes) - Math.min(...changes);
const sources: AggregateSource[] = got.map((q) => ({
...q,
outlier: Math.abs(q.day_change_pct - consensusChange) > TOL_PP,
}));
// Confidence = how many independent sources corroborate the served
// number, out of the target. 3 agree → 100; 2 of 3 → 67 (1 flagged);
// a lone source → 33. Honest about thin or split evidence.
const corroborating = sources.filter((s) => !s.outlier).length;
const confidence = Math.round((100 * corroborating) / TARGET_SOURCES);
return {
ticker: ticker.toUpperCase(),
price: consensusPrice,
day_change_pct: consensusChange,
confidence,
source_count: got.length,
spread_pp: spread,
sources,
price_date: got.find((q) => q.price_date)?.price_date,
ts: Date.now(),
};
}
// ── SQLite time-series cache ─────────────────────────────────────────
let _db: Database | null = null;
function db(): Database {
if (_db) return _db;
// Ensure the directory exists (Bun has no mkdir on Database open).
const dir = DB_PATH.slice(0, DB_PATH.lastIndexOf("/"));
try {
require("node:fs").mkdirSync(dir, { recursive: true });
} catch {}
const d = new Database(DB_PATH, { create: true });
d.exec("PRAGMA journal_mode = WAL;");
// The serving path (gateway process) and the hourly ingest (timer
// process) both write this db. WAL allows concurrent readers, but two
// writers still serialize — busy_timeout makes the loser wait instead
// of throwing SQLITE_BUSY.
d.exec("PRAGMA busy_timeout = 5000;");
d.exec(`
CREATE TABLE IF NOT EXISTS quote_observations (
ticker TEXT NOT NULL, ts INTEGER NOT NULL, source TEXT NOT NULL,
price REAL, day_change_pct REAL,
PRIMARY KEY (ticker, ts, source)
);
CREATE TABLE IF NOT EXISTS quote_aggregates (
ticker TEXT NOT NULL, ts INTEGER NOT NULL,
price REAL, day_change_pct REAL, confidence INTEGER,
source_count INTEGER, spread_pp REAL, sources TEXT, price_date TEXT,
PRIMARY KEY (ticker, ts)
);
CREATE INDEX IF NOT EXISTS idx_agg_ticker_ts ON quote_aggregates(ticker, ts DESC);
`);
_db = d;
return d;
}
// One row per ticker per HOUR bucket — re-running within the hour
// overwrites (idempotent), so a manual re-trigger never double-counts.
function hourBucket(tsMs: number): number {
return Math.floor(tsMs / 3_600_000) * 3_600_000;
}
export function writeAggregate(agg: AggregateQuote): void {
const d = db();
const bucket = hourBucket(agg.ts);
const obs = d.prepare(
"INSERT OR REPLACE INTO quote_observations (ticker, ts, source, price, day_change_pct) VALUES (?, ?, ?, ?, ?)",
);
const tx = d.transaction((a: AggregateQuote) => {
for (const s of a.sources) obs.run(a.ticker, bucket, s.source, s.price, s.day_change_pct);
d.prepare(
`INSERT OR REPLACE INTO quote_aggregates
(ticker, ts, price, day_change_pct, confidence, source_count, spread_pp, sources, price_date)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
a.ticker,
bucket,
a.price,
a.day_change_pct,
a.confidence,
a.source_count,
a.spread_pp,
JSON.stringify(a.sources),
a.price_date ?? null,
);
});
tx(agg);
}
function rowToAggregate(row: any): AggregateQuote {
return {
ticker: row.ticker,
price: row.price,
day_change_pct: row.day_change_pct,
confidence: row.confidence,
source_count: row.source_count,
spread_pp: row.spread_pp,
sources: JSON.parse(row.sources || "[]"),
price_date: row.price_date ?? undefined,
ts: row.ts,
};
}
export function readCachedAggregate(ticker: string): AggregateQuote | null {
const row = db()
.prepare("SELECT * FROM quote_aggregates WHERE ticker = ? ORDER BY ts DESC LIMIT 1")
.get(ticker.toUpperCase()) as any;
return row ? rowToAggregate(row) : null;
}
// Distinct tickers currently tracked in the cache — the universe the
// hourly job refreshes when called with no explicit list.
export function trackedTickers(): string[] {
return (db().prepare("SELECT DISTINCT ticker FROM quote_aggregates").all() as any[]).map(
(r) => r.ticker,
);
}
// ── serving + ingest ────────────────────────────────────────────────
// Serve aggregates for a request. Cache-first; tickers never seen are
// aggregated live ONCE and cached (so the page never blocks on a cold
// symbol), then flagged stale if the cached row is older than the hourly
// window. Refreshing warm tickers is the ingest job's responsibility, not
// the request path — keeps serving fast.
export async function serveQuotes(
tickers: string[],
): Promise<Record<string, (AggregateQuote & { stale: boolean }) | null>> {
const out: Record<string, (AggregateQuote & { stale: boolean }) | null> = {};
const cold: string[] = [];
for (const t of tickers) {
const c = readCachedAggregate(t);
if (c) out[t] = { ...c, stale: Date.now() - c.ts > STALE_AFTER_MS };
else cold.push(t);
}
if (cold.length) {
const fresh = await Promise.all(cold.map((t) => aggregateTicker(t)));
for (let i = 0; i < cold.length; i++) {
const a = fresh[i];
if (a) {
writeAggregate(a);
out[cold[i]] = { ...a, stale: false };
} else {
out[cold[i]] = null;
}
}
}
return out;
}
// The hourly ingest. Aggregates each ticker from all sources and writes
// the snapshot. With no list, refreshes everything already tracked.
export async function ingestQuotes(
tickers?: string[],
): Promise<{ requested: number; written: number; failed: string[] }> {
const list = (tickers?.length ? tickers : trackedTickers()).map((t) => t.toUpperCase());
const uniq = Array.from(new Set(list)).filter((t) => t && !t.includes("."));
let written = 0;
const failed: string[] = [];
// Serial fan-out to stay polite to the free endpoints. One retry on a
// total miss: all three fetches share a call, so a transient blip drops
// them together (the most liquid ticker fails as easily as a thin one).
for (const t of uniq) {
let a = await aggregateTicker(t);
if (!a) {
await new Promise((r) => setTimeout(r, 600));
a = await aggregateTicker(t);
}
if (a) {
writeAggregate(a);
written++;
} else {
failed.push(t);
}
}
return { requested: uniq.length, written, failed };
}
+115 -212
View File
@@ -135,6 +135,26 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
.ft{text-align:center;padding:24px;color:#3d444d;font-size:11px;border-top:1px solid #171d27;margin-top:32px}
.ft a{color:#545d68;text-decoration:none;transition:color 0.15s}
.ft a:hover{color:#e6edf3}
/* ── UX pass 2026-05-30: clearer interactive affordances ── */
[onclick]{cursor:pointer}
[role="button"]{cursor:pointer}
.iworker{cursor:pointer}
.iworker:hover{background:#1c2333;transform:translateY(-1px);box-shadow:0 2px 8px rgba(0,0,0,.3),0 0 0 1px #30363d;transition:background .15s,transform .15s,box-shadow .15s}
.iworker:active{transform:translateY(0);box-shadow:none}
details>summary{cursor:pointer}
details>summary:hover{color:#e6edf3;background:rgba(58,90,127,.15);border-radius:6px}
a:focus-visible,button:focus-visible,summary:focus-visible,[onclick]:focus-visible{outline:2px solid #388bfd;outline-offset:2px;border-radius:4px}
/* ③ example-search disclosure (2026-05-30) — collapsed by default, CSS-only hover/focus */
.samples-disclosure{font-size:11px}
.samples-summary{display:inline-flex;align-items:center;gap:6px;cursor:pointer;color:#8b949e;list-style:none;padding:4px 0;user-select:none}
.samples-summary::-webkit-details-marker{display:none}
.samples-summary::before{content:'▸';display:inline-block;font-size:10px;transition:transform .15s}
.samples-disclosure[open] .samples-summary::before{transform:rotate(90deg)}
.samples-summary:hover{color:#e6edf3}
.samples-row{display:flex;flex-wrap:wrap;gap:6px;margin-top:8px}
.search-chip{font-size:11px;padding:5px 10px;background:#161b22;border:1px solid #30363d;border-radius:14px;color:#8b949e;cursor:pointer;transition:border-color .15s,color .15s,background .15s}
.search-chip:hover{border-color:#58a6ff;color:#e6edf3;background:#1c2333}
.search-chip:focus-visible{outline:2px solid #388bfd;outline-offset:2px}
.ld{color:#3d444d;text-align:center;padding:40px;font-size:13px}
/* Reused pulse keyframe — shared by status dots in the market-pulse
@@ -336,6 +356,18 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
line-height: 1;
}
.contract-card.expanded .card-toggle .chevron { transform: rotate(180deg) }
/* Persistent top-of-card expand affordance (2026-05-30 UX) — always
visible, button-styled so users immediately see the card expands. */
.contract-card .card-top-toggle {
display: inline-flex; align-items: center; gap: 6px;
padding: 4px 10px; margin-bottom: 10px;
font-size: 11px; font-weight: 600; color: #8b949e;
background: #161b22; border: 1px solid #30363d; border-radius: 7px;
cursor: pointer; user-select: none;
transition: background .15s, border-color .15s, color .15s;
}
.contract-card .card-top-toggle:hover { background: #1c2333; border-color: #58a6ff; color: #e6edf3 }
.contract-card .card-top-toggle .chevron { display:inline-block; font-size:12px; line-height:1 }
@media (prefers-reduced-motion: reduce) {
.contract-card { transition: none }
.contract-card:hover { transform: none }
@@ -591,12 +623,12 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<nav>
<a href="." class="active">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="profiler">Profiler</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
<a href="alerts">Alerts</a>
<a href="workspaces">Workspaces</a>
<a href="profiler" target="_blank" rel="noopener">Profiler ↗</a>
<a href="proof" target="_blank" rel="noopener">Architecture ↗</a>
<a href="spec" target="_blank" rel="noopener">Spec ↗</a>
<a href="onboard" target="_blank" rel="noopener">Onboard ↗</a>
<a href="alerts" target="_blank" rel="noopener">Alerts ↗</a>
<a href="workspaces" target="_blank" rel="noopener">Workspaces ↗</a>
</nav>
<div class="rt" id="status">Loading...</div>
</div>
@@ -711,10 +743,15 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<div class="section" id="market-section">
<div class="section-header">
<span class="section-title">Market Intelligence</span>
<span class="section-meta">Public permit data · Updated live</span>
<span class="section-title">Market Intelligence — Chicago permit map</span>
<span class="section-meta">Where the construction money is · live from data.cityofchicago.org</span>
</div>
<div id="market"></div>
<p style="color:#8b949e;font-size:11px;margin:0 0 10px;line-height:1.5;max-width:680px">
The geographic companion to ① Live Market: every active building permit plotted
by location, sized by construction value. ① tells you <em>when</em> the shifts are;
this tells you <em>where</em> the work is.
</p>
<div id="market"><div style="color:#8b949e;font-size:11px;padding:8px 0">Loading Chicago permit map…</div></div>
</div>
<!-- ═══ ③ Worker Search — natural-language lookup over the 500K bench ═══ -->
@@ -747,12 +784,18 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
</div>
<!-- ═══ ④ System Activity — what the substrate has learned to do ═══ -->
<div class="section" id="learning-section">
<div class="section-header">
<!-- 2026-05-30: demoted to a collapsed-by-default <details> (tier 3). It's
capability/reference info, not the day's staffing work, so it no longer
competes for attention on every load. Mirrors the ⓪ legacy-bridge pattern:
ontoggle flips the ▸/▾ hint. loadLearning() still populates #learning. -->
<details class="section" id="learning-section"
ontoggle="var s=this.querySelector('.lr-toggle-hint');if(s)s.textContent=this.open?'▾ click to collapse':'▸ click to expand'">
<summary style="list-style:none;cursor:pointer;display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;outline:none">
<span class="section-title" style="color:#e6edf3;font-size:13px">④ System Activity — what the substrate has learned to do</span>
<span class="section-meta">Live capabilities · pulled from each subsystem on load</span>
</div>
<p style="color:#8b949e;font-size:11px;margin:0 0 12px;line-height:1.5;max-width:780px">
<span class="lr-toggle-hint" style="margin-left:auto;font-size:10px;color:#545d68">▸ click to expand</span>
</summary>
<p style="color:#8b949e;font-size:11px;margin:10px 0 12px;line-height:1.5;max-width:780px">
Each tile is a capability the system has acquired and the live metric proving
it's running. Operational learning (fills, playbooks, hot-swaps) compounds
inside each capability; what changes here is the <em>set of things the
@@ -760,25 +803,14 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
capability replicates by config, not code.
</p>
<div id="learning"></div>
</div>
</details>
<!-- ═══ ⑤ Substrate signals — compact architecture-health strip (bottom) ═══ -->
<div class="section" id="arch-signals-section">
<div class="section-header">
<span class="section-title" style="color:#e6edf3;font-size:13px">⑤ Substrate Signals — architecture health</span>
<span class="section-meta">Live probes of the index, memory, and pathway layers</span>
</div>
<p style="color:#8b949e;font-size:11px;margin:0 0 10px;line-height:1.5;max-width:680px">
These tiles measure the architecture itself, not the staffing workload.
Instant-search latency, index shape, playbook-memory depth, pathway-matrix
compounding — four probes that answer "is the substrate healthy right now?"
</p>
<div id="arch-signals" style="display:grid;grid-template-columns:repeat(auto-fit,minmax(200px,1fr));gap:10px">
<div style="color:#8b949e;font-size:11px;grid-column:1/-1">Probing substrate…</div>
</div>
</div>
<!-- ⑤ Substrate Signals (architecture-health probes) moved to the Architecture
page (proof.html, Chapter 11) on 2026-05-30 — they measure the substrate,
not the staffing workload, so they belong with the architecture story and
keep this operator dashboard focused on the day's work. Link in the footer. -->
<div class="ft" id="footer">Staffing Co-Pilot · Hybrid SQL + Vector Search · loading scale… · <a href="console">Console</a> · <a href="proof">Architecture</a></div>
<div class="ft" id="footer">Staffing Co-Pilot · Hybrid SQL + Vector Search · loading scale… · <a href="console">Console</a> · <a href="proof" target="_blank" rel="noopener">Architecture ↗</a></div>
</div>
<!-- ═══ Mobile section-jump dock — visible < 768px only via CSS ═══ -->
@@ -792,7 +824,7 @@ body{font-family:'Inter',-apple-system,system-ui,'Segoe UI',sans-serif;backgroun
<a href="#live-contracts-section" title="Open contracts + candidates">②<span class="dock-label">Jobs</span></a>
<a href="#worker-search-section" title="Search 500K workers">③<span class="dock-label">Search</span></a>
<a href="#learning-section" title="What the substrate has learned">④<span class="dock-label">Learn</span></a>
<a href="#arch-signals-section" title="Live architecture health">⑤<span class="dock-label">Stack</span></a>
<a href="proof" title="Architecture &amp; live substrate signals" target="_blank" rel="noopener">✦<span class="dock-label">Arch</span></a>
</nav>
<script>
var P=location.pathname.indexOf('/lakehouse')>=0?'/lakehouse':'';
@@ -826,7 +858,7 @@ function styleSkelStages(){
}
window.addEventListener('load',function(){
styleSkelStages(); startSkelStages();
loadSystemSummary();loadLegacyBridge();loadDay();loadStaffingForecast();loadLiveContracts();loadMarket();loadLearning();loadWorkerSearchSamples();loadArchSignals();loadStaffers();
loadSystemSummary();loadLegacyBridge();loadDay();loadStaffingForecast();loadLiveContracts();loadMarket();loadLearning();loadWorkerSearchSamples();loadStaffers();
});
// Per-staffer hot-swap dropdown — runs independently of the simulation
@@ -989,6 +1021,15 @@ function loadWorkerSearchSamples(){
{tag:'Specialist', q:'maintenance tech comfortable with Allen-Bradley PLCs'},
{tag:'High-volume', q:'warehouse associates available next week near Indianapolis'}
];
// 2026-05-30: wrap the 6 example chips in a collapsed disclosure so they
// don't consume vertical space on every load. The chip keeps its tag+query
// two-span layout and scroll-into-view click; the disclosure just gates them.
el.textContent='';
var det=document.createElement('details');det.className='samples-disclosure';
var sum=document.createElement('summary');sum.className='samples-summary';
sum.textContent='Example searches';
det.appendChild(sum);
var row=document.createElement('div');row.className='samples-row';
samples.forEach(function(s){
var chip=document.createElement('button');
chip.type='button';
@@ -1002,8 +1043,10 @@ function loadWorkerSearchSamples(){
var input=document.getElementById('sq');
if(input){input.value=s.q;doSearch();input.scrollIntoView({behavior:'smooth',block:'center'});}
};
el.appendChild(chip);
row.appendChild(chip);
});
det.appendChild(row);
el.appendChild(det);
}
function loadSystemSummary(){
@@ -1023,7 +1066,7 @@ function loadSystemSummary(){
foot.appendChild(document.createTextNode('Staffing Co-Pilot · Hybrid SQL + Vector Search · '+totalRows+' rows across '+ds+' datasets · '+chunks+' vector chunks · '));
var a1=document.createElement('a');a1.href='console';a1.textContent='Console';foot.appendChild(a1);
foot.appendChild(document.createTextNode(' · '));
var a2=document.createElement('a');a2.href='proof';a2.textContent='Architecture';foot.appendChild(a2);
var a2=document.createElement('a');a2.href='proof';a2.target='_blank';a2.rel='noopener';a2.textContent='Architecture ↗';foot.appendChild(a2);
}
// Also update the collapsible search box label if not yet populated
var sum=document.querySelector('.sa summary');
@@ -1033,56 +1076,9 @@ function loadSystemSummary(){
}).catch(function(){/* non-fatal */});
}
// ─── Substrate signals: render the 4 architecture-health tiles ───
function loadArchSignals(){
var el=document.getElementById('arch-signals');
api('/intelligence/arch_signals',{}).then(function(s){
el.textContent='';
function tile(label, big, sub, accent){
var t=document.createElement('div');
t.style.cssText='background:#0d1117;border:1px solid #171d27;border-radius:10px;padding:14px 16px;border-left:3px solid '+(accent||'#58a6ff');
var l=document.createElement('div');l.style.cssText='font-size:10px;text-transform:uppercase;letter-spacing:1px;color:#545d68;margin-bottom:6px';l.textContent=label;
var b=document.createElement('div');b.style.cssText='font-size:20px;font-weight:600;color:#e6edf3;line-height:1.1';b.textContent=big;
var u=document.createElement('div');u.style.cssText='font-size:10px;color:#8b949e;margin-top:6px;line-height:1.4';u.textContent=sub;
t.appendChild(l);t.appendChild(b);t.appendChild(u);
return t;
}
var idx=s.index||{};
var pbm=s.playbook_memory||{};
var pwm=s.pathway_memory||{};
// Tile 1 — instant search (the "we cleverly indexed at ingest" claim)
var latencyColor=s.instant_search_probe_ms<100?'#2ea043':s.instant_search_probe_ms<500?'#d29922':'#f85149';
el.appendChild(tile(
'Instant Search',
(s.instant_search_probe_ms||'?')+'ms',
'Live /vectors/hybrid probe · 500K-row index · '+(idx.chunk_count||0).toLocaleString()+' chunks',
latencyColor
));
// Tile 2 — index shape (hot-swap claim)
el.appendChild(tile(
'Index',
(idx.dimensions||768)+'d · '+(idx.model||'?'),
(idx.source||'?')+' → '+(idx.name||'?')+' · '+(idx.backend||'parquet'),
'#58a6ff'
));
// Tile 3 — self-regulating memory
el.appendChild(tile(
'Playbook Memory',
(pbm.entries||0).toLocaleString()+' entries',
pbm.entries>0?'Meta-index active · boosts candidates from past fills':'Empty · POST /vectors/playbook_memory/rebuild to populate',
pbm.entries>0?'#2ea043':'#d29922'
));
// Tile 4 — ADR-021 pathway compounding
el.appendChild(tile(
'Pathway Matrix',
(pwm.total_pathways||0)+' traces',
pwm.retired+' retired · '+pwm.total_replays+' replays · ADR-021 compounding',
'#58a6ff'
));
}).catch(function(e){
el.textContent='substrate signals unavailable: '+e.message;
});
}
// ⑤ loadArchSignals() was MOVED to proof.html (Architecture, Chapter 11)
// on 2026-05-30 — the architecture-health probes live with the rest of the
// architecture story now, so the dashboard stays focused on staffing work.
// ─── Live Market hero: 24/7 shift clock (left) + Chicago permit pulse (right) ──
// Combines two previously-separate tiles into one coherent "what's happening in
@@ -2534,6 +2530,15 @@ function loadLiveContracts(){
right.appendChild(rate);
}
hdr.appendChild(left);hdr.appendChild(right);card.appendChild(hdr);
// Persistent expand affordance at the TOP of the card (2026-05-30
// UX). The whole card is already clickable; this makes the toggle
// obvious and shows expand state in BOTH collapsed and expanded
// form. Text + chevron are flipped by the card click handler below.
var topToggle=document.createElement('div');topToggle.className='card-top-toggle';
var topChev=document.createElement('span');topChev.className='chevron';topChev.textContent='▸';
var topLabel=document.createElement('span');topLabel.textContent='Show details';
topToggle.appendChild(topChev);topToggle.appendChild(topLabel);
card.appendChild(topToggle);
// Architecture pill row — instant-search latency + shift coverage
// + pool-size proof that the index actually fired on this call.
// This is the "our substrate is better" surface J asked for.
@@ -2853,6 +2858,8 @@ function loadLiveContracts(){
toggleLabel.textContent=nowExpanded
? 'click to collapse'
: 'click for full curve, economics, candidates, project index';
topLabel.textContent=nowExpanded?'Hide details':'Show details';
topChev.textContent=nowExpanded?'▾':'▸';
// Auto-open the Project Index ONLY on user-driven expand. This
// avoids firing 20+ OSHA scrapes on page load — each open
// triggers /intelligence/permit_entities, which live-scrapes
@@ -3369,134 +3376,30 @@ function addBigMeter(parent,label,val,desc){
d.appendChild(lb);d.appendChild(row);d.appendChild(ds);parent.appendChild(d);
}
// First-name → gender hint for face-pool selection. Built from the
// most common 200 US given names. Lookup is best-effort — undefined
// returns means "no hint, draw from full pool." Don't surface this
// anywhere user-visible; it's purely a face-pool selector.
var FEMALE_NAMES = new Set(['Mary','Patricia','Jennifer','Linda','Elizabeth','Barbara','Susan','Jessica','Sarah','Karen','Lisa','Nancy','Betty','Sandra','Margaret','Ashley','Kimberly','Emily','Donna','Michelle','Carol','Amanda','Melissa','Deborah','Stephanie','Dorothy','Rebecca','Sharon','Laura','Cynthia','Amy','Kathleen','Angela','Shirley','Brenda','Emma','Anna','Pamela','Nicole','Samantha','Katherine','Christine','Helen','Debra','Rachel','Carolyn','Janet','Maria','Catherine','Heather','Diane','Olivia','Julie','Joyce','Victoria','Ruth','Virginia','Lauren','Kelly','Christina','Joan','Evelyn','Judith','Andrea','Hannah','Megan','Cheryl','Jacqueline','Martha','Madison','Teresa','Gloria','Sara','Janice','Ann','Kathryn','Abigail','Sophia','Frances','Jean','Alice','Judy','Isabella','Julia','Grace','Amber','Denise','Danielle','Marilyn','Beverly','Charlotte','Natalie','Theresa','Diana','Brittany','Doris','Kayla','Alexis','Lori','Marie','Carmen','Aisha','Rosa','Kim','Mia','Audrey','Erin','Tina','Vanessa','Tara','Wendy','Tanya','Maya','Crystal','Yvonne','Kara','Shannon','Brianna','Faith','Caroline','Carla','Tracey','Tracy','Rita','Dawn','Tiffany','Stacy','Stacey','Gina','Bonnie','Tammy','Joanne','Jamie','Tonya','Alyssa','Ariana','Elena','Ellie','Erica','Erika','Felicia','Holly','Jenna','Jenny','Krista','Kristen','Kristin','Krystal','Lana','Leah','Lucy','Mallory','Melinda','Meredith','Misty','Monica','Mya','Naomi','Paige','Patrice','Paula','Renee','Rhonda','Robin','Roxanne','Sadie','Selena','Shari','Shauna','Sierra','Skylar','Sonia','Stella','Tamara','Taryn','Trina','Veronica','Vivian','Whitney','Yolanda','Zoe','Lakeisha','Latoya','Tasha',
// Hispanic female
'Esperanza','Luz','Lucia','Camila','Valentina','Mariana','Catalina','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Consuelo','Dolores','Mercedes','Marisol','Guadalupe','Lupita','Inez','Itzel','Yesenia','Monserrat','Renata','Alejandra','Alma','Belen','Blanca','Esmeralda','Imelda','Lourdes','Magdalena','Olga','Refugio','Rocio','Susana','Anita','Fatima',
// South Asian female
'Priya','Anjali','Neha','Kavya','Pooja','Divya','Meera','Lakshmi','Rani','Asha','Saanvi','Aanya','Aaradhya','Shreya','Riya','Tanvi','Ishita','Shivani','Padma','Sita','Geeta','Rekha','Amira',
// East Asian female
'Mei','Sakura','Aiko','Sora','Chiyo','Hana','Eun','Xiu','Lan','Hua','Min','Xin','Ying','Zhen','Yan',
// Black female (additional)
'Imani','Keisha','Lakisha','Kenya','Tamika','Latanya','Latrice','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Maliyah','Imari','Nia','Zuri','Talia','Jada','Ebony','Dominique',
// Middle Eastern female
'Layla','Yasmin','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Dunia','Iman','Lina','Mona','Noor','Rana','Sabrina','Soha','Zara'
]);
var MALE_NAMES = new Set(['James','Robert','John','Michael','David','William','Richard','Joseph','Thomas','Charles','Christopher','Daniel','Matthew','Anthony','Mark','Donald','Steven','Paul','Andrew','Joshua','Kenneth','Kevin','Brian','George','Edward','Ronald','Timothy','Jason','Jeffrey','Ryan','Jacob','Gary','Nicholas','Eric','Jonathan','Stephen','Larry','Justin','Scott','Brandon','Benjamin','Samuel','Gregory','Frank','Alexander','Raymond','Patrick','Jack','Dennis','Jerry','Tyler','Aaron','Jose','Adam','Henry','Nathan','Douglas','Zachary','Peter','Kyle','Walter','Ethan','Jeremy','Harold','Keith','Christian','Roger','Noah','Gerald','Carl','Terry','Sean','Austin','Arthur','Lawrence','Jesse','Dylan','Bryan','Joe','Jordan','Billy','Bruce','Albert','Willie','Gabriel','Logan','Alan','Juan','Wayne','Roy','Ralph','Randy','Eugene','Vincent','Russell','Elijah','Louis','Bobby','Philip','Johnny','Marcus','Antonio','Carlos','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Victor','Jamal','Xavier','DeShawn','Dwayne','Jermaine','Malik','Tyrone','Devon','Andre','Anwar','Brent','Calvin','Casey','Cody','Cole','Cory','Curt','Dale','Damon','Darius','Darrell','Dean','Derek','Donnie','Drew','Earl','Eddie','Floyd','Glenn','Greg','Howard','Ivan','Jared','Jay','Jeff','Joel','Johnnie','Lance','Lee','Leonard','Lloyd','Mario','Martin','Mason','Maurice','Max','Mitchell','Morgan','Nick','Norman','Oliver','Owen','Pete','Quincy','Rafael','Reggie','Rex','Ricky','Rod','Russ','Salvatore','Shane','Shaun','Stanley','Steve','Theodore','Todd','Travis','Trevor','Troy','Wade','Warren','Wesley',
// Hispanic male
'Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Joaquin','Ignacio','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Felipe','Gerardo','Jaime','Leonardo','Luis','Pablo','Ramon','Reynaldo','Vincente','Javier','Esteban','Eduardo','Fernando','Humberto','Ernestino','Cristian','Hernan',
// South Asian male
'Raj','Anil','Rohan','Vikram','Arjun','Sanjay','Ravi','Krishna','Pradeep','Sunil','Amit','Deepak','Ashok','Manoj','Rahul','Vijay','Suresh','Naveen','Anand','Nikhil','Aditya','Karan','Rajesh','Ramesh','Kishore','Mohan','Ajay','Aarav','Ishaan',
// East Asian male
'Wei','Yi','Jin','Hiroshi','Akira','Kenji','Haruto','Hyun','Yoon','Kai','Long','Hong','Hao','Tao','Bao','Cheng','Feng','Qiang','Jian','Dong','Bin','Lei','Hui','Yu','Yuan',
// Black male (additional)
'Demetrius','Tyrese','Trevon','Kareem','DaQuan','Tyrell','Kwame','Khalil','Rashid','Terrell','Chauncey','Cedric','Imari','Jalen','Jaylen',
// Middle Eastern male
'Omar','Khalid','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Ibrahim','Mahmoud','Saif','Bilal','Faisal','Hamza','Imran','Sami','Wael','Yasin','Zaid'
]);
function guessGenderFromFirstName(name){
if(!name) return null;
var clean = name.replace(/[^A-Za-z]/g,'');
if(!clean) return null;
// Title-case
var c = clean[0].toUpperCase() + clean.slice(1).toLowerCase();
if(FEMALE_NAMES.has(c)) return 'woman';
if(MALE_NAMES.has(c)) return 'man';
return null;
}
// First-name → ethnicity hint. The synthetic-data generator built
// workers_500k from a multi-cultural name pool (Raj, DeShawn, Jamal,
// Mei, Wei, Carmen, Esperanza, Aisha, etc.). The hint is used ONLY
// to bias face-pool selection toward visually-aligned StyleGAN faces;
// it never surfaces in any worker-facing label or report. Buckets
// match common deepface "race" categories so the pool tags align
// when deepface is later run over the headshot pool.
var NAMES_SOUTH_ASIAN = new Set(['Raj','Anil','Rohan','Vikram','Arjun','Sanjay','Ravi','Krishna','Pradeep','Sunil','Amit','Deepak','Ashok','Manoj','Rahul','Vijay','Suresh','Naveen','Anand','Nikhil','Aditya','Karan','Rajesh','Ramesh','Kishore','Mohan','Ajay','Priya','Anjali','Neha','Kavya','Pooja','Divya','Meera','Lakshmi','Rani','Asha','Saanvi','Aanya','Aaradhya','Shreya','Riya','Tanvi','Ishita','Aarav','Ishaan','Shivani','Padma','Sita','Geeta','Rekha']);
var NAMES_EAST_ASIAN = new Set(['Wei','Mei','Yi','Jin','Chen','Lin','Liu','Wang','Zhang','Yang','Wu','Zhao','Sun','Hiroshi','Yuki','Akira','Kenji','Sakura','Aiko','Haruto','Sora','Chiyo','Hana','Hyun','Eun','Yoon','Kai','Long','Hong','Xiu','Lan','Hua','Hao','Tao','Bao','Cheng','Feng','Qiang','Jian','Dong','Bin','Min','Lei','Hui','Yu','Xin','Ying','Zhen','Yuan','Yan']);
var NAMES_HISPANIC = new Set(['Carmen','Carlos','Maria','Diego','Hector','Jorge','Julio','Manuel','Miguel','Pedro','Raul','Ricardo','Roberto','Sergio','Antonio','Esperanza','Luz','Sofia','Lucia','Isabella','Camila','Valentina','Mariana','Elena','Rosa','Catalina','Esteban','Fernando','Eduardo','Javier','Alejandro','Andres','Mateo','Santiago','Sebastian','Emilio','Tomas','Cristina','Daniela','Gabriela','Ximena','Adriana','Beatriz','Pilar','Consuelo','Dolores','Mercedes','Xavier','Marisol','Guadalupe','Lupita','Inez','Itzel','Yolanda','Yesenia','Monserrat','Renata','Ximena','Joaquin','Ignacio','Rafael','Salvador','Cesar','Arturo','Armando','Hugo','Marco','Alejandra','Alma','Belen','Blanca','Esmeralda','Fatima','Gloria','Imelda','Lourdes','Magdalena','Olga','Paula','Refugio','Rocio','Susana','Teresa','Veronica','Anita','Ernestino','Felipe','Gerardo','Humberto','Jaime','Leonardo','Luis','Pablo','Ramon','Reynaldo','Vincente']);
var NAMES_BLACK = new Set(['DeShawn','Jamal','Aisha','Latoya','Tyrone','Malik','Imani','Keisha','Tariq','Lakisha','Kenya','Tamika','Shaquille','Andre','Marcus','Demetrius','Jermaine','Reggie','Tyrese','Darius','Trevon','Kareem','Damon','Jalen','Jaylen','Dwayne','DaQuan','Latanya','Latrice','Aaliyah','Kiara','Janelle','Jasmine','Tanisha','Yolanda','Maurice','Tyrell','Kwame','Khalil','Rashid','Terrell','Chauncey','Cedric','Maliyah','Imari','Nia','Zuri','Talia','Jada','Ebony','Dominique']);
var NAMES_MIDDLE_EASTERN = new Set(['Layla','Omar','Khalid','Fatima','Yasmin','Hassan','Hussein','Ahmed','Mohamed','Mohammed','Ali','Karim','Yusuf','Yara','Nadia','Zainab','Rania','Samira','Mariam','Salma','Ibrahim','Mahmoud','Saif','Anwar','Bilal','Faisal','Hamza','Imran','Jamal','Rashid','Sami','Tariq','Wael','Yasin','Zaid','Amira','Dunia','Iman','Lina','Mona','Noor','Rana','Sabrina','Soha','Yara','Zara']);
// Surname dictionaries — surname is more diagnostic than first name
// for hispanic and asian. Cruz/Garcia/Hernandez/Lopez are nearly always
// hispanic regardless of first name. Patel/Singh/Kumar are South Asian.
// Chen/Wang/Liu/Nguyen are East Asian. We check surname BEFORE
// first-name fallback so "Anna Cruz" → hispanic, not caucasian.
var SURNAMES_HISPANIC = new Set(['Garcia','Rodriguez','Martinez','Hernandez','Lopez','Gonzalez','Perez','Sanchez','Ramirez','Torres','Flores','Rivera','Gomez','Diaz','Reyes','Cruz','Morales','Ortiz','Gutierrez','Chavez','Ramos','Ruiz','Alvarez','Mendoza','Vasquez','Castillo','Jimenez','Moreno','Romero','Herrera','Medina','Aguilar','Vargas','Castro','Fernandez','Guzman','Munoz','Salazar','Ortega','Delgado','Estrada','Ayala','Pena','Cabrera','Alvarado','Espinoza','Padilla','Cardenas','Cortes','Cortez','Ibarra','Vega','Soto','Lara','Navarro','Campos','Acosta','Rios','Marquez','Velasquez','Velazquez','Sandoval','Maldonado','Solis','Rojas','Pacheco','Mejia','Beltran','Santiago','Cervantes','Lozano','Carrillo','Galvan','Trevino','Galvez','Santana','Galvan','Robles','Valencia','Carrasco','Tapia','Lugo','Barajas','Bautista','Quintero','Salinas','Avila','Macias','Velasco','Gallegos','Calderon']);
var SURNAMES_SOUTH_ASIAN = new Set(['Patel','Singh','Kumar','Sharma','Gupta','Shah','Khan','Mehta','Desai','Joshi','Reddy','Nair','Iyer','Verma','Agarwal','Kapoor','Chopra','Malhotra','Bhatt','Bhattacharya','Banerjee','Chatterjee','Mukherjee','Das','Sen','Bose','Roy','Saha','Sinha','Trivedi','Pandey','Mishra','Tiwari','Yadav','Chauhan','Rana','Thakur','Pillai','Menon','Krishnan','Subramanian','Raman','Rao','Naidu','Nayak','Mohan','Pradhan','Acharya','Devi','Kaur','Khanna']);
var SURNAMES_EAST_ASIAN = new Set(['Chen','Wang','Li','Liu','Yang','Huang','Zhao','Wu','Zhou','Xu','Zhu','Sun','Ma','Lin','Lee','Kim','Park','Choi','Jung','Kang','Cho','Yoon','Han','Lim','Oh','Nakamura','Tanaka','Suzuki','Yamamoto','Sato','Watanabe','Takahashi','Kobayashi','Yoshida','Saito','Nguyen','Tran','Le','Pham','Hoang','Phan','Vu','Vo','Dang','Bui','Do','Ho','Ngo','Duong','Truong','Lam','Trinh','Mai','Cao','Lam','Wong','Tang','Tan','Cheng','Lau','Leung','Ng','Cheung','Yu','Yip','Yam','Lo','Hsu','Tsai','Hsieh','Lin']);
var SURNAMES_MIDDLE_EASTERN = new Set(['Khan','Ahmed','Hussein','Hassan','Ali','Mahmoud','Mohamed','Mohammed','Saleh','Aziz','Karim','Hamad','Najjar','Haddad','Khoury','Mansour','Hakim','Zaman','Rahman','Iqbal','Malik','Sheikh','Siddiqui','Qureshi','Abbasi','Bukhari','Naqvi','Tahir','Anwar','Aslam','Saeed','Rizvi','Farooqi']);
var SURNAMES_BLACK = new Set(['Washington','Jefferson','Booker','Tubman','Robinson','Jackson','Williams','Brown','Davis','Smith','Johnson','Thomas','Lewis','Walker','Wright','Edwards','Carter','Mitchell','Roberts','Phillips','Bell','Coleman','Patterson','Graham','Bailey','Reed','Cook','Morgan','Bryant','Russell','Hayes','Howard','Ward','Foster','Long','Hill','Murphy','Rivers','Banks','Boyd','Glover','Harper','Jenkins','Wallace','Mason','Spencer','Crawford','Greene','Holmes','Stewart','Pierce','Hardy','Sims','Sutton','Wells','Burke','Hines','Hudson','Mosley','Dawson','Mathis','Lyons','Newton','Watts','Whitaker','Wilkerson']);
function guessEthnicityFromName(firstName, lastName){
// Try surname FIRST — it's the stronger signal for hispanic/asian
// identity ("Anna Cruz" → hispanic via surname). Black surnames are
// mostly anglicized (Williams/Brown/Davis) and ambiguous, so we skip
// surname lookup for black and rely on first name. Caucasian is the
// implicit default — never explicitly tagged.
if(lastName){
var s = lastName.replace(/[^A-Za-z]/g,'');
if(s){
var sc = s[0].toUpperCase() + s.slice(1).toLowerCase();
if(SURNAMES_HISPANIC.has(sc)) return 'hispanic';
if(SURNAMES_MIDDLE_EASTERN.has(sc)) return 'middle_eastern';
if(SURNAMES_SOUTH_ASIAN.has(sc)) return 'south_asian';
if(SURNAMES_EAST_ASIAN.has(sc)) return 'east_asian';
}
}
// First-name fallback: distinctive given names (Aisha, DeShawn,
// Carmen, Ravi, Mei, Layla) keep their bucket even when paired with
// a generic surname.
if(firstName){
var clean = firstName.replace(/[^A-Za-z]/g,'');
if(clean){
var c = clean[0].toUpperCase() + clean.slice(1).toLowerCase();
if(NAMES_MIDDLE_EASTERN.has(c)) return 'middle_eastern';
if(NAMES_BLACK.has(c)) return 'black';
if(NAMES_HISPANIC.has(c)) return 'hispanic';
if(NAMES_SOUTH_ASIAN.has(c)) return 'south_asian';
if(NAMES_EAST_ASIAN.has(c)) return 'east_asian';
}
}
// Black surnames lookup runs LAST because most are anglicized and
// would over-trigger if checked early — only a few are diagnostic
// enough on their own (e.g., when first name is also generic).
if(lastName){
var s2 = lastName.replace(/[^A-Za-z]/g,'');
if(s2){
var sc2 = s2[0].toUpperCase() + s2.slice(1).toLowerCase();
if(SURNAMES_BLACK.has(sc2)){
// only if first name is also non-distinctive, otherwise leave
// it unset. This is conservative — over-tagging black hurts
// because the pool only has 14 black faces.
}
}
}
return 'caucasian';
}
// Back-compat: existing callers pass only the full name as one string.
function guessEthnicityFromFirstName(name){
if(!name) return 'caucasian';
var parts = String(name).trim().split(/\s+/);
var first = parts[0] || '';
var last = parts.length > 1 ? parts[parts.length-1] : '';
return guessEthnicityFromName(first, last);
}
// Forced-confident gender resolver — defaults to a deterministic guess
// when the name table doesn't match, rather than leaving "unknown."
// We're authoring the synthetic data; we own the confidence call.
function genderFor(name){
var g = guessGenderFromFirstName(name);
if(g) return g;
if(!name) return 'man';
// hash-based fallback so unknown names still spread roughly 50/50
var s = String(name);
var h = 0; for (var i=0;i<s.length;i++) h = (h*31 + s.charCodeAt(i))|0;
return (Math.abs(h) & 1) ? 'man' : 'woman';
}
// Name → ethnicity / gender inference — REMOVED 2026-05-03 per Phase 1.6
// BIPA gates (docs/PHASE_1_6_BIPA_GATES.md, Gate 4) and the audit-trail
// PRD's protected-attribute exclusion rule (docs/AUDIT_TRAIL_PRD.md §4).
//
// What was here: FEMALE_NAMES / MALE_NAMES / NAMES_HISPANIC / NAMES_BLACK /
// NAMES_SOUTH_ASIAN / NAMES_EAST_ASIAN / NAMES_MIDDLE_EASTERN lookup sets
// plus parallel SURNAMES_* sets, plus guessGenderFromFirstName(),
// guessEthnicityFromName(), guessEthnicityFromFirstName(), and
// genderFor(). Used historically to bias synthetic-face-pool bucketing.
//
// Why removed: hard-coded inferred-attribute classification from candidate
// names is (1) a Title VII / IL Human Rights Act discriminatory feature
// engineering risk, and (2) when combined with deepface-extracted
// classifications (per Phase 1.5 walk), forms the "biometric information
// derived from a biometric identifier" pattern under BIPA's broad reading.
// Removing both forecloses both arguments.
//
// Replacement: face-pool routing for synthetic faces uses deterministic
// hash of candidate_id (in mcp-server/index.ts /headshots/ routes), no
// demographic inference. Real-photo intake requires the consent gate
// (Phase 1.6 Gate 3) and is NOT served by this UI surface.
//
// Do NOT re-introduce. The Phase 1.6 unit test asserts these symbols
// stay absent from this file.
// Role classification — sober, no illustrations. Maps a role string
// to a short uppercase label and a "band" used as a left-edge color
+1
View File
@@ -102,6 +102,7 @@ h2{color:#e6edf3;font-size:22px;font-weight:700;letter-spacing:-0.3px;margin-bot
<nav>
<a href=".">Dashboard</a>
<a href="console">Walkthrough</a>
<a href="profiler">Profiler</a>
<a href="proof">Architecture</a>
<a href="spec">Spec</a>
<a href="onboard">Onboard</a>
+7
View File
@@ -0,0 +1,7 @@
{
"_comment": "consent_versions allowlist for the biometric consent endpoint. Loaded by the gateway at startup from the path in LH_CONSENT_VERSIONS_FILE (default: /etc/lakehouse/consent_versions.json). Hashes are SHA-256 of the rendered consent template text (lowercase hex). Generate from /biometric/intake — the JS computes the hash client-side and you can capture it from a smoke run, OR compute server-side from the markdown file. Counsel-tier deployment SHOULD have at least one entry; absent file = permissive (any non-empty hash accepted, v1 compat).",
"_doc": "docs/PHASE_1_6_BIPA_GATES.md §1 Gate 2",
"versions": [
"REPLACE_WITH_REAL_SHA256_OF_SIGNED_CONSENT_TEMPLATE_v1"
]
}
+36 -4
View File
@@ -19,6 +19,10 @@ TARGET_DIR=/etc/systemd/system
UNITS=(
lakehouse-auditor.service
lakehouse-context7-bridge.service
lakehouse-retention-sweep.service
lakehouse-retention-sweep.timer
lakehouse-quote-ingest.service
lakehouse-quote-ingest.timer
)
if [[ $EUID -ne 0 ]]; then
@@ -41,9 +45,34 @@ echo "→ systemctl daemon-reload"
systemctl daemon-reload
for unit in "${UNITS[@]}"; do
echo "→ enable + (re)start $unit"
systemctl enable "$unit" >/dev/null
systemctl restart "$unit"
# For .timer units: enable + start the timer (which fires its
# paired oneshot service on schedule). For long-running .service
# units that DON'T have a timer: enable + restart so changes
# land. For oneshot .service units that ARE driven by a timer,
# do NOT enable/start them directly — the timer pulls them in.
base="${unit%.*}"
case "$unit" in
*.timer)
echo "→ enable + (re)start $unit"
systemctl enable "$unit" >/dev/null
systemctl restart "$unit"
;;
*.service)
# Skip if a paired .timer exists in this install set.
paired_timer="${base}.timer"
paired_in_set=0
for u2 in "${UNITS[@]}"; do
[[ "$u2" == "$paired_timer" ]] && paired_in_set=1 && break
done
if [[ $paired_in_set -eq 1 ]]; then
echo "→ skip direct start of $unit (driven by $paired_timer)"
else
echo "→ enable + (re)start $unit"
systemctl enable "$unit" >/dev/null
systemctl restart "$unit"
fi
;;
esac
done
echo ""
@@ -51,9 +80,12 @@ echo "─── post-install status ───"
for unit in "${UNITS[@]}"; do
active=$(systemctl is-active "$unit" 2>/dev/null || true)
enabled=$(systemctl is-enabled "$unit" 2>/dev/null || true)
printf " %-40s active=%s enabled=%s\n" "$unit" "$active" "$enabled"
printf " %-44s active=%s enabled=%s\n" "$unit" "$active" "$enabled"
done
echo ""
echo "Live logs: journalctl -u lakehouse-auditor.service -f"
echo " journalctl -u lakehouse-retention-sweep.service -f"
echo "Pause: touch /home/profit/lakehouse/auditor.paused"
echo "Resume: rm /home/profit/lakehouse/auditor.paused"
echo "Sweep test: systemctl start lakehouse-retention-sweep.service # one-shot, completes immediately"
echo "Next sweep: systemctl list-timers lakehouse-retention-sweep.timer"
@@ -0,0 +1,17 @@
[Unit]
Description=Lakehouse quote ingest — multi-source consensus snapshot into the local SQLite cache (Yahoo+Nasdaq+CNBC)
After=network-online.target lakehouse-agent.service
Wants=network-online.target
[Service]
# Oneshot driven by lakehouse-quote-ingest.timer. Runs as root to match
# the lakehouse-agent (mcp-server) service that owns the cache db — both
# write data/_quote_cache/quotes.db, so they must share a user (see
# README "Why both services run as root"). busy_timeout in quotes.ts
# serializes the concurrent writes.
Type=oneshot
WorkingDirectory=/home/profit/lakehouse/mcp-server
ExecStart=/home/profit/.bun/bin/bun run /home/profit/lakehouse/mcp-server/quote_ingest.ts
# No args → refreshes every ticker already tracked in the cache. Cold
# tickers (new basket entries) warm on-demand via the serving path.
TimeoutStartSec=300
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Hourly trigger for Lakehouse multi-source quote ingest
[Timer]
OnCalendar=hourly
Persistent=true
# Spread the wake so we don't hit the free quote endpoints on the dot.
RandomizedDelaySec=120
[Install]
WantedBy=timers.target
@@ -0,0 +1,33 @@
[Unit]
Description=Lakehouse retention sweep — flag biometric + general PII subjects past their retention_until clock (BIPA + general retention compliance)
Documentation=file:///home/profit/lakehouse/docs/PHASE_1_6_BIPA_GATES.md
After=lakehouse.service
[Service]
Type=oneshot
User=root
Group=root
WorkingDirectory=/home/profit/lakehouse
# Use the release binary built alongside catalogd. retention_sweep
# deliberately does NOT auto-mutate state — it writes a date-stamped
# JSONL report to data/_catalog/subjects/_retention_sweep_<date>.jsonl
# (see crates/catalogd/src/bin/retention_sweep.rs). Operators run
# actual destruction via the destruction runbook + verify scripts.
#
# Without --apply the sweep is dry-run only; we want the persisted
# report so the audit trail captures every daily check.
ExecStart=/home/profit/lakehouse/target/release/retention_sweep \
--storage-root /home/profit/lakehouse/data \
--apply
StandardOutput=append:/var/log/lakehouse/retention_sweep.log
StandardError=append:/var/log/lakehouse/retention_sweep.log
# Bound the runtime — sweep over 100k subjects should complete in
# seconds; if it ever hits this we want a hard stop, not a runaway.
TimeoutStartSec=10min
# Guardrails. retention_sweep is read-mostly + writes one JSONL.
CPUQuota=100%
MemoryMax=2G
Nice=15
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,18 @@
[Unit]
Description=Daily trigger for lakehouse-retention-sweep.service
Documentation=file:///home/profit/lakehouse/docs/PHASE_1_6_BIPA_GATES.md
[Timer]
# Daily at 03:00 UTC. The sweep is idempotent across runs (it produces
# a per-day JSONL report and does NOT mutate manifest state), so it's
# safe to overlap with operator activity.
OnCalendar=*-*-* 03:00:00 UTC
# Persistent: if the box was off at 03:00, run on next boot. Without
# this we'd silently miss days, which would silently let retention
# expire without flagging.
Persistent=true
AccuracySec=1min
Unit=lakehouse-retention-sweep.service
[Install]
WantedBy=timers.target
+27
View File
@@ -0,0 +1,27 @@
# Pinned toolchain — this workspace only builds inside a NARROW rustc band.
#
# Established empirically 2026-07-28 in a clean Debian 13 container:
#
# rustc 1.85.0 (what `apt install rustc` gives on Debian trixie)
# -> FAILS: "rustc 1.85.0 is not supported by the following packages"
# aws-config, aws-sdk-*, aws-smithy-* all require >= 1.91.1
#
# rustc 1.97.1 (what `rustup default stable` gives today)
# -> FAILS: ethnum, error[E0512] "cannot transmute between types of
# different sizes, or dependently-sized types"
#
# rustc 1.94.0 -> builds clean (2m39s, workspace, exit 0)
#
# So BOTH obvious reviewer paths break: distro rustc is too old, rustup stable
# is too new. Without this pin the build reproduced on our host only because the
# toolchain happened to be restored from the previous machine. Pinning makes it
# reproducible by construction rather than by accident.
#
# Native prerequisites (also verified in the clean container, minimal set):
# protobuf-compiler -- lance-encoding / lance-file build scripts need protoc
# pkg-config
# libssl-dev
# NOT required, despite being installed on the dev host: cmake, clang, libclang-dev.
[toolchain]
channel = "1.94.0"
profile = "minimal"
-77
View File
@@ -1,77 +0,0 @@
#!/usr/bin/env bash
# A/B test of T3 overseer: does it actually make subsequent runs better?
# Chains Run B (T3 seed) → Run C (T3 + read-back) → Run D (T3 cloud).
# Run A is assumed already complete (launched separately). Aggregates
# metrics at the end into ab_scorecard.json.
set -e
cd "$(dirname "$0")/.."
export OLLAMA_CLOUD_KEY="$(python3 -c "import json; print(json.load(open('/root/llm_team_config.json'))['providers']['ollama_cloud']['api_key'])")"
echo "▶ A/B test start at $(date -Iseconds)"
echo "▶ prior lessons dir: $(ls data/_playbook_lessons 2>/dev/null | wc -l) files"
# Run B — T3 enabled local, no prior lessons should exist yet
echo "──── RUN B: T3 local, seeds first lesson ────"
bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_B.log 2>&1 || true
echo " B exit=$?"
ls data/_playbook_lessons/*.json 2>/dev/null | head -5
# Run C — T3 enabled local, B's lesson should load
echo "──── RUN C: T3 local, reads B's lesson ────"
bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_C.log 2>&1 || true
echo " C exit=$?"
# Run D — T3 enabled CLOUD (gpt-oss:120b), reads B+C lessons
echo "──── RUN D: T3 cloud, reads B+C lessons ────"
LH_OVERVIEW_CLOUD=1 bun tests/multi-agent/scenario.ts > /tmp/lakehouse_ab_D.log 2>&1 || true
echo " D exit=$?"
echo "▶ all runs done at $(date -Iseconds)"
echo "▶ scorecard:"
ls -1dt tests/multi-agent/playbooks/scenario-* | head -4 | tac | python3 -c "
import sys, os, json
runs = [l.strip() for l in sys.stdin if l.strip()]
labels = ['A(no-T3)','B(T3-seed)','C(T3-read)','D(T3-cloud)']
# Prepend Run A: most recent BEFORE the ab_t3_test kicked off is Run A
# (launched separately). But we only picked up the most recent 4 runs.
# Actually: ab_t3_test runs B/C/D, so recent 3 = B,C,D. Run A is the one
# BEFORE those — find it separately.
# Reread to include Run A:
import subprocess
all_runs = subprocess.check_output(['bash','-c','ls -1dt tests/multi-agent/playbooks/scenario-* | head -8']).decode().strip().split('\n')
# The 4 most recent are D, C, B, A (reverse chronological).
top4 = list(reversed(all_runs[:4])) # oldest first → A,B,C,D
rows = []
for i, path in enumerate(top4):
try:
results = json.load(open(os.path.join(path, 'results.json')))
except FileNotFoundError:
continue
ok = sum(1 for r in results if r.get('ok'))
turns = sum(r.get('turns', 0) for r in results)
gaps = sum(len(r.get('gap_signals', [])) for r in results)
cites = sum(len(r.get('playbook_citations') or []) for r in results)
prior = []
try:
prior = json.load(open(os.path.join(path, 'prior_lessons.json')))
except FileNotFoundError:
pass
rows.append({
'label': labels[i] if i < len(labels) else f'run{i}',
'path': path,
'ok_events': ok,
'total_events': len(results),
'total_turns': turns,
'total_gaps': gaps,
'total_citations': cites,
'prior_lessons_loaded': len(prior),
})
scorecard = {'generated_at': __import__('datetime').datetime.utcnow().isoformat()+'Z', 'runs': rows}
open('tests/multi-agent/playbooks/ab_scorecard.json','w').write(json.dumps(scorecard, indent=2))
print(json.dumps(scorecard, indent=2))
"
echo "▶ saved: tests/multi-agent/playbooks/ab_scorecard.json"
-121
View File
@@ -1,121 +0,0 @@
#!/usr/bin/env bun
/**
* Pass 2: matrix corpus + relevance threshold sweep.
*
* For each (corpus, threshold) combination, run codereview_matrix_only
* on the same N files. Compares which corpus actually adds grounded
* findings vs codereview_isolation (matrix-off baseline).
*
* Output: data/_kb/mode_experiments.jsonl gets one row per call,
* tagged via the force_matrix_corpus + force_relevance_threshold
* fields visible in `sources`. Aggregator can then group by corpus.
*
* Usage: bun run scripts/mode_pass2_corpus_sweep.ts
*/
const GATEWAY = process.env.LH_GATEWAY ?? "http://localhost:3100";
const MODEL = process.env.LH_MODEL ?? "openai/gpt-oss-120b:free";
const FILES = (process.env.LH_FILES ?? [
"crates/queryd/src/delta.rs",
"crates/queryd/src/service.rs",
"crates/vectord/src/pathway_memory.rs",
"crates/gateway/src/v1/mode.rs",
"crates/aibridge/src/client.rs",
].join(",")).split(",");
const CORPORA = (process.env.LH_CORPORA ?? [
"distilled_procedural_v20260423102847",
"distilled_factual_v20260423095819",
"distilled_config_hint_v20260423102847",
"kb_team_runs_v1",
].join(",")).split(",");
const THRESHOLDS = (process.env.LH_THRESHOLDS ?? "0.2,0.3,0.4,0.5").split(",").map(Number);
interface Result {
corpus: string;
threshold: number;
file: string;
ok: boolean;
matrix_kept?: number;
matrix_dropped?: number;
response_chars?: number;
latency_ms?: number;
error?: string;
}
async function runOne(corpus: string, threshold: number, file: string): Promise<Result> {
try {
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
task_class: "scrum_review",
file_path: file,
force_mode: "codereview_matrix_only",
force_model: MODEL,
force_matrix_corpus: corpus,
force_relevance_threshold: threshold,
}),
signal: AbortSignal.timeout(180_000),
});
if (!r.ok) {
const body = await r.text().catch(() => "");
return { corpus, threshold, file, ok: false, error: `HTTP ${r.status}: ${body.slice(0, 150)}` };
}
const j: any = await r.json();
return {
corpus, threshold, file, ok: true,
matrix_kept: j.sources?.matrix_chunks_kept,
matrix_dropped: j.sources?.matrix_chunks_dropped,
response_chars: (j.response ?? "").length,
latency_ms: j.latency_ms,
};
} catch (e: any) {
return { corpus, threshold, file, ok: false, error: e.message };
}
}
async function main() {
const total = CORPORA.length * THRESHOLDS.length * FILES.length;
console.log(`[pass2] corpora=${CORPORA.length} × thresholds=${THRESHOLDS.length} × files=${FILES.length} = ${total} runs`);
console.log(`[pass2] model=${MODEL}\n`);
let i = 0;
const results: Result[] = [];
for (const corpus of CORPORA) {
for (const threshold of THRESHOLDS) {
for (const file of FILES) {
i++;
process.stdout.write(` [${i}/${total}] corpus=${corpus.slice(0, 30).padEnd(30)} thr=${threshold.toFixed(1)} ${file.slice(-32).padStart(32)} ... `);
const r = await runOne(corpus, threshold, file);
results.push(r);
if (r.ok) {
const total_chunks = (r.matrix_kept ?? 0) + (r.matrix_dropped ?? 0);
console.log(`✓ k=${r.matrix_kept}/${total_chunks} resp=${r.response_chars} ${((r.latency_ms ?? 0) / 1000).toFixed(1)}s`);
} else {
console.log(`✗ ${r.error}`);
}
}
}
}
console.log(`\n[pass2] complete · ${results.filter(r => r.ok).length}/${results.length} succeeded`);
// Per-corpus×threshold roll-up of kept-rate (the matrix usefulness proxy).
console.log(`\n[pass2] kept-rate by corpus × threshold (avg chunks kept per call):`);
console.log(` ${"corpus".padEnd(40)} ${THRESHOLDS.map(t => `thr=${t.toFixed(1)}`).join(" ").padStart(35)}`);
for (const corpus of CORPORA) {
const cells = THRESHOLDS.map(t => {
const matched = results.filter(r => r.ok && r.corpus === corpus && r.threshold === t);
if (matched.length === 0) return " — ";
const avgKept = matched.reduce((s, r) => s + (r.matrix_kept ?? 0), 0) / matched.length;
return avgKept.toFixed(1).padStart(5);
}).join(" ");
console.log(` ${corpus.slice(0, 40).padEnd(40)} ${cells}`);
}
console.log(`\n[pass2] aggregate findings/groundedness with: bun run scripts/mode_compare.ts`);
}
main().catch(e => { console.error(e); process.exit(1); });
-109
View File
@@ -1,109 +0,0 @@
#!/usr/bin/env bun
/**
* Pass 3: variance test.
*
* Runs codereview_lakehouse on the SAME file N times at each of M
* temperatures. Measures run-to-run stability of grounded finding
* count, response size, and latency. Anything <100% groundedness
* is a leak; track which symbols got hallucinated.
*
* Output appends to data/_kb/mode_experiments.jsonl. The aggregator
* can group by ts and identify variance buckets.
*
* Usage: bun run scripts/mode_pass3_variance.ts
*/
const GATEWAY = process.env.LH_GATEWAY ?? "http://localhost:3100";
const MODEL = process.env.LH_MODEL ?? "openai/gpt-oss-120b:free";
const FILES = (process.env.LH_FILES ?? [
"crates/queryd/src/delta.rs",
"crates/vectord/src/pathway_memory.rs",
"crates/gateway/src/v1/mode.rs",
].join(",")).split(",");
const TEMPS = (process.env.LH_TEMPS ?? "0.0,0.1,0.3").split(",").map(Number);
const REPS = Number(process.env.LH_REPS ?? 5);
interface Result {
file: string;
temp: number;
rep: number;
ok: boolean;
response_chars?: number;
latency_ms?: number;
error?: string;
}
async function runOne(file: string, temp: number, rep: number): Promise<Result> {
try {
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
task_class: "scrum_review",
file_path: file,
force_mode: "codereview_lakehouse",
force_model: MODEL,
force_temperature: temp,
}),
signal: AbortSignal.timeout(180_000),
});
if (!r.ok) {
const body = await r.text().catch(() => "");
return { file, temp, rep, ok: false, error: `HTTP ${r.status}: ${body.slice(0, 150)}` };
}
const j: any = await r.json();
return {
file, temp, rep, ok: true,
response_chars: (j.response ?? "").length,
latency_ms: j.latency_ms,
};
} catch (e: any) {
return { file, temp, rep, ok: false, error: e.message };
}
}
async function main() {
const total = FILES.length * TEMPS.length * REPS;
console.log(`[pass3] files=${FILES.length} × temps=${TEMPS.length} × reps=${REPS} = ${total} runs`);
console.log(`[pass3] model=${MODEL}\n`);
let i = 0;
const results: Result[] = [];
for (const file of FILES) {
for (const temp of TEMPS) {
for (let rep = 1; rep <= REPS; rep++) {
i++;
process.stdout.write(` [${i}/${total}] temp=${temp.toFixed(1)} rep=${rep}/${REPS} ${file.slice(-32).padStart(32)} ... `);
const r = await runOne(file, temp, rep);
results.push(r);
if (r.ok) {
console.log(`✓ resp=${r.response_chars} ${((r.latency_ms ?? 0) / 1000).toFixed(1)}s`);
} else {
console.log(`✗ ${r.error}`);
}
}
}
}
console.log(`\n[pass3] complete · ${results.filter(r => r.ok).length}/${results.length} succeeded`);
// Per-file × temp variance summary (response_chars stddev as a quick
// proxy for output instability).
console.log(`\n[pass3] response_chars variance (mean ± stddev) by file × temp:`);
console.log(` ${"file".padEnd(40)} ${TEMPS.map(t => `temp=${t.toFixed(1)}`.padStart(20)).join(" ")}`);
for (const file of FILES) {
const cells = TEMPS.map(t => {
const xs = results.filter(r => r.ok && r.file === file && r.temp === t).map(r => r.response_chars ?? 0);
if (xs.length === 0) return " — ";
const mean = xs.reduce((s, x) => s + x, 0) / xs.length;
const sd = Math.sqrt(xs.reduce((s, x) => s + Math.pow(x - mean, 2), 0) / xs.length);
return `${Math.round(mean).toString().padStart(7)} ± ${Math.round(sd).toString().padEnd(6)}`.padStart(20);
}).join(" ");
console.log(` ${file.slice(0, 40).padEnd(40)} ${cells}`);
}
console.log(`\n[pass3] grounding variance via: bun run scripts/mode_compare.ts (look for grounded-N column drift)`);
}
main().catch(e => { console.error(e); process.exit(1); });
-127
View File
@@ -1,127 +0,0 @@
#!/usr/bin/env bun
/**
* Pass 4: staffing_inference_lakehouse cross-domain validation.
*
* Runs the staffing-domain mode against synthetic fill requests.
* Validates that the modes-as-prompt-molders architecture generalizes
* beyond code review — the composer pattern (file_content + bug
* fingerprints + relevance-filtered matrix + domain framing) should
* produce grounded staffing recommendations the same way it produces
* grounded code reviews.
*
* Each fill request is posted as `file_content` (since the runner's
* shape expects file content; for staffing it's the request payload).
* file_path is set to a synthetic path under requests/ so pathway
* memory bucketing groups requests by geo+role.
*
* Usage: bun run scripts/mode_pass4_staffing.ts
*/
const GATEWAY = process.env.LH_GATEWAY ?? "http://localhost:3100";
const MODEL = process.env.LH_MODEL ?? "openai/gpt-oss-120b:free";
interface FillRequest {
city: string;
state: string;
role: string;
count: number;
deadline: string;
notes?: string;
}
const REQUESTS: FillRequest[] = [
{ city: "Toledo", state: "OH", role: "Welder", count: 2, deadline: "2026-04-29", notes: "OSHA 10 required" },
{ city: "Nashville", state: "TN", role: "Forklift Operator", count: 3, deadline: "2026-05-01" },
{ city: "Chicago", state: "IL", role: "Assembler", count: 5, deadline: "2026-04-30", notes: "second shift" },
{ city: "South Bend", state: "IN", role: "Electrician", count: 1, deadline: "2026-04-28", notes: "journeyman license" },
{ city: "Murfreesboro", state: "TN", role: "Packaging Operator", count: 4, deadline: "2026-05-02" },
];
function requestToPayload(req: FillRequest): string {
return [
`# Fill Request`,
`Role: ${req.role} × ${req.count}`,
`Location: ${req.city}, ${req.state}`,
`Deadline: ${req.deadline}`,
req.notes ? `Notes: ${req.notes}` : "",
"",
"Recommend candidates from the matrix data. Cite playbook references.",
].filter(Boolean).join("\n");
}
interface Result {
req: FillRequest;
ok: boolean;
response_chars?: number;
bug_fingerprints?: number;
matrix_kept?: number;
matrix_dropped?: number;
latency_ms?: number;
error?: string;
preview?: string;
}
async function runOne(req: FillRequest): Promise<Result> {
const payload = requestToPayload(req);
const file_path = `requests/${req.role.toLowerCase().replace(/\s+/g, "_")}_${req.city.toLowerCase().replace(/\s+/g, "_")}_${req.state}.md`;
try {
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
task_class: "staffing_inference",
file_path,
file_content: payload,
force_mode: "staffing_inference_lakehouse",
force_model: MODEL,
}),
signal: AbortSignal.timeout(180_000),
});
if (!r.ok) {
const body = await r.text().catch(() => "");
return { req, ok: false, error: `HTTP ${r.status}: ${body.slice(0, 200)}` };
}
const j: any = await r.json();
return {
req, ok: true,
response_chars: (j.response ?? "").length,
bug_fingerprints: j.sources?.bug_fingerprints_count,
matrix_kept: j.sources?.matrix_chunks_kept,
matrix_dropped: j.sources?.matrix_chunks_dropped,
latency_ms: j.latency_ms,
preview: (j.response ?? "").slice(0, 400),
};
} catch (e: any) {
return { req, ok: false, error: e.message };
}
}
async function main() {
console.log(`[pass4] requests=${REQUESTS.length} model=${MODEL} mode=staffing_inference_lakehouse\n`);
let i = 0;
const results: Result[] = [];
for (const req of REQUESTS) {
i++;
process.stdout.write(` [${i}/${REQUESTS.length}] ${req.role.padEnd(22)} × ${req.count} in ${req.city}, ${req.state} ... `);
const r = await runOne(req);
results.push(r);
if (r.ok) {
console.log(`✓ resp=${r.response_chars} bug=${r.bug_fingerprints ?? 0} mtx=${r.matrix_kept ?? 0}/${(r.matrix_kept ?? 0) + (r.matrix_dropped ?? 0)} ${((r.latency_ms ?? 0) / 1000).toFixed(1)}s`);
} else {
console.log(`✗ ${r.error}`);
}
}
console.log(`\n[pass4] complete · ${results.filter(r => r.ok).length}/${results.length} succeeded\n`);
// Show first successful response head to verify the framing actually
// produced staffing-style output (verdict + ranked candidates) not
// generic prose.
const first = results.find(r => r.ok && r.preview);
if (first) {
console.log(`[pass4] first successful response preview (${first.req.city} ${first.req.role}):`);
console.log(first.preview!.split("\n").map(l => " | " + l).join("\n"));
}
}
main().catch(e => { console.error(e); process.exit(1); });
-169
View File
@@ -1,169 +0,0 @@
#!/usr/bin/env bun
/**
* Pass 5 variance summarizer. Reads data/_kb/mode_experiments.jsonl
* since a timestamp, groups by (mode|corpus), reports mean ± stddev
* of grounded finding count, plus a head-to-head wins/losses table
* vs the isolation baseline.
*
* Usage:
* bun run scripts/mode_pass5_summarize.ts # default 2h
* bun run scripts/mode_pass5_summarize.ts --since 2026-04-26T22 # explicit
*/
import { readFileSync, existsSync } from "node:fs";
const argSince = (() => {
const i = Bun.argv.indexOf("--since");
return i >= 0 ? Bun.argv[i + 1] : new Date(Date.now() - 2 * 60 * 60 * 1000).toISOString();
})();
const JSONL = "data/_kb/mode_experiments.jsonl";
if (!existsSync(JSONL)) { console.error(`no ${JSONL}`); process.exit(1); }
interface Row {
ts: string; mode: string; file_path: string; response: string;
sources: { matrix_corpus?: string | string[] | null };
latency_ms: number;
}
function corpusKey(c: any): string {
if (!c) return "";
if (typeof c === "string") return c;
if (Array.isArray(c)) return c.length === 0 ? "" : [...c].sort().join("+");
return "";
}
const condKey = (r: Row) => {
const c = corpusKey(r.sources?.matrix_corpus);
return c ? `${r.mode}|${c}` : r.mode;
};
// Reuse the same grounding logic as mode_compare — symbols cited in
// findings rows must appear in the focus file, and any line numbers
// must fall within EOF.
function extractFindings(md: string): { symbols: string[]; lines: number[] }[] {
const sec = /(?:^|\n)#{1,3}[^\na-zA-Z]*(?:Ranked\s+)?Findings?[^\n]*\n/i;
const m = md.match(sec);
let section = md;
if (m && m.index !== undefined) {
const after = md.slice(m.index + m[0].length);
const stop = after.search(/\n#{1,3}[^\na-zA-Z]*(?:Patch|Suggestion|Reference|Summary|Concrete)/i);
section = stop >= 0 ? after.slice(0, stop) : after;
}
// Three row shapes:
// 1) numbered: `| 1 | ... |`
// 2) path-with-line: `| service.rs:106 | ... |`
// 3) path-with-sym: `| crates/vectord/src/pathway_memory.rs:load_fn (≈L220) | ... |`
// Pick whichever shape matches the most rows (ties favor numbered).
const numbered = section.split("\n").filter(l => /^\|\s*\*?\*?\d+\*?\*?\s*\|/.test(l));
const pathRows = section.split("\n").filter(l => /^\|\s*[a-z_/\.][a-z_/\.0-9]*\.(rs|ts|py)\b/i.test(l));
const rows = numbered.length >= pathRows.length ? numbered : pathRows;
return rows.map(row => {
const sym = new Set<string>();
for (const t of row.matchAll(/`([A-Za-z_][A-Za-z0-9_:]*)`/g)) sym.add(t[1]);
for (const t of row.matchAll(/\b([a-z][a-z0-9_]{4,})\b/g)) sym.add(t[1]);
const lines: number[] = [];
for (const t of row.matchAll(/[:\-](\d{2,5})/g)) lines.push(parseInt(t[1]));
return { symbols: [...sym], lines };
});
}
function grounded(md: string, file: string): { total: number; grounded: number; oob: number } {
const content = readFileSync(file, "utf8");
const eof = content.split("\n").length;
const findings = extractFindings(md);
let g = 0, oob = 0;
for (const f of findings) {
const symHit = f.symbols.length > 0 && f.symbols.some(s => content.includes(s));
const lineOob = f.lines.length > 0 && f.lines.some(l => l > eof);
if (lineOob) oob++;
if (symHit && !lineOob) g++;
}
return { total: findings.length, grounded: g, oob };
}
const lines = readFileSync(JSONL, "utf8").split("\n").filter(Boolean);
const rows: Row[] = [];
for (const l of lines) {
try {
const r: Row = JSON.parse(l);
if (r.ts < argSince) continue;
rows.push(r);
} catch {}
}
if (rows.length === 0) { console.error(`no rows since ${argSince}`); process.exit(1); }
// Group: condition → file → array of grounded counts
type CellArr = { grnd: number[]; total: number[]; oob: number[]; ms: number[] };
const byCond: Record<string, Record<string, CellArr>> = {};
for (const r of rows) {
const k = condKey(r);
byCond[k] ??= {};
byCond[k][r.file_path] ??= { grnd: [], total: [], oob: [], ms: [] };
const g = grounded(r.response, r.file_path);
byCond[k][r.file_path].grnd.push(g.grounded);
byCond[k][r.file_path].total.push(g.total);
byCond[k][r.file_path].oob.push(g.oob);
byCond[k][r.file_path].ms.push(r.latency_ms);
}
function stats(xs: number[]): { n: number; mean: number; sd: number; min: number; max: number } {
const n = xs.length;
if (n === 0) return { n: 0, mean: 0, sd: 0, min: 0, max: 0 };
const mean = xs.reduce((s, x) => s + x, 0) / n;
const variance = n === 1 ? 0 : xs.reduce((s, x) => s + (x - mean) ** 2, 0) / (n - 1);
return { n, mean, sd: Math.sqrt(variance), min: Math.min(...xs), max: Math.max(...xs) };
}
const conditions = Object.keys(byCond).sort();
const files = [...new Set(rows.map(r => r.file_path))].sort();
console.log(`\n═══ Pass 5 variance — since ${argSince} ═══\n`);
console.log(` ${rows.length} rows · ${conditions.length} conditions · ${files.length} files\n`);
for (const file of files) {
console.log(`📄 ${file}`);
console.log(` ${"condition".padEnd(56)} n ${"grounded mean ± sd".padStart(20)} ${"range".padStart(8)} ${"oob".padStart(4)} ${"avg ms".padStart(7)}`);
console.log(` ${"─".repeat(56)} ─── ${"─".repeat(20)} ${"─".repeat(8)} ${"─".repeat(4)} ${"─".repeat(7)}`);
for (const c of conditions) {
const cell = byCond[c]?.[file];
if (!cell || cell.grnd.length === 0) continue;
const s = stats(cell.grnd);
const oobSum = cell.oob.reduce((a, b) => a + b, 0);
const msMean = cell.ms.reduce((a, b) => a + b, 0) / cell.ms.length;
const meanSd = `${s.mean.toFixed(1)} ± ${s.sd.toFixed(1)}`;
const range = `[${s.min}-${s.max}]`;
console.log(` ${c.padEnd(56)} ${String(s.n).padStart(3)} ${meanSd.padStart(20)} ${range.padStart(8)} ${String(oobSum).padStart(4)} ${Math.round(msMean / 1000).toString().padStart(5)}s`);
}
console.log("");
}
// Head-to-head: for each condition vs isolation baseline, count rep-by-rep
// wins across the same file. Requires equal rep counts.
console.log(`═══ Head-to-head: each condition vs isolation, rep-by-rep ═══\n`);
const isoKey = conditions.find(c => c.startsWith("codereview_isolation"));
if (!isoKey) {
console.log(" no isolation rows in window");
} else {
console.log(` baseline: ${isoKey}\n`);
console.log(` ${"challenger".padEnd(56)} wins losses ties Δ mean grnd`);
console.log(` ${"─".repeat(56)} ${"─".repeat(4)} ${"─".repeat(6)} ${"─".repeat(4)} ${"─".repeat(12)}`);
for (const c of conditions) {
if (c === isoKey) continue;
let wins = 0, losses = 0, ties = 0, deltaSum = 0, n = 0;
for (const file of files) {
const isoArr = byCond[isoKey]?.[file]?.grnd ?? [];
const cArr = byCond[c]?.[file]?.grnd ?? [];
const k = Math.min(isoArr.length, cArr.length);
for (let i = 0; i < k; i++) {
if (cArr[i] > isoArr[i]) wins++;
else if (cArr[i] < isoArr[i]) losses++;
else ties++;
deltaSum += cArr[i] - isoArr[i];
n++;
}
}
const dMean = n > 0 ? (deltaSum / n).toFixed(2) : "—";
console.log(` ${c.padEnd(56)} ${String(wins).padStart(4)} ${String(losses).padStart(6)} ${String(ties).padStart(4)} ${dMean.padStart(12)}`);
}
}
-105
View File
@@ -1,105 +0,0 @@
#!/usr/bin/env bun
/**
* Pass 5: variance test for the 2026-04-26 paid-model bake-off.
*
* The pass-4 single-rep sweep showed isolation beating every matrix
* condition by 1.0-1.4 grounded findings/file on grok-4.1-fast. This
* harness runs N reps × M conditions on the file where the effect was
* sharpest (pathway_memory.rs, 1355 lines) so we can decide whether
* the deltas are real signal or run-to-run noise.
*
* Conditions:
* 1. codereview_isolation — no matrix
* 2. codereview_lakehouse + corpus=lakehouse_arch_v1 — A only
* 3. codereview_lakehouse + corpus=lakehouse_symbols_v1 — C only
* 4. codereview_lakehouse (modes.toml default) — A+C composed
*
* Output appends per-call to data/_kb/mode_experiments.jsonl. Aggregate
* with `bun run scripts/mode_compare.ts --since <ts>` and read the
* grounded column with multiple rows per (mode|corpus) key.
*
* Usage:
* bun run scripts/mode_pass5_variance_paid.ts
* LH_REPS=3 LH_FILE=crates/queryd/src/delta.rs bun run scripts/mode_pass5_variance_paid.ts
*/
const GATEWAY = process.env.LH_GATEWAY ?? "http://localhost:3100";
const MODEL = process.env.LH_MODEL ?? "x-ai/grok-4.1-fast";
const FILE = process.env.LH_FILE ?? "crates/vectord/src/pathway_memory.rs";
const REPS = Number(process.env.LH_REPS ?? 5);
interface Condition {
label: string;
mode: string;
corpus?: string | string[];
}
const ALL_CONDITIONS: Condition[] = [
{ label: "isolation ", mode: "codereview_isolation" },
{ label: "arch_only ", mode: "codereview_lakehouse", corpus: "lakehouse_arch_v1" },
{ label: "symbols_only ", mode: "codereview_lakehouse", corpus: "lakehouse_symbols_v1" },
{ label: "composed (A+C) ", mode: "codereview_lakehouse" /* uses modes.toml default */ },
];
// Optional whitelist via env: LH_CONDITIONS=isolation,composed limits the
// run to a subset (matches against the trimmed `label`). Useful when only
// the head-to-head pair matters and saves ~50% latency on slow rungs.
const wantedLabels = (process.env.LH_CONDITIONS ?? "")
.split(",").map(s => s.trim().toLowerCase()).filter(Boolean);
const CONDITIONS: Condition[] = wantedLabels.length === 0
? ALL_CONDITIONS
: ALL_CONDITIONS.filter(c => wantedLabels.some(w => c.label.trim().toLowerCase().startsWith(w)));
async function runOne(c: Condition, rep: number): Promise<{ ok: boolean; latency_ms?: number; resp_chars?: number; error?: string }> {
const body: any = {
task_class: "scrum_review",
file_path: FILE,
force_mode: c.mode,
force_model: MODEL,
};
if (c.corpus !== undefined) body.force_matrix_corpus = c.corpus;
try {
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(240_000),
});
if (!r.ok) {
const txt = await r.text().catch(() => "");
return { ok: false, error: `HTTP ${r.status}: ${txt.slice(0, 160)}` };
}
const j: any = await r.json();
return { ok: true, latency_ms: j.latency_ms, resp_chars: (j.response ?? "").length };
} catch (e: any) {
return { ok: false, error: e.message };
}
}
async function main() {
const total = CONDITIONS.length * REPS;
console.log(`[pass5] file=${FILE}`);
console.log(`[pass5] model=${MODEL} · ${CONDITIONS.length} conditions × ${REPS} reps = ${total} runs`);
console.log("");
let i = 0;
const startTs = new Date().toISOString();
for (let rep = 1; rep <= REPS; rep++) {
for (const c of CONDITIONS) {
i++;
process.stdout.write(` [${i}/${total}] rep=${rep} ${c.label}... `);
const r = await runOne(c, rep);
if (r.ok) {
console.log(`✓ ${r.resp_chars} chars · ${((r.latency_ms ?? 0) / 1000).toFixed(1)}s`);
} else {
console.log(`✗ ${r.error}`);
}
}
}
console.log(`\n[pass5] complete · started ${startTs}`);
console.log(`[pass5] aggregate: bun run scripts/mode_compare.ts --since ${startTs}`);
}
main().catch(e => { console.error(e); process.exit(1); });
-458
View File
@@ -1,458 +0,0 @@
#!/bin/bash
# OVERNIGHT PROOF — the test that settles it
# Runs unattended: embed 500K, build indexes, measure recall,
# autonomous agent test, sustained load. ~3 hours total.
#
# Monitor: tail -f /home/profit/lakehouse/logs/overnight_proof.log
set -uo pipefail
LOG="/home/profit/lakehouse/logs/overnight_proof.log"
STATE="/tmp/overnight_proof_state"
LOCK="/tmp/overnight_proof.lock"
LH="http://localhost:3100"
GW="http://localhost:3700"
mkdir -p /home/profit/lakehouse/logs
if [ -f "$LOCK" ] && kill -0 "$(cat $LOCK)" 2>/dev/null; then
echo "$(date) Already running" >> "$LOG"
exit 0
fi
echo $$ > "$LOCK"
trap "rm -f $LOCK" EXIT
log() { echo "$(date '+%H:%M:%S') $1" | tee -a "$LOG"; }
touch "$STATE"
step=$(cat "$STATE" 2>/dev/null || echo "embed")
log "═══ OVERNIGHT PROOF: step=$step ═══"
case "$step" in
embed)
log "STEP 1/5: Embedding 500K workers through Ollama (~40 min)"
log " This is the real test — actual nomic-embed-text embeddings, not random vectors"
python3 << 'PYEOF' >> "$LOG" 2>&1
import json, time, sys
from urllib.request import Request, urlopen
from urllib.error import HTTPError
LH = "http://localhost:3100"
def post(path, body, timeout=300):
r = Request(f"{LH}{path}", json.dumps(body).encode(), headers={"Content-Type": "application/json"})
try: return json.loads(urlopen(r, timeout=timeout).read())
except HTTPError as e: return {"error": e.read().decode()[:200]}
except Exception as e: return {"error": str(e)}
# Fetch 500K resume_text for embedding
print("Fetching resume texts from workers_500k...")
r = post("/query/sql", {"sql": "SELECT worker_id, resume_text FROM workers_500k LIMIT 500000"})
if "error" in r:
print(f"SQL error: {r['error']}")
sys.exit(1)
rows = r.get("rows", [])
print(f"Got {len(rows)} rows")
# Build docs for embedding
docs = []
for row in rows:
wid = row.get("worker_id", "")
text = row.get("resume_text", "")
if text and len(text) > 20:
docs.append({"id": f"W500K-{wid}", "text": text})
print(f"{len(docs)} docs ready for embedding")
# Chunk into batches of 50K to avoid timeout issues
BATCH = 50000
for batch_start in range(0, len(docs), BATCH):
batch = docs[batch_start:batch_start + BATCH]
batch_num = batch_start // BATCH + 1
total_batches = (len(docs) + BATCH - 1) // BATCH
idx_name = f"workers_500k_v{batch_num}"
print(f"\nBatch {batch_num}/{total_batches}: {len(batch)} docs → index '{idx_name}'")
t0 = time.time()
r = post("/vectors/index", {
"index_name": idx_name,
"source": "workers_500k",
"documents": batch,
"chunk_size": 500,
"overlap": 50,
}, timeout=600)
if "error" in r:
print(f" Index creation error: {r['error']}")
continue
job_id = r.get("job_id")
chunks = r.get("chunks", 0)
print(f" Job {job_id}: {chunks} chunks, embedding in background...")
# Wait for this batch to complete
for _ in range(600): # 50 min max per batch
time.sleep(5)
status = post(f"/vectors/jobs/{job_id}", None) if job_id else {"status": "unknown"}
if isinstance(status, dict):
state = status.get("status", "unknown")
progress = status.get("embedded_chunks", 0)
if state == "completed":
elapsed = time.time() - t0
rate = chunks / elapsed if elapsed > 0 else 0
print(f" DONE: {chunks} chunks in {elapsed:.0f}s ({rate:.0f}/sec)")
break
elif state == "failed":
print(f" FAILED: {status.get('error', 'unknown')}")
break
sys.stdout.write(f"\r {state}: {progress}/{chunks} chunks...")
sys.stdout.flush()
print()
print("\nAll batches submitted. Checking indexes...")
r = post("/vectors/indexes", None)
if not isinstance(r, list): r = []
for idx in r:
if "500k" in idx.get("index_name", ""):
print(f" {idx['index_name']}: {idx['chunk_count']} chunks")
print("STEP 1 COMPLETE")
PYEOF
if grep -q "STEP 1 COMPLETE" "$LOG"; then
echo "build_indexes" > "$STATE"
log "Embedding complete — moving to index build"
else
log "Embedding may still be running — will check on next heartbeat"
echo "check_embed" > "$STATE"
fi
;;
check_embed)
log "Checking embedding job status..."
python3 -c "
import json
from urllib.request import urlopen
r = json.loads(urlopen('http://localhost:3100/vectors/jobs', timeout=30).read())
running = [j for j in r if j.get('status') == 'running']
completed = [j for j in r if j.get('status') == 'completed' and '500k' in j.get('index_name','')]
print(f'Running: {len(running)}, Completed 500K: {len(completed)}')
if not running:
print('ALL_DONE')
" >> "$LOG" 2>&1
if grep -q "ALL_DONE" "$LOG"; then
echo "build_indexes" > "$STATE"
fi
;;
build_indexes)
log "STEP 2/5: Building HNSW + Lance on real 500K embeddings"
python3 << 'PYEOF' >> "$LOG" 2>&1
import json, time
from urllib.request import Request, urlopen
LH = "http://localhost:3100"
def post(path, body, timeout=600):
r = Request(f"{LH}{path}", json.dumps(body).encode(), headers={"Content-Type": "application/json"})
return json.loads(urlopen(r, timeout=timeout).read())
# Find the first 500K index
indexes = json.loads(urlopen(f"{LH}/vectors/indexes", timeout=30).read())
idx_500k = [i for i in indexes if "500k" in i.get("index_name","")]
if not idx_500k:
print("No 500K index found — embedding may not be complete")
exit(1)
idx_name = idx_500k[0]["index_name"]
chunks = idx_500k[0]["chunk_count"]
print(f"Using index: {idx_name} ({chunks} chunks)")
# Build HNSW
print(f"Building HNSW on {chunks} real embeddings...")
t0 = time.time()
r = post("/vectors/hnsw/build", {"index_name": idx_name})
print(f" HNSW: {r.get('vectors',0)} vectors in {time.time()-t0:.0f}s")
# Migrate to Lance
print(f"Migrating to Lance...")
t0 = time.time()
r = post(f"/vectors/lance/migrate/{idx_name}", {})
stats = r.get("stats", {})
print(f" Lance: {stats.get('rows_written',0)} rows in {stats.get('duration_secs',0):.1f}s")
# Build IVF_PQ on Lance
# sqrt(50K) ≈ 224 partitions for a 50K batch
print(f"Building IVF_PQ on Lance...")
t0 = time.time()
r = post(f"/vectors/lance/index/{idx_name}", {"num_partitions": 224, "num_bits": 8, "num_sub_vectors": 192})
print(f" IVF_PQ: built in {r.get('build_time_secs',0):.0f}s")
# Build scalar btree
print(f"Building scalar btree on doc_id...")
r = post(f"/vectors/lance/scalar-index/{idx_name}/doc_id", {})
print(f" Btree: built in {r.get('build_time_secs',0):.1f}s")
print("STEP 2 COMPLETE")
PYEOF
if grep -q "STEP 2 COMPLETE" "$LOG"; then
echo "recall_test" > "$STATE"
fi
;;
recall_test)
log "STEP 3/5: Measuring recall on REAL embeddings"
python3 << 'PYEOF' >> "$LOG" 2>&1
import json, time
from urllib.request import Request, urlopen
LH = "http://localhost:3100"
def post(path, body, timeout=300):
r = Request(f"{LH}{path}", json.dumps(body).encode(), headers={"Content-Type": "application/json"})
return json.loads(urlopen(r, timeout=timeout).read())
# Find 500K index
indexes = json.loads(urlopen(f"{LH}/vectors/indexes", timeout=30).read())
idx_500k = [i for i in indexes if "500k" in i.get("index_name","")]
if not idx_500k:
print("No 500K index — skipping recall")
exit(0)
idx_name = idx_500k[0]["index_name"]
# Auto-generate eval harness
print(f"Generating eval harness for {idx_name}...")
r = post(f"/vectors/hnsw/evals/{idx_name}_recall/autogen", {
"index_name": idx_name, "sample_count": 50, "k": 10,
})
print(f" Harness: {len(r.get('queries',[]))} queries, k={r.get('k',10)}")
# HNSW recall
print("Measuring HNSW recall...")
r = post("/vectors/hnsw/trial", {
"index_name": idx_name,
"harness": f"{idx_name}_recall",
"config": {"ef_construction": 80, "ef_search": 30, "seed": 42},
})
print(f" HNSW recall@10: {r.get('metrics',{}).get('recall_at_k',0):.4f}")
print(f" HNSW p50: {r.get('metrics',{}).get('search_latency_p50_us',0):.0f}us")
# Lance recall
print("Measuring Lance IVF_PQ recall...")
r = post(f"/vectors/lance/recall/{idx_name}", {
"harness": f"{idx_name}_recall", "top_k": 10,
})
print(f" Lance recall@10: {r.get('mean_recall',0):.4f}")
print(f" Lance p50: {r.get('latency_p50_us',0):.0f}us")
print("STEP 3 COMPLETE")
PYEOF
if grep -q "STEP 3 COMPLETE" "$LOG"; then
echo "autonomous_test" > "$STATE"
fi
;;
autonomous_test)
log "STEP 4/5: 100 staffing questions — LOCAL MODEL ONLY, no human steering"
python3 << 'PYEOF' >> "$LOG" 2>&1
import json, time, random
from urllib.request import Request, urlopen
from urllib.error import HTTPError
GW = "http://localhost:3700"
LH = "http://localhost:3100"
random.seed(2026)
def gw(path, body=None, timeout=180):
data = json.dumps(body).encode() if body else None
method = "POST" if body else "GET"
r = Request(f"{GW}{path}", data=data, method=method, headers={"Content-Type":"application/json"} if body else {})
try: return json.loads(urlopen(r, timeout=timeout).read())
except HTTPError as e: return {"error": e.read().decode()[:200]}
except Exception as e: return {"error": str(e)}
def sql(query):
r = gw("/sql", {"sql": query})
return r.get("rows", []) if "error" not in r else []
ROLES = ["Forklift Operator","Machine Operator","Assembler","Loader","Quality Tech",
"Welder","Sanitation Worker","Shipping Clerk","Production Worker","Maintenance Tech"]
STATES = ["IL","IN","OH","MO","TN","KY","WI","MI"]
print("═══ 100 AUTONOMOUS OPERATIONS ═══")
passed = 0
failed = 0
total_ms = 0
# Mix of operation types
for i in range(100):
op_type = random.choices(["match","count","aggregate","lookup"], weights=[50,25,15,10])[0]
role = random.choice(ROLES)
state = random.choice(STATES)
rel = round(random.uniform(0.6, 0.9), 2)
t0 = time.time()
ok = False
detail = ""
if op_type == "match":
r = gw("/search", {
"question": f"Find {role} workers in {state}",
"sql_filter": f"role = '{role}' AND state = '{state}' AND CAST(reliability AS DOUBLE) >= {rel}",
"dataset": "workers_500k", "top_k": 5, "generate": False,
})
matched = len(r.get("sources", []))
ok = matched > 0 or r.get("sql_matches", 0) == 0 # 0 matches is ok if SQL found 0
detail = f"match: {matched} results (sql={r.get('sql_matches',0)})"
elif op_type == "count":
truth = sql(f"SELECT COUNT(*) cnt FROM workers_500k WHERE role = '{role}' AND state = '{state}'")
expected = truth[0]["cnt"] if truth else 0
# Use keyword classifier logic: count → SQL
r = gw("/sql", {"sql": f"SELECT COUNT(*) cnt FROM workers_500k WHERE role = '{role}' AND state = '{state}'"})
got = r.get("rows", [{}])[0].get("cnt", -1) if "error" not in r else -1
ok = got == expected
detail = f"count: got={got} expected={expected}"
elif op_type == "aggregate":
r = gw("/sql", {"sql": f"SELECT ROUND(AVG(CAST(reliability AS DOUBLE)),3) avg FROM workers_500k WHERE role = '{role}' AND state = '{state}'"})
ok = "error" not in r and r.get("rows")
detail = f"aggregate: {r.get('rows',[{}])[0] if ok else r.get('error','?')[:40]}"
elif op_type == "lookup":
wid = random.randint(1, 500000)
r = gw(f"/worker/{wid}")
ok = r.get("rows") and len(r["rows"]) > 0
detail = f"lookup: worker {wid} {'found' if ok else 'not found'}"
ms = (time.time()-t0)*1000
total_ms += ms
if ok: passed += 1
else: failed += 1
if i % 20 == 0 or not ok:
icon = "OK" if ok else "FAIL"
print(f" [{i+1:3d}/100] {icon} {op_type:10s} {detail[:50]:50s} ({ms:.0f}ms)")
pct = passed / 100 * 100
print(f"\n═══ RESULT: {passed}/100 passed ({pct:.0f}%) in {total_ms/1000:.1f}s ═══")
print(f" avg latency: {total_ms/100:.0f}ms per operation")
# Log to playbook
gw("/log", {
"operation": f"autonomous_100: {passed}/100 ({pct:.0f}%)",
"approach": "keyword routing + SQL + hybrid, local model only",
"result": f"passed={passed} failed={failed} avg_ms={total_ms/100:.0f}",
"context": "overnight proof step 4",
})
if pct >= 90:
print("STEP 4 COMPLETE — AUTONOMOUS TEST PASSED")
else:
print(f"STEP 4 COMPLETE — {pct:.0f}% (below 90% target)")
PYEOF
echo "sustained_load" > "$STATE"
;;
sustained_load)
log "STEP 5/5: Sustained load — 30 minutes of continuous operations"
python3 << 'PYEOF' >> "$LOG" 2>&1
import json, time, random, concurrent.futures
from urllib.request import Request, urlopen
from urllib.error import HTTPError
GW = "http://localhost:3700"
random.seed(42)
def gw(path, body=None):
data = json.dumps(body).encode() if body else None
r = Request(f"{GW}{path}", data=data, method="POST" if body else "GET",
headers={"Content-Type":"application/json"} if body else {})
try: return json.loads(urlopen(r, timeout=60).read())
except: return {"error": "timeout"}
ROLES = ["Forklift Operator","Machine Operator","Assembler","Loader","Quality Tech"]
STATES = ["IL","IN","OH","MO"]
print("═══ SUSTAINED LOAD: 30 minutes ═══")
duration = 30 * 60 # 30 minutes
t_start = time.time()
ops = 0
errors = 0
cycle = 0
while time.time() - t_start < duration:
cycle += 1
batch_ops = 0
batch_errors = 0
# Fire 10 concurrent operations
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
futures = []
for _ in range(10):
role = random.choice(ROLES)
state = random.choice(STATES)
futures.append(pool.submit(gw, "/sql", {
"sql": f"SELECT COUNT(*) FROM workers_500k WHERE role = '{role}' AND state = '{state}'"
}))
for f in concurrent.futures.as_completed(futures):
r = f.result()
batch_ops += 1
if "error" in r: batch_errors += 1
ops += batch_ops
errors += batch_errors
elapsed = time.time() - t_start
remaining = duration - elapsed
if cycle % 30 == 0: # Log every ~30 cycles
rate = ops / elapsed
print(f" {elapsed/60:.0f}min: {ops} ops ({rate:.0f}/sec) errors={errors} remaining={remaining/60:.0f}min")
time.sleep(1) # 1 sec between batches
elapsed = time.time() - t_start
rate = ops / elapsed
print(f"\n═══ SUSTAINED LOAD COMPLETE ═══")
print(f" Duration: {elapsed/60:.1f} minutes")
print(f" Operations: {ops}")
print(f" Rate: {rate:.0f} ops/sec")
print(f" Errors: {errors} ({100*errors/max(ops,1):.1f}%)")
print(f" STEP 5 COMPLETE")
PYEOF
echo "report" > "$STATE"
;;
report)
log ""
log "═══════════════════════════════════════════════════════"
log " OVERNIGHT PROOF — COMPLETE"
log "═══════════════════════════════════════════════════════"
log " Step 1: 500K real embeddings via Ollama"
log " Step 2: HNSW + Lance indexes on real data"
log " Step 3: Recall measured on real embeddings"
log " Step 4: 100 autonomous operations (no human)"
log " Step 5: 30 min sustained concurrent load"
log ""
log " Full log: $LOG"
log "═══════════════════════════════════════════════════════"
echo "done" > "$STATE"
;;
done)
log "Overnight proof already complete."
;;
esac
+267
View File
@@ -0,0 +1,267 @@
#!/usr/bin/env bash
# attest_pre_identityd_biometric_state — one-shot defense artifact.
#
# Specification: docs/PHASE_1_6_BIPA_GATES.md §2 (Cryptographic
# attestation that no biometric data exists pre-identityd).
#
# Why this exists: in a BIPA dispute, plaintiffs may argue that the
# EXISTENCE of biometric schema fields constitutes constructive notice
# of intent to collect. The defense: prove that no biometric data was
# actually collected from real candidates before the identity service +
# consent gate (Phase 1.6 Gates 1-3) shipped.
#
# This script produces a defensible record of:
# 1. workers_500k.parquet schema has NO column named photo / biometric_*
# / face_* / image_*
# 2. data/_kb/*.jsonl and data/_pathway_memory/state.json contain NO
# base64 image magic bytes (JPEG /9j/, PNG iVBOR), no data:image/*
# MIME prefixes, and no field-name patterns that imply biometric
# payload (photo, biometric, deepface_*)
# 3. data/headshots/manifest.jsonl rows are entirely synthetic — count
# matches the face_pool size, and every row's source is a synthetic
# generator (not a real candidate upload)
#
# Output:
# docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_<DATE>.md
# — markdown attestation document with all evidence + a SHA-256
# hash of the evidence summary. Ready for J + counsel signature.
#
# Exit codes:
# 0 — clean, attestation written, ready for signature
# 1 — evidence FAILED, attestation NOT written; investigate before signing
# 2 — script error (missing tools, unreadable files)
set -uo pipefail
cd "$(dirname "$0")/../.."
# Dependency gate: pyarrow is required to read the parquet schema. Fail
# fast with a clear message rather than letting python3 -c emit a stack
# trace that gets captured into the attestation as "evidence". (Caught
# 2026-05-03 kimi scrum WARN python3-reliance.)
if ! python3 -c "import pyarrow" 2>/dev/null; then
echo "[attest] FAIL: python3 -c 'import pyarrow' failed." >&2
echo "[attest] pyarrow is required to verify workers_500k.parquet schema." >&2
echo "[attest] Install with: pip install pyarrow" >&2
exit 2
fi
DATE="${OVERRIDE_DATE:-$(date -u +%Y-%m-%d)}"
OUT_DIR="docs/attestations"
OUT="$OUT_DIR/BIPA_PRE_IDENTITYD_ATTESTATION_${DATE}.md"
mkdir -p "$OUT_DIR"
WORKERS_PARQUET="${WORKERS_PARQUET:-data/datasets/workers_500k.parquet}"
KB_DIR="${KB_DIR:-data/_kb}"
PATHWAY_STATE="${PATHWAY_STATE:-data/_pathway_memory/state.json}"
HEADSHOTS_MANIFEST="${HEADSHOTS_MANIFEST:-data/headshots/manifest.jsonl}"
PASS=0
FAIL=0
EVIDENCE=$(mktemp)
note() { echo "$1" >> "$EVIDENCE"; }
mark_pass() { PASS=$((PASS+1)); note " - PASS: $1"; }
mark_fail() { FAIL=$((FAIL+1)); note " - FAIL: $1"; }
# ── Check 1: workers_500k.parquet schema ────────────────────────────
note "## Check 1 — workers_500k.parquet schema (no biometric columns)"
note ""
note "**Source:** \`$WORKERS_PARQUET\`"
note ""
if [ ! -r "$WORKERS_PARQUET" ]; then
echo "[attest] FAIL: cannot read $WORKERS_PARQUET" >&2
rm -f "$EVIDENCE"
exit 2
fi
# Hash NAME + TYPE + nullability per column, not just names. A schema
# fingerprint over names alone would not invalidate if a column got
# repurposed (e.g. resume_text reused to hold base64 photo bytes under
# its existing name). Including types catches that class of evasion.
# (Caught 2026-05-03 opus scrum WARN on attestation:18.)
SCHEMA=$(python3 -c "
import sys, pyarrow.parquet as pq
schema = pq.read_schema('$WORKERS_PARQUET')
for f in schema:
print(f'{f.name}\t{f.type}\tnullable={f.nullable}')
" 2>&1)
# Bash assigns + propagates the substitution's exit through \$?.
# Verified: X=\$(false); echo \$? -> 1. opus 2026-05-03 BLOCK on this
# location was a false positive — the check IS the python3 exit gate.
if [ $? -ne 0 ]; then
echo "[attest] FAIL: schema read error: $SCHEMA" >&2
rm -f "$EVIDENCE"
exit 2
fi
SCHEMA_HASH=$(echo "$SCHEMA" | sha256sum | awk '{print $1}')
SCHEMA_LINES=$(echo "$SCHEMA" | wc -l)
note "**Schema columns** ($SCHEMA_LINES total):"
note ""
note '```'
note "$SCHEMA"
note '```'
note ""
note "**Schema SHA-256:** \`$SCHEMA_HASH\`"
note ""
# Forbidden column patterns (case-insensitive)
FORBIDDEN_COLS=$(echo "$SCHEMA" | grep -iE "^(photo|biometric|face|image)([_].*)?$" || true)
if [ -z "$FORBIDDEN_COLS" ]; then
mark_pass "no biometric / photo / face / image column present"
else
mark_fail "forbidden columns present: $FORBIDDEN_COLS"
fi
note ""
# ── Check 2: KB JSONL + pathway state — no base64 image / MIME ──────
note "## Check 2 — KB + pathway memory contain no biometric payloads"
note ""
note "**Sources scanned:**"
note "- \`$KB_DIR/*.jsonl\` (knowledge base)"
note "- \`$PATHWAY_STATE\` (pathway memory state)"
note ""
SCAN_PATHS=()
if [ -d "$KB_DIR" ]; then
while IFS= read -r f; do SCAN_PATHS+=("$f"); done < <(find "$KB_DIR" -maxdepth 2 -type f -name "*.jsonl")
fi
if [ -r "$PATHWAY_STATE" ]; then
SCAN_PATHS+=("$PATHWAY_STATE")
fi
# Forbidden patterns:
# data:image/ — explicit MIME embed
# "photo": — bare photo field
# "biometric" — field name
# "deepface_ — deepface output prefix
# /9j/[A-Za-z0-9+/]{40,} — JPEG base64 magic + length floor (false-positive guard)
# iVBORw0KGgo[A-Za-z0-9+/]{20,} — PNG base64 magic + length floor
PATTERN_FILE=$(mktemp)
cat > "$PATTERN_FILE" <<'PATTERNS'
data:image/
"photo"\s*:
"biometric"
"deepface_
/9j/[A-Za-z0-9+/=]{40,}
iVBORw0KGgo[A-Za-z0-9+/=]{20,}
PATTERNS
HITS=0
HIT_DETAIL=$(mktemp)
for path in "${SCAN_PATHS[@]}"; do
if grep -aHEf "$PATTERN_FILE" "$path" > "$HIT_DETAIL.tmp" 2>/dev/null; then
if [ -s "$HIT_DETAIL.tmp" ]; then
HITS=$((HITS + $(wc -l < "$HIT_DETAIL.tmp")))
cat "$HIT_DETAIL.tmp" >> "$HIT_DETAIL"
fi
fi
done
rm -f "$PATTERN_FILE" "$HIT_DETAIL.tmp"
note "**Files scanned:** ${#SCAN_PATHS[@]}"
note "**Forbidden-pattern hits:** $HITS"
note ""
if [ "$HITS" -eq 0 ]; then
mark_pass "no biometric payload patterns found in scanned files"
else
mark_fail "$HITS forbidden-pattern hits — see detail below"
note ""
note "### Detail (first 20 hits)"
note ""
note '```'
head -20 "$HIT_DETAIL" >> "$EVIDENCE"
note '```'
fi
rm -f "$HIT_DETAIL"
note ""
# ── Check 3: headshots manifest is synthetic-only ───────────────────
note "## Check 3 — Headshots manifest is synthetic-only"
note ""
note "**Source:** \`$HEADSHOTS_MANIFEST\`"
note ""
if [ ! -r "$HEADSHOTS_MANIFEST" ]; then
note "**SKIP** — manifest not present (no headshot UI deployed)."
note ""
mark_pass "no headshots manifest = no headshot data exists at all"
else
TOTAL_ROWS=$(wc -l < "$HEADSHOTS_MANIFEST")
# A row is non-synthetic if it lacks the synthetic markers (source: tag,
# archetype: tag, deterministic id pattern). The Phase 1.5 walk
# established that the synthetic face pool uses generated portraits
# with archetype tags. Anything else (real candidate upload) would
# be a Phase 1.6 violation.
NON_SYNTHETIC=$(grep -cE '"source"[[:space:]]*:[[:space:]]*"(real|candidate_upload|photo_upload)"' "$HEADSHOTS_MANIFEST" 2>/dev/null) || NON_SYNTHETIC=0
# Strip any newlines / whitespace defensively in case grep -c returned weirdly.
NON_SYNTHETIC=$(printf '%s' "$NON_SYNTHETIC" | tr -d '[:space:]')
: "${NON_SYNTHETIC:=0}"
note "**Total rows:** $TOTAL_ROWS"
note "**Rows tagged real/candidate_upload/photo_upload:** $NON_SYNTHETIC"
note ""
if [ "$NON_SYNTHETIC" = "0" ]; then
mark_pass "all $TOTAL_ROWS rows are synthetic (no real-candidate uploads)"
else
mark_fail "$NON_SYNTHETIC rows tagged as non-synthetic — investigate"
fi
fi
note ""
# ── Summary + final hash ────────────────────────────────────────────
TOTAL=$((PASS + FAIL))
note "## Summary"
note ""
note "**$PASS / $TOTAL** evidence checks pass."
note ""
if [ "$FAIL" -gt 0 ]; then
note "**Status: NOT READY FOR SIGNATURE** — at least one check failed. Resolve before counsel review."
note ""
fi
# Compute the evidence hash so any modification to the attestation
# document is detectable post-signature.
EVIDENCE_HASH=$(sha256sum "$EVIDENCE" | awk '{print $1}')
# ── Render final attestation document ───────────────────────────────
{
echo "# BIPA Pre-IdentityD Biometric Attestation"
echo
echo "**Date:** $DATE"
echo "**Spec:** docs/PHASE_1_6_BIPA_GATES.md §2"
echo "**Generator:** scripts/staffing/attest_pre_identityd_biometric_state.sh"
echo
echo "## Purpose"
echo
echo "This is a one-time defense artifact establishing that, as of"
echo "$DATE, no biometric identifiers or biometric information"
echo "from real candidates have been collected, processed, or stored"
echo "by the Lakehouse system. It is intended to be signed by J"
echo "(operator of record) and outside counsel, then anchored to a"
echo "tamper-evident store (filesystem with backups + version control)."
echo
echo "## Evidence"
echo
cat "$EVIDENCE"
echo
echo "---"
echo
echo "## Attestation"
echo
echo "I, the undersigned, attest that the above evidence accurately"
echo "reflects the state of the Lakehouse system as of $DATE."
echo "No biometric identifiers or biometric information from real"
echo "candidates have been collected, processed, or stored prior to"
echo "the deployment of the Phase 1.6 BIPA pre-launch gates."
echo
echo "**Evidence SHA-256:** \`$EVIDENCE_HASH\`"
echo
echo "---"
echo
echo "**Operator (J):** _______________________________ Date: __________"
echo
echo "**Outside counsel:** ___________________________ Date: __________"
echo
} > "$OUT"
rm -f "$EVIDENCE"
echo "[attest] $PASS / $TOTAL checks pass — attestation: $OUT"
echo "[attest] evidence SHA-256: $EVIDENCE_HASH"
[ "$FAIL" -eq 0 ]
+263
View File
@@ -0,0 +1,263 @@
#!/usr/bin/env bash
# biometric_destruction_report — monthly destruction event aggregation.
#
# Specification: docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §5.
# Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5.
#
# Why this exists: counsel and operations review need a periodic
# attestation that destructions have happened in a defensible cadence.
# This script produces an anonymized monthly report aggregating
# per-subject audit logs.
#
# Output is anonymized — counts, timings, scope/trigger breakdowns,
# and chain attestations. Candidate IDs are hashed (sha256-prefix) so
# the report can be shared with counsel without exposing identifiers.
#
# Usage:
# biometric_destruction_report.sh \
# [--month YYYY-MM] \
# [--audit-dir data/_catalog/subjects] \
# [--output reports/biometric/destruction_<month>.md]
#
# Defaults:
# --month — current UTC month (YYYY-MM)
# --audit-dir — data/_catalog/subjects
# --output — reports/biometric/destruction_<month>.md
#
# Exit codes:
# 0 — report written successfully (whether or not events were found)
# 1 — report written but with anomalies that need review
# 2 — script error (missing tools, unreadable audit dir)
set -uo pipefail
cd "$(dirname "$0")/../.."
MONTH=""
AUDIT_DIR="data/_catalog/subjects"
OUT=""
while [ "$#" -gt 0 ]; do
case "$1" in
--month) MONTH="$2"; shift 2 ;;
--audit-dir) AUDIT_DIR="$2"; shift 2 ;;
--output) OUT="$2"; shift 2 ;;
-h|--help)
sed -n '2,30p' "$0" | sed 's/^# \?//'
exit 0 ;;
*) echo "unknown flag: $1" >&2; exit 2 ;;
esac
done
# Default month = current UTC YYYY-MM. Validate format defensively
# so a malformed --month value (e.g. "May 2026") doesn't silently
# match nothing in the JSONL filter.
if [ -z "$MONTH" ]; then
MONTH=$(date -u +%Y-%m)
fi
if ! echo "$MONTH" | grep -qE '^[0-9]{4}-(0[1-9]|1[0-2])$'; then
echo "[report] FAIL: --month must be YYYY-MM, got '$MONTH'" >&2
exit 2
fi
if [ -z "$OUT" ]; then
OUT="reports/biometric/destruction_${MONTH}.md"
fi
# Dependency gates.
for cmd in jq sha256sum; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "[report] FAIL: required tool '$cmd' not found in PATH" >&2
exit 2
fi
done
if [ ! -d "$AUDIT_DIR" ]; then
echo "[report] FAIL: audit dir not found at $AUDIT_DIR" >&2
exit 2
fi
mkdir -p "$(dirname "$OUT")"
# Aggregator storage.
EVENTS=$(mktemp)
ANOMALIES=$(mktemp)
trap 'rm -f "$EVENTS" "$ANOMALIES"' EXIT
# Iterate every per-subject audit log under AUDIT_DIR. Each file is
# JSONL — one row per line. We extract erasure rows in the requested
# month + emit a normalized one-line record per event.
TOTAL_FILES=0
TOTAL_ROWS_SCANNED=0
SHARDS_WITH_EVENTS=0
for f in "$AUDIT_DIR"/*.audit.jsonl; do
[ -e "$f" ] || continue
TOTAL_FILES=$((TOTAL_FILES + 1))
# File-level row count (cheap).
ROWS=$(wc -l < "$f" 2>/dev/null || echo 0)
TOTAL_ROWS_SCANNED=$((TOTAL_ROWS_SCANNED + ROWS))
# Filter rows for the month + erasure kinds.
HAD_EVENT=0
while IFS= read -r line; do
[ -n "$line" ] || continue
KIND=$(printf '%s' "$line" | jq -r '.accessor.kind // ""' 2>/dev/null || echo "")
case "$KIND" in
biometric_erasure|full_erasure) ;;
*) continue ;;
esac
TS=$(printf '%s' "$line" | jq -r '.ts // ""' 2>/dev/null || echo "")
case "$TS" in
"${MONTH}-"*) ;; # only this month
*) continue ;;
esac
HAD_EVENT=1
CID=$(printf '%s' "$line" | jq -r '.candidate_id // ""' 2>/dev/null || echo "")
PURPOSE=$(printf '%s' "$line" | jq -r '.accessor.purpose // ""' 2>/dev/null || echo "")
RESULT=$(printf '%s' "$line" | jq -r '.result // ""' 2>/dev/null || echo "")
# accessor.purpose has shape "trigger=<name>;..." per biometric_endpoint
TRIGGER=$(printf '%s' "$PURPOSE" | sed -nE 's/.*trigger=([a-z_]+).*/\1/p')
[ -n "$TRIGGER" ] || TRIGGER="unknown"
# Hash candidate_id so the report stays anonymized.
CID_HASH=$(printf '%s' "$CID" | sha256sum | awk '{print substr($1,1,12)}')
# Anomaly: erasure row but result not in {erased, success}.
case "$RESULT" in
erased|success) ;;
*)
echo " - candidate_hash=$CID_HASH ts=$TS kind=$KIND result=$RESULT trigger=$TRIGGER (unexpected result)" >> "$ANOMALIES"
;;
esac
# Tab-separated event line: ts, kind, trigger, result, cid_hash
printf '%s\t%s\t%s\t%s\t%s\n' "$TS" "$KIND" "$TRIGGER" "$RESULT" "$CID_HASH" >> "$EVENTS"
done < "$f"
if [ "$HAD_EVENT" = "1" ]; then
SHARDS_WITH_EVENTS=$((SHARDS_WITH_EVENTS + 1))
fi
done
EVENT_COUNT=$(wc -l < "$EVENTS" 2>/dev/null || echo 0)
EVENT_COUNT=$(printf '%s' "$EVENT_COUNT" | tr -d '[:space:]')
: "${EVENT_COUNT:=0}"
# Compute breakdowns.
COUNT_BIOMETRIC_ONLY=0
COUNT_FULL=0
if [ "$EVENT_COUNT" != "0" ]; then
COUNT_BIOMETRIC_ONLY=$(awk -F '\t' '$2=="biometric_erasure"' "$EVENTS" | wc -l | tr -d '[:space:]')
COUNT_FULL=$(awk -F '\t' '$2=="full_erasure"' "$EVENTS" | wc -l | tr -d '[:space:]')
fi
ANOMALY_COUNT=$(wc -l < "$ANOMALIES" 2>/dev/null || echo 0)
ANOMALY_COUNT=$(printf '%s' "$ANOMALY_COUNT" | tr -d '[:space:]')
: "${ANOMALY_COUNT:=0}"
# Render the report.
GENERATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
{
echo "# Biometric Destruction Report — $MONTH"
echo
echo "**Generated:** $GENERATED_AT"
echo "**Audit dir scanned:** \`$AUDIT_DIR\`"
echo "**Spec:** docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §5"
echo "**Generator:** scripts/staffing/biometric_destruction_report.sh"
echo
echo "## Scope"
echo
echo "- **Subject audit shards scanned:** $TOTAL_FILES"
echo "- **Audit rows scanned (all kinds):** $TOTAL_ROWS_SCANNED"
echo "- **Shards containing $MONTH erasure events:** $SHARDS_WITH_EVENTS"
echo
echo "## Destruction events in $MONTH"
echo
echo "- **Total events:** $EVENT_COUNT"
echo "- **By scope:**"
echo " - \`biometric_erasure\` (BiometricCollection cleared, manifest retained): $COUNT_BIOMETRIC_ONLY"
echo " - \`full_erasure\` (manifest + biometric data cleared): $COUNT_FULL"
echo
if [ "$EVENT_COUNT" = "0" ]; then
echo "**No destruction events recorded for $MONTH.** This is correct"
echo "for a month with no retention expiries / withdrawal requests"
echo "/ RTBF requests / court orders."
echo
else
echo "### By trigger"
echo
echo "| Trigger | Count |"
echo "|---|---|"
awk -F '\t' '{print $3}' "$EVENTS" | sort | uniq -c | \
sort -rn | awk '{ printf("| %s | %d |\n", $2, $1); }'
echo
echo "### Event detail (anonymized)"
echo
echo "Candidate IDs are hashed (sha256-12-prefix) so this report can"
echo "be shared with outside counsel without exposing identifiers."
echo
echo "| ts | kind | trigger | result | candidate_hash |"
echo "|---|---|---|---|---|"
sort -k1,1 "$EVENTS" | awk -F '\t' '{
printf("| %s | %s | %s | %s | %s |\n", $1, $2, $3, $4, $5);
}'
echo
fi
if [ "$ANOMALY_COUNT" != "0" ]; then
echo "## Anomalies ($ANOMALY_COUNT)"
echo
echo "Events whose audit row deviates from expected shape (kind/result"
echo "mismatch, missing trigger, etc.). These do NOT necessarily mean"
echo "the destruction failed — the BIPA-load-bearing surface is the"
echo "audit chain, which still verifies cryptographically. They are"
echo "logged here so an operator can investigate and confirm."
echo
echo '```'
cat "$ANOMALIES"
echo '```'
echo
fi
echo "## Cryptographic attestation"
echo
echo "This report was produced by aggregating per-subject HMAC-chained"
echo "audit logs. The chain itself is the BIPA-defensible substrate;"
echo "this report is a derived view, not the chain of record. To verify"
echo "any individual event, run:"
echo
echo '```bash'
echo "./scripts/staffing/verify_biometric_erasure.sh <candidate_id>"
echo '```'
echo "(operator must un-hash the candidate ID through their own"
echo " operator log to perform spot-checks)."
echo
echo "**Cross-runtime parity:** the same audit logs are byte-identical"
echo "under Rust + Go (per scripts/cutover/parity/subject_audit_parity.sh)."
echo "If counsel needs cross-runtime attestation, that probe provides it."
echo
EVIDENCE_HASH=$(sha256sum "$EVENTS" 2>/dev/null | awk '{print $1}')
: "${EVIDENCE_HASH:=$(echo -n '' | sha256sum | awk '{print $1}')}"
echo "**Events SHA-256:** \`$EVIDENCE_HASH\`"
echo
echo "---"
echo
echo "**Operator (J):** _______________________________ Date: __________"
echo
} > "$OUT"
echo "[report] $EVENT_COUNT destruction events in $MONTH ($COUNT_BIOMETRIC_ONLY biometric_only, $COUNT_FULL full)"
echo "[report] anomalies: $ANOMALY_COUNT"
echo "[report] output: $OUT"
# Exit 1 if anomalies present (review needed) but report still written.
if [ "$ANOMALY_COUNT" != "0" ]; then
exit 1
fi
exit 0
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# bundle_counsel_packet — assemble the counsel-review packet tarball.
#
# Specification: docs/counsel/COUNSEL_REVIEW_PACKET_<DATE>.md §9.
#
# Why this exists: the cover note references a list of documents.
# Counsel needs them as a single transmittable artifact, with per-file
# integrity hashes so they can verify nothing changed in transit.
#
# Output:
# reports/counsel/counsel_packet_<DATE>.tar.gz
# reports/counsel/counsel_packet_<DATE>.manifest.txt (sha256 per file)
#
# Usage:
# bundle_counsel_packet.sh [--date YYYY-MM-DD]
#
# Exit codes:
# 0 — packet bundled successfully
# 1 — one or more referenced documents are missing
# 2 — script error (missing tools, write failure)
set -uo pipefail
cd "$(dirname "$0")/../.."
DATE="$(date -u +%Y-%m-%d)"
while [ "$#" -gt 0 ]; do
case "$1" in
--date) DATE="$2"; shift 2 ;;
-h|--help)
sed -n '2,20p' "$0" | sed 's/^# \?//'
exit 0 ;;
*) echo "unknown flag: $1" >&2; exit 2 ;;
esac
done
# Dependency gate.
for cmd in tar sha256sum; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "[bundle] FAIL: required tool '$cmd' not found in PATH" >&2
exit 2
fi
done
# Files in the packet. Order is the recommended counsel-review order
# from the cover note §6.
FILES=(
"docs/counsel/COUNSEL_REVIEW_PACKET_${DATE}.md"
"docs/policies/consent/biometric_retention_schedule_v1.md"
"docs/policies/consent/biometric_consent_template_v1.md"
"docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md"
"docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md"
"docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md"
"docs/specs/GATE_3B_DEEPFACE_DESIGN.md"
"docs/PHASE_1_6_BIPA_GATES.md"
)
# Verify all referenced files exist before opening the tarball.
MISSING=0
for f in "${FILES[@]}"; do
if [ ! -r "$f" ]; then
echo "[bundle] MISSING: $f" >&2
MISSING=$((MISSING + 1))
fi
done
if [ "$MISSING" -gt 0 ]; then
echo "[bundle] FAIL: $MISSING required documents missing — aborting" >&2
exit 1
fi
OUT_DIR="reports/counsel"
mkdir -p "$OUT_DIR"
TARBALL="$OUT_DIR/counsel_packet_${DATE}.tar.gz"
MANIFEST="$OUT_DIR/counsel_packet_${DATE}.manifest.txt"
# Build the manifest first — counsel uses this to verify integrity.
{
echo "# Counsel Packet Manifest — $DATE"
echo "# Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "# Each file is listed with its SHA-256 hash. To verify on receipt:"
echo "# tar xzf counsel_packet_${DATE}.tar.gz"
echo "# sha256sum -c counsel_packet_${DATE}.manifest.txt"
echo "# (re-format the lines below with two spaces between hash and path"
echo "# for sha256sum -c compatibility — sha256sum's strict format)"
echo
for f in "${FILES[@]}"; do
sha256sum "$f"
done
} > "$MANIFEST"
# Build the tarball — include the manifest itself.
tar -czf "$TARBALL" "${FILES[@]}" "$MANIFEST"
PACKET_HASH=$(sha256sum "$TARBALL" | awk '{print $1}')
echo "[bundle] packet: $TARBALL"
echo "[bundle] manifest: $MANIFEST"
echo "[bundle] tarball SHA-256: $PACKET_HASH"
echo "[bundle] files: ${#FILES[@]}"
+170
View File
@@ -0,0 +1,170 @@
#!/usr/bin/env bash
# seed_consent_version — hash a consent template + add to allowlist.
#
# Specification: docs/PHASE_1_6_BIPA_GATES.md (consent_versions
# allowlist) + docs/policies/consent/biometric_consent_template_v1.md
#
# Why this exists: when counsel returns a signed consent template,
# the operator needs to flip the gateway from permissive mode
# (any non-empty hash accepted, v1 compat) to strict mode (only
# accepted hashes). That requires:
# 1. Compute SHA-256 of the binding text (the markdown source)
# 2. Atomically write it into /etc/lakehouse/consent_versions.json
# 3. Restart the gateway so the in-memory allowlist re-reads
#
# This script does all three. Idempotent: re-running with the same
# template is a no-op (the hash is already in the allowlist).
#
# Usage:
# seed_consent_version.sh <path-to-consent-template.md> [--no-restart] [--label "v1 signed 2026-06-15 by Counsel-Name"]
#
# Environment:
# ALLOWLIST_FILE — default /etc/lakehouse/consent_versions.json
# GATEWAY_SERVICE — default lakehouse.service (used for restart)
#
# Exit codes:
# 0 — hash seeded; allowlist now contains it (whether new or already present)
# 1 — file present but allowlist write/restart failed
# 2 — script error (missing tools, missing input, bad path)
set -uo pipefail
if [ "$#" -lt 1 ]; then
cat >&2 <<'USAGE'
usage: seed_consent_version.sh <path-to-consent-template.md> [--no-restart] [--label "free-text"]
Computes SHA-256 of the markdown file and merges it into the
gateway's consent_versions allowlist. By default also restarts
the gateway so the new entry takes effect immediately.
USAGE
exit 2
fi
TEMPLATE_PATH="$1"
shift
DO_RESTART=1
LABEL=""
while [ "$#" -gt 0 ]; do
case "$1" in
--no-restart) DO_RESTART=0; shift ;;
--label) LABEL="$2"; shift 2 ;;
-h|--help) sed -n '2,20p' "$0" | sed 's/^# \?//'; exit 0 ;;
*) echo "[seed] unknown flag: $1" >&2; exit 2 ;;
esac
done
ALLOWLIST_FILE="${ALLOWLIST_FILE:-/etc/lakehouse/consent_versions.json}"
GATEWAY_SERVICE="${GATEWAY_SERVICE:-lakehouse.service}"
for cmd in sha256sum jq install; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "[seed] FAIL: required tool '$cmd' not found in PATH" >&2
exit 2
fi
done
if [ ! -r "$TEMPLATE_PATH" ]; then
echo "[seed] FAIL: cannot read $TEMPLATE_PATH" >&2
exit 2
fi
HASH=$(sha256sum "$TEMPLATE_PATH" | awk '{print $1}')
SIZE=$(stat -c '%s' "$TEMPLATE_PATH")
ABS=$(readlink -f "$TEMPLATE_PATH")
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
[ -n "$LABEL" ] || LABEL="auto-seeded by seed_consent_version.sh from $ABS"
echo "[seed] template: $ABS"
echo "[seed] size: $SIZE bytes"
echo "[seed] sha256: $HASH"
echo "[seed] allowlist: $ALLOWLIST_FILE"
echo "[seed] label: $LABEL"
echo "[seed] timestamp: $NOW"
# Build the new allowlist contents. Two cases:
# (a) file absent → create with single entry + audit metadata
# (b) file present → parse, check if hash already exists, add if not
TMP_OUT=$(mktemp)
trap 'rm -f "$TMP_OUT"' EXIT
if [ ! -e "$ALLOWLIST_FILE" ]; then
echo "[seed] allowlist file absent — creating fresh"
jq -n --arg hash "$HASH" --arg label "$LABEL" --arg ts "$NOW" --arg src "$ABS" '
{
"_doc": "docs/PHASE_1_6_BIPA_GATES.md consent_versions allowlist. Strict-mode gate for /biometric/subject/{id}/consent. Hashes are SHA-256 of the binding consent template markdown.",
versions: [$hash],
_meta: {
seeded_at: [{ ts: $ts, hash: $hash, label: $label, source_path: $src }]
}
}
' > "$TMP_OUT"
elif [ ! -r "$ALLOWLIST_FILE" ]; then
echo "[seed] FAIL: $ALLOWLIST_FILE exists but is not readable" >&2
exit 1
else
if ! jq empty "$ALLOWLIST_FILE" 2>/dev/null; then
echo "[seed] FAIL: $ALLOWLIST_FILE is not valid JSON" >&2
exit 1
fi
ALREADY=$(jq -r --arg h "$HASH" '(.versions // []) | index($h) | tostring' "$ALLOWLIST_FILE")
if [ "$ALREADY" != "null" ]; then
echo "[seed] hash already in allowlist (idempotent no-op for versions array)"
# Still bump the metadata so the audit trail captures the re-attempt.
jq --arg hash "$HASH" --arg label "$LABEL" --arg ts "$NOW" --arg src "$ABS" '
.versions = (.versions // []) |
._meta = ((._meta // {})
| .seeded_at = ((.seeded_at // []) + [{ ts: $ts, hash: $hash, label: ($label + " [reseed]"), source_path: $src }]))
' "$ALLOWLIST_FILE" > "$TMP_OUT"
else
echo "[seed] adding hash to existing allowlist (now $(($(jq -r '(.versions // []) | length' "$ALLOWLIST_FILE") + 1)) entries)"
jq --arg hash "$HASH" --arg label "$LABEL" --arg ts "$NOW" --arg src "$ABS" '
.versions = ((.versions // []) + [$hash]) |
._meta = ((._meta // {})
| .seeded_at = ((.seeded_at // []) + [{ ts: $ts, hash: $hash, label: $label, source_path: $src }]))
' "$ALLOWLIST_FILE" > "$TMP_OUT"
fi
fi
# Atomic move so a crashed write can't leave a half-file.
PARENT=$(dirname "$ALLOWLIST_FILE")
if [ ! -d "$PARENT" ]; then
echo "[seed] FAIL: parent dir $PARENT does not exist (run as root after sudo mkdir -p)" >&2
exit 1
fi
if ! sudo install -m 0644 -o root -g root "$TMP_OUT" "$ALLOWLIST_FILE"; then
echo "[seed] FAIL: install to $ALLOWLIST_FILE failed (run with sudo?)" >&2
exit 1
fi
echo "[seed] allowlist written to $ALLOWLIST_FILE"
echo "[seed] current allowlist contents:"
sudo cat "$ALLOWLIST_FILE" | jq '{ versions, latest_seed: ._meta.seeded_at[-1] }'
# Restart unless --no-restart.
if [ "$DO_RESTART" = "1" ]; then
echo
echo "[seed] restarting $GATEWAY_SERVICE so the new allowlist loads…"
if ! sudo systemctl restart "$GATEWAY_SERVICE"; then
echo "[seed] FAIL: systemctl restart $GATEWAY_SERVICE failed" >&2
exit 1
fi
sleep 2
if ! curl -sf http://localhost:3100/biometric/health >/dev/null; then
echo "[seed] WARN: /biometric/health not responding 200 after restart" >&2
else
echo "[seed] gateway healthy post-restart. allowlist live."
fi
else
echo "[seed] --no-restart set; gateway must be restarted manually for the change to load."
fi
echo
echo "[seed] To verify strict mode is now active, probe with a known-bad hash:"
echo
echo ' TOKEN=$(cat /etc/lakehouse/legal_audit.token)'
echo ' curl -sS -X POST http://localhost:3100/biometric/subject/WORKER-1/consent \'
echo ' -H "X-Lakehouse-Legal-Token: $TOKEN" \'
echo ' -H "Content-Type: application/json" \'
echo " -d '{\"consent_version_hash\":\"deadbeef\",\"consent_collection_method\":\"electronic_signature\",\"operator_of_record\":\"smoke\"}' | jq"
echo
echo ' Expected: HTTP 400 + error="consent_version_unknown"'
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env bash
# subject_timeline — pretty-print a subject's full BIPA lifecycle.
#
# Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §6
# + docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §3.
#
# Why this exists: when an operator gets a question like "what
# happened to candidate X's biometric data" — counsel inquiry,
# subject access request, or just routine triage — they need a
# one-shot view of the full lineage. /audit/subject/{id} returns
# the raw JSON; this wraps it in a human-readable timeline.
#
# Output:
# - Manifest summary (status, biometric status, retention_until)
# - Audit chain (chronological, kind + result + ts + hmac prefix)
# - Chain verification status (HMAC chain integrity)
# - On-disk photo presence + size if applicable
#
# Usage:
# subject_timeline.sh <candidate_id>
#
# Environment:
# GATEWAY_URL — default http://localhost:3100
# LEGAL_TOKEN_FILE — default /etc/lakehouse/legal_audit.token
# UPLOADS_ROOT — default data/biometric/uploads (relative to repo)
#
# Exit codes:
# 0 — timeline printed (chain may or may not verify; that's a fact, not a script error)
# 1 — chain verification failed (still prints, but flagged)
# 2 — script error (missing tools, network failure, bad token, subject not found)
set -uo pipefail
cd "$(dirname "$0")/../.."
if [ "$#" -lt 1 ]; then
echo "usage: subject_timeline.sh <candidate_id>" >&2
exit 2
fi
CANDIDATE_ID="$1"
GATEWAY_URL="${GATEWAY_URL:-http://localhost:3100}"
LEGAL_TOKEN_FILE="${LEGAL_TOKEN_FILE:-/etc/lakehouse/legal_audit.token}"
UPLOADS_ROOT="${UPLOADS_ROOT:-data/biometric/uploads}"
for cmd in curl jq; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "[timeline] FAIL: required tool '$cmd' not found" >&2
exit 2
fi
done
if [ ! -r "$LEGAL_TOKEN_FILE" ]; then
echo "[timeline] FAIL: cannot read legal token at $LEGAL_TOKEN_FILE" >&2
exit 2
fi
LEGAL_TOKEN=$(tr -d '[:space:]' < "$LEGAL_TOKEN_FILE")
[ -n "$LEGAL_TOKEN" ] || { echo "[timeline] FAIL: legal token file is empty" >&2; exit 2; }
# safe_id matches catalogd::biometric_endpoint::sanitize_for_path
SAFE_ID=$(printf '%s' "$CANDIDATE_ID" | sed 's/[^A-Za-z0-9_.\-]/_/g')
RESP_FILE=$(mktemp)
trap 'rm -f "$RESP_FILE"' EXIT
HTTP_CODE=$(curl -sS -o "$RESP_FILE" -w '%{http_code}' \
-H "X-Lakehouse-Legal-Token: $LEGAL_TOKEN" \
-H "Accept: application/json" \
"$GATEWAY_URL/audit/subject/$CANDIDATE_ID")
if [ "$HTTP_CODE" != "200" ]; then
echo "[timeline] FAIL: GET /audit/subject/$CANDIDATE_ID returned HTTP $HTTP_CODE" >&2
echo "[timeline] response:" >&2
cat "$RESP_FILE" >&2
echo >&2
exit 2
fi
# ── Header ──────────────────────────────────────────────────────────
printf '\n'
printf '═══ Subject Timeline — %s ═══\n' "$CANDIDATE_ID"
printf '\n'
# ── Manifest summary ───────────────────────────────────────────────
printf 'Manifest\n'
printf ' candidate_id : %s\n' "$(jq -r '.manifest.candidate_id' < "$RESP_FILE")"
printf ' subject status : %s\n' "$(jq -r '.manifest.status' < "$RESP_FILE")"
printf ' vertical : %s\n' "$(jq -r '.manifest.vertical' < "$RESP_FILE")"
printf ' general_pii : %s (until %s)\n' \
"$(jq -r '.manifest.consent.general_pii.status' < "$RESP_FILE")" \
"$(jq -r '.manifest.retention.general_pii_until' < "$RESP_FILE")"
printf ' biometric : %s\n' "$(jq -r '.manifest.consent.biometric.status' < "$RESP_FILE")"
RET=$(jq -r '.manifest.consent.biometric.retention_until // "—"' < "$RESP_FILE")
printf ' biometric retent. : %s\n' "$RET"
BC_PRESENT=$(jq -r '.manifest.biometric_collection != null' < "$RESP_FILE")
if [ "$BC_PRESENT" = "true" ]; then
printf ' photo data_path : %s\n' "$(jq -r '.manifest.biometric_collection.data_path' < "$RESP_FILE")"
printf ' photo template : %s\n' "$(jq -r '.manifest.biometric_collection.template_hash' < "$RESP_FILE")"
printf ' photo collected : %s\n' "$(jq -r '.manifest.biometric_collection.collected_at' < "$RESP_FILE")"
printf ' consent_ver_hash : %s\n' "$(jq -r '.manifest.biometric_collection.consent_version_hash' < "$RESP_FILE")"
fi
# ── On-disk photo state ────────────────────────────────────────────
printf '\nOn disk\n'
PHOTO_DIR="$UPLOADS_ROOT/$SAFE_ID"
if [ -d "$PHOTO_DIR" ]; then
COUNT=$(find "$PHOTO_DIR" -maxdepth 1 -type f 2>/dev/null | wc -l | tr -d '[:space:]')
printf ' uploads dir : %s (%s file(s))\n' "$PHOTO_DIR" "${COUNT:-0}"
if [ "${COUNT:-0}" != "0" ]; then
while IFS= read -r f; do
printf ' - %s (%s bytes)\n' "$f" "$(stat -c '%s' "$f" 2>/dev/null || echo '?')"
done < <(find "$PHOTO_DIR" -maxdepth 1 -type f 2>/dev/null)
fi
else
printf ' uploads dir : %s (absent)\n' "$PHOTO_DIR"
fi
# ── Audit chain ────────────────────────────────────────────────────
printf '\nAudit chain\n'
ROWS_TOTAL=$(jq -r '.audit_log.chain_rows_total' < "$RESP_FILE")
VERIFIED=$(jq -r '.audit_log.chain_verified' < "$RESP_FILE")
ROOT=$(jq -r '.audit_log.chain_root // "—"' < "$RESP_FILE")
ERROR=$(jq -r '.audit_log.chain_verification_error // ""' < "$RESP_FILE")
printf ' rows total : %s\n' "$ROWS_TOTAL"
printf ' verified : %s\n' "$VERIFIED"
printf ' chain root (last) : %s\n' "$ROOT"
if [ -n "$ERROR" ] && [ "$ERROR" != "null" ]; then
printf ' verification err : %s\n' "$ERROR"
fi
if [ "$ROWS_TOTAL" != "0" ]; then
printf '\n events (chronological):\n'
jq -r '
.audit_log.rows
| sort_by(.ts)
| .[]
| " \(.ts) | \(.accessor.kind | ascii_upcase) | result=\(.result) | hmac=\(.row_hmac[0:16])… | trace=\(.accessor.trace_id // "—")"
' < "$RESP_FILE"
fi
# ── Footer ─────────────────────────────────────────────────────────
printf '\n'
if [ "$VERIFIED" = "true" ]; then
printf 'Status: chain verified end-to-end.\n'
printf '\n'
exit 0
else
printf 'Status: CHAIN VERIFICATION FAILED. Investigate before quoting this timeline\n'
printf ' in any external response. Likely causes: post-rotation legacy chain\n'
printf ' (expected) or actual tampering (escalate to engineering + counsel).\n'
printf '\n'
exit 1
fi
+266
View File
@@ -0,0 +1,266 @@
#!/usr/bin/env bash
# verify_biometric_erasure — confirm that a biometric erasure completed cleanly.
#
# Specification: docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §3 step 3.
# Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5.
#
# Why this exists: when an operator runs the erasure curl call against
# /biometric/subject/{id}/erase, they need a defensible artifact proving
# destruction completed. This script produces that artifact by checking
# four things:
#
# 1. SubjectManifest.biometric_collection is null (catalogd cleared the row)
# 2. data/biometric/uploads/<safe_id>/ is empty or absent (photo file gone)
# 3. Most recent audit row has accessor.kind in {biometric_erasure, full_erasure}
# AND result is "erased" or "success" (the chain logged the erasure intent)
# 4. audit_log.chain_verified is true (HMAC chain still intact end-to-end)
#
# All four must pass for an operator to mark the destruction complete.
#
# Usage:
# verify_biometric_erasure.sh <candidate_id> [--from ISO] [--to ISO]
#
# Environment:
# GATEWAY_URL — default http://localhost:3100
# LEGAL_TOKEN_FILE — default /etc/lakehouse/legal_audit.token
# UPLOADS_ROOT — default data/biometric/uploads (relative to repo root)
# OUT_DIR — default reports/biometric (where the verification report lands)
#
# Exit codes:
# 0 — all four checks pass; erasure verified
# 1 — one or more checks failed; do NOT mark destruction complete; escalate
# 2 — script error (missing tools, network failure, bad token)
set -uo pipefail
cd "$(dirname "$0")/../.."
if [ "$#" -lt 1 ]; then
echo "usage: verify_biometric_erasure.sh <candidate_id> [--from ISO] [--to ISO]" >&2
exit 2
fi
CANDIDATE_ID="$1"
shift
FROM=""
TO=""
while [ "$#" -gt 0 ]; do
case "$1" in
--from) FROM="$2"; shift 2 ;;
--to) TO="$2"; shift 2 ;;
*) echo "unknown flag: $1" >&2; exit 2 ;;
esac
done
GATEWAY_URL="${GATEWAY_URL:-http://localhost:3100}"
LEGAL_TOKEN_FILE="${LEGAL_TOKEN_FILE:-/etc/lakehouse/legal_audit.token}"
UPLOADS_ROOT="${UPLOADS_ROOT:-data/biometric/uploads}"
OUT_DIR="${OUT_DIR:-reports/biometric}"
# Dependency gates — fail fast with clear errors rather than producing
# a misleading "evidence" file from missing tools.
for cmd in curl jq sha256sum; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "[verify] FAIL: required tool '$cmd' not found in PATH" >&2
exit 2
fi
done
if [ ! -r "$LEGAL_TOKEN_FILE" ]; then
echo "[verify] FAIL: cannot read legal token at $LEGAL_TOKEN_FILE" >&2
echo "[verify] This script requires legal-tier auth to query /audit/subject/." >&2
exit 2
fi
LEGAL_TOKEN=$(tr -d '[:space:]' < "$LEGAL_TOKEN_FILE")
if [ -z "$LEGAL_TOKEN" ]; then
echo "[verify] FAIL: legal token file is empty" >&2
exit 2
fi
# safe_id matches catalogd::biometric_endpoint::sanitize_for_path:
# any non-[A-Za-z0-9_.-] char is replaced with underscore.
SAFE_ID=$(printf '%s' "$CANDIDATE_ID" | sed 's/[^A-Za-z0-9_.\-]/_/g')
mkdir -p "$OUT_DIR"
DATE=$(date -u +%Y-%m-%dT%H-%M-%SZ)
OUT="$OUT_DIR/erasure_verify_${SAFE_ID}_${DATE}.md"
EVIDENCE=$(mktemp)
trap 'rm -f "$EVIDENCE"' EXIT
PASS=0
FAIL=0
note() { echo "$1" >> "$EVIDENCE"; }
mark_pass() { PASS=$((PASS+1)); note " - PASS: $1"; }
mark_fail() { FAIL=$((FAIL+1)); note " - FAIL: $1"; }
note "## Verification target"
note ""
note "- **candidate_id:** \`$CANDIDATE_ID\`"
note "- **safe_id (filesystem):** \`$SAFE_ID\`"
note "- **gateway:** \`$GATEWAY_URL\`"
note "- **uploads root:** \`$UPLOADS_ROOT\`"
note "- **window:** ${FROM:-unbounded} → ${TO:-unbounded}"
note ""
# ── Fetch the audit response ────────────────────────────────────────
QUERY=""
if [ -n "$FROM" ]; then QUERY="from=$FROM"; fi
if [ -n "$TO" ]; then
if [ -n "$QUERY" ]; then QUERY="${QUERY}&to=$TO"; else QUERY="to=$TO"; fi
fi
URL="$GATEWAY_URL/audit/subject/$CANDIDATE_ID"
if [ -n "$QUERY" ]; then URL="$URL?$QUERY"; fi
RESP_FILE=$(mktemp)
HTTP_CODE=$(curl -sS -o "$RESP_FILE" -w '%{http_code}' \
-H "X-Lakehouse-Legal-Token: $LEGAL_TOKEN" \
-H "Accept: application/json" \
"$URL" 2>&1) || HTTP_CODE="000"
if [ "$HTTP_CODE" != "200" ]; then
echo "[verify] FAIL: GET $URL returned HTTP $HTTP_CODE" >&2
echo "[verify] response head:" >&2
head -c 500 "$RESP_FILE" >&2
echo >&2
rm -f "$RESP_FILE"
exit 2
fi
# Schema sanity — refuse to evaluate against an unrecognized response shape.
SCHEMA=$(jq -r '.schema // ""' < "$RESP_FILE")
if [ "$SCHEMA" != "subject_audit_response.v1" ]; then
echo "[verify] FAIL: unexpected response schema '$SCHEMA' (want subject_audit_response.v1)" >&2
rm -f "$RESP_FILE"
exit 2
fi
# ── Check 1: manifest.biometric_collection is null ──────────────────
note "## Check 1 — Subject manifest biometric_collection is null"
note ""
BIO_COLL=$(jq -c '.manifest.biometric_collection // null' < "$RESP_FILE")
note "**manifest.biometric_collection:** \`$BIO_COLL\`"
note ""
if [ "$BIO_COLL" = "null" ]; then
mark_pass "biometric_collection field is null on the subject manifest"
else
mark_fail "biometric_collection is still populated — erasure incomplete"
fi
note ""
# ── Check 2: filesystem uploads dir is empty/absent ─────────────────
note "## Check 2 — Quarantined upload directory empty or absent"
note ""
UPLOAD_DIR="$UPLOADS_ROOT/$SAFE_ID"
note "**path:** \`$UPLOAD_DIR\`"
if [ ! -e "$UPLOAD_DIR" ]; then
note "**state:** absent (directory was removed during erasure or never existed)"
note ""
mark_pass "upload directory is absent"
elif [ ! -d "$UPLOAD_DIR" ]; then
note "**state:** path exists but is not a directory — investigate"
note ""
mark_fail "upload path exists and is not a directory: $UPLOAD_DIR"
else
REMAINING=$(find "$UPLOAD_DIR" -maxdepth 1 -mindepth 1 2>/dev/null | wc -l | tr -d '[:space:]')
: "${REMAINING:=0}"
note "**state:** directory exists with $REMAINING remaining entries"
note ""
if [ "$REMAINING" = "0" ]; then
mark_pass "upload directory is empty (no residual photo files)"
else
mark_fail "$REMAINING file(s) remain under $UPLOAD_DIR — must be unlinked"
note "### Residual files"
note ""
note '```'
find "$UPLOAD_DIR" -maxdepth 2 >> "$EVIDENCE"
note '```'
note ""
fi
fi
# ── Check 3: most recent audit row reflects erasure ─────────────────
note "## Check 3 — Audit log records the erasure event"
note ""
ROW_COUNT=$(jq '.audit_log.rows | length' < "$RESP_FILE")
note "**rows in window:** $ROW_COUNT"
if [ "$ROW_COUNT" = "0" ]; then
mark_fail "no audit rows in the requested window — erasure should have appended one"
note ""
else
LAST_KIND=$(jq -r '.audit_log.rows | last | .accessor.kind // ""' < "$RESP_FILE")
LAST_RESULT=$(jq -r '.audit_log.rows | last | .result // ""' < "$RESP_FILE")
LAST_TS=$(jq -r '.audit_log.rows | last | .ts // ""' < "$RESP_FILE")
note "**last row:** ts=\`$LAST_TS\` accessor.kind=\`$LAST_KIND\` result=\`$LAST_RESULT\`"
note ""
case "$LAST_KIND" in
biometric_erasure|full_erasure)
case "$LAST_RESULT" in
erased|success)
mark_pass "last audit row is an erasure event ($LAST_KIND/$LAST_RESULT)"
;;
*)
mark_fail "last row kind is $LAST_KIND but result is '$LAST_RESULT' (expected erased/success)"
;;
esac
;;
*)
mark_fail "last audit row accessor.kind is '$LAST_KIND' (expected biometric_erasure or full_erasure)"
;;
esac
fi
note ""
# ── Check 4: HMAC chain verifies end-to-end ─────────────────────────
note "## Check 4 — HMAC chain integrity"
note ""
CHAIN_VERIFIED=$(jq -r '.audit_log.chain_verified' < "$RESP_FILE")
CHAIN_ROOT=$(jq -r '.audit_log.chain_root // ""' < "$RESP_FILE")
CHAIN_ROWS=$(jq -r '.audit_log.chain_rows_total // 0' < "$RESP_FILE")
CHAIN_ERR=$(jq -r '.audit_log.chain_verification_error // ""' < "$RESP_FILE")
note "**chain_verified:** \`$CHAIN_VERIFIED\`"
note "**chain_rows_total:** $CHAIN_ROWS"
note "**chain_root:** \`$CHAIN_ROOT\`"
if [ -n "$CHAIN_ERR" ]; then
note "**chain_verification_error:** \`$CHAIN_ERR\`"
fi
note ""
if [ "$CHAIN_VERIFIED" = "true" ]; then
mark_pass "chain verifies end-to-end ($CHAIN_ROWS rows)"
else
mark_fail "chain integrity broken — destruction is NOT defensible until investigated"
fi
note ""
# ── Render report ───────────────────────────────────────────────────
TOTAL=$((PASS + FAIL))
note "## Summary"
note ""
note "**$PASS / $TOTAL** verification checks pass."
note ""
if [ "$FAIL" -gt 0 ]; then
note "**Status: ERASURE NOT VERIFIED.** Do NOT mark destruction complete. Escalate to engineering before responding to candidate / counsel."
note ""
fi
# Hash response body so the report has a tamper-evident anchor.
RESP_HASH=$(sha256sum "$RESP_FILE" | awk '{print $1}')
EVIDENCE_HASH=$(sha256sum "$EVIDENCE" | awk '{print $1}')
{
echo "# Biometric Erasure Verification — $CANDIDATE_ID"
echo
echo "**Date:** $DATE"
echo "**Spec:** docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §3 step 3"
echo "**Generator:** scripts/staffing/verify_biometric_erasure.sh"
echo
cat "$EVIDENCE"
echo "---"
echo
echo "**Audit response SHA-256:** \`$RESP_HASH\`"
echo "**Evidence summary SHA-256:** \`$EVIDENCE_HASH\`"
echo
} > "$OUT"
rm -f "$RESP_FILE"
echo "[verify] $PASS / $TOTAL checks pass — report: $OUT"
echo "[verify] response hash: $RESP_HASH"
[ "$FAIL" -eq 0 ]
Executable
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env bash
# scrum — find gaps in current work-in-progress, push to KB.
#
# Usage:
# ./scrum auto-bundle current diff vs origin/main, auto-label
# ./scrum my_label same, with explicit label
# ./scrum --staged bundle staged-only diff (pre-commit check)
# ./scrum --since=COMMIT bundle from a specific commit
#
# Output:
# findings → data/_kb/scrum_findings.jsonl (one row per scrum run, KB-queryable)
# verdicts → reports/scrum/_evidence/$(date +%Y-%m-%d)/verdicts/<label>_*.md
#
# This is a TOOL J runs to find gaps. Findings flow to the KB. The KB
# informs how WE work on the code. Findings DO NOT auto-flow into
# architecture or design docs — that's J's call after reading them.
#
# Cloud models are used here BY DESIGN (3-lineage cross-review needs
# distinct training corpora). This is dev tooling, not the runtime
# hot path — PRD line 70 applies to customer requests, not to J's
# dev tools.
set -euo pipefail
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
LABEL=""
DIFF_MODE="branch"
SINCE_COMMIT=""
while [ $# -gt 0 ]; do
case "$1" in
--staged) DIFF_MODE="staged"; shift ;;
--since=*) DIFF_MODE="since"; SINCE_COMMIT="${1#--since=}"; shift ;;
-h|--help) sed -n '1,18p' "$0"; exit 0 ;;
*) LABEL="$1"; shift ;;
esac
done
[ -z "$LABEL" ] && LABEL="scrum_$(date +%Y%m%d_%H%M%S)"
# Build the diff
DIFF_FILE=$(mktemp -t "scrum_${LABEL}.XXXXXX.diff")
case "$DIFF_MODE" in
branch) git diff origin/main...HEAD > "$DIFF_FILE" 2>/dev/null || git diff main...HEAD > "$DIFF_FILE" ;;
staged) git diff --staged > "$DIFF_FILE" ;;
since) git diff "$SINCE_COMMIT"...HEAD > "$DIFF_FILE" ;;
esac
DIFF_SIZE=$(wc -c < "$DIFF_FILE")
DIFF_LINES=$(wc -l < "$DIFF_FILE")
if [ "$DIFF_SIZE" -lt 50 ]; then
echo "[scrum] no diff to review (mode=$DIFF_MODE)"
rm -f "$DIFF_FILE"
exit 0
fi
if [ "$DIFF_SIZE" -gt 100000 ]; then
echo "[scrum] diff is ${DIFF_SIZE} bytes (>100KB) — kimi/qwen will give up"
echo " split into smaller bundles by file/component, then re-run"
echo " per-component diffs are typically <60KB"
rm -f "$DIFF_FILE"
exit 1
fi
echo "[scrum] $LABEL — $DIFF_LINES lines / $DIFF_SIZE bytes — running 3 reviewers"
# Locate the underlying scrum_review.sh (lives in golangLAKEHOUSE)
SCRUM_REVIEW=""
for path in \
"/home/profit/golangLAKEHOUSE/scripts/scrum_review.sh" \
"$(git rev-parse --show-toplevel)/scripts/scrum_review.sh" \
"../golangLAKEHOUSE/scripts/scrum_review.sh"
do
if [ -f "$path" ]; then SCRUM_REVIEW="$path"; break; fi
done
if [ -z "$SCRUM_REVIEW" ]; then
echo "[scrum] could not locate scrum_review.sh"
echo " expected at /home/profit/golangLAKEHOUSE/scripts/scrum_review.sh"
rm -f "$DIFF_FILE"
exit 1
fi
# Default to the live Go gateway on :4110 (chatd-routed multi-provider).
# Set LH_GATEWAY=http://127.0.0.1:3100 to use the Rust gateway instead.
export LH_GATEWAY="${LH_GATEWAY:-http://127.0.0.1:4110}"
# Run from the scrum_review.sh directory so its `cd "$(dirname $0)/.."`
# works correctly (lands in golangLAKEHOUSE root, where reports/ lives).
SCRUM_DIR=$(dirname "$SCRUM_REVIEW")
SCRUM_REPO=$(dirname "$SCRUM_DIR")
# Run the underlying review. We treat ANY tally-file produced as a usable
# result, even if scrum_review.sh exits non-zero on its own aggregation
# step (its tally-counting can mis-fire while the verdicts themselves are
# fine — the per-reviewer markdown files are what matter).
( cd "$SCRUM_REPO" && bash "$SCRUM_REVIEW" "$DIFF_FILE" "$LABEL" ) || true
TALLY="$SCRUM_REPO/reports/scrum/_evidence/$(date +%Y-%m-%d)/verdicts/${LABEL}_tally.md"
if [ ! -f "$TALLY" ]; then
# Build a minimal tally from the per-reviewer files if scrum_review.sh
# didn't produce one. Lets findings still flow to the KB.
VERDICTS_DIR="$SCRUM_REPO/reports/scrum/_evidence/$(date +%Y-%m-%d)/verdicts"
if ls "$VERDICTS_DIR/${LABEL}_"*.md >/dev/null 2>&1; then
{
echo "# Convergence tally — $LABEL (auto-rebuilt by ./scrum wrapper)"
echo
echo "scrum_review.sh did not emit a tally; per-reviewer verdicts:"
echo
for f in "$VERDICTS_DIR/${LABEL}_"*.md; do
echo "- $(basename "$f")"
done
} > "$TALLY"
else
echo "[scrum] no per-reviewer verdicts produced — bailing"
rm -f "$DIFF_FILE"
exit 1
fi
fi
# Push to KB — one JSONL row per scrum run, queryable via DuckDB or jq
KB_DIR="$(git rev-parse --show-toplevel)/data/_kb"
mkdir -p "$KB_DIR"
KB_FILE="$KB_DIR/scrum_findings.jsonl"
# Extract finding count + convergent count from the tally.
# Tally row shape: `| <reviewers> | <severity> | <where> | <hits> |`
# Convergent rows have "+" in the Reviewers (first) column.
FINDINGS=$(awk -F'|' '/^\| / && NF>=5 && $2 !~ /^[ -]*$/ && $2 !~ /Reviewers/ {n++} END{print n+0}' "$TALLY" 2>/dev/null)
CONVERGENT=$(awk -F'|' '/^\| / && NF>=5 && $2 ~ /\+/ {n++} END{print n+0}' "$TALLY" 2>/dev/null)
[ -z "$FINDINGS" ] && FINDINGS=0
[ -z "$CONVERGENT" ] && CONVERGENT=0
# Build a single JSONL row
python3 <<PYEOF >> "$KB_FILE"
import json, sys, datetime
from pathlib import Path
tally = Path("$TALLY").read_text()
row = {
"schema": "scrum_finding.v1",
"ts": datetime.datetime.now(datetime.UTC).isoformat().replace("+00:00", "Z"),
"label": "$LABEL",
"diff_mode": "$DIFF_MODE",
"diff_bytes": $DIFF_SIZE,
"diff_lines": $DIFF_LINES,
"findings_total": $FINDINGS,
"findings_convergent": $CONVERGENT,
"tally_path": "$TALLY",
"tally_excerpt": tally[:2000],
"branch": "$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)",
"head_sha": "$(git rev-parse HEAD 2>/dev/null || echo unknown)",
}
print(json.dumps(row))
PYEOF
rm -f "$DIFF_FILE"
echo ""
echo "──────────────────────────────────────────────────────────────────"
echo "[scrum] $LABEL: $FINDINGS findings ($CONVERGENT convergent)"
echo "[scrum] tally: $TALLY"
echo "[scrum] verdicts: $SCRUM_REPO/reports/scrum/_evidence/$(date +%Y-%m-%d)/verdicts/"
echo "[scrum] KB: $KB_FILE"
echo ""
echo "[scrum] read tally: cat '$TALLY'"
echo "[scrum] query KB: jq -c 'select(.label == \"$LABEL\")' '$KB_FILE'"
-214
View File
@@ -1,214 +0,0 @@
#!/usr/bin/env bun
// Autonomous scrum loop — wraps scrum_master_pipeline.ts + scrum_applier.ts
// in a goal-driven retry loop. Observer is POSTed an iteration summary at
// every boundary so it can build meta-commentary outside the loop's epistemic
// scope.
//
// Usage:
// LOOP_TARGETS="crates/a/src/x.rs,crates/b/src/y.rs" \
// LOOP_MAX_ITERS=5 \
// LOOP_PUSH=1 \
// bun run tests/real-world/autonomous_loop.ts
//
// Stop conditions: max_iters reached OR 2 consecutive iters with 0 commits.
import { spawn } from "node:child_process";
import { appendFile, readFile } from "node:fs/promises";
import { existsSync } from "node:fs";
const REPO = "/home/profit/lakehouse";
const OBSERVER = process.env.LOOP_OBSERVER ?? "http://localhost:3800";
const BRANCH = process.env.LOOP_BRANCH ?? "scrum/auto-apply-19814";
const MAX_ITERS = Number(process.env.LOOP_MAX_ITERS ?? 3);
const PUSH = process.env.LOOP_PUSH === "1";
const MIN_CONF = process.env.LOOP_MIN_CONF ?? "85";
// Optional override — when unset, let scrum_applier.ts use ITS default
// (currently x-ai/grok-4.1-fast on openrouter). The prior hardcoded
// qwen3-coder:480b default was clobbering the applier's own default
// and forcing every iter to hit the throttled ollama_cloud account.
const APPLIER_MODEL = process.env.LOOP_APPLIER_MODEL;
const APPLIER_PROVIDER = process.env.LOOP_APPLIER_PROVIDER;
const TARGETS = (process.env.LOOP_TARGETS ?? "crates/queryd/src/service.rs,crates/gateway/src/main.rs,crates/gateway/src/v1/mod.rs")
.split(",").map(s => s.trim()).filter(Boolean);
const FORENSIC = process.env.LH_SCRUM_FORENSIC ?? `${REPO}/docs/SCRUM_FORENSIC_PROMPT.md`;
const PROPOSAL = process.env.LH_SCRUM_PROPOSAL ?? `${REPO}/docs/SCRUM_FIX_WAVE.md`;
const LOOP_ID = `loop_${Date.now().toString(36)}`;
const JOURNAL = `${REPO}/data/_kb/autonomous_loops.jsonl`;
interface IterResult {
iter: number;
scrum_reviews_added: number;
applier_outcomes: Record<string, number>;
commits_landed: number;
commit_shas: string[];
build_status: "green" | "red" | "unknown";
duration_ms: number;
}
function log(msg: string) {
const ts = new Date().toISOString().slice(11, 19);
console.log(`[loop ${LOOP_ID} ${ts}] ${msg}`);
}
function runCmd(cmd: string, args: string[], env: Record<string, string> = {}): Promise<{ code: number; stdout: string; stderr: string }> {
return new Promise((resolve) => {
const child = spawn(cmd, args, { cwd: REPO, env: { ...process.env, ...env } });
let stdout = "", stderr = "";
child.stdout.on("data", (d) => { stdout += d; process.stdout.write(d); });
child.stderr.on("data", (d) => { stderr += d; process.stderr.write(d); });
child.on("close", (code) => resolve({ code: code ?? -1, stdout, stderr }));
});
}
async function countLines(path: string): Promise<number> {
if (!existsSync(path)) return 0;
const text = await readFile(path, "utf8");
return text.split("\n").filter(Boolean).length;
}
async function gitHeadSha(): Promise<string> {
const r = await runCmd("git", ["rev-parse", "HEAD"]);
return r.stdout.trim();
}
async function commitsSince(baseSha: string): Promise<string[]> {
const r = await runCmd("git", ["log", "--oneline", `${baseSha}..HEAD`]);
return r.stdout.trim().split("\n").filter(Boolean);
}
async function cargoCheckGreen(): Promise<boolean> {
log("cargo check --workspace …");
const r = await runCmd("cargo", ["check", "--workspace", "--quiet"]);
return r.code === 0;
}
async function postObserver(payload: object) {
try {
const r = await fetch(`${OBSERVER}/event`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(5000),
});
if (!r.ok) log(`observer POST returned ${r.status}`);
} catch (e: any) {
log(`observer POST failed: ${e.message}`);
}
}
async function runIter(iter: number, baseSha: string): Promise<IterResult> {
const t0 = Date.now();
log(`══ iter ${iter} start (base ${baseSha.slice(0, 8)}) targets=${TARGETS.length}`);
const reviewsBefore = await countLines(`${REPO}/data/_kb/scrum_reviews.jsonl`);
const applyBefore = await countLines(`${REPO}/data/_kb/auto_apply.jsonl`);
log(`scrum_master_pipeline.ts → ${TARGETS.length} files`);
await runCmd("bun", ["run", "tests/real-world/scrum_master_pipeline.ts"], {
LH_SCRUM_FILES: TARGETS.join(","),
LH_SCRUM_FORENSIC: FORENSIC,
LH_SCRUM_PROPOSAL: PROPOSAL,
});
log(`scrum_applier.ts COMMIT=1 MIN_CONF=${MIN_CONF} files=${TARGETS.length}`);
// Only forward model/provider when explicitly overridden — otherwise
// let scrum_applier.ts use its own defaults (Grok 4.1 fast on openrouter).
const applierEnv: Record<string, string> = {
LH_APPLIER_COMMIT: "1",
LH_APPLIER_MIN_CONF: MIN_CONF,
LH_APPLIER_MAX_FILES: String(TARGETS.length),
LH_APPLIER_BRANCH: BRANCH,
// Constrain applier to THIS iter's targets so it patches what we
// just reviewed instead of the highest-confidence file from history.
LH_APPLIER_FILES: TARGETS.join(","),
};
if (APPLIER_MODEL) applierEnv.LH_APPLIER_MODEL = APPLIER_MODEL;
if (APPLIER_PROVIDER) applierEnv.LH_APPLIER_PROVIDER = APPLIER_PROVIDER;
await runCmd("bun", ["run", "tests/real-world/scrum_applier.ts"], applierEnv);
const reviewsAfter = await countLines(`${REPO}/data/_kb/scrum_reviews.jsonl`);
const applyAfterText = existsSync(`${REPO}/data/_kb/auto_apply.jsonl`)
? await readFile(`${REPO}/data/_kb/auto_apply.jsonl`, "utf8")
: "";
const applyRows = applyAfterText.split("\n").filter(Boolean).slice(applyBefore);
const outcomes: Record<string, number> = {};
for (const line of applyRows) {
try {
const o = JSON.parse(line);
outcomes[o.action ?? "?"] = (outcomes[o.action ?? "?"] ?? 0) + 1;
} catch { /* skip malformed */ }
}
const commitShas = await commitsSince(baseSha);
const buildStatus = commitShas.length > 0 ? (await cargoCheckGreen() ? "green" : "red") : "unknown";
const result: IterResult = {
iter,
scrum_reviews_added: reviewsAfter - reviewsBefore,
applier_outcomes: outcomes,
commits_landed: commitShas.length,
commit_shas: commitShas.map(s => s.split(" ")[0]),
build_status: buildStatus,
duration_ms: Date.now() - t0,
};
log(`iter ${iter} done — reviews+${result.scrum_reviews_added} commits=${result.commits_landed} build=${buildStatus} (${(result.duration_ms / 1000).toFixed(1)}s)`);
await postObserver({
source: "autonomous_loop",
loop_id: LOOP_ID,
event_kind: "iteration_complete",
iter,
targets: TARGETS,
success: buildStatus !== "red",
scrum_reviews_added: result.scrum_reviews_added,
applier_outcomes: result.applier_outcomes,
commits_landed: result.commits_landed,
commit_shas: result.commit_shas,
build_status: buildStatus,
duration_ms: result.duration_ms,
ts: new Date().toISOString(),
});
await appendFile(JOURNAL, JSON.stringify({ loop_id: LOOP_ID, ...result, ts: new Date().toISOString() }) + "\n");
return result;
}
async function main() {
log(`autonomous loop starting · branch=${BRANCH} max_iters=${MAX_ITERS} push=${PUSH}`);
log(`targets: ${TARGETS.join(", ")}`);
const branchR = await runCmd("git", ["branch", "--show-current"]);
if (branchR.stdout.trim() !== BRANCH) {
log(`ERROR: on branch ${branchR.stdout.trim()}, expected ${BRANCH}. Refusing to run.`);
process.exit(1);
}
let consecutiveZero = 0;
for (let iter = 1; iter <= MAX_ITERS; iter++) {
const baseSha = await gitHeadSha();
const result = await runIter(iter, baseSha);
if (PUSH && result.commits_landed > 0) {
log(`git push origin ${BRANCH}`);
const pushR = await runCmd("git", ["push", "origin", BRANCH]);
if (pushR.code !== 0) log(`push failed (continuing): ${pushR.stderr.slice(0, 200)}`);
}
consecutiveZero = result.commits_landed === 0 ? consecutiveZero + 1 : 0;
if (consecutiveZero >= 2) {
log(`STOP: 2 consecutive iters with 0 commits. Loop converged or stuck.`);
break;
}
}
log(`loop ${LOOP_ID} complete. Journal: ${JOURNAL}`);
}
main().catch((e) => {
log(`FATAL: ${e.message}`);
process.exit(1);
});
@@ -1,224 +0,0 @@
// consensus_reducer_design.ts — N=3 design consultation.
//
// J's ask: enhance the tree-split reducer to preserve FULL backtrack-able
// context (endpoints tried, attempt count per model in the ladder, KB
// sources retrieved, context7 bridge hits, MCP observer signals, audit
// verdicts) instead of collapsing to a summary. Then index the full
// context through our existing vectord matrix indexing (HNSW + Lance +
// playbook_memory) so successful pathways become hot-swappable — the
// system asks "what did we try, what worked, in what order" for a
// similar task class and gets a ranked playbook back.
//
// Before building, consult three diverse models and print their proposals
// side-by-side so we can pick the convergent design.
const GATEWAY = "http://localhost:3100";
const DESIGN_BRIEF = `
# Context — Lakehouse signal→commit loop
We run 6x scrum-master iterations that audit Rust crates for PRD
alignment, produce findings + confidence, and feed an auto-applier that
lands small mechanical commits through a cargo-green-and-warning-stable
gate. Key components:
- \`tests/real-world/scrum_master_pipeline.ts\` — orchestrator. 9-rung
model LADDER (kimi-k2:1t → qwen3-coder:480b → deepseek → mistral-large
→ gpt-oss:120b → qwen3.5:397b → openrouter free rescues → local
qwen3.5:latest). Each target file retrieves 5 PRD chunks + 5
proposal-doc chunks via vectord RAG, tree-splits large files into 3.5K
shards, asks each rung in order, accepts first response passing
structural checks.
- \`mcp-server/observer.ts\` — receives scrum \`/event\` emissions
(file, verdict, critical_failures_count, gradient_tier, attempts,
reviewer_model, tree_split_fired). Escalates failure clusters to LLM
Team by POSTing to /v1/chat with qwen3-coder:480b.
- \`context7-bridge\` — external library docs lookup.
- \`auditor/audit.ts\` — independent N=3 consensus re-check of scrum
findings; writes to data/_kb/audit_facts.jsonl via LLM Team
\`/api/run?mode=extract\`.
- \`crates/vectord/src/playbook_memory.rs\` — indexing for proven
playbooks: PlaybookEntry, DocRef, FailureRecord, BoostEntry,
PatternReport. Uses HNSW index + Lance columnar backend + promotion
pipeline. Already battle-tested for workers/staffing queries.
- Tree-split REDUCER: after shards return map-style summaries, they are
concatenated with internal §N§ markers and fed to a reviewer model to
produce ONE file-level review. Currently the reducer sees summaries,
not the full context behind each shard's conclusion.
# The problem
The reducer currently TRUNCATES to a short summary. When the auditor or
a future iteration wants to backtrack WHY the reducer concluded what it
did — which attempt succeeded, which failed, what KB chunks were
retrieved, what observer signal classified the file as LOOPING vs
CONVERGING — that context is lost. So:
1. Auditor can't verify citation provenance beyond the summary line.
2. Applier can't tell a "tried X, failed, qwen fixed it" playbook from a
"tried X and it was easy" playbook — they look identical downstream.
3. The matrix indexing is only used for RAG chunks during the scrum
pass, NOT for storing the full end-to-end pathway of a successful
review.
# The design question
Propose an enhanced reducer + indexing design that:
(a) Preserves the FULL backtrack context per reviewed file:
- every ladder attempt (model, ms, accepted_y/n, reject_reason)
- every retrieved KB chunk (source doc, chunk id, cosine score, rank)
- every observer signal (class, priors, prior-iter outcomes)
- every context7 bridge hit (library, version pulled)
- every sub-pipeline call (LLM Team extract results, audit consensus)
(b) Stores this pathway into vectord's matrix indexing alongside the
review verdict so it becomes retrievable by similarity. When a new
file's fingerprint (task_class + file-path prefix + signal class)
matches a past successful pathway, the system can hot-swap by
replaying or short-circuiting to the model/KB combo that worked.
(c) Surfaces the matrix-index hit rate as a feedback signal on the
scrum's UI — "this file was solved 3 times before by the same ladder
rung; consider short-circuiting to rung 5."
(d) Is compatible with the existing playbook_memory.rs primitives
(PlaybookEntry, DocRef, FailureRecord, BoostEntry) — extend don't
replace. The indexing layer is in production for workers/staffing;
we want the reducer pathway to piggyback on proven infrastructure.
# Constraints
- NO new crate. Extend vectord + scrum_master_pipeline.
- Full context can be LARGE — a reviewed file might have 5 retrievals,
4 ladder attempts, 8 observer priors. Design the embedding /
fingerprint so similar-but-not-identical pathways cluster.
- The reducer summary is still needed for the reviewer LLM input —
don't remove it, ADD the full-context sidecar.
- Audit trail: every pathway must be replayable deterministically from
what's stored (i.e., enough context to re-run without the original
prompt cache).
# Required output (STRICT JSON, no prose, no markdown fences):
{
"approach": "one-paragraph summary of your proposed design",
"data_model": {
"new_fields_on_playbook_entry": [...],
"new_types": [ {"name": "...", "purpose": "...", "fields": [...]} ]
},
"storage_strategy": {
"what_to_vectorize": "the text that becomes the embedding",
"fingerprint_key": "the deterministic key for similarity retrieval",
"backend": "HNSW, Lance, playbook_memory — pick"
},
"reducer_changes": {
"inputs_added": [...],
"outputs_added": [...],
"compatibility_notes": "how existing callers stay working"
},
"hot_swap_logic": "concrete rule for when to skip the ladder and replay a past pathway",
"ui_signal": "what to surface so J sees whether matrix indexing is earning its keep",
"risks": [...],
"why_this_beats_summarization": "one-paragraph argument"
}
`.trim();
interface Probe {
name: string;
provider: "ollama" | "ollama_cloud" | "openrouter";
model: string;
}
// Round-3 probe set — 4 probes covering the remaining ladder rungs +
// architecture/provider diversity. J wanted all 4 of the untouched
// options so the aggregated 10-model signal is saturated across the
// usable ladder.
const PROBES: Probe[] = [
{ name: "qwen35-397b", provider: "ollama_cloud", model: "qwen3.5:397b" },
{ name: "openrouter-gpt-oss", provider: "openrouter", model: "openai/gpt-oss-120b:free" },
{ name: "openrouter-gemma3", provider: "openrouter", model: "google/gemma-3-27b-it:free" },
{ name: "qwen3-coder-480b-2", provider: "ollama_cloud", model: "qwen3-coder:480b" }, // second probe of the coding specialist — stability check
];
async function ask(p: Probe): Promise<{ name: string; raw: string; ms: number; error?: string }> {
const started = Date.now();
try {
const r = await fetch(`${GATEWAY}/v1/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
provider: p.provider,
model: p.model,
messages: [
{ role: "system", content: "You are a senior architect. Output STRICT JSON only." },
{ role: "user", content: DESIGN_BRIEF },
],
max_tokens: 3000,
temperature: 0,
}),
});
const ms = Date.now() - started;
if (!r.ok) return { name: p.name, raw: "", ms, error: `HTTP ${r.status}: ${(await r.text()).slice(0, 200)}` };
const j = await r.json();
const content = j.content ?? j.message?.content ?? j.choices?.[0]?.message?.content ?? "";
return { name: p.name, raw: String(content), ms };
} catch (e: any) {
return { name: p.name, raw: "", ms: Date.now() - started, error: String(e).slice(0, 200) };
}
}
function extractJson(raw: string): any | null {
let s = raw.trim();
const fence = s.match(/^```(?:json)?\s*/);
if (fence) s = s.slice(fence[0].length);
if (s.endsWith("```")) s = s.slice(0, -3).trim();
const first = s.indexOf("{");
const last = s.lastIndexOf("}");
if (first < 0 || last <= first) return null;
try {
return JSON.parse(s.slice(first, last + 1));
} catch {
return null;
}
}
function summarize(obj: any, max = 240): string {
if (!obj) return "(no JSON parsed)";
if (typeof obj === "string") return obj.length > max ? obj.slice(0, max) + "…" : obj;
if (Array.isArray(obj)) return obj.map((x) => summarize(x, max)).join("; ");
return Object.entries(obj)
.map(([k, v]) => `${k}=${summarize(v, max)}`)
.join(" | ");
}
async function main() {
console.log(`\n── N=3 design consensus ──`);
console.log(`models: ${PROBES.map((p) => p.model).join(", ")}\n`);
const results = await Promise.all(PROBES.map(ask));
for (const r of results) {
console.log(`\n── ${r.name} (${r.ms}ms) ──`);
if (r.error) { console.log(` ERROR: ${r.error}`); continue; }
const j = extractJson(r.raw);
if (!j) {
console.log(` raw (no JSON): ${r.raw.slice(0, 600)}…`);
continue;
}
console.log(` approach: ${summarize(j.approach, 400)}`);
console.log(` fingerprint: ${summarize(j.storage_strategy?.fingerprint_key, 200)}`);
console.log(` vectorize: ${summarize(j.storage_strategy?.what_to_vectorize, 200)}`);
console.log(` backend: ${summarize(j.storage_strategy?.backend, 200)}`);
console.log(` hot_swap: ${summarize(j.hot_swap_logic, 300)}`);
console.log(` new_types: ${summarize(j.data_model?.new_types, 400)}`);
console.log(` risks: ${summarize(j.risks, 300)}`);
console.log(` why>summary: ${summarize(j.why_this_beats_summarization, 300)}`);
}
// Write full JSON to disk so we can inspect later.
const outPath = `/home/profit/lakehouse/data/_kb/consensus_reducer_design_${Date.now().toString(36)}.json`;
await Bun.write(outPath, JSON.stringify(results, null, 2));
console.log(`\nfull responses → ${outPath}`);
}
await main();
-528
View File
@@ -1,528 +0,0 @@
// Real-world architecture stress test — 6 iterations of the full pipeline
// against the PRD as a corpus. Goal: prove at scale what Phase 21
// promised (context continuation + tree-split), plus Phase 19
// compounding across iterations.
//
// Run: bun run tests/real-world/enrich_prd_pipeline.ts
//
// No mocks. No skipped layers. On any error, the test triggers
// cloud-rescue rather than fail — it's the architecture's job to
// recover. The test FAILS only if we can't complete 6 iterations.
import { readFile, writeFile, mkdir } from "node:fs/promises";
import { createHash } from "node:crypto";
const PRD_PATH = "/home/profit/lakehouse/docs/PRD.md";
const CHUNK_SIZE = 800; // chars per chunk — ~200 tokens
const CHUNK_OVERLAP = 120;
const TOP_K_RETRIEVE = 12; // chunks per iteration — pulled up to force overflow
const CONTEXT_BUDGET_CHARS = 4000; // tight budget — forces tree-split on every iter
const INJECT_FAIL_ON_ITER = 3; // force the TASK-retry loop on iter 3
// Continuation controls (per-cloud-call) — used for output-overflow.
// Separate from the task-retry loop (per-task) — that handles errors
// across attempts.
const PRIMARY_MAX_TOKENS = 150; // tight — forces truncation
const CONTINUATION_MAX_TOKENS = 300; // each continuation doubles headroom
const MAX_CONTINUATIONS = 6; // max stitch pieces per cloud call
// Task-level retry loop (J's clarification, 2026-04-22):
// When a TASK errors, retry the whole task up to 6 times. Each
// retry gets prior attempts' failures injected as learning context,
// so attempt N+1 is informed by what N failed at. The loop caps at
// 6 to avoid infinite spinning on genuinely unsolvable tasks.
const MAX_TASK_RETRIES = 6;
// To FORCE the retry loop on iter INJECT_FAIL_ON_ITER, cycle through
// 5 deliberately-invalid models + 1 valid one. Attempts 1-5 will
// 502/404 from Ollama Cloud; attempt 6 finally succeeds. Proves the
// loop fires all 6 with compounding failure context.
const FORCE_RETRY_MODEL_SEQUENCE = [
"deliberately-invalid-model-attempt-1",
"deliberately-invalid-model-attempt-2",
"deliberately-invalid-model-attempt-3",
"deliberately-invalid-model-attempt-4",
"deliberately-invalid-model-attempt-5",
"gpt-oss:20b", // 6th attempt succeeds
];
const GATEWAY = "http://localhost:3100";
const SIDECAR = "http://localhost:3200";
const CLOUD_MODEL = "gpt-oss:120b";
const RESCUE_MODEL = "gpt-oss:20b"; // fallback local cloud model via sidecar
const RUN_NONCE = Date.now().toString(36);
const OUT_DIR = `/home/profit/lakehouse/tests/real-world/runs/${RUN_NONCE}`;
// The 6 progressively-compounding questions. #6 explicitly requires
// synthesis across prior 5 answers.
const QUESTIONS: string[] = [
"Summarize the Lakehouse project's one-paragraph thesis: what problem does it solve, what's the unique approach?",
"How does Phase 19 playbook memory turn successful fills into a signal that boosts future rankings?",
"Explain the role of Phase 24 observer in the learning loop — what does it see, and what does it feed into?",
"What's the VRAM-aware profile swap mechanism in Phase 17, and why does it matter for multi-model serving?",
"How do Phase 25 validity windows and Phase 27 playbook versioning interact when a schema drifts?",
"Synthesize the prior 5 answers: how do the pieces (playbook memory, observer, profile swap, validity windows, versioning) compose into a system that measurably gets smarter over time? Cite specific prior answers.",
];
type Chunk = { id: string; text: string; embedding: number[]; offset: number };
interface IterationResult {
iteration: number;
question: string;
retrieval_top_k: number;
context_chars_before_budget: number;
tree_split_fired: boolean;
cloud_calls_total: number;
continuation_retries: number;
rescue_triggered: boolean;
// Task-level retry telemetry
task_attempts_made: number; // how many attempts fired (1 = first succeeded)
task_retry_history: Array<{ n: number; model: string; error: string }>;
playbook_id: string | null;
tokens_prompt: number;
tokens_completion: number;
citations_from_prior_iterations: string[];
duration_ms: number;
answer_preview: string;
errors_recovered: string[];
}
function log(msg: string) { console.log(`[enrich] ${msg}`); }
function sleep(ms: number) { return new Promise(r => setTimeout(r, ms)); }
function cosine(a: number[], b: number[]): number {
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }
if (na === 0 || nb === 0) return 0;
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
function hash(s: string): string { return createHash("sha256").update(s).digest("hex").slice(0, 10); }
async function embedBatch(texts: string[]): Promise<number[][]> {
// Sidecar /embed accepts a list. On partial failure, retry individually.
const r = await fetch(`${SIDECAR}/embed`, {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ texts }),
signal: AbortSignal.timeout(120000),
});
if (!r.ok) throw new Error(`embed batch ${r.status}: ${await r.text()}`);
const j: any = await r.json();
return j.embeddings;
}
function chunkText(text: string): Array<{ text: string; offset: number }> {
const out: Array<{ text: string; offset: number }> = [];
let i = 0;
while (i < text.length) {
const end = Math.min(i + CHUNK_SIZE, text.length);
const slice = text.slice(i, end).trim();
if (slice.length > 50) out.push({ text: slice, offset: i });
if (end >= text.length) break;
i = end - CHUNK_OVERLAP;
}
return out;
}
async function chat(opts: {
provider: "ollama" | "ollama_cloud",
model: string,
messages: Array<{ role: string; content: string }>,
max_tokens: number,
think: boolean,
}): Promise<{ content: string; prompt_tokens: number; completion_tokens: number; finish_reason: string }> {
const r = await fetch(`${GATEWAY}/v1/chat`, {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ ...opts }),
signal: AbortSignal.timeout(180000),
});
if (!r.ok) throw new Error(`/v1/chat ${r.status}: ${await r.text()}`);
const j: any = await r.json();
return {
content: j.choices?.[0]?.message?.content ?? "",
prompt_tokens: j.usage?.prompt_tokens ?? 0,
completion_tokens: j.usage?.completion_tokens ?? 0,
finish_reason: j.choices?.[0]?.finish_reason ?? "?",
};
}
// ─── Tree-split over oversized chunk set ──────────────────────────
async function treeSplitSummarize(
chunks: Chunk[],
question: string,
): Promise<{ scratchpad: string; cloud_calls: number }> {
// Shard into groups fitting within half the budget each.
const perShard = Math.max(1, Math.floor((CONTEXT_BUDGET_CHARS / 2) / CHUNK_SIZE));
const shards: Chunk[][] = [];
for (let i = 0; i < chunks.length; i += perShard) {
shards.push(chunks.slice(i, i + perShard));
}
log(` tree-split: ${chunks.length} chunks → ${shards.length} shards of up to ${perShard}`);
let scratchpad = "";
let cloud_calls = 0;
for (let si = 0; si < shards.length; si++) {
const shard = shards[si];
const shardText = shard.map(c => `[chunk @${c.offset}]\n${c.text}`).join("\n\n");
const userMsg = `Question: ${question}\n\nShard ${si + 1}/${shards.length} of source material:\n\n${shardText}\n\nScratchpad so far:\n${scratchpad || "(empty)"}\n\nUpdate the scratchpad: extract only facts from THIS shard that help answer the question. Be terse. No prose.`;
const r = await chat({
provider: "ollama_cloud",
model: CLOUD_MODEL,
messages: [
{ role: "system", content: "You maintain a concise factual scratchpad across multiple shards of source text. No prose outside the scratchpad. Each shard, append ≤80 words of relevant facts." },
{ role: "user", content: userMsg },
],
max_tokens: 500,
think: false,
});
cloud_calls += 1;
scratchpad += `\n--- shard ${si + 1} notes ---\n${r.content.trim()}`;
if (scratchpad.length > CONTEXT_BUDGET_CHARS) {
// truncate oldest halves
scratchpad = scratchpad.slice(-CONTEXT_BUDGET_CHARS);
log(` tree-split: scratchpad truncated to ${scratchpad.length} chars`);
}
}
return { scratchpad, cloud_calls };
}
// ─── Continuable generate — up to max_continuations stitches ──────
//
// Two failure modes handled:
// A) Empty response — typically thinking model burned the budget
// on hidden reasoning. Retry with 2× max_tokens.
// B) Truncated response (finish_reason=length) — answer got cut off
// mid-sentence. Pass the partial back as scratchpad and ask the
// model to continue from where it stopped.
//
// Stitching: keep appending content across retries; prompt_tokens and
// completion_tokens accumulate; finish_reason reflects the LAST call.
// Loop exits on the first call that finishes cleanly (stop) with
// non-empty content, OR when retries hit the cap.
async function generateContinuable(
opts: Parameters<typeof chat>[0] & { max_continuations?: number },
): Promise<{ content: string; prompt_tokens: number; completion_tokens: number; continuation_retries: number; finish_reason: string }> {
const maxCont = opts.max_continuations ?? 1;
let total = await chat(opts);
let retries = 0;
while (retries < maxCont && (total.content.length === 0 || total.finish_reason === "length")) {
retries += 1;
const mode = total.content.length === 0 ? "empty" : "truncated";
log(` continuation retry ${retries}/${maxCont} (${mode}: finish=${total.finish_reason}, content=${total.content.length} chars)`);
// Continuation prompt — branch on failure mode:
// empty → retry with 2× tokens, same prompt (thinking budget)
// length → pass the partial as assistant turn, ask to continue
const continuationMessages = total.content.length === 0
? opts.messages
: [
...opts.messages,
{ role: "assistant", content: total.content },
{ role: "user", content: "Continue from exactly where you stopped. Do not repeat. Finish the answer." },
];
const continued = await chat({
...opts,
max_tokens: CONTINUATION_MAX_TOKENS,
messages: continuationMessages,
});
total = {
content: total.content + continued.content,
prompt_tokens: total.prompt_tokens + continued.prompt_tokens,
completion_tokens: total.completion_tokens + continued.completion_tokens,
finish_reason: continued.finish_reason,
};
}
return { ...total, continuation_retries: retries };
}
// ─── Single iteration: retrieve → budget-check → chat → seed ─────
async function runIteration(
iteration: number,
question: string,
allChunks: Chunk[],
priorPlaybookIds: string[],
priorAnswers: string[],
): Promise<IterationResult> {
const started = Date.now();
const errorsRecovered: string[] = [];
log(`iter ${iteration}: "${question.slice(0, 70)}..."`);
// 1. Embed the question
const qEmb = (await embedBatch([question]))[0];
// 2. Retrieve top-K chunks by cosine
const scored = allChunks
.map(c => ({ c, score: cosine(qEmb, c.embedding) }))
.sort((a, b) => b.score - a.score)
.slice(0, TOP_K_RETRIEVE);
const chunks = scored.map(x => x.c);
log(` retrieved top ${chunks.length} chunks (score range ${scored[0].score.toFixed(3)} .. ${scored[scored.length - 1].score.toFixed(3)})`);
// 3. Context budget check — tree-split if over
const contextChars = chunks.map(c => c.text).join("\n\n").length;
let contextForPrompt: string;
let treeSplit = false;
let cloudCallsTotal = 0;
if (contextChars > CONTEXT_BUDGET_CHARS) {
treeSplit = true;
log(` context ${contextChars} chars > budget ${CONTEXT_BUDGET_CHARS} → tree-split`);
const { scratchpad, cloud_calls } = await treeSplitSummarize(chunks, question);
contextForPrompt = `Distilled scratchpad from ${chunks.length} source chunks (too large to fit directly):\n${scratchpad}`;
cloudCallsTotal += cloud_calls;
} else {
contextForPrompt = chunks.map(c => `[chunk @${c.offset}]\n${c.text}`).join("\n\n");
}
// 4. Seed prompt with prior iteration answers (real compounding).
// Not just IDs — the model needs the CONTENT to synthesize.
let citationBlock = "";
let citationsReceived: string[] = [];
if (priorPlaybookIds.length > 0 && priorAnswers.length > 0) {
const lines = priorAnswers.map((ans, i) => {
const pid = priorPlaybookIds[i]?.slice(0, 12) ?? "unknown";
// Trim each prior answer to ~400 chars so we don't blow budget
return `[pb:${pid}] iter ${i + 1} answer:\n${ans.slice(0, 400)}\n`;
});
citationBlock = `\n\n═══ PRIOR ITERATIONS (compounding context) ═══\n${lines.join("\n")}═══ end prior iterations ═══\n\nYour answer MUST cite specific prior iterations using [pb:ID] notation when drawing on them. Synthesis questions require explicit cross-iteration reasoning.`;
citationsReceived = priorPlaybookIds.slice();
}
// 5. TASK-LEVEL RETRY LOOP — per J's clarification 2026-04-22.
// Try the task up to MAX_TASK_RETRIES times. Each retry:
// a) Picks a model (normally CLOUD_MODEL; on INJECT_FAIL_ON_ITER,
// cycles through 5 invalid models + 1 valid to force full loop)
// b) Injects prior attempt errors as learning context
// c) If the attempt succeeds (non-empty, >100 chars), loop exits
// d) Otherwise, records failure and tries again with the learning
//
// Cap at 6 so we don't spin forever on unsolvable tasks.
let result: { content: string; prompt_tokens: number; completion_tokens: number; continuation_retries: number; finish_reason: string } | null = null;
let rescueTriggered = false;
const taskAttemptHistory: Array<{ n: number; model: string; error: string }> = [];
const forceRetries = iteration === INJECT_FAIL_ON_ITER;
if (forceRetries) log(` FORCING TASK-RETRY LOOP — iter ${iteration} will cycle through 5 invalid models + 1 valid`);
for (let attempt = 1; attempt <= MAX_TASK_RETRIES; attempt++) {
const modelForAttempt = forceRetries
? FORCE_RETRY_MODEL_SEQUENCE[attempt - 1]
: CLOUD_MODEL;
// Compose a prior-attempts learning block for attempts 2+
const learningBlock = taskAttemptHistory.length > 0
? `\n\n═══ PRIOR ATTEMPTS THIS TASK (do NOT repeat these failures; adjust approach) ═══\n${taskAttemptHistory.map(a => `Attempt ${a.n} (model ${a.model}) failed: ${a.error.slice(0, 160)}`).join("\n")}\n═══ end prior attempts ═══\n`
: "";
log(` task attempt ${attempt}/${MAX_TASK_RETRIES}: model=${modelForAttempt}${learningBlock ? " [with prior-failure context]" : ""}`);
try {
const r = await generateContinuable({
provider: "ollama_cloud",
model: modelForAttempt,
messages: [
{ role: "system", content: "You answer questions about the Lakehouse PRD using only the provided source material and prior iteration answers. Be specific. Cite chunk offsets OR [pb:ID] markers. Write a detailed 250-word answer." },
{ role: "user", content: `Question: ${question}\n\nSource material:\n${contextForPrompt}${citationBlock}${learningBlock}` },
],
max_tokens: PRIMARY_MAX_TOKENS,
think: false,
max_continuations: MAX_CONTINUATIONS,
});
cloudCallsTotal += 1 + r.continuation_retries;
if (r.content && r.content.length > 100) {
// Acceptable answer — exit loop
result = r;
if (attempt > 1) {
log(` task attempt ${attempt} SUCCEEDED (${r.content.length} chars) after ${attempt - 1} prior failures`);
rescueTriggered = true;
}
break;
}
// Thin response — count as failure with learning signal
const err = `thin-answer: ${r.content.length} chars, finish=${r.finish_reason}`;
taskAttemptHistory.push({ n: attempt, model: modelForAttempt, error: err });
errorsRecovered.push(`attempt ${attempt}: ${err}`);
} catch (e) {
const err = (e as Error).message;
taskAttemptHistory.push({ n: attempt, model: modelForAttempt, error: err });
errorsRecovered.push(`attempt ${attempt}: ${err.slice(0, 120)}`);
cloudCallsTotal += 1;
}
}
// Last-ditch: if all 6 task attempts failed, try the local fallback
// once more so we at least return SOMETHING. This is the "don't get
// caught in a loop, accept best-so-far" rule J stated explicitly.
if (!result) {
errorsRecovered.push(`all ${MAX_TASK_RETRIES} task attempts failed — local fallback`);
rescueTriggered = true;
try {
result = await generateContinuable({
provider: "ollama",
model: "qwen3.5:latest",
messages: [{ role: "user", content: `Q: ${question}\n\n${contextForPrompt.slice(0, 4000)}` }],
max_tokens: 300,
think: false,
max_continuations: 2,
});
cloudCallsTotal += 1 + result.continuation_retries;
} catch (e) {
// Absolute last resort — fabricate a skeleton result
result = {
content: `[task failed after ${MAX_TASK_RETRIES} retries + local fallback: ${(e as Error).message}]`,
prompt_tokens: 0,
completion_tokens: 0,
continuation_retries: 0,
finish_reason: "error",
};
}
}
if (result.content.length === 0) {
errorsRecovered.push("even rescue returned empty — last-ditch local fallback");
rescueTriggered = true;
result = await generateContinuable({
provider: "ollama",
model: "qwen3.5:latest",
messages: [{ role: "user", content: `Q: ${question}\n\n${contextForPrompt.slice(0, 4000)}` }],
max_tokens: 300,
think: false,
});
cloudCallsTotal += 1;
}
// 6. Seed playbook with the answer
let playbook_id: string | null = null;
try {
const ts = new Date().toISOString();
const seedOp = `TEST: enrich_prd_run_${RUN_NONCE} iter${iteration} in Corpus, PRD`;
const r = await fetch(`${GATEWAY}/vectors/playbook_memory/seed`, {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({
operation: seedOp,
approach: `q="${question.slice(0, 80)}" context_chars=${contextChars} tree_split=${treeSplit}`,
context: result.content.slice(0, 600),
endorsed_names: [`iter${iteration}_${RUN_NONCE}`],
append: true,
}),
signal: AbortSignal.timeout(15000),
});
if (r.ok) {
const j: any = await r.json();
playbook_id = j.outcome?.playbook_id ?? null;
} else {
errorsRecovered.push(`seed ${r.status}: ${(await r.text()).slice(0, 100)}`);
}
} catch (e) {
errorsRecovered.push(`seed exception: ${(e as Error).message}`);
}
return {
iteration,
question,
retrieval_top_k: chunks.length,
context_chars_before_budget: contextChars,
tree_split_fired: treeSplit,
cloud_calls_total: cloudCallsTotal,
continuation_retries: result.continuation_retries,
rescue_triggered: rescueTriggered,
task_attempts_made: taskAttemptHistory.length + 1, // +1 for the successful attempt
task_retry_history: taskAttemptHistory,
playbook_id,
tokens_prompt: result.prompt_tokens,
tokens_completion: result.completion_tokens,
citations_from_prior_iterations: citationsReceived,
duration_ms: Date.now() - started,
answer_preview: result.content.slice(0, 500),
errors_recovered: errorsRecovered,
};
}
async function main() {
await mkdir(OUT_DIR, { recursive: true });
log(`run nonce: ${RUN_NONCE}`);
log(`output dir: ${OUT_DIR}`);
// ─── Phase 1: load, chunk, embed the PRD ───────────────────────
log(`loading PRD from ${PRD_PATH}`);
const prd = await readFile(PRD_PATH, "utf8");
log(`PRD: ${prd.length} chars, ${prd.split("\n").length} lines`);
const raw_chunks = chunkText(prd);
log(`chunked into ${raw_chunks.length} pieces (size ${CHUNK_SIZE}, overlap ${CHUNK_OVERLAP})`);
// Embed in batches of 32 to avoid sidecar overload
const allChunks: Chunk[] = [];
const BATCH = 32;
const t0 = Date.now();
for (let i = 0; i < raw_chunks.length; i += BATCH) {
const batch = raw_chunks.slice(i, i + BATCH);
const embs = await embedBatch(batch.map(b => b.text));
for (let j = 0; j < batch.length; j++) {
allChunks.push({
id: hash(batch[j].text),
text: batch[j].text,
embedding: embs[j].map(x => Number(x)),
offset: batch[j].offset,
});
}
log(` embedded ${allChunks.length}/${raw_chunks.length}`);
}
log(`embedded all ${allChunks.length} chunks in ${((Date.now() - t0) / 1000).toFixed(1)}s`);
// ─── Phase 2: 6 iterations ─────────────────────────────────────
const results: IterationResult[] = [];
const priorIds: string[] = [];
const priorAnswers: string[] = [];
for (let i = 1; i <= QUESTIONS.length; i++) {
const q = QUESTIONS[i - 1];
const r = await runIteration(i, q, allChunks, priorIds, priorAnswers);
results.push(r);
if (r.playbook_id) priorIds.push(r.playbook_id);
priorAnswers.push(r.answer_preview);
log(` → iter ${i}: ${r.errors_recovered.length} errors recovered, ${r.continuation_retries} continuations, tree-split=${r.tree_split_fired}, rescue=${r.rescue_triggered}, ${r.duration_ms}ms`);
await writeFile(`${OUT_DIR}/iter_${i}.json`, JSON.stringify(r, null, 2));
}
// Check whether iter 6 actually cited prior pb:IDs in its answer.
// Playbook IDs look like `pb-seed-<hex>` so the regex needs to allow
// hyphens + letters inside the brackets, not just hex chars.
const iter6 = results[5];
const citationsHonored = iter6 ? (iter6.answer_preview.match(/\[pb:[\w-]+\]/gi)?.length ?? 0) : 0;
// ─── Phase 3: summary ──────────────────────────────────────────
const summary = {
run_nonce: RUN_NONCE,
ran_at: new Date().toISOString(),
prd_chars: prd.length,
prd_chunks: allChunks.length,
iterations: results.length,
total_cloud_calls: results.reduce((s, r) => s + r.cloud_calls_total, 0),
total_continuation_retries: results.reduce((s, r) => s + r.continuation_retries, 0),
total_errors_recovered: results.reduce((s, r) => s + r.errors_recovered.length, 0),
tree_splits_fired: results.filter(r => r.tree_split_fired).length,
rescues_triggered: results.filter(r => r.rescue_triggered).length,
iter6_received_prior_ids: results[5]?.citations_from_prior_iterations.length ?? 0,
iter6_actually_cited_in_answer: citationsHonored,
iter3_task_attempts: results[2]?.task_attempts_made ?? 0,
iter3_task_retries: results[2]?.task_retry_history.length ?? 0,
max_task_attempts_any_iter: Math.max(...results.map(r => r.task_attempts_made)),
total_duration_ms: results.reduce((s, r) => s + r.duration_ms, 0),
overall: results.length === 6 && results.every(r => r.playbook_id !== null) ? "PASS" : "PARTIAL",
};
await writeFile(`${OUT_DIR}/summary.json`, JSON.stringify(summary, null, 2));
log("");
log(`══════ SUMMARY ${summary.overall} ══════`);
log(` 6 iterations, ${summary.total_cloud_calls} cloud calls, ${summary.total_errors_recovered} errors recovered`);
log(` tree-splits: ${summary.tree_splits_fired}/6 continuations: ${summary.total_continuation_retries} rescues: ${summary.rescues_triggered}`);
log(` iter 6 received ${summary.iter6_received_prior_ids} prior IDs, cited ${summary.iter6_actually_cited_in_answer} [pb:...] markers in its answer`);
log(` iter 3 task-retry loop: ${summary.iter3_task_attempts} attempts (${summary.iter3_task_retries} prior-failure retries before success)`);
log(` total duration: ${(summary.total_duration_ms / 1000).toFixed(1)}s`);
log("");
for (const r of results) {
const flags = [
r.tree_split_fired ? "tree-split" : "",
r.continuation_retries > 0 ? `cont=${r.continuation_retries}` : "",
r.rescue_triggered ? "rescued" : "",
r.errors_recovered.length > 0 ? `err=${r.errors_recovered.length}` : "",
].filter(Boolean).join(" ");
log(` iter ${r.iteration}: ${r.tokens_prompt}+${r.tokens_completion} tok, ${r.duration_ms}ms ${flags ? `[${flags}]` : ""}`);
}
log("");
log(`artifacts: ${OUT_DIR}/{iter_1..6.json, summary.json}`);
process.exit(summary.overall === "PASS" ? 0 : 1);
}
main().catch(e => { console.error("[enrich] fatal:", e); process.exit(2); });
-267
View File
@@ -1,267 +0,0 @@
// Hard-task escalation test. The task is deliberately constructed so
// that a local 7B model (qwen3.5:latest) will miss at least one of the
// validation rules. Watch the escalation ladder:
// 1. qwen3.5:latest (local 7B) — likely fails
// 2. qwen3:latest (local 7B) — likely fails differently
// 3. gpt-oss:20b (cloud 20B) — may fail
// 4. gpt-oss:120b (cloud 120B) — should succeed
// 5. gpt-oss:120b w/ prior-attempt errors injected — retry with context
// 6. absolute last ditch: return best-so-far with failure annotation
//
// Each attempt:
// - Calls the model via /v1/chat
// - Validates the output against a strict rubric
// - On fail: records the specific rubric violations + the partial
// output, injects both into the next attempt's prompt as "here's
// what's wrong, fix it specifically"
// - On success: exit loop
//
// Run: bun run tests/real-world/hard_task_escalation.ts
import { writeFile, mkdir } from "node:fs/promises";
const GATEWAY = "http://localhost:3100";
const OUT_DIR = `/home/profit/lakehouse/tests/real-world/runs/hard_task_${Date.now().toString(36)}`;
const MAX_ATTEMPTS = 6;
// The hard task. Specific enough that a small model will miss at
// least one rule. Not purely knowledge-based — it's a code-generation
// task with strict structural constraints.
const TASK = `Write a complete Rust async function with the EXACT signature:
pub async fn check_drift_batched(refs: Vec<DocRef>) -> Result<Vec<String>, String>
It must:
1. Group refs by tool name (case-insensitive — use .to_ascii_lowercase())
2. Issue parallel HTTP GET requests to http://localhost:3900/docs/{tool}/diff?since={snippet_hash}
3. Use reqwest and a JoinSet/Semaphore to cap concurrent in-flight requests at 4
4. On HTTP 5xx, retry with exponential backoff: sleep 250ms, then 500ms, then 1000ms, then give up on that tool
5. Parse the response JSON: {"drifted": bool, ...}. Return a Vec<String> of tool names where drifted == true
6. All errors bubble via ? or Result — NO .unwrap(), NO .expect(), NO panic!()
7. Include rustdoc /// comments on the function and each internal helper
Assume this struct is already imported:
pub struct DocRef { pub tool: String, pub snippet_hash: Option<String>, pub version_seen: String }
Output ONLY the Rust code. No prose, no markdown fences, no explanation. Start directly with the /// doc comment.`;
// Escalation ladder — small-local → large-local → cloud → specialist
// cloud → trillion-param cloud. Corrected 2026-04-22 per J:
// gpt-oss:20b is LOCAL (ollama list confirms 13 GB on disk), and the
// final escalation tier should be kimi-k2:1t (the biggest model
// we have access to on Ollama Cloud).
const LADDER: Array<{ provider: "ollama" | "ollama_cloud"; model: string; note: string }> = [
{ provider: "ollama", model: "qwen3.5:latest", note: "local 7B" },
{ provider: "ollama", model: "qwen3:latest", note: "local 7B (different) " },
{ provider: "ollama", model: "gpt-oss:20b", note: "local 20B" }, // FIXED: local, not cloud
{ provider: "ollama_cloud", model: "gpt-oss:120b", note: "cloud 120B" },
{ provider: "ollama_cloud", model: "devstral-2:123b", note: "cloud 123B (coding specialist)" },
// NOTE 2026-04-22 — J wanted Kimi as the last escalation but Kimi
// K2.5/K2.6 both return "this model requires a subscription" on our
// current Ollama Cloud key. mistral-large-3:675b is the biggest
// model actually provisioned on this key (verified via direct curl
// to ollama.com/api/generate). Upgrade path: Ollama Cloud Pro →
// swap this line to kimi-k2.5 or kimi-k2.6:cloud.
{ provider: "ollama_cloud", model: "mistral-large-3:675b", note: "cloud 675B (biggest available on current key; kimi-k2.x needs pro subscription)" },
];
// Validation rubric — the answer must pass all of these to be accepted.
interface RubricResult {
passed: boolean;
violations: string[];
passed_rules: string[];
}
function validate(code: string): RubricResult {
const violations: string[] = [];
const passed: string[] = [];
const check = (rule: string, ok: boolean) => { ok ? passed.push(rule) : violations.push(rule); };
check("has pub async fn check_drift_batched signature",
/pub\s+async\s+fn\s+check_drift_batched\s*\(/.test(code));
check("takes Vec<DocRef> argument",
/refs\s*:\s*Vec\s*<\s*DocRef\s*>/.test(code));
check("returns Result<Vec<String>, String>",
/Result\s*<\s*Vec\s*<\s*String\s*>\s*,\s*String\s*>/.test(code));
check("uses reqwest",
/\breqwest\b/i.test(code));
check("references JoinSet or Semaphore for concurrency",
/\bJoinSet\b|\bSemaphore\b/i.test(code));
check("bounds concurrency at 4",
/\b4\b/.test(code) && (/Semaphore\s*::\s*new\s*\(\s*4\b/.test(code) || /permits\s*:\s*4\b/.test(code) || /limit\s*:\s*4\b/.test(code) || /max\s*:\s*4\b/.test(code) || /capacity\s*:\s*4\b/.test(code)));
// Exponential backoff — models express this several ways. Accept
// any recognizable doubling pattern starting at 250ms. 2026-04-22:
// devstral-2:123b wrote `retry_delay *= 2` which my earlier regex
// rejected even though the code is correct. Broadening rubric to
// match all idiomatic doubling forms.
const hasSeed250 = /Duration\s*::\s*from_millis\s*\(\s*250\b/.test(code)
|| /millis\s*\(\s*250\b/.test(code);
const hasDoublingPattern = /250\s*\*\s*2/.test(code) // 250 * 2^n literal
|| /<<\s*\d+/.test(code) // bit-shift
|| /\.pow\s*\(/.test(code) // 2u32.pow(attempt)
|| /\*=\s*2\b/.test(code) // delay *= 2 ← was missing
|| /\*\s*2\s*;/.test(code) // delay = delay * 2;
|| /saturating_mul\s*\(\s*2\b/.test(code); // saturating doubling
check("has 250ms backoff seed",
hasSeed250);
check("reaches 500ms backoff (literal or doubling from 250)",
/Duration\s*::\s*from_millis\s*\(\s*500\b/.test(code)
|| /millis\s*\(\s*500\b/.test(code)
|| (hasSeed250 && hasDoublingPattern));
check("reaches 1000ms backoff (literal or doubling to 1000)",
/Duration\s*::\s*from_millis\s*\(\s*1000\b/.test(code)
|| /millis\s*\(\s*1000\b/.test(code)
|| (hasSeed250 && hasDoublingPattern));
check("case-insensitive tool grouping (to_ascii_lowercase)",
/to_ascii_lowercase|to_lowercase/.test(code));
check("NO .unwrap() — all errors bubble via ?",
!/\.unwrap\s*\(\s*\)/.test(code));
check("NO .expect(...) — all errors bubble via ?",
!/\.expect\s*\(/.test(code));
check("NO panic!() / unimplemented!() / todo!()",
!/\bpanic!\s*\(|\bunimplemented!\s*\(|\btodo!\s*\(/.test(code));
check("has rustdoc /// comments",
/\/\/\//.test(code));
check("reasonable length (> 500 chars)",
code.length > 500);
return { passed: violations.length === 0, violations, passed_rules: passed };
}
function log(msg: string) { console.log(`[hard] ${msg}`); }
async function chat(opts: {
provider: "ollama" | "ollama_cloud",
model: string,
prompt: string,
}): Promise<{ content: string; error?: string }> {
try {
const r = await fetch(`${GATEWAY}/v1/chat`, {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({
provider: opts.provider,
model: opts.model,
messages: [{ role: "user", content: opts.prompt }],
max_tokens: 2500,
temperature: 0.2,
think: false,
}),
signal: AbortSignal.timeout(240000),
});
if (!r.ok) return { content: "", error: `/v1/chat ${r.status}: ${(await r.text()).slice(0, 300)}` };
const j: any = await r.json();
return { content: j.choices?.[0]?.message?.content ?? "" };
} catch (e) {
return { content: "", error: (e as Error).message };
}
}
interface AttemptRecord {
n: number;
provider: string;
model: string;
duration_ms: number;
content_chars: number;
error: string | null;
rubric_violations: string[];
rubric_passed: string[];
accepted: boolean;
}
function extractCode(raw: string): string {
// Strip common fence wrappers
const m = raw.match(/```(?:rust)?\s*\n([\s\S]*?)```/);
if (m) return m[1].trim();
return raw.trim();
}
async function main() {
await mkdir(OUT_DIR, { recursive: true });
log(`output: ${OUT_DIR}`);
log(`task: ${TASK.slice(0, 120)}...`);
log("");
const attempts: AttemptRecord[] = [];
let acceptedCode: string | null = null;
for (let i = 0; i < MAX_ATTEMPTS; i++) {
const n = i + 1;
const rung = LADDER[i] ?? LADDER[LADDER.length - 1];
// Build the prompt: base task + prior failures' learning blocks
let priorLearning = "";
if (attempts.length > 0) {
priorLearning = `\n\n═══ PRIOR ATTEMPTS FAILED. Fix these exact issues: ═══\n`;
for (const a of attempts) {
priorLearning += `Attempt ${a.n} (${a.provider}/${a.model}, ${a.content_chars} chars) violations:\n`;
for (const v of a.rubric_violations) priorLearning += ` - ${v}\n`;
if (a.error) priorLearning += ` [error: ${a.error.slice(0, 120)}]\n`;
}
priorLearning += `═══ end prior attempts ═══\n\nDO NOT repeat the above violations. Address each one explicitly.`;
}
log(`attempt ${n}/${MAX_ATTEMPTS}: ${rung.provider}::${rung.model}${priorLearning ? " [w/ learning]" : ""}`);
const t0 = Date.now();
const r = await chat({ provider: rung.provider, model: rung.model, prompt: TASK + priorLearning });
const dur = Date.now() - t0;
const code = extractCode(r.content);
const rubric = code ? validate(code) : { passed: false, violations: ["empty response"], passed_rules: [] };
const record: AttemptRecord = {
n,
provider: rung.provider,
model: rung.model,
duration_ms: dur,
content_chars: code.length,
error: r.error ?? null,
rubric_violations: rubric.violations,
rubric_passed: rubric.passed_rules,
accepted: rubric.passed,
};
attempts.push(record);
log(` → ${dur}ms, ${code.length} chars, ${rubric.passed_rules.length} rules passed / ${rubric.violations.length} failed${r.error ? `, err: ${r.error.slice(0, 80)}` : ""}`);
for (const v of rubric.violations.slice(0, 5)) log(` ✗ ${v}`);
await writeFile(`${OUT_DIR}/attempt_${n}.txt`, code);
await writeFile(`${OUT_DIR}/attempt_${n}.json`, JSON.stringify(record, null, 2));
if (rubric.passed) {
log(` ✅ ACCEPTED on attempt ${n}`);
acceptedCode = code;
break;
}
}
const summary = {
task: TASK.slice(0, 200),
total_attempts: attempts.length,
accepted: acceptedCode !== null,
accepted_on_attempt: acceptedCode ? attempts.findIndex(a => a.accepted) + 1 : null,
escalation_path: attempts.map(a => `${a.provider}/${a.model}`),
per_attempt_pass_counts: attempts.map(a => a.rubric_passed.length),
per_attempt_violation_counts: attempts.map(a => a.rubric_violations.length),
total_duration_ms: attempts.reduce((s, a) => s + a.duration_ms, 0),
};
await writeFile(`${OUT_DIR}/summary.json`, JSON.stringify(summary, null, 2));
log("");
log(`═══ RESULT ═══`);
log(`attempts: ${summary.total_attempts}`);
log(`accepted: ${summary.accepted} ${summary.accepted ? `on attempt ${summary.accepted_on_attempt}` : ""}`);
log(`escalation path:`);
for (const [i, a] of attempts.entries()) {
const mark = a.accepted ? "✅" : "❌";
log(` ${mark} attempt ${i + 1}: ${a.provider}/${a.model} — ${a.rubric_passed.length}/${a.rubric_passed.length + a.rubric_violations.length} rules passed, ${a.duration_ms}ms`);
}
log("");
log(`total time: ${(summary.total_duration_ms / 1000).toFixed(1)}s`);
log(`artifacts: ${OUT_DIR}/{attempt_1..N.{txt,json}, summary.json}`);
process.exit(summary.accepted ? 0 : 1);
}
main().catch(e => { console.error("[hard] fatal:", e); process.exit(2); });
-186
View File
@@ -1,186 +0,0 @@
// Nine-consecutive audit runner — empirical test of the predictive-
// compounding property. Runs the audit pipeline 9 times against the
// same PR (each time with a new diff from Gitea), captures the
// verdict + audit_lessons state after each run, and reports whether
// the KB stabilizes or drifts.
//
// What we expect (favorable compounding):
// - signature_count grows sublinearly (same patterns recur, so
// distinct-signature count stabilizes fast)
// - verdict settles on a stable value after run 2-3 (first audit
// establishes baseline, rest repeat)
// - confidence stays LOW for all signatures (same PR repeatedly)
// - NO new recurring findings fire because confidence < 0.3 on
// same-PR noise (kb_index rating policy)
//
// What would indicate drift (the thing we want to prove DOESN'T happen):
// - signature_count grows linearly — each run produces new signatures
// - verdict oscillates (block → approve → block ...)
// - confidence inflates — kb_index rating escalates on repeated runs
//
// Run: bun run tests/real-world/nine_consecutive_audits.ts
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { aggregate } from "../../auditor/kb_index.ts";
import { getPrSnapshot } from "../../auditor/gitea.ts";
import { auditPr } from "../../auditor/audit.ts";
const REPO = "/home/profit/lakehouse";
const AUDIT_LESSONS = `${REPO}/data/_kb/audit_lessons.jsonl`;
const VERDICTS_DIR = `${REPO}/data/_auditor/verdicts`;
const POLL_INTERVAL_MS = 5_000;
const RUNS = Number(process.env.LH_AUDIT_RUNS ?? 9);
const TARGET_PR = Number(process.env.LH_AUDIT_PR ?? 8);
const SKIP_INFERENCE = process.env.LH_AUDITOR_SKIP_INFERENCE !== "0";
const RESET_KB = process.env.LH_RESET_KB === "1";
async function waitForVerdict(prNum: number, sha: string, deadlineMs: number): Promise<any> {
const short = sha.slice(0, 12);
const path = join(VERDICTS_DIR, `${prNum}-${short}.json`);
const start = Date.now();
while (Date.now() - start < deadlineMs) {
try {
const raw = await readFile(path, "utf8");
return JSON.parse(raw);
} catch { /* not yet */ }
await Bun.sleep(POLL_INTERVAL_MS);
}
throw new Error(`no verdict file after ${deadlineMs}ms: ${path}`);
}
async function captureAggState(): Promise<{ sig_count: number; max_count: number; max_confidence: number; top3: Array<{ sig: string; count: number; conf: number; summary: string }> }> {
const agg = await aggregate<any>(AUDIT_LESSONS, {
keyFn: (r) => r?.signature,
scopeFn: (r) => (r?.pr_number !== undefined ? `pr-${r.pr_number}` : undefined),
});
const list = Array.from(agg.values()).sort((a, b) => b.count - a.count);
const recurring = list.filter(r => r.count >= 2);
const recurringMaxCount = recurring.length > 0 ? Math.max(...recurring.map(a => a.count)) : 0;
const recurringMaxConf = recurring.length > 0 ? Math.max(...recurring.map(a => a.confidence)) : 0;
return {
sig_count: list.length,
max_count: list[0]?.count ?? 0,
max_confidence: recurringMaxConf,
recurring_max_count: recurringMaxCount,
top3: list.slice(0, 3).map(a => ({
sig: a.signature,
count: a.count,
conf: a.confidence,
summary: a.representative_summary.slice(0, 80),
})),
};
}
interface RunRecord {
run: number;
sha: string;
verdict_overall: string;
findings_total: number;
findings_block: number;
findings_warn: number;
findings_info: number;
audit_duration_ms: number;
claims_total: number;
claims_empirical: number;
kb_sig_count_after: number;
kb_max_count_after: number;
kb_max_confidence_after: number;
kb_recurring_max_count: number;
}
async function main() {
console.log(`[nine] target PR: #${TARGET_PR}`);
console.log(`[nine] runs: ${RUNS}`);
console.log(`[nine] skip_inference: ${SKIP_INFERENCE}`);
console.log(`[nine] reset_kb: ${RESET_KB}`);
console.log(`[nine] audit_lessons.jsonl: ${AUDIT_LESSONS}`);
if (RESET_KB) {
console.log("[nine] clearing audit_lessons.jsonl for clean test...");
await writeFile(AUDIT_LESSONS, "");
}
console.log("");
const pr = await getPrSnapshot(TARGET_PR);
console.log(`[nine] PR #${pr.number}: "${pr.title}" (head=${pr.head_sha.slice(0, 12)})`);
console.log(`[nine] files in diff: ${pr.files.length}`);
console.log("");
const baseline = await captureAggState();
console.log(`[nine] baseline: sig_count=${baseline.sig_count} max_count=${baseline.max_count} max_conf=${baseline.max_confidence.toFixed(2)}`);
console.log("");
const records: RunRecord[] = [];
for (let n = 1; n <= RUNS; n++) {
const t0 = Date.now();
console.log(`─── run ${n}/${RUNS} ───`);
const verdict = await auditPr(pr, {
dry_run: true,
skip_dynamic: true,
skip_inference: SKIP_INFERENCE,
});
console.log(` sha ${verdict.head_sha.slice(0, 12)}`);
const after = await captureAggState();
const rec: RunRecord = {
run: n,
sha: verdict.head_sha.slice(0, 12),
verdict_overall: String(verdict.overall),
findings_total: Number(verdict.metrics?.findings_total ?? 0),
findings_block: Number(verdict.metrics?.findings_block ?? 0),
findings_warn: Number(verdict.metrics?.findings_warn ?? 0),
findings_info: Number(verdict.metrics?.findings_info ?? 0),
audit_duration_ms: Number(verdict.metrics?.audit_duration_ms ?? 0),
claims_total: Number(verdict.metrics?.claims_total ?? 0),
claims_empirical: Number(verdict.metrics?.claims_empirical ?? 0),
kb_sig_count_after: after.sig_count,
kb_max_count_after: after.max_count,
kb_max_confidence_after: after.max_confidence,
kb_recurring_max_count: after.recurring_max_count,
};
records.push(rec);
console.log(` verdict=${rec.verdict_overall} findings=${rec.findings_total} (b=${rec.findings_block} w=${rec.findings_warn})`);
console.log(` kb after: sig=${rec.kb_sig_count_after} max_count=${rec.kb_max_count_after} recurring_max=${rec.kb_recurring_max_count} max_conf=${rec.kb_max_confidence_after.toFixed(2)}`);
console.log(` elapsed: ${((Date.now() - t0) / 1000).toFixed(1)}s`);
console.log("");
}
console.log("═══ FINAL ═══");
console.log("run | verdict | find | block warn info | dur_s | kb_sig max_count max_conf");
for (const r of records) {
console.log(
` ${String(r.run).padStart(1)} | ${r.verdict_overall.padEnd(16)} | ${String(r.findings_total).padStart(4)} | ${String(r.findings_block).padStart(5)} ${String(r.findings_warn).padStart(5)} ${String(r.findings_info).padStart(5)} | ${(r.audit_duration_ms / 1000).toFixed(1).padStart(5)} | ${String(r.kb_sig_count_after).padStart(6)} ${String(r.kb_max_count_after).padStart(9)} ${r.kb_max_confidence_after.toFixed(2)}`,
);
}
console.log("");
console.log("═══ COMPOUNDING PROPERTY ═══");
const sigDelta = records[records.length - 1].kb_sig_count_after - baseline.sig_count;
const maxConf = records[records.length - 1].kb_max_confidence_after;
const recurringMax = records[records.length - 1].kb_recurring_max_count;
console.log(` signatures added over ${RUNS} runs: ${sigDelta}`);
console.log(` max recurring count after run ${RUNS}: ${recurringMax} (same-PR recurrences per signature)`);
console.log(` max confidence after run ${RUNS}: ${maxConf.toFixed(2)} (expect LOW — same-PR should not inflate)`);
const verdictSet = new Set(records.map(r => r.verdict_overall));
if (verdictSet.size === 1) {
console.log(` verdict stable: all ${RUNS} runs returned '${[...verdictSet][0]}' ✓`);
} else {
console.log(` verdict oscillated across runs: ${[...verdictSet].join(" | ")} ✗`);
}
if (maxConf < 0.6 && recurringMax < 5) {
console.log(` confidence policy holding: same-PR noise stays below escalation threshold ✓`);
} else {
console.log(` ⚠ cross-cutting pattern detected (conf=${maxConf.toFixed(2)}, recurring=${recurringMax}) — kb_index policy escalated`);
}
const jsonOut = `${REPO}/tests/real-world/runs/nine_consecutive_${Date.now().toString(36)}.json`;
await Bun.write(jsonOut, JSON.stringify({ target_pr: TARGET_PR, baseline, records }, null, 2));
console.log("");
console.log(` report: ${jsonOut}`);
}
main().catch(e => { console.error("[nine] fatal:", e); process.exit(1); });