profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

phase 1.6: ops dashboard + consent_versions allowlist + subject timeline tool

Browse Source 
Closes the afternoon's "all four" wave (per J's request to do all the
items in one pass instead of pick-one-of-options):

(1) Live demo on WORKER-100 — full lifecycle exercised end-to-end
    against the running gateway. 3 audit rows landed in correct
    order (consent_grant → biometric_collection →
    consent_withdrawal), chain_verified=true, photo on disk at
    data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg
    (180 bytes JFIF). retention_until=2026-06-04 (30d from
    withdrawal per consent template v1 §2).

(2) GET /biometric/stats — read-only aggregate over all subjects.
    Returns counts by biometric.status + subject.status, photo
    count, oldest_active_retention_until, and the last 20
    state-change events (consent_grant / collection / withdrawal /
    erasure — validator_lookup and other noise filtered out).
    Walks per-subject audit logs via the existing writer; cheap
    for 100 subjects, would want an event-stream index at 100k.
    Legal-tier auth (same posture as /audit). 4 unit tests.

(3) /biometric/dashboard mcp-server frontend. Auto-refreshes
    /biometric/stats every 15s, neo-brutalist tile layout for
    the per-status counts + retention horizon block + recent
    events table with kind badges + event-kind breakdown pills.
    sessionStorage-backed token; logout button clears state.
    DOM-built throughout (textContent + createElement) — never
    innerHTML on audit-row values, since trace_id et al. could
    in theory carry operator-supplied strings.

(4) consent_versions allowlist. BiometricEndpointState gains
    `allowed_consent_versions: Option<Arc<HashSet<String>>>`,
    loaded at startup from /etc/lakehouse/consent_versions.json
    (override via LH_CONSENT_VERSIONS_FILE). process_consent
    refuses unknown hashes with HTTP 400 consent_version_unknown
    when configured. Resolution semantics:
      - Missing file → permissive (v1 compat, warn-log)
      - Parse error → permissive (error-log; broken config
        silently going strict would be worse)
      - Empty array → strict, refuse all (deliberate freeze
        mode for "counsel hasn't signed v1 yet")
      - Populated → strict, lowercase-normalized comparison
    5 unit tests (known/unknown/case/empty/none-permissive).
    Example template at ops/consent_versions.example.json with
    a counsel-tier deployment note.

(5) scripts/staffing/subject_timeline.sh — operator one-shot
    pretty-print of any subject's full BIPA lifecycle. Curls
    /audit/subject/{id} with legal token; renders manifest
    summary + on-disk photo state + chronological audit chain
    with kind badges + chain verification status. Smoke-tested
    on WORKER-100 (3 rows verified).

(6) STATE_OF_PLAY.md refresh. New section "afternoon wave"
    captures all four commits (76cb5ac, 7f0f500, 68d226c, this
    one) + the live demo evidence + the v1 endpoint matrix +
    UI/CLI inventory + the production-cutover blocking set
    (counsel calendar only — eng substrate is done).

Verified live post-restart:
- /audit/health + /biometric/health both 200
- /biometric/stats returns 100 subjects, 2 withdrawn (WORKER-2 from
  earlier scrum + WORKER-100 from today's demo), 1 photo on record,
  6 recent state-change events
- /biometric/intake + /biometric/withdraw + /biometric/dashboard
  all 200 on mcp-server :3700
- subject_timeline.sh on WORKER-100: chain_verified=true,
  chain_root=a47563ff937d50de…
- 88/88 catalogd lib tests + 55/55 biometric_endpoint tests green

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-05 15:27:52 -05:00
co-authored by Claude Opus 4.7
parent 68d226c314
commit 87b034f5f9
7 changed files with 1070 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATE_OF_PLAY.md
+60 -1
View File
@@ -7,7 +7,66 @@
---
## WHAT LANDED 2026-05-05 (doc reconciliation wave — Gate 3b decision + counsel packet ready)
## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard)
After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`.
| Commit | What | Verified |
|---|---|---|
| `76cb5ac` | `POST /biometric/subject/{id}/consent` — Gate 2 backend. Flips status NeverCollected→Given, computes retention_until = given_at + 18mo (per retention schedule v1 §4), captures consent_version_hash + collection_method (closed set: electronic_signature/paper/click_acceptance) + operator + evidence path to audit row. State machine: 409 already_given, 409 post-withdrawal-requires-erase, 403 subject_inactive. | 12 unit tests; live POST returns 401/400/404 on guards |
| `7f0f500` | Candidate intake UI at `/biometric/intake?candidate_id=XXX` (mcp-server :3700, exposed via nginx at devop.live/lakehouse/biometric/intake). 4-screen flow: operator auth → consent template render + click-accept → photo capture (file or webcam getUserMedia) → confirmation showing audit hmac. SHA-256 of rendered consent block becomes consent_version_hash. sessionStorage-backed token (clears on tab close), neo-brutalist style matching onboard.html. | 22631-byte HTML, mcp-server route returns 200 |
| `68d226c` | Three things in one wave: <br>(a) `POST /biometric/subject/{id}/withdraw` — BIPA right of withdrawal. Flips Given→Withdrawn, accelerates retention_until from 18mo→30d (per consent template v1 §2 SLA), audit kind=biometric_consent_withdrawal. State machine 409s on NeverCollected/Pending (nothing_to_withdraw), Withdrawn (already_withdrawn), Expired (already_expired). 12 unit tests. <br>(b) Withdraw UI at `/biometric/withdraw` — 3-screen operator flow (token+name auth → reason+evidence form → confirmation showing 30-day clock + verify curl recipe). <br>(c) `lakehouse-retention-sweep.{service,timer}` systemd units in `ops/systemd/`. Daily 03:00 UTC, Persistent=true, install.sh updated to handle paired timer+oneshot service. <br>Plus operator_of_record bug fix in intake UI (was hardcoded `'intake_ui_operator'`). | 46/46 biometric_endpoint + 71/71 catalogd lib tests; manual sweep run: 100 subjects, 0 overdue, exit 0 |
| **(current HEAD post-this-wave)** | Stats endpoint `GET /biometric/stats` (legal-tier auth, returns subject counts by status + photo count + oldest active retention + last 20 state-change events with anonymizable trace_ids) + ops dashboard at `/biometric/dashboard` (single-page, polls /stats every 15s, table + status tiles, XSS-safe DOM construction not innerHTML). Plus consent_versions allowlist: `BiometricEndpointState.allowed_consent_versions: Option<Arc<HashSet<String>>>`, loaded from `/etc/lakehouse/consent_versions.json` (`LH_CONSENT_VERSIONS_FILE` override), missing-file = permissive (v1 compat), present + populated = strict mode (refuses unknown hashes with 400 consent_version_unknown). Plus `scripts/staffing/subject_timeline.sh` — pretty-prints any subject's full BIPA lifecycle from /audit/subject/{id} (manifest + on-disk photo + chronological audit chain + verification status). | 5 new allowlist unit tests + 4 stats tests; live demo on WORKER-100 ran end-to-end (consent → photo → withdraw, chain verified=true, chain_root=a47563ff…) |
### Live demo evidence (WORKER-100, 2026-05-05 20:12 UTC)
The full lifecycle was exercised against the live gateway as a verification artifact. The audit chain on WORKER-100 now contains 3 rows:
```
20:12:33.054 BIOMETRIC_CONSENT_GRANT result=given hmac=9c6f4153341e97d2… trace=live-demo-2026-05-05
20:12:47.957 BIOMETRIC_COLLECTION result=success hmac=856be6173c88277c… trace=live-demo-2026-05-05
20:15:27.298 BIOMETRIC_CONSENT_WITHDRAWAL result=withdrawn hmac=a47563ff937d50de… trace=live-demo-2026-05-05
```
`chain_verified=true`, chain_root = a47563ff937d50de43b09a0c903cff954233836c219a928ee8ca2aa6792272dd. Photo file at `data/biometric/uploads/WORKER-100/1778011967957907731_027b6bb1.jpg` (180 bytes — a minimal real JFIF JPEG), retention_until=2026-06-04 (= 30 days from withdrawal). Retention sweep will flag this subject on or after that date; operator runs `/biometric/subject/WORKER-100/erase` to destroy.
To re-verify: `./scripts/staffing/subject_timeline.sh WORKER-100`.
### Endpoint matrix (v1 BIPA lifecycle complete)
| Event | Endpoint | Method | Auth | Status flip | retention_until |
|---|---|---|---|---|---|
| consent given | `/biometric/subject/{id}/consent` | POST | legal | NeverCollected/Pending → Given | now + 18mo |
| photo collected | `/biometric/subject/{id}/photo` | POST | legal + consent gate | (no change) | (no change) |
| consent withdrawn | `/biometric/subject/{id}/withdraw` | POST | legal | Given → Withdrawn | now + 30d |
| destruction | `/biometric/subject/{id}/erase` | POST | legal | (manifest cleared) | n/a |
| audit read | `/audit/subject/{id}` | GET | legal | (read-only) | (read-only) |
| ops aggregates | `/biometric/stats` | GET | legal | (read-only) | (read-only) |
UIs:
- `/biometric/intake?candidate_id=X` — operator-driven consent + photo
- `/biometric/withdraw` — operator-driven withdrawal recording
- `/biometric/dashboard` — read-only ops aggregate, auto-refresh
CLI tools:
- `scripts/staffing/verify_biometric_erasure.sh <id>` — post-erasure verification
- `scripts/staffing/biometric_destruction_report.sh --month YYYY-MM` — anonymized monthly report
- `scripts/staffing/subject_timeline.sh <id>` — full lifecycle pretty-print (NEW 2026-05-05)
- `scripts/staffing/bundle_counsel_packet.sh` — counsel review tarball
- `scripts/staffing/attest_pre_identityd_biometric_state.sh` — defense attestation generator
### What's blocking production cutover NOW
**Counsel calendar.** Engineering substrate is done end-to-end: every state transition has a defensible endpoint, every endpoint has tests + live verification, every UI is reachable, retention sweep is scheduled, allowlist hardening is wired. The remaining work is signature/review:
1. Counsel review of consent template v1 (revised for Option C — classifications deferred)
2. Counsel review of retention schedule v1 (revised for Option C)
3. Counsel review of destruction runbook
4. Counsel + J signatures on §2 attestation
5. Once counsel signs the consent template, populate `/etc/lakehouse/consent_versions.json` with the signed hash to flip the gateway from permissive to strict mode
Counsel-review packet at `reports/counsel/counsel_packet_2026-05-05.tar.gz` (regenerable via `bundle_counsel_packet.sh` to pick up the latest doc state).
---
## WHAT LANDED 2026-05-05 (morning wave — doc reconciliation + Gate 3b decision + counsel packet)
This was a **doc-only wave**, not code. Background: J asked for an audit of the BIPA/biometric documentation before production cutover. Audit found moderate fragmentation between docs and shipped code (post-`identityd` collapse, post-Gate-3a-ship, pre-Gate-3b-decision). Closed it in one pass.