profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

phase 1.6: lock Gate 3b = C, reconcile docs to shipped state, fix double-upload file leak

Browse Source 
Four threads landing together — all driven by the audit J asked for before
production cutover.

(1) Gate 3b DECIDED: Option C (defer classifications). `BiometricCollection.classifications`
    stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status
    flipped from "draft / awaits product" to DECIDED. Consent template + retention
    schedule revised to remove all "automated facial-classification" / "deepface"
    language so disclosed scope matches implemented scope.

(2) Endpoint-path drift reconciled across 3 docs. `PHASE_1_6_BIPA_GATES.md`,
    `BIPA_DESTRUCTION_RUNBOOK.md`, and `biometric_retention_schedule_v1.md` had
    references to legacy `/v1/identity/subjects/*` paths (proposed under a separate
    identityd daemon, never shipped) — corrected to actual shipped routes
    `/biometric/subject/*` (catalogd-local). Schema block in PHASE_1_6_BIPA_GATES
    rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate
    (not the proposed Postgres `subjects` table).

(3) New operational artifacts:
    - `scripts/staffing/verify_biometric_erasure.sh` — checks 4 things post-erasure
      (manifest cleared, uploads dir empty, audit row matches, chain verified).
      Smoke-tested live against WORKER-2.
    - `scripts/staffing/biometric_destruction_report.sh` — monthly anonymized
      destruction-event aggregation. Smoke-tested clean.
    - `scripts/staffing/bundle_counsel_packet.sh` — tarballs the counsel-review
      packet with per-file SHA-256 manifest.
    - `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` — formal rotation procedure
      operationalized after the 2026-05-05 /tmp wipe incident.
    - `docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md` — cover note bundling
      all eng-staged BIPA docs for counsel review with per-doc questions, sign-off
      checklist, recommended review sequence.

(4) Double-upload file leak fixed in `crates/catalogd/src/biometric_endpoint.rs`.
    `verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo
    file. Investigation showed the file was 13-byte test-fixture bytes (zero PII,
    no biometric content); audit timeline showed two consecutive uploads followed
    by one erasure — the second upload had silently overwritten manifest.data_path,
    orphaning the first file. Patched `process_upload` to refuse a second upload
    with HTTP 409 + `error: "biometric_already_collected"` when
    `biometric_collection.is_some()` on the manifest. Operator must explicitly
    POST `/biometric/subject/{id}/erase` first.

    Tests: new `second_upload_without_erase_returns_409` (asserts 409 + manifest
    pointer unchanged + first file untouched on disk). Replaced
    `repeated_uploads_grow_the_chain` with `upload_erase_upload_grows_the_chain_cleanly`
    (covers the legitimate re-collection cycle: chain grows to 3 rows). Updated
    `content_type_with_parameters_accepted` to use 2 distinct subjects (was
    using 1 subject with 2 uploads to test ct parsing — would now 409).

    22/22 biometric_endpoint tests + 59/59 catalogd lib tests green post-patch.

Production posture: gateway needs `cargo build --release -p gateway` +
`systemctl restart lakehouse.service` to pick up the new 409 in live traffic.

Counsel calendar is now the only remaining blocker for first real-photo intake.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-05 06:19:40 -05:00
co-authored by Claude Opus 4.7
parent 03e8a91d97
commit b2c34b80b3
13 changed files with 1451 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PHASE_1_6_BIPA_GATES.md
+27 -17
View File
@@ -22,7 +22,7 @@ Each gate is a deliverable that must ship before real-photo intake. None is opti
- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file
- Linked from public privacy policy at the deployment URL
- Specifies:
- Categories of biometric data collected (facial geometry derived from candidate photos, age estimate, gender classification, race classification — per Phase 1.5 deepface walk)
- Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`)
- Purpose of collection (identity matching for staffing operations)
- Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin)
- Destruction procedure: per Gate 5 below
@@ -67,7 +67,7 @@ Each gate is a deliverable that must ship before real-photo intake. None is opti
**What ships:**
A new endpoint (proposed: `POST /v1/identity/subjects/{candidate_id}/photo`) with the following behavior:
An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior:
1. Caller authenticates with service-tier token
2. Endpoint queries identityd for `subjects.biometric_consent_status`
@@ -75,18 +75,23 @@ A new endpoint (proposed: `POST /v1/identity/subjects/{candidate_id}/photo`) wit
4. If status = `'given'`:
a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`)
b. deepface tagging runs against the photo
c. Classifications (gender, race, age) stored to `subjects` table fields (NEW columns — see schema additions below)
c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1.
d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule
e. `pii_access_log` row written with `purpose_token='biometric_collection'`
5. Response: `{candidate_id, retention_until, consent_version}`
**Schema additions to identityd `subjects`:**
**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):**
```sql
ALTER TABLE subjects ADD COLUMN biometric_classifications JSONB; -- {gender, race, age} from deepface
ALTER TABLE subjects ADD COLUMN biometric_data_path TEXT; -- quarantined path
ALTER TABLE subjects ADD COLUMN biometric_collected_at TIMESTAMPTZ;
ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the photo bytes (for integrity, NOT for re-derivation)
The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/<id>.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option<JSON>`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type.
```rust
// crates/catalogd/src/subject_manifest.rs (paraphrased)
pub struct BiometricCollection {
pub data_path: String, // quarantined path
pub template_hash: String, // SHA-256 of original bytes (integrity, NOT re-derivation)
pub collected_at: DateTime<Utc>,
pub classifications: Option<Value>, // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md)
}
```
**Engineering acceptance:**
@@ -130,8 +135,8 @@ ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the
- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing
- Specifies:
- Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request
- Procedure: identityd `POST /v1/identity/subjects/{id}/erase` (legal-tier auth)
- Erasure scope: `subjects.biometric_*` columns ciphertext-deleted, `biometric_data_path` files securely overwritten + unlinked, deepface classifications nulled
- Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly
- Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads/<id>/*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails
- Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed
- Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction)
- Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request
@@ -140,7 +145,7 @@ ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the
**Engineering acceptance:**
- Runbook committed
- `POST /v1/identity/subjects/{id}/erase` endpoint includes biometric-specific erasure path
- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full)
- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock)
- Erasure events are logged with cryptographic attestation
@@ -188,7 +193,7 @@ of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code:
|---|---|---|---|---|
| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** |
| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** |
| 3 | Photo-upload endpoint with consent enforcement | DONE for the consent-gate substrate (`crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 10 unit tests, live-verified end-to-end). Deepface classification deferred to **Gate 3b** (own session — needs Python subprocess design after sidecar drop). | n/a until 3b | **3a DONE, 3b deferred** |
| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option<JSON> = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** |
| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** |
| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** |
@@ -212,10 +217,14 @@ re-promotes to blocking and a separate training program must be authored.
expected operator population size, or restore it to the blocking set.
**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel
review of the engineering scaffolds. Gate 3 (photo-upload endpoint)
is the only remaining engineering work; it's deferred to its own
session because it crosses into identityd photo intake and deepface
integration scope that hasn't been designed yet.
review of the engineering scaffolds. Gate 3 substrate is fully
shipped; Gate 3b deepface classification was DECIDED on 2026-05-05
as Option C (defer) — `BiometricCollection.classifications` stays
`None` in v1, consent + retention docs revised to match this
narrower scope. If a future product requirement surfaces a real
need for classifications, the substrate is forward-compatible
(`Option<JSON>`) and either Option A (~1 day) or Option B (~5 days)
of the design doc can be picked up then under a v2 consent template.
---
@@ -258,4 +267,5 @@ integration scope that hasn't been designed yet.
## Change log
- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup.
- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill.
docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md
+260
View File
@@ -0,0 +1,260 @@
# Counsel Review Packet — Phase 1.6 BIPA Pre-Launch
**Date assembled:** 2026-05-05
**For:** outside counsel
**From:** J, operator of record
**Scope:** documents that engineering has staged for legal sufficiency review
before the staffing platform begins collecting any real candidate
biometric data (BIPA §15(a)(b)).
> **What this packet is.** The Phase 1.6 BIPA gates outline what
> engineering must ship before real-photo intake. As of 2026-05-05,
> all engineering substrate is shipped and verified live (see §1
> below for the inventory). What remains is binding-text authoring
> + counsel sign-off on five documents, plus operational notification
> obligations counsel may want to layer on top.
>
> **What this packet is NOT.** Not a request for counsel to write
> binding text from scratch. The documents are eng-staged in
> reasonable plain language; the request is for counsel to render
> them into legally-sufficient text and attest where signatures
> are required.
---
## 1. Engineering substrate — shipped + verified
For factual context on what counsel is reviewing AGAINST. None of
this requires sign-off here; it's the system the documents bind to.
| Component | Where it lives | Verification |
|---|---|---|
| Subject manifest registry | `crates/catalogd/src/registry.rs`, `data/_catalog/subjects/<id>.json` | 17 unit tests + 100 backfilled WORKER manifests in production |
| Per-subject HMAC audit chain (SHA-256) | `crates/catalogd/src/subject_audit.rs`, `data/_catalog/subjects/<id>.audit.jsonl` | Tamper-detection + concurrent-append race tests pass |
| Photo upload (consent-gated) | `POST /biometric/subject/{id}/photo` | 11 unit tests + live roundtrip 200 |
| Erasure (two-scope) | `POST /biometric/subject/{id}/erase` (`biometric_only` / `full`) | 21 unit tests; transactional rollback on audit failure |
| Legal-tier audit read | `GET /audit/subject/{id}` (X-Lakehouse-Legal-Token header) | Constant-time auth, chain re-verification per request |
| Retention sweep (BIPA-aware clock) | `crates/catalogd/src/bin/retention_sweep` | 8 unit tests; live verified against 100 backfilled subjects |
| Cross-runtime parity (Rust ↔ Go) | `scripts/cutover/parity/subject_audit_parity.sh` | 6/6 byte-identical assertions pass |
**Key insight for counsel:** the audit chain is the BIPA-defensible
substrate. Every state-changing event (consent given, photo uploaded,
photo erased, legal-tier read) appends to a per-subject HMAC-chained
JSONL log. The chain verifies end-to-end on every legal-tier read.
A tampered chain is detectable; a forged chain requires the HMAC
signing key, which is held under root-only mode 0400 at
`/etc/lakehouse/subject_audit.key` and rotated per the runbook in
attachment §6 below.
**Gate 3b (deepface classification) — decided 2026-05-05: Option C
(defer).** The system collects only the photograph, not derived
demographic information. The consent template + retention schedule
in this packet were revised the same day to match.
---
## 2. Documents requiring counsel review + sign-off
In recommended review order:
| # | Document | Path | Counsel ask | Sign-off |
|---|---|---|---|---|
| A | Biometric Retention Schedule v1 | `docs/policies/consent/biometric_retention_schedule_v1.md` | Render into binding language; confirm 18-month operational ceiling vs. BIPA 3-year statutory cap | Counsel + J |
| B | Biometric Consent Template v1 | `docs/policies/consent/biometric_consent_template_v1.md` | Render Disclosures 1-3 into binding consent language; specify electronic vs. paper signature mechanism | Counsel + J |
| C | BIPA Destruction Runbook | `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` | Confirm 30-day SLA from trigger; confirm two-operator (operator + witness) requirement; confirm legal-hold check procedure | Counsel attestation |
| D | BIPA Pre-IdentityD Attestation | `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md` | Sign as countersigning party; J signs as operator-of-record | Counsel + J |
| E | Legal-Tier Audit Key Rotation | `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` | Confirm rotation cadence; opine on candidate-notification obligation when rotation is compromise-driven | Counsel notes |
| F | Gate 3b Deepface Design (FYI) | `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` | Decision-of-record showing classifications were *deliberately deferred*, not omitted by oversight. No sign-off needed; provided for audit-trail completeness. | None |
The five documents requiring sign-off are A, B, C, D, E. Document F
is included so the audit trail shows the Gate 3b decision was
deliberate.
---
## 3. Specific questions for counsel — by document
### Document A — Retention Schedule
1. The schedule sets an **18-month** operational ceiling against the
BIPA 3-year statutory cap. Is the safety margin appropriate, or
should we move to a tighter window (12 months) given the
plaintiff-friendly Illinois posture?
2. The schedule references the **catalogd-local** storage substrate
rather than a separate identityd Postgres table. Does the
public-facing language need to mention the storage architecture
at all, or is "we keep the photo and a SHA-256 hash" sufficient?
3. Public publication URL — counsel to specify (placeholder marked
in §7 of the schedule).
4. Confirm whether existing consent under v1 carries forward when
a future v2 is published, or whether re-consent is required.
### Document B — Consent Template
1. Disclosure 1 says "we do NOT run automated facial-classification
in v1." Does that disclosure need to mention the *possibility* of
future classification, or is silence-with-supersession-clause
adequate?
2. Plain-language summary in §1 — counsel to confirm it's appropriate
to include alongside the binding disclosure, or recommend an
alternative comprehension aid.
3. Withdrawal SLA is set to **30 days** in §2. Counsel to confirm
against jurisdiction (Illinois primary; secondary deployments
would inherit).
4. Contact for withdrawal — counsel to specify the channel
(placeholder in §3).
5. Sign-off mechanism: electronic signature service, in-app
click-acceptance with timestamp, paper form? Each has different
evidentiary weight.
### Document C — Destruction Runbook
1. Confirm 30-day SLA from each of four triggers (retention expiry,
consent withdrawal, RTBF, court order). Some interpretations
prefer 7 or 14 days for withdrawal/RTBF.
2. Two-operator requirement (operator-of-record + witness): is the
witness role acceptable for counsel's defensibility view, or
should we elevate to dual-control with cryptographic split-key?
3. Legal-hold check procedure (§2 step 3) — counsel to specify the
actual procedure for confirming no hold is in force before
erasing.
4. Backup-window disclosure (§4) — confirm 30-day backup retention
is acceptable.
5. Candidate notification template (§3 step 4) — counsel to supply.
### Document D — Pre-IdentityD Attestation
1. Both signature lines blank — J signs as operator-of-record;
counsel signs as the countersigning legal party.
2. The attestation hash anchors the evidence; once signed, the
hash itself becomes a tamper-evident witness. Counsel to confirm
storage location for the signed copy (firm files?).
### Document E — Key Rotation Runbook
1. Recommended rotation cadence — 90 days suggested in §1.
Counsel to confirm or override.
2. Custody schedule for `/etc/lakehouse/_archived/` raw key files —
§7.2 question; suggested 1-year retention but counsel-driven.
3. Candidate-notification obligation when rotation is
compromise-driven (§7.3) — counsel call.
---
## 4. Engineering changes counsel should know about (recent)
These reconciled doc/code drift after a rapid wave on 2026-05-03:
- **Endpoint paths:** the original v1 spec proposed
`/v1/identity/subjects/*` under a separate identityd daemon. That
daemon was collapsed into catalogd; endpoints actually shipped at
`/biometric/subject/*` (catalogd-local). Documents in this packet
reference the catalogd-local routes; legacy references in
`IDENTITY_SERVICE_DESIGN.md` are flagged "do NOT implement
as-written" in that doc's deprecation header.
- **No identityd Postgres database:** the original spec proposed
encrypted-at-rest Postgres + HashiCorp Vault + S3 Object Lock for
PII storage. The shipped substrate is local JSON manifests +
per-subject HMAC-chained JSONL, sized for J's local-only
deployment per `PRD.md` line 70 ("Everything runs locally — no
cloud APIs").
- **Gate 3b deferral (Option C, 2026-05-05):** classifications
(gender / race / age inference) were deliberately deferred. The
consent template and retention schedule in this packet do NOT
disclose collection of derived demographic data, because we are
not collecting it. If a future product requirement reverses this,
we will publish a v2 consent + v2 retention with re-consent.
- **Key rotation 2026-05-05:** the prior `LH_SUBJECT_AUDIT_KEY` was
lost when a `/tmp` wipe on reboot disabled the audit and biometric
endpoints. The new key is at `/etc/lakehouse/subject_audit.key`
(mode 0400). Pre-rotation audit chains tamper-detect under the
new key — this is correct, expected behavior, not a bug.
---
## 5. Open eng items NOT awaiting counsel
For transparency. These are engineering work items, not legal items:
1. **Residual photo unlink on erasure.** During verification of the
one historical erasure event (`WORKER-2`), the verify script
surfaced a stranded photo file that was not unlinked when
`BiometricCollection` was cleared from the manifest. Engineering
investigates; if the bug is real, the fix is `crates/catalogd/src/biometric_endpoint.rs`
in the erasure handler. This does NOT affect the current packet —
no real candidate photos have been collected yet (per §1
attestation), so the residual is from a synthetic test event.
2. **Phase 1.6 §3 employee training.** Currently deferred per
acknowledgement coverage in §7 of the destruction runbook
(single-operator population). Re-promotes to blocking if the
operator population grows; counsel may want to opine on the
threshold.
---
## 6. Sign-off sequence
Recommended order so a hold-up on one doc doesn't block others:
1. **First wave (parallel):** A (retention schedule) + B (consent
template). These two have the tightest interdependence (consent
v1 references retention v1 by hash); review them together.
2. **Second wave:** C (destruction runbook). Depends on A's retention
period being fixed.
3. **Third wave:** D (pre-identityd attestation). Sign once A + B + C
are settled; the attestation snapshot is the boundary between
pre-Phase-1.6 and post-Phase-1.6 system state.
4. **Fourth wave:** E (key rotation). Independent of A-D; can be
reviewed in parallel any time.
---
## 7. After sign-off — engineering steps
Once each document is signed:
| Document | Engineering action | Trigger |
|---|---|---|
| A retention schedule | Hash + commit; reference in `consent_versions` table | Counsel signature |
| B consent template | Hash + commit; reference in candidate-facing intake UI | Counsel signature |
| C destruction runbook | Adopt; operator acknowledgment recorded in §7 | Counsel attestation |
| D pre-identityd attestation | Anchor hash to filesystem + git; counsel keeps original signature | Both signatures |
| E key rotation | Adopt; rotation event log seeded with counsel-approved cadence | Counsel notes |
The HARD blocker for first real-candidate photo collection is
A + B + D signed. C and E are operationally important but do not
block the *first* photo (they govern destruction + key handling
which apply to any state, not the boundary state).
---
## 8. Cover-note hash
This packet is itself a snapshot. Future-Claude / future-J will refer
back to this packet to know what counsel saw on 2026-05-05.
**Packet attached files (referenced by path):**
- `docs/policies/consent/biometric_retention_schedule_v1.md`
- `docs/policies/consent/biometric_consent_template_v1.md`
- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
- `docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md`
- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md`
- `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`
- `docs/PHASE_1_6_BIPA_GATES.md` (the spec they all reference)
Per-file SHA-256 hashes are produced by the bundler script (next
section); the bundler also creates a tarball ready for transmission.
---
## 9. Generating the bundle for transmission
```bash
./scripts/staffing/bundle_counsel_packet.sh
```
Produces `reports/counsel/counsel_packet_<DATE>.tar.gz` with all
referenced documents + a manifest listing per-file SHA-256 hashes.
Counsel can verify file integrity on receipt by re-running
sha256sum against each file in the tarball.
docs/policies/consent/biometric_consent_template_v1.md
+10 -8
View File
@@ -3,6 +3,7 @@
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 2 (BIPA §15(b)(1)-(3))
**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before deployment
**Version:** v1 (initial; supersession requires a new version + new hash)
**Updated 2026-05-05:** Disclosure 1 + plain-language summary revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). Pending J's product confirmation of Gate 3b; if Gate 3b chooses Option A or B, this template needs counsel re-authoring.
> This is the consent template a candidate signs (electronically or
> on paper) BEFORE Lakehouse collects, stores, or processes any
@@ -25,11 +26,12 @@ content; counsel provides the legally-sufficient wording.
### Disclosure 1 — Notice of collection (§15(b)(1))
Lakehouse will collect, store, and use my **biometric identifier**
(facial geometry derived from a photograph of me) and **biometric
information** (gender, race, and age classifications derived from
that photograph by an automated facial-classification model called
deepface).
Lakehouse will collect and store my **biometric identifier** (a
photograph of me from which facial geometry is implicit). The
photograph itself is the data we keep — we do NOT run automated
facial-classification (gender / race / age inference) against it
in v1. If at a later date we add automated classification, we will
re-collect consent under a superseding template before doing so.
### Disclosure 2 — Specific purpose and length of term (§15(b)(2))
@@ -66,9 +68,9 @@ is appropriate to include or whether a different plain-language
section is preferred.
> **What you're agreeing to:** if you upload a photo of yourself,
> we'll keep that photo and a few descriptive labels about the photo
> (estimated age, perceived gender, perceived race) to help your
> staffing coordinator recognize you when you arrive at job sites.
> we'll keep that photo so your staffing coordinator can recognize
> you when you arrive at job sites. We don't run automated guesses
> about your age, gender, or race against the photo.
>
> **How long we keep it:** at most 18 months after your last
> placement or interaction with us, then it's permanently destroyed.
docs/policies/consent/biometric_retention_schedule_v1.md
+18 -13
View File
@@ -3,6 +3,7 @@
**Spec:** docs/PHASE_1_6_BIPA_GATES.md §1 Gate 1 (BIPA §15(a))
**Status:** Engineering scaffold — ⚖ COUNSEL must author the binding text before public publication
**Version:** v1 (initial; supersession requires a new version + new hash)
**Updated 2026-05-05:** §1 + §2 revised to match the Gate 3b deferral recommendation in `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` (Option C — defer classifications). §5 destruction-trigger endpoint corrected to the shipped catalogd-local route. Pending J's product confirmation of Gate 3b.
> This is a publicly-available retention schedule for biometric identifiers
> and biometric information collected by the Lakehouse staffing platform.
@@ -15,12 +16,15 @@
This schedule applies to:
- **Biometric identifiers** as defined in 740 ILCS 14/10: facial geometry
derived from candidate photographs.
- **Biometric identifiers** as defined in 740 ILCS 14/10: candidate
photographs from which facial geometry is implicit.
- **Biometric information** as defined in 740 ILCS 14/10: any information
derived from a biometric identifier, including but not limited to
the gender, race, and age classifications produced by the deepface
model when applied to a candidate photograph.
derived from a biometric identifier. **In v1 of this schedule, no
derived information is collected** — automated facial-classification
(gender, race, age inference) is deferred per
`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` Option C. If a future version
of this schedule introduces classification, that is a superseding
v2 schedule with re-consent under the matching v2 consent template.
**Out of scope** (explicitly NOT biometric data under this schedule):
@@ -39,14 +43,15 @@ This schedule applies to:
| Category | Source | Storage location |
|---|---|---|
| Photograph (raw bytes) | Candidate upload via the consent-gated photo endpoint | Quarantined under `data/biometric/uploads/<candidate_id>/<ts>.<ext>`; encrypted at rest |
| Facial geometry classifications | deepface inference run against the photograph | `subjects.biometric_classifications` (JSONB on the identityd `subjects` row) |
| Photograph integrity hash | SHA-256 of the original bytes | `subjects.biometric_template_hash` |
| Photograph (raw bytes) | Candidate upload via the consent-gated photo endpoint | Quarantined under `data/biometric/uploads/<candidate_id>/<ts>_<uuid>.<ext>`; mode 0700 dir / 0600 file |
| Photograph integrity hash | SHA-256 of the original bytes | `SubjectManifest.biometric_collection.template_hash` (catalogd JSON manifest at `data/_catalog/subjects/<id>.json`) |
We do NOT collect raw biometric template vectors that could be used
to re-derive a face from the encoded form. The deepface output is
stored as discrete classification labels (e.g. `{"age_estimate": 32,
"gender": "...", "race": "..."}`), not as a re-identifiable embedding.
to re-derive a face from the encoded form. We do NOT run automated
facial-classification (gender, race, age inference) in v1 — see
`docs/specs/GATE_3B_DEEPFACE_DESIGN.md` for the deferral rationale.
The `BiometricCollection.classifications` field on the subject
manifest exists in the schema but is `None` for every subject.
---
@@ -104,8 +109,8 @@ Runbook** (`docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`) when:
- Retention period under §4 expires
- Candidate withdraws biometric consent under the consent template (Gate 2)
- Candidate exercises a right-to-be-forgotten request
- An identityd `POST /v1/identity/subjects/{id}/erase` is invoked under
legal-tier authentication
- A catalogd-local `POST /biometric/subject/{id}/erase` is invoked
under legal-tier authentication (shipped `848a458`)
Every destruction event is recorded as an append-only audit row in
the affected subject's per-subject HMAC-chained audit log (see
docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md
+27 -11
View File
@@ -66,10 +66,11 @@ Before initiating destruction, the operator MUST:
Invoke the legal-tier erasure endpoint:
```bash
curl -sf -X POST "http://localhost:3100/v1/identity/subjects/${CANDIDATE_ID}/erase" \
curl -sf -X POST "http://localhost:3100/biometric/subject/${CANDIDATE_ID}/erase" \
-H "Authorization: Bearer $(cat /etc/lakehouse/legal_audit.token)" \
-H "Content-Type: application/json" \
-d '{
"scope": "biometric_only|full",
"trigger": "retention_expiry|consent_withdrawal|rtbf|court_order",
"trigger_evidence_path": "<path to signed artifact>",
"operator_of_record": "<operator name>",
@@ -77,17 +78,25 @@ curl -sf -X POST "http://localhost:3100/v1/identity/subjects/${CANDIDATE_ID}/era
}'
```
⚖ ENGINEERING — `POST /v1/identity/subjects/{id}/erase` is Phase 1.6
Gate 3 dependent. Until it ships, the manual procedure is:
The endpoint is **shipped** (commit `848a458`, 21 unit tests). It is
served from catalogd-local at `/biometric/subject/{id}/erase` (the
original v1 spec proposed `/v1/identity/subjects/{id}/erase` under a
separate identityd daemon — that daemon was collapsed into catalogd
per the architecture pivot).
a. Set `SubjectManifest.consent.biometric.status = "withdrawn"` and
`SubjectManifest.status = "erased"` via direct registry write
(operator-of-record only).
b. Securely overwrite + unlink the quarantined photo path:
`shred -uvz data/biometric/uploads/${CANDIDATE_ID}/*.jpg`
(or equivalent for the configured backend).
c. NULL the deepface classification fields on the subject row.
d. Append the destruction-event audit row (Step 2 below).
The endpoint exposes two scopes:
- **`scope: "biometric_only"`** — clears `BiometricCollection` from
the SubjectManifest (drops `data_path`, `template_hash`, and
`classifications` together) + securely unlinks the quarantined
photo file. Subject manifest itself remains. Use for retention
expiry / consent withdrawal where only biometric data must go.
- **`scope: "full"`** — full subject erasure (manifest + biometric
files). Use for court-ordered erasure or full RTBF requests.
In both scopes, the audit row is appended BEFORE photo unlink so
the chain has legal proof of intent even if the file delete fails
(transactional rollback on audit failure).
### Step 2 — Append the destruction-event audit row
@@ -224,5 +233,12 @@ training program.
## 8. Change log
- 2026-05-05 — Endpoint path reconciled with shipped state:
`/v1/identity/subjects/{id}/erase` (legacy proposal under a
separate identityd daemon) → `/biometric/subject/{id}/erase`
(catalogd-local, shipped `848a458`). Step 1 manual-fallback
block removed (the endpoint is no longer "TODO"). Two-scope
body shape (`biometric_only` / `full`) documented to match
the implementation.
- 2026-05-03 — Initial scaffold. ⚖ COUNSEL review required before
adoption.
docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md
+308
View File
@@ -0,0 +1,308 @@
# Legal-Tier Audit Key & Token Rotation Runbook
**Spec companion:** `docs/PHASE_1_6_BIPA_GATES.md` §2 + `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`
**Audience:** Operators with root on the gateway host (J + named operators)
**Status:** Engineering-authored — ⚖ counsel review encouraged before formal adoption
> This runbook covers rotation of the two crypto-credentials that gate
> the Phase 1.6 audit substrate:
>
> 1. **`LH_SUBJECT_AUDIT_KEY`** — the 32-byte HMAC-SHA256 signing key
> that chains every per-subject audit row. If this key changes, all
> pre-rotation chain rows tamper-detect under the new key. That is
> correct, expected, BIPA-defensible behavior — the chain integrity
> it provided pre-rotation remains intact in the archive of the old
> key, and post-rotation chains remain intact going forward.
>
> 2. **`LH_LEGAL_AUDIT_TOKEN`** — the 32+-character bearer token that
> authorizes calls to `/audit/subject/{id}` and
> `/biometric/subject/{id}/erase`. Rotation does NOT touch any audit
> history; only access to the legal-tier endpoints flips.
>
> Both live at `/etc/lakehouse/` (mode 0400, owned by root) and are
> loaded by the gateway via systemd `Environment=` directives in
> `/etc/systemd/system/lakehouse.service.d/audit_env.conf`. They are
> NOT loaded from `/tmp` — a 2026-05-05 reboot incident wiped a
> `/tmp`-resident key and caused `/audit` + `/biometric` to fail-closed
> (which is what they should do); the rotation fix moved them to the
> persistent path.
---
## 1. When to rotate
Rotate **immediately** when any of the following is true:
| Trigger | Urgency | Notes |
|---|---|---|
| Suspected operator credential compromise | Within 1 hour | Token mismatch is fail-closed by default; immediate rotation closes the window. |
| Operator with legal-tier access leaves the team | Within 24 hours | Treat as compromise. |
| Key/token file's filesystem permissions were ever weakened (mode > 0400, group readable, etc.) | Within 24 hours | Filesystem audit may have leaked the bytes. |
| Token was ever transmitted over an untrusted channel (printed in CI log, sent over SMS, etc.) | Within 24 hours | Same reasoning. |
| Scheduled rotation (recommended) | Every 90 days | BIPA does not mandate a rotation cadence; counsel may set one. |
Do **not** rotate when:
- A subject's audit chain tamper-detects in isolation. That is normal
if the audit log was edited (which would itself be the BIPA finding,
not the key). Investigate the chain, not the key.
- Cross-runtime parity drift appears. That's an HMAC-input-shape bug
(Go vs Rust serialization), not a key issue. See
`STATE_OF_PLAY.md` "three runtime-divergence classes" entry.
---
## 2. Pre-rotation checks (5 minutes)
Before generating new credentials, capture a clean baseline so you can
prove the rotation cause and sequence afterward.
### 2.1. Take the engineering snapshot
```bash
# Confirm the canonical files exist with correct permissions.
ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
# Hash the existing key + token (NEVER the bytes themselves) so the
# old credential is identifiable in retrospect without storing it.
sha256sum /etc/lakehouse/subject_audit.key
sha256sum /etc/lakehouse/legal_audit.token
# Confirm the gateway is currently using these files.
sudo systemctl cat lakehouse.service | grep -E "Environment.*AUDIT"
# Verify the audit endpoint is healthy with the current credentials.
curl -sf http://localhost:3100/audit/health
```
If `/audit/health` is already 503, the rotation is **recovery**, not
preventive — note this in the rotation event record (§5).
### 2.2. Capture a known-good chain root
Pick one or two subjects with non-empty audit logs and record their
chain roots **under the current key**:
```bash
TOKEN=$(cat /etc/lakehouse/legal_audit.token)
for cid in WORKER-2 WORKER-100; do
curl -sf -H "X-Lakehouse-Legal-Token: $TOKEN" \
"http://localhost:3100/audit/subject/$cid" \
| jq '{cid: .candidate_id, verified: .audit_log.chain_verified, root: .audit_log.chain_root, rows: .audit_log.chain_rows_total}'
done
```
Save the output. Post-rotation, those chains will tamper-detect under
the new key — that is **expected** and the saved snapshot is the proof
that the chain WAS intact under the old key, before rotation.
---
## 3. Generation + rotation
### 3.1. Generate the new key
```bash
# 32 random bytes as hex = 64 chars. Either format works for HMAC-SHA256;
# we follow the existing convention (44-char base64-ish with no padding).
sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
/etc/lakehouse/subject_audit.key.new
sudo install -m 0400 -o root -g root <(openssl rand -base64 33 | tr -d '\n=' | head -c 44) \
/etc/lakehouse/legal_audit.token.new
# Sanity: confirm 44-char content + correct mode.
sudo wc -c /etc/lakehouse/subject_audit.key.new /etc/lakehouse/legal_audit.token.new
sudo ls -la /etc/lakehouse/*.new
```
Both must be `mode 0400`, owned by root, exactly **44 chars** (the
audit endpoint refuses tokens shorter than 32 chars at load — see
`crates/catalogd/src/audit_endpoint.rs:73`).
### 3.2. Atomic swap
The gateway reads these files **once at boot** (per
`crates/catalogd/src/audit_endpoint.rs::AuditEndpointState::new` and
the equivalent for the writer). Atomic mv → restart is required.
```bash
# Move the old credentials to a quarantine path with timestamp so the
# old hashes remain identifiable post-rotation.
TS=$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p /etc/lakehouse/_archived
sudo install -d -m 0700 -o root -g root /etc/lakehouse/_archived
sudo mv /etc/lakehouse/subject_audit.key /etc/lakehouse/_archived/subject_audit.key.$TS
sudo mv /etc/lakehouse/legal_audit.token /etc/lakehouse/_archived/legal_audit.token.$TS
sudo mv /etc/lakehouse/subject_audit.key.new /etc/lakehouse/subject_audit.key
sudo mv /etc/lakehouse/legal_audit.token.new /etc/lakehouse/legal_audit.token
sudo ls -la /etc/lakehouse/subject_audit.key /etc/lakehouse/legal_audit.token
```
### 3.3. Restart the gateway
```bash
sudo systemctl restart lakehouse.service
sleep 2
sudo systemctl status lakehouse.service --no-pager | head -10
```
Wait for the gateway to bind port 3100 cleanly. If it doesn't, check
`journalctl -u lakehouse.service -n 50 --no-pager` for the failure
mode — the most common cause is the new file having wrong mode/owner.
---
## 4. Post-rotation verification (5 minutes)
### 4.1. Health probes
```bash
# Audit endpoint must be 200, not 503.
curl -sf http://localhost:3100/audit/health
# Expect: "audit endpoint ready"
# /v1/health must list the gateway's full provider set.
curl -sf http://localhost:3100/v1/health | jq '.providers, .worker_count'
```
### 4.2. Confirm the new token works
```bash
NEW_TOKEN=$(cat /etc/lakehouse/legal_audit.token)
curl -sS -o /dev/null -w '%{http_code}\n' \
-H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-100
# Expect: 200
```
If 401, the file the gateway loaded does NOT match the file you wrote.
Check ownership / mode / for trailing whitespace differences with
`hexdump -C /etc/lakehouse/legal_audit.token | head`.
### 4.3. Confirm the new chain works
Append-only chains are key-tied. Any *new* audit row written
post-rotation is signed under the new key and verifies cleanly:
```bash
# Issue a /v1/validate call against any worker — it spawns an audit row.
curl -sf -X POST http://localhost:3100/v1/validate \
-H 'Content-Type: application/json' \
-d '{"mode":"fill","candidate_id":"WORKER-100","worker_id":"WORKER-100","fields":["exists"]}' >/dev/null
# Read the chain back. Last row must verify under the new key.
curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-100 \
| jq '.audit_log | {verified: .chain_verified, rows: .chain_rows_total, last_kind: .rows[-1].accessor.kind}'
```
`chain_verified: true` confirms the new key is signing + verifying.
### 4.4. Confirm pre-rotation chains tamper-detect (expected)
```bash
curl -sf -H "X-Lakehouse-Legal-Token: $NEW_TOKEN" \
http://localhost:3100/audit/subject/WORKER-2 \
| jq '.audit_log | {verified: .chain_verified, error: .chain_verification_error}'
```
For any subject whose chain was written under the old key, this
returns `chain_verified: false` with an HMAC-mismatch error. **This
is correct behavior**, not a bug. The old chain was correctly signed
under the old key and verified under it; the new key cannot retroactively
verify rows it didn't sign. The pre-rotation snapshot you captured in
§2.2 is the defensible proof that those rows WERE valid pre-rotation.
If, instead, you see a chain that *should* verify post-rotation
returning `verified: false`, that's the rotation having gone wrong —
likely an old-key file that didn't get archived cleanly. Restore from
`/etc/lakehouse/_archived/<ts>/`, then re-attempt.
---
## 5. Record the rotation event
Append a row to the rotation log:
```bash
sudo tee -a /etc/lakehouse/_archived/rotation_log.jsonl <<EOF
{"ts":"$(date -u +%Y-%m-%dT%H:%M:%SZ)","operator":"<your name>","reason":"<scheduled|compromise|cred_loss|recovery>","old_key_sha256":"<hash from §2.1>","new_key_sha256":"$(sha256sum /etc/lakehouse/subject_audit.key | awk '{print $1}')","old_token_sha256":"<hash from §2.1>","new_token_sha256":"$(sha256sum /etc/lakehouse/legal_audit.token | awk '{print $1}')","witness":"<witness name or N/A for routine>"}
EOF
sudo chmod 0600 /etc/lakehouse/_archived/rotation_log.jsonl
sudo chown root:root /etc/lakehouse/_archived/rotation_log.jsonl
```
This file is the operator-side record of when the key changed and why.
It does NOT contain the key itself — only hashes — so it is safe to
back up and share with counsel on request.
---
## 6. Recovery from a lost key
If the active `subject_audit.key` is destroyed (filesystem corruption,
accidental delete, /tmp wipe per the 2026-05-05 incident), the gateway
will fail-closed at startup:
- `/audit/subject/{id}` → 503 ("audit endpoint disabled (legal token missing)" or equivalent for the signing key)
- `/biometric/subject/{id}/photo` → 503 (same fail-closed posture)
This is correct behavior — a server that cannot HMAC-sign new audit
rows must not accept new biometric writes.
**Recovery is rotation.** Generate a new key per §3.1, atomic-swap
per §3.2, restart per §3.3, verify per §4. Pre-loss chains tamper-detect
under the new key (the old key is gone — there is no way to verify
them). Treat the loss event as the BIPA-defensible boundary: pre-loss
chain verification was provided by the working key; post-loss new
chains are signed under the new key.
If a counsel-grade attestation of the pre-loss chains is needed, the
`/etc/lakehouse/_archived/` folder contains the historical hashes;
combined with the cross-runtime parity probe (Go reader gives the
same byte-identical view as Rust), the chain history pre-loss is
preservable as long as the on-disk JSONL files were not also lost.
---
## 7. ⚖ counsel notes
These are areas where counsel may want to opine before this runbook
is formally adopted:
1. **Rotation cadence.** BIPA itself does not require periodic rotation;
counsel may set a 90-day schedule to satisfy a separate compliance
posture (SOC2, internal policy).
2. **Custody of `/etc/lakehouse/_archived/`.** The archived hashes do
NOT contain the keys, but the archived raw key files DO. Counsel
may want a more aggressive destruction schedule for the raw archived
keys — say 1 year — to reduce a long-tail compromise surface.
3. **Notification obligations on rotation due to compromise.** §1
triggers a rotation; §1 does not address whether candidates whose
biometric data was protected by the compromised key must be notified.
This is a counsel call.
---
## 8. Operator acknowledgment
| Operator | Date acknowledged | Signature |
|---|---|---|
| J | _____ | _______________ |
| _____ | _____ | _______________ |
---
## 9. Change log
- 2026-05-05 — Initial runbook authored after the /tmp wipe incident
on the same day (key was at `/tmp/subject_audit.key` and was deleted
on reboot, disabling `/audit` + `/biometric` until the key was
regenerated at `/etc/lakehouse/subject_audit.key`). Recovery of
that incident produced a working procedure; this runbook captures
it as the canonical playbook for any future rotation.
docs/specs/GATE_3B_DEEPFACE_DESIGN.md
+5 -1
View File
@@ -1,6 +1,8 @@
# Gate 3b — Deepface Classification Integration (Design)
**Status:** Design draft — 2026-05-03 morning · **Companion to:** [`PHASE_1_6_BIPA_GATES.md`](../PHASE_1_6_BIPA_GATES.md) Gate 3 · **Depends on:** Gate 3a (photo upload) which is shipped (`f1fa6e4`)
**Status:** **DECIDED 2026-05-05 — Option C (defer classifications)** · Original design draft 2026-05-03 morning · **Companion to:** [`PHASE_1_6_BIPA_GATES.md`](../PHASE_1_6_BIPA_GATES.md) Gate 3 · **Depends on:** Gate 3a (photo upload) which is shipped (`f1fa6e4`)
> **Decision summary (2026-05-05):** J accepted Option C. `BiometricCollection.classifications` remains `Option<JSON> = None` in v1. The consent template and retention schedule were revised the same day to remove all "automated facial-classification" language so the disclosed scope matches the implemented scope. If a real product requirement for classifications surfaces later, this doc's Option A (Python subprocess) or Option B (ONNX-in-Rust) is picked up under a v2 consent template + v2 retention schedule.
> **What this is.** Three options for how `BiometricCollection.classifications` (currently `Option<JSON>`, always `None`) gets populated by an automated facial-attribute classifier. Phase 1.6 Gate 3a ships the consent-gated upload + audit chain + transactional rollback; Gate 3b adds the classification step. The substrate is ready — what's missing is the design choice for HOW classification happens.
>
@@ -152,6 +154,8 @@ Reasoning:
⚖ J — pick A / B / C. The substrate accommodates any choice; the cost is the design-doc → counsel-coordination → engineering loop, which differs by an order of magnitude across the options.
**[2026-05-05] J's decision: Option C.** Reasoning recorded in change log below. Consent + retention doc revisions for Option C shipped same day; counsel review of revised text is the remaining work.
---
## Open questions for J