phase 1.6: lock Gate 3b = C, reconcile docs to shipped state, fix double-upload file leak
Four threads landing together — all driven by the audit J asked for before
production cutover.
(1) Gate 3b DECIDED: Option C (defer classifications). `BiometricCollection.classifications`
stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status
flipped from "draft / awaits product" to DECIDED. Consent template + retention
schedule revised to remove all "automated facial-classification" / "deepface"
language so disclosed scope matches implemented scope.
(2) Endpoint-path drift reconciled across 3 docs. `PHASE_1_6_BIPA_GATES.md`,
`BIPA_DESTRUCTION_RUNBOOK.md`, and `biometric_retention_schedule_v1.md` had
references to legacy `/v1/identity/subjects/*` paths (proposed under a separate
identityd daemon, never shipped) — corrected to actual shipped routes
`/biometric/subject/*` (catalogd-local). Schema block in PHASE_1_6_BIPA_GATES
rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate
(not the proposed Postgres `subjects` table).
(3) New operational artifacts:
- `scripts/staffing/verify_biometric_erasure.sh` — checks 4 things post-erasure
(manifest cleared, uploads dir empty, audit row matches, chain verified).
Smoke-tested live against WORKER-2.
- `scripts/staffing/biometric_destruction_report.sh` — monthly anonymized
destruction-event aggregation. Smoke-tested clean.
- `scripts/staffing/bundle_counsel_packet.sh` — tarballs the counsel-review
packet with per-file SHA-256 manifest.
- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` — formal rotation procedure
operationalized after the 2026-05-05 /tmp wipe incident.
- `docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md` — cover note bundling
all eng-staged BIPA docs for counsel review with per-doc questions, sign-off
checklist, recommended review sequence.
(4) Double-upload file leak fixed in `crates/catalogd/src/biometric_endpoint.rs`.
`verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo
file. Investigation showed the file was 13-byte test-fixture bytes (zero PII,
no biometric content); audit timeline showed two consecutive uploads followed
by one erasure — the second upload had silently overwritten manifest.data_path,
orphaning the first file. Patched `process_upload` to refuse a second upload
with HTTP 409 + `error: "biometric_already_collected"` when
`biometric_collection.is_some()` on the manifest. Operator must explicitly
POST `/biometric/subject/{id}/erase` first.
Tests: new `second_upload_without_erase_returns_409` (asserts 409 + manifest
pointer unchanged + first file untouched on disk). Replaced
`repeated_uploads_grow_the_chain` with `upload_erase_upload_grows_the_chain_cleanly`
(covers the legitimate re-collection cycle: chain grows to 3 rows). Updated
`content_type_with_parameters_accepted` to use 2 distinct subjects (was
using 1 subject with 2 uploads to test ct parsing — would now 409).
22/22 biometric_endpoint tests + 59/59 catalogd lib tests green post-patch.
Production posture: gateway needs `cargo build --release -p gateway` +
`systemctl restart lakehouse.service` to pick up the new 409 in live traffic.
Counsel calendar is now the only remaining blocker for first real-photo intake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -22,7 +22,7 @@ Each gate is a deliverable that must ship before real-photo intake. None is opti
|
||||
- `docs/policies/consent/biometric_retention_schedule_v1.md` — public file
|
||||
- Linked from public privacy policy at the deployment URL
|
||||
- Specifies:
|
||||
- Categories of biometric data collected (facial geometry derived from candidate photos, age estimate, gender classification, race classification — per Phase 1.5 deepface walk)
|
||||
- Categories of biometric data collected (facial photograph for staff identification at job sites; classifications deferred per Gate 3b — see `docs/specs/GATE_3B_DEEPFACE_DESIGN.md`)
|
||||
- Purpose of collection (identity matching for staffing operations)
|
||||
- Maximum retention: BIPA §15(a) caps at "3 years from the individual's last interaction with the private entity, whichever occurs first" — recommend 18-24 months as the operational ceiling (provides safety margin)
|
||||
- Destruction procedure: per Gate 5 below
|
||||
@@ -67,7 +67,7 @@ Each gate is a deliverable that must ship before real-photo intake. None is opti
|
||||
|
||||
**What ships:**
|
||||
|
||||
A new endpoint (proposed: `POST /v1/identity/subjects/{candidate_id}/photo`) with the following behavior:
|
||||
An endpoint at `POST /biometric/subject/{candidate_id}/photo` (catalogd-local — the original v1 spec named this `/v1/identity/subjects/{candidate_id}/photo` under a separate identityd daemon; that daemon was collapsed into catalogd per the architecture pivot. See `IDENTITY_SERVICE_DESIGN.md` deprecation header.) with the following behavior:
|
||||
|
||||
1. Caller authenticates with service-tier token
|
||||
2. Endpoint queries identityd for `subjects.biometric_consent_status`
|
||||
@@ -75,18 +75,23 @@ A new endpoint (proposed: `POST /v1/identity/subjects/{candidate_id}/photo`) wit
|
||||
4. If status = `'given'`:
|
||||
a. Photo bytes accepted, stored to a quarantined path under `data/biometric/uploads/{candidate_id}/{ts}.{ext}` (NOT `data/headshots/`)
|
||||
b. deepface tagging runs against the photo
|
||||
c. Classifications (gender, race, age) stored to `subjects` table fields (NEW columns — see schema additions below)
|
||||
c. Classifications (gender, race, age) — **DEFERRED to Gate 3b** (`docs/specs/GATE_3B_DEEPFACE_DESIGN.md`). `BiometricCollection.classifications` remains `None` in v1.
|
||||
d. Original photo bytes encrypted under DEK + retained per Gate 1 schedule
|
||||
e. `pii_access_log` row written with `purpose_token='biometric_collection'`
|
||||
5. Response: `{candidate_id, retention_until, consent_version}`
|
||||
|
||||
**Schema additions to identityd `subjects`:**
|
||||
**Schema (as shipped — catalogd `SubjectManifest.biometric_collection`):**
|
||||
|
||||
```sql
|
||||
ALTER TABLE subjects ADD COLUMN biometric_classifications JSONB; -- {gender, race, age} from deepface
|
||||
ALTER TABLE subjects ADD COLUMN biometric_data_path TEXT; -- quarantined path
|
||||
ALTER TABLE subjects ADD COLUMN biometric_collected_at TIMESTAMPTZ;
|
||||
ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the photo bytes (for integrity, NOT for re-derivation)
|
||||
The original spec proposed JSONB columns on a Postgres `subjects` table under identityd. The shipped implementation collapses this into a per-subject JSON manifest at `data/_catalog/subjects/<id>.json`, with the `BiometricCollection` struct holding `data_path`, `template_hash`, `collected_at`, and `classifications: Option<JSON>`. See `crates/catalogd/src/subject_manifest.rs` for the canonical type.
|
||||
|
||||
```rust
|
||||
// crates/catalogd/src/subject_manifest.rs (paraphrased)
|
||||
pub struct BiometricCollection {
|
||||
pub data_path: String, // quarantined path
|
||||
pub template_hash: String, // SHA-256 of original bytes (integrity, NOT re-derivation)
|
||||
pub collected_at: DateTime<Utc>,
|
||||
pub classifications: Option<Value>, // None until Gate 3b ships (deferred — see GATE_3B_DEEPFACE_DESIGN.md)
|
||||
}
|
||||
```
|
||||
|
||||
**Engineering acceptance:**
|
||||
@@ -130,8 +135,8 @@ ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the
|
||||
- `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md` — operator-facing
|
||||
- Specifies:
|
||||
- Triggers: retention expiry (per Gate 1), withdrawal, RTBF request, candidate request
|
||||
- Procedure: identityd `POST /v1/identity/subjects/{id}/erase` (legal-tier auth)
|
||||
- Erasure scope: `subjects.biometric_*` columns ciphertext-deleted, `biometric_data_path` files securely overwritten + unlinked, deepface classifications nulled
|
||||
- Procedure: catalogd-local `POST /biometric/subject/{id}/erase` (legal-tier auth) — formerly proposed under identityd; now serves from catalogd directly
|
||||
- Erasure scope: `BiometricCollection` set to `None` on the subject manifest (drops `data_path`, `template_hash`, `classifications` together), quarantined photo files at `data/biometric/uploads/<id>/*` securely unlinked, audit row appended BEFORE photo unlink so the chain proves intent even if file delete fails
|
||||
- Backup window: per `IDENTITY_SERVICE_DESIGN` v3-B12, residual exists in DB backups for 30 days max; subject is informed
|
||||
- Witnessed: every erasure event written to `pii_access_log` with `purpose_token='biometric_erasure'` and the legal-tier JWT signature (proves authorized destruction)
|
||||
- Reporting: monthly internal report of erasures + retention-expiry sweeps; available to counsel on request
|
||||
@@ -140,7 +145,7 @@ ALTER TABLE subjects ADD COLUMN biometric_template_hash TEXT; -- hash of the
|
||||
|
||||
**Engineering acceptance:**
|
||||
- Runbook committed
|
||||
- `POST /v1/identity/subjects/{id}/erase` endpoint includes biometric-specific erasure path
|
||||
- `POST /biometric/subject/{id}/erase` endpoint includes biometric-specific erasure path (shipped `848a458` — 21 unit tests, two scopes: biometric_only / full)
|
||||
- Daily sweep job destroys biometric data past `biometric_retention_until` (separate from general retention sweep — biometric has stricter clock)
|
||||
- Erasure events are logged with cryptographic attestation
|
||||
|
||||
@@ -188,7 +193,7 @@ of 2026-05-03 — scaffolds vs. counsel sign-off vs. shipped code:
|
||||
|---|---|---|---|---|
|
||||
| 1 | Public retention schedule | scaffolded at `docs/policies/consent/biometric_retention_schedule_v1.md` | pending | **eng-staged** |
|
||||
| 2 | Consent template | scaffolded at `docs/policies/consent/biometric_consent_template_v1.md` | pending | **eng-staged** |
|
||||
| 3 | Photo-upload endpoint with consent enforcement | DONE for the consent-gate substrate (`crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 10 unit tests, live-verified end-to-end). Deepface classification deferred to **Gate 3b** (own session — needs Python subprocess design after sidecar drop). | n/a until 3b | **3a DONE, 3b deferred** |
|
||||
| 3 | Photo-upload endpoint with consent enforcement | DONE — `crates/catalogd/src/biometric_endpoint.rs` mounted at `/biometric/subject/{id}/photo`, 11 unit tests, live-verified end-to-end. **Gate 3b DECIDED 2026-05-05: Option C (defer classifications).** `BiometricCollection.classifications` stays `Option<JSON> = None` in v1; consent + retention docs revised to match. See `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` §6 + change log. | reviewed under Gate 2 (matching consent text) | **DONE — 3a shipped, 3b deferred per design doc** |
|
||||
| 4 | Name → ethnicity inference removed | DONE — `mcp-server/search.html:3372` removal note + `mcp-server/phase_1_6_gate_4.test.ts` absence test (3/3 green) | none required | **DONE** |
|
||||
| 5 | Destruction runbook | scaffolded at `docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md`; erasure endpoint + verify/report scripts marked TODO | pending | **eng-staged** |
|
||||
|
||||
@@ -212,10 +217,14 @@ re-promotes to blocking and a separate training program must be authored.
|
||||
expected operator population size, or restore it to the blocking set.
|
||||
|
||||
**Calendar bottleneck:** Items 1, 2, 5, 6 (and #7) await counsel
|
||||
review of the engineering scaffolds. Gate 3 (photo-upload endpoint)
|
||||
is the only remaining engineering work; it's deferred to its own
|
||||
session because it crosses into identityd photo intake and deepface
|
||||
integration scope that hasn't been designed yet.
|
||||
review of the engineering scaffolds. Gate 3 substrate is fully
|
||||
shipped; Gate 3b deepface classification was DECIDED on 2026-05-05
|
||||
as Option C (defer) — `BiometricCollection.classifications` stays
|
||||
`None` in v1, consent + retention docs revised to match this
|
||||
narrower scope. If a future product requirement surfaces a real
|
||||
need for classifications, the substrate is forward-compatible
|
||||
(`Option<JSON>`) and either Option A (~1 day) or Option B (~5 days)
|
||||
of the design doc can be picked up then under a v2 consent template.
|
||||
|
||||
---
|
||||
|
||||
@@ -258,4 +267,5 @@ integration scope that hasn't been designed yet.
|
||||
|
||||
## Change log
|
||||
|
||||
- 2026-05-05 — Reconciled with shipped state: endpoint paths corrected from the legacy identityd v1 spec (`/v1/identity/subjects/*`) to the catalogd-local routes that actually shipped (`/biometric/subject/*`). Schema block rewritten to reflect the JSON `SubjectManifest.biometric_collection` substrate that replaced the proposed Postgres columns. Gate 3b deepface deferral marked in-line where Disclosure 1 / Gate 3 step 5c / Gate 5 erasure scope previously assumed classifications were collected. No legal text changed; this was doc/code drift cleanup.
|
||||
- 2026-05-03 — Initial draft. Authored after `IDENTITY_SERVICE_DESIGN` v3 §5 Step 0 named Phase 1.6 as a hard prerequisite to backfill.
|
||||
|
||||
Reference in New Issue
Block a user