phase 1.6: lock Gate 3b = C, reconcile docs to shipped state, fix double-upload file leak
Four threads landing together — all driven by the audit J asked for before
production cutover.
(1) Gate 3b DECIDED: Option C (defer classifications). `BiometricCollection.classifications`
stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status
flipped from "draft / awaits product" to DECIDED. Consent template + retention
schedule revised to remove all "automated facial-classification" / "deepface"
language so disclosed scope matches implemented scope.
(2) Endpoint-path drift reconciled across 3 docs. `PHASE_1_6_BIPA_GATES.md`,
`BIPA_DESTRUCTION_RUNBOOK.md`, and `biometric_retention_schedule_v1.md` had
references to legacy `/v1/identity/subjects/*` paths (proposed under a separate
identityd daemon, never shipped) — corrected to actual shipped routes
`/biometric/subject/*` (catalogd-local). Schema block in PHASE_1_6_BIPA_GATES
rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate
(not the proposed Postgres `subjects` table).
(3) New operational artifacts:
- `scripts/staffing/verify_biometric_erasure.sh` — checks 4 things post-erasure
(manifest cleared, uploads dir empty, audit row matches, chain verified).
Smoke-tested live against WORKER-2.
- `scripts/staffing/biometric_destruction_report.sh` — monthly anonymized
destruction-event aggregation. Smoke-tested clean.
- `scripts/staffing/bundle_counsel_packet.sh` — tarballs the counsel-review
packet with per-file SHA-256 manifest.
- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` — formal rotation procedure
operationalized after the 2026-05-05 /tmp wipe incident.
- `docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md` — cover note bundling
all eng-staged BIPA docs for counsel review with per-doc questions, sign-off
checklist, recommended review sequence.
(4) Double-upload file leak fixed in `crates/catalogd/src/biometric_endpoint.rs`.
`verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo
file. Investigation showed the file was 13-byte test-fixture bytes (zero PII,
no biometric content); audit timeline showed two consecutive uploads followed
by one erasure — the second upload had silently overwritten manifest.data_path,
orphaning the first file. Patched `process_upload` to refuse a second upload
with HTTP 409 + `error: "biometric_already_collected"` when
`biometric_collection.is_some()` on the manifest. Operator must explicitly
POST `/biometric/subject/{id}/erase` first.
Tests: new `second_upload_without_erase_returns_409` (asserts 409 + manifest
pointer unchanged + first file untouched on disk). Replaced
`repeated_uploads_grow_the_chain` with `upload_erase_upload_grows_the_chain_cleanly`
(covers the legitimate re-collection cycle: chain grows to 3 rows). Updated
`content_type_with_parameters_accepted` to use 2 distinct subjects (was
using 1 subject with 2 uploads to test ct parsing — would now 409).
22/22 biometric_endpoint tests + 59/59 catalogd lib tests green post-patch.
Production posture: gateway needs `cargo build --release -p gateway` +
`systemctl restart lakehouse.service` to pick up the new 409 in live traffic.
Counsel calendar is now the only remaining blocker for first real-photo intake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+263
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env bash
|
||||
# biometric_destruction_report — monthly destruction event aggregation.
|
||||
#
|
||||
# Specification: docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §5.
|
||||
# Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5.
|
||||
#
|
||||
# Why this exists: counsel and operations review need a periodic
|
||||
# attestation that destructions have happened in a defensible cadence.
|
||||
# This script produces an anonymized monthly report aggregating
|
||||
# per-subject audit logs.
|
||||
#
|
||||
# Output is anonymized — counts, timings, scope/trigger breakdowns,
|
||||
# and chain attestations. Candidate IDs are hashed (sha256-prefix) so
|
||||
# the report can be shared with counsel without exposing identifiers.
|
||||
#
|
||||
# Usage:
|
||||
# biometric_destruction_report.sh \
|
||||
# [--month YYYY-MM] \
|
||||
# [--audit-dir data/_catalog/subjects] \
|
||||
# [--output reports/biometric/destruction_<month>.md]
|
||||
#
|
||||
# Defaults:
|
||||
# --month — current UTC month (YYYY-MM)
|
||||
# --audit-dir — data/_catalog/subjects
|
||||
# --output — reports/biometric/destruction_<month>.md
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — report written successfully (whether or not events were found)
|
||||
# 1 — report written but with anomalies that need review
|
||||
# 2 — script error (missing tools, unreadable audit dir)
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
MONTH=""
|
||||
AUDIT_DIR="data/_catalog/subjects"
|
||||
OUT=""
|
||||
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--month) MONTH="$2"; shift 2 ;;
|
||||
--audit-dir) AUDIT_DIR="$2"; shift 2 ;;
|
||||
--output) OUT="$2"; shift 2 ;;
|
||||
-h|--help)
|
||||
sed -n '2,30p' "$0" | sed 's/^# \?//'
|
||||
exit 0 ;;
|
||||
*) echo "unknown flag: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Default month = current UTC YYYY-MM. Validate format defensively
|
||||
# so a malformed --month value (e.g. "May 2026") doesn't silently
|
||||
# match nothing in the JSONL filter.
|
||||
if [ -z "$MONTH" ]; then
|
||||
MONTH=$(date -u +%Y-%m)
|
||||
fi
|
||||
if ! echo "$MONTH" | grep -qE '^[0-9]{4}-(0[1-9]|1[0-2])$'; then
|
||||
echo "[report] FAIL: --month must be YYYY-MM, got '$MONTH'" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ -z "$OUT" ]; then
|
||||
OUT="reports/biometric/destruction_${MONTH}.md"
|
||||
fi
|
||||
|
||||
# Dependency gates.
|
||||
for cmd in jq sha256sum; do
|
||||
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||
echo "[report] FAIL: required tool '$cmd' not found in PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
if [ ! -d "$AUDIT_DIR" ]; then
|
||||
echo "[report] FAIL: audit dir not found at $AUDIT_DIR" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "$OUT")"
|
||||
|
||||
# Aggregator storage.
|
||||
EVENTS=$(mktemp)
|
||||
ANOMALIES=$(mktemp)
|
||||
trap 'rm -f "$EVENTS" "$ANOMALIES"' EXIT
|
||||
|
||||
# Iterate every per-subject audit log under AUDIT_DIR. Each file is
|
||||
# JSONL — one row per line. We extract erasure rows in the requested
|
||||
# month + emit a normalized one-line record per event.
|
||||
TOTAL_FILES=0
|
||||
TOTAL_ROWS_SCANNED=0
|
||||
SHARDS_WITH_EVENTS=0
|
||||
|
||||
for f in "$AUDIT_DIR"/*.audit.jsonl; do
|
||||
[ -e "$f" ] || continue
|
||||
TOTAL_FILES=$((TOTAL_FILES + 1))
|
||||
|
||||
# File-level row count (cheap).
|
||||
ROWS=$(wc -l < "$f" 2>/dev/null || echo 0)
|
||||
TOTAL_ROWS_SCANNED=$((TOTAL_ROWS_SCANNED + ROWS))
|
||||
|
||||
# Filter rows for the month + erasure kinds.
|
||||
HAD_EVENT=0
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
KIND=$(printf '%s' "$line" | jq -r '.accessor.kind // ""' 2>/dev/null || echo "")
|
||||
case "$KIND" in
|
||||
biometric_erasure|full_erasure) ;;
|
||||
*) continue ;;
|
||||
esac
|
||||
|
||||
TS=$(printf '%s' "$line" | jq -r '.ts // ""' 2>/dev/null || echo "")
|
||||
case "$TS" in
|
||||
"${MONTH}-"*) ;; # only this month
|
||||
*) continue ;;
|
||||
esac
|
||||
|
||||
HAD_EVENT=1
|
||||
CID=$(printf '%s' "$line" | jq -r '.candidate_id // ""' 2>/dev/null || echo "")
|
||||
PURPOSE=$(printf '%s' "$line" | jq -r '.accessor.purpose // ""' 2>/dev/null || echo "")
|
||||
RESULT=$(printf '%s' "$line" | jq -r '.result // ""' 2>/dev/null || echo "")
|
||||
# accessor.purpose has shape "trigger=<name>;..." per biometric_endpoint
|
||||
TRIGGER=$(printf '%s' "$PURPOSE" | sed -nE 's/.*trigger=([a-z_]+).*/\1/p')
|
||||
[ -n "$TRIGGER" ] || TRIGGER="unknown"
|
||||
|
||||
# Hash candidate_id so the report stays anonymized.
|
||||
CID_HASH=$(printf '%s' "$CID" | sha256sum | awk '{print substr($1,1,12)}')
|
||||
|
||||
# Anomaly: erasure row but result not in {erased, success}.
|
||||
case "$RESULT" in
|
||||
erased|success) ;;
|
||||
*)
|
||||
echo " - candidate_hash=$CID_HASH ts=$TS kind=$KIND result=$RESULT trigger=$TRIGGER (unexpected result)" >> "$ANOMALIES"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Tab-separated event line: ts, kind, trigger, result, cid_hash
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "$TS" "$KIND" "$TRIGGER" "$RESULT" "$CID_HASH" >> "$EVENTS"
|
||||
done < "$f"
|
||||
|
||||
if [ "$HAD_EVENT" = "1" ]; then
|
||||
SHARDS_WITH_EVENTS=$((SHARDS_WITH_EVENTS + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
EVENT_COUNT=$(wc -l < "$EVENTS" 2>/dev/null || echo 0)
|
||||
EVENT_COUNT=$(printf '%s' "$EVENT_COUNT" | tr -d '[:space:]')
|
||||
: "${EVENT_COUNT:=0}"
|
||||
|
||||
# Compute breakdowns.
|
||||
COUNT_BIOMETRIC_ONLY=0
|
||||
COUNT_FULL=0
|
||||
if [ "$EVENT_COUNT" != "0" ]; then
|
||||
COUNT_BIOMETRIC_ONLY=$(awk -F '\t' '$2=="biometric_erasure"' "$EVENTS" | wc -l | tr -d '[:space:]')
|
||||
COUNT_FULL=$(awk -F '\t' '$2=="full_erasure"' "$EVENTS" | wc -l | tr -d '[:space:]')
|
||||
fi
|
||||
|
||||
ANOMALY_COUNT=$(wc -l < "$ANOMALIES" 2>/dev/null || echo 0)
|
||||
ANOMALY_COUNT=$(printf '%s' "$ANOMALY_COUNT" | tr -d '[:space:]')
|
||||
: "${ANOMALY_COUNT:=0}"
|
||||
|
||||
# Render the report.
|
||||
GENERATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
|
||||
{
|
||||
echo "# Biometric Destruction Report — $MONTH"
|
||||
echo
|
||||
echo "**Generated:** $GENERATED_AT"
|
||||
echo "**Audit dir scanned:** \`$AUDIT_DIR\`"
|
||||
echo "**Spec:** docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §5"
|
||||
echo "**Generator:** scripts/staffing/biometric_destruction_report.sh"
|
||||
echo
|
||||
echo "## Scope"
|
||||
echo
|
||||
echo "- **Subject audit shards scanned:** $TOTAL_FILES"
|
||||
echo "- **Audit rows scanned (all kinds):** $TOTAL_ROWS_SCANNED"
|
||||
echo "- **Shards containing $MONTH erasure events:** $SHARDS_WITH_EVENTS"
|
||||
echo
|
||||
echo "## Destruction events in $MONTH"
|
||||
echo
|
||||
echo "- **Total events:** $EVENT_COUNT"
|
||||
echo "- **By scope:**"
|
||||
echo " - \`biometric_erasure\` (BiometricCollection cleared, manifest retained): $COUNT_BIOMETRIC_ONLY"
|
||||
echo " - \`full_erasure\` (manifest + biometric data cleared): $COUNT_FULL"
|
||||
echo
|
||||
|
||||
if [ "$EVENT_COUNT" = "0" ]; then
|
||||
echo "**No destruction events recorded for $MONTH.** This is correct"
|
||||
echo "for a month with no retention expiries / withdrawal requests"
|
||||
echo "/ RTBF requests / court orders."
|
||||
echo
|
||||
else
|
||||
echo "### By trigger"
|
||||
echo
|
||||
echo "| Trigger | Count |"
|
||||
echo "|---|---|"
|
||||
awk -F '\t' '{print $3}' "$EVENTS" | sort | uniq -c | \
|
||||
sort -rn | awk '{ printf("| %s | %d |\n", $2, $1); }'
|
||||
echo
|
||||
echo "### Event detail (anonymized)"
|
||||
echo
|
||||
echo "Candidate IDs are hashed (sha256-12-prefix) so this report can"
|
||||
echo "be shared with outside counsel without exposing identifiers."
|
||||
echo
|
||||
echo "| ts | kind | trigger | result | candidate_hash |"
|
||||
echo "|---|---|---|---|---|"
|
||||
sort -k1,1 "$EVENTS" | awk -F '\t' '{
|
||||
printf("| %s | %s | %s | %s | %s |\n", $1, $2, $3, $4, $5);
|
||||
}'
|
||||
echo
|
||||
fi
|
||||
|
||||
if [ "$ANOMALY_COUNT" != "0" ]; then
|
||||
echo "## Anomalies ($ANOMALY_COUNT)"
|
||||
echo
|
||||
echo "Events whose audit row deviates from expected shape (kind/result"
|
||||
echo "mismatch, missing trigger, etc.). These do NOT necessarily mean"
|
||||
echo "the destruction failed — the BIPA-load-bearing surface is the"
|
||||
echo "audit chain, which still verifies cryptographically. They are"
|
||||
echo "logged here so an operator can investigate and confirm."
|
||||
echo
|
||||
echo '```'
|
||||
cat "$ANOMALIES"
|
||||
echo '```'
|
||||
echo
|
||||
fi
|
||||
|
||||
echo "## Cryptographic attestation"
|
||||
echo
|
||||
echo "This report was produced by aggregating per-subject HMAC-chained"
|
||||
echo "audit logs. The chain itself is the BIPA-defensible substrate;"
|
||||
echo "this report is a derived view, not the chain of record. To verify"
|
||||
echo "any individual event, run:"
|
||||
echo
|
||||
echo '```bash'
|
||||
echo "./scripts/staffing/verify_biometric_erasure.sh <candidate_id>"
|
||||
echo '```'
|
||||
echo "(operator must un-hash the candidate ID through their own"
|
||||
echo " operator log to perform spot-checks)."
|
||||
echo
|
||||
echo "**Cross-runtime parity:** the same audit logs are byte-identical"
|
||||
echo "under Rust + Go (per scripts/cutover/parity/subject_audit_parity.sh)."
|
||||
echo "If counsel needs cross-runtime attestation, that probe provides it."
|
||||
echo
|
||||
|
||||
EVIDENCE_HASH=$(sha256sum "$EVENTS" 2>/dev/null | awk '{print $1}')
|
||||
: "${EVIDENCE_HASH:=$(echo -n '' | sha256sum | awk '{print $1}')}"
|
||||
echo "**Events SHA-256:** \`$EVIDENCE_HASH\`"
|
||||
echo
|
||||
echo "---"
|
||||
echo
|
||||
echo "**Operator (J):** _______________________________ Date: __________"
|
||||
echo
|
||||
} > "$OUT"
|
||||
|
||||
echo "[report] $EVENT_COUNT destruction events in $MONTH ($COUNT_BIOMETRIC_ONLY biometric_only, $COUNT_FULL full)"
|
||||
echo "[report] anomalies: $ANOMALY_COUNT"
|
||||
echo "[report] output: $OUT"
|
||||
|
||||
# Exit 1 if anomalies present (review needed) but report still written.
|
||||
if [ "$ANOMALY_COUNT" != "0" ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
Executable
+99
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
# bundle_counsel_packet — assemble the counsel-review packet tarball.
|
||||
#
|
||||
# Specification: docs/counsel/COUNSEL_REVIEW_PACKET_<DATE>.md §9.
|
||||
#
|
||||
# Why this exists: the cover note references a list of documents.
|
||||
# Counsel needs them as a single transmittable artifact, with per-file
|
||||
# integrity hashes so they can verify nothing changed in transit.
|
||||
#
|
||||
# Output:
|
||||
# reports/counsel/counsel_packet_<DATE>.tar.gz
|
||||
# reports/counsel/counsel_packet_<DATE>.manifest.txt (sha256 per file)
|
||||
#
|
||||
# Usage:
|
||||
# bundle_counsel_packet.sh [--date YYYY-MM-DD]
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — packet bundled successfully
|
||||
# 1 — one or more referenced documents are missing
|
||||
# 2 — script error (missing tools, write failure)
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--date) DATE="$2"; shift 2 ;;
|
||||
-h|--help)
|
||||
sed -n '2,20p' "$0" | sed 's/^# \?//'
|
||||
exit 0 ;;
|
||||
*) echo "unknown flag: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Dependency gate.
|
||||
for cmd in tar sha256sum; do
|
||||
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||
echo "[bundle] FAIL: required tool '$cmd' not found in PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
# Files in the packet. Order is the recommended counsel-review order
|
||||
# from the cover note §6.
|
||||
FILES=(
|
||||
"docs/counsel/COUNSEL_REVIEW_PACKET_${DATE}.md"
|
||||
"docs/policies/consent/biometric_retention_schedule_v1.md"
|
||||
"docs/policies/consent/biometric_consent_template_v1.md"
|
||||
"docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md"
|
||||
"docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md"
|
||||
"docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md"
|
||||
"docs/specs/GATE_3B_DEEPFACE_DESIGN.md"
|
||||
"docs/PHASE_1_6_BIPA_GATES.md"
|
||||
)
|
||||
|
||||
# Verify all referenced files exist before opening the tarball.
|
||||
MISSING=0
|
||||
for f in "${FILES[@]}"; do
|
||||
if [ ! -r "$f" ]; then
|
||||
echo "[bundle] MISSING: $f" >&2
|
||||
MISSING=$((MISSING + 1))
|
||||
fi
|
||||
done
|
||||
if [ "$MISSING" -gt 0 ]; then
|
||||
echo "[bundle] FAIL: $MISSING required documents missing — aborting" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
OUT_DIR="reports/counsel"
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
TARBALL="$OUT_DIR/counsel_packet_${DATE}.tar.gz"
|
||||
MANIFEST="$OUT_DIR/counsel_packet_${DATE}.manifest.txt"
|
||||
|
||||
# Build the manifest first — counsel uses this to verify integrity.
|
||||
{
|
||||
echo "# Counsel Packet Manifest — $DATE"
|
||||
echo "# Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "# Each file is listed with its SHA-256 hash. To verify on receipt:"
|
||||
echo "# tar xzf counsel_packet_${DATE}.tar.gz"
|
||||
echo "# sha256sum -c counsel_packet_${DATE}.manifest.txt"
|
||||
echo "# (re-format the lines below with two spaces between hash and path"
|
||||
echo "# for sha256sum -c compatibility — sha256sum's strict format)"
|
||||
echo
|
||||
for f in "${FILES[@]}"; do
|
||||
sha256sum "$f"
|
||||
done
|
||||
} > "$MANIFEST"
|
||||
|
||||
# Build the tarball — include the manifest itself.
|
||||
tar -czf "$TARBALL" "${FILES[@]}" "$MANIFEST"
|
||||
|
||||
PACKET_HASH=$(sha256sum "$TARBALL" | awk '{print $1}')
|
||||
|
||||
echo "[bundle] packet: $TARBALL"
|
||||
echo "[bundle] manifest: $MANIFEST"
|
||||
echo "[bundle] tarball SHA-256: $PACKET_HASH"
|
||||
echo "[bundle] files: ${#FILES[@]}"
|
||||
Executable
+266
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env bash
|
||||
# verify_biometric_erasure — confirm that a biometric erasure completed cleanly.
|
||||
#
|
||||
# Specification: docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §3 step 3.
|
||||
# Spec: docs/PHASE_1_6_BIPA_GATES.md §1 Gate 5.
|
||||
#
|
||||
# Why this exists: when an operator runs the erasure curl call against
|
||||
# /biometric/subject/{id}/erase, they need a defensible artifact proving
|
||||
# destruction completed. This script produces that artifact by checking
|
||||
# four things:
|
||||
#
|
||||
# 1. SubjectManifest.biometric_collection is null (catalogd cleared the row)
|
||||
# 2. data/biometric/uploads/<safe_id>/ is empty or absent (photo file gone)
|
||||
# 3. Most recent audit row has accessor.kind in {biometric_erasure, full_erasure}
|
||||
# AND result is "erased" or "success" (the chain logged the erasure intent)
|
||||
# 4. audit_log.chain_verified is true (HMAC chain still intact end-to-end)
|
||||
#
|
||||
# All four must pass for an operator to mark the destruction complete.
|
||||
#
|
||||
# Usage:
|
||||
# verify_biometric_erasure.sh <candidate_id> [--from ISO] [--to ISO]
|
||||
#
|
||||
# Environment:
|
||||
# GATEWAY_URL — default http://localhost:3100
|
||||
# LEGAL_TOKEN_FILE — default /etc/lakehouse/legal_audit.token
|
||||
# UPLOADS_ROOT — default data/biometric/uploads (relative to repo root)
|
||||
# OUT_DIR — default reports/biometric (where the verification report lands)
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — all four checks pass; erasure verified
|
||||
# 1 — one or more checks failed; do NOT mark destruction complete; escalate
|
||||
# 2 — script error (missing tools, network failure, bad token)
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
if [ "$#" -lt 1 ]; then
|
||||
echo "usage: verify_biometric_erasure.sh <candidate_id> [--from ISO] [--to ISO]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
CANDIDATE_ID="$1"
|
||||
shift
|
||||
FROM=""
|
||||
TO=""
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--from) FROM="$2"; shift 2 ;;
|
||||
--to) TO="$2"; shift 2 ;;
|
||||
*) echo "unknown flag: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
GATEWAY_URL="${GATEWAY_URL:-http://localhost:3100}"
|
||||
LEGAL_TOKEN_FILE="${LEGAL_TOKEN_FILE:-/etc/lakehouse/legal_audit.token}"
|
||||
UPLOADS_ROOT="${UPLOADS_ROOT:-data/biometric/uploads}"
|
||||
OUT_DIR="${OUT_DIR:-reports/biometric}"
|
||||
|
||||
# Dependency gates — fail fast with clear errors rather than producing
|
||||
# a misleading "evidence" file from missing tools.
|
||||
for cmd in curl jq sha256sum; do
|
||||
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||
echo "[verify] FAIL: required tool '$cmd' not found in PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
if [ ! -r "$LEGAL_TOKEN_FILE" ]; then
|
||||
echo "[verify] FAIL: cannot read legal token at $LEGAL_TOKEN_FILE" >&2
|
||||
echo "[verify] This script requires legal-tier auth to query /audit/subject/." >&2
|
||||
exit 2
|
||||
fi
|
||||
LEGAL_TOKEN=$(tr -d '[:space:]' < "$LEGAL_TOKEN_FILE")
|
||||
if [ -z "$LEGAL_TOKEN" ]; then
|
||||
echo "[verify] FAIL: legal token file is empty" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# safe_id matches catalogd::biometric_endpoint::sanitize_for_path:
|
||||
# any non-[A-Za-z0-9_.-] char is replaced with underscore.
|
||||
SAFE_ID=$(printf '%s' "$CANDIDATE_ID" | sed 's/[^A-Za-z0-9_.\-]/_/g')
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
DATE=$(date -u +%Y-%m-%dT%H-%M-%SZ)
|
||||
OUT="$OUT_DIR/erasure_verify_${SAFE_ID}_${DATE}.md"
|
||||
EVIDENCE=$(mktemp)
|
||||
trap 'rm -f "$EVIDENCE"' EXIT
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
note() { echo "$1" >> "$EVIDENCE"; }
|
||||
mark_pass() { PASS=$((PASS+1)); note " - PASS: $1"; }
|
||||
mark_fail() { FAIL=$((FAIL+1)); note " - FAIL: $1"; }
|
||||
|
||||
note "## Verification target"
|
||||
note ""
|
||||
note "- **candidate_id:** \`$CANDIDATE_ID\`"
|
||||
note "- **safe_id (filesystem):** \`$SAFE_ID\`"
|
||||
note "- **gateway:** \`$GATEWAY_URL\`"
|
||||
note "- **uploads root:** \`$UPLOADS_ROOT\`"
|
||||
note "- **window:** ${FROM:-unbounded} → ${TO:-unbounded}"
|
||||
note ""
|
||||
|
||||
# ── Fetch the audit response ────────────────────────────────────────
|
||||
QUERY=""
|
||||
if [ -n "$FROM" ]; then QUERY="from=$FROM"; fi
|
||||
if [ -n "$TO" ]; then
|
||||
if [ -n "$QUERY" ]; then QUERY="${QUERY}&to=$TO"; else QUERY="to=$TO"; fi
|
||||
fi
|
||||
URL="$GATEWAY_URL/audit/subject/$CANDIDATE_ID"
|
||||
if [ -n "$QUERY" ]; then URL="$URL?$QUERY"; fi
|
||||
|
||||
RESP_FILE=$(mktemp)
|
||||
HTTP_CODE=$(curl -sS -o "$RESP_FILE" -w '%{http_code}' \
|
||||
-H "X-Lakehouse-Legal-Token: $LEGAL_TOKEN" \
|
||||
-H "Accept: application/json" \
|
||||
"$URL" 2>&1) || HTTP_CODE="000"
|
||||
|
||||
if [ "$HTTP_CODE" != "200" ]; then
|
||||
echo "[verify] FAIL: GET $URL returned HTTP $HTTP_CODE" >&2
|
||||
echo "[verify] response head:" >&2
|
||||
head -c 500 "$RESP_FILE" >&2
|
||||
echo >&2
|
||||
rm -f "$RESP_FILE"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Schema sanity — refuse to evaluate against an unrecognized response shape.
|
||||
SCHEMA=$(jq -r '.schema // ""' < "$RESP_FILE")
|
||||
if [ "$SCHEMA" != "subject_audit_response.v1" ]; then
|
||||
echo "[verify] FAIL: unexpected response schema '$SCHEMA' (want subject_audit_response.v1)" >&2
|
||||
rm -f "$RESP_FILE"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ── Check 1: manifest.biometric_collection is null ──────────────────
|
||||
note "## Check 1 — Subject manifest biometric_collection is null"
|
||||
note ""
|
||||
BIO_COLL=$(jq -c '.manifest.biometric_collection // null' < "$RESP_FILE")
|
||||
note "**manifest.biometric_collection:** \`$BIO_COLL\`"
|
||||
note ""
|
||||
if [ "$BIO_COLL" = "null" ]; then
|
||||
mark_pass "biometric_collection field is null on the subject manifest"
|
||||
else
|
||||
mark_fail "biometric_collection is still populated — erasure incomplete"
|
||||
fi
|
||||
note ""
|
||||
|
||||
# ── Check 2: filesystem uploads dir is empty/absent ─────────────────
|
||||
note "## Check 2 — Quarantined upload directory empty or absent"
|
||||
note ""
|
||||
UPLOAD_DIR="$UPLOADS_ROOT/$SAFE_ID"
|
||||
note "**path:** \`$UPLOAD_DIR\`"
|
||||
if [ ! -e "$UPLOAD_DIR" ]; then
|
||||
note "**state:** absent (directory was removed during erasure or never existed)"
|
||||
note ""
|
||||
mark_pass "upload directory is absent"
|
||||
elif [ ! -d "$UPLOAD_DIR" ]; then
|
||||
note "**state:** path exists but is not a directory — investigate"
|
||||
note ""
|
||||
mark_fail "upload path exists and is not a directory: $UPLOAD_DIR"
|
||||
else
|
||||
REMAINING=$(find "$UPLOAD_DIR" -maxdepth 1 -mindepth 1 2>/dev/null | wc -l | tr -d '[:space:]')
|
||||
: "${REMAINING:=0}"
|
||||
note "**state:** directory exists with $REMAINING remaining entries"
|
||||
note ""
|
||||
if [ "$REMAINING" = "0" ]; then
|
||||
mark_pass "upload directory is empty (no residual photo files)"
|
||||
else
|
||||
mark_fail "$REMAINING file(s) remain under $UPLOAD_DIR — must be unlinked"
|
||||
note "### Residual files"
|
||||
note ""
|
||||
note '```'
|
||||
find "$UPLOAD_DIR" -maxdepth 2 >> "$EVIDENCE"
|
||||
note '```'
|
||||
note ""
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Check 3: most recent audit row reflects erasure ─────────────────
|
||||
note "## Check 3 — Audit log records the erasure event"
|
||||
note ""
|
||||
ROW_COUNT=$(jq '.audit_log.rows | length' < "$RESP_FILE")
|
||||
note "**rows in window:** $ROW_COUNT"
|
||||
if [ "$ROW_COUNT" = "0" ]; then
|
||||
mark_fail "no audit rows in the requested window — erasure should have appended one"
|
||||
note ""
|
||||
else
|
||||
LAST_KIND=$(jq -r '.audit_log.rows | last | .accessor.kind // ""' < "$RESP_FILE")
|
||||
LAST_RESULT=$(jq -r '.audit_log.rows | last | .result // ""' < "$RESP_FILE")
|
||||
LAST_TS=$(jq -r '.audit_log.rows | last | .ts // ""' < "$RESP_FILE")
|
||||
note "**last row:** ts=\`$LAST_TS\` accessor.kind=\`$LAST_KIND\` result=\`$LAST_RESULT\`"
|
||||
note ""
|
||||
case "$LAST_KIND" in
|
||||
biometric_erasure|full_erasure)
|
||||
case "$LAST_RESULT" in
|
||||
erased|success)
|
||||
mark_pass "last audit row is an erasure event ($LAST_KIND/$LAST_RESULT)"
|
||||
;;
|
||||
*)
|
||||
mark_fail "last row kind is $LAST_KIND but result is '$LAST_RESULT' (expected erased/success)"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
*)
|
||||
mark_fail "last audit row accessor.kind is '$LAST_KIND' (expected biometric_erasure or full_erasure)"
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
note ""
|
||||
|
||||
# ── Check 4: HMAC chain verifies end-to-end ─────────────────────────
|
||||
note "## Check 4 — HMAC chain integrity"
|
||||
note ""
|
||||
CHAIN_VERIFIED=$(jq -r '.audit_log.chain_verified' < "$RESP_FILE")
|
||||
CHAIN_ROOT=$(jq -r '.audit_log.chain_root // ""' < "$RESP_FILE")
|
||||
CHAIN_ROWS=$(jq -r '.audit_log.chain_rows_total // 0' < "$RESP_FILE")
|
||||
CHAIN_ERR=$(jq -r '.audit_log.chain_verification_error // ""' < "$RESP_FILE")
|
||||
note "**chain_verified:** \`$CHAIN_VERIFIED\`"
|
||||
note "**chain_rows_total:** $CHAIN_ROWS"
|
||||
note "**chain_root:** \`$CHAIN_ROOT\`"
|
||||
if [ -n "$CHAIN_ERR" ]; then
|
||||
note "**chain_verification_error:** \`$CHAIN_ERR\`"
|
||||
fi
|
||||
note ""
|
||||
if [ "$CHAIN_VERIFIED" = "true" ]; then
|
||||
mark_pass "chain verifies end-to-end ($CHAIN_ROWS rows)"
|
||||
else
|
||||
mark_fail "chain integrity broken — destruction is NOT defensible until investigated"
|
||||
fi
|
||||
note ""
|
||||
|
||||
# ── Render report ───────────────────────────────────────────────────
|
||||
TOTAL=$((PASS + FAIL))
|
||||
note "## Summary"
|
||||
note ""
|
||||
note "**$PASS / $TOTAL** verification checks pass."
|
||||
note ""
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
note "**Status: ERASURE NOT VERIFIED.** Do NOT mark destruction complete. Escalate to engineering before responding to candidate / counsel."
|
||||
note ""
|
||||
fi
|
||||
|
||||
# Hash response body so the report has a tamper-evident anchor.
|
||||
RESP_HASH=$(sha256sum "$RESP_FILE" | awk '{print $1}')
|
||||
EVIDENCE_HASH=$(sha256sum "$EVIDENCE" | awk '{print $1}')
|
||||
|
||||
{
|
||||
echo "# Biometric Erasure Verification — $CANDIDATE_ID"
|
||||
echo
|
||||
echo "**Date:** $DATE"
|
||||
echo "**Spec:** docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md §3 step 3"
|
||||
echo "**Generator:** scripts/staffing/verify_biometric_erasure.sh"
|
||||
echo
|
||||
cat "$EVIDENCE"
|
||||
echo "---"
|
||||
echo
|
||||
echo "**Audit response SHA-256:** \`$RESP_HASH\`"
|
||||
echo "**Evidence summary SHA-256:** \`$EVIDENCE_HASH\`"
|
||||
echo
|
||||
} > "$OUT"
|
||||
|
||||
rm -f "$RESP_FILE"
|
||||
echo "[verify] $PASS / $TOTAL checks pass — report: $OUT"
|
||||
echo "[verify] response hash: $RESP_HASH"
|
||||
[ "$FAIL" -eq 0 ]
|
||||
Reference in New Issue
Block a user