catalogd: Step 1 — SubjectManifest type + Registry CRUD

Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 1.
Mirrors the existing AiView put/get/list/delete pattern. NOT a separate
daemon, NOT new infrastructure — extends catalogd's manifest layer with
a fourth manifest type (subject) alongside dataset/view/tombstone/profile.

shared/types.rs additions:
- SubjectManifest (the wire format from spec §2)
- SubjectStatus enum: pending_consent | active | withdrawn |
  retention_expired | erased
- SubjectVertical enum: unknown | general | healthcare | finance | other
  (default = Unknown for fail-closed routing per spec §2.1)
- ConsentStatus enum: pending_backfill_review | pending_first_contact |
  given | withdrawn | expired
- BiometricConsentStatus enum: never_collected | pending | given |
  withdrawn | expired
- GeneralPiiConsent + BiometricConsent + SubjectConsent
- SubjectRetention (general_pii_until + policy)
- SubjectDatasetRef (name + key_column + key_value pointing at existing
  catalogd dataset manifests)

catalogd/registry.rs additions:
- subjects: Arc<RwLock<HashMap<String, SubjectManifest>>> field on Registry
- put_subject() — validates dataset refs, persists to
  _catalog/subjects/<id>.json, updates in-memory cache
- get_subject() / list_subjects() / delete_subject() / subjects_count()
- rebuild() now loads subject manifests at startup alongside views +
  profiles + tombstones

Tests (5/5 passing):
- put_subject_with_no_dataset_refs_succeeds
- put_subject_rejects_dangling_dataset_ref (validation works)
- put_subject_with_valid_dataset_ref_succeeds
- subject_round_trips_through_object_store (persistence works)
- delete_subject_removes_in_memory_and_persistence

NOT in this commit (future steps):
- Step 2: SubjectAuditWriter with HMAC chain
- Step 3: Backfill ETL from workers_500k.parquet
- Steps 4-5: Wire gateway tool registry + validator to write audit rows
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep

cargo check --workspace clean. cargo test -p catalogd subject 5/5 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-05-03 03:13:08 -05:00
co-authored by Claude Opus 4.7
parent ed1fcd3c26
commit d25990982c
2 changed files with 342 additions and 1 deletions
+146
View File
@@ -364,3 +364,149 @@ pub enum Redaction {
keep_suffix: usize,
},
}
// ─── Subject manifests (compliance/access metadata layer) ────────────
// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
//
// A SubjectManifest is a per-person record threading existing catalogd
// primitives (datasets, views, tombstones) together by subject token.
// Stored at _catalog/subjects/<candidate_id>.json. NOT a separate
// daemon — extends catalogd's existing manifest pattern.
//
// The audit log per subject (HMAC-chained JSONL at
// _catalog/subjects/<candidate_id>.audit.jsonl) is owned by a separate
// SubjectAuditWriter (Step 2) — not modeled here.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum SubjectStatus {
PendingConsent,
Active,
Withdrawn,
RetentionExpired,
Erased,
}
/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed:
/// the gateway treats Unknown as healthcare-equivalent until reclassified
/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1).
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum SubjectVertical {
Unknown,
General,
Healthcare,
Finance,
Other,
}
impl Default for SubjectVertical {
fn default() -> Self { SubjectVertical::Unknown }
}
/// State machine for general-PII consent. BIPA-specific biometric
/// consent is tracked separately on `BiometricConsent`.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum ConsentStatus {
/// Backfilled from existing data sources before consent was tracked.
/// SHOULD route as if no consent exists until reviewed.
PendingBackfillReview,
/// New subject; awaiting consent UX.
PendingFirstContact,
Given,
Withdrawn,
Expired,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum BiometricConsentStatus {
/// Default. No biometric data exists for this subject.
NeverCollected,
Pending,
Given,
Withdrawn,
Expired,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GeneralPiiConsent {
pub status: ConsentStatus,
/// Reference to the consent template version (matches a row in the
/// future consent_versions table). Empty when status is pending_*.
#[serde(default)]
pub version: String,
#[serde(default)]
pub given_at: Option<chrono::DateTime<chrono::Utc>>,
#[serde(default)]
pub withdrawn_at: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BiometricConsent {
pub status: BiometricConsentStatus,
/// BIPA: max 3 years from last interaction. Implementation MUST
/// enforce daily expiration sweep against this field.
#[serde(default)]
pub retention_until: Option<chrono::DateTime<chrono::Utc>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectConsent {
pub general_pii: GeneralPiiConsent,
pub biometric: BiometricConsent,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectRetention {
pub general_pii_until: chrono::DateTime<chrono::Utc>,
/// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026").
#[serde(default)]
pub policy: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectDatasetRef {
/// catalogd dataset name (must resolve to an existing manifest).
pub name: String,
/// Column in that dataset that holds the subject's identifier.
pub key_column: String,
/// Specific identifier value (typically same as candidate_id; differs
/// only when a dataset uses a foreign-key with a different format).
pub key_value: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SubjectManifest {
/// Schema version tag — always "subject_manifest.v1" for v1.
#[serde(default = "default_subject_manifest_schema")]
pub schema: String,
/// Subject identifier (the canonical token, e.g. "CAND-000001").
pub candidate_id: String,
pub created_at: chrono::DateTime<chrono::Utc>,
pub updated_at: chrono::DateTime<chrono::Utc>,
pub status: SubjectStatus,
#[serde(default)]
pub vertical: SubjectVertical,
pub consent: SubjectConsent,
pub retention: SubjectRetention,
/// Datasets that contain rows for this subject. Used by audit-response
/// to know what to project. Each entry MUST reference an existing
/// catalogd dataset manifest (validated on put_subject).
pub datasets: Vec<SubjectDatasetRef>,
/// Names of existing AiViews that safely project this subject's data.
/// Used by service-tier readers (no PII exposure).
#[serde(default)]
pub safe_views: Vec<String>,
/// Path (relative to repo root) of the per-subject audit JSONL.
/// Owned by SubjectAuditWriter (Step 2). Empty until first write.
#[serde(default)]
pub audit_log_path: String,
/// SHA-256 of the most recent HMAC chain checkpoint of the audit log.
/// Empty until first audit row written.
#[serde(default)]
pub audit_log_chain_root: String,
}
fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() }