catalogd: Step 1 — SubjectManifest type + Registry CRUD
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 1.
Mirrors the existing AiView put/get/list/delete pattern. NOT a separate
daemon, NOT new infrastructure — extends catalogd's manifest layer with
a fourth manifest type (subject) alongside dataset/view/tombstone/profile.
shared/types.rs additions:
- SubjectManifest (the wire format from spec §2)
- SubjectStatus enum: pending_consent | active | withdrawn |
retention_expired | erased
- SubjectVertical enum: unknown | general | healthcare | finance | other
(default = Unknown for fail-closed routing per spec §2.1)
- ConsentStatus enum: pending_backfill_review | pending_first_contact |
given | withdrawn | expired
- BiometricConsentStatus enum: never_collected | pending | given |
withdrawn | expired
- GeneralPiiConsent + BiometricConsent + SubjectConsent
- SubjectRetention (general_pii_until + policy)
- SubjectDatasetRef (name + key_column + key_value pointing at existing
catalogd dataset manifests)
catalogd/registry.rs additions:
- subjects: Arc<RwLock<HashMap<String, SubjectManifest>>> field on Registry
- put_subject() — validates dataset refs, persists to
_catalog/subjects/<id>.json, updates in-memory cache
- get_subject() / list_subjects() / delete_subject() / subjects_count()
- rebuild() now loads subject manifests at startup alongside views +
profiles + tombstones
Tests (5/5 passing):
- put_subject_with_no_dataset_refs_succeeds
- put_subject_rejects_dangling_dataset_ref (validation works)
- put_subject_with_valid_dataset_ref_succeeds
- subject_round_trips_through_object_store (persistence works)
- delete_subject_removes_in_memory_and_persistence
NOT in this commit (future steps):
- Step 2: SubjectAuditWriter with HMAC chain
- Step 3: Backfill ETL from workers_500k.parquet
- Steps 4-5: Wire gateway tool registry + validator to write audit rows
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
cargo check --workspace clean. cargo test -p catalogd subject 5/5 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -364,3 +364,149 @@ pub enum Redaction {
|
||||
keep_suffix: usize,
|
||||
},
|
||||
}
|
||||
|
||||
// ─── Subject manifests (compliance/access metadata layer) ────────────
|
||||
// Specification: docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md
|
||||
//
|
||||
// A SubjectManifest is a per-person record threading existing catalogd
|
||||
// primitives (datasets, views, tombstones) together by subject token.
|
||||
// Stored at _catalog/subjects/<candidate_id>.json. NOT a separate
|
||||
// daemon — extends catalogd's existing manifest pattern.
|
||||
//
|
||||
// The audit log per subject (HMAC-chained JSONL at
|
||||
// _catalog/subjects/<candidate_id>.audit.jsonl) is owned by a separate
|
||||
// SubjectAuditWriter (Step 2) — not modeled here.
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SubjectStatus {
|
||||
PendingConsent,
|
||||
Active,
|
||||
Withdrawn,
|
||||
RetentionExpired,
|
||||
Erased,
|
||||
}
|
||||
|
||||
/// Vertical drives healthcare-PHI routing. Default Unknown is fail-closed:
|
||||
/// the gateway treats Unknown as healthcare-equivalent until reclassified
|
||||
/// (per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §2.1).
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SubjectVertical {
|
||||
Unknown,
|
||||
General,
|
||||
Healthcare,
|
||||
Finance,
|
||||
Other,
|
||||
}
|
||||
|
||||
impl Default for SubjectVertical {
|
||||
fn default() -> Self { SubjectVertical::Unknown }
|
||||
}
|
||||
|
||||
/// State machine for general-PII consent. BIPA-specific biometric
|
||||
/// consent is tracked separately on `BiometricConsent`.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ConsentStatus {
|
||||
/// Backfilled from existing data sources before consent was tracked.
|
||||
/// SHOULD route as if no consent exists until reviewed.
|
||||
PendingBackfillReview,
|
||||
/// New subject; awaiting consent UX.
|
||||
PendingFirstContact,
|
||||
Given,
|
||||
Withdrawn,
|
||||
Expired,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum BiometricConsentStatus {
|
||||
/// Default. No biometric data exists for this subject.
|
||||
NeverCollected,
|
||||
Pending,
|
||||
Given,
|
||||
Withdrawn,
|
||||
Expired,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct GeneralPiiConsent {
|
||||
pub status: ConsentStatus,
|
||||
/// Reference to the consent template version (matches a row in the
|
||||
/// future consent_versions table). Empty when status is pending_*.
|
||||
#[serde(default)]
|
||||
pub version: String,
|
||||
#[serde(default)]
|
||||
pub given_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
#[serde(default)]
|
||||
pub withdrawn_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct BiometricConsent {
|
||||
pub status: BiometricConsentStatus,
|
||||
/// BIPA: max 3 years from last interaction. Implementation MUST
|
||||
/// enforce daily expiration sweep against this field.
|
||||
#[serde(default)]
|
||||
pub retention_until: Option<chrono::DateTime<chrono::Utc>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SubjectConsent {
|
||||
pub general_pii: GeneralPiiConsent,
|
||||
pub biometric: BiometricConsent,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SubjectRetention {
|
||||
pub general_pii_until: chrono::DateTime<chrono::Utc>,
|
||||
/// Free-text policy reference (e.g. "4_year_default", "client_xyz_contract_2026").
|
||||
#[serde(default)]
|
||||
pub policy: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SubjectDatasetRef {
|
||||
/// catalogd dataset name (must resolve to an existing manifest).
|
||||
pub name: String,
|
||||
/// Column in that dataset that holds the subject's identifier.
|
||||
pub key_column: String,
|
||||
/// Specific identifier value (typically same as candidate_id; differs
|
||||
/// only when a dataset uses a foreign-key with a different format).
|
||||
pub key_value: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SubjectManifest {
|
||||
/// Schema version tag — always "subject_manifest.v1" for v1.
|
||||
#[serde(default = "default_subject_manifest_schema")]
|
||||
pub schema: String,
|
||||
/// Subject identifier (the canonical token, e.g. "CAND-000001").
|
||||
pub candidate_id: String,
|
||||
pub created_at: chrono::DateTime<chrono::Utc>,
|
||||
pub updated_at: chrono::DateTime<chrono::Utc>,
|
||||
pub status: SubjectStatus,
|
||||
#[serde(default)]
|
||||
pub vertical: SubjectVertical,
|
||||
pub consent: SubjectConsent,
|
||||
pub retention: SubjectRetention,
|
||||
/// Datasets that contain rows for this subject. Used by audit-response
|
||||
/// to know what to project. Each entry MUST reference an existing
|
||||
/// catalogd dataset manifest (validated on put_subject).
|
||||
pub datasets: Vec<SubjectDatasetRef>,
|
||||
/// Names of existing AiViews that safely project this subject's data.
|
||||
/// Used by service-tier readers (no PII exposure).
|
||||
#[serde(default)]
|
||||
pub safe_views: Vec<String>,
|
||||
/// Path (relative to repo root) of the per-subject audit JSONL.
|
||||
/// Owned by SubjectAuditWriter (Step 2). Empty until first write.
|
||||
#[serde(default)]
|
||||
pub audit_log_path: String,
|
||||
/// SHA-256 of the most recent HMAC chain checkpoint of the audit log.
|
||||
/// Empty until first audit row written.
|
||||
#[serde(default)]
|
||||
pub audit_log_chain_root: String,
|
||||
}
|
||||
|
||||
fn default_subject_manifest_schema() -> String { "subject_manifest.v1".into() }
|
||||
|
||||
Reference in New Issue
Block a user