gateway: Step 5 — wire SubjectAuditWriter into validator WorkerLookup

Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5. Every WorkerLookup.find() call from the validator path now produces one audit row in the per-subject HMAC-chained JSONL. Failures are non-blocking — validator continues whether audit succeeds or fails. Approach: decorator pattern. WorkerLookup is a sync trait by design (validator's contract is "in-memory snapshot, no per-call I/O") and audit writes are async, so we can't expand the trait. Instead, a new AuditingWorkerLookup wraps the inner lookup, captures a tokio::runtime::Handle at construction, and spawns audit writes from sync find() onto that handle. The chain stays intact under spawn fan- out because the writer's per-subject Mutex (shipped in the previous scrum-fix commit) serializes same-subject appends regardless of how the spawn calls arrive. Files changed: crates/gateway/src/v1/auditing_worker_lookup.rs (NEW, 175 LOC): - AuditingWorkerLookup<inner: dyn WorkerLookup, audit: Option<Arc<Writer>>> - new() captures Tokio Handle if audit is Some - find() runs inner lookup, then spawns audit append with: accessor.kind = "validator_lookup" accessor.purpose = "validator_worker_lookup" fields_accessed = ["exists"] (validator only proves existence of a subject; downstream code reads policy fields separately and would have its own audit if those become PII) result = "success" if found, "not_found" otherwise - Audit-disabled path (audit: None) is a transparent passthrough — zero overhead, no panic, no runtime requirement. crates/gateway/src/v1/mod.rs: + pub mod auditing_worker_lookup; crates/gateway/src/main.rs: - Hoisted subject_audit_writer construction OUT of the V1State literal (declaration-order constraint: validate_workers needs access to the writer). The hoisted Arc is then reused for the V1State.subject_audit field. - validate_workers now wraps the raw lookup with AuditingWorkerLookup::new(raw, subject_audit_writer.clone()) Tests (4/4 passing): - find_existing_subject_writes_success_audit_row - find_missing_subject_writes_not_found_audit_row (phantom-id case) - audit_disabled_means_no_writes_no_overhead (None pathway) - many_finds_to_same_subject_produce_intact_chain (30 sequential spawns on the same subject — chain verifies all 30, regression against the race we fixed in catalogd subject_audit) Also catches the iterate.rs:324 phantom-ID check transparently — that codepath calls state.validate_workers.find(...) which now goes through the wrapper, so every phantom-id rejection logs an audit row for free. NOT in this commit (future steps): - Step 6: /audit/subject/{id} HTTP endpoint - Step 7: Daily retention sweep - Threading X-Lakehouse-Trace-Id from request through to audit row (currently audit row's accessor.trace_id is empty) cargo build --release clean. cargo test -p gateway auditing_worker_lookup 4/4 PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 03:43:40 -05:00
parent e38f3573ff
commit cd8c59a53d
3 changed files with 237 additions and 26 deletions
@@ -255,6 +255,33 @@ async fn main() {
        // surface over the existing aibridge → Ollama path. Future
        // phases add provider adapters (39), routing engine (40),
        // session state (41), etc. All without changing this mount.
+        ;
+
+    // Build subject-audit writer BEFORE the V1State literal so the
+    // validate_workers wrapper can reference it. None = audit disabled
+    // (key file missing or unreadable); the gateway still serves PII
+    // paths but produces no audit trail per spec §3.2.
+    let subject_audit_writer: Option<std::sync::Arc<catalogd::subject_audit::SubjectAuditWriter>> = {
+        let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY")
+            .unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into());
+        match catalogd::subject_audit::SubjectAuditWriter::from_key_file(
+            store.clone(), std::path::Path::new(&key_path)
+        ).await {
+            Ok(w) => {
+                tracing::info!("v1: subject audit log enabled (key: {})", key_path);
+                Some(std::sync::Arc::new(w))
+            }
+            Err(e) => {
+                tracing::warn!(
+                    "v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)",
+                    e
+                );
+                None
+            }
+        }
+    };
+
+    app = app
        .nest("/v1", v1::router(v1::V1State {
            ai_client: ai_client.clone(),
            usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
@@ -334,7 +361,7 @@ async fn main() {
                let path_str = std::env::var("LH_WORKERS_PARQUET")
                    .unwrap_or_else(|_| "/home/profit/lakehouse/data/datasets/workers_500k.parquet".into());
                let path = std::path::Path::new(&path_str);
-                if path.exists() {
+                let raw: std::sync::Arc<dyn validator::WorkerLookup> = if path.exists() {
                    match validator::staffing::parquet_lookup::load_workers_parquet(path) {
                        Ok(lookup) => {
                            tracing::info!("v1: workers parquet loaded from {} — /v1/validate worker-existence checks enabled", path_str);
@@ -348,7 +375,16 @@ async fn main() {
                } else {
                    tracing::warn!("v1: workers parquet at {} not found — /v1/validate worker-existence checks will fail Consistency", path_str);
                    std::sync::Arc::new(validator::InMemoryWorkerLookup::new())
-                }
+                };
+                // Wrap with subject-audit decorator. When subject_audit_writer
+                // is None, the wrapper is a transparent passthrough — no
+                // audit, no overhead. When Some, every find() call fires an
+                // audit row via the per-subject HMAC chain. (Step 5 of
+                // SUBJECT_MANIFESTS_ON_CATALOGD spec.)
+                std::sync::Arc::new(v1::auditing_worker_lookup::AuditingWorkerLookup::new(
+                    raw,
+                    subject_audit_writer.clone(),
+                ))
            },
            // Phase 40 early deliverable — Langfuse trace emitter.
            // Defaults match mcp-server/tracing.ts conventions so
@@ -378,30 +414,10 @@ async fn main() {
                }
                s
            },
-            // Per-subject audit log writer (HMAC-chained JSONL). Loaded
-            // from /etc/lakehouse/subject_audit.key when present and
-            // ≥32 bytes. Missing key = audit disabled (warning, not
-            // error — gateway still serves PII paths but no audit
-            // trail). See docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
-            subject_audit: {
-                let key_path = std::env::var("LH_SUBJECT_AUDIT_KEY")
-                    .unwrap_or_else(|_| "/etc/lakehouse/subject_audit.key".into());
-                match catalogd::subject_audit::SubjectAuditWriter::from_key_file(
-                    store.clone(), std::path::Path::new(&key_path)
-                ).await {
-                    Ok(w) => {
-                        tracing::info!("v1: subject audit log enabled (key: {})", key_path);
-                        Some(std::sync::Arc::new(w))
-                    }
-                    Err(e) => {
-                        tracing::warn!(
-                            "v1: subject audit log DISABLED — {} (set LH_SUBJECT_AUDIT_KEY or place ≥32-byte key at /etc/lakehouse/subject_audit.key)",
-                            e
-                        );
-                        None
-                    }
-                }
-            },
+            // subject_audit was constructed BEFORE the V1State literal
+            // (see `subject_audit_writer` above) so validate_workers
+            // could wrap with it. Reuse the same Arc here.
+            subject_audit: subject_audit_writer.clone(),
        }));

    // Auth middleware (if enabled) — P5-001 fix 2026-04-23: