validator: Phase 43 v2 — real worker-existence + PII + name-consistency checks
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end:"

The Phase 43 scaffolds (FillValidator, EmailValidator) shipped with
TODO(phase-43 v2) markers for the actual cross-roster checks. This is
those checks landing.

The PRD calls for "the 0→85% pattern reproduces on real staffing
tasks — the iteration loop with validation in place is what made
small models successful." Worker-existence is the load-bearing check:
when the executor emits {candidate_id: "W-FAKE", name: "Imaginary"},
schema-only validation passes, and only roster lookup catches it.

Architecture:

- New `WorkerLookup` trait + `WorkerRecord` struct in lib.rs. Sync by
  design — validators hold an in-memory snapshot, no per-call I/O on
  the validation hot path. Production wraps a parquet snapshot;
  tests use `InMemoryWorkerLookup`.
- Validators take `Arc<dyn WorkerLookup>` at construction so the
  same shape covers prod + tests + future devops scaffolds.
- Contract metadata travels under JSON `_context` key alongside the
  validated payload (target_count, city, state, role, client_id for
  fills; candidate_id for emails). Keeps the Validator trait
  signature stable and lets the executor serialize context inline.

FillValidator (11 tests, was 4):
- Schema (existing)
- Completeness — endorsed count == target_count
- Worker existence — phantom candidate_id fails Consistency
- Status — non-active worker fails Consistency
- Geo/role match — city/state/role mismatch with contract fails
  Consistency
- Client blacklist — fails Policy
- Duplicate candidate_id within one fill — fails Consistency
- Name mismatch — Warning (not Error) since recruiters sometimes
  send roster updates through the proposal layer

EmailValidator (11 tests, was 4):
- Schema + length (existing)
- SSN scan (NNN-NN-NNNN) — fails Policy
- Salary disclosure (keyword + $-amount within ~40 chars) — fails
  Policy. Std-only scan, no regex dep added.
- Worker name consistency — when _context.candidate_id resolves,
  body must contain the worker's first name (Warning if missing)
- Phantom candidate_id in _context — fails Consistency
- Phone NNN-NNN-NNNN does NOT trip the SSN detector (verified by
  test); the SSN scanner explicitly rejects sequences embedded in
  longer digit runs

Pre-existing issue (NOT from this change, NOT fixed here):
crates/vectord/src/pathway_memory.rs:927 has a stale PathwayTrace
struct initializer that fails `cargo check --tests` with E0063 on
6 missing fields. `cargo check --workspace` (production) is green;
only the vectord test target is broken. Tracked for a separate fix.

Verification:
  cargo test -p validator      31 pass / 0 fail (was 13)
  cargo check --workspace      green

Next: wire `Arc<dyn WorkerLookup>` into the gateway execution loop
(generate → validate → observer-correct → retry, bounded by
max_iterations=3 per Phase 43 PRD). Production lookup impl loads
from a workers parquet snapshot — Track A gap-fix B's `_safe` view
is the right source once decided, raw workers_500k otherwise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-27 06:56:28 -05:00
co-authored by Claude Opus 4.7
parent 8aa7ee974f
commit 00c8408335
3 changed files with 656 additions and 49 deletions
+73
View File
@@ -93,3 +93,76 @@ pub trait Validator: Send + Sync {
/// Human-readable name for logs + Langfuse traces.
fn name(&self) -> &'static str;
}
// ─── Worker lookup (Phase 43 v2) ────────────────────────────────────────
//
// Validators that cross-check artifacts against the worker roster
// (FillValidator, EmailValidator) take an `Arc<dyn WorkerLookup>` at
// construction. Keeping the trait sync + in-memory mirrors the
// lakehouse pattern of "load truth into memory, validate against
// snapshot, refresh periodically" rather than per-call DB hits.
//
// Production impl: wrap a parquet snapshot loaded from
// `data/datasets/workers_500k.parquet` (or its safe view counterpart
// once Track A.B lands). Tests use `InMemoryWorkerLookup`.
/// One worker row from the staffing roster — the fields validators
/// actually read. Anything not on this struct (resume_text, scores,
/// communications) is intentionally hidden from the validator path.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct WorkerRecord {
pub candidate_id: String,
pub name: String,
/// Free-form. Validators check for `"active"` (any other value
/// fails the status check). Common values from existing data:
/// "active", "inactive", "placed", "blacklisted".
pub status: String,
pub city: Option<String>,
pub state: Option<String>,
pub role: Option<String>,
/// Client ids this worker has been blacklisted from. Populated
/// from joining a blacklist table; empty when not provided.
#[serde(default)]
pub blacklisted_clients: Vec<String>,
}
/// Worker lookup contract. Sync by design — implementations should
/// hold an in-memory snapshot, not perform per-call I/O.
pub trait WorkerLookup: Send + Sync {
fn find(&self, candidate_id: &str) -> Option<WorkerRecord>;
}
/// HashMap-backed lookup. Used by validator unit tests + as a
/// reasonable bootstrap impl for production once the parquet loader
/// fills it on startup.
pub struct InMemoryWorkerLookup {
rows: std::collections::HashMap<String, WorkerRecord>,
}
impl InMemoryWorkerLookup {
pub fn new() -> Self {
Self { rows: Default::default() }
}
pub fn from_records(records: Vec<WorkerRecord>) -> Self {
let mut rows = std::collections::HashMap::with_capacity(records.len());
for r in records {
rows.insert(r.candidate_id.clone(), r);
}
Self { rows }
}
pub fn insert(&mut self, record: WorkerRecord) {
self.rows.insert(record.candidate_id.clone(), record);
}
pub fn len(&self) -> usize { self.rows.len() }
pub fn is_empty(&self) -> bool { self.rows.is_empty() }
}
impl Default for InMemoryWorkerLookup {
fn default() -> Self { Self::new() }
}
impl WorkerLookup for InMemoryWorkerLookup {
fn find(&self, candidate_id: &str) -> Option<WorkerRecord> {
self.rows.get(candidate_id).cloned()
}
}