real_006 diagnosis: Q43 leak is cross-city, not cross-role

Traced w-279's path through the substrate. The leak source is Q49 (Packers in Indianapolis IN for Midway Distribution), NOT Q18 as the initial reading suspected. Q49 recorded w-279 with role=Packers, client=Midway Distribution, city=Indianapolis. Q43 (Packer in Chicago IL for Midway Distribution) ran later. roleEqual("Packer","Packers") → both normalize to "packer" → role gate passes (correctly, by design — they ARE the same role under plural-strip). Cosine distance between Q49's recorded query and Q43's query is small enough to fit inside the 0.20 inject threshold because role + client + count + time-token dominate the embedding (only the city and singular/plural noun differ). Inject fires, w-279 surfaces at Q43's warm top-1 in Chicago, judge correctly rates 2/5 — wrong city. The role gate IS working. What's missing is a CITY gate. Real_002's fix targeted cross-role bleed (Forklift → CNC). real_006 surfaced cross-city bleed within same role + same client — a hole prior tests structurally couldn't reach because they all sourced from rows 0-9 where no such pair existed. Concrete fix surface documented (1 new field, 2 gate checks, 1 regex, ~5 tests). Half a session of work, same shape as real_002. Not implementing tonight — diagnosis only. The 18 unit-level role-gate tests still pass, confirming the gate is doing what it was specified to do. The bug is a missing specification, not a broken implementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
real_006: distribution-shift test on rows 10-59 of fill_events
2026-05-05 05:13:19 -05:00 · 2026-05-05 04:54:03 -05:00 · 2026-05-05 03:27:47 -05:00 · 2026-05-03 05:27:14 -05:00 · 2026-05-03 04:55:49 -05:00 · 2026-05-03 04:29:53 -05:00
18 changed files with 1647 additions and 26 deletions
@@ -1,7 +1,7 @@
 # STATE OF PLAY — Lakehouse-Go
 **Last verified:** 2026-05-02 ~11:30 CDT
-**Verified by:** **production-readiness gauntlet** — 21/21 smoke chain green, per-component scrum across 4 bundles, **3 cross-runtime parity probes all green post-fix** (validator: **6/6 match** after wire-format alignment shipped; materializer: 2/2 after omitempty fix; extract_json: 12/12). All findings surfaced by the parity probes have been actioned. Disposition: `reports/cutover/gauntlet_2026-05-02/disposition.md`.
+**Verified by:** **production-readiness gauntlet** — 21/21 smoke chain green, per-component scrum across 4 bundles, **6 cross-runtime parity probes all green: 38/38 byte-identical assertions** (validator 6/6 after wire-format alignment, materializer 2/2 after omitempty fix, extract_json 12/12, session_log 4/4, embed 8/8 cosine, subject_audit 6/6 after the three runtime-divergence-class fixes documented at `lakehouse/STATE_OF_PLAY.md` lines 36-39). All findings surfaced by the parity probes have been actioned. Disposition: `reports/cutover/gauntlet_2026-05-02/disposition.md`. Subject-audit probe added 2026-05-03 (`262a77a`).
 > **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo) but ASK if ambiguous — the Rust system at `/home/profit/lakehouse/` is also receiving real work as of 2026-05-02 (Lance backend hardening, observability parity, sidecar drop — see `lakehouse/STATE_OF_PLAY.md`). The cross-runtime decisions tracker at `docs/ARCHITECTURE_COMPARISON.md` is the single source of truth for what's open across both. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
@@ -59,6 +59,7 @@ Don't:
 | 2026-05-02 | **extract_json parity probe — 12/12 match across edge cases** | New `scripts/cutover/parity/extract_json_parity.sh` runs identical model-output strings through Rust `gateway::v1::iterate::extract_json` AND Go `validator.ExtractJSON`. 12 fixtures: fenced/unfenced blocks, nested objects, unicode, escaped quotes, top-level array, malformed JSON. Substrate gate: `cargo test -p gateway extract_json` PASS before probe. Result: **12/12 match.** Algorithms genuinely equivalent. Rust side gained `pub` on `extract_json` + new `bin/parity_extract_json` (~30 LOC). |
 | 2026-05-02 | **Validator wire-format alignment — DONE** | Custom `MarshalJSON`/`UnmarshalJSON` on Go's `validator.ValidationError` emits the Rust serde-tagged-enum shape `{"Schema":{"field":"x","reason":"y"}}`. UnmarshalJSON also accepts the legacy flat shape (migration safety) and rejects unknown variants (drift guard for future Rust enum additions). 4 new pinning tests in `types_test.go`. Re-run validator parity probe: **6/6 match** (was 1/6). |
 | 2026-05-02 | **Lance backend gauntlet (4-pack + root-cause fix) — DONE** | Lance crate had zero tests + no smoke when audited this morning. Shipped: (a) `sanitize_lance_err` over all 5 routes (search/doc/index/append/migrate) — missing-index now 404 not 500, no `/home/` or `/root/.cargo/` paths leaked; (b) 7 unit tests in `crates/vectord-lance` with synth Parquet helper; (c) 9-probe `scripts/lance_smoke.sh` against live `:3100`; (d) 10M re-bench (`reports/lance_10m_rebench_2026-05-02.md`) — search warm ~20ms, search cold ~46ms median. Bench surfaced doc-fetch p50 ~100ms (300x slower than ADR-019 100K projection); root-caused to lance-bench bypassing IndexMeta → warming auto-build never fired → no `doc_id` btree. **Fix shipped (commit `5d30b3d`)**: `lance_migrate` HTTP handler now auto-builds the btree inline (1.2s on 10M, +269MB), drops doc-fetch to ~5ms (20x). Live verified 9/9 smoke + post-restart doc-fetch 4-15ms. |
 | 2026-05-03 | **Subject manifests + per-subject HMAC audit log — DONE on Rust + Go** | Local-first compliance substrate per `lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md` Steps 1-8. Rust shipped: `SubjectManifest` type + Registry CRUD (`crates/catalogd/src/registry.rs`), `SubjectAuditWriter` with HMAC-SHA256 chain + per-subject Mutex serialization + canonical-JSON via BTreeMap (`subject_audit.rs`), backfill ETL (`bin/backfill_subjects`), gateway tool dispatch + validator decorator wiring, legal-tier `/audit/subject/{id}` endpoint with constant-time-eq token + tampering detection, daily `bin/retention_sweep` (BIPA-aware, idempotent, no auto-mutation). Go shipped: identical `internal/catalogd/subject.go` reader + `VerifyChain` over RAW LINE BYTES (avoids time-precision drift), 11 unit tests. **6th cross-runtime parity probe**: `scripts/cutover/parity/subject_audit_parity.sh` — 6/6 byte-identical assertions across known-answer fixture + 5 real production audit logs. Surfaced + closed three drift classes in authoring loop: (1) Go `omitempty` stripping `trace_id:""`; (2) `time.RFC3339Nano` truncating trailing-zero nanoseconds where chrono AutoSi keeps 9 digits; (3) Go `json.Marshal` HTML-escaping `<>&` where serde keeps literal — fixed via `marshalNoEscapeHTML` + raw-bytes canonicalization. Two cross-lineage scrums caught real bugs each round (chain corruption race, schema-evolution HMAC drift, hardcoded "success" classifier, token min length, chain_root from windowed slice, tampering detection, HTML escape divergence). |
 | _open_ | Decide Lance vs Parquet+HNSW for primary | Lance verified production-ready at 10M (this morning's gauntlet). HNSW at 10M doesn't fit RAM (~60GB for vectors+graph), so the comparison is between Lance and Parquet+HNSW-with-spilling. Decide once we have a 10M ingest scenario where the Parquet path is bottlenecked. |
 | _open_ | Pick Go primary vs Rust primary | Both viable. Go has perf edge after today; Rust has production deploy + producer-side completeness. |
@@ -0,0 +1,371 @@
 // Subject manifests + per-subject audit-log chain verification.
 //
 // Specification: /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md.
 //
 // This file is the Go side of Step 8 (cross-runtime parity). The Rust
 // side is the writer (crates/catalogd/src/subject_audit.rs); Go is a
 // READER + VERIFIER only — both sides serialize / hash with identical
 // algorithms so a chain written by Rust verifies under Go.
 //
 // Algorithm (must match Rust crates/catalogd/src/subject_audit.rs exactly):
 //   row_hmac = HMAC-SHA256(key, prev_chain_hash_bytes || canonical_row_bytes)
 //
 // where:
 //   - prev_chain_hash_bytes is the ASCII bytes of the previous row's
 //     row_hmac field, OR the literal ASCII string "GENESIS" for the
 //     first row (no hex decode!)
 //   - canonical_row_bytes is the row JSON with the row_hmac field
 //     dropped, with all object keys sorted alphabetically at every
 //     nesting depth, with no insignificant whitespace
 //   - the final hash is rendered as 64 lowercase hex characters
 package catalogd
 import (
 	"bytes"
 	"crypto/hmac"
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"fmt"
 	"os"
 	"sort"
 	"strings"
 	"time"
 )
 const GenesisHash = "GENESIS"
 // SubjectManifest mirrors crates/shared/src/types.rs::SubjectManifest.
 // Keep field tags byte-identical so manifest JSON written by Rust round-trips.
 //
 // Vertical: omitempty matches Rust's serde-default deserializer accepting
 // missing field (default Unknown). For the OTHER fields we deliberately
 // avoid omitempty since Rust always emits them.
 type SubjectManifest struct {
 	Schema              string                `json:"schema"`
 	CandidateID         string                `json:"candidate_id"`
 	CreatedAt           time.Time             `json:"created_at"`
 	UpdatedAt           time.Time             `json:"updated_at"`
 	Status              string                `json:"status"`
 	Vertical            string                `json:"vertical,omitempty"`
 	Consent             SubjectConsent        `json:"consent"`
 	Retention           SubjectRetention      `json:"retention"`
 	Datasets            []SubjectDatasetRef   `json:"datasets"`
 	SafeViews           []string              `json:"safe_views"`
 	AuditLogPath        string                `json:"audit_log_path"`
 	AuditLogChainRoot   string                `json:"audit_log_chain_root"`
 	BiometricCollection *BiometricCollection  `json:"biometric_collection,omitempty"`
 }
 // BiometricCollection mirrors crates/shared/src/types.rs::BiometricCollection.
 // Written by the Phase 1.6 Gate 3 photo-upload endpoint. None until the
 // first biometric upload; cleared on BIPA erasure.
 type BiometricCollection struct {
 	DataPath            string          `json:"data_path"`
 	TemplateHash        string          `json:"template_hash"`
 	CollectedAt         time.Time       `json:"collected_at"`
 	ConsentVersionHash  string          `json:"consent_version_hash,omitempty"`
 	Classifications     json.RawMessage `json:"classifications,omitempty"`
 }
 type SubjectConsent struct {
 	GeneralPii GeneralPiiConsent `json:"general_pii"`
 	Biometric  BiometricConsent  `json:"biometric"`
 }
 type GeneralPiiConsent struct {
 	Status       string     `json:"status"`
 	Version      string     `json:"version,omitempty"`
 	GivenAt      *time.Time `json:"given_at,omitempty"`
 	WithdrawnAt  *time.Time `json:"withdrawn_at,omitempty"`
 }
 type BiometricConsent struct {
 	Status         string     `json:"status"`
 	RetentionUntil *time.Time `json:"retention_until,omitempty"`
 }
 type SubjectRetention struct {
 	GeneralPiiUntil time.Time `json:"general_pii_until"`
 	Policy          string    `json:"policy,omitempty"`
 }
 type SubjectDatasetRef struct {
 	Name      string `json:"name"`
 	KeyColumn string `json:"key_column"`
 	KeyValue  string `json:"key_value"`
 }
 // SubjectAuditRow mirrors crates/shared/src/types.rs::SubjectAuditRow.
 // JSON field order does NOT matter (canonicalizer sorts at hash time)
 // but field names + types MUST match for round-trip.
 type SubjectAuditRow struct {
 	Schema         string         `json:"schema"`
 	Ts             time.Time      `json:"ts"`
 	CandidateID    string         `json:"candidate_id"`
 	Accessor       AuditAccessor  `json:"accessor"`
 	FieldsAccessed []string       `json:"fields_accessed"`
 	Result         string         `json:"result"`
 	PrevChainHash  string         `json:"prev_chain_hash"`
 	RowHmac        string         `json:"row_hmac"`
 }
 // AuditAccessor mirrors crates/shared/src/types.rs::AuditAccessor.
 //
 // IMPORTANT: trace_id has NO omitempty — Rust's serde always writes
 // the field (even empty) because `#[serde(default)]` only affects
 // READS. Stripping it here produces different canonical bytes than
 // the writer used → HMAC mismatch on real production rows. (Caught
 // 2026-05-03 by subject_audit_parity.sh on a live WORKER-1 row with
 // trace_id="".)
 type AuditAccessor struct {
 	Kind    string `json:"kind"`
 	Daemon  string `json:"daemon"`
 	Purpose string `json:"purpose"`
 	TraceID string `json:"trace_id"`
 }
 // LoadSubjectManifest reads + parses a subject manifest JSON file.
 func LoadSubjectManifest(path string) (*SubjectManifest, error) {
 	bytes, err := os.ReadFile(path)
 	if err != nil {
 		return nil, fmt.Errorf("read manifest %s: %w", path, err)
 	}
 	var m SubjectManifest
 	if err := json.Unmarshal(bytes, &m); err != nil {
 		return nil, fmt.Errorf("parse manifest %s: %w", path, err)
 	}
 	return &m, nil
 }
 // AuditLogEntry is one parsed audit row PLUS the raw line bytes from
 // the file. The raw bytes are load-bearing: we MUST canonicalize from
 // them (not from the re-marshaled struct) to avoid time-precision drift.
 //
 // Why: Rust's chrono RFC3339-AutoSi serializes nanoseconds with 9 digits
 // (e.g. "461439210"). Go's time.RFC3339Nano strips trailing zeros (e.g.
 // "46143921"). Round-tripping through time.Time changes the byte sequence
 // and breaks the HMAC. Read once, hash from the original bytes.
 // (Caught 2026-05-03 by subject_audit_parity.sh on a real WORKER-5 row
 // whose nanoseconds happened to end in 0.)
 type AuditLogEntry struct {
 	Row SubjectAuditRow
 	Raw []byte
 }
 // ReadAuditLog parses an audit JSONL file. Returns one AuditLogEntry
 // per non-empty line. Defensive: blank lines are skipped, unparseable
 // lines are skipped (matches Rust read_rows_in_range).
 func ReadAuditLog(path string) ([]AuditLogEntry, error) {
 	bytes, err := os.ReadFile(path)
 	if err != nil {
 		if os.IsNotExist(err) {
 			return nil, nil
 		}
 		return nil, fmt.Errorf("read audit log %s: %w", path, err)
 	}
 	var entries []AuditLogEntry
 	for _, line := range strings.Split(string(bytes), "\n") {
 		trimmed := strings.TrimSpace(line)
 		if trimmed == "" {
 			continue
 		}
 		var row SubjectAuditRow
 		if err := json.Unmarshal([]byte(trimmed), &row); err != nil {
 			continue
 		}
 		entries = append(entries, AuditLogEntry{Row: row, Raw: []byte(trimmed)})
 	}
 	return entries, nil
 }
 // canonicalRowBytesFromRaw is the production canonicalizer used by
 // VerifyChain. It accepts the raw line bytes from the audit JSONL,
 // drops the row_hmac key, sorts all keys alphabetically at every
 // nesting depth, and re-emits compact JSON. Numbers + strings pass
 // through verbatim (json.Number preserves number text, strings are
 // re-JSON-escaped identically to Rust's serde_json).
 //
 // CRITICAL: hashing must always go through this function. NEVER hash
 // the bytes produced by canonicalRowBytesFromStruct against a real
 // production chain — the time-precision drift will give wrong HMACs
 // for any row whose nanoseconds end in 0.
 func canonicalRowBytesFromRaw(rawLine []byte) ([]byte, error) {
 	dec := json.NewDecoder(strings.NewReader(string(rawLine)))
 	dec.UseNumber()
 	var v any
 	if err := dec.Decode(&v); err != nil {
 		return nil, fmt.Errorf("decode raw line: %w", err)
 	}
 	if obj, ok := v.(map[string]any); ok {
 		delete(obj, "row_hmac")
 	}
 	var buf strings.Builder
 	if err := writeCanonical(&buf, v); err != nil {
 		return nil, err
 	}
 	return []byte(buf.String()), nil
 }
 // canonicalRowBytesFromStruct is for the parity probe's known-answer
 // vector + tests where we WANT to canonicalize a Go-built row from
 // scratch (no original bytes exist yet). Production verification uses
 // canonicalRowBytesFromRaw against the file bytes; this function is
 // only safe for synthetic inputs whose ts has no trailing-zero nanos.
 func canonicalRowBytesFromStruct(row *SubjectAuditRow) ([]byte, error) {
 	raw, err := json.Marshal(row)
 	if err != nil {
 		return nil, fmt.Errorf("marshal row: %w", err)
 	}
 	return canonicalRowBytesFromRaw(raw)
 }
 // marshalNoEscapeHTML wraps json.Encoder with HTML escaping disabled.
 //
 // Why: Go's json.Marshal escapes `<`, `>`, `&` to `<`, `>`,
 // `&` by default. Rust's serde_json::to_vec keeps them literal.
 // Any string field containing one of those characters would produce
 // different canonical bytes across runtimes → broken HMAC chain.
 // (Caught 2026-05-03 by opus scrum WARN on parity_subject_audit.rs:
 //  canonical_json — initially undetected because no production audit
 //  field contained `<>&`, but realistic for purpose strings like
 //  "error & retry" or trace_id "<HTTP-Request-Id>".)
 //
 // Also strips the trailing newline json.Encoder appends — that newline
 // is meaningful to JSONL consumers but is junk for hash input.
 func marshalNoEscapeHTML(v any) ([]byte, error) {
 	var buf bytes.Buffer
 	enc := json.NewEncoder(&buf)
 	enc.SetEscapeHTML(false)
 	if err := enc.Encode(v); err != nil {
 		return nil, err
 	}
 	out := buf.Bytes()
 	return bytes.TrimRight(out, "\n"), nil
 }
 // writeCanonical recursively writes v as canonical JSON: object keys
 // sorted alphabetically, no insignificant whitespace. Arrays preserve
 // element order (semantically significant per spec §3).
 //
 // All scalar emission goes through marshalNoEscapeHTML so the byte
 // sequence matches Rust's serde_json output character-for-character.
 func writeCanonical(buf *strings.Builder, v any) error {
 	switch t := v.(type) {
 	case map[string]any:
 		keys := make([]string, 0, len(t))
 		for k := range t {
 			keys = append(keys, k)
 		}
 		sort.Strings(keys)
 		buf.WriteByte('{')
 		for i, k := range keys {
 			if i > 0 {
 				buf.WriteByte(',')
 			}
 			ks, err := marshalNoEscapeHTML(k)
 			if err != nil {
 				return fmt.Errorf("marshal key: %w", err)
 			}
 			buf.Write(ks)
 			buf.WriteByte(':')
 			if err := writeCanonical(buf, t[k]); err != nil {
 				return err
 			}
 		}
 		buf.WriteByte('}')
 	case []any:
 		buf.WriteByte('[')
 		for i, elem := range t {
 			if i > 0 {
 				buf.WriteByte(',')
 			}
 			if err := writeCanonical(buf, elem); err != nil {
 				return err
 			}
 		}
 		buf.WriteByte(']')
 	default:
 		bs, err := marshalNoEscapeHTML(v)
 		if err != nil {
 			return fmt.Errorf("marshal scalar: %w", err)
 		}
 		buf.Write(bs)
 	}
 	return nil
 }
 // CanonicalAndHmac is a helper for the parity probe: canonicalizes the
 // row from scratch (struct → JSON → canonical), computes the HMAC
 // against the given prev_hash, returns both. Used only for the
 // known-answer fixture; production verification uses VerifyChain
 // (which canonicalizes from the original file bytes to dodge time
 // precision drift).
 func CanonicalAndHmac(row *SubjectAuditRow, key []byte, prevHash string) (canonical []byte, hmacHex string, err error) {
 	canonical, err = canonicalRowBytesFromStruct(row)
 	if err != nil {
 		return nil, "", err
 	}
 	hmacHex = computeRowHMAC(key, prevHash, canonical)
 	return canonical, hmacHex, nil
 }
 // computeRowHMAC = HMAC-SHA256(key, prev_hash_ascii_bytes || canonical_row_bytes)
 // rendered as 64-char lowercase hex.
 func computeRowHMAC(key []byte, prevHash string, canonical []byte) string {
 	mac := hmac.New(sha256.New, key)
 	mac.Write([]byte(prevHash))
 	mac.Write(canonical)
 	return hex.EncodeToString(mac.Sum(nil))
 }
 // VerifyChain replays the chain for a subject's audit log. Returns the
 // number of rows verified, the chain tip (last row's row_hmac, or
 // GENESIS when log is empty), and an error describing the first chain
 // break found.
 //
 // CRITICAL: canonicalizes from each entry's RAW LINE BYTES, not from
 // the parsed struct. Round-tripping through encoding/json + time.Time
 // strips trailing zeros from RFC3339 nanos and produces different
 // canonical bytes than the Rust writer used (which used chrono's
 // AutoSi 9-digit format). Catches: 2026-05-03 WORKER-5 nanos
 // 461439210 → Rust emits 9 digits, Go's time.RFC3339Nano emits 8.
 //
 // Special case: empty/missing log returns (0, GENESIS, nil).
 func VerifyChain(entries []AuditLogEntry, key []byte) (count int, chainTip string, err error) {
 	prev := GenesisHash
 	for i, entry := range entries {
 		if entry.Row.PrevChainHash != prev {
 			return i, prev, fmt.Errorf("chain break at row %d: prev_chain_hash=%q expected=%q",
 				i+1, entry.Row.PrevChainHash, prev)
 		}
 		claimed := entry.Row.RowHmac
 		canonical, cerr := canonicalRowBytesFromRaw(entry.Raw)
 		if cerr != nil {
 			return i, prev, fmt.Errorf("canonicalize row %d: %w", i+1, cerr)
 		}
 		recomputed := computeRowHMAC(key, prev, canonical)
 		if recomputed != claimed {
 			return i, prev, fmt.Errorf("hmac mismatch at row %d: stored=%s recomputed=%s",
 				i+1, claimed, recomputed)
 		}
 		prev = claimed
 		count = i + 1
 	}
 	chainTip = prev
 	return count, chainTip, nil
 }
 // LoadKeyFile reads a signing key file. Refuses keys shorter than 32
 // bytes (matches Rust: 2026-05-03 kimi BLOCK fix lifted the minimum
 // from 16 to 32 to align with HMAC-SHA256 best practice).
 func LoadKeyFile(path string) ([]byte, error) {
 	key, err := os.ReadFile(path)
 	if err != nil {
 		return nil, fmt.Errorf("read key file %s: %w", path, err)
 	}
 	if len(key) < 32 {
 		return nil, fmt.Errorf("signing key is %d bytes; recommend ≥32 bytes for HMAC-SHA256", len(key))
 	}
 	return key, nil
 }
@@ -0,0 +1,364 @@
 package catalogd
 import (
 	"crypto/hmac"
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"strings"
 	"testing"
 	"time"
 )
 // deterministicKey is the same fixture key the Rust tests use:
 // (0u8..32).collect() — so a Rust-written chain verifies under Go.
 func deterministicKey() []byte {
 	k := make([]byte, 32)
 	for i := range k {
 		k[i] = byte(i)
 	}
 	return k
 }
 func mkRow(candidateID string, fields []string, prevHash, ts string) SubjectAuditRow {
 	t, _ := time.Parse(time.RFC3339Nano, ts)
 	return SubjectAuditRow{
 		Schema:      "subject_audit.v1",
 		Ts:          t,
 		CandidateID: candidateID,
 		Accessor: AuditAccessor{
 			Kind:    "gateway_lookup",
 			Daemon:  "gateway",
 			Purpose: "fill_validation",
 			TraceID: "",
 		},
 		FieldsAccessed: fields,
 		Result:         "success",
 		PrevChainHash:  prevHash,
 		RowHmac:        "", // computed below
 	}
 }
 // TestCanonicalJSON_KeysSortedAlphabetically asserts the same property
 // the Rust unit test asserts (subject_audit::tests::canonical_json_sorts_keys_alphabetically).
 func TestCanonicalJSON_KeysSortedAlphabetically(t *testing.T) {
 	v := map[string]any{
 		"z": 1,
 		"a": 2,
 		"m": map[string]any{"y": 1, "b": 2},
 	}
 	var buf strings.Builder
 	if err := writeCanonical(&buf, v); err != nil {
 		t.Fatalf("canonical: %v", err)
 	}
 	s := buf.String()
 	a, m, z := strings.Index(s, "\"a\""), strings.Index(s, "\"m\""), strings.Index(s, "\"z\"")
 	if !(a < m && m < z) {
 		t.Fatalf("top-level keys out of order: %s", s)
 	}
 	b, y := strings.Index(s, "\"b\""), strings.Index(s, "\"y\"")
 	if !(b < y) {
 		t.Fatalf("nested keys out of order: %s", s)
 	}
 }
 // TestCanonicalJSON_ArraysPreserveOrder asserts arrays are NOT sorted —
 // matches Rust subject_audit::tests::canonical_json_arrays_preserve_order.
 func TestCanonicalJSON_ArraysPreserveOrder(t *testing.T) {
 	v := map[string]any{"k": []any{"c", "a", "b"}}
 	var buf strings.Builder
 	if err := writeCanonical(&buf, v); err != nil {
 		t.Fatalf("canonical: %v", err)
 	}
 	if !strings.Contains(buf.String(), "\"c\",\"a\",\"b\"") {
 		t.Fatalf("array order altered: %s", buf.String())
 	}
 }
 // buildEntry produces an AuditLogEntry by computing the HMAC against
 // the row's struct-derived canonical bytes, then storing the resulting
 // row JSON as the raw bytes. Test-only — production reads raw bytes
 // straight from disk so the time-precision drift doesn't apply.
 func buildEntry(row SubjectAuditRow, key []byte, prev string) AuditLogEntry {
 	canon, err := canonicalRowBytesFromStruct(&row)
 	if err != nil {
 		panic(err)
 	}
 	row.RowHmac = computeRowHMAC(key, prev, canon)
 	raw, err := json.Marshal(row)
 	if err != nil {
 		panic(err)
 	}
 	return AuditLogEntry{Row: row, Raw: raw}
 }
 // TestVerifyChain_ReplaysAndReachesTip writes 3 rows with HMACs computed
 // the same way Rust would, then verifies they chain. This is the local
 // half of the parity contract — the cross-runtime half (Rust writes,
 // Go verifies) is covered by scripts/cutover/parity/subject_audit_parity.sh.
 func TestVerifyChain_ReplaysAndReachesTip(t *testing.T) {
 	key := deterministicKey()
 	r1 := mkRow("CAND-PARITY", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
 	e1 := buildEntry(r1, key, GenesisHash)
 	r2 := mkRow("CAND-PARITY", []string{"phone"}, e1.Row.RowHmac, "2026-05-03T12:00:01Z")
 	e2 := buildEntry(r2, key, e1.Row.RowHmac)
 	r3 := mkRow("CAND-PARITY", []string{"email"}, e2.Row.RowHmac, "2026-05-03T12:00:02Z")
 	e3 := buildEntry(r3, key, e2.Row.RowHmac)
 	count, tip, err := VerifyChain([]AuditLogEntry{e1, e2, e3}, key)
 	if err != nil {
 		t.Fatalf("verify failed: %v", err)
 	}
 	if count != 3 {
 		t.Fatalf("expected 3 rows verified, got %d", count)
 	}
 	if tip != e3.Row.RowHmac {
 		t.Fatalf("chain tip wrong: tip=%s expected=%s", tip, e3.Row.RowHmac)
 	}
 }
 // TestVerifyChain_EmptyLogIsTriviallyValid mirrors Rust's empty-log
 // special case: 0 rows, GENESIS tip, no error.
 func TestVerifyChain_EmptyLogIsTriviallyValid(t *testing.T) {
 	count, tip, err := VerifyChain(nil, deterministicKey())
 	if err != nil {
 		t.Fatalf("empty log returned error: %v", err)
 	}
 	if count != 0 {
 		t.Fatalf("expected 0 rows on empty log, got %d", count)
 	}
 	if tip != GenesisHash {
 		t.Fatalf("expected GENESIS tip on empty log, got %q", tip)
 	}
 }
 // TestVerifyChain_TamperDetected: tamper the raw line's `result` field
 // (the canonicalizer sees the new bytes; HMAC mismatches the stored hash).
 func TestVerifyChain_TamperDetected(t *testing.T) {
 	key := deterministicKey()
 	r1 := mkRow("CAND-T", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
 	e1 := buildEntry(r1, key, GenesisHash)
 	// Tamper: replace "success" with "denied" in the raw bytes ONLY.
 	// The struct's row_hmac (used as the "stored" comparator) stays put.
 	e1.Raw = []byte(strings.Replace(string(e1.Raw), `"success"`, `"denied"`, 1))
 	_, _, err := VerifyChain([]AuditLogEntry{e1}, key)
 	if err == nil {
 		t.Fatal("expected hmac mismatch after tamper, got nil")
 	}
 	if !strings.Contains(err.Error(), "hmac mismatch") {
 		t.Fatalf("expected hmac mismatch, got: %v", err)
 	}
 }
 // TestVerifyChain_BadKeyRejectsValidRows: same rows + wrong key = mismatch.
 func TestVerifyChain_BadKeyRejectsValidRows(t *testing.T) {
 	good := deterministicKey()
 	r1 := mkRow("CAND-BK", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
 	e1 := buildEntry(r1, good, GenesisHash)
 	bad := make([]byte, 32)
 	for i := range bad {
 		bad[i] = 0xff
 	}
 	_, _, err := VerifyChain([]AuditLogEntry{e1}, bad)
 	if err == nil {
 		t.Fatal("expected hmac mismatch with wrong key")
 	}
 }
 // TestComputeRowHMAC_StableAcrossRuns: same row + same key always = same hash.
 func TestComputeRowHMAC_StableAcrossRuns(t *testing.T) {
 	key := deterministicKey()
 	r := mkRow("CAND-S", []string{"a", "b"}, GenesisHash, "2026-05-03T12:00:00Z")
 	c1, _ := canonicalRowBytesFromStruct(&r)
 	c2, _ := canonicalRowBytesFromStruct(&r)
 	if string(c1) != string(c2) {
 		t.Fatalf("canonical bytes unstable across runs:\n  c1=%s\n  c2=%s", c1, c2)
 	}
 	h1 := computeRowHMAC(key, GenesisHash, c1)
 	h2 := computeRowHMAC(key, GenesisHash, c2)
 	if h1 != h2 {
 		t.Fatalf("hmac unstable across runs: %s vs %s", h1, h2)
 	}
 	if len(h1) != 64 {
 		t.Fatalf("hmac wrong length %d", len(h1))
 	}
 	// Sanity: hex-decodable.
 	if _, err := hex.DecodeString(h1); err != nil {
 		t.Fatalf("hmac not hex: %v", err)
 	}
 }
 // TestKnownAnswerVector matches a Go-computed reference. The same
 // inputs must produce this exact byte string under Rust as well —
 // scripts/cutover/parity/subject_audit_parity.sh runs the Rust helper
 // against this exact fixture and asserts byte-identical output.
 //
 // If you change the fixture, rebuild Rust's parity_subject_audit + Go's
 // helper and update both sides together.
 func TestKnownAnswerVector(t *testing.T) {
 	key := deterministicKey()
 	r := SubjectAuditRow{
 		Schema:      "subject_audit.v1",
 		Ts:          time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC),
 		CandidateID: "WORKER-FIXED",
 		Accessor: AuditAccessor{
 			Kind:    "gateway_lookup",
 			Daemon:  "gateway",
 			Purpose: "parity_test",
 			TraceID: "trace-fixed",
 		},
 		FieldsAccessed: []string{"name"},
 		Result:         "success",
 		PrevChainHash:  GenesisHash,
 		RowHmac:        "",
 	}
 	canon, err := canonicalRowBytesFromStruct(&r)
 	if err != nil {
 		t.Fatalf("canonical: %v", err)
 	}
 	t.Logf("canonical bytes: %s", canon)
 	hmacHex := computeRowHMAC(key, GenesisHash, canon)
 	t.Logf("hmac: %s", hmacHex)
 	// Sanity: round-trip through encoding/json + canonicalization is stable.
 	again, err := canonicalRowBytesFromStruct(&r)
 	if err != nil {
 		t.Fatalf("canonical 2: %v", err)
 	}
 	if string(canon) != string(again) {
 		t.Fatalf("canonical drift: %s vs %s", canon, again)
 	}
 	// Sanity: real HMAC against the canonical bytes.
 	mac := hmac.New(sha256.New, key)
 	mac.Write([]byte(GenesisHash))
 	mac.Write(canon)
 	expected := hex.EncodeToString(mac.Sum(nil))
 	if hmacHex != expected {
 		t.Fatalf("computeRowHMAC drift: %s vs %s", hmacHex, expected)
 	}
 }
 // TestVerifyChain_HtmlChars_NotEscaped is the regression test for the
 // 2026-05-03 opus scrum WARN: Go's json.Marshal escapes `<`, `>`, `&`
 // to `<`, `>`, `&` by default; Rust's serde_json keeps
 // them literal. Audit rows with these chars in any string field would
 // silently break the chain across runtimes. Fix is in writeCanonical's
 // marshalNoEscapeHTML helper. This test asserts canonical bytes contain
 // the literal `<`, `>`, `&` (proving the fix is in place).
 func TestVerifyChain_HtmlChars_NotEscaped(t *testing.T) {
 	r := mkRow("CAND-HTML", []string{"name"}, GenesisHash, "2026-05-03T12:00:00Z")
 	r.Accessor.Purpose = "error & retry"   // & must NOT be &
 	r.Accessor.TraceID = "<HTTP-Req-Id>"   // < and > must NOT be < / >
 	canon, err := canonicalRowBytesFromStruct(&r)
 	if err != nil {
 		t.Fatalf("canonical: %v", err)
 	}
 	s := string(canon)
 	// FAIL if the bytes contain Go's HTML-safe < / > / &
 	// escape sequences (six raw chars each: backslash, u, 0, 0, hex, hex).
 	// Those wouldn't match Rust's literal-char output and would silently
 	// break the cross-runtime HMAC chain. Note: the strings below are
 	// raw-string literals — the backslash + u006xx is six literal bytes,
 	// NOT a Go-source unicode escape.
 	if strings.Contains(s, "\\u003c") || strings.Contains(s, "\\u003e") || strings.Contains(s, "\\u0026") {
 		t.Fatalf("canonical bytes contain Go HTML-escape sequences (would diverge from Rust):\n%s", s)
 	}
 	// PASS only if the literal chars survived round-trip.
 	if !strings.Contains(s, "\"<HTTP-Req-Id>\"") || !strings.Contains(s, "\"error & retry\"") {
 		t.Fatalf("canonical bytes missing literal <>&:\n%s", s)
 	}
 }
 // TestVerifyChain_RawBytesPreserveTimePrecision is the regression test
 // for the 2026-05-03 WORKER-5 finding: when a row's nanoseconds end in
 // 0, time.RFC3339Nano strips the trailing zero on re-marshal, producing
 // different canonical bytes than Rust's chrono AutoSi (which always
 // emits 9 digits). VerifyChain MUST canonicalize from the raw line
 // bytes to avoid this drift. Test feeds a hand-crafted raw line whose
 // ts has a trailing-zero nano value and asserts verify succeeds when
 // the chain hash was computed against THOSE EXACT bytes.
 func TestVerifyChain_RawBytesPreserveTimePrecision(t *testing.T) {
 	key := deterministicKey()
 	// Hand-crafted raw line exactly as Rust would write it, with
 	// nanoseconds=461439210 (trailing zero present).
 	rawNoHmac := `{"schema":"subject_audit.v1","ts":"2026-05-03T09:12:47.461439210Z","candidate_id":"WORKER-5","accessor":{"kind":"validator_lookup","daemon":"gateway","purpose":"validator_worker_lookup","trace_id":""},"fields_accessed":["exists"],"result":"not_found","prev_chain_hash":"GENESIS"}`
 	canonical, err := canonicalRowBytesFromRaw([]byte(rawNoHmac))
 	if err != nil {
 		t.Fatalf("canonicalize raw: %v", err)
 	}
 	hmacHex := computeRowHMAC(key, GenesisHash, canonical)
 	// Compose the full row by injecting row_hmac at the end (matches
 	// what the Rust writer produces — declaration order + appended hmac).
 	rawFull := strings.TrimSuffix(rawNoHmac, "}") + `,"row_hmac":"` + hmacHex + `"}`
 	var row SubjectAuditRow
 	if err := json.Unmarshal([]byte(rawFull), &row); err != nil {
 		t.Fatalf("unmarshal: %v", err)
 	}
 	entry := AuditLogEntry{Row: row, Raw: []byte(rawFull)}
 	count, tip, err := VerifyChain([]AuditLogEntry{entry}, key)
 	if err != nil {
 		t.Fatalf("verify failed (regression: time-precision drift): %v", err)
 	}
 	if count != 1 {
 		t.Fatalf("expected 1 row verified, got %d", count)
 	}
 	if tip != hmacHex {
 		t.Fatalf("tip mismatch: %s vs %s", tip, hmacHex)
 	}
 }
 // TestSubjectManifest_RoundTripJSON: parse a fixture JSON identical in
 // shape to what crates/catalogd/src/registry.rs::put_subject writes to
 // data/_catalog/subjects/<id>.json. If this fails, the Go reader is
 // out of sync with the Rust writer (a Step 8 contract violation).
 func TestSubjectManifest_RoundTripJSON(t *testing.T) {
 	src := `{
  "schema": "subject_manifest.v1",
  "candidate_id": "WORKER-1",
  "created_at": "2026-05-03T08:22:24.571647177Z",
  "updated_at": "2026-05-03T08:22:24.571647177Z",
  "status": "active",
  "vertical": "unknown",
  "consent": {
    "general_pii": {
      "status": "pending_backfill_review",
      "version": ""
    },
    "biometric": {
      "status": "never_collected"
    }
  },
  "retention": {
    "general_pii_until": "2030-05-02T08:22:24.571647177Z",
    "policy": "4_year_default"
  },
  "datasets": [
    {"name": "workers_500k", "key_column": "worker_id", "key_value": "1"}
  ],
  "safe_views": ["workers_safe"],
  "audit_log_path": "_catalog/subjects/WORKER-1.audit.jsonl",
  "audit_log_chain_root": ""
 }`
 	var m SubjectManifest
 	if err := json.Unmarshal([]byte(src), &m); err != nil {
 		t.Fatalf("unmarshal: %v", err)
 	}
 	if m.CandidateID != "WORKER-1" {
 		t.Fatalf("candidate_id wrong: %s", m.CandidateID)
 	}
 	if m.Status != "active" {
 		t.Fatalf("status wrong: %s", m.Status)
 	}
 	if m.Consent.GeneralPii.Status != "pending_backfill_review" {
 		t.Fatalf("general_pii.status wrong: %s", m.Consent.GeneralPii.Status)
 	}
 	if m.Consent.Biometric.Status != "never_collected" {
 		t.Fatalf("biometric.status wrong: %s", m.Consent.Biometric.Status)
 	}
 	if m.Retention.Policy != "4_year_default" {
 		t.Fatalf("retention.policy wrong: %s", m.Retention.Policy)
 	}
 	if len(m.Datasets) != 1 || m.Datasets[0].Name != "workers_500k" {
 		t.Fatalf("datasets wrong: %+v", m.Datasets)
 	}
 }
@@ -246,6 +246,13 @@ type LogConfig struct {
 // the corpus-downgrade rule: it's already weak, no need to downgrade
 // further. Strong (paid / cloud / frontier) models trigger the gate.
 type ModelsConfig struct {
 	// ⚠ UNWIRED scaffolding — these tier fields parse from TOML and
 	// Resolve() reads them, but as of 2026-05-03 NO OTHER CODE calls
 	// Resolve(). Setting these in lakehouse.toml has no runtime effect
 	// on routing. Customer hot path uses local Ollama via direct model
 	// name per PRD line 70 (everything runs locally). If you wire up
 	// Resolve() consumers, document them here so the next person knows
 	// the field is live, not dead scaffolding.
 	LocalFast      string `toml:"local_fast"`
 	LocalEmbed     string `toml:"local_embed"`
 	LocalJudge     string `toml:"local_judge"`
@@ -260,6 +267,9 @@ type ModelsConfig struct {
 	FrontierStrong string `toml:"frontier_strong"`
 	FrontierFree   string `toml:"frontier_free"`
 	// WeakModels IS live — internal/workflow/modes.go reads it at
 	// startup, internal/matrix/downgrade.go uses the list for the
 	// strong-model auto-downgrade gate. Don't lump with unwired fields.
 	WeakModels []string `toml:"weak_models"`
 }
@@ -1,6 +1,6 @@
 # /v1/embed cross-runtime parity probe
-**Date:** 2026-05-03T03:13:12Z
+**Date:** 2026-05-05T08:22:55Z
 **Rust:** `http://127.0.0.1:3100/ai/embed`  ·  **Go:** `http://127.0.0.1:4110/v1/embed`
 **Model:** `nomic-embed-text` (forced — overrides each side's default)
 **Match metric:** cosine similarity ≥ `0.99999`
@@ -1,6 +1,6 @@
 # extract_json parity probe — Rust vs Go
-**Date:** 2026-05-03T03:13:10Z
+**Date:** 2026-05-05T08:23:07Z
 **Rust helper:** `/home/profit/lakehouse/target/release/parity_extract_json` (links live `gateway::v1::iterate::extract_json`)
 **Go helper:** `./bin/parity_extract_json_go` (links live `internal/validator.ExtractJSON`)
@@ -1,6 +1,6 @@
 # Materializer parity probe — Bun vs Go
-**Date:** 2026-05-03T03:13:11Z
+**Date:** 2026-05-05T08:23:08Z
 **Bun:** `/home/profit/lakehouse/scripts/distillation/build_evidence_index.ts`
 **Go:**  `./bin/materializer`
@@ -1,6 +1,6 @@
 # session_log parity probe — Rust gateway vs Go validatord
-**Date:** 2026-05-03T03:13:10Z
+**Date:** 2026-05-05T08:23:08Z
 **Rust helper:** `/home/profit/lakehouse/target/release/parity_session_log`
 **Go helper:** `./bin/parity_session_log_go`
@@ -0,0 +1,31 @@
 # subject_audit_parity
 **Generated:** 2026-05-05 08:23:08 UTC
 **Spec:** /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8
 **Rust helper:** `/home/profit/lakehouse/target/release/parity_subject_audit`
 **Go helper:** `./bin/subject_audit_helper`
 **Audit dir:** `/home/profit/lakehouse/data/_catalog/subjects`
 ## Phase 1 — Known-answer vector
 Hardcoded fixture row, identical inputs, byte-compare canonical-JSON + HMAC.
 **MATCH** ✓
 ```json
 {"mode":"known_answer","canonical":"{\"accessor\":{\"daemon\":\"gateway\",\"kind\":\"gateway_lookup\",\"purpose\":\"parity_test\",\"trace_id\":\"trace-fixed\"},\"candidate_id\":\"WORKER-FIXED\",\"fields_accessed\":[\"name\"],\"prev_chain_hash\":\"GENESIS\",\"result\":\"success\",\"schema\":\"subject_audit.v1\",\"ts\":\"2026-05-03T12:00:00Z\"}","hmac":"f730fa038c847c27386b92eb1939ec64c62086c0a92617ac0bdf9f650c390b96","canonical_bytes_len":266}
 ```
 ## Phase 2 — Real production audit logs
 Every `*.audit.jsonl` in `/home/profit/lakehouse/data/_catalog/subjects` verified by both runtimes.
 **SKIP** — signing key not readable at `/tmp/lakehouse_audit/subject_audit.key`.
 Set `KEY_PATH=...` or seed /tmp/lakehouse_audit/subject_audit.key (see systemd unit).
 ## Summary
 **1 / 1** parity assertions passed.
 **Status: PARITY** — every Rust assertion matches Go byte-for-byte.
@@ -1,6 +1,6 @@
 # Validator parity probe — Rust :3100 vs Go :4110
-**Date:** 2026-05-03T03:13:05Z
+**Date:** 2026-05-05T08:23:08Z
 **Rust gateway:** `http://127.0.0.1:3100`  ·  **Go gateway:** `http://127.0.0.1:4110`
 Identical `POST /v1/validate` request → both runtimes. Match
@@ -0,0 +1,151 @@
 # Playbook-Lift Reality Test — Run real_006
 **Generated:** 2026-05-05T09:50:08.929241389Z
 **Judge:** `qwen2.5:latest` (Ollama, resolved from env JUDGE_MODEL=qwen2.5:latest)
 **Corpora:** `workers,ethereal_workers`
 **Workers limit:** 5000
 **Queries:** `tests/reality/real_coord_queries_v3.txt` (50 executed)
 **K per pass:** 10
 **Paraphrase pass:** ENABLED
 **Re-judge pass:** ENABLED
 **Evidence:** `reports/reality-tests/playbook_lift_real_006.json`
 ---
 ## Headline
 | Metric | Value |
 |---|---:|
 | Total queries run | 50 |
 | Cold-pass discoveries (judge-best ≠ top-1) | 9 |
 | Warm-pass lifts (recorded playbook → top-1) | 9 |
 | No change (judge-best already top-1, no playbook needed) | 41 |
 | Playbook boosts triggered (warm pass) | 10 |
 | Mean Δ top-1 distance (warm − cold) | -0.05014307 |
 | **Paraphrase pass — recorded answer at rank 0 (top-1)** | **6 / 9** |
 | Paraphrase pass — recorded answer at any rank in top-K | 9 / 9 |
 | **Quality lift** (warm top-1 rating > cold top-1 rating) | **11 / 50** |
 | Quality neutral (warm top-1 rating = cold top-1 rating) | 36 / 50 |
 | Quality regressed (warm top-1 rating < cold top-1 rating) | 3 / 50 |
 **Verbatim lift rate:** 9 of 9 discoveries became top-1 after warm pass.
 ---
 ## Per-query results
 | # | Query | Cold top-1 | Cold judge-best (rank/rating) | Recorded? | Warm top-1 | Judge-best warm rank | Lift |
 |---|---|---|---|---|---|---|---|
 | 1 | Need 1 Loader in Kansas City MO starting at 17:30 for Corner | w-4806 | 0/5 | — | w-4806 | 0 | no |
 | 2 | Need 2 Assemblers in Cincinnati OH starting at 14:30 for Gre | w-4371 | 0/4 | — | w-4371 | 0 | no |
 | 3 | Need 1 Forklift Operator in Lexington KY starting at 08:30 f | e-8263 | 1/5 | ✓ w-4636 | w-4636 | 0 | **YES** |
 | 4 | Need 2 Assemblers in Flint MI starting at 08:30 for Centenni | e-7186 | 1/4 | ✓ e-9319 | e-9319 | 0 | **YES** |
 | 5 | Need 2 Welders in Indianapolis IN starting at 10:00 for Nort | e-5834 | 0/4 | — | e-5834 | 0 | no |
 | 6 | Need 2 Material Handlers in Cincinnati OH starting at 13:00  | e-4871 | 4/2 | — | e-4871 | 4 | no |
 | 7 | Need 3 Pickers in Flint MI starting at 17:00 for Centennial  | e-5571 | 0/2 | — | e-5571 | 0 | no |
 | 8 | Need 3 Packers in Indianapolis IN starting at 09:00 for Heri | w-279 | 0/4 | — | w-279 | 0 | no |
 | 9 | Need 3 Assemblers in Columbus OH starting at 17:30 for River | w-281 | 0/4 | — | w-281 | 0 | no |
 | 10 | Need 5 Machine Operators in Cleveland OH starting at 14:30 f | e-8279 | 0/4 | — | e-8279 | 0 | no |
 | 11 | Need 2 Assemblers in Grand Rapids MI starting at 13:00 for C | w-4502 | 0/3 | — | w-4502 | 0 | no |
 | 12 | Need 2 Pickers in Akron OH starting at 10:30 for Summit Indu | e-5655 | 2/2 | — | e-5655 | 2 | no |
 | 13 | Need 3 Quality Techs in Lexington KY starting at 12:30 for K | e-6369 | 0/4 | — | e-6369 | 0 | no |
 | 14 | Need 4 Assemblers in Gary IN starting at 12:00 for Heritage  | e-1315 | 0/2 | — | e-1315 | 0 | no |
 | 15 | Need 3 Packers in Toledo OH starting at 16:00 for Cornerston | e-4887 | 1/2 | — | e-4887 | 1 | no |
 | 16 | Need 4 Warehouse Associates in Fort Wayne IN starting at 13: | w-4434 | 0/4 | — | w-4434 | 0 | no |
 | 17 | Need 4 Assemblers in Columbus OH starting at 13:00 for Midwa | w-281 | 0/4 | — | w-281 | 0 | no |
 | 18 | Need 2 Shipping Clerks in Chicago IL starting at 17:00 for M | w-4504 | 1/4 | ✓ w-1522 | w-1522 | 0 | **YES** |
 | 19 | Need 2 Machine Operators in Chicago IL starting at 11:00 for | e-1251 | 0/4 | — | e-1251 | 0 | no |
 | 20 | Need 3 CNC Operators in Grand Rapids MI starting at 10:00 fo | w-792 | 0/3 | — | w-792 | 0 | no |
 | 21 | Need 1 Warehouse Associate in Lexington KY starting at 09:30 | e-2331 | 0/4 | — | e-2331 | 0 | no |
 | 22 | Need 2 Material Handlers in Gary IN starting at 11:00 for He | e-18 | 0/2 | — | e-18 | 0 | no |
 | 23 | Need 5 Assemblers in Fort Wayne IN starting at 16:30 for Mid | e-6271 | 7/3 | — | e-6271 | 7 | no |
 | 24 | Need 1 Loader in Cincinnati OH starting at 08:00 for Summit  | e-8843 | 1/5 | ✓ w-4473 | w-4473 | 0 | **YES** |
 | 25 | Need 3 Quality Techs in Columbus OH starting at 12:00 for Ri | w-281 | 0/2 | — | w-281 | 0 | no |
 | 26 | Need 1 Machine Operator in Columbus OH starting at 09:30 for | w-4815 | 0/4 | — | w-4815 | 0 | no |
 | 27 | Need 3 Machine Operators in Madison WI starting at 12:00 for | w-2027 | 0/4 | — | w-2027 | 0 | no |
 | 28 | Need 2 Material Handlers in Kansas City MO starting at 11:30 | e-6774 | 0/3 | — | e-6774 | 0 | no |
 | 29 | Need 3 Loaders in Flint MI starting at 16:00 for Parallel Ma | w-4875 | 0/2 | — | w-4875 | 0 | no |
 | 30 | Need 2 Welders in Louisville KY starting at 13:00 for Horizo | w-2267 | 0/4 | — | w-2267 | 0 | no |
 | 31 | Need 1 CNC Operator in Flint MI starting at 10:30 for Horizo | e-317 | 7/3 | — | e-317 | 7 | no |
 | 32 | Need 1 Material Handler in Columbus OH starting at 15:30 for | e-8676 | 1/4 | ✓ w-2589 | w-2589 | 0 | **YES** |
 | 33 | Need 2 Forklift Operators in Louisville KY starting at 14:30 | w-1830 | 0/4 | — | w-1830 | 0 | no |
 | 34 | Need 2 Warehouse Associates in Chicago IL starting at 10:00  | w-4743 | 7/4 | ✓ e-9171 | e-9171 | 0 | **YES** |
 | 35 | Need 2 Material Handlers in Gary IN starting at 15:00 for Pa | w-4236 | 1/2 | — | w-4236 | 1 | no |
 | 36 | Need 1 Forklift Operator in Grand Rapids MI starting at 10:0 | w-3227 | 3/2 | — | w-3227 | 3 | no |
 | 37 | Need 2 Pickers in Louisville KY starting at 12:30 for Corner | e-6489 | 2/4 | ✓ e-7622 | e-7622 | 0 | **YES** |
 | 38 | Need 2 Loaders in Indianapolis IN starting at 17:30 for Midw | e-9877 | 0/4 | — | e-9877 | 0 | no |
 | 39 | Need 1 Shipping Clerk in Indianapolis IN starting at 12:30 f | w-4635 | 0/5 | — | w-4635 | 0 | no |
 | 40 | Need 2 Assemblers in Cincinnati OH starting at 08:00 for Key | w-4945 | 0/4 | — | w-4945 | 0 | no |
 | 41 | Need 5 Quality Techs in Kansas City MO starting at 11:30 for | e-5633 | 0/4 | — | e-5633 | 0 | no |
 | 42 | Need 2 Machine Operators in Gary IN starting at 10:00 for He | e-1089 | 0/2 | — | e-1089 | 0 | no |
 | 43 | Need 1 Packer in Chicago IL starting at 09:30 for Midway Dis | e-7746 | 0/5 | — | w-279 | 1 | no |
 | 44 | Need 2 Pickers in Lexington KY starting at 17:30 for Vanguar | e-3375 | 0/4 | — | e-3375 | 0 | no |
 | 45 | Need 2 Maintenance Techs in Grand Rapids MI starting at 17:0 | e-6083 | 0/2 | — | e-6083 | 0 | no |
 | 46 | Need 1 Material Handler in Detroit MI starting at 10:30 for  | w-3286 | 0/5 | — | w-3286 | 0 | no |
 | 47 | Need 1 Welder in Akron OH starting at 15:00 for Summit Indus | e-6149 | 0/2 | — | e-6149 | 0 | no |
 | 48 | Need 1 Shipping Clerk in Cincinnati OH starting at 13:30 for | e-4218 | 3/5 | ✓ w-3488 | w-3488 | 0 | **YES** |
 | 49 | Need 5 Packers in Indianapolis IN starting at 10:30 for Midw | e-2746 | 2/4 | ✓ w-279 | w-279 | 0 | **YES** |
 | 50 | Need 1 Forklift Operator in Louisville KY starting at 10:30  | w-1830 | 0/4 | — | w-1830 | 0 | no |
 ---
 ## Paraphrase pass — does the playbook help similar-but-different queries?
 For each query whose Pass 1 cold pass recorded a playbook entry, the
 judge model rephrased the query, and the rephrased version was sent
 through warm matrix.search. The recorded answer ID's rank in those
 results tests whether cosine on the embedded paraphrase finds the
 recorded query's vector.
 | # | Original (≤40c) | Paraphrase (≤60c) | Recorded answer | Paraphrase top-1 | Recorded rank | Paraphrase lift |
 |---|---|---|---|---|---|---|
 | 3 | Need 1 Forklift Operator in Lexington KY | Vanguard Components requires a Forklift Operator in Lexingto | w-4636 | w-4636 | 0 | **YES** |
 | 4 | Need 2 Assemblers in Flint MI starting a | Centennial Packaging requires 2 Assemblers to start at 08:30 | e-9319 | e-9319 | 0 | **YES** |
 | 18 | Need 2 Shipping Clerks in Chicago IL sta | Looking for 2 Shipping Clerks in Chicago, IL to start at 5:0 | w-1522 | w-4504 | 1 | no |
 | 24 | Need 1 Loader in Cincinnati OH starting  | Summit Industrial requires 1 Loader position from 08:00 onwa | w-4473 | w-4473 | 0 | **YES** |
 | 32 | Need 1 Material Handler in Columbus OH s | Looking for a Material Handler in Columbus, OH who can start | w-2589 | w-2589 | 0 | **YES** |
 | 34 | Need 2 Warehouse Associates in Chicago I | Looking for 2 Warehouse Associates to work from 10:00 onward | e-9171 | e-9171 | 0 | **YES** |
 | 37 | Need 2 Pickers in Louisville KY starting | Looking for 2 Pickers in Louisville, KY to start at 12:30 fo | e-7622 | e-6489 | 2 | no |
 | 48 | Need 1 Shipping Clerk in Cincinnati OH s | Summit Industrial requires a Shipping Clerk in Cincinnati, O | w-3488 | w-3488 | 0 | **YES** |
 | 49 | Need 5 Packers in Indianapolis IN starti | Looking for 5 packers in Indianapolis, IN to start at 10:30  | w-279 | e-2746 | 1 | no |
 ---
 ## Honesty caveats
 1. **Judge IS the ground truth proxy.** Without human-labeled relevance, the LLM
   judge's verdict is what defines "best." If `qwen2.5:latest` rates badly,
   the lift number is meaningless. To validate the judge itself, sample 5–10
   verdicts manually and check agreement.
 2. **Score-1.0 boost = distance halved.** Playbook math is
   `distance' = distance × (1 - 0.5 × score)`. Lift requires the judge-best
   result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise
   even halving doesn't promote it. Tight clusters → little visible lift.
 3. **Verbatim vs paraphrase.** The verbatim lift rate (above) is the cheap
   case — same query, recorded playbook, expected boost. The paraphrase
   pass (when enabled) is the actual learning property: similar-but-different
   queries hitting a recorded playbook. Compare verbatim and paraphrase
   lift rates — paraphrase should be lower (semantic-distance gates some
   playbook hits) but non-zero is the meaningful signal.
 4. **Multi-corpus skew.** Default corpora=`workers,ethereal_workers` — if all judge-best
   results land in one corpus, the matrix layer's purpose isn't being tested.
   Check per-corpus distribution in the JSON.
 5. **Judge resolution.** This run used `qwen2.5:latest` from
   env JUDGE_MODEL=qwen2.5:latest.
   Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
 6. **Paraphrase generation also uses the judge.** The same model that rates
   relevance also rephrases queries. A judge that's bad at rating staffing
   queries is probably also bad at rephrasing them. Worth sanity-checking
   a sample of `paraphrase_query` values in the JSON before trusting the
   paraphrase lift number.
 ## Next moves
 - If lift rate ≥ 50% of discoveries: matrix layer + playbook is doing real
  work. Move to paraphrase queries + tag-based boost (currently ignored).
 - If lift rate < 20%: investigate why — judge variance, distance gap too
  wide, or playbook math too gentle. The score=1.0 / 0.5× formula may need
  retuning.
 - If discovery rate (cold judge-best ≠ top-1) is itself low: cosine is
  already close to optimal on this query distribution. Either the corpus
  is too narrow or the queries are too easy.
@@ -0,0 +1,234 @@
 # Reality test real_006 — distribution-shift findings
 **Run:** 2026-05-05 04:41:46 → 04:50:08 CDT (8m22s driver wall, ~14 min including ingest)
 **Judge:** `qwen2.5:latest` (Ollama, local) — anchor's recommended judge, ~9s/query
 **Queries:** 50 from `tests/reality/real_coord_queries_v3.txt` (rows 10-59 of fill_events.parquet, single `need` style)
 **Corpora:** `workers,ethereal_workers` (5K + 10K)
 **Local-only:** zero cloud calls per PRD line 70.
 Companion to `playbook_lift_real_006.{json,md}`. That's the harness output; this is the reading.
 ---
 ## Why this test exists
 real_001-005 all sourced their queries from the **first 10 rows** of
 `fill_events.parquet`. `gen_real_queries.go` had `-limit N` but no
 `-offset N`, so every "real" reality test ran on the same memorized
 slice. The published "8 / 10 cold-pass top-1 = judge-best" was a
 property of those 10 rows, not measured generalization. real_006
 closes the methodology gap: new `-offset` flag samples rows 10-59 (5×
 the count, never seen by the substrate).
 ---
 ## Headline — substrate generalizes (mostly)
 | Metric | real_001 (10 queries, rows 0-9) | real_006 (50 queries, rows 10-59) | Verdict |
 |---|---:|---:|---|
 | Cold-pass top-1 = judge-best (rank match) | 8 / 10 (80%) | **41 / 50 (82%)** | **HOLDS** |
 | Cold-pass top-1 = judge-best AND rating ≥ 2 | 8 / 10 (80%) | 34 / 50 (68%) | -12 pts |
 | Mean cold top-1 judge rating | ~3.3 | 3.08 | -7% |
 | Discoveries (judge promoted non-top-1) | 2 / 10 | 9 / 50 (18%) | comparable |
 | Verbatim lift (discovery → warm top-1) | 2 / 2 (100%) | 9 / 9 (100%) | **HOLDS** |
 | Paraphrase recovery → top-1 | n/a (disabled) | 6 / 9 (67%) | new |
 | Quality regressed on rejudge | 0 (test absent) | 3 / 50 (6%) | new |
 **Reading:** the substrate's *rank* behavior generalizes cleanly — the
 top-1 worker is judge-approved at the same rate on fresh data as on
 memorized data. The *quality* of top-1 (rating ≥ 2) drops 12 points,
 which means 7 of the 41 "no-discovery" queries had cold top-1 the
 judge rated 1 (irrelevant) but the corpus had nothing better. Honest
 signal: parts of the v3 slice are in territory the workers corpus
 doesn't cover well.
 The verbatim-lift property (discovery → warm top-1) is **clean at
 9/9**, matching real_001's 2/2 perfectly. When the playbook records,
 the recorded answer comes back next time. That's the load-bearing
 learning property.
 ---
 ## Cluster analysis — the cross-pollination question
 real_001 found that same-(client, city) clusters cause Shape A boost
 to bleed across roles. Real_002's role-gate fix (`roleEqual`) was
 supposed to close that. real_006 has *more* cluster opportunities than
 real_001 did:
 | Cluster | Count | Result |
 |---|---:|---|
 | Riverfront Steel + Columbus OH | 4 | mostly clean — see below |
 | Heritage Foods + Gary IN | 3 | **clean** — distinct workers per role, no boost firing |
 | Cornerstone Fabrication + Louisville KY | 3 | clean |
 | Midway Distribution + Chicago IL | 3 | **bleed: Q43 regressed** |
 ### Heritage Foods + Gary IN (3 queries, all clean)
 ```
 Q14 Assemblers       → e-1315
 Q22 Material Handler → e-18
 Q42 Machine Operator → e-1089
 ```
 Three different roles → three different workers. Zero boosts fired,
 zero playbooks recorded. **Role-disambiguation works at the cosine
 level for this cluster.** Comparable to real_002's role-gate
 demonstration.
 ### Riverfront Steel + Columbus OH (4 queries, partial)
 ```
 Q9  Assemblers       → w-281    (cold = warm, no boost)
 Q25 Quality Techs    → w-281    (cold = warm, no boost) ← same worker as Q9!
 Q26 Machine Operator → w-4815   (clean)
 Q32 Material Handler → e-8676 → w-2589  (judge promoted, playbook recorded)
 ```
 Q9 and Q25 both surface `w-281` cold-pass for *different roles* —
 that's a **cosine-level confusion** in the workers corpus, not a
 playbook bleed. The substrate isn't breaking; the corpus contains a
 worker whose resume embeds close to both "Assemblers" and "Quality
 Techs" in this client+city. Judge-rating Q25 dropped 2 → 1 on
 rejudge, which is the LLM's own consistency drift, not a substrate
 fault. Worth noting but not a bug.
 ### Midway Distribution + Chicago IL (3 queries) — the regression
 ```
 Q18 Shipping Clerks   → cold w-4504 → warm w-1522  (boost=1, playbook recorded)
 Q19 Machine Operators → cold = warm e-1251         (clean)
 Q43 Packer            → cold e-7746 (rating 5) → warm w-279 (rating 2)  ← regressed
 ```
 **Diagnosis (2026-05-05 follow-up): the leak source isn't Q18 — it's Q49.**
 Three queries in real_006 touch `w-279`:
 | # | role-extract | client | city | result | playbook? |
 |---|---|---|---|---|---|
 | Q8 Packers Indianapolis IN Heritage Foods | Packers | Heritage Foods | Indianapolis | w-279 (cold = judge-best) | no |
 | **Q49** Packers **Indianapolis IN** Midway Distribution | Packers | **Midway Distribution** | Indianapolis | cold e-2746 → warm w-279 (judge-best) | **yes — recorded** |
 | Q43 Packer **Chicago IL** Midway Distribution | Packer | **Midway Distribution** | Chicago | cold e-7746 (rating 5) → warm w-279 (rating 2) | no |
 Q49 recorded `w-279` with role=`Packers`, client=`Midway Distribution`,
 city=`Indianapolis`. When Q43 ran with role=`Packer`,
 client=`Midway Distribution`, city=`Chicago`:
 - `roleEqual("Packer", "Packers")` → both normalize to `"packer"` →
  **gate passes (correctly, by design)**
 - Q49's recorded query embedding is close enough to Q43's that the
  playbook hit's distance falls inside `DefaultPlaybookMaxInjectDistance = 0.20`
  (role + client + count + time-token dominate cosine; only the city
  and the singular/plural noun differ)
 - Inject fires; `w-279` (an Indianapolis worker) surfaces at Q43's
  warm top-1 in **Chicago**
 - Judge correctly rates this 2/5 — wrong city
 **The role gate IS working as designed.** What's missing is a
 **city gate** (or more generally, a metadata-equality gate on the
 demand attributes that don't appear in the role field). Real_002's
 fix anticipated cross-role bleed (Forklift → CNC); it didn't
 anticipate cross-city bleed within the same client+role.
 **Why prior tests missed this:** real_001-005 sourced from rows 0-9
 of fill_events.parquet. Among those 10 rows there was no
 Midway-Distribution × Packer × (different cities) pair. real_006
 includes rows 10-59 which contain Q43 (Chicago) and Q49 (Indianapolis)
 on the same client+role — a structurally new combination the
 substrate hadn't been tested against.
 The methodology gap closing on itself: the offset-flag fix that
 surfaced real_006's headline number (-12 pts strict) also surfaced
 a real cross-city leak the gate doesn't catch.
 **The other regressions** (Q4 Centennial Packaging Flint, Q25
 Riverfront Steel Quality Techs) are smaller (3→2 and 2→1) and look
 like judge consistency drift on borderline candidates. Q43 is the
 structural one.
 ### Concrete fix surface (next session)
 A city gate alongside the role gate would close this:
 1. **`PlaybookEntry`** gains `City string` (or generalize to
   `Metadata map[string]string`). Recorded at `playbookRecord` time
   from the same query the role extractor parses.
 2. **`InjectPlaybookMisses` + `ApplyPlaybookBoost`** add a
   `cityEqual(queryCity, hit.Entry.City)` check after the role
   check. Same "permissive on empty" semantics as `roleEqual`.
 3. **Bin extractor** adds a city-extract regex (e.g.
   `\s+in\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\s+[A-Z]{2}`) to capture
   the city + state token from the standard query shape.
 4. **Unit tests** mirror the existing role-gate tests, locking the
   exact Q43/Q49 scenario as a regression gate (and add an integration-
   level test that record(role=Packers, city=Indianapolis) followed by
   search(role=Packer, city=Chicago) doesn't surface the recorded
   answer).
 Estimated scope: 1 new field, 2 new gate checks, 1 new regex, ~5
 tests. Same shape as real_002's role-gate fix. Half a session.
 Open question: same-metro normalization ("Detroit MI" ≈ "Dearborn MI"?)
 would help with real-world dispatch where coordinators legitimately
 route across nearby cities. Punt that to future work — strict equality
 closes the structural bleed without over-engineering.
 ---
 ## What this confirms vs falsifies
 **Confirmed:**
 - Substrate generalizes at the rank level (82% cold-top-1 = judge-best)
 - Verbatim lift works (9/9 discoveries → warm top-1)
 - Role-disambiguation works at cosine level for clean role-distinct
  query distributions (Heritage Foods cluster is the proof)
 - Paraphrase recovery is real (6/9 → top-1, 9/9 any-rank)
 **Falsified / weakened:**
 - "8/10 cold-pass top-1 = judge-best" was 12 points optimistic on
  the strict (rating ≥ 2) interpretation. Real number on broader
  data is ~68%, not 80%. Headline rank-match number (82%) holds.
 - Real_002's role-gate fix is **not structurally airtight**. Q43
  shows the cluster-bleed pattern can still fire under conditions
  the prior tests didn't reach. Open question: which path is
  leaking — extractor failure, gate scope, or cosine drift?
 ---
 ## Next moves (informed by this evidence)
 1. **Diagnose Q43 specifically**: re-run the role extractor on its
   query text, check whether Q18's playbook entry has a role field
   recorded, look at the warm-pass top-K to see whether `w-279`
   reaches there via boost, inject, or cosine-only.
 2. **Strengthen the corpus for the role-city combos that scored
   low rating** (the 7 queries where cold top-1 was rating=1). The
   workers corpus has gaps the v3 slice surfaced.
 3. **Don't ship the "80% generalizes" framing as-is.** The number
   real_006 measured (82% rank, 68% rating ≥ 2) is the honest one
   to publish.
 This is what reality tests are for. Numbers from the memorized slice
 gave a clean story; numbers from the held-out slice show where it
 needs work.
 ---
 ## Repro
 ```bash
 cd /home/profit/golangLAKEHOUSE
 PATH=/usr/local/go/bin:$PATH go build -o bin/gen_real_queries ./scripts/cutover/gen_real_queries.go
 ./bin/gen_real_queries -limit 50 -offset 10 -styles need > tests/reality/real_coord_queries_v3.txt
 PATH=/usr/local/go/bin:$PATH \
  RUN_ID=real_006 \
  JUDGE_MODEL=qwen2.5:latest \
  QUERIES_FILE=tests/reality/real_coord_queries_v3.txt \
  WITH_PARAPHRASE=1 \
  WITH_REJUDGE=1 \
  bash scripts/playbook_lift.sh
 ```
 Local-only. No cloud calls.
@@ -29,6 +29,7 @@ import (
 func main() {
 	src := flag.String("src", "/home/profit/lakehouse/data/datasets/fill_events.parquet", "fill_events parquet path")
 	limit := flag.Int("limit", 10, "number of source rows to read")
 	offset := flag.Int("offset", 0, "skip the first N rows (lets reality tests sample beyond the memorized real_001 slice)")
 	styles := flag.String("styles", "need", "comma-separated styles to emit per row (need|client_first|looking|shorthand|all)")
 	flag.Parse()
@@ -58,16 +59,24 @@ func main() {
 	at := tbl.Column(10).Data().Chunk(0)
 	deadline := tbl.Column(12).Data().Chunk(0)
-	n := int(tbl.NumRows())
+	totalRows := int(tbl.NumRows())
-	if *limit < n {
+	start := *offset
-		n = *limit
+	if start < 0 {
 		start = 0
 	}
 	if start > totalRows {
 		start = totalRows
 	}
 	end := start + *limit
 	if end > totalRows {
 		end = totalRows
 	}
 	stylesList := parseStyles(*styles)
 	fmt.Println("# Real-shape coordinator queries — generated from fill_events.parquet")
 	fmt.Println("# (real-shape demand data; queries built mechanically from event rows).")
-	fmt.Printf("# Source: %s (%d rows total, %d emitted, styles=%v)\n", *src, tbl.NumRows(), n, stylesList)
+	fmt.Printf("# Source: %s (%d rows total, rows [%d,%d) emitted, styles=%v)\n", *src, totalRows, start, end, stylesList)
 	fmt.Println("#")
 	fmt.Println("# Styles:")
 	fmt.Println("#   need:         'Need N {role}{s} in {city} {state} starting at {at} for {client}'")
@@ -80,7 +89,7 @@ func main() {
 	fmt.Println("# substrate's bleed behavior when the role gate is silently disabled.")
 	fmt.Println()
-	for i := 0; i < n; i++ {
+	for i := start; i < end; i++ {
 		ev := event{
 			client: client.ValueStr(i),
 			city:   city.ValueStr(i),
@@ -0,0 +1,168 @@
 // Cross-runtime parity helper — Go side.
 //
 // Specification: /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.
 //
 // Counterpart of crates/catalogd/src/bin/parity_subject_audit.rs.
 // Both helpers MUST produce byte-identical output for the same input.
 //
 // Modes:
 //
 //   --known-answer
 //       Print canonical-JSON + HMAC for a hardcoded fixture. Compared
 //       byte-for-byte against the Rust helper's output. If they
 //       differ, the canonical-JSON or HMAC algorithm has drifted.
 //
 //   --verify <audit_log_path> --key <key_path>
 //       Replay the HMAC chain on a real audit JSONL. Print one JSON
 //       object: {mode, count, tip, verified, error}.
 package main
 import (
 	"encoding/json"
 	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
 	"time"
 	cat "git.agentview.dev/profit/golangLAKEHOUSE/internal/catalogd"
 )
 const genesis = "GENESIS"
 func deterministicKey() []byte {
 	k := make([]byte, 32)
 	for i := range k {
 		k[i] = byte(i)
 	}
 	return k
 }
 // knownAnswerOut is intentionally identical to KnownAnswerOut in the
 // Rust helper so a stdout diff is a one-line semantic comparison.
 type knownAnswerOut struct {
 	Mode              string `json:"mode"`
 	Canonical         string `json:"canonical"`
 	Hmac              string `json:"hmac"`
 	CanonicalBytesLen int    `json:"canonical_bytes_len"`
 }
 type verifyOut struct {
 	Mode     string `json:"mode"`
 	Count    int    `json:"count"`
 	Tip      string `json:"tip"`
 	Verified bool   `json:"verified"`
 	Error    *string `json:"error"`
 }
 func runKnownAnswer() {
 	row := cat.SubjectAuditRow{
 		Schema:      "subject_audit.v1",
 		Ts:          time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC),
 		CandidateID: "WORKER-FIXED",
 		Accessor: cat.AuditAccessor{
 			Kind:    "gateway_lookup",
 			Daemon:  "gateway",
 			Purpose: "parity_test",
 			TraceID: "trace-fixed",
 		},
 		FieldsAccessed: []string{"name"},
 		Result:         "success",
 		PrevChainHash:  genesis,
 	}
 	canonical, hmacHex, err := cat.CanonicalAndHmac(&row, deterministicKey(), genesis)
 	if err != nil {
 		die("canonical/hmac: %v", err)
 	}
 	out := knownAnswerOut{
 		Mode:              "known_answer",
 		Canonical:         string(canonical),
 		Hmac:              hmacHex,
 		CanonicalBytesLen: len(canonical),
 	}
 	emit(out)
 }
 func runVerify(auditPath, keyPath string) {
 	entries, err := cat.ReadAuditLog(auditPath)
 	if err != nil {
 		die("read audit log: %v", err)
 	}
 	key, err := os.ReadFile(keyPath)
 	if err != nil {
 		die("read key: %v", err)
 	}
 	count, tip, verr := cat.VerifyChain(entries, key)
 	out := verifyOut{
 		Mode:     "verify",
 		Count:    count,
 		Tip:      tip,
 		Verified: verr == nil,
 	}
 	if verr != nil {
 		s := verr.Error()
 		out.Error = &s
 		// Reset count + tip to match the Rust helper's error semantics.
 		out.Count = 0
 		out.Tip = genesis
 	}
 	emit(out)
 }
 func emit(v any) {
 	bs, err := json.Marshal(v)
 	if err != nil {
 		die("marshal output: %v", err)
 	}
 	fmt.Println(string(bs))
 }
 func die(format string, a ...any) {
 	fmt.Fprintf(os.Stderr, format+"\n", a...)
 	os.Exit(2)
 }
 func main() {
 	args := os.Args[1:]
 	var (
 		knownAnswer bool
 		auditPath   string
 		keyPath     string
 	)
 	for i := 0; i < len(args); i++ {
 		switch args[i] {
 		case "--known-answer":
 			knownAnswer = true
 		case "--verify":
 			if i+1 >= len(args) {
 				die("--verify needs a path")
 			}
 			auditPath = args[i+1]
 			i++
 		case "--key":
 			if i+1 >= len(args) {
 				die("--key needs a path")
 			}
 			keyPath = args[i+1]
 			i++
 		case "-h", "--help":
 			fmt.Fprintln(os.Stderr, "subject_audit_helper --known-answer")
 			fmt.Fprintln(os.Stderr, "subject_audit_helper --verify <audit_log> --key <key_file>")
 			os.Exit(0)
 		default:
 			die("unknown arg: %s", args[i])
 		}
 	}
 	if knownAnswer {
 		runKnownAnswer()
 		return
 	}
 	if auditPath == "" || keyPath == "" {
 		die("need --known-answer OR (--verify <path> --key <path>)")
 	}
 	// Sanity: file naming convention <candidate_id>.audit.jsonl.
 	if !strings.HasSuffix(filepath.Base(auditPath), ".audit.jsonl") {
 		die("audit log path must end with .audit.jsonl")
 	}
 	runVerify(auditPath, keyPath)
 }
@@ -0,0 +1,203 @@
 #!/usr/bin/env bash
 # subject_audit_parity — verify Rust and Go produce byte-identical
 # canonical JSON + HMAC-SHA256 chain hashes for subject audit logs.
 #
 # Why: the SubjectManifest + audit-log substrate (Rust crates/catalogd
 # subject_audit.rs) is consumed by the legal-tier endpoint /audit/subject/{id}
 # AND by the Go internal/catalogd reader. A canonical-JSON drift between
 # the two runtimes would mean a chain written by Rust does not verify
 # under Go (or worse, verifies as tampered) — silently breaking the
 # defensible-audit guarantee.
 #
 # The probe runs in two phases:
 #
 #   Phase 1 — Known-answer vector
 #     Both helpers serialize the same hardcoded SubjectAuditRow,
 #     emit canonical bytes + HMAC. Outputs MUST be byte-identical.
 #     This catches algorithm drift independent of any real data.
 #
 #   Phase 2 — Real production audit logs
 #     Walk every *.audit.jsonl in the live data dir. Run BOTH helpers
 #     against each one with the production signing key. Outputs MUST
 #     be byte-identical. This catches real-world drift (e.g. struct
 #     field tag mismatches that only fire on certain field values).
 #
 # Outputs: reports/cutover/gauntlet_2026-05-02/parity/subject_audit_parity.md
 #
 # Env overrides:
 #   RUST_REPO=/home/profit/lakehouse
 #   RUST_BIN=$RUST_REPO/target/release/parity_subject_audit
 #   GO_BIN=./bin/subject_audit_helper
 #   AUDIT_DIR=$RUST_REPO/data/_catalog/subjects
 #   KEY_PATH=/tmp/lakehouse_audit/subject_audit.key
 set -uo pipefail
 cd "$(dirname "$0")/../../.."
 RUST_REPO="${RUST_REPO:-/home/profit/lakehouse}"
 RUST_BIN="${RUST_BIN:-$RUST_REPO/target/release/parity_subject_audit}"
 GO_BIN="${GO_BIN:-./bin/subject_audit_helper}"
 AUDIT_DIR="${AUDIT_DIR:-$RUST_REPO/data/_catalog/subjects}"
 KEY_PATH="${KEY_PATH:-/tmp/lakehouse_audit/subject_audit.key}"
 OUT_DIR="reports/cutover/gauntlet_2026-05-02/parity"
 mkdir -p "$OUT_DIR"
 OUT="$OUT_DIR/subject_audit_parity.md"
 export PATH="$PATH:/usr/local/go/bin"
 # ── Build / verify both sides ───────────────────────────────────────
 if [ ! -x "$RUST_BIN" ]; then
  echo "[subject-audit-parity] building Rust helper..."
  (cd "$RUST_REPO" && cargo build -p catalogd --bin parity_subject_audit --release 2>&1 | tail -3)
 fi
 if [ ! -x "$RUST_BIN" ]; then
  echo "[subject-audit-parity] SKIP: $RUST_BIN missing"
  exit 0
 fi
 if [ ! -x "$GO_BIN" ]; then
  echo "[subject-audit-parity] building Go helper..."
  go build -o "$GO_BIN" ./scripts/cutover/parity/subject_audit_helper/
 fi
 if [ ! -x "$GO_BIN" ]; then
  echo "[subject-audit-parity] FAIL: $GO_BIN missing after build"
  exit 1
 fi
 # ── Report header ───────────────────────────────────────────────────
 {
  echo "# subject_audit_parity"
  echo
  echo "**Generated:** $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
  echo "**Spec:** /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8"
  echo "**Rust helper:** \`$RUST_BIN\`"
  echo "**Go helper:** \`$GO_BIN\`"
  echo "**Audit dir:** \`$AUDIT_DIR\`"
  echo
 } > "$OUT"
 PASS=0
 FAIL=0
 # ── Phase 1 — Known-answer vector ───────────────────────────────────
 {
  echo "## Phase 1 — Known-answer vector"
  echo
  echo "Hardcoded fixture row, identical inputs, byte-compare canonical-JSON + HMAC."
  echo
 } >> "$OUT"
 RUST_KA="$(mktemp)"; GO_KA="$(mktemp)"
 "$RUST_BIN" --known-answer > "$RUST_KA" 2>&1 || true
 "$GO_BIN" --known-answer > "$GO_KA" 2>&1 || true
 if diff -q "$RUST_KA" "$GO_KA" >/dev/null 2>&1; then
  PASS=$((PASS+1))
  {
    echo "**MATCH** ✓"
    echo
    echo '```json'
    cat "$RUST_KA"
    echo '```'
    echo
  } >> "$OUT"
 else
  FAIL=$((FAIL+1))
  {
    echo "**MISMATCH** ✗"
    echo
    echo "### Rust"
    echo '```json'
    cat "$RUST_KA"
    echo '```'
    echo
    echo "### Go"
    echo '```json'
    cat "$GO_KA"
    echo '```'
    echo
    echo "### Diff"
    echo '```diff'
    diff "$RUST_KA" "$GO_KA" || true
    echo '```'
    echo
  } >> "$OUT"
 fi
 # ── Phase 2 — Real production audit logs ────────────────────────────
 {
  echo "## Phase 2 — Real production audit logs"
  echo
  echo "Every \`*.audit.jsonl\` in \`$AUDIT_DIR\` verified by both runtimes."
  echo
 } >> "$OUT"
 if [ ! -r "$KEY_PATH" ]; then
  {
    echo "**SKIP** — signing key not readable at \`$KEY_PATH\`."
    echo "Set \`KEY_PATH=...\` or seed /tmp/lakehouse_audit/subject_audit.key (see systemd unit)."
    echo
  } >> "$OUT"
 else
  shopt -s nullglob
  LOGS=( "$AUDIT_DIR"/*.audit.jsonl )
  if [ "${#LOGS[@]}" -eq 0 ]; then
    {
      echo "**SKIP** — no \`*.audit.jsonl\` files under \`$AUDIT_DIR\`."
      echo "(Trigger one by hitting the gateway with /v1/validate on a candidate_id.)"
      echo
    } >> "$OUT"
  else
    {
      echo "| Audit log | Rust verified | Go verified | Result |"
      echo "|---|---|---|---|"
    } >> "$OUT"
    for log in "${LOGS[@]}"; do
      label="$(basename "$log")"
      RUST_OUT="$(mktemp)"; GO_OUT="$(mktemp)"
      "$RUST_BIN" --verify "$log" --key "$KEY_PATH" > "$RUST_OUT" 2>&1 || true
      "$GO_BIN" --verify "$log" --key "$KEY_PATH" > "$GO_OUT" 2>&1 || true
      rust_count=$(jq -r '.count // 0' < "$RUST_OUT" 2>/dev/null || echo "?")
      go_count=$(jq -r '.count // 0' < "$GO_OUT" 2>/dev/null || echo "?")
      rust_ok=$(jq -r '.verified // false' < "$RUST_OUT" 2>/dev/null || echo "?")
      go_ok=$(jq -r '.verified // false' < "$GO_OUT" 2>/dev/null || echo "?")
      if diff -q "$RUST_OUT" "$GO_OUT" >/dev/null 2>&1; then
        PASS=$((PASS+1))
        echo "| \`$label\` | $rust_count rows ($rust_ok) | $go_count rows ($go_ok) | **MATCH** ✓ |" >> "$OUT"
      else
        FAIL=$((FAIL+1))
        {
          echo "| \`$label\` | $rust_count rows ($rust_ok) | $go_count rows ($go_ok) | **MISMATCH** ✗ |"
          echo
          echo "### Diff for \`$label\`"
          echo '```diff'
          diff "$RUST_OUT" "$GO_OUT" || true
          echo '```'
          echo
        } >> "$OUT"
      fi
      rm -f "$RUST_OUT" "$GO_OUT"
    done
  fi
 fi
 rm -f "$RUST_KA" "$GO_KA"
 # ── Summary ─────────────────────────────────────────────────────────
 TOTAL=$((PASS + FAIL))
 {
  echo
  echo "## Summary"
  echo
  echo "**$PASS / $TOTAL** parity assertions passed."
  echo
  if [ "$FAIL" -gt 0 ]; then
    echo "**Status: DIVERGED** — Rust and Go disagree on at least one canonical-JSON or HMAC computation."
    echo "Investigate the diff above before declaring cross-runtime parity."
  else
    echo "**Status: PARITY** — every Rust assertion matches Go byte-for-byte."
  fi
 } >> "$OUT"
 echo "[subject-audit-parity] $PASS / $TOTAL pass — report: $OUT"
 [ "$FAIL" -eq 0 ]
@@ -135,32 +135,47 @@ start observerd 4219
 start matrixd   4218
 start gateway   4110
-# chatd is started independently — its provider key files come from
+# chatd + validatord are started independently — both sit OUTSIDE
-# /etc/lakehouse/{ollama_cloud,openrouter,opencode,kimi}.env; if
+# the persistent :4xxx range and are SHARED with smoke harnesses
-# chatd is already up (long-running from a prior session) we don't
+# (no bucket / port conflict because chatd has no S3 state and
-# touch it. chatd uses no S3, so no temp-toml override needed.
+# validatord's state is JSONL files, not MinIO). Provider key files
-if ! curl -sSf -m 1 http://127.0.0.1:3220/health >/dev/null 2>&1; then
+# come from /etc/lakehouse/{ollama_cloud,openrouter,opencode,kimi}.env.
-  echo "[gostack] chatd :3220 not up; starting"
+# If either is already up (long-running from a prior session) we
-  nohup ./bin/chatd -config lakehouse.toml > /tmp/gostack-logs/chatd.log 2>&1 & disown
+# don't touch it. The matching smokes (chatd_smoke / validatord_smoke)
 # do `pkill -f "bin/<name>$"` on teardown — re-running this script
 # brings them back. No temp-toml override needed for either.
 start_shared() {
  local bin="$1"
  local port="$2"
  if curl -sSf -m 1 "http://127.0.0.1:$port/health" >/dev/null 2>&1; then
    echo "  $bin :$port already up (skipping)"
    return 0
  fi
  echo "[gostack] $bin :$port not up; starting"
  nohup ./bin/"$bin" -config lakehouse.toml > "/tmp/gostack-logs/$bin.log" 2>&1 & disown
  for _ in $(seq 1 50); do
-    if curl -sSf -m 1 "http://127.0.0.1:3220/health" >/dev/null 2>&1; then
+    if curl -sSf -m 1 "http://127.0.0.1:$port/health" >/dev/null 2>&1; then
-      echo "  chatd :3220 up"
+      echo "  $bin :$port up"
-      break
+      return 0
    fi
    sleep 0.1
  done
-else
+  echo "  $bin :$port FAILED — log tail:"
-  echo "  chatd :3220 already up (skipping)"
+  tail -20 "/tmp/gostack-logs/$bin.log"
-fi
+  return 1
 }
 start_shared chatd      3220
 start_shared validatord 3221
 echo
 echo "[gostack] ready · sweep:"
-for p in 4110 4211 4212 4213 4214 4215 4216 4217 4218 4219 3220; do
+for p in 4110 4211 4212 4213 4214 4215 4216 4217 4218 4219 3220 3221; do
  curl -sSf -m 1 "http://127.0.0.1:$p/health" 2>/dev/null | head -c 80
  echo
 done
 echo
 echo "[gostack] persistent stack: ports :4110+:4211-:4219 · bucket=$PERSISTENT_BUCKET"
 echo "[gostack] smoke harnesses:  ports :3110+:3211-:3219 · bucket=lakehouse-go-primary"
-echo "[gostack] shared:           chatd at :3220 (read-mostly LLM dispatch)"
+echo "[gostack] shared:           chatd :3220 + validatord :3221 (no S3 / JSONL-only state)"
 echo "[gostack] tear down via: pkill -f 'bin/persistent-'"
@@ -0,0 +1,64 @@
 # Real-shape coordinator queries — generated from fill_events.parquet
 # (real-shape demand data; queries built mechanically from event rows).
 # Source: /home/profit/lakehouse/data/datasets/fill_events.parquet (123 rows total, rows [10,60) emitted, styles=[need])
 #
 # Styles:
 #   need:         'Need N {role}{s} in {city} {state} starting at {at} for {client}'
 #                 — matches scripts/playbook_lift's extractRoleFromNeed regex
 #   client_first: '{client} needs N {role}{s} in {city} {state} at {at}'
 #   looking:      'Looking for N {role}{s} at {client} in {city} {state} for {at} shift'
 #   shorthand:    'N {role}{s} {city} {state} {at} {client}'
 #
 # Only 'need' currently extracts a role. The other three test the
 # substrate's bleed behavior when the role gate is silently disabled.
 Need 1 Loader in Kansas City MO starting at 17:30 for Cornerstone Fabrication
 Need 2 Assemblers in Cincinnati OH starting at 14:30 for Great Lakes Mfg
 Need 1 Forklift Operator in Lexington KY starting at 08:30 for Vanguard Components, deadline 2026-05-20
 Need 2 Assemblers in Flint MI starting at 08:30 for Centennial Packaging
 Need 2 Welders in Indianapolis IN starting at 10:00 for Northland Logistics
 Need 2 Material Handlers in Cincinnati OH starting at 13:00 for Great Lakes Mfg
 Need 3 Pickers in Flint MI starting at 17:00 for Centennial Packaging
 Need 3 Packers in Indianapolis IN starting at 09:00 for Heritage Foods, deadline 2026-05-04
 Need 3 Assemblers in Columbus OH starting at 17:30 for Riverfront Steel
 Need 5 Machine Operators in Cleveland OH starting at 14:30 for Apex Warehouse
 Need 2 Assemblers in Grand Rapids MI starting at 13:00 for Cornerstone Fabrication, deadline 2026-05-21
 Need 2 Pickers in Akron OH starting at 10:30 for Summit Industrial
 Need 3 Quality Techs in Lexington KY starting at 12:30 for Keystone Plastics, deadline 2026-06-14
 Need 4 Assemblers in Gary IN starting at 12:00 for Heritage Foods
 Need 3 Packers in Toledo OH starting at 16:00 for Cornerstone Fabrication, deadline 2026-06-02
 Need 4 Warehouse Associates in Fort Wayne IN starting at 13:00 for Cornerstone Fabrication, deadline 2026-05-17
 Need 4 Assemblers in Columbus OH starting at 13:00 for Midway Distribution
 Need 2 Shipping Clerks in Chicago IL starting at 17:00 for Midway Distribution
 Need 2 Machine Operators in Chicago IL starting at 11:00 for Midway Distribution
 Need 3 CNC Operators in Grand Rapids MI starting at 10:00 for Parallel Machining, deadline 2026-06-14
 Need 1 Warehouse Associate in Lexington KY starting at 09:30 for Keystone Plastics, deadline 2026-06-14
 Need 2 Material Handlers in Gary IN starting at 11:00 for Heritage Foods
 Need 5 Assemblers in Fort Wayne IN starting at 16:30 for Midway Distribution, deadline 2026-06-09
 Need 1 Loader in Cincinnati OH starting at 08:00 for Summit Industrial
 Need 3 Quality Techs in Columbus OH starting at 12:00 for Riverfront Steel
 Need 1 Machine Operator in Columbus OH starting at 09:30 for Riverfront Steel
 Need 3 Machine Operators in Madison WI starting at 12:00 for Great Lakes Mfg, deadline 2026-05-24
 Need 2 Material Handlers in Kansas City MO starting at 11:30 for Parallel Machining
 Need 3 Loaders in Flint MI starting at 16:00 for Parallel Machining
 Need 2 Welders in Louisville KY starting at 13:00 for Horizon Supply, deadline 2026-06-04
 Need 1 CNC Operator in Flint MI starting at 10:30 for Horizon Supply
 Need 1 Material Handler in Columbus OH starting at 15:30 for Riverfront Steel
 Need 2 Forklift Operators in Louisville KY starting at 14:30 for Cornerstone Fabrication, deadline 2026-06-08
 Need 2 Warehouse Associates in Chicago IL starting at 10:00 for Northland Logistics
 Need 2 Material Handlers in Gary IN starting at 15:00 for Parallel Machining
 Need 1 Forklift Operator in Grand Rapids MI starting at 10:00 for Cornerstone Fabrication, deadline 2026-05-21
 Need 2 Pickers in Louisville KY starting at 12:30 for Cornerstone Fabrication, deadline 2026-06-08
 Need 2 Loaders in Indianapolis IN starting at 17:30 for Midway Distribution
 Need 1 Shipping Clerk in Indianapolis IN starting at 12:30 for Northland Logistics
 Need 2 Assemblers in Cincinnati OH starting at 08:00 for Keystone Plastics, deadline 2026-05-26
 Need 5 Quality Techs in Kansas City MO starting at 11:30 for Summit Industrial, deadline 2026-05-23
 Need 2 Machine Operators in Gary IN starting at 10:00 for Heritage Foods
 Need 1 Packer in Chicago IL starting at 09:30 for Midway Distribution
 Need 2 Pickers in Lexington KY starting at 17:30 for Vanguard Components, deadline 2026-05-20
 Need 2 Maintenance Techs in Grand Rapids MI starting at 17:00 for Pioneer Assembly, deadline 2026-05-20
 Need 1 Material Handler in Detroit MI starting at 10:30 for Summit Industrial
 Need 1 Welder in Akron OH starting at 15:00 for Summit Industrial
 Need 1 Shipping Clerk in Cincinnati OH starting at 13:30 for Summit Industrial
 Need 5 Packers in Indianapolis IN starting at 10:30 for Midway Distribution
 Need 1 Forklift Operator in Louisville KY starting at 10:30 for Cornerstone Fabrication, deadline 2026-06-08
Author	SHA1	Message	Date
rootandClaude Opus 4.7	eb4308d8fd	real_006 diagnosis: Q43 leak is cross-city, not cross-role Traced w-279's path through the substrate. The leak source is Q49 (Packers in Indianapolis IN for Midway Distribution), NOT Q18 as the initial reading suspected. Q49 recorded w-279 with role=Packers, client=Midway Distribution, city=Indianapolis. Q43 (Packer in Chicago IL for Midway Distribution) ran later. roleEqual("Packer","Packers") → both normalize to "packer" → role gate passes (correctly, by design — they ARE the same role under plural-strip). Cosine distance between Q49's recorded query and Q43's query is small enough to fit inside the 0.20 inject threshold because role + client + count + time-token dominate the embedding (only the city and singular/plural noun differ). Inject fires, w-279 surfaces at Q43's warm top-1 in Chicago, judge correctly rates 2/5 — wrong city. The role gate IS working. What's missing is a CITY gate. Real_002's fix targeted cross-role bleed (Forklift → CNC). real_006 surfaced cross-city bleed within same role + same client — a hole prior tests structurally couldn't reach because they all sourced from rows 0-9 where no such pair existed. Concrete fix surface documented (1 new field, 2 gate checks, 1 regex, ~5 tests). Half a session of work, same shape as real_002. Not implementing tonight — diagnosis only. The 18 unit-level role-gate tests still pass, confirming the gate is doing what it was specified to do. The bug is a missing specification, not a broken implementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 05:13:19 -05:00
rootandClaude Opus 4.7	95f155b017	real_006: distribution-shift test on rows 10-59 of fill_events Methodology fix: gen_real_queries.go gains -offset N flag. Every prior real_NNN test sourced queries from rows 0-9 of fill_events.parquet (default -limit 10), so the substrate's published "8/10 cold-pass top-1 = judge-best" was measured on a memorized slice, not held-out data. real_006 samples 50 fresh rows (offset 10, never seen by the workers or ethereal_workers corpora). Same harness, same local qwen2.5:latest judge, same K=10. ~14 min wall total. Local-only, no cloud calls. Headline findings: - Cold-pass top-1 = judge-best (rank match): 41/50 (82%) vs real_001's 8/10 (80%) — substrate generalizes at rank level. - Strict (rating ≥ 2): 34/50 (68%) — 12-point drop from real_001's 80%. ~7 of 41 "no-discovery" queries had cold top-1 the judge rated 1; the corpus has gaps for some role-city combos in the v3 slice. - Verbatim lift: 9/9 discoveries → warm top-1 (clean, matches real_001 2/2) - Paraphrase recovery: 6/9 → top-1, 9/9 any-rank - Quality regressed: 3/50 — Q43 is the structural one Q43 (Packer at Midway Distribution / Chicago IL) regressed from rating 5 to rating 2 on warm pass with `warm_boosted_count=0` and `playbook_recorded=false`. Q18 (Shipping Clerks at the same client+city) recorded a playbook entry. The regression suggests Q18's recording leaked into Q43 via the warm-pass playbook corpus retrieval surface even though the role gate from real_002 should have blocked it. Three possible paths: extractor failed on one query, gate fires on boost path but not Shape B inject, or cosine drift puts the recorded worker close enough to Q43's embedding that warm-pass retrieval picks it up directly. Diagnosis is the next move. Three same-(client, city) clusters tested: - Heritage Foods Gary IN × 3 distinct roles: clean, distinct workers - Riverfront Steel Columbus OH × 4: cosine-level confusion (Q9/Q25 surface same worker w-281 for Assemblers vs Quality Techs at cold- pass), but no playbook bleed - Midway Distribution Chicago IL × 3: Q43 regression as above What this confirms: substrate works on the fresh distribution at the rank level, verbatim lift is real, paraphrase recovery is real. What this falsifies: real_002's role-gate fix is not structurally airtight. The bleed pattern can still fire under conditions the prior tests didn't reach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 04:54:03 -05:00
rootandClaude Opus 4.7	0e530f4436	drift fix: validatord in start_go_stack + parity refresh Two anchor-vs-reality drifts found during /read-mem audit: 1. start_go_stack.sh never started validatord :3221, even though it shipped 2026-05-02 (`f9e7241`) and STATE_OF_PLAY claims it as part of the persistent stack. Cold-boot quietly omitted it, leaving /v1/iterate unreachable on the persistent gateway. Fix: factored chatd's conditional-start block into a start_shared helper, called for both chatd :3220 and validatord :3221. Same shared-with-smokes posture as chatd (no S3 / JSONL-only state, no temp-toml override needed). 2. STATE_OF_PLAY header claimed 3 parity probes / 32 assertions. Reality is 6 probes / 38 assertions since subject_audit landed in `262a77a` (2026-05-03). Header refreshed; cross-references the three runtime-divergence classes documented at lakehouse/STATE_OF_PLAY.md lines 36-39. Parity reports regenerated as verification artifact (all 6 still green: 8+12+2+4+1+6). Same pattern as `c0a55b1`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 03:27:47 -05:00
root	c0a55b1182	parity reports: regenerated 2026-05-03 morning verification All 6 probes re-run post-restart for today's verification: validator(6/6) + extract_json(12/12) + session_log(4/4) + materializer(2/2) + embed(8/8) + subject_audit(6/6) = 38/38	2026-05-03 05:27:14 -05:00
rootandClaude Opus 4.7	df2a9d1f77	catalogd: BiometricCollection on Go SubjectManifest reader Mirrors lakehouse f1fa6e4 (Phase 1.6 Gate 3a). Adds: type BiometricCollection struct { DataPath string TemplateHash string CollectedAt time.Time ConsentVersionHash string (omitempty) Classifications json.RawMessage (omitempty) } SubjectManifest gains *BiometricCollection biometric_collection,omitempty field so manifests written by the Rust biometric upload endpoint round-trip through the Go reader without losing the new collection record. Cross-runtime parity probe (subject_audit_parity.sh) still 6/6 byte-identical after the field addition — the omitempty pointer means manifests without a collection record produce identical bytes to pre-Gate-3a manifests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 04:55:49 -05:00
rootandClaude Opus 4.7	857ca4c971	catalogd: HTML-safe escape fix + decisions tracker entry Per 2026-05-03 step_7_8_retention_and_parity scrum (opus WARN on parity_subject_audit.rs:canonical_json): Go's json.Marshal HTML-escapes < > & to < > & by default. Rust's serde_json::to_vec keeps them literal. Any audit row with these chars in any string field would silently produce different canonical bytes across runtimes → broken HMAC chain. Latent because no production audit field has carried <>& yet, but realistic for purpose strings ("error & retry") or trace_id values ("<HTTP-Request-Id>"). Fix: marshalNoEscapeHTML helper wraps json.Encoder.SetEscapeHTML(false) + trims trailing newline. Routed through writeCanonical for both keys and scalar values. Regression test: TestVerifyChain_HtmlChars_NotEscaped (purpose has &, trace_id has <>) asserts the canonical bytes contain literal chars, not escape sequences. 11 unit tests pass including the new one; parity probe still 6/6 byte-identical against live production audit logs. Decisions tracker: added 2026-05-03 entry for SUBJECT_MANIFESTS_ON_CATALOGD Steps 1-8 closure + 6th cross-runtime parity probe (was 5). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 04:29:53 -05:00
rootandClaude Opus 4.7	262a77a52a	subject-audit parity (Step 8) — Go reader + cross-runtime probe Per /home/profit/lakehouse/docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8. Go side reads SubjectManifest + verifies HMAC chain on per-subject audit JSONL files using IDENTICAL canonical-JSON + HMAC-SHA256 algorithm to crates/catalogd/src/subject_audit.rs. A Rust-written chain now verifies under Go and vice versa. Files: - internal/catalogd/subject.go SubjectManifest, SubjectAuditRow, AuditAccessor, AuditLogEntry LoadSubjectManifest, LoadKeyFile (32-byte minimum, matches Rust) ReadAuditLog, VerifyChain canonicalRowBytesFromRaw (production), canonicalRowBytesFromStruct (tests) computeRowHMAC, CanonicalAndHmac (parity helper) - internal/catalogd/subject_test.go (10 unit tests) - scripts/cutover/parity/subject_audit_helper/main.go CLI helper mirroring crates/catalogd/src/bin/parity_subject_audit.rs - scripts/cutover/parity/subject_audit_parity.sh Two-phase probe: known-answer + every real audit log Two real bugs caught + fixed by the probe authoring loop: 1. omitempty on AuditAccessor.TraceID stripped the field when empty, producing different canonical bytes than Rust (which always writes the field). Removed omitempty. Rust + Go now produce identical bytes for rows with trace_id="" (the common production case). 2. time.RFC3339Nano strips trailing zeros from nanoseconds, producing "...46143921" where Rust's chrono AutoSi produces "...461439210". Hashing through the parsed-then-re-marshaled struct breaks the chain on any row whose nanos end in 0. Fixed by canonicalizing from the RAW LINE BYTES (preserves the original timestamp string byte-for-byte). Test TestVerifyChain_RawBytesPreserveTimePrecision regression-locks this with a hand-crafted nanos=461439210 row. Live verification (6 / 6 byte-identical assertions): - Phase 1 known-answer: canonical bytes (266) + HMAC match - Phase 2 real logs: WORKER-1..5 audit JSONL all verify under both runtimes with identical (count, tip, verified, error) output Report: reports/cutover/gauntlet_2026-05-02/parity/subject_audit_parity.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 04:17:15 -05:00
rootandClaude Opus 4.7	22c0b42e96	config: mark unwired ModelsConfig tier fields as scaffolding Surfaced during today's local-only audit. The ModelsConfig struct parses 11 tier fields from lakehouse.toml (LocalFast/LocalEmbed/ LocalJudge/LocalReview/CloudJudge/CloudReview/CloudStrong/ FrontierReview/FrontierArch/FrontierStrong/FrontierFree) and exposes Resolve(tier) → model. As of 2026-05-03, NO code calls Resolve(). Operators setting these in lakehouse.toml see no runtime effect — silent config drift. Not removing the fields (would silently swallow operator's existing TOML and make it harder to wire later). Adding a clear UNWIRED warning comment instead. WeakModels IS live (consumed by internal/workflow/modes.go + internal/matrix/downgrade.go) — split out with its own comment so it's not lumped with the dead fields. If a future commit wires up Resolve() consumers, replace the UNWIRED comment with the consumer reference. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 02:54:10 -05:00