0c74b82fc8a38dae9351f953d2ffcaad591dc8ed
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rootandClaude Opus 4.7
|
0c74b82fc8 |
phase 1.6 gate 4: REMOVE name → ethnicity / gender inference
Per docs/PHASE_1_6_BIPA_GATES.md Gate 4 + AUDIT_TRAIL_PRD §4 protected- attribute exclusion rule. The lookup tables + inference functions in search.html (3375-3499) and console.html (245-311) were dead code in the rendering path — headshot rendering disabled 2026-04-28 left these functions defined but unused. Removing them forecloses both Title VII discriminatory-feature-engineering AND BIPA biometric-information- derived-from-biometric-identifier arguments. Removed: - FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN - SURNAMES_HISPANIC, SURNAMES_SOUTH_ASIAN, SURNAMES_EAST_ASIAN, SURNAMES_MIDDLE_EASTERN, SURNAMES_BLACK - guessGenderFromFirstName(), guessEthnicityFromName(), guessEthnicityFromFirstName(), genderFor() From both search.html and console.html. Replacement: deprecation comment block referencing the BIPA gates doc. Verified: zero live consumers anywhere in mcp-server/. Searched for genderFor()/guessEthnicityFromName()/guessEthnicityFromFirstName()/ guessGenderFromFirstName() call sites — none remain. Per J 2026-05-03: this kind of test-code-leaked-into-main is exactly what J wants cleaned up. The face-pool inference was meant as a testing tool for synthetic icon generation but ended up as production-shape inference logic in the customer-facing UI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
cd440d4cee |
audit phase 1.6: BIPA pre-launch gates — block identity-service backfill
Per IDENTITY_SERVICE_DESIGN v3 §5 Step 0, Phase 1.6 is hard
prerequisite to identityd backfill. This doc specifies the 5 gates +
2 supporting deliverables that must ship before real-photo intake.
Five gates (BIPA §15 compliance):
1. Public retention schedule — counsel writes; engineering files+hash
2. Informed written consent — counsel writes template; engineering
wires identityd consent-status enforcement
3. Photo-upload endpoint with consent enforcement — POST /v1/identity/
subjects/{id}/photo with hard 403 when biometric_consent_status
!= 'given'; quarantined storage path; deepface output isolated
to identityd subjects table (not synthetic-face manifest)
4. Deprecate name → ethnicity inference (mcp-server/search.html
lookup tables removed; Phase 1.5 §1B finding closed)
5. Destruction runbook — operator-facing; ties to identityd
/erase endpoint with biometric-specific erasure path; daily
sweep job for biometric_retention_until expiry
Plus:
- Cryptographic attestation that no biometric data exists
pre-identityd (per v3-B11) — defends against
infrastructure-as-notice plaintiff argument
- Employee BIPA-handling training acknowledgment
Engineering effort: ~4-5 days (one week to stage everything ready).
Counsel effort: ~3-6 weeks calendar (review cycles dominate).
Calendar bottleneck is counsel, not engineering.
Phase 1.6 exit = 7 checked gates + signoffs. Until done, identityd
backfill cannot proceed (per identity service design v3 §5 Step 0).
5 open questions for J + counsel: photo-upload UX, consent
mechanism (DocuSign/click/paper), named operator list, named
counsel for sign-off, public privacy policy URL.
No code changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
8129ddd883 |
identity service: v3 amendments — second-pass scrum BUILD-WITH-CHANGES
Re-scrummed v2 across opus + kimi + gemini. All 3 verdict:
BUILD-WITH-CHANGES. v1 blockers verified RESOLVED. 12 new v2
findings folded as v3 amendments in §12.
Convergent v2 findings (≥2 reviewers):
v3-A1: mTLS CA root must NOT live in identityd (opus + gemini).
v3 fix: Vault PKI for CA, identityd as intermediate.
v3-A2: Dual-control public key registry must be tamper-evident
(opus + gemini). v3 fix: Vault KV with separate access
policies + server-issued nonces for replay protection.
Single-reviewer v3 amendments (10 more):
- B1: Step 8 fallback-to-SQL needs explicit 14-day time bound
- B2: NER drop-on-detect needs Prometheus alerting
- B3: legal-tier notification transport spec'd (signed Slack/email,
no PII in body, failure non-blocking)
- B4: Step 6 human review SLA flagged — ~7 months at 500/day for
~100k unknown rows; operational decision needed
- B5: Memory zeroing in Go is best-effort (Rust uses zeroize crate);
documented as not cryptographic-grade
- B6: purpose_definitions needs versioning + emergency revocation
(purpose_versions + purpose_revocations tables)
- B7: Cache invalidation needs erasure_generation atomicity
(subjects.erasure_generation int; gateway rejects stale-gen
cache hits) — replaces best-effort pub/sub
- B8: 15-min cooling-off period for dual-control issuance to
prevent emergency-bypass culture
- B9: NER calibrated test set with target recall ≥99.5% on
synthetic adversarial PII
- B10: S3 Object Lock in separate AWS account with write-only IAM;
root credentials held by external party
- B11: BIPA infrastructure-as-notice attestation in Phase 1.6 doc
- B12: Backup retention vs ciphertext-deletion erasure window
documented in RTBF runbook
Estimate revised v2 12-15d → v3 17-20d. Worth it — the cost is what
buys "I would build this" from 3 independent senior security
architects across 3 model lineages.
Must-have v3 items (block implementation): A1, A2, B1, B6, B7, B11.
Should-have (ship in Phase 5 if calendar tight): B2-B5, B8-B10, B12.
Re-scrum NOT recommended for v3 — diminishing returns; must-have
items are concrete fixes with clear acceptance criteria.
No code changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
298fadce41 |
identity service: v2 — fold cross-lineage scrum findings + 4 'would not build' blocker fixes
Scrummed v1 across opus + kimi + gemini lineages via the new model fleet. 3/3 reviewers said 'I would NOT build v1 as written.' 4 convergent blockers, all resolved in v2: 1. Migration order wrong — backfill before validation creates dark database; if backfill bug, no production traffic catches it. v2 inserts BIPA-prereq Step 0 + shadow-write before backfill + shadow-read before cutover. 9-step migration with cryptographic attestation of completeness at quarantine. 2. Master key on disk + legal token static file = 'security theater' per all 3. v2: HashiCorp Vault Transit / AWS KMS for KEK (not sealed file). Legal token: split-secret short-lived JWT (max 24h), dual-control issuance (J + counsel both sign), revocable in <60s. 3. consent_status='inferred_existing' is BIPA prima facie violation (kimi+gemini explicit). v2 backfill uses 'pending_backfill_review'; biometric data NEVER backfilled — separate consent stream. 4. Healthcare default 'general' = HIPAA exposure window for every misclassified subject. v2 default 'unknown' with fail-closed routing (treat unknown as healthcare-equivalent until classified by manual review). Auto-escalation to healthcare on resume_text pattern match. Plus 12 single-reviewer additions: - mTLS mandatory between gateway↔identityd (kimi) - External anchor for audit chain: S3 Object Lock 7-year compliance mode, hourly + on-event commits (all 3) - Audit-log signing key separate from encryption KEK (opus) - Field-level authorization via purpose_definitions table (kimi) - Per-row encryption keys deferred to Phase 7 (kimi simplification) - pii_access_log itself needs legal-tier read auth (opus) - Synchronous cache invalidation pub/sub on RTBF (opus) - Outbound NER pass for Langfuse defense-in-depth (opus TOCTOU) - model_version_hash per decision row (gemini) - /vertical minimal-disclosure endpoint (kimi HIPAA min-necessary) - Auto-escalation healthcare on resume_text pattern (kimi) - Rate limiting + token revocation list (opus) - Oracle tests in audit_parity.sh (kimi SOC2 CC4.1) Architecturally simplified per scrum: - Per-row encryption keys deferred to Phase 7 (single DEK + HSM- wrapped KEK + ciphertext deletion is equivalent practical erasure with less complexity) - PDF render deferred (JSON ships first) - Training-safe export deferred (not critical path) Estimated effort revised 8-10 → 12-15 days. Worth it — every addition was a 3/3-reviewer convergent finding. Re-scrum recommended before implementation starts to verify v2 addresses the v1 blockers. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
565ea4b32a |
audit phase 2: IDENTITY_SERVICE_DESIGN.md — full design doc
Incorporates J's confirmed answers (2026-05-03): - separate daemon (identityd) on :3225 / :4225 - signed JSON with PDF render for legal export - legal-only credential separate from admin token - Langfuse self-hosted (drops cross-border concern) - EU placeholder fields, not enforced - healthcare vertical routing — local-only models for healthcare PHI - training-safe export with hashed pseudonyms Plus Phase 1 + 1.5 findings + scrum-driven priorities: - UUID v7 candidate_id (drops kimi enumeration risk) - per-row encryption with per-subject keys (crypto-erasure target) - pii_access_log with Merkle-style integrity hash chain (FRE 901) - subject_id top-level promotion in all JSONL sinks - Langfuse boundary redaction layer (scrum C2 priority) - adverse-impact comparator pool in audit response (scrum C3) - BIPA-specific consent + retention metadata (scrum C4) - vertical detection at gateway boundary (J answer 10) Implementation single-language: Go (one identityd, both runtimes call it via HTTP). Postgres backing store, isolated schema. Master key in sealed file v1, vault migration path documented. 8-step migration path: stand up empty → backfill from parquet → behind feature flag → cut over reads incrementally → quarantine PII columns in workers_500k. Each step its own commit + gate + rollback. 6 open questions for J before implementation: master key location, Postgres shared vs isolated, vertical backfill default, legal token issuance procedure, crypto-erasure sweep cadence, EU enforcement timeline. Estimated 8-10 working days total. Largest single phase in the audit program. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
fd429f4185 |
audit phase 1.5: BIPA schema audit + outcomes.jsonl content sample
Two follow-up walks per AUDIT_PHASE_1_DISCOVERY §10/C4 + gemini scrum flag. Read-only. No code changes. BIPA findings: - scripts/staffing/tag_face_pool.py uses deepface to extract gender + race + age from face images. Output persists to data/headshots/ manifest.jsonl. For synthetic faces this is fine; for real candidate photos this becomes a regulated biometric database (740 ILCS 14/10). - mcp-server/index.ts:1408 ComfyUI prompt EXPLICITLY embeds protected attributes (age + race + gender) into model prompt — system-level encoding of protected-attribute features into AI workflow. - mcp-server/search.html:3375-3432 has hard-coded FEMALE_NAMES / MALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup tables — name-based ethnicity inference. Title VII / disparate-impact risk separate from BIPA. - data/headshots/manifest.jsonl is TRACKED IN GIT today (synthetic classifications). For real photos, this would be biometric data in version control — serious failure. - No consent flow, no public retention schedule, no deletion procedure, no employee training documented. All required by BIPA §15(a)/(b) before real-photo intake. outcomes.jsonl sample: - 39/101 rows persist candidate names in fills[*].name field today - Sample names: "Carmen I. Garcia", "Jamal Z. Jones", "Jacob N. Patel" (synthetic but real shape) - 0 hits for "culture fit" / "communication" / etc proxy phrases — synthetic data doesn't generate them. When real models reason about real candidates, they will. Append-only persistence makes RTBF cryptographic-erasure-only. Recommends Phase 1.6 (NEW) — BIPA pre-launch gates between Phase 1.5 and Phase 2: BIPA_COMPLIANCE_POLICY.md, consent gate at upload endpoint, quarantine real-photo classifications to data/biometric/, deprecate name->ethnicity lookup tables, unit test that synthetic manifest stays synthetic. 4-8 hours of design + one code commit. 5 open questions for J: where do real photos enter, will deepface tagging path stay for real photos, consent UX, retention duration floor, designated privacy officer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
64bda21614 |
audit PRD: J answered 5 open questions — fold into §10, revise phase plan
Conversation 2026-05-03 — J confirmed: - Photos/video YES → BIPA in full force ($1k-$5k per violation) - Langfuse self-hosted → drops GDPR Art. 44 cross-border concern - EU not in scope now but placeholder needed → design EU-compatible - Healthcare vertical YES → HIPAA BAA needed with model providers, PHI redaction at gateway boundary OR local-only routing for those requests, vertical-detection at boundary is Phase 2 requirement - Training/RAG MAY re-run on outcomes → design as if it will, training- safe export interface needed, crypto-erasure becomes load-bearing evidence chain §10 updated with answered/pending status per question. New §10.5 "Effect on phase plan" introduces: - Phase 1.5 (NEW) — BIPA photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample, BEFORE Phase 2 design - Phase 2 design must now include: EU-placeholder fields, vertical detection, training-safe export, BIPA consent metadata - Phase 9 rehearsal must cover discrimination + BIPA + healthcare PHI 3 questions still pending J's call before Phase 2 design ships: identity service daemon vs in-process, JSON vs signed PDF for legal export, audit endpoint auth model. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
627a5f0c3d |
audit phase 1: §10 scrum-review findings + walk back §1F over-claim
Ran cross-lineage scrum on the discovery doc with the new model fleet
(opus + kimi-k2.6 + gemini-3-flash via Go gateway :4110, custom
"senior security architect" prompt). 3/3 reviewers responded with
substantive 800-1200 word reviews. Saved at /tmp/audit_scrum/.
5 convergent findings (≥2 reviewers) added as §10/C1-C5:
C1. §1F matrix-indexer "good for audit defensibility" claim is over-
claimed — walked back in TL;DR. Trace bodies unverified; treat as
SUSPECTED PII sink until §8.1 sampling completes.
C2. §1E (Langfuse) is the most dangerous leak — fix FIRST, ahead of
view-routing. Boundary-crossing leak (GDPR Art. 44 / CPRA sale /
SOC2 disposal). All 3 reviewers converge on this priority.
C3. Discrimination defense requires the FULL CANDIDATE POOL, not just
fills. EEOC UGESP (1978): need adverse-impact stats on everyone
who could have been picked. Phase 1 worked example missed this.
C4. BIPA / biometric exposure understated in findings (in PRD §10.5
but not translated to actionables). $1k-$5k per-violation regime.
C5. candidate_id must be promoted to top-level field in all JSONL
sinks. Grepping natural-language strings is not defensible audit
strategy. 3/3 reviewers converge.
11 single-reviewer high-value catches added as §10 single-reviewer
section: opus on LLM provider egress (8th PII path), Art. 22 right-
to-explanation, special-category data, DPIA/ROPA/DPA inventory; kimi
on sequential ID enumeration risk, Langfuse retention config, CCPA
de-identified-in-place vs crypto-shred, Bun common-mode failure,
cryptographic audit-trail integrity (Merkle/FRE 901), HIPAA BAA,
revised SELECT * effort estimate; gemini on data residency, "culture
fit" reasoning proxies, comparator-pool snapshot.
§9 reordered: sample first → defense-layer second → Langfuse
boundary third (was view-routing first per original draft;
boundary-crossing leak is higher priority per scrum).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
505ea93726 |
audit phase 1: discovery walk complete — subject + PII surface map
Read-only walk of both runtimes per AUDIT_TRAIL_PRD.md §8 phase 1.
Fills "UNKNOWN" cells in PRD §3 + §7 with file:line evidence.
Headline findings:
- candidates_safe + workers_safe views EXIST as a defense layer but
are BYPASSED — tool registry SQL templates query raw tables
- PII traverses 7+ persistence/transmission paths per fill scenario:
SQL → tool_result → LogEntry → /v1/respond → Langfuse → outcomes.jsonl
→ overseer_corrections.jsonl
- candidate_id is stable but co-located with PII in workers_500k.parquet
(no separate identity service)
- /audit/subject/{id} endpoint does not exist
- Append-only persistence is universal — RTBF requires crypto-erasure
- Pathway memory is structurally subject-agnostic in fingerprints
(defensive); trace bodies may leak PII (needs sampling)
- Go side mirrors Rust PII shape — parity in the leak too
- Worked example (John Martinez audit today): NOT POSSIBLE to produce
complete-and-defensible response
Recommends 4 cheap high-value moves before Phase 2 design starts:
defense-layer enforcement (rewrite 3 SQL templates to _safe views),
sample state.json/Langfuse to confirm pathway memory is clean, walk
Bun mcp-server tool surface, schema-audit for protected-attribute
proxies. None are commitments — J's call.
No code changes in this commit. Companion to AUDIT_TRAIL_PRD.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
b2d717ae44 |
audit PRD: add §10.5 jurisdictional surface (IL + IN, federal, SOC2)
J flagged that the staffing system targets Chicago + Indiana — added a jurisdictional checklist section to the audit-trail PRD so counsel has a working starting point. Covered: - Federal: Title VII, ADEA, ADA, EEOC, OFCCP, FCRA, Section 1981 - Illinois: BIPA (high risk if any candidate photos), AI Video Interview Act (820 ILCS 42), Illinois Human Rights Act (broader than Title VII), PIPA breach notification, Day and Temporary Labor Services Act (directly applies — staffing industry-specific recordkeeping), Cook County + City of Chicago Human Rights Ordinances (additional protected classes including source of income, parental status, credit history) - Indiana: Data Breach Disclosure, Civil Rights Law (lighter than IL), Genetic Information Privacy Act - SOC 2 Type II as the typical SaaS sale gate (Privacy + Security TSCs most relevant; 6-9 month effort to first report) - HIPAA / PCI / ISO 27001 noted as out of current scope but flagged Phase reordering implications captured: - BIPA risk on real candidate photos may need to be resolved BEFORE audit-trail work (class-action exposure) - SOC 2 Type II prep runs in parallel, not after - IL Day and Temporary Labor Services recordkeeping may override our proposed 4-year retention SLA 7 open questions added that counsel must answer before the §8 phases can be locked in. Document is explicit (multiple times) that this is NOT legal advice — a research-grade checklist for J's counsel conversation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
c170ebc86e |
docs: AUDIT_TRAIL_PRD — production-readiness gate for staffing client
J flagged that smoke + parity tests prove the surface compiles, NOT that an audit response can be produced for a specific person — and the staffing client won't sign without defensible discrimination-claim response capability. New docs/AUDIT_TRAIL_PRD.md captures: - worked example: John Martinez at Warehouse B requests audit - subject audit response output format (per-decision row schema) - surface map: where decisions happen today, where the gaps are - PII handling rules (tokenization, protected-attribute exclusion, inferred-attribute risk) - identity service design intent (separate daemon, audited reads) - retention + right-to-be-forgotten policy intent - 9-phase implementation sequence with explicit per-phase exit criteria - cross-runtime requirement (both Rust + Go must satisfy) - 7 open questions blocking phase 2+ that need J's call STATE_OF_PLAY + PRD updated with explicit "production-ready blocker" section pointing at the new doc. The "substrate is shipped" framing gets a caveat: substrate ≠ production-ready until audit phase 9 exits. No code changes. This is the planning artifact J asked for before we start building. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
5368aca4d4 |
docs: sync ADR-019 + PRD + DECISIONS with 2026-05-02 substrate changes
ADR-019: closed the "re-bench when 10M corpus exists" follow-up. Added "Follow-up: 10M re-bench (2026-05-02)" section with the post-fix numbers (search ~20ms warm / ~46ms cold, doc-fetch ~5ms post-btree). Documented the lance-bench-bypassing-IndexMeta bug + 2-layer fix + gauntlet (7 unit + 12 sanitize + 10 smoke probes). Reframes the strategic question as "Lance vs Parquet+HNSW-with-spilling" since HNSW doesn't fit RAM at 10M. DECISIONS: added ADR-022 — drop Python sidecar from Rust hot path. Captures the rationale (236× embed perf gap was pure overhead), co-shipped LRU cache, dev-only Python that survives, cross-runtime parity verification, and the operator runbook signal (ps -ef ABSENT post-deploy). PRD: updated AI Boundary table line + aibridge crate description to reflect direct Ollama path (was: Python FastAPI sidecar → Ollama). Both lines reference ADR-022 for the full rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
e9d17f7d5a |
sanitize: drop over-broad path-missing branch + UTF-8-safe redaction
Re-scrum of yesterday's sanitizer fix surfaced 2 more real bugs in the fix itself (opus, both WARN, neither caught by kimi/qwen): W1 (service.rs:1949) — `mentions_path_missing` standalone branch was too aggressive. A registry-internal error like "/root/.cargo/.../x.rs: no such file or directory" would 404 because it triggers without dataset context. That's a real 500. Dropped the standalone branch; require dataset context AND missing-shape phrase. Lance's actual "Dataset at path X was not found" still satisfies it. W2 (service.rs:2018) — `out.push(bytes[i] as char)` corrupted multi-byte UTF-8 by casting raw bytes to char (only sound for ASCII < 128). A path containing user-supplied non-ASCII names produced Latin-1 mojibake. Rewrote redact_paths to track byte indices and emit unmatched runs as &str slices via push_str(&s[range]) — preserves multi-byte sequences verbatim. Step advance is now per-char, not per-byte, via small utf8_char_len helper. Two new regression tests: - is_not_found_does_not_match_unrelated_path_missing - redact_preserves_multibyte_utf8 (uses 工作 + café in input) 12/12 sanitize tests PASS. Smoke 10/10 PASS. Loop closure for opus re-scrum on the 2026-05-02 fix bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
ac7c996596 |
sweep up scrum WARNs — model const, stale config, temp_path entropy, smoke gate
Four findings deferred from the 2026-05-02 scrum, all 1-5 line fixes: W1 (kimi WARN @ scrum_master_pipeline.ts:1143) — `gemini-3-flash-preview` hardcoded twice in MAP and REDUCE phases. Extracted TREE_SPLIT_MODEL + TREE_SPLIT_PROVIDER constants near the existing config block. Diverging the two would break tree-split coherence (per-shard digests must come from the same model the reducer collapses). W2 (qwen WARN @ providers.toml:30) — stale `kimi-k2:1t` reference in operator-facing comments after PR #13 noted it's upstream-broken. Reframed as historical context ("was X here pre-2026-05-03 — that model is broken") so future operators don't paste-route from the comment. W3 (opus WARN @ vectord-lance/src/lib.rs:622) — temp_path() entropy was only pid+nanos, which collide under tokio scheduling when multiple tests in the same cargo process create temp dirs back-to-back. Added per-process AtomicU64 sequence counter — guarantees uniqueness regardless of clock. W4 (opus INFO @ scripts/lance_smoke.sh:38) — `|| echo '{}'` swallowed curl transport failures (gateway down, network broken, timeout), surfacing as misleading "no method field" jq errors at the next probe. Now captures $? separately, gates a "curl reachable" probe, and only falls back to empty body for the dependent jq parse. Smoke went 9 → 10 probes. Verified: vectord-lance 7/7 tests PASS, gateway cargo check clean, lance_smoke.sh 10/10 PASS against live gateway. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
7bb66f08c3 |
lance: scrum-driven sanitizer + smoke-gate fixes (opus 2026-05-02 BLOCK)
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across"
Cross-lineage scrum on the lance wave (4 bundles, 33 distinct findings)
surfaced 1 real BLOCK and 2 real WARNs from opus that the kimi/qwen
lineages missed. Per feedback_cross_lineage_review.md, opus is the
load-bearing reviewer; cross-lineage convergence is noise unless verified.
BLOCK fix — sanitize_lance_err path-stripping was unsound:
err.split("/home/").next().unwrap_or(&err)
returns Some("") when err STARTS with "/home/", erasing the entire
message. Replaced truncation with redact_paths() — a hand-rolled scanner
that walks the input once, replacing path-shaped substrings with
[REDACTED] while preserving surrounding error context. Catches:
- absolute paths under /root/.cargo, /home, /var, /tmp, /etc, /usr, /opt
- relative variants (Lance occasionally strips leading slash —
observed live "Dataset at path home/profit/lakehouse/data/lance/x
was not found")
- multiple occurrences in one error
- preserves quote/comma/whitespace terminators
WARN fix #1 — is_not_found heuristic was too broad:
lower.contains("not found")
caught real 500s like "column not found", "field not found in schema".
Narrowed to require dataset-shape phrasing AND exclude the
column/field/schema patterns explicitly.
WARN fix #2 — lance_smoke.sh `grep -qvE` was an unsound regression gate.
bash -c "echo '$BODY' | grep -qvE 'pat'"
With -v -q, exits 0 if ANY line lacks the pattern — so a multi-line
body with one leak line + any clean line FALSE-PASSES. Replaced with
the correct "pattern absent" form: `! grep -qE 'pat'`. Also expanded
the pattern set (added /var/, /tmp/) since the scrum surfaced these
as additional leak vectors.
Also unblocks pre-existing pathway_memory test compile error (stale
PathwayTrace init missing 6 Mem0-versioning fields added in
|
||
|
root
|
a294a61ee4 |
Merge remote-tracking branch 'origin/main' into demo/post-pr11-polish-2026-04-28
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across"
|
||
|
rootandClaude Opus 4.7
|
0af62861d2 |
STATE_OF_PLAY: refresh for 2026-05-02 wave (Lance gauntlet + parity + housekeeping)
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:"
Anchor was 5 days stale. Adds the 12-commit wave (Lance backend hardening, sidecar drop, observability parity, gitignore cleanup, gray-zone content add) with verification status for each. Updates DO NOT RELITIGATE with the 4 new things this wave makes load-bearing: - python sidecar dropped from hot path (don't wire it back) - lance gauntlet shipped (don't re-discover the bugs we just fixed) - 32/32 cross-runtime parity (don't build a 6th probe for already-covered surface) - ARCHITECTURE_COMPARISON.md is the single source of truth for cross-runtime decisions Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
41b0a99ed2 |
chore: add real content that was sitting untracked
Surfaced by today's untracked-files audit. None of these are accidents —
multiple are referenced by name in CLAUDE.md and memory files but were
never added.
Categories:
- docs/PHASE_AUDIT_GUIDE.md (106 LOC) — Claude Code phase audit guidance
- ops/systemd/lakehouse-langfuse-bridge.service — Langfuse bridge unit
- package.json — top-level npm manifest
- scripts/e2e_pipeline_check.sh + production_smoke.sh — real test scripts
- reports/kimi/audit-last-week*.md — the "Two reports live" CLAUDE.md cites
- tests/multi-agent/scenarios/ — 44 staffing scenarios (cutover decision A)
- tests/multi-agent/playbooks/ — 102 playbook records
- tests/battery/, tests/agent_test/PRD.md, tests/real-world/* — real tests
- sidecar/sidecar/{lab_ui,pipeline_lab}.py — 888 LOC dev-only UIs that
remain in service post-sidecar-drop (commit
|
||
|
rootandClaude Opus 4.7
|
6e34ef7baf |
gitignore: stop tracking runtime data, logs, build artifacts, scratch
Untracked count was 100+; almost all were data/_*/ daemon state, generated parquets under data/datasets and data/vectors, the 33GB data/lance/ tree, node_modules, exports, logs, per-run distillation reports, and test scratchpads. None of these are content — all regenerate from inputs. Now down to 33 untracked items, all real content (scripts, systemd unit, test scenarios, dev-only sidecar UIs, kimi audit reports). Those need J's call on what to track vs leave parked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
044650a1da |
lance-bench: also build doc_id btree post-IVF — match gateway's migrate behavior
The bench's own measure_random_access_lance uses take(row_position) —
doesn't need the btree. But datasets written by this bench are commonly
queried via /vectors/lance/doc/<name>/<doc_id> downstream, and without
the btree that path falls back to a full table scan. Building inline
keeps bench-produced datasets immediately production-shape and removes
a footgun (the same one that made scale_test_10m's doc-fetch ~100ms
until commit
|
||
|
rootandClaude Opus 4.7
|
5d30b3da89 |
lance: auto-build doc_id btree in migrate handler (root-cause for 10M doc-fetch slowness)
scale_test_10m doc-fetch p50 was ~100ms — full table scan over 35GB. Root cause: the auto-build at service.rs:1492-1503 only fires for IndexMeta- registered indexes during set_active_profile warming. lance-bench writes datasets through /vectors/lance/migrate/* directly, bypassing IndexMeta, so its datasets never get the doc_id btree that ADR-019 depends on. Fix: build the btree inline at the end of lance_migrate. Costs ~1.2s on 10M rows (+269MB on disk), drops doc-fetch from ~100ms to ~5ms (20x). Failure is non-fatal — logs a warning and the dataset stays queryable. Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across 5 calls, smoke 9/9 PASS, vectord-lance 7/7 unit tests PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
7594725c25 |
lance backend: 4-pack — bug fix + smoke + tests + 10M re-bench
lakehouse/auditor 12 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Surfaced by the 2026-05-02 audit (vectord-lance + lance-bench + glue
existed and worked but had no tests, no smoke, leaked server paths
on missing-index search, and the ADR-019 10M re-bench was deferred).
## 1. Fix: missing-index search returned 500 + leaked filesystem path
Pre-fix:
$ POST /vectors/lance/search/no-such-index
HTTP 500
Dataset at path home/profit/lakehouse/data/lance/no-such-index was
not found: Not found: home/profit/lakehouse/data/lance/no-such-index/
_versions, /root/.cargo/registry/src/index.crates.io-...-1949cf8c.../
lance-table-4.0.0/src/io/commit.rs:364:26, ...
Post-fix:
HTTP 404
lance dataset not found: no-such-index
Added `sanitize_lance_err()` in crates/vectord/src/service.rs that:
- maps "not found" / "no such file" patterns → 404 (was 500)
- strips /home/ and /root/.cargo/ paths from any error body
Applied to all 5 lance handlers: search, get_doc, build_index,
append, migrate. The store_for() handle is cheap-and-stateless;
the actual disk hit happens inside the operation, which is where
the leak originated.
## 2. scripts/lance_smoke.sh — first regression gate
9-probe smoke against the live HTTP surface. Exercises only read
paths (no state mutation in CI). Specifically locks the sanitizer
fix — a future regression that re-introduces the path leak fires
the smoke immediately. 9/9 PASS against the live :3100 today.
## 3. Unit tests on vectord-lance/src/lib.rs (was: zero tests)
7 tests covering the public LanceVectorStore API:
- fresh_store_reports_no_state — handle is lazy
- migrate_then_count_and_fetch — Parquet → Lance round-trip
- get_by_doc_id_missing_returns_none — Ok(None) vs Err contract
that lets the HTTP handler return 404 cleanly
- append_grows_count_and_new_rows_fetchable — ADR-019's
structural-difference claim verified at the unit level
- append_dim_mismatch_errors — guards against silently breaking
search by accepting inconsistent-dim rows
- search_returns_nearest — exact-vector match → top-1
- stats_reports_post_migrate_state — locks the field shape
7/7 PASS. cargo test -p vectord-lance --lib green.
## 4. 10M re-bench (deferred from ADR-019)
reports/lance_10m_rebench_2026-05-02.md captures the numbers driven
against the live :3100 over data/lance/scale_test_10m (33GB / 10M
vectors, IVF_PQ confirmed via response method tag).
Headline:
Search cold (10 diverse queries): median ~32ms, mean ~46ms
Search warm (5x same query): ~20ms p50
Doc fetch (5x same id): ~100ms p50
Search latency at 10M is acceptable for batch / async workloads,
too slow for sub-10ms voice/recommendation paths. ADR-019's "Lance
pulls ahead at 10M" claim remains unverified-but-not-refuted — at
this scale HNSW doesn't operationally exist (10M × 768d × 4 bytes =
30GB just for vectors).
Real finding: doc-fetch at 10M is 300x slower than the 100K number
ADR-019 cited (311μs → ~100ms). Likely cause: scalar btree index
on doc_id may not be built for this dataset. Follow-up to
investigate whether forcing build_scalar_index brings it back to
the load-bearing O(1) range. Captured in the report.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
98b6647f2a |
gateway: IterateResponse echoes trace_id + enable session_log_path
lakehouse/auditor 14 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Closes the 2026-05-02 cross-runtime parity gap: Go's
validator.IterateResponse carried trace_id back to callers; Rust's
didn't. A caller pivoting from response → Langfuse → session log
worked on Go but failed on Rust because the join key wasn't visible
in the response body.
## Changes
crates/gateway/src/v1/iterate.rs:
- IterateResponse + IterateFailure gain `trace_id: Option<String>`
(skip-serializing-if-none preserves backward-compat for any
consumer parsing the response without the field)
- Both return sites populated with the resolved trace_id
lakehouse.toml:
- [gateway].session_log_path set to /tmp/lakehouse-validator/sessions.jsonl
— same path Go validatord writes to. The two daemons now co-write
one unified longitudinal log; rows tag daemon="gateway" vs
daemon="validatord" so producers stay distinguishable in DuckDB
queries. Append-write is atomic at the row sizes both runtimes
produce, so concurrent writes from both daemons are safe.
## Verification
Post-restart of lakehouse.service:
POST /v1/iterate with X-Lakehouse-Trace-Id: rust-fix1-test
→ response.trace_id = "rust-fix1-test" ✓ (was: field absent)
→ sessions.jsonl latest row daemon=gateway, session_id=rust-fix1-test ✓ (was: no row)
Cross-runtime drive — same prompt to Rust :3100 and Go :4110:
Rust: trace_id=unified-rust-001, daemon=gateway, accepted
Go: trace_id=unified-go-001, daemon=validatord, accepted
Same file, distinct daemons, one query covers both:
SELECT daemon, COUNT(*) FROM read_json_auto('sessions.jsonl', format='nd') GROUP BY daemon
→ gateway: 2, validatord: 19
All 4 parity probes still 6/6 + 12/12 + 4/4 + 2/2 against live
:3100 + :4110 stacks. Cargo test 4/4 PASS for v1::iterate module.
## Architecture invariant
The "unified longitudinal log" thesis is now demonstrated. Operators
running both runtimes in production point both daemons at the same
session_log_path and DuckDB queries naturally span both producers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
57bde63a06 |
gateway: trace-id propagation + coordinator session JSONL (Rust parity)
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Cross-runtime parity with the Go-side observability wave (commits
d6d2fdf + 1a3a82a in golangLAKEHOUSE). The two layers J flagged:
the LIVE per-call view (Langfuse) and the LONGITUDINAL forensic view
(JSONL queryable via DuckDB). Hard correctness gate (FillValidator
phantom-rejection) was already in place; this is the observability
on top.
## Trace-id propagation
X-Lakehouse-Trace-Id header constant declared in
crates/gateway/src/v1/iterate.rs (matches Go's shared.TraceIDHeader
byte-for-byte). When set on an inbound /v1/iterate request, the
handler reuses it; the chat + validate self-loopback hops forward
the same header so chatd's trace emit nests under the parent rather
than minting a fresh top-level trace per call.
ChatTrace gains a parent_trace_id field. emit_chat_inner skips the
trace-create event when parent is set, only emits the
generation-create which attaches to the existing trace tree. Result:
an iterate session with N retries shows in Langfuse as ONE tree, not
N+1 disconnected traces.
emit_attempt_span (new) writes one Langfuse span per iteration
attempt with input={iteration, model, provider, prompt} and
output={verdict, raw, error}. WARNING level on non-accepted
verdicts. The returned span id is stamped on the corresponding
SessionRecord attempt for cross-log correlation.
## Coordinator session JSONL
crates/gateway/src/v1/session_log.rs — new writer matching Go's
internal/validator/session_log.go schema byte-for-byte:
- SessionRecord with schema=session.iterate.v1
- SessionAttemptRecord per retry
- SessionLogger.append: tokio Mutex serialized append-only
- Best-effort posture (slog.Warn on error, never blocks request)
iterate.rs builds + appends a row on EVERY code path:
- accepted: write_session_accepted with grounded_in_roster bool
derived from validate_workers WorkerLookup (matches Go's
handlers.rosterCheckFor("fill") semantics)
- max-iter-exhausted: write_session_failure
- infra-error: write_infra_error (so a missing /v1/iterate event
never silently disappears from the longitudinal log)
[gateway].session_log_path config field (empty = disabled).
Production: /var/lib/lakehouse/gateway/sessions.jsonl. Operators who
want a unified longitudinal stream can point both Rust and Go
loggers at the same path — write-append is safe at the row sizes we
produce.
## Cross-runtime parity probe
crates/gateway/src/bin/parity_session_log: tiny stdin/stdout helper
that round-trips a fixture through SessionRecord serde.
golangLAKEHOUSE/scripts/cutover/parity/session_log_parity.sh feeds
4 fixtures through both helpers and diffs the rows after stripping
timestamp + daemon (the two fields that legitimately differ between
producers).
Result: **4/4 byte-equal** including the unicode-prompt fixture
("Café résumé ⭐ 你好"). Schema parity holds. The non-trivial-equal
guard in the probe rejects the case where both sides fail
identically — protecting against a regression where one side
silently stops producing valid JSON.
## Verification
- cargo test -p gateway --lib: 90/90 PASS (3 new session_log tests
including concurrent-append safety)
- cargo check --workspace: clean
- session_log_parity.sh: 4/4 fixtures byte-equal
- Both runtimes can append to the same path; DuckDB sees one stream
- The Go-side validatord smoke remains 5/5 (unchanged)
## Architecture invariant
Don't propose to "wire trace-id propagation in Rust" or "add Rust
session log" — both are now shipped on the demo/post-pr11-polish
branch. The longitudinal log + Langfuse tree together cover the
multi-call observability concern J flagged 2026-05-02.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
ba928b1d64 |
aibridge: drop Python sidecar from hot path; AiClient → direct Ollama
lakehouse/auditor 11 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
The "drop Python sidecar from Rust aibridge" item from the architecture_comparison decisions tracker. Universal-win cleanup — removes 1 process + 1 runtime + 1 hop from every embed/generate request, with no behavior change. ## What was on the hot path before gateway → AiClient → http://:3200 (FastAPI sidecar) ├── embed.py → http://:11434 (Ollama) ├── generate.py → http://:11434 ├── rerank.py → http://:11434 (loops generate) └── admin.py → http://:11434 (/api/ps + nvidia-smi) The sidecar's hot-path code (~120 LOC across embed.py / generate.py / rerank.py / admin.py) was pure pass-through: each route translated its request body to Ollama's wire format and returned Ollama's response in a sidecar envelope. Zero logic, one full HTTP hop of overhead. ## What's on the hot path now gateway → AiClient → http://:11434 (Ollama directly) Inline rewrites in crates/aibridge/src/client.rs: - embed_uncached: per-text loop to /api/embed; computes dimension from response[0].length (matches the sidecar's prior shape) - generate (direct path): translates GenerateRequest → /api/generate (model, prompt, stream:false, options:{temperature, num_predict}, system, think); maps response → GenerateResponse using Ollama's field names (response, prompt_eval_count, eval_count) - rerank: per-doc loop with the same score-prompt the sidecar used; parses leading number, clamps 0-10, sorts desc - unload_model: /api/generate with prompt:"", keep_alive:0 - preload_model: /api/generate with prompt:" ", keep_alive:"5m", num_predict:1 - vram_snapshot: GET /api/ps + std::process::Command nvidia-smi; same envelope shape as the sidecar's /admin/vram so callers keep parsing - health: GET /api/version, wrapped in a sidecar-shaped envelope ({status, ollama_url, ollama_version}) Public AiClient API is unchanged — Request/Response types untouched. Callers (gateway routes, vectord, etc.) require zero updates. ## Config changes - crates/shared/src/config.rs: default_sidecar_url() bumps to :11434. The TOML field stays `[sidecar].url` for migration compat (operators with existing configs don't need to rename anything). - lakehouse.toml + config/providers.toml: bumped to localhost:11434 with comments explaining the 2026-05-02 transition. ## What stays Python sidecar/sidecar/lab_ui.py (385 LOC) + pipeline_lab.py (503 LOC) are dev-mode Streamlit-shape UIs for prompt experimentation. Not on the runtime hot path; continue running for ad-hoc work. The embed/generate/rerank/admin routes inside sidecar can be retired, but operators who want to keep the sidecar process running for the lab UI face no breakage — those routes still call Ollama and work. ## Verification - cargo check --workspace: clean - cargo test -p aibridge --lib: 32/32 PASS - Live smoke against test gateway on :3199 with new config: /ai/embed → 768-dim vector for "forklift operator" ✓ /v1/chat → provider=ollama, model=qwen2.5:latest, content=OK ✓ - nvidia-smi parsing tested via std::process::Command path - Live `lakehouse.service` (port :3100) NOT yet restarted — deploy step is operator-driven (sudo systemctl restart lakehouse.service) ## Architecture comparison update (Captured separately in golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md decisions tracker.) The "drop Python sidecar" line moves from _open_ to DONE. The Rust process model now has 1 mega-binary instead of 1 mega-binary + 1 sidecar process — a small but real reduction in ops surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
654797a429 |
gateway: pub extract_json + parity_extract_json bin (cross-runtime probe)
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Supports the 2026-05-02 cross-runtime parity probe at
golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh which
feeds identical model-output strings through both runtimes' extract_json
and diffs results.
## Changes
- crates/gateway/src/v1/iterate.rs: extract_json gains `pub` + a
comment pointing at the Go counterpart and the parity probe path
- crates/gateway/src/lib.rs: NEW thin lib facade re-exporting the
modules so sub-binaries can reuse them. main.rs is unchanged
(still uses local mod declarations)
- crates/gateway/src/bin/parity_extract_json.rs: NEW ~30-LOC binary
that reads stdin, calls extract_json, prints {matched, value} JSON
## Probe result (logged in golangLAKEHOUSE)
12/12 match across fenced blocks, nested objects, unicode, escaped
quotes, top-level array, malformed JSON. Both runtimes' algorithms
are genuinely equivalent.
Substrate gate the probe enforces: `cargo test -p gateway extract_json`
PASS before any parity comparison runs. So a future divergence in
the live extract_json fires either as a Rust test failure (live
behavior changed) or a probe diff (Go behavior changed) — never
silently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
c5654d417c |
docs: pointer to ARCHITECTURE_COMPARISON.md source in golangLAKEHOUSE
lakehouse/auditor 18 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Per J's request: the parallel-runtime comparison is a living source
file maintained at /home/profit/golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md.
This file is a pointer reachable from the Rust repo's docs/ so the
comparison is discoverable from either side.
Doesn't contain authoritative content — just the link + a quick
status summary + update guidance ('source lives in golangLAKEHOUSE,
don't drift two copies').
|
||
|
root
|
150cc3b681 |
aibridge: LRU embed cache - 236x RPS gain on warm workloads. Per architecture_comparison.md universal-win for Rust side. Cache key (model,text), default 4096 entries, in-process inside gateway. Load test: 128 RPS -> 30k+ RPS, p50 78ms -> 129us.
lakehouse/auditor 20 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
|
||
|
rootandClaude Opus 4.7
|
9eed982f1a |
mcp-server: /_go/* pass-through for G5 cutover slice
Adds an opt-in pass-through that routes Bun mcp-server requests
to the Go gateway when GO_LAKEHOUSE_URL is set. /_go/v1/embed,
/_go/v1/matrix/search etc. flow through Bun frontend → Go
backend without touching any existing tool. Off-by-default
(empty GO_LAKEHOUSE_URL → 503 with rationale); enabled via
systemd drop-in at:
/etc/systemd/system/lakehouse-agent.service.d/go-cutover.conf
This is the first slice of real Bun-fronted traffic hitting the
Go substrate. The /api/* pass-through (Rust gateway) and every
existing tool are unmodified — fully additive cutover step.
Reversible: unset GO_LAKEHOUSE_URL or remove the systemd drop-in
and restart lakehouse-agent.service.
Verified end-to-end against persistent Go stack on :4110:
/_go/health → {"status":"ok","service":"gateway"}
/_go/v1/embed → nomic-embed-text-v2-moe vectors (dim=768)
/_go/v1/matrix/search → 3/3 Forklift Operators (role+geo match)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
3d068681f5 |
distillation: regenerated acceptance + audit reports (run_hash refresh)
lakehouse/auditor 17 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:"
Phase 6 acceptance + Phase 8 full-audit reports re-run; bit-for-bit reproducibility property still holds (run 1 hash == run 2 hash), just at a new value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
8de94eba08 |
cleanup: bump qwen2.5 → qwen3.5:latest in active defaults
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:"
stronger local rung is now the small-model-pipeline tier-1 default across both Rust legacy + Go rewrite (cf. golangLAKEHOUSE phase 1). same JSON-clean property as qwen2.5, more capacity. ollama still serves both side-by-side; rollback is a 4-line revert if a workload regresses. active-default sites: - lakehouse.toml [ai] gen_model + rerank_model → qwen3.5:latest - mcp-server/observer.ts diagnose call (Phase 44 /v1/chat path) → qwen3.5:latest - mcp-server/index.ts model roster doc → qwen3.5:latest first - crates/vectord/src/rag.rs ContinuableOpts + RagResponse.model → qwen3.5:latest skipped: execution_loop/mod.rs comments describing historic qwen2.5 tool_call quirks — those are documentation of past behavior, not active defaults. data/_catalog/profiles/*.json are runtime-generated (gitignored), not in scope for tracked changes. cargo check -p vectord: clean. no behavioral change in the audit pipeline — same JSON-clean local model, same think=Some(false) posture, just stronger upstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
a00e9bb438 |
infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths
lakehouse/auditor 2 blocking issues: State field rename likely incomplete — `opencode_key` may not exist on `self.state`
Ollama Pro plan went live today (39-model fleet on the same OLLAMA_CLOUD_KEY) and OpenCode Zen was already wired in the gateway but not consumed. Routing every gpt-oss call site to faster / stronger replacements: | Site | gpt-oss → replacement | Why | |---|---|---| | ollama_cloud default | gpt-oss:120b → deepseek-v3.2 | newest DeepSeek revision; live-probed `pong` | | openrouter default | openai/gpt-oss-120b:free → x-ai/grok-4.1-fast | already the scrum LADDER's PRIMARY | | modes.toml staffing_inference | openai/gpt-oss-120b:free → kimi-k2.6 | coding-specialized, on Ollama Pro | | modes.toml doc_drift_check | gpt-oss:120b → gemini-3-flash-preview | speed leader for factual checks | | scrum_master_pipeline tree-split MAP+REDUCE | gpt-oss:120b → gemini-3-flash-preview | latency-dominated path (5-20× per file) | | bot/propose.ts CLOUD_MODEL | gpt-oss:120b → deepseek-v3.2 | same Ollama key, faster | | mcp-server/observer.ts overseer label fallback | gpt-oss:120b → claude-opus-4-7 | matches new overseer model | | crates/gateway/src/execution_loop overseer escalation | ollama_cloud/gpt-oss:120b → opencode/claude-opus-4-7 | frontier reasoning matters here — fires only after local self-correct fails twice; Zen pay-per-token cost is bounded | Verification: - `cargo check -p gateway --tests` — clean - Live probes through localhost:3100/v1/chat: - `opencode/claude-opus-4-7` → "pong" - `gemini-3-flash-preview` (ollama_cloud) → "pong" - `kimi-k2.6` (ollama_cloud) → "pong" - `deepseek-v3.2` (ollama_cloud) → "Pong! 🏓" Notes: - kimi-k2:1t still upstream-broken (HTTP 500 on Ollama Pro probe today, matches yesterday's memory). Replacement table never picks it. - The Rust changes need a `systemctl restart lakehouse.service` to take effect on the running gateway. TS callers reload on next run. - aibridge/src/context.rs still has gpt-oss:{20b,120b} in its window- size lookup table; harmless and kept for callers that pass it explicitly as an override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
d475fc7fff |
infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths
Ollama Pro plan went live today (39-model fleet on the same OLLAMA_CLOUD_KEY) and OpenCode Zen was already wired in the gateway but not consumed. Routing every gpt-oss call site to faster / stronger replacements: | Site | gpt-oss → replacement | Why | |---|---|---| | ollama_cloud default | gpt-oss:120b → deepseek-v3.2 | newest DeepSeek revision; live-probed `pong` | | openrouter default | openai/gpt-oss-120b:free → x-ai/grok-4.1-fast | already the scrum LADDER's PRIMARY | | modes.toml staffing_inference | openai/gpt-oss-120b:free → kimi-k2.6 | coding-specialized, on Ollama Pro | | modes.toml doc_drift_check | gpt-oss:120b → gemini-3-flash-preview | speed leader for factual checks | | scrum_master_pipeline tree-split MAP+REDUCE | gpt-oss:120b → gemini-3-flash-preview | latency-dominated path (5-20× per file) | | bot/propose.ts CLOUD_MODEL | gpt-oss:120b → deepseek-v3.2 | same Ollama key, faster | | mcp-server/observer.ts overseer label fallback | gpt-oss:120b → claude-opus-4-7 | matches new overseer model | | crates/gateway/src/execution_loop overseer escalation | ollama_cloud/gpt-oss:120b → opencode/claude-opus-4-7 | frontier reasoning matters here — fires only after local self-correct fails twice; Zen pay-per-token cost is bounded | Verification: - `cargo check -p gateway --tests` — clean - Live probes through localhost:3100/v1/chat: - `opencode/claude-opus-4-7` → "pong" - `gemini-3-flash-preview` (ollama_cloud) → "pong" - `kimi-k2.6` (ollama_cloud) → "pong" - `deepseek-v3.2` (ollama_cloud) → "Pong! 🏓" Notes: - kimi-k2:1t still upstream-broken (HTTP 500 on Ollama Pro probe today, matches yesterday's memory). Replacement table never picks it. - The Rust changes need a `systemctl restart lakehouse.service` to take effect on the running gateway. TS callers reload on next run. - aibridge/src/context.rs still has gpt-oss:{20b,120b} in its window- size lookup table; harmless and kept for callers that pass it explicitly as an override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
f4dc1b29e3 |
demo: search.html — Live Market explainer rewrite + fp-bar viewport-paint + compact contract cards
lakehouse/auditor 18 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:"
Four UI changes landing together since they all polish Section ① and
Section ② of the public demo:
1. Section ① (Live Market — Chicago) explainer rewritten data-source-
first ("Live from City of Chicago Open Data...") with bolded dial
names so a skimmer can map the visual to the prose. Drops the
"internal calendar" jargon and the slightly-overclaiming "rest of
the page is reacting" framing — downstream sections read the same
feed but don't react to the per-shift filter, so the new copy says
"this row is its heartbeat" instead.
2. Fill-probability bar gets a left-to-right paint reveal (clip-path
inset animation) so the green→gold→orange→red gradient reads as a
*timeline growing* instead of a static heatmap with a "danger zone"
at the right. Followed by a 30%-wide shimmer sweep on a 3.4s loop
for live-signal feel.
3. Paint trigger moved from on-render to IntersectionObserver — by
the time the user scrolls to Section ② the on-render animation had
already finished. Now each bar paints in over 2.8s when it enters
viewport (threshold 0.2, 350ms entry delay). Single shared observer,
unobserve()s after firing so the watch list trends to zero.
4. Contract cards now compact-by-default with click-to-expand. New
summary strip shows revenue / margin / fill-by-1wk / top candidate
so scanners get the punchline without expanding. Click anywhere on
the card surface (excluding inner content) to expand the full FP
curve, economics grid, candidates list, and Project Index. Project
Index auto-opens with the parent card so users actually find the
build signals — but only on user-driven expand (avoiding 20× OSHA
scrapes on page load). grid-template-rows: 0fr → 1fr animation
handles the smooth height transition.
All four animations honor prefers-reduced-motion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
f892230699 |
demo: search.html UX polish — skeleton loader, card-in stagger, hero takeover, B&W faces
Search results no longer pop in as a single block. New behavior:
- Skeleton list pre-claims the vertical space results will occupy
with shimmering placeholder cards, so arriving results fade in
over the skeleton instead of pushing layout. Sweep is staggered
per row for a "rolling wave" not "everything blinking together".
- Domain-language stage caption ("matching against permits",
"ranking by reliability") rotates on a fixed schedule so users
read progress, not a stuck spinner.
- @keyframes card-in: real worker cards rise 4px and fade in over
350ms with nth-child stagger across the first ~12 rows. Honors
prefers-reduced-motion.
- Avatar imgs filter through grayscale + slight contrast/blur to
pull the SDXL Turbo color cast (which screams "AI generated" at
small sizes). Cert icons get the same treatment.
- Once-per-session hero takeover compresses the Section ⓪ strip
("Not a CRM — an index that learns from you") into a centered
hero on first paint, dismissed by clicking anywhere. Stats
hydrate from live endpoints.
console.html: mirrors the avatar B&W filter for visual consistency,
and removes the headshot insertion entirely — back to monogram
initials. The console (internal staffer view) doesn't need synthetic
faces; the public demo at /lakehouse/ does.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
4b92d1da91 |
demo: icon recipe pipeline + role-aware portraits + ComfyUI negative-prompt override
Adds two single-source-of-truth recipe files that drive both the
hot-path render server and the offline pre-render scripts:
- role_scenes.ts: per-role-band scene clauses (clothing + backdrop).
Forklift operators look like forklift operators instead of
collapsing to interchangeable studio shots. SCENES_VERSION mixes
into the headshot cache key so a coordinator tweak refreshes every
matching face on next view.
- icon_recipes.ts: cert / role-prop / status / hazard / empty icons
with deterministic per-recipe seeds + fuzzy text resolver.
ICONS_VERSION suffix on the cached file means edits don't
overwrite in place — misfires are recoverable.
Routes (mcp-server/index.ts):
- GET /headshots/_scenes — exposes SCENES + version to the
pre-render script so prompts don't drift between batch and hot-path.
- GET /icons/_recipes — same idea for icons.
- GET /icons/cert?text=... — resolves free-text cert names to a
recipe and 302s to the rendered icon. 404 (not 500) when no recipe
matches so the front-end can hang `onerror="this.remove()"`.
- GET /icons/render/{category}/{slug} — cache-or-render at 256² (8
steps) for crisper edges than 512² when downsampled to 14px.
ComfyUI portrait support (scripts/serve_imagegen.py):
The editorial workflow had `human, person, face` baked into its
negative prompt — actively sabotaging portraits. _comfyui_generate
now accepts negative_prompt/cfg/sampler/scheduler overrides, and
those mix into the cache key so portrait calls don't collapse into
hero-shot cache hits.
scripts/staffing/render_role_pool.py: pre-renders the role-aware
face pool by reading SCENES from /headshots/_scenes — single source
of truth verified at run time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
1745881426 |
staffing: face pool fetch preserves prior tags + --shrink gate + atomic manifest write
fetch_face_pool was wiping 952 hand-classified rows when re-run from a Python without deepface installed (it reset every gender to None). Now: - Loads existing manifest by id and overlays only fetch-owned fields, so gender/race/age/excluded survive a refetch. - deepface pass tags only records that don't already have a gender; deepface unavailable means "leave existing tags alone" not "reset". - New --shrink flag required to drop ids >= --count. Default refuses to shrink the pool silently. - Atomic write via tmp + os.replace so an interrupted run can't corrupt the manifest. - Dedupes duplicate id lines (root cause of the 2497-row manifest backing a 1000-face pool). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
a05174d2fa |
ops: track tif_polygons.ts orphan import
entity.ts imports findTifDistrict from ./tif_polygons.js but the source file was never committed — only present in the working tree. Adding it so a fresh clone compiles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
f9a408e4c4 |
Surname → ethnicity routing + ComfyUI fallback for sparse pool buckets + cache-buster
Three problems J flagged ("not matching properly", "same faces", "still
showing old icons") had three different roots:
1. MISMATCH: front-end was first-name only, so "Anna Cruz" / "Patricia
Garcia" / "John Jimenez" all defaulted to caucasian. Added
SURNAMES_HISPANIC / _SOUTH_ASIAN / _EAST_ASIAN / _MIDDLE_EASTERN
dicts to both search.html and console.html. Surname is checked
FIRST (stronger signal for hispanic + asian than first names),
then first-name fallback. Cruz → hispanic, Patel → south_asian,
Nguyen → east_asian, regardless of first name.
2. SAME FACES: pool buckets are uneven — woman/south_asian=3,
man/black=4, woman/middle_eastern=2 — so any worker in those
buckets collapses to 2-4 photos no matter how good the hash is.
/headshots/:key now 302-redirects to /headshots/generate/:key
when the gender × race intersection is below 30 faces. ComfyUI
on-demand gives infinite uniqueness for the sparse buckets
(deterministic-per-worker via djb2 seed). Dense buckets still
serve from the pool — no GPU cost there.
3. STALE CACHE: Cache-Control was max-age=86400, immutable — pinned
old photos in browsers for 24h after any server-side update.
Dropped to max-age=3600, must-revalidate, and added a v=2
cache-buster query param to all front-end /headshots/ URLs so
existing cached entries are bypassed on next page load.
Also surfacing X-Face-Pool-Bucket / Bucket-Size headers for diagnosis.
Verified: playwright run shows surname routing correct (Torres,
Rivera, Alvarez, Gutierrez, Patel, Nguyen, Omar all bucketed
correctly), sparse buckets 302 to ComfyUI, dense buckets stay on
the thumb pool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
a3b65f314e |
Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs
Worker cards now ship a real photo per person instead of monogram tiles:
- fetch_face_pool.py pulls 1000 faces from thispersondoesnotexist.com
- tag_face_pool.py runs deepface for gender/race/age, excludes <22yo
- manifest.jsonl: 952 servable, gender/race buckets populated
- /headshots/_thumbs/ pre-resized to 384px webp (587KB -> 11KB,
60x smaller; without this Chrome's parallel-connection budget
drops ~75% of tiles in a 40-card grid)
- /headshots/:key gender x race x age intersection bucketing with
gender-only fallback when intersection is sparse
- /headshots/generate/:key ComfyUI on-demand for the contractor
profile spotlight (cold ~1.5s, cached ~1ms; worker-derived
djb2 seed makes faces deterministic-per-worker but unique
across workers sharing the same prompt)
- serve_imagegen.py _cache_key() now includes seed (was caching
by prompt only -> 3 different worker seeds collapsed to 1
cached image; verified fix produces 3 distinct md5s)
- confidence-default name resolution: Xavier->man+hispanic,
Aisha->woman+black, etc. Every worker resolves to a bucket.
End-to-end: playwright run on /?q=forklift+operators+IL -> 21/21
cards loaded, 0 broken, all 384px webp.
Cache + binary pool gitignored; manifest tracked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
10ed3bc630 |
demo: real synthetic headshots — fetch pool + serve route + UI wire
Three layers shipped:
1. SCRIPT — scripts/staffing/fetch_face_pool.py
Pulls N synthetic StyleGAN faces from thispersondoesnotexist.com
into data/headshots/face_NNNN.jpg, writes manifest.jsonl. Idempotent:
re-running skips existing files. Optional gender tagging via deepface
(currently unavailable on this box; the script handles ImportError
gracefully and tags everything as untagged). Fetched 198 faces with
concurrency=3 in ~67s.
2. SERVER — /headshots/:key route in mcp-server/index.ts
Loads manifest at first hit, caches in globalThis._faces. Hashes the
key with djb2-style mixing → pool index → returns the JPG. Same
key always gets the same face (deterministic). Accepts
?g=man|woman&e=caucasian|black|hispanic|south_asian|east_asian|middle_eastern
to bias pool selection — the gender/ethnicity buckets fall back to
the full pool when no tagged matches exist. Cache-Control:
86400 immutable so faces ride the browser cache after first hit.
/headshots/__reload re-reads the manifest without restart.
3. UI — search.html + console.html worker cards
Re-added overlay <img> on top of the monogram .av circle. img.src
= /headshots/<encoded-key>?g=<hint>&e=<hint>. img.onerror removes
the failed image so the monogram stays visible if the face pool
isn't fetched / CDN is blocked. .av now has overflow:hidden +
position:relative to clip the img to a perfect circle.
Forced-confident name resolution (J: "we're CREATING the profile,
created as though you truly have the information Xavier is more
likely Hispanic and he's a male"):
genderFor(name) — looks up MALE_NAMES + FEMALE_NAMES,
falls back to a deterministic hash split
so unknown names spread ~50/50. Sets now
include cross-cultural names: Alejandro/
Andres/Mateo/Santiago/Joaquin/Cesar/Hugo/
Felipe/Gerardo/Salvador/Ramon (Hispanic),
Raj/Anil/Vikram/Krishna/Pradeep (South
Asian), Wei/Yi/Hiroshi/Akira/Hyun (East
Asian), Demetrius/Kareem/DaQuan/Khalil
(Black), Omar/Khalid/Hassan/Ahmed/Bilal
(Middle Eastern). FEMALE_NAMES extended
in parallel.
guessEthnicityFromFirstName(name)
— confident default of 'caucasian' for any
name not in the cultural buckets so every
worker resolves to a category the face
pool can be biased toward. Order: ME → Black
→ Hispanic → South Asian → East Asian →
Caucasian (matters where names overlap,
e.g. Aisha appears in ME + Black, biases
toward ME for visual fit).
Both helpers also ported into console.html so the triage backfills
and try-it-yourself rendering get the same hint stack.
Privacy note in the script + route comments: the synthetic data uses
the worker's name as the seed; production should hash worker_id (not
name) to avoid leaking PII to a third-party CDN. The fetch URL itself
is referenced once per pool build, not per-worker.
.gitignore — added data/headshots/face_*.jpg (~100MB for 198 faces;
the manifest + script are tracked). Re-running the script on a fresh
checkout rebuilds the pool from scratch.
Verified end-to-end via playwright on devop.live/lakehouse:
forklift query → 10 worker cards
10/10 with face images (real synthetic headshots, not monograms)
0/10 broken
Alejandro G. Nelson → ?g=man&e=hispanic
Patricia K. Garcia → ?g=woman&e=caucasian
Each name → unique face, deterministic across loads.
Console triage backfills get the same treatment.
|
||
|
root
|
cdf5f5926a |
demo: console — sober worker cards (mirror dashboard styling)
J: "can you update Staffer's Console too the same look." Console
rendered worker rows in three places (Chapter 4 permit-contract
candidates, Chapter 8 triage backfills, Chapter 9 try-it-yourself
results) with the original 28px square avatar + flat backgrounds —
inconsistent with the new dashboard design.
Three changes:
1. CSS — .worker now has a 3px left-edge border that color-codes the
role family, and .av is a 32px circle with a muted dark background
+ 1px ring + monogram initials. Five role-band colors mirror
search.html: warehouse blue / production amber / trades purple /
driver green / lead orange. Plus a .role-pill style matching the
dashboard's small uppercase chip.
2. Helpers — added ROLE_BANDS regex table + roleBand() classifier and
a new workerRow(name, role, detail, opts) builder. Same regex
patterns as search.html so a "Forklift Operator" classifies
identically on every page. opts.endorsed adds the green endorsed
chip; opts.score appends a rank badge.
3. Replaced the three inline avatar+row constructors with workerRow()
calls. Net: console.html lost ~20 lines of duplicated DOM building
while gaining role bands + pills.
Verified end-to-end via playwright on devop.live/lakehouse/console:
Chapter 8 triage scenario "Marcus running late site 4422":
5 backfill rows render with [warehouse] band + WAREHOUSE pill +
monogram avatars (SBC, ETW, SHC, WMG, MEB).
Same sober look as the dashboard worker cards. No emojis, no
cartoons, color-coded role family on the left edge.
|
||
|
root
|
f92b55615f |
demo: worker cards — sober monogram avatars + role bands (no cartoons)
J: "It's two cartoonish right now the website looks like it was made
by first grade teacher." Pulled the DiceBear personas-style headshots
and the emoji role badges. They were generative-illustration playful;
this is supposed to read like a staffing tool, not a kindergarten
attendance sheet.
Replacement design — restraint, signal, no glyphs:
Avatar: 40px circle, monogram initials, muted dark background
(#161b22), 1px ring (#21262d), white-ish text. No image,
no emoji. Looks like a pre-photo placeholder slot in a
real ATS.
Role band: the role gets classified into one of five families:
WAREHOUSE / PRODUCTION / SKILLED TRADE / DRIVER / LEAD
(regex-based; falls back to the first word of the role
for unknown families). Each family has a single muted
color: blue / amber / purple / green / orange. The
color appears as:
- a 3px left border on the .iworker card
- a 2px left border + matching text color on a small
uppercase pill in the detail line
That's it. No images, no emojis, no per-role illustrations. The
staffer sees role-family at a glance via the band color, name and
initials prominently, full role + city + zip in the detail string
behind the pill. Five colors total instead of an eight-color rainbow.
CSS:
.iworker[data-role-band="warehouse"] etc. → 3px left border
.role-pill[data-rb="warehouse"] etc. → matching pill border
JS:
ROLE_BANDS = 6 regex → band+label entries (warehouse, production,
trades, driver, lead, quality)
roleBand(role) = first matching entry, fallback to first
word of role uppercased
Verified end-to-end via playwright on devop.live/lakehouse:
forklift query → 10 cards
every card → monogram avatar + WAREHOUSE pill (blue band)
no images, no emojis, no rainbow
Restart sequence after these edits:
pkill -9 -f "/home/profit/lakehouse/mcp-server/index.ts"
( setsid bun run /home/profit/lakehouse/mcp-server/index.ts \
> /tmp/mcp-server.log 2>&1 < /dev/null & disown )
|
||
|
root
|
d571d62e9b |
demo: spec — refresh repo layout + model fleet + per-staffer + paths 8-9
J: "how about devop.live/lakehouse/spec." Spec was anchored on
2026-04-21 state (v2 footer): mistral mentioned in the model matrix,
13 crates not 15, missing validator/truth/auditor crates, no mention
of OpenCode 40-model fleet, no pathway memory, no per-staffer
hot-swap, no Construction Activity Signal Engine, ADR count was 20.
Footer claimed Phases 19-25.
Edits, in order:
Ch1 Repository layout
+ crates/truth/ (ADR-021 rule store)
+ crates/validator/ (Phase 43 — schema/completeness/policy gates)
+ auditor/ (cross-lineage Kimi↔Haiku/Opus auto-promote)
+ scripts/distillation/ (frozen substrate v1.0.0 at
|
||
|
root
|
631b0329b1 |
demo: proof — full architecture-page rewrite for current state
J: "needs a rewrite." Old version was anchored on a dual-agent mistral+qwen2.5 loop that hasn't been the model story for weeks, called the system 13 crates (it's 15), referenced "Local 7B models" in the honest-limits section, and had no mention of: - the 40-model OpenCode fleet via one sk-* key - the 9-rung cloud-first ladder - N=3 consensus + cross-architecture tie-breaker - auditor cross-lineage (Kimi K2.6 ↔ Haiku 4.5, Opus auto-promote) - distillation v1.0.0 frozen substrate ( |
||
|
root
|
4c46cf6a21 |
demo: console — three new chapters reflecting recent shipments
J: "it's outdated." Console walkthrough was stuck on the original 6
chapters (legacy-bridge / permits / catalog / ranking demo / playbook
memory / try-it-yourself). Three weeks of new work weren't visible.
Three new chapters added between the existing playbook-memory chapter
and the input box; all pull live data from the running system:
Chapter 6 — Three coordinators, three views of the same corpus
Renders Maria/Devon/Aisha cards from /staffers with their
territories. Frames the per-staffer hot-swap as the relevance
gradient that compounds independently per coordinator. Same query
"forklift operators" returns 89 IN / 16 WI / 167 IL workers
depending on who's acting.
Chapter 7 — The hidden signal — public issuers in your contractor graph
Pulls /intelligence/profiler_index, builds the basket, shows
issuer count + attributed build value + contractor count as the
three top metrics. Lists top 8 issuers with attribution counts
and direct-link to the profiler. This is the BAI / Signal Engine
pitch in walkthrough form: every contractor name is also a forward
indicator on a public equity. Cross-metro replication explicit
in the closing paragraph.
Chapter 8 — When something breaks — triage in one shot
Live triage demo against /intelligence/chat with body
{message:"Marcus running late site 4422"}. Renders the worker
card + draft SMS + 5 backfills + duration_ms. The 250ms-vs-20min
moment, made concrete with real Quincy IL workers.
Chapter 9 (was 6) — Try it yourself
Updated input examples to demonstrate each new route:
"8 production workers near 60607" → headcount + zip parser
"Marcus running late site 4422" → triage handler
"Marcus" → bare-name lookup
"what came in last night" → temporal route
"reliable forklift operators with OSHA certs" → hybrid SQL+vector
Each is a click-to-run link beneath the input.
Two new accent classes: .accent-g (green for issuer-count) and
.accent-r (red for triage event).
Verified end-to-end on devop.live/lakehouse/console: 9 chapters
render, ch6 shows 3 staffer personas, ch7 shows 11 issuers / $347M /
200 contractors, ch8 shows Marcus V. Campbell + draft SMS + 5
backfills.
|
||
|
root
|
6366487b45 |
ops: persist runtime fixes — iterate.rs unused state, catalog cleanup
Two load-bearing runtime changes that were never committed: 1. crates/gateway/src/v1/iterate.rs — `state` → `_state` on the unused route-state parameter. Cleared the one cargo workspace warning. Fix was made earlier this session but the working-tree change never made it into a commit. 2. data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json — DELETED. This was the dead manifest for `client_workerskjkk`, a typo dataset whose parquet was deleted but whose catalog entry stayed registered. Every SQL query failed schema inference on the missing file before reaching its target table — that's the bug that made /system/summary report 0 workers and the demo show zero bench. Deleting the manifest keeps the fix on disk; committing the deletion keeps it in git so a fresh checkout doesn't regress. 3. data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json — runtime catalog metadata update from the successful_playbooks_live write path. Ride-along change. Reports under reports/distillation/phase[68]-*.md are auto-regenerated by the audit cycle each run; skipping those. |
||
|
root
|
db81fd8836 |
demo: System Activity panel — capability index reflects every recent shipment
Old panel showed playbook ops + search counts and went empty in a
fresh demo (no operations yet). J: "update System Activity to coincide
with all of our recent updates."
Rebuilt as a live capability index — each tile is a thing the
substrate has learned to do, with the metric proving it's running.
Pulled in parallel from /staffers, /system/summary,
/api/vectors/playbook_memory/stats, /api/vectors/pathway/stats,
/intelligence/profiler_index, /intelligence/activity. Each probe
catches its own error so a single missing endpoint doesn't collapse
the panel.
Nine capability cards (verified end-to-end on devop.live/lakehouse):
1. Per-staffer hot-swap index 3 personas (Maria/Devon/Aisha)
2. Construction Activity Signal Engine 11 issuers · $347M attributed
build value · network 11/14
3. Late-worker / no-show triage one-shot — name+late → backfills+SMS
4. Permit → staffing bridge 24/day, every Chicago permit ≥$250K
5. Hybrid SQL + vector search 500K workers · 5,474 playbook entries
6. Schema-agnostic ingestion 36 datasets · 2.98M rows
7. Contractor profile + project index 6 wired · 12 queued sources
8. Pathway memory 88 traces · 11/11 replays · 100%
9. Ticker association network 11 tickers · 3 direct + 11 associated
Each card carries:
- capability title + ship date pill ("baseline" or "shipped 2026-04-27")
- big metric (live, not pre-baked)
- sub-context line in coordinator language
- "why a staffer cares" explanation
- optional "Open →" deep link to the surface (Profiler, Contractor)
Header + intro paragraph reframed: "what the substrate has learned to
do" instead of "what the substrate has learned." Operational learning
(fills, playbooks, hot-swaps) compounds INSIDE each capability; the
panel surfaces the set of capabilities the corpus knows how to express.
Closing operational-stats row at the bottom shows fills/searches/
recent playbooks when /intelligence/activity has any.
|
||
|
root
|
a789000982 |
demo: profiler — Construction Activity Signal Engine narrative + BAI
J's prompt: shoot for the stars, frame the data corpus's value as a
predictive signal, not just a contractor directory. The thesis is
that every name in this corpus is also a forward indicator on public
equities — permits filed today predict construction starts in ~45
days, staffing in ~30, revenue recognition months later. The
associated-ticker network surfaces this signal before any 10-Q does.
Two new layers above the basket:
1. HERO THESIS PANEL — "Chicago Construction Activity Signal Engine"
header + 3-line value statement, then 4 live metrics:
- BAI (Building Activity Index) — attribution-weighted average of
day-change % across surfaced issuers. Weight = attribution count
so issuers we have more depth on count more. Today: +0.76%
(9 issuers · top contributors FCBC +2.4%, ACRE +1.7%, JPM +1.5%).
Color-coded green/red.
- Indexed build value — total $ of permits attributable to ANY
public issuer in this view. Today: $344M.
- Network depth — issuers / attribution edges. Today: 9 / 15.
This is the "we see what nobody else sees" metric: how many
contractors are bridges from a private builder back to a public
equity holder.
- Market replication roadmap — chips showing "Chicago — live ·
NYC DOB — adapter ready · LA County · Houston BCD · Boston ISD
· DC DCRA". Frames the corpus as metro-agnostic from day one.
2. PER-TICKER ACTIVITY MAP — when a basket card is clicked, a leaflet
map appears below the basket plotting that ticker's geocoded permit
activity. Pulls /intelligence/contractor_profile for up to 6
attributed contractors, merges their geocoded permits, plots on a
dark Chicago tile layer. Color-banded by permit cost (green <$100K,
amber $100K-$1M, red ≥$1M). Click TGT → 23 Target permits across
Chicago; click JPM → JPMorgan-adjacent contractor activity. Cached
per ticker so toggling is instant.
Verified end-to-end on devop.live/lakehouse/profiler:
Default load: hero panel renders with all 4 metrics, basket strip
with 9 issuers + live prices in 669ms.
Click TGT : signal map activates, "23 geocoded permits across
1 contractor", table filters to 2 rows.
Tooltip on basket cards: full reason path including matched name +
contributors attributed to that ticker.
Architecture-side: zero new server code — all metrics computed
client-side from the existing profiler_index + ticker_quotes payloads.
The corpus already had the value; the page just needed to articulate it.
|
||
|
root
|
aa56fbce61 |
demo: profiler — scrolling ticker basket with live prices + click-to-filter
J asked: "kind of like a scrolling ticker that has all of the companies
and their stock prices and where they fit in the map." Implemented as
a horizontal-scroll strip at the top of /profiler:
9 public issuers in this view · quotes via Stooq · 669ms
┌────┬────┬────┬────┬────┬────┬────┐
│TGT │JPM │BALY│ACRE│FCBC│NREF│LSBK│ ← live price + day-change per
│129 │311 │... │... │... │... │... │ ticker, color-banded by
│+.17│+1.5│... │... │... │... │... │ attribution kind
└────┴────┴────┴────┴────┴────┴────┘
Each card carries:
- ticker + live price + day-change % (red/green)
- attribution count + kind (exact / direct / parent / associated)
- left bar color = strongest attribution kind (green for direct
issuer, amber for parent, blue for co-permit associated, gradient
when both direct and associated apply)
- tooltip on hover lists the contractors attributed to this ticker
- click toggles a filter on the table below — clicking TGT cuts the
200-row list down to just TARGET CORPORATION + TORNOW, KYLE F
(Target's primary co-permit contractor)
Server-side:
- entity.ts exports fetchStooqQuote (was internal)
- new POST /intelligence/ticker_quotes — accepts {tickers: [...]},
fans out to Stooq.us in parallel, returns
{ticker, price, price_date, open, high, low, day_change_pct,
stooq_url} per symbol or null for non-US listings (HOC.DE, SKA-B.ST,
LLC.AX). Capped at 50 symbols per call.
Front-end:
- mcp-server/profiler.html — new .basket-wrap section above the
controls. buildBasket() runs after profiler_index loads:
1. Aggregates unique tickers from .tickers.direct + .associated
across all surfaced contractors
2. Renders shells immediately (ticker symbol + "—" placeholder)
3. Batch-fetches quotes via /intelligence/ticker_quotes
4. Updates each card with price + day-change in place
Click on a card sets a tickerFilter; render() skips rows whose
attributions don't include that ticker. "clear filter" button on
the basket strip resets it.
Verified end-to-end on devop.live/lakehouse/profiler:
Default load → 9 issuers, live prices populated in 669ms
TGT click → table filters to TARGET CORPORATION + TORNOW, KYLE F
(the contractor who runs 3 of Target's recent permits
gets the TGT correlation indicator)
JPM card → $311.63, +1.55% — JPMorgan-adjacent contractors
Tooltip → list of contractors attributed to the ticker
|
||
|
root
|
ba41ad2846 |
demo: profiler index — ticker associations (direct, parent, co-permit)
J's framing: "if a contractor works for Target, future Target contracts
mean money flows back to the contractor — the ticker is an associated
indicator." Now the profiler index attaches three flavors of ticker per
contractor and renders them as colored pills:
green DIRECT contractor IS the public issuer (Target Corp → TGT)
amber PARENT contractor is a subsidiary of a public parent
(Turner Construction → HOC.DE via Hochtief AG)
blue ASSOCIATED contractor co-appears on permits with a public
entity (TORNOW, KYLE F → TGT, 3 shared permits with
TARGET CORPORATION)
The associated flavor is the correlation signal J described — it pulls
the ticker for whoever the contractor has been working *with*, not
just what they are themselves. Most contractors are private; the
associated link is how the moat shows up.
Server-side:
- entity.ts new export `lookupTickerLite(name)` — cheap in-memory
resolver that does only the SEC tickers index lookup + curated
KNOWN_PARENT_MAP check, no per-call SEC profile or Stooq fetch.
~10ms per name after the index is loaded once.
- /intelligence/profiler_index now runs a third Socrata pull
(5K permit pairs in window) to build a co-occurrence map. For each
contractor in the result, attaches:
.tickers.direct[] — name matches a public issuer
.tickers.associated[] — top 5 co-permit partners that resolve
to a ticker, with partner_name +
co_permits count + partner_via reason
Front-end:
- mcp-server/profiler.html — new .ticker-pill styles (3 colors per
attribution kind), pills render under the contractor name in the
table. Hover title gives the full reason path.
Verified end-to-end on the public URL:
search="tornow" → blue TGT pill, hint "Associated via co-permits
with TARGET CORPORATION (3 shared permits) —
TARGET CORP"
search="target" → green TGT × 2 (TARGET CORPORATION +
CORPORATION TARGET name variants both resolve
direct to the same issuer)
default top 200 → 15 ticker pills surface across the page including
JPM (via JPMORGAN CHASE BANK co-permits) and
parent-link tickers for the construction majors.
|
||
|
root
|
f6a7621b2d |
demo: profiler index — directory of every Chicago contractor
J asked for "a profiler index that shows a history of everyone." This
is a /profiler directory page (also reachable via /contractors) that
ranks every contractor who's filed a Chicago permit, by total permit
value. Rows are clickable into the full /contractor profile.
Defaults: since 2025-06-01, min permit cost $250K, top 200 contractors
by total_cost. Server pulls two Socrata GROUP BY queries (one keyed on
contact_1_name, one on contact_2_name), merges them so contractors
listed in either applicant or contractor slot appear once with combined
counts/cost. ~300ms cold.
UI: live search box, since-date selector, min-cost selector, sortable
columns (name / permits / total_cost / last_filed). Live numbers as of
this write: 200 contractors, 1,702 permits, $14.22B aggregate. Filter
"Target" returns TARGET CORPORATION + CORPORATION TARGET (name variants
from Socrata).
Also fixes J's other complaint — "no new contracts, Target is gone":
/intelligence/permit_contracts was hard-capped at $limit=6 + only
the most recent 6 over $250K, so any day with 6 fresh permits would
push older contractors (Target) off the panel entirely. Now defaults
to 24 (caller can pass body.limit up to 100), so 2-3 days of permits
stay on the panel. Added body.contractor — passes a name into the
WHERE so the staffer can pin a specific contractor to the panel
("Target Corporation" → 3 of their permits over $250K).
Server-side:
- new POST /intelligence/profiler_index — paginated contractor index
(since, min_cost, search, limit) with merged contact_1+contact_2
aggregations
- /intelligence/permit_contracts — body.limit + body.contractor
- /profiler and /contractors routes serve profiler.html
Front-end:
- new mcp-server/profiler.html — sortable table, live filter, deep
links to /contractor?name=... (prefix-aware via P, so /lakehouse
works on devop.live)
- search.html + console.html nav: added "Profiler" link
Verified end-to-end via playwright on the public URL.
|
||
|
root
|
31d8ef918c |
demo: contractor links — respect the /lakehouse path prefix
J reported https://devop.live/contractor?name=3115%20W%20POLK%20ST.%20LLC returned 404. Cause: the anchor href was a bare /contractor, which on devop.live routes to the LLM Team UI (port 5000) at the main site root, not the lakehouse mcp-server (which lives under /lakehouse/*). Every page that renders a contractor link now uses the same prefix detector the dashboard already had: var P = location.pathname.indexOf('/lakehouse') >= 0 ? '/lakehouse' : ''; Files updated: - search.html: entity-brief anchor + preview anchor → P+/contractor - console.html: permit-card contractor list → P+/contractor - contractor.html: history.replaceState + back-link + the /intelligence/contractor_profile fetch all use P prefix. The page is reachable at /lakehouse/contractor on the public URL and bare /contractor on localhost; both work without further config. Verified: https://devop.live/lakehouse/contractor?name=3115%20W%20POLK%20ST.%20LLC → 200, 29.9 KB, full profile renders. Contractor has 1 permit on file (a small LLC), 1 geocoded so the heat map plots one marker. |
||
|
root
|
a1066db87b |
demo: contractor profile — heat map, project index, 12 awaiting sources
The contractor.html click-target J asked for: a separate page (not a
modal, not a fall-through search) showing every angle on a contractor.
Reachable from the Co-Pilot dashboard, the staffers console, and the
search box — all anchor-wrap contractor names to /contractor?name=...
What's new on the page:
1. PROJECT INDEX — build-signal score
Single 0-100 number with the drivers laid out beneath. Driver list
is staffer-readable: "59 Chicago permits in 180d (+30) · OSHA 20
inspections (-25) · federal contractor (+15)". Score weights are
placeholders to be replaced by an ML model once the 12 awaiting
sources ship — the current 6 wired signals would not give a real
model enough features.
2. HEAT MAP — every Chicago permit they've been contact_1 or contact_2
on, last 24 months, plotted on a leaflet dark map. Color by cost
(green <$100K, amber $100K-$1M, red ≥$1M), radius proportional to
cost so the staffer sees where money + activity concentrates. Click
a marker for permit detail (cost, date, work type, address, permit
ID). All 50 of Turner Construction's geocoded recent permits in
Chicago plot end-to-end.
3. ACTIVITY TIMELINE — monthly permit count, bar chart, with the
first/last month labels so the staffer sees momentum. Tooltip on
each bar gives the count and total cost for that month.
4. 12 AWAITING SOURCES — placeholder cards for the public datasets
that would 3× the build-signal feature count. Each card has:
- source name (real, e.g. DOL Wage & Hour, EPA ECHO, MSHA, BBB)
- one-liner in coordinator language ("Has this contractor stiffed
workers? Will they pay our staffing invoices?")
- "Would show:" sample shape so the engineering scope is concrete
Order is staffing-decision relevance:
1. DOL Wage & Hour (WHD violations)
2. State Licensure Boards (active license + expiry)
3. Surety Bond Capacity (bonding ceiling)
4. EPA ECHO Compliance (env violations at sites)
5. DOT/FMCSA Carrier Safety (crash + OOS rates)
6. BBB Complaints + Rating
7. PACER Civil Suits (FLSA / Title VII / ADA)
8. UCC Lien Filings (cash flow distress)
9. D&B / Credit Bureau (PAYDEX, payment behavior)
10. State UI Employer Claims (workforce stability)
11. MSHA Mine Safety (excavation / aggregate / heavy)
12. Registered Apprenticeships (DOL RAPIDS pipeline)
Server-side: entity.ts fetchContractorHistory now pulls the 50 most
recent permits with id + lat/lng + work_description, so the heat map
and timeline have what they need without a second SQL hop. The
ContractorHistory.recent_permits type gained the optional fields.
Front-end: contractor.html got 4 new render sections, leaflet wiring
(stylesheet + script in head), placeholder grid CSS, and a PLACEHOLDERS
const at the bottom with the 12 sources. All popup HTML is built via
DOM construction (textContent + appendChild) — no innerHTML, no XSS.
console.html: contractor names from /intelligence/permit_contracts now
anchor-wrapped to /contractor?name=... so the click-through J described
works from the staffers console too. Click stops propagation so the
permit details element doesn't toggle on the same click.
Verified end-to-end via playwright — Turner Construction profile shows:
PIX score "Mixed signals — review drivers below"
Heat map: "50 permits plotted · green/amber/red"
4 section labels in order
12 placeholder cards in the documented order
|
||
|
root
|
5f0beffe80 |
demo: G — per-staffer hot-swap index (synthetic coordinator personas)
Same corpus, different relevance gradient per staffer. Three personas defined in mcp-server/index.ts STAFFERS roster (Maria/IL, Devon/IN, Aisha/WI), each with a primary state + secondary cities. Server-side: /intelligence/chat smart_search accepts a staffer_id body field; when set, defaults state to the staffer's territory and labels the playbook context as theirs. The playbook patterns query also defaults its geo to the staffer's primary city/state, so the recurring-skills/cert breakdowns reflect what they actually fill, not the global IL prior. Front-end: a staffer selector dropdown beside the existing state/role filters. Picking a staffer auto-pins state to their territory, shows a greeting line, relabels the MEMORY panel as MARIA'S/DEVON'S/AISHA'S MEMORY, and sends staffer_id to chat for scoping. Dropdown is populated from /staffers (NOT /api/staffers — the generic /api/* passthrough sends everything under /api/ to the Rust gateway, which doesn't own the roster). loadStaffers runs at window-load independently of loadDay's Promise.all so the dropdown populates even if simulation/SQL inits error out. Verified end-to-end via playwright. Same q="forklift operators": no staffer → 509 workers across MI/OH/IA, MEMORY label as Devon → 89 IN-only (Fort Wayne, Terre Haute), DEVON'S MEMORY as Aisha → 16 WI-only (Milwaukee, Madison, Green Bay), AISHA'S MEMORY As Maria with q="8 production workers near 60607": tags: headcount: 8 · zip 60607 → Chicago, IL · role: production · city: Chicago 20 workers, MARIA'S MEMORY label, top results in Chicago zips Closes the demo-side build of A-G from the persona plan: A. zip → city/state, B. headcount, C. bare-name, D. temporal, E. late-worker triage, F. contractor anchor, G. per-staffer index. |
||
|
root
|
677065de76 |
demo: P2 — staffer-language routes (zip, headcount, name, late-triage, ingest log)
Built from a playwright run as three personas:
Maria — "8 production workers near 60607 by next Friday, prior-fill at this client"
Devon — "what came in last night?"
Aisha — "Marcus running late site 4422"
Each one previously fell through to smart_search and returned irrelevant
results (geo wrong, headcount ignored, no triage, no temporal). Now:
A. Zip code → city/state lookup. Chicago zips (606xx, 607xx, 608xx)
resolve to {city: Chicago, state: IL}; 13 metro prefixes covered.
Maria's "near 60607" now returns Chicago workers, not Dayton/Green Bay.
B. Headcount parser. "8 production workers" / "12 forklift operators" /
"5 welders" set top_k 1..200, capped 5..25 for SQL+vector LIMIT.
Allows 0-2 role words between the count and the worker noun so
"8 production workers" matches as well as "8 workers".
C. Bare-name profile lookup. Single short capitalized phrase
("Marcus" / "Sarah Lopez") triggers a profile route. Per-token LIKE
AND-joined so "Marcus Rivera" matches "Marcus L. Rivera" without
hardcoding middle initials.
E. Late-worker / no-show triage. Pattern: <Name> (running late|late|
no show|sick|out today|called out|can't make it) — pulls profile +
reliability + responsiveness + recent calls, sources 5 same-role
same-geo backfills sorted by responsiveness, drafts a client SMS
the coordinator can copy. Front-end renders triage card + Copy SMS
button + green backfill list.
F. Contractor name preview anchor. The PROJECT INDEX preview line on
each permit card now wraps contact_1_name and contact_2_name in
anchors to /contractor?name=... — clicking a contractor finally
navigates instead of doing nothing. Click handler stops propagation
so the details element doesn't toggle.
D. Temporal "what came in" route. last night / today / past N hours /
recent — surfaces datasets from the catalog whose updated_at is
within the window, samples one row per dataset to detect worker-
shape, groups by role for worker tables. Schema-agnostic — drop
any dataset and it shows up. Currently sparse because no fresh
ingest has happened today; will populate as ingest runs.
Server: /intelligence/chat smart_search route accepts structured
state/role from the search-form dropdowns (P1 from prior commit) and
now ALSO honors b.state, b.role, q.match for headcount + zip + name +
triage patterns BEFORE falling through to NL parsing.
Front-end: doSearch dispatches on response.type and renders triage,
profile, ingest_log, and miss states with type-specific UI. All DOM
construction uses textContent / appendChild — no innerHTML, no XSS.
Verified end-to-end via playwright drive of devop.live/lakehouse:
Maria → 8 Chicago Production Workers (60685, 60662, 60634)
tags: "headcount: 8 · zip 60607 → Chicago, IL · ..."
Aisha → Marcus V. Campbell card + draft SMS + 5 Quincy IL backfills
"I'm dispatching Scott B. Cooper (96% reliability) to cover."
Devon → ingest_log surfaces successful_playbooks_live (last 1h)
Marcus → 5 profiles (Adams Louisville KY, Jenkins Green Bay WI, ...)
Screenshots: /tmp/persona_v2/{01_maria,02_aisha,03_devon,04_marcus}.png
Restart sequence after these edits: pkill -9 -f "mcp-server/index.ts" ;
cd /home/profit/lakehouse ; bun run mcp-server/index.ts. The bun on
:3700 is not systemd-managed (pre-existing convention).
|
||
|
root
|
fb99e92a60 |
demo: P1 — search filter now actually filters by state and role
The Co-Pilot search box read state and role from the dropdowns (#sst, #srl)
but appended them to the message string as ' in '+st. The server's NL
parser then matched the literal preposition "in" against the case-insensitive
regex /\b(IL|IN|...)\b/i and assigned state IN (Indiana) to every search.
Result: typing "forklift in IL" returned Indiana workers. Same for WI, TX,
any state — all silently became Indiana. That was the "cached/generic
response" the legacy staffing client was seeing.
Two prongs:
1. search.html doSearch() now passes structured fields:
{message, state, role}
instead of munging into the message text. Dropdown selections bypass
NL parsing entirely.
2. /intelligence/chat smart_search route accepts those structured fields
and prefers them over regex archaeology. Falls back to NL parsing only
when fields aren't provided. Fixed the regex too: the prepositional
form (?:in|from)\s+(STATE) wins, the standalone form requires uppercase
(drops /i flag) so the lowercase preposition "in" can no longer match.
Verified live:
- POST /intelligence/chat {"message":"forklift","state":"IL"}
→ 167 IL forklift operators (Galesburg, Joliet, ...)
- POST /intelligence/chat {"message":"forklift","state":"WI","role":"Forklift Operator"}
→ 16 WI Forklift Operators (Milwaukee, Madison, ...)
- POST /intelligence/chat {"message":"forklift in IL"} (NL fallback)
→ 167 IL workers (regex now correctly distinguishes preposition from state code)
Playwright drove the live UI through devop.live/lakehouse and confirmed the
front-end posts the structured body and the result panel renders the right
state. Restart sequence: kill old bun :3700, bun run mcp-server/index.ts.
|
||
|
root
|
c3c9c2174a |
staffing: B+C — safe views (candidates/workers/jobs) + workers_500k_v9 build script
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
Decision B from reports/staffing/synthetic-data-gap-report.md §7 (plus C: client_workerskjkk.parquet typo file removed from data/datasets/ — was never tracked, no git effect). PII enforcement was UNVERIFIED in workers_500k_v8 (the corpus staffing_inference mode embeds chunks from). Verified 2026-04-27 by inspecting data/vectors/meta/workers_500k_v8.json — `source: "workers_500k"` confirms v8 was built directly from the raw table, so the LLM has been seeing names / emails / phones / resume_text for every staffing query. This commit closes the boundary at the catalog metadata layer: candidates_safe (overhauled — was failing SQL invalid 434×/day on a nonexistent `vertical` column reference, copy-pasted from job_orders): drops last_name, email, phone, hourly_rate_usd candidate_id masked (keep first 3, last 2) row_filter: status != 'blocked' workers_safe (NEW): drops name, email, phone, zip, communications, resume_text keeps role, city, state, skills, certifications, archetype, scores resume_text + communications carry verbatim PII (full names) and there is no in-view text scrubber, so they are dropped wholesale. Skills + certifications + scores carry the matching signal for staffing inference. jobs_safe (NEW): drops description (often quotes client names verbatim) client_id masked (keep first 3, last 2) bill_rate / pay_rate kept — commercial info, not PII per staffing PRD scripts/staffing/build_workers_v9.sh (NEW): POSTs /vectors/index to rebuild workers_500k_v9 from `workers_safe` rather than the raw table. Embedded text is constructed from the view projection so PII never enters the corpus by construction. 30+ minute background job — not run inline. After it completes, flip config/modes.toml `staffing_inference` matrix_corpus from workers_500k_v8 to workers_500k_v9 and restart gateway. Distillation v1.0.0 substrate untouched. audit-full passed clean (16/16 required) before this commit; will re-verify after. |
||
|
root
|
940737daa7 |
staffing: D — workers_500k.phone int → string fixup script
Decision D from reports/staffing/synthetic-data-gap-report.md §7. Phones in workers_500k.parquet are 11-digit US numbers stored as int64 (e.g. 13122277740). Numerically fine, but breaks join keys against any other source that carries phone as string. Script casts the column to string in place, with non-destructive backup at data/datasets/workers_500k.parquet.bak-<date> before write. Idempotent: if phone is already string, exits 0 with "no-op". Safe to re-run. The .parquet itself is too large to commit (75MB) and follows project convention of staying out of git. The script makes the conversion reproducible from the source dataset. |
||
|
root
|
d56f08e740 |
staffing: A — fill_events.parquet from 44 scenarios + 64 lessons (deterministic)
Decision A from reports/staffing/synthetic-data-gap-report.md §7. Walks tests/multi-agent/scenarios/scen_*.json and data/_playbook_lessons/*.json, normalizes to a single fill_events.parquet at data/datasets/fill_events.parquet. One row per scenario event, lesson outcomes joined by (client, date) where the tuple matches. rows: 123 scenarios contributing: 40 events with outcome data: 62 unique (client, date) tuples: 40 Reproducibility: event_id is SHA1(client|date|role|at|city) truncated to 16 hex chars; rows sorted by event_id before write so re-runs produce bit-identical output. Verified. Pure normalization — no LLM, no new data, no distillation substrate mutation. |
||
|
rootandClaude Opus 4.7
|
ca7375ea2b |
auditor: layer-2 path-traversal guard — symlink resolution before read
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
Kimi's audit on |
||
|
rootandClaude Opus 4.7
|
2d9cb128bf |
auditor: BLOCK fix from kimi_architect on dd77632 — path-traversal guard
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
The grounding step in computeGrounding() resolves model-provided
file:line citations against REPO_ROOT and reads the file. Pre-fix:
no check that the resolved path stays inside REPO_ROOT. A model
output emitting `../../../../etc/passwd:1` would have resolved to
`/etc/passwd` and we'd have called fs.readFile() on it.
Verified the vulnerability with a 3-case smoke:
../../../../etc/passwd:1 → resolves to /etc/passwd → REFUSED
/etc/passwd:1 → absolute path → REFUSED
auditor/checks/...:1 → repo-relative → ALLOWED
Fix: after resolve(REPO_ROOT, relpath), require the absolute path
starts with `REPO_ROOT + "/"` (or equals REPO_ROOT exactly).
Anything else gets `[grounding: path escapes repo root, refusing]`
in the evidence trail and the finding is marked unverified rather
than read.
Caveats:
- Doesn't blanket-block absolute paths (would need legitimate
/home/profit/lakehouse/... citations to work). Only escapes get
rejected, regardless of how they were specified.
- Symlinks aren't followed/canonicalized; if REPO_ROOT contains a
symlink to /etc, that's a separate config concern not a code bug.
Verification:
bun build auditor/checks/kimi_architect.ts compiles
Resolution-only smoke (3 cases) all expected
Daemon will pick up the fix on next push (auto-reset fires)
This was the only BLOCK in the
|
||
|
rootandClaude Opus 4.7
|
dd77632d0e |
auditor: 2 BLOCK fixes from kimi_architect on a50e9586 audit
lakehouse/auditor 10 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
Lands 2 of the 3 BLOCKs from the auto-reset commit's audit: 1. static.ts:67-130 — backtick state-machine ordering `inMultilineBacktick` was updated AFTER pattern checks ran on a line, so any block-pattern hit on a line that opened a backtick block was evaluated under stale "outside-backtick" semantics. Net effect: false-positive BLOCK findings on hardcoded-string patterns sitting inside multi-line template literals (where they are legitimately quoted, not executed). Fix: compute state-at-line-start BEFORE pattern checks; carry state-at-line-end forward for the next iteration. Pattern checks now use `stateAtLineStart` consistently. 2. static.ts:223-228 — parentStructHasSerdeDerive bounds check The function walked backward from `fieldLineIdx` without validating it against `lines.length`. If a malformed diff fed in an out-of-range fieldLineIdx, the loop's implicit upper bound (`fieldLineIdx - 80`) could still be > 0, leading to undefined- slot reads or silently wrong results. Fix: defensive bail (`if (fieldLineIdx < 0 || >= lines.length) return false`) before the loop runs. SKIPPED with rationale: - BLOCK on types.ts:96 (requireSha256 "optional-chaining bypass") Investigated: requireString correctly catches null/undefined/object via `typeof !== "string"`; the call site at line 96 is just an invocation of the function defined at line 81-88. The full code paths (null, undefined, object, short string, valid hex) all produce correct error/success outcomes. Kimi's rationale was truncated at 200 chars; no bypass found in the actual code. Treating as a confabulation. Verification: bun build auditor/checks/static.ts compiles Daemon restart needed to activate; auto-reset cap will fire [1/3] on the new SHA. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
a50e9586f2 |
auditor: cap auto-resets on new head SHA (was per-PR-forever, now per-push)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
Operator feedback: manual jq-edit-state.json + restart isn't
sustainable. Each push should naturally get a fresh budget; old
counter discarded the moment the SHA moves. Cap intent shifts
from "PR exhaustion" to "per-push attempt limit" — bounded
recovery from transient upstream errors, not a forever limit.
Mechanism:
- The dedup branch above (`last === pr.head_sha → continue`)
unchanged.
- New branch: when `last` exists AND we have a non-zero count,
AND we've fallen through to here (which means SHA != last,
i.e. a new push), drop the counter to 0 BEFORE the cap check.
- Cap check fires only on same-SHA retries (transient errors that
consumed multiple attempts).
Net behavior:
- push code → 3 audits run → cap → quiet → push more code →
cap auto-resets → 3 more audits → cap → quiet
- No manual jq ever needed in steady state.
- Operator clears state.audit_count_per_pr.<N> = 0 only if a
single SHA somehow needs MORE than the cap.
Pre-existing manual reset still works (state edit + daemon
restart for the change to take effect). Documented in the new
log line that fires on the rare same-SHA-burned-cap case.
Verified compile (bun build auditor/index.ts → green). Daemon
restart needed to activate; current cycle 4616's `[1/3]` audit
on
|
||
|
rootandClaude Opus 4.7
|
6ed48c1a69 |
gateway+validator: /v1/health reports honest worker count for production
lakehouse/auditor 12 blocking issues: cloud: claim not backed — "Verified live (current synthetic data):"
Adds `fn len() -> usize` (default 0) to the WorkerLookup trait. The InMemoryWorkerLookup overrides with HashMap size; ParquetWorkerLookup constructs an InMemoryWorkerLookup so it inherits the count. /v1/health now reports `workers_count` (exact integer) alongside `workers_loaded` (derived bool: count > 0). The previous placeholder true was a known caveat in the prior commit's body — this closes it. Production switchover use case: J swaps workers_500k.parquet → real Chicago contractor data, restarts the gateway, and verifies the swap with one curl: curl http://localhost:3100/v1/health | jq .workers_count Expected: matches the row count of the new file. Mismatch (or 0) means the file is missing / unreadable / had a schema mismatch and the gateway fell back to the empty InMemoryWorkerLookup. Operator catches the drift before traffic reaches the validators. Verified live (current synthetic data): workers_count: 500000 (matches workers_500k.parquet row count) workers_loaded: true When the Chicago data lands, the same curl is the single source of truth that the new dataset is hot. Removes the restart-and-pray failure mode. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
74ad77211f |
gateway: /v1/health — production operational status endpoint
Adds GET /v1/health that returns a JSON snapshot of subsystem state
so operators (and load balancers, and the lakehouse-auditor
service) can verify the gateway is fully booted before routing
traffic. Phase 42-45 closures are now production-deployable; this
endpoint is the canary that proves it.
Returns 200 always — fields are observed-state, not pass/fail
gates. Monitoring tools evaluate the booleans + counts against
their own thresholds.
Shape:
{
"status": "ok",
"workers_loaded": bool,
"providers_configured": {
"ollama_cloud": bool, "openrouter": bool, "kimi": bool,
"opencode": bool, "gemini": bool, "claude": bool,
},
"langfuse_configured": bool,
"usage_total_requests": N,
"usage_by_provider": ["ollama_cloud", "openrouter", ...]
}
Verified live:
curl http://localhost:3100/v1/health
→ 4 providers configured (kimi, ollama_cloud, opencode, openrouter)
→ 2 not configured (claude, gemini — keys not wired)
→ langfuse_configured: true
→ workers_loaded: true (500K-row workers_500k.parquet snapshot)
Caveat: workers_loaded is a placeholder true — WorkerLookup trait
doesn't have a len() method yet, so we can't honestly report row
count from the runtime probe. The boot log line "loaded workers
parquet snapshot rows=N" is the source of truth on count. Future
follow-up: add `fn len(&self) -> usize` to WorkerLookup so /v1/health
can report the exact figure.
Pre-production checklist context: J flagged production switchover
incoming — synthetic profiles will be replaced with real Chicago
data soon. /v1/health gives the operator a single curl to verify
the gateway sees the new data after the parquet swap (boot log +
this endpoint).
Hot-swap reload (POST /v1/admin/reload-workers) deferred to a
follow-up — requires V1State.validate_workers to wrap in RwLock
or ArcSwap so write traffic doesn't block the steady-state
read path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
2cac64636c |
docs: PHASES tracker — mark Phases 42/43/44/45 complete
Today's work shipped four Phase closures (Truth Layer, Validation Pipeline, Caller Migration, Doc-Drift Detection); the canonical tracker now reflects them. Foundation for production switchover (real Chicago data replaces synthetic test data soon). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
6cafa7ec0e |
vectord: Phase 45 closure — /doc_drift/scan + doc_drift_corrections.jsonl writes
Phase 45 (doc-drift detection + context7 integration) was mostly
already shipped in prior sessions: DocRef struct, doc_drift module,
/doc_drift/check + /doc_drift/resolve endpoints, mcp-server's
context7_bridge.ts, boost exclusion in compute_boost_for_filtered
_with_role. The two missing pieces this commit lands:
1. POST /vectors/playbook_memory/doc_drift/scan — batch scan across
ALL active playbooks. Iterates the snapshot, filters out retired
+ already-flagged + no-doc_refs, runs check_all_refs on the rest,
flags drifted entries via PlaybookMemory::flag_doc_drift.
2. Per-detection write to data/_kb/doc_drift_corrections.jsonl. One
row per drifted playbook with playbook_id + scanned_at +
drifted_tools[] + per_tool[] + recommended_action. Downstream
consumers (overview model, operator dashboard, scrum_master
prompt enrichment) read this file to surface "this playbook
compounded the wrong way" signals to humans.
Idempotent by design:
- Already-flagged entries with no resolved_at are counted as
`already_flagged` and skipped (no double-flag, no duplicate row).
- Re-scanning after resolve_doc_drift() unflags an entry brings it
back into the eligible set on the next scan.
Aggregate response shape:
{
"scanned": N, // playbooks with doc_refs we checked
"newly_flagged": N, // drift detected this scan
"already_flagged": N, // skipped (still under review)
"skipped_retired": N,
"skipped_no_refs": N, // pre-Phase-45 playbooks
"drifted_by_tool": {tool: count},
"corrections_written": N,
}
Verified live:
POST /doc_drift/scan
→ scanned=4, newly_flagged=4, drifted_by_tool={docker:4, terraform:1},
corrections_written=4
POST /doc_drift/scan (re-run)
→ scanned=0, newly_flagged=0, already_flagged=6 (idempotent)
data/_kb/doc_drift_corrections.jsonl
→ 5 rows total (existing seed + this scan)
Phase 45 closure status:
DocRef + PlaybookEntry.doc_refs ✅ prior session
doc_drift module + check_all_refs ✅ prior session
/doc_drift/check + /resolve ✅ prior session
mcp-server/context7_bridge.ts ✅ prior session
boost exclusion in compute_boost_* ✅ prior session
/doc_drift/scan + corrections.jsonl ✅ THIS COMMIT
The 0→85% thesis stays valid against external doc drift. Popular
playbooks can no longer compound the wrong way as Docker / Terraform
/ React / etc. patch their docs — the scan flags drift, the boost
filter excludes the playbook, the operator reviews the corrections
.jsonl, and a revise call (Phase 27) supersedes the stale entry
with corrected operation/approach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
98db129b8f |
gateway: /v1/iterate — Phase 43 v3 part 3 (generate → validate → retry loop)
Closes the Phase 43 PRD's "iteration loop with validation in place"
structurally. Single endpoint that wraps the 0→85% pattern any
caller can post against without re-implementing it.
POST /v1/iterate
{
"kind":"fill" | "email" | "playbook",
"prompt":"...",
"system":"...", (optional)
"provider":"ollama_cloud",
"model":"kimi-k2.6",
"context":{...}, (target_count/city/state/role/...)
"max_iterations":3, (default 3)
"temperature":0.2, (default 0.2)
"max_tokens":4096 (default 4096)
}
→ 200 + IterateResponse (artifact accepted)
{artifact, validation, iterations, history:[{iteration,raw,status}]}
→ 422 + IterateFailure (max iter reached)
{error, iterations, history}
The loop:
1. Generate via gateway-internal HTTP loopback to /v1/chat with the
given provider/model. Model output is the model's free-form text.
2. Extract a JSON object from the output — handles fenced blocks
(```json ... ```), bare braces, and prose-with-embedded-JSON.
On no extractable JSON: append "your response wasn't valid JSON"
to the prompt and retry.
3. POST the extracted artifact to /v1/validate (server-side reuse of
the FillValidator/EmailValidator/PlaybookValidator stack from
Phase 43 v3 part 2).
4. On 200 + Report: success — return artifact + history.
5. On 422 + ValidationError: append the specific error JSON to the
prompt as corrective context and retry. This is the "observer
correction" piece in PRD shape, simplified — the validator's own
structured error IS the feedback signal.
6. Cap at max_iterations.
Verified end-to-end with kimi-k2.6 via ollama_cloud:
Request: fill 1 Welder in Toledo, model picks W-1 (actually
Louisville, KY — wrong city)
iter 0: model emits {fills:[W-1,"W-1"]} → 422 Consistency
("city 'Louisville' doesn't match contract city 'Toledo'")
iter 1: prompt now includes the error → model emits same answer
(didn't pick a different worker — model lacks roster
access; would need hybrid_search upstream)
max=2: 422 IterateFailure with full history
The negative test demonstrates the LOOP MECHANICS work:
- Generation → validation → retry-with-error-context → cap
- The model's failure trace is queryable; downstream tooling can
inspect history[] to see exactly where each iteration broke
- A production executor would do hybrid_search to find Toledo
workers before posting; /v1/iterate is the validation+retry
layer downstream
JSON extractor handles three shapes:
- Fenced: ```json {...} ``` (preferred — explicit signal)
- Bare: plain text + {...} + plain text
- Multi: picks the first balanced {...}
Unit tests cover all three plus the no-JSON fallback.
Phase 43 closure status:
v1: scaffolds ✅ (older commit)
v2: real validators ✅
|
||
|
rootandClaude Opus 4.7
|
5d93a715c3 |
gateway: Phase 44 part 3 — split AiClient so vectord routes through /v1/chat
Builds two AiClient instances at boot:
- `ai_client_direct = AiClient::new(sidecar_url)` — direct sidecar
transport. Used by V1State (gateway's own /v1/chat ollama_arm
needs this — calling /v1/chat from itself would self-loop) and
by the legacy /ai proxy.
- `ai_client_observable = AiClient::new_with_gateway(sidecar_url,
${gateway_host}:${gateway_port})` — routes generate() through
/v1/chat with provider="ollama". Used by:
vectord::agent (autotune background loop)
vectord::service (the /vectors HTTP surface — RAG, summary,
playbook synthesis, etc.)
Net result: every LLM call from a vectord module now lands in
/v1/usage and Langfuse traces. The autotune agent's hourly cycle
becomes observable; /vectors RAG calls show provider+model+latency
in the usage report. Phase 44 PRD's gate ("/v1/usage accounts for
every LLM call in the system within a 1-minute window") is now
satisfied for the gateway-hosted services.
Cost: one localhost HTTP hop per vectord-originated LLM call. At
~1-3ms RTT for in-process loopback, negligible against the LLM
call's own 30-90s wall-clock.
Phase 44 part 4 (deferred):
- Standalone consumers that build their own AiClient (test
harnesses, bot/propose, etc) — the TS-side already migrated in
part 1 + the regression guard at scripts/check_phase44_callers.sh
catches new direct callers. Rust standalone harnesses (if any
surface) follow the same pattern: construct via new_with_gateway
to opt into observability.
- Direct sidecar callers in standalone tools (scripts/serve_lab.py
is one) — Python-side; out of Rust scope.
Verified:
cargo build --release -p gateway compiles
systemctl restart lakehouse active
/v1/chat sanity PONG, finish=stop
When the autotune agent next cycles or any /vectors RAG endpoint
fires, /v1/usage will show the provider=ollama tick — first
real-world data should land within the next agent cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
7b88fb9269 |
aibridge: Phase 44 part 2 — opt-in /v1/chat routing for AiClient.generate()
The Phase 44 PRD's "AiClient becomes a thin /v1/chat client" was a
chicken-and-egg problem: the gateway's own /v1/chat ollama_arm calls
AiClient.generate() to reach the sidecar. If AiClient unconditionally
routed through /v1/chat, gateway → /v1/chat → ollama → AiClient →
/v1/chat would loop forever.
Solution: opt-in routing.
- `AiClient::new(base_url)` — direct-sidecar, gateway-internal use
(gateway's own /v1/chat handlers, ollama::chat in mod.rs)
- `AiClient::new_with_gateway(base_url, gateway_url)` — routes
generate() through ${gateway_url}/v1/chat with provider="ollama"
so the call lands in /v1/usage + Langfuse traces
Shape translation in generate_via_gateway():
GenerateRequest {prompt, system, model, temperature, max_tokens, think}
→ /v1/chat {messages: [system?, user], provider:"ollama", ...}
/v1/chat response choices[0].message.content + usage.{prompt,completion}_tokens
→ GenerateResponse {text, model, tokens_evaluated, tokens_generated}
embed(), rerank(), and admin methods (health, unload_model, etc.) stay
direct-to-sidecar — no /v1/embed equivalent yet, no point round-trip.
Transitive migration: aibridge::continuation::generate_continuable
goes through TextGenerator::generate_text() → AiClient.generate(), so
every caller of generate_continuable inherits the routing decision
made at AiClient construction. Phase 21's continuation loop, hot-
path JSON emitters, etc. all gain observability for free when the
construction site opts in.
Verified end-to-end:
curl /v1/chat with the exact JSON shape AiClient sends
→ "PONG-AIBRIDGE", finish=stop, 27/7 tokens
/v1/usage after the call
→ requests=1, by_provider.ollama.requests=1, tokens tracked
Phase 44 part 3 (next):
- Migrate vectord's AiClient construction site so vectord modules
(rag, autotune, harness, refresh, supervisor, playbook_memory)
flow through /v1/chat. Currently the gateway's main.rs constructs
one AiClient via `new()` and shares it via V1State; vectord
inherits direct-sidecar transport. Migration requires constructing
a SEPARATE AiClient with `new_with_gateway` for vectord's state
bag (V1State.ai_client must stay direct to avoid the self-loop).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
47776b07cd |
auditor: 2 fixes from kimi_architect on ebd9ab7 audit
The auditor's own audit on commit |
||
|
rootandClaude Opus 4.7
|
86123fce4c |
gateway: /v1/validate endpoint — Phase 43 v3 part 2
Closes the Phase 43 PRD's "any caller can validate" surface. The
validator crate (FillValidator + EmailValidator + PlaybookValidator
+ WorkerLookup) is now reachable over HTTP at /v1/validate.
Request/response:
POST /v1/validate
{"kind":"fill"|"email"|"playbook", "artifact":{...}, "context":{...}?}
→ 200 + Report on success
→ 422 + ValidationError on validation failure
→ 400 on bad kind
Boot-time wiring (main.rs):
- Load workers_500k.parquet into a shared Arc<dyn WorkerLookup>
- Path overridable via LH_WORKERS_PARQUET env
- Missing file: warn + fall back to empty InMemoryWorkerLookup so the
endpoint stays live (validators just fail Consistency on every
worker-existence check, which is the correct behavior when the
roster isn't configured)
- Boot log line: "workers parquet loaded from <path>" or
"workers parquet at <path> not found"
- Live boot timing: 500K rows loaded in ~1.4s
V1State gains `validate_workers: Arc<dyn validator::WorkerLookup>`.
The `_context` JSON key is auto-injected from `request.context` so
callers can either embed `_context` directly in `artifact` or split
it cleanly via the `context` field.
Verified live (gateway + 500K worker snapshot):
POST {kind:"fill", phantom W-FAKE-99999} → 422 Consistency
("does not exist in
worker roster")
POST {kind:"fill", real W-1, "Anyone"} → 200 OK + Warning
("differs from
roster name 'Donald
Green'")
POST {kind:"email", body has 123-45-6789} → 422 Policy ("SSN-
shaped sequence")
POST {kind:"nonsense"} → 400 Bad Request
The "0→85% with iteration" thesis can now run end-to-end on real
staffing data: an executor emits a fill_proposal, posts to
/v1/validate, gets a structured ValidationError on phantom IDs or
inactive workers, observer-corrects, retries. Closure of that loop
in a scrum harness is the next commit (separate scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
ebd9ab7c77 |
validator: Phase 43 v3 — production WorkerLookup backed by workers_500k.parquet
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Verified end-to-end:"
Closes the Phase 43 v2 loose end. The validator scaffolds (FillValidator,
EmailValidator) take Arc<dyn WorkerLookup> at construction; this commit
ships the parquet-snapshot impl that production code wires in.
Schema mapping (workers_500k.parquet → WorkerRecord):
worker_id (int64) → candidate_id = "W-{id}" (matches what the
staffing executor
emits)
name (string) → name (already concatenated upstream)
role (string) → role
city, state (string) → city, state
availability (double) → status: "active" if >0 else "inactive"
Workers_500k has no `status` column; we derive from `availability`
since 0.0 means vacationing/suspended/etc in this dataset's
convention. Once Track A.B's `_safe` view ships with proper status,
flip the loader to read it directly — schema mapping is in one
function (load_workers_parquet), so the swap is trivial.
In-memory snapshot model:
- Loads all 500K rows at startup → ~75MB resident
- Sync .find() — no per-call I/O on the validation hot path
- Refresh = call load_workers_parquet again to rebuild
- Caller-driven refresh (no auto-watch) — operators pick the cadence
Why workers_500k and not candidates.parquet:
candidates.parquet has the right shape (string candidate_id, status,
first/last_name) but lacks `role` — and the staffing executor matches
the W-* convention from workers_500k_v8 corpus. So the production
data path goes through workers_500k. The schema mismatch between the
two parquets is documented in `reports/staffing/synthetic-data-gap-
report.md` (gap A); resolution is operator's call.
Errors are typed (LookupLoadError):
- Open: file not found / permission
- Parse: invalid parquet
- MissingColumn: schema doesn't have required field
- BadRow: row missing worker_id or name
Schema check happens before iteration, so a wrong-shape file fails
loud immediately rather than silently building an empty lookup.
Verification:
cargo build -p validator compiles
cargo test -p validator 33 pass / 0 fail
(was 31; +2 for parquet)
load_real_workers_500k smoke test passes against the
live 500K-row file:
W-1 resolves, status +
role + city/state all
populated.
Phase 43 v3 part 2 (next):
- /v1/validate gateway endpoint that takes a JSON artifact + dispatches
to FillValidator/EmailValidator/PlaybookValidator with a shared
WorkerLookup loaded from the parquet at gateway startup.
- That closes the "any caller can validate" surface; execution-loop
wiring (Phase 43 PRD's "generate → validate → correct → retry")
becomes a thin wrapper on top of /v1/validate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
f6af0fd409 |
phase 44 (part 1): migrate TS callers to /v1/chat + add regression guard
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end:"
Migrates the four TypeScript /generate callers to the gateway's /v1/chat surface so every LLM call lands on /v1/usage and Langfuse: tests/multi-agent/agent.ts::generate() provider="ollama" tests/agent_test/agent_harness.ts::callAgent provider="ollama" bot/propose.ts::generateProposal provider="ollama_cloud" mcp-server/observer.ts (error analysis) provider="ollama" Each migration follows the same pattern as the prior generateCloud() migration (already on /v1/chat from 2026-04-24): replace `fetch(SIDECAR/generate)` with `fetch(GATEWAY/v1/chat)`, swap the prompt-style body for OpenAI-compat messages array, extract content from `choices[0].message.content` instead of `text`. Same upstream models in every case — gateway is the new home for the call, transport otherwise unchanged. Adds scripts/check_phase44_callers.sh — fail-loud regression guard that exits non-zero if any non-adapter file fetches /generate or api/generate. Adapter files (crates/gateway, crates/aibridge, sidecar/) are exempt. Pre-tightening regex flagged prose mentions in comments; the shipped regex requires `fetch(...)` or `client.post(...)` shape so comments don't trip it. Verification: bun build mcp-server/observer.ts compiles bun build tests/multi-agent/agent.ts compiles bun build tests/agent_test/agent_harness.ts compiles bun build bot/propose.ts compiles ./scripts/check_phase44_callers.sh ✅ clean systemctl restart lakehouse-observer active Phase 44 part 2 (deferred): - crates/aibridge/src/client.rs:118 still posts to sidecar /generate directly. AiClient is the foundational Rust LLM caller used by 8+ vectord modules; migrating it is a workspace-wide refactor that needs its own commit. Plan: keep AiClient as the local- transport layer for the gateway's `provider=ollama` arm, but introduce a thin `/v1/chat` wrapper for external callers (vectord autotune, agent, rag, refresh, supervisor, playbook_memory). - tests/real-world/hard_task_escalation.ts: comment mentions /api/generate but doesn't actually call it. Comment is left intentionally as historical context; regex no longer flags it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
bfe1ea9d1c |
auditor: alternate Kimi K2.6 ↔ Haiku 4.5, drop Opus from auto-promotion
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Verified end-to-end:"
Operator can't sustain Opus's ~$0.30/audit on the daemon. New strategy: - Even-numbered audits per PR use kimi-k2.6 via ollama_cloud (effectively free under the Ollama Pro flat subscription) - Odd-numbered audits use claude-haiku-4-5 via opencode/Zen (~$0.04/audit) - Frontier models (Opus, GPT-5.5-pro, Gemini 3.1-pro) are NOT in auto-promotion. Operator hands distilled findings to a frontier model manually when a load-bearing decision needs it. Mirrors the lakehouse playbook-memory pattern: cheap models do the volume, the validated subset compounds, only the compounded bundle gets handed to a frontier model. Same logic at the auditor layer. Audit-index derivation: count of existing kimi_verdicts files for the PR. So if the dir has 4 verdicts for PR #11 already, the 5th audit is index 4 (even) → Kimi, the 6th is index 5 (odd) → Haiku. Across an active PR's lifetime the audits naturally interleave the two lineages. Cost projection at observed cadence (5-10 pushes/day): - Old (Haiku default + Opus auto on big diffs): $1-3/day - New (Kimi/Haiku alternating, no Opus): $0.10-0.40/day - $31.68 budget lasts: ~3 months instead of ~10 days Override knobs: LH_AUDITOR_KIMI_MODEL=<X> pins to model X (no alternation) LH_AUDITOR_KIMI_PROVIDER=<P> provider for default model LH_AUDITOR_KIMI_ALT_MODEL=<X> sets the odd-index alternate LH_AUDITOR_KIMI_ALT_PROVIDER=<P> provider for alternate The OPUS_THRESHOLD env knobs from the prior auto-promotion commit are now no-ops (unset, no longer referenced). Verification: bun build auditor/checks/kimi_architect.ts compiles systemctl restart lakehouse-auditor active systemctl show env Haiku pin removed, Kimi default + cap=3 set Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
dc6dd1d30c |
auditor: per-PR audit cap (default 3) — daemon halts further audits until reset
Adds MAX_AUDITS_PER_PR (env LH_AUDITOR_MAX_AUDITS_PER_PR, default 3).
The poller increments a per-PR counter on each successful audit; when
the counter reaches the cap it skips that PR with a "capped" log line
until the operator manually clears state.audit_count_per_pr[<PR#>].
Why:
"I don't want it to continuously loop even if it finds a problem.
We need a maximum until we can come back."
Without this, the daemon polls every 90s and audits every new head
SHA. If each fix-commit surfaces new findings (which is what
kimi_architect is designed to do), the audit loop runs unbounded
while the operator is away. At ~$0.30/audit on Opus and 5-10 pushes
a day, that's $1-3/day idle burn — fine for a couple days, painful
for weeks.
Cap mechanics:
- Counter starts at 0 per PR (or whatever exists in state.json)
- Increments only on successful audit (failures don't count)
- Comparison is >= so cap=3 means audits 1, 2, 3 run; 4+ skip
- Skip is logged: "capped at N/M audits — clear state.json
audit_count_per_pr.<N> to resume"
- New `cycles_skipped_capped` counter on State for observability
Reset:
jq '.audit_count_per_pr = (.audit_count_per_pr - {"11": 4})' \
/home/profit/lakehouse/data/_auditor/state.json > /tmp/s.json && \
mv /tmp/s.json /home/profit/lakehouse/data/_auditor/state.json
- Daemon picks up the change on the next cycle (no restart needed —
state is reloaded each cycle)
- Or set the entry to 0 if you want to keep the key
Disable cap: LH_AUDITOR_MAX_AUDITS_PER_PR=0
Reduce cap: LH_AUDITOR_MAX_AUDITS_PER_PR=1 (one audit per PR head, then pause)
Pre-existing PR audits today (4 on PR #11) are NOT seeded into the
counter by this commit — operator decides post-deploy whether to set
state.audit_count_per_pr.11 to today's actual count or leave at 0.
Setting to 4 (or 3) immediately halts further audits on PR #11.
Verification:
bun build auditor/index.ts compiles
systemctl restart lakehouse-auditor active
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
19a65b87e3 |
auditor: 3 fixes from Opus self-audit on 454da15 + tree-split deletion
lakehouse/auditor 14 blocking issues: cloud: claim not backed — "Verified end-to-end:"
The post-fix audit on commit
|
||
|
rootandClaude Opus 4.7
|
454da15301 |
auditor + aibridge: 6 fixes from Opus 4.7 self-audit on PR #11
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end:"
The kimi_architect auditor on commit
|
||
|
rootandClaude Opus 4.7
|
00c8408335 |
validator: Phase 43 v2 — real worker-existence + PII + name-consistency checks
lakehouse/auditor 16 blocking issues: cloud: claim not backed — "Verified end-to-end:"
The Phase 43 scaffolds (FillValidator, EmailValidator) shipped with
TODO(phase-43 v2) markers for the actual cross-roster checks. This is
those checks landing.
The PRD calls for "the 0→85% pattern reproduces on real staffing
tasks — the iteration loop with validation in place is what made
small models successful." Worker-existence is the load-bearing check:
when the executor emits {candidate_id: "W-FAKE", name: "Imaginary"},
schema-only validation passes, and only roster lookup catches it.
Architecture:
- New `WorkerLookup` trait + `WorkerRecord` struct in lib.rs. Sync by
design — validators hold an in-memory snapshot, no per-call I/O on
the validation hot path. Production wraps a parquet snapshot;
tests use `InMemoryWorkerLookup`.
- Validators take `Arc<dyn WorkerLookup>` at construction so the
same shape covers prod + tests + future devops scaffolds.
- Contract metadata travels under JSON `_context` key alongside the
validated payload (target_count, city, state, role, client_id for
fills; candidate_id for emails). Keeps the Validator trait
signature stable and lets the executor serialize context inline.
FillValidator (11 tests, was 4):
- Schema (existing)
- Completeness — endorsed count == target_count
- Worker existence — phantom candidate_id fails Consistency
- Status — non-active worker fails Consistency
- Geo/role match — city/state/role mismatch with contract fails
Consistency
- Client blacklist — fails Policy
- Duplicate candidate_id within one fill — fails Consistency
- Name mismatch — Warning (not Error) since recruiters sometimes
send roster updates through the proposal layer
EmailValidator (11 tests, was 4):
- Schema + length (existing)
- SSN scan (NNN-NN-NNNN) — fails Policy
- Salary disclosure (keyword + $-amount within ~40 chars) — fails
Policy. Std-only scan, no regex dep added.
- Worker name consistency — when _context.candidate_id resolves,
body must contain the worker's first name (Warning if missing)
- Phantom candidate_id in _context — fails Consistency
- Phone NNN-NNN-NNNN does NOT trip the SSN detector (verified by
test); the SSN scanner explicitly rejects sequences embedded in
longer digit runs
Pre-existing issue (NOT from this change, NOT fixed here):
crates/vectord/src/pathway_memory.rs:927 has a stale PathwayTrace
struct initializer that fails `cargo check --tests` with E0063 on
6 missing fields. `cargo check --workspace` (production) is green;
only the vectord test target is broken. Tracked for a separate fix.
Verification:
cargo test -p validator 31 pass / 0 fail (was 13)
cargo check --workspace green
Next: wire `Arc<dyn WorkerLookup>` into the gateway execution loop
(generate → validate → observer-correct → retry, bounded by
max_iterations=3 per Phase 43 PRD). Production lookup impl loads
from a workers parquet snapshot — Track A gap-fix B's `_safe` view
is the right source once decided, raw workers_500k otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
8aa7ee974f |
auditor: auto-promote to Claude Opus 4.7 on big diffs (>100k chars)
Smart-routing in kimi_architect: default model (Haiku 4.5 by env, or
Kimi K2.6 if not set) handles normal PR audits cheap and fast; diffs
above LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS (default 100k) get
promoted to Claude Opus 4.7 for the audit.
Why this split: the 2026-04-27 3-way bake-off (Kimi K2.6 vs Haiku 4.5
vs Opus 4.7 on the same 32KB diff, all 3 lineages, same prompt and
grounding rules) showed Opus is the only model that:
- escalates severity to `block` on real architectural risks
- catches cross-file ramifications (gateway/auditor timeout
mismatch, cache invalidation by env-var change, line-citation
drift after diff truncation)
- costs ~5x what Haiku does per audit (~$0.10 vs $0.02)
So: pay for Opus when the diff is big enough to have those risks,
stay on Haiku when it isn't. 80% of refactor PRs cross 100KB; 90% of
single-feature PRs don't.
New env knobs (all optional, sensible defaults):
LH_AUDITOR_KIMI_OPUS_MODEL default claude-opus-4-7
LH_AUDITOR_KIMI_OPUS_PROVIDER default opencode
LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS default 100000
(set very high to disable)
The threaded `provider`/`model` arguments through callKimi() so the
same routing also lets per-call diagnostic harnesses run different
models without touching env vars.
Verified end-to-end:
small diff (1KB) -> default model (KIMI_MODEL env), 7 findings, 28s
big diff (163KB) -> claude-opus-4-7, 10 findings, 48s
Bake-off report at reports/kimi/cross-lineage-bakeoff.md captures
the full comparison: which findings each lineage caught vs missed,
3-way consensus on load-bearing bugs, recommended model-by-diff-size
table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
bc698eb6da |
gateway: OpenCode (Zen + Go) provider adapter
Wires opencode.ai as a /v1/chat provider. One sk-* key reaches 40
models across Anthropic, OpenAI, Google, Moonshot, DeepSeek, Zhipu,
Alibaba, Minimax — billed against either the user's Zen balance
(pay-per-token premium models) or Go subscription (flat-rate
Kimi/GLM/DeepSeek/etc.). The unified /zen/v1 endpoint routes both;
upstream picks the billing tier based on model id.
Notable adapter quirks:
- Strip "opencode/" prefix on outbound (mirrors openrouter/kimi
pattern). Caller can use {provider:"opencode", model:"X"} or
{model:"opencode/X"}.
- Drop temperature for claude-*, gpt-5*, o1/o3/o4 models. Anthropic
and OpenAI's reasoning lineage rejects temperature with 400
"deprecated for this model". OCChatBody now serializes temperature
as Option<f64> with skip_serializing_if so omitting it produces
clean JSON.
- max_tokens.filter(|&n| n > 0) catches Some(0) — defensive after
the same trap bit kimi.rs (empty env -> Number("") -> 0 -> 503).
- 600s default upstream timeout; reasoning models on big audit
prompts legitimately take 3-5 min. Override OPENCODE_TIMEOUT_SECS.
Key handling:
- /etc/lakehouse/opencode.env (0600 root) loaded via systemd
EnvironmentFile. Same pattern as kimi.env.
- OPENCODE_API_KEY env first, file scrape as fallback.
Verified end-to-end:
opencode/claude-opus-4-7 -> "I'm Claude, made by Anthropic."
opencode/kimi-k2.6 -> PONG-K26-GO
opencode/deepseek-v4-pro -> PONG-DS-V4
opencode/glm-5.1 -> PONG-GLM
opencode/minimax-m2.5-free -> PONG-FREE
Pricing reference (per audit @ ~14k in / 6k out):
claude-opus-4-7 ~$0.22 (Zen)
claude-haiku-4-5 ~$0.04 (Zen)
gpt-5.5-pro ~$1.50 (Zen)
gemini-3-flash ~$0.03 (Zen)
kimi-k2.6 / glm / deepseek / qwen / minimax / mimo: covered by Go
subscription ($10/mo, $60/mo cap).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
ff5de76241 |
auditor + gateway: 2 fixes from kimi_architect's first real run
Acted on 2 of 10 findings Kimi caught when auditing its own integration
on PR #11 head
|
||
|
rootandClaude Opus 4.7
|
3eaac413e6 |
auditor: route kimi_architect through ollama_cloud/kimi-k2.6 (TOS-clean primary)
Two changes:
1. Default provider now ollama_cloud/kimi-k2.6 (env-overridable via
LH_AUDITOR_KIMI_PROVIDER + LH_AUDITOR_KIMI_MODEL). Ollama Cloud Pro
exposes kimi-k2.6 legitimately, so we no longer need the User-Agent-
spoof path through api.kimi.com. Smoke test 2026-04-27:
api.kimi.com 368s 8 findings 8/8 grounded
ollama_cloud 54s 10 findings 10/10 grounded
The kimi.rs adapter (provider=kimi) stays wired as a fallback when
Ollama Cloud is upstream-broken.
2. Switch HTTP transport from Bun's native fetch to curl via Bun.spawn.
Bun fetch has an undocumented ~300s ceiling that AbortController +
setTimeout cannot override; curl honors -m for end-to-end max
transfer time without a hard intrinsic limit. Required for Kimi's
reasoning-heavy responses on big audit prompts.
3. Bug fix Kimi caught in this very file (turtles all the way down):
Number(process.env.LH_AUDITOR_KIMI_MAX_TOKENS ?? 128_000) yields 0
when env is set to empty string — `??` only catches null/undefined.
Switched to Number(env) || 128_000 so empty/0/NaN all fall back.
Same pattern probably exists in other files; future audit pass.
4. Bumped MAX_TOKENS default 12K -> 128K. Kimi K2.6's reasoning_content
counts against this budget but isn't surfaced in OpenAI-shape content;
12K silently produced finish_reason=length with empty content when
reasoning consumed the budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
8d02c7f441 |
auditor: integrate Kimi second-pass review (off by default, LH_AUDITOR_KIMI=1)
Adds kimi_architect as a fifth check kind in the auditor. Runs sequentially after static/dynamic/inference/kb_query, consumes their findings as context, and asks Kimi For Coding "what did everyone miss?" — targeting load-bearing issues that deepseek N=3 voting can't see (compile errors, false telemetry, schema bypasses, determinism leaks). 7/7 grounded on the distillation v1.0.0 audit experiment 2026-04-27. Off by default. Enable on the lakehouse-auditor service: systemctl edit lakehouse-auditor.service Environment=LH_AUDITOR_KIMI=1 Tunable env (all optional): LH_AUDITOR_KIMI_MODEL default kimi-for-coding LH_AUDITOR_KIMI_MAX_TOKENS default 12000 LH_GATEWAY_URL default http://localhost:3100 Guardrails: - Failure-isolated. Any Kimi error / 429 / TOS revocation returns a single info-level skip-finding so the existing pipeline never blocks on a Kimi outage. - Cost-bounded. Cached verdicts at data/_auditor/kimi_verdicts/<pr>- <sha>.json with 24h TTL — re-audits within the window return cached findings instead of re-calling upstream. New commits produce new SHAs so caching is per-head, not per-day. - 6min upstream timeout (vs 2min for openrouter inference) — Kimi is a reasoning model and the audit prompt is large. - Grounding verification baked in. Every finding's cited file:line is greppped against the actual file before the verdict is persisted. Per-finding evidence carries [grounding: verified at FILE:LINE] or [grounding: line N > EOF] / [grounding: file not found]. Confab- ulation rate goes into data/_kb/kimi_audits.jsonl as grounding_rate for "is this still valuable" tracking. Persisted artifacts: data/_auditor/kimi_verdicts/<pr>-<sha>.json full verdict + raw Kimi response + grounding data/_kb/kimi_audits.jsonl one row per call: latency, tokens, findings, grounding rate Verdict-rendering: kimi_architect now appears in the per-check sections of the human-readable comment posted to PRs (auditor/audit.ts checkOrder), after kb_query. Verification: bun build auditor/checks/kimi_architect.ts compiles bun build auditor/audit.ts compiles parser sanity (3-finding fixture) 3/3 lifted correctly Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
643dd2d520 |
gateway: direct Kimi For Coding provider adapter (api.kimi.com)
Wires kimi-for-coding (Kimi K2.6 underneath) as a first-class /v1/chat
provider so consumers can target it via {provider:"kimi"} or model
prefix kimi/<model>. Bypasses the upstream-broken kimi-k2:1t on Ollama
Cloud and the rate-limited moonshotai/kimi-k2.6 path through OpenRouter.
Adapter shape mirrors openrouter.rs (OpenAI-compatible Chat Completions).
Differences from generic OpenAI providers:
- api.kimi.com is a SEPARATE account system from api.moonshot.ai and
api.moonshot.cn. sk-kimi-* keys are NOT interchangeable across them.
- Endpoint is User-Agent-gated to "approved" coding agents (Kimi CLI,
Claude Code, Roo Code, Kilo Code, ...). Requests from generic clients
return 403 access_terminated_error. Adapter sends User-Agent:
claude-code/1.0.0. Per Moonshot TOS this is a tampering-class action
that may result in seat suspension; J authorized 2026-04-27 with
awareness of the risk.
- kimi-for-coding is a reasoning model — reasoning_content counts
against max_tokens. Default 800-token budget yields empty visible
content with finish_reason=length. Code-review workloads need
max_tokens >= 1500.
- Default 600s upstream timeout (vs 180s for openrouter.rs) — code
audits with full file context legitimately take 3-5 minutes.
Override via KIMI_TIMEOUT_SECS env.
Key handling:
- /etc/lakehouse/kimi.env (0600 root) loaded via systemd EnvironmentFile
- KIMI_API_KEY env first, then file scrape as fallback
- /etc/systemd/system/lakehouse.service NOT included in this commit
(system file outside repo); operator must add EnvironmentFile=-
/etc/lakehouse/kimi.env to the lakehouse.service unit
NOT in scrum_master_pipeline LADDER. The 9-rung ladder is for
unattended automatic recovery; placing Kimi there would hammer a
TOS-gated endpoint with hostility-policy potential. Kimi is
addressable via /v1/chat for explicit invocations only — auditor
integration in a follow-up commit.
Verification:
cargo check -p gateway --tests compiles
curl /v1/chat provider=kimi 200 OK, content="PONG"
curl /v1/chat model="kimi/kimi-for-coding" 200 OK (prefix routing)
Kimi audit on distillation last-week 7/7 grounded findings
(reports/kimi/audit-last-week-full.md)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
d77622fc6b |
distillation: fix 7 grounding bugs found by Kimi audit
Kimi For Coding (api.kimi.com, kimi-for-coding) ran a forensic audit on
distillation v1.0.0 with full file content. 7/7 flags verified real on
grep. Substrate now matches what v1.0.0 claimed: deterministic, no
schema bypasses, Rust tests compile.
Fixes:
- mode.rs:1035,1042 matrix_corpus Some/None -> vec![..]/vec![]; cargo
check --tests now compiles (was silently broken;
only bun tests were running)
- scorer.ts:30 SCORER_VERSION env override removed - identical
input now produces identical version stamp, not
env-dependent drift
- transforms.ts:181 auto_apply wall-clock fallback (new Date()) ->
deterministic recorded_at fallback
- replay.ts:378 recorded_run_id Date.now() -> sha256(recorded_at);
replay rows now reproducible given recorded_at
- receipts.ts:454,495 input_hash_match hardcoded true was misleading
telemetry; bumped DRIFT_REPORT_SCHEMA_VERSION 1->2,
field is now boolean|null with honest null when
not computed at this layer
- score_runs.ts:89-100,159 dedup keyed only on sig_hash made
scorer-version bumps invisible. Composite
sig_hash:scorer_version forces re-scoring
- export_sft.ts:126 (ev as any).contractor bypass emitted "<contractor>"
placeholder for every contract_analyses SFT row.
Added typed EvidenceRecord.metadata bucket;
transforms.ts populates metadata.contractor;
exporter reads typed value
Verification (all green):
cargo check -p gateway --tests compiles
bun test tests/distillation/ 145 pass / 0 fail
bun acceptance 22/22 invariants
bun audit-full 16/16 required checks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
d11632a6fa |
staffing: recon + synthetic-data gap report (Phase 0, no implementation)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Phase 8 done-criteria (per spec):"
Spec mandates these two docs before any staffing audit runner ships:
docs/recon/staffing-lakehouse-distillation-recon.md
reports/staffing/synthetic-data-gap-report.md
NO distillation core touched. Distillation v1.0.0 (commit
|
||
|
rootandClaude Opus 4.7
|
e7636f202b |
distillation: regenerate v1.0.0 release artifacts
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Phase 8 done-criteria (per spec):"
Auto-generated by `./scripts/distill release-freeze` — RELEASE-READY (6/6 gates). Captures the v1.0.0 manifest + the latest acceptance + audit reports re-run during the freeze. reports/distillation/release-freeze.md human-readable manifest reports/distillation/release-manifest.json machine-readable manifest reports/distillation/phase6-acceptance-report.md re-run during freeze (22/22 invariants) reports/distillation/phase8-full-audit-report.md re-run during freeze (16/16 required) Pre-tag state: branch: scrum/auto-apply-19814 head: <prior commit before this one> full pipeline: 145 distillation tests pass · 0 fail acceptance: 22/22 invariants on fixture, bit-identical reproducibility audit-full: 16/16 required across Phases 0-7 Tag command awaiting operator confirmation: git tag -a distillation-v1.0.0 -m "distillation v1.0.0 — 8-phase substrate frozen" git push origin distillation-v1.0.0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
rootandClaude Opus 4.7
|
73f242e3e4 |
distillation: Phase 9 — release freeze and operator handoff
Final phase. Adds:
scripts/distillation/release_freeze.ts ~330 lines, 6 release gates
docs/distillation/operator-handoff.md durable cold-start operator doc
docs/distillation/recovery-runbook.md failure-mode runbook by symptom
scripts/distillation/distill.ts +release-freeze subcommand
The release_freeze orchestrator runs every gate the system has:
1. Clean git state (tolerates auto-regenerated reports)
2. Full test suite (bun test tests/distillation auditor/schemas/distillation)
3. Phase commit verification (every Phase 0-8 commit resolves)
4. Acceptance gate (22-invariant fixture E2E)
5. audit-full (Phases 0-7 verified + drift detection)
6. Tag availability check (distillation-v1.0.0 not yet existing)
Outputs:
reports/distillation/release-freeze.md human-readable manifest
reports/distillation/release-manifest.json machine-readable manifest
Manifest captures:
- git_head + git_branch + released_at
- phase→commit map for all 9 commits (Phase 0+1+2 scaffold through Phase 8 audit)
- dataset counts at freeze (RAG/SFT/Preference/evidence/scored/quarantined)
- latest audit baseline row
- per-gate pass/fail with detail
Operator handoff doc covers:
- phase map with commits + report locations
- known-good commands
- how to rerun audit-full + inspect drift
- how to restore from last-good (git checkout distillation-v1.0.0)
- how to add future phases without contaminating corpus
- what NOT to modify casually (with file:reason mapping)
- cumulative commits at v1.0.0
Recovery runbook covers, by symptom:
- audit-full exit non-zero (per-phase diagnostics)
- drift table flags warn (intentional vs regression)
- acceptance fail vs audit-full pass divergence
- run-all empty exports (counter-bisection order)
- hash mismatch on identical input (determinism violation; CRITICAL)
- replay logs growing unbounded (rotation guidance)
- nuclear restore via git checkout distillation-v1.0.0
Spec constraints (per now.md Phase 9):
- DO NOT add new intelligence features ✓ (zero new logic)
- DO NOT change scoring/export logic ✓ (zero touches)
- DO NOT weaken gates ✓ (gates only added, never relaxed beyond the
auto-regen tolerance documented in checkCleanGit)
- DO NOT retrain anything ✓ (no model touches)
CLI:
./scripts/distill release-freeze # exit 0 = release-ready
Tag creation deferred to operator confirmation (the release-freeze
report prints the exact `git tag` command). Per CLAUDE.md guidance,
destructive/visible operations like tags require explicit user
authorization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
5bdd159966 |
distillation: Phase 8 — full system audit
lakehouse/auditor 14 blocking issues: cloud: claim not backed — "Phase 8 done-criteria (per spec):"
Meta-audit script that runs deterministic checks across Phases 0-7
and compares to a baseline (auto-grown from prior runs). Pure
observability — no pipeline modification. Single command:
./scripts/distill audit-full
Files (2 new + 1 modified):
scripts/distillation/audit_full.ts ~430 lines, 8 phase checks + drift
scripts/distillation/distill.ts +audit-full subcommand
reports/distillation/phase8-full-audit-report.md (autogenerated by run)
Real-data audit on commit
|
||
|
rootandClaude Opus 4.7
|
681f39d5fa |
distillation: Phase 7 — replay-driven local model bootstrapping
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "probes; multi-hour outage). deepseek is the proven drop-in from"
Runtime layer that takes a task → retrieves matching playbooks/RAG
records → builds a structured context bundle → feeds it to a LOCAL
model (qwen3.5:latest, ~7B class) → validates output → escalates only
when needed → logs the full run as new evidence. NOT model training.
Pure runtime behavior shaping via retrieval against the Phase 0-6
distillation substrate.
Files (3 new + 1 modified):
scripts/distillation/replay.ts ~370 lines
tests/distillation/replay.test.ts 10 tests, 19 expects
scripts/distillation/distill.ts +replay subcommand
reports/distillation/phase7-replay-report.md
Test metrics: 145 cumulative distillation tests pass · 0 fail · 372 expects · 618ms
Real-data A/B on 3 tasks (same qwen3.5:latest local model, with vs
without retrieval) — proves the spec claim "local model improves
with retrieval":
Task 1 "Audit phase 38 provider routing":
WITH retrieval: cited V1State, openrouter, /v1/chat, ProviderAdapter,
PRD.md line ranges — REAL Lakehouse internals
WITHOUT retrieval: invented "P99999, Z99999 placeholder codes" and
"production routing table" — pure fabrication
Task 2 "Verify pr_audit mode wired":
WITH: correct crates/gateway/src/main.rs path + lakehouse_answers_v1
WITHOUT: same assertion, no proof, asserts confidently
Task 3 "Audit phase 40 PRD circuit breaker drift":
WITH: anchored on the actual audit finding "no breaker class found"
WITHOUT: invented "0.0% failure rate vs 5.0% threshold" and signed
off as PASS on broken code — exact failure mode the
distillation pipeline was built to prevent
Both runs passed the structural validation gate (length, no hedges,
checklist token overlap) — the difference is grounding, supplied by
the retrieval layer pulling from exports/rag/playbooks.jsonl (446
records from earlier Phase 4 export).
Architecture:
jaccard token overlap against rag corpus → top-K (default 8) split
into accepted exemplars (top 3) + partial-warnings (top 2) + extracted
validation_steps (lines starting verify|check|assert|ensure|confirm)
→ prompt assembly → qwen3.5:latest via /v1/chat (or OpenRouter
for namespaced/free models) → deterministic validation gate →
escalation to deepseek-v3.1:671b on fail with --allow-escalation
→ log to data/_kb/replay_runs.jsonl
Spec invariants enforced:
- never bypass retrieval (--no-retrieval is explicit baseline, not default)
- never discard provenance (task_hash + rag_ids + full bundle logged)
- never allow free-form hallucinated output (validation gate is
deterministic code, never an LLM)
- log every run as new evidence (replay_run.v1 schema, append-only
to data/_kb/replay_runs.jsonl)
CLI:
./scripts/distill replay --task "<input>" [--local-only]
[--allow-escalation]
[--no-retrieval]
What this unlocks:
The substrate for "small-model bootstrapping" and "local inference
dominance" J flagged after Phase 5. Phase 8+ closes the loop:
schedule replay runs on common tasks, score outputs, feed accepted
ones back into corpus, measure escalation rate decreasing over time.
Known limitations (documented in report):
- Validation gate is structural not semantic (catches hedges/empty
but not plausible-wrong). Phase 13 wiring: run auditor against
every replay output.
- Retrieval is jaccard keyword. Works at 446 corpus, scale via
/vectors/search HNSW retrieval once corpus crosses ~10k.
- Convergence claim is architectural (deterministic retrieval +
low-temp call); longitudinal empirical study is Phase 8+.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
20a039c379 |
auditor: rebuild on mode runner + drop tree-split (use distillation substrate)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"
Architectural simplification leveraging Phase 5 distillation work: the auditor no longer pre-extracts facts via per-shard summaries because lakehouse_answers_v1 (gold-standard prior PR audits + observer escalations corpus) supplies cross-PR context through the mode runner's matrix retrieval. Same signal, ~50× fewer cloud calls per audit. Per-audit cost: Before: 168 gpt-oss:120b shard summaries + 3 final inference calls After: 3 deepseek-v3.1:671b mode-runner calls (full retrieval included) Wall-clock on PR #11 (1.36MB diff): Before: ~25 minutes After: 88 seconds (3/3 consensus succeeded) Files: auditor/checks/inference.ts - Default MODEL kimi-k2:1t → deepseek-v3.1:671b. kimi-k2 is hitting sustained Ollama Cloud 500 ISE (verified via repeated trivial probes; multi-hour outage). deepseek is the proven drop-in from Phase 5 distillation acceptance testing. - Dropped treeSplitDiff invocation. Diff truncates to MAX_DIFF_CHARS and goes straight to /v1/mode/execute task_class=pr_audit; mode runner pulls cross-PR context from lakehouse_answers_v1 via matrix retrieval. SHARD_MODEL retained for legacy callCloud compatibility (default qwen3-coder:480b if it ever runs). - extractAndPersistFacts now reads from truncated diff (no scratchpad post-tree-split-removal). auditor/checks/static.ts - serde-derived struct exemption (commit |
||
|
rootandClaude Opus 4.7
|
1b433a9308 |
distillation: Phase 6 — acceptance gate suite
End-to-end fixture-driven gate. Runs the entire pipeline (collect →
score → export-rag → export-sft → export-preference) on a deterministic
fixture, asserts 22 invariants, runs a SECOND time with the same
recorded_at, and verifies hash reproducibility. Exits non-zero on any
failure. Pure observability — no scoring/filtering/schema changes.
Files (3 new + 1 modified + 6 fixture jsonls):
scripts/distillation/acceptance.ts 330 lines, runner + 22 checks
reports/distillation/phase6-acceptance-report.md autogenerated by run
scripts/distillation/distill.ts +run-all, +receipts, +acceptance subcommands
tests/fixtures/distillation/acceptance/data/_kb/
scrum_reviews.jsonl 5 rows (accepted/partial/needs_human/scratchpad/missing-provenance)
audits.jsonl 3 rows (info/high+PRD-drift/medium severity)
auto_apply.jsonl 2 rows (committed, build_red_reverted)
contract_analyses.jsonl 2 rows (accept, reject)
observer_reviews.jsonl 2 rows (accept, reject — pair candidates)
distilled_facts.jsonl 1 extraction-class row
Spec cases covered (now.md Phase 6):
✓ accepted — Row #1 scrum, #6 audit-info, #11 contract-accept, #14 obs-accept
✓ partially_accepted — Row #2 scrum (3 attempts), #8 audit-medium
✓ rejected — #7 audit-high, #10 auto_apply build_red, #12 contract-reject, #15 obs-reject
✓ needs_human_review — #3 scrum (no markers), #13 distilled extraction-class
✓ missing provenance — Row #5 scrum (no reviewed_at) → routed to skips
✓ valid preference pair — observer_reviews accept+reject on same file
✓ invalid preference pair — quarantine reasons populated when generated
✓ scratchpad / tree-split — Row #4 scrum tree_split_fired=true with multi-shard text
✓ PRD drift — Row #7 audit severity=high, topic="PRD drift: circuit breaker shipped claim"
Acceptance run results (run_id: acceptance-run-1-stable):
22/22 invariants PASS
Pipeline counts:
collect: 14 records out, 1 skipped (missing-provenance fixture)
score: accepted=6 rejected=4 quarantined=4
export-rag: 7 rows (5 acc + 2 partial, ZERO rejected)
export-sft: 5 rows (all 'accepted', ZERO partial without --include-partial)
export-preference: 2 pairs (zero self-pairs, zero identical-text)
Hash reproducibility — bit-for-bit identical:
run_hash: 3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2
Two runs of the entire pipeline on the same fixture with the same
recorded_at produce byte-identical outputs.
The 22 invariants:
1-4. Receipts + summary.json + summary.md + drift.json exist
5-7. StageReceipt + RunSummary + DriftReport schemas all valid
8-10. SFT contains accepted only — no rejected/needs_human/partial leak
11-12. RAG contains accepted+partial — zero rejected
13-15. Preference: ≥1 pair, zero self-pairs, zero identical text
16. Every export row has 64-char hex provenance.sig_hash
17. Phase 2 missing-provenance row routed to distillation_skips.jsonl
18. SFT quarantine populated (6 unsafe_sft_category entries)
19. Scratchpad/tree-split fixture row materialized
20. PRD drift fixture row materialized
21. Per-stage output_hash identical across runs (0 mismatches)
22. run_hash identical across runs (bit-for-bit)
CLI:
./scripts/distill.ts acceptance # exits 0 on pass, 1 on fail
./scripts/distill.ts run-all # full pipeline with receipts
./scripts/distill.ts receipts --run-id <id>
Cumulative test metrics:
135 distillation tests pass · 0 fail · 353 expect() calls · 1411ms
(Phase 6 adds the runtime acceptance gate, not new unit tests —
the acceptance script IS the integration test, callable from CI.)
What this proves:
- Distillation pipeline is SAFE (contamination firewall held under
adversarial fixture)
- Distillation pipeline is REPRODUCIBLE (identical input → bit-identical
output across two runs)
- Distillation pipeline is GATED (every now.md invariant has a
deterministic assertion that exits non-zero on failure)
The 6-phase distillation substrate is now training-safe. RAG (446),
SFT (351 strict-accepted), and Preference (83 paired) datasets on
real lakehouse data each carry full provenance back to source rows
through the verified Phase 2 → Phase 3 → Phase 4 chain, with Phase 5
receipts capturing every input/output sha256 + per-stage validation,
and Phase 6 proving the whole chain is gate-tight on a deterministic
fixture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
2cf359a646 |
distillation: Phase 5 — receipts harness (system-level observability)
Forensic-grade per-stage receipts wrapping all 5 implemented pipeline
stages. Pure additive observability — does NOT modify scoring,
filtering, or schemas (spec non-negotiable).
Files (6 new):
auditor/schemas/distillation/stage_receipt.ts StageReceipt v1
auditor/schemas/distillation/run_summary.ts RunSummary v1
auditor/schemas/distillation/drift_report.ts DriftReport v1, severity {ok|warn|alert}
scripts/distillation/receipts.ts runAllWithReceipts + buildDrift + CLI
tests/distillation/receipts.test.ts 18 tests (schema, hash, drift, aggregation)
reports/distillation/phase5-receipts-report.md acceptance report
Stages wrapped:
collect (build_evidence_index → data/evidence/)
score (score_runs → data/scored-runs/)
export-rag (exports/rag/playbooks.jsonl)
export-sft (exports/sft/instruction_response.jsonl)
export-preference (exports/preference/chosen_rejected.jsonl)
Reserved (not yet implemented): extract-playbooks, index.
Output tree (per run_id):
reports/distillation/<run_id>/
collect.json score.json export-rag.json export-sft.json export-preference.json
summary.json summary.md drift.json
Test metrics: 135 distillation tests pass · 0 fail · 353 expects · 1.5s
(Phase 5 added 18; total 117→135)
Real-data run-all (run_id=78072357-835d-...):
total_records_in: 5,277 (across 5 stages)
total_records_out: 4,319
datasets: rag=448 sft=353 preference=83
total_quarantined: 1,937 (score's partial+human + each export's quarantine)
overall_passed: false (collect skipped 2 outcomes.jsonl rows missing created_at —
carry-over from Phase 2; faithfully propagated)
run_hash: 7a14d8cdd6980048a075efe97043683a4f9aabb38ec1faa8982c9887593090e0
Drift detection (second run):
prior_run_id detected automatically
severity=ok (no count or category swung >20%)
flags: ["run_hash differs from prior run"] — expected, since recorded_at
is baked into provenance and changes per run. No false alert.
Contamination firewall — verified at receipt level:
export-sft validation.errors: [] (re-reads SFT output, fails loud if any
quality_score is rejected/needs_human_review)
export-preference validation.errors: [] (re-reads, fails loud if any
chosen_run_id == rejected_run_id or chosen text == rejected text)
Invariants enforced (proven by tests + real run):
- Every stage emits ONE receipt per run (5/5 on disk)
- All receipts share run_id (uuid generated per run-all)
- aggregateIoHash is order-independent + collision-free across path/content
- Schema validators gate every receipt before write (defense in depth)
- Drift detection: pct_change > 20% → warn; new error class → warn
- Failure propagation: any stage validation.passed=false → overall_passed=false
- Self-validation: harness throws if RunSummary/DriftReport fail their own schema
CLI:
bun run scripts/distillation/receipts.ts run-all
bun run scripts/distillation/receipts.ts read --run-id <id>
Spec acceptance gate (now.md Phase 5):
[x] every stage emits receipts
[x] summary files exist
[x] drift detection works (severity ok|warn|alert)
[x] hashes stable across identical runs
[x] tests pass (18 new + 117 cumulative = 135)
[x] real pipeline run produces full receipt tree (8 files)
[x] failures visible and explicit
Known gaps (carry-overs):
- deterministic_violation flag exists in DriftReport but not yet populated
(requires comparing input_hash AND output_hash across runs; current
implementation compares output only)
- recorded_at baked into provenance means identical source produces different
output_hash on different runs — workaround: --recorded-at pin for repro tests
- drift threshold hard-coded at 20%; should be env-overridable for noisy datasets
- stages still continue running even if upstream stage failed; exports use stale
scored-runs in that case. Acceptable because export validation_pass reflects
health, but future tightening could short-circuit.
Phase 6 (acceptance gate suite) unblocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
68b6697bcb |
distillation: Phase 4 — dataset export layer
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Build the contamination firewall: RAG, SFT, and Preference exporters
that turn scored evidence into clean training datasets without
leaking rejected, unvalidated, hallucinated, or provenance-free
records.
Files (8 new + 4 schema updates):
scripts/distillation/quarantine.ts shared QuarantineWriter, 11-reason taxonomy
scripts/distillation/export_rag.ts RAG exporter (--include-review opt-in)
scripts/distillation/export_sft.ts SFT exporter (--include-partial opt-in, SFT_NEVER constant)
scripts/distillation/export_preference.ts preference exporter, same task_id pairing
scripts/distillation/distill.ts CLI dispatcher (build-evidence/score/export-*)
tests/distillation/exports.test.ts 15 contamination-firewall tests
reports/distillation/phase4-export-report.md acceptance report
Schema field-name alignment with now.md:
rag_sample.ts +source_category, exported_at→created_at
sft_sample.ts +id, exported_at→created_at, partially_accepted at schema (CLI gates)
preference_sample.ts +id, source_run_ids→chosen_run_id+rejected_run_id, +created_at
Test metrics: 117 distillation tests pass · 0 fail · 315 expects · 327ms
Real-data export run (1052 scored input rows):
RAG: 446 exported (351 acc + 95 partial), 606 quarantined
SFT: 351 exported (all 'accepted'), 701 quarantined
Preference: 83 pairs exported, 16 quarantined
CONTAMINATION FIREWALL — verified held on real data:
- SFT output: 351/351 quality_score='accepted' (ZERO leaked)
- RAG output: 351 acc + 95 partial (ZERO rejected leaked)
- Preference: 0 self-pairs (chosen_run_id != rejected_run_id)
- 536 rejected+needs_human_review records caught at unsafe_sft_category
gate, exact match to scored-runs forbidden-category total
Defense in depth (the firewall is two layers, not one):
1. Schema layer (Phase 1): SftSample.quality_score enum forbids
rejected/needs_human at write time
2. Exporter layer: SFT_NEVER constant in export_sft.ts checks
category before synthesis. Even if synthesis produced a row
with quality_score=rejected, validateSftSample would reject it.
Quarantine reasons (11): missing_provenance, missing_source_run_id,
empty_content, schema_violation, unsafe_sft_category,
unsafe_rag_category, invalid_preference_pairing,
hallucinated_file_path, duplicate_id, self_pairing,
category_disallowed.
Bug surfaced + fixed during testing: module-level evidenceCache
shared state across test runs (tests wipe TMP, cache holds stale
empty Map). Moved cache to per-call scope. Same pattern bit Phase 2
materializer would have hit if its tests had multiple runs sharing
state — preventive fix.
Pairing logic v1: same task_id with category gap. accepted×rejected
preferred, accepted×partially_accepted as fallback. MAX_PAIRS_PER_TASK=5
cap prevents one hot task from dominating. Future: cross-source
pairing (scrum_reviews chosen vs observer_reviews rejected on same
file) to grow dataset beyond 83.
CLI: ./scripts/distill.ts {build-evidence|score|export-rag|export-sft|export-preference|export-all|health}
Flags: --dry-run, --include-partial (SFT only), --include-review (RAG only)
Carry-overs to Phase 5 (Receipts Harness):
- Each exporter currently writes results but no per-stage receipt.json.
Phase 5 wraps build_evidence_index + score_runs + export_* in a
withReceipt() helper that captures git_sha + sha256 of inputs/outputs
+ record_counts + validation_pass.
- reports/distillation/latest.md aggregating most-recent run of each stage.
Carry-overs to Phase 3 v2:
- mode_experiments scoring (168 needs_human_review): derive markers from
validation_results.grounded_fraction
- extraction-class JOIN: distilled_*/audit_facts/observer_escalations
→ JOIN to verdict-bearing parent by task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
c989253e9b |
distillation: Phase 3 — deterministic Success Scorer
Pure scoreRecord function + score_runs.ts CLI + 38 tests.
Reads data/evidence/YYYY/MM/DD/*.jsonl, emits data/scored-runs/
mirror partition with one ScoredRun per EvidenceRecord. ZERO model
calls. scorer_version stamped on every output (default v1.0.0).
Three-class scoring strategy (taxonomy from Phase 2 evidence_health.md):
CLASS A (verdict-bearing): direct mapping from existing markers.
scrum_reviews: accepted_on_attempt_1 → accepted; 2-3 → partial;
4+ → partial with high-cost reason
observer_reviews: accept|reject|cycle → category
audits: severity info/low → accepted, medium → partial,
high/critical → rejected (legacy markers also handled)
contract_analyses: failure_markers + observer_verdict
CLASS B (telemetry-rich): partial markers, fall back to needs_human
auto_apply: committed → accepted; *_reverted → rejected
outcomes: all_events_ok → accepted; gap_signals > 0 → partial
mode_experiments: empty text → rejected; latency > 120s → partial
CLASS C (extraction): needs_human (Phase 3 v2 will JOIN to parents)
Real-data run on 1052 evidence rows:
accepted=384 (37%) · partial=132 (13%) · rejected=57 (5%) · needs_human=479 (45%)
Verdict-bearing sources land 0% needs_human:
scrum_reviews (172): 111 acc · 61 part · 0 rej · 0 hum
audits (264): 217 acc · 29 part · 18 rej · 0 hum
observer_reviews (44): 22 acc · 3 part · 19 rej · 0 hum
contract_analyses (2): 1 acc · 0 part · 1 rej · 0 hum
BUG SURFACED + FIXED:
Phase 2 transform for audits.jsonl assumed PR-verdict shape (recon
misnamed it). Real schema: per-finding stream
{finding_id, phase, resolution, severity, topic, ts, evidence}.
Updated transform to derive markers from severity. 264 findings
went 0% scoreable → 100% scoreable. Pre-fix audits scored all 263
needs_human; post-fix 217 acc + 29 partial + 18 rej. This is
exactly the kind of bug that real-data scoring is supposed to
surface — synthetic tests passed before the run, real data
revealed the assumption mismatch.
Score-readiness:
Pre-fix: 309/1051 = 29% specific category
Post-fix: 573/1052 = 55% specific category
Matches Phase 2 evidence_health.md prediction (~54% scoreable)
Test metrics:
51 distillation tests pass (10 evidence_record + 30 schemas + 8 realdata
+ 9 build_evidence_index + 30 scorer + 8 score_runs + 21 inferred from earlier
files; bun test reports 51 across 3 phase-3 files alone)
192 expect() calls
399ms total
Receipts:
reports/distillation/2026-04-27T03-44-26-602Z/receipt.json
- record_counts.cat_accepted=384, cat_partially_accepted=132,
cat_rejected=57, cat_needs_human_review=479
- validation_pass=true (0 skips)
- self-validates against Receipt schema before write
Carry-overs to Phase 4+:
- mode_experiments 166 needs_human: derive grounding from validation_results
- extraction-class 207 rows: JOIN to verdict-bearing parent by task_id
- audit_discrepancies transform (still missing — Phase 4c needs)
- model_trust transform (needed for ModelLedgerEntry aggregation)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
1ea802943f |
distillation: Phase 2 — Evidence View materializer + health audit
Phase 2 ships the JOIN script that turns 12 source JSONL streams
into unified data/evidence/YYYY/MM/DD/<source>.jsonl rows conforming
to EvidenceRecord v1, plus a high-level health audit proving the
substrate is real before Phase 3 reads from it.
Files:
scripts/distillation/build_evidence_index.ts materializeAll() + cli
scripts/distillation/check_evidence_health.ts provenance + coverage audit
tests/distillation/build_evidence_index.test.ts 9 acceptance tests
Test metrics:
9/9 pass · 85 expect() calls · 323ms
Real-data run (2026-04-27T03:33:53Z):
1053 rows read from 12 source streams
1051 written (99.8%) to data/evidence/2026/04/27/
2 skipped (outcomes.jsonl rows missing created_at — schema-level catch)
0 deduped on first run
Sources covered (priority order from recon):
TIER 1 (validated 100% in Phase 1, 8 sources):
distilled_facts/procedures/config_hints, contract_analyses,
mode_experiments, scrum_reviews, observer_escalations, audit_facts
TIER 2 (added by Phase 2):
auto_apply, observer_reviews, audits, outcomes
High-level audit results:
Provenance round-trip: 30/30 sampled rows trace cleanly to source
rows with matching canonicalSha256(orderedKeys(row)). Every output
has source_file + line_offset + sig_hash + recorded_at. Proven.
Score-readiness: 54% aggregate scoreable. Three-class taxonomy
emerges from coverage matrix:
- Verdict-bearing (100% scoreable): scrum_reviews, observer_reviews,
audits, contract_analyses — direct scoring inputs
- Telemetry-rich (0-70%): mode_experiments, audit_facts, outcomes
— Phase 3 will derive markers from latency/grounding/retrieval
- Pure-extraction (0%): distilled_*, observer_escalations
— context for OTHER scoring, not scoreable themselves
Invariants enforced (proven by tests + real-data audit):
- ZERO model calls in materializer (deterministic only)
- canonicalSha256(orderedKeys(row)) per source row → stable sig_hash
- Schema validator gates output: rejected rows go to skips, never to evidence/
- JSON.parse failures caught + logged, never crash the run
- Missing source files tallied as rows_present=false, never error
- Idempotent: second run on identical input writes 0 rows (proven on
real data: 1053 read, 0 written, 1051 deduped)
- Bit-stable: identical input produces byte-identical output (proven
by tests/distillation/build_evidence_index.test.ts case 3)
- Receipt self-validates against schema before write
- validation_pass = boolean (skipped == 0), never inferred
Receipt at:
reports/distillation/2026-04-27T03-33-53-972Z/receipt.json
- schema_version=1, git_sha pinned, sha256 on every input/output
- record_counts: {in:1053, out:1051, skipped:2, deduped:0}
- validation_pass=false (skipped > 0; spec says explicit, never inferred)
Skips at:
data/_kb/distillation_skips.jsonl (2 rows from outcomes.jsonl,
reason: timestamp field missing — schema layer caught it cleanly)
Health audit at:
data/_kb/evidence_health.md
Phase 2 done-criteria all met:
✓ tests pass
✓ ≥1 row from each Tier-1 source on real data (8/8 + 4 Tier 2 bonus)
✓ data/_kb/distillation_skips.jsonl populated with reasons
✓ Receipt JSON written + self-validates
✓ Provenance round-trip proven on real sampled rows
✓ Score-readiness coverage measured
Carry-overs to Phase 3:
- audit_discrepancies transform (needed before Phase 4c preference data)
- model_trust transform (needed before ModelLedgerEntry aggregation)
- outcomes.jsonl created_at: 2 rows fail materialization, decide
transform-side fix vs source-side fix
- 11 untested streams from recon still have no transform; add as
Phase 3+ consumers need them
- mode_experiments + distilled_* are 0% scoreable; Phase 3 must
JOIN to adjacent verdict-bearing records, NOT score in isolation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
27b1d27605 |
distillation: Phase 0 recon + Phase 1 schemas + Phase 2 transforms scaffold
lakehouse/auditor 9 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Phase 0 — docs/recon/local-distillation-recon.md
Inventories the 23 KB JSONL streams + 20 vector corpora + auditor's
kb_index.ts as substrate for the now.md distillation pipeline. Maps
spec modules to existing producers, identifies real gaps, lists 9
schemas to formalize. ZERO implementation in recon — gating doc only.
Phase 1 — auditor/schemas/distillation/
9 schemas + foundation types + 48 tests passing in 502ms:
types.ts shared validators + canonicalSha256
evidence_record.ts EVIDENCE_SCHEMA_VERSION=1, ModelRole enum
scored_run.ts 4 categories pinned, anchor_grounding ∈ [0,1]
receipt.ts git_sha 40-char, sha256 file refs, validation_pass:bool
playbook.ts non-empty source_run_ids + acceptance_criteria
scratchpad_summary.ts validation_status enum, hash sha256
model_ledger.ts success_rate ∈ [0,1], sample_count ≥ 1
rag_sample.ts success_score ∈ {accepted, partially_accepted}
sft_sample.ts quality_score MUST be 'accepted' (no leak)
preference_sample.ts chosen != rejected, source_run_ids must differ
evidence_record.test.ts 10 tests, JSON-fixture round-trip
schemas.test.ts 30 tests, inline fixtures
realdata.test.ts 8 tests, real-JSONL probe
Real-data validation probe (one of the 3 notables from recon):
46 rows across 7 sources, 100% pass. distilled_facts/procedures alive.
Report at data/_kb/realdata_validation_report.md (also written by the
test). Confirms schema fits existing producers without migration.
Phase 2 scaffold — scripts/distillation/transforms.ts
Promoted PROBES from realdata.test.ts into a real TRANSFORMS array
covering 12 source streams (8 Tier 1 validated + 4 Tier 2 from
recon's untested-streams list). Pure functions: no I/O, no model
calls, no clock reads. Caller supplies recorded_at + sig_hash so
materializer is deterministic by construction.
Spec non-negotiables enforced at schema layer (defense in depth):
- provenance{source_file, sig_hash, recorded_at} required everywhere
- schema_version mismatch hard-rejects (forward-compat gate)
- SFT no-leak: validateSftSample REJECTS partially_accepted, rejected,
needs_human_review — three explicit tests
- Every score has WHY (reasons non-empty)
- Every playbook traces to source (source_run_ids non-empty)
- Every preference has WHY (reason non-empty)
- Receipts substantive (git_sha 40-char, sha256 64-char, validation_pass:bool)
Branch carries uncommitted auditor rebuild work (mode.rs + modes.toml
+ inference.ts + static.ts) blocked on upstream Ollama Cloud kimi-k2
500 ISE; held pending recon-driven design decisions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
f753e11157 |
docs: SCRUM_MASTER_SPEC timeline — productization wave + verified live state
lakehouse/auditor 9 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Splits the existing 04-25/26 section into two waves:
- experiment wave (mode-runner build-out, pre-productization)
- productization wave (OpenAI-compat, Archon, answers corpus,
staffing native runner, multi-corpus + downgrade gate, observer
paid escalation, /v1/chat → observer event wiring)
Adds verified-live block at the end with the numbers a fresh session
needs to anchor on: pathway memory 88 traces / 11 successful replays
at 100% (probation gate crossed), strong-model auto-downgrade firing
on grok-4.1-fast, and the auditor blind spot at static.ts:117 (now
fixed in
|