c989253e9b0392439947df4b4e440d8c36ff7fca
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rootandClaude Opus 4.7
|
27b1d27605 |
distillation: Phase 0 recon + Phase 1 schemas + Phase 2 transforms scaffold
lakehouse/auditor 9 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Phase 0 — docs/recon/local-distillation-recon.md
Inventories the 23 KB JSONL streams + 20 vector corpora + auditor's
kb_index.ts as substrate for the now.md distillation pipeline. Maps
spec modules to existing producers, identifies real gaps, lists 9
schemas to formalize. ZERO implementation in recon — gating doc only.
Phase 1 — auditor/schemas/distillation/
9 schemas + foundation types + 48 tests passing in 502ms:
types.ts shared validators + canonicalSha256
evidence_record.ts EVIDENCE_SCHEMA_VERSION=1, ModelRole enum
scored_run.ts 4 categories pinned, anchor_grounding ∈ [0,1]
receipt.ts git_sha 40-char, sha256 file refs, validation_pass:bool
playbook.ts non-empty source_run_ids + acceptance_criteria
scratchpad_summary.ts validation_status enum, hash sha256
model_ledger.ts success_rate ∈ [0,1], sample_count ≥ 1
rag_sample.ts success_score ∈ {accepted, partially_accepted}
sft_sample.ts quality_score MUST be 'accepted' (no leak)
preference_sample.ts chosen != rejected, source_run_ids must differ
evidence_record.test.ts 10 tests, JSON-fixture round-trip
schemas.test.ts 30 tests, inline fixtures
realdata.test.ts 8 tests, real-JSONL probe
Real-data validation probe (one of the 3 notables from recon):
46 rows across 7 sources, 100% pass. distilled_facts/procedures alive.
Report at data/_kb/realdata_validation_report.md (also written by the
test). Confirms schema fits existing producers without migration.
Phase 2 scaffold — scripts/distillation/transforms.ts
Promoted PROBES from realdata.test.ts into a real TRANSFORMS array
covering 12 source streams (8 Tier 1 validated + 4 Tier 2 from
recon's untested-streams list). Pure functions: no I/O, no model
calls, no clock reads. Caller supplies recorded_at + sig_hash so
materializer is deterministic by construction.
Spec non-negotiables enforced at schema layer (defense in depth):
- provenance{source_file, sig_hash, recorded_at} required everywhere
- schema_version mismatch hard-rejects (forward-compat gate)
- SFT no-leak: validateSftSample REJECTS partially_accepted, rejected,
needs_human_review — three explicit tests
- Every score has WHY (reasons non-empty)
- Every playbook traces to source (source_run_ids non-empty)
- Every preference has WHY (reason non-empty)
- Receipts substantive (git_sha 40-char, sha256 64-char, validation_pass:bool)
Branch carries uncommitted auditor rebuild work (mode.rs + modes.toml
+ inference.ts + static.ts) blocked on upstream Ollama Cloud kimi-k2
500 ISE; held pending recon-driven design decisions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
rootandClaude Opus 4.7
|
107a68224d |
auditor: skip serde-derived structs in unread-field check
Fields on structs that derive Serialize or Deserialize ARE read — by
the macro, on every JSON round-trip — but the static check only
looked for explicit `.field` references in the diff. Result: every
new response/request struct shipped through `/v1/*` was flagged as
"placeholder state without a consumer."
PR #11 head
|
||
 profit
|
7c1745611a |
Audit pipeline PR #9: determinism + fact extraction + verifier gate + KB stats + context injection (PR #9)
Bundles PR #9's work for the audit pipeline: - N=3 consensus on cloud inference (gpt-oss:120b parallel) with qwen3-coder:480b tie-breaker - audit_discrepancies.jsonl logs N-run disagreements - scrum_master reviews route through llm_team fact extraction; source="scrum_review" - Verifier-gated persistence: drops INCORRECT, keeps UNVERIFIABLE/UNCHECKED; schema_version:2 - scrum_master_reviewed flag on accepted reviews - auditor/kb_stats.ts: on-demand observability script - claim_parser history/proof pattern class (verified-on-PR, was-flipping, the-proven-X) - claim_parser quoted-string guard (mirrors static.ts fix) - fact_extractor project context injection via docs/AUDITOR_CONTEXT.md - Fixed verifier-verdict parser to handle multiple gemma2 output formats Empirical: 3-run determinism test on unchanged PR #9 SHA showed 7/7 warn findings stable; block count oscillation eliminated; llm_team quality scores 8-9 on context-injected extract runs. See PR #9 for full run-by-run commit history. |
||
|
profit
|
156dae6732 |
Auditor self-test branch: real-world pipelines + cohesion Phase C + KB index (PR #8)
Bundles 12 commits validating the auditor + scrum_master architecture end-to-end: - enrich_prd_pipeline / hard_task_escalation / scrum_master_pipeline stress tests - Tree-split + scrum_reviews.jsonl + kb_query surfacing - Verdict → audit_lessons feedback loop (closed) - kb_index aggregator with confidence-based severity policy - 9-run + 5-run empirical tests proved the predictive-compounding property - Level 1 correction: temp=0 cloud inference for deterministic per-claim verdicts - audit_one.ts dry-run CLI - Fixes: static quoted-string guard, empirical-claim classification, symbol-resolver gate, repo-file size cap See PR #8 for run-by-run commit history. |
||
|
profit
|
c33c1bcbc5 |
Auditor: poller + live end-to-end proof
lakehouse/auditor all checks passed (4 findings, all info)
auditor/index.ts (task #9) — the top-level poller. 90s interval, dedupes by head SHA via data/_auditor/state.json, supports --once for CLI testing. Env gates: LH_AUDITOR_RUN_DYNAMIC=1 to include the hybrid fixture (default off; it mutates live state), LH_AUDITOR_SKIP_INFERENCE=1 for fast runs without cloud calls. Single-shot run proof (task #10): cycle 1: 2 open PRs audit PR #2 |
||
|
profit
|
039ed32411 |
Auditor: KB query check + verdict orchestrator + Gitea poster
lakehouse/auditor all checks passed (4 findings, all info)
auditor/checks/kb_query.ts (task #7) — reads data/_kb/outcomes.jsonl, error_corrections.jsonl, data/_observer/ops.jsonl, data/_bot/cycles/*. Cheap/offline: no model calls, tail-reads only. Fail-rate >30% in recent scenario outcomes → warn; otherwise info. Live-proven: 1 finding emitted against current KB state (69 scenario runs, 27.7% fail rate — below warn threshold). auditor/audit.ts (task #8) — orchestrator. Runs static + dynamic + inference + kb_query in parallel, calls assembleVerdict, persists to data/_auditor/verdicts/, posts to Gitea (commit status + issue comment). AuditOptions supports skip_dynamic/skip_inference/dry_run for iteration. auditor/gitea.ts — added postIssueComment (author can comment on own PR, unlike postReview which self-review-blocks). static.ts — skip BLOCK_PATTERNS scan on auditor/checks/* and auditor/fixtures/* because those files legitimately contain the patterns as regex/string-literal data. WARN/INFO patterns (TODO comments, hardcoded placeholders) still run. Live-proven: dry-run audit of PR #1 after fix went from 13 block findings to 0 from static; 11 warn from inference still fire on real overreach claims. Dry-run audit against PR #1, skip_dynamic=true: verdict: block (BEFORE the static fix) verdict: request_changes (AFTER — inference correctly flagged "tasks 1-9 complete" as not backed; 0 false-positive blocks from static self-match) 42.5s total across checks (mostly cloud inference: 36s) 26 claims, 39KB diff Tasks 5 + 6 + 7 + 8 complete. Remaining: #9 (poller) + #10 (end-to-end proof) + #12 (upsert UPDATE merge fix). |
||
|
profit
|
efc7b5ac44 |
Auditor: dynamic + inference checks
auditor/checks/dynamic.ts — wraps runHybridFixture, maps layer results to Findings. Placeholder-style errors (404/unimplemented/ slice N) → info; other failures → warn. Always emits a summary finding with real numbers (shipped/placeholder phase counts + per- layer latency). Live-tested against current stack: 2 info findings, 0 warnings — all shipped layers actually work. auditor/checks/inference.ts — wraps the run_codereview reviewer pattern from llm_team_ui.py, adapted for claim-vs-diff verification. Calls /v1/chat provider=ollama_cloud model=gpt-oss:120b. Requests strict JSON response with claim_verdicts[] and unflagged_gaps[]. A strong claim marked "not backed" by cloud → BLOCK severity; moderate → warn; weak → info. Cloud-unreachable or unparseable-output → info (never blocks on the reviewer being down). Live-tested against PR #1 (this PR, 20 claims, 39KB diff): - 36.9s round-trip - 7 block + 23 warn + 2 info findings - gpt-oss:120b correctly flagged "Fully-functional auditor (tasks 1-9 complete)" as not-backed (only 6/10 tasks done at that commit) — accurate catch - Some false positives from the original 15KB truncation threshold (cloud missed gitea.ts, flagged "no Gitea client present") - Bumped MAX_DIFF_CHARS from 15000 to 40000 to fit the full PR diff in context; reviewer precision improves accordingly Tasks 5 + 6 completed. Remaining: #7 (KB query), #8 (verdict + Gitea poster), #9 (poller), #10 (end-to-end proof), #12 (upsert UPDATE-drops-doc_refs). |
||
|
profit
|
c5da680add |
Fixture: unique-per-run nonce eliminates state-pollution false positive
After the serde fix (PR #2, fix/upsert-outcome-serde) landed on main, re-running this fixture STILL reported "doc_refs field is empty" — but with a different root cause than the panic. Root cause: pre-fix runs panicked on response serialization but had already added entries to state (panic happened between upsert_entry returning and the handler's serde_json::json! of the response). So state.json was polluted with __auditor_test_worker__ entries from those runs, WITHOUT doc_refs (doc_refs wasn't even wired at the time those state rows were written). The fixture's `find(endorsed_names.includes(TEST_WORKER_NAME))` was picking the oldest polluted entry, not the fresh one. Compounding: discovered a secondary bug while investigating — upsert_entry's UPDATE branch only merges endorsed_names. doc_refs, schema_fingerprint, valid_until on an UPDATE are silently dropped. Filed as task #12, separate PR to follow. Fix in this fixture: use a nonce suffix on both TEST_WORKER_NAME and TEST_OPERATION so every run is guaranteed to hit the ADD path in upsert_entry, sidestepping the UPDATE bug AND eliminating state pollution entirely. Live re-run after this edit: ✓ Phase 38 /v1/chat 449ms, 42 tokens ✓ Phase 40 Langfuse trace 20ms ✓ Phase 45.1 seed + doc_refs 239ms, doc_refs.length=1 persisted ✓ Phase 45.2 bridge diff 2ms, drifted=true ✗ Phase 45.3 drift-check HONEST 404 (endpoint not built) shipped_phases: [38, 40, 45.1, 45.2] (was [38, 40, 45.2]) placeholder: [45.3] (was [45.1, 45.3]) One fewer placeholder — exactly because the serde fix merged on fix/upsert-outcome-serde and the fixture now cleanly exercises the path. The loop is: fixture finds bug → PR fixes bug → fixture re-run confirms fix → one fewer placeholder. |
||
|
profit
|
5bbcaf4c33 |
Fix: layer-2 Langfuse filter used meaningless ternary
Caught by running a side-test through LLM Team's run_codereview
flow (gpt-oss:120b reviewer) against this fixture, 2026-04-22.
BEFORE:
const ourStart = Date.parse(
l1.evidence.match(/tokens=/) ? result.ran_at : result.ran_at
);
// Both branches return result.ran_at — the ternary is meaningless.
// result.ran_at is the fixture start time, NOT the moment we fired
// /v1/chat. Any trace created between fixture-start and chat-fetch
// would false-negative.
AFTER:
const chat_request_sent_ms = Date.now(); // captured before layer 1
// ...
const recent = items.filter(t =>
Date.parse(t.timestamp) >= chat_request_sent_ms
);
Re-ran the fixture against the live stack — layers 1,2,4 still pass
(no regression); layer 2 trace matched at age=2494ms which is within
the chat-to-trace propagation window. Layers 3,5 still fail for the
original unrelated reasons (UpsertOutcome serde panic + Phase 45
slice 3 endpoint not built).
First concrete act-on-finding from a code-checker run. The process
works.
|
||
|
profit
|
9c893fbb8c |
Auditor: hybrid fixture — found a pre-existing bug on first live run
auditor/fixtures/hybrid_38_40_45.ts — the never-before-run hybrid
test. Exercises Phase 38 /v1/chat → Phase 40 Langfuse → Phase 45
slice 1 seed+doc_refs → Phase 45 slice 2 bridge drift → (expected-
fail) Phase 45 slice 3 drift-check endpoint.
auditor/fixtures/cli.ts — standalone runner. Human-readable summary
to stderr, machine-readable JSON to stdout, exit code 0/1/2 for
pass / fail / partial_pass.
Live run results — honest measurements, not hand-waved:
✓ Phase 38 /v1/chat returns 9 visible tokens, 6.7s latency
("docker run is a common Docker command.")
✓ Phase 40 Langfuse trace 18a8a0b7 landed in 2.5s
✗ Phase 45.1 seed endpoint returns empty reply — discovered a
PRE-EXISTING BUG unrelated to doc_refs:
playbook_memory.rs:257 UpsertOutcome has newtype
variants Added(String) and Noop(String) under
#[serde(tag="mode")] — serde panics on serialize.
panicked at crates/vectord/src/service.rs:2323:
Error("cannot serialize tagged newtype variant
UpsertOutcome::Added containing a string")
Reproduced: curl /seed with AND without doc_refs
both get "Empty reply from server" (socket closed
mid-response). This bug has existed since Phase 26
shipped (commit
|
||
|
profit
|
b933334ae2 |
Auditor: static diff check — catches own Phase 45 placeholder
auditor/checks/static.ts — grep-style scan of PR diffs, no AST,
no LLM. High-signal patterns only.
Severity grading:
- BLOCK — unimplemented!(), todo!(), panic!("not implemented"),
throw new Error("not implemented")
- WARN — TODO/FIXME/XXX/HACK in added lines;
new pub struct fields with <2 mentions in the diff
(added but nobody reads it — placeholder state)
- INFO — hardcoded "placeholder"/"dummy"/"foobar"/"changeme"/"xxx"
strings in added lines
Live-proven — the existential test J asked for:
vs PR #1 (scaffold): 0 findings (all scaffold fields cross-
reference within the diff)
vs commit
|
||
|
profit
|
bfe8985233 |
Auditor: claim parser
auditor/claim_parser.ts — reads PR body + commit messages, extracts
ship-claims. Regex-based, intentionally not LLM-driven: the parser's
job is to surface claim substrates, not to judge them (that's the
inference check's job, runs later with cloud model).
Three strength tiers:
- strong — "verified end-to-end", "live-proven", "production-ready",
"phase N shipped", "proven"
- moderate — "shipped", "landed", "green", "passing", "works",
"complete", "done"
- weak — "should work", "expected to", "probably"
Live-proven against PR #1 (this PR): 4 claims extracted from
1 commit (2 strong, 2 moderate). "live-proven" correctly tagged as
strong (it IS a stronger claim than "shipped").
Next: static diff check consumes these claims + the PR diff to find
placeholder patterns — empty fns, TODO, unwired fields, etc.
|
||
|
profitandClaude Opus 4.7
|
f48dd2f20b |
Auditor scaffold: types + Gitea client + policy stub + README
All-Bun sub-agent that watches open PRs on Gitea, reads ship-claims, and hard-blocks merges when the code doesn't back the claim. First commit of N; this is the skeleton. Dynamic/static/inference/kb checks + poller land in follow-up commits on this same branch. - auditor/types.ts — Claim, Finding, Verdict, PrSnapshot shapes - auditor/gitea.ts — minimal API client (listOpenPrs, getPrDiff, postCommitStatus, postReview). Live-proven: returned 0 open PRs against our repo (which IS the current state — every commit today went to main directly, which is the problem this auditor is meant to prevent) - auditor/policy.ts — stub `assembleVerdict` + severity rules. Intentionally conservative defaults: strong claim + zero evidence = block, not warn. - auditor/README.md — how to run + the hard-block mechanism Workflow discipline change: starting with this branch, no more direct pushes to main. Every change lands as a PR. When this auditor is fully built and running, it'll review its own completion PR — the recursive self-test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |