181c35b8298eb0c5b6739175f9b32be9bb8b3737
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 profit
|
181c35b829 |
scrum_master fact extraction + verifier gate + schema_version bump
Three bundled changes that round out the KB enrichment pipeline (PR #9 commits B/C/D compressed into one — they all touch the same persist surfaces so splitting them would just add noise): B. scrum_master reviews now route accepted review bodies through fact_extractor (same llm_team extract pipeline as inference) and append to data/_kb/audit_facts.jsonl tagged source:"scrum_review". One KB, two producers — downstream consumers can filter by source when they care about provenance. Skips reviews <120 chars (one-liners / LGTM-type comments with no extractable knowledge). C. Verifier-gated fact persistence. fact_extractor now parses the verifier's free-form prose into per-fact verdicts (CORRECT / INCORRECT / UNVERIFIABLE / UNCHECKED). Facts marked INCORRECT are dropped on write; CORRECT + UNVERIFIABLE + UNCHECKED are kept (dropping UNVERIFIABLE would lose ~90% of real signal — the verifier's prior-knowledge base doesn't know Lakehouse internals, so domain-specific facts read as UNVERIFIABLE by default). verifier_verdicts array is persisted alongside facts so downstream queries can surface high-confidence facts (CORRECT) separately from provisional ones (UNVERIFIABLE). schema_version:2 added to both scrum_reviews.jsonl and audit_facts.jsonl writes. Old (v1) rows remain readable; new rows get the field so the forward-compat reader in kb_query can differentiate. D. scrum_master_reviewed:true flag added to scrum_reviews.jsonl rows on accept. Future kb_query surfacing can filter by this (e.g., "show me PRs where a scrum review exists vs only inference" as governance signal). Also carried into audit_facts.jsonl when the scrum_review source path writes there. |
||
|
profit
|
2afad0f83f |
auditor/inference: N=3 consensus + qwen3-coder:480b tie-breaker
Closes the determinism gap observed in the 3-run baseline test: 1 of 8 findings (the "proven escalation ladder" block) was flipping across identical-state audits. Root cause: cloud non-determinism at temp=0 is real in practice even though it shouldn't be in theory. Fix: run the primary reviewer (gpt-oss:120b) N=3 times in PARALLEL (Promise.all, wall-clock ≈ single call because they're independent HTTP requests). Aggregate votes per claim_idx. Majority wins. On a 1-1-1 split, call a tie-breaker model with different architecture: qwen3-coder:480b — newer coding specialist, 4x params of the primary, distinct training lineage. Every case where the 3 runs disagreed (even when majority resolved) is logged to data/_kb/audit_discrepancies.jsonl with the vote counts and resolution type. This is how we measure consensus drift over time — a dashboard metric is literally `wc -l audit_discrepancies` relative to audit count. Verified: 2 back-to-back audits on unchanged PR #8 produced identical 8 findings each (1 block + 7 warn). consensus=3/3 on every claim, zero discrepancies logged. Cost: 3x primary tokens (7K per audit vs 2K), wall-clock ~unchanged because calls are parallel. New env vars: LH_AUDITOR_CONSENSUS_N default 3 LH_AUDITOR_TIEBREAKER_MODEL default qwen3-coder:480b Factored the cloud call into runCloudInference() helper so the consensus loop is clean and the tie-breaker reuses the same prompt shape as the primary. |
||
|
profit
|
77650c4ba3 |
auditor: inference curation layer + llm_team fact extraction → KB
Closes the cycle J asked for: curated cloud output lands structured knowledge in the KB so future audits have architectural context, not just a log of per-finding signatures. Three pieces: 1. Inference curation (tree-split) — when diff > 30KB, shard at 4.5KB, summarize each shard via cloud (temp=0, think=false on small shards; think=true on main call). Merge into scratchpad. The cloud verification then runs against the scratchpad, not truncated raw. Eliminates the 40KB MAX_DIFF_CHARS truncation path for large PRs (PR #8 is 102KB — was losing 62KB). Anti-false-positive guard in the prompt: cloud is told scratchpad absence is NOT diff absence, so it doesn't flag curated-out symbols as missing. unflagged_gaps section is dropped entirely when curated (scratchpad can't ground them). 2. fact_extractor — TS client for llm_team_ui's extract-facts mode at localhost:5000/api/run. Sends curated scratchpad through qwen2.5 extractor + gemma2 verifier, parses SSE stream, returns structured {facts, entities, relationships, verification, llm_team_run_id}. Best-effort: if llm_team is down, extraction fails silently and the audit still completes. AWAITED so CLI tools (audit_one.ts) don't exit before extraction lands — the systemd poller has 90s headroom so the extra ~15s doesn't matter. 3. audit_facts.jsonl + checkAuditFacts() — one row per curated audit with the extraction result. kb_query tails the jsonl, explodes entity rows, aggregates by entity name with distinct-PR counting, surfaces entities recurring in 2+ PRs as info findings. Filters out short names (<3 chars, extractor truncation artifacts) and generic types (string/number/etc.) so signal isn't drowned. Verified end-to-end on PR #8: 102KB diff → 23 shards → 1KB scratchpad → qwen2.5 extracted 4 facts + 6 entities + 6 relationships (real code-level knowledge: AggregateOptions<T> type, aggregate<T> async function with real signature, typed relationships). llm_team_run_id cross-references to llm_team's own team_runs table. Also: audit.ts passes (pr_number, head_sha) as InferenceContext so extracted facts are scope-tagged for the KB index. |
||
|
profit
|
47f1ca73e7 |
auditor: Level 1 correction — keep think=true, only temp=0 is needed
lakehouse/auditor 4 warnings — see review
The previous Level 1 commit set think=false which broke the cloud inference check on real PR audits. gpt-oss:120b is a reasoning model; at think=false on large prompts (40KB diff + 14 claims) it returned empty content — verified by inspecting verdict 8-8e4ebbe4b38a which showed "cloud returned unparseable output — skipped" with 13421 tokens used and head:<empty>. Small-prompt tests passed because the model could respond without needing to think. Real audits with the full diff + claims context require the reasoning channel to produce any output at all. The determinism we need comes from temp=0 (greedy sampling). The reasoning trace at think=true varies in prose but greedy sampling converges to the same FINAL classification from identical starting state, so signatures remain stable. max_tokens restored to 3000 for the think trace + response. |
||
|
profit
|
1e00eb4472 |
auditor: inference temp=0, think=false — kill signature creep
9-run empirical test showed 20 of 27 audit_lessons signatures were singletons (count=1) — the cloud producing slightly-different summary phrasings for the SAME underlying claim on each audit, each hashing to a fresh signature. That's the creep J flagged — not explosive, but steady ~2 new sigs per run, unbounded over hundreds of runs. Root cause: temperature=0.2 + think=true was letting variable prose leak into the classification output. Fix: temp=0 (greedy sample → identical input yields identical output on same model version), think=false (no reasoning trace variance), max_tokens 3000→1500 (tighter bound prevents tail wander). The compounding policy itself was validated by the 9 runs: - 7 recurring claims (the legitimate signals) all at conf 0.08-0.20 - ratingSeverity() correctly held them at info (below 0.3 threshold) - cross-PR signal test separately confirmed conf=1.00 → sev=block Also: LH_AUDIT_RUNS env so the test can validate with smaller N. |
||
|
profit
|
9d12a814e3 |
auditor: kb_index aggregator + nine-consecutive empirical test
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Phase 1 — definition-layer over append-only JSONL scratchpads.
auditor/kb_index.ts is the single shared aggregator:
aggregate<T>(jsonlPath, { keyFn, scopeFn, checkFn, tailLimit })
→ Map<signature, {count, distinct_scopes, confidence,
first_seen, last_seen, representative_summary, ...}>
ratingSeverity(agg) — confidence × count severity policy shared
across all KB readers. Kills the "same unfixed PR inflates its
own recurrence score" failure mode by design: confidence =
distinct_scopes/count, so same-scope noise stays below the 0.3
escalation threshold no matter how many times it repeats.
checkAuditLessons now routes through aggregate + ratingSeverity.
Net effect: the recurrence detector's bespoke Map/Set bookkeeping is
gone; same behavior, shared discipline, reusable by scrum/observer.
Also: symbolsExistInRepo now skips files >500KB so the audit can't
get stuck slurping a fixture.
Phase 2 — nine-consecutive audit runner.
tests/real-world/nine_consecutive_audits.ts pushes 9 empty commits,
waits for each verdict, captures the audit_lessons aggregate state
after each run, reports:
- sig_count trajectory (should stabilize, not grow linearly)
- max_count trajectory (same-signature repeat rate)
- max_confidence trajectory (must stay LOW on same-PR noise)
- verdict_stable across runs (must NOT oscillate)
This is the empirical proof that the KB compounds favorably:
noise doesn't escalate itself, and signal stays distinguishable.
Unit-tested both failure modes: same-PR × 9 repeats = conf=0.11
(info); cross-PR × 5 distinct = conf=1.00 (block). The rating
function correctly discriminates.
|
||
|
profit
|
f4be27a879 |
auditor: fix two false-positive classes from cloud inference
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Observed on PR #8 audit (
|
||
|
profit
|
efc7b5ac44 |
Auditor: dynamic + inference checks
auditor/checks/dynamic.ts — wraps runHybridFixture, maps layer results to Findings. Placeholder-style errors (404/unimplemented/ slice N) → info; other failures → warn. Always emits a summary finding with real numbers (shipped/placeholder phase counts + per- layer latency). Live-tested against current stack: 2 info findings, 0 warnings — all shipped layers actually work. auditor/checks/inference.ts — wraps the run_codereview reviewer pattern from llm_team_ui.py, adapted for claim-vs-diff verification. Calls /v1/chat provider=ollama_cloud model=gpt-oss:120b. Requests strict JSON response with claim_verdicts[] and unflagged_gaps[]. A strong claim marked "not backed" by cloud → BLOCK severity; moderate → warn; weak → info. Cloud-unreachable or unparseable-output → info (never blocks on the reviewer being down). Live-tested against PR #1 (this PR, 20 claims, 39KB diff): - 36.9s round-trip - 7 block + 23 warn + 2 info findings - gpt-oss:120b correctly flagged "Fully-functional auditor (tasks 1-9 complete)" as not-backed (only 6/10 tasks done at that commit) — accurate catch - Some false positives from the original 15KB truncation threshold (cloud missed gitea.ts, flagged "no Gitea client present") - Bumped MAX_DIFF_CHARS from 15000 to 40000 to fit the full PR diff in context; reviewer precision improves accordingly Tasks 5 + 6 completed. Remaining: #7 (KB query), #8 (verdict + Gitea poster), #9 (poller), #10 (end-to-end proof), #12 (upsert UPDATE-drops-doc_refs). |