auditor: kb_index aggregator + nine-consecutive empirical test
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"

Phase 1 — definition-layer over append-only JSONL scratchpads.

auditor/kb_index.ts is the single shared aggregator:

  aggregate<T>(jsonlPath, { keyFn, scopeFn, checkFn, tailLimit })
      → Map<signature, {count, distinct_scopes, confidence,
                        first_seen, last_seen, representative_summary, ...}>

  ratingSeverity(agg) — confidence × count severity policy shared
    across all KB readers. Kills the "same unfixed PR inflates its
    own recurrence score" failure mode by design: confidence =
    distinct_scopes/count, so same-scope noise stays below the 0.3
    escalation threshold no matter how many times it repeats.

checkAuditLessons now routes through aggregate + ratingSeverity.
Net effect: the recurrence detector's bespoke Map/Set bookkeeping is
gone; same behavior, shared discipline, reusable by scrum/observer.

Also: symbolsExistInRepo now skips files >500KB so the audit can't
get stuck slurping a fixture.

Phase 2 — nine-consecutive audit runner.

tests/real-world/nine_consecutive_audits.ts pushes 9 empty commits,
waits for each verdict, captures the audit_lessons aggregate state
after each run, reports:

  - sig_count trajectory (should stabilize, not grow linearly)
  - max_count trajectory (same-signature repeat rate)
  - max_confidence trajectory (must stay LOW on same-PR noise)
  - verdict_stable across runs (must NOT oscillate)

This is the empirical proof that the KB compounds favorably:
noise doesn't escalate itself, and signal stays distinguishable.

Unit-tested both failure modes: same-PR × 9 repeats = conf=0.11
(info); cross-PR × 5 distinct = conf=1.00 (block). The rating
function correctly discriminates.
This commit is contained in:
profit
2026-04-22 21:49:46 -05:00
parent f4be27a879
commit 9d12a814e3
4 changed files with 369 additions and 44 deletions
+5 -1
View File
@@ -260,10 +260,13 @@ function extractSymbols(text: string): string[] {
// Scan the repo for at least one definition of each symbol. Uses Bun's
// Glob to walk TS/Rust/Python/JS sources; ignores node_modules, data/,
// and target/.
// and target/. Skips files > 500KB — those are fixtures/snapshots that
// won't contain a definition line and slurping them slows the audit.
async function symbolsExistInRepo(symbols: string[]): Promise<string[]> {
const patterns = ["**/*.ts", "**/*.tsx", "**/*.rs", "**/*.py", "**/*.js"];
const skip = (p: string) => p.includes("/node_modules/") || p.startsWith("data/") || p.includes("/target/") || p.startsWith("dist/");
const MAX_FILE_BYTES = 500_000;
const { stat } = await import("node:fs/promises");
const resolved = new Set<string>();
const toFind = new Set(symbols);
for (const pat of patterns) {
@@ -271,6 +274,7 @@ async function symbolsExistInRepo(symbols: string[]): Promise<string[]> {
const glob = new Glob(pat);
for await (const f of glob.scan({ cwd: REPO_ROOT, onlyFiles: true })) {
if (skip(f)) continue;
try { const s = await stat(`${REPO_ROOT}/${f}`); if (s.size > MAX_FILE_BYTES) continue; } catch { continue; }
let content: string;
try { content = await readFile(`${REPO_ROOT}/${f}`, "utf8"); } catch { continue; }
for (const sym of Array.from(toFind)) {