auditor: kb_index aggregator + nine-consecutive empirical test
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Phase 1 — definition-layer over append-only JSONL scratchpads.
auditor/kb_index.ts is the single shared aggregator:
aggregate<T>(jsonlPath, { keyFn, scopeFn, checkFn, tailLimit })
→ Map<signature, {count, distinct_scopes, confidence,
first_seen, last_seen, representative_summary, ...}>
ratingSeverity(agg) — confidence × count severity policy shared
across all KB readers. Kills the "same unfixed PR inflates its
own recurrence score" failure mode by design: confidence =
distinct_scopes/count, so same-scope noise stays below the 0.3
escalation threshold no matter how many times it repeats.
checkAuditLessons now routes through aggregate + ratingSeverity.
Net effect: the recurrence detector's bespoke Map/Set bookkeeping is
gone; same behavior, shared discipline, reusable by scrum/observer.
Also: symbolsExistInRepo now skips files >500KB so the audit can't
get stuck slurping a fixture.
Phase 2 — nine-consecutive audit runner.
tests/real-world/nine_consecutive_audits.ts pushes 9 empty commits,
waits for each verdict, captures the audit_lessons aggregate state
after each run, reports:
- sig_count trajectory (should stabilize, not grow linearly)
- max_count trajectory (same-signature repeat rate)
- max_confidence trajectory (must stay LOW on same-PR noise)
- verdict_stable across runs (must NOT oscillate)
This is the empirical proof that the KB compounds favorably:
noise doesn't escalate itself, and signal stays distinguishable.
Unit-tested both failure modes: same-PR × 9 repeats = conf=0.11
(info); cross-PR × 5 distinct = conf=1.00 (block). The rating
function correctly discriminates.
This commit is contained in:
@@ -260,10 +260,13 @@ function extractSymbols(text: string): string[] {
|
||||
|
||||
// Scan the repo for at least one definition of each symbol. Uses Bun's
|
||||
// Glob to walk TS/Rust/Python/JS sources; ignores node_modules, data/,
|
||||
// and target/.
|
||||
// and target/. Skips files > 500KB — those are fixtures/snapshots that
|
||||
// won't contain a definition line and slurping them slows the audit.
|
||||
async function symbolsExistInRepo(symbols: string[]): Promise<string[]> {
|
||||
const patterns = ["**/*.ts", "**/*.tsx", "**/*.rs", "**/*.py", "**/*.js"];
|
||||
const skip = (p: string) => p.includes("/node_modules/") || p.startsWith("data/") || p.includes("/target/") || p.startsWith("dist/");
|
||||
const MAX_FILE_BYTES = 500_000;
|
||||
const { stat } = await import("node:fs/promises");
|
||||
const resolved = new Set<string>();
|
||||
const toFind = new Set(symbols);
|
||||
for (const pat of patterns) {
|
||||
@@ -271,6 +274,7 @@ async function symbolsExistInRepo(symbols: string[]): Promise<string[]> {
|
||||
const glob = new Glob(pat);
|
||||
for await (const f of glob.scan({ cwd: REPO_ROOT, onlyFiles: true })) {
|
||||
if (skip(f)) continue;
|
||||
try { const s = await stat(`${REPO_ROOT}/${f}`); if (s.size > MAX_FILE_BYTES) continue; } catch { continue; }
|
||||
let content: string;
|
||||
try { content = await readFile(`${REPO_ROOT}/${f}`, "utf8"); } catch { continue; }
|
||||
for (const sym of Array.from(toFind)) {
|
||||
|
||||
Reference in New Issue
Block a user