auditor: rebuild on mode runner + drop tree-split (use distillation substrate)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"
Architectural simplification leveraging Phase 5 distillation work: the auditor no longer pre-extracts facts via per-shard summaries because lakehouse_answers_v1 (gold-standard prior PR audits + observer escalations corpus) supplies cross-PR context through the mode runner's matrix retrieval. Same signal, ~50× fewer cloud calls per audit. Per-audit cost: Before: 168 gpt-oss:120b shard summaries + 3 final inference calls After: 3 deepseek-v3.1:671b mode-runner calls (full retrieval included) Wall-clock on PR #11 (1.36MB diff): Before: ~25 minutes After: 88 seconds (3/3 consensus succeeded) Files: auditor/checks/inference.ts - Default MODEL kimi-k2:1t → deepseek-v3.1:671b. kimi-k2 is hitting sustained Ollama Cloud 500 ISE (verified via repeated trivial probes; multi-hour outage). deepseek is the proven drop-in from Phase 5 distillation acceptance testing. - Dropped treeSplitDiff invocation. Diff truncates to MAX_DIFF_CHARS and goes straight to /v1/mode/execute task_class=pr_audit; mode runner pulls cross-PR context from lakehouse_answers_v1 via matrix retrieval. SHARD_MODEL retained for legacy callCloud compatibility (default qwen3-coder:480b if it ever runs). - extractAndPersistFacts now reads from truncated diff (no scratchpad post-tree-split-removal). auditor/checks/static.ts - serde-derived struct exemption (commit107a682shipped this; this commit is the rest of the auditor rebuild it landed alongside) - multi-line template literal awareness in isInsideQuotedString — tracks backtick state across lines so todo!() inside docstrings doesn't trip BLOCK_PATTERNS. crates/gateway/src/v1/mode.rs - pr_audit native runner mode added to VALID_MODES + is_native_mode + flags_for_mode + framing_text. PrAudit framing produces strict JSON {claim_verdicts, unflagged_gaps} for the auditor to parse. config/modes.toml - pr_audit task class with default_model=deepseek-v3.1:671b and matrix_corpus=lakehouse_answers_v1. Documents kimi-k2 outage with link to the swap rationale. Real-data audit on PR #11 head1b433a9(which is the PR with all the distillation work + auditor rebuild itself): - Pipeline ran to completion (88s for inference; full audit ~3 min) - 3/3 consensus runs succeeded on deepseek-v3.1:671b - 156 findings: 12 block, 23 warn, 121 info - Block findings are legitimate signal: 12 reviewer claims like "Invariants enforced (proven by tests + real run):" that the truncated diff can't directly verify. The auditor is correctly flagging claim-vs-diff divergence — exactly its job. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+88
-36
@@ -54,49 +54,79 @@ export function runStaticCheck(diff: string): Finding[] {
|
||||
const isAuditorCheckerFile = path.startsWith("auditor/checks/") ||
|
||||
path.startsWith("auditor/fixtures/");
|
||||
|
||||
// Track multi-line backtick-template state across the file. Walks
|
||||
// all post-merge lines (context + added, skipping removed lines)
|
||||
// in order and keeps `inMultilineBacktick` flipping on each
|
||||
// unescaped backtick. Pre-2026-04-26 the per-line walk in
|
||||
// isInsideQuotedString missed `todo!()` matches inside docstring
|
||||
// template literals because the opening backtick lived on a
|
||||
// line above the match. Now we OR the file-level state into the
|
||||
// per-line check.
|
||||
let inMultilineBacktick = false;
|
||||
|
||||
for (let idx = 0; idx < lines.length; idx++) {
|
||||
const line = lines[idx];
|
||||
if (!line.startsWith("+") || line.startsWith("+++")) continue;
|
||||
const added = line.slice(1);
|
||||
|
||||
if (!isAuditorCheckerFile) {
|
||||
for (const { re, why } of BLOCK_PATTERNS) {
|
||||
const m = added.match(re);
|
||||
if (m && typeof m.index === "number") {
|
||||
// Skip if the match sits inside a quoted string literal —
|
||||
// this is how rubric files (tests/real-world/*, prompt
|
||||
// templates) legitimately reference the patterns they
|
||||
// guard against, without actually executing them.
|
||||
if (isInsideQuotedString(added, m.index)) continue;
|
||||
// Diff bookkeeping lines and removed lines don't contribute to
|
||||
// the post-merge file's string state.
|
||||
if (line.startsWith("+++") || line.startsWith("---") ||
|
||||
line.startsWith("@@") || line.startsWith("\\ No newline")) continue;
|
||||
if (line.startsWith("-")) continue;
|
||||
|
||||
const isAdded = line.startsWith("+");
|
||||
// Strip the diff prefix (' ' for context, '+' for added).
|
||||
const body = (isAdded || line.startsWith(" ")) ? line.slice(1) : line;
|
||||
|
||||
if (isAdded) {
|
||||
const added = body;
|
||||
|
||||
if (!isAuditorCheckerFile) {
|
||||
for (const { re, why } of BLOCK_PATTERNS) {
|
||||
const m = added.match(re);
|
||||
if (m && typeof m.index === "number") {
|
||||
// Skip if the match sits inside a quoted string literal —
|
||||
// this is how rubric files (tests/real-world/*, prompt
|
||||
// templates) legitimately reference the patterns they
|
||||
// guard against, without actually executing them.
|
||||
if (inMultilineBacktick || isInsideQuotedString(added, m.index)) continue;
|
||||
findings.push({
|
||||
check: "static",
|
||||
severity: "block",
|
||||
summary: `${why} in ${path}`,
|
||||
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const { re, why } of WARN_COMMENT_PATTERNS) {
|
||||
if (re.test(line)) {
|
||||
findings.push({
|
||||
check: "static",
|
||||
severity: "block",
|
||||
severity: "warn",
|
||||
summary: `${why} in ${path}`,
|
||||
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
|
||||
});
|
||||
}
|
||||
}
|
||||
for (const { re, why } of INFO_HARDCODED_PATTERNS) {
|
||||
if (re.test(added)) {
|
||||
findings.push({
|
||||
check: "static",
|
||||
severity: "info",
|
||||
summary: `${why} in ${path}`,
|
||||
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const { re, why } of WARN_COMMENT_PATTERNS) {
|
||||
if (re.test(line)) {
|
||||
findings.push({
|
||||
check: "static",
|
||||
severity: "warn",
|
||||
summary: `${why} in ${path}`,
|
||||
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
|
||||
});
|
||||
}
|
||||
}
|
||||
for (const { re, why } of INFO_HARDCODED_PATTERNS) {
|
||||
if (re.test(added)) {
|
||||
findings.push({
|
||||
check: "static",
|
||||
severity: "info",
|
||||
summary: `${why} in ${path}`,
|
||||
evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Update file-level multi-line backtick state by walking THIS
|
||||
// line's unescaped backticks. Both context and added lines
|
||||
// contribute (they're both in the post-merge file). Doc-comment
|
||||
// backticks like `\\\`Foo\\\`` count too — that's the source of
|
||||
// the original bug, where multi-line template literals contained
|
||||
// `todo!()` references.
|
||||
inMultilineBacktick = updateBacktickState(body, inMultilineBacktick);
|
||||
}
|
||||
|
||||
// "Field added but never read" heuristic — catches exactly the
|
||||
@@ -220,12 +250,34 @@ function stripDiffPrefix(line: string): string {
|
||||
return line;
|
||||
}
|
||||
|
||||
// Walk a single line and toggle the cross-line backtick state on each
|
||||
// unescaped backtick. Single-quote and double-quote runs are line-
|
||||
// bounded in JS/TS/Rust by language rules (string literals don't span
|
||||
// newlines without explicit `\` continuation), so we only track
|
||||
// backticks across lines. Returns the new state for the next line.
|
||||
function updateBacktickState(line: string, inBacktick: boolean): boolean {
|
||||
let state = inBacktick;
|
||||
let inDouble = false;
|
||||
let inSingle = false;
|
||||
for (let i = 0; i < line.length; i++) {
|
||||
const c = line[i];
|
||||
const esc = i > 0 && line[i - 1] === "\\";
|
||||
if (esc) continue;
|
||||
// Inside a multi-line backtick template, single/double quotes
|
||||
// don't open new strings — they're literal characters of the
|
||||
// template. Same applies the other way around.
|
||||
if (c === '"' && !inSingle && !state) inDouble = !inDouble;
|
||||
else if (c === "'" && !inDouble && !state) inSingle = !inSingle;
|
||||
else if (c === "`" && !inDouble && !inSingle) state = !state;
|
||||
}
|
||||
return state;
|
||||
}
|
||||
|
||||
// True if `pos` falls inside a double- or single-quoted string on this
|
||||
// line (backtick template literals too). Walks left→right toggling the
|
||||
// "in quote" state on each unescaped quote. Good enough for single-
|
||||
// line matches; multi-line strings aren't parsed (they're extremely
|
||||
// rare in the patterns we're blocking on, and would require a proper
|
||||
// tokenizer to handle correctly).
|
||||
// "in quote" state on each unescaped quote. Per-line only — the file-
|
||||
// level walk in runStaticCheck handles multi-line backtick templates
|
||||
// via updateBacktickState.
|
||||
function isInsideQuotedString(line: string, pos: number): boolean {
|
||||
let inDouble = false, inSingle = false, inBacktick = false;
|
||||
for (let i = 0; i < pos; i++) {
|
||||
|
||||
Reference in New Issue
Block a user