auditor: rebuild on mode runner + drop tree-split (use distillation substrate)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"
Architectural simplification leveraging Phase 5 distillation work: the auditor no longer pre-extracts facts via per-shard summaries because lakehouse_answers_v1 (gold-standard prior PR audits + observer escalations corpus) supplies cross-PR context through the mode runner's matrix retrieval. Same signal, ~50× fewer cloud calls per audit. Per-audit cost: Before: 168 gpt-oss:120b shard summaries + 3 final inference calls After: 3 deepseek-v3.1:671b mode-runner calls (full retrieval included) Wall-clock on PR #11 (1.36MB diff): Before: ~25 minutes After: 88 seconds (3/3 consensus succeeded) Files: auditor/checks/inference.ts - Default MODEL kimi-k2:1t → deepseek-v3.1:671b. kimi-k2 is hitting sustained Ollama Cloud 500 ISE (verified via repeated trivial probes; multi-hour outage). deepseek is the proven drop-in from Phase 5 distillation acceptance testing. - Dropped treeSplitDiff invocation. Diff truncates to MAX_DIFF_CHARS and goes straight to /v1/mode/execute task_class=pr_audit; mode runner pulls cross-PR context from lakehouse_answers_v1 via matrix retrieval. SHARD_MODEL retained for legacy callCloud compatibility (default qwen3-coder:480b if it ever runs). - extractAndPersistFacts now reads from truncated diff (no scratchpad post-tree-split-removal). auditor/checks/static.ts - serde-derived struct exemption (commit107a682shipped this; this commit is the rest of the auditor rebuild it landed alongside) - multi-line template literal awareness in isInsideQuotedString — tracks backtick state across lines so todo!() inside docstrings doesn't trip BLOCK_PATTERNS. crates/gateway/src/v1/mode.rs - pr_audit native runner mode added to VALID_MODES + is_native_mode + flags_for_mode + framing_text. PrAudit framing produces strict JSON {claim_verdicts, unflagged_gaps} for the auditor to parse. config/modes.toml - pr_audit task class with default_model=deepseek-v3.1:671b and matrix_corpus=lakehouse_answers_v1. Documents kimi-k2 outage with link to the swap rationale. Real-data audit on PR #11 head1b433a9(which is the PR with all the distillation work + auditor rebuild itself): - Pipeline ran to completion (88s for inference; full audit ~3 min) - 3/3 consensus runs succeeded on deepseek-v3.1:671b - 156 findings: 12 block, 23 warn, 121 info - Block findings are legitimate signal: 12 reviewer claims like "Invariants enforced (proven by tests + real run):" that the truncated diff can't directly verify. The auditor is correctly flagging claim-vs-diff divergence — exactly its job. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -52,6 +52,7 @@ const VALID_MODES: &[&str] = &[
|
||||
"codereview_matrix_only", // file + matrix only (no pathway)
|
||||
"codereview_playbook_only", // pathway only, NO file content (lossy ceiling)
|
||||
"staffing_inference_lakehouse", // staffing-domain composer (Pass 4)
|
||||
"pr_audit", // PR-wide claim-vs-diff verifier (auditor)
|
||||
];
|
||||
|
||||
/// Whether a mode is handled natively in this gateway vs proxied to
|
||||
@@ -65,6 +66,7 @@ fn is_native_mode(mode: &str) -> bool {
|
||||
| "codereview_matrix_only"
|
||||
| "codereview_playbook_only"
|
||||
| "staffing_inference_lakehouse"
|
||||
| "pr_audit"
|
||||
)
|
||||
}
|
||||
|
||||
@@ -85,6 +87,7 @@ pub enum ReviewerFraming {
|
||||
Adversarial, // forensic, ranked findings + verdict (lakehouse default)
|
||||
Generic, // "review this" — no codebase priors (null baseline)
|
||||
Staffing, // staffing-domain coordinator framing (Pass 4)
|
||||
PrAudit, // PR-wide claim verification — JSON-shaped {claim_verdicts}
|
||||
}
|
||||
|
||||
fn flags_for_mode(mode: &str) -> EnrichmentFlags {
|
||||
@@ -129,6 +132,21 @@ fn flags_for_mode(mode: &str) -> EnrichmentFlags {
|
||||
use_relevance_filter: true,
|
||||
framing: ReviewerFraming::Staffing,
|
||||
},
|
||||
"pr_audit" => EnrichmentFlags {
|
||||
// PR-wide claim verification. file_content = the diff text
|
||||
// (or curated scratchpad for huge PRs — auditor handles the
|
||||
// tree-split BEFORE calling). bug_fingerprints surface
|
||||
// prior PR-level patterns. matrix corpus pulls
|
||||
// lakehouse_answers_v1 — prior accepted scrum reviews +
|
||||
// observer escalations — so the reviewer sees how similar
|
||||
// claims were resolved before. relevance filter on to drop
|
||||
// adjacency pollution from the answer corpus.
|
||||
include_file_content: true,
|
||||
include_bug_fingerprints: true,
|
||||
include_matrix_chunks: true,
|
||||
use_relevance_filter: true,
|
||||
framing: ReviewerFraming::PrAudit,
|
||||
},
|
||||
// Default (codereview_lakehouse): everything on.
|
||||
_ => EnrichmentFlags {
|
||||
include_file_content: true,
|
||||
@@ -510,11 +528,28 @@ fill citations from the playbook, (3) risks (double-booking, eligibility gaps, g
|
||||
with severity + confidence percent, (4) playbook reference IDs you used. Be precise — only \
|
||||
recommend candidates whose names appear in the matrix data; do NOT fabricate workers.";
|
||||
|
||||
const FRAMING_PR_AUDIT: &str = "You are an adversarial PR claim verifier for the Lakehouse \
|
||||
codebase (Rust + DataFusion + Parquet + object storage). Caller passes ship-claims from a PR \
|
||||
description and the unified diff (or a curated scratchpad of it for huge PRs). Your job: for \
|
||||
each claim, decide whether the diff actually backs it. Be ruthless — claim-diff divergence \
|
||||
is the failure mode this auditor exists to prevent. Output ONLY a single JSON object with \
|
||||
this exact shape:\n\
|
||||
{\n\
|
||||
\"claim_verdicts\": [\n\
|
||||
{\"claim_idx\": <integer matching the input list>, \"backed\": <true|false>, \"evidence\": \"<one-line citation: file:line or `quote`, max 240 chars>\"}\n\
|
||||
],\n\
|
||||
\"unflagged_gaps\": [\"<one-line description of substantive code change in diff that no claim covers>\"]\n\
|
||||
}\n\
|
||||
No markdown, no preamble, no explanation outside the JSON. Every input claim must appear in \
|
||||
claim_verdicts exactly once. Lean toward backed=false when in doubt — false positives waste \
|
||||
human time but false negatives ship broken claims.";
|
||||
|
||||
fn framing_text(f: ReviewerFraming) -> &'static str {
|
||||
match f {
|
||||
ReviewerFraming::Adversarial => FRAMING_ADVERSARIAL,
|
||||
ReviewerFraming::Generic => FRAMING_GENERIC,
|
||||
ReviewerFraming::Staffing => FRAMING_STAFFING,
|
||||
ReviewerFraming::PrAudit => FRAMING_PR_AUDIT,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user