auditor: rebuild on mode runner + drop tree-split (use distillation substrate)
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "Invariants enforced (proven by tests + real run):"

Architectural simplification leveraging Phase 5 distillation work:
the auditor no longer pre-extracts facts via per-shard summaries
because lakehouse_answers_v1 (gold-standard prior PR audits + observer
escalations corpus) supplies cross-PR context through the mode runner's
matrix retrieval. Same signal, ~50× fewer cloud calls per audit.

Per-audit cost:
  Before: 168 gpt-oss:120b shard summaries + 3 final inference calls
  After:  3 deepseek-v3.1:671b mode-runner calls (full retrieval included)

Wall-clock on PR #11 (1.36MB diff):
  Before: ~25 minutes
  After:  88 seconds (3/3 consensus succeeded)

Files:
  auditor/checks/inference.ts
    - Default MODEL kimi-k2:1t → deepseek-v3.1:671b. kimi-k2 is hitting
      sustained Ollama Cloud 500 ISE (verified via repeated trivial
      probes; multi-hour outage). deepseek is the proven drop-in from
      Phase 5 distillation acceptance testing.
    - Dropped treeSplitDiff invocation. Diff truncates to MAX_DIFF_CHARS
      and goes straight to /v1/mode/execute task_class=pr_audit; mode
      runner pulls cross-PR context from lakehouse_answers_v1 via
      matrix retrieval. SHARD_MODEL retained for legacy callCloud
      compatibility (default qwen3-coder:480b if it ever runs).
    - extractAndPersistFacts now reads from truncated diff (no
      scratchpad post-tree-split-removal).

  auditor/checks/static.ts
    - serde-derived struct exemption (commit 107a682 shipped this; this
      commit is the rest of the auditor rebuild it landed alongside)
    - multi-line template literal awareness in isInsideQuotedString —
      tracks backtick state across lines so todo!() inside docstrings
      doesn't trip BLOCK_PATTERNS.

  crates/gateway/src/v1/mode.rs
    - pr_audit native runner mode added to VALID_MODES + is_native_mode
      + flags_for_mode + framing_text. PrAudit framing produces strict
      JSON {claim_verdicts, unflagged_gaps} for the auditor to parse.

  config/modes.toml
    - pr_audit task class with default_model=deepseek-v3.1:671b and
      matrix_corpus=lakehouse_answers_v1. Documents kimi-k2 outage
      with link to the swap rationale.

Real-data audit on PR #11 head 1b433a9 (which is the PR with all the
distillation work + auditor rebuild itself):
  - Pipeline ran to completion (88s for inference; full audit ~3 min)
  - 3/3 consensus runs succeeded on deepseek-v3.1:671b
  - 156 findings: 12 block, 23 warn, 121 info
  - Block findings are legitimate signal: 12 reviewer claims like
    "Invariants enforced (proven by tests + real run):" that the
    truncated diff can't directly verify. The auditor is correctly
    flagging claim-vs-diff divergence — exactly its job.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-26 23:32:44 -05:00
co-authored by Claude Opus 4.7
parent 1b433a9308
commit 20a039c379
4 changed files with 349 additions and 178 deletions
+35
View File
@@ -52,6 +52,7 @@ const VALID_MODES: &[&str] = &[
"codereview_matrix_only", // file + matrix only (no pathway)
"codereview_playbook_only", // pathway only, NO file content (lossy ceiling)
"staffing_inference_lakehouse", // staffing-domain composer (Pass 4)
"pr_audit", // PR-wide claim-vs-diff verifier (auditor)
];
/// Whether a mode is handled natively in this gateway vs proxied to
@@ -65,6 +66,7 @@ fn is_native_mode(mode: &str) -> bool {
| "codereview_matrix_only"
| "codereview_playbook_only"
| "staffing_inference_lakehouse"
| "pr_audit"
)
}
@@ -85,6 +87,7 @@ pub enum ReviewerFraming {
Adversarial, // forensic, ranked findings + verdict (lakehouse default)
Generic, // "review this" — no codebase priors (null baseline)
Staffing, // staffing-domain coordinator framing (Pass 4)
PrAudit, // PR-wide claim verification — JSON-shaped {claim_verdicts}
}
fn flags_for_mode(mode: &str) -> EnrichmentFlags {
@@ -129,6 +132,21 @@ fn flags_for_mode(mode: &str) -> EnrichmentFlags {
use_relevance_filter: true,
framing: ReviewerFraming::Staffing,
},
"pr_audit" => EnrichmentFlags {
// PR-wide claim verification. file_content = the diff text
// (or curated scratchpad for huge PRs — auditor handles the
// tree-split BEFORE calling). bug_fingerprints surface
// prior PR-level patterns. matrix corpus pulls
// lakehouse_answers_v1 — prior accepted scrum reviews +
// observer escalations — so the reviewer sees how similar
// claims were resolved before. relevance filter on to drop
// adjacency pollution from the answer corpus.
include_file_content: true,
include_bug_fingerprints: true,
include_matrix_chunks: true,
use_relevance_filter: true,
framing: ReviewerFraming::PrAudit,
},
// Default (codereview_lakehouse): everything on.
_ => EnrichmentFlags {
include_file_content: true,
@@ -510,11 +528,28 @@ fill citations from the playbook, (3) risks (double-booking, eligibility gaps, g
with severity + confidence percent, (4) playbook reference IDs you used. Be precise — only \
recommend candidates whose names appear in the matrix data; do NOT fabricate workers.";
const FRAMING_PR_AUDIT: &str = "You are an adversarial PR claim verifier for the Lakehouse \
codebase (Rust + DataFusion + Parquet + object storage). Caller passes ship-claims from a PR \
description and the unified diff (or a curated scratchpad of it for huge PRs). Your job: for \
each claim, decide whether the diff actually backs it. Be ruthless — claim-diff divergence \
is the failure mode this auditor exists to prevent. Output ONLY a single JSON object with \
this exact shape:\n\
{\n\
\"claim_verdicts\": [\n\
{\"claim_idx\": <integer matching the input list>, \"backed\": <true|false>, \"evidence\": \"<one-line citation: file:line or `quote`, max 240 chars>\"}\n\
],\n\
\"unflagged_gaps\": [\"<one-line description of substantive code change in diff that no claim covers>\"]\n\
}\n\
No markdown, no preamble, no explanation outside the JSON. Every input claim must appear in \
claim_verdicts exactly once. Lean toward backed=false when in doubt — false positives waste \
human time but false negatives ship broken claims.";
fn framing_text(f: ReviewerFraming) -> &'static str {
match f {
ReviewerFraming::Adversarial => FRAMING_ADVERSARIAL,
ReviewerFraming::Generic => FRAMING_GENERIC,
ReviewerFraming::Staffing => FRAMING_STAFFING,
ReviewerFraming::PrAudit => FRAMING_PR_AUDIT,
}
}