auditor: auto-promote to Claude Opus 4.7 on big diffs (>100k chars)

Smart-routing in kimi_architect: default model (Haiku 4.5 by env, or
Kimi K2.6 if not set) handles normal PR audits cheap and fast; diffs
above LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS (default 100k) get
promoted to Claude Opus 4.7 for the audit.

Why this split: the 2026-04-27 3-way bake-off (Kimi K2.6 vs Haiku 4.5
vs Opus 4.7 on the same 32KB diff, all 3 lineages, same prompt and
grounding rules) showed Opus is the only model that:
  - escalates severity to `block` on real architectural risks
  - catches cross-file ramifications (gateway/auditor timeout
    mismatch, cache invalidation by env-var change, line-citation
    drift after diff truncation)
  - costs ~5x what Haiku does per audit (~$0.10 vs $0.02)

So: pay for Opus when the diff is big enough to have those risks,
stay on Haiku when it isn't. 80% of refactor PRs cross 100KB; 90% of
single-feature PRs don't.

New env knobs (all optional, sensible defaults):
  LH_AUDITOR_KIMI_OPUS_MODEL              default claude-opus-4-7
  LH_AUDITOR_KIMI_OPUS_PROVIDER           default opencode
  LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS    default 100000
                                          (set very high to disable)

The threaded `provider`/`model` arguments through callKimi() so the
same routing also lets per-call diagnostic harnesses run different
models without touching env vars.

Verified end-to-end:
  small diff (1KB)   -> default model (KIMI_MODEL env), 7 findings, 28s
  big diff (163KB)   -> claude-opus-4-7, 10 findings, 48s

Bake-off report at reports/kimi/cross-lineage-bakeoff.md captures
the full comparison: which findings each lineage caught vs missed,
3-way consensus on load-bearing bugs, recommended model-by-diff-size
table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-27 06:48:38 -05:00
co-authored by Claude Opus 4.7
parent bc698eb6da
commit 8aa7ee974f
2 changed files with 107 additions and 6 deletions
+26 -6
View File
@@ -50,6 +50,25 @@ const MAX_PRIOR_FINDINGS = 50;
// gateway as a fallback for when Ollama Cloud is upstream-broken.
const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "ollama_cloud";
const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6";
// Big-diff promotion: when the diff exceeds OPUS_THRESHOLD_CHARS, swap
// to OPUS_MODEL for that audit. 2026-04-27 3-way bake-off (Kimi vs
// Haiku vs Opus on a 32K diff) showed Opus is the only model that
// catches cross-file ramifications + escalates `block` severity on
// real architectural risks. ~5x the spend per audit, only worth it
// when the diff is big enough to have those risks.
//
// Defaults: Haiku for normal diffs (fast, cheap, ~$0.02), Opus for
// > 100k chars. Disable promotion: set OPUS_THRESHOLD_CHARS very high.
const OPUS_MODEL = process.env.LH_AUDITOR_KIMI_OPUS_MODEL ?? "claude-opus-4-7";
const OPUS_PROVIDER = process.env.LH_AUDITOR_KIMI_OPUS_PROVIDER ?? "opencode";
const OPUS_THRESHOLD_CHARS = Number(process.env.LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS) || 100_000;
function selectModel(diffLen: number): { provider: string; model: string; promoted: boolean } {
if (diffLen > OPUS_THRESHOLD_CHARS) {
return { provider: OPUS_PROVIDER, model: OPUS_MODEL, promoted: true };
}
return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
}
// 128K — Kimi K2.6 emits reasoning_content that counts against this
// budget but isn't surfaced in the OpenAI-shape `content` field.
// Capping low silently produces empty content with finish_reason=length
@@ -98,11 +117,12 @@ export async function runKimiArchitectCheck(
return fs2;
}
const selected = selectModel(diff.length);
let response: { content: string; usage: any; finish_reason: string; latency_ms: number };
try {
response = await callKimi(buildPrompt(diff, priorFindings, ctx));
response = await callKimi(buildPrompt(diff, priorFindings, ctx), selected.provider, selected.model);
} catch (e) {
return [skipFinding(`kimi call failed: ${(e as Error).message.slice(0, 200)}`)];
return [skipFinding(`kimi call failed (${selected.model}): ${(e as Error).message.slice(0, 200)}`)];
}
const findings = parseFindings(response.content);
@@ -112,7 +132,7 @@ export async function runKimiArchitectCheck(
pr_number: ctx.pr_number,
head_sha: ctx.head_sha,
cached_at: new Date().toISOString(),
model: KIMI_MODEL,
model: selected.model,
latency_ms: response.latency_ms,
finish_reason: response.finish_reason,
usage: {
@@ -185,11 +205,11 @@ ${truncatedDiff}
`;
}
async function callKimi(prompt: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
async function callKimi(prompt: string, provider: string, model: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
const t0 = Date.now();
const body = JSON.stringify({
provider: KIMI_PROVIDER,
model: KIMI_MODEL,
provider,
model,
messages: [{ role: "user", content: prompt }],
max_tokens: MAX_TOKENS,
temperature: 0.2,