auditor: auto-promote to Claude Opus 4.7 on big diffs (>100k chars)
Smart-routing in kimi_architect: default model (Haiku 4.5 by env, or
Kimi K2.6 if not set) handles normal PR audits cheap and fast; diffs
above LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS (default 100k) get
promoted to Claude Opus 4.7 for the audit.
Why this split: the 2026-04-27 3-way bake-off (Kimi K2.6 vs Haiku 4.5
vs Opus 4.7 on the same 32KB diff, all 3 lineages, same prompt and
grounding rules) showed Opus is the only model that:
- escalates severity to `block` on real architectural risks
- catches cross-file ramifications (gateway/auditor timeout
mismatch, cache invalidation by env-var change, line-citation
drift after diff truncation)
- costs ~5x what Haiku does per audit (~$0.10 vs $0.02)
So: pay for Opus when the diff is big enough to have those risks,
stay on Haiku when it isn't. 80% of refactor PRs cross 100KB; 90% of
single-feature PRs don't.
New env knobs (all optional, sensible defaults):
LH_AUDITOR_KIMI_OPUS_MODEL default claude-opus-4-7
LH_AUDITOR_KIMI_OPUS_PROVIDER default opencode
LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS default 100000
(set very high to disable)
The threaded `provider`/`model` arguments through callKimi() so the
same routing also lets per-call diagnostic harnesses run different
models without touching env vars.
Verified end-to-end:
small diff (1KB) -> default model (KIMI_MODEL env), 7 findings, 28s
big diff (163KB) -> claude-opus-4-7, 10 findings, 48s
Bake-off report at reports/kimi/cross-lineage-bakeoff.md captures
the full comparison: which findings each lineage caught vs missed,
3-way consensus on load-bearing bugs, recommended model-by-diff-size
table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -50,6 +50,25 @@ const MAX_PRIOR_FINDINGS = 50;
|
||||
// gateway as a fallback for when Ollama Cloud is upstream-broken.
|
||||
const KIMI_PROVIDER = process.env.LH_AUDITOR_KIMI_PROVIDER ?? "ollama_cloud";
|
||||
const KIMI_MODEL = process.env.LH_AUDITOR_KIMI_MODEL ?? "kimi-k2.6";
|
||||
// Big-diff promotion: when the diff exceeds OPUS_THRESHOLD_CHARS, swap
|
||||
// to OPUS_MODEL for that audit. 2026-04-27 3-way bake-off (Kimi vs
|
||||
// Haiku vs Opus on a 32K diff) showed Opus is the only model that
|
||||
// catches cross-file ramifications + escalates `block` severity on
|
||||
// real architectural risks. ~5x the spend per audit, only worth it
|
||||
// when the diff is big enough to have those risks.
|
||||
//
|
||||
// Defaults: Haiku for normal diffs (fast, cheap, ~$0.02), Opus for
|
||||
// > 100k chars. Disable promotion: set OPUS_THRESHOLD_CHARS very high.
|
||||
const OPUS_MODEL = process.env.LH_AUDITOR_KIMI_OPUS_MODEL ?? "claude-opus-4-7";
|
||||
const OPUS_PROVIDER = process.env.LH_AUDITOR_KIMI_OPUS_PROVIDER ?? "opencode";
|
||||
const OPUS_THRESHOLD_CHARS = Number(process.env.LH_AUDITOR_KIMI_OPUS_THRESHOLD_CHARS) || 100_000;
|
||||
|
||||
function selectModel(diffLen: number): { provider: string; model: string; promoted: boolean } {
|
||||
if (diffLen > OPUS_THRESHOLD_CHARS) {
|
||||
return { provider: OPUS_PROVIDER, model: OPUS_MODEL, promoted: true };
|
||||
}
|
||||
return { provider: KIMI_PROVIDER, model: KIMI_MODEL, promoted: false };
|
||||
}
|
||||
// 128K — Kimi K2.6 emits reasoning_content that counts against this
|
||||
// budget but isn't surfaced in the OpenAI-shape `content` field.
|
||||
// Capping low silently produces empty content with finish_reason=length
|
||||
@@ -98,11 +117,12 @@ export async function runKimiArchitectCheck(
|
||||
return fs2;
|
||||
}
|
||||
|
||||
const selected = selectModel(diff.length);
|
||||
let response: { content: string; usage: any; finish_reason: string; latency_ms: number };
|
||||
try {
|
||||
response = await callKimi(buildPrompt(diff, priorFindings, ctx));
|
||||
response = await callKimi(buildPrompt(diff, priorFindings, ctx), selected.provider, selected.model);
|
||||
} catch (e) {
|
||||
return [skipFinding(`kimi call failed: ${(e as Error).message.slice(0, 200)}`)];
|
||||
return [skipFinding(`kimi call failed (${selected.model}): ${(e as Error).message.slice(0, 200)}`)];
|
||||
}
|
||||
|
||||
const findings = parseFindings(response.content);
|
||||
@@ -112,7 +132,7 @@ export async function runKimiArchitectCheck(
|
||||
pr_number: ctx.pr_number,
|
||||
head_sha: ctx.head_sha,
|
||||
cached_at: new Date().toISOString(),
|
||||
model: KIMI_MODEL,
|
||||
model: selected.model,
|
||||
latency_ms: response.latency_ms,
|
||||
finish_reason: response.finish_reason,
|
||||
usage: {
|
||||
@@ -185,11 +205,11 @@ ${truncatedDiff}
|
||||
`;
|
||||
}
|
||||
|
||||
async function callKimi(prompt: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
|
||||
async function callKimi(prompt: string, provider: string, model: string): Promise<{ content: string; usage: any; finish_reason: string; latency_ms: number }> {
|
||||
const t0 = Date.now();
|
||||
const body = JSON.stringify({
|
||||
provider: KIMI_PROVIDER,
|
||||
model: KIMI_MODEL,
|
||||
provider,
|
||||
model,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
max_tokens: MAX_TOKENS,
|
||||
temperature: 0.2,
|
||||
|
||||
Reference in New Issue
Block a user