v1/mode: parameterized runner + 5 enrichment-experiment modes
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts

J's directive (2026-04-26): "Create different modes so we can really
dial in the architecture before it gets further along — pinpoint the
failures and strengths equally so I know what direction to go in.
Loop theater happens when we don't pinpoint the most accurate path."

Refactored execute() to switch on mode name → EnrichmentFlags preset.
Five native modes designed as deliberate experiments — each isolates
one architectural axis so the comparison matrix reads off what's
doing work vs what's adding latency for nothing:

  codereview_lakehouse     — all enrichment on (ceiling)
  codereview_null          — raw file + generic prompt (baseline)
  codereview_isolation     — file + pathway only (no matrix)
  codereview_matrix_only   — file + matrix only (no pathway)
  codereview_playbook_only — pathway only, NO file content (lossy ceiling)

Each call appends a row to data/_kb/mode_experiments.jsonl with full
sources + response. LH_MODE_LOG_OFF=1 to suppress.

scripts/mode_experiment.ts — sweeps files × modes serially, prints
live progress with per-call enrichment stats. Defaults to OpenRouter
free model so cloud quota doesn't gate experiments.

scripts/mode_compare.ts — reads the JSONL, outputs per-file matrix
+ per-mode aggregate + mode-vs-baseline win/loss with avg finding
delta. Heuristic finding-count from markdown table rows; pathway
citation count from preamble references.

scrum_master_pipeline.ts gets a mode-runner fast path gated by
LH_USE_MODE_RUNNER=1: try /v1/mode/execute first, fall through to
the existing ladder if response < LH_MODE_MIN_CHARS (default 2000)
or anything errors. Off by default until A/B-validated.

First experiment results (2 files × 5 modes via gpt-oss-120b:free):
  - codereview_null produces 12.6KB response with ZERO findings
    (proves adversarial framing is load-bearing)
  - codereview_playbook_only produces MORE findings than lakehouse
    on average (12 vs 9) at 73% the latency — pathway memory is
    the dominant signal driver
  - codereview_matrix_only underperforms isolation by ~0.5 findings
    while costing the same latency — matrix corpus likely
    underperforming for scrum_review task class

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-26 01:36:42 -05:00
co-authored by Claude Opus 4.7
parent 86f63a083d
commit 7c47734287
5 changed files with 526 additions and 20 deletions
+54
View File
@@ -1438,6 +1438,56 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
// Collect attempts for the pathway trace sidecar.
const pathwayAttempts: LadderAttemptRec[] = [];
// ─── Mode runner fast path (J 2026-04-26) ───
// Modes are prompt-molders, not model-pickers. /v1/mode/execute
// composes pathway memory + relevance-filtered matrix chunks +
// focus-file context into ONE prompt designed for one-shot success.
// Try it first; if the response is substantive, skip the ladder
// entirely. If anything goes wrong, fall through unchanged.
//
// Off by default until we've A/B-validated quality vs the ladder.
// LH_USE_MODE_RUNNER=1 enables. LH_MODE_MIN_CHARS controls the
// success bar (default 2000 — anything shorter is treated as a
// thin response and falls through).
if (process.env.LH_USE_MODE_RUNNER === "1") {
const minChars = Number(process.env.LH_MODE_MIN_CHARS ?? 2000);
log(` ⚡ mode runner enabled — trying /v1/mode/execute (min_chars=${minChars})`);
const t0 = Date.now();
try {
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
task_class: taskClass,
file_path: rel,
file_content: content,
}),
signal: AbortSignal.timeout(180_000),
});
const modeMs = Date.now() - t0;
if (r.ok) {
const j: any = await r.json();
const respChars = (j.response ?? "").length;
if (respChars >= minChars) {
log(` ✓ mode ${j.mode} → ${j.model} | ${j.enriched_prompt_chars} prompt chars → ${respChars} resp chars in ${modeMs}ms`);
log(` sources: ${j.sources?.bug_fingerprints_count ?? 0} fingerprints, ${j.sources?.matrix_chunks_kept ?? 0}/${(j.sources?.matrix_chunks_kept ?? 0) + (j.sources?.matrix_chunks_dropped ?? 0)} matrix chunks kept`);
accepted = j.response;
acceptedModel = `mode_runner/${j.mode}/${j.model}`;
acceptedOn = 1;
history.push({ n: 1, model: j.model, status: "accepted", chars: respChars });
pathwayAttempts.push({ rung: 0, model: j.model, latency_ms: modeMs, accepted: true, reject_reason: null });
} else {
log(` ✗ mode runner returned ${respChars} chars (<${minChars}), falling through to ladder`);
}
} else {
const body = await r.text().catch(() => "");
log(` ✗ mode runner HTTP ${r.status}: ${body.slice(0, 200)} — falling through to ladder`);
}
} catch (e: any) {
log(` ✗ mode runner err: ${e.message} — falling through to ladder`);
}
}
// Single-model strategy with same-model retry. modelIdx advances
// only on PROVIDER errors. Quality rejects from observer keep the
// same model and retry with enriched context (history feeds back
@@ -1448,6 +1498,10 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
let qualityRetriesOnCurrentModel = 0;
for (let step = 0; step < MAX_ATTEMPTS; step++) {
// Mode runner already produced an acceptable response — short-circuit
// the ladder. Falls through to the post-loop bookkeeping which
// handles {history, pathwayAttempts, hotSwap replay, etc}.
if (accepted) break;
if (modelIdx >= ladderOrder.length) {
log(` ✗ all ${ladderOrder.length} fallback models exhausted, marking UNRESOLVED`);
break;