v1/mode: parameterized runner + 5 enrichment-experiment modes
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
J's directive (2026-04-26): "Create different modes so we can really
dial in the architecture before it gets further along — pinpoint the
failures and strengths equally so I know what direction to go in.
Loop theater happens when we don't pinpoint the most accurate path."
Refactored execute() to switch on mode name → EnrichmentFlags preset.
Five native modes designed as deliberate experiments — each isolates
one architectural axis so the comparison matrix reads off what's
doing work vs what's adding latency for nothing:
codereview_lakehouse — all enrichment on (ceiling)
codereview_null — raw file + generic prompt (baseline)
codereview_isolation — file + pathway only (no matrix)
codereview_matrix_only — file + matrix only (no pathway)
codereview_playbook_only — pathway only, NO file content (lossy ceiling)
Each call appends a row to data/_kb/mode_experiments.jsonl with full
sources + response. LH_MODE_LOG_OFF=1 to suppress.
scripts/mode_experiment.ts — sweeps files × modes serially, prints
live progress with per-call enrichment stats. Defaults to OpenRouter
free model so cloud quota doesn't gate experiments.
scripts/mode_compare.ts — reads the JSONL, outputs per-file matrix
+ per-mode aggregate + mode-vs-baseline win/loss with avg finding
delta. Heuristic finding-count from markdown table rows; pathway
citation count from preamble references.
scrum_master_pipeline.ts gets a mode-runner fast path gated by
LH_USE_MODE_RUNNER=1: try /v1/mode/execute first, fall through to
the existing ladder if response < LH_MODE_MIN_CHARS (default 2000)
or anything errors. Off by default until A/B-validated.
First experiment results (2 files × 5 modes via gpt-oss-120b:free):
- codereview_null produces 12.6KB response with ZERO findings
(proves adversarial framing is load-bearing)
- codereview_playbook_only produces MORE findings than lakehouse
on average (12 vs 9) at 73% the latency — pathway memory is
the dominant signal driver
- codereview_matrix_only underperforms isolation by ~0.5 findings
while costing the same latency — matrix corpus likely
underperforming for scrum_review task class
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1438,6 +1438,56 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
|
||||
// Collect attempts for the pathway trace sidecar.
|
||||
const pathwayAttempts: LadderAttemptRec[] = [];
|
||||
|
||||
// ─── Mode runner fast path (J 2026-04-26) ───
|
||||
// Modes are prompt-molders, not model-pickers. /v1/mode/execute
|
||||
// composes pathway memory + relevance-filtered matrix chunks +
|
||||
// focus-file context into ONE prompt designed for one-shot success.
|
||||
// Try it first; if the response is substantive, skip the ladder
|
||||
// entirely. If anything goes wrong, fall through unchanged.
|
||||
//
|
||||
// Off by default until we've A/B-validated quality vs the ladder.
|
||||
// LH_USE_MODE_RUNNER=1 enables. LH_MODE_MIN_CHARS controls the
|
||||
// success bar (default 2000 — anything shorter is treated as a
|
||||
// thin response and falls through).
|
||||
if (process.env.LH_USE_MODE_RUNNER === "1") {
|
||||
const minChars = Number(process.env.LH_MODE_MIN_CHARS ?? 2000);
|
||||
log(` ⚡ mode runner enabled — trying /v1/mode/execute (min_chars=${minChars})`);
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
const r = await fetch(`${GATEWAY}/v1/mode/execute`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
task_class: taskClass,
|
||||
file_path: rel,
|
||||
file_content: content,
|
||||
}),
|
||||
signal: AbortSignal.timeout(180_000),
|
||||
});
|
||||
const modeMs = Date.now() - t0;
|
||||
if (r.ok) {
|
||||
const j: any = await r.json();
|
||||
const respChars = (j.response ?? "").length;
|
||||
if (respChars >= minChars) {
|
||||
log(` ✓ mode ${j.mode} → ${j.model} | ${j.enriched_prompt_chars} prompt chars → ${respChars} resp chars in ${modeMs}ms`);
|
||||
log(` sources: ${j.sources?.bug_fingerprints_count ?? 0} fingerprints, ${j.sources?.matrix_chunks_kept ?? 0}/${(j.sources?.matrix_chunks_kept ?? 0) + (j.sources?.matrix_chunks_dropped ?? 0)} matrix chunks kept`);
|
||||
accepted = j.response;
|
||||
acceptedModel = `mode_runner/${j.mode}/${j.model}`;
|
||||
acceptedOn = 1;
|
||||
history.push({ n: 1, model: j.model, status: "accepted", chars: respChars });
|
||||
pathwayAttempts.push({ rung: 0, model: j.model, latency_ms: modeMs, accepted: true, reject_reason: null });
|
||||
} else {
|
||||
log(` ✗ mode runner returned ${respChars} chars (<${minChars}), falling through to ladder`);
|
||||
}
|
||||
} else {
|
||||
const body = await r.text().catch(() => "");
|
||||
log(` ✗ mode runner HTTP ${r.status}: ${body.slice(0, 200)} — falling through to ladder`);
|
||||
}
|
||||
} catch (e: any) {
|
||||
log(` ✗ mode runner err: ${e.message} — falling through to ladder`);
|
||||
}
|
||||
}
|
||||
|
||||
// Single-model strategy with same-model retry. modelIdx advances
|
||||
// only on PROVIDER errors. Quality rejects from observer keep the
|
||||
// same model and retry with enriched context (history feeds back
|
||||
@@ -1448,6 +1498,10 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
|
||||
let qualityRetriesOnCurrentModel = 0;
|
||||
|
||||
for (let step = 0; step < MAX_ATTEMPTS; step++) {
|
||||
// Mode runner already produced an acceptable response — short-circuit
|
||||
// the ladder. Falls through to the post-loop bookkeeping which
|
||||
// handles {history, pathwayAttempts, hotSwap replay, etc}.
|
||||
if (accepted) break;
|
||||
if (modelIdx >= ladderOrder.length) {
|
||||
log(` ✗ all ${ladderOrder.length} fallback models exhausted, marking UNRESOLVED`);
|
||||
break;
|
||||
|
||||
Reference in New Issue
Block a user