profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

v1/mode: model-aware enrichment downgrade + 3 corpora + variance harness
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts

Browse Source 
Pass 5 (5 reps × 4 conditions × 1 file on grok-4.1-fast) showed composing
matrix corpora is anti-additive on strong models — composed lakehouse_arch
+ symbols LOST 5/5 head-to-head vs codereview_isolation (Δ −1.8 grounded
findings, p=0.031). Default flips to isolation; matrix path now auto-
downgrades when the resolved model is strong.

Mode runner:
- matrix_corpus is Vec<String> (string OR array via deserialize_string_or_vec)
- top_k=6 from each corpus, merge by score, take top 8 globally
- chunk tag prefers doc_id over source so reviewer sees [adr:009] vs [lakehouse_arch]
- is_weak_model() gate auto-downgrades codereview_lakehouse → codereview_isolation
  for strong models (default-strong; weak = :free suffix or local last-resort)
- LH_FORCE_FULL_ENRICHMENT=1 bypasses for diagnostic runs
- EnrichmentSources.downgraded_from records when the gate fires

Three corpora indexed via /vectors/index (5849 chunks total):
- lakehouse_arch_v1 — ADRs + phases + PRD + scrum spec (93 docs, 2119 chunks)
- scrum_findings_v1 — past scrum_reviews.jsonl (168 docs, 1260 chunks; EXCLUDED
  from defaults — 24% out-of-bounds line citations from cross-file drift)
- lakehouse_symbols_v1 — regex-extracted pub items + /// docs (656 docs, 2470 chunks)

Experiment infra:
- scripts/build_*_corpus.ts — re-runnable when source content changes
- scripts/mode_pass5_variance_paid.ts — N reps × M conditions on one file
- scripts/mode_pass5_summarize.ts — mean ± σ + head-to-head, parser handles
  numbered + path-with-line + path-with-symbol finding tables
- scripts/mode_compare.ts — groups by mode|corpus when sweeps span corpora
- scripts/mode_experiment.ts — default model bumped to x-ai/grok-4.1-fast,
  --corpus flag for per-call override

Decisions + open follow-ups: docs/MODE_RUNNER_TUNING_PLAN.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-26 17:29:17 -05:00
co-authored by Claude Opus 4.7
parent 56bf30cfd8
commit 2dbc8dbc83
10 changed files with 1043 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gateway/src/v1/mode.rs
+170 -45
View File
@@ -147,8 +147,31 @@ pub struct TaskClassEntry {
#[serde(default)]
pub fallback_modes: Vec<String>,
pub default_model: String,
#[serde(default)]
pub matrix_corpus: Option<String>,
/// One or more corpora the mode runner queries (top-k per corpus,
/// merged by score before the relevance filter). Accepts a single
/// string or an array in modes.toml — `deserialize_string_or_vec`
/// handles both shapes for backward compat.
#[serde(default, deserialize_with = "deserialize_string_or_vec")]
pub matrix_corpus: Vec<String>,
}
/// Accept `key = "x"` or `key = ["x", "y"]` in TOML/JSON. Empty string or
/// missing field → empty vec.
fn deserialize_string_or_vec<'de, D>(d: D) -> Result<Vec<String>, D::Error>
where D: serde::Deserializer<'de> {
use serde::de::Error;
let v = serde_json::Value::deserialize(d).map_err(D::Error::custom)?;
match v {
serde_json::Value::Null => Ok(vec![]),
serde_json::Value::String(s) if s.is_empty() => Ok(vec![]),
serde_json::Value::String(s) => Ok(vec![s]),
serde_json::Value::Array(a) => a
.into_iter()
.map(|x| x.as_str().map(String::from)
.ok_or_else(|| D::Error::custom("matrix_corpus array must contain strings")))
.collect(),
other => Err(D::Error::custom(format!("matrix_corpus must be string or array, got {other:?}"))),
}
}
#[derive(Clone, Debug, Deserialize)]
@@ -234,7 +257,7 @@ pub struct DecisionTrace {
pub task_class_matched: bool,
pub source: &'static str, // "config" | "default" | "force_mode"
pub fallbacks: Vec<String>,
pub matrix_corpus: Option<String>,
pub matrix_corpus: Vec<String>,
pub notes: Vec<String>,
}
@@ -279,7 +302,7 @@ pub async fn route(
task_class_matched: cfg.lookup(&req.task_class).is_some(),
source: "force_mode",
fallbacks: vec![],
matrix_corpus: None,
matrix_corpus: vec![],
notes,
},
}));
@@ -349,7 +372,7 @@ pub async fn route(
task_class_matched: false,
source: "default",
fallbacks: cfg.default.fallback_modes.clone(),
matrix_corpus: None,
matrix_corpus: vec![],
notes,
},
}))
@@ -419,11 +442,13 @@ pub struct ExecuteRequest {
/// runner uses its built-in forensic-review framing.
#[serde(default)]
pub user_question: Option<String>,
/// Override the matrix corpus the runner queries. Defaults to the
/// task_class's matrix_corpus from modes.toml. Use for the corpus-
/// tightening experiment (Pass 2 of the 2026-04-26 mode sweep).
#[serde(default)]
pub force_matrix_corpus: Option<String>,
/// Override the matrix corpus (or corpora) the runner queries.
/// Accepts a single string or array — same semantics as
/// modes.toml's `matrix_corpus`. Empty/missing → use the task
/// class default. Multi-corpus path: top-k retrieved from each,
/// merged and re-sorted by score before the relevance filter.
#[serde(default, deserialize_with = "deserialize_string_or_vec")]
pub force_matrix_corpus: Vec<String>,
/// Override the relevance filter threshold (default 0.3). Setting
/// to 0 keeps every chunk; raising rejects more aggressively. Used
/// to find the threshold sweet spot per task class.
@@ -441,8 +466,13 @@ pub struct EnrichmentSources {
pub bug_fingerprints_count: usize,
pub matrix_chunks_kept: usize,
pub matrix_chunks_dropped: usize,
pub matrix_corpus: Option<String>,
pub matrix_corpus: Vec<String>,
pub relevance_filter_used: bool,
/// Set when the model-aware downgrade fires — records the mode the
/// caller was originally routed to before is_weak_model() flipped
/// it. None means no downgrade happened.
#[serde(skip_serializing_if = "Option::is_none")]
pub downgraded_from: Option<String>,
pub enrichment_warnings: Vec<String>,
/// Which enrichment knobs the runner used for this mode. Lets
/// the comparison aggregator group runs by signal-set.
@@ -488,6 +518,30 @@ fn framing_text(f: ReviewerFraming) -> &'static str {
}
}
/// Strong-model heuristic for the model-aware enrichment downgrade.
///
/// Pass 5 variance test (2026-04-26, see docs/MODE_RUNNER_TUNING_PLAN.md)
/// proved that on `x-ai/grok-4.1-fast`, composing matrix corpora into the
/// `codereview_lakehouse` prompt LOST 5/5 head-to-head reps against the
/// matrix-free `codereview_isolation` mode. Strong models have enough
/// native capacity that bug fingerprints + adversarial framing + file
/// content carry them; matrix chunks displace depth-of-analysis.
///
/// We default to "strong" (downgrade matrix off) because most production
/// traffic uses paid models. The explicit `weak` predicate keeps the
/// list small and easy to extend — anything matching `:free` (OpenRouter
/// free tier) or the local last-resort qwen3.5 stays on the full
/// `codereview_lakehouse` path where matrix demonstrably helped during
/// the 2026-04-26 free-tier bake-off.
fn is_weak_model(model: &str) -> bool {
if model.ends_with(":free") || model.contains(":free/") {
return true;
}
// Local last-resort rung from the scrum ladder. Other local models
// can be added here as we test them.
matches!(model, "qwen3.5:latest" | "qwen3:latest")
}
pub async fn execute(
State(_state): State<V1State>,
Json(req): Json<ExecuteRequest>,
@@ -507,7 +561,40 @@ pub async fn execute(
.clone()
.or_else(|| tc.map(|t| t.default_model.clone()))
.unwrap_or_else(|| cfg.default.default_model.clone());
let matrix_corpus = tc.and_then(|t| t.matrix_corpus.clone());
let matrix_corpus: Vec<String> = tc
.map(|t| t.matrix_corpus.clone())
.unwrap_or_default();
// Model-aware enrichment downgrade (2026-04-26 pass 5 finding).
// If a caller resolves `codereview_lakehouse` against a strong
// model, downgrade to `codereview_isolation` so we don't pollute
// the prompt with matrix chunks the model would do better without.
// `LH_FORCE_FULL_ENRICHMENT=1` bypasses for diagnostic runs.
// `force_mode` from the caller is treated as opt-in to the chosen
// mode and skips the downgrade — experiments need to inspect exact
// mode behavior on whatever model they pass.
let force_full = std::env::var("LH_FORCE_FULL_ENRICHMENT")
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false);
let downgraded_from = if mode == "codereview_lakehouse"
&& req.force_mode.is_none()
&& !force_full
&& !is_weak_model(&model)
{
tracing::info!(
target: "v1::mode",
"downgrade codereview_lakehouse -> codereview_isolation for strong model {}",
model
);
Some(mode.clone())
} else {
None
};
let mode = if downgraded_from.is_some() {
"codereview_isolation".to_string()
} else {
mode
};
if !is_native_mode(&mode) {
// Native execute is the only path implemented; LLM-Team proxy
@@ -525,12 +612,17 @@ pub async fn execute(
}
// Caller can override the matrix corpus per-call (Pass 2 corpus
// tightening). Falls back to modes.toml default.
let matrix_corpus = req.force_matrix_corpus.clone().or(matrix_corpus);
// tightening). Empty force_matrix_corpus falls back to modes.toml.
let matrix_corpus: Vec<String> = if req.force_matrix_corpus.is_empty() {
matrix_corpus
} else {
req.force_matrix_corpus.clone()
};
let flags = flags_for_mode(&mode);
let mut sources = EnrichmentSources {
matrix_corpus: matrix_corpus.clone(),
flags: Some(flags),
downgraded_from: downgraded_from.clone(),
..Default::default()
};
@@ -613,38 +705,64 @@ pub async fn execute(
}
}
// Step 3: matrix corpus search (if configured for this task class).
// Step 3: matrix corpus search. Multi-corpus path: query top_k from
// each, merge, re-sort by score, take top 8 overall before the
// relevance filter — orthogonal corpora (e.g. arch + symbols) get
// composed without one swamping the other on chunk count alone.
let mut raw_chunks: Vec<serde_json::Value> = vec![];
if flags.include_matrix_chunks {
if let Some(corpus) = &matrix_corpus {
let body = serde_json::json!({
"index_name": corpus,
"query": format!("{} {}\n{}", req.task_class, req.file_path, &file_content[..file_content.len().min(500)]),
"top_k": 8,
});
match client
.post("http://localhost:3100/vectors/search")
.json(&body)
.send()
.await
{
Ok(r) if r.status().is_success() => {
if let Ok(j) = r.json::<serde_json::Value>().await {
raw_chunks = j
.get("results")
.and_then(|v| v.as_array())
.cloned()
.unwrap_or_default();
if flags.include_matrix_chunks && !matrix_corpus.is_empty() {
let query_str = format!(
"{} {}\n{}",
req.task_class,
req.file_path,
&file_content[..file_content.len().min(500)]
);
let per_corpus_k = if matrix_corpus.len() == 1 { 8 } else { 6 };
for corpus in &matrix_corpus {
let body = serde_json::json!({
"index_name": corpus,
"query": query_str,
"top_k": per_corpus_k,
});
match client
.post("http://localhost:3100/vectors/search")
.json(&body)
.send()
.await
{
Ok(r) if r.status().is_success() => {
if let Ok(j) = r.json::<serde_json::Value>().await {
if let Some(arr) = j.get("results").and_then(|v| v.as_array()) {
for mut c in arr.iter().cloned() {
// Tag the corpus origin on each chunk so
// dropped/kept telemetry can attribute
// signal back to its source corpus.
if let serde_json::Value::Object(ref mut obj) = c {
obj.insert(
"corpus".to_string(),
serde_json::Value::String(corpus.clone()),
);
}
raw_chunks.push(c);
}
}
}
}
Ok(r) => sources
.enrichment_warnings
.push(format!("matrix_search[{}] HTTP {}", corpus, r.status())),
Err(e) => sources
.enrichment_warnings
.push(format!("matrix_search[{}] err: {e}", corpus)),
}
Ok(r) => sources
.enrichment_warnings
.push(format!("matrix_search HTTP {}", r.status())),
Err(e) => sources
.enrichment_warnings
.push(format!("matrix_search err: {e}")),
}
} // close `if let Some(corpus)`
// Sort merged chunks by score desc and take the global top 8.
raw_chunks.sort_by(|a, b| {
let sa = a.get("score").and_then(|v| v.as_f64()).unwrap_or(0.0);
let sb = b.get("score").and_then(|v| v.as_f64()).unwrap_or(0.0);
sb.partial_cmp(&sa).unwrap_or(std::cmp::Ordering::Equal)
});
raw_chunks.truncate(8);
}
// Step 4: relevance filter — drop adjacency pollution.
@@ -709,9 +827,16 @@ pub async fn execute(
if flags.include_matrix_chunks && !kept_chunks.is_empty() {
user_prompt.push_str("📁 RELATED CONTEXT (matrix chunks):\n");
for c in &kept_chunks {
let src = c.get("source").and_then(|v| v.as_str()).unwrap_or("?");
let txt = c.get("text").and_then(|v| v.as_str()).unwrap_or("");
user_prompt.push_str(&format!(" [{}] {}\n", src, &txt[..txt.len().min(280)]));
// Prefer doc_id for the tag — corpus builders encode origin
// in doc_id (e.g. `adr:017`, `phase:19`) so the reviewer sees
// useful provenance instead of a generic source label.
let tag = c.get("doc_id").and_then(|v| v.as_str())
.filter(|s| !s.is_empty())
.or_else(|| c.get("source").and_then(|v| v.as_str()))
.unwrap_or("?");
let txt = c.get("text").or_else(|| c.get("chunk_text"))
.and_then(|v| v.as_str()).unwrap_or("");
user_prompt.push_str(&format!(" [{}] {}\n", tag, &txt[..txt.len().min(280)]));
}
user_prompt.push_str("\n");
}