pathway_memory: consensus-designed sidecar + hot-swap learning loop

10-probe N=3 consensus (kimi-k2:1t / gpt-oss:120b / qwen3.5:latest /
deepseek-v3.1:671b / qwen3-coder:480b / mistral-large-3:675b /
qwen3.5:397b + 2 stability re-probes; 2 openrouter probes 429'd) locked
the design across three rounds. Full JSON responses in
data/_kb/consensus_reducer_design_{mocq3akn,mocq6pi1,mocqatik}.json.

What it does

Preserves FULL backtrack context per reviewed file (ladder attempts +
latencies + reject reasons, KB chunks with provenance + cosine + rank,
observer signals, context7 bridge hits, sub-pipeline calls, audit
consensus) and indexes them by narrow fingerprint for hot-swap of
proven review pathways.

When scrum reviews a file:
  1. narrow fingerprint = task_class + file_prefix + signal_class
  2. query_hot_swap checks pathway memory for a match that passes
     probation (≥3 replays @ ≥80% success) + audit gate + similarity
     (≥0.90 cosine on normalized-metadata-token embedding)
  3. if hot-swap eligible, recommended model tried first in the ladder
  4. replay outcome reported back, updating the pathway's success_rate
  5. pathways below 0.80 after ≥3 replays retire permanently (sticky)
  6. full PathwayTrace always inserted at end of review — hot-swap
     grows with use, it doesn't bootstrap from nothing

Gate design is load-bearing:
  - narrow fingerprint (6 of 8 consensus models converged on the same
    3-field composition; lock) — enables generalization within crate
  - probation ≥3 replays — binomial tail at 80% is ~5%, below is noise
  - success rate ≥0.80 — mistral + qwen3-coder independently proposed
    this exact threshold across two rounds
  - similarity ≥0.90 — middle of the 0.85/0.95 consensus spread
  - bootstrap: null audit_consensus ALLOWED (auditor → pathway update
    not wired yet; probation + success_rate gates alone enforce safety
    during bootstrap; explicit audit FAIL still blocks)
  - retirement is sticky — prevents oscillation on noise

Files

  + crates/vectord/src/pathway_memory.rs  (new, 600 lines + 18 tests)
    PathwayTrace, LadderAttempt, KbChunkRef, ObserverSignal, BridgeHit,
    SubPipelineCall, AuditConsensus, HotSwapCandidate, PathwayMemory,
    PathwayMemoryStats. 18/18 tests green.
    Cosine + 32-bucket L2-normalized embedding; mirror of TS impl.
  M crates/vectord/src/lib.rs
    pub mod pathway_memory;
  M crates/vectord/src/service.rs
    VectorState grows pathway_memory field;
    4 HTTP handlers (/pathway/insert, /pathway/query,
    /pathway/record_replay, /pathway/stats).
  M crates/gateway/src/main.rs
    Construct PathwayMemory + load from storage on boot,
    wire into VectorState.
  M tests/real-world/scrum_master_pipeline.ts
    Byte-matching TS bucket-hash (verified same bucket indices as
    Rust); pre-ladder hot-swap query; ladder reorder on hit;
    per-attempt latency capture; post-accept trace insert
    (fire-and-forget); replay outcome recording;
    observer /event emits pathway_hot_swap_hit, pathway_similarity,
    rungs_saved per review for the VCP UI.
  M ui/server.ts
    /data/pathway_stats aggregates /vectors/pathway/stats +
    scrum_reviews.jsonl window for the value metric.
  M ui/ui.js
    Three new metric cards:
      · pathway reuse rate (activity: is it firing?)
      · avg rungs saved (value: is it earning its keep?)
      · pathways tracked (stability: retirement = learning)

What's not in this commit (queued)

  - auditor → pathway audit_consensus update wire (explicit audit-fail
    block activates when this lands)
  - bridge_hits + sub_pipeline_calls population from context7 / LLM
    Team extract results (fields wired, callers not yet)
  - replay log (PathwayReplayOutcome {matched_id, succeeded, ts}) as
    a separate jsonl for forensic audit of why specific replays failed

Why > summarization

Summaries discard the causal chain. With this, auditor can verify
citation provenance, applier can distinguish lucky from learned paths,
and the matrix indexing actually stores end-to-end pathways instead of
just RAG chunks — which is what J meant by "why aren't we using it
for everything."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/real-world/scrum_master_pipeline.ts

@@ -157,6 +157,158 @@ async function embedBatch(texts: string[]): Promise<number[][]> {
   return (await r.json() as any).embeddings;
 }
 
+// ─── Pathway memory (2026-04-24 consensus design) ───────────────────
+//
+// Mirrors vectord/src/pathway_memory.rs. The bucket-hash vector MUST
+// byte-match the Rust implementation so traces written from TypeScript
+// are searchable against the same embedding space. Verified by running
+// both implementations on the same input tokens and asserting matching
+// bucket indices.
+
+function filePrefix(path: string): string {
+  return path.split("/").slice(0, 2).join("/");
+}
+
+function computePathwayId(taskClass: string, filePath: string, signalClass: string | null): string {
+  const h = createHash("sha256");
+  h.update(taskClass);
+  h.update("|");
+  h.update(filePrefix(filePath));
+  h.update("|");
+  h.update(signalClass ?? "");
+  return h.digest("hex");
+}
+
+// 32-bucket L2-normalized token hash. Same algorithm as Rust.
+function buildPathwayVec(tokens: string[]): number[] {
+  const buckets = new Array(32).fill(0);
+  for (const t of tokens) {
+    const d = createHash("sha256").update(t, "utf8").digest();
+    const b1 = d[0] % 32;
+    const b2 = d[8] % 32;
+    buckets[b1] += 1;
+    buckets[b2] += 1;
+  }
+  let norm = 0;
+  for (const v of buckets) norm += v * v;
+  norm = Math.sqrt(norm);
+  if (norm > 0) for (let i = 0; i < buckets.length; i++) buckets[i] /= norm;
+  return buckets;
+}
+
+// Build the minimal query vector for a pre-ladder hot-swap check. We
+// don't yet know the ladder attempts or KB chunks — the query vec is
+// computed from what we CAN know up front: task/file/signal. This is
+// a weaker embedding than the one computed at trace-insert time, but
+// similarity still discriminates between task/file/signal combinations.
+function buildQueryVec(taskClass: string, filePath: string, signalClass: string | null): number[] {
+  const tokens = [taskClass, filePath];
+  if (signalClass) tokens.push(`signal:${signalClass}`);
+  return buildPathwayVec(tokens);
+}
+
+interface HotSwapCandidate {
+  pathway_id: string;
+  similarity: number;
+  replay_count: number;
+  success_rate: number;
+  recommended_rung: number;
+  recommended_model: string;
+}
+
+async function queryHotSwap(taskClass: string, filePath: string, signalClass: string | null): Promise<HotSwapCandidate | null> {
+  try {
+    const query_vec = buildQueryVec(taskClass, filePath, signalClass);
+    const r = await fetch(`${GATEWAY}/vectors/pathway/query`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({ task_class: taskClass, file_path: filePath, signal_class: signalClass, query_vec }),
+      signal: AbortSignal.timeout(5000),
+    });
+    if (!r.ok) return null;
+    const j = await r.json() as { candidate: HotSwapCandidate | null };
+    return j.candidate ?? null;
+  } catch {
+    // Pathway service unavailable → run full ladder. Hot-swap is
+    // always an optimization, never a correctness requirement.
+    return null;
+  }
+}
+
+interface LadderAttemptRec {
+  rung: number;
+  model: string;
+  latency_ms: number;
+  accepted: boolean;
+  reject_reason: string | null;
+}
+
+interface PathwayTracePayload {
+  pathway_id: string;
+  task_class: string;
+  file_path: string;
+  signal_class: string | null;
+  created_at: string;
+  ladder_attempts: LadderAttemptRec[];
+  kb_chunks: { source_doc: string; chunk_id: string; cosine_score: number; rank: number }[];
+  observer_signals: { class: string; priors: string[]; prior_iter_outcomes: string[] }[];
+  bridge_hits: { library: string; version: string }[];
+  sub_pipeline_calls: { pipeline: string; result_summary: string }[];
+  audit_consensus: { pass: boolean; models: string[]; disagreements: number } | null;
+  reducer_summary: string;
+  final_verdict: string;
+  pathway_vec: number[];
+  replay_count: number;
+  replays_succeeded: number;
+  retired: boolean;
+}
+
+async function writePathwayTrace(trace: PathwayTracePayload): Promise<void> {
+  try {
+    await fetch(`${GATEWAY}/vectors/pathway/insert`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify(trace),
+      signal: AbortSignal.timeout(10000),
+    });
+  } catch {
+    // Fire-and-forget: scrum runs shouldn't fail if pathway insert fails.
+  }
+}
+
+async function recordPathwayReplay(pathwayId: string, succeeded: boolean): Promise<void> {
+  try {
+    await fetch(`${GATEWAY}/vectors/pathway/record_replay`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({ pathway_id: pathwayId, succeeded }),
+      signal: AbortSignal.timeout(5000),
+    });
+  } catch {
+    // Fire-and-forget. Not critical.
+  }
+}
+
+// Deterministic signal_class lookup from scrum_reviews.jsonl history.
+// First-time files get `null`. Files seen before get the signal class
+// the observer assigned on their most-recent review (if any). Keeps the
+// pathway fingerprint stable across iterations for LOOPING files.
+async function lookupSignalClass(filePath: string): Promise<string | null> {
+  try {
+    const { readFile } = await import("node:fs/promises");
+    const raw = await readFile(SCRUM_REVIEWS_JSONL, "utf8").catch(() => "");
+    if (!raw) return null;
+    const lines = raw.trim().split("\n").reverse();
+    for (const line of lines) {
+      try {
+        const r = JSON.parse(line);
+        if (r.file === filePath && r.signal_class) return r.signal_class;
+      } catch {}
+    }
+    return null;
+  } catch { return null; }
+}
+
 async function chat(opts: {
   provider: "ollama" | "ollama_cloud",
   model: string,
@@ -389,32 +541,63 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
   let acceptedModel = "";
   let acceptedOn = 0;
 
-  for (let i = 0; i < MAX_ATTEMPTS; i++) {
-    const n = i + 1;
+  // Pathway hot-swap pre-check. If a proven pathway exists for this
+  // (task, file_prefix, signal) combo with ≥3 replays at ≥80% success,
+  // skip the ladder and try its winning rung first. On success we
+  // record a positive replay; on failure we fall through to the full
+  // ladder and record a negative replay. Fire-and-forget — pathway
+  // service unavailable → null candidate → business as usual.
+  const signalClass = await lookupSignalClass(rel);
+  const taskClass = "scrum_review";
+  const hotSwap = await queryHotSwap(taskClass, rel, signalClass);
+  let hotSwapOrderedIndices: number[] | null = null;
+  if (hotSwap) {
+    // Reorder the ladder to try the recommended model first. Rung
+    // indices are preserved in the output so the trace still reflects
+    // the true ladder position the model sits at.
+    const recommendedIdx = LADDER.findIndex(r => r.model === hotSwap.recommended_model);
+    if (recommendedIdx >= 0) {
+      log(`  🔥 hot-swap candidate: ${hotSwap.recommended_model} (rung ${hotSwap.recommended_rung}, sim=${hotSwap.similarity.toFixed(3)}, success_rate=${hotSwap.success_rate.toFixed(2)}, ${hotSwap.replay_count} replays)`);
+      hotSwapOrderedIndices = [recommendedIdx, ...LADDER.map((_, i) => i).filter(i => i !== recommendedIdx)];
+    }
+  }
+  const ladderOrder = hotSwapOrderedIndices ?? LADDER.map((_, i) => i);
+
+  // Collect attempts for the pathway trace sidecar.
+  const pathwayAttempts: LadderAttemptRec[] = [];
+
+  for (let step = 0; step < MAX_ATTEMPTS; step++) {
+    const i = ladderOrder[step];
+    const n = step + 1;
     const rung = LADDER[i];
     const learning = history.length > 0
       ? `\n\n═══ PRIOR ATTEMPTS FAILED. Specific issues to fix: ═══\n${history.map(h => `Attempt ${h.n} (${h.model}, ${h.chars} chars): ${h.status} — ${h.error ?? "thin/unstructured answer"}`).join("\n")}\n═══`
       : "";
 
     log(`  attempt ${n}/${MAX_ATTEMPTS}: ${rung.provider}::${rung.model}${learning ? " [w/ learning]" : ""}`);
+    const attemptStarted = Date.now();
     const r = await chat({
       provider: rung.provider,
       model: rung.model,
       prompt: baseTask + learning,
       max_tokens: 1500,
     });
+    const attemptMs = Date.now() - attemptStarted;
 
     if (r.error) {
       history.push({ n, model: rung.model, status: "error", chars: 0, error: r.error.slice(0, 180) });
+      pathwayAttempts.push({ rung: i + 1, model: rung.model, latency_ms: attemptMs, accepted: false, reject_reason: `error: ${r.error.slice(0, 100)}` });
       log(`    ✗ error: ${r.error.slice(0, 80)}`);
       continue;
     }
     if (!isAcceptable(r.content)) {
       history.push({ n, model: rung.model, status: "thin", chars: r.content.length, error: `thin/unstructured (${r.content.length} chars)` });
+      pathwayAttempts.push({ rung: i + 1, model: rung.model, latency_ms: attemptMs, accepted: false, reject_reason: `thin (${r.content.length} chars)` });
       log(`    ✗ thin/unstructured (${r.content.length} chars)`);
       continue;
     }
     history.push({ n, model: rung.model, status: "accepted", chars: r.content.length });
+    pathwayAttempts.push({ rung: i + 1, model: rung.model, latency_ms: attemptMs, accepted: true, reject_reason: null });
     accepted = r.content;
     acceptedModel = `${rung.provider}/${rung.model}`;
     acceptedOn = n;
@@ -422,6 +605,15 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
     break;
   }
 
+  // Hot-swap bookkeeping: if we tried the recommended model first,
+  // report whether it worked so the pathway's success_rate updates.
+  if (hotSwap) {
+    const replaySucceeded = acceptedModel.endsWith(`/${hotSwap.recommended_model}`);
+    log(`  pathway replay ${replaySucceeded ? "✓" : "✗"} (${hotSwap.pathway_id.slice(0, 12)}…)`);
+    // Fire and forget — don't await; observer can handle it.
+    recordPathwayReplay(hotSwap.pathway_id, replaySucceeded);
+  }
+
   const review: FileReview = {
     file: rel,
     file_bytes: content.length,
@@ -599,6 +791,54 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
       console.error(`[scrum] failed to append scrum_reviews.jsonl: ${(e as Error).message}`);
     }
 
+    // Pathway trace sidecar (consensus-designed 2026-04-24). Captures
+    // FULL context (ladder attempts, KB chunks, observer signal, verdict)
+    // for similarity-based hot-swap in future iterations. First-review
+    // pathways start in probation (replay_count=0); they become
+    // hot-swappable only after ≥3 replays at ≥80% success.
+    try {
+      const pathwayTrace: PathwayTracePayload = {
+        pathway_id: computePathwayId(taskClass, rel, signalClass),
+        task_class: taskClass,
+        file_path: rel,
+        signal_class: signalClass,
+        created_at: row.reviewed_at,
+        ladder_attempts: pathwayAttempts,
+        kb_chunks: [
+          ...topPrd.map((c, idx) => ({
+            source_doc: "PRD.md", chunk_id: `prd@${c.offset}`, cosine_score: (c as any)._score ?? 0, rank: idx,
+          })),
+          ...topPlan.map((c, idx) => ({
+            source_doc: "cohesion_plan", chunk_id: `plan@${c.offset}`, cosine_score: (c as any)._score ?? 0, rank: idx,
+          })),
+        ],
+        observer_signals: signalClass ? [{ class: signalClass, priors: [], prior_iter_outcomes: [] }] : [],
+        bridge_hits: [],      // context7 not wired into scrum yet; empty for v1
+        sub_pipeline_calls: [], // LLM Team extract happens after this row; out of scope for v1
+        audit_consensus: null, // set by auditor's later N=3 pass, via /pathway/insert update
+        reducer_summary: accepted.slice(0, 4000),
+        final_verdict: verdict ?? "accepted",
+        // Vec built from the full attempts/chunks — richer than the
+        // query-time vector. The similarity gate will still discriminate
+        // between pathways with the same fingerprint but different
+        // ladder/KB profiles.
+        pathway_vec: buildPathwayVec([
+          taskClass,
+          rel,
+          ...(signalClass ? [`signal:${signalClass}`] : []),
+          ...pathwayAttempts.flatMap(a => [`rung:${a.rung}`, `model:${a.model}`, `accepted:${a.accepted}`]),
+          ...topPrd.map(c => `kb:PRD.md`),
+          ...topPlan.map(c => `kb:cohesion_plan`),
+        ]),
+        replay_count: 0,
+        replays_succeeded: 0,
+        retired: false,
+      };
+      writePathwayTrace(pathwayTrace); // fire-and-forget
+    } catch (e) {
+      console.error(`[scrum] pathway trace failed: ${(e as Error).message}`);
+    }
+
     // Close the scrum → observer loop (fix 2026-04-24). Architecture
     // audit surfaced: observer ring had 2000 ops, 1999 from Langfuse,
     // zero from scrum. Observer's analyzeErrors + PLAYBOOK_BUILDER loops
@@ -643,6 +883,20 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
           critical_failures_count,
           verified_components_count,
           missing_components_count,
+          // Pathway fields: emitted on every review so the observer
+          // can build a full picture of hot-swap performance over time.
+          // `pathway_hot_swap_hit` flags whether the first-tried rung
+          // this review was a pathway recommendation vs the default
+          // ladder top. `rungs_saved` quantifies the compute we avoided
+          // when a hot-swap landed — this is the value metric the VCP
+          // UI surfaces ("avg_rungs_saved_per_commit").
+          pathway_hot_swap_hit: hotSwap !== null,
+          pathway_id: hotSwap?.pathway_id ?? null,
+          pathway_similarity: hotSwap?.similarity ?? null,
+          pathway_success_rate: hotSwap?.success_rate ?? null,
+          rungs_saved: hotSwap && acceptedModel.endsWith(`/${hotSwap.recommended_model}`)
+            ? Math.max(0, hotSwap.recommended_rung - 1)
+            : 0,
           ts: row.reviewed_at,
         }),
         signal: AbortSignal.timeout(3000),