pathway_memory: consensus-designed sidecar + hot-swap learning loop

10-probe N=3 consensus (kimi-k2:1t / gpt-oss:120b / qwen3.5:latest /
deepseek-v3.1:671b / qwen3-coder:480b / mistral-large-3:675b /
qwen3.5:397b + 2 stability re-probes; 2 openrouter probes 429'd) locked
the design across three rounds. Full JSON responses in
data/_kb/consensus_reducer_design_{mocq3akn,mocq6pi1,mocqatik}.json.

What it does

Preserves FULL backtrack context per reviewed file (ladder attempts +
latencies + reject reasons, KB chunks with provenance + cosine + rank,
observer signals, context7 bridge hits, sub-pipeline calls, audit
consensus) and indexes them by narrow fingerprint for hot-swap of
proven review pathways.

When scrum reviews a file:
  1. narrow fingerprint = task_class + file_prefix + signal_class
  2. query_hot_swap checks pathway memory for a match that passes
     probation (≥3 replays @ ≥80% success) + audit gate + similarity
     (≥0.90 cosine on normalized-metadata-token embedding)
  3. if hot-swap eligible, recommended model tried first in the ladder
  4. replay outcome reported back, updating the pathway's success_rate
  5. pathways below 0.80 after ≥3 replays retire permanently (sticky)
  6. full PathwayTrace always inserted at end of review — hot-swap
     grows with use, it doesn't bootstrap from nothing

Gate design is load-bearing:
  - narrow fingerprint (6 of 8 consensus models converged on the same
    3-field composition; lock) — enables generalization within crate
  - probation ≥3 replays — binomial tail at 80% is ~5%, below is noise
  - success rate ≥0.80 — mistral + qwen3-coder independently proposed
    this exact threshold across two rounds
  - similarity ≥0.90 — middle of the 0.85/0.95 consensus spread
  - bootstrap: null audit_consensus ALLOWED (auditor → pathway update
    not wired yet; probation + success_rate gates alone enforce safety
    during bootstrap; explicit audit FAIL still blocks)
  - retirement is sticky — prevents oscillation on noise

Files

  + crates/vectord/src/pathway_memory.rs  (new, 600 lines + 18 tests)
    PathwayTrace, LadderAttempt, KbChunkRef, ObserverSignal, BridgeHit,
    SubPipelineCall, AuditConsensus, HotSwapCandidate, PathwayMemory,
    PathwayMemoryStats. 18/18 tests green.
    Cosine + 32-bucket L2-normalized embedding; mirror of TS impl.
  M crates/vectord/src/lib.rs
    pub mod pathway_memory;
  M crates/vectord/src/service.rs
    VectorState grows pathway_memory field;
    4 HTTP handlers (/pathway/insert, /pathway/query,
    /pathway/record_replay, /pathway/stats).
  M crates/gateway/src/main.rs
    Construct PathwayMemory + load from storage on boot,
    wire into VectorState.
  M tests/real-world/scrum_master_pipeline.ts
    Byte-matching TS bucket-hash (verified same bucket indices as
    Rust); pre-ladder hot-swap query; ladder reorder on hit;
    per-attempt latency capture; post-accept trace insert
    (fire-and-forget); replay outcome recording;
    observer /event emits pathway_hot_swap_hit, pathway_similarity,
    rungs_saved per review for the VCP UI.
  M ui/server.ts
    /data/pathway_stats aggregates /vectors/pathway/stats +
    scrum_reviews.jsonl window for the value metric.
  M ui/ui.js
    Three new metric cards:
      · pathway reuse rate (activity: is it firing?)
      · avg rungs saved (value: is it earning its keep?)
      · pathways tracked (stability: retirement = learning)

What's not in this commit (queued)

  - auditor → pathway audit_consensus update wire (explicit audit-fail
    block activates when this lands)
  - bridge_hits + sub_pipeline_calls population from context7 / LLM
    Team extract results (fields wired, callers not yet)
  - replay log (PathwayReplayOutcome {matched_id, succeeded, ts}) as
    a separate jsonl for forensic audit of why specific replays failed

Why > summarization

Summaries discard the causal chain. With this, auditor can verify
citation provenance, applier can distinguish lucky from learned paths,
and the matrix indexing actually stores end-to-end pathways instead of
just RAG chunks — which is what J meant by "why aren't we using it
for everything."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ui/server.ts

@@ -349,6 +349,44 @@ Bun.serve({
     if (path === "/data/outcomes") return Response.json(await tailJsonl(`${KB}/outcomes.jsonl`, 30));
     if (path === "/data/audit_facts") return Response.json(await tailJsonl(`${KB}/audit_facts.jsonl`, 30));
 
+    // Pathway memory — consensus-designed sidecar (2026-04-24). Two
+    // exposed metrics: reuse_rate (activity — is it firing?) and
+    // avg_rungs_saved_per_commit (value — is it earning its keep?).
+    // Round-3 consensus (qwen3.5:397b) pointed out that activity
+    // without value tells us nothing; the UI needs both to judge the
+    // health of the hot-swap learning loop.
+    if (path === "/data/pathway_stats") {
+      try {
+        const r = await fetch("http://localhost:3100/vectors/pathway/stats", { signal: AbortSignal.timeout(3000) });
+        if (!r.ok) return Response.json({ error: `vectord ${r.status}`, stats: null });
+        const stats = await r.json();
+        // Tail recent scrum events to compute avg_rungs_saved_per_commit
+        // (a committed review = any row in scrum_reviews.jsonl; rungs_saved
+        // only populates when pathway memory fired AND the recommended
+        // model actually produced the accept).
+        const reviews = await tailJsonl(`${KB}/scrum_reviews.jsonl`, 200);
+        let totalCommits = 0;
+        let totalRungsSaved = 0;
+        let hotSwapHits = 0;
+        for (const r of reviews) {
+          totalCommits++;
+          if (r.pathway_hot_swap_hit) hotSwapHits++;
+          if (typeof r.rungs_saved === "number") totalRungsSaved += r.rungs_saved;
+        }
+        return Response.json({
+          stats,
+          scrum_window: {
+            reviews: totalCommits,
+            hot_swap_hits: hotSwapHits,
+            pathway_reuse_rate: totalCommits ? hotSwapHits / totalCommits : 0,
+            avg_rungs_saved_per_commit: totalCommits ? totalRungsSaved / totalCommits : 0,
+          },
+        });
+      } catch (e) {
+        return Response.json({ error: (e as Error).message, stats: null });
+      }
+    }
+
     if (path.startsWith("/data/file/")) {
       const relpath = decodeURIComponent(path.slice("/data/file/".length));
       return Response.json(await fileHistory(relpath));