ADR-021: semantic-correctness layer lands in pathway_memory (A+B+C)

Phase A — data model (vectord/src/pathway_memory.rs):
  + SemanticFlag enum (9 variants: UnitMismatch, TypeConfusion,
    NullableConfusion, OffByOne, StaleReference, PseudoImpl, DeadCode,
    WarningNoise, BoundaryViolation) as #[serde(tag = "kind")]
  + TypeHint { source, symbol, type_repr }
  + BugFingerprint { flag, pattern_key, example, occurrences }
  + PathwayTrace gains semantic_flags, type_hints_used, bug_fingerprints
    all #[serde(default)] for back-compat deserialization of pre-ADR-021
    traces on disk
  + build_pathway_vec now tokenizes flag:{variant} + bug:{flag}:{key}
    so traces with different bug histories cluster separately in the
    similarity gate (proven by pathway_vec_differs_when_bug_fingerprint_added
    test)

Phase B — producer (scrum_master_pipeline.ts):
  + Prompt addendum: each finding must carry `**Flag: <CATEGORY>**` tag
    alongside the existing Confidence: NN% tag. 9 category choices plus
    `None` for improvements that aren't bug-shaped.
  + Parser extracts tagged flags from reviewer markdown; falls back to
    bare-word match if reviewer omits the label. Deduplicated per trace.
  + PathwayTracePayload gains semantic_flags / type_hints_used /
    bug_fingerprints fields. Wire format matches Rust serde tagged enum
    so TS and Rust interop directly.

Phase C — pre-review enrichment:
  + new `/vectors/pathway/bug_fingerprints` endpoint aggregates
    occurrences by (flag, pattern_key) across traces sharing a narrow
    fingerprint, sorts by frequency, returns top-K.
  + scrum calls it before the ladder and prepends a PATHWAY MEMORY
    preamble to the reviewer prompt ("these patterns appeared N times
    on this file area before — check for recurrences"). Empty on
    fresh install; grows as the matrix index learns.

Tests: 27 pathway_memory tests green (was 18). New tests:
  - pathway_trace_deserializes_without_new_fields_backcompat
  - semantic_flag_serializes_as_tagged_enum
  - bug_fingerprint_roundtrips_through_serde
  - pathway_vec_differs_when_bug_fingerprint_added
  - semantic_flag_discriminates_by_variant
  - bug_fingerprints_aggregate_by_pattern_key (sums occurrences, sorts desc)
  - bug_fingerprints_empty_for_unseen_fingerprint
  - bug_fingerprints_respects_limit
  - insert_preserves_semantic_fields (roundtrip via persist + reload)

Workspace warnings unchanged at 11.

What's still queued (not this commit):
  - type_hints_used population from catalogd column types + Arrow schema
  - bug_fingerprint extraction from reviewer output (Phase D — for now
    semantic_flags populate but the fingerprint key requires parsing
    code-shape from the finding; next iteration's work)
  - auditor → pathway audit_consensus update wire (explicit-fail gate)

Why this commit matters: the mechanical applier's gates are syntactic
(warning count, patch size, rationale-token alignment). The
queryd/delta.rs base_rows bug (86901f8) was found by human reading —
unit mismatch between row counts and file counts. At 100 bugs this
deep, humans can't catch them all; the matrix index has to learn the
shapes. This commit gives it the fields to learn into and the surface
to read from.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/real-world/scrum_master_pipeline.ts

@@ -34,10 +34,12 @@ const FILE_TREE_SPLIT_THRESHOLD = 6000;
 const FILE_SHARD_SIZE = 3500;
 // Appended jsonl so auditor's kb_query can surface scrum findings for
 // files touched by a PR under review. Part of cohesion plan Phase C.
-const SCRUM_REVIEWS_JSONL = "/home/profit/lakehouse/data/_kb/scrum_reviews.jsonl";
+const SCRUM_REVIEWS_JSONL = process.env.LH_SCRUM_REVIEWS_OUT
+  || "/home/profit/lakehouse/data/_kb/scrum_reviews.jsonl";
 const OUT_DIR = `/home/profit/lakehouse/tests/real-world/runs/scrum_${Date.now().toString(36)}`;
 
-const PRD_PATH = "/home/profit/lakehouse/docs/PRD.md";
+const PRD_PATH = process.env.LH_SCRUM_PRD
+  || "/home/profit/lakehouse/docs/PRD.md";
 // Using CONTROL_PLANE_PRD as the "suggested changes" doc since it
 // describes the Phase 38-44 target architecture and is on main.
 // Override via LH_SCRUM_PROPOSAL env to point at a fix-wave doc
@@ -258,6 +260,11 @@ interface PathwayTracePayload {
   reducer_summary: string;
   final_verdict: string;
   pathway_vec: number[];
+  // ADR-021 semantic-correctness layer. `kind` field matches the Rust
+  // serde(tag = "kind") wire format — TS and Rust interop directly.
+  semantic_flags: { kind: string }[];
+  type_hints_used: { source: string; symbol: string; type_repr: string }[];
+  bug_fingerprints: { flag: { kind: string }; pattern_key: string; example: string; occurrences: number }[];
   replay_count: number;
   replays_succeeded: number;
   retired: boolean;
@@ -289,6 +296,33 @@ async function recordPathwayReplay(pathwayId: string, succeeded: boolean): Promi
   }
 }
 
+// ADR-021 Phase C: pre-review enrichment. Fetch aggregated bug
+// fingerprints for this narrow fingerprint (same key as hot-swap —
+// task_class + file_prefix + signal_class) so the reviewer prompt
+// can explicitly warn "this file area has had these bug patterns
+// before." Empty on fresh install; grows as the matrix index learns.
+interface BugFingerprintRow {
+  flag: { kind: string };
+  pattern_key: string;
+  example: string;
+  occurrences: number;
+}
+async function fetchBugFingerprints(taskClass: string, filePath: string, signalClass: string | null, limit: number): Promise<BugFingerprintRow[]> {
+  try {
+    const r = await fetch(`${GATEWAY}/vectors/pathway/bug_fingerprints`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({ task_class: taskClass, file_path: filePath, signal_class: signalClass, limit }),
+      signal: AbortSignal.timeout(5000),
+    });
+    if (!r.ok) return [];
+    const j = await r.json() as { fingerprints: BugFingerprintRow[] };
+    return j.fingerprints ?? [];
+  } catch {
+    return [];
+  }
+}
+
 // Deterministic signal_class lookup from scrum_reviews.jsonl history.
 // First-time files get `null`. Files seen before get the signal class
 // the observer assigned on their most-recent review (if any). Keeps the
@@ -534,6 +568,23 @@ Attach a self-assessed **Confidence: NN%** to every suggested change AND every g
 - <50%: genuinely uncertain — include regardless so downstream knows to investigate before applying
 Format each finding as: \`**1.** <change>. **Confidence: NN%.**\` (in tables, add a final "Confidence" column.) Low confidence is valuable signal — do not round up.
 
+**Per-finding semantic-flag tag (ADR-021, required on every finding):**
+Also attach a \`**Flag: <CATEGORY>**\` on each finding so the pathway-memory matrix index can cluster bug classes over time. Pick the ONE tag that best fits; if none fits, use \`None\`. Allowed categories:
+- \`UnitMismatch\` — operation combines values with different units (e.g. row_count - file_count, bytes - rows)
+- \`TypeConfusion\` — same type, wrong role (e.g. treating a PK as a row index)
+- \`NullableConfusion\` — unwrap-without-check or nullable-treated-as-non-null
+- \`OffByOne\` — loop / range / slice boundary mistake
+- \`StaleReference\` — calls a deprecated / removed / moved symbol
+- \`PseudoImpl\` — stub / todo!() / function named for work it doesn't do
+- \`DeadCode\` — unreachable or uncalled code
+- \`WarningNoise\` — compiles green but would add a cargo warning
+- \`BoundaryViolation\` — crosses a crate/layer boundary it shouldn't
+- \`None\` — improvement or nicety that doesn't fit a bug category
+
+In tables, add a "Flag" column. Examples:
+  \`**1.** Rewrite base_rows calc. **Confidence: 90%.** **Flag: UnitMismatch.**\`
+  \`**2.** Extract retry loop. **Confidence: 75%.** **Flag: None.**\`
+
 Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-offset when relevant.`;
 
   const history: FileReview["attempts_history"] = [];
@@ -550,6 +601,24 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
   const signalClass = await lookupSignalClass(rel);
   const taskClass = "scrum_review";
   const hotSwap = await queryHotSwap(taskClass, rel, signalClass);
+
+  // ADR-021 Phase C: pre-review enrichment. Pull aggregated bug
+  // fingerprints the matrix index has learned for this narrow
+  // fingerprint and prepend to the reviewer prompt as historical
+  // context. This is the compounding mechanism — iter-N reviewer
+  // sees what iter-(N-1) and earlier found, so the grammar of bugs
+  // accumulates instead of being re-discovered each iteration.
+  const pastFingerprints = await fetchBugFingerprints(taskClass, rel, signalClass, 5);
+  let pathwayPreamble = "";
+  if (pastFingerprints.length > 0) {
+    pathwayPreamble = "═══ PATHWAY MEMORY — BUGS PREVIOUSLY FOUND ON THIS FILE AREA (ADR-021) ═══\n" +
+      "The matrix index has flagged these patterns on the same task_class + file_prefix + signal_class before. Check this file for recurrences of the same shape:\n\n" +
+      pastFingerprints.map((fp, i) =>
+        `${i + 1}. [${fp.flag.kind}] pattern=\`${fp.pattern_key}\` occurrences=${fp.occurrences}\n   example: ${fp.example.slice(0, 160)}`
+      ).join("\n") +
+      "\n═══\n\n";
+    log(`  📚 pathway memory: ${pastFingerprints.length} historical bug pattern(s) prepended to prompt`);
+  }
   let hotSwapOrderedIndices: number[] | null = null;
   if (hotSwap) {
     // Reorder the ladder to try the recommended model first. Rung
@@ -574,12 +643,12 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
       ? `\n\n═══ PRIOR ATTEMPTS FAILED. Specific issues to fix: ═══\n${history.map(h => `Attempt ${h.n} (${h.model}, ${h.chars} chars): ${h.status} — ${h.error ?? "thin/unstructured answer"}`).join("\n")}\n═══`
       : "";
 
-    log(`  attempt ${n}/${MAX_ATTEMPTS}: ${rung.provider}::${rung.model}${learning ? " [w/ learning]" : ""}`);
+    log(`  attempt ${n}/${MAX_ATTEMPTS}: ${rung.provider}::${rung.model}${learning ? " [w/ learning]" : ""}${pathwayPreamble ? " [w/ pathway memory]" : ""}`);
     const attemptStarted = Date.now();
     const r = await chat({
       provider: rung.provider,
       model: rung.model,
-      prompt: baseTask + learning,
+      prompt: pathwayPreamble + baseTask + learning,
       max_tokens: 1500,
     });
     const attemptMs = Date.now() - attemptStarted;
@@ -660,6 +729,34 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
       : null;
     const conf_min = confidences.length ? Math.min(...confidences) : null;
 
+    // ADR-021 Phase B: extract per-finding semantic flags. Reviewer is
+    // prompted to tag each finding with one of 9 categories plus None.
+    // Patterns: "**Flag: UnitMismatch**", "Flag: OffByOne", table cell
+    // with the flag word, or bare-word match. Deduplicated per-trace
+    // so repeats in one review count once.
+    const FLAG_VARIANTS = [
+      "UnitMismatch", "TypeConfusion", "NullableConfusion", "OffByOne",
+      "StaleReference", "PseudoImpl", "DeadCode", "WarningNoise", "BoundaryViolation",
+    ];
+    const flagMatches = new Set<string>();
+    // Prefer matches anchored to the "Flag:" keyword; fall back to
+    // bare-word matches so older reviewers that mention a category
+    // without the "Flag:" prefix still contribute signal.
+    const patFlagLabeled = /(?:Flag[*:\s]*\s*)([A-Z][A-Za-z]+)/g;
+    for (const m of accepted.matchAll(patFlagLabeled)) {
+      if (FLAG_VARIANTS.includes(m[1])) flagMatches.add(m[1]);
+    }
+    // Second pass — bare-word matches for each variant, but ONLY if
+    // the labeled pass produced nothing. This avoids flagging every
+    // file that happens to mention "DeadCode" in a code sample.
+    if (flagMatches.size === 0) {
+      for (const v of FLAG_VARIANTS) {
+        const re = new RegExp(`\\b${v}\\b`);
+        if (re.test(accepted)) flagMatches.add(v);
+      }
+    }
+    const semantic_flags_arr = [...flagMatches].map(k => ({ kind: k }));
+
     // Score extraction — regex accepts decimals ("Score: 4.5/10") and
     // surrounding punctuation ("4/10 — mid"). iter 3 had 4 unparseable
     // scores because the prior regex /(\d)\s*\/\s*10/ missed decimals.
@@ -822,6 +919,9 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
         // query-time vector. The similarity gate will still discriminate
         // between pathways with the same fingerprint but different
         // ladder/KB profiles.
+        // Include semantic flag tokens in the embedding so traces with
+        // different bug histories cluster separately — matches Rust's
+        // build_pathway_vec exactly (flag:<Variant> token shape).
         pathway_vec: buildPathwayVec([
           taskClass,
           rel,
@@ -829,7 +929,11 @@ Respond with markdown. Be specific, not generic. Cite file-region + PRD-chunk-of
           ...pathwayAttempts.flatMap(a => [`rung:${a.rung}`, `model:${a.model}`, `accepted:${a.accepted}`]),
           ...topPrd.map(c => `kb:PRD.md`),
           ...topPlan.map(c => `kb:cohesion_plan`),
+          ...semantic_flags_arr.map(f => `flag:${f.kind}`),
         ]),
+        semantic_flags: semantic_flags_arr,
+        type_hints_used: [], // Phase C — pre-review enrichment from catalogd/arrow/truth
+        bug_fingerprints: [], // Phase C — fingerprint extraction from prompt responses
         replay_count: 0,
         replays_succeeded: 0,
         retired: false,