distillation: Phase 7 — replay-driven local model bootstrapping

Runtime layer that takes a task → retrieves matching playbooks/RAG
records → builds a structured context bundle → feeds it to a LOCAL
model (qwen3.5:latest, ~7B class) → validates output → escalates only
when needed → logs the full run as new evidence. NOT model training.
Pure runtime behavior shaping via retrieval against the Phase 0-6
distillation substrate.

Files (3 new + 1 modified):
  scripts/distillation/replay.ts             ~370 lines
  tests/distillation/replay.test.ts          10 tests, 19 expects
  scripts/distillation/distill.ts            +replay subcommand
  reports/distillation/phase7-replay-report.md

Test metrics: 145 cumulative distillation tests pass · 0 fail · 372 expects · 618ms

Real-data A/B on 3 tasks (same qwen3.5:latest local model, with vs
without retrieval) — proves the spec claim "local model improves
with retrieval":

Task 1 "Audit phase 38 provider routing":
  WITH retrieval:    cited V1State, openrouter, /v1/chat, ProviderAdapter,
                      PRD.md line ranges — REAL Lakehouse internals
  WITHOUT retrieval: invented "P99999, Z99999 placeholder codes" and
                      "production routing table" — pure fabrication

Task 2 "Verify pr_audit mode wired":
  WITH:    correct crates/gateway/src/main.rs path + lakehouse_answers_v1
  WITHOUT: same assertion, no proof, asserts confidently

Task 3 "Audit phase 40 PRD circuit breaker drift":
  WITH:    anchored on the actual audit finding "no breaker class found"
  WITHOUT: invented "0.0% failure rate vs 5.0% threshold" and signed
            off as PASS on broken code — exact failure mode the
            distillation pipeline was built to prevent

Both runs passed the structural validation gate (length, no hedges,
checklist token overlap) — the difference is grounding, supplied by
the retrieval layer pulling from exports/rag/playbooks.jsonl (446
records from earlier Phase 4 export).

Architecture:
  jaccard token overlap against rag corpus → top-K (default 8) split
  into accepted exemplars (top 3) + partial-warnings (top 2) + extracted
  validation_steps (lines starting verify|check|assert|ensure|confirm)
  → prompt assembly → qwen3.5:latest via /v1/chat (or OpenRouter
  for namespaced/free models) → deterministic validation gate →
  escalation to deepseek-v3.1:671b on fail with --allow-escalation
  → log to data/_kb/replay_runs.jsonl

Spec invariants enforced:
  - never bypass retrieval (--no-retrieval is explicit baseline, not default)
  - never discard provenance (task_hash + rag_ids + full bundle logged)
  - never allow free-form hallucinated output (validation gate is
    deterministic code, never an LLM)
  - log every run as new evidence (replay_run.v1 schema, append-only
    to data/_kb/replay_runs.jsonl)

CLI:
  ./scripts/distill replay --task "<input>" [--local-only]
                                            [--allow-escalation]
                                            [--no-retrieval]

What this unlocks:
  The substrate for "small-model bootstrapping" and "local inference
  dominance" J flagged after Phase 5. Phase 8+ closes the loop:
  schedule replay runs on common tasks, score outputs, feed accepted
  ones back into corpus, measure escalation rate decreasing over time.

Known limitations (documented in report):
  - Validation gate is structural not semantic (catches hedges/empty
    but not plausible-wrong). Phase 13 wiring: run auditor against
    every replay output.
  - Retrieval is jaccard keyword. Works at 446 corpus, scale via
    /vectors/search HNSW retrieval once corpus crosses ~10k.
  - Convergence claim is architectural (deterministic retrieval +
    low-temp call); longitudinal empirical study is Phase 8+.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

scripts/distillation/distill.ts

@@ -21,6 +21,7 @@ import { exportRag } from "./export_rag";
 import { exportSft } from "./export_sft";
 import { exportPreference } from "./export_preference";
 import { runAllWithReceipts } from "./receipts";
+import { replay } from "./replay";
 import { TRANSFORMS } from "./transforms";
 import { spawnSync } from "node:child_process";
 
@@ -87,6 +88,30 @@ async function main() {
       if (!r.summary.overall_passed) process.exit(1);
       break;
     }
+    case "replay": {
+      const taskIdx = process.argv.indexOf("--task");
+      if (taskIdx < 0 || !process.argv[taskIdx + 1]) {
+        console.error("usage: distill.ts replay --task \"<input>\" [--local-only] [--allow-escalation] [--no-retrieval]");
+        process.exit(2);
+      }
+      const r = await replay({
+        task: process.argv[taskIdx + 1],
+        local_only: process.argv.includes("--local-only"),
+        allow_escalation: process.argv.includes("--allow-escalation"),
+        no_retrieval: process.argv.includes("--no-retrieval"),
+      }, DEFAULT_ROOT);
+      console.log(`[replay] run_id=${r.recorded_run_id}`);
+      console.log(`[replay] retrieval: ${r.context_bundle ? r.context_bundle.retrieved_playbooks.length + " playbooks" : "DISABLED"}`);
+      console.log(`[replay] escalation_path: ${r.escalation_path.join(" → ")}`);
+      console.log(`[replay] model_used: ${r.model_used} · ${r.duration_ms}ms`);
+      console.log(`[replay] validation: ${r.validation_result.passed ? "PASS" : "FAIL"}${r.validation_result.reasons.length ? " (" + r.validation_result.reasons.join("; ") + ")" : ""}`);
+      console.log("");
+      console.log("─── response ───");
+      console.log(r.model_response.slice(0, 1500));
+      if (r.model_response.length > 1500) console.log(`... [${r.model_response.length - 1500} more chars]`);
+      if (!r.validation_result.passed && !process.argv.includes("--allow-escalation")) process.exit(1);
+      break;
+    }
     case "acceptance": {
       // Phase 6 — fixture-driven end-to-end gate. Spawns the dedicated
       // acceptance script so its non-zero exit propagates.
@@ -125,8 +150,10 @@ async function main() {
       console.log("  run-all            full pipeline with structured receipts (Phase 5)");
       console.log("  receipts           read summary for a run (--run-id <id>)");
       console.log("  acceptance         fixture-driven end-to-end gate (Phase 6)");
+      console.log("  replay             retrieval-driven local-model bootstrap (Phase 7) — needs --task");
       console.log("");
-      console.log("Flags: --dry-run, --include-partial, --include-review");
+      console.log("Flags: --dry-run, --include-partial, --include-review,");
+      console.log("       --task \"<text>\", --local-only, --allow-escalation, --no-retrieval");
       break;
     }
     default: