Cohesion: Python inventory + integration plan + Phase A verdict indexing

Three artifacts in one PR:

1. docs/PYTHON_INVENTORY.md — every .py file in the repo classified:
   Production (sidecar routers + 3 systemd services), Documented
   (kb_measure, kb_staffer_report), Manual (one-off tools), Dead
   (sidecar/sidecar/lab_ui.py + pipeline_lab.py are genuinely
   not imported anywhere).

2. docs/COHESION_INTEGRATION_PLAN.md — the "smarter DB" loop J
   called out as missing. Six phases A-F. Phase A ships here; B-F
   are named + sequenced for follow-up PRs. Each phase adds ONE
   wire of the loop; no single PR does them all.

3. Phase A wire (auditor verdicts → observer + KB):
   - auditor/audit.ts: after assembleVerdict, fire-and-forget POST
     to :3800/event with source="auditor" AND append to
     data/_kb/outcomes.jsonl with kind="audit". Errors log + drop
     — the verdict is still on disk at _auditor/verdicts/.
   - mcp-server/observer.ts: extend source union to include
     "auditor" | "bot" (was "mcp" | "scenario" only, which silently
     coerced my first auditor POST to source="scenario"). Accept
     body.ok OR body.success. Accept body.audit_duration_ms as a
     fallback for duration_ms. Uses body.one_liner as
     output_summary when set.

Live-verified after observer restart:
   re-audit PR #6 → verdict=request_changes, 4 findings (1 warn)
     observer: by_source={'auditor': 1}  (previously coerced to 'scenario')
     _kb/outcomes.jsonl tail: kind=audit sig=pr6-7fe47bab
       pr=6 overall=request_changes

The shape of the loop is now visible to downstream consumers. Phase
B (auditor's kb_query check reads these audit rows for history)
lands in a follow-up PR. Phase C-F similar.

NOT in this PR:
- Actually deleting lab_ui.py + pipeline_lab.py (operator decision,
  called out in the inventory doc)
- Cleaning up the 5 overlapping Python scripts (same)
- Phases B-F of the cohesion plan (separate PRs per wire)
- Integration test that asserts "smarter DB" across runs (Phase F)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.mcp.json

@@ -0,0 +1,12 @@
+{
+  "mcpServers": {
+    "lakehouse": {
+      "command": "bun",
+      "args": ["run", "/home/profit/lakehouse/mcp-server/index.ts"],
+      "env": {
+        "LAKEHOUSE_URL": "http://localhost:3100",
+        "MCP_TRANSPORT": "stdio"
+      }
+    }
+  }
+}

auditor/README.md

@@ -0,0 +1,53 @@
+# Lakehouse Claim Auditor
+
+A Bun sub-agent that watches open PRs on Gitea, reads the ship-claims
+in commit messages and PR bodies, and **hard-blocks** merges when the
+code doesn't back the claim.
+
+Rationale: when "compiles + one curl works" gets called "phase shipped,"
+placeholder code accumulates. This auditor runs every 90s, fetches
+each open PR, and subjects it to four checks:
+
+1. **Static diff** — grep/parse looking for placeholder patterns
+2. **Dynamic** — runs the never-before-executed hybrid test fixture
+3. **Cloud inference** — asks `gpt-oss:120b` via `/v1/chat` to
+   identify gaps in the diff
+4. **KB query** — looks up `data/_kb/` + observer for prior failure
+   patterns on similar claims
+
+Verdict is assembled, posted to Gitea as:
+- A **failing commit status** (hard block — branch protection
+  prevents merge)
+- A **review comment** explaining every finding
+
+## Run manually
+
+```bash
+cd /home/profit/lakehouse
+bun run auditor/index.ts
+```
+
+Defaults: polls every 90s, stops on `auditor.paused` file present.
+
+## State
+
+- `data/_auditor/state.json` — last-audited head SHA per PR
+- `data/_auditor/verdicts/{pr}-{sha}.json` — per-run verdict record
+
+## Where YOU edit
+
+`auditor/policy.ts` — the verdict assembler. Controls which findings
+block vs warn vs inform. All other code is mechanical: fetching,
+running checks, posting to Gitea.
+
+## Hard-block mechanism
+
+1. Commit status is posted as `failure` with context `lakehouse/auditor`
+2. If `main` branch protection requires `lakehouse/auditor` status
+   to pass, Gitea prevents merge
+3. When code is fixed and re-audit passes, status flips to `success`,
+   merge unblocks
+
+Enable branch protection (one-time, via Gitea UI or API):
+- `POST /repos/profit/lakehouse/branch_protections`
+- `{"branch_name": "main", "required_status_checks": {"contexts": ["lakehouse/auditor"]}}`

auditor/audit.ts

@@ -0,0 +1,243 @@
+// Orchestrator — runs all four checks on a PR, assembles a verdict,
+// posts to Gitea. This is task #8's integration layer; the poller
+// (task #9) calls this once per PR on every fresh head SHA.
+//
+// Hard-block mechanism: commit status posted with state="failure"
+// and context="lakehouse/auditor". If `main` branch protection
+// requires that context to pass, merge is physically impossible
+// until the auditor re-audits a fixed commit and flips the status
+// to "success".
+//
+// Human-readable reasoning: posted as a PR issue comment (not a
+// review — reviews have self-review restrictions on Gitea and the
+// auditor currently uses the same PAT as the PR author).
+
+import { readFile, writeFile, mkdir } from "node:fs/promises";
+import { join } from "node:path";
+import type { PrSnapshot, Verdict, Finding } from "./types.ts";
+import { getPrDiff, postCommitStatus, postIssueComment } from "./gitea.ts";
+import { parseClaims } from "./claim_parser.ts";
+import { assembleVerdict } from "./policy.ts";
+import { runStaticCheck } from "./checks/static.ts";
+import { runDynamicCheck } from "./checks/dynamic.ts";
+import { runInferenceCheck } from "./checks/inference.ts";
+import { runKbCheck } from "./checks/kb_query.ts";
+
+const VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/verdicts";
+
+export interface AuditOptions {
+  // Skip the cloud inference call (fast path for iteration). Default false.
+  skip_inference?: boolean;
+  // Skip the dynamic check (avoid running the hybrid fixture every PR,
+  // since it hits live services and mutates playbook state). Default false
+  // on `main`-branch-target PRs, true when auditing feature branches
+  // where the fixture would pollute state. Caller decides.
+  skip_dynamic?: boolean;
+  // Skip Gitea posting — useful for dry-runs / local testing.
+  // Default false.
+  dry_run?: boolean;
+}
+
+export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<Verdict> {
+  const t0 = Date.now();
+  const diff = await getPrDiff(pr.number);
+  const { claims } = parseClaims(pr);
+
+  // Run checks in parallel where they don't share mutable state.
+  // Static + kb_query + inference are all read-only. Dynamic mutates
+  // playbook state (nonce-scoped per run, but still live) so if
+  // skip_dynamic is false we still run it in parallel — the mutation
+  // is namespaced.
+  const [staticFindings, dynamicFindings, inferenceFindings, kbFindings] = await Promise.all([
+    runStaticCheck(diff),
+    opts.skip_dynamic ? Promise.resolve(stubFinding("dynamic", "skipped by options")) : runDynamicCheck(),
+    opts.skip_inference ? Promise.resolve(stubFinding("inference", "skipped by options")) : runInferenceCheck(claims, diff),
+    runKbCheck(claims),
+  ]);
+
+  const allFindings: Finding[] = [
+    ...staticFindings,
+    ...dynamicFindings,
+    ...inferenceFindings,
+    ...kbFindings,
+  ];
+
+  const duration_ms = Date.now() - t0;
+  const metrics = {
+    audit_duration_ms: duration_ms,
+    findings_total: allFindings.length,
+    findings_block: allFindings.filter(f => f.severity === "block").length,
+    findings_warn: allFindings.filter(f => f.severity === "warn").length,
+    findings_info: allFindings.filter(f => f.severity === "info").length,
+    claims_strong: claims.filter(c => c.strength === "strong").length,
+    claims_moderate: claims.filter(c => c.strength === "moderate").length,
+    claims_weak: claims.filter(c => c.strength === "weak").length,
+    claims_total: claims.length,
+    diff_bytes: diff.length,
+  };
+
+  const verdict = assembleVerdict(allFindings, metrics, pr.number, pr.head_sha);
+
+  await persistVerdict(verdict);
+
+  // Phase A of the cohesion plan (docs/COHESION_INTEGRATION_PLAN.md):
+  // make every audit verdict visible to the observer + KB. Enables
+  // future Phase B (kb_query sees prior audit history) without a
+  // separate backfill. Fire-and-forget: observer/KB failures don't
+  // block the Gitea post.
+  indexVerdictToObserver(verdict).catch(e =>
+    console.error(`[auditor] observer indexing failed: ${(e as Error).message}`));
+  appendVerdictToKbOutcomes(verdict).catch(e =>
+    console.error(`[auditor] kb outcomes append failed: ${(e as Error).message}`));
+
+  if (!opts.dry_run) {
+    await postToGitea(verdict);
+  }
+
+  return verdict;
+}
+
+// Phase A — verdict indexing.
+//
+// Two destinations, both append-only + non-blocking:
+//   1. observer :3800/event — ring buffer + data/_observer/ops.jsonl
+//   2. data/_kb/outcomes.jsonl — same file scenarios write to, with
+//      kind:"audit" so readers can filter
+//
+// Errors log + drop. The verdict is still on disk at
+// _auditor/verdicts/{pr}-{sha}.json; observer + KB are a convenience
+// surface, not a source of truth.
+
+const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
+const KB_OUTCOMES = "/home/profit/lakehouse/data/_kb/outcomes.jsonl";
+
+async function indexVerdictToObserver(v: Verdict): Promise<void> {
+  const payload = {
+    source: "auditor",
+    event_kind: "audit",
+    ok: v.overall === "approve",
+    sig_hash: `pr${v.pr_number}-${v.head_sha.slice(0, 8)}`,
+    pr_number: v.pr_number,
+    head_sha: v.head_sha,
+    overall: v.overall,
+    one_liner: v.one_liner,
+    findings_block: v.metrics.findings_block,
+    findings_warn: v.metrics.findings_warn,
+    audit_duration_ms: v.metrics.audit_duration_ms,
+    audited_at: v.audited_at,
+  };
+  const r = await fetch(`${OBSERVER_URL}/event`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify(payload),
+    signal: AbortSignal.timeout(3000),
+  });
+  if (!r.ok) throw new Error(`observer ${r.status}: ${await r.text()}`);
+}
+
+async function appendVerdictToKbOutcomes(v: Verdict): Promise<void> {
+  const { appendFile, mkdir } = await import("node:fs/promises");
+  const { dirname } = await import("node:path");
+  await mkdir(dirname(KB_OUTCOMES), { recursive: true });
+  const row = {
+    kind: "audit",
+    sig_hash: `pr${v.pr_number}-${v.head_sha.slice(0, 8)}`,
+    audited_at: v.audited_at,
+    pr_number: v.pr_number,
+    head_sha: v.head_sha,
+    overall: v.overall,
+    one_liner: v.one_liner,
+    ok_events: v.overall === "approve" ? 1 : 0,
+    total_events: 1,
+    findings: {
+      block: v.metrics.findings_block,
+      warn: v.metrics.findings_warn,
+      info: v.metrics.findings_info,
+    },
+    elapsed_secs: (v.metrics.audit_duration_ms ?? 0) / 1000,
+  };
+  await appendFile(KB_OUTCOMES, JSON.stringify(row) + "\n");
+}
+
+async function persistVerdict(v: Verdict): Promise<void> {
+  await mkdir(VERDICTS_DIR, { recursive: true });
+  const filename = `${v.pr_number}-${v.head_sha.slice(0, 12)}.json`;
+  await writeFile(join(VERDICTS_DIR, filename), JSON.stringify(v, null, 2));
+}
+
+export async function postToGitea(v: Verdict): Promise<void> {
+  // 1. Commit status — the hard block signal (if branch protection
+  //    is configured to require lakehouse/auditor on main).
+  const state = v.overall === "approve" ? "success" : "failure";
+  await postCommitStatus({
+    sha: v.head_sha,
+    state,
+    context: "lakehouse/auditor",
+    description: v.one_liner,
+    target_url: "", // no URL yet; could point to a verdicts dashboard
+  });
+
+  // 2. Issue comment — the reasoning. Gated so we don't spam the PR
+  //    with identical comments on re-audits of the same SHA. Caller
+  //    (poller) ensures we only re-audit fresh SHAs, but a dedup
+  //    marker inside the body keeps it idempotent if re-run.
+  const body = formatReviewBody(v);
+  await postIssueComment({ pr_number: v.pr_number, body });
+}
+
+function formatReviewBody(v: Verdict): string {
+  const byCheck: Record<string, Finding[]> = {};
+  for (const f of v.findings) {
+    (byCheck[f.check] ||= []).push(f);
+  }
+
+  const verdictEmoji =
+    v.overall === "approve" ? "✅" :
+    v.overall === "request_changes" ? "⚠️" :
+    "🛑";
+
+  const lines: string[] = [];
+  lines.push(`## Auditor verdict: ${verdictEmoji} \`${v.overall}\``);
+  lines.push("");
+  lines.push(`**One-liner:** ${v.one_liner}`);
+  lines.push(`**Head SHA:** \`${v.head_sha.slice(0, 12)}\``);
+  lines.push(`**Audited at:** ${v.audited_at}`);
+  lines.push("");
+
+  // Per-check sections, only if the check produced findings.
+  const checkOrder = ["static", "dynamic", "inference", "kb_query"] as const;
+  for (const check of checkOrder) {
+    const fs = byCheck[check] ?? [];
+    if (fs.length === 0) continue;
+    const bySev = {
+      block: fs.filter(f => f.severity === "block").length,
+      warn: fs.filter(f => f.severity === "warn").length,
+      info: fs.filter(f => f.severity === "info").length,
+    };
+    lines.push(`<details><summary><b>${check}</b> — ${fs.length} findings (${bySev.block} block, ${bySev.warn} warn, ${bySev.info} info)</summary>`);
+    lines.push("");
+    for (const f of fs) {
+      const mark = f.severity === "block" ? "🛑" : f.severity === "warn" ? "⚠️" : "ℹ️";
+      lines.push(`${mark} **${f.severity}** — ${f.summary}`);
+      for (const e of f.evidence.slice(0, 3)) {
+        lines.push(`  - \`${e.slice(0, 180).replace(/\n/g, " ")}\``);
+      }
+    }
+    lines.push("");
+    lines.push("</details>");
+    lines.push("");
+  }
+
+  lines.push("### Metrics");
+  lines.push("```json");
+  lines.push(JSON.stringify(v.metrics, null, 2));
+  lines.push("```");
+  lines.push("");
+  lines.push(`<sub>Lakehouse auditor · SHA ${v.head_sha.slice(0, 8)} · re-audit on new commit flips the status automatically.</sub>`);
+
+  return lines.join("\n");
+}
+
+function stubFinding(check: "dynamic" | "inference", why: string): Finding[] {
+  return [{ check, severity: "info", summary: `${check} check skipped — ${why}`, evidence: [why] }];
+}

auditor/checks/dynamic.ts

@@ -0,0 +1,91 @@
+// Dynamic execution check — runs the hybrid fixture and maps its
+// layer results to auditor Findings.
+//
+// A layer that fails with a "not implemented / 404 / slice N" error
+// gets severity=info (honest placeholder signal). A layer that fails
+// any other way gets severity=warn (something actually broke).
+// An info-level summary finding is always emitted carrying the real
+// numbers — shipped/placeholder phase counts, per-layer latency.
+
+import { runHybridFixture } from "../fixtures/hybrid_38_40_45.ts";
+import type { Finding } from "../types.ts";
+
+const PLACEHOLDER_MARKERS = [
+  "unimplemented",
+  " 404 ", "(404)", " 405 ", "(405)",
+  "slice 3", "slice 4", "slice 5",
+  "endpoint not built", "not yet",
+];
+
+function isPlaceholderFailure(err?: string): boolean {
+  if (!err) return false;
+  const low = err.toLowerCase();
+  return PLACEHOLDER_MARKERS.some(m => low.includes(m.toLowerCase()));
+}
+
+export async function runDynamicCheck(): Promise<Finding[]> {
+  const findings: Finding[] = [];
+
+  let result;
+  try {
+    result = await runHybridFixture();
+  } catch (e) {
+    // Fixture itself crashed — can't run dynamic check at all.
+    return [
+      {
+        check: "dynamic",
+        severity: "warn",
+        summary: `hybrid fixture crashed before completing: ${(e as Error).message.slice(0, 140)}`,
+        evidence: [(e as Error).message],
+      },
+    ];
+  }
+
+  // Per-layer findings for every non-ok layer.
+  for (const layer of result.layers) {
+    if (layer.ok) continue;
+    const placeholder = isPlaceholderFailure(layer.error);
+    findings.push({
+      check: "dynamic",
+      severity: placeholder ? "info" : "warn",
+      summary: placeholder
+        ? `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) honestly reports unimplemented`
+        : `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) failed — not a placeholder, a real failure`,
+      evidence: [
+        `evidence: ${layer.evidence.slice(0, 160)}`,
+        ...(layer.error ? [`error: ${layer.error.slice(0, 160)}`] : []),
+        `latency_ms: ${layer.latency_ms}`,
+      ],
+    });
+  }
+
+  // One overall summary with real numbers so the report shows what
+  // DID pass plus per-layer timing.
+  const metrics_preview = Object.entries(result.real_numbers)
+    .slice(0, 10)
+    .map(([k, v]) => `${k}=${v}`);
+  findings.push({
+    check: "dynamic",
+    severity: "info",
+    summary: `hybrid fixture overall=${result.overall}, shipped [${result.shipped_phases.join(", ")}], placeholder [${result.placeholder_phases.join(", ")}]`,
+    evidence: metrics_preview.length > 0 ? metrics_preview : ["no metrics emitted"],
+  });
+
+  // If the fixture ran at all but nothing passed, elevate one of the
+  // summary findings to warn — something more than "all honest
+  // placeholders" is wrong.
+  if (result.overall === "fail") {
+    findings.push({
+      check: "dynamic",
+      severity: "warn",
+      summary: `hybrid fixture: 0 layers passed (overall=fail)`,
+      evidence: [
+        "a total fixture fail usually means a precondition service is down",
+        "(gateway /health / sidecar / Langfuse /v1/chat) — NOT necessarily",
+        "the PR's code problem. Check service status before blaming the PR.",
+      ],
+    });
+  }
+
+  return findings;
+}

auditor/checks/inference.ts

@@ -0,0 +1,206 @@
+// Cloud inference check — wraps the proven run_codereview pattern
+// from llm_team_ui.py (same 3-stage framing, same cloud model) to
+// critique a PR's claims against its diff.
+//
+// Proved out 2026-04-22 via /tmp/codereview_runner.py — gpt-oss:120b
+// caught a real ternary bug in auditor/fixtures/hybrid_38_40_45.ts
+// that unit tests missed. This module reuses the reviewer prompt
+// shape (bugs / security / performance / style / edge cases) and
+// adds claim-vs-diff specific framing.
+//
+// Call surface: runInferenceCheck(claims, diff) → Finding[].
+// Cloud latency budget: ~60s (gpt-oss:120b reviewer typically 35-50s
+// with a 15KB diff + claim list).
+
+import type { Claim, Finding } from "../types.ts";
+
+const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
+const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "gpt-oss:120b";
+// 40KB comfortably fits gpt-oss:120b's context. PR #1 (~39KB) was
+// previously truncated at 15KB causing the reviewer to miss later
+// files (gitea.ts, policy.ts) and flag "no Gitea client present" as a
+// block finding when the file was simply outside the truncation window.
+const MAX_DIFF_CHARS = 40000;
+const CALL_TIMEOUT_MS = 120_000;
+
+export async function runInferenceCheck(claims: Claim[], diff: string): Promise<Finding[]> {
+  if (claims.length === 0) {
+    return [{
+      check: "inference",
+      severity: "info",
+      summary: "no ship-claims extracted — skipping cloud inference",
+      evidence: ["parser returned empty claim list; nothing to verify against cloud"],
+    }];
+  }
+
+  const truncated = diff.length > MAX_DIFF_CHARS
+    ? diff.slice(0, MAX_DIFF_CHARS) + `\n...[${diff.length - MAX_DIFF_CHARS} more chars truncated]`
+    : diff;
+
+  // Build the reviewer prompt in the same shape as run_codereview's
+  // review stage (llm_team_ui.py:10950), adapted for claim verification:
+  //   "Task: ..."
+  //   "Code: ..."
+  //   "Review: bugs/security/perf/style/edge. Provide corrected code."
+  // We add: claim list upfront + ask for structured JSON verdict.
+  const systemMsg = [
+    "You review pull-request diffs against the author's own ship-claims.",
+    "For each claim, decide: is it backed by actual code in the diff, or is",
+    "it placeholder / aspirational / unwired?",
+    "",
+    "A claim is BACKED when the diff contains a real code path that delivers",
+    "the claimed behavior. A claim is NOT BACKED when:",
+    "  - the claim asserts functionality but the diff only adds types/fields",
+    "    with no consumer",
+    "  - the claim mentions tests but no test function was added",
+    "  - the claim claims integration but the integration point is a stub",
+    "  - the diff contains unimplemented!() / todo!() / TODO comments",
+    "  - the claim says 'works end-to-end' but the diff has no end-to-end test",
+    "",
+    "Respond with strict JSON only. No prose before or after. Shape:",
+    "{",
+    '  "claim_verdicts": [',
+    '    {"claim_idx": 0, "backed": false, "evidence": "short reason"}',
+    "  ],",
+    '  "unflagged_gaps": [',
+    '    {"location": "file:line", "summary": "short description"}',
+    "  ]",
+    "}",
+  ].join("\n");
+
+  const userMsg = [
+    `Ship-claims the author made (numbered 0..N-1):`,
+    claims.map((c, i) => `  ${i}. [${c.strength}] "${c.text}" at ${c.location}`).join("\n"),
+    "",
+    `Diff:`,
+    "```",
+    truncated,
+    "```",
+    "",
+    `For each numbered claim above, emit a claim_verdicts entry. For gaps the`,
+    `author DIDN'T claim but that look like placeholder code, emit unflagged_gaps.`,
+    `Strict JSON only, matching the shape described. No prose outside JSON.`,
+  ].join("\n");
+
+  let resp: Response;
+  try {
+    resp = await fetch(`${GATEWAY}/v1/chat`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({
+        provider: "ollama_cloud",
+        model: MODEL,
+        messages: [
+          { role: "system", content: systemMsg },
+          { role: "user", content: userMsg },
+        ],
+        max_tokens: 3000,
+        temperature: 0.2,
+        think: true, // T3 overseer should reason — JSON shape is still required
+      }),
+      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
+    });
+  } catch (e) {
+    // Cloud unreachable → soft-fail. Don't block a PR because the
+    // reviewer model is down. Static + dynamic + kb still run.
+    return [{
+      check: "inference",
+      severity: "info",
+      summary: "cloud inference unreachable — skipped",
+      evidence: [`fetch failed: ${(e as Error).message.slice(0, 180)}`],
+    }];
+  }
+
+  if (!resp.ok) {
+    return [{
+      check: "inference",
+      severity: "info",
+      summary: `cloud inference returned ${resp.status} — skipped`,
+      evidence: [`body: ${(await resp.text()).slice(0, 200)}`],
+    }];
+  }
+
+  const body: any = await resp.json();
+  const content: string = body?.choices?.[0]?.message?.content ?? "";
+  const usage = body?.usage ?? {};
+
+  const parsed = extractJson(content);
+  if (!parsed) {
+    return [{
+      check: "inference",
+      severity: "info",
+      summary: "cloud returned unparseable output — skipped",
+      evidence: [
+        `head: ${content.slice(0, 200)}`,
+        `tokens: ${usage.total_tokens ?? "?"}`,
+      ],
+    }];
+  }
+
+  const findings: Finding[] = [];
+
+  // One summary info finding so the verdict layer knows the check ran.
+  findings.push({
+    check: "inference",
+    severity: "info",
+    summary: `cloud review completed (model=${MODEL}, tokens=${usage.total_tokens ?? "?"})`,
+    evidence: [
+      `claim_verdicts: ${parsed.claim_verdicts?.length ?? 0}, unflagged_gaps: ${parsed.unflagged_gaps?.length ?? 0}`,
+    ],
+  });
+
+  for (const v of parsed.claim_verdicts ?? []) {
+    if (v?.backed === false) {
+      const idx = typeof v.claim_idx === "number" ? v.claim_idx : -1;
+      const claim = claims[idx];
+      if (!claim) continue;
+      // Strong+unbacked = BLOCK. That's the whole point of the auditor.
+      const sev: Finding["severity"] = claim.strength === "strong" ? "block"
+        : claim.strength === "moderate" ? "warn"
+        : "info";
+      findings.push({
+        check: "inference",
+        severity: sev,
+        claim_text: claim.text,
+        summary: `cloud: claim not backed — "${claim.text.slice(0, 100)}"`,
+        evidence: [
+          `at ${claim.location}`,
+          `cloud reason: ${String(v.evidence ?? "no reason given").slice(0, 200)}`,
+        ],
+      });
+    }
+  }
+
+  for (const g of parsed.unflagged_gaps ?? []) {
+    findings.push({
+      check: "inference",
+      severity: "warn",
+      summary: `cloud-flagged gap not in any claim: ${String(g?.summary ?? "?").slice(0, 120)}`,
+      evidence: [`location: ${String(g?.location ?? "?").slice(0, 140)}`],
+    });
+  }
+
+  return findings;
+}
+
+// Lift the first balanced JSON object out of the response. Tolerates
+// leading prose, code fences, and model reasoning preamble when the
+// cloud model ignored "strict JSON only."
+function extractJson(text: string): any | null {
+  const cleaned = text.replace(/^```(?:json)?\s*/im, "").replace(/```\s*$/im, "");
+  let depth = 0;
+  let start = -1;
+  for (let i = 0; i < cleaned.length; i++) {
+    const c = cleaned[i];
+    if (c === "{") {
+      if (depth === 0) start = i;
+      depth++;
+    } else if (c === "}") {
+      depth--;
+      if (depth === 0 && start >= 0) {
+        try { return JSON.parse(cleaned.slice(start, i + 1)); } catch { start = -1; }
+      }
+    }
+  }
+  return null;
+}

auditor/checks/kb_query.ts

@@ -0,0 +1,183 @@
+// Local-KB check — reads data/_kb/ + data/_observer/ + data/_bot/
+// for prior evidence bearing on this PR's claims. Cheap, offline,
+// no model calls. The point: if a claim like "Phase X shipped" has
+// a historical record of failing on the same signature before, the
+// auditor surfaces that pattern before the cloud check has to
+// infer it.
+//
+// What this check reads (all file-backed, append-only or periodic):
+//   data/_kb/outcomes.jsonl         — per-scenario outcomes (kb.ts)
+//   data/_kb/error_corrections.jsonl — fail→succeed deltas on same sig
+//   data/_observer/ops.jsonl         — observer ring → disk stream
+//   data/_bot/cycles/*.json          — bot cycle results
+//
+// Each JSONL line / per-cycle file is small; this check reads tails
+// only (last N lines or last M files) to stay cheap on large corpora.
+
+import { readFile, readdir, stat } from "node:fs/promises";
+import { join } from "node:path";
+import type { Claim, Finding } from "../types.ts";
+
+const KB_DIR = "/home/profit/lakehouse/data/_kb";
+const OBSERVER_OPS = "/home/profit/lakehouse/data/_observer/ops.jsonl";
+const BOT_CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
+const TAIL_LINES = 500;
+const MAX_BOT_CYCLE_FILES = 30;
+
+export async function runKbCheck(claims: Claim[]): Promise<Finding[]> {
+  const findings: Finding[] = [];
+
+  // 1. Recent scenario outcomes: are strong-claim-style phrases showing
+  //    up alongside failed events? That's "we claimed it worked" +
+  //    "it didn't" in the KB.
+  const scenarioFindings = await checkScenarioOutcomes(claims);
+  findings.push(...scenarioFindings);
+
+  // 2. Error corrections: any of the claims text overlap a
+  //    recently-observed fail→succeed pair? If yes, add context.
+  const correctionFindings = await checkErrorCorrections(claims);
+  findings.push(...correctionFindings);
+
+  // 3. Bot cycles: any prior bot cycle ended in tests_failed or
+  //    apply_failed on a file this PR is also touching?
+  const botFindings = await checkBotCycles();
+  findings.push(...botFindings);
+
+  // 4. Observer: count recent error events. High volume = shared
+  //    infra problem, worth flagging (context for other findings).
+  const obsFindings = await checkObserverStream();
+  findings.push(...obsFindings);
+
+  return findings;
+}
+
+async function tailJsonl<T = any>(path: string, n: number): Promise<T[]> {
+  try {
+    const raw = await readFile(path, "utf8");
+    const lines = raw.split("\n").filter(l => l.length > 0);
+    const slice = lines.slice(-n);
+    const out: T[] = [];
+    for (const line of slice) {
+      try { out.push(JSON.parse(line)); } catch { /* skip malformed */ }
+    }
+    return out;
+  } catch {
+    return [];
+  }
+}
+
+async function checkScenarioOutcomes(_claims: Claim[]): Promise<Finding[]> {
+  const outcomes = await tailJsonl<any>(join(KB_DIR, "outcomes.jsonl"), TAIL_LINES);
+  if (outcomes.length === 0) return [];
+  const totalEvents = outcomes.reduce((s, o) => s + (o.total_events ?? 0), 0);
+  const okEvents = outcomes.reduce((s, o) => s + (o.ok_events ?? 0), 0);
+  const failRate = totalEvents > 0 ? 1 - okEvents / totalEvents : 0;
+
+  if (totalEvents === 0) {
+    return [{
+      check: "kb_query",
+      severity: "info",
+      summary: `KB: no scenario outcomes on file — learning loop is empty`,
+      evidence: [`data/_kb/outcomes.jsonl has ${outcomes.length} entries with 0 total events`],
+    }];
+  }
+
+  const recent = outcomes.slice(-10);
+  const recentFailSigs: string[] = recent
+    .filter(o => (o.ok_events ?? 0) < (o.total_events ?? 0))
+    .map(o => o.sig_hash)
+    .filter(s => typeof s === "string");
+
+  const findings: Finding[] = [{
+    check: "kb_query",
+    severity: failRate > 0.3 ? "warn" : "info",
+    summary: `KB: ${outcomes.length} recent scenario runs, ${okEvents}/${totalEvents} events ok (fail rate ${(failRate * 100).toFixed(1)}%)`,
+    evidence: [
+      `most recent: ${recent[recent.length - 1]?.run_id ?? "?"}`,
+      `recent failing sigs: ${recentFailSigs.length > 0 ? recentFailSigs.slice(-3).join(", ") : "none"}`,
+    ],
+  }];
+  return findings;
+}
+
+async function checkErrorCorrections(_claims: Claim[]): Promise<Finding[]> {
+  const corrections = await tailJsonl<any>(join(KB_DIR, "error_corrections.jsonl"), TAIL_LINES);
+  if (corrections.length === 0) return [];
+  return [{
+    check: "kb_query",
+    severity: "info",
+    summary: `KB: ${corrections.length} error corrections on file (fail→succeed pairs)`,
+    evidence: [
+      corrections.length > 0
+        ? `most recent: ${String(corrections[corrections.length - 1]?.sig_hash ?? "?").slice(0, 24)}`
+        : "none",
+    ],
+  }];
+}
+
+async function checkBotCycles(): Promise<Finding[]> {
+  let entries: string[] = [];
+  try { entries = await readdir(BOT_CYCLES_DIR); }
+  catch { return []; }
+
+  const jsonFiles = entries.filter(e => e.endsWith(".json"));
+  if (jsonFiles.length === 0) return [];
+
+  // Sort by mtime desc, take most recent N
+  const withStat = await Promise.all(
+    jsonFiles.map(async name => {
+      try { return { name, mtime: (await stat(join(BOT_CYCLES_DIR, name))).mtimeMs }; }
+      catch { return { name, mtime: 0 }; }
+    }),
+  );
+  const recent = withStat.sort((a, b) => b.mtime - a.mtime).slice(0, MAX_BOT_CYCLE_FILES);
+
+  const outcomes: Record<string, number> = {};
+  for (const { name } of recent) {
+    try {
+      const r = JSON.parse(await readFile(join(BOT_CYCLES_DIR, name), "utf8"));
+      const o = String(r.outcome ?? "unknown");
+      outcomes[o] = (outcomes[o] ?? 0) + 1;
+    } catch { /* skip */ }
+  }
+
+  const summary = Object.entries(outcomes)
+    .sort((a, b) => b[1] - a[1])
+    .map(([k, v]) => `${k}=${v}`)
+    .join(", ");
+
+  const failCount = (outcomes["tests_failed"] ?? 0) + (outcomes["apply_failed"] ?? 0) + (outcomes["model_failed"] ?? 0);
+  return [{
+    check: "kb_query",
+    severity: failCount > recent.length / 2 ? "warn" : "info",
+    summary: `KB: bot recorded ${recent.length} recent cycles — ${summary || "no outcomes parsed"}`,
+    evidence: [
+      `dir: ${BOT_CYCLES_DIR}`,
+      `fail-class total: ${failCount} / ${recent.length}`,
+    ],
+  }];
+}
+
+async function checkObserverStream(): Promise<Finding[]> {
+  const ops = await tailJsonl<any>(OBSERVER_OPS, TAIL_LINES);
+  if (ops.length === 0) return [];
+  const failures = ops.filter(o => o.ok === false).length;
+  return [{
+    check: "kb_query",
+    severity: "info",
+    summary: `KB: observer stream ${ops.length} recent ops, ${failures} failures`,
+    evidence: [
+      `source: ${OBSERVER_OPS}`,
+      `by source: ${observerBySource(ops)}`,
+    ],
+  }];
+}
+
+function observerBySource(ops: any[]): string {
+  const c: Record<string, number> = {};
+  for (const o of ops) {
+    const s = String(o.source ?? "unknown");
+    c[s] = (c[s] ?? 0) + 1;
+  }
+  return Object.entries(c).sort((a, b) => b[1] - a[1]).map(([k, v]) => `${k}=${v}`).join(", ") || "empty";
+}

auditor/checks/static.ts

@@ -0,0 +1,159 @@
+// Static diff check — grep-style, no AST, no LLM. Looks for patterns
+// that are high-signal evidence of placeholder code.
+//
+// Findings are severity-graded:
+//   block — explicit non-impl markers (unimplemented!, todo!,
+//           panic!("not implemented"), throw new Error("not implemented"))
+//   warn  — TODO / FIXME / XXX / HACK comments on added lines,
+//           new struct fields with no read-site anywhere in the diff,
+//           suspiciously-empty function bodies ({ Ok(()) } / {} when
+//           the commit message claims the fn "implements" something)
+//   info  — hardcoded "test" / "dummy" / "placeholder" strings in
+//           added lines (could be real, just flag for inspection)
+//
+// Consumes: raw unified diff text from Gitea.
+
+import type { Finding } from "../types.ts";
+
+// Rust + TypeScript patterns that almost always indicate "this is
+// not actually implemented yet."
+const BLOCK_PATTERNS: Array<{ re: RegExp; why: string }> = [
+  { re: /\bunimplemented!\s*\(/, why: "unimplemented!() macro call" },
+  { re: /\btodo!\s*\(/, why: "todo!() macro call" },
+  { re: /panic!\s*\(\s*"(?:not implemented|TODO|not yet|unimpl)/i, why: "panic! with not-implemented message" },
+  { re: /throw\s+new\s+Error\s*\(\s*['"](?:not implemented|TODO|unimpl)/i, why: "throw Error 'not implemented'" },
+];
+
+const WARN_COMMENT_PATTERNS: Array<{ re: RegExp; why: string }> = [
+  { re: /^\+.*\/\/\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
+  { re: /^\+.*#\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
+];
+
+const INFO_HARDCODED_PATTERNS: Array<{ re: RegExp; why: string }> = [
+  { re: /"(?:placeholder|dummy|foobar|xxx|replaceme|changeme)"/i, why: "suspicious hardcoded string" },
+];
+
+export function runStaticCheck(diff: string): Finding[] {
+  const findings: Finding[] = [];
+
+  // Per-file walk: only look at ADDED lines (prefix '+' but not '+++'
+  // which is the diff header).
+  const perFile = splitDiffByFile(diff);
+
+  for (const [path, lines] of perFile) {
+    // Skip diff bookkeeping + pure-delete files
+    if (!lines.some(l => l.startsWith("+") && !l.startsWith("+++"))) continue;
+
+    // The auditor's own check files literally contain the BLOCK
+    // patterns as regex definitions (BLOCK_PATTERNS in this file,
+    // prompt examples in inference.ts). Skipping BLOCK scan on these
+    // specific paths prevents the checker from self-flagging its own
+    // string literals. WARN/INFO patterns still run — those genuinely
+    // could indicate problems in the checker's own code (TODO
+    // comments don't self-define).
+    const isAuditorCheckerFile = path.startsWith("auditor/checks/") ||
+                                 path.startsWith("auditor/fixtures/");
+
+    for (let idx = 0; idx < lines.length; idx++) {
+      const line = lines[idx];
+      if (!line.startsWith("+") || line.startsWith("+++")) continue;
+      const added = line.slice(1);
+
+      if (!isAuditorCheckerFile) {
+        for (const { re, why } of BLOCK_PATTERNS) {
+          if (re.test(added)) {
+            findings.push({
+              check: "static",
+              severity: "block",
+              summary: `${why} in ${path}`,
+              evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
+            });
+          }
+        }
+      }
+      for (const { re, why } of WARN_COMMENT_PATTERNS) {
+        if (re.test(line)) {
+          findings.push({
+            check: "static",
+            severity: "warn",
+            summary: `${why} in ${path}`,
+            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
+          });
+        }
+      }
+      for (const { re, why } of INFO_HARDCODED_PATTERNS) {
+        if (re.test(added)) {
+          findings.push({
+            check: "static",
+            severity: "info",
+            summary: `${why} in ${path}`,
+            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
+          });
+        }
+      }
+    }
+
+    // "Field added but never read" heuristic — catches exactly the
+    // Phase 45 DocRef placeholder pattern. Limited to the diff itself:
+    // we're not doing a full-codebase grep here (too noisy; callers
+    // elsewhere might exist). The point is: if NEITHER this diff nor
+    // any other line in the diff reads the field, the PR is shipping
+    // state without a consumer.
+    const addedLines = lines.filter(l => l.startsWith("+") && !l.startsWith("+++"))
+      .map(l => l.slice(1));
+    const newFields = extractNewFields(addedLines);
+    for (const field of newFields) {
+      const readPattern = new RegExp(`[\\.:]\\s*${escape(field)}\\b|\\b${escape(field)}\\s*:`);
+      // The definition line itself matches readPattern — filter it out
+      // by requiring at least TWO lines in the diff mention the field
+      // (one defines, one reads).
+      const hits = addedLines.filter(l => readPattern.test(l));
+      if (hits.length < 2) {
+        findings.push({
+          check: "static",
+          severity: "warn",
+          summary: `field '${field}' added in ${path} but no read-site in the diff — could be placeholder state without a consumer`,
+          evidence: [`${path}: added '${field}' with no reader; rest of diff has ${hits.length - 1} mentions`],
+        });
+      }
+    }
+  }
+
+  return findings;
+}
+
+function splitDiffByFile(diff: string): Map<string, string[]> {
+  const out = new Map<string, string[]>();
+  let current: string | null = null;
+  let buf: string[] = [];
+  for (const line of diff.split(/\r?\n/)) {
+    const m = line.match(/^diff --git a\/(\S+) b\/(\S+)/);
+    if (m) {
+      if (current) out.set(current, buf);
+      current = m[2];
+      buf = [];
+      continue;
+    }
+    buf.push(line);
+  }
+  if (current) out.set(current, buf);
+  return out;
+}
+
+// Extract new `pub name: Type,` fields from added lines. Rust syntax.
+// Narrowly-scoped: only matches at the start of a trimmed line,
+// requires `pub ` prefix, ignores `pub fn` / `pub struct` / etc.
+function extractNewFields(addedLines: string[]): string[] {
+  const fields = new Set<string>();
+  for (const line of addedLines) {
+    const t = line.trim();
+    // pub NAME: Type,
+    const m = t.match(/^pub\s+(?!fn\b|struct\b|enum\b|mod\b|use\b|trait\b|impl\b|const\b|static\b|type\b)(\w+)\s*:/);
+    if (m) fields.add(m[1]);
+  }
+  return Array.from(fields);
+}
+
+function escape(s: string): string {
+  return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+}

auditor/claim_parser.ts

@@ -0,0 +1,119 @@
+// Claim parser — reads commit messages + PR body, extracts ship-claims.
+//
+// A "ship-claim" is any phrase that asserts functionality is working,
+// tested, complete, or landed. These are the assertions the downstream
+// checks (static/dynamic/inference/kb) try to falsify.
+//
+// Heuristic approach (regex + strength grading) — intentionally NOT
+// using an LLM here. Reason: the inference check already asks a cloud
+// model "does this match the claim?". The parser's job is to surface
+// the claim substrates, not judge them. Over-engineering the parser
+// risks false-negatives when the cloud model was going to catch it
+// anyway.
+
+import type { Claim, PrSnapshot } from "./types.ts";
+
+// Strong claims: explicit end-to-end + verification vocabulary
+const STRONG_PATTERNS: RegExp[] = [
+  /\bverified\s+(end[- ]to[- ]end|live|in\s+production|against)\b/i,
+  /\btested\s+(live|end[- ]to[- ]end|against|with)\b/i,
+  /\bworks\s+(end[- ]to[- ]end|live|in\s+production)\b/i,
+  /\bproduction[- ]ready\b/i,
+  /\bfully\s+(functional|wired|working)\b/i,
+  /\bphase\s+\d+(\.\d+)?\s+(shipped|complete|done|landed)\b/i,
+  /\bground\s+truth\b/i,
+  /\bproven\b/i,
+];
+
+// Moderate claims: asserted completion or pass but without the strong
+// verification qualifier.
+const MODERATE_PATTERNS: RegExp[] = [
+  /\bshipped\b/i,
+  /\blanded\b/i,
+  /\bgreen\b/i,
+  /\b(tests?\s+)?pass(ing|ed)\b/i,
+  /\bcomplet(e|ed)\b/i,
+  /\bdone\b/i,
+  /\bwired\b/i,
+  /\bfixed\b/i,
+  /\bworks\b/i,
+];
+
+// Weak claims: aspirational or hedged. Usually low-risk but recorded
+// for completeness.
+const WEAK_PATTERNS: RegExp[] = [
+  /\bshould\s+work\b/i,
+  /\bexpected\s+to\b/i,
+  /\bintended\s+to\b/i,
+  /\bwill\s+(work|handle|support)\b/i,
+  /\bprobably\b/i,
+];
+
+export interface ParsedClaims {
+  claims: Claim[];
+  commits_scanned: number;
+}
+
+export function parseClaims(pr: PrSnapshot): ParsedClaims {
+  const claims: Claim[] = [];
+
+  // PR body — every matching line becomes a claim at location "pr_body:N"
+  if (pr.body) {
+    scanText(pr.body, "pr_body", pr.head_sha, claims);
+  }
+
+  // Each commit message gets its own scan.
+  for (const c of pr.commits) {
+    if (!c.message) continue;
+    scanText(c.message, `commit:${c.sha.slice(0, 8)}`, c.sha, claims);
+  }
+
+  return { claims, commits_scanned: pr.commits.length };
+}
+
+function scanText(text: string, location_prefix: string, commit_sha: string, out: Claim[]): void {
+  const lines = text.split(/\r?\n/);
+  for (let i = 0; i < lines.length; i++) {
+    const line = lines[i];
+    if (line.length < 3) continue;
+
+    // Strong patterns first — if a line matches strong, it's strong,
+    // don't double-count as moderate.
+    const strong = firstMatch(line, STRONG_PATTERNS);
+    if (strong) {
+      out.push({
+        text: line.trim().slice(0, 200),
+        commit_sha,
+        location: `${location_prefix}:${i + 1}`,
+        strength: "strong",
+      });
+      continue;
+    }
+    const moderate = firstMatch(line, MODERATE_PATTERNS);
+    if (moderate) {
+      out.push({
+        text: line.trim().slice(0, 200),
+        commit_sha,
+        location: `${location_prefix}:${i + 1}`,
+        strength: "moderate",
+      });
+      continue;
+    }
+    const weak = firstMatch(line, WEAK_PATTERNS);
+    if (weak) {
+      out.push({
+        text: line.trim().slice(0, 200),
+        commit_sha,
+        location: `${location_prefix}:${i + 1}`,
+        strength: "weak",
+      });
+    }
+  }
+}
+
+function firstMatch(text: string, patterns: RegExp[]): RegExp | null {
+  for (const p of patterns) {
+    if (p.test(text)) return p;
+  }
+  return null;
+}

auditor/fixtures/cli.ts

@@ -0,0 +1,49 @@
+// Standalone runner for auditor fixtures. Invoke:
+//
+//   bun run auditor/fixtures/cli.ts hybrid_38_40_45
+//
+// Prints human-readable per-layer breakdown + JSON result so the
+// output can be captured by the dynamic check without re-running.
+
+import { runHybridFixture, type FixtureResult } from "./hybrid_38_40_45.ts";
+
+const fixtureArg = process.argv[2] ?? "hybrid_38_40_45";
+
+let result: FixtureResult;
+switch (fixtureArg) {
+  case "hybrid_38_40_45":
+    result = await runHybridFixture();
+    break;
+  default:
+    console.error(`unknown fixture: ${fixtureArg}`);
+    process.exit(2);
+}
+
+// Human-readable summary
+console.error(""); // blank line to stderr so stdout JSON stays clean
+console.error(`─── Fixture: ${result.fixture} ───`);
+console.error(`  overall: ${result.overall.toUpperCase()}`);
+console.error(`  shipped:     [${result.shipped_phases.join(", ") || "—"}]`);
+console.error(`  placeholder: [${result.placeholder_phases.join(", ") || "—"}]`);
+console.error("");
+for (const l of result.layers) {
+  const mark = l.ok ? "✓" : "✗";
+  const phaseStr = `Phase ${l.phase}`.padEnd(11);
+  console.error(`  ${mark} ${phaseStr} ${l.layer.padEnd(30)} ${String(l.latency_ms).padStart(6)}ms`);
+  if (l.ok) {
+    console.error(`      ${l.evidence.slice(0, 200)}`);
+  } else {
+    console.error(`      ERROR: ${l.error?.slice(0, 240) ?? "unknown"}`);
+  }
+}
+console.error("");
+for (const n of result.notes) console.error(`  note: ${n}`);
+console.error("");
+
+// Machine-readable output on stdout.
+console.log(JSON.stringify(result, null, 2));
+
+// Exit code reflects overall: 0 pass, 2 partial, 1 fail.
+// Dynamic check reads this AND the JSON; partial-pass is treated
+// as informative (some layers shipped), not blocking on its own.
+process.exit(result.overall === "pass" ? 0 : result.overall === "partial_pass" ? 2 : 1);

auditor/fixtures/hybrid_38_40_45.ts

@@ -0,0 +1,335 @@
+// The never-run hybrid test fixture. Exercises the full stack end-to-end
+// across Phase 38 (/v1/chat), Phase 40 (Langfuse tracing), Phase 45
+// slice 1 (DocRef on playbook), and Phase 45 slice 2 (context7 bridge
+// drift detection).
+//
+// Returns HONEST per-layer results. A layer that's unimplemented or
+// broken returns ok=false with a real error; overall verdict flags
+// which phases are genuinely shipped vs still placeholder.
+//
+// Deterministic: always runs the same payload so a run's output is
+// comparable across audits. Uses TEST-prefixed names so playbook
+// state doesn't get polluted with production-looking fixture data.
+//
+// Preconditions:
+//   - gateway up on :3100
+//   - Python sidecar up on :3200
+//   - Ollama local + Ollama Cloud key loaded
+//   - Langfuse up on :3001
+//   - context7 bridge up on :3900 (manual-start per mcp-server/README)
+//     If bridge isn't running, that layer reports ok=false with a
+//     clear error — the fixture doesn't try to auto-start it.
+
+import { readFile } from "node:fs/promises";
+
+const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
+const LANGFUSE = process.env.LANGFUSE_URL ?? "http://localhost:3001";
+const BRIDGE = process.env.CONTEXT7_BRIDGE_URL ?? "http://localhost:3900";
+const FIXTURE_ID = "auditor:hybrid_38_40_45:v1";
+const TEST_WORKER_NAME = "__auditor_test_worker__";
+const STALE_HASH = "stale-hash-for-drift-proof";
+// Unique-per-run identifiers so each fixture invocation hits the ADD
+// path in upsert_entry (not UPDATE/NOOP on a leftover from a prior
+// run). Catches state pollution + avoids interaction with task #12
+// (UPDATE branch silently drops doc_refs).
+const RUN_NONCE = Date.now().toString(36);
+const TEST_WORKER_NAME_VERSIONED = `__auditor_test_worker_${RUN_NONCE}__`;
+const TEST_OPERATION_VERSIONED = `TEST: fill: __auditor_${RUN_NONCE}__ x1 in TestCity, TC`;
+
+export interface LayerResult {
+  layer: string;
+  phase: string;
+  ok: boolean;
+  latency_ms: number;
+  evidence: string;
+  error?: string;
+  // Layer-specific numbers that downstream checks can aggregate.
+  metrics?: Record<string, number | string | boolean>;
+}
+
+export interface FixtureResult {
+  fixture: string;
+  ran_at: string;
+  overall: "pass" | "partial_pass" | "fail";
+  layers: LayerResult[];
+  real_numbers: Record<string, number>;
+  shipped_phases: string[];
+  placeholder_phases: string[];
+  notes: string[];
+}
+
+// Wraps a layer's async body with timing + structured error capture.
+// On throw: returns ok=false with the error string as evidence, so
+// downstream verdict code treats it as a layer failure rather than
+// a fixture crash.
+async function measureLayer(
+  layer: string,
+  phase: string,
+  body: () => Promise<{ evidence: string; metrics?: Record<string, number | string | boolean> }>,
+): Promise<LayerResult> {
+  const t0 = Date.now();
+  try {
+    const { evidence, metrics } = await body();
+    return {
+      layer, phase, ok: true,
+      latency_ms: Date.now() - t0,
+      evidence, metrics,
+    };
+  } catch (e) {
+    return {
+      layer, phase, ok: false,
+      latency_ms: Date.now() - t0,
+      evidence: `FAILED: ${(e as Error).message.slice(0, 200)}`,
+      error: (e as Error).message,
+    };
+  }
+}
+
+async function langfuseAuth(): Promise<{ pk: string; sk: string } | null> {
+  // Default credentials match mcp-server/tracing.ts. Same sources the
+  // Rust side uses in gateway/src/v1/langfuse_trace.rs.
+  let pk = process.env.LANGFUSE_PUBLIC_KEY ?? "pk-lf-staffing";
+  let sk = process.env.LANGFUSE_SECRET_KEY ?? "sk-lf-staffing-secret";
+  if (!pk || !sk) return null;
+  return { pk, sk };
+}
+
+export async function runHybridFixture(): Promise<FixtureResult> {
+  const result: FixtureResult = {
+    fixture: FIXTURE_ID,
+    ran_at: new Date().toISOString(),
+    overall: "fail",
+    layers: [],
+    real_numbers: {},
+    shipped_phases: [],
+    placeholder_phases: [],
+    notes: [],
+  };
+
+  // ========================================================================
+  // Layer 1 — Phase 38: POST /v1/chat returns valid OpenAI shape
+  // ========================================================================
+  // Captured HERE, immediately before the chat layer runs, so layer 2's
+  // Langfuse-trace filter uses the actual moment the chat call was
+  // attempted — not the fixture start time. Earlier draft had a
+  // meaningless ternary returning result.ran_at on both branches; the
+  // LLM-Team codereview (2026-04-22) caught this and flagged it as a
+  // false-negative window on traces created between fixture-start and
+  // chat-fetch.
+  const chat_request_sent_ms = Date.now();
+
+  const l1 = await measureLayer("phase38_chat", "38", async () => {
+    const r = await fetch(`${GATEWAY}/v1/chat`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({
+        model: "qwen3.5:latest",
+        messages: [
+          { role: "system", content: "Respond in 8 words or fewer." },
+          { role: "user", content: "Name one Docker command." },
+        ],
+        max_tokens: 60,
+        temperature: 0.2,
+      }),
+      signal: AbortSignal.timeout(60000),
+    });
+    if (!r.ok) throw new Error(`gateway ${r.status}: ${await r.text()}`);
+    const j: any = await r.json();
+    const content = j.choices?.[0]?.message?.content;
+    if (typeof content !== "string" || content.length === 0) {
+      throw new Error(`empty content — think-false default may not be wired: ${JSON.stringify(j).slice(0, 200)}`);
+    }
+    return {
+      evidence: `content="${content.slice(0, 60)}" tokens=${j.usage?.total_tokens}`,
+      metrics: {
+        prompt_tokens: j.usage?.prompt_tokens ?? 0,
+        completion_tokens: j.usage?.completion_tokens ?? 0,
+        total_tokens: j.usage?.total_tokens ?? 0,
+        content_nonempty: true,
+      },
+    };
+  });
+  result.layers.push(l1);
+
+  // ========================================================================
+  // Layer 2 — Phase 40: Langfuse trace lands within 3s
+  // ========================================================================
+  // Fire-and-forget tracing means we need a brief sleep before query.
+  await new Promise(res => setTimeout(res, 2500));
+  const l2 = await measureLayer("phase40_langfuse_trace", "40", async () => {
+    const auth = await langfuseAuth();
+    if (!auth) throw new Error("Langfuse credentials not in env");
+    const r = await fetch(`${LANGFUSE}/api/public/traces?limit=5&name=v1.chat:ollama`, {
+      headers: { Authorization: `Basic ${btoa(`${auth.pk}:${auth.sk}`)}` },
+      signal: AbortSignal.timeout(10000),
+    });
+    if (!r.ok) throw new Error(`langfuse ${r.status}: ${await r.text()}`);
+    const j: any = await r.json();
+    const items = Array.isArray(j.data) ? j.data : [];
+    // Filter on the chat-request timestamp captured above. A Langfuse
+    // trace must be newer than the moment we fired /v1/chat to plausibly
+    // belong to our request. Using fixture start time (result.ran_at)
+    // was wrong and could false-negative on slow fixtures.
+    const recent = items.filter((t: any) => Date.parse(t.timestamp) >= chat_request_sent_ms);
+    if (recent.length === 0) {
+      throw new Error(`no v1.chat:ollama trace since ${new Date(chat_request_sent_ms).toISOString()} (${items.length} older traces visible, Langfuse reachable — tracing is not firing)`);
+    }
+    const trace = recent[0];
+    return {
+      evidence: `trace ${trace.id.slice(0, 8)} name=${trace.name} age=${Date.now() - Date.parse(trace.timestamp)}ms`,
+      metrics: {
+        trace_age_ms: Date.now() - Date.parse(trace.timestamp),
+        trace_latency_reported: Number(trace.latency ?? 0),
+        recent_trace_count: recent.length,
+      },
+    };
+  });
+  result.layers.push(l2);
+
+  // ========================================================================
+  // Layer 3 — Phase 45 slice 1: /seed accepts doc_refs and persists them
+  // ========================================================================
+  // Seed a playbook with doc_refs referencing Docker at a stale hash.
+  // Ignore the cleanup concern for v1 — this fixture pollutes the
+  // in-memory playbook state; operator can retire test entries.
+  const l3 = await measureLayer("phase45_seed_with_doc_refs", "45.1", async () => {
+    const r = await fetch(`${GATEWAY}/vectors/playbook_memory/seed`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify({
+        operation: TEST_OPERATION_VERSIONED,
+        approach: "auditor hybrid fixture run",
+        context: "doc_refs integration test — retire after audit",
+        endorsed_names: [TEST_WORKER_NAME_VERSIONED],
+        append: true,
+        doc_refs: [
+          {
+            tool: "docker",
+            version_seen: "24.0.7",
+            snippet_hash: STALE_HASH,
+            source_url: "https://context7.com/docker/docs",
+            seen_at: new Date().toISOString(),
+          },
+        ],
+      }),
+      signal: AbortSignal.timeout(30000),
+    });
+    if (!r.ok) throw new Error(`seed ${r.status}: ${await r.text()}`);
+    const j: any = await r.json();
+    // Verify doc_refs actually persisted — read state.json directly.
+    // That's the canonical source of truth; the seed response may or
+    // may not echo doc_refs back.
+    const state_raw = await readFile("/home/profit/lakehouse/data/_playbook_memory/state.json", "utf8");
+    const state = JSON.parse(state_raw);
+    const entries = state.entries ?? [];
+    const ours = entries.find((e: any) => (e.endorsed_names ?? []).includes(TEST_WORKER_NAME_VERSIONED));
+    if (!ours) throw new Error(`seeded entry not found in state.json (worker ${TEST_WORKER_NAME_VERSIONED} not present)`);
+    const docRefs = ours.doc_refs ?? [];
+    if (docRefs.length === 0) {
+      throw new Error("entry saved but doc_refs field is empty — the field accepted by the API isn't being persisted");
+    }
+    return {
+      evidence: `playbook ${ours.playbook_id.slice(0, 16)} persisted with doc_refs.length=${docRefs.length}; first tool=${docRefs[0]?.tool} hash=${docRefs[0]?.snippet_hash}`,
+      metrics: {
+        doc_refs_stored: docRefs.length,
+        entries_after: j.entries_after ?? -1,
+        playbook_id: ours.playbook_id,
+        mode: j.outcome?.mode ?? "unknown",
+      },
+    };
+  });
+  result.layers.push(l3);
+
+  // ========================================================================
+  // Layer 4 — Phase 45 slice 2: context7 bridge detects drift vs stale hash
+  // ========================================================================
+  const l4 = await measureLayer("phase45_bridge_diff", "45.2", async () => {
+    // Health check first — if bridge isn't running, fail with a clear
+    // message rather than mysterious connection refused.
+    try {
+      const h = await fetch(`${BRIDGE}/health`, { signal: AbortSignal.timeout(3000) });
+      if (!h.ok) throw new Error(`bridge /health ${h.status}`);
+    } catch (e) {
+      throw new Error(`context7 bridge at ${BRIDGE} is not reachable (bridge is manual-start; run 'bun run mcp-server/context7_bridge.ts'): ${(e as Error).message}`);
+    }
+    const r = await fetch(`${BRIDGE}/docs/docker/diff?since=${encodeURIComponent(STALE_HASH)}`, {
+      signal: AbortSignal.timeout(30000),
+    });
+    if (!r.ok) throw new Error(`bridge diff ${r.status}: ${await r.text()}`);
+    const j: any = await r.json();
+    if (j.drifted !== true) {
+      throw new Error(`expected drifted=true against ${STALE_HASH}; got ${JSON.stringify(j)}`);
+    }
+    return {
+      evidence: `bridge confirms drift: current=${j.current_snippet_hash} previous=${j.previous_snippet_hash} upstream_updated=${j.last_updated_upstream}`,
+      metrics: {
+        drifted: true,
+        current_hash_length: String(j.current_snippet_hash ?? "").length,
+        library_id: j.library_id,
+      },
+    };
+  });
+  result.layers.push(l4);
+
+  // ========================================================================
+  // Layer 5 — Phase 45 slice 3: /doc_drift/check/{id} endpoint
+  //                              (EXPECTED TO FAIL — endpoint unimplemented)
+  // ========================================================================
+  const l5 = await measureLayer("phase45_slice3_drift_flag", "45.3", async () => {
+    const pid = String(l3.metrics?.playbook_id ?? "");
+    if (!pid) throw new Error("layer 3 did not produce a playbook_id to check");
+    const r = await fetch(`${GATEWAY}/vectors/playbook_memory/doc_drift/check/${encodeURIComponent(pid)}`, {
+      method: "POST",
+      signal: AbortSignal.timeout(10000),
+    });
+    if (r.status === 404 || r.status === 405) {
+      // This is the HONEST signal: the endpoint doesn't exist yet.
+      // Fail this layer with a clear marker — not a silent pass.
+      throw new Error(`endpoint unimplemented (${r.status}) — Phase 45 slice 3 is still placeholder. doc_refs are persisted (layer 3) and the bridge can answer drift questions (layer 4), but nothing ties them together into a playbook flag yet.`);
+    }
+    if (!r.ok) throw new Error(`drift/check ${r.status}: ${await r.text()}`);
+    const j: any = await r.json();
+    // If the endpoint DID exist, assert the flag got set.
+    if (!j.flagged) {
+      throw new Error(`endpoint responded but flag not set: ${JSON.stringify(j).slice(0, 200)}`);
+    }
+    return {
+      evidence: `playbook flagged: ${JSON.stringify(j).slice(0, 160)}`,
+      metrics: { flagged: true },
+    };
+  });
+  result.layers.push(l5);
+
+  // ========================================================================
+  // Verdict assembly
+  // ========================================================================
+  for (const l of result.layers) {
+    if (l.ok) result.shipped_phases.push(l.phase);
+    else result.placeholder_phases.push(l.phase);
+    result.real_numbers[`${l.layer}_latency_ms`] = l.latency_ms;
+    if (l.metrics) {
+      for (const [k, v] of Object.entries(l.metrics)) {
+        if (typeof v === "number") result.real_numbers[`${l.layer}.${k}`] = v;
+      }
+    }
+  }
+
+  const anyFail = result.placeholder_phases.length > 0;
+  const allPass = result.layers.every(l => l.ok);
+  result.overall = allPass ? "pass" : (result.shipped_phases.length > 0 ? "partial_pass" : "fail");
+
+  if (!l5.ok) {
+    result.notes.push(
+      "Phase 45 slice 3 (doc_drift/check endpoint) is the expected-fail layer. " +
+      "Layer 5 failing with a 404/405 is the CORRECT honest signal — don't mask this " +
+      "to get a green result. When slice 3 ships, layer 5 flips to ok=true automatically."
+    );
+  }
+  if (anyFail) {
+    result.notes.push(
+      `${result.placeholder_phases.length} layer(s) failed. Check evidence per layer for specifics.`
+    );
+  }
+
+  return result;
+}

auditor/gitea.ts

@@ -0,0 +1,145 @@
+// Gitea API client. Minimal surface — only what the auditor needs:
+// list open PRs, get commits + files for a PR, fetch a diff, post a
+// commit status, post a review.
+//
+// Auth: reads PAT from ~/.git-credentials (set up by the credential
+// helper flow in 2026-04-22 session). Gitea's "token" auth scheme
+// matches what `git fetch` is already using.
+
+import { readFile } from "node:fs/promises";
+import type { PrSnapshot } from "./types.ts";
+
+const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
+const OWNER = "profit";
+const REPO = "lakehouse";
+const CRED_FILE = "/home/profit/.git-credentials";
+
+let cachedPat: string | null = null;
+
+async function getPat(): Promise<string> {
+  if (cachedPat) return cachedPat;
+  const raw = await readFile(CRED_FILE, "utf8");
+  for (const line of raw.split("\n")) {
+    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
+    if (m) { cachedPat = m[1]; return m[1]; }
+  }
+  throw new Error(`no Gitea PAT in ${CRED_FILE}`);
+}
+
+async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
+  const pat = await getPat();
+  const url = `${HOST}/api/v1${path}`;
+  const headers = new Headers(init.headers);
+  headers.set("Authorization", `token ${pat}`);
+  if (init.body && !headers.has("content-type")) {
+    headers.set("content-type", "application/json");
+  }
+  return fetch(url, { ...init, headers, signal: AbortSignal.timeout(20000) });
+}
+
+export async function listOpenPrs(): Promise<PrSnapshot[]> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls?state=open&page=1&limit=50`);
+  if (!r.ok) throw new Error(`listOpenPrs ${r.status}: ${await r.text()}`);
+  const rows = (await r.json()) as any[];
+  return Promise.all(rows.map(row => snapshotFromPr(row)));
+}
+
+export async function getPrSnapshot(num: number): Promise<PrSnapshot> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}`);
+  if (!r.ok) throw new Error(`getPr ${num} ${r.status}: ${await r.text()}`);
+  return snapshotFromPr((await r.json()) as any);
+}
+
+async function snapshotFromPr(row: any): Promise<PrSnapshot> {
+  const num = row.number;
+  const commitsResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/commits`);
+  const commits = commitsResp.ok ? ((await commitsResp.json()) as any[]) : [];
+  const filesResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/files`);
+  const files = filesResp.ok ? ((await filesResp.json()) as any[]) : [];
+  return {
+    number: num,
+    head_sha: row.head?.sha ?? "",
+    base_sha: row.base?.sha ?? "",
+    title: row.title ?? "",
+    body: row.body ?? "",
+    state: row.state === "open" ? "open" : (row.merged ? "merged" : "closed"),
+    author: row.user?.login ?? "",
+    commits: commits.map(c => ({
+      sha: (c.sha ?? "").slice(0, 12),
+      message: c.commit?.message ?? "",
+      author: c.commit?.author?.name ?? "",
+    })),
+    files: files.map(f => ({
+      path: f.filename ?? "",
+      additions: f.additions ?? 0,
+      deletions: f.deletions ?? 0,
+    })),
+  };
+}
+
+/// Returns the unified diff text of the PR. Used by static checks.
+export async function getPrDiff(num: number): Promise<string> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}.diff`);
+  if (!r.ok) throw new Error(`getDiff ${num} ${r.status}: ${await r.text()}`);
+  return await r.text();
+}
+
+/// Hard-block mechanism: post a failing commit status on the PR head
+/// SHA. Branch protection (if enabled on `main`) treats this as a
+/// required-check fail and prevents merge. The description is shown
+/// in the Gitea UI next to the red X.
+export async function postCommitStatus(args: {
+  sha: string;
+  state: "success" | "pending" | "failure" | "error";
+  context: string;
+  description: string;
+  target_url?: string;
+}): Promise<void> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/statuses/${args.sha}`, {
+    method: "POST",
+    body: JSON.stringify({
+      state: args.state,
+      context: args.context,
+      description: args.description.slice(0, 140),
+      target_url: args.target_url ?? "",
+    }),
+  });
+  if (!r.ok) throw new Error(`postCommitStatus ${r.status}: ${await r.text()}`);
+}
+
+/// Post a review comment. Gitea typically blocks self-review
+/// (author posting a review on their own PR). Prefer
+/// `postIssueComment` when running with the author's PAT.
+export async function postReview(args: {
+  pr_number: number;
+  commit_id: string;
+  body: string;
+  event: "APPROVE" | "REQUEST_CHANGES" | "COMMENT";
+}): Promise<void> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${args.pr_number}/reviews`, {
+    method: "POST",
+    body: JSON.stringify({
+      commit_id: args.commit_id,
+      body: args.body,
+      event: args.event,
+    }),
+  });
+  if (!r.ok) throw new Error(`postReview ${r.status}: ${await r.text()}`);
+}
+
+/// Plain issue comment. Works for the auditor's own PAT because
+/// Gitea allows authors to comment on their own PRs (just not
+/// review them). Auditor uses this for the reasoning body; the
+/// actual block signal is the commit status.
+export async function postIssueComment(args: {
+  pr_number: number;
+  body: string;
+}): Promise<{ id: number; html_url: string }> {
+  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/issues/${args.pr_number}/comments`, {
+    method: "POST",
+    body: JSON.stringify({ body: args.body }),
+  });
+  if (!r.ok) throw new Error(`postIssueComment ${r.status}: ${await r.text()}`);
+  const j = await r.json() as any;
+  return { id: j.id, html_url: j.html_url };
+}

auditor/index.ts

@@ -0,0 +1,147 @@
+// Auditor poller — the top-level entry. Polls Gitea for open PRs on
+// a fixed interval, dedupes by head SHA, runs audit + posts verdict
+// for each new (pr, sha) pair.
+//
+// Run manually:
+//   bun run auditor/index.ts
+//
+// Stop:
+//   touch auditor.paused        (skips next cycle)
+//   pkill -f auditor/index.ts   (kills in-flight)
+//
+// State:
+//   data/_auditor/state.json           — last-audited SHA per PR
+//   data/_auditor/verdicts/{id}.json   — per-run verdict records
+//
+// This entry runs forever. A systemd unit would wrap it once the
+// workflow is trusted (same pattern as mcp-server, observer).
+
+import { readFile, writeFile, mkdir, access } from "node:fs/promises";
+import { listOpenPrs } from "./gitea.ts";
+import { auditPr } from "./audit.ts";
+
+const POLL_INTERVAL_MS = 90_000; // 90s — enough budget for audit runs to complete
+const PAUSE_FILE = "/home/profit/lakehouse/auditor.paused";
+const STATE_FILE = "/home/profit/lakehouse/data/_auditor/state.json";
+
+interface State {
+  // Map: PR number → last-audited head SHA. Lets us dedupe audits
+  // across restarts (poller can crash/restart without re-auditing
+  // all open PRs from scratch).
+  last_audited: Record<string, string>;
+  started_at: string;
+  cycles_total: number;
+  cycles_skipped_paused: number;
+  audits_run: number;
+  last_cycle_at?: string;
+}
+
+async function fileExists(path: string): Promise<boolean> {
+  try { await access(path); return true; } catch { return false; }
+}
+
+async function loadState(): Promise<State> {
+  try {
+    const raw = await readFile(STATE_FILE, "utf8");
+    const s = JSON.parse(raw);
+    return {
+      last_audited: s.last_audited ?? {},
+      started_at: s.started_at ?? new Date().toISOString(),
+      cycles_total: s.cycles_total ?? 0,
+      cycles_skipped_paused: s.cycles_skipped_paused ?? 0,
+      audits_run: s.audits_run ?? 0,
+      last_cycle_at: s.last_cycle_at,
+    };
+  } catch {
+    return {
+      last_audited: {},
+      started_at: new Date().toISOString(),
+      cycles_total: 0,
+      cycles_skipped_paused: 0,
+      audits_run: 0,
+    };
+  }
+}
+
+async function saveState(s: State): Promise<void> {
+  await mkdir("/home/profit/lakehouse/data/_auditor", { recursive: true });
+  await writeFile(STATE_FILE, JSON.stringify(s, null, 2));
+}
+
+async function runCycle(state: State): Promise<State> {
+  state.cycles_total += 1;
+  state.last_cycle_at = new Date().toISOString();
+
+  if (await fileExists(PAUSE_FILE)) {
+    state.cycles_skipped_paused += 1;
+    console.log(`[auditor] cycle ${state.cycles_total}: paused (touch ${PAUSE_FILE} exists)`);
+    return state;
+  }
+
+  let prs;
+  try {
+    prs = await listOpenPrs();
+  } catch (e) {
+    console.error(`[auditor] listOpenPrs failed: ${(e as Error).message}`);
+    return state;
+  }
+
+  console.log(`[auditor] cycle ${state.cycles_total}: ${prs.length} open PR(s)`);
+
+  for (const pr of prs) {
+    const last = state.last_audited[String(pr.number)];
+    if (last === pr.head_sha) {
+      console.log(`[auditor]   skip PR #${pr.number} (SHA ${pr.head_sha.slice(0, 8)} already audited)`);
+      continue;
+    }
+    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)}`);
+    try {
+      // Skip dynamic by default: it mutates live playbook state and
+      // re-runs on every PR update would pollute quickly. Operator
+      // can run dynamic via `bun run auditor/fixtures/cli.ts` manually
+      // OR set LH_AUDITOR_RUN_DYNAMIC=1 to opt in.
+      const run_dynamic = process.env.LH_AUDITOR_RUN_DYNAMIC === "1";
+      const verdict = await auditPr(pr, {
+        skip_dynamic: !run_dynamic,
+        skip_inference: process.env.LH_AUDITOR_SKIP_INFERENCE === "1",
+      });
+      console.log(`[auditor]     verdict=${verdict.overall} findings=${verdict.metrics.findings_total} (block=${verdict.metrics.findings_block} warn=${verdict.metrics.findings_warn})`);
+      state.last_audited[String(pr.number)] = pr.head_sha;
+      state.audits_run += 1;
+    } catch (e) {
+      console.error(`[auditor]     audit failed: ${(e as Error).message}`);
+    }
+  }
+
+  return state;
+}
+
+async function main(): Promise<void> {
+  console.log(`[auditor] starting poller — interval ${POLL_INTERVAL_MS / 1000}s`);
+  console.log(`[auditor] pause file: ${PAUSE_FILE}`);
+  console.log(`[auditor] state file: ${STATE_FILE}`);
+
+  let state = await loadState();
+  console.log(`[auditor] loaded state: ${Object.keys(state.last_audited).length} PRs previously audited, ${state.cycles_total} cycles so far`);
+
+  // Single-shot mode for CLI testing: `bun run auditor/index.ts --once`
+  const once = process.argv.includes("--once");
+  if (once) {
+    state = await runCycle(state);
+    await saveState(state);
+    console.log(`[auditor] single-shot complete. total audits: ${state.audits_run}`);
+    return;
+  }
+
+  // Loop.
+  while (true) {
+    state = await runCycle(state);
+    await saveState(state);
+    await new Promise(res => setTimeout(res, POLL_INTERVAL_MS));
+  }
+}
+
+main().catch(e => {
+  console.error("[auditor] fatal:", e);
+  process.exit(1);
+});

auditor/policy.ts

@@ -0,0 +1,62 @@
+// ═══════════════════════════════════════════════════════════════════
+// YOU WRITE THIS FILE. Policy decides what blocks vs what's a comment.
+// Defaults are opinionated on the "stop clicking past placeholder"
+// side — easier to loosen than to tighten when you're watching the
+// auditor behave in live PRs.
+// ═══════════════════════════════════════════════════════════════════
+
+import type { Finding, Verdict } from "./types.ts";
+
+/// Translate the four-check output into a single verdict. This is the
+/// single pane of glass the auditor operates on — tune thresholds here.
+export function assembleVerdict(
+  findings: Finding[],
+  metrics: Record<string, number>,
+  pr_number: number,
+  head_sha: string,
+): Verdict {
+  const blocking = findings.filter(f => f.severity === "block");
+  const warning = findings.filter(f => f.severity === "warn");
+
+  let overall: Verdict["overall"];
+  let one_liner: string;
+
+  if (blocking.length > 0) {
+    overall = "block";
+    one_liner = `${blocking.length} blocking issue${blocking.length > 1 ? "s" : ""}: ${blocking[0].summary}`;
+  } else if (warning.length >= 3) {
+    // Three or more warnings is a block — death by a thousand cuts.
+    overall = "request_changes";
+    one_liner = `${warning.length} warnings — see review`;
+  } else if (warning.length > 0) {
+    overall = "request_changes";
+    one_liner = warning[0].summary;
+  } else {
+    overall = "approve";
+    one_liner = `all checks passed (${findings.length} findings, all info)`;
+  }
+
+  return {
+    pr_number,
+    head_sha,
+    audited_at: new Date().toISOString(),
+    overall,
+    findings,
+    metrics,
+    one_liner,
+  };
+}
+
+/// Which strength-of-claim warrants which severity when evidence is
+/// weak? A "Phase X shipped" claim with zero integration tests is a
+/// blocker. A "should work" claim with no test is a warn.
+export function severityFromClaimEvidence(
+  claim_strength: "weak" | "moderate" | "strong",
+  evidence_grade: "none" | "partial" | "full",
+): "info" | "warn" | "block" {
+  if (evidence_grade === "full") return "info";
+  if (claim_strength === "strong" && evidence_grade === "none") return "block";
+  if (claim_strength === "strong" && evidence_grade === "partial") return "warn";
+  if (claim_strength === "moderate" && evidence_grade === "none") return "warn";
+  return "info";
+}

auditor/types.ts

@@ -0,0 +1,65 @@
+// Shared types for the claim-auditor. Every field exists for a reason;
+// if something can't be verified from a check, it goes into `evidence`
+// so the verdict is inspectable, not a black box.
+
+export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
+
+export type Severity = "info" | "warn" | "block";
+
+export interface Claim {
+  // Verbatim phrase that raised the claim — e.g. "Phase 38 shipped",
+  // "verified end-to-end", "works after restart". Used as the "what
+  // does the author assert" input to downstream checks.
+  text: string;
+  // Where it came from. `commit_sha` is the short hash; `location`
+  // is a file:line for in-diff claims, or "pr_body" / "commit_message".
+  commit_sha: string;
+  location: string;
+  // Heuristic rating of how strong the claim is. "green+tested"
+  // is strong; "should work" is weak. Drives sensitivity — stronger
+  // claims get harder-blocked on weak evidence.
+  strength: "weak" | "moderate" | "strong";
+}
+
+export interface Finding {
+  check: CheckKind;
+  severity: Severity;
+  claim_text?: string;
+  // Free-form short description: "field added but never read", "no
+  // test covers this code path", "cloud model says placeholder".
+  summary: string;
+  // Concrete evidence: file paths, line numbers, log excerpts, test
+  // output, cloud-model verdict. No handwaving.
+  evidence: string[];
+}
+
+export interface Verdict {
+  pr_number: number;
+  head_sha: string;
+  audited_at: string;
+  overall: "approve" | "request_changes" | "block";
+  findings: Finding[];
+  // Real numbers that downstream policy can gate on. e.g. if the
+  // hybrid test produced latency numbers or token counts, they
+  // surface here so /auditor/history is queryable.
+  metrics: Record<string, number>;
+  // Short one-line justification for the `overall` verdict. What
+  // gets posted as the commit-status description in Gitea (max 140
+  // chars) must fit here.
+  one_liner: string;
+}
+
+export interface PrSnapshot {
+  number: number;
+  head_sha: string;
+  base_sha: string;
+  title: string;
+  body: string;
+  state: "open" | "closed" | "merged";
+  author: string;
+  // Array of commit messages in the PR (not diffs — those are
+  // fetched on-demand per-check).
+  commits: Array<{ sha: string; message: string; author: string }>;
+  // File paths touched by the PR, with lines-added / lines-removed.
+  files: Array<{ path: string; additions: number; deletions: number }>;
+}

bot/README.md

@@ -0,0 +1,127 @@
+# Lakehouse PR-Bot
+
+A local sub-agent that reads the PRD, asks a cloud model for a small change
+proposal, applies it, runs tests, and opens a draft PR on Gitea. Manual-only
+right now — no systemd, no cron. Run one cycle at a time and watch.
+
+## Run one cycle
+
+```bash
+cd /home/profit/lakehouse
+bun run bot/cycle.ts
+```
+
+Prerequisites:
+- Working tree must be clean (`git status` shows nothing). The bot refuses on a dirty tree so its changes don't mix with in-flight work.
+- Sidecar running on :3200, gateway on :3100, observer on :3800.
+- At least one PRD line tagged `[bot-eligible]` (see below).
+- Gitea PAT configured in `~/.git-credentials` (already set up).
+
+## Tag a gap as bot-eligible
+
+Add `[bot-eligible]` to any PRD line the bot is allowed to work on:
+
+```md
+- [ ] Add a unit test for parse_city_state covering "South Bend, IN" edge case [bot-eligible]
+```
+
+The bot scans `docs/PRD.md` for these tags. Each tagged line becomes a candidate.
+Start small — one tag at a time — until you trust the loop.
+
+## Stop it mid-cycle
+
+Create the pause file:
+
+```bash
+touch /home/profit/lakehouse/bot.paused
+```
+
+The next `bot/cycle.ts` invocation exits immediately with `skipped_pause`.
+(It does NOT kill a cycle already in-flight — use `Ctrl-C` or `pkill -f bot/cycle.ts` for that.)
+
+## Budget
+
+- **20 cloud calls/day, 160k tokens/day** (hard ceiling, see `bot/cost.ts`).
+- Tracked in `data/_bot/cost-YYYY-MM-DD.json`.
+- Resets at UTC midnight.
+
+## Cycle outcomes
+
+Every run writes a result to `data/_bot/cycles/{cycle_id}.json` and POSTs an event to the observer on :3800.
+Possible outcomes:
+
+| Outcome | Meaning |
+|---|---|
+| `ok` | PR opened |
+| `cycle_noop` | proposal applied, every file was **identical to current content** (Mem0 NOOP); no test run, no PR |
+| `skipped_pause` | pause file present |
+| `skipped_cost` | daily budget exhausted |
+| `skipped_policy` | `policy.shouldRunCycle` said no |
+| `skipped_dirty_tree` | uncommitted changes present |
+| `skipped_no_gap` | no `[bot-eligible]` tags in PRD |
+| `model_failed` | sidecar/cloud model errored or returned unparseable JSON |
+| `proposal_rejected` | `policy.scoreProposal` rejected it (size, path, etc.) |
+| `apply_failed` | file write errored |
+| `tests_failed` | cargo or bun test red |
+| `pr_skipped_by_policy` | green tests, but `policy.shouldOpenPR` said no |
+| `pr_failed` | Gitea API or git push errored |
+
+## Mem0-aligned apply semantics
+
+`apply.ts` categorizes every proposed file into one of three modes:
+
+| Mode | Trigger | Action |
+|---|---|---|
+| **ADD** | `is_new: true` and file doesn't exist | Create the file |
+| **UPDATE** | `is_new: false`, file exists, content **differs** from current | Overwrite |
+| **NOOP** | `is_new: false`, file exists, content **matches** current exactly | Skip (no write, no diff) |
+
+If every file in the proposal is NOOP, the cycle short-circuits to `cycle_noop` **before** running tests or opening a PR. Mismatched shapes (`is_new:true` but file exists, or `is_new:false` but file missing) become `apply_failed` — model state confusion is surfaced, not papered over.
+
+PR bodies report the three counts separately so reviewers see what actually changed vs. what was confirmed identical.
+
+## How the bot compounds over cycles
+
+Every finished cycle persists a `CycleResult` to `data/_bot/cycles/{cycle_id}.json`. At the **start** of the next cycle, `bot/kb.ts::loadHistory(gap_id)` scans that directory, filters to prior cycles on the same gap, and returns the five most recent outcomes (`ok`, `tests_failed`, `proposal_rejected`, `cycle_noop`, etc.) with their reasons and touched files.
+
+Those outcomes are summarized into a compact block and **injected into the cloud prompt** before asking for a new proposal. The model sees things like:
+
+```
+Prior attempts on this gap (3 most recent):
+- 2026-04-22 03:15 UTC — tests_failed
+    reason: cargo test -p vectord::lance failed on field_type_coerce
+- 2026-04-22 02:48 UTC — proposal_rejected
+    reason: touched forbidden path (docs/ADR-): docs/ADR-019-update.md
+- 2026-04-22 01:23 UTC — ok PR: https://git.agentview.dev/profit/lakehouse/pulls/142
+    reason: PR #142 opened
+
+Learn from these: build on what worked, avoid paths that failed.
+```
+
+This is the same compounding pattern `scenario.ts` uses via `kb.loadRecommendation` — the bot's cycles are the bot's memory. No embedding, no separate jsonl, no cross-run orchestration required. First cycle on a new gap skips the block cleanly.
+
+The observer events POSTed on every cycle carry the same data, so `GET :3800/stats?source=bot` aggregates "% of bot cycles on similar gaps that landed a PR" without extra plumbing.
+
+## Where YOU edit
+
+`bot/policy.ts` — four small functions that define what the bot does. The rest
+(`propose.ts`, `apply.ts`, `test.ts`, `pr.ts`, `cycle.ts`, `kb.ts`) is mechanical
+orchestration — you shouldn't need to touch it unless a pipeline changes.
+
+One policy upgrade the history opens up: in `shouldRunCycle`, you can now bail
+out if the same gap has failed N times in a row. Example addition:
+
+```ts
+import { loadHistory, statsFor } from "./kb.ts";
+// inside shouldRunCycle — but you'd need the gap, which is picked later.
+// A more natural place is scoreProposal: if prior N failures on this
+// gap's path+summary pattern, reject before testing.
+```
+
+## Hard-coded guardrails (not in policy — can't be disabled)
+
+- Bot **never deletes files** (`apply.ts` has no delete path).
+- Bot **never touches** `.git/`, `secrets`, `lakehouse.toml`, `docs/ADR-*`, `docs/DECISIONS.md`, `docs/PRD.md`, `/etc/`, `/root/`, `Cargo.lock`.
+- All paths validated for traversal (`path/../`) and repo-escape.
+- PRs always open as **draft** — never auto-merge.
+- Budget check is before the policy function runs — no way to override the daily cap from `policy.ts`.

bot/apply.ts

@@ -0,0 +1,86 @@
+// Apply a proposal to disk with Mem0-aligned ADD / UPDATE / NOOP
+// semantics. Whole-file writes only — never deletes. All paths
+// validated repo-relative + path-traversal-free.
+//
+// Three outcomes per file:
+//   ADD    — file didn't exist, new content written (is_new must be true)
+//   UPDATE — file existed, content differs from current, overwritten
+//   NOOP   — file existed, content identical to current, nothing written
+//
+// The NOOP case is important: a cloud proposal that "re-derives" an
+// existing file shouldn't churn the repo or burn test cycles.
+
+import { writeFile, readFile, mkdir, access } from "node:fs/promises";
+import { dirname, resolve, relative } from "node:path";
+import type { ApplyOutcome, Proposal } from "./types.ts";
+
+const REPO_ROOT = "/home/profit/lakehouse";
+
+export async function applyProposal(p: Proposal): Promise<ApplyOutcome> {
+  const added: string[] = [];
+  const updated: string[] = [];
+  const noop: string[] = [];
+  const errors: string[] = [];
+
+  for (const f of p.files) {
+    const err = validatePath(f.path);
+    if (err) {
+      errors.push(`${f.path}: ${err}`);
+      continue;
+    }
+    const abs = resolve(REPO_ROOT, f.path);
+    try {
+      const exists = await fileExists(abs);
+
+      if (f.is_new && exists) {
+        errors.push(`${f.path}: is_new=true but file exists`);
+        continue;
+      }
+      if (!f.is_new && !exists) {
+        // Proposal claims to update a file that doesn't exist.
+        // Refuse rather than silently ADD — model is confused about
+        // repo state, and we want that surfaced, not papered over.
+        errors.push(`${f.path}: is_new=false but file does not exist`);
+        continue;
+      }
+
+      if (!f.is_new) {
+        // UPDATE candidate — compare to detect NOOP.
+        const current = await readFile(abs, "utf8");
+        if (current === f.content) {
+          noop.push(f.path);
+          continue;
+        }
+        await writeFile(abs, f.content);
+        updated.push(f.path);
+      } else {
+        // ADD — file doesn't exist yet.
+        await mkdir(dirname(abs), { recursive: true });
+        await writeFile(abs, f.content);
+        added.push(f.path);
+      }
+    } catch (e) {
+      errors.push(`${f.path}: ${(e as Error).message}`);
+    }
+  }
+  return { added, updated, noop, errors };
+}
+
+function validatePath(p: string): string | null {
+  if (!p) return "empty path";
+  if (p.startsWith("/")) return "must be repo-relative";
+  if (p.startsWith("..") || p.includes("/../")) return "path traversal";
+  const abs = resolve(REPO_ROOT, p);
+  const rel = relative(REPO_ROOT, abs);
+  if (rel.startsWith("..")) return "resolves outside repo";
+  return null;
+}
+
+async function fileExists(abs: string): Promise<boolean> {
+  try {
+    await access(abs);
+    return true;
+  } catch {
+    return false;
+  }
+}

bot/cost.ts

@@ -0,0 +1,52 @@
+// Daily cost tracker. Resets at UTC midnight by keying the file by date.
+// Hard ceilings are the policy surface — the bot refuses to call the
+// cloud once the budget is exhausted for the day.
+
+import { readFile, writeFile, mkdir } from "node:fs/promises";
+import { join } from "node:path";
+import type { CostState } from "./types.ts";
+
+const COST_DIR = "/home/profit/lakehouse/data/_bot";
+
+export const DAILY_CALLS_BUDGET = 20;
+export const DAILY_TOKENS_BUDGET = 8000 * DAILY_CALLS_BUDGET; // 160k tokens/day ceiling
+
+function todayUtc(): string {
+  return new Date().toISOString().slice(0, 10);
+}
+
+function costFile(date: string): string {
+  return join(COST_DIR, `cost-${date}.json`);
+}
+
+export async function readCost(): Promise<CostState> {
+  const date = todayUtc();
+  try {
+    const raw = await readFile(costFile(date), "utf8");
+    return JSON.parse(raw) as CostState;
+  } catch {
+    return { date, calls: 0, tokens: 0 };
+  }
+}
+
+export async function recordCost(calls: number, tokens: number): Promise<CostState> {
+  await mkdir(COST_DIR, { recursive: true });
+  const current = await readCost();
+  const updated: CostState = {
+    date: current.date,
+    calls: current.calls + calls,
+    tokens: current.tokens + tokens,
+  };
+  await writeFile(costFile(updated.date), JSON.stringify(updated, null, 2));
+  return updated;
+}
+
+export function budgetCheck(c: CostState): { ok: boolean; reason: string } {
+  if (c.calls >= DAILY_CALLS_BUDGET) {
+    return { ok: false, reason: `daily call budget exhausted (${c.calls}/${DAILY_CALLS_BUDGET})` };
+  }
+  if (c.tokens >= DAILY_TOKENS_BUDGET) {
+    return { ok: false, reason: `daily token budget exhausted (${c.tokens}/${DAILY_TOKENS_BUDGET})` };
+  }
+  return { ok: true, reason: `${c.calls}/${DAILY_CALLS_BUDGET} calls, ${c.tokens}/${DAILY_TOKENS_BUDGET} tokens used today` };
+}

bot/cycle.ts

@@ -0,0 +1,345 @@
+// Single-cycle orchestrator. This is the manual entry point:
+//   bun run bot/cycle.ts
+//
+// Steps (each step can short-circuit with a clear outcome):
+//   1. Pause-file check         → skipped_pause
+//   2. Cost budget check        → skipped_cost
+//   3. Gather CycleContext      (dirty tree, last cycle, autotune status)
+//   4. policy.shouldRunCycle    → skipped_policy
+//   5. findGaps + policy.pickGap → skipped_no_gap
+//   6. generateProposal         → model_failed
+//   7. policy.scoreProposal     → proposal_rejected
+//   8. git checkout new branch
+//   9. applyProposal            → apply_failed
+//  10. runTests                 → tests_failed
+//  11. policy.shouldOpenPR      → pr_skipped_by_policy
+//  12. git commit + push
+//  13. openPr                   → pr_failed / ok
+// 14. recordCost + postEvent (every outcome)
+// 15. write cycle result to data/_bot/cycles/{cycle_id}.json
+
+import { readFile, writeFile, mkdir, access } from "node:fs/promises";
+import { createHash, randomUUID } from "node:crypto";
+import { spawnSync } from "node:child_process";
+import { join } from "node:path";
+
+import type { CycleContext, CycleResult, CycleOutcome, Proposal } from "./types.ts";
+import * as policy from "./policy.ts";
+import { readCost, recordCost, budgetCheck, DAILY_CALLS_BUDGET, DAILY_TOKENS_BUDGET } from "./cost.ts";
+import { findGaps, generateProposal } from "./propose.ts";
+import { applyProposal } from "./apply.ts";
+import { runTests } from "./test.ts";
+import { openPr } from "./pr.ts";
+import { postEvent } from "./observer.ts";
+import { loadHistory, summarizeHistory, statsFor } from "./kb.ts";
+
+const REPO_ROOT = "/home/profit/lakehouse";
+const PAUSE_FILE = `${REPO_ROOT}/bot.paused`;
+const CYCLES_DIR = `${REPO_ROOT}/data/_bot/cycles`;
+const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
+
+function log(msg: string) { console.log(`[bot] ${msg}`); }
+
+async function fileExists(p: string): Promise<boolean> {
+  try { await access(p); return true; } catch { return false; }
+}
+
+function git(args: string[]): { code: number; stdout: string; stderr: string } {
+  const r = spawnSync("git", args, { cwd: REPO_ROOT, encoding: "utf8" });
+  return { code: r.status ?? -1, stdout: r.stdout ?? "", stderr: r.stderr ?? "" };
+}
+
+async function gatherContext(cost: { calls: number; tokens: number }): Promise<CycleContext> {
+  // Last cycle: newest file in CYCLES_DIR.
+  let lastCycleAt: string | null = null;
+  let lastCycleGapId: string | null = null;
+  try {
+    const { readdir, stat } = await import("node:fs/promises");
+    const entries = await readdir(CYCLES_DIR).catch(() => [] as string[]);
+    let newestTs = 0;
+    let newestPath: string | null = null;
+    for (const e of entries) {
+      if (!e.endsWith(".json")) continue;
+      const p = join(CYCLES_DIR, e);
+      const s = await stat(p);
+      if (s.mtimeMs > newestTs) { newestTs = s.mtimeMs; newestPath = p; }
+    }
+    if (newestPath) {
+      const r = JSON.parse(await readFile(newestPath, "utf8")) as CycleResult;
+      lastCycleAt = r.started_at;
+      lastCycleGapId = r.gap?.id ?? null;
+    }
+  } catch {}
+
+  // Autotune status — if the agent is busy, skip this cycle to avoid
+  // contention on shared Ollama GPU memory.
+  let autotuneBusy = false;
+  try {
+    const r = await fetch(`${GATEWAY_URL}/vectors/agent/status`, { signal: AbortSignal.timeout(2000) });
+    if (r.ok) {
+      const j = await r.json() as any;
+      autotuneBusy = Boolean(j.running) && (j.queue_depth ?? 0) > 0;
+    }
+  } catch {}
+
+  // Dirty tree check.
+  const status = git(["status", "--porcelain"]);
+  const workingTreeDirty = status.code === 0 && status.stdout.trim().length > 0;
+
+  return {
+    startedAt: new Date().toISOString(),
+    dailyCallsUsed: cost.calls,
+    dailyCallsBudget: DAILY_CALLS_BUDGET,
+    dailyTokensUsed: cost.tokens,
+    dailyTokensBudget: DAILY_TOKENS_BUDGET,
+    lastCycleAt,
+    lastCycleGapId,
+    autotuneBusy,
+    workingTreeDirty,
+  };
+}
+
+async function persistResult(res: CycleResult): Promise<void> {
+  await mkdir(CYCLES_DIR, { recursive: true });
+  await writeFile(join(CYCLES_DIR, `${res.cycle_id}.json`), JSON.stringify(res, null, 2));
+}
+
+function sigHash(gap_id: string | null, proposalSummary: string): string {
+  const h = createHash("sha256");
+  h.update(gap_id ?? "no-gap");
+  h.update("|");
+  h.update(proposalSummary);
+  return h.digest("hex").slice(0, 16);
+}
+
+function emit(res: CycleResult): Promise<void> {
+  return postEvent({
+    source: "bot",
+    cycle_id: res.cycle_id,
+    sig_hash: sigHash(res.gap?.id ?? null, res.proposal?.summary ?? ""),
+    event_kind: res.outcome,
+    ok: res.outcome === "ok",
+    staffer_id: res.proposal?.model_used,
+    turns: res.cloud_calls,
+    duration_ms: new Date(res.ended_at).getTime() - new Date(res.started_at).getTime(),
+    extra: {
+      reason: res.reason,
+      pr_url: res.prUrl,
+      files_added: res.filesAdded,
+      files_updated: res.filesUpdated,
+      files_noop: res.filesNoop,
+      tests_green: res.testsGreen,
+    },
+  });
+}
+
+async function main() {
+  const cycle_id = `bot-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomUUID().slice(0, 8)}`;
+  const started_at = new Date().toISOString();
+  log(`cycle ${cycle_id} starting`);
+
+  const finish = async (outcome: CycleOutcome, reason: string, extra: Partial<CycleResult> = {}): Promise<CycleResult> => {
+    const res: CycleResult = {
+      cycle_id,
+      started_at,
+      ended_at: new Date().toISOString(),
+      outcome,
+      reason,
+      gap: null,
+      proposal: null,
+      filesAdded: [],
+      filesUpdated: [],
+      filesNoop: [],
+      testsGreen: null,
+      testsOutput: "",
+      prUrl: null,
+      tokens_used: 0,
+      cloud_calls: 0,
+      ...extra,
+    };
+    await persistResult(res);
+    await emit(res);
+    log(`→ ${outcome}: ${reason}`);
+    return res;
+  };
+
+  // 1. Pause.
+  if (await fileExists(PAUSE_FILE)) {
+    return finish("skipped_pause", `pause file at ${PAUSE_FILE}`);
+  }
+
+  // 2. Cost.
+  const cost = await readCost();
+  const bc = budgetCheck(cost);
+  if (!bc.ok) return finish("skipped_cost", bc.reason);
+  log(`cost: ${bc.reason}`);
+
+  // 3+4. Context + policy.shouldRunCycle.
+  const ctx = await gatherContext(cost);
+  const srq = policy.shouldRunCycle(ctx);
+  if (!srq.run) {
+    return finish(ctx.workingTreeDirty ? "skipped_dirty_tree" : "skipped_policy", srq.reason);
+  }
+
+  // 5. Gaps + policy.pickGap.
+  const gaps = await findGaps();
+  log(`found ${gaps.length} [bot-eligible] gap(s) in PRD`);
+  const gapPick = policy.pickGap(gaps);
+  if (!gapPick.gap) return finish("skipped_no_gap", gapPick.reason);
+  const gap = gapPick.gap;
+  log(`picked gap ${gap.id}: "${gap.prd_line.slice(0, 80)}"`);
+
+  // 6. KB lookup + Proposal.
+  // Bot cycles compound: prior cycles on this gap inform the current
+  // proposal. Summary is a compact block injected into the cloud prompt
+  // so the model can build on past successes and steer around past
+  // failures. Empty string when this is the first cycle on this gap.
+  const history = await loadHistory(gap.id, 5);
+  const stats = statsFor(history);
+  if (history.length > 0) {
+    log(`kb: ${stats.attempts} prior cycle(s) — ${stats.pr_opened} pr / ${stats.tests_failed} tests_failed / ${stats.proposal_rejected} rejected / ${stats.noop} noop`);
+  } else {
+    log(`kb: first cycle on this gap`);
+  }
+  const historySummary = summarizeHistory(history);
+
+  let proposal: Proposal;
+  try {
+    proposal = await generateProposal(gap, historySummary);
+    log(`proposal: ${proposal.summary} (${proposal.files.length} files, ~${proposal.estimated_loc} LOC, ${proposal.tokens_used} tokens)`);
+  } catch (e) {
+    await recordCost(1, 0);
+    return finish("model_failed", (e as Error).message, { gap, cloud_calls: 1 });
+  }
+  await recordCost(1, proposal.tokens_used);
+
+  // 7. Score.
+  const score = policy.scoreProposal(proposal);
+  if (!score.accept) {
+    return finish("proposal_rejected", score.reason, { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
+  }
+  log(`proposal accepted: ${score.reason}`);
+
+  // 8. Branch.
+  const dayStamp = new Date().toISOString().slice(0, 10).replace(/-/g, "");
+  const branch = `bot/cycle-${dayStamp}-${gap.id}`;
+  // Branch must be fresh. If one already exists from a prior failed cycle
+  // for the same gap, bail — don't silently overwrite or reuse stale state.
+  const check = git(["rev-parse", "--verify", `refs/heads/${branch}`]);
+  if (check.code === 0) {
+    return finish("apply_failed",
+      `branch ${branch} already exists — delete it manually if you want a fresh cycle for this gap`,
+      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
+  }
+  const co = git(["checkout", "-b", branch]);
+  if (co.code !== 0) {
+    return finish("apply_failed", `git checkout -b failed: ${co.stderr}`,
+      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
+  }
+  log(`on branch ${branch}`);
+
+  // 9. Apply — Mem0 ADD/UPDATE/NOOP semantics.
+  const ap = await applyProposal(proposal);
+  if (ap.errors.length > 0) {
+    git(["checkout", "main"]);
+    git(["branch", "-D", branch]);
+    return finish("apply_failed", ap.errors.join("; "),
+      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
+  }
+  const changed = [...ap.added, ...ap.updated];
+  log(`apply: ${ap.added.length} add, ${ap.updated.length} update, ${ap.noop.length} noop`);
+
+  // Short-circuit: if every proposed file was identical to what's on
+  // disk, there's nothing to test or PR. Clean up the branch and exit.
+  if (changed.length === 0) {
+    git(["checkout", "main"]);
+    git(["branch", "-D", branch]);
+    return finish("cycle_noop",
+      `all ${proposal.files.length} proposed file(s) were identical to current state`,
+      { gap, proposal, filesNoop: ap.noop,
+        tokens_used: proposal.tokens_used, cloud_calls: 1 });
+  }
+
+  // 10. Test.
+  log(`running tests (this will take a while)...`);
+  const testRes = await runTests();
+  if (!testRes.green) {
+    git(["checkout", "."]);
+    git(["checkout", "main"]);
+    git(["branch", "-D", branch]);
+    return finish("tests_failed", "cargo or bun test failed", {
+      gap, proposal,
+      filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
+      testsGreen: false, testsOutput: testRes.output,
+      tokens_used: proposal.tokens_used, cloud_calls: 1,
+    });
+  }
+  log(`tests green`);
+
+  // 11. PR gate.
+  const partial: CycleResult = {
+    cycle_id, started_at,
+    ended_at: new Date().toISOString(),
+    outcome: "ok", reason: "",
+    gap, proposal,
+    filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
+    testsGreen: true, testsOutput: testRes.output,
+    prUrl: null, tokens_used: proposal.tokens_used, cloud_calls: 1,
+  };
+  const prGate = policy.shouldOpenPR(partial);
+  if (!prGate.open) {
+    git(["checkout", "main"]);
+    git(["branch", "-D", branch]);
+    return finish("pr_skipped_by_policy", prGate.reason, partial);
+  }
+
+  // 12. Commit + push.
+  git(["add", ...changed]);
+  const commitMsg = [
+    `bot: ${proposal.summary}`,
+    "",
+    proposal.rationale,
+    "",
+    `PRD gap: ${gap.prd_line}`,
+    `Cycle: ${cycle_id}`,
+    "",
+    `Co-Authored-By: ${proposal.model_used} (cloud) <bot@lakehouse.local>`,
+  ].join("\n");
+  const ci = git(["commit", "-m", commitMsg]);
+  if (ci.code !== 0) {
+    return finish("pr_failed", `git commit failed: ${ci.stderr}`, partial);
+  }
+  const ps = git(["push", "-u", "origin", branch]);
+  if (ps.code !== 0) {
+    return finish("pr_failed", `git push failed: ${ps.stderr}`, partial);
+  }
+
+  // 13. Open PR.
+  try {
+    const pr = await openPr({
+      branch,
+      title: `bot: ${proposal.summary}`,
+      body: [
+        `**Gap** (PRD line ${gap.line_number}): ${gap.prd_line}`,
+        "",
+        `**Rationale**`,
+        proposal.rationale,
+        "",
+        `**Cycle**: \`${cycle_id}\``,
+        `**Model**: \`${proposal.model_used}\``,
+        `**Tokens**: ${proposal.tokens_used}`,
+        `**Added**: ${ap.added.map(f => `\`${f}\``).join(", ") || "_none_"}`,
+        `**Updated**: ${ap.updated.map(f => `\`${f}\``).join(", ") || "_none_"}`,
+        ap.noop.length > 0 ? `**NOOP (identical to current)**: ${ap.noop.map(f => `\`${f}\``).join(", ")}` : "",
+      ].join("\n"),
+    });
+    return finish("ok", `PR #${pr.number} opened`, { ...partial, prUrl: pr.html_url });
+  } catch (e) {
+    return finish("pr_failed", (e as Error).message, partial);
+  }
+}
+
+main().catch(e => {
+  console.error("[bot] fatal:", e);
+  process.exit(1);
+});

bot/kb.ts

@@ -0,0 +1,101 @@
+// Bot-local knowledge base. Every finished cycle already persists a
+// CycleResult to data/_bot/cycles/{id}.json — that IS the outcome log.
+// KB here just reads that dir, filters to prior cycles on the same gap,
+// and produces a short summary the cloud model can condition on.
+//
+// No separate jsonl, no new write path, no embedding calls. The bot's
+// "memory" is the same primary artifact that the observer consumes.
+//
+// Future: embedding-based neighbor matching across gaps (cheap once
+// sidecar is local), cross-pollination with scenario KB's
+// pathway_recommendations. Not required for the feedback loop to work
+// on a single gap — that's the floor we're building first.
+
+import { readdir, readFile } from "node:fs/promises";
+import { join } from "node:path";
+import type { CycleResult } from "./types.ts";
+
+const CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
+
+export interface HistoryEntry {
+  cycle_id: string;
+  ended_at: string;
+  outcome: string;
+  reason: string;
+  pr_url: string | null;
+  tests_green: boolean | null;
+  files_added: string[];
+  files_updated: string[];
+  tokens_used: number;
+}
+
+export async function loadHistory(gap_id: string, max: number = 5): Promise<HistoryEntry[]> {
+  let entries: string[] = [];
+  try {
+    entries = await readdir(CYCLES_DIR);
+  } catch {
+    return [];
+  }
+  const matches: HistoryEntry[] = [];
+  for (const e of entries) {
+    if (!e.endsWith(".json")) continue;
+    try {
+      const raw = await readFile(join(CYCLES_DIR, e), "utf8");
+      const r = JSON.parse(raw) as CycleResult;
+      if (r.gap?.id !== gap_id) continue;
+      matches.push({
+        cycle_id: r.cycle_id,
+        ended_at: r.ended_at,
+        outcome: r.outcome,
+        reason: r.reason,
+        pr_url: r.prUrl,
+        tests_green: r.testsGreen,
+        files_added: r.filesAdded ?? [],
+        files_updated: r.filesUpdated ?? [],
+        tokens_used: r.tokens_used,
+      });
+    } catch {
+      // Skip unreadable / malformed cycle files. Don't fail the current
+      // cycle because an old one is corrupt.
+    }
+  }
+  matches.sort((a, b) => b.ended_at.localeCompare(a.ended_at));
+  return matches.slice(0, max);
+}
+
+// Compact prompt-ready summary. Empty string when there's no history —
+// caller can skip the "prior attempts" block entirely.
+export function summarizeHistory(h: HistoryEntry[]): string {
+  if (h.length === 0) return "";
+  const lines = h.map(e => {
+    const when = e.ended_at.slice(0, 16).replace("T", " ");
+    const files = [...e.files_added, ...e.files_updated];
+    const filesStr = files.length > 0 ? ` touched: ${files.join(", ")}` : "";
+    const prStr = e.pr_url ? ` PR: ${e.pr_url}` : "";
+    return `- ${when} UTC — ${e.outcome}${prStr}${filesStr}\n    reason: ${e.reason}`;
+  });
+  return [
+    `Prior attempts on this gap (${h.length} most recent):`,
+    ...lines,
+    "",
+    "Learn from these: build on what worked, avoid paths that failed.",
+  ].join("\n");
+}
+
+// Aggregate stats for telemetry — lets the bot expose "% of cycles on
+// this gap that landed a PR" without re-parsing the raw history.
+export function statsFor(h: HistoryEntry[]): {
+  attempts: number;
+  pr_opened: number;
+  tests_failed: number;
+  proposal_rejected: number;
+  noop: number;
+} {
+  return {
+    attempts: h.length,
+    pr_opened: h.filter(e => e.outcome === "ok").length,
+    tests_failed: h.filter(e => e.outcome === "tests_failed").length,
+    proposal_rejected: h.filter(e => e.outcome === "proposal_rejected").length,
+    noop: h.filter(e => e.outcome === "cycle_noop").length,
+  };
+}

bot/observer.ts

@@ -0,0 +1,34 @@
+// POST bot cycle outcomes to the observer on :3800 so they accumulate
+// in the KB alongside scenario + autotune events. Non-fatal on failure:
+// the cycle result is also written to disk so observability isn't a
+// single point of failure.
+
+const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
+
+export interface BotObserverEvent {
+  source: "bot";
+  cycle_id: string;
+  sig_hash: string;          // stable hash of the gap + proposal for dedup
+  event_kind: string;        // cycle outcome (ok, tests_failed, ...)
+  ok: boolean;
+  staffer_id?: string;       // the model used, e.g. "gpt-oss:120b"
+  turns?: number;            // number of cloud calls this cycle
+  duration_ms?: number;
+  extra?: Record<string, any>;
+}
+
+export async function postEvent(ev: BotObserverEvent): Promise<void> {
+  try {
+    const r = await fetch(`${OBSERVER_URL}/event`, {
+      method: "POST",
+      headers: { "content-type": "application/json" },
+      body: JSON.stringify(ev),
+      signal: AbortSignal.timeout(3000),
+    });
+    if (!r.ok) {
+      console.error(`[bot/observer] ${r.status}: ${await r.text()}`);
+    }
+  } catch (e) {
+    console.error(`[bot/observer] POST failed: ${(e as Error).message}`);
+  }
+}

bot/policy.ts

@@ -0,0 +1,70 @@
+// ═══════════════════════════════════════════════════════════════════
+// YOU WRITE THIS FILE. These four functions shape the bot's entire
+// behavior. Defaults below are safe and intentionally conservative —
+// redline to taste. Each function is ~5-10 lines.
+// ═══════════════════════════════════════════════════════════════════
+
+import type { CycleContext, CycleResult, Gap, Proposal } from "./types.ts";
+
+// Should this cycle even fire right now?
+// Called AFTER the pause-file check + cost-cap check have already passed.
+// Use this for judgment calls (cooldown, shared-resource conflicts).
+export function shouldRunCycle(ctx: CycleContext): { run: boolean; reason: string } {
+  if (ctx.workingTreeDirty) {
+    return { run: false, reason: "dirty working tree — commit or stash your work first" };
+  }
+  if (ctx.lastCycleAt) {
+    const minsAgo = (Date.now() - new Date(ctx.lastCycleAt).getTime()) / 60000;
+    if (minsAgo < 10) return { run: false, reason: `last cycle ${minsAgo.toFixed(1)}min ago (<10min cooldown)` };
+  }
+  if (ctx.autotuneBusy) {
+    return { run: false, reason: "autotune agent is running — avoid Ollama contention" };
+  }
+  return { run: true, reason: "ready" };
+}
+
+// Which [bot-eligible] gap from the PRD does this cycle attack?
+// Default: first in document order. Alternatives: random for exploration,
+// smallest-context, tag-priority (e.g. [bot-eligible:high] wins).
+export function pickGap(gaps: Gap[]): { gap: Gap | null; reason: string } {
+  if (gaps.length === 0) return { gap: null, reason: "no [bot-eligible] gaps in PRD" };
+  return { gap: gaps[0], reason: "first eligible gap in document order" };
+}
+
+// Cheap gate BEFORE burning test cycles. Reject if proposal smells wrong.
+// This catches runaway cloud proposals before we apply+test them.
+export function scoreProposal(p: Proposal): { accept: boolean; reason: string } {
+  if (p.files.length === 0) return { accept: false, reason: "empty proposal" };
+  if (p.files.length > 5) return { accept: false, reason: `>5 files (${p.files.length})` };
+  if (p.estimated_loc > 200) return { accept: false, reason: `>200 LOC (${p.estimated_loc})` };
+
+  // Paths the bot is NEVER allowed to touch. Extend cautiously.
+  const forbidden = [
+    ".git/", "secrets", "lakehouse.toml",
+    "docs/ADR-", "docs/DECISIONS.md", "docs/PRD.md",
+    "/etc/", "/root/", "Cargo.lock",
+  ];
+  for (const f of p.files) {
+    const hit = forbidden.find(prefix => f.path.includes(prefix));
+    if (hit) return { accept: false, reason: `touches forbidden path (${hit}): ${f.path}` };
+  }
+  return { accept: true, reason: "passes size + path filters" };
+}
+
+// Tests are green and changes applied. Do we actually open a PR?
+// Mem0-aligned: the cycle already short-circuited with `cycle_noop` if
+// every file was identical to current state, so by the time we're here
+// at least one file was actually added or updated.
+//
+// Stricter options below — uncomment to require a test file touched, etc.
+export function shouldOpenPR(c: CycleResult): { open: boolean; reason: string } {
+  if (!c.testsGreen) return { open: false, reason: "tests red" };
+  const realChanges = c.filesAdded.length + c.filesUpdated.length;
+  if (realChanges === 0) return { open: false, reason: "nothing actually changed on disk" };
+
+  // Stricter option — require at least one test file in the diff:
+  // const hasTest = [...c.filesAdded, ...c.filesUpdated].some(f => /test|_tests/i.test(f));
+  // if (!hasTest) return { open: false, reason: "no test file touched" };
+
+  return { open: true, reason: `green + ${c.filesAdded.length} add / ${c.filesUpdated.length} update` };
+}

bot/pr.ts

@@ -0,0 +1,58 @@
+// Gitea PR open. Reads PAT from ~/.git-credentials (set up by the
+// credential-helper flow). All PRs open as DRAFT so J reviews before
+// merge — never auto-merge.
+
+import { readFile } from "node:fs/promises";
+
+const GITEA_HOST = "https://git.agentview.dev";
+const REPO_OWNER = "profit";
+const REPO_NAME = "lakehouse";
+const CRED_FILE = "/home/profit/.git-credentials";
+
+async function readPat(): Promise<string> {
+  const raw = await readFile(CRED_FILE, "utf8");
+  for (const line of raw.split("\n")) {
+    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
+    if (m) return m[1];
+  }
+  throw new Error(`no Gitea PAT found in ${CRED_FILE}`);
+}
+
+export interface OpenPrInput {
+  branch: string;    // head branch (just pushed)
+  base?: string;     // default "main"
+  title: string;
+  body: string;
+}
+
+export interface OpenPrResult {
+  number: number;
+  html_url: string;
+}
+
+export async function openPr(input: OpenPrInput): Promise<OpenPrResult> {
+  const pat = await readPat();
+  const url = `${GITEA_HOST}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/pulls`;
+  const payload = {
+    title: input.title,
+    body: input.body,
+    head: input.branch,
+    base: input.base ?? "main",
+    // Gitea uses "draft" flag — always on for bot PRs.
+    draft: true,
+  };
+  const r = await fetch(url, {
+    method: "POST",
+    headers: {
+      "content-type": "application/json",
+      "Authorization": `token ${pat}`,
+    },
+    body: JSON.stringify(payload),
+    signal: AbortSignal.timeout(30000),
+  });
+  if (!r.ok) {
+    throw new Error(`Gitea ${r.status}: ${await r.text()}`);
+  }
+  const j = await r.json() as any;
+  return { number: j.number, html_url: j.html_url };
+}

bot/propose.ts

@@ -0,0 +1,141 @@
+// Gap detection + cloud proposal.
+//
+// Gap detection: scan docs/PRD.md for lines tagged [bot-eligible].
+// Each match becomes a Gap with surrounding context.
+//
+// Proposal: one-shot call to the T3 cloud model via the Python sidecar's
+// /generate endpoint. Asks for a structured JSON response with file
+// contents. Truncation-resistant via Phase 21's generate_continuable —
+// for now we pass max_tokens high and rely on the model completing in
+// one pass; swap to the Rust continuation wrapper if we see truncation.
+
+import { readFile } from "node:fs/promises";
+import { createHash } from "node:crypto";
+import type { Gap, Proposal } from "./types.ts";
+
+const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
+const REPO_ROOT = "/home/profit/lakehouse";
+const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
+const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
+const MAX_TOKENS = 6000;
+
+export async function findGaps(): Promise<Gap[]> {
+  const prd = await readFile(PRD_PATH, "utf8");
+  const lines = prd.split("\n");
+  const gaps: Gap[] = [];
+  for (let i = 0; i < lines.length; i++) {
+    if (!lines[i].includes("[bot-eligible]")) continue;
+    const contextLines = lines.slice(i, Math.min(i + 6, lines.length)).join("\n");
+    const id = createHash("sha256").update(lines[i]).digest("hex").slice(0, 12);
+    gaps.push({
+      id,
+      prd_line: lines[i].trim(),
+      context: contextLines,
+      source_file: "docs/PRD.md",
+      line_number: i + 1,
+    });
+  }
+  return gaps;
+}
+
+const SYSTEM_PROMPT = `You are an assistant that proposes small, testable code changes to the Lakehouse repo.
+The Lakehouse is a Rust-first data platform with 13 crates + Bun/TypeScript test harness.
+You will be given one PRD gap tagged [bot-eligible] and must respond with a STRICT JSON object — no prose.
+
+Rules:
+- Response MUST be a single JSON object, no markdown fences, no commentary.
+- Change MUST be small: <200 lines total, ≤5 files.
+- Include at least one test file (new or modified) that proves the change.
+- NEVER touch .git/, secrets, lakehouse.toml, docs/ADR-*, docs/DECISIONS.md, docs/PRD.md, /etc/, /root/, Cargo.lock.
+- Paths MUST be repo-relative (no leading /).
+- Whole-file contents only — no patches, no diffs.
+
+Response shape:
+{
+  "summary": "one line",
+  "rationale": "why this addresses the gap",
+  "files": [ { "path": "crates/foo/src/bar.rs", "content": "<full file>", "is_new": false } ],
+  "estimated_loc": 42
+}`;
+
+export async function generateProposal(gap: Gap, historySummary: string = ""): Promise<Proposal> {
+  const sections = [
+    `PRD gap (line ${gap.line_number}):`,
+    "```",
+    gap.context,
+    "```",
+    "",
+  ];
+  if (historySummary) {
+    sections.push(historySummary, "");
+  }
+  sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
+  const userPrompt = sections.join("\n");
+
+  const r = await fetch(`${SIDECAR_URL}/generate`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({
+      model: CLOUD_MODEL,
+      system: SYSTEM_PROMPT,
+      prompt: userPrompt,
+      temperature: 0.2,
+      max_tokens: MAX_TOKENS,
+      think: false,
+    }),
+    signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
+  });
+  if (!r.ok) {
+    throw new Error(`sidecar ${r.status}: ${await r.text()}`);
+  }
+  const j = await r.json() as any;
+  const raw: string = j.text ?? j.response ?? "";
+  const usage = j.usage ?? {};
+  const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
+
+  const parsed = extractJson(raw);
+  if (!parsed || typeof parsed !== "object") {
+    throw new Error(`model returned no JSON object. Raw head: ${raw.slice(0, 300)}`);
+  }
+  if (!Array.isArray(parsed.files)) {
+    throw new Error(`proposal.files not an array: ${JSON.stringify(parsed).slice(0, 200)}`);
+  }
+
+  return {
+    summary: String(parsed.summary ?? "").trim(),
+    rationale: String(parsed.rationale ?? "").trim(),
+    files: parsed.files.map((f: any) => ({
+      path: String(f.path ?? ""),
+      content: String(f.content ?? ""),
+      is_new: Boolean(f.is_new),
+    })).filter((f: any) => f.path && f.content !== undefined),
+    estimated_loc: Number(parsed.estimated_loc ?? 0),
+    model_used: CLOUD_MODEL,
+    tokens_used: tokens,
+  };
+}
+
+// Find the first balanced JSON object in the string. Tolerates leading
+// "```json" fences even though we asked the model not to emit them.
+function extractJson(text: string): any | null {
+  const cleaned = text.replace(/^```(?:json)?\s*/i, "").replace(/```\s*$/i, "").trim();
+  let depth = 0;
+  let start = -1;
+  for (let i = 0; i < cleaned.length; i++) {
+    const c = cleaned[i];
+    if (c === "{") {
+      if (depth === 0) start = i;
+      depth++;
+    } else if (c === "}") {
+      depth--;
+      if (depth === 0 && start >= 0) {
+        try {
+          return JSON.parse(cleaned.slice(start, i + 1));
+        } catch {
+          start = -1;
+        }
+      }
+    }
+  }
+  return null;
+}

bot/test.ts

@@ -0,0 +1,46 @@
+// Run the test gate: `cargo test --workspace` + `bun test`.
+// Returns { green, output } — output is last ~4KB of combined stdout/stderr.
+
+import { spawn } from "node:child_process";
+
+const REPO_ROOT = "/home/profit/lakehouse";
+const TEST_TIMEOUT_MS = 15 * 60 * 1000; // 15 min cargo + 2 min bun, generous
+
+export async function runTests(): Promise<{ green: boolean; output: string }> {
+  const cargoOut = await runCmd("cargo", ["test", "--workspace", "--quiet", "--", "--test-threads=1"], REPO_ROOT, TEST_TIMEOUT_MS);
+  if (cargoOut.code !== 0) {
+    return { green: false, output: tail(`cargo test (${cargoOut.code}):\n${cargoOut.combined}`) };
+  }
+
+  const bunOut = await runCmd("bun", ["test", "tests/multi-agent"], REPO_ROOT, 120000);
+  const bunGreen = bunOut.code === 0;
+  const combined = [
+    `cargo test: OK`,
+    `bun test (${bunOut.code}):`,
+    bunOut.combined,
+  ].join("\n");
+  return { green: bunGreen, output: tail(combined) };
+}
+
+function tail(s: string, n = 4096): string {
+  return s.length > n ? "…" + s.slice(-n) : s;
+}
+
+function runCmd(
+  cmd: string,
+  args: string[],
+  cwd: string,
+  timeoutMs: number,
+): Promise<{ code: number; combined: string }> {
+  return new Promise(resolveP => {
+    const child = spawn(cmd, args, { cwd, env: { ...process.env } });
+    let combined = "";
+    child.stdout.on("data", d => { combined += d.toString(); });
+    child.stderr.on("data", d => { combined += d.toString(); });
+    const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
+    child.on("close", code => {
+      clearTimeout(timer);
+      resolveP({ code: code ?? -1, combined });
+    });
+  });
+}

bot/types.ts

@@ -0,0 +1,88 @@
+// Shared types for the PR-bot. Small and explicit — the bot's behavior
+// should be readable from these shapes alone.
+
+export interface Gap {
+  id: string;              // hash of prd_line — stable key for dedup
+  prd_line: string;        // the [bot-eligible]-tagged line, verbatim
+  context: string;         // surrounding context from the PRD (next ~5 lines)
+  source_file: string;     // e.g. "docs/PRD.md"
+  line_number: number;
+}
+
+export interface ProposalFile {
+  path: string;            // repo-relative (no leading slash)
+  content: string;         // full file content — the bot writes whole files, not patches
+  is_new: boolean;         // true = create, false = overwrite existing
+}
+
+export interface Proposal {
+  summary: string;         // one-line
+  rationale: string;       // why this change addresses the gap
+  files: ProposalFile[];
+  estimated_loc: number;   // total added+changed lines across all files
+  model_used: string;
+  tokens_used: number;
+}
+
+export interface CycleContext {
+  startedAt: string;       // ISO
+  dailyCallsUsed: number;
+  dailyCallsBudget: number;
+  dailyTokensUsed: number;
+  dailyTokensBudget: number;
+  lastCycleAt: string | null;
+  lastCycleGapId: string | null;
+  autotuneBusy: boolean;
+  workingTreeDirty: boolean;
+}
+
+export type CycleOutcome =
+  | "ok"                    // PR opened
+  | "skipped_pause"
+  | "skipped_cost"
+  | "skipped_policy"
+  | "skipped_no_gap"
+  | "skipped_dirty_tree"
+  | "proposal_rejected"
+  | "apply_failed"
+  | "cycle_noop"            // proposal applied but every file was identical to what's on disk
+  | "tests_failed"
+  | "pr_skipped_by_policy"
+  | "pr_failed"
+  | "model_failed";
+
+// Mem0-aligned apply outcomes. Three shapes instead of the binary
+// "written" / "errored". NOOP is a first-class outcome — identical
+// content shouldn't waste test cycles or open an empty PR.
+export type ApplyMode = "add" | "update" | "noop";
+
+export interface ApplyOutcome {
+  added: string[];
+  updated: string[];
+  noop: string[];
+  errors: string[];
+}
+
+export interface CycleResult {
+  cycle_id: string;
+  started_at: string;
+  ended_at: string;
+  outcome: CycleOutcome;
+  reason: string;
+  gap: Gap | null;
+  proposal: Proposal | null;
+  filesAdded: string[];
+  filesUpdated: string[];
+  filesNoop: string[];
+  testsGreen: boolean | null;     // null = not run
+  testsOutput: string;
+  prUrl: string | null;
+  tokens_used: number;
+  cloud_calls: number;
+}
+
+export interface CostState {
+  date: string;          // YYYY-MM-DD UTC
+  calls: number;
+  tokens: number;
+}

bun.lock

@@ -0,0 +1,18 @@
+{
+  "lockfileVersion": 1,
+  "configVersion": 1,
+  "workspaces": {
+    "": {
+      "dependencies": {
+        "langfuse": "^3.38.20",
+      },
+    },
+  },
+  "packages": {
+    "langfuse": ["langfuse@3.38.20", "", { "dependencies": { "langfuse-core": "^3.38.20" } }, "sha512-MAmBAASSzJtmK1O9HQegA1mFsQhT8Yf+OJRGvE7FXkyv3g/eiBE0glLD0Ohg3pkxhoPdggM5SejK7ue9ctlaMA=="],
+
+    "langfuse-core": ["langfuse-core@3.38.20", "", { "dependencies": { "mustache": "^4.2.0" } }, "sha512-zBKVmQN/1oT5VWZUBYlWzvokIlkC/6mnpgr/2atMyTeAm+jR3ia7w2iJMjlrF5/oG8ukO1s8+LDRCzJpF1QeEA=="],
+
+    "mustache": ["mustache@4.2.0", "", { "bin": { "mustache": "bin/mustache" } }, "sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ=="],
+  }
+}

config/models.json

@@ -0,0 +1,193 @@
+{
+  "_description": "Lakehouse model matrix — authoritative routing for all agent tiers. Local models do the heavy lifting; cloud models are consulted sparingly for overview, strategic, and gatekeeper decisions. Read by tests/multi-agent/scenario.ts and mcp-server/index.ts.",
+  "version": 1,
+  "updated": "2026-04-21",
+  "providers": {
+    "ollama_local": {
+      "base_url": "http://localhost:11434",
+      "key_env": null
+    },
+    "ollama_cloud": {
+      "base_url": "https://ollama.com",
+      "key_env": "OLLAMA_CLOUD_KEY",
+      "key_source": "/root/llm_team_config.json → providers.ollama_cloud.api_key",
+      "rate_budget": {
+        "calls_per_hour": 200,
+        "calls_per_day": 2000,
+        "notes": "Paid tier — generous. Policy: keep overview calls ≤ 3/scenario, strategic ≤ 10/day, gatekeeper ≤ 5/day."
+      }
+    }
+  },
+  "tiers": {
+    "t1_hot": {
+      "purpose": "Per tool call — SQL generation, hybrid_search, sql(). Runs 50-200 times per scenario. Latency-sensitive.",
+      "kind": "local_fast",
+      "primary": { "model": "qwen3.5:latest", "provider": "ollama_local", "context_window": 262144 },
+      "fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
+      "max_tokens": 1000,
+      "temperature": 0.3,
+      "never_route_cloud": true,
+      "context_budget": {
+        "system_prompt_cap": 4000,
+        "prior_context_cap": 6000,
+        "tool_results_cap": 8000,
+        "safety_margin": 2000,
+        "overflow_policy": "summarize_oldest_tool_results_via_t3"
+      },
+      "rationale": "qwen3.5:latest is a 9.7B thinking model with 262K context that emits clean JSON (verified 2026-04-21 after mistral A/B wipeout). Thinking budget requires max_tokens ≥ 400; we hold 1000 to keep propose_done payloads intact. qwen2.5 stays as fallback."
+    },
+    "t2_review": {
+      "purpose": "Per step consensus — executor ↔ reviewer loop critique. 5-14 calls per event.",
+      "kind": "local_balanced",
+      "primary": { "model": "qwen3:latest", "provider": "ollama_local", "context_window": 40960 },
+      "fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
+      "max_tokens": 800,
+      "temperature": 0.3,
+      "never_route_cloud": true,
+      "context_budget": {
+        "system_prompt_cap": 2000,
+        "recent_turns_cap": 4000,
+        "safety_margin": 1000
+      },
+      "rationale": "Reviewer only needs to detect schema violations and drift — a 7B model is sufficient."
+    },
+    "t3_overview": {
+      "purpose": "Mid-day checkpoint after every misplacement + every Nth event, and cross-day lesson. 1-3 calls per scenario.",
+      "kind": "thinking_cloud",
+      "primary": { "model": "gpt-oss:120b", "provider": "ollama_cloud", "context_window": 131072 },
+      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
+      "max_tokens": 900,
+      "temperature": 0.2,
+      "cloud_budget_per_scenario": 5,
+      "env_flag": "LH_OVERVIEW_CLOUD=1",
+      "context_budget": {
+        "event_digest_cap": 30000,
+        "checkpoint_cap": 8000,
+        "lesson_corpus_cap": 40000,
+        "safety_margin": 8000,
+        "overflow_policy": "chunk_lessons_via_cosine_topk"
+      },
+      "rationale": "Same prompt family as local 20b (gpt-oss series) — prompts port directly. 120b is faster via cloud than 20b local in practice, and lessons are noticeably more specific."
+    },
+    "t4_strategic": {
+      "purpose": "Daily playbook board re-ranking, weekly gap audit, pattern discovery across accumulated playbooks. 1-10 calls per day.",
+      "kind": "thinking_cloud_large",
+      "primary": { "model": "kimi-k2.6", "provider": "ollama_cloud", "context_window": 200000 },
+      "secondary": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
+      "fallback": { "model": "glm-4.7", "provider": "ollama_cloud", "context_window": 131072 },
+      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
+      "max_tokens": 2000,
+      "temperature": 0.2,
+      "cloud_budget_per_day": 10,
+      "context_budget": {
+        "playbook_corpus_cap": 80000,
+        "pattern_history_cap": 20000,
+        "safety_margin": 16000,
+        "overflow_policy": "two_pass_map_reduce"
+      },
+      "rationale": "J named qwen3.5 specifically. GLM-4.7 is a promising alternate for debate phase. Runs after all scenarios complete for the day."
+    },
+    "t5_gatekeeper": {
+      "purpose": "MUST route here: architecture changes, new client onboarding, schema migrations, playbook retirements, index rebuilds, autotune config changes.",
+      "kind": "thinking_cloud_deepest",
+      "primary": { "model": "kimi-k2-thinking", "provider": "ollama_cloud", "context_window": 200000 },
+      "fallback": { "model": "deepseek-v3.1:671b", "provider": "ollama_cloud", "context_window": 131072 },
+      "secondary_fallback": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
+      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
+      "max_tokens": 4000,
+      "temperature": 0.1,
+      "cloud_budget_per_day": 5,
+      "audit_log": true,
+      "context_budget": {
+        "decision_doc_cap": 50000,
+        "evidence_bundle_cap": 100000,
+        "prior_gatekeeper_decisions_cap": 20000,
+        "safety_margin": 20000,
+        "overflow_policy": "escalate_to_kimi_k2_1t_or_split_decision"
+      },
+      "rationale": "Highest-stakes decisions — reasoning depth matters more than latency. Audit log so J can always see what the gatekeeper was asked and what it answered. No human approval required today; escalate later if mis-decisions show up."
+    }
+  },
+
+  "embeddings": {
+    "_description": "Vector embedding models for hybrid search + playbook_memory similarity. Always local — embedding calls are too high-volume for cloud. qwen3-embedding is the new primary (2026-04-21); nomic-embed-text-v2-moe is the MoE alternative for corpus-scale embedding; original nomic-embed-text stays as legacy fallback while vectord indexes are being re-embedded.",
+    "primary": { "model": "qwen3-embedding", "provider": "ollama_local", "dim": 1024, "context_window": 32768 },
+    "alternate": { "model": "nomic-embed-text-v2-moe", "provider": "ollama_local", "dim": 768, "context_window": 2048, "notes": "MoE variant — better for long documents, slower per-call, roughly matches nomic-embed-text on short text." },
+    "legacy": { "model": "nomic-embed-text:latest", "provider": "ollama_local", "dim": 768, "context_window": 8192, "notes": "Used by existing vectord indexes. Do NOT switch live until indexes are rebuilt, or hybrid search recall will crater." },
+    "migration_plan": "Re-embed workers_500k (and all production indexes) with qwen3-embedding, keep the old index under a _v1 suffix for rollback, flip the profile pointer only after recall benchmarks match or exceed legacy. Touches vectord::agent + autotune."
+  },
+
+  "context_management": {
+    "_description": "Rule zero: NEVER call a model with more tokens than its context_window minus safety_margin. Every call goes through the budget checker first. If over budget → chunk, summarize, or escalate. This is the stability floor.",
+    "token_estimator": {
+      "method": "chars_div_4",
+      "note": "Rough, biased safe by ~15%. For production, swap to tiktoken or the provider's tokenizer endpoint."
+    },
+    "overflow_policies": {
+      "summarize_oldest_tool_results_via_t3": {
+        "when": "T1 conversation history + tool results exceed context_budget.tool_results_cap",
+        "how": "Send oldest N tool results to T3 with prompt 'summarize these into 500 tokens that preserve what the executor needs to know'; replace them with the summary in the running conversation."
+      },
+      "chunk_lessons_via_cosine_topk": {
+        "when": "lesson corpus in data/_playbook_lessons/*.json exceeds lesson_corpus_cap",
+        "how": "Embed the current scenario spec, cosine-rank all lessons, take top-K until budget exhausted. Fall back to date-sorted if embeddings unavailable."
+      },
+      "two_pass_map_reduce": {
+        "when": "T4 playbook corpus exceeds playbook_corpus_cap",
+        "how": "Pass 1: chunk playbooks into ≤30K token shards, run primary model on each shard to emit 'shard summary'. Pass 2: feed all summaries to primary model for global synthesis. Logged as two audit entries."
+      },
+      "escalate_to_kimi_k2_1t_or_split_decision": {
+        "when": "T5 decision evidence exceeds decision_doc_cap + evidence_bundle_cap",
+        "how": "Prefer kimi-k2:1t which has 1M context. If still over, split decision into sub-decisions (e.g. 'retire playbooks by city' instead of 'retire playbooks globally') and loop."
+      }
+    },
+    "chunking_cache": {
+      "_description": "Precomputed shards of the playbook corpus, indexed by (corpus_version, shard_id). Avoids re-chunking on every T4 run.",
+      "location": "data/_chunk_cache/",
+      "invalidation": "Key includes corpus hash. When playbook_memory changes, the hash changes, the cache misses, and chunks regenerate.",
+      "implementation_status": "SPEC — next sprint."
+    },
+    "implementation_status": "context_window + context_budget fields WIRED in config. Enforcement helper NOT yet wired in agent.ts. Next: add estimateTokens() + budgetCheck() helpers, route all generate() calls through them, emit a metric when overflow policy fires."
+  },
+  "experimental_rotation": {
+    "enabled": false,
+    "purpose": "Sample newer models on a schedule to collect comparison data without rate-limit risk.",
+    "candidates": [
+      { "model": "minimax-m2.7",      "notes": "Newer minimax; unknown output stability" },
+      { "model": "glm-5",             "notes": "GLM next-gen; larger context" },
+      { "model": "glm-5.1",           "notes": "Incremental on GLM-5" },
+      { "model": "qwen3-next:80b",    "notes": "Qwen's experimental successor; smaller than 3.5" },
+      { "model": "qwen3-coder-next",  "notes": "Coder-optimized — good for SQL gen T1 experiments" },
+      { "model": "deepseek-v3.2",     "notes": "Smaller deepseek; reasoning/coding" },
+      { "model": "nemotron-3-super",  "notes": "NVIDIA 230B; general" },
+      { "model": "cogito-2.1:671b",   "notes": "671B general" },
+      { "model": "mistral-large-3:675b", "notes": "Mistral's flagship; good T3 candidate" }
+    ],
+    "rotation": "weekly",
+    "sample_rate": 0.1,
+    "apply_to_tier": "t4_strategic",
+    "notes": "When enabled, T4 routes 10% of calls to a rotating experimental model. Log comparison in /data/_model_eval/ — if the experimental consistently beats primary across 3 rotations, promote it to primary."
+  },
+  "playbook_versioning": {
+    "enabled": true,
+    "purpose": "A playbook can work, then break when architecture changes. Versioning lets us pin which change retired which playbook.",
+    "dataset": "playbook_memory",
+    "schema_additions": {
+      "version": "integer — auto-increment per operation",
+      "parent_id": "string — previous version entry_id for same operation (null for v1)",
+      "retired_at": "timestamp — set when success_rate drops or architecture changes",
+      "retirement_reason": "string — e.g. 'schema_migration:workers_500k 2026-05-03'",
+      "architecture_snapshot": "object — crate versions, index name, schema fingerprint at seed time"
+    },
+    "retire_triggers": [
+      "success_rate < 0.3 over last 20 citations",
+      "schema_fingerprint mismatch detected at retrieval time",
+      "architecture change event emitted by ingestd/vectord",
+      "T5 gatekeeper explicitly retires via /vectors/playbook_memory/retire"
+    ],
+    "read_back_policy": "retrieval returns only non-retired versions. History endpoint /vectors/playbook_memory/history/{operation} returns the full chain.",
+    "ui_surface": "mcp-server to render a diff view: side-by-side of playbook versions with a timeline of what changed and when.",
+    "implementation_status": "SPEC — not yet wired. Target: next sprint. Touches gateway + catalogd + mcp-server."
+  },
+  "matrix_index_hybrid_search_note": "Phase 22 candidate: elevate the hybrid_search T1 tool to consult T3 when a pool returns <3 matches OR when the same (role, city) combo has failed N times in 24h. Consult result is a reformulated sql_filter the executor retries with. Keeps T1 fast on the happy path, escalates to T3 only on low-recall signals."
+}

crates/aibridge/src/client.rs

@@ -25,7 +25,7 @@ pub struct EmbedResponse {
     pub dimensions: usize,
 }
 
-#[derive(Serialize, Deserialize)]
+#[derive(Clone, Serialize, Deserialize)]
 pub struct GenerateRequest {
     pub prompt: String,
     #[serde(skip_serializing_if = "Option::is_none")]
@@ -36,6 +36,14 @@ pub struct GenerateRequest {
     pub temperature: Option<f64>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub max_tokens: Option<u32>,
+    /// Phase 21 — per-call opt-out of hidden reasoning. Thinking models
+    /// (qwen3.5, gpt-oss, etc) burn tokens on reasoning before the
+    /// visible response starts; setting this to `false` on hot-path
+    /// JSON emitters avoids empty returns when the budget is tight.
+    /// Sidecar forwards this to Ollama's `think` parameter; if the
+    /// sidecar drops an unknown field the request still succeeds.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub think: Option<bool>,
 }
 
 #[derive(Deserialize, Serialize, Clone)]

crates/aibridge/src/context.rs

@@ -0,0 +1,194 @@
+//! Phase 21 — context-budget accounting for model calls.
+//!
+//! Ports `assertContextBudget` + `estimateTokens` + `CONTEXT_WINDOWS`
+//! from `tests/multi-agent/agent.ts` so Rust-side callers (gateway
+//! tool surfaces, future Rust agents) get the same loud-fail behavior
+//! on window overflow instead of silent truncation.
+//!
+//! The token estimator is deliberately the same chars/4 heuristic as
+//! the TS side. It's biased ~15% safe — pessimistic on English, correct
+//! within a factor of 2 on code. Swap to a provider tokenizer only when
+//! the estimator drives a decision (we're nowhere near that yet).
+
+use std::collections::HashMap;
+use std::sync::OnceLock;
+
+/// Rough token count. `chars / 4` ceiling. See module docs for why
+/// this heuristic is sufficient.
+pub fn estimate_tokens(text: &str) -> usize {
+    (text.chars().count() + 3) / 4
+}
+
+/// Phase 21 — per-model context windows, mirroring the TS table in
+/// `tests/multi-agent/agent.ts`. Anchored on each model's documented
+/// max; unknown models fall back to `DEFAULT_CONTEXT_WINDOW`.
+pub const DEFAULT_CONTEXT_WINDOW: usize = 32_768;
+pub const DEFAULT_SAFETY_MARGIN: usize = 2_000;
+pub const DEFAULT_MAX_TOKENS: usize = 800;
+
+fn known_windows() -> &'static HashMap<&'static str, usize> {
+    static TABLE: OnceLock<HashMap<&'static str, usize>> = OnceLock::new();
+    TABLE.get_or_init(|| {
+        let mut m = HashMap::new();
+        m.insert("mistral:latest", 32_768);
+        m.insert("qwen2.5:latest", 32_768);
+        m.insert("qwen3:latest", 40_960);
+        m.insert("qwen3.5:latest", 262_144);
+        m.insert("qwen3-embedding", 32_768);
+        m.insert("nomic-embed-text-v2-moe", 2_048);
+        m.insert("gpt-oss:20b", 131_072);
+        m.insert("gpt-oss:120b", 131_072);
+        m.insert("qwen3.5:397b", 131_072);
+        m.insert("kimi-k2-thinking", 200_000);
+        m.insert("kimi-k2.6", 200_000);
+        m.insert("kimi-k2:1t", 1_048_576);
+        m.insert("deepseek-v3.1:671b", 131_072);
+        m.insert("glm-4.7", 131_072);
+        m
+    })
+}
+
+pub fn context_window_for(model: &str) -> usize {
+    known_windows().get(model).copied().unwrap_or(DEFAULT_CONTEXT_WINDOW)
+}
+
+/// Result of a budget check — exposes the numbers so callers can log
+/// how much headroom remains without re-running the estimator.
+#[derive(Debug, Clone, Copy)]
+pub struct BudgetCheck {
+    pub estimated: usize,
+    pub window: usize,
+    pub remaining: i64,
+}
+
+/// Inputs to `assert_context_budget`. `bypass` exists for call sites
+/// that handle their own overflow (continuation's second pass already
+/// counted the partial; T5 gatekeeper prompts have a separate policy).
+#[derive(Debug, Clone, Default)]
+pub struct BudgetOpts<'a> {
+    pub system: Option<&'a str>,
+    pub max_tokens: Option<usize>,
+    pub safety_margin: Option<usize>,
+    pub bypass: bool,
+}
+
+/// Phase 21's loud-fail primitive. Returns a `BudgetCheck` on success
+/// and the same struct plus over-by count on failure. The whole point
+/// is to stop silent truncation — callers that expect overflow should
+/// chunk BEFORE calling or set `bypass: true`.
+pub fn assert_context_budget(
+    model: &str,
+    prompt: &str,
+    opts: BudgetOpts,
+) -> Result<BudgetCheck, (BudgetCheck, usize)> {
+    let window = context_window_for(model);
+    let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
+    let max_tokens = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS);
+    let sys_tokens = opts.system.map(estimate_tokens).unwrap_or(0);
+    let estimated = estimate_tokens(prompt) + sys_tokens + max_tokens;
+    let remaining = window as i64 - estimated as i64 - safety as i64;
+    let check = BudgetCheck { estimated, window, remaining };
+    if remaining < 0 && !opts.bypass {
+        return Err((check, (-remaining) as usize));
+    }
+    Ok(check)
+}
+
+/// Convenience — format an overflow error the same way the TS side
+/// does. Exposed so downstream crates render consistent messages.
+pub fn overflow_message(model: &str, check: &BudgetCheck, over_by: usize, safety: usize) -> String {
+    format!(
+        "context overflow: model={} est={}t window={}t safety={}t over={}t. \
+         Chunk the prompt (see config/models.json overflow_policies) or set \
+         bypass:true if you know the risk.",
+        model, check.estimated, check.window, safety, over_by,
+    )
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn estimate_tokens_ceiling_divides_by_four() {
+        assert_eq!(estimate_tokens(""), 0);
+        assert_eq!(estimate_tokens("abc"), 1);    // 3 → ceil(3/4) = 1
+        assert_eq!(estimate_tokens("abcd"), 1);   // 4 → ceil(4/4) = 1
+        assert_eq!(estimate_tokens("abcde"), 2);  // 5 → ceil(5/4) = 2
+        assert_eq!(estimate_tokens(&"x".repeat(400)), 100);
+    }
+
+    #[test]
+    fn context_window_known_and_fallback() {
+        assert_eq!(context_window_for("qwen3.5:latest"), 262_144);
+        assert_eq!(context_window_for("kimi-k2:1t"), 1_048_576);
+        assert_eq!(context_window_for("some-unreleased-model"), DEFAULT_CONTEXT_WINDOW);
+    }
+
+    #[test]
+    fn budget_passes_well_under_window() {
+        let check = assert_context_budget(
+            "qwen3:latest",
+            &"x".repeat(4_000), // ~1000 tokens
+            BudgetOpts { max_tokens: Some(500), ..Default::default() },
+        ).expect("well under 40K window");
+        assert!(check.remaining > 30_000);
+    }
+
+    #[test]
+    fn budget_fails_when_prompt_overflows_window() {
+        let huge = "x".repeat(200_000); // ~50K tokens, over qwen3's 40K
+        let err = assert_context_budget(
+            "qwen3:latest",
+            &huge,
+            BudgetOpts::default(),
+        ).expect_err("should overflow qwen3's 40K window");
+        assert!(err.1 > 0, "over_by must be positive");
+    }
+
+    #[test]
+    fn budget_bypass_returns_ok_even_over() {
+        let huge = "x".repeat(200_000);
+        let check = assert_context_budget(
+            "qwen3:latest",
+            &huge,
+            BudgetOpts { bypass: true, ..Default::default() },
+        ).expect("bypass must suppress the error");
+        assert!(check.remaining < 0, "check still reports negative remaining");
+    }
+
+    #[test]
+    fn budget_counts_system_prompt() {
+        // 10K-char system prompt → ~2500 tokens. With a big max_tokens
+        // this should push us closer to the window.
+        let sys = "s".repeat(10_000);
+        let prompt = "p".repeat(4_000);
+        let with_sys = assert_context_budget(
+            "qwen3:latest",
+            &prompt,
+            BudgetOpts {
+                system: Some(&sys),
+                max_tokens: Some(500),
+                ..Default::default()
+            },
+        ).unwrap();
+        let without_sys = assert_context_budget(
+            "qwen3:latest",
+            &prompt,
+            BudgetOpts { max_tokens: Some(500), ..Default::default() },
+        ).unwrap();
+        assert!(with_sys.estimated > without_sys.estimated,
+            "system prompt should raise estimate");
+        assert_eq!(with_sys.estimated - without_sys.estimated, estimate_tokens(&sys));
+    }
+
+    #[test]
+    fn overflow_message_includes_numbers() {
+        let check = BudgetCheck { estimated: 42_000, window: 40_960, remaining: -1_040 };
+        let msg = overflow_message("qwen3:latest", &check, 3_040, 2_000);
+        assert!(msg.contains("qwen3:latest"));
+        assert!(msg.contains("42000t"));
+        assert!(msg.contains("40960t"));
+        assert!(msg.contains("3040t"));
+    }
+}

crates/aibridge/src/continuation.rs

@@ -0,0 +1,438 @@
+//! Phase 21 — OUTPUT-overflow handler. Ports `generateContinuable`
+//! from `tests/multi-agent/agent.ts`.
+//!
+//! Two failure modes to repair:
+//!
+//! * EMPTY response — thinking model ate the entire budget on hidden
+//!   reasoning before emitting a token. Fix: retry the original prompt
+//!   with 2× the budget, geometric up to `BUDGET_CAP`.
+//!
+//! * TRUNCATED non-empty — model got most of the way but hit
+//!   max_tokens before closing the structure. Fix: continue with the
+//!   partial response in the prompt as scratchpad, so the model knows
+//!   where to pick up without restarting.
+//!
+//! `TextGenerator` abstracts the sidecar so tests can inject canned
+//! responses without a live Ollama.
+
+use std::future::Future;
+
+use crate::client::{AiClient, GenerateRequest, GenerateResponse};
+
+/// Shape classifier for `is_structurally_complete`. JSON responses
+/// must parse; text responses just need to be non-empty.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ResponseShape {
+    Json,
+    Text,
+}
+
+/// Trait that `generate_continuable` + `generate_tree_split` call. The
+/// real implementation forwards to `AiClient::generate`; tests supply a
+/// mock with a scripted sequence of responses.
+pub trait TextGenerator: Send + Sync {
+    fn generate_text(
+        &self,
+        req: GenerateRequest,
+    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send;
+}
+
+impl TextGenerator for AiClient {
+    fn generate_text(
+        &self,
+        req: GenerateRequest,
+    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
+        self.generate(req)
+    }
+}
+
+/// Strip a surrounding ```json``` fence if present. Leaves the inner
+/// content alone otherwise. Returns a slice of `s`.
+fn strip_json_fence(s: &str) -> &str {
+    let t = s.trim();
+    if let Some(rest) = t.strip_prefix("```json") {
+        rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
+    } else if let Some(rest) = t.strip_prefix("```") {
+        rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
+    } else {
+        t
+    }
+}
+
+/// Port of the TS brace-balance + JSON.parse check. Returns true when
+/// the outermost `{...}` block is balanced and parses. Text shape is
+/// satisfied by any non-empty, non-whitespace payload.
+pub fn is_structurally_complete(text: &str, shape: ResponseShape) -> bool {
+    if text.trim().is_empty() { return false; }
+    if shape == ResponseShape::Text { return true; }
+
+    let s = strip_json_fence(text);
+    let Some(start) = s.find('{') else { return false };
+    let Some(end) = s.rfind('}') else { return false };
+    if end <= start { return false; }
+    let slice = &s[start..=end];
+
+    // Balance check — cheaper than parse, catches truncated nests.
+    // String state tracked because `{` inside a string doesn't count.
+    let mut depth: i32 = 0;
+    let mut in_str = false;
+    let mut esc = false;
+    for c in slice.chars() {
+        if esc { esc = false; continue; }
+        if c == '\\' { esc = true; continue; }
+        if c == '"' { in_str = !in_str; continue; }
+        if in_str { continue; }
+        if c == '{' { depth += 1; }
+        else if c == '}' {
+            depth -= 1;
+            if depth < 0 { return false; }
+        }
+    }
+    if depth != 0 { return false; }
+    // Parse check is the tie-breaker — balanced but invalid JSON (e.g.
+    // trailing comma before `}`) shouldn't count as complete.
+    serde_json::from_str::<serde_json::Value>(slice).is_ok()
+}
+
+/// Knobs for `generate_continuable`. All optional with sensible
+/// defaults that match the TS version.
+#[derive(Debug, Clone)]
+pub struct ContinuableOpts {
+    pub model: String,
+    pub max_tokens: Option<u32>,
+    pub temperature: Option<f64>,
+    pub system: Option<String>,
+    pub shape: ResponseShape,
+    pub max_continuations: usize,
+    pub think: Option<bool>,
+    /// Geometric-backoff ceiling for the empty-response retry path.
+    /// Matches TS's `budgetCap = 8000`.
+    pub budget_cap: u32,
+    /// Maximum empty-response retries before giving up. Matches TS's
+    /// hardcoded `retry < 3`.
+    pub max_empty_retries: usize,
+}
+
+impl ContinuableOpts {
+    pub fn new(model: impl Into<String>) -> Self {
+        Self {
+            model: model.into(),
+            max_tokens: None,
+            temperature: None,
+            system: None,
+            shape: ResponseShape::Json,
+            max_continuations: 3,
+            think: None,
+            budget_cap: 8_000,
+            max_empty_retries: 3,
+        }
+    }
+}
+
+/// Outcome of a `generate_continuable` call. Carries the combined
+/// text plus diagnostic counters so observability downstream can
+/// report "how many continuations did that query cost".
+#[derive(Debug, Clone)]
+pub struct ContinuableOutcome {
+    pub text: String,
+    pub empty_retries: usize,
+    pub continuations: usize,
+    pub final_complete: bool,
+}
+
+fn make_request(opts: &ContinuableOpts, prompt: String, current_max: u32) -> GenerateRequest {
+    GenerateRequest {
+        prompt,
+        model: Some(opts.model.clone()),
+        system: opts.system.clone(),
+        temperature: opts.temperature,
+        max_tokens: Some(current_max),
+        think: opts.think,
+    }
+}
+
+fn continuation_prompt(original: &str, partial: &str) -> String {
+    format!(
+        "{original}\n\n\
+         PARTIAL RESPONSE SO FAR (continue from here — do NOT restart, \
+         do NOT repeat what's already there, emit ONLY the remaining \
+         tokens to close the structure):\n{partial}"
+    )
+}
+
+/// Phase 21 — output-overflow safe generate. See module docs for the
+/// two failure modes repaired. On final-still-incomplete, returns the
+/// combined text with `final_complete: false` so the caller's parser
+/// can throw with the raw text for forensics rather than silently
+/// truncating.
+pub async fn generate_continuable<G: TextGenerator>(
+    generator: &G,
+    prompt: &str,
+    opts: &ContinuableOpts,
+) -> Result<ContinuableOutcome, String> {
+    let initial_max = opts.max_tokens.unwrap_or(800);
+    let mut current_max = initial_max;
+    let mut combined = String::new();
+    let mut empty_retries = 0usize;
+    let mut continuations = 0usize;
+
+    // Phase 21(a) — empty-response backoff loop.
+    for retry in 0..opts.max_empty_retries {
+        let req = make_request(opts, prompt.to_string(), current_max);
+        let resp = generator.generate_text(req).await?;
+        if !resp.text.trim().is_empty() {
+            combined = resp.text;
+            break;
+        }
+        empty_retries = retry + 1;
+        current_max = (current_max.saturating_mul(2)).min(opts.budget_cap);
+    }
+
+    // Phase 21(b) — structural-completion continuation loop. Runs on
+    // the truncated-non-empty case; empty + exhausted retries falls
+    // through with empty combined and final_complete=false.
+    for _ in 0..opts.max_continuations {
+        if is_structurally_complete(&combined, opts.shape) {
+            return Ok(ContinuableOutcome {
+                text: combined,
+                empty_retries,
+                continuations,
+                final_complete: true,
+            });
+        }
+        if combined.trim().is_empty() {
+            // Nothing to continue from — continuing "" is identical to
+            // the initial call and would loop. Bail so the caller sees
+            // the failure rather than burning N extra calls.
+            break;
+        }
+        let cont_prompt = continuation_prompt(prompt, &combined);
+        let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
+        let resp = generator.generate_text(req).await?;
+        combined.push_str(&resp.text);
+        continuations += 1;
+    }
+
+    let final_complete = is_structurally_complete(&combined, opts.shape);
+    Ok(ContinuableOutcome {
+        text: combined,
+        empty_retries,
+        continuations,
+        final_complete,
+    })
+}
+
+/// Scripted generator for unit tests. Returns responses from `script`
+/// in order; extra calls reuse the last entry so tests don't have to
+/// count past what they actually assert on.
+#[cfg(test)]
+pub struct ScriptedGenerator {
+    script: Vec<Result<String, String>>,
+    calls: std::sync::Arc<std::sync::Mutex<Vec<GenerateRequest>>>,
+}
+
+#[cfg(test)]
+impl ScriptedGenerator {
+    pub fn new<I>(script: I) -> Self
+    where
+        I: IntoIterator<Item = Result<String, String>>,
+    {
+        Self {
+            script: script.into_iter().collect(),
+            calls: std::sync::Arc::new(std::sync::Mutex::new(Vec::new())),
+        }
+    }
+
+    pub fn calls(&self) -> Vec<GenerateRequest> {
+        self.calls.lock().unwrap().clone()
+    }
+
+    pub fn call_count(&self) -> usize {
+        self.calls.lock().unwrap().len()
+    }
+}
+
+#[cfg(test)]
+impl TextGenerator for ScriptedGenerator {
+    fn generate_text(
+        &self,
+        req: GenerateRequest,
+    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
+        let i = {
+            let mut calls = self.calls.lock().unwrap();
+            let i = calls.len();
+            calls.push(req.clone());
+            i
+        };
+        let model = req.model.clone().unwrap_or_default();
+        let entry = self.script.get(i)
+            .cloned()
+            .unwrap_or_else(|| self.script.last().cloned().unwrap_or(Ok(String::new())));
+        async move {
+            entry.map(|text| GenerateResponse {
+                text,
+                model,
+                tokens_evaluated: None,
+                tokens_generated: None,
+            })
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn structural_complete_rejects_empty_and_text_mismatch() {
+        assert!(!is_structurally_complete("", ResponseShape::Text));
+        assert!(!is_structurally_complete("   ", ResponseShape::Text));
+        assert!(is_structurally_complete("any content", ResponseShape::Text));
+    }
+
+    #[test]
+    fn structural_complete_handles_balanced_json() {
+        assert!(is_structurally_complete(r#"{"a": 1}"#, ResponseShape::Json));
+        assert!(is_structurally_complete(
+            r#"```json
+{"a": 1, "b": [1, 2, 3]}
+```"#,
+            ResponseShape::Json,
+        ));
+    }
+
+    #[test]
+    fn structural_complete_rejects_truncated_json() {
+        assert!(!is_structurally_complete(r#"{"a": 1"#, ResponseShape::Json));
+        assert!(!is_structurally_complete(r#"{"a": {"b": 1"#, ResponseShape::Json));
+        // Trailing comma — balanced but unparseable
+        assert!(!is_structurally_complete(r#"{"a": 1,}"#, ResponseShape::Json));
+    }
+
+    #[test]
+    fn structural_complete_ignores_braces_inside_strings() {
+        assert!(is_structurally_complete(r#"{"s": "has } inside"}"#, ResponseShape::Json));
+        assert!(is_structurally_complete(r#"{"s": "escaped \" quote"}"#, ResponseShape::Json));
+    }
+
+    #[tokio::test]
+    async fn continuable_returns_first_response_when_complete() {
+        let generator = ScriptedGenerator::new(vec![Ok(r#"{"ok": true}"#.to_string())]);
+        let opts = ContinuableOpts::new("qwen3:latest");
+        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
+        assert!(out.final_complete);
+        assert_eq!(out.empty_retries, 0);
+        assert_eq!(out.continuations, 0);
+        assert_eq!(generator.call_count(), 1);
+    }
+
+    #[tokio::test]
+    async fn continuable_retries_on_empty_with_doubled_budget() {
+        // Two empties, then a good response. Third call should see 4×
+        // the initial budget (2× twice).
+        let generator = ScriptedGenerator::new(vec![
+            Ok("".to_string()),
+            Ok("".to_string()),
+            Ok(r#"{"ok": true}"#.to_string()),
+        ]);
+        let mut opts = ContinuableOpts::new("qwen3:latest");
+        opts.max_tokens = Some(100);
+        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
+        assert!(out.final_complete);
+        assert_eq!(out.empty_retries, 2);
+
+        let calls = generator.calls();
+        assert_eq!(calls.len(), 3);
+        assert_eq!(calls[0].max_tokens, Some(100));
+        assert_eq!(calls[1].max_tokens, Some(200));
+        assert_eq!(calls[2].max_tokens, Some(400));
+    }
+
+    #[tokio::test]
+    async fn continuable_caps_budget_at_budget_cap() {
+        let generator = ScriptedGenerator::new(vec![
+            Ok("".to_string()),
+            Ok("".to_string()),
+            Ok(r#"{"ok":1}"#.to_string()),
+        ]);
+        let mut opts = ContinuableOpts::new("qwen3:latest");
+        opts.max_tokens = Some(5_000);
+        opts.budget_cap = 7_000;
+        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
+        assert!(out.final_complete);
+        let calls = generator.calls();
+        // 5000 → doubled would be 10000; cap pulls it to 7000.
+        assert_eq!(calls[1].max_tokens, Some(7_000));
+        assert_eq!(calls[2].max_tokens, Some(7_000));
+    }
+
+    #[tokio::test]
+    async fn continuable_glues_truncated_response() {
+        // First call returns balanced-open `{...`; continuation closes
+        // it with `...}`. Combined must parse.
+        let generator = ScriptedGenerator::new(vec![
+            Ok(r#"{"fills": [{"candidate_id": "C-001""#.to_string()),
+            Ok(r#", "name": "Alice"}]}"#.to_string()),
+        ]);
+        let opts = ContinuableOpts::new("qwen3:latest");
+        let out = generate_continuable(&generator, "ORIGINAL", &opts).await.unwrap();
+        assert!(out.final_complete, "combined must parse: {}", out.text);
+        assert_eq!(out.continuations, 1);
+
+        let calls = generator.calls();
+        assert_eq!(calls.len(), 2);
+        // Continuation prompt must contain the partial — that's the
+        // scratchpad primitive J called out.
+        let cont_prompt = &calls[1].prompt;
+        assert!(cont_prompt.contains("ORIGINAL"),
+            "continuation must include original prompt");
+        assert!(cont_prompt.contains(r#"{"fills": [{"candidate_id": "C-001""#),
+            "continuation must include partial");
+    }
+
+    #[tokio::test]
+    async fn continuable_does_not_loop_on_persistent_empty() {
+        // All three retries return empty; we must NOT then enter the
+        // continuation loop with an empty partial (would burn 3 more
+        // calls continuing from "").
+        let generator = ScriptedGenerator::new(vec![
+            Ok("".to_string()),
+            Ok("".to_string()),
+            Ok("".to_string()),
+        ]);
+        let opts = ContinuableOpts::new("qwen3:latest");
+        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
+        assert!(!out.final_complete);
+        assert_eq!(out.empty_retries, 3);
+        assert_eq!(out.continuations, 0, "must not continue from empty");
+        assert_eq!(generator.call_count(), 3);
+    }
+
+    #[tokio::test]
+    async fn continuable_returns_raw_on_exhausted_continuations() {
+        // Three continuations that never complete — caller's parser
+        // will throw. We must return the combined text so forensics
+        // has the raw content, not a lossy truncation.
+        let generator = ScriptedGenerator::new(vec![
+            Ok(r#"{"a": ["#.to_string()),
+            Ok("1,".to_string()),
+            Ok("2,".to_string()),
+            Ok("3,".to_string()),
+        ]);
+        let mut opts = ContinuableOpts::new("qwen3:latest");
+        opts.max_continuations = 3;
+        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
+        assert!(!out.final_complete);
+        assert_eq!(out.continuations, 3);
+        assert!(out.text.contains(r#"{"a": ["#));
+        assert!(out.text.contains("3,"));
+    }
+
+    #[tokio::test]
+    async fn continuable_propagates_generator_errors() {
+        let generator = ScriptedGenerator::new(vec![Err("sidecar 503".to_string())]);
+        let opts = ContinuableOpts::new("qwen3:latest");
+        let err = generate_continuable(&generator, "test", &opts).await.unwrap_err();
+        assert!(err.contains("503"));
+    }
+}

crates/aibridge/src/lib.rs

@@ -1,2 +1,5 @@
 pub mod client;
+pub mod context;
+pub mod continuation;
 pub mod service;
+pub mod tree_split;

crates/aibridge/src/tree_split.rs

@@ -0,0 +1,326 @@
+//! Phase 21 — INPUT-overflow handler. Ports `generateTreeSplit` from
+//! `tests/multi-agent/agent.ts`.
+//!
+//! When the input corpus exceeds the model's window (200 playbooks
+//! pasted into a T4 strategic prompt, a long retrospective digest, a
+//! cross-corpus summarization), raising `max_tokens` doesn't help —
+//! the prompt itself is the problem. The answer is map-reduce:
+//!
+//!   1. Caller shards the input at semantic boundaries (records,
+//!      paragraphs, playbook entries).
+//!   2. For each shard, build a map prompt that includes the running
+//!      scratchpad and run it through `generate_continuable`.
+//!   3. Append the map output to the scratchpad (oldest-first
+//!      truncation when it outgrows `scratchpad_budget`).
+//!   4. Build a reduce prompt from the final scratchpad and run it.
+//!
+//! Every shard prompt and the reduce prompt go through
+//! `assert_context_budget` first — if a single shard still overflows
+//! we bubble the error up rather than silently truncating. That's the
+//! whole point of Phase 21.
+
+use crate::context::{assert_context_budget, BudgetOpts, estimate_tokens, overflow_message,
+                     DEFAULT_MAX_TOKENS, DEFAULT_SAFETY_MARGIN};
+use crate::continuation::{generate_continuable, ContinuableOpts, ResponseShape, TextGenerator};
+
+/// Callback signatures — caller supplies closures that stitch the
+/// scratchpad into each shard's prompt and build the final reduce
+/// prompt. Kept as `Fn` (not `FnMut`) so the map loop can call them
+/// by reference.
+pub type MapPromptFn<'a> = dyn Fn(&str, &str) -> String + Send + Sync + 'a;
+pub type ReducePromptFn<'a> = dyn Fn(&str) -> String + Send + Sync + 'a;
+
+/// Knobs for `generate_tree_split`.
+#[derive(Debug, Clone)]
+pub struct TreeSplitOpts {
+    pub model: String,
+    pub system: Option<String>,
+    pub temperature: Option<f64>,
+    /// max_tokens for map AND reduce (reduce defaults are usually
+    /// higher; caller overrides for just reduce by calling through
+    /// continuable directly if needed).
+    pub max_tokens: Option<u32>,
+    pub reduce_max_tokens: Option<u32>,
+    pub think: Option<bool>,
+    /// Soft ceiling on scratchpad size (estimated tokens). When it
+    /// grows past this, the oldest shard digest gets dropped. Default
+    /// 6000, matching the TS implementation.
+    pub scratchpad_budget: usize,
+    pub safety_margin: Option<usize>,
+}
+
+impl TreeSplitOpts {
+    pub fn new(model: impl Into<String>) -> Self {
+        Self {
+            model: model.into(),
+            system: None,
+            temperature: None,
+            max_tokens: None,
+            reduce_max_tokens: None,
+            think: None,
+            scratchpad_budget: 6_000,
+            safety_margin: None,
+        }
+    }
+}
+
+/// Result — final reduce response plus the accumulated scratchpad so
+/// the caller can inspect what was kept vs truncated.
+#[derive(Debug, Clone)]
+pub struct TreeSplitResult {
+    pub response: String,
+    pub scratchpad: String,
+    pub shards_processed: usize,
+    pub scratchpad_truncations: usize,
+    pub total_continuations: usize,
+}
+
+/// Drop shard-digest blocks from the head of `scratchpad` until its
+/// estimated-token count fits the budget. Digest blocks are delimited
+/// by `\n— shard N/M digest —\n` so we can find the first one and
+/// chop everything before its successor.
+fn truncate_scratchpad(scratchpad: &mut String, budget_tokens: usize) -> bool {
+    if estimate_tokens(scratchpad) <= budget_tokens { return false; }
+    // Find the second delimiter — everything before it gets dropped.
+    const DELIM_PREFIX: &str = "\n— shard ";
+    let mut cursor = 0;
+    let mut truncated = false;
+    while estimate_tokens(&scratchpad[cursor..]) > budget_tokens {
+        // Skip past a leading delimiter (if we're sitting on one from
+        // a previous iteration), then find the next.
+        let search_from = cursor + if scratchpad[cursor..].starts_with(DELIM_PREFIX) {
+            DELIM_PREFIX.len()
+        } else { 0 };
+        let Some(rel_next) = scratchpad[search_from..].find(DELIM_PREFIX) else { break };
+        cursor = search_from + rel_next;
+        truncated = true;
+    }
+    if cursor > 0 {
+        scratchpad.drain(..cursor);
+    }
+    truncated
+}
+
+/// Phase 21 — map-reduce over shards with a running scratchpad. See
+/// module docs.
+pub async fn generate_tree_split<G: TextGenerator>(
+    generator: &G,
+    shards: &[String],
+    map_prompt: &MapPromptFn<'_>,
+    reduce_prompt: &ReducePromptFn<'_>,
+    opts: &TreeSplitOpts,
+) -> Result<TreeSplitResult, String> {
+    let mut scratchpad = String::new();
+    let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
+    let map_max = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS as u32);
+    let reduce_max = opts.reduce_max_tokens.unwrap_or(1_500);
+    let mut truncations = 0usize;
+    let mut total_continuations = 0usize;
+
+    for (i, shard) in shards.iter().enumerate() {
+        let shard_prompt = map_prompt(shard, &scratchpad);
+        // Loud-fail on per-shard overflow — caller sharded too
+        // coarsely. Silent truncation is exactly the mode J rejected.
+        let budget = BudgetOpts {
+            system: opts.system.as_deref(),
+            max_tokens: Some(map_max as usize),
+            safety_margin: Some(safety),
+            bypass: false,
+        };
+        let check = assert_context_budget(&opts.model, &shard_prompt, budget)
+            .map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
+        let _ = check;
+
+        let mut cont_opts = ContinuableOpts::new(&opts.model);
+        cont_opts.max_tokens = Some(map_max);
+        cont_opts.temperature = opts.temperature;
+        cont_opts.system = opts.system.clone();
+        cont_opts.shape = ResponseShape::Text;
+        cont_opts.think = opts.think;
+
+        let outcome = generate_continuable(generator, &shard_prompt, &cont_opts).await?;
+        total_continuations += outcome.continuations;
+
+        // Append this shard's digest and, if needed, drop oldest.
+        scratchpad.push_str(&format!(
+            "\n— shard {}/{} digest —\n{}",
+            i + 1, shards.len(), outcome.text.trim(),
+        ));
+        if truncate_scratchpad(&mut scratchpad, opts.scratchpad_budget) {
+            truncations += 1;
+        }
+    }
+
+    // Reduce pass. Budget check first — if the scratchpad is still too
+    // big for the reduce prompt we fail loud with numbers.
+    let reduce_p = reduce_prompt(&scratchpad);
+    let budget = BudgetOpts {
+        system: opts.system.as_deref(),
+        max_tokens: Some(reduce_max as usize),
+        safety_margin: Some(safety),
+        bypass: false,
+    };
+    assert_context_budget(&opts.model, &reduce_p, budget)
+        .map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
+
+    let mut cont_opts = ContinuableOpts::new(&opts.model);
+    cont_opts.max_tokens = Some(reduce_max);
+    cont_opts.temperature = opts.temperature;
+    cont_opts.system = opts.system.clone();
+    cont_opts.shape = ResponseShape::Text;
+    cont_opts.think = opts.think;
+
+    let outcome = generate_continuable(generator, &reduce_p, &cont_opts).await?;
+    total_continuations += outcome.continuations;
+
+    Ok(TreeSplitResult {
+        response: outcome.text,
+        scratchpad,
+        shards_processed: shards.len(),
+        scratchpad_truncations: truncations,
+        total_continuations,
+    })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::continuation::ScriptedGenerator;
+
+    fn simple_map(shard: &str, scratchpad: &str) -> String {
+        format!("SCRATCHPAD:\n{scratchpad}\n---\nSHARD:\n{shard}\n---\nDIGEST:")
+    }
+
+    fn simple_reduce(scratchpad: &str) -> String {
+        format!("SCRATCHPAD:\n{scratchpad}\n---\nFINAL:")
+    }
+
+    #[tokio::test]
+    async fn tree_split_runs_map_then_reduce() {
+        // 3 shards → 3 map calls → 1 reduce call = 4 responses scripted.
+        let generator = ScriptedGenerator::new(vec![
+            Ok("digest-1".to_string()),
+            Ok("digest-2".to_string()),
+            Ok("digest-3".to_string()),
+            Ok("FINAL ANSWER".to_string()),
+        ]);
+        let shards: Vec<String> = ["a", "b", "c"].iter().map(|s| s.to_string()).collect();
+        let opts = TreeSplitOpts::new("qwen3:latest");
+        let map_fn: &MapPromptFn = &simple_map;
+        let reduce_fn: &ReducePromptFn = &simple_reduce;
+        let result = generate_tree_split(&generator, &shards, map_fn, reduce_fn, &opts)
+            .await
+            .unwrap();
+        assert_eq!(result.shards_processed, 3);
+        assert_eq!(result.response, "FINAL ANSWER");
+        assert_eq!(generator.call_count(), 4);
+        // Scratchpad must carry all three digests in order.
+        assert!(result.scratchpad.contains("digest-1"));
+        assert!(result.scratchpad.contains("digest-2"));
+        assert!(result.scratchpad.contains("digest-3"));
+    }
+
+    #[tokio::test]
+    async fn tree_split_reduce_prompt_sees_full_scratchpad() {
+        let generator = ScriptedGenerator::new(vec![
+            Ok("summary-A".to_string()),
+            Ok("summary-B".to_string()),
+            Ok("REDUCED".to_string()),
+        ]);
+        let shards = vec!["input-one".to_string(), "input-two".to_string()];
+        let opts = TreeSplitOpts::new("qwen3:latest");
+        let _ = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
+            .await
+            .unwrap();
+        // Third call = reduce. Its prompt must include both digests.
+        let calls = generator.calls();
+        let reduce_prompt = &calls[2].prompt;
+        assert!(reduce_prompt.contains("summary-A"),
+            "reduce prompt must see first shard digest");
+        assert!(reduce_prompt.contains("summary-B"),
+            "reduce prompt must see second shard digest");
+    }
+
+    #[tokio::test]
+    async fn tree_split_loud_fails_on_shard_overflow() {
+        let generator = ScriptedGenerator::new(vec![Ok("digest".to_string())]);
+        // One gigantic shard — well over qwen3's 40K window even as a
+        // prompt. The budget check must reject before any generate call.
+        let shards = vec!["x".repeat(200_000)];
+        let opts = TreeSplitOpts::new("qwen3:latest");
+        let err = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
+            .await
+            .expect_err("shard-sized overflow must be rejected");
+        assert!(err.contains("overflow"), "error should mention overflow: {err}");
+        assert_eq!(generator.call_count(), 0, "generate must not be called on overflow");
+    }
+
+    #[tokio::test]
+    async fn tree_split_truncates_scratchpad_when_over_budget() {
+        // Tight budget so each shard trips truncation. qwen3's 40K
+        // window is fine; the budget we care about is the scratchpad
+        // cap, not the model window.
+        let generator = ScriptedGenerator::new(vec![
+            Ok("A".repeat(2_000)),
+            Ok("B".repeat(2_000)),
+            Ok("C".repeat(2_000)),
+            Ok("D".repeat(2_000)),
+            Ok("FINAL".to_string()),
+        ]);
+        let shards: Vec<String> = (0..4).map(|i| format!("shard{i}")).collect();
+        let mut opts = TreeSplitOpts::new("qwen3:latest");
+        opts.scratchpad_budget = 1_000; // ~4000 chars — one digest barely fits
+        let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
+            .await
+            .unwrap();
+        assert!(result.scratchpad_truncations > 0,
+            "tight budget must trigger truncation");
+        // Scratchpad should still fit roughly within the budget
+        // (post-truncation); the estimator uses chars/4 so the bound
+        // is ~budget*4 chars. Give some slack for the delimiter.
+        let scratchpad_tokens = estimate_tokens(&result.scratchpad);
+        assert!(scratchpad_tokens <= opts.scratchpad_budget * 2,
+            "scratchpad {} tokens vs budget {}", scratchpad_tokens, opts.scratchpad_budget);
+    }
+
+    #[tokio::test]
+    async fn tree_split_reports_continuations_from_map_and_reduce() {
+        // First shard: truncated-then-continued. Reduce: truncated-then-continued.
+        // 1 shard: 2 map calls (initial + continuation), then 2 reduce calls.
+        let generator = ScriptedGenerator::new(vec![
+            Ok("partial".to_string()),         // map shape=text, non-empty → complete on first pass
+            Ok("reduce-out".to_string()),
+        ]);
+        let shards = vec!["only".to_string()];
+        let opts = TreeSplitOpts::new("qwen3:latest");
+        let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
+            .await
+            .unwrap();
+        // Text shape treats non-empty as complete → 0 continuations.
+        assert_eq!(result.total_continuations, 0);
+        assert_eq!(result.shards_processed, 1);
+    }
+
+    #[test]
+    fn truncate_scratchpad_noop_when_under_budget() {
+        let mut s = "\n— shard 1/1 digest —\nshort".to_string();
+        let truncated = truncate_scratchpad(&mut s, 1_000);
+        assert!(!truncated);
+        assert!(s.contains("short"));
+    }
+
+    #[test]
+    fn truncate_scratchpad_drops_oldest_first() {
+        let mut s = format!(
+            "\n— shard 1/3 digest —\n{}\n— shard 2/3 digest —\n{}\n— shard 3/3 digest —\nshort",
+            "x".repeat(4_000), // ~1000 tokens
+            "y".repeat(4_000), // ~1000 tokens
+        );
+        let truncated = truncate_scratchpad(&mut s, 500); // ~2000 chars
+        assert!(truncated);
+        assert!(!s.contains(&"x".repeat(4_000)),
+            "oldest digest should be dropped");
+        assert!(s.contains("short"),
+            "newest digest should survive");
+    }
+}

crates/catalogd/src/grpc.rs

@@ -54,7 +54,16 @@ impl CatalogService for CatalogGrpc {
         let manifest = self.registry
             .register(req.name, SchemaFingerprint(req.schema_fingerprint), objects)
             .await
-            .map_err(|e| Status::internal(e))?;
+            .map_err(|e| {
+                // Mirror the HTTP side's 409 mapping: schema drift is a
+                // precondition failure, not a server error. Keeps gRPC
+                // and HTTP callers seeing the same diagnostic signal.
+                if e.contains("different schema") {
+                    Status::failed_precondition(e)
+                } else {
+                    Status::internal(e)
+                }
+            })?;
 
         Ok(Response::new(manifest_to_proto(&manifest)))
     }

crates/catalogd/src/registry.rs

@@ -28,6 +28,27 @@ pub struct MigrateBucketsReport {
     pub manifests_persisted: usize,
 }
 
+#[derive(Debug, Clone, Default, serde::Serialize)]
+pub struct DedupeReport {
+    pub groups: usize,
+    pub removed: usize,
+    pub kept: Vec<DedupeKept>,
+    pub errors: Vec<DedupeError>,
+}
+
+#[derive(Debug, Clone, serde::Serialize)]
+pub struct DedupeKept {
+    pub name: String,
+    pub kept_id: String,
+    pub removed: usize,
+}
+
+#[derive(Debug, Clone, serde::Serialize)]
+pub struct DedupeError {
+    pub manifest_id: String,
+    pub error: String,
+}
+
 /// Partial metadata update — only set fields are applied.
 #[derive(Debug, Clone, Default, serde::Deserialize)]
 pub struct MetadataUpdate {
@@ -212,13 +233,65 @@ impl Registry {
         Ok(())
     }
 
-    /// Register a new dataset. Persists manifest to storage before updating memory.
+    /// Register a dataset. Idempotent on `name`:
+    ///   - No existing dataset by this name → create new manifest.
+    ///   - Exists with same schema fingerprint → update objects + `updated_at`
+    ///     in place (re-ingest of identical-shape data).
+    ///   - Exists with different schema fingerprint → reject as `Err`
+    ///     (callers map to HTTP 409 / gRPC FAILED_PRECONDITION). Schema
+    ///     evolution is not yet exposed as an HTTP endpoint; callers must
+    ///     register under a new name or delete the existing manifest first.
+    ///
+    /// Concurrency: the write lock is held across the storage write. That
+    /// serializes concurrent registers but is correctness-mandatory —
+    /// dropping the lock across the check→insert sequence creates a TOCTOU
+    /// window where two callers with the same name both see "no existing"
+    /// and both insert, reintroducing the duplicate-manifest bug this
+    /// function exists to prevent. Register is a metadata-only hop on the
+    /// ingest path, so the serialization cost is acceptable. Precedent:
+    /// `update_metadata` also holds the write lock across its I/O.
+    ///
+    /// Legacy state: if multiple manifests with the same name already exist
+    /// (from before this was idempotent), the most recently updated one is
+    /// treated as canonical. Run `dedupe_by_name` to clean up the rest.
     pub async fn register(
         &self,
         name: String,
         schema_fingerprint: SchemaFingerprint,
         objects: Vec<ObjectRef>,
     ) -> Result<DatasetManifest, String> {
+        let mut datasets = self.datasets.write().await;
+
+        let existing = datasets
+            .values()
+            .filter(|d| d.name == name)
+            .max_by_key(|d| d.updated_at)
+            .cloned();
+
+        if let Some(mut manifest) = existing {
+            if manifest.schema_fingerprint != schema_fingerprint {
+                return Err(format!(
+                    "dataset '{}' already exists with a different schema \
+                     (existing fingerprint: {}, new: {}). Schema-evolution \
+                     migration is not yet exposed as an HTTP endpoint; \
+                     register under a new name or delete the existing \
+                     manifest first.",
+                    name, manifest.schema_fingerprint.0, schema_fingerprint.0,
+                ));
+            }
+            manifest.objects = objects;
+            manifest.updated_at = chrono::Utc::now();
+
+            let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
+            let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
+            ops::put(&self.store, &manifest_key, json.into()).await?;
+
+            datasets.insert(manifest.id.clone(), manifest.clone());
+
+            tracing::info!("re-registered (idempotent): {} ({})", manifest.name, manifest.id);
+            return Ok(manifest);
+        }
+
         let now = chrono::Utc::now();
         let manifest = DatasetManifest {
             id: DatasetId::new(),
@@ -240,12 +313,10 @@ impl Registry {
             embedding_refresh_policy: None,
         };
 
-        // Write-ahead: persist before in-memory update
         let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
         let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
         ops::put(&self.store, &manifest_key, json.into()).await?;
 
-        let mut datasets = self.datasets.write().await;
         datasets.insert(manifest.id.clone(), manifest.clone());
 
         tracing::info!("registered dataset: {} ({})", manifest.name, manifest.id);
@@ -380,6 +451,72 @@ impl Registry {
         Ok(manifest.clone())
     }
 
+    /// Collapse duplicate manifests that share a `name`. Winner per group:
+    ///   1. Prefer a manifest with a non-null `row_count` (already resynced).
+    ///   2. Break ties by newest `updated_at`.
+    /// Losers are removed from the in-memory registry AND their manifest
+    /// JSON is deleted from object storage. Datasets with a single manifest
+    /// are untouched. Parquet data files are never touched — only catalog
+    /// metadata. See `register` for the prevention side of this fix.
+    ///
+    /// Operator-only. Do not run while ingest is active: a concurrent
+    /// `register` between the snapshot and the delete sweep can create a
+    /// new manifest under a name we just deduped, producing a transient
+    /// count of 2 until the next dedupe run. Safe outside ingest windows.
+    pub async fn dedupe_by_name(&self) -> DedupeReport {
+        let groups: Vec<(String, Vec<DatasetManifest>)> = {
+            let datasets = self.datasets.read().await;
+            let mut by_name: HashMap<String, Vec<DatasetManifest>> = HashMap::new();
+            for m in datasets.values() {
+                by_name.entry(m.name.clone()).or_default().push(m.clone());
+            }
+            by_name.into_iter().filter(|(_, v)| v.len() > 1).collect()
+        };
+
+        let mut report = DedupeReport::default();
+        let mut to_delete: Vec<DatasetId> = Vec::new();
+
+        for (name, mut manifests) in groups {
+            report.groups += 1;
+            manifests.sort_by(|a, b| {
+                b.row_count.is_some().cmp(&a.row_count.is_some())
+                    .then(b.updated_at.cmp(&a.updated_at))
+            });
+            let winner = &manifests[0];
+            report.kept.push(DedupeKept {
+                name: name.clone(),
+                kept_id: winner.id.to_string(),
+                removed: manifests.len() - 1,
+            });
+            for loser in &manifests[1..] {
+                to_delete.push(loser.id.clone());
+            }
+        }
+
+        {
+            let mut datasets = self.datasets.write().await;
+            for id in &to_delete {
+                datasets.remove(id);
+            }
+        }
+        for id in &to_delete {
+            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
+            match ops::delete(&self.store, &key).await {
+                Ok(_) => report.removed += 1,
+                Err(e) => report.errors.push(DedupeError {
+                    manifest_id: id.to_string(),
+                    error: e,
+                }),
+            }
+        }
+
+        tracing::info!(
+            "dedupe: {} groups collapsed, {} manifests removed, {} errors",
+            report.groups, report.removed, report.errors.len(),
+        );
+        report
+    }
+
     /// Resync every dataset that currently has a null row_count.
     /// Returns (successes, failures) where each entry is (name, detail).
     pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) {
@@ -549,6 +686,46 @@ impl Registry {
         Ok(())
     }
 
+    /// Remove a dataset from the catalog by name. Deletes the manifest
+    /// from both the in-memory registry and object storage.
+    ///
+    /// Scope is metadata-only: the underlying parquet files, vector
+    /// indexes, tombstones, trial journals, and AiViews that reference
+    /// this dataset are NOT touched. Caller is responsible for any
+    /// cascade cleanup. This mirrors how a DROP TABLE in a typical
+    /// warehouse separates "forget about this dataset" from "actually
+    /// reclaim the bytes".
+    ///
+    /// Returns Ok(()) when the dataset existed and was removed; an Err
+    /// with "dataset not found" when no manifest by that name exists.
+    /// Legacy state with duplicate names: all matching manifests are
+    /// removed (effectively a post-hoc dedupe on that name).
+    pub async fn delete_dataset(&self, name: &str) -> Result<usize, String> {
+        let ids_to_remove: Vec<DatasetId> = {
+            let datasets = self.datasets.read().await;
+            datasets.values().filter(|d| d.name == name).map(|d| d.id.clone()).collect()
+        };
+        if ids_to_remove.is_empty() {
+            return Err(format!("dataset not found: {name}"));
+        }
+
+        for id in &ids_to_remove {
+            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
+            if let Err(e) = ops::delete(&self.store, &key).await {
+                // Storage delete failed — log and keep going. The in-memory
+                // remove below is still correct; on restart, the missing
+                // storage object just means nothing to rehydrate.
+                tracing::warn!("delete_dataset '{name}': storage delete of {key} failed: {e}");
+            }
+        }
+        let mut datasets = self.datasets.write().await;
+        for id in &ids_to_remove {
+            datasets.remove(id);
+        }
+        tracing::info!("deleted dataset '{}' ({} manifest(s))", name, ids_to_remove.len());
+        Ok(ids_to_remove.len())
+    }
+
     /// List datasets whose `embedding_stale_since` is set — they need a refresh.
     pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
         let datasets = self.datasets.read().await;
@@ -579,3 +756,275 @@ impl Registry {
         Ok(manifest.clone())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use object_store::memory::InMemory;
+
+    fn fixture() -> Registry {
+        Registry::new(Arc::new(InMemory::new()))
+    }
+
+    fn fp(s: &str) -> SchemaFingerprint {
+        SchemaFingerprint(s.to_string())
+    }
+
+    fn obj(key: &str) -> ObjectRef {
+        ObjectRef {
+            bucket: "primary".to_string(),
+            key: key.to_string(),
+            size_bytes: 100,
+            created_at: chrono::Utc::now(),
+        }
+    }
+
+    #[tokio::test]
+    async fn fresh_register_creates_new_manifest() {
+        let reg = fixture();
+        let m = reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
+        assert_eq!(m.name, "a");
+        assert_eq!(m.schema_fingerprint, fp("f1"));
+        assert_eq!(reg.list().await.len(), 1);
+        assert_eq!(reg.get_by_name("a").await.unwrap().id, m.id);
+    }
+
+    #[tokio::test]
+    async fn re_register_same_fingerprint_is_idempotent() {
+        let reg = fixture();
+        let first = reg.register("a".into(), fp("f1"), vec![obj("old.parquet")]).await.unwrap();
+        let second = reg.register("a".into(), fp("f1"), vec![obj("new.parquet")]).await.unwrap();
+
+        assert_eq!(first.id, second.id, "same name+fp should reuse ID");
+        assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
+        let fetched = reg.get_by_name("a").await.unwrap();
+        assert_eq!(fetched.objects[0].key, "new.parquet", "objects replaced");
+        assert!(fetched.updated_at >= first.updated_at, "updated_at bumped");
+    }
+
+    #[tokio::test]
+    async fn re_register_different_fingerprint_rejects() {
+        let reg = fixture();
+        reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
+        let err = reg.register("a".into(), fp("f2"), vec![obj("a.parquet")]).await.unwrap_err();
+
+        assert!(err.contains("different schema"), "error mentions schema drift: {err}");
+        assert_eq!(reg.list().await.len(), 1, "failed register did not mutate state");
+        assert_eq!(reg.get_by_name("a").await.unwrap().schema_fingerprint, fp("f1"));
+    }
+
+    #[tokio::test]
+    async fn delete_dataset_removes_manifest_and_in_memory_entry() {
+        let reg = fixture();
+        reg.register("to_delete".into(), fp("f1"), vec![obj("d.parquet")]).await.unwrap();
+        reg.register("keepme".into(),    fp("f2"), vec![obj("k.parquet")]).await.unwrap();
+        assert_eq!(reg.list().await.len(), 2);
+
+        let removed = reg.delete_dataset("to_delete").await.unwrap();
+        assert_eq!(removed, 1, "one manifest removed");
+        assert!(reg.get_by_name("to_delete").await.is_none());
+        assert!(reg.get_by_name("keepme").await.is_some(), "unrelated dataset untouched");
+        assert_eq!(reg.list().await.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn delete_dataset_404s_on_unknown_name() {
+        let reg = fixture();
+        let err = reg.delete_dataset("nope").await.unwrap_err();
+        assert!(err.starts_with("dataset not found"), "error wording: {err}");
+    }
+
+    #[tokio::test]
+    async fn delete_dataset_removes_all_dupes_with_matching_name() {
+        // Legacy zombies: direct insert so we can seed duplicates,
+        // mirroring pre-idempotent-register state. delete_dataset should
+        // sweep every manifest sharing that name in a single call.
+        let reg = fixture();
+        let now = chrono::Utc::now();
+        for _ in 0..3 {
+            let m = DatasetManifest {
+                id: DatasetId::new(),
+                name: "zombie".into(),
+                schema_fingerprint: fp("fp"),
+                objects: vec![obj("z.parquet")],
+                created_at: now, updated_at: now,
+                row_count: None, description: String::new(), owner: String::new(),
+                sensitivity: None, columns: vec![], lineage: None, freshness: None,
+                tags: vec![], last_embedded_at: None, embedding_stale_since: None,
+                embedding_refresh_policy: None,
+            };
+            reg.datasets.write().await.insert(m.id.clone(), m);
+        }
+        assert_eq!(reg.list().await.len(), 3);
+
+        let removed = reg.delete_dataset("zombie").await.unwrap();
+        assert_eq!(removed, 3, "all three manifests removed in one sweep");
+        assert_eq!(reg.list().await.len(), 0);
+    }
+
+    #[tokio::test]
+    async fn dedupe_no_duplicates_is_noop() {
+        let reg = fixture();
+        reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
+        reg.register("b".into(), fp("f2"), vec![obj("b.parquet")]).await.unwrap();
+
+        let report = reg.dedupe_by_name().await;
+        assert_eq!(report.groups, 0);
+        assert_eq!(report.removed, 0);
+        assert_eq!(reg.list().await.len(), 2);
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 8)]
+    async fn concurrent_register_stress_many_workers_one_manifest() {
+        // Stress variant: fire 32 concurrent registers for the same new
+        // name across 8 worker threads. Single-manifest invariant must
+        // hold regardless of how the scheduler interleaves them. This
+        // catches race regressions the 2-call variant below might miss
+        // under a forgiving scheduler.
+        let reg = fixture();
+        let mut handles = Vec::new();
+        for i in 0..32 {
+            let r = reg.clone();
+            handles.push(tokio::spawn(async move {
+                r.register(
+                    "stress".into(),
+                    fp("stress-fp"),
+                    vec![obj(&format!("{i}.parquet"))],
+                ).await
+            }));
+        }
+        for h in handles {
+            h.await.unwrap().expect("each register succeeds (idempotent)");
+        }
+        let all = reg.list().await;
+        let stress_manifests: Vec<_> = all.iter().filter(|d| d.name == "stress").collect();
+        assert_eq!(stress_manifests.len(), 1,
+            "32 concurrent registers produced {} manifests for 'stress' — expected 1",
+            stress_manifests.len());
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    async fn concurrent_register_same_new_name_collapses_to_one_manifest() {
+        // Pins the TOCTOU fix: without the write-lock held across the
+        // check→insert sequence, two parallel registers of the same
+        // previously-unknown name would both see "no existing" and both
+        // insert. Multi-thread flavor is required to actually exercise
+        // the race — under single-threaded tokio the scheduler doesn't
+        // preempt between awaits on the same task's critical section.
+        // Under the current impl, whichever acquires the write lock
+        // first creates the manifest; the second observes it and takes
+        // the idempotent-update branch.
+        let reg = fixture();
+        let a = reg.clone();
+        let b = reg.clone();
+
+        let (ra, rb) = tokio::join!(
+            async move {
+                a.register("race".into(), fp("shared-fp"), vec![obj("a.parquet")]).await
+            },
+            async move {
+                b.register("race".into(), fp("shared-fp"), vec![obj("b.parquet")]).await
+            },
+        );
+
+        let m_a = ra.expect("first register succeeds");
+        let m_b = rb.expect("second register succeeds (same fp = idempotent)");
+        assert_eq!(m_a.id, m_b.id, "both calls resolve to a single DatasetId");
+        assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    async fn concurrent_register_same_name_different_fp_one_wins_one_rejects() {
+        // Reverse case: concurrent registers with conflicting fingerprints.
+        // Deterministic outcome: one call's fingerprint gets persisted as
+        // canonical, the other is rejected on seeing the now-established
+        // fingerprint. Which one wins is scheduler-dependent — we only
+        // assert exactly one success and exactly one rejection.
+        let reg = fixture();
+        let a = reg.clone();
+        let b = reg.clone();
+
+        let (ra, rb) = tokio::join!(
+            async move { a.register("race".into(), fp("fp1"), vec![obj("a.parquet")]).await },
+            async move { b.register("race".into(), fp("fp2"), vec![obj("b.parquet")]).await },
+        );
+
+        let successes = [&ra, &rb].iter().filter(|r| r.is_ok()).count();
+        let rejections = [&ra, &rb].iter().filter(|r| r.is_err()).count();
+        assert_eq!(successes, 1, "exactly one register wins");
+        assert_eq!(rejections, 1, "the other is rejected on fingerprint");
+        let rejection_msg = [&ra, &rb].iter()
+            .find_map(|r| r.as_ref().err())
+            .unwrap();
+        assert!(rejection_msg.contains("different schema"));
+        assert_eq!(reg.list().await.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn dedupe_collapses_dupes_preferring_non_null_row_count() {
+        let reg = fixture();
+
+        // Simulate legacy pre-idempotent-register state by inserting
+        // manifests with the same `name` directly. Mirrors what happened
+        // with the 308× successful_playbooks duplication.
+        let now = chrono::Utc::now();
+        let old_with_rows = DatasetManifest {
+            id: DatasetId::new(),
+            name: "dupes".into(),
+            schema_fingerprint: fp("f1"),
+            objects: vec![obj("dupes.parquet")],
+            created_at: now - chrono::Duration::hours(2),
+            updated_at: now - chrono::Duration::hours(2),
+            row_count: Some(42),
+            description: String::new(),
+            owner: String::new(),
+            sensitivity: None,
+            columns: vec![],
+            lineage: None,
+            freshness: None,
+            tags: vec![],
+            last_embedded_at: None,
+            embedding_stale_since: None,
+            embedding_refresh_policy: None,
+        };
+        let newer_no_rows = DatasetManifest {
+            id: DatasetId::new(),
+            updated_at: now, // newer, but null row_count
+            row_count: None,
+            ..old_with_rows.clone()
+        };
+        let oldest = DatasetManifest {
+            id: DatasetId::new(),
+            updated_at: now - chrono::Duration::hours(5),
+            row_count: None,
+            ..old_with_rows.clone()
+        };
+
+        let kept_id = old_with_rows.id.clone();
+        let loser_a = newer_no_rows.id.clone();
+        let loser_b = oldest.id.clone();
+
+        {
+            let mut d = reg.datasets.write().await;
+            d.insert(old_with_rows.id.clone(), old_with_rows);
+            d.insert(newer_no_rows.id.clone(), newer_no_rows);
+            d.insert(oldest.id.clone(), oldest);
+        }
+        assert_eq!(reg.list().await.len(), 3);
+
+        let report = reg.dedupe_by_name().await;
+        assert_eq!(report.groups, 1);
+        assert_eq!(report.removed, 2);
+        assert_eq!(report.errors.len(), 0);
+        assert_eq!(report.kept.len(), 1);
+        assert_eq!(report.kept[0].name, "dupes");
+        assert_eq!(report.kept[0].kept_id, kept_id.to_string(),
+            "should keep the manifest with non-null row_count");
+
+        let remaining = reg.list().await;
+        assert_eq!(remaining.len(), 1);
+        assert_eq!(remaining[0].id, kept_id);
+        assert!(reg.get(&loser_a).await.is_none());
+        assert!(reg.get(&loser_b).await.is_none());
+    }
+}

crates/catalogd/src/service.rs

@@ -17,11 +17,12 @@ pub fn router(registry: Registry) -> Router {
         .route("/datasets", post(create_dataset))
         .route("/datasets", get(list_datasets))
         .route("/datasets/{id}", get(get_dataset))
-        .route("/datasets/by-name/{name}", get(get_dataset_by_name))
+        .route("/datasets/by-name/{name}", get(get_dataset_by_name).delete(delete_dataset_by_name))
         .route("/datasets/by-name/{name}/metadata", post(update_metadata))
         .route("/datasets/by-name/{name}/resync", post(resync_dataset))
         .route("/resync-missing", post(resync_all_missing))
         .route("/migrate-buckets", post(migrate_buckets))
+        .route("/dedupe", post(dedupe_by_name))
         // Phase D: AI-safe views
         .route("/views", post(create_view).get(list_views))
         .route("/views/{name}", get(get_view).delete(delete_view))
@@ -121,7 +122,14 @@ async fn create_dataset(
             let resp = DatasetResponse::from(&manifest);
             Ok((StatusCode::CREATED, Json(resp)))
         }
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => {
+            let status = if e.contains("different schema") {
+                StatusCode::CONFLICT
+            } else {
+                StatusCode::INTERNAL_SERVER_ERROR
+            };
+            Err((status, e))
+        }
     }
 }
 
@@ -153,6 +161,25 @@ async fn get_dataset_by_name(
     }
 }
 
+/// Remove a dataset manifest by name. Metadata only — parquet files,
+/// vector indexes, and tombstones are NOT cascade-deleted. See
+/// `Registry::delete_dataset` for the full scope.
+async fn delete_dataset_by_name(
+    State(registry): State<Registry>,
+    Path(name): Path<String>,
+) -> impl IntoResponse {
+    match registry.delete_dataset(&name).await {
+        Ok(removed) => {
+            let body = serde_json::json!({ "name": name, "manifests_removed": removed });
+            Ok(Json(body))
+        }
+        Err(e) if e.starts_with("dataset not found") => {
+            Err((StatusCode::NOT_FOUND, e))
+        }
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}
+
 async fn update_metadata(
     State(registry): State<Registry>,
     Path(name): Path<String>,
@@ -214,6 +241,12 @@ async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse
     }
 }
 
+/// Collapse duplicate manifests by `name` — see `Registry::dedupe_by_name`.
+/// Safe to run repeatedly; single-manifest datasets are untouched.
+async fn dedupe_by_name(State(registry): State<Registry>) -> impl IntoResponse {
+    Json(registry.dedupe_by_name().await)
+}
+
 // --- Phase D: AI-safe views ---
 
 #[derive(Deserialize)]

crates/gateway/Cargo.toml

@@ -28,3 +28,4 @@ opentelemetry-stdout = { workspace = true }
 tracing-opentelemetry = { workspace = true }
 arrow = { workspace = true }
 chrono = { workspace = true }
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

crates/gateway/src/main.rs

@@ -3,6 +3,7 @@ mod access_service;
 mod auth;
 mod observability;
 mod tools;
+mod v1;
 
 use axum::{Router, extract::DefaultBodyLimit, routing::get};
 use proto::lakehouse::catalog_service_server::CatalogServiceServer;
@@ -86,6 +87,11 @@ async fn main() {
     // primary. Journal / registry resolve per-call via the bucket registry.
     let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone());
     let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone());
+    let hs = vectord::harness::HarnessStore::new(bucket_registry.clone(), index_reg.clone());
+    // Phase 19: playbook memory. Load cached state; empty-on-miss is fine,
+    // operators call POST /vectors/playbook_memory/rebuild to populate.
+    let pbm = vectord::playbook_memory::PlaybookMemory::new(store.clone());
+    let _ = pbm.load_from_storage().await;
 
     // Phase 16.2: spawn the autotune agent. When config.agent.enabled=false
     // this returns a handle that drops triggers silently — no surprise load.
@@ -107,6 +113,7 @@ async fn main() {
             embedding_cache: emb_cache.clone(),
             trial_journal: tj.clone(),
             promotion_registry: pr.clone(),
+            harness_store: hs.clone(),
         },
     );
 
@@ -155,6 +162,7 @@ async fn main() {
             hnsw_store: hnsw,
             embedding_cache: emb_cache,
             trial_journal: tj,
+            harness_store: hs,
             catalog: registry.clone(),
             promotion_registry: pr,
             agent_handle,
@@ -163,6 +171,8 @@ async fn main() {
             lance: vectord::lance_backend::LanceRegistry::new(
                 bucket_registry.clone(), index_reg.clone(),
             ),
+            playbook_memory: pbm,
+            embed_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(1)),
         }))
         .nest("/workspaces", queryd::workspace_service::router(workspace_mgr))
         .nest("/journal", journald::service::router(journal))
@@ -174,6 +184,38 @@ async fn main() {
                 registry: tool_reg,
                 query_fn: tools::QueryExecutor::new(engine.clone()),
             }
+        }))
+        // Phase 38 — Universal API skeleton. Thin OpenAI-compatible
+        // surface over the existing aibridge → Ollama path. Future
+        // phases add provider adapters (39), routing engine (40),
+        // session state (41), etc. All without changing this mount.
+        .nest("/v1", v1::router(v1::V1State {
+            ai_client: ai_client.clone(),
+            usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
+            // Phase 39 first slice — Ollama Cloud adapter. Key resolved
+            // from OLLAMA_CLOUD_KEY env, then /root/llm_team_config.json,
+            // then OLLAMA_CLOUD_API_KEY env. None = cloud routes 503.
+            ollama_cloud_key: {
+                let k = v1::ollama_cloud::resolve_cloud_key();
+                if k.is_some() {
+                    tracing::info!("v1: Ollama Cloud key loaded — /v1/chat provider=ollama_cloud enabled");
+                } else {
+                    tracing::warn!("v1: no Ollama Cloud key in env or /root/llm_team_config.json — cloud routes will 503");
+                }
+                k
+            },
+            // Phase 40 early deliverable — Langfuse trace emitter.
+            // Defaults match mcp-server/tracing.ts conventions so
+            // gateway traces land in the same staffing project.
+            langfuse: {
+                let c = v1::langfuse_trace::LangfuseClient::from_env_or_defaults();
+                if c.is_some() {
+                    tracing::info!("v1: Langfuse tracing enabled — /v1/chat calls will appear at localhost:3001");
+                } else {
+                    tracing::warn!("v1: Langfuse keys missing — /v1/chat calls will not be traced");
+                }
+                c
+            },
         }));
 
     // Auth middleware (if enabled)
@@ -190,7 +232,7 @@ async fn main() {
     }
 
     app = app
-        .layer(DefaultBodyLimit::max(256 * 1024 * 1024)) // 256MB
+        .layer(DefaultBodyLimit::max(512 * 1024 * 1024)) // 512MB — supports 500K worker CSV
         .layer(CorsLayer::new()
             .allow_origin(Any)
             .allow_methods(Any)

crates/gateway/src/tools/registry.rs

@@ -103,7 +103,7 @@ impl ToolRegistry {
                     ParamDef { name: "limit".into(), param_type: "integer".into(), required: false, description: "Max results".into(), default: Some(serde_json::json!(20)) },
                 ],
                 returns: "List of candidates with id, name, phone, email, skills, experience".into(),
-                sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, zip, vertical, skills, years_experience FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
+                sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, skills, years_experience, status FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
                 category: "candidates".into(),
             },
             ToolDef {

crates/gateway/src/v1/langfuse_trace.rs

@@ -0,0 +1,226 @@
+//! Phase 40 (early deliverable) — Langfuse tracing from the Rust gateway.
+//!
+//! Langfuse is already running at `http://localhost:3001` with the
+//! staffing project — `mcp-server/tracing.ts` has been emitting traces
+//! for months. This file bridges the gap: every `/v1/chat` call now
+//! appears in the same Langfuse project, alongside traces from the
+//! Bun side.
+//!
+//! Design:
+//! - Fire-and-forget — emit spawns a tokio task. Response to client
+//!   never waits on Langfuse, never fails because of it. If Langfuse
+//!   is down or slow, we silently drop traces (and `tracing::warn!`).
+//! - Two events per call — `trace-create` + `generation-create`. The
+//!   generation nests under the trace so Langfuse UI shows the whole
+//!   thing as one clickable row.
+//! - No SDK — plain HTTP POST to `/api/public/ingestion` with Basic
+//!   auth. Keeps the Rust dependency graph small.
+
+use serde::Serialize;
+use std::sync::Arc;
+use std::time::Duration;
+
+use super::Message;
+
+const INGESTION_PATH: &str = "/api/public/ingestion";
+const EMIT_TIMEOUT_SECS: u64 = 5;
+
+#[derive(Clone)]
+pub struct LangfuseClient {
+    inner: Arc<Inner>,
+}
+
+struct Inner {
+    base_url: String,
+    public_key: String,
+    secret_key: String,
+    http: reqwest::Client,
+}
+
+impl LangfuseClient {
+    pub fn new(base_url: String, public_key: String, secret_key: String) -> Self {
+        let http = reqwest::Client::builder()
+            .timeout(Duration::from_secs(EMIT_TIMEOUT_SECS))
+            .build()
+            .expect("langfuse client init");
+        Self { inner: Arc::new(Inner { base_url, public_key, secret_key, http }) }
+    }
+
+    /// Resolve a LangfuseClient from the same conventions the TS side
+    /// already uses. Returns None if keys are missing — caller logs a
+    /// warning and omits tracing (non-fatal).
+    ///
+    /// Sources checked in order:
+    ///   1. LANGFUSE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY env
+    ///   2. /root/llm_team_config.json (if it carries langfuse keys)
+    ///   3. Hardcoded staffing-project defaults matching mcp-server
+    ///      (pk-lf-staffing / sk-lf-staffing-secret / localhost:3001)
+    pub fn from_env_or_defaults() -> Option<Self> {
+        let base = std::env::var("LANGFUSE_URL")
+            .unwrap_or_else(|_| "http://localhost:3001".to_string());
+        let pk = std::env::var("LANGFUSE_PUBLIC_KEY")
+            .unwrap_or_else(|_| "pk-lf-staffing".to_string());
+        let sk = std::env::var("LANGFUSE_SECRET_KEY")
+            .unwrap_or_else(|_| "sk-lf-staffing-secret".to_string());
+        if pk.trim().is_empty() || sk.trim().is_empty() { return None; }
+        Some(Self::new(base, pk, sk))
+    }
+
+    /// Fire-and-forget emit. Never blocks, never errors upward.
+    pub fn emit_chat(&self, ev: ChatTrace) {
+        let this = self.clone();
+        tokio::spawn(async move {
+            if let Err(e) = this.emit_chat_inner(ev).await {
+                tracing::warn!(target: "v1.langfuse", "trace drop: {e}");
+            }
+        });
+    }
+
+    async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
+        let trace_id = uuid_v7_like();
+        let gen_id = uuid_v7_like();
+        let trace_ts = ev.start_time.clone();
+
+        let batch = IngestionBatch {
+            batch: vec![
+                IngestionEvent {
+                    id: uuid_v7_like(),
+                    timestamp: trace_ts.clone(),
+                    kind: "trace-create",
+                    body: serde_json::json!({
+                        "id": trace_id,
+                        "name": format!("v1.chat:{}", ev.provider),
+                        "input": serde_json::json!({
+                            "model": ev.model,
+                            "messages": ev.input,
+                        }),
+                        "metadata": serde_json::json!({
+                            "provider": ev.provider,
+                            "think": ev.think,
+                        }),
+                    }),
+                },
+                IngestionEvent {
+                    id: uuid_v7_like(),
+                    timestamp: ev.end_time.clone(),
+                    kind: "generation-create",
+                    body: serde_json::json!({
+                        "id": gen_id,
+                        "traceId": trace_id,
+                        "name": "chat",
+                        "model": ev.model,
+                        "modelParameters": serde_json::json!({
+                            "temperature": ev.temperature,
+                            "max_tokens": ev.max_tokens,
+                            "think": ev.think,
+                        }),
+                        "input": ev.input,
+                        "output": ev.output,
+                        "usage": serde_json::json!({
+                            "input": ev.prompt_tokens,
+                            "output": ev.completion_tokens,
+                            "total": ev.prompt_tokens + ev.completion_tokens,
+                            "unit": "TOKENS",
+                        }),
+                        "startTime": ev.start_time,
+                        "endTime": ev.end_time,
+                        "metadata": serde_json::json!({
+                            "provider": ev.provider,
+                            "latency_ms": ev.latency_ms,
+                        }),
+                    }),
+                },
+            ],
+        };
+
+        let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
+        let resp = self.inner.http
+            .post(url)
+            .basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
+            .json(&batch)
+            .send()
+            .await
+            .map_err(|e| format!("POST failed: {e}"))?;
+        if !resp.status().is_success() {
+            return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+        }
+        Ok(())
+    }
+}
+
+/// Everything the v1.chat handler collects for one completed call.
+pub struct ChatTrace {
+    pub provider: String,
+    pub model: String,
+    pub input: Vec<Message>,
+    pub output: String,
+    pub prompt_tokens: u32,
+    pub completion_tokens: u32,
+    pub temperature: Option<f64>,
+    pub max_tokens: Option<u32>,
+    pub think: Option<bool>,
+    pub start_time: String,
+    pub end_time: String,
+    pub latency_ms: u64,
+}
+
+#[derive(Serialize)]
+struct IngestionBatch {
+    batch: Vec<IngestionEvent>,
+}
+
+#[derive(Serialize)]
+struct IngestionEvent {
+    id: String,
+    timestamp: String,
+    #[serde(rename = "type")]
+    kind: &'static str,
+    body: serde_json::Value,
+}
+
+/// UUIDv7-ish identifier — not strictly UUIDv7 but time-ordered and
+/// unique enough for event deduplication. Avoids a uuid crate dep.
+fn uuid_v7_like() -> String {
+    let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
+    let rand = std::time::SystemTime::now()
+        .duration_since(std::time::UNIX_EPOCH)
+        .map(|d| d.subsec_nanos())
+        .unwrap_or(0);
+    format!("{:016x}-{:08x}", ts, rand)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn from_env_or_defaults_returns_some_when_defaults_present() {
+        // Defaults are hardcoded non-empty — should always produce Some.
+        let c = LangfuseClient::from_env_or_defaults();
+        assert!(c.is_some());
+    }
+
+    #[test]
+    fn uuid_v7_like_produces_distinct_ids() {
+        let a = uuid_v7_like();
+        // Different nanosecond timestamps — but they might collide on
+        // back-to-back calls. Loop a bit to ensure uniqueness surface.
+        let b = uuid_v7_like();
+        let c = uuid_v7_like();
+        assert!(a != b || b != c, "three consecutive ids should have at least one distinct pair");
+        assert!(a.contains('-'));
+    }
+
+    #[test]
+    fn ingestion_batch_serializes_with_rename() {
+        let ev = IngestionEvent {
+            id: "e1".into(),
+            timestamp: "2026-04-22T00:00:00Z".into(),
+            kind: "trace-create",
+            body: serde_json::json!({"name": "t"}),
+        };
+        let json = serde_json::to_string(&IngestionBatch { batch: vec![ev] }).unwrap();
+        assert!(json.contains("\"type\":\"trace-create\""));
+        assert!(json.contains("\"batch\":["));
+    }
+}

crates/gateway/src/v1/mod.rs

@@ -0,0 +1,256 @@
+//! Phase 38 — Universal API skeleton (`/v1/*`).
+//!
+//! OpenAI-compatible shape on top of the existing aibridge → Ollama
+//! path. This is the thin slice: single provider, stateless, no
+//! streaming. Phase 39 replaces the direct Ollama call with a
+//! `ProviderAdapter` trait dispatch; Phase 40 adds routing + fallback.
+//!
+//! The shape matches OpenAI's `/v1/chat/completions` closely enough
+//! that clients using openai-compatible SDKs can point at us with the
+//! URL swap alone. We keep the endpoint path `/v1/chat` (not
+//! `/v1/chat/completions`) because our PRD declares the terser form;
+//! adding the alias is one line in Phase 39 when it matters.
+
+pub mod ollama;
+pub mod ollama_cloud;
+pub mod langfuse_trace;
+
+use axum::{
+    Router,
+    extract::State,
+    http::StatusCode,
+    response::IntoResponse,
+    routing::{get, post},
+    Json,
+};
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use tokio::sync::RwLock;
+
+#[derive(Clone)]
+pub struct V1State {
+    pub ai_client: aibridge::client::AiClient,
+    pub usage: Arc<RwLock<Usage>>,
+    /// Ollama Cloud bearer token. Loaded at startup via
+    /// `ollama_cloud::resolve_cloud_key()`. None = cloud routes 503.
+    pub ollama_cloud_key: Option<String>,
+    /// Phase 40 early deliverable — Langfuse client. None = tracing
+    /// disabled (keys missing or container unreachable). Traces are
+    /// fire-and-forget: never block the response path.
+    pub langfuse: Option<langfuse_trace::LangfuseClient>,
+}
+
+#[derive(Default, Clone, Serialize)]
+pub struct Usage {
+    pub requests: u64,
+    pub prompt_tokens: u64,
+    pub completion_tokens: u64,
+    pub total_tokens: u64,
+}
+
+pub fn router(state: V1State) -> Router {
+    Router::new()
+        .route("/chat", post(chat))
+        .route("/usage", get(usage))
+        .route("/sessions", get(sessions))
+        .with_state(state)
+}
+
+// -- Shared types (OpenAI-compatible) --
+
+#[derive(Serialize, Deserialize, Clone, Debug)]
+pub struct Message {
+    pub role: String,
+    pub content: String,
+}
+
+#[derive(Deserialize, Debug)]
+pub struct ChatRequest {
+    pub model: String,
+    pub messages: Vec<Message>,
+    #[serde(default)]
+    pub temperature: Option<f64>,
+    #[serde(default)]
+    pub max_tokens: Option<u32>,
+    /// Accepted for shape-compat but ignored in the thin slice —
+    /// Phase 38 returns non-streaming even when the client asked for it.
+    /// Phase 39+ wires real streaming.
+    #[serde(default)]
+    pub stream: Option<bool>,
+    /// Non-OpenAI extension. Passes through to the provider's thinking
+    /// toggle. Default: **false** — hot-path discipline for thinking
+    /// models (qwen3.5, qwen3, gpt-oss) that otherwise burn the token
+    /// budget on hidden reasoning before visible output starts,
+    /// producing empty responses. Set true explicitly when calling an
+    /// overseer / reasoning-heavy path.
+    #[serde(default)]
+    pub think: Option<bool>,
+    /// Non-OpenAI extension. Selects the provider adapter. Accepted:
+    ///   - None / "ollama" / "local" → local Ollama via sidecar (default)
+    ///   - "ollama_cloud" / "cloud" → Ollama Cloud direct HTTPS
+    /// Phase 40 adds a routing engine that picks this automatically
+    /// from the model name; Phase 38/39 requires explicit selection.
+    #[serde(default)]
+    pub provider: Option<String>,
+}
+
+#[derive(Serialize)]
+pub struct ChatResponse {
+    pub id: String,
+    pub object: &'static str,
+    pub created: i64,
+    pub model: String,
+    pub choices: Vec<Choice>,
+    pub usage: UsageBlock,
+}
+
+#[derive(Serialize)]
+pub struct Choice {
+    pub index: u32,
+    pub message: Message,
+    pub finish_reason: String,
+}
+
+#[derive(Serialize, Deserialize, Clone)]
+pub struct UsageBlock {
+    pub prompt_tokens: u32,
+    pub completion_tokens: u32,
+    pub total_tokens: u32,
+}
+
+// -- Handlers --
+
+async fn chat(
+    State(state): State<V1State>,
+    Json(req): Json<ChatRequest>,
+) -> Result<Json<ChatResponse>, (StatusCode, String)> {
+    if req.messages.is_empty() {
+        return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
+    }
+    if req.stream.unwrap_or(false) {
+        tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
+    }
+
+    let provider = req.provider.as_deref().unwrap_or("ollama").to_ascii_lowercase();
+    let start_time = chrono::Utc::now();
+    let start_instant = std::time::Instant::now();
+
+    let resp = match provider.as_str() {
+        "ollama" | "local" | "" => ollama::chat(&state.ai_client, &req)
+            .await
+            .map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama local: {e}")))?,
+        "ollama_cloud" | "cloud" => {
+            let key = state.ollama_cloud_key.as_deref().ok_or((
+                StatusCode::SERVICE_UNAVAILABLE,
+                "OLLAMA_CLOUD_KEY not configured — set env or populate /root/llm_team_config.json".to_string(),
+            ))?;
+            ollama_cloud::chat(key, &req)
+                .await
+                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama cloud: {e}")))?
+        }
+        other => {
+            return Err((
+                StatusCode::BAD_REQUEST,
+                format!("unknown provider '{other}' — supported: ollama, ollama_cloud"),
+            ));
+        }
+    };
+
+    let end_time = chrono::Utc::now();
+    let latency_ms = start_instant.elapsed().as_millis() as u64;
+
+    // Phase 40 — emit Langfuse trace. Fire-and-forget: the clone is
+    // cheap (Arc inside), the tokio::spawn never blocks us, a dead
+    // Langfuse just logs a warn. Client-visible response latency is
+    // untouched.
+    if let Some(lf) = &state.langfuse {
+        let output = resp.choices.first()
+            .map(|c| c.message.content.clone())
+            .unwrap_or_default();
+        lf.emit_chat(langfuse_trace::ChatTrace {
+            provider: provider.clone(),
+            model: resp.model.clone(),
+            input: req.messages.clone(),
+            output,
+            prompt_tokens: resp.usage.prompt_tokens,
+            completion_tokens: resp.usage.completion_tokens,
+            temperature: req.temperature,
+            max_tokens: req.max_tokens,
+            think: req.think,
+            start_time: start_time.to_rfc3339(),
+            end_time: end_time.to_rfc3339(),
+            latency_ms,
+        });
+    }
+
+    {
+        let mut u = state.usage.write().await;
+        u.requests += 1;
+        u.prompt_tokens += resp.usage.prompt_tokens as u64;
+        u.completion_tokens += resp.usage.completion_tokens as u64;
+        u.total_tokens += resp.usage.total_tokens as u64;
+    }
+
+    Ok(Json(resp))
+}
+
+async fn usage(State(state): State<V1State>) -> impl IntoResponse {
+    let snapshot = state.usage.read().await.clone();
+    Json(snapshot)
+}
+
+// Phase 38 is stateless — no session persistence yet. Return an empty
+// list in OpenAI-ish shape so clients that probe this endpoint don't
+// 404. Real session state lands in Phase 41 with the profile-system
+// expansion.
+async fn sessions() -> impl IntoResponse {
+    Json(serde_json::json!({
+        "data": [],
+        "object": "list",
+        "note": "Phase 38: stateless. Session state lands in Phase 41.",
+    }))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn chat_request_parses_openai_shape() {
+        let raw = r#"{
+            "model": "qwen3.5:latest",
+            "messages": [
+                {"role": "system", "content": "You are helpful."},
+                {"role": "user", "content": "Hi"}
+            ],
+            "temperature": 0.2,
+            "max_tokens": 100
+        }"#;
+        let r: ChatRequest = serde_json::from_str(raw).unwrap();
+        assert_eq!(r.model, "qwen3.5:latest");
+        assert_eq!(r.messages.len(), 2);
+        assert_eq!(r.messages[0].role, "system");
+        assert_eq!(r.messages[1].content, "Hi");
+        assert_eq!(r.temperature, Some(0.2));
+        assert_eq!(r.max_tokens, Some(100));
+    }
+
+    #[test]
+    fn chat_request_accepts_minimal() {
+        let raw = r#"{
+            "model": "any",
+            "messages": [{"role": "user", "content": "hi"}]
+        }"#;
+        let r: ChatRequest = serde_json::from_str(raw).unwrap();
+        assert_eq!(r.temperature, None);
+        assert_eq!(r.max_tokens, None);
+        assert_eq!(r.stream, None);
+    }
+
+    #[test]
+    fn usage_counter_default_is_zero() {
+        let u = Usage::default();
+        assert_eq!(u.requests, 0);
+        assert_eq!(u.total_tokens, 0);
+    }
+}

crates/gateway/src/v1/ollama.rs

@@ -0,0 +1,160 @@
+//! Phase 38 — Ollama shape adapter.
+//!
+//! Translates `/v1/chat` (OpenAI-compatible) requests into the
+//! existing aibridge `GenerateRequest` shape, and the `GenerateResponse`
+//! back into an OpenAI-compatible `ChatResponse`. This is a bridge,
+//! not a new client — aibridge + the Python sidecar stay as-is.
+//!
+//! Phase 39 replaces this direct call with a `ProviderAdapter` trait
+//! dispatch so the same `/v1/chat` handler routes to any provider.
+
+use aibridge::client::{AiClient, GenerateRequest};
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+pub async fn chat(client: &AiClient, req: &ChatRequest) -> Result<ChatResponse, String> {
+    let (system, prompt) = flatten_messages(&req.messages);
+
+    let gen_req = GenerateRequest {
+        prompt,
+        model: Some(req.model.clone()),
+        system: if system.is_empty() { None } else { Some(system) },
+        temperature: req.temperature,
+        max_tokens: req.max_tokens,
+        // Phase 38 default: think=false. Hot-path discipline for
+        // thinking models (qwen3.5, qwen3, gpt-oss) which otherwise
+        // burn the max_tokens budget on hidden reasoning before any
+        // visible output, producing empty responses. Callers that
+        // actually want reasoning (overseers, T3+ tiers) opt in via
+        // the `think: true` extension field. Phase 40 routing engine
+        // flips this per task class.
+        think: Some(req.think.unwrap_or(false)),
+    };
+
+    let t0 = std::time::Instant::now();
+    let resp = client.generate(gen_req).await.map_err(|e| e.to_string())?;
+    let latency_ms = t0.elapsed().as_millis();
+
+    // Prefer sidecar-reported token counts when present. Fall back to
+    // chars/4 estimate (biased safe ~15%, matches Phase 21 convention
+    // in crates/aibridge/src/context.rs::estimate_tokens).
+    let prompt_tokens = resp.tokens_evaluated
+        .map(|n| n as u32)
+        .unwrap_or_else(|| estimate_prompt_tokens(&req.messages));
+    let completion_tokens = resp.tokens_generated
+        .map(|n| n as u32)
+        .unwrap_or_else(|| ((resp.text.chars().count() + 3) / 4) as u32);
+
+    tracing::info!(
+        target: "v1.chat",
+        model = %req.model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "ollama chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model: resp.model,
+        choices: vec![Choice {
+            index: 0,
+            message: Message {
+                role: "assistant".into(),
+                content: resp.text,
+            },
+            finish_reason: "stop".into(),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+/// Public re-export of the flattener so sibling adapters (Ollama Cloud,
+/// future OpenRouter) can reuse the same shape collapse without
+/// duplicating the logic. Keeps `(system, prompt)` format consistent
+/// across providers.
+pub fn flatten_messages_public(messages: &[Message]) -> (String, String) {
+    flatten_messages(messages)
+}
+
+/// Collapse a message array into (system, prompt). Multiple system
+/// messages concatenate with a newline — matches OpenAI's documented
+/// behavior. Non-system messages become role-labeled turns.
+fn flatten_messages(messages: &[Message]) -> (String, String) {
+    let mut system = String::new();
+    let mut prompt = String::new();
+    for m in messages {
+        if m.role == "system" {
+            if !system.is_empty() { system.push('\n'); }
+            system.push_str(&m.content);
+        } else {
+            prompt.push_str(&m.role);
+            prompt.push_str(": ");
+            prompt.push_str(&m.content);
+            prompt.push_str("\n\n");
+        }
+    }
+    prompt.push_str("assistant:");
+    (system, prompt)
+}
+
+fn estimate_prompt_tokens(messages: &[Message]) -> u32 {
+    let chars: usize = messages.iter().map(|m| m.content.chars().count()).sum();
+    ((chars + 3) / 4) as u32
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn flatten_separates_system_from_turns() {
+        let msgs = vec![
+            Message { role: "system".into(), content: "Rules here.".into() },
+            Message { role: "user".into(), content: "Q1".into() },
+            Message { role: "assistant".into(), content: "A1".into() },
+            Message { role: "user".into(), content: "Q2".into() },
+        ];
+        let (system, prompt) = flatten_messages(&msgs);
+        assert_eq!(system, "Rules here.");
+        assert!(prompt.contains("user: Q1"));
+        assert!(prompt.contains("assistant: A1"));
+        assert!(prompt.contains("user: Q2"));
+        assert!(prompt.trim_end().ends_with("assistant:"));
+    }
+
+    #[test]
+    fn flatten_concatenates_multiple_system_messages() {
+        let msgs = vec![
+            Message { role: "system".into(), content: "First.".into() },
+            Message { role: "system".into(), content: "Second.".into() },
+            Message { role: "user".into(), content: "Hi".into() },
+        ];
+        let (system, _) = flatten_messages(&msgs);
+        assert_eq!(system, "First.\nSecond.");
+    }
+
+    #[test]
+    fn flatten_with_no_system_returns_empty_system() {
+        let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
+        let (system, prompt) = flatten_messages(&msgs);
+        assert!(system.is_empty());
+        assert!(prompt.contains("user: hi"));
+    }
+
+    #[test]
+    fn estimate_tokens_chars_div_4_ceiling() {
+        let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
+        // 8 chars / 4 = 2, with ceiling → 2
+        assert_eq!(estimate_prompt_tokens(&msgs), 2);
+
+        let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
+        // 9 chars → (9+3)/4 = 3 (ceiling)
+        assert_eq!(estimate_prompt_tokens(&msgs2), 3);
+    }
+}

crates/gateway/src/v1/ollama_cloud.rs

@@ -0,0 +1,189 @@
+//! Phase 39 (first slice) — Ollama Cloud adapter.
+//!
+//! Direct HTTPS call to `https://ollama.com/api/generate` with Bearer
+//! auth. Mirrors the `tests/multi-agent/agent.ts::generateCloud` pattern
+//! exactly (same endpoint, same body shape, same Bearer header), so
+//! cloud calls from Rust behave identically to the TS hot path.
+//!
+//! Key sourcing priority:
+//!   1. Env var `OLLAMA_CLOUD_KEY` (matches agent.ts convention)
+//!   2. `/root/llm_team_config.json` → providers.ollama_cloud.api_key
+//!   3. Env var `OLLAMA_CLOUD_API_KEY` (LLM Team UI convention)
+//!
+//! First hit wins. Key is loaded once at gateway startup by
+//! `resolve_cloud_key()` and stored on `V1State.ollama_cloud_key`.
+
+use std::time::Duration;
+use serde::{Deserialize, Serialize};
+
+use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
+
+const CLOUD_BASE_URL: &str = "https://ollama.com";
+const CLOUD_TIMEOUT_SECS: u64 = 180;
+
+/// Read the Ollama Cloud key from the three sanctioned sources. Returns
+/// None if none is set — callers must 503 rather than attempt a call.
+pub fn resolve_cloud_key() -> Option<String> {
+    if let Ok(k) = std::env::var("OLLAMA_CLOUD_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    if let Ok(raw) = std::fs::read_to_string("/root/llm_team_config.json") {
+        if let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) {
+            if let Some(k) = v.pointer("/providers/ollama_cloud/api_key").and_then(|x| x.as_str()) {
+                if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+            }
+        }
+    }
+    if let Ok(k) = std::env::var("OLLAMA_CLOUD_API_KEY") {
+        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
+    }
+    None
+}
+
+pub async fn chat(
+    key: &str,
+    req: &ChatRequest,
+) -> Result<ChatResponse, String> {
+    let (system, prompt) = super::ollama::flatten_messages_public(&req.messages);
+
+    let body = CloudGenerateBody {
+        model: req.model.clone(),
+        prompt,
+        system: if system.is_empty() { None } else { Some(system) },
+        stream: false,
+        think: Some(req.think.unwrap_or(false)),
+        options: CloudOptions {
+            // Thinking cloud models need headroom — floor 400 to give
+            // qwen3.5:397b / gpt-oss:120b reasoning room. Matches
+            // agent.ts `Math.max(opts.max_tokens ?? 800, 400)` policy.
+            num_predict: std::cmp::max(req.max_tokens.unwrap_or(800), 400),
+            temperature: req.temperature.unwrap_or(0.3),
+        },
+    };
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(CLOUD_TIMEOUT_SECS))
+        .build()
+        .map_err(|e| format!("build client: {e}"))?;
+
+    let t0 = std::time::Instant::now();
+    let resp = client
+        .post(format!("{}/api/generate", CLOUD_BASE_URL))
+        .bearer_auth(key)
+        .json(&body)
+        .send()
+        .await
+        .map_err(|e| format!("ollama.com unreachable: {e}"))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        let body = resp.text().await.unwrap_or_else(|_| "?".into());
+        return Err(format!("ollama.com {}: {}", status, body));
+    }
+
+    let parsed: CloudGenerateResponse = resp.json().await
+        .map_err(|e| format!("invalid cloud response: {e}"))?;
+
+    let latency_ms = t0.elapsed().as_millis();
+    let text = parsed.response.unwrap_or_default();
+
+    let prompt_tokens = parsed.prompt_eval_count.unwrap_or_else(|| {
+        let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
+        ((chars + 3) / 4) as u32
+    });
+    let completion_tokens = parsed.eval_count.unwrap_or_else(|| {
+        ((text.chars().count() + 3) / 4) as u32
+    });
+
+    tracing::info!(
+        target: "v1.chat",
+        provider = "ollama_cloud",
+        model = %req.model,
+        prompt_tokens,
+        completion_tokens,
+        latency_ms = latency_ms as u64,
+        "cloud chat completed",
+    );
+
+    Ok(ChatResponse {
+        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
+        object: "chat.completion",
+        created: chrono::Utc::now().timestamp(),
+        model: parsed.model.unwrap_or_else(|| req.model.clone()),
+        choices: vec![Choice {
+            index: 0,
+            message: Message { role: "assistant".into(), content: text },
+            finish_reason: "stop".into(),
+        }],
+        usage: UsageBlock {
+            prompt_tokens,
+            completion_tokens,
+            total_tokens: prompt_tokens + completion_tokens,
+        },
+    })
+}
+
+// -- Ollama /api/generate wire shapes --
+
+#[derive(Serialize)]
+struct CloudGenerateBody {
+    model: String,
+    prompt: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    system: Option<String>,
+    stream: bool,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    think: Option<bool>,
+    options: CloudOptions,
+}
+
+#[derive(Serialize)]
+struct CloudOptions {
+    num_predict: u32,
+    temperature: f64,
+}
+
+#[derive(Deserialize)]
+struct CloudGenerateResponse {
+    #[serde(default)]
+    response: Option<String>,
+    #[serde(default)]
+    model: Option<String>,
+    #[serde(default)]
+    prompt_eval_count: Option<u32>,
+    #[serde(default)]
+    eval_count: Option<u32>,
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_cloud_key_returns_none_when_no_sources_set() {
+        // Only check shape — we can't reliably unset env vars in a test
+        // that runs alongside others, and the file path is on disk.
+        // If all three sources are empty the function returns None; if
+        // any are set we expect Some. This just smoke-tests the call
+        // doesn't panic.
+        let _ = resolve_cloud_key();
+    }
+
+    #[test]
+    fn cloud_body_serializes_compact() {
+        let body = CloudGenerateBody {
+            model: "gpt-oss:120b".into(),
+            prompt: "user: hi\n\nassistant:".into(),
+            system: Some("Be terse.".into()),
+            stream: false,
+            think: Some(false),
+            options: CloudOptions { num_predict: 400, temperature: 0.3 },
+        };
+        let json = serde_json::to_string(&body).unwrap();
+        assert!(json.contains("\"model\":\"gpt-oss:120b\""));
+        assert!(json.contains("\"stream\":false"));
+        assert!(json.contains("\"num_predict\":400"));
+        assert!(json.contains("\"think\":false"));
+        assert!(json.contains("\"system\":\"Be terse.\""));
+    }
+}

crates/ingestd/Cargo.toml

@@ -24,3 +24,4 @@ object_store = { workspace = true }
 tokio-postgres = { workspace = true }
 mysql_async = { workspace = true }
 uuid = { workspace = true }
+croner = "3"

crates/ingestd/src/schedule.rs

@@ -17,8 +17,8 @@
 //! shouldn't pile up runs). Different schedules run concurrently.
 //!
 //! What's deliberately not in scope here:
-//! - Cron expressions (the trigger enum has the variant but parsing is
-//!   stubbed). Intervals cover 90% of operational scheduling; cron is
+//! - Sub-minute cron precision. 5-field Unix cron is supported; seconds
+//!   are pinned to 0. Intervals cover sub-minute cases; cron is
 //!   easy to bolt on later.
 //! - Backoff / retry policies. A failed run records the failure and
 //!   schedules `next_run_at` as if it succeeded — no exponential backoff.
@@ -85,8 +85,12 @@ impl ScheduleKind {
 pub enum ScheduleTrigger {
     /// Run every N seconds. Time anchor = compute_next_run_at decision.
     Interval { secs: u64 },
-    /// Cron expression — parsing not implemented yet. Defining the
-    /// variant now so the JSON shape is forward-compatible.
+    /// 5-field Unix cron expression: "min hour dom month dow".
+    /// Seconds are implicitly 0 (minute-granularity scheduling).
+    /// Examples: "15 14 * * *" = 14:15 UTC daily, "0 */6 * * *" = every
+    /// 6 hours on the hour, "0 9 * * 1-5" = 09:00 UTC weekdays.
+    /// Timezone: UTC always — the scheduler's clock is UTC and we don't
+    /// carry a per-schedule timezone field.
     Cron { expr: String },
 }
 
@@ -162,22 +166,55 @@ impl ScheduleDef {
 //      Parse the expression, find the next match after `now`. Most
 //    expressive but pulls in a cron parser dep.
 //
-// Cron is currently stubbed — falling back to "now + 1 hour" so a
-// mistakenly-configured Cron schedule doesn't melt the GPU.
 pub fn compute_next_run_at(
     trigger: &ScheduleTrigger,
     completed_at: DateTime<Utc>,
     _previous_next: DateTime<Utc>,
 ) -> DateTime<Utc> {
-    // TODO(J): pick a strategy. Starter = since-last-completion.
     match trigger {
         ScheduleTrigger::Interval { secs } => {
             completed_at + Duration::seconds(*secs as i64)
         }
-        ScheduleTrigger::Cron { expr: _ } => {
-            // Cron parsing not implemented — fall back to a safe 1h
-            // window so a bad config can't fire-loop the system.
-            completed_at + Duration::hours(1)
+        ScheduleTrigger::Cron { expr } => {
+            next_cron_fire(expr, completed_at)
+                // Creation-time validation (see `validate_trigger`) rejects
+                // unparseable expressions, so this fallback is only reached
+                // if the schedule JSON was hand-edited on disk. One-hour
+                // window keeps a bad expr from fire-looping the system.
+                .unwrap_or(completed_at + Duration::hours(1))
+        }
+    }
+}
+
+/// Parse a Vixie/POSIX cron expression and return the next fire time
+/// strictly after `after`. Accepts 5-field (`min hour dom month dow`) and
+/// 6-field (with leading `sec`) forms natively — croner handles both.
+/// Day-of-week follows Unix convention: 0=Sun, 1=Mon, …, 6=Sat (7=Sun as
+/// synonym). Returns None if the expression fails to parse or produces no
+/// upcoming match (which can happen for year-bounded patterns in the past).
+fn next_cron_fire(expr: &str, after: DateTime<Utc>) -> Option<DateTime<Utc>> {
+    use std::str::FromStr;
+    let cron = croner::Cron::from_str(expr).ok()?;
+    cron.find_next_occurrence(&after, false).ok()
+}
+
+/// Reject a ScheduleTrigger whose content can't be interpreted — used by
+/// the HTTP handlers to fail fast at create/patch time rather than falling
+/// back silently at fire time. Interval triggers get a sanity gate
+/// (no zero secs); cron triggers get a full parse through croner.
+pub fn validate_trigger(trigger: &ScheduleTrigger) -> Result<(), String> {
+    match trigger {
+        ScheduleTrigger::Interval { secs } => {
+            if *secs == 0 {
+                return Err("interval secs must be > 0".into());
+            }
+            Ok(())
+        }
+        ScheduleTrigger::Cron { expr } => {
+            use std::str::FromStr;
+            croner::Cron::from_str(expr)
+                .map_err(|e| format!("invalid cron expression '{expr}': {e}"))?;
+            Ok(())
         }
     }
 }
@@ -551,6 +588,7 @@ fn redact_dsn(dsn: &str) -> String {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use chrono::Timelike;
 
     fn mk(secs: u64) -> ScheduleDef {
         ScheduleDef {
@@ -600,12 +638,73 @@ mod tests {
     }
 
     #[test]
-    fn cron_falls_back_to_one_hour() {
+    fn cron_every_minute_fires_within_sixty_seconds() {
         let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() };
         let now = Utc::now();
         let next = compute_next_run_at(&trig, now, now);
         let delta = next - now;
-        assert_eq!(delta.num_seconds(), 3600);
+        assert!(delta.num_seconds() > 0 && delta.num_seconds() <= 60,
+            "expected next fire within 60s, got {}s", delta.num_seconds());
+    }
+
+    #[test]
+    fn cron_daily_at_1415_utc_is_within_24h() {
+        // 14:15 UTC daily — whether we're before or after 14:15 today, the
+        // next fire is at most 24h out.
+        let trig = ScheduleTrigger::Cron { expr: "15 14 * * *".into() };
+        let now: DateTime<Utc> = "2026-04-20T10:00:00Z".parse().unwrap();
+        let next = compute_next_run_at(&trig, now, now);
+        assert!(next > now);
+        assert!((next - now).num_hours() <= 24);
+        // And the minute-of-hour is 15.
+        assert_eq!(next.minute(), 15);
+        assert_eq!(next.hour(), 14);
+    }
+
+    #[test]
+    fn cron_weekday_skips_weekend() {
+        // 09:00 UTC weekdays only. On a Saturday, next fire is Monday.
+        let trig = ScheduleTrigger::Cron { expr: "0 9 * * 1-5".into() };
+        // 2026-04-18 is a Saturday.
+        let sat: DateTime<Utc> = "2026-04-18T10:00:00Z".parse().unwrap();
+        let next = compute_next_run_at(&trig, sat, sat);
+        // Monday 2026-04-20 at 09:00 UTC.
+        assert_eq!(next, "2026-04-20T09:00:00Z".parse::<DateTime<Utc>>().unwrap());
+    }
+
+    #[test]
+    fn cron_six_field_seconds_granularity() {
+        // 6-field (seconds included): fire every minute at 30s past the minute.
+        let trig = ScheduleTrigger::Cron { expr: "30 * * * * *".into() };
+        let now = Utc::now();
+        let next = compute_next_run_at(&trig, now, now);
+        assert_eq!(next.second(), 30);
+    }
+
+    #[test]
+    fn validate_rejects_bad_cron() {
+        let bad = ScheduleTrigger::Cron { expr: "not a cron".into() };
+        let err = validate_trigger(&bad).unwrap_err();
+        assert!(err.contains("cron"), "error should mention cron, got: {err}");
+    }
+
+    #[test]
+    fn validate_rejects_wrong_field_count() {
+        // 4 fields — neither 5 nor 6.
+        let bad = ScheduleTrigger::Cron { expr: "* * * *".into() };
+        assert!(validate_trigger(&bad).is_err());
+    }
+
+    #[test]
+    fn validate_rejects_zero_interval() {
+        let bad = ScheduleTrigger::Interval { secs: 0 };
+        assert!(validate_trigger(&bad).is_err());
+    }
+
+    #[test]
+    fn validate_accepts_good_cron() {
+        let good = ScheduleTrigger::Cron { expr: "0 */6 * * *".into() };
+        assert!(validate_trigger(&good).is_ok());
     }
 
     #[test]

crates/ingestd/src/service.rs

@@ -510,6 +510,9 @@ async fn create_schedule(
     if state.schedules.get(&req.id).await.is_some() {
         return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id)));
     }
+    if let Err(e) = schedule::validate_trigger(&req.trigger) {
+        return Err((StatusCode::BAD_REQUEST, e));
+    }
     let now = chrono::Utc::now();
     let def = schedule::ScheduleDef {
         id: req.id,
@@ -564,6 +567,11 @@ async fn patch_schedule(
     let Some(mut def) = state.schedules.get(&id).await else {
         return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
     };
+    if let Some(t) = &req.trigger {
+        if let Err(e) = schedule::validate_trigger(t) {
+            return Err((StatusCode::BAD_REQUEST, e));
+        }
+    }
     if let Some(e) = req.enabled { def.enabled = e; }
     if let Some(t) = req.trigger { def.trigger = t; }
     match state.schedules.put(def).await {

crates/storaged/src/append_log.rs

@@ -18,7 +18,7 @@
 ///   to small JSONL events.
 ///
 /// Storage layout:
-/// ```
+/// ```text
 /// {prefix}/
 ///   batch_0001776319628000123.jsonl
 ///   batch_0001776319745987654.jsonl
@@ -221,15 +221,15 @@ pub struct CompactStats {
     pub new_key: Option<String>,
 }
 
-// Proactively flush on drop, best-effort.
-// We can't `.await` in Drop; we spawn the flush on the tokio runtime if one
-// is available. If the runtime is already shutting down the flush is lost —
-// acceptable for these observability journals, which are hints not records.
+// Log unflushed-buffer size on drop. We can't `.await` from a sync `Drop`,
+// so a real flush isn't possible here — callers are responsible for calling
+// `.flush()` before dropping if durability matters. These journals are
+// observability hints; a few lost buffered events at shutdown are
+// acceptable per ADR-018.
 impl Drop for AppendLog {
     fn drop(&mut self) {
         let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
         if buf_len == 0 { return; }
-        // Can't spawn from sync Drop on every runtime shape; log + move on.
         tracing::debug!(
             "append_log '{}' dropping with {} unflushed events",
             self.prefix, buf_len,
@@ -237,3 +237,94 @@ impl Drop for AppendLog {
     }
 }
 
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use object_store::memory::InMemory;
+
+    fn mk(threshold: usize) -> AppendLog {
+        AppendLog::new(Arc::new(InMemory::new()), "prefix")
+            .with_flush_threshold(threshold)
+    }
+
+    #[tokio::test]
+    async fn append_stays_buffered_below_threshold() {
+        let log = mk(5);
+        log.append(b"one".to_vec()).await.unwrap();
+        log.append(b"two".to_vec()).await.unwrap();
+        assert_eq!(log.file_count().await.unwrap(), 0, "no files until threshold");
+        let all = log.read_all().await.unwrap();
+        assert_eq!(all, vec![b"one".to_vec(), b"two".to_vec()],
+            "read_all surfaces unflushed buffer");
+    }
+
+    #[tokio::test]
+    async fn append_auto_flushes_on_threshold() {
+        let log = mk(3);
+        for i in 0..3 {
+            log.append(format!("evt{i}").into_bytes()).await.unwrap();
+        }
+        assert_eq!(log.file_count().await.unwrap(), 1, "threshold triggered one flush");
+
+        // A fourth append stays buffered until the next threshold.
+        log.append(b"evt3".to_vec()).await.unwrap();
+        assert_eq!(log.file_count().await.unwrap(), 1, "below threshold again");
+    }
+
+    #[tokio::test]
+    async fn flush_empty_is_noop() {
+        let log = mk(32);
+        log.flush().await.unwrap();
+        log.flush().await.unwrap();
+        assert_eq!(log.file_count().await.unwrap(), 0);
+    }
+
+    #[tokio::test]
+    async fn read_all_orders_events_across_flushes() {
+        let log = mk(1); // flush-on-every-append
+        for i in 0..5 {
+            log.append(format!("e{i}").into_bytes()).await.unwrap();
+            // Spread writes out so timestamps sort strictly.
+            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
+        }
+        let all = log.read_all().await.unwrap();
+        let strs: Vec<String> = all.into_iter()
+            .map(|v| String::from_utf8(v).unwrap())
+            .collect();
+        assert_eq!(strs, vec!["e0", "e1", "e2", "e3", "e4"],
+            "lex sort of batch keys == chronological event order");
+    }
+
+    #[tokio::test]
+    async fn compact_merges_multiple_files_into_one() {
+        let log = mk(1); // force file-per-append
+        for i in 0..4 {
+            log.append(format!("e{i}").into_bytes()).await.unwrap();
+            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
+        }
+        assert_eq!(log.file_count().await.unwrap(), 4);
+
+        let stats = log.compact().await.unwrap();
+        assert_eq!(stats.merged_files, 4);
+        assert_eq!(stats.events, 4);
+        assert!(stats.new_key.is_some());
+
+        assert_eq!(log.file_count().await.unwrap(), 1, "originals deleted, 1 survivor");
+        let all = log.read_all().await.unwrap();
+        assert_eq!(all.len(), 4, "no events lost in compaction");
+    }
+
+    #[tokio::test]
+    async fn compact_with_single_file_is_noop() {
+        let log = mk(1);
+        log.append(b"only".to_vec()).await.unwrap();
+        assert_eq!(log.file_count().await.unwrap(), 1);
+
+        let stats = log.compact().await.unwrap();
+        assert_eq!(stats.merged_files, 1);
+        assert_eq!(stats.events, 0, "nothing to consolidate");
+        assert!(stats.new_key.is_none(), "no new file written");
+        assert_eq!(log.file_count().await.unwrap(), 1, "original untouched");
+    }
+}
+

crates/storaged/src/error_journal.rs

@@ -198,3 +198,92 @@ impl ErrorJournal {
         self.log.file_count().await
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use object_store::memory::InMemory;
+
+    fn mk() -> ErrorJournal {
+        ErrorJournal::new(Arc::new(InMemory::new()))
+    }
+
+    #[tokio::test]
+    async fn append_updates_ring_immediately() {
+        let j = mk();
+        j.append(BucketErrorEvent::new_read("tenant_a", "k1", "timeout")).await;
+        j.append(BucketErrorEvent::new_write("tenant_b", "k2", "denied")).await;
+
+        let recent = j.recent(10).await;
+        assert_eq!(recent.len(), 2);
+        assert_eq!(recent[0].target, "tenant_a");
+        assert_eq!(recent[1].target, "tenant_b");
+    }
+
+    #[tokio::test]
+    async fn load_recent_rehydrates_ring_from_disk() {
+        // Use a shared InMemory store so the second journal reads the first
+        // journal's writes — mirrors gateway restart semantics.
+        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        {
+            let j1 = ErrorJournal {
+                log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX).with_flush_threshold(1)),
+                ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
+            };
+            j1.append(BucketErrorEvent::new_read("t", "k", "err")).await;
+            j1.flush().await.unwrap();
+        }
+
+        let j2 = ErrorJournal {
+            log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX)),
+            ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
+        };
+        assert_eq!(j2.recent(10).await.len(), 0, "fresh ring is empty");
+
+        let loaded = j2.load_recent().await.unwrap();
+        assert_eq!(loaded, 1);
+        assert_eq!(j2.recent(10).await[0].target, "t");
+    }
+
+    #[tokio::test]
+    async fn mark_rescued_updates_most_recent_match() {
+        let j = mk();
+        j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
+        j.append(BucketErrorEvent::new_read("t2", "k", "err")).await;
+        j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
+
+        j.mark_rescued_last("t1", "k").await;
+
+        let recent = j.recent(10).await;
+        assert!(!recent[0].rescued, "older t1 event not touched");
+        assert!(!recent[1].rescued, "t2 not touched");
+        assert!(recent[2].rescued, "most recent t1 was rescued");
+    }
+
+    #[tokio::test]
+    async fn health_flags_buckets_with_three_or_more_errors() {
+        let j = mk();
+        for _ in 0..3 { j.append(BucketErrorEvent::new_read("broken", "k", "e")).await; }
+        j.append(BucketErrorEvent::new_read("flaky", "k", "e")).await;
+
+        let h = j.health(60).await;
+        assert_eq!(h.total_errors, 4);
+        assert_eq!(h.per_bucket.get("broken"), Some(&3));
+        assert_eq!(h.per_bucket.get("flaky"), Some(&1));
+        assert!(h.unhealthy_buckets.contains(&"broken".to_string()));
+        assert!(!h.unhealthy_buckets.contains(&"flaky".to_string()));
+    }
+
+    #[tokio::test]
+    async fn filter_by_bucket_returns_chronological_order() {
+        let j = mk();
+        j.append(BucketErrorEvent::new_read("a", "k1", "e")).await;
+        j.append(BucketErrorEvent::new_read("b", "k2", "e")).await;
+        j.append(BucketErrorEvent::new_read("a", "k3", "e")).await;
+
+        let only_a = j.filter(Some("a"), None, 10).await;
+        assert_eq!(only_a.len(), 2);
+        assert_eq!(only_a[0].key, "k1", "oldest first");
+        assert_eq!(only_a[1].key, "k3");
+    }
+}

crates/ui/assets/style.css

@@ -205,3 +205,13 @@ tr:hover td { background: var(--accent-glow); }
     padding: 8px 12px; border-bottom: 1px solid var(--border); font-size: 13px;
 }
 .table-item:hover { background: var(--accent-glow); }
+
+/* Phase 19 — Playbook panel */
+.boosted-row { background: rgba(120, 200, 120, 0.10); }
+.boosted-row td { border-top: 1px solid rgba(120, 200, 120, 0.30); }
+.mono-cell {
+    font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
+    font-size: 11px; color: var(--text-dim);
+    max-width: 220px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
+}
+.panel-section + .panel-section { margin-top: 18px; }

crates/ui/src/main.rs

@@ -178,9 +178,116 @@ enum Tab {
     Explore,
     Sql,
     Ingest,
+    Playbook,
     Status,
 }
 
+// --- Playbook memory types (Phase 19) ---
+
+#[derive(Debug, Clone, Deserialize, PartialEq)]
+struct PlaybookStats {
+    entries: usize,
+    entries_with_embeddings: usize,
+    #[serde(default)]
+    total_names_endorsed: usize,
+    #[serde(default)]
+    sample: Vec<PlaybookSample>,
+}
+
+#[derive(Debug, Clone, Deserialize, PartialEq)]
+struct PlaybookSample {
+    id: String,
+    operation: String,
+    #[serde(default)]
+    city: Option<String>,
+    #[serde(default)]
+    state: Option<String>,
+    #[serde(default)]
+    endorsed: Vec<String>,
+}
+
+#[derive(Debug, Clone, Deserialize, PartialEq)]
+struct HybridResp {
+    #[serde(default)]
+    sql_matches: usize,
+    #[serde(default)]
+    vector_reranked: usize,
+    #[serde(default)]
+    method: String,
+    #[serde(default)]
+    duration_ms: u64,
+    #[serde(default)]
+    answer: Option<String>,
+    #[serde(default)]
+    sources: Vec<HybridSource>,
+}
+
+#[derive(Debug, Clone, Deserialize, PartialEq)]
+struct HybridSource {
+    doc_id: String,
+    chunk_text: String,
+    score: f32,
+    #[serde(default)]
+    sql_verified: bool,
+    #[serde(default)]
+    playbook_boost: f32,
+    #[serde(default)]
+    playbook_citations: Vec<String>,
+}
+
+#[derive(Debug, Clone, Deserialize, PartialEq)]
+struct IndexInfo {
+    index_name: String,
+    source: String,
+    #[serde(default)]
+    chunk_count: usize,
+    #[serde(default)]
+    vector_backend: String,
+}
+
+async fn fetch_playbook_stats() -> Result<PlaybookStats, String> {
+    let resp = reqwest::get(&format!("{}/vectors/playbook_memory/stats", api_base()))
+        .await.map_err(|e| e.to_string())?;
+    if !resp.status().is_success() {
+        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+    }
+    resp.json().await.map_err(|e| e.to_string())
+}
+
+async fn rebuild_playbook_memory() -> Result<serde_json::Value, String> {
+    let client = reqwest::Client::new();
+    let resp = client.post(&format!("{}/vectors/playbook_memory/rebuild", api_base()))
+        .json(&serde_json::json!({}))
+        .send().await.map_err(|e| e.to_string())?;
+    if !resp.status().is_success() {
+        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+    }
+    resp.json().await.map_err(|e| e.to_string())
+}
+
+async fn fetch_indexes() -> Result<Vec<IndexInfo>, String> {
+    let resp = reqwest::get(&format!("{}/vectors/indexes", api_base()))
+        .await.map_err(|e| e.to_string())?;
+    resp.json().await.map_err(|e| e.to_string())
+}
+
+async fn hybrid_search(index_name: &str, question: &str, use_playbook: bool, top_k: usize) -> Result<HybridResp, String> {
+    let client = reqwest::Client::new();
+    let resp = client.post(&format!("{}/vectors/hybrid", api_base()))
+        .json(&serde_json::json!({
+            "index_name": index_name,
+            "question": question,
+            "top_k": top_k,
+            "generate": false,
+            "use_playbook_memory": use_playbook,
+        }))
+        .send().await.map_err(|e| e.to_string())?;
+    if !resp.status().is_success() {
+        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
+    }
+    resp.json().await.map_err(|e| e.to_string())
+}
+
 // --- App ---
 
 #[component]
@@ -239,6 +346,11 @@ fn App() -> Element {
                         onclick: move |_| active_tab.set(Tab::Ingest),
                         "Ingest"
                     }
+                    button {
+                        class: if *active_tab.read() == Tab::Playbook { "tab active" } else { "tab" },
+                        onclick: move |_| active_tab.set(Tab::Playbook),
+                        "Playbook"
+                    }
                     button {
                         class: if *active_tab.read() == Tab::Status { "tab active" } else { "tab" },
                         onclick: move |_| active_tab.set(Tab::Status),
@@ -260,6 +372,7 @@ fn App() -> Element {
                     Tab::Explore => rsx! { ExplorePanel { datasets: datasets.read().clone() } },
                     Tab::Sql => rsx! { SqlPanel {} },
                     Tab::Ingest => rsx! { IngestPanel {} },
+                    Tab::Playbook => rsx! { PlaybookPanel {} },
                     Tab::Status => rsx! { StatusPanel {} },
                 }
             }
@@ -354,14 +467,14 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
         div { class: "panel ask-panel",
             div { class: "ask-hero",
                 h2 { "Ask your data anything" }
-                p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI." }
+                p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI over the staffing dataset." }
             }
 
             div { class: "ask-input-row",
                 input {
                     class: "ask-input",
                     value: "{question}",
-                    placeholder: "e.g. Which department has the highest average salary?",
+                    placeholder: "e.g. Which clients placed the most candidates last quarter?",
                     oninput: move |e| question.set(e.value()),
                     onkeydown: move |e| {
                         if e.key() == Key::Enter {
@@ -432,10 +545,12 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
 
             div { class: "ask-examples",
                 "Try: "
-                button { class: "example-btn", onclick: move |_| question.set("Which department has the highest average salary?".into()), "highest avg salary by dept" }
-                button { class: "example-btn", onclick: move |_| question.set("Show me the top 3 most expensive products".into()), "top 3 expensive products" }
-                button { class: "example-btn", onclick: move |_| question.set("How many events per action type?".into()), "events by action" }
-                button { class: "example-btn", onclick: move |_| question.set("List all employees who earn more than 90000".into()), "employees > 90k" }
+                button { class: "example-btn", onclick: move |_| question.set("How many candidates do we have by city?".into()), "candidates by city" }
+                button { class: "example-btn", onclick: move |_| question.set("Top 10 clients by total placements".into()), "top clients by placements" }
+                button { class: "example-btn", onclick: move |_| question.set("Open job orders ordered by bill rate descending".into()), "open jobs by rate" }
+                button { class: "example-btn", onclick: move |_| question.set("Recruiters with the highest placement count".into()), "top recruiters" }
+                button { class: "example-btn", onclick: move |_| question.set("Total billed hours per client last month".into()), "hours per client" }
+                button { class: "example-btn", onclick: move |_| question.set("Cold leads: candidates we called more than 5 times but never placed".into()), "cold leads" }
             }
 
             if let Some(sql) = generated_sql.read().as_ref() {
@@ -578,7 +693,7 @@ fn ExplorePanel(datasets: Vec<Dataset>) -> Element {
 
 #[component]
 fn SqlPanel() -> Element {
-    let mut query_text = use_signal(|| String::from("SELECT * FROM employees LIMIT 10"));
+    let mut query_text = use_signal(|| String::from("SELECT candidate_id, first_name, last_name, city, status FROM candidates LIMIT 10"));
     let mut result = use_signal(|| None::<Result<QueryResponse, String>>);
     let mut loading = use_signal(|| false);
 
@@ -727,7 +842,7 @@ fn DashboardPanel() -> Element {
                     }
                     div { class: "stat-card accent",
                         div { class: "stat-value", "{s[\"hnsw_loaded\"]}" }
-                        div { class: "stat-label", "HNSW Indexes (27ms)" }
+                        div { class: "stat-label", "HNSW Indexes Loaded" }
                     }
                     div { class: "stat-card",
                         div { class: "stat-value", "{s[\"tools\"]}" }
@@ -750,27 +865,27 @@ fn DashboardPanel() -> Element {
                     div { class: "arch-grid",
                         div { class: "arch-card",
                             div { class: "arch-title", "Ingest" }
-                            div { class: "arch-items", "CSV, JSON, PDF, Text, PostgreSQL, File Watcher" }
+                            div { class: "arch-items", "CSV · JSON · PDF (+OCR) · Text · Postgres · MySQL · Inbox watcher · Cron schedules" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Storage" }
-                            div { class: "arch-items", "Parquet on Object Storage, Delta Writes, Compaction" }
+                            div { class: "arch-items", "Parquet on Object Storage · Delta writes · Compaction · Tombstones · Multi-bucket federation + rescue" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Query" }
-                            div { class: "arch-items", "DataFusion SQL, MemCache (9.8x), Hot/Cold" }
+                            div { class: "arch-items", "DataFusion SQL · MemCache (9.8× hot) · Merge-on-read · AI-safe views" }
                         }
                         div { class: "arch-card",
-                            div { class: "arch-title", "AI" }
-                            div { class: "arch-items", "Ollama (local), Embed, Generate, RAG, HNSW" }
+                            div { class: "arch-title", "AI / Vector" }
+                            div { class: "arch-items", "Ollama (local) · Embed/Generate/RAG · HNSW (Parquet) · Lance IVF_PQ · Hybrid SQL+vector · Profile-scoped" }
+                        }
+                        div { class: "arch-card",
+                            div { class: "arch-title", "Learning loop" }
+                            div { class: "arch-items", "Playbook memory · Endorsement boost · Multi-agent orchestrator · Autotune agent (Pareto-promote)" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Governance" }
-                            div { class: "arch-items", "Event Journal, PII Detection, Tool Registry, Access Control" }
-                        }
-                        div { class: "arch-card",
-                            div { class: "arch-title", "Agents" }
-                            div { class: "arch-items", "Workspaces, Handoff, Shortlists, Activity Logs" }
+                            div { class: "arch-items", "Event journal · PII detection · Tool registry · Access control · Audit log · Catalog v2 metadata" }
                         }
                     }
                 }
@@ -779,20 +894,23 @@ fn DashboardPanel() -> Element {
                     h3 { "Build Progression" }
                     div { class: "phase-list",
                         {rsx! {
-                            PhaseItem { num: "0-5", name: "Foundation", detail: "Storage, Catalog, DataFusion, AI, UI, gRPC" }
-                            PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV/JSON/PDF/Text auto-schema" }
-                            PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed, Search, LLM Answers" }
-                            PhaseItem { num: "8", name: "Hot Cache", detail: "9.8x speedup, Delta Writes" }
-                            PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract, Instant Handoff" }
-                            PhaseItem { num: "9", name: "Event Journal", detail: "Append-only Mutation History" }
-                            PhaseItem { num: "10", name: "Rich Catalog", detail: "PII Detection, Lineage" }
-                            PhaseItem { num: "11", name: "Embedding Versioning", detail: "Model-proof Vectors" }
-                            PhaseItem { num: "12", name: "Tool Registry", detail: "6 Governed Actions + Audit" }
-                            PhaseItem { num: "13", name: "Access Control", detail: "Role-based, Field-level" }
-                            PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff Detection, AI Migration" }
-                            PhaseItem { num: "15", name: "HNSW Index", detail: "100K Search in 27ms" }
-                            PhaseItem { num: "16", name: "File Watcher", detail: "Auto-ingest from Inbox" }
-                            PhaseItem { num: "17", name: "DB Connector", detail: "PostgreSQL Import" }
+                            PhaseItem { num: "0-5", name: "Foundation", detail: "Storage · Catalog · DataFusion · Ollama · UI · gRPC" }
+                            PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV · JSON · PDF · Text · auto-schema · dedupe" }
+                            PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed · brute-force cosine · LLM grounded answers" }
+                            PhaseItem { num: "8", name: "Hot Cache + Deltas", detail: "MemTable LRU · 9.8× speedup · merge-on-read · compaction" }
+                            PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract · daily/weekly/monthly tiers · zero-copy handoff" }
+                            PhaseItem { num: "9", name: "Event Journal", detail: "Append-only mutation log · time-travel · audit" }
+                            PhaseItem { num: "10", name: "Rich Catalog v2", detail: "PII auto-detection · lineage · freshness SLA · sensitivity" }
+                            PhaseItem { num: "11", name: "Embedding Versioning", detail: "Per-index model+version · A/B · incremental re-embed" }
+                            PhaseItem { num: "12", name: "Tool Registry", detail: "Governed actions · param validation · audit · MCP-ready" }
+                            PhaseItem { num: "13", name: "Access Control", detail: "Roles · field-level sensitivity · column masking · query audit" }
+                            PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff detection · AI migration prompts · versioned schemas" }
+                            PhaseItem { num: "15", name: "HNSW + Trials", detail: "100K vectors · p50 873µs · trial journal · eval harness" }
+                            PhaseItem { num: "16", name: "Hot-swap + Autotune", detail: "Promotion registry · rollback · ε-greedy agent · Pareto winner" }
+                            PhaseItem { num: "17", name: "Model Profiles + VRAM", detail: "ModelProfile manifests · scoped search · sequential model swap" }
+                            PhaseItem { num: "18", name: "Lance hybrid backend", detail: "IVF_PQ build 14× faster · random fetch 112× · S3-native · per-profile routing" }
+                            PhaseItem { num: "19", name: "Playbook memory", detail: "Feedback loop · endorsement boost (cap 0.25) · orchestrator write-through · citations" }
+                            PhaseItem { num: "+", name: "Federation + Schedules", detail: "Multi-bucket · rescue fallback · error journal · MySQL · PDF OCR · cron ingest · catalog dedupe" }
                         }}
                     }
                 }
@@ -816,6 +934,285 @@ fn PhaseItem(num: String, name: String, detail: String) -> Element {
     }
 }
 
+// === PLAYBOOK — Phase 19 meta-index feedback loop ===
+
+#[component]
+fn PlaybookPanel() -> Element {
+    let mut stats = use_signal(|| None::<Result<PlaybookStats, String>>);
+    let mut indexes = use_signal(Vec::<IndexInfo>::new);
+    let mut rebuild_status = use_signal(|| None::<Result<String, String>>);
+    let mut rebuilding = use_signal(|| false);
+    let mut loaded = use_signal(|| false);
+
+    // Comparison state
+    let mut selected_index = use_signal(|| String::new());
+    let mut question = use_signal(|| String::from("reliable assembler in Detroit"));
+    let mut top_k = use_signal(|| 10usize);
+    let mut compare_loading = use_signal(|| false);
+    let mut hits_off = use_signal(|| None::<Result<HybridResp, String>>);
+    let mut hits_on = use_signal(|| None::<Result<HybridResp, String>>);
+
+    let load_all = move || {
+        spawn(async move {
+            stats.set(Some(fetch_playbook_stats().await));
+            if let Ok(ix) = fetch_indexes().await {
+                if selected_index.read().is_empty() {
+                    if let Some(default) = ix.iter().find(|i| i.source == "workers_500k").or_else(|| ix.first()) {
+                        selected_index.set(default.index_name.clone());
+                    }
+                }
+                indexes.set(ix);
+            }
+        });
+    };
+
+    use_effect(move || {
+        if !*loaded.read() {
+            loaded.set(true);
+            load_all();
+        }
+    });
+
+    let do_rebuild = move |_| {
+        spawn(async move {
+            rebuilding.set(true);
+            rebuild_status.set(None);
+            match rebuild_playbook_memory().await {
+                Ok(v) => rebuild_status.set(Some(Ok(format!("rebuild ok — {}", v)))),
+                Err(e) => rebuild_status.set(Some(Err(e))),
+            }
+            // Refresh stats afterward
+            stats.set(Some(fetch_playbook_stats().await));
+            rebuilding.set(false);
+        });
+    };
+
+    let do_compare = move |_| {
+        let idx = selected_index.read().clone();
+        let q = question.read().clone();
+        let k = *top_k.read();
+        if idx.is_empty() || q.trim().is_empty() { return; }
+        spawn(async move {
+            compare_loading.set(true);
+            hits_off.set(None);
+            hits_on.set(None);
+            // Run both sequentially so the embedding cache is shared
+            hits_off.set(Some(hybrid_search(&idx, &q, false, k).await));
+            hits_on.set(Some(hybrid_search(&idx, &q, true, k).await));
+            compare_loading.set(false);
+        });
+    };
+
+    rsx! {
+        div { class: "panel",
+            div { class: "ask-hero",
+                h2 { "Playbook Memory" }
+                p { class: "subtitle",
+                    "Phase 19 feedback loop: past successful playbooks boost future search rankings. \
+                     Endorsed workers from semantically similar past operations re-rank toward the top, \
+                     with citations back to the playbook that endorsed them."
+                }
+            }
+
+            // Stats card
+            div { class: "panel-section",
+                match stats.read().as_ref() {
+                    None => rsx! { div { class: "loading", "loading playbook stats..." } },
+                    Some(Err(e)) => rsx! { div { class: "error", "stats: {e}" } },
+                    Some(Ok(s)) => rsx! {
+                        div { class: "stat-grid",
+                            div { class: "stat-card",
+                                div { class: "stat-value", "{s.entries}" }
+                                div { class: "stat-label", "Playbooks in Memory" }
+                            }
+                            div { class: "stat-card",
+                                div { class: "stat-value", "{s.entries_with_embeddings}" }
+                                div { class: "stat-label", "Embedded" }
+                            }
+                            div { class: "stat-card accent",
+                                div { class: "stat-value", "{s.total_names_endorsed}" }
+                                div { class: "stat-label", "Endorsed Worker-Tags" }
+                            }
+                        }
+                    }
+                }
+                div { class: "sql-actions",
+                    button {
+                        class: "btn",
+                        disabled: *rebuilding.read(),
+                        onclick: do_rebuild,
+                        if *rebuilding.read() { "rebuilding from successful_playbooks..." } else { "Rebuild from successful_playbooks" }
+                    }
+                }
+                if let Some(s) = rebuild_status.read().as_ref() {
+                    match s {
+                        Ok(msg) => rsx! { div { class: "result-box", "{msg}" } },
+                        Err(e) => rsx! { div { class: "error", "{e}" } },
+                    }
+                }
+            }
+
+            // Sample playbooks
+            if let Some(Ok(s)) = stats.read().as_ref() {
+                if !s.sample.is_empty() {
+                    div { class: "panel-section",
+                        h3 { "Sample playbooks" }
+                        div { class: "table-wrap",
+                            table {
+                                thead { tr {
+                                    th { "ID" }
+                                    th { "Operation" }
+                                    th { "Location" }
+                                    th { "Endorsed" }
+                                } }
+                                tbody {
+                                    for pb in s.sample.iter() {
+                                        {
+                                            let loc = match (&pb.city, &pb.state) {
+                                                (Some(c), Some(st)) => format!("{c}, {st}"),
+                                                _ => "—".into(),
+                                            };
+                                            let endorsed = if pb.endorsed.is_empty() {
+                                                "—".to_string()
+                                            } else {
+                                                pb.endorsed.join(", ")
+                                            };
+                                            let pid = pb.id.clone();
+                                            let op = pb.operation.clone();
+                                            rsx! {
+                                                tr {
+                                                    td { class: "mono-cell", title: "{pid}", "{pid}" }
+                                                    td { "{op}" }
+                                                    td { "{loc}" }
+                                                    td { "{endorsed}" }
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+
+            // Side-by-side comparison: boost OFF vs ON
+            div { class: "panel-section",
+                h3 { "See the boost — search compared" }
+                p { class: "hint",
+                    "Run the same query against the same index twice — once with playbook boost OFF and once ON. \
+                     Hits with non-zero playbook_boost and citations are workers that past similar playbooks endorsed."
+                }
+                div { class: "form-row",
+                    label { "Index" }
+                    select {
+                        value: "{selected_index}",
+                        onchange: move |e| selected_index.set(e.value()),
+                        for ix in indexes.read().iter() {
+                            option { value: "{ix.index_name}", "{ix.index_name} ({ix.source}, {ix.chunk_count} chunks, {ix.vector_backend})" }
+                        }
+                    }
+                }
+                div { class: "form-row",
+                    label { "Question" }
+                    input {
+                        value: "{question}",
+                        oninput: move |e| question.set(e.value()),
+                        placeholder: "e.g. reliable assembler in Detroit"
+                    }
+                }
+                div { class: "form-row",
+                    label { "Top K" }
+                    input {
+                        r#type: "number",
+                        value: "{top_k}",
+                        oninput: move |e| {
+                            if let Ok(n) = e.value().parse::<usize>() { top_k.set(n.clamp(1, 50)); }
+                        }
+                    }
+                }
+                button {
+                    class: "btn btn-ask",
+                    disabled: *compare_loading.read(),
+                    onclick: do_compare,
+                    if *compare_loading.read() { "running both queries..." } else { "Run comparison" }
+                }
+
+                div { class: "explore-grid",
+                    div { class: "ds-detail",
+                        h3 { "Boost OFF (vanilla)" }
+                        match hits_off.read().as_ref() {
+                            None => rsx! { div { class: "empty", "—" } },
+                            Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
+                            Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
+                        }
+                    }
+                    div { class: "ds-detail",
+                        h3 { "Boost ON (Phase 19)" }
+                        match hits_on.read().as_ref() {
+                            None => rsx! { div { class: "empty", "—" } },
+                            Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
+                            Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
+                        }
+                    }
+                }
+            }
+        }
+    }
+}
+
+#[component]
+fn HybridHitTable(resp: HybridResp) -> Element {
+    rsx! {
+        div { class: "results-info",
+            "{resp.sources.len()} hits · {resp.duration_ms}ms · method={resp.method}"
+        }
+        if resp.sources.is_empty() {
+            div { class: "empty-sm", "no hits" }
+        } else {
+            div { class: "table-wrap",
+                table {
+                    thead { tr {
+                        th { "#" }
+                        th { "Doc" }
+                        th { "Score" }
+                        th { "Boost" }
+                        th { "Citations" }
+                        th { "Snippet" }
+                    } }
+                    tbody {
+                        for (i, h) in resp.sources.iter().enumerate() {
+                            {
+                                let snippet: String = h.chunk_text.chars().take(120).collect();
+                                let cites = if h.playbook_citations.is_empty() {
+                                    "—".to_string()
+                                } else {
+                                    h.playbook_citations.join(", ")
+                                };
+                                let row_class = if h.playbook_boost > 0.0 { "boosted-row" } else { "" };
+                                let rank = i + 1;
+                                let did = h.doc_id.clone();
+                                let score = format!("{:.3}", h.score);
+                                let boost = if h.playbook_boost > 0.0 { format!("+{:.3}", h.playbook_boost) } else { "—".into() };
+                                rsx! {
+                                    tr { class: "{row_class}",
+                                        td { "{rank}" }
+                                        td { class: "mono-cell", "{did}" }
+                                        td { "{score}" }
+                                        td { "{boost}" }
+                                        td { class: "mono-cell", title: "{cites}", "{cites}" }
+                                        td { "{snippet}" }
+                                    }
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+}
+
 // === INGEST — Data on-ramp ===
 
 #[component]

crates/vectord-lance/src/lib.rs

@@ -357,7 +357,25 @@ impl LanceVectorStore {
     /// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ
     /// index if one exists; otherwise does a full scan (slow but
     /// correct — useful during development before index build).
-    pub async fn search(&self, query: &[f32], top_k: usize) -> Result<Vec<Hit>, String> {
+    ///
+    /// `nprobes` tells Lance how many IVF partitions to probe per query.
+    /// Lance's built-in default is **1**, which caps recall well below
+    /// what the index is actually capable of. Passing `None` keeps that
+    /// default (only sensible when latency trumps recall); for real
+    /// workloads set `nprobes` to 5–10% of `num_partitions` (e.g. 20–30
+    /// on a 316-partition index).
+    ///
+    /// `refine_factor` re-ranks the PQ-approximate top-k by computing
+    /// exact distances on `top_k * refine_factor` candidates, then
+    /// trimming to `top_k`. Cheap way to buy back recall lost to product
+    /// quantization. `None` skips the re-rank pass.
+    pub async fn search(
+        &self,
+        query: &[f32],
+        top_k: usize,
+        nprobes: Option<usize>,
+        refine_factor: Option<u32>,
+    ) -> Result<Vec<Hit>, String> {
         use lance::dataset::Dataset;
 
         let dataset = Dataset::open(&self.path).await.map_err(e)?;
@@ -365,6 +383,8 @@ impl LanceVectorStore {
 
         let mut scanner = dataset.scan();
         scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?;
+        if let Some(n) = nprobes { scanner.nprobes(n); }
+        if let Some(f) = refine_factor { scanner.refine(f); }
         scanner.project(&["doc_id", "chunk_text"]).map_err(e)?;
 
         let mut stream = scanner.try_into_stream().await.map_err(e)?;

crates/vectord/Cargo.toml

@@ -8,6 +8,8 @@ shared = { path = "../shared" }
 storaged = { path = "../storaged" }
 aibridge = { path = "../aibridge" }
 catalogd = { path = "../catalogd" }
+queryd = { path = "../queryd" }
+futures = { workspace = true }
 # ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps.
 # Public API uses only std types so no version conflict propagates here.
 vectord-lance = { path = "../vectord-lance" }
@@ -23,3 +25,5 @@ arrow = { workspace = true }
 chrono = { workspace = true }
 instant-distance = { workspace = true }
 uuid = { workspace = true }
+sha2 = { workspace = true }
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

crates/vectord/src/agent.rs

@@ -204,6 +204,7 @@ pub struct AgentDeps {
     pub embedding_cache: EmbeddingCache,
     pub trial_journal: TrialJournal,
     pub promotion_registry: PromotionRegistry,
+    pub harness_store: crate::harness::HarnessStore,
 }
 
 // -------- Spawn --------
@@ -354,12 +355,11 @@ async fn run_one_cycle(
     // Read history.
     let history = deps.trial_journal.list(&event.index_name).await
         .map_err(|e| format!("read journal: {e}"))?;
-    if history.is_empty() {
-        return Err(format!(
-            "no trials yet for '{}' — seed with at least one POST /hnsw/trial first",
-            event.index_name,
-        ));
-    }
+
+    // Bootstrap mode: first ever visit to this index. Auto-generate a
+    // harness (if missing) and seed with the default config — the Phase 15
+    // known-good ec=80/es=30. Subsequent visits go through the proposer.
+    let is_bootstrap = history.is_empty();
 
     // Current champion (if any) is the promoted config.
     let champion = deps.promotion_registry.get_current(&event.index_name).await;
@@ -367,24 +367,26 @@ async fn run_one_cycle(
         history.iter().find(|t| t.id == p.trial_id).cloned()
     });
 
-    // Propose the next config.
-    let Some(next_config) = propose_next_config(&history, champion_trial.as_ref()) else {
-        return Err("proposer returned None — search space exhausted".into());
+    let (next_config, harness_name) = if is_bootstrap {
+        let name = ensure_auto_harness(&event.index_name, deps).await?;
+        (HnswConfig::default(), name)
+    } else {
+        let Some(cfg) = propose_next_config(&history, champion_trial.as_ref()) else {
+            return Err("proposer returned None — search space exhausted".into());
+        };
+        // Validate bounds defensively.
+        if !(10..=400).contains(&cfg.ef_construction) {
+            return Err(format!("proposed ef_construction={} out of bounds", cfg.ef_construction));
+        }
+        if !(10..=200).contains(&cfg.ef_search) {
+            return Err(format!("proposed ef_search={} out of bounds", cfg.ef_search));
+        }
+        // A future refinement: remember per-index "canonical harness" on
+        // the index metadata. For now: latest wins.
+        let hname = history.last().unwrap().eval_set.clone();
+        (cfg, hname)
     };
-
-    // Validate bounds defensively.
-    if !(10..=400).contains(&next_config.ef_construction) {
-        return Err(format!("proposed ef_construction={} out of bounds", next_config.ef_construction));
-    }
-    if !(10..=200).contains(&next_config.ef_search) {
-        return Err(format!("proposed ef_search={} out of bounds", next_config.ef_search));
-    }
-
-    // Need a harness to measure. Use the most recent one in history.
-    // (A future refinement: remember per-index "canonical harness" on
-    // the index metadata. For now: latest wins.)
-    let harness_name = history.last().unwrap().eval_set.clone();
-    let mut harness_set = harness::EvalSet::load(&deps.store, &harness_name).await
+    let mut harness_set = deps.harness_store.load_for_index(&event.index_name, &harness_name).await
         .map_err(|e| format!("load harness '{harness_name}': {e}"))?;
 
     let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await
@@ -393,7 +395,7 @@ async fn run_one_cycle(
     if !harness_set.ground_truth_built {
         harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await
             .map_err(|e| format!("ground truth: {e}"))?;
-        harness_set.save(&deps.store).await.ok();
+        deps.harness_store.save(&harness_set).await.ok();
     }
 
     // Build + bench.
@@ -428,10 +430,13 @@ async fn run_one_cycle(
     let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
     let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
 
-    let note = match &event.reason {
-        TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
-        TriggerReason::Manual => "agent: manual".to_string(),
-        TriggerReason::Periodic => "agent: periodic".to_string(),
+    let note = {
+        let base = match &event.reason {
+            TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
+            TriggerReason::Manual => "agent: manual".to_string(),
+            TriggerReason::Periodic => "agent: periodic".to_string(),
+        };
+        if is_bootstrap { format!("{base} bootstrap") } else { base }
     };
 
     let trial = Trial {
@@ -597,19 +602,92 @@ pub fn propose_next_config(history: &[Trial], champion: Option<&Trial>) -> Optio
 
 // -------- Helpers --------
 
-/// Find an index to poke on a periodic wake. Strategy: the one with the
-/// most recent trial. If nothing's been trialed yet, return None.
+/// Minimum vectors for an index to be worth auto-tuning. Below this,
+/// HNSW's win over brute-force is too small to matter and trial budget
+/// is better spent on bigger indexes.
+const AUTOTUNE_MIN_VECTORS: usize = 1_000;
+
+/// Find an index to poke on a periodic wake. Strategy: least-recently-tuned
+/// wins — pick the index (from `IndexRegistry`, not just promoted ones)
+/// whose most-recent trial is oldest, so trial budget spreads across the
+/// whole portfolio. Never-trialed indexes sort first (None < Some), which
+/// is exactly what we want: bootstrap them on their first visit.
+///
+/// Why not "most recently promoted" (the original strategy): a converged
+/// index like `threat_intel_v1` sits at recall=1.0 on a minimal graph and
+/// can't be improved — but it was always the freshest promotion, so the
+/// agent burned every trial on it while 40+ other indexes got zero
+/// attention. Rotating by last-trial-time fixes that without any explicit
+/// convergence detection.
+///
+/// Why IndexRegistry instead of PromotionRegistry: on a fresh system only
+/// a handful of indexes have ever been promoted, so promotion-based
+/// picking starves new ones. IndexRegistry lists every live index the
+/// platform knows about; filtering by `AUTOTUNE_MIN_VECTORS` keeps the
+/// proposer off indexes where the graph config doesn't matter.
+///
+/// Cost: N extra journal reads per periodic tick, where N = eligible
+/// indexes. Fine at dozens; if the portfolio grows into thousands, cache
+/// `last_trial_at` on the IndexMeta.
 async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> {
-    // `/agent` runs against any index that has a trial journal. We don't
-    // have a "list all journals" helper, so we derive candidates from the
-    // promotion registry (indexes with a human ever promoting are live).
-    let promos = deps.promotion_registry.list_all().await.ok()?;
-    // Prefer the one most recently promoted — it's the one a human cares
-    // about right now.
-    promos.into_iter()
-        .filter_map(|f| f.current.map(|c| (f.index_name, c.promoted_at)))
-        .max_by_key(|(_, at)| *at)
-        .map(|(name, _)| name)
+    let candidates: Vec<String> = deps
+        .index_registry
+        .list(None, None)
+        .await
+        .into_iter()
+        .filter(|m| m.chunk_count >= AUTOTUNE_MIN_VECTORS)
+        .map(|m| m.index_name)
+        .collect();
+    if candidates.is_empty() { return None; }
+
+    let mut scored: Vec<(String, Option<DateTime<Utc>>)> = Vec::with_capacity(candidates.len());
+    for name in candidates {
+        let last_trial_at = deps
+            .trial_journal
+            .list(&name)
+            .await
+            .ok()
+            .and_then(|trials| trials.into_iter().map(|t| t.created_at).max());
+        scored.push((name, last_trial_at));
+    }
+
+    scored.into_iter().min_by_key(|(_, last)| *last).map(|(n, _)| n)
+}
+
+/// On the first visit to an index, load or synthesize an eval harness so
+/// bootstrap trials have something to measure recall against. Returns the
+/// harness name. Synthetic harnesses sample 20 chunks and use their first
+/// 200 chars as self-queries — the ground truth is computed by brute-force
+/// cosine, so recall numbers are real (a config that finds the chunk in
+/// top-k counts as a hit).
+async fn ensure_auto_harness(index_name: &str, deps: &AgentDeps) -> Result<String, String> {
+    let name = format!("{index_name}_auto");
+
+    if deps.harness_store.load_for_index(index_name, &name).await.is_ok() {
+        return Ok(name);
+    }
+
+    let embeddings = deps
+        .embedding_cache
+        .get_or_load(index_name)
+        .await
+        .map_err(|e| format!("load embeddings for auto-harness: {e}"))?;
+    if embeddings.is_empty() {
+        return Err(format!("index '{index_name}' has no embeddings — cannot bootstrap"));
+    }
+
+    let mut eval = harness::synthetic_from_chunks(&name, index_name, &embeddings, 20, 10);
+    harness::compute_ground_truth(&mut eval, &embeddings, &deps.ai_client)
+        .await
+        .map_err(|e| format!("auto-harness ground truth: {e}"))?;
+    deps.harness_store.save(&eval).await
+        .map_err(|e| format!("save auto-harness: {e}"))?;
+
+    tracing::info!(
+        "agent: bootstrapped harness '{}' for index '{}' (20 self-queries, k=10)",
+        name, index_name
+    );
+    Ok(name)
 }
 
 async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool {

crates/vectord/src/autotune.rs

@@ -28,7 +28,7 @@ use aibridge::client::AiClient;
 use catalogd::registry::Registry as CatalogRegistry;
 
 use crate::embedding_cache::EmbeddingCache;
-use crate::harness;
+use crate::harness::{self, HarnessStore};
 use crate::hnsw::HnswStore;
 use crate::index_registry::IndexRegistry;
 use crate::jobs::JobTracker;
@@ -128,7 +128,7 @@ fn pick_winner(trials: &[Trial], min_recall: f32) -> Option<&Trial> {
 #[allow(clippy::too_many_arguments)]
 pub async fn run_autotune(
     req: AutotuneRequest,
-    store: &Arc<dyn ObjectStore>,
+    _store: &Arc<dyn ObjectStore>,
     catalog: &CatalogRegistry,
     ai_client: &AiClient,
     embedding_cache: &EmbeddingCache,
@@ -136,6 +136,7 @@ pub async fn run_autotune(
     index_registry: &IndexRegistry,
     trial_journal: &TrialJournal,
     promotion_registry: &PromotionRegistry,
+    harness_store: &HarnessStore,
     _job_tracker: &JobTracker,
 ) -> Result<AutotuneResult, String> {
     let t0 = std::time::Instant::now();
@@ -146,8 +147,11 @@ pub async fn run_autotune(
     }
     let _ = catalog; // reserved for future audit emission
 
-    // Load the harness once, compute ground truth once.
-    let mut harness_set = harness::EvalSet::load(store, &req.harness)
+    // Load the harness once, compute ground truth once. Harness resolves
+    // to the index's bucket via HarnessStore, with primary as fallback for
+    // pre-federation evals.
+    let mut harness_set = harness_store
+        .load_for_index(&req.index_name, &req.harness)
         .await
         .map_err(|e| format!("load harness: {e}"))?;
 
@@ -160,7 +164,7 @@ pub async fn run_autotune(
         harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client)
             .await
             .map_err(|e| format!("ground truth: {e}"))?;
-        harness_set.save(store).await.ok();
+        harness_store.save(&harness_set).await.ok();
     }
 
     let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid));

crates/vectord/src/doc_drift.rs

@@ -0,0 +1,166 @@
+// Phase 45 slice 3 — context7 bridge client for doc-drift detection.
+//
+// Calls the Bun context7 bridge on :3900 (mcp-server/context7_bridge.ts)
+// which itself wraps context7's public API. For each DocRef on a
+// playbook, queries /docs/:tool/diff?since=<snippet_hash> and parses
+// the response.
+//
+// Kept deliberately thin: no caching here (bridge caches for 5 min),
+// no retry logic (transient fail = report as "unknown" drift status,
+// don't flag). The handler decides whether to flag the playbook based
+// on whether ANY tool came back drifted=true.
+
+use serde::Deserialize;
+use std::time::Duration;
+
+use crate::playbook_memory::DocRef;
+
+const DEFAULT_BRIDGE_URL: &str = "http://localhost:3900";
+const CALL_TIMEOUT_SECS: u64 = 15;
+
+#[derive(Debug, Clone)]
+pub struct DriftCheck {
+    pub tool: String,
+    pub version_seen: String,
+    pub outcome: DriftOutcome,
+}
+
+#[derive(Debug, Clone)]
+pub enum DriftOutcome {
+    /// Bridge reports drift (current snippet_hash != previous).
+    Drifted { current_snippet_hash: String, source_url: Option<String> },
+    /// Bridge reports no drift — the stored snippet_hash still matches
+    /// the current context7 docs.
+    Unchanged,
+    /// Bridge unreachable, 404 on the tool, or returned unparseable
+    /// data. We deliberately don't flag on these — a down bridge
+    /// shouldn't silently mark every playbook drift-flagged.
+    Unknown { reason: String },
+}
+
+#[derive(Debug, Clone, Deserialize)]
+struct BridgeDiffResponse {
+    drifted: bool,
+    current_snippet_hash: Option<String>,
+    source_url: Option<String>,
+}
+
+pub struct DriftCheckerConfig {
+    pub bridge_url: String,
+}
+
+impl Default for DriftCheckerConfig {
+    fn default() -> Self {
+        Self {
+            bridge_url: std::env::var("LH_BRIDGE_URL")
+                .unwrap_or_else(|_| DEFAULT_BRIDGE_URL.to_string()),
+        }
+    }
+}
+
+/// For every doc_ref, ask the bridge whether it drifted against the
+/// recorded snippet_hash. Returns per-tool outcomes.
+pub async fn check_all_refs(
+    cfg: &DriftCheckerConfig,
+    doc_refs: &[DocRef],
+) -> Vec<DriftCheck> {
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(CALL_TIMEOUT_SECS))
+        .build()
+        .expect("reqwest client build");
+
+    let mut out = Vec::with_capacity(doc_refs.len());
+    for r in doc_refs {
+        let hash = r.snippet_hash.as_deref().unwrap_or("");
+        if hash.is_empty() {
+            // No hash to compare against — can't detect drift. Report
+            // unknown so the caller isn't forced to flag.
+            out.push(DriftCheck {
+                tool: r.tool.clone(),
+                version_seen: r.version_seen.clone(),
+                outcome: DriftOutcome::Unknown {
+                    reason: "no snippet_hash recorded on doc_ref".into(),
+                },
+            });
+            continue;
+        }
+        let url = format!(
+            "{}/docs/{}/diff?since={}",
+            cfg.bridge_url.trim_end_matches('/'),
+            urlencoding_minimal(&r.tool),
+            urlencoding_minimal(hash),
+        );
+        let outcome = match client.get(&url).send().await {
+            Err(e) => DriftOutcome::Unknown {
+                reason: format!("bridge unreachable: {}", e),
+            },
+            Ok(resp) => {
+                if resp.status() == reqwest::StatusCode::NOT_FOUND {
+                    DriftOutcome::Unknown {
+                        reason: format!("bridge 404 — no context7 library for tool '{}'", r.tool),
+                    }
+                } else if !resp.status().is_success() {
+                    let status = resp.status();
+                    DriftOutcome::Unknown {
+                        reason: format!("bridge {}: {}", status, resp.text().await.unwrap_or_default()),
+                    }
+                } else {
+                    match resp.json::<BridgeDiffResponse>().await {
+                        Err(e) => DriftOutcome::Unknown {
+                            reason: format!("bridge response parse: {}", e),
+                        },
+                        Ok(body) => {
+                            if body.drifted {
+                                DriftOutcome::Drifted {
+                                    current_snippet_hash: body.current_snippet_hash.unwrap_or_default(),
+                                    source_url: body.source_url,
+                                }
+                            } else {
+                                DriftOutcome::Unchanged
+                            }
+                        }
+                    }
+                }
+            }
+        };
+        out.push(DriftCheck {
+            tool: r.tool.clone(),
+            version_seen: r.version_seen.clone(),
+            outcome,
+        });
+    }
+    out
+}
+
+/// Minimal URL path+query encoder for ASCII tool names. We're not
+/// pulling in the `urlencoding` crate for this single use — tool
+/// names are short alphanumeric (docker/terraform/react), snippet
+/// hashes are hex. Handles space + `/` + `?` + `&` defensively so
+/// unusual tool names don't corrupt the URL.
+fn urlencoding_minimal(s: &str) -> String {
+    let mut out = String::with_capacity(s.len());
+    for b in s.bytes() {
+        match b {
+            b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => out.push(b as char),
+            _ => out.push_str(&format!("%{:02X}", b)),
+        }
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn urlencoding_handles_ascii_safe_chars_passthrough() {
+        assert_eq!(urlencoding_minimal("docker"), "docker");
+        assert_eq!(urlencoding_minimal("next.js"), "next.js");
+    }
+
+    #[test]
+    fn urlencoding_encodes_slash_and_space() {
+        assert_eq!(urlencoding_minimal("foo/bar"), "foo%2Fbar");
+        assert_eq!(urlencoding_minimal("foo bar"), "foo%20bar");
+    }
+}

crates/vectord/src/harness.rs

@@ -10,11 +10,14 @@
 use chrono::{DateTime, Utc};
 use object_store::ObjectStore;
 use serde::{Deserialize, Serialize};
+use std::collections::HashSet;
 use std::sync::Arc;
 
 use aibridge::client::{AiClient, EmbedRequest};
 use storaged::ops;
+use storaged::registry::BucketRegistry;
 
+use crate::index_registry::IndexRegistry;
 use crate::store::StoredEmbedding;
 
 /// A single eval query with optional pre-computed ground truth.
@@ -85,6 +88,108 @@ impl EvalSet {
     }
 }
 
+/// Federation-aware wrapper around EvalSet persistence. Mirrors the
+/// `TrialJournal` / `PromotionRegistry` pattern: harness files colocate
+/// with their index's bucket (looked up via `IndexMeta.bucket`), falling
+/// back to `primary` for indexes the registry has never seen. Legacy
+/// harnesses predating federation remain discoverable — lookups transparently
+/// try the resolved bucket first, then `primary` as a fallback. Cross-bucket
+/// listing dedupes across every registered bucket so `GET /hnsw/evals`
+/// returns a complete picture.
+#[derive(Clone)]
+pub struct HarnessStore {
+    buckets: Arc<BucketRegistry>,
+    index_registry: IndexRegistry,
+}
+
+impl HarnessStore {
+    pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
+        Self { buckets, index_registry }
+    }
+
+    /// Resolve which bucket holds this index's eval artifacts. Indexes the
+    /// registry has never heard of fall through to `primary`.
+    async fn bucket_for_index(&self, index_name: &str) -> String {
+        self.index_registry
+            .get(index_name)
+            .await
+            .map(|m| m.bucket)
+            .unwrap_or_else(|| "primary".to_string())
+    }
+
+    /// Save to the bucket that owns `eval.index_name`. Writes under the
+    /// standard `_hnsw_evals/{name}.json` prefix of the resolved bucket.
+    pub async fn save(&self, eval: &EvalSet) -> Result<(), String> {
+        let bucket = self.bucket_for_index(&eval.index_name).await;
+        let store = self.buckets.get(&bucket)?;
+        eval.save(&store).await
+    }
+
+    /// Load a harness by name, given the index it belongs to. Tries the
+    /// index's bucket first; if the file is absent AND the resolved bucket
+    /// isn't `primary`, falls through to `primary` so pre-federation evals
+    /// remain reachable without migration.
+    pub async fn load_for_index(
+        &self,
+        index_name: &str,
+        harness_name: &str,
+    ) -> Result<EvalSet, String> {
+        let bucket = self.bucket_for_index(index_name).await;
+        let primary_store = self.buckets.get("primary")?;
+        let store = self.buckets.get(&bucket)?;
+
+        match EvalSet::load(&store, harness_name).await {
+            Ok(e) => Ok(e),
+            Err(e) if bucket != "primary" => EvalSet::load(&primary_store, harness_name)
+                .await
+                .map_err(|primary_err| format!("{bucket}: {e}; primary fallback: {primary_err}")),
+            Err(e) => Err(e),
+        }
+    }
+
+    /// Find a harness by name without knowing which index it belongs to —
+    /// used by `GET /hnsw/evals/{name}`. Scans every registered bucket;
+    /// first hit wins. Primary is searched first so pre-federation evals
+    /// with the same name as a federated one resolve deterministically.
+    pub async fn get_any(&self, harness_name: &str) -> Result<EvalSet, String> {
+        let bucket_infos = self.buckets.list().await;
+        let mut ordered: Vec<String> = bucket_infos.iter().map(|b| b.name.clone()).collect();
+        ordered.sort_by_key(|n| if n == "primary" { 0 } else { 1 });
+
+        let mut last_err = None;
+        for b in ordered {
+            let store = match self.buckets.get(&b) {
+                Ok(s) => s,
+                Err(e) => { last_err = Some(e); continue; }
+            };
+            match EvalSet::load(&store, harness_name).await {
+                Ok(e) => return Ok(e),
+                Err(e) => { last_err = Some(e); }
+            }
+        }
+        Err(last_err.unwrap_or_else(|| format!("no buckets registered for eval '{harness_name}'")))
+    }
+
+    /// Union of every harness name across every registered bucket.
+    /// Duplicates (same name in multiple buckets — pathological but
+    /// possible after a manual migration) are collapsed.
+    pub async fn list_all(&self) -> Vec<String> {
+        let mut all: HashSet<String> = HashSet::new();
+        for b in self.buckets.list().await {
+            let store = match self.buckets.get(&b.name) {
+                Ok(s) => s,
+                Err(_) => continue,
+            };
+            if let Ok(names) = EvalSet::list(&store).await {
+                all.extend(names);
+            }
+        }
+        let mut out: Vec<String> = all.into_iter().collect();
+        out.sort();
+        out
+    }
+}
+
 /// Cosine similarity for two same-length f32 slices.
 fn cosine(a: &[f32], b: &[f32]) -> f32 {
     let mut dot = 0.0f32;
@@ -127,13 +232,15 @@ pub fn recall_at_k(predicted: &[String], ground_truth: &[String], k: usize) -> f
     if ground_truth.is_empty() || k == 0 {
         return 0.0;
     }
+    // Set-intersection recall@k. Previous implementation counted duplicates
+    // in `predicted` (a corpus with repeated chunks — e.g. cached LLM
+    // responses — returns the same doc_id multiple times via HNSW), which
+    // inflated recall above 1.0 and poisoned promotion decisions.
     let gt_set: std::collections::HashSet<&String> =
         ground_truth.iter().take(k).collect();
-    let hits = predicted
-        .iter()
-        .take(k)
-        .filter(|d| gt_set.contains(d))
-        .count();
+    let pred_set: std::collections::HashSet<&String> =
+        predicted.iter().take(k).collect();
+    let hits = pred_set.intersection(&gt_set).count();
     hits as f32 / gt_set.len() as f32
 }
 
@@ -214,3 +321,67 @@ pub fn synthetic_from_chunks(
         ground_truth_built: false,
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn s(v: &[&str]) -> Vec<String> { v.iter().map(|x| x.to_string()).collect() }
+
+    #[test]
+    fn recall_empty_ground_truth_is_zero() {
+        assert_eq!(recall_at_k(&s(&["a", "b"]), &[], 10), 0.0);
+    }
+
+    #[test]
+    fn recall_k_zero_is_zero() {
+        assert_eq!(recall_at_k(&s(&["a"]), &s(&["a"]), 0), 0.0);
+    }
+
+    #[test]
+    fn recall_perfect_match_equals_one() {
+        let pred = s(&["a", "b", "c"]);
+        let gt = s(&["a", "b", "c"]);
+        assert!((recall_at_k(&pred, &gt, 3) - 1.0).abs() < 1e-6);
+    }
+
+    #[test]
+    fn recall_half_match() {
+        let pred = s(&["a", "b", "x", "y"]);
+        let gt = s(&["a", "b", "c", "d"]);
+        assert!((recall_at_k(&pred, &gt, 4) - 0.5).abs() < 1e-6);
+    }
+
+    #[test]
+    fn recall_duplicates_in_predicted_do_not_inflate() {
+        // Regression guard: the previous implementation counted each
+        // duplicate in `predicted` separately, which could push recall
+        // above 1.0 on corpora with repeated chunks (cached responses etc).
+        // Set-intersection semantics keep it bounded in [0, 1].
+        let pred = s(&["a", "a", "a", "a"]);
+        let gt = s(&["a", "b", "c", "d"]);
+        let r = recall_at_k(&pred, &gt, 4);
+        assert!(r <= 1.0, "recall {r} must not exceed 1.0");
+        // One unique match out of four in gt = 0.25.
+        assert!((r - 0.25).abs() < 1e-6);
+    }
+
+    #[test]
+    fn recall_duplicates_in_ground_truth_handled() {
+        // gt with dupes reduces effective |gt|; matching one hits all.
+        let pred = s(&["x"]);
+        let gt = s(&["x", "x", "x"]);
+        let r = recall_at_k(&pred, &gt, 3);
+        assert!(r <= 1.0);
+        assert!((r - 1.0).abs() < 1e-6); // |pred ∩ gt| / |gt_set| = 1/1
+    }
+
+    #[test]
+    fn recall_respects_k_bound() {
+        // k=2 means only the first 2 of pred and gt count toward the set.
+        let pred = s(&["a", "b", "c", "d"]);
+        let gt = s(&["a", "b", "c", "d"]);
+        let r = recall_at_k(&pred, &gt, 2);
+        assert!((r - 1.0).abs() < 1e-6);
+    }
+}

crates/vectord/src/index_registry.rs

@@ -40,6 +40,12 @@ pub struct IndexMeta {
     /// `Lance` means it points at a Lance dataset directory.
     #[serde(default)]
     pub vector_backend: shared::types::VectorBackend,
+    /// ADR-020: prefix prepended to doc_ids during embedding. If set,
+    /// hybrid search strips this prefix to match against SQL primary keys.
+    /// None = doc_ids ARE the raw primary keys (no stripping needed).
+    /// Existing indexes: "W-", "CAND-", "W500K-", etc.
+    #[serde(default)]
+    pub id_prefix: Option<String>,
 }
 
 fn default_bucket() -> String { "primary".to_string() }

crates/vectord/src/jobs.rs

@@ -22,7 +22,12 @@ pub struct Job {
     pub status: JobStatus,
     pub index_name: String,
     pub total_chunks: usize,
+    /// How many chunks have been embedded so far.
+    /// Also serialized as "processed" for backward-compat with monitoring scripts.
     pub embedded_chunks: usize,
+    #[serde(rename = "processed")]
+    pub processed_alias: usize,
+    pub total: usize,
     pub progress_pct: f32,
     pub storage_key: Option<String>,
     pub error: Option<String>,
@@ -53,6 +58,8 @@ impl JobTracker {
             index_name: index_name.to_string(),
             total_chunks,
             embedded_chunks: 0,
+            processed_alias: 0,
+            total: total_chunks,
             progress_pct: 0.0,
             storage_key: None,
             error: None,
@@ -69,6 +76,7 @@ impl JobTracker {
         let mut jobs = self.jobs.write().await;
         if let Some(job) = jobs.get_mut(id) {
             job.embedded_chunks = embedded;
+            job.processed_alias = embedded;  // keep alias in sync
             job.progress_pct = if job.total_chunks > 0 {
                 (embedded as f32 / job.total_chunks as f32) * 100.0
             } else {
@@ -84,6 +92,7 @@ impl JobTracker {
         if let Some(job) = jobs.get_mut(id) {
             job.status = JobStatus::Completed;
             job.embedded_chunks = job.total_chunks;
+            job.processed_alias = job.total_chunks;
             job.progress_pct = 100.0;
             job.storage_key = Some(storage_key);
             job.completed_at = Some(chrono::Utc::now().to_rfc3339());

crates/vectord/src/lib.rs

@@ -7,6 +7,8 @@ pub mod harness;
 pub mod hnsw;
 pub mod index_registry;
 pub mod jobs;
+pub mod playbook_memory;
+pub mod doc_drift;
 pub mod promotion;
 pub mod refresh;
 pub mod store;

crates/vectord/src/rag.rs

@@ -1,12 +1,92 @@
-/// RAG pipeline: question → embed → search → retrieve → generate answer.
+/// RAG pipeline: question → embed → search → rerank → generate answer.
+///
+/// The rerank step (added 2026-04-17) uses the LLM as a cross-encoder
+/// between retrieval and generation. This catches cases where the
+/// embedding model scores documents as similar but the content isn't
+/// actually relevant to the question — a known weakness of small
+/// general-purpose embed models on domain-specific text.
 
 use object_store::ObjectStore;
 use std::sync::Arc;
 
 use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
+use aibridge::continuation::{generate_continuable, ContinuableOpts, ResponseShape};
 use crate::search::{self, SearchResult};
 use crate::store;
 
+/// Cross-encoder rerank: ask the LLM to re-sort retrieved chunks by
+/// relevance. Falls back to the original order if the model returns
+/// garbage (which happens ~5% of the time with 7B models).
+async fn rerank(
+    question: &str,
+    mut results: Vec<SearchResult>,
+    ai_client: &AiClient,
+) -> Vec<SearchResult> {
+    if results.len() <= 1 {
+        return results;
+    }
+
+    let chunk_list: String = results.iter().enumerate().map(|(i, r)| {
+        let text: String = r.chunk_text.chars().take(200).collect();
+        format!("[{i}] {text}")
+    }).collect::<Vec<_>>().join("\n");
+
+    let resp = ai_client.generate(GenerateRequest {
+        prompt: format!(
+            "Rank these text chunks by relevance to the question.\n\
+            Return ONLY a comma-separated list of indices, most relevant first.\n\n\
+            Question: {question}\n\n\
+            Chunks:\n{chunk_list}\n\n\
+            Ranking:"
+        ),
+        model: None,
+        system: None,
+        temperature: Some(0.0),
+        max_tokens: Some(50),
+        // Reranker returns a comma-separated int list — pure structured
+        // output, zero benefit from hidden reasoning. Opt out to avoid
+        // the empty-response failure mode Phase 21 catalogued.
+        think: Some(false),
+    }).await;
+
+    match resp {
+        Ok(gen_resp) => {
+            let text = gen_resp.text.trim();
+            let indices: Vec<usize> = text
+                .split(|c: char| c == ',' || c.is_whitespace())
+                .filter_map(|s| s.trim().parse::<usize>().ok())
+                .filter(|&i| i < results.len())
+                .collect();
+
+            if indices.is_empty() {
+                tracing::debug!("reranker returned unparseable output: {text}");
+                return results;
+            }
+
+            let mut reranked: Vec<SearchResult> = Vec::with_capacity(results.len());
+            let mut used = vec![false; results.len()];
+            for &i in &indices {
+                if !used[i] {
+                    reranked.push(results[i].clone());
+                    used[i] = true;
+                }
+            }
+            // Append any the model didn't mention (preserves all results).
+            for (i, r) in results.drain(..).enumerate() {
+                if !used[i] {
+                    reranked.push(r);
+                }
+            }
+            tracing::info!("reranker reordered {}/{} chunks", indices.len(), reranked.len());
+            reranked
+        }
+        Err(e) => {
+            tracing::debug!("reranker failed (using original order): {e}");
+            results
+        }
+    }
+}
+
 /// Full RAG answer with provenance.
 #[derive(Debug, Clone, serde::Serialize)]
 pub struct RagResponse {
@@ -51,34 +131,53 @@ pub async fn query(
         });
     }
 
-    // 3. Build context from retrieved chunks
+    // 3. Rerank: ask the LLM to sort retrieved chunks by relevance.
+    //    This cross-encoder step catches cases where embedding similarity
+    //    is high but semantic relevance is low — common with small embed
+    //    models on domain-specific text.
+    let results = rerank(question, results, ai_client).await;
+
+    // 4. Build context from (reranked) top chunks
     let context: String = results.iter().enumerate().map(|(i, r)| {
         format!("[{}] (source: {}, doc: {}) {}", i + 1, r.source, r.doc_id, r.chunk_text)
     }).collect::<Vec<_>>().join("\n\n");
 
-    // 4. Generate answer
+    // 5. Generate answer
     tracing::info!("RAG: generating answer from {} chunks", results.len());
     let prompt = format!(
-        "You are a helpful assistant answering questions based on retrieved documents from a data system.\n\n\
-        Use ONLY the following context to answer. If the context doesn't contain enough information, say so.\n\
+        "You are a helpful assistant answering questions about a staffing database.\n\n\
+        Use ONLY the following retrieved records to answer. Be specific — cite names,\n\
+        numbers, skills, and cities from the records. If the context doesn't contain\n\
+        enough information to fully answer, say what you can and note what's missing.\n\
         Cite sources by their number [1], [2], etc.\n\n\
         Context:\n{context}\n\n\
         Question: {question}\n\n\
         Answer:"
     );
 
-    let gen_resp = ai_client.generate(GenerateRequest {
-        prompt,
-        model: None,
-        system: None,
-        temperature: Some(0.2),
-        max_tokens: Some(512),
-    }).await?;
+    // Route the answer call through Phase 21's generate_continuable so
+    // a thinking-model empty-response or a mid-JSON truncation self-
+    // repairs instead of silently returning half an answer. Shape is
+    // Text (the answer is prose, not JSON), think is Some(false) to
+    // opt out of hidden reasoning on the hot path. This is the first
+    // production caller of the Phase 21 primitives — see audit finding
+    // "Phase 21 Rust primitives are wired but not CALLED by any
+    // production surface" from 2026-04-21.
+    let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
+    cont_opts.max_tokens = Some(512);
+    cont_opts.temperature = Some(0.2);
+    cont_opts.shape = ResponseShape::Text;
+    cont_opts.think = Some(false);
+    let outcome = generate_continuable(ai_client, &prompt, &cont_opts).await?;
 
     Ok(RagResponse {
-        answer: gen_resp.text.trim().to_string(),
-        model: gen_resp.model,
+        answer: outcome.text.trim().to_string(),
+        // generate_continuable doesn't surface the model name (sidecar
+        // echoes whatever Ollama loaded). Use the configured tier model
+        // for now; if RAG needs to report the actual resolved model,
+        // the runner can add a post-call ps probe later.
+        model: "qwen2.5:latest".to_string(),
         sources: results,
-        tokens_generated: gen_resp.tokens_generated,
+        tokens_generated: None,
     })
 }

crates/vectord/src/refresh.rs

@@ -307,6 +307,7 @@ async fn try_update_index_meta(
         chunks_per_sec: 0.0,
         bucket: "primary".to_string(),
         vector_backend: shared::types::VectorBackend::Parquet,
+        id_prefix: None,
     };
     index_registry.register(meta).await
 }

data/_catalog/manifests/07c119c3-2e59-463c-89a8-7980a849b6d6.json

@@ -0,0 +1,159 @@
+{
+  "id": "07c119c3-2e59-463c-89a8-7980a849b6d6",
+  "name": "workers_500k",
+  "schema_fingerprint": "71686e497f5ee2fc77d50b13c39a46d89ca3d7673e9a0879e67c2db476764b25",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/workers_500k.parquet",
+      "size_bytes": 323794818,
+      "created_at": "2026-04-17T05:57:41.972944716Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:57:41.972945960Z",
+  "updated_at": "2026-04-17T05:57:41.973093967Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Int64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responsiveness",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "engagement",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "compliance",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "communications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "resume_text",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "workers_500k.csv",
+    "ingest_job": "ingest-1776405461972",
+    "ingest_timestamp": "2026-04-17T05:57:41.972944716Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 500000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/0efdef2d-52bc-4154-82d4-056bb3a1915d.json

@@ -0,0 +1,159 @@
+{
+  "id": "0efdef2d-52bc-4154-82d4-056bb3a1915d",
+  "name": "ethereal_workers",
+  "schema_fingerprint": "9a1286ffada5390b459b217f56263e5ebffec1520ea29bab0be2efc6d5381adc",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/ethereal_workers.parquet",
+      "size_bytes": 6991716,
+      "created_at": "2026-04-17T03:23:29.399966984Z"
+    }
+  ],
+  "created_at": "2026-04-17T03:23:29.399967772Z",
+  "updated_at": "2026-04-17T03:23:29.400227081Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Int64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responsiveness",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "engagement",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "compliance",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "communications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "resume_text",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "ethereal_workers.csv",
+    "ingest_job": "ingest-1776396209399",
+    "ingest_timestamp": "2026-04-17T03:23:29.399966984Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 10000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/0fd78303-9ad4-45fd-90d7-db95607d9ab1.json

@@ -1,23 +1,97 @@
 {
   "id": "0fd78303-9ad4-45fd-90d7-db95607d9ab1",
   "name": "timesheets",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "806946533e133a4dd032f500644c9f702a46151fbb5bbd3b503c207dac86893b",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/timesheets.parquet",
       "size_bytes": 17539932,
       "created_at": "2026-03-27T14:42:43.922019299Z"
     }
   ],
   "created_at": "2026-03-27T14:42:43.922025703Z",
-  "updated_at": "2026-03-27T14:42:43.922025703Z",
+  "updated_at": "2026-04-17T02:45:58.282412872Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "timesheet_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "placement_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "candidate_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "client_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "hours_regular",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "hours_overtime",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "bill_total",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pay_total",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "week_ending",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "approved",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 1000000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/1339f3d6-7677-47fb-8182-5f8e43f27cde.json

@@ -1,23 +1,104 @@
 {
   "id": "1339f3d6-7677-47fb-8182-5f8e43f27cde",
   "name": "job_orders",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "579a0593ca7c0e22ba53702821757f189077e1ddff7fd592c0097feed446a99a",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/job_orders.parquet",
       "size_bytes": 905534,
       "created_at": "2026-03-27T14:42:38.935718195Z"
     }
   ],
   "created_at": "2026-03-27T14:42:38.935724058Z",
-  "updated_at": "2026-03-27T14:42:38.935724058Z",
+  "updated_at": "2026-04-17T02:45:57.664902260Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "job_order_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "client_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "title",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "vertical",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "bill_rate",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pay_rate",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "status",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "description",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 15000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/15d889ef-2dba-4d1f-8ce0-22f7def35155.json

@@ -0,0 +1,54 @@
+{
+  "id": "15d889ef-2dba-4d1f-8ce0-22f7def35155",
+  "name": "scanned_resume",
+  "schema_fingerprint": "d3bc44bcafc990711442db3eecf25c0758232e3fccf6345200b1813fae5868c2",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/scanned_resume.parquet",
+      "size_bytes": 3097,
+      "created_at": "2026-04-17T01:44:33.927277187Z"
+    }
+  ],
+  "created_at": "2026-04-17T01:44:33.927280048Z",
+  "updated_at": "2026-04-17T01:44:33.927528900Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "source_file",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "page_number",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "text_content",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "pdf",
+    "source_file": "test_scanned_resume.pdf",
+    "ingest_job": "ingest-1776390273927",
+    "ingest_timestamp": "2026-04-17T01:44:33.927277187Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 1,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/17119168-cfd2-43b2-a4dc-ff2a2c5fd086.json

@@ -0,0 +1,90 @@
+{
+  "id": "17119168-cfd2-43b2-a4dc-ff2a2c5fd086",
+  "name": "demo_customers",
+  "schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/demo_customers.parquet",
+      "size_bytes": 4583,
+      "created_at": "2026-04-17T01:35:00.165218923Z"
+    }
+  ],
+  "created_at": "2026-04-17T01:35:00.165220510Z",
+  "updated_at": "2026-04-17T05:56:44.964702014Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "tier",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "lifetime_spend",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "is_active",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "signed_up_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "notes",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 11,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/1d8a065e-59c1-45ce-967b-398bc8370cbb.json

@@ -1,23 +1,132 @@
 {
   "id": "1d8a065e-59c1-45ce-967b-398bc8370cbb",
   "name": "candidates",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "206360ab312e4c13679ed0ab4ccb3702250cc3e2176cc669d41471584c556c73",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/candidates.parquet",
       "size_bytes": 10592165,
       "created_at": "2026-03-27T14:42:38.823368759Z"
     }
   ],
   "created_at": "2026-03-27T14:42:38.823374843Z",
-  "updated_at": "2026-03-27T14:42:38.823374843Z",
+  "updated_at": "2026-04-17T02:45:57.722237378Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "candidate_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "first_name",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "last_name",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "phone",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "vertical",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "resume_summary",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "status",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "source",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "min_pay_rate",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "years_experience",
+      "data_type": "Int64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 100000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/2b8e8b01-a44e-49c5-a103-a1b55898bee1.json

@@ -0,0 +1,110 @@
+{
+  "id": "2b8e8b01-a44e-49c5-a103-a1b55898bee1",
+  "name": "kb_memory_entries",
+  "schema_fingerprint": "15dbebd0abb906577e11cbb73083abeb8961b8c603a2472cafa8c46100c3fb1c",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_memory_entries.parquet",
+      "size_bytes": 8795,
+      "created_at": "2026-04-17T05:22:06.356227657Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.356228933Z",
+  "updated_at": "2026-04-17T05:22:06.356776285Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "content",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "content_type",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "source",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "session_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "embedding",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "metadata",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "importance_score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "accessed_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "access_count",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "postgresql",
+    "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
+    "ingest_job": "pg-stream-1776403326356",
+    "ingest_timestamp": "2026-04-17T05:22:06.356227657Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 13,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/2db2d92f-1171-4f2f-a707-ad387b6cd464.json

@@ -0,0 +1,117 @@
+{
+  "id": "2db2d92f-1171-4f2f-a707-ad387b6cd464",
+  "name": "onboard_live_1776729919",
+  "schema_fingerprint": "9cdf2fc924bad875e63bd12a82dfb1cd080c5cbba71f671b4a2d41e746df187d",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/onboard_live_1776729919.parquet",
+      "size_bytes": 31444,
+      "created_at": "2026-04-21T00:05:19.717239967Z"
+    }
+  ],
+  "created_at": "2026-04-21T00:05:19.717241621Z",
+  "updated_at": "2026-04-21T00:05:19.717370050Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "sample_1776729919.csv",
+    "ingest_job": "ingest-1776729919717",
+    "ingest_timestamp": "2026-04-21T00:05:19.717239967Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 173,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/3178f0da-5985-4731-b343-162dd415f3be.json

@@ -0,0 +1,75 @@
+{
+  "id": "3178f0da-5985-4731-b343-162dd415f3be",
+  "name": "kb_self_reports",
+  "schema_fingerprint": "76382b028b92bb0b361306dced3d773af0ee8de6aa160cedfc4d649f2860167a",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_self_reports.parquet",
+      "size_bytes": 51134,
+      "created_at": "2026-04-17T04:14:34.355419647Z"
+    }
+  ],
+  "created_at": "2026-04-17T04:14:34.355420701Z",
+  "updated_at": "2026-04-17T06:06:21.358604060Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "report_type",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "model",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "report",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "data_size",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "postgresql",
+    "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
+    "ingest_job": "pg-stream-1776399274355",
+    "ingest_timestamp": "2026-04-17T04:14:34.355419647Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 11,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json

@@ -0,0 +1,26 @@
+{
+  "id": "32ee74a0-59b4-4e5b-8edb-70c9347a4bf3",
+  "name": "successful_playbooks_live",
+  "schema_fingerprint": "4e28e94650d9f5689e952674aebfd09db70edfcba617713081c0a83bf59121ce",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/successful_playbooks_live.parquet",
+      "size_bytes": 284341,
+      "created_at": "2026-04-22T03:28:28.343840136Z"
+    }
+  ],
+  "created_at": "2026-04-20T11:07:57.308050648Z",
+  "updated_at": "2026-04-22T03:28:28.343843823Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": null,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/33544879-fcfa-4458-b782-a4ca8dafed08.json

@@ -0,0 +1,159 @@
+{
+  "id": "33544879-fcfa-4458-b782-a4ca8dafed08",
+  "name": "workers_100k",
+  "schema_fingerprint": "1cc2a0caf174aa9bf67b1e2cb7858a312a0c94d604d906f807eb5610864ff70e",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/workers_100k.parquet",
+      "size_bytes": 65225188,
+      "created_at": "2026-04-17T04:54:03.306737596Z"
+    }
+  ],
+  "created_at": "2026-04-17T04:54:03.306740279Z",
+  "updated_at": "2026-04-17T04:54:03.306927852Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Int64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responsiveness",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "engagement",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "compliance",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "communications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "resume_text",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "workers_100k.csv",
+    "ingest_job": "ingest-1776401643306",
+    "ingest_timestamp": "2026-04-17T04:54:03.306737596Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 100000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/35f36c38-5cd3-438e-b93d-7ff7720f0346.json

@@ -0,0 +1,90 @@
+{
+  "id": "35f36c38-5cd3-438e-b93d-7ff7720f0346",
+  "name": "kb_meta_runs",
+  "schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_meta_runs.parquet",
+      "size_bytes": 886387,
+      "created_at": "2026-04-17T05:22:06.314413387Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.314414515Z",
+  "updated_at": "2026-04-17T06:06:21.300610797Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pipeline_id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "iteration",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "stage_results",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "final_output",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "model_config",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "duration_ms",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 48,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/3b12ae24-17d4-4325-92bf-f3155982f3bf.json

@@ -4,7 +4,7 @@
   "schema_fingerprint": "30c0e31f0963e6f4af02131bbb9ea246fbbd068b849b833565a4b28211fbc90b",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/users.parquet",
       "size_bytes": 2012,
       "created_at": "2026-03-28T01:38:59.904968123Z"

data/_catalog/manifests/3c2579b4-f3f3-4875-95fa-58d8b49ad94c.json

@@ -4,7 +4,7 @@
   "schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/meta_runs.parquet",
       "size_bytes": 729773,
       "created_at": "2026-03-28T01:38:57.380576453Z"

data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json

@@ -0,0 +1,117 @@
+{
+  "id": "564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7",
+  "name": "client_workerskjkk",
+  "schema_fingerprint": "cdfe85348885ddf329e5e6e9bf0e2c75c92d1a86fdb0fd3875ed46e3f93c4d82",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/client_workerskjkk.parquet",
+      "size_bytes": 32201,
+      "created_at": "2026-04-21T00:49:04.623625149Z"
+    }
+  ],
+  "created_at": "2026-04-21T00:49:04.623626738Z",
+  "updated_at": "2026-04-21T00:49:04.623901788Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "staffing_roster_sample.csv",
+    "ingest_job": "ingest-1776732544623",
+    "ingest_timestamp": "2026-04-21T00:49:04.623625149Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 180,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/57e557a8-4754-47c9-9636-e01d39fc20f2.json

@@ -0,0 +1,54 @@
+{
+  "id": "57e557a8-4754-47c9-9636-e01d39fc20f2",
+  "name": "animals",
+  "schema_fingerprint": "b024b907f151889aca5407fc46fe1821d2600f587cb1a26742843597a66436f1",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/animals.parquet",
+      "size_bytes": 1107,
+      "created_at": "2026-04-16T13:47:57.033455363Z"
+    }
+  ],
+  "created_at": "2026-04-16T13:47:57.033456007Z",
+  "updated_at": "2026-04-16T13:47:57.033631067Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "animal_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "species",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "habitat",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "animals.csv",
+    "ingest_job": "ingest-1776347277033",
+    "ingest_timestamp": "2026-04-16T13:47:57.033455363Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 3,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/5a35646f-b357-4636-b88f-203aba0d7435.json

@@ -0,0 +1,61 @@
+{
+  "id": "5a35646f-b357-4636-b88f-203aba0d7435",
+  "name": "people_test",
+  "schema_fingerprint": "bba35b6cd1e399dc0504c760c767ab391f7ed81a41573c15b3bd4eb0e3bcc382",
+  "objects": [
+    {
+      "bucket": "testing",
+      "key": "datasets/people_test.parquet",
+      "size_bytes": 1402,
+      "created_at": "2026-04-16T13:47:32.669517398Z"
+    }
+  ],
+  "created_at": "2026-04-16T13:47:32.669518210Z",
+  "updated_at": "2026-04-16T13:47:32.669787451Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "person_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "people.csv",
+    "ingest_job": "ingest-1776347252669",
+    "ingest_timestamp": "2026-04-16T13:47:32.669517398Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 3,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/5bab2fac-8bdd-41b7-a993-ed48fecc60b3.json

@@ -0,0 +1,117 @@
+{
+  "id": "5bab2fac-8bdd-41b7-a993-ed48fecc60b3",
+  "name": "onboard_demo2",
+  "schema_fingerprint": "274575423f7d2648cea49a628a2e827dbd8822cef1f4eebfcf8e0a5f8602fbf7",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/onboard_demo2.parquet",
+      "size_bytes": 7833,
+      "created_at": "2026-04-20T23:13:36.886222056Z"
+    }
+  ],
+  "created_at": "2026-04-20T23:13:36.886223472Z",
+  "updated_at": "2026-04-20T23:13:36.886426834Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "worker_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "email",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "skills",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "certifications",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "availability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reliability",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archetype",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "staffing_roster_sample.csv",
+    "ingest_job": "ingest-1776726816886",
+    "ingest_timestamp": "2026-04-20T23:13:36.886222056Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 25,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/909a1fd1-0924-4a5e-a527-d249a07540a1.json

@@ -0,0 +1,104 @@
+{
+  "id": "909a1fd1-0924-4a5e-a527-d249a07540a1",
+  "name": "kb_response_cache",
+  "schema_fingerprint": "c90d7be310b5025d2c4d398cf07692d8d9bb46ed591c0a87b339bafcac9ddeed",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_response_cache.parquet",
+      "size_bytes": 8464841,
+      "created_at": "2026-04-17T05:22:06.168071783Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.168072339Z",
+  "updated_at": "2026-04-17T06:06:21.356504054Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "cache_key",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "prompt",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "mode",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "models_used",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "best_run_id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "best_score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responses",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "hit_count",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "updated_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 197,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/94a8bd16-6756-43af-b951-09a9e6b8300f.json

@@ -1,23 +1,69 @@
 {
   "id": "94a8bd16-6756-43af-b951-09a9e6b8300f",
   "name": "clients",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "1ba20a49d967c85c41640a9b9a61287ae9fa483daf6c71a197591fe3228c76d2",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/clients.parquet",
       "size_bytes": 21971,
       "created_at": "2026-03-27T14:42:38.830329102Z"
     }
   ],
   "created_at": "2026-03-27T14:42:38.830331694Z",
-  "updated_at": "2026-03-27T14:42:38.830331694Z",
+  "updated_at": "2026-04-17T02:45:58.138164775Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "client_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "company_name",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "vertical",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "zip",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 2000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/9861b817-f571-496b-9d7b-c4d385e51ca5.json

@@ -0,0 +1,75 @@
+{
+  "id": "9861b817-f571-496b-9d7b-c4d385e51ca5",
+  "name": "sparse_workers",
+  "schema_fingerprint": "1055d5e17fa3285a1cac7d4c949ae03699cd046af671f22d0d1cf4d0024c66a4",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/sparse_workers.parquet",
+      "size_bytes": 7619,
+      "created_at": "2026-04-17T20:31:24.968688194Z"
+    }
+  ],
+  "created_at": "2026-04-17T20:31:24.968689336Z",
+  "updated_at": "2026-04-17T20:31:24.968969734Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": "pii",
+  "columns": [
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "phone",
+      "data_type": "Int64",
+      "sensitivity": "pii",
+      "description": "",
+      "is_pii": true
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "state",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "notes",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "sparse_workers.csv",
+    "ingest_job": "ingest-1776457884968",
+    "ingest_timestamp": "2026-04-17T20:31:24.968688194Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 200,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/9c4d9116-1d9d-4afd-a8d1-c514a678e5fa.json

@@ -1,23 +1,83 @@
 {
   "id": "9c4d9116-1d9d-4afd-a8d1-c514a678e5fa",
   "name": "call_log",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "d1f82ed776afc484747f8f58b30ccd00d10bca73703b1479a67647f98eeff7e2",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/call_log.parquet",
       "size_bytes": 35951077,
       "created_at": "2026-03-27T14:42:47.395548205Z"
     }
   ],
   "created_at": "2026-03-27T14:42:47.395555326Z",
-  "updated_at": "2026-03-27T14:42:47.395555326Z",
+  "updated_at": "2026-04-17T02:45:58.137293143Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "call_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "from_number",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "to_number",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "duration_seconds",
+      "data_type": "Int64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "timestamp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "recruiter",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "candidate_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "disposition",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 800000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/a2206c79-28d6-40f2-b97b-61a5df5bee06.json

@@ -0,0 +1,237 @@
+{
+  "id": "a2206c79-28d6-40f2-b97b-61a5df5bee06",
+  "name": "kb_threat_intel",
+  "schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_threat_intel.parquet",
+      "size_bytes": 247318,
+      "created_at": "2026-04-17T04:23:16.149697926Z"
+    }
+  ],
+  "created_at": "2026-04-17T04:23:16.149698245Z",
+  "updated_at": "2026-04-17T05:56:44.994303201Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "ip",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "threat_level",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "classification",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "confidence",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "summary",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "indicators",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "recommendation",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pattern",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "attack_type",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "likely_automated",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "country",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "country_code",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "city",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "isp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "org",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "asn",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "is_proxy",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "is_hosting",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "open_ports",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "blocklist_count",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "blocklist_total",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "blocklists_blocked",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "reverse_dns",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "traceroute",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "log_count",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "banned",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "enriched_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "updated_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "raw_data",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 54,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/a99da7e2-8f42-4ff3-8fa7-679d4e514581.json

@@ -0,0 +1,82 @@
+{
+  "id": "a99da7e2-8f42-4ff3-8fa7-679d4e514581",
+  "name": "kb_response_cache_history",
+  "schema_fingerprint": "9a05c209a51f9543bd7dc9387695b0c67a6abc135b53eaeec458140712bfcf50",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_response_cache_history.parquet",
+      "size_bytes": 8396568,
+      "created_at": "2026-04-17T04:14:34.260241351Z"
+    }
+  ],
+  "created_at": "2026-04-17T04:14:34.260242474Z",
+  "updated_at": "2026-04-17T06:06:21.334716946Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "cache_key",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "run_id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responses",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "models_used",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "recorded_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "postgresql",
+    "source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
+    "ingest_job": "pg-stream-1776403326260",
+    "ingest_timestamp": "2026-04-17T05:22:06.260249210Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 207,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/c2f0acdb-df65-4c21-9633-08af4248c85b.json

@@ -0,0 +1,97 @@
+{
+  "id": "c2f0acdb-df65-4c21-9633-08af4248c85b",
+  "name": "kb_lab_trials",
+  "schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_lab_trials.parquet",
+      "size_bytes": 68221,
+      "created_at": "2026-04-17T05:22:06.382106104Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.382107354Z",
+  "updated_at": "2026-04-17T06:06:21.358304760Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "experiment_id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "trial_num",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "config_diff",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "config_snapshot",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "scores",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "avg_score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "improved",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "duration_ms",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 41,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/d35c7941-37e2-4bde-8226-5cf69c74931a.json

@@ -1,23 +1,83 @@
 {
   "id": "d35c7941-37e2-4bde-8226-5cf69c74931a",
   "name": "placements",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "962535a1810c07183a882c15a76a11a02ad3afef844e71975b0eaa48943487f6",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/placements.parquet",
       "size_bytes": 1213820,
       "created_at": "2026-03-27T14:42:39.040983450Z"
     }
   ],
   "created_at": "2026-03-27T14:42:39.040989351Z",
-  "updated_at": "2026-03-27T14:42:39.040989351Z",
+  "updated_at": "2026-04-17T02:45:58.147407036Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "placement_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "candidate_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "job_order_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "client_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "bill_rate",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pay_rate",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "recruiter",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "status",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 50000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/d668a036-7b50-4845-8194-6a03ef370c8d.json

@@ -0,0 +1,97 @@
+{
+  "id": "d668a036-7b50-4845-8194-6a03ef370c8d",
+  "name": "kb_pipeline_runs",
+  "schema_fingerprint": "c019b81feb58ff2aefe4cbe700056a100e25716f2aa8e8415ba8f20656812f75",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_pipeline_runs.parquet",
+      "size_bytes": 1518415,
+      "created_at": "2026-04-17T05:22:06.289208160Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.289209657Z",
+  "updated_at": "2026-04-17T06:06:21.311384739Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "pipeline",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "topic",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "status",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "steps",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "result",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "models_used",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "duration_ms",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "completed_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 195,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/d92c5d57-14f9-4ce0-9c4f-19c6db94a170.json

@@ -0,0 +1,118 @@
+{
+  "id": "d92c5d57-14f9-4ce0-9c4f-19c6db94a170",
+  "name": "kb_team_runs",
+  "schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_team_runs.parquet",
+      "size_bytes": 19293718,
+      "created_at": "2026-04-17T05:22:06.071898629Z"
+    }
+  ],
+  "created_at": "2026-04-17T05:22:06.071899235Z",
+  "updated_at": "2026-04-17T05:56:44.992569105Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "mode",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "prompt",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "config",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "responses",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "models_used",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "archived",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "tags",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "notes",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "quality_score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "score_method",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "score_metadata",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 590,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/dab56959-d79b-4c68-a6bd-e5fc487621ff.json

@@ -0,0 +1,68 @@
+{
+  "id": "dab56959-d79b-4c68-a6bd-e5fc487621ff",
+  "name": "successful_playbooks",
+  "schema_fingerprint": "30b52caca7b6303852cd648882eae9ed87205b80e5e570f6fda08dc8735d9b06",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/successful_playbooks.parquet",
+      "size_bytes": 2540,
+      "created_at": "2026-04-17T22:25:34.467634304Z"
+    }
+  ],
+  "created_at": "2026-04-17T22:25:34.467635066Z",
+  "updated_at": "2026-04-19T08:44:08.533058251Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "timestamp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "operation",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "approach",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "result",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "context",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "csv",
+    "source_file": "playbook.csv",
+    "ingest_job": "ingest-1776588248532",
+    "ingest_timestamp": "2026-04-19T08:44:08.532686879Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 29,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/e2a8f88a-59f6-40c7-a45b-e23d8f3533b6.json

@@ -4,7 +4,7 @@
   "schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/lab_trials.parquet",
       "size_bytes": 64646,
       "created_at": "2026-03-28T01:14:03.026116573Z"

data/_catalog/manifests/e959ca11-9f6b-4843-864a-cc3f50a8aa60.json

@@ -1,23 +1,83 @@
 {
   "id": "e959ca11-9f6b-4843-864a-cc3f50a8aa60",
   "name": "email_log",
-  "schema_fingerprint": "auto",
+  "schema_fingerprint": "b28cbe711bba32611306f9a3186f67c1510c3e39b0c8812d765fe2b44bd65634",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/email_log.parquet",
       "size_bytes": 16768671,
       "created_at": "2026-03-27T14:42:49.271082991Z"
     }
   ],
   "created_at": "2026-03-27T14:42:49.271091077Z",
-  "updated_at": "2026-03-27T14:42:49.271091077Z",
+  "updated_at": "2026-04-17T02:45:57.881479855Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
-  "columns": [],
+  "columns": [
+    {
+      "name": "email_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "from_addr",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "to_addr",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "subject",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "timestamp",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "recruiter",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "candidate_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "opened",
+      "data_type": "Boolean",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
   "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": null
+  "row_count": 500000,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/eeb0d288-e227-478d-9407-78a43b623c5c.json

@@ -0,0 +1,118 @@
+{
+  "id": "eeb0d288-e227-478d-9407-78a43b623c5c",
+  "name": "kb_meta_pipelines",
+  "schema_fingerprint": "cabe1d7fc442e1dfcfaabf663509d590c6edc84b445a91acaf0ae68b94aff518",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/kb_meta_pipelines.parquet",
+      "size_bytes": 32085,
+      "created_at": "2026-04-17T04:14:34.315211960Z"
+    }
+  ],
+  "created_at": "2026-04-17T04:14:34.315212764Z",
+  "updated_at": "2026-04-17T06:06:21.357779083Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "id",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "name",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "data_source",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "stages",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "status",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "current_stage",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "total_stages",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "results",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "best_score",
+      "data_type": "Float64",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "iterations",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "config",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "created_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "updated_at",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": null,
+  "freshness": null,
+  "tags": [],
+  "row_count": 6,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}

data/_catalog/manifests/ef6c0acf-de7a-4be0-877e-e0c2889f390f.json

@@ -1,17 +1,17 @@
 {
-  "id": "e7304f05-5278-4e17-961a-51f2588fd2aa",
+  "id": "ef6c0acf-de7a-4be0-877e-e0c2889f390f",
   "name": "threat_intel",
   "schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/threat_intel.parquet",
-      "size_bytes": 111130,
-      "created_at": "2026-03-28T01:14:03.054140697Z"
+      "size_bytes": 247112,
+      "created_at": "2026-04-19T11:19:40.517865325Z"
     }
   ],
-  "created_at": "2026-03-28T01:14:03.054141294Z",
-  "updated_at": "2026-03-28T01:14:03.054427047Z",
+  "created_at": "2026-04-16T07:59:13.882669807Z",
+  "updated_at": "2026-04-19T11:19:40.517873046Z",
   "description": "",
   "owner": "",
   "sensitivity": null,
@@ -227,14 +227,11 @@
       "is_pii": false
     }
   ],
-  "lineage": {
-    "source_system": "postgresql",
-    "source_file": "127.0.0.1:5432/knowledge_base.threat_intel",
-    "ingest_job": "pg-import-1774660443054",
-    "ingest_timestamp": "2026-03-28T01:14:03.054140697Z",
-    "parent_datasets": []
-  },
+  "lineage": null,
   "freshness": null,
   "tags": [],
-  "row_count": 20
+  "row_count": 20,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
 }

data/_catalog/manifests/f0b6f408-71a5-4365-bd1d-98c1e176096a.json

@@ -4,7 +4,7 @@
   "schema_fingerprint": "0fcee1ac176b9bc45ab392bbd4401042803eb80d0de48b1fa6a5e20ffb27fa8d",
   "objects": [
     {
-      "bucket": "data",
+      "bucket": "primary",
       "key": "datasets/new_candidates.parquet",
       "size_bytes": 2731,
       "created_at": "2026-03-28T01:04:22.787759218Z"

data/_catalog/manifests/f2fda821-2f86-4c81-af58-488d184d8243.json

@@ -0,0 +1,47 @@
+{
+  "id": "f2fda821-2f86-4c81-af58-488d184d8243",
+  "name": "resumes",
+  "schema_fingerprint": "a2b2c191412faf317800637da5887d30d749d5ba0222d0a8edd12cde08daeb8a",
+  "objects": [
+    {
+      "bucket": "primary",
+      "key": "datasets/resumes.parquet",
+      "size_bytes": 3159,
+      "created_at": "2026-04-19T11:26:29.607950059Z"
+    }
+  ],
+  "created_at": "2026-04-19T11:26:29.607951447Z",
+  "updated_at": "2026-04-19T11:26:29.608054435Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "doc_id",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "resume_text",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "ndjson",
+    "source_file": "resumes.ndjson",
+    "ingest_job": "ingest-1776597989607",
+    "ingest_timestamp": "2026-04-19T11:26:29.607950059Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 10,
+  "last_embedded_at": null,
+  "embedding_stale_since": null,
+  "embedding_refresh_policy": null
+}