Observer KB UI: now shows LLM Team extracted facts + entities

- Knowledge tab: stats (categories, verified facts, entities, decisions, insights, questions),
  auto-generated categories, key decisions, key insights
- Facts & Entities tab (NEW): 279 extracted facts from LLM Team extract mode,
  264 entities as clickable chips (click → search), verified entries from secondary model
- Search tab: semantic HNSW search across all knowledge
- Ask AI tab: RAG with cited sources
- Ingest tab: upload with progress
- Entity chips link to semantic search
- Category drill-down shows related content via HNSW

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.mcp.json

@@ -1,12 +0,0 @@
-{
-  "mcpServers": {
-    "lakehouse": {
-      "command": "bun",
-      "args": ["run", "/home/profit/lakehouse/mcp-server/index.ts"],
-      "env": {
-        "LAKEHOUSE_URL": "http://localhost:3100",
-        "MCP_TRANSPORT": "stdio"
-      }
-    }
-  }
-}

Cargo.toml

@@ -12,8 +12,6 @@ members = [
     "crates/journald",
     "crates/gateway",
     "crates/ui",
-    "crates/lance-bench",
-    "crates/vectord-lance",
 ]
 
 [workspace.dependencies]
@@ -47,5 +45,4 @@ csv = "1"
 lopdf = "0.35"
 encoding_rs = "0.8"
 instant-distance = "0.6"
-tokio-postgres = { version = "0.7", features = ["with-serde_json-1", "with-chrono-0_4", "with-uuid-1"] }
-mysql_async = { version = "0.34", default-features = false, features = ["minimal"] }
+tokio-postgres = { version = "0.7", features = ["with-serde_json-1", "with-chrono-0_4"] }

apps/observer-kb/docs/PRD.md

@@ -0,0 +1,183 @@
+# PRD: Observer Knowledge Base
+
+**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
+**Created:** 2026-03-28
+**Owner:** J
+**Branch:** `observer-kb`
+**Parent:** Lakehouse core (crates/)
+
+---
+
+## Problem
+
+AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.
+
+## Vision
+
+A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.
+
+---
+
+## Architecture
+
+### Multi-Pass Processing via LLM Team Modes
+
+The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.
+
+```
+Raw PDF Upload
+    │
+    ▼ Mode: EXTRACT
+    Structured facts, entities, relationships
+    │
+    ▼ Mode: VALIDATOR
+    Fact-check claims, score 1-10 confidence
+    │
+    ▼ Mode: HALLUCINATION
+    Verify each claim, flag likely hallucinations
+    │
+    ▼ Mode: CONSENSUS
+    Multiple passes iterate until extraction converges
+    │
+    ▼ Mode: REDTEAM
+    Find flaws in knowledge, patch them
+    │
+    ▼ Mode: PIPELINE (custom)
+    Clean → Q&A structure → topic grouping → value ranking
+    │
+    ▼ RENDER
+    Well-curated document: questions as anchors, branching connections,
+    what was asked → answered → directed
+```
+
+### Integration with devop.live (LLM Team)
+
+The Observer KB calls the LLM Team API to run modes against extracted content:
+
+| Mode | Purpose in KB |
+|---|---|
+| **extract** | Structured facts, entities, relationships from raw text |
+| **validator** | Fact-check extracted claims, confidence scores |
+| **hallucination** | Verify each factual claim independently |
+| **drift** | Process N times, find consistent vs uncertain content |
+| **redteam** | Find flaws in extracted knowledge, improve |
+| **consensus** | Iterate until multiple passes agree on correct extraction |
+| **pipeline** | Chain: clean → extract → validate → organize → render |
+| **research** | Generate follow-up questions, research in parallel |
+| **blindassembly** | Split content into parts, process independently, reassemble |
+
+### Data Flow
+
+```
+Upload Portal (Observer KB :3400)
+    │
+    ├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
+    │
+    ▼
+Processing Pipeline
+    │
+    ├── Call devop.live API (:5000) with each mode
+    ├── Each mode processes content through local Ollama models
+    ├── Results stored back in Lakehouse (Parquet + catalog)
+    │
+    ▼
+Knowledge Store
+    │
+    ├── knowledge_facts: verified claims with confidence scores
+    ├── knowledge_qa: question → answer pairs with context
+    ├── knowledge_topics: auto-generated categories
+    ├── knowledge_connections: how topics relate across conversations
+    ├── knowledge_contradictions: flagged conflicts between conversations
+    ├── knowledge_validated: claims that passed hallucination check
+    │
+    ▼
+Rendering
+    │
+    ├── Structured knowledge view (web UI)
+    ├── Exportable curated document (PDF/markdown)
+    └── Question-anchored navigation (Q → A → follow-up → related)
+```
+
+---
+
+## Processing Pipeline Detail
+
+### Pass 1: Clean + Structure
+- Strip PDF artifacts (headers, footers, page numbers)
+- Identify conversation turns (user/AI)
+- Fix encoding issues (ligatures, special chars)
+- Normalize formatting
+
+### Pass 2: Q&A Extraction
+- Identify questions asked by user
+- Map each question to its answer
+- Track follow-up questions and how they branch
+- Build question→answer→follow-up chains
+
+### Pass 3: Fact Extraction + Validation
+- Run EXTRACT mode: pull structured facts, entities, relationships
+- Run VALIDATOR mode: score each fact 1-10 confidence
+- Run HALLUCINATION mode: verify claims independently
+- Flag contradictions between conversations
+
+### Pass 4: Organize + Curate
+- Run CONSENSUS mode: multiple passes converge on correct categorization
+- Auto-generate topic categories from content (not predefined)
+- Rank content by value (decisions > insights > discussion > noise)
+- Connect related content across conversations
+
+### Pass 5: Render
+- Question-anchored view: each question is a node, answers branch from it
+- Topic-grouped view: knowledge organized by auto-generated categories
+- Timeline view: how thinking evolved across conversations
+- Export as clean markdown/PDF
+
+---
+
+## Current State (as of 2026-03-28)
+
+### What's Built
+- PDF ingestion (8 PDFs, 477 pages extracted)
+- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
+- Semantic search via HNSW (27ms over 477 chunks)
+- RAG (question → search → LLM answer with sources)
+- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
+- Observer KB web UI (knowledge-first, not page-first)
+- Upload portal with drag & drop
+
+### What's Not Built Yet
+- Integration with devop.live LLM Team modes
+- Multi-pass processing pipeline (extract → validate → hallucination check)
+- Q&A chain extraction (question → answer → follow-up mapping)
+- Contradiction detection between conversations
+- Curated document export (PDF/markdown)
+- Question-anchored navigation
+- Auto-processing on upload (currently manual script)
+
+### What Needs to Happen Next Session
+1. **Integrate with devop.live API** — call modes programmatically
+2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
+3. **Q&A chain extraction** — questions as anchors, answers branching
+4. **Auto-process on upload** — no manual script, pipeline runs automatically
+5. **Render curated output** — readable, hand-it-to-someone quality
+
+---
+
+## Tech Stack
+
+- **Storage:** Lakehouse storaged (Parquet on object storage)
+- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
+- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
+- **Vectors:** Lakehouse vectord (HNSW search, RAG)
+- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
+- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
+- **UI:** Pure HTML/JS on :3400
+
+---
+
+## Non-Goals (for now)
+
+- Real-time conversation ingestion (PDF exports only)
+- Multi-user collaboration
+- Cloud deployment
+- Editing extracted knowledge from UI (read-only for now)

apps/observer-kb/index.html

@@ -0,0 +1,286 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Observer KB</title>
+<style>
+:root{--bg:#0a0a0f;--s1:#12121a;--s2:#1a1a28;--b:#2a2a3a;--t:#e0e0e8;--d:#888898;--a:#6c5ce7;--ok:#00d2a0;--err:#ff6b6b;--warn:#ffa94d;--f:'JetBrains Mono','Fira Code',monospace}
+*{margin:0;padding:0;box-sizing:border-box}body{font-family:var(--f);background:var(--bg);color:var(--t);font-size:13px;min-height:100vh}
+.app{max-width:1200px;margin:0 auto;padding:20px}
+header{display:flex;align-items:center;gap:20px;padding:16px 0;border-bottom:1px solid var(--b);margin-bottom:20px}
+header h1{font-size:20px;color:var(--a)}
+.tabs{display:flex;gap:2px}.tab{background:none;border:none;color:var(--d);font-family:var(--f);font-size:12px;padding:10px 16px;cursor:pointer;border-bottom:2px solid transparent}
+.tab:hover{color:var(--t)}.tab.active{color:var(--a);border-bottom-color:var(--a)}
+.hdr-stats{margin-left:auto;font-size:11px;color:var(--d)}
+.btn{background:var(--a);color:white;border:none;padding:8px 16px;border-radius:6px;font-family:var(--f);font-size:12px;cursor:pointer;font-weight:600}
+.btn:hover{opacity:.9}.btn:disabled{opacity:.5}.btn-sm{padding:5px 10px;font-size:11px}
+.hidden{display:none}
+.stats-bar{display:grid;grid-template-columns:repeat(auto-fill,minmax(130px,1fr));gap:8px;margin-bottom:20px}
+.stat{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:12px;text-align:center}
+.stat-val{font-size:22px;font-weight:700;color:var(--a)}.stat-label{font-size:10px;color:var(--d);text-transform:uppercase;margin-top:2px}
+.section{margin-bottom:24px}.section h2{font-size:13px;color:var(--d);text-transform:uppercase;letter-spacing:.08em;margin-bottom:10px;border-bottom:1px solid var(--b);padding-bottom:6px}
+.entry{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:14px;margin-bottom:8px}
+.badge{font-size:9px;text-transform:uppercase;letter-spacing:.08em;padding:2px 8px;border-radius:3px;display:inline-block;margin-bottom:6px;font-weight:600}
+.badge-fact{background:var(--a);color:white}.badge-entity{background:var(--ok);color:white}
+.badge-decision{background:#e67e22;color:white}.badge-insight{background:#3498db;color:white}
+.badge-question{background:var(--warn);color:#000}.badge-verified{background:var(--ok);color:white}
+.entry-text{font-size:12px;line-height:1.6}.entry-src{font-size:10px;color:var(--d);margin-top:4px}
+.cat-grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(250px,1fr));gap:8px;margin-bottom:16px}
+.cat-card{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:14px;cursor:pointer;border-left:3px solid var(--a);transition:all .15s}
+.cat-card:hover{border-color:var(--a)}.cat-title{font-weight:600;font-size:13px}.cat-count{font-size:10px;color:var(--d);margin-top:2px}
+.search-row{display:flex;gap:8px;margin-bottom:16px}
+.search-row input{flex:1;background:var(--s1);border:2px solid var(--b);color:var(--t);font-family:var(--f);font-size:14px;padding:12px;border-radius:8px;outline:none}
+.search-row input:focus{border-color:var(--a)}
+.ask-input{width:100%;background:var(--s1);border:2px solid var(--b);color:var(--t);font-family:var(--f);font-size:13px;padding:12px;border-radius:8px;resize:vertical;min-height:80px;outline:none;margin-bottom:8px}
+.ask-input:focus{border-color:var(--a)}
+.ai-answer{background:var(--s2);border:1px solid var(--b);border-radius:8px;padding:16px;margin-top:12px;line-height:1.7;white-space:pre-wrap}
+.ai-meta{font-size:10px;color:var(--d);margin-top:6px}
+.upload-zone{border:2px dashed var(--b);border-radius:12px;padding:36px;text-align:center;cursor:pointer;margin-bottom:16px}
+.upload-zone:hover{border-color:var(--a);background:rgba(108,92,231,.03)}
+.upload-zone h3{margin-bottom:4px}.upload-zone p{color:var(--d);font-size:11px}
+.file-item{display:flex;align-items:center;justify-content:space-between;padding:8px 12px;border-bottom:1px solid var(--b);font-size:12px}
+.done{color:var(--ok)}.failed{color:var(--err)}.uploading{color:var(--a)}
+.result{background:var(--s2);border:1px solid var(--b);border-radius:8px;padding:12px;margin-bottom:6px;border-left:3px solid var(--a)}
+.result-score{font-size:10px;color:var(--a);margin-bottom:2px}.result-text{font-size:12px;line-height:1.5}
+.empty{color:var(--d);text-align:center;padding:32px}
+</style>
+</head>
+<body>
+<div class="app">
+<header>
+<h1>OBSERVER KB</h1>
+<div class="tabs">
+<button class="tab active" onclick="showTab('knowledge')">Knowledge</button>
+<button class="tab" onclick="showTab('facts')">Facts &amp; Entities</button>
+<button class="tab" onclick="showTab('explore')">Search</button>
+<button class="tab" onclick="showTab('ask')">Ask AI</button>
+<button class="tab" onclick="showTab('upload')">Ingest</button>
+</div>
+<div class="hdr-stats" id="hdr-stats"></div>
+</header>
+
+<div id="tab-knowledge"></div>
+<div id="tab-facts" class="hidden"></div>
+<div id="tab-explore" class="hidden"></div>
+<div id="tab-ask" class="hidden"></div>
+<div id="tab-upload" class="hidden"></div>
+</div>
+
+<script>
+const API=`http://${location.hostname}:3100`;
+const $=id=>document.getElementById(id);
+async function q(sql){return(await(await fetch(API+'/query/sql',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({sql})})).json())}
+async function searchHNSW(query,k=10){try{return(await(await fetch(API+'/vectors/hnsw/search',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',query,top_k:k})})).json()).results||[]}catch{return[]}}
+
+function showTab(name){
+['knowledge','facts','explore','ask','upload'].forEach((t,i)=>{
+$('tab-'+t).classList.toggle('hidden',t!==name);
+document.querySelectorAll('.tab')[i].classList.toggle('active',t===name);
+});
+({knowledge:loadKnowledge,facts:loadFacts,explore:loadExplore,ask:loadAsk,upload:loadUpload})[name]();
+}
+
+function makeEntry(type,text,source){
+const e=document.createElement('div');e.className='entry';
+const b=document.createElement('span');b.className='badge badge-'+type;b.textContent=type;
+const t=document.createElement('div');t.className='entry-text';t.textContent=text;
+const s=document.createElement('div');s.className='entry-src';s.textContent=source;
+e.append(b,t,s);return e;
+}
+
+// === KNOWLEDGE (categories + decisions + insights) ===
+async function loadKnowledge(){
+const el=$('tab-knowledge');el.textContent='';
+try{
+const[topics,decisions,insights,questions,extractions]=await Promise.all([
+q("SELECT topic,COUNT(*) as n,COUNT(DISTINCT conversation) as c FROM knowledge_topics GROUP BY topic ORDER BY n DESC LIMIT 15"),
+q("SELECT conversation,page,decision FROM knowledge_decisions LIMIT 12"),
+q("SELECT conversation,page,insight FROM knowledge_insights LIMIT 8"),
+q("SELECT COUNT(*) as n FROM knowledge_questions"),
+q("SELECT type,COUNT(*) as n FROM processed_extractions GROUP BY type"),
+]);
+
+const factCount=extractions.rows.find(r=>r.type==='fact')?.n||0;
+const entityCount=extractions.rows.find(r=>r.type==='entity')?.n||0;
+
+// Stats
+const sb=document.createElement('div');sb.className='stats-bar';
+[{v:topics.row_count,l:'Categories'},{v:factCount,l:'Verified Facts'},{v:entityCount,l:'Entities'},{v:decisions.row_count,l:'Decisions'},{v:insights.row_count,l:'Insights'},{v:questions.rows[0]?.n||0,l:'Open Questions'}].forEach(s=>{
+const d=document.createElement('div');d.className='stat';
+const v=document.createElement('div');v.className='stat-val';v.textContent=s.v;
+const l=document.createElement('div');l.className='stat-label';l.textContent=s.l;
+d.append(v,l);sb.appendChild(d);
+});
+el.appendChild(sb);
+$('hdr-stats').textContent=factCount+' facts · '+entityCount+' entities · '+decisions.row_count+' decisions';
+
+// Categories
+const cs=document.createElement('div');cs.className='section';
+cs.insertAdjacentHTML('beforeend','<h2>Auto-Generated Categories</h2>');
+const cg=document.createElement('div');cg.className='cat-grid';
+topics.rows.forEach(t=>{
+const c=document.createElement('div');c.className='cat-card';
+const n=document.createElement('div');n.className='cat-title';n.textContent=t.topic;
+const m=document.createElement('div');m.className='cat-count';m.textContent=t.n+' mentions · '+t.c+' conversations';
+c.append(n,m);c.onclick=()=>drillTopic(t.topic);cg.appendChild(c);
+});
+cs.appendChild(cg);el.appendChild(cs);
+
+// Decisions
+const ds=document.createElement('div');ds.className='section';
+ds.insertAdjacentHTML('beforeend','<h2>Key Decisions</h2>');
+decisions.rows.forEach(d=>ds.appendChild(makeEntry('decision',d.decision,d.conversation+' p'+d.page)));
+el.appendChild(ds);
+
+// Insights
+const is=document.createElement('div');is.className='section';
+is.insertAdjacentHTML('beforeend','<h2>Key Insights</h2>');
+insights.rows.forEach(d=>is.appendChild(makeEntry('insight',d.insight,d.conversation+' p'+d.page)));
+el.appendChild(is);
+}catch(e){el.textContent='Error: '+e}
+}
+
+async function drillTopic(topic){
+const el=$('tab-knowledge');el.textContent='';
+const back=document.createElement('button');back.className='btn btn-sm';back.textContent='← Back';back.onclick=loadKnowledge;
+const h=document.createElement('h2');h.style.cssText='font-size:18px;margin:12px 0';h.textContent=topic;
+el.append(back,h);
+const results=await searchHNSW(topic,10);
+const s=document.createElement('div');s.className='section';
+s.insertAdjacentHTML('beforeend','<h2>Related Content</h2>');
+results.forEach(r=>{
+const e=document.createElement('div');e.className='result';
+const sc=document.createElement('div');sc.className='result-score';sc.textContent='['+r.score.toFixed(3)+'] '+r.doc_id;
+const t=document.createElement('div');t.className='result-text';t.textContent=r.chunk_text.substring(0,300);
+e.append(sc,t);s.appendChild(e);
+});
+el.appendChild(s);
+}
+
+// === FACTS & ENTITIES (from LLM Team extract mode) ===
+async function loadFacts(){
+const el=$('tab-facts');el.textContent='';
+try{
+const[facts,entities,verified]=await Promise.all([
+q("SELECT conversation,page,content FROM processed_extractions WHERE type='fact' ORDER BY conversation,page LIMIT 30"),
+q("SELECT conversation,content,COUNT(*) as n FROM processed_extractions WHERE type='entity' GROUP BY conversation,content ORDER BY n DESC LIMIT 30"),
+q("SELECT conversation,page,content FROM processed_validated LIMIT 10"),
+]);
+
+const fs=document.createElement('div');fs.className='section';
+fs.insertAdjacentHTML('beforeend','<h2>Extracted Facts ('+facts.row_count+' total — via LLM Team Extract Mode)</h2>');
+facts.rows.forEach(r=>fs.appendChild(makeEntry('fact',r.content,r.conversation+' p'+r.page)));
+el.appendChild(fs);
+
+const es=document.createElement('div');es.className='section';
+es.insertAdjacentHTML('beforeend','<h2>Entities ('+entities.row_count+' unique)</h2>');
+const eg=document.createElement('div');eg.style.cssText='display:flex;flex-wrap:wrap;gap:6px';
+entities.rows.forEach(r=>{
+const chip=document.createElement('span');
+chip.style.cssText='background:var(--s2);border:1px solid var(--b);padding:4px 10px;border-radius:4px;font-size:11px;cursor:pointer';
+chip.textContent=r.content+(r.n>1?' ('+r.n+')':'');
+chip.onclick=()=>{showTab('explore');setTimeout(()=>{document.getElementById('searchInput').value=r.content;doSearch()},100)};
+eg.appendChild(chip);
+});
+es.appendChild(eg);el.appendChild(es);
+
+if(verified.rows.length){
+const vs=document.createElement('div');vs.className='section';
+vs.insertAdjacentHTML('beforeend','<h2>Verified by Secondary Model</h2>');
+verified.rows.forEach(r=>vs.appendChild(makeEntry('verified',r.content,r.conversation+' p'+r.page)));
+el.appendChild(vs);
+}
+}catch(e){el.textContent='Error: '+e}
+}
+
+// === SEARCH ===
+function loadExplore(){
+const el=$('tab-explore');el.textContent='';
+const row=document.createElement('div');row.className='search-row';
+const input=document.createElement('input');input.id='searchInput';input.placeholder='Search by meaning across all knowledge...';
+const btn=document.createElement('button');btn.className='btn';btn.textContent='Search';
+btn.onclick=doSearch;input.onkeydown=e=>{if(e.key==='Enter')doSearch()};
+row.append(input,btn);
+const res=document.createElement('div');res.id='searchResults';
+const hint=document.createElement('div');hint.className='empty';hint.textContent='Try: "object storage architecture", "data ingestion trade-offs", "CRM options"';
+el.append(row,res,hint);
+}
+async function doSearch(){
+const query=document.getElementById('searchInput').value;if(!query)return;
+const el=$('searchResults');el.textContent='searching...';
+const results=await searchHNSW(query,15);
+el.textContent='';
+if(!results.length){el.textContent='No results';return}
+results.forEach(r=>{
+const e=document.createElement('div');e.className='result';
+const s=document.createElement('div');s.className='result-score';s.textContent='['+r.score.toFixed(3)+'] '+r.doc_id.replace(/_/g,' ');
+const t=document.createElement('div');t.className='result-text';t.textContent=r.chunk_text.substring(0,400);
+e.append(s,t);el.appendChild(e);
+});
+}
+
+// === ASK AI ===
+function loadAsk(){
+const el=$('tab-ask');el.textContent='';
+const ta=document.createElement('textarea');ta.className='ask-input';ta.id='askInput';
+ta.placeholder='Ask a question about your knowledge base...\n\nExamples:\n- What architecture decisions were made about Iceberg?\n- Compare Spark vs Flink approaches\n- What approaches were abandoned and why?';
+const btn=document.createElement('button');btn.className='btn';btn.id='askBtn';btn.textContent='Ask AI';
+btn.onclick=doAsk;
+const resp=document.createElement('div');resp.id='askResp';
+el.append(ta,btn,resp);
+}
+async function doAsk(){
+const question=$('askInput').value;if(!question)return;
+$('askBtn').disabled=true;$('askBtn').textContent='thinking...';
+const el=$('askResp');el.textContent='searching knowledge...';
+try{
+const r=await(await fetch(API+'/vectors/rag',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',question,top_k:5})})).json();
+el.textContent='';
+const ans=document.createElement('div');ans.className='ai-answer';ans.textContent=r.answer;
+const meta=document.createElement('div');meta.className='ai-meta';meta.textContent='Model: '+r.model+(r.tokens_generated?' · '+r.tokens_generated+' tokens':'');
+el.append(ans,meta);
+if(r.sources?.length){
+const sh=document.createElement('h2');sh.style.cssText='font-size:11px;color:var(--d);margin-top:14px';sh.textContent='SOURCES';el.appendChild(sh);
+r.sources.forEach(s=>{
+const e=document.createElement('div');e.className='result';e.style.marginTop='4px';
+const sc=document.createElement('div');sc.className='result-score';sc.textContent='['+s.score.toFixed(3)+'] '+s.doc_id;
+const t=document.createElement('div');t.className='result-text';t.style.fontSize='11px';t.textContent=s.chunk_text.substring(0,200);
+e.append(sc,t);el.appendChild(e);
+});
+}
+}catch(e){el.textContent='Error: '+e}
+$('askBtn').disabled=false;$('askBtn').textContent='Ask AI';
+}
+
+// === UPLOAD ===
+function loadUpload(){
+const el=$('tab-upload');el.textContent='';
+const zone=document.createElement('div');zone.className='upload-zone';
+zone.onclick=()=>{const i=document.createElement('input');i.type='file';i.multiple=true;i.accept='.pdf,.txt,.csv,.json';i.onchange=e=>[...e.target.files].forEach(uploadFile);i.click()};
+zone.ondragover=e=>{e.preventDefault();zone.style.borderColor='var(--a)'};
+zone.ondragleave=()=>zone.style.borderColor='';
+zone.ondrop=e=>{e.preventDefault();zone.style.borderColor='';[...e.dataTransfer.files].forEach(uploadFile)};
+const h=document.createElement('h3');h.textContent='Drop files to ingest';
+const p=document.createElement('p');p.textContent='PDF, CSV, JSON, Text — auto-extracted and indexed';
+zone.append(h,p);
+const list=document.createElement('div');list.id='upList';
+el.append(zone,list);
+}
+function uploadFile(file){
+const item=document.createElement('div');item.className='file-item';
+const name=document.createElement('span');name.textContent=file.name;
+const status=document.createElement('span');status.className='uploading';status.textContent='uploading...';
+item.append(name,status);$('upList').prepend(item);
+const fd=new FormData();fd.append('file',file);
+const xhr=new XMLHttpRequest();xhr.open('POST',API+'/ingest/file');
+xhr.upload.onprogress=e=>{if(e.lengthComputable)status.textContent=Math.round(e.loaded/e.total*100)+'%'};
+xhr.onload=()=>{status.className=xhr.status<300?'done':'failed';status.textContent=xhr.status<300?'ingested':'failed'};
+xhr.onerror=()=>{status.className='failed';status.textContent='error'};
+xhr.send(fd);
+}
+
+loadKnowledge();
+</script>
+</body>
+</html>

auditor/README.md

@@ -1,53 +0,0 @@
-# Lakehouse Claim Auditor
-
-A Bun sub-agent that watches open PRs on Gitea, reads the ship-claims
-in commit messages and PR bodies, and **hard-blocks** merges when the
-code doesn't back the claim.
-
-Rationale: when "compiles + one curl works" gets called "phase shipped,"
-placeholder code accumulates. This auditor runs every 90s, fetches
-each open PR, and subjects it to four checks:
-
-1. **Static diff** — grep/parse looking for placeholder patterns
-2. **Dynamic** — runs the never-before-executed hybrid test fixture
-3. **Cloud inference** — asks `gpt-oss:120b` via `/v1/chat` to
-   identify gaps in the diff
-4. **KB query** — looks up `data/_kb/` + observer for prior failure
-   patterns on similar claims
-
-Verdict is assembled, posted to Gitea as:
-- A **failing commit status** (hard block — branch protection
-  prevents merge)
-- A **review comment** explaining every finding
-
-## Run manually
-
-```bash
-cd /home/profit/lakehouse
-bun run auditor/index.ts
-```
-
-Defaults: polls every 90s, stops on `auditor.paused` file present.
-
-## State
-
-- `data/_auditor/state.json` — last-audited head SHA per PR
-- `data/_auditor/verdicts/{pr}-{sha}.json` — per-run verdict record
-
-## Where YOU edit
-
-`auditor/policy.ts` — the verdict assembler. Controls which findings
-block vs warn vs inform. All other code is mechanical: fetching,
-running checks, posting to Gitea.
-
-## Hard-block mechanism
-
-1. Commit status is posted as `failure` with context `lakehouse/auditor`
-2. If `main` branch protection requires `lakehouse/auditor` status
-   to pass, Gitea prevents merge
-3. When code is fixed and re-audit passes, status flips to `success`,
-   merge unblocks
-
-Enable branch protection (one-time, via Gitea UI or API):
-- `POST /repos/profit/lakehouse/branch_protections`
-- `{"branch_name": "main", "required_status_checks": {"contexts": ["lakehouse/auditor"]}}`

auditor/audit.ts

@@ -1,171 +0,0 @@
-// Orchestrator — runs all four checks on a PR, assembles a verdict,
-// posts to Gitea. This is task #8's integration layer; the poller
-// (task #9) calls this once per PR on every fresh head SHA.
-//
-// Hard-block mechanism: commit status posted with state="failure"
-// and context="lakehouse/auditor". If `main` branch protection
-// requires that context to pass, merge is physically impossible
-// until the auditor re-audits a fixed commit and flips the status
-// to "success".
-//
-// Human-readable reasoning: posted as a PR issue comment (not a
-// review — reviews have self-review restrictions on Gitea and the
-// auditor currently uses the same PAT as the PR author).
-
-import { readFile, writeFile, mkdir } from "node:fs/promises";
-import { join } from "node:path";
-import type { PrSnapshot, Verdict, Finding } from "./types.ts";
-import { getPrDiff, postCommitStatus, postIssueComment } from "./gitea.ts";
-import { parseClaims } from "./claim_parser.ts";
-import { assembleVerdict } from "./policy.ts";
-import { runStaticCheck } from "./checks/static.ts";
-import { runDynamicCheck } from "./checks/dynamic.ts";
-import { runInferenceCheck } from "./checks/inference.ts";
-import { runKbCheck } from "./checks/kb_query.ts";
-
-const VERDICTS_DIR = "/home/profit/lakehouse/data/_auditor/verdicts";
-
-export interface AuditOptions {
-  // Skip the cloud inference call (fast path for iteration). Default false.
-  skip_inference?: boolean;
-  // Skip the dynamic check (avoid running the hybrid fixture every PR,
-  // since it hits live services and mutates playbook state). Default false
-  // on `main`-branch-target PRs, true when auditing feature branches
-  // where the fixture would pollute state. Caller decides.
-  skip_dynamic?: boolean;
-  // Skip Gitea posting — useful for dry-runs / local testing.
-  // Default false.
-  dry_run?: boolean;
-}
-
-export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<Verdict> {
-  const t0 = Date.now();
-  const diff = await getPrDiff(pr.number);
-  const { claims } = parseClaims(pr);
-
-  // Run checks in parallel where they don't share mutable state.
-  // Static + kb_query + inference are all read-only. Dynamic mutates
-  // playbook state (nonce-scoped per run, but still live) so if
-  // skip_dynamic is false we still run it in parallel — the mutation
-  // is namespaced.
-  const [staticFindings, dynamicFindings, inferenceFindings, kbFindings] = await Promise.all([
-    runStaticCheck(diff),
-    opts.skip_dynamic ? Promise.resolve(stubFinding("dynamic", "skipped by options")) : runDynamicCheck(),
-    opts.skip_inference ? Promise.resolve(stubFinding("inference", "skipped by options")) : runInferenceCheck(claims, diff),
-    runKbCheck(claims),
-  ]);
-
-  const allFindings: Finding[] = [
-    ...staticFindings,
-    ...dynamicFindings,
-    ...inferenceFindings,
-    ...kbFindings,
-  ];
-
-  const duration_ms = Date.now() - t0;
-  const metrics = {
-    audit_duration_ms: duration_ms,
-    findings_total: allFindings.length,
-    findings_block: allFindings.filter(f => f.severity === "block").length,
-    findings_warn: allFindings.filter(f => f.severity === "warn").length,
-    findings_info: allFindings.filter(f => f.severity === "info").length,
-    claims_strong: claims.filter(c => c.strength === "strong").length,
-    claims_moderate: claims.filter(c => c.strength === "moderate").length,
-    claims_weak: claims.filter(c => c.strength === "weak").length,
-    claims_total: claims.length,
-    diff_bytes: diff.length,
-  };
-
-  const verdict = assembleVerdict(allFindings, metrics, pr.number, pr.head_sha);
-
-  await persistVerdict(verdict);
-
-  if (!opts.dry_run) {
-    await postToGitea(verdict);
-  }
-
-  return verdict;
-}
-
-async function persistVerdict(v: Verdict): Promise<void> {
-  await mkdir(VERDICTS_DIR, { recursive: true });
-  const filename = `${v.pr_number}-${v.head_sha.slice(0, 12)}.json`;
-  await writeFile(join(VERDICTS_DIR, filename), JSON.stringify(v, null, 2));
-}
-
-export async function postToGitea(v: Verdict): Promise<void> {
-  // 1. Commit status — the hard block signal (if branch protection
-  //    is configured to require lakehouse/auditor on main).
-  const state = v.overall === "approve" ? "success" : "failure";
-  await postCommitStatus({
-    sha: v.head_sha,
-    state,
-    context: "lakehouse/auditor",
-    description: v.one_liner,
-    target_url: "", // no URL yet; could point to a verdicts dashboard
-  });
-
-  // 2. Issue comment — the reasoning. Gated so we don't spam the PR
-  //    with identical comments on re-audits of the same SHA. Caller
-  //    (poller) ensures we only re-audit fresh SHAs, but a dedup
-  //    marker inside the body keeps it idempotent if re-run.
-  const body = formatReviewBody(v);
-  await postIssueComment({ pr_number: v.pr_number, body });
-}
-
-function formatReviewBody(v: Verdict): string {
-  const byCheck: Record<string, Finding[]> = {};
-  for (const f of v.findings) {
-    (byCheck[f.check] ||= []).push(f);
-  }
-
-  const verdictEmoji =
-    v.overall === "approve" ? "✅" :
-    v.overall === "request_changes" ? "⚠️" :
-    "🛑";
-
-  const lines: string[] = [];
-  lines.push(`## Auditor verdict: ${verdictEmoji} \`${v.overall}\``);
-  lines.push("");
-  lines.push(`**One-liner:** ${v.one_liner}`);
-  lines.push(`**Head SHA:** \`${v.head_sha.slice(0, 12)}\``);
-  lines.push(`**Audited at:** ${v.audited_at}`);
-  lines.push("");
-
-  // Per-check sections, only if the check produced findings.
-  const checkOrder = ["static", "dynamic", "inference", "kb_query"] as const;
-  for (const check of checkOrder) {
-    const fs = byCheck[check] ?? [];
-    if (fs.length === 0) continue;
-    const bySev = {
-      block: fs.filter(f => f.severity === "block").length,
-      warn: fs.filter(f => f.severity === "warn").length,
-      info: fs.filter(f => f.severity === "info").length,
-    };
-    lines.push(`<details><summary><b>${check}</b> — ${fs.length} findings (${bySev.block} block, ${bySev.warn} warn, ${bySev.info} info)</summary>`);
-    lines.push("");
-    for (const f of fs) {
-      const mark = f.severity === "block" ? "🛑" : f.severity === "warn" ? "⚠️" : "ℹ️";
-      lines.push(`${mark} **${f.severity}** — ${f.summary}`);
-      for (const e of f.evidence.slice(0, 3)) {
-        lines.push(`  - \`${e.slice(0, 180).replace(/\n/g, " ")}\``);
-      }
-    }
-    lines.push("");
-    lines.push("</details>");
-    lines.push("");
-  }
-
-  lines.push("### Metrics");
-  lines.push("```json");
-  lines.push(JSON.stringify(v.metrics, null, 2));
-  lines.push("```");
-  lines.push("");
-  lines.push(`<sub>Lakehouse auditor · SHA ${v.head_sha.slice(0, 8)} · re-audit on new commit flips the status automatically.</sub>`);
-
-  return lines.join("\n");
-}
-
-function stubFinding(check: "dynamic" | "inference", why: string): Finding[] {
-  return [{ check, severity: "info", summary: `${check} check skipped — ${why}`, evidence: [why] }];
-}

auditor/checks/dynamic.ts

@@ -1,91 +0,0 @@
-// Dynamic execution check — runs the hybrid fixture and maps its
-// layer results to auditor Findings.
-//
-// A layer that fails with a "not implemented / 404 / slice N" error
-// gets severity=info (honest placeholder signal). A layer that fails
-// any other way gets severity=warn (something actually broke).
-// An info-level summary finding is always emitted carrying the real
-// numbers — shipped/placeholder phase counts, per-layer latency.
-
-import { runHybridFixture } from "../fixtures/hybrid_38_40_45.ts";
-import type { Finding } from "../types.ts";
-
-const PLACEHOLDER_MARKERS = [
-  "unimplemented",
-  " 404 ", "(404)", " 405 ", "(405)",
-  "slice 3", "slice 4", "slice 5",
-  "endpoint not built", "not yet",
-];
-
-function isPlaceholderFailure(err?: string): boolean {
-  if (!err) return false;
-  const low = err.toLowerCase();
-  return PLACEHOLDER_MARKERS.some(m => low.includes(m.toLowerCase()));
-}
-
-export async function runDynamicCheck(): Promise<Finding[]> {
-  const findings: Finding[] = [];
-
-  let result;
-  try {
-    result = await runHybridFixture();
-  } catch (e) {
-    // Fixture itself crashed — can't run dynamic check at all.
-    return [
-      {
-        check: "dynamic",
-        severity: "warn",
-        summary: `hybrid fixture crashed before completing: ${(e as Error).message.slice(0, 140)}`,
-        evidence: [(e as Error).message],
-      },
-    ];
-  }
-
-  // Per-layer findings for every non-ok layer.
-  for (const layer of result.layers) {
-    if (layer.ok) continue;
-    const placeholder = isPlaceholderFailure(layer.error);
-    findings.push({
-      check: "dynamic",
-      severity: placeholder ? "info" : "warn",
-      summary: placeholder
-        ? `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) honestly reports unimplemented`
-        : `hybrid fixture layer ${layer.layer} (Phase ${layer.phase}) failed — not a placeholder, a real failure`,
-      evidence: [
-        `evidence: ${layer.evidence.slice(0, 160)}`,
-        ...(layer.error ? [`error: ${layer.error.slice(0, 160)}`] : []),
-        `latency_ms: ${layer.latency_ms}`,
-      ],
-    });
-  }
-
-  // One overall summary with real numbers so the report shows what
-  // DID pass plus per-layer timing.
-  const metrics_preview = Object.entries(result.real_numbers)
-    .slice(0, 10)
-    .map(([k, v]) => `${k}=${v}`);
-  findings.push({
-    check: "dynamic",
-    severity: "info",
-    summary: `hybrid fixture overall=${result.overall}, shipped [${result.shipped_phases.join(", ")}], placeholder [${result.placeholder_phases.join(", ")}]`,
-    evidence: metrics_preview.length > 0 ? metrics_preview : ["no metrics emitted"],
-  });
-
-  // If the fixture ran at all but nothing passed, elevate one of the
-  // summary findings to warn — something more than "all honest
-  // placeholders" is wrong.
-  if (result.overall === "fail") {
-    findings.push({
-      check: "dynamic",
-      severity: "warn",
-      summary: `hybrid fixture: 0 layers passed (overall=fail)`,
-      evidence: [
-        "a total fixture fail usually means a precondition service is down",
-        "(gateway /health / sidecar / Langfuse /v1/chat) — NOT necessarily",
-        "the PR's code problem. Check service status before blaming the PR.",
-      ],
-    });
-  }
-
-  return findings;
-}

auditor/checks/inference.ts

@@ -1,206 +0,0 @@
-// Cloud inference check — wraps the proven run_codereview pattern
-// from llm_team_ui.py (same 3-stage framing, same cloud model) to
-// critique a PR's claims against its diff.
-//
-// Proved out 2026-04-22 via /tmp/codereview_runner.py — gpt-oss:120b
-// caught a real ternary bug in auditor/fixtures/hybrid_38_40_45.ts
-// that unit tests missed. This module reuses the reviewer prompt
-// shape (bugs / security / performance / style / edge cases) and
-// adds claim-vs-diff specific framing.
-//
-// Call surface: runInferenceCheck(claims, diff) → Finding[].
-// Cloud latency budget: ~60s (gpt-oss:120b reviewer typically 35-50s
-// with a 15KB diff + claim list).
-
-import type { Claim, Finding } from "../types.ts";
-
-const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
-const MODEL = process.env.LH_AUDITOR_REVIEW_MODEL ?? "gpt-oss:120b";
-// 40KB comfortably fits gpt-oss:120b's context. PR #1 (~39KB) was
-// previously truncated at 15KB causing the reviewer to miss later
-// files (gitea.ts, policy.ts) and flag "no Gitea client present" as a
-// block finding when the file was simply outside the truncation window.
-const MAX_DIFF_CHARS = 40000;
-const CALL_TIMEOUT_MS = 120_000;
-
-export async function runInferenceCheck(claims: Claim[], diff: string): Promise<Finding[]> {
-  if (claims.length === 0) {
-    return [{
-      check: "inference",
-      severity: "info",
-      summary: "no ship-claims extracted — skipping cloud inference",
-      evidence: ["parser returned empty claim list; nothing to verify against cloud"],
-    }];
-  }
-
-  const truncated = diff.length > MAX_DIFF_CHARS
-    ? diff.slice(0, MAX_DIFF_CHARS) + `\n...[${diff.length - MAX_DIFF_CHARS} more chars truncated]`
-    : diff;
-
-  // Build the reviewer prompt in the same shape as run_codereview's
-  // review stage (llm_team_ui.py:10950), adapted for claim verification:
-  //   "Task: ..."
-  //   "Code: ..."
-  //   "Review: bugs/security/perf/style/edge. Provide corrected code."
-  // We add: claim list upfront + ask for structured JSON verdict.
-  const systemMsg = [
-    "You review pull-request diffs against the author's own ship-claims.",
-    "For each claim, decide: is it backed by actual code in the diff, or is",
-    "it placeholder / aspirational / unwired?",
-    "",
-    "A claim is BACKED when the diff contains a real code path that delivers",
-    "the claimed behavior. A claim is NOT BACKED when:",
-    "  - the claim asserts functionality but the diff only adds types/fields",
-    "    with no consumer",
-    "  - the claim mentions tests but no test function was added",
-    "  - the claim claims integration but the integration point is a stub",
-    "  - the diff contains unimplemented!() / todo!() / TODO comments",
-    "  - the claim says 'works end-to-end' but the diff has no end-to-end test",
-    "",
-    "Respond with strict JSON only. No prose before or after. Shape:",
-    "{",
-    '  "claim_verdicts": [',
-    '    {"claim_idx": 0, "backed": false, "evidence": "short reason"}',
-    "  ],",
-    '  "unflagged_gaps": [',
-    '    {"location": "file:line", "summary": "short description"}',
-    "  ]",
-    "}",
-  ].join("\n");
-
-  const userMsg = [
-    `Ship-claims the author made (numbered 0..N-1):`,
-    claims.map((c, i) => `  ${i}. [${c.strength}] "${c.text}" at ${c.location}`).join("\n"),
-    "",
-    `Diff:`,
-    "```",
-    truncated,
-    "```",
-    "",
-    `For each numbered claim above, emit a claim_verdicts entry. For gaps the`,
-    `author DIDN'T claim but that look like placeholder code, emit unflagged_gaps.`,
-    `Strict JSON only, matching the shape described. No prose outside JSON.`,
-  ].join("\n");
-
-  let resp: Response;
-  try {
-    resp = await fetch(`${GATEWAY}/v1/chat`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        provider: "ollama_cloud",
-        model: MODEL,
-        messages: [
-          { role: "system", content: systemMsg },
-          { role: "user", content: userMsg },
-        ],
-        max_tokens: 3000,
-        temperature: 0.2,
-        think: true, // T3 overseer should reason — JSON shape is still required
-      }),
-      signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
-    });
-  } catch (e) {
-    // Cloud unreachable → soft-fail. Don't block a PR because the
-    // reviewer model is down. Static + dynamic + kb still run.
-    return [{
-      check: "inference",
-      severity: "info",
-      summary: "cloud inference unreachable — skipped",
-      evidence: [`fetch failed: ${(e as Error).message.slice(0, 180)}`],
-    }];
-  }
-
-  if (!resp.ok) {
-    return [{
-      check: "inference",
-      severity: "info",
-      summary: `cloud inference returned ${resp.status} — skipped`,
-      evidence: [`body: ${(await resp.text()).slice(0, 200)}`],
-    }];
-  }
-
-  const body: any = await resp.json();
-  const content: string = body?.choices?.[0]?.message?.content ?? "";
-  const usage = body?.usage ?? {};
-
-  const parsed = extractJson(content);
-  if (!parsed) {
-    return [{
-      check: "inference",
-      severity: "info",
-      summary: "cloud returned unparseable output — skipped",
-      evidence: [
-        `head: ${content.slice(0, 200)}`,
-        `tokens: ${usage.total_tokens ?? "?"}`,
-      ],
-    }];
-  }
-
-  const findings: Finding[] = [];
-
-  // One summary info finding so the verdict layer knows the check ran.
-  findings.push({
-    check: "inference",
-    severity: "info",
-    summary: `cloud review completed (model=${MODEL}, tokens=${usage.total_tokens ?? "?"})`,
-    evidence: [
-      `claim_verdicts: ${parsed.claim_verdicts?.length ?? 0}, unflagged_gaps: ${parsed.unflagged_gaps?.length ?? 0}`,
-    ],
-  });
-
-  for (const v of parsed.claim_verdicts ?? []) {
-    if (v?.backed === false) {
-      const idx = typeof v.claim_idx === "number" ? v.claim_idx : -1;
-      const claim = claims[idx];
-      if (!claim) continue;
-      // Strong+unbacked = BLOCK. That's the whole point of the auditor.
-      const sev: Finding["severity"] = claim.strength === "strong" ? "block"
-        : claim.strength === "moderate" ? "warn"
-        : "info";
-      findings.push({
-        check: "inference",
-        severity: sev,
-        claim_text: claim.text,
-        summary: `cloud: claim not backed — "${claim.text.slice(0, 100)}"`,
-        evidence: [
-          `at ${claim.location}`,
-          `cloud reason: ${String(v.evidence ?? "no reason given").slice(0, 200)}`,
-        ],
-      });
-    }
-  }
-
-  for (const g of parsed.unflagged_gaps ?? []) {
-    findings.push({
-      check: "inference",
-      severity: "warn",
-      summary: `cloud-flagged gap not in any claim: ${String(g?.summary ?? "?").slice(0, 120)}`,
-      evidence: [`location: ${String(g?.location ?? "?").slice(0, 140)}`],
-    });
-  }
-
-  return findings;
-}
-
-// Lift the first balanced JSON object out of the response. Tolerates
-// leading prose, code fences, and model reasoning preamble when the
-// cloud model ignored "strict JSON only."
-function extractJson(text: string): any | null {
-  const cleaned = text.replace(/^```(?:json)?\s*/im, "").replace(/```\s*$/im, "");
-  let depth = 0;
-  let start = -1;
-  for (let i = 0; i < cleaned.length; i++) {
-    const c = cleaned[i];
-    if (c === "{") {
-      if (depth === 0) start = i;
-      depth++;
-    } else if (c === "}") {
-      depth--;
-      if (depth === 0 && start >= 0) {
-        try { return JSON.parse(cleaned.slice(start, i + 1)); } catch { start = -1; }
-      }
-    }
-  }
-  return null;
-}

auditor/checks/kb_query.ts

@@ -1,183 +0,0 @@
-// Local-KB check — reads data/_kb/ + data/_observer/ + data/_bot/
-// for prior evidence bearing on this PR's claims. Cheap, offline,
-// no model calls. The point: if a claim like "Phase X shipped" has
-// a historical record of failing on the same signature before, the
-// auditor surfaces that pattern before the cloud check has to
-// infer it.
-//
-// What this check reads (all file-backed, append-only or periodic):
-//   data/_kb/outcomes.jsonl         — per-scenario outcomes (kb.ts)
-//   data/_kb/error_corrections.jsonl — fail→succeed deltas on same sig
-//   data/_observer/ops.jsonl         — observer ring → disk stream
-//   data/_bot/cycles/*.json          — bot cycle results
-//
-// Each JSONL line / per-cycle file is small; this check reads tails
-// only (last N lines or last M files) to stay cheap on large corpora.
-
-import { readFile, readdir, stat } from "node:fs/promises";
-import { join } from "node:path";
-import type { Claim, Finding } from "../types.ts";
-
-const KB_DIR = "/home/profit/lakehouse/data/_kb";
-const OBSERVER_OPS = "/home/profit/lakehouse/data/_observer/ops.jsonl";
-const BOT_CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
-const TAIL_LINES = 500;
-const MAX_BOT_CYCLE_FILES = 30;
-
-export async function runKbCheck(claims: Claim[]): Promise<Finding[]> {
-  const findings: Finding[] = [];
-
-  // 1. Recent scenario outcomes: are strong-claim-style phrases showing
-  //    up alongside failed events? That's "we claimed it worked" +
-  //    "it didn't" in the KB.
-  const scenarioFindings = await checkScenarioOutcomes(claims);
-  findings.push(...scenarioFindings);
-
-  // 2. Error corrections: any of the claims text overlap a
-  //    recently-observed fail→succeed pair? If yes, add context.
-  const correctionFindings = await checkErrorCorrections(claims);
-  findings.push(...correctionFindings);
-
-  // 3. Bot cycles: any prior bot cycle ended in tests_failed or
-  //    apply_failed on a file this PR is also touching?
-  const botFindings = await checkBotCycles();
-  findings.push(...botFindings);
-
-  // 4. Observer: count recent error events. High volume = shared
-  //    infra problem, worth flagging (context for other findings).
-  const obsFindings = await checkObserverStream();
-  findings.push(...obsFindings);
-
-  return findings;
-}
-
-async function tailJsonl<T = any>(path: string, n: number): Promise<T[]> {
-  try {
-    const raw = await readFile(path, "utf8");
-    const lines = raw.split("\n").filter(l => l.length > 0);
-    const slice = lines.slice(-n);
-    const out: T[] = [];
-    for (const line of slice) {
-      try { out.push(JSON.parse(line)); } catch { /* skip malformed */ }
-    }
-    return out;
-  } catch {
-    return [];
-  }
-}
-
-async function checkScenarioOutcomes(_claims: Claim[]): Promise<Finding[]> {
-  const outcomes = await tailJsonl<any>(join(KB_DIR, "outcomes.jsonl"), TAIL_LINES);
-  if (outcomes.length === 0) return [];
-  const totalEvents = outcomes.reduce((s, o) => s + (o.total_events ?? 0), 0);
-  const okEvents = outcomes.reduce((s, o) => s + (o.ok_events ?? 0), 0);
-  const failRate = totalEvents > 0 ? 1 - okEvents / totalEvents : 0;
-
-  if (totalEvents === 0) {
-    return [{
-      check: "kb_query",
-      severity: "info",
-      summary: `KB: no scenario outcomes on file — learning loop is empty`,
-      evidence: [`data/_kb/outcomes.jsonl has ${outcomes.length} entries with 0 total events`],
-    }];
-  }
-
-  const recent = outcomes.slice(-10);
-  const recentFailSigs: string[] = recent
-    .filter(o => (o.ok_events ?? 0) < (o.total_events ?? 0))
-    .map(o => o.sig_hash)
-    .filter(s => typeof s === "string");
-
-  const findings: Finding[] = [{
-    check: "kb_query",
-    severity: failRate > 0.3 ? "warn" : "info",
-    summary: `KB: ${outcomes.length} recent scenario runs, ${okEvents}/${totalEvents} events ok (fail rate ${(failRate * 100).toFixed(1)}%)`,
-    evidence: [
-      `most recent: ${recent[recent.length - 1]?.run_id ?? "?"}`,
-      `recent failing sigs: ${recentFailSigs.length > 0 ? recentFailSigs.slice(-3).join(", ") : "none"}`,
-    ],
-  }];
-  return findings;
-}
-
-async function checkErrorCorrections(_claims: Claim[]): Promise<Finding[]> {
-  const corrections = await tailJsonl<any>(join(KB_DIR, "error_corrections.jsonl"), TAIL_LINES);
-  if (corrections.length === 0) return [];
-  return [{
-    check: "kb_query",
-    severity: "info",
-    summary: `KB: ${corrections.length} error corrections on file (fail→succeed pairs)`,
-    evidence: [
-      corrections.length > 0
-        ? `most recent: ${String(corrections[corrections.length - 1]?.sig_hash ?? "?").slice(0, 24)}`
-        : "none",
-    ],
-  }];
-}
-
-async function checkBotCycles(): Promise<Finding[]> {
-  let entries: string[] = [];
-  try { entries = await readdir(BOT_CYCLES_DIR); }
-  catch { return []; }
-
-  const jsonFiles = entries.filter(e => e.endsWith(".json"));
-  if (jsonFiles.length === 0) return [];
-
-  // Sort by mtime desc, take most recent N
-  const withStat = await Promise.all(
-    jsonFiles.map(async name => {
-      try { return { name, mtime: (await stat(join(BOT_CYCLES_DIR, name))).mtimeMs }; }
-      catch { return { name, mtime: 0 }; }
-    }),
-  );
-  const recent = withStat.sort((a, b) => b.mtime - a.mtime).slice(0, MAX_BOT_CYCLE_FILES);
-
-  const outcomes: Record<string, number> = {};
-  for (const { name } of recent) {
-    try {
-      const r = JSON.parse(await readFile(join(BOT_CYCLES_DIR, name), "utf8"));
-      const o = String(r.outcome ?? "unknown");
-      outcomes[o] = (outcomes[o] ?? 0) + 1;
-    } catch { /* skip */ }
-  }
-
-  const summary = Object.entries(outcomes)
-    .sort((a, b) => b[1] - a[1])
-    .map(([k, v]) => `${k}=${v}`)
-    .join(", ");
-
-  const failCount = (outcomes["tests_failed"] ?? 0) + (outcomes["apply_failed"] ?? 0) + (outcomes["model_failed"] ?? 0);
-  return [{
-    check: "kb_query",
-    severity: failCount > recent.length / 2 ? "warn" : "info",
-    summary: `KB: bot recorded ${recent.length} recent cycles — ${summary || "no outcomes parsed"}`,
-    evidence: [
-      `dir: ${BOT_CYCLES_DIR}`,
-      `fail-class total: ${failCount} / ${recent.length}`,
-    ],
-  }];
-}
-
-async function checkObserverStream(): Promise<Finding[]> {
-  const ops = await tailJsonl<any>(OBSERVER_OPS, TAIL_LINES);
-  if (ops.length === 0) return [];
-  const failures = ops.filter(o => o.ok === false).length;
-  return [{
-    check: "kb_query",
-    severity: "info",
-    summary: `KB: observer stream ${ops.length} recent ops, ${failures} failures`,
-    evidence: [
-      `source: ${OBSERVER_OPS}`,
-      `by source: ${observerBySource(ops)}`,
-    ],
-  }];
-}
-
-function observerBySource(ops: any[]): string {
-  const c: Record<string, number> = {};
-  for (const o of ops) {
-    const s = String(o.source ?? "unknown");
-    c[s] = (c[s] ?? 0) + 1;
-  }
-  return Object.entries(c).sort((a, b) => b[1] - a[1]).map(([k, v]) => `${k}=${v}`).join(", ") || "empty";
-}

auditor/checks/static.ts

@@ -1,159 +0,0 @@
-// Static diff check — grep-style, no AST, no LLM. Looks for patterns
-// that are high-signal evidence of placeholder code.
-//
-// Findings are severity-graded:
-//   block — explicit non-impl markers (unimplemented!, todo!,
-//           panic!("not implemented"), throw new Error("not implemented"))
-//   warn  — TODO / FIXME / XXX / HACK comments on added lines,
-//           new struct fields with no read-site anywhere in the diff,
-//           suspiciously-empty function bodies ({ Ok(()) } / {} when
-//           the commit message claims the fn "implements" something)
-//   info  — hardcoded "test" / "dummy" / "placeholder" strings in
-//           added lines (could be real, just flag for inspection)
-//
-// Consumes: raw unified diff text from Gitea.
-
-import type { Finding } from "../types.ts";
-
-// Rust + TypeScript patterns that almost always indicate "this is
-// not actually implemented yet."
-const BLOCK_PATTERNS: Array<{ re: RegExp; why: string }> = [
-  { re: /\bunimplemented!\s*\(/, why: "unimplemented!() macro call" },
-  { re: /\btodo!\s*\(/, why: "todo!() macro call" },
-  { re: /panic!\s*\(\s*"(?:not implemented|TODO|not yet|unimpl)/i, why: "panic! with not-implemented message" },
-  { re: /throw\s+new\s+Error\s*\(\s*['"](?:not implemented|TODO|unimpl)/i, why: "throw Error 'not implemented'" },
-];
-
-const WARN_COMMENT_PATTERNS: Array<{ re: RegExp; why: string }> = [
-  { re: /^\+.*\/\/\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
-  { re: /^\+.*#\s*(TODO|FIXME|XXX|HACK)\b/i, why: "TODO/FIXME/XXX/HACK comment added" },
-];
-
-const INFO_HARDCODED_PATTERNS: Array<{ re: RegExp; why: string }> = [
-  { re: /"(?:placeholder|dummy|foobar|xxx|replaceme|changeme)"/i, why: "suspicious hardcoded string" },
-];
-
-export function runStaticCheck(diff: string): Finding[] {
-  const findings: Finding[] = [];
-
-  // Per-file walk: only look at ADDED lines (prefix '+' but not '+++'
-  // which is the diff header).
-  const perFile = splitDiffByFile(diff);
-
-  for (const [path, lines] of perFile) {
-    // Skip diff bookkeeping + pure-delete files
-    if (!lines.some(l => l.startsWith("+") && !l.startsWith("+++"))) continue;
-
-    // The auditor's own check files literally contain the BLOCK
-    // patterns as regex definitions (BLOCK_PATTERNS in this file,
-    // prompt examples in inference.ts). Skipping BLOCK scan on these
-    // specific paths prevents the checker from self-flagging its own
-    // string literals. WARN/INFO patterns still run — those genuinely
-    // could indicate problems in the checker's own code (TODO
-    // comments don't self-define).
-    const isAuditorCheckerFile = path.startsWith("auditor/checks/") ||
-                                 path.startsWith("auditor/fixtures/");
-
-    for (let idx = 0; idx < lines.length; idx++) {
-      const line = lines[idx];
-      if (!line.startsWith("+") || line.startsWith("+++")) continue;
-      const added = line.slice(1);
-
-      if (!isAuditorCheckerFile) {
-        for (const { re, why } of BLOCK_PATTERNS) {
-          if (re.test(added)) {
-            findings.push({
-              check: "static",
-              severity: "block",
-              summary: `${why} in ${path}`,
-              evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
-            });
-          }
-        }
-      }
-      for (const { re, why } of WARN_COMMENT_PATTERNS) {
-        if (re.test(line)) {
-          findings.push({
-            check: "static",
-            severity: "warn",
-            summary: `${why} in ${path}`,
-            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
-          });
-        }
-      }
-      for (const { re, why } of INFO_HARDCODED_PATTERNS) {
-        if (re.test(added)) {
-          findings.push({
-            check: "static",
-            severity: "info",
-            summary: `${why} in ${path}`,
-            evidence: [`${path}:+${idx + 1}: ${added.trim().slice(0, 160)}`],
-          });
-        }
-      }
-    }
-
-    // "Field added but never read" heuristic — catches exactly the
-    // Phase 45 DocRef placeholder pattern. Limited to the diff itself:
-    // we're not doing a full-codebase grep here (too noisy; callers
-    // elsewhere might exist). The point is: if NEITHER this diff nor
-    // any other line in the diff reads the field, the PR is shipping
-    // state without a consumer.
-    const addedLines = lines.filter(l => l.startsWith("+") && !l.startsWith("+++"))
-      .map(l => l.slice(1));
-    const newFields = extractNewFields(addedLines);
-    for (const field of newFields) {
-      const readPattern = new RegExp(`[\\.:]\\s*${escape(field)}\\b|\\b${escape(field)}\\s*:`);
-      // The definition line itself matches readPattern — filter it out
-      // by requiring at least TWO lines in the diff mention the field
-      // (one defines, one reads).
-      const hits = addedLines.filter(l => readPattern.test(l));
-      if (hits.length < 2) {
-        findings.push({
-          check: "static",
-          severity: "warn",
-          summary: `field '${field}' added in ${path} but no read-site in the diff — could be placeholder state without a consumer`,
-          evidence: [`${path}: added '${field}' with no reader; rest of diff has ${hits.length - 1} mentions`],
-        });
-      }
-    }
-  }
-
-  return findings;
-}
-
-function splitDiffByFile(diff: string): Map<string, string[]> {
-  const out = new Map<string, string[]>();
-  let current: string | null = null;
-  let buf: string[] = [];
-  for (const line of diff.split(/\r?\n/)) {
-    const m = line.match(/^diff --git a\/(\S+) b\/(\S+)/);
-    if (m) {
-      if (current) out.set(current, buf);
-      current = m[2];
-      buf = [];
-      continue;
-    }
-    buf.push(line);
-  }
-  if (current) out.set(current, buf);
-  return out;
-}
-
-// Extract new `pub name: Type,` fields from added lines. Rust syntax.
-// Narrowly-scoped: only matches at the start of a trimmed line,
-// requires `pub ` prefix, ignores `pub fn` / `pub struct` / etc.
-function extractNewFields(addedLines: string[]): string[] {
-  const fields = new Set<string>();
-  for (const line of addedLines) {
-    const t = line.trim();
-    // pub NAME: Type,
-    const m = t.match(/^pub\s+(?!fn\b|struct\b|enum\b|mod\b|use\b|trait\b|impl\b|const\b|static\b|type\b)(\w+)\s*:/);
-    if (m) fields.add(m[1]);
-  }
-  return Array.from(fields);
-}
-
-function escape(s: string): string {
-  return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
-}

auditor/claim_parser.ts

@@ -1,119 +0,0 @@
-// Claim parser — reads commit messages + PR body, extracts ship-claims.
-//
-// A "ship-claim" is any phrase that asserts functionality is working,
-// tested, complete, or landed. These are the assertions the downstream
-// checks (static/dynamic/inference/kb) try to falsify.
-//
-// Heuristic approach (regex + strength grading) — intentionally NOT
-// using an LLM here. Reason: the inference check already asks a cloud
-// model "does this match the claim?". The parser's job is to surface
-// the claim substrates, not judge them. Over-engineering the parser
-// risks false-negatives when the cloud model was going to catch it
-// anyway.
-
-import type { Claim, PrSnapshot } from "./types.ts";
-
-// Strong claims: explicit end-to-end + verification vocabulary
-const STRONG_PATTERNS: RegExp[] = [
-  /\bverified\s+(end[- ]to[- ]end|live|in\s+production|against)\b/i,
-  /\btested\s+(live|end[- ]to[- ]end|against|with)\b/i,
-  /\bworks\s+(end[- ]to[- ]end|live|in\s+production)\b/i,
-  /\bproduction[- ]ready\b/i,
-  /\bfully\s+(functional|wired|working)\b/i,
-  /\bphase\s+\d+(\.\d+)?\s+(shipped|complete|done|landed)\b/i,
-  /\bground\s+truth\b/i,
-  /\bproven\b/i,
-];
-
-// Moderate claims: asserted completion or pass but without the strong
-// verification qualifier.
-const MODERATE_PATTERNS: RegExp[] = [
-  /\bshipped\b/i,
-  /\blanded\b/i,
-  /\bgreen\b/i,
-  /\b(tests?\s+)?pass(ing|ed)\b/i,
-  /\bcomplet(e|ed)\b/i,
-  /\bdone\b/i,
-  /\bwired\b/i,
-  /\bfixed\b/i,
-  /\bworks\b/i,
-];
-
-// Weak claims: aspirational or hedged. Usually low-risk but recorded
-// for completeness.
-const WEAK_PATTERNS: RegExp[] = [
-  /\bshould\s+work\b/i,
-  /\bexpected\s+to\b/i,
-  /\bintended\s+to\b/i,
-  /\bwill\s+(work|handle|support)\b/i,
-  /\bprobably\b/i,
-];
-
-export interface ParsedClaims {
-  claims: Claim[];
-  commits_scanned: number;
-}
-
-export function parseClaims(pr: PrSnapshot): ParsedClaims {
-  const claims: Claim[] = [];
-
-  // PR body — every matching line becomes a claim at location "pr_body:N"
-  if (pr.body) {
-    scanText(pr.body, "pr_body", pr.head_sha, claims);
-  }
-
-  // Each commit message gets its own scan.
-  for (const c of pr.commits) {
-    if (!c.message) continue;
-    scanText(c.message, `commit:${c.sha.slice(0, 8)}`, c.sha, claims);
-  }
-
-  return { claims, commits_scanned: pr.commits.length };
-}
-
-function scanText(text: string, location_prefix: string, commit_sha: string, out: Claim[]): void {
-  const lines = text.split(/\r?\n/);
-  for (let i = 0; i < lines.length; i++) {
-    const line = lines[i];
-    if (line.length < 3) continue;
-
-    // Strong patterns first — if a line matches strong, it's strong,
-    // don't double-count as moderate.
-    const strong = firstMatch(line, STRONG_PATTERNS);
-    if (strong) {
-      out.push({
-        text: line.trim().slice(0, 200),
-        commit_sha,
-        location: `${location_prefix}:${i + 1}`,
-        strength: "strong",
-      });
-      continue;
-    }
-    const moderate = firstMatch(line, MODERATE_PATTERNS);
-    if (moderate) {
-      out.push({
-        text: line.trim().slice(0, 200),
-        commit_sha,
-        location: `${location_prefix}:${i + 1}`,
-        strength: "moderate",
-      });
-      continue;
-    }
-    const weak = firstMatch(line, WEAK_PATTERNS);
-    if (weak) {
-      out.push({
-        text: line.trim().slice(0, 200),
-        commit_sha,
-        location: `${location_prefix}:${i + 1}`,
-        strength: "weak",
-      });
-    }
-  }
-}
-
-function firstMatch(text: string, patterns: RegExp[]): RegExp | null {
-  for (const p of patterns) {
-    if (p.test(text)) return p;
-  }
-  return null;
-}

auditor/fixtures/cli.ts

@@ -1,49 +0,0 @@
-// Standalone runner for auditor fixtures. Invoke:
-//
-//   bun run auditor/fixtures/cli.ts hybrid_38_40_45
-//
-// Prints human-readable per-layer breakdown + JSON result so the
-// output can be captured by the dynamic check without re-running.
-
-import { runHybridFixture, type FixtureResult } from "./hybrid_38_40_45.ts";
-
-const fixtureArg = process.argv[2] ?? "hybrid_38_40_45";
-
-let result: FixtureResult;
-switch (fixtureArg) {
-  case "hybrid_38_40_45":
-    result = await runHybridFixture();
-    break;
-  default:
-    console.error(`unknown fixture: ${fixtureArg}`);
-    process.exit(2);
-}
-
-// Human-readable summary
-console.error(""); // blank line to stderr so stdout JSON stays clean
-console.error(`─── Fixture: ${result.fixture} ───`);
-console.error(`  overall: ${result.overall.toUpperCase()}`);
-console.error(`  shipped:     [${result.shipped_phases.join(", ") || "—"}]`);
-console.error(`  placeholder: [${result.placeholder_phases.join(", ") || "—"}]`);
-console.error("");
-for (const l of result.layers) {
-  const mark = l.ok ? "✓" : "✗";
-  const phaseStr = `Phase ${l.phase}`.padEnd(11);
-  console.error(`  ${mark} ${phaseStr} ${l.layer.padEnd(30)} ${String(l.latency_ms).padStart(6)}ms`);
-  if (l.ok) {
-    console.error(`      ${l.evidence.slice(0, 200)}`);
-  } else {
-    console.error(`      ERROR: ${l.error?.slice(0, 240) ?? "unknown"}`);
-  }
-}
-console.error("");
-for (const n of result.notes) console.error(`  note: ${n}`);
-console.error("");
-
-// Machine-readable output on stdout.
-console.log(JSON.stringify(result, null, 2));
-
-// Exit code reflects overall: 0 pass, 2 partial, 1 fail.
-// Dynamic check reads this AND the JSON; partial-pass is treated
-// as informative (some layers shipped), not blocking on its own.
-process.exit(result.overall === "pass" ? 0 : result.overall === "partial_pass" ? 2 : 1);

auditor/fixtures/hybrid_38_40_45.ts

@@ -1,335 +0,0 @@
-// The never-run hybrid test fixture. Exercises the full stack end-to-end
-// across Phase 38 (/v1/chat), Phase 40 (Langfuse tracing), Phase 45
-// slice 1 (DocRef on playbook), and Phase 45 slice 2 (context7 bridge
-// drift detection).
-//
-// Returns HONEST per-layer results. A layer that's unimplemented or
-// broken returns ok=false with a real error; overall verdict flags
-// which phases are genuinely shipped vs still placeholder.
-//
-// Deterministic: always runs the same payload so a run's output is
-// comparable across audits. Uses TEST-prefixed names so playbook
-// state doesn't get polluted with production-looking fixture data.
-//
-// Preconditions:
-//   - gateway up on :3100
-//   - Python sidecar up on :3200
-//   - Ollama local + Ollama Cloud key loaded
-//   - Langfuse up on :3001
-//   - context7 bridge up on :3900 (manual-start per mcp-server/README)
-//     If bridge isn't running, that layer reports ok=false with a
-//     clear error — the fixture doesn't try to auto-start it.
-
-import { readFile } from "node:fs/promises";
-
-const GATEWAY = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
-const LANGFUSE = process.env.LANGFUSE_URL ?? "http://localhost:3001";
-const BRIDGE = process.env.CONTEXT7_BRIDGE_URL ?? "http://localhost:3900";
-const FIXTURE_ID = "auditor:hybrid_38_40_45:v1";
-const TEST_WORKER_NAME = "__auditor_test_worker__";
-const STALE_HASH = "stale-hash-for-drift-proof";
-// Unique-per-run identifiers so each fixture invocation hits the ADD
-// path in upsert_entry (not UPDATE/NOOP on a leftover from a prior
-// run). Catches state pollution + avoids interaction with task #12
-// (UPDATE branch silently drops doc_refs).
-const RUN_NONCE = Date.now().toString(36);
-const TEST_WORKER_NAME_VERSIONED = `__auditor_test_worker_${RUN_NONCE}__`;
-const TEST_OPERATION_VERSIONED = `TEST: fill: __auditor_${RUN_NONCE}__ x1 in TestCity, TC`;
-
-export interface LayerResult {
-  layer: string;
-  phase: string;
-  ok: boolean;
-  latency_ms: number;
-  evidence: string;
-  error?: string;
-  // Layer-specific numbers that downstream checks can aggregate.
-  metrics?: Record<string, number | string | boolean>;
-}
-
-export interface FixtureResult {
-  fixture: string;
-  ran_at: string;
-  overall: "pass" | "partial_pass" | "fail";
-  layers: LayerResult[];
-  real_numbers: Record<string, number>;
-  shipped_phases: string[];
-  placeholder_phases: string[];
-  notes: string[];
-}
-
-// Wraps a layer's async body with timing + structured error capture.
-// On throw: returns ok=false with the error string as evidence, so
-// downstream verdict code treats it as a layer failure rather than
-// a fixture crash.
-async function measureLayer(
-  layer: string,
-  phase: string,
-  body: () => Promise<{ evidence: string; metrics?: Record<string, number | string | boolean> }>,
-): Promise<LayerResult> {
-  const t0 = Date.now();
-  try {
-    const { evidence, metrics } = await body();
-    return {
-      layer, phase, ok: true,
-      latency_ms: Date.now() - t0,
-      evidence, metrics,
-    };
-  } catch (e) {
-    return {
-      layer, phase, ok: false,
-      latency_ms: Date.now() - t0,
-      evidence: `FAILED: ${(e as Error).message.slice(0, 200)}`,
-      error: (e as Error).message,
-    };
-  }
-}
-
-async function langfuseAuth(): Promise<{ pk: string; sk: string } | null> {
-  // Default credentials match mcp-server/tracing.ts. Same sources the
-  // Rust side uses in gateway/src/v1/langfuse_trace.rs.
-  let pk = process.env.LANGFUSE_PUBLIC_KEY ?? "pk-lf-staffing";
-  let sk = process.env.LANGFUSE_SECRET_KEY ?? "sk-lf-staffing-secret";
-  if (!pk || !sk) return null;
-  return { pk, sk };
-}
-
-export async function runHybridFixture(): Promise<FixtureResult> {
-  const result: FixtureResult = {
-    fixture: FIXTURE_ID,
-    ran_at: new Date().toISOString(),
-    overall: "fail",
-    layers: [],
-    real_numbers: {},
-    shipped_phases: [],
-    placeholder_phases: [],
-    notes: [],
-  };
-
-  // ========================================================================
-  // Layer 1 — Phase 38: POST /v1/chat returns valid OpenAI shape
-  // ========================================================================
-  // Captured HERE, immediately before the chat layer runs, so layer 2's
-  // Langfuse-trace filter uses the actual moment the chat call was
-  // attempted — not the fixture start time. Earlier draft had a
-  // meaningless ternary returning result.ran_at on both branches; the
-  // LLM-Team codereview (2026-04-22) caught this and flagged it as a
-  // false-negative window on traces created between fixture-start and
-  // chat-fetch.
-  const chat_request_sent_ms = Date.now();
-
-  const l1 = await measureLayer("phase38_chat", "38", async () => {
-    const r = await fetch(`${GATEWAY}/v1/chat`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        model: "qwen3.5:latest",
-        messages: [
-          { role: "system", content: "Respond in 8 words or fewer." },
-          { role: "user", content: "Name one Docker command." },
-        ],
-        max_tokens: 60,
-        temperature: 0.2,
-      }),
-      signal: AbortSignal.timeout(60000),
-    });
-    if (!r.ok) throw new Error(`gateway ${r.status}: ${await r.text()}`);
-    const j: any = await r.json();
-    const content = j.choices?.[0]?.message?.content;
-    if (typeof content !== "string" || content.length === 0) {
-      throw new Error(`empty content — think-false default may not be wired: ${JSON.stringify(j).slice(0, 200)}`);
-    }
-    return {
-      evidence: `content="${content.slice(0, 60)}" tokens=${j.usage?.total_tokens}`,
-      metrics: {
-        prompt_tokens: j.usage?.prompt_tokens ?? 0,
-        completion_tokens: j.usage?.completion_tokens ?? 0,
-        total_tokens: j.usage?.total_tokens ?? 0,
-        content_nonempty: true,
-      },
-    };
-  });
-  result.layers.push(l1);
-
-  // ========================================================================
-  // Layer 2 — Phase 40: Langfuse trace lands within 3s
-  // ========================================================================
-  // Fire-and-forget tracing means we need a brief sleep before query.
-  await new Promise(res => setTimeout(res, 2500));
-  const l2 = await measureLayer("phase40_langfuse_trace", "40", async () => {
-    const auth = await langfuseAuth();
-    if (!auth) throw new Error("Langfuse credentials not in env");
-    const r = await fetch(`${LANGFUSE}/api/public/traces?limit=5&name=v1.chat:ollama`, {
-      headers: { Authorization: `Basic ${btoa(`${auth.pk}:${auth.sk}`)}` },
-      signal: AbortSignal.timeout(10000),
-    });
-    if (!r.ok) throw new Error(`langfuse ${r.status}: ${await r.text()}`);
-    const j: any = await r.json();
-    const items = Array.isArray(j.data) ? j.data : [];
-    // Filter on the chat-request timestamp captured above. A Langfuse
-    // trace must be newer than the moment we fired /v1/chat to plausibly
-    // belong to our request. Using fixture start time (result.ran_at)
-    // was wrong and could false-negative on slow fixtures.
-    const recent = items.filter((t: any) => Date.parse(t.timestamp) >= chat_request_sent_ms);
-    if (recent.length === 0) {
-      throw new Error(`no v1.chat:ollama trace since ${new Date(chat_request_sent_ms).toISOString()} (${items.length} older traces visible, Langfuse reachable — tracing is not firing)`);
-    }
-    const trace = recent[0];
-    return {
-      evidence: `trace ${trace.id.slice(0, 8)} name=${trace.name} age=${Date.now() - Date.parse(trace.timestamp)}ms`,
-      metrics: {
-        trace_age_ms: Date.now() - Date.parse(trace.timestamp),
-        trace_latency_reported: Number(trace.latency ?? 0),
-        recent_trace_count: recent.length,
-      },
-    };
-  });
-  result.layers.push(l2);
-
-  // ========================================================================
-  // Layer 3 — Phase 45 slice 1: /seed accepts doc_refs and persists them
-  // ========================================================================
-  // Seed a playbook with doc_refs referencing Docker at a stale hash.
-  // Ignore the cleanup concern for v1 — this fixture pollutes the
-  // in-memory playbook state; operator can retire test entries.
-  const l3 = await measureLayer("phase45_seed_with_doc_refs", "45.1", async () => {
-    const r = await fetch(`${GATEWAY}/vectors/playbook_memory/seed`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({
-        operation: TEST_OPERATION_VERSIONED,
-        approach: "auditor hybrid fixture run",
-        context: "doc_refs integration test — retire after audit",
-        endorsed_names: [TEST_WORKER_NAME_VERSIONED],
-        append: true,
-        doc_refs: [
-          {
-            tool: "docker",
-            version_seen: "24.0.7",
-            snippet_hash: STALE_HASH,
-            source_url: "https://context7.com/docker/docs",
-            seen_at: new Date().toISOString(),
-          },
-        ],
-      }),
-      signal: AbortSignal.timeout(30000),
-    });
-    if (!r.ok) throw new Error(`seed ${r.status}: ${await r.text()}`);
-    const j: any = await r.json();
-    // Verify doc_refs actually persisted — read state.json directly.
-    // That's the canonical source of truth; the seed response may or
-    // may not echo doc_refs back.
-    const state_raw = await readFile("/home/profit/lakehouse/data/_playbook_memory/state.json", "utf8");
-    const state = JSON.parse(state_raw);
-    const entries = state.entries ?? [];
-    const ours = entries.find((e: any) => (e.endorsed_names ?? []).includes(TEST_WORKER_NAME_VERSIONED));
-    if (!ours) throw new Error(`seeded entry not found in state.json (worker ${TEST_WORKER_NAME_VERSIONED} not present)`);
-    const docRefs = ours.doc_refs ?? [];
-    if (docRefs.length === 0) {
-      throw new Error("entry saved but doc_refs field is empty — the field accepted by the API isn't being persisted");
-    }
-    return {
-      evidence: `playbook ${ours.playbook_id.slice(0, 16)} persisted with doc_refs.length=${docRefs.length}; first tool=${docRefs[0]?.tool} hash=${docRefs[0]?.snippet_hash}`,
-      metrics: {
-        doc_refs_stored: docRefs.length,
-        entries_after: j.entries_after ?? -1,
-        playbook_id: ours.playbook_id,
-        mode: j.outcome?.mode ?? "unknown",
-      },
-    };
-  });
-  result.layers.push(l3);
-
-  // ========================================================================
-  // Layer 4 — Phase 45 slice 2: context7 bridge detects drift vs stale hash
-  // ========================================================================
-  const l4 = await measureLayer("phase45_bridge_diff", "45.2", async () => {
-    // Health check first — if bridge isn't running, fail with a clear
-    // message rather than mysterious connection refused.
-    try {
-      const h = await fetch(`${BRIDGE}/health`, { signal: AbortSignal.timeout(3000) });
-      if (!h.ok) throw new Error(`bridge /health ${h.status}`);
-    } catch (e) {
-      throw new Error(`context7 bridge at ${BRIDGE} is not reachable (bridge is manual-start; run 'bun run mcp-server/context7_bridge.ts'): ${(e as Error).message}`);
-    }
-    const r = await fetch(`${BRIDGE}/docs/docker/diff?since=${encodeURIComponent(STALE_HASH)}`, {
-      signal: AbortSignal.timeout(30000),
-    });
-    if (!r.ok) throw new Error(`bridge diff ${r.status}: ${await r.text()}`);
-    const j: any = await r.json();
-    if (j.drifted !== true) {
-      throw new Error(`expected drifted=true against ${STALE_HASH}; got ${JSON.stringify(j)}`);
-    }
-    return {
-      evidence: `bridge confirms drift: current=${j.current_snippet_hash} previous=${j.previous_snippet_hash} upstream_updated=${j.last_updated_upstream}`,
-      metrics: {
-        drifted: true,
-        current_hash_length: String(j.current_snippet_hash ?? "").length,
-        library_id: j.library_id,
-      },
-    };
-  });
-  result.layers.push(l4);
-
-  // ========================================================================
-  // Layer 5 — Phase 45 slice 3: /doc_drift/check/{id} endpoint
-  //                              (EXPECTED TO FAIL — endpoint unimplemented)
-  // ========================================================================
-  const l5 = await measureLayer("phase45_slice3_drift_flag", "45.3", async () => {
-    const pid = String(l3.metrics?.playbook_id ?? "");
-    if (!pid) throw new Error("layer 3 did not produce a playbook_id to check");
-    const r = await fetch(`${GATEWAY}/vectors/playbook_memory/doc_drift/check/${encodeURIComponent(pid)}`, {
-      method: "POST",
-      signal: AbortSignal.timeout(10000),
-    });
-    if (r.status === 404 || r.status === 405) {
-      // This is the HONEST signal: the endpoint doesn't exist yet.
-      // Fail this layer with a clear marker — not a silent pass.
-      throw new Error(`endpoint unimplemented (${r.status}) — Phase 45 slice 3 is still placeholder. doc_refs are persisted (layer 3) and the bridge can answer drift questions (layer 4), but nothing ties them together into a playbook flag yet.`);
-    }
-    if (!r.ok) throw new Error(`drift/check ${r.status}: ${await r.text()}`);
-    const j: any = await r.json();
-    // If the endpoint DID exist, assert the flag got set.
-    if (!j.flagged) {
-      throw new Error(`endpoint responded but flag not set: ${JSON.stringify(j).slice(0, 200)}`);
-    }
-    return {
-      evidence: `playbook flagged: ${JSON.stringify(j).slice(0, 160)}`,
-      metrics: { flagged: true },
-    };
-  });
-  result.layers.push(l5);
-
-  // ========================================================================
-  // Verdict assembly
-  // ========================================================================
-  for (const l of result.layers) {
-    if (l.ok) result.shipped_phases.push(l.phase);
-    else result.placeholder_phases.push(l.phase);
-    result.real_numbers[`${l.layer}_latency_ms`] = l.latency_ms;
-    if (l.metrics) {
-      for (const [k, v] of Object.entries(l.metrics)) {
-        if (typeof v === "number") result.real_numbers[`${l.layer}.${k}`] = v;
-      }
-    }
-  }
-
-  const anyFail = result.placeholder_phases.length > 0;
-  const allPass = result.layers.every(l => l.ok);
-  result.overall = allPass ? "pass" : (result.shipped_phases.length > 0 ? "partial_pass" : "fail");
-
-  if (!l5.ok) {
-    result.notes.push(
-      "Phase 45 slice 3 (doc_drift/check endpoint) is the expected-fail layer. " +
-      "Layer 5 failing with a 404/405 is the CORRECT honest signal — don't mask this " +
-      "to get a green result. When slice 3 ships, layer 5 flips to ok=true automatically."
-    );
-  }
-  if (anyFail) {
-    result.notes.push(
-      `${result.placeholder_phases.length} layer(s) failed. Check evidence per layer for specifics.`
-    );
-  }
-
-  return result;
-}

auditor/gitea.ts

@@ -1,145 +0,0 @@
-// Gitea API client. Minimal surface — only what the auditor needs:
-// list open PRs, get commits + files for a PR, fetch a diff, post a
-// commit status, post a review.
-//
-// Auth: reads PAT from ~/.git-credentials (set up by the credential
-// helper flow in 2026-04-22 session). Gitea's "token" auth scheme
-// matches what `git fetch` is already using.
-
-import { readFile } from "node:fs/promises";
-import type { PrSnapshot } from "./types.ts";
-
-const HOST = process.env.GITEA_HOST ?? "https://git.agentview.dev";
-const OWNER = "profit";
-const REPO = "lakehouse";
-const CRED_FILE = "/home/profit/.git-credentials";
-
-let cachedPat: string | null = null;
-
-async function getPat(): Promise<string> {
-  if (cachedPat) return cachedPat;
-  const raw = await readFile(CRED_FILE, "utf8");
-  for (const line of raw.split("\n")) {
-    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
-    if (m) { cachedPat = m[1]; return m[1]; }
-  }
-  throw new Error(`no Gitea PAT in ${CRED_FILE}`);
-}
-
-async function giteaFetch(path: string, init: RequestInit = {}): Promise<Response> {
-  const pat = await getPat();
-  const url = `${HOST}/api/v1${path}`;
-  const headers = new Headers(init.headers);
-  headers.set("Authorization", `token ${pat}`);
-  if (init.body && !headers.has("content-type")) {
-    headers.set("content-type", "application/json");
-  }
-  return fetch(url, { ...init, headers, signal: AbortSignal.timeout(20000) });
-}
-
-export async function listOpenPrs(): Promise<PrSnapshot[]> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls?state=open&page=1&limit=50`);
-  if (!r.ok) throw new Error(`listOpenPrs ${r.status}: ${await r.text()}`);
-  const rows = (await r.json()) as any[];
-  return Promise.all(rows.map(row => snapshotFromPr(row)));
-}
-
-export async function getPrSnapshot(num: number): Promise<PrSnapshot> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}`);
-  if (!r.ok) throw new Error(`getPr ${num} ${r.status}: ${await r.text()}`);
-  return snapshotFromPr((await r.json()) as any);
-}
-
-async function snapshotFromPr(row: any): Promise<PrSnapshot> {
-  const num = row.number;
-  const commitsResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/commits`);
-  const commits = commitsResp.ok ? ((await commitsResp.json()) as any[]) : [];
-  const filesResp = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}/files`);
-  const files = filesResp.ok ? ((await filesResp.json()) as any[]) : [];
-  return {
-    number: num,
-    head_sha: row.head?.sha ?? "",
-    base_sha: row.base?.sha ?? "",
-    title: row.title ?? "",
-    body: row.body ?? "",
-    state: row.state === "open" ? "open" : (row.merged ? "merged" : "closed"),
-    author: row.user?.login ?? "",
-    commits: commits.map(c => ({
-      sha: (c.sha ?? "").slice(0, 12),
-      message: c.commit?.message ?? "",
-      author: c.commit?.author?.name ?? "",
-    })),
-    files: files.map(f => ({
-      path: f.filename ?? "",
-      additions: f.additions ?? 0,
-      deletions: f.deletions ?? 0,
-    })),
-  };
-}
-
-/// Returns the unified diff text of the PR. Used by static checks.
-export async function getPrDiff(num: number): Promise<string> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${num}.diff`);
-  if (!r.ok) throw new Error(`getDiff ${num} ${r.status}: ${await r.text()}`);
-  return await r.text();
-}
-
-/// Hard-block mechanism: post a failing commit status on the PR head
-/// SHA. Branch protection (if enabled on `main`) treats this as a
-/// required-check fail and prevents merge. The description is shown
-/// in the Gitea UI next to the red X.
-export async function postCommitStatus(args: {
-  sha: string;
-  state: "success" | "pending" | "failure" | "error";
-  context: string;
-  description: string;
-  target_url?: string;
-}): Promise<void> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/statuses/${args.sha}`, {
-    method: "POST",
-    body: JSON.stringify({
-      state: args.state,
-      context: args.context,
-      description: args.description.slice(0, 140),
-      target_url: args.target_url ?? "",
-    }),
-  });
-  if (!r.ok) throw new Error(`postCommitStatus ${r.status}: ${await r.text()}`);
-}
-
-/// Post a review comment. Gitea typically blocks self-review
-/// (author posting a review on their own PR). Prefer
-/// `postIssueComment` when running with the author's PAT.
-export async function postReview(args: {
-  pr_number: number;
-  commit_id: string;
-  body: string;
-  event: "APPROVE" | "REQUEST_CHANGES" | "COMMENT";
-}): Promise<void> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/pulls/${args.pr_number}/reviews`, {
-    method: "POST",
-    body: JSON.stringify({
-      commit_id: args.commit_id,
-      body: args.body,
-      event: args.event,
-    }),
-  });
-  if (!r.ok) throw new Error(`postReview ${r.status}: ${await r.text()}`);
-}
-
-/// Plain issue comment. Works for the auditor's own PAT because
-/// Gitea allows authors to comment on their own PRs (just not
-/// review them). Auditor uses this for the reasoning body; the
-/// actual block signal is the commit status.
-export async function postIssueComment(args: {
-  pr_number: number;
-  body: string;
-}): Promise<{ id: number; html_url: string }> {
-  const r = await giteaFetch(`/repos/${OWNER}/${REPO}/issues/${args.pr_number}/comments`, {
-    method: "POST",
-    body: JSON.stringify({ body: args.body }),
-  });
-  if (!r.ok) throw new Error(`postIssueComment ${r.status}: ${await r.text()}`);
-  const j = await r.json() as any;
-  return { id: j.id, html_url: j.html_url };
-}

auditor/index.ts

@@ -1,147 +0,0 @@
-// Auditor poller — the top-level entry. Polls Gitea for open PRs on
-// a fixed interval, dedupes by head SHA, runs audit + posts verdict
-// for each new (pr, sha) pair.
-//
-// Run manually:
-//   bun run auditor/index.ts
-//
-// Stop:
-//   touch auditor.paused        (skips next cycle)
-//   pkill -f auditor/index.ts   (kills in-flight)
-//
-// State:
-//   data/_auditor/state.json           — last-audited SHA per PR
-//   data/_auditor/verdicts/{id}.json   — per-run verdict records
-//
-// This entry runs forever. A systemd unit would wrap it once the
-// workflow is trusted (same pattern as mcp-server, observer).
-
-import { readFile, writeFile, mkdir, access } from "node:fs/promises";
-import { listOpenPrs } from "./gitea.ts";
-import { auditPr } from "./audit.ts";
-
-const POLL_INTERVAL_MS = 90_000; // 90s — enough budget for audit runs to complete
-const PAUSE_FILE = "/home/profit/lakehouse/auditor.paused";
-const STATE_FILE = "/home/profit/lakehouse/data/_auditor/state.json";
-
-interface State {
-  // Map: PR number → last-audited head SHA. Lets us dedupe audits
-  // across restarts (poller can crash/restart without re-auditing
-  // all open PRs from scratch).
-  last_audited: Record<string, string>;
-  started_at: string;
-  cycles_total: number;
-  cycles_skipped_paused: number;
-  audits_run: number;
-  last_cycle_at?: string;
-}
-
-async function fileExists(path: string): Promise<boolean> {
-  try { await access(path); return true; } catch { return false; }
-}
-
-async function loadState(): Promise<State> {
-  try {
-    const raw = await readFile(STATE_FILE, "utf8");
-    const s = JSON.parse(raw);
-    return {
-      last_audited: s.last_audited ?? {},
-      started_at: s.started_at ?? new Date().toISOString(),
-      cycles_total: s.cycles_total ?? 0,
-      cycles_skipped_paused: s.cycles_skipped_paused ?? 0,
-      audits_run: s.audits_run ?? 0,
-      last_cycle_at: s.last_cycle_at,
-    };
-  } catch {
-    return {
-      last_audited: {},
-      started_at: new Date().toISOString(),
-      cycles_total: 0,
-      cycles_skipped_paused: 0,
-      audits_run: 0,
-    };
-  }
-}
-
-async function saveState(s: State): Promise<void> {
-  await mkdir("/home/profit/lakehouse/data/_auditor", { recursive: true });
-  await writeFile(STATE_FILE, JSON.stringify(s, null, 2));
-}
-
-async function runCycle(state: State): Promise<State> {
-  state.cycles_total += 1;
-  state.last_cycle_at = new Date().toISOString();
-
-  if (await fileExists(PAUSE_FILE)) {
-    state.cycles_skipped_paused += 1;
-    console.log(`[auditor] cycle ${state.cycles_total}: paused (touch ${PAUSE_FILE} exists)`);
-    return state;
-  }
-
-  let prs;
-  try {
-    prs = await listOpenPrs();
-  } catch (e) {
-    console.error(`[auditor] listOpenPrs failed: ${(e as Error).message}`);
-    return state;
-  }
-
-  console.log(`[auditor] cycle ${state.cycles_total}: ${prs.length} open PR(s)`);
-
-  for (const pr of prs) {
-    const last = state.last_audited[String(pr.number)];
-    if (last === pr.head_sha) {
-      console.log(`[auditor]   skip PR #${pr.number} (SHA ${pr.head_sha.slice(0, 8)} already audited)`);
-      continue;
-    }
-    console.log(`[auditor]   audit PR #${pr.number} (${pr.head_sha.slice(0, 8)}) — ${pr.title.slice(0, 60)}`);
-    try {
-      // Skip dynamic by default: it mutates live playbook state and
-      // re-runs on every PR update would pollute quickly. Operator
-      // can run dynamic via `bun run auditor/fixtures/cli.ts` manually
-      // OR set LH_AUDITOR_RUN_DYNAMIC=1 to opt in.
-      const run_dynamic = process.env.LH_AUDITOR_RUN_DYNAMIC === "1";
-      const verdict = await auditPr(pr, {
-        skip_dynamic: !run_dynamic,
-        skip_inference: process.env.LH_AUDITOR_SKIP_INFERENCE === "1",
-      });
-      console.log(`[auditor]     verdict=${verdict.overall} findings=${verdict.metrics.findings_total} (block=${verdict.metrics.findings_block} warn=${verdict.metrics.findings_warn})`);
-      state.last_audited[String(pr.number)] = pr.head_sha;
-      state.audits_run += 1;
-    } catch (e) {
-      console.error(`[auditor]     audit failed: ${(e as Error).message}`);
-    }
-  }
-
-  return state;
-}
-
-async function main(): Promise<void> {
-  console.log(`[auditor] starting poller — interval ${POLL_INTERVAL_MS / 1000}s`);
-  console.log(`[auditor] pause file: ${PAUSE_FILE}`);
-  console.log(`[auditor] state file: ${STATE_FILE}`);
-
-  let state = await loadState();
-  console.log(`[auditor] loaded state: ${Object.keys(state.last_audited).length} PRs previously audited, ${state.cycles_total} cycles so far`);
-
-  // Single-shot mode for CLI testing: `bun run auditor/index.ts --once`
-  const once = process.argv.includes("--once");
-  if (once) {
-    state = await runCycle(state);
-    await saveState(state);
-    console.log(`[auditor] single-shot complete. total audits: ${state.audits_run}`);
-    return;
-  }
-
-  // Loop.
-  while (true) {
-    state = await runCycle(state);
-    await saveState(state);
-    await new Promise(res => setTimeout(res, POLL_INTERVAL_MS));
-  }
-}
-
-main().catch(e => {
-  console.error("[auditor] fatal:", e);
-  process.exit(1);
-});

auditor/policy.ts

@@ -1,62 +0,0 @@
-// ═══════════════════════════════════════════════════════════════════
-// YOU WRITE THIS FILE. Policy decides what blocks vs what's a comment.
-// Defaults are opinionated on the "stop clicking past placeholder"
-// side — easier to loosen than to tighten when you're watching the
-// auditor behave in live PRs.
-// ═══════════════════════════════════════════════════════════════════
-
-import type { Finding, Verdict } from "./types.ts";
-
-/// Translate the four-check output into a single verdict. This is the
-/// single pane of glass the auditor operates on — tune thresholds here.
-export function assembleVerdict(
-  findings: Finding[],
-  metrics: Record<string, number>,
-  pr_number: number,
-  head_sha: string,
-): Verdict {
-  const blocking = findings.filter(f => f.severity === "block");
-  const warning = findings.filter(f => f.severity === "warn");
-
-  let overall: Verdict["overall"];
-  let one_liner: string;
-
-  if (blocking.length > 0) {
-    overall = "block";
-    one_liner = `${blocking.length} blocking issue${blocking.length > 1 ? "s" : ""}: ${blocking[0].summary}`;
-  } else if (warning.length >= 3) {
-    // Three or more warnings is a block — death by a thousand cuts.
-    overall = "request_changes";
-    one_liner = `${warning.length} warnings — see review`;
-  } else if (warning.length > 0) {
-    overall = "request_changes";
-    one_liner = warning[0].summary;
-  } else {
-    overall = "approve";
-    one_liner = `all checks passed (${findings.length} findings, all info)`;
-  }
-
-  return {
-    pr_number,
-    head_sha,
-    audited_at: new Date().toISOString(),
-    overall,
-    findings,
-    metrics,
-    one_liner,
-  };
-}
-
-/// Which strength-of-claim warrants which severity when evidence is
-/// weak? A "Phase X shipped" claim with zero integration tests is a
-/// blocker. A "should work" claim with no test is a warn.
-export function severityFromClaimEvidence(
-  claim_strength: "weak" | "moderate" | "strong",
-  evidence_grade: "none" | "partial" | "full",
-): "info" | "warn" | "block" {
-  if (evidence_grade === "full") return "info";
-  if (claim_strength === "strong" && evidence_grade === "none") return "block";
-  if (claim_strength === "strong" && evidence_grade === "partial") return "warn";
-  if (claim_strength === "moderate" && evidence_grade === "none") return "warn";
-  return "info";
-}

auditor/types.ts

@@ -1,65 +0,0 @@
-// Shared types for the claim-auditor. Every field exists for a reason;
-// if something can't be verified from a check, it goes into `evidence`
-// so the verdict is inspectable, not a black box.
-
-export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
-
-export type Severity = "info" | "warn" | "block";
-
-export interface Claim {
-  // Verbatim phrase that raised the claim — e.g. "Phase 38 shipped",
-  // "verified end-to-end", "works after restart". Used as the "what
-  // does the author assert" input to downstream checks.
-  text: string;
-  // Where it came from. `commit_sha` is the short hash; `location`
-  // is a file:line for in-diff claims, or "pr_body" / "commit_message".
-  commit_sha: string;
-  location: string;
-  // Heuristic rating of how strong the claim is. "green+tested"
-  // is strong; "should work" is weak. Drives sensitivity — stronger
-  // claims get harder-blocked on weak evidence.
-  strength: "weak" | "moderate" | "strong";
-}
-
-export interface Finding {
-  check: CheckKind;
-  severity: Severity;
-  claim_text?: string;
-  // Free-form short description: "field added but never read", "no
-  // test covers this code path", "cloud model says placeholder".
-  summary: string;
-  // Concrete evidence: file paths, line numbers, log excerpts, test
-  // output, cloud-model verdict. No handwaving.
-  evidence: string[];
-}
-
-export interface Verdict {
-  pr_number: number;
-  head_sha: string;
-  audited_at: string;
-  overall: "approve" | "request_changes" | "block";
-  findings: Finding[];
-  // Real numbers that downstream policy can gate on. e.g. if the
-  // hybrid test produced latency numbers or token counts, they
-  // surface here so /auditor/history is queryable.
-  metrics: Record<string, number>;
-  // Short one-line justification for the `overall` verdict. What
-  // gets posted as the commit-status description in Gitea (max 140
-  // chars) must fit here.
-  one_liner: string;
-}
-
-export interface PrSnapshot {
-  number: number;
-  head_sha: string;
-  base_sha: string;
-  title: string;
-  body: string;
-  state: "open" | "closed" | "merged";
-  author: string;
-  // Array of commit messages in the PR (not diffs — those are
-  // fetched on-demand per-check).
-  commits: Array<{ sha: string; message: string; author: string }>;
-  // File paths touched by the PR, with lines-added / lines-removed.
-  files: Array<{ path: string; additions: number; deletions: number }>;
-}

bot/README.md

@@ -1,127 +0,0 @@
-# Lakehouse PR-Bot
-
-A local sub-agent that reads the PRD, asks a cloud model for a small change
-proposal, applies it, runs tests, and opens a draft PR on Gitea. Manual-only
-right now — no systemd, no cron. Run one cycle at a time and watch.
-
-## Run one cycle
-
-```bash
-cd /home/profit/lakehouse
-bun run bot/cycle.ts
-```
-
-Prerequisites:
-- Working tree must be clean (`git status` shows nothing). The bot refuses on a dirty tree so its changes don't mix with in-flight work.
-- Sidecar running on :3200, gateway on :3100, observer on :3800.
-- At least one PRD line tagged `[bot-eligible]` (see below).
-- Gitea PAT configured in `~/.git-credentials` (already set up).
-
-## Tag a gap as bot-eligible
-
-Add `[bot-eligible]` to any PRD line the bot is allowed to work on:
-
-```md
-- [ ] Add a unit test for parse_city_state covering "South Bend, IN" edge case [bot-eligible]
-```
-
-The bot scans `docs/PRD.md` for these tags. Each tagged line becomes a candidate.
-Start small — one tag at a time — until you trust the loop.
-
-## Stop it mid-cycle
-
-Create the pause file:
-
-```bash
-touch /home/profit/lakehouse/bot.paused
-```
-
-The next `bot/cycle.ts` invocation exits immediately with `skipped_pause`.
-(It does NOT kill a cycle already in-flight — use `Ctrl-C` or `pkill -f bot/cycle.ts` for that.)
-
-## Budget
-
-- **20 cloud calls/day, 160k tokens/day** (hard ceiling, see `bot/cost.ts`).
-- Tracked in `data/_bot/cost-YYYY-MM-DD.json`.
-- Resets at UTC midnight.
-
-## Cycle outcomes
-
-Every run writes a result to `data/_bot/cycles/{cycle_id}.json` and POSTs an event to the observer on :3800.
-Possible outcomes:
-
-| Outcome | Meaning |
-|---|---|
-| `ok` | PR opened |
-| `cycle_noop` | proposal applied, every file was **identical to current content** (Mem0 NOOP); no test run, no PR |
-| `skipped_pause` | pause file present |
-| `skipped_cost` | daily budget exhausted |
-| `skipped_policy` | `policy.shouldRunCycle` said no |
-| `skipped_dirty_tree` | uncommitted changes present |
-| `skipped_no_gap` | no `[bot-eligible]` tags in PRD |
-| `model_failed` | sidecar/cloud model errored or returned unparseable JSON |
-| `proposal_rejected` | `policy.scoreProposal` rejected it (size, path, etc.) |
-| `apply_failed` | file write errored |
-| `tests_failed` | cargo or bun test red |
-| `pr_skipped_by_policy` | green tests, but `policy.shouldOpenPR` said no |
-| `pr_failed` | Gitea API or git push errored |
-
-## Mem0-aligned apply semantics
-
-`apply.ts` categorizes every proposed file into one of three modes:
-
-| Mode | Trigger | Action |
-|---|---|---|
-| **ADD** | `is_new: true` and file doesn't exist | Create the file |
-| **UPDATE** | `is_new: false`, file exists, content **differs** from current | Overwrite |
-| **NOOP** | `is_new: false`, file exists, content **matches** current exactly | Skip (no write, no diff) |
-
-If every file in the proposal is NOOP, the cycle short-circuits to `cycle_noop` **before** running tests or opening a PR. Mismatched shapes (`is_new:true` but file exists, or `is_new:false` but file missing) become `apply_failed` — model state confusion is surfaced, not papered over.
-
-PR bodies report the three counts separately so reviewers see what actually changed vs. what was confirmed identical.
-
-## How the bot compounds over cycles
-
-Every finished cycle persists a `CycleResult` to `data/_bot/cycles/{cycle_id}.json`. At the **start** of the next cycle, `bot/kb.ts::loadHistory(gap_id)` scans that directory, filters to prior cycles on the same gap, and returns the five most recent outcomes (`ok`, `tests_failed`, `proposal_rejected`, `cycle_noop`, etc.) with their reasons and touched files.
-
-Those outcomes are summarized into a compact block and **injected into the cloud prompt** before asking for a new proposal. The model sees things like:
-
-```
-Prior attempts on this gap (3 most recent):
-- 2026-04-22 03:15 UTC — tests_failed
-    reason: cargo test -p vectord::lance failed on field_type_coerce
-- 2026-04-22 02:48 UTC — proposal_rejected
-    reason: touched forbidden path (docs/ADR-): docs/ADR-019-update.md
-- 2026-04-22 01:23 UTC — ok PR: https://git.agentview.dev/profit/lakehouse/pulls/142
-    reason: PR #142 opened
-
-Learn from these: build on what worked, avoid paths that failed.
-```
-
-This is the same compounding pattern `scenario.ts` uses via `kb.loadRecommendation` — the bot's cycles are the bot's memory. No embedding, no separate jsonl, no cross-run orchestration required. First cycle on a new gap skips the block cleanly.
-
-The observer events POSTed on every cycle carry the same data, so `GET :3800/stats?source=bot` aggregates "% of bot cycles on similar gaps that landed a PR" without extra plumbing.
-
-## Where YOU edit
-
-`bot/policy.ts` — four small functions that define what the bot does. The rest
-(`propose.ts`, `apply.ts`, `test.ts`, `pr.ts`, `cycle.ts`, `kb.ts`) is mechanical
-orchestration — you shouldn't need to touch it unless a pipeline changes.
-
-One policy upgrade the history opens up: in `shouldRunCycle`, you can now bail
-out if the same gap has failed N times in a row. Example addition:
-
-```ts
-import { loadHistory, statsFor } from "./kb.ts";
-// inside shouldRunCycle — but you'd need the gap, which is picked later.
-// A more natural place is scoreProposal: if prior N failures on this
-// gap's path+summary pattern, reject before testing.
-```
-
-## Hard-coded guardrails (not in policy — can't be disabled)
-
-- Bot **never deletes files** (`apply.ts` has no delete path).
-- Bot **never touches** `.git/`, `secrets`, `lakehouse.toml`, `docs/ADR-*`, `docs/DECISIONS.md`, `docs/PRD.md`, `/etc/`, `/root/`, `Cargo.lock`.
-- All paths validated for traversal (`path/../`) and repo-escape.
-- PRs always open as **draft** — never auto-merge.
-- Budget check is before the policy function runs — no way to override the daily cap from `policy.ts`.

bot/apply.ts

@@ -1,86 +0,0 @@
-// Apply a proposal to disk with Mem0-aligned ADD / UPDATE / NOOP
-// semantics. Whole-file writes only — never deletes. All paths
-// validated repo-relative + path-traversal-free.
-//
-// Three outcomes per file:
-//   ADD    — file didn't exist, new content written (is_new must be true)
-//   UPDATE — file existed, content differs from current, overwritten
-//   NOOP   — file existed, content identical to current, nothing written
-//
-// The NOOP case is important: a cloud proposal that "re-derives" an
-// existing file shouldn't churn the repo or burn test cycles.
-
-import { writeFile, readFile, mkdir, access } from "node:fs/promises";
-import { dirname, resolve, relative } from "node:path";
-import type { ApplyOutcome, Proposal } from "./types.ts";
-
-const REPO_ROOT = "/home/profit/lakehouse";
-
-export async function applyProposal(p: Proposal): Promise<ApplyOutcome> {
-  const added: string[] = [];
-  const updated: string[] = [];
-  const noop: string[] = [];
-  const errors: string[] = [];
-
-  for (const f of p.files) {
-    const err = validatePath(f.path);
-    if (err) {
-      errors.push(`${f.path}: ${err}`);
-      continue;
-    }
-    const abs = resolve(REPO_ROOT, f.path);
-    try {
-      const exists = await fileExists(abs);
-
-      if (f.is_new && exists) {
-        errors.push(`${f.path}: is_new=true but file exists`);
-        continue;
-      }
-      if (!f.is_new && !exists) {
-        // Proposal claims to update a file that doesn't exist.
-        // Refuse rather than silently ADD — model is confused about
-        // repo state, and we want that surfaced, not papered over.
-        errors.push(`${f.path}: is_new=false but file does not exist`);
-        continue;
-      }
-
-      if (!f.is_new) {
-        // UPDATE candidate — compare to detect NOOP.
-        const current = await readFile(abs, "utf8");
-        if (current === f.content) {
-          noop.push(f.path);
-          continue;
-        }
-        await writeFile(abs, f.content);
-        updated.push(f.path);
-      } else {
-        // ADD — file doesn't exist yet.
-        await mkdir(dirname(abs), { recursive: true });
-        await writeFile(abs, f.content);
-        added.push(f.path);
-      }
-    } catch (e) {
-      errors.push(`${f.path}: ${(e as Error).message}`);
-    }
-  }
-  return { added, updated, noop, errors };
-}
-
-function validatePath(p: string): string | null {
-  if (!p) return "empty path";
-  if (p.startsWith("/")) return "must be repo-relative";
-  if (p.startsWith("..") || p.includes("/../")) return "path traversal";
-  const abs = resolve(REPO_ROOT, p);
-  const rel = relative(REPO_ROOT, abs);
-  if (rel.startsWith("..")) return "resolves outside repo";
-  return null;
-}
-
-async function fileExists(abs: string): Promise<boolean> {
-  try {
-    await access(abs);
-    return true;
-  } catch {
-    return false;
-  }
-}

bot/cost.ts

@@ -1,52 +0,0 @@
-// Daily cost tracker. Resets at UTC midnight by keying the file by date.
-// Hard ceilings are the policy surface — the bot refuses to call the
-// cloud once the budget is exhausted for the day.
-
-import { readFile, writeFile, mkdir } from "node:fs/promises";
-import { join } from "node:path";
-import type { CostState } from "./types.ts";
-
-const COST_DIR = "/home/profit/lakehouse/data/_bot";
-
-export const DAILY_CALLS_BUDGET = 20;
-export const DAILY_TOKENS_BUDGET = 8000 * DAILY_CALLS_BUDGET; // 160k tokens/day ceiling
-
-function todayUtc(): string {
-  return new Date().toISOString().slice(0, 10);
-}
-
-function costFile(date: string): string {
-  return join(COST_DIR, `cost-${date}.json`);
-}
-
-export async function readCost(): Promise<CostState> {
-  const date = todayUtc();
-  try {
-    const raw = await readFile(costFile(date), "utf8");
-    return JSON.parse(raw) as CostState;
-  } catch {
-    return { date, calls: 0, tokens: 0 };
-  }
-}
-
-export async function recordCost(calls: number, tokens: number): Promise<CostState> {
-  await mkdir(COST_DIR, { recursive: true });
-  const current = await readCost();
-  const updated: CostState = {
-    date: current.date,
-    calls: current.calls + calls,
-    tokens: current.tokens + tokens,
-  };
-  await writeFile(costFile(updated.date), JSON.stringify(updated, null, 2));
-  return updated;
-}
-
-export function budgetCheck(c: CostState): { ok: boolean; reason: string } {
-  if (c.calls >= DAILY_CALLS_BUDGET) {
-    return { ok: false, reason: `daily call budget exhausted (${c.calls}/${DAILY_CALLS_BUDGET})` };
-  }
-  if (c.tokens >= DAILY_TOKENS_BUDGET) {
-    return { ok: false, reason: `daily token budget exhausted (${c.tokens}/${DAILY_TOKENS_BUDGET})` };
-  }
-  return { ok: true, reason: `${c.calls}/${DAILY_CALLS_BUDGET} calls, ${c.tokens}/${DAILY_TOKENS_BUDGET} tokens used today` };
-}

bot/cycle.ts

@@ -1,345 +0,0 @@
-// Single-cycle orchestrator. This is the manual entry point:
-//   bun run bot/cycle.ts
-//
-// Steps (each step can short-circuit with a clear outcome):
-//   1. Pause-file check         → skipped_pause
-//   2. Cost budget check        → skipped_cost
-//   3. Gather CycleContext      (dirty tree, last cycle, autotune status)
-//   4. policy.shouldRunCycle    → skipped_policy
-//   5. findGaps + policy.pickGap → skipped_no_gap
-//   6. generateProposal         → model_failed
-//   7. policy.scoreProposal     → proposal_rejected
-//   8. git checkout new branch
-//   9. applyProposal            → apply_failed
-//  10. runTests                 → tests_failed
-//  11. policy.shouldOpenPR      → pr_skipped_by_policy
-//  12. git commit + push
-//  13. openPr                   → pr_failed / ok
-// 14. recordCost + postEvent (every outcome)
-// 15. write cycle result to data/_bot/cycles/{cycle_id}.json
-
-import { readFile, writeFile, mkdir, access } from "node:fs/promises";
-import { createHash, randomUUID } from "node:crypto";
-import { spawnSync } from "node:child_process";
-import { join } from "node:path";
-
-import type { CycleContext, CycleResult, CycleOutcome, Proposal } from "./types.ts";
-import * as policy from "./policy.ts";
-import { readCost, recordCost, budgetCheck, DAILY_CALLS_BUDGET, DAILY_TOKENS_BUDGET } from "./cost.ts";
-import { findGaps, generateProposal } from "./propose.ts";
-import { applyProposal } from "./apply.ts";
-import { runTests } from "./test.ts";
-import { openPr } from "./pr.ts";
-import { postEvent } from "./observer.ts";
-import { loadHistory, summarizeHistory, statsFor } from "./kb.ts";
-
-const REPO_ROOT = "/home/profit/lakehouse";
-const PAUSE_FILE = `${REPO_ROOT}/bot.paused`;
-const CYCLES_DIR = `${REPO_ROOT}/data/_bot/cycles`;
-const GATEWAY_URL = process.env.LH_GATEWAY_URL ?? "http://localhost:3100";
-
-function log(msg: string) { console.log(`[bot] ${msg}`); }
-
-async function fileExists(p: string): Promise<boolean> {
-  try { await access(p); return true; } catch { return false; }
-}
-
-function git(args: string[]): { code: number; stdout: string; stderr: string } {
-  const r = spawnSync("git", args, { cwd: REPO_ROOT, encoding: "utf8" });
-  return { code: r.status ?? -1, stdout: r.stdout ?? "", stderr: r.stderr ?? "" };
-}
-
-async function gatherContext(cost: { calls: number; tokens: number }): Promise<CycleContext> {
-  // Last cycle: newest file in CYCLES_DIR.
-  let lastCycleAt: string | null = null;
-  let lastCycleGapId: string | null = null;
-  try {
-    const { readdir, stat } = await import("node:fs/promises");
-    const entries = await readdir(CYCLES_DIR).catch(() => [] as string[]);
-    let newestTs = 0;
-    let newestPath: string | null = null;
-    for (const e of entries) {
-      if (!e.endsWith(".json")) continue;
-      const p = join(CYCLES_DIR, e);
-      const s = await stat(p);
-      if (s.mtimeMs > newestTs) { newestTs = s.mtimeMs; newestPath = p; }
-    }
-    if (newestPath) {
-      const r = JSON.parse(await readFile(newestPath, "utf8")) as CycleResult;
-      lastCycleAt = r.started_at;
-      lastCycleGapId = r.gap?.id ?? null;
-    }
-  } catch {}
-
-  // Autotune status — if the agent is busy, skip this cycle to avoid
-  // contention on shared Ollama GPU memory.
-  let autotuneBusy = false;
-  try {
-    const r = await fetch(`${GATEWAY_URL}/vectors/agent/status`, { signal: AbortSignal.timeout(2000) });
-    if (r.ok) {
-      const j = await r.json() as any;
-      autotuneBusy = Boolean(j.running) && (j.queue_depth ?? 0) > 0;
-    }
-  } catch {}
-
-  // Dirty tree check.
-  const status = git(["status", "--porcelain"]);
-  const workingTreeDirty = status.code === 0 && status.stdout.trim().length > 0;
-
-  return {
-    startedAt: new Date().toISOString(),
-    dailyCallsUsed: cost.calls,
-    dailyCallsBudget: DAILY_CALLS_BUDGET,
-    dailyTokensUsed: cost.tokens,
-    dailyTokensBudget: DAILY_TOKENS_BUDGET,
-    lastCycleAt,
-    lastCycleGapId,
-    autotuneBusy,
-    workingTreeDirty,
-  };
-}
-
-async function persistResult(res: CycleResult): Promise<void> {
-  await mkdir(CYCLES_DIR, { recursive: true });
-  await writeFile(join(CYCLES_DIR, `${res.cycle_id}.json`), JSON.stringify(res, null, 2));
-}
-
-function sigHash(gap_id: string | null, proposalSummary: string): string {
-  const h = createHash("sha256");
-  h.update(gap_id ?? "no-gap");
-  h.update("|");
-  h.update(proposalSummary);
-  return h.digest("hex").slice(0, 16);
-}
-
-function emit(res: CycleResult): Promise<void> {
-  return postEvent({
-    source: "bot",
-    cycle_id: res.cycle_id,
-    sig_hash: sigHash(res.gap?.id ?? null, res.proposal?.summary ?? ""),
-    event_kind: res.outcome,
-    ok: res.outcome === "ok",
-    staffer_id: res.proposal?.model_used,
-    turns: res.cloud_calls,
-    duration_ms: new Date(res.ended_at).getTime() - new Date(res.started_at).getTime(),
-    extra: {
-      reason: res.reason,
-      pr_url: res.prUrl,
-      files_added: res.filesAdded,
-      files_updated: res.filesUpdated,
-      files_noop: res.filesNoop,
-      tests_green: res.testsGreen,
-    },
-  });
-}
-
-async function main() {
-  const cycle_id = `bot-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomUUID().slice(0, 8)}`;
-  const started_at = new Date().toISOString();
-  log(`cycle ${cycle_id} starting`);
-
-  const finish = async (outcome: CycleOutcome, reason: string, extra: Partial<CycleResult> = {}): Promise<CycleResult> => {
-    const res: CycleResult = {
-      cycle_id,
-      started_at,
-      ended_at: new Date().toISOString(),
-      outcome,
-      reason,
-      gap: null,
-      proposal: null,
-      filesAdded: [],
-      filesUpdated: [],
-      filesNoop: [],
-      testsGreen: null,
-      testsOutput: "",
-      prUrl: null,
-      tokens_used: 0,
-      cloud_calls: 0,
-      ...extra,
-    };
-    await persistResult(res);
-    await emit(res);
-    log(`→ ${outcome}: ${reason}`);
-    return res;
-  };
-
-  // 1. Pause.
-  if (await fileExists(PAUSE_FILE)) {
-    return finish("skipped_pause", `pause file at ${PAUSE_FILE}`);
-  }
-
-  // 2. Cost.
-  const cost = await readCost();
-  const bc = budgetCheck(cost);
-  if (!bc.ok) return finish("skipped_cost", bc.reason);
-  log(`cost: ${bc.reason}`);
-
-  // 3+4. Context + policy.shouldRunCycle.
-  const ctx = await gatherContext(cost);
-  const srq = policy.shouldRunCycle(ctx);
-  if (!srq.run) {
-    return finish(ctx.workingTreeDirty ? "skipped_dirty_tree" : "skipped_policy", srq.reason);
-  }
-
-  // 5. Gaps + policy.pickGap.
-  const gaps = await findGaps();
-  log(`found ${gaps.length} [bot-eligible] gap(s) in PRD`);
-  const gapPick = policy.pickGap(gaps);
-  if (!gapPick.gap) return finish("skipped_no_gap", gapPick.reason);
-  const gap = gapPick.gap;
-  log(`picked gap ${gap.id}: "${gap.prd_line.slice(0, 80)}"`);
-
-  // 6. KB lookup + Proposal.
-  // Bot cycles compound: prior cycles on this gap inform the current
-  // proposal. Summary is a compact block injected into the cloud prompt
-  // so the model can build on past successes and steer around past
-  // failures. Empty string when this is the first cycle on this gap.
-  const history = await loadHistory(gap.id, 5);
-  const stats = statsFor(history);
-  if (history.length > 0) {
-    log(`kb: ${stats.attempts} prior cycle(s) — ${stats.pr_opened} pr / ${stats.tests_failed} tests_failed / ${stats.proposal_rejected} rejected / ${stats.noop} noop`);
-  } else {
-    log(`kb: first cycle on this gap`);
-  }
-  const historySummary = summarizeHistory(history);
-
-  let proposal: Proposal;
-  try {
-    proposal = await generateProposal(gap, historySummary);
-    log(`proposal: ${proposal.summary} (${proposal.files.length} files, ~${proposal.estimated_loc} LOC, ${proposal.tokens_used} tokens)`);
-  } catch (e) {
-    await recordCost(1, 0);
-    return finish("model_failed", (e as Error).message, { gap, cloud_calls: 1 });
-  }
-  await recordCost(1, proposal.tokens_used);
-
-  // 7. Score.
-  const score = policy.scoreProposal(proposal);
-  if (!score.accept) {
-    return finish("proposal_rejected", score.reason, { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
-  }
-  log(`proposal accepted: ${score.reason}`);
-
-  // 8. Branch.
-  const dayStamp = new Date().toISOString().slice(0, 10).replace(/-/g, "");
-  const branch = `bot/cycle-${dayStamp}-${gap.id}`;
-  // Branch must be fresh. If one already exists from a prior failed cycle
-  // for the same gap, bail — don't silently overwrite or reuse stale state.
-  const check = git(["rev-parse", "--verify", `refs/heads/${branch}`]);
-  if (check.code === 0) {
-    return finish("apply_failed",
-      `branch ${branch} already exists — delete it manually if you want a fresh cycle for this gap`,
-      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
-  }
-  const co = git(["checkout", "-b", branch]);
-  if (co.code !== 0) {
-    return finish("apply_failed", `git checkout -b failed: ${co.stderr}`,
-      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
-  }
-  log(`on branch ${branch}`);
-
-  // 9. Apply — Mem0 ADD/UPDATE/NOOP semantics.
-  const ap = await applyProposal(proposal);
-  if (ap.errors.length > 0) {
-    git(["checkout", "main"]);
-    git(["branch", "-D", branch]);
-    return finish("apply_failed", ap.errors.join("; "),
-      { gap, proposal, tokens_used: proposal.tokens_used, cloud_calls: 1 });
-  }
-  const changed = [...ap.added, ...ap.updated];
-  log(`apply: ${ap.added.length} add, ${ap.updated.length} update, ${ap.noop.length} noop`);
-
-  // Short-circuit: if every proposed file was identical to what's on
-  // disk, there's nothing to test or PR. Clean up the branch and exit.
-  if (changed.length === 0) {
-    git(["checkout", "main"]);
-    git(["branch", "-D", branch]);
-    return finish("cycle_noop",
-      `all ${proposal.files.length} proposed file(s) were identical to current state`,
-      { gap, proposal, filesNoop: ap.noop,
-        tokens_used: proposal.tokens_used, cloud_calls: 1 });
-  }
-
-  // 10. Test.
-  log(`running tests (this will take a while)...`);
-  const testRes = await runTests();
-  if (!testRes.green) {
-    git(["checkout", "."]);
-    git(["checkout", "main"]);
-    git(["branch", "-D", branch]);
-    return finish("tests_failed", "cargo or bun test failed", {
-      gap, proposal,
-      filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
-      testsGreen: false, testsOutput: testRes.output,
-      tokens_used: proposal.tokens_used, cloud_calls: 1,
-    });
-  }
-  log(`tests green`);
-
-  // 11. PR gate.
-  const partial: CycleResult = {
-    cycle_id, started_at,
-    ended_at: new Date().toISOString(),
-    outcome: "ok", reason: "",
-    gap, proposal,
-    filesAdded: ap.added, filesUpdated: ap.updated, filesNoop: ap.noop,
-    testsGreen: true, testsOutput: testRes.output,
-    prUrl: null, tokens_used: proposal.tokens_used, cloud_calls: 1,
-  };
-  const prGate = policy.shouldOpenPR(partial);
-  if (!prGate.open) {
-    git(["checkout", "main"]);
-    git(["branch", "-D", branch]);
-    return finish("pr_skipped_by_policy", prGate.reason, partial);
-  }
-
-  // 12. Commit + push.
-  git(["add", ...changed]);
-  const commitMsg = [
-    `bot: ${proposal.summary}`,
-    "",
-    proposal.rationale,
-    "",
-    `PRD gap: ${gap.prd_line}`,
-    `Cycle: ${cycle_id}`,
-    "",
-    `Co-Authored-By: ${proposal.model_used} (cloud) <bot@lakehouse.local>`,
-  ].join("\n");
-  const ci = git(["commit", "-m", commitMsg]);
-  if (ci.code !== 0) {
-    return finish("pr_failed", `git commit failed: ${ci.stderr}`, partial);
-  }
-  const ps = git(["push", "-u", "origin", branch]);
-  if (ps.code !== 0) {
-    return finish("pr_failed", `git push failed: ${ps.stderr}`, partial);
-  }
-
-  // 13. Open PR.
-  try {
-    const pr = await openPr({
-      branch,
-      title: `bot: ${proposal.summary}`,
-      body: [
-        `**Gap** (PRD line ${gap.line_number}): ${gap.prd_line}`,
-        "",
-        `**Rationale**`,
-        proposal.rationale,
-        "",
-        `**Cycle**: \`${cycle_id}\``,
-        `**Model**: \`${proposal.model_used}\``,
-        `**Tokens**: ${proposal.tokens_used}`,
-        `**Added**: ${ap.added.map(f => `\`${f}\``).join(", ") || "_none_"}`,
-        `**Updated**: ${ap.updated.map(f => `\`${f}\``).join(", ") || "_none_"}`,
-        ap.noop.length > 0 ? `**NOOP (identical to current)**: ${ap.noop.map(f => `\`${f}\``).join(", ")}` : "",
-      ].join("\n"),
-    });
-    return finish("ok", `PR #${pr.number} opened`, { ...partial, prUrl: pr.html_url });
-  } catch (e) {
-    return finish("pr_failed", (e as Error).message, partial);
-  }
-}
-
-main().catch(e => {
-  console.error("[bot] fatal:", e);
-  process.exit(1);
-});

bot/kb.ts

@@ -1,101 +0,0 @@
-// Bot-local knowledge base. Every finished cycle already persists a
-// CycleResult to data/_bot/cycles/{id}.json — that IS the outcome log.
-// KB here just reads that dir, filters to prior cycles on the same gap,
-// and produces a short summary the cloud model can condition on.
-//
-// No separate jsonl, no new write path, no embedding calls. The bot's
-// "memory" is the same primary artifact that the observer consumes.
-//
-// Future: embedding-based neighbor matching across gaps (cheap once
-// sidecar is local), cross-pollination with scenario KB's
-// pathway_recommendations. Not required for the feedback loop to work
-// on a single gap — that's the floor we're building first.
-
-import { readdir, readFile } from "node:fs/promises";
-import { join } from "node:path";
-import type { CycleResult } from "./types.ts";
-
-const CYCLES_DIR = "/home/profit/lakehouse/data/_bot/cycles";
-
-export interface HistoryEntry {
-  cycle_id: string;
-  ended_at: string;
-  outcome: string;
-  reason: string;
-  pr_url: string | null;
-  tests_green: boolean | null;
-  files_added: string[];
-  files_updated: string[];
-  tokens_used: number;
-}
-
-export async function loadHistory(gap_id: string, max: number = 5): Promise<HistoryEntry[]> {
-  let entries: string[] = [];
-  try {
-    entries = await readdir(CYCLES_DIR);
-  } catch {
-    return [];
-  }
-  const matches: HistoryEntry[] = [];
-  for (const e of entries) {
-    if (!e.endsWith(".json")) continue;
-    try {
-      const raw = await readFile(join(CYCLES_DIR, e), "utf8");
-      const r = JSON.parse(raw) as CycleResult;
-      if (r.gap?.id !== gap_id) continue;
-      matches.push({
-        cycle_id: r.cycle_id,
-        ended_at: r.ended_at,
-        outcome: r.outcome,
-        reason: r.reason,
-        pr_url: r.prUrl,
-        tests_green: r.testsGreen,
-        files_added: r.filesAdded ?? [],
-        files_updated: r.filesUpdated ?? [],
-        tokens_used: r.tokens_used,
-      });
-    } catch {
-      // Skip unreadable / malformed cycle files. Don't fail the current
-      // cycle because an old one is corrupt.
-    }
-  }
-  matches.sort((a, b) => b.ended_at.localeCompare(a.ended_at));
-  return matches.slice(0, max);
-}
-
-// Compact prompt-ready summary. Empty string when there's no history —
-// caller can skip the "prior attempts" block entirely.
-export function summarizeHistory(h: HistoryEntry[]): string {
-  if (h.length === 0) return "";
-  const lines = h.map(e => {
-    const when = e.ended_at.slice(0, 16).replace("T", " ");
-    const files = [...e.files_added, ...e.files_updated];
-    const filesStr = files.length > 0 ? ` touched: ${files.join(", ")}` : "";
-    const prStr = e.pr_url ? ` PR: ${e.pr_url}` : "";
-    return `- ${when} UTC — ${e.outcome}${prStr}${filesStr}\n    reason: ${e.reason}`;
-  });
-  return [
-    `Prior attempts on this gap (${h.length} most recent):`,
-    ...lines,
-    "",
-    "Learn from these: build on what worked, avoid paths that failed.",
-  ].join("\n");
-}
-
-// Aggregate stats for telemetry — lets the bot expose "% of cycles on
-// this gap that landed a PR" without re-parsing the raw history.
-export function statsFor(h: HistoryEntry[]): {
-  attempts: number;
-  pr_opened: number;
-  tests_failed: number;
-  proposal_rejected: number;
-  noop: number;
-} {
-  return {
-    attempts: h.length,
-    pr_opened: h.filter(e => e.outcome === "ok").length,
-    tests_failed: h.filter(e => e.outcome === "tests_failed").length,
-    proposal_rejected: h.filter(e => e.outcome === "proposal_rejected").length,
-    noop: h.filter(e => e.outcome === "cycle_noop").length,
-  };
-}

bot/observer.ts

@@ -1,34 +0,0 @@
-// POST bot cycle outcomes to the observer on :3800 so they accumulate
-// in the KB alongside scenario + autotune events. Non-fatal on failure:
-// the cycle result is also written to disk so observability isn't a
-// single point of failure.
-
-const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
-
-export interface BotObserverEvent {
-  source: "bot";
-  cycle_id: string;
-  sig_hash: string;          // stable hash of the gap + proposal for dedup
-  event_kind: string;        // cycle outcome (ok, tests_failed, ...)
-  ok: boolean;
-  staffer_id?: string;       // the model used, e.g. "gpt-oss:120b"
-  turns?: number;            // number of cloud calls this cycle
-  duration_ms?: number;
-  extra?: Record<string, any>;
-}
-
-export async function postEvent(ev: BotObserverEvent): Promise<void> {
-  try {
-    const r = await fetch(`${OBSERVER_URL}/event`, {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify(ev),
-      signal: AbortSignal.timeout(3000),
-    });
-    if (!r.ok) {
-      console.error(`[bot/observer] ${r.status}: ${await r.text()}`);
-    }
-  } catch (e) {
-    console.error(`[bot/observer] POST failed: ${(e as Error).message}`);
-  }
-}

bot/policy.ts

@@ -1,70 +0,0 @@
-// ═══════════════════════════════════════════════════════════════════
-// YOU WRITE THIS FILE. These four functions shape the bot's entire
-// behavior. Defaults below are safe and intentionally conservative —
-// redline to taste. Each function is ~5-10 lines.
-// ═══════════════════════════════════════════════════════════════════
-
-import type { CycleContext, CycleResult, Gap, Proposal } from "./types.ts";
-
-// Should this cycle even fire right now?
-// Called AFTER the pause-file check + cost-cap check have already passed.
-// Use this for judgment calls (cooldown, shared-resource conflicts).
-export function shouldRunCycle(ctx: CycleContext): { run: boolean; reason: string } {
-  if (ctx.workingTreeDirty) {
-    return { run: false, reason: "dirty working tree — commit or stash your work first" };
-  }
-  if (ctx.lastCycleAt) {
-    const minsAgo = (Date.now() - new Date(ctx.lastCycleAt).getTime()) / 60000;
-    if (minsAgo < 10) return { run: false, reason: `last cycle ${minsAgo.toFixed(1)}min ago (<10min cooldown)` };
-  }
-  if (ctx.autotuneBusy) {
-    return { run: false, reason: "autotune agent is running — avoid Ollama contention" };
-  }
-  return { run: true, reason: "ready" };
-}
-
-// Which [bot-eligible] gap from the PRD does this cycle attack?
-// Default: first in document order. Alternatives: random for exploration,
-// smallest-context, tag-priority (e.g. [bot-eligible:high] wins).
-export function pickGap(gaps: Gap[]): { gap: Gap | null; reason: string } {
-  if (gaps.length === 0) return { gap: null, reason: "no [bot-eligible] gaps in PRD" };
-  return { gap: gaps[0], reason: "first eligible gap in document order" };
-}
-
-// Cheap gate BEFORE burning test cycles. Reject if proposal smells wrong.
-// This catches runaway cloud proposals before we apply+test them.
-export function scoreProposal(p: Proposal): { accept: boolean; reason: string } {
-  if (p.files.length === 0) return { accept: false, reason: "empty proposal" };
-  if (p.files.length > 5) return { accept: false, reason: `>5 files (${p.files.length})` };
-  if (p.estimated_loc > 200) return { accept: false, reason: `>200 LOC (${p.estimated_loc})` };
-
-  // Paths the bot is NEVER allowed to touch. Extend cautiously.
-  const forbidden = [
-    ".git/", "secrets", "lakehouse.toml",
-    "docs/ADR-", "docs/DECISIONS.md", "docs/PRD.md",
-    "/etc/", "/root/", "Cargo.lock",
-  ];
-  for (const f of p.files) {
-    const hit = forbidden.find(prefix => f.path.includes(prefix));
-    if (hit) return { accept: false, reason: `touches forbidden path (${hit}): ${f.path}` };
-  }
-  return { accept: true, reason: "passes size + path filters" };
-}
-
-// Tests are green and changes applied. Do we actually open a PR?
-// Mem0-aligned: the cycle already short-circuited with `cycle_noop` if
-// every file was identical to current state, so by the time we're here
-// at least one file was actually added or updated.
-//
-// Stricter options below — uncomment to require a test file touched, etc.
-export function shouldOpenPR(c: CycleResult): { open: boolean; reason: string } {
-  if (!c.testsGreen) return { open: false, reason: "tests red" };
-  const realChanges = c.filesAdded.length + c.filesUpdated.length;
-  if (realChanges === 0) return { open: false, reason: "nothing actually changed on disk" };
-
-  // Stricter option — require at least one test file in the diff:
-  // const hasTest = [...c.filesAdded, ...c.filesUpdated].some(f => /test|_tests/i.test(f));
-  // if (!hasTest) return { open: false, reason: "no test file touched" };
-
-  return { open: true, reason: `green + ${c.filesAdded.length} add / ${c.filesUpdated.length} update` };
-}

bot/pr.ts

@@ -1,58 +0,0 @@
-// Gitea PR open. Reads PAT from ~/.git-credentials (set up by the
-// credential-helper flow). All PRs open as DRAFT so J reviews before
-// merge — never auto-merge.
-
-import { readFile } from "node:fs/promises";
-
-const GITEA_HOST = "https://git.agentview.dev";
-const REPO_OWNER = "profit";
-const REPO_NAME = "lakehouse";
-const CRED_FILE = "/home/profit/.git-credentials";
-
-async function readPat(): Promise<string> {
-  const raw = await readFile(CRED_FILE, "utf8");
-  for (const line of raw.split("\n")) {
-    const m = line.match(/^https:\/\/[^:]+:([^@]+)@git\.agentview\.dev/);
-    if (m) return m[1];
-  }
-  throw new Error(`no Gitea PAT found in ${CRED_FILE}`);
-}
-
-export interface OpenPrInput {
-  branch: string;    // head branch (just pushed)
-  base?: string;     // default "main"
-  title: string;
-  body: string;
-}
-
-export interface OpenPrResult {
-  number: number;
-  html_url: string;
-}
-
-export async function openPr(input: OpenPrInput): Promise<OpenPrResult> {
-  const pat = await readPat();
-  const url = `${GITEA_HOST}/api/v1/repos/${REPO_OWNER}/${REPO_NAME}/pulls`;
-  const payload = {
-    title: input.title,
-    body: input.body,
-    head: input.branch,
-    base: input.base ?? "main",
-    // Gitea uses "draft" flag — always on for bot PRs.
-    draft: true,
-  };
-  const r = await fetch(url, {
-    method: "POST",
-    headers: {
-      "content-type": "application/json",
-      "Authorization": `token ${pat}`,
-    },
-    body: JSON.stringify(payload),
-    signal: AbortSignal.timeout(30000),
-  });
-  if (!r.ok) {
-    throw new Error(`Gitea ${r.status}: ${await r.text()}`);
-  }
-  const j = await r.json() as any;
-  return { number: j.number, html_url: j.html_url };
-}

bot/propose.ts

@@ -1,141 +0,0 @@
-// Gap detection + cloud proposal.
-//
-// Gap detection: scan docs/PRD.md for lines tagged [bot-eligible].
-// Each match becomes a Gap with surrounding context.
-//
-// Proposal: one-shot call to the T3 cloud model via the Python sidecar's
-// /generate endpoint. Asks for a structured JSON response with file
-// contents. Truncation-resistant via Phase 21's generate_continuable —
-// for now we pass max_tokens high and rely on the model completing in
-// one pass; swap to the Rust continuation wrapper if we see truncation.
-
-import { readFile } from "node:fs/promises";
-import { createHash } from "node:crypto";
-import type { Gap, Proposal } from "./types.ts";
-
-const SIDECAR_URL = process.env.LH_SIDECAR_URL ?? "http://localhost:3200";
-const REPO_ROOT = "/home/profit/lakehouse";
-const PRD_PATH = `${REPO_ROOT}/docs/PRD.md`;
-const CLOUD_MODEL = process.env.LH_BOT_MODEL ?? "gpt-oss:120b";
-const MAX_TOKENS = 6000;
-
-export async function findGaps(): Promise<Gap[]> {
-  const prd = await readFile(PRD_PATH, "utf8");
-  const lines = prd.split("\n");
-  const gaps: Gap[] = [];
-  for (let i = 0; i < lines.length; i++) {
-    if (!lines[i].includes("[bot-eligible]")) continue;
-    const contextLines = lines.slice(i, Math.min(i + 6, lines.length)).join("\n");
-    const id = createHash("sha256").update(lines[i]).digest("hex").slice(0, 12);
-    gaps.push({
-      id,
-      prd_line: lines[i].trim(),
-      context: contextLines,
-      source_file: "docs/PRD.md",
-      line_number: i + 1,
-    });
-  }
-  return gaps;
-}
-
-const SYSTEM_PROMPT = `You are an assistant that proposes small, testable code changes to the Lakehouse repo.
-The Lakehouse is a Rust-first data platform with 13 crates + Bun/TypeScript test harness.
-You will be given one PRD gap tagged [bot-eligible] and must respond with a STRICT JSON object — no prose.
-
-Rules:
-- Response MUST be a single JSON object, no markdown fences, no commentary.
-- Change MUST be small: <200 lines total, ≤5 files.
-- Include at least one test file (new or modified) that proves the change.
-- NEVER touch .git/, secrets, lakehouse.toml, docs/ADR-*, docs/DECISIONS.md, docs/PRD.md, /etc/, /root/, Cargo.lock.
-- Paths MUST be repo-relative (no leading /).
-- Whole-file contents only — no patches, no diffs.
-
-Response shape:
-{
-  "summary": "one line",
-  "rationale": "why this addresses the gap",
-  "files": [ { "path": "crates/foo/src/bar.rs", "content": "<full file>", "is_new": false } ],
-  "estimated_loc": 42
-}`;
-
-export async function generateProposal(gap: Gap, historySummary: string = ""): Promise<Proposal> {
-  const sections = [
-    `PRD gap (line ${gap.line_number}):`,
-    "```",
-    gap.context,
-    "```",
-    "",
-  ];
-  if (historySummary) {
-    sections.push(historySummary, "");
-  }
-  sections.push("Propose a small change that addresses this gap. Respond with the JSON object only.");
-  const userPrompt = sections.join("\n");
-
-  const r = await fetch(`${SIDECAR_URL}/generate`, {
-    method: "POST",
-    headers: { "content-type": "application/json" },
-    body: JSON.stringify({
-      model: CLOUD_MODEL,
-      system: SYSTEM_PROMPT,
-      prompt: userPrompt,
-      temperature: 0.2,
-      max_tokens: MAX_TOKENS,
-      think: false,
-    }),
-    signal: AbortSignal.timeout(180000), // cloud T3 can be slow — 3 min
-  });
-  if (!r.ok) {
-    throw new Error(`sidecar ${r.status}: ${await r.text()}`);
-  }
-  const j = await r.json() as any;
-  const raw: string = j.text ?? j.response ?? "";
-  const usage = j.usage ?? {};
-  const tokens = (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0);
-
-  const parsed = extractJson(raw);
-  if (!parsed || typeof parsed !== "object") {
-    throw new Error(`model returned no JSON object. Raw head: ${raw.slice(0, 300)}`);
-  }
-  if (!Array.isArray(parsed.files)) {
-    throw new Error(`proposal.files not an array: ${JSON.stringify(parsed).slice(0, 200)}`);
-  }
-
-  return {
-    summary: String(parsed.summary ?? "").trim(),
-    rationale: String(parsed.rationale ?? "").trim(),
-    files: parsed.files.map((f: any) => ({
-      path: String(f.path ?? ""),
-      content: String(f.content ?? ""),
-      is_new: Boolean(f.is_new),
-    })).filter((f: any) => f.path && f.content !== undefined),
-    estimated_loc: Number(parsed.estimated_loc ?? 0),
-    model_used: CLOUD_MODEL,
-    tokens_used: tokens,
-  };
-}
-
-// Find the first balanced JSON object in the string. Tolerates leading
-// "```json" fences even though we asked the model not to emit them.
-function extractJson(text: string): any | null {
-  const cleaned = text.replace(/^```(?:json)?\s*/i, "").replace(/```\s*$/i, "").trim();
-  let depth = 0;
-  let start = -1;
-  for (let i = 0; i < cleaned.length; i++) {
-    const c = cleaned[i];
-    if (c === "{") {
-      if (depth === 0) start = i;
-      depth++;
-    } else if (c === "}") {
-      depth--;
-      if (depth === 0 && start >= 0) {
-        try {
-          return JSON.parse(cleaned.slice(start, i + 1));
-        } catch {
-          start = -1;
-        }
-      }
-    }
-  }
-  return null;
-}

bot/test.ts

@@ -1,46 +0,0 @@
-// Run the test gate: `cargo test --workspace` + `bun test`.
-// Returns { green, output } — output is last ~4KB of combined stdout/stderr.
-
-import { spawn } from "node:child_process";
-
-const REPO_ROOT = "/home/profit/lakehouse";
-const TEST_TIMEOUT_MS = 15 * 60 * 1000; // 15 min cargo + 2 min bun, generous
-
-export async function runTests(): Promise<{ green: boolean; output: string }> {
-  const cargoOut = await runCmd("cargo", ["test", "--workspace", "--quiet", "--", "--test-threads=1"], REPO_ROOT, TEST_TIMEOUT_MS);
-  if (cargoOut.code !== 0) {
-    return { green: false, output: tail(`cargo test (${cargoOut.code}):\n${cargoOut.combined}`) };
-  }
-
-  const bunOut = await runCmd("bun", ["test", "tests/multi-agent"], REPO_ROOT, 120000);
-  const bunGreen = bunOut.code === 0;
-  const combined = [
-    `cargo test: OK`,
-    `bun test (${bunOut.code}):`,
-    bunOut.combined,
-  ].join("\n");
-  return { green: bunGreen, output: tail(combined) };
-}
-
-function tail(s: string, n = 4096): string {
-  return s.length > n ? "…" + s.slice(-n) : s;
-}
-
-function runCmd(
-  cmd: string,
-  args: string[],
-  cwd: string,
-  timeoutMs: number,
-): Promise<{ code: number; combined: string }> {
-  return new Promise(resolveP => {
-    const child = spawn(cmd, args, { cwd, env: { ...process.env } });
-    let combined = "";
-    child.stdout.on("data", d => { combined += d.toString(); });
-    child.stderr.on("data", d => { combined += d.toString(); });
-    const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
-    child.on("close", code => {
-      clearTimeout(timer);
-      resolveP({ code: code ?? -1, combined });
-    });
-  });
-}

bot/types.ts

@@ -1,88 +0,0 @@
-// Shared types for the PR-bot. Small and explicit — the bot's behavior
-// should be readable from these shapes alone.
-
-export interface Gap {
-  id: string;              // hash of prd_line — stable key for dedup
-  prd_line: string;        // the [bot-eligible]-tagged line, verbatim
-  context: string;         // surrounding context from the PRD (next ~5 lines)
-  source_file: string;     // e.g. "docs/PRD.md"
-  line_number: number;
-}
-
-export interface ProposalFile {
-  path: string;            // repo-relative (no leading slash)
-  content: string;         // full file content — the bot writes whole files, not patches
-  is_new: boolean;         // true = create, false = overwrite existing
-}
-
-export interface Proposal {
-  summary: string;         // one-line
-  rationale: string;       // why this change addresses the gap
-  files: ProposalFile[];
-  estimated_loc: number;   // total added+changed lines across all files
-  model_used: string;
-  tokens_used: number;
-}
-
-export interface CycleContext {
-  startedAt: string;       // ISO
-  dailyCallsUsed: number;
-  dailyCallsBudget: number;
-  dailyTokensUsed: number;
-  dailyTokensBudget: number;
-  lastCycleAt: string | null;
-  lastCycleGapId: string | null;
-  autotuneBusy: boolean;
-  workingTreeDirty: boolean;
-}
-
-export type CycleOutcome =
-  | "ok"                    // PR opened
-  | "skipped_pause"
-  | "skipped_cost"
-  | "skipped_policy"
-  | "skipped_no_gap"
-  | "skipped_dirty_tree"
-  | "proposal_rejected"
-  | "apply_failed"
-  | "cycle_noop"            // proposal applied but every file was identical to what's on disk
-  | "tests_failed"
-  | "pr_skipped_by_policy"
-  | "pr_failed"
-  | "model_failed";
-
-// Mem0-aligned apply outcomes. Three shapes instead of the binary
-// "written" / "errored". NOOP is a first-class outcome — identical
-// content shouldn't waste test cycles or open an empty PR.
-export type ApplyMode = "add" | "update" | "noop";
-
-export interface ApplyOutcome {
-  added: string[];
-  updated: string[];
-  noop: string[];
-  errors: string[];
-}
-
-export interface CycleResult {
-  cycle_id: string;
-  started_at: string;
-  ended_at: string;
-  outcome: CycleOutcome;
-  reason: string;
-  gap: Gap | null;
-  proposal: Proposal | null;
-  filesAdded: string[];
-  filesUpdated: string[];
-  filesNoop: string[];
-  testsGreen: boolean | null;     // null = not run
-  testsOutput: string;
-  prUrl: string | null;
-  tokens_used: number;
-  cloud_calls: number;
-}
-
-export interface CostState {
-  date: string;          // YYYY-MM-DD UTC
-  calls: number;
-  tokens: number;
-}

bun.lock

@@ -1,18 +0,0 @@
-{
-  "lockfileVersion": 1,
-  "configVersion": 1,
-  "workspaces": {
-    "": {
-      "dependencies": {
-        "langfuse": "^3.38.20",
-      },
-    },
-  },
-  "packages": {
-    "langfuse": ["langfuse@3.38.20", "", { "dependencies": { "langfuse-core": "^3.38.20" } }, "sha512-MAmBAASSzJtmK1O9HQegA1mFsQhT8Yf+OJRGvE7FXkyv3g/eiBE0glLD0Ohg3pkxhoPdggM5SejK7ue9ctlaMA=="],
-
-    "langfuse-core": ["langfuse-core@3.38.20", "", { "dependencies": { "mustache": "^4.2.0" } }, "sha512-zBKVmQN/1oT5VWZUBYlWzvokIlkC/6mnpgr/2atMyTeAm+jR3ia7w2iJMjlrF5/oG8ukO1s8+LDRCzJpF1QeEA=="],
-
-    "mustache": ["mustache@4.2.0", "", { "bin": { "mustache": "bin/mustache" } }, "sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ=="],
-  }
-}

config/models.json

@@ -1,193 +0,0 @@
-{
-  "_description": "Lakehouse model matrix — authoritative routing for all agent tiers. Local models do the heavy lifting; cloud models are consulted sparingly for overview, strategic, and gatekeeper decisions. Read by tests/multi-agent/scenario.ts and mcp-server/index.ts.",
-  "version": 1,
-  "updated": "2026-04-21",
-  "providers": {
-    "ollama_local": {
-      "base_url": "http://localhost:11434",
-      "key_env": null
-    },
-    "ollama_cloud": {
-      "base_url": "https://ollama.com",
-      "key_env": "OLLAMA_CLOUD_KEY",
-      "key_source": "/root/llm_team_config.json → providers.ollama_cloud.api_key",
-      "rate_budget": {
-        "calls_per_hour": 200,
-        "calls_per_day": 2000,
-        "notes": "Paid tier — generous. Policy: keep overview calls ≤ 3/scenario, strategic ≤ 10/day, gatekeeper ≤ 5/day."
-      }
-    }
-  },
-  "tiers": {
-    "t1_hot": {
-      "purpose": "Per tool call — SQL generation, hybrid_search, sql(). Runs 50-200 times per scenario. Latency-sensitive.",
-      "kind": "local_fast",
-      "primary": { "model": "qwen3.5:latest", "provider": "ollama_local", "context_window": 262144 },
-      "fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
-      "max_tokens": 1000,
-      "temperature": 0.3,
-      "never_route_cloud": true,
-      "context_budget": {
-        "system_prompt_cap": 4000,
-        "prior_context_cap": 6000,
-        "tool_results_cap": 8000,
-        "safety_margin": 2000,
-        "overflow_policy": "summarize_oldest_tool_results_via_t3"
-      },
-      "rationale": "qwen3.5:latest is a 9.7B thinking model with 262K context that emits clean JSON (verified 2026-04-21 after mistral A/B wipeout). Thinking budget requires max_tokens ≥ 400; we hold 1000 to keep propose_done payloads intact. qwen2.5 stays as fallback."
-    },
-    "t2_review": {
-      "purpose": "Per step consensus — executor ↔ reviewer loop critique. 5-14 calls per event.",
-      "kind": "local_balanced",
-      "primary": { "model": "qwen3:latest", "provider": "ollama_local", "context_window": 40960 },
-      "fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
-      "max_tokens": 800,
-      "temperature": 0.3,
-      "never_route_cloud": true,
-      "context_budget": {
-        "system_prompt_cap": 2000,
-        "recent_turns_cap": 4000,
-        "safety_margin": 1000
-      },
-      "rationale": "Reviewer only needs to detect schema violations and drift — a 7B model is sufficient."
-    },
-    "t3_overview": {
-      "purpose": "Mid-day checkpoint after every misplacement + every Nth event, and cross-day lesson. 1-3 calls per scenario.",
-      "kind": "thinking_cloud",
-      "primary": { "model": "gpt-oss:120b", "provider": "ollama_cloud", "context_window": 131072 },
-      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
-      "max_tokens": 900,
-      "temperature": 0.2,
-      "cloud_budget_per_scenario": 5,
-      "env_flag": "LH_OVERVIEW_CLOUD=1",
-      "context_budget": {
-        "event_digest_cap": 30000,
-        "checkpoint_cap": 8000,
-        "lesson_corpus_cap": 40000,
-        "safety_margin": 8000,
-        "overflow_policy": "chunk_lessons_via_cosine_topk"
-      },
-      "rationale": "Same prompt family as local 20b (gpt-oss series) — prompts port directly. 120b is faster via cloud than 20b local in practice, and lessons are noticeably more specific."
-    },
-    "t4_strategic": {
-      "purpose": "Daily playbook board re-ranking, weekly gap audit, pattern discovery across accumulated playbooks. 1-10 calls per day.",
-      "kind": "thinking_cloud_large",
-      "primary": { "model": "kimi-k2.6", "provider": "ollama_cloud", "context_window": 200000 },
-      "secondary": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
-      "fallback": { "model": "glm-4.7", "provider": "ollama_cloud", "context_window": 131072 },
-      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
-      "max_tokens": 2000,
-      "temperature": 0.2,
-      "cloud_budget_per_day": 10,
-      "context_budget": {
-        "playbook_corpus_cap": 80000,
-        "pattern_history_cap": 20000,
-        "safety_margin": 16000,
-        "overflow_policy": "two_pass_map_reduce"
-      },
-      "rationale": "J named qwen3.5 specifically. GLM-4.7 is a promising alternate for debate phase. Runs after all scenarios complete for the day."
-    },
-    "t5_gatekeeper": {
-      "purpose": "MUST route here: architecture changes, new client onboarding, schema migrations, playbook retirements, index rebuilds, autotune config changes.",
-      "kind": "thinking_cloud_deepest",
-      "primary": { "model": "kimi-k2-thinking", "provider": "ollama_cloud", "context_window": 200000 },
-      "fallback": { "model": "deepseek-v3.1:671b", "provider": "ollama_cloud", "context_window": 131072 },
-      "secondary_fallback": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
-      "local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
-      "max_tokens": 4000,
-      "temperature": 0.1,
-      "cloud_budget_per_day": 5,
-      "audit_log": true,
-      "context_budget": {
-        "decision_doc_cap": 50000,
-        "evidence_bundle_cap": 100000,
-        "prior_gatekeeper_decisions_cap": 20000,
-        "safety_margin": 20000,
-        "overflow_policy": "escalate_to_kimi_k2_1t_or_split_decision"
-      },
-      "rationale": "Highest-stakes decisions — reasoning depth matters more than latency. Audit log so J can always see what the gatekeeper was asked and what it answered. No human approval required today; escalate later if mis-decisions show up."
-    }
-  },
-
-  "embeddings": {
-    "_description": "Vector embedding models for hybrid search + playbook_memory similarity. Always local — embedding calls are too high-volume for cloud. qwen3-embedding is the new primary (2026-04-21); nomic-embed-text-v2-moe is the MoE alternative for corpus-scale embedding; original nomic-embed-text stays as legacy fallback while vectord indexes are being re-embedded.",
-    "primary": { "model": "qwen3-embedding", "provider": "ollama_local", "dim": 1024, "context_window": 32768 },
-    "alternate": { "model": "nomic-embed-text-v2-moe", "provider": "ollama_local", "dim": 768, "context_window": 2048, "notes": "MoE variant — better for long documents, slower per-call, roughly matches nomic-embed-text on short text." },
-    "legacy": { "model": "nomic-embed-text:latest", "provider": "ollama_local", "dim": 768, "context_window": 8192, "notes": "Used by existing vectord indexes. Do NOT switch live until indexes are rebuilt, or hybrid search recall will crater." },
-    "migration_plan": "Re-embed workers_500k (and all production indexes) with qwen3-embedding, keep the old index under a _v1 suffix for rollback, flip the profile pointer only after recall benchmarks match or exceed legacy. Touches vectord::agent + autotune."
-  },
-
-  "context_management": {
-    "_description": "Rule zero: NEVER call a model with more tokens than its context_window minus safety_margin. Every call goes through the budget checker first. If over budget → chunk, summarize, or escalate. This is the stability floor.",
-    "token_estimator": {
-      "method": "chars_div_4",
-      "note": "Rough, biased safe by ~15%. For production, swap to tiktoken or the provider's tokenizer endpoint."
-    },
-    "overflow_policies": {
-      "summarize_oldest_tool_results_via_t3": {
-        "when": "T1 conversation history + tool results exceed context_budget.tool_results_cap",
-        "how": "Send oldest N tool results to T3 with prompt 'summarize these into 500 tokens that preserve what the executor needs to know'; replace them with the summary in the running conversation."
-      },
-      "chunk_lessons_via_cosine_topk": {
-        "when": "lesson corpus in data/_playbook_lessons/*.json exceeds lesson_corpus_cap",
-        "how": "Embed the current scenario spec, cosine-rank all lessons, take top-K until budget exhausted. Fall back to date-sorted if embeddings unavailable."
-      },
-      "two_pass_map_reduce": {
-        "when": "T4 playbook corpus exceeds playbook_corpus_cap",
-        "how": "Pass 1: chunk playbooks into ≤30K token shards, run primary model on each shard to emit 'shard summary'. Pass 2: feed all summaries to primary model for global synthesis. Logged as two audit entries."
-      },
-      "escalate_to_kimi_k2_1t_or_split_decision": {
-        "when": "T5 decision evidence exceeds decision_doc_cap + evidence_bundle_cap",
-        "how": "Prefer kimi-k2:1t which has 1M context. If still over, split decision into sub-decisions (e.g. 'retire playbooks by city' instead of 'retire playbooks globally') and loop."
-      }
-    },
-    "chunking_cache": {
-      "_description": "Precomputed shards of the playbook corpus, indexed by (corpus_version, shard_id). Avoids re-chunking on every T4 run.",
-      "location": "data/_chunk_cache/",
-      "invalidation": "Key includes corpus hash. When playbook_memory changes, the hash changes, the cache misses, and chunks regenerate.",
-      "implementation_status": "SPEC — next sprint."
-    },
-    "implementation_status": "context_window + context_budget fields WIRED in config. Enforcement helper NOT yet wired in agent.ts. Next: add estimateTokens() + budgetCheck() helpers, route all generate() calls through them, emit a metric when overflow policy fires."
-  },
-  "experimental_rotation": {
-    "enabled": false,
-    "purpose": "Sample newer models on a schedule to collect comparison data without rate-limit risk.",
-    "candidates": [
-      { "model": "minimax-m2.7",      "notes": "Newer minimax; unknown output stability" },
-      { "model": "glm-5",             "notes": "GLM next-gen; larger context" },
-      { "model": "glm-5.1",           "notes": "Incremental on GLM-5" },
-      { "model": "qwen3-next:80b",    "notes": "Qwen's experimental successor; smaller than 3.5" },
-      { "model": "qwen3-coder-next",  "notes": "Coder-optimized — good for SQL gen T1 experiments" },
-      { "model": "deepseek-v3.2",     "notes": "Smaller deepseek; reasoning/coding" },
-      { "model": "nemotron-3-super",  "notes": "NVIDIA 230B; general" },
-      { "model": "cogito-2.1:671b",   "notes": "671B general" },
-      { "model": "mistral-large-3:675b", "notes": "Mistral's flagship; good T3 candidate" }
-    ],
-    "rotation": "weekly",
-    "sample_rate": 0.1,
-    "apply_to_tier": "t4_strategic",
-    "notes": "When enabled, T4 routes 10% of calls to a rotating experimental model. Log comparison in /data/_model_eval/ — if the experimental consistently beats primary across 3 rotations, promote it to primary."
-  },
-  "playbook_versioning": {
-    "enabled": true,
-    "purpose": "A playbook can work, then break when architecture changes. Versioning lets us pin which change retired which playbook.",
-    "dataset": "playbook_memory",
-    "schema_additions": {
-      "version": "integer — auto-increment per operation",
-      "parent_id": "string — previous version entry_id for same operation (null for v1)",
-      "retired_at": "timestamp — set when success_rate drops or architecture changes",
-      "retirement_reason": "string — e.g. 'schema_migration:workers_500k 2026-05-03'",
-      "architecture_snapshot": "object — crate versions, index name, schema fingerprint at seed time"
-    },
-    "retire_triggers": [
-      "success_rate < 0.3 over last 20 citations",
-      "schema_fingerprint mismatch detected at retrieval time",
-      "architecture change event emitted by ingestd/vectord",
-      "T5 gatekeeper explicitly retires via /vectors/playbook_memory/retire"
-    ],
-    "read_back_policy": "retrieval returns only non-retired versions. History endpoint /vectors/playbook_memory/history/{operation} returns the full chain.",
-    "ui_surface": "mcp-server to render a diff view: side-by-side of playbook versions with a timeline of what changed and when.",
-    "implementation_status": "SPEC — not yet wired. Target: next sprint. Touches gateway + catalogd + mcp-server."
-  },
-  "matrix_index_hybrid_search_note": "Phase 22 candidate: elevate the hybrid_search T1 tool to consult T3 when a pool returns <3 matches OR when the same (role, city) combo has failed N times in 24h. Consult result is a reformulated sql_filter the executor retries with. Keeps T1 fast on the happy path, escalates to T3 only on low-recall signals."
-}

crates/aibridge/src/client.rs

@@ -25,7 +25,7 @@ pub struct EmbedResponse {
     pub dimensions: usize,
 }
 
-#[derive(Clone, Serialize, Deserialize)]
+#[derive(Serialize, Deserialize)]
 pub struct GenerateRequest {
     pub prompt: String,
     #[serde(skip_serializing_if = "Option::is_none")]
@@ -36,14 +36,6 @@ pub struct GenerateRequest {
     pub temperature: Option<f64>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub max_tokens: Option<u32>,
-    /// Phase 21 — per-call opt-out of hidden reasoning. Thinking models
-    /// (qwen3.5, gpt-oss, etc) burn tokens on reasoning before the
-    /// visible response starts; setting this to `false` on hot-path
-    /// JSON emitters avoids empty returns when the budget is tight.
-    /// Sidecar forwards this to Ollama's `think` parameter; if the
-    /// sidecar drops an unknown field the request still succeeds.
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub think: Option<bool>,
 }
 
 #[derive(Deserialize, Serialize, Clone)]
@@ -142,47 +134,4 @@ impl AiClient {
         }
         resp.json().await.map_err(|e| format!("rerank parse error: {e}"))
     }
-
-    /// Force Ollama to unload the named model from VRAM (keep_alive=0).
-    /// Used for predictable profile swaps — without this, Ollama holds a
-    /// model for its configured TTL (default 5min) and the previous
-    /// profile's model can linger in VRAM next to the new one.
-    pub async fn unload_model(&self, model: &str) -> Result<serde_json::Value, String> {
-        let resp = self.client
-            .post(format!("{}/admin/unload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
-            .send().await
-            .map_err(|e| format!("unload request failed: {e}"))?;
-        if !resp.status().is_success() {
-            let text = resp.text().await.unwrap_or_default();
-            return Err(format!("unload error: {text}"));
-        }
-        resp.json().await.map_err(|e| format!("unload parse error: {e}"))
-    }
-
-    /// Ask Ollama to load the named model into VRAM proactively. Makes
-    /// the first real request after profile activation fast (no cold-load
-    /// latency).
-    pub async fn preload_model(&self, model: &str) -> Result<serde_json::Value, String> {
-        let resp = self.client
-            .post(format!("{}/admin/preload", self.base_url))
-            .json(&serde_json::json!({ "model": model }))
-            .send().await
-            .map_err(|e| format!("preload request failed: {e}"))?;
-        if !resp.status().is_success() {
-            let text = resp.text().await.unwrap_or_default();
-            return Err(format!("preload error: {text}"));
-        }
-        resp.json().await.map_err(|e| format!("preload parse error: {e}"))
-    }
-
-    /// GPU + loaded-model snapshot from the sidecar. Combines nvidia-smi
-    /// output (if available) with Ollama's /api/ps.
-    pub async fn vram_snapshot(&self) -> Result<serde_json::Value, String> {
-        let resp = self.client
-            .get(format!("{}/admin/vram", self.base_url))
-            .send().await
-            .map_err(|e| format!("vram request failed: {e}"))?;
-        resp.json().await.map_err(|e| format!("vram parse error: {e}"))
-    }
 }

crates/aibridge/src/context.rs

@@ -1,194 +0,0 @@
-//! Phase 21 — context-budget accounting for model calls.
-//!
-//! Ports `assertContextBudget` + `estimateTokens` + `CONTEXT_WINDOWS`
-//! from `tests/multi-agent/agent.ts` so Rust-side callers (gateway
-//! tool surfaces, future Rust agents) get the same loud-fail behavior
-//! on window overflow instead of silent truncation.
-//!
-//! The token estimator is deliberately the same chars/4 heuristic as
-//! the TS side. It's biased ~15% safe — pessimistic on English, correct
-//! within a factor of 2 on code. Swap to a provider tokenizer only when
-//! the estimator drives a decision (we're nowhere near that yet).
-
-use std::collections::HashMap;
-use std::sync::OnceLock;
-
-/// Rough token count. `chars / 4` ceiling. See module docs for why
-/// this heuristic is sufficient.
-pub fn estimate_tokens(text: &str) -> usize {
-    (text.chars().count() + 3) / 4
-}
-
-/// Phase 21 — per-model context windows, mirroring the TS table in
-/// `tests/multi-agent/agent.ts`. Anchored on each model's documented
-/// max; unknown models fall back to `DEFAULT_CONTEXT_WINDOW`.
-pub const DEFAULT_CONTEXT_WINDOW: usize = 32_768;
-pub const DEFAULT_SAFETY_MARGIN: usize = 2_000;
-pub const DEFAULT_MAX_TOKENS: usize = 800;
-
-fn known_windows() -> &'static HashMap<&'static str, usize> {
-    static TABLE: OnceLock<HashMap<&'static str, usize>> = OnceLock::new();
-    TABLE.get_or_init(|| {
-        let mut m = HashMap::new();
-        m.insert("mistral:latest", 32_768);
-        m.insert("qwen2.5:latest", 32_768);
-        m.insert("qwen3:latest", 40_960);
-        m.insert("qwen3.5:latest", 262_144);
-        m.insert("qwen3-embedding", 32_768);
-        m.insert("nomic-embed-text-v2-moe", 2_048);
-        m.insert("gpt-oss:20b", 131_072);
-        m.insert("gpt-oss:120b", 131_072);
-        m.insert("qwen3.5:397b", 131_072);
-        m.insert("kimi-k2-thinking", 200_000);
-        m.insert("kimi-k2.6", 200_000);
-        m.insert("kimi-k2:1t", 1_048_576);
-        m.insert("deepseek-v3.1:671b", 131_072);
-        m.insert("glm-4.7", 131_072);
-        m
-    })
-}
-
-pub fn context_window_for(model: &str) -> usize {
-    known_windows().get(model).copied().unwrap_or(DEFAULT_CONTEXT_WINDOW)
-}
-
-/// Result of a budget check — exposes the numbers so callers can log
-/// how much headroom remains without re-running the estimator.
-#[derive(Debug, Clone, Copy)]
-pub struct BudgetCheck {
-    pub estimated: usize,
-    pub window: usize,
-    pub remaining: i64,
-}
-
-/// Inputs to `assert_context_budget`. `bypass` exists for call sites
-/// that handle their own overflow (continuation's second pass already
-/// counted the partial; T5 gatekeeper prompts have a separate policy).
-#[derive(Debug, Clone, Default)]
-pub struct BudgetOpts<'a> {
-    pub system: Option<&'a str>,
-    pub max_tokens: Option<usize>,
-    pub safety_margin: Option<usize>,
-    pub bypass: bool,
-}
-
-/// Phase 21's loud-fail primitive. Returns a `BudgetCheck` on success
-/// and the same struct plus over-by count on failure. The whole point
-/// is to stop silent truncation — callers that expect overflow should
-/// chunk BEFORE calling or set `bypass: true`.
-pub fn assert_context_budget(
-    model: &str,
-    prompt: &str,
-    opts: BudgetOpts,
-) -> Result<BudgetCheck, (BudgetCheck, usize)> {
-    let window = context_window_for(model);
-    let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
-    let max_tokens = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS);
-    let sys_tokens = opts.system.map(estimate_tokens).unwrap_or(0);
-    let estimated = estimate_tokens(prompt) + sys_tokens + max_tokens;
-    let remaining = window as i64 - estimated as i64 - safety as i64;
-    let check = BudgetCheck { estimated, window, remaining };
-    if remaining < 0 && !opts.bypass {
-        return Err((check, (-remaining) as usize));
-    }
-    Ok(check)
-}
-
-/// Convenience — format an overflow error the same way the TS side
-/// does. Exposed so downstream crates render consistent messages.
-pub fn overflow_message(model: &str, check: &BudgetCheck, over_by: usize, safety: usize) -> String {
-    format!(
-        "context overflow: model={} est={}t window={}t safety={}t over={}t. \
-         Chunk the prompt (see config/models.json overflow_policies) or set \
-         bypass:true if you know the risk.",
-        model, check.estimated, check.window, safety, over_by,
-    )
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn estimate_tokens_ceiling_divides_by_four() {
-        assert_eq!(estimate_tokens(""), 0);
-        assert_eq!(estimate_tokens("abc"), 1);    // 3 → ceil(3/4) = 1
-        assert_eq!(estimate_tokens("abcd"), 1);   // 4 → ceil(4/4) = 1
-        assert_eq!(estimate_tokens("abcde"), 2);  // 5 → ceil(5/4) = 2
-        assert_eq!(estimate_tokens(&"x".repeat(400)), 100);
-    }
-
-    #[test]
-    fn context_window_known_and_fallback() {
-        assert_eq!(context_window_for("qwen3.5:latest"), 262_144);
-        assert_eq!(context_window_for("kimi-k2:1t"), 1_048_576);
-        assert_eq!(context_window_for("some-unreleased-model"), DEFAULT_CONTEXT_WINDOW);
-    }
-
-    #[test]
-    fn budget_passes_well_under_window() {
-        let check = assert_context_budget(
-            "qwen3:latest",
-            &"x".repeat(4_000), // ~1000 tokens
-            BudgetOpts { max_tokens: Some(500), ..Default::default() },
-        ).expect("well under 40K window");
-        assert!(check.remaining > 30_000);
-    }
-
-    #[test]
-    fn budget_fails_when_prompt_overflows_window() {
-        let huge = "x".repeat(200_000); // ~50K tokens, over qwen3's 40K
-        let err = assert_context_budget(
-            "qwen3:latest",
-            &huge,
-            BudgetOpts::default(),
-        ).expect_err("should overflow qwen3's 40K window");
-        assert!(err.1 > 0, "over_by must be positive");
-    }
-
-    #[test]
-    fn budget_bypass_returns_ok_even_over() {
-        let huge = "x".repeat(200_000);
-        let check = assert_context_budget(
-            "qwen3:latest",
-            &huge,
-            BudgetOpts { bypass: true, ..Default::default() },
-        ).expect("bypass must suppress the error");
-        assert!(check.remaining < 0, "check still reports negative remaining");
-    }
-
-    #[test]
-    fn budget_counts_system_prompt() {
-        // 10K-char system prompt → ~2500 tokens. With a big max_tokens
-        // this should push us closer to the window.
-        let sys = "s".repeat(10_000);
-        let prompt = "p".repeat(4_000);
-        let with_sys = assert_context_budget(
-            "qwen3:latest",
-            &prompt,
-            BudgetOpts {
-                system: Some(&sys),
-                max_tokens: Some(500),
-                ..Default::default()
-            },
-        ).unwrap();
-        let without_sys = assert_context_budget(
-            "qwen3:latest",
-            &prompt,
-            BudgetOpts { max_tokens: Some(500), ..Default::default() },
-        ).unwrap();
-        assert!(with_sys.estimated > without_sys.estimated,
-            "system prompt should raise estimate");
-        assert_eq!(with_sys.estimated - without_sys.estimated, estimate_tokens(&sys));
-    }
-
-    #[test]
-    fn overflow_message_includes_numbers() {
-        let check = BudgetCheck { estimated: 42_000, window: 40_960, remaining: -1_040 };
-        let msg = overflow_message("qwen3:latest", &check, 3_040, 2_000);
-        assert!(msg.contains("qwen3:latest"));
-        assert!(msg.contains("42000t"));
-        assert!(msg.contains("40960t"));
-        assert!(msg.contains("3040t"));
-    }
-}

crates/aibridge/src/continuation.rs

@@ -1,438 +0,0 @@
-//! Phase 21 — OUTPUT-overflow handler. Ports `generateContinuable`
-//! from `tests/multi-agent/agent.ts`.
-//!
-//! Two failure modes to repair:
-//!
-//! * EMPTY response — thinking model ate the entire budget on hidden
-//!   reasoning before emitting a token. Fix: retry the original prompt
-//!   with 2× the budget, geometric up to `BUDGET_CAP`.
-//!
-//! * TRUNCATED non-empty — model got most of the way but hit
-//!   max_tokens before closing the structure. Fix: continue with the
-//!   partial response in the prompt as scratchpad, so the model knows
-//!   where to pick up without restarting.
-//!
-//! `TextGenerator` abstracts the sidecar so tests can inject canned
-//! responses without a live Ollama.
-
-use std::future::Future;
-
-use crate::client::{AiClient, GenerateRequest, GenerateResponse};
-
-/// Shape classifier for `is_structurally_complete`. JSON responses
-/// must parse; text responses just need to be non-empty.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum ResponseShape {
-    Json,
-    Text,
-}
-
-/// Trait that `generate_continuable` + `generate_tree_split` call. The
-/// real implementation forwards to `AiClient::generate`; tests supply a
-/// mock with a scripted sequence of responses.
-pub trait TextGenerator: Send + Sync {
-    fn generate_text(
-        &self,
-        req: GenerateRequest,
-    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send;
-}
-
-impl TextGenerator for AiClient {
-    fn generate_text(
-        &self,
-        req: GenerateRequest,
-    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
-        self.generate(req)
-    }
-}
-
-/// Strip a surrounding ```json``` fence if present. Leaves the inner
-/// content alone otherwise. Returns a slice of `s`.
-fn strip_json_fence(s: &str) -> &str {
-    let t = s.trim();
-    if let Some(rest) = t.strip_prefix("```json") {
-        rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
-    } else if let Some(rest) = t.strip_prefix("```") {
-        rest.trim_start_matches('\n').strip_suffix("```").unwrap_or(rest).trim()
-    } else {
-        t
-    }
-}
-
-/// Port of the TS brace-balance + JSON.parse check. Returns true when
-/// the outermost `{...}` block is balanced and parses. Text shape is
-/// satisfied by any non-empty, non-whitespace payload.
-pub fn is_structurally_complete(text: &str, shape: ResponseShape) -> bool {
-    if text.trim().is_empty() { return false; }
-    if shape == ResponseShape::Text { return true; }
-
-    let s = strip_json_fence(text);
-    let Some(start) = s.find('{') else { return false };
-    let Some(end) = s.rfind('}') else { return false };
-    if end <= start { return false; }
-    let slice = &s[start..=end];
-
-    // Balance check — cheaper than parse, catches truncated nests.
-    // String state tracked because `{` inside a string doesn't count.
-    let mut depth: i32 = 0;
-    let mut in_str = false;
-    let mut esc = false;
-    for c in slice.chars() {
-        if esc { esc = false; continue; }
-        if c == '\\' { esc = true; continue; }
-        if c == '"' { in_str = !in_str; continue; }
-        if in_str { continue; }
-        if c == '{' { depth += 1; }
-        else if c == '}' {
-            depth -= 1;
-            if depth < 0 { return false; }
-        }
-    }
-    if depth != 0 { return false; }
-    // Parse check is the tie-breaker — balanced but invalid JSON (e.g.
-    // trailing comma before `}`) shouldn't count as complete.
-    serde_json::from_str::<serde_json::Value>(slice).is_ok()
-}
-
-/// Knobs for `generate_continuable`. All optional with sensible
-/// defaults that match the TS version.
-#[derive(Debug, Clone)]
-pub struct ContinuableOpts {
-    pub model: String,
-    pub max_tokens: Option<u32>,
-    pub temperature: Option<f64>,
-    pub system: Option<String>,
-    pub shape: ResponseShape,
-    pub max_continuations: usize,
-    pub think: Option<bool>,
-    /// Geometric-backoff ceiling for the empty-response retry path.
-    /// Matches TS's `budgetCap = 8000`.
-    pub budget_cap: u32,
-    /// Maximum empty-response retries before giving up. Matches TS's
-    /// hardcoded `retry < 3`.
-    pub max_empty_retries: usize,
-}
-
-impl ContinuableOpts {
-    pub fn new(model: impl Into<String>) -> Self {
-        Self {
-            model: model.into(),
-            max_tokens: None,
-            temperature: None,
-            system: None,
-            shape: ResponseShape::Json,
-            max_continuations: 3,
-            think: None,
-            budget_cap: 8_000,
-            max_empty_retries: 3,
-        }
-    }
-}
-
-/// Outcome of a `generate_continuable` call. Carries the combined
-/// text plus diagnostic counters so observability downstream can
-/// report "how many continuations did that query cost".
-#[derive(Debug, Clone)]
-pub struct ContinuableOutcome {
-    pub text: String,
-    pub empty_retries: usize,
-    pub continuations: usize,
-    pub final_complete: bool,
-}
-
-fn make_request(opts: &ContinuableOpts, prompt: String, current_max: u32) -> GenerateRequest {
-    GenerateRequest {
-        prompt,
-        model: Some(opts.model.clone()),
-        system: opts.system.clone(),
-        temperature: opts.temperature,
-        max_tokens: Some(current_max),
-        think: opts.think,
-    }
-}
-
-fn continuation_prompt(original: &str, partial: &str) -> String {
-    format!(
-        "{original}\n\n\
-         PARTIAL RESPONSE SO FAR (continue from here — do NOT restart, \
-         do NOT repeat what's already there, emit ONLY the remaining \
-         tokens to close the structure):\n{partial}"
-    )
-}
-
-/// Phase 21 — output-overflow safe generate. See module docs for the
-/// two failure modes repaired. On final-still-incomplete, returns the
-/// combined text with `final_complete: false` so the caller's parser
-/// can throw with the raw text for forensics rather than silently
-/// truncating.
-pub async fn generate_continuable<G: TextGenerator>(
-    generator: &G,
-    prompt: &str,
-    opts: &ContinuableOpts,
-) -> Result<ContinuableOutcome, String> {
-    let initial_max = opts.max_tokens.unwrap_or(800);
-    let mut current_max = initial_max;
-    let mut combined = String::new();
-    let mut empty_retries = 0usize;
-    let mut continuations = 0usize;
-
-    // Phase 21(a) — empty-response backoff loop.
-    for retry in 0..opts.max_empty_retries {
-        let req = make_request(opts, prompt.to_string(), current_max);
-        let resp = generator.generate_text(req).await?;
-        if !resp.text.trim().is_empty() {
-            combined = resp.text;
-            break;
-        }
-        empty_retries = retry + 1;
-        current_max = (current_max.saturating_mul(2)).min(opts.budget_cap);
-    }
-
-    // Phase 21(b) — structural-completion continuation loop. Runs on
-    // the truncated-non-empty case; empty + exhausted retries falls
-    // through with empty combined and final_complete=false.
-    for _ in 0..opts.max_continuations {
-        if is_structurally_complete(&combined, opts.shape) {
-            return Ok(ContinuableOutcome {
-                text: combined,
-                empty_retries,
-                continuations,
-                final_complete: true,
-            });
-        }
-        if combined.trim().is_empty() {
-            // Nothing to continue from — continuing "" is identical to
-            // the initial call and would loop. Bail so the caller sees
-            // the failure rather than burning N extra calls.
-            break;
-        }
-        let cont_prompt = continuation_prompt(prompt, &combined);
-        let req = make_request(opts, cont_prompt, current_max.min(opts.budget_cap));
-        let resp = generator.generate_text(req).await?;
-        combined.push_str(&resp.text);
-        continuations += 1;
-    }
-
-    let final_complete = is_structurally_complete(&combined, opts.shape);
-    Ok(ContinuableOutcome {
-        text: combined,
-        empty_retries,
-        continuations,
-        final_complete,
-    })
-}
-
-/// Scripted generator for unit tests. Returns responses from `script`
-/// in order; extra calls reuse the last entry so tests don't have to
-/// count past what they actually assert on.
-#[cfg(test)]
-pub struct ScriptedGenerator {
-    script: Vec<Result<String, String>>,
-    calls: std::sync::Arc<std::sync::Mutex<Vec<GenerateRequest>>>,
-}
-
-#[cfg(test)]
-impl ScriptedGenerator {
-    pub fn new<I>(script: I) -> Self
-    where
-        I: IntoIterator<Item = Result<String, String>>,
-    {
-        Self {
-            script: script.into_iter().collect(),
-            calls: std::sync::Arc::new(std::sync::Mutex::new(Vec::new())),
-        }
-    }
-
-    pub fn calls(&self) -> Vec<GenerateRequest> {
-        self.calls.lock().unwrap().clone()
-    }
-
-    pub fn call_count(&self) -> usize {
-        self.calls.lock().unwrap().len()
-    }
-}
-
-#[cfg(test)]
-impl TextGenerator for ScriptedGenerator {
-    fn generate_text(
-        &self,
-        req: GenerateRequest,
-    ) -> impl Future<Output = Result<GenerateResponse, String>> + Send {
-        let i = {
-            let mut calls = self.calls.lock().unwrap();
-            let i = calls.len();
-            calls.push(req.clone());
-            i
-        };
-        let model = req.model.clone().unwrap_or_default();
-        let entry = self.script.get(i)
-            .cloned()
-            .unwrap_or_else(|| self.script.last().cloned().unwrap_or(Ok(String::new())));
-        async move {
-            entry.map(|text| GenerateResponse {
-                text,
-                model,
-                tokens_evaluated: None,
-                tokens_generated: None,
-            })
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn structural_complete_rejects_empty_and_text_mismatch() {
-        assert!(!is_structurally_complete("", ResponseShape::Text));
-        assert!(!is_structurally_complete("   ", ResponseShape::Text));
-        assert!(is_structurally_complete("any content", ResponseShape::Text));
-    }
-
-    #[test]
-    fn structural_complete_handles_balanced_json() {
-        assert!(is_structurally_complete(r#"{"a": 1}"#, ResponseShape::Json));
-        assert!(is_structurally_complete(
-            r#"```json
-{"a": 1, "b": [1, 2, 3]}
-```"#,
-            ResponseShape::Json,
-        ));
-    }
-
-    #[test]
-    fn structural_complete_rejects_truncated_json() {
-        assert!(!is_structurally_complete(r#"{"a": 1"#, ResponseShape::Json));
-        assert!(!is_structurally_complete(r#"{"a": {"b": 1"#, ResponseShape::Json));
-        // Trailing comma — balanced but unparseable
-        assert!(!is_structurally_complete(r#"{"a": 1,}"#, ResponseShape::Json));
-    }
-
-    #[test]
-    fn structural_complete_ignores_braces_inside_strings() {
-        assert!(is_structurally_complete(r#"{"s": "has } inside"}"#, ResponseShape::Json));
-        assert!(is_structurally_complete(r#"{"s": "escaped \" quote"}"#, ResponseShape::Json));
-    }
-
-    #[tokio::test]
-    async fn continuable_returns_first_response_when_complete() {
-        let generator = ScriptedGenerator::new(vec![Ok(r#"{"ok": true}"#.to_string())]);
-        let opts = ContinuableOpts::new("qwen3:latest");
-        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
-        assert!(out.final_complete);
-        assert_eq!(out.empty_retries, 0);
-        assert_eq!(out.continuations, 0);
-        assert_eq!(generator.call_count(), 1);
-    }
-
-    #[tokio::test]
-    async fn continuable_retries_on_empty_with_doubled_budget() {
-        // Two empties, then a good response. Third call should see 4×
-        // the initial budget (2× twice).
-        let generator = ScriptedGenerator::new(vec![
-            Ok("".to_string()),
-            Ok("".to_string()),
-            Ok(r#"{"ok": true}"#.to_string()),
-        ]);
-        let mut opts = ContinuableOpts::new("qwen3:latest");
-        opts.max_tokens = Some(100);
-        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
-        assert!(out.final_complete);
-        assert_eq!(out.empty_retries, 2);
-
-        let calls = generator.calls();
-        assert_eq!(calls.len(), 3);
-        assert_eq!(calls[0].max_tokens, Some(100));
-        assert_eq!(calls[1].max_tokens, Some(200));
-        assert_eq!(calls[2].max_tokens, Some(400));
-    }
-
-    #[tokio::test]
-    async fn continuable_caps_budget_at_budget_cap() {
-        let generator = ScriptedGenerator::new(vec![
-            Ok("".to_string()),
-            Ok("".to_string()),
-            Ok(r#"{"ok":1}"#.to_string()),
-        ]);
-        let mut opts = ContinuableOpts::new("qwen3:latest");
-        opts.max_tokens = Some(5_000);
-        opts.budget_cap = 7_000;
-        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
-        assert!(out.final_complete);
-        let calls = generator.calls();
-        // 5000 → doubled would be 10000; cap pulls it to 7000.
-        assert_eq!(calls[1].max_tokens, Some(7_000));
-        assert_eq!(calls[2].max_tokens, Some(7_000));
-    }
-
-    #[tokio::test]
-    async fn continuable_glues_truncated_response() {
-        // First call returns balanced-open `{...`; continuation closes
-        // it with `...}`. Combined must parse.
-        let generator = ScriptedGenerator::new(vec![
-            Ok(r#"{"fills": [{"candidate_id": "C-001""#.to_string()),
-            Ok(r#", "name": "Alice"}]}"#.to_string()),
-        ]);
-        let opts = ContinuableOpts::new("qwen3:latest");
-        let out = generate_continuable(&generator, "ORIGINAL", &opts).await.unwrap();
-        assert!(out.final_complete, "combined must parse: {}", out.text);
-        assert_eq!(out.continuations, 1);
-
-        let calls = generator.calls();
-        assert_eq!(calls.len(), 2);
-        // Continuation prompt must contain the partial — that's the
-        // scratchpad primitive J called out.
-        let cont_prompt = &calls[1].prompt;
-        assert!(cont_prompt.contains("ORIGINAL"),
-            "continuation must include original prompt");
-        assert!(cont_prompt.contains(r#"{"fills": [{"candidate_id": "C-001""#),
-            "continuation must include partial");
-    }
-
-    #[tokio::test]
-    async fn continuable_does_not_loop_on_persistent_empty() {
-        // All three retries return empty; we must NOT then enter the
-        // continuation loop with an empty partial (would burn 3 more
-        // calls continuing from "").
-        let generator = ScriptedGenerator::new(vec![
-            Ok("".to_string()),
-            Ok("".to_string()),
-            Ok("".to_string()),
-        ]);
-        let opts = ContinuableOpts::new("qwen3:latest");
-        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
-        assert!(!out.final_complete);
-        assert_eq!(out.empty_retries, 3);
-        assert_eq!(out.continuations, 0, "must not continue from empty");
-        assert_eq!(generator.call_count(), 3);
-    }
-
-    #[tokio::test]
-    async fn continuable_returns_raw_on_exhausted_continuations() {
-        // Three continuations that never complete — caller's parser
-        // will throw. We must return the combined text so forensics
-        // has the raw content, not a lossy truncation.
-        let generator = ScriptedGenerator::new(vec![
-            Ok(r#"{"a": ["#.to_string()),
-            Ok("1,".to_string()),
-            Ok("2,".to_string()),
-            Ok("3,".to_string()),
-        ]);
-        let mut opts = ContinuableOpts::new("qwen3:latest");
-        opts.max_continuations = 3;
-        let out = generate_continuable(&generator, "test", &opts).await.unwrap();
-        assert!(!out.final_complete);
-        assert_eq!(out.continuations, 3);
-        assert!(out.text.contains(r#"{"a": ["#));
-        assert!(out.text.contains("3,"));
-    }
-
-    #[tokio::test]
-    async fn continuable_propagates_generator_errors() {
-        let generator = ScriptedGenerator::new(vec![Err("sidecar 503".to_string())]);
-        let opts = ContinuableOpts::new("qwen3:latest");
-        let err = generate_continuable(&generator, "test", &opts).await.unwrap_err();
-        assert!(err.contains("503"));
-    }
-}

crates/aibridge/src/lib.rs

@@ -1,5 +1,2 @@
 pub mod client;
-pub mod context;
-pub mod continuation;
 pub mod service;
-pub mod tree_split;

crates/aibridge/src/service.rs

@@ -16,17 +16,9 @@ pub fn router(client: AiClient) -> Router {
         .route("/embed", post(embed))
         .route("/generate", post(generate))
         .route("/rerank", post(rerank))
-        .route("/vram", get(vram))
         .with_state(client)
 }
 
-async fn vram(State(client): State<AiClient>) -> impl IntoResponse {
-    match client.vram_snapshot().await {
-        Ok(snap) => Ok(Json(snap)),
-        Err(e) => Err((StatusCode::BAD_GATEWAY, e)),
-    }
-}
-
 async fn health(State(client): State<AiClient>) -> impl IntoResponse {
     match client.health().await {
         Ok(info) => Ok(Json(info)),

crates/aibridge/src/tree_split.rs

@@ -1,326 +0,0 @@
-//! Phase 21 — INPUT-overflow handler. Ports `generateTreeSplit` from
-//! `tests/multi-agent/agent.ts`.
-//!
-//! When the input corpus exceeds the model's window (200 playbooks
-//! pasted into a T4 strategic prompt, a long retrospective digest, a
-//! cross-corpus summarization), raising `max_tokens` doesn't help —
-//! the prompt itself is the problem. The answer is map-reduce:
-//!
-//!   1. Caller shards the input at semantic boundaries (records,
-//!      paragraphs, playbook entries).
-//!   2. For each shard, build a map prompt that includes the running
-//!      scratchpad and run it through `generate_continuable`.
-//!   3. Append the map output to the scratchpad (oldest-first
-//!      truncation when it outgrows `scratchpad_budget`).
-//!   4. Build a reduce prompt from the final scratchpad and run it.
-//!
-//! Every shard prompt and the reduce prompt go through
-//! `assert_context_budget` first — if a single shard still overflows
-//! we bubble the error up rather than silently truncating. That's the
-//! whole point of Phase 21.
-
-use crate::context::{assert_context_budget, BudgetOpts, estimate_tokens, overflow_message,
-                     DEFAULT_MAX_TOKENS, DEFAULT_SAFETY_MARGIN};
-use crate::continuation::{generate_continuable, ContinuableOpts, ResponseShape, TextGenerator};
-
-/// Callback signatures — caller supplies closures that stitch the
-/// scratchpad into each shard's prompt and build the final reduce
-/// prompt. Kept as `Fn` (not `FnMut`) so the map loop can call them
-/// by reference.
-pub type MapPromptFn<'a> = dyn Fn(&str, &str) -> String + Send + Sync + 'a;
-pub type ReducePromptFn<'a> = dyn Fn(&str) -> String + Send + Sync + 'a;
-
-/// Knobs for `generate_tree_split`.
-#[derive(Debug, Clone)]
-pub struct TreeSplitOpts {
-    pub model: String,
-    pub system: Option<String>,
-    pub temperature: Option<f64>,
-    /// max_tokens for map AND reduce (reduce defaults are usually
-    /// higher; caller overrides for just reduce by calling through
-    /// continuable directly if needed).
-    pub max_tokens: Option<u32>,
-    pub reduce_max_tokens: Option<u32>,
-    pub think: Option<bool>,
-    /// Soft ceiling on scratchpad size (estimated tokens). When it
-    /// grows past this, the oldest shard digest gets dropped. Default
-    /// 6000, matching the TS implementation.
-    pub scratchpad_budget: usize,
-    pub safety_margin: Option<usize>,
-}
-
-impl TreeSplitOpts {
-    pub fn new(model: impl Into<String>) -> Self {
-        Self {
-            model: model.into(),
-            system: None,
-            temperature: None,
-            max_tokens: None,
-            reduce_max_tokens: None,
-            think: None,
-            scratchpad_budget: 6_000,
-            safety_margin: None,
-        }
-    }
-}
-
-/// Result — final reduce response plus the accumulated scratchpad so
-/// the caller can inspect what was kept vs truncated.
-#[derive(Debug, Clone)]
-pub struct TreeSplitResult {
-    pub response: String,
-    pub scratchpad: String,
-    pub shards_processed: usize,
-    pub scratchpad_truncations: usize,
-    pub total_continuations: usize,
-}
-
-/// Drop shard-digest blocks from the head of `scratchpad` until its
-/// estimated-token count fits the budget. Digest blocks are delimited
-/// by `\n— shard N/M digest —\n` so we can find the first one and
-/// chop everything before its successor.
-fn truncate_scratchpad(scratchpad: &mut String, budget_tokens: usize) -> bool {
-    if estimate_tokens(scratchpad) <= budget_tokens { return false; }
-    // Find the second delimiter — everything before it gets dropped.
-    const DELIM_PREFIX: &str = "\n— shard ";
-    let mut cursor = 0;
-    let mut truncated = false;
-    while estimate_tokens(&scratchpad[cursor..]) > budget_tokens {
-        // Skip past a leading delimiter (if we're sitting on one from
-        // a previous iteration), then find the next.
-        let search_from = cursor + if scratchpad[cursor..].starts_with(DELIM_PREFIX) {
-            DELIM_PREFIX.len()
-        } else { 0 };
-        let Some(rel_next) = scratchpad[search_from..].find(DELIM_PREFIX) else { break };
-        cursor = search_from + rel_next;
-        truncated = true;
-    }
-    if cursor > 0 {
-        scratchpad.drain(..cursor);
-    }
-    truncated
-}
-
-/// Phase 21 — map-reduce over shards with a running scratchpad. See
-/// module docs.
-pub async fn generate_tree_split<G: TextGenerator>(
-    generator: &G,
-    shards: &[String],
-    map_prompt: &MapPromptFn<'_>,
-    reduce_prompt: &ReducePromptFn<'_>,
-    opts: &TreeSplitOpts,
-) -> Result<TreeSplitResult, String> {
-    let mut scratchpad = String::new();
-    let safety = opts.safety_margin.unwrap_or(DEFAULT_SAFETY_MARGIN);
-    let map_max = opts.max_tokens.unwrap_or(DEFAULT_MAX_TOKENS as u32);
-    let reduce_max = opts.reduce_max_tokens.unwrap_or(1_500);
-    let mut truncations = 0usize;
-    let mut total_continuations = 0usize;
-
-    for (i, shard) in shards.iter().enumerate() {
-        let shard_prompt = map_prompt(shard, &scratchpad);
-        // Loud-fail on per-shard overflow — caller sharded too
-        // coarsely. Silent truncation is exactly the mode J rejected.
-        let budget = BudgetOpts {
-            system: opts.system.as_deref(),
-            max_tokens: Some(map_max as usize),
-            safety_margin: Some(safety),
-            bypass: false,
-        };
-        let check = assert_context_budget(&opts.model, &shard_prompt, budget)
-            .map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
-        let _ = check;
-
-        let mut cont_opts = ContinuableOpts::new(&opts.model);
-        cont_opts.max_tokens = Some(map_max);
-        cont_opts.temperature = opts.temperature;
-        cont_opts.system = opts.system.clone();
-        cont_opts.shape = ResponseShape::Text;
-        cont_opts.think = opts.think;
-
-        let outcome = generate_continuable(generator, &shard_prompt, &cont_opts).await?;
-        total_continuations += outcome.continuations;
-
-        // Append this shard's digest and, if needed, drop oldest.
-        scratchpad.push_str(&format!(
-            "\n— shard {}/{} digest —\n{}",
-            i + 1, shards.len(), outcome.text.trim(),
-        ));
-        if truncate_scratchpad(&mut scratchpad, opts.scratchpad_budget) {
-            truncations += 1;
-        }
-    }
-
-    // Reduce pass. Budget check first — if the scratchpad is still too
-    // big for the reduce prompt we fail loud with numbers.
-    let reduce_p = reduce_prompt(&scratchpad);
-    let budget = BudgetOpts {
-        system: opts.system.as_deref(),
-        max_tokens: Some(reduce_max as usize),
-        safety_margin: Some(safety),
-        bypass: false,
-    };
-    assert_context_budget(&opts.model, &reduce_p, budget)
-        .map_err(|(c, over)| overflow_message(&opts.model, &c, over, safety))?;
-
-    let mut cont_opts = ContinuableOpts::new(&opts.model);
-    cont_opts.max_tokens = Some(reduce_max);
-    cont_opts.temperature = opts.temperature;
-    cont_opts.system = opts.system.clone();
-    cont_opts.shape = ResponseShape::Text;
-    cont_opts.think = opts.think;
-
-    let outcome = generate_continuable(generator, &reduce_p, &cont_opts).await?;
-    total_continuations += outcome.continuations;
-
-    Ok(TreeSplitResult {
-        response: outcome.text,
-        scratchpad,
-        shards_processed: shards.len(),
-        scratchpad_truncations: truncations,
-        total_continuations,
-    })
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::continuation::ScriptedGenerator;
-
-    fn simple_map(shard: &str, scratchpad: &str) -> String {
-        format!("SCRATCHPAD:\n{scratchpad}\n---\nSHARD:\n{shard}\n---\nDIGEST:")
-    }
-
-    fn simple_reduce(scratchpad: &str) -> String {
-        format!("SCRATCHPAD:\n{scratchpad}\n---\nFINAL:")
-    }
-
-    #[tokio::test]
-    async fn tree_split_runs_map_then_reduce() {
-        // 3 shards → 3 map calls → 1 reduce call = 4 responses scripted.
-        let generator = ScriptedGenerator::new(vec![
-            Ok("digest-1".to_string()),
-            Ok("digest-2".to_string()),
-            Ok("digest-3".to_string()),
-            Ok("FINAL ANSWER".to_string()),
-        ]);
-        let shards: Vec<String> = ["a", "b", "c"].iter().map(|s| s.to_string()).collect();
-        let opts = TreeSplitOpts::new("qwen3:latest");
-        let map_fn: &MapPromptFn = &simple_map;
-        let reduce_fn: &ReducePromptFn = &simple_reduce;
-        let result = generate_tree_split(&generator, &shards, map_fn, reduce_fn, &opts)
-            .await
-            .unwrap();
-        assert_eq!(result.shards_processed, 3);
-        assert_eq!(result.response, "FINAL ANSWER");
-        assert_eq!(generator.call_count(), 4);
-        // Scratchpad must carry all three digests in order.
-        assert!(result.scratchpad.contains("digest-1"));
-        assert!(result.scratchpad.contains("digest-2"));
-        assert!(result.scratchpad.contains("digest-3"));
-    }
-
-    #[tokio::test]
-    async fn tree_split_reduce_prompt_sees_full_scratchpad() {
-        let generator = ScriptedGenerator::new(vec![
-            Ok("summary-A".to_string()),
-            Ok("summary-B".to_string()),
-            Ok("REDUCED".to_string()),
-        ]);
-        let shards = vec!["input-one".to_string(), "input-two".to_string()];
-        let opts = TreeSplitOpts::new("qwen3:latest");
-        let _ = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
-            .await
-            .unwrap();
-        // Third call = reduce. Its prompt must include both digests.
-        let calls = generator.calls();
-        let reduce_prompt = &calls[2].prompt;
-        assert!(reduce_prompt.contains("summary-A"),
-            "reduce prompt must see first shard digest");
-        assert!(reduce_prompt.contains("summary-B"),
-            "reduce prompt must see second shard digest");
-    }
-
-    #[tokio::test]
-    async fn tree_split_loud_fails_on_shard_overflow() {
-        let generator = ScriptedGenerator::new(vec![Ok("digest".to_string())]);
-        // One gigantic shard — well over qwen3's 40K window even as a
-        // prompt. The budget check must reject before any generate call.
-        let shards = vec!["x".repeat(200_000)];
-        let opts = TreeSplitOpts::new("qwen3:latest");
-        let err = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
-            .await
-            .expect_err("shard-sized overflow must be rejected");
-        assert!(err.contains("overflow"), "error should mention overflow: {err}");
-        assert_eq!(generator.call_count(), 0, "generate must not be called on overflow");
-    }
-
-    #[tokio::test]
-    async fn tree_split_truncates_scratchpad_when_over_budget() {
-        // Tight budget so each shard trips truncation. qwen3's 40K
-        // window is fine; the budget we care about is the scratchpad
-        // cap, not the model window.
-        let generator = ScriptedGenerator::new(vec![
-            Ok("A".repeat(2_000)),
-            Ok("B".repeat(2_000)),
-            Ok("C".repeat(2_000)),
-            Ok("D".repeat(2_000)),
-            Ok("FINAL".to_string()),
-        ]);
-        let shards: Vec<String> = (0..4).map(|i| format!("shard{i}")).collect();
-        let mut opts = TreeSplitOpts::new("qwen3:latest");
-        opts.scratchpad_budget = 1_000; // ~4000 chars — one digest barely fits
-        let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
-            .await
-            .unwrap();
-        assert!(result.scratchpad_truncations > 0,
-            "tight budget must trigger truncation");
-        // Scratchpad should still fit roughly within the budget
-        // (post-truncation); the estimator uses chars/4 so the bound
-        // is ~budget*4 chars. Give some slack for the delimiter.
-        let scratchpad_tokens = estimate_tokens(&result.scratchpad);
-        assert!(scratchpad_tokens <= opts.scratchpad_budget * 2,
-            "scratchpad {} tokens vs budget {}", scratchpad_tokens, opts.scratchpad_budget);
-    }
-
-    #[tokio::test]
-    async fn tree_split_reports_continuations_from_map_and_reduce() {
-        // First shard: truncated-then-continued. Reduce: truncated-then-continued.
-        // 1 shard: 2 map calls (initial + continuation), then 2 reduce calls.
-        let generator = ScriptedGenerator::new(vec![
-            Ok("partial".to_string()),         // map shape=text, non-empty → complete on first pass
-            Ok("reduce-out".to_string()),
-        ]);
-        let shards = vec!["only".to_string()];
-        let opts = TreeSplitOpts::new("qwen3:latest");
-        let result = generate_tree_split(&generator, &shards, &simple_map, &simple_reduce, &opts)
-            .await
-            .unwrap();
-        // Text shape treats non-empty as complete → 0 continuations.
-        assert_eq!(result.total_continuations, 0);
-        assert_eq!(result.shards_processed, 1);
-    }
-
-    #[test]
-    fn truncate_scratchpad_noop_when_under_budget() {
-        let mut s = "\n— shard 1/1 digest —\nshort".to_string();
-        let truncated = truncate_scratchpad(&mut s, 1_000);
-        assert!(!truncated);
-        assert!(s.contains("short"));
-    }
-
-    #[test]
-    fn truncate_scratchpad_drops_oldest_first() {
-        let mut s = format!(
-            "\n— shard 1/3 digest —\n{}\n— shard 2/3 digest —\n{}\n— shard 3/3 digest —\nshort",
-            "x".repeat(4_000), // ~1000 tokens
-            "y".repeat(4_000), // ~1000 tokens
-        );
-        let truncated = truncate_scratchpad(&mut s, 500); // ~2000 chars
-        assert!(truncated);
-        assert!(!s.contains(&"x".repeat(4_000)),
-            "oldest digest should be dropped");
-        assert!(s.contains("short"),
-            "newest digest should survive");
-    }
-}

crates/catalogd/Cargo.toml

@@ -15,6 +15,5 @@ bytes = { workspace = true }
 chrono = { workspace = true }
 uuid = { workspace = true }
 object_store = { workspace = true }
-arrow = { workspace = true }
 proto = { path = "../proto" }
 tonic = { workspace = true }

crates/catalogd/src/grpc.rs

@@ -54,16 +54,7 @@ impl CatalogService for CatalogGrpc {
         let manifest = self.registry
             .register(req.name, SchemaFingerprint(req.schema_fingerprint), objects)
             .await
-            .map_err(|e| {
-                // Mirror the HTTP side's 409 mapping: schema drift is a
-                // precondition failure, not a server error. Keeps gRPC
-                // and HTTP callers seeing the same diagnostic signal.
-                if e.contains("different schema") {
-                    Status::failed_precondition(e)
-                } else {
-                    Status::internal(e)
-                }
-            })?;
+            .map_err(|e| Status::internal(e))?;
 
         Ok(Response::new(manifest_to_proto(&manifest)))
     }

crates/catalogd/src/lib.rs

@@ -1,4 +1,3 @@
 pub mod registry;
 pub mod service;
 pub mod grpc;
-pub mod tombstones;

crates/catalogd/src/registry.rs

@@ -1,9 +1,7 @@
 use shared::types::{
-    AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile,
-    ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone,
+    DatasetId, DatasetManifest, ObjectRef, SchemaFingerprint,
+    ColumnMeta, Lineage, FreshnessContract, Sensitivity,
 };
-
-use crate::tombstones::TombstoneStore;
 use std::collections::HashMap;
 use std::sync::Arc;
 use tokio::sync::RwLock;
@@ -11,44 +9,6 @@ use tokio::sync::RwLock;
 use storaged::ops;
 use object_store::ObjectStore;
 
-/// Make a view name safe for use as an object storage key.
-/// Allows letters, digits, `_`, `-`, `.`. Anything else becomes `_`.
-fn sanitize_view_name(name: &str) -> String {
-    name.chars()
-        .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' { c } else { '_' })
-        .collect()
-}
-
-#[derive(Debug, Clone, Default, serde::Serialize)]
-pub struct MigrateBucketsReport {
-    pub refs_examined: usize,
-    pub refs_renamed: usize,    // legacy "data"/"local" → "primary"
-    pub refs_stamped: usize,    // empty → "primary"
-    pub refs_unchanged: usize,  // already canonical
-    pub manifests_persisted: usize,
-}
-
-#[derive(Debug, Clone, Default, serde::Serialize)]
-pub struct DedupeReport {
-    pub groups: usize,
-    pub removed: usize,
-    pub kept: Vec<DedupeKept>,
-    pub errors: Vec<DedupeError>,
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct DedupeKept {
-    pub name: String,
-    pub kept_id: String,
-    pub removed: usize,
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct DedupeError {
-    pub manifest_id: String,
-    pub error: String,
-}
-
 /// Partial metadata update — only set fields are applied.
 #[derive(Debug, Clone, Default, serde::Deserialize)]
 pub struct MetadataUpdate {
@@ -60,23 +20,14 @@ pub struct MetadataUpdate {
     pub lineage: Option<Lineage>,
     pub freshness: Option<FreshnessContract>,
     pub row_count: Option<u64>,
-    // Phase C embedding freshness
-    pub embedding_refresh_policy: Option<RefreshPolicy>,
 }
 
 const MANIFEST_PREFIX: &str = "_catalog/manifests";
-const VIEW_PREFIX: &str = "_catalog/views";
-const PROFILE_PREFIX: &str = "_catalog/profiles";
 
 /// In-memory dataset registry backed by manifest persistence in object storage.
-/// Also tracks AiViews (Phase D) — safe projections over base datasets.
-/// And tombstones (Phase E) — soft-delete markers applied at query time.
 #[derive(Clone)]
 pub struct Registry {
     datasets: Arc<RwLock<HashMap<DatasetId, DatasetManifest>>>,
-    views: Arc<RwLock<HashMap<String, AiView>>>,
-    profiles: Arc<RwLock<HashMap<String, ModelProfile>>>,
-    tombstones: TombstoneStore,
     store: Arc<dyn ObjectStore>,
 }
 
@@ -84,47 +35,11 @@ impl Registry {
     pub fn new(store: Arc<dyn ObjectStore>) -> Self {
         Self {
             datasets: Arc::new(RwLock::new(HashMap::new())),
-            views: Arc::new(RwLock::new(HashMap::new())),
-            profiles: Arc::new(RwLock::new(HashMap::new())),
-            tombstones: TombstoneStore::new(store.clone()),
             store,
         }
     }
 
-    pub fn tombstones(&self) -> &TombstoneStore {
-        &self.tombstones
-    }
-
-    /// Add a tombstone for a dataset row. Validates that the dataset
-    /// exists. Idempotent at the endpoint layer — callers dedup if needed.
-    pub async fn add_tombstone(
-        &self,
-        dataset: &str,
-        row_key_column: &str,
-        row_key_value: &str,
-        actor: &str,
-        reason: &str,
-    ) -> Result<Tombstone, String> {
-        if self.get_by_name(dataset).await.is_none() {
-            return Err(format!("dataset not found: {dataset}"));
-        }
-        let ts = Tombstone {
-            dataset: dataset.to_string(),
-            row_key_column: row_key_column.to_string(),
-            row_key_value: row_key_value.to_string(),
-            deleted_at: chrono::Utc::now(),
-            actor: actor.to_string(),
-            reason: reason.to_string(),
-        };
-        self.tombstones.append(&ts).await?;
-        Ok(ts)
-    }
-
-    pub async fn list_tombstones(&self, dataset: &str) -> Result<Vec<Tombstone>, String> {
-        self.tombstones.list(dataset).await
-    }
-
-    /// Rebuild in-memory index from persisted manifests + views on startup.
+    /// Rebuild in-memory index from persisted manifests on startup.
     pub async fn rebuild(&self) -> Result<usize, String> {
         let keys = ops::list(&self.store, Some(MANIFEST_PREFIX)).await?;
         let mut datasets = self.datasets.write().await;
@@ -137,161 +52,16 @@ impl Registry {
         }
         let count = datasets.len();
         tracing::info!("catalog rebuilt: {count} datasets loaded");
-
-        // Phase D: load AiView definitions alongside manifests.
-        let view_keys = ops::list(&self.store, Some(VIEW_PREFIX)).await.unwrap_or_default();
-        let mut views = self.views.write().await;
-        views.clear();
-        for key in &view_keys {
-            if !key.ends_with(".json") { continue; }
-            let data = match ops::get(&self.store, key).await {
-                Ok(d) => d,
-                Err(e) => {
-                    tracing::warn!("view '{key}': read failed: {e}");
-                    continue;
-                }
-            };
-            match serde_json::from_slice::<AiView>(&data) {
-                Ok(view) => { views.insert(view.name.clone(), view); }
-                Err(e) => tracing::warn!("view '{key}': parse failed: {e}"),
-            }
-        }
-        if !views.is_empty() {
-            tracing::info!("catalog: {} views loaded", views.len());
-        }
-
-        // Phase 17: load model profiles.
-        let profile_keys = ops::list(&self.store, Some(PROFILE_PREFIX)).await.unwrap_or_default();
-        let mut profiles = self.profiles.write().await;
-        profiles.clear();
-        for key in &profile_keys {
-            if !key.ends_with(".json") { continue; }
-            let data = match ops::get(&self.store, key).await {
-                Ok(d) => d,
-                Err(e) => { tracing::warn!("profile '{key}': read failed: {e}"); continue; }
-            };
-            match serde_json::from_slice::<ModelProfile>(&data) {
-                Ok(p) => { profiles.insert(p.id.clone(), p); }
-                Err(e) => tracing::warn!("profile '{key}': parse failed: {e}"),
-            }
-        }
-        if !profiles.is_empty() {
-            tracing::info!("catalog: {} model profiles loaded", profiles.len());
-        }
-
         Ok(count)
     }
 
-    // --- Phase 17: Model profiles ---
-
-    /// Create or replace a model profile. Validates id slug + ensures
-    /// every bound_dataset exists (as either a raw dataset or an AiView).
-    pub async fn put_profile(&self, profile: ModelProfile) -> Result<ModelProfile, String> {
-        if profile.id.is_empty() {
-            return Err("profile id is empty".into());
-        }
-        if !profile.id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') {
-            return Err(format!(
-                "profile id '{}' must be alphanumeric + '-' or '_' only", profile.id,
-            ));
-        }
-        for binding in &profile.bound_datasets {
-            let exists = self.get_by_name(binding).await.is_some()
-                || self.get_view(binding).await.is_some();
-            if !exists {
-                return Err(format!(
-                    "bound dataset '{}' not found as dataset or view", binding,
-                ));
-            }
-        }
-
-        let key = format!("{PROFILE_PREFIX}/{}.json", profile.id);
-        let json = serde_json::to_vec_pretty(&profile).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &key, json.into()).await?;
-
-        let mut profiles = self.profiles.write().await;
-        profiles.insert(profile.id.clone(), profile.clone());
-        tracing::info!(
-            "profile registered: {} -> ollama={} bindings={:?}",
-            profile.id, profile.ollama_name, profile.bound_datasets,
-        );
-        Ok(profile)
-    }
-
-    pub async fn get_profile(&self, id: &str) -> Option<ModelProfile> {
-        self.profiles.read().await.get(id).cloned()
-    }
-
-    pub async fn list_profiles(&self) -> Vec<ModelProfile> {
-        self.profiles.read().await.values().cloned().collect()
-    }
-
-    pub async fn delete_profile(&self, id: &str) -> Result<(), String> {
-        let key = format!("{PROFILE_PREFIX}/{id}.json");
-        ops::delete(&self.store, &key).await?;
-        self.profiles.write().await.remove(id);
-        Ok(())
-    }
-
-    /// Register a dataset. Idempotent on `name`:
-    ///   - No existing dataset by this name → create new manifest.
-    ///   - Exists with same schema fingerprint → update objects + `updated_at`
-    ///     in place (re-ingest of identical-shape data).
-    ///   - Exists with different schema fingerprint → reject as `Err`
-    ///     (callers map to HTTP 409 / gRPC FAILED_PRECONDITION). Schema
-    ///     evolution is not yet exposed as an HTTP endpoint; callers must
-    ///     register under a new name or delete the existing manifest first.
-    ///
-    /// Concurrency: the write lock is held across the storage write. That
-    /// serializes concurrent registers but is correctness-mandatory —
-    /// dropping the lock across the check→insert sequence creates a TOCTOU
-    /// window where two callers with the same name both see "no existing"
-    /// and both insert, reintroducing the duplicate-manifest bug this
-    /// function exists to prevent. Register is a metadata-only hop on the
-    /// ingest path, so the serialization cost is acceptable. Precedent:
-    /// `update_metadata` also holds the write lock across its I/O.
-    ///
-    /// Legacy state: if multiple manifests with the same name already exist
-    /// (from before this was idempotent), the most recently updated one is
-    /// treated as canonical. Run `dedupe_by_name` to clean up the rest.
+    /// Register a new dataset. Persists manifest to storage before updating memory.
     pub async fn register(
         &self,
         name: String,
         schema_fingerprint: SchemaFingerprint,
         objects: Vec<ObjectRef>,
     ) -> Result<DatasetManifest, String> {
-        let mut datasets = self.datasets.write().await;
-
-        let existing = datasets
-            .values()
-            .filter(|d| d.name == name)
-            .max_by_key(|d| d.updated_at)
-            .cloned();
-
-        if let Some(mut manifest) = existing {
-            if manifest.schema_fingerprint != schema_fingerprint {
-                return Err(format!(
-                    "dataset '{}' already exists with a different schema \
-                     (existing fingerprint: {}, new: {}). Schema-evolution \
-                     migration is not yet exposed as an HTTP endpoint; \
-                     register under a new name or delete the existing \
-                     manifest first.",
-                    name, manifest.schema_fingerprint.0, schema_fingerprint.0,
-                ));
-            }
-            manifest.objects = objects;
-            manifest.updated_at = chrono::Utc::now();
-
-            let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
-            let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
-            ops::put(&self.store, &manifest_key, json.into()).await?;
-
-            datasets.insert(manifest.id.clone(), manifest.clone());
-
-            tracing::info!("re-registered (idempotent): {} ({})", manifest.name, manifest.id);
-            return Ok(manifest);
-        }
-
         let now = chrono::Utc::now();
         let manifest = DatasetManifest {
             id: DatasetId::new(),
@@ -308,15 +78,14 @@ impl Registry {
             freshness: None,
             tags: vec![],
             row_count: None,
-            last_embedded_at: None,
-            embedding_stale_since: None,
-            embedding_refresh_policy: None,
         };
 
+        // Write-ahead: persist before in-memory update
         let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
         let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
         ops::put(&self.store, &manifest_key, json.into()).await?;
 
+        let mut datasets = self.datasets.write().await;
         datasets.insert(manifest.id.clone(), manifest.clone());
 
         tracing::info!("registered dataset: {} ({})", manifest.name, manifest.id);
@@ -342,7 +111,6 @@ impl Registry {
         if let Some(lineage) = updates.lineage { manifest.lineage = Some(lineage); }
         if let Some(freshness) = updates.freshness { manifest.freshness = Some(freshness); }
         if let Some(count) = updates.row_count { manifest.row_count = Some(count); }
-        if let Some(policy) = updates.embedding_refresh_policy { manifest.embedding_refresh_policy = Some(policy); }
         manifest.updated_at = chrono::Utc::now();
 
         // Persist
@@ -372,370 +140,6 @@ impl Registry {
         datasets.values().cloned().collect()
     }
 
-    /// Re-read the parquet footer(s) for a dataset and repopulate `row_count`
-    /// and `columns` from reality. Use this to repair manifests whose
-    /// metadata was lost (e.g. migrated from a pre-Phase 10 catalog).
-    ///
-    /// Does NOT touch owner/description/sensitivity/lineage/tags — only
-    /// the structural facts that parquet can tell us authoritatively.
-    /// The existing `schema_fingerprint` is updated if the recomputed one
-    /// differs; a warning is logged so drift is visible.
-    pub async fn resync_from_parquet(&self, name: &str) -> Result<DatasetManifest, String> {
-        use shared::arrow_helpers::{fingerprint_schema, parquet_to_record_batches};
-
-        // Snapshot the target manifest so we don't hold the write lock during IO.
-        let (id, objects, old_fp) = {
-            let datasets = self.datasets.read().await;
-            let m = datasets
-                .values()
-                .find(|d| d.name == name)
-                .ok_or_else(|| format!("dataset not found: {name}"))?;
-            (m.id.clone(), m.objects.clone(), m.schema_fingerprint.clone())
-        };
-
-        if objects.is_empty() {
-            return Err(format!("dataset '{name}' has no object references to resync from"));
-        }
-
-        let mut total_rows: u64 = 0;
-        let mut first_schema: Option<arrow::datatypes::SchemaRef> = None;
-        for obj in &objects {
-            let data = ops::get(&self.store, &obj.key).await
-                .map_err(|e| format!("read {}: {e}", obj.key))?;
-            let (schema, batches) = parquet_to_record_batches(&data)
-                .map_err(|e| format!("parse {}: {e}", obj.key))?;
-            let rows: u64 = batches.iter().map(|b| b.num_rows() as u64).sum();
-            total_rows += rows;
-            if first_schema.is_none() {
-                first_schema = Some(schema);
-            }
-        }
-
-        let schema = first_schema.ok_or("no schema recovered")?;
-        let new_fp = fingerprint_schema(&schema);
-        if new_fp != old_fp {
-            tracing::warn!(
-                "dataset '{}' schema fingerprint drift: {} -> {} (updating to match parquet reality)",
-                name, old_fp.0, new_fp.0,
-            );
-        }
-
-        let columns: Vec<ColumnMeta> = schema
-            .fields()
-            .iter()
-            .map(|f| ColumnMeta {
-                name: f.name().clone(),
-                data_type: f.data_type().to_string(),
-                sensitivity: None,
-                description: String::new(),
-                is_pii: false,
-            })
-            .collect();
-
-        // Apply updates.
-        let mut datasets = self.datasets.write().await;
-        let manifest = datasets
-            .get_mut(&id)
-            .ok_or_else(|| format!("dataset disappeared during resync: {name}"))?;
-        manifest.row_count = Some(total_rows);
-        manifest.columns = columns;
-        manifest.schema_fingerprint = new_fp;
-        manifest.updated_at = chrono::Utc::now();
-
-        // Persist.
-        let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
-        let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &manifest_key, json.into()).await?;
-
-        tracing::info!("resynced '{name}': row_count={total_rows}, {} columns", manifest.columns.len());
-        Ok(manifest.clone())
-    }
-
-    /// Collapse duplicate manifests that share a `name`. Winner per group:
-    ///   1. Prefer a manifest with a non-null `row_count` (already resynced).
-    ///   2. Break ties by newest `updated_at`.
-    /// Losers are removed from the in-memory registry AND their manifest
-    /// JSON is deleted from object storage. Datasets with a single manifest
-    /// are untouched. Parquet data files are never touched — only catalog
-    /// metadata. See `register` for the prevention side of this fix.
-    ///
-    /// Operator-only. Do not run while ingest is active: a concurrent
-    /// `register` between the snapshot and the delete sweep can create a
-    /// new manifest under a name we just deduped, producing a transient
-    /// count of 2 until the next dedupe run. Safe outside ingest windows.
-    pub async fn dedupe_by_name(&self) -> DedupeReport {
-        let groups: Vec<(String, Vec<DatasetManifest>)> = {
-            let datasets = self.datasets.read().await;
-            let mut by_name: HashMap<String, Vec<DatasetManifest>> = HashMap::new();
-            for m in datasets.values() {
-                by_name.entry(m.name.clone()).or_default().push(m.clone());
-            }
-            by_name.into_iter().filter(|(_, v)| v.len() > 1).collect()
-        };
-
-        let mut report = DedupeReport::default();
-        let mut to_delete: Vec<DatasetId> = Vec::new();
-
-        for (name, mut manifests) in groups {
-            report.groups += 1;
-            manifests.sort_by(|a, b| {
-                b.row_count.is_some().cmp(&a.row_count.is_some())
-                    .then(b.updated_at.cmp(&a.updated_at))
-            });
-            let winner = &manifests[0];
-            report.kept.push(DedupeKept {
-                name: name.clone(),
-                kept_id: winner.id.to_string(),
-                removed: manifests.len() - 1,
-            });
-            for loser in &manifests[1..] {
-                to_delete.push(loser.id.clone());
-            }
-        }
-
-        {
-            let mut datasets = self.datasets.write().await;
-            for id in &to_delete {
-                datasets.remove(id);
-            }
-        }
-        for id in &to_delete {
-            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
-            match ops::delete(&self.store, &key).await {
-                Ok(_) => report.removed += 1,
-                Err(e) => report.errors.push(DedupeError {
-                    manifest_id: id.to_string(),
-                    error: e,
-                }),
-            }
-        }
-
-        tracing::info!(
-            "dedupe: {} groups collapsed, {} manifests removed, {} errors",
-            report.groups, report.removed, report.errors.len(),
-        );
-        report
-    }
-
-    /// Resync every dataset that currently has a null row_count.
-    /// Returns (successes, failures) where each entry is (name, detail).
-    pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) {
-        let names: Vec<String> = {
-            let datasets = self.datasets.read().await;
-            datasets
-                .values()
-                .filter(|d| d.row_count.is_none() || d.columns.is_empty())
-                .map(|d| d.name.clone())
-                .collect()
-        };
-
-        let mut ok = Vec::new();
-        let mut err = Vec::new();
-        for name in names {
-            match self.resync_from_parquet(&name).await {
-                Ok(m) => ok.push((name, m.row_count.unwrap_or(0))),
-                Err(e) => err.push((name, e)),
-            }
-        }
-        (ok, err)
-    }
-
-    /// Mark a dataset's embeddings as stale (row-level data has been written
-    /// since the last embedding refresh). Idempotent — setting stale when
-    /// already stale is a no-op. Only marks stale if the dataset has been
-    /// embedded before — a never-embedded dataset doesn't need a stale flag
-    /// (it just needs an initial index build). Called from the ingest path.
-    pub async fn mark_embeddings_stale(&self, name: &str) -> Result<(), String> {
-        let mut datasets = self.datasets.write().await;
-        let manifest = datasets
-            .values_mut()
-            .find(|d| d.name == name)
-            .ok_or_else(|| format!("dataset not found: {name}"))?;
-
-        if manifest.last_embedded_at.is_none() {
-            return Ok(()); // never embedded -> no stale semantics yet
-        }
-        if manifest.embedding_stale_since.is_none() {
-            manifest.embedding_stale_since = Some(chrono::Utc::now());
-            manifest.updated_at = chrono::Utc::now();
-
-            let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
-            let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
-            ops::put(&self.store, &key, json.into()).await?;
-            tracing::info!("marked embeddings stale for dataset '{name}'");
-        }
-        Ok(())
-    }
-
-    /// Clear the stale marker and set `last_embedded_at = now`.
-    /// Called by the embedding refresh pipeline once it finishes.
-    pub async fn clear_embeddings_stale(&self, name: &str) -> Result<(), String> {
-        let mut datasets = self.datasets.write().await;
-        let manifest = datasets
-            .values_mut()
-            .find(|d| d.name == name)
-            .ok_or_else(|| format!("dataset not found: {name}"))?;
-
-        let now = chrono::Utc::now();
-        manifest.embedding_stale_since = None;
-        manifest.last_embedded_at = Some(now);
-        manifest.updated_at = now;
-
-        let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
-        let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &key, json.into()).await?;
-        Ok(())
-    }
-
-    /// Federation layer 2: stamp `bucket = "primary"` on every ObjectRef
-    /// whose `bucket` field is empty or matches a legacy value (`"data"`,
-    /// `"local"`). One-shot migration; re-running is a safe no-op once
-    /// every ref is canonical.
-    pub async fn migrate_buckets_to_primary(&self) -> Result<MigrateBucketsReport, String> {
-        let mut datasets = self.datasets.write().await;
-        let mut report = MigrateBucketsReport::default();
-        let mut to_persist: Vec<(DatasetId, DatasetManifest)> = Vec::new();
-
-        for manifest in datasets.values_mut() {
-            let mut changed = false;
-            for obj in manifest.objects.iter_mut() {
-                report.refs_examined += 1;
-                if obj.bucket.is_empty() {
-                    obj.bucket = "primary".to_string();
-                    report.refs_stamped += 1;
-                    changed = true;
-                } else if obj.bucket == "data" || obj.bucket == "local" {
-                    obj.bucket = "primary".to_string();
-                    report.refs_renamed += 1;
-                    changed = true;
-                } else {
-                    report.refs_unchanged += 1;
-                }
-            }
-            if changed {
-                manifest.updated_at = chrono::Utc::now();
-                to_persist.push((manifest.id.clone(), manifest.clone()));
-            }
-        }
-
-        // Persist updated manifests after we've finished mutating the map.
-        for (id, manifest) in to_persist {
-            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
-            let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
-            ops::put(&self.store, &key, json.into()).await?;
-            report.manifests_persisted += 1;
-        }
-
-        tracing::info!(
-            "bucket migration: examined {} refs, renamed {}, stamped {}, unchanged {}, persisted {} manifests",
-            report.refs_examined,
-            report.refs_renamed,
-            report.refs_stamped,
-            report.refs_unchanged,
-            report.manifests_persisted,
-        );
-        Ok(report)
-    }
-
-    // --- Phase D: AI-safe views ---
-
-    /// Create or replace a named view. Validates that the base dataset
-    /// exists and that the column whitelist is non-empty. Persists to
-    /// `_catalog/views/{name}.json`.
-    pub async fn put_view(&self, mut view: AiView) -> Result<AiView, String> {
-        if view.name.is_empty() {
-            return Err("view name is empty".into());
-        }
-        if view.columns.is_empty() {
-            return Err("view must whitelist at least one column".into());
-        }
-        // Base dataset must exist (read-side will fail anyway, fail-fast here
-        // so operators don't end up with dangling views).
-        if self.get_by_name(&view.base_dataset).await.is_none() {
-            return Err(format!(
-                "base dataset '{}' not found in catalog",
-                view.base_dataset
-            ));
-        }
-        if view.created_at.timestamp() == 0 {
-            view.created_at = chrono::Utc::now();
-        }
-
-        let key = format!("{VIEW_PREFIX}/{}.json", sanitize_view_name(&view.name));
-        let json = serde_json::to_vec_pretty(&view).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &key, json.into()).await?;
-
-        let mut views = self.views.write().await;
-        views.insert(view.name.clone(), view.clone());
-        tracing::info!("view registered: {} over '{}'", view.name, view.base_dataset);
-        Ok(view)
-    }
-
-    pub async fn get_view(&self, name: &str) -> Option<AiView> {
-        self.views.read().await.get(name).cloned()
-    }
-
-    pub async fn list_views(&self) -> Vec<AiView> {
-        self.views.read().await.values().cloned().collect()
-    }
-
-    pub async fn delete_view(&self, name: &str) -> Result<(), String> {
-        let key = format!("{VIEW_PREFIX}/{}.json", sanitize_view_name(name));
-        ops::delete(&self.store, &key).await?;
-        self.views.write().await.remove(name);
-        Ok(())
-    }
-
-    /// Remove a dataset from the catalog by name. Deletes the manifest
-    /// from both the in-memory registry and object storage.
-    ///
-    /// Scope is metadata-only: the underlying parquet files, vector
-    /// indexes, tombstones, trial journals, and AiViews that reference
-    /// this dataset are NOT touched. Caller is responsible for any
-    /// cascade cleanup. This mirrors how a DROP TABLE in a typical
-    /// warehouse separates "forget about this dataset" from "actually
-    /// reclaim the bytes".
-    ///
-    /// Returns Ok(()) when the dataset existed and was removed; an Err
-    /// with "dataset not found" when no manifest by that name exists.
-    /// Legacy state with duplicate names: all matching manifests are
-    /// removed (effectively a post-hoc dedupe on that name).
-    pub async fn delete_dataset(&self, name: &str) -> Result<usize, String> {
-        let ids_to_remove: Vec<DatasetId> = {
-            let datasets = self.datasets.read().await;
-            datasets.values().filter(|d| d.name == name).map(|d| d.id.clone()).collect()
-        };
-        if ids_to_remove.is_empty() {
-            return Err(format!("dataset not found: {name}"));
-        }
-
-        for id in &ids_to_remove {
-            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
-            if let Err(e) = ops::delete(&self.store, &key).await {
-                // Storage delete failed — log and keep going. The in-memory
-                // remove below is still correct; on restart, the missing
-                // storage object just means nothing to rehydrate.
-                tracing::warn!("delete_dataset '{name}': storage delete of {key} failed: {e}");
-            }
-        }
-        let mut datasets = self.datasets.write().await;
-        for id in &ids_to_remove {
-            datasets.remove(id);
-        }
-        tracing::info!("deleted dataset '{}' ({} manifest(s))", name, ids_to_remove.len());
-        Ok(ids_to_remove.len())
-    }
-
-    /// List datasets whose `embedding_stale_since` is set — they need a refresh.
-    pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
-        let datasets = self.datasets.read().await;
-        datasets
-            .values()
-            .filter(|d| d.embedding_stale_since.is_some())
-            .cloned()
-            .collect()
-    }
-
     /// Add objects to an existing dataset.
     pub async fn add_objects(
         &self,
@@ -756,275 +160,3 @@ impl Registry {
         Ok(manifest.clone())
     }
 }
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use object_store::memory::InMemory;
-
-    fn fixture() -> Registry {
-        Registry::new(Arc::new(InMemory::new()))
-    }
-
-    fn fp(s: &str) -> SchemaFingerprint {
-        SchemaFingerprint(s.to_string())
-    }
-
-    fn obj(key: &str) -> ObjectRef {
-        ObjectRef {
-            bucket: "primary".to_string(),
-            key: key.to_string(),
-            size_bytes: 100,
-            created_at: chrono::Utc::now(),
-        }
-    }
-
-    #[tokio::test]
-    async fn fresh_register_creates_new_manifest() {
-        let reg = fixture();
-        let m = reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
-        assert_eq!(m.name, "a");
-        assert_eq!(m.schema_fingerprint, fp("f1"));
-        assert_eq!(reg.list().await.len(), 1);
-        assert_eq!(reg.get_by_name("a").await.unwrap().id, m.id);
-    }
-
-    #[tokio::test]
-    async fn re_register_same_fingerprint_is_idempotent() {
-        let reg = fixture();
-        let first = reg.register("a".into(), fp("f1"), vec![obj("old.parquet")]).await.unwrap();
-        let second = reg.register("a".into(), fp("f1"), vec![obj("new.parquet")]).await.unwrap();
-
-        assert_eq!(first.id, second.id, "same name+fp should reuse ID");
-        assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
-        let fetched = reg.get_by_name("a").await.unwrap();
-        assert_eq!(fetched.objects[0].key, "new.parquet", "objects replaced");
-        assert!(fetched.updated_at >= first.updated_at, "updated_at bumped");
-    }
-
-    #[tokio::test]
-    async fn re_register_different_fingerprint_rejects() {
-        let reg = fixture();
-        reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
-        let err = reg.register("a".into(), fp("f2"), vec![obj("a.parquet")]).await.unwrap_err();
-
-        assert!(err.contains("different schema"), "error mentions schema drift: {err}");
-        assert_eq!(reg.list().await.len(), 1, "failed register did not mutate state");
-        assert_eq!(reg.get_by_name("a").await.unwrap().schema_fingerprint, fp("f1"));
-    }
-
-    #[tokio::test]
-    async fn delete_dataset_removes_manifest_and_in_memory_entry() {
-        let reg = fixture();
-        reg.register("to_delete".into(), fp("f1"), vec![obj("d.parquet")]).await.unwrap();
-        reg.register("keepme".into(),    fp("f2"), vec![obj("k.parquet")]).await.unwrap();
-        assert_eq!(reg.list().await.len(), 2);
-
-        let removed = reg.delete_dataset("to_delete").await.unwrap();
-        assert_eq!(removed, 1, "one manifest removed");
-        assert!(reg.get_by_name("to_delete").await.is_none());
-        assert!(reg.get_by_name("keepme").await.is_some(), "unrelated dataset untouched");
-        assert_eq!(reg.list().await.len(), 1);
-    }
-
-    #[tokio::test]
-    async fn delete_dataset_404s_on_unknown_name() {
-        let reg = fixture();
-        let err = reg.delete_dataset("nope").await.unwrap_err();
-        assert!(err.starts_with("dataset not found"), "error wording: {err}");
-    }
-
-    #[tokio::test]
-    async fn delete_dataset_removes_all_dupes_with_matching_name() {
-        // Legacy zombies: direct insert so we can seed duplicates,
-        // mirroring pre-idempotent-register state. delete_dataset should
-        // sweep every manifest sharing that name in a single call.
-        let reg = fixture();
-        let now = chrono::Utc::now();
-        for _ in 0..3 {
-            let m = DatasetManifest {
-                id: DatasetId::new(),
-                name: "zombie".into(),
-                schema_fingerprint: fp("fp"),
-                objects: vec![obj("z.parquet")],
-                created_at: now, updated_at: now,
-                row_count: None, description: String::new(), owner: String::new(),
-                sensitivity: None, columns: vec![], lineage: None, freshness: None,
-                tags: vec![], last_embedded_at: None, embedding_stale_since: None,
-                embedding_refresh_policy: None,
-            };
-            reg.datasets.write().await.insert(m.id.clone(), m);
-        }
-        assert_eq!(reg.list().await.len(), 3);
-
-        let removed = reg.delete_dataset("zombie").await.unwrap();
-        assert_eq!(removed, 3, "all three manifests removed in one sweep");
-        assert_eq!(reg.list().await.len(), 0);
-    }
-
-    #[tokio::test]
-    async fn dedupe_no_duplicates_is_noop() {
-        let reg = fixture();
-        reg.register("a".into(), fp("f1"), vec![obj("a.parquet")]).await.unwrap();
-        reg.register("b".into(), fp("f2"), vec![obj("b.parquet")]).await.unwrap();
-
-        let report = reg.dedupe_by_name().await;
-        assert_eq!(report.groups, 0);
-        assert_eq!(report.removed, 0);
-        assert_eq!(reg.list().await.len(), 2);
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 8)]
-    async fn concurrent_register_stress_many_workers_one_manifest() {
-        // Stress variant: fire 32 concurrent registers for the same new
-        // name across 8 worker threads. Single-manifest invariant must
-        // hold regardless of how the scheduler interleaves them. This
-        // catches race regressions the 2-call variant below might miss
-        // under a forgiving scheduler.
-        let reg = fixture();
-        let mut handles = Vec::new();
-        for i in 0..32 {
-            let r = reg.clone();
-            handles.push(tokio::spawn(async move {
-                r.register(
-                    "stress".into(),
-                    fp("stress-fp"),
-                    vec![obj(&format!("{i}.parquet"))],
-                ).await
-            }));
-        }
-        for h in handles {
-            h.await.unwrap().expect("each register succeeds (idempotent)");
-        }
-        let all = reg.list().await;
-        let stress_manifests: Vec<_> = all.iter().filter(|d| d.name == "stress").collect();
-        assert_eq!(stress_manifests.len(), 1,
-            "32 concurrent registers produced {} manifests for 'stress' — expected 1",
-            stress_manifests.len());
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    async fn concurrent_register_same_new_name_collapses_to_one_manifest() {
-        // Pins the TOCTOU fix: without the write-lock held across the
-        // check→insert sequence, two parallel registers of the same
-        // previously-unknown name would both see "no existing" and both
-        // insert. Multi-thread flavor is required to actually exercise
-        // the race — under single-threaded tokio the scheduler doesn't
-        // preempt between awaits on the same task's critical section.
-        // Under the current impl, whichever acquires the write lock
-        // first creates the manifest; the second observes it and takes
-        // the idempotent-update branch.
-        let reg = fixture();
-        let a = reg.clone();
-        let b = reg.clone();
-
-        let (ra, rb) = tokio::join!(
-            async move {
-                a.register("race".into(), fp("shared-fp"), vec![obj("a.parquet")]).await
-            },
-            async move {
-                b.register("race".into(), fp("shared-fp"), vec![obj("b.parquet")]).await
-            },
-        );
-
-        let m_a = ra.expect("first register succeeds");
-        let m_b = rb.expect("second register succeeds (same fp = idempotent)");
-        assert_eq!(m_a.id, m_b.id, "both calls resolve to a single DatasetId");
-        assert_eq!(reg.list().await.len(), 1, "no duplicate manifest created");
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    async fn concurrent_register_same_name_different_fp_one_wins_one_rejects() {
-        // Reverse case: concurrent registers with conflicting fingerprints.
-        // Deterministic outcome: one call's fingerprint gets persisted as
-        // canonical, the other is rejected on seeing the now-established
-        // fingerprint. Which one wins is scheduler-dependent — we only
-        // assert exactly one success and exactly one rejection.
-        let reg = fixture();
-        let a = reg.clone();
-        let b = reg.clone();
-
-        let (ra, rb) = tokio::join!(
-            async move { a.register("race".into(), fp("fp1"), vec![obj("a.parquet")]).await },
-            async move { b.register("race".into(), fp("fp2"), vec![obj("b.parquet")]).await },
-        );
-
-        let successes = [&ra, &rb].iter().filter(|r| r.is_ok()).count();
-        let rejections = [&ra, &rb].iter().filter(|r| r.is_err()).count();
-        assert_eq!(successes, 1, "exactly one register wins");
-        assert_eq!(rejections, 1, "the other is rejected on fingerprint");
-        let rejection_msg = [&ra, &rb].iter()
-            .find_map(|r| r.as_ref().err())
-            .unwrap();
-        assert!(rejection_msg.contains("different schema"));
-        assert_eq!(reg.list().await.len(), 1);
-    }
-
-    #[tokio::test]
-    async fn dedupe_collapses_dupes_preferring_non_null_row_count() {
-        let reg = fixture();
-
-        // Simulate legacy pre-idempotent-register state by inserting
-        // manifests with the same `name` directly. Mirrors what happened
-        // with the 308× successful_playbooks duplication.
-        let now = chrono::Utc::now();
-        let old_with_rows = DatasetManifest {
-            id: DatasetId::new(),
-            name: "dupes".into(),
-            schema_fingerprint: fp("f1"),
-            objects: vec![obj("dupes.parquet")],
-            created_at: now - chrono::Duration::hours(2),
-            updated_at: now - chrono::Duration::hours(2),
-            row_count: Some(42),
-            description: String::new(),
-            owner: String::new(),
-            sensitivity: None,
-            columns: vec![],
-            lineage: None,
-            freshness: None,
-            tags: vec![],
-            last_embedded_at: None,
-            embedding_stale_since: None,
-            embedding_refresh_policy: None,
-        };
-        let newer_no_rows = DatasetManifest {
-            id: DatasetId::new(),
-            updated_at: now, // newer, but null row_count
-            row_count: None,
-            ..old_with_rows.clone()
-        };
-        let oldest = DatasetManifest {
-            id: DatasetId::new(),
-            updated_at: now - chrono::Duration::hours(5),
-            row_count: None,
-            ..old_with_rows.clone()
-        };
-
-        let kept_id = old_with_rows.id.clone();
-        let loser_a = newer_no_rows.id.clone();
-        let loser_b = oldest.id.clone();
-
-        {
-            let mut d = reg.datasets.write().await;
-            d.insert(old_with_rows.id.clone(), old_with_rows);
-            d.insert(newer_no_rows.id.clone(), newer_no_rows);
-            d.insert(oldest.id.clone(), oldest);
-        }
-        assert_eq!(reg.list().await.len(), 3);
-
-        let report = reg.dedupe_by_name().await;
-        assert_eq!(report.groups, 1);
-        assert_eq!(report.removed, 2);
-        assert_eq!(report.errors.len(), 0);
-        assert_eq!(report.kept.len(), 1);
-        assert_eq!(report.kept[0].name, "dupes");
-        assert_eq!(report.kept[0].kept_id, kept_id.to_string(),
-            "should keep the manifest with non-null row_count");
-
-        let remaining = reg.list().await;
-        assert_eq!(remaining.len(), 1);
-        assert_eq!(remaining[0].id, kept_id);
-        assert!(reg.get(&loser_a).await.is_none());
-        assert!(reg.get(&loser_b).await.is_none());
-    }
-}

crates/catalogd/src/service.rs

@@ -17,20 +17,8 @@ pub fn router(registry: Registry) -> Router {
         .route("/datasets", post(create_dataset))
         .route("/datasets", get(list_datasets))
         .route("/datasets/{id}", get(get_dataset))
-        .route("/datasets/by-name/{name}", get(get_dataset_by_name).delete(delete_dataset_by_name))
+        .route("/datasets/by-name/{name}", get(get_dataset_by_name))
         .route("/datasets/by-name/{name}/metadata", post(update_metadata))
-        .route("/datasets/by-name/{name}/resync", post(resync_dataset))
-        .route("/resync-missing", post(resync_all_missing))
-        .route("/migrate-buckets", post(migrate_buckets))
-        .route("/dedupe", post(dedupe_by_name))
-        // Phase D: AI-safe views
-        .route("/views", post(create_view).get(list_views))
-        .route("/views/{name}", get(get_view).delete(delete_view))
-        // Phase E: soft-delete tombstones
-        .route("/datasets/by-name/{name}/tombstone", post(tombstone_rows).get(list_tombstones))
-        // Phase 17: model profiles
-        .route("/profiles", post(create_profile).get(list_profiles))
-        .route("/profiles/{id}", get(get_profile).delete(delete_profile))
         .with_state(registry)
 }
 
@@ -122,14 +110,7 @@ async fn create_dataset(
             let resp = DatasetResponse::from(&manifest);
             Ok((StatusCode::CREATED, Json(resp)))
         }
-        Err(e) => {
-            let status = if e.contains("different schema") {
-                StatusCode::CONFLICT
-            } else {
-                StatusCode::INTERNAL_SERVER_ERROR
-            };
-            Err((status, e))
-        }
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
     }
 }
 
@@ -161,25 +142,6 @@ async fn get_dataset_by_name(
     }
 }
 
-/// Remove a dataset manifest by name. Metadata only — parquet files,
-/// vector indexes, and tombstones are NOT cascade-deleted. See
-/// `Registry::delete_dataset` for the full scope.
-async fn delete_dataset_by_name(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match registry.delete_dataset(&name).await {
-        Ok(removed) => {
-            let body = serde_json::json!({ "name": name, "manifests_removed": removed });
-            Ok(Json(body))
-        }
-        Err(e) if e.starts_with("dataset not found") => {
-            Err((StatusCode::NOT_FOUND, e))
-        }
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
 async fn update_metadata(
     State(registry): State<Registry>,
     Path(name): Path<String>,
@@ -190,260 +152,3 @@ async fn update_metadata(
         Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
     }
 }
-
-/// Re-read parquet footers for a single dataset and repopulate row_count
-/// and columns from reality. Useful for repairing manifests whose metadata
-/// was lost or never backfilled.
-async fn resync_dataset(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match registry.resync_from_parquet(&name).await {
-        Ok(manifest) => Ok(Json(DatasetResponse::from(&manifest))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-#[derive(Serialize)]
-struct ResyncAllResponse {
-    succeeded: Vec<ResyncOk>,
-    failed: Vec<ResyncErr>,
-}
-
-#[derive(Serialize)]
-struct ResyncOk {
-    name: String,
-    row_count: u64,
-}
-
-#[derive(Serialize)]
-struct ResyncErr {
-    name: String,
-    error: String,
-}
-
-/// Resync every dataset that currently has null row_count or empty columns.
-async fn resync_all_missing(State(registry): State<Registry>) -> impl IntoResponse {
-    let (ok, err) = registry.resync_missing().await;
-    Json(ResyncAllResponse {
-        succeeded: ok.into_iter().map(|(name, row_count)| ResyncOk { name, row_count }).collect(),
-        failed: err.into_iter().map(|(name, error)| ResyncErr { name, error }).collect(),
-    })
-}
-
-/// Federation layer 2 one-shot: normalize every ObjectRef.bucket field
-/// to the canonical "primary" value. Idempotent — re-running once
-/// everything is canonical is a safe no-op.
-async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse {
-    match registry.migrate_buckets_to_primary().await {
-        Ok(report) => Ok(Json(report)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-/// Collapse duplicate manifests by `name` — see `Registry::dedupe_by_name`.
-/// Safe to run repeatedly; single-manifest datasets are untouched.
-async fn dedupe_by_name(State(registry): State<Registry>) -> impl IntoResponse {
-    Json(registry.dedupe_by_name().await)
-}
-
-// --- Phase D: AI-safe views ---
-
-#[derive(Deserialize)]
-struct CreateViewRequest {
-    name: String,
-    base_dataset: String,
-    columns: Vec<String>,
-    #[serde(default)]
-    row_filter: Option<String>,
-    #[serde(default)]
-    column_redactions: std::collections::HashMap<String, shared::types::Redaction>,
-    #[serde(default)]
-    description: String,
-    #[serde(default)]
-    created_by: String,
-}
-
-async fn create_view(
-    State(registry): State<Registry>,
-    Json(req): Json<CreateViewRequest>,
-) -> impl IntoResponse {
-    let view = shared::types::AiView {
-        name: req.name,
-        base_dataset: req.base_dataset,
-        columns: req.columns,
-        row_filter: req.row_filter,
-        column_redactions: req.column_redactions,
-        created_at: chrono::Utc::now(),
-        created_by: req.created_by,
-        description: req.description,
-    };
-    match registry.put_view(view).await {
-        Ok(v) => Ok((StatusCode::CREATED, Json(v))),
-        Err(e) => Err((StatusCode::BAD_REQUEST, e)),
-    }
-}
-
-async fn list_views(State(registry): State<Registry>) -> impl IntoResponse {
-    Json(registry.list_views().await)
-}
-
-async fn get_view(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match registry.get_view(&name).await {
-        Some(v) => Ok(Json(v)),
-        None => Err((StatusCode::NOT_FOUND, format!("view not found: {name}"))),
-    }
-}
-
-async fn delete_view(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match registry.delete_view(&name).await {
-        Ok(()) => Ok(StatusCode::NO_CONTENT),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-// --- Phase E: soft-delete tombstones ---
-
-#[derive(Deserialize)]
-struct TombstoneRequest {
-    row_key_column: String,
-    row_key_values: Vec<String>,
-    #[serde(default)]
-    actor: String,
-    #[serde(default)]
-    reason: String,
-}
-
-#[derive(Serialize)]
-struct TombstoneResponse {
-    dataset: String,
-    row_key_column: String,
-    rows_tombstoned: usize,
-    failures: Vec<String>,
-}
-
-async fn tombstone_rows(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-    Json(req): Json<TombstoneRequest>,
-) -> impl IntoResponse {
-    if req.row_key_values.is_empty() {
-        return Err((StatusCode::BAD_REQUEST, "row_key_values is empty".to_string()));
-    }
-
-    let mut ok = 0;
-    let mut failures = Vec::new();
-    for value in &req.row_key_values {
-        match registry
-            .add_tombstone(&name, &req.row_key_column, value, &req.actor, &req.reason)
-            .await
-        {
-            Ok(_) => ok += 1,
-            Err(e) => failures.push(format!("{value}: {e}")),
-        }
-    }
-
-    let status = if ok > 0 && failures.is_empty() {
-        StatusCode::CREATED
-    } else if ok > 0 {
-        StatusCode::MULTI_STATUS
-    } else {
-        StatusCode::BAD_REQUEST
-    };
-
-    Ok((status, Json(TombstoneResponse {
-        dataset: name,
-        row_key_column: req.row_key_column,
-        rows_tombstoned: ok,
-        failures,
-    })))
-}
-
-async fn list_tombstones(
-    State(registry): State<Registry>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match registry.list_tombstones(&name).await {
-        Ok(ts) => Ok(Json(ts)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-// --- Phase 17: Model profiles ---
-
-#[derive(Deserialize)]
-struct CreateProfileRequest {
-    id: String,
-    ollama_name: String,
-    #[serde(default)]
-    description: String,
-    bound_datasets: Vec<String>,
-    #[serde(default)]
-    hnsw_config: shared::types::ProfileHnswConfig,
-    #[serde(default = "default_embed_model_req")]
-    embed_model: String,
-    #[serde(default)]
-    created_by: String,
-    /// Federation: optional per-profile bucket (`profile:{id}` by convention).
-    /// Omitting keeps artifacts in primary.
-    #[serde(default)]
-    bucket: Option<String>,
-    /// ADR-019 hybrid: which vector backend to route this profile's
-    /// indexes to. Defaults to Parquet+HNSW.
-    #[serde(default)]
-    vector_backend: shared::types::VectorBackend,
-}
-
-fn default_embed_model_req() -> String { "nomic-embed-text".to_string() }
-
-async fn create_profile(
-    State(registry): State<Registry>,
-    Json(req): Json<CreateProfileRequest>,
-) -> impl IntoResponse {
-    let profile = shared::types::ModelProfile {
-        id: req.id,
-        ollama_name: req.ollama_name,
-        description: req.description,
-        bound_datasets: req.bound_datasets,
-        hnsw_config: req.hnsw_config,
-        embed_model: req.embed_model,
-        created_at: chrono::Utc::now(),
-        created_by: req.created_by,
-        bucket: req.bucket,
-        vector_backend: req.vector_backend,
-    };
-    match registry.put_profile(profile).await {
-        Ok(p) => Ok((StatusCode::CREATED, Json(p))),
-        Err(e) => Err((StatusCode::BAD_REQUEST, e)),
-    }
-}
-
-async fn list_profiles(State(registry): State<Registry>) -> impl IntoResponse {
-    Json(registry.list_profiles().await)
-}
-
-async fn get_profile(
-    State(registry): State<Registry>,
-    Path(id): Path<String>,
-) -> impl IntoResponse {
-    match registry.get_profile(&id).await {
-        Some(p) => Ok(Json(p)),
-        None => Err((StatusCode::NOT_FOUND, format!("profile not found: {id}"))),
-    }
-}
-
-async fn delete_profile(
-    State(registry): State<Registry>,
-    Path(id): Path<String>,
-) -> impl IntoResponse {
-    match registry.delete_profile(&id).await {
-        Ok(()) => Ok(StatusCode::NO_CONTENT),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}

crates/catalogd/src/tombstones.rs

@@ -1,151 +0,0 @@
-//! Soft-delete tombstone storage (Phase E).
-//!
-//! One append-log per dataset at `_catalog/tombstones/{dataset}/batch_*.jsonl`.
-//! Uses the shared `storaged::append_log::AppendLog` pattern so appends
-//! are write-once (never rewrites existing files) and can be compacted.
-//!
-//! The store exposes a per-dataset cache of active tombstones for the
-//! hot path (queryd filter construction) — that's why events are pulled
-//! into an in-memory map on every call rather than scanning object
-//! storage repeatedly.
-
-use object_store::ObjectStore;
-use shared::types::Tombstone;
-use std::collections::HashMap;
-use std::sync::Arc;
-use storaged::append_log::{AppendLog, CompactStats};
-use tokio::sync::RwLock;
-
-const TOMBSTONE_PREFIX: &str = "_catalog/tombstones";
-
-#[derive(Clone)]
-pub struct TombstoneStore {
-    store: Arc<dyn ObjectStore>,
-    logs: Arc<RwLock<HashMap<String, Arc<AppendLog>>>>,
-}
-
-impl TombstoneStore {
-    pub fn new(store: Arc<dyn ObjectStore>) -> Self {
-        Self {
-            store,
-            logs: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    fn prefix_for(dataset: &str) -> String {
-        // Sanitize dataset name for filesystem safety.
-        let safe: String = dataset
-            .chars()
-            .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
-            .collect();
-        format!("{TOMBSTONE_PREFIX}/{}", safe)
-    }
-
-    async fn log_for(&self, dataset: &str) -> Arc<AppendLog> {
-        if let Some(log) = self.logs.read().await.get(dataset) {
-            return log.clone();
-        }
-        let mut guard = self.logs.write().await;
-        if let Some(log) = guard.get(dataset) {
-            return log.clone();
-        }
-        // Threshold of 1 — every tombstone is high-value. Compliance/audit
-        // doesn't tolerate "lost on restart"; we trade a small file count
-        // for guaranteed durability. Compaction merges later if volume grows.
-        let log = Arc::new(
-            AppendLog::new(self.store.clone(), Self::prefix_for(dataset))
-                .with_flush_threshold(1),
-        );
-        guard.insert(dataset.to_string(), log.clone());
-        log
-    }
-
-    /// Append one tombstone. Validates that the `row_key_column` matches
-    /// the column already used for this dataset (all tombstones for a
-    /// dataset share one key column so the query filter is well-defined).
-    /// Forces a flush so the tombstone is durable before this call returns.
-    pub async fn append(&self, ts: &Tombstone) -> Result<(), String> {
-        let existing = self.list(&ts.dataset).await?;
-        if let Some(prior) = existing.first() {
-            if prior.row_key_column != ts.row_key_column {
-                return Err(format!(
-                    "dataset '{}' already uses '{}' as tombstone key; cannot mix with '{}'",
-                    ts.dataset, prior.row_key_column, ts.row_key_column,
-                ));
-            }
-        }
-        let line = serde_json::to_vec(ts).map_err(|e| e.to_string())?;
-        let log = self.log_for(&ts.dataset).await;
-        log.append(line).await?;
-        // Belt-and-suspenders: explicit flush in case the threshold is
-        // ever raised. Tombstones must be durable on return.
-        log.flush().await
-    }
-
-    /// All tombstones for a dataset (chronological).
-    pub async fn list(&self, dataset: &str) -> Result<Vec<Tombstone>, String> {
-        let log = self.log_for(dataset).await;
-        let lines = log.read_all().await?;
-        let mut out = Vec::with_capacity(lines.len());
-        for line in lines {
-            match serde_json::from_slice::<Tombstone>(&line) {
-                Ok(t) => out.push(t),
-                Err(e) => tracing::warn!("tombstones/{}: skip malformed entry: {e}", dataset),
-            }
-        }
-        Ok(out)
-    }
-
-    /// Per-dataset grouped view used by queryd — returns a map of
-    /// `{dataset -> (row_key_column, set_of_values)}` for every dataset
-    /// that has any tombstones.
-    pub async fn all_grouped(
-        &self,
-        datasets: &[String],
-    ) -> Result<HashMap<String, (String, Vec<String>)>, String> {
-        let mut grouped = HashMap::new();
-        for dataset in datasets {
-            let ts = match self.list(dataset).await {
-                Ok(ts) => ts,
-                Err(_) => continue,
-            };
-            if ts.is_empty() { continue; }
-            let col = ts[0].row_key_column.clone();
-            let values: Vec<String> = ts.iter().map(|t| t.row_key_value.clone()).collect();
-            grouped.insert(dataset.clone(), (col, values));
-        }
-        Ok(grouped)
-    }
-
-    pub async fn compact(&self, dataset: &str) -> Result<CompactStats, String> {
-        let log = self.log_for(dataset).await;
-        log.compact().await
-    }
-
-    /// Remove every tombstone for a dataset. Called after a successful
-    /// parquet compaction has physically deleted those rows — the
-    /// tombstones have done their job and the journal can be cleared.
-    ///
-    /// Implementation: drop the per-dataset AppendLog from the cache and
-    /// delete all its batch files. Next write starts fresh.
-    pub async fn clear(&self, dataset: &str) -> Result<usize, String> {
-        let prefix = format!("{}/", Self::prefix_for(dataset));
-        let keys = storaged::ops::list(&self.store, Some(&prefix)).await?;
-        let matching: Vec<String> = keys
-            .into_iter()
-            .filter(|k| {
-                let basename = k.rsplit('/').next().unwrap_or(k);
-                basename.starts_with("batch_") && basename.ends_with(".jsonl")
-            })
-            .collect();
-        let count = matching.len();
-        for key in &matching {
-            let _ = storaged::ops::delete(&self.store, key).await;
-        }
-        self.logs.write().await.remove(dataset);
-        if count > 0 {
-            tracing::info!("cleared {count} tombstone batch files for '{}'", dataset);
-        }
-        Ok(count)
-    }
-}

crates/gateway/Cargo.toml

@@ -28,4 +28,3 @@ opentelemetry-stdout = { workspace = true }
 tracing-opentelemetry = { workspace = true }
 arrow = { workspace = true }
 chrono = { workspace = true }
-reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

crates/gateway/src/main.rs

@@ -3,7 +3,6 @@ mod access_service;
 mod auth;
 mod observability;
 mod tools;
-mod v1;
 
 use axum::{Router, extract::DefaultBodyLimit, routing::get};
 use proto::lakehouse::catalog_service_server::CatalogServiceServer;
@@ -25,30 +24,8 @@ async fn main() {
     tracing::info!("config loaded: gateway={}:{}, storage={}",
         config.gateway.host, config.gateway.port, config.storage.root);
 
-    // Secrets provider
-    let secrets = std::sync::Arc::new(shared::secrets::FileSecretsProvider::from_env());
-    if let Err(e) = secrets.load().await {
-        tracing::warn!("secrets load: {e} — bucket credentials may be missing");
-    }
-
-    // Federation: bucket registry (primary + any [[storage.buckets]] entries + rescue)
-    let bucket_registry = std::sync::Arc::new(
-        storaged::registry::BucketRegistry::from_config(&config.storage, secrets.clone())
-            .await
-            .expect("bucket registry init failed")
-    );
-    let buckets = bucket_registry.list().await;
-    tracing::info!(
-        "bucket registry: {} buckets [{}], rescue={:?}",
-        buckets.len(),
-        buckets.iter().map(|b| format!("{}:{}", b.name, b.backend)).collect::<Vec<_>>().join(", "),
-        bucket_registry.rescue_name(),
-    );
-
-    // Back-compat: existing components that expect a single Arc<ObjectStore>
-    // get the primary bucket's store. As we migrate call sites to be bucket-
-    // aware, they'll call bucket_registry.get(name) directly.
-    let store = bucket_registry.default_store();
+    // Storage backend
+    let store = storaged::backend::init_local(&config.storage.root);
 
     // Catalog
     let registry = catalogd::registry::Registry::new(store.clone());
@@ -58,7 +35,7 @@ async fn main() {
 
     // Query engine with 16GB memory cache
     let cache = queryd::cache::MemCache::new(16 * 1024 * 1024 * 1024);
-    let engine = queryd::context::QueryEngine::new(registry.clone(), bucket_registry.clone(), cache);
+    let engine = queryd::context::QueryEngine::new(registry.clone(), store.clone(), cache);
 
     // Event journal — append-only mutation log (flush every 100 events)
     let journal = journald::journal::Journal::new(store.clone(), 100);
@@ -76,103 +53,27 @@ async fn main() {
     // AI sidecar client
     let ai_client = aibridge::client::AiClient::new(&config.sidecar.url);
 
-    // Vector service components — built before the router because both the
-    // /vectors service AND ingestd need the agent handle to enqueue triggers.
-    let index_reg = vectord::index_registry::IndexRegistry::new(store.clone());
-    let _ = index_reg.rebuild().await;
-    let hnsw = vectord::hnsw::HnswStore::new();
-    let emb_cache = vectord::embedding_cache::EmbeddingCache::new(store.clone());
-    // Phase B (federation layer 2): trial journals + promotion files now
-    // live in each index's recorded bucket (IndexMeta.bucket), not always
-    // primary. Journal / registry resolve per-call via the bucket registry.
-    let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone());
-    let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone());
-    let hs = vectord::harness::HarnessStore::new(bucket_registry.clone(), index_reg.clone());
-    // Phase 19: playbook memory. Load cached state; empty-on-miss is fine,
-    // operators call POST /vectors/playbook_memory/rebuild to populate.
-    let pbm = vectord::playbook_memory::PlaybookMemory::new(store.clone());
-    let _ = pbm.load_from_storage().await;
-
-    // Phase 16.2: spawn the autotune agent. When config.agent.enabled=false
-    // this returns a handle that drops triggers silently — no surprise load.
-    let agent_cfg = vectord::agent::AgentConfig {
-        enabled: config.agent.enabled,
-        cycle_interval_secs: config.agent.cycle_interval_secs,
-        cooldown_between_trials_secs: config.agent.cooldown_between_trials_secs,
-        min_recall: config.agent.min_recall,
-        max_trials_per_hour: config.agent.max_trials_per_hour,
-    };
-    let agent_handle = vectord::agent::spawn(
-        agent_cfg,
-        vectord::agent::AgentDeps {
-            store: store.clone(),
-            ai_client: ai_client.clone(),
-            catalog: registry.clone(),
-            index_registry: index_reg.clone(),
-            hnsw_store: hnsw.clone(),
-            embedding_cache: emb_cache.clone(),
-            trial_journal: tj.clone(),
-            promotion_registry: pr.clone(),
-            harness_store: hs.clone(),
-        },
-    );
-
     // HTTP router
     let mut app = Router::new()
         .route("/health", get(health))
-        .nest("/storage", storaged::service::router(store.clone())
-            .merge(storaged::federation_service::router(bucket_registry.clone())))
+        .nest("/storage", storaged::service::router(store.clone()))
         .nest("/catalog", catalogd::service::router(registry.clone()))
         .nest("/query", queryd::service::router(engine.clone()))
         .nest("/ai", aibridge::service::router(ai_client.clone()))
-        ;
-
-    // Phase E: scheduled ingest
-    let sched_store = ingestd::schedule::ScheduleStore::new(store.clone());
-    if let Ok(n) = sched_store.rebuild().await {
-        if n > 0 { tracing::info!("rebuilt {n} persisted schedule(s)"); }
-    }
-    let scheduler = ingestd::schedule::Scheduler {
-        store: sched_store.clone(),
-        ingest: ingestd::schedule::SchedulerIngestDeps {
-            store: store.clone(),
-            registry: registry.clone(),
-            buckets: bucket_registry.clone(),
-            agent_handle: Some(agent_handle.clone()),
-            index_registry: index_reg.clone(),
-        },
-        tick_secs: 10,
-    };
-    scheduler.spawn();
-
-    app = app
         .nest("/ingest", ingestd::service::router(ingestd::service::IngestState {
             store: store.clone(),
             registry: registry.clone(),
-            buckets: bucket_registry.clone(),
-            agent_handle: agent_handle.clone(),
-            index_registry: index_reg.clone(),
-            schedules: sched_store,
         }))
-        .nest("/vectors", vectord::service::router(vectord::service::VectorState {
-            store: store.clone(),
-            ai_client: ai_client.clone(),
-            job_tracker: vectord::jobs::JobTracker::new(),
-            index_registry: index_reg.clone(),
-            hnsw_store: hnsw,
-            embedding_cache: emb_cache,
-            trial_journal: tj,
-            harness_store: hs,
-            catalog: registry.clone(),
-            promotion_registry: pr,
-            agent_handle,
-            bucket_registry: bucket_registry.clone(),
-            active_profile: std::sync::Arc::new(tokio::sync::RwLock::new(None)),
-            lance: vectord::lance_backend::LanceRegistry::new(
-                bucket_registry.clone(), index_reg.clone(),
-            ),
-            playbook_memory: pbm,
-            embed_semaphore: std::sync::Arc::new(tokio::sync::Semaphore::new(1)),
+        .nest("/vectors", vectord::service::router({
+            let index_reg = vectord::index_registry::IndexRegistry::new(store.clone());
+            let _ = index_reg.rebuild().await;
+            vectord::service::VectorState {
+                store: store.clone(),
+                ai_client: ai_client.clone(),
+                job_tracker: vectord::jobs::JobTracker::new(),
+                index_registry: index_reg,
+                hnsw_store: vectord::hnsw::HnswStore::new(),
+            }
         }))
         .nest("/workspaces", queryd::workspace_service::router(workspace_mgr))
         .nest("/journal", journald::service::router(journal))
@@ -184,38 +85,6 @@ async fn main() {
                 registry: tool_reg,
                 query_fn: tools::QueryExecutor::new(engine.clone()),
             }
-        }))
-        // Phase 38 — Universal API skeleton. Thin OpenAI-compatible
-        // surface over the existing aibridge → Ollama path. Future
-        // phases add provider adapters (39), routing engine (40),
-        // session state (41), etc. All without changing this mount.
-        .nest("/v1", v1::router(v1::V1State {
-            ai_client: ai_client.clone(),
-            usage: std::sync::Arc::new(tokio::sync::RwLock::new(v1::Usage::default())),
-            // Phase 39 first slice — Ollama Cloud adapter. Key resolved
-            // from OLLAMA_CLOUD_KEY env, then /root/llm_team_config.json,
-            // then OLLAMA_CLOUD_API_KEY env. None = cloud routes 503.
-            ollama_cloud_key: {
-                let k = v1::ollama_cloud::resolve_cloud_key();
-                if k.is_some() {
-                    tracing::info!("v1: Ollama Cloud key loaded — /v1/chat provider=ollama_cloud enabled");
-                } else {
-                    tracing::warn!("v1: no Ollama Cloud key in env or /root/llm_team_config.json — cloud routes will 503");
-                }
-                k
-            },
-            // Phase 40 early deliverable — Langfuse trace emitter.
-            // Defaults match mcp-server/tracing.ts conventions so
-            // gateway traces land in the same staffing project.
-            langfuse: {
-                let c = v1::langfuse_trace::LangfuseClient::from_env_or_defaults();
-                if c.is_some() {
-                    tracing::info!("v1: Langfuse tracing enabled — /v1/chat calls will appear at localhost:3001");
-                } else {
-                    tracing::warn!("v1: Langfuse keys missing — /v1/chat calls will not be traced");
-                }
-                c
-            },
         }));
 
     // Auth middleware (if enabled)
@@ -232,7 +101,7 @@ async fn main() {
     }
 
     app = app
-        .layer(DefaultBodyLimit::max(512 * 1024 * 1024)) // 512MB — supports 500K worker CSV
+        .layer(DefaultBodyLimit::max(256 * 1024 * 1024)) // 256MB
         .layer(CorsLayer::new()
             .allow_origin(Any)
             .allow_methods(Any)

crates/gateway/src/tools/registry.rs

@@ -103,7 +103,7 @@ impl ToolRegistry {
                     ParamDef { name: "limit".into(), param_type: "integer".into(), required: false, description: "Max results".into(), default: Some(serde_json::json!(20)) },
                 ],
                 returns: "List of candidates with id, name, phone, email, skills, experience".into(),
-                sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, skills, years_experience, status FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
+                sql_template: "SELECT candidate_id, first_name, last_name, phone, email, city, state, zip, vertical, skills, years_experience FROM candidates WHERE 1=1 {skills_filter} {city_filter} {state_filter} {years_filter} {status_filter} ORDER BY years_experience DESC LIMIT {limit}".into(),
                 category: "candidates".into(),
             },
             ToolDef {

crates/gateway/src/v1/langfuse_trace.rs

@@ -1,226 +0,0 @@
-//! Phase 40 (early deliverable) — Langfuse tracing from the Rust gateway.
-//!
-//! Langfuse is already running at `http://localhost:3001` with the
-//! staffing project — `mcp-server/tracing.ts` has been emitting traces
-//! for months. This file bridges the gap: every `/v1/chat` call now
-//! appears in the same Langfuse project, alongside traces from the
-//! Bun side.
-//!
-//! Design:
-//! - Fire-and-forget — emit spawns a tokio task. Response to client
-//!   never waits on Langfuse, never fails because of it. If Langfuse
-//!   is down or slow, we silently drop traces (and `tracing::warn!`).
-//! - Two events per call — `trace-create` + `generation-create`. The
-//!   generation nests under the trace so Langfuse UI shows the whole
-//!   thing as one clickable row.
-//! - No SDK — plain HTTP POST to `/api/public/ingestion` with Basic
-//!   auth. Keeps the Rust dependency graph small.
-
-use serde::Serialize;
-use std::sync::Arc;
-use std::time::Duration;
-
-use super::Message;
-
-const INGESTION_PATH: &str = "/api/public/ingestion";
-const EMIT_TIMEOUT_SECS: u64 = 5;
-
-#[derive(Clone)]
-pub struct LangfuseClient {
-    inner: Arc<Inner>,
-}
-
-struct Inner {
-    base_url: String,
-    public_key: String,
-    secret_key: String,
-    http: reqwest::Client,
-}
-
-impl LangfuseClient {
-    pub fn new(base_url: String, public_key: String, secret_key: String) -> Self {
-        let http = reqwest::Client::builder()
-            .timeout(Duration::from_secs(EMIT_TIMEOUT_SECS))
-            .build()
-            .expect("langfuse client init");
-        Self { inner: Arc::new(Inner { base_url, public_key, secret_key, http }) }
-    }
-
-    /// Resolve a LangfuseClient from the same conventions the TS side
-    /// already uses. Returns None if keys are missing — caller logs a
-    /// warning and omits tracing (non-fatal).
-    ///
-    /// Sources checked in order:
-    ///   1. LANGFUSE_URL + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY env
-    ///   2. /root/llm_team_config.json (if it carries langfuse keys)
-    ///   3. Hardcoded staffing-project defaults matching mcp-server
-    ///      (pk-lf-staffing / sk-lf-staffing-secret / localhost:3001)
-    pub fn from_env_or_defaults() -> Option<Self> {
-        let base = std::env::var("LANGFUSE_URL")
-            .unwrap_or_else(|_| "http://localhost:3001".to_string());
-        let pk = std::env::var("LANGFUSE_PUBLIC_KEY")
-            .unwrap_or_else(|_| "pk-lf-staffing".to_string());
-        let sk = std::env::var("LANGFUSE_SECRET_KEY")
-            .unwrap_or_else(|_| "sk-lf-staffing-secret".to_string());
-        if pk.trim().is_empty() || sk.trim().is_empty() { return None; }
-        Some(Self::new(base, pk, sk))
-    }
-
-    /// Fire-and-forget emit. Never blocks, never errors upward.
-    pub fn emit_chat(&self, ev: ChatTrace) {
-        let this = self.clone();
-        tokio::spawn(async move {
-            if let Err(e) = this.emit_chat_inner(ev).await {
-                tracing::warn!(target: "v1.langfuse", "trace drop: {e}");
-            }
-        });
-    }
-
-    async fn emit_chat_inner(&self, ev: ChatTrace) -> Result<(), String> {
-        let trace_id = uuid_v7_like();
-        let gen_id = uuid_v7_like();
-        let trace_ts = ev.start_time.clone();
-
-        let batch = IngestionBatch {
-            batch: vec![
-                IngestionEvent {
-                    id: uuid_v7_like(),
-                    timestamp: trace_ts.clone(),
-                    kind: "trace-create",
-                    body: serde_json::json!({
-                        "id": trace_id,
-                        "name": format!("v1.chat:{}", ev.provider),
-                        "input": serde_json::json!({
-                            "model": ev.model,
-                            "messages": ev.input,
-                        }),
-                        "metadata": serde_json::json!({
-                            "provider": ev.provider,
-                            "think": ev.think,
-                        }),
-                    }),
-                },
-                IngestionEvent {
-                    id: uuid_v7_like(),
-                    timestamp: ev.end_time.clone(),
-                    kind: "generation-create",
-                    body: serde_json::json!({
-                        "id": gen_id,
-                        "traceId": trace_id,
-                        "name": "chat",
-                        "model": ev.model,
-                        "modelParameters": serde_json::json!({
-                            "temperature": ev.temperature,
-                            "max_tokens": ev.max_tokens,
-                            "think": ev.think,
-                        }),
-                        "input": ev.input,
-                        "output": ev.output,
-                        "usage": serde_json::json!({
-                            "input": ev.prompt_tokens,
-                            "output": ev.completion_tokens,
-                            "total": ev.prompt_tokens + ev.completion_tokens,
-                            "unit": "TOKENS",
-                        }),
-                        "startTime": ev.start_time,
-                        "endTime": ev.end_time,
-                        "metadata": serde_json::json!({
-                            "provider": ev.provider,
-                            "latency_ms": ev.latency_ms,
-                        }),
-                    }),
-                },
-            ],
-        };
-
-        let url = format!("{}{}", self.inner.base_url.trim_end_matches('/'), INGESTION_PATH);
-        let resp = self.inner.http
-            .post(url)
-            .basic_auth(&self.inner.public_key, Some(&self.inner.secret_key))
-            .json(&batch)
-            .send()
-            .await
-            .map_err(|e| format!("POST failed: {e}"))?;
-        if !resp.status().is_success() {
-            return Err(format!("{}: {}", resp.status(), resp.text().await.unwrap_or_default()));
-        }
-        Ok(())
-    }
-}
-
-/// Everything the v1.chat handler collects for one completed call.
-pub struct ChatTrace {
-    pub provider: String,
-    pub model: String,
-    pub input: Vec<Message>,
-    pub output: String,
-    pub prompt_tokens: u32,
-    pub completion_tokens: u32,
-    pub temperature: Option<f64>,
-    pub max_tokens: Option<u32>,
-    pub think: Option<bool>,
-    pub start_time: String,
-    pub end_time: String,
-    pub latency_ms: u64,
-}
-
-#[derive(Serialize)]
-struct IngestionBatch {
-    batch: Vec<IngestionEvent>,
-}
-
-#[derive(Serialize)]
-struct IngestionEvent {
-    id: String,
-    timestamp: String,
-    #[serde(rename = "type")]
-    kind: &'static str,
-    body: serde_json::Value,
-}
-
-/// UUIDv7-ish identifier — not strictly UUIDv7 but time-ordered and
-/// unique enough for event deduplication. Avoids a uuid crate dep.
-fn uuid_v7_like() -> String {
-    let ts = chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0);
-    let rand = std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .map(|d| d.subsec_nanos())
-        .unwrap_or(0);
-    format!("{:016x}-{:08x}", ts, rand)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn from_env_or_defaults_returns_some_when_defaults_present() {
-        // Defaults are hardcoded non-empty — should always produce Some.
-        let c = LangfuseClient::from_env_or_defaults();
-        assert!(c.is_some());
-    }
-
-    #[test]
-    fn uuid_v7_like_produces_distinct_ids() {
-        let a = uuid_v7_like();
-        // Different nanosecond timestamps — but they might collide on
-        // back-to-back calls. Loop a bit to ensure uniqueness surface.
-        let b = uuid_v7_like();
-        let c = uuid_v7_like();
-        assert!(a != b || b != c, "three consecutive ids should have at least one distinct pair");
-        assert!(a.contains('-'));
-    }
-
-    #[test]
-    fn ingestion_batch_serializes_with_rename() {
-        let ev = IngestionEvent {
-            id: "e1".into(),
-            timestamp: "2026-04-22T00:00:00Z".into(),
-            kind: "trace-create",
-            body: serde_json::json!({"name": "t"}),
-        };
-        let json = serde_json::to_string(&IngestionBatch { batch: vec![ev] }).unwrap();
-        assert!(json.contains("\"type\":\"trace-create\""));
-        assert!(json.contains("\"batch\":["));
-    }
-}

crates/gateway/src/v1/mod.rs

@@ -1,256 +0,0 @@
-//! Phase 38 — Universal API skeleton (`/v1/*`).
-//!
-//! OpenAI-compatible shape on top of the existing aibridge → Ollama
-//! path. This is the thin slice: single provider, stateless, no
-//! streaming. Phase 39 replaces the direct Ollama call with a
-//! `ProviderAdapter` trait dispatch; Phase 40 adds routing + fallback.
-//!
-//! The shape matches OpenAI's `/v1/chat/completions` closely enough
-//! that clients using openai-compatible SDKs can point at us with the
-//! URL swap alone. We keep the endpoint path `/v1/chat` (not
-//! `/v1/chat/completions`) because our PRD declares the terser form;
-//! adding the alias is one line in Phase 39 when it matters.
-
-pub mod ollama;
-pub mod ollama_cloud;
-pub mod langfuse_trace;
-
-use axum::{
-    Router,
-    extract::State,
-    http::StatusCode,
-    response::IntoResponse,
-    routing::{get, post},
-    Json,
-};
-use serde::{Deserialize, Serialize};
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-#[derive(Clone)]
-pub struct V1State {
-    pub ai_client: aibridge::client::AiClient,
-    pub usage: Arc<RwLock<Usage>>,
-    /// Ollama Cloud bearer token. Loaded at startup via
-    /// `ollama_cloud::resolve_cloud_key()`. None = cloud routes 503.
-    pub ollama_cloud_key: Option<String>,
-    /// Phase 40 early deliverable — Langfuse client. None = tracing
-    /// disabled (keys missing or container unreachable). Traces are
-    /// fire-and-forget: never block the response path.
-    pub langfuse: Option<langfuse_trace::LangfuseClient>,
-}
-
-#[derive(Default, Clone, Serialize)]
-pub struct Usage {
-    pub requests: u64,
-    pub prompt_tokens: u64,
-    pub completion_tokens: u64,
-    pub total_tokens: u64,
-}
-
-pub fn router(state: V1State) -> Router {
-    Router::new()
-        .route("/chat", post(chat))
-        .route("/usage", get(usage))
-        .route("/sessions", get(sessions))
-        .with_state(state)
-}
-
-// -- Shared types (OpenAI-compatible) --
-
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct Message {
-    pub role: String,
-    pub content: String,
-}
-
-#[derive(Deserialize, Debug)]
-pub struct ChatRequest {
-    pub model: String,
-    pub messages: Vec<Message>,
-    #[serde(default)]
-    pub temperature: Option<f64>,
-    #[serde(default)]
-    pub max_tokens: Option<u32>,
-    /// Accepted for shape-compat but ignored in the thin slice —
-    /// Phase 38 returns non-streaming even when the client asked for it.
-    /// Phase 39+ wires real streaming.
-    #[serde(default)]
-    pub stream: Option<bool>,
-    /// Non-OpenAI extension. Passes through to the provider's thinking
-    /// toggle. Default: **false** — hot-path discipline for thinking
-    /// models (qwen3.5, qwen3, gpt-oss) that otherwise burn the token
-    /// budget on hidden reasoning before visible output starts,
-    /// producing empty responses. Set true explicitly when calling an
-    /// overseer / reasoning-heavy path.
-    #[serde(default)]
-    pub think: Option<bool>,
-    /// Non-OpenAI extension. Selects the provider adapter. Accepted:
-    ///   - None / "ollama" / "local" → local Ollama via sidecar (default)
-    ///   - "ollama_cloud" / "cloud" → Ollama Cloud direct HTTPS
-    /// Phase 40 adds a routing engine that picks this automatically
-    /// from the model name; Phase 38/39 requires explicit selection.
-    #[serde(default)]
-    pub provider: Option<String>,
-}
-
-#[derive(Serialize)]
-pub struct ChatResponse {
-    pub id: String,
-    pub object: &'static str,
-    pub created: i64,
-    pub model: String,
-    pub choices: Vec<Choice>,
-    pub usage: UsageBlock,
-}
-
-#[derive(Serialize)]
-pub struct Choice {
-    pub index: u32,
-    pub message: Message,
-    pub finish_reason: String,
-}
-
-#[derive(Serialize, Deserialize, Clone)]
-pub struct UsageBlock {
-    pub prompt_tokens: u32,
-    pub completion_tokens: u32,
-    pub total_tokens: u32,
-}
-
-// -- Handlers --
-
-async fn chat(
-    State(state): State<V1State>,
-    Json(req): Json<ChatRequest>,
-) -> Result<Json<ChatResponse>, (StatusCode, String)> {
-    if req.messages.is_empty() {
-        return Err((StatusCode::BAD_REQUEST, "messages must be non-empty".into()));
-    }
-    if req.stream.unwrap_or(false) {
-        tracing::warn!("/v1/chat: stream=true requested but Phase 38 returns non-streaming");
-    }
-
-    let provider = req.provider.as_deref().unwrap_or("ollama").to_ascii_lowercase();
-    let start_time = chrono::Utc::now();
-    let start_instant = std::time::Instant::now();
-
-    let resp = match provider.as_str() {
-        "ollama" | "local" | "" => ollama::chat(&state.ai_client, &req)
-            .await
-            .map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama local: {e}")))?,
-        "ollama_cloud" | "cloud" => {
-            let key = state.ollama_cloud_key.as_deref().ok_or((
-                StatusCode::SERVICE_UNAVAILABLE,
-                "OLLAMA_CLOUD_KEY not configured — set env or populate /root/llm_team_config.json".to_string(),
-            ))?;
-            ollama_cloud::chat(key, &req)
-                .await
-                .map_err(|e| (StatusCode::BAD_GATEWAY, format!("ollama cloud: {e}")))?
-        }
-        other => {
-            return Err((
-                StatusCode::BAD_REQUEST,
-                format!("unknown provider '{other}' — supported: ollama, ollama_cloud"),
-            ));
-        }
-    };
-
-    let end_time = chrono::Utc::now();
-    let latency_ms = start_instant.elapsed().as_millis() as u64;
-
-    // Phase 40 — emit Langfuse trace. Fire-and-forget: the clone is
-    // cheap (Arc inside), the tokio::spawn never blocks us, a dead
-    // Langfuse just logs a warn. Client-visible response latency is
-    // untouched.
-    if let Some(lf) = &state.langfuse {
-        let output = resp.choices.first()
-            .map(|c| c.message.content.clone())
-            .unwrap_or_default();
-        lf.emit_chat(langfuse_trace::ChatTrace {
-            provider: provider.clone(),
-            model: resp.model.clone(),
-            input: req.messages.clone(),
-            output,
-            prompt_tokens: resp.usage.prompt_tokens,
-            completion_tokens: resp.usage.completion_tokens,
-            temperature: req.temperature,
-            max_tokens: req.max_tokens,
-            think: req.think,
-            start_time: start_time.to_rfc3339(),
-            end_time: end_time.to_rfc3339(),
-            latency_ms,
-        });
-    }
-
-    {
-        let mut u = state.usage.write().await;
-        u.requests += 1;
-        u.prompt_tokens += resp.usage.prompt_tokens as u64;
-        u.completion_tokens += resp.usage.completion_tokens as u64;
-        u.total_tokens += resp.usage.total_tokens as u64;
-    }
-
-    Ok(Json(resp))
-}
-
-async fn usage(State(state): State<V1State>) -> impl IntoResponse {
-    let snapshot = state.usage.read().await.clone();
-    Json(snapshot)
-}
-
-// Phase 38 is stateless — no session persistence yet. Return an empty
-// list in OpenAI-ish shape so clients that probe this endpoint don't
-// 404. Real session state lands in Phase 41 with the profile-system
-// expansion.
-async fn sessions() -> impl IntoResponse {
-    Json(serde_json::json!({
-        "data": [],
-        "object": "list",
-        "note": "Phase 38: stateless. Session state lands in Phase 41.",
-    }))
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn chat_request_parses_openai_shape() {
-        let raw = r#"{
-            "model": "qwen3.5:latest",
-            "messages": [
-                {"role": "system", "content": "You are helpful."},
-                {"role": "user", "content": "Hi"}
-            ],
-            "temperature": 0.2,
-            "max_tokens": 100
-        }"#;
-        let r: ChatRequest = serde_json::from_str(raw).unwrap();
-        assert_eq!(r.model, "qwen3.5:latest");
-        assert_eq!(r.messages.len(), 2);
-        assert_eq!(r.messages[0].role, "system");
-        assert_eq!(r.messages[1].content, "Hi");
-        assert_eq!(r.temperature, Some(0.2));
-        assert_eq!(r.max_tokens, Some(100));
-    }
-
-    #[test]
-    fn chat_request_accepts_minimal() {
-        let raw = r#"{
-            "model": "any",
-            "messages": [{"role": "user", "content": "hi"}]
-        }"#;
-        let r: ChatRequest = serde_json::from_str(raw).unwrap();
-        assert_eq!(r.temperature, None);
-        assert_eq!(r.max_tokens, None);
-        assert_eq!(r.stream, None);
-    }
-
-    #[test]
-    fn usage_counter_default_is_zero() {
-        let u = Usage::default();
-        assert_eq!(u.requests, 0);
-        assert_eq!(u.total_tokens, 0);
-    }
-}

crates/gateway/src/v1/ollama.rs

@@ -1,160 +0,0 @@
-//! Phase 38 — Ollama shape adapter.
-//!
-//! Translates `/v1/chat` (OpenAI-compatible) requests into the
-//! existing aibridge `GenerateRequest` shape, and the `GenerateResponse`
-//! back into an OpenAI-compatible `ChatResponse`. This is a bridge,
-//! not a new client — aibridge + the Python sidecar stay as-is.
-//!
-//! Phase 39 replaces this direct call with a `ProviderAdapter` trait
-//! dispatch so the same `/v1/chat` handler routes to any provider.
-
-use aibridge::client::{AiClient, GenerateRequest};
-use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
-
-pub async fn chat(client: &AiClient, req: &ChatRequest) -> Result<ChatResponse, String> {
-    let (system, prompt) = flatten_messages(&req.messages);
-
-    let gen_req = GenerateRequest {
-        prompt,
-        model: Some(req.model.clone()),
-        system: if system.is_empty() { None } else { Some(system) },
-        temperature: req.temperature,
-        max_tokens: req.max_tokens,
-        // Phase 38 default: think=false. Hot-path discipline for
-        // thinking models (qwen3.5, qwen3, gpt-oss) which otherwise
-        // burn the max_tokens budget on hidden reasoning before any
-        // visible output, producing empty responses. Callers that
-        // actually want reasoning (overseers, T3+ tiers) opt in via
-        // the `think: true` extension field. Phase 40 routing engine
-        // flips this per task class.
-        think: Some(req.think.unwrap_or(false)),
-    };
-
-    let t0 = std::time::Instant::now();
-    let resp = client.generate(gen_req).await.map_err(|e| e.to_string())?;
-    let latency_ms = t0.elapsed().as_millis();
-
-    // Prefer sidecar-reported token counts when present. Fall back to
-    // chars/4 estimate (biased safe ~15%, matches Phase 21 convention
-    // in crates/aibridge/src/context.rs::estimate_tokens).
-    let prompt_tokens = resp.tokens_evaluated
-        .map(|n| n as u32)
-        .unwrap_or_else(|| estimate_prompt_tokens(&req.messages));
-    let completion_tokens = resp.tokens_generated
-        .map(|n| n as u32)
-        .unwrap_or_else(|| ((resp.text.chars().count() + 3) / 4) as u32);
-
-    tracing::info!(
-        target: "v1.chat",
-        model = %req.model,
-        prompt_tokens,
-        completion_tokens,
-        latency_ms = latency_ms as u64,
-        "ollama chat completed",
-    );
-
-    Ok(ChatResponse {
-        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
-        object: "chat.completion",
-        created: chrono::Utc::now().timestamp(),
-        model: resp.model,
-        choices: vec![Choice {
-            index: 0,
-            message: Message {
-                role: "assistant".into(),
-                content: resp.text,
-            },
-            finish_reason: "stop".into(),
-        }],
-        usage: UsageBlock {
-            prompt_tokens,
-            completion_tokens,
-            total_tokens: prompt_tokens + completion_tokens,
-        },
-    })
-}
-
-/// Public re-export of the flattener so sibling adapters (Ollama Cloud,
-/// future OpenRouter) can reuse the same shape collapse without
-/// duplicating the logic. Keeps `(system, prompt)` format consistent
-/// across providers.
-pub fn flatten_messages_public(messages: &[Message]) -> (String, String) {
-    flatten_messages(messages)
-}
-
-/// Collapse a message array into (system, prompt). Multiple system
-/// messages concatenate with a newline — matches OpenAI's documented
-/// behavior. Non-system messages become role-labeled turns.
-fn flatten_messages(messages: &[Message]) -> (String, String) {
-    let mut system = String::new();
-    let mut prompt = String::new();
-    for m in messages {
-        if m.role == "system" {
-            if !system.is_empty() { system.push('\n'); }
-            system.push_str(&m.content);
-        } else {
-            prompt.push_str(&m.role);
-            prompt.push_str(": ");
-            prompt.push_str(&m.content);
-            prompt.push_str("\n\n");
-        }
-    }
-    prompt.push_str("assistant:");
-    (system, prompt)
-}
-
-fn estimate_prompt_tokens(messages: &[Message]) -> u32 {
-    let chars: usize = messages.iter().map(|m| m.content.chars().count()).sum();
-    ((chars + 3) / 4) as u32
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn flatten_separates_system_from_turns() {
-        let msgs = vec![
-            Message { role: "system".into(), content: "Rules here.".into() },
-            Message { role: "user".into(), content: "Q1".into() },
-            Message { role: "assistant".into(), content: "A1".into() },
-            Message { role: "user".into(), content: "Q2".into() },
-        ];
-        let (system, prompt) = flatten_messages(&msgs);
-        assert_eq!(system, "Rules here.");
-        assert!(prompt.contains("user: Q1"));
-        assert!(prompt.contains("assistant: A1"));
-        assert!(prompt.contains("user: Q2"));
-        assert!(prompt.trim_end().ends_with("assistant:"));
-    }
-
-    #[test]
-    fn flatten_concatenates_multiple_system_messages() {
-        let msgs = vec![
-            Message { role: "system".into(), content: "First.".into() },
-            Message { role: "system".into(), content: "Second.".into() },
-            Message { role: "user".into(), content: "Hi".into() },
-        ];
-        let (system, _) = flatten_messages(&msgs);
-        assert_eq!(system, "First.\nSecond.");
-    }
-
-    #[test]
-    fn flatten_with_no_system_returns_empty_system() {
-        let msgs = vec![Message { role: "user".into(), content: "hi".into() }];
-        let (system, prompt) = flatten_messages(&msgs);
-        assert!(system.is_empty());
-        assert!(prompt.contains("user: hi"));
-    }
-
-    #[test]
-    fn estimate_tokens_chars_div_4_ceiling() {
-        let msgs = vec![Message { role: "user".into(), content: "abcdefgh".into() }];
-        // 8 chars / 4 = 2, with ceiling → 2
-        assert_eq!(estimate_prompt_tokens(&msgs), 2);
-
-        let msgs2 = vec![Message { role: "user".into(), content: "abcdefghi".into() }];
-        // 9 chars → (9+3)/4 = 3 (ceiling)
-        assert_eq!(estimate_prompt_tokens(&msgs2), 3);
-    }
-}

crates/gateway/src/v1/ollama_cloud.rs

@@ -1,189 +0,0 @@
-//! Phase 39 (first slice) — Ollama Cloud adapter.
-//!
-//! Direct HTTPS call to `https://ollama.com/api/generate` with Bearer
-//! auth. Mirrors the `tests/multi-agent/agent.ts::generateCloud` pattern
-//! exactly (same endpoint, same body shape, same Bearer header), so
-//! cloud calls from Rust behave identically to the TS hot path.
-//!
-//! Key sourcing priority:
-//!   1. Env var `OLLAMA_CLOUD_KEY` (matches agent.ts convention)
-//!   2. `/root/llm_team_config.json` → providers.ollama_cloud.api_key
-//!   3. Env var `OLLAMA_CLOUD_API_KEY` (LLM Team UI convention)
-//!
-//! First hit wins. Key is loaded once at gateway startup by
-//! `resolve_cloud_key()` and stored on `V1State.ollama_cloud_key`.
-
-use std::time::Duration;
-use serde::{Deserialize, Serialize};
-
-use super::{ChatRequest, ChatResponse, Choice, Message, UsageBlock};
-
-const CLOUD_BASE_URL: &str = "https://ollama.com";
-const CLOUD_TIMEOUT_SECS: u64 = 180;
-
-/// Read the Ollama Cloud key from the three sanctioned sources. Returns
-/// None if none is set — callers must 503 rather than attempt a call.
-pub fn resolve_cloud_key() -> Option<String> {
-    if let Ok(k) = std::env::var("OLLAMA_CLOUD_KEY") {
-        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
-    }
-    if let Ok(raw) = std::fs::read_to_string("/root/llm_team_config.json") {
-        if let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) {
-            if let Some(k) = v.pointer("/providers/ollama_cloud/api_key").and_then(|x| x.as_str()) {
-                if !k.trim().is_empty() { return Some(k.trim().to_string()); }
-            }
-        }
-    }
-    if let Ok(k) = std::env::var("OLLAMA_CLOUD_API_KEY") {
-        if !k.trim().is_empty() { return Some(k.trim().to_string()); }
-    }
-    None
-}
-
-pub async fn chat(
-    key: &str,
-    req: &ChatRequest,
-) -> Result<ChatResponse, String> {
-    let (system, prompt) = super::ollama::flatten_messages_public(&req.messages);
-
-    let body = CloudGenerateBody {
-        model: req.model.clone(),
-        prompt,
-        system: if system.is_empty() { None } else { Some(system) },
-        stream: false,
-        think: Some(req.think.unwrap_or(false)),
-        options: CloudOptions {
-            // Thinking cloud models need headroom — floor 400 to give
-            // qwen3.5:397b / gpt-oss:120b reasoning room. Matches
-            // agent.ts `Math.max(opts.max_tokens ?? 800, 400)` policy.
-            num_predict: std::cmp::max(req.max_tokens.unwrap_or(800), 400),
-            temperature: req.temperature.unwrap_or(0.3),
-        },
-    };
-
-    let client = reqwest::Client::builder()
-        .timeout(Duration::from_secs(CLOUD_TIMEOUT_SECS))
-        .build()
-        .map_err(|e| format!("build client: {e}"))?;
-
-    let t0 = std::time::Instant::now();
-    let resp = client
-        .post(format!("{}/api/generate", CLOUD_BASE_URL))
-        .bearer_auth(key)
-        .json(&body)
-        .send()
-        .await
-        .map_err(|e| format!("ollama.com unreachable: {e}"))?;
-
-    let status = resp.status();
-    if !status.is_success() {
-        let body = resp.text().await.unwrap_or_else(|_| "?".into());
-        return Err(format!("ollama.com {}: {}", status, body));
-    }
-
-    let parsed: CloudGenerateResponse = resp.json().await
-        .map_err(|e| format!("invalid cloud response: {e}"))?;
-
-    let latency_ms = t0.elapsed().as_millis();
-    let text = parsed.response.unwrap_or_default();
-
-    let prompt_tokens = parsed.prompt_eval_count.unwrap_or_else(|| {
-        let chars: usize = req.messages.iter().map(|m| m.content.chars().count()).sum();
-        ((chars + 3) / 4) as u32
-    });
-    let completion_tokens = parsed.eval_count.unwrap_or_else(|| {
-        ((text.chars().count() + 3) / 4) as u32
-    });
-
-    tracing::info!(
-        target: "v1.chat",
-        provider = "ollama_cloud",
-        model = %req.model,
-        prompt_tokens,
-        completion_tokens,
-        latency_ms = latency_ms as u64,
-        "cloud chat completed",
-    );
-
-    Ok(ChatResponse {
-        id: format!("chatcmpl-{}", chrono::Utc::now().timestamp_nanos_opt().unwrap_or(0)),
-        object: "chat.completion",
-        created: chrono::Utc::now().timestamp(),
-        model: parsed.model.unwrap_or_else(|| req.model.clone()),
-        choices: vec![Choice {
-            index: 0,
-            message: Message { role: "assistant".into(), content: text },
-            finish_reason: "stop".into(),
-        }],
-        usage: UsageBlock {
-            prompt_tokens,
-            completion_tokens,
-            total_tokens: prompt_tokens + completion_tokens,
-        },
-    })
-}
-
-// -- Ollama /api/generate wire shapes --
-
-#[derive(Serialize)]
-struct CloudGenerateBody {
-    model: String,
-    prompt: String,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    system: Option<String>,
-    stream: bool,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    think: Option<bool>,
-    options: CloudOptions,
-}
-
-#[derive(Serialize)]
-struct CloudOptions {
-    num_predict: u32,
-    temperature: f64,
-}
-
-#[derive(Deserialize)]
-struct CloudGenerateResponse {
-    #[serde(default)]
-    response: Option<String>,
-    #[serde(default)]
-    model: Option<String>,
-    #[serde(default)]
-    prompt_eval_count: Option<u32>,
-    #[serde(default)]
-    eval_count: Option<u32>,
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn resolve_cloud_key_returns_none_when_no_sources_set() {
-        // Only check shape — we can't reliably unset env vars in a test
-        // that runs alongside others, and the file path is on disk.
-        // If all three sources are empty the function returns None; if
-        // any are set we expect Some. This just smoke-tests the call
-        // doesn't panic.
-        let _ = resolve_cloud_key();
-    }
-
-    #[test]
-    fn cloud_body_serializes_compact() {
-        let body = CloudGenerateBody {
-            model: "gpt-oss:120b".into(),
-            prompt: "user: hi\n\nassistant:".into(),
-            system: Some("Be terse.".into()),
-            stream: false,
-            think: Some(false),
-            options: CloudOptions { num_predict: 400, temperature: 0.3 },
-        };
-        let json = serde_json::to_string(&body).unwrap();
-        assert!(json.contains("\"model\":\"gpt-oss:120b\""));
-        assert!(json.contains("\"stream\":false"));
-        assert!(json.contains("\"num_predict\":400"));
-        assert!(json.contains("\"think\":false"));
-        assert!(json.contains("\"system\":\"Be terse.\""));
-    }
-}

crates/ingestd/Cargo.toml

@@ -7,7 +7,6 @@ edition = "2024"
 shared = { path = "../shared" }
 storaged = { path = "../storaged" }
 catalogd = { path = "../catalogd" }
-vectord = { path = "../vectord" }
 tokio = { workspace = true }
 axum = { workspace = true, features = ["multipart"] }
 lopdf = { workspace = true }
@@ -22,6 +21,3 @@ csv = { workspace = true }
 chrono = { workspace = true }
 object_store = { workspace = true }
 tokio-postgres = { workspace = true }
-mysql_async = { workspace = true }
-uuid = { workspace = true }
-croner = "3"

crates/ingestd/src/chatgpt_parser.rs

@@ -0,0 +1,235 @@
+/// ChatGPT PDF conversation parser.
+/// Extracts structured conversation turns from ChatGPT export PDFs.
+/// Handles: user/assistant turns, timestamps, conversation titles.
+
+use arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray};
+use arrow::datatypes::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// A single conversation turn.
+#[derive(Debug, Clone)]
+pub struct ConversationTurn {
+    pub source_file: String,
+    pub conversation_title: String,
+    pub turn_number: i32,
+    pub role: String,        // "user" or "assistant"
+    pub content: String,
+    pub page_number: i32,
+}
+
+/// Parse a ChatGPT PDF into structured conversation turns.
+pub fn parse_chatgpt_pdf(content: &[u8], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
+    let doc = lopdf::Document::load_mem(content)
+        .map_err(|e| format!("PDF load error: {e}"))?;
+
+    // Extract all text
+    let pages = doc.get_pages();
+    let mut all_text = String::new();
+    let mut page_texts: Vec<(i32, String)> = Vec::new();
+
+    for (&page_num, _) in pages.iter() {
+        let text = doc.extract_text(&[page_num]).unwrap_or_default();
+        let text = text.trim().to_string();
+        if !text.is_empty() {
+            page_texts.push((page_num as i32, text.clone()));
+            all_text.push_str(&text);
+            all_text.push('\n');
+        }
+    }
+
+    if all_text.trim().is_empty() {
+        return Err("PDF contains no extractable text".into());
+    }
+
+    // Try to parse as ChatGPT conversation format
+    let turns = extract_turns(&all_text, filename, &page_texts);
+
+    if turns.is_empty() {
+        // Fall back to page-based extraction if we can't detect conversation structure
+        return parse_as_raw_pages(&page_texts, filename);
+    }
+
+    tracing::info!("parsed ChatGPT PDF '{}': {} turns", filename, turns.len());
+    turns_to_batch(&turns)
+}
+
+/// Extract conversation turns from ChatGPT PDF text.
+/// ChatGPT PDFs typically have patterns like:
+/// - "You said:" / "ChatGPT said:" (newer format)
+/// - "User:" / "Assistant:"
+/// - Or just alternating blocks separated by formatting
+fn extract_turns(text: &str, filename: &str, page_texts: &[(i32, String)]) -> Vec<ConversationTurn> {
+    let mut turns = Vec::new();
+    let mut turn_num = 0;
+
+    // Try to extract conversation title from first line
+    let title = text.lines().next()
+        .map(|l| l.trim().to_string())
+        .filter(|l| l.len() < 200 && !l.is_empty())
+        .unwrap_or_else(|| filename.replace(".pdf", ""));
+
+    // Strategy 1: Look for "You said:" / "ChatGPT said:" markers (common format)
+    let you_markers = ["You said:", "You:", "User:", "Human:"];
+    let ai_markers = ["ChatGPT said:", "ChatGPT:", "Assistant:", "AI:"];
+
+    let has_markers = you_markers.iter().any(|m| text.contains(m))
+        && ai_markers.iter().any(|m| text.contains(m));
+
+    if has_markers {
+        let lines: Vec<&str> = text.lines().collect();
+        let mut current_role = String::new();
+        let mut current_content = String::new();
+        let mut current_page = 1i32;
+
+        for line in &lines {
+            let trimmed = line.trim();
+
+            // Check for role markers
+            let mut found_marker = false;
+            for marker in &you_markers {
+                if trimmed.starts_with(marker) || trimmed == *marker {
+                    // Save previous turn
+                    if !current_content.trim().is_empty() && !current_role.is_empty() {
+                        turn_num += 1;
+                        turns.push(ConversationTurn {
+                            source_file: filename.to_string(),
+                            conversation_title: title.clone(),
+                            turn_number: turn_num,
+                            role: current_role.clone(),
+                            content: current_content.trim().to_string(),
+                            page_number: current_page,
+                        });
+                    }
+                    current_role = "user".to_string();
+                    current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
+                    current_content.push('\n');
+                    found_marker = true;
+                    break;
+                }
+            }
+            if !found_marker {
+                for marker in &ai_markers {
+                    if trimmed.starts_with(marker) || trimmed == *marker {
+                        if !current_content.trim().is_empty() && !current_role.is_empty() {
+                            turn_num += 1;
+                            turns.push(ConversationTurn {
+                                source_file: filename.to_string(),
+                                conversation_title: title.clone(),
+                                turn_number: turn_num,
+                                role: current_role.clone(),
+                                content: current_content.trim().to_string(),
+                                page_number: current_page,
+                            });
+                        }
+                        current_role = "assistant".to_string();
+                        current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
+                        current_content.push('\n');
+                        found_marker = true;
+                        break;
+                    }
+                }
+            }
+
+            if !found_marker && !current_role.is_empty() {
+                current_content.push_str(trimmed);
+                current_content.push('\n');
+            }
+        }
+
+        // Save last turn
+        if !current_content.trim().is_empty() && !current_role.is_empty() {
+            turn_num += 1;
+            turns.push(ConversationTurn {
+                source_file: filename.to_string(),
+                conversation_title: title.clone(),
+                turn_number: turn_num,
+                role: current_role,
+                content: current_content.trim().to_string(),
+                page_number: current_page,
+            });
+        }
+
+        return turns;
+    }
+
+    // Strategy 2: Split by paragraph blocks, alternate user/assistant
+    // This is a heuristic fallback for PDFs without clear markers
+    let paragraphs: Vec<&str> = text.split("\n\n")
+        .map(|p| p.trim())
+        .filter(|p| p.len() > 20) // skip short fragments
+        .collect();
+
+    if paragraphs.len() >= 2 {
+        for (i, para) in paragraphs.iter().enumerate() {
+            turn_num += 1;
+            turns.push(ConversationTurn {
+                source_file: filename.to_string(),
+                conversation_title: title.clone(),
+                turn_number: turn_num,
+                role: if i % 2 == 0 { "user" } else { "assistant" }.to_string(),
+                content: para.to_string(),
+                page_number: 1,
+            });
+        }
+    }
+
+    turns
+}
+
+/// Fallback: parse as raw pages with full text.
+fn parse_as_raw_pages(page_texts: &[(i32, String)], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("source_file", DataType::Utf8, false),
+        Field::new("conversation_title", DataType::Utf8, false),
+        Field::new("turn_number", DataType::Int32, false),
+        Field::new("role", DataType::Utf8, false),
+        Field::new("content", DataType::Utf8, false),
+        Field::new("page_number", DataType::Int32, false),
+    ]));
+
+    let sources: Vec<&str> = page_texts.iter().map(|_| filename).collect();
+    let titles: Vec<String> = page_texts.iter().map(|_| filename.replace(".pdf", "")).collect();
+    let turn_nums: Vec<i32> = (1..=page_texts.len() as i32).collect();
+    let roles: Vec<&str> = page_texts.iter().map(|_| "page").collect();
+    let contents: Vec<&str> = page_texts.iter().map(|(_, t)| t.as_str()).collect();
+    let pages: Vec<i32> = page_texts.iter().map(|(p, _)| *p).collect();
+
+    let arrays: Vec<ArrayRef> = vec![
+        Arc::new(StringArray::from(sources)),
+        Arc::new(StringArray::from(titles.iter().map(|s| s.as_str()).collect::<Vec<_>>())),
+        Arc::new(Int32Array::from(turn_nums)),
+        Arc::new(StringArray::from(roles)),
+        Arc::new(StringArray::from(contents)),
+        Arc::new(Int32Array::from(pages)),
+    ];
+
+    let batch = RecordBatch::try_new(schema.clone(), arrays)
+        .map_err(|e| format!("RecordBatch error: {e}"))?;
+
+    Ok((schema, vec![batch]))
+}
+
+fn turns_to_batch(turns: &[ConversationTurn]) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("source_file", DataType::Utf8, false),
+        Field::new("conversation_title", DataType::Utf8, false),
+        Field::new("turn_number", DataType::Int32, false),
+        Field::new("role", DataType::Utf8, false),
+        Field::new("content", DataType::Utf8, false),
+        Field::new("page_number", DataType::Int32, false),
+    ]));
+
+    let arrays: Vec<ArrayRef> = vec![
+        Arc::new(StringArray::from(turns.iter().map(|t| t.source_file.as_str()).collect::<Vec<_>>())),
+        Arc::new(StringArray::from(turns.iter().map(|t| t.conversation_title.as_str()).collect::<Vec<_>>())),
+        Arc::new(Int32Array::from(turns.iter().map(|t| t.turn_number).collect::<Vec<_>>())),
+        Arc::new(StringArray::from(turns.iter().map(|t| t.role.as_str()).collect::<Vec<_>>())),
+        Arc::new(StringArray::from(turns.iter().map(|t| t.content.as_str()).collect::<Vec<_>>())),
+        Arc::new(Int32Array::from(turns.iter().map(|t| t.page_number).collect::<Vec<_>>())),
+    ];
+
+    let batch = RecordBatch::try_new(schema.clone(), arrays)
+        .map_err(|e| format!("RecordBatch error: {e}"))?;
+
+    Ok((schema, vec![batch]))
+}

crates/ingestd/src/lib.rs

@@ -1,7 +1,5 @@
+pub mod chatgpt_parser;
 pub mod db_ingest;
-pub mod my_stream;
-pub mod pg_stream;
-pub mod schedule;
 pub mod detect;
 pub mod csv_ingest;
 pub mod json_ingest;

crates/ingestd/src/my_stream.rs

@@ -1,410 +0,0 @@
-/// Streaming MySQL ingest.
-///
-/// Mirrors `pg_stream` for MySQL sources. Same OFFSET-paginated strategy,
-/// same Parquet-streaming shape. Uses `mysql_async` (pure-rust) so we
-/// don't need a C client library at build time.
-///
-/// Type mapping follows ADR-010 (default to string on ambiguity):
-/// - Booleans (TINYINT(1)) and integer types map to Arrow Int32/Int64
-/// - Floating point and decimals → Float64
-/// - Everything else (text, varchar, json, date, timestamp) → Utf8
-///
-/// What's deliberately not supported (yet):
-/// - TLS connections — `minimal` feature is plain TCP only. Upgrade
-///   when a tenant actually needs it.
-/// - Keyset pagination — OFFSET scans are O(N²) at multi-million-row
-///   scale; same caveat as `pg_stream`.
-/// - BINARY/BLOB columns — currently rendered as base64 or empty string
-///   via Display-fallback.
-
-use arrow::array::{ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray};
-use arrow::datatypes::{DataType, Field, Schema};
-use mysql_async::prelude::*;
-use mysql_async::{Conn, Opts, Row, Value};
-use parquet::arrow::ArrowWriter;
-use std::sync::Arc;
-
-/// Request shape for MySQL streaming ingest.
-#[derive(Debug, Clone, serde::Deserialize)]
-pub struct MyStreamRequest {
-    /// mysql://user:pass@host:port/db
-    pub dsn: String,
-    pub table: String,
-    #[serde(default)]
-    pub dataset_name: Option<String>,
-    /// Rows per fetch. Default 10_000.
-    #[serde(default)]
-    pub batch_size: Option<usize>,
-    /// Column to ORDER BY for stable pagination. If omitted, the first
-    /// column returned by the schema probe is used.
-    #[serde(default)]
-    pub order_by: Option<String>,
-    /// Hard cap on total rows (for sampling / previews).
-    #[serde(default)]
-    pub limit: Option<usize>,
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct MyStreamResult {
-    pub table: String,
-    pub rows: usize,
-    pub batches: usize,
-    pub columns: usize,
-    pub schema: Vec<String>,
-    pub parquet_bytes: u64,
-    pub duration_secs: f32,
-}
-
-/// Parsed DSN pieces. Kept local (rather than reusing pg's DbConfig) so
-/// the MySQL connector doesn't depend on the PG path.
-#[derive(Debug, Clone)]
-pub struct MyConfig {
-    pub host: String,
-    pub port: u16,
-    pub user: String,
-    pub password: String,
-    pub database: String,
-}
-
-impl MyConfig {
-    /// Build a mysql_async `Opts` from the parsed config.
-    pub fn to_opts(&self) -> Opts {
-        let url = if self.password.is_empty() {
-            format!(
-                "mysql://{}@{}:{}/{}",
-                percent(&self.user), percent(&self.host), self.port, percent(&self.database),
-            )
-        } else {
-            format!(
-                "mysql://{}:{}@{}:{}/{}",
-                percent(&self.user), percent(&self.password),
-                percent(&self.host), self.port, percent(&self.database),
-            )
-        };
-        Opts::from_url(&url).expect("MyConfig.to_opts: rebuilt URL should parse")
-    }
-}
-
-/// Minimal URL-encoder for the few characters that commonly appear in
-/// MySQL passwords. mysql_async's URL parser expects valid URL-encoded
-/// components.
-fn percent(s: &str) -> String {
-    let mut out = String::with_capacity(s.len());
-    for c in s.chars() {
-        match c {
-            '@' | ':' | '/' | '?' | '#' | '[' | ']' | ' ' | '%' => {
-                out.push_str(&format!("%{:02X}", c as u32));
-            }
-            _ => out.push(c),
-        }
-    }
-    out
-}
-
-/// Parse a mysql:// DSN.
-/// Supports: mysql://[user[:password]@]host[:port][/db]
-pub fn parse_dsn(dsn: &str) -> Result<MyConfig, String> {
-    let rest = dsn
-        .strip_prefix("mysql://")
-        .ok_or_else(|| "DSN must start with mysql://".to_string())?;
-
-    let (auth_host, database) = match rest.split_once('/') {
-        Some((ah, db)) => (ah, db.split('?').next().unwrap_or(db).to_string()),
-        None => (rest, String::new()),
-    };
-
-    let (userpass, hostport) = match auth_host.rsplit_once('@') {
-        Some((up, hp)) => (Some(up), hp),
-        None => (None, auth_host),
-    };
-
-    let (user, password) = match userpass {
-        Some(up) => match up.split_once(':') {
-            Some((u, p)) => (u.to_string(), p.to_string()),
-            None => (up.to_string(), String::new()),
-        },
-        None => ("root".to_string(), String::new()),
-    };
-
-    let (host, port) = match hostport.rsplit_once(':') {
-        Some((h, p)) => (
-            h.to_string(),
-            p.parse::<u16>().map_err(|_| format!("invalid port in DSN: {p}"))?,
-        ),
-        None => (hostport.to_string(), 3306),
-    };
-
-    if host.is_empty() {
-        return Err("DSN has no host".into());
-    }
-    if database.is_empty() {
-        return Err("DSN has no database path (mysql://... /db)".into());
-    }
-
-    Ok(MyConfig { host, port, user, password, database })
-}
-
-/// Stream a MySQL table as Parquet bytes.
-pub async fn stream_table_to_parquet(
-    req: &MyStreamRequest,
-) -> Result<(bytes::Bytes, MyStreamResult), String> {
-    let t0 = std::time::Instant::now();
-    let config = parse_dsn(&req.dsn)?;
-    let batch_size = req.batch_size.unwrap_or(10_000).max(1);
-
-    let mut conn = Conn::new(config.to_opts()).await
-        .map_err(|e| format!("mysql connect: {e}"))?;
-
-    // Probe columns via information_schema — gives real type names that
-    // we can map to Arrow dtypes without needing to fetch a row first.
-    let columns = probe_columns(&mut conn, &config.database, &req.table).await?;
-    if columns.is_empty() {
-        return Err(format!("table '{}' not found or has no columns", req.table));
-    }
-
-    let arrow_fields: Vec<Field> = columns
-        .iter()
-        .map(|(name, ty)| Field::new(name, mysql_type_to_arrow(ty), true))
-        .collect();
-    let schema = Arc::new(Schema::new(arrow_fields));
-    let schema_report: Vec<String> = columns
-        .iter()
-        .map(|(n, t)| format!("{}:{}", n, t))
-        .collect();
-
-    let order_col = req.order_by.clone().unwrap_or_else(|| columns[0].0.clone());
-
-    let mut buf: Vec<u8> = Vec::with_capacity(1024 * 1024);
-    let mut writer = ArrowWriter::try_new(&mut buf, schema.clone(), None)
-        .map_err(|e| format!("ArrowWriter init: {e}"))?;
-
-    let mut total_rows: usize = 0;
-    let mut batch_count: usize = 0;
-    let row_cap = req.limit.unwrap_or(usize::MAX);
-
-    loop {
-        let remaining = row_cap.saturating_sub(total_rows);
-        if remaining == 0 { break; }
-        let fetch = remaining.min(batch_size);
-
-        // Backticks are MySQL's identifier quote. Forbid backticks in
-        // table/column names to prevent injection — neither pg nor us
-        // should be accepting those anyway.
-        if req.table.contains('`') || order_col.contains('`') {
-            return Err("table or order_by column contains backticks — refused".into());
-        }
-        let sql = format!(
-            "SELECT * FROM `{}` ORDER BY `{}` LIMIT {} OFFSET {}",
-            req.table, order_col, fetch, total_rows,
-        );
-
-        let rows: Vec<Row> = conn.query(&sql).await
-            .map_err(|e| format!("fetch batch at offset {total_rows}: {e}"))?;
-
-        if rows.is_empty() { break; }
-        let n = rows.len();
-
-        let arrays: Vec<ArrayRef> = columns
-            .iter()
-            .enumerate()
-            .map(|(idx, (_, ty))| rows_to_column(&rows, idx, ty))
-            .collect::<Result<_, _>>()?;
-        let batch = RecordBatch::try_new(schema.clone(), arrays)
-            .map_err(|e| format!("RecordBatch: {e}"))?;
-        writer.write(&batch).map_err(|e| format!("ArrowWriter::write: {e}"))?;
-
-        total_rows += n;
-        batch_count += 1;
-        tracing::info!(
-            "mysql stream '{}': fetched batch {} ({} rows, total {})",
-            req.table, batch_count, n, total_rows,
-        );
-
-        if n < fetch { break; }
-    }
-
-    writer.close().map_err(|e| format!("ArrowWriter::close: {e}"))?;
-    conn.disconnect().await.ok();
-
-    let result = MyStreamResult {
-        table: req.table.clone(),
-        rows: total_rows,
-        batches: batch_count,
-        columns: columns.len(),
-        schema: schema_report,
-        parquet_bytes: buf.len() as u64,
-        duration_secs: t0.elapsed().as_secs_f32(),
-    };
-
-    Ok((bytes::Bytes::from(buf), result))
-}
-
-async fn probe_columns(
-    conn: &mut Conn,
-    schema: &str,
-    table: &str,
-) -> Result<Vec<(String, String)>, String> {
-    let sql = format!(
-        "SELECT COLUMN_NAME, DATA_TYPE FROM information_schema.columns \
-         WHERE table_schema = '{}' AND table_name = '{}' \
-         ORDER BY ORDINAL_POSITION",
-        schema.replace('\'', "''"),
-        table.replace('\'', "''"),
-    );
-    let rows: Vec<(String, String)> = conn.query(&sql).await
-        .map_err(|e| format!("probe columns: {e}"))?;
-    Ok(rows)
-}
-
-/// MySQL data type string → Arrow DataType. Conservative: anything we
-/// don't recognize becomes Utf8 (ADR-010).
-fn mysql_type_to_arrow(ty: &str) -> DataType {
-    match ty.to_lowercase().as_str() {
-        "tinyint" | "smallint" | "mediumint" | "int" | "integer" => DataType::Int32,
-        "bigint" => DataType::Int64,
-        "float" | "double" | "decimal" | "numeric" | "real" => DataType::Float64,
-        "bit" | "bool" | "boolean" => DataType::Boolean,
-        _ => DataType::Utf8,
-    }
-}
-
-/// Convert a single column slice of MySQL rows into an Arrow array.
-fn rows_to_column(
-    rows: &[Row],
-    idx: usize,
-    ty: &str,
-) -> Result<ArrayRef, String> {
-    let arrow_ty = mysql_type_to_arrow(ty);
-    match arrow_ty {
-        DataType::Boolean => {
-            let v: Vec<Option<bool>> = rows.iter().map(|r| cell_as_bool(r, idx)).collect();
-            Ok(Arc::new(BooleanArray::from(v)))
-        }
-        DataType::Int32 => {
-            let v: Vec<Option<i32>> = rows.iter().map(|r| cell_as_i64(r, idx).map(|n| n as i32)).collect();
-            Ok(Arc::new(Int32Array::from(v)))
-        }
-        DataType::Int64 => {
-            let v: Vec<Option<i64>> = rows.iter().map(|r| cell_as_i64(r, idx)).collect();
-            Ok(Arc::new(Int64Array::from(v)))
-        }
-        DataType::Float64 => {
-            let v: Vec<Option<f64>> = rows.iter().map(|r| cell_as_f64(r, idx)).collect();
-            Ok(Arc::new(Float64Array::from(v)))
-        }
-        _ => {
-            let v: Vec<Option<String>> = rows.iter().map(|r| cell_as_string(r, idx)).collect();
-            Ok(Arc::new(StringArray::from(v)))
-        }
-    }
-}
-
-fn cell(r: &Row, idx: usize) -> &Value { r.as_ref(idx).unwrap_or(&Value::NULL) }
-
-fn cell_as_bool(r: &Row, idx: usize) -> Option<bool> {
-    match cell(r, idx) {
-        Value::NULL => None,
-        Value::Int(n) => Some(*n != 0),
-        Value::UInt(n) => Some(*n != 0),
-        Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| {
-            match s.to_ascii_lowercase().as_str() {
-                "true" | "1" | "y" | "yes" => Some(true),
-                "false" | "0" | "n" | "no" => Some(false),
-                _ => None,
-            }
-        }),
-        _ => None,
-    }
-}
-
-fn cell_as_i64(r: &Row, idx: usize) -> Option<i64> {
-    match cell(r, idx) {
-        Value::NULL => None,
-        Value::Int(n) => Some(*n),
-        Value::UInt(n) => i64::try_from(*n).ok(),
-        Value::Float(f) => Some(*f as i64),
-        Value::Double(f) => Some(*f as i64),
-        Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| s.parse().ok()),
-        _ => None,
-    }
-}
-
-fn cell_as_f64(r: &Row, idx: usize) -> Option<f64> {
-    match cell(r, idx) {
-        Value::NULL => None,
-        Value::Int(n) => Some(*n as f64),
-        Value::UInt(n) => Some(*n as f64),
-        Value::Float(f) => Some(*f as f64),
-        Value::Double(f) => Some(*f),
-        Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| s.parse().ok()),
-        _ => None,
-    }
-}
-
-fn cell_as_string(r: &Row, idx: usize) -> Option<String> {
-    match cell(r, idx) {
-        Value::NULL => None,
-        Value::Bytes(b) => Some(String::from_utf8_lossy(b).into_owned()),
-        Value::Int(n) => Some(n.to_string()),
-        Value::UInt(n) => Some(n.to_string()),
-        Value::Float(f) => Some(f.to_string()),
-        Value::Double(f) => Some(f.to_string()),
-        Value::Date(y, mo, d, h, mi, s, _us) => {
-            Some(format!("{:04}-{:02}-{:02} {:02}:{:02}:{:02}", y, mo, d, h, mi, s))
-        }
-        Value::Time(neg, days, h, mi, s, _us) => {
-            let sign = if *neg { "-" } else { "" };
-            Some(format!("{}{}d {:02}:{:02}:{:02}", sign, days, h, mi, s))
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn parse_dsn_full() {
-        let c = parse_dsn("mysql://daisy:secret@db.example.com:3307/my_db").unwrap();
-        assert_eq!(c.host, "db.example.com");
-        assert_eq!(c.port, 3307);
-        assert_eq!(c.user, "daisy");
-        assert_eq!(c.password, "secret");
-        assert_eq!(c.database, "my_db");
-    }
-
-    #[test]
-    fn parse_dsn_default_port() {
-        let c = parse_dsn("mysql://root@localhost/shop").unwrap();
-        assert_eq!(c.port, 3306);
-        assert_eq!(c.user, "root");
-        assert_eq!(c.password, "");
-    }
-
-    #[test]
-    fn parse_dsn_no_auth() {
-        let c = parse_dsn("mysql://127.0.0.1:3306/analytics").unwrap();
-        assert_eq!(c.user, "root");
-        assert_eq!(c.host, "127.0.0.1");
-    }
-
-    #[test]
-    fn parse_dsn_rejects_non_mysql() {
-        assert!(parse_dsn("postgresql://host/db").is_err());
-    }
-
-    #[test]
-    fn parse_dsn_requires_db() {
-        assert!(parse_dsn("mysql://localhost:3306").is_err());
-    }
-
-    #[test]
-    fn type_map_basics() {
-        assert_eq!(mysql_type_to_arrow("int"), DataType::Int32);
-        assert_eq!(mysql_type_to_arrow("BIGINT"), DataType::Int64);
-        assert_eq!(mysql_type_to_arrow("decimal"), DataType::Float64);
-        assert_eq!(mysql_type_to_arrow("varchar"), DataType::Utf8);
-        assert_eq!(mysql_type_to_arrow("json"), DataType::Utf8);
-        assert_eq!(mysql_type_to_arrow("bool"), DataType::Boolean);
-    }
-}

crates/ingestd/src/pdf_ingest.rs

@@ -4,19 +4,8 @@ use arrow::record_batch::RecordBatch;
 use std::sync::Arc;
 
 /// Extract text from a PDF file.
-///
-/// Two-tier approach:
-/// 1. **Text extraction (lopdf)** — fast, works on digital/native PDFs.
-///    Returns one row per page: (source_file, page_number, text_content).
-/// 2. **OCR fallback (Tesseract)** — if text extraction yields zero
-///    pages (common with scanned documents, image-only PDFs, or PDFs
-///    exported from scanners without an OCR layer), we extract embedded
-///    images from the PDF and run Tesseract on each. Falls back
-///    gracefully if Tesseract isn't installed (returns an error
-///    directing the operator to install it).
-///
-/// Schema is identical for both paths so downstream consumers (chunker,
-/// vectord, queryd) don't need to know how the text was produced.
+/// Returns one row per page: (page_number, text_content).
+/// This handles text-based PDFs. Scanned/image PDFs need OCR (not implemented yet).
 pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
     let doc = lopdf::Document::load_mem(content)
         .map_err(|e| format!("PDF load error: {e}"))?;
@@ -37,35 +26,13 @@ pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>,
         }
     }
 
-    // Tier 2: OCR fallback for scanned pages.
     if page_numbers.is_empty() {
-        tracing::info!(
-            "no extractable text in '{}' — attempting OCR on {} pages",
-            source_filename, pages.len(),
-        );
-        let ocr_results = ocr_pdf_images(&doc, source_filename)?;
-        if ocr_results.is_empty() {
-            return Err(
-                "PDF contains no extractable text and OCR produced no output. \
-                 Ensure tesseract is installed (`apt install tesseract-ocr tesseract-ocr-eng`)."
-                .into(),
-            );
-        }
-        for (pg, text) in ocr_results {
-            page_numbers.push(pg);
-            page_texts.push(text);
-            sources.push(source_filename.to_string());
-        }
-        tracing::info!(
-            "OCR recovered {} pages from '{}'", page_numbers.len(), source_filename,
-        );
-    } else {
-        tracing::info!(
-            "extracted {} pages with text from PDF '{}'",
-            page_numbers.len(), source_filename,
-        );
+        // PDF has no extractable text — likely scanned/image
+        return Err("PDF contains no extractable text (may be scanned/image — OCR not yet supported)".into());
     }
 
+    tracing::info!("extracted {} pages with text from PDF '{}'", page_numbers.len(), source_filename);
+
     let schema = Arc::new(Schema::new(vec![
         Field::new("source_file", DataType::Utf8, false),
         Field::new("page_number", DataType::Int32, false),
@@ -83,197 +50,3 @@ pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>,
 
     Ok((schema, vec![batch]))
 }
-
-/// Extract embedded images from the PDF, write each to a temp file,
-/// run Tesseract, collect the OCR'd text. Returns (page_number, text)
-/// pairs for pages that yielded non-empty output.
-///
-/// Image extraction strategy: walk every page's resources looking for
-/// XObject streams with /Subtype /Image. Decode the raw stream bytes
-/// (lopdf handles FlateDecode/DCTDecode) and write them as-is to a
-/// temp file with the appropriate extension. Tesseract handles JPEG,
-/// PNG, TIFF, and BMP natively.
-fn ocr_pdf_images(
-    doc: &lopdf::Document,
-    source_filename: &str,
-) -> Result<Vec<(i32, String)>, String> {
-    if !tesseract_available() {
-        return Err(
-            "tesseract not found on PATH — install with `apt install tesseract-ocr tesseract-ocr-eng`"
-                .into(),
-        );
-    }
-
-    let pages = doc.get_pages();
-    let mut results: Vec<(i32, String)> = Vec::new();
-
-    for (&page_num, &page_id) in &pages {
-        let images = extract_page_images(doc, page_id);
-        if images.is_empty() {
-            continue;
-        }
-
-        // OCR each image on this page and concatenate the text.
-        let mut page_text = String::new();
-        for (idx, img_bytes) in images.iter().enumerate() {
-            match ocr_image_bytes(img_bytes, page_num, idx) {
-                Ok(text) if !text.trim().is_empty() => {
-                    if !page_text.is_empty() { page_text.push('\n'); }
-                    page_text.push_str(text.trim());
-                }
-                Ok(_) => {} // empty OCR — skip
-                Err(e) => {
-                    tracing::warn!(
-                        "OCR failed on '{}' page {} image {}: {e}",
-                        source_filename, page_num, idx,
-                    );
-                }
-            }
-        }
-        if !page_text.is_empty() {
-            results.push((page_num as i32, page_text));
-        }
-    }
-    Ok(results)
-}
-
-/// Extract raw image bytes from a PDF page's XObject resources.
-/// Handles the common case: page /Resources → /XObject dict → streams
-/// with /Subtype /Image.
-fn extract_page_images(
-    doc: &lopdf::Document,
-    page_id: lopdf::ObjectId,
-) -> Vec<Vec<u8>> {
-    let mut images = Vec::new();
-
-    let page_obj = match doc.get_object(page_id) {
-        Ok(o) => o,
-        Err(_) => return images,
-    };
-    let page_dict = match page_obj.as_dict() {
-        Ok(d) => d,
-        Err(_) => return images,
-    };
-
-    // Resolve /Resources → /XObject
-    let resources = page_dict
-        .get(b"Resources")
-        .ok()
-        .and_then(|r| doc.dereference(r).ok())
-        .map(|(_, o)| o.clone());
-    let xobjects = resources
-        .as_ref()
-        .and_then(|r| r.as_dict().ok())
-        .and_then(|d| d.get(b"XObject").ok())
-        .and_then(|x| doc.dereference(x).ok())
-        .map(|(_, o)| o.clone());
-
-    let xobj_dict = match xobjects.as_ref().and_then(|x| x.as_dict().ok()) {
-        Some(d) => d,
-        None => return images,
-    };
-
-    for (_name, obj_ref) in xobj_dict.iter() {
-        let obj = match doc.dereference(obj_ref) {
-            Ok((_, o)) => o.clone(),
-            Err(_) => continue,
-        };
-        let stream = match obj.as_stream() {
-            Ok(s) => s,
-            Err(_) => continue,
-        };
-        // Check /Subtype == /Image
-        let is_image = stream.dict.get(b"Subtype")
-            .ok()
-            .and_then(|s| s.as_name().ok())
-            .map(|n| n == b"Image")
-            .unwrap_or(false);
-        if !is_image {
-            continue;
-        }
-        // Get decoded content — lopdf handles FlateDecode/DCTDecode.
-        let content = match stream.decompressed_content() {
-            Ok(c) => c,
-            Err(_) => {
-                // Try raw content as fallback (some DCT streams are stored raw).
-                stream.content.clone()
-            }
-        };
-        if !content.is_empty() {
-            images.push(content);
-        }
-    }
-    images
-}
-
-/// Run Tesseract on raw image bytes. Writes to a temp file, shells out,
-/// reads the output. Returns the OCR'd text.
-fn ocr_image_bytes(
-    img_bytes: &[u8],
-    page_num: u32,
-    img_idx: usize,
-) -> Result<String, String> {
-    use std::io::Write;
-    use std::process::Command;
-
-    let tmp_dir = std::env::temp_dir();
-    // Detect image format from magic bytes to pick the right extension.
-    let ext = if img_bytes.starts_with(&[0xFF, 0xD8]) {
-        "jpg"
-    } else if img_bytes.starts_with(&[0x89, 0x50, 0x4E, 0x47]) {
-        "png"
-    } else if img_bytes.starts_with(b"II") || img_bytes.starts_with(b"MM") {
-        "tiff"
-    } else {
-        // Tesseract can often handle raw pixel data with the right options,
-        // but writing as PNG is safer. For truly unknown formats, try PNG
-        // wrapping; if Tesseract can't read it, it'll error cleanly.
-        "png"
-    };
-
-    let img_path = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}.{ext}"));
-    let out_base = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}_out"));
-    let out_path = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}_out.txt"));
-
-    // Write image bytes to temp file.
-    let mut f = std::fs::File::create(&img_path)
-        .map_err(|e| format!("create temp image: {e}"))?;
-    f.write_all(img_bytes)
-        .map_err(|e| format!("write temp image: {e}"))?;
-    drop(f);
-
-    // Run Tesseract.
-    let status = Command::new("tesseract")
-        .arg(&img_path)
-        .arg(&out_base)
-        .arg("--oem").arg("3")         // LSTM + legacy combined
-        .arg("--psm").arg("6")         // assume uniform block of text
-        .arg("-l").arg("eng")
-        .stdout(std::process::Stdio::null())
-        .stderr(std::process::Stdio::null())
-        .status()
-        .map_err(|e| format!("tesseract exec: {e}"))?;
-
-    // Cleanup input regardless of outcome.
-    let _ = std::fs::remove_file(&img_path);
-
-    if !status.success() {
-        let _ = std::fs::remove_file(&out_path);
-        return Err(format!("tesseract exited with status {status}"));
-    }
-
-    let text = std::fs::read_to_string(&out_path).unwrap_or_default();
-    let _ = std::fs::remove_file(&out_path);
-    Ok(text)
-}
-
-/// Quick check: is `tesseract` on PATH?
-fn tesseract_available() -> bool {
-    std::process::Command::new("tesseract")
-        .arg("--version")
-        .stdout(std::process::Stdio::null())
-        .stderr(std::process::Stdio::null())
-        .status()
-        .map(|s| s.success())
-        .unwrap_or(false)
-}

crates/ingestd/src/pg_stream.rs

@@ -1,308 +0,0 @@
-/// Streaming PostgreSQL ingest.
-///
-/// The original `db_ingest::import_postgres_table` loads every row into
-/// memory before emitting Parquet — fine for small tables, blows up on
-/// 1M+ rows. This module paginates via `ORDER BY <pk> LIMIT N OFFSET M`
-/// and streams batches into an `ArrowWriter`, closing once exhausted.
-///
-/// Pagination is OFFSET-based (not keyset) — simpler, works on any table
-/// without needing to know the PK type. OFFSET N scans N rows per fetch,
-/// which becomes quadratic for very large tables. Upgrade to keyset when
-/// a user actually ingests something that hurts.
-
-use arrow::array::{ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray};
-use arrow::datatypes::{DataType, Field, Schema};
-use parquet::arrow::ArrowWriter;
-use std::sync::Arc;
-use tokio_postgres::{Client, NoTls, types::Type as PgType};
-
-use crate::db_ingest::DbConfig;
-
-/// Request shape for streaming ingest — takes a DSN string and optional
-/// tuning knobs.
-#[derive(Debug, Clone, serde::Deserialize)]
-pub struct PgStreamRequest {
-    /// postgresql://user:pass@host:port/db — alternative to DbConfig.
-    pub dsn: String,
-    pub table: String,
-    #[serde(default)]
-    pub dataset_name: Option<String>,
-    /// Rows per fetch. Default 10_000.
-    #[serde(default)]
-    pub batch_size: Option<usize>,
-    /// Column to ORDER BY for stable pagination. If omitted, the first
-    /// column returned by the schema probe is used.
-    #[serde(default)]
-    pub order_by: Option<String>,
-    /// Hard cap on total rows (for sampling / previews).
-    #[serde(default)]
-    pub limit: Option<usize>,
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct PgStreamResult {
-    pub table: String,
-    pub rows: usize,
-    pub batches: usize,
-    pub columns: usize,
-    pub schema: Vec<String>,
-    pub parquet_bytes: u64,
-    pub duration_secs: f32,
-}
-
-/// Parse a postgresql:// DSN into a DbConfig.
-/// Supports: postgresql://[user[:password]@]host[:port][/db]
-/// Does NOT support: query parameters (sslmode etc) — add if needed.
-pub fn parse_dsn(dsn: &str) -> Result<DbConfig, String> {
-    let rest = dsn
-        .strip_prefix("postgresql://")
-        .or_else(|| dsn.strip_prefix("postgres://"))
-        .ok_or_else(|| "DSN must start with postgresql:// or postgres://".to_string())?;
-
-    // Split off path (database) first.
-    let (auth_host, database) = match rest.split_once('/') {
-        Some((ah, db)) => (ah, db.split('?').next().unwrap_or(db).to_string()),
-        None => (rest, "postgres".to_string()),
-    };
-
-    // Split user[:password] from host[:port].
-    let (userpass, hostport) = match auth_host.rsplit_once('@') {
-        Some((up, hp)) => (Some(up), hp),
-        None => (None, auth_host),
-    };
-
-    let (user, password) = match userpass {
-        Some(up) => match up.split_once(':') {
-            Some((u, p)) => (u.to_string(), p.to_string()),
-            None => (up.to_string(), String::new()),
-        },
-        None => ("postgres".to_string(), String::new()),
-    };
-
-    let (host, port) = match hostport.rsplit_once(':') {
-        Some((h, p)) => (
-            h.to_string(),
-            p.parse::<u16>().map_err(|_| format!("invalid port in DSN: {p}"))?,
-        ),
-        None => (hostport.to_string(), 5432),
-    };
-
-    if host.is_empty() {
-        return Err("DSN has no host".into());
-    }
-
-    Ok(DbConfig { host, port, database, user, password })
-}
-
-/// Stream a Postgres table as Parquet bytes.
-///
-/// Returns the full Parquet payload + summary stats. The payload is
-/// written to an in-memory buffer via `ArrowWriter` so each batch gets
-/// columnar compression as it arrives — memory footprint stays roughly
-/// at one batch plus Parquet footer state.
-pub async fn stream_table_to_parquet(
-    req: &PgStreamRequest,
-) -> Result<(bytes::Bytes, PgStreamResult), String> {
-    let t0 = std::time::Instant::now();
-    let config = parse_dsn(&req.dsn)?;
-    let batch_size = req.batch_size.unwrap_or(10_000).max(1);
-
-    let (client, connection) = tokio_postgres::connect(&config.connection_string(), NoTls)
-        .await
-        .map_err(|e| format!("postgres connect: {e}"))?;
-    tokio::spawn(async move {
-        if let Err(e) = connection.await {
-            tracing::error!("pg connection: {e}");
-        }
-    });
-
-    // Probe schema.
-    let columns = probe_columns(&client, &req.table).await?;
-    if columns.is_empty() {
-        return Err(format!("table '{}' not found or has no columns", req.table));
-    }
-
-    let arrow_fields: Vec<Field> = columns
-        .iter()
-        .map(|(name, pg)| Field::new(name, pg_type_to_arrow(pg), true))
-        .collect();
-    let schema = Arc::new(Schema::new(arrow_fields));
-    let schema_report: Vec<String> = columns
-        .iter()
-        .map(|(n, t)| format!("{}:{}", n, t))
-        .collect();
-
-    // Pagination key: user-specified or first column.
-    let order_col = req.order_by.clone().unwrap_or_else(|| columns[0].0.clone());
-
-    // Stream batches into ArrowWriter.
-    let mut buf: Vec<u8> = Vec::with_capacity(1024 * 1024);
-    let mut writer = ArrowWriter::try_new(&mut buf, schema.clone(), None)
-        .map_err(|e| format!("ArrowWriter init: {e}"))?;
-
-    let mut total_rows: usize = 0;
-    let mut batch_count: usize = 0;
-    let row_cap = req.limit.unwrap_or(usize::MAX);
-
-    loop {
-        let remaining = row_cap.saturating_sub(total_rows);
-        if remaining == 0 { break; }
-        let fetch = remaining.min(batch_size);
-
-        let sql = format!(
-            "SELECT * FROM \"{}\" ORDER BY \"{}\" LIMIT {} OFFSET {}",
-            req.table, order_col, fetch, total_rows,
-        );
-
-        let rows = client
-            .query(&sql, &[])
-            .await
-            .map_err(|e| format!("fetch batch at offset {total_rows}: {e}"))?;
-
-        if rows.is_empty() { break; }
-
-        let n = rows.len();
-        let arrays: Vec<ArrayRef> = columns
-            .iter()
-            .enumerate()
-            .map(|(idx, (_, pg))| rows_to_column(&rows, idx, pg))
-            .collect::<Result<_, _>>()?;
-        let batch = RecordBatch::try_new(schema.clone(), arrays)
-            .map_err(|e| format!("RecordBatch: {e}"))?;
-        writer.write(&batch).map_err(|e| format!("ArrowWriter::write: {e}"))?;
-
-        total_rows += n;
-        batch_count += 1;
-        tracing::info!(
-            "pg stream '{}': fetched batch {} ({} rows, total {})",
-            req.table, batch_count, n, total_rows,
-        );
-
-        if n < fetch { break; } // short read = end of table
-    }
-
-    writer.close().map_err(|e| format!("ArrowWriter::close: {e}"))?;
-    let parquet_bytes = buf.len() as u64;
-    let duration = t0.elapsed().as_secs_f32();
-
-    let result = PgStreamResult {
-        table: req.table.clone(),
-        rows: total_rows,
-        batches: batch_count,
-        columns: columns.len(),
-        schema: schema_report,
-        parquet_bytes,
-        duration_secs: duration,
-    };
-
-    Ok((bytes::Bytes::from(buf), result))
-}
-
-async fn probe_columns(client: &Client, table: &str) -> Result<Vec<(String, PgType)>, String> {
-    let stmt = client
-        .prepare(&format!("SELECT * FROM \"{}\" LIMIT 0", table))
-        .await
-        .map_err(|e| format!("prepare schema probe: {e}"))?;
-    Ok(stmt
-        .columns()
-        .iter()
-        .map(|c| (c.name().to_string(), c.type_().clone()))
-        .collect())
-}
-
-fn pg_type_to_arrow(pg: &PgType) -> DataType {
-    match *pg {
-        PgType::BOOL => DataType::Boolean,
-        PgType::INT2 | PgType::INT4 => DataType::Int32,
-        PgType::INT8 | PgType::OID => DataType::Int64,
-        PgType::FLOAT4 | PgType::FLOAT8 | PgType::NUMERIC => DataType::Float64,
-        _ => DataType::Utf8,
-    }
-}
-
-fn rows_to_column(
-    rows: &[tokio_postgres::Row],
-    idx: usize,
-    pg: &PgType,
-) -> Result<ArrayRef, String> {
-    match *pg {
-        PgType::BOOL => {
-            let v: Vec<Option<bool>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
-            Ok(Arc::new(BooleanArray::from(v)))
-        }
-        PgType::INT2 => {
-            let v: Vec<Option<i32>> = rows.iter()
-                .map(|r| r.try_get::<_, i16>(idx).ok().map(|x| x as i32)).collect();
-            Ok(Arc::new(Int32Array::from(v)))
-        }
-        PgType::INT4 => {
-            let v: Vec<Option<i32>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
-            Ok(Arc::new(Int32Array::from(v)))
-        }
-        PgType::INT8 | PgType::OID => {
-            let v: Vec<Option<i64>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
-            Ok(Arc::new(Int64Array::from(v)))
-        }
-        PgType::FLOAT4 => {
-            let v: Vec<Option<f64>> = rows.iter()
-                .map(|r| r.try_get::<_, f32>(idx).ok().map(|x| x as f64)).collect();
-            Ok(Arc::new(Float64Array::from(v)))
-        }
-        PgType::FLOAT8 => {
-            let v: Vec<Option<f64>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
-            Ok(Arc::new(Float64Array::from(v)))
-        }
-        _ => {
-            // Safe-default per ADR-010: everything else becomes string.
-            let v: Vec<Option<String>> = rows.iter().map(|r| {
-                r.try_get::<_, String>(idx).ok()
-                    .or_else(|| r.try_get::<_, serde_json::Value>(idx).ok().map(|j| j.to_string()))
-                    .or_else(|| {
-                        // Timestamps + uuid, rendered via Display.
-                        r.try_get::<_, chrono::DateTime<chrono::Utc>>(idx).ok()
-                            .map(|t| t.to_rfc3339())
-                    })
-                    .or_else(|| r.try_get::<_, uuid::Uuid>(idx).ok().map(|u| u.to_string()))
-                    .or(Some(String::new()))
-            }).collect();
-            Ok(Arc::new(StringArray::from(v)))
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn parse_dsn_full() {
-        let c = parse_dsn("postgresql://daisy:secret@db.example.com:6543/my_db").unwrap();
-        assert_eq!(c.host, "db.example.com");
-        assert_eq!(c.port, 6543);
-        assert_eq!(c.user, "daisy");
-        assert_eq!(c.password, "secret");
-        assert_eq!(c.database, "my_db");
-    }
-
-    #[test]
-    fn parse_dsn_minimal() {
-        let c = parse_dsn("postgresql://localhost/knowledge_base").unwrap();
-        assert_eq!(c.host, "localhost");
-        assert_eq!(c.port, 5432);
-        assert_eq!(c.user, "postgres");
-        assert_eq!(c.password, "");
-        assert_eq!(c.database, "knowledge_base");
-    }
-
-    #[test]
-    fn parse_dsn_no_password() {
-        let c = parse_dsn("postgres://postgres@127.0.0.1:5432/mydb").unwrap();
-        assert_eq!(c.user, "postgres");
-        assert_eq!(c.password, "");
-    }
-
-    #[test]
-    fn parse_dsn_rejects_non_pg() {
-        assert!(parse_dsn("http://host/db").is_err());
-    }
-}

crates/ingestd/src/pipeline.rs

@@ -28,29 +28,12 @@ pub struct IngestResult {
 }
 
 /// Full ingest pipeline: detect → parse → dedup → store → register.
-///
-/// Bucket name (federation layer 2) is recorded on the ObjectRef so the
-/// catalog knows which bucket holds the data — defaults to "primary" via
-/// the ingest service's resolve_bucket helper.
 pub async fn ingest_file(
     filename: &str,
     content: &[u8],
     dataset_name: Option<&str>,
     store: &Arc<dyn ObjectStore>,
     registry: &Registry,
-) -> Result<IngestResult, String> {
-    ingest_file_to_bucket(filename, content, dataset_name, "primary", store, registry).await
-}
-
-/// Same as `ingest_file` but with explicit target bucket name on the
-/// ObjectRef. Header-aware ingest endpoints call this directly.
-pub async fn ingest_file_to_bucket(
-    filename: &str,
-    content: &[u8],
-    dataset_name: Option<&str>,
-    bucket: &str,
-    store: &Arc<dyn ObjectStore>,
-    registry: &Registry,
 ) -> Result<IngestResult, String> {
     // 1. Detect file type
     let file_type = detect_file_type(filename, content);
@@ -60,7 +43,13 @@ pub async fn ingest_file_to_bucket(
     let (schema, batches) = match file_type {
         FileType::Csv => csv_ingest::parse_csv(content)?,
         FileType::Json | FileType::NdJson => json_ingest::parse_json(content)?,
-        FileType::Pdf => pdf_ingest::parse_pdf(content, filename)?,
+        FileType::Pdf => {
+            // Try ChatGPT parser first, fall back to generic PDF
+            match crate::chatgpt_parser::parse_chatgpt_pdf(content, filename) {
+                Ok(result) => result,
+                Err(_) => pdf_ingest::parse_pdf(content, filename)?,
+            }
+        }
         FileType::Text => parse_text_file(content, filename)?,
         FileType::Unknown => return Err(format!("unknown file type for '{filename}'")),
     };
@@ -115,7 +104,7 @@ pub async fn ingest_file_to_bucket(
     let schema_fp = fingerprint_schema(&schema);
     let now = chrono::Utc::now();
     let obj_ref = ObjectRef {
-        bucket: bucket.to_string(),
+        bucket: "data".to_string(),
         key: storage_key.clone(),
         size_bytes: parquet_size,
         created_at: now,
@@ -164,10 +153,6 @@ pub async fn ingest_file_to_bucket(
         ..Default::default()
     }).await;
 
-    // Phase C: if this dataset already had embeddings, they're now stale.
-    // mark_embeddings_stale is a no-op for never-embedded datasets.
-    let _ = registry.mark_embeddings_stale(&safe_name).await;
-
     Ok(IngestResult {
         dataset_name: safe_name,
         file_type: format!("{:?}", file_type),

crates/ingestd/src/schedule.rs

@@ -1,725 +0,0 @@
-//! Scheduled ingest — the production-substrate piece that turns the
-//! lakehouse from "manual API toolkit" into a system that runs itself.
-//!
-//! A `ScheduleDef` declares "what to ingest" + "when to fire." A
-//! background `Scheduler` task wakes periodically, asks each enabled
-//! schedule whether it's due, and runs the matching ingest path. Outcomes
-//! land back on the schedule so operators can see what happened without
-//! `journalctl` archaeology.
-//!
-//! Storage shape: one JSON file per schedule at
-//! `primary://_schedules/{id}.json`. Serializable via serde, rebuilt on
-//! startup. Same write-once shape as the rest of the catalog — no DB.
-//!
-//! Concurrency model: single-flight per schedule_id. The scheduler holds
-//! a `HashSet<String>` of currently-running ids; a tick that finds a
-//! schedule already in flight skips it (no queueing — a slow ingest
-//! shouldn't pile up runs). Different schedules run concurrently.
-//!
-//! What's deliberately not in scope here:
-//! - Sub-minute cron precision. 5-field Unix cron is supported; seconds
-//!   are pinned to 0. Intervals cover sub-minute cases; cron is
-//!   easy to bolt on later.
-//! - Backoff / retry policies. A failed run records the failure and
-//!   schedules `next_run_at` as if it succeeded — no exponential backoff.
-//! - Distributed coordination. Single-machine scheduler; no leader
-//!   election. If you run two gateway instances they'll both fire.
-
-use chrono::{DateTime, Duration, Utc};
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::collections::{HashMap, HashSet};
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use storaged::ops;
-
-const PREFIX: &str = "_schedules";
-
-// =================== Public types ===================
-
-/// What kind of ingest the schedule fires.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(tag = "type", rename_all = "snake_case")]
-pub enum ScheduleKind {
-    /// Pull a MySQL/MariaDB table via the my_stream path.
-    Mysql {
-        /// `mysql://user:pass@host:port/db` — DSN with full credentials.
-        /// Stored encrypted-at-rest is a future concern; for now it lives
-        /// in plain JSON in the catalog. Don't put production root creds
-        /// here without disk-level encryption.
-        dsn: String,
-        table: String,
-        #[serde(default)]
-        dataset_name: Option<String>,
-        #[serde(default)]
-        batch_size: Option<usize>,
-        #[serde(default)]
-        order_by: Option<String>,
-    },
-    /// Pull a Postgres table via the pg_stream path.
-    Postgres {
-        dsn: String,
-        table: String,
-        #[serde(default)]
-        dataset_name: Option<String>,
-        #[serde(default)]
-        batch_size: Option<usize>,
-        #[serde(default)]
-        order_by: Option<String>,
-    },
-}
-
-impl ScheduleKind {
-    pub fn label(&self) -> String {
-        match self {
-            ScheduleKind::Mysql { table, .. } => format!("mysql:{table}"),
-            ScheduleKind::Postgres { table, .. } => format!("postgres:{table}"),
-        }
-    }
-}
-
-/// When the schedule fires.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(tag = "type", rename_all = "snake_case")]
-pub enum ScheduleTrigger {
-    /// Run every N seconds. Time anchor = compute_next_run_at decision.
-    Interval { secs: u64 },
-    /// 5-field Unix cron expression: "min hour dom month dow".
-    /// Seconds are implicitly 0 (minute-granularity scheduling).
-    /// Examples: "15 14 * * *" = 14:15 UTC daily, "0 */6 * * *" = every
-    /// 6 hours on the hour, "0 9 * * 1-5" = 09:00 UTC weekdays.
-    /// Timezone: UTC always — the scheduler's clock is UTC and we don't
-    /// carry a per-schedule timezone field.
-    Cron { expr: String },
-}
-
-/// Outcome of a single run, kept on the schedule for observability.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct RunOutcome {
-    pub at: DateTime<Utc>,
-    pub success: bool,
-    pub message: String,
-    pub rows: Option<usize>,
-    pub duration_secs: f32,
-}
-
-/// One scheduled ingest definition.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ScheduleDef {
-    pub id: String,
-    pub kind: ScheduleKind,
-    pub trigger: ScheduleTrigger,
-    pub enabled: bool,
-    pub created_at: DateTime<Utc>,
-    #[serde(default)]
-    pub created_by: String,
-    /// When the next run is due. Recomputed after each completion via
-    /// `compute_next_run_at`. On creation: set to "now" so the schedule
-    /// fires on the next scheduler tick.
-    pub next_run_at: DateTime<Utc>,
-    #[serde(default)]
-    pub last_run_at: Option<DateTime<Utc>>,
-    #[serde(default)]
-    pub last_outcome: Option<RunOutcome>,
-    /// Total successful runs and total failures since creation. Useful
-    /// for at-a-glance "is this schedule healthy?" checks.
-    #[serde(default)]
-    pub run_count: u64,
-    #[serde(default)]
-    pub failure_count: u64,
-}
-
-impl ScheduleDef {
-    /// True if this schedule should fire at `now`. Honors `enabled`.
-    pub fn is_due(&self, now: DateTime<Utc>) -> bool {
-        self.enabled && now >= self.next_run_at
-    }
-}
-
-// =================== Trigger semantics ===================
-//
-// J: THIS IS YOURS TO IMPLEMENT
-// =====================================================
-//
-// `compute_next_run_at` decides when a schedule fires next. The current
-// state is the "since last completion" semantics — simple but has real
-// trade-offs vs other strategies:
-//
-// 1. SINCE LAST COMPLETION (current default below):
-//      next = completed_at + interval
-//    Easy to reason about. A long-running ingest delays the next fire
-//    by however long it took. Drift is bounded — the schedule never
-//    "falls behind" trying to catch up.
-//
-// 2. ANCHORED INTERVAL:
-//      next = previous_target + interval, regardless of when run finished
-//    Keeps wall-clock alignment ("every 30min on the :00 and :30").
-//    Risk: if ingest takes longer than the interval, the next run is
-//    "due immediately" and you can fire back-to-back forever.
-//
-// 3. ANCHORED + SKIP:
-//      Anchored, but skip a tick if the previous one is still running.
-//    Best of both, slightly more code.
-//
-// 4. CRON:
-//      Parse the expression, find the next match after `now`. Most
-//    expressive but pulls in a cron parser dep.
-//
-pub fn compute_next_run_at(
-    trigger: &ScheduleTrigger,
-    completed_at: DateTime<Utc>,
-    _previous_next: DateTime<Utc>,
-) -> DateTime<Utc> {
-    match trigger {
-        ScheduleTrigger::Interval { secs } => {
-            completed_at + Duration::seconds(*secs as i64)
-        }
-        ScheduleTrigger::Cron { expr } => {
-            next_cron_fire(expr, completed_at)
-                // Creation-time validation (see `validate_trigger`) rejects
-                // unparseable expressions, so this fallback is only reached
-                // if the schedule JSON was hand-edited on disk. One-hour
-                // window keeps a bad expr from fire-looping the system.
-                .unwrap_or(completed_at + Duration::hours(1))
-        }
-    }
-}
-
-/// Parse a Vixie/POSIX cron expression and return the next fire time
-/// strictly after `after`. Accepts 5-field (`min hour dom month dow`) and
-/// 6-field (with leading `sec`) forms natively — croner handles both.
-/// Day-of-week follows Unix convention: 0=Sun, 1=Mon, …, 6=Sat (7=Sun as
-/// synonym). Returns None if the expression fails to parse or produces no
-/// upcoming match (which can happen for year-bounded patterns in the past).
-fn next_cron_fire(expr: &str, after: DateTime<Utc>) -> Option<DateTime<Utc>> {
-    use std::str::FromStr;
-    let cron = croner::Cron::from_str(expr).ok()?;
-    cron.find_next_occurrence(&after, false).ok()
-}
-
-/// Reject a ScheduleTrigger whose content can't be interpreted — used by
-/// the HTTP handlers to fail fast at create/patch time rather than falling
-/// back silently at fire time. Interval triggers get a sanity gate
-/// (no zero secs); cron triggers get a full parse through croner.
-pub fn validate_trigger(trigger: &ScheduleTrigger) -> Result<(), String> {
-    match trigger {
-        ScheduleTrigger::Interval { secs } => {
-            if *secs == 0 {
-                return Err("interval secs must be > 0".into());
-            }
-            Ok(())
-        }
-        ScheduleTrigger::Cron { expr } => {
-            use std::str::FromStr;
-            croner::Cron::from_str(expr)
-                .map_err(|e| format!("invalid cron expression '{expr}': {e}"))?;
-            Ok(())
-        }
-    }
-}
-
-// =================== Persistence ===================
-
-/// Storage path for a schedule's JSON record.
-fn schedule_key(id: &str) -> String {
-    let safe: String = id
-        .chars()
-        .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
-        .collect();
-    format!("{PREFIX}/{safe}.json")
-}
-
-/// In-memory + on-disk schedule registry. The Scheduler reads from here;
-/// HTTP CRUD writes through here.
-#[derive(Clone)]
-pub struct ScheduleStore {
-    /// Bucket where schedule JSON lives. Always primary today — a
-    /// future enhancement could move to per-profile schedule buckets.
-    store: Arc<dyn ObjectStore>,
-    schedules: Arc<RwLock<HashMap<String, ScheduleDef>>>,
-}
-
-impl ScheduleStore {
-    pub fn new(store: Arc<dyn ObjectStore>) -> Self {
-        Self {
-            store,
-            schedules: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    /// Load every persisted schedule into memory. Call on startup.
-    pub async fn rebuild(&self) -> Result<usize, String> {
-        let keys = ops::list(&self.store, Some(&format!("{PREFIX}/")))
-            .await
-            .unwrap_or_default();
-        let mut count = 0usize;
-        for key in keys {
-            if !key.ends_with(".json") { continue; }
-            match ops::get(&self.store, &key).await {
-                Ok(bytes) => match serde_json::from_slice::<ScheduleDef>(&bytes) {
-                    Ok(def) => {
-                        self.schedules.write().await.insert(def.id.clone(), def);
-                        count += 1;
-                    }
-                    Err(e) => tracing::warn!("schedule {key}: malformed: {e}"),
-                },
-                Err(e) => tracing::warn!("schedule {key}: read failed: {e}"),
-            }
-        }
-        Ok(count)
-    }
-
-    pub async fn list(&self) -> Vec<ScheduleDef> {
-        let mut out: Vec<ScheduleDef> = self.schedules.read().await.values().cloned().collect();
-        out.sort_by(|a, b| a.id.cmp(&b.id));
-        out
-    }
-
-    pub async fn get(&self, id: &str) -> Option<ScheduleDef> {
-        self.schedules.read().await.get(id).cloned()
-    }
-
-    pub async fn put(&self, def: ScheduleDef) -> Result<ScheduleDef, String> {
-        let json = serde_json::to_vec_pretty(&def).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &schedule_key(&def.id), json.into()).await?;
-        self.schedules.write().await.insert(def.id.clone(), def.clone());
-        Ok(def)
-    }
-
-    pub async fn delete(&self, id: &str) -> Result<bool, String> {
-        // Object_store's delete is idempotent — silently OKs missing keys.
-        let _ = ops::delete(&self.store, &schedule_key(id)).await;
-        Ok(self.schedules.write().await.remove(id).is_some())
-    }
-
-    /// Update completion metadata + persist. Used by the scheduler
-    /// after a run finishes (success or failure).
-    pub async fn record_run(
-        &self,
-        id: &str,
-        outcome: RunOutcome,
-    ) -> Result<(), String> {
-        let mut guard = self.schedules.write().await;
-        let Some(def) = guard.get_mut(id) else {
-            return Err(format!("schedule '{id}' not registered"));
-        };
-        def.last_run_at = Some(outcome.at);
-        if outcome.success {
-            def.run_count += 1;
-        } else {
-            def.failure_count += 1;
-        }
-        def.next_run_at = compute_next_run_at(&def.trigger, outcome.at, def.next_run_at);
-        def.last_outcome = Some(outcome);
-
-        let json = serde_json::to_vec_pretty(def).map_err(|e| e.to_string())?;
-        let key = schedule_key(id);
-        let store = self.store.clone();
-        // Drop the write lock before async I/O.
-        drop(guard);
-        ops::put(&store, &key, json.into()).await?;
-        Ok(())
-    }
-}
-
-// =================== Scheduler task ===================
-
-/// Long-running task that fires due schedules. Spawned at gateway
-/// startup. Stop semantics: drop the handle (the loop exits when its
-/// store is the only reference left, or sooner if you wire a stop signal).
-pub struct Scheduler {
-    pub store: ScheduleStore,
-    pub ingest: SchedulerIngestDeps,
-    /// Polling cadence for the loop itself. The schedules' own intervals
-    /// are independent — this just controls how granularly we check
-    /// "is anything due?" Default 10s.
-    pub tick_secs: u64,
-}
-
-/// What the scheduler needs to actually run an ingest. We pass in only
-/// what's needed — keeps the dep set narrow.
-#[derive(Clone)]
-pub struct SchedulerIngestDeps {
-    pub store: Arc<dyn ObjectStore>,
-    pub registry: catalogd::registry::Registry,
-    pub buckets: Arc<storaged::registry::BucketRegistry>,
-    /// Optional Phase 16.5 hook — push a DatasetAppended event if any
-    /// HNSW index is bound to the dataset that just got refreshed.
-    pub agent_handle: Option<vectord::agent::AgentHandle>,
-    pub index_registry: vectord::index_registry::IndexRegistry,
-}
-
-impl Scheduler {
-    /// Spawn the loop in the background. Returns immediately.
-    pub fn spawn(self) {
-        tokio::spawn(async move {
-            self.run().await;
-        });
-    }
-
-    async fn run(self) {
-        tracing::info!("scheduler started — tick={}s", self.tick_secs);
-        let in_flight: Arc<RwLock<HashSet<String>>> = Arc::new(RwLock::new(HashSet::new()));
-
-        loop {
-            tokio::time::sleep(std::time::Duration::from_secs(self.tick_secs)).await;
-            let now = Utc::now();
-            let due: Vec<ScheduleDef> = self.store.list().await
-                .into_iter()
-                .filter(|s| s.is_due(now))
-                .collect();
-            if due.is_empty() { continue; }
-
-            for def in due {
-                {
-                    let in_flight_guard = in_flight.read().await;
-                    if in_flight_guard.contains(&def.id) {
-                        // Previous run still going — skip this tick.
-                        // We don't reschedule; next tick re-evaluates.
-                        continue;
-                    }
-                }
-
-                let store = self.store.clone();
-                let ingest = self.ingest.clone();
-                let in_flight = in_flight.clone();
-                let id = def.id.clone();
-
-                in_flight.write().await.insert(id.clone());
-
-                tokio::spawn(async move {
-                    let t0 = std::time::Instant::now();
-                    let outcome = run_schedule(&def, &ingest).await;
-                    let elapsed = t0.elapsed().as_secs_f32();
-                    let outcome = RunOutcome {
-                        at: Utc::now(),
-                        success: outcome.is_ok(),
-                        message: match &outcome {
-                            Ok(s) => s.clone(),
-                            Err(e) => e.clone(),
-                        },
-                        rows: outcome.as_ref().ok().and_then(parse_rows_from_message),
-                        duration_secs: elapsed,
-                    };
-                    if let Err(e) = store.record_run(&id, outcome).await {
-                        tracing::warn!("scheduler: record_run({id}): {e}");
-                    }
-                    in_flight.write().await.remove(&id);
-                });
-            }
-        }
-    }
-}
-
-/// Pluck a "rows=N" hint out of a success message — best-effort, no
-/// hard contract. Lets us surface row counts in the schedule status
-/// endpoint without forcing every ingest path to return structured
-/// stats here.
-fn parse_rows_from_message(msg: &String) -> Option<usize> {
-    msg.split_whitespace()
-        .find_map(|tok| tok.strip_prefix("rows=").and_then(|n| n.parse().ok()))
-}
-
-/// Fire one schedule's ingest. Returns Ok(success_message) or Err(reason).
-pub async fn run_schedule(
-    def: &ScheduleDef,
-    deps: &SchedulerIngestDeps,
-) -> Result<String, String> {
-    tracing::info!("scheduler: firing '{}' ({})", def.id, def.kind.label());
-    match &def.kind {
-        ScheduleKind::Mysql { dsn, table, dataset_name, batch_size, order_by } => {
-            let req = crate::my_stream::MyStreamRequest {
-                dsn: dsn.clone(),
-                table: table.clone(),
-                dataset_name: dataset_name.clone(),
-                batch_size: *batch_size,
-                order_by: order_by.clone(),
-                limit: None,
-            };
-            run_mysql(req, deps).await
-        }
-        ScheduleKind::Postgres { dsn, table, dataset_name, batch_size, order_by } => {
-            let req = crate::pg_stream::PgStreamRequest {
-                dsn: dsn.clone(),
-                table: table.clone(),
-                dataset_name: dataset_name.clone(),
-                batch_size: *batch_size,
-                order_by: order_by.clone(),
-                limit: None,
-            };
-            run_postgres(req, deps).await
-        }
-    }
-}
-
-async fn run_mysql(
-    req: crate::my_stream::MyStreamRequest,
-    deps: &SchedulerIngestDeps,
-) -> Result<String, String> {
-    let (parquet, result) = crate::my_stream::stream_table_to_parquet(&req).await?;
-    if result.rows == 0 {
-        return Ok(format!("rows=0 — table empty"));
-    }
-    persist_and_register(
-        &parquet, &result.table, req.dataset_name.as_deref(),
-        result.rows, "mysql", &req.dsn, deps,
-    ).await
-}
-
-async fn run_postgres(
-    req: crate::pg_stream::PgStreamRequest,
-    deps: &SchedulerIngestDeps,
-) -> Result<String, String> {
-    let (parquet, result) = crate::pg_stream::stream_table_to_parquet(&req).await?;
-    if result.rows == 0 {
-        return Ok(format!("rows=0 — table empty"));
-    }
-    persist_and_register(
-        &parquet, &result.table, req.dataset_name.as_deref(),
-        result.rows, "postgresql", &req.dsn, deps,
-    ).await
-}
-
-/// Shared write+register path. Mirrors the HTTP ingest handlers in
-/// service.rs so the scheduled path produces identical catalog output:
-/// PII detection, lineage with redacted DSN, mark-stale + agent trigger.
-async fn persist_and_register(
-    parquet: &bytes::Bytes,
-    table: &str,
-    dataset_name_override: Option<&str>,
-    rows: usize,
-    source_system: &str,
-    dsn: &str,
-    deps: &SchedulerIngestDeps,
-) -> Result<String, String> {
-    use shared::types::{ColumnMeta, Lineage, ObjectRef, SchemaFingerprint, Sensitivity};
-
-    let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(parquet)
-        .map_err(|e| format!("reparse parquet: {e}"))?;
-
-    let dataset_name = dataset_name_override.unwrap_or(table).to_string();
-    let storage_key = format!("datasets/{}.parquet", dataset_name);
-    let parquet_size = parquet.len() as u64;
-    let bucket = "primary".to_string();
-
-    let target_store = deps.buckets.get(&bucket)?;
-    ops::put(&target_store, &storage_key, parquet.clone()).await?;
-
-    let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
-    let now = Utc::now();
-    deps.registry.register(
-        dataset_name.clone(),
-        SchemaFingerprint(schema_fp.0),
-        vec![ObjectRef {
-            bucket: bucket.clone(),
-            key: storage_key.clone(),
-            size_bytes: parquet_size,
-            created_at: now,
-        }],
-    ).await?;
-
-    let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
-    let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
-    let columns: Vec<ColumnMeta> = schema.fields().iter().map(|f| {
-        let sens = shared::pii::detect_sensitivity(f.name());
-        ColumnMeta {
-            name: f.name().clone(),
-            data_type: f.data_type().to_string(),
-            sensitivity: sens.clone(),
-            description: String::new(),
-            is_pii: matches!(sens, Some(Sensitivity::Pii)),
-        }
-    }).collect();
-
-    let lineage = Lineage {
-        source_system: source_system.to_string(),
-        source_file: format!("dsn: {}", redact_dsn(dsn)),
-        ingest_job: format!("scheduled-{}-{}", source_system, now.timestamp_millis()),
-        ingest_timestamp: now,
-        parent_datasets: vec![],
-    };
-
-    let _ = deps.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
-        sensitivity,
-        columns: Some(columns),
-        lineage: Some(lineage),
-        row_count: Some(rows as u64),
-        ..Default::default()
-    }).await;
-
-    let _ = deps.registry.mark_embeddings_stale(&dataset_name).await;
-
-    if let Some(agent) = &deps.agent_handle {
-        let bound = deps.index_registry.list(Some(&dataset_name), None).await;
-        for meta in bound {
-            let event = vectord::agent::TriggerEvent::dataset_appended(
-                meta.index_name.clone(), &dataset_name,
-            );
-            let _ = agent.enqueue(event).await;
-        }
-    }
-
-    Ok(format!("rows={rows} dataset={dataset_name} bytes={parquet_size}"))
-}
-
-/// Same generic redactor used in service.rs — duplicated here to avoid
-/// pulling service::redact_dsn behind a `pub` visibility just for this.
-fn redact_dsn(dsn: &str) -> String {
-    let scheme_end = match dsn.find("://") {
-        Some(i) => i + 3,
-        None => return dsn.to_string(),
-    };
-    let at_idx = match dsn.rfind('@') {
-        Some(i) if i > scheme_end => i,
-        _ => return dsn.to_string(),
-    };
-    let userpass = &dsn[scheme_end..at_idx];
-    let colon_offset = match userpass.find(':') {
-        Some(i) => i,
-        None => return dsn.to_string(),
-    };
-    let colon_idx = scheme_end + colon_offset;
-    format!("{}:***{}", &dsn[..colon_idx], &dsn[at_idx..])
-}
-
-// =================== Tests ===================
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use chrono::Timelike;
-
-    fn mk(secs: u64) -> ScheduleDef {
-        ScheduleDef {
-            id: "test".into(),
-            kind: ScheduleKind::Mysql {
-                dsn: "mysql://x@host/db".into(),
-                table: "t".into(),
-                dataset_name: None,
-                batch_size: None,
-                order_by: None,
-            },
-            trigger: ScheduleTrigger::Interval { secs },
-            enabled: true,
-            created_at: Utc::now(),
-            created_by: "tester".into(),
-            next_run_at: Utc::now(),
-            last_run_at: None,
-            last_outcome: None,
-            run_count: 0,
-            failure_count: 0,
-        }
-    }
-
-    #[test]
-    fn is_due_respects_enabled() {
-        let mut s = mk(60);
-        s.next_run_at = Utc::now() - Duration::seconds(5);
-        assert!(s.is_due(Utc::now()));
-        s.enabled = false;
-        assert!(!s.is_due(Utc::now()));
-    }
-
-    #[test]
-    fn is_due_respects_future_time() {
-        let mut s = mk(60);
-        s.next_run_at = Utc::now() + Duration::seconds(60);
-        assert!(!s.is_due(Utc::now()));
-    }
-
-    #[test]
-    fn next_run_since_last_completion_advances() {
-        let s = mk(120);
-        let completed = Utc::now();
-        let next = compute_next_run_at(&s.trigger, completed, completed);
-        let delta = next - completed;
-        assert_eq!(delta.num_seconds(), 120);
-    }
-
-    #[test]
-    fn cron_every_minute_fires_within_sixty_seconds() {
-        let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() };
-        let now = Utc::now();
-        let next = compute_next_run_at(&trig, now, now);
-        let delta = next - now;
-        assert!(delta.num_seconds() > 0 && delta.num_seconds() <= 60,
-            "expected next fire within 60s, got {}s", delta.num_seconds());
-    }
-
-    #[test]
-    fn cron_daily_at_1415_utc_is_within_24h() {
-        // 14:15 UTC daily — whether we're before or after 14:15 today, the
-        // next fire is at most 24h out.
-        let trig = ScheduleTrigger::Cron { expr: "15 14 * * *".into() };
-        let now: DateTime<Utc> = "2026-04-20T10:00:00Z".parse().unwrap();
-        let next = compute_next_run_at(&trig, now, now);
-        assert!(next > now);
-        assert!((next - now).num_hours() <= 24);
-        // And the minute-of-hour is 15.
-        assert_eq!(next.minute(), 15);
-        assert_eq!(next.hour(), 14);
-    }
-
-    #[test]
-    fn cron_weekday_skips_weekend() {
-        // 09:00 UTC weekdays only. On a Saturday, next fire is Monday.
-        let trig = ScheduleTrigger::Cron { expr: "0 9 * * 1-5".into() };
-        // 2026-04-18 is a Saturday.
-        let sat: DateTime<Utc> = "2026-04-18T10:00:00Z".parse().unwrap();
-        let next = compute_next_run_at(&trig, sat, sat);
-        // Monday 2026-04-20 at 09:00 UTC.
-        assert_eq!(next, "2026-04-20T09:00:00Z".parse::<DateTime<Utc>>().unwrap());
-    }
-
-    #[test]
-    fn cron_six_field_seconds_granularity() {
-        // 6-field (seconds included): fire every minute at 30s past the minute.
-        let trig = ScheduleTrigger::Cron { expr: "30 * * * * *".into() };
-        let now = Utc::now();
-        let next = compute_next_run_at(&trig, now, now);
-        assert_eq!(next.second(), 30);
-    }
-
-    #[test]
-    fn validate_rejects_bad_cron() {
-        let bad = ScheduleTrigger::Cron { expr: "not a cron".into() };
-        let err = validate_trigger(&bad).unwrap_err();
-        assert!(err.contains("cron"), "error should mention cron, got: {err}");
-    }
-
-    #[test]
-    fn validate_rejects_wrong_field_count() {
-        // 4 fields — neither 5 nor 6.
-        let bad = ScheduleTrigger::Cron { expr: "* * * *".into() };
-        assert!(validate_trigger(&bad).is_err());
-    }
-
-    #[test]
-    fn validate_rejects_zero_interval() {
-        let bad = ScheduleTrigger::Interval { secs: 0 };
-        assert!(validate_trigger(&bad).is_err());
-    }
-
-    #[test]
-    fn validate_accepts_good_cron() {
-        let good = ScheduleTrigger::Cron { expr: "0 */6 * * *".into() };
-        assert!(validate_trigger(&good).is_ok());
-    }
-
-    #[test]
-    fn schedule_kind_label() {
-        let k = ScheduleKind::Mysql {
-            dsn: "x".into(), table: "customers".into(),
-            dataset_name: None, batch_size: None, order_by: None,
-        };
-        assert_eq!(k.label(), "mysql:customers");
-    }
-
-    #[test]
-    fn redact_dsn_handles_mysql_and_postgres() {
-        assert_eq!(redact_dsn("mysql://u:secret@h/db"), "mysql://u:***@h/db");
-        assert_eq!(redact_dsn("postgresql://u:secret@h:5432/db"), "postgresql://u:***@h:5432/db");
-        assert_eq!(redact_dsn("mysql://u@h/db"), "mysql://u@h/db"); // no password
-    }
-}

crates/ingestd/src/service.rs

@@ -1,9 +1,9 @@
 use axum::{
     Json, Router,
-    extract::{Multipart, Path, Query, State},
-    http::{HeaderMap, StatusCode},
+    extract::{Multipart, Query, State},
+    http::StatusCode,
     response::IntoResponse,
-    routing::{delete, get, patch, post},
+    routing::{get, post},
 };
 use bytes::Bytes;
 use object_store::ObjectStore;
@@ -11,74 +11,15 @@ use serde::Deserialize;
 use std::sync::Arc;
 
 use catalogd::registry::Registry;
-use crate::{db_ingest, my_stream, pg_stream, pipeline, schedule};
+use crate::{db_ingest, pipeline};
 use shared::arrow_helpers::record_batch_to_parquet;
 use shared::types::{ObjectRef, SchemaFingerprint};
 use storaged::ops;
-use storaged::registry::BucketRegistry;
 
 #[derive(Clone)]
 pub struct IngestState {
     pub store: Arc<dyn ObjectStore>,
     pub registry: Registry,
-    /// Federation layer 2: lookup target bucket from request headers.
-    pub buckets: Arc<BucketRegistry>,
-    /// Phase 16.5: when ingest marks a dataset's embeddings stale, push
-    /// a `DatasetAppended` trigger so the autotune agent can schedule
-    /// re-trials. Holds the agent handle (no-op when agent disabled).
-    pub agent_handle: vectord::agent::AgentHandle,
-    /// Used to look up which HNSW indexes are attached to the
-    /// just-ingested dataset. Each matching index gets one trigger.
-    pub index_registry: vectord::index_registry::IndexRegistry,
-    /// Scheduled-ingest registry. The scheduler task runs against this
-    /// store; HTTP CRUD endpoints write through it.
-    pub schedules: schedule::ScheduleStore,
-}
-
-/// Push `DatasetAppended` triggers for every HNSW index bound to this
-/// dataset. Called after a successful ingest. Logs failures but never
-/// fails the ingest — the agent is best-effort, not load-bearing.
-async fn notify_agent_on_append(state: &IngestState, dataset_name: &str) {
-    let indexes = state.index_registry.list(Some(dataset_name), None).await;
-    if indexes.is_empty() {
-        return;
-    }
-    for meta in indexes {
-        let event = vectord::agent::TriggerEvent::dataset_appended(
-            meta.index_name.clone(), dataset_name,
-        );
-        if let Err(e) = state.agent_handle.enqueue(event).await {
-            tracing::warn!("agent enqueue failed for '{}': {}", meta.index_name, e);
-        } else {
-            tracing::info!(
-                "agent: enqueued DatasetAppended({}) for index '{}'",
-                dataset_name, meta.index_name,
-            );
-        }
-    }
-}
-
-/// Resolve the target bucket from `X-Lakehouse-Bucket` header.
-/// Returns `(bucket_name, store_for_writes)`. Falls back to "primary"
-/// when the header is absent. Returns Err with the canonical bucket
-/// list when the header names an unknown bucket.
-fn resolve_bucket(
-    headers: &HeaderMap,
-    buckets: &BucketRegistry,
-) -> Result<(String, Arc<dyn ObjectStore>), (StatusCode, String)> {
-    let target = headers
-        .get("x-lakehouse-bucket")
-        .and_then(|h| h.to_str().ok())
-        .map(|s| s.to_string())
-        .unwrap_or_else(|| buckets.default_name().to_string());
-
-    match buckets.get(&target) {
-        Ok(store) => Ok((target, store)),
-        Err(_) => Err((
-            StatusCode::NOT_FOUND,
-            format!("unknown bucket '{}' — use GET /storage/buckets to list", target),
-        )),
-    }
 }
 
 pub fn router(state: IngestState) -> Router {
@@ -87,14 +28,6 @@ pub fn router(state: IngestState) -> Router {
         .route("/file", post(ingest_file))
         .route("/postgres/tables", post(list_pg_tables))
         .route("/postgres/import", post(import_pg_table))
-        .route("/db", post(ingest_db_stream))
-        .route("/mysql", post(ingest_mysql_stream))
-        // Phase E: scheduled ingest
-        .route("/schedules", get(list_schedules).post(create_schedule))
-        .route("/schedules/{id}", get(get_schedule)
-            .patch(patch_schedule)
-            .delete(delete_schedule))
-        .route("/schedules/{id}/run-now", post(run_schedule_now))
         .with_state(state)
 }
 
@@ -110,11 +43,8 @@ struct IngestQuery {
 async fn ingest_file(
     State(state): State<IngestState>,
     Query(query): Query<IngestQuery>,
-    headers: HeaderMap,
     mut multipart: Multipart,
 ) -> impl IntoResponse {
-    let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
-
     let field = match multipart.next_field().await {
         Ok(Some(f)) => f,
         Ok(None) => return Err((StatusCode::BAD_REQUEST, "no file uploaded".to_string())),
@@ -125,18 +55,12 @@ async fn ingest_file(
     let content = field.bytes().await
         .map_err(|e| (StatusCode::BAD_REQUEST, format!("read error: {e}")))?;
 
-    tracing::info!(
-        "ingest '{}' ({} bytes) -> bucket={}",
-        filename, content.len(), bucket,
-    );
+    tracing::info!("received file '{}' ({} bytes) for ingest", filename, content.len());
 
     let dataset_name = query.name.as_deref();
 
-    match pipeline::ingest_file_to_bucket(&filename, &content, dataset_name, &bucket, &store, &state.registry).await {
+    match pipeline::ingest_file(&filename, &content, dataset_name, &state.store, &state.registry).await {
         Ok(result) => {
-            if !result.deduplicated {
-                notify_agent_on_append(&state, &result.dataset_name).await;
-            }
             if result.deduplicated {
                 Ok((StatusCode::OK, Json(result)))
             } else {
@@ -264,369 +188,3 @@ async fn import_pg_table(
         "size_bytes": parquet_size,
     }))))
 }
-
-/// Streaming DSN-based ingest. Paginates via ORDER BY + LIMIT/OFFSET so
-/// large tables don't blow up memory. This is the path the task spec calls
-/// `POST /ingest/db` with `{dsn, table, dataset_name}`.
-async fn ingest_db_stream(
-    State(state): State<IngestState>,
-    headers: HeaderMap,
-    Json(req): Json<pg_stream::PgStreamRequest>,
-) -> Result<(StatusCode, Json<serde_json::Value>), (StatusCode, String)> {
-    let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
-    tracing::info!(
-        "pg stream ingest: table='{}' dataset='{:?}' bucket='{}' batch_size={:?}",
-        req.table, req.dataset_name, bucket, req.batch_size,
-    );
-
-    // Stream from Postgres into Parquet bytes.
-    let (parquet, stream_result) = pg_stream::stream_table_to_parquet(&req)
-        .await
-        .map_err(|e| (StatusCode::BAD_GATEWAY, e))?;
-
-    if stream_result.rows == 0 {
-        return Ok((StatusCode::OK, Json(serde_json::json!({
-            "table": req.table,
-            "rows": 0,
-            "message": "table is empty",
-        }))));
-    }
-
-    // Recover schema from the parquet footer — keeps a single source of truth
-    // for the schema fingerprint + column metadata.
-    let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(&parquet)
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("reparse parquet: {e}")))?;
-
-    let dataset_name = req.dataset_name.clone().unwrap_or_else(|| req.table.clone());
-    let storage_key = format!("datasets/{}.parquet", dataset_name);
-    let size_bytes = parquet.len() as u64;
-
-    ops::put(&store, &storage_key, parquet)
-        .await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
-
-    let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
-    let now = chrono::Utc::now();
-    state.registry.register(
-        dataset_name.clone(),
-        SchemaFingerprint(schema_fp.0),
-        vec![ObjectRef {
-            bucket: bucket.clone(),
-            key: storage_key.clone(),
-            size_bytes,
-            created_at: now,
-        }],
-    ).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
-
-    // Rich metadata: PII auto-detect, lineage, row_count, columns.
-    let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
-    let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
-    let columns: Vec<shared::types::ColumnMeta> = schema.fields().iter().map(|f| {
-        let sens = shared::pii::detect_sensitivity(f.name());
-        shared::types::ColumnMeta {
-            name: f.name().clone(),
-            data_type: f.data_type().to_string(),
-            sensitivity: sens.clone(),
-            description: String::new(),
-            is_pii: matches!(sens, Some(shared::types::Sensitivity::Pii)),
-        }
-    }).collect();
-
-    let lineage = shared::types::Lineage {
-        source_system: "postgresql".to_string(),
-        source_file: format!("dsn: {}", redact_dsn(&req.dsn)),
-        ingest_job: format!("pg-stream-{}", now.timestamp_millis()),
-        ingest_timestamp: now,
-        parent_datasets: vec![],
-    };
-
-    let _ = state.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
-        sensitivity,
-        columns: Some(columns),
-        lineage: Some(lineage),
-        row_count: Some(stream_result.rows as u64),
-        ..Default::default()
-    }).await;
-
-    // Phase C: mark embeddings stale if the dataset already had a vector
-    // index. No-op for newly-created datasets.
-    let _ = state.registry.mark_embeddings_stale(&dataset_name).await;
-
-    // Phase 16.5: notify autotune agent. No-op if no indexes attached.
-    notify_agent_on_append(&state, &dataset_name).await;
-
-    Ok((StatusCode::CREATED, Json(serde_json::json!({
-        "dataset_name": dataset_name,
-        "table": stream_result.table,
-        "rows": stream_result.rows,
-        "batches": stream_result.batches,
-        "columns": stream_result.columns,
-        "schema": stream_result.schema,
-        "storage_key": storage_key,
-        "size_bytes": size_bytes,
-        "duration_secs": stream_result.duration_secs,
-    }))))
-}
-
-/// Streaming MySQL ingest. Mirrors `ingest_db_stream` — same pagination +
-/// Parquet-footer schema recovery + PII/lineage metadata. Different driver,
-/// otherwise identical.
-async fn ingest_mysql_stream(
-    State(state): State<IngestState>,
-    headers: HeaderMap,
-    Json(req): Json<my_stream::MyStreamRequest>,
-) -> Result<(StatusCode, Json<serde_json::Value>), (StatusCode, String)> {
-    let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
-    tracing::info!(
-        "mysql stream ingest: table='{}' dataset='{:?}' bucket='{}' batch_size={:?}",
-        req.table, req.dataset_name, bucket, req.batch_size,
-    );
-
-    let (parquet, stream_result) = my_stream::stream_table_to_parquet(&req)
-        .await
-        .map_err(|e| (StatusCode::BAD_GATEWAY, e))?;
-
-    if stream_result.rows == 0 {
-        return Ok((StatusCode::OK, Json(serde_json::json!({
-            "table": req.table,
-            "rows": 0,
-            "message": "table is empty",
-        }))));
-    }
-
-    let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(&parquet)
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("reparse parquet: {e}")))?;
-
-    let dataset_name = req.dataset_name.clone().unwrap_or_else(|| req.table.clone());
-    let storage_key = format!("datasets/{}.parquet", dataset_name);
-    let size_bytes = parquet.len() as u64;
-
-    ops::put(&store, &storage_key, parquet)
-        .await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
-
-    let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
-    let now = chrono::Utc::now();
-    state.registry.register(
-        dataset_name.clone(),
-        SchemaFingerprint(schema_fp.0),
-        vec![ObjectRef {
-            bucket: bucket.clone(),
-            key: storage_key.clone(),
-            size_bytes,
-            created_at: now,
-        }],
-    ).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
-
-    let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
-    let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
-    let columns: Vec<shared::types::ColumnMeta> = schema.fields().iter().map(|f| {
-        let sens = shared::pii::detect_sensitivity(f.name());
-        shared::types::ColumnMeta {
-            name: f.name().clone(),
-            data_type: f.data_type().to_string(),
-            sensitivity: sens.clone(),
-            description: String::new(),
-            is_pii: matches!(sens, Some(shared::types::Sensitivity::Pii)),
-        }
-    }).collect();
-
-    let lineage = shared::types::Lineage {
-        source_system: "mysql".to_string(),
-        source_file: format!("dsn: {}", redact_dsn(&req.dsn)),
-        ingest_job: format!("mysql-stream-{}", now.timestamp_millis()),
-        ingest_timestamp: now,
-        parent_datasets: vec![],
-    };
-
-    let _ = state.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
-        sensitivity,
-        columns: Some(columns),
-        lineage: Some(lineage),
-        row_count: Some(stream_result.rows as u64),
-        ..Default::default()
-    }).await;
-
-    // Phase C: mark stale + Phase 16.5: notify agent.
-    let _ = state.registry.mark_embeddings_stale(&dataset_name).await;
-    notify_agent_on_append(&state, &dataset_name).await;
-
-    Ok((StatusCode::CREATED, Json(serde_json::json!({
-        "dataset_name": dataset_name,
-        "table": stream_result.table,
-        "rows": stream_result.rows,
-        "batches": stream_result.batches,
-        "columns": stream_result.columns,
-        "schema": stream_result.schema,
-        "storage_key": storage_key,
-        "size_bytes": size_bytes,
-        "duration_secs": stream_result.duration_secs,
-    }))))
-}
-
-/// Redact the password in any `scheme://user:pass@host/...` DSN. Works
-/// for postgres, mysql, and any other scheme using the same shape.
-fn redact_dsn(dsn: &str) -> String {
-    let scheme_end = match dsn.find("://") {
-        Some(i) => i + 3,
-        None => return dsn.to_string(),
-    };
-    let at_idx = match dsn.rfind('@') {
-        Some(i) if i > scheme_end => i,
-        _ => return dsn.to_string(),
-    };
-    // Find the password separator (first `:` after scheme, before `@`).
-    let userpass = &dsn[scheme_end..at_idx];
-    let colon_offset = match userpass.find(':') {
-        Some(i) => i,
-        None => return dsn.to_string(), // no password in DSN
-    };
-    let colon_idx = scheme_end + colon_offset;
-    format!("{}:***{}", &dsn[..colon_idx], &dsn[at_idx..])
-}
-
-// --- Phase E: Scheduled ingest CRUD endpoints ---
-
-#[derive(Deserialize)]
-struct CreateScheduleRequest {
-    /// Stable id used in URLs and on disk. Required so callers can
-    /// PATCH/DELETE deterministically. Sanitized on storage but the
-    /// caller's id is what gets persisted as the canonical key.
-    id: String,
-    kind: schedule::ScheduleKind,
-    trigger: schedule::ScheduleTrigger,
-    #[serde(default = "default_enabled")]
-    enabled: bool,
-    #[serde(default)]
-    created_by: String,
-}
-
-fn default_enabled() -> bool { true }
-
-async fn create_schedule(
-    State(state): State<IngestState>,
-    Json(req): Json<CreateScheduleRequest>,
-) -> impl IntoResponse {
-    if state.schedules.get(&req.id).await.is_some() {
-        return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id)));
-    }
-    if let Err(e) = schedule::validate_trigger(&req.trigger) {
-        return Err((StatusCode::BAD_REQUEST, e));
-    }
-    let now = chrono::Utc::now();
-    let def = schedule::ScheduleDef {
-        id: req.id,
-        kind: req.kind,
-        trigger: req.trigger,
-        enabled: req.enabled,
-        created_at: now,
-        created_by: req.created_by,
-        // Fire on the next tick — operators usually want to verify
-        // their schedule works without waiting for the full interval.
-        next_run_at: now,
-        last_run_at: None,
-        last_outcome: None,
-        run_count: 0,
-        failure_count: 0,
-    };
-    match state.schedules.put(def).await {
-        Ok(d) => Ok((StatusCode::CREATED, Json(d))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-async fn list_schedules(State(state): State<IngestState>) -> impl IntoResponse {
-    Json(state.schedules.list().await)
-}
-
-async fn get_schedule(
-    State(state): State<IngestState>,
-    Path(id): Path<String>,
-) -> impl IntoResponse {
-    match state.schedules.get(&id).await {
-        Some(d) => Ok(Json(d)),
-        None => Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found"))),
-    }
-}
-
-#[derive(Deserialize)]
-struct PatchScheduleRequest {
-    /// Toggle on/off without changing anything else.
-    #[serde(default)]
-    enabled: Option<bool>,
-    /// Replace the trigger spec — useful for bumping intervals.
-    #[serde(default)]
-    trigger: Option<schedule::ScheduleTrigger>,
-}
-
-async fn patch_schedule(
-    State(state): State<IngestState>,
-    Path(id): Path<String>,
-    Json(req): Json<PatchScheduleRequest>,
-) -> impl IntoResponse {
-    let Some(mut def) = state.schedules.get(&id).await else {
-        return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
-    };
-    if let Some(t) = &req.trigger {
-        if let Err(e) = schedule::validate_trigger(t) {
-            return Err((StatusCode::BAD_REQUEST, e));
-        }
-    }
-    if let Some(e) = req.enabled { def.enabled = e; }
-    if let Some(t) = req.trigger { def.trigger = t; }
-    match state.schedules.put(def).await {
-        Ok(d) => Ok(Json(d)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-async fn delete_schedule(
-    State(state): State<IngestState>,
-    Path(id): Path<String>,
-) -> impl IntoResponse {
-    match state.schedules.delete(&id).await {
-        Ok(true) => Ok(Json(serde_json::json!({ "deleted": id }))),
-        Ok(false) => Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found"))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
-    }
-}
-
-/// Force a schedule to fire right now, regardless of its next_run_at.
-/// Records the outcome and reschedules. Useful for "test my new schedule
-/// works without waiting an hour."
-async fn run_schedule_now(
-    State(state): State<IngestState>,
-    Path(id): Path<String>,
-) -> impl IntoResponse {
-    let Some(def) = state.schedules.get(&id).await else {
-        return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
-    };
-    let deps = schedule::SchedulerIngestDeps {
-        store: state.store.clone(),
-        registry: state.registry.clone(),
-        buckets: state.buckets.clone(),
-        agent_handle: Some(state.agent_handle.clone()),
-        index_registry: state.index_registry.clone(),
-    };
-
-    let t0 = std::time::Instant::now();
-    let result = schedule::run_schedule(&def, &deps).await;
-    let elapsed = t0.elapsed().as_secs_f32();
-    let outcome = schedule::RunOutcome {
-        at: chrono::Utc::now(),
-        success: result.is_ok(),
-        message: match &result {
-            Ok(s) => s.clone(),
-            Err(e) => e.clone(),
-        },
-        rows: result.as_ref().ok().and_then(|m: &String| {
-            m.split_whitespace()
-                .find_map(|tok| tok.strip_prefix("rows=").and_then(|n| n.parse().ok()))
-        }),
-        duration_secs: elapsed,
-    };
-    if let Err(e) = state.schedules.record_run(&id, outcome.clone()).await {
-        tracing::warn!("schedule {id}: record_run failed: {e}");
-    }
-    Ok(Json(outcome))
-}

crates/lance-bench/Cargo.toml

@@ -1,42 +0,0 @@
-[package]
-name = "lance-bench"
-version = "0.1.0"
-edition = "2024"
-
-# Standalone pilot for Phase B (see docs/EXECUTION_PLAN.md).
-# Deliberately NOT sharing workspace deps — Lance 4.x pulls in its own
-# DataFusion and Arrow versions incompatible with the rest of the stack.
-# Isolating the pilot means we don't force a workspace-wide upgrade until
-# we've decided Lance is worth it.
-
-[dependencies]
-# Only the features we actually need — the default brings in AWS/Azure/GCP/HF etc
-# which is ~200 extra crates we don't care about for a local pilot.
-lance = { version = "4.0", default-features = false }
-# Lance exposes DatasetIndexExt, IndexType, and IvfBuildParams through
-# its sub-crates which must be imported directly — lance itself doesn't
-# re-export them at a convenient path.
-lance-index = { version = "4.0", default-features = false }
-lance-linalg = { version = "4.0", default-features = false }
-
-# Arrow re-exported by Lance; pin to a range Lance picks so types match.
-arrow = "57"
-arrow-array = "57"
-arrow-schema = "57"
-
-# Also need to read the EXISTING Parquet vector files so we can compare.
-# These live in data/vectors/*.parquet. Lance's internal Parquet reading
-# might differ from ours; using our format's Arrow/Parquet versions for
-# the read side keeps the inputs identical.
-parquet = "57"
-
-tokio = { version = "1", features = ["full"] }
-futures = "0.3"
-serde = { version = "1", features = ["derive"] }
-serde_json = "1"
-anyhow = "1"
-bytes = "1"
-
-[[bin]]
-name = "lance-bench"
-path = "src/main.rs"

crates/lance-bench/src/main.rs

@@ -1,633 +0,0 @@
-//! Phase B: Lance pilot benchmark.
-//!
-//! Standalone binary that compares Lance vector storage against our
-//! Parquet-with-binary-blob + in-RAM HNSW approach. See
-//! docs/EXECUTION_PLAN.md for the decision rules this fuels.
-//!
-//! Inputs:
-//!   data/vectors/resumes_100k_v2.parquet — existing 100K × 768d embeddings
-//!
-//! Output:
-//!   A JSON report printed to stdout with measurements for:
-//!     - Cold load time (parquet → arrow) vs Lance open + scan
-//!     - Disk size
-//!     - Vector search latency (p50 / p95 / p99)
-//!     - Single-row random access
-//!     - Append cost (adding 10K rows)
-//!
-//! Usage:
-//!   cargo run --bin lance-bench -- \
-//!     --parquet data/vectors/resumes_100k_v2.parquet \
-//!     --lance-out /tmp/lance_resumes_100k_v2 \
-//!     --json-out /tmp/lance_bench.json
-
-use anyhow::{Context, Result};
-use arrow_array::{Array, ArrayRef, BinaryArray, FixedSizeListArray, Float32Array, RecordBatch, RecordBatchIterator};
-use arrow_schema::{DataType, Field, Schema};
-use serde::Serialize;
-use std::sync::Arc;
-use std::time::Instant;
-
-#[derive(Debug, Serialize)]
-struct BenchReport {
-    vectors: usize,
-    dimensions: usize,
-    parquet_path: String,
-    lance_path: String,
-
-    // Parquet baseline
-    parquet_disk_bytes: u64,
-    parquet_cold_load_secs: f32,
-
-    // Lance numbers
-    lance_write_secs: f32,
-    lance_disk_bytes: u64,
-    lance_cold_open_secs: f32,
-
-    // Index + search
-    lance_index_build_secs: Option<f32>,
-    lance_index_disk_bytes: Option<u64>,
-    lance_search_p50_us: Option<f32>,
-    lance_search_p95_us: Option<f32>,
-    lance_search_p99_us: Option<f32>,
-
-    // Architectural features Parquet+sidecar can't cheaply do
-    lance_random_row_access_us: Option<f32>,   // fetch one row by row_id
-    parquet_random_row_access_us: Option<f32>, // for comparison — full scan cost
-    lance_append_10k_secs: Option<f32>,        // add 10K new rows
-    lance_append_disk_bytes_added: Option<u64>,
-
-    // Head-to-head reference (from our own measurements)
-    reference_hnsw_p50_us: f32,
-    reference_hnsw_p95_us: f32,
-    reference_brute_force_us: f32,
-    reference_hnsw_build_secs: f32,
-}
-
-#[tokio::main]
-async fn main() -> Result<()> {
-    // Simple positional args: parquet_in, lance_out.
-    let args: Vec<String> = std::env::args().collect();
-    let parquet_path = args
-        .get(1)
-        .cloned()
-        .unwrap_or_else(|| "data/vectors/resumes_100k_v2.parquet".to_string());
-    let lance_path = args
-        .get(2)
-        .cloned()
-        .unwrap_or_else(|| "/tmp/lance_bench_dataset".to_string());
-
-    eprintln!("=== Phase B Lance pilot ===");
-    eprintln!("input  parquet: {}", parquet_path);
-    eprintln!("output lance:   {}", lance_path);
-
-    // --- 1. Cold-load the existing Parquet vector index into memory
-    eprintln!("\n[1/4] reading Parquet baseline...");
-    let t0 = Instant::now();
-    let (schema, batches, total_rows) = read_parquet_vectors(&parquet_path)
-        .context("read parquet")?;
-    let parquet_cold_load_secs = t0.elapsed().as_secs_f32();
-    let parquet_disk_bytes = std::fs::metadata(&parquet_path)?.len();
-
-    let dims = detect_vector_dims(&batches)?;
-    eprintln!(
-        "  loaded {} rows, {} columns, vectors={}d, disk={:.1} MB, cold load={:.2}s",
-        total_rows,
-        schema.fields().len(),
-        dims,
-        parquet_disk_bytes as f64 / 1_000_000.0,
-        parquet_cold_load_secs,
-    );
-
-    // --- 2. Convert from binary-blob-of-f32 to Lance's FixedSizeList<Float32>
-    eprintln!("\n[2/4] converting binary-blob vectors to Arrow FixedSizeList...");
-    let t0 = Instant::now();
-    let (lance_schema, lance_batches) = convert_to_fixed_size_list(&schema, batches, dims)?;
-    eprintln!("  conversion took {:.2}s", t0.elapsed().as_secs_f32());
-
-    // --- 3. Write as Lance dataset
-    eprintln!("\n[3/4] writing Lance dataset...");
-    let t0 = Instant::now();
-    // Clean up any prior run
-    let _ = std::fs::remove_dir_all(&lance_path);
-    write_lance_dataset(&lance_path, lance_schema.clone(), lance_batches).await?;
-    let lance_write_secs = t0.elapsed().as_secs_f32();
-    let lance_disk_bytes = dir_size_bytes(&lance_path);
-    eprintln!(
-        "  write took {:.2}s, disk={:.1} MB",
-        lance_write_secs,
-        lance_disk_bytes as f64 / 1_000_000.0,
-    );
-
-    // --- 4. Cold open + scan the Lance dataset
-    eprintln!("\n[4/6] cold-opening Lance dataset...");
-    let t0 = Instant::now();
-    let scanned_rows = cold_open_and_scan_lance(&lance_path).await?;
-    let lance_cold_open_secs = t0.elapsed().as_secs_f32();
-    eprintln!(
-        "  open + full scan: {} rows in {:.2}s",
-        scanned_rows, lance_cold_open_secs,
-    );
-
-    // --- 5. Build a vector index on the Lance dataset
-    eprintln!("\n[5/6] building Lance vector index (IVF_PQ)...");
-    let t0 = Instant::now();
-    let index_built = build_lance_vector_index(&lance_path, dims).await;
-    let (lance_index_build_secs, lance_index_disk_bytes) = match index_built {
-        Ok(()) => {
-            let secs = t0.elapsed().as_secs_f32();
-            let disk = dir_size_bytes(&lance_path) - lance_disk_bytes;
-            eprintln!("  built in {:.2}s, index adds {:.1} MB on disk", secs, disk as f64 / 1e6);
-            (Some(secs), Some(disk))
-        }
-        Err(e) => {
-            eprintln!("  index build failed: {e:#}");
-            (None, None)
-        }
-    };
-
-    // --- 6. Run search queries, measure latency
-    eprintln!("\n[6/6] running vector search benchmarks...");
-    let search_stats = if lance_index_build_secs.is_some() {
-        run_search_benchmarks(&lance_path, dims).await.ok()
-    } else {
-        None
-    };
-    let (lance_search_p50, lance_search_p95, lance_search_p99) = match search_stats {
-        Some((p50, p95, p99)) => {
-            eprintln!("  p50={:.0}us p95={:.0}us p99={:.0}us", p50, p95, p99);
-            (Some(p50), Some(p95), Some(p99))
-        }
-        None => (None, None, None),
-    };
-
-    // --- Random access comparison
-    eprintln!("\n[7/8] random row access — Lance vs full-scan Parquet...");
-    let lance_random = measure_random_access_lance(&lance_path).await.ok();
-    let parquet_random = measure_random_access_parquet(&parquet_path).ok();
-    if let Some(us) = lance_random {
-        eprintln!("  Lance random-fetch avg: {:.0}us", us);
-    }
-    if let Some(us) = parquet_random {
-        eprintln!("  Parquet full-scan-to-row avg: {:.0}us", us);
-    }
-
-    // --- Append cost
-    eprintln!("\n[8/8] append 10K new rows to existing dataset...");
-    let t0 = Instant::now();
-    let pre_append_bytes = dir_size_bytes(&lance_path);
-    let append_result = append_10k_rows(&lance_path, dims).await;
-    let (lance_append_secs, lance_append_bytes) = match append_result {
-        Ok(()) => {
-            let secs = t0.elapsed().as_secs_f32();
-            let bytes = dir_size_bytes(&lance_path).saturating_sub(pre_append_bytes);
-            eprintln!("  append took {:.2}s, added {:.1} MB", secs, bytes as f64 / 1e6);
-            (Some(secs), Some(bytes))
-        }
-        Err(e) => {
-            eprintln!("  append failed: {e:#}");
-            (None, None)
-        }
-    };
-
-    // --- Report
-    let report = BenchReport {
-        vectors: total_rows,
-        dimensions: dims,
-        parquet_path: parquet_path.clone(),
-        lance_path: lance_path.clone(),
-        parquet_disk_bytes,
-        parquet_cold_load_secs,
-        lance_write_secs,
-        lance_disk_bytes,
-        lance_cold_open_secs,
-        lance_index_build_secs,
-        lance_index_disk_bytes,
-        lance_search_p50_us: lance_search_p50,
-        lance_search_p95_us: lance_search_p95,
-        lance_search_p99_us: lance_search_p99,
-        lance_random_row_access_us: lance_random,
-        parquet_random_row_access_us: parquet_random,
-        lance_append_10k_secs: lance_append_secs,
-        lance_append_disk_bytes_added: lance_append_bytes,
-        // From our Phase 15 trial on the SAME index (ec=80 es=30, recall=1.00):
-        reference_hnsw_p50_us: 873.0,
-        reference_hnsw_p95_us: 1413.0,
-        reference_brute_force_us: 43983.0,
-        reference_hnsw_build_secs: 230.0,
-    };
-
-    let json = serde_json::to_string_pretty(&report)?;
-    println!("{}", json);
-
-    eprintln!("\n=== Summary ===");
-    eprintln!("  Parquet cold load:  {:.2}s", report.parquet_cold_load_secs);
-    eprintln!("  Lance cold open:    {:.2}s  ({})",
-        report.lance_cold_open_secs,
-        format_ratio(report.parquet_cold_load_secs, report.lance_cold_open_secs));
-    eprintln!("  Parquet disk:       {:.1} MB", report.parquet_disk_bytes as f64 / 1e6);
-    eprintln!("  Lance disk:         {:.1} MB  ({})",
-        report.lance_disk_bytes as f64 / 1e6,
-        format_ratio(report.parquet_disk_bytes as f32, report.lance_disk_bytes as f32));
-    if let (Some(p50), Some(p95)) = (report.lance_search_p50_us, report.lance_search_p95_us) {
-        eprintln!("  Lance search p50:   {:.0}us  vs our HNSW {:.0}us  ({})",
-            p50, report.reference_hnsw_p50_us,
-            format_ratio(report.reference_hnsw_p50_us, p50));
-        eprintln!("  Lance search p95:   {:.0}us  vs our HNSW {:.0}us  ({})",
-            p95, report.reference_hnsw_p95_us,
-            format_ratio(report.reference_hnsw_p95_us, p95));
-        eprintln!("  Speedup vs brute force: {:.1}× (Lance) vs {:.1}× (HNSW)",
-            report.reference_brute_force_us / p50,
-            report.reference_brute_force_us / report.reference_hnsw_p50_us);
-    }
-    if let Some(build) = report.lance_index_build_secs {
-        eprintln!("  Index build:        {:.1}s (Lance IVF_PQ) vs {:.0}s (our HNSW ec=80)  ({}× faster)",
-            build, report.reference_hnsw_build_secs, report.reference_hnsw_build_secs / build);
-    }
-    if let (Some(lance_us), Some(parquet_us)) = (report.lance_random_row_access_us, report.parquet_random_row_access_us) {
-        eprintln!("  Random row access:  {:.0}us (Lance) vs {:.0}us (Parquet scan)  ({})",
-            lance_us, parquet_us, format_ratio(parquet_us, lance_us));
-    }
-    if let Some(append_secs) = report.lance_append_10k_secs {
-        eprintln!("  Append 10K rows:    {:.2}s (Lance native)  [Parquet would require full rewrite]",
-            append_secs);
-    }
-
-    Ok(())
-}
-
-fn format_ratio(baseline: f32, candidate: f32) -> String {
-    if candidate == 0.0 { return "inf".into(); }
-    let ratio = baseline / candidate;
-    if ratio >= 1.0 {
-        format!("{:.2}× faster/smaller", ratio)
-    } else {
-        format!("{:.2}× slower/larger", 1.0 / ratio)
-    }
-}
-
-fn dir_size_bytes(path: &str) -> u64 {
-    fn recurse(p: &std::path::Path) -> u64 {
-        let Ok(meta) = std::fs::metadata(p) else { return 0; };
-        if meta.is_file() { return meta.len(); }
-        let Ok(entries) = std::fs::read_dir(p) else { return 0; };
-        entries
-            .filter_map(|e| e.ok())
-            .map(|e| recurse(&e.path()))
-            .sum()
-    }
-    recurse(std::path::Path::new(path))
-}
-
-/// Read the existing vector Parquet (binary-blob format: source, doc_id,
-/// chunk_idx, chunk_text, vector as Binary bytes).
-fn read_parquet_vectors(path: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>, usize)> {
-    use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
-    use std::fs::File;
-
-    let file = File::open(path).with_context(|| format!("open {path}"))?;
-    let builder = ParquetRecordBatchReaderBuilder::try_new(file)?;
-    let schema = builder.schema().clone();
-    let reader = builder.build()?;
-    let batches: Vec<RecordBatch> = reader.collect::<Result<Vec<_>, _>>()?;
-    let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
-    Ok((schema, batches, rows))
-}
-
-fn detect_vector_dims(batches: &[RecordBatch]) -> Result<usize> {
-    for batch in batches {
-        let vector_col_idx = batch
-            .schema()
-            .index_of("vector")
-            .context("no 'vector' column in parquet")?;
-        let col = batch.column(vector_col_idx);
-        if let Some(binary) = col.as_any().downcast_ref::<BinaryArray>() {
-            for i in 0..binary.len() {
-                if !binary.is_null(i) {
-                    let bytes = binary.value(i);
-                    return Ok(bytes.len() / 4); // f32 = 4 bytes
-                }
-            }
-        }
-    }
-    anyhow::bail!("could not determine vector dimensions")
-}
-
-/// Convert our binary-blob vector representation into Arrow's native
-/// FixedSizeList<Float32> — that's what Lance expects for vector columns.
-fn convert_to_fixed_size_list(
-    schema: &Arc<Schema>,
-    batches: Vec<RecordBatch>,
-    dims: usize,
-) -> Result<(Arc<Schema>, Vec<RecordBatch>)> {
-    // New schema keeps everything identical but replaces the vector column
-    // with a FixedSizeList<Float32, dims>.
-    let new_fields: Vec<Arc<Field>> = schema
-        .fields()
-        .iter()
-        .map(|f| {
-            if f.name() == "vector" {
-                Arc::new(Field::new(
-                    "vector",
-                    DataType::FixedSizeList(
-                        Arc::new(Field::new("item", DataType::Float32, true)),
-                        dims as i32,
-                    ),
-                    false,
-                ))
-            } else {
-                f.clone()
-            }
-        })
-        .collect();
-    let new_schema = Arc::new(Schema::new(new_fields));
-
-    let mut new_batches = Vec::with_capacity(batches.len());
-    for batch in batches {
-        let vector_idx = batch.schema().index_of("vector")?;
-        let mut new_arrays: Vec<ArrayRef> = Vec::with_capacity(batch.num_columns());
-        for (i, col) in batch.columns().iter().enumerate() {
-            if i == vector_idx {
-                let binary = col
-                    .as_any()
-                    .downcast_ref::<BinaryArray>()
-                    .context("vector column must be Binary")?;
-                let fsl = binary_to_fixed_size_list(binary, dims)?;
-                new_arrays.push(Arc::new(fsl));
-            } else {
-                new_arrays.push(col.clone());
-            }
-        }
-        new_batches.push(RecordBatch::try_new(new_schema.clone(), new_arrays)?);
-    }
-
-    Ok((new_schema, new_batches))
-}
-
-fn binary_to_fixed_size_list(binary: &BinaryArray, dims: usize) -> Result<FixedSizeListArray> {
-    let n = binary.len();
-    let mut all_floats: Vec<f32> = Vec::with_capacity(n * dims);
-    for i in 0..n {
-        if binary.is_null(i) {
-            all_floats.extend(std::iter::repeat(0.0).take(dims));
-            continue;
-        }
-        let bytes = binary.value(i);
-        if bytes.len() != dims * 4 {
-            anyhow::bail!(
-                "row {} has {} bytes, expected {} ({} × f32)",
-                i, bytes.len(), dims * 4, dims,
-            );
-        }
-        for chunk in bytes.chunks_exact(4) {
-            all_floats.push(f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]));
-        }
-    }
-    let values = Float32Array::from(all_floats);
-    let field = Arc::new(Field::new("item", DataType::Float32, true));
-    FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None)
-        .context("build FixedSizeListArray")
-}
-
-/// Write batches into a Lance dataset at the given path.
-async fn write_lance_dataset(
-    path: &str,
-    schema: Arc<Schema>,
-    batches: Vec<RecordBatch>,
-) -> Result<()> {
-    use lance::dataset::{Dataset, WriteParams};
-
-    let reader = RecordBatchIterator::new(batches.into_iter().map(Ok), schema);
-    Dataset::write(reader, path, Some(WriteParams::default()))
-        .await
-        .context("Dataset::write")?;
-    Ok(())
-}
-
-/// Open a Lance dataset cold (from disk) and scan it fully — measuring the
-/// equivalent of our "load embeddings from Parquet" cost.
-async fn cold_open_and_scan_lance(path: &str) -> Result<usize> {
-    use futures::StreamExt;
-    use lance::dataset::Dataset;
-
-    let dataset = Dataset::open(path).await.context("Dataset::open")?;
-    let scanner = dataset.scan();
-    let mut stream = scanner.try_into_stream().await?;
-    let mut total = 0usize;
-    while let Some(batch) = stream.next().await {
-        let batch = batch?;
-        total += batch.num_rows();
-    }
-    Ok(total)
-}
-
-/// Build an IVF_PQ vector index on the `vector` column. IVF_PQ (Inverted File
-/// with Product Quantization) is Lance's native ANN index — comparable to
-/// HNSW in intent, but on-disk and compatible with Lance's random-access
-/// model.
-async fn build_lance_vector_index(path: &str, _dims: usize) -> Result<()> {
-    use lance::dataset::Dataset;
-    use lance::index::vector::VectorIndexParams;
-    use lance_index::{DatasetIndexExt, IndexType};
-    use lance_linalg::distance::MetricType;
-
-    let mut dataset = Dataset::open(path).await?;
-
-    // IVF_PQ with ~sqrt(N) partitions is a reasonable default for 100K.
-    // num_sub_vectors must divide dims evenly: 768/48 = 16 dims per subvector.
-    // num_bits = 8 gives 256 codes per subvector (good recall/size trade).
-    // max_iterations = 50 is plenty for this scale.
-    let params = VectorIndexParams::ivf_pq(
-        316,               // num_partitions (~sqrt(100000))
-        8,                 // num_bits
-        48,                // num_sub_vectors
-        MetricType::Cosine,
-        50,                // max_iterations
-    );
-
-    dataset
-        .create_index(
-            &["vector"],
-            IndexType::Vector,
-            Some("vec_idx".into()),
-            &params,
-            true,
-        )
-        .await
-        .context("create_index")?;
-
-    Ok(())
-}
-
-/// Run N vector searches against the Lance dataset and return (p50, p95, p99) latencies in us.
-/// Uses a handful of random rows as queries — same pattern as our harness::synthetic_from_chunks.
-async fn run_search_benchmarks(path: &str, _dims: usize) -> Result<(f32, f32, f32)> {
-    use futures::StreamExt;
-    use lance::dataset::Dataset;
-
-    let dataset = Dataset::open(path).await?;
-
-    // Pick 20 representative query vectors from the data itself.
-    // (Synthetic — same pattern as our existing harness.)
-    let query_vectors = sample_query_vectors(&dataset, 20).await?;
-
-    let mut latencies_us: Vec<f32> = Vec::with_capacity(query_vectors.len());
-    for (i, qv) in query_vectors.iter().enumerate() {
-        let qarr = Arc::new(Float32Array::from(qv.clone())) as ArrayRef;
-
-        let t0 = Instant::now();
-        let mut scanner = dataset.scan();
-        scanner
-            .nearest("vector", qarr.as_any().downcast_ref::<Float32Array>().unwrap(), 10)
-            .context("scanner.nearest")?;
-        let mut stream = scanner.try_into_stream().await?;
-        let mut hits = 0;
-        while let Some(batch) = stream.next().await {
-            let batch = batch?;
-            hits += batch.num_rows();
-        }
-        let us = t0.elapsed().as_micros() as f32;
-        latencies_us.push(us);
-        if i == 0 {
-            eprintln!("  first query: {} hits in {:.0}us (includes any lazy init)", hits, us);
-        }
-    }
-
-    latencies_us.sort_by(|a, b| a.partial_cmp(b).unwrap());
-    let p = |pct: f32| -> f32 {
-        let idx = ((latencies_us.len() as f32 - 1.0) * pct).round() as usize;
-        latencies_us[idx.min(latencies_us.len() - 1)]
-    };
-    Ok((p(0.50), p(0.95), p(0.99)))
-}
-
-/// Random row access via Lance's `take` — fetch 20 random rows by index, measure avg latency.
-async fn measure_random_access_lance(path: &str) -> Result<f32> {
-    use lance::dataset::Dataset;
-    let dataset = Dataset::open(path).await?;
-    let n = dataset.count_rows(None).await?;
-    let indices: Vec<u64> = (0..20).map(|i| ((i as u64) * (n as u64 / 23)) % (n as u64)).collect();
-
-    // Full-schema projection — Lance's Schema implements Into<ProjectionRequest>.
-    let schema = dataset.schema().clone();
-    let mut total_us: u128 = 0;
-    for idx in &indices {
-        let t0 = Instant::now();
-        let _batch = dataset.take(&[*idx], schema.clone()).await?;
-        total_us += t0.elapsed().as_micros();
-    }
-    Ok(total_us as f32 / indices.len() as f32)
-}
-
-/// Random row access for Parquet — full scan + filter. There's no random-access
-/// primitive in vanilla Parquet, so this is the cost of finding one specific row.
-/// This is the cost our current design pays for "get doc X's full text for RAG."
-fn measure_random_access_parquet(path: &str) -> Result<f32> {
-    use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
-    use std::fs::File;
-
-    // We simulate 5 lookups — full scan each time. 20 would be painful.
-    let iters = 5;
-    let mut total_us: u128 = 0;
-    for _ in 0..iters {
-        let t0 = Instant::now();
-        let file = File::open(path)?;
-        let builder = ParquetRecordBatchReaderBuilder::try_new(file)?;
-        let reader = builder.build()?;
-        // Iterate until we've conceptually found a row — we stop early if
-        // we wanted row 50000, but we have to at least read its batch.
-        let mut seen = 0usize;
-        for b in reader {
-            let b = b?;
-            seen += b.num_rows();
-            if seen > 50000 { break; }
-        }
-        total_us += t0.elapsed().as_micros();
-    }
-    Ok(total_us as f32 / iters as f32)
-}
-
-/// Append 10K new rows to the existing Lance dataset.
-/// Measures the "ingest delta" cost without full rewrite.
-async fn append_10k_rows(path: &str, dims: usize) -> Result<()> {
-    use lance::dataset::{Dataset, WriteMode, WriteParams};
-
-    let dataset = Dataset::open(path).await?;
-    let schema = dataset.schema();
-    let arrow_schema: Arc<Schema> = Arc::new(schema.into());
-
-    // Build a 10K row batch with random-ish data matching the existing schema.
-    let n = 10_000;
-    let arrays: Vec<ArrayRef> = arrow_schema
-        .fields()
-        .iter()
-        .map(|f| -> Result<ArrayRef> {
-            match f.data_type() {
-                DataType::Utf8 => {
-                    let vals: Vec<String> = (0..n).map(|i| format!("appended-{}", i)).collect();
-                    Ok(Arc::new(arrow_array::StringArray::from(vals)))
-                }
-                DataType::Int32 => {
-                    let vals: Vec<i32> = (0..n as i32).collect();
-                    Ok(Arc::new(arrow_array::Int32Array::from(vals)))
-                }
-                DataType::FixedSizeList(_, _) => {
-                    let floats: Vec<f32> = (0..n * dims).map(|i| (i as f32).sin()).collect();
-                    let values = Float32Array::from(floats);
-                    let field = Arc::new(Field::new("item", DataType::Float32, true));
-                    let fsl = FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None)?;
-                    Ok(Arc::new(fsl))
-                }
-                other => anyhow::bail!("unsupported append column type: {:?}", other),
-            }
-        })
-        .collect::<Result<Vec<_>>>()?;
-
-    let batch = RecordBatch::try_new(arrow_schema.clone(), arrays)?;
-    let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), arrow_schema);
-    let params = WriteParams { mode: WriteMode::Append, ..Default::default() };
-    Dataset::write(reader, path, Some(params)).await?;
-    Ok(())
-}
-
-/// Grab a few existing vectors from the dataset to use as self-similar queries.
-async fn sample_query_vectors(
-    dataset: &lance::dataset::Dataset,
-    count: usize,
-) -> Result<Vec<Vec<f32>>> {
-    use futures::StreamExt;
-
-    // Just take the first `count` rows; good enough for latency measurement.
-    let scanner = dataset.scan();
-    let mut scanner = scanner;
-    scanner.limit(Some(count as i64), None)?;
-    scanner.project(&["vector"])?;
-    let mut stream = scanner.try_into_stream().await?;
-
-    let mut out = Vec::with_capacity(count);
-    while let Some(batch) = stream.next().await {
-        let batch = batch?;
-        let vector_col = batch
-            .column(0)
-            .as_any()
-            .downcast_ref::<FixedSizeListArray>()
-            .context("vector column must be FixedSizeList")?;
-
-        for row in 0..vector_col.len() {
-            if out.len() >= count { break; }
-            let values = vector_col.value(row);
-            let f32_arr = values
-                .as_any()
-                .downcast_ref::<Float32Array>()
-                .context("inner array must be Float32")?;
-            let mut v = Vec::with_capacity(f32_arr.len());
-            for i in 0..f32_arr.len() {
-                v.push(f32_arr.value(i));
-            }
-            out.push(v);
-        }
-        if out.len() >= count { break; }
-    }
-    Ok(out)
-}
-

crates/queryd/Cargo.toml

@@ -15,7 +15,6 @@ tracing = { workspace = true }
 datafusion = { workspace = true }
 object_store = { workspace = true }
 arrow = { workspace = true }
-parquet = { workspace = true }
 bytes = { workspace = true }
 futures = { workspace = true }
 url = { workspace = true }

crates/queryd/src/context.rs

@@ -5,7 +5,6 @@ use datafusion::datasource::file_format::parquet::ParquetFormat;
 use datafusion::prelude::*;
 use object_store::ObjectStore;
 use std::sync::Arc;
-use storaged::registry::BucketRegistry;
 use url::Url;
 
 use crate::cache::MemCache;
@@ -13,61 +12,27 @@ use crate::delta;
 
 const STORE_SCHEME: &str = "lakehouse";
 
-/// URL scheme used to register a bucket's object store with DataFusion.
-/// Bucket names safely embed in URL host components after sanitizing
-/// `:` (used in `profile:user`) to `-`.
-fn bucket_scheme(bucket: &str) -> String {
-    format!("{STORE_SCHEME}-{}", bucket.replace(':', "-"))
-}
-
-/// Build a `ListingTableUrl`-shaped string for an object in a bucket.
-fn bucket_object_url(bucket: &str, key: &str) -> String {
-    format!("{}://data/{}", bucket_scheme(bucket), key)
-}
-
-/// Build the registration base URL for a bucket.
-fn bucket_base_url(bucket: &str) -> String {
-    format!("{}://data/", bucket_scheme(bucket))
-}
-
 /// Query engine with in-memory cache and delta merge support.
-/// Federation layer 2: holds a `BucketRegistry` so cross-bucket queries
-/// route reads to the right ObjectStore based on `ObjectRef.bucket`.
 #[derive(Clone)]
 pub struct QueryEngine {
     registry: Registry,
-    buckets: Arc<BucketRegistry>,
-    /// Primary bucket store cached for backward-compat callers (compact,
-    /// workspace ops) that haven't been migrated to bucket-aware paths.
     store: Arc<dyn ObjectStore>,
     cache: MemCache,
 }
 
 impl QueryEngine {
-    pub fn new(registry: Registry, buckets: Arc<BucketRegistry>, cache: MemCache) -> Self {
-        let store = buckets.default_store();
-        Self { registry, buckets, store, cache }
+    pub fn new(registry: Registry, store: Arc<dyn ObjectStore>, cache: MemCache) -> Self {
+        Self { registry, store, cache }
     }
 
     pub fn cache(&self) -> &MemCache {
         &self.cache
     }
 
-    /// Backward-compat: returns the primary bucket's store. Bucket-aware
-    /// callers should use `engine.buckets()` directly.
     pub fn store(&self) -> &Arc<dyn ObjectStore> {
         &self.store
     }
 
-    pub fn buckets(&self) -> &Arc<BucketRegistry> {
-        &self.buckets
-    }
-
-    /// Catalog registry — used by compact/tombstone integration.
-    pub fn catalog(&self) -> &Registry {
-        &self.registry
-    }
-
     /// Execute a SQL query. Uses cache for hot data, falls back to Parquet.
     pub async fn query(&self, sql: &str) -> Result<Vec<arrow::array::RecordBatch>, String> {
         let ctx = self.build_context().await?;
@@ -78,16 +43,11 @@ impl QueryEngine {
 
     /// Pin a dataset into the memory cache.
     pub async fn pin_dataset(&self, name: &str) -> Result<(), String> {
+        // Read from Parquet
         let ctx = SessionContext::new();
-
-        // Register every bucket so a multi-bucket dataset can be pinned.
-        for info in self.buckets.list().await {
-            let url = Url::parse(&bucket_base_url(&info.name))
-                .map_err(|e| format!("invalid store url: {e}"))?;
-            let store = self.buckets.get(&info.name)
-                .map_err(|e| format!("registry inconsistency: {e}"))?;
-            ctx.runtime_env().register_object_store(&url, store);
-        }
+        let base_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
+            .map_err(|e| format!("invalid store url: {e}"))?;
+        ctx.runtime_env().register_object_store(&base_url, self.store.clone());
 
         let dataset = self.registry.get_by_name(name).await
             .ok_or_else(|| format!("dataset not found: {name}"))?;
@@ -98,10 +58,7 @@ impl QueryEngine {
 
         let opts = ListingOptions::new(Arc::new(ParquetFormat::default()));
         let table_paths: Vec<ListingTableUrl> = dataset.objects.iter()
-            .filter_map(|o| {
-                let bucket = if o.bucket.is_empty() { "primary" } else { &o.bucket };
-                ListingTableUrl::parse(&bucket_object_url(bucket, &o.key)).ok()
-            })
+            .filter_map(|o| ListingTableUrl::parse(&format!("{STORE_SCHEME}://data/{}", o.key)).ok())
             .collect();
 
         let schema = opts.infer_schema(&ctx.state(), &table_paths[0]).await
@@ -122,41 +79,9 @@ impl QueryEngine {
     async fn build_context(&self) -> Result<SessionContext, String> {
         let ctx = SessionContext::new();
 
-        // Phase E: snapshot tombstones by dataset before registering tables
-        // so we can wrap tombstoned tables in a filter view. The underlying
-        // base table is registered under an internal name `__raw__{dataset}`
-        // and the public `{dataset}` name becomes the filtered view.
-        let all_dataset_names: Vec<String> = self.registry.list().await
-            .iter().map(|d| d.name.clone()).collect();
-        let tombstones_by_dataset = self
-            .registry
-            .tombstones()
-            .all_grouped(&all_dataset_names)
-            .await
-            .unwrap_or_default();
-
-        // Federation layer 2: register every configured bucket as its own
-        // DataFusion ObjectStore under a distinct URL scheme. Each
-        // dataset's ObjectRef.bucket determines which store DataFusion
-        // routes to at scan time.
-        let bucket_infos = self.buckets.list().await;
-        for info in &bucket_infos {
-            let url_str = bucket_base_url(&info.name);
-            let url = Url::parse(&url_str)
-                .map_err(|e| format!("invalid store url '{url_str}': {e}"))?;
-            // unknown-bucket here would be a config invariant violation,
-            // since we just listed them — propagate as an error to surface it.
-            let store = self.buckets.get(&info.name)
-                .map_err(|e| format!("registry inconsistency: {e}"))?;
-            ctx.runtime_env().register_object_store(&url, store);
-        }
-
-        // Backward-compat: also register the legacy `lakehouse://data/`
-        // URL pointing at primary, in case any code path still constructs
-        // ListingTableUrls without a bucket prefix.
-        let legacy_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
-            .map_err(|e| format!("invalid legacy store url: {e}"))?;
-        ctx.runtime_env().register_object_store(&legacy_url, self.store.clone());
+        let base_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
+            .map_err(|e| format!("invalid store url: {e}"))?;
+        ctx.runtime_env().register_object_store(&base_url, self.store.clone());
 
         let datasets = self.registry.list().await;
         for dataset in &datasets {
@@ -164,9 +89,9 @@ impl QueryEngine {
                 continue;
             }
 
-            // Check cache first — cached batches are bucket-agnostic since
-            // they're already materialized in memory.
+            // Check cache first
             if let Some(cached) = self.cache.get(&dataset.name).await {
+                // Load any delta files and merge
                 let delta_batches = delta::load_deltas(&self.store, &dataset.name).await.unwrap_or_default();
 
                 let mut all_batches = cached.batches;
@@ -181,15 +106,10 @@ impl QueryEngine {
                 continue;
             }
 
-            // Build ListingTable URLs that include the per-object bucket
-            // scheme. DataFusion routes each scan to the right store
-            // automatically based on the URL.
+            // Fall back to Parquet file
             let opts = ListingOptions::new(Arc::new(ParquetFormat::default()));
             let table_paths: Vec<ListingTableUrl> = dataset.objects.iter()
-                .filter_map(|o| {
-                    let bucket = if o.bucket.is_empty() { "primary" } else { &o.bucket };
-                    ListingTableUrl::parse(&bucket_object_url(bucket, &o.key)).ok()
-                })
+                .filter_map(|o| ListingTableUrl::parse(&format!("{STORE_SCHEME}://data/{}", o.key)).ok())
                 .collect();
 
             if table_paths.is_empty() {
@@ -204,123 +124,8 @@ impl QueryEngine {
             let table = ListingTable::try_new(config)
                 .map_err(|e| format!("table creation failed for {}: {e}", dataset.name))?;
 
-            // Decide the registration name: if this dataset has any
-            // tombstones, the raw table gets an internal name and the
-            // public name becomes a filtered view.
-            let tombstone_entry = tombstones_by_dataset.get(&dataset.name);
-            let register_name = if tombstone_entry.is_some() {
-                format!("__raw__{}", dataset.name)
-            } else {
-                dataset.name.clone()
-            };
-
-            if let Err(e) = ctx.register_table(register_name.as_str(), Arc::new(table)) {
-                let msg = e.to_string();
-                if msg.contains("already exists") {
-                    tracing::debug!("skip duplicate manifest registration: {}", register_name);
-                    continue;
-                } else {
-                    return Err(format!("table registration failed for {}: {}", register_name, msg));
-                }
-            }
-
-            // If there are tombstones, register the public name as a
-            // filtered view that excludes tombstoned row_key_values.
-            if let Some((key_col, values)) = tombstone_entry {
-                // Build WHERE NOT IN (...) — quote values to be SQL-safe.
-                // For string keys this is a literal list; for integer keys
-                // CAST(col AS VARCHAR) makes the comparison unambiguous.
-                let quoted: Vec<String> = values.iter()
-                    .map(|v| format!("'{}'", v.replace('\'', "''")))
-                    .collect();
-                let sql = format!(
-                    "SELECT * FROM \"{}\" WHERE CAST(\"{}\" AS VARCHAR) NOT IN ({})",
-                    register_name, key_col, quoted.join(", "),
-                );
-                tracing::debug!(
-                    "tombstone filter for '{}': {} row_keys excluded",
-                    dataset.name, values.len(),
-                );
-                match ctx.sql(&sql).await {
-                    Ok(df) => {
-                        if let Err(e) = ctx.register_table(dataset.name.as_str(), df.into_view()) {
-                            let msg = e.to_string();
-                            if msg.contains("already exists") {
-                                tracing::debug!("skip duplicate tombstone view: {}", dataset.name);
-                            } else {
-                                tracing::warn!("tombstone view registration failed for {}: {msg}", dataset.name);
-                            }
-                        }
-                    }
-                    Err(e) => {
-                        tracing::warn!("tombstone view SQL failed for '{}': {e}", dataset.name);
-                    }
-                }
-            }
-        }
-
-        // Phase D: register every AiView as a DataFusion view that wraps
-        // the base table with the safe projection + filter + redactions.
-        for view in self.registry.list_views().await {
-            // Build the SELECT clause with column whitelist and any
-            // redaction expressions. Each column is either the raw column
-            // name or an expression aliased back to the column name so
-            // downstream queries see the same shape.
-            let select_cols: Vec<String> = view.columns.iter().map(|col| {
-                match view.column_redactions.get(col) {
-                    None => format!("\"{}\"", col),
-                    Some(shared::types::Redaction::Null) => {
-                        format!("CAST(NULL AS VARCHAR) AS \"{}\"", col)
-                    }
-                    Some(shared::types::Redaction::Hash) => {
-                        // DataFusion has digest('sha256', value) — but column
-                        // type might not be string. Cast first to be safe.
-                        format!("digest(CAST(\"{}\" AS VARCHAR), 'sha256') AS \"{}\"", col, col)
-                    }
-                    Some(shared::types::Redaction::Mask { keep_prefix, keep_suffix }) => {
-                        // Keep first N + last M chars, replace middle with stars.
-                        // CASE NULL through with COALESCE for null-safe behavior.
-                        format!(
-                            "CASE WHEN \"{c}\" IS NULL THEN NULL ELSE \
-                                concat(\
-                                    substr(CAST(\"{c}\" AS VARCHAR), 1, {p}), \
-                                    repeat('*', greatest(0, length(CAST(\"{c}\" AS VARCHAR)) - {p} - {s})), \
-                                    substr(CAST(\"{c}\" AS VARCHAR), greatest(1, length(CAST(\"{c}\" AS VARCHAR)) - {s} + 1))\
-                                ) END AS \"{c}\"",
-                            c = col, p = keep_prefix, s = keep_suffix,
-                        )
-                    }
-                }
-            }).collect();
-
-            let where_clause = view.row_filter
-                .as_deref()
-                .map(|f| format!(" WHERE {}", f))
-                .unwrap_or_default();
-            let view_sql = format!(
-                "SELECT {} FROM \"{}\"{}",
-                select_cols.join(", "),
-                view.base_dataset,
-                where_clause,
-            );
-
-            tracing::debug!("registering AiView '{}': {}", view.name, view_sql);
-
-            let df = match ctx.sql(&view_sql).await {
-                Ok(df) => df,
-                Err(e) => {
-                    tracing::warn!("view '{}' SQL invalid, skipping: {e}", view.name);
-                    continue;
-                }
-            };
-            if let Err(e) = ctx.register_table(view.name.as_str(), df.into_view()) {
-                let msg = e.to_string();
-                if msg.contains("already exists") {
-                    tracing::debug!("skip duplicate view registration: {}", view.name);
-                } else {
-                    tracing::warn!("view '{}' registration failed: {e}", view.name);
-                }
-            }
+            ctx.register_table(&dataset.name, Arc::new(table))
+                .map_err(|e| format!("table registration failed for {}: {e}", dataset.name))?;
         }
 
         Ok(ctx)

crates/queryd/src/delta.rs

@@ -3,15 +3,10 @@
 /// we write small delta files. At query time, deltas are merged with the base.
 /// Periodic compaction merges deltas into the base file.
 
-use arrow::array::{Array, BooleanArray, Int32Array, Int64Array, RecordBatch, StringArray};
-use arrow::compute::filter_record_batch;
+use arrow::array::RecordBatch;
+use arrow::datatypes::SchemaRef;
 use bytes::Bytes;
 use object_store::ObjectStore;
-use parquet::arrow::ArrowWriter;
-use parquet::basic::Compression;
-use parquet::file::properties::WriterProperties;
-use shared::types::Tombstone;
-use std::collections::HashSet;
 use std::sync::Arc;
 
 use shared::arrow_helpers::{parquet_to_record_batches, record_batch_to_parquet};
@@ -63,207 +58,74 @@ pub async fn load_deltas(
 }
 
 /// Compact: merge base Parquet + all deltas into a single new base file.
-/// Optionally filters out tombstoned rows — this is the physical-deletion
-/// half of Phase E (the query-time filter handles the "immediate hide",
-/// compaction handles the "actually remove from disk" requirement).
-///
-/// Tombstones must share one `row_key_column` (enforced at tombstone write).
-/// If non-empty, every row in the merged output whose row_key_column value
-/// matches a tombstone is dropped before the new base is written.
+/// Optionally deduplicates by a primary key column.
 pub async fn compact(
     store: &Arc<dyn ObjectStore>,
     dataset_name: &str,
     base_key: &str,
     primary_key_col: Option<&str>,
-    tombstones: &[Tombstone],
 ) -> Result<CompactResult, String> {
     // Load base
     let base_data = ops::get(store, base_key).await?;
-    let (_schema, mut base_batches) = parquet_to_record_batches(&base_data)?;
+    let (schema, mut base_batches) = parquet_to_record_batches(&base_data)?;
 
     // Load deltas
     let delta_batches = load_deltas(store, dataset_name).await?;
     let delta_count = delta_batches.len();
 
-    let has_tombstones = !tombstones.is_empty();
-    let nothing_to_do = delta_batches.is_empty() && !has_tombstones;
-    if nothing_to_do {
+    if delta_batches.is_empty() {
         return Ok(CompactResult {
             base_rows: base_batches.iter().map(|b| b.num_rows()).sum(),
             delta_rows: 0,
             final_rows: base_batches.iter().map(|b| b.num_rows()).sum(),
             deltas_merged: 0,
-            tombstones_applied: 0,
-            rows_dropped_by_tombstones: 0,
         });
     }
 
     base_batches.extend(delta_batches);
-    let pre_filter_rows: usize = base_batches.iter().map(|b| b.num_rows()).sum();
+
+    let base_rows: usize = base_batches.iter().map(|b| b.num_rows()).sum();
 
     // If primary key specified, deduplicate (keep last occurrence)
-    let merged_batches = if let Some(_pk) = primary_key_col {
-        // Current simplification — full dedup requires sort by PK; leave
-        // that work for a follow-up. Tombstone filter still runs below.
+    let final_batches = if let Some(_pk) = primary_key_col {
+        // For now, just concatenate. Full dedup requires sorting by PK
+        // and keeping the last row per key — this is a simplification.
+        // TODO: implement proper merge with dedup
         base_batches
     } else {
         base_batches
     };
 
-    // Tombstone filter: drop rows whose row_key matches.
-    let (final_batches, dropped) = if has_tombstones {
-        apply_tombstone_filter(merged_batches, tombstones)?
-    } else {
-        (merged_batches, 0)
-    };
-
     let final_rows: usize = final_batches.iter().map(|b| b.num_rows()).sum();
 
-    // Write merged base atomically: build ONE Parquet file (single
-    // footer!) via ArrowWriter, stage it under a temporary key, read
-    // back to verify row count matches what we intended, THEN swap
-    // into base_key. If anything goes wrong between build and swap,
-    // the original base_key is untouched.
-    //
-    // Why this matters: concatenating N record_batch_to_parquet()
-    // outputs produces N independent Parquet files glued together —
-    // readers see only the first footer. That silently corrupted the
-    // candidates table before this code was fixed.
-    let schema_ref = final_batches
-        .first()
-        .map(|b| b.schema())
-        .ok_or_else(|| "no batches to write".to_string())?;
-
-    // Use Snappy to match the compression of files written by ingest —
-    // otherwise compaction would inflate file size 3× on each rewrite.
-    let props = WriterProperties::builder()
-        .set_compression(Compression::SNAPPY)
-        .build();
-
-    let mut buf: Vec<u8> = Vec::with_capacity(16 * 1024 * 1024);
-    {
-        let mut writer = ArrowWriter::try_new(&mut buf, schema_ref.clone(), Some(props))
-            .map_err(|e| format!("ArrowWriter init: {e}"))?;
-        for batch in &final_batches {
-            writer.write(batch).map_err(|e| format!("write batch: {e}"))?;
-        }
-        writer.close().map_err(|e| format!("close writer: {e}"))?;
+    // Write merged base
+    let mut merged_parquet = Vec::new();
+    for batch in &final_batches {
+        let pq = record_batch_to_parquet(batch)?;
+        merged_parquet.extend_from_slice(&pq);
     }
+    ops::put(store, base_key, Bytes::from(merged_parquet)).await?;
 
-    // Verify before committing — if the written bytes don't parse as
-    // the row count we expect, DON'T overwrite the base file.
-    let (_, verify_batches) = parquet_to_record_batches(&buf)
-        .map_err(|e| format!("verify parse: {e}"))?;
-    let verify_rows: usize = verify_batches.iter().map(|b| b.num_rows()).sum();
-    if verify_rows != final_rows {
-        return Err(format!(
-            "compact verification failed: wrote {} bytes that parse as {} rows (expected {}); base_key '{}' untouched",
-            buf.len(), verify_rows, final_rows, base_key,
-        ));
-    }
-
-    // Stage under a temp key first, then promote via a second put
-    // over base_key. Object stores don't have atomic rename across
-    // arbitrary backends; this is "write twice + delete temp" which
-    // is the widely-adopted pattern.
-    let temp_key = format!("{base_key}.compact-{}.tmp", chrono::Utc::now().timestamp_millis());
-    ops::put(store, &temp_key, Bytes::from(buf.clone())).await?;
-    ops::put(store, base_key, Bytes::from(buf)).await?;
-    let _ = ops::delete(store, &temp_key).await;
-
-    // Only now that base is durably updated, delete delta files.
+    // Delete delta files
     let delta_keys = list_deltas(store, dataset_name).await?;
     for key in &delta_keys {
         let _ = ops::delete(store, key).await;
     }
 
-    tracing::info!(
-        "compacted '{}': {} deltas merged, {} tombstones applied ({} rows dropped), {} → {} rows",
-        dataset_name, delta_count, tombstones.len(), dropped, pre_filter_rows, final_rows,
-    );
+    tracing::info!("compacted '{}': {} deltas merged, {} → {} rows", dataset_name, delta_count, base_rows, final_rows);
 
     Ok(CompactResult {
-        base_rows: pre_filter_rows - delta_count,  // rough base-before-deltas
-        delta_rows: delta_count,
+        base_rows,
+        delta_rows: final_rows - base_rows + delta_count, // approximate
         final_rows,
         deltas_merged: delta_count,
-        tombstones_applied: tombstones.len(),
-        rows_dropped_by_tombstones: dropped,
     })
 }
 
-/// Filter every batch in `batches` to exclude rows whose `row_key_column`
-/// value appears in the tombstone set. All tombstones must share one
-/// row_key_column — enforced at tombstone write time.
-fn apply_tombstone_filter(
-    batches: Vec<RecordBatch>,
-    tombstones: &[Tombstone],
-) -> Result<(Vec<RecordBatch>, usize), String> {
-    let col_name = &tombstones[0].row_key_column;
-    let bad_values: HashSet<String> = tombstones
-        .iter()
-        .map(|t| t.row_key_value.clone())
-        .collect();
-
-    let mut out = Vec::with_capacity(batches.len());
-    let mut total_dropped = 0usize;
-
-    for batch in batches {
-        let col_idx = match batch.schema().index_of(col_name) {
-            Ok(i) => i,
-            Err(_) => {
-                // Column not in this batch — nothing to drop, pass through.
-                out.push(batch);
-                continue;
-            }
-        };
-        let col = batch.column(col_idx);
-
-        // Build a "keep" mask: true where row is NOT tombstoned.
-        let mask: Vec<bool> = if let Some(s) = col.as_any().downcast_ref::<StringArray>() {
-            (0..s.len()).map(|i| {
-                s.is_null(i) || !bad_values.contains(s.value(i))
-            }).collect()
-        } else if let Some(a) = col.as_any().downcast_ref::<Int32Array>() {
-            (0..a.len()).map(|i| {
-                a.is_null(i) || !bad_values.contains(&a.value(i).to_string())
-            }).collect()
-        } else if let Some(a) = col.as_any().downcast_ref::<Int64Array>() {
-            (0..a.len()).map(|i| {
-                a.is_null(i) || !bad_values.contains(&a.value(i).to_string())
-            }).collect()
-        } else {
-            // Key column of an unsupported type — log and pass through
-            // rather than panic. Operator has to handle this manually.
-            tracing::warn!(
-                "tombstone filter: column '{}' has unsupported type {:?}; passing rows through",
-                col_name, col.data_type(),
-            );
-            out.push(batch);
-            continue;
-        };
-
-        let kept = mask.iter().filter(|b| **b).count();
-        let dropped = mask.len() - kept;
-        total_dropped += dropped;
-        if dropped == 0 {
-            out.push(batch);
-            continue;
-        }
-        let bool_arr = BooleanArray::from(mask);
-        let filtered = filter_record_batch(&batch, &bool_arr)
-            .map_err(|e| format!("filter batch: {e}"))?;
-        out.push(filtered);
-    }
-    Ok((out, total_dropped))
-}
-
 #[derive(Debug, Clone, serde::Serialize)]
 pub struct CompactResult {
     pub base_rows: usize,
     pub delta_rows: usize,
     pub final_rows: usize,
     pub deltas_merged: usize,
-    pub tombstones_applied: usize,
-    pub rows_dropped_by_tombstones: usize,
 }

crates/queryd/src/service.rs

@@ -184,31 +184,13 @@ async fn compact_dataset(
     State(state): State<QueryState>,
     Json(req): Json<CompactRequest>,
 ) -> impl IntoResponse {
-    // Phase E: pull tombstones for this dataset and let compact physically
-    // drop those rows. After a successful rewrite, clear the tombstone log
-    // — the rows are gone from disk, the tombstones have done their job.
-    let tombstones = state
-        .engine
-        .catalog()
-        .list_tombstones(&req.dataset)
-        .await
-        .unwrap_or_default();
-
     match delta::compact(
         state.engine.store(),
         &req.dataset,
         &req.base_key,
         req.primary_key.as_deref(),
-        &tombstones,
     ).await {
-        Ok(result) => {
-            if result.rows_dropped_by_tombstones > 0 {
-                if let Err(e) = state.engine.catalog().tombstones().clear(&req.dataset).await {
-                    tracing::warn!("post-compact tombstone clear failed: {e}");
-                }
-            }
-            Ok(Json(result))
-        }
+        Ok(result) => Ok(Json(result)),
         Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
     }
 }

crates/shared/Cargo.toml

@@ -15,5 +15,3 @@ bytes = { workspace = true }
 sha2 = { workspace = true }
 toml = { workspace = true }
 tracing = { workspace = true }
-tokio = { workspace = true }
-async-trait = "0.1"

crates/shared/src/config.rs

@@ -16,46 +16,8 @@ pub struct Config {
     pub auth: AuthConfig,
     #[serde(default)]
     pub observability: ObservabilityConfig,
-    #[serde(default)]
-    pub agent: AgentSettings,
 }
 
-/// Phase 16.2 — background autotune agent settings.
-///
-/// Duplicated from `vectord::agent::AgentConfig` because `shared` can't
-/// depend on `vectord` (vectord already depends on shared). The gateway
-/// copies these into the vectord config at startup.
-#[derive(Debug, Clone, Deserialize)]
-pub struct AgentSettings {
-    #[serde(default)]
-    pub enabled: bool,
-    #[serde(default = "default_cycle_interval_secs")]
-    pub cycle_interval_secs: u64,
-    #[serde(default = "default_cooldown_secs")]
-    pub cooldown_between_trials_secs: u64,
-    #[serde(default = "default_min_recall")]
-    pub min_recall: f32,
-    #[serde(default = "default_max_trials_per_hour")]
-    pub max_trials_per_hour: u32,
-}
-
-impl Default for AgentSettings {
-    fn default() -> Self {
-        Self {
-            enabled: false,
-            cycle_interval_secs: default_cycle_interval_secs(),
-            cooldown_between_trials_secs: default_cooldown_secs(),
-            min_recall: default_min_recall(),
-            max_trials_per_hour: default_max_trials_per_hour(),
-        }
-    }
-}
-
-fn default_cycle_interval_secs() -> u64 { 60 }
-fn default_cooldown_secs() -> u64 { 30 }
-fn default_min_recall() -> f32 { 0.9 }
-fn default_max_trials_per_hour() -> u32 { 30 }
-
 #[derive(Debug, Clone, Deserialize)]
 pub struct GatewayConfig {
     #[serde(default = "default_host")]
@@ -66,39 +28,8 @@ pub struct GatewayConfig {
 
 #[derive(Debug, Clone, Deserialize)]
 pub struct StorageConfig {
-    /// Legacy single-backend root. If `buckets` is empty, this is used to
-    /// create an implicit `primary` bucket at this path — preserves the
-    /// pre-federation config shape.
     #[serde(default = "default_storage_root")]
     pub root: String,
-
-    /// Where profile buckets are rooted when auto-provisioned.
-    #[serde(default = "default_profile_root")]
-    pub profile_root: String,
-
-    /// Name of the bucket used for read fallback when a target bucket is
-    /// unreachable. If `None`, no fallback — reads fail hard.
-    #[serde(default)]
-    pub rescue_bucket: Option<String>,
-
-    /// Explicitly configured buckets. Empty = backward-compat single-bucket
-    /// mode driven by `root`.
-    #[serde(default)]
-    pub buckets: Vec<BucketConfig>,
-}
-
-#[derive(Debug, Clone, Deserialize)]
-pub struct BucketConfig {
-    pub name: String,
-    pub backend: String,           // "local" | "s3"
-    /// Local filesystem root (for backend = "local")
-    pub root: Option<String>,
-    /// S3 bucket name (for backend = "s3")
-    pub bucket: Option<String>,
-    pub region: Option<String>,
-    pub endpoint: Option<String>,
-    /// Handle for the secrets provider — never the literal credential.
-    pub secret_ref: Option<String>,
 }
 
 #[derive(Debug, Clone, Deserialize, Default)]
@@ -147,7 +78,6 @@ pub struct ObservabilityConfig {
 fn default_host() -> String { "0.0.0.0".to_string() }
 fn default_gateway_port() -> u16 { 3100 }
 fn default_storage_root() -> String { "./data".to_string() }
-fn default_profile_root() -> String { "./data/_profiles".to_string() }
 fn default_manifest_prefix() -> String { "_catalog/manifests".to_string() }
 fn default_sidecar_url() -> String { "http://localhost:3200".to_string() }
 fn default_embed_model() -> String { "nomic-embed-text".to_string() }
@@ -185,19 +115,13 @@ impl Default for Config {
     fn default() -> Self {
         Self {
             gateway: GatewayConfig { host: default_host(), port: default_gateway_port() },
-            storage: StorageConfig {
-                root: default_storage_root(),
-                profile_root: default_profile_root(),
-                rescue_bucket: None,
-                buckets: Vec::new(),
-            },
+            storage: StorageConfig { root: default_storage_root() },
             catalog: CatalogConfig::default(),
             query: QueryConfig::default(),
             sidecar: SidecarConfig { url: default_sidecar_url() },
             ai: AiConfig::default(),
             auth: AuthConfig::default(),
             observability: ObservabilityConfig::default(),
-            agent: AgentSettings::default(),
         }
     }
 }

crates/shared/src/lib.rs

@@ -3,4 +3,3 @@ pub mod errors;
 pub mod arrow_helpers;
 pub mod config;
 pub mod pii;
-pub mod secrets;

crates/shared/src/secrets.rs

@@ -1,144 +0,0 @@
-/// Secrets layer for bucket credentials.
-///
-/// Credentials never appear in `lakehouse.toml`. Each bucket entry has an
-/// opaque `secret_ref` handle, resolved at startup by a `SecretsProvider`.
-///
-/// MVP ships `FileSecretsProvider` — reads a TOML file at a path given by
-/// `LAKEHOUSE_SECRETS` env var, defaulting to `/etc/lakehouse/secrets.toml`.
-/// The file should be root-owned, mode 0600, never in git.
-///
-/// Future providers (Vault, SOPS, OS keyring) implement the same trait.
-
-use async_trait::async_trait;
-use serde::Deserialize;
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-
-#[derive(Debug, Clone)]
-pub struct BucketCredentials {
-    pub access_key: String,
-    pub secret_key: String,
-}
-
-// Redact credentials when debug-printing — NEVER let these end up in logs.
-impl BucketCredentials {
-    pub fn redacted(&self) -> String {
-        let prefix: String = self.access_key.chars().take(4).collect();
-        format!("access_key={}... (redacted)", prefix)
-    }
-}
-
-#[async_trait]
-pub trait SecretsProvider: Send + Sync {
-    /// Resolve a handle to credentials. Returns Err if the handle is unknown.
-    async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String>;
-
-    /// List known handles (for diagnostics only, never values).
-    async fn list_handles(&self) -> Vec<String>;
-}
-
-#[derive(Debug, Deserialize)]
-struct RawSecretEntry {
-    access_key: String,
-    secret_key: String,
-}
-
-/// File-backed secrets provider.
-pub struct FileSecretsProvider {
-    path: PathBuf,
-    cache: tokio::sync::RwLock<HashMap<String, BucketCredentials>>,
-}
-
-impl FileSecretsProvider {
-    pub fn new(path: impl Into<PathBuf>) -> Self {
-        Self {
-            path: path.into(),
-            cache: tokio::sync::RwLock::new(HashMap::new()),
-        }
-    }
-
-    /// Default path: env var `LAKEHOUSE_SECRETS` or `/etc/lakehouse/secrets.toml`.
-    pub fn from_env() -> Self {
-        let path = std::env::var("LAKEHOUSE_SECRETS")
-            .unwrap_or_else(|_| "/etc/lakehouse/secrets.toml".to_string());
-        Self::new(path)
-    }
-
-    /// Arc wrapper for convenient injection.
-    pub fn shared(path: impl Into<PathBuf>) -> Arc<dyn SecretsProvider> {
-        Arc::new(Self::new(path))
-    }
-
-    /// Load and populate the cache. Call once at startup.
-    pub async fn load(&self) -> Result<usize, String> {
-        if !self.path.exists() {
-            tracing::info!(
-                "secrets file {} not present — running with zero credentials configured",
-                self.path.display()
-            );
-            return Ok(0);
-        }
-        check_permissions(&self.path)?;
-        let content = std::fs::read_to_string(&self.path)
-            .map_err(|e| format!("read secrets file: {e}"))?;
-        let raw: HashMap<String, RawSecretEntry> = toml::from_str(&content)
-            .map_err(|e| format!("parse secrets file: {e}"))?;
-
-        let mut cache = self.cache.write().await;
-        for (handle, entry) in raw {
-            cache.insert(handle.clone(), BucketCredentials {
-                access_key: entry.access_key,
-                secret_key: entry.secret_key,
-            });
-        }
-        let count = cache.len();
-        tracing::info!("secrets: loaded {count} handles from {}", self.path.display());
-        Ok(count)
-    }
-}
-
-#[async_trait]
-impl SecretsProvider for FileSecretsProvider {
-    async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String> {
-        let cache = self.cache.read().await;
-        cache
-            .get(handle)
-            .cloned()
-            .ok_or_else(|| format!("secret handle not found: {handle}"))
-    }
-
-    async fn list_handles(&self) -> Vec<String> {
-        self.cache.read().await.keys().cloned().collect()
-    }
-}
-
-/// Reject world-readable secrets files. Fail closed.
-fn check_permissions(path: &Path) -> Result<(), String> {
-    #[cfg(unix)]
-    {
-        use std::os::unix::fs::PermissionsExt;
-        let meta = std::fs::metadata(path)
-            .map_err(|e| format!("stat secrets file: {e}"))?;
-        let mode = meta.permissions().mode() & 0o777;
-        if mode & 0o044 != 0 {
-            return Err(format!(
-                "secrets file {} is group/world-readable (mode {:o}); chmod 600 to proceed",
-                path.display(), mode
-            ));
-        }
-    }
-    let _ = path;
-    Ok(())
-}
-
-/// A zero-handles provider for tests / purely-local setups.
-pub struct EmptySecretsProvider;
-
-#[async_trait]
-impl SecretsProvider for EmptySecretsProvider {
-    async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String> {
-        Err(format!("no secrets provider configured (wanted handle '{handle}')"))
-    }
-    async fn list_handles(&self) -> Vec<String> { Vec::new() }
-}

crates/shared/src/types.rs

@@ -116,237 +116,4 @@ pub struct DatasetManifest {
     /// Row count (updated on ingest/compact)
     #[serde(default)]
     pub row_count: Option<u64>,
-
-    // --- Embedding freshness (Phase C) ---
-
-    /// When the attached vector index was last refreshed. `None` means this
-    /// dataset has no vector index yet, or has never been embedded.
-    #[serde(default)]
-    pub last_embedded_at: Option<chrono::DateTime<chrono::Utc>>,
-
-    /// When data was written that hasn't yet been embedded. `Some(t)` means
-    /// the vector index is out-of-date as of timestamp `t`. Cleared on refresh.
-    #[serde(default)]
-    pub embedding_stale_since: Option<chrono::DateTime<chrono::Utc>>,
-
-    /// How this dataset wants stale embeddings handled.
-    #[serde(default)]
-    pub embedding_refresh_policy: Option<RefreshPolicy>,
-}
-
-/// Controls what happens when new data is written to a dataset with an
-/// attached vector index.
-///
-/// - `Manual` (default): data writes set `embedding_stale_since`; nothing
-///   embeds until an operator or agent calls `/vectors/refresh/{dataset}`.
-/// - `OnAppend`: ingest fires a background refresh immediately after writing.
-///   Suitable for datasets where vector freshness matters more than ingest
-///   latency.
-/// - `Scheduled(cron)`: a timer or external scheduler triggers refresh at
-///   the named cadence. The scheduler itself is not in this ADR scope —
-///   this just declares the intent so operators can see which policy a
-///   dataset expects.
-#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
-#[serde(tag = "kind", rename_all = "snake_case")]
-pub enum RefreshPolicy {
-    Manual,
-    OnAppend,
-    Scheduled { cron: String },
-}
-
-impl Default for RefreshPolicy {
-    fn default() -> Self { Self::Manual }
-}
-
-/// AI-safe view of a dataset (Phase D).
-///
-/// A view is a named projection over a base dataset that:
-/// - whitelists the columns visible (everything else is invisible),
-/// - optionally applies a row filter (a SQL WHERE clause fragment),
-/// - optionally redacts column values (e.g. mask the last 4 of an SSN).
-///
-/// Queries that reference the view name see only the safe projection.
-/// The base table remains separately accessible by its raw name — this
-/// is a *capability surface*, not a data classification fence. Use
-/// access control (Phase 13) on top to enforce who can query which view.
-///
-/// Stored as JSON at `_catalog/views/{name}.json` in the primary bucket
-/// alongside dataset manifests. The view itself doesn't store data —
-/// queryd materializes it as a DataFusion view at SessionContext build
-/// time.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct AiView {
-    pub name: String,
-    /// Source dataset name. Must exist in the catalog.
-    pub base_dataset: String,
-    /// Whitelisted columns. Empty list is invalid (would expose nothing).
-    pub columns: Vec<String>,
-    /// Optional WHERE clause fragment (without the `WHERE` keyword), e.g.
-    /// `status != 'blocked' AND city = 'Chicago'`. SQL injection here is
-    /// the operator's problem — only trust this from authenticated admins.
-    #[serde(default)]
-    pub row_filter: Option<String>,
-    /// Per-column redactions: `{column_name -> Redaction}`.
-    #[serde(default)]
-    pub column_redactions: std::collections::HashMap<String, Redaction>,
-    pub created_at: chrono::DateTime<chrono::Utc>,
-    #[serde(default)]
-    pub created_by: String,
-    #[serde(default)]
-    pub description: String,
-}
-
-/// Per-model configuration for HNSW builds. Declared inline so the PRD
-/// invariant 9 ("every reader gets its own profile") can carry tuning
-/// preferences without forcing a round-trip to vectord's trial system.
-/// Mirrors `vectord::trial::HnswConfig` to avoid a cross-crate dep cycle.
-#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
-pub struct ProfileHnswConfig {
-    pub ef_construction: usize,
-    pub ef_search: usize,
-    #[serde(default)]
-    pub seed: Option<u64>,
-}
-
-impl Default for ProfileHnswConfig {
-    /// Matches the production HNSW default from Phase 15 trial sweep
-    /// (see docs/ADR and vectord::trial::HnswConfig::default).
-    fn default() -> Self {
-        Self { ef_construction: 80, ef_search: 30, seed: None }
-    }
-}
-
-/// A named client of the substrate — a local LLM, an AI agent, or a
-/// human role that should see a specific scoped view of the data.
-/// Profiles are the first-class "every reader gets its own profile"
-/// concept from PRD invariant 9.
-///
-/// Bound datasets can be raw catalog tables OR `AiView` names — either
-/// works at query time since both register as DataFusion tables with
-/// the same name. Mixing raw and views lets you compose access rules:
-/// bind a model to `candidates_safe` (PII-redacted view) and `placements`
-/// (raw table, no PII), and the model gets exactly what it needs.
-///
-/// Stored at `_catalog/profiles/{id}.json`. Activation (`/profile/{id}/
-/// activate`) pre-loads the embedding cache and builds HNSW with the
-/// profile's config — first search after activate is warm.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ModelProfile {
-    /// Unique id — slug form (alphanumeric + `-`, `_`). Used in URLs,
-    /// bucket names (`profile:{id}`), and trial journal prefixes.
-    pub id: String,
-    /// Ollama model tag (e.g. "qwen2.5:7b", "nomic-embed-text:latest").
-    /// Used when this profile is the "generate" or "embed" caller.
-    pub ollama_name: String,
-    /// Free-text description surfaced to operators. Who/what is this
-    /// profile for?
-    #[serde(default)]
-    pub description: String,
-    /// Names of datasets or AiViews this profile can read. A search
-    /// against an index whose `source` isn't in this list is refused.
-    pub bound_datasets: Vec<String>,
-    /// Per-profile HNSW config, used when activate builds the graph.
-    #[serde(default)]
-    pub hnsw_config: ProfileHnswConfig,
-    /// Which embedding model this profile expects. Defaults to
-    /// `nomic-embed-text` (the only one currently installed).
-    #[serde(default = "default_embed_model")]
-    pub embed_model: String,
-    pub created_at: chrono::DateTime<chrono::Utc>,
-    #[serde(default)]
-    pub created_by: String,
-    /// Federation layer 2: which bucket this profile's artifacts
-    /// (trial journals, promotions for its indexes) live in. If set and
-    /// not yet registered, `POST /vectors/profile/{id}/activate` auto-
-    /// provisions a local bucket under `storage.profile_root` and
-    /// registers it before warming indexes. `None` = share the primary
-    /// bucket (pre-federation behavior).
-    #[serde(default)]
-    pub bucket: Option<String>,
-    /// ADR-019 hybrid: which storage backend serves this profile's
-    /// indexes. Default Parquet so pre-existing profiles unchanged.
-    /// New profiles can opt into Lance for append-heavy / random-access
-    /// / 5M+ vector workloads.
-    #[serde(default)]
-    pub vector_backend: VectorBackend,
-}
-
-fn default_embed_model() -> String { "nomic-embed-text".to_string() }
-
-/// Per-profile / per-index storage backend choice.
-///
-/// `Parquet` (default) — vectors as binary-blob in Parquet + in-RAM HNSW
-/// for ANN. Best when:
-/// - Index ≤ ~5M vectors (fits in RAM)
-/// - Search latency dominates (HNSW p50 < 1ms at 100K)
-/// - Workload is mostly read, not append
-///
-/// `Lance` — vectors as native FixedSizeList in a Lance dataset, IVF_PQ
-/// on-disk index. Best when:
-/// - Index will exceed RAM ceiling (5M+ vectors per index)
-/// - Heavy random-row access for RAG (Lance: 311us; Parquet: ~35ms full
-///   scan per fetch)
-/// - Append-heavy ingest (Lance: 0.08s for 10K rows; Parquet: full
-///   ~330MB rewrite per change)
-/// - Need to rebuild index frequently (Lance IVF_PQ: 16s; HNSW: 230s
-///   at the same scale)
-///
-/// Defaults to Parquet for backward compat with pre-ADR-019 indexes.
-/// See `docs/ADR-019-vector-storage.md` for the full scorecard.
-#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
-#[serde(rename_all = "lowercase")]
-pub enum VectorBackend {
-    Parquet,
-    Lance,
-}
-
-impl Default for VectorBackend {
-    fn default() -> Self { VectorBackend::Parquet }
-}
-
-/// Soft-delete marker (Phase E).
-///
-/// Tombstones live beside the dataset in `_catalog/tombstones/{dataset}/`
-/// as append-log JSONL. Query-time filter in queryd reads all tombstones
-/// for each dataset and wraps the base table in a DataFusion view that
-/// excludes tombstoned rows. Physical deletion happens later (compaction),
-/// so the row count immediately reflects the delete but data is still on
-/// disk until compact runs. That's deliberate — it gives a reversal
-/// window and keeps the event journal audit trail intact.
-///
-/// All tombstones for a given dataset must use the same `row_key_column`
-/// (enforced on write); otherwise the query-time filter can't be built
-/// as a single WHERE clause.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct Tombstone {
-    pub dataset: String,
-    /// Column name that identifies the row (e.g. "candidate_id").
-    pub row_key_column: String,
-    /// Value of that column for the tombstoned row.
-    pub row_key_value: String,
-    pub deleted_at: chrono::DateTime<chrono::Utc>,
-    /// Human / system actor responsible for the delete (audit).
-    #[serde(default)]
-    pub actor: String,
-    /// Why (e.g. "GDPR request #1234", "user-requested erasure").
-    #[serde(default)]
-    pub reason: String,
-}
-
-/// How a column's values should be transformed before being returned.
-/// `Mask` is the most common — keeps a few visible chars, replaces the
-/// rest with `*`. `Hash` returns SHA-256 of the value for join keys you
-/// want to preserve referential integrity on without exposing the value.
-/// `Null` returns NULL unconditionally.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(tag = "kind", rename_all = "snake_case")]
-pub enum Redaction {
-    Null,
-    Hash,
-    Mask {
-        /// Number of leading characters left visible.
-        keep_prefix: usize,
-        /// Number of trailing characters left visible.
-        keep_suffix: usize,
-    },
 }

crates/storaged/Cargo.toml

@@ -13,4 +13,3 @@ tracing = { workspace = true }
 object_store = { workspace = true }
 bytes = { workspace = true }
 futures = { workspace = true }
-chrono = { workspace = true }

crates/storaged/src/append_log.rs

@@ -1,330 +0,0 @@
-/// Write-once batched append log.
-///
-/// Problem we're fixing: the error journal and HNSW trial journal both
-/// previously did read-modify-write of their whole JSONL file on every
-/// event. That's O(N²) cumulative work and generates huge churn at scale.
-/// It's exactly the pattern llms3.com flags as the "small-file /
-/// rewrite-amplification" anti-pattern.
-///
-/// This helper implements the pattern object storage actually wants:
-///
-/// - Events accumulate in an **in-memory buffer** (reads see them immediately).
-/// - When the buffer hits a threshold, or `flush()` is called, the buffer is
-///   written **as one new object** with a timestamp-sorted key.
-/// - Existing objects are never rewritten.
-/// - Reads enumerate all batch files, sort by key, and concat in order.
-/// - An explicit `compact()` reads every batch file, writes one consolidated
-///   file, and deletes the originals — the LSM-tree compaction idea applied
-///   to small JSONL events.
-///
-/// Storage layout:
-/// ```text
-/// {prefix}/
-///   batch_0001776319628000123.jsonl
-///   batch_0001776319745987654.jsonl
-///   batch_compacted_00001776319800000000.jsonl   (after compact())
-/// ```
-/// Key format: the zero-padded epoch microsecond of the write, so lexical
-/// sort == chronological sort.
-
-use bytes::Bytes;
-use chrono::Utc;
-use object_store::ObjectStore;
-use std::sync::Arc;
-use tokio::sync::Mutex;
-
-use crate::ops;
-
-const DEFAULT_FLUSH_THRESHOLD: usize = 32;
-
-pub struct AppendLog {
-    store: Arc<dyn ObjectStore>,
-    prefix: String,
-    buffer: Mutex<Vec<Vec<u8>>>,
-    flush_threshold: usize,
-}
-
-impl AppendLog {
-    /// Create a new append log rooted at `prefix` in the given object store.
-    /// Events auto-flush when buffer reaches `flush_threshold` (default 32).
-    pub fn new(store: Arc<dyn ObjectStore>, prefix: impl Into<String>) -> Self {
-        Self {
-            store,
-            prefix: prefix.into(),
-            buffer: Mutex::new(Vec::new()),
-            flush_threshold: DEFAULT_FLUSH_THRESHOLD,
-        }
-    }
-
-    pub fn with_flush_threshold(mut self, threshold: usize) -> Self {
-        self.flush_threshold = threshold.max(1);
-        self
-    }
-
-    /// Add an event. The returned future completes either immediately
-    /// (buffered) or after a flush, depending on whether the buffer hit the
-    /// threshold. Callers don't need to care either way.
-    pub async fn append(&self, line: Vec<u8>) -> Result<(), String> {
-        let should_flush = {
-            let mut buf = self.buffer.lock().await;
-            buf.push(line);
-            buf.len() >= self.flush_threshold
-        };
-        if should_flush {
-            self.flush().await?;
-        }
-        Ok(())
-    }
-
-    /// Force-flush the in-memory buffer to object storage as a single new
-    /// batch file. Safe to call anytime (idempotent no-op when buffer empty).
-    pub async fn flush(&self) -> Result<(), String> {
-        let batch = {
-            let mut buf = self.buffer.lock().await;
-            if buf.is_empty() { return Ok(()); }
-            std::mem::take(&mut *buf)
-        };
-
-        let ts_us = Utc::now().timestamp_micros().max(0) as u128;
-        let key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
-
-        let mut body = Vec::with_capacity(batch.iter().map(|b| b.len() + 1).sum());
-        for line in batch {
-            body.extend_from_slice(&line);
-            if !body.ends_with(b"\n") { body.push(b'\n'); }
-        }
-        ops::put(&self.store, &key, Bytes::from(body)).await
-    }
-
-    /// Read every event across all batch files + unflushed in-memory buffer.
-    /// Events are returned in chronological order.
-    pub async fn read_all(&self) -> Result<Vec<Vec<u8>>, String> {
-        let mut keys = self.list_batch_keys().await?;
-        keys.sort();
-
-        let mut out = Vec::new();
-        for key in keys {
-            let bytes = ops::get(&self.store, &key).await?;
-            for line in bytes.split(|b| *b == b'\n') {
-                if !line.is_empty() {
-                    out.push(line.to_vec());
-                }
-            }
-        }
-
-        // Include unflushed events so callers see the latest state
-        // whether or not someone ran flush() recently.
-        let buf = self.buffer.lock().await;
-        for line in buf.iter() {
-            out.push(line.clone());
-        }
-
-        Ok(out)
-    }
-
-    /// Consolidate all current batch files into one compacted file, then
-    /// delete the originals. Safe to call while appends are in flight:
-    /// new batches written during compaction get a higher timestamp and
-    /// survive. Fails closed — if anything goes wrong mid-delete, the
-    /// compacted file coexists with originals and next read sees duplicates
-    /// (which the dedup caller must handle) rather than data loss.
-    pub async fn compact(&self) -> Result<CompactStats, String> {
-        // Snapshot which files to compact BEFORE we write the new one.
-        let mut originals = self.list_batch_keys().await?;
-        originals.sort();
-
-        if originals.len() < 2 {
-            return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
-        }
-
-        // Gather all existing events.
-        let mut events = Vec::new();
-        for key in &originals {
-            let bytes = ops::get(&self.store, key).await?;
-            for line in bytes.split(|b| *b == b'\n') {
-                if !line.is_empty() {
-                    events.push(line.to_vec());
-                }
-            }
-        }
-
-        let total_events = events.len();
-        if total_events == 0 {
-            // Clean up empty files without writing a new one.
-            for key in &originals {
-                let _ = ops::delete(&self.store, key).await;
-            }
-            return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
-        }
-
-        // Name the compacted file with the SAME `batch_{ts}` format so it
-        // sorts chronologically with future batches. Using a distinct prefix
-        // ("batch_compacted_") would break lex ordering: later `batch_N`
-        // files would sort BEFORE the compacted file because 'c' > digits.
-        // Timestamp = now, so any appends arriving during compaction (which
-        // get the current wall-clock time) sort AFTER this file.
-        let ts_us = Utc::now().timestamp_micros().max(0) as u128;
-        let new_key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
-
-        let mut body = Vec::new();
-        for line in &events {
-            body.extend_from_slice(line);
-            body.push(b'\n');
-        }
-        ops::put(&self.store, &new_key, Bytes::from(body)).await?;
-
-        // Only delete originals once the consolidated file is persisted.
-        let mut failures = 0;
-        for key in &originals {
-            if ops::delete(&self.store, key).await.is_err() {
-                failures += 1;
-            }
-        }
-        if failures > 0 {
-            tracing::warn!(
-                "compact '{}': {} original files failed to delete — consolidated file {} has the data",
-                self.prefix, failures, new_key,
-            );
-        }
-
-        Ok(CompactStats {
-            merged_files: originals.len(),
-            events: total_events,
-            new_key: Some(new_key),
-        })
-    }
-
-    /// How many batch files exist on disk right now.
-    pub async fn file_count(&self) -> Result<usize, String> {
-        Ok(self.list_batch_keys().await?.len())
-    }
-
-    async fn list_batch_keys(&self) -> Result<Vec<String>, String> {
-        let prefix_with_slash = format!("{}/", self.prefix);
-        // list the prefix then filter for keys that match our naming scheme;
-        // unrelated files at the same prefix won't be touched.
-        let raw = ops::list(&self.store, Some(&prefix_with_slash)).await?;
-        Ok(raw
-            .into_iter()
-            .filter(|k| {
-                let basename = k.rsplit('/').next().unwrap_or(k);
-                basename.starts_with("batch_") && basename.ends_with(".jsonl")
-            })
-            .collect())
-    }
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct CompactStats {
-    pub merged_files: usize,
-    pub events: usize,
-    pub new_key: Option<String>,
-}
-
-// Log unflushed-buffer size on drop. We can't `.await` from a sync `Drop`,
-// so a real flush isn't possible here — callers are responsible for calling
-// `.flush()` before dropping if durability matters. These journals are
-// observability hints; a few lost buffered events at shutdown are
-// acceptable per ADR-018.
-impl Drop for AppendLog {
-    fn drop(&mut self) {
-        let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
-        if buf_len == 0 { return; }
-        tracing::debug!(
-            "append_log '{}' dropping with {} unflushed events",
-            self.prefix, buf_len,
-        );
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use object_store::memory::InMemory;
-
-    fn mk(threshold: usize) -> AppendLog {
-        AppendLog::new(Arc::new(InMemory::new()), "prefix")
-            .with_flush_threshold(threshold)
-    }
-
-    #[tokio::test]
-    async fn append_stays_buffered_below_threshold() {
-        let log = mk(5);
-        log.append(b"one".to_vec()).await.unwrap();
-        log.append(b"two".to_vec()).await.unwrap();
-        assert_eq!(log.file_count().await.unwrap(), 0, "no files until threshold");
-        let all = log.read_all().await.unwrap();
-        assert_eq!(all, vec![b"one".to_vec(), b"two".to_vec()],
-            "read_all surfaces unflushed buffer");
-    }
-
-    #[tokio::test]
-    async fn append_auto_flushes_on_threshold() {
-        let log = mk(3);
-        for i in 0..3 {
-            log.append(format!("evt{i}").into_bytes()).await.unwrap();
-        }
-        assert_eq!(log.file_count().await.unwrap(), 1, "threshold triggered one flush");
-
-        // A fourth append stays buffered until the next threshold.
-        log.append(b"evt3".to_vec()).await.unwrap();
-        assert_eq!(log.file_count().await.unwrap(), 1, "below threshold again");
-    }
-
-    #[tokio::test]
-    async fn flush_empty_is_noop() {
-        let log = mk(32);
-        log.flush().await.unwrap();
-        log.flush().await.unwrap();
-        assert_eq!(log.file_count().await.unwrap(), 0);
-    }
-
-    #[tokio::test]
-    async fn read_all_orders_events_across_flushes() {
-        let log = mk(1); // flush-on-every-append
-        for i in 0..5 {
-            log.append(format!("e{i}").into_bytes()).await.unwrap();
-            // Spread writes out so timestamps sort strictly.
-            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
-        }
-        let all = log.read_all().await.unwrap();
-        let strs: Vec<String> = all.into_iter()
-            .map(|v| String::from_utf8(v).unwrap())
-            .collect();
-        assert_eq!(strs, vec!["e0", "e1", "e2", "e3", "e4"],
-            "lex sort of batch keys == chronological event order");
-    }
-
-    #[tokio::test]
-    async fn compact_merges_multiple_files_into_one() {
-        let log = mk(1); // force file-per-append
-        for i in 0..4 {
-            log.append(format!("e{i}").into_bytes()).await.unwrap();
-            tokio::time::sleep(std::time::Duration::from_millis(2)).await;
-        }
-        assert_eq!(log.file_count().await.unwrap(), 4);
-
-        let stats = log.compact().await.unwrap();
-        assert_eq!(stats.merged_files, 4);
-        assert_eq!(stats.events, 4);
-        assert!(stats.new_key.is_some());
-
-        assert_eq!(log.file_count().await.unwrap(), 1, "originals deleted, 1 survivor");
-        let all = log.read_all().await.unwrap();
-        assert_eq!(all.len(), 4, "no events lost in compaction");
-    }
-
-    #[tokio::test]
-    async fn compact_with_single_file_is_noop() {
-        let log = mk(1);
-        log.append(b"only".to_vec()).await.unwrap();
-        assert_eq!(log.file_count().await.unwrap(), 1);
-
-        let stats = log.compact().await.unwrap();
-        assert_eq!(stats.merged_files, 1);
-        assert_eq!(stats.events, 0, "nothing to consolidate");
-        assert!(stats.new_key.is_none(), "no new file written");
-        assert_eq!(log.file_count().await.unwrap(), 1, "original untouched");
-    }
-}
-

crates/storaged/src/error_journal.rs

@@ -1,289 +0,0 @@
-/// Bucket operation error journal.
-///
-/// Every bucket op failure and every rescue fallback lands here. Goal:
-/// answering "is anything broken?" with one HTTP call.
-///
-/// Storage: batched write-once files under `_errors/bucket_errors/` in the
-/// primary bucket. Uses the shared `AppendLog` helper so we never rewrite
-/// existing files — see `append_log.rs` for the full pattern rationale.
-///
-/// In-memory ring buffer holds the last N events for fast response; on
-/// startup `load_recent` hydrates it from all batch files.
-
-use chrono::{DateTime, Utc};
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::collections::VecDeque;
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use crate::append_log::{AppendLog, CompactStats};
-
-const JOURNAL_PREFIX: &str = "_errors/bucket_errors";
-const RING_CAPACITY: usize = 2000;
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-pub enum BucketOp {
-    Read,
-    Write,
-    Delete,
-    List,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct BucketErrorEvent {
-    pub ts: DateTime<Utc>,
-    pub op: BucketOp,
-    pub target: String,
-    pub key: String,
-    pub error: String,
-    #[serde(default)]
-    pub rescued: bool,
-}
-
-impl BucketErrorEvent {
-    pub fn new_read(target: &str, key: &str, error: &str) -> Self {
-        Self { ts: Utc::now(), op: BucketOp::Read, target: target.into(), key: key.into(), error: error.into(), rescued: false }
-    }
-    pub fn new_write(target: &str, key: &str, error: &str) -> Self {
-        Self { ts: Utc::now(), op: BucketOp::Write, target: target.into(), key: key.into(), error: error.into(), rescued: false }
-    }
-    pub fn new_delete(target: &str, key: &str, error: &str) -> Self {
-        Self { ts: Utc::now(), op: BucketOp::Delete, target: target.into(), key: key.into(), error: error.into(), rescued: false }
-    }
-    pub fn new_list(target: &str, prefix: &str, error: &str) -> Self {
-        Self { ts: Utc::now(), op: BucketOp::List, target: target.into(), key: prefix.into(), error: error.into(), rescued: false }
-    }
-}
-
-#[derive(Clone)]
-pub struct ErrorJournal {
-    log: Arc<AppendLog>,
-    ring: Arc<RwLock<VecDeque<BucketErrorEvent>>>,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct HealthReport {
-    pub period_minutes: i64,
-    pub total_errors: usize,
-    pub per_bucket: std::collections::HashMap<String, usize>,
-    pub unhealthy_buckets: Vec<String>,
-}
-
-impl ErrorJournal {
-    pub fn new(store: Arc<dyn ObjectStore>) -> Self {
-        // Keep flush threshold lowish — operators checking /storage/errors
-        // after a recent incident want to see fresh rows even without an
-        // explicit flush.
-        let log = Arc::new(
-            AppendLog::new(store, JOURNAL_PREFIX).with_flush_threshold(8),
-        );
-        Self {
-            log,
-            ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
-        }
-    }
-
-    /// Hydrate the ring buffer from existing batch files. Call once at
-    /// startup. Tolerates malformed lines (skipped with a warning) and
-    /// missing files (returns 0).
-    pub async fn load_recent(&self) -> Result<usize, String> {
-        let lines = self.log.read_all().await.unwrap_or_default();
-        let mut ring = self.ring.write().await;
-        for line in lines {
-            match serde_json::from_slice::<BucketErrorEvent>(&line) {
-                Ok(ev) => {
-                    if ring.len() >= RING_CAPACITY { ring.pop_front(); }
-                    ring.push_back(ev);
-                }
-                Err(e) => tracing::warn!("error journal: skip malformed line ({e})"),
-            }
-        }
-        Ok(ring.len())
-    }
-
-    /// Append an event. In-memory ring updated immediately; persistence
-    /// happens in batches via the underlying AppendLog.
-    pub async fn append(&self, event: BucketErrorEvent) {
-        {
-            let mut ring = self.ring.write().await;
-            if ring.len() >= RING_CAPACITY { ring.pop_front(); }
-            ring.push_back(event.clone());
-        }
-
-        match serde_json::to_vec(&event) {
-            Ok(line) => {
-                if let Err(e) = self.log.append(line).await {
-                    tracing::error!("error journal persist failed: {e}");
-                }
-            }
-            Err(e) => tracing::error!("error journal serialize failed: {e}"),
-        }
-    }
-
-    /// Mark the most recent matching in-memory event as rescued.
-    /// Only updates the ring buffer — the JSONL line already persisted
-    /// records the failure fact; rescue status travels in response headers.
-    pub async fn mark_rescued_last(&self, target: &str, key: &str) {
-        let mut ring = self.ring.write().await;
-        for ev in ring.iter_mut().rev() {
-            if matches!(ev.op, BucketOp::Read) && ev.target == target && ev.key == key && !ev.rescued {
-                ev.rescued = true;
-                break;
-            }
-        }
-    }
-
-    pub async fn recent(&self, limit: usize) -> Vec<BucketErrorEvent> {
-        let ring = self.ring.read().await;
-        let start = ring.len().saturating_sub(limit);
-        ring.iter().skip(start).cloned().collect()
-    }
-
-    pub async fn filter(
-        &self,
-        bucket: Option<&str>,
-        since: Option<DateTime<Utc>>,
-        limit: usize,
-    ) -> Vec<BucketErrorEvent> {
-        let ring = self.ring.read().await;
-        ring.iter()
-            .rev()
-            .filter(|ev| bucket.map_or(true, |b| ev.target == b))
-            .filter(|ev| since.map_or(true, |s| ev.ts >= s))
-            .take(limit)
-            .cloned()
-            .collect::<Vec<_>>()
-            .into_iter()
-            .rev()
-            .collect()
-    }
-
-    /// Summarize errors in the last N minutes.
-    pub async fn health(&self, period_minutes: i64) -> HealthReport {
-        use std::collections::HashMap;
-        let cutoff = Utc::now() - chrono::Duration::minutes(period_minutes);
-        let ring = self.ring.read().await;
-        let recent: Vec<_> = ring.iter().filter(|ev| ev.ts >= cutoff).collect();
-        let mut per_bucket: HashMap<String, usize> = HashMap::new();
-        for ev in &recent {
-            *per_bucket.entry(ev.target.clone()).or_insert(0) += 1;
-        }
-        let unhealthy_buckets: Vec<String> = per_bucket
-            .iter()
-            .filter(|(_, c)| **c >= 3)
-            .map(|(k, _)| k.clone())
-            .collect();
-        HealthReport {
-            period_minutes,
-            total_errors: recent.len(),
-            per_bucket,
-            unhealthy_buckets,
-        }
-    }
-
-    /// Force an immediate flush of buffered events to object storage.
-    pub async fn flush(&self) -> Result<(), String> {
-        self.log.flush().await
-    }
-
-    /// Consolidate all batch files into one. Operator cleanup.
-    pub async fn compact(&self) -> Result<CompactStats, String> {
-        self.log.compact().await
-    }
-
-    /// How many JSONL batch files currently exist.
-    pub async fn file_count(&self) -> Result<usize, String> {
-        self.log.file_count().await
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use object_store::memory::InMemory;
-
-    fn mk() -> ErrorJournal {
-        ErrorJournal::new(Arc::new(InMemory::new()))
-    }
-
-    #[tokio::test]
-    async fn append_updates_ring_immediately() {
-        let j = mk();
-        j.append(BucketErrorEvent::new_read("tenant_a", "k1", "timeout")).await;
-        j.append(BucketErrorEvent::new_write("tenant_b", "k2", "denied")).await;
-
-        let recent = j.recent(10).await;
-        assert_eq!(recent.len(), 2);
-        assert_eq!(recent[0].target, "tenant_a");
-        assert_eq!(recent[1].target, "tenant_b");
-    }
-
-    #[tokio::test]
-    async fn load_recent_rehydrates_ring_from_disk() {
-        // Use a shared InMemory store so the second journal reads the first
-        // journal's writes — mirrors gateway restart semantics.
-        let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
-        {
-            let j1 = ErrorJournal {
-                log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX).with_flush_threshold(1)),
-                ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
-            };
-            j1.append(BucketErrorEvent::new_read("t", "k", "err")).await;
-            j1.flush().await.unwrap();
-        }
-
-        let j2 = ErrorJournal {
-            log: Arc::new(AppendLog::new(store.clone(), JOURNAL_PREFIX)),
-            ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
-        };
-        assert_eq!(j2.recent(10).await.len(), 0, "fresh ring is empty");
-
-        let loaded = j2.load_recent().await.unwrap();
-        assert_eq!(loaded, 1);
-        assert_eq!(j2.recent(10).await[0].target, "t");
-    }
-
-    #[tokio::test]
-    async fn mark_rescued_updates_most_recent_match() {
-        let j = mk();
-        j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
-        j.append(BucketErrorEvent::new_read("t2", "k", "err")).await;
-        j.append(BucketErrorEvent::new_read("t1", "k", "err")).await;
-
-        j.mark_rescued_last("t1", "k").await;
-
-        let recent = j.recent(10).await;
-        assert!(!recent[0].rescued, "older t1 event not touched");
-        assert!(!recent[1].rescued, "t2 not touched");
-        assert!(recent[2].rescued, "most recent t1 was rescued");
-    }
-
-    #[tokio::test]
-    async fn health_flags_buckets_with_three_or_more_errors() {
-        let j = mk();
-        for _ in 0..3 { j.append(BucketErrorEvent::new_read("broken", "k", "e")).await; }
-        j.append(BucketErrorEvent::new_read("flaky", "k", "e")).await;
-
-        let h = j.health(60).await;
-        assert_eq!(h.total_errors, 4);
-        assert_eq!(h.per_bucket.get("broken"), Some(&3));
-        assert_eq!(h.per_bucket.get("flaky"), Some(&1));
-        assert!(h.unhealthy_buckets.contains(&"broken".to_string()));
-        assert!(!h.unhealthy_buckets.contains(&"flaky".to_string()));
-    }
-
-    #[tokio::test]
-    async fn filter_by_bucket_returns_chronological_order() {
-        let j = mk();
-        j.append(BucketErrorEvent::new_read("a", "k1", "e")).await;
-        j.append(BucketErrorEvent::new_read("b", "k2", "e")).await;
-        j.append(BucketErrorEvent::new_read("a", "k3", "e")).await;
-
-        let only_a = j.filter(Some("a"), None, 10).await;
-        assert_eq!(only_a.len(), 2);
-        assert_eq!(only_a[0].key, "k1", "oldest first");
-        assert_eq!(only_a[1].key, "k3");
-    }
-}

crates/storaged/src/federation_service.rs

@@ -1,179 +0,0 @@
-/// HTTP surface for federation operator endpoints.
-///
-/// - `GET /buckets` — configured buckets with reachability status
-/// - `GET /errors` — recent bucket op failures (filterable)
-/// - `GET /bucket-health` — aggregated errors-per-bucket in the last 5 minutes
-///
-/// Mounted under `/storage` by the gateway. `/storage/health` is already
-/// claimed by the existing storage router for service liveness, so the
-/// aggregated bucket health lives at `/storage/bucket-health`.
-
-use axum::{
-    Json, Router,
-    extract::{Path, Query, State},
-    http::StatusCode,
-    response::IntoResponse,
-    routing::{delete, get, post, put},
-};
-use chrono::{DateTime, Utc};
-use serde::Deserialize;
-use shared::config::BucketConfig;
-use std::sync::Arc;
-
-use crate::error_journal::BucketErrorEvent;
-use crate::registry::{BucketInfo, BucketRegistry};
-
-pub fn router(registry: Arc<BucketRegistry>) -> Router {
-    Router::new()
-        .route("/buckets", get(list_buckets).post(add_bucket))
-        .route("/buckets/{name}", delete(remove_bucket))
-        .route("/errors", get(list_errors))
-        .route("/errors/flush", post(flush_errors))
-        .route("/errors/compact", post(compact_errors))
-        .route("/bucket-health", get(get_health))
-        .route("/buckets/{bucket}/objects/{*key}", put(put_bucket_object))
-        .route("/buckets/{bucket}/objects/{*key}", get(get_bucket_object))
-        .with_state(registry)
-}
-
-/// Provision + register a bucket at runtime. Body is a `BucketConfig`.
-///
-/// Federation layer 2: profile buckets (`profile:alice`) and tenant
-/// buckets can be added after startup without a service restart.
-async fn add_bucket(
-    State(reg): State<Arc<BucketRegistry>>,
-    Json(bc): Json<BucketConfig>,
-) -> impl IntoResponse {
-    // Safety net: local backends must land somewhere under the configured
-    // profile_root (for profile:*) or be otherwise explicitly rooted.
-    // Refuse empty / missing root to avoid "bucket created at ./" surprises.
-    if bc.backend == "local" {
-        match bc.root.as_deref() {
-            Some(r) if !r.trim().is_empty() => {}
-            _ => return Err((StatusCode::BAD_REQUEST,
-                "local bucket requires a non-empty 'root' path".to_string())),
-        }
-    }
-    match reg.add_bucket(bc).await {
-        Ok(info) => Ok((StatusCode::CREATED, Json(info))),
-        Err(e) => {
-            let code = if e.contains("already registered") {
-                StatusCode::CONFLICT
-            } else {
-                StatusCode::BAD_REQUEST
-            };
-            Err((code, e))
-        }
-    }
-}
-
-/// Unregister a bucket. Refused for primary / rescue / unknown names.
-async fn remove_bucket(
-    State(reg): State<Arc<BucketRegistry>>,
-    Path(name): Path<String>,
-) -> impl IntoResponse {
-    match reg.remove_bucket(&name) {
-        Ok(()) => Ok((StatusCode::OK, format!("unregistered: {name}"))),
-        Err(e) => {
-            let code = if e.contains("cannot remove") {
-                StatusCode::FORBIDDEN
-            } else if e.contains("not registered") {
-                StatusCode::NOT_FOUND
-            } else {
-                StatusCode::BAD_REQUEST
-            };
-            Err((code, e))
-        }
-    }
-}
-
-async fn list_buckets(State(reg): State<Arc<BucketRegistry>>) -> Json<Vec<BucketInfo>> {
-    Json(reg.list().await)
-}
-
-#[derive(Deserialize)]
-struct ErrorQuery {
-    bucket: Option<String>,
-    since: Option<DateTime<Utc>>,
-    #[serde(default = "default_limit")]
-    limit: usize,
-}
-
-fn default_limit() -> usize { 50 }
-
-async fn list_errors(
-    State(reg): State<Arc<BucketRegistry>>,
-    Query(q): Query<ErrorQuery>,
-) -> Json<Vec<BucketErrorEvent>> {
-    Json(reg.journal().filter(q.bucket.as_deref(), q.since, q.limit).await)
-}
-
-#[derive(Deserialize)]
-struct HealthQuery {
-    #[serde(default = "default_period")]
-    minutes: i64,
-}
-
-fn default_period() -> i64 { 5 }
-
-async fn get_health(
-    State(reg): State<Arc<BucketRegistry>>,
-    Query(q): Query<HealthQuery>,
-) -> impl IntoResponse {
-    Json(reg.journal().health(q.minutes).await)
-}
-
-/// Bucket-aware write. Hard-fails if the target bucket is unreachable;
-/// never falls back to rescue for writes.
-async fn put_bucket_object(
-    State(reg): State<Arc<BucketRegistry>>,
-    Path((bucket, key)): Path<(String, String)>,
-    body: bytes::Bytes,
-) -> impl IntoResponse {
-    match reg.write_smart(&bucket, &key, body).await {
-        Ok(()) => (StatusCode::CREATED, format!("stored: {bucket}/{key}")).into_response(),
-        Err(e) => (StatusCode::SERVICE_UNAVAILABLE, e).into_response(),
-    }
-}
-
-/// Bucket-aware read. Falls through to the rescue bucket if the target
-/// is unreachable or the key is missing; emits observability headers so
-/// callers can detect the fallback.
-async fn flush_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
-    match reg.journal().flush().await {
-        Ok(()) => (StatusCode::OK, "flushed").into_response(),
-        Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
-    }
-}
-
-async fn compact_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
-    match reg.journal().compact().await {
-        Ok(stats) => Json(stats).into_response(),
-        Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
-    }
-}
-
-async fn get_bucket_object(
-    State(reg): State<Arc<BucketRegistry>>,
-    Path((bucket, key)): Path<(String, String)>,
-) -> impl IntoResponse {
-    match reg.read_smart(&bucket, &key).await {
-        Ok(outcome) => {
-            let mut resp = outcome.data.into_response();
-            if outcome.rescued {
-                let headers = resp.headers_mut();
-                headers.insert("x-lakehouse-rescue-used", "true".parse().unwrap());
-                headers.insert(
-                    "x-lakehouse-original-bucket",
-                    outcome.original_bucket.parse().unwrap(),
-                );
-                headers.insert(
-                    "x-lakehouse-served-by",
-                    outcome.served_by.parse().unwrap(),
-                );
-            }
-            resp
-        }
-        Err(e) => (StatusCode::NOT_FOUND, e).into_response(),
-    }
-}

crates/storaged/src/lib.rs

@@ -1,7 +1,3 @@
-pub mod append_log;
 pub mod backend;
-pub mod error_journal;
-pub mod federation_service;
 pub mod ops;
-pub mod registry;
 pub mod service;

crates/storaged/src/registry.rs

@@ -1,377 +0,0 @@
-/// Multi-backend bucket registry — the federation foundation.
-///
-/// Federation rule: every `ObjectRef` belongs to exactly one named bucket.
-/// The registry resolves bucket names to `object_store` backends, handles
-/// rescue-bucket fallback on read failure, writes every failure to the
-/// error journal, and exposes a health summary for operators.
-///
-/// Existing call sites can keep using `ops::*` with `registry.get(name)`.
-/// New bucket-aware call sites use `registry.read_smart` / `write_smart`
-/// which handle fallback + journaling automatically.
-
-use object_store::ObjectStore;
-use object_store::local::LocalFileSystem;
-use serde::Serialize;
-use shared::config::{BucketConfig, StorageConfig};
-use shared::secrets::{BucketCredentials, SecretsProvider};
-use std::collections::HashMap;
-use std::sync::{Arc, RwLock};
-
-use crate::error_journal::{BucketErrorEvent, ErrorJournal};
-
-/// A registered bucket — the store handle + its configuration.
-pub struct BucketEntry {
-    pub name: String,
-    pub backend: String,
-    pub store: Arc<dyn ObjectStore>,
-    pub config: BucketConfig,
-}
-
-/// Read outcome — may have been rescued.
-#[derive(Debug, Clone)]
-pub struct ReadOutcome {
-    pub data: bytes::Bytes,
-    pub rescued: bool,
-    pub original_bucket: String,
-    pub served_by: String,
-}
-
-/// Summary entry for GET /storage/buckets.
-#[derive(Debug, Clone, Serialize)]
-pub struct BucketInfo {
-    pub name: String,
-    pub backend: String,
-    pub reachable: bool,
-    pub role: BucketRole,
-}
-
-#[derive(Debug, Clone, Serialize, PartialEq)]
-#[serde(rename_all = "lowercase")]
-pub enum BucketRole {
-    Primary,
-    Rescue,
-    Profile,
-    Tenant,
-}
-
-pub struct BucketRegistry {
-    /// RwLock because federation layer 2 adds runtime bucket lifecycle
-    /// (`POST /storage/buckets` / `DELETE /storage/buckets/{name}`).
-    /// Almost all accesses are reads — writes happen at provision /
-    /// deactivate time only.
-    buckets: RwLock<HashMap<String, Arc<BucketEntry>>>,
-    default: String,
-    rescue: Option<String>,
-    profile_root: String,
-    /// Held so runtime `add_bucket` can resolve secret_ref handles.
-    secrets: Arc<dyn SecretsProvider>,
-    journal: ErrorJournal,
-}
-
-impl BucketRegistry {
-    /// Build the registry from storage config + secrets provider.
-    /// Back-compat: if `buckets` is empty, synthesize a `primary` bucket from
-    /// the legacy `root` field so pre-federation configs keep working.
-    pub async fn from_config(
-        cfg: &StorageConfig,
-        secrets: Arc<dyn SecretsProvider>,
-    ) -> Result<Self, String> {
-        let mut buckets: HashMap<String, Arc<BucketEntry>> = HashMap::new();
-
-        let bucket_configs: Vec<BucketConfig> = if cfg.buckets.is_empty() {
-            vec![BucketConfig {
-                name: "primary".to_string(),
-                backend: "local".to_string(),
-                root: Some(cfg.root.clone()),
-                bucket: None,
-                region: None,
-                endpoint: None,
-                secret_ref: None,
-            }]
-        } else {
-            cfg.buckets.clone()
-        };
-
-        for bc in bucket_configs {
-            let store = build_store(&bc, secrets.as_ref()).await?;
-            let entry = Arc::new(BucketEntry {
-                name: bc.name.clone(),
-                backend: bc.backend.clone(),
-                store,
-                config: bc.clone(),
-            });
-            buckets.insert(bc.name.clone(), entry);
-        }
-
-        // Ensure `primary` always exists — it's where error journals live.
-        if !buckets.contains_key("primary") {
-            return Err("no bucket named 'primary' configured — required as error-journal home".into());
-        }
-
-        // Rescue bucket is optional but, if named, must exist.
-        if let Some(r) = &cfg.rescue_bucket {
-            if !buckets.contains_key(r) {
-                return Err(format!("rescue_bucket '{r}' not found among configured buckets"));
-            }
-        }
-
-        let journal = ErrorJournal::new(buckets.get("primary").unwrap().store.clone());
-        let _ = journal.load_recent().await;
-
-        Ok(Self {
-            buckets: RwLock::new(buckets),
-            default: "primary".to_string(),
-            rescue: cfg.rescue_bucket.clone(),
-            profile_root: cfg.profile_root.clone(),
-            secrets,
-            journal,
-        })
-    }
-
-    pub fn default_name(&self) -> &str { &self.default }
-
-    pub fn rescue_name(&self) -> Option<&str> { self.rescue.as_deref() }
-
-    pub fn profile_root(&self) -> &str { &self.profile_root }
-
-    pub fn journal(&self) -> &ErrorJournal { &self.journal }
-
-    /// Resolve a bucket name to its object store. Existing call sites use
-    /// this as a drop-in replacement for the old single-store pattern.
-    ///
-    /// Uses `std::sync::RwLock` — every caller holds the guard for
-    /// microseconds (clones an Arc and drops), so there's no async-
-    /// blocking concern. Never hold across an await.
-    pub fn get(&self, bucket: &str) -> Result<Arc<dyn ObjectStore>, String> {
-        let guard = self.buckets.read().expect("bucket registry lock poisoned");
-        guard
-            .get(bucket)
-            .map(|e| e.store.clone())
-            .ok_or_else(|| format!("unknown bucket: {bucket}"))
-    }
-
-    /// The default bucket's store — use for code paths that don't yet know
-    /// about buckets.
-    pub fn default_store(&self) -> Arc<dyn ObjectStore> {
-        let guard = self.buckets.read().expect("bucket registry lock poisoned");
-        guard.get(&self.default).unwrap().store.clone()
-    }
-
-    /// True if this bucket name is registered.
-    pub fn contains(&self, bucket: &str) -> bool {
-        let guard = self.buckets.read().expect("bucket registry lock poisoned");
-        guard.contains_key(bucket)
-    }
-
-    /// List all registered buckets. Checks reachability by doing a trivial
-    /// `list` with limit 1 on each.
-    pub async fn list(&self) -> Vec<BucketInfo> {
-        // Snapshot (name, backend, store_clone) under the lock, then probe
-        // outside — probing can be slow (network), don't hold the lock.
-        let snapshot: Vec<(String, String, Arc<dyn ObjectStore>)> = {
-            let guard = self.buckets.read().expect("bucket registry lock poisoned");
-            guard
-                .iter()
-                .map(|(n, e)| (n.clone(), e.backend.clone(), e.store.clone()))
-                .collect()
-        };
-
-        let mut out = Vec::with_capacity(snapshot.len());
-        for (name, backend, store) in snapshot {
-            let reachable = probe(&store).await;
-            let role = self.classify(&name);
-            out.push(BucketInfo { name, backend, reachable, role });
-        }
-        out.sort_by(|a, b| a.name.cmp(&b.name));
-        out
-    }
-
-    fn classify(&self, name: &str) -> BucketRole {
-        if name == self.default { BucketRole::Primary }
-        else if Some(name) == self.rescue.as_deref() { BucketRole::Rescue }
-        else if name.starts_with("profile:") { BucketRole::Profile }
-        else { BucketRole::Tenant }
-    }
-
-    /// Provision + register a bucket at runtime (federation layer 2).
-    /// Returns Err if the name is already registered. Local backends get
-    /// their root directory created automatically.
-    pub async fn add_bucket(&self, bc: BucketConfig) -> Result<BucketInfo, String> {
-        if self.contains(&bc.name) {
-            return Err(format!("bucket '{}' already registered", bc.name));
-        }
-        let store = build_store(&bc, self.secrets.as_ref()).await?;
-        let reachable = probe(&store).await;
-        let entry = Arc::new(BucketEntry {
-            name: bc.name.clone(),
-            backend: bc.backend.clone(),
-            store,
-            config: bc.clone(),
-        });
-        let role = self.classify(&bc.name);
-        let info = BucketInfo {
-            name: bc.name.clone(),
-            backend: bc.backend.clone(),
-            reachable,
-            role,
-        };
-        {
-            let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
-            guard.insert(bc.name.clone(), entry);
-        }
-        tracing::info!("registered bucket '{}' backend={} reachable={}",
-            bc.name, bc.backend, reachable);
-        Ok(info)
-    }
-
-    /// Unregister a bucket. Refuses to remove `primary` or the configured
-    /// rescue bucket — those are load-bearing. Caller is responsible for
-    /// any final flush of bucket-local state before calling this.
-    pub fn remove_bucket(&self, name: &str) -> Result<(), String> {
-        if name == self.default {
-            return Err("cannot remove primary bucket".into());
-        }
-        if Some(name) == self.rescue.as_deref() {
-            return Err(format!("cannot remove rescue bucket '{name}'"));
-        }
-        let removed = {
-            let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
-            guard.remove(name)
-        };
-        if removed.is_none() {
-            return Err(format!("bucket '{name}' not registered"));
-        }
-        tracing::info!("unregistered bucket '{}'", name);
-        Ok(())
-    }
-
-    /// Read with rescue-bucket fallback. If the target bucket fails and a
-    /// rescue is configured, retries against rescue. Records every failure
-    /// in the error journal.
-    pub async fn read_smart(&self, bucket: &str, key: &str) -> Result<ReadOutcome, String> {
-        let target_store = {
-            let guard = self.buckets.read().expect("bucket registry lock poisoned");
-            guard.get(bucket).map(|e| e.store.clone())
-                .ok_or_else(|| format!("unknown bucket: {bucket}"))?
-        };
-
-        match crate::ops::get(&target_store, key).await {
-            Ok(data) => Ok(ReadOutcome {
-                data, rescued: false,
-                original_bucket: bucket.to_string(),
-                served_by: bucket.to_string(),
-            }),
-            Err(err) => {
-                // Record failure regardless of what happens next.
-                self.journal.append(BucketErrorEvent::new_read(bucket, key, &err)).await;
-
-                // Try rescue, if any.
-                if let Some(rescue_name) = &self.rescue {
-                    if rescue_name != bucket {
-                        let rescue_store = {
-                            let guard = self.buckets.read().expect("bucket registry lock poisoned");
-                            guard.get(rescue_name).map(|e| e.store.clone())
-                        };
-                        if let Some(rescue_store) = rescue_store {
-                            match crate::ops::get(&rescue_store, key).await {
-                                Ok(data) => {
-                                    self.journal.mark_rescued_last(bucket, key).await;
-                                    return Ok(ReadOutcome {
-                                        data, rescued: true,
-                                        original_bucket: bucket.to_string(),
-                                        served_by: rescue_name.clone(),
-                                    });
-                                }
-                                Err(rescue_err) => {
-                                    return Err(format!(
-                                        "read '{key}' failed in '{bucket}' ({err}); rescue '{rescue_name}' also failed ({rescue_err})"
-                                    ));
-                                }
-                            }
-                        }
-                    }
-                }
-                Err(format!("read '{key}' failed in '{bucket}': {err}"))
-            }
-        }
-    }
-
-    /// Write always goes to target. No rescue fallback for writes — writes
-    /// that silently vanish are the worst possible failure.
-    pub async fn write_smart(
-        &self,
-        bucket: &str,
-        key: &str,
-        data: bytes::Bytes,
-    ) -> Result<(), String> {
-        let target_store = {
-            let guard = self.buckets.read().expect("bucket registry lock poisoned");
-            guard.get(bucket).map(|e| e.store.clone())
-                .ok_or_else(|| format!("unknown bucket: {bucket}"))?
-        };
-        match crate::ops::put(&target_store, key, data).await {
-            Ok(()) => Ok(()),
-            Err(err) => {
-                self.journal.append(BucketErrorEvent::new_write(bucket, key, &err)).await;
-                Err(format!("write '{key}' failed in '{bucket}': {err}"))
-            }
-        }
-    }
-}
-
-/// Trivial reachability check — try to list with limit 0.
-async fn probe(store: &Arc<dyn ObjectStore>) -> bool {
-    use futures::StreamExt;
-    let mut stream = store.list(None);
-    // Pulling the first item confirms the store responds. Empty bucket = ok.
-    match stream.next().await {
-        Some(Ok(_)) => true,
-        None => true,          // empty but reachable
-        Some(Err(_)) => false,
-    }
-}
-
-/// Build a concrete ObjectStore from a BucketConfig.
-async fn build_store(
-    bc: &BucketConfig,
-    secrets: &dyn SecretsProvider,
-) -> Result<Arc<dyn ObjectStore>, String> {
-    match bc.backend.as_str() {
-        "local" => {
-            let root = bc.root.as_deref()
-                .ok_or_else(|| format!("bucket '{}' is backend=local but has no root", bc.name))?;
-            std::fs::create_dir_all(root)
-                .map_err(|e| format!("create bucket dir '{root}': {e}"))?;
-            let fs = LocalFileSystem::new_with_prefix(root)
-                .map_err(|e| format!("init local bucket '{}': {e}", bc.name))?;
-            Ok(Arc::new(fs))
-        }
-        "s3" => {
-            let handle = bc.secret_ref.as_deref()
-                .ok_or_else(|| format!("s3 bucket '{}' has no secret_ref", bc.name))?;
-            let creds: BucketCredentials = secrets.resolve(handle).await?;
-            let s3_bucket = bc.bucket.as_deref()
-                .ok_or_else(|| format!("s3 bucket '{}' has no `bucket` name", bc.name))?;
-            let region = bc.region.as_deref().unwrap_or("us-east-1");
-
-            let mut builder = object_store::aws::AmazonS3Builder::new()
-                .with_bucket_name(s3_bucket)
-                .with_region(region)
-                .with_access_key_id(&creds.access_key)
-                .with_secret_access_key(&creds.secret_key);
-            if let Some(endpoint) = &bc.endpoint {
-                builder = builder.with_endpoint(endpoint);
-                // MinIO and other S3-compatible services often run on plain
-                // HTTP. object_store refuses HTTP by default — opt in when
-                // a custom endpoint is configured (TLS endpoints work either way).
-                if endpoint.starts_with("http://") {
-                    builder = builder.with_allow_http(true);
-                }
-            }
-            let s3 = builder.build()
-                .map_err(|e| format!("init s3 bucket '{}': {e}", bc.name))?;
-            Ok(Arc::new(s3))
-        }
-        other => Err(format!("unknown backend '{other}' for bucket '{}'", bc.name)),
-    }
-}

crates/ui/assets/style.css

@@ -205,13 +205,3 @@ tr:hover td { background: var(--accent-glow); }
     padding: 8px 12px; border-bottom: 1px solid var(--border); font-size: 13px;
 }
 .table-item:hover { background: var(--accent-glow); }
-
-/* Phase 19 — Playbook panel */
-.boosted-row { background: rgba(120, 200, 120, 0.10); }
-.boosted-row td { border-top: 1px solid rgba(120, 200, 120, 0.30); }
-.mono-cell {
-    font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
-    font-size: 11px; color: var(--text-dim);
-    max-width: 220px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
-}
-.panel-section + .panel-section { margin-top: 18px; }

crates/ui/src/main.rs

@@ -178,116 +178,9 @@ enum Tab {
     Explore,
     Sql,
     Ingest,
-    Playbook,
     Status,
 }
 
-// --- Playbook memory types (Phase 19) ---
-
-#[derive(Debug, Clone, Deserialize, PartialEq)]
-struct PlaybookStats {
-    entries: usize,
-    entries_with_embeddings: usize,
-    #[serde(default)]
-    total_names_endorsed: usize,
-    #[serde(default)]
-    sample: Vec<PlaybookSample>,
-}
-
-#[derive(Debug, Clone, Deserialize, PartialEq)]
-struct PlaybookSample {
-    id: String,
-    operation: String,
-    #[serde(default)]
-    city: Option<String>,
-    #[serde(default)]
-    state: Option<String>,
-    #[serde(default)]
-    endorsed: Vec<String>,
-}
-
-#[derive(Debug, Clone, Deserialize, PartialEq)]
-struct HybridResp {
-    #[serde(default)]
-    sql_matches: usize,
-    #[serde(default)]
-    vector_reranked: usize,
-    #[serde(default)]
-    method: String,
-    #[serde(default)]
-    duration_ms: u64,
-    #[serde(default)]
-    answer: Option<String>,
-    #[serde(default)]
-    sources: Vec<HybridSource>,
-}
-
-#[derive(Debug, Clone, Deserialize, PartialEq)]
-struct HybridSource {
-    doc_id: String,
-    chunk_text: String,
-    score: f32,
-    #[serde(default)]
-    sql_verified: bool,
-    #[serde(default)]
-    playbook_boost: f32,
-    #[serde(default)]
-    playbook_citations: Vec<String>,
-}
-
-#[derive(Debug, Clone, Deserialize, PartialEq)]
-struct IndexInfo {
-    index_name: String,
-    source: String,
-    #[serde(default)]
-    chunk_count: usize,
-    #[serde(default)]
-    vector_backend: String,
-}
-
-async fn fetch_playbook_stats() -> Result<PlaybookStats, String> {
-    let resp = reqwest::get(&format!("{}/vectors/playbook_memory/stats", api_base()))
-        .await.map_err(|e| e.to_string())?;
-    if !resp.status().is_success() {
-        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
-    }
-    resp.json().await.map_err(|e| e.to_string())
-}
-
-async fn rebuild_playbook_memory() -> Result<serde_json::Value, String> {
-    let client = reqwest::Client::new();
-    let resp = client.post(&format!("{}/vectors/playbook_memory/rebuild", api_base()))
-        .json(&serde_json::json!({}))
-        .send().await.map_err(|e| e.to_string())?;
-    if !resp.status().is_success() {
-        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
-    }
-    resp.json().await.map_err(|e| e.to_string())
-}
-
-async fn fetch_indexes() -> Result<Vec<IndexInfo>, String> {
-    let resp = reqwest::get(&format!("{}/vectors/indexes", api_base()))
-        .await.map_err(|e| e.to_string())?;
-    resp.json().await.map_err(|e| e.to_string())
-}
-
-async fn hybrid_search(index_name: &str, question: &str, use_playbook: bool, top_k: usize) -> Result<HybridResp, String> {
-    let client = reqwest::Client::new();
-    let resp = client.post(&format!("{}/vectors/hybrid", api_base()))
-        .json(&serde_json::json!({
-            "index_name": index_name,
-            "question": question,
-            "top_k": top_k,
-            "generate": false,
-            "use_playbook_memory": use_playbook,
-        }))
-        .send().await.map_err(|e| e.to_string())?;
-    if !resp.status().is_success() {
-        return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
-    }
-    resp.json().await.map_err(|e| e.to_string())
-}
-
 // --- App ---
 
 #[component]
@@ -346,11 +239,6 @@ fn App() -> Element {
                         onclick: move |_| active_tab.set(Tab::Ingest),
                         "Ingest"
                     }
-                    button {
-                        class: if *active_tab.read() == Tab::Playbook { "tab active" } else { "tab" },
-                        onclick: move |_| active_tab.set(Tab::Playbook),
-                        "Playbook"
-                    }
                     button {
                         class: if *active_tab.read() == Tab::Status { "tab active" } else { "tab" },
                         onclick: move |_| active_tab.set(Tab::Status),
@@ -372,7 +260,6 @@ fn App() -> Element {
                     Tab::Explore => rsx! { ExplorePanel { datasets: datasets.read().clone() } },
                     Tab::Sql => rsx! { SqlPanel {} },
                     Tab::Ingest => rsx! { IngestPanel {} },
-                    Tab::Playbook => rsx! { PlaybookPanel {} },
                     Tab::Status => rsx! { StatusPanel {} },
                 }
             }
@@ -467,14 +354,14 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
         div { class: "panel ask-panel",
             div { class: "ask-hero",
                 h2 { "Ask your data anything" }
-                p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI over the staffing dataset." }
+                p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI." }
             }
 
             div { class: "ask-input-row",
                 input {
                     class: "ask-input",
                     value: "{question}",
-                    placeholder: "e.g. Which clients placed the most candidates last quarter?",
+                    placeholder: "e.g. Which department has the highest average salary?",
                     oninput: move |e| question.set(e.value()),
                     onkeydown: move |e| {
                         if e.key() == Key::Enter {
@@ -545,12 +432,10 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
 
             div { class: "ask-examples",
                 "Try: "
-                button { class: "example-btn", onclick: move |_| question.set("How many candidates do we have by city?".into()), "candidates by city" }
-                button { class: "example-btn", onclick: move |_| question.set("Top 10 clients by total placements".into()), "top clients by placements" }
-                button { class: "example-btn", onclick: move |_| question.set("Open job orders ordered by bill rate descending".into()), "open jobs by rate" }
-                button { class: "example-btn", onclick: move |_| question.set("Recruiters with the highest placement count".into()), "top recruiters" }
-                button { class: "example-btn", onclick: move |_| question.set("Total billed hours per client last month".into()), "hours per client" }
-                button { class: "example-btn", onclick: move |_| question.set("Cold leads: candidates we called more than 5 times but never placed".into()), "cold leads" }
+                button { class: "example-btn", onclick: move |_| question.set("Which department has the highest average salary?".into()), "highest avg salary by dept" }
+                button { class: "example-btn", onclick: move |_| question.set("Show me the top 3 most expensive products".into()), "top 3 expensive products" }
+                button { class: "example-btn", onclick: move |_| question.set("How many events per action type?".into()), "events by action" }
+                button { class: "example-btn", onclick: move |_| question.set("List all employees who earn more than 90000".into()), "employees > 90k" }
             }
 
             if let Some(sql) = generated_sql.read().as_ref() {
@@ -693,7 +578,7 @@ fn ExplorePanel(datasets: Vec<Dataset>) -> Element {
 
 #[component]
 fn SqlPanel() -> Element {
-    let mut query_text = use_signal(|| String::from("SELECT candidate_id, first_name, last_name, city, status FROM candidates LIMIT 10"));
+    let mut query_text = use_signal(|| String::from("SELECT * FROM employees LIMIT 10"));
     let mut result = use_signal(|| None::<Result<QueryResponse, String>>);
     let mut loading = use_signal(|| false);
 
@@ -842,7 +727,7 @@ fn DashboardPanel() -> Element {
                     }
                     div { class: "stat-card accent",
                         div { class: "stat-value", "{s[\"hnsw_loaded\"]}" }
-                        div { class: "stat-label", "HNSW Indexes Loaded" }
+                        div { class: "stat-label", "HNSW Indexes (27ms)" }
                     }
                     div { class: "stat-card",
                         div { class: "stat-value", "{s[\"tools\"]}" }
@@ -865,27 +750,27 @@ fn DashboardPanel() -> Element {
                     div { class: "arch-grid",
                         div { class: "arch-card",
                             div { class: "arch-title", "Ingest" }
-                            div { class: "arch-items", "CSV · JSON · PDF (+OCR) · Text · Postgres · MySQL · Inbox watcher · Cron schedules" }
+                            div { class: "arch-items", "CSV, JSON, PDF, Text, PostgreSQL, File Watcher" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Storage" }
-                            div { class: "arch-items", "Parquet on Object Storage · Delta writes · Compaction · Tombstones · Multi-bucket federation + rescue" }
+                            div { class: "arch-items", "Parquet on Object Storage, Delta Writes, Compaction" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Query" }
-                            div { class: "arch-items", "DataFusion SQL · MemCache (9.8× hot) · Merge-on-read · AI-safe views" }
+                            div { class: "arch-items", "DataFusion SQL, MemCache (9.8x), Hot/Cold" }
                         }
                         div { class: "arch-card",
-                            div { class: "arch-title", "AI / Vector" }
-                            div { class: "arch-items", "Ollama (local) · Embed/Generate/RAG · HNSW (Parquet) · Lance IVF_PQ · Hybrid SQL+vector · Profile-scoped" }
-                        }
-                        div { class: "arch-card",
-                            div { class: "arch-title", "Learning loop" }
-                            div { class: "arch-items", "Playbook memory · Endorsement boost · Multi-agent orchestrator · Autotune agent (Pareto-promote)" }
+                            div { class: "arch-title", "AI" }
+                            div { class: "arch-items", "Ollama (local), Embed, Generate, RAG, HNSW" }
                         }
                         div { class: "arch-card",
                             div { class: "arch-title", "Governance" }
-                            div { class: "arch-items", "Event journal · PII detection · Tool registry · Access control · Audit log · Catalog v2 metadata" }
+                            div { class: "arch-items", "Event Journal, PII Detection, Tool Registry, Access Control" }
+                        }
+                        div { class: "arch-card",
+                            div { class: "arch-title", "Agents" }
+                            div { class: "arch-items", "Workspaces, Handoff, Shortlists, Activity Logs" }
                         }
                     }
                 }
@@ -894,23 +779,20 @@ fn DashboardPanel() -> Element {
                     h3 { "Build Progression" }
                     div { class: "phase-list",
                         {rsx! {
-                            PhaseItem { num: "0-5", name: "Foundation", detail: "Storage · Catalog · DataFusion · Ollama · UI · gRPC" }
-                            PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV · JSON · PDF · Text · auto-schema · dedupe" }
-                            PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed · brute-force cosine · LLM grounded answers" }
-                            PhaseItem { num: "8", name: "Hot Cache + Deltas", detail: "MemTable LRU · 9.8× speedup · merge-on-read · compaction" }
-                            PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract · daily/weekly/monthly tiers · zero-copy handoff" }
-                            PhaseItem { num: "9", name: "Event Journal", detail: "Append-only mutation log · time-travel · audit" }
-                            PhaseItem { num: "10", name: "Rich Catalog v2", detail: "PII auto-detection · lineage · freshness SLA · sensitivity" }
-                            PhaseItem { num: "11", name: "Embedding Versioning", detail: "Per-index model+version · A/B · incremental re-embed" }
-                            PhaseItem { num: "12", name: "Tool Registry", detail: "Governed actions · param validation · audit · MCP-ready" }
-                            PhaseItem { num: "13", name: "Access Control", detail: "Roles · field-level sensitivity · column masking · query audit" }
-                            PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff detection · AI migration prompts · versioned schemas" }
-                            PhaseItem { num: "15", name: "HNSW + Trials", detail: "100K vectors · p50 873µs · trial journal · eval harness" }
-                            PhaseItem { num: "16", name: "Hot-swap + Autotune", detail: "Promotion registry · rollback · ε-greedy agent · Pareto winner" }
-                            PhaseItem { num: "17", name: "Model Profiles + VRAM", detail: "ModelProfile manifests · scoped search · sequential model swap" }
-                            PhaseItem { num: "18", name: "Lance hybrid backend", detail: "IVF_PQ build 14× faster · random fetch 112× · S3-native · per-profile routing" }
-                            PhaseItem { num: "19", name: "Playbook memory", detail: "Feedback loop · endorsement boost (cap 0.25) · orchestrator write-through · citations" }
-                            PhaseItem { num: "+", name: "Federation + Schedules", detail: "Multi-bucket · rescue fallback · error journal · MySQL · PDF OCR · cron ingest · catalog dedupe" }
+                            PhaseItem { num: "0-5", name: "Foundation", detail: "Storage, Catalog, DataFusion, AI, UI, gRPC" }
+                            PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV/JSON/PDF/Text auto-schema" }
+                            PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed, Search, LLM Answers" }
+                            PhaseItem { num: "8", name: "Hot Cache", detail: "9.8x speedup, Delta Writes" }
+                            PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract, Instant Handoff" }
+                            PhaseItem { num: "9", name: "Event Journal", detail: "Append-only Mutation History" }
+                            PhaseItem { num: "10", name: "Rich Catalog", detail: "PII Detection, Lineage" }
+                            PhaseItem { num: "11", name: "Embedding Versioning", detail: "Model-proof Vectors" }
+                            PhaseItem { num: "12", name: "Tool Registry", detail: "6 Governed Actions + Audit" }
+                            PhaseItem { num: "13", name: "Access Control", detail: "Role-based, Field-level" }
+                            PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff Detection, AI Migration" }
+                            PhaseItem { num: "15", name: "HNSW Index", detail: "100K Search in 27ms" }
+                            PhaseItem { num: "16", name: "File Watcher", detail: "Auto-ingest from Inbox" }
+                            PhaseItem { num: "17", name: "DB Connector", detail: "PostgreSQL Import" }
                         }}
                     }
                 }
@@ -934,285 +816,6 @@ fn PhaseItem(num: String, name: String, detail: String) -> Element {
     }
 }
 
-// === PLAYBOOK — Phase 19 meta-index feedback loop ===
-
-#[component]
-fn PlaybookPanel() -> Element {
-    let mut stats = use_signal(|| None::<Result<PlaybookStats, String>>);
-    let mut indexes = use_signal(Vec::<IndexInfo>::new);
-    let mut rebuild_status = use_signal(|| None::<Result<String, String>>);
-    let mut rebuilding = use_signal(|| false);
-    let mut loaded = use_signal(|| false);
-
-    // Comparison state
-    let mut selected_index = use_signal(|| String::new());
-    let mut question = use_signal(|| String::from("reliable assembler in Detroit"));
-    let mut top_k = use_signal(|| 10usize);
-    let mut compare_loading = use_signal(|| false);
-    let mut hits_off = use_signal(|| None::<Result<HybridResp, String>>);
-    let mut hits_on = use_signal(|| None::<Result<HybridResp, String>>);
-
-    let load_all = move || {
-        spawn(async move {
-            stats.set(Some(fetch_playbook_stats().await));
-            if let Ok(ix) = fetch_indexes().await {
-                if selected_index.read().is_empty() {
-                    if let Some(default) = ix.iter().find(|i| i.source == "workers_500k").or_else(|| ix.first()) {
-                        selected_index.set(default.index_name.clone());
-                    }
-                }
-                indexes.set(ix);
-            }
-        });
-    };
-
-    use_effect(move || {
-        if !*loaded.read() {
-            loaded.set(true);
-            load_all();
-        }
-    });
-
-    let do_rebuild = move |_| {
-        spawn(async move {
-            rebuilding.set(true);
-            rebuild_status.set(None);
-            match rebuild_playbook_memory().await {
-                Ok(v) => rebuild_status.set(Some(Ok(format!("rebuild ok — {}", v)))),
-                Err(e) => rebuild_status.set(Some(Err(e))),
-            }
-            // Refresh stats afterward
-            stats.set(Some(fetch_playbook_stats().await));
-            rebuilding.set(false);
-        });
-    };
-
-    let do_compare = move |_| {
-        let idx = selected_index.read().clone();
-        let q = question.read().clone();
-        let k = *top_k.read();
-        if idx.is_empty() || q.trim().is_empty() { return; }
-        spawn(async move {
-            compare_loading.set(true);
-            hits_off.set(None);
-            hits_on.set(None);
-            // Run both sequentially so the embedding cache is shared
-            hits_off.set(Some(hybrid_search(&idx, &q, false, k).await));
-            hits_on.set(Some(hybrid_search(&idx, &q, true, k).await));
-            compare_loading.set(false);
-        });
-    };
-
-    rsx! {
-        div { class: "panel",
-            div { class: "ask-hero",
-                h2 { "Playbook Memory" }
-                p { class: "subtitle",
-                    "Phase 19 feedback loop: past successful playbooks boost future search rankings. \
-                     Endorsed workers from semantically similar past operations re-rank toward the top, \
-                     with citations back to the playbook that endorsed them."
-                }
-            }
-
-            // Stats card
-            div { class: "panel-section",
-                match stats.read().as_ref() {
-                    None => rsx! { div { class: "loading", "loading playbook stats..." } },
-                    Some(Err(e)) => rsx! { div { class: "error", "stats: {e}" } },
-                    Some(Ok(s)) => rsx! {
-                        div { class: "stat-grid",
-                            div { class: "stat-card",
-                                div { class: "stat-value", "{s.entries}" }
-                                div { class: "stat-label", "Playbooks in Memory" }
-                            }
-                            div { class: "stat-card",
-                                div { class: "stat-value", "{s.entries_with_embeddings}" }
-                                div { class: "stat-label", "Embedded" }
-                            }
-                            div { class: "stat-card accent",
-                                div { class: "stat-value", "{s.total_names_endorsed}" }
-                                div { class: "stat-label", "Endorsed Worker-Tags" }
-                            }
-                        }
-                    }
-                }
-                div { class: "sql-actions",
-                    button {
-                        class: "btn",
-                        disabled: *rebuilding.read(),
-                        onclick: do_rebuild,
-                        if *rebuilding.read() { "rebuilding from successful_playbooks..." } else { "Rebuild from successful_playbooks" }
-                    }
-                }
-                if let Some(s) = rebuild_status.read().as_ref() {
-                    match s {
-                        Ok(msg) => rsx! { div { class: "result-box", "{msg}" } },
-                        Err(e) => rsx! { div { class: "error", "{e}" } },
-                    }
-                }
-            }
-
-            // Sample playbooks
-            if let Some(Ok(s)) = stats.read().as_ref() {
-                if !s.sample.is_empty() {
-                    div { class: "panel-section",
-                        h3 { "Sample playbooks" }
-                        div { class: "table-wrap",
-                            table {
-                                thead { tr {
-                                    th { "ID" }
-                                    th { "Operation" }
-                                    th { "Location" }
-                                    th { "Endorsed" }
-                                } }
-                                tbody {
-                                    for pb in s.sample.iter() {
-                                        {
-                                            let loc = match (&pb.city, &pb.state) {
-                                                (Some(c), Some(st)) => format!("{c}, {st}"),
-                                                _ => "—".into(),
-                                            };
-                                            let endorsed = if pb.endorsed.is_empty() {
-                                                "—".to_string()
-                                            } else {
-                                                pb.endorsed.join(", ")
-                                            };
-                                            let pid = pb.id.clone();
-                                            let op = pb.operation.clone();
-                                            rsx! {
-                                                tr {
-                                                    td { class: "mono-cell", title: "{pid}", "{pid}" }
-                                                    td { "{op}" }
-                                                    td { "{loc}" }
-                                                    td { "{endorsed}" }
-                                                }
-                                            }
-                                        }
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-
-            // Side-by-side comparison: boost OFF vs ON
-            div { class: "panel-section",
-                h3 { "See the boost — search compared" }
-                p { class: "hint",
-                    "Run the same query against the same index twice — once with playbook boost OFF and once ON. \
-                     Hits with non-zero playbook_boost and citations are workers that past similar playbooks endorsed."
-                }
-                div { class: "form-row",
-                    label { "Index" }
-                    select {
-                        value: "{selected_index}",
-                        onchange: move |e| selected_index.set(e.value()),
-                        for ix in indexes.read().iter() {
-                            option { value: "{ix.index_name}", "{ix.index_name} ({ix.source}, {ix.chunk_count} chunks, {ix.vector_backend})" }
-                        }
-                    }
-                }
-                div { class: "form-row",
-                    label { "Question" }
-                    input {
-                        value: "{question}",
-                        oninput: move |e| question.set(e.value()),
-                        placeholder: "e.g. reliable assembler in Detroit"
-                    }
-                }
-                div { class: "form-row",
-                    label { "Top K" }
-                    input {
-                        r#type: "number",
-                        value: "{top_k}",
-                        oninput: move |e| {
-                            if let Ok(n) = e.value().parse::<usize>() { top_k.set(n.clamp(1, 50)); }
-                        }
-                    }
-                }
-                button {
-                    class: "btn btn-ask",
-                    disabled: *compare_loading.read(),
-                    onclick: do_compare,
-                    if *compare_loading.read() { "running both queries..." } else { "Run comparison" }
-                }
-
-                div { class: "explore-grid",
-                    div { class: "ds-detail",
-                        h3 { "Boost OFF (vanilla)" }
-                        match hits_off.read().as_ref() {
-                            None => rsx! { div { class: "empty", "—" } },
-                            Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
-                            Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
-                        }
-                    }
-                    div { class: "ds-detail",
-                        h3 { "Boost ON (Phase 19)" }
-                        match hits_on.read().as_ref() {
-                            None => rsx! { div { class: "empty", "—" } },
-                            Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
-                            Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
-                        }
-                    }
-                }
-            }
-        }
-    }
-}
-
-#[component]
-fn HybridHitTable(resp: HybridResp) -> Element {
-    rsx! {
-        div { class: "results-info",
-            "{resp.sources.len()} hits · {resp.duration_ms}ms · method={resp.method}"
-        }
-        if resp.sources.is_empty() {
-            div { class: "empty-sm", "no hits" }
-        } else {
-            div { class: "table-wrap",
-                table {
-                    thead { tr {
-                        th { "#" }
-                        th { "Doc" }
-                        th { "Score" }
-                        th { "Boost" }
-                        th { "Citations" }
-                        th { "Snippet" }
-                    } }
-                    tbody {
-                        for (i, h) in resp.sources.iter().enumerate() {
-                            {
-                                let snippet: String = h.chunk_text.chars().take(120).collect();
-                                let cites = if h.playbook_citations.is_empty() {
-                                    "—".to_string()
-                                } else {
-                                    h.playbook_citations.join(", ")
-                                };
-                                let row_class = if h.playbook_boost > 0.0 { "boosted-row" } else { "" };
-                                let rank = i + 1;
-                                let did = h.doc_id.clone();
-                                let score = format!("{:.3}", h.score);
-                                let boost = if h.playbook_boost > 0.0 { format!("+{:.3}", h.playbook_boost) } else { "—".into() };
-                                rsx! {
-                                    tr { class: "{row_class}",
-                                        td { "{rank}" }
-                                        td { class: "mono-cell", "{did}" }
-                                        td { "{score}" }
-                                        td { "{boost}" }
-                                        td { class: "mono-cell", title: "{cites}", "{cites}" }
-                                        td { "{snippet}" }
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-    }
-}
-
 // === INGEST — Data on-ramp ===
 
 #[component]

crates/vectord-lance/Cargo.toml

@@ -1,37 +0,0 @@
-[package]
-name = "vectord-lance"
-version = "0.1.0"
-edition = "2024"
-
-# Firewall crate: the Lance stack (Arrow 57, DataFusion 52) is isolated
-# from the rest of vectord (Arrow 55, DataFusion 47). Public API uses
-# only std types — no Arrow types cross the crate boundary — so no
-# version conflict propagates outward.
-#
-# See docs/ADR-019-vector-storage.md for the rationale: "productionizing
-# will need either workspace-wide upgrade or a firewall via a dedicated
-# vectord-lance crate." This is that firewall.
-
-[dependencies]
-# S3 support: Lance delegates to its internal object_store crate when
-# given s3:// URIs. The "dynamodb" feature enables DynamoDB-based
-# commit locking for multi-writer S3; we don't need that (single-writer)
-# so just the base AWS/S3 feature is enough.
-# Lance 4.0 feature "aws" enables S3-compatible storage via its internal
-# object_store + opendal crates. Reads AWS_* env vars for credentials.
-lance = { version = "4.0", default-features = false, features = ["aws"] }
-lance-index = { version = "4.0", default-features = false }
-lance-linalg = { version = "4.0", default-features = false }
-
-# These Arrow/Parquet versions MUST match Lance's expectations — Lance
-# re-exports their types through its API so any mismatch is a hard
-# compile error. Keep in sync with lance-bench.
-arrow = "57"
-arrow-array = "57"
-arrow-schema = "57"
-parquet = "57"
-
-tokio = { version = "1", features = ["full"] }
-futures = "0.3"
-serde = { version = "1", features = ["derive"] }
-bytes = "1"

crates/vectord-lance/src/lib.rs

@@ -1,605 +0,0 @@
-//! Production Lance vector backend (ADR-019 — hybrid architecture).
-//!
-//! This is the firewall crate. It owns its own Arrow 57 / DataFusion 52
-//! / Lance 4 dependency tree. The public API uses only std types
-//! (`Vec<f32>`, `Vec<String>`, `String`, `bool`) so nothing Arrow-shaped
-//! crosses the crate boundary. That keeps vectord (Arrow 55) from
-//! picking up an incompatible dep.
-//!
-//! Responsibilities:
-//! - Migrate an existing binary-blob Parquet vector file into a Lance
-//!   dataset with `FixedSizeList<Float32, dims>`. One-time cost.
-//! - Append new rows natively (no full-file rewrite — Lance's structural win).
-//! - Build an IVF_PQ ANN index on the vector column.
-//! - Vector search (`search`) using the IVF_PQ index when present,
-//!   falling back to full scan otherwise.
-//! - Random-access row fetch by `doc_id` (`get_by_doc_id`) — the O(1)
-//!   lookup that Parquet-on-object-store can't cheaply do.
-//! - Cheap count + stats introspection.
-
-use arrow_array::{
-    Array, ArrayRef, BinaryArray, FixedSizeListArray, Float32Array, Int32Array,
-    RecordBatch, RecordBatchIterator, StringArray,
-};
-use arrow_schema::{DataType, Field, Schema};
-use futures::StreamExt;
-use serde::Serialize;
-use std::sync::Arc;
-use std::time::Instant;
-
-// ================= Public types =================
-
-/// One search result. Mirrors vectord's existing `SearchResult` shape
-/// structurally but carries simpler types so this crate stays firewalled.
-#[derive(Debug, Clone, Serialize)]
-pub struct Hit {
-    pub doc_id: String,
-    pub chunk_text: String,
-    pub score: f32,
-    /// Optional — set by search_with_vector, not by index-only search.
-    pub distance: Option<f32>,
-}
-
-/// A fully-hydrated row fetched by `get_by_doc_id` — includes the vector
-/// so callers can do downstream work (rerank, cite, etc.) without a
-/// second round trip.
-#[derive(Debug, Clone, Serialize)]
-pub struct Row {
-    pub doc_id: String,
-    pub chunk_text: String,
-    pub vector: Vec<f32>,
-    pub source: Option<String>,
-    pub chunk_idx: Option<i32>,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct MigrationStats {
-    pub rows_written: usize,
-    pub dimensions: usize,
-    pub disk_bytes: u64,
-    pub duration_secs: f32,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct AppendStats {
-    pub rows_appended: usize,
-    pub disk_bytes_added: u64,
-    pub duration_secs: f32,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct IndexStats {
-    pub name: String,
-    pub num_partitions: u32,
-    pub num_bits: u32,
-    pub num_sub_vectors: u32,
-    pub build_time_secs: f32,
-    pub disk_bytes_added: u64,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct ScalarIndexStats {
-    pub name: String,
-    pub column: String,
-    pub build_time_secs: f32,
-    pub disk_bytes_added: u64,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct DatasetStats {
-    pub path: String,
-    pub rows: usize,
-    pub disk_bytes: u64,
-    pub has_vector_index: bool,
-    pub has_doc_id_index: bool,
-}
-
-// ================= The backend =================
-
-/// Thin wrapper around a Lance dataset path. Lance handles the heavy
-/// lifting — we just expose a narrow API.
-#[derive(Clone)]
-pub struct LanceVectorStore {
-    /// Local filesystem path or object-store URI (file:///..., s3://...).
-    /// Lance's internal URI parsing handles both.
-    path: String,
-}
-
-impl LanceVectorStore {
-    pub fn new(path: impl Into<String>) -> Self {
-        Self { path: path.into() }
-    }
-
-    pub fn path(&self) -> &str { &self.path }
-
-    /// Row count via Lance's fast metadata path (no scan).
-    pub async fn count(&self) -> Result<usize, String> {
-        let dataset = open_or_err(&self.path).await?;
-        let n = dataset.count_rows(None).await.map_err(e)?;
-        Ok(n)
-    }
-
-    /// True if the on-disk dataset exists AND has at least one `vector`
-    /// column index attached.
-    pub async fn has_vector_index(&self) -> Result<bool, String> {
-        use lance_index::DatasetIndexExt;
-        let dataset = match lance::dataset::Dataset::open(&self.path).await {
-            Ok(d) => d,
-            Err(_) => return Ok(false),
-        };
-        let indexes = dataset.load_indices().await.map_err(e)?;
-        Ok(indexes.iter().any(|ix| {
-            ix.fields.iter().any(|fid| {
-                dataset.schema().field_by_id(*fid)
-                    .map(|f| f.name == "vector")
-                    .unwrap_or(false)
-            })
-        }))
-    }
-
-    pub async fn stats(&self) -> Result<DatasetStats, String> {
-        let rows = self.count().await.unwrap_or(0);
-        let disk_bytes = dir_size_bytes(&strip_file_uri(&self.path));
-        let has_vector_index = self.has_vector_index().await.unwrap_or(false);
-        let has_doc_id_index = self.has_scalar_index("doc_id").await.unwrap_or(false);
-        Ok(DatasetStats {
-            path: self.path.clone(),
-            rows,
-            disk_bytes,
-            has_vector_index,
-            has_doc_id_index,
-        })
-    }
-
-    /// Migrate a vectord-format Parquet file into a Lance dataset.
-    ///
-    /// Input schema (vectord's binary-blob format):
-    ///   - source      : Utf8
-    ///   - doc_id      : Utf8
-    ///   - chunk_idx   : Int32
-    ///   - chunk_text  : Utf8
-    ///   - vector      : Binary (raw f32 little-endian bytes)
-    ///
-    /// Output schema (Lance-friendly):
-    ///   - source      : Utf8
-    ///   - doc_id      : Utf8
-    ///   - chunk_idx   : Int32
-    ///   - chunk_text  : Utf8
-    ///   - vector      : FixedSizeList<Float32, dims>
-    ///
-    /// Idempotent at the file level — if the target exists, it's
-    /// overwritten. Caller must manage destination paths.
-    pub async fn migrate_from_parquet_bytes(
-        &self,
-        parquet_bytes: &[u8],
-    ) -> Result<MigrationStats, String> {
-        let t0 = Instant::now();
-        let (schema, batches, rows) = read_parquet(parquet_bytes)?;
-        let dims = detect_vector_dims(&batches)?;
-        let (new_schema, new_batches) = convert_to_fixed_size_list(&schema, batches, dims)?;
-
-        // Overwrite any prior dataset at this path.
-        let _ = std::fs::remove_dir_all(&strip_file_uri(&self.path));
-
-        write_dataset(&self.path, new_schema, new_batches).await?;
-        let disk_bytes = dir_size_bytes(&strip_file_uri(&self.path));
-
-        Ok(MigrationStats {
-            rows_written: rows,
-            dimensions: dims,
-            disk_bytes,
-            duration_secs: t0.elapsed().as_secs_f32(),
-        })
-    }
-
-    /// Native Lance append — does NOT rewrite existing files. New rows
-    /// land as a separate fragment; readers union across fragments at
-    /// query time. Contrast: our Parquet path requires rewriting the
-    /// entire vector file to add rows.
-    pub async fn append(
-        &self,
-        source: Option<String>,
-        doc_ids: Vec<String>,
-        chunk_idxs: Vec<i32>,
-        chunk_texts: Vec<String>,
-        vectors: Vec<Vec<f32>>,
-    ) -> Result<AppendStats, String> {
-        let n = doc_ids.len();
-        if n == 0 {
-            return Ok(AppendStats { rows_appended: 0, disk_bytes_added: 0, duration_secs: 0.0 });
-        }
-        if chunk_idxs.len() != n || chunk_texts.len() != n || vectors.len() != n {
-            return Err(format!(
-                "append: length mismatch (doc_ids={n}, chunk_idxs={}, chunk_texts={}, vectors={})",
-                chunk_idxs.len(), chunk_texts.len(), vectors.len(),
-            ));
-        }
-        let dims = vectors[0].len();
-        for (i, v) in vectors.iter().enumerate() {
-            if v.len() != dims {
-                return Err(format!("append: row {i} has {} dims, expected {}", v.len(), dims));
-            }
-        }
-
-        let t0 = Instant::now();
-        let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
-
-        let src_arr = StringArray::from(
-            (0..n).map(|_| source.clone()).collect::<Vec<_>>()
-        );
-        let doc_id_arr = StringArray::from(doc_ids);
-        let chunk_idx_arr = Int32Array::from(chunk_idxs);
-        let chunk_text_arr = StringArray::from(chunk_texts);
-
-        let mut flat: Vec<f32> = Vec::with_capacity(n * dims);
-        for v in vectors { flat.extend(v); }
-        let values = Float32Array::from(flat);
-        let item_field = Arc::new(Field::new("item", DataType::Float32, true));
-        let vec_arr = FixedSizeListArray::try_new(
-            item_field.clone(), dims as i32, Arc::new(values), None,
-        ).map_err(e)?;
-
-        let schema = Arc::new(Schema::new(vec![
-            Field::new("source", DataType::Utf8, true),
-            Field::new("doc_id", DataType::Utf8, false),
-            Field::new("chunk_idx", DataType::Int32, true),
-            Field::new("chunk_text", DataType::Utf8, true),
-            Field::new("vector", DataType::FixedSizeList(item_field, dims as i32), false),
-        ]));
-
-        let arrays: Vec<ArrayRef> = vec![
-            Arc::new(src_arr), Arc::new(doc_id_arr), Arc::new(chunk_idx_arr),
-            Arc::new(chunk_text_arr), Arc::new(vec_arr),
-        ];
-        let batch = RecordBatch::try_new(schema.clone(), arrays).map_err(e)?;
-        let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema);
-
-        use lance::dataset::{Dataset, WriteMode, WriteParams};
-        let params = WriteParams { mode: WriteMode::Append, ..Default::default() };
-        Dataset::write(reader, &self.path, Some(params)).await.map_err(e)?;
-
-        Ok(AppendStats {
-            rows_appended: n,
-            disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
-            duration_secs: t0.elapsed().as_secs_f32(),
-        })
-    }
-
-    /// Build an IVF_PQ vector index. Replaces any prior index with the
-    /// same name. Callers pass explicit params — sensible defaults for
-    /// ~100K × 768d: num_partitions=316 (≈√N), num_bits=8, num_sub_vectors=48.
-    pub async fn build_index(
-        &self,
-        num_partitions: u32,
-        num_bits: u32,
-        num_sub_vectors: u32,
-    ) -> Result<IndexStats, String> {
-        use lance::dataset::Dataset;
-        use lance::index::vector::VectorIndexParams;
-        use lance_index::{DatasetIndexExt, IndexType};
-        use lance_linalg::distance::MetricType;
-
-        let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
-        let t0 = Instant::now();
-
-        let mut dataset = Dataset::open(&self.path).await.map_err(e)?;
-        let params = VectorIndexParams::ivf_pq(
-            num_partitions as usize,
-            num_bits as u8,
-            num_sub_vectors as usize,
-            MetricType::Cosine,
-            50, // max_iterations — same as bench
-        );
-        dataset.create_index(
-            &["vector"],
-            IndexType::Vector,
-            Some("vec_idx".into()),
-            &params,
-            true, // replace
-        ).await.map_err(e)?;
-
-        Ok(IndexStats {
-            name: "vec_idx".into(),
-            num_partitions,
-            num_bits,
-            num_sub_vectors,
-            build_time_secs: t0.elapsed().as_secs_f32(),
-            disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
-        })
-    }
-
-    /// Build a scalar btree index on `doc_id`. Makes `get_by_doc_id`
-    /// O(log N) instead of a filter-scan of every fragment. Small index
-    /// footprint — a few hundred KB on 100K rows.
-    pub async fn build_scalar_index(&self, column: &str) -> Result<ScalarIndexStats, String> {
-        use lance::dataset::Dataset;
-        use lance_index::{DatasetIndexExt, IndexType};
-        use lance_index::scalar::ScalarIndexParams;
-
-        let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
-        let t0 = Instant::now();
-
-        let mut dataset = Dataset::open(&self.path).await.map_err(e)?;
-        let idx_name = format!("{column}_btree");
-        dataset.create_index(
-            &[column],
-            IndexType::Scalar,
-            Some(idx_name.clone()),
-            &ScalarIndexParams::default(),
-            true,
-        ).await.map_err(e)?;
-
-        Ok(ScalarIndexStats {
-            name: idx_name,
-            column: column.to_string(),
-            build_time_secs: t0.elapsed().as_secs_f32(),
-            disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
-        })
-    }
-
-    /// True if a scalar index exists on the named column.
-    pub async fn has_scalar_index(&self, column: &str) -> Result<bool, String> {
-        use lance_index::DatasetIndexExt;
-        let dataset = match lance::dataset::Dataset::open(&self.path).await {
-            Ok(d) => d,
-            Err(_) => return Ok(false),
-        };
-        let indexes = dataset.load_indices().await.map_err(e)?;
-        Ok(indexes.iter().any(|ix| {
-            ix.fields.iter().any(|fid| {
-                dataset.schema().field_by_id(*fid)
-                    .map(|f| f.name == column)
-                    .unwrap_or(false)
-            })
-        }))
-    }
-
-    /// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ
-    /// index if one exists; otherwise does a full scan (slow but
-    /// correct — useful during development before index build).
-    ///
-    /// `nprobes` tells Lance how many IVF partitions to probe per query.
-    /// Lance's built-in default is **1**, which caps recall well below
-    /// what the index is actually capable of. Passing `None` keeps that
-    /// default (only sensible when latency trumps recall); for real
-    /// workloads set `nprobes` to 5–10% of `num_partitions` (e.g. 20–30
-    /// on a 316-partition index).
-    ///
-    /// `refine_factor` re-ranks the PQ-approximate top-k by computing
-    /// exact distances on `top_k * refine_factor` candidates, then
-    /// trimming to `top_k`. Cheap way to buy back recall lost to product
-    /// quantization. `None` skips the re-rank pass.
-    pub async fn search(
-        &self,
-        query: &[f32],
-        top_k: usize,
-        nprobes: Option<usize>,
-        refine_factor: Option<u32>,
-    ) -> Result<Vec<Hit>, String> {
-        use lance::dataset::Dataset;
-
-        let dataset = Dataset::open(&self.path).await.map_err(e)?;
-        let qarr = Float32Array::from(query.to_vec());
-
-        let mut scanner = dataset.scan();
-        scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?;
-        if let Some(n) = nprobes { scanner.nprobes(n); }
-        if let Some(f) = refine_factor { scanner.refine(f); }
-        scanner.project(&["doc_id", "chunk_text"]).map_err(e)?;
-
-        let mut stream = scanner.try_into_stream().await.map_err(e)?;
-        let mut hits: Vec<Hit> = Vec::with_capacity(top_k);
-
-        while let Some(batch) = stream.next().await {
-            let batch = batch.map_err(e)?;
-            let doc_ids = batch.column_by_name("doc_id")
-                .ok_or_else(|| "no doc_id column in search result".to_string())?
-                .as_any().downcast_ref::<StringArray>()
-                .ok_or_else(|| "doc_id is not StringArray".to_string())?;
-            let chunk_texts = batch.column_by_name("chunk_text")
-                .and_then(|c| c.as_any().downcast_ref::<StringArray>());
-            // Lance tacks on a `_distance` column for nearest() queries.
-            let distances = batch.column_by_name("_distance")
-                .and_then(|c| c.as_any().downcast_ref::<Float32Array>());
-            for row in 0..batch.num_rows() {
-                let d = distances.map(|a| a.value(row));
-                hits.push(Hit {
-                    doc_id: doc_ids.value(row).to_string(),
-                    chunk_text: chunk_texts.map(|a| a.value(row).to_string()).unwrap_or_default(),
-                    score: d.map(|d| 1.0 - d).unwrap_or(0.0), // cosine distance → similarity
-                    distance: d,
-                });
-            }
-        }
-        Ok(hits)
-    }
-
-    /// Fetch one row by doc_id. Implementation: Lance filter-pushdown
-    /// scan — O(1) with partition pruning on a proper btree index,
-    /// O(N) on vector-only datasets (still far faster than reading
-    /// the whole Parquet file). We don't build a scalar index on
-    /// doc_id yet; that's a future optimization.
-    pub async fn get_by_doc_id(&self, doc_id: &str) -> Result<Option<Row>, String> {
-        use lance::dataset::Dataset;
-
-        let dataset = Dataset::open(&self.path).await.map_err(e)?;
-        let filter = format!("doc_id = '{}'", doc_id.replace('\'', "''"));
-        let mut scanner = dataset.scan();
-        scanner.filter(&filter).map_err(e)?;
-        scanner.limit(Some(1), None).map_err(e)?;
-        let mut stream = scanner.try_into_stream().await.map_err(e)?;
-
-        while let Some(batch) = stream.next().await {
-            let batch = batch.map_err(e)?;
-            if batch.num_rows() == 0 { continue; }
-            return Ok(Some(row_from_batch(&batch, 0)?));
-        }
-        Ok(None)
-    }
-}
-
-// ================= Internal helpers =================
-
-fn e<T: std::fmt::Display>(err: T) -> String { err.to_string() }
-
-/// `file:///abs/path` → `/abs/path`. Leave other URI schemes as-is for
-/// helpers that only work on local paths (dir_size_bytes, remove_dir_all).
-fn strip_file_uri(uri: &str) -> String {
-    uri.strip_prefix("file://").unwrap_or(uri).to_string()
-}
-
-async fn open_or_err(path: &str) -> Result<lance::dataset::Dataset, String> {
-    lance::dataset::Dataset::open(path).await.map_err(e)
-}
-
-fn dir_size_bytes(path: &str) -> u64 {
-    fn recurse(p: &std::path::Path) -> u64 {
-        let Ok(meta) = std::fs::metadata(p) else { return 0; };
-        if meta.is_file() { return meta.len(); }
-        let Ok(entries) = std::fs::read_dir(p) else { return 0; };
-        entries.filter_map(|e| e.ok()).map(|e| recurse(&e.path())).sum()
-    }
-    recurse(std::path::Path::new(path))
-}
-
-fn read_parquet(bytes: &[u8]) -> Result<(Arc<Schema>, Vec<RecordBatch>, usize), String> {
-    use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
-    let builder = ParquetRecordBatchReaderBuilder::try_new(bytes::Bytes::copy_from_slice(bytes))
-        .map_err(e)?;
-    let schema = builder.schema().clone();
-    let reader = builder.build().map_err(e)?;
-    let batches: Vec<RecordBatch> = reader.collect::<Result<_, _>>().map_err(e)?;
-    let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
-    Ok((schema, batches, rows))
-}
-
-fn detect_vector_dims(batches: &[RecordBatch]) -> Result<usize, String> {
-    for batch in batches {
-        let idx = batch.schema().index_of("vector")
-            .map_err(|_| "no 'vector' column".to_string())?;
-        let col = batch.column(idx);
-        if let Some(binary) = col.as_any().downcast_ref::<BinaryArray>() {
-            for i in 0..binary.len() {
-                if !binary.is_null(i) {
-                    return Ok(binary.value(i).len() / 4);
-                }
-            }
-        } else if let Some(fsl) = col.as_any().downcast_ref::<FixedSizeListArray>() {
-            return Ok(fsl.value_length() as usize);
-        }
-    }
-    Err("could not determine vector dimensions".into())
-}
-
-fn convert_to_fixed_size_list(
-    schema: &Arc<Schema>,
-    batches: Vec<RecordBatch>,
-    dims: usize,
-) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
-    let new_fields: Vec<Arc<Field>> = schema
-        .fields()
-        .iter()
-        .map(|f| {
-            if f.name() == "vector" {
-                Arc::new(Field::new(
-                    "vector",
-                    DataType::FixedSizeList(
-                        Arc::new(Field::new("item", DataType::Float32, true)),
-                        dims as i32,
-                    ),
-                    false,
-                ))
-            } else {
-                f.clone()
-            }
-        })
-        .collect();
-    let new_schema = Arc::new(Schema::new(new_fields));
-
-    let mut out = Vec::with_capacity(batches.len());
-    for batch in batches {
-        let vec_idx = batch.schema().index_of("vector").map_err(e)?;
-        let mut new_cols: Vec<ArrayRef> = Vec::with_capacity(batch.num_columns());
-        for (i, col) in batch.columns().iter().enumerate() {
-            if i == vec_idx {
-                if let Some(bin) = col.as_any().downcast_ref::<BinaryArray>() {
-                    new_cols.push(Arc::new(binary_to_fsl(bin, dims)?));
-                } else if col.as_any().is::<FixedSizeListArray>() {
-                    // Already in the right shape — just clone.
-                    new_cols.push(col.clone());
-                } else {
-                    return Err("vector column is neither Binary nor FixedSizeList".into());
-                }
-            } else {
-                new_cols.push(col.clone());
-            }
-        }
-        out.push(RecordBatch::try_new(new_schema.clone(), new_cols).map_err(e)?);
-    }
-    Ok((new_schema, out))
-}
-
-fn binary_to_fsl(bin: &BinaryArray, dims: usize) -> Result<FixedSizeListArray, String> {
-    let n = bin.len();
-    let mut flat: Vec<f32> = Vec::with_capacity(n * dims);
-    for i in 0..n {
-        if bin.is_null(i) {
-            flat.extend(std::iter::repeat(0.0).take(dims));
-            continue;
-        }
-        let b = bin.value(i);
-        if b.len() != dims * 4 {
-            return Err(format!(
-                "row {i}: {} bytes vs expected {} ({} × f32)",
-                b.len(), dims * 4, dims,
-            ));
-        }
-        for c in b.chunks_exact(4) {
-            flat.push(f32::from_le_bytes([c[0], c[1], c[2], c[3]]));
-        }
-    }
-    let values = Float32Array::from(flat);
-    let field = Arc::new(Field::new("item", DataType::Float32, true));
-    FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None).map_err(e)
-}
-
-async fn write_dataset(
-    path: &str,
-    schema: Arc<Schema>,
-    batches: Vec<RecordBatch>,
-) -> Result<(), String> {
-    use lance::dataset::{Dataset, WriteParams};
-    let reader = RecordBatchIterator::new(batches.into_iter().map(Ok), schema);
-    Dataset::write(reader, path, Some(WriteParams::default()))
-        .await.map_err(e)?;
-    Ok(())
-}
-
-fn row_from_batch(batch: &RecordBatch, row: usize) -> Result<Row, String> {
-    let doc_id = batch.column_by_name("doc_id")
-        .and_then(|c| c.as_any().downcast_ref::<StringArray>())
-        .map(|a| a.value(row).to_string())
-        .ok_or_else(|| "missing doc_id".to_string())?;
-    let chunk_text = batch.column_by_name("chunk_text")
-        .and_then(|c| c.as_any().downcast_ref::<StringArray>())
-        .map(|a| a.value(row).to_string())
-        .unwrap_or_default();
-    let source = batch.column_by_name("source")
-        .and_then(|c| c.as_any().downcast_ref::<StringArray>())
-        .and_then(|a| if a.is_null(row) { None } else { Some(a.value(row).to_string()) });
-    let chunk_idx = batch.column_by_name("chunk_idx")
-        .and_then(|c| c.as_any().downcast_ref::<Int32Array>())
-        .and_then(|a| if a.is_null(row) { None } else { Some(a.value(row)) });
-
-    let vec_col = batch.column_by_name("vector")
-        .ok_or_else(|| "no vector column in row fetch".to_string())?;
-    let fsl = vec_col.as_any().downcast_ref::<FixedSizeListArray>()
-        .ok_or_else(|| "vector column is not FixedSizeList".to_string())?;
-    let inner = fsl.value(row);
-    let floats = inner.as_any().downcast_ref::<Float32Array>()
-        .ok_or_else(|| "vector inner not Float32".to_string())?;
-    let mut v = Vec::with_capacity(floats.len());
-    for i in 0..floats.len() { v.push(floats.value(i)); }
-
-    Ok(Row { doc_id, chunk_text, vector: v, source, chunk_idx })
-}

crates/vectord/Cargo.toml

@@ -7,12 +7,6 @@ edition = "2024"
 shared = { path = "../shared" }
 storaged = { path = "../storaged" }
 aibridge = { path = "../aibridge" }
-catalogd = { path = "../catalogd" }
-queryd = { path = "../queryd" }
-futures = { workspace = true }
-# ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps.
-# Public API uses only std types so no version conflict propagates here.
-vectord-lance = { path = "../vectord-lance" }
 tokio = { workspace = true }
 axum = { workspace = true }
 serde = { workspace = true }
@@ -24,5 +18,3 @@ parquet = { workspace = true }
 arrow = { workspace = true }
 chrono = { workspace = true }
 instant-distance = { workspace = true }
-uuid = { workspace = true }
-sha2 = { workspace = true }

crates/vectord/src/agent.rs

@@ -1,789 +0,0 @@
-//! Phase 16.2 + 16.5 — The autotune agent.
-//!
-//! A long-running tokio task that watches the trial journal and
-//! autonomously proposes + runs new HNSW configs. Distinct from
-//! `autotune::run_autotune` which is synchronous (one HTTP call, grid
-//! of trials, done). The agent is the continuous version: it sleeps,
-//! wakes on triggers, proposes configs based on prior trial history,
-//! runs them one at a time, and auto-promotes when it finds an
-//! improvement.
-//!
-//! Design invariants:
-//! - Trials are data (ADR-018). Every proposal reads the journal; every
-//!   attempt appends to it. The journal is the agent's memory.
-//! - One trial at a time. Bounded Ollama load — the agent never fires
-//!   multiple parallel embeddings and respects `cooldown_between_trials_secs`.
-//! - Rate-limited. `max_trials_per_hour` is a hard ceiling so a
-//!   misbehaving proposal function can't saturate the system.
-//! - Never promotes below `min_recall`. Same safety gate as
-//!   `run_autotune` — we will not make the index worse.
-//! - Triggered OR periodic. Ingest enqueues a `DatasetAppended` event
-//!   when a new batch lands; the agent also wakes periodically to keep
-//!   exploring even when nothing changed externally.
-//! - Graceful shutdown via the `stop_tx` signal — the handle's Drop
-//!   doesn't force-kill, but `stop()` requests a clean exit after the
-//!   current trial.
-
-use chrono::{DateTime, Utc};
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::collections::VecDeque;
-use std::sync::Arc;
-use tokio::sync::{Mutex, RwLock, mpsc, oneshot};
-
-use aibridge::client::AiClient;
-use catalogd::registry::Registry as CatalogRegistry;
-
-use crate::embedding_cache::EmbeddingCache;
-use crate::harness;
-use crate::hnsw::HnswStore;
-use crate::index_registry::IndexRegistry;
-use crate::promotion::{PromotionEntry, PromotionRegistry};
-use crate::trial::{HnswConfig, Trial, TrialJournal, TrialMetrics};
-
-// -------- Public-facing types --------
-
-/// Runtime configuration for the agent. Mirrored in shared::config under
-/// `[agent]`. Defaults are conservative — designed to tune slowly in the
-/// background without fighting real workloads for GPU time.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct AgentConfig {
-    /// Master switch. When false, `spawn` returns a handle but the loop
-    /// doesn't run.
-    pub enabled: bool,
-    /// Periodic wake-up — even if the trigger queue is empty, every N
-    /// seconds the agent picks an index with trials and proposes one
-    /// more config. Keeps exploration alive on idle indexes.
-    pub cycle_interval_secs: u64,
-    /// Minimum gap between two trials on the SAME index. Prevents the
-    /// agent from hammering Ollama when a hot index has many pending
-    /// triggers in a row.
-    pub cooldown_between_trials_secs: u64,
-    /// Below this recall, a proposal is never promoted — even if it
-    /// beats the champion on latency.
-    pub min_recall: f32,
-    /// Budget cap: hard ceiling on trials per hour across all indexes.
-    /// When hit, the agent idles until the hour window rolls.
-    pub max_trials_per_hour: u32,
-}
-
-impl Default for AgentConfig {
-    fn default() -> Self {
-        Self {
-            enabled: false,            // opt-in — don't auto-start until J turns it on
-            cycle_interval_secs: 60,
-            cooldown_between_trials_secs: 30,
-            min_recall: 0.9,
-            max_trials_per_hour: 30,
-        }
-    }
-}
-
-/// What caused the agent to look at a particular index. Recorded in the
-/// trial's note field so we can tell "new data arrived" trials from
-/// "periodic exploration" trials in the journal.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub enum TriggerReason {
-    /// Ingest just appended to a dataset that has attached HNSW indexes.
-    DatasetAppended { dataset: String },
-    /// A human or another agent hit `/agent/enqueue/{index}`.
-    Manual,
-    /// Periodic wake — no external event, just keep exploring.
-    Periodic,
-}
-
-/// One unit of work for the agent.
-#[derive(Debug, Clone)]
-pub struct TriggerEvent {
-    pub index_name: String,
-    pub reason: TriggerReason,
-    pub enqueued_at: DateTime<Utc>,
-}
-
-impl TriggerEvent {
-    pub fn manual(index_name: impl Into<String>) -> Self {
-        Self { index_name: index_name.into(), reason: TriggerReason::Manual, enqueued_at: Utc::now() }
-    }
-    pub fn dataset_appended(index_name: impl Into<String>, dataset: impl Into<String>) -> Self {
-        Self {
-            index_name: index_name.into(),
-            reason: TriggerReason::DatasetAppended { dataset: dataset.into() },
-            enqueued_at: Utc::now(),
-        }
-    }
-    pub fn periodic(index_name: impl Into<String>) -> Self {
-        Self { index_name: index_name.into(), reason: TriggerReason::Periodic, enqueued_at: Utc::now() }
-    }
-}
-
-/// Observable snapshot of the agent's state — what `/agent/status` returns.
-#[derive(Debug, Clone, Serialize)]
-pub struct AgentStatus {
-    pub running: bool,
-    pub config: AgentConfig,
-    pub queue_depth: usize,
-    pub trials_run: u64,
-    pub promotions: u64,
-    pub trials_in_last_hour: u32,
-    pub last_event: Option<AgentEvent>,
-    pub started_at: Option<DateTime<Utc>>,
-}
-
-/// Last thing that happened — useful for "why didn't it do anything?" debugging.
-#[derive(Debug, Clone, Serialize)]
-pub struct AgentEvent {
-    pub at: DateTime<Utc>,
-    pub kind: String,      // "trial_completed" | "promoted" | "skipped_rate_limit" | etc
-    pub index_name: Option<String>,
-    pub detail: String,
-}
-
-/// Handle returned by `spawn`. Holds the trigger sender + shared status +
-/// stop signal.
-#[derive(Clone)]
-pub struct AgentHandle {
-    trigger_tx: mpsc::Sender<TriggerEvent>,
-    inner: Arc<AgentInner>,
-}
-
-struct AgentInner {
-    status: RwLock<AgentStatus>,
-    stop_tx: Mutex<Option<oneshot::Sender<()>>>,
-    queue_len: Mutex<usize>,     // mirror of the channel capacity — for status reporting
-    recent_trials: Mutex<VecDeque<DateTime<Utc>>>,  // ring of recent trial timestamps for rate limit
-}
-
-impl AgentHandle {
-    /// Enqueue a trigger. Returns Err if the agent isn't running or the
-    /// queue is full (backpressure — dropping events is correct here
-    /// since periodic exploration will pick up the slack).
-    pub async fn enqueue(&self, event: TriggerEvent) -> Result<(), String> {
-        self.trigger_tx.try_send(event).map_err(|e| format!("enqueue: {e}"))?;
-        let mut guard = self.inner.queue_len.lock().await;
-        *guard = guard.saturating_add(1);
-        // Update queue_depth in status for observability.
-        let mut s = self.inner.status.write().await;
-        s.queue_depth = *guard;
-        Ok(())
-    }
-
-    pub async fn status(&self) -> AgentStatus {
-        let mut s = self.inner.status.read().await.clone();
-        // Refresh rate-limit window from ring buffer.
-        let cutoff = Utc::now() - chrono::Duration::hours(1);
-        let ring = self.inner.recent_trials.lock().await;
-        s.trials_in_last_hour = ring.iter().filter(|t| **t >= cutoff).count() as u32;
-        s
-    }
-
-    /// Request a graceful stop. Returns immediately — the loop exits
-    /// after its current trial completes.
-    pub async fn stop(&self) -> bool {
-        let mut guard = self.inner.stop_tx.lock().await;
-        if let Some(tx) = guard.take() {
-            let _ = tx.send(());
-            true
-        } else {
-            false
-        }
-    }
-}
-
-// -------- Agent state holder --------
-
-/// Everything the agent needs to run a trial. Mirrors the fields of
-/// `VectorState` the agent actually uses. Kept separate so the service
-/// layer builds it explicitly — no clone of unneeded state.
-#[derive(Clone)]
-pub struct AgentDeps {
-    pub store: Arc<dyn ObjectStore>,
-    pub ai_client: AiClient,
-    pub catalog: CatalogRegistry,
-    pub index_registry: IndexRegistry,
-    pub hnsw_store: HnswStore,
-    pub embedding_cache: EmbeddingCache,
-    pub trial_journal: TrialJournal,
-    pub promotion_registry: PromotionRegistry,
-    pub harness_store: crate::harness::HarnessStore,
-}
-
-// -------- Spawn --------
-
-/// Start the agent loop in a background tokio task. Returns a handle
-/// the caller uses to enqueue events and read status.
-pub fn spawn(config: AgentConfig, deps: AgentDeps) -> AgentHandle {
-    let (trigger_tx, trigger_rx) = mpsc::channel::<TriggerEvent>(256);
-    let (stop_tx, stop_rx) = oneshot::channel::<()>();
-
-    let status = AgentStatus {
-        running: config.enabled,
-        config: config.clone(),
-        queue_depth: 0,
-        trials_run: 0,
-        promotions: 0,
-        trials_in_last_hour: 0,
-        last_event: None,
-        started_at: if config.enabled { Some(Utc::now()) } else { None },
-    };
-
-    let inner = Arc::new(AgentInner {
-        status: RwLock::new(status),
-        stop_tx: Mutex::new(Some(stop_tx)),
-        queue_len: Mutex::new(0),
-        recent_trials: Mutex::new(VecDeque::with_capacity(64)),
-    });
-
-    if config.enabled {
-        tracing::info!(
-            "autotune agent started (cycle={}s, cooldown={}s, cap={}/hr, min_recall={})",
-            config.cycle_interval_secs, config.cooldown_between_trials_secs,
-            config.max_trials_per_hour, config.min_recall,
-        );
-        let loop_inner = inner.clone();
-        let loop_deps = deps.clone();
-        let loop_config = config.clone();
-        tokio::spawn(async move {
-            run_loop(loop_config, loop_deps, trigger_rx, stop_rx, loop_inner).await;
-        });
-    } else {
-        // Agent disabled — still drain the channel so sends don't back up.
-        tokio::spawn(async move {
-            let mut rx = trigger_rx;
-            while rx.recv().await.is_some() {}
-        });
-        tracing::info!("autotune agent configured but disabled (set [agent].enabled=true)");
-    }
-
-    AgentHandle { trigger_tx, inner }
-}
-
-// -------- Main loop --------
-
-async fn run_loop(
-    config: AgentConfig,
-    deps: AgentDeps,
-    mut trigger_rx: mpsc::Receiver<TriggerEvent>,
-    mut stop_rx: oneshot::Receiver<()>,
-    inner: Arc<AgentInner>,
-) {
-    let mut periodic = tokio::time::interval(std::time::Duration::from_secs(config.cycle_interval_secs));
-    // First tick fires immediately — skip it so we don't double-fire on startup.
-    periodic.tick().await;
-
-    loop {
-        let event = tokio::select! {
-            _ = &mut stop_rx => {
-                tracing::info!("autotune agent: stop signal received");
-                let mut s = inner.status.write().await;
-                s.running = false;
-                return;
-            }
-            maybe = trigger_rx.recv() => match maybe {
-                Some(ev) => {
-                    let mut guard = inner.queue_len.lock().await;
-                    *guard = guard.saturating_sub(1);
-                    let mut s = inner.status.write().await;
-                    s.queue_depth = *guard;
-                    ev
-                }
-                None => {
-                    tracing::info!("autotune agent: trigger channel closed");
-                    return;
-                }
-            },
-            _ = periodic.tick() => {
-                // Periodic wake — pick an index with existing trials.
-                // If nothing's been tuned yet there's nothing to propose.
-                match pick_periodic_target(&deps).await {
-                    Some(idx) => TriggerEvent::periodic(idx),
-                    None => continue,
-                }
-            }
-        };
-
-        // Rate limit: count recent trials, skip if over budget.
-        if over_rate_limit(&inner, config.max_trials_per_hour).await {
-            record_event(&inner, "skipped_rate_limit", Some(&event.index_name),
-                format!("hit cap of {}/hour", config.max_trials_per_hour)).await;
-            continue;
-        }
-
-        // Per-index cooldown.
-        if cooling_down(&inner, &event.index_name, config.cooldown_between_trials_secs).await {
-            record_event(&inner, "skipped_cooldown", Some(&event.index_name),
-                format!("last trial too recent (<{}s)", config.cooldown_between_trials_secs)).await;
-            continue;
-        }
-
-        // Run one trial.
-        match run_one_cycle(&event, &deps, config.min_recall).await {
-            Ok(outcome) => {
-                mark_recent_trial(&inner).await;
-                {
-                    let mut s = inner.status.write().await;
-                    s.trials_run += 1;
-                    if outcome.promoted { s.promotions += 1; }
-                }
-                record_event(&inner, if outcome.promoted { "promoted" } else { "trial_completed" },
-                    Some(&event.index_name),
-                    format!("config=ec{}/es{} recall={:.3} p50={:.0}us {}",
-                        outcome.trial.config.ef_construction,
-                        outcome.trial.config.ef_search,
-                        outcome.trial.metrics.recall_at_k,
-                        outcome.trial.metrics.search_latency_p50_us,
-                        if outcome.promoted { "★ PROMOTED" } else { "" })).await;
-            }
-            Err(e) => {
-                record_event(&inner, "trial_error", Some(&event.index_name), e).await;
-            }
-        }
-    }
-}
-
-/// Result of one cycle — ran a trial, maybe promoted it.
-struct CycleOutcome {
-    trial: Trial,
-    promoted: bool,
-}
-
-/// Core cycle: propose → build → bench → record → maybe promote.
-async fn run_one_cycle(
-    event: &TriggerEvent,
-    deps: &AgentDeps,
-    min_recall: f32,
-) -> Result<CycleOutcome, String> {
-    // Read history.
-    let history = deps.trial_journal.list(&event.index_name).await
-        .map_err(|e| format!("read journal: {e}"))?;
-
-    // Bootstrap mode: first ever visit to this index. Auto-generate a
-    // harness (if missing) and seed with the default config — the Phase 15
-    // known-good ec=80/es=30. Subsequent visits go through the proposer.
-    let is_bootstrap = history.is_empty();
-
-    // Current champion (if any) is the promoted config.
-    let champion = deps.promotion_registry.get_current(&event.index_name).await;
-    let champion_trial = champion.as_ref().and_then(|p| {
-        history.iter().find(|t| t.id == p.trial_id).cloned()
-    });
-
-    let (next_config, harness_name) = if is_bootstrap {
-        let name = ensure_auto_harness(&event.index_name, deps).await?;
-        (HnswConfig::default(), name)
-    } else {
-        let Some(cfg) = propose_next_config(&history, champion_trial.as_ref()) else {
-            return Err("proposer returned None — search space exhausted".into());
-        };
-        // Validate bounds defensively.
-        if !(10..=400).contains(&cfg.ef_construction) {
-            return Err(format!("proposed ef_construction={} out of bounds", cfg.ef_construction));
-        }
-        if !(10..=200).contains(&cfg.ef_search) {
-            return Err(format!("proposed ef_search={} out of bounds", cfg.ef_search));
-        }
-        // A future refinement: remember per-index "canonical harness" on
-        // the index metadata. For now: latest wins.
-        let hname = history.last().unwrap().eval_set.clone();
-        (cfg, hname)
-    };
-    let mut harness_set = deps.harness_store.load_for_index(&event.index_name, &harness_name).await
-        .map_err(|e| format!("load harness '{harness_name}': {e}"))?;
-
-    let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await
-        .map_err(|e| format!("embeddings: {e}"))?;
-
-    if !harness_set.ground_truth_built {
-        harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await
-            .map_err(|e| format!("ground truth: {e}"))?;
-        deps.harness_store.save(&harness_set).await.ok();
-    }
-
-    // Build + bench.
-    let trial_id = Trial::new_id();
-    let slot = format!("{}__{}", event.index_name, trial_id);
-    let build = deps.hnsw_store
-        .build_index_with_config(&slot, (*embeddings).clone(), &next_config)
-        .await?;
-
-    let query_vectors: Vec<Vec<f32>> = harness_set.queries
-        .iter().filter_map(|q| q.query_embedding.clone()).collect();
-    let bench = deps.hnsw_store.bench_search(&slot, &query_vectors, harness_set.k).await?;
-
-    let mut recalls = Vec::with_capacity(harness_set.queries.len());
-    for (q, hits) in harness_set.queries.iter().zip(bench.retrieved.iter()) {
-        if let Some(gt) = &q.ground_truth {
-            recalls.push(harness::recall_at_k(hits, gt, harness_set.k));
-        }
-    }
-    let mean_recall = if recalls.is_empty() { 0.0 } else {
-        recalls.iter().sum::<f32>() / recalls.len() as f32
-    };
-
-    let mut lats = bench.latencies_us.clone();
-    lats.sort_unstable_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
-    let p = |pct: f32| -> f32 {
-        if lats.is_empty() { return 0.0; }
-        let idx = ((lats.len() as f32 - 1.0) * pct).round() as usize;
-        lats[idx.min(lats.len() - 1)]
-    };
-
-    let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
-    let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
-
-    let note = {
-        let base = match &event.reason {
-            TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
-            TriggerReason::Manual => "agent: manual".to_string(),
-            TriggerReason::Periodic => "agent: periodic".to_string(),
-        };
-        if is_bootstrap { format!("{base} bootstrap") } else { base }
-    };
-
-    let trial = Trial {
-        id: trial_id,
-        index_name: event.index_name.clone(),
-        eval_set: harness_set.name.clone(),
-        config: next_config.clone(),
-        metrics: TrialMetrics {
-            build_time_secs: build.build_time_secs,
-            search_latency_p50_us: p(0.50),
-            search_latency_p95_us: p(0.95),
-            search_latency_p99_us: p(0.99),
-            recall_at_k: mean_recall,
-            memory_bytes,
-            vectors: build.vectors,
-            eval_queries: harness_set.queries.len(),
-            brute_force_latency_us: 0.0,
-        },
-        created_at: Utc::now(),
-        note: Some(note),
-    };
-    deps.trial_journal.append(&trial).await.ok();
-    deps.hnsw_store.drop(&slot).await;
-
-    // Promotion decision: the new trial must meet recall gate AND beat
-    // the current champion (higher recall OR same recall + lower p50).
-    let promoted = if trial.metrics.recall_at_k < min_recall {
-        false
-    } else {
-        let beats = match &champion_trial {
-            None => true,  // no champion yet — anything passing the gate wins
-            Some(c) => beats_champion(&trial, c),
-        };
-        if beats {
-            let entry = PromotionEntry {
-                config: trial.config.clone(),
-                trial_id: trial.id.clone(),
-                promoted_at: Utc::now(),
-                promoted_by: "agent".to_string(),
-                note: Some(format!(
-                    "auto-promote: recall={:.3} p50={:.0}us (was {:.3}/{:.0}us)",
-                    trial.metrics.recall_at_k, trial.metrics.search_latency_p50_us,
-                    champion_trial.as_ref().map(|t| t.metrics.recall_at_k).unwrap_or(0.0),
-                    champion_trial.as_ref().map(|t| t.metrics.search_latency_p50_us).unwrap_or(0.0),
-                )),
-            };
-            deps.promotion_registry.promote(&event.index_name, entry).await.is_ok()
-        } else {
-            false
-        }
-    };
-
-    Ok(CycleOutcome { trial, promoted })
-}
-
-/// Champion-beat test: strictly higher recall, OR equal recall with
-/// lower p50. Same rule as autotune::pick_winner — kept consistent so the
-/// agent and the synchronous autotune agree on what "better" means.
-fn beats_champion(candidate: &Trial, champion: &Trial) -> bool {
-    if candidate.metrics.recall_at_k > champion.metrics.recall_at_k {
-        return true;
-    }
-    if (candidate.metrics.recall_at_k - champion.metrics.recall_at_k).abs() < 1e-4
-        && candidate.metrics.search_latency_p50_us < champion.metrics.search_latency_p50_us {
-        return true;
-    }
-    false
-}
-
-/// Propose the next HnswConfig given trial history and the current
-/// champion.
-///
-/// ============================================================
-/// J: THIS IS YOURS TO IMPLEMENT
-/// ============================================================
-///
-/// Inputs:
-/// - `history`: every trial ever run on this index, oldest first
-/// - `champion`: the currently-promoted trial, if any
-///
-/// Output:
-/// - `Some(HnswConfig)` with the config to try next
-/// - `None` if you decide there's nothing worth trying (agent logs
-///   "search space exhausted" and moves on)
-///
-/// Hard bounds the caller enforces: ef_construction ∈ [10, 400],
-/// ef_search ∈ [10, 200]. Stay inside those — configs outside get
-/// rejected and count as a wasted cycle.
-///
-/// Design options (pick one, or mix):
-///
-/// 1. LOCAL REFINEMENT (exploit-heavy):
-///    Sample near champion ± small delta. Converges fast, risks local
-///    minima. Good for "we know roughly where the sweet spot is."
-///
-/// 2. ε-GREEDY (mixed):
-///    With prob ε, random sample from full bounds (explore). Otherwise
-///    refinement around champion (exploit). ε=0.2 is a reasonable start.
-///    Good for long-running tune with no prior knowledge.
-///
-/// 3. COARSE→FINE (annealed):
-///    First N trials: wide random. Then shrink the neighborhood around
-///    champion as more trials accumulate. Mimics simulated annealing.
-///
-/// 4. DEDUP-AWARE:
-///    Whatever strategy, skip configs already in history. Prevents the
-///    agent from re-running the same (ec, es) pair twice.
-///
-/// A simple starter implementation is provided below (local refinement
-/// + dedup). Replace with your preferred strategy.
-pub fn propose_next_config(history: &[Trial], champion: Option<&Trial>) -> Option<HnswConfig> {
-    // ε-greedy around the champion, dedup-aware.
-    //
-    // - With probability ε (≈0.25), sample a random config from the full
-    //   bounds. Keeps exploration alive so we don't get stuck hill-climbing
-    //   one axis.
-    // - Otherwise: perturb the champion symmetrically on BOTH axes (not
-    //   just +20 / +40 like the starter did). Prefers small moves first
-    //   so recall stays near the current level.
-    // - Always skip configs already in history — no point re-running.
-    // - Deterministic per-history: RNG is seeded from history length so
-    //   the same journal state always proposes the same next config.
-    //   Makes tests + offline replay reproducible.
-
-    let base = champion
-        .map(|t| t.config.clone())
-        .or_else(|| history.last().map(|t| t.config.clone()))
-        .unwrap_or_default();
-
-    let tried = |ec: usize, es: usize| -> bool {
-        history.iter().any(|t|
-            t.config.ef_construction == ec && t.config.ef_search == es
-        )
-    };
-    let clamp = |ec: i32, es: i32| -> (usize, usize) {
-        (ec.clamp(10, 400) as usize, es.clamp(10, 200) as usize)
-    };
-
-    // Tiny xorshift — no rand crate dep. Seeded from history length so the
-    // proposer is deterministic for a given journal state.
-    let mut rng = (history.len() as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
-    let mut next_u = || {
-        rng ^= rng << 13; rng ^= rng >> 7; rng ^= rng << 17;
-        rng
-    };
-
-    for _attempt in 0..32 {
-        let explore = (next_u() % 100) < 25;  // ε = 0.25
-        let (ec, es) = if explore {
-            clamp(10 + (next_u() % 391) as i32, 10 + (next_u() % 191) as i32)
-        } else {
-            // Symmetric perturbation — signed steps on both axes.
-            let dec = [-40, -20, -10, 10, 20, 40][(next_u() % 6) as usize];
-            let des = [-20, -10, -5, 5, 10, 20][(next_u() % 6) as usize];
-            clamp(base.ef_construction as i32 + dec, base.ef_search as i32 + des)
-        };
-        if !tried(ec, es) {
-            return Some(HnswConfig { ef_construction: ec, ef_search: es, seed: Some(42) });
-        }
-    }
-    None  // 32 attempts all landed on duplicates — likely saturated
-}
-
-// -------- Helpers --------
-
-/// Minimum vectors for an index to be worth auto-tuning. Below this,
-/// HNSW's win over brute-force is too small to matter and trial budget
-/// is better spent on bigger indexes.
-const AUTOTUNE_MIN_VECTORS: usize = 1_000;
-
-/// Find an index to poke on a periodic wake. Strategy: least-recently-tuned
-/// wins — pick the index (from `IndexRegistry`, not just promoted ones)
-/// whose most-recent trial is oldest, so trial budget spreads across the
-/// whole portfolio. Never-trialed indexes sort first (None < Some), which
-/// is exactly what we want: bootstrap them on their first visit.
-///
-/// Why not "most recently promoted" (the original strategy): a converged
-/// index like `threat_intel_v1` sits at recall=1.0 on a minimal graph and
-/// can't be improved — but it was always the freshest promotion, so the
-/// agent burned every trial on it while 40+ other indexes got zero
-/// attention. Rotating by last-trial-time fixes that without any explicit
-/// convergence detection.
-///
-/// Why IndexRegistry instead of PromotionRegistry: on a fresh system only
-/// a handful of indexes have ever been promoted, so promotion-based
-/// picking starves new ones. IndexRegistry lists every live index the
-/// platform knows about; filtering by `AUTOTUNE_MIN_VECTORS` keeps the
-/// proposer off indexes where the graph config doesn't matter.
-///
-/// Cost: N extra journal reads per periodic tick, where N = eligible
-/// indexes. Fine at dozens; if the portfolio grows into thousands, cache
-/// `last_trial_at` on the IndexMeta.
-async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> {
-    let candidates: Vec<String> = deps
-        .index_registry
-        .list(None, None)
-        .await
-        .into_iter()
-        .filter(|m| m.chunk_count >= AUTOTUNE_MIN_VECTORS)
-        .map(|m| m.index_name)
-        .collect();
-    if candidates.is_empty() { return None; }
-
-    let mut scored: Vec<(String, Option<DateTime<Utc>>)> = Vec::with_capacity(candidates.len());
-    for name in candidates {
-        let last_trial_at = deps
-            .trial_journal
-            .list(&name)
-            .await
-            .ok()
-            .and_then(|trials| trials.into_iter().map(|t| t.created_at).max());
-        scored.push((name, last_trial_at));
-    }
-
-    scored.into_iter().min_by_key(|(_, last)| *last).map(|(n, _)| n)
-}
-
-/// On the first visit to an index, load or synthesize an eval harness so
-/// bootstrap trials have something to measure recall against. Returns the
-/// harness name. Synthetic harnesses sample 20 chunks and use their first
-/// 200 chars as self-queries — the ground truth is computed by brute-force
-/// cosine, so recall numbers are real (a config that finds the chunk in
-/// top-k counts as a hit).
-async fn ensure_auto_harness(index_name: &str, deps: &AgentDeps) -> Result<String, String> {
-    let name = format!("{index_name}_auto");
-
-    if deps.harness_store.load_for_index(index_name, &name).await.is_ok() {
-        return Ok(name);
-    }
-
-    let embeddings = deps
-        .embedding_cache
-        .get_or_load(index_name)
-        .await
-        .map_err(|e| format!("load embeddings for auto-harness: {e}"))?;
-    if embeddings.is_empty() {
-        return Err(format!("index '{index_name}' has no embeddings — cannot bootstrap"));
-    }
-
-    let mut eval = harness::synthetic_from_chunks(&name, index_name, &embeddings, 20, 10);
-    harness::compute_ground_truth(&mut eval, &embeddings, &deps.ai_client)
-        .await
-        .map_err(|e| format!("auto-harness ground truth: {e}"))?;
-    deps.harness_store.save(&eval).await
-        .map_err(|e| format!("save auto-harness: {e}"))?;
-
-    tracing::info!(
-        "agent: bootstrapped harness '{}' for index '{}' (20 self-queries, k=10)",
-        name, index_name
-    );
-    Ok(name)
-}
-
-async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool {
-    let cutoff = Utc::now() - chrono::Duration::hours(1);
-    let ring = inner.recent_trials.lock().await;
-    ring.iter().filter(|t| **t >= cutoff).count() as u32 >= cap
-}
-
-async fn cooling_down(inner: &Arc<AgentInner>, _index: &str, cooldown_secs: u64) -> bool {
-    // Minimal impl: gate on global most-recent trial rather than per-index.
-    // Per-index cooldown would be easy to add — keep a HashMap<String, DateTime>
-    // — but for Phase 16.2 MVP, global is fine. Ollama is the shared resource.
-    let ring = inner.recent_trials.lock().await;
-    if let Some(last) = ring.back() {
-        let since = Utc::now().signed_duration_since(*last);
-        return since < chrono::Duration::seconds(cooldown_secs as i64);
-    }
-    false
-}
-
-async fn mark_recent_trial(inner: &Arc<AgentInner>) {
-    let mut ring = inner.recent_trials.lock().await;
-    ring.push_back(Utc::now());
-    // Keep bounded.
-    while ring.len() > 256 {
-        ring.pop_front();
-    }
-}
-
-async fn record_event(
-    inner: &Arc<AgentInner>,
-    kind: &str,
-    index: Option<&str>,
-    detail: String,
-) {
-    tracing::info!("agent: {} {}{}", kind,
-        index.map(|i| format!("[{i}] ")).unwrap_or_default(), detail);
-    let mut s = inner.status.write().await;
-    s.last_event = Some(AgentEvent {
-        at: Utc::now(),
-        kind: kind.to_string(),
-        index_name: index.map(String::from),
-        detail,
-    });
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn mk_trial(ec: usize, es: usize, recall: f32, p50: f32) -> Trial {
-        Trial {
-            id: format!("t-{ec}-{es}"),
-            index_name: "test".into(),
-            eval_set: "eval".into(),
-            config: HnswConfig { ef_construction: ec, ef_search: es, seed: Some(42) },
-            metrics: TrialMetrics {
-                build_time_secs: 1.0,
-                search_latency_p50_us: p50,
-                search_latency_p95_us: p50 * 1.5,
-                search_latency_p99_us: p50 * 2.0,
-                recall_at_k: recall,
-                memory_bytes: 0, vectors: 1000, eval_queries: 10,
-                brute_force_latency_us: 100.0,
-            },
-            created_at: Utc::now(),
-            note: None,
-        }
-    }
-
-    #[test]
-    fn propose_skips_duplicates() {
-        let hist = vec![
-            mk_trial(80, 30, 1.0, 500.0),
-            mk_trial(100, 30, 1.0, 520.0),   // ec+20
-        ];
-        let next = propose_next_config(&hist, Some(&hist[0])).unwrap();
-        // ec+20 is taken, so the proposer should skip it.
-        assert!(next.ef_construction != 100 || next.ef_search != 30);
-    }
-
-    #[test]
-    fn beats_champion_strict_recall() {
-        let champ = mk_trial(80, 30, 0.95, 500.0);
-        let better_recall = mk_trial(80, 30, 0.99, 600.0);
-        let worse_recall = mk_trial(80, 30, 0.90, 100.0);
-        assert!(beats_champion(&better_recall, &champ));
-        assert!(!beats_champion(&worse_recall, &champ));
-    }
-
-    #[test]
-    fn beats_champion_same_recall_lower_latency() {
-        let champ = mk_trial(80, 30, 1.0, 500.0);
-        let faster = mk_trial(60, 30, 1.0, 400.0);
-        let slower = mk_trial(60, 30, 1.0, 600.0);
-        assert!(beats_champion(&faster, &champ));
-        assert!(!beats_champion(&slower, &champ));
-    }
-}

crates/vectord/src/autotune.rs

@@ -1,332 +0,0 @@
-//! Phase 16: Autotune — the agent loop for HNSW tuning.
-//!
-//! Given an index and a harness, run a grid of (ef_construction,
-//! ef_search) trials, pick the Pareto winner, and promote it.
-//!
-//! This is deliberately NOT a sophisticated optimizer — random-grid
-//! search is good enough when the config space is small (a dozen
-//! combinations) and the work per trial is seconds, not hours. Think
-//! of it as a "smart operator" rather than Bayesian optimization.
-//!
-//! Score: we want high recall and low p50 latency. The function below
-//! picks the Pareto-dominating trial, breaking ties by prefer-lower-
-//! build-time (cheaper to rebuild on data changes).
-//!
-//! Safeguards (PRD risk mitigation):
-//! - Never promote a trial with recall < `min_recall` (default 0.9)
-//! - Hard config bounds: ef_construction ∈ [10, 400], ef_search ∈ [10, 200]
-//! - Concurrency cap: one autotune per index at a time (tracked by
-//!   JobTracker — caller is responsible for not firing in parallel
-//!   for the same index)
-
-use chrono::Utc;
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::sync::Arc;
-
-use aibridge::client::AiClient;
-use catalogd::registry::Registry as CatalogRegistry;
-
-use crate::embedding_cache::EmbeddingCache;
-use crate::harness::{self, HarnessStore};
-use crate::hnsw::HnswStore;
-use crate::index_registry::IndexRegistry;
-use crate::jobs::JobTracker;
-use crate::promotion::{PromotionEntry, PromotionRegistry};
-use crate::trial::{HnswConfig, Trial, TrialJournal, TrialMetrics};
-
-#[derive(Debug, Clone, Deserialize)]
-pub struct AutotuneRequest {
-    pub index_name: String,
-    pub harness: String,
-    /// Minimum recall a trial must achieve to be eligible for promotion.
-    /// Default 0.9 — below that, the tuning is not producing useful
-    /// answers. Never promote a bad index.
-    #[serde(default = "default_min_recall")]
-    pub min_recall: f32,
-    /// Override the default config grid (rarely needed). When omitted,
-    /// a small curated grid runs.
-    #[serde(default)]
-    pub grid: Option<Vec<HnswConfig>>,
-}
-
-fn default_min_recall() -> f32 { 0.9 }
-
-#[derive(Debug, Clone, Serialize)]
-pub struct AutotuneResult {
-    pub index_name: String,
-    pub trials_run: usize,
-    pub trials_eligible: usize,
-    pub winner: Option<WinnerReport>,
-    pub promoted: bool,
-    pub duration_secs: f32,
-    pub skipped: Vec<String>,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct WinnerReport {
-    pub trial_id: String,
-    pub config: HnswConfig,
-    pub recall_at_k: f32,
-    pub search_latency_p50_us: f32,
-    pub build_time_secs: f32,
-}
-
-/// Default exploration grid. Covers the practical range of settings
-/// from "fast build, OK recall" to "thorough build, great recall".
-fn default_grid() -> Vec<HnswConfig> {
-    vec![
-        HnswConfig { ef_construction: 20, ef_search: 30, seed: Some(42) },
-        HnswConfig { ef_construction: 40, ef_search: 30, seed: Some(42) },
-        HnswConfig { ef_construction: 80, ef_search: 30, seed: Some(42) },
-        HnswConfig { ef_construction: 80, ef_search: 60, seed: Some(42) },
-        HnswConfig { ef_construction: 160, ef_search: 30, seed: Some(42) },
-    ]
-}
-
-/// Config-space bounds. Rejects anything outside these before running a
-/// trial — defends against an agent getting creative with absurd values.
-fn sanitize_grid(grid: Vec<HnswConfig>) -> (Vec<HnswConfig>, Vec<String>) {
-    let mut ok = Vec::new();
-    let mut rejected = Vec::new();
-    for cfg in grid {
-        if !(10..=400).contains(&cfg.ef_construction) {
-            rejected.push(format!(
-                "ef_construction={} out of bounds [10,400]", cfg.ef_construction,
-            ));
-            continue;
-        }
-        if !(10..=200).contains(&cfg.ef_search) {
-            rejected.push(format!(
-                "ef_search={} out of bounds [10,200]", cfg.ef_search,
-            ));
-            continue;
-        }
-        ok.push(cfg);
-    }
-    (ok, rejected)
-}
-
-/// Pareto winner: highest recall, then lowest p50 latency among
-/// eligible trials (recall >= min_recall).
-fn pick_winner(trials: &[Trial], min_recall: f32) -> Option<&Trial> {
-    trials
-        .iter()
-        .filter(|t| t.metrics.recall_at_k >= min_recall)
-        .min_by(|a, b| {
-            // Compare by (−recall, p50_latency) so minimum is the
-            // best trial: higher recall wins, then lower latency.
-            let a_key = (-a.metrics.recall_at_k, a.metrics.search_latency_p50_us);
-            let b_key = (-b.metrics.recall_at_k, b.metrics.search_latency_p50_us);
-            a_key.partial_cmp(&b_key).unwrap_or(std::cmp::Ordering::Equal)
-        })
-}
-
-/// Run the autotune loop. Synchronous — caller can wrap in tokio::spawn
-/// to make it fire-and-forget. Uses the existing trial flow so every
-/// autotune step lands in the trial journal.
-#[allow(clippy::too_many_arguments)]
-pub async fn run_autotune(
-    req: AutotuneRequest,
-    _store: &Arc<dyn ObjectStore>,
-    catalog: &CatalogRegistry,
-    ai_client: &AiClient,
-    embedding_cache: &EmbeddingCache,
-    hnsw_store: &HnswStore,
-    index_registry: &IndexRegistry,
-    trial_journal: &TrialJournal,
-    promotion_registry: &PromotionRegistry,
-    harness_store: &HarnessStore,
-    _job_tracker: &JobTracker,
-) -> Result<AutotuneResult, String> {
-    let t0 = std::time::Instant::now();
-
-    // Sanity: the index has to exist.
-    if index_registry.get(&req.index_name).await.is_none() {
-        return Err(format!("index not found: {}", req.index_name));
-    }
-    let _ = catalog; // reserved for future audit emission
-
-    // Load the harness once, compute ground truth once. Harness resolves
-    // to the index's bucket via HarnessStore, with primary as fallback for
-    // pre-federation evals.
-    let mut harness_set = harness_store
-        .load_for_index(&req.index_name, &req.harness)
-        .await
-        .map_err(|e| format!("load harness: {e}"))?;
-
-    let embeddings = embedding_cache
-        .get_or_load(&req.index_name)
-        .await
-        .map_err(|e| format!("embeddings: {e}"))?;
-
-    if !harness_set.ground_truth_built {
-        harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client)
-            .await
-            .map_err(|e| format!("ground truth: {e}"))?;
-        harness_store.save(&harness_set).await.ok();
-    }
-
-    let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid));
-    if grid.is_empty() {
-        return Err("all configs rejected by bounds check".into());
-    }
-
-    tracing::info!(
-        "autotune '{}': {} trial configs, min_recall={}",
-        req.index_name, grid.len(), req.min_recall,
-    );
-
-    // Run each trial through the existing pipeline so every run lands
-    // in the trial journal and contributes to the long-term tuning
-    // memory. This is the llms3.com-style "trials are data" pattern.
-    let mut run_trials: Vec<Trial> = Vec::with_capacity(grid.len());
-    for config in &grid {
-        let trial = run_single_trial(
-            &req.index_name,
-            &harness_set,
-            &embeddings,
-            config,
-            hnsw_store,
-            trial_journal,
-            ai_client,
-        ).await;
-        match trial {
-            Ok(t) => run_trials.push(t),
-            Err(e) => tracing::warn!("autotune: trial ec={} es={} failed: {e}",
-                config.ef_construction, config.ef_search),
-        }
-    }
-
-    let trials_eligible = run_trials
-        .iter()
-        .filter(|t| t.metrics.recall_at_k >= req.min_recall)
-        .count();
-
-    let winner = pick_winner(&run_trials, req.min_recall);
-    let mut promoted = false;
-
-    let winner_report = if let Some(w) = winner {
-        let entry = PromotionEntry {
-            config: w.config.clone(),
-            trial_id: w.id.clone(),
-            promoted_at: Utc::now(),
-            promoted_by: "autotune".to_string(),
-            note: Some(format!(
-                "autotune winner: recall={:.3} p50={:.0}us",
-                w.metrics.recall_at_k, w.metrics.search_latency_p50_us,
-            )),
-        };
-        if let Err(e) = promotion_registry.promote(&req.index_name, entry).await {
-            tracing::warn!("autotune: promote failed: {e}");
-        } else {
-            promoted = true;
-        }
-        Some(WinnerReport {
-            trial_id: w.id.clone(),
-            config: w.config.clone(),
-            recall_at_k: w.metrics.recall_at_k,
-            search_latency_p50_us: w.metrics.search_latency_p50_us,
-            build_time_secs: w.metrics.build_time_secs,
-        })
-    } else {
-        tracing::warn!(
-            "autotune '{}': no trial met min_recall={}; not promoting",
-            req.index_name, req.min_recall,
-        );
-        None
-    };
-
-    Ok(AutotuneResult {
-        index_name: req.index_name,
-        trials_run: run_trials.len(),
-        trials_eligible,
-        winner: winner_report,
-        promoted,
-        duration_secs: t0.elapsed().as_secs_f32(),
-        skipped: rejected,
-    })
-}
-
-/// Run one trial and journal it. Returns the recorded Trial.
-#[allow(clippy::too_many_arguments)]
-async fn run_single_trial(
-    index_name: &str,
-    harness_set: &harness::EvalSet,
-    embeddings: &Arc<Vec<crate::store::StoredEmbedding>>,
-    config: &HnswConfig,
-    hnsw_store: &HnswStore,
-    trial_journal: &TrialJournal,
-    _ai_client: &AiClient,
-) -> Result<Trial, String> {
-    let trial_id = Trial::new_id();
-    let slot = format!("{}__{}", index_name, trial_id);
-
-    let build = hnsw_store
-        .build_index_with_config(&slot, (**embeddings).clone(), config)
-        .await?;
-
-    let query_vectors: Vec<Vec<f32>> = harness_set
-        .queries
-        .iter()
-        .filter_map(|q| q.query_embedding.clone())
-        .collect();
-    let bench = hnsw_store
-        .bench_search(&slot, &query_vectors, harness_set.k)
-        .await?;
-
-    let mut recalls = Vec::with_capacity(harness_set.queries.len());
-    for (q, hits) in harness_set.queries.iter().zip(bench.retrieved.iter()) {
-        if let Some(gt) = &q.ground_truth {
-            recalls.push(harness::recall_at_k(hits, gt, harness_set.k));
-        }
-    }
-    let mean_recall = if recalls.is_empty() {
-        0.0
-    } else {
-        recalls.iter().sum::<f32>() / recalls.len() as f32
-    };
-
-    let mut lats = bench.latencies_us.clone();
-    lats.sort_unstable_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
-    let p = |pct: f32| -> f32 {
-        if lats.is_empty() { return 0.0; }
-        let idx = ((lats.len() as f32 - 1.0) * pct).round() as usize;
-        lats[idx.min(lats.len() - 1)]
-    };
-
-    let brute_us = if let Some(qv) = query_vectors.first() {
-        let t = std::time::Instant::now();
-        let _ = harness::brute_force_top_k(qv, embeddings, harness_set.k);
-        t.elapsed().as_micros() as f32
-    } else {
-        0.0
-    };
-
-    let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
-    let memory_bytes =
-        (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
-
-    let trial = Trial {
-        id: trial_id.clone(),
-        index_name: index_name.to_string(),
-        eval_set: harness_set.name.clone(),
-        config: config.clone(),
-        metrics: TrialMetrics {
-            build_time_secs: build.build_time_secs,
-            search_latency_p50_us: p(0.50),
-            search_latency_p95_us: p(0.95),
-            search_latency_p99_us: p(0.99),
-            recall_at_k: mean_recall,
-            memory_bytes,
-            vectors: build.vectors,
-            eval_queries: harness_set.queries.len(),
-            brute_force_latency_us: brute_us,
-        },
-        created_at: Utc::now(),
-        note: Some("autotune".to_string()),
-    };
-    let _ = trial_journal.append(&trial).await;
-    let _ = hnsw_store.drop(&slot).await;
-    Ok(trial)
-}

crates/vectord/src/chunker.rs

@@ -15,28 +15,9 @@ pub struct TextChunk {
     pub text: String,
 }
 
-/// Round `idx` UP to the nearest UTF-8 char boundary in `s`.
-/// Needed because `&str[a..b]` panics if `a` or `b` lands mid-character —
-/// and any text imported from Postgres/user data will contain multi-byte
-/// chars like `\u{202f}` (narrow no-break space), em-dashes, etc.
-fn ceil_char_boundary(s: &str, mut idx: usize) -> usize {
-    if idx >= s.len() { return s.len(); }
-    while idx <= s.len() && !s.is_char_boundary(idx) {
-        idx += 1;
-    }
-    idx
-}
-
 /// Split text into overlapping chunks.
-/// - `chunk_size`: target bytes per chunk (UTF-8 boundary-respecting)
-/// - `overlap`: bytes of overlap between consecutive chunks
-///
-/// Safety contract:
-/// - Never slices at a byte that isn't a UTF-8 char boundary
-/// - Always advances forward (monotonic `start`) — no infinite loops on
-///   pathological inputs
-/// - Bounded iteration — caps total chunks at `text.len()` to prevent
-///   runaway allocation on degenerate text/parameter combinations
+/// - `chunk_size`: target characters per chunk (not tokens — chars are a good proxy)
+/// - `overlap`: characters of overlap between consecutive chunks
 pub fn chunk_text(
     text: &str,
     source: &str,
@@ -59,78 +40,46 @@ pub fn chunk_text(
         }];
     }
 
+    // Work with char indices, not byte indices, to handle Unicode safely
+    let chars: Vec<char> = text.chars().collect();
+    let char_len = chars.len();
     let mut chunks = Vec::new();
-    let mut start = 0;
+    let mut start = 0usize; // char index
     let mut idx = 0u32;
 
-    // Safety cap: a well-formed chunk_size advances `start` by at least
-    // `chunk_size - overlap` each iteration. Even in degenerate cases we
-    // should never emit more chunks than there are bytes of text.
-    let max_chunks = text.len();
-    let mut iterations = 0;
+    while start < char_len {
+        let end = (start + chunk_size).min(char_len);
+        let chunk_str: String = chars[start..end].iter().collect();
 
-    while start < text.len() {
-        iterations += 1;
-        if iterations > max_chunks {
-            tracing::warn!(
-                "chunker: exceeded safety cap ({}) on doc '{}' of length {} — stopping",
-                max_chunks, doc_id, text.len(),
-            );
-            break;
-        }
-
-        // UTF-8 safe upper bound: move forward to a char boundary.
-        let end = ceil_char_boundary(text, (start + chunk_size).min(text.len()));
-
-        let chunk_text_slice = &text[start..end];
-        let actual_end = if end < text.len() {
-            // Look for last sentence boundary inside the slice.
-            if let Some(pos) = chunk_text_slice.rfind(". ") {
-                start + pos + 2
-            } else if let Some(pos) = chunk_text_slice.rfind('\n') {
-                start + pos + 1
-            } else if let Some(pos) = chunk_text_slice.rfind(' ') {
-                start + pos + 1
+        let actual_end = if end < char_len {
+            // Look for last sentence boundary
+            if let Some(pos) = chunk_str.rfind(". ") {
+                start + chunk_str[..pos + 2].chars().count()
+            } else if let Some(pos) = chunk_str.rfind('\n') {
+                start + chunk_str[..pos + 1].chars().count()
+            } else if let Some(pos) = chunk_str.rfind(' ') {
+                start + chunk_str[..pos + 1].chars().count()
             } else {
                 end
             }
         } else {
             end
         };
-        // All the rfind'd positions are at char boundaries (they match
-        // ASCII byte patterns), so actual_end should already be safe.
-        // Defensive belt-and-suspenders: round up anyway.
-        let actual_end = ceil_char_boundary(text, actual_end);
 
-        let chunk = text[start..actual_end].trim();
-        if !chunk.is_empty() {
+        let chunk: String = chars[start..actual_end].iter().collect();
+        let trimmed = chunk.trim();
+        if !trimmed.is_empty() {
             chunks.push(TextChunk {
                 source: source.to_string(),
                 doc_id: doc_id.to_string(),
                 chunk_idx: idx,
-                text: chunk.to_string(),
+                text: trimmed.to_string(),
             });
             idx += 1;
         }
 
-        // Advance with overlap — must strictly progress to avoid infinite
-        // loops on degenerate inputs (e.g. chunk_size ≤ overlap, or
-        // boundary-finding returning a position at `start`).
-        if actual_end >= text.len() {
-            break;
-        }
-        let tentative = if actual_end > overlap { actual_end - overlap } else { actual_end };
-        let next_start = ceil_char_boundary(text, tentative);
-        if next_start <= start {
-            // No progress — force forward one char to guarantee termination.
-            let mut forced = start + 1;
-            while forced < text.len() && !text.is_char_boundary(forced) {
-                forced += 1;
-            }
-            start = forced;
-        } else {
-            start = next_start;
-        }
+        if actual_end >= char_len { break; }
+        start = if actual_end > overlap { actual_end - overlap } else { actual_end };
     }
 
     chunks
@@ -180,45 +129,4 @@ mod tests {
         assert_eq!(chunks[0].source, "candidates");
         assert_eq!(chunks[0].doc_id, "CAND-001");
     }
-
-    #[test]
-    fn handles_multibyte_utf8_at_chunk_boundary() {
-        // Regression: multi-byte unicode (narrow no-break space \u{202f},
-        // 3 bytes) landing inside a chunk-size window used to panic with
-        // "byte index N is not a char boundary". Found on pg-imported
-        // data that contained editorial unicode punctuation.
-        let text = "**Claim 1**\u{202f}\u{2013} *Each large language model has its own personality \
-                   and way of interacting.*\u{202f}\u{2013}\u{202f}Verdict:\u{202f}VERIFIED. \
-                   Reasoning: widely accepted observation that different models exhibit distinct response \
-                   styles stemming from variations in training data.".repeat(5);
-        let chunks = chunk_text(&text, "test", "unicode-1", 500, 50);
-        // Must not panic, must produce at least one chunk.
-        assert!(!chunks.is_empty());
-        // All chunks must be valid UTF-8 substrings of the original.
-        for chunk in &chunks {
-            assert!(!chunk.text.is_empty());
-            assert!(text.contains(&chunk.text));
-        }
-    }
-
-    #[test]
-    fn no_infinite_loop_on_no_spaces() {
-        // Long text with no spaces, sentences, or newlines — chunker must
-        // still advance via the hard `end` fallback.
-        let text = "a".repeat(5000);
-        let chunks = chunk_text(&text, "test", "noSpace", 500, 50);
-        assert!(!chunks.is_empty());
-        // Expected ~5000/450 chunks (with overlap=50 → step=450)
-        assert!(chunks.len() < 50, "chunk count {} seems excessive", chunks.len());
-    }
-
-    #[test]
-    fn no_infinite_loop_on_degenerate_params() {
-        // chunk_size equals overlap — previously would have made zero
-        // forward progress. Our safety forced-advance must catch this.
-        let text = "a b c d e f g h i j k l m n o p q r s t u v w x y z".repeat(20);
-        let chunks = chunk_text(&text, "test", "degen", 100, 100);
-        assert!(!chunks.is_empty());
-        assert!(chunks.len() <= text.len(), "runaway chunk count {}", chunks.len());
-    }
 }

crates/vectord/src/embedding_cache.rs

@@ -1,98 +0,0 @@
-/// In-memory cache for StoredEmbedding vectors.
-///
-/// Rationale: loading 100K embeddings from Parquet takes ~2-5s. When an AI agent
-/// iterates on HNSW parameters, each trial would repeat that cost. The cache
-/// pins embeddings in memory so trials reuse them.
-///
-/// This is a pure performance layer — the Parquet file is still the source of
-/// truth (ADR-008). Eviction is safe; worst case is one slow reload.
-
-use object_store::ObjectStore;
-use serde::Serialize;
-use std::collections::HashMap;
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use crate::store::{self, StoredEmbedding};
-
-#[derive(Clone)]
-pub struct EmbeddingCache {
-    store: Arc<dyn ObjectStore>,
-    cache: Arc<RwLock<HashMap<String, Arc<Vec<StoredEmbedding>>>>>,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct CacheEntry {
-    pub index_name: String,
-    pub vectors: usize,
-    pub dimensions: usize,
-    pub memory_bytes: u64,
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct CacheStats {
-    pub entries: Vec<CacheEntry>,
-    pub total_memory_bytes: u64,
-}
-
-impl EmbeddingCache {
-    pub fn new(store: Arc<dyn ObjectStore>) -> Self {
-        Self {
-            store,
-            cache: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    /// Return cached embeddings, loading from object storage on first request.
-    pub async fn get_or_load(
-        &self,
-        index_name: &str,
-    ) -> Result<Arc<Vec<StoredEmbedding>>, String> {
-        if let Some(cached) = self.cache.read().await.get(index_name) {
-            return Ok(cached.clone());
-        }
-
-        // Load under a write lock so concurrent callers only hit disk once.
-        let mut guard = self.cache.write().await;
-        if let Some(cached) = guard.get(index_name) {
-            return Ok(cached.clone());
-        }
-        tracing::info!("embedding_cache: loading '{index_name}' from object storage");
-        let t0 = std::time::Instant::now();
-        let loaded = store::load_embeddings(&self.store, index_name).await?;
-        let n = loaded.len();
-        let arc = Arc::new(loaded);
-        guard.insert(index_name.to_string(), arc.clone());
-        tracing::info!(
-            "embedding_cache: loaded '{index_name}' — {n} vectors in {:.2}s",
-            t0.elapsed().as_secs_f32()
-        );
-        Ok(arc)
-    }
-
-    pub async fn evict(&self, index_name: &str) -> bool {
-        self.cache.write().await.remove(index_name).is_some()
-    }
-
-    pub async fn stats(&self) -> CacheStats {
-        let cache = self.cache.read().await;
-        let mut entries = Vec::with_capacity(cache.len());
-        let mut total: u64 = 0;
-        for (name, embs) in cache.iter() {
-            let dims = embs.first().map(|e| e.vector.len()).unwrap_or(0);
-            // Rough estimate: vector data + chunk_text + metadata overhead.
-            let vec_bytes = (embs.len() * dims * std::mem::size_of::<f32>()) as u64;
-            let text_bytes: u64 = embs.iter().map(|e| e.chunk_text.len() as u64).sum();
-            let overhead = (embs.len() * 128) as u64; // strings + struct overhead
-            let mem = vec_bytes + text_bytes + overhead;
-            total += mem;
-            entries.push(CacheEntry {
-                index_name: name.clone(),
-                vectors: embs.len(),
-                dimensions: dims,
-                memory_bytes: mem,
-            });
-        }
-        CacheStats { entries, total_memory_bytes: total }
-    }
-}

crates/vectord/src/harness.rs

@@ -1,387 +0,0 @@
-/// Eval harness for HNSW tuning.
-///
-/// An EvalSet is a named collection of queries against a specific vector index.
-/// Each query has a query_text and (optionally pre-computed) ground_truth
-/// doc_ids — the "correct" top-k results that brute-force cosine returns.
-/// HNSW trials are scored by how well they recreate that top-k (recall@k).
-///
-/// Storage: `_hnsw_evals/{name}.json` as a single JSON document. Small.
-
-use chrono::{DateTime, Utc};
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::collections::HashSet;
-use std::sync::Arc;
-
-use aibridge::client::{AiClient, EmbedRequest};
-use storaged::ops;
-use storaged::registry::BucketRegistry;
-
-use crate::index_registry::IndexRegistry;
-use crate::store::StoredEmbedding;
-
-/// A single eval query with optional pre-computed ground truth.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct EvalQuery {
-    pub id: String,
-    pub query_text: String,
-    /// Ordered list of doc_ids that brute-force returns as top-k.
-    /// `None` means the ground truth hasn't been computed yet; `compute_ground_truth`
-    /// fills it in against the current embedding set.
-    #[serde(default)]
-    pub ground_truth: Option<Vec<String>>,
-    /// Optional pre-computed query embedding. Filling this avoids re-embedding
-    /// the same query on every trial.
-    #[serde(default)]
-    pub query_embedding: Option<Vec<f32>>,
-}
-
-/// A named eval set tied to a specific vector index.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct EvalSet {
-    pub name: String,
-    pub index_name: String,
-    pub k: usize, // top-k used for recall calculation
-    pub queries: Vec<EvalQuery>,
-    pub created_at: DateTime<Utc>,
-    pub ground_truth_built: bool,
-}
-
-impl EvalSet {
-    pub fn new(name: &str, index_name: &str, k: usize) -> Self {
-        Self {
-            name: name.to_string(),
-            index_name: index_name.to_string(),
-            k,
-            queries: Vec::new(),
-            created_at: Utc::now(),
-            ground_truth_built: false,
-        }
-    }
-
-    pub fn storage_key(&self) -> String {
-        format!("_hnsw_evals/{}.json", self.name)
-    }
-
-    pub async fn save(&self, store: &Arc<dyn ObjectStore>) -> Result<(), String> {
-        let json = serde_json::to_vec_pretty(self).map_err(|e| e.to_string())?;
-        ops::put(store, &self.storage_key(), json.into()).await
-    }
-
-    pub async fn load(store: &Arc<dyn ObjectStore>, name: &str) -> Result<Self, String> {
-        let key = format!("_hnsw_evals/{}.json", name);
-        let data = ops::get(store, &key).await?;
-        serde_json::from_slice(&data).map_err(|e| format!("parse eval set: {e}"))
-    }
-
-    pub async fn list(store: &Arc<dyn ObjectStore>) -> Result<Vec<String>, String> {
-        let keys = ops::list(store, Some("_hnsw_evals/")).await?;
-        Ok(keys
-            .into_iter()
-            .filter(|k| k.ends_with(".json"))
-            .map(|k| {
-                k.trim_start_matches("_hnsw_evals/")
-                    .trim_end_matches(".json")
-                    .to_string()
-            })
-            .collect())
-    }
-}
-
-/// Federation-aware wrapper around EvalSet persistence. Mirrors the
-/// `TrialJournal` / `PromotionRegistry` pattern: harness files colocate
-/// with their index's bucket (looked up via `IndexMeta.bucket`), falling
-/// back to `primary` for indexes the registry has never seen. Legacy
-/// harnesses predating federation remain discoverable — lookups transparently
-/// try the resolved bucket first, then `primary` as a fallback. Cross-bucket
-/// listing dedupes across every registered bucket so `GET /hnsw/evals`
-/// returns a complete picture.
-#[derive(Clone)]
-pub struct HarnessStore {
-    buckets: Arc<BucketRegistry>,
-    index_registry: IndexRegistry,
-}
-
-impl HarnessStore {
-    pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
-        Self { buckets, index_registry }
-    }
-
-    /// Resolve which bucket holds this index's eval artifacts. Indexes the
-    /// registry has never heard of fall through to `primary`.
-    async fn bucket_for_index(&self, index_name: &str) -> String {
-        self.index_registry
-            .get(index_name)
-            .await
-            .map(|m| m.bucket)
-            .unwrap_or_else(|| "primary".to_string())
-    }
-
-    /// Save to the bucket that owns `eval.index_name`. Writes under the
-    /// standard `_hnsw_evals/{name}.json` prefix of the resolved bucket.
-    pub async fn save(&self, eval: &EvalSet) -> Result<(), String> {
-        let bucket = self.bucket_for_index(&eval.index_name).await;
-        let store = self.buckets.get(&bucket)?;
-        eval.save(&store).await
-    }
-
-    /// Load a harness by name, given the index it belongs to. Tries the
-    /// index's bucket first; if the file is absent AND the resolved bucket
-    /// isn't `primary`, falls through to `primary` so pre-federation evals
-    /// remain reachable without migration.
-    pub async fn load_for_index(
-        &self,
-        index_name: &str,
-        harness_name: &str,
-    ) -> Result<EvalSet, String> {
-        let bucket = self.bucket_for_index(index_name).await;
-        let primary_store = self.buckets.get("primary")?;
-        let store = self.buckets.get(&bucket)?;
-
-        match EvalSet::load(&store, harness_name).await {
-            Ok(e) => Ok(e),
-            Err(e) if bucket != "primary" => EvalSet::load(&primary_store, harness_name)
-                .await
-                .map_err(|primary_err| format!("{bucket}: {e}; primary fallback: {primary_err}")),
-            Err(e) => Err(e),
-        }
-    }
-
-    /// Find a harness by name without knowing which index it belongs to —
-    /// used by `GET /hnsw/evals/{name}`. Scans every registered bucket;
-    /// first hit wins. Primary is searched first so pre-federation evals
-    /// with the same name as a federated one resolve deterministically.
-    pub async fn get_any(&self, harness_name: &str) -> Result<EvalSet, String> {
-        let bucket_infos = self.buckets.list().await;
-        let mut ordered: Vec<String> = bucket_infos.iter().map(|b| b.name.clone()).collect();
-        ordered.sort_by_key(|n| if n == "primary" { 0 } else { 1 });
-
-        let mut last_err = None;
-        for b in ordered {
-            let store = match self.buckets.get(&b) {
-                Ok(s) => s,
-                Err(e) => { last_err = Some(e); continue; }
-            };
-            match EvalSet::load(&store, harness_name).await {
-                Ok(e) => return Ok(e),
-                Err(e) => { last_err = Some(e); }
-            }
-        }
-        Err(last_err.unwrap_or_else(|| format!("no buckets registered for eval '{harness_name}'")))
-    }
-
-    /// Union of every harness name across every registered bucket.
-    /// Duplicates (same name in multiple buckets — pathological but
-    /// possible after a manual migration) are collapsed.
-    pub async fn list_all(&self) -> Vec<String> {
-        let mut all: HashSet<String> = HashSet::new();
-        for b in self.buckets.list().await {
-            let store = match self.buckets.get(&b.name) {
-                Ok(s) => s,
-                Err(_) => continue,
-            };
-            if let Ok(names) = EvalSet::list(&store).await {
-                all.extend(names);
-            }
-        }
-        let mut out: Vec<String> = all.into_iter().collect();
-        out.sort();
-        out
-    }
-}
-
-/// Cosine similarity for two same-length f32 slices.
-fn cosine(a: &[f32], b: &[f32]) -> f32 {
-    let mut dot = 0.0f32;
-    let mut na = 0.0f32;
-    let mut nb = 0.0f32;
-    for i in 0..a.len() {
-        dot += a[i] * b[i];
-        na += a[i] * a[i];
-        nb += b[i] * b[i];
-    }
-    if na == 0.0 || nb == 0.0 {
-        return 0.0;
-    }
-    dot / (na.sqrt() * nb.sqrt())
-}
-
-/// Brute-force top-k search for a single query against all embeddings.
-/// This is the ground-truth oracle that HNSW trials must approximate.
-pub fn brute_force_top_k(
-    query: &[f32],
-    embeddings: &[StoredEmbedding],
-    k: usize,
-) -> Vec<String> {
-    let mut scored: Vec<(f32, usize)> = embeddings
-        .iter()
-        .enumerate()
-        .map(|(i, e)| (cosine(query, &e.vector), i))
-        .collect();
-    // Partial sort — we only need top-k.
-    scored.sort_unstable_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal));
-    scored
-        .into_iter()
-        .take(k)
-        .map(|(_, i)| embeddings[i].doc_id.clone())
-        .collect()
-}
-
-/// Recall@k — fraction of ground-truth doc_ids that appear in predicted.
-pub fn recall_at_k(predicted: &[String], ground_truth: &[String], k: usize) -> f32 {
-    if ground_truth.is_empty() || k == 0 {
-        return 0.0;
-    }
-    // Set-intersection recall@k. Previous implementation counted duplicates
-    // in `predicted` (a corpus with repeated chunks — e.g. cached LLM
-    // responses — returns the same doc_id multiple times via HNSW), which
-    // inflated recall above 1.0 and poisoned promotion decisions.
-    let gt_set: std::collections::HashSet<&String> =
-        ground_truth.iter().take(k).collect();
-    let pred_set: std::collections::HashSet<&String> =
-        predicted.iter().take(k).collect();
-    let hits = pred_set.intersection(&gt_set).count();
-    hits as f32 / gt_set.len() as f32
-}
-
-/// Populate query_embedding and ground_truth for every query that lacks them.
-pub async fn compute_ground_truth(
-    eval: &mut EvalSet,
-    embeddings: &[StoredEmbedding],
-    ai_client: &AiClient,
-) -> Result<(), String> {
-    let need_embed: Vec<(usize, String)> = eval
-        .queries
-        .iter()
-        .enumerate()
-        .filter(|(_, q)| q.query_embedding.is_none())
-        .map(|(i, q)| (i, q.query_text.clone()))
-        .collect();
-
-    if !need_embed.is_empty() {
-        // Embed in one batch to keep things simple; for very large eval sets
-        // we'd batch this in chunks of 32.
-        let texts: Vec<String> = need_embed.iter().map(|(_, t)| t.clone()).collect();
-        let resp = ai_client
-            .embed(EmbedRequest { texts, model: None })
-            .await
-            .map_err(|e| format!("embed queries: {e}"))?;
-        for ((idx, _), vec) in need_embed.iter().zip(resp.embeddings.iter()) {
-            let v: Vec<f32> = vec.iter().map(|&x| x as f32).collect();
-            eval.queries[*idx].query_embedding = Some(v);
-        }
-    }
-
-    for q in eval.queries.iter_mut() {
-        if q.ground_truth.is_some() {
-            continue;
-        }
-        let emb = q.query_embedding.as_ref().ok_or("missing embedding")?;
-        q.ground_truth = Some(brute_force_top_k(emb, embeddings, eval.k));
-    }
-    eval.ground_truth_built = true;
-    Ok(())
-}
-
-/// Auto-generate a synthetic eval set by sampling every Nth chunk's text as
-/// its own query. Useful for a quick-start eval when the user doesn't have
-/// real natural-language queries yet.
-pub fn synthetic_from_chunks(
-    eval_name: &str,
-    index_name: &str,
-    embeddings: &[StoredEmbedding],
-    sample_count: usize,
-    k: usize,
-) -> EvalSet {
-    let n = embeddings.len();
-    let sample_count = sample_count.min(n);
-    let stride = (n / sample_count.max(1)).max(1);
-
-    let mut queries = Vec::with_capacity(sample_count);
-    for i in 0..sample_count {
-        let idx = (i * stride).min(n - 1);
-        let chunk = &embeddings[idx];
-        // Use the first ~200 chars of the chunk as the "query" — it should find
-        // itself and nearby chunks as top results.
-        let query_text: String = chunk.chunk_text.chars().take(200).collect();
-        queries.push(EvalQuery {
-            id: format!("syn-{}", i),
-            query_text,
-            ground_truth: None,
-            query_embedding: None,
-        });
-    }
-
-    EvalSet {
-        name: eval_name.to_string(),
-        index_name: index_name.to_string(),
-        k,
-        queries,
-        created_at: Utc::now(),
-        ground_truth_built: false,
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn s(v: &[&str]) -> Vec<String> { v.iter().map(|x| x.to_string()).collect() }
-
-    #[test]
-    fn recall_empty_ground_truth_is_zero() {
-        assert_eq!(recall_at_k(&s(&["a", "b"]), &[], 10), 0.0);
-    }
-
-    #[test]
-    fn recall_k_zero_is_zero() {
-        assert_eq!(recall_at_k(&s(&["a"]), &s(&["a"]), 0), 0.0);
-    }
-
-    #[test]
-    fn recall_perfect_match_equals_one() {
-        let pred = s(&["a", "b", "c"]);
-        let gt = s(&["a", "b", "c"]);
-        assert!((recall_at_k(&pred, &gt, 3) - 1.0).abs() < 1e-6);
-    }
-
-    #[test]
-    fn recall_half_match() {
-        let pred = s(&["a", "b", "x", "y"]);
-        let gt = s(&["a", "b", "c", "d"]);
-        assert!((recall_at_k(&pred, &gt, 4) - 0.5).abs() < 1e-6);
-    }
-
-    #[test]
-    fn recall_duplicates_in_predicted_do_not_inflate() {
-        // Regression guard: the previous implementation counted each
-        // duplicate in `predicted` separately, which could push recall
-        // above 1.0 on corpora with repeated chunks (cached responses etc).
-        // Set-intersection semantics keep it bounded in [0, 1].
-        let pred = s(&["a", "a", "a", "a"]);
-        let gt = s(&["a", "b", "c", "d"]);
-        let r = recall_at_k(&pred, &gt, 4);
-        assert!(r <= 1.0, "recall {r} must not exceed 1.0");
-        // One unique match out of four in gt = 0.25.
-        assert!((r - 0.25).abs() < 1e-6);
-    }
-
-    #[test]
-    fn recall_duplicates_in_ground_truth_handled() {
-        // gt with dupes reduces effective |gt|; matching one hits all.
-        let pred = s(&["x"]);
-        let gt = s(&["x", "x", "x"]);
-        let r = recall_at_k(&pred, &gt, 3);
-        assert!(r <= 1.0);
-        assert!((r - 1.0).abs() < 1e-6); // |pred ∩ gt| / |gt_set| = 1/1
-    }
-
-    #[test]
-    fn recall_respects_k_bound() {
-        // k=2 means only the first 2 of pred and gt count toward the set.
-        let pred = s(&["a", "b", "c", "d"]);
-        let gt = s(&["a", "b", "c", "d"]);
-        let r = recall_at_k(&pred, &gt, 2);
-        assert!((r - 1.0).abs() < 1e-6);
-    }
-}

crates/vectord/src/hnsw.rs

@@ -7,7 +7,6 @@ use std::sync::Arc;
 use tokio::sync::RwLock;
 
 use crate::store::StoredEmbedding;
-use crate::trial::HnswConfig;
 
 /// A vector point for HNSW — wraps f32 slice with cosine distance.
 #[derive(Clone)]
@@ -63,32 +62,18 @@ impl HnswStore {
         }
     }
 
-    /// Build an HNSW index from stored embeddings with default config.
+    /// Build an HNSW index from stored embeddings.
     pub async fn build_index(
         &self,
         index_name: &str,
         embeddings: Vec<StoredEmbedding>,
-    ) -> Result<BuildStats, String> {
-        self.build_index_with_config(index_name, embeddings, &HnswConfig::default()).await
-    }
-
-    /// Build an HNSW index from stored embeddings with explicit config.
-    /// Used by the trial system — each trial calls this with different params.
-    pub async fn build_index_with_config(
-        &self,
-        index_name: &str,
-        embeddings: Vec<StoredEmbedding>,
-        config: &HnswConfig,
     ) -> Result<BuildStats, String> {
         let n = embeddings.len();
         if n == 0 {
             return Err("no embeddings to index".into());
         }
 
-        tracing::info!(
-            "building HNSW '{}': {} vectors, ef_construction={} ef_search={} seed={:?}",
-            index_name, n, config.ef_construction, config.ef_search, config.seed,
-        );
+        tracing::info!("building HNSW index '{}' from {} vectors", index_name, n);
         let start = std::time::Instant::now();
 
         // Separate points and metadata
@@ -107,17 +92,14 @@ impl HnswStore {
             values.push(i);
         }
 
-        // Build HNSW — the expensive part
-        let mut builder = Builder::default()
-            .ef_construction(config.ef_construction)
-            .ef_search(config.ef_search);
-        if let Some(seed) = config.seed {
-            builder = builder.seed(seed);
-        }
-        let map = builder.build(points, values);
+        // Build HNSW — this is the expensive part
+        let map = Builder::default()
+            .ef_construction(40)   // balanced for 100K scale
+            .ef_search(50)         // fast search with good recall
+            .build(points, values);
 
         let build_time = start.elapsed().as_secs_f32();
-        tracing::info!("HNSW '{}' built: {} vectors in {:.1}s", index_name, n, build_time);
+        tracing::info!("HNSW index '{}' built: {} vectors in {:.1}s", index_name, n, build_time);
 
         let index = Arc::new(HnswIndex { map, metadata });
         self.indexes.write().await.insert(index_name.to_string(), index);
@@ -129,43 +111,6 @@ impl HnswStore {
         })
     }
 
-    /// Run a batch of search queries and return raw per-query latencies in microseconds.
-    /// Also returns the retrieved doc_ids per query (for recall calculation).
-    pub async fn bench_search(
-        &self,
-        index_name: &str,
-        query_vectors: &[Vec<f32>],
-        top_k: usize,
-    ) -> Result<BenchResult, String> {
-        let indexes = self.indexes.read().await;
-        let index = indexes
-            .get(index_name)
-            .ok_or_else(|| format!("HNSW index not found: {index_name}"))?
-            .clone();
-        drop(indexes);
-
-        let mut latencies_us = Vec::with_capacity(query_vectors.len());
-        let mut retrieved: Vec<Vec<String>> = Vec::with_capacity(query_vectors.len());
-
-        for qv in query_vectors {
-            let query_point = VecPoint(qv.clone());
-            let t0 = std::time::Instant::now();
-            let mut search = Search::default();
-            let results = index.map.search(&query_point, &mut search);
-            let ids: Vec<String> = results
-                .take(top_k)
-                .map(|item| {
-                    let meta_idx = *item.value;
-                    index.metadata[meta_idx].doc_id.clone()
-                })
-                .collect();
-            latencies_us.push(t0.elapsed().as_micros() as f32);
-            retrieved.push(ids);
-        }
-
-        Ok(BenchResult { latencies_us, retrieved })
-    }
-
     /// Search an HNSW index. Returns approximate nearest neighbors.
     pub async fn search(
         &self,
@@ -221,9 +166,3 @@ pub struct BuildStats {
     pub vectors: usize,
     pub build_time_secs: f32,
 }
-
-#[derive(Debug, Clone)]
-pub struct BenchResult {
-    pub latencies_us: Vec<f32>,
-    pub retrieved: Vec<Vec<String>>,
-}

crates/vectord/src/index_registry.rs

@@ -29,27 +29,8 @@ pub struct IndexMeta {
     pub created_at: DateTime<Utc>,
     pub build_time_secs: f32,
     pub chunks_per_sec: f32,
-    /// Federation layer 2: which bucket holds this index's artifacts
-    /// (trial journal + promotion file). Defaults to "primary" for
-    /// pre-federation indexes — the serde default keeps old metadata
-    /// files readable without migration.
-    #[serde(default = "default_bucket")]
-    pub bucket: String,
-    /// ADR-019: which physical backend stores this index. `Parquet`
-    /// means storage_key points at our binary-blob Parquet file;
-    /// `Lance` means it points at a Lance dataset directory.
-    #[serde(default)]
-    pub vector_backend: shared::types::VectorBackend,
-    /// ADR-020: prefix prepended to doc_ids during embedding. If set,
-    /// hybrid search strips this prefix to match against SQL primary keys.
-    /// None = doc_ids ARE the raw primary keys (no stripping needed).
-    /// Existing indexes: "W-", "CAND-", "W500K-", etc.
-    #[serde(default)]
-    pub id_prefix: Option<String>,
 }
 
-fn default_bucket() -> String { "primary".to_string() }
-
 /// Registry of all vector indexes.
 #[derive(Clone)]
 pub struct IndexRegistry {

crates/vectord/src/jobs.rs

@@ -22,12 +22,7 @@ pub struct Job {
     pub status: JobStatus,
     pub index_name: String,
     pub total_chunks: usize,
-    /// How many chunks have been embedded so far.
-    /// Also serialized as "processed" for backward-compat with monitoring scripts.
     pub embedded_chunks: usize,
-    #[serde(rename = "processed")]
-    pub processed_alias: usize,
-    pub total: usize,
     pub progress_pct: f32,
     pub storage_key: Option<String>,
     pub error: Option<String>,
@@ -58,8 +53,6 @@ impl JobTracker {
             index_name: index_name.to_string(),
             total_chunks,
             embedded_chunks: 0,
-            processed_alias: 0,
-            total: total_chunks,
             progress_pct: 0.0,
             storage_key: None,
             error: None,
@@ -76,7 +69,6 @@ impl JobTracker {
         let mut jobs = self.jobs.write().await;
         if let Some(job) = jobs.get_mut(id) {
             job.embedded_chunks = embedded;
-            job.processed_alias = embedded;  // keep alias in sync
             job.progress_pct = if job.total_chunks > 0 {
                 (embedded as f32 / job.total_chunks as f32) * 100.0
             } else {
@@ -92,7 +84,6 @@ impl JobTracker {
         if let Some(job) = jobs.get_mut(id) {
             job.status = JobStatus::Completed;
             job.embedded_chunks = job.total_chunks;
-            job.processed_alias = job.total_chunks;
             job.progress_pct = 100.0;
             job.storage_key = Some(storage_key);
             job.completed_at = Some(chrono::Utc::now().to_rfc3339());

crates/vectord/src/lance_backend.rs

@@ -1,131 +0,0 @@
-//! ADR-019 hybrid: routing layer for the Lance vector backend.
-//!
-//! Holds a `LanceRegistry` which maps an index name to its
-//! `LanceVectorStore` instance, lazy-creating on first touch. Path
-//! resolution: the index's bucket gives us a bucket root; we append
-//! `lance/{index_name}` and use that as the dataset URI. For local
-//! buckets that means a directory under the bucket's root.
-//!
-//! S3 buckets: in principle Lance accepts s3://... URIs, but the
-//! firewall crate (`vectord-lance`) doesn't pull the S3 feature in by
-//! default. When we promote profile buckets onto S3 in a future phase,
-//! enable that feature and update `lance_uri_for` accordingly.
-
-use std::collections::HashMap;
-use std::path::PathBuf;
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use storaged::registry::BucketRegistry;
-use vectord_lance::LanceVectorStore;
-
-use crate::index_registry::IndexRegistry;
-
-/// Convert a bucket+index pair into the URI Lance should use as the
-/// dataset path. Supports both local (filesystem) and S3 buckets.
-///
-/// **Local buckets:** path resolution mirrors lakehouse.toml's convention.
-/// Returns an absolute filesystem path.
-///
-/// **S3 buckets:** returns `s3://{s3_bucket}/lance/{index_name}`. Lance's
-/// internal object_store crate reads `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
-/// / `AWS_ENDPOINT` from environment (or the S3 feature's default chain).
-/// For MinIO: set `AWS_ENDPOINT=http://localhost:9000` and
-/// `AWS_ALLOW_HTTP=true` before starting the gateway.
-///
-/// Refuses unknown buckets so a typo doesn't silently land Lance data
-/// in a directory / prefix the rest of the system can't see.
-pub fn lance_uri_for(
-    buckets: &BucketRegistry,
-    bucket: &str,
-    index_name: &str,
-) -> Result<String, String> {
-    if !buckets.contains(bucket) {
-        return Err(format!("bucket '{bucket}' not registered"));
-    }
-    // Check if this bucket is S3-backed by looking for a bucket config
-    // with backend="s3". BucketRegistry exposes backend type through
-    // the list() info, but that's async. The simpler signal: if the
-    // bucket name matches one we know is S3 (configured via lakehouse.toml
-    // with backend="s3"), use the s3:// URI scheme.
-    //
-    // For the synchronous path, we check a naming convention: buckets
-    // whose name starts with "s3:" are treated as S3 targets. The rest
-    // of the name is the S3 bucket name. Convention-based, explicit,
-    // no async needed.
-    //
-    // Additionally, any bucket registered with backend="s3" in the
-    // config will have its BucketConfig.bucket field set — that's the
-    // actual S3 bucket name. We can't access BucketConfig synchronously
-    // from the current registry API, so for now the naming convention
-    // is the primary signal.
-    if bucket.starts_with("s3:") {
-        let s3_bucket = &bucket["s3:".len()..];
-        return Ok(format!("s3://{s3_bucket}/lance/{index_name}"));
-    }
-
-    // Local path resolution.
-    let root: PathBuf = match bucket {
-        "primary" => PathBuf::from("./data"),
-        "rescue"  => PathBuf::from("./data/_rescue"),
-        "testing" => PathBuf::from("./data/_testing"),
-        b if b.starts_with("profile:") => {
-            let safe = b.replace(':', "_");
-            PathBuf::from(format!("./data/_profiles/{safe}"))
-        }
-        b => PathBuf::from(format!("./data/_buckets/{}", b.replace(':', "_"))),
-    };
-    let _ = std::fs::create_dir_all(root.join("lance"));
-    let abs = match std::fs::canonicalize(&root) {
-        Ok(p) => p.join("lance").join(index_name),
-        Err(_) => root.join("lance").join(index_name),
-    };
-    Ok(abs.to_string_lossy().to_string())
-}
-
-/// Lookup-by-index registry of `LanceVectorStore` handles. One handle
-/// per index, lazy-created on first call. Cheap to keep alive — it's
-/// just a path string + handle to Lance's metadata cache.
-#[derive(Clone)]
-pub struct LanceRegistry {
-    buckets: Arc<BucketRegistry>,
-    indexes: IndexRegistry,
-    stores: Arc<RwLock<HashMap<String, LanceVectorStore>>>,
-}
-
-impl LanceRegistry {
-    pub fn new(buckets: Arc<BucketRegistry>, indexes: IndexRegistry) -> Self {
-        Self {
-            buckets,
-            indexes,
-            stores: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    /// Get (or lazy-create) the LanceVectorStore for an index.
-    /// Resolves bucket → URI via the registry; backend stays whatever
-    /// IndexMeta says (no enforcement here — caller decides whether to
-    /// route to Lance based on profile/index backend choice).
-    pub async fn store_for(&self, index_name: &str) -> Result<LanceVectorStore, String> {
-        if let Some(s) = self.stores.read().await.get(index_name) {
-            return Ok(s.clone());
-        }
-        let bucket = self.indexes
-            .get(index_name).await
-            .map(|m| m.bucket)
-            .unwrap_or_else(|| "primary".to_string());
-        let uri = lance_uri_for(&self.buckets, &bucket, index_name)?;
-        let store = LanceVectorStore::new(uri);
-        self.stores.write().await.insert(index_name.to_string(), store.clone());
-        Ok(store)
-    }
-
-    /// For freshly-created indexes that don't have IndexMeta yet — caller
-    /// supplies the bucket explicitly.
-    pub async fn store_for_new(&self, index_name: &str, bucket: &str) -> Result<LanceVectorStore, String> {
-        let uri = lance_uri_for(&self.buckets, bucket, index_name)?;
-        let store = LanceVectorStore::new(uri);
-        self.stores.write().await.insert(index_name.to_string(), store.clone());
-        Ok(store)
-    }
-}

crates/vectord/src/lib.rs

@@ -1,18 +1,9 @@
-pub mod agent;
-pub mod autotune;
-pub mod lance_backend;
 pub mod chunker;
-pub mod embedding_cache;
-pub mod harness;
 pub mod hnsw;
 pub mod index_registry;
 pub mod jobs;
-pub mod playbook_memory;
-pub mod promotion;
-pub mod refresh;
 pub mod store;
 pub mod search;
 pub mod rag;
 pub mod supervisor;
 pub mod service;
-pub mod trial;

crates/vectord/src/promotion.rs

@@ -1,220 +0,0 @@
-//! Phase 16: Promoted HNSW configs — the "active generation" pointer.
-//!
-//! An index's HNSW config used at build time normally defaults to the
-//! system-wide default (`HnswConfig::default()`). An operator or the
-//! autotune agent can *promote* a specific trial's config — subsequent
-//! HNSW builds against that index use the promoted config instead.
-//!
-//! Every promotion is history-tracked so `rollback` can revert. The
-//! history file lives at `primary://_hnsw_promotions/{index_name}.json`
-//! and is small (< few KB) so we rewrite it on every promotion rather
-//! than append-log.
-//!
-//! Not included here:
-//! - Atomic graph rebuild on promote — promotion only updates the sticky
-//!   default. Next activation (or search that triggers lazy build) picks
-//!   up the new config. That's "zero-downtime swap after build" which is
-//!   what ADR-019 actually claimed; an instant-swap requires a
-//!   pre-built graph pool which we don't have yet.
-//! - Agent loop — lives in `vectord::autotune`.
-
-use chrono::{DateTime, Utc};
-use serde::{Deserialize, Serialize};
-use std::collections::HashMap;
-use std::sync::Arc;
-use storaged::ops;
-use storaged::registry::BucketRegistry;
-use tokio::sync::RwLock;
-
-use crate::index_registry::IndexRegistry;
-use crate::trial::HnswConfig;
-
-const PROMOTION_PREFIX: &str = "_hnsw_promotions";
-
-/// One promotion record. The `trial_id` is the origin of the config —
-/// lets operators trace back "why was this config picked?" to the exact
-/// trial in the trial journal.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct PromotionEntry {
-    pub config: HnswConfig,
-    pub trial_id: String,
-    pub promoted_at: DateTime<Utc>,
-    #[serde(default)]
-    pub promoted_by: String,
-    #[serde(default)]
-    pub note: Option<String>,
-}
-
-/// Serialized form of an index's promotion history.
-#[derive(Debug, Clone, Serialize, Deserialize, Default)]
-pub struct PromotionFile {
-    pub index_name: String,
-    pub current: Option<PromotionEntry>,
-    #[serde(default)]
-    pub history: Vec<PromotionEntry>,
-}
-
-#[derive(Clone)]
-pub struct PromotionRegistry {
-    buckets: Arc<BucketRegistry>,
-    index_registry: IndexRegistry,
-    cache: Arc<RwLock<HashMap<String, PromotionFile>>>,
-}
-
-impl PromotionRegistry {
-    pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
-        Self {
-            buckets,
-            index_registry,
-            cache: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    fn key(index_name: &str) -> String {
-        // Sanitize for object-store safety.
-        let safe: String = index_name
-            .chars()
-            .map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
-            .collect();
-        format!("{PROMOTION_PREFIX}/{safe}.json")
-    }
-
-    /// Resolve which bucket's store holds this index's promotion file.
-    /// Same rules as TrialJournal::bucket_for — follows IndexMeta.bucket,
-    /// defaults to primary when metadata is missing.
-    async fn store_for(&self, index_name: &str) -> Result<Arc<dyn object_store::ObjectStore>, String> {
-        let bucket = self.index_registry
-            .get(index_name)
-            .await
-            .map(|m| m.bucket)
-            .unwrap_or_else(|| "primary".to_string());
-        self.buckets.get(&bucket)
-    }
-
-    /// Load (and cache) the promotion file for an index.
-    pub async fn load(&self, index_name: &str) -> Result<PromotionFile, String> {
-        if let Some(cached) = self.cache.read().await.get(index_name) {
-            return Ok(cached.clone());
-        }
-        let store = self.store_for(index_name).await?;
-        let key = Self::key(index_name);
-        let file = match ops::get(&store, &key).await {
-            Ok(bytes) => serde_json::from_slice::<PromotionFile>(&bytes)
-                .map_err(|e| format!("parse promotion file: {e}"))?,
-            Err(_) => PromotionFile {
-                index_name: index_name.to_string(),
-                current: None,
-                history: Vec::new(),
-            },
-        };
-        self.cache.write().await.insert(index_name.to_string(), file.clone());
-        Ok(file)
-    }
-
-    /// Promote a config to the active slot. Pushes the current promotion
-    /// (if any) onto the history stack. Persists before returning — the
-    /// config is durable by the time this call completes.
-    pub async fn promote(
-        &self,
-        index_name: &str,
-        entry: PromotionEntry,
-    ) -> Result<PromotionFile, String> {
-        let mut file = self.load(index_name).await?;
-        if let Some(prior) = file.current.take() {
-            file.history.push(prior);
-            // Cap history to something sensible so this file doesn't grow
-            // unbounded. 50 entries = 50 promotions — way more than any
-            // sane workflow needs.
-            const HISTORY_CAP: usize = 50;
-            if file.history.len() > HISTORY_CAP {
-                let drop = file.history.len() - HISTORY_CAP;
-                file.history.drain(0..drop);
-            }
-        }
-        file.current = Some(entry);
-        file.index_name = index_name.to_string();
-
-        let store = self.store_for(index_name).await?;
-        let key = Self::key(index_name);
-        let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
-        ops::put(&store, &key, json.into()).await?;
-
-        self.cache.write().await.insert(index_name.to_string(), file.clone());
-        tracing::info!(
-            "promoted '{}' to config {:?} (trial={})",
-            index_name, file.current.as_ref().unwrap().config, file.current.as_ref().unwrap().trial_id,
-        );
-        Ok(file)
-    }
-
-    /// Pop the latest promotion back onto the current slot (if any
-    /// history exists). If current is set but history is empty, the
-    /// current promotion is cleared — the index falls back to defaults.
-    pub async fn rollback(&self, index_name: &str) -> Result<PromotionFile, String> {
-        let mut file = self.load(index_name).await?;
-        match file.history.pop() {
-            Some(prev) => {
-                file.current = Some(prev);
-            }
-            None => {
-                if file.current.is_none() {
-                    return Err(format!("no promotion to rollback for '{index_name}'"));
-                }
-                file.current = None;
-            }
-        }
-        let store = self.store_for(index_name).await?;
-        let key = Self::key(index_name);
-        let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
-        ops::put(&store, &key, json.into()).await?;
-        self.cache.write().await.insert(index_name.to_string(), file.clone());
-        tracing::info!("rolled back promotion for '{}'", index_name);
-        Ok(file)
-    }
-
-    /// Get the currently-promoted config (if any). Callers use this to
-    /// pick the right HnswConfig at build time.
-    pub async fn get_current(&self, index_name: &str) -> Option<PromotionEntry> {
-        self.load(index_name).await.ok().and_then(|f| f.current)
-    }
-
-    /// Convenience: return the promoted config or the provided default.
-    pub async fn config_or(&self, index_name: &str, default: HnswConfig) -> HnswConfig {
-        match self.get_current(index_name).await {
-            Some(entry) => entry.config,
-            None => default,
-        }
-    }
-
-    /// List every index that has a promotion recorded (for operator UI).
-    ///
-    /// Federation: scans EVERY registered bucket for promotion files.
-    /// Per-profile buckets each have their own `_hnsw_promotions/` so we
-    /// aggregate across them. Dedups by index_name — if the same index
-    /// somehow has promotion files in multiple buckets, the one from the
-    /// bucket recorded in IndexMeta wins.
-    pub async fn list_all(&self) -> Result<Vec<PromotionFile>, String> {
-        let bucket_infos = self.buckets.list().await;
-        let mut by_name: HashMap<String, PromotionFile> = HashMap::new();
-
-        for b in &bucket_infos {
-            let store = match self.buckets.get(&b.name) {
-                Ok(s) => s,
-                Err(_) => continue,
-            };
-            let keys = ops::list(&store, Some(&format!("{PROMOTION_PREFIX}/")))
-                .await.unwrap_or_default();
-            for key in keys {
-                if !key.ends_with(".json") { continue; }
-                let bytes = match ops::get(&store, &key).await {
-                    Ok(b) => b,
-                    Err(_) => continue,
-                };
-                if let Ok(f) = serde_json::from_slice::<PromotionFile>(&bytes) {
-                    by_name.insert(f.index_name.clone(), f);
-                }
-            }
-        }
-        Ok(by_name.into_values().collect())
-    }
-}

crates/vectord/src/rag.rs

@@ -1,92 +1,12 @@
-/// RAG pipeline: question → embed → search → rerank → generate answer.
-///
-/// The rerank step (added 2026-04-17) uses the LLM as a cross-encoder
-/// between retrieval and generation. This catches cases where the
-/// embedding model scores documents as similar but the content isn't
-/// actually relevant to the question — a known weakness of small
-/// general-purpose embed models on domain-specific text.
+/// RAG pipeline: question → embed → search → retrieve → generate answer.
 
 use object_store::ObjectStore;
 use std::sync::Arc;
 
 use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
-use aibridge::continuation::{generate_continuable, ContinuableOpts, ResponseShape};
 use crate::search::{self, SearchResult};
 use crate::store;
 
-/// Cross-encoder rerank: ask the LLM to re-sort retrieved chunks by
-/// relevance. Falls back to the original order if the model returns
-/// garbage (which happens ~5% of the time with 7B models).
-async fn rerank(
-    question: &str,
-    mut results: Vec<SearchResult>,
-    ai_client: &AiClient,
-) -> Vec<SearchResult> {
-    if results.len() <= 1 {
-        return results;
-    }
-
-    let chunk_list: String = results.iter().enumerate().map(|(i, r)| {
-        let text: String = r.chunk_text.chars().take(200).collect();
-        format!("[{i}] {text}")
-    }).collect::<Vec<_>>().join("\n");
-
-    let resp = ai_client.generate(GenerateRequest {
-        prompt: format!(
-            "Rank these text chunks by relevance to the question.\n\
-            Return ONLY a comma-separated list of indices, most relevant first.\n\n\
-            Question: {question}\n\n\
-            Chunks:\n{chunk_list}\n\n\
-            Ranking:"
-        ),
-        model: None,
-        system: None,
-        temperature: Some(0.0),
-        max_tokens: Some(50),
-        // Reranker returns a comma-separated int list — pure structured
-        // output, zero benefit from hidden reasoning. Opt out to avoid
-        // the empty-response failure mode Phase 21 catalogued.
-        think: Some(false),
-    }).await;
-
-    match resp {
-        Ok(gen_resp) => {
-            let text = gen_resp.text.trim();
-            let indices: Vec<usize> = text
-                .split(|c: char| c == ',' || c.is_whitespace())
-                .filter_map(|s| s.trim().parse::<usize>().ok())
-                .filter(|&i| i < results.len())
-                .collect();
-
-            if indices.is_empty() {
-                tracing::debug!("reranker returned unparseable output: {text}");
-                return results;
-            }
-
-            let mut reranked: Vec<SearchResult> = Vec::with_capacity(results.len());
-            let mut used = vec![false; results.len()];
-            for &i in &indices {
-                if !used[i] {
-                    reranked.push(results[i].clone());
-                    used[i] = true;
-                }
-            }
-            // Append any the model didn't mention (preserves all results).
-            for (i, r) in results.drain(..).enumerate() {
-                if !used[i] {
-                    reranked.push(r);
-                }
-            }
-            tracing::info!("reranker reordered {}/{} chunks", indices.len(), reranked.len());
-            reranked
-        }
-        Err(e) => {
-            tracing::debug!("reranker failed (using original order): {e}");
-            results
-        }
-    }
-}
-
 /// Full RAG answer with provenance.
 #[derive(Debug, Clone, serde::Serialize)]
 pub struct RagResponse {
@@ -131,53 +51,34 @@ pub async fn query(
         });
     }
 
-    // 3. Rerank: ask the LLM to sort retrieved chunks by relevance.
-    //    This cross-encoder step catches cases where embedding similarity
-    //    is high but semantic relevance is low — common with small embed
-    //    models on domain-specific text.
-    let results = rerank(question, results, ai_client).await;
-
-    // 4. Build context from (reranked) top chunks
+    // 3. Build context from retrieved chunks
     let context: String = results.iter().enumerate().map(|(i, r)| {
         format!("[{}] (source: {}, doc: {}) {}", i + 1, r.source, r.doc_id, r.chunk_text)
     }).collect::<Vec<_>>().join("\n\n");
 
-    // 5. Generate answer
+    // 4. Generate answer
     tracing::info!("RAG: generating answer from {} chunks", results.len());
     let prompt = format!(
-        "You are a helpful assistant answering questions about a staffing database.\n\n\
-        Use ONLY the following retrieved records to answer. Be specific — cite names,\n\
-        numbers, skills, and cities from the records. If the context doesn't contain\n\
-        enough information to fully answer, say what you can and note what's missing.\n\
+        "You are a helpful assistant answering questions based on retrieved documents from a data system.\n\n\
+        Use ONLY the following context to answer. If the context doesn't contain enough information, say so.\n\
         Cite sources by their number [1], [2], etc.\n\n\
         Context:\n{context}\n\n\
         Question: {question}\n\n\
         Answer:"
     );
 
-    // Route the answer call through Phase 21's generate_continuable so
-    // a thinking-model empty-response or a mid-JSON truncation self-
-    // repairs instead of silently returning half an answer. Shape is
-    // Text (the answer is prose, not JSON), think is Some(false) to
-    // opt out of hidden reasoning on the hot path. This is the first
-    // production caller of the Phase 21 primitives — see audit finding
-    // "Phase 21 Rust primitives are wired but not CALLED by any
-    // production surface" from 2026-04-21.
-    let mut cont_opts = ContinuableOpts::new("qwen2.5:latest");
-    cont_opts.max_tokens = Some(512);
-    cont_opts.temperature = Some(0.2);
-    cont_opts.shape = ResponseShape::Text;
-    cont_opts.think = Some(false);
-    let outcome = generate_continuable(ai_client, &prompt, &cont_opts).await?;
+    let gen_resp = ai_client.generate(GenerateRequest {
+        prompt,
+        model: None,
+        system: None,
+        temperature: Some(0.2),
+        max_tokens: Some(512),
+    }).await?;
 
     Ok(RagResponse {
-        answer: outcome.text.trim().to_string(),
-        // generate_continuable doesn't surface the model name (sidecar
-        // echoes whatever Ollama loaded). Use the configured tier model
-        // for now; if RAG needs to report the actual resolved model,
-        // the runner can add a post-call ps probe later.
-        model: "qwen2.5:latest".to_string(),
+        answer: gen_resp.text.trim().to_string(),
+        model: gen_resp.model,
         sources: results,
-        tokens_generated: None,
+        tokens_generated: gen_resp.tokens_generated,
     })
 }

crates/vectord/src/refresh.rs

@@ -1,313 +0,0 @@
-//! Phase C: Decoupled embedding refresh.
-//!
-//! When a dataset's row-level data changes, the vector index it feeds is
-//! stale. Historically we coupled ingest and embedding — writing new rows
-//! also re-embedded every row, which is fine at 500 rows but blows up at
-//! 100K+. The llms3.com architecture calls out "asynchronous vector
-//! refresh cycles independent from transactional mutations" as the right
-//! pattern.
-//!
-//! This module implements the refresh side. The ingest path marks
-//! embeddings stale (see catalogd::registry::mark_embeddings_stale); this
-//! code clears that staleness by embedding only rows whose `doc_id` isn't
-//! already in the existing vector index.
-//!
-//! Scope — MVP:
-//! - Reads the dataset's Parquet, extracts (doc_id, text) pairs from named
-//!   columns
-//! - Loads existing embeddings via EmbeddingCache
-//! - Filters to rows whose doc_id is NOT in the existing set
-//! - Chunks, embeds via Ollama, appends to the index parquet
-//! - Clears stale flag on success
-//!
-//! Not in MVP:
-//! - UPDATE semantics (same doc_id, new content) — would need content-hash
-//!   comparison per row
-//! - Large-scale resilience (batching with checkpoints like the
-//!   supervisor) — MVP does it inline
-//! - Lance backend — ADR-019 makes this straightforward later; MVP stays
-//!   on Parquet sidecar indexes
-
-use std::collections::HashSet;
-use std::sync::Arc;
-
-use aibridge::client::{AiClient, EmbedRequest};
-use arrow::array::{Array, Int32Array, Int64Array, StringArray};
-use catalogd::registry::Registry;
-use object_store::ObjectStore;
-
-use crate::chunker::{self, TextChunk};
-use crate::embedding_cache::EmbeddingCache;
-use crate::index_registry::IndexRegistry;
-use crate::store::{self, StoredEmbedding};
-
-#[derive(Debug, Clone, serde::Deserialize)]
-pub struct RefreshRequest {
-    pub index_name: String,
-    /// Column with the document id (row identity).
-    pub id_column: String,
-    /// Column with the text to embed.
-    pub text_column: String,
-    /// Chunk size (chars). Defaults to 500.
-    #[serde(default)]
-    pub chunk_size: Option<usize>,
-    /// Overlap (chars). Defaults to 50.
-    #[serde(default)]
-    pub overlap: Option<usize>,
-}
-
-#[derive(Debug, Clone, serde::Serialize)]
-pub struct RefreshResult {
-    pub index_name: String,
-    pub dataset_name: String,
-    pub pre_existing_docs: usize,
-    pub dataset_docs: usize,
-    pub new_docs_embedded: usize,
-    pub new_chunks_embedded: usize,
-    pub total_embeddings_after: usize,
-    pub duration_secs: f32,
-    pub stale_cleared: bool,
-}
-
-/// Full refresh pipeline. Takes dataset_name as URL param, body has the
-/// column selectors.
-pub async fn refresh_index(
-    dataset_name: &str,
-    req: &RefreshRequest,
-    store_: &Arc<dyn ObjectStore>,
-    registry: &Registry,
-    ai_client: &AiClient,
-    embedding_cache: &EmbeddingCache,
-    index_registry: &IndexRegistry,
-) -> Result<RefreshResult, String> {
-    let t0 = std::time::Instant::now();
-
-    // 1. Find the dataset manifest, pull its object storage key
-    let manifest = registry
-        .get_by_name(dataset_name)
-        .await
-        .ok_or_else(|| format!("dataset not found: {dataset_name}"))?;
-
-    if manifest.objects.is_empty() {
-        return Err(format!("dataset '{dataset_name}' has no object references"));
-    }
-
-    // 2. Read dataset rows — extract (doc_id, text) pairs from the
-    //    specified columns. Multi-object datasets: concat across all.
-    let mut doc_id_to_text: Vec<(String, String)> = Vec::new();
-    for obj in &manifest.objects {
-        let data = storaged::ops::get(store_, &obj.key).await
-            .map_err(|e| format!("read {}: {e}", obj.key))?;
-        let (schema, batches) = shared::arrow_helpers::parquet_to_record_batches(&data)
-            .map_err(|e| format!("parse {}: {e}", obj.key))?;
-
-        let id_idx = schema.index_of(&req.id_column)
-            .map_err(|_| format!("id column '{}' not in dataset schema", req.id_column))?;
-        let text_idx = schema.index_of(&req.text_column)
-            .map_err(|_| format!("text column '{}' not in dataset schema", req.text_column))?;
-
-        for batch in &batches {
-            let text_col = batch
-                .column(text_idx)
-                .as_any()
-                .downcast_ref::<StringArray>()
-                .ok_or_else(|| format!("text column '{}' is not Utf8", req.text_column))?;
-
-            // Accept Utf8, Int32, or Int64 as id — that covers CSV-derived
-            // and Postgres-imported schemas without forcing upstream casts.
-            let id_reader: Box<dyn Fn(usize) -> Option<String>> = {
-                let col = batch.column(id_idx);
-                if let Some(s) = col.as_any().downcast_ref::<StringArray>() {
-                    let s = s.clone();
-                    Box::new(move |row| if s.is_null(row) { None } else { Some(s.value(row).to_string()) })
-                } else if let Some(a) = col.as_any().downcast_ref::<Int32Array>() {
-                    let a = a.clone();
-                    Box::new(move |row| if a.is_null(row) { None } else { Some(a.value(row).to_string()) })
-                } else if let Some(a) = col.as_any().downcast_ref::<Int64Array>() {
-                    let a = a.clone();
-                    Box::new(move |row| if a.is_null(row) { None } else { Some(a.value(row).to_string()) })
-                } else {
-                    return Err(format!(
-                        "id column '{}' must be Utf8, Int32, or Int64 — got {}",
-                        req.id_column,
-                        col.data_type(),
-                    ));
-                }
-            };
-
-            for row in 0..batch.num_rows() {
-                if text_col.is_null(row) { continue; }
-                let Some(id) = id_reader(row) else { continue; };
-                let text = text_col.value(row).to_string();
-                if text.trim().is_empty() { continue; }
-                doc_id_to_text.push((id, text));
-            }
-        }
-    }
-    let dataset_docs = doc_id_to_text.len();
-    tracing::info!("refresh '{}': dataset has {dataset_docs} rows", dataset_name);
-
-    // 3. Load existing embeddings (empty if no index yet)
-    let existing: Vec<StoredEmbedding> = match embedding_cache.get_or_load(&req.index_name).await {
-        Ok(arc) => arc.as_ref().clone(),
-        Err(_) => Vec::new(), // first-time index build
-    };
-    let pre_existing_docs: HashSet<String> = existing
-        .iter()
-        .map(|e| e.doc_id.clone())
-        .collect();
-    let pre_existing_count = pre_existing_docs.len();
-
-    // 4. Delta — rows whose doc_id isn't already indexed
-    let new_rows: Vec<(String, String)> = doc_id_to_text
-        .into_iter()
-        .filter(|(id, _)| !pre_existing_docs.contains(id))
-        .collect();
-    let new_docs = new_rows.len();
-
-    if new_docs == 0 {
-        tracing::info!("refresh '{}': no new docs to embed", dataset_name);
-        // Even on no-op, make sure index metadata is registered so
-        // downstream discovery (profile activation, A/B search) sees
-        // indexes built by earlier refresh calls that predated the
-        // auto-register behavior.
-        let _ = try_update_index_meta(index_registry, &req.index_name, existing.len()).await;
-        registry.clear_embeddings_stale(dataset_name).await?;
-        return Ok(RefreshResult {
-            index_name: req.index_name.clone(),
-            dataset_name: dataset_name.to_string(),
-            pre_existing_docs: pre_existing_count,
-            dataset_docs,
-            new_docs_embedded: 0,
-            new_chunks_embedded: 0,
-            total_embeddings_after: existing.len(),
-            duration_secs: t0.elapsed().as_secs_f32(),
-            stale_cleared: true,
-        });
-    }
-
-    // 5. Chunk the new rows
-    let chunk_size = req.chunk_size.unwrap_or(500);
-    let overlap = req.overlap.unwrap_or(50);
-    let doc_ids: Vec<String> = new_rows.iter().map(|(id, _)| id.clone()).collect();
-    let texts: Vec<String> = new_rows.iter().map(|(_, t)| t.clone()).collect();
-    let chunks: Vec<TextChunk> = chunker::chunk_column(
-        dataset_name, &doc_ids, &texts, chunk_size, overlap,
-    );
-    let new_chunks = chunks.len();
-    tracing::info!("refresh '{}': {} new docs -> {} chunks", dataset_name, new_docs, new_chunks);
-
-    // 6. Embed via Ollama (batched)
-    let batch_size = 32;
-    let mut all_vectors: Vec<Vec<f64>> = Vec::with_capacity(new_chunks);
-    for batch in chunks.chunks(batch_size) {
-        let batch_texts: Vec<String> = batch.iter().map(|c| c.text.clone()).collect();
-        let resp = ai_client
-            .embed(EmbedRequest { texts: batch_texts, model: None })
-            .await
-            .map_err(|e| format!("embed: {e}"))?;
-        all_vectors.extend(resp.embeddings);
-    }
-
-    // 7. Combine existing + new and write back as a single parquet
-    //    (MVP — append-as-rewrite. ADR-019 points to Lance for true native
-    //    append; for the Parquet sidecar we rewrite at refresh time.)
-    let mut new_stored: Vec<StoredEmbedding> = existing.clone();
-    for (chunk, vector) in chunks.into_iter().zip(all_vectors.iter()) {
-        new_stored.push(StoredEmbedding {
-            source: chunk.source,
-            doc_id: chunk.doc_id,
-            chunk_idx: chunk.chunk_idx,
-            chunk_text: chunk.text,
-            vector: vector.iter().map(|&x| x as f32).collect(),
-        });
-    }
-
-    // We have to reconstruct chunks + vectors for store_embeddings — but
-    // store_embeddings takes &[TextChunk] and vectors separately. Convert
-    // the combined StoredEmbedding back to that shape.
-    let combined_chunks: Vec<TextChunk> = new_stored
-        .iter()
-        .map(|e| TextChunk {
-            source: e.source.clone(),
-            doc_id: e.doc_id.clone(),
-            chunk_idx: e.chunk_idx,
-            text: e.chunk_text.clone(),
-        })
-        .collect();
-    let combined_vectors: Vec<Vec<f64>> = new_stored
-        .iter()
-        .map(|e| e.vector.iter().map(|&f| f as f64).collect())
-        .collect();
-
-    let _key = store::store_embeddings(
-        store_, &req.index_name, &combined_chunks, &combined_vectors,
-    ).await?;
-
-    // 8. Evict embedding cache — next read will pick up the new file
-    let _ = embedding_cache.evict(&req.index_name).await;
-
-    // 9. Update index registry metadata (row/chunk counts; others unchanged
-    //    so we don't disturb the existing entry if present)
-    let _ = try_update_index_meta(index_registry, &req.index_name, new_stored.len()).await;
-
-    // 10. Clear stale flag
-    registry.clear_embeddings_stale(dataset_name).await?;
-
-    let total = new_stored.len();
-    Ok(RefreshResult {
-        index_name: req.index_name.clone(),
-        dataset_name: dataset_name.to_string(),
-        pre_existing_docs: pre_existing_count,
-        dataset_docs,
-        new_docs_embedded: new_docs,
-        new_chunks_embedded: new_chunks,
-        total_embeddings_after: total,
-        duration_secs: t0.elapsed().as_secs_f32(),
-        stale_cleared: true,
-    })
-}
-
-/// Best-effort refresh of index registry metadata.
-/// - If the index already exists, bump the chunk_count.
-/// - If it's brand new (first-time build via refresh), REGISTER it so
-///   downstream discovery (profile activation, A/B search) can find it.
-async fn try_update_index_meta(
-    index_registry: &IndexRegistry,
-    index_name: &str,
-    chunk_count: usize,
-) -> Result<(), String> {
-    if let Some(mut meta) = index_registry.get(index_name).await {
-        meta.chunk_count = chunk_count;
-        return index_registry.register(meta).await;
-    }
-    // First-time registration — infer reasonable defaults. Convention:
-    // index name `{source}_v1` or `{source}_vN` implies a source dataset
-    // named by stripping the `_vN` suffix. Otherwise use the index name
-    // as the source and let the caller patch later.
-    let source = match index_name.rsplit_once('_') {
-        Some((base, suffix)) if suffix.starts_with('v') && suffix[1..].chars().all(|c| c.is_ascii_digit()) => {
-            base.to_string()
-        }
-        _ => index_name.to_string(),
-    };
-    let meta = crate::index_registry::IndexMeta {
-        index_name: index_name.to_string(),
-        source,
-        model_name: "nomic-embed-text".to_string(),
-        model_version: "latest".to_string(),
-        dimensions: 768,
-        chunk_count,
-        doc_count: chunk_count,
-        chunk_size: 500,
-        overlap: 50,
-        storage_key: format!("vectors/{index_name}.parquet"),
-        created_at: chrono::Utc::now(),
-        build_time_secs: 0.0,
-        chunks_per_sec: 0.0,
-        bucket: "primary".to_string(),
-        vector_backend: shared::types::VectorBackend::Parquet,
-        id_prefix: None,
-    };
-    index_registry.register(meta).await
-}

crates/vectord/src/trial.rs

@@ -1,240 +0,0 @@
-/// Trial journal for HNSW parameter tuning.
-///
-/// Every HNSW build+eval is recorded as a Trial. The journal is append-only
-/// and stored under `_hnsw_trials/{index_name}/` as batched JSONL files —
-/// an AI agent iterating on configs reads prior trials to decide what to
-/// try next, and writes a new trial on each attempt.
-///
-/// Storage uses the shared `storaged::append_log::AppendLog` so appends are
-/// write-once (new file per batch) rather than rewriting a single growing
-/// JSONL on every event. See `append_log.rs` for the full rationale.
-
-use chrono::{DateTime, Utc};
-use object_store::ObjectStore;
-use serde::{Deserialize, Serialize};
-use std::collections::HashMap;
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use storaged::append_log::AppendLog;
-use storaged::registry::BucketRegistry;
-
-use crate::index_registry::IndexRegistry;
-
-/// HNSW build/search parameters the agent can tune.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct HnswConfig {
-    pub ef_construction: usize,
-    pub ef_search: usize,
-    #[serde(default)]
-    pub seed: Option<u64>,
-}
-
-impl Default for HnswConfig {
-    /// Production default, locked in 2026-04-16 based on trial grid against
-    /// resumes_100k_v2 (100K vectors, 20 queries, recall@10):
-    ///   ec=20 es=30  → recall 0.960, p50 509us, build 8s
-    ///   ec=80 es=30  → recall 1.000, p50 873us, build 230s  ← sweet spot
-    ///   ec=200 es=30 → recall 1.000, p50 874us, build 106s  (no recall gain)
-    ///
-    /// `ec=80` is the smallest value that reaches 100% recall. Higher values
-    /// waste build time. `es=30` gives faster search than `es=100` with no
-    /// recall penalty at this scale.
-    fn default() -> Self {
-        Self {
-            ef_construction: 80,
-            ef_search: 30,
-            seed: None,
-        }
-    }
-}
-
-/// Metrics collected on every trial. All latencies in microseconds.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct TrialMetrics {
-    pub build_time_secs: f32,
-    pub search_latency_p50_us: f32,
-    pub search_latency_p95_us: f32,
-    pub search_latency_p99_us: f32,
-    pub recall_at_k: f32,
-    pub memory_bytes: u64,
-    pub vectors: usize,
-    pub eval_queries: usize,
-    /// Brute-force latency for comparison — how much speedup did HNSW buy us?
-    pub brute_force_latency_us: f32,
-}
-
-/// A single tuning attempt.
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct Trial {
-    pub id: String,
-    pub index_name: String,
-    pub eval_set: String,
-    pub config: HnswConfig,
-    pub metrics: TrialMetrics,
-    pub created_at: DateTime<Utc>,
-    /// Free-form note — the agent can record why it tried this config.
-    #[serde(default)]
-    pub note: Option<String>,
-}
-
-impl Trial {
-    pub fn new_id() -> String {
-        format!(
-            "trial-{}-{}",
-            Utc::now().timestamp_millis(),
-            &uuid::Uuid::new_v4().to_string()[..8]
-        )
-    }
-}
-
-/// Per-index append log, lazy-created on first write.
-///
-/// Federation layer 2: the journal resolves each index's bucket from the
-/// index registry and writes its JSONL batches to THAT bucket, not
-/// primary. Back-compat is preserved by `IndexMeta::bucket` defaulting
-/// to "primary" for pre-federation indexes. Indexes the registry has
-/// never heard of (edge case — trials run before first register) fall
-/// through to primary as well.
-#[derive(Clone)]
-pub struct TrialJournal {
-    buckets: Arc<BucketRegistry>,
-    index_registry: IndexRegistry,
-    /// Cache per (bucket, index) AppendLog so the in-memory buffer persists
-    /// across calls. Keyed by `(bucket, index_name)` so moving an index
-    /// between buckets is clean — the old journal stays intact.
-    logs: Arc<RwLock<HashMap<(String, String), Arc<AppendLog>>>>,
-}
-
-impl TrialJournal {
-    pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
-        Self {
-            buckets,
-            index_registry,
-            logs: Arc::new(RwLock::new(HashMap::new())),
-        }
-    }
-
-    fn prefix(index_name: &str) -> String {
-        format!("_hnsw_trials/{}", index_name)
-    }
-
-    /// Resolve which bucket holds this index's trial artifacts.
-    /// Falls back to primary for indexes without recorded metadata.
-    async fn bucket_for(&self, index_name: &str) -> String {
-        self.index_registry
-            .get(index_name)
-            .await
-            .map(|m| m.bucket)
-            .unwrap_or_else(|| "primary".to_string())
-    }
-
-    async fn log_for(&self, index_name: &str) -> Result<Arc<AppendLog>, String> {
-        let bucket = self.bucket_for(index_name).await;
-        let key = (bucket.clone(), index_name.to_string());
-
-        if let Some(log) = self.logs.read().await.get(&key) {
-            return Ok(log.clone());
-        }
-        let mut guard = self.logs.write().await;
-        if let Some(log) = guard.get(&key) {
-            return Ok(log.clone());
-        }
-        let store = self.buckets.get(&bucket)?;
-        // Trials arrive one at a time during human/agent iteration — a low
-        // threshold gives "hit /trials and see my latest attempt" immediacy
-        // without creating one file per event.
-        let log = Arc::new(
-            AppendLog::new(store, Self::prefix(index_name))
-                .with_flush_threshold(4),
-        );
-        guard.insert(key, log.clone());
-        Ok(log)
-    }
-
-    /// Append a trial record. In-memory buffered; persisted in batches.
-    pub async fn append(&self, trial: &Trial) -> Result<(), String> {
-        let line = serde_json::to_vec(trial).map_err(|e| e.to_string())?;
-        let log = self.log_for(&trial.index_name).await?;
-        log.append(line).await
-    }
-
-    /// Read all trials for an index (flushed batches + unflushed buffer).
-    pub async fn list(&self, index_name: &str) -> Result<Vec<Trial>, String> {
-        let log = self.log_for(index_name).await?;
-        let lines = log.read_all().await?;
-        let mut trials = Vec::with_capacity(lines.len());
-        for line in lines {
-            match serde_json::from_slice::<Trial>(&line) {
-                Ok(t) => trials.push(t),
-                Err(e) => tracing::warn!("trial journal: skip malformed line: {e}"),
-            }
-        }
-        Ok(trials)
-    }
-
-    /// Explicit flush for callers that want write-through semantics
-    /// (e.g. an agent that wants to commit a trial before querying stats).
-    pub async fn flush(&self, index_name: &str) -> Result<(), String> {
-        let log = self.log_for(index_name).await?;
-        log.flush().await
-    }
-
-    /// Compact all batch files for an index into one.
-    pub async fn compact(&self, index_name: &str) -> Result<storaged::append_log::CompactStats, String> {
-        let log = self.log_for(index_name).await?;
-        log.compact().await
-    }
-
-    /// Current champion for an index by the named metric.
-    /// Valid metrics: `recall`, `latency`, `pareto`.
-    ///
-    /// The `pareto` strategy is a placeholder — J should tune the scoring
-    /// function to match what matters in production. Right now it's a simple
-    /// weighted sum.
-    pub async fn best(
-        &self,
-        index_name: &str,
-        metric: &str,
-    ) -> Result<Option<Trial>, String> {
-        let trials = self.list(index_name).await?;
-        if trials.is_empty() {
-            return Ok(None);
-        }
-
-        let best = match metric {
-            "recall" => trials
-                .into_iter()
-                .max_by(|a, b| {
-                    a.metrics
-                        .recall_at_k
-                        .partial_cmp(&b.metrics.recall_at_k)
-                        .unwrap_or(std::cmp::Ordering::Equal)
-                })
-                .unwrap(),
-            "latency" => trials
-                .into_iter()
-                .min_by(|a, b| {
-                    a.metrics
-                        .search_latency_p95_us
-                        .partial_cmp(&b.metrics.search_latency_p95_us)
-                        .unwrap_or(std::cmp::Ordering::Equal)
-                })
-                .unwrap(),
-            "pareto" | _ => trials
-                .into_iter()
-                .max_by(|a, b| pareto_score(a).partial_cmp(&pareto_score(b)).unwrap())
-                .unwrap(),
-        };
-        Ok(Some(best))
-    }
-}
-
-/// Simple Pareto-style score: reward recall, penalize p95 latency.
-/// Tunable — J should swap this in production to match what matters.
-fn pareto_score(t: &Trial) -> f32 {
-    // Recall is [0, 1]. Latency is us — assume 100us baseline.
-    let recall = t.metrics.recall_at_k;
-    let latency_penalty = (t.metrics.search_latency_p95_us / 1000.0).min(1.0); // cap at 1ms
-    recall - 0.2 * latency_penalty
-}

data/_catalog/manifests/07c119c3-2e59-463c-89a8-7980a849b6d6.json

@@ -1,159 +0,0 @@
-{
-  "id": "07c119c3-2e59-463c-89a8-7980a849b6d6",
-  "name": "workers_500k",
-  "schema_fingerprint": "71686e497f5ee2fc77d50b13c39a46d89ca3d7673e9a0879e67c2db476764b25",
-  "objects": [
-    {
-      "bucket": "primary",
-      "key": "datasets/workers_500k.parquet",
-      "size_bytes": 323794818,
-      "created_at": "2026-04-17T05:57:41.972944716Z"
-    }
-  ],
-  "created_at": "2026-04-17T05:57:41.972945960Z",
-  "updated_at": "2026-04-17T05:57:41.973093967Z",
-  "description": "",
-  "owner": "",
-  "sensitivity": "pii",
-  "columns": [
-    {
-      "name": "worker_id",
-      "data_type": "Int64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "name",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "role",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "email",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "phone",
-      "data_type": "Int64",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "city",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "state",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "zip",
-      "data_type": "Int64",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "skills",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "certifications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "archetype",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "reliability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "responsiveness",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "engagement",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "compliance",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "availability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "communications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "resume_text",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    }
-  ],
-  "lineage": {
-    "source_system": "csv",
-    "source_file": "workers_500k.csv",
-    "ingest_job": "ingest-1776405461972",
-    "ingest_timestamp": "2026-04-17T05:57:41.972944716Z",
-    "parent_datasets": []
-  },
-  "freshness": null,
-  "tags": [],
-  "row_count": 500000,
-  "last_embedded_at": null,
-  "embedding_stale_since": null,
-  "embedding_refresh_policy": null
-}

data/_catalog/manifests/0ae8f29b-bf70-4c9d-8b64-6ea7cb7f76a1.json

@@ -0,0 +1,72 @@
+{
+  "id": "0ae8f29b-bf70-4c9d-8b64-6ea7cb7f76a1",
+  "name": "rust_in_tech_stack",
+  "schema_fingerprint": "314ee8ede68287240d7cec1403d252787c8ceb4ee2afa72cffcb0853ac3ac889",
+  "objects": [
+    {
+      "bucket": "data",
+      "key": "datasets/rust_in_tech_stack.parquet",
+      "size_bytes": 99623,
+      "created_at": "2026-03-28T03:32:50.769271381Z"
+    }
+  ],
+  "created_at": "2026-03-28T03:32:50.769272195Z",
+  "updated_at": "2026-03-28T03:32:50.769710478Z",
+  "description": "",
+  "owner": "",
+  "sensitivity": null,
+  "columns": [
+    {
+      "name": "source_file",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "conversation_title",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "turn_number",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "role",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "content",
+      "data_type": "Utf8",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    },
+    {
+      "name": "page_number",
+      "data_type": "Int32",
+      "sensitivity": null,
+      "description": "",
+      "is_pii": false
+    }
+  ],
+  "lineage": {
+    "source_system": "pdf",
+    "source_file": "Rust in Tech Stack.pdf",
+    "ingest_job": "ingest-1774668770769",
+    "ingest_timestamp": "2026-03-28T03:32:50.769271381Z",
+    "parent_datasets": []
+  },
+  "freshness": null,
+  "tags": [],
+  "row_count": 49
+}

data/_catalog/manifests/0efdef2d-52bc-4154-82d4-056bb3a1915d.json

@@ -1,159 +0,0 @@
-{
-  "id": "0efdef2d-52bc-4154-82d4-056bb3a1915d",
-  "name": "ethereal_workers",
-  "schema_fingerprint": "9a1286ffada5390b459b217f56263e5ebffec1520ea29bab0be2efc6d5381adc",
-  "objects": [
-    {
-      "bucket": "primary",
-      "key": "datasets/ethereal_workers.parquet",
-      "size_bytes": 6991716,
-      "created_at": "2026-04-17T03:23:29.399966984Z"
-    }
-  ],
-  "created_at": "2026-04-17T03:23:29.399967772Z",
-  "updated_at": "2026-04-17T03:23:29.400227081Z",
-  "description": "",
-  "owner": "",
-  "sensitivity": "pii",
-  "columns": [
-    {
-      "name": "worker_id",
-      "data_type": "Int64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "name",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "role",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "email",
-      "data_type": "Utf8",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "phone",
-      "data_type": "Int64",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "city",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "state",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "zip",
-      "data_type": "Int64",
-      "sensitivity": "pii",
-      "description": "",
-      "is_pii": true
-    },
-    {
-      "name": "skills",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "certifications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "archetype",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "reliability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "responsiveness",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "engagement",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "compliance",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "availability",
-      "data_type": "Float64",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "communications",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    },
-    {
-      "name": "resume_text",
-      "data_type": "Utf8",
-      "sensitivity": null,
-      "description": "",
-      "is_pii": false
-    }
-  ],
-  "lineage": {
-    "source_system": "csv",
-    "source_file": "ethereal_workers.csv",
-    "ingest_job": "ingest-1776396209399",
-    "ingest_timestamp": "2026-04-17T03:23:29.399966984Z",
-    "parent_datasets": []
-  },
-  "freshness": null,
-  "tags": [],
-  "row_count": 10000,
-  "last_embedded_at": null,
-  "embedding_stale_since": null,
-  "embedding_refresh_policy": null
-}