Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 rootandClaude Opus 4.6
|
4d64e4a618 | ||
|
rootandClaude Opus 4.6
|
3f19ad5d93 | ||
|
rootandClaude Opus 4.6
|
bd68b662b7 | ||
|
rootandClaude Opus 4.6
|
3cf000ccf3 | ||
|
rootandClaude Opus 4.6
|
1dcbe8e04c | ||
|
rootandClaude Opus 4.6
|
2189a6dbee | ||
|
rootandClaude Opus 4.6
|
bf033b2d9d | ||
|
rootandClaude Opus 4.6
|
0756075c6d | ||
|
rootandClaude Opus 4.6
|
8ca4ba5717 | ||
|
rootandClaude Opus 4.6
|
c49b30053b |
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"lakehouse": {
|
||||
"command": "bun",
|
||||
"args": ["run", "/home/profit/lakehouse/mcp-server/index.ts"],
|
||||
"env": {
|
||||
"LAKEHOUSE_URL": "http://localhost:3100",
|
||||
"MCP_TRANSPORT": "stdio"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Generated
+399
-3647
File diff suppressed because it is too large
Load Diff
+1
-4
@@ -12,8 +12,6 @@ members = [
|
||||
"crates/journald",
|
||||
"crates/gateway",
|
||||
"crates/ui",
|
||||
"crates/lance-bench",
|
||||
"crates/vectord-lance",
|
||||
]
|
||||
|
||||
[workspace.dependencies]
|
||||
@@ -47,5 +45,4 @@ csv = "1"
|
||||
lopdf = "0.35"
|
||||
encoding_rs = "0.8"
|
||||
instant-distance = "0.6"
|
||||
tokio-postgres = { version = "0.7", features = ["with-serde_json-1", "with-chrono-0_4", "with-uuid-1"] }
|
||||
mysql_async = { version = "0.34", default-features = false, features = ["minimal"] }
|
||||
tokio-postgres = { version = "0.7", features = ["with-serde_json-1", "with-chrono-0_4"] }
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
# PRD: Observer Knowledge Base
|
||||
|
||||
**Status:** Active — extraction pipeline proven, needs multi-pass processing via LLM Team modes
|
||||
**Created:** 2026-03-28
|
||||
**Owner:** J
|
||||
**Branch:** `observer-kb`
|
||||
**Parent:** Lakehouse core (crates/)
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
AI conversations generate massive amounts of knowledge that's lost to context windows, hallucination, and scope drift. Current approach (single-pass extraction into categories) produces disorganized, hard-to-validate output. Reading raw pages is not useful — the system needs to process, clean, structure, and curate knowledge like an editorial team.
|
||||
|
||||
## Vision
|
||||
|
||||
A publishing pipeline that takes raw AI conversation exports and produces well-curated, validated knowledge documents. The output should be something you hand to a team — organized, filtered, and clearly presented. Questions anchor the content, with branching connections showing what was asked, what was answered, and how reasoning was directed.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Multi-Pass Processing via LLM Team Modes
|
||||
|
||||
The key insight: devop.live already has 20 analysis modes that process content through structured workflows. The Observer KB should route uploaded data through these modes automatically.
|
||||
|
||||
```
|
||||
Raw PDF Upload
|
||||
│
|
||||
▼ Mode: EXTRACT
|
||||
Structured facts, entities, relationships
|
||||
│
|
||||
▼ Mode: VALIDATOR
|
||||
Fact-check claims, score 1-10 confidence
|
||||
│
|
||||
▼ Mode: HALLUCINATION
|
||||
Verify each claim, flag likely hallucinations
|
||||
│
|
||||
▼ Mode: CONSENSUS
|
||||
Multiple passes iterate until extraction converges
|
||||
│
|
||||
▼ Mode: REDTEAM
|
||||
Find flaws in knowledge, patch them
|
||||
│
|
||||
▼ Mode: PIPELINE (custom)
|
||||
Clean → Q&A structure → topic grouping → value ranking
|
||||
│
|
||||
▼ RENDER
|
||||
Well-curated document: questions as anchors, branching connections,
|
||||
what was asked → answered → directed
|
||||
```
|
||||
|
||||
### Integration with devop.live (LLM Team)
|
||||
|
||||
The Observer KB calls the LLM Team API to run modes against extracted content:
|
||||
|
||||
| Mode | Purpose in KB |
|
||||
|---|---|
|
||||
| **extract** | Structured facts, entities, relationships from raw text |
|
||||
| **validator** | Fact-check extracted claims, confidence scores |
|
||||
| **hallucination** | Verify each factual claim independently |
|
||||
| **drift** | Process N times, find consistent vs uncertain content |
|
||||
| **redteam** | Find flaws in extracted knowledge, improve |
|
||||
| **consensus** | Iterate until multiple passes agree on correct extraction |
|
||||
| **pipeline** | Chain: clean → extract → validate → organize → render |
|
||||
| **research** | Generate follow-up questions, research in parallel |
|
||||
| **blindassembly** | Split content into parts, process independently, reassemble |
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Upload Portal (Observer KB :3400)
|
||||
│
|
||||
├── PDF → Lakehouse ingestd (text extraction, Parquet storage)
|
||||
│
|
||||
▼
|
||||
Processing Pipeline
|
||||
│
|
||||
├── Call devop.live API (:5000) with each mode
|
||||
├── Each mode processes content through local Ollama models
|
||||
├── Results stored back in Lakehouse (Parquet + catalog)
|
||||
│
|
||||
▼
|
||||
Knowledge Store
|
||||
│
|
||||
├── knowledge_facts: verified claims with confidence scores
|
||||
├── knowledge_qa: question → answer pairs with context
|
||||
├── knowledge_topics: auto-generated categories
|
||||
├── knowledge_connections: how topics relate across conversations
|
||||
├── knowledge_contradictions: flagged conflicts between conversations
|
||||
├── knowledge_validated: claims that passed hallucination check
|
||||
│
|
||||
▼
|
||||
Rendering
|
||||
│
|
||||
├── Structured knowledge view (web UI)
|
||||
├── Exportable curated document (PDF/markdown)
|
||||
└── Question-anchored navigation (Q → A → follow-up → related)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Processing Pipeline Detail
|
||||
|
||||
### Pass 1: Clean + Structure
|
||||
- Strip PDF artifacts (headers, footers, page numbers)
|
||||
- Identify conversation turns (user/AI)
|
||||
- Fix encoding issues (ligatures, special chars)
|
||||
- Normalize formatting
|
||||
|
||||
### Pass 2: Q&A Extraction
|
||||
- Identify questions asked by user
|
||||
- Map each question to its answer
|
||||
- Track follow-up questions and how they branch
|
||||
- Build question→answer→follow-up chains
|
||||
|
||||
### Pass 3: Fact Extraction + Validation
|
||||
- Run EXTRACT mode: pull structured facts, entities, relationships
|
||||
- Run VALIDATOR mode: score each fact 1-10 confidence
|
||||
- Run HALLUCINATION mode: verify claims independently
|
||||
- Flag contradictions between conversations
|
||||
|
||||
### Pass 4: Organize + Curate
|
||||
- Run CONSENSUS mode: multiple passes converge on correct categorization
|
||||
- Auto-generate topic categories from content (not predefined)
|
||||
- Rank content by value (decisions > insights > discussion > noise)
|
||||
- Connect related content across conversations
|
||||
|
||||
### Pass 5: Render
|
||||
- Question-anchored view: each question is a node, answers branch from it
|
||||
- Topic-grouped view: knowledge organized by auto-generated categories
|
||||
- Timeline view: how thinking evolved across conversations
|
||||
- Export as clean markdown/PDF
|
||||
|
||||
---
|
||||
|
||||
## Current State (as of 2026-03-28)
|
||||
|
||||
### What's Built
|
||||
- PDF ingestion (8 PDFs, 477 pages extracted)
|
||||
- Single-pass knowledge extraction (496 topics, 300 decisions, 100 insights)
|
||||
- Semantic search via HNSW (27ms over 477 chunks)
|
||||
- RAG (question → search → LLM answer with sources)
|
||||
- 5-pass deep analysis (corpus overview, patterns, evolution, gaps, synthesis)
|
||||
- Observer KB web UI (knowledge-first, not page-first)
|
||||
- Upload portal with drag & drop
|
||||
|
||||
### What's Not Built Yet
|
||||
- Integration with devop.live LLM Team modes
|
||||
- Multi-pass processing pipeline (extract → validate → hallucination check)
|
||||
- Q&A chain extraction (question → answer → follow-up mapping)
|
||||
- Contradiction detection between conversations
|
||||
- Curated document export (PDF/markdown)
|
||||
- Question-anchored navigation
|
||||
- Auto-processing on upload (currently manual script)
|
||||
|
||||
### What Needs to Happen Next Session
|
||||
1. **Integrate with devop.live API** — call modes programmatically
|
||||
2. **Build the multi-pass pipeline** — extract → validate → hallucinate-check → organize
|
||||
3. **Q&A chain extraction** — questions as anchors, answers branching
|
||||
4. **Auto-process on upload** — no manual script, pipeline runs automatically
|
||||
5. **Render curated output** — readable, hand-it-to-someone quality
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Storage:** Lakehouse storaged (Parquet on object storage)
|
||||
- **Catalog:** Lakehouse catalogd (metadata, PII, lineage)
|
||||
- **Query:** Lakehouse queryd (DataFusion SQL, hot cache, paged results)
|
||||
- **Vectors:** Lakehouse vectord (HNSW search, RAG)
|
||||
- **AI Processing:** devop.live LLM Team API (20 modes, multiple models)
|
||||
- **AI Runtime:** Ollama (local models — qwen2.5, mistral, gemma2, nomic-embed-text)
|
||||
- **UI:** Pure HTML/JS on :3400
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals (for now)
|
||||
|
||||
- Real-time conversation ingestion (PDF exports only)
|
||||
- Multi-user collaboration
|
||||
- Cloud deployment
|
||||
- Editing extracted knowledge from UI (read-only for now)
|
||||
@@ -0,0 +1,286 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Observer KB</title>
|
||||
<style>
|
||||
:root{--bg:#0a0a0f;--s1:#12121a;--s2:#1a1a28;--b:#2a2a3a;--t:#e0e0e8;--d:#888898;--a:#6c5ce7;--ok:#00d2a0;--err:#ff6b6b;--warn:#ffa94d;--f:'JetBrains Mono','Fira Code',monospace}
|
||||
*{margin:0;padding:0;box-sizing:border-box}body{font-family:var(--f);background:var(--bg);color:var(--t);font-size:13px;min-height:100vh}
|
||||
.app{max-width:1200px;margin:0 auto;padding:20px}
|
||||
header{display:flex;align-items:center;gap:20px;padding:16px 0;border-bottom:1px solid var(--b);margin-bottom:20px}
|
||||
header h1{font-size:20px;color:var(--a)}
|
||||
.tabs{display:flex;gap:2px}.tab{background:none;border:none;color:var(--d);font-family:var(--f);font-size:12px;padding:10px 16px;cursor:pointer;border-bottom:2px solid transparent}
|
||||
.tab:hover{color:var(--t)}.tab.active{color:var(--a);border-bottom-color:var(--a)}
|
||||
.hdr-stats{margin-left:auto;font-size:11px;color:var(--d)}
|
||||
.btn{background:var(--a);color:white;border:none;padding:8px 16px;border-radius:6px;font-family:var(--f);font-size:12px;cursor:pointer;font-weight:600}
|
||||
.btn:hover{opacity:.9}.btn:disabled{opacity:.5}.btn-sm{padding:5px 10px;font-size:11px}
|
||||
.hidden{display:none}
|
||||
.stats-bar{display:grid;grid-template-columns:repeat(auto-fill,minmax(130px,1fr));gap:8px;margin-bottom:20px}
|
||||
.stat{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:12px;text-align:center}
|
||||
.stat-val{font-size:22px;font-weight:700;color:var(--a)}.stat-label{font-size:10px;color:var(--d);text-transform:uppercase;margin-top:2px}
|
||||
.section{margin-bottom:24px}.section h2{font-size:13px;color:var(--d);text-transform:uppercase;letter-spacing:.08em;margin-bottom:10px;border-bottom:1px solid var(--b);padding-bottom:6px}
|
||||
.entry{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:14px;margin-bottom:8px}
|
||||
.badge{font-size:9px;text-transform:uppercase;letter-spacing:.08em;padding:2px 8px;border-radius:3px;display:inline-block;margin-bottom:6px;font-weight:600}
|
||||
.badge-fact{background:var(--a);color:white}.badge-entity{background:var(--ok);color:white}
|
||||
.badge-decision{background:#e67e22;color:white}.badge-insight{background:#3498db;color:white}
|
||||
.badge-question{background:var(--warn);color:#000}.badge-verified{background:var(--ok);color:white}
|
||||
.entry-text{font-size:12px;line-height:1.6}.entry-src{font-size:10px;color:var(--d);margin-top:4px}
|
||||
.cat-grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(250px,1fr));gap:8px;margin-bottom:16px}
|
||||
.cat-card{background:var(--s1);border:1px solid var(--b);border-radius:8px;padding:14px;cursor:pointer;border-left:3px solid var(--a);transition:all .15s}
|
||||
.cat-card:hover{border-color:var(--a)}.cat-title{font-weight:600;font-size:13px}.cat-count{font-size:10px;color:var(--d);margin-top:2px}
|
||||
.search-row{display:flex;gap:8px;margin-bottom:16px}
|
||||
.search-row input{flex:1;background:var(--s1);border:2px solid var(--b);color:var(--t);font-family:var(--f);font-size:14px;padding:12px;border-radius:8px;outline:none}
|
||||
.search-row input:focus{border-color:var(--a)}
|
||||
.ask-input{width:100%;background:var(--s1);border:2px solid var(--b);color:var(--t);font-family:var(--f);font-size:13px;padding:12px;border-radius:8px;resize:vertical;min-height:80px;outline:none;margin-bottom:8px}
|
||||
.ask-input:focus{border-color:var(--a)}
|
||||
.ai-answer{background:var(--s2);border:1px solid var(--b);border-radius:8px;padding:16px;margin-top:12px;line-height:1.7;white-space:pre-wrap}
|
||||
.ai-meta{font-size:10px;color:var(--d);margin-top:6px}
|
||||
.upload-zone{border:2px dashed var(--b);border-radius:12px;padding:36px;text-align:center;cursor:pointer;margin-bottom:16px}
|
||||
.upload-zone:hover{border-color:var(--a);background:rgba(108,92,231,.03)}
|
||||
.upload-zone h3{margin-bottom:4px}.upload-zone p{color:var(--d);font-size:11px}
|
||||
.file-item{display:flex;align-items:center;justify-content:space-between;padding:8px 12px;border-bottom:1px solid var(--b);font-size:12px}
|
||||
.done{color:var(--ok)}.failed{color:var(--err)}.uploading{color:var(--a)}
|
||||
.result{background:var(--s2);border:1px solid var(--b);border-radius:8px;padding:12px;margin-bottom:6px;border-left:3px solid var(--a)}
|
||||
.result-score{font-size:10px;color:var(--a);margin-bottom:2px}.result-text{font-size:12px;line-height:1.5}
|
||||
.empty{color:var(--d);text-align:center;padding:32px}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
<header>
|
||||
<h1>OBSERVER KB</h1>
|
||||
<div class="tabs">
|
||||
<button class="tab active" onclick="showTab('knowledge')">Knowledge</button>
|
||||
<button class="tab" onclick="showTab('facts')">Facts & Entities</button>
|
||||
<button class="tab" onclick="showTab('explore')">Search</button>
|
||||
<button class="tab" onclick="showTab('ask')">Ask AI</button>
|
||||
<button class="tab" onclick="showTab('upload')">Ingest</button>
|
||||
</div>
|
||||
<div class="hdr-stats" id="hdr-stats"></div>
|
||||
</header>
|
||||
|
||||
<div id="tab-knowledge"></div>
|
||||
<div id="tab-facts" class="hidden"></div>
|
||||
<div id="tab-explore" class="hidden"></div>
|
||||
<div id="tab-ask" class="hidden"></div>
|
||||
<div id="tab-upload" class="hidden"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const API=`http://${location.hostname}:3100`;
|
||||
const $=id=>document.getElementById(id);
|
||||
async function q(sql){return(await(await fetch(API+'/query/sql',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({sql})})).json())}
|
||||
async function searchHNSW(query,k=10){try{return(await(await fetch(API+'/vectors/hnsw/search',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',query,top_k:k})})).json()).results||[]}catch{return[]}}
|
||||
|
||||
function showTab(name){
|
||||
['knowledge','facts','explore','ask','upload'].forEach((t,i)=>{
|
||||
$('tab-'+t).classList.toggle('hidden',t!==name);
|
||||
document.querySelectorAll('.tab')[i].classList.toggle('active',t===name);
|
||||
});
|
||||
({knowledge:loadKnowledge,facts:loadFacts,explore:loadExplore,ask:loadAsk,upload:loadUpload})[name]();
|
||||
}
|
||||
|
||||
function makeEntry(type,text,source){
|
||||
const e=document.createElement('div');e.className='entry';
|
||||
const b=document.createElement('span');b.className='badge badge-'+type;b.textContent=type;
|
||||
const t=document.createElement('div');t.className='entry-text';t.textContent=text;
|
||||
const s=document.createElement('div');s.className='entry-src';s.textContent=source;
|
||||
e.append(b,t,s);return e;
|
||||
}
|
||||
|
||||
// === KNOWLEDGE (categories + decisions + insights) ===
|
||||
async function loadKnowledge(){
|
||||
const el=$('tab-knowledge');el.textContent='';
|
||||
try{
|
||||
const[topics,decisions,insights,questions,extractions]=await Promise.all([
|
||||
q("SELECT topic,COUNT(*) as n,COUNT(DISTINCT conversation) as c FROM knowledge_topics GROUP BY topic ORDER BY n DESC LIMIT 15"),
|
||||
q("SELECT conversation,page,decision FROM knowledge_decisions LIMIT 12"),
|
||||
q("SELECT conversation,page,insight FROM knowledge_insights LIMIT 8"),
|
||||
q("SELECT COUNT(*) as n FROM knowledge_questions"),
|
||||
q("SELECT type,COUNT(*) as n FROM processed_extractions GROUP BY type"),
|
||||
]);
|
||||
|
||||
const factCount=extractions.rows.find(r=>r.type==='fact')?.n||0;
|
||||
const entityCount=extractions.rows.find(r=>r.type==='entity')?.n||0;
|
||||
|
||||
// Stats
|
||||
const sb=document.createElement('div');sb.className='stats-bar';
|
||||
[{v:topics.row_count,l:'Categories'},{v:factCount,l:'Verified Facts'},{v:entityCount,l:'Entities'},{v:decisions.row_count,l:'Decisions'},{v:insights.row_count,l:'Insights'},{v:questions.rows[0]?.n||0,l:'Open Questions'}].forEach(s=>{
|
||||
const d=document.createElement('div');d.className='stat';
|
||||
const v=document.createElement('div');v.className='stat-val';v.textContent=s.v;
|
||||
const l=document.createElement('div');l.className='stat-label';l.textContent=s.l;
|
||||
d.append(v,l);sb.appendChild(d);
|
||||
});
|
||||
el.appendChild(sb);
|
||||
$('hdr-stats').textContent=factCount+' facts · '+entityCount+' entities · '+decisions.row_count+' decisions';
|
||||
|
||||
// Categories
|
||||
const cs=document.createElement('div');cs.className='section';
|
||||
cs.insertAdjacentHTML('beforeend','<h2>Auto-Generated Categories</h2>');
|
||||
const cg=document.createElement('div');cg.className='cat-grid';
|
||||
topics.rows.forEach(t=>{
|
||||
const c=document.createElement('div');c.className='cat-card';
|
||||
const n=document.createElement('div');n.className='cat-title';n.textContent=t.topic;
|
||||
const m=document.createElement('div');m.className='cat-count';m.textContent=t.n+' mentions · '+t.c+' conversations';
|
||||
c.append(n,m);c.onclick=()=>drillTopic(t.topic);cg.appendChild(c);
|
||||
});
|
||||
cs.appendChild(cg);el.appendChild(cs);
|
||||
|
||||
// Decisions
|
||||
const ds=document.createElement('div');ds.className='section';
|
||||
ds.insertAdjacentHTML('beforeend','<h2>Key Decisions</h2>');
|
||||
decisions.rows.forEach(d=>ds.appendChild(makeEntry('decision',d.decision,d.conversation+' p'+d.page)));
|
||||
el.appendChild(ds);
|
||||
|
||||
// Insights
|
||||
const is=document.createElement('div');is.className='section';
|
||||
is.insertAdjacentHTML('beforeend','<h2>Key Insights</h2>');
|
||||
insights.rows.forEach(d=>is.appendChild(makeEntry('insight',d.insight,d.conversation+' p'+d.page)));
|
||||
el.appendChild(is);
|
||||
}catch(e){el.textContent='Error: '+e}
|
||||
}
|
||||
|
||||
async function drillTopic(topic){
|
||||
const el=$('tab-knowledge');el.textContent='';
|
||||
const back=document.createElement('button');back.className='btn btn-sm';back.textContent='← Back';back.onclick=loadKnowledge;
|
||||
const h=document.createElement('h2');h.style.cssText='font-size:18px;margin:12px 0';h.textContent=topic;
|
||||
el.append(back,h);
|
||||
const results=await searchHNSW(topic,10);
|
||||
const s=document.createElement('div');s.className='section';
|
||||
s.insertAdjacentHTML('beforeend','<h2>Related Content</h2>');
|
||||
results.forEach(r=>{
|
||||
const e=document.createElement('div');e.className='result';
|
||||
const sc=document.createElement('div');sc.className='result-score';sc.textContent='['+r.score.toFixed(3)+'] '+r.doc_id;
|
||||
const t=document.createElement('div');t.className='result-text';t.textContent=r.chunk_text.substring(0,300);
|
||||
e.append(sc,t);s.appendChild(e);
|
||||
});
|
||||
el.appendChild(s);
|
||||
}
|
||||
|
||||
// === FACTS & ENTITIES (from LLM Team extract mode) ===
|
||||
async function loadFacts(){
|
||||
const el=$('tab-facts');el.textContent='';
|
||||
try{
|
||||
const[facts,entities,verified]=await Promise.all([
|
||||
q("SELECT conversation,page,content FROM processed_extractions WHERE type='fact' ORDER BY conversation,page LIMIT 30"),
|
||||
q("SELECT conversation,content,COUNT(*) as n FROM processed_extractions WHERE type='entity' GROUP BY conversation,content ORDER BY n DESC LIMIT 30"),
|
||||
q("SELECT conversation,page,content FROM processed_validated LIMIT 10"),
|
||||
]);
|
||||
|
||||
const fs=document.createElement('div');fs.className='section';
|
||||
fs.insertAdjacentHTML('beforeend','<h2>Extracted Facts ('+facts.row_count+' total — via LLM Team Extract Mode)</h2>');
|
||||
facts.rows.forEach(r=>fs.appendChild(makeEntry('fact',r.content,r.conversation+' p'+r.page)));
|
||||
el.appendChild(fs);
|
||||
|
||||
const es=document.createElement('div');es.className='section';
|
||||
es.insertAdjacentHTML('beforeend','<h2>Entities ('+entities.row_count+' unique)</h2>');
|
||||
const eg=document.createElement('div');eg.style.cssText='display:flex;flex-wrap:wrap;gap:6px';
|
||||
entities.rows.forEach(r=>{
|
||||
const chip=document.createElement('span');
|
||||
chip.style.cssText='background:var(--s2);border:1px solid var(--b);padding:4px 10px;border-radius:4px;font-size:11px;cursor:pointer';
|
||||
chip.textContent=r.content+(r.n>1?' ('+r.n+')':'');
|
||||
chip.onclick=()=>{showTab('explore');setTimeout(()=>{document.getElementById('searchInput').value=r.content;doSearch()},100)};
|
||||
eg.appendChild(chip);
|
||||
});
|
||||
es.appendChild(eg);el.appendChild(es);
|
||||
|
||||
if(verified.rows.length){
|
||||
const vs=document.createElement('div');vs.className='section';
|
||||
vs.insertAdjacentHTML('beforeend','<h2>Verified by Secondary Model</h2>');
|
||||
verified.rows.forEach(r=>vs.appendChild(makeEntry('verified',r.content,r.conversation+' p'+r.page)));
|
||||
el.appendChild(vs);
|
||||
}
|
||||
}catch(e){el.textContent='Error: '+e}
|
||||
}
|
||||
|
||||
// === SEARCH ===
|
||||
function loadExplore(){
|
||||
const el=$('tab-explore');el.textContent='';
|
||||
const row=document.createElement('div');row.className='search-row';
|
||||
const input=document.createElement('input');input.id='searchInput';input.placeholder='Search by meaning across all knowledge...';
|
||||
const btn=document.createElement('button');btn.className='btn';btn.textContent='Search';
|
||||
btn.onclick=doSearch;input.onkeydown=e=>{if(e.key==='Enter')doSearch()};
|
||||
row.append(input,btn);
|
||||
const res=document.createElement('div');res.id='searchResults';
|
||||
const hint=document.createElement('div');hint.className='empty';hint.textContent='Try: "object storage architecture", "data ingestion trade-offs", "CRM options"';
|
||||
el.append(row,res,hint);
|
||||
}
|
||||
async function doSearch(){
|
||||
const query=document.getElementById('searchInput').value;if(!query)return;
|
||||
const el=$('searchResults');el.textContent='searching...';
|
||||
const results=await searchHNSW(query,15);
|
||||
el.textContent='';
|
||||
if(!results.length){el.textContent='No results';return}
|
||||
results.forEach(r=>{
|
||||
const e=document.createElement('div');e.className='result';
|
||||
const s=document.createElement('div');s.className='result-score';s.textContent='['+r.score.toFixed(3)+'] '+r.doc_id.replace(/_/g,' ');
|
||||
const t=document.createElement('div');t.className='result-text';t.textContent=r.chunk_text.substring(0,400);
|
||||
e.append(s,t);el.appendChild(e);
|
||||
});
|
||||
}
|
||||
|
||||
// === ASK AI ===
|
||||
function loadAsk(){
|
||||
const el=$('tab-ask');el.textContent='';
|
||||
const ta=document.createElement('textarea');ta.className='ask-input';ta.id='askInput';
|
||||
ta.placeholder='Ask a question about your knowledge base...\n\nExamples:\n- What architecture decisions were made about Iceberg?\n- Compare Spark vs Flink approaches\n- What approaches were abandoned and why?';
|
||||
const btn=document.createElement('button');btn.className='btn';btn.id='askBtn';btn.textContent='Ask AI';
|
||||
btn.onclick=doAsk;
|
||||
const resp=document.createElement('div');resp.id='askResp';
|
||||
el.append(ta,btn,resp);
|
||||
}
|
||||
async function doAsk(){
|
||||
const question=$('askInput').value;if(!question)return;
|
||||
$('askBtn').disabled=true;$('askBtn').textContent='thinking...';
|
||||
const el=$('askResp');el.textContent='searching knowledge...';
|
||||
try{
|
||||
const r=await(await fetch(API+'/vectors/rag',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({index_name:'observer_kb',question,top_k:5})})).json();
|
||||
el.textContent='';
|
||||
const ans=document.createElement('div');ans.className='ai-answer';ans.textContent=r.answer;
|
||||
const meta=document.createElement('div');meta.className='ai-meta';meta.textContent='Model: '+r.model+(r.tokens_generated?' · '+r.tokens_generated+' tokens':'');
|
||||
el.append(ans,meta);
|
||||
if(r.sources?.length){
|
||||
const sh=document.createElement('h2');sh.style.cssText='font-size:11px;color:var(--d);margin-top:14px';sh.textContent='SOURCES';el.appendChild(sh);
|
||||
r.sources.forEach(s=>{
|
||||
const e=document.createElement('div');e.className='result';e.style.marginTop='4px';
|
||||
const sc=document.createElement('div');sc.className='result-score';sc.textContent='['+s.score.toFixed(3)+'] '+s.doc_id;
|
||||
const t=document.createElement('div');t.className='result-text';t.style.fontSize='11px';t.textContent=s.chunk_text.substring(0,200);
|
||||
e.append(sc,t);el.appendChild(e);
|
||||
});
|
||||
}
|
||||
}catch(e){el.textContent='Error: '+e}
|
||||
$('askBtn').disabled=false;$('askBtn').textContent='Ask AI';
|
||||
}
|
||||
|
||||
// === UPLOAD ===
|
||||
function loadUpload(){
|
||||
const el=$('tab-upload');el.textContent='';
|
||||
const zone=document.createElement('div');zone.className='upload-zone';
|
||||
zone.onclick=()=>{const i=document.createElement('input');i.type='file';i.multiple=true;i.accept='.pdf,.txt,.csv,.json';i.onchange=e=>[...e.target.files].forEach(uploadFile);i.click()};
|
||||
zone.ondragover=e=>{e.preventDefault();zone.style.borderColor='var(--a)'};
|
||||
zone.ondragleave=()=>zone.style.borderColor='';
|
||||
zone.ondrop=e=>{e.preventDefault();zone.style.borderColor='';[...e.dataTransfer.files].forEach(uploadFile)};
|
||||
const h=document.createElement('h3');h.textContent='Drop files to ingest';
|
||||
const p=document.createElement('p');p.textContent='PDF, CSV, JSON, Text — auto-extracted and indexed';
|
||||
zone.append(h,p);
|
||||
const list=document.createElement('div');list.id='upList';
|
||||
el.append(zone,list);
|
||||
}
|
||||
function uploadFile(file){
|
||||
const item=document.createElement('div');item.className='file-item';
|
||||
const name=document.createElement('span');name.textContent=file.name;
|
||||
const status=document.createElement('span');status.className='uploading';status.textContent='uploading...';
|
||||
item.append(name,status);$('upList').prepend(item);
|
||||
const fd=new FormData();fd.append('file',file);
|
||||
const xhr=new XMLHttpRequest();xhr.open('POST',API+'/ingest/file');
|
||||
xhr.upload.onprogress=e=>{if(e.lengthComputable)status.textContent=Math.round(e.loaded/e.total*100)+'%'};
|
||||
xhr.onload=()=>{status.className=xhr.status<300?'done':'failed';status.textContent=xhr.status<300?'ingested':'failed'};
|
||||
xhr.onerror=()=>{status.className='failed';status.textContent='error'};
|
||||
xhr.send(fd);
|
||||
}
|
||||
|
||||
loadKnowledge();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,193 +0,0 @@
|
||||
{
|
||||
"_description": "Lakehouse model matrix — authoritative routing for all agent tiers. Local models do the heavy lifting; cloud models are consulted sparingly for overview, strategic, and gatekeeper decisions. Read by tests/multi-agent/scenario.ts and mcp-server/index.ts.",
|
||||
"version": 1,
|
||||
"updated": "2026-04-21",
|
||||
"providers": {
|
||||
"ollama_local": {
|
||||
"base_url": "http://localhost:11434",
|
||||
"key_env": null
|
||||
},
|
||||
"ollama_cloud": {
|
||||
"base_url": "https://ollama.com",
|
||||
"key_env": "OLLAMA_CLOUD_KEY",
|
||||
"key_source": "/root/llm_team_config.json → providers.ollama_cloud.api_key",
|
||||
"rate_budget": {
|
||||
"calls_per_hour": 200,
|
||||
"calls_per_day": 2000,
|
||||
"notes": "Paid tier — generous. Policy: keep overview calls ≤ 3/scenario, strategic ≤ 10/day, gatekeeper ≤ 5/day."
|
||||
}
|
||||
}
|
||||
},
|
||||
"tiers": {
|
||||
"t1_hot": {
|
||||
"purpose": "Per tool call — SQL generation, hybrid_search, sql(). Runs 50-200 times per scenario. Latency-sensitive.",
|
||||
"kind": "local_fast",
|
||||
"primary": { "model": "qwen3.5:latest", "provider": "ollama_local", "context_window": 262144 },
|
||||
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
|
||||
"max_tokens": 1000,
|
||||
"temperature": 0.3,
|
||||
"never_route_cloud": true,
|
||||
"context_budget": {
|
||||
"system_prompt_cap": 4000,
|
||||
"prior_context_cap": 6000,
|
||||
"tool_results_cap": 8000,
|
||||
"safety_margin": 2000,
|
||||
"overflow_policy": "summarize_oldest_tool_results_via_t3"
|
||||
},
|
||||
"rationale": "qwen3.5:latest is a 9.7B thinking model with 262K context that emits clean JSON (verified 2026-04-21 after mistral A/B wipeout). Thinking budget requires max_tokens ≥ 400; we hold 1000 to keep propose_done payloads intact. qwen2.5 stays as fallback."
|
||||
},
|
||||
"t2_review": {
|
||||
"purpose": "Per step consensus — executor ↔ reviewer loop critique. 5-14 calls per event.",
|
||||
"kind": "local_balanced",
|
||||
"primary": { "model": "qwen3:latest", "provider": "ollama_local", "context_window": 40960 },
|
||||
"fallback": { "model": "qwen2.5:latest", "provider": "ollama_local", "context_window": 32768 },
|
||||
"max_tokens": 800,
|
||||
"temperature": 0.3,
|
||||
"never_route_cloud": true,
|
||||
"context_budget": {
|
||||
"system_prompt_cap": 2000,
|
||||
"recent_turns_cap": 4000,
|
||||
"safety_margin": 1000
|
||||
},
|
||||
"rationale": "Reviewer only needs to detect schema violations and drift — a 7B model is sufficient."
|
||||
},
|
||||
"t3_overview": {
|
||||
"purpose": "Mid-day checkpoint after every misplacement + every Nth event, and cross-day lesson. 1-3 calls per scenario.",
|
||||
"kind": "thinking_cloud",
|
||||
"primary": { "model": "gpt-oss:120b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 900,
|
||||
"temperature": 0.2,
|
||||
"cloud_budget_per_scenario": 5,
|
||||
"env_flag": "LH_OVERVIEW_CLOUD=1",
|
||||
"context_budget": {
|
||||
"event_digest_cap": 30000,
|
||||
"checkpoint_cap": 8000,
|
||||
"lesson_corpus_cap": 40000,
|
||||
"safety_margin": 8000,
|
||||
"overflow_policy": "chunk_lessons_via_cosine_topk"
|
||||
},
|
||||
"rationale": "Same prompt family as local 20b (gpt-oss series) — prompts port directly. 120b is faster via cloud than 20b local in practice, and lessons are noticeably more specific."
|
||||
},
|
||||
"t4_strategic": {
|
||||
"purpose": "Daily playbook board re-ranking, weekly gap audit, pattern discovery across accumulated playbooks. 1-10 calls per day.",
|
||||
"kind": "thinking_cloud_large",
|
||||
"primary": { "model": "kimi-k2.6", "provider": "ollama_cloud", "context_window": 200000 },
|
||||
"secondary": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"fallback": { "model": "glm-4.7", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 2000,
|
||||
"temperature": 0.2,
|
||||
"cloud_budget_per_day": 10,
|
||||
"context_budget": {
|
||||
"playbook_corpus_cap": 80000,
|
||||
"pattern_history_cap": 20000,
|
||||
"safety_margin": 16000,
|
||||
"overflow_policy": "two_pass_map_reduce"
|
||||
},
|
||||
"rationale": "J named qwen3.5 specifically. GLM-4.7 is a promising alternate for debate phase. Runs after all scenarios complete for the day."
|
||||
},
|
||||
"t5_gatekeeper": {
|
||||
"purpose": "MUST route here: architecture changes, new client onboarding, schema migrations, playbook retirements, index rebuilds, autotune config changes.",
|
||||
"kind": "thinking_cloud_deepest",
|
||||
"primary": { "model": "kimi-k2-thinking", "provider": "ollama_cloud", "context_window": 200000 },
|
||||
"fallback": { "model": "deepseek-v3.1:671b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"secondary_fallback": { "model": "qwen3.5:397b", "provider": "ollama_cloud", "context_window": 131072 },
|
||||
"local_fallback": { "model": "gpt-oss:20b", "provider": "ollama_local", "context_window": 131072 },
|
||||
"max_tokens": 4000,
|
||||
"temperature": 0.1,
|
||||
"cloud_budget_per_day": 5,
|
||||
"audit_log": true,
|
||||
"context_budget": {
|
||||
"decision_doc_cap": 50000,
|
||||
"evidence_bundle_cap": 100000,
|
||||
"prior_gatekeeper_decisions_cap": 20000,
|
||||
"safety_margin": 20000,
|
||||
"overflow_policy": "escalate_to_kimi_k2_1t_or_split_decision"
|
||||
},
|
||||
"rationale": "Highest-stakes decisions — reasoning depth matters more than latency. Audit log so J can always see what the gatekeeper was asked and what it answered. No human approval required today; escalate later if mis-decisions show up."
|
||||
}
|
||||
},
|
||||
|
||||
"embeddings": {
|
||||
"_description": "Vector embedding models for hybrid search + playbook_memory similarity. Always local — embedding calls are too high-volume for cloud. qwen3-embedding is the new primary (2026-04-21); nomic-embed-text-v2-moe is the MoE alternative for corpus-scale embedding; original nomic-embed-text stays as legacy fallback while vectord indexes are being re-embedded.",
|
||||
"primary": { "model": "qwen3-embedding", "provider": "ollama_local", "dim": 1024, "context_window": 32768 },
|
||||
"alternate": { "model": "nomic-embed-text-v2-moe", "provider": "ollama_local", "dim": 768, "context_window": 2048, "notes": "MoE variant — better for long documents, slower per-call, roughly matches nomic-embed-text on short text." },
|
||||
"legacy": { "model": "nomic-embed-text:latest", "provider": "ollama_local", "dim": 768, "context_window": 8192, "notes": "Used by existing vectord indexes. Do NOT switch live until indexes are rebuilt, or hybrid search recall will crater." },
|
||||
"migration_plan": "Re-embed workers_500k (and all production indexes) with qwen3-embedding, keep the old index under a _v1 suffix for rollback, flip the profile pointer only after recall benchmarks match or exceed legacy. Touches vectord::agent + autotune."
|
||||
},
|
||||
|
||||
"context_management": {
|
||||
"_description": "Rule zero: NEVER call a model with more tokens than its context_window minus safety_margin. Every call goes through the budget checker first. If over budget → chunk, summarize, or escalate. This is the stability floor.",
|
||||
"token_estimator": {
|
||||
"method": "chars_div_4",
|
||||
"note": "Rough, biased safe by ~15%. For production, swap to tiktoken or the provider's tokenizer endpoint."
|
||||
},
|
||||
"overflow_policies": {
|
||||
"summarize_oldest_tool_results_via_t3": {
|
||||
"when": "T1 conversation history + tool results exceed context_budget.tool_results_cap",
|
||||
"how": "Send oldest N tool results to T3 with prompt 'summarize these into 500 tokens that preserve what the executor needs to know'; replace them with the summary in the running conversation."
|
||||
},
|
||||
"chunk_lessons_via_cosine_topk": {
|
||||
"when": "lesson corpus in data/_playbook_lessons/*.json exceeds lesson_corpus_cap",
|
||||
"how": "Embed the current scenario spec, cosine-rank all lessons, take top-K until budget exhausted. Fall back to date-sorted if embeddings unavailable."
|
||||
},
|
||||
"two_pass_map_reduce": {
|
||||
"when": "T4 playbook corpus exceeds playbook_corpus_cap",
|
||||
"how": "Pass 1: chunk playbooks into ≤30K token shards, run primary model on each shard to emit 'shard summary'. Pass 2: feed all summaries to primary model for global synthesis. Logged as two audit entries."
|
||||
},
|
||||
"escalate_to_kimi_k2_1t_or_split_decision": {
|
||||
"when": "T5 decision evidence exceeds decision_doc_cap + evidence_bundle_cap",
|
||||
"how": "Prefer kimi-k2:1t which has 1M context. If still over, split decision into sub-decisions (e.g. 'retire playbooks by city' instead of 'retire playbooks globally') and loop."
|
||||
}
|
||||
},
|
||||
"chunking_cache": {
|
||||
"_description": "Precomputed shards of the playbook corpus, indexed by (corpus_version, shard_id). Avoids re-chunking on every T4 run.",
|
||||
"location": "data/_chunk_cache/",
|
||||
"invalidation": "Key includes corpus hash. When playbook_memory changes, the hash changes, the cache misses, and chunks regenerate.",
|
||||
"implementation_status": "SPEC — next sprint."
|
||||
},
|
||||
"implementation_status": "context_window + context_budget fields WIRED in config. Enforcement helper NOT yet wired in agent.ts. Next: add estimateTokens() + budgetCheck() helpers, route all generate() calls through them, emit a metric when overflow policy fires."
|
||||
},
|
||||
"experimental_rotation": {
|
||||
"enabled": false,
|
||||
"purpose": "Sample newer models on a schedule to collect comparison data without rate-limit risk.",
|
||||
"candidates": [
|
||||
{ "model": "minimax-m2.7", "notes": "Newer minimax; unknown output stability" },
|
||||
{ "model": "glm-5", "notes": "GLM next-gen; larger context" },
|
||||
{ "model": "glm-5.1", "notes": "Incremental on GLM-5" },
|
||||
{ "model": "qwen3-next:80b", "notes": "Qwen's experimental successor; smaller than 3.5" },
|
||||
{ "model": "qwen3-coder-next", "notes": "Coder-optimized — good for SQL gen T1 experiments" },
|
||||
{ "model": "deepseek-v3.2", "notes": "Smaller deepseek; reasoning/coding" },
|
||||
{ "model": "nemotron-3-super", "notes": "NVIDIA 230B; general" },
|
||||
{ "model": "cogito-2.1:671b", "notes": "671B general" },
|
||||
{ "model": "mistral-large-3:675b", "notes": "Mistral's flagship; good T3 candidate" }
|
||||
],
|
||||
"rotation": "weekly",
|
||||
"sample_rate": 0.1,
|
||||
"apply_to_tier": "t4_strategic",
|
||||
"notes": "When enabled, T4 routes 10% of calls to a rotating experimental model. Log comparison in /data/_model_eval/ — if the experimental consistently beats primary across 3 rotations, promote it to primary."
|
||||
},
|
||||
"playbook_versioning": {
|
||||
"enabled": true,
|
||||
"purpose": "A playbook can work, then break when architecture changes. Versioning lets us pin which change retired which playbook.",
|
||||
"dataset": "playbook_memory",
|
||||
"schema_additions": {
|
||||
"version": "integer — auto-increment per operation",
|
||||
"parent_id": "string — previous version entry_id for same operation (null for v1)",
|
||||
"retired_at": "timestamp — set when success_rate drops or architecture changes",
|
||||
"retirement_reason": "string — e.g. 'schema_migration:workers_500k 2026-05-03'",
|
||||
"architecture_snapshot": "object — crate versions, index name, schema fingerprint at seed time"
|
||||
},
|
||||
"retire_triggers": [
|
||||
"success_rate < 0.3 over last 20 citations",
|
||||
"schema_fingerprint mismatch detected at retrieval time",
|
||||
"architecture change event emitted by ingestd/vectord",
|
||||
"T5 gatekeeper explicitly retires via /vectors/playbook_memory/retire"
|
||||
],
|
||||
"read_back_policy": "retrieval returns only non-retired versions. History endpoint /vectors/playbook_memory/history/{operation} returns the full chain.",
|
||||
"ui_surface": "mcp-server to render a diff view: side-by-side of playbook versions with a timeline of what changed and when.",
|
||||
"implementation_status": "SPEC — not yet wired. Target: next sprint. Touches gateway + catalogd + mcp-server."
|
||||
},
|
||||
"matrix_index_hybrid_search_note": "Phase 22 candidate: elevate the hybrid_search T1 tool to consult T3 when a pool returns <3 matches OR when the same (role, city) combo has failed N times in 24h. Consult result is a reformulated sql_filter the executor retries with. Keeps T1 fast on the happy path, escalates to T3 only on low-recall signals."
|
||||
}
|
||||
@@ -134,47 +134,4 @@ impl AiClient {
|
||||
}
|
||||
resp.json().await.map_err(|e| format!("rerank parse error: {e}"))
|
||||
}
|
||||
|
||||
/// Force Ollama to unload the named model from VRAM (keep_alive=0).
|
||||
/// Used for predictable profile swaps — without this, Ollama holds a
|
||||
/// model for its configured TTL (default 5min) and the previous
|
||||
/// profile's model can linger in VRAM next to the new one.
|
||||
pub async fn unload_model(&self, model: &str) -> Result<serde_json::Value, String> {
|
||||
let resp = self.client
|
||||
.post(format!("{}/admin/unload", self.base_url))
|
||||
.json(&serde_json::json!({ "model": model }))
|
||||
.send().await
|
||||
.map_err(|e| format!("unload request failed: {e}"))?;
|
||||
if !resp.status().is_success() {
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
return Err(format!("unload error: {text}"));
|
||||
}
|
||||
resp.json().await.map_err(|e| format!("unload parse error: {e}"))
|
||||
}
|
||||
|
||||
/// Ask Ollama to load the named model into VRAM proactively. Makes
|
||||
/// the first real request after profile activation fast (no cold-load
|
||||
/// latency).
|
||||
pub async fn preload_model(&self, model: &str) -> Result<serde_json::Value, String> {
|
||||
let resp = self.client
|
||||
.post(format!("{}/admin/preload", self.base_url))
|
||||
.json(&serde_json::json!({ "model": model }))
|
||||
.send().await
|
||||
.map_err(|e| format!("preload request failed: {e}"))?;
|
||||
if !resp.status().is_success() {
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
return Err(format!("preload error: {text}"));
|
||||
}
|
||||
resp.json().await.map_err(|e| format!("preload parse error: {e}"))
|
||||
}
|
||||
|
||||
/// GPU + loaded-model snapshot from the sidecar. Combines nvidia-smi
|
||||
/// output (if available) with Ollama's /api/ps.
|
||||
pub async fn vram_snapshot(&self) -> Result<serde_json::Value, String> {
|
||||
let resp = self.client
|
||||
.get(format!("{}/admin/vram", self.base_url))
|
||||
.send().await
|
||||
.map_err(|e| format!("vram request failed: {e}"))?;
|
||||
resp.json().await.map_err(|e| format!("vram parse error: {e}"))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,17 +16,9 @@ pub fn router(client: AiClient) -> Router {
|
||||
.route("/embed", post(embed))
|
||||
.route("/generate", post(generate))
|
||||
.route("/rerank", post(rerank))
|
||||
.route("/vram", get(vram))
|
||||
.with_state(client)
|
||||
}
|
||||
|
||||
async fn vram(State(client): State<AiClient>) -> impl IntoResponse {
|
||||
match client.vram_snapshot().await {
|
||||
Ok(snap) => Ok(Json(snap)),
|
||||
Err(e) => Err((StatusCode::BAD_GATEWAY, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn health(State(client): State<AiClient>) -> impl IntoResponse {
|
||||
match client.health().await {
|
||||
Ok(info) => Ok(Json(info)),
|
||||
|
||||
@@ -15,6 +15,5 @@ bytes = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
uuid = { workspace = true }
|
||||
object_store = { workspace = true }
|
||||
arrow = { workspace = true }
|
||||
proto = { path = "../proto" }
|
||||
tonic = { workspace = true }
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
pub mod registry;
|
||||
pub mod service;
|
||||
pub mod grpc;
|
||||
pub mod tombstones;
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
use shared::types::{
|
||||
AiView, ColumnMeta, DatasetId, DatasetManifest, FreshnessContract, Lineage, ModelProfile,
|
||||
ObjectRef, RefreshPolicy, SchemaFingerprint, Sensitivity, Tombstone,
|
||||
DatasetId, DatasetManifest, ObjectRef, SchemaFingerprint,
|
||||
ColumnMeta, Lineage, FreshnessContract, Sensitivity,
|
||||
};
|
||||
|
||||
use crate::tombstones::TombstoneStore;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
@@ -11,23 +9,6 @@ use tokio::sync::RwLock;
|
||||
use storaged::ops;
|
||||
use object_store::ObjectStore;
|
||||
|
||||
/// Make a view name safe for use as an object storage key.
|
||||
/// Allows letters, digits, `_`, `-`, `.`. Anything else becomes `_`.
|
||||
fn sanitize_view_name(name: &str) -> String {
|
||||
name.chars()
|
||||
.map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.' { c } else { '_' })
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default, serde::Serialize)]
|
||||
pub struct MigrateBucketsReport {
|
||||
pub refs_examined: usize,
|
||||
pub refs_renamed: usize, // legacy "data"/"local" → "primary"
|
||||
pub refs_stamped: usize, // empty → "primary"
|
||||
pub refs_unchanged: usize, // already canonical
|
||||
pub manifests_persisted: usize,
|
||||
}
|
||||
|
||||
/// Partial metadata update — only set fields are applied.
|
||||
#[derive(Debug, Clone, Default, serde::Deserialize)]
|
||||
pub struct MetadataUpdate {
|
||||
@@ -39,23 +20,14 @@ pub struct MetadataUpdate {
|
||||
pub lineage: Option<Lineage>,
|
||||
pub freshness: Option<FreshnessContract>,
|
||||
pub row_count: Option<u64>,
|
||||
// Phase C embedding freshness
|
||||
pub embedding_refresh_policy: Option<RefreshPolicy>,
|
||||
}
|
||||
|
||||
const MANIFEST_PREFIX: &str = "_catalog/manifests";
|
||||
const VIEW_PREFIX: &str = "_catalog/views";
|
||||
const PROFILE_PREFIX: &str = "_catalog/profiles";
|
||||
|
||||
/// In-memory dataset registry backed by manifest persistence in object storage.
|
||||
/// Also tracks AiViews (Phase D) — safe projections over base datasets.
|
||||
/// And tombstones (Phase E) — soft-delete markers applied at query time.
|
||||
#[derive(Clone)]
|
||||
pub struct Registry {
|
||||
datasets: Arc<RwLock<HashMap<DatasetId, DatasetManifest>>>,
|
||||
views: Arc<RwLock<HashMap<String, AiView>>>,
|
||||
profiles: Arc<RwLock<HashMap<String, ModelProfile>>>,
|
||||
tombstones: TombstoneStore,
|
||||
store: Arc<dyn ObjectStore>,
|
||||
}
|
||||
|
||||
@@ -63,47 +35,11 @@ impl Registry {
|
||||
pub fn new(store: Arc<dyn ObjectStore>) -> Self {
|
||||
Self {
|
||||
datasets: Arc::new(RwLock::new(HashMap::new())),
|
||||
views: Arc::new(RwLock::new(HashMap::new())),
|
||||
profiles: Arc::new(RwLock::new(HashMap::new())),
|
||||
tombstones: TombstoneStore::new(store.clone()),
|
||||
store,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn tombstones(&self) -> &TombstoneStore {
|
||||
&self.tombstones
|
||||
}
|
||||
|
||||
/// Add a tombstone for a dataset row. Validates that the dataset
|
||||
/// exists. Idempotent at the endpoint layer — callers dedup if needed.
|
||||
pub async fn add_tombstone(
|
||||
&self,
|
||||
dataset: &str,
|
||||
row_key_column: &str,
|
||||
row_key_value: &str,
|
||||
actor: &str,
|
||||
reason: &str,
|
||||
) -> Result<Tombstone, String> {
|
||||
if self.get_by_name(dataset).await.is_none() {
|
||||
return Err(format!("dataset not found: {dataset}"));
|
||||
}
|
||||
let ts = Tombstone {
|
||||
dataset: dataset.to_string(),
|
||||
row_key_column: row_key_column.to_string(),
|
||||
row_key_value: row_key_value.to_string(),
|
||||
deleted_at: chrono::Utc::now(),
|
||||
actor: actor.to_string(),
|
||||
reason: reason.to_string(),
|
||||
};
|
||||
self.tombstones.append(&ts).await?;
|
||||
Ok(ts)
|
||||
}
|
||||
|
||||
pub async fn list_tombstones(&self, dataset: &str) -> Result<Vec<Tombstone>, String> {
|
||||
self.tombstones.list(dataset).await
|
||||
}
|
||||
|
||||
/// Rebuild in-memory index from persisted manifests + views on startup.
|
||||
/// Rebuild in-memory index from persisted manifests on startup.
|
||||
pub async fn rebuild(&self) -> Result<usize, String> {
|
||||
let keys = ops::list(&self.store, Some(MANIFEST_PREFIX)).await?;
|
||||
let mut datasets = self.datasets.write().await;
|
||||
@@ -116,102 +52,9 @@ impl Registry {
|
||||
}
|
||||
let count = datasets.len();
|
||||
tracing::info!("catalog rebuilt: {count} datasets loaded");
|
||||
|
||||
// Phase D: load AiView definitions alongside manifests.
|
||||
let view_keys = ops::list(&self.store, Some(VIEW_PREFIX)).await.unwrap_or_default();
|
||||
let mut views = self.views.write().await;
|
||||
views.clear();
|
||||
for key in &view_keys {
|
||||
if !key.ends_with(".json") { continue; }
|
||||
let data = match ops::get(&self.store, key).await {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
tracing::warn!("view '{key}': read failed: {e}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
match serde_json::from_slice::<AiView>(&data) {
|
||||
Ok(view) => { views.insert(view.name.clone(), view); }
|
||||
Err(e) => tracing::warn!("view '{key}': parse failed: {e}"),
|
||||
}
|
||||
}
|
||||
if !views.is_empty() {
|
||||
tracing::info!("catalog: {} views loaded", views.len());
|
||||
}
|
||||
|
||||
// Phase 17: load model profiles.
|
||||
let profile_keys = ops::list(&self.store, Some(PROFILE_PREFIX)).await.unwrap_or_default();
|
||||
let mut profiles = self.profiles.write().await;
|
||||
profiles.clear();
|
||||
for key in &profile_keys {
|
||||
if !key.ends_with(".json") { continue; }
|
||||
let data = match ops::get(&self.store, key).await {
|
||||
Ok(d) => d,
|
||||
Err(e) => { tracing::warn!("profile '{key}': read failed: {e}"); continue; }
|
||||
};
|
||||
match serde_json::from_slice::<ModelProfile>(&data) {
|
||||
Ok(p) => { profiles.insert(p.id.clone(), p); }
|
||||
Err(e) => tracing::warn!("profile '{key}': parse failed: {e}"),
|
||||
}
|
||||
}
|
||||
if !profiles.is_empty() {
|
||||
tracing::info!("catalog: {} model profiles loaded", profiles.len());
|
||||
}
|
||||
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
// --- Phase 17: Model profiles ---
|
||||
|
||||
/// Create or replace a model profile. Validates id slug + ensures
|
||||
/// every bound_dataset exists (as either a raw dataset or an AiView).
|
||||
pub async fn put_profile(&self, profile: ModelProfile) -> Result<ModelProfile, String> {
|
||||
if profile.id.is_empty() {
|
||||
return Err("profile id is empty".into());
|
||||
}
|
||||
if !profile.id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') {
|
||||
return Err(format!(
|
||||
"profile id '{}' must be alphanumeric + '-' or '_' only", profile.id,
|
||||
));
|
||||
}
|
||||
for binding in &profile.bound_datasets {
|
||||
let exists = self.get_by_name(binding).await.is_some()
|
||||
|| self.get_view(binding).await.is_some();
|
||||
if !exists {
|
||||
return Err(format!(
|
||||
"bound dataset '{}' not found as dataset or view", binding,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
let key = format!("{PROFILE_PREFIX}/{}.json", profile.id);
|
||||
let json = serde_json::to_vec_pretty(&profile).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &key, json.into()).await?;
|
||||
|
||||
let mut profiles = self.profiles.write().await;
|
||||
profiles.insert(profile.id.clone(), profile.clone());
|
||||
tracing::info!(
|
||||
"profile registered: {} -> ollama={} bindings={:?}",
|
||||
profile.id, profile.ollama_name, profile.bound_datasets,
|
||||
);
|
||||
Ok(profile)
|
||||
}
|
||||
|
||||
pub async fn get_profile(&self, id: &str) -> Option<ModelProfile> {
|
||||
self.profiles.read().await.get(id).cloned()
|
||||
}
|
||||
|
||||
pub async fn list_profiles(&self) -> Vec<ModelProfile> {
|
||||
self.profiles.read().await.values().cloned().collect()
|
||||
}
|
||||
|
||||
pub async fn delete_profile(&self, id: &str) -> Result<(), String> {
|
||||
let key = format!("{PROFILE_PREFIX}/{id}.json");
|
||||
ops::delete(&self.store, &key).await?;
|
||||
self.profiles.write().await.remove(id);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Register a new dataset. Persists manifest to storage before updating memory.
|
||||
pub async fn register(
|
||||
&self,
|
||||
@@ -235,9 +78,6 @@ impl Registry {
|
||||
freshness: None,
|
||||
tags: vec![],
|
||||
row_count: None,
|
||||
last_embedded_at: None,
|
||||
embedding_stale_since: None,
|
||||
embedding_refresh_policy: None,
|
||||
};
|
||||
|
||||
// Write-ahead: persist before in-memory update
|
||||
@@ -271,7 +111,6 @@ impl Registry {
|
||||
if let Some(lineage) = updates.lineage { manifest.lineage = Some(lineage); }
|
||||
if let Some(freshness) = updates.freshness { manifest.freshness = Some(freshness); }
|
||||
if let Some(count) = updates.row_count { manifest.row_count = Some(count); }
|
||||
if let Some(policy) = updates.embedding_refresh_policy { manifest.embedding_refresh_policy = Some(policy); }
|
||||
manifest.updated_at = chrono::Utc::now();
|
||||
|
||||
// Persist
|
||||
@@ -301,264 +140,6 @@ impl Registry {
|
||||
datasets.values().cloned().collect()
|
||||
}
|
||||
|
||||
/// Re-read the parquet footer(s) for a dataset and repopulate `row_count`
|
||||
/// and `columns` from reality. Use this to repair manifests whose
|
||||
/// metadata was lost (e.g. migrated from a pre-Phase 10 catalog).
|
||||
///
|
||||
/// Does NOT touch owner/description/sensitivity/lineage/tags — only
|
||||
/// the structural facts that parquet can tell us authoritatively.
|
||||
/// The existing `schema_fingerprint` is updated if the recomputed one
|
||||
/// differs; a warning is logged so drift is visible.
|
||||
pub async fn resync_from_parquet(&self, name: &str) -> Result<DatasetManifest, String> {
|
||||
use shared::arrow_helpers::{fingerprint_schema, parquet_to_record_batches};
|
||||
|
||||
// Snapshot the target manifest so we don't hold the write lock during IO.
|
||||
let (id, objects, old_fp) = {
|
||||
let datasets = self.datasets.read().await;
|
||||
let m = datasets
|
||||
.values()
|
||||
.find(|d| d.name == name)
|
||||
.ok_or_else(|| format!("dataset not found: {name}"))?;
|
||||
(m.id.clone(), m.objects.clone(), m.schema_fingerprint.clone())
|
||||
};
|
||||
|
||||
if objects.is_empty() {
|
||||
return Err(format!("dataset '{name}' has no object references to resync from"));
|
||||
}
|
||||
|
||||
let mut total_rows: u64 = 0;
|
||||
let mut first_schema: Option<arrow::datatypes::SchemaRef> = None;
|
||||
for obj in &objects {
|
||||
let data = ops::get(&self.store, &obj.key).await
|
||||
.map_err(|e| format!("read {}: {e}", obj.key))?;
|
||||
let (schema, batches) = parquet_to_record_batches(&data)
|
||||
.map_err(|e| format!("parse {}: {e}", obj.key))?;
|
||||
let rows: u64 = batches.iter().map(|b| b.num_rows() as u64).sum();
|
||||
total_rows += rows;
|
||||
if first_schema.is_none() {
|
||||
first_schema = Some(schema);
|
||||
}
|
||||
}
|
||||
|
||||
let schema = first_schema.ok_or("no schema recovered")?;
|
||||
let new_fp = fingerprint_schema(&schema);
|
||||
if new_fp != old_fp {
|
||||
tracing::warn!(
|
||||
"dataset '{}' schema fingerprint drift: {} -> {} (updating to match parquet reality)",
|
||||
name, old_fp.0, new_fp.0,
|
||||
);
|
||||
}
|
||||
|
||||
let columns: Vec<ColumnMeta> = schema
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|f| ColumnMeta {
|
||||
name: f.name().clone(),
|
||||
data_type: f.data_type().to_string(),
|
||||
sensitivity: None,
|
||||
description: String::new(),
|
||||
is_pii: false,
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Apply updates.
|
||||
let mut datasets = self.datasets.write().await;
|
||||
let manifest = datasets
|
||||
.get_mut(&id)
|
||||
.ok_or_else(|| format!("dataset disappeared during resync: {name}"))?;
|
||||
manifest.row_count = Some(total_rows);
|
||||
manifest.columns = columns;
|
||||
manifest.schema_fingerprint = new_fp;
|
||||
manifest.updated_at = chrono::Utc::now();
|
||||
|
||||
// Persist.
|
||||
let manifest_key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
|
||||
let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &manifest_key, json.into()).await?;
|
||||
|
||||
tracing::info!("resynced '{name}': row_count={total_rows}, {} columns", manifest.columns.len());
|
||||
Ok(manifest.clone())
|
||||
}
|
||||
|
||||
/// Resync every dataset that currently has a null row_count.
|
||||
/// Returns (successes, failures) where each entry is (name, detail).
|
||||
pub async fn resync_missing(&self) -> (Vec<(String, u64)>, Vec<(String, String)>) {
|
||||
let names: Vec<String> = {
|
||||
let datasets = self.datasets.read().await;
|
||||
datasets
|
||||
.values()
|
||||
.filter(|d| d.row_count.is_none() || d.columns.is_empty())
|
||||
.map(|d| d.name.clone())
|
||||
.collect()
|
||||
};
|
||||
|
||||
let mut ok = Vec::new();
|
||||
let mut err = Vec::new();
|
||||
for name in names {
|
||||
match self.resync_from_parquet(&name).await {
|
||||
Ok(m) => ok.push((name, m.row_count.unwrap_or(0))),
|
||||
Err(e) => err.push((name, e)),
|
||||
}
|
||||
}
|
||||
(ok, err)
|
||||
}
|
||||
|
||||
/// Mark a dataset's embeddings as stale (row-level data has been written
|
||||
/// since the last embedding refresh). Idempotent — setting stale when
|
||||
/// already stale is a no-op. Only marks stale if the dataset has been
|
||||
/// embedded before — a never-embedded dataset doesn't need a stale flag
|
||||
/// (it just needs an initial index build). Called from the ingest path.
|
||||
pub async fn mark_embeddings_stale(&self, name: &str) -> Result<(), String> {
|
||||
let mut datasets = self.datasets.write().await;
|
||||
let manifest = datasets
|
||||
.values_mut()
|
||||
.find(|d| d.name == name)
|
||||
.ok_or_else(|| format!("dataset not found: {name}"))?;
|
||||
|
||||
if manifest.last_embedded_at.is_none() {
|
||||
return Ok(()); // never embedded -> no stale semantics yet
|
||||
}
|
||||
if manifest.embedding_stale_since.is_none() {
|
||||
manifest.embedding_stale_since = Some(chrono::Utc::now());
|
||||
manifest.updated_at = chrono::Utc::now();
|
||||
|
||||
let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
|
||||
let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &key, json.into()).await?;
|
||||
tracing::info!("marked embeddings stale for dataset '{name}'");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Clear the stale marker and set `last_embedded_at = now`.
|
||||
/// Called by the embedding refresh pipeline once it finishes.
|
||||
pub async fn clear_embeddings_stale(&self, name: &str) -> Result<(), String> {
|
||||
let mut datasets = self.datasets.write().await;
|
||||
let manifest = datasets
|
||||
.values_mut()
|
||||
.find(|d| d.name == name)
|
||||
.ok_or_else(|| format!("dataset not found: {name}"))?;
|
||||
|
||||
let now = chrono::Utc::now();
|
||||
manifest.embedding_stale_since = None;
|
||||
manifest.last_embedded_at = Some(now);
|
||||
manifest.updated_at = now;
|
||||
|
||||
let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
|
||||
let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &key, json.into()).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Federation layer 2: stamp `bucket = "primary"` on every ObjectRef
|
||||
/// whose `bucket` field is empty or matches a legacy value (`"data"`,
|
||||
/// `"local"`). One-shot migration; re-running is a safe no-op once
|
||||
/// every ref is canonical.
|
||||
pub async fn migrate_buckets_to_primary(&self) -> Result<MigrateBucketsReport, String> {
|
||||
let mut datasets = self.datasets.write().await;
|
||||
let mut report = MigrateBucketsReport::default();
|
||||
let mut to_persist: Vec<(DatasetId, DatasetManifest)> = Vec::new();
|
||||
|
||||
for manifest in datasets.values_mut() {
|
||||
let mut changed = false;
|
||||
for obj in manifest.objects.iter_mut() {
|
||||
report.refs_examined += 1;
|
||||
if obj.bucket.is_empty() {
|
||||
obj.bucket = "primary".to_string();
|
||||
report.refs_stamped += 1;
|
||||
changed = true;
|
||||
} else if obj.bucket == "data" || obj.bucket == "local" {
|
||||
obj.bucket = "primary".to_string();
|
||||
report.refs_renamed += 1;
|
||||
changed = true;
|
||||
} else {
|
||||
report.refs_unchanged += 1;
|
||||
}
|
||||
}
|
||||
if changed {
|
||||
manifest.updated_at = chrono::Utc::now();
|
||||
to_persist.push((manifest.id.clone(), manifest.clone()));
|
||||
}
|
||||
}
|
||||
|
||||
// Persist updated manifests after we've finished mutating the map.
|
||||
for (id, manifest) in to_persist {
|
||||
let key = format!("{MANIFEST_PREFIX}/{}.json", id);
|
||||
let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &key, json.into()).await?;
|
||||
report.manifests_persisted += 1;
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"bucket migration: examined {} refs, renamed {}, stamped {}, unchanged {}, persisted {} manifests",
|
||||
report.refs_examined,
|
||||
report.refs_renamed,
|
||||
report.refs_stamped,
|
||||
report.refs_unchanged,
|
||||
report.manifests_persisted,
|
||||
);
|
||||
Ok(report)
|
||||
}
|
||||
|
||||
// --- Phase D: AI-safe views ---
|
||||
|
||||
/// Create or replace a named view. Validates that the base dataset
|
||||
/// exists and that the column whitelist is non-empty. Persists to
|
||||
/// `_catalog/views/{name}.json`.
|
||||
pub async fn put_view(&self, mut view: AiView) -> Result<AiView, String> {
|
||||
if view.name.is_empty() {
|
||||
return Err("view name is empty".into());
|
||||
}
|
||||
if view.columns.is_empty() {
|
||||
return Err("view must whitelist at least one column".into());
|
||||
}
|
||||
// Base dataset must exist (read-side will fail anyway, fail-fast here
|
||||
// so operators don't end up with dangling views).
|
||||
if self.get_by_name(&view.base_dataset).await.is_none() {
|
||||
return Err(format!(
|
||||
"base dataset '{}' not found in catalog",
|
||||
view.base_dataset
|
||||
));
|
||||
}
|
||||
if view.created_at.timestamp() == 0 {
|
||||
view.created_at = chrono::Utc::now();
|
||||
}
|
||||
|
||||
let key = format!("{VIEW_PREFIX}/{}.json", sanitize_view_name(&view.name));
|
||||
let json = serde_json::to_vec_pretty(&view).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &key, json.into()).await?;
|
||||
|
||||
let mut views = self.views.write().await;
|
||||
views.insert(view.name.clone(), view.clone());
|
||||
tracing::info!("view registered: {} over '{}'", view.name, view.base_dataset);
|
||||
Ok(view)
|
||||
}
|
||||
|
||||
pub async fn get_view(&self, name: &str) -> Option<AiView> {
|
||||
self.views.read().await.get(name).cloned()
|
||||
}
|
||||
|
||||
pub async fn list_views(&self) -> Vec<AiView> {
|
||||
self.views.read().await.values().cloned().collect()
|
||||
}
|
||||
|
||||
pub async fn delete_view(&self, name: &str) -> Result<(), String> {
|
||||
let key = format!("{VIEW_PREFIX}/{}.json", sanitize_view_name(name));
|
||||
ops::delete(&self.store, &key).await?;
|
||||
self.views.write().await.remove(name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// List datasets whose `embedding_stale_since` is set — they need a refresh.
|
||||
pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
|
||||
let datasets = self.datasets.read().await;
|
||||
datasets
|
||||
.values()
|
||||
.filter(|d| d.embedding_stale_since.is_some())
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Add objects to an existing dataset.
|
||||
pub async fn add_objects(
|
||||
&self,
|
||||
|
||||
@@ -19,17 +19,6 @@ pub fn router(registry: Registry) -> Router {
|
||||
.route("/datasets/{id}", get(get_dataset))
|
||||
.route("/datasets/by-name/{name}", get(get_dataset_by_name))
|
||||
.route("/datasets/by-name/{name}/metadata", post(update_metadata))
|
||||
.route("/datasets/by-name/{name}/resync", post(resync_dataset))
|
||||
.route("/resync-missing", post(resync_all_missing))
|
||||
.route("/migrate-buckets", post(migrate_buckets))
|
||||
// Phase D: AI-safe views
|
||||
.route("/views", post(create_view).get(list_views))
|
||||
.route("/views/{name}", get(get_view).delete(delete_view))
|
||||
// Phase E: soft-delete tombstones
|
||||
.route("/datasets/by-name/{name}/tombstone", post(tombstone_rows).get(list_tombstones))
|
||||
// Phase 17: model profiles
|
||||
.route("/profiles", post(create_profile).get(list_profiles))
|
||||
.route("/profiles/{id}", get(get_profile).delete(delete_profile))
|
||||
.with_state(registry)
|
||||
}
|
||||
|
||||
@@ -163,254 +152,3 @@ async fn update_metadata(
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Re-read parquet footers for a single dataset and repopulate row_count
|
||||
/// and columns from reality. Useful for repairing manifests whose metadata
|
||||
/// was lost or never backfilled.
|
||||
async fn resync_dataset(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.resync_from_parquet(&name).await {
|
||||
Ok(manifest) => Ok(Json(DatasetResponse::from(&manifest))),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ResyncAllResponse {
|
||||
succeeded: Vec<ResyncOk>,
|
||||
failed: Vec<ResyncErr>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ResyncOk {
|
||||
name: String,
|
||||
row_count: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ResyncErr {
|
||||
name: String,
|
||||
error: String,
|
||||
}
|
||||
|
||||
/// Resync every dataset that currently has null row_count or empty columns.
|
||||
async fn resync_all_missing(State(registry): State<Registry>) -> impl IntoResponse {
|
||||
let (ok, err) = registry.resync_missing().await;
|
||||
Json(ResyncAllResponse {
|
||||
succeeded: ok.into_iter().map(|(name, row_count)| ResyncOk { name, row_count }).collect(),
|
||||
failed: err.into_iter().map(|(name, error)| ResyncErr { name, error }).collect(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Federation layer 2 one-shot: normalize every ObjectRef.bucket field
|
||||
/// to the canonical "primary" value. Idempotent — re-running once
|
||||
/// everything is canonical is a safe no-op.
|
||||
async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse {
|
||||
match registry.migrate_buckets_to_primary().await {
|
||||
Ok(report) => Ok(Json(report)),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
// --- Phase D: AI-safe views ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct CreateViewRequest {
|
||||
name: String,
|
||||
base_dataset: String,
|
||||
columns: Vec<String>,
|
||||
#[serde(default)]
|
||||
row_filter: Option<String>,
|
||||
#[serde(default)]
|
||||
column_redactions: std::collections::HashMap<String, shared::types::Redaction>,
|
||||
#[serde(default)]
|
||||
description: String,
|
||||
#[serde(default)]
|
||||
created_by: String,
|
||||
}
|
||||
|
||||
async fn create_view(
|
||||
State(registry): State<Registry>,
|
||||
Json(req): Json<CreateViewRequest>,
|
||||
) -> impl IntoResponse {
|
||||
let view = shared::types::AiView {
|
||||
name: req.name,
|
||||
base_dataset: req.base_dataset,
|
||||
columns: req.columns,
|
||||
row_filter: req.row_filter,
|
||||
column_redactions: req.column_redactions,
|
||||
created_at: chrono::Utc::now(),
|
||||
created_by: req.created_by,
|
||||
description: req.description,
|
||||
};
|
||||
match registry.put_view(view).await {
|
||||
Ok(v) => Ok((StatusCode::CREATED, Json(v))),
|
||||
Err(e) => Err((StatusCode::BAD_REQUEST, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_views(State(registry): State<Registry>) -> impl IntoResponse {
|
||||
Json(registry.list_views().await)
|
||||
}
|
||||
|
||||
async fn get_view(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.get_view(&name).await {
|
||||
Some(v) => Ok(Json(v)),
|
||||
None => Err((StatusCode::NOT_FOUND, format!("view not found: {name}"))),
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete_view(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.delete_view(&name).await {
|
||||
Ok(()) => Ok(StatusCode::NO_CONTENT),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
// --- Phase E: soft-delete tombstones ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct TombstoneRequest {
|
||||
row_key_column: String,
|
||||
row_key_values: Vec<String>,
|
||||
#[serde(default)]
|
||||
actor: String,
|
||||
#[serde(default)]
|
||||
reason: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct TombstoneResponse {
|
||||
dataset: String,
|
||||
row_key_column: String,
|
||||
rows_tombstoned: usize,
|
||||
failures: Vec<String>,
|
||||
}
|
||||
|
||||
async fn tombstone_rows(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
Json(req): Json<TombstoneRequest>,
|
||||
) -> impl IntoResponse {
|
||||
if req.row_key_values.is_empty() {
|
||||
return Err((StatusCode::BAD_REQUEST, "row_key_values is empty".to_string()));
|
||||
}
|
||||
|
||||
let mut ok = 0;
|
||||
let mut failures = Vec::new();
|
||||
for value in &req.row_key_values {
|
||||
match registry
|
||||
.add_tombstone(&name, &req.row_key_column, value, &req.actor, &req.reason)
|
||||
.await
|
||||
{
|
||||
Ok(_) => ok += 1,
|
||||
Err(e) => failures.push(format!("{value}: {e}")),
|
||||
}
|
||||
}
|
||||
|
||||
let status = if ok > 0 && failures.is_empty() {
|
||||
StatusCode::CREATED
|
||||
} else if ok > 0 {
|
||||
StatusCode::MULTI_STATUS
|
||||
} else {
|
||||
StatusCode::BAD_REQUEST
|
||||
};
|
||||
|
||||
Ok((status, Json(TombstoneResponse {
|
||||
dataset: name,
|
||||
row_key_column: req.row_key_column,
|
||||
rows_tombstoned: ok,
|
||||
failures,
|
||||
})))
|
||||
}
|
||||
|
||||
async fn list_tombstones(
|
||||
State(registry): State<Registry>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.list_tombstones(&name).await {
|
||||
Ok(ts) => Ok(Json(ts)),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
// --- Phase 17: Model profiles ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct CreateProfileRequest {
|
||||
id: String,
|
||||
ollama_name: String,
|
||||
#[serde(default)]
|
||||
description: String,
|
||||
bound_datasets: Vec<String>,
|
||||
#[serde(default)]
|
||||
hnsw_config: shared::types::ProfileHnswConfig,
|
||||
#[serde(default = "default_embed_model_req")]
|
||||
embed_model: String,
|
||||
#[serde(default)]
|
||||
created_by: String,
|
||||
/// Federation: optional per-profile bucket (`profile:{id}` by convention).
|
||||
/// Omitting keeps artifacts in primary.
|
||||
#[serde(default)]
|
||||
bucket: Option<String>,
|
||||
/// ADR-019 hybrid: which vector backend to route this profile's
|
||||
/// indexes to. Defaults to Parquet+HNSW.
|
||||
#[serde(default)]
|
||||
vector_backend: shared::types::VectorBackend,
|
||||
}
|
||||
|
||||
fn default_embed_model_req() -> String { "nomic-embed-text".to_string() }
|
||||
|
||||
async fn create_profile(
|
||||
State(registry): State<Registry>,
|
||||
Json(req): Json<CreateProfileRequest>,
|
||||
) -> impl IntoResponse {
|
||||
let profile = shared::types::ModelProfile {
|
||||
id: req.id,
|
||||
ollama_name: req.ollama_name,
|
||||
description: req.description,
|
||||
bound_datasets: req.bound_datasets,
|
||||
hnsw_config: req.hnsw_config,
|
||||
embed_model: req.embed_model,
|
||||
created_at: chrono::Utc::now(),
|
||||
created_by: req.created_by,
|
||||
bucket: req.bucket,
|
||||
vector_backend: req.vector_backend,
|
||||
};
|
||||
match registry.put_profile(profile).await {
|
||||
Ok(p) => Ok((StatusCode::CREATED, Json(p))),
|
||||
Err(e) => Err((StatusCode::BAD_REQUEST, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_profiles(State(registry): State<Registry>) -> impl IntoResponse {
|
||||
Json(registry.list_profiles().await)
|
||||
}
|
||||
|
||||
async fn get_profile(
|
||||
State(registry): State<Registry>,
|
||||
Path(id): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.get_profile(&id).await {
|
||||
Some(p) => Ok(Json(p)),
|
||||
None => Err((StatusCode::NOT_FOUND, format!("profile not found: {id}"))),
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete_profile(
|
||||
State(registry): State<Registry>,
|
||||
Path(id): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match registry.delete_profile(&id).await {
|
||||
Ok(()) => Ok(StatusCode::NO_CONTENT),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,151 +0,0 @@
|
||||
//! Soft-delete tombstone storage (Phase E).
|
||||
//!
|
||||
//! One append-log per dataset at `_catalog/tombstones/{dataset}/batch_*.jsonl`.
|
||||
//! Uses the shared `storaged::append_log::AppendLog` pattern so appends
|
||||
//! are write-once (never rewrites existing files) and can be compacted.
|
||||
//!
|
||||
//! The store exposes a per-dataset cache of active tombstones for the
|
||||
//! hot path (queryd filter construction) — that's why events are pulled
|
||||
//! into an in-memory map on every call rather than scanning object
|
||||
//! storage repeatedly.
|
||||
|
||||
use object_store::ObjectStore;
|
||||
use shared::types::Tombstone;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use storaged::append_log::{AppendLog, CompactStats};
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
const TOMBSTONE_PREFIX: &str = "_catalog/tombstones";
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct TombstoneStore {
|
||||
store: Arc<dyn ObjectStore>,
|
||||
logs: Arc<RwLock<HashMap<String, Arc<AppendLog>>>>,
|
||||
}
|
||||
|
||||
impl TombstoneStore {
|
||||
pub fn new(store: Arc<dyn ObjectStore>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
logs: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
fn prefix_for(dataset: &str) -> String {
|
||||
// Sanitize dataset name for filesystem safety.
|
||||
let safe: String = dataset
|
||||
.chars()
|
||||
.map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
|
||||
.collect();
|
||||
format!("{TOMBSTONE_PREFIX}/{}", safe)
|
||||
}
|
||||
|
||||
async fn log_for(&self, dataset: &str) -> Arc<AppendLog> {
|
||||
if let Some(log) = self.logs.read().await.get(dataset) {
|
||||
return log.clone();
|
||||
}
|
||||
let mut guard = self.logs.write().await;
|
||||
if let Some(log) = guard.get(dataset) {
|
||||
return log.clone();
|
||||
}
|
||||
// Threshold of 1 — every tombstone is high-value. Compliance/audit
|
||||
// doesn't tolerate "lost on restart"; we trade a small file count
|
||||
// for guaranteed durability. Compaction merges later if volume grows.
|
||||
let log = Arc::new(
|
||||
AppendLog::new(self.store.clone(), Self::prefix_for(dataset))
|
||||
.with_flush_threshold(1),
|
||||
);
|
||||
guard.insert(dataset.to_string(), log.clone());
|
||||
log
|
||||
}
|
||||
|
||||
/// Append one tombstone. Validates that the `row_key_column` matches
|
||||
/// the column already used for this dataset (all tombstones for a
|
||||
/// dataset share one key column so the query filter is well-defined).
|
||||
/// Forces a flush so the tombstone is durable before this call returns.
|
||||
pub async fn append(&self, ts: &Tombstone) -> Result<(), String> {
|
||||
let existing = self.list(&ts.dataset).await?;
|
||||
if let Some(prior) = existing.first() {
|
||||
if prior.row_key_column != ts.row_key_column {
|
||||
return Err(format!(
|
||||
"dataset '{}' already uses '{}' as tombstone key; cannot mix with '{}'",
|
||||
ts.dataset, prior.row_key_column, ts.row_key_column,
|
||||
));
|
||||
}
|
||||
}
|
||||
let line = serde_json::to_vec(ts).map_err(|e| e.to_string())?;
|
||||
let log = self.log_for(&ts.dataset).await;
|
||||
log.append(line).await?;
|
||||
// Belt-and-suspenders: explicit flush in case the threshold is
|
||||
// ever raised. Tombstones must be durable on return.
|
||||
log.flush().await
|
||||
}
|
||||
|
||||
/// All tombstones for a dataset (chronological).
|
||||
pub async fn list(&self, dataset: &str) -> Result<Vec<Tombstone>, String> {
|
||||
let log = self.log_for(dataset).await;
|
||||
let lines = log.read_all().await?;
|
||||
let mut out = Vec::with_capacity(lines.len());
|
||||
for line in lines {
|
||||
match serde_json::from_slice::<Tombstone>(&line) {
|
||||
Ok(t) => out.push(t),
|
||||
Err(e) => tracing::warn!("tombstones/{}: skip malformed entry: {e}", dataset),
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Per-dataset grouped view used by queryd — returns a map of
|
||||
/// `{dataset -> (row_key_column, set_of_values)}` for every dataset
|
||||
/// that has any tombstones.
|
||||
pub async fn all_grouped(
|
||||
&self,
|
||||
datasets: &[String],
|
||||
) -> Result<HashMap<String, (String, Vec<String>)>, String> {
|
||||
let mut grouped = HashMap::new();
|
||||
for dataset in datasets {
|
||||
let ts = match self.list(dataset).await {
|
||||
Ok(ts) => ts,
|
||||
Err(_) => continue,
|
||||
};
|
||||
if ts.is_empty() { continue; }
|
||||
let col = ts[0].row_key_column.clone();
|
||||
let values: Vec<String> = ts.iter().map(|t| t.row_key_value.clone()).collect();
|
||||
grouped.insert(dataset.clone(), (col, values));
|
||||
}
|
||||
Ok(grouped)
|
||||
}
|
||||
|
||||
pub async fn compact(&self, dataset: &str) -> Result<CompactStats, String> {
|
||||
let log = self.log_for(dataset).await;
|
||||
log.compact().await
|
||||
}
|
||||
|
||||
/// Remove every tombstone for a dataset. Called after a successful
|
||||
/// parquet compaction has physically deleted those rows — the
|
||||
/// tombstones have done their job and the journal can be cleared.
|
||||
///
|
||||
/// Implementation: drop the per-dataset AppendLog from the cache and
|
||||
/// delete all its batch files. Next write starts fresh.
|
||||
pub async fn clear(&self, dataset: &str) -> Result<usize, String> {
|
||||
let prefix = format!("{}/", Self::prefix_for(dataset));
|
||||
let keys = storaged::ops::list(&self.store, Some(&prefix)).await?;
|
||||
let matching: Vec<String> = keys
|
||||
.into_iter()
|
||||
.filter(|k| {
|
||||
let basename = k.rsplit('/').next().unwrap_or(k);
|
||||
basename.starts_with("batch_") && basename.ends_with(".jsonl")
|
||||
})
|
||||
.collect();
|
||||
let count = matching.len();
|
||||
for key in &matching {
|
||||
let _ = storaged::ops::delete(&self.store, key).await;
|
||||
}
|
||||
self.logs.write().await.remove(dataset);
|
||||
if count > 0 {
|
||||
tracing::info!("cleared {count} tombstone batch files for '{}'", dataset);
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
}
|
||||
+15
-104
@@ -24,30 +24,8 @@ async fn main() {
|
||||
tracing::info!("config loaded: gateway={}:{}, storage={}",
|
||||
config.gateway.host, config.gateway.port, config.storage.root);
|
||||
|
||||
// Secrets provider
|
||||
let secrets = std::sync::Arc::new(shared::secrets::FileSecretsProvider::from_env());
|
||||
if let Err(e) = secrets.load().await {
|
||||
tracing::warn!("secrets load: {e} — bucket credentials may be missing");
|
||||
}
|
||||
|
||||
// Federation: bucket registry (primary + any [[storage.buckets]] entries + rescue)
|
||||
let bucket_registry = std::sync::Arc::new(
|
||||
storaged::registry::BucketRegistry::from_config(&config.storage, secrets.clone())
|
||||
.await
|
||||
.expect("bucket registry init failed")
|
||||
);
|
||||
let buckets = bucket_registry.list().await;
|
||||
tracing::info!(
|
||||
"bucket registry: {} buckets [{}], rescue={:?}",
|
||||
buckets.len(),
|
||||
buckets.iter().map(|b| format!("{}:{}", b.name, b.backend)).collect::<Vec<_>>().join(", "),
|
||||
bucket_registry.rescue_name(),
|
||||
);
|
||||
|
||||
// Back-compat: existing components that expect a single Arc<ObjectStore>
|
||||
// get the primary bucket's store. As we migrate call sites to be bucket-
|
||||
// aware, they'll call bucket_registry.get(name) directly.
|
||||
let store = bucket_registry.default_store();
|
||||
// Storage backend
|
||||
let store = storaged::backend::init_local(&config.storage.root);
|
||||
|
||||
// Catalog
|
||||
let registry = catalogd::registry::Registry::new(store.clone());
|
||||
@@ -57,7 +35,7 @@ async fn main() {
|
||||
|
||||
// Query engine with 16GB memory cache
|
||||
let cache = queryd::cache::MemCache::new(16 * 1024 * 1024 * 1024);
|
||||
let engine = queryd::context::QueryEngine::new(registry.clone(), bucket_registry.clone(), cache);
|
||||
let engine = queryd::context::QueryEngine::new(registry.clone(), store.clone(), cache);
|
||||
|
||||
// Event journal — append-only mutation log (flush every 100 events)
|
||||
let journal = journald::journal::Journal::new(store.clone(), 100);
|
||||
@@ -75,94 +53,27 @@ async fn main() {
|
||||
// AI sidecar client
|
||||
let ai_client = aibridge::client::AiClient::new(&config.sidecar.url);
|
||||
|
||||
// Vector service components — built before the router because both the
|
||||
// /vectors service AND ingestd need the agent handle to enqueue triggers.
|
||||
let index_reg = vectord::index_registry::IndexRegistry::new(store.clone());
|
||||
let _ = index_reg.rebuild().await;
|
||||
let hnsw = vectord::hnsw::HnswStore::new();
|
||||
let emb_cache = vectord::embedding_cache::EmbeddingCache::new(store.clone());
|
||||
// Phase B (federation layer 2): trial journals + promotion files now
|
||||
// live in each index's recorded bucket (IndexMeta.bucket), not always
|
||||
// primary. Journal / registry resolve per-call via the bucket registry.
|
||||
let tj = vectord::trial::TrialJournal::new(bucket_registry.clone(), index_reg.clone());
|
||||
let pr = vectord::promotion::PromotionRegistry::new(bucket_registry.clone(), index_reg.clone());
|
||||
|
||||
// Phase 16.2: spawn the autotune agent. When config.agent.enabled=false
|
||||
// this returns a handle that drops triggers silently — no surprise load.
|
||||
let agent_cfg = vectord::agent::AgentConfig {
|
||||
enabled: config.agent.enabled,
|
||||
cycle_interval_secs: config.agent.cycle_interval_secs,
|
||||
cooldown_between_trials_secs: config.agent.cooldown_between_trials_secs,
|
||||
min_recall: config.agent.min_recall,
|
||||
max_trials_per_hour: config.agent.max_trials_per_hour,
|
||||
};
|
||||
let agent_handle = vectord::agent::spawn(
|
||||
agent_cfg,
|
||||
vectord::agent::AgentDeps {
|
||||
store: store.clone(),
|
||||
ai_client: ai_client.clone(),
|
||||
catalog: registry.clone(),
|
||||
index_registry: index_reg.clone(),
|
||||
hnsw_store: hnsw.clone(),
|
||||
embedding_cache: emb_cache.clone(),
|
||||
trial_journal: tj.clone(),
|
||||
promotion_registry: pr.clone(),
|
||||
},
|
||||
);
|
||||
|
||||
// HTTP router
|
||||
let mut app = Router::new()
|
||||
.route("/health", get(health))
|
||||
.nest("/storage", storaged::service::router(store.clone())
|
||||
.merge(storaged::federation_service::router(bucket_registry.clone())))
|
||||
.nest("/storage", storaged::service::router(store.clone()))
|
||||
.nest("/catalog", catalogd::service::router(registry.clone()))
|
||||
.nest("/query", queryd::service::router(engine.clone()))
|
||||
.nest("/ai", aibridge::service::router(ai_client.clone()))
|
||||
;
|
||||
|
||||
// Phase E: scheduled ingest
|
||||
let sched_store = ingestd::schedule::ScheduleStore::new(store.clone());
|
||||
if let Ok(n) = sched_store.rebuild().await {
|
||||
if n > 0 { tracing::info!("rebuilt {n} persisted schedule(s)"); }
|
||||
}
|
||||
let scheduler = ingestd::schedule::Scheduler {
|
||||
store: sched_store.clone(),
|
||||
ingest: ingestd::schedule::SchedulerIngestDeps {
|
||||
store: store.clone(),
|
||||
registry: registry.clone(),
|
||||
buckets: bucket_registry.clone(),
|
||||
agent_handle: Some(agent_handle.clone()),
|
||||
index_registry: index_reg.clone(),
|
||||
},
|
||||
tick_secs: 10,
|
||||
};
|
||||
scheduler.spawn();
|
||||
|
||||
app = app
|
||||
.nest("/ingest", ingestd::service::router(ingestd::service::IngestState {
|
||||
store: store.clone(),
|
||||
registry: registry.clone(),
|
||||
buckets: bucket_registry.clone(),
|
||||
agent_handle: agent_handle.clone(),
|
||||
index_registry: index_reg.clone(),
|
||||
schedules: sched_store,
|
||||
}))
|
||||
.nest("/vectors", vectord::service::router(vectord::service::VectorState {
|
||||
store: store.clone(),
|
||||
ai_client: ai_client.clone(),
|
||||
job_tracker: vectord::jobs::JobTracker::new(),
|
||||
index_registry: index_reg.clone(),
|
||||
hnsw_store: hnsw,
|
||||
embedding_cache: emb_cache,
|
||||
trial_journal: tj,
|
||||
catalog: registry.clone(),
|
||||
promotion_registry: pr,
|
||||
agent_handle,
|
||||
bucket_registry: bucket_registry.clone(),
|
||||
active_profile: std::sync::Arc::new(tokio::sync::RwLock::new(None)),
|
||||
lance: vectord::lance_backend::LanceRegistry::new(
|
||||
bucket_registry.clone(), index_reg.clone(),
|
||||
),
|
||||
.nest("/vectors", vectord::service::router({
|
||||
let index_reg = vectord::index_registry::IndexRegistry::new(store.clone());
|
||||
let _ = index_reg.rebuild().await;
|
||||
vectord::service::VectorState {
|
||||
store: store.clone(),
|
||||
ai_client: ai_client.clone(),
|
||||
job_tracker: vectord::jobs::JobTracker::new(),
|
||||
index_registry: index_reg,
|
||||
hnsw_store: vectord::hnsw::HnswStore::new(),
|
||||
}
|
||||
}))
|
||||
.nest("/workspaces", queryd::workspace_service::router(workspace_mgr))
|
||||
.nest("/journal", journald::service::router(journal))
|
||||
@@ -190,7 +101,7 @@ async fn main() {
|
||||
}
|
||||
|
||||
app = app
|
||||
.layer(DefaultBodyLimit::max(512 * 1024 * 1024)) // 512MB — supports 500K worker CSV
|
||||
.layer(DefaultBodyLimit::max(256 * 1024 * 1024)) // 256MB
|
||||
.layer(CorsLayer::new()
|
||||
.allow_origin(Any)
|
||||
.allow_methods(Any)
|
||||
|
||||
@@ -7,7 +7,6 @@ edition = "2024"
|
||||
shared = { path = "../shared" }
|
||||
storaged = { path = "../storaged" }
|
||||
catalogd = { path = "../catalogd" }
|
||||
vectord = { path = "../vectord" }
|
||||
tokio = { workspace = true }
|
||||
axum = { workspace = true, features = ["multipart"] }
|
||||
lopdf = { workspace = true }
|
||||
@@ -22,5 +21,3 @@ csv = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
object_store = { workspace = true }
|
||||
tokio-postgres = { workspace = true }
|
||||
mysql_async = { workspace = true }
|
||||
uuid = { workspace = true }
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
/// ChatGPT PDF conversation parser.
|
||||
/// Extracts structured conversation turns from ChatGPT export PDFs.
|
||||
/// Handles: user/assistant turns, timestamps, conversation titles.
|
||||
|
||||
use arrow::array::{ArrayRef, Int32Array, RecordBatch, StringArray};
|
||||
use arrow::datatypes::{DataType, Field, Schema};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// A single conversation turn.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ConversationTurn {
|
||||
pub source_file: String,
|
||||
pub conversation_title: String,
|
||||
pub turn_number: i32,
|
||||
pub role: String, // "user" or "assistant"
|
||||
pub content: String,
|
||||
pub page_number: i32,
|
||||
}
|
||||
|
||||
/// Parse a ChatGPT PDF into structured conversation turns.
|
||||
pub fn parse_chatgpt_pdf(content: &[u8], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||
let doc = lopdf::Document::load_mem(content)
|
||||
.map_err(|e| format!("PDF load error: {e}"))?;
|
||||
|
||||
// Extract all text
|
||||
let pages = doc.get_pages();
|
||||
let mut all_text = String::new();
|
||||
let mut page_texts: Vec<(i32, String)> = Vec::new();
|
||||
|
||||
for (&page_num, _) in pages.iter() {
|
||||
let text = doc.extract_text(&[page_num]).unwrap_or_default();
|
||||
let text = text.trim().to_string();
|
||||
if !text.is_empty() {
|
||||
page_texts.push((page_num as i32, text.clone()));
|
||||
all_text.push_str(&text);
|
||||
all_text.push('\n');
|
||||
}
|
||||
}
|
||||
|
||||
if all_text.trim().is_empty() {
|
||||
return Err("PDF contains no extractable text".into());
|
||||
}
|
||||
|
||||
// Try to parse as ChatGPT conversation format
|
||||
let turns = extract_turns(&all_text, filename, &page_texts);
|
||||
|
||||
if turns.is_empty() {
|
||||
// Fall back to page-based extraction if we can't detect conversation structure
|
||||
return parse_as_raw_pages(&page_texts, filename);
|
||||
}
|
||||
|
||||
tracing::info!("parsed ChatGPT PDF '{}': {} turns", filename, turns.len());
|
||||
turns_to_batch(&turns)
|
||||
}
|
||||
|
||||
/// Extract conversation turns from ChatGPT PDF text.
|
||||
/// ChatGPT PDFs typically have patterns like:
|
||||
/// - "You said:" / "ChatGPT said:" (newer format)
|
||||
/// - "User:" / "Assistant:"
|
||||
/// - Or just alternating blocks separated by formatting
|
||||
fn extract_turns(text: &str, filename: &str, page_texts: &[(i32, String)]) -> Vec<ConversationTurn> {
|
||||
let mut turns = Vec::new();
|
||||
let mut turn_num = 0;
|
||||
|
||||
// Try to extract conversation title from first line
|
||||
let title = text.lines().next()
|
||||
.map(|l| l.trim().to_string())
|
||||
.filter(|l| l.len() < 200 && !l.is_empty())
|
||||
.unwrap_or_else(|| filename.replace(".pdf", ""));
|
||||
|
||||
// Strategy 1: Look for "You said:" / "ChatGPT said:" markers (common format)
|
||||
let you_markers = ["You said:", "You:", "User:", "Human:"];
|
||||
let ai_markers = ["ChatGPT said:", "ChatGPT:", "Assistant:", "AI:"];
|
||||
|
||||
let has_markers = you_markers.iter().any(|m| text.contains(m))
|
||||
&& ai_markers.iter().any(|m| text.contains(m));
|
||||
|
||||
if has_markers {
|
||||
let lines: Vec<&str> = text.lines().collect();
|
||||
let mut current_role = String::new();
|
||||
let mut current_content = String::new();
|
||||
let mut current_page = 1i32;
|
||||
|
||||
for line in &lines {
|
||||
let trimmed = line.trim();
|
||||
|
||||
// Check for role markers
|
||||
let mut found_marker = false;
|
||||
for marker in &you_markers {
|
||||
if trimmed.starts_with(marker) || trimmed == *marker {
|
||||
// Save previous turn
|
||||
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||
turn_num += 1;
|
||||
turns.push(ConversationTurn {
|
||||
source_file: filename.to_string(),
|
||||
conversation_title: title.clone(),
|
||||
turn_number: turn_num,
|
||||
role: current_role.clone(),
|
||||
content: current_content.trim().to_string(),
|
||||
page_number: current_page,
|
||||
});
|
||||
}
|
||||
current_role = "user".to_string();
|
||||
current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
|
||||
current_content.push('\n');
|
||||
found_marker = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if !found_marker {
|
||||
for marker in &ai_markers {
|
||||
if trimmed.starts_with(marker) || trimmed == *marker {
|
||||
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||
turn_num += 1;
|
||||
turns.push(ConversationTurn {
|
||||
source_file: filename.to_string(),
|
||||
conversation_title: title.clone(),
|
||||
turn_number: turn_num,
|
||||
role: current_role.clone(),
|
||||
content: current_content.trim().to_string(),
|
||||
page_number: current_page,
|
||||
});
|
||||
}
|
||||
current_role = "assistant".to_string();
|
||||
current_content = trimmed.strip_prefix(marker).unwrap_or("").trim().to_string();
|
||||
current_content.push('\n');
|
||||
found_marker = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !found_marker && !current_role.is_empty() {
|
||||
current_content.push_str(trimmed);
|
||||
current_content.push('\n');
|
||||
}
|
||||
}
|
||||
|
||||
// Save last turn
|
||||
if !current_content.trim().is_empty() && !current_role.is_empty() {
|
||||
turn_num += 1;
|
||||
turns.push(ConversationTurn {
|
||||
source_file: filename.to_string(),
|
||||
conversation_title: title.clone(),
|
||||
turn_number: turn_num,
|
||||
role: current_role,
|
||||
content: current_content.trim().to_string(),
|
||||
page_number: current_page,
|
||||
});
|
||||
}
|
||||
|
||||
return turns;
|
||||
}
|
||||
|
||||
// Strategy 2: Split by paragraph blocks, alternate user/assistant
|
||||
// This is a heuristic fallback for PDFs without clear markers
|
||||
let paragraphs: Vec<&str> = text.split("\n\n")
|
||||
.map(|p| p.trim())
|
||||
.filter(|p| p.len() > 20) // skip short fragments
|
||||
.collect();
|
||||
|
||||
if paragraphs.len() >= 2 {
|
||||
for (i, para) in paragraphs.iter().enumerate() {
|
||||
turn_num += 1;
|
||||
turns.push(ConversationTurn {
|
||||
source_file: filename.to_string(),
|
||||
conversation_title: title.clone(),
|
||||
turn_number: turn_num,
|
||||
role: if i % 2 == 0 { "user" } else { "assistant" }.to_string(),
|
||||
content: para.to_string(),
|
||||
page_number: 1,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
turns
|
||||
}
|
||||
|
||||
/// Fallback: parse as raw pages with full text.
|
||||
fn parse_as_raw_pages(page_texts: &[(i32, String)], filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("source_file", DataType::Utf8, false),
|
||||
Field::new("conversation_title", DataType::Utf8, false),
|
||||
Field::new("turn_number", DataType::Int32, false),
|
||||
Field::new("role", DataType::Utf8, false),
|
||||
Field::new("content", DataType::Utf8, false),
|
||||
Field::new("page_number", DataType::Int32, false),
|
||||
]));
|
||||
|
||||
let sources: Vec<&str> = page_texts.iter().map(|_| filename).collect();
|
||||
let titles: Vec<String> = page_texts.iter().map(|_| filename.replace(".pdf", "")).collect();
|
||||
let turn_nums: Vec<i32> = (1..=page_texts.len() as i32).collect();
|
||||
let roles: Vec<&str> = page_texts.iter().map(|_| "page").collect();
|
||||
let contents: Vec<&str> = page_texts.iter().map(|(_, t)| t.as_str()).collect();
|
||||
let pages: Vec<i32> = page_texts.iter().map(|(p, _)| *p).collect();
|
||||
|
||||
let arrays: Vec<ArrayRef> = vec![
|
||||
Arc::new(StringArray::from(sources)),
|
||||
Arc::new(StringArray::from(titles.iter().map(|s| s.as_str()).collect::<Vec<_>>())),
|
||||
Arc::new(Int32Array::from(turn_nums)),
|
||||
Arc::new(StringArray::from(roles)),
|
||||
Arc::new(StringArray::from(contents)),
|
||||
Arc::new(Int32Array::from(pages)),
|
||||
];
|
||||
|
||||
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||
.map_err(|e| format!("RecordBatch error: {e}"))?;
|
||||
|
||||
Ok((schema, vec![batch]))
|
||||
}
|
||||
|
||||
fn turns_to_batch(turns: &[ConversationTurn]) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("source_file", DataType::Utf8, false),
|
||||
Field::new("conversation_title", DataType::Utf8, false),
|
||||
Field::new("turn_number", DataType::Int32, false),
|
||||
Field::new("role", DataType::Utf8, false),
|
||||
Field::new("content", DataType::Utf8, false),
|
||||
Field::new("page_number", DataType::Int32, false),
|
||||
]));
|
||||
|
||||
let arrays: Vec<ArrayRef> = vec![
|
||||
Arc::new(StringArray::from(turns.iter().map(|t| t.source_file.as_str()).collect::<Vec<_>>())),
|
||||
Arc::new(StringArray::from(turns.iter().map(|t| t.conversation_title.as_str()).collect::<Vec<_>>())),
|
||||
Arc::new(Int32Array::from(turns.iter().map(|t| t.turn_number).collect::<Vec<_>>())),
|
||||
Arc::new(StringArray::from(turns.iter().map(|t| t.role.as_str()).collect::<Vec<_>>())),
|
||||
Arc::new(StringArray::from(turns.iter().map(|t| t.content.as_str()).collect::<Vec<_>>())),
|
||||
Arc::new(Int32Array::from(turns.iter().map(|t| t.page_number).collect::<Vec<_>>())),
|
||||
];
|
||||
|
||||
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||
.map_err(|e| format!("RecordBatch error: {e}"))?;
|
||||
|
||||
Ok((schema, vec![batch]))
|
||||
}
|
||||
@@ -1,7 +1,5 @@
|
||||
pub mod chatgpt_parser;
|
||||
pub mod db_ingest;
|
||||
pub mod my_stream;
|
||||
pub mod pg_stream;
|
||||
pub mod schedule;
|
||||
pub mod detect;
|
||||
pub mod csv_ingest;
|
||||
pub mod json_ingest;
|
||||
|
||||
@@ -1,410 +0,0 @@
|
||||
/// Streaming MySQL ingest.
|
||||
///
|
||||
/// Mirrors `pg_stream` for MySQL sources. Same OFFSET-paginated strategy,
|
||||
/// same Parquet-streaming shape. Uses `mysql_async` (pure-rust) so we
|
||||
/// don't need a C client library at build time.
|
||||
///
|
||||
/// Type mapping follows ADR-010 (default to string on ambiguity):
|
||||
/// - Booleans (TINYINT(1)) and integer types map to Arrow Int32/Int64
|
||||
/// - Floating point and decimals → Float64
|
||||
/// - Everything else (text, varchar, json, date, timestamp) → Utf8
|
||||
///
|
||||
/// What's deliberately not supported (yet):
|
||||
/// - TLS connections — `minimal` feature is plain TCP only. Upgrade
|
||||
/// when a tenant actually needs it.
|
||||
/// - Keyset pagination — OFFSET scans are O(N²) at multi-million-row
|
||||
/// scale; same caveat as `pg_stream`.
|
||||
/// - BINARY/BLOB columns — currently rendered as base64 or empty string
|
||||
/// via Display-fallback.
|
||||
|
||||
use arrow::array::{ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray};
|
||||
use arrow::datatypes::{DataType, Field, Schema};
|
||||
use mysql_async::prelude::*;
|
||||
use mysql_async::{Conn, Opts, Row, Value};
|
||||
use parquet::arrow::ArrowWriter;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Request shape for MySQL streaming ingest.
|
||||
#[derive(Debug, Clone, serde::Deserialize)]
|
||||
pub struct MyStreamRequest {
|
||||
/// mysql://user:pass@host:port/db
|
||||
pub dsn: String,
|
||||
pub table: String,
|
||||
#[serde(default)]
|
||||
pub dataset_name: Option<String>,
|
||||
/// Rows per fetch. Default 10_000.
|
||||
#[serde(default)]
|
||||
pub batch_size: Option<usize>,
|
||||
/// Column to ORDER BY for stable pagination. If omitted, the first
|
||||
/// column returned by the schema probe is used.
|
||||
#[serde(default)]
|
||||
pub order_by: Option<String>,
|
||||
/// Hard cap on total rows (for sampling / previews).
|
||||
#[serde(default)]
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct MyStreamResult {
|
||||
pub table: String,
|
||||
pub rows: usize,
|
||||
pub batches: usize,
|
||||
pub columns: usize,
|
||||
pub schema: Vec<String>,
|
||||
pub parquet_bytes: u64,
|
||||
pub duration_secs: f32,
|
||||
}
|
||||
|
||||
/// Parsed DSN pieces. Kept local (rather than reusing pg's DbConfig) so
|
||||
/// the MySQL connector doesn't depend on the PG path.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct MyConfig {
|
||||
pub host: String,
|
||||
pub port: u16,
|
||||
pub user: String,
|
||||
pub password: String,
|
||||
pub database: String,
|
||||
}
|
||||
|
||||
impl MyConfig {
|
||||
/// Build a mysql_async `Opts` from the parsed config.
|
||||
pub fn to_opts(&self) -> Opts {
|
||||
let url = if self.password.is_empty() {
|
||||
format!(
|
||||
"mysql://{}@{}:{}/{}",
|
||||
percent(&self.user), percent(&self.host), self.port, percent(&self.database),
|
||||
)
|
||||
} else {
|
||||
format!(
|
||||
"mysql://{}:{}@{}:{}/{}",
|
||||
percent(&self.user), percent(&self.password),
|
||||
percent(&self.host), self.port, percent(&self.database),
|
||||
)
|
||||
};
|
||||
Opts::from_url(&url).expect("MyConfig.to_opts: rebuilt URL should parse")
|
||||
}
|
||||
}
|
||||
|
||||
/// Minimal URL-encoder for the few characters that commonly appear in
|
||||
/// MySQL passwords. mysql_async's URL parser expects valid URL-encoded
|
||||
/// components.
|
||||
fn percent(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
for c in s.chars() {
|
||||
match c {
|
||||
'@' | ':' | '/' | '?' | '#' | '[' | ']' | ' ' | '%' => {
|
||||
out.push_str(&format!("%{:02X}", c as u32));
|
||||
}
|
||||
_ => out.push(c),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Parse a mysql:// DSN.
|
||||
/// Supports: mysql://[user[:password]@]host[:port][/db]
|
||||
pub fn parse_dsn(dsn: &str) -> Result<MyConfig, String> {
|
||||
let rest = dsn
|
||||
.strip_prefix("mysql://")
|
||||
.ok_or_else(|| "DSN must start with mysql://".to_string())?;
|
||||
|
||||
let (auth_host, database) = match rest.split_once('/') {
|
||||
Some((ah, db)) => (ah, db.split('?').next().unwrap_or(db).to_string()),
|
||||
None => (rest, String::new()),
|
||||
};
|
||||
|
||||
let (userpass, hostport) = match auth_host.rsplit_once('@') {
|
||||
Some((up, hp)) => (Some(up), hp),
|
||||
None => (None, auth_host),
|
||||
};
|
||||
|
||||
let (user, password) = match userpass {
|
||||
Some(up) => match up.split_once(':') {
|
||||
Some((u, p)) => (u.to_string(), p.to_string()),
|
||||
None => (up.to_string(), String::new()),
|
||||
},
|
||||
None => ("root".to_string(), String::new()),
|
||||
};
|
||||
|
||||
let (host, port) = match hostport.rsplit_once(':') {
|
||||
Some((h, p)) => (
|
||||
h.to_string(),
|
||||
p.parse::<u16>().map_err(|_| format!("invalid port in DSN: {p}"))?,
|
||||
),
|
||||
None => (hostport.to_string(), 3306),
|
||||
};
|
||||
|
||||
if host.is_empty() {
|
||||
return Err("DSN has no host".into());
|
||||
}
|
||||
if database.is_empty() {
|
||||
return Err("DSN has no database path (mysql://... /db)".into());
|
||||
}
|
||||
|
||||
Ok(MyConfig { host, port, user, password, database })
|
||||
}
|
||||
|
||||
/// Stream a MySQL table as Parquet bytes.
|
||||
pub async fn stream_table_to_parquet(
|
||||
req: &MyStreamRequest,
|
||||
) -> Result<(bytes::Bytes, MyStreamResult), String> {
|
||||
let t0 = std::time::Instant::now();
|
||||
let config = parse_dsn(&req.dsn)?;
|
||||
let batch_size = req.batch_size.unwrap_or(10_000).max(1);
|
||||
|
||||
let mut conn = Conn::new(config.to_opts()).await
|
||||
.map_err(|e| format!("mysql connect: {e}"))?;
|
||||
|
||||
// Probe columns via information_schema — gives real type names that
|
||||
// we can map to Arrow dtypes without needing to fetch a row first.
|
||||
let columns = probe_columns(&mut conn, &config.database, &req.table).await?;
|
||||
if columns.is_empty() {
|
||||
return Err(format!("table '{}' not found or has no columns", req.table));
|
||||
}
|
||||
|
||||
let arrow_fields: Vec<Field> = columns
|
||||
.iter()
|
||||
.map(|(name, ty)| Field::new(name, mysql_type_to_arrow(ty), true))
|
||||
.collect();
|
||||
let schema = Arc::new(Schema::new(arrow_fields));
|
||||
let schema_report: Vec<String> = columns
|
||||
.iter()
|
||||
.map(|(n, t)| format!("{}:{}", n, t))
|
||||
.collect();
|
||||
|
||||
let order_col = req.order_by.clone().unwrap_or_else(|| columns[0].0.clone());
|
||||
|
||||
let mut buf: Vec<u8> = Vec::with_capacity(1024 * 1024);
|
||||
let mut writer = ArrowWriter::try_new(&mut buf, schema.clone(), None)
|
||||
.map_err(|e| format!("ArrowWriter init: {e}"))?;
|
||||
|
||||
let mut total_rows: usize = 0;
|
||||
let mut batch_count: usize = 0;
|
||||
let row_cap = req.limit.unwrap_or(usize::MAX);
|
||||
|
||||
loop {
|
||||
let remaining = row_cap.saturating_sub(total_rows);
|
||||
if remaining == 0 { break; }
|
||||
let fetch = remaining.min(batch_size);
|
||||
|
||||
// Backticks are MySQL's identifier quote. Forbid backticks in
|
||||
// table/column names to prevent injection — neither pg nor us
|
||||
// should be accepting those anyway.
|
||||
if req.table.contains('`') || order_col.contains('`') {
|
||||
return Err("table or order_by column contains backticks — refused".into());
|
||||
}
|
||||
let sql = format!(
|
||||
"SELECT * FROM `{}` ORDER BY `{}` LIMIT {} OFFSET {}",
|
||||
req.table, order_col, fetch, total_rows,
|
||||
);
|
||||
|
||||
let rows: Vec<Row> = conn.query(&sql).await
|
||||
.map_err(|e| format!("fetch batch at offset {total_rows}: {e}"))?;
|
||||
|
||||
if rows.is_empty() { break; }
|
||||
let n = rows.len();
|
||||
|
||||
let arrays: Vec<ArrayRef> = columns
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(idx, (_, ty))| rows_to_column(&rows, idx, ty))
|
||||
.collect::<Result<_, _>>()?;
|
||||
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||
.map_err(|e| format!("RecordBatch: {e}"))?;
|
||||
writer.write(&batch).map_err(|e| format!("ArrowWriter::write: {e}"))?;
|
||||
|
||||
total_rows += n;
|
||||
batch_count += 1;
|
||||
tracing::info!(
|
||||
"mysql stream '{}': fetched batch {} ({} rows, total {})",
|
||||
req.table, batch_count, n, total_rows,
|
||||
);
|
||||
|
||||
if n < fetch { break; }
|
||||
}
|
||||
|
||||
writer.close().map_err(|e| format!("ArrowWriter::close: {e}"))?;
|
||||
conn.disconnect().await.ok();
|
||||
|
||||
let result = MyStreamResult {
|
||||
table: req.table.clone(),
|
||||
rows: total_rows,
|
||||
batches: batch_count,
|
||||
columns: columns.len(),
|
||||
schema: schema_report,
|
||||
parquet_bytes: buf.len() as u64,
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
};
|
||||
|
||||
Ok((bytes::Bytes::from(buf), result))
|
||||
}
|
||||
|
||||
async fn probe_columns(
|
||||
conn: &mut Conn,
|
||||
schema: &str,
|
||||
table: &str,
|
||||
) -> Result<Vec<(String, String)>, String> {
|
||||
let sql = format!(
|
||||
"SELECT COLUMN_NAME, DATA_TYPE FROM information_schema.columns \
|
||||
WHERE table_schema = '{}' AND table_name = '{}' \
|
||||
ORDER BY ORDINAL_POSITION",
|
||||
schema.replace('\'', "''"),
|
||||
table.replace('\'', "''"),
|
||||
);
|
||||
let rows: Vec<(String, String)> = conn.query(&sql).await
|
||||
.map_err(|e| format!("probe columns: {e}"))?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// MySQL data type string → Arrow DataType. Conservative: anything we
|
||||
/// don't recognize becomes Utf8 (ADR-010).
|
||||
fn mysql_type_to_arrow(ty: &str) -> DataType {
|
||||
match ty.to_lowercase().as_str() {
|
||||
"tinyint" | "smallint" | "mediumint" | "int" | "integer" => DataType::Int32,
|
||||
"bigint" => DataType::Int64,
|
||||
"float" | "double" | "decimal" | "numeric" | "real" => DataType::Float64,
|
||||
"bit" | "bool" | "boolean" => DataType::Boolean,
|
||||
_ => DataType::Utf8,
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert a single column slice of MySQL rows into an Arrow array.
|
||||
fn rows_to_column(
|
||||
rows: &[Row],
|
||||
idx: usize,
|
||||
ty: &str,
|
||||
) -> Result<ArrayRef, String> {
|
||||
let arrow_ty = mysql_type_to_arrow(ty);
|
||||
match arrow_ty {
|
||||
DataType::Boolean => {
|
||||
let v: Vec<Option<bool>> = rows.iter().map(|r| cell_as_bool(r, idx)).collect();
|
||||
Ok(Arc::new(BooleanArray::from(v)))
|
||||
}
|
||||
DataType::Int32 => {
|
||||
let v: Vec<Option<i32>> = rows.iter().map(|r| cell_as_i64(r, idx).map(|n| n as i32)).collect();
|
||||
Ok(Arc::new(Int32Array::from(v)))
|
||||
}
|
||||
DataType::Int64 => {
|
||||
let v: Vec<Option<i64>> = rows.iter().map(|r| cell_as_i64(r, idx)).collect();
|
||||
Ok(Arc::new(Int64Array::from(v)))
|
||||
}
|
||||
DataType::Float64 => {
|
||||
let v: Vec<Option<f64>> = rows.iter().map(|r| cell_as_f64(r, idx)).collect();
|
||||
Ok(Arc::new(Float64Array::from(v)))
|
||||
}
|
||||
_ => {
|
||||
let v: Vec<Option<String>> = rows.iter().map(|r| cell_as_string(r, idx)).collect();
|
||||
Ok(Arc::new(StringArray::from(v)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn cell(r: &Row, idx: usize) -> &Value { r.as_ref(idx).unwrap_or(&Value::NULL) }
|
||||
|
||||
fn cell_as_bool(r: &Row, idx: usize) -> Option<bool> {
|
||||
match cell(r, idx) {
|
||||
Value::NULL => None,
|
||||
Value::Int(n) => Some(*n != 0),
|
||||
Value::UInt(n) => Some(*n != 0),
|
||||
Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| {
|
||||
match s.to_ascii_lowercase().as_str() {
|
||||
"true" | "1" | "y" | "yes" => Some(true),
|
||||
"false" | "0" | "n" | "no" => Some(false),
|
||||
_ => None,
|
||||
}
|
||||
}),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn cell_as_i64(r: &Row, idx: usize) -> Option<i64> {
|
||||
match cell(r, idx) {
|
||||
Value::NULL => None,
|
||||
Value::Int(n) => Some(*n),
|
||||
Value::UInt(n) => i64::try_from(*n).ok(),
|
||||
Value::Float(f) => Some(*f as i64),
|
||||
Value::Double(f) => Some(*f as i64),
|
||||
Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| s.parse().ok()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn cell_as_f64(r: &Row, idx: usize) -> Option<f64> {
|
||||
match cell(r, idx) {
|
||||
Value::NULL => None,
|
||||
Value::Int(n) => Some(*n as f64),
|
||||
Value::UInt(n) => Some(*n as f64),
|
||||
Value::Float(f) => Some(*f as f64),
|
||||
Value::Double(f) => Some(*f),
|
||||
Value::Bytes(b) => std::str::from_utf8(b).ok().and_then(|s| s.parse().ok()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn cell_as_string(r: &Row, idx: usize) -> Option<String> {
|
||||
match cell(r, idx) {
|
||||
Value::NULL => None,
|
||||
Value::Bytes(b) => Some(String::from_utf8_lossy(b).into_owned()),
|
||||
Value::Int(n) => Some(n.to_string()),
|
||||
Value::UInt(n) => Some(n.to_string()),
|
||||
Value::Float(f) => Some(f.to_string()),
|
||||
Value::Double(f) => Some(f.to_string()),
|
||||
Value::Date(y, mo, d, h, mi, s, _us) => {
|
||||
Some(format!("{:04}-{:02}-{:02} {:02}:{:02}:{:02}", y, mo, d, h, mi, s))
|
||||
}
|
||||
Value::Time(neg, days, h, mi, s, _us) => {
|
||||
let sign = if *neg { "-" } else { "" };
|
||||
Some(format!("{}{}d {:02}:{:02}:{:02}", sign, days, h, mi, s))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_full() {
|
||||
let c = parse_dsn("mysql://daisy:secret@db.example.com:3307/my_db").unwrap();
|
||||
assert_eq!(c.host, "db.example.com");
|
||||
assert_eq!(c.port, 3307);
|
||||
assert_eq!(c.user, "daisy");
|
||||
assert_eq!(c.password, "secret");
|
||||
assert_eq!(c.database, "my_db");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_default_port() {
|
||||
let c = parse_dsn("mysql://root@localhost/shop").unwrap();
|
||||
assert_eq!(c.port, 3306);
|
||||
assert_eq!(c.user, "root");
|
||||
assert_eq!(c.password, "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_no_auth() {
|
||||
let c = parse_dsn("mysql://127.0.0.1:3306/analytics").unwrap();
|
||||
assert_eq!(c.user, "root");
|
||||
assert_eq!(c.host, "127.0.0.1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_rejects_non_mysql() {
|
||||
assert!(parse_dsn("postgresql://host/db").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_requires_db() {
|
||||
assert!(parse_dsn("mysql://localhost:3306").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn type_map_basics() {
|
||||
assert_eq!(mysql_type_to_arrow("int"), DataType::Int32);
|
||||
assert_eq!(mysql_type_to_arrow("BIGINT"), DataType::Int64);
|
||||
assert_eq!(mysql_type_to_arrow("decimal"), DataType::Float64);
|
||||
assert_eq!(mysql_type_to_arrow("varchar"), DataType::Utf8);
|
||||
assert_eq!(mysql_type_to_arrow("json"), DataType::Utf8);
|
||||
assert_eq!(mysql_type_to_arrow("bool"), DataType::Boolean);
|
||||
}
|
||||
}
|
||||
@@ -4,19 +4,8 @@ use arrow::record_batch::RecordBatch;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Extract text from a PDF file.
|
||||
///
|
||||
/// Two-tier approach:
|
||||
/// 1. **Text extraction (lopdf)** — fast, works on digital/native PDFs.
|
||||
/// Returns one row per page: (source_file, page_number, text_content).
|
||||
/// 2. **OCR fallback (Tesseract)** — if text extraction yields zero
|
||||
/// pages (common with scanned documents, image-only PDFs, or PDFs
|
||||
/// exported from scanners without an OCR layer), we extract embedded
|
||||
/// images from the PDF and run Tesseract on each. Falls back
|
||||
/// gracefully if Tesseract isn't installed (returns an error
|
||||
/// directing the operator to install it).
|
||||
///
|
||||
/// Schema is identical for both paths so downstream consumers (chunker,
|
||||
/// vectord, queryd) don't need to know how the text was produced.
|
||||
/// Returns one row per page: (page_number, text_content).
|
||||
/// This handles text-based PDFs. Scanned/image PDFs need OCR (not implemented yet).
|
||||
pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||
let doc = lopdf::Document::load_mem(content)
|
||||
.map_err(|e| format!("PDF load error: {e}"))?;
|
||||
@@ -37,35 +26,13 @@ pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>,
|
||||
}
|
||||
}
|
||||
|
||||
// Tier 2: OCR fallback for scanned pages.
|
||||
if page_numbers.is_empty() {
|
||||
tracing::info!(
|
||||
"no extractable text in '{}' — attempting OCR on {} pages",
|
||||
source_filename, pages.len(),
|
||||
);
|
||||
let ocr_results = ocr_pdf_images(&doc, source_filename)?;
|
||||
if ocr_results.is_empty() {
|
||||
return Err(
|
||||
"PDF contains no extractable text and OCR produced no output. \
|
||||
Ensure tesseract is installed (`apt install tesseract-ocr tesseract-ocr-eng`)."
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
for (pg, text) in ocr_results {
|
||||
page_numbers.push(pg);
|
||||
page_texts.push(text);
|
||||
sources.push(source_filename.to_string());
|
||||
}
|
||||
tracing::info!(
|
||||
"OCR recovered {} pages from '{}'", page_numbers.len(), source_filename,
|
||||
);
|
||||
} else {
|
||||
tracing::info!(
|
||||
"extracted {} pages with text from PDF '{}'",
|
||||
page_numbers.len(), source_filename,
|
||||
);
|
||||
// PDF has no extractable text — likely scanned/image
|
||||
return Err("PDF contains no extractable text (may be scanned/image — OCR not yet supported)".into());
|
||||
}
|
||||
|
||||
tracing::info!("extracted {} pages with text from PDF '{}'", page_numbers.len(), source_filename);
|
||||
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("source_file", DataType::Utf8, false),
|
||||
Field::new("page_number", DataType::Int32, false),
|
||||
@@ -83,197 +50,3 @@ pub fn parse_pdf(content: &[u8], source_filename: &str) -> Result<(Arc<Schema>,
|
||||
|
||||
Ok((schema, vec![batch]))
|
||||
}
|
||||
|
||||
/// Extract embedded images from the PDF, write each to a temp file,
|
||||
/// run Tesseract, collect the OCR'd text. Returns (page_number, text)
|
||||
/// pairs for pages that yielded non-empty output.
|
||||
///
|
||||
/// Image extraction strategy: walk every page's resources looking for
|
||||
/// XObject streams with /Subtype /Image. Decode the raw stream bytes
|
||||
/// (lopdf handles FlateDecode/DCTDecode) and write them as-is to a
|
||||
/// temp file with the appropriate extension. Tesseract handles JPEG,
|
||||
/// PNG, TIFF, and BMP natively.
|
||||
fn ocr_pdf_images(
|
||||
doc: &lopdf::Document,
|
||||
source_filename: &str,
|
||||
) -> Result<Vec<(i32, String)>, String> {
|
||||
if !tesseract_available() {
|
||||
return Err(
|
||||
"tesseract not found on PATH — install with `apt install tesseract-ocr tesseract-ocr-eng`"
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
|
||||
let pages = doc.get_pages();
|
||||
let mut results: Vec<(i32, String)> = Vec::new();
|
||||
|
||||
for (&page_num, &page_id) in &pages {
|
||||
let images = extract_page_images(doc, page_id);
|
||||
if images.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
// OCR each image on this page and concatenate the text.
|
||||
let mut page_text = String::new();
|
||||
for (idx, img_bytes) in images.iter().enumerate() {
|
||||
match ocr_image_bytes(img_bytes, page_num, idx) {
|
||||
Ok(text) if !text.trim().is_empty() => {
|
||||
if !page_text.is_empty() { page_text.push('\n'); }
|
||||
page_text.push_str(text.trim());
|
||||
}
|
||||
Ok(_) => {} // empty OCR — skip
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"OCR failed on '{}' page {} image {}: {e}",
|
||||
source_filename, page_num, idx,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
if !page_text.is_empty() {
|
||||
results.push((page_num as i32, page_text));
|
||||
}
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
/// Extract raw image bytes from a PDF page's XObject resources.
|
||||
/// Handles the common case: page /Resources → /XObject dict → streams
|
||||
/// with /Subtype /Image.
|
||||
fn extract_page_images(
|
||||
doc: &lopdf::Document,
|
||||
page_id: lopdf::ObjectId,
|
||||
) -> Vec<Vec<u8>> {
|
||||
let mut images = Vec::new();
|
||||
|
||||
let page_obj = match doc.get_object(page_id) {
|
||||
Ok(o) => o,
|
||||
Err(_) => return images,
|
||||
};
|
||||
let page_dict = match page_obj.as_dict() {
|
||||
Ok(d) => d,
|
||||
Err(_) => return images,
|
||||
};
|
||||
|
||||
// Resolve /Resources → /XObject
|
||||
let resources = page_dict
|
||||
.get(b"Resources")
|
||||
.ok()
|
||||
.and_then(|r| doc.dereference(r).ok())
|
||||
.map(|(_, o)| o.clone());
|
||||
let xobjects = resources
|
||||
.as_ref()
|
||||
.and_then(|r| r.as_dict().ok())
|
||||
.and_then(|d| d.get(b"XObject").ok())
|
||||
.and_then(|x| doc.dereference(x).ok())
|
||||
.map(|(_, o)| o.clone());
|
||||
|
||||
let xobj_dict = match xobjects.as_ref().and_then(|x| x.as_dict().ok()) {
|
||||
Some(d) => d,
|
||||
None => return images,
|
||||
};
|
||||
|
||||
for (_name, obj_ref) in xobj_dict.iter() {
|
||||
let obj = match doc.dereference(obj_ref) {
|
||||
Ok((_, o)) => o.clone(),
|
||||
Err(_) => continue,
|
||||
};
|
||||
let stream = match obj.as_stream() {
|
||||
Ok(s) => s,
|
||||
Err(_) => continue,
|
||||
};
|
||||
// Check /Subtype == /Image
|
||||
let is_image = stream.dict.get(b"Subtype")
|
||||
.ok()
|
||||
.and_then(|s| s.as_name().ok())
|
||||
.map(|n| n == b"Image")
|
||||
.unwrap_or(false);
|
||||
if !is_image {
|
||||
continue;
|
||||
}
|
||||
// Get decoded content — lopdf handles FlateDecode/DCTDecode.
|
||||
let content = match stream.decompressed_content() {
|
||||
Ok(c) => c,
|
||||
Err(_) => {
|
||||
// Try raw content as fallback (some DCT streams are stored raw).
|
||||
stream.content.clone()
|
||||
}
|
||||
};
|
||||
if !content.is_empty() {
|
||||
images.push(content);
|
||||
}
|
||||
}
|
||||
images
|
||||
}
|
||||
|
||||
/// Run Tesseract on raw image bytes. Writes to a temp file, shells out,
|
||||
/// reads the output. Returns the OCR'd text.
|
||||
fn ocr_image_bytes(
|
||||
img_bytes: &[u8],
|
||||
page_num: u32,
|
||||
img_idx: usize,
|
||||
) -> Result<String, String> {
|
||||
use std::io::Write;
|
||||
use std::process::Command;
|
||||
|
||||
let tmp_dir = std::env::temp_dir();
|
||||
// Detect image format from magic bytes to pick the right extension.
|
||||
let ext = if img_bytes.starts_with(&[0xFF, 0xD8]) {
|
||||
"jpg"
|
||||
} else if img_bytes.starts_with(&[0x89, 0x50, 0x4E, 0x47]) {
|
||||
"png"
|
||||
} else if img_bytes.starts_with(b"II") || img_bytes.starts_with(b"MM") {
|
||||
"tiff"
|
||||
} else {
|
||||
// Tesseract can often handle raw pixel data with the right options,
|
||||
// but writing as PNG is safer. For truly unknown formats, try PNG
|
||||
// wrapping; if Tesseract can't read it, it'll error cleanly.
|
||||
"png"
|
||||
};
|
||||
|
||||
let img_path = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}.{ext}"));
|
||||
let out_base = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}_out"));
|
||||
let out_path = tmp_dir.join(format!("lh_ocr_p{page_num}_i{img_idx}_out.txt"));
|
||||
|
||||
// Write image bytes to temp file.
|
||||
let mut f = std::fs::File::create(&img_path)
|
||||
.map_err(|e| format!("create temp image: {e}"))?;
|
||||
f.write_all(img_bytes)
|
||||
.map_err(|e| format!("write temp image: {e}"))?;
|
||||
drop(f);
|
||||
|
||||
// Run Tesseract.
|
||||
let status = Command::new("tesseract")
|
||||
.arg(&img_path)
|
||||
.arg(&out_base)
|
||||
.arg("--oem").arg("3") // LSTM + legacy combined
|
||||
.arg("--psm").arg("6") // assume uniform block of text
|
||||
.arg("-l").arg("eng")
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.status()
|
||||
.map_err(|e| format!("tesseract exec: {e}"))?;
|
||||
|
||||
// Cleanup input regardless of outcome.
|
||||
let _ = std::fs::remove_file(&img_path);
|
||||
|
||||
if !status.success() {
|
||||
let _ = std::fs::remove_file(&out_path);
|
||||
return Err(format!("tesseract exited with status {status}"));
|
||||
}
|
||||
|
||||
let text = std::fs::read_to_string(&out_path).unwrap_or_default();
|
||||
let _ = std::fs::remove_file(&out_path);
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
/// Quick check: is `tesseract` on PATH?
|
||||
fn tesseract_available() -> bool {
|
||||
std::process::Command::new("tesseract")
|
||||
.arg("--version")
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.status()
|
||||
.map(|s| s.success())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
@@ -1,308 +0,0 @@
|
||||
/// Streaming PostgreSQL ingest.
|
||||
///
|
||||
/// The original `db_ingest::import_postgres_table` loads every row into
|
||||
/// memory before emitting Parquet — fine for small tables, blows up on
|
||||
/// 1M+ rows. This module paginates via `ORDER BY <pk> LIMIT N OFFSET M`
|
||||
/// and streams batches into an `ArrowWriter`, closing once exhausted.
|
||||
///
|
||||
/// Pagination is OFFSET-based (not keyset) — simpler, works on any table
|
||||
/// without needing to know the PK type. OFFSET N scans N rows per fetch,
|
||||
/// which becomes quadratic for very large tables. Upgrade to keyset when
|
||||
/// a user actually ingests something that hurts.
|
||||
|
||||
use arrow::array::{ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array, RecordBatch, StringArray};
|
||||
use arrow::datatypes::{DataType, Field, Schema};
|
||||
use parquet::arrow::ArrowWriter;
|
||||
use std::sync::Arc;
|
||||
use tokio_postgres::{Client, NoTls, types::Type as PgType};
|
||||
|
||||
use crate::db_ingest::DbConfig;
|
||||
|
||||
/// Request shape for streaming ingest — takes a DSN string and optional
|
||||
/// tuning knobs.
|
||||
#[derive(Debug, Clone, serde::Deserialize)]
|
||||
pub struct PgStreamRequest {
|
||||
/// postgresql://user:pass@host:port/db — alternative to DbConfig.
|
||||
pub dsn: String,
|
||||
pub table: String,
|
||||
#[serde(default)]
|
||||
pub dataset_name: Option<String>,
|
||||
/// Rows per fetch. Default 10_000.
|
||||
#[serde(default)]
|
||||
pub batch_size: Option<usize>,
|
||||
/// Column to ORDER BY for stable pagination. If omitted, the first
|
||||
/// column returned by the schema probe is used.
|
||||
#[serde(default)]
|
||||
pub order_by: Option<String>,
|
||||
/// Hard cap on total rows (for sampling / previews).
|
||||
#[serde(default)]
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct PgStreamResult {
|
||||
pub table: String,
|
||||
pub rows: usize,
|
||||
pub batches: usize,
|
||||
pub columns: usize,
|
||||
pub schema: Vec<String>,
|
||||
pub parquet_bytes: u64,
|
||||
pub duration_secs: f32,
|
||||
}
|
||||
|
||||
/// Parse a postgresql:// DSN into a DbConfig.
|
||||
/// Supports: postgresql://[user[:password]@]host[:port][/db]
|
||||
/// Does NOT support: query parameters (sslmode etc) — add if needed.
|
||||
pub fn parse_dsn(dsn: &str) -> Result<DbConfig, String> {
|
||||
let rest = dsn
|
||||
.strip_prefix("postgresql://")
|
||||
.or_else(|| dsn.strip_prefix("postgres://"))
|
||||
.ok_or_else(|| "DSN must start with postgresql:// or postgres://".to_string())?;
|
||||
|
||||
// Split off path (database) first.
|
||||
let (auth_host, database) = match rest.split_once('/') {
|
||||
Some((ah, db)) => (ah, db.split('?').next().unwrap_or(db).to_string()),
|
||||
None => (rest, "postgres".to_string()),
|
||||
};
|
||||
|
||||
// Split user[:password] from host[:port].
|
||||
let (userpass, hostport) = match auth_host.rsplit_once('@') {
|
||||
Some((up, hp)) => (Some(up), hp),
|
||||
None => (None, auth_host),
|
||||
};
|
||||
|
||||
let (user, password) = match userpass {
|
||||
Some(up) => match up.split_once(':') {
|
||||
Some((u, p)) => (u.to_string(), p.to_string()),
|
||||
None => (up.to_string(), String::new()),
|
||||
},
|
||||
None => ("postgres".to_string(), String::new()),
|
||||
};
|
||||
|
||||
let (host, port) = match hostport.rsplit_once(':') {
|
||||
Some((h, p)) => (
|
||||
h.to_string(),
|
||||
p.parse::<u16>().map_err(|_| format!("invalid port in DSN: {p}"))?,
|
||||
),
|
||||
None => (hostport.to_string(), 5432),
|
||||
};
|
||||
|
||||
if host.is_empty() {
|
||||
return Err("DSN has no host".into());
|
||||
}
|
||||
|
||||
Ok(DbConfig { host, port, database, user, password })
|
||||
}
|
||||
|
||||
/// Stream a Postgres table as Parquet bytes.
|
||||
///
|
||||
/// Returns the full Parquet payload + summary stats. The payload is
|
||||
/// written to an in-memory buffer via `ArrowWriter` so each batch gets
|
||||
/// columnar compression as it arrives — memory footprint stays roughly
|
||||
/// at one batch plus Parquet footer state.
|
||||
pub async fn stream_table_to_parquet(
|
||||
req: &PgStreamRequest,
|
||||
) -> Result<(bytes::Bytes, PgStreamResult), String> {
|
||||
let t0 = std::time::Instant::now();
|
||||
let config = parse_dsn(&req.dsn)?;
|
||||
let batch_size = req.batch_size.unwrap_or(10_000).max(1);
|
||||
|
||||
let (client, connection) = tokio_postgres::connect(&config.connection_string(), NoTls)
|
||||
.await
|
||||
.map_err(|e| format!("postgres connect: {e}"))?;
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = connection.await {
|
||||
tracing::error!("pg connection: {e}");
|
||||
}
|
||||
});
|
||||
|
||||
// Probe schema.
|
||||
let columns = probe_columns(&client, &req.table).await?;
|
||||
if columns.is_empty() {
|
||||
return Err(format!("table '{}' not found or has no columns", req.table));
|
||||
}
|
||||
|
||||
let arrow_fields: Vec<Field> = columns
|
||||
.iter()
|
||||
.map(|(name, pg)| Field::new(name, pg_type_to_arrow(pg), true))
|
||||
.collect();
|
||||
let schema = Arc::new(Schema::new(arrow_fields));
|
||||
let schema_report: Vec<String> = columns
|
||||
.iter()
|
||||
.map(|(n, t)| format!("{}:{}", n, t))
|
||||
.collect();
|
||||
|
||||
// Pagination key: user-specified or first column.
|
||||
let order_col = req.order_by.clone().unwrap_or_else(|| columns[0].0.clone());
|
||||
|
||||
// Stream batches into ArrowWriter.
|
||||
let mut buf: Vec<u8> = Vec::with_capacity(1024 * 1024);
|
||||
let mut writer = ArrowWriter::try_new(&mut buf, schema.clone(), None)
|
||||
.map_err(|e| format!("ArrowWriter init: {e}"))?;
|
||||
|
||||
let mut total_rows: usize = 0;
|
||||
let mut batch_count: usize = 0;
|
||||
let row_cap = req.limit.unwrap_or(usize::MAX);
|
||||
|
||||
loop {
|
||||
let remaining = row_cap.saturating_sub(total_rows);
|
||||
if remaining == 0 { break; }
|
||||
let fetch = remaining.min(batch_size);
|
||||
|
||||
let sql = format!(
|
||||
"SELECT * FROM \"{}\" ORDER BY \"{}\" LIMIT {} OFFSET {}",
|
||||
req.table, order_col, fetch, total_rows,
|
||||
);
|
||||
|
||||
let rows = client
|
||||
.query(&sql, &[])
|
||||
.await
|
||||
.map_err(|e| format!("fetch batch at offset {total_rows}: {e}"))?;
|
||||
|
||||
if rows.is_empty() { break; }
|
||||
|
||||
let n = rows.len();
|
||||
let arrays: Vec<ArrayRef> = columns
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(idx, (_, pg))| rows_to_column(&rows, idx, pg))
|
||||
.collect::<Result<_, _>>()?;
|
||||
let batch = RecordBatch::try_new(schema.clone(), arrays)
|
||||
.map_err(|e| format!("RecordBatch: {e}"))?;
|
||||
writer.write(&batch).map_err(|e| format!("ArrowWriter::write: {e}"))?;
|
||||
|
||||
total_rows += n;
|
||||
batch_count += 1;
|
||||
tracing::info!(
|
||||
"pg stream '{}': fetched batch {} ({} rows, total {})",
|
||||
req.table, batch_count, n, total_rows,
|
||||
);
|
||||
|
||||
if n < fetch { break; } // short read = end of table
|
||||
}
|
||||
|
||||
writer.close().map_err(|e| format!("ArrowWriter::close: {e}"))?;
|
||||
let parquet_bytes = buf.len() as u64;
|
||||
let duration = t0.elapsed().as_secs_f32();
|
||||
|
||||
let result = PgStreamResult {
|
||||
table: req.table.clone(),
|
||||
rows: total_rows,
|
||||
batches: batch_count,
|
||||
columns: columns.len(),
|
||||
schema: schema_report,
|
||||
parquet_bytes,
|
||||
duration_secs: duration,
|
||||
};
|
||||
|
||||
Ok((bytes::Bytes::from(buf), result))
|
||||
}
|
||||
|
||||
async fn probe_columns(client: &Client, table: &str) -> Result<Vec<(String, PgType)>, String> {
|
||||
let stmt = client
|
||||
.prepare(&format!("SELECT * FROM \"{}\" LIMIT 0", table))
|
||||
.await
|
||||
.map_err(|e| format!("prepare schema probe: {e}"))?;
|
||||
Ok(stmt
|
||||
.columns()
|
||||
.iter()
|
||||
.map(|c| (c.name().to_string(), c.type_().clone()))
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn pg_type_to_arrow(pg: &PgType) -> DataType {
|
||||
match *pg {
|
||||
PgType::BOOL => DataType::Boolean,
|
||||
PgType::INT2 | PgType::INT4 => DataType::Int32,
|
||||
PgType::INT8 | PgType::OID => DataType::Int64,
|
||||
PgType::FLOAT4 | PgType::FLOAT8 | PgType::NUMERIC => DataType::Float64,
|
||||
_ => DataType::Utf8,
|
||||
}
|
||||
}
|
||||
|
||||
fn rows_to_column(
|
||||
rows: &[tokio_postgres::Row],
|
||||
idx: usize,
|
||||
pg: &PgType,
|
||||
) -> Result<ArrayRef, String> {
|
||||
match *pg {
|
||||
PgType::BOOL => {
|
||||
let v: Vec<Option<bool>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
|
||||
Ok(Arc::new(BooleanArray::from(v)))
|
||||
}
|
||||
PgType::INT2 => {
|
||||
let v: Vec<Option<i32>> = rows.iter()
|
||||
.map(|r| r.try_get::<_, i16>(idx).ok().map(|x| x as i32)).collect();
|
||||
Ok(Arc::new(Int32Array::from(v)))
|
||||
}
|
||||
PgType::INT4 => {
|
||||
let v: Vec<Option<i32>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
|
||||
Ok(Arc::new(Int32Array::from(v)))
|
||||
}
|
||||
PgType::INT8 | PgType::OID => {
|
||||
let v: Vec<Option<i64>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
|
||||
Ok(Arc::new(Int64Array::from(v)))
|
||||
}
|
||||
PgType::FLOAT4 => {
|
||||
let v: Vec<Option<f64>> = rows.iter()
|
||||
.map(|r| r.try_get::<_, f32>(idx).ok().map(|x| x as f64)).collect();
|
||||
Ok(Arc::new(Float64Array::from(v)))
|
||||
}
|
||||
PgType::FLOAT8 => {
|
||||
let v: Vec<Option<f64>> = rows.iter().map(|r| r.try_get(idx).ok()).collect();
|
||||
Ok(Arc::new(Float64Array::from(v)))
|
||||
}
|
||||
_ => {
|
||||
// Safe-default per ADR-010: everything else becomes string.
|
||||
let v: Vec<Option<String>> = rows.iter().map(|r| {
|
||||
r.try_get::<_, String>(idx).ok()
|
||||
.or_else(|| r.try_get::<_, serde_json::Value>(idx).ok().map(|j| j.to_string()))
|
||||
.or_else(|| {
|
||||
// Timestamps + uuid, rendered via Display.
|
||||
r.try_get::<_, chrono::DateTime<chrono::Utc>>(idx).ok()
|
||||
.map(|t| t.to_rfc3339())
|
||||
})
|
||||
.or_else(|| r.try_get::<_, uuid::Uuid>(idx).ok().map(|u| u.to_string()))
|
||||
.or(Some(String::new()))
|
||||
}).collect();
|
||||
Ok(Arc::new(StringArray::from(v)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_full() {
|
||||
let c = parse_dsn("postgresql://daisy:secret@db.example.com:6543/my_db").unwrap();
|
||||
assert_eq!(c.host, "db.example.com");
|
||||
assert_eq!(c.port, 6543);
|
||||
assert_eq!(c.user, "daisy");
|
||||
assert_eq!(c.password, "secret");
|
||||
assert_eq!(c.database, "my_db");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_minimal() {
|
||||
let c = parse_dsn("postgresql://localhost/knowledge_base").unwrap();
|
||||
assert_eq!(c.host, "localhost");
|
||||
assert_eq!(c.port, 5432);
|
||||
assert_eq!(c.user, "postgres");
|
||||
assert_eq!(c.password, "");
|
||||
assert_eq!(c.database, "knowledge_base");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_no_password() {
|
||||
let c = parse_dsn("postgres://postgres@127.0.0.1:5432/mydb").unwrap();
|
||||
assert_eq!(c.user, "postgres");
|
||||
assert_eq!(c.password, "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_dsn_rejects_non_pg() {
|
||||
assert!(parse_dsn("http://host/db").is_err());
|
||||
}
|
||||
}
|
||||
@@ -28,29 +28,12 @@ pub struct IngestResult {
|
||||
}
|
||||
|
||||
/// Full ingest pipeline: detect → parse → dedup → store → register.
|
||||
///
|
||||
/// Bucket name (federation layer 2) is recorded on the ObjectRef so the
|
||||
/// catalog knows which bucket holds the data — defaults to "primary" via
|
||||
/// the ingest service's resolve_bucket helper.
|
||||
pub async fn ingest_file(
|
||||
filename: &str,
|
||||
content: &[u8],
|
||||
dataset_name: Option<&str>,
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
registry: &Registry,
|
||||
) -> Result<IngestResult, String> {
|
||||
ingest_file_to_bucket(filename, content, dataset_name, "primary", store, registry).await
|
||||
}
|
||||
|
||||
/// Same as `ingest_file` but with explicit target bucket name on the
|
||||
/// ObjectRef. Header-aware ingest endpoints call this directly.
|
||||
pub async fn ingest_file_to_bucket(
|
||||
filename: &str,
|
||||
content: &[u8],
|
||||
dataset_name: Option<&str>,
|
||||
bucket: &str,
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
registry: &Registry,
|
||||
) -> Result<IngestResult, String> {
|
||||
// 1. Detect file type
|
||||
let file_type = detect_file_type(filename, content);
|
||||
@@ -60,7 +43,13 @@ pub async fn ingest_file_to_bucket(
|
||||
let (schema, batches) = match file_type {
|
||||
FileType::Csv => csv_ingest::parse_csv(content)?,
|
||||
FileType::Json | FileType::NdJson => json_ingest::parse_json(content)?,
|
||||
FileType::Pdf => pdf_ingest::parse_pdf(content, filename)?,
|
||||
FileType::Pdf => {
|
||||
// Try ChatGPT parser first, fall back to generic PDF
|
||||
match crate::chatgpt_parser::parse_chatgpt_pdf(content, filename) {
|
||||
Ok(result) => result,
|
||||
Err(_) => pdf_ingest::parse_pdf(content, filename)?,
|
||||
}
|
||||
}
|
||||
FileType::Text => parse_text_file(content, filename)?,
|
||||
FileType::Unknown => return Err(format!("unknown file type for '{filename}'")),
|
||||
};
|
||||
@@ -115,7 +104,7 @@ pub async fn ingest_file_to_bucket(
|
||||
let schema_fp = fingerprint_schema(&schema);
|
||||
let now = chrono::Utc::now();
|
||||
let obj_ref = ObjectRef {
|
||||
bucket: bucket.to_string(),
|
||||
bucket: "data".to_string(),
|
||||
key: storage_key.clone(),
|
||||
size_bytes: parquet_size,
|
||||
created_at: now,
|
||||
@@ -164,10 +153,6 @@ pub async fn ingest_file_to_bucket(
|
||||
..Default::default()
|
||||
}).await;
|
||||
|
||||
// Phase C: if this dataset already had embeddings, they're now stale.
|
||||
// mark_embeddings_stale is a no-op for never-embedded datasets.
|
||||
let _ = registry.mark_embeddings_stale(&safe_name).await;
|
||||
|
||||
Ok(IngestResult {
|
||||
dataset_name: safe_name,
|
||||
file_type: format!("{:?}", file_type),
|
||||
|
||||
@@ -1,626 +0,0 @@
|
||||
//! Scheduled ingest — the production-substrate piece that turns the
|
||||
//! lakehouse from "manual API toolkit" into a system that runs itself.
|
||||
//!
|
||||
//! A `ScheduleDef` declares "what to ingest" + "when to fire." A
|
||||
//! background `Scheduler` task wakes periodically, asks each enabled
|
||||
//! schedule whether it's due, and runs the matching ingest path. Outcomes
|
||||
//! land back on the schedule so operators can see what happened without
|
||||
//! `journalctl` archaeology.
|
||||
//!
|
||||
//! Storage shape: one JSON file per schedule at
|
||||
//! `primary://_schedules/{id}.json`. Serializable via serde, rebuilt on
|
||||
//! startup. Same write-once shape as the rest of the catalog — no DB.
|
||||
//!
|
||||
//! Concurrency model: single-flight per schedule_id. The scheduler holds
|
||||
//! a `HashSet<String>` of currently-running ids; a tick that finds a
|
||||
//! schedule already in flight skips it (no queueing — a slow ingest
|
||||
//! shouldn't pile up runs). Different schedules run concurrently.
|
||||
//!
|
||||
//! What's deliberately not in scope here:
|
||||
//! - Cron expressions (the trigger enum has the variant but parsing is
|
||||
//! stubbed). Intervals cover 90% of operational scheduling; cron is
|
||||
//! easy to bolt on later.
|
||||
//! - Backoff / retry policies. A failed run records the failure and
|
||||
//! schedules `next_run_at` as if it succeeded — no exponential backoff.
|
||||
//! - Distributed coordination. Single-machine scheduler; no leader
|
||||
//! election. If you run two gateway instances they'll both fire.
|
||||
|
||||
use chrono::{DateTime, Duration, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use storaged::ops;
|
||||
|
||||
const PREFIX: &str = "_schedules";
|
||||
|
||||
// =================== Public types ===================
|
||||
|
||||
/// What kind of ingest the schedule fires.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(tag = "type", rename_all = "snake_case")]
|
||||
pub enum ScheduleKind {
|
||||
/// Pull a MySQL/MariaDB table via the my_stream path.
|
||||
Mysql {
|
||||
/// `mysql://user:pass@host:port/db` — DSN with full credentials.
|
||||
/// Stored encrypted-at-rest is a future concern; for now it lives
|
||||
/// in plain JSON in the catalog. Don't put production root creds
|
||||
/// here without disk-level encryption.
|
||||
dsn: String,
|
||||
table: String,
|
||||
#[serde(default)]
|
||||
dataset_name: Option<String>,
|
||||
#[serde(default)]
|
||||
batch_size: Option<usize>,
|
||||
#[serde(default)]
|
||||
order_by: Option<String>,
|
||||
},
|
||||
/// Pull a Postgres table via the pg_stream path.
|
||||
Postgres {
|
||||
dsn: String,
|
||||
table: String,
|
||||
#[serde(default)]
|
||||
dataset_name: Option<String>,
|
||||
#[serde(default)]
|
||||
batch_size: Option<usize>,
|
||||
#[serde(default)]
|
||||
order_by: Option<String>,
|
||||
},
|
||||
}
|
||||
|
||||
impl ScheduleKind {
|
||||
pub fn label(&self) -> String {
|
||||
match self {
|
||||
ScheduleKind::Mysql { table, .. } => format!("mysql:{table}"),
|
||||
ScheduleKind::Postgres { table, .. } => format!("postgres:{table}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// When the schedule fires.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(tag = "type", rename_all = "snake_case")]
|
||||
pub enum ScheduleTrigger {
|
||||
/// Run every N seconds. Time anchor = compute_next_run_at decision.
|
||||
Interval { secs: u64 },
|
||||
/// Cron expression — parsing not implemented yet. Defining the
|
||||
/// variant now so the JSON shape is forward-compatible.
|
||||
Cron { expr: String },
|
||||
}
|
||||
|
||||
/// Outcome of a single run, kept on the schedule for observability.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RunOutcome {
|
||||
pub at: DateTime<Utc>,
|
||||
pub success: bool,
|
||||
pub message: String,
|
||||
pub rows: Option<usize>,
|
||||
pub duration_secs: f32,
|
||||
}
|
||||
|
||||
/// One scheduled ingest definition.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ScheduleDef {
|
||||
pub id: String,
|
||||
pub kind: ScheduleKind,
|
||||
pub trigger: ScheduleTrigger,
|
||||
pub enabled: bool,
|
||||
pub created_at: DateTime<Utc>,
|
||||
#[serde(default)]
|
||||
pub created_by: String,
|
||||
/// When the next run is due. Recomputed after each completion via
|
||||
/// `compute_next_run_at`. On creation: set to "now" so the schedule
|
||||
/// fires on the next scheduler tick.
|
||||
pub next_run_at: DateTime<Utc>,
|
||||
#[serde(default)]
|
||||
pub last_run_at: Option<DateTime<Utc>>,
|
||||
#[serde(default)]
|
||||
pub last_outcome: Option<RunOutcome>,
|
||||
/// Total successful runs and total failures since creation. Useful
|
||||
/// for at-a-glance "is this schedule healthy?" checks.
|
||||
#[serde(default)]
|
||||
pub run_count: u64,
|
||||
#[serde(default)]
|
||||
pub failure_count: u64,
|
||||
}
|
||||
|
||||
impl ScheduleDef {
|
||||
/// True if this schedule should fire at `now`. Honors `enabled`.
|
||||
pub fn is_due(&self, now: DateTime<Utc>) -> bool {
|
||||
self.enabled && now >= self.next_run_at
|
||||
}
|
||||
}
|
||||
|
||||
// =================== Trigger semantics ===================
|
||||
//
|
||||
// J: THIS IS YOURS TO IMPLEMENT
|
||||
// =====================================================
|
||||
//
|
||||
// `compute_next_run_at` decides when a schedule fires next. The current
|
||||
// state is the "since last completion" semantics — simple but has real
|
||||
// trade-offs vs other strategies:
|
||||
//
|
||||
// 1. SINCE LAST COMPLETION (current default below):
|
||||
// next = completed_at + interval
|
||||
// Easy to reason about. A long-running ingest delays the next fire
|
||||
// by however long it took. Drift is bounded — the schedule never
|
||||
// "falls behind" trying to catch up.
|
||||
//
|
||||
// 2. ANCHORED INTERVAL:
|
||||
// next = previous_target + interval, regardless of when run finished
|
||||
// Keeps wall-clock alignment ("every 30min on the :00 and :30").
|
||||
// Risk: if ingest takes longer than the interval, the next run is
|
||||
// "due immediately" and you can fire back-to-back forever.
|
||||
//
|
||||
// 3. ANCHORED + SKIP:
|
||||
// Anchored, but skip a tick if the previous one is still running.
|
||||
// Best of both, slightly more code.
|
||||
//
|
||||
// 4. CRON:
|
||||
// Parse the expression, find the next match after `now`. Most
|
||||
// expressive but pulls in a cron parser dep.
|
||||
//
|
||||
// Cron is currently stubbed — falling back to "now + 1 hour" so a
|
||||
// mistakenly-configured Cron schedule doesn't melt the GPU.
|
||||
pub fn compute_next_run_at(
|
||||
trigger: &ScheduleTrigger,
|
||||
completed_at: DateTime<Utc>,
|
||||
_previous_next: DateTime<Utc>,
|
||||
) -> DateTime<Utc> {
|
||||
// TODO(J): pick a strategy. Starter = since-last-completion.
|
||||
match trigger {
|
||||
ScheduleTrigger::Interval { secs } => {
|
||||
completed_at + Duration::seconds(*secs as i64)
|
||||
}
|
||||
ScheduleTrigger::Cron { expr: _ } => {
|
||||
// Cron parsing not implemented — fall back to a safe 1h
|
||||
// window so a bad config can't fire-loop the system.
|
||||
completed_at + Duration::hours(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// =================== Persistence ===================
|
||||
|
||||
/// Storage path for a schedule's JSON record.
|
||||
fn schedule_key(id: &str) -> String {
|
||||
let safe: String = id
|
||||
.chars()
|
||||
.map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
|
||||
.collect();
|
||||
format!("{PREFIX}/{safe}.json")
|
||||
}
|
||||
|
||||
/// In-memory + on-disk schedule registry. The Scheduler reads from here;
|
||||
/// HTTP CRUD writes through here.
|
||||
#[derive(Clone)]
|
||||
pub struct ScheduleStore {
|
||||
/// Bucket where schedule JSON lives. Always primary today — a
|
||||
/// future enhancement could move to per-profile schedule buckets.
|
||||
store: Arc<dyn ObjectStore>,
|
||||
schedules: Arc<RwLock<HashMap<String, ScheduleDef>>>,
|
||||
}
|
||||
|
||||
impl ScheduleStore {
|
||||
pub fn new(store: Arc<dyn ObjectStore>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
schedules: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Load every persisted schedule into memory. Call on startup.
|
||||
pub async fn rebuild(&self) -> Result<usize, String> {
|
||||
let keys = ops::list(&self.store, Some(&format!("{PREFIX}/")))
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
let mut count = 0usize;
|
||||
for key in keys {
|
||||
if !key.ends_with(".json") { continue; }
|
||||
match ops::get(&self.store, &key).await {
|
||||
Ok(bytes) => match serde_json::from_slice::<ScheduleDef>(&bytes) {
|
||||
Ok(def) => {
|
||||
self.schedules.write().await.insert(def.id.clone(), def);
|
||||
count += 1;
|
||||
}
|
||||
Err(e) => tracing::warn!("schedule {key}: malformed: {e}"),
|
||||
},
|
||||
Err(e) => tracing::warn!("schedule {key}: read failed: {e}"),
|
||||
}
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
pub async fn list(&self) -> Vec<ScheduleDef> {
|
||||
let mut out: Vec<ScheduleDef> = self.schedules.read().await.values().cloned().collect();
|
||||
out.sort_by(|a, b| a.id.cmp(&b.id));
|
||||
out
|
||||
}
|
||||
|
||||
pub async fn get(&self, id: &str) -> Option<ScheduleDef> {
|
||||
self.schedules.read().await.get(id).cloned()
|
||||
}
|
||||
|
||||
pub async fn put(&self, def: ScheduleDef) -> Result<ScheduleDef, String> {
|
||||
let json = serde_json::to_vec_pretty(&def).map_err(|e| e.to_string())?;
|
||||
ops::put(&self.store, &schedule_key(&def.id), json.into()).await?;
|
||||
self.schedules.write().await.insert(def.id.clone(), def.clone());
|
||||
Ok(def)
|
||||
}
|
||||
|
||||
pub async fn delete(&self, id: &str) -> Result<bool, String> {
|
||||
// Object_store's delete is idempotent — silently OKs missing keys.
|
||||
let _ = ops::delete(&self.store, &schedule_key(id)).await;
|
||||
Ok(self.schedules.write().await.remove(id).is_some())
|
||||
}
|
||||
|
||||
/// Update completion metadata + persist. Used by the scheduler
|
||||
/// after a run finishes (success or failure).
|
||||
pub async fn record_run(
|
||||
&self,
|
||||
id: &str,
|
||||
outcome: RunOutcome,
|
||||
) -> Result<(), String> {
|
||||
let mut guard = self.schedules.write().await;
|
||||
let Some(def) = guard.get_mut(id) else {
|
||||
return Err(format!("schedule '{id}' not registered"));
|
||||
};
|
||||
def.last_run_at = Some(outcome.at);
|
||||
if outcome.success {
|
||||
def.run_count += 1;
|
||||
} else {
|
||||
def.failure_count += 1;
|
||||
}
|
||||
def.next_run_at = compute_next_run_at(&def.trigger, outcome.at, def.next_run_at);
|
||||
def.last_outcome = Some(outcome);
|
||||
|
||||
let json = serde_json::to_vec_pretty(def).map_err(|e| e.to_string())?;
|
||||
let key = schedule_key(id);
|
||||
let store = self.store.clone();
|
||||
// Drop the write lock before async I/O.
|
||||
drop(guard);
|
||||
ops::put(&store, &key, json.into()).await?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// =================== Scheduler task ===================
|
||||
|
||||
/// Long-running task that fires due schedules. Spawned at gateway
|
||||
/// startup. Stop semantics: drop the handle (the loop exits when its
|
||||
/// store is the only reference left, or sooner if you wire a stop signal).
|
||||
pub struct Scheduler {
|
||||
pub store: ScheduleStore,
|
||||
pub ingest: SchedulerIngestDeps,
|
||||
/// Polling cadence for the loop itself. The schedules' own intervals
|
||||
/// are independent — this just controls how granularly we check
|
||||
/// "is anything due?" Default 10s.
|
||||
pub tick_secs: u64,
|
||||
}
|
||||
|
||||
/// What the scheduler needs to actually run an ingest. We pass in only
|
||||
/// what's needed — keeps the dep set narrow.
|
||||
#[derive(Clone)]
|
||||
pub struct SchedulerIngestDeps {
|
||||
pub store: Arc<dyn ObjectStore>,
|
||||
pub registry: catalogd::registry::Registry,
|
||||
pub buckets: Arc<storaged::registry::BucketRegistry>,
|
||||
/// Optional Phase 16.5 hook — push a DatasetAppended event if any
|
||||
/// HNSW index is bound to the dataset that just got refreshed.
|
||||
pub agent_handle: Option<vectord::agent::AgentHandle>,
|
||||
pub index_registry: vectord::index_registry::IndexRegistry,
|
||||
}
|
||||
|
||||
impl Scheduler {
|
||||
/// Spawn the loop in the background. Returns immediately.
|
||||
pub fn spawn(self) {
|
||||
tokio::spawn(async move {
|
||||
self.run().await;
|
||||
});
|
||||
}
|
||||
|
||||
async fn run(self) {
|
||||
tracing::info!("scheduler started — tick={}s", self.tick_secs);
|
||||
let in_flight: Arc<RwLock<HashSet<String>>> = Arc::new(RwLock::new(HashSet::new()));
|
||||
|
||||
loop {
|
||||
tokio::time::sleep(std::time::Duration::from_secs(self.tick_secs)).await;
|
||||
let now = Utc::now();
|
||||
let due: Vec<ScheduleDef> = self.store.list().await
|
||||
.into_iter()
|
||||
.filter(|s| s.is_due(now))
|
||||
.collect();
|
||||
if due.is_empty() { continue; }
|
||||
|
||||
for def in due {
|
||||
{
|
||||
let in_flight_guard = in_flight.read().await;
|
||||
if in_flight_guard.contains(&def.id) {
|
||||
// Previous run still going — skip this tick.
|
||||
// We don't reschedule; next tick re-evaluates.
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
let store = self.store.clone();
|
||||
let ingest = self.ingest.clone();
|
||||
let in_flight = in_flight.clone();
|
||||
let id = def.id.clone();
|
||||
|
||||
in_flight.write().await.insert(id.clone());
|
||||
|
||||
tokio::spawn(async move {
|
||||
let t0 = std::time::Instant::now();
|
||||
let outcome = run_schedule(&def, &ingest).await;
|
||||
let elapsed = t0.elapsed().as_secs_f32();
|
||||
let outcome = RunOutcome {
|
||||
at: Utc::now(),
|
||||
success: outcome.is_ok(),
|
||||
message: match &outcome {
|
||||
Ok(s) => s.clone(),
|
||||
Err(e) => e.clone(),
|
||||
},
|
||||
rows: outcome.as_ref().ok().and_then(parse_rows_from_message),
|
||||
duration_secs: elapsed,
|
||||
};
|
||||
if let Err(e) = store.record_run(&id, outcome).await {
|
||||
tracing::warn!("scheduler: record_run({id}): {e}");
|
||||
}
|
||||
in_flight.write().await.remove(&id);
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Pluck a "rows=N" hint out of a success message — best-effort, no
|
||||
/// hard contract. Lets us surface row counts in the schedule status
|
||||
/// endpoint without forcing every ingest path to return structured
|
||||
/// stats here.
|
||||
fn parse_rows_from_message(msg: &String) -> Option<usize> {
|
||||
msg.split_whitespace()
|
||||
.find_map(|tok| tok.strip_prefix("rows=").and_then(|n| n.parse().ok()))
|
||||
}
|
||||
|
||||
/// Fire one schedule's ingest. Returns Ok(success_message) or Err(reason).
|
||||
pub async fn run_schedule(
|
||||
def: &ScheduleDef,
|
||||
deps: &SchedulerIngestDeps,
|
||||
) -> Result<String, String> {
|
||||
tracing::info!("scheduler: firing '{}' ({})", def.id, def.kind.label());
|
||||
match &def.kind {
|
||||
ScheduleKind::Mysql { dsn, table, dataset_name, batch_size, order_by } => {
|
||||
let req = crate::my_stream::MyStreamRequest {
|
||||
dsn: dsn.clone(),
|
||||
table: table.clone(),
|
||||
dataset_name: dataset_name.clone(),
|
||||
batch_size: *batch_size,
|
||||
order_by: order_by.clone(),
|
||||
limit: None,
|
||||
};
|
||||
run_mysql(req, deps).await
|
||||
}
|
||||
ScheduleKind::Postgres { dsn, table, dataset_name, batch_size, order_by } => {
|
||||
let req = crate::pg_stream::PgStreamRequest {
|
||||
dsn: dsn.clone(),
|
||||
table: table.clone(),
|
||||
dataset_name: dataset_name.clone(),
|
||||
batch_size: *batch_size,
|
||||
order_by: order_by.clone(),
|
||||
limit: None,
|
||||
};
|
||||
run_postgres(req, deps).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_mysql(
|
||||
req: crate::my_stream::MyStreamRequest,
|
||||
deps: &SchedulerIngestDeps,
|
||||
) -> Result<String, String> {
|
||||
let (parquet, result) = crate::my_stream::stream_table_to_parquet(&req).await?;
|
||||
if result.rows == 0 {
|
||||
return Ok(format!("rows=0 — table empty"));
|
||||
}
|
||||
persist_and_register(
|
||||
&parquet, &result.table, req.dataset_name.as_deref(),
|
||||
result.rows, "mysql", &req.dsn, deps,
|
||||
).await
|
||||
}
|
||||
|
||||
async fn run_postgres(
|
||||
req: crate::pg_stream::PgStreamRequest,
|
||||
deps: &SchedulerIngestDeps,
|
||||
) -> Result<String, String> {
|
||||
let (parquet, result) = crate::pg_stream::stream_table_to_parquet(&req).await?;
|
||||
if result.rows == 0 {
|
||||
return Ok(format!("rows=0 — table empty"));
|
||||
}
|
||||
persist_and_register(
|
||||
&parquet, &result.table, req.dataset_name.as_deref(),
|
||||
result.rows, "postgresql", &req.dsn, deps,
|
||||
).await
|
||||
}
|
||||
|
||||
/// Shared write+register path. Mirrors the HTTP ingest handlers in
|
||||
/// service.rs so the scheduled path produces identical catalog output:
|
||||
/// PII detection, lineage with redacted DSN, mark-stale + agent trigger.
|
||||
async fn persist_and_register(
|
||||
parquet: &bytes::Bytes,
|
||||
table: &str,
|
||||
dataset_name_override: Option<&str>,
|
||||
rows: usize,
|
||||
source_system: &str,
|
||||
dsn: &str,
|
||||
deps: &SchedulerIngestDeps,
|
||||
) -> Result<String, String> {
|
||||
use shared::types::{ColumnMeta, Lineage, ObjectRef, SchemaFingerprint, Sensitivity};
|
||||
|
||||
let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(parquet)
|
||||
.map_err(|e| format!("reparse parquet: {e}"))?;
|
||||
|
||||
let dataset_name = dataset_name_override.unwrap_or(table).to_string();
|
||||
let storage_key = format!("datasets/{}.parquet", dataset_name);
|
||||
let parquet_size = parquet.len() as u64;
|
||||
let bucket = "primary".to_string();
|
||||
|
||||
let target_store = deps.buckets.get(&bucket)?;
|
||||
ops::put(&target_store, &storage_key, parquet.clone()).await?;
|
||||
|
||||
let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
|
||||
let now = Utc::now();
|
||||
deps.registry.register(
|
||||
dataset_name.clone(),
|
||||
SchemaFingerprint(schema_fp.0),
|
||||
vec![ObjectRef {
|
||||
bucket: bucket.clone(),
|
||||
key: storage_key.clone(),
|
||||
size_bytes: parquet_size,
|
||||
created_at: now,
|
||||
}],
|
||||
).await?;
|
||||
|
||||
let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
|
||||
let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
|
||||
let columns: Vec<ColumnMeta> = schema.fields().iter().map(|f| {
|
||||
let sens = shared::pii::detect_sensitivity(f.name());
|
||||
ColumnMeta {
|
||||
name: f.name().clone(),
|
||||
data_type: f.data_type().to_string(),
|
||||
sensitivity: sens.clone(),
|
||||
description: String::new(),
|
||||
is_pii: matches!(sens, Some(Sensitivity::Pii)),
|
||||
}
|
||||
}).collect();
|
||||
|
||||
let lineage = Lineage {
|
||||
source_system: source_system.to_string(),
|
||||
source_file: format!("dsn: {}", redact_dsn(dsn)),
|
||||
ingest_job: format!("scheduled-{}-{}", source_system, now.timestamp_millis()),
|
||||
ingest_timestamp: now,
|
||||
parent_datasets: vec![],
|
||||
};
|
||||
|
||||
let _ = deps.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
|
||||
sensitivity,
|
||||
columns: Some(columns),
|
||||
lineage: Some(lineage),
|
||||
row_count: Some(rows as u64),
|
||||
..Default::default()
|
||||
}).await;
|
||||
|
||||
let _ = deps.registry.mark_embeddings_stale(&dataset_name).await;
|
||||
|
||||
if let Some(agent) = &deps.agent_handle {
|
||||
let bound = deps.index_registry.list(Some(&dataset_name), None).await;
|
||||
for meta in bound {
|
||||
let event = vectord::agent::TriggerEvent::dataset_appended(
|
||||
meta.index_name.clone(), &dataset_name,
|
||||
);
|
||||
let _ = agent.enqueue(event).await;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(format!("rows={rows} dataset={dataset_name} bytes={parquet_size}"))
|
||||
}
|
||||
|
||||
/// Same generic redactor used in service.rs — duplicated here to avoid
|
||||
/// pulling service::redact_dsn behind a `pub` visibility just for this.
|
||||
fn redact_dsn(dsn: &str) -> String {
|
||||
let scheme_end = match dsn.find("://") {
|
||||
Some(i) => i + 3,
|
||||
None => return dsn.to_string(),
|
||||
};
|
||||
let at_idx = match dsn.rfind('@') {
|
||||
Some(i) if i > scheme_end => i,
|
||||
_ => return dsn.to_string(),
|
||||
};
|
||||
let userpass = &dsn[scheme_end..at_idx];
|
||||
let colon_offset = match userpass.find(':') {
|
||||
Some(i) => i,
|
||||
None => return dsn.to_string(),
|
||||
};
|
||||
let colon_idx = scheme_end + colon_offset;
|
||||
format!("{}:***{}", &dsn[..colon_idx], &dsn[at_idx..])
|
||||
}
|
||||
|
||||
// =================== Tests ===================
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn mk(secs: u64) -> ScheduleDef {
|
||||
ScheduleDef {
|
||||
id: "test".into(),
|
||||
kind: ScheduleKind::Mysql {
|
||||
dsn: "mysql://x@host/db".into(),
|
||||
table: "t".into(),
|
||||
dataset_name: None,
|
||||
batch_size: None,
|
||||
order_by: None,
|
||||
},
|
||||
trigger: ScheduleTrigger::Interval { secs },
|
||||
enabled: true,
|
||||
created_at: Utc::now(),
|
||||
created_by: "tester".into(),
|
||||
next_run_at: Utc::now(),
|
||||
last_run_at: None,
|
||||
last_outcome: None,
|
||||
run_count: 0,
|
||||
failure_count: 0,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_due_respects_enabled() {
|
||||
let mut s = mk(60);
|
||||
s.next_run_at = Utc::now() - Duration::seconds(5);
|
||||
assert!(s.is_due(Utc::now()));
|
||||
s.enabled = false;
|
||||
assert!(!s.is_due(Utc::now()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_due_respects_future_time() {
|
||||
let mut s = mk(60);
|
||||
s.next_run_at = Utc::now() + Duration::seconds(60);
|
||||
assert!(!s.is_due(Utc::now()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn next_run_since_last_completion_advances() {
|
||||
let s = mk(120);
|
||||
let completed = Utc::now();
|
||||
let next = compute_next_run_at(&s.trigger, completed, completed);
|
||||
let delta = next - completed;
|
||||
assert_eq!(delta.num_seconds(), 120);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cron_falls_back_to_one_hour() {
|
||||
let trig = ScheduleTrigger::Cron { expr: "* * * * *".into() };
|
||||
let now = Utc::now();
|
||||
let next = compute_next_run_at(&trig, now, now);
|
||||
let delta = next - now;
|
||||
assert_eq!(delta.num_seconds(), 3600);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schedule_kind_label() {
|
||||
let k = ScheduleKind::Mysql {
|
||||
dsn: "x".into(), table: "customers".into(),
|
||||
dataset_name: None, batch_size: None, order_by: None,
|
||||
};
|
||||
assert_eq!(k.label(), "mysql:customers");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redact_dsn_handles_mysql_and_postgres() {
|
||||
assert_eq!(redact_dsn("mysql://u:secret@h/db"), "mysql://u:***@h/db");
|
||||
assert_eq!(redact_dsn("postgresql://u:secret@h:5432/db"), "postgresql://u:***@h:5432/db");
|
||||
assert_eq!(redact_dsn("mysql://u@h/db"), "mysql://u@h/db"); // no password
|
||||
}
|
||||
}
|
||||
@@ -1,9 +1,9 @@
|
||||
use axum::{
|
||||
Json, Router,
|
||||
extract::{Multipart, Path, Query, State},
|
||||
http::{HeaderMap, StatusCode},
|
||||
extract::{Multipart, Query, State},
|
||||
http::StatusCode,
|
||||
response::IntoResponse,
|
||||
routing::{delete, get, patch, post},
|
||||
routing::{get, post},
|
||||
};
|
||||
use bytes::Bytes;
|
||||
use object_store::ObjectStore;
|
||||
@@ -11,74 +11,15 @@ use serde::Deserialize;
|
||||
use std::sync::Arc;
|
||||
|
||||
use catalogd::registry::Registry;
|
||||
use crate::{db_ingest, my_stream, pg_stream, pipeline, schedule};
|
||||
use crate::{db_ingest, pipeline};
|
||||
use shared::arrow_helpers::record_batch_to_parquet;
|
||||
use shared::types::{ObjectRef, SchemaFingerprint};
|
||||
use storaged::ops;
|
||||
use storaged::registry::BucketRegistry;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct IngestState {
|
||||
pub store: Arc<dyn ObjectStore>,
|
||||
pub registry: Registry,
|
||||
/// Federation layer 2: lookup target bucket from request headers.
|
||||
pub buckets: Arc<BucketRegistry>,
|
||||
/// Phase 16.5: when ingest marks a dataset's embeddings stale, push
|
||||
/// a `DatasetAppended` trigger so the autotune agent can schedule
|
||||
/// re-trials. Holds the agent handle (no-op when agent disabled).
|
||||
pub agent_handle: vectord::agent::AgentHandle,
|
||||
/// Used to look up which HNSW indexes are attached to the
|
||||
/// just-ingested dataset. Each matching index gets one trigger.
|
||||
pub index_registry: vectord::index_registry::IndexRegistry,
|
||||
/// Scheduled-ingest registry. The scheduler task runs against this
|
||||
/// store; HTTP CRUD endpoints write through it.
|
||||
pub schedules: schedule::ScheduleStore,
|
||||
}
|
||||
|
||||
/// Push `DatasetAppended` triggers for every HNSW index bound to this
|
||||
/// dataset. Called after a successful ingest. Logs failures but never
|
||||
/// fails the ingest — the agent is best-effort, not load-bearing.
|
||||
async fn notify_agent_on_append(state: &IngestState, dataset_name: &str) {
|
||||
let indexes = state.index_registry.list(Some(dataset_name), None).await;
|
||||
if indexes.is_empty() {
|
||||
return;
|
||||
}
|
||||
for meta in indexes {
|
||||
let event = vectord::agent::TriggerEvent::dataset_appended(
|
||||
meta.index_name.clone(), dataset_name,
|
||||
);
|
||||
if let Err(e) = state.agent_handle.enqueue(event).await {
|
||||
tracing::warn!("agent enqueue failed for '{}': {}", meta.index_name, e);
|
||||
} else {
|
||||
tracing::info!(
|
||||
"agent: enqueued DatasetAppended({}) for index '{}'",
|
||||
dataset_name, meta.index_name,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve the target bucket from `X-Lakehouse-Bucket` header.
|
||||
/// Returns `(bucket_name, store_for_writes)`. Falls back to "primary"
|
||||
/// when the header is absent. Returns Err with the canonical bucket
|
||||
/// list when the header names an unknown bucket.
|
||||
fn resolve_bucket(
|
||||
headers: &HeaderMap,
|
||||
buckets: &BucketRegistry,
|
||||
) -> Result<(String, Arc<dyn ObjectStore>), (StatusCode, String)> {
|
||||
let target = headers
|
||||
.get("x-lakehouse-bucket")
|
||||
.and_then(|h| h.to_str().ok())
|
||||
.map(|s| s.to_string())
|
||||
.unwrap_or_else(|| buckets.default_name().to_string());
|
||||
|
||||
match buckets.get(&target) {
|
||||
Ok(store) => Ok((target, store)),
|
||||
Err(_) => Err((
|
||||
StatusCode::NOT_FOUND,
|
||||
format!("unknown bucket '{}' — use GET /storage/buckets to list", target),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn router(state: IngestState) -> Router {
|
||||
@@ -87,14 +28,6 @@ pub fn router(state: IngestState) -> Router {
|
||||
.route("/file", post(ingest_file))
|
||||
.route("/postgres/tables", post(list_pg_tables))
|
||||
.route("/postgres/import", post(import_pg_table))
|
||||
.route("/db", post(ingest_db_stream))
|
||||
.route("/mysql", post(ingest_mysql_stream))
|
||||
// Phase E: scheduled ingest
|
||||
.route("/schedules", get(list_schedules).post(create_schedule))
|
||||
.route("/schedules/{id}", get(get_schedule)
|
||||
.patch(patch_schedule)
|
||||
.delete(delete_schedule))
|
||||
.route("/schedules/{id}/run-now", post(run_schedule_now))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
@@ -110,11 +43,8 @@ struct IngestQuery {
|
||||
async fn ingest_file(
|
||||
State(state): State<IngestState>,
|
||||
Query(query): Query<IngestQuery>,
|
||||
headers: HeaderMap,
|
||||
mut multipart: Multipart,
|
||||
) -> impl IntoResponse {
|
||||
let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
|
||||
|
||||
let field = match multipart.next_field().await {
|
||||
Ok(Some(f)) => f,
|
||||
Ok(None) => return Err((StatusCode::BAD_REQUEST, "no file uploaded".to_string())),
|
||||
@@ -125,18 +55,12 @@ async fn ingest_file(
|
||||
let content = field.bytes().await
|
||||
.map_err(|e| (StatusCode::BAD_REQUEST, format!("read error: {e}")))?;
|
||||
|
||||
tracing::info!(
|
||||
"ingest '{}' ({} bytes) -> bucket={}",
|
||||
filename, content.len(), bucket,
|
||||
);
|
||||
tracing::info!("received file '{}' ({} bytes) for ingest", filename, content.len());
|
||||
|
||||
let dataset_name = query.name.as_deref();
|
||||
|
||||
match pipeline::ingest_file_to_bucket(&filename, &content, dataset_name, &bucket, &store, &state.registry).await {
|
||||
match pipeline::ingest_file(&filename, &content, dataset_name, &state.store, &state.registry).await {
|
||||
Ok(result) => {
|
||||
if !result.deduplicated {
|
||||
notify_agent_on_append(&state, &result.dataset_name).await;
|
||||
}
|
||||
if result.deduplicated {
|
||||
Ok((StatusCode::OK, Json(result)))
|
||||
} else {
|
||||
@@ -264,361 +188,3 @@ async fn import_pg_table(
|
||||
"size_bytes": parquet_size,
|
||||
}))))
|
||||
}
|
||||
|
||||
/// Streaming DSN-based ingest. Paginates via ORDER BY + LIMIT/OFFSET so
|
||||
/// large tables don't blow up memory. This is the path the task spec calls
|
||||
/// `POST /ingest/db` with `{dsn, table, dataset_name}`.
|
||||
async fn ingest_db_stream(
|
||||
State(state): State<IngestState>,
|
||||
headers: HeaderMap,
|
||||
Json(req): Json<pg_stream::PgStreamRequest>,
|
||||
) -> Result<(StatusCode, Json<serde_json::Value>), (StatusCode, String)> {
|
||||
let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
|
||||
tracing::info!(
|
||||
"pg stream ingest: table='{}' dataset='{:?}' bucket='{}' batch_size={:?}",
|
||||
req.table, req.dataset_name, bucket, req.batch_size,
|
||||
);
|
||||
|
||||
// Stream from Postgres into Parquet bytes.
|
||||
let (parquet, stream_result) = pg_stream::stream_table_to_parquet(&req)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::BAD_GATEWAY, e))?;
|
||||
|
||||
if stream_result.rows == 0 {
|
||||
return Ok((StatusCode::OK, Json(serde_json::json!({
|
||||
"table": req.table,
|
||||
"rows": 0,
|
||||
"message": "table is empty",
|
||||
}))));
|
||||
}
|
||||
|
||||
// Recover schema from the parquet footer — keeps a single source of truth
|
||||
// for the schema fingerprint + column metadata.
|
||||
let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(&parquet)
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("reparse parquet: {e}")))?;
|
||||
|
||||
let dataset_name = req.dataset_name.clone().unwrap_or_else(|| req.table.clone());
|
||||
let storage_key = format!("datasets/{}.parquet", dataset_name);
|
||||
let size_bytes = parquet.len() as u64;
|
||||
|
||||
ops::put(&store, &storage_key, parquet)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
|
||||
|
||||
let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
|
||||
let now = chrono::Utc::now();
|
||||
state.registry.register(
|
||||
dataset_name.clone(),
|
||||
SchemaFingerprint(schema_fp.0),
|
||||
vec![ObjectRef {
|
||||
bucket: bucket.clone(),
|
||||
key: storage_key.clone(),
|
||||
size_bytes,
|
||||
created_at: now,
|
||||
}],
|
||||
).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
|
||||
|
||||
// Rich metadata: PII auto-detect, lineage, row_count, columns.
|
||||
let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
|
||||
let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
|
||||
let columns: Vec<shared::types::ColumnMeta> = schema.fields().iter().map(|f| {
|
||||
let sens = shared::pii::detect_sensitivity(f.name());
|
||||
shared::types::ColumnMeta {
|
||||
name: f.name().clone(),
|
||||
data_type: f.data_type().to_string(),
|
||||
sensitivity: sens.clone(),
|
||||
description: String::new(),
|
||||
is_pii: matches!(sens, Some(shared::types::Sensitivity::Pii)),
|
||||
}
|
||||
}).collect();
|
||||
|
||||
let lineage = shared::types::Lineage {
|
||||
source_system: "postgresql".to_string(),
|
||||
source_file: format!("dsn: {}", redact_dsn(&req.dsn)),
|
||||
ingest_job: format!("pg-stream-{}", now.timestamp_millis()),
|
||||
ingest_timestamp: now,
|
||||
parent_datasets: vec![],
|
||||
};
|
||||
|
||||
let _ = state.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
|
||||
sensitivity,
|
||||
columns: Some(columns),
|
||||
lineage: Some(lineage),
|
||||
row_count: Some(stream_result.rows as u64),
|
||||
..Default::default()
|
||||
}).await;
|
||||
|
||||
// Phase C: mark embeddings stale if the dataset already had a vector
|
||||
// index. No-op for newly-created datasets.
|
||||
let _ = state.registry.mark_embeddings_stale(&dataset_name).await;
|
||||
|
||||
// Phase 16.5: notify autotune agent. No-op if no indexes attached.
|
||||
notify_agent_on_append(&state, &dataset_name).await;
|
||||
|
||||
Ok((StatusCode::CREATED, Json(serde_json::json!({
|
||||
"dataset_name": dataset_name,
|
||||
"table": stream_result.table,
|
||||
"rows": stream_result.rows,
|
||||
"batches": stream_result.batches,
|
||||
"columns": stream_result.columns,
|
||||
"schema": stream_result.schema,
|
||||
"storage_key": storage_key,
|
||||
"size_bytes": size_bytes,
|
||||
"duration_secs": stream_result.duration_secs,
|
||||
}))))
|
||||
}
|
||||
|
||||
/// Streaming MySQL ingest. Mirrors `ingest_db_stream` — same pagination +
|
||||
/// Parquet-footer schema recovery + PII/lineage metadata. Different driver,
|
||||
/// otherwise identical.
|
||||
async fn ingest_mysql_stream(
|
||||
State(state): State<IngestState>,
|
||||
headers: HeaderMap,
|
||||
Json(req): Json<my_stream::MyStreamRequest>,
|
||||
) -> Result<(StatusCode, Json<serde_json::Value>), (StatusCode, String)> {
|
||||
let (bucket, store) = resolve_bucket(&headers, &state.buckets)?;
|
||||
tracing::info!(
|
||||
"mysql stream ingest: table='{}' dataset='{:?}' bucket='{}' batch_size={:?}",
|
||||
req.table, req.dataset_name, bucket, req.batch_size,
|
||||
);
|
||||
|
||||
let (parquet, stream_result) = my_stream::stream_table_to_parquet(&req)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::BAD_GATEWAY, e))?;
|
||||
|
||||
if stream_result.rows == 0 {
|
||||
return Ok((StatusCode::OK, Json(serde_json::json!({
|
||||
"table": req.table,
|
||||
"rows": 0,
|
||||
"message": "table is empty",
|
||||
}))));
|
||||
}
|
||||
|
||||
let (schema, _) = shared::arrow_helpers::parquet_to_record_batches(&parquet)
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("reparse parquet: {e}")))?;
|
||||
|
||||
let dataset_name = req.dataset_name.clone().unwrap_or_else(|| req.table.clone());
|
||||
let storage_key = format!("datasets/{}.parquet", dataset_name);
|
||||
let size_bytes = parquet.len() as u64;
|
||||
|
||||
ops::put(&store, &storage_key, parquet)
|
||||
.await
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
|
||||
|
||||
let schema_fp = shared::arrow_helpers::fingerprint_schema(&schema);
|
||||
let now = chrono::Utc::now();
|
||||
state.registry.register(
|
||||
dataset_name.clone(),
|
||||
SchemaFingerprint(schema_fp.0),
|
||||
vec![ObjectRef {
|
||||
bucket: bucket.clone(),
|
||||
key: storage_key.clone(),
|
||||
size_bytes,
|
||||
created_at: now,
|
||||
}],
|
||||
).await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
|
||||
|
||||
let col_names: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
|
||||
let sensitivity = shared::pii::detect_dataset_sensitivity(&col_names);
|
||||
let columns: Vec<shared::types::ColumnMeta> = schema.fields().iter().map(|f| {
|
||||
let sens = shared::pii::detect_sensitivity(f.name());
|
||||
shared::types::ColumnMeta {
|
||||
name: f.name().clone(),
|
||||
data_type: f.data_type().to_string(),
|
||||
sensitivity: sens.clone(),
|
||||
description: String::new(),
|
||||
is_pii: matches!(sens, Some(shared::types::Sensitivity::Pii)),
|
||||
}
|
||||
}).collect();
|
||||
|
||||
let lineage = shared::types::Lineage {
|
||||
source_system: "mysql".to_string(),
|
||||
source_file: format!("dsn: {}", redact_dsn(&req.dsn)),
|
||||
ingest_job: format!("mysql-stream-{}", now.timestamp_millis()),
|
||||
ingest_timestamp: now,
|
||||
parent_datasets: vec![],
|
||||
};
|
||||
|
||||
let _ = state.registry.update_metadata(&dataset_name, catalogd::registry::MetadataUpdate {
|
||||
sensitivity,
|
||||
columns: Some(columns),
|
||||
lineage: Some(lineage),
|
||||
row_count: Some(stream_result.rows as u64),
|
||||
..Default::default()
|
||||
}).await;
|
||||
|
||||
// Phase C: mark stale + Phase 16.5: notify agent.
|
||||
let _ = state.registry.mark_embeddings_stale(&dataset_name).await;
|
||||
notify_agent_on_append(&state, &dataset_name).await;
|
||||
|
||||
Ok((StatusCode::CREATED, Json(serde_json::json!({
|
||||
"dataset_name": dataset_name,
|
||||
"table": stream_result.table,
|
||||
"rows": stream_result.rows,
|
||||
"batches": stream_result.batches,
|
||||
"columns": stream_result.columns,
|
||||
"schema": stream_result.schema,
|
||||
"storage_key": storage_key,
|
||||
"size_bytes": size_bytes,
|
||||
"duration_secs": stream_result.duration_secs,
|
||||
}))))
|
||||
}
|
||||
|
||||
/// Redact the password in any `scheme://user:pass@host/...` DSN. Works
|
||||
/// for postgres, mysql, and any other scheme using the same shape.
|
||||
fn redact_dsn(dsn: &str) -> String {
|
||||
let scheme_end = match dsn.find("://") {
|
||||
Some(i) => i + 3,
|
||||
None => return dsn.to_string(),
|
||||
};
|
||||
let at_idx = match dsn.rfind('@') {
|
||||
Some(i) if i > scheme_end => i,
|
||||
_ => return dsn.to_string(),
|
||||
};
|
||||
// Find the password separator (first `:` after scheme, before `@`).
|
||||
let userpass = &dsn[scheme_end..at_idx];
|
||||
let colon_offset = match userpass.find(':') {
|
||||
Some(i) => i,
|
||||
None => return dsn.to_string(), // no password in DSN
|
||||
};
|
||||
let colon_idx = scheme_end + colon_offset;
|
||||
format!("{}:***{}", &dsn[..colon_idx], &dsn[at_idx..])
|
||||
}
|
||||
|
||||
// --- Phase E: Scheduled ingest CRUD endpoints ---
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct CreateScheduleRequest {
|
||||
/// Stable id used in URLs and on disk. Required so callers can
|
||||
/// PATCH/DELETE deterministically. Sanitized on storage but the
|
||||
/// caller's id is what gets persisted as the canonical key.
|
||||
id: String,
|
||||
kind: schedule::ScheduleKind,
|
||||
trigger: schedule::ScheduleTrigger,
|
||||
#[serde(default = "default_enabled")]
|
||||
enabled: bool,
|
||||
#[serde(default)]
|
||||
created_by: String,
|
||||
}
|
||||
|
||||
fn default_enabled() -> bool { true }
|
||||
|
||||
async fn create_schedule(
|
||||
State(state): State<IngestState>,
|
||||
Json(req): Json<CreateScheduleRequest>,
|
||||
) -> impl IntoResponse {
|
||||
if state.schedules.get(&req.id).await.is_some() {
|
||||
return Err((StatusCode::CONFLICT, format!("schedule '{}' already exists", req.id)));
|
||||
}
|
||||
let now = chrono::Utc::now();
|
||||
let def = schedule::ScheduleDef {
|
||||
id: req.id,
|
||||
kind: req.kind,
|
||||
trigger: req.trigger,
|
||||
enabled: req.enabled,
|
||||
created_at: now,
|
||||
created_by: req.created_by,
|
||||
// Fire on the next tick — operators usually want to verify
|
||||
// their schedule works without waiting for the full interval.
|
||||
next_run_at: now,
|
||||
last_run_at: None,
|
||||
last_outcome: None,
|
||||
run_count: 0,
|
||||
failure_count: 0,
|
||||
};
|
||||
match state.schedules.put(def).await {
|
||||
Ok(d) => Ok((StatusCode::CREATED, Json(d))),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_schedules(State(state): State<IngestState>) -> impl IntoResponse {
|
||||
Json(state.schedules.list().await)
|
||||
}
|
||||
|
||||
async fn get_schedule(
|
||||
State(state): State<IngestState>,
|
||||
Path(id): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match state.schedules.get(&id).await {
|
||||
Some(d) => Ok(Json(d)),
|
||||
None => Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found"))),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct PatchScheduleRequest {
|
||||
/// Toggle on/off without changing anything else.
|
||||
#[serde(default)]
|
||||
enabled: Option<bool>,
|
||||
/// Replace the trigger spec — useful for bumping intervals.
|
||||
#[serde(default)]
|
||||
trigger: Option<schedule::ScheduleTrigger>,
|
||||
}
|
||||
|
||||
async fn patch_schedule(
|
||||
State(state): State<IngestState>,
|
||||
Path(id): Path<String>,
|
||||
Json(req): Json<PatchScheduleRequest>,
|
||||
) -> impl IntoResponse {
|
||||
let Some(mut def) = state.schedules.get(&id).await else {
|
||||
return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
|
||||
};
|
||||
if let Some(e) = req.enabled { def.enabled = e; }
|
||||
if let Some(t) = req.trigger { def.trigger = t; }
|
||||
match state.schedules.put(def).await {
|
||||
Ok(d) => Ok(Json(d)),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete_schedule(
|
||||
State(state): State<IngestState>,
|
||||
Path(id): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match state.schedules.delete(&id).await {
|
||||
Ok(true) => Ok(Json(serde_json::json!({ "deleted": id }))),
|
||||
Ok(false) => Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found"))),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Force a schedule to fire right now, regardless of its next_run_at.
|
||||
/// Records the outcome and reschedules. Useful for "test my new schedule
|
||||
/// works without waiting an hour."
|
||||
async fn run_schedule_now(
|
||||
State(state): State<IngestState>,
|
||||
Path(id): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
let Some(def) = state.schedules.get(&id).await else {
|
||||
return Err((StatusCode::NOT_FOUND, format!("schedule '{id}' not found")));
|
||||
};
|
||||
let deps = schedule::SchedulerIngestDeps {
|
||||
store: state.store.clone(),
|
||||
registry: state.registry.clone(),
|
||||
buckets: state.buckets.clone(),
|
||||
agent_handle: Some(state.agent_handle.clone()),
|
||||
index_registry: state.index_registry.clone(),
|
||||
};
|
||||
|
||||
let t0 = std::time::Instant::now();
|
||||
let result = schedule::run_schedule(&def, &deps).await;
|
||||
let elapsed = t0.elapsed().as_secs_f32();
|
||||
let outcome = schedule::RunOutcome {
|
||||
at: chrono::Utc::now(),
|
||||
success: result.is_ok(),
|
||||
message: match &result {
|
||||
Ok(s) => s.clone(),
|
||||
Err(e) => e.clone(),
|
||||
},
|
||||
rows: result.as_ref().ok().and_then(|m: &String| {
|
||||
m.split_whitespace()
|
||||
.find_map(|tok| tok.strip_prefix("rows=").and_then(|n| n.parse().ok()))
|
||||
}),
|
||||
duration_secs: elapsed,
|
||||
};
|
||||
if let Err(e) = state.schedules.record_run(&id, outcome.clone()).await {
|
||||
tracing::warn!("schedule {id}: record_run failed: {e}");
|
||||
}
|
||||
Ok(Json(outcome))
|
||||
}
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
[package]
|
||||
name = "lance-bench"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
# Standalone pilot for Phase B (see docs/EXECUTION_PLAN.md).
|
||||
# Deliberately NOT sharing workspace deps — Lance 4.x pulls in its own
|
||||
# DataFusion and Arrow versions incompatible with the rest of the stack.
|
||||
# Isolating the pilot means we don't force a workspace-wide upgrade until
|
||||
# we've decided Lance is worth it.
|
||||
|
||||
[dependencies]
|
||||
# Only the features we actually need — the default brings in AWS/Azure/GCP/HF etc
|
||||
# which is ~200 extra crates we don't care about for a local pilot.
|
||||
lance = { version = "4.0", default-features = false }
|
||||
# Lance exposes DatasetIndexExt, IndexType, and IvfBuildParams through
|
||||
# its sub-crates which must be imported directly — lance itself doesn't
|
||||
# re-export them at a convenient path.
|
||||
lance-index = { version = "4.0", default-features = false }
|
||||
lance-linalg = { version = "4.0", default-features = false }
|
||||
|
||||
# Arrow re-exported by Lance; pin to a range Lance picks so types match.
|
||||
arrow = "57"
|
||||
arrow-array = "57"
|
||||
arrow-schema = "57"
|
||||
|
||||
# Also need to read the EXISTING Parquet vector files so we can compare.
|
||||
# These live in data/vectors/*.parquet. Lance's internal Parquet reading
|
||||
# might differ from ours; using our format's Arrow/Parquet versions for
|
||||
# the read side keeps the inputs identical.
|
||||
parquet = "57"
|
||||
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
futures = "0.3"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
anyhow = "1"
|
||||
bytes = "1"
|
||||
|
||||
[[bin]]
|
||||
name = "lance-bench"
|
||||
path = "src/main.rs"
|
||||
@@ -1,633 +0,0 @@
|
||||
//! Phase B: Lance pilot benchmark.
|
||||
//!
|
||||
//! Standalone binary that compares Lance vector storage against our
|
||||
//! Parquet-with-binary-blob + in-RAM HNSW approach. See
|
||||
//! docs/EXECUTION_PLAN.md for the decision rules this fuels.
|
||||
//!
|
||||
//! Inputs:
|
||||
//! data/vectors/resumes_100k_v2.parquet — existing 100K × 768d embeddings
|
||||
//!
|
||||
//! Output:
|
||||
//! A JSON report printed to stdout with measurements for:
|
||||
//! - Cold load time (parquet → arrow) vs Lance open + scan
|
||||
//! - Disk size
|
||||
//! - Vector search latency (p50 / p95 / p99)
|
||||
//! - Single-row random access
|
||||
//! - Append cost (adding 10K rows)
|
||||
//!
|
||||
//! Usage:
|
||||
//! cargo run --bin lance-bench -- \
|
||||
//! --parquet data/vectors/resumes_100k_v2.parquet \
|
||||
//! --lance-out /tmp/lance_resumes_100k_v2 \
|
||||
//! --json-out /tmp/lance_bench.json
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use arrow_array::{Array, ArrayRef, BinaryArray, FixedSizeListArray, Float32Array, RecordBatch, RecordBatchIterator};
|
||||
use arrow_schema::{DataType, Field, Schema};
|
||||
use serde::Serialize;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct BenchReport {
|
||||
vectors: usize,
|
||||
dimensions: usize,
|
||||
parquet_path: String,
|
||||
lance_path: String,
|
||||
|
||||
// Parquet baseline
|
||||
parquet_disk_bytes: u64,
|
||||
parquet_cold_load_secs: f32,
|
||||
|
||||
// Lance numbers
|
||||
lance_write_secs: f32,
|
||||
lance_disk_bytes: u64,
|
||||
lance_cold_open_secs: f32,
|
||||
|
||||
// Index + search
|
||||
lance_index_build_secs: Option<f32>,
|
||||
lance_index_disk_bytes: Option<u64>,
|
||||
lance_search_p50_us: Option<f32>,
|
||||
lance_search_p95_us: Option<f32>,
|
||||
lance_search_p99_us: Option<f32>,
|
||||
|
||||
// Architectural features Parquet+sidecar can't cheaply do
|
||||
lance_random_row_access_us: Option<f32>, // fetch one row by row_id
|
||||
parquet_random_row_access_us: Option<f32>, // for comparison — full scan cost
|
||||
lance_append_10k_secs: Option<f32>, // add 10K new rows
|
||||
lance_append_disk_bytes_added: Option<u64>,
|
||||
|
||||
// Head-to-head reference (from our own measurements)
|
||||
reference_hnsw_p50_us: f32,
|
||||
reference_hnsw_p95_us: f32,
|
||||
reference_brute_force_us: f32,
|
||||
reference_hnsw_build_secs: f32,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
// Simple positional args: parquet_in, lance_out.
|
||||
let args: Vec<String> = std::env::args().collect();
|
||||
let parquet_path = args
|
||||
.get(1)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "data/vectors/resumes_100k_v2.parquet".to_string());
|
||||
let lance_path = args
|
||||
.get(2)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "/tmp/lance_bench_dataset".to_string());
|
||||
|
||||
eprintln!("=== Phase B Lance pilot ===");
|
||||
eprintln!("input parquet: {}", parquet_path);
|
||||
eprintln!("output lance: {}", lance_path);
|
||||
|
||||
// --- 1. Cold-load the existing Parquet vector index into memory
|
||||
eprintln!("\n[1/4] reading Parquet baseline...");
|
||||
let t0 = Instant::now();
|
||||
let (schema, batches, total_rows) = read_parquet_vectors(&parquet_path)
|
||||
.context("read parquet")?;
|
||||
let parquet_cold_load_secs = t0.elapsed().as_secs_f32();
|
||||
let parquet_disk_bytes = std::fs::metadata(&parquet_path)?.len();
|
||||
|
||||
let dims = detect_vector_dims(&batches)?;
|
||||
eprintln!(
|
||||
" loaded {} rows, {} columns, vectors={}d, disk={:.1} MB, cold load={:.2}s",
|
||||
total_rows,
|
||||
schema.fields().len(),
|
||||
dims,
|
||||
parquet_disk_bytes as f64 / 1_000_000.0,
|
||||
parquet_cold_load_secs,
|
||||
);
|
||||
|
||||
// --- 2. Convert from binary-blob-of-f32 to Lance's FixedSizeList<Float32>
|
||||
eprintln!("\n[2/4] converting binary-blob vectors to Arrow FixedSizeList...");
|
||||
let t0 = Instant::now();
|
||||
let (lance_schema, lance_batches) = convert_to_fixed_size_list(&schema, batches, dims)?;
|
||||
eprintln!(" conversion took {:.2}s", t0.elapsed().as_secs_f32());
|
||||
|
||||
// --- 3. Write as Lance dataset
|
||||
eprintln!("\n[3/4] writing Lance dataset...");
|
||||
let t0 = Instant::now();
|
||||
// Clean up any prior run
|
||||
let _ = std::fs::remove_dir_all(&lance_path);
|
||||
write_lance_dataset(&lance_path, lance_schema.clone(), lance_batches).await?;
|
||||
let lance_write_secs = t0.elapsed().as_secs_f32();
|
||||
let lance_disk_bytes = dir_size_bytes(&lance_path);
|
||||
eprintln!(
|
||||
" write took {:.2}s, disk={:.1} MB",
|
||||
lance_write_secs,
|
||||
lance_disk_bytes as f64 / 1_000_000.0,
|
||||
);
|
||||
|
||||
// --- 4. Cold open + scan the Lance dataset
|
||||
eprintln!("\n[4/6] cold-opening Lance dataset...");
|
||||
let t0 = Instant::now();
|
||||
let scanned_rows = cold_open_and_scan_lance(&lance_path).await?;
|
||||
let lance_cold_open_secs = t0.elapsed().as_secs_f32();
|
||||
eprintln!(
|
||||
" open + full scan: {} rows in {:.2}s",
|
||||
scanned_rows, lance_cold_open_secs,
|
||||
);
|
||||
|
||||
// --- 5. Build a vector index on the Lance dataset
|
||||
eprintln!("\n[5/6] building Lance vector index (IVF_PQ)...");
|
||||
let t0 = Instant::now();
|
||||
let index_built = build_lance_vector_index(&lance_path, dims).await;
|
||||
let (lance_index_build_secs, lance_index_disk_bytes) = match index_built {
|
||||
Ok(()) => {
|
||||
let secs = t0.elapsed().as_secs_f32();
|
||||
let disk = dir_size_bytes(&lance_path) - lance_disk_bytes;
|
||||
eprintln!(" built in {:.2}s, index adds {:.1} MB on disk", secs, disk as f64 / 1e6);
|
||||
(Some(secs), Some(disk))
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!(" index build failed: {e:#}");
|
||||
(None, None)
|
||||
}
|
||||
};
|
||||
|
||||
// --- 6. Run search queries, measure latency
|
||||
eprintln!("\n[6/6] running vector search benchmarks...");
|
||||
let search_stats = if lance_index_build_secs.is_some() {
|
||||
run_search_benchmarks(&lance_path, dims).await.ok()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let (lance_search_p50, lance_search_p95, lance_search_p99) = match search_stats {
|
||||
Some((p50, p95, p99)) => {
|
||||
eprintln!(" p50={:.0}us p95={:.0}us p99={:.0}us", p50, p95, p99);
|
||||
(Some(p50), Some(p95), Some(p99))
|
||||
}
|
||||
None => (None, None, None),
|
||||
};
|
||||
|
||||
// --- Random access comparison
|
||||
eprintln!("\n[7/8] random row access — Lance vs full-scan Parquet...");
|
||||
let lance_random = measure_random_access_lance(&lance_path).await.ok();
|
||||
let parquet_random = measure_random_access_parquet(&parquet_path).ok();
|
||||
if let Some(us) = lance_random {
|
||||
eprintln!(" Lance random-fetch avg: {:.0}us", us);
|
||||
}
|
||||
if let Some(us) = parquet_random {
|
||||
eprintln!(" Parquet full-scan-to-row avg: {:.0}us", us);
|
||||
}
|
||||
|
||||
// --- Append cost
|
||||
eprintln!("\n[8/8] append 10K new rows to existing dataset...");
|
||||
let t0 = Instant::now();
|
||||
let pre_append_bytes = dir_size_bytes(&lance_path);
|
||||
let append_result = append_10k_rows(&lance_path, dims).await;
|
||||
let (lance_append_secs, lance_append_bytes) = match append_result {
|
||||
Ok(()) => {
|
||||
let secs = t0.elapsed().as_secs_f32();
|
||||
let bytes = dir_size_bytes(&lance_path).saturating_sub(pre_append_bytes);
|
||||
eprintln!(" append took {:.2}s, added {:.1} MB", secs, bytes as f64 / 1e6);
|
||||
(Some(secs), Some(bytes))
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!(" append failed: {e:#}");
|
||||
(None, None)
|
||||
}
|
||||
};
|
||||
|
||||
// --- Report
|
||||
let report = BenchReport {
|
||||
vectors: total_rows,
|
||||
dimensions: dims,
|
||||
parquet_path: parquet_path.clone(),
|
||||
lance_path: lance_path.clone(),
|
||||
parquet_disk_bytes,
|
||||
parquet_cold_load_secs,
|
||||
lance_write_secs,
|
||||
lance_disk_bytes,
|
||||
lance_cold_open_secs,
|
||||
lance_index_build_secs,
|
||||
lance_index_disk_bytes,
|
||||
lance_search_p50_us: lance_search_p50,
|
||||
lance_search_p95_us: lance_search_p95,
|
||||
lance_search_p99_us: lance_search_p99,
|
||||
lance_random_row_access_us: lance_random,
|
||||
parquet_random_row_access_us: parquet_random,
|
||||
lance_append_10k_secs: lance_append_secs,
|
||||
lance_append_disk_bytes_added: lance_append_bytes,
|
||||
// From our Phase 15 trial on the SAME index (ec=80 es=30, recall=1.00):
|
||||
reference_hnsw_p50_us: 873.0,
|
||||
reference_hnsw_p95_us: 1413.0,
|
||||
reference_brute_force_us: 43983.0,
|
||||
reference_hnsw_build_secs: 230.0,
|
||||
};
|
||||
|
||||
let json = serde_json::to_string_pretty(&report)?;
|
||||
println!("{}", json);
|
||||
|
||||
eprintln!("\n=== Summary ===");
|
||||
eprintln!(" Parquet cold load: {:.2}s", report.parquet_cold_load_secs);
|
||||
eprintln!(" Lance cold open: {:.2}s ({})",
|
||||
report.lance_cold_open_secs,
|
||||
format_ratio(report.parquet_cold_load_secs, report.lance_cold_open_secs));
|
||||
eprintln!(" Parquet disk: {:.1} MB", report.parquet_disk_bytes as f64 / 1e6);
|
||||
eprintln!(" Lance disk: {:.1} MB ({})",
|
||||
report.lance_disk_bytes as f64 / 1e6,
|
||||
format_ratio(report.parquet_disk_bytes as f32, report.lance_disk_bytes as f32));
|
||||
if let (Some(p50), Some(p95)) = (report.lance_search_p50_us, report.lance_search_p95_us) {
|
||||
eprintln!(" Lance search p50: {:.0}us vs our HNSW {:.0}us ({})",
|
||||
p50, report.reference_hnsw_p50_us,
|
||||
format_ratio(report.reference_hnsw_p50_us, p50));
|
||||
eprintln!(" Lance search p95: {:.0}us vs our HNSW {:.0}us ({})",
|
||||
p95, report.reference_hnsw_p95_us,
|
||||
format_ratio(report.reference_hnsw_p95_us, p95));
|
||||
eprintln!(" Speedup vs brute force: {:.1}× (Lance) vs {:.1}× (HNSW)",
|
||||
report.reference_brute_force_us / p50,
|
||||
report.reference_brute_force_us / report.reference_hnsw_p50_us);
|
||||
}
|
||||
if let Some(build) = report.lance_index_build_secs {
|
||||
eprintln!(" Index build: {:.1}s (Lance IVF_PQ) vs {:.0}s (our HNSW ec=80) ({}× faster)",
|
||||
build, report.reference_hnsw_build_secs, report.reference_hnsw_build_secs / build);
|
||||
}
|
||||
if let (Some(lance_us), Some(parquet_us)) = (report.lance_random_row_access_us, report.parquet_random_row_access_us) {
|
||||
eprintln!(" Random row access: {:.0}us (Lance) vs {:.0}us (Parquet scan) ({})",
|
||||
lance_us, parquet_us, format_ratio(parquet_us, lance_us));
|
||||
}
|
||||
if let Some(append_secs) = report.lance_append_10k_secs {
|
||||
eprintln!(" Append 10K rows: {:.2}s (Lance native) [Parquet would require full rewrite]",
|
||||
append_secs);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn format_ratio(baseline: f32, candidate: f32) -> String {
|
||||
if candidate == 0.0 { return "inf".into(); }
|
||||
let ratio = baseline / candidate;
|
||||
if ratio >= 1.0 {
|
||||
format!("{:.2}× faster/smaller", ratio)
|
||||
} else {
|
||||
format!("{:.2}× slower/larger", 1.0 / ratio)
|
||||
}
|
||||
}
|
||||
|
||||
fn dir_size_bytes(path: &str) -> u64 {
|
||||
fn recurse(p: &std::path::Path) -> u64 {
|
||||
let Ok(meta) = std::fs::metadata(p) else { return 0; };
|
||||
if meta.is_file() { return meta.len(); }
|
||||
let Ok(entries) = std::fs::read_dir(p) else { return 0; };
|
||||
entries
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| recurse(&e.path()))
|
||||
.sum()
|
||||
}
|
||||
recurse(std::path::Path::new(path))
|
||||
}
|
||||
|
||||
/// Read the existing vector Parquet (binary-blob format: source, doc_id,
|
||||
/// chunk_idx, chunk_text, vector as Binary bytes).
|
||||
fn read_parquet_vectors(path: &str) -> Result<(Arc<Schema>, Vec<RecordBatch>, usize)> {
|
||||
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
|
||||
use std::fs::File;
|
||||
|
||||
let file = File::open(path).with_context(|| format!("open {path}"))?;
|
||||
let builder = ParquetRecordBatchReaderBuilder::try_new(file)?;
|
||||
let schema = builder.schema().clone();
|
||||
let reader = builder.build()?;
|
||||
let batches: Vec<RecordBatch> = reader.collect::<Result<Vec<_>, _>>()?;
|
||||
let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
|
||||
Ok((schema, batches, rows))
|
||||
}
|
||||
|
||||
fn detect_vector_dims(batches: &[RecordBatch]) -> Result<usize> {
|
||||
for batch in batches {
|
||||
let vector_col_idx = batch
|
||||
.schema()
|
||||
.index_of("vector")
|
||||
.context("no 'vector' column in parquet")?;
|
||||
let col = batch.column(vector_col_idx);
|
||||
if let Some(binary) = col.as_any().downcast_ref::<BinaryArray>() {
|
||||
for i in 0..binary.len() {
|
||||
if !binary.is_null(i) {
|
||||
let bytes = binary.value(i);
|
||||
return Ok(bytes.len() / 4); // f32 = 4 bytes
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
anyhow::bail!("could not determine vector dimensions")
|
||||
}
|
||||
|
||||
/// Convert our binary-blob vector representation into Arrow's native
|
||||
/// FixedSizeList<Float32> — that's what Lance expects for vector columns.
|
||||
fn convert_to_fixed_size_list(
|
||||
schema: &Arc<Schema>,
|
||||
batches: Vec<RecordBatch>,
|
||||
dims: usize,
|
||||
) -> Result<(Arc<Schema>, Vec<RecordBatch>)> {
|
||||
// New schema keeps everything identical but replaces the vector column
|
||||
// with a FixedSizeList<Float32, dims>.
|
||||
let new_fields: Vec<Arc<Field>> = schema
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|f| {
|
||||
if f.name() == "vector" {
|
||||
Arc::new(Field::new(
|
||||
"vector",
|
||||
DataType::FixedSizeList(
|
||||
Arc::new(Field::new("item", DataType::Float32, true)),
|
||||
dims as i32,
|
||||
),
|
||||
false,
|
||||
))
|
||||
} else {
|
||||
f.clone()
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
let new_schema = Arc::new(Schema::new(new_fields));
|
||||
|
||||
let mut new_batches = Vec::with_capacity(batches.len());
|
||||
for batch in batches {
|
||||
let vector_idx = batch.schema().index_of("vector")?;
|
||||
let mut new_arrays: Vec<ArrayRef> = Vec::with_capacity(batch.num_columns());
|
||||
for (i, col) in batch.columns().iter().enumerate() {
|
||||
if i == vector_idx {
|
||||
let binary = col
|
||||
.as_any()
|
||||
.downcast_ref::<BinaryArray>()
|
||||
.context("vector column must be Binary")?;
|
||||
let fsl = binary_to_fixed_size_list(binary, dims)?;
|
||||
new_arrays.push(Arc::new(fsl));
|
||||
} else {
|
||||
new_arrays.push(col.clone());
|
||||
}
|
||||
}
|
||||
new_batches.push(RecordBatch::try_new(new_schema.clone(), new_arrays)?);
|
||||
}
|
||||
|
||||
Ok((new_schema, new_batches))
|
||||
}
|
||||
|
||||
fn binary_to_fixed_size_list(binary: &BinaryArray, dims: usize) -> Result<FixedSizeListArray> {
|
||||
let n = binary.len();
|
||||
let mut all_floats: Vec<f32> = Vec::with_capacity(n * dims);
|
||||
for i in 0..n {
|
||||
if binary.is_null(i) {
|
||||
all_floats.extend(std::iter::repeat(0.0).take(dims));
|
||||
continue;
|
||||
}
|
||||
let bytes = binary.value(i);
|
||||
if bytes.len() != dims * 4 {
|
||||
anyhow::bail!(
|
||||
"row {} has {} bytes, expected {} ({} × f32)",
|
||||
i, bytes.len(), dims * 4, dims,
|
||||
);
|
||||
}
|
||||
for chunk in bytes.chunks_exact(4) {
|
||||
all_floats.push(f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]));
|
||||
}
|
||||
}
|
||||
let values = Float32Array::from(all_floats);
|
||||
let field = Arc::new(Field::new("item", DataType::Float32, true));
|
||||
FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None)
|
||||
.context("build FixedSizeListArray")
|
||||
}
|
||||
|
||||
/// Write batches into a Lance dataset at the given path.
|
||||
async fn write_lance_dataset(
|
||||
path: &str,
|
||||
schema: Arc<Schema>,
|
||||
batches: Vec<RecordBatch>,
|
||||
) -> Result<()> {
|
||||
use lance::dataset::{Dataset, WriteParams};
|
||||
|
||||
let reader = RecordBatchIterator::new(batches.into_iter().map(Ok), schema);
|
||||
Dataset::write(reader, path, Some(WriteParams::default()))
|
||||
.await
|
||||
.context("Dataset::write")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Open a Lance dataset cold (from disk) and scan it fully — measuring the
|
||||
/// equivalent of our "load embeddings from Parquet" cost.
|
||||
async fn cold_open_and_scan_lance(path: &str) -> Result<usize> {
|
||||
use futures::StreamExt;
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
let dataset = Dataset::open(path).await.context("Dataset::open")?;
|
||||
let scanner = dataset.scan();
|
||||
let mut stream = scanner.try_into_stream().await?;
|
||||
let mut total = 0usize;
|
||||
while let Some(batch) = stream.next().await {
|
||||
let batch = batch?;
|
||||
total += batch.num_rows();
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
/// Build an IVF_PQ vector index on the `vector` column. IVF_PQ (Inverted File
|
||||
/// with Product Quantization) is Lance's native ANN index — comparable to
|
||||
/// HNSW in intent, but on-disk and compatible with Lance's random-access
|
||||
/// model.
|
||||
async fn build_lance_vector_index(path: &str, _dims: usize) -> Result<()> {
|
||||
use lance::dataset::Dataset;
|
||||
use lance::index::vector::VectorIndexParams;
|
||||
use lance_index::{DatasetIndexExt, IndexType};
|
||||
use lance_linalg::distance::MetricType;
|
||||
|
||||
let mut dataset = Dataset::open(path).await?;
|
||||
|
||||
// IVF_PQ with ~sqrt(N) partitions is a reasonable default for 100K.
|
||||
// num_sub_vectors must divide dims evenly: 768/48 = 16 dims per subvector.
|
||||
// num_bits = 8 gives 256 codes per subvector (good recall/size trade).
|
||||
// max_iterations = 50 is plenty for this scale.
|
||||
let params = VectorIndexParams::ivf_pq(
|
||||
316, // num_partitions (~sqrt(100000))
|
||||
8, // num_bits
|
||||
48, // num_sub_vectors
|
||||
MetricType::Cosine,
|
||||
50, // max_iterations
|
||||
);
|
||||
|
||||
dataset
|
||||
.create_index(
|
||||
&["vector"],
|
||||
IndexType::Vector,
|
||||
Some("vec_idx".into()),
|
||||
¶ms,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
.context("create_index")?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Run N vector searches against the Lance dataset and return (p50, p95, p99) latencies in us.
|
||||
/// Uses a handful of random rows as queries — same pattern as our harness::synthetic_from_chunks.
|
||||
async fn run_search_benchmarks(path: &str, _dims: usize) -> Result<(f32, f32, f32)> {
|
||||
use futures::StreamExt;
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
let dataset = Dataset::open(path).await?;
|
||||
|
||||
// Pick 20 representative query vectors from the data itself.
|
||||
// (Synthetic — same pattern as our existing harness.)
|
||||
let query_vectors = sample_query_vectors(&dataset, 20).await?;
|
||||
|
||||
let mut latencies_us: Vec<f32> = Vec::with_capacity(query_vectors.len());
|
||||
for (i, qv) in query_vectors.iter().enumerate() {
|
||||
let qarr = Arc::new(Float32Array::from(qv.clone())) as ArrayRef;
|
||||
|
||||
let t0 = Instant::now();
|
||||
let mut scanner = dataset.scan();
|
||||
scanner
|
||||
.nearest("vector", qarr.as_any().downcast_ref::<Float32Array>().unwrap(), 10)
|
||||
.context("scanner.nearest")?;
|
||||
let mut stream = scanner.try_into_stream().await?;
|
||||
let mut hits = 0;
|
||||
while let Some(batch) = stream.next().await {
|
||||
let batch = batch?;
|
||||
hits += batch.num_rows();
|
||||
}
|
||||
let us = t0.elapsed().as_micros() as f32;
|
||||
latencies_us.push(us);
|
||||
if i == 0 {
|
||||
eprintln!(" first query: {} hits in {:.0}us (includes any lazy init)", hits, us);
|
||||
}
|
||||
}
|
||||
|
||||
latencies_us.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
let p = |pct: f32| -> f32 {
|
||||
let idx = ((latencies_us.len() as f32 - 1.0) * pct).round() as usize;
|
||||
latencies_us[idx.min(latencies_us.len() - 1)]
|
||||
};
|
||||
Ok((p(0.50), p(0.95), p(0.99)))
|
||||
}
|
||||
|
||||
/// Random row access via Lance's `take` — fetch 20 random rows by index, measure avg latency.
|
||||
async fn measure_random_access_lance(path: &str) -> Result<f32> {
|
||||
use lance::dataset::Dataset;
|
||||
let dataset = Dataset::open(path).await?;
|
||||
let n = dataset.count_rows(None).await?;
|
||||
let indices: Vec<u64> = (0..20).map(|i| ((i as u64) * (n as u64 / 23)) % (n as u64)).collect();
|
||||
|
||||
// Full-schema projection — Lance's Schema implements Into<ProjectionRequest>.
|
||||
let schema = dataset.schema().clone();
|
||||
let mut total_us: u128 = 0;
|
||||
for idx in &indices {
|
||||
let t0 = Instant::now();
|
||||
let _batch = dataset.take(&[*idx], schema.clone()).await?;
|
||||
total_us += t0.elapsed().as_micros();
|
||||
}
|
||||
Ok(total_us as f32 / indices.len() as f32)
|
||||
}
|
||||
|
||||
/// Random row access for Parquet — full scan + filter. There's no random-access
|
||||
/// primitive in vanilla Parquet, so this is the cost of finding one specific row.
|
||||
/// This is the cost our current design pays for "get doc X's full text for RAG."
|
||||
fn measure_random_access_parquet(path: &str) -> Result<f32> {
|
||||
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
|
||||
use std::fs::File;
|
||||
|
||||
// We simulate 5 lookups — full scan each time. 20 would be painful.
|
||||
let iters = 5;
|
||||
let mut total_us: u128 = 0;
|
||||
for _ in 0..iters {
|
||||
let t0 = Instant::now();
|
||||
let file = File::open(path)?;
|
||||
let builder = ParquetRecordBatchReaderBuilder::try_new(file)?;
|
||||
let reader = builder.build()?;
|
||||
// Iterate until we've conceptually found a row — we stop early if
|
||||
// we wanted row 50000, but we have to at least read its batch.
|
||||
let mut seen = 0usize;
|
||||
for b in reader {
|
||||
let b = b?;
|
||||
seen += b.num_rows();
|
||||
if seen > 50000 { break; }
|
||||
}
|
||||
total_us += t0.elapsed().as_micros();
|
||||
}
|
||||
Ok(total_us as f32 / iters as f32)
|
||||
}
|
||||
|
||||
/// Append 10K new rows to the existing Lance dataset.
|
||||
/// Measures the "ingest delta" cost without full rewrite.
|
||||
async fn append_10k_rows(path: &str, dims: usize) -> Result<()> {
|
||||
use lance::dataset::{Dataset, WriteMode, WriteParams};
|
||||
|
||||
let dataset = Dataset::open(path).await?;
|
||||
let schema = dataset.schema();
|
||||
let arrow_schema: Arc<Schema> = Arc::new(schema.into());
|
||||
|
||||
// Build a 10K row batch with random-ish data matching the existing schema.
|
||||
let n = 10_000;
|
||||
let arrays: Vec<ArrayRef> = arrow_schema
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|f| -> Result<ArrayRef> {
|
||||
match f.data_type() {
|
||||
DataType::Utf8 => {
|
||||
let vals: Vec<String> = (0..n).map(|i| format!("appended-{}", i)).collect();
|
||||
Ok(Arc::new(arrow_array::StringArray::from(vals)))
|
||||
}
|
||||
DataType::Int32 => {
|
||||
let vals: Vec<i32> = (0..n as i32).collect();
|
||||
Ok(Arc::new(arrow_array::Int32Array::from(vals)))
|
||||
}
|
||||
DataType::FixedSizeList(_, _) => {
|
||||
let floats: Vec<f32> = (0..n * dims).map(|i| (i as f32).sin()).collect();
|
||||
let values = Float32Array::from(floats);
|
||||
let field = Arc::new(Field::new("item", DataType::Float32, true));
|
||||
let fsl = FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None)?;
|
||||
Ok(Arc::new(fsl))
|
||||
}
|
||||
other => anyhow::bail!("unsupported append column type: {:?}", other),
|
||||
}
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
|
||||
let batch = RecordBatch::try_new(arrow_schema.clone(), arrays)?;
|
||||
let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), arrow_schema);
|
||||
let params = WriteParams { mode: WriteMode::Append, ..Default::default() };
|
||||
Dataset::write(reader, path, Some(params)).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Grab a few existing vectors from the dataset to use as self-similar queries.
|
||||
async fn sample_query_vectors(
|
||||
dataset: &lance::dataset::Dataset,
|
||||
count: usize,
|
||||
) -> Result<Vec<Vec<f32>>> {
|
||||
use futures::StreamExt;
|
||||
|
||||
// Just take the first `count` rows; good enough for latency measurement.
|
||||
let scanner = dataset.scan();
|
||||
let mut scanner = scanner;
|
||||
scanner.limit(Some(count as i64), None)?;
|
||||
scanner.project(&["vector"])?;
|
||||
let mut stream = scanner.try_into_stream().await?;
|
||||
|
||||
let mut out = Vec::with_capacity(count);
|
||||
while let Some(batch) = stream.next().await {
|
||||
let batch = batch?;
|
||||
let vector_col = batch
|
||||
.column(0)
|
||||
.as_any()
|
||||
.downcast_ref::<FixedSizeListArray>()
|
||||
.context("vector column must be FixedSizeList")?;
|
||||
|
||||
for row in 0..vector_col.len() {
|
||||
if out.len() >= count { break; }
|
||||
let values = vector_col.value(row);
|
||||
let f32_arr = values
|
||||
.as_any()
|
||||
.downcast_ref::<Float32Array>()
|
||||
.context("inner array must be Float32")?;
|
||||
let mut v = Vec::with_capacity(f32_arr.len());
|
||||
for i in 0..f32_arr.len() {
|
||||
v.push(f32_arr.value(i));
|
||||
}
|
||||
out.push(v);
|
||||
}
|
||||
if out.len() >= count { break; }
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
@@ -15,7 +15,6 @@ tracing = { workspace = true }
|
||||
datafusion = { workspace = true }
|
||||
object_store = { workspace = true }
|
||||
arrow = { workspace = true }
|
||||
parquet = { workspace = true }
|
||||
bytes = { workspace = true }
|
||||
futures = { workspace = true }
|
||||
url = { workspace = true }
|
||||
|
||||
+16
-211
@@ -5,7 +5,6 @@ use datafusion::datasource::file_format::parquet::ParquetFormat;
|
||||
use datafusion::prelude::*;
|
||||
use object_store::ObjectStore;
|
||||
use std::sync::Arc;
|
||||
use storaged::registry::BucketRegistry;
|
||||
use url::Url;
|
||||
|
||||
use crate::cache::MemCache;
|
||||
@@ -13,61 +12,27 @@ use crate::delta;
|
||||
|
||||
const STORE_SCHEME: &str = "lakehouse";
|
||||
|
||||
/// URL scheme used to register a bucket's object store with DataFusion.
|
||||
/// Bucket names safely embed in URL host components after sanitizing
|
||||
/// `:` (used in `profile:user`) to `-`.
|
||||
fn bucket_scheme(bucket: &str) -> String {
|
||||
format!("{STORE_SCHEME}-{}", bucket.replace(':', "-"))
|
||||
}
|
||||
|
||||
/// Build a `ListingTableUrl`-shaped string for an object in a bucket.
|
||||
fn bucket_object_url(bucket: &str, key: &str) -> String {
|
||||
format!("{}://data/{}", bucket_scheme(bucket), key)
|
||||
}
|
||||
|
||||
/// Build the registration base URL for a bucket.
|
||||
fn bucket_base_url(bucket: &str) -> String {
|
||||
format!("{}://data/", bucket_scheme(bucket))
|
||||
}
|
||||
|
||||
/// Query engine with in-memory cache and delta merge support.
|
||||
/// Federation layer 2: holds a `BucketRegistry` so cross-bucket queries
|
||||
/// route reads to the right ObjectStore based on `ObjectRef.bucket`.
|
||||
#[derive(Clone)]
|
||||
pub struct QueryEngine {
|
||||
registry: Registry,
|
||||
buckets: Arc<BucketRegistry>,
|
||||
/// Primary bucket store cached for backward-compat callers (compact,
|
||||
/// workspace ops) that haven't been migrated to bucket-aware paths.
|
||||
store: Arc<dyn ObjectStore>,
|
||||
cache: MemCache,
|
||||
}
|
||||
|
||||
impl QueryEngine {
|
||||
pub fn new(registry: Registry, buckets: Arc<BucketRegistry>, cache: MemCache) -> Self {
|
||||
let store = buckets.default_store();
|
||||
Self { registry, buckets, store, cache }
|
||||
pub fn new(registry: Registry, store: Arc<dyn ObjectStore>, cache: MemCache) -> Self {
|
||||
Self { registry, store, cache }
|
||||
}
|
||||
|
||||
pub fn cache(&self) -> &MemCache {
|
||||
&self.cache
|
||||
}
|
||||
|
||||
/// Backward-compat: returns the primary bucket's store. Bucket-aware
|
||||
/// callers should use `engine.buckets()` directly.
|
||||
pub fn store(&self) -> &Arc<dyn ObjectStore> {
|
||||
&self.store
|
||||
}
|
||||
|
||||
pub fn buckets(&self) -> &Arc<BucketRegistry> {
|
||||
&self.buckets
|
||||
}
|
||||
|
||||
/// Catalog registry — used by compact/tombstone integration.
|
||||
pub fn catalog(&self) -> &Registry {
|
||||
&self.registry
|
||||
}
|
||||
|
||||
/// Execute a SQL query. Uses cache for hot data, falls back to Parquet.
|
||||
pub async fn query(&self, sql: &str) -> Result<Vec<arrow::array::RecordBatch>, String> {
|
||||
let ctx = self.build_context().await?;
|
||||
@@ -78,16 +43,11 @@ impl QueryEngine {
|
||||
|
||||
/// Pin a dataset into the memory cache.
|
||||
pub async fn pin_dataset(&self, name: &str) -> Result<(), String> {
|
||||
// Read from Parquet
|
||||
let ctx = SessionContext::new();
|
||||
|
||||
// Register every bucket so a multi-bucket dataset can be pinned.
|
||||
for info in self.buckets.list().await {
|
||||
let url = Url::parse(&bucket_base_url(&info.name))
|
||||
.map_err(|e| format!("invalid store url: {e}"))?;
|
||||
let store = self.buckets.get(&info.name)
|
||||
.map_err(|e| format!("registry inconsistency: {e}"))?;
|
||||
ctx.runtime_env().register_object_store(&url, store);
|
||||
}
|
||||
let base_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
|
||||
.map_err(|e| format!("invalid store url: {e}"))?;
|
||||
ctx.runtime_env().register_object_store(&base_url, self.store.clone());
|
||||
|
||||
let dataset = self.registry.get_by_name(name).await
|
||||
.ok_or_else(|| format!("dataset not found: {name}"))?;
|
||||
@@ -98,10 +58,7 @@ impl QueryEngine {
|
||||
|
||||
let opts = ListingOptions::new(Arc::new(ParquetFormat::default()));
|
||||
let table_paths: Vec<ListingTableUrl> = dataset.objects.iter()
|
||||
.filter_map(|o| {
|
||||
let bucket = if o.bucket.is_empty() { "primary" } else { &o.bucket };
|
||||
ListingTableUrl::parse(&bucket_object_url(bucket, &o.key)).ok()
|
||||
})
|
||||
.filter_map(|o| ListingTableUrl::parse(&format!("{STORE_SCHEME}://data/{}", o.key)).ok())
|
||||
.collect();
|
||||
|
||||
let schema = opts.infer_schema(&ctx.state(), &table_paths[0]).await
|
||||
@@ -122,41 +79,9 @@ impl QueryEngine {
|
||||
async fn build_context(&self) -> Result<SessionContext, String> {
|
||||
let ctx = SessionContext::new();
|
||||
|
||||
// Phase E: snapshot tombstones by dataset before registering tables
|
||||
// so we can wrap tombstoned tables in a filter view. The underlying
|
||||
// base table is registered under an internal name `__raw__{dataset}`
|
||||
// and the public `{dataset}` name becomes the filtered view.
|
||||
let all_dataset_names: Vec<String> = self.registry.list().await
|
||||
.iter().map(|d| d.name.clone()).collect();
|
||||
let tombstones_by_dataset = self
|
||||
.registry
|
||||
.tombstones()
|
||||
.all_grouped(&all_dataset_names)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
|
||||
// Federation layer 2: register every configured bucket as its own
|
||||
// DataFusion ObjectStore under a distinct URL scheme. Each
|
||||
// dataset's ObjectRef.bucket determines which store DataFusion
|
||||
// routes to at scan time.
|
||||
let bucket_infos = self.buckets.list().await;
|
||||
for info in &bucket_infos {
|
||||
let url_str = bucket_base_url(&info.name);
|
||||
let url = Url::parse(&url_str)
|
||||
.map_err(|e| format!("invalid store url '{url_str}': {e}"))?;
|
||||
// unknown-bucket here would be a config invariant violation,
|
||||
// since we just listed them — propagate as an error to surface it.
|
||||
let store = self.buckets.get(&info.name)
|
||||
.map_err(|e| format!("registry inconsistency: {e}"))?;
|
||||
ctx.runtime_env().register_object_store(&url, store);
|
||||
}
|
||||
|
||||
// Backward-compat: also register the legacy `lakehouse://data/`
|
||||
// URL pointing at primary, in case any code path still constructs
|
||||
// ListingTableUrls without a bucket prefix.
|
||||
let legacy_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
|
||||
.map_err(|e| format!("invalid legacy store url: {e}"))?;
|
||||
ctx.runtime_env().register_object_store(&legacy_url, self.store.clone());
|
||||
let base_url = Url::parse(&format!("{STORE_SCHEME}://data/"))
|
||||
.map_err(|e| format!("invalid store url: {e}"))?;
|
||||
ctx.runtime_env().register_object_store(&base_url, self.store.clone());
|
||||
|
||||
let datasets = self.registry.list().await;
|
||||
for dataset in &datasets {
|
||||
@@ -164,9 +89,9 @@ impl QueryEngine {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Check cache first — cached batches are bucket-agnostic since
|
||||
// they're already materialized in memory.
|
||||
// Check cache first
|
||||
if let Some(cached) = self.cache.get(&dataset.name).await {
|
||||
// Load any delta files and merge
|
||||
let delta_batches = delta::load_deltas(&self.store, &dataset.name).await.unwrap_or_default();
|
||||
|
||||
let mut all_batches = cached.batches;
|
||||
@@ -181,15 +106,10 @@ impl QueryEngine {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Build ListingTable URLs that include the per-object bucket
|
||||
// scheme. DataFusion routes each scan to the right store
|
||||
// automatically based on the URL.
|
||||
// Fall back to Parquet file
|
||||
let opts = ListingOptions::new(Arc::new(ParquetFormat::default()));
|
||||
let table_paths: Vec<ListingTableUrl> = dataset.objects.iter()
|
||||
.filter_map(|o| {
|
||||
let bucket = if o.bucket.is_empty() { "primary" } else { &o.bucket };
|
||||
ListingTableUrl::parse(&bucket_object_url(bucket, &o.key)).ok()
|
||||
})
|
||||
.filter_map(|o| ListingTableUrl::parse(&format!("{STORE_SCHEME}://data/{}", o.key)).ok())
|
||||
.collect();
|
||||
|
||||
if table_paths.is_empty() {
|
||||
@@ -204,123 +124,8 @@ impl QueryEngine {
|
||||
let table = ListingTable::try_new(config)
|
||||
.map_err(|e| format!("table creation failed for {}: {e}", dataset.name))?;
|
||||
|
||||
// Decide the registration name: if this dataset has any
|
||||
// tombstones, the raw table gets an internal name and the
|
||||
// public name becomes a filtered view.
|
||||
let tombstone_entry = tombstones_by_dataset.get(&dataset.name);
|
||||
let register_name = if tombstone_entry.is_some() {
|
||||
format!("__raw__{}", dataset.name)
|
||||
} else {
|
||||
dataset.name.clone()
|
||||
};
|
||||
|
||||
if let Err(e) = ctx.register_table(register_name.as_str(), Arc::new(table)) {
|
||||
let msg = e.to_string();
|
||||
if msg.contains("already exists") {
|
||||
tracing::debug!("skip duplicate manifest registration: {}", register_name);
|
||||
continue;
|
||||
} else {
|
||||
return Err(format!("table registration failed for {}: {}", register_name, msg));
|
||||
}
|
||||
}
|
||||
|
||||
// If there are tombstones, register the public name as a
|
||||
// filtered view that excludes tombstoned row_key_values.
|
||||
if let Some((key_col, values)) = tombstone_entry {
|
||||
// Build WHERE NOT IN (...) — quote values to be SQL-safe.
|
||||
// For string keys this is a literal list; for integer keys
|
||||
// CAST(col AS VARCHAR) makes the comparison unambiguous.
|
||||
let quoted: Vec<String> = values.iter()
|
||||
.map(|v| format!("'{}'", v.replace('\'', "''")))
|
||||
.collect();
|
||||
let sql = format!(
|
||||
"SELECT * FROM \"{}\" WHERE CAST(\"{}\" AS VARCHAR) NOT IN ({})",
|
||||
register_name, key_col, quoted.join(", "),
|
||||
);
|
||||
tracing::debug!(
|
||||
"tombstone filter for '{}': {} row_keys excluded",
|
||||
dataset.name, values.len(),
|
||||
);
|
||||
match ctx.sql(&sql).await {
|
||||
Ok(df) => {
|
||||
if let Err(e) = ctx.register_table(dataset.name.as_str(), df.into_view()) {
|
||||
let msg = e.to_string();
|
||||
if msg.contains("already exists") {
|
||||
tracing::debug!("skip duplicate tombstone view: {}", dataset.name);
|
||||
} else {
|
||||
tracing::warn!("tombstone view registration failed for {}: {msg}", dataset.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!("tombstone view SQL failed for '{}': {e}", dataset.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Phase D: register every AiView as a DataFusion view that wraps
|
||||
// the base table with the safe projection + filter + redactions.
|
||||
for view in self.registry.list_views().await {
|
||||
// Build the SELECT clause with column whitelist and any
|
||||
// redaction expressions. Each column is either the raw column
|
||||
// name or an expression aliased back to the column name so
|
||||
// downstream queries see the same shape.
|
||||
let select_cols: Vec<String> = view.columns.iter().map(|col| {
|
||||
match view.column_redactions.get(col) {
|
||||
None => format!("\"{}\"", col),
|
||||
Some(shared::types::Redaction::Null) => {
|
||||
format!("CAST(NULL AS VARCHAR) AS \"{}\"", col)
|
||||
}
|
||||
Some(shared::types::Redaction::Hash) => {
|
||||
// DataFusion has digest('sha256', value) — but column
|
||||
// type might not be string. Cast first to be safe.
|
||||
format!("digest(CAST(\"{}\" AS VARCHAR), 'sha256') AS \"{}\"", col, col)
|
||||
}
|
||||
Some(shared::types::Redaction::Mask { keep_prefix, keep_suffix }) => {
|
||||
// Keep first N + last M chars, replace middle with stars.
|
||||
// CASE NULL through with COALESCE for null-safe behavior.
|
||||
format!(
|
||||
"CASE WHEN \"{c}\" IS NULL THEN NULL ELSE \
|
||||
concat(\
|
||||
substr(CAST(\"{c}\" AS VARCHAR), 1, {p}), \
|
||||
repeat('*', greatest(0, length(CAST(\"{c}\" AS VARCHAR)) - {p} - {s})), \
|
||||
substr(CAST(\"{c}\" AS VARCHAR), greatest(1, length(CAST(\"{c}\" AS VARCHAR)) - {s} + 1))\
|
||||
) END AS \"{c}\"",
|
||||
c = col, p = keep_prefix, s = keep_suffix,
|
||||
)
|
||||
}
|
||||
}
|
||||
}).collect();
|
||||
|
||||
let where_clause = view.row_filter
|
||||
.as_deref()
|
||||
.map(|f| format!(" WHERE {}", f))
|
||||
.unwrap_or_default();
|
||||
let view_sql = format!(
|
||||
"SELECT {} FROM \"{}\"{}",
|
||||
select_cols.join(", "),
|
||||
view.base_dataset,
|
||||
where_clause,
|
||||
);
|
||||
|
||||
tracing::debug!("registering AiView '{}': {}", view.name, view_sql);
|
||||
|
||||
let df = match ctx.sql(&view_sql).await {
|
||||
Ok(df) => df,
|
||||
Err(e) => {
|
||||
tracing::warn!("view '{}' SQL invalid, skipping: {e}", view.name);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if let Err(e) = ctx.register_table(view.name.as_str(), df.into_view()) {
|
||||
let msg = e.to_string();
|
||||
if msg.contains("already exists") {
|
||||
tracing::debug!("skip duplicate view registration: {}", view.name);
|
||||
} else {
|
||||
tracing::warn!("view '{}' registration failed: {e}", view.name);
|
||||
}
|
||||
}
|
||||
ctx.register_table(&dataset.name, Arc::new(table))
|
||||
.map_err(|e| format!("table registration failed for {}: {e}", dataset.name))?;
|
||||
}
|
||||
|
||||
Ok(ctx)
|
||||
|
||||
+21
-159
@@ -3,15 +3,10 @@
|
||||
/// we write small delta files. At query time, deltas are merged with the base.
|
||||
/// Periodic compaction merges deltas into the base file.
|
||||
|
||||
use arrow::array::{Array, BooleanArray, Int32Array, Int64Array, RecordBatch, StringArray};
|
||||
use arrow::compute::filter_record_batch;
|
||||
use arrow::array::RecordBatch;
|
||||
use arrow::datatypes::SchemaRef;
|
||||
use bytes::Bytes;
|
||||
use object_store::ObjectStore;
|
||||
use parquet::arrow::ArrowWriter;
|
||||
use parquet::basic::Compression;
|
||||
use parquet::file::properties::WriterProperties;
|
||||
use shared::types::Tombstone;
|
||||
use std::collections::HashSet;
|
||||
use std::sync::Arc;
|
||||
|
||||
use shared::arrow_helpers::{parquet_to_record_batches, record_batch_to_parquet};
|
||||
@@ -63,207 +58,74 @@ pub async fn load_deltas(
|
||||
}
|
||||
|
||||
/// Compact: merge base Parquet + all deltas into a single new base file.
|
||||
/// Optionally filters out tombstoned rows — this is the physical-deletion
|
||||
/// half of Phase E (the query-time filter handles the "immediate hide",
|
||||
/// compaction handles the "actually remove from disk" requirement).
|
||||
///
|
||||
/// Tombstones must share one `row_key_column` (enforced at tombstone write).
|
||||
/// If non-empty, every row in the merged output whose row_key_column value
|
||||
/// matches a tombstone is dropped before the new base is written.
|
||||
/// Optionally deduplicates by a primary key column.
|
||||
pub async fn compact(
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
dataset_name: &str,
|
||||
base_key: &str,
|
||||
primary_key_col: Option<&str>,
|
||||
tombstones: &[Tombstone],
|
||||
) -> Result<CompactResult, String> {
|
||||
// Load base
|
||||
let base_data = ops::get(store, base_key).await?;
|
||||
let (_schema, mut base_batches) = parquet_to_record_batches(&base_data)?;
|
||||
let (schema, mut base_batches) = parquet_to_record_batches(&base_data)?;
|
||||
|
||||
// Load deltas
|
||||
let delta_batches = load_deltas(store, dataset_name).await?;
|
||||
let delta_count = delta_batches.len();
|
||||
|
||||
let has_tombstones = !tombstones.is_empty();
|
||||
let nothing_to_do = delta_batches.is_empty() && !has_tombstones;
|
||||
if nothing_to_do {
|
||||
if delta_batches.is_empty() {
|
||||
return Ok(CompactResult {
|
||||
base_rows: base_batches.iter().map(|b| b.num_rows()).sum(),
|
||||
delta_rows: 0,
|
||||
final_rows: base_batches.iter().map(|b| b.num_rows()).sum(),
|
||||
deltas_merged: 0,
|
||||
tombstones_applied: 0,
|
||||
rows_dropped_by_tombstones: 0,
|
||||
});
|
||||
}
|
||||
|
||||
base_batches.extend(delta_batches);
|
||||
let pre_filter_rows: usize = base_batches.iter().map(|b| b.num_rows()).sum();
|
||||
|
||||
let base_rows: usize = base_batches.iter().map(|b| b.num_rows()).sum();
|
||||
|
||||
// If primary key specified, deduplicate (keep last occurrence)
|
||||
let merged_batches = if let Some(_pk) = primary_key_col {
|
||||
// Current simplification — full dedup requires sort by PK; leave
|
||||
// that work for a follow-up. Tombstone filter still runs below.
|
||||
let final_batches = if let Some(_pk) = primary_key_col {
|
||||
// For now, just concatenate. Full dedup requires sorting by PK
|
||||
// and keeping the last row per key — this is a simplification.
|
||||
// TODO: implement proper merge with dedup
|
||||
base_batches
|
||||
} else {
|
||||
base_batches
|
||||
};
|
||||
|
||||
// Tombstone filter: drop rows whose row_key matches.
|
||||
let (final_batches, dropped) = if has_tombstones {
|
||||
apply_tombstone_filter(merged_batches, tombstones)?
|
||||
} else {
|
||||
(merged_batches, 0)
|
||||
};
|
||||
|
||||
let final_rows: usize = final_batches.iter().map(|b| b.num_rows()).sum();
|
||||
|
||||
// Write merged base atomically: build ONE Parquet file (single
|
||||
// footer!) via ArrowWriter, stage it under a temporary key, read
|
||||
// back to verify row count matches what we intended, THEN swap
|
||||
// into base_key. If anything goes wrong between build and swap,
|
||||
// the original base_key is untouched.
|
||||
//
|
||||
// Why this matters: concatenating N record_batch_to_parquet()
|
||||
// outputs produces N independent Parquet files glued together —
|
||||
// readers see only the first footer. That silently corrupted the
|
||||
// candidates table before this code was fixed.
|
||||
let schema_ref = final_batches
|
||||
.first()
|
||||
.map(|b| b.schema())
|
||||
.ok_or_else(|| "no batches to write".to_string())?;
|
||||
|
||||
// Use Snappy to match the compression of files written by ingest —
|
||||
// otherwise compaction would inflate file size 3× on each rewrite.
|
||||
let props = WriterProperties::builder()
|
||||
.set_compression(Compression::SNAPPY)
|
||||
.build();
|
||||
|
||||
let mut buf: Vec<u8> = Vec::with_capacity(16 * 1024 * 1024);
|
||||
{
|
||||
let mut writer = ArrowWriter::try_new(&mut buf, schema_ref.clone(), Some(props))
|
||||
.map_err(|e| format!("ArrowWriter init: {e}"))?;
|
||||
for batch in &final_batches {
|
||||
writer.write(batch).map_err(|e| format!("write batch: {e}"))?;
|
||||
}
|
||||
writer.close().map_err(|e| format!("close writer: {e}"))?;
|
||||
// Write merged base
|
||||
let mut merged_parquet = Vec::new();
|
||||
for batch in &final_batches {
|
||||
let pq = record_batch_to_parquet(batch)?;
|
||||
merged_parquet.extend_from_slice(&pq);
|
||||
}
|
||||
ops::put(store, base_key, Bytes::from(merged_parquet)).await?;
|
||||
|
||||
// Verify before committing — if the written bytes don't parse as
|
||||
// the row count we expect, DON'T overwrite the base file.
|
||||
let (_, verify_batches) = parquet_to_record_batches(&buf)
|
||||
.map_err(|e| format!("verify parse: {e}"))?;
|
||||
let verify_rows: usize = verify_batches.iter().map(|b| b.num_rows()).sum();
|
||||
if verify_rows != final_rows {
|
||||
return Err(format!(
|
||||
"compact verification failed: wrote {} bytes that parse as {} rows (expected {}); base_key '{}' untouched",
|
||||
buf.len(), verify_rows, final_rows, base_key,
|
||||
));
|
||||
}
|
||||
|
||||
// Stage under a temp key first, then promote via a second put
|
||||
// over base_key. Object stores don't have atomic rename across
|
||||
// arbitrary backends; this is "write twice + delete temp" which
|
||||
// is the widely-adopted pattern.
|
||||
let temp_key = format!("{base_key}.compact-{}.tmp", chrono::Utc::now().timestamp_millis());
|
||||
ops::put(store, &temp_key, Bytes::from(buf.clone())).await?;
|
||||
ops::put(store, base_key, Bytes::from(buf)).await?;
|
||||
let _ = ops::delete(store, &temp_key).await;
|
||||
|
||||
// Only now that base is durably updated, delete delta files.
|
||||
// Delete delta files
|
||||
let delta_keys = list_deltas(store, dataset_name).await?;
|
||||
for key in &delta_keys {
|
||||
let _ = ops::delete(store, key).await;
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"compacted '{}': {} deltas merged, {} tombstones applied ({} rows dropped), {} → {} rows",
|
||||
dataset_name, delta_count, tombstones.len(), dropped, pre_filter_rows, final_rows,
|
||||
);
|
||||
tracing::info!("compacted '{}': {} deltas merged, {} → {} rows", dataset_name, delta_count, base_rows, final_rows);
|
||||
|
||||
Ok(CompactResult {
|
||||
base_rows: pre_filter_rows - delta_count, // rough base-before-deltas
|
||||
delta_rows: delta_count,
|
||||
base_rows,
|
||||
delta_rows: final_rows - base_rows + delta_count, // approximate
|
||||
final_rows,
|
||||
deltas_merged: delta_count,
|
||||
tombstones_applied: tombstones.len(),
|
||||
rows_dropped_by_tombstones: dropped,
|
||||
})
|
||||
}
|
||||
|
||||
/// Filter every batch in `batches` to exclude rows whose `row_key_column`
|
||||
/// value appears in the tombstone set. All tombstones must share one
|
||||
/// row_key_column — enforced at tombstone write time.
|
||||
fn apply_tombstone_filter(
|
||||
batches: Vec<RecordBatch>,
|
||||
tombstones: &[Tombstone],
|
||||
) -> Result<(Vec<RecordBatch>, usize), String> {
|
||||
let col_name = &tombstones[0].row_key_column;
|
||||
let bad_values: HashSet<String> = tombstones
|
||||
.iter()
|
||||
.map(|t| t.row_key_value.clone())
|
||||
.collect();
|
||||
|
||||
let mut out = Vec::with_capacity(batches.len());
|
||||
let mut total_dropped = 0usize;
|
||||
|
||||
for batch in batches {
|
||||
let col_idx = match batch.schema().index_of(col_name) {
|
||||
Ok(i) => i,
|
||||
Err(_) => {
|
||||
// Column not in this batch — nothing to drop, pass through.
|
||||
out.push(batch);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let col = batch.column(col_idx);
|
||||
|
||||
// Build a "keep" mask: true where row is NOT tombstoned.
|
||||
let mask: Vec<bool> = if let Some(s) = col.as_any().downcast_ref::<StringArray>() {
|
||||
(0..s.len()).map(|i| {
|
||||
s.is_null(i) || !bad_values.contains(s.value(i))
|
||||
}).collect()
|
||||
} else if let Some(a) = col.as_any().downcast_ref::<Int32Array>() {
|
||||
(0..a.len()).map(|i| {
|
||||
a.is_null(i) || !bad_values.contains(&a.value(i).to_string())
|
||||
}).collect()
|
||||
} else if let Some(a) = col.as_any().downcast_ref::<Int64Array>() {
|
||||
(0..a.len()).map(|i| {
|
||||
a.is_null(i) || !bad_values.contains(&a.value(i).to_string())
|
||||
}).collect()
|
||||
} else {
|
||||
// Key column of an unsupported type — log and pass through
|
||||
// rather than panic. Operator has to handle this manually.
|
||||
tracing::warn!(
|
||||
"tombstone filter: column '{}' has unsupported type {:?}; passing rows through",
|
||||
col_name, col.data_type(),
|
||||
);
|
||||
out.push(batch);
|
||||
continue;
|
||||
};
|
||||
|
||||
let kept = mask.iter().filter(|b| **b).count();
|
||||
let dropped = mask.len() - kept;
|
||||
total_dropped += dropped;
|
||||
if dropped == 0 {
|
||||
out.push(batch);
|
||||
continue;
|
||||
}
|
||||
let bool_arr = BooleanArray::from(mask);
|
||||
let filtered = filter_record_batch(&batch, &bool_arr)
|
||||
.map_err(|e| format!("filter batch: {e}"))?;
|
||||
out.push(filtered);
|
||||
}
|
||||
Ok((out, total_dropped))
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct CompactResult {
|
||||
pub base_rows: usize,
|
||||
pub delta_rows: usize,
|
||||
pub final_rows: usize,
|
||||
pub deltas_merged: usize,
|
||||
pub tombstones_applied: usize,
|
||||
pub rows_dropped_by_tombstones: usize,
|
||||
}
|
||||
|
||||
@@ -184,31 +184,13 @@ async fn compact_dataset(
|
||||
State(state): State<QueryState>,
|
||||
Json(req): Json<CompactRequest>,
|
||||
) -> impl IntoResponse {
|
||||
// Phase E: pull tombstones for this dataset and let compact physically
|
||||
// drop those rows. After a successful rewrite, clear the tombstone log
|
||||
// — the rows are gone from disk, the tombstones have done their job.
|
||||
let tombstones = state
|
||||
.engine
|
||||
.catalog()
|
||||
.list_tombstones(&req.dataset)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
|
||||
match delta::compact(
|
||||
state.engine.store(),
|
||||
&req.dataset,
|
||||
&req.base_key,
|
||||
req.primary_key.as_deref(),
|
||||
&tombstones,
|
||||
).await {
|
||||
Ok(result) => {
|
||||
if result.rows_dropped_by_tombstones > 0 {
|
||||
if let Err(e) = state.engine.catalog().tombstones().clear(&req.dataset).await {
|
||||
tracing::warn!("post-compact tombstone clear failed: {e}");
|
||||
}
|
||||
}
|
||||
Ok(Json(result))
|
||||
}
|
||||
Ok(result) => Ok(Json(result)),
|
||||
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,5 +15,3 @@ bytes = { workspace = true }
|
||||
sha2 = { workspace = true }
|
||||
toml = { workspace = true }
|
||||
tracing = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
async-trait = "0.1"
|
||||
|
||||
@@ -16,46 +16,8 @@ pub struct Config {
|
||||
pub auth: AuthConfig,
|
||||
#[serde(default)]
|
||||
pub observability: ObservabilityConfig,
|
||||
#[serde(default)]
|
||||
pub agent: AgentSettings,
|
||||
}
|
||||
|
||||
/// Phase 16.2 — background autotune agent settings.
|
||||
///
|
||||
/// Duplicated from `vectord::agent::AgentConfig` because `shared` can't
|
||||
/// depend on `vectord` (vectord already depends on shared). The gateway
|
||||
/// copies these into the vectord config at startup.
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct AgentSettings {
|
||||
#[serde(default)]
|
||||
pub enabled: bool,
|
||||
#[serde(default = "default_cycle_interval_secs")]
|
||||
pub cycle_interval_secs: u64,
|
||||
#[serde(default = "default_cooldown_secs")]
|
||||
pub cooldown_between_trials_secs: u64,
|
||||
#[serde(default = "default_min_recall")]
|
||||
pub min_recall: f32,
|
||||
#[serde(default = "default_max_trials_per_hour")]
|
||||
pub max_trials_per_hour: u32,
|
||||
}
|
||||
|
||||
impl Default for AgentSettings {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
cycle_interval_secs: default_cycle_interval_secs(),
|
||||
cooldown_between_trials_secs: default_cooldown_secs(),
|
||||
min_recall: default_min_recall(),
|
||||
max_trials_per_hour: default_max_trials_per_hour(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn default_cycle_interval_secs() -> u64 { 60 }
|
||||
fn default_cooldown_secs() -> u64 { 30 }
|
||||
fn default_min_recall() -> f32 { 0.9 }
|
||||
fn default_max_trials_per_hour() -> u32 { 30 }
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct GatewayConfig {
|
||||
#[serde(default = "default_host")]
|
||||
@@ -66,39 +28,8 @@ pub struct GatewayConfig {
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct StorageConfig {
|
||||
/// Legacy single-backend root. If `buckets` is empty, this is used to
|
||||
/// create an implicit `primary` bucket at this path — preserves the
|
||||
/// pre-federation config shape.
|
||||
#[serde(default = "default_storage_root")]
|
||||
pub root: String,
|
||||
|
||||
/// Where profile buckets are rooted when auto-provisioned.
|
||||
#[serde(default = "default_profile_root")]
|
||||
pub profile_root: String,
|
||||
|
||||
/// Name of the bucket used for read fallback when a target bucket is
|
||||
/// unreachable. If `None`, no fallback — reads fail hard.
|
||||
#[serde(default)]
|
||||
pub rescue_bucket: Option<String>,
|
||||
|
||||
/// Explicitly configured buckets. Empty = backward-compat single-bucket
|
||||
/// mode driven by `root`.
|
||||
#[serde(default)]
|
||||
pub buckets: Vec<BucketConfig>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct BucketConfig {
|
||||
pub name: String,
|
||||
pub backend: String, // "local" | "s3"
|
||||
/// Local filesystem root (for backend = "local")
|
||||
pub root: Option<String>,
|
||||
/// S3 bucket name (for backend = "s3")
|
||||
pub bucket: Option<String>,
|
||||
pub region: Option<String>,
|
||||
pub endpoint: Option<String>,
|
||||
/// Handle for the secrets provider — never the literal credential.
|
||||
pub secret_ref: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Default)]
|
||||
@@ -147,7 +78,6 @@ pub struct ObservabilityConfig {
|
||||
fn default_host() -> String { "0.0.0.0".to_string() }
|
||||
fn default_gateway_port() -> u16 { 3100 }
|
||||
fn default_storage_root() -> String { "./data".to_string() }
|
||||
fn default_profile_root() -> String { "./data/_profiles".to_string() }
|
||||
fn default_manifest_prefix() -> String { "_catalog/manifests".to_string() }
|
||||
fn default_sidecar_url() -> String { "http://localhost:3200".to_string() }
|
||||
fn default_embed_model() -> String { "nomic-embed-text".to_string() }
|
||||
@@ -185,19 +115,13 @@ impl Default for Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
gateway: GatewayConfig { host: default_host(), port: default_gateway_port() },
|
||||
storage: StorageConfig {
|
||||
root: default_storage_root(),
|
||||
profile_root: default_profile_root(),
|
||||
rescue_bucket: None,
|
||||
buckets: Vec::new(),
|
||||
},
|
||||
storage: StorageConfig { root: default_storage_root() },
|
||||
catalog: CatalogConfig::default(),
|
||||
query: QueryConfig::default(),
|
||||
sidecar: SidecarConfig { url: default_sidecar_url() },
|
||||
ai: AiConfig::default(),
|
||||
auth: AuthConfig::default(),
|
||||
observability: ObservabilityConfig::default(),
|
||||
agent: AgentSettings::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,4 +3,3 @@ pub mod errors;
|
||||
pub mod arrow_helpers;
|
||||
pub mod config;
|
||||
pub mod pii;
|
||||
pub mod secrets;
|
||||
|
||||
@@ -1,144 +0,0 @@
|
||||
/// Secrets layer for bucket credentials.
|
||||
///
|
||||
/// Credentials never appear in `lakehouse.toml`. Each bucket entry has an
|
||||
/// opaque `secret_ref` handle, resolved at startup by a `SecretsProvider`.
|
||||
///
|
||||
/// MVP ships `FileSecretsProvider` — reads a TOML file at a path given by
|
||||
/// `LAKEHOUSE_SECRETS` env var, defaulting to `/etc/lakehouse/secrets.toml`.
|
||||
/// The file should be root-owned, mode 0600, never in git.
|
||||
///
|
||||
/// Future providers (Vault, SOPS, OS keyring) implement the same trait.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use serde::Deserialize;
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BucketCredentials {
|
||||
pub access_key: String,
|
||||
pub secret_key: String,
|
||||
}
|
||||
|
||||
// Redact credentials when debug-printing — NEVER let these end up in logs.
|
||||
impl BucketCredentials {
|
||||
pub fn redacted(&self) -> String {
|
||||
let prefix: String = self.access_key.chars().take(4).collect();
|
||||
format!("access_key={}... (redacted)", prefix)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait SecretsProvider: Send + Sync {
|
||||
/// Resolve a handle to credentials. Returns Err if the handle is unknown.
|
||||
async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String>;
|
||||
|
||||
/// List known handles (for diagnostics only, never values).
|
||||
async fn list_handles(&self) -> Vec<String>;
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct RawSecretEntry {
|
||||
access_key: String,
|
||||
secret_key: String,
|
||||
}
|
||||
|
||||
/// File-backed secrets provider.
|
||||
pub struct FileSecretsProvider {
|
||||
path: PathBuf,
|
||||
cache: tokio::sync::RwLock<HashMap<String, BucketCredentials>>,
|
||||
}
|
||||
|
||||
impl FileSecretsProvider {
|
||||
pub fn new(path: impl Into<PathBuf>) -> Self {
|
||||
Self {
|
||||
path: path.into(),
|
||||
cache: tokio::sync::RwLock::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Default path: env var `LAKEHOUSE_SECRETS` or `/etc/lakehouse/secrets.toml`.
|
||||
pub fn from_env() -> Self {
|
||||
let path = std::env::var("LAKEHOUSE_SECRETS")
|
||||
.unwrap_or_else(|_| "/etc/lakehouse/secrets.toml".to_string());
|
||||
Self::new(path)
|
||||
}
|
||||
|
||||
/// Arc wrapper for convenient injection.
|
||||
pub fn shared(path: impl Into<PathBuf>) -> Arc<dyn SecretsProvider> {
|
||||
Arc::new(Self::new(path))
|
||||
}
|
||||
|
||||
/// Load and populate the cache. Call once at startup.
|
||||
pub async fn load(&self) -> Result<usize, String> {
|
||||
if !self.path.exists() {
|
||||
tracing::info!(
|
||||
"secrets file {} not present — running with zero credentials configured",
|
||||
self.path.display()
|
||||
);
|
||||
return Ok(0);
|
||||
}
|
||||
check_permissions(&self.path)?;
|
||||
let content = std::fs::read_to_string(&self.path)
|
||||
.map_err(|e| format!("read secrets file: {e}"))?;
|
||||
let raw: HashMap<String, RawSecretEntry> = toml::from_str(&content)
|
||||
.map_err(|e| format!("parse secrets file: {e}"))?;
|
||||
|
||||
let mut cache = self.cache.write().await;
|
||||
for (handle, entry) in raw {
|
||||
cache.insert(handle.clone(), BucketCredentials {
|
||||
access_key: entry.access_key,
|
||||
secret_key: entry.secret_key,
|
||||
});
|
||||
}
|
||||
let count = cache.len();
|
||||
tracing::info!("secrets: loaded {count} handles from {}", self.path.display());
|
||||
Ok(count)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl SecretsProvider for FileSecretsProvider {
|
||||
async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String> {
|
||||
let cache = self.cache.read().await;
|
||||
cache
|
||||
.get(handle)
|
||||
.cloned()
|
||||
.ok_or_else(|| format!("secret handle not found: {handle}"))
|
||||
}
|
||||
|
||||
async fn list_handles(&self) -> Vec<String> {
|
||||
self.cache.read().await.keys().cloned().collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Reject world-readable secrets files. Fail closed.
|
||||
fn check_permissions(path: &Path) -> Result<(), String> {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let meta = std::fs::metadata(path)
|
||||
.map_err(|e| format!("stat secrets file: {e}"))?;
|
||||
let mode = meta.permissions().mode() & 0o777;
|
||||
if mode & 0o044 != 0 {
|
||||
return Err(format!(
|
||||
"secrets file {} is group/world-readable (mode {:o}); chmod 600 to proceed",
|
||||
path.display(), mode
|
||||
));
|
||||
}
|
||||
}
|
||||
let _ = path;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A zero-handles provider for tests / purely-local setups.
|
||||
pub struct EmptySecretsProvider;
|
||||
|
||||
#[async_trait]
|
||||
impl SecretsProvider for EmptySecretsProvider {
|
||||
async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String> {
|
||||
Err(format!("no secrets provider configured (wanted handle '{handle}')"))
|
||||
}
|
||||
async fn list_handles(&self) -> Vec<String> { Vec::new() }
|
||||
}
|
||||
@@ -116,237 +116,4 @@ pub struct DatasetManifest {
|
||||
/// Row count (updated on ingest/compact)
|
||||
#[serde(default)]
|
||||
pub row_count: Option<u64>,
|
||||
|
||||
// --- Embedding freshness (Phase C) ---
|
||||
|
||||
/// When the attached vector index was last refreshed. `None` means this
|
||||
/// dataset has no vector index yet, or has never been embedded.
|
||||
#[serde(default)]
|
||||
pub last_embedded_at: Option<chrono::DateTime<chrono::Utc>>,
|
||||
|
||||
/// When data was written that hasn't yet been embedded. `Some(t)` means
|
||||
/// the vector index is out-of-date as of timestamp `t`. Cleared on refresh.
|
||||
#[serde(default)]
|
||||
pub embedding_stale_since: Option<chrono::DateTime<chrono::Utc>>,
|
||||
|
||||
/// How this dataset wants stale embeddings handled.
|
||||
#[serde(default)]
|
||||
pub embedding_refresh_policy: Option<RefreshPolicy>,
|
||||
}
|
||||
|
||||
/// Controls what happens when new data is written to a dataset with an
|
||||
/// attached vector index.
|
||||
///
|
||||
/// - `Manual` (default): data writes set `embedding_stale_since`; nothing
|
||||
/// embeds until an operator or agent calls `/vectors/refresh/{dataset}`.
|
||||
/// - `OnAppend`: ingest fires a background refresh immediately after writing.
|
||||
/// Suitable for datasets where vector freshness matters more than ingest
|
||||
/// latency.
|
||||
/// - `Scheduled(cron)`: a timer or external scheduler triggers refresh at
|
||||
/// the named cadence. The scheduler itself is not in this ADR scope —
|
||||
/// this just declares the intent so operators can see which policy a
|
||||
/// dataset expects.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(tag = "kind", rename_all = "snake_case")]
|
||||
pub enum RefreshPolicy {
|
||||
Manual,
|
||||
OnAppend,
|
||||
Scheduled { cron: String },
|
||||
}
|
||||
|
||||
impl Default for RefreshPolicy {
|
||||
fn default() -> Self { Self::Manual }
|
||||
}
|
||||
|
||||
/// AI-safe view of a dataset (Phase D).
|
||||
///
|
||||
/// A view is a named projection over a base dataset that:
|
||||
/// - whitelists the columns visible (everything else is invisible),
|
||||
/// - optionally applies a row filter (a SQL WHERE clause fragment),
|
||||
/// - optionally redacts column values (e.g. mask the last 4 of an SSN).
|
||||
///
|
||||
/// Queries that reference the view name see only the safe projection.
|
||||
/// The base table remains separately accessible by its raw name — this
|
||||
/// is a *capability surface*, not a data classification fence. Use
|
||||
/// access control (Phase 13) on top to enforce who can query which view.
|
||||
///
|
||||
/// Stored as JSON at `_catalog/views/{name}.json` in the primary bucket
|
||||
/// alongside dataset manifests. The view itself doesn't store data —
|
||||
/// queryd materializes it as a DataFusion view at SessionContext build
|
||||
/// time.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct AiView {
|
||||
pub name: String,
|
||||
/// Source dataset name. Must exist in the catalog.
|
||||
pub base_dataset: String,
|
||||
/// Whitelisted columns. Empty list is invalid (would expose nothing).
|
||||
pub columns: Vec<String>,
|
||||
/// Optional WHERE clause fragment (without the `WHERE` keyword), e.g.
|
||||
/// `status != 'blocked' AND city = 'Chicago'`. SQL injection here is
|
||||
/// the operator's problem — only trust this from authenticated admins.
|
||||
#[serde(default)]
|
||||
pub row_filter: Option<String>,
|
||||
/// Per-column redactions: `{column_name -> Redaction}`.
|
||||
#[serde(default)]
|
||||
pub column_redactions: std::collections::HashMap<String, Redaction>,
|
||||
pub created_at: chrono::DateTime<chrono::Utc>,
|
||||
#[serde(default)]
|
||||
pub created_by: String,
|
||||
#[serde(default)]
|
||||
pub description: String,
|
||||
}
|
||||
|
||||
/// Per-model configuration for HNSW builds. Declared inline so the PRD
|
||||
/// invariant 9 ("every reader gets its own profile") can carry tuning
|
||||
/// preferences without forcing a round-trip to vectord's trial system.
|
||||
/// Mirrors `vectord::trial::HnswConfig` to avoid a cross-crate dep cycle.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub struct ProfileHnswConfig {
|
||||
pub ef_construction: usize,
|
||||
pub ef_search: usize,
|
||||
#[serde(default)]
|
||||
pub seed: Option<u64>,
|
||||
}
|
||||
|
||||
impl Default for ProfileHnswConfig {
|
||||
/// Matches the production HNSW default from Phase 15 trial sweep
|
||||
/// (see docs/ADR and vectord::trial::HnswConfig::default).
|
||||
fn default() -> Self {
|
||||
Self { ef_construction: 80, ef_search: 30, seed: None }
|
||||
}
|
||||
}
|
||||
|
||||
/// A named client of the substrate — a local LLM, an AI agent, or a
|
||||
/// human role that should see a specific scoped view of the data.
|
||||
/// Profiles are the first-class "every reader gets its own profile"
|
||||
/// concept from PRD invariant 9.
|
||||
///
|
||||
/// Bound datasets can be raw catalog tables OR `AiView` names — either
|
||||
/// works at query time since both register as DataFusion tables with
|
||||
/// the same name. Mixing raw and views lets you compose access rules:
|
||||
/// bind a model to `candidates_safe` (PII-redacted view) and `placements`
|
||||
/// (raw table, no PII), and the model gets exactly what it needs.
|
||||
///
|
||||
/// Stored at `_catalog/profiles/{id}.json`. Activation (`/profile/{id}/
|
||||
/// activate`) pre-loads the embedding cache and builds HNSW with the
|
||||
/// profile's config — first search after activate is warm.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ModelProfile {
|
||||
/// Unique id — slug form (alphanumeric + `-`, `_`). Used in URLs,
|
||||
/// bucket names (`profile:{id}`), and trial journal prefixes.
|
||||
pub id: String,
|
||||
/// Ollama model tag (e.g. "qwen2.5:7b", "nomic-embed-text:latest").
|
||||
/// Used when this profile is the "generate" or "embed" caller.
|
||||
pub ollama_name: String,
|
||||
/// Free-text description surfaced to operators. Who/what is this
|
||||
/// profile for?
|
||||
#[serde(default)]
|
||||
pub description: String,
|
||||
/// Names of datasets or AiViews this profile can read. A search
|
||||
/// against an index whose `source` isn't in this list is refused.
|
||||
pub bound_datasets: Vec<String>,
|
||||
/// Per-profile HNSW config, used when activate builds the graph.
|
||||
#[serde(default)]
|
||||
pub hnsw_config: ProfileHnswConfig,
|
||||
/// Which embedding model this profile expects. Defaults to
|
||||
/// `nomic-embed-text` (the only one currently installed).
|
||||
#[serde(default = "default_embed_model")]
|
||||
pub embed_model: String,
|
||||
pub created_at: chrono::DateTime<chrono::Utc>,
|
||||
#[serde(default)]
|
||||
pub created_by: String,
|
||||
/// Federation layer 2: which bucket this profile's artifacts
|
||||
/// (trial journals, promotions for its indexes) live in. If set and
|
||||
/// not yet registered, `POST /vectors/profile/{id}/activate` auto-
|
||||
/// provisions a local bucket under `storage.profile_root` and
|
||||
/// registers it before warming indexes. `None` = share the primary
|
||||
/// bucket (pre-federation behavior).
|
||||
#[serde(default)]
|
||||
pub bucket: Option<String>,
|
||||
/// ADR-019 hybrid: which storage backend serves this profile's
|
||||
/// indexes. Default Parquet so pre-existing profiles unchanged.
|
||||
/// New profiles can opt into Lance for append-heavy / random-access
|
||||
/// / 5M+ vector workloads.
|
||||
#[serde(default)]
|
||||
pub vector_backend: VectorBackend,
|
||||
}
|
||||
|
||||
fn default_embed_model() -> String { "nomic-embed-text".to_string() }
|
||||
|
||||
/// Per-profile / per-index storage backend choice.
|
||||
///
|
||||
/// `Parquet` (default) — vectors as binary-blob in Parquet + in-RAM HNSW
|
||||
/// for ANN. Best when:
|
||||
/// - Index ≤ ~5M vectors (fits in RAM)
|
||||
/// - Search latency dominates (HNSW p50 < 1ms at 100K)
|
||||
/// - Workload is mostly read, not append
|
||||
///
|
||||
/// `Lance` — vectors as native FixedSizeList in a Lance dataset, IVF_PQ
|
||||
/// on-disk index. Best when:
|
||||
/// - Index will exceed RAM ceiling (5M+ vectors per index)
|
||||
/// - Heavy random-row access for RAG (Lance: 311us; Parquet: ~35ms full
|
||||
/// scan per fetch)
|
||||
/// - Append-heavy ingest (Lance: 0.08s for 10K rows; Parquet: full
|
||||
/// ~330MB rewrite per change)
|
||||
/// - Need to rebuild index frequently (Lance IVF_PQ: 16s; HNSW: 230s
|
||||
/// at the same scale)
|
||||
///
|
||||
/// Defaults to Parquet for backward compat with pre-ADR-019 indexes.
|
||||
/// See `docs/ADR-019-vector-storage.md` for the full scorecard.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum VectorBackend {
|
||||
Parquet,
|
||||
Lance,
|
||||
}
|
||||
|
||||
impl Default for VectorBackend {
|
||||
fn default() -> Self { VectorBackend::Parquet }
|
||||
}
|
||||
|
||||
/// Soft-delete marker (Phase E).
|
||||
///
|
||||
/// Tombstones live beside the dataset in `_catalog/tombstones/{dataset}/`
|
||||
/// as append-log JSONL. Query-time filter in queryd reads all tombstones
|
||||
/// for each dataset and wraps the base table in a DataFusion view that
|
||||
/// excludes tombstoned rows. Physical deletion happens later (compaction),
|
||||
/// so the row count immediately reflects the delete but data is still on
|
||||
/// disk until compact runs. That's deliberate — it gives a reversal
|
||||
/// window and keeps the event journal audit trail intact.
|
||||
///
|
||||
/// All tombstones for a given dataset must use the same `row_key_column`
|
||||
/// (enforced on write); otherwise the query-time filter can't be built
|
||||
/// as a single WHERE clause.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Tombstone {
|
||||
pub dataset: String,
|
||||
/// Column name that identifies the row (e.g. "candidate_id").
|
||||
pub row_key_column: String,
|
||||
/// Value of that column for the tombstoned row.
|
||||
pub row_key_value: String,
|
||||
pub deleted_at: chrono::DateTime<chrono::Utc>,
|
||||
/// Human / system actor responsible for the delete (audit).
|
||||
#[serde(default)]
|
||||
pub actor: String,
|
||||
/// Why (e.g. "GDPR request #1234", "user-requested erasure").
|
||||
#[serde(default)]
|
||||
pub reason: String,
|
||||
}
|
||||
|
||||
/// How a column's values should be transformed before being returned.
|
||||
/// `Mask` is the most common — keeps a few visible chars, replaces the
|
||||
/// rest with `*`. `Hash` returns SHA-256 of the value for join keys you
|
||||
/// want to preserve referential integrity on without exposing the value.
|
||||
/// `Null` returns NULL unconditionally.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(tag = "kind", rename_all = "snake_case")]
|
||||
pub enum Redaction {
|
||||
Null,
|
||||
Hash,
|
||||
Mask {
|
||||
/// Number of leading characters left visible.
|
||||
keep_prefix: usize,
|
||||
/// Number of trailing characters left visible.
|
||||
keep_suffix: usize,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -13,4 +13,3 @@ tracing = { workspace = true }
|
||||
object_store = { workspace = true }
|
||||
bytes = { workspace = true }
|
||||
futures = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
|
||||
@@ -1,239 +0,0 @@
|
||||
/// Write-once batched append log.
|
||||
///
|
||||
/// Problem we're fixing: the error journal and HNSW trial journal both
|
||||
/// previously did read-modify-write of their whole JSONL file on every
|
||||
/// event. That's O(N²) cumulative work and generates huge churn at scale.
|
||||
/// It's exactly the pattern llms3.com flags as the "small-file /
|
||||
/// rewrite-amplification" anti-pattern.
|
||||
///
|
||||
/// This helper implements the pattern object storage actually wants:
|
||||
///
|
||||
/// - Events accumulate in an **in-memory buffer** (reads see them immediately).
|
||||
/// - When the buffer hits a threshold, or `flush()` is called, the buffer is
|
||||
/// written **as one new object** with a timestamp-sorted key.
|
||||
/// - Existing objects are never rewritten.
|
||||
/// - Reads enumerate all batch files, sort by key, and concat in order.
|
||||
/// - An explicit `compact()` reads every batch file, writes one consolidated
|
||||
/// file, and deletes the originals — the LSM-tree compaction idea applied
|
||||
/// to small JSONL events.
|
||||
///
|
||||
/// Storage layout:
|
||||
/// ```
|
||||
/// {prefix}/
|
||||
/// batch_0001776319628000123.jsonl
|
||||
/// batch_0001776319745987654.jsonl
|
||||
/// batch_compacted_00001776319800000000.jsonl (after compact())
|
||||
/// ```
|
||||
/// Key format: the zero-padded epoch microsecond of the write, so lexical
|
||||
/// sort == chronological sort.
|
||||
|
||||
use bytes::Bytes;
|
||||
use chrono::Utc;
|
||||
use object_store::ObjectStore;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
use crate::ops;
|
||||
|
||||
const DEFAULT_FLUSH_THRESHOLD: usize = 32;
|
||||
|
||||
pub struct AppendLog {
|
||||
store: Arc<dyn ObjectStore>,
|
||||
prefix: String,
|
||||
buffer: Mutex<Vec<Vec<u8>>>,
|
||||
flush_threshold: usize,
|
||||
}
|
||||
|
||||
impl AppendLog {
|
||||
/// Create a new append log rooted at `prefix` in the given object store.
|
||||
/// Events auto-flush when buffer reaches `flush_threshold` (default 32).
|
||||
pub fn new(store: Arc<dyn ObjectStore>, prefix: impl Into<String>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
prefix: prefix.into(),
|
||||
buffer: Mutex::new(Vec::new()),
|
||||
flush_threshold: DEFAULT_FLUSH_THRESHOLD,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_flush_threshold(mut self, threshold: usize) -> Self {
|
||||
self.flush_threshold = threshold.max(1);
|
||||
self
|
||||
}
|
||||
|
||||
/// Add an event. The returned future completes either immediately
|
||||
/// (buffered) or after a flush, depending on whether the buffer hit the
|
||||
/// threshold. Callers don't need to care either way.
|
||||
pub async fn append(&self, line: Vec<u8>) -> Result<(), String> {
|
||||
let should_flush = {
|
||||
let mut buf = self.buffer.lock().await;
|
||||
buf.push(line);
|
||||
buf.len() >= self.flush_threshold
|
||||
};
|
||||
if should_flush {
|
||||
self.flush().await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Force-flush the in-memory buffer to object storage as a single new
|
||||
/// batch file. Safe to call anytime (idempotent no-op when buffer empty).
|
||||
pub async fn flush(&self) -> Result<(), String> {
|
||||
let batch = {
|
||||
let mut buf = self.buffer.lock().await;
|
||||
if buf.is_empty() { return Ok(()); }
|
||||
std::mem::take(&mut *buf)
|
||||
};
|
||||
|
||||
let ts_us = Utc::now().timestamp_micros().max(0) as u128;
|
||||
let key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
|
||||
|
||||
let mut body = Vec::with_capacity(batch.iter().map(|b| b.len() + 1).sum());
|
||||
for line in batch {
|
||||
body.extend_from_slice(&line);
|
||||
if !body.ends_with(b"\n") { body.push(b'\n'); }
|
||||
}
|
||||
ops::put(&self.store, &key, Bytes::from(body)).await
|
||||
}
|
||||
|
||||
/// Read every event across all batch files + unflushed in-memory buffer.
|
||||
/// Events are returned in chronological order.
|
||||
pub async fn read_all(&self) -> Result<Vec<Vec<u8>>, String> {
|
||||
let mut keys = self.list_batch_keys().await?;
|
||||
keys.sort();
|
||||
|
||||
let mut out = Vec::new();
|
||||
for key in keys {
|
||||
let bytes = ops::get(&self.store, &key).await?;
|
||||
for line in bytes.split(|b| *b == b'\n') {
|
||||
if !line.is_empty() {
|
||||
out.push(line.to_vec());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Include unflushed events so callers see the latest state
|
||||
// whether or not someone ran flush() recently.
|
||||
let buf = self.buffer.lock().await;
|
||||
for line in buf.iter() {
|
||||
out.push(line.clone());
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Consolidate all current batch files into one compacted file, then
|
||||
/// delete the originals. Safe to call while appends are in flight:
|
||||
/// new batches written during compaction get a higher timestamp and
|
||||
/// survive. Fails closed — if anything goes wrong mid-delete, the
|
||||
/// compacted file coexists with originals and next read sees duplicates
|
||||
/// (which the dedup caller must handle) rather than data loss.
|
||||
pub async fn compact(&self) -> Result<CompactStats, String> {
|
||||
// Snapshot which files to compact BEFORE we write the new one.
|
||||
let mut originals = self.list_batch_keys().await?;
|
||||
originals.sort();
|
||||
|
||||
if originals.len() < 2 {
|
||||
return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
|
||||
}
|
||||
|
||||
// Gather all existing events.
|
||||
let mut events = Vec::new();
|
||||
for key in &originals {
|
||||
let bytes = ops::get(&self.store, key).await?;
|
||||
for line in bytes.split(|b| *b == b'\n') {
|
||||
if !line.is_empty() {
|
||||
events.push(line.to_vec());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let total_events = events.len();
|
||||
if total_events == 0 {
|
||||
// Clean up empty files without writing a new one.
|
||||
for key in &originals {
|
||||
let _ = ops::delete(&self.store, key).await;
|
||||
}
|
||||
return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
|
||||
}
|
||||
|
||||
// Name the compacted file with the SAME `batch_{ts}` format so it
|
||||
// sorts chronologically with future batches. Using a distinct prefix
|
||||
// ("batch_compacted_") would break lex ordering: later `batch_N`
|
||||
// files would sort BEFORE the compacted file because 'c' > digits.
|
||||
// Timestamp = now, so any appends arriving during compaction (which
|
||||
// get the current wall-clock time) sort AFTER this file.
|
||||
let ts_us = Utc::now().timestamp_micros().max(0) as u128;
|
||||
let new_key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
|
||||
|
||||
let mut body = Vec::new();
|
||||
for line in &events {
|
||||
body.extend_from_slice(line);
|
||||
body.push(b'\n');
|
||||
}
|
||||
ops::put(&self.store, &new_key, Bytes::from(body)).await?;
|
||||
|
||||
// Only delete originals once the consolidated file is persisted.
|
||||
let mut failures = 0;
|
||||
for key in &originals {
|
||||
if ops::delete(&self.store, key).await.is_err() {
|
||||
failures += 1;
|
||||
}
|
||||
}
|
||||
if failures > 0 {
|
||||
tracing::warn!(
|
||||
"compact '{}': {} original files failed to delete — consolidated file {} has the data",
|
||||
self.prefix, failures, new_key,
|
||||
);
|
||||
}
|
||||
|
||||
Ok(CompactStats {
|
||||
merged_files: originals.len(),
|
||||
events: total_events,
|
||||
new_key: Some(new_key),
|
||||
})
|
||||
}
|
||||
|
||||
/// How many batch files exist on disk right now.
|
||||
pub async fn file_count(&self) -> Result<usize, String> {
|
||||
Ok(self.list_batch_keys().await?.len())
|
||||
}
|
||||
|
||||
async fn list_batch_keys(&self) -> Result<Vec<String>, String> {
|
||||
let prefix_with_slash = format!("{}/", self.prefix);
|
||||
// list the prefix then filter for keys that match our naming scheme;
|
||||
// unrelated files at the same prefix won't be touched.
|
||||
let raw = ops::list(&self.store, Some(&prefix_with_slash)).await?;
|
||||
Ok(raw
|
||||
.into_iter()
|
||||
.filter(|k| {
|
||||
let basename = k.rsplit('/').next().unwrap_or(k);
|
||||
basename.starts_with("batch_") && basename.ends_with(".jsonl")
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct CompactStats {
|
||||
pub merged_files: usize,
|
||||
pub events: usize,
|
||||
pub new_key: Option<String>,
|
||||
}
|
||||
|
||||
// Proactively flush on drop, best-effort.
|
||||
// We can't `.await` in Drop; we spawn the flush on the tokio runtime if one
|
||||
// is available. If the runtime is already shutting down the flush is lost —
|
||||
// acceptable for these observability journals, which are hints not records.
|
||||
impl Drop for AppendLog {
|
||||
fn drop(&mut self) {
|
||||
let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
|
||||
if buf_len == 0 { return; }
|
||||
// Can't spawn from sync Drop on every runtime shape; log + move on.
|
||||
tracing::debug!(
|
||||
"append_log '{}' dropping with {} unflushed events",
|
||||
self.prefix, buf_len,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,200 +0,0 @@
|
||||
/// Bucket operation error journal.
|
||||
///
|
||||
/// Every bucket op failure and every rescue fallback lands here. Goal:
|
||||
/// answering "is anything broken?" with one HTTP call.
|
||||
///
|
||||
/// Storage: batched write-once files under `_errors/bucket_errors/` in the
|
||||
/// primary bucket. Uses the shared `AppendLog` helper so we never rewrite
|
||||
/// existing files — see `append_log.rs` for the full pattern rationale.
|
||||
///
|
||||
/// In-memory ring buffer holds the last N events for fast response; on
|
||||
/// startup `load_recent` hydrates it from all batch files.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::append_log::{AppendLog, CompactStats};
|
||||
|
||||
const JOURNAL_PREFIX: &str = "_errors/bucket_errors";
|
||||
const RING_CAPACITY: usize = 2000;
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum BucketOp {
|
||||
Read,
|
||||
Write,
|
||||
Delete,
|
||||
List,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct BucketErrorEvent {
|
||||
pub ts: DateTime<Utc>,
|
||||
pub op: BucketOp,
|
||||
pub target: String,
|
||||
pub key: String,
|
||||
pub error: String,
|
||||
#[serde(default)]
|
||||
pub rescued: bool,
|
||||
}
|
||||
|
||||
impl BucketErrorEvent {
|
||||
pub fn new_read(target: &str, key: &str, error: &str) -> Self {
|
||||
Self { ts: Utc::now(), op: BucketOp::Read, target: target.into(), key: key.into(), error: error.into(), rescued: false }
|
||||
}
|
||||
pub fn new_write(target: &str, key: &str, error: &str) -> Self {
|
||||
Self { ts: Utc::now(), op: BucketOp::Write, target: target.into(), key: key.into(), error: error.into(), rescued: false }
|
||||
}
|
||||
pub fn new_delete(target: &str, key: &str, error: &str) -> Self {
|
||||
Self { ts: Utc::now(), op: BucketOp::Delete, target: target.into(), key: key.into(), error: error.into(), rescued: false }
|
||||
}
|
||||
pub fn new_list(target: &str, prefix: &str, error: &str) -> Self {
|
||||
Self { ts: Utc::now(), op: BucketOp::List, target: target.into(), key: prefix.into(), error: error.into(), rescued: false }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct ErrorJournal {
|
||||
log: Arc<AppendLog>,
|
||||
ring: Arc<RwLock<VecDeque<BucketErrorEvent>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct HealthReport {
|
||||
pub period_minutes: i64,
|
||||
pub total_errors: usize,
|
||||
pub per_bucket: std::collections::HashMap<String, usize>,
|
||||
pub unhealthy_buckets: Vec<String>,
|
||||
}
|
||||
|
||||
impl ErrorJournal {
|
||||
pub fn new(store: Arc<dyn ObjectStore>) -> Self {
|
||||
// Keep flush threshold lowish — operators checking /storage/errors
|
||||
// after a recent incident want to see fresh rows even without an
|
||||
// explicit flush.
|
||||
let log = Arc::new(
|
||||
AppendLog::new(store, JOURNAL_PREFIX).with_flush_threshold(8),
|
||||
);
|
||||
Self {
|
||||
log,
|
||||
ring: Arc::new(RwLock::new(VecDeque::with_capacity(RING_CAPACITY))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Hydrate the ring buffer from existing batch files. Call once at
|
||||
/// startup. Tolerates malformed lines (skipped with a warning) and
|
||||
/// missing files (returns 0).
|
||||
pub async fn load_recent(&self) -> Result<usize, String> {
|
||||
let lines = self.log.read_all().await.unwrap_or_default();
|
||||
let mut ring = self.ring.write().await;
|
||||
for line in lines {
|
||||
match serde_json::from_slice::<BucketErrorEvent>(&line) {
|
||||
Ok(ev) => {
|
||||
if ring.len() >= RING_CAPACITY { ring.pop_front(); }
|
||||
ring.push_back(ev);
|
||||
}
|
||||
Err(e) => tracing::warn!("error journal: skip malformed line ({e})"),
|
||||
}
|
||||
}
|
||||
Ok(ring.len())
|
||||
}
|
||||
|
||||
/// Append an event. In-memory ring updated immediately; persistence
|
||||
/// happens in batches via the underlying AppendLog.
|
||||
pub async fn append(&self, event: BucketErrorEvent) {
|
||||
{
|
||||
let mut ring = self.ring.write().await;
|
||||
if ring.len() >= RING_CAPACITY { ring.pop_front(); }
|
||||
ring.push_back(event.clone());
|
||||
}
|
||||
|
||||
match serde_json::to_vec(&event) {
|
||||
Ok(line) => {
|
||||
if let Err(e) = self.log.append(line).await {
|
||||
tracing::error!("error journal persist failed: {e}");
|
||||
}
|
||||
}
|
||||
Err(e) => tracing::error!("error journal serialize failed: {e}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Mark the most recent matching in-memory event as rescued.
|
||||
/// Only updates the ring buffer — the JSONL line already persisted
|
||||
/// records the failure fact; rescue status travels in response headers.
|
||||
pub async fn mark_rescued_last(&self, target: &str, key: &str) {
|
||||
let mut ring = self.ring.write().await;
|
||||
for ev in ring.iter_mut().rev() {
|
||||
if matches!(ev.op, BucketOp::Read) && ev.target == target && ev.key == key && !ev.rescued {
|
||||
ev.rescued = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn recent(&self, limit: usize) -> Vec<BucketErrorEvent> {
|
||||
let ring = self.ring.read().await;
|
||||
let start = ring.len().saturating_sub(limit);
|
||||
ring.iter().skip(start).cloned().collect()
|
||||
}
|
||||
|
||||
pub async fn filter(
|
||||
&self,
|
||||
bucket: Option<&str>,
|
||||
since: Option<DateTime<Utc>>,
|
||||
limit: usize,
|
||||
) -> Vec<BucketErrorEvent> {
|
||||
let ring = self.ring.read().await;
|
||||
ring.iter()
|
||||
.rev()
|
||||
.filter(|ev| bucket.map_or(true, |b| ev.target == b))
|
||||
.filter(|ev| since.map_or(true, |s| ev.ts >= s))
|
||||
.take(limit)
|
||||
.cloned()
|
||||
.collect::<Vec<_>>()
|
||||
.into_iter()
|
||||
.rev()
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Summarize errors in the last N minutes.
|
||||
pub async fn health(&self, period_minutes: i64) -> HealthReport {
|
||||
use std::collections::HashMap;
|
||||
let cutoff = Utc::now() - chrono::Duration::minutes(period_minutes);
|
||||
let ring = self.ring.read().await;
|
||||
let recent: Vec<_> = ring.iter().filter(|ev| ev.ts >= cutoff).collect();
|
||||
let mut per_bucket: HashMap<String, usize> = HashMap::new();
|
||||
for ev in &recent {
|
||||
*per_bucket.entry(ev.target.clone()).or_insert(0) += 1;
|
||||
}
|
||||
let unhealthy_buckets: Vec<String> = per_bucket
|
||||
.iter()
|
||||
.filter(|(_, c)| **c >= 3)
|
||||
.map(|(k, _)| k.clone())
|
||||
.collect();
|
||||
HealthReport {
|
||||
period_minutes,
|
||||
total_errors: recent.len(),
|
||||
per_bucket,
|
||||
unhealthy_buckets,
|
||||
}
|
||||
}
|
||||
|
||||
/// Force an immediate flush of buffered events to object storage.
|
||||
pub async fn flush(&self) -> Result<(), String> {
|
||||
self.log.flush().await
|
||||
}
|
||||
|
||||
/// Consolidate all batch files into one. Operator cleanup.
|
||||
pub async fn compact(&self) -> Result<CompactStats, String> {
|
||||
self.log.compact().await
|
||||
}
|
||||
|
||||
/// How many JSONL batch files currently exist.
|
||||
pub async fn file_count(&self) -> Result<usize, String> {
|
||||
self.log.file_count().await
|
||||
}
|
||||
}
|
||||
@@ -1,179 +0,0 @@
|
||||
/// HTTP surface for federation operator endpoints.
|
||||
///
|
||||
/// - `GET /buckets` — configured buckets with reachability status
|
||||
/// - `GET /errors` — recent bucket op failures (filterable)
|
||||
/// - `GET /bucket-health` — aggregated errors-per-bucket in the last 5 minutes
|
||||
///
|
||||
/// Mounted under `/storage` by the gateway. `/storage/health` is already
|
||||
/// claimed by the existing storage router for service liveness, so the
|
||||
/// aggregated bucket health lives at `/storage/bucket-health`.
|
||||
|
||||
use axum::{
|
||||
Json, Router,
|
||||
extract::{Path, Query, State},
|
||||
http::StatusCode,
|
||||
response::IntoResponse,
|
||||
routing::{delete, get, post, put},
|
||||
};
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::Deserialize;
|
||||
use shared::config::BucketConfig;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::error_journal::BucketErrorEvent;
|
||||
use crate::registry::{BucketInfo, BucketRegistry};
|
||||
|
||||
pub fn router(registry: Arc<BucketRegistry>) -> Router {
|
||||
Router::new()
|
||||
.route("/buckets", get(list_buckets).post(add_bucket))
|
||||
.route("/buckets/{name}", delete(remove_bucket))
|
||||
.route("/errors", get(list_errors))
|
||||
.route("/errors/flush", post(flush_errors))
|
||||
.route("/errors/compact", post(compact_errors))
|
||||
.route("/bucket-health", get(get_health))
|
||||
.route("/buckets/{bucket}/objects/{*key}", put(put_bucket_object))
|
||||
.route("/buckets/{bucket}/objects/{*key}", get(get_bucket_object))
|
||||
.with_state(registry)
|
||||
}
|
||||
|
||||
/// Provision + register a bucket at runtime. Body is a `BucketConfig`.
|
||||
///
|
||||
/// Federation layer 2: profile buckets (`profile:alice`) and tenant
|
||||
/// buckets can be added after startup without a service restart.
|
||||
async fn add_bucket(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Json(bc): Json<BucketConfig>,
|
||||
) -> impl IntoResponse {
|
||||
// Safety net: local backends must land somewhere under the configured
|
||||
// profile_root (for profile:*) or be otherwise explicitly rooted.
|
||||
// Refuse empty / missing root to avoid "bucket created at ./" surprises.
|
||||
if bc.backend == "local" {
|
||||
match bc.root.as_deref() {
|
||||
Some(r) if !r.trim().is_empty() => {}
|
||||
_ => return Err((StatusCode::BAD_REQUEST,
|
||||
"local bucket requires a non-empty 'root' path".to_string())),
|
||||
}
|
||||
}
|
||||
match reg.add_bucket(bc).await {
|
||||
Ok(info) => Ok((StatusCode::CREATED, Json(info))),
|
||||
Err(e) => {
|
||||
let code = if e.contains("already registered") {
|
||||
StatusCode::CONFLICT
|
||||
} else {
|
||||
StatusCode::BAD_REQUEST
|
||||
};
|
||||
Err((code, e))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Unregister a bucket. Refused for primary / rescue / unknown names.
|
||||
async fn remove_bucket(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Path(name): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
match reg.remove_bucket(&name) {
|
||||
Ok(()) => Ok((StatusCode::OK, format!("unregistered: {name}"))),
|
||||
Err(e) => {
|
||||
let code = if e.contains("cannot remove") {
|
||||
StatusCode::FORBIDDEN
|
||||
} else if e.contains("not registered") {
|
||||
StatusCode::NOT_FOUND
|
||||
} else {
|
||||
StatusCode::BAD_REQUEST
|
||||
};
|
||||
Err((code, e))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_buckets(State(reg): State<Arc<BucketRegistry>>) -> Json<Vec<BucketInfo>> {
|
||||
Json(reg.list().await)
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct ErrorQuery {
|
||||
bucket: Option<String>,
|
||||
since: Option<DateTime<Utc>>,
|
||||
#[serde(default = "default_limit")]
|
||||
limit: usize,
|
||||
}
|
||||
|
||||
fn default_limit() -> usize { 50 }
|
||||
|
||||
async fn list_errors(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Query(q): Query<ErrorQuery>,
|
||||
) -> Json<Vec<BucketErrorEvent>> {
|
||||
Json(reg.journal().filter(q.bucket.as_deref(), q.since, q.limit).await)
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct HealthQuery {
|
||||
#[serde(default = "default_period")]
|
||||
minutes: i64,
|
||||
}
|
||||
|
||||
fn default_period() -> i64 { 5 }
|
||||
|
||||
async fn get_health(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Query(q): Query<HealthQuery>,
|
||||
) -> impl IntoResponse {
|
||||
Json(reg.journal().health(q.minutes).await)
|
||||
}
|
||||
|
||||
/// Bucket-aware write. Hard-fails if the target bucket is unreachable;
|
||||
/// never falls back to rescue for writes.
|
||||
async fn put_bucket_object(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Path((bucket, key)): Path<(String, String)>,
|
||||
body: bytes::Bytes,
|
||||
) -> impl IntoResponse {
|
||||
match reg.write_smart(&bucket, &key, body).await {
|
||||
Ok(()) => (StatusCode::CREATED, format!("stored: {bucket}/{key}")).into_response(),
|
||||
Err(e) => (StatusCode::SERVICE_UNAVAILABLE, e).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Bucket-aware read. Falls through to the rescue bucket if the target
|
||||
/// is unreachable or the key is missing; emits observability headers so
|
||||
/// callers can detect the fallback.
|
||||
async fn flush_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
|
||||
match reg.journal().flush().await {
|
||||
Ok(()) => (StatusCode::OK, "flushed").into_response(),
|
||||
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn compact_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
|
||||
match reg.journal().compact().await {
|
||||
Ok(stats) => Json(stats).into_response(),
|
||||
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn get_bucket_object(
|
||||
State(reg): State<Arc<BucketRegistry>>,
|
||||
Path((bucket, key)): Path<(String, String)>,
|
||||
) -> impl IntoResponse {
|
||||
match reg.read_smart(&bucket, &key).await {
|
||||
Ok(outcome) => {
|
||||
let mut resp = outcome.data.into_response();
|
||||
if outcome.rescued {
|
||||
let headers = resp.headers_mut();
|
||||
headers.insert("x-lakehouse-rescue-used", "true".parse().unwrap());
|
||||
headers.insert(
|
||||
"x-lakehouse-original-bucket",
|
||||
outcome.original_bucket.parse().unwrap(),
|
||||
);
|
||||
headers.insert(
|
||||
"x-lakehouse-served-by",
|
||||
outcome.served_by.parse().unwrap(),
|
||||
);
|
||||
}
|
||||
resp
|
||||
}
|
||||
Err(e) => (StatusCode::NOT_FOUND, e).into_response(),
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,3 @@
|
||||
pub mod append_log;
|
||||
pub mod backend;
|
||||
pub mod error_journal;
|
||||
pub mod federation_service;
|
||||
pub mod ops;
|
||||
pub mod registry;
|
||||
pub mod service;
|
||||
|
||||
@@ -1,377 +0,0 @@
|
||||
/// Multi-backend bucket registry — the federation foundation.
|
||||
///
|
||||
/// Federation rule: every `ObjectRef` belongs to exactly one named bucket.
|
||||
/// The registry resolves bucket names to `object_store` backends, handles
|
||||
/// rescue-bucket fallback on read failure, writes every failure to the
|
||||
/// error journal, and exposes a health summary for operators.
|
||||
///
|
||||
/// Existing call sites can keep using `ops::*` with `registry.get(name)`.
|
||||
/// New bucket-aware call sites use `registry.read_smart` / `write_smart`
|
||||
/// which handle fallback + journaling automatically.
|
||||
|
||||
use object_store::ObjectStore;
|
||||
use object_store::local::LocalFileSystem;
|
||||
use serde::Serialize;
|
||||
use shared::config::{BucketConfig, StorageConfig};
|
||||
use shared::secrets::{BucketCredentials, SecretsProvider};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, RwLock};
|
||||
|
||||
use crate::error_journal::{BucketErrorEvent, ErrorJournal};
|
||||
|
||||
/// A registered bucket — the store handle + its configuration.
|
||||
pub struct BucketEntry {
|
||||
pub name: String,
|
||||
pub backend: String,
|
||||
pub store: Arc<dyn ObjectStore>,
|
||||
pub config: BucketConfig,
|
||||
}
|
||||
|
||||
/// Read outcome — may have been rescued.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ReadOutcome {
|
||||
pub data: bytes::Bytes,
|
||||
pub rescued: bool,
|
||||
pub original_bucket: String,
|
||||
pub served_by: String,
|
||||
}
|
||||
|
||||
/// Summary entry for GET /storage/buckets.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct BucketInfo {
|
||||
pub name: String,
|
||||
pub backend: String,
|
||||
pub reachable: bool,
|
||||
pub role: BucketRole,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, PartialEq)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum BucketRole {
|
||||
Primary,
|
||||
Rescue,
|
||||
Profile,
|
||||
Tenant,
|
||||
}
|
||||
|
||||
pub struct BucketRegistry {
|
||||
/// RwLock because federation layer 2 adds runtime bucket lifecycle
|
||||
/// (`POST /storage/buckets` / `DELETE /storage/buckets/{name}`).
|
||||
/// Almost all accesses are reads — writes happen at provision /
|
||||
/// deactivate time only.
|
||||
buckets: RwLock<HashMap<String, Arc<BucketEntry>>>,
|
||||
default: String,
|
||||
rescue: Option<String>,
|
||||
profile_root: String,
|
||||
/// Held so runtime `add_bucket` can resolve secret_ref handles.
|
||||
secrets: Arc<dyn SecretsProvider>,
|
||||
journal: ErrorJournal,
|
||||
}
|
||||
|
||||
impl BucketRegistry {
|
||||
/// Build the registry from storage config + secrets provider.
|
||||
/// Back-compat: if `buckets` is empty, synthesize a `primary` bucket from
|
||||
/// the legacy `root` field so pre-federation configs keep working.
|
||||
pub async fn from_config(
|
||||
cfg: &StorageConfig,
|
||||
secrets: Arc<dyn SecretsProvider>,
|
||||
) -> Result<Self, String> {
|
||||
let mut buckets: HashMap<String, Arc<BucketEntry>> = HashMap::new();
|
||||
|
||||
let bucket_configs: Vec<BucketConfig> = if cfg.buckets.is_empty() {
|
||||
vec![BucketConfig {
|
||||
name: "primary".to_string(),
|
||||
backend: "local".to_string(),
|
||||
root: Some(cfg.root.clone()),
|
||||
bucket: None,
|
||||
region: None,
|
||||
endpoint: None,
|
||||
secret_ref: None,
|
||||
}]
|
||||
} else {
|
||||
cfg.buckets.clone()
|
||||
};
|
||||
|
||||
for bc in bucket_configs {
|
||||
let store = build_store(&bc, secrets.as_ref()).await?;
|
||||
let entry = Arc::new(BucketEntry {
|
||||
name: bc.name.clone(),
|
||||
backend: bc.backend.clone(),
|
||||
store,
|
||||
config: bc.clone(),
|
||||
});
|
||||
buckets.insert(bc.name.clone(), entry);
|
||||
}
|
||||
|
||||
// Ensure `primary` always exists — it's where error journals live.
|
||||
if !buckets.contains_key("primary") {
|
||||
return Err("no bucket named 'primary' configured — required as error-journal home".into());
|
||||
}
|
||||
|
||||
// Rescue bucket is optional but, if named, must exist.
|
||||
if let Some(r) = &cfg.rescue_bucket {
|
||||
if !buckets.contains_key(r) {
|
||||
return Err(format!("rescue_bucket '{r}' not found among configured buckets"));
|
||||
}
|
||||
}
|
||||
|
||||
let journal = ErrorJournal::new(buckets.get("primary").unwrap().store.clone());
|
||||
let _ = journal.load_recent().await;
|
||||
|
||||
Ok(Self {
|
||||
buckets: RwLock::new(buckets),
|
||||
default: "primary".to_string(),
|
||||
rescue: cfg.rescue_bucket.clone(),
|
||||
profile_root: cfg.profile_root.clone(),
|
||||
secrets,
|
||||
journal,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn default_name(&self) -> &str { &self.default }
|
||||
|
||||
pub fn rescue_name(&self) -> Option<&str> { self.rescue.as_deref() }
|
||||
|
||||
pub fn profile_root(&self) -> &str { &self.profile_root }
|
||||
|
||||
pub fn journal(&self) -> &ErrorJournal { &self.journal }
|
||||
|
||||
/// Resolve a bucket name to its object store. Existing call sites use
|
||||
/// this as a drop-in replacement for the old single-store pattern.
|
||||
///
|
||||
/// Uses `std::sync::RwLock` — every caller holds the guard for
|
||||
/// microseconds (clones an Arc and drops), so there's no async-
|
||||
/// blocking concern. Never hold across an await.
|
||||
pub fn get(&self, bucket: &str) -> Result<Arc<dyn ObjectStore>, String> {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard
|
||||
.get(bucket)
|
||||
.map(|e| e.store.clone())
|
||||
.ok_or_else(|| format!("unknown bucket: {bucket}"))
|
||||
}
|
||||
|
||||
/// The default bucket's store — use for code paths that don't yet know
|
||||
/// about buckets.
|
||||
pub fn default_store(&self) -> Arc<dyn ObjectStore> {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard.get(&self.default).unwrap().store.clone()
|
||||
}
|
||||
|
||||
/// True if this bucket name is registered.
|
||||
pub fn contains(&self, bucket: &str) -> bool {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard.contains_key(bucket)
|
||||
}
|
||||
|
||||
/// List all registered buckets. Checks reachability by doing a trivial
|
||||
/// `list` with limit 1 on each.
|
||||
pub async fn list(&self) -> Vec<BucketInfo> {
|
||||
// Snapshot (name, backend, store_clone) under the lock, then probe
|
||||
// outside — probing can be slow (network), don't hold the lock.
|
||||
let snapshot: Vec<(String, String, Arc<dyn ObjectStore>)> = {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard
|
||||
.iter()
|
||||
.map(|(n, e)| (n.clone(), e.backend.clone(), e.store.clone()))
|
||||
.collect()
|
||||
};
|
||||
|
||||
let mut out = Vec::with_capacity(snapshot.len());
|
||||
for (name, backend, store) in snapshot {
|
||||
let reachable = probe(&store).await;
|
||||
let role = self.classify(&name);
|
||||
out.push(BucketInfo { name, backend, reachable, role });
|
||||
}
|
||||
out.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
out
|
||||
}
|
||||
|
||||
fn classify(&self, name: &str) -> BucketRole {
|
||||
if name == self.default { BucketRole::Primary }
|
||||
else if Some(name) == self.rescue.as_deref() { BucketRole::Rescue }
|
||||
else if name.starts_with("profile:") { BucketRole::Profile }
|
||||
else { BucketRole::Tenant }
|
||||
}
|
||||
|
||||
/// Provision + register a bucket at runtime (federation layer 2).
|
||||
/// Returns Err if the name is already registered. Local backends get
|
||||
/// their root directory created automatically.
|
||||
pub async fn add_bucket(&self, bc: BucketConfig) -> Result<BucketInfo, String> {
|
||||
if self.contains(&bc.name) {
|
||||
return Err(format!("bucket '{}' already registered", bc.name));
|
||||
}
|
||||
let store = build_store(&bc, self.secrets.as_ref()).await?;
|
||||
let reachable = probe(&store).await;
|
||||
let entry = Arc::new(BucketEntry {
|
||||
name: bc.name.clone(),
|
||||
backend: bc.backend.clone(),
|
||||
store,
|
||||
config: bc.clone(),
|
||||
});
|
||||
let role = self.classify(&bc.name);
|
||||
let info = BucketInfo {
|
||||
name: bc.name.clone(),
|
||||
backend: bc.backend.clone(),
|
||||
reachable,
|
||||
role,
|
||||
};
|
||||
{
|
||||
let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
|
||||
guard.insert(bc.name.clone(), entry);
|
||||
}
|
||||
tracing::info!("registered bucket '{}' backend={} reachable={}",
|
||||
bc.name, bc.backend, reachable);
|
||||
Ok(info)
|
||||
}
|
||||
|
||||
/// Unregister a bucket. Refuses to remove `primary` or the configured
|
||||
/// rescue bucket — those are load-bearing. Caller is responsible for
|
||||
/// any final flush of bucket-local state before calling this.
|
||||
pub fn remove_bucket(&self, name: &str) -> Result<(), String> {
|
||||
if name == self.default {
|
||||
return Err("cannot remove primary bucket".into());
|
||||
}
|
||||
if Some(name) == self.rescue.as_deref() {
|
||||
return Err(format!("cannot remove rescue bucket '{name}'"));
|
||||
}
|
||||
let removed = {
|
||||
let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
|
||||
guard.remove(name)
|
||||
};
|
||||
if removed.is_none() {
|
||||
return Err(format!("bucket '{name}' not registered"));
|
||||
}
|
||||
tracing::info!("unregistered bucket '{}'", name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read with rescue-bucket fallback. If the target bucket fails and a
|
||||
/// rescue is configured, retries against rescue. Records every failure
|
||||
/// in the error journal.
|
||||
pub async fn read_smart(&self, bucket: &str, key: &str) -> Result<ReadOutcome, String> {
|
||||
let target_store = {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard.get(bucket).map(|e| e.store.clone())
|
||||
.ok_or_else(|| format!("unknown bucket: {bucket}"))?
|
||||
};
|
||||
|
||||
match crate::ops::get(&target_store, key).await {
|
||||
Ok(data) => Ok(ReadOutcome {
|
||||
data, rescued: false,
|
||||
original_bucket: bucket.to_string(),
|
||||
served_by: bucket.to_string(),
|
||||
}),
|
||||
Err(err) => {
|
||||
// Record failure regardless of what happens next.
|
||||
self.journal.append(BucketErrorEvent::new_read(bucket, key, &err)).await;
|
||||
|
||||
// Try rescue, if any.
|
||||
if let Some(rescue_name) = &self.rescue {
|
||||
if rescue_name != bucket {
|
||||
let rescue_store = {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard.get(rescue_name).map(|e| e.store.clone())
|
||||
};
|
||||
if let Some(rescue_store) = rescue_store {
|
||||
match crate::ops::get(&rescue_store, key).await {
|
||||
Ok(data) => {
|
||||
self.journal.mark_rescued_last(bucket, key).await;
|
||||
return Ok(ReadOutcome {
|
||||
data, rescued: true,
|
||||
original_bucket: bucket.to_string(),
|
||||
served_by: rescue_name.clone(),
|
||||
});
|
||||
}
|
||||
Err(rescue_err) => {
|
||||
return Err(format!(
|
||||
"read '{key}' failed in '{bucket}' ({err}); rescue '{rescue_name}' also failed ({rescue_err})"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(format!("read '{key}' failed in '{bucket}': {err}"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Write always goes to target. No rescue fallback for writes — writes
|
||||
/// that silently vanish are the worst possible failure.
|
||||
pub async fn write_smart(
|
||||
&self,
|
||||
bucket: &str,
|
||||
key: &str,
|
||||
data: bytes::Bytes,
|
||||
) -> Result<(), String> {
|
||||
let target_store = {
|
||||
let guard = self.buckets.read().expect("bucket registry lock poisoned");
|
||||
guard.get(bucket).map(|e| e.store.clone())
|
||||
.ok_or_else(|| format!("unknown bucket: {bucket}"))?
|
||||
};
|
||||
match crate::ops::put(&target_store, key, data).await {
|
||||
Ok(()) => Ok(()),
|
||||
Err(err) => {
|
||||
self.journal.append(BucketErrorEvent::new_write(bucket, key, &err)).await;
|
||||
Err(format!("write '{key}' failed in '{bucket}': {err}"))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Trivial reachability check — try to list with limit 0.
|
||||
async fn probe(store: &Arc<dyn ObjectStore>) -> bool {
|
||||
use futures::StreamExt;
|
||||
let mut stream = store.list(None);
|
||||
// Pulling the first item confirms the store responds. Empty bucket = ok.
|
||||
match stream.next().await {
|
||||
Some(Ok(_)) => true,
|
||||
None => true, // empty but reachable
|
||||
Some(Err(_)) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a concrete ObjectStore from a BucketConfig.
|
||||
async fn build_store(
|
||||
bc: &BucketConfig,
|
||||
secrets: &dyn SecretsProvider,
|
||||
) -> Result<Arc<dyn ObjectStore>, String> {
|
||||
match bc.backend.as_str() {
|
||||
"local" => {
|
||||
let root = bc.root.as_deref()
|
||||
.ok_or_else(|| format!("bucket '{}' is backend=local but has no root", bc.name))?;
|
||||
std::fs::create_dir_all(root)
|
||||
.map_err(|e| format!("create bucket dir '{root}': {e}"))?;
|
||||
let fs = LocalFileSystem::new_with_prefix(root)
|
||||
.map_err(|e| format!("init local bucket '{}': {e}", bc.name))?;
|
||||
Ok(Arc::new(fs))
|
||||
}
|
||||
"s3" => {
|
||||
let handle = bc.secret_ref.as_deref()
|
||||
.ok_or_else(|| format!("s3 bucket '{}' has no secret_ref", bc.name))?;
|
||||
let creds: BucketCredentials = secrets.resolve(handle).await?;
|
||||
let s3_bucket = bc.bucket.as_deref()
|
||||
.ok_or_else(|| format!("s3 bucket '{}' has no `bucket` name", bc.name))?;
|
||||
let region = bc.region.as_deref().unwrap_or("us-east-1");
|
||||
|
||||
let mut builder = object_store::aws::AmazonS3Builder::new()
|
||||
.with_bucket_name(s3_bucket)
|
||||
.with_region(region)
|
||||
.with_access_key_id(&creds.access_key)
|
||||
.with_secret_access_key(&creds.secret_key);
|
||||
if let Some(endpoint) = &bc.endpoint {
|
||||
builder = builder.with_endpoint(endpoint);
|
||||
// MinIO and other S3-compatible services often run on plain
|
||||
// HTTP. object_store refuses HTTP by default — opt in when
|
||||
// a custom endpoint is configured (TLS endpoints work either way).
|
||||
if endpoint.starts_with("http://") {
|
||||
builder = builder.with_allow_http(true);
|
||||
}
|
||||
}
|
||||
let s3 = builder.build()
|
||||
.map_err(|e| format!("init s3 bucket '{}': {e}", bc.name))?;
|
||||
Ok(Arc::new(s3))
|
||||
}
|
||||
other => Err(format!("unknown backend '{other}' for bucket '{}'", bc.name)),
|
||||
}
|
||||
}
|
||||
@@ -205,13 +205,3 @@ tr:hover td { background: var(--accent-glow); }
|
||||
padding: 8px 12px; border-bottom: 1px solid var(--border); font-size: 13px;
|
||||
}
|
||||
.table-item:hover { background: var(--accent-glow); }
|
||||
|
||||
/* Phase 19 — Playbook panel */
|
||||
.boosted-row { background: rgba(120, 200, 120, 0.10); }
|
||||
.boosted-row td { border-top: 1px solid rgba(120, 200, 120, 0.30); }
|
||||
.mono-cell {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 11px; color: var(--text-dim);
|
||||
max-width: 220px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||
}
|
||||
.panel-section + .panel-section { margin-top: 18px; }
|
||||
|
||||
+32
-429
@@ -178,116 +178,9 @@ enum Tab {
|
||||
Explore,
|
||||
Sql,
|
||||
Ingest,
|
||||
Playbook,
|
||||
Status,
|
||||
}
|
||||
|
||||
// --- Playbook memory types (Phase 19) ---
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct PlaybookStats {
|
||||
entries: usize,
|
||||
entries_with_embeddings: usize,
|
||||
#[serde(default)]
|
||||
total_names_endorsed: usize,
|
||||
#[serde(default)]
|
||||
sample: Vec<PlaybookSample>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct PlaybookSample {
|
||||
id: String,
|
||||
operation: String,
|
||||
#[serde(default)]
|
||||
city: Option<String>,
|
||||
#[serde(default)]
|
||||
state: Option<String>,
|
||||
#[serde(default)]
|
||||
endorsed: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct HybridResp {
|
||||
#[serde(default)]
|
||||
sql_matches: usize,
|
||||
#[serde(default)]
|
||||
vector_reranked: usize,
|
||||
#[serde(default)]
|
||||
method: String,
|
||||
#[serde(default)]
|
||||
duration_ms: u64,
|
||||
#[serde(default)]
|
||||
answer: Option<String>,
|
||||
#[serde(default)]
|
||||
sources: Vec<HybridSource>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct HybridSource {
|
||||
doc_id: String,
|
||||
chunk_text: String,
|
||||
score: f32,
|
||||
#[serde(default)]
|
||||
sql_verified: bool,
|
||||
#[serde(default)]
|
||||
playbook_boost: f32,
|
||||
#[serde(default)]
|
||||
playbook_citations: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, PartialEq)]
|
||||
struct IndexInfo {
|
||||
index_name: String,
|
||||
source: String,
|
||||
#[serde(default)]
|
||||
chunk_count: usize,
|
||||
#[serde(default)]
|
||||
vector_backend: String,
|
||||
}
|
||||
|
||||
async fn fetch_playbook_stats() -> Result<PlaybookStats, String> {
|
||||
let resp = reqwest::get(&format!("{}/vectors/playbook_memory/stats", api_base()))
|
||||
.await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn rebuild_playbook_memory() -> Result<serde_json::Value, String> {
|
||||
let client = reqwest::Client::new();
|
||||
let resp = client.post(&format!("{}/vectors/playbook_memory/rebuild", api_base()))
|
||||
.json(&serde_json::json!({}))
|
||||
.send().await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn fetch_indexes() -> Result<Vec<IndexInfo>, String> {
|
||||
let resp = reqwest::get(&format!("{}/vectors/indexes", api_base()))
|
||||
.await.map_err(|e| e.to_string())?;
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
async fn hybrid_search(index_name: &str, question: &str, use_playbook: bool, top_k: usize) -> Result<HybridResp, String> {
|
||||
let client = reqwest::Client::new();
|
||||
let resp = client.post(&format!("{}/vectors/hybrid", api_base()))
|
||||
.json(&serde_json::json!({
|
||||
"index_name": index_name,
|
||||
"question": question,
|
||||
"top_k": top_k,
|
||||
"generate": false,
|
||||
"use_playbook_memory": use_playbook,
|
||||
}))
|
||||
.send().await.map_err(|e| e.to_string())?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(format!("HTTP {}: {}", resp.status(), resp.text().await.unwrap_or_default()));
|
||||
}
|
||||
resp.json().await.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
// --- App ---
|
||||
|
||||
#[component]
|
||||
@@ -346,11 +239,6 @@ fn App() -> Element {
|
||||
onclick: move |_| active_tab.set(Tab::Ingest),
|
||||
"Ingest"
|
||||
}
|
||||
button {
|
||||
class: if *active_tab.read() == Tab::Playbook { "tab active" } else { "tab" },
|
||||
onclick: move |_| active_tab.set(Tab::Playbook),
|
||||
"Playbook"
|
||||
}
|
||||
button {
|
||||
class: if *active_tab.read() == Tab::Status { "tab active" } else { "tab" },
|
||||
onclick: move |_| active_tab.set(Tab::Status),
|
||||
@@ -372,7 +260,6 @@ fn App() -> Element {
|
||||
Tab::Explore => rsx! { ExplorePanel { datasets: datasets.read().clone() } },
|
||||
Tab::Sql => rsx! { SqlPanel {} },
|
||||
Tab::Ingest => rsx! { IngestPanel {} },
|
||||
Tab::Playbook => rsx! { PlaybookPanel {} },
|
||||
Tab::Status => rsx! { StatusPanel {} },
|
||||
}
|
||||
}
|
||||
@@ -467,14 +354,14 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
|
||||
div { class: "panel ask-panel",
|
||||
div { class: "ask-hero",
|
||||
h2 { "Ask your data anything" }
|
||||
p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI over the staffing dataset." }
|
||||
p { class: "subtitle", "Natural language → SQL → Results. Powered by local AI." }
|
||||
}
|
||||
|
||||
div { class: "ask-input-row",
|
||||
input {
|
||||
class: "ask-input",
|
||||
value: "{question}",
|
||||
placeholder: "e.g. Which clients placed the most candidates last quarter?",
|
||||
placeholder: "e.g. Which department has the highest average salary?",
|
||||
oninput: move |e| question.set(e.value()),
|
||||
onkeydown: move |e| {
|
||||
if e.key() == Key::Enter {
|
||||
@@ -545,12 +432,10 @@ fn AskPanel(datasets: Vec<Dataset>) -> Element {
|
||||
|
||||
div { class: "ask-examples",
|
||||
"Try: "
|
||||
button { class: "example-btn", onclick: move |_| question.set("How many candidates do we have by city?".into()), "candidates by city" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Top 10 clients by total placements".into()), "top clients by placements" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Open job orders ordered by bill rate descending".into()), "open jobs by rate" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Recruiters with the highest placement count".into()), "top recruiters" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Total billed hours per client last month".into()), "hours per client" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Cold leads: candidates we called more than 5 times but never placed".into()), "cold leads" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Which department has the highest average salary?".into()), "highest avg salary by dept" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("Show me the top 3 most expensive products".into()), "top 3 expensive products" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("How many events per action type?".into()), "events by action" }
|
||||
button { class: "example-btn", onclick: move |_| question.set("List all employees who earn more than 90000".into()), "employees > 90k" }
|
||||
}
|
||||
|
||||
if let Some(sql) = generated_sql.read().as_ref() {
|
||||
@@ -693,7 +578,7 @@ fn ExplorePanel(datasets: Vec<Dataset>) -> Element {
|
||||
|
||||
#[component]
|
||||
fn SqlPanel() -> Element {
|
||||
let mut query_text = use_signal(|| String::from("SELECT candidate_id, first_name, last_name, city, status FROM candidates LIMIT 10"));
|
||||
let mut query_text = use_signal(|| String::from("SELECT * FROM employees LIMIT 10"));
|
||||
let mut result = use_signal(|| None::<Result<QueryResponse, String>>);
|
||||
let mut loading = use_signal(|| false);
|
||||
|
||||
@@ -842,7 +727,7 @@ fn DashboardPanel() -> Element {
|
||||
}
|
||||
div { class: "stat-card accent",
|
||||
div { class: "stat-value", "{s[\"hnsw_loaded\"]}" }
|
||||
div { class: "stat-label", "HNSW Indexes Loaded" }
|
||||
div { class: "stat-label", "HNSW Indexes (27ms)" }
|
||||
}
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s[\"tools\"]}" }
|
||||
@@ -865,27 +750,27 @@ fn DashboardPanel() -> Element {
|
||||
div { class: "arch-grid",
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Ingest" }
|
||||
div { class: "arch-items", "CSV · JSON · PDF (+OCR) · Text · Postgres · MySQL · Inbox watcher · Cron schedules" }
|
||||
div { class: "arch-items", "CSV, JSON, PDF, Text, PostgreSQL, File Watcher" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Storage" }
|
||||
div { class: "arch-items", "Parquet on Object Storage · Delta writes · Compaction · Tombstones · Multi-bucket federation + rescue" }
|
||||
div { class: "arch-items", "Parquet on Object Storage, Delta Writes, Compaction" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Query" }
|
||||
div { class: "arch-items", "DataFusion SQL · MemCache (9.8× hot) · Merge-on-read · AI-safe views" }
|
||||
div { class: "arch-items", "DataFusion SQL, MemCache (9.8x), Hot/Cold" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "AI / Vector" }
|
||||
div { class: "arch-items", "Ollama (local) · Embed/Generate/RAG · HNSW (Parquet) · Lance IVF_PQ · Hybrid SQL+vector · Profile-scoped" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Learning loop" }
|
||||
div { class: "arch-items", "Playbook memory · Endorsement boost · Multi-agent orchestrator · Autotune agent (Pareto-promote)" }
|
||||
div { class: "arch-title", "AI" }
|
||||
div { class: "arch-items", "Ollama (local), Embed, Generate, RAG, HNSW" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Governance" }
|
||||
div { class: "arch-items", "Event journal · PII detection · Tool registry · Access control · Audit log · Catalog v2 metadata" }
|
||||
div { class: "arch-items", "Event Journal, PII Detection, Tool Registry, Access Control" }
|
||||
}
|
||||
div { class: "arch-card",
|
||||
div { class: "arch-title", "Agents" }
|
||||
div { class: "arch-items", "Workspaces, Handoff, Shortlists, Activity Logs" }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -894,23 +779,20 @@ fn DashboardPanel() -> Element {
|
||||
h3 { "Build Progression" }
|
||||
div { class: "phase-list",
|
||||
{rsx! {
|
||||
PhaseItem { num: "0-5", name: "Foundation", detail: "Storage · Catalog · DataFusion · Ollama · UI · gRPC" }
|
||||
PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV · JSON · PDF · Text · auto-schema · dedupe" }
|
||||
PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed · brute-force cosine · LLM grounded answers" }
|
||||
PhaseItem { num: "8", name: "Hot Cache + Deltas", detail: "MemTable LRU · 9.8× speedup · merge-on-read · compaction" }
|
||||
PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract · daily/weekly/monthly tiers · zero-copy handoff" }
|
||||
PhaseItem { num: "9", name: "Event Journal", detail: "Append-only mutation log · time-travel · audit" }
|
||||
PhaseItem { num: "10", name: "Rich Catalog v2", detail: "PII auto-detection · lineage · freshness SLA · sensitivity" }
|
||||
PhaseItem { num: "11", name: "Embedding Versioning", detail: "Per-index model+version · A/B · incremental re-embed" }
|
||||
PhaseItem { num: "12", name: "Tool Registry", detail: "Governed actions · param validation · audit · MCP-ready" }
|
||||
PhaseItem { num: "13", name: "Access Control", detail: "Roles · field-level sensitivity · column masking · query audit" }
|
||||
PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff detection · AI migration prompts · versioned schemas" }
|
||||
PhaseItem { num: "15", name: "HNSW + Trials", detail: "100K vectors · p50 873µs · trial journal · eval harness" }
|
||||
PhaseItem { num: "16", name: "Hot-swap + Autotune", detail: "Promotion registry · rollback · ε-greedy agent · Pareto winner" }
|
||||
PhaseItem { num: "17", name: "Model Profiles + VRAM", detail: "ModelProfile manifests · scoped search · sequential model swap" }
|
||||
PhaseItem { num: "18", name: "Lance hybrid backend", detail: "IVF_PQ build 14× faster · random fetch 112× · S3-native · per-profile routing" }
|
||||
PhaseItem { num: "19", name: "Playbook memory", detail: "Feedback loop · endorsement boost (cap 0.25) · orchestrator write-through · citations" }
|
||||
PhaseItem { num: "+", name: "Federation + Schedules", detail: "Multi-bucket · rescue fallback · error journal · MySQL · PDF OCR · cron ingest · catalog dedupe" }
|
||||
PhaseItem { num: "0-5", name: "Foundation", detail: "Storage, Catalog, DataFusion, AI, UI, gRPC" }
|
||||
PhaseItem { num: "6", name: "Ingest Pipeline", detail: "CSV/JSON/PDF/Text auto-schema" }
|
||||
PhaseItem { num: "7", name: "Vector + RAG", detail: "Embed, Search, LLM Answers" }
|
||||
PhaseItem { num: "8", name: "Hot Cache", detail: "9.8x speedup, Delta Writes" }
|
||||
PhaseItem { num: "8.5", name: "Agent Workspaces", detail: "Per-contract, Instant Handoff" }
|
||||
PhaseItem { num: "9", name: "Event Journal", detail: "Append-only Mutation History" }
|
||||
PhaseItem { num: "10", name: "Rich Catalog", detail: "PII Detection, Lineage" }
|
||||
PhaseItem { num: "11", name: "Embedding Versioning", detail: "Model-proof Vectors" }
|
||||
PhaseItem { num: "12", name: "Tool Registry", detail: "6 Governed Actions + Audit" }
|
||||
PhaseItem { num: "13", name: "Access Control", detail: "Role-based, Field-level" }
|
||||
PhaseItem { num: "14", name: "Schema Evolution", detail: "Diff Detection, AI Migration" }
|
||||
PhaseItem { num: "15", name: "HNSW Index", detail: "100K Search in 27ms" }
|
||||
PhaseItem { num: "16", name: "File Watcher", detail: "Auto-ingest from Inbox" }
|
||||
PhaseItem { num: "17", name: "DB Connector", detail: "PostgreSQL Import" }
|
||||
}}
|
||||
}
|
||||
}
|
||||
@@ -934,285 +816,6 @@ fn PhaseItem(num: String, name: String, detail: String) -> Element {
|
||||
}
|
||||
}
|
||||
|
||||
// === PLAYBOOK — Phase 19 meta-index feedback loop ===
|
||||
|
||||
#[component]
|
||||
fn PlaybookPanel() -> Element {
|
||||
let mut stats = use_signal(|| None::<Result<PlaybookStats, String>>);
|
||||
let mut indexes = use_signal(Vec::<IndexInfo>::new);
|
||||
let mut rebuild_status = use_signal(|| None::<Result<String, String>>);
|
||||
let mut rebuilding = use_signal(|| false);
|
||||
let mut loaded = use_signal(|| false);
|
||||
|
||||
// Comparison state
|
||||
let mut selected_index = use_signal(|| String::new());
|
||||
let mut question = use_signal(|| String::from("reliable assembler in Detroit"));
|
||||
let mut top_k = use_signal(|| 10usize);
|
||||
let mut compare_loading = use_signal(|| false);
|
||||
let mut hits_off = use_signal(|| None::<Result<HybridResp, String>>);
|
||||
let mut hits_on = use_signal(|| None::<Result<HybridResp, String>>);
|
||||
|
||||
let load_all = move || {
|
||||
spawn(async move {
|
||||
stats.set(Some(fetch_playbook_stats().await));
|
||||
if let Ok(ix) = fetch_indexes().await {
|
||||
if selected_index.read().is_empty() {
|
||||
if let Some(default) = ix.iter().find(|i| i.source == "workers_500k").or_else(|| ix.first()) {
|
||||
selected_index.set(default.index_name.clone());
|
||||
}
|
||||
}
|
||||
indexes.set(ix);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
use_effect(move || {
|
||||
if !*loaded.read() {
|
||||
loaded.set(true);
|
||||
load_all();
|
||||
}
|
||||
});
|
||||
|
||||
let do_rebuild = move |_| {
|
||||
spawn(async move {
|
||||
rebuilding.set(true);
|
||||
rebuild_status.set(None);
|
||||
match rebuild_playbook_memory().await {
|
||||
Ok(v) => rebuild_status.set(Some(Ok(format!("rebuild ok — {}", v)))),
|
||||
Err(e) => rebuild_status.set(Some(Err(e))),
|
||||
}
|
||||
// Refresh stats afterward
|
||||
stats.set(Some(fetch_playbook_stats().await));
|
||||
rebuilding.set(false);
|
||||
});
|
||||
};
|
||||
|
||||
let do_compare = move |_| {
|
||||
let idx = selected_index.read().clone();
|
||||
let q = question.read().clone();
|
||||
let k = *top_k.read();
|
||||
if idx.is_empty() || q.trim().is_empty() { return; }
|
||||
spawn(async move {
|
||||
compare_loading.set(true);
|
||||
hits_off.set(None);
|
||||
hits_on.set(None);
|
||||
// Run both sequentially so the embedding cache is shared
|
||||
hits_off.set(Some(hybrid_search(&idx, &q, false, k).await));
|
||||
hits_on.set(Some(hybrid_search(&idx, &q, true, k).await));
|
||||
compare_loading.set(false);
|
||||
});
|
||||
};
|
||||
|
||||
rsx! {
|
||||
div { class: "panel",
|
||||
div { class: "ask-hero",
|
||||
h2 { "Playbook Memory" }
|
||||
p { class: "subtitle",
|
||||
"Phase 19 feedback loop: past successful playbooks boost future search rankings. \
|
||||
Endorsed workers from semantically similar past operations re-rank toward the top, \
|
||||
with citations back to the playbook that endorsed them."
|
||||
}
|
||||
}
|
||||
|
||||
// Stats card
|
||||
div { class: "panel-section",
|
||||
match stats.read().as_ref() {
|
||||
None => rsx! { div { class: "loading", "loading playbook stats..." } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "stats: {e}" } },
|
||||
Some(Ok(s)) => rsx! {
|
||||
div { class: "stat-grid",
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s.entries}" }
|
||||
div { class: "stat-label", "Playbooks in Memory" }
|
||||
}
|
||||
div { class: "stat-card",
|
||||
div { class: "stat-value", "{s.entries_with_embeddings}" }
|
||||
div { class: "stat-label", "Embedded" }
|
||||
}
|
||||
div { class: "stat-card accent",
|
||||
div { class: "stat-value", "{s.total_names_endorsed}" }
|
||||
div { class: "stat-label", "Endorsed Worker-Tags" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
div { class: "sql-actions",
|
||||
button {
|
||||
class: "btn",
|
||||
disabled: *rebuilding.read(),
|
||||
onclick: do_rebuild,
|
||||
if *rebuilding.read() { "rebuilding from successful_playbooks..." } else { "Rebuild from successful_playbooks" }
|
||||
}
|
||||
}
|
||||
if let Some(s) = rebuild_status.read().as_ref() {
|
||||
match s {
|
||||
Ok(msg) => rsx! { div { class: "result-box", "{msg}" } },
|
||||
Err(e) => rsx! { div { class: "error", "{e}" } },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sample playbooks
|
||||
if let Some(Ok(s)) = stats.read().as_ref() {
|
||||
if !s.sample.is_empty() {
|
||||
div { class: "panel-section",
|
||||
h3 { "Sample playbooks" }
|
||||
div { class: "table-wrap",
|
||||
table {
|
||||
thead { tr {
|
||||
th { "ID" }
|
||||
th { "Operation" }
|
||||
th { "Location" }
|
||||
th { "Endorsed" }
|
||||
} }
|
||||
tbody {
|
||||
for pb in s.sample.iter() {
|
||||
{
|
||||
let loc = match (&pb.city, &pb.state) {
|
||||
(Some(c), Some(st)) => format!("{c}, {st}"),
|
||||
_ => "—".into(),
|
||||
};
|
||||
let endorsed = if pb.endorsed.is_empty() {
|
||||
"—".to_string()
|
||||
} else {
|
||||
pb.endorsed.join(", ")
|
||||
};
|
||||
let pid = pb.id.clone();
|
||||
let op = pb.operation.clone();
|
||||
rsx! {
|
||||
tr {
|
||||
td { class: "mono-cell", title: "{pid}", "{pid}" }
|
||||
td { "{op}" }
|
||||
td { "{loc}" }
|
||||
td { "{endorsed}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Side-by-side comparison: boost OFF vs ON
|
||||
div { class: "panel-section",
|
||||
h3 { "See the boost — search compared" }
|
||||
p { class: "hint",
|
||||
"Run the same query against the same index twice — once with playbook boost OFF and once ON. \
|
||||
Hits with non-zero playbook_boost and citations are workers that past similar playbooks endorsed."
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Index" }
|
||||
select {
|
||||
value: "{selected_index}",
|
||||
onchange: move |e| selected_index.set(e.value()),
|
||||
for ix in indexes.read().iter() {
|
||||
option { value: "{ix.index_name}", "{ix.index_name} ({ix.source}, {ix.chunk_count} chunks, {ix.vector_backend})" }
|
||||
}
|
||||
}
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Question" }
|
||||
input {
|
||||
value: "{question}",
|
||||
oninput: move |e| question.set(e.value()),
|
||||
placeholder: "e.g. reliable assembler in Detroit"
|
||||
}
|
||||
}
|
||||
div { class: "form-row",
|
||||
label { "Top K" }
|
||||
input {
|
||||
r#type: "number",
|
||||
value: "{top_k}",
|
||||
oninput: move |e| {
|
||||
if let Ok(n) = e.value().parse::<usize>() { top_k.set(n.clamp(1, 50)); }
|
||||
}
|
||||
}
|
||||
}
|
||||
button {
|
||||
class: "btn btn-ask",
|
||||
disabled: *compare_loading.read(),
|
||||
onclick: do_compare,
|
||||
if *compare_loading.read() { "running both queries..." } else { "Run comparison" }
|
||||
}
|
||||
|
||||
div { class: "explore-grid",
|
||||
div { class: "ds-detail",
|
||||
h3 { "Boost OFF (vanilla)" }
|
||||
match hits_off.read().as_ref() {
|
||||
None => rsx! { div { class: "empty", "—" } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
|
||||
Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
|
||||
}
|
||||
}
|
||||
div { class: "ds-detail",
|
||||
h3 { "Boost ON (Phase 19)" }
|
||||
match hits_on.read().as_ref() {
|
||||
None => rsx! { div { class: "empty", "—" } },
|
||||
Some(Err(e)) => rsx! { div { class: "error", "{e}" } },
|
||||
Some(Ok(r)) => rsx! { HybridHitTable { resp: r.clone() } },
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[component]
|
||||
fn HybridHitTable(resp: HybridResp) -> Element {
|
||||
rsx! {
|
||||
div { class: "results-info",
|
||||
"{resp.sources.len()} hits · {resp.duration_ms}ms · method={resp.method}"
|
||||
}
|
||||
if resp.sources.is_empty() {
|
||||
div { class: "empty-sm", "no hits" }
|
||||
} else {
|
||||
div { class: "table-wrap",
|
||||
table {
|
||||
thead { tr {
|
||||
th { "#" }
|
||||
th { "Doc" }
|
||||
th { "Score" }
|
||||
th { "Boost" }
|
||||
th { "Citations" }
|
||||
th { "Snippet" }
|
||||
} }
|
||||
tbody {
|
||||
for (i, h) in resp.sources.iter().enumerate() {
|
||||
{
|
||||
let snippet: String = h.chunk_text.chars().take(120).collect();
|
||||
let cites = if h.playbook_citations.is_empty() {
|
||||
"—".to_string()
|
||||
} else {
|
||||
h.playbook_citations.join(", ")
|
||||
};
|
||||
let row_class = if h.playbook_boost > 0.0 { "boosted-row" } else { "" };
|
||||
let rank = i + 1;
|
||||
let did = h.doc_id.clone();
|
||||
let score = format!("{:.3}", h.score);
|
||||
let boost = if h.playbook_boost > 0.0 { format!("+{:.3}", h.playbook_boost) } else { "—".into() };
|
||||
rsx! {
|
||||
tr { class: "{row_class}",
|
||||
td { "{rank}" }
|
||||
td { class: "mono-cell", "{did}" }
|
||||
td { "{score}" }
|
||||
td { "{boost}" }
|
||||
td { class: "mono-cell", title: "{cites}", "{cites}" }
|
||||
td { "{snippet}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// === INGEST — Data on-ramp ===
|
||||
|
||||
#[component]
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
[package]
|
||||
name = "vectord-lance"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
# Firewall crate: the Lance stack (Arrow 57, DataFusion 52) is isolated
|
||||
# from the rest of vectord (Arrow 55, DataFusion 47). Public API uses
|
||||
# only std types — no Arrow types cross the crate boundary — so no
|
||||
# version conflict propagates outward.
|
||||
#
|
||||
# See docs/ADR-019-vector-storage.md for the rationale: "productionizing
|
||||
# will need either workspace-wide upgrade or a firewall via a dedicated
|
||||
# vectord-lance crate." This is that firewall.
|
||||
|
||||
[dependencies]
|
||||
# S3 support: Lance delegates to its internal object_store crate when
|
||||
# given s3:// URIs. The "dynamodb" feature enables DynamoDB-based
|
||||
# commit locking for multi-writer S3; we don't need that (single-writer)
|
||||
# so just the base AWS/S3 feature is enough.
|
||||
# Lance 4.0 feature "aws" enables S3-compatible storage via its internal
|
||||
# object_store + opendal crates. Reads AWS_* env vars for credentials.
|
||||
lance = { version = "4.0", default-features = false, features = ["aws"] }
|
||||
lance-index = { version = "4.0", default-features = false }
|
||||
lance-linalg = { version = "4.0", default-features = false }
|
||||
|
||||
# These Arrow/Parquet versions MUST match Lance's expectations — Lance
|
||||
# re-exports their types through its API so any mismatch is a hard
|
||||
# compile error. Keep in sync with lance-bench.
|
||||
arrow = "57"
|
||||
arrow-array = "57"
|
||||
arrow-schema = "57"
|
||||
parquet = "57"
|
||||
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
futures = "0.3"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
bytes = "1"
|
||||
@@ -1,585 +0,0 @@
|
||||
//! Production Lance vector backend (ADR-019 — hybrid architecture).
|
||||
//!
|
||||
//! This is the firewall crate. It owns its own Arrow 57 / DataFusion 52
|
||||
//! / Lance 4 dependency tree. The public API uses only std types
|
||||
//! (`Vec<f32>`, `Vec<String>`, `String`, `bool`) so nothing Arrow-shaped
|
||||
//! crosses the crate boundary. That keeps vectord (Arrow 55) from
|
||||
//! picking up an incompatible dep.
|
||||
//!
|
||||
//! Responsibilities:
|
||||
//! - Migrate an existing binary-blob Parquet vector file into a Lance
|
||||
//! dataset with `FixedSizeList<Float32, dims>`. One-time cost.
|
||||
//! - Append new rows natively (no full-file rewrite — Lance's structural win).
|
||||
//! - Build an IVF_PQ ANN index on the vector column.
|
||||
//! - Vector search (`search`) using the IVF_PQ index when present,
|
||||
//! falling back to full scan otherwise.
|
||||
//! - Random-access row fetch by `doc_id` (`get_by_doc_id`) — the O(1)
|
||||
//! lookup that Parquet-on-object-store can't cheaply do.
|
||||
//! - Cheap count + stats introspection.
|
||||
|
||||
use arrow_array::{
|
||||
Array, ArrayRef, BinaryArray, FixedSizeListArray, Float32Array, Int32Array,
|
||||
RecordBatch, RecordBatchIterator, StringArray,
|
||||
};
|
||||
use arrow_schema::{DataType, Field, Schema};
|
||||
use futures::StreamExt;
|
||||
use serde::Serialize;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
// ================= Public types =================
|
||||
|
||||
/// One search result. Mirrors vectord's existing `SearchResult` shape
|
||||
/// structurally but carries simpler types so this crate stays firewalled.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Hit {
|
||||
pub doc_id: String,
|
||||
pub chunk_text: String,
|
||||
pub score: f32,
|
||||
/// Optional — set by search_with_vector, not by index-only search.
|
||||
pub distance: Option<f32>,
|
||||
}
|
||||
|
||||
/// A fully-hydrated row fetched by `get_by_doc_id` — includes the vector
|
||||
/// so callers can do downstream work (rerank, cite, etc.) without a
|
||||
/// second round trip.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Row {
|
||||
pub doc_id: String,
|
||||
pub chunk_text: String,
|
||||
pub vector: Vec<f32>,
|
||||
pub source: Option<String>,
|
||||
pub chunk_idx: Option<i32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct MigrationStats {
|
||||
pub rows_written: usize,
|
||||
pub dimensions: usize,
|
||||
pub disk_bytes: u64,
|
||||
pub duration_secs: f32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct AppendStats {
|
||||
pub rows_appended: usize,
|
||||
pub disk_bytes_added: u64,
|
||||
pub duration_secs: f32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct IndexStats {
|
||||
pub name: String,
|
||||
pub num_partitions: u32,
|
||||
pub num_bits: u32,
|
||||
pub num_sub_vectors: u32,
|
||||
pub build_time_secs: f32,
|
||||
pub disk_bytes_added: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct ScalarIndexStats {
|
||||
pub name: String,
|
||||
pub column: String,
|
||||
pub build_time_secs: f32,
|
||||
pub disk_bytes_added: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct DatasetStats {
|
||||
pub path: String,
|
||||
pub rows: usize,
|
||||
pub disk_bytes: u64,
|
||||
pub has_vector_index: bool,
|
||||
pub has_doc_id_index: bool,
|
||||
}
|
||||
|
||||
// ================= The backend =================
|
||||
|
||||
/// Thin wrapper around a Lance dataset path. Lance handles the heavy
|
||||
/// lifting — we just expose a narrow API.
|
||||
#[derive(Clone)]
|
||||
pub struct LanceVectorStore {
|
||||
/// Local filesystem path or object-store URI (file:///..., s3://...).
|
||||
/// Lance's internal URI parsing handles both.
|
||||
path: String,
|
||||
}
|
||||
|
||||
impl LanceVectorStore {
|
||||
pub fn new(path: impl Into<String>) -> Self {
|
||||
Self { path: path.into() }
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &str { &self.path }
|
||||
|
||||
/// Row count via Lance's fast metadata path (no scan).
|
||||
pub async fn count(&self) -> Result<usize, String> {
|
||||
let dataset = open_or_err(&self.path).await?;
|
||||
let n = dataset.count_rows(None).await.map_err(e)?;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
/// True if the on-disk dataset exists AND has at least one `vector`
|
||||
/// column index attached.
|
||||
pub async fn has_vector_index(&self) -> Result<bool, String> {
|
||||
use lance_index::DatasetIndexExt;
|
||||
let dataset = match lance::dataset::Dataset::open(&self.path).await {
|
||||
Ok(d) => d,
|
||||
Err(_) => return Ok(false),
|
||||
};
|
||||
let indexes = dataset.load_indices().await.map_err(e)?;
|
||||
Ok(indexes.iter().any(|ix| {
|
||||
ix.fields.iter().any(|fid| {
|
||||
dataset.schema().field_by_id(*fid)
|
||||
.map(|f| f.name == "vector")
|
||||
.unwrap_or(false)
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
pub async fn stats(&self) -> Result<DatasetStats, String> {
|
||||
let rows = self.count().await.unwrap_or(0);
|
||||
let disk_bytes = dir_size_bytes(&strip_file_uri(&self.path));
|
||||
let has_vector_index = self.has_vector_index().await.unwrap_or(false);
|
||||
let has_doc_id_index = self.has_scalar_index("doc_id").await.unwrap_or(false);
|
||||
Ok(DatasetStats {
|
||||
path: self.path.clone(),
|
||||
rows,
|
||||
disk_bytes,
|
||||
has_vector_index,
|
||||
has_doc_id_index,
|
||||
})
|
||||
}
|
||||
|
||||
/// Migrate a vectord-format Parquet file into a Lance dataset.
|
||||
///
|
||||
/// Input schema (vectord's binary-blob format):
|
||||
/// - source : Utf8
|
||||
/// - doc_id : Utf8
|
||||
/// - chunk_idx : Int32
|
||||
/// - chunk_text : Utf8
|
||||
/// - vector : Binary (raw f32 little-endian bytes)
|
||||
///
|
||||
/// Output schema (Lance-friendly):
|
||||
/// - source : Utf8
|
||||
/// - doc_id : Utf8
|
||||
/// - chunk_idx : Int32
|
||||
/// - chunk_text : Utf8
|
||||
/// - vector : FixedSizeList<Float32, dims>
|
||||
///
|
||||
/// Idempotent at the file level — if the target exists, it's
|
||||
/// overwritten. Caller must manage destination paths.
|
||||
pub async fn migrate_from_parquet_bytes(
|
||||
&self,
|
||||
parquet_bytes: &[u8],
|
||||
) -> Result<MigrationStats, String> {
|
||||
let t0 = Instant::now();
|
||||
let (schema, batches, rows) = read_parquet(parquet_bytes)?;
|
||||
let dims = detect_vector_dims(&batches)?;
|
||||
let (new_schema, new_batches) = convert_to_fixed_size_list(&schema, batches, dims)?;
|
||||
|
||||
// Overwrite any prior dataset at this path.
|
||||
let _ = std::fs::remove_dir_all(&strip_file_uri(&self.path));
|
||||
|
||||
write_dataset(&self.path, new_schema, new_batches).await?;
|
||||
let disk_bytes = dir_size_bytes(&strip_file_uri(&self.path));
|
||||
|
||||
Ok(MigrationStats {
|
||||
rows_written: rows,
|
||||
dimensions: dims,
|
||||
disk_bytes,
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Native Lance append — does NOT rewrite existing files. New rows
|
||||
/// land as a separate fragment; readers union across fragments at
|
||||
/// query time. Contrast: our Parquet path requires rewriting the
|
||||
/// entire vector file to add rows.
|
||||
pub async fn append(
|
||||
&self,
|
||||
source: Option<String>,
|
||||
doc_ids: Vec<String>,
|
||||
chunk_idxs: Vec<i32>,
|
||||
chunk_texts: Vec<String>,
|
||||
vectors: Vec<Vec<f32>>,
|
||||
) -> Result<AppendStats, String> {
|
||||
let n = doc_ids.len();
|
||||
if n == 0 {
|
||||
return Ok(AppendStats { rows_appended: 0, disk_bytes_added: 0, duration_secs: 0.0 });
|
||||
}
|
||||
if chunk_idxs.len() != n || chunk_texts.len() != n || vectors.len() != n {
|
||||
return Err(format!(
|
||||
"append: length mismatch (doc_ids={n}, chunk_idxs={}, chunk_texts={}, vectors={})",
|
||||
chunk_idxs.len(), chunk_texts.len(), vectors.len(),
|
||||
));
|
||||
}
|
||||
let dims = vectors[0].len();
|
||||
for (i, v) in vectors.iter().enumerate() {
|
||||
if v.len() != dims {
|
||||
return Err(format!("append: row {i} has {} dims, expected {}", v.len(), dims));
|
||||
}
|
||||
}
|
||||
|
||||
let t0 = Instant::now();
|
||||
let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
|
||||
|
||||
let src_arr = StringArray::from(
|
||||
(0..n).map(|_| source.clone()).collect::<Vec<_>>()
|
||||
);
|
||||
let doc_id_arr = StringArray::from(doc_ids);
|
||||
let chunk_idx_arr = Int32Array::from(chunk_idxs);
|
||||
let chunk_text_arr = StringArray::from(chunk_texts);
|
||||
|
||||
let mut flat: Vec<f32> = Vec::with_capacity(n * dims);
|
||||
for v in vectors { flat.extend(v); }
|
||||
let values = Float32Array::from(flat);
|
||||
let item_field = Arc::new(Field::new("item", DataType::Float32, true));
|
||||
let vec_arr = FixedSizeListArray::try_new(
|
||||
item_field.clone(), dims as i32, Arc::new(values), None,
|
||||
).map_err(e)?;
|
||||
|
||||
let schema = Arc::new(Schema::new(vec![
|
||||
Field::new("source", DataType::Utf8, true),
|
||||
Field::new("doc_id", DataType::Utf8, false),
|
||||
Field::new("chunk_idx", DataType::Int32, true),
|
||||
Field::new("chunk_text", DataType::Utf8, true),
|
||||
Field::new("vector", DataType::FixedSizeList(item_field, dims as i32), false),
|
||||
]));
|
||||
|
||||
let arrays: Vec<ArrayRef> = vec![
|
||||
Arc::new(src_arr), Arc::new(doc_id_arr), Arc::new(chunk_idx_arr),
|
||||
Arc::new(chunk_text_arr), Arc::new(vec_arr),
|
||||
];
|
||||
let batch = RecordBatch::try_new(schema.clone(), arrays).map_err(e)?;
|
||||
let reader = RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema);
|
||||
|
||||
use lance::dataset::{Dataset, WriteMode, WriteParams};
|
||||
let params = WriteParams { mode: WriteMode::Append, ..Default::default() };
|
||||
Dataset::write(reader, &self.path, Some(params)).await.map_err(e)?;
|
||||
|
||||
Ok(AppendStats {
|
||||
rows_appended: n,
|
||||
disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Build an IVF_PQ vector index. Replaces any prior index with the
|
||||
/// same name. Callers pass explicit params — sensible defaults for
|
||||
/// ~100K × 768d: num_partitions=316 (≈√N), num_bits=8, num_sub_vectors=48.
|
||||
pub async fn build_index(
|
||||
&self,
|
||||
num_partitions: u32,
|
||||
num_bits: u32,
|
||||
num_sub_vectors: u32,
|
||||
) -> Result<IndexStats, String> {
|
||||
use lance::dataset::Dataset;
|
||||
use lance::index::vector::VectorIndexParams;
|
||||
use lance_index::{DatasetIndexExt, IndexType};
|
||||
use lance_linalg::distance::MetricType;
|
||||
|
||||
let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
|
||||
let t0 = Instant::now();
|
||||
|
||||
let mut dataset = Dataset::open(&self.path).await.map_err(e)?;
|
||||
let params = VectorIndexParams::ivf_pq(
|
||||
num_partitions as usize,
|
||||
num_bits as u8,
|
||||
num_sub_vectors as usize,
|
||||
MetricType::Cosine,
|
||||
50, // max_iterations — same as bench
|
||||
);
|
||||
dataset.create_index(
|
||||
&["vector"],
|
||||
IndexType::Vector,
|
||||
Some("vec_idx".into()),
|
||||
¶ms,
|
||||
true, // replace
|
||||
).await.map_err(e)?;
|
||||
|
||||
Ok(IndexStats {
|
||||
name: "vec_idx".into(),
|
||||
num_partitions,
|
||||
num_bits,
|
||||
num_sub_vectors,
|
||||
build_time_secs: t0.elapsed().as_secs_f32(),
|
||||
disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
|
||||
})
|
||||
}
|
||||
|
||||
/// Build a scalar btree index on `doc_id`. Makes `get_by_doc_id`
|
||||
/// O(log N) instead of a filter-scan of every fragment. Small index
|
||||
/// footprint — a few hundred KB on 100K rows.
|
||||
pub async fn build_scalar_index(&self, column: &str) -> Result<ScalarIndexStats, String> {
|
||||
use lance::dataset::Dataset;
|
||||
use lance_index::{DatasetIndexExt, IndexType};
|
||||
use lance_index::scalar::ScalarIndexParams;
|
||||
|
||||
let pre_bytes = dir_size_bytes(&strip_file_uri(&self.path));
|
||||
let t0 = Instant::now();
|
||||
|
||||
let mut dataset = Dataset::open(&self.path).await.map_err(e)?;
|
||||
let idx_name = format!("{column}_btree");
|
||||
dataset.create_index(
|
||||
&[column],
|
||||
IndexType::Scalar,
|
||||
Some(idx_name.clone()),
|
||||
&ScalarIndexParams::default(),
|
||||
true,
|
||||
).await.map_err(e)?;
|
||||
|
||||
Ok(ScalarIndexStats {
|
||||
name: idx_name,
|
||||
column: column.to_string(),
|
||||
build_time_secs: t0.elapsed().as_secs_f32(),
|
||||
disk_bytes_added: dir_size_bytes(&strip_file_uri(&self.path)).saturating_sub(pre_bytes),
|
||||
})
|
||||
}
|
||||
|
||||
/// True if a scalar index exists on the named column.
|
||||
pub async fn has_scalar_index(&self, column: &str) -> Result<bool, String> {
|
||||
use lance_index::DatasetIndexExt;
|
||||
let dataset = match lance::dataset::Dataset::open(&self.path).await {
|
||||
Ok(d) => d,
|
||||
Err(_) => return Ok(false),
|
||||
};
|
||||
let indexes = dataset.load_indices().await.map_err(e)?;
|
||||
Ok(indexes.iter().any(|ix| {
|
||||
ix.fields.iter().any(|fid| {
|
||||
dataset.schema().field_by_id(*fid)
|
||||
.map(|f| f.name == column)
|
||||
.unwrap_or(false)
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
/// Search for top_k nearest neighbors of `query`. Uses the IVF_PQ
|
||||
/// index if one exists; otherwise does a full scan (slow but
|
||||
/// correct — useful during development before index build).
|
||||
pub async fn search(&self, query: &[f32], top_k: usize) -> Result<Vec<Hit>, String> {
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
let dataset = Dataset::open(&self.path).await.map_err(e)?;
|
||||
let qarr = Float32Array::from(query.to_vec());
|
||||
|
||||
let mut scanner = dataset.scan();
|
||||
scanner.nearest("vector", &qarr, top_k as usize).map_err(e)?;
|
||||
scanner.project(&["doc_id", "chunk_text"]).map_err(e)?;
|
||||
|
||||
let mut stream = scanner.try_into_stream().await.map_err(e)?;
|
||||
let mut hits: Vec<Hit> = Vec::with_capacity(top_k);
|
||||
|
||||
while let Some(batch) = stream.next().await {
|
||||
let batch = batch.map_err(e)?;
|
||||
let doc_ids = batch.column_by_name("doc_id")
|
||||
.ok_or_else(|| "no doc_id column in search result".to_string())?
|
||||
.as_any().downcast_ref::<StringArray>()
|
||||
.ok_or_else(|| "doc_id is not StringArray".to_string())?;
|
||||
let chunk_texts = batch.column_by_name("chunk_text")
|
||||
.and_then(|c| c.as_any().downcast_ref::<StringArray>());
|
||||
// Lance tacks on a `_distance` column for nearest() queries.
|
||||
let distances = batch.column_by_name("_distance")
|
||||
.and_then(|c| c.as_any().downcast_ref::<Float32Array>());
|
||||
for row in 0..batch.num_rows() {
|
||||
let d = distances.map(|a| a.value(row));
|
||||
hits.push(Hit {
|
||||
doc_id: doc_ids.value(row).to_string(),
|
||||
chunk_text: chunk_texts.map(|a| a.value(row).to_string()).unwrap_or_default(),
|
||||
score: d.map(|d| 1.0 - d).unwrap_or(0.0), // cosine distance → similarity
|
||||
distance: d,
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(hits)
|
||||
}
|
||||
|
||||
/// Fetch one row by doc_id. Implementation: Lance filter-pushdown
|
||||
/// scan — O(1) with partition pruning on a proper btree index,
|
||||
/// O(N) on vector-only datasets (still far faster than reading
|
||||
/// the whole Parquet file). We don't build a scalar index on
|
||||
/// doc_id yet; that's a future optimization.
|
||||
pub async fn get_by_doc_id(&self, doc_id: &str) -> Result<Option<Row>, String> {
|
||||
use lance::dataset::Dataset;
|
||||
|
||||
let dataset = Dataset::open(&self.path).await.map_err(e)?;
|
||||
let filter = format!("doc_id = '{}'", doc_id.replace('\'', "''"));
|
||||
let mut scanner = dataset.scan();
|
||||
scanner.filter(&filter).map_err(e)?;
|
||||
scanner.limit(Some(1), None).map_err(e)?;
|
||||
let mut stream = scanner.try_into_stream().await.map_err(e)?;
|
||||
|
||||
while let Some(batch) = stream.next().await {
|
||||
let batch = batch.map_err(e)?;
|
||||
if batch.num_rows() == 0 { continue; }
|
||||
return Ok(Some(row_from_batch(&batch, 0)?));
|
||||
}
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
// ================= Internal helpers =================
|
||||
|
||||
fn e<T: std::fmt::Display>(err: T) -> String { err.to_string() }
|
||||
|
||||
/// `file:///abs/path` → `/abs/path`. Leave other URI schemes as-is for
|
||||
/// helpers that only work on local paths (dir_size_bytes, remove_dir_all).
|
||||
fn strip_file_uri(uri: &str) -> String {
|
||||
uri.strip_prefix("file://").unwrap_or(uri).to_string()
|
||||
}
|
||||
|
||||
async fn open_or_err(path: &str) -> Result<lance::dataset::Dataset, String> {
|
||||
lance::dataset::Dataset::open(path).await.map_err(e)
|
||||
}
|
||||
|
||||
fn dir_size_bytes(path: &str) -> u64 {
|
||||
fn recurse(p: &std::path::Path) -> u64 {
|
||||
let Ok(meta) = std::fs::metadata(p) else { return 0; };
|
||||
if meta.is_file() { return meta.len(); }
|
||||
let Ok(entries) = std::fs::read_dir(p) else { return 0; };
|
||||
entries.filter_map(|e| e.ok()).map(|e| recurse(&e.path())).sum()
|
||||
}
|
||||
recurse(std::path::Path::new(path))
|
||||
}
|
||||
|
||||
fn read_parquet(bytes: &[u8]) -> Result<(Arc<Schema>, Vec<RecordBatch>, usize), String> {
|
||||
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
|
||||
let builder = ParquetRecordBatchReaderBuilder::try_new(bytes::Bytes::copy_from_slice(bytes))
|
||||
.map_err(e)?;
|
||||
let schema = builder.schema().clone();
|
||||
let reader = builder.build().map_err(e)?;
|
||||
let batches: Vec<RecordBatch> = reader.collect::<Result<_, _>>().map_err(e)?;
|
||||
let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
|
||||
Ok((schema, batches, rows))
|
||||
}
|
||||
|
||||
fn detect_vector_dims(batches: &[RecordBatch]) -> Result<usize, String> {
|
||||
for batch in batches {
|
||||
let idx = batch.schema().index_of("vector")
|
||||
.map_err(|_| "no 'vector' column".to_string())?;
|
||||
let col = batch.column(idx);
|
||||
if let Some(binary) = col.as_any().downcast_ref::<BinaryArray>() {
|
||||
for i in 0..binary.len() {
|
||||
if !binary.is_null(i) {
|
||||
return Ok(binary.value(i).len() / 4);
|
||||
}
|
||||
}
|
||||
} else if let Some(fsl) = col.as_any().downcast_ref::<FixedSizeListArray>() {
|
||||
return Ok(fsl.value_length() as usize);
|
||||
}
|
||||
}
|
||||
Err("could not determine vector dimensions".into())
|
||||
}
|
||||
|
||||
fn convert_to_fixed_size_list(
|
||||
schema: &Arc<Schema>,
|
||||
batches: Vec<RecordBatch>,
|
||||
dims: usize,
|
||||
) -> Result<(Arc<Schema>, Vec<RecordBatch>), String> {
|
||||
let new_fields: Vec<Arc<Field>> = schema
|
||||
.fields()
|
||||
.iter()
|
||||
.map(|f| {
|
||||
if f.name() == "vector" {
|
||||
Arc::new(Field::new(
|
||||
"vector",
|
||||
DataType::FixedSizeList(
|
||||
Arc::new(Field::new("item", DataType::Float32, true)),
|
||||
dims as i32,
|
||||
),
|
||||
false,
|
||||
))
|
||||
} else {
|
||||
f.clone()
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
let new_schema = Arc::new(Schema::new(new_fields));
|
||||
|
||||
let mut out = Vec::with_capacity(batches.len());
|
||||
for batch in batches {
|
||||
let vec_idx = batch.schema().index_of("vector").map_err(e)?;
|
||||
let mut new_cols: Vec<ArrayRef> = Vec::with_capacity(batch.num_columns());
|
||||
for (i, col) in batch.columns().iter().enumerate() {
|
||||
if i == vec_idx {
|
||||
if let Some(bin) = col.as_any().downcast_ref::<BinaryArray>() {
|
||||
new_cols.push(Arc::new(binary_to_fsl(bin, dims)?));
|
||||
} else if col.as_any().is::<FixedSizeListArray>() {
|
||||
// Already in the right shape — just clone.
|
||||
new_cols.push(col.clone());
|
||||
} else {
|
||||
return Err("vector column is neither Binary nor FixedSizeList".into());
|
||||
}
|
||||
} else {
|
||||
new_cols.push(col.clone());
|
||||
}
|
||||
}
|
||||
out.push(RecordBatch::try_new(new_schema.clone(), new_cols).map_err(e)?);
|
||||
}
|
||||
Ok((new_schema, out))
|
||||
}
|
||||
|
||||
fn binary_to_fsl(bin: &BinaryArray, dims: usize) -> Result<FixedSizeListArray, String> {
|
||||
let n = bin.len();
|
||||
let mut flat: Vec<f32> = Vec::with_capacity(n * dims);
|
||||
for i in 0..n {
|
||||
if bin.is_null(i) {
|
||||
flat.extend(std::iter::repeat(0.0).take(dims));
|
||||
continue;
|
||||
}
|
||||
let b = bin.value(i);
|
||||
if b.len() != dims * 4 {
|
||||
return Err(format!(
|
||||
"row {i}: {} bytes vs expected {} ({} × f32)",
|
||||
b.len(), dims * 4, dims,
|
||||
));
|
||||
}
|
||||
for c in b.chunks_exact(4) {
|
||||
flat.push(f32::from_le_bytes([c[0], c[1], c[2], c[3]]));
|
||||
}
|
||||
}
|
||||
let values = Float32Array::from(flat);
|
||||
let field = Arc::new(Field::new("item", DataType::Float32, true));
|
||||
FixedSizeListArray::try_new(field, dims as i32, Arc::new(values), None).map_err(e)
|
||||
}
|
||||
|
||||
async fn write_dataset(
|
||||
path: &str,
|
||||
schema: Arc<Schema>,
|
||||
batches: Vec<RecordBatch>,
|
||||
) -> Result<(), String> {
|
||||
use lance::dataset::{Dataset, WriteParams};
|
||||
let reader = RecordBatchIterator::new(batches.into_iter().map(Ok), schema);
|
||||
Dataset::write(reader, path, Some(WriteParams::default()))
|
||||
.await.map_err(e)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn row_from_batch(batch: &RecordBatch, row: usize) -> Result<Row, String> {
|
||||
let doc_id = batch.column_by_name("doc_id")
|
||||
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
|
||||
.map(|a| a.value(row).to_string())
|
||||
.ok_or_else(|| "missing doc_id".to_string())?;
|
||||
let chunk_text = batch.column_by_name("chunk_text")
|
||||
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
|
||||
.map(|a| a.value(row).to_string())
|
||||
.unwrap_or_default();
|
||||
let source = batch.column_by_name("source")
|
||||
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
|
||||
.and_then(|a| if a.is_null(row) { None } else { Some(a.value(row).to_string()) });
|
||||
let chunk_idx = batch.column_by_name("chunk_idx")
|
||||
.and_then(|c| c.as_any().downcast_ref::<Int32Array>())
|
||||
.and_then(|a| if a.is_null(row) { None } else { Some(a.value(row)) });
|
||||
|
||||
let vec_col = batch.column_by_name("vector")
|
||||
.ok_or_else(|| "no vector column in row fetch".to_string())?;
|
||||
let fsl = vec_col.as_any().downcast_ref::<FixedSizeListArray>()
|
||||
.ok_or_else(|| "vector column is not FixedSizeList".to_string())?;
|
||||
let inner = fsl.value(row);
|
||||
let floats = inner.as_any().downcast_ref::<Float32Array>()
|
||||
.ok_or_else(|| "vector inner not Float32".to_string())?;
|
||||
let mut v = Vec::with_capacity(floats.len());
|
||||
for i in 0..floats.len() { v.push(floats.value(i)); }
|
||||
|
||||
Ok(Row { doc_id, chunk_text, vector: v, source, chunk_idx })
|
||||
}
|
||||
@@ -7,11 +7,6 @@ edition = "2024"
|
||||
shared = { path = "../shared" }
|
||||
storaged = { path = "../storaged" }
|
||||
aibridge = { path = "../aibridge" }
|
||||
catalogd = { path = "../catalogd" }
|
||||
queryd = { path = "../queryd" }
|
||||
# ADR-019 firewall — vectord-lance owns its own Arrow 57 / Lance 4 deps.
|
||||
# Public API uses only std types so no version conflict propagates here.
|
||||
vectord-lance = { path = "../vectord-lance" }
|
||||
tokio = { workspace = true }
|
||||
axum = { workspace = true }
|
||||
serde = { workspace = true }
|
||||
@@ -23,4 +18,3 @@ parquet = { workspace = true }
|
||||
arrow = { workspace = true }
|
||||
chrono = { workspace = true }
|
||||
instant-distance = { workspace = true }
|
||||
uuid = { workspace = true }
|
||||
|
||||
@@ -1,711 +0,0 @@
|
||||
//! Phase 16.2 + 16.5 — The autotune agent.
|
||||
//!
|
||||
//! A long-running tokio task that watches the trial journal and
|
||||
//! autonomously proposes + runs new HNSW configs. Distinct from
|
||||
//! `autotune::run_autotune` which is synchronous (one HTTP call, grid
|
||||
//! of trials, done). The agent is the continuous version: it sleeps,
|
||||
//! wakes on triggers, proposes configs based on prior trial history,
|
||||
//! runs them one at a time, and auto-promotes when it finds an
|
||||
//! improvement.
|
||||
//!
|
||||
//! Design invariants:
|
||||
//! - Trials are data (ADR-018). Every proposal reads the journal; every
|
||||
//! attempt appends to it. The journal is the agent's memory.
|
||||
//! - One trial at a time. Bounded Ollama load — the agent never fires
|
||||
//! multiple parallel embeddings and respects `cooldown_between_trials_secs`.
|
||||
//! - Rate-limited. `max_trials_per_hour` is a hard ceiling so a
|
||||
//! misbehaving proposal function can't saturate the system.
|
||||
//! - Never promotes below `min_recall`. Same safety gate as
|
||||
//! `run_autotune` — we will not make the index worse.
|
||||
//! - Triggered OR periodic. Ingest enqueues a `DatasetAppended` event
|
||||
//! when a new batch lands; the agent also wakes periodically to keep
|
||||
//! exploring even when nothing changed externally.
|
||||
//! - Graceful shutdown via the `stop_tx` signal — the handle's Drop
|
||||
//! doesn't force-kill, but `stop()` requests a clean exit after the
|
||||
//! current trial.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::{Mutex, RwLock, mpsc, oneshot};
|
||||
|
||||
use aibridge::client::AiClient;
|
||||
use catalogd::registry::Registry as CatalogRegistry;
|
||||
|
||||
use crate::embedding_cache::EmbeddingCache;
|
||||
use crate::harness;
|
||||
use crate::hnsw::HnswStore;
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::promotion::{PromotionEntry, PromotionRegistry};
|
||||
use crate::trial::{HnswConfig, Trial, TrialJournal, TrialMetrics};
|
||||
|
||||
// -------- Public-facing types --------
|
||||
|
||||
/// Runtime configuration for the agent. Mirrored in shared::config under
|
||||
/// `[agent]`. Defaults are conservative — designed to tune slowly in the
|
||||
/// background without fighting real workloads for GPU time.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct AgentConfig {
|
||||
/// Master switch. When false, `spawn` returns a handle but the loop
|
||||
/// doesn't run.
|
||||
pub enabled: bool,
|
||||
/// Periodic wake-up — even if the trigger queue is empty, every N
|
||||
/// seconds the agent picks an index with trials and proposes one
|
||||
/// more config. Keeps exploration alive on idle indexes.
|
||||
pub cycle_interval_secs: u64,
|
||||
/// Minimum gap between two trials on the SAME index. Prevents the
|
||||
/// agent from hammering Ollama when a hot index has many pending
|
||||
/// triggers in a row.
|
||||
pub cooldown_between_trials_secs: u64,
|
||||
/// Below this recall, a proposal is never promoted — even if it
|
||||
/// beats the champion on latency.
|
||||
pub min_recall: f32,
|
||||
/// Budget cap: hard ceiling on trials per hour across all indexes.
|
||||
/// When hit, the agent idles until the hour window rolls.
|
||||
pub max_trials_per_hour: u32,
|
||||
}
|
||||
|
||||
impl Default for AgentConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false, // opt-in — don't auto-start until J turns it on
|
||||
cycle_interval_secs: 60,
|
||||
cooldown_between_trials_secs: 30,
|
||||
min_recall: 0.9,
|
||||
max_trials_per_hour: 30,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What caused the agent to look at a particular index. Recorded in the
|
||||
/// trial's note field so we can tell "new data arrived" trials from
|
||||
/// "periodic exploration" trials in the journal.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub enum TriggerReason {
|
||||
/// Ingest just appended to a dataset that has attached HNSW indexes.
|
||||
DatasetAppended { dataset: String },
|
||||
/// A human or another agent hit `/agent/enqueue/{index}`.
|
||||
Manual,
|
||||
/// Periodic wake — no external event, just keep exploring.
|
||||
Periodic,
|
||||
}
|
||||
|
||||
/// One unit of work for the agent.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct TriggerEvent {
|
||||
pub index_name: String,
|
||||
pub reason: TriggerReason,
|
||||
pub enqueued_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
impl TriggerEvent {
|
||||
pub fn manual(index_name: impl Into<String>) -> Self {
|
||||
Self { index_name: index_name.into(), reason: TriggerReason::Manual, enqueued_at: Utc::now() }
|
||||
}
|
||||
pub fn dataset_appended(index_name: impl Into<String>, dataset: impl Into<String>) -> Self {
|
||||
Self {
|
||||
index_name: index_name.into(),
|
||||
reason: TriggerReason::DatasetAppended { dataset: dataset.into() },
|
||||
enqueued_at: Utc::now(),
|
||||
}
|
||||
}
|
||||
pub fn periodic(index_name: impl Into<String>) -> Self {
|
||||
Self { index_name: index_name.into(), reason: TriggerReason::Periodic, enqueued_at: Utc::now() }
|
||||
}
|
||||
}
|
||||
|
||||
/// Observable snapshot of the agent's state — what `/agent/status` returns.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct AgentStatus {
|
||||
pub running: bool,
|
||||
pub config: AgentConfig,
|
||||
pub queue_depth: usize,
|
||||
pub trials_run: u64,
|
||||
pub promotions: u64,
|
||||
pub trials_in_last_hour: u32,
|
||||
pub last_event: Option<AgentEvent>,
|
||||
pub started_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
/// Last thing that happened — useful for "why didn't it do anything?" debugging.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct AgentEvent {
|
||||
pub at: DateTime<Utc>,
|
||||
pub kind: String, // "trial_completed" | "promoted" | "skipped_rate_limit" | etc
|
||||
pub index_name: Option<String>,
|
||||
pub detail: String,
|
||||
}
|
||||
|
||||
/// Handle returned by `spawn`. Holds the trigger sender + shared status +
|
||||
/// stop signal.
|
||||
#[derive(Clone)]
|
||||
pub struct AgentHandle {
|
||||
trigger_tx: mpsc::Sender<TriggerEvent>,
|
||||
inner: Arc<AgentInner>,
|
||||
}
|
||||
|
||||
struct AgentInner {
|
||||
status: RwLock<AgentStatus>,
|
||||
stop_tx: Mutex<Option<oneshot::Sender<()>>>,
|
||||
queue_len: Mutex<usize>, // mirror of the channel capacity — for status reporting
|
||||
recent_trials: Mutex<VecDeque<DateTime<Utc>>>, // ring of recent trial timestamps for rate limit
|
||||
}
|
||||
|
||||
impl AgentHandle {
|
||||
/// Enqueue a trigger. Returns Err if the agent isn't running or the
|
||||
/// queue is full (backpressure — dropping events is correct here
|
||||
/// since periodic exploration will pick up the slack).
|
||||
pub async fn enqueue(&self, event: TriggerEvent) -> Result<(), String> {
|
||||
self.trigger_tx.try_send(event).map_err(|e| format!("enqueue: {e}"))?;
|
||||
let mut guard = self.inner.queue_len.lock().await;
|
||||
*guard = guard.saturating_add(1);
|
||||
// Update queue_depth in status for observability.
|
||||
let mut s = self.inner.status.write().await;
|
||||
s.queue_depth = *guard;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn status(&self) -> AgentStatus {
|
||||
let mut s = self.inner.status.read().await.clone();
|
||||
// Refresh rate-limit window from ring buffer.
|
||||
let cutoff = Utc::now() - chrono::Duration::hours(1);
|
||||
let ring = self.inner.recent_trials.lock().await;
|
||||
s.trials_in_last_hour = ring.iter().filter(|t| **t >= cutoff).count() as u32;
|
||||
s
|
||||
}
|
||||
|
||||
/// Request a graceful stop. Returns immediately — the loop exits
|
||||
/// after its current trial completes.
|
||||
pub async fn stop(&self) -> bool {
|
||||
let mut guard = self.inner.stop_tx.lock().await;
|
||||
if let Some(tx) = guard.take() {
|
||||
let _ = tx.send(());
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -------- Agent state holder --------
|
||||
|
||||
/// Everything the agent needs to run a trial. Mirrors the fields of
|
||||
/// `VectorState` the agent actually uses. Kept separate so the service
|
||||
/// layer builds it explicitly — no clone of unneeded state.
|
||||
#[derive(Clone)]
|
||||
pub struct AgentDeps {
|
||||
pub store: Arc<dyn ObjectStore>,
|
||||
pub ai_client: AiClient,
|
||||
pub catalog: CatalogRegistry,
|
||||
pub index_registry: IndexRegistry,
|
||||
pub hnsw_store: HnswStore,
|
||||
pub embedding_cache: EmbeddingCache,
|
||||
pub trial_journal: TrialJournal,
|
||||
pub promotion_registry: PromotionRegistry,
|
||||
}
|
||||
|
||||
// -------- Spawn --------
|
||||
|
||||
/// Start the agent loop in a background tokio task. Returns a handle
|
||||
/// the caller uses to enqueue events and read status.
|
||||
pub fn spawn(config: AgentConfig, deps: AgentDeps) -> AgentHandle {
|
||||
let (trigger_tx, trigger_rx) = mpsc::channel::<TriggerEvent>(256);
|
||||
let (stop_tx, stop_rx) = oneshot::channel::<()>();
|
||||
|
||||
let status = AgentStatus {
|
||||
running: config.enabled,
|
||||
config: config.clone(),
|
||||
queue_depth: 0,
|
||||
trials_run: 0,
|
||||
promotions: 0,
|
||||
trials_in_last_hour: 0,
|
||||
last_event: None,
|
||||
started_at: if config.enabled { Some(Utc::now()) } else { None },
|
||||
};
|
||||
|
||||
let inner = Arc::new(AgentInner {
|
||||
status: RwLock::new(status),
|
||||
stop_tx: Mutex::new(Some(stop_tx)),
|
||||
queue_len: Mutex::new(0),
|
||||
recent_trials: Mutex::new(VecDeque::with_capacity(64)),
|
||||
});
|
||||
|
||||
if config.enabled {
|
||||
tracing::info!(
|
||||
"autotune agent started (cycle={}s, cooldown={}s, cap={}/hr, min_recall={})",
|
||||
config.cycle_interval_secs, config.cooldown_between_trials_secs,
|
||||
config.max_trials_per_hour, config.min_recall,
|
||||
);
|
||||
let loop_inner = inner.clone();
|
||||
let loop_deps = deps.clone();
|
||||
let loop_config = config.clone();
|
||||
tokio::spawn(async move {
|
||||
run_loop(loop_config, loop_deps, trigger_rx, stop_rx, loop_inner).await;
|
||||
});
|
||||
} else {
|
||||
// Agent disabled — still drain the channel so sends don't back up.
|
||||
tokio::spawn(async move {
|
||||
let mut rx = trigger_rx;
|
||||
while rx.recv().await.is_some() {}
|
||||
});
|
||||
tracing::info!("autotune agent configured but disabled (set [agent].enabled=true)");
|
||||
}
|
||||
|
||||
AgentHandle { trigger_tx, inner }
|
||||
}
|
||||
|
||||
// -------- Main loop --------
|
||||
|
||||
async fn run_loop(
|
||||
config: AgentConfig,
|
||||
deps: AgentDeps,
|
||||
mut trigger_rx: mpsc::Receiver<TriggerEvent>,
|
||||
mut stop_rx: oneshot::Receiver<()>,
|
||||
inner: Arc<AgentInner>,
|
||||
) {
|
||||
let mut periodic = tokio::time::interval(std::time::Duration::from_secs(config.cycle_interval_secs));
|
||||
// First tick fires immediately — skip it so we don't double-fire on startup.
|
||||
periodic.tick().await;
|
||||
|
||||
loop {
|
||||
let event = tokio::select! {
|
||||
_ = &mut stop_rx => {
|
||||
tracing::info!("autotune agent: stop signal received");
|
||||
let mut s = inner.status.write().await;
|
||||
s.running = false;
|
||||
return;
|
||||
}
|
||||
maybe = trigger_rx.recv() => match maybe {
|
||||
Some(ev) => {
|
||||
let mut guard = inner.queue_len.lock().await;
|
||||
*guard = guard.saturating_sub(1);
|
||||
let mut s = inner.status.write().await;
|
||||
s.queue_depth = *guard;
|
||||
ev
|
||||
}
|
||||
None => {
|
||||
tracing::info!("autotune agent: trigger channel closed");
|
||||
return;
|
||||
}
|
||||
},
|
||||
_ = periodic.tick() => {
|
||||
// Periodic wake — pick an index with existing trials.
|
||||
// If nothing's been tuned yet there's nothing to propose.
|
||||
match pick_periodic_target(&deps).await {
|
||||
Some(idx) => TriggerEvent::periodic(idx),
|
||||
None => continue,
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Rate limit: count recent trials, skip if over budget.
|
||||
if over_rate_limit(&inner, config.max_trials_per_hour).await {
|
||||
record_event(&inner, "skipped_rate_limit", Some(&event.index_name),
|
||||
format!("hit cap of {}/hour", config.max_trials_per_hour)).await;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Per-index cooldown.
|
||||
if cooling_down(&inner, &event.index_name, config.cooldown_between_trials_secs).await {
|
||||
record_event(&inner, "skipped_cooldown", Some(&event.index_name),
|
||||
format!("last trial too recent (<{}s)", config.cooldown_between_trials_secs)).await;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Run one trial.
|
||||
match run_one_cycle(&event, &deps, config.min_recall).await {
|
||||
Ok(outcome) => {
|
||||
mark_recent_trial(&inner).await;
|
||||
{
|
||||
let mut s = inner.status.write().await;
|
||||
s.trials_run += 1;
|
||||
if outcome.promoted { s.promotions += 1; }
|
||||
}
|
||||
record_event(&inner, if outcome.promoted { "promoted" } else { "trial_completed" },
|
||||
Some(&event.index_name),
|
||||
format!("config=ec{}/es{} recall={:.3} p50={:.0}us {}",
|
||||
outcome.trial.config.ef_construction,
|
||||
outcome.trial.config.ef_search,
|
||||
outcome.trial.metrics.recall_at_k,
|
||||
outcome.trial.metrics.search_latency_p50_us,
|
||||
if outcome.promoted { "★ PROMOTED" } else { "" })).await;
|
||||
}
|
||||
Err(e) => {
|
||||
record_event(&inner, "trial_error", Some(&event.index_name), e).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Result of one cycle — ran a trial, maybe promoted it.
|
||||
struct CycleOutcome {
|
||||
trial: Trial,
|
||||
promoted: bool,
|
||||
}
|
||||
|
||||
/// Core cycle: propose → build → bench → record → maybe promote.
|
||||
async fn run_one_cycle(
|
||||
event: &TriggerEvent,
|
||||
deps: &AgentDeps,
|
||||
min_recall: f32,
|
||||
) -> Result<CycleOutcome, String> {
|
||||
// Read history.
|
||||
let history = deps.trial_journal.list(&event.index_name).await
|
||||
.map_err(|e| format!("read journal: {e}"))?;
|
||||
if history.is_empty() {
|
||||
return Err(format!(
|
||||
"no trials yet for '{}' — seed with at least one POST /hnsw/trial first",
|
||||
event.index_name,
|
||||
));
|
||||
}
|
||||
|
||||
// Current champion (if any) is the promoted config.
|
||||
let champion = deps.promotion_registry.get_current(&event.index_name).await;
|
||||
let champion_trial = champion.as_ref().and_then(|p| {
|
||||
history.iter().find(|t| t.id == p.trial_id).cloned()
|
||||
});
|
||||
|
||||
// Propose the next config.
|
||||
let Some(next_config) = propose_next_config(&history, champion_trial.as_ref()) else {
|
||||
return Err("proposer returned None — search space exhausted".into());
|
||||
};
|
||||
|
||||
// Validate bounds defensively.
|
||||
if !(10..=400).contains(&next_config.ef_construction) {
|
||||
return Err(format!("proposed ef_construction={} out of bounds", next_config.ef_construction));
|
||||
}
|
||||
if !(10..=200).contains(&next_config.ef_search) {
|
||||
return Err(format!("proposed ef_search={} out of bounds", next_config.ef_search));
|
||||
}
|
||||
|
||||
// Need a harness to measure. Use the most recent one in history.
|
||||
// (A future refinement: remember per-index "canonical harness" on
|
||||
// the index metadata. For now: latest wins.)
|
||||
let harness_name = history.last().unwrap().eval_set.clone();
|
||||
let mut harness_set = harness::EvalSet::load(&deps.store, &harness_name).await
|
||||
.map_err(|e| format!("load harness '{harness_name}': {e}"))?;
|
||||
|
||||
let embeddings = deps.embedding_cache.get_or_load(&event.index_name).await
|
||||
.map_err(|e| format!("embeddings: {e}"))?;
|
||||
|
||||
if !harness_set.ground_truth_built {
|
||||
harness::compute_ground_truth(&mut harness_set, &embeddings, &deps.ai_client).await
|
||||
.map_err(|e| format!("ground truth: {e}"))?;
|
||||
harness_set.save(&deps.store).await.ok();
|
||||
}
|
||||
|
||||
// Build + bench.
|
||||
let trial_id = Trial::new_id();
|
||||
let slot = format!("{}__{}", event.index_name, trial_id);
|
||||
let build = deps.hnsw_store
|
||||
.build_index_with_config(&slot, (*embeddings).clone(), &next_config)
|
||||
.await?;
|
||||
|
||||
let query_vectors: Vec<Vec<f32>> = harness_set.queries
|
||||
.iter().filter_map(|q| q.query_embedding.clone()).collect();
|
||||
let bench = deps.hnsw_store.bench_search(&slot, &query_vectors, harness_set.k).await?;
|
||||
|
||||
let mut recalls = Vec::with_capacity(harness_set.queries.len());
|
||||
for (q, hits) in harness_set.queries.iter().zip(bench.retrieved.iter()) {
|
||||
if let Some(gt) = &q.ground_truth {
|
||||
recalls.push(harness::recall_at_k(hits, gt, harness_set.k));
|
||||
}
|
||||
}
|
||||
let mean_recall = if recalls.is_empty() { 0.0 } else {
|
||||
recalls.iter().sum::<f32>() / recalls.len() as f32
|
||||
};
|
||||
|
||||
let mut lats = bench.latencies_us.clone();
|
||||
lats.sort_unstable_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||
let p = |pct: f32| -> f32 {
|
||||
if lats.is_empty() { return 0.0; }
|
||||
let idx = ((lats.len() as f32 - 1.0) * pct).round() as usize;
|
||||
lats[idx.min(lats.len() - 1)]
|
||||
};
|
||||
|
||||
let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
|
||||
let memory_bytes = (embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
|
||||
|
||||
let note = match &event.reason {
|
||||
TriggerReason::DatasetAppended { dataset } => format!("agent: dataset_appended({dataset})"),
|
||||
TriggerReason::Manual => "agent: manual".to_string(),
|
||||
TriggerReason::Periodic => "agent: periodic".to_string(),
|
||||
};
|
||||
|
||||
let trial = Trial {
|
||||
id: trial_id,
|
||||
index_name: event.index_name.clone(),
|
||||
eval_set: harness_set.name.clone(),
|
||||
config: next_config.clone(),
|
||||
metrics: TrialMetrics {
|
||||
build_time_secs: build.build_time_secs,
|
||||
search_latency_p50_us: p(0.50),
|
||||
search_latency_p95_us: p(0.95),
|
||||
search_latency_p99_us: p(0.99),
|
||||
recall_at_k: mean_recall,
|
||||
memory_bytes,
|
||||
vectors: build.vectors,
|
||||
eval_queries: harness_set.queries.len(),
|
||||
brute_force_latency_us: 0.0,
|
||||
},
|
||||
created_at: Utc::now(),
|
||||
note: Some(note),
|
||||
};
|
||||
deps.trial_journal.append(&trial).await.ok();
|
||||
deps.hnsw_store.drop(&slot).await;
|
||||
|
||||
// Promotion decision: the new trial must meet recall gate AND beat
|
||||
// the current champion (higher recall OR same recall + lower p50).
|
||||
let promoted = if trial.metrics.recall_at_k < min_recall {
|
||||
false
|
||||
} else {
|
||||
let beats = match &champion_trial {
|
||||
None => true, // no champion yet — anything passing the gate wins
|
||||
Some(c) => beats_champion(&trial, c),
|
||||
};
|
||||
if beats {
|
||||
let entry = PromotionEntry {
|
||||
config: trial.config.clone(),
|
||||
trial_id: trial.id.clone(),
|
||||
promoted_at: Utc::now(),
|
||||
promoted_by: "agent".to_string(),
|
||||
note: Some(format!(
|
||||
"auto-promote: recall={:.3} p50={:.0}us (was {:.3}/{:.0}us)",
|
||||
trial.metrics.recall_at_k, trial.metrics.search_latency_p50_us,
|
||||
champion_trial.as_ref().map(|t| t.metrics.recall_at_k).unwrap_or(0.0),
|
||||
champion_trial.as_ref().map(|t| t.metrics.search_latency_p50_us).unwrap_or(0.0),
|
||||
)),
|
||||
};
|
||||
deps.promotion_registry.promote(&event.index_name, entry).await.is_ok()
|
||||
} else {
|
||||
false
|
||||
}
|
||||
};
|
||||
|
||||
Ok(CycleOutcome { trial, promoted })
|
||||
}
|
||||
|
||||
/// Champion-beat test: strictly higher recall, OR equal recall with
|
||||
/// lower p50. Same rule as autotune::pick_winner — kept consistent so the
|
||||
/// agent and the synchronous autotune agree on what "better" means.
|
||||
fn beats_champion(candidate: &Trial, champion: &Trial) -> bool {
|
||||
if candidate.metrics.recall_at_k > champion.metrics.recall_at_k {
|
||||
return true;
|
||||
}
|
||||
if (candidate.metrics.recall_at_k - champion.metrics.recall_at_k).abs() < 1e-4
|
||||
&& candidate.metrics.search_latency_p50_us < champion.metrics.search_latency_p50_us {
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Propose the next HnswConfig given trial history and the current
|
||||
/// champion.
|
||||
///
|
||||
/// ============================================================
|
||||
/// J: THIS IS YOURS TO IMPLEMENT
|
||||
/// ============================================================
|
||||
///
|
||||
/// Inputs:
|
||||
/// - `history`: every trial ever run on this index, oldest first
|
||||
/// - `champion`: the currently-promoted trial, if any
|
||||
///
|
||||
/// Output:
|
||||
/// - `Some(HnswConfig)` with the config to try next
|
||||
/// - `None` if you decide there's nothing worth trying (agent logs
|
||||
/// "search space exhausted" and moves on)
|
||||
///
|
||||
/// Hard bounds the caller enforces: ef_construction ∈ [10, 400],
|
||||
/// ef_search ∈ [10, 200]. Stay inside those — configs outside get
|
||||
/// rejected and count as a wasted cycle.
|
||||
///
|
||||
/// Design options (pick one, or mix):
|
||||
///
|
||||
/// 1. LOCAL REFINEMENT (exploit-heavy):
|
||||
/// Sample near champion ± small delta. Converges fast, risks local
|
||||
/// minima. Good for "we know roughly where the sweet spot is."
|
||||
///
|
||||
/// 2. ε-GREEDY (mixed):
|
||||
/// With prob ε, random sample from full bounds (explore). Otherwise
|
||||
/// refinement around champion (exploit). ε=0.2 is a reasonable start.
|
||||
/// Good for long-running tune with no prior knowledge.
|
||||
///
|
||||
/// 3. COARSE→FINE (annealed):
|
||||
/// First N trials: wide random. Then shrink the neighborhood around
|
||||
/// champion as more trials accumulate. Mimics simulated annealing.
|
||||
///
|
||||
/// 4. DEDUP-AWARE:
|
||||
/// Whatever strategy, skip configs already in history. Prevents the
|
||||
/// agent from re-running the same (ec, es) pair twice.
|
||||
///
|
||||
/// A simple starter implementation is provided below (local refinement
|
||||
/// + dedup). Replace with your preferred strategy.
|
||||
pub fn propose_next_config(history: &[Trial], champion: Option<&Trial>) -> Option<HnswConfig> {
|
||||
// ε-greedy around the champion, dedup-aware.
|
||||
//
|
||||
// - With probability ε (≈0.25), sample a random config from the full
|
||||
// bounds. Keeps exploration alive so we don't get stuck hill-climbing
|
||||
// one axis.
|
||||
// - Otherwise: perturb the champion symmetrically on BOTH axes (not
|
||||
// just +20 / +40 like the starter did). Prefers small moves first
|
||||
// so recall stays near the current level.
|
||||
// - Always skip configs already in history — no point re-running.
|
||||
// - Deterministic per-history: RNG is seeded from history length so
|
||||
// the same journal state always proposes the same next config.
|
||||
// Makes tests + offline replay reproducible.
|
||||
|
||||
let base = champion
|
||||
.map(|t| t.config.clone())
|
||||
.or_else(|| history.last().map(|t| t.config.clone()))
|
||||
.unwrap_or_default();
|
||||
|
||||
let tried = |ec: usize, es: usize| -> bool {
|
||||
history.iter().any(|t|
|
||||
t.config.ef_construction == ec && t.config.ef_search == es
|
||||
)
|
||||
};
|
||||
let clamp = |ec: i32, es: i32| -> (usize, usize) {
|
||||
(ec.clamp(10, 400) as usize, es.clamp(10, 200) as usize)
|
||||
};
|
||||
|
||||
// Tiny xorshift — no rand crate dep. Seeded from history length so the
|
||||
// proposer is deterministic for a given journal state.
|
||||
let mut rng = (history.len() as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
|
||||
let mut next_u = || {
|
||||
rng ^= rng << 13; rng ^= rng >> 7; rng ^= rng << 17;
|
||||
rng
|
||||
};
|
||||
|
||||
for _attempt in 0..32 {
|
||||
let explore = (next_u() % 100) < 25; // ε = 0.25
|
||||
let (ec, es) = if explore {
|
||||
clamp(10 + (next_u() % 391) as i32, 10 + (next_u() % 191) as i32)
|
||||
} else {
|
||||
// Symmetric perturbation — signed steps on both axes.
|
||||
let dec = [-40, -20, -10, 10, 20, 40][(next_u() % 6) as usize];
|
||||
let des = [-20, -10, -5, 5, 10, 20][(next_u() % 6) as usize];
|
||||
clamp(base.ef_construction as i32 + dec, base.ef_search as i32 + des)
|
||||
};
|
||||
if !tried(ec, es) {
|
||||
return Some(HnswConfig { ef_construction: ec, ef_search: es, seed: Some(42) });
|
||||
}
|
||||
}
|
||||
None // 32 attempts all landed on duplicates — likely saturated
|
||||
}
|
||||
|
||||
// -------- Helpers --------
|
||||
|
||||
/// Find an index to poke on a periodic wake. Strategy: the one with the
|
||||
/// most recent trial. If nothing's been trialed yet, return None.
|
||||
async fn pick_periodic_target(deps: &AgentDeps) -> Option<String> {
|
||||
// `/agent` runs against any index that has a trial journal. We don't
|
||||
// have a "list all journals" helper, so we derive candidates from the
|
||||
// promotion registry (indexes with a human ever promoting are live).
|
||||
let promos = deps.promotion_registry.list_all().await.ok()?;
|
||||
// Prefer the one most recently promoted — it's the one a human cares
|
||||
// about right now.
|
||||
promos.into_iter()
|
||||
.filter_map(|f| f.current.map(|c| (f.index_name, c.promoted_at)))
|
||||
.max_by_key(|(_, at)| *at)
|
||||
.map(|(name, _)| name)
|
||||
}
|
||||
|
||||
async fn over_rate_limit(inner: &Arc<AgentInner>, cap: u32) -> bool {
|
||||
let cutoff = Utc::now() - chrono::Duration::hours(1);
|
||||
let ring = inner.recent_trials.lock().await;
|
||||
ring.iter().filter(|t| **t >= cutoff).count() as u32 >= cap
|
||||
}
|
||||
|
||||
async fn cooling_down(inner: &Arc<AgentInner>, _index: &str, cooldown_secs: u64) -> bool {
|
||||
// Minimal impl: gate on global most-recent trial rather than per-index.
|
||||
// Per-index cooldown would be easy to add — keep a HashMap<String, DateTime>
|
||||
// — but for Phase 16.2 MVP, global is fine. Ollama is the shared resource.
|
||||
let ring = inner.recent_trials.lock().await;
|
||||
if let Some(last) = ring.back() {
|
||||
let since = Utc::now().signed_duration_since(*last);
|
||||
return since < chrono::Duration::seconds(cooldown_secs as i64);
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
async fn mark_recent_trial(inner: &Arc<AgentInner>) {
|
||||
let mut ring = inner.recent_trials.lock().await;
|
||||
ring.push_back(Utc::now());
|
||||
// Keep bounded.
|
||||
while ring.len() > 256 {
|
||||
ring.pop_front();
|
||||
}
|
||||
}
|
||||
|
||||
async fn record_event(
|
||||
inner: &Arc<AgentInner>,
|
||||
kind: &str,
|
||||
index: Option<&str>,
|
||||
detail: String,
|
||||
) {
|
||||
tracing::info!("agent: {} {}{}", kind,
|
||||
index.map(|i| format!("[{i}] ")).unwrap_or_default(), detail);
|
||||
let mut s = inner.status.write().await;
|
||||
s.last_event = Some(AgentEvent {
|
||||
at: Utc::now(),
|
||||
kind: kind.to_string(),
|
||||
index_name: index.map(String::from),
|
||||
detail,
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn mk_trial(ec: usize, es: usize, recall: f32, p50: f32) -> Trial {
|
||||
Trial {
|
||||
id: format!("t-{ec}-{es}"),
|
||||
index_name: "test".into(),
|
||||
eval_set: "eval".into(),
|
||||
config: HnswConfig { ef_construction: ec, ef_search: es, seed: Some(42) },
|
||||
metrics: TrialMetrics {
|
||||
build_time_secs: 1.0,
|
||||
search_latency_p50_us: p50,
|
||||
search_latency_p95_us: p50 * 1.5,
|
||||
search_latency_p99_us: p50 * 2.0,
|
||||
recall_at_k: recall,
|
||||
memory_bytes: 0, vectors: 1000, eval_queries: 10,
|
||||
brute_force_latency_us: 100.0,
|
||||
},
|
||||
created_at: Utc::now(),
|
||||
note: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn propose_skips_duplicates() {
|
||||
let hist = vec![
|
||||
mk_trial(80, 30, 1.0, 500.0),
|
||||
mk_trial(100, 30, 1.0, 520.0), // ec+20
|
||||
];
|
||||
let next = propose_next_config(&hist, Some(&hist[0])).unwrap();
|
||||
// ec+20 is taken, so the proposer should skip it.
|
||||
assert!(next.ef_construction != 100 || next.ef_search != 30);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn beats_champion_strict_recall() {
|
||||
let champ = mk_trial(80, 30, 0.95, 500.0);
|
||||
let better_recall = mk_trial(80, 30, 0.99, 600.0);
|
||||
let worse_recall = mk_trial(80, 30, 0.90, 100.0);
|
||||
assert!(beats_champion(&better_recall, &champ));
|
||||
assert!(!beats_champion(&worse_recall, &champ));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn beats_champion_same_recall_lower_latency() {
|
||||
let champ = mk_trial(80, 30, 1.0, 500.0);
|
||||
let faster = mk_trial(60, 30, 1.0, 400.0);
|
||||
let slower = mk_trial(60, 30, 1.0, 600.0);
|
||||
assert!(beats_champion(&faster, &champ));
|
||||
assert!(!beats_champion(&slower, &champ));
|
||||
}
|
||||
}
|
||||
@@ -1,328 +0,0 @@
|
||||
//! Phase 16: Autotune — the agent loop for HNSW tuning.
|
||||
//!
|
||||
//! Given an index and a harness, run a grid of (ef_construction,
|
||||
//! ef_search) trials, pick the Pareto winner, and promote it.
|
||||
//!
|
||||
//! This is deliberately NOT a sophisticated optimizer — random-grid
|
||||
//! search is good enough when the config space is small (a dozen
|
||||
//! combinations) and the work per trial is seconds, not hours. Think
|
||||
//! of it as a "smart operator" rather than Bayesian optimization.
|
||||
//!
|
||||
//! Score: we want high recall and low p50 latency. The function below
|
||||
//! picks the Pareto-dominating trial, breaking ties by prefer-lower-
|
||||
//! build-time (cheaper to rebuild on data changes).
|
||||
//!
|
||||
//! Safeguards (PRD risk mitigation):
|
||||
//! - Never promote a trial with recall < `min_recall` (default 0.9)
|
||||
//! - Hard config bounds: ef_construction ∈ [10, 400], ef_search ∈ [10, 200]
|
||||
//! - Concurrency cap: one autotune per index at a time (tracked by
|
||||
//! JobTracker — caller is responsible for not firing in parallel
|
||||
//! for the same index)
|
||||
|
||||
use chrono::Utc;
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::sync::Arc;
|
||||
|
||||
use aibridge::client::AiClient;
|
||||
use catalogd::registry::Registry as CatalogRegistry;
|
||||
|
||||
use crate::embedding_cache::EmbeddingCache;
|
||||
use crate::harness;
|
||||
use crate::hnsw::HnswStore;
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::jobs::JobTracker;
|
||||
use crate::promotion::{PromotionEntry, PromotionRegistry};
|
||||
use crate::trial::{HnswConfig, Trial, TrialJournal, TrialMetrics};
|
||||
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct AutotuneRequest {
|
||||
pub index_name: String,
|
||||
pub harness: String,
|
||||
/// Minimum recall a trial must achieve to be eligible for promotion.
|
||||
/// Default 0.9 — below that, the tuning is not producing useful
|
||||
/// answers. Never promote a bad index.
|
||||
#[serde(default = "default_min_recall")]
|
||||
pub min_recall: f32,
|
||||
/// Override the default config grid (rarely needed). When omitted,
|
||||
/// a small curated grid runs.
|
||||
#[serde(default)]
|
||||
pub grid: Option<Vec<HnswConfig>>,
|
||||
}
|
||||
|
||||
fn default_min_recall() -> f32 { 0.9 }
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct AutotuneResult {
|
||||
pub index_name: String,
|
||||
pub trials_run: usize,
|
||||
pub trials_eligible: usize,
|
||||
pub winner: Option<WinnerReport>,
|
||||
pub promoted: bool,
|
||||
pub duration_secs: f32,
|
||||
pub skipped: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct WinnerReport {
|
||||
pub trial_id: String,
|
||||
pub config: HnswConfig,
|
||||
pub recall_at_k: f32,
|
||||
pub search_latency_p50_us: f32,
|
||||
pub build_time_secs: f32,
|
||||
}
|
||||
|
||||
/// Default exploration grid. Covers the practical range of settings
|
||||
/// from "fast build, OK recall" to "thorough build, great recall".
|
||||
fn default_grid() -> Vec<HnswConfig> {
|
||||
vec![
|
||||
HnswConfig { ef_construction: 20, ef_search: 30, seed: Some(42) },
|
||||
HnswConfig { ef_construction: 40, ef_search: 30, seed: Some(42) },
|
||||
HnswConfig { ef_construction: 80, ef_search: 30, seed: Some(42) },
|
||||
HnswConfig { ef_construction: 80, ef_search: 60, seed: Some(42) },
|
||||
HnswConfig { ef_construction: 160, ef_search: 30, seed: Some(42) },
|
||||
]
|
||||
}
|
||||
|
||||
/// Config-space bounds. Rejects anything outside these before running a
|
||||
/// trial — defends against an agent getting creative with absurd values.
|
||||
fn sanitize_grid(grid: Vec<HnswConfig>) -> (Vec<HnswConfig>, Vec<String>) {
|
||||
let mut ok = Vec::new();
|
||||
let mut rejected = Vec::new();
|
||||
for cfg in grid {
|
||||
if !(10..=400).contains(&cfg.ef_construction) {
|
||||
rejected.push(format!(
|
||||
"ef_construction={} out of bounds [10,400]", cfg.ef_construction,
|
||||
));
|
||||
continue;
|
||||
}
|
||||
if !(10..=200).contains(&cfg.ef_search) {
|
||||
rejected.push(format!(
|
||||
"ef_search={} out of bounds [10,200]", cfg.ef_search,
|
||||
));
|
||||
continue;
|
||||
}
|
||||
ok.push(cfg);
|
||||
}
|
||||
(ok, rejected)
|
||||
}
|
||||
|
||||
/// Pareto winner: highest recall, then lowest p50 latency among
|
||||
/// eligible trials (recall >= min_recall).
|
||||
fn pick_winner(trials: &[Trial], min_recall: f32) -> Option<&Trial> {
|
||||
trials
|
||||
.iter()
|
||||
.filter(|t| t.metrics.recall_at_k >= min_recall)
|
||||
.min_by(|a, b| {
|
||||
// Compare by (−recall, p50_latency) so minimum is the
|
||||
// best trial: higher recall wins, then lower latency.
|
||||
let a_key = (-a.metrics.recall_at_k, a.metrics.search_latency_p50_us);
|
||||
let b_key = (-b.metrics.recall_at_k, b.metrics.search_latency_p50_us);
|
||||
a_key.partial_cmp(&b_key).unwrap_or(std::cmp::Ordering::Equal)
|
||||
})
|
||||
}
|
||||
|
||||
/// Run the autotune loop. Synchronous — caller can wrap in tokio::spawn
|
||||
/// to make it fire-and-forget. Uses the existing trial flow so every
|
||||
/// autotune step lands in the trial journal.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn run_autotune(
|
||||
req: AutotuneRequest,
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
catalog: &CatalogRegistry,
|
||||
ai_client: &AiClient,
|
||||
embedding_cache: &EmbeddingCache,
|
||||
hnsw_store: &HnswStore,
|
||||
index_registry: &IndexRegistry,
|
||||
trial_journal: &TrialJournal,
|
||||
promotion_registry: &PromotionRegistry,
|
||||
_job_tracker: &JobTracker,
|
||||
) -> Result<AutotuneResult, String> {
|
||||
let t0 = std::time::Instant::now();
|
||||
|
||||
// Sanity: the index has to exist.
|
||||
if index_registry.get(&req.index_name).await.is_none() {
|
||||
return Err(format!("index not found: {}", req.index_name));
|
||||
}
|
||||
let _ = catalog; // reserved for future audit emission
|
||||
|
||||
// Load the harness once, compute ground truth once.
|
||||
let mut harness_set = harness::EvalSet::load(store, &req.harness)
|
||||
.await
|
||||
.map_err(|e| format!("load harness: {e}"))?;
|
||||
|
||||
let embeddings = embedding_cache
|
||||
.get_or_load(&req.index_name)
|
||||
.await
|
||||
.map_err(|e| format!("embeddings: {e}"))?;
|
||||
|
||||
if !harness_set.ground_truth_built {
|
||||
harness::compute_ground_truth(&mut harness_set, &embeddings, ai_client)
|
||||
.await
|
||||
.map_err(|e| format!("ground truth: {e}"))?;
|
||||
harness_set.save(store).await.ok();
|
||||
}
|
||||
|
||||
let (grid, rejected) = sanitize_grid(req.grid.clone().unwrap_or_else(default_grid));
|
||||
if grid.is_empty() {
|
||||
return Err("all configs rejected by bounds check".into());
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"autotune '{}': {} trial configs, min_recall={}",
|
||||
req.index_name, grid.len(), req.min_recall,
|
||||
);
|
||||
|
||||
// Run each trial through the existing pipeline so every run lands
|
||||
// in the trial journal and contributes to the long-term tuning
|
||||
// memory. This is the llms3.com-style "trials are data" pattern.
|
||||
let mut run_trials: Vec<Trial> = Vec::with_capacity(grid.len());
|
||||
for config in &grid {
|
||||
let trial = run_single_trial(
|
||||
&req.index_name,
|
||||
&harness_set,
|
||||
&embeddings,
|
||||
config,
|
||||
hnsw_store,
|
||||
trial_journal,
|
||||
ai_client,
|
||||
).await;
|
||||
match trial {
|
||||
Ok(t) => run_trials.push(t),
|
||||
Err(e) => tracing::warn!("autotune: trial ec={} es={} failed: {e}",
|
||||
config.ef_construction, config.ef_search),
|
||||
}
|
||||
}
|
||||
|
||||
let trials_eligible = run_trials
|
||||
.iter()
|
||||
.filter(|t| t.metrics.recall_at_k >= req.min_recall)
|
||||
.count();
|
||||
|
||||
let winner = pick_winner(&run_trials, req.min_recall);
|
||||
let mut promoted = false;
|
||||
|
||||
let winner_report = if let Some(w) = winner {
|
||||
let entry = PromotionEntry {
|
||||
config: w.config.clone(),
|
||||
trial_id: w.id.clone(),
|
||||
promoted_at: Utc::now(),
|
||||
promoted_by: "autotune".to_string(),
|
||||
note: Some(format!(
|
||||
"autotune winner: recall={:.3} p50={:.0}us",
|
||||
w.metrics.recall_at_k, w.metrics.search_latency_p50_us,
|
||||
)),
|
||||
};
|
||||
if let Err(e) = promotion_registry.promote(&req.index_name, entry).await {
|
||||
tracing::warn!("autotune: promote failed: {e}");
|
||||
} else {
|
||||
promoted = true;
|
||||
}
|
||||
Some(WinnerReport {
|
||||
trial_id: w.id.clone(),
|
||||
config: w.config.clone(),
|
||||
recall_at_k: w.metrics.recall_at_k,
|
||||
search_latency_p50_us: w.metrics.search_latency_p50_us,
|
||||
build_time_secs: w.metrics.build_time_secs,
|
||||
})
|
||||
} else {
|
||||
tracing::warn!(
|
||||
"autotune '{}': no trial met min_recall={}; not promoting",
|
||||
req.index_name, req.min_recall,
|
||||
);
|
||||
None
|
||||
};
|
||||
|
||||
Ok(AutotuneResult {
|
||||
index_name: req.index_name,
|
||||
trials_run: run_trials.len(),
|
||||
trials_eligible,
|
||||
winner: winner_report,
|
||||
promoted,
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
skipped: rejected,
|
||||
})
|
||||
}
|
||||
|
||||
/// Run one trial and journal it. Returns the recorded Trial.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn run_single_trial(
|
||||
index_name: &str,
|
||||
harness_set: &harness::EvalSet,
|
||||
embeddings: &Arc<Vec<crate::store::StoredEmbedding>>,
|
||||
config: &HnswConfig,
|
||||
hnsw_store: &HnswStore,
|
||||
trial_journal: &TrialJournal,
|
||||
_ai_client: &AiClient,
|
||||
) -> Result<Trial, String> {
|
||||
let trial_id = Trial::new_id();
|
||||
let slot = format!("{}__{}", index_name, trial_id);
|
||||
|
||||
let build = hnsw_store
|
||||
.build_index_with_config(&slot, (**embeddings).clone(), config)
|
||||
.await?;
|
||||
|
||||
let query_vectors: Vec<Vec<f32>> = harness_set
|
||||
.queries
|
||||
.iter()
|
||||
.filter_map(|q| q.query_embedding.clone())
|
||||
.collect();
|
||||
let bench = hnsw_store
|
||||
.bench_search(&slot, &query_vectors, harness_set.k)
|
||||
.await?;
|
||||
|
||||
let mut recalls = Vec::with_capacity(harness_set.queries.len());
|
||||
for (q, hits) in harness_set.queries.iter().zip(bench.retrieved.iter()) {
|
||||
if let Some(gt) = &q.ground_truth {
|
||||
recalls.push(harness::recall_at_k(hits, gt, harness_set.k));
|
||||
}
|
||||
}
|
||||
let mean_recall = if recalls.is_empty() {
|
||||
0.0
|
||||
} else {
|
||||
recalls.iter().sum::<f32>() / recalls.len() as f32
|
||||
};
|
||||
|
||||
let mut lats = bench.latencies_us.clone();
|
||||
lats.sort_unstable_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||
let p = |pct: f32| -> f32 {
|
||||
if lats.is_empty() { return 0.0; }
|
||||
let idx = ((lats.len() as f32 - 1.0) * pct).round() as usize;
|
||||
lats[idx.min(lats.len() - 1)]
|
||||
};
|
||||
|
||||
let brute_us = if let Some(qv) = query_vectors.first() {
|
||||
let t = std::time::Instant::now();
|
||||
let _ = harness::brute_force_top_k(qv, embeddings, harness_set.k);
|
||||
t.elapsed().as_micros() as f32
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
let dims = embeddings.first().map(|e| e.vector.len()).unwrap_or(0);
|
||||
let memory_bytes =
|
||||
(embeddings.len() * dims * std::mem::size_of::<f32>() + embeddings.len() * 128) as u64;
|
||||
|
||||
let trial = Trial {
|
||||
id: trial_id.clone(),
|
||||
index_name: index_name.to_string(),
|
||||
eval_set: harness_set.name.clone(),
|
||||
config: config.clone(),
|
||||
metrics: TrialMetrics {
|
||||
build_time_secs: build.build_time_secs,
|
||||
search_latency_p50_us: p(0.50),
|
||||
search_latency_p95_us: p(0.95),
|
||||
search_latency_p99_us: p(0.99),
|
||||
recall_at_k: mean_recall,
|
||||
memory_bytes,
|
||||
vectors: build.vectors,
|
||||
eval_queries: harness_set.queries.len(),
|
||||
brute_force_latency_us: brute_us,
|
||||
},
|
||||
created_at: Utc::now(),
|
||||
note: Some("autotune".to_string()),
|
||||
};
|
||||
let _ = trial_journal.append(&trial).await;
|
||||
let _ = hnsw_store.drop(&slot).await;
|
||||
Ok(trial)
|
||||
}
|
||||
+23
-115
@@ -15,28 +15,9 @@ pub struct TextChunk {
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
/// Round `idx` UP to the nearest UTF-8 char boundary in `s`.
|
||||
/// Needed because `&str[a..b]` panics if `a` or `b` lands mid-character —
|
||||
/// and any text imported from Postgres/user data will contain multi-byte
|
||||
/// chars like `\u{202f}` (narrow no-break space), em-dashes, etc.
|
||||
fn ceil_char_boundary(s: &str, mut idx: usize) -> usize {
|
||||
if idx >= s.len() { return s.len(); }
|
||||
while idx <= s.len() && !s.is_char_boundary(idx) {
|
||||
idx += 1;
|
||||
}
|
||||
idx
|
||||
}
|
||||
|
||||
/// Split text into overlapping chunks.
|
||||
/// - `chunk_size`: target bytes per chunk (UTF-8 boundary-respecting)
|
||||
/// - `overlap`: bytes of overlap between consecutive chunks
|
||||
///
|
||||
/// Safety contract:
|
||||
/// - Never slices at a byte that isn't a UTF-8 char boundary
|
||||
/// - Always advances forward (monotonic `start`) — no infinite loops on
|
||||
/// pathological inputs
|
||||
/// - Bounded iteration — caps total chunks at `text.len()` to prevent
|
||||
/// runaway allocation on degenerate text/parameter combinations
|
||||
/// - `chunk_size`: target characters per chunk (not tokens — chars are a good proxy)
|
||||
/// - `overlap`: characters of overlap between consecutive chunks
|
||||
pub fn chunk_text(
|
||||
text: &str,
|
||||
source: &str,
|
||||
@@ -59,78 +40,46 @@ pub fn chunk_text(
|
||||
}];
|
||||
}
|
||||
|
||||
// Work with char indices, not byte indices, to handle Unicode safely
|
||||
let chars: Vec<char> = text.chars().collect();
|
||||
let char_len = chars.len();
|
||||
let mut chunks = Vec::new();
|
||||
let mut start = 0;
|
||||
let mut start = 0usize; // char index
|
||||
let mut idx = 0u32;
|
||||
|
||||
// Safety cap: a well-formed chunk_size advances `start` by at least
|
||||
// `chunk_size - overlap` each iteration. Even in degenerate cases we
|
||||
// should never emit more chunks than there are bytes of text.
|
||||
let max_chunks = text.len();
|
||||
let mut iterations = 0;
|
||||
while start < char_len {
|
||||
let end = (start + chunk_size).min(char_len);
|
||||
let chunk_str: String = chars[start..end].iter().collect();
|
||||
|
||||
while start < text.len() {
|
||||
iterations += 1;
|
||||
if iterations > max_chunks {
|
||||
tracing::warn!(
|
||||
"chunker: exceeded safety cap ({}) on doc '{}' of length {} — stopping",
|
||||
max_chunks, doc_id, text.len(),
|
||||
);
|
||||
break;
|
||||
}
|
||||
|
||||
// UTF-8 safe upper bound: move forward to a char boundary.
|
||||
let end = ceil_char_boundary(text, (start + chunk_size).min(text.len()));
|
||||
|
||||
let chunk_text_slice = &text[start..end];
|
||||
let actual_end = if end < text.len() {
|
||||
// Look for last sentence boundary inside the slice.
|
||||
if let Some(pos) = chunk_text_slice.rfind(". ") {
|
||||
start + pos + 2
|
||||
} else if let Some(pos) = chunk_text_slice.rfind('\n') {
|
||||
start + pos + 1
|
||||
} else if let Some(pos) = chunk_text_slice.rfind(' ') {
|
||||
start + pos + 1
|
||||
let actual_end = if end < char_len {
|
||||
// Look for last sentence boundary
|
||||
if let Some(pos) = chunk_str.rfind(". ") {
|
||||
start + chunk_str[..pos + 2].chars().count()
|
||||
} else if let Some(pos) = chunk_str.rfind('\n') {
|
||||
start + chunk_str[..pos + 1].chars().count()
|
||||
} else if let Some(pos) = chunk_str.rfind(' ') {
|
||||
start + chunk_str[..pos + 1].chars().count()
|
||||
} else {
|
||||
end
|
||||
}
|
||||
} else {
|
||||
end
|
||||
};
|
||||
// All the rfind'd positions are at char boundaries (they match
|
||||
// ASCII byte patterns), so actual_end should already be safe.
|
||||
// Defensive belt-and-suspenders: round up anyway.
|
||||
let actual_end = ceil_char_boundary(text, actual_end);
|
||||
|
||||
let chunk = text[start..actual_end].trim();
|
||||
if !chunk.is_empty() {
|
||||
let chunk: String = chars[start..actual_end].iter().collect();
|
||||
let trimmed = chunk.trim();
|
||||
if !trimmed.is_empty() {
|
||||
chunks.push(TextChunk {
|
||||
source: source.to_string(),
|
||||
doc_id: doc_id.to_string(),
|
||||
chunk_idx: idx,
|
||||
text: chunk.to_string(),
|
||||
text: trimmed.to_string(),
|
||||
});
|
||||
idx += 1;
|
||||
}
|
||||
|
||||
// Advance with overlap — must strictly progress to avoid infinite
|
||||
// loops on degenerate inputs (e.g. chunk_size ≤ overlap, or
|
||||
// boundary-finding returning a position at `start`).
|
||||
if actual_end >= text.len() {
|
||||
break;
|
||||
}
|
||||
let tentative = if actual_end > overlap { actual_end - overlap } else { actual_end };
|
||||
let next_start = ceil_char_boundary(text, tentative);
|
||||
if next_start <= start {
|
||||
// No progress — force forward one char to guarantee termination.
|
||||
let mut forced = start + 1;
|
||||
while forced < text.len() && !text.is_char_boundary(forced) {
|
||||
forced += 1;
|
||||
}
|
||||
start = forced;
|
||||
} else {
|
||||
start = next_start;
|
||||
}
|
||||
if actual_end >= char_len { break; }
|
||||
start = if actual_end > overlap { actual_end - overlap } else { actual_end };
|
||||
}
|
||||
|
||||
chunks
|
||||
@@ -180,45 +129,4 @@ mod tests {
|
||||
assert_eq!(chunks[0].source, "candidates");
|
||||
assert_eq!(chunks[0].doc_id, "CAND-001");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handles_multibyte_utf8_at_chunk_boundary() {
|
||||
// Regression: multi-byte unicode (narrow no-break space \u{202f},
|
||||
// 3 bytes) landing inside a chunk-size window used to panic with
|
||||
// "byte index N is not a char boundary". Found on pg-imported
|
||||
// data that contained editorial unicode punctuation.
|
||||
let text = "**Claim 1**\u{202f}\u{2013} *Each large language model has its own personality \
|
||||
and way of interacting.*\u{202f}\u{2013}\u{202f}Verdict:\u{202f}VERIFIED. \
|
||||
Reasoning: widely accepted observation that different models exhibit distinct response \
|
||||
styles stemming from variations in training data.".repeat(5);
|
||||
let chunks = chunk_text(&text, "test", "unicode-1", 500, 50);
|
||||
// Must not panic, must produce at least one chunk.
|
||||
assert!(!chunks.is_empty());
|
||||
// All chunks must be valid UTF-8 substrings of the original.
|
||||
for chunk in &chunks {
|
||||
assert!(!chunk.text.is_empty());
|
||||
assert!(text.contains(&chunk.text));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_infinite_loop_on_no_spaces() {
|
||||
// Long text with no spaces, sentences, or newlines — chunker must
|
||||
// still advance via the hard `end` fallback.
|
||||
let text = "a".repeat(5000);
|
||||
let chunks = chunk_text(&text, "test", "noSpace", 500, 50);
|
||||
assert!(!chunks.is_empty());
|
||||
// Expected ~5000/450 chunks (with overlap=50 → step=450)
|
||||
assert!(chunks.len() < 50, "chunk count {} seems excessive", chunks.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_infinite_loop_on_degenerate_params() {
|
||||
// chunk_size equals overlap — previously would have made zero
|
||||
// forward progress. Our safety forced-advance must catch this.
|
||||
let text = "a b c d e f g h i j k l m n o p q r s t u v w x y z".repeat(20);
|
||||
let chunks = chunk_text(&text, "test", "degen", 100, 100);
|
||||
assert!(!chunks.is_empty());
|
||||
assert!(chunks.len() <= text.len(), "runaway chunk count {}", chunks.len());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
/// In-memory cache for StoredEmbedding vectors.
|
||||
///
|
||||
/// Rationale: loading 100K embeddings from Parquet takes ~2-5s. When an AI agent
|
||||
/// iterates on HNSW parameters, each trial would repeat that cost. The cache
|
||||
/// pins embeddings in memory so trials reuse them.
|
||||
///
|
||||
/// This is a pure performance layer — the Parquet file is still the source of
|
||||
/// truth (ADR-008). Eviction is safe; worst case is one slow reload.
|
||||
|
||||
use object_store::ObjectStore;
|
||||
use serde::Serialize;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::store::{self, StoredEmbedding};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct EmbeddingCache {
|
||||
store: Arc<dyn ObjectStore>,
|
||||
cache: Arc<RwLock<HashMap<String, Arc<Vec<StoredEmbedding>>>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct CacheEntry {
|
||||
pub index_name: String,
|
||||
pub vectors: usize,
|
||||
pub dimensions: usize,
|
||||
pub memory_bytes: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct CacheStats {
|
||||
pub entries: Vec<CacheEntry>,
|
||||
pub total_memory_bytes: u64,
|
||||
}
|
||||
|
||||
impl EmbeddingCache {
|
||||
pub fn new(store: Arc<dyn ObjectStore>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
cache: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Return cached embeddings, loading from object storage on first request.
|
||||
pub async fn get_or_load(
|
||||
&self,
|
||||
index_name: &str,
|
||||
) -> Result<Arc<Vec<StoredEmbedding>>, String> {
|
||||
if let Some(cached) = self.cache.read().await.get(index_name) {
|
||||
return Ok(cached.clone());
|
||||
}
|
||||
|
||||
// Load under a write lock so concurrent callers only hit disk once.
|
||||
let mut guard = self.cache.write().await;
|
||||
if let Some(cached) = guard.get(index_name) {
|
||||
return Ok(cached.clone());
|
||||
}
|
||||
tracing::info!("embedding_cache: loading '{index_name}' from object storage");
|
||||
let t0 = std::time::Instant::now();
|
||||
let loaded = store::load_embeddings(&self.store, index_name).await?;
|
||||
let n = loaded.len();
|
||||
let arc = Arc::new(loaded);
|
||||
guard.insert(index_name.to_string(), arc.clone());
|
||||
tracing::info!(
|
||||
"embedding_cache: loaded '{index_name}' — {n} vectors in {:.2}s",
|
||||
t0.elapsed().as_secs_f32()
|
||||
);
|
||||
Ok(arc)
|
||||
}
|
||||
|
||||
pub async fn evict(&self, index_name: &str) -> bool {
|
||||
self.cache.write().await.remove(index_name).is_some()
|
||||
}
|
||||
|
||||
pub async fn stats(&self) -> CacheStats {
|
||||
let cache = self.cache.read().await;
|
||||
let mut entries = Vec::with_capacity(cache.len());
|
||||
let mut total: u64 = 0;
|
||||
for (name, embs) in cache.iter() {
|
||||
let dims = embs.first().map(|e| e.vector.len()).unwrap_or(0);
|
||||
// Rough estimate: vector data + chunk_text + metadata overhead.
|
||||
let vec_bytes = (embs.len() * dims * std::mem::size_of::<f32>()) as u64;
|
||||
let text_bytes: u64 = embs.iter().map(|e| e.chunk_text.len() as u64).sum();
|
||||
let overhead = (embs.len() * 128) as u64; // strings + struct overhead
|
||||
let mem = vec_bytes + text_bytes + overhead;
|
||||
total += mem;
|
||||
entries.push(CacheEntry {
|
||||
index_name: name.clone(),
|
||||
vectors: embs.len(),
|
||||
dimensions: dims,
|
||||
memory_bytes: mem,
|
||||
});
|
||||
}
|
||||
CacheStats { entries, total_memory_bytes: total }
|
||||
}
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
/// Eval harness for HNSW tuning.
|
||||
///
|
||||
/// An EvalSet is a named collection of queries against a specific vector index.
|
||||
/// Each query has a query_text and (optionally pre-computed) ground_truth
|
||||
/// doc_ids — the "correct" top-k results that brute-force cosine returns.
|
||||
/// HNSW trials are scored by how well they recreate that top-k (recall@k).
|
||||
///
|
||||
/// Storage: `_hnsw_evals/{name}.json` as a single JSON document. Small.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::sync::Arc;
|
||||
|
||||
use aibridge::client::{AiClient, EmbedRequest};
|
||||
use storaged::ops;
|
||||
|
||||
use crate::store::StoredEmbedding;
|
||||
|
||||
/// A single eval query with optional pre-computed ground truth.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EvalQuery {
|
||||
pub id: String,
|
||||
pub query_text: String,
|
||||
/// Ordered list of doc_ids that brute-force returns as top-k.
|
||||
/// `None` means the ground truth hasn't been computed yet; `compute_ground_truth`
|
||||
/// fills it in against the current embedding set.
|
||||
#[serde(default)]
|
||||
pub ground_truth: Option<Vec<String>>,
|
||||
/// Optional pre-computed query embedding. Filling this avoids re-embedding
|
||||
/// the same query on every trial.
|
||||
#[serde(default)]
|
||||
pub query_embedding: Option<Vec<f32>>,
|
||||
}
|
||||
|
||||
/// A named eval set tied to a specific vector index.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EvalSet {
|
||||
pub name: String,
|
||||
pub index_name: String,
|
||||
pub k: usize, // top-k used for recall calculation
|
||||
pub queries: Vec<EvalQuery>,
|
||||
pub created_at: DateTime<Utc>,
|
||||
pub ground_truth_built: bool,
|
||||
}
|
||||
|
||||
impl EvalSet {
|
||||
pub fn new(name: &str, index_name: &str, k: usize) -> Self {
|
||||
Self {
|
||||
name: name.to_string(),
|
||||
index_name: index_name.to_string(),
|
||||
k,
|
||||
queries: Vec::new(),
|
||||
created_at: Utc::now(),
|
||||
ground_truth_built: false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn storage_key(&self) -> String {
|
||||
format!("_hnsw_evals/{}.json", self.name)
|
||||
}
|
||||
|
||||
pub async fn save(&self, store: &Arc<dyn ObjectStore>) -> Result<(), String> {
|
||||
let json = serde_json::to_vec_pretty(self).map_err(|e| e.to_string())?;
|
||||
ops::put(store, &self.storage_key(), json.into()).await
|
||||
}
|
||||
|
||||
pub async fn load(store: &Arc<dyn ObjectStore>, name: &str) -> Result<Self, String> {
|
||||
let key = format!("_hnsw_evals/{}.json", name);
|
||||
let data = ops::get(store, &key).await?;
|
||||
serde_json::from_slice(&data).map_err(|e| format!("parse eval set: {e}"))
|
||||
}
|
||||
|
||||
pub async fn list(store: &Arc<dyn ObjectStore>) -> Result<Vec<String>, String> {
|
||||
let keys = ops::list(store, Some("_hnsw_evals/")).await?;
|
||||
Ok(keys
|
||||
.into_iter()
|
||||
.filter(|k| k.ends_with(".json"))
|
||||
.map(|k| {
|
||||
k.trim_start_matches("_hnsw_evals/")
|
||||
.trim_end_matches(".json")
|
||||
.to_string()
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
|
||||
/// Cosine similarity for two same-length f32 slices.
|
||||
fn cosine(a: &[f32], b: &[f32]) -> f32 {
|
||||
let mut dot = 0.0f32;
|
||||
let mut na = 0.0f32;
|
||||
let mut nb = 0.0f32;
|
||||
for i in 0..a.len() {
|
||||
dot += a[i] * b[i];
|
||||
na += a[i] * a[i];
|
||||
nb += b[i] * b[i];
|
||||
}
|
||||
if na == 0.0 || nb == 0.0 {
|
||||
return 0.0;
|
||||
}
|
||||
dot / (na.sqrt() * nb.sqrt())
|
||||
}
|
||||
|
||||
/// Brute-force top-k search for a single query against all embeddings.
|
||||
/// This is the ground-truth oracle that HNSW trials must approximate.
|
||||
pub fn brute_force_top_k(
|
||||
query: &[f32],
|
||||
embeddings: &[StoredEmbedding],
|
||||
k: usize,
|
||||
) -> Vec<String> {
|
||||
let mut scored: Vec<(f32, usize)> = embeddings
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, e)| (cosine(query, &e.vector), i))
|
||||
.collect();
|
||||
// Partial sort — we only need top-k.
|
||||
scored.sort_unstable_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal));
|
||||
scored
|
||||
.into_iter()
|
||||
.take(k)
|
||||
.map(|(_, i)| embeddings[i].doc_id.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Recall@k — fraction of ground-truth doc_ids that appear in predicted.
|
||||
pub fn recall_at_k(predicted: &[String], ground_truth: &[String], k: usize) -> f32 {
|
||||
if ground_truth.is_empty() || k == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
let gt_set: std::collections::HashSet<&String> =
|
||||
ground_truth.iter().take(k).collect();
|
||||
let hits = predicted
|
||||
.iter()
|
||||
.take(k)
|
||||
.filter(|d| gt_set.contains(d))
|
||||
.count();
|
||||
hits as f32 / gt_set.len() as f32
|
||||
}
|
||||
|
||||
/// Populate query_embedding and ground_truth for every query that lacks them.
|
||||
pub async fn compute_ground_truth(
|
||||
eval: &mut EvalSet,
|
||||
embeddings: &[StoredEmbedding],
|
||||
ai_client: &AiClient,
|
||||
) -> Result<(), String> {
|
||||
let need_embed: Vec<(usize, String)> = eval
|
||||
.queries
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, q)| q.query_embedding.is_none())
|
||||
.map(|(i, q)| (i, q.query_text.clone()))
|
||||
.collect();
|
||||
|
||||
if !need_embed.is_empty() {
|
||||
// Embed in one batch to keep things simple; for very large eval sets
|
||||
// we'd batch this in chunks of 32.
|
||||
let texts: Vec<String> = need_embed.iter().map(|(_, t)| t.clone()).collect();
|
||||
let resp = ai_client
|
||||
.embed(EmbedRequest { texts, model: None })
|
||||
.await
|
||||
.map_err(|e| format!("embed queries: {e}"))?;
|
||||
for ((idx, _), vec) in need_embed.iter().zip(resp.embeddings.iter()) {
|
||||
let v: Vec<f32> = vec.iter().map(|&x| x as f32).collect();
|
||||
eval.queries[*idx].query_embedding = Some(v);
|
||||
}
|
||||
}
|
||||
|
||||
for q in eval.queries.iter_mut() {
|
||||
if q.ground_truth.is_some() {
|
||||
continue;
|
||||
}
|
||||
let emb = q.query_embedding.as_ref().ok_or("missing embedding")?;
|
||||
q.ground_truth = Some(brute_force_top_k(emb, embeddings, eval.k));
|
||||
}
|
||||
eval.ground_truth_built = true;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Auto-generate a synthetic eval set by sampling every Nth chunk's text as
|
||||
/// its own query. Useful for a quick-start eval when the user doesn't have
|
||||
/// real natural-language queries yet.
|
||||
pub fn synthetic_from_chunks(
|
||||
eval_name: &str,
|
||||
index_name: &str,
|
||||
embeddings: &[StoredEmbedding],
|
||||
sample_count: usize,
|
||||
k: usize,
|
||||
) -> EvalSet {
|
||||
let n = embeddings.len();
|
||||
let sample_count = sample_count.min(n);
|
||||
let stride = (n / sample_count.max(1)).max(1);
|
||||
|
||||
let mut queries = Vec::with_capacity(sample_count);
|
||||
for i in 0..sample_count {
|
||||
let idx = (i * stride).min(n - 1);
|
||||
let chunk = &embeddings[idx];
|
||||
// Use the first ~200 chars of the chunk as the "query" — it should find
|
||||
// itself and nearby chunks as top results.
|
||||
let query_text: String = chunk.chunk_text.chars().take(200).collect();
|
||||
queries.push(EvalQuery {
|
||||
id: format!("syn-{}", i),
|
||||
query_text,
|
||||
ground_truth: None,
|
||||
query_embedding: None,
|
||||
});
|
||||
}
|
||||
|
||||
EvalSet {
|
||||
name: eval_name.to_string(),
|
||||
index_name: index_name.to_string(),
|
||||
k,
|
||||
queries,
|
||||
created_at: Utc::now(),
|
||||
ground_truth_built: false,
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,6 @@ use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::store::StoredEmbedding;
|
||||
use crate::trial::HnswConfig;
|
||||
|
||||
/// A vector point for HNSW — wraps f32 slice with cosine distance.
|
||||
#[derive(Clone)]
|
||||
@@ -63,32 +62,18 @@ impl HnswStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Build an HNSW index from stored embeddings with default config.
|
||||
/// Build an HNSW index from stored embeddings.
|
||||
pub async fn build_index(
|
||||
&self,
|
||||
index_name: &str,
|
||||
embeddings: Vec<StoredEmbedding>,
|
||||
) -> Result<BuildStats, String> {
|
||||
self.build_index_with_config(index_name, embeddings, &HnswConfig::default()).await
|
||||
}
|
||||
|
||||
/// Build an HNSW index from stored embeddings with explicit config.
|
||||
/// Used by the trial system — each trial calls this with different params.
|
||||
pub async fn build_index_with_config(
|
||||
&self,
|
||||
index_name: &str,
|
||||
embeddings: Vec<StoredEmbedding>,
|
||||
config: &HnswConfig,
|
||||
) -> Result<BuildStats, String> {
|
||||
let n = embeddings.len();
|
||||
if n == 0 {
|
||||
return Err("no embeddings to index".into());
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"building HNSW '{}': {} vectors, ef_construction={} ef_search={} seed={:?}",
|
||||
index_name, n, config.ef_construction, config.ef_search, config.seed,
|
||||
);
|
||||
tracing::info!("building HNSW index '{}' from {} vectors", index_name, n);
|
||||
let start = std::time::Instant::now();
|
||||
|
||||
// Separate points and metadata
|
||||
@@ -107,17 +92,14 @@ impl HnswStore {
|
||||
values.push(i);
|
||||
}
|
||||
|
||||
// Build HNSW — the expensive part
|
||||
let mut builder = Builder::default()
|
||||
.ef_construction(config.ef_construction)
|
||||
.ef_search(config.ef_search);
|
||||
if let Some(seed) = config.seed {
|
||||
builder = builder.seed(seed);
|
||||
}
|
||||
let map = builder.build(points, values);
|
||||
// Build HNSW — this is the expensive part
|
||||
let map = Builder::default()
|
||||
.ef_construction(40) // balanced for 100K scale
|
||||
.ef_search(50) // fast search with good recall
|
||||
.build(points, values);
|
||||
|
||||
let build_time = start.elapsed().as_secs_f32();
|
||||
tracing::info!("HNSW '{}' built: {} vectors in {:.1}s", index_name, n, build_time);
|
||||
tracing::info!("HNSW index '{}' built: {} vectors in {:.1}s", index_name, n, build_time);
|
||||
|
||||
let index = Arc::new(HnswIndex { map, metadata });
|
||||
self.indexes.write().await.insert(index_name.to_string(), index);
|
||||
@@ -129,43 +111,6 @@ impl HnswStore {
|
||||
})
|
||||
}
|
||||
|
||||
/// Run a batch of search queries and return raw per-query latencies in microseconds.
|
||||
/// Also returns the retrieved doc_ids per query (for recall calculation).
|
||||
pub async fn bench_search(
|
||||
&self,
|
||||
index_name: &str,
|
||||
query_vectors: &[Vec<f32>],
|
||||
top_k: usize,
|
||||
) -> Result<BenchResult, String> {
|
||||
let indexes = self.indexes.read().await;
|
||||
let index = indexes
|
||||
.get(index_name)
|
||||
.ok_or_else(|| format!("HNSW index not found: {index_name}"))?
|
||||
.clone();
|
||||
drop(indexes);
|
||||
|
||||
let mut latencies_us = Vec::with_capacity(query_vectors.len());
|
||||
let mut retrieved: Vec<Vec<String>> = Vec::with_capacity(query_vectors.len());
|
||||
|
||||
for qv in query_vectors {
|
||||
let query_point = VecPoint(qv.clone());
|
||||
let t0 = std::time::Instant::now();
|
||||
let mut search = Search::default();
|
||||
let results = index.map.search(&query_point, &mut search);
|
||||
let ids: Vec<String> = results
|
||||
.take(top_k)
|
||||
.map(|item| {
|
||||
let meta_idx = *item.value;
|
||||
index.metadata[meta_idx].doc_id.clone()
|
||||
})
|
||||
.collect();
|
||||
latencies_us.push(t0.elapsed().as_micros() as f32);
|
||||
retrieved.push(ids);
|
||||
}
|
||||
|
||||
Ok(BenchResult { latencies_us, retrieved })
|
||||
}
|
||||
|
||||
/// Search an HNSW index. Returns approximate nearest neighbors.
|
||||
pub async fn search(
|
||||
&self,
|
||||
@@ -221,9 +166,3 @@ pub struct BuildStats {
|
||||
pub vectors: usize,
|
||||
pub build_time_secs: f32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BenchResult {
|
||||
pub latencies_us: Vec<f32>,
|
||||
pub retrieved: Vec<Vec<String>>,
|
||||
}
|
||||
|
||||
@@ -29,27 +29,8 @@ pub struct IndexMeta {
|
||||
pub created_at: DateTime<Utc>,
|
||||
pub build_time_secs: f32,
|
||||
pub chunks_per_sec: f32,
|
||||
/// Federation layer 2: which bucket holds this index's artifacts
|
||||
/// (trial journal + promotion file). Defaults to "primary" for
|
||||
/// pre-federation indexes — the serde default keeps old metadata
|
||||
/// files readable without migration.
|
||||
#[serde(default = "default_bucket")]
|
||||
pub bucket: String,
|
||||
/// ADR-019: which physical backend stores this index. `Parquet`
|
||||
/// means storage_key points at our binary-blob Parquet file;
|
||||
/// `Lance` means it points at a Lance dataset directory.
|
||||
#[serde(default)]
|
||||
pub vector_backend: shared::types::VectorBackend,
|
||||
/// ADR-020: prefix prepended to doc_ids during embedding. If set,
|
||||
/// hybrid search strips this prefix to match against SQL primary keys.
|
||||
/// None = doc_ids ARE the raw primary keys (no stripping needed).
|
||||
/// Existing indexes: "W-", "CAND-", "W500K-", etc.
|
||||
#[serde(default)]
|
||||
pub id_prefix: Option<String>,
|
||||
}
|
||||
|
||||
fn default_bucket() -> String { "primary".to_string() }
|
||||
|
||||
/// Registry of all vector indexes.
|
||||
#[derive(Clone)]
|
||||
pub struct IndexRegistry {
|
||||
|
||||
@@ -22,12 +22,7 @@ pub struct Job {
|
||||
pub status: JobStatus,
|
||||
pub index_name: String,
|
||||
pub total_chunks: usize,
|
||||
/// How many chunks have been embedded so far.
|
||||
/// Also serialized as "processed" for backward-compat with monitoring scripts.
|
||||
pub embedded_chunks: usize,
|
||||
#[serde(rename = "processed")]
|
||||
pub processed_alias: usize,
|
||||
pub total: usize,
|
||||
pub progress_pct: f32,
|
||||
pub storage_key: Option<String>,
|
||||
pub error: Option<String>,
|
||||
@@ -58,8 +53,6 @@ impl JobTracker {
|
||||
index_name: index_name.to_string(),
|
||||
total_chunks,
|
||||
embedded_chunks: 0,
|
||||
processed_alias: 0,
|
||||
total: total_chunks,
|
||||
progress_pct: 0.0,
|
||||
storage_key: None,
|
||||
error: None,
|
||||
@@ -76,7 +69,6 @@ impl JobTracker {
|
||||
let mut jobs = self.jobs.write().await;
|
||||
if let Some(job) = jobs.get_mut(id) {
|
||||
job.embedded_chunks = embedded;
|
||||
job.processed_alias = embedded; // keep alias in sync
|
||||
job.progress_pct = if job.total_chunks > 0 {
|
||||
(embedded as f32 / job.total_chunks as f32) * 100.0
|
||||
} else {
|
||||
@@ -92,7 +84,6 @@ impl JobTracker {
|
||||
if let Some(job) = jobs.get_mut(id) {
|
||||
job.status = JobStatus::Completed;
|
||||
job.embedded_chunks = job.total_chunks;
|
||||
job.processed_alias = job.total_chunks;
|
||||
job.progress_pct = 100.0;
|
||||
job.storage_key = Some(storage_key);
|
||||
job.completed_at = Some(chrono::Utc::now().to_rfc3339());
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
//! ADR-019 hybrid: routing layer for the Lance vector backend.
|
||||
//!
|
||||
//! Holds a `LanceRegistry` which maps an index name to its
|
||||
//! `LanceVectorStore` instance, lazy-creating on first touch. Path
|
||||
//! resolution: the index's bucket gives us a bucket root; we append
|
||||
//! `lance/{index_name}` and use that as the dataset URI. For local
|
||||
//! buckets that means a directory under the bucket's root.
|
||||
//!
|
||||
//! S3 buckets: in principle Lance accepts s3://... URIs, but the
|
||||
//! firewall crate (`vectord-lance`) doesn't pull the S3 feature in by
|
||||
//! default. When we promote profile buckets onto S3 in a future phase,
|
||||
//! enable that feature and update `lance_uri_for` accordingly.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use storaged::registry::BucketRegistry;
|
||||
use vectord_lance::LanceVectorStore;
|
||||
|
||||
use crate::index_registry::IndexRegistry;
|
||||
|
||||
/// Convert a bucket+index pair into the URI Lance should use as the
|
||||
/// dataset path. Supports both local (filesystem) and S3 buckets.
|
||||
///
|
||||
/// **Local buckets:** path resolution mirrors lakehouse.toml's convention.
|
||||
/// Returns an absolute filesystem path.
|
||||
///
|
||||
/// **S3 buckets:** returns `s3://{s3_bucket}/lance/{index_name}`. Lance's
|
||||
/// internal object_store crate reads `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
|
||||
/// / `AWS_ENDPOINT` from environment (or the S3 feature's default chain).
|
||||
/// For MinIO: set `AWS_ENDPOINT=http://localhost:9000` and
|
||||
/// `AWS_ALLOW_HTTP=true` before starting the gateway.
|
||||
///
|
||||
/// Refuses unknown buckets so a typo doesn't silently land Lance data
|
||||
/// in a directory / prefix the rest of the system can't see.
|
||||
pub fn lance_uri_for(
|
||||
buckets: &BucketRegistry,
|
||||
bucket: &str,
|
||||
index_name: &str,
|
||||
) -> Result<String, String> {
|
||||
if !buckets.contains(bucket) {
|
||||
return Err(format!("bucket '{bucket}' not registered"));
|
||||
}
|
||||
// Check if this bucket is S3-backed by looking for a bucket config
|
||||
// with backend="s3". BucketRegistry exposes backend type through
|
||||
// the list() info, but that's async. The simpler signal: if the
|
||||
// bucket name matches one we know is S3 (configured via lakehouse.toml
|
||||
// with backend="s3"), use the s3:// URI scheme.
|
||||
//
|
||||
// For the synchronous path, we check a naming convention: buckets
|
||||
// whose name starts with "s3:" are treated as S3 targets. The rest
|
||||
// of the name is the S3 bucket name. Convention-based, explicit,
|
||||
// no async needed.
|
||||
//
|
||||
// Additionally, any bucket registered with backend="s3" in the
|
||||
// config will have its BucketConfig.bucket field set — that's the
|
||||
// actual S3 bucket name. We can't access BucketConfig synchronously
|
||||
// from the current registry API, so for now the naming convention
|
||||
// is the primary signal.
|
||||
if bucket.starts_with("s3:") {
|
||||
let s3_bucket = &bucket["s3:".len()..];
|
||||
return Ok(format!("s3://{s3_bucket}/lance/{index_name}"));
|
||||
}
|
||||
|
||||
// Local path resolution.
|
||||
let root: PathBuf = match bucket {
|
||||
"primary" => PathBuf::from("./data"),
|
||||
"rescue" => PathBuf::from("./data/_rescue"),
|
||||
"testing" => PathBuf::from("./data/_testing"),
|
||||
b if b.starts_with("profile:") => {
|
||||
let safe = b.replace(':', "_");
|
||||
PathBuf::from(format!("./data/_profiles/{safe}"))
|
||||
}
|
||||
b => PathBuf::from(format!("./data/_buckets/{}", b.replace(':', "_"))),
|
||||
};
|
||||
let _ = std::fs::create_dir_all(root.join("lance"));
|
||||
let abs = match std::fs::canonicalize(&root) {
|
||||
Ok(p) => p.join("lance").join(index_name),
|
||||
Err(_) => root.join("lance").join(index_name),
|
||||
};
|
||||
Ok(abs.to_string_lossy().to_string())
|
||||
}
|
||||
|
||||
/// Lookup-by-index registry of `LanceVectorStore` handles. One handle
|
||||
/// per index, lazy-created on first call. Cheap to keep alive — it's
|
||||
/// just a path string + handle to Lance's metadata cache.
|
||||
#[derive(Clone)]
|
||||
pub struct LanceRegistry {
|
||||
buckets: Arc<BucketRegistry>,
|
||||
indexes: IndexRegistry,
|
||||
stores: Arc<RwLock<HashMap<String, LanceVectorStore>>>,
|
||||
}
|
||||
|
||||
impl LanceRegistry {
|
||||
pub fn new(buckets: Arc<BucketRegistry>, indexes: IndexRegistry) -> Self {
|
||||
Self {
|
||||
buckets,
|
||||
indexes,
|
||||
stores: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Get (or lazy-create) the LanceVectorStore for an index.
|
||||
/// Resolves bucket → URI via the registry; backend stays whatever
|
||||
/// IndexMeta says (no enforcement here — caller decides whether to
|
||||
/// route to Lance based on profile/index backend choice).
|
||||
pub async fn store_for(&self, index_name: &str) -> Result<LanceVectorStore, String> {
|
||||
if let Some(s) = self.stores.read().await.get(index_name) {
|
||||
return Ok(s.clone());
|
||||
}
|
||||
let bucket = self.indexes
|
||||
.get(index_name).await
|
||||
.map(|m| m.bucket)
|
||||
.unwrap_or_else(|| "primary".to_string());
|
||||
let uri = lance_uri_for(&self.buckets, &bucket, index_name)?;
|
||||
let store = LanceVectorStore::new(uri);
|
||||
self.stores.write().await.insert(index_name.to_string(), store.clone());
|
||||
Ok(store)
|
||||
}
|
||||
|
||||
/// For freshly-created indexes that don't have IndexMeta yet — caller
|
||||
/// supplies the bucket explicitly.
|
||||
pub async fn store_for_new(&self, index_name: &str, bucket: &str) -> Result<LanceVectorStore, String> {
|
||||
let uri = lance_uri_for(&self.buckets, bucket, index_name)?;
|
||||
let store = LanceVectorStore::new(uri);
|
||||
self.stores.write().await.insert(index_name.to_string(), store.clone());
|
||||
Ok(store)
|
||||
}
|
||||
}
|
||||
@@ -1,18 +1,9 @@
|
||||
pub mod agent;
|
||||
pub mod autotune;
|
||||
pub mod lance_backend;
|
||||
pub mod chunker;
|
||||
pub mod embedding_cache;
|
||||
pub mod harness;
|
||||
pub mod hnsw;
|
||||
pub mod index_registry;
|
||||
pub mod jobs;
|
||||
pub mod playbook_memory;
|
||||
pub mod promotion;
|
||||
pub mod refresh;
|
||||
pub mod store;
|
||||
pub mod search;
|
||||
pub mod rag;
|
||||
pub mod supervisor;
|
||||
pub mod service;
|
||||
pub mod trial;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,220 +0,0 @@
|
||||
//! Phase 16: Promoted HNSW configs — the "active generation" pointer.
|
||||
//!
|
||||
//! An index's HNSW config used at build time normally defaults to the
|
||||
//! system-wide default (`HnswConfig::default()`). An operator or the
|
||||
//! autotune agent can *promote* a specific trial's config — subsequent
|
||||
//! HNSW builds against that index use the promoted config instead.
|
||||
//!
|
||||
//! Every promotion is history-tracked so `rollback` can revert. The
|
||||
//! history file lives at `primary://_hnsw_promotions/{index_name}.json`
|
||||
//! and is small (< few KB) so we rewrite it on every promotion rather
|
||||
//! than append-log.
|
||||
//!
|
||||
//! Not included here:
|
||||
//! - Atomic graph rebuild on promote — promotion only updates the sticky
|
||||
//! default. Next activation (or search that triggers lazy build) picks
|
||||
//! up the new config. That's "zero-downtime swap after build" which is
|
||||
//! what ADR-019 actually claimed; an instant-swap requires a
|
||||
//! pre-built graph pool which we don't have yet.
|
||||
//! - Agent loop — lives in `vectord::autotune`.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use storaged::ops;
|
||||
use storaged::registry::BucketRegistry;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::trial::HnswConfig;
|
||||
|
||||
const PROMOTION_PREFIX: &str = "_hnsw_promotions";
|
||||
|
||||
/// One promotion record. The `trial_id` is the origin of the config —
|
||||
/// lets operators trace back "why was this config picked?" to the exact
|
||||
/// trial in the trial journal.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct PromotionEntry {
|
||||
pub config: HnswConfig,
|
||||
pub trial_id: String,
|
||||
pub promoted_at: DateTime<Utc>,
|
||||
#[serde(default)]
|
||||
pub promoted_by: String,
|
||||
#[serde(default)]
|
||||
pub note: Option<String>,
|
||||
}
|
||||
|
||||
/// Serialized form of an index's promotion history.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct PromotionFile {
|
||||
pub index_name: String,
|
||||
pub current: Option<PromotionEntry>,
|
||||
#[serde(default)]
|
||||
pub history: Vec<PromotionEntry>,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct PromotionRegistry {
|
||||
buckets: Arc<BucketRegistry>,
|
||||
index_registry: IndexRegistry,
|
||||
cache: Arc<RwLock<HashMap<String, PromotionFile>>>,
|
||||
}
|
||||
|
||||
impl PromotionRegistry {
|
||||
pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
|
||||
Self {
|
||||
buckets,
|
||||
index_registry,
|
||||
cache: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
fn key(index_name: &str) -> String {
|
||||
// Sanitize for object-store safety.
|
||||
let safe: String = index_name
|
||||
.chars()
|
||||
.map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' })
|
||||
.collect();
|
||||
format!("{PROMOTION_PREFIX}/{safe}.json")
|
||||
}
|
||||
|
||||
/// Resolve which bucket's store holds this index's promotion file.
|
||||
/// Same rules as TrialJournal::bucket_for — follows IndexMeta.bucket,
|
||||
/// defaults to primary when metadata is missing.
|
||||
async fn store_for(&self, index_name: &str) -> Result<Arc<dyn object_store::ObjectStore>, String> {
|
||||
let bucket = self.index_registry
|
||||
.get(index_name)
|
||||
.await
|
||||
.map(|m| m.bucket)
|
||||
.unwrap_or_else(|| "primary".to_string());
|
||||
self.buckets.get(&bucket)
|
||||
}
|
||||
|
||||
/// Load (and cache) the promotion file for an index.
|
||||
pub async fn load(&self, index_name: &str) -> Result<PromotionFile, String> {
|
||||
if let Some(cached) = self.cache.read().await.get(index_name) {
|
||||
return Ok(cached.clone());
|
||||
}
|
||||
let store = self.store_for(index_name).await?;
|
||||
let key = Self::key(index_name);
|
||||
let file = match ops::get(&store, &key).await {
|
||||
Ok(bytes) => serde_json::from_slice::<PromotionFile>(&bytes)
|
||||
.map_err(|e| format!("parse promotion file: {e}"))?,
|
||||
Err(_) => PromotionFile {
|
||||
index_name: index_name.to_string(),
|
||||
current: None,
|
||||
history: Vec::new(),
|
||||
},
|
||||
};
|
||||
self.cache.write().await.insert(index_name.to_string(), file.clone());
|
||||
Ok(file)
|
||||
}
|
||||
|
||||
/// Promote a config to the active slot. Pushes the current promotion
|
||||
/// (if any) onto the history stack. Persists before returning — the
|
||||
/// config is durable by the time this call completes.
|
||||
pub async fn promote(
|
||||
&self,
|
||||
index_name: &str,
|
||||
entry: PromotionEntry,
|
||||
) -> Result<PromotionFile, String> {
|
||||
let mut file = self.load(index_name).await?;
|
||||
if let Some(prior) = file.current.take() {
|
||||
file.history.push(prior);
|
||||
// Cap history to something sensible so this file doesn't grow
|
||||
// unbounded. 50 entries = 50 promotions — way more than any
|
||||
// sane workflow needs.
|
||||
const HISTORY_CAP: usize = 50;
|
||||
if file.history.len() > HISTORY_CAP {
|
||||
let drop = file.history.len() - HISTORY_CAP;
|
||||
file.history.drain(0..drop);
|
||||
}
|
||||
}
|
||||
file.current = Some(entry);
|
||||
file.index_name = index_name.to_string();
|
||||
|
||||
let store = self.store_for(index_name).await?;
|
||||
let key = Self::key(index_name);
|
||||
let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
|
||||
ops::put(&store, &key, json.into()).await?;
|
||||
|
||||
self.cache.write().await.insert(index_name.to_string(), file.clone());
|
||||
tracing::info!(
|
||||
"promoted '{}' to config {:?} (trial={})",
|
||||
index_name, file.current.as_ref().unwrap().config, file.current.as_ref().unwrap().trial_id,
|
||||
);
|
||||
Ok(file)
|
||||
}
|
||||
|
||||
/// Pop the latest promotion back onto the current slot (if any
|
||||
/// history exists). If current is set but history is empty, the
|
||||
/// current promotion is cleared — the index falls back to defaults.
|
||||
pub async fn rollback(&self, index_name: &str) -> Result<PromotionFile, String> {
|
||||
let mut file = self.load(index_name).await?;
|
||||
match file.history.pop() {
|
||||
Some(prev) => {
|
||||
file.current = Some(prev);
|
||||
}
|
||||
None => {
|
||||
if file.current.is_none() {
|
||||
return Err(format!("no promotion to rollback for '{index_name}'"));
|
||||
}
|
||||
file.current = None;
|
||||
}
|
||||
}
|
||||
let store = self.store_for(index_name).await?;
|
||||
let key = Self::key(index_name);
|
||||
let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
|
||||
ops::put(&store, &key, json.into()).await?;
|
||||
self.cache.write().await.insert(index_name.to_string(), file.clone());
|
||||
tracing::info!("rolled back promotion for '{}'", index_name);
|
||||
Ok(file)
|
||||
}
|
||||
|
||||
/// Get the currently-promoted config (if any). Callers use this to
|
||||
/// pick the right HnswConfig at build time.
|
||||
pub async fn get_current(&self, index_name: &str) -> Option<PromotionEntry> {
|
||||
self.load(index_name).await.ok().and_then(|f| f.current)
|
||||
}
|
||||
|
||||
/// Convenience: return the promoted config or the provided default.
|
||||
pub async fn config_or(&self, index_name: &str, default: HnswConfig) -> HnswConfig {
|
||||
match self.get_current(index_name).await {
|
||||
Some(entry) => entry.config,
|
||||
None => default,
|
||||
}
|
||||
}
|
||||
|
||||
/// List every index that has a promotion recorded (for operator UI).
|
||||
///
|
||||
/// Federation: scans EVERY registered bucket for promotion files.
|
||||
/// Per-profile buckets each have their own `_hnsw_promotions/` so we
|
||||
/// aggregate across them. Dedups by index_name — if the same index
|
||||
/// somehow has promotion files in multiple buckets, the one from the
|
||||
/// bucket recorded in IndexMeta wins.
|
||||
pub async fn list_all(&self) -> Result<Vec<PromotionFile>, String> {
|
||||
let bucket_infos = self.buckets.list().await;
|
||||
let mut by_name: HashMap<String, PromotionFile> = HashMap::new();
|
||||
|
||||
for b in &bucket_infos {
|
||||
let store = match self.buckets.get(&b.name) {
|
||||
Ok(s) => s,
|
||||
Err(_) => continue,
|
||||
};
|
||||
let keys = ops::list(&store, Some(&format!("{PROMOTION_PREFIX}/")))
|
||||
.await.unwrap_or_default();
|
||||
for key in keys {
|
||||
if !key.ends_with(".json") { continue; }
|
||||
let bytes = match ops::get(&store, &key).await {
|
||||
Ok(b) => b,
|
||||
Err(_) => continue,
|
||||
};
|
||||
if let Ok(f) = serde_json::from_slice::<PromotionFile>(&bytes) {
|
||||
by_name.insert(f.index_name.clone(), f);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(by_name.into_values().collect())
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,4 @@
|
||||
/// RAG pipeline: question → embed → search → rerank → generate answer.
|
||||
///
|
||||
/// The rerank step (added 2026-04-17) uses the LLM as a cross-encoder
|
||||
/// between retrieval and generation. This catches cases where the
|
||||
/// embedding model scores documents as similar but the content isn't
|
||||
/// actually relevant to the question — a known weakness of small
|
||||
/// general-purpose embed models on domain-specific text.
|
||||
/// RAG pipeline: question → embed → search → retrieve → generate answer.
|
||||
|
||||
use object_store::ObjectStore;
|
||||
use std::sync::Arc;
|
||||
@@ -13,75 +7,6 @@ use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
|
||||
use crate::search::{self, SearchResult};
|
||||
use crate::store;
|
||||
|
||||
/// Cross-encoder rerank: ask the LLM to re-sort retrieved chunks by
|
||||
/// relevance. Falls back to the original order if the model returns
|
||||
/// garbage (which happens ~5% of the time with 7B models).
|
||||
async fn rerank(
|
||||
question: &str,
|
||||
mut results: Vec<SearchResult>,
|
||||
ai_client: &AiClient,
|
||||
) -> Vec<SearchResult> {
|
||||
if results.len() <= 1 {
|
||||
return results;
|
||||
}
|
||||
|
||||
let chunk_list: String = results.iter().enumerate().map(|(i, r)| {
|
||||
let text: String = r.chunk_text.chars().take(200).collect();
|
||||
format!("[{i}] {text}")
|
||||
}).collect::<Vec<_>>().join("\n");
|
||||
|
||||
let resp = ai_client.generate(GenerateRequest {
|
||||
prompt: format!(
|
||||
"Rank these text chunks by relevance to the question.\n\
|
||||
Return ONLY a comma-separated list of indices, most relevant first.\n\n\
|
||||
Question: {question}\n\n\
|
||||
Chunks:\n{chunk_list}\n\n\
|
||||
Ranking:"
|
||||
),
|
||||
model: None,
|
||||
system: None,
|
||||
temperature: Some(0.0),
|
||||
max_tokens: Some(50),
|
||||
}).await;
|
||||
|
||||
match resp {
|
||||
Ok(gen_resp) => {
|
||||
let text = gen_resp.text.trim();
|
||||
let indices: Vec<usize> = text
|
||||
.split(|c: char| c == ',' || c.is_whitespace())
|
||||
.filter_map(|s| s.trim().parse::<usize>().ok())
|
||||
.filter(|&i| i < results.len())
|
||||
.collect();
|
||||
|
||||
if indices.is_empty() {
|
||||
tracing::debug!("reranker returned unparseable output: {text}");
|
||||
return results;
|
||||
}
|
||||
|
||||
let mut reranked: Vec<SearchResult> = Vec::with_capacity(results.len());
|
||||
let mut used = vec![false; results.len()];
|
||||
for &i in &indices {
|
||||
if !used[i] {
|
||||
reranked.push(results[i].clone());
|
||||
used[i] = true;
|
||||
}
|
||||
}
|
||||
// Append any the model didn't mention (preserves all results).
|
||||
for (i, r) in results.drain(..).enumerate() {
|
||||
if !used[i] {
|
||||
reranked.push(r);
|
||||
}
|
||||
}
|
||||
tracing::info!("reranker reordered {}/{} chunks", indices.len(), reranked.len());
|
||||
reranked
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("reranker failed (using original order): {e}");
|
||||
results
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Full RAG answer with provenance.
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct RagResponse {
|
||||
@@ -126,24 +51,16 @@ pub async fn query(
|
||||
});
|
||||
}
|
||||
|
||||
// 3. Rerank: ask the LLM to sort retrieved chunks by relevance.
|
||||
// This cross-encoder step catches cases where embedding similarity
|
||||
// is high but semantic relevance is low — common with small embed
|
||||
// models on domain-specific text.
|
||||
let results = rerank(question, results, ai_client).await;
|
||||
|
||||
// 4. Build context from (reranked) top chunks
|
||||
// 3. Build context from retrieved chunks
|
||||
let context: String = results.iter().enumerate().map(|(i, r)| {
|
||||
format!("[{}] (source: {}, doc: {}) {}", i + 1, r.source, r.doc_id, r.chunk_text)
|
||||
}).collect::<Vec<_>>().join("\n\n");
|
||||
|
||||
// 5. Generate answer
|
||||
// 4. Generate answer
|
||||
tracing::info!("RAG: generating answer from {} chunks", results.len());
|
||||
let prompt = format!(
|
||||
"You are a helpful assistant answering questions about a staffing database.\n\n\
|
||||
Use ONLY the following retrieved records to answer. Be specific — cite names,\n\
|
||||
numbers, skills, and cities from the records. If the context doesn't contain\n\
|
||||
enough information to fully answer, say what you can and note what's missing.\n\
|
||||
"You are a helpful assistant answering questions based on retrieved documents from a data system.\n\n\
|
||||
Use ONLY the following context to answer. If the context doesn't contain enough information, say so.\n\
|
||||
Cite sources by their number [1], [2], etc.\n\n\
|
||||
Context:\n{context}\n\n\
|
||||
Question: {question}\n\n\
|
||||
|
||||
@@ -1,313 +0,0 @@
|
||||
//! Phase C: Decoupled embedding refresh.
|
||||
//!
|
||||
//! When a dataset's row-level data changes, the vector index it feeds is
|
||||
//! stale. Historically we coupled ingest and embedding — writing new rows
|
||||
//! also re-embedded every row, which is fine at 500 rows but blows up at
|
||||
//! 100K+. The llms3.com architecture calls out "asynchronous vector
|
||||
//! refresh cycles independent from transactional mutations" as the right
|
||||
//! pattern.
|
||||
//!
|
||||
//! This module implements the refresh side. The ingest path marks
|
||||
//! embeddings stale (see catalogd::registry::mark_embeddings_stale); this
|
||||
//! code clears that staleness by embedding only rows whose `doc_id` isn't
|
||||
//! already in the existing vector index.
|
||||
//!
|
||||
//! Scope — MVP:
|
||||
//! - Reads the dataset's Parquet, extracts (doc_id, text) pairs from named
|
||||
//! columns
|
||||
//! - Loads existing embeddings via EmbeddingCache
|
||||
//! - Filters to rows whose doc_id is NOT in the existing set
|
||||
//! - Chunks, embeds via Ollama, appends to the index parquet
|
||||
//! - Clears stale flag on success
|
||||
//!
|
||||
//! Not in MVP:
|
||||
//! - UPDATE semantics (same doc_id, new content) — would need content-hash
|
||||
//! comparison per row
|
||||
//! - Large-scale resilience (batching with checkpoints like the
|
||||
//! supervisor) — MVP does it inline
|
||||
//! - Lance backend — ADR-019 makes this straightforward later; MVP stays
|
||||
//! on Parquet sidecar indexes
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::sync::Arc;
|
||||
|
||||
use aibridge::client::{AiClient, EmbedRequest};
|
||||
use arrow::array::{Array, Int32Array, Int64Array, StringArray};
|
||||
use catalogd::registry::Registry;
|
||||
use object_store::ObjectStore;
|
||||
|
||||
use crate::chunker::{self, TextChunk};
|
||||
use crate::embedding_cache::EmbeddingCache;
|
||||
use crate::index_registry::IndexRegistry;
|
||||
use crate::store::{self, StoredEmbedding};
|
||||
|
||||
#[derive(Debug, Clone, serde::Deserialize)]
|
||||
pub struct RefreshRequest {
|
||||
pub index_name: String,
|
||||
/// Column with the document id (row identity).
|
||||
pub id_column: String,
|
||||
/// Column with the text to embed.
|
||||
pub text_column: String,
|
||||
/// Chunk size (chars). Defaults to 500.
|
||||
#[serde(default)]
|
||||
pub chunk_size: Option<usize>,
|
||||
/// Overlap (chars). Defaults to 50.
|
||||
#[serde(default)]
|
||||
pub overlap: Option<usize>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
pub struct RefreshResult {
|
||||
pub index_name: String,
|
||||
pub dataset_name: String,
|
||||
pub pre_existing_docs: usize,
|
||||
pub dataset_docs: usize,
|
||||
pub new_docs_embedded: usize,
|
||||
pub new_chunks_embedded: usize,
|
||||
pub total_embeddings_after: usize,
|
||||
pub duration_secs: f32,
|
||||
pub stale_cleared: bool,
|
||||
}
|
||||
|
||||
/// Full refresh pipeline. Takes dataset_name as URL param, body has the
|
||||
/// column selectors.
|
||||
pub async fn refresh_index(
|
||||
dataset_name: &str,
|
||||
req: &RefreshRequest,
|
||||
store_: &Arc<dyn ObjectStore>,
|
||||
registry: &Registry,
|
||||
ai_client: &AiClient,
|
||||
embedding_cache: &EmbeddingCache,
|
||||
index_registry: &IndexRegistry,
|
||||
) -> Result<RefreshResult, String> {
|
||||
let t0 = std::time::Instant::now();
|
||||
|
||||
// 1. Find the dataset manifest, pull its object storage key
|
||||
let manifest = registry
|
||||
.get_by_name(dataset_name)
|
||||
.await
|
||||
.ok_or_else(|| format!("dataset not found: {dataset_name}"))?;
|
||||
|
||||
if manifest.objects.is_empty() {
|
||||
return Err(format!("dataset '{dataset_name}' has no object references"));
|
||||
}
|
||||
|
||||
// 2. Read dataset rows — extract (doc_id, text) pairs from the
|
||||
// specified columns. Multi-object datasets: concat across all.
|
||||
let mut doc_id_to_text: Vec<(String, String)> = Vec::new();
|
||||
for obj in &manifest.objects {
|
||||
let data = storaged::ops::get(store_, &obj.key).await
|
||||
.map_err(|e| format!("read {}: {e}", obj.key))?;
|
||||
let (schema, batches) = shared::arrow_helpers::parquet_to_record_batches(&data)
|
||||
.map_err(|e| format!("parse {}: {e}", obj.key))?;
|
||||
|
||||
let id_idx = schema.index_of(&req.id_column)
|
||||
.map_err(|_| format!("id column '{}' not in dataset schema", req.id_column))?;
|
||||
let text_idx = schema.index_of(&req.text_column)
|
||||
.map_err(|_| format!("text column '{}' not in dataset schema", req.text_column))?;
|
||||
|
||||
for batch in &batches {
|
||||
let text_col = batch
|
||||
.column(text_idx)
|
||||
.as_any()
|
||||
.downcast_ref::<StringArray>()
|
||||
.ok_or_else(|| format!("text column '{}' is not Utf8", req.text_column))?;
|
||||
|
||||
// Accept Utf8, Int32, or Int64 as id — that covers CSV-derived
|
||||
// and Postgres-imported schemas without forcing upstream casts.
|
||||
let id_reader: Box<dyn Fn(usize) -> Option<String>> = {
|
||||
let col = batch.column(id_idx);
|
||||
if let Some(s) = col.as_any().downcast_ref::<StringArray>() {
|
||||
let s = s.clone();
|
||||
Box::new(move |row| if s.is_null(row) { None } else { Some(s.value(row).to_string()) })
|
||||
} else if let Some(a) = col.as_any().downcast_ref::<Int32Array>() {
|
||||
let a = a.clone();
|
||||
Box::new(move |row| if a.is_null(row) { None } else { Some(a.value(row).to_string()) })
|
||||
} else if let Some(a) = col.as_any().downcast_ref::<Int64Array>() {
|
||||
let a = a.clone();
|
||||
Box::new(move |row| if a.is_null(row) { None } else { Some(a.value(row).to_string()) })
|
||||
} else {
|
||||
return Err(format!(
|
||||
"id column '{}' must be Utf8, Int32, or Int64 — got {}",
|
||||
req.id_column,
|
||||
col.data_type(),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
for row in 0..batch.num_rows() {
|
||||
if text_col.is_null(row) { continue; }
|
||||
let Some(id) = id_reader(row) else { continue; };
|
||||
let text = text_col.value(row).to_string();
|
||||
if text.trim().is_empty() { continue; }
|
||||
doc_id_to_text.push((id, text));
|
||||
}
|
||||
}
|
||||
}
|
||||
let dataset_docs = doc_id_to_text.len();
|
||||
tracing::info!("refresh '{}': dataset has {dataset_docs} rows", dataset_name);
|
||||
|
||||
// 3. Load existing embeddings (empty if no index yet)
|
||||
let existing: Vec<StoredEmbedding> = match embedding_cache.get_or_load(&req.index_name).await {
|
||||
Ok(arc) => arc.as_ref().clone(),
|
||||
Err(_) => Vec::new(), // first-time index build
|
||||
};
|
||||
let pre_existing_docs: HashSet<String> = existing
|
||||
.iter()
|
||||
.map(|e| e.doc_id.clone())
|
||||
.collect();
|
||||
let pre_existing_count = pre_existing_docs.len();
|
||||
|
||||
// 4. Delta — rows whose doc_id isn't already indexed
|
||||
let new_rows: Vec<(String, String)> = doc_id_to_text
|
||||
.into_iter()
|
||||
.filter(|(id, _)| !pre_existing_docs.contains(id))
|
||||
.collect();
|
||||
let new_docs = new_rows.len();
|
||||
|
||||
if new_docs == 0 {
|
||||
tracing::info!("refresh '{}': no new docs to embed", dataset_name);
|
||||
// Even on no-op, make sure index metadata is registered so
|
||||
// downstream discovery (profile activation, A/B search) sees
|
||||
// indexes built by earlier refresh calls that predated the
|
||||
// auto-register behavior.
|
||||
let _ = try_update_index_meta(index_registry, &req.index_name, existing.len()).await;
|
||||
registry.clear_embeddings_stale(dataset_name).await?;
|
||||
return Ok(RefreshResult {
|
||||
index_name: req.index_name.clone(),
|
||||
dataset_name: dataset_name.to_string(),
|
||||
pre_existing_docs: pre_existing_count,
|
||||
dataset_docs,
|
||||
new_docs_embedded: 0,
|
||||
new_chunks_embedded: 0,
|
||||
total_embeddings_after: existing.len(),
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
stale_cleared: true,
|
||||
});
|
||||
}
|
||||
|
||||
// 5. Chunk the new rows
|
||||
let chunk_size = req.chunk_size.unwrap_or(500);
|
||||
let overlap = req.overlap.unwrap_or(50);
|
||||
let doc_ids: Vec<String> = new_rows.iter().map(|(id, _)| id.clone()).collect();
|
||||
let texts: Vec<String> = new_rows.iter().map(|(_, t)| t.clone()).collect();
|
||||
let chunks: Vec<TextChunk> = chunker::chunk_column(
|
||||
dataset_name, &doc_ids, &texts, chunk_size, overlap,
|
||||
);
|
||||
let new_chunks = chunks.len();
|
||||
tracing::info!("refresh '{}': {} new docs -> {} chunks", dataset_name, new_docs, new_chunks);
|
||||
|
||||
// 6. Embed via Ollama (batched)
|
||||
let batch_size = 32;
|
||||
let mut all_vectors: Vec<Vec<f64>> = Vec::with_capacity(new_chunks);
|
||||
for batch in chunks.chunks(batch_size) {
|
||||
let batch_texts: Vec<String> = batch.iter().map(|c| c.text.clone()).collect();
|
||||
let resp = ai_client
|
||||
.embed(EmbedRequest { texts: batch_texts, model: None })
|
||||
.await
|
||||
.map_err(|e| format!("embed: {e}"))?;
|
||||
all_vectors.extend(resp.embeddings);
|
||||
}
|
||||
|
||||
// 7. Combine existing + new and write back as a single parquet
|
||||
// (MVP — append-as-rewrite. ADR-019 points to Lance for true native
|
||||
// append; for the Parquet sidecar we rewrite at refresh time.)
|
||||
let mut new_stored: Vec<StoredEmbedding> = existing.clone();
|
||||
for (chunk, vector) in chunks.into_iter().zip(all_vectors.iter()) {
|
||||
new_stored.push(StoredEmbedding {
|
||||
source: chunk.source,
|
||||
doc_id: chunk.doc_id,
|
||||
chunk_idx: chunk.chunk_idx,
|
||||
chunk_text: chunk.text,
|
||||
vector: vector.iter().map(|&x| x as f32).collect(),
|
||||
});
|
||||
}
|
||||
|
||||
// We have to reconstruct chunks + vectors for store_embeddings — but
|
||||
// store_embeddings takes &[TextChunk] and vectors separately. Convert
|
||||
// the combined StoredEmbedding back to that shape.
|
||||
let combined_chunks: Vec<TextChunk> = new_stored
|
||||
.iter()
|
||||
.map(|e| TextChunk {
|
||||
source: e.source.clone(),
|
||||
doc_id: e.doc_id.clone(),
|
||||
chunk_idx: e.chunk_idx,
|
||||
text: e.chunk_text.clone(),
|
||||
})
|
||||
.collect();
|
||||
let combined_vectors: Vec<Vec<f64>> = new_stored
|
||||
.iter()
|
||||
.map(|e| e.vector.iter().map(|&f| f as f64).collect())
|
||||
.collect();
|
||||
|
||||
let _key = store::store_embeddings(
|
||||
store_, &req.index_name, &combined_chunks, &combined_vectors,
|
||||
).await?;
|
||||
|
||||
// 8. Evict embedding cache — next read will pick up the new file
|
||||
let _ = embedding_cache.evict(&req.index_name).await;
|
||||
|
||||
// 9. Update index registry metadata (row/chunk counts; others unchanged
|
||||
// so we don't disturb the existing entry if present)
|
||||
let _ = try_update_index_meta(index_registry, &req.index_name, new_stored.len()).await;
|
||||
|
||||
// 10. Clear stale flag
|
||||
registry.clear_embeddings_stale(dataset_name).await?;
|
||||
|
||||
let total = new_stored.len();
|
||||
Ok(RefreshResult {
|
||||
index_name: req.index_name.clone(),
|
||||
dataset_name: dataset_name.to_string(),
|
||||
pre_existing_docs: pre_existing_count,
|
||||
dataset_docs,
|
||||
new_docs_embedded: new_docs,
|
||||
new_chunks_embedded: new_chunks,
|
||||
total_embeddings_after: total,
|
||||
duration_secs: t0.elapsed().as_secs_f32(),
|
||||
stale_cleared: true,
|
||||
})
|
||||
}
|
||||
|
||||
/// Best-effort refresh of index registry metadata.
|
||||
/// - If the index already exists, bump the chunk_count.
|
||||
/// - If it's brand new (first-time build via refresh), REGISTER it so
|
||||
/// downstream discovery (profile activation, A/B search) can find it.
|
||||
async fn try_update_index_meta(
|
||||
index_registry: &IndexRegistry,
|
||||
index_name: &str,
|
||||
chunk_count: usize,
|
||||
) -> Result<(), String> {
|
||||
if let Some(mut meta) = index_registry.get(index_name).await {
|
||||
meta.chunk_count = chunk_count;
|
||||
return index_registry.register(meta).await;
|
||||
}
|
||||
// First-time registration — infer reasonable defaults. Convention:
|
||||
// index name `{source}_v1` or `{source}_vN` implies a source dataset
|
||||
// named by stripping the `_vN` suffix. Otherwise use the index name
|
||||
// as the source and let the caller patch later.
|
||||
let source = match index_name.rsplit_once('_') {
|
||||
Some((base, suffix)) if suffix.starts_with('v') && suffix[1..].chars().all(|c| c.is_ascii_digit()) => {
|
||||
base.to_string()
|
||||
}
|
||||
_ => index_name.to_string(),
|
||||
};
|
||||
let meta = crate::index_registry::IndexMeta {
|
||||
index_name: index_name.to_string(),
|
||||
source,
|
||||
model_name: "nomic-embed-text".to_string(),
|
||||
model_version: "latest".to_string(),
|
||||
dimensions: 768,
|
||||
chunk_count,
|
||||
doc_count: chunk_count,
|
||||
chunk_size: 500,
|
||||
overlap: 50,
|
||||
storage_key: format!("vectors/{index_name}.parquet"),
|
||||
created_at: chrono::Utc::now(),
|
||||
build_time_secs: 0.0,
|
||||
chunks_per_sec: 0.0,
|
||||
bucket: "primary".to_string(),
|
||||
vector_backend: shared::types::VectorBackend::Parquet,
|
||||
id_prefix: None,
|
||||
};
|
||||
index_registry.register(meta).await
|
||||
}
|
||||
+11
-2268
File diff suppressed because it is too large
Load Diff
@@ -1,240 +0,0 @@
|
||||
/// Trial journal for HNSW parameter tuning.
|
||||
///
|
||||
/// Every HNSW build+eval is recorded as a Trial. The journal is append-only
|
||||
/// and stored under `_hnsw_trials/{index_name}/` as batched JSONL files —
|
||||
/// an AI agent iterating on configs reads prior trials to decide what to
|
||||
/// try next, and writes a new trial on each attempt.
|
||||
///
|
||||
/// Storage uses the shared `storaged::append_log::AppendLog` so appends are
|
||||
/// write-once (new file per batch) rather than rewriting a single growing
|
||||
/// JSONL on every event. See `append_log.rs` for the full rationale.
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use object_store::ObjectStore;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use storaged::append_log::AppendLog;
|
||||
use storaged::registry::BucketRegistry;
|
||||
|
||||
use crate::index_registry::IndexRegistry;
|
||||
|
||||
/// HNSW build/search parameters the agent can tune.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct HnswConfig {
|
||||
pub ef_construction: usize,
|
||||
pub ef_search: usize,
|
||||
#[serde(default)]
|
||||
pub seed: Option<u64>,
|
||||
}
|
||||
|
||||
impl Default for HnswConfig {
|
||||
/// Production default, locked in 2026-04-16 based on trial grid against
|
||||
/// resumes_100k_v2 (100K vectors, 20 queries, recall@10):
|
||||
/// ec=20 es=30 → recall 0.960, p50 509us, build 8s
|
||||
/// ec=80 es=30 → recall 1.000, p50 873us, build 230s ← sweet spot
|
||||
/// ec=200 es=30 → recall 1.000, p50 874us, build 106s (no recall gain)
|
||||
///
|
||||
/// `ec=80` is the smallest value that reaches 100% recall. Higher values
|
||||
/// waste build time. `es=30` gives faster search than `es=100` with no
|
||||
/// recall penalty at this scale.
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
ef_construction: 80,
|
||||
ef_search: 30,
|
||||
seed: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Metrics collected on every trial. All latencies in microseconds.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TrialMetrics {
|
||||
pub build_time_secs: f32,
|
||||
pub search_latency_p50_us: f32,
|
||||
pub search_latency_p95_us: f32,
|
||||
pub search_latency_p99_us: f32,
|
||||
pub recall_at_k: f32,
|
||||
pub memory_bytes: u64,
|
||||
pub vectors: usize,
|
||||
pub eval_queries: usize,
|
||||
/// Brute-force latency for comparison — how much speedup did HNSW buy us?
|
||||
pub brute_force_latency_us: f32,
|
||||
}
|
||||
|
||||
/// A single tuning attempt.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Trial {
|
||||
pub id: String,
|
||||
pub index_name: String,
|
||||
pub eval_set: String,
|
||||
pub config: HnswConfig,
|
||||
pub metrics: TrialMetrics,
|
||||
pub created_at: DateTime<Utc>,
|
||||
/// Free-form note — the agent can record why it tried this config.
|
||||
#[serde(default)]
|
||||
pub note: Option<String>,
|
||||
}
|
||||
|
||||
impl Trial {
|
||||
pub fn new_id() -> String {
|
||||
format!(
|
||||
"trial-{}-{}",
|
||||
Utc::now().timestamp_millis(),
|
||||
&uuid::Uuid::new_v4().to_string()[..8]
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-index append log, lazy-created on first write.
|
||||
///
|
||||
/// Federation layer 2: the journal resolves each index's bucket from the
|
||||
/// index registry and writes its JSONL batches to THAT bucket, not
|
||||
/// primary. Back-compat is preserved by `IndexMeta::bucket` defaulting
|
||||
/// to "primary" for pre-federation indexes. Indexes the registry has
|
||||
/// never heard of (edge case — trials run before first register) fall
|
||||
/// through to primary as well.
|
||||
#[derive(Clone)]
|
||||
pub struct TrialJournal {
|
||||
buckets: Arc<BucketRegistry>,
|
||||
index_registry: IndexRegistry,
|
||||
/// Cache per (bucket, index) AppendLog so the in-memory buffer persists
|
||||
/// across calls. Keyed by `(bucket, index_name)` so moving an index
|
||||
/// between buckets is clean — the old journal stays intact.
|
||||
logs: Arc<RwLock<HashMap<(String, String), Arc<AppendLog>>>>,
|
||||
}
|
||||
|
||||
impl TrialJournal {
|
||||
pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
|
||||
Self {
|
||||
buckets,
|
||||
index_registry,
|
||||
logs: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
fn prefix(index_name: &str) -> String {
|
||||
format!("_hnsw_trials/{}", index_name)
|
||||
}
|
||||
|
||||
/// Resolve which bucket holds this index's trial artifacts.
|
||||
/// Falls back to primary for indexes without recorded metadata.
|
||||
async fn bucket_for(&self, index_name: &str) -> String {
|
||||
self.index_registry
|
||||
.get(index_name)
|
||||
.await
|
||||
.map(|m| m.bucket)
|
||||
.unwrap_or_else(|| "primary".to_string())
|
||||
}
|
||||
|
||||
async fn log_for(&self, index_name: &str) -> Result<Arc<AppendLog>, String> {
|
||||
let bucket = self.bucket_for(index_name).await;
|
||||
let key = (bucket.clone(), index_name.to_string());
|
||||
|
||||
if let Some(log) = self.logs.read().await.get(&key) {
|
||||
return Ok(log.clone());
|
||||
}
|
||||
let mut guard = self.logs.write().await;
|
||||
if let Some(log) = guard.get(&key) {
|
||||
return Ok(log.clone());
|
||||
}
|
||||
let store = self.buckets.get(&bucket)?;
|
||||
// Trials arrive one at a time during human/agent iteration — a low
|
||||
// threshold gives "hit /trials and see my latest attempt" immediacy
|
||||
// without creating one file per event.
|
||||
let log = Arc::new(
|
||||
AppendLog::new(store, Self::prefix(index_name))
|
||||
.with_flush_threshold(4),
|
||||
);
|
||||
guard.insert(key, log.clone());
|
||||
Ok(log)
|
||||
}
|
||||
|
||||
/// Append a trial record. In-memory buffered; persisted in batches.
|
||||
pub async fn append(&self, trial: &Trial) -> Result<(), String> {
|
||||
let line = serde_json::to_vec(trial).map_err(|e| e.to_string())?;
|
||||
let log = self.log_for(&trial.index_name).await?;
|
||||
log.append(line).await
|
||||
}
|
||||
|
||||
/// Read all trials for an index (flushed batches + unflushed buffer).
|
||||
pub async fn list(&self, index_name: &str) -> Result<Vec<Trial>, String> {
|
||||
let log = self.log_for(index_name).await?;
|
||||
let lines = log.read_all().await?;
|
||||
let mut trials = Vec::with_capacity(lines.len());
|
||||
for line in lines {
|
||||
match serde_json::from_slice::<Trial>(&line) {
|
||||
Ok(t) => trials.push(t),
|
||||
Err(e) => tracing::warn!("trial journal: skip malformed line: {e}"),
|
||||
}
|
||||
}
|
||||
Ok(trials)
|
||||
}
|
||||
|
||||
/// Explicit flush for callers that want write-through semantics
|
||||
/// (e.g. an agent that wants to commit a trial before querying stats).
|
||||
pub async fn flush(&self, index_name: &str) -> Result<(), String> {
|
||||
let log = self.log_for(index_name).await?;
|
||||
log.flush().await
|
||||
}
|
||||
|
||||
/// Compact all batch files for an index into one.
|
||||
pub async fn compact(&self, index_name: &str) -> Result<storaged::append_log::CompactStats, String> {
|
||||
let log = self.log_for(index_name).await?;
|
||||
log.compact().await
|
||||
}
|
||||
|
||||
/// Current champion for an index by the named metric.
|
||||
/// Valid metrics: `recall`, `latency`, `pareto`.
|
||||
///
|
||||
/// The `pareto` strategy is a placeholder — J should tune the scoring
|
||||
/// function to match what matters in production. Right now it's a simple
|
||||
/// weighted sum.
|
||||
pub async fn best(
|
||||
&self,
|
||||
index_name: &str,
|
||||
metric: &str,
|
||||
) -> Result<Option<Trial>, String> {
|
||||
let trials = self.list(index_name).await?;
|
||||
if trials.is_empty() {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let best = match metric {
|
||||
"recall" => trials
|
||||
.into_iter()
|
||||
.max_by(|a, b| {
|
||||
a.metrics
|
||||
.recall_at_k
|
||||
.partial_cmp(&b.metrics.recall_at_k)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
})
|
||||
.unwrap(),
|
||||
"latency" => trials
|
||||
.into_iter()
|
||||
.min_by(|a, b| {
|
||||
a.metrics
|
||||
.search_latency_p95_us
|
||||
.partial_cmp(&b.metrics.search_latency_p95_us)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
})
|
||||
.unwrap(),
|
||||
"pareto" | _ => trials
|
||||
.into_iter()
|
||||
.max_by(|a, b| pareto_score(a).partial_cmp(&pareto_score(b)).unwrap())
|
||||
.unwrap(),
|
||||
};
|
||||
Ok(Some(best))
|
||||
}
|
||||
}
|
||||
|
||||
/// Simple Pareto-style score: reward recall, penalize p95 latency.
|
||||
/// Tunable — J should swap this in production to match what matters.
|
||||
fn pareto_score(t: &Trial) -> f32 {
|
||||
// Recall is [0, 1]. Latency is us — assume 100us baseline.
|
||||
let recall = t.metrics.recall_at_k;
|
||||
let latency_penalty = (t.metrics.search_latency_p95_us / 1000.0).min(1.0); // cap at 1ms
|
||||
recall - 0.2 * latency_penalty
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
{
|
||||
"id": "03c62234-f9f8-40e9-a27e-d5e09ab2713d",
|
||||
"name": "kb_pipeline_runs",
|
||||
"schema_fingerprint": "c019b81feb58ff2aefe4cbe700056a100e25716f2aa8e8415ba8f20656812f75",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_pipeline_runs.parquet",
|
||||
"size_bytes": 1518415,
|
||||
"created_at": "2026-04-17T02:54:43.281447563Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.281448143Z",
|
||||
"updated_at": "2026-04-17T05:56:44.999492793Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pipeline",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "topic",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "steps",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "result",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "completed_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776403326289",
|
||||
"ingest_timestamp": "2026-04-17T05:22:06.289208160Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 195,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "0ae8f29b-bf70-4c9d-8b64-6ea7cb7f76a1",
|
||||
"name": "rust_in_tech_stack",
|
||||
"schema_fingerprint": "314ee8ede68287240d7cec1403d252787c8ceb4ee2afa72cffcb0853ac3ac889",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/rust_in_tech_stack.parquet",
|
||||
"size_bytes": 99623,
|
||||
"created_at": "2026-03-28T03:32:50.769271381Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:32:50.769272195Z",
|
||||
"updated_at": "2026-03-28T03:32:50.769710478Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "Rust in Tech Stack.pdf",
|
||||
"ingest_job": "ingest-1774668770769",
|
||||
"ingest_timestamp": "2026-03-28T03:32:50.769271381Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 49
|
||||
}
|
||||
@@ -1,159 +0,0 @@
|
||||
{
|
||||
"id": "0efdef2d-52bc-4154-82d4-056bb3a1915d",
|
||||
"name": "ethereal_workers",
|
||||
"schema_fingerprint": "9a1286ffada5390b459b217f56263e5ebffec1520ea29bab0be2efc6d5381adc",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/ethereal_workers.parquet",
|
||||
"size_bytes": 6991716,
|
||||
"created_at": "2026-04-17T03:23:29.399966984Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T03:23:29.399967772Z",
|
||||
"updated_at": "2026-04-17T03:23:29.400227081Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "worker_id",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "certifications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archetype",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reliability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responsiveness",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "engagement",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "compliance",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "availability",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "communications",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_text",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "ethereal_workers.csv",
|
||||
"ingest_job": "ingest-1776396209399",
|
||||
"ingest_timestamp": "2026-04-17T03:23:29.399966984Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 10000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,97 +1,23 @@
|
||||
{
|
||||
"id": "0fd78303-9ad4-45fd-90d7-db95607d9ab1",
|
||||
"name": "timesheets",
|
||||
"schema_fingerprint": "806946533e133a4dd032f500644c9f702a46151fbb5bbd3b503c207dac86893b",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/timesheets.parquet",
|
||||
"size_bytes": 17539932,
|
||||
"created_at": "2026-03-27T14:42:43.922019299Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:43.922025703Z",
|
||||
"updated_at": "2026-04-17T02:45:58.282412872Z",
|
||||
"updated_at": "2026-03-27T14:42:43.922025703Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "timesheet_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "placement_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hours_regular",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hours_overtime",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_total",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_total",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "week_ending",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "approved",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 1000000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -1,104 +1,23 @@
|
||||
{
|
||||
"id": "1339f3d6-7677-47fb-8182-5f8e43f27cde",
|
||||
"name": "job_orders",
|
||||
"schema_fingerprint": "579a0593ca7c0e22ba53702821757f189077e1ddff7fd592c0097feed446a99a",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/job_orders.parquet",
|
||||
"size_bytes": 905534,
|
||||
"created_at": "2026-03-27T14:42:38.935718195Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.935724058Z",
|
||||
"updated_at": "2026-04-17T02:45:57.664902260Z",
|
||||
"updated_at": "2026-03-27T14:42:38.935724058Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "job_order_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "description",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 15000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -1,110 +0,0 @@
|
||||
{
|
||||
"id": "14698884-071c-4adb-ae50-cfb8d885656c",
|
||||
"name": "kb_response_cache",
|
||||
"schema_fingerprint": "c90d7be310b5025d2c4d398cf07692d8d9bb46ed591c0a87b339bafcac9ddeed",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_response_cache.parquet",
|
||||
"size_bytes": 8360233,
|
||||
"created_at": "2026-04-17T02:54:43.145342968Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.145343876Z",
|
||||
"updated_at": "2026-04-17T02:54:43.145629696Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "cache_key",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "prompt",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_run_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responses",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "hit_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776394483145",
|
||||
"ingest_timestamp": "2026-04-17T02:54:43.145342968Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 195,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
{
|
||||
"id": "15d889ef-2dba-4d1f-8ce0-22f7def35155",
|
||||
"name": "scanned_resume",
|
||||
"schema_fingerprint": "d3bc44bcafc990711442db3eecf25c0758232e3fccf6345200b1813fae5868c2",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/scanned_resume.parquet",
|
||||
"size_bytes": 3097,
|
||||
"created_at": "2026-04-17T01:44:33.927277187Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:44:33.927280048Z",
|
||||
"updated_at": "2026-04-17T01:44:33.927528900Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "text_content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "test_scanned_resume.pdf",
|
||||
"ingest_job": "ingest-1776390273927",
|
||||
"ingest_timestamp": "2026-04-17T01:44:33.927277187Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 1,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,90 +0,0 @@
|
||||
{
|
||||
"id": "17119168-cfd2-43b2-a4dc-ff2a2c5fd086",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4583,
|
||||
"created_at": "2026-04-17T01:35:00.165218923Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:35:00.165220510Z",
|
||||
"updated_at": "2026-04-17T05:56:44.964702014Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,132 +1,23 @@
|
||||
{
|
||||
"id": "1d8a065e-59c1-45ce-967b-398bc8370cbb",
|
||||
"name": "candidates",
|
||||
"schema_fingerprint": "206360ab312e4c13679ed0ab4ccb3702250cc3e2176cc669d41471584c556c73",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/candidates.parquet",
|
||||
"size_bytes": 10592165,
|
||||
"created_at": "2026-03-27T14:42:38.823368759Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.823374843Z",
|
||||
"updated_at": "2026-04-17T02:45:57.722237378Z",
|
||||
"updated_at": "2026-03-27T14:42:38.823374843Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "first_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "last_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "phone",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "resume_summary",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "min_pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "years_experience",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 100000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"id": "24a05512-dc06-4884-9889-2ea978abdaec",
|
||||
"name": "processed_extractions",
|
||||
"schema_fingerprint": "c2853ba4c0cc5a897b37680fefea2708f51d51774226b00b72444e79d594c874",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/processed_extractions.parquet",
|
||||
"size_bytes": 28175,
|
||||
"created_at": "2026-03-28T05:16:57.437041979Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T05:16:57.437042648Z",
|
||||
"updated_at": "2026-03-28T05:16:57.437403737Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "processed_extractions.csv",
|
||||
"ingest_job": "ingest-1774675017437",
|
||||
"ingest_timestamp": "2026-03-28T05:16:57.437041979Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 543
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
{
|
||||
"id": "36b0f141-93a1-4776-beef-b0b8d17a12a9",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4583,
|
||||
"created_at": "2026-04-17T01:35:01.636480725Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:35:01.636481698Z",
|
||||
"updated_at": "2026-04-17T01:35:01.636776595Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "mysql",
|
||||
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
|
||||
"ingest_job": "scheduled-mysql-1776389701636",
|
||||
"ingest_timestamp": "2026-04-17T01:35:01.636480725Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "30c0e31f0963e6f4af02131bbb9ea246fbbd068b849b833565a4b28211fbc90b",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/users.parquet",
|
||||
"size_bytes": 2012,
|
||||
"created_at": "2026-03-28T01:38:59.904968123Z"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/meta_runs.parquet",
|
||||
"size_bytes": 729773,
|
||||
"created_at": "2026-03-28T01:38:57.380576453Z"
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"id": "3ead1a7a-810b-49d1-add8-752bab52dfae",
|
||||
"name": "processed_validated",
|
||||
"schema_fingerprint": "8078de214059a2a4582203b32c1172fcc311c2d3c762a3b075c07268308f1841",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/processed_validated.parquet",
|
||||
"size_bytes": 15271,
|
||||
"created_at": "2026-03-28T05:16:57.439472748Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T05:16:57.439473552Z",
|
||||
"updated_at": "2026-03-28T05:16:57.439749766Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "processed_validated.csv",
|
||||
"ingest_job": "ingest-1774675017439",
|
||||
"ingest_timestamp": "2026-03-28T05:16:57.439472748Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 60
|
||||
}
|
||||
@@ -1,110 +0,0 @@
|
||||
{
|
||||
"id": "443d63f1-b0ed-4d4b-8e5a-ce59c097b97b",
|
||||
"name": "kb_memory_entries",
|
||||
"schema_fingerprint": "15dbebd0abb906577e11cbb73083abeb8961b8c603a2472cafa8c46100c3fb1c",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_memory_entries.parquet",
|
||||
"size_bytes": 8795,
|
||||
"created_at": "2026-04-17T02:54:43.330897823Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.330898330Z",
|
||||
"updated_at": "2026-04-17T02:54:43.330972517Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "session_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "embedding",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "metadata",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "importance_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "accessed_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "access_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776394483330",
|
||||
"ingest_timestamp": "2026-04-17T02:54:43.330897823Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 13,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "54cc1ae9-14fe-40f6-88e4-7c623f57b577",
|
||||
"name": "crm_development_options",
|
||||
"schema_fingerprint": "bb04328fa99161403fa0993f59bf14adcf08eb551c7b0c5e9b9e305e5e0c8556",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/crm_development_options.parquet",
|
||||
"size_bytes": 50528,
|
||||
"created_at": "2026-03-28T03:30:29.036504470Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:30:29.036504841Z",
|
||||
"updated_at": "2026-03-28T03:30:29.036642992Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "CRM Development Options.pdf",
|
||||
"ingest_job": "ingest-1774668629036",
|
||||
"ingest_timestamp": "2026-03-28T03:30:29.036504470Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 25
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
{
|
||||
"id": "57e557a8-4754-47c9-9636-e01d39fc20f2",
|
||||
"name": "animals",
|
||||
"schema_fingerprint": "b024b907f151889aca5407fc46fe1821d2600f587cb1a26742843597a66436f1",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/animals.parquet",
|
||||
"size_bytes": 1107,
|
||||
"created_at": "2026-04-16T13:47:57.033455363Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-16T13:47:57.033456007Z",
|
||||
"updated_at": "2026-04-16T13:47:57.033631067Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "animal_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "species",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "habitat",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "animals.csv",
|
||||
"ingest_job": "ingest-1776347277033",
|
||||
"ingest_timestamp": "2026-04-16T13:47:57.033455363Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 3,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,61 +0,0 @@
|
||||
{
|
||||
"id": "5a35646f-b357-4636-b88f-203aba0d7435",
|
||||
"name": "people_test",
|
||||
"schema_fingerprint": "bba35b6cd1e399dc0504c760c767ab391f7ed81a41573c15b3bd4eb0e3bcc382",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "testing",
|
||||
"key": "datasets/people_test.parquet",
|
||||
"size_bytes": 1402,
|
||||
"created_at": "2026-04-16T13:47:32.669517398Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-16T13:47:32.669518210Z",
|
||||
"updated_at": "2026-04-16T13:47:32.669787451Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "person_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "people.csv",
|
||||
"ingest_job": "ingest-1776347252669",
|
||||
"ingest_timestamp": "2026-04-16T13:47:32.669517398Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 3,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "5d4bbdb1-fb53-4cea-86d5-309fe067111c",
|
||||
"name": "legacy_staffing_crm_search",
|
||||
"schema_fingerprint": "87b3fa58cae74468bf7fbbe32397b1a88bc281832bb0226b9eae5ce2501cda4e",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/legacy_staffing_crm_search.parquet",
|
||||
"size_bytes": 2722188,
|
||||
"created_at": "2026-03-28T03:34:14.444384147Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:34:14.444384964Z",
|
||||
"updated_at": "2026-03-28T03:34:14.444631183Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "Legacy Staffing CRM Search.pdf",
|
||||
"ingest_job": "ingest-1774668854444",
|
||||
"ingest_timestamp": "2026-03-28T03:34:14.444384147Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 1
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "667478c3-47f9-4076-bab2-8293999d4db2",
|
||||
"name": "knowledge_insights",
|
||||
"schema_fingerprint": "36896987c7df0e78f5282208e3b8bbbaf9c017588d0bb42a20a7dbe97d8d4fa0",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/knowledge_insights.parquet",
|
||||
"size_bytes": 13099,
|
||||
"created_at": "2026-03-28T04:06:05.361963023Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T04:06:05.361963273Z",
|
||||
"updated_at": "2026-03-28T04:06:05.362127023Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "insight",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "knowledge_insights.csv",
|
||||
"ingest_job": "ingest-1774670765361",
|
||||
"ingest_timestamp": "2026-03-28T04:06:05.361963023Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 100
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
{
|
||||
"id": "765985e6-ea16-46b5-a6c0-ddaff346827f",
|
||||
"name": "kb_meta_pipelines",
|
||||
"schema_fingerprint": "cabe1d7fc442e1dfcfaabf663509d590c6edc84b445a91acaf0ae68b94aff518",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_meta_pipelines.parquet",
|
||||
"size_bytes": 32085,
|
||||
"created_at": "2026-04-17T02:54:43.307966396Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.307966799Z",
|
||||
"updated_at": "2026-04-17T04:23:16.140272666Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "data_source",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "stages",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "current_stage",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "total_stages",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "results",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "best_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "iterations",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776399796139",
|
||||
"ingest_timestamp": "2026-04-17T04:23:16.139992290Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 6,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "799c825c-950e-480e-a38e-4407377e43e3",
|
||||
"name": "knowledge_topics",
|
||||
"schema_fingerprint": "356c37c8fef56ce7f7882c80f1301cc2f70bcbf874719cb0628b42179874494c",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/knowledge_topics.parquet",
|
||||
"size_bytes": 14252,
|
||||
"created_at": "2026-03-28T04:06:05.358571822Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T04:06:05.358572461Z",
|
||||
"updated_at": "2026-03-28T04:06:05.358833280Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "topic",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "knowledge_topics.csv",
|
||||
"ingest_job": "ingest-1774670765358",
|
||||
"ingest_timestamp": "2026-03-28T04:06:05.358571822Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 496
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
{
|
||||
"id": "7ad27f97-622e-49e3-9c38-327cb2334fa1",
|
||||
"name": "kb_team_runs",
|
||||
"schema_fingerprint": "d704ee2b9b434774aed2258da2fddcdcbab226547a011ba24d4281253657bdd3",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_team_runs.parquet",
|
||||
"size_bytes": 19185025,
|
||||
"created_at": "2026-04-17T02:54:43.043742588Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.043743165Z",
|
||||
"updated_at": "2026-04-17T02:54:43.043968762Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "mode",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "prompt",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "responses",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "models_used",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "archived",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tags",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "quality_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score_method",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score_metadata",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776394483043",
|
||||
"ingest_timestamp": "2026-04-17T02:54:43.043742588Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 588,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "7ca19535-838e-4f79-aa6f-d8ebaa2e6edc",
|
||||
"name": "code_sorting_assistance",
|
||||
"schema_fingerprint": "47e317d38bbad23688f6c274cbecc3dfd7076e42a6a0d09002a0bef835789bc2",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/code_sorting_assistance.parquet",
|
||||
"size_bytes": 122034,
|
||||
"created_at": "2026-03-28T03:31:08.585681049Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:31:08.585682007Z",
|
||||
"updated_at": "2026-03-28T03:31:08.585978498Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "Code Sorting Assistance.pdf",
|
||||
"ingest_job": "ingest-1774668668585",
|
||||
"ingest_timestamp": "2026-03-28T03:31:08.585681049Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 70
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "80eaf246-778b-4987-9bef-62a6cccb6021",
|
||||
"name": "knowledge_questions",
|
||||
"schema_fingerprint": "d86b4ddf7a9694118aeff3d8aa41312e0c3091b0f15cad1079ee9561c29f9c91",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/knowledge_questions.parquet",
|
||||
"size_bytes": 17253,
|
||||
"created_at": "2026-03-28T04:06:05.363389625Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T04:06:05.363390263Z",
|
||||
"updated_at": "2026-03-28T04:06:05.363439745Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "conversation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "question",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "csv",
|
||||
"source_file": "knowledge_questions.csv",
|
||||
"ingest_job": "ingest-1774670765363",
|
||||
"ingest_timestamp": "2026-03-28T04:06:05.363389625Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 200
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
{
|
||||
"id": "8192d934-fc90-46dd-b8bd-c443a4743b19",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4519,
|
||||
"created_at": "2026-04-17T01:00:48.361437263Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:00:48.361437860Z",
|
||||
"updated_at": "2026-04-17T01:00:48.361681724Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": "pii",
|
||||
"description": "",
|
||||
"is_pii": true
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "mysql",
|
||||
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
|
||||
"ingest_job": "mysql-stream-1776387648361",
|
||||
"ingest_timestamp": "2026-04-17T01:00:48.361437263Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 10,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,90 +0,0 @@
|
||||
{
|
||||
"id": "8621894c-f9c6-4eea-8e71-e80b9cdc9a36",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4583,
|
||||
"created_at": "2026-04-17T01:34:30.158848713Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:34:30.158849587Z",
|
||||
"updated_at": "2026-04-17T05:56:27.642214035Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
{
|
||||
"id": "8f37454b-27c8-4f61-aeca-8e48070db552",
|
||||
"name": "demo_customers",
|
||||
"schema_fingerprint": "7af9708f2366f790e0ae03db6f681f5a4d4b8af4d1527b091bedfee1be81d304",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/demo_customers.parquet",
|
||||
"size_bytes": 4519,
|
||||
"created_at": "2026-04-17T01:34:00.160615345Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T01:34:00.160616631Z",
|
||||
"updated_at": "2026-04-17T02:45:58.283076939Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": "pii",
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "email",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "tier",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "lifetime_spend",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_active",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "signed_up_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "notes",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "mysql",
|
||||
"source_file": "dsn: mysql://lh:***@127.0.0.1:3306/lh_demo",
|
||||
"ingest_job": "scheduled-mysql-1776389700165",
|
||||
"ingest_timestamp": "2026-04-17T01:35:00.165218923Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "8f5c82bf-93ec-4ce7-b2d1-61e559242add",
|
||||
"name": "llms3_update_prompt",
|
||||
"schema_fingerprint": "77cfcdb5e22f095302bb816abd4e33380a6b357cd7d23efee5076083020fbf32",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/llms3_update_prompt.parquet",
|
||||
"size_bytes": 39396,
|
||||
"created_at": "2026-03-28T03:28:04.752204191Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:28:04.752204734Z",
|
||||
"updated_at": "2026-03-28T03:28:04.752281926Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "LLMS3 Update Prompt.pdf",
|
||||
"ingest_job": "ingest-1774668484752",
|
||||
"ingest_timestamp": "2026-03-28T03:28:04.752204191Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 6
|
||||
}
|
||||
@@ -1,69 +1,23 @@
|
||||
{
|
||||
"id": "94a8bd16-6756-43af-b951-09a9e6b8300f",
|
||||
"name": "clients",
|
||||
"schema_fingerprint": "1ba20a49d967c85c41640a9b9a61287ae9fa483daf6c71a197591fe3228c76d2",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/clients.parquet",
|
||||
"size_bytes": 21971,
|
||||
"created_at": "2026-03-27T14:42:38.830329102Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:38.830331694Z",
|
||||
"updated_at": "2026-04-17T02:45:58.138164775Z",
|
||||
"updated_at": "2026-03-27T14:42:38.830331694Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "company_name",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "vertical",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "state",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "zip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 2000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -1,83 +1,23 @@
|
||||
{
|
||||
"id": "9c4d9116-1d9d-4afd-a8d1-c514a678e5fa",
|
||||
"name": "call_log",
|
||||
"schema_fingerprint": "d1f82ed776afc484747f8f58b30ccd00d10bca73703b1479a67647f98eeff7e2",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/call_log.parquet",
|
||||
"size_bytes": 35951077,
|
||||
"created_at": "2026-03-27T14:42:47.395548205Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:47.395555326Z",
|
||||
"updated_at": "2026-04-17T02:45:58.137293143Z",
|
||||
"updated_at": "2026-03-27T14:42:47.395555326Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "call_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "from_number",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "to_number",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_seconds",
|
||||
"data_type": "Int64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "timestamp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recruiter",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "disposition",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 800000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
{
|
||||
"id": "a5858f94-267f-4382-ba32-0934e1b984f7",
|
||||
"name": "kb_lab_trials",
|
||||
"schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_lab_trials.parquet",
|
||||
"size_bytes": 68221,
|
||||
"created_at": "2026-04-17T02:54:43.423304357Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.423304931Z",
|
||||
"updated_at": "2026-04-17T04:23:16.166692762Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "experiment_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "trial_num",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config_diff",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "config_snapshot",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "scores",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "avg_score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "improved",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776399796166",
|
||||
"ingest_timestamp": "2026-04-17T04:23:16.166457198Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 41,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,75 +0,0 @@
|
||||
{
|
||||
"id": "ab2b610a-cee7-40e1-9dab-c709e2292709",
|
||||
"name": "kb_self_reports",
|
||||
"schema_fingerprint": "76382b028b92bb0b361306dced3d773af0ee8de6aa160cedfc4d649f2860167a",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_self_reports.parquet",
|
||||
"size_bytes": 51134,
|
||||
"created_at": "2026-04-17T02:54:43.335457986Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.335458749Z",
|
||||
"updated_at": "2026-04-17T02:54:43.335536869Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "report_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "report",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "data_size",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776394483335",
|
||||
"ingest_timestamp": "2026-04-17T02:54:43.335457986Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 11,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,243 +0,0 @@
|
||||
{
|
||||
"id": "b076cff9-c522-48fc-b892-b7cba7b29c9b",
|
||||
"name": "kb_threat_intel",
|
||||
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_threat_intel.parquet",
|
||||
"size_bytes": 247112,
|
||||
"created_at": "2026-04-17T02:54:43.321496407Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.321497035Z",
|
||||
"updated_at": "2026-04-17T02:54:43.321758360Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "ip",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "threat_level",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "classification",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "confidence",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "summary",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "indicators",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recommendation",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pattern",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "attack_type",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "likely_automated",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "country",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "country_code",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "city",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "isp",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "org",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "asn",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_proxy",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "is_hosting",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "open_ports",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklist_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklist_total",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "blocklists_blocked",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "reverse_dns",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "traceroute",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "log_count",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "banned",
|
||||
"data_type": "Boolean",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "enriched_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "updated_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "raw_data",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776394483321",
|
||||
"ingest_timestamp": "2026-04-17T02:54:43.321496407Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 54,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
{
|
||||
"id": "ce3fe55b-61cd-4ac4-95de-b91f9186d6e3",
|
||||
"name": "kb_meta_runs",
|
||||
"schema_fingerprint": "68f2c0d7a3ceb0aaa3c17c64900704519c72d213161bc9e5179c42ee53f6d0df",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"key": "datasets/kb_meta_runs.parquet",
|
||||
"size_bytes": 886387,
|
||||
"created_at": "2026-04-17T02:54:43.299467359Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-04-17T02:54:43.299468094Z",
|
||||
"updated_at": "2026-04-17T04:23:16.134134852Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pipeline_id",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "iteration",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "stage_results",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "final_output",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "score",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "model_config",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "duration_ms",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "created_at",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://kbuser@localhost:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776399796133",
|
||||
"ingest_timestamp": "2026-04-17T04:23:16.133868872Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 48,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
}
|
||||
@@ -1,83 +1,23 @@
|
||||
{
|
||||
"id": "d35c7941-37e2-4bde-8226-5cf69c74931a",
|
||||
"name": "placements",
|
||||
"schema_fingerprint": "962535a1810c07183a882c15a76a11a02ad3afef844e71975b0eaa48943487f6",
|
||||
"schema_fingerprint": "auto",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/placements.parquet",
|
||||
"size_bytes": 1213820,
|
||||
"created_at": "2026-03-27T14:42:39.040983450Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-27T14:42:39.040989351Z",
|
||||
"updated_at": "2026-04-17T02:45:58.147407036Z",
|
||||
"updated_at": "2026-03-27T14:42:39.040989351Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "placement_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "job_order_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "client_id",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "bill_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "pay_rate",
|
||||
"data_type": "Float64",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "recruiter",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "status",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"columns": [],
|
||||
"lineage": null,
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 50000,
|
||||
"last_embedded_at": null,
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": null
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"id": "d609d775-1a8e-4114-8aee-3238017229a7",
|
||||
"name": "hybrid_indexing_setup",
|
||||
"schema_fingerprint": "dfb09f4395d131e5fda5dab65f8c9dbe395d46d0e828f504e053cdb1cf23f24b",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "data",
|
||||
"key": "datasets/hybrid_indexing_setup.parquet",
|
||||
"size_bytes": 131530,
|
||||
"created_at": "2026-03-28T03:30:28.978970613Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T03:30:28.978971805Z",
|
||||
"updated_at": "2026-03-28T03:30:28.979244113Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
"columns": [
|
||||
{
|
||||
"name": "source_file",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "conversation_title",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "turn_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "role",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"data_type": "Utf8",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
},
|
||||
{
|
||||
"name": "page_number",
|
||||
"data_type": "Int32",
|
||||
"sensitivity": null,
|
||||
"description": "",
|
||||
"is_pii": false
|
||||
}
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "pdf",
|
||||
"source_file": "Hybrid Indexing Setup.pdf",
|
||||
"ingest_job": "ingest-1774668628978",
|
||||
"ingest_timestamp": "2026-03-28T03:30:28.978970613Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 69
|
||||
}
|
||||
@@ -4,7 +4,7 @@
|
||||
"schema_fingerprint": "1d5782349402439a7e44efd0ccab9ae64ac3044221adef9e828b60b8bbb44dd5",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/lab_trials.parquet",
|
||||
"size_bytes": 64646,
|
||||
"created_at": "2026-03-28T01:14:03.026116573Z"
|
||||
|
||||
@@ -4,14 +4,14 @@
|
||||
"schema_fingerprint": "df1e126046147b3de42086880e10c3501a3a615ecddf336bc24957a24c321241",
|
||||
"objects": [
|
||||
{
|
||||
"bucket": "primary",
|
||||
"bucket": "data",
|
||||
"key": "datasets/threat_intel.parquet",
|
||||
"size_bytes": 111130,
|
||||
"created_at": "2026-03-28T01:14:03.054140697Z"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-03-28T01:14:03.054141294Z",
|
||||
"updated_at": "2026-04-17T05:56:45.009810081Z",
|
||||
"updated_at": "2026-03-28T01:14:03.054427047Z",
|
||||
"description": "",
|
||||
"owner": "",
|
||||
"sensitivity": null,
|
||||
@@ -229,15 +229,12 @@
|
||||
],
|
||||
"lineage": {
|
||||
"source_system": "postgresql",
|
||||
"source_file": "dsn: postgresql://postgres@127.0.0.1:5432/knowledge_base",
|
||||
"ingest_job": "pg-stream-1776326353882",
|
||||
"ingest_timestamp": "2026-04-16T07:59:13.882669337Z",
|
||||
"source_file": "127.0.0.1:5432/knowledge_base.threat_intel",
|
||||
"ingest_job": "pg-import-1774660443054",
|
||||
"ingest_timestamp": "2026-03-28T01:14:03.054140697Z",
|
||||
"parent_datasets": []
|
||||
},
|
||||
"freshness": null,
|
||||
"tags": [],
|
||||
"row_count": 20,
|
||||
"last_embedded_at": "2026-04-16T15:08:32.348412159Z",
|
||||
"embedding_stale_since": null,
|
||||
"embedding_refresh_policy": null
|
||||
"row_count": 20
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user