PRD v2: production roadmap with ingest, vector search, hot cache phases
- Phase 6: Ingest pipeline (CSV/JSON → schema detect → Parquet → catalog) - Phase 7: Vector index + RAG (embed → HNSW → semantic search → LLM answer) - Phase 8: Hot cache + incremental updates (MemTable, delta files, merge-on-read) - ADR-008 through ADR-011: embeddings as Parquet, delta files not Delta Lake, schema defaults to string, not a CRM replacement - Staffing company reference dataset (286K rows, 7 tables) - Honest risk assessment: vector search at scale and incremental updates are hard Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
+161
-79
@@ -1,6 +1,6 @@
|
||||
# PRD: Lakehouse — Rust-First Object Storage System
|
||||
|
||||
**Status:** Active
|
||||
**Status:** Active — Phase 0-5 complete, entering production path
|
||||
**Created:** 2026-03-27
|
||||
**Owner:** J
|
||||
|
||||
@@ -8,20 +8,23 @@
|
||||
|
||||
## Problem
|
||||
|
||||
Traditional data platforms couple storage, compute, and metadata into monolithic databases. This creates vendor lock-in, scaling bottlenecks, and opaque data access. AI workloads bolt onto these systems awkwardly, sharing resources with transactional queries.
|
||||
Legacy data systems silo information across CRMs, databases, spreadsheets, and file shares. Querying across them requires manual ETL, pre-defined schemas, and expensive database licenses. When AI enters the picture, these systems can't handle the dual requirement of fast analytical queries AND semantic retrieval over unstructured text.
|
||||
|
||||
A staffing company (our reference case) has candidate records in an ATS, client data in a CRM, timesheets in billing software, call logs from a phone system, and email records from Exchange. Answering "find every Java developer in Chicago who was called 5+ times but never placed" requires querying across all of them — and no single system can do it.
|
||||
|
||||
We need a system where:
|
||||
- Object storage is the source of truth (not a database)
|
||||
- Metadata, access, and execution are controlled by Rust services
|
||||
- Queries run directly over object storage via Arrow/Parquet
|
||||
- AI inference is isolated and swappable
|
||||
- The entire system is rebuildable from repository + docs alone
|
||||
- Any data source (CSV, DB export, PDF, JSON) can be ingested without pre-defined schemas
|
||||
- Structured data is queryable via SQL at scale (millions of rows, sub-second)
|
||||
- Unstructured data is searchable via AI embeddings (semantic retrieval)
|
||||
- An LLM can answer natural language questions against all of it
|
||||
- Everything runs locally — no cloud APIs, total data privacy
|
||||
- The system is rebuildable from repository + object storage alone
|
||||
|
||||
---
|
||||
|
||||
## Solution
|
||||
|
||||
A modular Rust service mesh over S3-compatible object storage.
|
||||
A modular Rust service mesh over S3-compatible object storage, with a local AI layer for embeddings and generation.
|
||||
|
||||
### Locked Stack
|
||||
|
||||
@@ -30,14 +33,15 @@ A modular Rust service mesh over S3-compatible object storage.
|
||||
| Frontend | Dioxus | Yes |
|
||||
| API | Axum + Tokio | Yes |
|
||||
| Object Storage Interface | Apache Arrow `object_store` | Yes |
|
||||
| Storage Backend | RustFS (fallback: SeaweedFS) | Yes |
|
||||
| Storage Backend | LocalFileSystem → RustFS → S3 | Yes |
|
||||
| Query Engine | DataFusion | Yes |
|
||||
| Data Format | Parquet + Arrow | Yes |
|
||||
| RPC (internal) | tonic (gRPC) | Yes |
|
||||
| AI Runtime | Ollama (local models) | Yes |
|
||||
| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes |
|
||||
| Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** |
|
||||
|
||||
No new frameworks. No exceptions.
|
||||
No new frameworks without documented ADR.
|
||||
|
||||
---
|
||||
|
||||
@@ -47,84 +51,160 @@ No new frameworks. No exceptions.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| **gateway** | HTTP ingress, routing, auth envelope, middleware |
|
||||
| **catalogd** | Metadata control plane — dataset registry, schema versions, manifest index |
|
||||
| **storaged** | Object I/O — read/write/list/delete via `object_store` crate |
|
||||
| **queryd** | SQL execution — DataFusion over registered Parquet datasets |
|
||||
| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits |
|
||||
| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests |
|
||||
| **storaged** | Object I/O — read/write/list/delete via `object_store` |
|
||||
| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache |
|
||||
| **ingestd** | *NEW* — Ingest pipeline: CSV/JSON/DB → normalize → Parquet → catalog |
|
||||
| **vectord** | *NEW* — Embedding store + vector index: chunk → embed → index → search |
|
||||
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
|
||||
| **ui** | Dioxus frontend — dataset browser, query editor, results viewer |
|
||||
| **shared** | Types, errors, Arrow helpers, protobuf definitions |
|
||||
|
||||
### AI Sidecar
|
||||
|
||||
Python FastAPI process that adapts Ollama's HTTP API into Arrow-compatible formats:
|
||||
- `POST /embed` → `nomic-embed-text` via Ollama
|
||||
- `POST /generate` → configurable model (qwen2.5, mistral, gemma2, llama3.2)
|
||||
- `POST /rerank` → cross-encoder reranking via generate endpoint
|
||||
|
||||
No mocks. No stubs. Real models from day one. Ollama manages model lifecycle, GPU scheduling, caching. Sidecar is stateless passthrough.
|
||||
| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
|
||||
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions |
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Client → gateway → catalogd (metadata lookup)
|
||||
→ storaged (object read/write)
|
||||
→ queryd (SQL execution over Parquet)
|
||||
→ aibridge → sidecar → Ollama (inference)
|
||||
Raw data → ingestd (normalize, chunk, detect schema)
|
||||
├→ storaged (Parquet files to object storage)
|
||||
├→ catalogd (register dataset + schema)
|
||||
├→ vectord (embed text chunks, build index)
|
||||
└→ queryd (auto-register as queryable table)
|
||||
|
||||
User question → gateway
|
||||
├→ vectord (semantic search for relevant chunks) ← RAG path
|
||||
├→ queryd (SQL over structured data) ← Analytics path
|
||||
└→ aibridge → Ollama (generate answer from context)
|
||||
```
|
||||
|
||||
### Query Paths
|
||||
|
||||
**Analytical (SQL):** "What's the average bill rate for .NET devs in Chicago?"
|
||||
→ DataFusion scans Parquet columnar, returns in <200ms
|
||||
|
||||
**Semantic (RAG):** "Find candidates who could do data engineering work"
|
||||
→ Embed question → vector search across resume embeddings → retrieve top chunks → LLM answers
|
||||
|
||||
**Hybrid:** "Which clients are we losing money on, and why?"
|
||||
→ SQL for margin calculations + RAG over client notes/emails for context → LLM synthesizes
|
||||
|
||||
### Invariants
|
||||
|
||||
1. Object storage = source of truth for all data
|
||||
2. catalogd = sole metadata authority (datasets, schemas, manifests)
|
||||
3. No raw data stored in catalog — only pointers (bucket, key, schema fingerprint)
|
||||
4. storaged never interprets data — dumb pipe with presigned URLs
|
||||
5. queryd registers tables via catalog pointers, not by scanning storage
|
||||
6. aibridge is stateless — Python sidecar is replaceable without touching Rust
|
||||
7. All services are modular and independently replaceable
|
||||
|
||||
### Dependency Graph
|
||||
|
||||
```
|
||||
shared ← storaged ← catalogd ← queryd
|
||||
shared ← aibridge
|
||||
gateway → {storaged, catalogd, queryd, aibridge}
|
||||
ui → gateway (HTTP only, no crate dependency)
|
||||
```
|
||||
2. catalogd = sole metadata authority
|
||||
3. No raw data in catalog — only pointers
|
||||
4. vectord stores embeddings AS Parquet (portable, not a proprietary format)
|
||||
5. ingestd is idempotent — re-ingesting the same file is a no-op
|
||||
6. Hot cache is a performance layer, not a source of truth — eviction is safe
|
||||
7. All services modular and independently replaceable
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0: Bootstrap
|
||||
Workspace compiles, gateway serves health check, structured logging works.
|
||||
### Phase 0-5: Foundation ✅ COMPLETE
|
||||
|
||||
**Gate:** `cargo build` clean, `GET /health` returns 200, logs on stdout, docs committed.
|
||||
- Rust workspace, Axum gateway, object storage, catalog, DataFusion query engine
|
||||
- Python sidecar with real Ollama models (embed, generate, rerank)
|
||||
- Dioxus UI with Ask (NL→SQL), Explore, SQL, System tabs
|
||||
- gRPC, OpenTelemetry, auth middleware, TOML config
|
||||
- Validated with 286K row staffing company dataset across 7 tables
|
||||
- Cross-reference queries (JOINs across candidates, placements, timesheets, calls) in <150ms
|
||||
|
||||
### Phase 1: Storage + Catalog
|
||||
Write Parquet to object storage, register in catalog, read back.
|
||||
### Phase 6: Ingest Pipeline
|
||||
|
||||
**Gate:** Upload Parquet → register dataset → retrieve metadata → read back. All via gateway HTTP.
|
||||
Build the data on-ramp. Accept messy real-world data, normalize it, make it queryable.
|
||||
|
||||
### Phase 2: Query Engine
|
||||
SQL queries over registered Parquet datasets via DataFusion.
|
||||
| Step | Deliverable | Gate |
|
||||
|---|---|---|
|
||||
| 6.1 | `ingestd` crate with CSV parser → Arrow RecordBatch → Parquet | CSV file → queryable dataset |
|
||||
| 6.2 | JSON ingest (newline-delimited JSON, nested objects) | JSON file → flat Parquet |
|
||||
| 6.3 | Schema detection — infer column types from data | No manual schema definition needed |
|
||||
| 6.4 | Deduplication — detect and skip already-ingested files (content hash) | Re-ingest same file = no-op |
|
||||
| 6.5 | Text chunking — split large text fields for embedding | Long text → overlapping chunks |
|
||||
| 6.6 | Auto-registration — ingest writes to storage AND registers in catalog | Single API call: file in → queryable |
|
||||
| 6.7 | Gateway endpoint: `POST /ingest` with file upload | Upload CSV from browser → query in seconds |
|
||||
|
||||
**Gate:** `SELECT * FROM dataset LIMIT 10` returns correct results. Resolution goes through catalog.
|
||||
**Gate:** Upload a raw CSV or JSON file → auto-detected schema → stored as Parquet → registered → immediately queryable via SQL. No manual steps.
|
||||
|
||||
### Phase 3: AI Integration
|
||||
Python sidecar with real Ollama models. Embeddings, generation, reranking.
|
||||
**Risk:** Schema detection on messy data (mixed types, nulls, inconsistent formatting). Mitigation: conservative type inference (default to string), let user override.
|
||||
|
||||
**Gate:** Rust sends text → Python → Ollama → real embeddings return as Arrow-compatible floats.
|
||||
### Phase 7: Vector Index + RAG Pipeline
|
||||
|
||||
### Phase 4: Frontend
|
||||
Dioxus UI: dataset browser, query editor, results table.
|
||||
Make unstructured data searchable by meaning, not just keywords.
|
||||
|
||||
**Gate:** User can browse datasets and run queries from browser.
|
||||
| Step | Deliverable | Gate |
|
||||
|---|---|---|
|
||||
| 7.1 | `vectord` crate with embedding storage as Parquet (doc_id, chunk_text, vector) | Embeddings stored as portable Parquet |
|
||||
| 7.2 | Chunking strategy — configurable chunk size + overlap for text columns | Large text fields split into embeddable chunks |
|
||||
| 7.3 | Brute-force vector search via DataFusion (cosine similarity SQL) | Semantic search works, correctness verified |
|
||||
| 7.4 | HNSW index for fast approximate nearest neighbor | Search over 100K+ vectors in <50ms |
|
||||
| 7.5 | RAG endpoint: `POST /rag` — question → embed → search → retrieve → generate | Natural language question → grounded answer |
|
||||
| 7.6 | Auto-embed on ingest — text columns automatically embedded during ingest | No separate embedding step needed |
|
||||
| 7.7 | Hybrid search — combine SQL filters with vector similarity | "Java devs in Chicago" (SQL) + "who could do data engineering" (semantic) |
|
||||
|
||||
### Phase 5: Hardening
|
||||
gRPC internals, OpenTelemetry, auth, config-driven startup.
|
||||
**Gate:** Ingest 15K candidate resumes → auto-embed → ask "find someone who could handle our Kubernetes migration" → system returns relevant candidates ranked by semantic match, with LLM explanation.
|
||||
|
||||
**Gate:** Services communicate via gRPC. Traces propagate. Auth enforced. System restartable from repo + config.
|
||||
**Risk: HNSW in Rust at scale.** This is the hardest technical problem. Options:
|
||||
- `hora` crate — Rust-native ANN, but less mature than FAISS
|
||||
- Store HNSW index as a serialized file alongside Parquet data
|
||||
- Fallback: brute-force scan is fine up to ~100K vectors; optimize later
|
||||
- Nuclear option: use Qdrant as an external vector store (breaks "no new services" rule)
|
||||
|
||||
**Decision needed:** Evaluate `hora` vs external Qdrant vs brute-force at J's data scale.
|
||||
|
||||
### Phase 8: Hot Cache + Incremental Updates
|
||||
|
||||
Make frequently-accessed data fast, and handle real-time updates without full rewrite.
|
||||
|
||||
| Step | Deliverable | Gate |
|
||||
|---|---|---|
|
||||
| 8.1 | MemTable hot cache — pin active datasets in memory | Queries on hot data: <10ms |
|
||||
| 8.2 | Cache policy — LRU eviction based on access patterns | Memory-bounded, auto-manages |
|
||||
| 8.3 | Incremental writes — append new rows without rewriting entire Parquet file | Update one candidate's phone → no full table rewrite |
|
||||
| 8.4 | Merge-on-read — query combines base Parquet + delta files | Correct results from base + updates |
|
||||
| 8.5 | Compaction — periodic merge of delta files into base Parquet | Prevent delta file proliferation |
|
||||
| 8.6 | Upsert semantics — insert or update by primary key | Same candidate ID → update in place |
|
||||
|
||||
**Gate:** Update a single row in a 15K-row dataset. Query reflects the change immediately. No full Parquet rewrite. Memory cache serves hot data in <10ms.
|
||||
|
||||
**Risk: This is the Delta Lake problem.** Full ACID transactions over Parquet files is what Databricks spent years building. We're NOT building Delta Lake — we're building a pragmatic version:
|
||||
- Append-only delta files (easy)
|
||||
- Merge-on-read (moderate)
|
||||
- Compaction (moderate)
|
||||
- Full ACID isolation (NOT attempting — single-writer model instead)
|
||||
|
||||
### Phase 9+: Future (not designed yet)
|
||||
|
||||
- Database connector ingest (PostgreSQL, MySQL, MSSQL → Parquet)
|
||||
- PDF/document ingest (OCR → text → chunks → embed)
|
||||
- Scheduled ingest (cron-based file watching)
|
||||
- Multi-node query distribution
|
||||
- Row-level access control
|
||||
- Audit log (who queried what, when)
|
||||
|
||||
---
|
||||
|
||||
## Reference Dataset: Staffing Company
|
||||
|
||||
Validated with realistic staffing company data:
|
||||
|
||||
| Table | Rows | Description |
|
||||
|---|---|---|
|
||||
| candidates | 15,000 | Names, phones, emails, zip, skills, resume text, availability |
|
||||
| clients | 500 | Companies, contacts, verticals, bill rates |
|
||||
| job_orders | 3,000 | Positions with descriptions, requirements, rates |
|
||||
| placements | 8,000 | Candidate↔job matches with dates, rates, recruiters |
|
||||
| timesheets | 120,000 | Weekly hours, bill/pay totals, approvals |
|
||||
| call_log | 80,000 | Phone CDR — who called whom, duration, disposition |
|
||||
| email_log | 60,000 | Email tracking — subject, opened, direction |
|
||||
| **Total** | **286,500** | **7 tables, cross-referenced** |
|
||||
|
||||
Proven queries:
|
||||
- Candidate search by skills + location + availability: 80ms
|
||||
- Revenue by client with profit margins (JOIN 120K timesheets): 142ms
|
||||
- Cold lead detection (candidates called 5+ times, never placed): 94ms
|
||||
- Margin analysis by vertical (JOIN placements → job orders): 53ms
|
||||
- Natural language → AI-generated SQL → execution → results: ~3s (model inference)
|
||||
|
||||
---
|
||||
|
||||
@@ -132,24 +212,22 @@ gRPC internals, OpenTelemetry, auth, config-driven startup.
|
||||
|
||||
| Model | Use |
|
||||
|---|---|
|
||||
| `nomic-embed-text` | Embeddings (768d) |
|
||||
| `qwen2.5` | Code generation, structured output |
|
||||
| `mistral` | General generation |
|
||||
| `nomic-embed-text` | Embeddings (768d) — semantic search, RAG retrieval |
|
||||
| `qwen2.5` | SQL generation, structured output, summarization |
|
||||
| `mistral` | General generation, longer context |
|
||||
| `gemma2` | General generation |
|
||||
| `llama3.2` | General generation |
|
||||
|
||||
Model selection via environment variables. No hardcoded model names in Rust code.
|
||||
| `llama3.2` | General generation, lightweight |
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Multi-tenancy
|
||||
- Streaming ingestion / CDC
|
||||
- Custom file formats
|
||||
- Query caching / materialized views
|
||||
- Wrapping `object_store` with another abstraction
|
||||
- Cloud deployment (local-first)
|
||||
- Multi-tenancy (single-owner system)
|
||||
- Cloud deployment (local-first, always)
|
||||
- Full ACID transactions (single-writer model is sufficient)
|
||||
- Real-time streaming / CDC (batch ingest is the model)
|
||||
- Replacing the CRM (this is the analytical layer BEHIND the CRM)
|
||||
- Custom file formats (Parquet is the format, period)
|
||||
|
||||
---
|
||||
|
||||
@@ -157,11 +235,13 @@ Model selection via environment variables. No hardcoded model names in Rust code
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| RustFS immaturity | High | Start with LocalFileSystem, test against MinIO, RustFS last. SeaweedFS fallback. |
|
||||
| DataFusion table registration overhead | Medium | Lazy registration + LRU cache of SessionContext instances. |
|
||||
| Catalog consistency without DB | Medium | Write-ahead: persist manifest before in-memory update. Rebuild from storage on restart. |
|
||||
| Dioxus WASM gaps | Medium | Phase 4 is last. Fallback to plain HTML if blocked. |
|
||||
| Schema evolution | Medium | Schema fingerprinting in Phase 1. Validate before query. |
|
||||
| Vector search in Rust at scale | **High** | Start brute-force, evaluate `hora` crate, Qdrant as fallback |
|
||||
| Incremental updates on Parquet | **High** | Delta files + merge-on-read, NOT full Delta Lake |
|
||||
| Legacy data messiness | **High** | Conservative schema detection, default to string, user overrides |
|
||||
| Schema evolution across ingests | **Medium** | Schema fingerprinting + versioned manifests |
|
||||
| Memory pressure from hot cache | **Medium** | LRU eviction, configurable memory limit |
|
||||
| HNSW index persistence | **Medium** | Serialize alongside Parquet, rebuild on startup |
|
||||
| Python sidecar as bottleneck | **Low** | Can replace with direct Ollama HTTP from Rust later |
|
||||
|
||||
---
|
||||
|
||||
@@ -173,3 +253,5 @@ Model selection via environment variables. No hardcoded model names in Rust code
|
||||
4. No silent architecture drift
|
||||
5. Always work in smallest valid step
|
||||
6. Always verify before moving on
|
||||
7. **New:** Flag when something is genuinely hard vs just engineering work
|
||||
8. **New:** If a phase reveals the approach is wrong, update the PRD before continuing
|
||||
|
||||
Reference in New Issue
Block a user