profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Browse Source 
Four shipped features and a PRD realignment, all measured end-to-end:

HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
  TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
  /hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
  at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
  80/30 = 1.00 recall in 230s

Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
  and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows

Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
  /etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
  rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
  anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
  /storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
  X-Lakehouse-Rescue-Used observability headers on fallback

Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
  into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
  (586 rows × 13 cols, 6 batches, 196ms end-to-end)

PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
  trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
  bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
  decision rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 01:50:05 -05:00
co-authored by Claude Opus 4.6
parent 84407eeb51
commit dbe00d018f
33 changed files with 3305 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PRD.md
+149 -24
View File
@@ -1,22 +1,41 @@
# PRD: Lakehouse — Rust-First Object Storage System
# PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores
**Status:** Active — Phase 0-5 complete, entering production path
**Status:** Active — Phase 0-14 complete; federation foundation + HNSW trial system shipped 2026-04-16; entering Phase 16 (hot-swap + model profiles)
**Created:** 2026-03-27
**Last reframed:** 2026-04-16 — from "staffing analytics platform" to "dual-use knowledge substrate" (see §Problem below)
**Owner:** J
---
## Problem
### Use case 1 — Staffing analytics (reference implementation)
Legacy data systems silo information across CRMs, databases, spreadsheets, and file shares. Querying across them requires manual ETL, pre-defined schemas, and expensive database licenses. When AI enters the picture, these systems can't handle the dual requirement of fast analytical queries AND semantic retrieval over unstructured text.
A staffing company (our reference case) has candidate records in an ATS, client data in a CRM, timesheets in billing software, call logs from a phone system, and email records from Exchange. Answering "find every Java developer in Chicago who was called 5+ times but never placed" requires querying across all of them — and no single system can do it.
We need a system where:
- Any data source (CSV, DB export, PDF, JSON) can be ingested without pre-defined schemas
### Use case 2 — Local AI knowledge substrate (the second half)
Local LLM workloads need a substrate for ingesting, indexing, and retrieving large knowledge corpora. Each running model (or agent) has its own context — documents it cares about, a vector index tuned to its domain, a scoped view of the catalog. That infrastructure is architecturally identical to the staffing problem: ingest messy data, index it, query it, hand it to an AI. Building one substrate that serves both prevents fragmentation.
Concretely this means a running Ollama model like `qwen2.5:7b` or `claude-code-local` should be able to:
- Bind to a named set of datasets
- Get a scoped vector index pre-warmed for its domain
- Issue searches that only see its bound data
- Have its trial/tuning history isolated from other models
- Swap between knowledge generations (today's, yesterday's) without rebuild
The same infrastructure that lets a recruiter query 2.47M rows of staffing data also lets a local 7B model answer questions grounded in a 500K-chunk documentation corpus. Same substrate, different tenant.
### Shared requirements
- Any data source (CSV, DB export, PDF, JSON, Postgres table) can be ingested without pre-defined schemas
- Structured data is queryable via SQL at scale (millions of rows, sub-second)
- Unstructured data is searchable via AI embeddings (semantic retrieval)
- An LLM can answer natural language questions against all of it
- Unstructured data is searchable via AI embeddings with per-profile indexes
- An LLM can answer natural language questions against scoped data
- Indexes can be hot-swapped between generations without rebuild downtime
- Trials are first-class data — the system remembers how it was tuned
- Everything runs locally — no cloud APIs, total data privacy
- The system is rebuildable from repository + object storage alone
@@ -51,15 +70,22 @@ No new frameworks without documented ADR.
| Service | Responsibility |
|---|---|
| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits |
| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests |
| **storaged** | Object I/O — read/write/list/delete via `object_store` |
| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache |
| **ingestd** | *NEW* — Ingest pipeline: CSV/JSON/DB → normalize → Parquet → catalog |
| **vectord** | *NEW* — Embedding store + vector index: chunk → embed → index → search |
| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits, `X-Lakehouse-Bucket` header routing |
| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests, per-dataset resync from parquet footers |
| **storaged** | Object I/O — `BucketRegistry` (multi-backend), rescue fallback, error journal, append-log batching pattern |
| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache, delta merge-on-read |
| **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog |
| **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) |
| **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal |
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions |
| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
**Federation building blocks** (shipped 2026-04-16):
- `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced)
- `storaged::registry::BucketRegistry` — multi-bucket resolution with `rescue_bucket` read fallback
- `storaged::append_log::AppendLog` — write-once batched append pattern (no RMW, no small-file problem)
- `storaged::error_journal::ErrorJournal` — bucket operation failure log at `primary://_errors/bucket_errors/batch_*.jsonl`
### Data Flow
@@ -92,10 +118,14 @@ User question → gateway
1. Object storage = source of truth for all data
2. catalogd = sole metadata authority
3. No raw data in catalog — only pointers
4. vectord stores embeddings AS Parquet (portable, not a proprietary format)
4. vectord stores embeddings AS Parquet (portable, not a proprietary format) — see ADR-018 for the Parquet-vs-Lance trade review
5. ingestd is idempotent — re-ingesting the same file is a no-op
6. Hot cache is a performance layer, not a source of truth — eviction is safe
7. All services modular and independently replaceable
8. **Indexes are hot-swappable.** A new index generation can be built in the background while the current one serves queries. Promotion is atomic (pointer swap). Rollback to a prior generation is always possible. (Phase 16)
9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17)
10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern.
11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?"
---
@@ -262,19 +292,90 @@ Per-contract overlays with daily/weekly/monthly tiers and instant handoff.
| 14.3 | Migration preview: show how old data maps to new schema before applying | Human approves before data transforms |
| 14.4 | Versioned schemas in catalog: v1, v2, v3 coexist | Queries specify version or use latest |
### Phase 15+: Horizon
### Phase 15: Infrastructure horizon items
- [x] HNSW vector index with trial system (shipped 2026-04-16)
- [x] Federation foundation — ADR-017 (shipped 2026-04-16)
- [x] Database connector ingest — Postgres batch with streaming (shipped 2026-04-16)
- [ ] Federation layer 2 — X-Lakehouse-Bucket middleware, catalog migration, cross-bucket SQL in queryd
- [ ] PDF OCR for scanned documents (Tesseract integration)
- [ ] Scheduled ingest (cron-based file watching, S3 event triggers)
- [ ] Multi-node query distribution (DataFusion supports this architecturally)
### Phase 16: Hot-Swap Index Generations
Make indexes upgrade-in-place without dropping queries.
| Step | Deliverable | Gate |
|---|---|---|
| 16.1 | "Active generation" pointer per logical index name | `/vectors/search` routes to current champion automatically |
| 16.2 | Background trial runner: watches trial journal, proposes configs (random search / Bayesian), fires `/hnsw/trial` | Agent autonomously tunes without human POSTing each config |
| 16.3 | Promotion endpoint: `POST /hnsw/promote/{index}/{trial_id}` atomically swaps active pointer | Next search hits new config, zero downtime |
| 16.4 | Rollback: `POST /hnsw/rollback/{index}` reverts to previous generation | Bad promotion recoverable in milliseconds |
| 16.5 | Dataset-append triggers: when `POST /ingest/file` writes to a dataset with attached vector indexes, schedule automatic re-trial (not full rebuild) | New docs get embedded + indexed without manual intervention |
**Gate:** Run the trial agent for 10 minutes against `resumes_100k_v2` with a fresh eval set. It explores the `ef_construction × ef_search` space, promotes the Pareto winner, continues running. Zero human clicks. All trials and promotions appear in `/hnsw/trials/resumes_100k_v2`.
**Risk:** Agent loops into a bad region (e.g. always proposes ef_construction=1). Mitigation: a hardcoded config space constraint + minimum-quality gate (don't promote anything with recall <0.9).
### Phase 17: Model Profiles + Dataset Bindings
Make "different models see different data" real instead of a config string.
| Step | Deliverable | Gate |
|---|---|---|
| 17.1 | `ModelProfile` manifest: id, ollama_name, bucket, bound_datasets[], hnsw_config, embed_model | `GET /models` lists profiles; `POST /models` creates one |
| 17.2 | Profile activation endpoint: `POST /profile/{id}/activate` — warms EmbeddingCache for bound indexes, builds HNSW with profile's config | Next search against bound indexes is <1ms cold |
| 17.3 | Model-scoped search: `POST /search?model=X` filters to bound datasets only | Model A can't see Model B's datasets unless explicitly shared |
| 17.4 | VRAM-aware activation: only one (or small N) model loaded at a time on 16GB A4000 | Activating model B unloads model A via Ollama's keep_alive=0 |
| 17.5 | Audit: every tool invocation by a model is logged with model identity | `GET /models/{id}/audit` shows exactly what each model touched |
**Gate:** Two model profiles defined: `staffing-recruiter` (bound to candidates/placements/timesheets) and `docs-assistant` (bound to a documentation corpus). Activate staffing-recruiter, search for candidates — works. Switch to docs-assistant, same search — returns zero from staffing (not bound) but finds docs. VRAM shows only one embedding model loaded at a time.
**VRAM reality:** 16GB A4000 realistically holds 1-2 loaded models concurrently. "Multi-model" in practice means sequential swap between profiles, not parallel serving. The profile abstraction makes this swap clean.
### Phase 18: Storage format decision (Lance evaluation)
The question raised 2026-04-16 after J's LLMS3 knowledge base identified Lance as `alternative_to` Parquet for vector workloads. Current stack: Parquet with binary-blob vector columns + in-RAM HNSW sidecar. Evaluated against: Lance native vector format with disk-resident indexes.
| Step | Deliverable | Decision criteria |
|---|---|---|
| 18.1 | Parallel Lance-backed vector index for `resumes_100k_v2` behind feature flag | Both implementations coexist, benchmarkable |
| 18.2 | Head-to-head benchmark: cold-load, search latency, disk size, append cost | See criteria below |
| 18.3 | ADR-019 documenting the decision with measured data | Commit or reject with evidence |
**Decision rules:**
- Lance wins on cold-load by ≥2× AND matches search latency → migrate vector layer to Lance. Dataset Parquet stays.
- Lance is within 50% of current → stay on current stack, document ceiling explicitly.
- Lance loses → close the door, move on.
### Phase 19+: Further horizon
- Federated multi-bucket query (client A's S3 + client B's S3 + yours)
- Database connector ingest (PostgreSQL, MySQL, MSSQL → Parquet via CDC)
- PDF OCR for scanned documents (Tesseract integration)
- Scheduled ingest (cron-based file watching, S3 event triggers)
- Specialized fine-tuned models per domain (staffing matcher, resume parser)
- Multi-node query distribution (DataFusion supports this architecturally)
- Video/audio transcript ingest + multimodal embeddings
- True distributed query (DataFusion multi-node) — only if single-machine ceilings bite
---
## Reference Dataset: Staffing Company
## Known ceilings (honest)
The current stack has measurable limits. Documenting them so future decisions aren't based on wishful thinking.
| Dimension | Current ceiling | Breaks at | Escape hatch |
|---|---|---|---|
| Vector count per index | ~5M vectors on 128GB RAM | 10M+ (serious web crawl) | Phase 18 Lance migration OR mmap'd embeddings |
| Concurrent active indexes | ~50-100 at 100K vectors each | 10M×50 configurations | Lance disk-resident + per-profile activation |
| Rows per dataset | 2.47M proven, probably 100M+ fine | Approaches DataFusion memory limits | DataFusion predicate pushdown + partition pruning (existing) |
| Concurrent loaded models | 1-2 on 16GB VRAM (A4000) | 3+ models simultaneous | Not our problem — architectural, driven by Ollama |
| Trial journal growth per index | Thousands of trials, batched JSONL | High-frequency auto-tuning agent | Compaction via `/hnsw/trials/{idx}/compact` |
| Error journal growth | Bounded by ring buffer (2000 events in-memory) + batched JSONL on disk | Continuous failure scenarios | Compaction + retention policy (TODO) |
---
## Reference Workloads
### Workload 1: Staffing Company
Scale-tested on 128GB RAM server:
@@ -315,6 +416,27 @@ Scale-tested on 128GB RAM server:
- Instant handoff between agents — zero data copy
- Full activity timeline preserved across handoffs
### Workload 2: Local LLM Knowledge Base
The second use case this substrate is built for. Reference corpus: the running `knowledge_base` Postgres database (586 team runs, response cache history, pipeline runs, threat intel) + LLMS3.com published corpus (~243 enriched documents).
Target scale on same 128GB server:
- Documents: 10K-100K per model profile
- Chunks after chunking: 500K-5M per profile
- Embedding dimensions: 768 (nomic-embed-text)
- Query latency: <100ms semantic search, <3s end-to-end RAG including LLM generation
- Concurrent model profiles: 2-5 configured, 1-2 active at a time (VRAM-bound)
Measured to date (Phase 7 + Phase 16 prep):
- 100K candidate-resume chunks embedded in 10 min via Ollama nomic-embed-text
- HNSW search at 100% recall, ~1ms p50 on 100K vectors (ec=80 es=30 locked as default)
- Trial journal instrumented and working for parameter tuning
Gaps still to close for this workload:
- Model profiles (Phase 17) — today, "model" is a string, not a first-class entity
- Hot-swap generations (Phase 16) — today, rebuild = downtime
- Scale past 5M vectors — needs Phase 18 Lance evaluation to decide path
---
## Available Local Models
@@ -331,12 +453,15 @@ Scale-tested on 128GB RAM server:
## Non-Goals
- Multi-tenancy (single-owner system)
- Cloud deployment (local-first, always)
- Full ACID transactions (single-writer model is sufficient)
- Real-time streaming / CDC (batch ingest is the model)
- Replacing the CRM (this is the analytical layer BEHIND the CRM)
- Custom file formats (Parquet is the format, period)
- Real-time streaming / CDC (batch ingest is the model; scheduled refresh, not transactional replication)
- Replacing the CRM (this is the analytical + AI layer BEHIND the CRM)
- Custom file formats — Parquet for datasets + sidecar indexes for vectors (see ADR-018 for why we stayed Parquet instead of migrating to Lance, and the ceilings that choice implies)
- Hard multi-tenant isolation (profiles and federation provide soft isolation; this is not a SaaS platform with adversarial tenants — operator is single-trust)
Removed from prior non-goals (2026-04-16):
- ~~Multi-tenancy (single-owner system)~~ — federation + profile buckets are now first-class; soft multi-tenancy is a design goal. Hard adversarial multi-tenancy (adversarial tenants on shared infrastructure) remains out of scope.
---