Phase 0: bootstrap Rust workspace
- Cargo workspace with 6 crates: shared, storaged, catalogd, queryd, aibridge, gateway - shared: types (DatasetId, ObjectRef, SchemaFingerprint, DatasetManifest) + error enum - gateway: Axum HTTP entrypoint with nested service routers + tracing - All services expose /health stubs - justfile with build/test/run recipes - PRD, phase tracker, and ADR docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,26 @@
|
||||
# Architecture Decision Records
|
||||
|
||||
## ADR-001: Object storage as source of truth
|
||||
**Date:** 2026-03-27
|
||||
**Decision:** All data lives in S3-compatible object storage. No traditional database.
|
||||
**Rationale:** Eliminates DB operational overhead, enables infinite scale at storage tier, forces clean separation of data and metadata.
|
||||
|
||||
## ADR-002: Catalog metadata persistence
|
||||
**Date:** 2026-03-27
|
||||
**Decision:** catalogd persists manifests as Parquet files in object storage. In-memory index rebuilt on startup.
|
||||
**Rationale:** No external DB dependency. Storage is already the source of truth. Write-ahead pattern ensures consistency.
|
||||
|
||||
## ADR-003: Real models only (no mocks)
|
||||
**Date:** 2026-03-27
|
||||
**Decision:** AI sidecar hits Ollama with real models from Phase 3 onward. No stub/mock endpoints.
|
||||
**Rationale:** Local Ollama instance available with nomic-embed-text, qwen2.5, mistral, gemma2, llama3.2. Mocks hide integration bugs.
|
||||
|
||||
## ADR-004: Python sidecar as Ollama adapter
|
||||
**Date:** 2026-03-27
|
||||
**Decision:** Python FastAPI sidecar is a thin HTTP adapter over Ollama's API. No model loading in Python.
|
||||
**Rationale:** Ollama handles model lifecycle, GPU scheduling, caching. Sidecar stays stateless and lightweight — no torch/transformers deps.
|
||||
|
||||
## ADR-005: HTTP-first, gRPC later
|
||||
**Date:** 2026-03-27
|
||||
**Decision:** All inter-service communication uses HTTP through Phase 4. gRPC migration in Phase 5.
|
||||
**Rationale:** HTTP is simpler to debug, test, and iterate on. gRPC adds protobuf compilation and streaming complexity before APIs stabilize.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Phase Tracker
|
||||
|
||||
## Phase 0: Bootstrap
|
||||
- [ ] 0.1 — Cargo workspace with all crate stubs compiling
|
||||
- [ ] 0.2 — `shared` crate: error types, ObjectRef, DatasetId
|
||||
- [ ] 0.3 — `gateway` with Axum: GET /health → 200
|
||||
- [ ] 0.4 — tracing + tracing-subscriber wired in gateway
|
||||
- [ ] 0.5 — justfile with build, test, run recipes
|
||||
- [ ] 0.6 — docs committed to git
|
||||
|
||||
**Gate:** All crates compile. Gateway runs. Logs emit. Docs committed.
|
||||
|
||||
## Phase 1: Storage + Catalog
|
||||
- [ ] 1.1 — storaged: object_store backend init (LocalFileSystem → S3)
|
||||
- [ ] 1.2 — storaged: Axum endpoints (PUT/GET/DELETE /objects/{key})
|
||||
- [ ] 1.3 — shared/arrow.rs: RecordBatch ↔ Parquet helpers
|
||||
- [ ] 1.4 — catalogd/registry.rs: in-memory index + manifest persistence
|
||||
- [ ] 1.5 — catalogd/schema.rs: schema fingerprinting
|
||||
- [ ] 1.6 — catalogd service: POST/GET /datasets endpoints
|
||||
- [ ] 1.7 — gateway routes to storaged + catalogd
|
||||
|
||||
**Gate:** Upload Parquet → register → metadata → read back. All via gateway.
|
||||
|
||||
## Phase 2: Query Engine
|
||||
- [ ] 2.1 — queryd: SessionContext + object_store config
|
||||
- [ ] 2.2 — queryd: ListingTable from catalog ObjectRefs
|
||||
- [ ] 2.3 — queryd service: POST /query → Arrow IPC or JSON
|
||||
- [ ] 2.4 — queryd → catalogd wiring
|
||||
- [ ] 2.5 — gateway routes /query
|
||||
|
||||
**Gate:** SQL over Parquet returns correct results via catalog resolution.
|
||||
|
||||
## Phase 3: AI Integration
|
||||
- [ ] 3.1 — Python sidecar: FastAPI + Ollama (embed/generate/rerank)
|
||||
- [ ] 3.2 — Dockerfile for sidecar
|
||||
- [ ] 3.3 — aibridge/client.rs: HTTP client to sidecar
|
||||
- [ ] 3.4 — aibridge service: Axum proxy endpoints
|
||||
- [ ] 3.5 — Model config via env vars
|
||||
|
||||
**Gate:** Rust → Python → Ollama → real embeddings return.
|
||||
|
||||
## Phase 4: Frontend
|
||||
- [ ] 4.1 — Dioxus scaffold, WASM build
|
||||
- [ ] 4.2 — Dataset browser
|
||||
- [ ] 4.3 — Query editor + results table
|
||||
- [ ] 4.4 — Error display + loading states
|
||||
|
||||
**Gate:** Browse datasets and query from browser.
|
||||
|
||||
## Phase 5: Hardening
|
||||
- [ ] 5.1 — Proto definitions
|
||||
- [ ] 5.2 — Internal gRPC migration
|
||||
- [ ] 5.3 — OpenTelemetry tracing
|
||||
- [ ] 5.4 — Auth middleware
|
||||
- [ ] 5.5 — Config-driven startup
|
||||
|
||||
**Gate:** gRPC internals, traces, auth, restartable from repo + config.
|
||||
+175
@@ -0,0 +1,175 @@
|
||||
# PRD: Lakehouse — Rust-First Object Storage System
|
||||
|
||||
**Status:** Active
|
||||
**Created:** 2026-03-27
|
||||
**Owner:** J
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Traditional data platforms couple storage, compute, and metadata into monolithic databases. This creates vendor lock-in, scaling bottlenecks, and opaque data access. AI workloads bolt onto these systems awkwardly, sharing resources with transactional queries.
|
||||
|
||||
We need a system where:
|
||||
- Object storage is the source of truth (not a database)
|
||||
- Metadata, access, and execution are controlled by Rust services
|
||||
- Queries run directly over object storage via Arrow/Parquet
|
||||
- AI inference is isolated and swappable
|
||||
- The entire system is rebuildable from repository + docs alone
|
||||
|
||||
---
|
||||
|
||||
## Solution
|
||||
|
||||
A modular Rust service mesh over S3-compatible object storage.
|
||||
|
||||
### Locked Stack
|
||||
|
||||
| Layer | Technology | Locked |
|
||||
|---|---|---|
|
||||
| Frontend | Dioxus | Yes |
|
||||
| API | Axum + Tokio | Yes |
|
||||
| Object Storage Interface | Apache Arrow `object_store` | Yes |
|
||||
| Storage Backend | RustFS (fallback: SeaweedFS) | Yes |
|
||||
| Query Engine | DataFusion | Yes |
|
||||
| Data Format | Parquet + Arrow | Yes |
|
||||
| RPC (internal) | tonic (gRPC) | Yes |
|
||||
| AI Runtime | Ollama (local models) | Yes |
|
||||
| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes |
|
||||
|
||||
No new frameworks. No exceptions.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| **gateway** | HTTP ingress, routing, auth envelope, middleware |
|
||||
| **catalogd** | Metadata control plane — dataset registry, schema versions, manifest index |
|
||||
| **storaged** | Object I/O — read/write/list/delete via `object_store` crate |
|
||||
| **queryd** | SQL execution — DataFusion over registered Parquet datasets |
|
||||
| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
|
||||
| **ui** | Dioxus frontend — dataset browser, query editor, results viewer |
|
||||
| **shared** | Types, errors, Arrow helpers, protobuf definitions |
|
||||
|
||||
### AI Sidecar
|
||||
|
||||
Python FastAPI process that adapts Ollama's HTTP API into Arrow-compatible formats:
|
||||
- `POST /embed` → `nomic-embed-text` via Ollama
|
||||
- `POST /generate` → configurable model (qwen2.5, mistral, gemma2, llama3.2)
|
||||
- `POST /rerank` → cross-encoder reranking via generate endpoint
|
||||
|
||||
No mocks. No stubs. Real models from day one. Ollama manages model lifecycle, GPU scheduling, caching. Sidecar is stateless passthrough.
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Client → gateway → catalogd (metadata lookup)
|
||||
→ storaged (object read/write)
|
||||
→ queryd (SQL execution over Parquet)
|
||||
→ aibridge → sidecar → Ollama (inference)
|
||||
```
|
||||
|
||||
### Invariants
|
||||
|
||||
1. Object storage = source of truth for all data
|
||||
2. catalogd = sole metadata authority (datasets, schemas, manifests)
|
||||
3. No raw data stored in catalog — only pointers (bucket, key, schema fingerprint)
|
||||
4. storaged never interprets data — dumb pipe with presigned URLs
|
||||
5. queryd registers tables via catalog pointers, not by scanning storage
|
||||
6. aibridge is stateless — Python sidecar is replaceable without touching Rust
|
||||
7. All services are modular and independently replaceable
|
||||
|
||||
### Dependency Graph
|
||||
|
||||
```
|
||||
shared ← storaged ← catalogd ← queryd
|
||||
shared ← aibridge
|
||||
gateway → {storaged, catalogd, queryd, aibridge}
|
||||
ui → gateway (HTTP only, no crate dependency)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0: Bootstrap
|
||||
Workspace compiles, gateway serves health check, structured logging works.
|
||||
|
||||
**Gate:** `cargo build` clean, `GET /health` returns 200, logs on stdout, docs committed.
|
||||
|
||||
### Phase 1: Storage + Catalog
|
||||
Write Parquet to object storage, register in catalog, read back.
|
||||
|
||||
**Gate:** Upload Parquet → register dataset → retrieve metadata → read back. All via gateway HTTP.
|
||||
|
||||
### Phase 2: Query Engine
|
||||
SQL queries over registered Parquet datasets via DataFusion.
|
||||
|
||||
**Gate:** `SELECT * FROM dataset LIMIT 10` returns correct results. Resolution goes through catalog.
|
||||
|
||||
### Phase 3: AI Integration
|
||||
Python sidecar with real Ollama models. Embeddings, generation, reranking.
|
||||
|
||||
**Gate:** Rust sends text → Python → Ollama → real embeddings return as Arrow-compatible floats.
|
||||
|
||||
### Phase 4: Frontend
|
||||
Dioxus UI: dataset browser, query editor, results table.
|
||||
|
||||
**Gate:** User can browse datasets and run queries from browser.
|
||||
|
||||
### Phase 5: Hardening
|
||||
gRPC internals, OpenTelemetry, auth, config-driven startup.
|
||||
|
||||
**Gate:** Services communicate via gRPC. Traces propagate. Auth enforced. System restartable from repo + config.
|
||||
|
||||
---
|
||||
|
||||
## Available Local Models
|
||||
|
||||
| Model | Use |
|
||||
|---|---|
|
||||
| `nomic-embed-text` | Embeddings (768d) |
|
||||
| `qwen2.5` | Code generation, structured output |
|
||||
| `mistral` | General generation |
|
||||
| `gemma2` | General generation |
|
||||
| `llama3.2` | General generation |
|
||||
|
||||
Model selection via environment variables. No hardcoded model names in Rust code.
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Multi-tenancy
|
||||
- Streaming ingestion / CDC
|
||||
- Custom file formats
|
||||
- Query caching / materialized views
|
||||
- Wrapping `object_store` with another abstraction
|
||||
- Cloud deployment (local-first)
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| RustFS immaturity | High | Start with LocalFileSystem, test against MinIO, RustFS last. SeaweedFS fallback. |
|
||||
| DataFusion table registration overhead | Medium | Lazy registration + LRU cache of SessionContext instances. |
|
||||
| Catalog consistency without DB | Medium | Write-ahead: persist manifest before in-memory update. Rebuild from storage on restart. |
|
||||
| Dioxus WASM gaps | Medium | Phase 4 is last. Fallback to plain HTML if blocked. |
|
||||
| Schema evolution | Medium | Schema fingerprinting in Phase 1. Validate before query. |
|
||||
|
||||
---
|
||||
|
||||
## Operating Rules
|
||||
|
||||
1. PRD > architecture > phases > status > git
|
||||
2. Git is memory, not chat
|
||||
3. No undocumented changes
|
||||
4. No silent architecture drift
|
||||
5. Always work in smallest valid step
|
||||
6. Always verify before moving on
|
||||
Reference in New Issue
Block a user