profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Phases 16.2 + L2 + 17 VRAM gate + MySQL + 18 Lance hybrid milestone

Browse Source 
Five threads of work landing as one milestone — all individually
verified end-to-end against real data, full release build clean,
46 unit tests pass.

## Phase 16.2 / 16.5 — autotune agent + ingest triggers

`vectord::agent` is a long-running tokio task that watches the trial
journal and autonomously proposes + runs new HNSW configs. Distinct
from `autotune::run_autotune` (synchronous one-shot grid). Triggered
on POST /vectors/agent/enqueue/{idx} or by the periodic wake; ingest
paths now push DatasetAppended events when an index's source dataset
gets re-ingested. Rate-limited (max_trials_per_hour) and cooldown-
gated so it can't saturate Ollama under live load.

The proposer is ε-greedy around the current champion: with prob 0.25
sample random from full bounds, otherwise perturb champion ± small
delta on both axes. Dedup against history. Deterministic — RNG seeded
from history.len() so the same journal state proposes the same next
config (helps offline replay debugging).

`[agent]` config section in lakehouse.toml; opt-in via enabled=true.

## Federation Layer 2 — runtime bucket lifecycle + per-index scoping

`BucketRegistry.buckets` moved to `std::sync::RwLock<HashMap>` so
buckets can be added/removed after startup. POST /storage/buckets
provisions at runtime; DELETE /storage/buckets/{name} unregisters
(refuses primary/rescue with 403). Local-backend buckets get their
root directory auto-created.

`IndexMeta.bucket` (default "primary" via serde) records each index's
home bucket. `TrialJournal` and `PromotionRegistry` now hold
Arc<BucketRegistry> + IndexRegistry; they resolve target store per-
index via IndexMeta.bucket. PromotionRegistry::list_all scans every
bucket and dedups by index_name. Pre-federation indexes keep working
unchanged — they just default to primary.

`ModelProfile.bucket: Option<String>` declares per-profile artifact
home. POST /vectors/profile/{id}/activate auto-provisions the
profile's bucket under storage.profile_root if not yet registered.

EvalSets stay primary-only for now — noted gap, low-risk to extend
later with the same resolver pattern.

## Phase 17 — VRAM-aware two-profile gate

Sidecar gains POST /admin/unload (Ollama keep_alive=0 trick — forces
immediate VRAM release), POST /admin/preload (keep_alive=5m with
empty prompt, takes the slot warm), and GET /admin/vram (combines
nvidia-smi snapshot with Ollama /api/ps). Exposed via aibridge as
unload_model / preload_model / vram_snapshot.

`VectorState.active_profile` is the GPU-slot singleton —
Arc<RwLock<Option<ActiveProfileSlot>>>. activate_profile checks for
a previous profile with a different ollama_name and unloads it
before preloading the new one; same-model reactivations skip the
unload (Ollama no-ops). New routes: POST /vectors/profile/{id}/
deactivate (unload + clear slot), GET /vectors/profile/active.

Verified live: staffing-recruiter (qwen2.5) → docs-assistant
(mistral) swap freed qwen2.5 from VRAM and loaded mistral. nomic-
embed-text persists across swaps because both profiles use it —
free optimization that fell out of the design. Scoped search
correctly 403s cross-profile in both directions.

## MySQL streaming connector

`crates/ingestd/src/my_stream.rs` mirrors pg_stream.rs for MySQL.
Pure-rust `mysql_async` driver (default-features=false to avoid C
deps). Same OFFSET pagination, same Parquet-streaming write shape.
Type mapping per ADR-010: int/bigint → Int32/Int64, decimal/float
→ Float64, tinyint(1)/bool → Boolean, everything else → Utf8 with
fallback parsers for date/time/json/uuid via Display.

POST /ingest/mysql parallel to /ingest/db. Same PII auto-detection,
same lineage capture (source_system="mysql"), same agent-trigger
hook. `redact_dsn` generalized — was hardcoded to "postgresql://"
length, now works for any scheme://user:pass@host/path URL (latent
PII leak fix for MySQL DSNs).

Verified live against MariaDB on localhost: 10 rows × 9 columns of
test data round-tripped through datatypes int/varchar/decimal/
tinyint/datetime/text. PII detection auto-flagged name + email.
Aggregation queries through DataFusion match the source values
exactly.

## Phase 18 — Hybrid Parquet+HNSW ⊕ Lance backend (ADR-019)

`vectord-lance` is a new firewall crate. Lance pulls Arrow 57 and
DataFusion 52 — incompatible with the rest of the workspace's
Arrow 55 / DataFusion 47. The firewall isolates that dep tree:
public API uses only std types (Vec<f32>, Vec<String>, Hit, Row,
*Stats), so no Arrow types cross the crate boundary and nothing
propagates to vectord. The ADR-019 path that didn't ship until now.

`vectord::lance_backend::LanceRegistry` lazy-creates a
LanceVectorStore per index, resolving bucket → URI via the
conventional local-bucket layout. `IndexMeta.vector_backend` and
`ModelProfile.vector_backend` carry the choice (default Parquet so
existing indexes unchanged).

Six routes under /vectors/lance/*:
- migrate/{idx}: convert binary-blob Parquet → Lance FixedSizeList
- index/{idx}: build IVF_PQ
- search/{idx}: vector search (embed via sidecar)
- doc/{idx}/{doc_id}: random row fetch
- append/{idx}: native fragment append
- stats/{idx}: row count + index presence

Verified live on the real resumes_100k_v2 corpus (100K × 768d):
- Migrate: 0.57s
- Build IVF_PQ index: 16.2s (matches ADR-019 bench; 14× faster than
  HNSW's 230s for the same data)
- Search end-to-end (Ollama embed + Lance scan): 23-53ms
- Random doc_id fetch: 5-7ms (filter scan; faster than Parquet's
  ~35ms full-file scan, slower than the bench's 311us positional
  take — would close that gap with a scalar btree on doc_id)
- Append 100 rows: 3.3ms / +320KB on disk vs Parquet's required
  full ~330MB rewrite — the structural win
- Index survives append; both backends coexist cleanly

## Known follow-ups not in this milestone

- ModelProfile.vector_backend doesn't yet auto-route /vectors/profile/
  {id}/search to Lance; callers go through /vectors/lance/* directly
- Scalar btree on doc_id (closes the 5-7ms → ~300us gap)
- vectord-lance built default-features=false → no S3 yet
- IVF_PQ recall not measured (ADR-019 caveat) — needs a Lance-aware
  variant of the eval harness
- Watcher-path ingest doesn't push agent triggers (HTTP paths do)
- EvalSets still primary-only (federation gap)
- No PATCH endpoint to move an existing index between buckets
- The pre-existing storaged::append_log doctest fails to compile
  (malformed `{prefix}/` parses as code fence) — pre-existing bug,
  left for a focused fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-16 20:24:46 -05:00
co-authored by Claude Opus 4.6
parent 4e1c400f5d
commit 0d037cfac1
28 changed files with 3240 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/storaged/src/federation_service.rs
+57 -4
View File
@@ -13,10 +13,11 @@ use axum::{
extract::{Path, Query, State},
http::StatusCode,
response::IntoResponse,
routing::{get, put},
routing::{delete, get, post, put},
};
use chrono::{DateTime, Utc};
use serde::Deserialize;
use shared::config::BucketConfig;
use std::sync::Arc;
use crate::error_journal::BucketErrorEvent;
@@ -24,16 +25,68 @@ use crate::registry::{BucketInfo, BucketRegistry};
pub fn router(registry: Arc<BucketRegistry>) -> Router {
Router::new()
.route("/buckets", get(list_buckets))
.route("/buckets", get(list_buckets).post(add_bucket))
.route("/buckets/{name}", delete(remove_bucket))
.route("/errors", get(list_errors))
.route("/errors/flush", axum::routing::post(flush_errors))
.route("/errors/compact", axum::routing::post(compact_errors))
.route("/errors/flush", post(flush_errors))
.route("/errors/compact", post(compact_errors))
.route("/bucket-health", get(get_health))
.route("/buckets/{bucket}/objects/{*key}", put(put_bucket_object))
.route("/buckets/{bucket}/objects/{*key}", get(get_bucket_object))
.with_state(registry)
}
/// Provision + register a bucket at runtime. Body is a `BucketConfig`.
///
/// Federation layer 2: profile buckets (`profile:alice`) and tenant
/// buckets can be added after startup without a service restart.
async fn add_bucket(
State(reg): State<Arc<BucketRegistry>>,
Json(bc): Json<BucketConfig>,
) -> impl IntoResponse {
// Safety net: local backends must land somewhere under the configured
// profile_root (for profile:*) or be otherwise explicitly rooted.
// Refuse empty / missing root to avoid "bucket created at ./" surprises.
if bc.backend == "local" {
match bc.root.as_deref() {
Some(r) if !r.trim().is_empty() => {}
_ => return Err((StatusCode::BAD_REQUEST,
"local bucket requires a non-empty 'root' path".to_string())),
}
}
match reg.add_bucket(bc).await {
Ok(info) => Ok((StatusCode::CREATED, Json(info))),
Err(e) => {
let code = if e.contains("already registered") {
StatusCode::CONFLICT
} else {
StatusCode::BAD_REQUEST
};
Err((code, e))
}
}
}
/// Unregister a bucket. Refused for primary / rescue / unknown names.
async fn remove_bucket(
State(reg): State<Arc<BucketRegistry>>,
Path(name): Path<String>,
) -> impl IntoResponse {
match reg.remove_bucket(&name) {
Ok(()) => Ok((StatusCode::OK, format!("unregistered: {name}"))),
Err(e) => {
let code = if e.contains("cannot remove") {
StatusCode::FORBIDDEN
} else if e.contains("not registered") {
StatusCode::NOT_FOUND
} else {
StatusCode::BAD_REQUEST
};
Err((code, e))
}
}
}
async fn list_buckets(State(reg): State<Arc<BucketRegistry>>) -> Json<Vec<BucketInfo>> {
Json(reg.list().await)
}
crates/storaged/src/registry.rs
+111 -23
View File
@@ -15,7 +15,7 @@ use serde::Serialize;
use shared::config::{BucketConfig, StorageConfig};
use shared::secrets::{BucketCredentials, SecretsProvider};
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::{Arc, RwLock};
use crate::error_journal::{BucketErrorEvent, ErrorJournal};
@@ -55,10 +55,16 @@ pub enum BucketRole {
}
pub struct BucketRegistry {
buckets: HashMap<String, Arc<BucketEntry>>,
/// RwLock because federation layer 2 adds runtime bucket lifecycle
/// (`POST /storage/buckets` / `DELETE /storage/buckets/{name}`).
/// Almost all accesses are reads — writes happen at provision /
/// deactivate time only.
buckets: RwLock<HashMap<String, Arc<BucketEntry>>>,
default: String,
rescue: Option<String>,
profile_root: String,
/// Held so runtime `add_bucket` can resolve secret_ref handles.
secrets: Arc<dyn SecretsProvider>,
journal: ErrorJournal,
}
@@ -113,10 +119,11 @@ impl BucketRegistry {
let _ = journal.load_recent().await;
Ok(Self {
buckets,
buckets: RwLock::new(buckets),
default: "primary".to_string(),
rescue: cfg.rescue_bucket.clone(),
profile_root: cfg.profile_root.clone(),
secrets,
journal,
})
}
@@ -125,12 +132,19 @@ impl BucketRegistry {
pub fn rescue_name(&self) -> Option<&str> { self.rescue.as_deref() }
pub fn profile_root(&self) -> &str { &self.profile_root }
pub fn journal(&self) -> &ErrorJournal { &self.journal }
/// Resolve a bucket name to its object store. Existing call sites use
/// this as a drop-in replacement for the old single-store pattern.
///
/// Uses `std::sync::RwLock` — every caller holds the guard for
/// microseconds (clones an Arc and drops), so there's no async-
/// blocking concern. Never hold across an await.
pub fn get(&self, bucket: &str) -> Result<Arc<dyn ObjectStore>, String> {
self.buckets
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard
.get(bucket)
.map(|e| e.store.clone())
.ok_or_else(|| format!("unknown bucket: {bucket}"))
@@ -139,22 +153,34 @@ impl BucketRegistry {
/// The default bucket's store — use for code paths that don't yet know
/// about buckets.
pub fn default_store(&self) -> Arc<dyn ObjectStore> {
self.buckets.get(&self.default).unwrap().store.clone()
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard.get(&self.default).unwrap().store.clone()
}
/// True if this bucket name is registered.
pub fn contains(&self, bucket: &str) -> bool {
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard.contains_key(bucket)
}
/// List all registered buckets. Checks reachability by doing a trivial
/// `list` with limit 1 on each.
pub async fn list(&self) -> Vec<BucketInfo> {
let mut out = Vec::with_capacity(self.buckets.len());
for (name, entry) in &self.buckets {
let reachable = probe(&entry.store).await;
let role = self.classify(name);
out.push(BucketInfo {
name: name.clone(),
backend: entry.backend.clone(),
reachable,
role,
});
// Snapshot (name, backend, store_clone) under the lock, then probe
// outside — probing can be slow (network), don't hold the lock.
let snapshot: Vec<(String, String, Arc<dyn ObjectStore>)> = {
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard
.iter()
.map(|(n, e)| (n.clone(), e.backend.clone(), e.store.clone()))
.collect()
};
let mut out = Vec::with_capacity(snapshot.len());
for (name, backend, store) in snapshot {
let reachable = probe(&store).await;
let role = self.classify(&name);
out.push(BucketInfo { name, backend, reachable, role });
}
out.sort_by(|a, b| a.name.cmp(&b.name));
out
@@ -167,14 +193,69 @@ impl BucketRegistry {
else { BucketRole::Tenant }
}
/// Provision + register a bucket at runtime (federation layer 2).
/// Returns Err if the name is already registered. Local backends get
/// their root directory created automatically.
pub async fn add_bucket(&self, bc: BucketConfig) -> Result<BucketInfo, String> {
if self.contains(&bc.name) {
return Err(format!("bucket '{}' already registered", bc.name));
}
let store = build_store(&bc, self.secrets.as_ref()).await?;
let reachable = probe(&store).await;
let entry = Arc::new(BucketEntry {
name: bc.name.clone(),
backend: bc.backend.clone(),
store,
config: bc.clone(),
});
let role = self.classify(&bc.name);
let info = BucketInfo {
name: bc.name.clone(),
backend: bc.backend.clone(),
reachable,
role,
};
{
let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
guard.insert(bc.name.clone(), entry);
}
tracing::info!("registered bucket '{}' backend={} reachable={}",
bc.name, bc.backend, reachable);
Ok(info)
}
/// Unregister a bucket. Refuses to remove `primary` or the configured
/// rescue bucket — those are load-bearing. Caller is responsible for
/// any final flush of bucket-local state before calling this.
pub fn remove_bucket(&self, name: &str) -> Result<(), String> {
if name == self.default {
return Err("cannot remove primary bucket".into());
}
if Some(name) == self.rescue.as_deref() {
return Err(format!("cannot remove rescue bucket '{name}'"));
}
let removed = {
let mut guard = self.buckets.write().expect("bucket registry lock poisoned");
guard.remove(name)
};
if removed.is_none() {
return Err(format!("bucket '{name}' not registered"));
}
tracing::info!("unregistered bucket '{}'", name);
Ok(())
}
/// Read with rescue-bucket fallback. If the target bucket fails and a
/// rescue is configured, retries against rescue. Records every failure
/// in the error journal.
pub async fn read_smart(&self, bucket: &str, key: &str) -> Result<ReadOutcome, String> {
let target = self.buckets.get(bucket)
.ok_or_else(|| format!("unknown bucket: {bucket}"))?;
let target_store = {
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard.get(bucket).map(|e| e.store.clone())
.ok_or_else(|| format!("unknown bucket: {bucket}"))?
};
match crate::ops::get(&target.store, key).await {
match crate::ops::get(&target_store, key).await {
Ok(data) => Ok(ReadOutcome {
data, rescued: false,
original_bucket: bucket.to_string(),
@@ -187,8 +268,12 @@ impl BucketRegistry {
// Try rescue, if any.
if let Some(rescue_name) = &self.rescue {
if rescue_name != bucket {
if let Some(rescue) = self.buckets.get(rescue_name) {
match crate::ops::get(&rescue.store, key).await {
let rescue_store = {
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard.get(rescue_name).map(|e| e.store.clone())
};
if let Some(rescue_store) = rescue_store {
match crate::ops::get(&rescue_store, key).await {
Ok(data) => {
self.journal.mark_rescued_last(bucket, key).await;
return Ok(ReadOutcome {
@@ -219,9 +304,12 @@ impl BucketRegistry {
key: &str,
data: bytes::Bytes,
) -> Result<(), String> {
let target = self.buckets.get(bucket)
.ok_or_else(|| format!("unknown bucket: {bucket}"))?;
match crate::ops::put(&target.store, key, data).await {
let target_store = {
let guard = self.buckets.read().expect("bucket registry lock poisoned");
guard.get(bucket).map(|e| e.store.clone())
.ok_or_else(|| format!("unknown bucket: {bucket}"))?
};
match crate::ops::put(&target_store, key, data).await {
Ok(()) => Ok(()),
Err(err) => {
self.journal.append(BucketErrorEvent::new_write(bucket, key, &err)).await;