Phases 16.2 + L2 + 17 VRAM gate + MySQL + 18 Lance hybrid milestone

Five threads of work landing as one milestone — all individually
verified end-to-end against real data, full release build clean,
46 unit tests pass.

## Phase 16.2 / 16.5 — autotune agent + ingest triggers

`vectord::agent` is a long-running tokio task that watches the trial
journal and autonomously proposes + runs new HNSW configs. Distinct
from `autotune::run_autotune` (synchronous one-shot grid). Triggered
on POST /vectors/agent/enqueue/{idx} or by the periodic wake; ingest
paths now push DatasetAppended events when an index's source dataset
gets re-ingested. Rate-limited (max_trials_per_hour) and cooldown-
gated so it can't saturate Ollama under live load.

The proposer is ε-greedy around the current champion: with prob 0.25
sample random from full bounds, otherwise perturb champion ± small
delta on both axes. Dedup against history. Deterministic — RNG seeded
from history.len() so the same journal state proposes the same next
config (helps offline replay debugging).

`[agent]` config section in lakehouse.toml; opt-in via enabled=true.

## Federation Layer 2 — runtime bucket lifecycle + per-index scoping

`BucketRegistry.buckets` moved to `std::sync::RwLock<HashMap>` so
buckets can be added/removed after startup. POST /storage/buckets
provisions at runtime; DELETE /storage/buckets/{name} unregisters
(refuses primary/rescue with 403). Local-backend buckets get their
root directory auto-created.

`IndexMeta.bucket` (default "primary" via serde) records each index's
home bucket. `TrialJournal` and `PromotionRegistry` now hold
Arc<BucketRegistry> + IndexRegistry; they resolve target store per-
index via IndexMeta.bucket. PromotionRegistry::list_all scans every
bucket and dedups by index_name. Pre-federation indexes keep working
unchanged — they just default to primary.

`ModelProfile.bucket: Option<String>` declares per-profile artifact
home. POST /vectors/profile/{id}/activate auto-provisions the
profile's bucket under storage.profile_root if not yet registered.

EvalSets stay primary-only for now — noted gap, low-risk to extend
later with the same resolver pattern.

## Phase 17 — VRAM-aware two-profile gate

Sidecar gains POST /admin/unload (Ollama keep_alive=0 trick — forces
immediate VRAM release), POST /admin/preload (keep_alive=5m with
empty prompt, takes the slot warm), and GET /admin/vram (combines
nvidia-smi snapshot with Ollama /api/ps). Exposed via aibridge as
unload_model / preload_model / vram_snapshot.

`VectorState.active_profile` is the GPU-slot singleton —
Arc<RwLock<Option<ActiveProfileSlot>>>. activate_profile checks for
a previous profile with a different ollama_name and unloads it
before preloading the new one; same-model reactivations skip the
unload (Ollama no-ops). New routes: POST /vectors/profile/{id}/
deactivate (unload + clear slot), GET /vectors/profile/active.

Verified live: staffing-recruiter (qwen2.5) → docs-assistant
(mistral) swap freed qwen2.5 from VRAM and loaded mistral. nomic-
embed-text persists across swaps because both profiles use it —
free optimization that fell out of the design. Scoped search
correctly 403s cross-profile in both directions.

## MySQL streaming connector

`crates/ingestd/src/my_stream.rs` mirrors pg_stream.rs for MySQL.
Pure-rust `mysql_async` driver (default-features=false to avoid C
deps). Same OFFSET pagination, same Parquet-streaming write shape.
Type mapping per ADR-010: int/bigint → Int32/Int64, decimal/float
→ Float64, tinyint(1)/bool → Boolean, everything else → Utf8 with
fallback parsers for date/time/json/uuid via Display.

POST /ingest/mysql parallel to /ingest/db. Same PII auto-detection,
same lineage capture (source_system="mysql"), same agent-trigger
hook. `redact_dsn` generalized — was hardcoded to "postgresql://"
length, now works for any scheme://user:pass@host/path URL (latent
PII leak fix for MySQL DSNs).

Verified live against MariaDB on localhost: 10 rows × 9 columns of
test data round-tripped through datatypes int/varchar/decimal/
tinyint/datetime/text. PII detection auto-flagged name + email.
Aggregation queries through DataFusion match the source values
exactly.

## Phase 18 — Hybrid Parquet+HNSW ⊕ Lance backend (ADR-019)

`vectord-lance` is a new firewall crate. Lance pulls Arrow 57 and
DataFusion 52 — incompatible with the rest of the workspace's
Arrow 55 / DataFusion 47. The firewall isolates that dep tree:
public API uses only std types (Vec<f32>, Vec<String>, Hit, Row,
*Stats), so no Arrow types cross the crate boundary and nothing
propagates to vectord. The ADR-019 path that didn't ship until now.

`vectord::lance_backend::LanceRegistry` lazy-creates a
LanceVectorStore per index, resolving bucket → URI via the
conventional local-bucket layout. `IndexMeta.vector_backend` and
`ModelProfile.vector_backend` carry the choice (default Parquet so
existing indexes unchanged).

Six routes under /vectors/lance/*:
- migrate/{idx}: convert binary-blob Parquet → Lance FixedSizeList
- index/{idx}: build IVF_PQ
- search/{idx}: vector search (embed via sidecar)
- doc/{idx}/{doc_id}: random row fetch
- append/{idx}: native fragment append
- stats/{idx}: row count + index presence

Verified live on the real resumes_100k_v2 corpus (100K × 768d):
- Migrate: 0.57s
- Build IVF_PQ index: 16.2s (matches ADR-019 bench; 14× faster than
  HNSW's 230s for the same data)
- Search end-to-end (Ollama embed + Lance scan): 23-53ms
- Random doc_id fetch: 5-7ms (filter scan; faster than Parquet's
  ~35ms full-file scan, slower than the bench's 311us positional
  take — would close that gap with a scalar btree on doc_id)
- Append 100 rows: 3.3ms / +320KB on disk vs Parquet's required
  full ~330MB rewrite — the structural win
- Index survives append; both backends coexist cleanly

## Known follow-ups not in this milestone

- ModelProfile.vector_backend doesn't yet auto-route /vectors/profile/
  {id}/search to Lance; callers go through /vectors/lance/* directly
- Scalar btree on doc_id (closes the 5-7ms → ~300us gap)
- vectord-lance built default-features=false → no S3 yet
- IVF_PQ recall not measured (ADR-019 caveat) — needs a Lance-aware
  variant of the eval harness
- Watcher-path ingest doesn't push agent triggers (HTTP paths do)
- EvalSets still primary-only (federation gap)
- No PATCH endpoint to move an existing index between buckets
- The pre-existing storaged::append_log doctest fails to compile
  (malformed `{prefix}/` parses as code fence) — pre-existing bug,
  left for a focused fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/promotion.rs

@@ -19,13 +19,14 @@
 //! - Agent loop — lives in `vectord::autotune`.
 
 use chrono::{DateTime, Utc};
-use object_store::ObjectStore;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::Arc;
 use storaged::ops;
+use storaged::registry::BucketRegistry;
 use tokio::sync::RwLock;
 
+use crate::index_registry::IndexRegistry;
 use crate::trial::HnswConfig;
 
 const PROMOTION_PREFIX: &str = "_hnsw_promotions";
@@ -55,14 +56,16 @@ pub struct PromotionFile {
 
 #[derive(Clone)]
 pub struct PromotionRegistry {
-    store: Arc<dyn ObjectStore>,
+    buckets: Arc<BucketRegistry>,
+    index_registry: IndexRegistry,
     cache: Arc<RwLock<HashMap<String, PromotionFile>>>,
 }
 
 impl PromotionRegistry {
-    pub fn new(store: Arc<dyn ObjectStore>) -> Self {
+    pub fn new(buckets: Arc<BucketRegistry>, index_registry: IndexRegistry) -> Self {
         Self {
-            store,
+            buckets,
+            index_registry,
             cache: Arc::new(RwLock::new(HashMap::new())),
         }
     }
@@ -76,13 +79,26 @@ impl PromotionRegistry {
         format!("{PROMOTION_PREFIX}/{safe}.json")
     }
 
+    /// Resolve which bucket's store holds this index's promotion file.
+    /// Same rules as TrialJournal::bucket_for — follows IndexMeta.bucket,
+    /// defaults to primary when metadata is missing.
+    async fn store_for(&self, index_name: &str) -> Result<Arc<dyn object_store::ObjectStore>, String> {
+        let bucket = self.index_registry
+            .get(index_name)
+            .await
+            .map(|m| m.bucket)
+            .unwrap_or_else(|| "primary".to_string());
+        self.buckets.get(&bucket)
+    }
+
     /// Load (and cache) the promotion file for an index.
     pub async fn load(&self, index_name: &str) -> Result<PromotionFile, String> {
         if let Some(cached) = self.cache.read().await.get(index_name) {
             return Ok(cached.clone());
         }
+        let store = self.store_for(index_name).await?;
         let key = Self::key(index_name);
-        let file = match ops::get(&self.store, &key).await {
+        let file = match ops::get(&store, &key).await {
             Ok(bytes) => serde_json::from_slice::<PromotionFile>(&bytes)
                 .map_err(|e| format!("parse promotion file: {e}"))?,
             Err(_) => PromotionFile {
@@ -118,9 +134,10 @@ impl PromotionRegistry {
         file.current = Some(entry);
         file.index_name = index_name.to_string();
 
+        let store = self.store_for(index_name).await?;
         let key = Self::key(index_name);
         let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &key, json.into()).await?;
+        ops::put(&store, &key, json.into()).await?;
 
         self.cache.write().await.insert(index_name.to_string(), file.clone());
         tracing::info!(
@@ -146,9 +163,10 @@ impl PromotionRegistry {
                 file.current = None;
             }
         }
+        let store = self.store_for(index_name).await?;
         let key = Self::key(index_name);
         let json = serde_json::to_vec_pretty(&file).map_err(|e| e.to_string())?;
-        ops::put(&self.store, &key, json.into()).await?;
+        ops::put(&store, &key, json.into()).await?;
         self.cache.write().await.insert(index_name.to_string(), file.clone());
         tracing::info!("rolled back promotion for '{}'", index_name);
         Ok(file)
@@ -169,19 +187,34 @@ impl PromotionRegistry {
     }
 
     /// List every index that has a promotion recorded (for operator UI).
+    ///
+    /// Federation: scans EVERY registered bucket for promotion files.
+    /// Per-profile buckets each have their own `_hnsw_promotions/` so we
+    /// aggregate across them. Dedups by index_name — if the same index
+    /// somehow has promotion files in multiple buckets, the one from the
+    /// bucket recorded in IndexMeta wins.
     pub async fn list_all(&self) -> Result<Vec<PromotionFile>, String> {
-        let keys = ops::list(&self.store, Some(&format!("{PROMOTION_PREFIX}/"))).await.unwrap_or_default();
-        let mut out = Vec::new();
-        for key in keys {
-            if !key.ends_with(".json") { continue; }
-            let bytes = match ops::get(&self.store, &key).await {
-                Ok(b) => b,
+        let bucket_infos = self.buckets.list().await;
+        let mut by_name: HashMap<String, PromotionFile> = HashMap::new();
+
+        for b in &bucket_infos {
+            let store = match self.buckets.get(&b.name) {
+                Ok(s) => s,
                 Err(_) => continue,
             };
-            if let Ok(f) = serde_json::from_slice::<PromotionFile>(&bytes) {
-                out.push(f);
+            let keys = ops::list(&store, Some(&format!("{PROMOTION_PREFIX}/")))
+                .await.unwrap_or_default();
+            for key in keys {
+                if !key.ends_with(".json") { continue; }
+                let bytes = match ops::get(&store, &key).await {
+                    Ok(b) => b,
+                    Err(_) => continue,
+                };
+                if let Ok(f) = serde_json::from_slice::<PromotionFile>(&bytes) {
+                    by_name.insert(f.index_name.clone(), f);
+                }
             }
         }
-        Ok(out)
+        Ok(by_name.into_values().collect())
     }
 }