Phases 16.2 + L2 + 17 VRAM gate + MySQL + 18 Lance hybrid milestone

Five threads of work landing as one milestone — all individually
verified end-to-end against real data, full release build clean,
46 unit tests pass.

## Phase 16.2 / 16.5 — autotune agent + ingest triggers

`vectord::agent` is a long-running tokio task that watches the trial
journal and autonomously proposes + runs new HNSW configs. Distinct
from `autotune::run_autotune` (synchronous one-shot grid). Triggered
on POST /vectors/agent/enqueue/{idx} or by the periodic wake; ingest
paths now push DatasetAppended events when an index's source dataset
gets re-ingested. Rate-limited (max_trials_per_hour) and cooldown-
gated so it can't saturate Ollama under live load.

The proposer is ε-greedy around the current champion: with prob 0.25
sample random from full bounds, otherwise perturb champion ± small
delta on both axes. Dedup against history. Deterministic — RNG seeded
from history.len() so the same journal state proposes the same next
config (helps offline replay debugging).

`[agent]` config section in lakehouse.toml; opt-in via enabled=true.

## Federation Layer 2 — runtime bucket lifecycle + per-index scoping

`BucketRegistry.buckets` moved to `std::sync::RwLock<HashMap>` so
buckets can be added/removed after startup. POST /storage/buckets
provisions at runtime; DELETE /storage/buckets/{name} unregisters
(refuses primary/rescue with 403). Local-backend buckets get their
root directory auto-created.

`IndexMeta.bucket` (default "primary" via serde) records each index's
home bucket. `TrialJournal` and `PromotionRegistry` now hold
Arc<BucketRegistry> + IndexRegistry; they resolve target store per-
index via IndexMeta.bucket. PromotionRegistry::list_all scans every
bucket and dedups by index_name. Pre-federation indexes keep working
unchanged — they just default to primary.

`ModelProfile.bucket: Option<String>` declares per-profile artifact
home. POST /vectors/profile/{id}/activate auto-provisions the
profile's bucket under storage.profile_root if not yet registered.

EvalSets stay primary-only for now — noted gap, low-risk to extend
later with the same resolver pattern.

## Phase 17 — VRAM-aware two-profile gate

Sidecar gains POST /admin/unload (Ollama keep_alive=0 trick — forces
immediate VRAM release), POST /admin/preload (keep_alive=5m with
empty prompt, takes the slot warm), and GET /admin/vram (combines
nvidia-smi snapshot with Ollama /api/ps). Exposed via aibridge as
unload_model / preload_model / vram_snapshot.

`VectorState.active_profile` is the GPU-slot singleton —
Arc<RwLock<Option<ActiveProfileSlot>>>. activate_profile checks for
a previous profile with a different ollama_name and unloads it
before preloading the new one; same-model reactivations skip the
unload (Ollama no-ops). New routes: POST /vectors/profile/{id}/
deactivate (unload + clear slot), GET /vectors/profile/active.

Verified live: staffing-recruiter (qwen2.5) → docs-assistant
(mistral) swap freed qwen2.5 from VRAM and loaded mistral. nomic-
embed-text persists across swaps because both profiles use it —
free optimization that fell out of the design. Scoped search
correctly 403s cross-profile in both directions.

## MySQL streaming connector

`crates/ingestd/src/my_stream.rs` mirrors pg_stream.rs for MySQL.
Pure-rust `mysql_async` driver (default-features=false to avoid C
deps). Same OFFSET pagination, same Parquet-streaming write shape.
Type mapping per ADR-010: int/bigint → Int32/Int64, decimal/float
→ Float64, tinyint(1)/bool → Boolean, everything else → Utf8 with
fallback parsers for date/time/json/uuid via Display.

POST /ingest/mysql parallel to /ingest/db. Same PII auto-detection,
same lineage capture (source_system="mysql"), same agent-trigger
hook. `redact_dsn` generalized — was hardcoded to "postgresql://"
length, now works for any scheme://user:pass@host/path URL (latent
PII leak fix for MySQL DSNs).

Verified live against MariaDB on localhost: 10 rows × 9 columns of
test data round-tripped through datatypes int/varchar/decimal/
tinyint/datetime/text. PII detection auto-flagged name + email.
Aggregation queries through DataFusion match the source values
exactly.

## Phase 18 — Hybrid Parquet+HNSW ⊕ Lance backend (ADR-019)

`vectord-lance` is a new firewall crate. Lance pulls Arrow 57 and
DataFusion 52 — incompatible with the rest of the workspace's
Arrow 55 / DataFusion 47. The firewall isolates that dep tree:
public API uses only std types (Vec<f32>, Vec<String>, Hit, Row,
*Stats), so no Arrow types cross the crate boundary and nothing
propagates to vectord. The ADR-019 path that didn't ship until now.

`vectord::lance_backend::LanceRegistry` lazy-creates a
LanceVectorStore per index, resolving bucket → URI via the
conventional local-bucket layout. `IndexMeta.vector_backend` and
`ModelProfile.vector_backend` carry the choice (default Parquet so
existing indexes unchanged).

Six routes under /vectors/lance/*:
- migrate/{idx}: convert binary-blob Parquet → Lance FixedSizeList
- index/{idx}: build IVF_PQ
- search/{idx}: vector search (embed via sidecar)
- doc/{idx}/{doc_id}: random row fetch
- append/{idx}: native fragment append
- stats/{idx}: row count + index presence

Verified live on the real resumes_100k_v2 corpus (100K × 768d):
- Migrate: 0.57s
- Build IVF_PQ index: 16.2s (matches ADR-019 bench; 14× faster than
  HNSW's 230s for the same data)
- Search end-to-end (Ollama embed + Lance scan): 23-53ms
- Random doc_id fetch: 5-7ms (filter scan; faster than Parquet's
  ~35ms full-file scan, slower than the bench's 311us positional
  take — would close that gap with a scalar btree on doc_id)
- Append 100 rows: 3.3ms / +320KB on disk vs Parquet's required
  full ~330MB rewrite — the structural win
- Index survives append; both backends coexist cleanly

## Known follow-ups not in this milestone

- ModelProfile.vector_backend doesn't yet auto-route /vectors/profile/
  {id}/search to Lance; callers go through /vectors/lance/* directly
- Scalar btree on doc_id (closes the 5-7ms → ~300us gap)
- vectord-lance built default-features=false → no S3 yet
- IVF_PQ recall not measured (ADR-019 caveat) — needs a Lance-aware
  variant of the eval harness
- Watcher-path ingest doesn't push agent triggers (HTTP paths do)
- EvalSets still primary-only (federation gap)
- No PATCH endpoint to move an existing index between buckets
- The pre-existing storaged::append_log doctest fails to compile
  (malformed `{prefix}/` parses as code fence) — pre-existing bug,
  left for a focused fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/lance_backend.rs

@@ -0,0 +1,112 @@
+//! ADR-019 hybrid: routing layer for the Lance vector backend.
+//!
+//! Holds a `LanceRegistry` which maps an index name to its
+//! `LanceVectorStore` instance, lazy-creating on first touch. Path
+//! resolution: the index's bucket gives us a bucket root; we append
+//! `lance/{index_name}` and use that as the dataset URI. For local
+//! buckets that means a directory under the bucket's root.
+//!
+//! S3 buckets: in principle Lance accepts s3://... URIs, but the
+//! firewall crate (`vectord-lance`) doesn't pull the S3 feature in by
+//! default. When we promote profile buckets onto S3 in a future phase,
+//! enable that feature and update `lance_uri_for` accordingly.
+
+use std::collections::HashMap;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::sync::RwLock;
+
+use storaged::registry::BucketRegistry;
+use vectord_lance::LanceVectorStore;
+
+use crate::index_registry::IndexRegistry;
+
+/// Convert a bucket+index pair into the URI Lance should use as the
+/// dataset path. Local-only for MVP; S3 when we wire that backend.
+///
+/// Path resolution mirrors lakehouse.toml's convention for local
+/// buckets: ./data for primary, ./data/_rescue for rescue, ./data/_testing
+/// for testing, ./data/_profiles/{sanitized} for profile:* buckets, and
+/// ./data/_buckets/{sanitized} for everything else. Sanitization replaces
+/// `:` with `_` so paths are filesystem-safe.
+///
+/// Refuses unknown buckets so a typo doesn't silently land Lance data
+/// in a directory the rest of the system can't see.
+pub fn lance_uri_for(
+    buckets: &BucketRegistry,
+    bucket: &str,
+    index_name: &str,
+) -> Result<String, String> {
+    if !buckets.contains(bucket) {
+        return Err(format!("bucket '{bucket}' not registered"));
+    }
+    let root: PathBuf = match bucket {
+        "primary" => PathBuf::from("./data"),
+        "rescue"  => PathBuf::from("./data/_rescue"),
+        "testing" => PathBuf::from("./data/_testing"),
+        b if b.starts_with("profile:") => {
+            let safe = b.replace(':', "_");
+            PathBuf::from(format!("./data/_profiles/{safe}"))
+        }
+        b => PathBuf::from(format!("./data/_buckets/{}", b.replace(':', "_"))),
+    };
+    let dataset_dir = root.join("lance").join(index_name);
+    // Pre-create the parent so Lance's first write doesn't trip on a
+    // missing ancestor. Lance handles the dataset directory itself.
+    let _ = std::fs::create_dir_all(root.join("lance"));
+    // Canonicalize after the parent is guaranteed to exist; if the
+    // dataset dir hasn't been created yet, canonicalize the parent and
+    // tack on the leaf name.
+    let abs = match std::fs::canonicalize(&root) {
+        Ok(p) => p.join("lance").join(index_name),
+        Err(_) => dataset_dir.clone(),
+    };
+    Ok(abs.to_string_lossy().to_string())
+}
+
+/// Lookup-by-index registry of `LanceVectorStore` handles. One handle
+/// per index, lazy-created on first call. Cheap to keep alive — it's
+/// just a path string + handle to Lance's metadata cache.
+#[derive(Clone)]
+pub struct LanceRegistry {
+    buckets: Arc<BucketRegistry>,
+    indexes: IndexRegistry,
+    stores: Arc<RwLock<HashMap<String, LanceVectorStore>>>,
+}
+
+impl LanceRegistry {
+    pub fn new(buckets: Arc<BucketRegistry>, indexes: IndexRegistry) -> Self {
+        Self {
+            buckets,
+            indexes,
+            stores: Arc::new(RwLock::new(HashMap::new())),
+        }
+    }
+
+    /// Get (or lazy-create) the LanceVectorStore for an index.
+    /// Resolves bucket → URI via the registry; backend stays whatever
+    /// IndexMeta says (no enforcement here — caller decides whether to
+    /// route to Lance based on profile/index backend choice).
+    pub async fn store_for(&self, index_name: &str) -> Result<LanceVectorStore, String> {
+        if let Some(s) = self.stores.read().await.get(index_name) {
+            return Ok(s.clone());
+        }
+        let bucket = self.indexes
+            .get(index_name).await
+            .map(|m| m.bucket)
+            .unwrap_or_else(|| "primary".to_string());
+        let uri = lance_uri_for(&self.buckets, &bucket, index_name)?;
+        let store = LanceVectorStore::new(uri);
+        self.stores.write().await.insert(index_name.to_string(), store.clone());
+        Ok(store)
+    }
+
+    /// For freshly-created indexes that don't have IndexMeta yet — caller
+    /// supplies the bucket explicitly.
+    pub async fn store_for_new(&self, index_name: &str, bucket: &str) -> Result<LanceVectorStore, String> {
+        let uri = lance_uri_for(&self.buckets, bucket, index_name)?;
+        let store = LanceVectorStore::new(uri);
+        self.stores.write().await.insert(index_name.to_string(), store.clone());
+        Ok(store)
+    }
+}