lance backend: 4-pack — bug fix + smoke + tests + 10M re-bench

Surfaced by the 2026-05-02 audit (vectord-lance + lance-bench + glue
existed and worked but had no tests, no smoke, leaked server paths
on missing-index search, and the ADR-019 10M re-bench was deferred).

## 1. Fix: missing-index search returned 500 + leaked filesystem path

Pre-fix:
  $ POST /vectors/lance/search/no-such-index
  HTTP 500
  Dataset at path home/profit/lakehouse/data/lance/no-such-index was
  not found: Not found: home/profit/lakehouse/data/lance/no-such-index/
  _versions, /root/.cargo/registry/src/index.crates.io-...-1949cf8c.../
  lance-table-4.0.0/src/io/commit.rs:364:26, ...

Post-fix:
  HTTP 404
  lance dataset not found: no-such-index

Added `sanitize_lance_err()` in crates/vectord/src/service.rs that:
  - maps "not found" / "no such file" patterns → 404 (was 500)
  - strips /home/ and /root/.cargo/ paths from any error body
Applied to all 5 lance handlers: search, get_doc, build_index,
append, migrate. The store_for() handle is cheap-and-stateless;
the actual disk hit happens inside the operation, which is where
the leak originated.

## 2. scripts/lance_smoke.sh — first regression gate

9-probe smoke against the live HTTP surface. Exercises only read
paths (no state mutation in CI). Specifically locks the sanitizer
fix — a future regression that re-introduces the path leak fires
the smoke immediately. 9/9 PASS against the live :3100 today.

## 3. Unit tests on vectord-lance/src/lib.rs (was: zero tests)

7 tests covering the public LanceVectorStore API:
  - fresh_store_reports_no_state — handle is lazy
  - migrate_then_count_and_fetch — Parquet → Lance round-trip
  - get_by_doc_id_missing_returns_none — Ok(None) vs Err contract
    that lets the HTTP handler return 404 cleanly
  - append_grows_count_and_new_rows_fetchable — ADR-019's
    structural-difference claim verified at the unit level
  - append_dim_mismatch_errors — guards against silently breaking
    search by accepting inconsistent-dim rows
  - search_returns_nearest — exact-vector match → top-1
  - stats_reports_post_migrate_state — locks the field shape

7/7 PASS. cargo test -p vectord-lance --lib green.

## 4. 10M re-bench (deferred from ADR-019)

reports/lance_10m_rebench_2026-05-02.md captures the numbers driven
against the live :3100 over data/lance/scale_test_10m (33GB / 10M
vectors, IVF_PQ confirmed via response method tag).

Headline:
  Search cold (10 diverse queries):   median ~32ms, mean ~46ms
  Search warm (5x same query):        ~20ms p50
  Doc fetch (5x same id):             ~100ms p50

Search latency at 10M is acceptable for batch / async workloads,
too slow for sub-10ms voice/recommendation paths. ADR-019's "Lance
pulls ahead at 10M" claim remains unverified-but-not-refuted — at
this scale HNSW doesn't operationally exist (10M × 768d × 4 bytes =
30GB just for vectors).

Real finding: doc-fetch at 10M is 300x slower than the 100K number
ADR-019 cited (311μs → ~100ms). Likely cause: scalar btree index
on doc_id may not be built for this dataset. Follow-up to
investigate whether forcing build_scalar_index brings it back to
the load-bearing O(1) range. Captured in the report.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

crates/vectord/src/service.rs

@@ -1855,10 +1855,10 @@ async fn lance_migrate(
         .map_err(|e| (StatusCode::NOT_FOUND, format!("read parquet: {e}")))?;
 
     let lance_store = state.lance.store_for_new(&index_name, &bucket).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let stats = lance_store.migrate_from_parquet_bytes(&bytes).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     tracing::info!(
         "lance migrate '{}': {} rows, {}d, {} bytes on disk, {:.2}s",
@@ -1888,6 +1888,38 @@ fn default_partitions() -> u32 { 316 }   // ≈√100K — sane for the referenc
 fn default_bits() -> u32 { 8 }
 fn default_subvectors() -> u32 { 48 }    // 768/48 = 16 dims per subvector
 
+/// Sanitize a Lance backend error before returning it to the HTTP
+/// caller. Two responsibilities:
+///
+/// 1. Map "dataset not found" patterns to HTTP 404 instead of 500.
+///    A missing index isn't an internal failure — it's a resource
+///    lookup miss, and the response code should reflect that.
+/// 2. Strip server-side filesystem paths and Rust crate registry
+///    paths (`/root/.cargo/registry/src/index.crates.io-...`) from
+///    the message body. An attacker probing the surface shouldn't
+///    learn the server's directory layout or our exact dep versions.
+///
+/// Surfaced 2026-05-02 by the Lance backend audit: missing-index
+/// search returned 500 + leaked the lakehouse data path AND the
+/// .cargo/registry path with crate versions.
+fn sanitize_lance_err(err: String, index_name: &str) -> (StatusCode, String) {
+    let lower = err.to_lowercase();
+    let is_not_found = lower.contains("not found") || lower.contains("no such file");
+    let msg = if is_not_found {
+        format!("lance dataset not found: {index_name}")
+    } else {
+        // Generic 500 with the Lance error body trimmed of paths.
+        let cleaned = err
+            .split("/root/.cargo/").next().unwrap_or(&err)
+            .split("/home/").next().unwrap_or(&err)
+            .trim_end_matches([',', ' ', '\n', '\t'])
+            .to_string();
+        if cleaned.is_empty() { format!("lance backend error on {index_name}") } else { cleaned }
+    };
+    let status = if is_not_found { StatusCode::NOT_FOUND } else { StatusCode::INTERNAL_SERVER_ERROR };
+    (status, msg)
+}
+
 /// Build the IVF_PQ index on the Lance dataset.
 async fn lance_build_index(
     State(state): State<VectorState>,
@@ -1895,10 +1927,10 @@ async fn lance_build_index(
     Json(req): Json<LanceIndexRequest>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     match lance_store.build_index(req.num_partitions, req.num_bits, req.num_sub_vectors).await {
         Ok(stats) => Ok(Json(stats)),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -1947,13 +1979,13 @@ async fn lance_search(
     let qv: Vec<f32> = embed_resp.embeddings[0].iter().map(|&x| x as f32).collect();
 
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let t0 = std::time::Instant::now();
     let nprobes = req.nprobes.or(Some(LANCE_DEFAULT_NPROBES));
     let refine = req.refine_factor.or(Some(LANCE_DEFAULT_REFINE_FACTOR));
     let hits = lance_store.search(&qv, req.top_k, nprobes, refine).await
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     Ok(Json(serde_json::json!({
         "index_name": index_name,
@@ -1971,7 +2003,7 @@ async fn lance_get_doc(
     Path((index_name, doc_id)): Path<(String, String)>,
 ) -> impl IntoResponse {
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
     let t0 = std::time::Instant::now();
     match lance_store.get_by_doc_id(&doc_id).await {
         Ok(Some(row)) => Ok(Json(serde_json::json!({
@@ -1981,7 +2013,7 @@ async fn lance_get_doc(
             "row": row,
         }))),
         Ok(None) => Err((StatusCode::NOT_FOUND, format!("doc_id not found: {doc_id}"))),
-        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        Err(e) => Err(sanitize_lance_err(e, &index_name)),
     }
 }
 
@@ -2013,7 +2045,7 @@ async fn lance_append(
         return Err((StatusCode::BAD_REQUEST, "rows array is empty".into()));
     }
     let lance_store = state.lance.store_for(&index_name).await
-        .map_err(|e| (StatusCode::BAD_REQUEST, e))?;
+        .map_err(|e| sanitize_lance_err(e, &index_name))?;
 
     let mut doc_ids = Vec::with_capacity(req.rows.len());
     let mut chunk_idxs = Vec::with_capacity(req.rows.len());