lance backend: 4-pack — bug fix + smoke + tests + 10M re-bench

Surfaced by the 2026-05-02 audit (vectord-lance + lance-bench + glue
existed and worked but had no tests, no smoke, leaked server paths
on missing-index search, and the ADR-019 10M re-bench was deferred).

## 1. Fix: missing-index search returned 500 + leaked filesystem path

Pre-fix:
  $ POST /vectors/lance/search/no-such-index
  HTTP 500
  Dataset at path home/profit/lakehouse/data/lance/no-such-index was
  not found: Not found: home/profit/lakehouse/data/lance/no-such-index/
  _versions, /root/.cargo/registry/src/index.crates.io-...-1949cf8c.../
  lance-table-4.0.0/src/io/commit.rs:364:26, ...

Post-fix:
  HTTP 404
  lance dataset not found: no-such-index

Added `sanitize_lance_err()` in crates/vectord/src/service.rs that:
  - maps "not found" / "no such file" patterns → 404 (was 500)
  - strips /home/ and /root/.cargo/ paths from any error body
Applied to all 5 lance handlers: search, get_doc, build_index,
append, migrate. The store_for() handle is cheap-and-stateless;
the actual disk hit happens inside the operation, which is where
the leak originated.

## 2. scripts/lance_smoke.sh — first regression gate

9-probe smoke against the live HTTP surface. Exercises only read
paths (no state mutation in CI). Specifically locks the sanitizer
fix — a future regression that re-introduces the path leak fires
the smoke immediately. 9/9 PASS against the live :3100 today.

## 3. Unit tests on vectord-lance/src/lib.rs (was: zero tests)

7 tests covering the public LanceVectorStore API:
  - fresh_store_reports_no_state — handle is lazy
  - migrate_then_count_and_fetch — Parquet → Lance round-trip
  - get_by_doc_id_missing_returns_none — Ok(None) vs Err contract
    that lets the HTTP handler return 404 cleanly
  - append_grows_count_and_new_rows_fetchable — ADR-019's
    structural-difference claim verified at the unit level
  - append_dim_mismatch_errors — guards against silently breaking
    search by accepting inconsistent-dim rows
  - search_returns_nearest — exact-vector match → top-1
  - stats_reports_post_migrate_state — locks the field shape

7/7 PASS. cargo test -p vectord-lance --lib green.

## 4. 10M re-bench (deferred from ADR-019)

reports/lance_10m_rebench_2026-05-02.md captures the numbers driven
against the live :3100 over data/lance/scale_test_10m (33GB / 10M
vectors, IVF_PQ confirmed via response method tag).

Headline:
  Search cold (10 diverse queries):   median ~32ms, mean ~46ms
  Search warm (5x same query):        ~20ms p50
  Doc fetch (5x same id):             ~100ms p50

Search latency at 10M is acceptable for batch / async workloads,
too slow for sub-10ms voice/recommendation paths. ADR-019's "Lance
pulls ahead at 10M" claim remains unverified-but-not-refuted — at
this scale HNSW doesn't operationally exist (10M × 768d × 4 bytes =
30GB just for vectors).

Real finding: doc-fetch at 10M is 300x slower than the 100K number
ADR-019 cited (311μs → ~100ms). Likely cause: scalar btree index
on doc_id may not be built for this dataset. Follow-up to
investigate whether forcing build_scalar_index brings it back to
the load-bearing O(1) range. Captured in the report.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

crates/vectord-lance/src/lib.rs

@@ -603,3 +603,200 @@ fn row_from_batch(batch: &RecordBatch, row: usize) -> Result<Row, String> {
 
     Ok(Row { doc_id, chunk_text, vector: v, source, chunk_idx })
 }
+
+// =================== Tests ===================
+//
+// All tests run against a temp directory — never the production
+// data/lance/ tree. Lance reads/writes are async + filesystem-bound,
+// so we use #[tokio::test]. Each test uses a unique per-pid + per-
+// nanosecond temp dir so concurrent runs don't collide and a re-run
+// of a single test doesn't see prior state.
+//
+// Surfaced 2026-05-02 audit: vectord-lance had ZERO tests despite
+// being on the live HTTP path. These are the load-bearing locks for
+// the public API contract.
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn temp_path(label: &str) -> String {
+        let n = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .map(|d| d.subsec_nanos())
+            .unwrap_or(0);
+        let pid = std::process::id();
+        std::env::temp_dir()
+            .join(format!("vlance_test_{label}_{pid}_{n}"))
+            .to_string_lossy()
+            .to_string()
+    }
+
+    /// Build a minimal in-memory Parquet file matching vectord's
+    /// binary-blob schema. Used as input to migrate_from_parquet_bytes.
+    fn synth_parquet_bytes(n_rows: usize, dims: usize) -> Vec<u8> {
+        use parquet::arrow::ArrowWriter;
+        use std::io::Cursor;
+
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("source", DataType::Utf8, true),
+            Field::new("doc_id", DataType::Utf8, false),
+            Field::new("chunk_idx", DataType::Int32, true),
+            Field::new("chunk_text", DataType::Utf8, true),
+            Field::new("vector", DataType::Binary, false),
+        ]));
+
+        let sources: Vec<Option<&str>> = (0..n_rows).map(|_| Some("test")).collect();
+        let doc_ids: Vec<String> = (0..n_rows).map(|i| format!("DOC-{i:04}")).collect();
+        let chunk_idxs: Vec<Option<i32>> = (0..n_rows).map(|i| Some(i as i32)).collect();
+        let chunk_texts: Vec<String> = (0..n_rows).map(|i| format!("synth chunk {i}")).collect();
+        let vectors: Vec<Vec<u8>> = (0..n_rows).map(|i| {
+            let v: Vec<f32> = (0..dims).map(|j| (i * dims + j) as f32 * 0.01).collect();
+            let mut bytes = Vec::with_capacity(dims * 4);
+            for f in v { bytes.extend_from_slice(&f.to_le_bytes()); }
+            bytes
+        }).collect();
+
+        let batch = RecordBatch::try_new(schema.clone(), vec![
+            Arc::new(StringArray::from(sources)),
+            Arc::new(StringArray::from(doc_ids)),
+            Arc::new(Int32Array::from(chunk_idxs)),
+            Arc::new(StringArray::from(chunk_texts)),
+            Arc::new(BinaryArray::from(vectors.iter().map(|v| v.as_slice()).collect::<Vec<_>>())),
+        ]).expect("synth parquet batch");
+
+        let mut buf = Cursor::new(Vec::new());
+        let mut writer = ArrowWriter::try_new(&mut buf, schema, None).expect("arrow writer");
+        writer.write(&batch).expect("write batch");
+        writer.close().expect("close writer");
+        buf.into_inner()
+    }
+
+    #[tokio::test]
+    async fn fresh_store_reports_no_state() {
+        let path = temp_path("fresh");
+        let store = LanceVectorStore::new(path.clone());
+        assert_eq!(store.path(), path);
+        assert_eq!(store.count().await.unwrap_or(0), 0);
+        assert!(!store.has_vector_index().await.unwrap_or(true));
+    }
+
+    #[tokio::test]
+    async fn migrate_then_count_and_fetch() {
+        let path = temp_path("migrate_fetch");
+        let store = LanceVectorStore::new(path.clone());
+        let bytes = synth_parquet_bytes(8, 4);
+
+        let stats = store.migrate_from_parquet_bytes(&bytes).await.expect("migrate");
+        assert_eq!(stats.rows_written, 8);
+        assert_eq!(stats.dimensions, 4);
+        assert!(stats.disk_bytes > 0, "lance dataset should occupy disk");
+
+        assert_eq!(store.count().await.unwrap(), 8);
+
+        let row = store.get_by_doc_id("DOC-0003").await
+            .expect("get_by_doc_id Ok").expect("DOC-0003 exists");
+        assert_eq!(row.doc_id, "DOC-0003");
+        assert_eq!(row.chunk_text, "synth chunk 3");
+        assert_eq!(row.vector.len(), 4);
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Load-bearing contract: get_by_doc_id distinguishes "dataset
+    /// missing" (Err) from "id missing" (Ok(None)) so the HTTP
+    /// handler can return 404 without inspecting error strings.
+    #[tokio::test]
+    async fn get_by_doc_id_missing_returns_none() {
+        let path = temp_path("missing_id");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let row = store.get_by_doc_id("DOC-NEVER-EXISTS").await.expect("Ok");
+        assert!(row.is_none(), "missing id must return Ok(None), not Err");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Verifies the load-bearing structural-difference claim of
+    /// ADR-019: Lance appends without rewriting the whole file. Row
+    /// count grows; new rows are fetchable by their doc_ids.
+    #[tokio::test]
+    async fn append_grows_count_and_new_rows_fetchable() {
+        let path = temp_path("append");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+        assert_eq!(store.count().await.unwrap(), 4);
+
+        let stats = store.append(
+            Some("appended".into()),
+            vec!["NEW-A".into(), "NEW-B".into()],
+            vec![0, 0],
+            vec!["new chunk a".into(), "new chunk b".into()],
+            vec![vec![0.1, 0.2, 0.3, 0.4], vec![0.5, 0.6, 0.7, 0.8]],
+        ).await.expect("append");
+
+        assert_eq!(stats.rows_appended, 2);
+        assert_eq!(store.count().await.unwrap(), 6);
+
+        let new_a = store.get_by_doc_id("NEW-A").await.unwrap().expect("NEW-A");
+        assert_eq!(new_a.chunk_text, "new chunk a");
+        assert_eq!(new_a.source.as_deref(), Some("appended"));
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Without this guard a dim-mismatch row would land on disk and
+    /// silently break search at query time.
+    #[tokio::test]
+    async fn append_dim_mismatch_errors() {
+        let path = temp_path("dim_mismatch");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(4, 4)).await.expect("migrate");
+
+        let err = store.append(
+            None, vec!["X".into(), "Y".into()], vec![0, 0],
+            vec!["a".into(), "b".into()],
+            vec![vec![1.0, 2.0, 3.0, 4.0], vec![1.0, 2.0]],
+        ).await;
+        assert!(err.is_err(), "dim mismatch must error");
+        let msg = err.unwrap_err();
+        assert!(msg.contains("dim") || msg.contains("expected"),
+            "error must mention the dimension problem; got: {msg}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// Search round-trip: query the exact vector for one row, top-1
+    /// must be that row. Verifies the search path works on small
+    /// datasets where IVF training would normally be skipped.
+    #[tokio::test]
+    async fn search_returns_nearest() {
+        let path = temp_path("search");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(8, 4)).await.expect("migrate");
+
+        let target: Vec<f32> = (0..4).map(|j| (5 * 4 + j) as f32 * 0.01).collect();
+        let hits = store.search(&target, 3, None, None).await.expect("search");
+        assert!(!hits.is_empty(), "search must return at least 1 hit");
+        assert_eq!(hits[0].doc_id, "DOC-0005",
+            "exact-vector match should be top-1; got {hits:?}");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+
+    /// stats() summarizes the dataset state in one call. Locks the
+    /// field shape so downstream consumers don't break on a rename.
+    #[tokio::test]
+    async fn stats_reports_post_migrate_state() {
+        let path = temp_path("stats");
+        let store = LanceVectorStore::new(path.clone());
+        store.migrate_from_parquet_bytes(&synth_parquet_bytes(5, 4)).await.expect("migrate");
+
+        let s = store.stats().await.expect("stats");
+        assert_eq!(s.rows, 5);
+        assert!(s.disk_bytes > 0);
+        assert!(!s.has_vector_index, "no vector index built yet");
+
+        let _ = std::fs::remove_dir_all(&path);
+    }
+}