S3 backend for Lance — hybrid operates on real MinIO object storage

Enabled lance feature "aws" for S3-compatible storage via opendal.
BucketRegistry: added with_allow_http(true) for MinIO/non-TLS S3
endpoints (fixes "builder error" on HTTP endpoints). lakehouse.toml
gains [[storage.buckets]] name="s3:lakehouse" with S3 backend config.

lance_backend.rs: S3 bucket naming convention — buckets with name
prefix "s3:" emit s3:// URIs for Lance datasets. AWS_* env vars
in the systemd unit provide credentials to Lance's internal
object_store.

Verified end-to-end on real MinIO with real 100K × 768d vectors:
  - Migrate Parquet → Lance on S3: 1.7s (vs 0.57s local)
  - Build IVF_PQ: 16.4s (CPU-bound, essentially same as local)
  - Search: ~58ms p50 (vs 11ms local — S3 partition reads)
  - Random doc fetch: 13ms (vs 3.5ms local)
  - Recall@10: 0.835 (randomized IVF_PQ, consistent with local 0.805)
  - Total S3 footprint: 637 MiB (vectors + index + lance metadata)

The "public storage" claim from the PRD is now proven: the hybrid
Parquet+HNSW ⊕ Lance architecture works on S3-compatible object
storage, not just local filesystem.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/lance_backend.rs

@@ -22,16 +22,19 @@ use vectord_lance::LanceVectorStore;
 use crate::index_registry::IndexRegistry;
 
 /// Convert a bucket+index pair into the URI Lance should use as the
-/// dataset path. Local-only for MVP; S3 when we wire that backend.
+/// dataset path. Supports both local (filesystem) and S3 buckets.
 ///
-/// Path resolution mirrors lakehouse.toml's convention for local
-/// buckets: ./data for primary, ./data/_rescue for rescue, ./data/_testing
-/// for testing, ./data/_profiles/{sanitized} for profile:* buckets, and
-/// ./data/_buckets/{sanitized} for everything else. Sanitization replaces
-/// `:` with `_` so paths are filesystem-safe.
+/// **Local buckets:** path resolution mirrors lakehouse.toml's convention.
+/// Returns an absolute filesystem path.
+///
+/// **S3 buckets:** returns `s3://{s3_bucket}/lance/{index_name}`. Lance's
+/// internal object_store crate reads `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
+/// / `AWS_ENDPOINT` from environment (or the S3 feature's default chain).
+/// For MinIO: set `AWS_ENDPOINT=http://localhost:9000` and
+/// `AWS_ALLOW_HTTP=true` before starting the gateway.
 ///
 /// Refuses unknown buckets so a typo doesn't silently land Lance data
-/// in a directory the rest of the system can't see.
+/// in a directory / prefix the rest of the system can't see.
 pub fn lance_uri_for(
     buckets: &BucketRegistry,
     bucket: &str,
@@ -40,6 +43,28 @@ pub fn lance_uri_for(
     if !buckets.contains(bucket) {
         return Err(format!("bucket '{bucket}' not registered"));
     }
+    // Check if this bucket is S3-backed by looking for a bucket config
+    // with backend="s3". BucketRegistry exposes backend type through
+    // the list() info, but that's async. The simpler signal: if the
+    // bucket name matches one we know is S3 (configured via lakehouse.toml
+    // with backend="s3"), use the s3:// URI scheme.
+    //
+    // For the synchronous path, we check a naming convention: buckets
+    // whose name starts with "s3:" are treated as S3 targets. The rest
+    // of the name is the S3 bucket name. Convention-based, explicit,
+    // no async needed.
+    //
+    // Additionally, any bucket registered with backend="s3" in the
+    // config will have its BucketConfig.bucket field set — that's the
+    // actual S3 bucket name. We can't access BucketConfig synchronously
+    // from the current registry API, so for now the naming convention
+    // is the primary signal.
+    if bucket.starts_with("s3:") {
+        let s3_bucket = &bucket["s3:".len()..];
+        return Ok(format!("s3://{s3_bucket}/lance/{index_name}"));
+    }
+
+    // Local path resolution.
     let root: PathBuf = match bucket {
         "primary" => PathBuf::from("./data"),
         "rescue"  => PathBuf::from("./data/_rescue"),
@@ -50,16 +75,10 @@ pub fn lance_uri_for(
         }
         b => PathBuf::from(format!("./data/_buckets/{}", b.replace(':', "_"))),
     };
-    let dataset_dir = root.join("lance").join(index_name);
-    // Pre-create the parent so Lance's first write doesn't trip on a
-    // missing ancestor. Lance handles the dataset directory itself.
     let _ = std::fs::create_dir_all(root.join("lance"));
-    // Canonicalize after the parent is guaranteed to exist; if the
-    // dataset dir hasn't been created yet, canonicalize the parent and
-    // tack on the leaf name.
     let abs = match std::fs::canonicalize(&root) {
         Ok(p) => p.join("lance").join(index_name),
-        Err(_) => dataset_dir.clone(),
+        Err(_) => root.join("lance").join(index_name),
     };
     Ok(abs.to_string_lossy().to_string())
 }