Federation layer 2: header routing + cross-bucket SQL

Three pieces of the multi-bucket federation made real:

1. Catalog migration (POST /catalog/migrate-buckets)
   - One-shot normalizer for ObjectRef.bucket field
   - Empty -> "primary"; legacy "data"/"local" -> "primary"
   - Idempotent; re-running on canonical state is no-op
   - Ran on existing catalog: 12 refs renamed from "data", 2 already
     "primary", all 14 now canonical

2. X-Lakehouse-Bucket header middleware on ingest
   - resolve_bucket() helper extracts header, returns
     (bucket_name, store) or 404 with valid bucket list
   - ingest_file and ingest_db_stream now route writes per-request
   - Defaults to "primary" when header absent
   - pipeline::ingest_file_to_bucket records the actual bucket on the
     ObjectRef so catalog stays the source of truth for "where does this
     data live"
   - Verified: ingest with X-Lakehouse-Bucket: testing lands in
     data/_testing/, ingest without header lands in data/, bad header
     returns 404 with hint

3. queryd registers every bucket with DataFusion
   - QueryEngine now holds Arc<BucketRegistry> instead of single store
   - build_context iterates all buckets, registers each as a separate
     ObjectStore under URL scheme "lakehouse-{bucket}://"
   - ListingTable URLs include the per-object bucket scheme so
     DataFusion routes scans automatically based on ObjectRef.bucket
   - Profile bucket names like "profile:user" sanitized to
     "lakehouse-profile-user" since URL host segments can't contain ":"
   - Tolerant of duplicate manifest entries (pre-existing
     pipeline::ingest_file behavior creates a fresh dataset id per
     ingest); duplicates skipped with debug log
   - Backward compat: legacy "lakehouse://data/" URL still registered
     pointing at primary

Success gate: cross-bucket CROSS JOIN
  SELECT p.name, p.role, a.species
  FROM people_test p          (bucket: testing)
  CROSS JOIN animals a        (bucket: primary)
  LIMIT 5
returns rows correctly. DataFusion routed each scan to its bucket's
ObjectStore based on the URL scheme.

No regressions: SELECT COUNT(*) FROM candidates still returns 100000
from the primary bucket.

Deferred to Phase 17:
- POST /profile/{user}/activate (HNSW hot-load on profile switch)
- vectord storage paths becoming bucket-scoped (trial journals,
  eval sets per-profile)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/catalogd/src/registry.rs

@@ -9,6 +9,15 @@ use tokio::sync::RwLock;
 use storaged::ops;
 use object_store::ObjectStore;
 
+#[derive(Debug, Clone, Default, serde::Serialize)]
+pub struct MigrateBucketsReport {
+    pub refs_examined: usize,
+    pub refs_renamed: usize,    // legacy "data"/"local" → "primary"
+    pub refs_stamped: usize,    // empty → "primary"
+    pub refs_unchanged: usize,  // already canonical
+    pub manifests_persisted: usize,
+}
+
 /// Partial metadata update — only set fields are applied.
 #[derive(Debug, Clone, Default, serde::Deserialize)]
 pub struct MetadataUpdate {
@@ -295,6 +304,56 @@ impl Registry {
         Ok(())
     }
 
+    /// Federation layer 2: stamp `bucket = "primary"` on every ObjectRef
+    /// whose `bucket` field is empty or matches a legacy value (`"data"`,
+    /// `"local"`). One-shot migration; re-running is a safe no-op once
+    /// every ref is canonical.
+    pub async fn migrate_buckets_to_primary(&self) -> Result<MigrateBucketsReport, String> {
+        let mut datasets = self.datasets.write().await;
+        let mut report = MigrateBucketsReport::default();
+        let mut to_persist: Vec<(DatasetId, DatasetManifest)> = Vec::new();
+
+        for manifest in datasets.values_mut() {
+            let mut changed = false;
+            for obj in manifest.objects.iter_mut() {
+                report.refs_examined += 1;
+                if obj.bucket.is_empty() {
+                    obj.bucket = "primary".to_string();
+                    report.refs_stamped += 1;
+                    changed = true;
+                } else if obj.bucket == "data" || obj.bucket == "local" {
+                    obj.bucket = "primary".to_string();
+                    report.refs_renamed += 1;
+                    changed = true;
+                } else {
+                    report.refs_unchanged += 1;
+                }
+            }
+            if changed {
+                manifest.updated_at = chrono::Utc::now();
+                to_persist.push((manifest.id.clone(), manifest.clone()));
+            }
+        }
+
+        // Persist updated manifests after we've finished mutating the map.
+        for (id, manifest) in to_persist {
+            let key = format!("{MANIFEST_PREFIX}/{}.json", id);
+            let json = serde_json::to_vec_pretty(&manifest).map_err(|e| e.to_string())?;
+            ops::put(&self.store, &key, json.into()).await?;
+            report.manifests_persisted += 1;
+        }
+
+        tracing::info!(
+            "bucket migration: examined {} refs, renamed {}, stamped {}, unchanged {}, persisted {} manifests",
+            report.refs_examined,
+            report.refs_renamed,
+            report.refs_stamped,
+            report.refs_unchanged,
+            report.manifests_persisted,
+        );
+        Ok(report)
+    }
+
     /// List datasets whose `embedding_stale_since` is set — they need a refresh.
     pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
         let datasets = self.datasets.read().await;

crates/catalogd/src/service.rs

@@ -21,6 +21,7 @@ pub fn router(registry: Registry) -> Router {
         .route("/datasets/by-name/{name}/metadata", post(update_metadata))
         .route("/datasets/by-name/{name}/resync", post(resync_dataset))
         .route("/resync-missing", post(resync_all_missing))
+        .route("/migrate-buckets", post(migrate_buckets))
         .with_state(registry)
 }
 
@@ -194,3 +195,13 @@ async fn resync_all_missing(State(registry): State<Registry>) -> impl IntoRespon
         failed: err.into_iter().map(|(name, error)| ResyncErr { name, error }).collect(),
     })
 }
+
+/// Federation layer 2 one-shot: normalize every ObjectRef.bucket field
+/// to the canonical "primary" value. Idempotent — re-running once
+/// everything is canonical is a safe no-op.
+async fn migrate_buckets(State(registry): State<Registry>) -> impl IntoResponse {
+    match registry.migrate_buckets_to_primary().await {
+        Ok(report) => Ok(Json(report)),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}