Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end:

HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
  TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
  /hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
  at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
  80/30 = 1.00 recall in 230s

Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
  and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows

Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
  /etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
  rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
  anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
  /storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
  X-Lakehouse-Rescue-Used observability headers on fallback

Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
  into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
  (586 rows × 13 cols, 6 batches, 196ms end-to-end)

PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
  trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
  bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
  decision rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/gateway/src/main.rs

@@ -24,8 +24,30 @@ async fn main() {
     tracing::info!("config loaded: gateway={}:{}, storage={}",
         config.gateway.host, config.gateway.port, config.storage.root);
 
-    // Storage backend
-    let store = storaged::backend::init_local(&config.storage.root);
+    // Secrets provider
+    let secrets = std::sync::Arc::new(shared::secrets::FileSecretsProvider::from_env());
+    if let Err(e) = secrets.load().await {
+        tracing::warn!("secrets load: {e} — bucket credentials may be missing");
+    }
+
+    // Federation: bucket registry (primary + any [[storage.buckets]] entries + rescue)
+    let bucket_registry = std::sync::Arc::new(
+        storaged::registry::BucketRegistry::from_config(&config.storage, secrets.clone())
+            .await
+            .expect("bucket registry init failed")
+    );
+    let buckets = bucket_registry.list().await;
+    tracing::info!(
+        "bucket registry: {} buckets [{}], rescue={:?}",
+        buckets.len(),
+        buckets.iter().map(|b| format!("{}:{}", b.name, b.backend)).collect::<Vec<_>>().join(", "),
+        bucket_registry.rescue_name(),
+    );
+
+    // Back-compat: existing components that expect a single Arc<ObjectStore>
+    // get the primary bucket's store. As we migrate call sites to be bucket-
+    // aware, they'll call bucket_registry.get(name) directly.
+    let store = bucket_registry.default_store();
 
     // Catalog
     let registry = catalogd::registry::Registry::new(store.clone());
@@ -56,7 +78,8 @@ async fn main() {
     // HTTP router
     let mut app = Router::new()
         .route("/health", get(health))
-        .nest("/storage", storaged::service::router(store.clone()))
+        .nest("/storage", storaged::service::router(store.clone())
+            .merge(storaged::federation_service::router(bucket_registry.clone())))
         .nest("/catalog", catalogd::service::router(registry.clone()))
         .nest("/query", queryd::service::router(engine.clone()))
         .nest("/ai", aibridge::service::router(ai_client.clone()))
@@ -73,6 +96,8 @@ async fn main() {
                 job_tracker: vectord::jobs::JobTracker::new(),
                 index_registry: index_reg,
                 hnsw_store: vectord::hnsw::HnswStore::new(),
+                embedding_cache: vectord::embedding_cache::EmbeddingCache::new(store.clone()),
+                trial_journal: vectord::trial::TrialJournal::new(store.clone()),
             }
         }))
         .nest("/workspaces", queryd::workspace_service::router(workspace_mgr))