Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end:

HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
  TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
  /hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
  at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
  80/30 = 1.00 recall in 230s

Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
  and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows

Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
  /etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
  rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
  anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
  /storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
  X-Lakehouse-Rescue-Used observability headers on fallback

Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
  into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
  (586 rows × 13 cols, 6 batches, 196ms end-to-end)

PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
  trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
  bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
  decision rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/catalogd/src/service.rs

@@ -19,6 +19,8 @@ pub fn router(registry: Registry) -> Router {
         .route("/datasets/{id}", get(get_dataset))
         .route("/datasets/by-name/{name}", get(get_dataset_by_name))
         .route("/datasets/by-name/{name}/metadata", post(update_metadata))
+        .route("/datasets/by-name/{name}/resync", post(resync_dataset))
+        .route("/resync-missing", post(resync_all_missing))
         .with_state(registry)
 }
 
@@ -152,3 +154,43 @@ async fn update_metadata(
         Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
     }
 }
+
+/// Re-read parquet footers for a single dataset and repopulate row_count
+/// and columns from reality. Useful for repairing manifests whose metadata
+/// was lost or never backfilled.
+async fn resync_dataset(
+    State(registry): State<Registry>,
+    Path(name): Path<String>,
+) -> impl IntoResponse {
+    match registry.resync_from_parquet(&name).await {
+        Ok(manifest) => Ok(Json(DatasetResponse::from(&manifest))),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}
+
+#[derive(Serialize)]
+struct ResyncAllResponse {
+    succeeded: Vec<ResyncOk>,
+    failed: Vec<ResyncErr>,
+}
+
+#[derive(Serialize)]
+struct ResyncOk {
+    name: String,
+    row_count: u64,
+}
+
+#[derive(Serialize)]
+struct ResyncErr {
+    name: String,
+    error: String,
+}
+
+/// Resync every dataset that currently has null row_count or empty columns.
+async fn resync_all_missing(State(registry): State<Registry>) -> impl IntoResponse {
+    let (ok, err) = registry.resync_missing().await;
+    Json(ResyncAllResponse {
+        succeeded: ok.into_iter().map(|(name, row_count)| ResyncOk { name, row_count }).collect(),
+        failed: err.into_iter().map(|(name, error)| ResyncErr { name, error }).collect(),
+    })
+}