Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end:

HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
  TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
  /hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
  at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
  80/30 = 1.00 recall in 230s

Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
  and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows

Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
  /etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
  rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
  anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
  /storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
  X-Lakehouse-Rescue-Used observability headers on fallback

Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
  into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
  (586 rows × 13 cols, 6 batches, 196ms end-to-end)

PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
  trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
  bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
  decision rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs/PHASES.md

@@ -121,9 +121,35 @@
 - [x] AI migration prompt builder for complex cases
 - [x] 5 unit tests
 
-## Phase 15+: Horizon ⬜
-- [ ] HNSW vector index (100K search: 4.5s → <50ms)
-- [ ] Federated multi-bucket query
+## Phase 15+: Horizon
+- [x] HNSW vector index with iteration-friendly trial system (2026-04-16)
+  - `HnswStore.build_index_with_config` — parameterized ef_construction, ef_search, seed
+  - `EmbeddingCache` — pins 100K vectors in memory, shared across trials
+  - `harness::EvalSet` — named query sets with brute-force ground truth
+  - `TrialJournal` — append-only JSONL at `_hnsw_trials/{index}.jsonl`
+  - Endpoints: `/vectors/hnsw/trial`, `/hnsw/trials/{idx}`, `/hnsw/trials/{idx}/best?metric={recall|latency|pareto}`, `/hnsw/evals`, `/hnsw/evals/{name}/autogen`, `/hnsw/cache/stats`
+  - Measured on 100K resumes: **brute-force 44-54ms → HNSW 509us-1830us**, recall 0.92-1.00 depending on `ef_construction`. Sweet spot: ec=80 es=30 → p50=873us recall=1.00 — locked in as `HnswConfig::default()`
+- [x] Catalog manifest repair — `POST /catalog/resync-missing` restores row_count and columns from parquet footers (2026-04-16). All 7 staffing tables recovered to PRD-matching 2.47M rows.
+- [~] Federated multi-bucket query — **foundation complete 2026-04-16**, see ADR-017
+  - [x] `StorageConfig.buckets` + `rescue_bucket` + `profile_root` config shape
+  - [x] `SecretsProvider` trait + `FileSecretsProvider` (reads /etc/lakehouse/secrets.toml, checks 0600 perms)
+  - [x] `storaged::BucketRegistry` — multi-backend, rescue-aware, reachability probes
+  - [x] `storaged::error_journal::ErrorJournal` — append-only JSONL at `primary://_errors/bucket_errors.jsonl`
+  - [x] Endpoints: `GET /storage/buckets`, `GET /storage/errors`, `GET /storage/bucket-health`
+  - [x] Bucket-aware I/O: `PUT/GET /storage/buckets/{bucket}/objects/{*key}` with rescue fallback + `X-Lakehouse-Rescue-Used` observability headers
+  - [x] Backward compat: empty `[[storage.buckets]]` synthesizes a `primary` from legacy `root`
+  - [x] Three-bucket test (primary + rescue + testing) verified: normal reads, rescue fallback with headers, hard-fail missing, write to unknown bucket 503, error journal + health summary
+  - [ ] `X-Lakehouse-Bucket` header middleware on ingest/query/catalog endpoints
+  - [ ] Catalog migration: set `bucket = "primary"` on every legacy ObjectRef
+  - [ ] `queryd` registers every bucket with DataFusion for cross-bucket SQL
+  - [ ] Profile hot-load endpoints: `POST /profile/{user}/activate|deactivate`
+  - [ ] `vectord` bucket-scoped paths (trial journals, eval sets per-bucket)
+- [x] Database connector ingest (Postgres first) — 2026-04-16
+  - `pg_stream::stream_table_to_parquet` — ORDER BY + LIMIT/OFFSET pagination, configurable batch_size
+  - `parse_dsn` — postgresql:// and postgres:// URL scheme, user/password/host/port/db
+  - `POST /ingest/db` endpoint: `{dsn, table, dataset_name?, batch_size?, order_by?, limit?}` → streams to Parquet, registers in catalog with PII detection + redacted-password lineage
+  - Existing `POST /ingest/postgres/import` (structured config) preserved alongside
+  - 4 DSN-parser unit tests + live end-to-end test against `knowledge_base.team_runs` (586 rows, 13 cols, 6 batches, 196ms)
 - [ ] Database connector ingest (Postgres/MySQL)
 - [ ] PDF OCR (Tesseract)
 - [ ] Scheduled ingest (cron)
@@ -133,3 +159,4 @@
 ---
 
 **30 unit tests | 11 crates | 16 ADRs | 2.47M rows | 100K vectors | All built 2026-03-27**
+**HNSW trial system: 2026-04-16**