Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end: HNSW trial system (Phase 15 horizon item → complete) - vectord: EmbeddingCache, harness (eval sets + brute-force ground truth), TrialJournal, parameterized HnswConfig on build_index_with_config - /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best, /hnsw/evals/{name}/autogen, /hnsw/cache/stats - Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us at 100% recall@10. ec=80 es=30 locked as HnswConfig::default() - Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s, 80/30 = 1.00 recall in 230s Catalog manifest repair - catalogd: resync_from_parquet reads parquet footers to restore row_count and columns on drifted manifests - POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing - All 7 staffing tables recovered to PRD-matching 2,469,278 rows Federation foundation (ADR-017) - shared::secrets: SecretsProvider trait + FileSecretsProvider (reads /etc/lakehouse/secrets.toml, enforces 0600 perms) - storaged::registry::BucketRegistry — multi-bucket resolution with rescue_bucket read fallback and reachability probing - storaged::error_journal — bucket op failures visible in one HTTP call - storaged::append_log — write-once batched append pattern (fixes the RMW anti-pattern llms3.com calls out; errors and trial journals both use it) - /storage/buckets, /storage/errors, /storage/bucket-health, /storage/errors/{flush,compact} - Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with X-Lakehouse-Rescue-Used observability headers on fallback Postgres streaming ingest - ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination into ArrowWriter, lineage redacts password - POST /ingest/db — verified against live knowledge_base.team_runs (586 rows × 13 cols, 6 batches, 196ms end-to-end) PRD realignment (2026-04-16) - Dual use case: staffing analytics + local LLM knowledge substrate - Removed "multi-tenancy (single-owner system)" from non-goals - Added invariants 8-11: indexes hot-swappable, per-reader profiles, trials-as-data, operational failures findable in one HTTP call - New phases 16 (hot-swap generations), 17 (model profiles + dataset bindings), 18 (Lance vs Parquet+sidecar evaluation) - Known ceilings table documents the 5M vector wall and escape hatches - ADR-017 (federation), ADR-018 (append-log pattern) added - EXECUTION_PLAN.md sequences phases B-E with success gates and decision rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 01:50:05 -05:00
parent 84407eeb51
commit dbe00d018f
33 changed files with 3305 additions and 52 deletions
@@ -0,0 +1,322 @@
+# ADR-017: Federated Multi-Bucket Storage
+
+**Status:** Accepted — 2026-04-16
+**Owner:** J
+**Implements:** Phase 15 horizon item "Federated multi-bucket query"
+**Depends on:** ADR-001 (object storage as source of truth), ADR-009 (delta files)
+
+---
+
+## Problem
+
+Today every dataset lives in exactly one object storage backend (`/home/profit/lakehouse/data` on local disk). Three scenarios break that assumption:
+
+1. **Multi-tenant hosting** — Client A's staffing data must stay in Client A's S3 bucket. Client B's in Client B's. The lakehouse is the compute plane, their bucket is the data plane.
+2. **Data residency** — European clients require their data never leaves an EU-region bucket. Same lakehouse instance, different physical storage.
+3. **Cross-dataset analytics** — A user asks "average placement margin across all clients we manage." The query must scan Parquet across N buckets as a single logical table.
+
+None of this is possible with a single `object_store` instance. `DataFusion` *can* register multiple object stores under different URL schemes, but nothing in the current stack surfaces that capability.
+
+## Non-Goals (for the MVP)
+
+- **Not** building cross-bucket JOIN optimization. DataFusion pushes predicates down per-partition already; that's sufficient.
+- **Not** handling bucket-level auth policies (who-can-read-which-bucket). That's Phase 13's job.
+- **Not** supporting heterogeneous backends in a single logical dataset (e.g. half in S3, half local). One dataset = one bucket.
+- **Not** automatic replication across buckets. Each bucket stands alone.
+- **Not** scheduled sync/migration. Manual today.
+
+---
+
+## Design
+
+### Core invariant
+
+Every `ObjectRef` belongs to exactly one named bucket. The catalog is the single authority over which bucket holds which dataset. Queries transparently span buckets by virtue of DataFusion's multi-store capability — the catalog tells the query engine where each file lives.
+
+### Naming
+
+Three classes of bucket:
+
+- **`primary`** — system-wide default. Shared catalog, reference datasets, and vectors that aren't tenant-specific. Always present, always the fallback. Corresponds to today's `./data` directory.
+- **`profile:{user_id}`** — per-user workspace bucket. Each user's personal data, their vector indexes, their workspace state. Provisioned on first use. Pre-loaded ("hot loaded") when the user activates their profile.
+- **`{named}`** — tenant/client buckets. Explicit, configured up front. E.g. `client_a`, `client_eu`.
+
+Conventions:
+- **Bucket URL** = `{bucket_id}://{key}` internally (e.g. `primary://datasets/candidates.parquet`, `profile:daisy://vectors/notes.parquet`)
+- DataFusion sees each bucket as a separate registered object store; `ListingTable` paths include the bucket scheme
+- Profile buckets use the `profile:` prefix as both a namespace marker and a DataFusion URL scheme
+
+### Config shape — `lakehouse.toml`
+
+```toml
+[storage]
+# Backward compat — the existing [storage] block still creates a "primary"
+# bucket if no [[storage.buckets]] entries are defined.
+root = "./data"
+# Where profile buckets are rooted when not otherwise configured.
+# A profile:{user} bucket resolves to {profile_root}/{user}/ unless the user
+# config overrides.
+profile_root = "./data/_profiles"
+
+[[storage.buckets]]
+name = "primary"
+backend = "local"
+root = "./data"
+
+rescue_bucket = "rescue"           # single shared fallback for failed reads (see §Failure mode)
+
+[[storage.buckets]]
+name = "rescue"
+backend = "local"
+root = "./data/_rescue"
+
+[[storage.buckets]]
+name = "client_a"
+backend = "s3"
+bucket = "client-a-lakehouse"
+region = "us-east-1"
+endpoint = "https://s3.amazonaws.com"
+secret_ref = "client_a_aws"        # NOT the literal secret — a handle
+
+[[storage.buckets]]
+name = "client_eu"
+backend = "s3"
+bucket = "client-eu-lakehouse"
+region = "eu-west-1"
+secret_ref = "client_eu_aws"
+```
+
+Rules:
+- Credentials **never appear in this file**. `secret_ref` is a handle resolved through the secrets layer (see §Secrets).
+- If `[[storage.buckets]]` is absent, the existing `[storage]` block creates a single `"primary"` bucket. Zero-config upgrade.
+- Adding a bucket requires a restart (MVP); runtime addition via API is a polish item.
+
+### Secrets
+
+Credentials are never in `lakehouse.toml`. A pluggable `SecretsProvider` trait resolves `secret_ref` handles to credential structs at startup:
+
+```rust
+pub trait SecretsProvider: Send + Sync {
+    async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String>;
+}
+```
+
+MVP ships **one** implementation — `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (or a path given by `LAKEHOUSE_SECRETS` env var):
+
+```toml
+# /etc/lakehouse/secrets.toml — root:root, mode 0600, NEVER in git
+[client_a_aws]
+access_key = "AKIA..."
+secret_key = "wJalrXUt..."
+
+[client_eu_aws]
+access_key = "AKIA..."
+secret_key = "..."
+```
+
+Future providers plug into the same trait without touching core code:
+- `VaultSecretsProvider` — HashiCorp Vault
+- `SopsSecretsProvider` — age/gpg-encrypted files in git
+- `KeyringSecretsProvider` — OS-level keyring
+
+Startup fails fast if a `secret_ref` can't resolve — no silent fallback to anonymous S3.
+
+### Schema changes
+
+**`ObjectRef`** already has a `bucket: String` field — currently it carries the S3 bucket name or `"local"` inconsistently. Repurpose it as the **catalog bucket name**:
+
+```rust
+pub struct ObjectRef {
+    pub bucket: String,        // NOW: catalog bucket name, e.g. "primary" or "client_a"
+    pub key: String,           // object key within that bucket
+    pub size_bytes: u64,
+    pub created_at: DateTime<Utc>,
+}
+```
+
+Migration: a `resync-missing`-style one-shot sets `bucket = "primary"` on every existing ObjectRef whose value is empty or ambiguous.
+
+**`DatasetManifest`** — no new top-level field. `objects: Vec<ObjectRef>` is already where bucket info lives. Per the design invariant (one dataset, one bucket), all ObjectRefs inside a manifest share a bucket, so we can add a convenience accessor `manifest.bucket_id()` → `objects[0].bucket.clone()`.
+
+### Storage layer changes
+
+**`storaged::BucketRegistry`** — new struct, replaces the single `Arc<dyn ObjectStore>` currently threaded through gateway.
+
+```rust
+pub struct BucketRegistry {
+    buckets: HashMap<String, Arc<dyn ObjectStore>>,
+    default: String, // "primary"
+}
+
+impl BucketRegistry {
+    pub fn from_config(cfg: &StorageConfig) -> Result<Self, String>;
+    pub fn get(&self, bucket_id: &str) -> Result<&Arc<dyn ObjectStore>, String>;
+    pub fn default_store(&self) -> &Arc<dyn ObjectStore>;
+    pub fn list_buckets(&self) -> Vec<BucketInfo>;
+}
+```
+
+**`storaged::ops`** gets a bucket parameter on every call:
+```rust
+pub async fn put(registry: &BucketRegistry, bucket: &str, key: &str, data: Bytes) -> Result<(), String>;
+pub async fn get(registry: &BucketRegistry, bucket: &str, key: &str) -> Result<Bytes, String>;
+// etc.
+```
+
+Every existing call site passes `"primary"` initially; specific call sites (ingestd, vectord) learn to route.
+
+### Query layer changes
+
+**`queryd`** — on startup, register every bucket as a separate `object_store` URL with DataFusion:
+
+```rust
+for (name, store) in bucket_registry.iter() {
+    let url = format!("lakehouse-{}://", name).parse::<Url>()?;
+    session_ctx.register_object_store(&url, store.clone());
+}
+```
+
+**`ListingTable`** URLs reference the bucket:
+```rust
+let url = format!("lakehouse-{}://{}", obj.bucket, obj.key);
+```
+
+DataFusion handles cross-bucket scans natively — each partition gets routed to the correct store.
+
+### Gateway API additions
+
+| Endpoint | Purpose |
+|---|---|
+| `GET /storage/buckets` | List all configured buckets + their backend + reachability status |
+| `GET /storage/errors` | Recent bucket operation failures (filtered by bucket/time) |
+| `GET /storage/health` | Summary: which buckets have errored in the last 5 minutes |
+| `POST /ingest/file` + `X-Lakehouse-Bucket: client_a` | Header selects target bucket; absent → `primary` |
+| `POST /profile/{user}/activate` | Pre-load user's profile bucket: embedding caches warm, HNSW indexes rebuilt, workspace state hydrated |
+| `POST /profile/{user}/deactivate` | Evict profile bucket's cached state |
+| `POST /catalog/datasets/by-name/{name}/relocate` | Move a dataset to another bucket (polish) |
+
+**Routing rule for all existing endpoints:**
+1. `X-Lakehouse-Bucket: {name}` header → use that bucket
+2. No header → `primary`
+3. Unknown bucket name → 404
+
+Catalog and query endpoints span all buckets by default — a query sees all buckets the user has access to unless explicitly scoped.
+
+### Profile hot load
+
+When `POST /profile/{user}/activate` fires:
+
+1. Resolve `profile:{user}` bucket — create if missing (first-time activation)
+2. Scan the catalog for all datasets whose `owner == user` or which live in that bucket
+3. For each vector index belonging to the profile: pre-load embeddings into `EmbeddingCache`, pre-build HNSW with the locked-in default config (ec=80 es=30)
+4. Hydrate any saved workspace state (see Phase 8.5) into memory
+5. Return a manifest summary — what's hot, what wasn't found, total memory used
+
+Design contract: `/profile/{user}/activate` should be idempotent. Calling it twice is cheap; the second call is a no-op because the cache is already warm.
+
+### Failure mode — rescue bucket + loud errors
+
+The design goal is **failures stay findable**. When something breaks, you should know within seconds which bucket, which key, which operation, and what happened — without grepping logs.
+
+**Writes**: always go to the target bucket. If the target bucket is unreachable, the request fails hard with 503 and the failure is recorded in the error journal. No queueing, no silent fallback — writes that vanish silently are the worst possible failure mode.
+
+**Reads**:
+- First attempt: target bucket.
+- On failure: fall through to the single configured `rescue_bucket`.
+- If the key exists in rescue, serve it with `X-Lakehouse-Rescue-Used: true` and `X-Lakehouse-Original-Bucket: {target}` response headers so the caller knows fallback happened.
+- If the key isn't in rescue either: 404 with both buckets listed in the error body.
+- Every rescue fallback is appended to the **error journal** (see below).
+
+**Rescue bucket** is a single, well-known bucket configured at the storage level. Typical setup: a periodic sync from `primary` → `rescue` so that when `primary` has an outage, recent data is still reachable. Sync is outside this ADR's scope — operator's job.
+
+**Rest of the system keeps working**: a failed bucket never takes down the gateway. Other buckets remain queryable. Cross-bucket queries scan what they can and surface per-partition failures (which bucket, which key) in the error body.
+
+### Error journal — "find errors ez"
+
+A simple append-only log of every bucket operation that failed or fell through. Stored at `primary://_errors/bucket_errors.jsonl` (yes, primary — because we need it findable even if other buckets are down).
+
+```json
+{"ts":"2026-04-16T10:30:15Z","op":"read","target":"client_a","key":"datasets/candidates.parquet","error":"connection refused","rescued":true,"rescue_key_found":true}
+{"ts":"2026-04-16T10:30:18Z","op":"write","target":"client_a","key":"datasets/new.parquet","error":"connection refused","rescued":false}
+```
+
+Exposed via:
+- `GET /storage/errors?limit=50` — recent errors
+- `GET /storage/errors?bucket=client_a` — filter by bucket
+- `GET /storage/errors?since=2026-04-16T10:00` — filter by time
+- `GET /storage/health` — summary: which buckets have errored in the last 5 minutes
+
+The journal doesn't replace logs (tracing still gets everything), but it gives you a **single authoritative place** to answer the question "has anything been failing?" without tailing systemd journals. If the journal is empty, nothing is broken.
+
+### Vectors are bucket-scoped
+
+Vector indexes live in the same bucket as their source data:
+
+- Data in `primary` → vectors in `primary://vectors/{index_name}.parquet`
+- Data in `profile:daisy` → vectors in `profile:daisy://vectors/{index_name}.parquet`
+- Data in `client_a` → vectors in `client_a://vectors/{index_name}.parquet`
+
+The existing `_hnsw_trials/` and `_hnsw_evals/` prefixes also live per-bucket. The trial journal for a profile's index stays inside that profile's bucket — so tenants don't see each other's tuning history.
+
+**Profile hot load consequence:** activating a profile pre-loads ALL vector indexes in that profile's bucket into `EmbeddingCache` + builds HNSW with the default config. First query after activation is warm.
+
+---
+
+## Implementation phases
+
+### MVP (this is what I'd build next session)
+
+1. **`shared::config`** — extend `StorageConfig` with `buckets: Vec<BucketConfig>` and `profile_root`, with backward-compat fallback from single `[storage]` block
+2. **`shared::secrets`** — `SecretsProvider` trait + `FileSecretsProvider` impl reading `/etc/lakehouse/secrets.toml`
+3. **`storaged::BucketRegistry`** — multi-backend registry; mirrors-of tracking; lazy profile bucket creation
+4. **`storaged::ops`** — add `bucket: &str` param to every call; read path falls back to mirror on unreachable; write path hard-fails
+5. **`gateway`** — `X-Lakehouse-Bucket` header middleware → routing decision per request
+6. **`gateway`** — `GET /storage/buckets` + `POST /profile/{user}/activate|deactivate`
+7. **`catalogd`** — one-shot migration: every ObjectRef without an explicit bucket gets `"primary"`
+8. **`queryd::session`** — register every bucket as a DataFusion ObjectStore under its own URL scheme
+9. **`vectord`** — every vector/trial/eval storage path becomes `{bucket}://vectors/...`; profile activate pre-loads HNSW
+10. **`storaged::error_journal`** — append-only JSONL writer at `primary://_errors/bucket_errors.jsonl`, hooked into every ops call-site
+11. **Test harness** — configure `primary` + `profile:testuser` + a mock `client_a` pointing at a 2nd local dir + `rescue` as a 3rd local dir; verify cross-bucket join, profile activate, rescue fallback, error journal visibility
+
+**Success gate #1 — cross-bucket query:** `SELECT SUM(hours_regular) FROM primary.timesheets t1 JOIN "profile:testuser".timesheets t2 USING (placement_id)` returns a sensible number.
+
+**Success gate #2 — profile hot load:** `POST /profile/daisy/activate` pre-builds HNSW over her personal vector index; next `/search` call against that index is <1ms cold.
+
+**Success gate #3 — rescue fallback + error visibility:** rename `client_a`'s directory to simulate outage. GET for a key returns the copy from `rescue` with `X-Lakehouse-Rescue-Used: true` + `X-Lakehouse-Original-Bucket: client_a` headers. PUT returns 503. `GET /storage/errors` shows both events with full context. `GET /storage/health` flags `client_a` as unhealthy.
+
+### Polish (follow-up)
+
+8. **Bucket health check** — `GET /storage/buckets` shows reachable/unreachable
+9. **Bucket relocation** — `POST /catalog/datasets/{name}/relocate` (copy to new bucket, update manifest, delete from old)
+10. **Runtime bucket addition** — `POST /storage/buckets` adds a bucket without restart (stores in a separate `buckets.toml`)
+11. **S3 streaming optimization** — for very large Parquet files, stream the footer instead of loading the whole blob
+
+---
+
+## Decisions (2026-04-16)
+
+1. **Credentials** — Pluggable `SecretsProvider` trait. MVP ships `FileSecretsProvider` at `/etc/lakehouse/secrets.toml` (root:root, 0600). Future providers (Vault, SOPS, OS keyring) plug in without touching core.
+2. **Primary default + profile bucket model** — `primary` always exists as system-wide fallback. Per-user `profile:{user}` buckets are provisioned on first activate. Named tenant buckets configured explicitly.
+3. **Ingest routing** — `X-Lakehouse-Bucket` header. Absent → `primary`. Unknown → 404.
+4. **Failure** — Writes: hard fail on target, logged to error journal. Reads: fall through to a single shared `rescue_bucket`; response headers `X-Lakehouse-Rescue-Used` + `X-Lakehouse-Original-Bucket` make the fallback visible. Every failure appended to an error journal at `primary://_errors/bucket_errors.jsonl`, queryable via `GET /storage/errors`. Failed buckets never take down unrelated paths.
+5. **Vectors in their bucket** — Vectors, trial journals, and eval sets all live in the same bucket as their source data. Profile activate pre-loads the full vector stack for that profile using the locked-in HNSW default (ec=80 es=30).
+
+---
+
+## Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| DataFusion multi-store predicate pushdown lands on cross-bucket JOINs | **Medium** | Benchmark early. If bad, mark cross-bucket JOINs as a known sharp edge. |
+| S3 credentials accidentally logged | **High** | Never stringify credential fields. Add a test that redacts `secret_key` in every code path. |
+| Existing ObjectRefs have inconsistent `bucket` values (some `"local"`, some empty) | **Low** | One-shot migration normalizes to `"primary"`. |
+| Bucket config drift between restarts | **Medium** | Config-driven means restart = canonical. A runtime API would need careful state reconciliation; defer. |
+
+---
+
+## Decision log
+
+- **1 dataset = 1 bucket** (rather than datasets spanning buckets) — vastly simpler, matches the "tenant isolation" real use case. Revisit only if someone has an actual multi-bucket dataset requirement.
+- **Config-first, runtime-later** — YAML/TOML for the MVP; runtime bucket add/remove is polish. Deployment restart is cheap in this stack.
+- **`bucket: String` repurposed, no new field** — avoids a schema migration on every manifest.
+- **DataFusion's multi-store primitive handles cross-bucket queries** — don't reinvent. We're wiring it up, not building it.
@@ -79,3 +79,13 @@
 **Date:** 2026-03-27
 **Decision:** Each contract/search gets a named workspace with saved queries, shortlists, activity logs, and delta layers. Workspaces have daily/weekly/monthly tiers and support instant zero-copy handoff between agents.
 **Rationale:** Staffing workflows are inherently agent-centric — a recruiter works a contract, builds context, then may need to hand it off. The workspace captures that context in a structured, queryable, transferable format. Without it, handoff means "read the email thread and figure it out."
+
+## ADR-017: Federated multi-bucket storage
+**Date:** 2026-04-16
+**Decision:** Every `ObjectRef` belongs to exactly one named bucket. Three bucket classes: `primary` (system default, always present), `profile:{user}` (per-user/per-model workspace bucket), and named tenant buckets (`client_a`, `client_eu`). A single shared `rescue_bucket` handles read fallback on target failure. Writes hard-fail on unreachable target; no silent fallback. Every bucket op failure lands in an error journal at `primary://_errors/bucket_errors/`, queryable via `/storage/errors`. Credentials never live in `lakehouse.toml` — a pluggable `SecretsProvider` trait resolves opaque `secret_ref` handles.
+**Rationale:** The single-backend assumption breaks when we want: tenant data isolation, data residency (EU bucket), per-profile workspaces for local models. DataFusion already supports multiple registered object stores; catalogd is the cross-bucket metadata authority. Rescue bucket + visible error journal makes operational failures diagnosable in one HTTP call. See `docs/ADR-017-federation.md` for full design + success gates.
+
+## ADR-018: Write-once batched append pattern for journals
+**Date:** 2026-04-16
+**Decision:** All append-only journals (error journal, HNSW trial journal, future audit logs) use the `storaged::append_log::AppendLog` helper. Events accumulate in an in-memory buffer; on threshold or explicit `flush()`, the buffer is written as one new timestamped file (`batch_{epoch_us}.jsonl`). Existing files are never rewritten. `compact()` merges all batches into one with a fresh timestamp, preserving chronological sort order.
+**Rationale:** Object stores have no append primitive. Naive "read-modify-write the whole JSONL file on every event" is O(N²) cumulative work and creates the classic small-file / rewrite-amplification anti-pattern that llms3.com flags as the top lakehouse pitfall. Write-once batching is the LSM-tree idea applied to small JSONL events — bounded write amplification, append-only semantics, optional compaction for read efficiency. The in-memory ring buffer preserves O(1) recent-event reads for the `/storage/errors` and `/hnsw/trials` query endpoints.
@@ -0,0 +1,233 @@
+# Execution Plan — Phases B through E
+
+**Created:** 2026-04-16
+**Status:** Active planning document — update as phases complete or scope shifts
+**Owner:** J
+
+This plan sequences the work J and Claw agreed on during the 2026-04-16 reframe session, after stress-testing the "dual-use substrate" vision and aligning it with llms3.com's architectural patterns.
+
+---
+
+## The four phases, at a glance
+
+| Phase | Work | Prereq | Estimated cost | Risk |
+|---|---|---|---|---|
+| B | Lance pilot on one vector index | None | 1 focused session | Medium — new dep, unfamiliar surface |
+| C | Decoupled embedding refresh pipeline | Benefits from B's outcome | 1 focused session | Low — additive, doesn't break existing |
+| D | AI-safe views | Phase 13 (done) | 1 focused session | Low — builds on existing catalog + tool registry |
+| E | Soft deletes / tombstones | None | 1-2 focused sessions | Medium — touches query path, compaction |
+
+Each "focused session" ≈ 3-4 hours of coding + verification + doc update.
+
+---
+
+## Phase B — Lance pilot (the storage format question)
+
+### Why now
+
+J's LLMS3 knowledge base explicitly positions Lance as `alternative_to` Parquet for vector workloads. We admitted in the 2026-04-16 stress test that our Parquet-portability argument for vectors is weaker than advertised — our vector Parquet blobs aren't readable by DuckDB/Polars anyway. Lance could unlock: random-row access, disk-resident indexes, time-travel, better compression. Or it could disappoint and we lock in Parquet with a written reason why.
+
+**Commit to one answer backed by measurements.** No ambiguity after this phase.
+
+### Scope
+
+1. Add `lance` crate as a dep in `vectord`, behind a `feature = "lance"` flag initially so we can build without forcing users to install it
+2. New module `vectord::lance_store` — mirror of `vectord::store` but against Lance format
+3. New endpoint `POST /vectors/lance/index` — build a Lance index for a named source (parallel to the existing Parquet path)
+4. Benchmark script that runs against `resumes_100k_v2` (the existing 100K reference)
+
+### Measured dimensions
+
+All benchmarks against `resumes_100k_v2` (100K × 768d), cold start:
+
+| Metric | Parquet baseline | Lance | Threshold to migrate |
+|---|---|---|---|
+| Cold load from disk | ~2.8s (measured) | ? | ≥2× faster |
+| Search latency p50 | 873us (ec=80 es=30) | ? | Within 50% |
+| Disk size | 330MB | ? | Comparable or better |
+| Single-row random access | N/A (full scan) | ? | <10ms |
+| Append 10K new rows | Full rewrite (~3s) | ? | Incremental <500ms |
+
+### Decision rules
+
+- **Lance wins cold-load by ≥2× AND matches search latency:** migrate vector storage to Lance. Dataset tables stay Parquet. Update ADR-008 → ADR-019.
+- **Lance within 50% across board:** stay Parquet. Document ceiling honestly in PRD (already done). Lance revisit when we have a problem Parquet can't solve.
+- **Lance loses:** close the door, don't revisit without new evidence. Write ADR-019 as "Lance evaluated, rejected, here's why with numbers."
+
+### Success gate
+
+Benchmark output table posted to `docs/ADR-019-vector-storage.md` with measured numbers in each cell of the table above. Decision rule applied mechanically. No "let's defer the call."
+
+### Rollback
+
+The `feature = "lance"` flag means if the pilot goes badly, `cargo build` without the flag is unchanged. No production migration happens until ADR-019 commits to the change. Safe experiment.
+
+---
+
+## Phase C — Decoupled embedding refresh
+
+### Why now
+
+llms3.com's lakehouse architecture explicitly separates "transactional data mutations" from "asynchronous vector refresh cycles." Today we couple them — an ingest writes rows AND embeddings in one flow. That means:
+
+- Adding 1K rows to a 100K-row dataset forces re-embedding of ALL rows (or nothing)
+- No notion of "embeddings are stale, schedule a refresh tonight"
+- The embedding cost (Ollama-bound, the bottleneck) is synchronous with ingest
+
+### Scope
+
+1. Add fields to `DatasetManifest`:
+   - `last_embedded_at: Option<DateTime>`
+   - `embedding_stale_since: Option<DateTime>` (set when data written but embeddings not refreshed)
+   - `embedding_refresh_policy: RefreshPolicy` — `Manual` | `OnAppend` | `Scheduled(cron)`
+2. Decouple ingest from embed: ingest writes data + marks embeddings stale; embedding runs separately
+3. New endpoint: `POST /vectors/refresh/{dataset}` — diffs existing vectors vs current rows, only embeds new/changed (keyed by `doc_id`)
+4. Background scheduler (or cron trigger) — for `Scheduled` policy, re-runs refresh per schedule
+5. `GET /vectors/stale` — lists datasets with stale embeddings and how stale
+
+### Measured success
+
+- Ingest a 1K-row append to `kb_team_runs` (currently 586 rows, Postgres-sourced).
+- Catalog shows `embedding_stale_since = now`.
+- `POST /vectors/refresh/kb_team_runs` embeds only the 1K new rows, not all 1586.
+- Result: new rows searchable, old embeddings unchanged, total Ollama time ~5s instead of ~30s.
+
+### Dependencies on Phase B
+
+If Lance wins Phase B, this is dramatically easier — Lance supports native append. If we stay Parquet, we need a "vectors delta" Parquet file that merges at read time (same pattern as Phase 8's data delta files). ~100 extra LOC if we stay Parquet.
+
+### Rollback
+
+The `refresh_policy` field defaults to `Manual` for all existing datasets, so no behavior change for anything already in the system. Opt-in per dataset.
+
+---
+
+## Phase D — AI-safe views
+
+### Why now
+
+llms3.com's framing: "AI-safe views enforcing row/column security + PII tokenization before model exposure." Phase 13 gave us role-based column masking at query time. That's per-query enforcement. "Views" means pre-materialized: create `candidates_safe` once, bind model X to that view, model X can never accidentally see raw `candidates`.
+
+This is also the precondition for Phase 17 (model profiles) to be meaningfully safe. "Bind model to dataset" isn't enough — needs to be "bind model to a safe view of the dataset."
+
+### Scope
+
+1. New catalog entity: `AiView` with fields `name`, `base_dataset`, `columns[]` (whitelist), `row_filter` (optional SQL WHERE clause), `column_redactions[]` (PII tokenization rules)
+2. Persistence: `_catalog/views/{name}.json` alongside manifests
+3. Query-rewrite layer: when a query references `candidates_safe`, DataFusion sees an equivalent `SELECT (whitelisted cols) FROM candidates WHERE (row_filter)` — with redactions applied as expressions
+4. Endpoint: `POST /catalog/views` to create, `GET /catalog/views` to list, `GET /catalog/views/{name}/preview` to see what it looks like
+5. Tool registry integration: tools can bind to an AiView instead of a raw table; agent invocations go through the view
+
+### Measured success
+
+- Create view `candidates_safe` = `SELECT candidate_id, skills, city FROM candidates WHERE status != 'blocked'`.
+- Agent (tool registry) calls `search_candidates` bound to `candidates_safe`.
+- Agent cannot see `email`, `phone`, `ssn`, or `status='blocked'` rows, even if it writes raw SQL.
+- Audit log records agent accessed `candidates_safe`, not `candidates`.
+
+### Dependencies
+
+- Phase 13 already provides the sensitivity classification layer
+- Phase 12 tool registry already exists
+- This phase is the bridge between them for agent access
+
+### Rollback
+
+Views are additive. Dropping the feature = delete view definitions, tool registry falls back to direct table access. No data migration needed.
+
+---
+
+## Phase E — Soft deletes / tombstones
+
+### Why now
+
+GDPR/CCPA compliance for staffing data. Today, `ops::delete` physically deletes a parquet object — fine for whole datasets, useless for "delete one candidate's record." To delete one row we'd have to rewrite the whole `candidates.parquet` which at 100K rows is 10MB of churn per deletion.
+
+llms3.com lists "deletion vectors" as a core lakehouse pattern (Iceberg/Delta/Hudi all implement it). This is the single biggest compliance gap in the current system.
+
+### Scope
+
+1. New sidecar per dataset: `{dataset}_tombstones.parquet` with columns `{row_key, deleted_at, actor, reason}`
+2. Delete API: `POST /catalog/datasets/{name}/tombstone` with `{row_keys[], reason, actor}`
+3. Query-time filter: `queryd` automatically LEFT JOINs tombstones and filters out deleted rows
+4. Compaction integration: Phase 8 compaction reads base + delta + tombstones, writes a clean base without tombstoned rows, clears the tombstone sidecar
+5. Event journal integration (Phase 9): every tombstone emits a journal event with full context
+
+### Measured success
+
+- `POST /catalog/datasets/candidates/tombstone` with `{row_keys: ["CAND-123"], reason: "GDPR request", actor: "legal@company"}`
+- `SELECT COUNT(*) FROM candidates` drops by 1 immediately
+- `SELECT * FROM candidates WHERE candidate_id = 'CAND-123'` returns empty
+- `GET /journal/history/CAND-123` shows the tombstone event
+- After scheduled compaction, the tombstone is materialized — `candidates.parquet` no longer contains CAND-123, tombstone sidecar is emptied for that row key
+
+### Dependencies
+
+- Phase 8 delta/merge-on-read pattern (done) — tombstones are a third layer at read time
+- Phase 9 event journal (done) — tombstones emit journal events
+
+### Rollback
+
+If query rewrite becomes too complex, fallback: tombstones stored but applied only during compaction (not at query time). Queries return deleted rows until compaction runs. Less useful but safer.
+
+---
+
+## Cross-phase concerns
+
+### Phases that need federation layer 2 (task #5)
+
+Every phase above assumes the federation foundation (shipped 2026-04-16) but NOT federation layer 2 (cross-bucket SQL, profile activation, `X-Lakehouse-Bucket` header).
+
+**Implication:** Phases B-E can proceed on the `primary` bucket without blocking on federation layer 2. Federation layer 2 becomes valuable when we want multi-profile model scoping (Phase 17). Sequence:
+
+```
+A (done) → B → C (+D in parallel) → federation layer 2 → Phase 16 → Phase 17 → E
+```
+
+### Phases that need federation layer 2 FIRST
+
+None of B/C/D/E strictly need it. Phase 16 (hot-swap) benefits from it. Phase 17 (model profiles) depends on it heavily.
+
+### What NOT to build in B-E
+
+- Distributed query — wait for a real scale problem
+- Replacement of DataFusion — working fine, stay put
+- Iceberg/Delta Lake migration — explicitly out of scope per ADR-009
+- Live streaming / CDC — explicit non-goal
+
+### Definition of done for each phase
+
+Each phase completes when:
+1. Code shipped and building clean
+2. Success gate measurably passed
+3. Relevant ADR added to `docs/DECISIONS.md` (or updated)
+4. `docs/PHASES.md` checkbox flipped with measurement data
+5. PRD invariants checked — if a new invariant emerged, add it
+6. One regression test in the crate or HTTP integration test
+
+---
+
+## Session plan (what to do in what order)
+
+### Next session
+1. **Phase B — Lance pilot.** Single session. Answers the biggest open architectural question.
+2. Based on outcome, **write ADR-019** with the decision + data.
+3. **Update PHASES.md** with Phase 18 status (Lance evaluated).
+
+### Session after
+4. **Phase C — Decoupled embedding refresh.** Implementation shaped by B's outcome (append is easy on Lance, requires delta logic on Parquet).
+
+### Session after that
+5. **Federation layer 2** OR **Phase D (AI-safe views)** — J decides based on priority. Federation layer 2 unlocks model profiles (Phase 17); AI-safe views is standalone value.
+
+### Final session for this track
+6. **Phase E — Soft deletes.** The compliance-driven phase. Fits cleanly after everything else because it touches the query path and wants to be built after query optimizations stabilize.
+
+### Milestone checkpoint
+After Phase E, stop and reassess. We'll have:
+- Lance decision made and committed
+- Decoupled embedding pipeline
+- AI-safe view enforcement
+- Soft delete semantics
+
+That's a substantial capability increase. Plausible "pause, write a retrospective, decide on Phase 16/17" moment.
@@ -121,9 +121,35 @@
 - [x] AI migration prompt builder for complex cases
 - [x] 5 unit tests

-## Phase 15+: Horizon ⬜
- [ ] HNSW vector index (100K search: 4.5s → <50ms)
- [ ] Federated multi-bucket query
+## Phase 15+: Horizon
+- [x] HNSW vector index with iteration-friendly trial system (2026-04-16)
+  - `HnswStore.build_index_with_config` — parameterized ef_construction, ef_search, seed
+  - `EmbeddingCache` — pins 100K vectors in memory, shared across trials
+  - `harness::EvalSet` — named query sets with brute-force ground truth
+  - `TrialJournal` — append-only JSONL at `_hnsw_trials/{index}.jsonl`
+  - Endpoints: `/vectors/hnsw/trial`, `/hnsw/trials/{idx}`, `/hnsw/trials/{idx}/best?metric={recall|latency|pareto}`, `/hnsw/evals`, `/hnsw/evals/{name}/autogen`, `/hnsw/cache/stats`
+  - Measured on 100K resumes: **brute-force 44-54ms → HNSW 509us-1830us**, recall 0.92-1.00 depending on `ef_construction`. Sweet spot: ec=80 es=30 → p50=873us recall=1.00 — locked in as `HnswConfig::default()`
+- [x] Catalog manifest repair — `POST /catalog/resync-missing` restores row_count and columns from parquet footers (2026-04-16). All 7 staffing tables recovered to PRD-matching 2.47M rows.
+- [~] Federated multi-bucket query — **foundation complete 2026-04-16**, see ADR-017
+  - [x] `StorageConfig.buckets` + `rescue_bucket` + `profile_root` config shape
+  - [x] `SecretsProvider` trait + `FileSecretsProvider` (reads /etc/lakehouse/secrets.toml, checks 0600 perms)
+  - [x] `storaged::BucketRegistry` — multi-backend, rescue-aware, reachability probes
+  - [x] `storaged::error_journal::ErrorJournal` — append-only JSONL at `primary://_errors/bucket_errors.jsonl`
+  - [x] Endpoints: `GET /storage/buckets`, `GET /storage/errors`, `GET /storage/bucket-health`
+  - [x] Bucket-aware I/O: `PUT/GET /storage/buckets/{bucket}/objects/{*key}` with rescue fallback + `X-Lakehouse-Rescue-Used` observability headers
+  - [x] Backward compat: empty `[[storage.buckets]]` synthesizes a `primary` from legacy `root`
+  - [x] Three-bucket test (primary + rescue + testing) verified: normal reads, rescue fallback with headers, hard-fail missing, write to unknown bucket 503, error journal + health summary
+  - [ ] `X-Lakehouse-Bucket` header middleware on ingest/query/catalog endpoints
+  - [ ] Catalog migration: set `bucket = "primary"` on every legacy ObjectRef
+  - [ ] `queryd` registers every bucket with DataFusion for cross-bucket SQL
+  - [ ] Profile hot-load endpoints: `POST /profile/{user}/activate|deactivate`
+  - [ ] `vectord` bucket-scoped paths (trial journals, eval sets per-bucket)
+- [x] Database connector ingest (Postgres first) — 2026-04-16
+  - `pg_stream::stream_table_to_parquet` — ORDER BY + LIMIT/OFFSET pagination, configurable batch_size
+  - `parse_dsn` — postgresql:// and postgres:// URL scheme, user/password/host/port/db
+  - `POST /ingest/db` endpoint: `{dsn, table, dataset_name?, batch_size?, order_by?, limit?}` → streams to Parquet, registers in catalog with PII detection + redacted-password lineage
+  - Existing `POST /ingest/postgres/import` (structured config) preserved alongside
+  - 4 DSN-parser unit tests + live end-to-end test against `knowledge_base.team_runs` (586 rows, 13 cols, 6 batches, 196ms)
 - [ ] Database connector ingest (Postgres/MySQL)
 - [ ] PDF OCR (Tesseract)
 - [ ] Scheduled ingest (cron)
@@ -133,3 +159,4 @@
 ---

 **30 unit tests | 11 crates | 16 ADRs | 2.47M rows | 100K vectors | All built 2026-03-27**
+**HNSW trial system: 2026-04-16**
@@ -1,22 +1,41 @@
-# PRD: Lakehouse — Rust-First Object Storage System
+# PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores

-**Status:** Active — Phase 0-5 complete, entering production path
+**Status:** Active — Phase 0-14 complete; federation foundation + HNSW trial system shipped 2026-04-16; entering Phase 16 (hot-swap + model profiles)
 **Created:** 2026-03-27
+**Last reframed:** 2026-04-16 — from "staffing analytics platform" to "dual-use knowledge substrate" (see §Problem below)
 **Owner:** J

 ---

 ## Problem

+### Use case 1 — Staffing analytics (reference implementation)
+
 Legacy data systems silo information across CRMs, databases, spreadsheets, and file shares. Querying across them requires manual ETL, pre-defined schemas, and expensive database licenses. When AI enters the picture, these systems can't handle the dual requirement of fast analytical queries AND semantic retrieval over unstructured text.

 A staffing company (our reference case) has candidate records in an ATS, client data in a CRM, timesheets in billing software, call logs from a phone system, and email records from Exchange. Answering "find every Java developer in Chicago who was called 5+ times but never placed" requires querying across all of them — and no single system can do it.

-We need a system where:
- Any data source (CSV, DB export, PDF, JSON) can be ingested without pre-defined schemas
+### Use case 2 — Local AI knowledge substrate (the second half)
+
+Local LLM workloads need a substrate for ingesting, indexing, and retrieving large knowledge corpora. Each running model (or agent) has its own context — documents it cares about, a vector index tuned to its domain, a scoped view of the catalog. That infrastructure is architecturally identical to the staffing problem: ingest messy data, index it, query it, hand it to an AI. Building one substrate that serves both prevents fragmentation.
+
+Concretely this means a running Ollama model like `qwen2.5:7b` or `claude-code-local` should be able to:
+- Bind to a named set of datasets
+- Get a scoped vector index pre-warmed for its domain
+- Issue searches that only see its bound data
+- Have its trial/tuning history isolated from other models
+- Swap between knowledge generations (today's, yesterday's) without rebuild
+
+The same infrastructure that lets a recruiter query 2.47M rows of staffing data also lets a local 7B model answer questions grounded in a 500K-chunk documentation corpus. Same substrate, different tenant.
+
+### Shared requirements
+
+- Any data source (CSV, DB export, PDF, JSON, Postgres table) can be ingested without pre-defined schemas
 - Structured data is queryable via SQL at scale (millions of rows, sub-second)
- Unstructured data is searchable via AI embeddings (semantic retrieval)
- An LLM can answer natural language questions against all of it
+- Unstructured data is searchable via AI embeddings with per-profile indexes
+- An LLM can answer natural language questions against scoped data
+- Indexes can be hot-swapped between generations without rebuild downtime
+- Trials are first-class data — the system remembers how it was tuned
 - Everything runs locally — no cloud APIs, total data privacy
 - The system is rebuildable from repository + object storage alone

@@ -51,15 +70,22 @@ No new frameworks without documented ADR.

 | Service | Responsibility |
 |---|---|
-| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits |
-| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests |
-| **storaged** | Object I/O — read/write/list/delete via `object_store` |
-| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache |
-| **ingestd** | *NEW* — Ingest pipeline: CSV/JSON/DB → normalize → Parquet → catalog |
-| **vectord** | *NEW* — Embedding store + vector index: chunk → embed → index → search |
+| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits, `X-Lakehouse-Bucket` header routing |
+| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests, per-dataset resync from parquet footers |
+| **storaged** | Object I/O — `BucketRegistry` (multi-backend), rescue fallback, error journal, append-log batching pattern |
+| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache, delta merge-on-read |
+| **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog |
+| **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) |
+| **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal |
 | **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
 | **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
-| **shared** | Types, errors, Arrow helpers, config, protobuf definitions |
+| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
+
+**Federation building blocks** (shipped 2026-04-16):
+- `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced)
+- `storaged::registry::BucketRegistry` — multi-bucket resolution with `rescue_bucket` read fallback
+- `storaged::append_log::AppendLog` — write-once batched append pattern (no RMW, no small-file problem)
+- `storaged::error_journal::ErrorJournal` — bucket operation failure log at `primary://_errors/bucket_errors/batch_*.jsonl`

 ### Data Flow

@@ -92,10 +118,14 @@ User question → gateway
 1. Object storage = source of truth for all data
 2. catalogd = sole metadata authority
 3. No raw data in catalog — only pointers
-4. vectord stores embeddings AS Parquet (portable, not a proprietary format)
+4. vectord stores embeddings AS Parquet (portable, not a proprietary format) — see ADR-018 for the Parquet-vs-Lance trade review
 5. ingestd is idempotent — re-ingesting the same file is a no-op
 6. Hot cache is a performance layer, not a source of truth — eviction is safe
 7. All services modular and independently replaceable
+8. **Indexes are hot-swappable.** A new index generation can be built in the background while the current one serves queries. Promotion is atomic (pointer swap). Rollback to a prior generation is always possible. (Phase 16)
+9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17)
+10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern.
+11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?"

 ---

@@ -262,19 +292,90 @@ Per-contract overlays with daily/weekly/monthly tiers and instant handoff.
 | 14.3 | Migration preview: show how old data maps to new schema before applying | Human approves before data transforms |
 | 14.4 | Versioned schemas in catalog: v1, v2, v3 coexist | Queries specify version or use latest |

-### Phase 15+: Horizon
+### Phase 15: Infrastructure horizon items
+
+- [x] HNSW vector index with trial system (shipped 2026-04-16)
+- [x] Federation foundation — ADR-017 (shipped 2026-04-16)
+- [x] Database connector ingest — Postgres batch with streaming (shipped 2026-04-16)
+- [ ] Federation layer 2 — X-Lakehouse-Bucket middleware, catalog migration, cross-bucket SQL in queryd
+- [ ] PDF OCR for scanned documents (Tesseract integration)
+- [ ] Scheduled ingest (cron-based file watching, S3 event triggers)
+- [ ] Multi-node query distribution (DataFusion supports this architecturally)
+
+### Phase 16: Hot-Swap Index Generations
+
+Make indexes upgrade-in-place without dropping queries.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 16.1 | "Active generation" pointer per logical index name | `/vectors/search` routes to current champion automatically |
+| 16.2 | Background trial runner: watches trial journal, proposes configs (random search / Bayesian), fires `/hnsw/trial` | Agent autonomously tunes without human POSTing each config |
+| 16.3 | Promotion endpoint: `POST /hnsw/promote/{index}/{trial_id}` atomically swaps active pointer | Next search hits new config, zero downtime |
+| 16.4 | Rollback: `POST /hnsw/rollback/{index}` reverts to previous generation | Bad promotion recoverable in milliseconds |
+| 16.5 | Dataset-append triggers: when `POST /ingest/file` writes to a dataset with attached vector indexes, schedule automatic re-trial (not full rebuild) | New docs get embedded + indexed without manual intervention |
+
+**Gate:** Run the trial agent for 10 minutes against `resumes_100k_v2` with a fresh eval set. It explores the `ef_construction × ef_search` space, promotes the Pareto winner, continues running. Zero human clicks. All trials and promotions appear in `/hnsw/trials/resumes_100k_v2`.
+
+**Risk:** Agent loops into a bad region (e.g. always proposes ef_construction=1). Mitigation: a hardcoded config space constraint + minimum-quality gate (don't promote anything with recall <0.9).
+
+### Phase 17: Model Profiles + Dataset Bindings
+
+Make "different models see different data" real instead of a config string.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 17.1 | `ModelProfile` manifest: id, ollama_name, bucket, bound_datasets[], hnsw_config, embed_model | `GET /models` lists profiles; `POST /models` creates one |
+| 17.2 | Profile activation endpoint: `POST /profile/{id}/activate` — warms EmbeddingCache for bound indexes, builds HNSW with profile's config | Next search against bound indexes is <1ms cold |
+| 17.3 | Model-scoped search: `POST /search?model=X` filters to bound datasets only | Model A can't see Model B's datasets unless explicitly shared |
+| 17.4 | VRAM-aware activation: only one (or small N) model loaded at a time on 16GB A4000 | Activating model B unloads model A via Ollama's keep_alive=0 |
+| 17.5 | Audit: every tool invocation by a model is logged with model identity | `GET /models/{id}/audit` shows exactly what each model touched |
+
+**Gate:** Two model profiles defined: `staffing-recruiter` (bound to candidates/placements/timesheets) and `docs-assistant` (bound to a documentation corpus). Activate staffing-recruiter, search for candidates — works. Switch to docs-assistant, same search — returns zero from staffing (not bound) but finds docs. VRAM shows only one embedding model loaded at a time.
+
+**VRAM reality:** 16GB A4000 realistically holds 1-2 loaded models concurrently. "Multi-model" in practice means sequential swap between profiles, not parallel serving. The profile abstraction makes this swap clean.
+
+### Phase 18: Storage format decision (Lance evaluation)
+
+The question raised 2026-04-16 after J's LLMS3 knowledge base identified Lance as `alternative_to` Parquet for vector workloads. Current stack: Parquet with binary-blob vector columns + in-RAM HNSW sidecar. Evaluated against: Lance native vector format with disk-resident indexes.
+
+| Step | Deliverable | Decision criteria |
+|---|---|---|
+| 18.1 | Parallel Lance-backed vector index for `resumes_100k_v2` behind feature flag | Both implementations coexist, benchmarkable |
+| 18.2 | Head-to-head benchmark: cold-load, search latency, disk size, append cost | See criteria below |
+| 18.3 | ADR-019 documenting the decision with measured data | Commit or reject with evidence |
+
+**Decision rules:**
+- Lance wins on cold-load by ≥2× AND matches search latency → migrate vector layer to Lance. Dataset Parquet stays.
+- Lance is within 50% of current → stay on current stack, document ceiling explicitly.
+- Lance loses → close the door, move on.
+
+### Phase 19+: Further horizon

- Federated multi-bucket query (client A's S3 + client B's S3 + yours)
- Database connector ingest (PostgreSQL, MySQL, MSSQL → Parquet via CDC)
 - PDF OCR for scanned documents (Tesseract integration)
- Scheduled ingest (cron-based file watching, S3 event triggers)
 - Specialized fine-tuned models per domain (staffing matcher, resume parser)
- Multi-node query distribution (DataFusion supports this architecturally)
 - Video/audio transcript ingest + multimodal embeddings
+- True distributed query (DataFusion multi-node) — only if single-machine ceilings bite

 ---

-## Reference Dataset: Staffing Company
+## Known ceilings (honest)
+
+The current stack has measurable limits. Documenting them so future decisions aren't based on wishful thinking.
+
+| Dimension | Current ceiling | Breaks at | Escape hatch |
+|---|---|---|---|
+| Vector count per index | ~5M vectors on 128GB RAM | 10M+ (serious web crawl) | Phase 18 Lance migration OR mmap'd embeddings |
+| Concurrent active indexes | ~50-100 at 100K vectors each | 10M×50 configurations | Lance disk-resident + per-profile activation |
+| Rows per dataset | 2.47M proven, probably 100M+ fine | Approaches DataFusion memory limits | DataFusion predicate pushdown + partition pruning (existing) |
+| Concurrent loaded models | 1-2 on 16GB VRAM (A4000) | 3+ models simultaneous | Not our problem — architectural, driven by Ollama |
+| Trial journal growth per index | Thousands of trials, batched JSONL | High-frequency auto-tuning agent | Compaction via `/hnsw/trials/{idx}/compact` |
+| Error journal growth | Bounded by ring buffer (2000 events in-memory) + batched JSONL on disk | Continuous failure scenarios | Compaction + retention policy (TODO) |
+
+---
+
+## Reference Workloads
+
+### Workload 1: Staffing Company

 Scale-tested on 128GB RAM server:

@@ -315,6 +416,27 @@ Scale-tested on 128GB RAM server:
 - Instant handoff between agents — zero data copy
 - Full activity timeline preserved across handoffs

+### Workload 2: Local LLM Knowledge Base
+
+The second use case this substrate is built for. Reference corpus: the running `knowledge_base` Postgres database (586 team runs, response cache history, pipeline runs, threat intel) + LLMS3.com published corpus (~243 enriched documents).
+
+Target scale on same 128GB server:
+- Documents: 10K-100K per model profile
+- Chunks after chunking: 500K-5M per profile
+- Embedding dimensions: 768 (nomic-embed-text)
+- Query latency: <100ms semantic search, <3s end-to-end RAG including LLM generation
+- Concurrent model profiles: 2-5 configured, 1-2 active at a time (VRAM-bound)
+
+Measured to date (Phase 7 + Phase 16 prep):
+- 100K candidate-resume chunks embedded in 10 min via Ollama nomic-embed-text
+- HNSW search at 100% recall, ~1ms p50 on 100K vectors (ec=80 es=30 locked as default)
+- Trial journal instrumented and working for parameter tuning
+
+Gaps still to close for this workload:
+- Model profiles (Phase 17) — today, "model" is a string, not a first-class entity
+- Hot-swap generations (Phase 16) — today, rebuild = downtime
+- Scale past 5M vectors — needs Phase 18 Lance evaluation to decide path
+
 ---

 ## Available Local Models
@@ -331,12 +453,15 @@ Scale-tested on 128GB RAM server:

 ## Non-Goals

- Multi-tenancy (single-owner system)
 - Cloud deployment (local-first, always)
 - Full ACID transactions (single-writer model is sufficient)
- Real-time streaming / CDC (batch ingest is the model)
- Replacing the CRM (this is the analytical layer BEHIND the CRM)
- Custom file formats (Parquet is the format, period)
+- Real-time streaming / CDC (batch ingest is the model; scheduled refresh, not transactional replication)
+- Replacing the CRM (this is the analytical + AI layer BEHIND the CRM)
+- Custom file formats — Parquet for datasets + sidecar indexes for vectors (see ADR-018 for why we stayed Parquet instead of migrating to Lance, and the ceilings that choice implies)
+- Hard multi-tenant isolation (profiles and federation provide soft isolation; this is not a SaaS platform with adversarial tenants — operator is single-trust)
+
+Removed from prior non-goals (2026-04-16):
+- ~~Multi-tenancy (single-owner system)~~ — federation + profile buckets are now first-class; soft multi-tenancy is a design goal. Hard adversarial multi-tenancy (adversarial tenants on shared infrastructure) remains out of scope.

 ---