checkpoint: matrix-agent-validated (2026-04-25)

Architectural snapshot of the lakehouse codebase at the point where the full matrix-driven agent loop with Mem0 versioning + deletion was validated end-to-end. WHAT THIS REPO IS A clean single-commit snapshot of the lakehouse code. Heavy test data (.parquet datasets, vector indexes) excluded — see REPLICATION.md for regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse. WHAT WAS PROVEN - Vector retrieval across multi-corpora matrix (chicago_permits + entity briefs + sec_tickers + distilled procedural + llm_team runs) - Observer hand-review (cloud + heuristic fallback) gating each candidate - Local-model agent loop (qwen3.5:latest) with tool use + scratchpad - Playbook seal on success → next-iter retrieval surfaces it as preamble - Mem0 versioning + deletion in pathway_memory: * UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical * REVISE: chains versions, parent.superseded_at + superseded_by stamped * RETIRE: marks specific trace retired with reason, excluded from retrieval * HISTORY: walks chain root→tip, cycle-safe KEY DIRECTORIES - crates/vectord/src/pathway_memory.rs — Mem0 ops live here - crates/vectord/src/playbook_memory.rs — original Mem0 reference - tests/agent_test/ — local-model agent harness + PRD + session archives - scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus) - scripts/vectorize_raw_corpus.ts — corpus → vector indexes - scripts/analyze_chicago_contracts.ts — real inference pipeline - scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces Replication: see REPLICATION.md for Debian 13 clean install + cloud-only adaptation (no local Ollama). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 19:43:27 -05:00
commit ac01fffd9a
325 changed files with 82720 additions and 0 deletions
@@ -0,0 +1,322 @@
+# ADR-017: Federated Multi-Bucket Storage
+
+**Status:** Accepted — 2026-04-16
+**Owner:** J
+**Implements:** Phase 15 horizon item "Federated multi-bucket query"
+**Depends on:** ADR-001 (object storage as source of truth), ADR-009 (delta files)
+
+---
+
+## Problem
+
+Today every dataset lives in exactly one object storage backend (`/home/profit/lakehouse/data` on local disk). Three scenarios break that assumption:
+
+1. **Multi-tenant hosting** — Client A's staffing data must stay in Client A's S3 bucket. Client B's in Client B's. The lakehouse is the compute plane, their bucket is the data plane.
+2. **Data residency** — European clients require their data never leaves an EU-region bucket. Same lakehouse instance, different physical storage.
+3. **Cross-dataset analytics** — A user asks "average placement margin across all clients we manage." The query must scan Parquet across N buckets as a single logical table.
+
+None of this is possible with a single `object_store` instance. `DataFusion` *can* register multiple object stores under different URL schemes, but nothing in the current stack surfaces that capability.
+
+## Non-Goals (for the MVP)
+
+- **Not** building cross-bucket JOIN optimization. DataFusion pushes predicates down per-partition already; that's sufficient.
+- **Not** handling bucket-level auth policies (who-can-read-which-bucket). That's Phase 13's job.
+- **Not** supporting heterogeneous backends in a single logical dataset (e.g. half in S3, half local). One dataset = one bucket.
+- **Not** automatic replication across buckets. Each bucket stands alone.
+- **Not** scheduled sync/migration. Manual today.
+
+---
+
+## Design
+
+### Core invariant
+
+Every `ObjectRef` belongs to exactly one named bucket. The catalog is the single authority over which bucket holds which dataset. Queries transparently span buckets by virtue of DataFusion's multi-store capability — the catalog tells the query engine where each file lives.
+
+### Naming
+
+Three classes of bucket:
+
+- **`primary`** — system-wide default. Shared catalog, reference datasets, and vectors that aren't tenant-specific. Always present, always the fallback. Corresponds to today's `./data` directory.
+- **`profile:{user_id}`** — per-user workspace bucket. Each user's personal data, their vector indexes, their workspace state. Provisioned on first use. Pre-loaded ("hot loaded") when the user activates their profile.
+- **`{named}`** — tenant/client buckets. Explicit, configured up front. E.g. `client_a`, `client_eu`.
+
+Conventions:
+- **Bucket URL** = `{bucket_id}://{key}` internally (e.g. `primary://datasets/candidates.parquet`, `profile:daisy://vectors/notes.parquet`)
+- DataFusion sees each bucket as a separate registered object store; `ListingTable` paths include the bucket scheme
+- Profile buckets use the `profile:` prefix as both a namespace marker and a DataFusion URL scheme
+
+### Config shape — `lakehouse.toml`
+
+```toml
+[storage]
+# Backward compat — the existing [storage] block still creates a "primary"
+# bucket if no [[storage.buckets]] entries are defined.
+root = "./data"
+# Where profile buckets are rooted when not otherwise configured.
+# A profile:{user} bucket resolves to {profile_root}/{user}/ unless the user
+# config overrides.
+profile_root = "./data/_profiles"
+
+[[storage.buckets]]
+name = "primary"
+backend = "local"
+root = "./data"
+
+rescue_bucket = "rescue"           # single shared fallback for failed reads (see §Failure mode)
+
+[[storage.buckets]]
+name = "rescue"
+backend = "local"
+root = "./data/_rescue"
+
+[[storage.buckets]]
+name = "client_a"
+backend = "s3"
+bucket = "client-a-lakehouse"
+region = "us-east-1"
+endpoint = "https://s3.amazonaws.com"
+secret_ref = "client_a_aws"        # NOT the literal secret — a handle
+
+[[storage.buckets]]
+name = "client_eu"
+backend = "s3"
+bucket = "client-eu-lakehouse"
+region = "eu-west-1"
+secret_ref = "client_eu_aws"
+```
+
+Rules:
+- Credentials **never appear in this file**. `secret_ref` is a handle resolved through the secrets layer (see §Secrets).
+- If `[[storage.buckets]]` is absent, the existing `[storage]` block creates a single `"primary"` bucket. Zero-config upgrade.
+- Adding a bucket requires a restart (MVP); runtime addition via API is a polish item.
+
+### Secrets
+
+Credentials are never in `lakehouse.toml`. A pluggable `SecretsProvider` trait resolves `secret_ref` handles to credential structs at startup:
+
+```rust
+pub trait SecretsProvider: Send + Sync {
+    async fn resolve(&self, handle: &str) -> Result<BucketCredentials, String>;
+}
+```
+
+MVP ships **one** implementation — `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (or a path given by `LAKEHOUSE_SECRETS` env var):
+
+```toml
+# /etc/lakehouse/secrets.toml — root:root, mode 0600, NEVER in git
+[client_a_aws]
+access_key = "AKIA..."
+secret_key = "wJalrXUt..."
+
+[client_eu_aws]
+access_key = "AKIA..."
+secret_key = "..."
+```
+
+Future providers plug into the same trait without touching core code:
+- `VaultSecretsProvider` — HashiCorp Vault
+- `SopsSecretsProvider` — age/gpg-encrypted files in git
+- `KeyringSecretsProvider` — OS-level keyring
+
+Startup fails fast if a `secret_ref` can't resolve — no silent fallback to anonymous S3.
+
+### Schema changes
+
+**`ObjectRef`** already has a `bucket: String` field — currently it carries the S3 bucket name or `"local"` inconsistently. Repurpose it as the **catalog bucket name**:
+
+```rust
+pub struct ObjectRef {
+    pub bucket: String,        // NOW: catalog bucket name, e.g. "primary" or "client_a"
+    pub key: String,           // object key within that bucket
+    pub size_bytes: u64,
+    pub created_at: DateTime<Utc>,
+}
+```
+
+Migration: a `resync-missing`-style one-shot sets `bucket = "primary"` on every existing ObjectRef whose value is empty or ambiguous.
+
+**`DatasetManifest`** — no new top-level field. `objects: Vec<ObjectRef>` is already where bucket info lives. Per the design invariant (one dataset, one bucket), all ObjectRefs inside a manifest share a bucket, so we can add a convenience accessor `manifest.bucket_id()` → `objects[0].bucket.clone()`.
+
+### Storage layer changes
+
+**`storaged::BucketRegistry`** — new struct, replaces the single `Arc<dyn ObjectStore>` currently threaded through gateway.
+
+```rust
+pub struct BucketRegistry {
+    buckets: HashMap<String, Arc<dyn ObjectStore>>,
+    default: String, // "primary"
+}
+
+impl BucketRegistry {
+    pub fn from_config(cfg: &StorageConfig) -> Result<Self, String>;
+    pub fn get(&self, bucket_id: &str) -> Result<&Arc<dyn ObjectStore>, String>;
+    pub fn default_store(&self) -> &Arc<dyn ObjectStore>;
+    pub fn list_buckets(&self) -> Vec<BucketInfo>;
+}
+```
+
+**`storaged::ops`** gets a bucket parameter on every call:
+```rust
+pub async fn put(registry: &BucketRegistry, bucket: &str, key: &str, data: Bytes) -> Result<(), String>;
+pub async fn get(registry: &BucketRegistry, bucket: &str, key: &str) -> Result<Bytes, String>;
+// etc.
+```
+
+Every existing call site passes `"primary"` initially; specific call sites (ingestd, vectord) learn to route.
+
+### Query layer changes
+
+**`queryd`** — on startup, register every bucket as a separate `object_store` URL with DataFusion:
+
+```rust
+for (name, store) in bucket_registry.iter() {
+    let url = format!("lakehouse-{}://", name).parse::<Url>()?;
+    session_ctx.register_object_store(&url, store.clone());
+}
+```
+
+**`ListingTable`** URLs reference the bucket:
+```rust
+let url = format!("lakehouse-{}://{}", obj.bucket, obj.key);
+```
+
+DataFusion handles cross-bucket scans natively — each partition gets routed to the correct store.
+
+### Gateway API additions
+
+| Endpoint | Purpose |
+|---|---|
+| `GET /storage/buckets` | List all configured buckets + their backend + reachability status |
+| `GET /storage/errors` | Recent bucket operation failures (filtered by bucket/time) |
+| `GET /storage/health` | Summary: which buckets have errored in the last 5 minutes |
+| `POST /ingest/file` + `X-Lakehouse-Bucket: client_a` | Header selects target bucket; absent → `primary` |
+| `POST /profile/{user}/activate` | Pre-load user's profile bucket: embedding caches warm, HNSW indexes rebuilt, workspace state hydrated |
+| `POST /profile/{user}/deactivate` | Evict profile bucket's cached state |
+| `POST /catalog/datasets/by-name/{name}/relocate` | Move a dataset to another bucket (polish) |
+
+**Routing rule for all existing endpoints:**
+1. `X-Lakehouse-Bucket: {name}` header → use that bucket
+2. No header → `primary`
+3. Unknown bucket name → 404
+
+Catalog and query endpoints span all buckets by default — a query sees all buckets the user has access to unless explicitly scoped.
+
+### Profile hot load
+
+When `POST /profile/{user}/activate` fires:
+
+1. Resolve `profile:{user}` bucket — create if missing (first-time activation)
+2. Scan the catalog for all datasets whose `owner == user` or which live in that bucket
+3. For each vector index belonging to the profile: pre-load embeddings into `EmbeddingCache`, pre-build HNSW with the locked-in default config (ec=80 es=30)
+4. Hydrate any saved workspace state (see Phase 8.5) into memory
+5. Return a manifest summary — what's hot, what wasn't found, total memory used
+
+Design contract: `/profile/{user}/activate` should be idempotent. Calling it twice is cheap; the second call is a no-op because the cache is already warm.
+
+### Failure mode — rescue bucket + loud errors
+
+The design goal is **failures stay findable**. When something breaks, you should know within seconds which bucket, which key, which operation, and what happened — without grepping logs.
+
+**Writes**: always go to the target bucket. If the target bucket is unreachable, the request fails hard with 503 and the failure is recorded in the error journal. No queueing, no silent fallback — writes that vanish silently are the worst possible failure mode.
+
+**Reads**:
+- First attempt: target bucket.
+- On failure: fall through to the single configured `rescue_bucket`.
+- If the key exists in rescue, serve it with `X-Lakehouse-Rescue-Used: true` and `X-Lakehouse-Original-Bucket: {target}` response headers so the caller knows fallback happened.
+- If the key isn't in rescue either: 404 with both buckets listed in the error body.
+- Every rescue fallback is appended to the **error journal** (see below).
+
+**Rescue bucket** is a single, well-known bucket configured at the storage level. Typical setup: a periodic sync from `primary` → `rescue` so that when `primary` has an outage, recent data is still reachable. Sync is outside this ADR's scope — operator's job.
+
+**Rest of the system keeps working**: a failed bucket never takes down the gateway. Other buckets remain queryable. Cross-bucket queries scan what they can and surface per-partition failures (which bucket, which key) in the error body.
+
+### Error journal — "find errors ez"
+
+A simple append-only log of every bucket operation that failed or fell through. Stored at `primary://_errors/bucket_errors.jsonl` (yes, primary — because we need it findable even if other buckets are down).
+
+```json
+{"ts":"2026-04-16T10:30:15Z","op":"read","target":"client_a","key":"datasets/candidates.parquet","error":"connection refused","rescued":true,"rescue_key_found":true}
+{"ts":"2026-04-16T10:30:18Z","op":"write","target":"client_a","key":"datasets/new.parquet","error":"connection refused","rescued":false}
+```
+
+Exposed via:
+- `GET /storage/errors?limit=50` — recent errors
+- `GET /storage/errors?bucket=client_a` — filter by bucket
+- `GET /storage/errors?since=2026-04-16T10:00` — filter by time
+- `GET /storage/health` — summary: which buckets have errored in the last 5 minutes
+
+The journal doesn't replace logs (tracing still gets everything), but it gives you a **single authoritative place** to answer the question "has anything been failing?" without tailing systemd journals. If the journal is empty, nothing is broken.
+
+### Vectors are bucket-scoped
+
+Vector indexes live in the same bucket as their source data:
+
+- Data in `primary` → vectors in `primary://vectors/{index_name}.parquet`
+- Data in `profile:daisy` → vectors in `profile:daisy://vectors/{index_name}.parquet`
+- Data in `client_a` → vectors in `client_a://vectors/{index_name}.parquet`
+
+The existing `_hnsw_trials/` and `_hnsw_evals/` prefixes also live per-bucket. The trial journal for a profile's index stays inside that profile's bucket — so tenants don't see each other's tuning history.
+
+**Profile hot load consequence:** activating a profile pre-loads ALL vector indexes in that profile's bucket into `EmbeddingCache` + builds HNSW with the default config. First query after activation is warm.
+
+---
+
+## Implementation phases
+
+### MVP (this is what I'd build next session)
+
+1. **`shared::config`** — extend `StorageConfig` with `buckets: Vec<BucketConfig>` and `profile_root`, with backward-compat fallback from single `[storage]` block
+2. **`shared::secrets`** — `SecretsProvider` trait + `FileSecretsProvider` impl reading `/etc/lakehouse/secrets.toml`
+3. **`storaged::BucketRegistry`** — multi-backend registry; mirrors-of tracking; lazy profile bucket creation
+4. **`storaged::ops`** — add `bucket: &str` param to every call; read path falls back to mirror on unreachable; write path hard-fails
+5. **`gateway`** — `X-Lakehouse-Bucket` header middleware → routing decision per request
+6. **`gateway`** — `GET /storage/buckets` + `POST /profile/{user}/activate|deactivate`
+7. **`catalogd`** — one-shot migration: every ObjectRef without an explicit bucket gets `"primary"`
+8. **`queryd::session`** — register every bucket as a DataFusion ObjectStore under its own URL scheme
+9. **`vectord`** — every vector/trial/eval storage path becomes `{bucket}://vectors/...`; profile activate pre-loads HNSW
+10. **`storaged::error_journal`** — append-only JSONL writer at `primary://_errors/bucket_errors.jsonl`, hooked into every ops call-site
+11. **Test harness** — configure `primary` + `profile:testuser` + a mock `client_a` pointing at a 2nd local dir + `rescue` as a 3rd local dir; verify cross-bucket join, profile activate, rescue fallback, error journal visibility
+
+**Success gate #1 — cross-bucket query:** `SELECT SUM(hours_regular) FROM primary.timesheets t1 JOIN "profile:testuser".timesheets t2 USING (placement_id)` returns a sensible number.
+
+**Success gate #2 — profile hot load:** `POST /profile/daisy/activate` pre-builds HNSW over her personal vector index; next `/search` call against that index is <1ms cold.
+
+**Success gate #3 — rescue fallback + error visibility:** rename `client_a`'s directory to simulate outage. GET for a key returns the copy from `rescue` with `X-Lakehouse-Rescue-Used: true` + `X-Lakehouse-Original-Bucket: client_a` headers. PUT returns 503. `GET /storage/errors` shows both events with full context. `GET /storage/health` flags `client_a` as unhealthy.
+
+### Polish (follow-up)
+
+8. **Bucket health check** — `GET /storage/buckets` shows reachable/unreachable
+9. **Bucket relocation** — `POST /catalog/datasets/{name}/relocate` (copy to new bucket, update manifest, delete from old)
+10. **Runtime bucket addition** — `POST /storage/buckets` adds a bucket without restart (stores in a separate `buckets.toml`)
+11. **S3 streaming optimization** — for very large Parquet files, stream the footer instead of loading the whole blob
+
+---
+
+## Decisions (2026-04-16)
+
+1. **Credentials** — Pluggable `SecretsProvider` trait. MVP ships `FileSecretsProvider` at `/etc/lakehouse/secrets.toml` (root:root, 0600). Future providers (Vault, SOPS, OS keyring) plug in without touching core.
+2. **Primary default + profile bucket model** — `primary` always exists as system-wide fallback. Per-user `profile:{user}` buckets are provisioned on first activate. Named tenant buckets configured explicitly.
+3. **Ingest routing** — `X-Lakehouse-Bucket` header. Absent → `primary`. Unknown → 404.
+4. **Failure** — Writes: hard fail on target, logged to error journal. Reads: fall through to a single shared `rescue_bucket`; response headers `X-Lakehouse-Rescue-Used` + `X-Lakehouse-Original-Bucket` make the fallback visible. Every failure appended to an error journal at `primary://_errors/bucket_errors.jsonl`, queryable via `GET /storage/errors`. Failed buckets never take down unrelated paths.
+5. **Vectors in their bucket** — Vectors, trial journals, and eval sets all live in the same bucket as their source data. Profile activate pre-loads the full vector stack for that profile using the locked-in HNSW default (ec=80 es=30).
+
+---
+
+## Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| DataFusion multi-store predicate pushdown lands on cross-bucket JOINs | **Medium** | Benchmark early. If bad, mark cross-bucket JOINs as a known sharp edge. |
+| S3 credentials accidentally logged | **High** | Never stringify credential fields. Add a test that redacts `secret_key` in every code path. |
+| Existing ObjectRefs have inconsistent `bucket` values (some `"local"`, some empty) | **Low** | One-shot migration normalizes to `"primary"`. |
+| Bucket config drift between restarts | **Medium** | Config-driven means restart = canonical. A runtime API would need careful state reconciliation; defer. |
+
+---
+
+## Decision log
+
+- **1 dataset = 1 bucket** (rather than datasets spanning buckets) — vastly simpler, matches the "tenant isolation" real use case. Revisit only if someone has an actual multi-bucket dataset requirement.
+- **Config-first, runtime-later** — YAML/TOML for the MVP; runtime bucket add/remove is polish. Deployment restart is cheap in this stack.
+- **`bucket: String` repurposed, no new field** — avoids a schema migration on every manifest.
+- **DataFusion's multi-store primitive handles cross-bucket queries** — don't reinvent. We're wiring it up, not building it.
@@ -0,0 +1,105 @@
+# ADR-019: Vector Storage — Parquet+HNSW stays, Lance joins as second tier
+
+**Status:** Accepted — 2026-04-16
+**Implements:** Phase 18 from PRD (Lance evaluation)
+**Supersedes:** nothing (augments ADR-008)
+**Owner:** J
+
+---
+
+## Context
+
+Phase 18 of the PRD committed to settling "Parquet+sidecar vs Lance" with measurements, not vibes. This ADR records the benchmark outcome and the resulting architectural direction.
+
+Input data: `data/vectors/resumes_100k_v2.parquet` — 100,000 × 768d embeddings, the same index we tuned HNSW against in Phase 15.
+
+Benchmark harness: `crates/lance-bench/src/main.rs` — standalone binary, deliberately not integrated into the workspace's common deps to avoid forcing DataFusion/Arrow upgrades on the rest of the stack until we'd decided.
+
+## The scorecard
+
+All numbers measured on the same 128GB server, same 100K × 768d index, release build:
+
+| Dimension | Parquet + HNSW (current) | Lance 4.0 IVF_PQ (candidate) | Winner |
+|---|---|---|---|
+| Cold load | 0.17s | 0.13s | Lance, 1.27× — *does not clear 2× decision threshold* |
+| Disk size (data only) | 330.3 MB | 330.4 MB | Tie |
+| Index on-disk footprint | 0 (HNSW is RAM-only) | 7.4 MB | Lance |
+| Index build time | 230s (ec=80 es=30) | 16s | **Lance, 14× faster** |
+| Search p50 | 873us (recall@10 = 1.00) | 2229us (recall unmeasured, likely 0.85-0.95) | **Parquet+HNSW, 2.55× faster** |
+| Search p95 | 1413us | 4998us | **Parquet+HNSW, 3.54× faster** |
+| Speedup vs brute force (p50) | 50.4× | 19.7× | Parquet+HNSW |
+| Random row access (fetch by id) | ~35ms (full-file scan) | 311us | **Lance, 112× faster** |
+| Append 10K rows | Full-file rewrite (~330MB + re-embed + re-index) | 0.08s, +31MB delta | **Lance, structurally different** |
+
+## Applying the decision rules from EXECUTION_PLAN.md
+
+Original rules:
+- *Lance wins cold-load by ≥2× AND matches search latency → migrate*
+- *Within 50% across board → stay Parquet, document ceiling*
+- *Lance loses → close the door*
+
+Strict reading: cold-load is **1.27×, not ≥2×**. Search latency is **2.55× worse, not matching**. By the written rule, we stay.
+
+But the written rule missed something. It assumed Lance's value would show up as raw-speed wins across the whole table. The actual benchmark reveals Lance's value is **in capabilities the current stack doesn't have**, not in the metrics we scoped:
+
+1. **Random row access** is 112× faster. Our Parquet design can't do O(1) random access to a row — RAG text retrieval is a full-file scan today. Lance makes this native.
+2. **Append** is structurally different. Adding 10K rows is 0.08s on Lance; on our stack it's a full rewrite of the entire 330MB Parquet file plus re-embedding plus re-indexing.
+3. **Index build** is 14× faster. The HNSW `ec=80 es=30` production default takes 230s; Lance IVF_PQ takes 16s. Hot-swap generation (Phase 16) is much more feasible at 16s per build.
+
+## The decision
+
+**Hybrid architecture — neither replace nor reject.**
+
+### What stays
+
+- `vectord::store` with Parquet + binary-blob vectors → **primary vector backend**
+- `vectord::hnsw::HnswStore` → in-RAM HNSW for search at 100K-scale indexes
+- All Phase 15 trial infrastructure → keeps working, unchanged
+- Production default `ec=80 es=30` → still the right call for in-RAM use
+
+### What gets added
+
+- **`vectord::lance_store`** — second backend using Lance as the persistence layer
+  - Scope: indexes where *any* of the following apply:
+    - Corpus exceeds ~5M vectors (our in-RAM ceiling)
+    - Workload is append-heavy (incremental ingest from streaming sources)
+    - Text retrieval dominates (point lookups by doc_id for RAG)
+    - Hot-swap generations are required (Phase 16)
+  - Implemented as a standalone crate first (follow the pilot layout), promoted into vectord when the API stabilizes
+- **Profile-level configuration** — `ModelProfile.vector_backend: Parquet | Lance` so each profile picks the tier that matches its workload
+
+### What we keep watching (but don't act on yet)
+
+- **Lance search latency at scale.** 2229us at 100K is worse than HNSW. At 10M we expect Lance to pull ahead because HNSW doesn't fit in RAM. Re-benchmark when we have a 10M-vector corpus to test against.
+- **IVF_PQ recall.** We measured latency but not recall — I picked `num_partitions=316, nbits=8, num_sub_vectors=48` blindly. A proper recall sweep is part of Phase C when we integrate Lance into the trial system.
+- **Lance's own HNSW-on-disk variant** (`with_ivf_hnsw_pq_params`). Might close the in-RAM latency gap. Left for a future pilot.
+
+## Why this isn't moving the goalposts
+
+The EXECUTION_PLAN rule was "migrate or don't migrate." The evidence says neither is correct — one stack can't serve both the staffing SQL workload AND the LLM-brain append-heavy random-access workload at all scales. The honest answer is two backends, each doing what it's good at, selected per-profile.
+
+This matches the dual-use framing in the 2026-04-16 PRD update: different workloads, shared substrate, per-profile specialization. We wrote that principle into the PRD; the benchmark data just made it concrete for the vector tier.
+
+## Follow-up work (updates EXECUTION_PLAN.md)
+
+- **Phase C (decoupled embedding refresh)** gets easier — Lance's native append removes the need to invent a "vectors delta" Parquet layer. When we build Phase C, use Lance as the embedding-layer backend.
+- **Phase 16 (hot-swap)** becomes feasible — 16s index builds mean online re-trials are cheap. When we build Phase 16, Lance is the storage for index generations.
+- **Phase 17 (model profiles)** gains a new field: `vector_backend: Parquet | Lance`. Default Parquet for backward compatibility. Agents can opt into Lance.
+
+## Costs we accept
+
+- **Second dependency tree.** Lance pulls in DataFusion 52 and Arrow 57, while our main stack runs DataFusion 47 and Arrow 55. Keeping lance-bench isolated works for a pilot; productionizing will need either workspace-wide upgrade or a firewall via a dedicated `vectord-lance` crate.
+- **Second API surface.** Lance's vector-index API is different from our HNSW code. Per-profile abstraction cost is real.
+- **Operational complexity.** Two vector storage implementations to debug and monitor.
+
+Worth it because the alternative — forcing every workload through one backend — means either the staffing case or the LLM-brain case is served badly.
+
+## Ceilings this updates in PRD
+
+The PRD "Known ceilings" table had:
+
+> Vector count per index | ~5M vectors on 128GB RAM | 10M+ (serious web crawl) | Phase 18 Lance migration OR mmap'd embeddings
+
+Update to:
+
+> Vector count per index | ~5M vectors on 128GB RAM (Parquet+HNSW in-RAM) | Past 5M | Switch that profile's `vector_backend` to Lance; IVF_PQ keeps working on disk-resident quantized codes
@@ -0,0 +1,107 @@
+# ADR-020: Universal ID Mapping — the hybrid search identity problem
+
+**Status:** Proposed — 2026-04-17
+**Triggered by:** ID mismatch between vector doc_ids and SQL primary keys
+**Owner:** J
+
+---
+
+## Problem
+
+The hybrid search endpoint (`POST /vectors/hybrid`) filters SQL results
+by primary key, then matches those keys against vector doc_ids. This
+requires the two ID spaces to be compatible. Currently they're not:
+
+| Source | SQL primary key | Vector doc_id | Match? |
+|---|---|---|---|
+| ethereal_workers | `worker_id = 4925` | `W-4925` | Only with W- strip |
+| workers_500k | `worker_id = 41566` | `W500K-41566` | Only with W500K- strip |
+| candidates | `candidate_id = CAND-055035` | `CAND-055035` | Yes (same format) |
+| workers_5k_proof | `worker_id = 3200` | `W5K-3200` | Only with W5K- strip |
+
+Every time a new dataset is ingested and embedded, a new prefix
+appears and the hybrid search breaks until someone hardcodes another
+`strip_prefix` line.
+
+This violates PRD invariant: **"Any data source can be ingested
+without pre-defined schemas."** The ID format IS a pre-defined schema
+that the hybrid search depends on.
+
+## Root cause
+
+When the embedding pipeline creates a vector index, it generates
+doc_ids by concatenating a prefix + the source row's ID column. The
+prefix is chosen at index creation time and baked into the Parquet
+vector file. The SQL dataset knows nothing about this prefix.
+
+The hybrid search then has to reverse-engineer the prefix to match
+vector results against SQL rows. This is fragile and breaks on every
+new data source.
+
+## Solution: catalog-level ID mapping
+
+### Option A: Normalize doc_ids at embedding time (simplest)
+
+When creating a vector index, don't prefix the doc_id. Use the raw
+value from the source column. If the source has `worker_id = 4925`,
+the vector doc_id is just `"4925"` — no `W-`, no `W500K-`.
+
+**Pros:** Simplest. Hybrid search just compares strings directly.
+**Cons:** Doc_ids across different indexes could collide (two datasets
+both have worker_id=1). Need to scope by index name.
+
+### Option B: Catalog stores the mapping (most robust)
+
+The catalog maintains a mapping table:
+```
+index_name | doc_id_prefix | source_dataset | source_id_column
+-----------+---------------+----------------+-----------------
+resumes    | CAND-         | candidates     | candidate_id
+workers_v1 | W500K-        | workers_500k   | worker_id
+```
+
+Hybrid search reads this mapping and applies the prefix/strip
+automatically. New datasets register their mapping at index creation.
+
+**Pros:** Handles all cases. Self-describing. No code changes for new data.
+**Cons:** One more lookup per hybrid search (trivial perf cost).
+
+### Option C: Pass the mapping in the request (pragmatic)
+
+The hybrid search request already accepts `id_column`. Extend it
+with `id_prefix` so the caller says "vector doc_ids have prefix
+W500K- and the SQL column is worker_id."
+
+```json
+{
+  "index_name": "workers_500k_v1",
+  "id_column": "worker_id",
+  "id_prefix": "W500K-",
+  "sql_filter": "role = 'Forklift Operator' AND state = 'IL'",
+  ...
+}
+```
+
+**Pros:** Zero backend change. Caller already knows the context.
+**Cons:** Caller has to know the prefix — not self-service.
+
+## Recommendation
+
+**Option A for new indexes, Option B for registry.**
+
+1. Change the embedding pipeline to use RAW IDs (no prefix) as the
+   default. Existing indexes keep their prefixed IDs.
+2. Add an `id_prefix` column to IndexMeta so the catalog knows how
+   to map between vector and SQL IDs.
+3. Hybrid search reads IndexMeta.id_prefix and applies it automatically.
+4. Remove the hardcoded strip_prefix chain.
+
+This means: ingest a new dataset, embed it, hybrid search works
+immediately. No code changes. The system is truly "any data source."
+
+## Implementation
+
+1. `IndexMeta` gains `id_prefix: Option<String>` (default None = no prefix)
+2. Embedding pipeline: when `id_prefix` is None, use raw IDs from source
+3. Hybrid search: read `IndexMeta.id_prefix`, prepend to SQL IDs before matching
+4. Migration: existing indexes retain their prefix in IndexMeta
@@ -0,0 +1,93 @@
+# ADR-021: Sparse Data Trust Path — start with nothing, earn everything
+
+**Status:** Proposed — 2026-04-17
+**Triggered by:** Legacy staffing company pushback: "we don't have that data"
+**Owner:** J
+
+---
+
+## The Problem
+
+We demonstrated a system with rich worker profiles (18 fields,
+behavioral scores, certifications, communication history). The
+staffing company said: "We don't have any of that. We have a name,
+a phone number, and a contract."
+
+They're right. Our demo assumed data that doesn't exist in their
+world. Showing AI that only works with perfect data is worse than
+useless — it builds distrust.
+
+## Their Reality
+
+Day 1 data for a typical worker:
+- Name
+- Phone number
+- Maybe: city, role, one or two skills mentioned on a call
+
+Day 1 data for a contract:
+- Client name
+- Role needed
+- Headcount
+- Start date
+- Maybe: required certs, location
+
+That's it. No reliability scores. No availability metrics. No
+communication history. No certifications in a database. The staffing
+coordinator carries that knowledge in their HEAD.
+
+## The Trust Path
+
+### Phase 1: Work with what they have (Day 1)
+- Accept sparse profiles: name + phone + role. That's enough.
+- Match contracts to workers by role + location only. No scores.
+- The system is useful immediately: "here are the 12 people you
+  have tagged as Forklift Operators in IL." That's already faster
+  than scrolling a spreadsheet.
+- Don't show empty fields. Don't show 0% bars. Don't show what's
+  missing — show what's THERE.
+
+### Phase 2: Earn data through use (Week 1-4)
+- Every placement generates a timesheet → reliability starts building
+- Every call logged → communication history accumulates
+- Every check-in → availability becomes real
+- Every cert verified → certification database grows
+- The staffer doesn't "enter data" — they just do their job, and
+  the system learns from it.
+
+### Phase 3: AI starts helping (Month 2+)
+- Enough data to actually score workers → show reliability
+- Enough history to predict availability → surface it
+- Enough placements to know client preferences → suggest matches
+- The enrichment happened BECAUSE they used the system, not as a
+  prerequisite TO use it.
+
+## What This Means for the UI
+
+- Worker cards must gracefully degrade. Name only? Show name only.
+  Name + role? Show name + role. Full profile? Show everything.
+- Never show empty metrics. No "Reliability: 0%" — that looks like
+  the worker is unreliable. Just don't show it until there's data.
+- Lead with what the staffer KNOWS: "you placed this worker at
+  Company X last month" — that's information they trust because they
+  lived it.
+
+## What This Means for the Architecture
+
+- The vector index works on whatever text exists. A name + role is
+  200 characters. That's enough for an embedding. As more data
+  arrives, the embeddings get richer and search gets better.
+- The hybrid search works with sparse SQL filters too. "role =
+  'Forklift Operator'" is a valid filter even without reliability.
+- The playbook system captures the staffer's decisions: "you picked
+  this worker for this contract" → that IS the training data for
+  future AI recommendations.
+
+## Implementation
+
+1. Sparse profile ingest: accept CSV with as few as 2 columns
+   (name, phone). Everything else optional.
+2. Graceful UI degradation: worker cards only show fields that exist
+3. Progressive enrichment: timesheet ingest → auto-calculate
+   reliability; check-in ingest → auto-calculate availability
+4. Trust indicators: "based on 3 placements" not "Reliability: 87%"
+   — show WHERE the number comes from
@@ -0,0 +1,69 @@
+# Auditor Context — project preamble for fact extraction
+
+This file is read by `auditor/fact_extractor.ts` and prepended to the
+extract-facts prompt sent to llm_team. The goal: give the extractor +
+verifier enough grounding to ground domain-specific facts instead of
+marking them UNVERIFIABLE by default.
+
+Keep this short (< 400 words). Verifier only reads the first ~4KB of
+the prompt alongside the facts. Longer = noise, not signal.
+
+Update when: a new Phase lands, a crate is added/removed, the project's
+primary domain shifts (e.g. staffing → DevOps).
+
+---
+
+## What Lakehouse is
+
+Lakehouse is a Rust-first data platform over S3-compatible object
+storage. Primary use: a staffing company ingesting legacy CRM data for
+AI-powered worker matching, contract fulfillment, and playbook-driven
+coordination.
+
+Architecture: 13 Rust crates + a Python sidecar (Ollama) + TypeScript
+sub-agents (auditor, scrum_master, bot). Runs on a single server
+(Nvidia A4000, 128GB RAM). All services on localhost: gateway :3100,
+sidecar :3200, UI :3300, MCP :3700, observer :3800, MinIO :9000.
+
+## Key crates (each maps to a responsibility)
+
+- **shared** — types, Arrow helpers, PII utilities, SecretsProvider
+- **proto** — gRPC definitions
+- **storaged** — S3/MinIO I/O, AppendLog, ErrorJournal
+- **catalogd** — metadata authority (manifests, views, tombstones)
+- **queryd** — DataFusion SQL, MemTable cache, compaction
+- **ingestd** — CSV/JSON/PDF/Postgres/MySQL ingest
+- **vectord** — embeddings, HNSW index, **playbook_memory meta-index** (Phase 19+)
+- **vectord-lance** — Lance 4.0 firewall crate (separate Arrow version)
+- **journald** — append-only mutation event log
+- **aibridge** — Rust↔Python sidecar bridge, context budget + continuation
+- **gateway** — Axum HTTP :3100 + gRPC :3101 (Phase 38+ adds /v1/chat)
+- **ui** — Dioxus WASM (stale, pre-Phase-9)
+- **lance-bench** — standalone benchmark
+
+## Current architectural direction (Phase 38-44)
+
+Universal AI Control Plane: a `/v1/chat` OpenAI-compatible API that
+routes all LLM traffic through one layer for token accounting + provider
+fallback. Truth Layer + Validation Pipeline enforce staffing-domain
+invariants (worker eligibility, PII, contract rules). The Auditor
+(Phase A of cohesion plan) hard-blocks PR merges on placeholder code.
+
+## Auditor sub-agent role
+
+`auditor/` (TypeScript, Bun runtime) polls Gitea every 90s for open PRs.
+For each fresh head SHA it runs 4 checks in parallel: static (grep-style
+placeholder detection), dynamic (runs the hybrid fixture), inference
+(gpt-oss:120b cloud review with N=3 consensus + qwen3-coder:480b
+tie-breaker), and kb_query (reads `data/_kb/*.jsonl` for prior evidence).
+Verdicts post to Gitea as commit status + review comment. Findings
+append to `data/_kb/audit_lessons.jsonl` (path-agnostic signatures for
+dedup). Curated scratchpads from tree-split get routed through this
+extract-facts pipeline to populate `audit_facts.jsonl` — which is what
+you (the extractor) are currently producing.
+
+## Things that are NOT the auditor
+
+- The LLM Team UI at `/root/llm_team_ui.py` (devop.live:5000) — a separate product for human-facing multi-model experimentation
+- The scrum_master pipeline at `tests/real-world/scrum_master_pipeline.ts` — reviews files, not claims
+- The bot at `bot/` — will apply fixes, doesn't audit
@@ -0,0 +1,457 @@
+# PRD — Universal AI Control Plane
+
+**Status:** Long-horizon architecture target as of 2026-04-22. Lakehouse Phases 0-37 (`docs/PRD.md`) are preserved as the reference implementation and first domain-specific consumer. Phases 38+ (control-plane layers) are sequenced below.
+
+**Current domain: staffing.** The immediate proving ground is the staffing substrate already built — synthetic workers_500k, contracts, emails, SMS drafts, playbook memory. Everything Phase 38-44 ships is validated first against that domain. The DevOps / Terraform / Ansible framing from the original PRD draft stays as a **long-horizon target** — architecture-compatible but not in current scope. See §Long-horizon domains at the bottom.
+
+**Owner:** J
+
+**Cross-read:** `docs/PRD.md` for what's shipped (staffing + AI substrate, 13 crates, ~3M rows). This doc for the layered architecture those pieces now fit into.
+
+---
+
+## Phase Sequencing (Phases 38-44)
+
+Ship each phase before starting the next. Each ends with green tests + docs update.
+
+| Phase | Layer | What ships | Est. LOC | Risk |
+|---|---|---|---|---|
+| 38 | Layer 1 skeleton | `/v1/chat`, `/v1/usage`, `/v1/sessions` routes forwarding to existing `aibridge` → Ollama. Bot migrates as first consumer. | ~400 | Low — additive, no existing routes touched |
+| 39 | Layer 3 adapters | `aibridge::ProviderAdapter` trait; Ollama + one new (OpenRouter). `/v1/chat` routes by config. | ~500 | Low-medium |
+| 40 | Layer 2 engine | Rules-based routing (`config/routing.toml`), fallback chains, cost gating. Add Gemini + Claude adapters. | ~600 | Medium |
+| 41 | Profile split | Separate Retrieval / Memory / Execution / Observer profiles; Phase 17 backward-compat. Absorbs Phase 37 hot-swap-async. | ~300 | Medium |
+| 42 | Truth Layer | New `crates/truth`; Terraform/Ansible schemas; `/v1/context` serves rules to router + observer. | ~700 | Medium |
+| 43 | Validation pipeline | Syntax/lint/dry-run/policy gates per output type. Plugs into Layer 5 execution loop. | ~400 | Medium |
+| 44 | Caller migration | All internal callers route through `/v1/chat`. Direct sidecar access deprecated. | ~200 | Low |
+
+**Total ≈3100 LOC.** Phase 37 (hot-swap async) folds into Phase 41 — it's an Execution-Profile activation concern.
+
+---
+
+## Phase 38 — Universal API Skeleton
+
+**Goal:** OpenAI-compatible `/v1/*` surface exists and forwards to existing aibridge → Ollama. Nothing about multi-provider yet — just the SHAPE, so every downstream piece (adapters, routing, usage accounting) has a surface to plug into.
+
+**Ships:**
+- `crates/gateway/src/v1/mod.rs` — router + `/v1/chat`, `/v1/usage`, `/v1/sessions`
+- `crates/gateway/src/v1/ollama.rs` — shape adapter (OpenAI chat ↔ existing aibridge `GenerateRequest`)
+- One-line `nest("/v1", ...)` in `crates/gateway/src/main.rs`
+- Unit test: `POST /v1/chat` roundtrips through mocked provider
+
+**Gate:**
+- `curl -X POST localhost:3100/v1/chat -d '{"model":"qwen3.5:latest","messages":[{"role":"user","content":"hi"}]}'` returns valid OpenAI-shape response.
+- `GET localhost:3100/v1/usage` returns `{requests, prompt_tokens, completion_tokens, total_tokens}`.
+- `GET localhost:3100/v1/sessions` returns `{data:[]}` (stub; real impl Phase 41).
+- `cargo test -p gateway` green.
+
+**Non-goals (explicit):** streaming, tool calls, function calling, session state, multi-provider, fallback, cost gating.
+
+**Risk:** Low — additive, doesn't touch existing routes. Worst case: `/v1/*` returns 502 and we fix the adapter. No existing caller affected.
+
+---
+
+## Phase 39 — Provider Adapter Refactor
+
+**Goal:** `aibridge` grows a `ProviderAdapter` trait. Ollama implementation wraps existing sidecar code. One new provider lands as proof: **OpenRouter** (simplest — it's OpenAI-compatible, so adapter is mostly passthrough).
+
+**Ships:**
+- `crates/aibridge/src/provider.rs` — `ProviderAdapter` trait with `chat()` + `embed()` + `unload()` methods
+- `crates/aibridge/src/providers/ollama.rs` — existing sidecar code moved behind the trait
+- `crates/aibridge/src/providers/openrouter.rs` — new, HTTP client to `openrouter.ai/api/v1/chat/completions`
+- `config/providers.toml` — provider registry (name, base_url, auth, default_models)
+- `/v1/chat` routes by `model` field: prefix match (e.g. `openrouter/anthropic/claude-3.5-sonnet` → OpenRouter; bare names → Ollama)
+
+**Gate:**
+- `/v1/chat` with `model: "qwen3.5:latest"` hits Ollama → green
+- `/v1/chat` with `model: "openrouter/openai/gpt-4o-mini"` hits OpenRouter (key from secrets.toml) → green
+- Neither call leaks provider-specific fields upward. Response is always the `/v1/chat` shape.
+
+**Non-goals:** Fallback chain (Phase 40), cost gating (Phase 40), Gemini/Claude adapters (Phase 40).
+
+**Risk:** Low-medium. The trait extraction is mostly a rearrange; OpenRouter is thin. Biggest risk is secret-loading conventions — `SecretsProvider` is already in place, so reuse that path.
+
+---
+
+## Phase 40 — Routing & Policy Engine + Observability Recovery
+
+**Goal:** Replace hardcoded T1-T5 routing with a rules engine. Add Gemini + Claude adapters. Cost gating enforced at router level. **Reinstate Langfuse + Gitea MCP** — recovery of the observability + repo-ops stack J built previously (see `project_lost_stack` memory).
+
+**Ships — routing:**
+- `crates/aibridge/src/routing.rs` — rules engine (match on: task type, token budget, previous attempt failures, profile ID)
+- `config/routing.toml` — rules in TOML (human-editable, hot-reloadable)
+- `crates/aibridge/src/providers/gemini.rs` — `generativelanguage.googleapis.com` adapter
+- `crates/aibridge/src/providers/claude.rs` — `api.anthropic.com` adapter
+- Fallback chain support: if primary returns 5xx or times out, try next in chain
+- Cost gate: per-request budget + daily budget per-provider
+
+**Ships — observability (was lost, now restored):**
+- **Langfuse** self-hosted via Docker Compose. Single source of truth for every LLM call trace: prompt / response / tokens / cost / latency / provider / fallback chain / profile used. UI at `localhost:3000`. Keys in `/etc/lakehouse/secrets.toml`.
+- `crates/aibridge/src/langfuse.rs` — thin fire-and-forget trace emitter. Every `/v1/chat` call spawns a background task that POSTs to `langfuse/api/public/ingestion`. Non-blocking: trace failures never affect response.
+- **Langfuse → observer pipe** — `mcp-server/langfuse_bridge.ts` or similar. Polls Langfuse's trace API at interval, forwards completed traces to observer `:3800/event` with `source: "langfuse"`. KB now sees cost/latency deltas per model, not just outcome deltas.
+- **Gitea MCP reconnect** — the MCP server binary still installed at `/home/profit/.bun/install/cache/gitea-mcp@0.0.10/` gets wired into `mcp-server/index.ts` tool registry. Agents can open PRs, comment on issues, list commits via named tools. Closes Phase 28's repo-ops gap.
+
+**Gate:**
+- Rule like "local models for simple JSON emitters, cloud for reasoning" fires correctly by task type
+- Primary fails → fallback provider hits, response still matches `/v1/chat` shape
+- Daily budget hit → subsequent requests return 429 with clear retry-at header
+- `/v1/usage` reports per-provider breakdown
+- **Every `/v1/chat` call appears in Langfuse UI** with correct prompt, response, latency, token count within 2 seconds of the request completing
+- **Langfuse → observer pipe** delivers trace deltas to KB: `GET :3800/stats?source=langfuse` shows non-zero count after a few scenarios run
+- **Gitea MCP tools callable** — `list_prs`, `open_pr`, `comment_on_issue` exposed in `mcp-server/index.ts`, verifiable via a quick agent scenario
+
+**Non-goals:** Retrieval Profile split (Phase 41), Truth Layer (Phase 42). Langfuse self-hosted UI customization / SSO.
+
+**Risk:** Medium. Multi-provider auth + cost tracking is cross-cutting; Langfuse adds 4-5 Docker containers (PostgreSQL, ClickHouse, Redis, web, worker). Mitigation: every provider call wrapped in a single `dispatch()` function so observability flows through one point; Langfuse Docker Compose is their supported deployment path, well-tested.
+
+---
+
+## Phase 41 — Profile System Expansion (+ Phase 37 hot-swap async folded in)
+
+**Goal:** The existing `ModelProfile` (Phase 17) becomes **ExecutionProfile**. Three new profile types land alongside. Profile activation is async — returns job_id, work runs in background (Phase 37 deliverable).
+
+**Ships:**
+- `crates/shared/src/profiles/` — `ExecutionProfile`, `RetrievalProfile`, `MemoryProfile`, `ObserverProfile`
+- `crates/catalogd` gains per-profile-type CRUD endpoints (`/catalog/profiles/retrieval`, etc.)
+- `crates/vectord/src/activation.rs` — `ActivationTracker` with background-job pattern (Phase 37 content)
+- `POST /vectors/profile/{id}/activate` returns 202 + job_id, polling at `GET /vectors/profile/jobs/{id}`
+- Single-flight guard: refuse new activation if one is pending/running
+- Backward compat: `ModelProfile` still loads, aliased to ExecutionProfile
+
+**Gate:**
+- Activate a profile → returns 202 in <100ms → job completes in background → `/vectors/profile/jobs/{id}` shows progress + final report
+- `tests/multi-agent/run_stress.ts` Phase 3 (hot-swap stress) passes (was SKIPPED)
+- Retrieval + Memory + Observer profiles can be created independently of Execution profile
+
+**Non-goals:** Truth Layer (Phase 42), validation (Phase 43), caller migration (Phase 44).
+
+**Risk:** Medium. Schema change + async refactor. Mitigation: `#[serde(default)]` on all new fields; existing profiles load unchanged.
+
+---
+
+## Phase 42 — Truth Layer (staffing rules first)
+
+**Goal:** New `crates/truth` crate holds immutable task-class constraints. Served via `/v1/context` to router and observer. No layer can override truth. **Staffing rules ship first**; Terraform/Ansible rule shapes are scaffolded but unpopulated until the long-horizon phase.
+
+**Ships:**
+- `crates/truth/src/lib.rs` — `TruthStore` with schema loading (TOML/YAML rules)
+- `crates/truth/src/staffing.rs` — staffing rule shapes:
+  - Worker eligibility (active status, not blacklisted for client, geo match, role match, availability window)
+  - Contract invariants (deadline present, role/count/city/state populated, budget_per_hour_max ≥ 0)
+  - PII handling (redaction rules on fields tagged `PII` before any cloud call — covers existing Phase 10 sensitivity tags)
+  - Client blacklist enforcement (auto-applied before any fill proposal)
+  - Fill requirements (endorsed_names count matches target_count, no duplicate worker_ids within a single fill)
+- `crates/truth/src/devops.rs` — **scaffold only**: empty rule struct for Terraform/Ansible, populated in the long-horizon phase. Keeps the dispatcher signature stable so no refactor needed later.
+- `truth/` dir at repo root — rule files, versioned in git
+- `/v1/context` endpoint — returns applicable rules for a task class (`staffing.fill`, `staffing.rescue`, `staffing.sms_draft`, etc.)
+- Router consults truth before dispatching: if task violates a rule, hard-fail with structured error + rule citation (matches existing Phase 13 access-control pattern)
+
+**Gate:**
+- Submit a fill proposal where a worker is client-blacklisted — router returns 422 + rule citation, no cloud tokens burned
+- Submit a fill with `endorsed_names.length != target_count` — 422 before dispatch
+- Observer cannot promote a correction that violates truth (rejected at router gate)
+- PII redaction verified: SSN / salary fields stripped from prompts before cloud calls
+- Truth reload is explicit (no file-watch hot reload in this phase)
+
+**Non-goals:** Validation execution (Phase 43), policy learning / evolution (deferred), actual Terraform/Ansible rules (long-horizon phase).
+
+**Risk:** Medium. Domain-specific rule enumeration takes discovery — start with a minimal rule set (5-10 staffing rules, derived from existing Phase 10-13 work) and grow organically as real fills surface edge cases.
+
+---
+
+## Phase 43 — Validation Pipeline (staffing outputs first)
+
+**Goal:** Staffing outputs run through schema / completeness / consistency / policy gates. Plug into Layer 5 execution loop — failure triggers observer-correction iteration. This is where the **0→85% pattern reproduces on real staffing tasks** — the iteration loop with validation in place is what made small models successful.
+
+**Ships:**
+- `crates/validator/src/lib.rs` — `Validator` trait: `validate(artifact) -> Result<Report, ValidationError>` + `Artifact` enum over output types
+- `crates/validator/src/staffing/fill.rs` — fill-proposal validator:
+  - Schema compliance (propose_done shape matches `{fills: [{candidate_id, name}]}`)
+  - Completeness (endorsed count == target_count)
+  - Worker existence (every candidate_id present in workers_500k via SQL lookup)
+  - Status check (every worker has status=active, not_on_client_blacklist)
+  - Geo/role match (worker city/state/role matches contract)
+- `crates/validator/src/staffing/email.rs` — generated email/SMS drafts:
+  - Schema (TO/BODY fields present)
+  - Length (SMS ≤ 160 chars; email subject ≤ 78 chars)
+  - PII absence (no SSN / salary leaked into outgoing text)
+  - Worker-name consistency (name in message matches worker record)
+- `crates/validator/src/staffing/playbook.rs` — sealed playbook:
+  - Operation format (`fill: Role xN in City, ST`)
+  - endorsed_names non-empty, ≤ target_count × 2
+  - fingerprint populated (Phase 25 validity window requirement)
+- `crates/validator/src/devops.rs` — **scaffold only**: stubbed Terraform/Ansible validators (`terraform validate`, `ansible-lint`) for the long-horizon phase
+- Task execution loop in gateway: generate → validate → if fail, observer correction + retry (bounded by `max_iterations=3`)
+- Validation results logged to observer (`data/_observer/ops.jsonl`) + KB (`data/_kb/outcomes.jsonl`)
+
+**Gate:**
+- Generate a fill proposal → validator catches a phantom worker_id → observer + cloud rescue propose correction → retry → green. This reproduces the 0→85% pattern on the live staffing pipeline.
+- `/v1/usage` shows iteration count per task, provider fallback chain, and tokens-per-iteration. Cost attribution per task class visible.
+- Reproduces 14× citation-lift finding from Phase 19 refinement on similar geos after validation gates.
+
+**Non-goals:** Caller migration (Phase 44), Terraform/Ansible wired validation (long-horizon).
+
+**Risk:** Medium. Validation shapes have to match actual executor outputs; mitigation is using real scenario runs as test fixtures (we have ~100 of them in `tests/multi-agent/playbooks/`).
+
+---
+
+## Phase 44 — Caller Migration + Direct-Provider Deprecation
+
+**Goal:** Every internal LLM caller routes through `/v1/chat`. Direct sidecar / direct Ollama / direct OpenAI calls are removed or explicitly deprecated with a warning.
+
+**Ships:**
+- `aibridge::AiClient` becomes a thin `/v1/chat` client (was direct-to-sidecar)
+- `crates/vectord::agent` (autotune): routes through `/v1`
+- `crates/vectord::autotune`: routes through `/v1`
+- `tests/multi-agent/agent.ts::generate()`: routes through `/v1`
+- `bot/propose.ts`: routes through `/v1` (already proposed as Phase 38's test consumer, formalized here)
+- Lint rule / grep pre-commit hook: no `fetch.*:3200/generate` outside the provider adapters
+
+**Gate:**
+- `grep -r "localhost:3200/generate\|/api/generate"` returns only adapter files + deprecation shims
+- `/v1/usage` accounts for every LLM call in the system within a 1-minute window after hitting a fresh scenario
+- Full scenario passes end-to-end without any caller bypassing `/v1/*`
+
+**Non-goals:** New features. This phase is purely mechanical migration.
+
+**Risk:** Low. Mechanical. Tests catch regressions.
+
+---
+
+## Phase 45 — Doc-drift detection + context7 integration
+
+**Goal:** Playbooks know which external docs they were written against. When those docs change (Docker adds a feature, npm lib goes major, Terraform renames a resource), the playbook is automatically flagged. Small models never run confidently-outdated procedures — the drift signal reaches them before the next execution does.
+
+**Why this phase exists at all:** The 0→85% thesis depends on the hyperfocus lane staying valid. External doc drift invalidates the lane silently — popular playbooks can compound the wrong way, accumulating boost while growing more wrong. Phase 25 already retires playbooks on *internal* schema drift; Phase 45 is the same mechanism against *external* doc drift. This is the completion of the learning loop, not an optional add-on.
+
+**Ships:**
+- `shared::types::DocRef` — `{ tool: String, version_seen: String, snippet_hash: Option<String>, source_url: Option<String>, seen_at: DateTime<Utc> }`
+- `PlaybookEntry.doc_refs: Vec<DocRef>` — `#[serde(default)]` so pre-Phase-45 entries load as empty vec
+- `/vectors/playbook_memory/seed` + `/revise` accept `doc_refs` in the request body
+- `/vectors/playbook_memory/doc_drift/check/{id}` — manual drift check: looks up each `doc_refs[]` entry via the context7 bridge, returns per-tool `{version_seen, version_current, drifted: bool}` plus overall verdict
+- `/vectors/playbook_memory/doc_drift/scan` — batch scan across all active playbooks (scheduled path for Phase 45.2)
+- `mcp-server/context7_bridge.ts` — Bun HTTP bridge. Exposes `GET /docs/:tool/version` + `GET /docs/:tool/:version/diff?since=X` against the installed context7 MCP plugin. Gateway calls this over localhost.
+- `PlaybookMemory::compute_boost_for_filtered_with_role` — excludes entries where `doc_drift_flagged_at.is_some() && doc_drift_review.is_none()` (same rule as retired + superseded)
+- Overview model synthesis writes `data/_kb/doc_drift_corrections.jsonl` per detected drift: `{playbook_id, tool, version_seen, version_current, diff_summary, recommended_action, generated_at}`
+- Human-in-the-loop re-seal path: `/vectors/playbook_memory/doc_drift/resolve/{id}` — marks reviewed, optionally triggers `revise_entry` if procedure changed
+
+**Gate:**
+- Seal a playbook referencing Docker 24.x → doc_refs captured. Bump Docker version behind the scenes → `/doc_drift/check/{id}` returns `drifted: true, from: 24.0.7, to: 25.0.1, summary: "..."`. The boosted playbook count on next `/vectors/hybrid` query drops by 1 (drift-flagged skipped).
+- `doc_drift_corrections.jsonl` contains the overview model's synthesis for the drift with at least: summary of change, recommended action, cost/impact estimate.
+- Human calls `/doc_drift/resolve/{id}` after reviewing → playbook returns to active boost pool (or supersedes via Phase 27 if procedure materially changed).
+- Unit tests: DocRef serde default (legacy entries load as empty), drift check against mocked context7 bridge, boost exclusion when drifted+unreviewed.
+
+**Non-goals (explicit):**
+- Automatic re-seal without human review. Drift-detection → flag, not silent rewrite.
+- Cross-playbook propagation of one drift diagnosis. Each playbook reviewed individually (aggregation later if warranted).
+- Generating the updated procedure. T3 *suggests*; human or separate bot (see `bot/`) *writes*.
+
+**Risk:** Medium. The context7 bridge is new infrastructure (Bun ↔ context7 MCP plugin ↔ HTTP shape for gateway consumption). Mitigation: context7 plugin is already installed; its MCP tools return structured JSON; the bridge is thin adapter code. Start with single-tool drift check (Docker) before broadening.
+
+---
+
+## Long-horizon domains (not in current phase sequence)
+
+The architecture was drafted with DevOps execution (Terraform, Ansible) as the eventual target. **That remains aspirational, not current scope** — we don't start wiring `terraform validate` / `ansible-lint` until the staffing domain proves the six-layer architecture at scale.
+
+What "proves at scale" means concretely:
+- Phases 38-44 all shipped against staffing, green tests
+- Live staffing pipeline handles **multiple concurrent contracts** with emails + SMS + indexed playbooks via `/v1/*`
+- Observed **iteration success lift** (the 0→85% pattern) reproduced on varied staffing scenarios, not just the original proof-of-concept
+- Token + cost accounting stable across provider fallback chains under real load
+- Truth Layer rules prevent real fill errors before cloud burn (not just theoretical)
+
+When staffing hits that bar, the DevOps domain lights up by:
+- Populating `crates/truth/src/devops.rs` with real Terraform/Ansible rule shapes
+- Populating `crates/validator/src/devops.rs` with `terraform validate` / `ansible-lint` shell-out
+- Adding DevOps task classes to `/v1/context` rule lookup
+- No architectural changes needed — the dispatcher, router, and execution loop stay identical.
+
+Other candidate long-horizon domains (same pattern):
+- Code generation tasks (validation via `cargo check` / `bun test`)
+- SQL query generation (validation via EXPLAIN + schema compliance)
+- Data pipeline definitions (validation via lineage check + schema compliance)
+
+None of these are in the current roadmap. **Staffing first, production-proven, then expand.**
+
+---
+
+## 1. Purpose
+
+Design and implement a universal AI control-plane API that enables:
+
+- **deterministic high-stakes task execution** — the immediate domain is staffing fills (contracts, workers, emails, SMS) at scale; the same architecture extends later to DevOps (Terraform, Ansible) without redesign
+- iterative capability amplification via observer loops
+- hybrid local + cloud model orchestration
+- structured knowledge + memory + playbook reuse
+- controlled improvement over time through validated iteration
+
+The system prioritizes **validated pipeline success over raw model intelligence**.
+
+### Current scope — staffing at scale
+
+The architecture must make the already-built staffing substrate reliably answer millions of inputs: pull real data, graph it across contracts, handle multiple concurrent contracts, index emails + SMS + playbooks via the hybrid SQL+vector method, and get **faster and better each iteration** via the feedback loops (Phase 19 playbook boost, Phase 22 KB pathway recommender, Phase 24 observer, Phase 26 Mem0 upsert).
+
+DevOps is an eventual domain — see §Long-horizon domains.
+
+## 2. Core Objectives
+
+### 2.1 Functional Goals
+
+- Provide a single universal API for all AI interactions
+- Support multi-provider routing (local, flat-rate, token-based)
+- Enable iterative execution loops with observer correction
+- Store and reuse successful execution playbooks
+- Integrate: S3-based knowledge storage, LanceDB retrieval/indexing, Mem0 memory layer, MCP tool ecosystem
+
+### 2.2 Non-Functional Goals
+
+- Deterministic behavior under constrained execution
+- Full observability and cost accounting
+- Safe DevOps execution (no uncontrolled mutation)
+- Profile-driven routing and execution
+- Reproducibility of successful runs
+
+## 3. System Architecture
+
+### 3.1 Layer Overview
+
+**Layer 1 — Universal API**
+
+Single entry point for all applications. Endpoints:
+
+- `/v1/chat`
+- `/v1/respond`
+- `/v1/tools`
+- `/v1/context`
+- `/v1/usage`
+- `/v1/sessions`
+
+All programs must use this layer. No direct provider calls allowed.
+
+**Layer 2 — Routing & Policy Engine**
+
+Responsibilities: provider selection, fallback logic, cost gating, premium access control, profile enforcement.
+Routing based on: task type, constraints, execution profile, system health.
+
+**Layer 3 — Provider Adapter Layer**
+
+Normalizes all providers: Ollama (local), OpenRouter, Gemini (direct), Claude (direct or routed), future providers.
+Guarantee: no provider-specific logic leaks upward.
+
+**Layer 4 — Knowledge & Memory Plane**
+
+- Knowledge (S3 + LanceDB): raw documents, processed chunks, embeddings, index profiles
+- Memory (Mem0): extracted facts, entity-linked memory, session-aware retrieval
+- Playbooks: successful execution traces, reusable patterns, correction strategies
+
+**Layer 5 — Execution Loop**
+
+Each task runs through: Retrieval → Planning → Generation → Validation → Observer feedback → Iteration (if needed).
+
+**Layer 6 — Observability & Accounting**
+
+Every request logs: tokens (input/output), cost, latency, provider, fallback chain, profile used, iteration delta.
+
+## 4. Execution Model
+
+### 4.1 Iterative Loop
+
+Each task follows: **Attempt → Validate → Observe → Adjust → Retry**
+
+Constraints:
+
+- max iterations (default: 3)
+- minimum improvement threshold
+- cost ceiling per task
+
+### 4.2 Observer Role
+
+Observer can: analyze failure, suggest corrections, recommend profile changes.
+Observer cannot: modify truth layer, auto-promote changes, override constraints.
+
+### 4.3 Cloud Escalation
+
+Cloud models (Gemini, Claude) are used for: structural correction, reasoning gaps, complex decomposition.
+They are not used for: brute-force retries, bulk execution.
+
+## 5. Profile System
+
+### 5.1 Profile Types
+
+- **Retrieval Profile** — chunking strategy, embedding method, reranking rules
+- **Memory Profile** — memory weighting, context injection rules
+- **Execution Profile** — allowed providers, tool access, risk level
+- **Observer Profile** — mutation aggressiveness, iteration strategy
+
+### 5.2 Profile Constraints
+
+- only one major profile change per iteration
+- profiles must produce measurable deltas
+- promotion requires repeated success
+
+## 6. Truth Layer (Critical)
+
+Defines non-negotiable constraints:
+
+- Terraform rules
+- Ansible structure requirements
+- security policies
+- organization standards
+
+Rules:
+
+- immutable at runtime
+- referenced by all layers
+- cannot be overridden by observer
+
+## 7. Playbook System
+
+### 7.1 Playbook Definition
+
+Each successful run produces: task class, context used, steps executed, tools used, output artifacts, validation results, cost/latency, success score.
+
+### 7.2 Playbook Lifecycle
+
+- created on success
+- reused for similar tasks
+- decayed over time
+- pruned if ineffective
+
+## 8. Validation System
+
+All DevOps outputs must pass: syntax validation, linting, dry-run, policy compliance.
+Failure → iteration continues or task fails.
+
+## 9. MCP Integration
+
+MCP servers provide: tools, external data, execution capabilities.
+All MCP outputs must be: normalized, validated, schema-compliant.
+No direct MCP output reaches the model.
+
+## 10. Token Accounting & Budget Control
+
+Each request tracks: input tokens, output tokens, retries, fallback cost.
+Policies: premium providers gated, cost ceilings enforced, per-task budget limits.
+
+## 11. Failure Handling
+
+**Recoverable failures:** bad decomposition, missing steps, weak retrieval → observer + iteration.
+
+**Hard failures:** missing truth data, invalid task classification, unsafe execution → termination + error report.
+
+## 12. Success Criteria
+
+A task is successful only if:
+
+- output is valid
+- all validators pass
+- no policy violations
+- result is reproducible
+- cost within limits
+
+## 13. Key Risks & Mitigations
+
+- **Observer drift** → bounded authority, confidence tracking
+- **Memory poisoning** → validation layer, memory weighting
+- **Cost explosion** → token accounting, iteration caps
+- **Retrieval errors** → post-retrieval validation, profile tuning
@@ -0,0 +1,106 @@
+# Architecture Decision Records
+
+## ADR-001: Object storage as source of truth
+**Date:** 2026-03-27
+**Decision:** All data lives in S3-compatible object storage. No traditional database.
+**Rationale:** Eliminates DB operational overhead, enables infinite scale at storage tier, forces clean separation of data and metadata.
+
+## ADR-002: Catalog metadata persistence
+**Date:** 2026-03-27
+**Decision:** catalogd persists manifests as Parquet files in object storage. In-memory index rebuilt on startup.
+**Rationale:** No external DB dependency. Storage is already the source of truth. Write-ahead pattern ensures consistency.
+
+## ADR-003: Real models only (no mocks)
+**Date:** 2026-03-27
+**Decision:** AI sidecar hits Ollama with real models from Phase 3 onward. No stub/mock endpoints.
+**Rationale:** Local Ollama instance available with nomic-embed-text, qwen2.5, mistral, gemma2, llama3.2. Mocks hide integration bugs.
+
+## ADR-004: Python sidecar as Ollama adapter
+**Date:** 2026-03-27
+**Decision:** Python FastAPI sidecar is a thin HTTP adapter over Ollama's API. No model loading in Python.
+**Rationale:** Ollama handles model lifecycle, GPU scheduling, caching. Sidecar stays stateless and lightweight — no torch/transformers deps.
+
+## ADR-005: HTTP-first, gRPC later
+**Date:** 2026-03-27
+**Decision:** All inter-service communication uses HTTP through Phase 4. gRPC migration in Phase 5.
+**Rationale:** HTTP is simpler to debug, test, and iterate on. gRPC adds protobuf compilation and streaming complexity before APIs stabilize.
+
+## ADR-006: TOML config over environment variables
+**Date:** 2026-03-27
+**Decision:** System configuration via lakehouse.toml file, with sane defaults for all values.
+**Rationale:** Config files are versionable, self-documenting, and support structured data. Env vars remain available as overrides. System must be restartable from repo + config alone.
+
+## ADR-007: Dual HTTP+gRPC on gateway
+**Date:** 2026-03-27
+**Decision:** Gateway serves HTTP on :3100 (external) and gRPC on :3101 (internal). Both run in the same process.
+**Rationale:** Single binary simplifies deployment. HTTP stays for browser/curl access. gRPC provides typed contracts for service-to-service calls. No premature microservice split.
+
+## ADR-008: Embeddings stored as Parquet, not a proprietary vector DB
+**Date:** 2026-03-27
+**Decision:** Vector embeddings stored as Parquet files (doc_id, chunk_text, vector columns). Vector index (HNSW) serialized as a sidecar file.
+**Rationale:** Keeps all data in one portable format. No vendor lock-in to Pinecone/Weaviate/Qdrant. Vectors are queryable via DataFusion like any other data. Trade-off: brute-force search is fine up to ~100K vectors; HNSW needed beyond that.
+
+## ADR-009: Incremental updates via delta files, not Delta Lake
+**Date:** 2026-03-27
+**Decision:** Updates append to delta Parquet files. Queries merge base + deltas at read time. Periodic compaction merges deltas into base. Single-writer model (no concurrent writers).
+**Rationale:** Full ACID over Parquet (Delta Lake/Iceberg) is a multi-year project. Our use case is single-writer (one ingest pipeline) with read-heavy workloads. Merge-on-read with compaction is sufficient and dramatically simpler.
+
+## ADR-010: Schema detection defaults to string
+**Date:** 2026-03-27
+**Decision:** Ingest pipeline infers column types from data. When ambiguous or mixed, defaults to String rather than failing.
+**Rationale:** Legacy data is messy. A column with "123", "N/A", and "" is a string, not an integer. Downstream queries can CAST as needed. Better to ingest everything than reject on type errors.
+
+## ADR-011: This is not a CRM replacement
+**Date:** 2026-03-27
+**Decision:** The lakehouse is the analytical layer BEHIND operational systems. It ingests exports, not live data. CRM/ATS stays for daily operations.
+**Rationale:** Operational systems need single-record CRUD, permissions, UI workflows. The lakehouse answers cross-cutting questions that no single operational system can. They complement, not compete.
+
+## ADR-012: Event journal — append-only mutation history
+**Date:** 2026-03-27
+**Decision:** Every data mutation (insert, update, delete) is appended to an immutable event journal. The journal stores: entity, field, old_value, new_value, actor, timestamp, source, workspace_id. Events are never modified or deleted.
+**Rationale:** This is the single most important future-proofing decision. AI auditability ("why did the agent recommend this candidate?"), compliance ("who changed this PII field?"), and time-travel queries ("what did this record look like on March 1st?") all require mutation history. This is impossible to retrofit — once history is lost, it's gone. The cost to implement is low (append-only Parquet), the cost of NOT implementing grows every day.
+
+## ADR-013: Rich metadata is a product, not a byproduct
+**Date:** 2026-03-27
+**Decision:** Every dataset in the catalog carries: owner, sensitivity classification, lineage (source_system → ingest → dataset), freshness SLA, description, and tags. Auto-detected where possible, required on manual ingest.
+**Rationale:** Datasets without metadata become "mystery files" within months. As data volume grows and AI agents consume data, the metadata layer is what makes the platform discoverable, governable, and trustworthy. Legacy companies that skip this step end up with expensive data swamps instead of data platforms.
+
+## ADR-014: Embedding versioning — model-proof vector layer
+**Date:** 2026-03-27
+**Decision:** Every vector index tracks: model_name, model_version, dimensions, created_at. Multiple index versions for the same data coexist. Re-embedding on model upgrade is incremental (only new/changed docs).
+**Rationale:** Embedding models improve rapidly. nomic-embed-text today, something better in 6 months. Without version tracking, upgrading means re-embedding the entire corpus. With versioning, you can A/B test new models, migrate incrementally, and maintain backward compatibility.
+
+## ADR-015: Tool registry before raw SQL for agents
+**Date:** 2026-03-27
+**Decision:** AI agents interact with the system through named, governed tools (search_candidates, update_phone, create_placement) rather than raw SQL. Tools have parameter validation, permission checks, audit logging, and rate limits.
+**Rationale:** In 3 years, most data access will be by AI agents, not humans typing SQL. Giving agents raw SQL access is ungovernable — you can't audit, permission, or rate-limit individual operations. Named tools with contracts are the interface that scales. Building the governed interface first prevents the technical debt of retrofitting controls onto raw access.
+
+## ADR-016: Agent workspaces as first-class concept
+**Date:** 2026-03-27
+**Decision:** Each contract/search gets a named workspace with saved queries, shortlists, activity logs, and delta layers. Workspaces have daily/weekly/monthly tiers and support instant zero-copy handoff between agents.
+**Rationale:** Staffing workflows are inherently agent-centric — a recruiter works a contract, builds context, then may need to hand it off. The workspace captures that context in a structured, queryable, transferable format. Without it, handoff means "read the email thread and figure it out."
+
+## ADR-017: Federated multi-bucket storage
+**Date:** 2026-04-16
+**Decision:** Every `ObjectRef` belongs to exactly one named bucket. Three bucket classes: `primary` (system default, always present), `profile:{user}` (per-user/per-model workspace bucket), and named tenant buckets (`client_a`, `client_eu`). A single shared `rescue_bucket` handles read fallback on target failure. Writes hard-fail on unreachable target; no silent fallback. Every bucket op failure lands in an error journal at `primary://_errors/bucket_errors/`, queryable via `/storage/errors`. Credentials never live in `lakehouse.toml` — a pluggable `SecretsProvider` trait resolves opaque `secret_ref` handles.
+**Rationale:** The single-backend assumption breaks when we want: tenant data isolation, data residency (EU bucket), per-profile workspaces for local models. DataFusion already supports multiple registered object stores; catalogd is the cross-bucket metadata authority. Rescue bucket + visible error journal makes operational failures diagnosable in one HTTP call. See `docs/ADR-017-federation.md` for full design + success gates.
+
+## ADR-018: Write-once batched append pattern for journals
+**Date:** 2026-04-16
+**Decision:** All append-only journals (error journal, HNSW trial journal, future audit logs) use the `storaged::append_log::AppendLog` helper. Events accumulate in an in-memory buffer; on threshold or explicit `flush()`, the buffer is written as one new timestamped file (`batch_{epoch_us}.jsonl`). Existing files are never rewritten. `compact()` merges all batches into one with a fresh timestamp, preserving chronological sort order.
+**Rationale:** Object stores have no append primitive. Naive "read-modify-write the whole JSONL file on every event" is O(N²) cumulative work and creates the classic small-file / rewrite-amplification anti-pattern that llms3.com flags as the top lakehouse pitfall. Write-once batching is the LSM-tree idea applied to small JSONL events — bounded write amplification, append-only semantics, optional compaction for read efficiency. The in-memory ring buffer preserves O(1) recent-event reads for the `/storage/errors` and `/hnsw/trials` query endpoints.
+
+## ADR-019: Vector storage — Parquet+HNSW primary, Lance secondary (hybrid)
+**Date:** 2026-04-16
+**Decision:** Keep Parquet + binary-blob vectors + in-RAM HNSW as the primary vector backend. Add Lance as a second backend available per-profile for workloads where Lance wins architecturally. Per-profile `vector_backend: Parquet | Lance` field becomes part of Phase 17 model profiles. Implementation kicks off via the standalone `crates/lance-bench` crate and is promoted into `vectord::lance_store` when the API stabilizes.
+**Rationale:** Head-to-head benchmark on the 100K × 768d `resumes_100k_v2` index (see `docs/ADR-019-vector-storage.md` for the full scorecard). Parquet+HNSW wins current-scale search latency by 2.55× (873us vs 2229us p50). Lance wins index build time by 14× (16s vs 230s), random row access by 112× (311us vs ~35ms full-file scan), and append speed structurally (0.08s vs full Parquet rewrite). Neither strictly dominates — the dual-use PRD framing (staffing + LLM brain) means both workloads exist in the same system. Keeps ADR-008's "Parquet is the format" principle intact for dataset tables; adds Lance as a purpose-built vector-tier option without discarding the tuned HNSW stack.
+
+## ADR-020: `register()` is idempotent by name with a schema-fingerprint gate
+**Date:** 2026-04-19
+**Decision:** `catalogd::Registry::register(name, fingerprint, objects)` is idempotent on `name`. If no manifest for `name` exists, create one. If one exists with the same `schema_fingerprint`, reuse its `DatasetId`, replace `objects`, bump `updated_at`, and write through. If one exists with a different `schema_fingerprint`, reject with `409 Conflict` (HTTP) / `FAILED_PRECONDITION` (gRPC). A one-shot operator endpoint `POST /catalog/dedupe` collapses any pre-existing duplicates (preferring the manifest with a non-null `row_count`, then the most recently updated).
+**Rationale:** Registry was keyed by surrogate `DatasetId` with no uniqueness constraint on `name`, so every caller that re-registered (re-ingest, external cron, gRPC retry) silently created a parallel manifest pointing at the same parquet — accumulating 308× `successful_playbooks` in live state before detection. The fingerprint gate turns re-ingest into an explicit no-op (matching PRD invariant #5 "ingestd is idempotent — re-ingesting the same file is a no-op") while forcing schema drift to be visible instead of silently clobbering. 409 status separates policy rejections from server errors, which matters for the Phase 12 tool-consumer ecosystem. Concurrency: the write lock is held across the storage write to close the check→insert TOCTOU window; serializing registers is acceptable because registers-per-second is low on the ingest path. Audit: idempotent-register events are visible as bumps to the stored manifest's `updated_at` field and in `catalogd` tracing output (tracing is non-durable, operator view only); `DedupeReport` is the return-value audit for cleanup runs. No event-journal entries are emitted — ADR-012 scopes the journal to row-level mutations, not catalog-manifest operations.
+
+## ADR-021: Semantic-correctness layer on pathway_memory — matrix-indexed bug grammar
+**Date:** 2026-04-24
+**Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec<SemanticFlag>` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec<TypeHint>` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec<BugFingerprint>` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002.
+**Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon.
@@ -0,0 +1,233 @@
+# Execution Plan — Phases B through E
+
+**Created:** 2026-04-16
+**Status:** Active planning document — update as phases complete or scope shifts
+**Owner:** J
+
+This plan sequences the work J and Claw agreed on during the 2026-04-16 reframe session, after stress-testing the "dual-use substrate" vision and aligning it with llms3.com's architectural patterns.
+
+---
+
+## The four phases, at a glance
+
+| Phase | Work | Prereq | Estimated cost | Risk |
+|---|---|---|---|---|
+| B | Lance pilot on one vector index | None | 1 focused session | Medium — new dep, unfamiliar surface |
+| C | Decoupled embedding refresh pipeline | Benefits from B's outcome | 1 focused session | Low — additive, doesn't break existing |
+| D | AI-safe views | Phase 13 (done) | 1 focused session | Low — builds on existing catalog + tool registry |
+| E | Soft deletes / tombstones | None | 1-2 focused sessions | Medium — touches query path, compaction |
+
+Each "focused session" ≈ 3-4 hours of coding + verification + doc update.
+
+---
+
+## Phase B — Lance pilot (the storage format question)
+
+### Why now
+
+J's LLMS3 knowledge base explicitly positions Lance as `alternative_to` Parquet for vector workloads. We admitted in the 2026-04-16 stress test that our Parquet-portability argument for vectors is weaker than advertised — our vector Parquet blobs aren't readable by DuckDB/Polars anyway. Lance could unlock: random-row access, disk-resident indexes, time-travel, better compression. Or it could disappoint and we lock in Parquet with a written reason why.
+
+**Commit to one answer backed by measurements.** No ambiguity after this phase.
+
+### Scope
+
+1. Add `lance` crate as a dep in `vectord`, behind a `feature = "lance"` flag initially so we can build without forcing users to install it
+2. New module `vectord::lance_store` — mirror of `vectord::store` but against Lance format
+3. New endpoint `POST /vectors/lance/index` — build a Lance index for a named source (parallel to the existing Parquet path)
+4. Benchmark script that runs against `resumes_100k_v2` (the existing 100K reference)
+
+### Measured dimensions
+
+All benchmarks against `resumes_100k_v2` (100K × 768d), cold start:
+
+| Metric | Parquet baseline | Lance | Threshold to migrate |
+|---|---|---|---|
+| Cold load from disk | ~2.8s (measured) | ? | ≥2× faster |
+| Search latency p50 | 873us (ec=80 es=30) | ? | Within 50% |
+| Disk size | 330MB | ? | Comparable or better |
+| Single-row random access | N/A (full scan) | ? | <10ms |
+| Append 10K new rows | Full rewrite (~3s) | ? | Incremental <500ms |
+
+### Decision rules
+
+- **Lance wins cold-load by ≥2× AND matches search latency:** migrate vector storage to Lance. Dataset tables stay Parquet. Update ADR-008 → ADR-019.
+- **Lance within 50% across board:** stay Parquet. Document ceiling honestly in PRD (already done). Lance revisit when we have a problem Parquet can't solve.
+- **Lance loses:** close the door, don't revisit without new evidence. Write ADR-019 as "Lance evaluated, rejected, here's why with numbers."
+
+### Success gate
+
+Benchmark output table posted to `docs/ADR-019-vector-storage.md` with measured numbers in each cell of the table above. Decision rule applied mechanically. No "let's defer the call."
+
+### Rollback
+
+The `feature = "lance"` flag means if the pilot goes badly, `cargo build` without the flag is unchanged. No production migration happens until ADR-019 commits to the change. Safe experiment.
+
+---
+
+## Phase C — Decoupled embedding refresh
+
+### Why now
+
+llms3.com's lakehouse architecture explicitly separates "transactional data mutations" from "asynchronous vector refresh cycles." Today we couple them — an ingest writes rows AND embeddings in one flow. That means:
+
+- Adding 1K rows to a 100K-row dataset forces re-embedding of ALL rows (or nothing)
+- No notion of "embeddings are stale, schedule a refresh tonight"
+- The embedding cost (Ollama-bound, the bottleneck) is synchronous with ingest
+
+### Scope
+
+1. Add fields to `DatasetManifest`:
+   - `last_embedded_at: Option<DateTime>`
+   - `embedding_stale_since: Option<DateTime>` (set when data written but embeddings not refreshed)
+   - `embedding_refresh_policy: RefreshPolicy` — `Manual` | `OnAppend` | `Scheduled(cron)`
+2. Decouple ingest from embed: ingest writes data + marks embeddings stale; embedding runs separately
+3. New endpoint: `POST /vectors/refresh/{dataset}` — diffs existing vectors vs current rows, only embeds new/changed (keyed by `doc_id`)
+4. Background scheduler (or cron trigger) — for `Scheduled` policy, re-runs refresh per schedule
+5. `GET /vectors/stale` — lists datasets with stale embeddings and how stale
+
+### Measured success
+
+- Ingest a 1K-row append to `kb_team_runs` (currently 586 rows, Postgres-sourced).
+- Catalog shows `embedding_stale_since = now`.
+- `POST /vectors/refresh/kb_team_runs` embeds only the 1K new rows, not all 1586.
+- Result: new rows searchable, old embeddings unchanged, total Ollama time ~5s instead of ~30s.
+
+### Dependencies on Phase B
+
+If Lance wins Phase B, this is dramatically easier — Lance supports native append. If we stay Parquet, we need a "vectors delta" Parquet file that merges at read time (same pattern as Phase 8's data delta files). ~100 extra LOC if we stay Parquet.
+
+### Rollback
+
+The `refresh_policy` field defaults to `Manual` for all existing datasets, so no behavior change for anything already in the system. Opt-in per dataset.
+
+---
+
+## Phase D — AI-safe views
+
+### Why now
+
+llms3.com's framing: "AI-safe views enforcing row/column security + PII tokenization before model exposure." Phase 13 gave us role-based column masking at query time. That's per-query enforcement. "Views" means pre-materialized: create `candidates_safe` once, bind model X to that view, model X can never accidentally see raw `candidates`.
+
+This is also the precondition for Phase 17 (model profiles) to be meaningfully safe. "Bind model to dataset" isn't enough — needs to be "bind model to a safe view of the dataset."
+
+### Scope
+
+1. New catalog entity: `AiView` with fields `name`, `base_dataset`, `columns[]` (whitelist), `row_filter` (optional SQL WHERE clause), `column_redactions[]` (PII tokenization rules)
+2. Persistence: `_catalog/views/{name}.json` alongside manifests
+3. Query-rewrite layer: when a query references `candidates_safe`, DataFusion sees an equivalent `SELECT (whitelisted cols) FROM candidates WHERE (row_filter)` — with redactions applied as expressions
+4. Endpoint: `POST /catalog/views` to create, `GET /catalog/views` to list, `GET /catalog/views/{name}/preview` to see what it looks like
+5. Tool registry integration: tools can bind to an AiView instead of a raw table; agent invocations go through the view
+
+### Measured success
+
+- Create view `candidates_safe` = `SELECT candidate_id, skills, city FROM candidates WHERE status != 'blocked'`.
+- Agent (tool registry) calls `search_candidates` bound to `candidates_safe`.
+- Agent cannot see `email`, `phone`, `ssn`, or `status='blocked'` rows, even if it writes raw SQL.
+- Audit log records agent accessed `candidates_safe`, not `candidates`.
+
+### Dependencies
+
+- Phase 13 already provides the sensitivity classification layer
+- Phase 12 tool registry already exists
+- This phase is the bridge between them for agent access
+
+### Rollback
+
+Views are additive. Dropping the feature = delete view definitions, tool registry falls back to direct table access. No data migration needed.
+
+---
+
+## Phase E — Soft deletes / tombstones
+
+### Why now
+
+GDPR/CCPA compliance for staffing data. Today, `ops::delete` physically deletes a parquet object — fine for whole datasets, useless for "delete one candidate's record." To delete one row we'd have to rewrite the whole `candidates.parquet` which at 100K rows is 10MB of churn per deletion.
+
+llms3.com lists "deletion vectors" as a core lakehouse pattern (Iceberg/Delta/Hudi all implement it). This is the single biggest compliance gap in the current system.
+
+### Scope
+
+1. New sidecar per dataset: `{dataset}_tombstones.parquet` with columns `{row_key, deleted_at, actor, reason}`
+2. Delete API: `POST /catalog/datasets/{name}/tombstone` with `{row_keys[], reason, actor}`
+3. Query-time filter: `queryd` automatically LEFT JOINs tombstones and filters out deleted rows
+4. Compaction integration: Phase 8 compaction reads base + delta + tombstones, writes a clean base without tombstoned rows, clears the tombstone sidecar
+5. Event journal integration (Phase 9): every tombstone emits a journal event with full context
+
+### Measured success
+
+- `POST /catalog/datasets/candidates/tombstone` with `{row_keys: ["CAND-123"], reason: "GDPR request", actor: "legal@company"}`
+- `SELECT COUNT(*) FROM candidates` drops by 1 immediately
+- `SELECT * FROM candidates WHERE candidate_id = 'CAND-123'` returns empty
+- `GET /journal/history/CAND-123` shows the tombstone event
+- After scheduled compaction, the tombstone is materialized — `candidates.parquet` no longer contains CAND-123, tombstone sidecar is emptied for that row key
+
+### Dependencies
+
+- Phase 8 delta/merge-on-read pattern (done) — tombstones are a third layer at read time
+- Phase 9 event journal (done) — tombstones emit journal events
+
+### Rollback
+
+If query rewrite becomes too complex, fallback: tombstones stored but applied only during compaction (not at query time). Queries return deleted rows until compaction runs. Less useful but safer.
+
+---
+
+## Cross-phase concerns
+
+### Phases that need federation layer 2 (task #5)
+
+Every phase above assumes the federation foundation (shipped 2026-04-16) but NOT federation layer 2 (cross-bucket SQL, profile activation, `X-Lakehouse-Bucket` header).
+
+**Implication:** Phases B-E can proceed on the `primary` bucket without blocking on federation layer 2. Federation layer 2 becomes valuable when we want multi-profile model scoping (Phase 17). Sequence:
+
+```
+A (done) → B → C (+D in parallel) → federation layer 2 → Phase 16 → Phase 17 → E
+```
+
+### Phases that need federation layer 2 FIRST
+
+None of B/C/D/E strictly need it. Phase 16 (hot-swap) benefits from it. Phase 17 (model profiles) depends on it heavily.
+
+### What NOT to build in B-E
+
+- Distributed query — wait for a real scale problem
+- Replacement of DataFusion — working fine, stay put
+- Iceberg/Delta Lake migration — explicitly out of scope per ADR-009
+- Live streaming / CDC — explicit non-goal
+
+### Definition of done for each phase
+
+Each phase completes when:
+1. Code shipped and building clean
+2. Success gate measurably passed
+3. Relevant ADR added to `docs/DECISIONS.md` (or updated)
+4. `docs/PHASES.md` checkbox flipped with measurement data
+5. PRD invariants checked — if a new invariant emerged, add it
+6. One regression test in the crate or HTTP integration test
+
+---
+
+## Session plan (what to do in what order)
+
+### Next session
+1. **Phase B — Lance pilot.** Single session. Answers the biggest open architectural question.
+2. Based on outcome, **write ADR-019** with the decision + data.
+3. **Update PHASES.md** with Phase 18 status (Lance evaluated).
+
+### Session after
+4. **Phase C — Decoupled embedding refresh.** Implementation shaped by B's outcome (append is easy on Lance, requires delta logic on Parquet).
+
+### Session after that
+5. **Federation layer 2** OR **Phase D (AI-safe views)** — J decides based on priority. Federation layer 2 unlocks model profiles (Phase 17); AI-safe views is standalone value.
+
+### Final session for this track
+6. **Phase E — Soft deletes.** The compliance-driven phase. Fits cleanly after everything else because it touches the query path and wants to be built after query optimizations stabilize.
+
+### Milestone checkpoint
+After Phase E, stop and reassess. We'll have:
+- Lance decision made and committed
+- Decoupled embedding pipeline
+- AI-safe view enforcement
+- Soft delete semantics
+
+That's a substantial capability increase. Plausible "pause, write a retrospective, decide on Phase 16/17" moment.
@@ -0,0 +1,358 @@
+# Phase Tracker
+
+## Phase 0: Bootstrap ✅
+- [x] Cargo workspace with all crate stubs compiling
+- [x] `shared` crate: error types, ObjectRef, DatasetId
+- [x] `gateway` with Axum: GET /health → 200
+- [x] tracing + tracing-subscriber wired in gateway
+- [x] justfile with build, test, run recipes
+- [x] docs committed to git
+
+## Phase 1: Storage + Catalog ✅
+- [x] storaged: object_store backend init (LocalFileSystem)
+- [x] storaged: Axum endpoints (PUT/GET/DELETE/LIST)
+- [x] shared/arrow_helpers.rs: RecordBatch ↔ Parquet + schema fingerprinting
+- [x] catalogd/registry.rs: in-memory index + manifest persistence
+- [x] catalogd service: POST/GET /datasets + by-name
+- [x] gateway routes wired
+
+## Phase 2: Query Engine ✅
+- [x] queryd: SessionContext + object_store config
+- [x] queryd: ListingTable from catalog ObjectRefs
+- [x] queryd service: POST /query/sql → JSON
+- [x] queryd → catalogd wiring
+- [x] gateway routes /query
+
+## Phase 3: AI Integration ✅
+- [x] Python sidecar: FastAPI + Ollama (embed/generate/rerank)
+- [x] Dockerfile for sidecar
+- [x] aibridge/client.rs: HTTP client
+- [x] aibridge service: Axum proxy endpoints
+- [x] Model config via env vars
+
+## Phase 4: Frontend ✅
+- [x] Dioxus scaffold, WASM build
+- [x] Ask tab: natural language → AI SQL → results
+- [x] Explore tab: dataset browser + AI summary
+- [x] SQL tab: raw DataFusion editor
+- [x] System tab: health checks for all services
+
+## Phase 5: Hardening ✅
+- [x] Proto definitions (lakehouse.proto)
+- [x] Internal gRPC: CatalogService on :3101
+- [x] OpenTelemetry tracing: stdout exporter
+- [x] Auth middleware: X-API-Key (toggleable)
+- [x] Config-driven startup: lakehouse.toml
+
+## Phase 6: Ingest Pipeline ✅
+- [x] CSV ingest with auto schema detection
+- [x] JSON ingest (array + newline-delimited, nested flattening)
+- [x] PDF text extraction (lopdf)
+- [x] Text/SMS file ingest
+- [x] Content hash dedup (SHA-256)
+- [x] POST /ingest/file multipart upload
+- [x] 12 unit tests
+
+## Phase 7: Vector Index + RAG ✅
+- [x] chunker: configurable size + overlap, sentence-boundary aware
+- [x] store: embeddings as Parquet (binary f32 vectors)
+- [x] search: brute-force cosine similarity
+- [x] rag: embed → search → retrieve → LLM answer with citations
+- [x] POST /vectors/index, /search, /rag
+- [x] Background job system with progress tracking
+- [x] Dual-pipeline supervisor with checkpointing + retry
+- [x] 100K embeddings: 177/sec on A4000, zero failures
+- [x] 6 unit tests
+
+## Phase 8: Hot Cache + Incremental Updates ✅
+- [x] MemTable hot cache: LRU, configurable max (16GB)
+- [x] POST /query/cache/pin, /cache/evict, GET /cache/stats
+- [x] Delta store: append-only delta Parquet files
+- [x] Merge-on-read: queries combine base + deltas
+- [x] Compaction: POST /query/compact
+- [x] Benchmarked: 9.8x speedup (1M rows: 942ms → 96ms)
+
+## Phase 8.5: Agent Workspaces ✅
+- [x] WorkspaceManager with daily/weekly/monthly/pinned tiers
+- [x] Saved searches, shortlists, activity logs per workspace
+- [x] Instant zero-copy handoff between agents
+- [x] Persistence to object storage, rebuild on startup
+
+## Phase 9: Event Journal ✅
+- [x] journald crate: append-only mutation log
+- [x] Event schema: entity, field, old/new value, actor, source, workspace
+- [x] In-memory buffer with auto-flush to Parquet
+- [x] GET /journal/history/{entity_id}, /recent, /stats
+- [x] POST /journal/event, /update, /flush
+
+## Phase 10: Rich Catalog v2 ✅
+- [x] DatasetManifest: description, owner, sensitivity, columns, lineage, freshness, tags
+- [x] PII auto-detection: email, phone, SSN, salary, address, medical
+- [x] Column-level metadata with sensitivity flags
+- [x] Lineage tracking: source_system → ingest_job → dataset
+- [x] PATCH /catalog/datasets/by-name/{name}/metadata
+- [x] Backward compatible (serde default)
+
+## Phase 11: Embedding Versioning ✅
+- [x] IndexRegistry: model_name, model_version, dimensions per index
+- [x] Index metadata persisted as JSON, rebuilt on startup
+- [x] GET /vectors/indexes — list all (filter by source/model)
+- [x] GET /vectors/indexes/{name} — metadata
+- [x] Background jobs auto-register metadata on completion
+
+## Phase 12: Tool Registry ✅
+- [x] 6 built-in staffing tools (search_candidates, get_candidate, revenue_by_client, recruiter_performance, cold_leads, open_jobs)
+- [x] Parameter validation + SQL template substitution
+- [x] Permission levels: read / write / admin
+- [x] Full audit trail per invocation
+- [x] GET /tools, GET /tools/{name}, POST /tools/{name}/call, GET /tools/audit
+
+## Phase 13: Security & Access Control ✅
+- [x] Role-based access: admin, recruiter, analyst, agent
+- [x] Field-level sensitivity enforcement
+- [x] Column masking determination per agent
+- [x] Query audit logging
+- [x] GET/POST /access/roles, GET /access/audit, POST /access/check
+
+## Phase 14: Schema Evolution ✅
+- [x] Schema diff detection (added, removed, type changed, renamed)
+- [x] Fuzzy rename detection (shared word parts)
+- [x] Auto-generated migration rules with confidence scores
+- [x] AI migration prompt builder for complex cases
+- [x] 5 unit tests
+
+## Phase 15+: Horizon
+- [x] HNSW vector index with iteration-friendly trial system (2026-04-16)
+  - `HnswStore.build_index_with_config` — parameterized ef_construction, ef_search, seed
+  - `EmbeddingCache` — pins 100K vectors in memory, shared across trials
+  - `harness::EvalSet` — named query sets with brute-force ground truth
+  - `TrialJournal` — append-only JSONL at `_hnsw_trials/{index}.jsonl`
+  - Endpoints: `/vectors/hnsw/trial`, `/hnsw/trials/{idx}`, `/hnsw/trials/{idx}/best?metric={recall|latency|pareto}`, `/hnsw/evals`, `/hnsw/evals/{name}/autogen`, `/hnsw/cache/stats`
+  - Measured on 100K resumes: **brute-force 44-54ms → HNSW 509us-1830us**, recall 0.92-1.00 depending on `ef_construction`. Sweet spot: ec=80 es=30 → p50=873us recall=1.00 — locked in as `HnswConfig::default()`
+- [x] Catalog manifest repair — `POST /catalog/resync-missing` restores row_count and columns from parquet footers (2026-04-16). All 7 staffing tables recovered to PRD-matching 2.47M rows.
+- [~] Federated multi-bucket query — **foundation complete 2026-04-16**, see ADR-017
+  - [x] `StorageConfig.buckets` + `rescue_bucket` + `profile_root` config shape
+  - [x] `SecretsProvider` trait + `FileSecretsProvider` (reads /etc/lakehouse/secrets.toml, checks 0600 perms)
+  - [x] `storaged::BucketRegistry` — multi-backend, rescue-aware, reachability probes
+  - [x] `storaged::error_journal::ErrorJournal` — append-only JSONL at `primary://_errors/bucket_errors.jsonl`
+  - [x] Endpoints: `GET /storage/buckets`, `GET /storage/errors`, `GET /storage/bucket-health`
+  - [x] Bucket-aware I/O: `PUT/GET /storage/buckets/{bucket}/objects/{*key}` with rescue fallback + `X-Lakehouse-Rescue-Used` observability headers
+  - [x] Backward compat: empty `[[storage.buckets]]` synthesizes a `primary` from legacy `root`
+  - [x] Three-bucket test (primary + rescue + testing) verified: normal reads, rescue fallback with headers, hard-fail missing, write to unknown bucket 503, error journal + health summary
+  - [x] `X-Lakehouse-Bucket` header middleware on ingest endpoints (2026-04-16)
+  - [x] Catalog migration: `POST /catalog/migrate-buckets` stamps `bucket = "primary"` on legacy refs (12 renamed, 14 total now canonical)
+  - [x] `queryd` registers every bucket with DataFusion for cross-bucket SQL — verified with people_test (testing) × animals (primary) CROSS JOIN
+  - [x] Profile hot-load endpoints: bucket auto-provisioning on `POST /vectors/profile/{id}/activate` (2026-04-17)
+  - [x] `vectord` bucket-scoped paths: TrialJournal + PromotionRegistry resolve per-index via IndexMeta.bucket (2026-04-17)
+  - [x] Runtime bucket lifecycle: `POST /storage/buckets` (provision) + `DELETE /storage/buckets/{name}` (unregister, refuses primary/rescue) (2026-04-17)
+  - [x] ModelProfile.bucket field — per-profile artifact isolation (2026-04-17)
+- [x] Database connector ingest (Postgres first) — 2026-04-16
+  - `pg_stream::stream_table_to_parquet` — ORDER BY + LIMIT/OFFSET pagination, configurable batch_size
+  - `parse_dsn` — postgresql:// and postgres:// URL scheme, user/password/host/port/db
+  - `POST /ingest/db` endpoint: `{dsn, table, dataset_name?, batch_size?, order_by?, limit?}` → streams to Parquet, registers in catalog with PII detection + redacted-password lineage
+  - Existing `POST /ingest/postgres/import` (structured config) preserved alongside
+  - 4 DSN-parser unit tests + live end-to-end test against `knowledge_base.team_runs` (586 rows, 13 cols, 6 batches, 196ms)
+- [x] Phase B: Lance storage evaluation — 2026-04-16
+  - `crates/lance-bench` standalone pilot (Lance 4.0) avoids DataFusion/Arrow version conflict with main stack
+  - 8-dimension benchmark on resumes_100k_v2 — see docs/ADR-019-vector-storage.md for scorecard
+  - Decision: hybrid architecture. Parquet+HNSW stays primary (2.55× faster search at 100K in-RAM). Lance added as per-profile second backend for random access (112× faster), append (0.08s vs full rewrite), hot-swap (14× faster index builds), and scale past 5M RAM ceiling.
+- [x] Phase E.2 — Compaction integrates tombstones (physical deletion) — 2026-04-16
+  - `delta::compact` accepts `tombstones: &[Tombstone]` param, filters rows at merge time via arrow `filter_record_batch`
+  - CompactResult gains `tombstones_applied` + `rows_dropped_by_tombstones`
+  - Atomic write: ArrowWriter → single Parquet file (fixes latent bug where concatenated Parquet byte streams produced garbage — footer-only-first-segment visible), verify-parse before overwrite, temp_key staging, delete delta files AFTER base write succeeds
+  - Snappy compression on output matches ingest defaults (avoids 3× size inflation on every compact)
+  - `TombstoneStore::clear` drops all batch files for a dataset; called by queryd after successful compact
+  - Query engine exposes `catalog()` accessor so service handler can reach the tombstone store
+  - E2E verified on candidates (100K rows): tombstone 3 IDs → compact → 99,997 rows physically in parquet, tombstones empty, IDs gone from `__raw__candidates` too; file size 10.59 MB → 10.72 MB (proportional to data, not inflated)
+- [x] Phase 16: Hot-swap generations + autotune agent — 2026-04-16
+  - `vectord::promotion::PromotionRegistry` — per-index current config + history at `_hnsw_promotions/{index}.json`, cap 50 history entries
+  - Endpoints: `POST /vectors/hnsw/promote/{index}/{trial_id}`, `POST /vectors/hnsw/rollback/{index}`, `GET /vectors/hnsw/promoted/{index}`
+  - `vectord::autotune::run_autotune` — grid of trials (configurable or default 5 configs), Pareto winner selection (max recall, then min p50), min_recall safety gate (default 0.9), config bounds (ec ∈ [10,400], es ∈ [10,200])
+  - `POST /vectors/hnsw/autotune` — runs the full loop synchronously, journals every trial, auto-promotes winner
+  - `activate_profile` uses `promotion_registry.config_or(..., profile_default)` so newly-promoted configs flow automatically into next activation
+  - End-to-end: autogen harness for threat_intel_v1 (10 queries), autotune ran 5 trials (all recall=1.00, p50 64-68us), promoted ec=20 es=30 at recall=1.0 p50=64us as winner. Manual promote of ec=80 es=30 pushed autotune pick onto history. Rollback restored autotune winner. Second rollback cleared to None. Re-promote + restart verified persistence. Activation after promotion logged "building HNSW ef_construction=80 ef_search=30 seed=42" — config flowed through correctly.
+- [x] Phase 17: Model profiles + scoped search — 2026-04-16
+  - `shared::types::ModelProfile` — { id, ollama_name, description, bound_datasets, hnsw_config, embed_model, created_at, created_by }
+  - `shared::types::ProfileHnswConfig` — mirror of vectord's HnswConfig to avoid cross-crate dep cycle (defaults ec=80 es=30 matching Phase 15 winner)
+  - `Registry::{put_profile, get_profile, list_profiles, delete_profile}` persisted at `_catalog/profiles/{id}.json`, validates bindings exist (raw dataset OR AiView)
+  - Endpoints: `POST/GET /catalog/profiles`, `GET/DELETE /catalog/profiles/{id}`
+  - `POST /vectors/profile/{id}/activate` — warms EmbeddingCache + builds HNSW with profile's config for every bound dataset's vector index; reports warmed indexes + failures + duration
+  - `POST /vectors/profile/{id}/search` — rejects 403 if requested index's source isn't in profile.bound_datasets; falls through to HNSW if warm, brute-force otherwise
+  - Fixed refresh to register new index metadata (was silently no-op for first-time indexes)
+  - End-to-end: security-analyst profile bound to threat_intel → activate warms 54 vectors in 156ms → within-scope HNSW search works (0.625 score); out-of-scope search for candidates returns 403 with allowed bindings listed
+- [x] Phase E: Soft deletes (tombstones) — 2026-04-16
+  - `shared::types::Tombstone` — { dataset, row_key_column, row_key_value, deleted_at, actor, reason }
+  - `catalogd::tombstones::TombstoneStore` per-dataset append-log at `_catalog/tombstones/{dataset}/`, flush_threshold=1 + explicit flush so every tombstone is durable on return (compliance requirement)
+  - All tombstones for a dataset must share the same `row_key_column` (validated at write — query filter is built as a single WHERE NOT IN clause)
+  - `Registry::add_tombstone / list_tombstones`
+  - Endpoint: `POST /catalog/datasets/by-name/{name}/tombstone` accepting `{row_key_column, row_key_values[], actor, reason}`; companion `GET` lists active tombstones
+  - `queryd::context::build_context` wraps tombstoned tables: raw goes to `__raw__{name}`, public name becomes a DataFusion view with `WHERE CAST(col AS VARCHAR) NOT IN (...)` filter
+  - End-to-end on candidates: tombstone 3 IDs, COUNT drops 100,000 → 99,997, specific WHERE returns empty, AiView candidates_safe transitively excludes them too, restart preserves all tombstones
+  - Limits / not in MVP: journal integration (tombstones don't yet emit Phase 9 mutation events — covered by audit fields on the tombstone itself). Physical compaction integration was closed by Phase E.2 below.
+- [x] Phase D: AI-safe views — 2026-04-16
+  - `shared::types::AiView` — name, base_dataset, columns whitelist, optional row_filter, column_redactions
+  - `shared::types::Redaction` — Null | Hash | Mask { keep_prefix, keep_suffix }
+  - `Registry::put_view / get_view / list_views / delete_view` persisted to `_catalog/views/{name}.json`
+  - `queryd::context` registers each view as a DataFusion view with the safe projection + filter + redactions baked into the SELECT
+  - Endpoints: `POST/GET /catalog/views`, `GET/DELETE /catalog/views/{name}`
+  - End-to-end on candidates: `candidates_safe` view exposes 8 of 15 columns, masks `candidate_id` (CAN******01), filters out `status='blocked'`. `SELECT * FROM candidates_safe` returns whitelist only; `SELECT email FROM candidates_safe` fails. View survives restart.
+  - Capability surface — raw `candidates` still accessible by name; Phase 13 access control is the layer that enforces who can query what
+- [x] Phase C: Decoupled embedding refresh — 2026-04-16
+  - `DatasetManifest`: `last_embedded_at`, `embedding_stale_since`, `embedding_refresh_policy` (Manual | OnAppend | Scheduled)
+  - `Registry::mark_embeddings_stale` / `clear_embeddings_stale` / `stale_datasets`
+  - Ingest paths (CSV pipeline + Postgres streaming) auto-mark-stale when writing to an already-embedded dataset
+  - `vectord::refresh::refresh_index` — reads dataset, diffs doc_ids vs existing embeddings, embeds only new rows, writes combined index, clears stale
+  - `POST /vectors/refresh/{dataset}` + `GET /vectors/stale`
+  - Id columns accept `Utf8`, `Int32`, `Int64`
+  - End-to-end on threat_intel: initial 20-row embed 2.1s; re-ingest to 54 rows auto-marks stale; delta refresh embeds only 34 new in 970ms (6× faster than full re-embed); stale cleared
+- [x] Phase 16.2/16.5: Background autotune agent + ingest-triggered re-trials — 2026-04-17
+  - `vectord::agent` — ε-greedy proposer, rate-limited, cooldown-gated, tokio background task
+  - Ingest paths push `DatasetAppended` triggers to agent queue
+  - Endpoints: `GET /vectors/agent/status`, `POST /vectors/agent/stop`, `POST /vectors/agent/enqueue/{idx}`
+  - `[agent]` config section in lakehouse.toml (enabled, cycle_interval, cooldown, min_recall, max_trials/hr)
+  - 3 unit tests
+- [x] Phase 17 VRAM gate: Two-profile sequential swap — 2026-04-17
+  - Sidecar: `POST /admin/unload` (keep_alive=0), `POST /admin/preload`, `GET /admin/vram` (nvidia-smi + Ollama /api/ps)
+  - `AiClient::unload_model / preload_model / vram_snapshot`
+  - `VectorState.active_profile` singleton — activate swaps models, deactivate unloads
+  - Verified: staffing-recruiter (qwen2.5) ↔ docs-assistant (mistral) — only one model in VRAM at a time
+- [x] MySQL streaming connector — 2026-04-17
+  - `my_stream.rs` mirrors pg_stream: DSN parsing, OFFSET pagination, Arrow type mapping, Parquet streaming
+  - `POST /ingest/mysql` with PII detection, lineage, agent trigger
+  - Verified end-to-end on live MariaDB (10 rows, 9 columns, round-tripped all types)
+  - 6 DSN + type-mapping unit tests
+- [x] Phase 18 hybrid: vectord-lance production crate — 2026-04-17
+  - Firewall crate (Arrow 57 / Lance 4, separate from main Arrow 55 / DF 47 stack)
+  - Public API: migrate_from_parquet, build_index (IVF_PQ), search, get_by_doc_id, append, build_scalar_index, stats
+  - `lance_backend::LanceRegistry` resolves bucket → URI per index
+  - `VectorBackend { Parquet | Lance }` enum on ModelProfile + IndexMeta
+  - 8 HTTP endpoints under `/vectors/lance/*` (migrate, index, search, doc, append, stats, scalar-index, recall)
+  - Profile-driven routing: `POST /vectors/profile/{id}/search` auto-routes to Lance when profile.vector_backend=lance
+  - Auto-migrate + auto-index on activation
+  - Measured on real 100K × 768d: migrate 0.57s, IVF_PQ build 16.2s (14× faster than HNSW 230s), search 23ms, append 100 rows 3.3ms, doc_id fetch 3.5ms (with scalar btree)
+  - IVF_PQ recall@10 = 0.805 with Lance's default `nprobes=1` (the hidden cap — see 2026-04-20 tuning work below, which lifts it to 1.000). Measured via `/vectors/lance/recall/{idx}` harness.
+- [x] Phase E.3: Scheduled ingest — 2026-04-17
+  - `ingestd::schedule` module: ScheduleDef, ScheduleStore (JSON at `_schedules/{id}.json`), Scheduler tokio task
+  - Supports MySQL + Postgres sources on interval triggers (Cron variant defined, parsing stubbed)
+  - 6 CRUD endpoints under `/ingest/schedules/*` + run-now manual trigger
+  - Full catalog integration: PII, lineage, mark-stale, agent trigger
+  - 6 unit tests
+- [x] PDF OCR via Tesseract — 2026-04-17
+  - Two-tier: lopdf text extraction → Tesseract 5.5 fallback for scanned/image PDFs
+  - Extracts embedded XObject /Image streams, shells to tesseract --oem 3 --psm 6
+  - Same schema (source_file, page_number, text_content) — downstream unchanged
+- [x] Catalog hygiene — idempotent `register()` + dedupe + DELETE (2026-04-19, ADR-020)
+  - `catalogd::Registry::register` now gates on `(name, schema_fingerprint)`: same fp → reuse `DatasetId` and update objects in place; different fp → return error (409 Conflict on HTTP, `FAILED_PRECONDITION` on gRPC). First-time registration is unchanged.
+  - `POST /catalog/dedupe` one-shot operator endpoint collapses pre-existing duplicates; winner = non-null `row_count` first, newest `updated_at` second.
+  - `DELETE /catalog/datasets/by-name/{name}` removes the manifest from both in-memory registry and object storage (metadata-only — parquet files, vector indexes, tombstones are NOT cascade-deleted). Added to support test-harness cleanup; also plugs a real catalog hole where zombie entries from prior deletes would break DataFusion schema inference.
+  - Cleanup run on live catalog: 374 → 31 datasets, 343 orphan manifests removed, 0 errors. 308× `successful_playbooks` was the worst offender.
+  - Concurrency: write lock held across storage I/O in `register()` to close the check→insert TOCTOU window (32-worker multi-threaded stress test verifies single-manifest invariant).
+  - End-to-end verification: `scripts/e2e_pipeline_check.sh` runs 31 assertions across 12 pipeline stages (ingest → catalog → SQL+JOIN → dedup → idempotency → metadata → PII → vector embed → semantic search → cleanup) against the live gateway. Idempotent across repeat runs.
+  - Tests: 11 new in `catalogd` (was 0, includes 3 concurrency tests + 3 delete_dataset tests); 11 new in `storaged` for `AppendLog` + `ErrorJournal` (was 0). Fixed a broken doctest in `append_log.rs`.
+- [x] Autotune agent: portfolio rotation + auto-bootstrap (2026-04-20)
+  - `pick_periodic_target` now sources candidates from `IndexRegistry` (not just promoted indexes) and picks least-recently-tuned, so trial budget spreads across every index with ≥1000 vectors instead of fixating on one converged champion.
+  - `run_one_cycle` bootstraps on first visit: `ensure_auto_harness` auto-generates `{index}_auto` (20 synthetic self-queries, k=10, brute-force ground truth) if missing, then seeds with `HnswConfig::default()` (ec=80/es=30).
+  - Regression fix: `harness::recall_at_k` now uses set-intersection semantics. The prior impl counted duplicates in `predicted` — on corpora with repeated chunks (`kb_response_cache_agent`) this inflated recall above 1.0 and poisoned promotion decisions. +7 unit tests.
+- [x] Scheduled ingest: real cron parsing (2026-04-20)
+  - Vixie-compatible 5/6-field cron via `croner` crate. Day-of-week follows Unix convention (`1-5` = Mon-Fri). 6-field adds seconds granularity.
+  - `validate_trigger` in `ingestd::schedule` — create/patch handlers reject malformed expressions with `400 BAD_REQUEST` at creation time, not silently at fire time.
+  - Swapped away from the `cron` crate (0.16) which uses a non-Unix DOW convention (`1=Sun`) that would silently bite anyone writing `1-5` expecting weekdays. +9 unit tests.
+- [x] EvalSets federation (2026-04-20)
+  - `harness::HarnessStore` mirrors the TrialJournal / PromotionRegistry federation pattern: eval artifacts colocate with each index's recorded bucket; legacy evals in primary remain discoverable via a fallback path; cross-bucket listing dedupes.
+  - Every eval callsite (service.rs × 5, agent.rs × 3, autotune.rs × 1) now routes through `HarnessStore`. `VectorState` and `AgentDeps` each hold a shared instance.
+- [x] Index bucket-migrate PATCH (2026-04-20)
+  - `PATCH /vectors/indexes/{name}/bucket` copies an index's vector parquet + trial-journal batches + promotion file + auto-harness to `dest_bucket`, flips `IndexMeta.bucket` as the commit point, and evicts the `EmbeddingCache` so next load reads from the new bucket. Optional `delete_source: true` sweeps source artifacts.
+  - Lance-backed indexes refused with 400 — Lance URIs are bucket-specific and require rewriting the dataset, separate story. Round-trip verified: 390 artifacts, 0.04s.
+- [x] IVF_PQ recall tuning (2026-04-20)
+  - `LanceVectorStore::search` now accepts optional `nprobes` + `refine_factor`. Lance's built-in `nprobes=1` default was the hidden cap on recall — on 316-partition `resumes_100k_v2` it searched only 0.3% of partitions per query.
+  - Server defaults (`LANCE_DEFAULT_NPROBES=20`, `LANCE_DEFAULT_REFINE_FACTOR=5`) flow through the scoped-search path and the autotune harness. Measured on `resumes_100k_v2`: recall `0.805 → 1.000` at p50 ≈ 7.4ms. Even `nprobes=5, refine=5` saturates recall at p50 ≈ 4.7ms.
+  - `/vectors/lance/recall/{idx}` accepts per-request `nprobes` / `refine_factor` so operators can sweep the curve.
+- [x] **Phase 19: Playbook memory (meta-index)** — the feedback loop originally implied by the PRD but never built. Playbooks stop being write-only; they start shaping future rankings. (2026-04-20)
+  - [x] 19.1 — `POST /vectors/playbook_memory/rebuild` scans `successful_playbooks` via DataFusion, builds one `PlaybookEntry` per row (operation + approach + context embedded as one vector via nomic-embed-text)
+  - [x] 19.2 — Brute-force cosine search over in-memory embeddings (chosen over HNSW: successful_playbooks maxes around thousands of rows, overhead of a second indexed surface isn't worth it until that ceiling bites)
+  - [x] 19.3 — Endorsed names parsed out of `result` column, keyed by `(city, state, name)` tuple so shared names across cities don't cross-pollinate. Parsing via `parse_names` + `parse_city_state` helpers (7 unit tests)
+  - [x] 19.4 — `/vectors/hybrid?use_playbook_memory=true`: fetches `top_k * 5` candidates so endorsed workers outside the vanilla top-K can still climb. Boost is additive on vector score, each hit carries `playbook_boost` + `playbook_citations` in the response for explainability
+  - [x] 19.5 — Multi-agent orchestrator (`tests/multi-agent/orchestrator.ts`) auto-seeds `POST /vectors/playbook_memory/seed` on consensus_done, so the next query sees the new endorsement without a full `/rebuild`. Closes the feedback loop: two agents reach consensus → playbook sealed → next query re-ranks
+  - [x] 19.6 — `MAX_BOOST_PER_WORKER = 0.25` enforced in `compute_boost_for`; verified with unit test (100 identical playbooks → boost capped at 0.25) and live test (5 identical seeds → exactly 0.25). Time decay also wired: `BOOST_HALF_LIFE_DAYS = 30.0` — 30-day-old playbooks contribute half, 60-day a quarter, via `exp(-age_days / 30)` in the boost loop
+  - Real finding surfaced during build: the 32 bootstrap rows in `successful_playbooks` reference phantom worker names — 80 of 82 don't correspond to actual rows in `workers_500k`. `/seed` endpoint bypasses `successful_playbooks` so operators can prime memory with real fixtures; production path is the orchestrator write-through
+- [x] **Phase 19 refinement — geo + role prefilter on boost** (2026-04-21)
+  - Added `compute_boost_for_filtered` and `compute_boost_for_filtered_with_role` to `playbook_memory.rs`. SQL filter's `(city, state, role)` parsed in `service.rs`; exact role-matches in target geo skip cosine and earn similarity=1.0. Restored the feedback loop: matched=0 → matched=11 per query on the same Nashville test. Citation density on Riverfront Steel: 2 → 28 per run (14×).
+  - Rust unit tests: `extractor_tests::extract_target_geo_basic/_missing_state/_word_boundary`, `extract_target_role_basic/_none/_multi_word`. 6/6 pass.
+  - Diagnostic log: `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every call.
+- [x] **Phase 20: Model Matrix + Overseer Tiers** (2026-04-21)
+  - `config/models.json` — 5 tiers (t1_hot / t2_review / t3_overview / t4_strategic / t5_gatekeeper), each with context_window + context_budget + overflow_policy. Ollama Cloud bearer key from `/root/llm_team_config.json`.
+  - Hot path: qwen3.5:latest + qwen3:latest local with `think:false`. Mistral dropped after 0/14 fill on complex scenarios.
+  - T3 cloud: gpt-oss:120b via Ollama Cloud — verified 4-8s latency, strict JSON-shape output for remediation.
+- [x] **Phase 21: Scratchpad + Tree-Split Continuation** (2026-04-21)
+  - `tests/multi-agent/agent.ts`: `estimateTokens()`, `assertContextBudget()`, `generateContinuable()`, `generateTreeSplit()`. `think` flag plumbed through sidecar's `/generate`. Empty-response backoff + truncation-continuation, no max_tokens tourniquet.
+  - Rust port shipped (2026-04-21, companion to Phase 27):
+    - `crates/aibridge/src/context.rs` — `estimate_tokens` (chars/4 ceil, matches TS), `context_window_for`, `assert_context_budget` returning `Result<BudgetCheck, (BudgetCheck, usize over_by)>` so callers get the numbers back on both success and overflow. Windows table mirrors `config/models.json`.
+    - `crates/aibridge/src/continuation.rs` — `generate_continuable<G: TextGenerator>` handles the two failure modes from TS: (a) empty thinking-model response → geometric-backoff retry with 2× budget up to `budget_cap`; (b) truncated non-empty → continuation with partial-as-scratchpad. `is_structurally_complete` balances braces then JSON.parse-check for the JSON shape; text shape is "non-empty". Guards the degen case "all retries empty → bail, don't loop on empty partial" — the TS impl has this implicit, Rust makes it explicit.
+    - `crates/aibridge/src/tree_split.rs` — `generate_tree_split` map→reduce with running scratchpad. Per-shard + reduce-prompt budget checked through `assert_context_budget`; loud-fails with the overflow message rather than silently truncating. Scratchpad oldest-digest-first truncation once it exceeds `scratchpad_budget` (default 6000 tokens, matches TS).
+    - `TextGenerator` trait (native async-fn-in-trait, edition 2024) so `ScriptedGenerator` test double can inject canned sequences without a live Ollama. `AiClient` implements `TextGenerator`.
+    - `GenerateRequest` gained `think: Option<bool>` field — forwards to sidecar for per-call hidden-reasoning opt-out on hot-path JSON emitters.
+    - 25 aibridge unit tests (8 context + 10 continuation + 7 tree_split) — all green, no network required.
+- [x] **Phase 22: Internal Knowledge Library** (2026-04-21)
+  - `data/_kb/` — signatures.jsonl, outcomes.jsonl, pathway_recommendations.jsonl, error_corrections.jsonl, config_snapshots.jsonl. Event-driven cycle: indexRun → recommendFor → loadRecommendation.
+  - Item B cloud rescue: failed event → cloud remediation JSON → retry with pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot).
+  - `scripts/kb_measure.py` aggregator. Unit tests: `kb.test.ts` — 4/4 pass (signature stability, role/city/count invariants, digest shape).
+- [x] **Phase 23: Staffer identity + competence-weighted retrieval** (2026-04-21)
+  - ScenarioSpec gained `contract: ContractTerms` and `staffer: Staffer { id, name, tenure_months, role, tool_level }`.
+  - tool_level runtime overrides: full / local / basic / minimal. Basic + minimal route executor to Ollama Cloud `kimi-k2.5` (kimi-k2.6 pending pro-tier upgrade).
+  - `data/_kb/staffers.jsonl` — competence_score = 0.45·fill + 0.20·turn_eff + 0.20·cite + 0.15·rescue. Recomputed per run.
+  - `findNeighbors` now returns `weighted_score = cosine × max_staffer_competence`. `scripts/kb_staffer_report.py` — leaderboard + cross-staffer worker overlap (Rachel D. Lewis 12× across 4 staffers → auto-discovered high-value label).
+  - `gen_staffer_demo.ts` + `run_staffer_demo.sh` — 4 personas × 3 contracts = 12 runs.
+- [x] **Phase 27: Playbook versioning** (2026-04-21)
+  - `PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` fields. All `#[serde(default)]` so entries persisted before Phase 27 load as roots with version=1.
+  - `PlaybookMemory::revise_entry(parent_id, new_entry)` appends a new version, stamps `superseded_at`+`superseded_by` on the parent, inherits `parent_id` and sets `version = parent + 1` on the new entry. Rejects revising a retired or already-superseded parent with a clear error — the tip of the chain is the only valid revise target.
+  - `PlaybookMemory::history(playbook_id)` returns the full chain root→tip, walking `parent_id` backward then `superseded_by` forward. Cycle-safe. Works from any node in the chain.
+  - Superseded entries excluded from boost (same rule as retired): `compute_boost_for_filtered_with_role`, the active-entries prefilter, the geo-index rebuild, and the upsert existing-entry search all skip `superseded_at.is_some()`.
+  - Endpoints: `POST /vectors/playbook_memory/revise` + `GET /vectors/playbook_memory/history/{id}`.
+  - `status_counts` now returns `(total, retired, superseded, failures)`. `/status` JSON reports `superseded` as a distinct counter; `active = total - retired - superseded`.
+  - 8 unit tests under `mod version_tests` covering: chain-metadata stamping, retired-parent rejection, already-superseded-parent rejection, superseded endorsement exclusion from boost, history traversal from root/tip/middle, empty-for-unknown-id, superseded-status-count, legacy-entry-default-version round-trip. 26/26 playbook_memory tests pass.
+- [x] **Phase 24: Observer / Autotune integration** (2026-04-20, commit b95dd86)
+  - Closed the gap where `lakehouse-observer.service` wrapped MCP :3700 while `tests/multi-agent/scenario.ts` hit gateway :3100 directly — observer sat idle at 0 ops across 3600+ cycles.
+  - `observer.ts` gained a Bun HTTP listener on `OBSERVER_PORT` (default 3800) with `GET /health`, `GET /stats` (totals + by_source + recent scenario digest), and `POST /event` for scenario outcomes. Body shapes into `ObservedOp` with `source="scenario"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
+  - `recordExternalOp()` shared ring-buffer insert — ERROR_ANALYZER and PLAYBOOK_BUILDER loops now see both MCP-wrapped and scenario-posted ops through the same path.
+  - `persistOp()` swap: old path wrote via `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly Parquet write-through.
+- [x] **Phase 25: Validity windows + playbook retirement** (2026-04-21, commit e0a843d)
+  - `PlaybookEntry` gained four optional fields (`#[serde(default)]` so legacy entries load as never-expiring): `schema_fingerprint` (SHA-256 over target dataset columns at seed time), `valid_until` (RFC3339 hard expiry), `retired_at` (set by retire calls), `retirement_reason` (human string).
+  - `compute_boost_for_filtered_with_role` now skips retired + expired entries before geo/cosine — no silent boosting from stale playbooks. Unit-tested on expired `valid_until` + retired + schema-drift retirement.
+  - Two retirement paths: `retire_one(playbook_id, reason)` for manual, `retire_on_schema_drift(city, state, current_fingerprint, reason)` for batch schema-migration sweep. Legacy entries without a fingerprint skip drift retirement (safe).
+  - Endpoint: `POST /vectors/playbook_memory/retire` — accepts either `{playbook_id, reason}` or `{city, state, current_schema_fingerprint, reason}`.
+- [x] **Phase 26: Mem0 upsert + Letta geo hot cache** (2026-04-21, commit 640db8c)
+  - Mem0-style upsert: `/seed` with `append=true` (default) routes through `upsert_entry`, which decides ADD / UPDATE / NOOP on (operation, day, city, state). Same-day re-seed merges names (union, stable order) instead of duplicating the row. Identical re-seed is a no-op. Different-day same-operation is a fresh ADD. Playbook_id stays stable on UPDATE so prior citations remain valid.
+  - Letta-style hot cache: `PlaybookMemory` now holds a `geo_index: HashMap<(city_lower, state_upper), Vec<entry_idx>>` rebuilt on every mutation. Geo-filtered boost queries skip the full scan and hit the O(1) key lookup. At 1.9K entries the full scan was sub-ms; the index scales the same path to 100K+ without code changes.
+  - `UpsertOutcome` enum reported back to callers — `{mode: added|updated|noop, playbook_id, merged_names?}` + `entries_after`.
+- [x] **Phase 37: Hot-swap async** (2026-04-22)
+  - Extended `JobTracker` with `JobType::ProfileActivation` + `Embed` enum variants
+  - Made `activate_profile` return immediately with `job_id`, work runs in background via `tokio::spawn`
+  - Background jobs tracked via `POST /vectors/jobs/{id}` + `GET /vectors/jobs`
+- [x] **Phase 38: Universal API Skeleton** (2026-04-23)
+  - `/v1/chat` — OpenAI-compatible POST, forwards to local Ollama or Ollama Cloud
+  - `/v1/usage` — returns `{requests, prompt_tokens, completion_tokens, total_tokens}`
+  - `/v1/sessions` — returns `{data: [], note: "Phase 38: stateless"}`
+  - Langfuse trace integration (fire-and-forget, Phase 40 early)
+  - 12 unit tests green, curl gates pass
+- [x] **Phase 39: Provider Adapter Refactor** (2026-04-23)
+  - `ProviderAdapter` trait with `chat()` + `embed()` + `unload()` + `health()`
+  - `OllamaAdapter` — wraps existing AiClient
+  - `OpenRouterAdapter` — HTTP client to openrouter.ai
+  - `provider_key()` routing by model prefix (openrouter/* → OpenRouter)
+- [x] **Phase 40: Routing & Policy Engine** (2026-04-23)
+  - `RoutingEngine` with `RouteDecision` in aibridge::routing
+  - `config/routing.toml` — rules by model_pattern, fallback chain, cost gating
+  - Per-provider usage tracking: Usage.by_provider
+  - 12 gateway tests green, curl gates pass
+- [x] **Phase 41: Profile System Expansion** (2026-04-23)
+  - `ProfileType` enum: Execution, Retrieval, Memory, Observer
+  - Per-type endpoints: `/profiles/retrieval`, `/profiles/memory`, `/profiles/observer`
+  - `profile_type` field on ModelProfile
+  - Guard fix: automated scrumaudit.py finds real issues
+- [ ] Fine-tuned domain models (Phase 25+)
+- [ ] Multi-node query distribution (only if ceilings bite)
+
+---
+
+**145 unit tests | 13 crates | 21 ADRs | 2.47M rows | 100K vectors | Hybrid Parquet+HNSW ⊕ Lance | Phases 0-27 shipped**
+**Latest: 2026-04-21 — Phase 27 (playbook versioning: `version` + `parent_id` + `superseded_at` + `superseded_by` on `PlaybookEntry`, `/revise` + `/history` endpoints, 8 new tests). Doc-sync pass: Phase 24 observer + Phase 25 validity windows + Phase 26 Mem0/Letta now reflected in phase tracker. Phase 19.6 time decay noted as wired (was misdocumented as deferred). Phase E.2 tombstone-at-compaction noted as closed in Phase 8 MVP limits.**
@@ -0,0 +1,900 @@
+# PRD: Lakehouse — Rust-First Substrate for Versioned Knowledge Stores
+
+> ## ▶ Direction pivot — 2026-04-22
+>
+> **`docs/CONTROL_PLANE_PRD.md` is now the long-horizon architecture target.**
+> Lakehouse (this document) is being refactored to conform to the Universal AI
+> Control Plane architecture — six layers, `/v1/*` universal API, multi-provider
+> routing, Truth Layer for DevOps constraints.
+>
+> This PRD remains the source of truth for **everything Phases 0-37** — the
+> shipped staffing / AI-substrate system below is preserved as the reference
+> implementation and the first domain-specific consumer of the control plane.
+> All existing phases, ADRs, and invariants continue to apply.
+>
+> **Phases 38+ live in `CONTROL_PLANE_PRD.md`** (Universal API, provider
+> adapters, routing engine, expanded profile system, Truth Layer, validation
+> pipeline, caller migration).
+>
+> Cross-read: this PRD for what's shipped, CONTROL_PLANE_PRD for where it's going.
+
+---
+
+**Status:** Active — Phases 0-36 shipped, Phase 37 (hot-swap async) TODO; control-plane pivot in flight; E2E test 9.0/10; stress test passes
+**Created:** 2026-03-27
+**Last updated:** 2026-04-22 — Control-plane pivot; Gateway seed concurrency (Semaphore); stress test with 6 diverse tasks
+**Owner:** J
+
+---
+
+## Problem
+
+### Use case 1 — Staffing analytics (reference implementation)
+
+Legacy data systems silo information across CRMs, databases, spreadsheets, and file shares. Querying across them requires manual ETL, pre-defined schemas, and expensive database licenses. When AI enters the picture, these systems can't handle the dual requirement of fast analytical queries AND semantic retrieval over unstructured text.
+
+A staffing company (our reference case) has candidate records in an ATS, client data in a CRM, timesheets in billing software, call logs from a phone system, and email records from Exchange. Answering "find every Java developer in Chicago who was called 5+ times but never placed" requires querying across all of them — and no single system can do it.
+
+### Use case 2 — Local AI knowledge substrate (the second half)
+
+Local LLM workloads need a substrate for ingesting, indexing, and retrieving large knowledge corpora. Each running model (or agent) has its own context — documents it cares about, a vector index tuned to its domain, a scoped view of the catalog. That infrastructure is architecturally identical to the staffing problem: ingest messy data, index it, query it, hand it to an AI. Building one substrate that serves both prevents fragmentation.
+
+Concretely this means a running Ollama model like `qwen2.5:7b` or `claude-code-local` should be able to:
+- Bind to a named set of datasets
+- Get a scoped vector index pre-warmed for its domain
+- Issue searches that only see its bound data
+- Have its trial/tuning history isolated from other models
+- Swap between knowledge generations (today's, yesterday's) without rebuild
+
+The same infrastructure that lets a recruiter query 2.47M rows of staffing data also lets a local 7B model answer questions grounded in a 500K-chunk documentation corpus. Same substrate, different tenant.
+
+### Shared requirements
+
+- Any data source (CSV, DB export, PDF, JSON, Postgres table) can be ingested without pre-defined schemas
+- Structured data is queryable via SQL at scale (millions of rows, sub-second)
+- Unstructured data is searchable via AI embeddings with per-profile indexes
+- An LLM can answer natural language questions against scoped data
+- Indexes can be hot-swapped between generations without rebuild downtime
+- Trials are first-class data — the system remembers how it was tuned
+- Everything runs locally — no cloud APIs, total data privacy
+- The system is rebuildable from repository + object storage alone
+
+---
+
+## Solution
+
+A modular Rust service mesh over S3-compatible object storage, with a local AI layer for embeddings and generation.
+
+### Locked Stack
+
+| Layer | Technology | Locked |
+|---|---|---|
+| Frontend | Dioxus | Yes |
+| API | Axum + Tokio | Yes |
+| Object Storage Interface | Apache Arrow `object_store` | Yes |
+| Storage Backend | LocalFileSystem → RustFS → S3 | Yes |
+| Query Engine | DataFusion | Yes |
+| Data Format | Parquet + Arrow | Yes |
+| RPC (internal) | tonic (gRPC) | Yes |
+| AI Runtime | Ollama (local models) | Yes |
+| AI Boundary | Python FastAPI sidecar → Ollama HTTP API | Yes |
+| Vector Index | TBD — evaluate `hora`, `qdrant` crate, or HNSW from scratch | **Open** |
+
+No new frameworks without documented ADR.
+
+---
+
+## Architecture
+
+### Services
+
+| Service | Responsibility |
+|---|---|
+| **gateway** | HTTP/gRPC ingress, routing, auth, CORS, body limits, `X-Lakehouse-Bucket` header routing |
+| **catalogd** | Metadata control plane — dataset registry, schema versions, manifests, per-dataset resync from parquet footers |
+| **storaged** | Object I/O — `BucketRegistry` (multi-backend), rescue fallback, error journal, append-log batching pattern |
+| **queryd** | SQL execution — DataFusion over Parquet, MemTable hot cache, delta merge-on-read |
+| **ingestd** | Ingest pipeline: CSV / JSON / PDF / Postgres-stream → normalize → Parquet → catalog |
+| **vectord** | Embedding store + vector indexes + HNSW trial system (EmbeddingCache, trial journal, eval harness) |
+| **journald** | Append-only mutation event log (ADR-012) — distinct from storaged error journal |
+| **aibridge** | Rust↔Python boundary — HTTP client to FastAPI sidecar |
+| **ui** | Dioxus frontend — Ask, Explore, SQL, System tabs |
+| **shared** | Types, errors, Arrow helpers, config, protobuf definitions, **secrets provider trait**, **PII detection** |
+| **mcp-server** | Agent gateway (Bun) — MCP tools, intelligence endpoints, scenario observer (:3700) |
+| **observer** | Autonomous iteration loop — records ops, error analysis, playbook consolidation (:3800) |
+| **scenario** | Day-in-the-life orchestrator — T5 tiers, KB integration, continuation tree-split (:bun) |
+
+**Federation building blocks** (shipped 2026-04-16):
+- `shared::secrets::SecretsProvider` trait + `FileSecretsProvider` reading `/etc/lakehouse/secrets.toml` (0600 enforced)
+- `storaged::registry::BucketRegistry` — multi-bucket resolution with `rescue_bucket` read fallback
+- `storaged::append_log::AppendLog` — write-once batched append pattern (no RMW, no small-file problem)
+- `storaged::error_journal::ErrorJournal` — bucket operation failure log at `primary://_errors/bucket_errors/batch_*.jsonl`
+
+### Data Flow
+
+```
+Raw data → ingestd (normalize, chunk, detect schema)
+    ├→ storaged (Parquet files to object storage)
+    ├→ catalogd (register dataset + schema)
+    ├→ vectord (embed text chunks, build index)
+    └→ queryd  (auto-register as queryable table)
+
+User question → gateway
+    ├→ vectord (semantic search for relevant chunks)  ← RAG path
+    ├→ queryd  (SQL over structured data)             ← Analytics path
+    └→ aibridge → Ollama (generate answer from context)
+```
+
+### Query Paths
+
+**Analytical (SQL):** "What's the average bill rate for .NET devs in Chicago?"
+→ DataFusion scans Parquet columnar, returns in <200ms
+
+**Semantic (RAG):** "Find candidates who could do data engineering work"
+→ Embed question → vector search across resume embeddings → retrieve top chunks → LLM answers
+
+**Hybrid (shipped 2026-04-17):** "Find reliable forklift operators in Illinois with OSHA certs"
+→ `POST /vectors/hybrid` with `sql_filter` + `question`: SQL narrows to structurally-valid candidates (role, state, reliability, certs), brute-force cosine ranks by semantic relevance within the filtered set, LLM generates answer from SQL-verified records only. Zero hallucinations on the staffing simulation (16/16 positions filled, all workers verified against golden data).
+
+### Invariants
+
+1. Object storage = source of truth for all data
+2. catalogd = sole metadata authority
+3. No raw data in catalog — only pointers
+4. vectord stores embeddings AS Parquet (portable, not a proprietary format) — see ADR-018 for the Parquet-vs-Lance trade review
+5. ingestd is idempotent — re-ingesting the same file is a no-op
+6. Hot cache is a performance layer, not a source of truth — eviction is safe
+7. All services modular and independently replaceable
+8. **Indexes are hot-swappable.** A new index generation can be built in the background while the current one serves queries. Promotion is atomic (pointer swap). Rollback to a prior generation is always possible. (Phase 16)
+9. **Every reader gets its own profile.** Human operators, AI agents, and local models are all clients of the same substrate. Each has a named profile with its own bucket, vector indexes, trial history, and dataset bindings. Profiles are a first-class architectural concept, not a tenancy afterthought. (Phase 17)
+10. **Trials are data, not logs.** Every index build is a trial with measurable metrics. The trial journal IS the agent's memory for how to tune itself. Stored as write-once batched JSONL per the ADR-018 append-log pattern.
+11. **Operational failures are findable in one HTTP call.** The bucket error journal, trial journal, and audit log all expose `/storage/errors`, `/hnsw/trials`, `/access/audit` with structured filter + aggregation. No `grep` archaeology to answer "what broke?"
+12. **Playbooks feed the index, not just the log.** A completed playbook isn't just a record of what worked — it's a signal that shapes future rankings. Every `successful_playbooks` row contributes to the playbook-memory vector index, so semantically-similar future operations re-rank toward workers that have actually succeeded in comparable fills. This is the "system gets smarter over time" dimension that distinguishes this substrate from a static search engine. (Phase 19)
+
+---
+
+## Vision drift acknowledged (2026-04-20)
+
+The system as shipped through Phase 18 is a **hybrid SQL+vector search engine with a playbook log**. The original pitch (and the "staffing AI co-pilot" framing) implied a **meta-index that learns from playbooks over time** — hot-swap profiles weren't just routing, they were knowledge generations that compounded. That learning loop was never built; playbooks were write-only. Phase 19 closes that gap explicitly.
+
+The feedback signal is **statistical + semantic**, not neural. No model training — the index reads the playbook journal, computes operation-similarity, and boosts endorsed workers at query time. Rebuildable from `successful_playbooks` alone, same as every other derived index.
+
+---
+
+## Phases
+
+### Phase 0-5: Foundation ✅ COMPLETE
+
+- Rust workspace, Axum gateway, object storage, catalog, DataFusion query engine
+- Python sidecar with real Ollama models (embed, generate, rerank)
+- Dioxus UI with Ask (NL→SQL), Explore, SQL, System tabs
+- gRPC, OpenTelemetry, auth middleware, TOML config
+- Validated with 286K row staffing company dataset across 7 tables
+- Cross-reference queries (JOINs across candidates, placements, timesheets, calls) in <150ms
+
+### Phase 6: Ingest Pipeline
+
+Build the data on-ramp. Accept messy real-world data, normalize it, make it queryable.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 6.1 | `ingestd` crate with CSV parser → Arrow RecordBatch → Parquet | CSV file → queryable dataset |
+| 6.2 | JSON ingest (newline-delimited JSON, nested objects) | JSON file → flat Parquet |
+| 6.3 | Schema detection — infer column types from data | No manual schema definition needed |
+| 6.4 | Deduplication — detect and skip already-ingested files (content hash) | Re-ingest same file = no-op |
+| 6.5 | Text chunking — split large text fields for embedding | Long text → overlapping chunks |
+| 6.6 | Auto-registration — ingest writes to storage AND registers in catalog | Single API call: file in → queryable |
+| 6.7 | Gateway endpoint: `POST /ingest` with file upload | Upload CSV from browser → query in seconds |
+
+**Gate:** Upload a raw CSV or JSON file → auto-detected schema → stored as Parquet → registered → immediately queryable via SQL. No manual steps.
+
+**Risk:** Schema detection on messy data (mixed types, nulls, inconsistent formatting). Mitigation: conservative type inference (default to string), let user override.
+
+### Phase 7: Vector Index + RAG Pipeline
+
+Make unstructured data searchable by meaning, not just keywords.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 7.1 | `vectord` crate with embedding storage as Parquet (doc_id, chunk_text, vector) | Embeddings stored as portable Parquet |
+| 7.2 | Chunking strategy — configurable chunk size + overlap for text columns | Large text fields split into embeddable chunks |
+| 7.3 | Brute-force vector search via DataFusion (cosine similarity SQL) | Semantic search works, correctness verified |
+| 7.4 | HNSW index for fast approximate nearest neighbor | Search over 100K+ vectors in <50ms |
+| 7.5 | RAG endpoint: `POST /rag` — question → embed → search → retrieve → generate | Natural language question → grounded answer |
+| 7.6 | Auto-embed on ingest — text columns automatically embedded during ingest | No separate embedding step needed |
+| 7.7 | Hybrid search — combine SQL filters with vector similarity | "Java devs in Chicago" (SQL) + "who could do data engineering" (semantic) |
+
+**Gate:** Ingest 15K candidate resumes → auto-embed → ask "find someone who could handle our Kubernetes migration" → system returns relevant candidates ranked by semantic match, with LLM explanation.
+
+**Risk: HNSW in Rust at scale.** This is the hardest technical problem. Options:
+- `hora` crate — Rust-native ANN, but less mature than FAISS
+- Store HNSW index as a serialized file alongside Parquet data
+- Fallback: brute-force scan is fine up to ~100K vectors; optimize later
+- Nuclear option: use Qdrant as an external vector store (breaks "no new services" rule)
+
+**Decision needed:** Evaluate `hora` vs external Qdrant vs brute-force at J's data scale.
+
+### Phase 8: Hot Cache + Incremental Updates
+
+Make frequently-accessed data fast, and handle real-time updates without full rewrite.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 8.1 | MemTable hot cache — pin active datasets in memory | Queries on hot data: <10ms |
+| 8.2 | Cache policy — LRU eviction based on access patterns | Memory-bounded, auto-manages |
+| 8.3 | Incremental writes — append new rows without rewriting entire Parquet file | Update one candidate's phone → no full table rewrite |
+| 8.4 | Merge-on-read — query combines base Parquet + delta files | Correct results from base + updates |
+| 8.5 | Compaction — periodic merge of delta files into base Parquet | Prevent delta file proliferation |
+| 8.6 | Upsert semantics — insert or update by primary key | Same candidate ID → update in place |
+
+**Gate:** Update a single row in a 15K-row dataset. Query reflects the change immediately. No full Parquet rewrite. Memory cache serves hot data in <10ms.
+
+**Risk: This is the Delta Lake problem.** Full ACID transactions over Parquet files is what Databricks spent years building. We're NOT building Delta Lake — we're building a pragmatic version:
+- Append-only delta files (easy)
+- Merge-on-read (moderate)
+- Compaction (moderate)
+- Full ACID isolation (NOT attempting — single-writer model instead)
+
+### Phase 8.5: Agent Workspaces ✅ COMPLETE
+
+Per-contract overlays with daily/weekly/monthly tiers and instant handoff.
+
+- WorkspaceManager with saved searches, shortlists, activity logs
+- Zero-copy handoff between agents (pointer swap, not data copy)
+- Persisted to object storage, rebuilt on startup
+
+### Phase 9: Event Journal — Never Destroy Evidence
+
+**Principle:** Every mutation is appended, never overwritten. This is the one decision that's impossible to retrofit — once history is lost, it's gone forever.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 9.1 | `journald` crate: append-only event log as Parquet | Every write/update/delete logged with who, when, what, old value, new value |
+| 9.2 | Event schema: entity, field, old_value, new_value, actor, timestamp, source, workspace_id | Standardized across all mutations |
+| 9.3 | Journal query: `SELECT * FROM journal WHERE entity = 'CAND-001' ORDER BY timestamp` | Full history of any record |
+| 9.4 | Replay capability: rebuild any dataset's state at any point in time | Time-travel queries |
+| 9.5 | Journal compaction: roll old events into monthly summary Parquet files | Prevent unbounded growth |
+
+**Gate:** Change a candidate's phone number. Query shows the change. Journal shows old value, new value, who changed it, when, and why. Replay to yesterday's state.
+
+**Why now:** In 3 years, compliance, AI auditability, and "why did the agent recommend this candidate" all require mutation history. Adding it later means you only have history from that day forward.
+
+### Phase 10: Rich Catalog v2 — Metadata as Product
+
+**Principle:** Every dataset should be self-describing. A new team member (or AI agent) should understand what data exists, who owns it, how fresh it is, and what's sensitive — without asking anyone.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 10.1 | Catalog schema upgrade: add owner, sensitivity, freshness_sla, description, tags, lineage | `GET /catalog/datasets` returns rich metadata |
+| 10.2 | Sensitivity classification: PII, PHI, financial, public, internal | Sensitive fields tagged at ingest |
+| 10.3 | Lineage tracking: source_system → ingest_job → dataset → derived_dataset | "Where did this data come from?" answerable |
+| 10.4 | Freshness contracts: expected_update_frequency, last_updated, stale_after | Alert when data goes stale |
+| 10.5 | Dataset contracts: required columns, type expectations, validation rules | Ingest rejects data that breaks the contract |
+| 10.6 | Auto-documentation: AI generates dataset description from schema + sample data | New datasets self-describe on ingest |
+
+**Gate:** Ingest a CSV. System auto-detects PII columns (email, phone, SSN patterns), tags them, generates a description, sets owner, and tracks lineage back to the source file.
+
+**Why now:** Every dataset you ingest without metadata becomes a "mystery file" in 6 months. The metadata layer makes the difference between a searchable knowledge platform and a data graveyard.
+
+### Phase 11: Embedding Versioning — Model-Proof Vector Layer
+
+**Principle:** Embedding models will change. If you don't track which model created which vectors, upgrading means re-embedding everything from scratch.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 11.1 | Vector index metadata: model_name, model_version, dimensions, created_at | Every index knows its embedding model |
+| 11.2 | Multi-version indexes: same data, different models, coexist | Search specifies which model version |
+| 11.3 | Incremental re-embed: only new/changed docs get re-embedded on model upgrade | Model swap doesn't require full re-embed |
+| 11.4 | A/B search: query both old and new model, compare results | Validate model upgrade before committing |
+
+**Gate:** Upgrade from nomic-embed-text to a new model. Old index still works. New index builds incrementally. Compare search quality. Switch when ready.
+
+### Phase 12: Tool Registry — Agent-Safe Business Actions
+
+**Principle:** In 3 years, AI agents won't just query — they'll act. Instead of every agent getting raw SQL access, expose named, governed, audited business actions.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 12.1 | Tool definition: name, description, parameters, permissions, audit_level | `search_candidates(skills, city, min_years)` as a registered tool |
+| 12.2 | Tool execution: validates params, checks permissions, logs usage, runs query | Agent calls tool, gets results, action is logged |
+| 12.3 | Read vs write tools: read tools are permissive, write tools require confirmation | `get_candidate` = auto-approved, `update_phone` = requires review |
+| 12.4 | MCP-compatible interface: expose tools via Model Context Protocol | Any MCP-compatible agent (Claude, GPT, local) can use them |
+| 12.5 | Rate limiting + quotas per agent/tool | Prevent runaway agent from overwhelming the system |
+
+**Gate:** An AI agent calls `search_candidates(skills="Python,AWS", city="Chicago", available=true)` → gets results → calls `shortlist_candidate(workspace_id, candidate_id, reason)` → action is logged, auditable, reversible.
+
+**Why now:** The tool interface is cheap to build (it's just named endpoints with validation). But retrofitting audit logging and permission checks onto raw SQL access is a nightmare. Build the governed interface first.
+
+### Phase 13: Security & Access Control
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 13.1 | Field-level sensitivity tags (PII, PHI, financial) in catalog | Sensitive fields identified |
+| 13.2 | Row-level access policies (agent A sees their candidates only) | Policy evaluated at query time |
+| 13.3 | Column masking (show last 4 of SSN, redact salary for non-managers) | Masked results based on role |
+| 13.4 | Query audit log (who queried what, when, which fields) | Every data access recorded |
+| 13.5 | Policy-as-code (TOML/YAML rules, not hardcoded) | Non-engineer can update access rules |
+
+### Phase 14: Schema Evolution + AI Migration
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 14.1 | Schema diff detection: old schema vs new ingest → list changes | "Column renamed: first_name → full_name" |
+| 14.2 | AI-generated migration rules: LLM suggests column mappings | "full_name = concat(first_name, ' ', last_name)" |
+| 14.3 | Migration preview: show how old data maps to new schema before applying | Human approves before data transforms |
+| 14.4 | Versioned schemas in catalog: v1, v2, v3 coexist | Queries specify version or use latest |
+
+### Phase 15: Infrastructure horizon items
+
+- [x] HNSW vector index with trial system (shipped 2026-04-16)
+- [x] Federation foundation — ADR-017 (shipped 2026-04-16)
+- [x] Database connector ingest — Postgres batch with streaming (shipped 2026-04-16)
+- [x] Federation layer 2 — runtime bucket lifecycle, per-index bucket scoping, profile bucket auto-provisioning (shipped 2026-04-17)
+- [x] MySQL streaming connector — mirrors Postgres path, verified on live MariaDB (shipped 2026-04-17)
+- [x] PDF OCR for scanned documents — Tesseract 5.5 fallback when lopdf yields no text (shipped 2026-04-17)
+- [x] Scheduled ingest — interval-based per-source schedules with CRUD + run-now + auto-trigger agent (shipped 2026-04-17)
+- [ ] Multi-node query distribution (DataFusion supports this architecturally)
+
+### Phase 16: Hot-Swap Index Generations
+
+Make indexes upgrade-in-place without dropping queries.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 16.1 | "Active generation" pointer per logical index name | `/vectors/search` routes to current champion automatically |
+| 16.2 | Background trial runner: watches trial journal, proposes configs (random search / Bayesian), fires `/hnsw/trial` | Agent autonomously tunes without human POSTing each config |
+| 16.3 | Promotion endpoint: `POST /hnsw/promote/{index}/{trial_id}` atomically swaps active pointer | Next search hits new config, zero downtime |
+| 16.4 | Rollback: `POST /hnsw/rollback/{index}` reverts to previous generation | Bad promotion recoverable in milliseconds |
+| 16.5 | Dataset-append triggers: when `POST /ingest/file` writes to a dataset with attached vector indexes, schedule automatic re-trial (not full rebuild) | New docs get embedded + indexed without manual intervention |
+
+**Gate:** Run the trial agent for 10 minutes against `resumes_100k_v2` with a fresh eval set. It explores the `ef_construction × ef_search` space, promotes the Pareto winner, continues running. Zero human clicks. All trials and promotions appear in `/hnsw/trials/resumes_100k_v2`.
+
+**Risk:** Agent loops into a bad region (e.g. always proposes ef_construction=1). Mitigation: a hardcoded config space constraint + minimum-quality gate (don't promote anything with recall <0.9).
+
+### Phase 17: Model Profiles + Dataset Bindings
+
+Make "different models see different data" real instead of a config string.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 17.1 | `ModelProfile` manifest: id, ollama_name, bucket, bound_datasets[], hnsw_config, embed_model | `GET /models` lists profiles; `POST /models` creates one |
+| 17.2 | Profile activation endpoint: `POST /profile/{id}/activate` — warms EmbeddingCache for bound indexes, builds HNSW with profile's config | Next search against bound indexes is <1ms cold |
+| 17.3 | Model-scoped search: `POST /search?model=X` filters to bound datasets only | Model A can't see Model B's datasets unless explicitly shared |
+| 17.4 | VRAM-aware activation: only one (or small N) model loaded at a time on 16GB A4000 | Activating model B unloads model A via Ollama's keep_alive=0 |
+| 17.5 | Audit: every tool invocation by a model is logged with model identity | `GET /models/{id}/audit` shows exactly what each model touched |
+
+**Gate:** Two model profiles defined: `staffing-recruiter` (bound to candidates/placements/timesheets) and `docs-assistant` (bound to a documentation corpus). Activate staffing-recruiter, search for candidates — works. Switch to docs-assistant, same search — returns zero from staffing (not bound) but finds docs. VRAM shows only one embedding model loaded at a time.
+
+**VRAM reality:** 16GB A4000 realistically holds 1-2 loaded models concurrently. "Multi-model" in practice means sequential swap between profiles, not parallel serving. The profile abstraction makes this swap clean.
+
+### Phase 18: Storage format decision (Lance evaluation)
+
+The question raised 2026-04-16 after J's LLMS3 knowledge base identified Lance as `alternative_to` Parquet for vector workloads. Current stack: Parquet with binary-blob vector columns + in-RAM HNSW sidecar. Evaluated against: Lance native vector format with disk-resident indexes.
+
+| Step | Deliverable | Decision criteria |
+|---|---|---|
+| 18.1 | ✅ Parallel Lance-backed vector index for `resumes_100k_v2` in standalone `crates/lance-bench` | Built 2026-04-16 |
+| 18.2 | ✅ Head-to-head benchmark across 8 dimensions (cold-load, search latency, disk, index build, random access, append) | Complete |
+| 18.3 | ✅ ADR-019 committed with measured data and decision | See `docs/ADR-019-vector-storage.md` |
+
+**Outcome:** Hybrid architecture. Parquet+HNSW stays primary (2.55× faster search at 100K in-RAM). Lance joins as a second backend for Phase 16 hot-swap (14× faster index builds), Phase C/append workloads (0.08s vs full rewrite), RAG random-access retrieval (112× faster), and indexes past the ~5M RAM ceiling.
+
+Per-profile `vector_backend: Parquet | Lance` becomes part of Phase 17 (model profiles). See ADR-019 for the full scorecard and caveats.
+
+### Phase 19: Playbook memory (meta-index) — the feedback loop
+
+Make successful playbooks actually improve future searches. Today `successful_playbooks` is a write-only log; future-you looks at it and thinks "cool, we filled Toledo welders once" — but the index has no idea it happened, so the next Toledo-welder search ranks the same as if none of those fills had existed. Phase 19 closes the loop.
+
+| Step | Deliverable | Gate |
+|---|---|---|
+| 19.1 | Embed every `successful_playbooks` row — operation + approach + context → one chunk per playbook | A new dataset `playbook_memory` appears in catalog with N rows = row count of `successful_playbooks` |
+| 19.2 | Vector index on `playbook_memory` (HNSW or Lance — whichever `agent-parquet` profile uses) | `/vectors/search` against `playbook_memory` returns semantically similar past playbooks |
+| 19.3 | Endorsement extraction: each playbook row has `fills[]` (worker_ids it succeeded with). Parse them out at ingest time and store in a sidecar `playbook_endorsements` Parquet keyed by playbook_id | `SELECT * FROM playbook_endorsements WHERE playbook_id = 'X'` returns the worker_ids |
+| 19.4 | `/vectors/hybrid` gains opt-in `use_playbook_memory: bool`. When true: after hybrid ranks candidates, find top-K similar past playbooks (semantic search over `playbook_memory`), extract endorsed worker_ids, add a bounded boost to candidates in the endorsed set, re-rank | A search where the "right" worker is known from a prior playbook ranks higher with the flag than without |
+| 19.5 | Write-through from multi-agent orchestrator: when two agents seal a playbook, it appends to `successful_playbooks` AND triggers a refresh of `playbook_memory` (via existing Phase C stale-mark path). Next query sees the new signal. | Run the orchestrator → inspect `playbook_memory` → see a new row. Run the same query before/after → ranking differs. |
+| 19.6 | Ceiling-aware boost: cap the per-worker boost so one popular worker can't dominate future searches. Boost decays with time (optional) so stale playbooks matter less. | Synthetic test: 100 playbooks all filled with the same worker_id; the 101st search still returns a mix, not just that one worker |
+
+**Gate:** Run a real search before and after a new successful playbook lands. The endorsed workers from similar past operations rank higher in the second call. Demonstrable with a diff of the two result sets.
+
+**Why this is the right version of "meta-index":** The alternative — training a neural re-ranker on (query, candidate, outcome) triples — is a weeks-long ML story and requires labeled outcome data we don't really have. The statistical-semantic version here is rebuildable from the existing playbook journal, explainable ("boosted because of similar playbooks X, Y, Z"), and invalidatable (delete a playbook → boost goes away on next rebuild). It gets 80% of the payoff at 10% of the cost. Neural re-ranking stays as a future option if it bites.
+
+**Non-goals for this phase:**
+- Neural training / fine-tuning. Statistical feedback only.
+- Hard guarantees about recall lift magnitude. "Measurably better on the demo query" is the gate, not a universal quality claim.
+- Real-time recomputation on every playbook. Batched refresh via the existing stale-marking path is sufficient.
+
+### Phase 19 refinement (WIRED 2026-04-21): geo-filter + role prefilter on boost
+
+Item-3 diagnostic pass surfaced that `compute_boost_for` was ranking playbooks globally by cosine similarity, while candidates came from an SQL-filtered city. Result: boost map had 170 endorsed workers, 0 intersected the 50 Nashville-filtered candidates. Zero citations where there should have been dozens.
+
+Fix — in `crates/vectord/src/playbook_memory.rs`:
+- `compute_boost_for_filtered(target_geo)` — skip playbooks from other cities before cosine sort.
+- `compute_boost_for_filtered_with_role(target_geo, target_role)` — multi-strategy: exact (role, city, state) match earns similarity=1.0 and fills up to half the top_k; cosine fallback fills the rest. Mirrors Mem0/Zep 2026 guidance on parallel-strategy rerank.
+
+In `crates/vectord/src/service.rs`:
+- `extract_target_geo` and `extract_target_role` pull both from the executor's SQL filter.
+- `tracing::info!` emits `playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=?` on every hybrid_search. Silent-truncation class of bug now visible.
+
+Citation lift measured: avg citations per run 0.32 → 1.38 after geo filter; then 2 → 28 in the single-scenario Riverfront Steel re-run after role prefilter landed. 14× delta on same scenario.
+
+Unit tests: `extract_target_geo_basic`, `_missing_state_returns_none`, `_word_boundary` (rejects "civilian" substring), `extract_target_role_basic`, `_none_when_absent`, `_multi_word` — all pass (`cargo test -p vectord --lib extractor_tests`).
+
+### Phase 20: Model Matrix + Overseer Tiers (WIRED 2026-04-21)
+
+Five-tier routing declared in `config/models.json`. Hot path (T1/T2) stays local (qwen3.5 + qwen3 after mistral was dropped for 0/14 fill rate on complex scenarios). Cloud for overview (T3 gpt-oss:120b), strategic (T4 qwen3.5:397b), and gatekeeper (T5 kimi-k2-thinking). Every tier declares `context_window` + `context_budget` + `overflow_policy`.
+
+- T1 hot: 50-200 calls/scenario, local only — `qwen3.5:latest` executor, `think:false`
+- T2 review: 5-14 calls/event, local only — `qwen3:latest` reviewer, `think:false`
+- T3 overview: 1-3 calls/scenario, cloud primary — `gpt-oss:120b` on Ollama Cloud, thinking on
+- T4 strategic: 1-10 calls/day, cloud primary
+- T5 gatekeeper: 1-5 calls/day, audit-logged
+
+T3 checkpoints + cross-day lessons wired. Lessons archive to `data/_playbook_lessons/` and load back at next scenario start as `prior_lessons` in executor context. Cloud passthrough verified on stress_01 scenario with `LH_OVERVIEW_CLOUD=1` — `gpt-oss:120b` response latency consistently 4-8s, diagnosing city-pivot ("Gary IN → Chicago IL, 40mi") when target city has zero supply.
+
+`think:false` is the key mechanical finding — qwen3.5 burns ~650 tokens of hidden reasoning before emitting response; hot-path JSON emitters MUST disable thinking or continuation has to paper over empty returns. T3/T4 overseers KEEP thinking (that's the point).
+
+**Kimi-k2.6 upgrade path:** Current Ollama Cloud key returns 403 on kimi-k2.6 (`ollama run kimi-k2.6:cloud` requires `ollama signin` with pro-tier account). kimi-k2.5 substitutes on the current tier — same family, strong at tool calling. Swap to k2.6 is a one-line change in `applyToolLevel` once the subscription lands.
+
+### Phase 21: Scratchpad + Tree-Split Continuation
+
+**Why this is a phase and not an optimization:** bumping `max_tokens` until a response stops truncating is a tourniquet — J called this out explicitly. As playbooks accumulate into the hundreds and responses grow, eventually SOME request will exceed SOME model's window, and we can't solve it by raising a number. The stable answer is two primitives that let us handle arbitrary-size work without losing context: a scratchpad that glues multi-call responses together, and a tree split that shards oversized inputs and reduces them back.
+
+**Two primitives (WIRED 2026-04-21 in `tests/multi-agent/agent.ts`):**
+
+1. **`generateContinuable()`** — handles OUTPUT overflow. Calls the model; checks structural completeness (for JSON: matched braces + JSON.parse success; for text: non-empty). If incomplete, calls again with "continue from here" + the partial response as scratchpad. Up to `max_continuations` times. No `max_tokens` tuning needed — if thinking ate the initial budget, continuation picks up the slack.
+
+2. **`generateTreeSplit()`** — handles INPUT overflow. Caller passes an array of shards (semantic chunks of the corpus). For each shard: map call with running scratchpad digest. Final reduce call produces the answer. Scratchpad truncates oldest content if it approaches its own budget. If a single shard still overflows, `assertContextBudget` throws — caller must re-shard at finer granularity, NOT silently truncate.
+
+**Guarantees:**
+1. No agent call can silently truncate. Either it completes, continues, or throws with numbers.
+2. No corpus is too big — `generateTreeSplit` handles any size the caller can shard.
+3. Scratchpad is the glue between multi-call responses; context is never lost, only compacted.
+4. Token estimation uses `chars / 4` (biased safe ~15%) until we wire the provider's tokenizer.
+
+**What lives where now:**
+- `agent.ts::estimateTokens()` + `assertContextBudget()` + `generateContinuable()` + `generateTreeSplit()` — WIRED
+- `scenario.ts` executor + reviewer + overviewGenerate calls — migrated to `generateContinuable`
+- `config/models.json` — context_window + context_budget + overflow_policies per tier (declarative)
+
+**Next sprint (Rust side, so gateway tools share it):**
+- `crates/aibridge/src/continuation.rs` — port of `generateContinuable`
+- `crates/aibridge/src/tree_split.rs` — port of `generateTreeSplit`
+- `crates/storaged/src/chunk_cache.rs` — precomputed shards keyed by corpus hash (avoid re-chunking on every T4 run)
+- `/metrics` counter: `context_continuations_total{model,shape,succeeded}`
+
+**Status:** TS primitives WIRED. Rust port pending. The escalation path (tree split → bigger-context cloud model → kimi-k2:1t's 1M window → split decision into sub-decisions) is declared in `config/models.json` under `context_management.overflow_policies`.
+
+### Phase 21 status update (WIRED 2026-04-21 evening)
+
+Additional primitives landed after the initial commit:
+
+- **`think: boolean`** flag plumbed through `generate()`, `generateCloud()`, `generateContinuable()`, and into sidecar's `/generate` endpoint. Enables per-call opt-out of hidden reasoning for hot-path JSON emitters. Verified: qwen3.5 with `think:false` + `num_predict:400` returns clean `{"worker_id":...}` on first call; without `think:false`, 650 tokens eaten by reasoning, response empty.
+
+- **Cloud executor routing** — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags let per-staffer tool_level route executor to Ollama Cloud when weak local model (qwen2.5) would collapse. Verified on kimi-k2.5 via Ollama Cloud: clean JSON emission, think:false honored.
+
+Rust port of continuation + tree-split primitives remains queued for next sprint (`crates/aibridge/src/continuation.rs`, `tree_split.rs`).
+
+### Phase 22: Internal Knowledge Library (KB)
+
+Meta-layer over Phase 19 playbook_memory. Playbook memory answers "which WORKERS worked for this event." The KB answers "which CONFIG worked for this playbook signature." Subject changes from workers to the system itself — model choice, budget hints, overflow policies, pathway notes.
+
+**Files (`data/_kb/`):**
+- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count). Sig = stable hash of the sequence of (kind, role, count, city, state) across events.
+- `outcomes.jsonl` — per-run record: {sig, run_id, models, ok/total, turns, citations, per-event summary, elapsed}.
+- `pathway_recommendations.jsonl` — AI-synthesized for next run: {confidence, rationale, top_models, budget_hints, pathway_notes, neighbors_consulted}.
+- `error_corrections.jsonl` — detected fail→succeed pairs on same sig, diff of what changed.
+- `config_snapshots.jsonl` — history of models.json changes + why.
+
+**Cycle (event-driven, not wall-clock):**
+1. Scenario ends → `kb.indexRun()` extracts signature, embeds spec digest, appends outcome.
+2. `kb.recommendFor(nextSpec)` finds k-NN signatures via cosine, feeds their outcome history + recent error corrections to the overview model, writes a structured recommendation.
+3. Next scenario starts → `kb.loadRecommendation(spec)` pulls the newest rec for this sig, injects `pathway_notes` into `guidanceFor()` alongside prior lessons.
+
+**Why file-based for MVP:** Phase 19 playbook_memory is already a catalogd dataset. KB is a separate meta-layer; keep it file-based first to iterate without a gateway schema migration. Rust port (and promotion to vectord-indexed corpus for neighbor search at scale) lands once shape stabilizes — mirrors how Phase 21 primitives were TS-first → Rust next sprint.
+
+**What the overview model gets asked:**
+- Target scenario digest
+- Top-k neighbor signatures with avg ok rate, best model combo per neighbor
+- Recent error corrections (sig, before/after model set)
+
+**What it outputs (JSON-constrained):**
+- confidence (high/medium/low)
+- rationale (2-3 sentences)
+- top_models {executor, reviewer, overview}
+- budget_hints {executor_max_tokens, reviewer_max_tokens, executor_think}
+- pathway_notes (concrete pre-run advice)
+
+**Status (WIRED 2026-04-21):** `tests/multi-agent/kb.ts` holds all primitives. scenario.ts reads rec at start, indexes + recommends at end. Cold start gracefully writes a "low confidence, no history" rec so the second run has a floor to build on. `snapshotConfig()` wired to fire at every scenario start — active model set + tool_level + cloud flags hashed and appended to `config_snapshots.jsonl`.
+
+**Phase 22 item B — cloud rescue (WIRED):** When an event fails and cloud T3 is enabled, `requestCloudRemediation()` feeds the failure trace (SQL filters attempted, row counts, reviewer drift reasons, gap signals, contract terms) to cloud and parses a JSON remediation with new_city / new_state / new_role / new_count / rationale. Event retries once with the pivot. Verified 1/3 rescues succeeded on stress_01 (Gary IN → South Bend IN pivot filled a Welder that local drift-aborted). Sanitizer splits "City, ST" comma-packed outputs so downstream SQL doesn't get `Hammond, IN, IN`.
+
+### Phase 23: Staffer identity + competence-weighted retrieval (WIRED 2026-04-21)
+
+Answers "who handled this" as a first-class dimension of the matrix index. Senior staffers' playbooks rank higher than juniors' on similar scenarios via competence × similarity score. Auto-discovers "reliable performer" worker labels via cross-staffer endorsement overlap.
+
+**Schema (`scenario.ts` ScenarioSpec):**
+- `contract?: ContractTerms` — deadline, budget_per_hour_max, local_bonus_per_hour, local_bonus_radius_mi, fill_requirement. Propagates into T3 checkpoint + cloud rescue prompts so cloud reasons about trade-offs (pivot-within-radius before budget-pivot-further).
+- `staffer?: Staffer` — {id, name, tenure_months, role, tool_level}. tool_level controls subsystems available to this run:
+  - `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
+  - `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
+  - `basic` — **kimi-k2.5 cloud** exec + qwen3 local reviewer + local T3, no rescue
+  - `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue — tests whether playbook inheritance carries knowledge alone
+
+**KB staffer indexing (`data/_kb/staffers.jsonl`):**
+- Recomputed per-staffer on every run: total_runs, fill_rate, avg_turns_per_event, avg_citations_per_run, rescue_rate, competence_score.
+- `competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate`. Bounded 0..1.
+
+**Weighted neighbor retrieval:**
+- `findNeighbors` in `kb.ts` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3). Senior playbooks rank above junior playbooks on similar scenarios.
+- `pathway_recommendations` include `best_staffer_id` / `best_staffer_competence` so cloud knows WHOSE playbook it's synthesizing from.
+
+**Cross-staffer auto-discovery:**
+- `scripts/kb_staffer_report.py` emits leaderboard + workers endorsed across ≥2 staffers on same signature.
+- Validated output: Rachel D. Lewis (Welder Nashville) endorsed 12× across 4 staffers; Christina Watson (Machine Op Indianapolis) 11×. These are the highest-confidence "reliable performer" labels the system produced without human tagging.
+
+**Demo infrastructure:**
+- `tests/multi-agent/gen_staffer_demo.ts` — 4 personas × 3 contracts = 12 scenario specs.
+- `scripts/run_staffer_demo.sh` — sequential batch with cloud T3.
+- `scripts/kb_staffer_report.py` — leaderboard + top/bottom differential + cross-staffer overlap.
+
+### Phase 24: Observer / Autotune integration (SHIPPED 2026-04-20, commit b95dd86)
+
+The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
+
+**What shipped:**
+- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
+- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
+- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
+- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through.
+
+### Phase 25: Validity windows + playbook retirement (SHIPPED 2026-04-21, commit e0a843d)
+
+Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive. `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`. `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking. Two retirement paths: `retire_one(id, reason)` for manual, `retire_on_schema_drift(city, state, fp, reason)` for batch migration sweep. Endpoint: `POST /vectors/playbook_memory/retire`.
+
+### Phase 26: Mem0 upsert + Letta geo hot cache (SHIPPED 2026-04-21, commit 640db8c)
+
+Same-day re-seed no longer duplicates rows. `/seed` with `append=true` routes through `upsert_entry` which decides ADD / UPDATE / NOOP on `(operation, day, city, state)`. Playbook_id stays stable on UPDATE so existing citations remain valid. `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation; geo-filtered boost queries skip the scan and hit O(1) lookup — sub-ms at current scale, same code path scales to 100K+ entries.
+
+### Phase 27: Playbook versioning (SHIPPED 2026-04-21)
+
+`PlaybookEntry` gained `version: u32` (default 1), `parent_id`, `superseded_at`, `superseded_by` — all `#[serde(default)]` so pre-Phase-27 state loads as roots. `revise_entry(parent_id, new_entry)` appends a new version, stamps the parent superseded, rejects revising a retired or already-superseded parent. `history(id)` returns the root→tip chain from any node. Superseded entries excluded from boost (same rule as retired). Endpoints: `POST /vectors/playbook_memory/revise`, `GET /vectors/playbook_memory/history/{id}`. `/status` reports `superseded` as a distinct counter. 8 new tests; 51/51 vectord lib tests green.
+
+### Phase 28: Agent Gateway + MCP Server (SHIPPED 2026-04-21)
+
+The agent gateway wraps the Rust substrate with an MCP-first interface (Model Context Protocol), enabling Claude Code, GPT agents, and internal scripts to interact with the lakehouse through named tools rather than raw HTTP. Built on Bun (not Node), serving as the "front door" for all AI consumers.
+
+**Files:** `mcp-server/index.ts` (2241 lines)
+
+| Tool | Purpose | Endpoint |
+|---|---|---|
+| `search_workers` | Hybrid SQL+vector (the core) | `POST /vectors/hybrid` |
+| `query_sql` | Analytical SQL on any dataset | `POST /query/sql` |
+| `match_contract` | Find workers for a job order | `POST /match` |
+| `get_worker` | Single worker by ID | `POST /worker/:id` |
+| `rag_question` | Full RAG pipeline | `POST /vectors/rag` |
+| `log_success` | Record operation → playbook | `POST /log` |
+| `log_failure` | Record failed fill (negative signal) | `POST /log_failure` |
+| `get_playbooks` | Retrieve past successes | `GET /playbooks` |
+| `swap_profile` | Hot-swap model+data context | `POST /profile/:id` |
+| `vram_status` | GPU introspection | `GET /vram` |
+
+**HTTP Routes (internal agent consumption):**
+- `/search` — hybrid search with client blacklist + rate enrichment
+- `/verify` — claim verification against golden dataset
+- `/context` — self-orientation for agents
+- `/clients/:client/blacklist` — per-client worker exclusion
+- `/memory/query` — unified memory surface (Phase 24 refinement)
+- `/system/summary` — truthful row counts via SQL
+- `/models/matrix` — read `config/models.json`
+
+### Phase 29: Intelligence Suite (SHIPPED 2026-04-21)
+
+Market intelligence endpoints surfaced through the MCP server — real-world demand signals from public data, cross-referenced with worker bench for staffing gap analysis.
+
+**Files:** `mcp-server/index.ts` continued
+
+| Endpoint | Purpose | Data Source |
+|---|---|---|
+| `/intelligence/market` | Building permits → demand forecast | Chicago Socrata API |
+| `/intelligence/staffing_forecast` | 30-day demand by role | Chicago permits + bench |
+| `/intelligence/permit_contracts` | Permits + Phase 19 ranked candidates | Chicago permits + Workers 500K |
+| `/intelligence/activity` | Activity feed + learned patterns | successful_playbooks |
+| `/intelligence/brief` | Parallel analytics across 500K profiles | Workers 500K |
+| `/intelligence/chat` | Natural language → routed queries | Hybrid + RAG |
+
+**Market signals captured:**
+- Largest permits by cost (top 50)
+- Work type breakout (electrical, mechanical, masonry, plumbing)
+- Cross-reference with IL bench supply/available/reliable
+- Demand forecast using $150K/worker industry heuristic
+
+### Phase 30: Observer + Autotune Integration (SHIPPED 2026-04-20, commit b95dd86)
+
+The gap: `lakehouse-observer.service` wrapped MCP :3700, while `tests/multi-agent/scenario.ts` hit gateway :3100 directly. Observer idle at 0 ops across 3600+ cycles — scenarios invisible to ERROR_ANALYZER and PLAYBOOK_BUILDER, autotune running blind to outcomes.
+
+**Files:** `mcp-server/observer.ts` (335 lines)
+
+**What shipped:**
+- `observer.ts` Bun HTTP listener on `OBSERVER_PORT` (default 3800): `GET /health`, `GET /stats` (totals, by_source, recent scenario digest), `POST /event` for scenario outcomes.
+- `ObservedOp` carries provenance — `source="scenario" | "mcp"` + `staffer_id` + `sig_hash` + `event_kind` + geo + rescue flags.
+- `recordExternalOp()` — shared ring-buffer insert; main analyzer + playbook builder no longer care where the op came from.
+- `persistOp()` fix: old path POSTed to `/ingest/file?name=observed_operations` which has REPLACE semantics (wiped prior ops); now uses append-friendly write-through to `data/_observer/ops.jsonl`.
+
+**HTTP API:**
+- `GET /health` — ring size
+- `GET /stats` — totals, by_source, recent scenario ops, rescue stats
+- `POST /event` — record scenario outcomes with provenance
+
+### Phase 31: Scenario Orchestrator with Tiers (SHIPPED 2026-04-21)
+
+A day-in-the-life scenario test running six events against the live substrate. Exercises baseline_fill, recurring, expansion, emergency, misplacement. Routes through the same executor/reviewer loop as the single-task orchestrator with event-specific constraints.
+
+**Files:** `tests/multi-agent/scenario.ts` (1772 lines)
+
+**Model Matrix (configurable per-tier):**
+
+| Tier | Purpose | Model | Context Budget |
+|---|---|---|---|
+| T1 hot | 50-200 calls/scenario | qwen3.5:latest | 8K, think:false |
+| T2 review | 5-14 calls/event | qwen3:latest | 8K, think:false |
+| T3 overview | 1-3 calls/scenario | gpt-oss:120b | 32K, thinking on |
+| T4 strategic | 1-10 calls/day | gpt-oss:120b | 64K |
+| T5 gatekeeper | 1-5 calls/day | kimi-k2.6 | 128K, audit-logged |
+
+**Staffer tool_levels (Phase 23):**
+- `full` — qwen3.5 + qwen3 local + cloud T3 + cloud rescue
+- `local` — qwen3.5 + qwen3 local + local gpt-oss:20b T3 + rescue
+- `basic` — kimi-k2.5 cloud exec + qwen3 local reviewer, no rescue
+- `minimal` — kimi-k2.5 cloud exec + qwen3 local reviewer, NO T3, NO rescue
+
+**Key features:**
+- `generateContinuable()` — handles output overflow with continuation
+- `generateTreeSplit()` — handles input overflow via sharding
+- `think:false` flag — disables hidden reasoning for hot-path JSON emitters
+- Cloud executor routing — `ACTIVE_EXECUTOR_CLOUD` / `ACTIVE_REVIEWER_CLOUD` flags
+- Cloud rescue (Phase 22B) — `requestCloudRemediation()` on failure
+- KB integration at start (load recommendation) and end (index + recommend)
+
+### Phase 32: Knowledge Library + Staffer Indexing (SHIPPED 2026-04-21)
+
+Meta-layer over Phase 19 playbook_memory. Tracks which configs worked for which playbook signatures.
+
+**Files:** `tests/multi-agent/kb.ts` (600 lines)
+
+**Files under `data/_kb/`:**
+- `signatures.jsonl` — (sig_hash, embedding[], first_seen, last_seen, run_count)
+- `outcomes.jsonl` — per-run: {sig, run_id, models, ok/total, turns, citations}
+- `pathway_recommendations.jsonl` — AI-synthesized for next run
+- `error_corrections.jsonl` — detected fail→succeed pairs
+- `config_snapshots.jsonl` — history of model changes
+- `staffers.jsonl` — per-staffer competence scores
+
+**Staffer competence scoring:**
+```
+competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate
+```
+
+**Weighted neighbor retrieval:**
+- `findNeighbors` returns `weighted_score = cosine × max_staffer_competence` (floor 0.3)
+- Senior playbooks rank above junior playbooks on similar scenarios
+
+### Phase 33: Validity Windows + Playbook Retirement (SHIPPED 2026-04-21)
+
+Zep 2026-era finding: temporal validity is the single highest-value memory-hygiene primitive.
+
+**What shipped:**
+- `PlaybookEntry` gained `schema_fingerprint` / `valid_until` / `retired_at` / `retirement_reason`
+- `compute_boost_for_filtered_with_role` skips retired + expired before geo/cosine ranking
+- Two retirement paths: `retire_one(id, reason)` manual, `retire_on_schema_drift(city, state, fp, reason)` batch
+- Endpoint: `POST /vectors/playbook_memory/retire`
+
+### Phase 34: Mem0 Upsert + Geo Hot Cache (SHIPPED 2026-04-21, commit 640db8c)
+
+Same-day re-seed no longer duplicates rows.
+
+**What shipped:**
+- `/seed` with `append=true` routes through `upsert_entry` — decides ADD / UPDATE / NOOP on `(operation, day, city, state)`
+- Playbook_id stays stable on UPDATE so existing citations remain valid
+- `PlaybookMemory.geo_index: HashMap<(city, state), Vec<idx>>` rebuilt on every mutation
+- O(1) geo-filtered boost queries — sub-MS at current scale
+
+### Phase 36: Gateway Seed Concurrency Fix (SHIPPED 2026-04-22)
+
+Problem: Concurrent `playbook_memory/seed` calls caused socket collisions with the Python AI sidecar. Two staffing coordinators hitting seed simultaneously → one socket error.
+
+Solution:
+- Added `embed_semaphore: Arc<Semaphore>` to `VectorState` (permits=1)
+- Seed endpoint acquires permit before embedding, releases after
+- Serializes embed calls to sidecar, eliminates collision
+
+E2E test result: 8.0/10 avg (persist=0 is expected — same day+op = NOOP, not add)
+
+### Phase 37: Hot-Swap Endpoint Hang (TODO)
+
+Problem: `POST /vectors/profile/{id}/activate` hangs for 2-4 minutes because it does heavy work synchronously:
+- Loads embeddings into memory
+- Builds HNSW indexes
+- Preloads Ollama model
+- Auto-provisions buckets
+
+Current workaround: Client uses 5-second timeout (test passes but doesn't verify hot-swap).
+
+**Required fix**: Convert to async job pattern:
+1. Endpoint accepts request, spawns background task, returns job_id immediately
+2. Client polls `GET /jobs/{job_id}` for status
+3. Progress events streamed via SSE during activation
+4. Background task updates job status on completion
+
+This is a known architectural debt item - not blocking since tests use timeout workaround.
+
+### Phase 35+: Further horizon
+
+- Specialized fine-tuned models per domain (staffing matcher, resume parser)
+- Video/audio transcript ingest + multimodal embeddings
+- Neural re-ranker over (query, candidate, outcome) triples — only if Phase 19's statistical feedback plateaus
+- True distributed query (DataFusion multi-node) — only if single-machine ceilings bite
+
+---
+
+## Known ceilings (honest)
+
+The current stack has measurable limits. Documenting them so future decisions aren't based on wishful thinking.
+
+| Dimension | Current ceiling | Breaks at | Escape hatch |
+|---|---|---|---|
+| Vector count per index (Parquet+HNSW in-RAM) | ~5M on 128GB | Past 5M | Switch that profile's `vector_backend` to Lance per ADR-019 — IVF_PQ stays on disk-resident quantized codes |
+| Concurrent active indexes | ~50-100 at 100K vectors each | 10M×50 configurations | Lance disk-resident + per-profile activation |
+| Rows per dataset | 2.47M proven, probably 100M+ fine | Approaches DataFusion memory limits | DataFusion predicate pushdown + partition pruning (existing) |
+| Concurrent loaded models | 1-2 on 16GB VRAM (A4000) | 3+ models simultaneous | Not our problem — architectural, driven by Ollama |
+| Trial journal growth per index | Thousands of trials, batched JSONL | High-frequency auto-tuning agent | Compaction via `/hnsw/trials/{idx}/compact` |
+| Error journal growth | Bounded by ring buffer (2000 events in-memory) + batched JSONL on disk | Continuous failure scenarios | Compaction + retention policy (TODO) |
+
+---
+
+## Reference Workloads
+
+### Workload 1: Staffing Company
+
+Scale-tested on 128GB RAM server:
+
+| Table | Rows | Size | Description |
+|---|---|---|---|
+| candidates | 100,000 | 10.1 MB | Names, phones, emails, zip, skills, resume text |
+| clients | 2,000 | 33 KB | Companies, contacts, verticals |
+| job_orders | 15,000 | 0.9 MB | Positions with descriptions, requirements, rates |
+| placements | 50,000 | 1.2 MB | Candidate↔job matches with rates, recruiters |
+| timesheets | 1,000,000 | 16.7 MB | Weekly hours, bill/pay totals, approvals |
+| call_log | 800,000 | 34.3 MB | Phone CDR — who called whom, duration, disposition |
+| email_log | 500,000 | 16.0 MB | Email tracking — subject, opened, direction |
+| **Total** | **2,467,000** | **79 MB** | **7 tables, cross-referenced** |
+
+### Benchmarks (2.47M rows)
+
+| Query | Cold (Parquet) | Hot (MemCache) | Speedup |
+|---|---|---|---|
+| 100K candidate filter (skills+city+status) | 257ms | 21ms | 12x |
+| 1M timesheet aggregation + JOIN | 942ms | 96ms | 9.8x |
+| 800K call log cross-reference (cold leads) | 642ms | — | — |
+| Triple JOIN recruiter performance | 487ms | — | — |
+| 500K email open rate aggregation | 259ms | — | — |
+| COUNT all 2.47M rows | 84ms | — | — |
+| 10K vector semantic search (cosine) | 450ms | — | — |
+| Natural language → AI SQL → execute | ~3s | — | (model inference) |
+
+### Vector Search
+
+- 10K candidate resumes embedded in 204s (49 chunks/sec via Ollama)
+- Semantic search over 10K vectors: ~450ms (brute-force cosine)
+- RAG pipeline: question → embed → search → retrieve → LLM answer with citations
+- AI correctly refuses to hallucinate when context doesn't support an answer
+
+### Agent Workspaces
+
+- Create per-contract workspace with saved searches + shortlists
+- Instant handoff between agents — zero data copy
+- Full activity timeline preserved across handoffs
+
+### Workload 2: Local LLM Knowledge Base
+
+The second use case this substrate is built for. Reference corpus: the running `knowledge_base` Postgres database (586 team runs, response cache history, pipeline runs, threat intel) + LLMS3.com published corpus (~243 enriched documents).
+
+Target scale on same 128GB server:
+- Documents: 10K-100K per model profile
+- Chunks after chunking: 500K-5M per profile
+- Embedding dimensions: 768 (nomic-embed-text)
+- Query latency: <100ms semantic search, <3s end-to-end RAG including LLM generation
+- Concurrent model profiles: 2-5 configured, 1-2 active at a time (VRAM-bound)
+
+Measured to date (Phase 7 + Phase 16 prep):
+- 100K candidate-resume chunks embedded in 10 min via Ollama nomic-embed-text
+- HNSW search at 100% recall, ~1ms p50 on 100K vectors (ec=80 es=30 locked as default)
+- Trial journal instrumented and working for parameter tuning
+
+Gaps still to close for this workload:
+- Model profiles (Phase 17) — today, "model" is a string, not a first-class entity
+- Hot-swap generations (Phase 16) — today, rebuild = downtime
+- Scale past 5M vectors — needs Phase 18 Lance evaluation to decide path
+
+---
+
+## Available Local Models
+
+| Model | Use |
+|---|---|
+| `nomic-embed-text` | Embeddings (768d) — semantic search, RAG retrieval |
+| `qwen2.5` | SQL generation, structured output, summarization |
+| `mistral` | General generation, longer context |
+| `gemma2` | General generation |
+| `llama3.2` | General generation, lightweight |
+
+---
+
+## Non-Goals
+
+- Cloud deployment (local-first, always)
+- Full ACID transactions (single-writer model is sufficient)
+- Real-time streaming / CDC (batch ingest is the model; scheduled refresh, not transactional replication)
+- Replacing the CRM (this is the analytical + AI layer BEHIND the CRM)
+- Custom file formats — Parquet for datasets + sidecar indexes for vectors (see ADR-018 for why we stayed Parquet instead of migrating to Lance, and the ceilings that choice implies)
+- Hard multi-tenant isolation (profiles and federation provide soft isolation; this is not a SaaS platform with adversarial tenants — operator is single-trust)
+
+Removed from prior non-goals (2026-04-16):
+- ~~Multi-tenancy (single-owner system)~~ — federation + profile buckets are now first-class; soft multi-tenancy is a design goal. Hard adversarial multi-tenancy (adversarial tenants on shared infrastructure) remains out of scope.
+
+---
+
+## Risks
+
+### Technical Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| Vector search in Rust at scale | **High** | Start brute-force, evaluate `hora` crate, Qdrant as fallback |
+| Incremental updates on Parquet | **High** | Delta files + merge-on-read, NOT full Delta Lake |
+| Legacy data messiness | **High** | Conservative schema detection, default to string, user overrides |
+| 100K+ embedding timeout | **High** | Async background job with progress, not single HTTP request |
+| Schema evolution across ingests | **Medium** | Schema fingerprinting + versioned manifests (Phase 14) |
+| Memory pressure from hot cache | **Medium** | LRU eviction, configurable memory limit (tested: 408MB for 1.1M rows) |
+| HNSW index persistence | **Medium** | Serialize alongside Parquet, rebuild on startup |
+| Python sidecar as bottleneck | **Low** | Can replace with direct Ollama HTTP from Rust later |
+
+### Strategic Risks (Future-Proofing)
+
+| Risk | Impact | Phase |
+|---|---|---|
+| No mutation history → can't audit AI decisions | **Critical** — compliance, trust | Phase 9 (event journal) |
+| No metadata → datasets become mystery files | **High** — onboarding, discovery | Phase 10 (rich catalog) |
+| Embeddings locked to one model | **High** — can't upgrade models | Phase 11 (versioning) |
+| Raw SQL as only interface → ungoverned agent access | **High** — security, auditability | Phase 12 (tool registry) |
+| No sensitivity classification → compliance exposure | **Medium** — grows with data volume | Phase 13 (access control) |
+| No schema evolution handling → ingest breaks on format change | **Medium** — grows with source count | Phase 14 (AI migration) |
+
+---
+
+## Design Principles (Future-Proofing)
+
+These are the decisions that still look smart after the stack changes:
+
+1. **Store the truth openly.** Parquet on object storage. No proprietary formats. Any engine can read it.
+2. **Describe it richly.** Every dataset has an owner, lineage, sensitivity tags, freshness contract.
+3. **Never destroy evidence.** Every mutation is journaled. Rebuild any state at any point in time.
+4. **Secure it centrally.** Permissions live in the data layer, not application code.
+5. **Expose it through reusable interfaces.** Named tools with contracts, not raw SQL for every consumer.
+6. **Version everything.** Schemas, embeddings, models — all versioned, all coexist during migration.
+7. **Make unstructured data first-class.** Every document gets: storage, text extraction, entity tags, chunks, embeddings, linkage.
+8. **Separate storage from compute from intelligence.** Scale each independently. Replace any layer without touching the others.
+
+---
+
+## Operating Rules
+
+1. PRD > architecture > phases > status > git
+2. Git is memory, not chat
+3. No undocumented changes
+4. No silent architecture drift
+5. Always work in smallest valid step
+6. Always verify before moving on
+7. Flag when something is genuinely hard vs just engineering work
+8. If a phase reveals the approach is wrong, update the PRD before continuing
+9. **Cheap-now, expensive-later decisions get built first** (event journal, metadata, versioning)
+10. **Build the governed interface before the raw interface** (tools before SQL for agents)
@@ -0,0 +1,63 @@
+# Scrum Fix Wave — Phase-Sweep 2026-04-23
+
+**Purpose:** Direct the scrum-master pipeline at concrete findings from the Phase 0→42 audit sweep, not at the high-level vision alone. Findings live in `data/_kb/phase_sweep_findings.jsonl` (19 items).
+
+## What the auditor expects you to produce per file
+
+For each file the scrum sees: concrete code-level suggestions that close the listed findings. Not rewrites for style. Not vision drift. Land the invariant or admit the checkbox was premature.
+
+## Meta-pattern to fix (read this first)
+
+The sweep surfaced **one root cause repeated across 5 phases**: primitives exist, cross-cutting enforcement doesn't. Auth, journal, access control, truth rules — the machinery is built, nothing calls it from the actual request path.
+
+One PR-cluster retires the pattern:
+1. **Identity.** Auth middleware wires X-API-Key → extension `AgentIdentity { name, role, api_key_hash }`. (P5-001)
+2. **Request pipeline.** `/query/sql` and `/tools/*/call` read `AgentIdentity`, pass into queryd + tools handlers.
+3. **Access enforcement.** Handlers call `access.can_access()` / `masked_columns()` before returning data; log via `access.log_query()`. (P13-001)
+4. **Mutation journaling.** Every ingest / delta-write / tombstone-add / catalog-register calls the corresponding `journal.record_*`. (P9-001)
+5. **Truth enforcement.** `TruthStore::check()` rewritten to `evaluate(task_class, ctx) -> Vec<RuleOutcome>`, actually walking `RuleCondition` against context. (P42-001, P42-002)
+
+After this cluster lands, Phases 5, 9, 13, 42 become "truly shipped" rather than "machinery shipped."
+
+## Findings by severity
+
+### 🔴 High
+
+- **P9-001** `journald/src/journal.rs`, `crates/gateway/src/main.rs`, all mutation sites. Journal has zero internal callers. Every `ingestd::service::upload_file` success, every `queryd::delta::write_delta`, every `catalogd::tombstones::add`, every `catalogd::registry::register` should emit a journal event. Plus fix: `event_counter` resets on process restart — seed from max existing `event_id` on rebuild or switch to UUID v7.
+- **P13-001** `crates/gateway/src/main.rs` + `crates/queryd/src/service.rs` + `crates/gateway/src/tools/service.rs`. `AccessControl.can_access` / `masked_columns` / `log_query` have zero callers. Query path ignores role. Phase 17's `profile_scoped_search` (service.rs:1641) is the template — copy that shape.
+- **P42-001** `crates/truth/src/lib.rs:56`. `TruthStore::check(task_class)` ignores `RuleCondition` entirely. Signature needs `evaluate(task_class, ctx: &serde_json::Value) -> Vec<RuleOutcome>` that actually walks conditions. Update all 14 tests to exercise fail/pass semantics, not just storage.
+
+### 🟡 Medium
+
+- **P5-001** `crates/gateway/src/auth.rs` + `crates/gateway/src/main.rs:222-233`. `api_key_auth` is `#[allow(dead_code)]`. Wire with `axum::middleware::from_fn_with_state(api_key, auth::api_key_auth)` on protected routes. `/health` stays public.
+- **P10-001** Legacy datasets (including `candidates`, 2.47M rows) have no PII flags. Add `POST /catalog/resync-metadata` mirroring `/catalog/resync-missing`.
+- **P14-001** `crates/ingestd/src/schema_evolution.rs`. Module has 5 passing tests and zero callers. Add `POST /catalog/datasets/by-name/{name}/schema-diff`. When ADR-020 `register()` returns 409, include a `migration_rules[]` body.
+- **P20-001** `config/models.json` is spec-only — never loaded by Rust. Load into `shared::model_matrix::ModelMatrix` at startup; delegate `aibridge::context::context_window_for` to matrix.
+- **P21-001** `generate_continuable` has one prod caller (`rag.rs:171`). Audit every `ai_client.generate()` site. Convert the truncation-prone + thinking-empty-prone sites (auditor paths, reranker, autotune) to `generate_continuable`.
+- **P39-001** `ProviderAdapter` trait + adapters ship, zero callers. `/v1/chat` in `v1/mod.rs:152` uses hardcoded `match req.provider`. Replace with adapter dispatch.
+- **P40-001** `config/routing.toml` is spec-only. `RoutingEngine::new` has no callers. Add `RoutingEngine::from_toml(path)`; `V1State` carries `routing: Arc<RoutingEngine>`; `/v1/chat` consults it before provider match.
+- **P42-002** Truth has no enforcement site. `/v1/chat` or execution_loop should call `truth.evaluate(task_class, ctx)` post-response.
+
+### 🟢 Low
+
+- **P1-001** `crates/storaged/src/federation_service.rs:34-35`. Bucket-qualified routes wire only PUT + GET. Add DELETE + LIST on `/buckets/{bucket}/objects/{*key}`.
+- **P4-001** UI deploy stale. Either rebuild (`just ui-build` + restart `lakehouse-ui.service`) or amend `PHASES.md` to note pre-Phase-9 drift.
+- **P7-001** `vectord::index_registry`. Orphan index registrations (parquet deleted) still list in `/vectors/indexes`. Add a startup sweep + `POST /vectors/resync-missing`.
+- **P12-001** `crates/gateway/src/tools/service.rs`. Audit row has `row_count=null`. Propagate `QueryResponse.row_count` + add `latency_ms`.
+- **P20-002** `crates/gateway/src/v1/mod.rs`. No model-prefix auto-routing. Caller must set `provider` explicitly. Tie to P39-001 + P40-001 fix.
+- **P21-002** `crates/aibridge/src/context.rs`. `context_window_for` hardcoded HashMap duplicates `config/models.json`. Delegate once P20-001 lands.
+- **P38-001** `crates/gateway/src/execution_loop/mod.rs:1523`. Test `executor_prompt_includes_surfaced_candidates` fails on "W-1 Alice Smith" assertion. Either update prompt formatter or update test.
+- **P40-002** Cost gating absent. Add `cost_ceiling_usd_per_hour` to `RoutingEngine` rule, pre-request check against `Usage.by_provider`.
+
+## What "done" looks like for each file the scrum touches
+
+- Name the specific finding(s) the file participates in.
+- Show code-level diff (minimum: function signature + first 5 lines of body) for the fix.
+- Call out any test that needs updating + one new test that would catch the bug on reintroduction.
+- Flag if the fix is too big for one PR and should be split (most of the cross-cutting cluster wants a shared identity/middleware PR first, per-service PRs after).
+
+## Out of scope for this wave
+
+- New features beyond what the findings describe.
+- UI work (Phase 4 stale is known).
+- DevOps / long-horizon domain work (Terraform/Ansible — Phase 43+).
@@ -0,0 +1,198 @@
+# Scrum Master PR Loop — Forensic Validation Prompt (iter 2+)
+
+Adopted 2026-04-23 from J. Replaces the default scrum prompt starting iter 2. Iter 1 used the softer "fix-wave" framing; iter 2 onward uses this adversarial one.
+
+---
+
+You are acting as an adversarial **Scrum Master + Systems Auditor**.
+
+Your job is to **prove whether this system actually works**, not to describe it.
+
+You are auditing a system with the following architecture:
+
+- AI Gateway with per-model adapters
+- Output normalization + schema validation layer
+- Execution pipeline (Terraform / Ansible / shell)
+- Task-scoped execution memory (S3 + Apache Arrow/Parquet)
+- Relevance orchestration (context filtering, freshness validation, fact extraction)
+- Local → Cloud fallback loop for failed tasks
+- Iterative repair loop with stored execution evidence
+
+---
+
+## PRIMARY OBJECTIVE
+
+Determine if the system is:
+
+1. Executable (real, not pseudocode)
+2. Aligned with PRD contracts
+3. Deterministic enough to trust
+4. Protected from model output drift
+5. Actually closing the loop (fail → repair → reuse)
+
+---
+
+## NON-NEGOTIABLE RULES
+
+- Do NOT summarize
+- Do NOT explain architecture unless tied to failure
+- Do NOT assume code works — verify
+- Every claim MUST reference files, functions, or execution evidence
+- If something is unclear → mark as FAIL
+
+---
+
+## AUDIT PASSES (RUN ALL)
+
+### 1. PSEUDOCODE / FAKE IMPLEMENTATION DETECTION
+Find any:
+- TODO / stub / placeholder
+- hardcoded outputs where AI should decide
+- mocked execution paths
+- fake success returns
+
+Output exact file + line references.
+
+### 2. PRD CONTRACT VALIDATION
+Verify implementation exists for:
+
+- Gateway routing logic
+- Per-model adapters
+- Output normalization (strip, parse, canonicalize)
+- Schema validation layer
+- Repair loop (retry with modification)
+- Raw output storage
+- Execution memory persistence
+- Retrieval based on prior failures
+- Relevance filtering (freshness / protocol awareness)
+- Execution permission gate
+
+For each component:
+- status: implemented | partial | missing
+- include file references
+
+### 3. NORMALIZATION + VALIDATION PIPELINE
+Prove that:
+
+- Raw model output is NEVER executed directly
+- JSON extraction is enforced
+- Unknown fields are rejected or handled
+- Schema validation blocks bad output
+- Repair loop triggers on failure
+
+If any path bypasses validation → FAIL
+
+### 4. FAILURE → CLOUD → REPAIR LOOP
+Trace the loop:
+
+- Local model fails
+- Failure is classified
+- Context is packaged
+- Cloud model returns corrective instruction
+- Local model retries
+- Result is validated
+- Successful pattern is stored
+
+If any step is missing or non-deterministic → FAIL
+
+### 5. EXECUTION MEMORY (S3 / ARROW)
+Verify:
+
+- Raw runs are stored (input, raw output, normalized output)
+- Failures are recorded with signatures
+- Successful retries are recorded
+- Retrieval pulls based on:
+  - task similarity
+  - failure signature
+  - execution success history
+
+If memory is only logs and not reused → FAIL
+
+### 6. RELEVANCE ORCHESTRATION
+Verify:
+
+- Context is filtered before model input
+- Freshness or version awareness exists
+- Fact extraction reduces noise
+- Context inclusion is explainable
+
+If system blindly injects context → FAIL
+
+### 7. EXECUTION SAFETY
+Verify:
+
+- No shell / terraform / ansible execution without validation gate
+- No direct model-to-command execution
+- Clear permission boundary exists
+
+If AI can execute commands unchecked → CRITICAL FAIL
+
+### 8. TESTING + EVIDENCE
+Find:
+
+- real tests (not mocks)
+- execution logs
+- validation results
+- success/failure traces
+
+If no proof of execution → FAIL
+
+---
+
+## OUTPUT FORMAT (STRICT)
+
+Each finding in any array MUST include a `confidence` field (integer 0–100). The confidence represents your self-assessed probability that the finding is correct and actionable. Low confidence is valuable — do not inflate. A finding with confidence < 50 is still recorded (it signals investigation needed) but downstream consumers will weight it less.
+
+```json
+{
+  "verdict": "pass | fail | needs_patch",
+  "critical_failures": [
+    {"id": "CF-1", "file": "path:line", "description": "...", "confidence": 95}
+  ],
+  "pseudocode_flags": [
+    {"file": "path:line", "reason": "...", "confidence": 88}
+  ],
+  "prd_mismatches": [
+    {"component": "...", "status": "partial|missing", "file_ref": "...", "confidence": 80}
+  ],
+  "broken_pipelines": [
+    {"pipeline": "...", "break_point": "...", "confidence": 70}
+  ],
+  "missing_components": [
+    {"component": "...", "required_by": "PRD section X", "confidence": 85}
+  ],
+  "risk_points": [
+    {"area": "...", "risk": "...", "confidence": 60}
+  ],
+  "verified_components": [
+    {"component": "...", "evidence": "file:line or test name", "confidence": 95}
+  ],
+  "evidence": {
+    "files_inspected": [],
+    "execution_paths_traced": [],
+    "tests_found": [],
+    "tests_missing": []
+  },
+  "required_next_actions": [
+    {"action": "...", "file_hint": "...", "confidence": 75}
+  ]
+}
+```
+
+**Calibration guide:**
+- 90–100: pattern seen repeatedly in shipped code; mechanical; low regression risk
+- 70–89: confident in direction, API shape or naming may vary
+- 50–69: plausible fix but may not match conventions, could cascade
+- <50: genuinely uncertain — record anyway so downstream knows to investigate
+
+---
+
+## FINAL DIRECTIVE
+
+You are not reviewing code.
+
+You are answering:
+
+> "Can this system be trusted to execute real-world DevOps tasks without hallucinating, bypassing validation, or collapsing under edge cases?"
+
+If the answer is not provably yes, the verdict is FAIL.
@@ -0,0 +1,95 @@
+# Scrum Loop Notes — Observations across iterations
+
+Running notes from the 6x scrum loop (started 2026-04-23). One section per iteration. "Fix next loop" items accumulate here so the next scrum run picks them up — do not fix inline during a running iteration.
+
+## Iteration tracker
+
+| Iter | Status | Scrum started | Scrum finished | Fixes applied | Build green | Re-sweep findings |
+|---|---|---|---|---|---|---|
+| 1 | 🟡 scrum running | 2026-04-23 (brqz3jxgo) | - | - | - | - (baseline = 19) |
+| 2 | 🟡 scrum running | 2026-04-23 (bzs6miehr) | - | - | - | pending |
+| 3 | ⬜ queued | - | - | - | - | - |
+| 4 | ⬜ queued | - | - | - | - | - |
+| 5 | ⬜ queued | - | - | - | - | - |
+| 6 | ⬜ queued | - | - | - | - | - |
+
+## Iteration 1 — in flight
+
+**Target files:** 21 source files extracted from the 19 Phase 0→42 findings.
+**Ladder:** cloud-first per feedback_scrum_cloud_first.md (gpt-oss:120b → qwen3.5:397b → devstral-2:123b → mistral-large-3:675b → gpt-oss:20b → qwen3.5:latest).
+**Proposal:** `docs/SCRUM_FIX_WAVE.md` (via LH_SCRUM_PROPOSAL env).
+
+### Fix next loop — observations accumulating
+
+*Add items here as the scrum runs. Keep each item to one line with a pointer to file + reason. Don't fix inline.*
+
+**[ITER 2 OBSERVATIONS]**
+- **[FORENSIC vs thin-detector mismatch]** iter 2 first attempt on auth.rs triggered "thin/unstructured" rejection at 2031 chars. Cause: forensic prompt asks for strict JSON verdict output, scrum's thin-answer detector expects markdown with score + table. The detector logic needs a forensic-aware branch OR the forensic prompt should preserve markdown output shape while still applying the 8 audit passes. File: `tests/real-world/scrum_master_pipeline.ts`, function that scores accepted vs thin. Fix next loop: add `isForensicAcceptable(text)` that checks for `"verdict"` field + at least one of `critical_failures`/`pseudocode_flags`/`required_next_actions`.
+
+- **[OBSERVATION metric]** 11 `#[allow(dead_code)]` markers cluster in crates/gateway/{auth,access,tools/registry,execution_loop,v1/truth} + crates/aibridge/providers/openrouter + crates/vectord/service. Each one maps cleanly to an audit finding. The `execution_loop/mod.rs:85` comment even admits it: `// reserved for Phase 42 truth-gate (step 6)`. **Metric:** fewer `#[allow(dead_code)]` markers per iteration = less pseudo-real code. Baseline = 11. Target after iter 6: ≤ 2 (only ones that are genuinely optional helpers).
+- **[OBSERVATION gateway-as-router]** scrum_master_pipeline currently fetches `GATEWAY/v1/chat` directly but its LADDER is still a hardcoded const. Should be driven by `config/routing.toml` via RoutingEngine (blocked by P40-001 until iter 1 lands fix). File: `tests/real-world/scrum_master_pipeline.ts:53`.
+- **[OBSERVATION file-type]** iter 1 target list is `.rs` only. Iter 2 must include `tests/multi-agent/*.ts` (executor, observer, kb consumer), `auditor/checks/*.ts`, `sidecar/sidecar/*.py`, and `config/*.{json,toml}`. The scrum pipeline handles any text file.
+- **[OBSERVATION triangulation]** auth.rs scrum review (first file out) independently identified P5-001 exactly: flagged `#[allow(dead_code)]`, scored alignment 4/10, prescribed an `AgentIdentity { name, role, hashed_key }` type matching SCRUM_FIX_WAVE. Audit + scrum converged without seeing each other's output — strong signal the findings are real, not artifacts of one method.
+- **[RULE from J 2026-04-23]** Wiring-gap fixes happen AFTER the scrum completes, not inline. Accumulate observations, apply in one coherent pass. Matches feedback_audit_findings_log.md.
+- **[OBSERVATION oversize-file]** `crates/gateway/src/execution_loop/mod.rs` is 80,901 chars → 24 shards (scrum pipeline's tree-split kicks in at 6KB threshold). A single-file-of-this-size for an execution module is itself a smell — it's the Phase 43 scaffold we kept piling into. Split candidates: executor prompts, reviewer prompts, budget accounting, truth-gate hook, fixtures. Not a fix for this iter, but queue for iter 3.
+- **[OBSERVATION cost-tracking]** zero escalations across first 8 files — 0.0 dollar cloud spend above the minimum. Per-request cost on gpt-oss:120b via Ollama Cloud is effectively $0 in this environment (self-hosted or flat-rate per the llm_team_config key). If we add per-iter token totals to scrum_loop_metrics.jsonl we can show trajectory even when cost is flat.
+
+**[ITER 3 OBSERVATIONS]**
+- **[LARGE-HANDLER thin]** kimi-k2:1t went thin on `crates/gateway/src/tools/service.rs` (~11KB, single large axum handler). deepseek-v3.1:671b rescued on attempt 2 (92.8s, 5408 chars, accepted). Pattern: very large routing files challenge even 1T models. Fix next loop: raise tree-split threshold for handler files OR shard by function boundaries not byte count.
+- **[WRITE-ONLY INDICATORS STILL]** 8 KB files write-only after iter 3: `audits.jsonl` (189 rows/1.9MB — biggest waste), `phase_sweep_findings.jsonl` (35), `distilled_facts.jsonl` (17), `human_overrides.jsonl` (8), `classifications.jsonl` (5), `scrum_loop_metrics.jsonl` (2), `distilled_config_hints.jsonl` (2), `distilled_procedures.jsonl` (2). Fix next loop: extend `auditor/checks/kb_query.ts` to surface these on PR review, OR build a single "KB health dashboard" reader.
+- **[ISOLATED AUTOTUNE]** `crates/vectord/src/agent.rs` has zero refs to scrum/audit/human_override KB. It tunes HNSW but doesn't know which indexes are attached to files the scrum flagged. Fix next loop: add `TriggerEvent::CodeReviewFlag { index_name, gradient_tier }` that biases trial budget toward indexes of flagged files.
+- **[CONFIDENCE WELL-CALIBRATED]** kimi-k2:1t confidences span 75-98 across iter3 files, cluster 85-95. No 100% inflation; min 75 = honest edge-case uncertainty. Good signal — the model is calibrating, not performance-signaling. Do NOT "fix" this by prompt-boosting confidence.
+- **[SCRUM→OBSERVER WIRED]** 2026-04-24 fix landed in iter-3 source but applies to iter 4+ (bun loaded code pre-edit). Verify next loop: `curl :3800/stats` should show `by_source.scrum > 0` after iter 4 runs. If zero, observer /event payload schema mismatch.
+- **[LOW-CONFIDENCE BLOCK flag WORKING]** `crates/storaged/src/federation_service.rs` scored 3/10 with confidences **[40, 35, 50]** → avg 42, min 35. Permission Gradient correctly classifies this file as `block` tier (min <50) — exactly the signal we want. kimi-k2:1t is honestly saying "I'm not confident about these suggestions." Human must review before any auto-apply. Validates Layer #6 working end-to-end. **DO NOT fix this** — it's the system protecting us.
+- **[CONFIDENCE-COLUMN DROPOUT]** `crates/gateway/src/main.rs` accepted review has 0 confidences extracted despite the prompt requiring them. Either kimi-k2:1t omitted the Confidence column in its table format (regex didn't match), or the specific content ran into token budget and confidence got trimmed. Fix next loop: strengthen the prompt instruction to say "if you omit the Confidence column you MUST provide Confidence: NN% inline after every suggestion" AND post-accept sanity-check the row (reject if findings_count=0 across 5+ suggestions).
+- **[SCORE-PARSER MISS on some files]** `crates/queryd/src/service.rs` and `crates/queryd/src/delta.rs` reviews return score='?' — regex `(\d)\s*/\s*10` isn't matching. Models may have written score as "Score: 4.5/10" or "4/10 (mid)" which the parser can't see. Fix next loop: regex accepts decimals + surrounding context. Not critical — confidences still landed.
+- **[SCORE DISTRIBUTION]** across 12 iter-3 files: 3/10 (7 files), 4/10 (2), 6/10 (1), ?/? (2). vectord/index_registry.rs alone scored 6/10 — meaningfully higher. That's worth a closer look: what does that file do right? If we find the pattern there and replicate to peers, scores should rise. This is learning-loop signal.
+
+- **[FORENSIC JSON OUTPUT IS RICHER THAN MARKDOWN]** kimi-k2:1t emitted strict JSON-schema output (matching SCRUM_FORENSIC_PROMPT.md) for `vectord/src/index_registry.rs`. Structure separates critical_failures / pseudocode_flags / prd_mismatches / broken_pipelines / missing_components / risk_points / **verified_components** / required_next_actions — each with confidence per entry. Markdown reviews never captured `verified_components` (what's confirmed working). Fix next loop: scrum pipeline detects JSON format in accepted output and extracts structured fields into KB row (`verified_components_count`, `critical_failures_count`, `missing_components_count`, `verdict`). Downstream consumers then filter PR files by `verdict: needs_patch` or prioritize by `missing_components_count`. **This is the biggest next-iter KB quality jump available** — goes from "confidence as a scalar" to "confidence per specific claim with evidence field." Verified_components in particular is the PROOF-OF-LIFE signal that tells us what's real — flipside of P9/P13/P42 gaps.
+
+- **[🔴 TREE-SPLIT REVIEW SHARD-LEAKAGE]** `ingestd/src/service.rs` (7 shards, 24.3KB) accepted output is titled "Forensic Audit Report – ... (shard 3)". The review covers only the Postgres-import path (shard 3). The `ingest_file` handler where my P9-001 fix lives (journal.record_ingest call) is in shard 1 or 2 — that reviewer never saw the fix. **`tree_split_fired: true` is supposed to mean the output is the reducer-merged summary of all shards**, but this review retained shard-specific scope. Either (a) the reduce step didn't integrate shard summaries, (b) the accepted attempt was one individual shard response that slipped past the reducer, or (c) the reducer prompt doesn't instruct the model to present the file holistically. **This is a real correctness bug** — it means file-level findings can be ghost-negative (fix applied, reviewer blind to it) and ghost-positive (gap exists in unreviewed shard, reviewer gives clean bill). Fix next loop: examine the reduce-step prompt in tree-split path, ensure accepted output comes from reduce step not from any individual shard. Validate by running on a file with a known fix and confirming the review notices it.
+
+**[ITER 4 OBSERVATIONS]**
+- **[🔴 OBSERVER DROPS SCRUM RICH FIELDS]** Scrum→observer wiring works (by_source.scrum appears in /stats on iter 4 file 1). BUT observer.ts:262-283 `ObservedOp = {...}` literal only spreads known keys (endpoint, success, duration_ms, role, city, state, count, rescue_*). My scrum-specific fields (confidence_avg, confidence_min, gradient_tier, verdict, critical_failures_count, verified_components_count, missing_components_count, alignment_score, output_format, findings_count, attempts_made, thin_rejections) are silently discarded. Observer knows the scrum event happened but loses review-quality data. Fix next loop: add `metadata?: Record<string, any>` passthrough on ObservedOp, or declare scrum-specific fields explicitly. Preferred: metadata passthrough so future sources (auditor, kb_extractor) land the same way.
+- **[SCHEMA V4 LANDING CORRECTLY]** main.rs iter-4 KB row has alignment_score=3 (decimal parser fixed), output_format="markdown" (classifier works), verdict=null (correct — only forensic_json produces verdict), confidence_avg=91 (previous iter got 0 due to column dropout — run-to-run variance self-healed this). Structured counters (critical/verified/missing) = 0 on markdown rows, populated on forensic_json rows.
+- **[RING BUFFER EVICTING LANGFUSE]** observer ring hit 2000 cap; first 2 scrum events pushed 2 langfuse entries out (1999 → 1997). Not a bug — ring works as designed — but means old-context retention is bounded. If we care about historical Langfuse traces we need a larger ring OR a separate per-source ring.
+- **[UI Playwright probe found 2 real bugs]** (fixed 2026-04-24): (a) ui/server.ts tryFetch relied on content-type header to decide JSON vs text; observer Bun.serve returns JSON without `application/json` content-type, so stats were strings — UI showed "0 ops" instead of 2000. Fixed: always attempt JSON.parse, fall back to raw text. (b) ui.js renderNodeContext used Object.entries(n.health) which iterates characters on a string — gateway /health returns "lakehouse ok" and the panel showed rows like `0=l, 1=a, 2=k, ...`. Fixed: primitive-vs-object guard. **Both were invisible in functional tests — only a real browser render exposed them.** Worth adding a Playwright smoke test to CI for any future UI changes.
+
+### Iter 1 results
+
+*Populated after scrum finishes.*
+
+## Iteration 2 — queued
+
+**Prompt shape change (from J 2026-04-23):** iter 2+ uses `docs/SCRUM_FORENSIC_PROMPT.md` as the system prompt, replacing the softer iter-1 framing. Adversarial auditor tone with 8 audit passes. Strict JSON output format with `verdict: pass|fail|needs_patch`. If system can't prove itself, verdict is FAIL.
+
+**Scrum pipeline change:** `scrum_master_pipeline.ts` needs an env `LH_SCRUM_SYSTEM_PROMPT` (new) to inject the forensic frame alongside the proposal doc. The file-level loop still asks for suggestions per file but under the 8-pass adversarial lens.
+
+**Goal:** Self-host. Pipeline loads its ladder from `config/routing.toml` via the RoutingEngine that iter 1 wired. If that still isn't loaded, note gap, proceed with hardcoded ladder, flag for iter 3.
+**Target expansion:** beyond `.rs` to `.ts` (tests/multi-agent, auditor/), `.py` (sidecar), `.md` (docs).
+
+## Iterations 3-6 — queued
+
+**Goal:** measure trajectory. Each iteration reduces finding count, raises unit test count, reduces grep-for-fake-markers count. If any iteration doesn't improve, that's the data point.
+
+## Metrics per iteration
+
+Capture after each re-sweep:
+
+- `findings_total` (baseline: 19)
+- `findings_by_severity` (baseline: 3h / 8m / 8l)
+- `phases_partial_count` (baseline: 9)
+- `phases_real_count` (baseline: 25 of 35)
+- `rust_test_count` (baseline: 194+)
+- `gateway_test_fail_count` (baseline: 1 — P38-001)
+- `grep_hits_unimplemented` run: `grep -rEc 'todo!\(\)|unimplemented!\(\)|FIXME' crates/`
+- `grep_hits_pseudo` run: `grep -rEc '\"placeholder\"|\"stub\"|\"mock\"|\"fake\"' crates/`
+
+## Rules for this loop
+
+1. **Cloud-first for every iteration.** Per feedback_scrum_cloud_first.md, strategic review uses 120B+ tier.
+2. **One cross-cutting PR per iteration when possible.** Meta-pattern from audit: identity+auth+access+journal+truth share a pipe. Fix them together.
+3. **Build must be green before next iteration starts.** A broken build is evidence the last iteration regressed, not progressed.
+4. **Log findings to the jsonl as new rows per iteration** with `sweep_id: phase_sweep_2026-04-23-iterN`. Never overwrite prior iteration's findings — the trajectory is the whole point.
+5. **Don't fix things during an iteration.** Every observation goes into "Fix next loop" section above. Next iteration's scrum picks them up.
+
@@ -0,0 +1,275 @@
+# Scrum Master Pipeline — Spec + Current State
+
+**Status:** Active iteration on branch `scrum/auto-apply-19814` → PR #11 at git.agentview.dev/profit/lakehouse
+**Last iter:** 9 (2026-04-24)
+**Branch commit head:** `f4cff66` (ADR-021 Phase D fix)
+
+This doc is the single handoff artifact for the scrum-master + auto-apply + pathway-memory loop built during 2026-04-24 sessions. A fresh Claude Code session reading this + `docs/DECISIONS.md` (ADR-020 and ADR-021 specifically) + `MEMORY.md` should have the same context as the session that wrote it.
+
+## 1. What the loop is
+
+An autonomous review-and-commit pipeline that:
+
+1. **Scrum master** (`tests/real-world/scrum_master_pipeline.ts`) — walks a target-file list, asks a 9-rung escalation ladder of cloud models to produce a forensic audit against PRD + a change proposal doc, retries with learning context until acceptance, emits a structured review row.
+2. **Pathway memory** (`crates/vectord/src/pathway_memory.rs`) — stores the full backtrack context of each review (attempts, KB chunks, flags, bug fingerprints) indexed by a narrow fingerprint (`task_class + file_prefix + signal_class`). On every new review, it prepends historical bug patterns as a preamble so the reviewer preempts recurrences. Retired pathways auto-exclude themselves from hot-swap eligibility.
+3. **Auto-applier** (`tests/real-world/scrum_applier.ts`) — filters schema_v4 review rows by gradient_tier + confidence, asks `qwen3-coder:480b` for concrete `old_string/new_string` patches, runs `cargo check --workspace`, commits on green OR reverts on red/warning-count-up/rationale-mismatch.
+4. **Observer** (`mcp-server/observer.ts`) — receives per-file `/event` emissions, escalates failure clusters to LLM Team via `/v1/chat` with `qwen3-coder:480b`.
+5. **Auditor** (`auditor/audit.ts`) — external N=3 consensus re-check of scrum findings; writes to `data/_kb/audit_facts.jsonl`.
+
+The guiding principle: **every KB write has a reader, every PR claim is diff-verifiable.**
+
+## 2. The 9-rung ladder (cloud-first, strongest-model-first)
+
+Defined in `tests/real-world/scrum_master_pipeline.ts` at `const LADDER`:
+
+| # | Provider | Model | Role |
+|---|---|---|---|
+| 1 | ollama_cloud | `kimi-k2:1t` | flagship, 1T params |
+| 2 | ollama_cloud | `qwen3-coder:480b` | coding specialist, 480B |
+| 3 | ollama_cloud | `deepseek-v3.1:671b` | reasoning, 671B |
+| 4 | ollama_cloud | `mistral-large-3:675b` | deep analysis, 675B |
+| 5 | ollama_cloud | `gpt-oss:120b` | reliable workhorse |
+| 6 | ollama_cloud | `qwen3.5:397b` | dense 397B, final thinker |
+| 7 | openrouter | `openai/gpt-oss-120b:free` | free-tier rescue |
+| 8 | openrouter | `google/gemma-3-27b-it:free` | fastest rescue |
+| 9 | ollama | `qwen3.5:latest` | last-resort local |
+
+**Each attempt is evaluated by `isAcceptable()`** (chars ≥ 3800 AND not a malformed JSON-only dump). On reject, the next rung sees a learning preamble with the prior rejection reason.
+
+## 3. Tree-split reducer
+
+Files larger than `FILE_TREE_SPLIT_THRESHOLD = 6000` bytes get chunked into `FILE_SHARD_SIZE = 3500`-byte shards. Each shard gets summarized via a fast rung, summaries are concatenated with internal `§N§` markers, then fed as a SCRATCHPAD to the reviewer. The `§N§` markers are stripped before the reviewer sees the merged context so it cannot claim "(shard 3)" in titles.
+
+Bug regime this fixed: pre-tree-split iters had reviewers claim fields were "missing" because the field was past the 6KB context cutoff, not actually absent.
+
+## 4. Schema v4 KB rows
+
+`data/_kb/scrum_reviews.jsonl` — one row per accepted review. Fields:
+
+```json
+{
+  "file": "crates/queryd/src/service.rs",
+  "reviewed_at": "2026-04-24T11:06:56Z",
+  "accepted_model": "ollama_cloud/kimi-k2:1t",
+  "accepted_on_attempt": 1,
+  "attempts_made": 1,
+  "tree_split_fired": true,
+  "suggestions_preview": "<truncated-2000-char>",
+  "confidences_per_finding": [92, 90, 88, 85, 75],
+  "confidence_avg": 86,
+  "confidence_min": 75,
+  "findings_count": 5,
+  "gradient_tier": "dry_run",          // auto ≥90 / dry_run ≥70 / simulation ≥50 / block <50
+  "gradient_tier_avg": "dry_run",
+  "alignment_score": 3,                // 1-10 self-rated
+  "output_format": "forensic_json",
+  "verdict": "fail",                   // pass | needs_patch | fail
+  "critical_failures_count": 3,
+  "pseudocode_flags_count": 0,
+  "prd_mismatches_count": 4,
+  "missing_components_count": 6,
+  "verified_components_count": 2,
+  "risk_points_count": 3,
+  "schema_version": 4,
+  "scrum_master_reviewed": true,
+  // ADR-021 fields on pathway trace (NOT this row, see pathway_memory state.json)
+  "pathway_hot_swap_hit": false,
+  "pathway_id": null,
+  "pathway_similarity": null,
+  "pathway_success_rate": null,
+  "rungs_saved": 0
+}
+```
+
+## 5. Applier hardened gates (landed 5e8d87b)
+
+`tests/real-world/scrum_applier.ts` has **5 gates** between emitter output and commit. A patch must pass ALL:
+
+1. **Confidence gate** — emitter's self-reported `confidence >= MIN_CONF` (default 90; 85 with relaxed env). Rejected patches log reason `confidence NN < MM`.
+2. **Size gate** — max 6 lines changed per patch. Prevents cascading cross-file refactors.
+3. **Cargo-green gate** — `cargo check --workspace` must pass. Red build → `git checkout -- file`.
+4. **Warning-count gate** — workspace baseline warning count measured at start; after patch, new count must be `≤ baseline`. Catches unused-import additions and dead_code-after-allow-removed patterns. **THIS GATE CAUGHT 96b46cd's HashSet noise.**
+5. **Rationale-diff token alignment** — rationale text must share at least one non-stopword token with the patch's new_string. Catches "Add destructive SQL filter" claims on `use tracing;` diffs.
+
+Plus:
+- **Deny-list**: `config/`, `ops/`, `auditor/`, `docs/`, `data/`, `mcp-server/`, `ui/`, `sidecar/`, `scripts/` can't be auto-applied (human review required).
+- **Branch guard**: refuses to run on `main`.
+- **Dry-run workspace revert**: in `COMMIT=0` mode, file is reverted after check regardless of outcome — no state pollution between runs.
+
+Every decision logs to `data/_kb/auto_apply.jsonl` (action: `committed` / `build_red_reverted` / `warnings_increased_reverted` / `rationale_mismatch_reverted` / `all_rejected` / `no_patches` / `dry_run_would_commit`).
+
+## 6. Pathway memory (ADR-021)
+
+**Full spec: `docs/DECISIONS.md` ADR-021. Code: `crates/vectord/src/pathway_memory.rs`.**
+
+Three-layer matrix index for compounding semantic-correctness signal:
+
+### Fingerprint (narrow)
+`pathway_id = SHA256(task_class + "|" + file_prefix + "|" + signal_class)` — first 2 path segments (`crates/queryd`) so related files in the same crate share pathways.
+
+### Embedding (similarity vector)
+32-bucket L2-normalized token hash. Tokens include: task_class, file_path, signal_class, per-attempt model+rung+accepted flag, KB chunk source_docs, observer class, bridge libraries, sub-pipeline calls, **semantic_flags**, and **bug_fingerprints (flag+pattern_key)**.
+
+**TS and Rust implementations byte-match** — verified by smoke test showing cosine=1.0 on same input tokens. This is load-bearing for the TS-written traces to be searchable against the Rust-indexed space.
+
+### Hot-swap gate (5-factor AND)
+```
+narrow_fingerprint_matches
+AND audit_consensus.pass != false (null OK during bootstrap)
+AND replay_count >= 3 (probation)
+AND success_rate >= 0.80
+AND NOT retired
+AND similarity(query_vec, stored.pathway_vec) >= 0.90
+```
+
+Replay bookkeeping: on hot-swap, `replay_count++`; if the recommended model succeeded, `replays_succeeded++`; if `replay_count >= 3 AND success_rate < 0.80` → `retired = true` (sticky — prevents oscillation on noise).
+
+### Semantic-correctness layer (ADR-021)
+Each `PathwayTrace` carries:
+- `semantic_flags: Vec<SemanticFlag>` — one of 9 variants: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`
+- `bug_fingerprints: Vec<BugFingerprint>` — `{flag, pattern_key, example, occurrences}` where `pattern_key = "{Flag}:{sorted-top-3-identifiers-joined-by-hyphen}"`. Stable across prose variation.
+- `type_hints_used: Vec<TypeHint>` — `{source, symbol, type_repr}`. Phase E (not yet populated).
+
+**Pre-review enrichment**: scrum calls `POST /vectors/pathway/bug_fingerprints` with `{task_class, file_path, signal_class, limit}` — returns aggregated fingerprints sorted by occurrences descending. If any, a `📚 PATHWAY MEMORY` preamble is prepended to the reviewer prompt with "this file area had these patterns before — check for recurrences."
+
+**Post-review extractor** (Phase D, `scrum_master_pipeline.ts`): walks reviewer markdown line-by-line, finds lines containing a `SemanticFlag` variant, extracts identifier-shaped backtick-quoted tokens, filters out flag names + Rust keywords (self/mut/async/etc), sorts and takes top 3, builds `pattern_key = "{Flag}:{tokens}"`.
+
+### HTTP surface (on gateway port 3100)
+| Endpoint | Purpose |
+|---|---|
+| `POST /vectors/pathway/insert` | write a full PathwayTrace |
+| `POST /vectors/pathway/query` | hot-swap candidate check (returns `{candidate: null}` or `{candidate: {...}}`) |
+| `POST /vectors/pathway/record_replay` | update replay_count + success_rate after hot-swap |
+| `GET /vectors/pathway/stats` | totals + reuse_rate + replay_success_rate |
+| `POST /vectors/pathway/bug_fingerprints` | aggregated fingerprints by narrow fingerprint (for pre-review preamble) |
+
+### State persistence
+`data/_pathway_memory/state.json` — JSON dump of all buckets. Loaded at gateway boot (`crates/gateway/src/main.rs` has `pwm.load_from_storage().await`).
+
+## 7. Current state (2026-04-24 end of session)
+
+### Commits on branch `scrum/auto-apply-19814` since iter-5 baseline
+
+| # | SHA | Subject |
+|---|---|---|
+| 1 | `25ea3de` | observer fix — route LLM Team escalation to `/v1/chat` qwen3-coder |
+| 2 | `8b77d67` | OpenRouter rescue ladder + tree-split reducer + first auto-applier |
+| 3 | `96b46cd` | first auto-applied commit (later found misleading) |
+| 4 | `5e8d87b` | cleanup + applier hardening (warning + rationale + dry-run gates) |
+| 5 | `9cc0ceb` | P42-002 — truth gate into queryd `/sql` + `/paged` paths |
+| 6 | `2f8b347` | pathway_memory base (PathwayTrace, hot-swap, 18 tests) |
+| 7 | `86901f8` | queryd/delta.rs 6-line unit-mismatch fix |
+| 8 | `92df0e9` | ADR-021 spec |
+| 9 | `0a0843b` | ADR-021 Phases A+B+C (semantic_flags, prompt tags, preamble endpoint) |
+| 10 | `ee31424` | ADR-021 Phase D (fingerprint extractor) |
+| 11 | `f4cff66` | Phase D fix — strip flag names + Rust keywords from pattern_keys |
+
+### Matrix index state
+- **12 pathway traces** in `data/_pathway_memory/state.json`
+- **11 distinct bug fingerprints** across 4 Flag categories on `crates/queryd` narrow fingerprint (1 manually seeded + 10 extracted)
+- **0 hot-swaps fired** (probation requires ≥3 replays per pathway; none reached yet)
+
+### Active in-flight
+- Iter 9 complete; next iter 10+ will use cleaner fingerprint extractor (`f4cff66`)
+- 4 "noisy" pattern_keys from iter-9-file-1 pre-fix run (e.g., `DeadCode:DeadCode`) — dormant, won't match future output, acceptable dead entries
+
+### Queued (not yet implemented)
+- **Phase E** — `type_hints_used` population from `catalogd` column types, Arrow `RecordBatch.schema()`, Rust struct field types. Feeds typed context to reviewer prompt.
+- **Auditor → pathway audit_consensus wire** — activates the strict-audit gate (currently lenient: null bootstraps, only explicit `false` blocks).
+- **VCP UI cards** for "top bug fingerprints in last N iters" + "new patterns learned this iter"
+
+## 8. How to run a new iteration
+
+```bash
+# Default 3 files (playbook_memory.rs, doc_drift.rs, auditor/audit.ts)
+LH_SCRUM_FORENSIC=/home/profit/lakehouse/docs/SCRUM_FORENSIC_PROMPT.md \
+LH_SCRUM_PROPOSAL=/home/profit/lakehouse/docs/SCRUM_FIX_WAVE.md \
+bun run tests/real-world/scrum_master_pipeline.ts
+
+# Targeted files:
+LH_SCRUM_FILES="/home/profit/lakehouse/crates/queryd/src/delta.rs,/home/profit/lakehouse/crates/queryd/src/service.rs" \
+LH_SCRUM_FORENSIC=... LH_SCRUM_PROPOSAL=... \
+bun run tests/real-world/scrum_master_pipeline.ts
+
+# Dry-run auto-applier against the latest scrum output:
+LH_APPLIER_MIN_CONF=85 LH_APPLIER_MAX_FILES=10 \
+LH_APPLIER_MODEL=qwen3-coder:480b \
+LH_APPLIER_BRANCH=scrum/auto-apply-19814 \
+bun run tests/real-world/scrum_applier.ts
+
+# Actually commit (ONLY after dry-run looks clean):
+LH_APPLIER_COMMIT=1 LH_APPLIER_MIN_CONF=85 LH_APPLIER_MAX_FILES=10 \
+LH_APPLIER_MODEL=qwen3-coder:480b \
+LH_APPLIER_BRANCH=scrum/auto-apply-19814 \
+bun run tests/real-world/scrum_applier.ts
+```
+
+## 9. Verify services before running
+
+```bash
+# Gateway (port 3100) — must be up; pathway endpoints are here
+curl -s http://localhost:3100/health            # "lakehouse ok"
+curl -s http://localhost:3100/vectors/pathway/stats   # pathway memory totals
+
+# UI (port 3950) — VCP dashboard + /data/pathway_stats aggregation
+curl -s http://localhost:3950/data/pathway_stats
+
+# Observer (port 3800) — event receiver + LLM Team escalation
+curl -s http://localhost:3800/health 2>/dev/null || true
+
+# Sidecar (port 3200) — Python embed
+curl -s http://localhost:3200/health 2>/dev/null || true
+
+# LLM Team (port 5000) — /api/run?mode=extract ONLY registered mode
+# (others like code_review/patch/refactor return "Unknown mode")
+curl -s http://localhost:5000/health 2>/dev/null || true
+```
+
+If gateway missing new routes after code change: `cargo build --release -p gateway && sudo systemctl restart lakehouse.service`.
+
+If UI missing new routes: kill old `bun run ui/server.ts` and restart (not a systemd service right now).
+
+## 10. Where things live (code pointers)
+
+| Concern | File |
+|---|---|
+| Scrum orchestrator | `tests/real-world/scrum_master_pipeline.ts` |
+| Scrum ladder constant | same file, `const LADDER` line ~92 |
+| Tree-split reducer | same file, `async function treeSplitFile` |
+| Forensic prompt preamble (loaded via env) | `docs/SCRUM_FORENSIC_PROMPT.md` |
+| Fix-wave proposal preamble | `docs/SCRUM_FIX_WAVE.md` |
+| Scrum iter notes | `docs/SCRUM_LOOP_NOTES.md` |
+| Auto-applier | `tests/real-world/scrum_applier.ts` |
+| Applier audit trail | `data/_kb/auto_apply.jsonl` |
+| Scrum reviews KB | `data/_kb/scrum_reviews.jsonl` |
+| Model trust journal | `data/_kb/model_trust.jsonl` |
+| Pathway memory module | `crates/vectord/src/pathway_memory.rs` |
+| Pathway HTTP handlers | `crates/vectord/src/service.rs` (bottom) |
+| Pathway state on disk | `data/_pathway_memory/state.json` |
+| VCP UI server | `ui/server.ts` |
+| VCP UI client | `ui/ui.js` + `ui/ui.css` + `ui/index.html` |
+| Observer | `mcp-server/observer.ts` |
+| Auditor | `auditor/audit.ts` |
+| LLM Team extract client | `auditor/fact_extractor.ts` |
+| ADR-021 spec | `docs/DECISIONS.md` ADR-021 |
+
+## 11. Key memory files a fresh session should read
+
+From `/root/.claude/projects/-home-profit/memory/`:
+
+- `project_scrum_pipeline.md` — updated state of the scrum iterations
+- `project_first_auto_apply.md` — 96b46cd story + cleanup + hardening evidence from iter 7
+- `feedback_semantic_correctness_via_matrix.md` — J's insight on compounding, the ADR-021 rule
+- `feedback_endpoint_probe_discipline.md` — GET 405 is not endpoint validation
+- `reference_llm_team_modes.md` — only `extract` is registered on port 5000
+- `feedback_scrum_cloud_first.md` — scrum/audit/enrich pipelines use cloud first
+- `feedback_cloud_determinism.md` — cloud N=3 consensus + qwen3-coder tie-breaker
+
+## 12. Known gotchas
+
+- **Gateway restart needed after Rust route additions.** `sudo systemctl restart lakehouse.service` — the service is systemd-managed.
+- **UI server needs manual restart** after `ui/server.ts` changes (no systemd unit). Kill old `bun` pid, restart with `bun run ui/server.ts &`.
+- **LLM Team mode `code_review` doesn't exist** — only `extract` is registered in `/root/llm_team_ui.py`. Don't wire new features to "Unknown mode" endpoints. See `reference_llm_team_modes.md`.
+- **OpenRouter free-tier 429s during consensus probes** are normal (rate-limited upstream). In the production ladder they hit as last-resort rescue with seconds-to-minutes gap; different traffic pattern than rapid-fire consensus runs.
+- **Openrouter minimax-m2.5:free has a 45s timeout** — not in ladder, only for one-off probes.
+- **Probation period is 3 replays** before hot-swap can fire. On a fresh install, no hot-swap fires until a pathway has been re-visited ≥3 times.
@@ -0,0 +1,83 @@
+# Future Expansion — Advanced System Evolution Layers
+
+Adopted 2026-04-24 from J. The system stops optimizing for task completion. It optimizes for **provable execution, repeatable outcomes, resilience under drift, failure, and adversarial conditions.**
+
+## Layer roster + iteration mapping
+
+| # | Layer | Short form | Target iter |
+|---|---|---|---:|
+| 1 | Counterfactual Execution | Generate synthetic failure variants from each success | iter 5 |
+| 2 | Model Trust Profiling | Per-(model, task_type) success rate → routing weight | **iter 3** |
+| 3 | Execution DNA | Compress successful runs into reusable patterns | iter 4 |
+| 4 | Drift Sentinel | Re-validate historical tasks on a schedule | iter 5 |
+| 5 | Adversarial Injection | Inject poisoned context / malformed outputs / conflicts | iter 6 |
+| 6 | Permission Gradient | Confidence → execution tier (≥0.9 full, ≥0.7 dry-run, ≥0.5 sim, <0.5 block) | **iter 3** |
+| 7 | Multi-Agent Disagreement | Planner/Critic/Validator — disagreement = signal | iter 4 |
+| 8 | Temporal Context | Time-aware memory with decay_score + last_validated_at | iter 4 |
+| 9 | Execution Cost Intelligence | Tokens, iterations, cloud_calls, latency per task | **iter 3** |
+| 10 | Human Override as Data | Capture manual fixes as jsonl rows | **iter 3** |
+
+## Detail (J's original framing preserved)
+
+### 1. Counterfactual Execution Layer
+Simulate alternate failure paths for every successful task. Real Execution → Success → Generate Variations (env, version, inputs) → Simulate Failure Cases → Store Synthetic Failure Signatures. **Purpose:** pre-train against unseen failures before real exposure.
+
+### 2. Model Trust Profiling ← iter 3
+Per-(model, task_type) performance tracking.
+```
+{ "model": "...", "task_type": "...", "success_rate": 0.0, "failure_modes": [], "trust_score": 0.0 }
+```
+**Usage:** route by trust score, adjust validation strictness dynamically, per-model risk budgets.
+
+### 3. Execution DNA (Trace Compression)
+Successful executions → reusable fragments.
+```
+{ "dna_id": "hash", "task_signature": "...", "critical_steps": [], "failure_avoidance": [] }
+```
+Replaces doc retrieval with pattern retrieval; faster convergence on similar tasks.
+
+### 4. Drift Sentinel
+Select Historical Task → Re-run Current Env → Compare → If Failure → Mark Drifted → Trigger Re-learning. Detect silent decay; maintain long-term reliability.
+
+### 5. Adversarial Injection Engine
+Inject malformed outputs / outdated docs / conflicting instructions / poisoned memory. Verify validation catches, execution blocks unsafe actions, memory rejects corrupted data. Build system immunity.
+
+### 6. Permission Gradient Execution ← iter 3
+Confidence-based control replacing binary:
+- confidence ≥ 0.9 → full execution
+- confidence ≥ 0.7 → dry-run + diff
+- confidence ≥ 0.5 → simulation only
+- confidence < 0.5 → block
+Inputs: validation score, model trust score, memory match confidence. Risk-aware control; reduced catastrophic-failure surface.
+
+### 7. Multi-Agent Disagreement Engine
+Planner / Critic / Validator; disagreement triggers more context, bigger model, stricter validation. Disagreement is signal, not noise.
+
+### 8. Temporal Context Layer
+```
+{ "created_at": "ts", "last_validated_at": "ts", "decay_score": 0.0 }
+```
+Retrieval priority: recent + validated + high success rate. Avoid stale knowledge.
+
+### 9. Execution Cost Intelligence ← iter 3
+```
+{ "task": "...", "tokens_used": 0, "iterations": 0, "cloud_calls": 0, "latency_ms": 0 }
+```
+Optimize local vs cloud; reduce unnecessary iterations.
+
+### 10. Human Override as Data ← iter 3
+```
+{ "human_fix": "...", "reason": "...", "task_signature": "...", "validated": true }
+```
+Manual fixes become reusable knowledge.
+
+## Final Principle
+
+Memory is not passive recall. It is operational substrate:
+- failures become structured knowledge
+- successes become reusable execution patterns
+- all outputs are validated before reuse
+
+## System Directive
+
+Not speed. Not convenience. **Correctness. Verifiability. Resilience under change.**