docs: sync ADR-019 + PRD + DECISIONS with 2026-05-02 substrate changes

ADR-019: closed the "re-bench when 10M corpus exists" follow-up. Added
"Follow-up: 10M re-bench (2026-05-02)" section with the post-fix numbers
(search ~20ms warm / ~46ms cold, doc-fetch ~5ms post-btree). Documented
the lance-bench-bypassing-IndexMeta bug + 2-layer fix + gauntlet
(7 unit + 12 sanitize + 10 smoke probes). Reframes the strategic
question as "Lance vs Parquet+HNSW-with-spilling" since HNSW doesn't
fit RAM at 10M.

DECISIONS: added ADR-022 — drop Python sidecar from Rust hot path.
Captures the rationale (236× embed perf gap was pure overhead),
co-shipped LRU cache, dev-only Python that survives, cross-runtime
parity verification, and the operator runbook signal (ps -ef ABSENT
post-deploy).

PRD: updated AI Boundary table line + aibridge crate description to
reflect direct Ollama path (was: Python FastAPI sidecar → Ollama).
Both lines reference ADR-022 for the full rationale.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/DECISIONS.md

@@ -104,3 +104,8 @@
 **Date:** 2026-04-24
 **Decision:** Extend `pathway_memory::PathwayTrace` (ADR added 2026-04-24 in same commit as this one) with a semantic-correctness layer so the matrix index compounds recognition of unit/type/shape bugs across iterations. Three new fields: `semantic_flags: Vec<SemanticFlag>` (enum: `UnitMismatch`, `TypeConfusion`, `NullableConfusion`, `OffByOne`, `StaleReference`, `PseudoImpl`, `DeadCode`, `WarningNoise`, `BoundaryViolation`), `type_hints_used: Vec<TypeHint>` (schema/type context the reviewer was given — catalogd column types for SQL-touching code, Arrow `RecordBatch.schema()` accessors for Rust, Rust struct field types for everything else), and `bug_fingerprints: Vec<BugFingerprint>` (structural pattern hash, e.g. `{lhs_unit: "rows", rhs_unit: "files", op: "-"}` → stable SHA for similarity retrieval). Scrum pipeline pre-review: query matrix index for bug fingerprints flagged on this file's narrow fingerprint (same `task_class + file_prefix + signal_class` as hot-swap) and prepend them to the reviewer prompt as "watch for these patterns historically found here." Reviewer prompt explicitly tags each finding with a `semantic_flag`. `truth::evaluate()` gets a review-time task_class (`code_review.unit_check`) that consumes parsed-fact rules like `FieldContainsAny { field: "code_expression", needles: ["row_count - file_count", "bytes_read - row_count"] }` — the same primitive we use for SQL guard in P42-002.
 **Rationale:** The 2026-04-24 `queryd/src/delta.rs` `base_rows = pre_filter_rows - delta_count` bug (86901f8) was found by a human reading the code and noticing units didn't match. The hardened mechanical applier *cannot* catch this — its gates are syntactic (warning count, patch size, rationale-token alignment) not semantic. At 100 bugs this deep, no human catches them all; the signal→commit loop is capped by what humans can notice per iteration. We already ship the primitives: `catalogd` knows column types per dataset, Arrow `RecordBatch.schema()` is on every hot-path call, `truth::evaluate()` runs arbitrary field conditions at runtime, `shared/arrow_helpers` has typed row/byte/file accessors. All of this is used at RUNTIME; none is fed into the REVIEW pipeline. Semantic flags + bug fingerprints turn the matrix index from "what review happened" (current) into "what category of bug appeared where" (compounding) — so iter-20 scrum on `crates/queryd/src/` preempts review prompts with "this crate had a row/file unit mismatch in iter 7 (delta.rs:189); check every arithmetic on `*_count` variables." Non-goals: we are NOT building a full type-inference engine (reuse Rust's `rustdoc`-level type info for structs, Arrow's schema for RecordBatch, catalogd's column types for SQL — everything beyond is Phase 3). Non-goals: this is not a linter — clippy/rustc already catch syntactic issues; this catches SEMANTIC ones (same type, wrong units/role). Bootstrap path: start with the 9 `SemanticFlag` variants above; add new variants only when a bug is found that doesn't fit an existing one. Gate alignment with hot-swap: a pathway that repeatedly produces bugs of the same `SemanticFlag` variant on the same narrow fingerprint is more valuable as a "watch this file for X" signal than as a hot-swap candidate — retirement logic needs to consider both replay success_rate AND whether the pathway is serving as a bug-pattern beacon.
+
+## ADR-022: Drop Python sidecar from Rust hot path; AiClient talks Ollama directly
+**Date:** 2026-05-02
+**Decision:** `crates/aibridge` AiClient was rewritten (commit `ba928b1`) to call Ollama HTTP directly: per-text `/api/embed`, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `nvidia-smi` for vram_snapshot. Public AiClient API is unchanged — 0 callers updated. The `sidecar/sidecar/{embed,generate,rerank,admin}.py` FastAPI hot-path routers are retired (kept on disk as historical reference only). The `sidecar/sidecar/{lab_ui,pipeline_lab}.py` ad-hoc Streamlit-shape dev UIs (~888 LOC) keep running as ad-hoc tooling; they are NOT on the runtime hot path. The systemd unit `lakehouse-sidecar.service` stays running for those dev UIs but the gateway no longer depends on it for any production call.
+**Rationale:** The Python sidecar embedded path was 236× slower than direct Ollama on warm workloads — the `/api/embed` HTTP hop + Python serialization tax was pure overhead with no logic added (sidecar was a pass-through translator). Co-shipped commit `150cc3b` added an in-process LRU embed cache to the gateway, dropping warm p50 from 78ms → 129us (cache hit) without changing the AiClient public API. Together: the gateway is one mega-process again instead of mega-process + Python sidecar, the Python runtime cost is paid only by dev tooling, and embed RPS is 236× higher on warm workloads. The dev UIs stayed because they're genuinely useful for ad-hoc embedding-pipeline experiments and don't run on the request path. PRD's "AI Boundary | Python FastAPI sidecar → Ollama HTTP API" line is updated to reflect the direct path; ADR-019's hybrid storage architecture is unaffected (this ADR changes the AI substrate, not the vector substrate). See cross-runtime mirror in Go-side `internal/aibridge` (Go was already direct-to-Ollama from D1 onward — this brings Rust to parity). Verified: `cargo test -p aibridge` 32/32 PASS, live `/ai/embed` returns 768d vector + `/v1/chat` returns "OK"; cross-runtime parity probe `embed_parity.sh` 8/8 PASS post-deploy. Cost: the Python sidecar was the canonical "model-version is loaded" probe — that capability moved into AiClient via `/api/ps` polling. Operator runbook: `ps -ef | grep "uvicorn sidecar"` should be ABSENT post-deploy on the production gateway box; presence indicates the old binary is still running.