Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end:

HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
  TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
  /hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
  at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
  80/30 = 1.00 recall in 230s

Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
  and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows

Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
  /etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
  rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
  anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
  /storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
  X-Lakehouse-Rescue-Used observability headers on fallback

Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
  into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
  (586 rows × 13 cols, 6 batches, 196ms end-to-end)

PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
  trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
  bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
  decision rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/storaged/src/append_log.rs

@@ -0,0 +1,239 @@
+/// Write-once batched append log.
+///
+/// Problem we're fixing: the error journal and HNSW trial journal both
+/// previously did read-modify-write of their whole JSONL file on every
+/// event. That's O(N²) cumulative work and generates huge churn at scale.
+/// It's exactly the pattern llms3.com flags as the "small-file /
+/// rewrite-amplification" anti-pattern.
+///
+/// This helper implements the pattern object storage actually wants:
+///
+/// - Events accumulate in an **in-memory buffer** (reads see them immediately).
+/// - When the buffer hits a threshold, or `flush()` is called, the buffer is
+///   written **as one new object** with a timestamp-sorted key.
+/// - Existing objects are never rewritten.
+/// - Reads enumerate all batch files, sort by key, and concat in order.
+/// - An explicit `compact()` reads every batch file, writes one consolidated
+///   file, and deletes the originals — the LSM-tree compaction idea applied
+///   to small JSONL events.
+///
+/// Storage layout:
+/// ```
+/// {prefix}/
+///   batch_0001776319628000123.jsonl
+///   batch_0001776319745987654.jsonl
+///   batch_compacted_00001776319800000000.jsonl   (after compact())
+/// ```
+/// Key format: the zero-padded epoch microsecond of the write, so lexical
+/// sort == chronological sort.
+
+use bytes::Bytes;
+use chrono::Utc;
+use object_store::ObjectStore;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+use crate::ops;
+
+const DEFAULT_FLUSH_THRESHOLD: usize = 32;
+
+pub struct AppendLog {
+    store: Arc<dyn ObjectStore>,
+    prefix: String,
+    buffer: Mutex<Vec<Vec<u8>>>,
+    flush_threshold: usize,
+}
+
+impl AppendLog {
+    /// Create a new append log rooted at `prefix` in the given object store.
+    /// Events auto-flush when buffer reaches `flush_threshold` (default 32).
+    pub fn new(store: Arc<dyn ObjectStore>, prefix: impl Into<String>) -> Self {
+        Self {
+            store,
+            prefix: prefix.into(),
+            buffer: Mutex::new(Vec::new()),
+            flush_threshold: DEFAULT_FLUSH_THRESHOLD,
+        }
+    }
+
+    pub fn with_flush_threshold(mut self, threshold: usize) -> Self {
+        self.flush_threshold = threshold.max(1);
+        self
+    }
+
+    /// Add an event. The returned future completes either immediately
+    /// (buffered) or after a flush, depending on whether the buffer hit the
+    /// threshold. Callers don't need to care either way.
+    pub async fn append(&self, line: Vec<u8>) -> Result<(), String> {
+        let should_flush = {
+            let mut buf = self.buffer.lock().await;
+            buf.push(line);
+            buf.len() >= self.flush_threshold
+        };
+        if should_flush {
+            self.flush().await?;
+        }
+        Ok(())
+    }
+
+    /// Force-flush the in-memory buffer to object storage as a single new
+    /// batch file. Safe to call anytime (idempotent no-op when buffer empty).
+    pub async fn flush(&self) -> Result<(), String> {
+        let batch = {
+            let mut buf = self.buffer.lock().await;
+            if buf.is_empty() { return Ok(()); }
+            std::mem::take(&mut *buf)
+        };
+
+        let ts_us = Utc::now().timestamp_micros().max(0) as u128;
+        let key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
+
+        let mut body = Vec::with_capacity(batch.iter().map(|b| b.len() + 1).sum());
+        for line in batch {
+            body.extend_from_slice(&line);
+            if !body.ends_with(b"\n") { body.push(b'\n'); }
+        }
+        ops::put(&self.store, &key, Bytes::from(body)).await
+    }
+
+    /// Read every event across all batch files + unflushed in-memory buffer.
+    /// Events are returned in chronological order.
+    pub async fn read_all(&self) -> Result<Vec<Vec<u8>>, String> {
+        let mut keys = self.list_batch_keys().await?;
+        keys.sort();
+
+        let mut out = Vec::new();
+        for key in keys {
+            let bytes = ops::get(&self.store, &key).await?;
+            for line in bytes.split(|b| *b == b'\n') {
+                if !line.is_empty() {
+                    out.push(line.to_vec());
+                }
+            }
+        }
+
+        // Include unflushed events so callers see the latest state
+        // whether or not someone ran flush() recently.
+        let buf = self.buffer.lock().await;
+        for line in buf.iter() {
+            out.push(line.clone());
+        }
+
+        Ok(out)
+    }
+
+    /// Consolidate all current batch files into one compacted file, then
+    /// delete the originals. Safe to call while appends are in flight:
+    /// new batches written during compaction get a higher timestamp and
+    /// survive. Fails closed — if anything goes wrong mid-delete, the
+    /// compacted file coexists with originals and next read sees duplicates
+    /// (which the dedup caller must handle) rather than data loss.
+    pub async fn compact(&self) -> Result<CompactStats, String> {
+        // Snapshot which files to compact BEFORE we write the new one.
+        let mut originals = self.list_batch_keys().await?;
+        originals.sort();
+
+        if originals.len() < 2 {
+            return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
+        }
+
+        // Gather all existing events.
+        let mut events = Vec::new();
+        for key in &originals {
+            let bytes = ops::get(&self.store, key).await?;
+            for line in bytes.split(|b| *b == b'\n') {
+                if !line.is_empty() {
+                    events.push(line.to_vec());
+                }
+            }
+        }
+
+        let total_events = events.len();
+        if total_events == 0 {
+            // Clean up empty files without writing a new one.
+            for key in &originals {
+                let _ = ops::delete(&self.store, key).await;
+            }
+            return Ok(CompactStats { merged_files: originals.len(), events: 0, new_key: None });
+        }
+
+        // Name the compacted file with the SAME `batch_{ts}` format so it
+        // sorts chronologically with future batches. Using a distinct prefix
+        // ("batch_compacted_") would break lex ordering: later `batch_N`
+        // files would sort BEFORE the compacted file because 'c' > digits.
+        // Timestamp = now, so any appends arriving during compaction (which
+        // get the current wall-clock time) sort AFTER this file.
+        let ts_us = Utc::now().timestamp_micros().max(0) as u128;
+        let new_key = format!("{}/batch_{:019}.jsonl", self.prefix, ts_us);
+
+        let mut body = Vec::new();
+        for line in &events {
+            body.extend_from_slice(line);
+            body.push(b'\n');
+        }
+        ops::put(&self.store, &new_key, Bytes::from(body)).await?;
+
+        // Only delete originals once the consolidated file is persisted.
+        let mut failures = 0;
+        for key in &originals {
+            if ops::delete(&self.store, key).await.is_err() {
+                failures += 1;
+            }
+        }
+        if failures > 0 {
+            tracing::warn!(
+                "compact '{}': {} original files failed to delete — consolidated file {} has the data",
+                self.prefix, failures, new_key,
+            );
+        }
+
+        Ok(CompactStats {
+            merged_files: originals.len(),
+            events: total_events,
+            new_key: Some(new_key),
+        })
+    }
+
+    /// How many batch files exist on disk right now.
+    pub async fn file_count(&self) -> Result<usize, String> {
+        Ok(self.list_batch_keys().await?.len())
+    }
+
+    async fn list_batch_keys(&self) -> Result<Vec<String>, String> {
+        let prefix_with_slash = format!("{}/", self.prefix);
+        // list the prefix then filter for keys that match our naming scheme;
+        // unrelated files at the same prefix won't be touched.
+        let raw = ops::list(&self.store, Some(&prefix_with_slash)).await?;
+        Ok(raw
+            .into_iter()
+            .filter(|k| {
+                let basename = k.rsplit('/').next().unwrap_or(k);
+                basename.starts_with("batch_") && basename.ends_with(".jsonl")
+            })
+            .collect())
+    }
+}
+
+#[derive(Debug, Clone, serde::Serialize)]
+pub struct CompactStats {
+    pub merged_files: usize,
+    pub events: usize,
+    pub new_key: Option<String>,
+}
+
+// Proactively flush on drop, best-effort.
+// We can't `.await` in Drop; we spawn the flush on the tokio runtime if one
+// is available. If the runtime is already shutting down the flush is lost —
+// acceptable for these observability journals, which are hints not records.
+impl Drop for AppendLog {
+    fn drop(&mut self) {
+        let buf_len = self.buffer.try_lock().map(|b| b.len()).unwrap_or(0);
+        if buf_len == 0 { return; }
+        // Can't spawn from sync Drop on every runtime shape; log + move on.
+        tracing::debug!(
+            "append_log '{}' dropping with {} unflushed events",
+            self.prefix, buf_len,
+        );
+    }
+}
+