Federation foundation + HNSW trial system + Postgres streaming + PRD reframe

Four shipped features and a PRD realignment, all measured end-to-end: HNSW trial system (Phase 15 horizon item → complete) - vectord: EmbeddingCache, harness (eval sets + brute-force ground truth), TrialJournal, parameterized HnswConfig on build_index_with_config - /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best, /hnsw/evals/{name}/autogen, /hnsw/cache/stats - Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us at 100% recall@10. ec=80 es=30 locked as HnswConfig::default() - Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s, 80/30 = 1.00 recall in 230s Catalog manifest repair - catalogd: resync_from_parquet reads parquet footers to restore row_count and columns on drifted manifests - POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing - All 7 staffing tables recovered to PRD-matching 2,469,278 rows Federation foundation (ADR-017) - shared::secrets: SecretsProvider trait + FileSecretsProvider (reads /etc/lakehouse/secrets.toml, enforces 0600 perms) - storaged::registry::BucketRegistry — multi-bucket resolution with rescue_bucket read fallback and reachability probing - storaged::error_journal — bucket op failures visible in one HTTP call - storaged::append_log — write-once batched append pattern (fixes the RMW anti-pattern llms3.com calls out; errors and trial journals both use it) - /storage/buckets, /storage/errors, /storage/bucket-health, /storage/errors/{flush,compact} - Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with X-Lakehouse-Rescue-Used observability headers on fallback Postgres streaming ingest - ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination into ArrowWriter, lineage redacts password - POST /ingest/db — verified against live knowledge_base.team_runs (586 rows × 13 cols, 6 batches, 196ms end-to-end) PRD realignment (2026-04-16) - Dual use case: staffing analytics + local LLM knowledge substrate - Removed "multi-tenancy (single-owner system)" from non-goals - Added invariants 8-11: indexes hot-swappable, per-reader profiles, trials-as-data, operational failures findable in one HTTP call - New phases 16 (hot-swap generations), 17 (model profiles + dataset bindings), 18 (Lance vs Parquet+sidecar evaluation) - Known ceilings table documents the 5M vector wall and escape hatches - ADR-017 (federation), ADR-018 (append-log pattern) added - EXECUTION_PLAN.md sequences phases B-E with success gates and decision rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 01:50:05 -05:00
parent 84407eeb51
commit dbe00d018f
33 changed files with 3305 additions and 52 deletions
@@ -0,0 +1,126 @@
+/// HTTP surface for federation operator endpoints.
+///
+/// - `GET /buckets` — configured buckets with reachability status
+/// - `GET /errors` — recent bucket op failures (filterable)
+/// - `GET /bucket-health` — aggregated errors-per-bucket in the last 5 minutes
+///
+/// Mounted under `/storage` by the gateway. `/storage/health` is already
+/// claimed by the existing storage router for service liveness, so the
+/// aggregated bucket health lives at `/storage/bucket-health`.
+
+use axum::{
+    Json, Router,
+    extract::{Path, Query, State},
+    http::StatusCode,
+    response::IntoResponse,
+    routing::{get, put},
+};
+use chrono::{DateTime, Utc};
+use serde::Deserialize;
+use std::sync::Arc;
+
+use crate::error_journal::BucketErrorEvent;
+use crate::registry::{BucketInfo, BucketRegistry};
+
+pub fn router(registry: Arc<BucketRegistry>) -> Router {
+    Router::new()
+        .route("/buckets", get(list_buckets))
+        .route("/errors", get(list_errors))
+        .route("/errors/flush", axum::routing::post(flush_errors))
+        .route("/errors/compact", axum::routing::post(compact_errors))
+        .route("/bucket-health", get(get_health))
+        .route("/buckets/{bucket}/objects/{*key}", put(put_bucket_object))
+        .route("/buckets/{bucket}/objects/{*key}", get(get_bucket_object))
+        .with_state(registry)
+}
+
+async fn list_buckets(State(reg): State<Arc<BucketRegistry>>) -> Json<Vec<BucketInfo>> {
+    Json(reg.list().await)
+}
+
+#[derive(Deserialize)]
+struct ErrorQuery {
+    bucket: Option<String>,
+    since: Option<DateTime<Utc>>,
+    #[serde(default = "default_limit")]
+    limit: usize,
+}
+
+fn default_limit() -> usize { 50 }
+
+async fn list_errors(
+    State(reg): State<Arc<BucketRegistry>>,
+    Query(q): Query<ErrorQuery>,
+) -> Json<Vec<BucketErrorEvent>> {
+    Json(reg.journal().filter(q.bucket.as_deref(), q.since, q.limit).await)
+}
+
+#[derive(Deserialize)]
+struct HealthQuery {
+    #[serde(default = "default_period")]
+    minutes: i64,
+}
+
+fn default_period() -> i64 { 5 }
+
+async fn get_health(
+    State(reg): State<Arc<BucketRegistry>>,
+    Query(q): Query<HealthQuery>,
+) -> impl IntoResponse {
+    Json(reg.journal().health(q.minutes).await)
+}
+
+/// Bucket-aware write. Hard-fails if the target bucket is unreachable;
+/// never falls back to rescue for writes.
+async fn put_bucket_object(
+    State(reg): State<Arc<BucketRegistry>>,
+    Path((bucket, key)): Path<(String, String)>,
+    body: bytes::Bytes,
+) -> impl IntoResponse {
+    match reg.write_smart(&bucket, &key, body).await {
+        Ok(()) => (StatusCode::CREATED, format!("stored: {bucket}/{key}")).into_response(),
+        Err(e) => (StatusCode::SERVICE_UNAVAILABLE, e).into_response(),
+    }
+}
+
+/// Bucket-aware read. Falls through to the rescue bucket if the target
+/// is unreachable or the key is missing; emits observability headers so
+/// callers can detect the fallback.
+async fn flush_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
+    match reg.journal().flush().await {
+        Ok(()) => (StatusCode::OK, "flushed").into_response(),
+        Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
+    }
+}
+
+async fn compact_errors(State(reg): State<Arc<BucketRegistry>>) -> impl IntoResponse {
+    match reg.journal().compact().await {
+        Ok(stats) => Json(stats).into_response(),
+        Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e).into_response(),
+    }
+}
+
+async fn get_bucket_object(
+    State(reg): State<Arc<BucketRegistry>>,
+    Path((bucket, key)): Path<(String, String)>,
+) -> impl IntoResponse {
+    match reg.read_smart(&bucket, &key).await {
+        Ok(outcome) => {
+            let mut resp = outcome.data.into_response();
+            if outcome.rescued {
+                let headers = resp.headers_mut();
+                headers.insert("x-lakehouse-rescue-used", "true".parse().unwrap());
+                headers.insert(
+                    "x-lakehouse-original-bucket",
+                    outcome.original_bucket.parse().unwrap(),
+                );
+                headers.insert(
+                    "x-lakehouse-served-by",
+                    outcome.served_by.parse().unwrap(),
+                );
+            }
+            resp
+        }
+        Err(e) => (StatusCode::NOT_FOUND, e).into_response(),
+    }
+}