Phase C: Decoupled embedding refresh

Implements the llms3.com-inspired pattern: embeddings refresh
asynchronously, decoupled from transactional row writes. New rows arrive,
ingest marks the vector index stale, a later refresh embeds only the
delta (doc_ids not already in the index).

Schema additions (DatasetManifest):
- last_embedded_at: Option<DateTime> - when the index was last refreshed
- embedding_stale_since: Option<DateTime> - set when data written, cleared on refresh
- embedding_refresh_policy: Option<RefreshPolicy> - Manual | OnAppend | Scheduled

Ingest paths (pipeline::ingest_file + pg_stream) call
registry.mark_embeddings_stale after writing. No-op if the dataset has
never been embedded — stale semantics only kick in once last_embedded_at
is set.

Refresh pipeline (vectord::refresh::refresh_index):
- Reads the dataset Parquet, extracts (doc_id, text) pairs
- Accepts Utf8 / Int32 / Int64 id columns (covers both CSV and pg schemas)
- Loads existing embeddings via EmbeddingCache (empty on first-time build)
- Filters to rows whose doc_id is NOT in the existing set
- Chunks (chunker::chunk_column), embeds via Ollama (batches of 32),
  writes combined index, clears stale flag

Endpoints:
- POST /vectors/refresh/{dataset_name} - body {index_name, id_column,
  text_column, chunk_size?, overlap?}
- GET /vectors/stale - lists datasets whose embedding_stale_since is set

End-to-end verified on threat_intel (knowledge_base.threat_intel):
- Initial refresh: 20 rows -> 20 chunks -> embedded in 2.1s,
  last_embedded_at set
- Idempotent second refresh: 0 new docs -> 1.8ms (pure delta check)
- Re-ingest to 54 rows: mark_embeddings_stale fires -> stale_since set
- /vectors/stale surfaces threat_intel with timestamps + policy
- Delta refresh: 34 new docs embedded in 970ms (6x faster than full
  re-embed); stale_cleared = true

Not in MVP scope:
- UPDATE semantics (same doc_id, different content) - would need
  per-row content hashing
- OnAppend policy auto-trigger - just declares intent; actual scheduler
  deferred
- Scheduler runtime - the Scheduled(cron) variant declares the intent so
  operators can see which datasets expect what, but the cron itself is
  separate

Per ADR-019: when a profile switches to vector_backend=Lance, this
refresh path benefits — Lance's native append replaces our "read all +
rewrite" Parquet rebuild pattern. Current MVP works well enough at
~500-5K rows to validate the architecture; Lance unblocks the 5M+ case.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/service.rs

@@ -10,7 +10,8 @@ use serde::{Deserialize, Serialize};
 use std::sync::Arc;
 
 use aibridge::client::{AiClient, EmbedRequest};
-use crate::{chunker, embedding_cache, harness, hnsw, index_registry, jobs, rag, search, store, supervisor, trial};
+use catalogd::registry::Registry as CatalogRegistry;
+use crate::{chunker, embedding_cache, harness, hnsw, index_registry, jobs, rag, refresh, search, store, supervisor, trial};
 
 #[derive(Clone)]
 pub struct VectorState {
@@ -21,6 +22,9 @@ pub struct VectorState {
     pub hnsw_store: hnsw::HnswStore,
     pub embedding_cache: embedding_cache::EmbeddingCache,
     pub trial_journal: trial::TrialJournal,
+    /// Catalog registry — needed by the Phase C refresh path to mark/clear
+    /// staleness and look up dataset manifests.
+    pub catalog: CatalogRegistry,
 }
 
 pub fn router(state: VectorState) -> Router {
@@ -47,6 +51,9 @@ pub fn router(state: VectorState) -> Router {
         // Cache management
         .route("/hnsw/cache/stats", get(cache_stats))
         .route("/hnsw/cache/{index_name}", axum::routing::delete(cache_evict))
+        // Phase C: embedding refresh
+        .route("/refresh/{dataset_name}", post(refresh_dataset))
+        .route("/stale", get(list_stale))
         .with_state(state)
 }
 
@@ -666,3 +673,61 @@ async fn cache_evict(
     let ok = state.embedding_cache.evict(&index_name).await;
     Json(serde_json::json!({ "evicted": ok, "index_name": index_name }))
 }
+
+// --- Phase C: embedding refresh ---
+//
+// Decouples "new row data arrived" from "re-embed everything." Ingest marks
+// a dataset's embeddings stale (see catalogd::registry::mark_embeddings_stale);
+// `/vectors/refresh/{dataset}` diffs existing embeddings against current
+// rows, embeds only the new ones, appends to the index, and clears the
+// stale flag.
+
+async fn refresh_dataset(
+    State(state): State<VectorState>,
+    Path(dataset_name): Path<String>,
+    Json(req): Json<refresh::RefreshRequest>,
+) -> Result<Json<refresh::RefreshResult>, (StatusCode, String)> {
+    tracing::info!(
+        "refresh requested for dataset '{}' -> index '{}'",
+        dataset_name, req.index_name,
+    );
+    match refresh::refresh_index(
+        &dataset_name,
+        &req,
+        &state.store,
+        &state.catalog,
+        &state.ai_client,
+        &state.embedding_cache,
+        &state.index_registry,
+    )
+    .await
+    {
+        Ok(result) => Ok(Json(result)),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}
+
+#[derive(Serialize)]
+struct StaleEntry {
+    dataset_name: String,
+    last_embedded_at: Option<String>,
+    stale_since: String,
+    refresh_policy: Option<shared::types::RefreshPolicy>,
+}
+
+async fn list_stale(State(state): State<VectorState>) -> impl IntoResponse {
+    let datasets = state.catalog.stale_datasets().await;
+    let entries: Vec<StaleEntry> = datasets
+        .into_iter()
+        .map(|d| StaleEntry {
+            dataset_name: d.name,
+            last_embedded_at: d.last_embedded_at.map(|t| t.to_rfc3339()),
+            stale_since: d
+                .embedding_stale_since
+                .map(|t| t.to_rfc3339())
+                .unwrap_or_default(),
+            refresh_policy: d.embedding_refresh_policy,
+        })
+        .collect();
+    Json(entries)
+}