Phase C: Decoupled embedding refresh

Implements the llms3.com-inspired pattern: embeddings refresh
asynchronously, decoupled from transactional row writes. New rows arrive,
ingest marks the vector index stale, a later refresh embeds only the
delta (doc_ids not already in the index).

Schema additions (DatasetManifest):
- last_embedded_at: Option<DateTime> - when the index was last refreshed
- embedding_stale_since: Option<DateTime> - set when data written, cleared on refresh
- embedding_refresh_policy: Option<RefreshPolicy> - Manual | OnAppend | Scheduled

Ingest paths (pipeline::ingest_file + pg_stream) call
registry.mark_embeddings_stale after writing. No-op if the dataset has
never been embedded — stale semantics only kick in once last_embedded_at
is set.

Refresh pipeline (vectord::refresh::refresh_index):
- Reads the dataset Parquet, extracts (doc_id, text) pairs
- Accepts Utf8 / Int32 / Int64 id columns (covers both CSV and pg schemas)
- Loads existing embeddings via EmbeddingCache (empty on first-time build)
- Filters to rows whose doc_id is NOT in the existing set
- Chunks (chunker::chunk_column), embeds via Ollama (batches of 32),
  writes combined index, clears stale flag

Endpoints:
- POST /vectors/refresh/{dataset_name} - body {index_name, id_column,
  text_column, chunk_size?, overlap?}
- GET /vectors/stale - lists datasets whose embedding_stale_since is set

End-to-end verified on threat_intel (knowledge_base.threat_intel):
- Initial refresh: 20 rows -> 20 chunks -> embedded in 2.1s,
  last_embedded_at set
- Idempotent second refresh: 0 new docs -> 1.8ms (pure delta check)
- Re-ingest to 54 rows: mark_embeddings_stale fires -> stale_since set
- /vectors/stale surfaces threat_intel with timestamps + policy
- Delta refresh: 34 new docs embedded in 970ms (6x faster than full
  re-embed); stale_cleared = true

Not in MVP scope:
- UPDATE semantics (same doc_id, different content) - would need
  per-row content hashing
- OnAppend policy auto-trigger - just declares intent; actual scheduler
  deferred
- Scheduler runtime - the Scheduled(cron) variant declares the intent so
  operators can see which datasets expect what, but the cron itself is
  separate

Per ADR-019: when a profile switches to vector_backend=Lance, this
refresh path benefits — Lance's native append replaces our "read all +
rewrite" Parquet rebuild pattern. Current MVP works well enough at
~500-5K rows to validate the architecture; Lance unblocks the 5M+ case.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/catalogd/src/registry.rs

@@ -1,6 +1,6 @@
 use shared::types::{
     DatasetId, DatasetManifest, ObjectRef, SchemaFingerprint,
-    ColumnMeta, Lineage, FreshnessContract, Sensitivity,
+    ColumnMeta, Lineage, FreshnessContract, RefreshPolicy, Sensitivity,
 };
 use std::collections::HashMap;
 use std::sync::Arc;
@@ -20,6 +20,8 @@ pub struct MetadataUpdate {
     pub lineage: Option<Lineage>,
     pub freshness: Option<FreshnessContract>,
     pub row_count: Option<u64>,
+    // Phase C embedding freshness
+    pub embedding_refresh_policy: Option<RefreshPolicy>,
 }
 
 const MANIFEST_PREFIX: &str = "_catalog/manifests";
@@ -78,6 +80,9 @@ impl Registry {
             freshness: None,
             tags: vec![],
             row_count: None,
+            last_embedded_at: None,
+            embedding_stale_since: None,
+            embedding_refresh_policy: None,
         };
 
         // Write-ahead: persist before in-memory update
@@ -111,6 +116,7 @@ impl Registry {
         if let Some(lineage) = updates.lineage { manifest.lineage = Some(lineage); }
         if let Some(freshness) = updates.freshness { manifest.freshness = Some(freshness); }
         if let Some(count) = updates.row_count { manifest.row_count = Some(count); }
+        if let Some(policy) = updates.embedding_refresh_policy { manifest.embedding_refresh_policy = Some(policy); }
         manifest.updated_at = chrono::Utc::now();
 
         // Persist
@@ -242,6 +248,63 @@ impl Registry {
         (ok, err)
     }
 
+    /// Mark a dataset's embeddings as stale (row-level data has been written
+    /// since the last embedding refresh). Idempotent — setting stale when
+    /// already stale is a no-op. Only marks stale if the dataset has been
+    /// embedded before — a never-embedded dataset doesn't need a stale flag
+    /// (it just needs an initial index build). Called from the ingest path.
+    pub async fn mark_embeddings_stale(&self, name: &str) -> Result<(), String> {
+        let mut datasets = self.datasets.write().await;
+        let manifest = datasets
+            .values_mut()
+            .find(|d| d.name == name)
+            .ok_or_else(|| format!("dataset not found: {name}"))?;
+
+        if manifest.last_embedded_at.is_none() {
+            return Ok(()); // never embedded -> no stale semantics yet
+        }
+        if manifest.embedding_stale_since.is_none() {
+            manifest.embedding_stale_since = Some(chrono::Utc::now());
+            manifest.updated_at = chrono::Utc::now();
+
+            let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
+            let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
+            ops::put(&self.store, &key, json.into()).await?;
+            tracing::info!("marked embeddings stale for dataset '{name}'");
+        }
+        Ok(())
+    }
+
+    /// Clear the stale marker and set `last_embedded_at = now`.
+    /// Called by the embedding refresh pipeline once it finishes.
+    pub async fn clear_embeddings_stale(&self, name: &str) -> Result<(), String> {
+        let mut datasets = self.datasets.write().await;
+        let manifest = datasets
+            .values_mut()
+            .find(|d| d.name == name)
+            .ok_or_else(|| format!("dataset not found: {name}"))?;
+
+        let now = chrono::Utc::now();
+        manifest.embedding_stale_since = None;
+        manifest.last_embedded_at = Some(now);
+        manifest.updated_at = now;
+
+        let key = format!("{MANIFEST_PREFIX}/{}.json", manifest.id);
+        let json = serde_json::to_vec_pretty(manifest).map_err(|e| e.to_string())?;
+        ops::put(&self.store, &key, json.into()).await?;
+        Ok(())
+    }
+
+    /// List datasets whose `embedding_stale_since` is set — they need a refresh.
+    pub async fn stale_datasets(&self) -> Vec<DatasetManifest> {
+        let datasets = self.datasets.read().await;
+        datasets
+            .values()
+            .filter(|d| d.embedding_stale_since.is_some())
+            .cloned()
+            .collect()
+    }
+
     /// Add objects to an existing dataset.
     pub async fn add_objects(
         &self,