Phase 16: Hot-swap generations + autotune agent loop

Closes the self-iteration loop from the PRD reframe: an agent can
tune HNSW configs autonomously and the winner flows through to the
next profile activation without human intervention.

Three primitives:

1. PromotionRegistry (vectord::promotion)
   - Per-index current + history at _hnsw_promotions/{index}.json
   - promote(index, entry) atomically swaps current, pushes prior
     onto history (capped at 50)
   - rollback() pops history back onto current; clears current if
     history exhausted
   - config_or(index, default) — the read side used at build time,
     returns promoted config if set else caller's default
   - Full cache + persistence; writes are durable on return

2. Autotune (vectord::autotune)
   - run_autotune(request, ...) — synchronous agent loop
   - Default grid: 5 configs covering the practical range
     (ec=20/40/80/80/160, es=30/30/30/60/30) with seed=42 for
     reproducibility
   - Every trial goes through the existing trial-journal pipeline
     so autotune runs land alongside manual trials in the
     "trials are data" log
   - Winner: max recall first, then min p50 latency; must clear
     min_recall gate (default 0.9) or no promotion happens
   - Config bounds (ec ∈ [10,400], es ∈ [10,200]) reject absurd
     values from the request's optional custom grid
   - On winner: promote with note "autotune winner: recall=X p50=Y"

3. Wiring
   - VectorState gains promotion_registry
   - activate_profile now calls promotion_registry.config_or(...)
     so newly-promoted configs are picked up on next activation —
     the "hot-swap" is: autotune promotes -> profile activates ->
     HNSW rebuilt with new config
   - New endpoints:
       POST /vectors/hnsw/promote/{index}/{trial_id}
             ?promoted_by=...&note=...
       POST /vectors/hnsw/rollback/{index}
       GET  /vectors/hnsw/promoted/{index}
       POST /vectors/hnsw/autotune  { index_name, harness,
                                      min_recall?, grid? }

End-to-end verified on threat_intel_v1 (54 vectors):
- autogen harness 'threat_intel_smoke' (10 queries)
- POST /autotune -> 5 trials in 620ms, winner ec=20 es=30
  recall=1.00 p50=64us auto-promoted
- Manual promote of ec=80 es=30 -> history depth 1
- Rollback -> back to ec=20 es=30 autotune winner
- Second rollback -> current cleared
- Re-promote + restart -> persistence verified
- Profile activation after promotion logged:
  "building HNSW ef_construction=80 ef_search=30 seed=Some(42)"
  proving the hot-swap loop is closed.

Deferred:
- Bayesian optimization (random-grid is fine at this config-space size)
- Append-triggered autotune (Phase 17.5 — refresh OnAppend policy
  can schedule autotune after appending sufficient new rows)
- Concurrent autotune per index guard (JobTracker integration)

PRD invariants satisfied: invariant 8 (hot-swappable indexes) is now
real code — promote is atomic, rollback is always available, the
active generation is a persistent pointer not a runtime convention.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/service.rs

@@ -11,7 +11,7 @@ use std::sync::Arc;
 
 use aibridge::client::{AiClient, EmbedRequest};
 use catalogd::registry::Registry as CatalogRegistry;
-use crate::{chunker, embedding_cache, harness, hnsw, index_registry, jobs, rag, refresh, search, store, supervisor, trial};
+use crate::{autotune, chunker, embedding_cache, harness, hnsw, index_registry, jobs, promotion, rag, refresh, search, store, supervisor, trial};
 
 #[derive(Clone)]
 pub struct VectorState {
@@ -25,6 +25,8 @@ pub struct VectorState {
     /// Catalog registry — needed by the Phase C refresh path to mark/clear
     /// staleness and look up dataset manifests.
     pub catalog: CatalogRegistry,
+    /// Phase 16: promoted HNSW configs. Activation + autotune read/write here.
+    pub promotion_registry: promotion::PromotionRegistry,
 }
 
 pub fn router(state: VectorState) -> Router {
@@ -58,6 +60,11 @@ pub fn router(state: VectorState) -> Router {
         // model's bound data. First search after activate is warm.
         .route("/profile/{id}/activate", post(activate_profile))
         .route("/profile/{id}/search", post(profile_scoped_search))
+        // Phase 16: promotion + autotune
+        .route("/hnsw/promote/{index}/{trial_id}", post(promote_trial))
+        .route("/hnsw/rollback/{index}", post(rollback_promotion))
+        .route("/hnsw/promoted/{index}", get(get_promoted))
+        .route("/hnsw/autotune", post(run_autotune_endpoint))
         .with_state(state)
 }
 
@@ -806,12 +813,19 @@ async fn activate_profile(
             };
             total_vectors += embeddings.len();
 
-            // Build HNSW with the profile's config.
-            let cfg = trial::HnswConfig {
+            // Build HNSW with the index's PROMOTED config if one exists
+            // (Phase 16 hot-swap), otherwise fall back to the profile's
+            // declared defaults. This means autotune's winner is picked
+            // up on the next activation automatically.
+            let profile_default = trial::HnswConfig {
                 ef_construction: profile.hnsw_config.ef_construction,
                 ef_search: profile.hnsw_config.ef_search,
                 seed: profile.hnsw_config.seed,
             };
+            let cfg = state
+                .promotion_registry
+                .config_or(&meta.index_name, profile_default)
+                .await;
             let build_t = std::time::Instant::now();
             match state
                 .hnsw_store
@@ -919,3 +933,83 @@ async fn profile_scoped_search(
         })))
     }
 }
+
+// --- Phase 16: Promotion + autotune ---
+
+#[derive(Deserialize)]
+struct PromoteQuery {
+    #[serde(default)]
+    promoted_by: String,
+    #[serde(default)]
+    note: Option<String>,
+}
+
+async fn promote_trial(
+    State(state): State<VectorState>,
+    Path((index_name, trial_id)): Path<(String, String)>,
+    Query(q): Query<PromoteQuery>,
+) -> impl IntoResponse {
+    // Pull the trial from the journal to get its config.
+    let trials = state
+        .trial_journal
+        .list(&index_name)
+        .await
+        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
+    let trial = trials
+        .iter()
+        .find(|t| t.id == trial_id)
+        .ok_or_else(|| (StatusCode::NOT_FOUND, format!("trial not found: {trial_id}")))?;
+
+    let entry = promotion::PromotionEntry {
+        config: trial.config.clone(),
+        trial_id: trial.id.clone(),
+        promoted_at: chrono::Utc::now(),
+        promoted_by: q.promoted_by,
+        note: q.note,
+    };
+    match state.promotion_registry.promote(&index_name, entry).await {
+        Ok(file) => Ok(Json(file)),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}
+
+async fn rollback_promotion(
+    State(state): State<VectorState>,
+    Path(index_name): Path<String>,
+) -> impl IntoResponse {
+    match state.promotion_registry.rollback(&index_name).await {
+        Ok(file) => Ok(Json(file)),
+        Err(e) => Err((StatusCode::NOT_FOUND, e)),
+    }
+}
+
+async fn get_promoted(
+    State(state): State<VectorState>,
+    Path(index_name): Path<String>,
+) -> impl IntoResponse {
+    match state.promotion_registry.load(&index_name).await {
+        Ok(file) => Ok(Json(file)),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}
+
+async fn run_autotune_endpoint(
+    State(state): State<VectorState>,
+    Json(req): Json<autotune::AutotuneRequest>,
+) -> impl IntoResponse {
+    match autotune::run_autotune(
+        req,
+        &state.store,
+        &state.catalog,
+        &state.ai_client,
+        &state.embedding_cache,
+        &state.hnsw_store,
+        &state.index_registry,
+        &state.trial_journal,
+        &state.promotion_registry,
+        &state.job_tracker,
+    ).await {
+        Ok(result) => Ok(Json(result)),
+        Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+    }
+}