Phase 17: Model profiles + scoped search — the LLM-brain keystone

Implements PRD invariant 9 ("every reader gets its own profile") and
completes the multi-model substrate vision. Local models (or agents)
bind to a named set of datasets; activation pre-loads their vector
indexes into memory; search enforces scope.

Schema (shared::types):
- ModelProfile { id, ollama_name, description, bound_datasets,
                 hnsw_config, embed_model, created_at, created_by }
- ProfileHnswConfig mirrors vectord::trial::HnswConfig to avoid a
  cross-crate dep cycle. Default (ec=80, es=30) matches the Phase 15
  trial winner.
- bound_datasets can reference raw dataset names OR AiView names
  (both register as DataFusion tables with the same name, so mixing
  raw tables and PII-redacted views composes naturally)

Catalog (catalogd::registry):
- put_profile validates id is a slug (alphanumeric + -_ only) and
  every binding resolves to an existing dataset or view
- Persistence at _catalog/profiles/{id}.json, loaded on rebuild
- get_profile / list_profiles / delete_profile

HTTP endpoints:
- POST /catalog/profiles  (create/update)
- GET  /catalog/profiles  (list)
- GET/DELETE /catalog/profiles/{id}
- POST /vectors/profile/{id}/activate  (HNSW hot-load)
- POST /vectors/profile/{id}/search    (scope-enforced)

Activation (vectord::service::activate_profile):
- For each bound dataset, find vector indexes with matching source
- Pre-load embeddings into EmbeddingCache
- Build HNSW with profile's config
- Report warmed indexes + per-binding failures + duration
- Failures on individual bindings don't abort — "substrate keeps
  working" per ADR-017

Scoped search (vectord::service::profile_scoped_search):
- Look up profile, verify index.source ∈ profile.bound_datasets
- Returns 403 with allowed bindings list if out-of-scope
- Uses HNSW if index is warm, brute-force cosine otherwise (graceful
  degradation — no "must activate first" friction)

Bug fix surfaced during testing: vectord::refresh::try_update_index_meta
was a no-op for first-time indexes, so threat_intel_v1 and
kb_team_runs_v1 (both built via refresh after Phase C shipped) didn't
show up in the index registry. Now it auto-infers the source from the
index name convention (`{source}_vN`) and registers new metadata with
reasonable defaults.

End-to-end verified:
- Created security-analyst profile bound to [threat_intel]
- POST /vectors/profile/security-analyst/activate → warmed
  threat_intel_v1 (54 vectors) in 156ms, HNSW built
- Within-scope search: method=hnsw, returned relevant IP indicators
- Out-of-scope: tried to search resumes_100k_v2 (source=candidates)
  → 403 "profile 'security-analyst' is not bound to 'candidates' —
    allowed bindings: [\"threat_intel\"]"
- staffing-recruiter profile created bound to candidates + placements;
  search without activation fell through to brute_force (graceful)

Deferred (Phase 17 followups):
- VRAM-aware activation (unload-then-load via Ollama keep_alive=0)
  — Ollama already handles this; we don't need to reinvent
- Model-identity in audit trail — Phase 13 has role-based audit;
  adding model_id is ~20 LOC when we want it
- Profile bucket pre-load (profile:user bucket mount) — Phase 17.5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

crates/vectord/src/refresh.rs

@@ -167,6 +167,11 @@ pub async fn refresh_index(
 
     if new_docs == 0 {
         tracing::info!("refresh '{}': no new docs to embed", dataset_name);
+        // Even on no-op, make sure index metadata is registered so
+        // downstream discovery (profile activation, A/B search) sees
+        // indexes built by earlier refresh calls that predated the
+        // auto-register behavior.
+        let _ = try_update_index_meta(index_registry, &req.index_name, existing.len()).await;
         registry.clear_embeddings_stale(dataset_name).await?;
         return Ok(RefreshResult {
             index_name: req.index_name.clone(),
@@ -263,8 +268,10 @@ pub async fn refresh_index(
     })
 }
 
-/// Best-effort refresh of index registry metadata. If the index exists,
-/// bump the chunk_count; if not, this is a no-op.
+/// Best-effort refresh of index registry metadata.
+/// - If the index already exists, bump the chunk_count.
+/// - If it's brand new (first-time build via refresh), REGISTER it so
+///   downstream discovery (profile activation, A/B search) can find it.
 async fn try_update_index_meta(
     index_registry: &IndexRegistry,
     index_name: &str,
@@ -272,8 +279,32 @@ async fn try_update_index_meta(
 ) -> Result<(), String> {
     if let Some(mut meta) = index_registry.get(index_name).await {
         meta.chunk_count = chunk_count;
-        index_registry.register(meta).await
-    } else {
-        Ok(())
+        return index_registry.register(meta).await;
     }
+    // First-time registration — infer reasonable defaults. Convention:
+    // index name `{source}_v1` or `{source}_vN` implies a source dataset
+    // named by stripping the `_vN` suffix. Otherwise use the index name
+    // as the source and let the caller patch later.
+    let source = match index_name.rsplit_once('_') {
+        Some((base, suffix)) if suffix.starts_with('v') && suffix[1..].chars().all(|c| c.is_ascii_digit()) => {
+            base.to_string()
+        }
+        _ => index_name.to_string(),
+    };
+    let meta = crate::index_registry::IndexMeta {
+        index_name: index_name.to_string(),
+        source,
+        model_name: "nomic-embed-text".to_string(),
+        model_version: "latest".to_string(),
+        dimensions: 768,
+        chunk_count,
+        doc_count: chunk_count,
+        chunk_size: 500,
+        overlap: 50,
+        storage_key: format!("vectors/{index_name}.parquet"),
+        created_at: chrono::Utc::now(),
+        build_time_secs: 0.0,
+        chunks_per_sec: 0.0,
+    };
+    index_registry.register(meta).await
 }

crates/vectord/src/service.rs

@@ -54,6 +54,10 @@ pub fn router(state: VectorState) -> Router {
         // Phase C: embedding refresh
         .route("/refresh/{dataset_name}", post(refresh_dataset))
         .route("/stale", get(list_stale))
+        // Phase 17: profile activation — pre-load caches + HNSW for this
+        // model's bound data. First search after activate is warm.
+        .route("/profile/{id}/activate", post(activate_profile))
+        .route("/profile/{id}/search", post(profile_scoped_search))
         .with_state(state)
 }
 
@@ -731,3 +735,187 @@ async fn list_stale(State(state): State<VectorState>) -> impl IntoResponse {
         .collect();
     Json(entries)
 }
+
+// --- Phase 17: Model profile activation + scoped search ---
+
+#[derive(Serialize)]
+struct ActivateReport {
+    profile_id: String,
+    ollama_name: String,
+    indexes_warmed: Vec<WarmedIndex>,
+    failures: Vec<String>,
+    total_vectors: usize,
+    duration_secs: f32,
+}
+
+#[derive(Serialize)]
+struct WarmedIndex {
+    index_name: String,
+    source: String,
+    vectors: usize,
+    hnsw_build_secs: f32,
+}
+
+/// Warm this profile's indexes. For every bound dataset, find the
+/// matching vector index (any index whose `source` equals the dataset
+/// or view name), load its embeddings into EmbeddingCache, build HNSW
+/// with the profile's config. Next `/profile/{id}/search` call is then
+/// <1ms cold.
+///
+/// Failures on individual indexes don't stop the activation — they get
+/// reported in the response. This matches the "substrate keeps working"
+/// philosophy from ADR-017: one bad binding shouldn't take down the
+/// whole profile.
+async fn activate_profile(
+    State(state): State<VectorState>,
+    Path(profile_id): Path<String>,
+) -> impl IntoResponse {
+    let t0 = std::time::Instant::now();
+
+    let profile = match state.catalog.get_profile(&profile_id).await {
+        Some(p) => p,
+        None => return Err((StatusCode::NOT_FOUND, format!("profile not found: {profile_id}"))),
+    };
+
+    let mut warmed = Vec::new();
+    let mut failures = Vec::new();
+    let mut total_vectors = 0usize;
+
+    let all_indexes = state.index_registry.list(None, None).await;
+
+    for binding in &profile.bound_datasets {
+        // Find every index whose source matches this binding.
+        let matched: Vec<_> = all_indexes
+            .iter()
+            .filter(|m| &m.source == binding)
+            .collect();
+        if matched.is_empty() {
+            failures.push(format!(
+                "no vector index found for binding '{}'", binding,
+            ));
+            continue;
+        }
+        for meta in matched {
+            // Pre-load embeddings into cache.
+            let embeddings = match state.embedding_cache.get_or_load(&meta.index_name).await {
+                Ok(arc) => arc,
+                Err(e) => {
+                    failures.push(format!("{}: load failed: {}", meta.index_name, e));
+                    continue;
+                }
+            };
+            total_vectors += embeddings.len();
+
+            // Build HNSW with the profile's config.
+            let cfg = trial::HnswConfig {
+                ef_construction: profile.hnsw_config.ef_construction,
+                ef_search: profile.hnsw_config.ef_search,
+                seed: profile.hnsw_config.seed,
+            };
+            let build_t = std::time::Instant::now();
+            match state
+                .hnsw_store
+                .build_index_with_config(&meta.index_name, (*embeddings).clone(), &cfg)
+                .await
+            {
+                Ok(_) => {
+                    warmed.push(WarmedIndex {
+                        index_name: meta.index_name.clone(),
+                        source: meta.source.clone(),
+                        vectors: embeddings.len(),
+                        hnsw_build_secs: build_t.elapsed().as_secs_f32(),
+                    });
+                }
+                Err(e) => {
+                    failures.push(format!("{}: HNSW build failed: {}", meta.index_name, e));
+                }
+            }
+        }
+    }
+
+    Ok(Json(ActivateReport {
+        profile_id: profile.id,
+        ollama_name: profile.ollama_name,
+        indexes_warmed: warmed,
+        failures,
+        total_vectors,
+        duration_secs: t0.elapsed().as_secs_f32(),
+    }))
+}
+
+#[derive(Deserialize)]
+struct ProfileSearchRequest {
+    index_name: String,
+    query: String,
+    top_k: Option<usize>,
+}
+
+/// Search scoped to a profile — refuses if the requested index's source
+/// isn't in the profile's bound_datasets. Reuses the existing HNSW
+/// search path when the index is warm; falls back to brute-force cosine
+/// if it's not (handled by the existing search code path).
+async fn profile_scoped_search(
+    State(state): State<VectorState>,
+    Path(profile_id): Path<String>,
+    Json(req): Json<ProfileSearchRequest>,
+) -> impl IntoResponse {
+    let profile = match state.catalog.get_profile(&profile_id).await {
+        Some(p) => p,
+        None => return Err((StatusCode::NOT_FOUND, format!("profile not found: {profile_id}"))),
+    };
+
+    // Verify the index is in scope for this profile.
+    let index_meta = match state.index_registry.get(&req.index_name).await {
+        Some(m) => m,
+        None => return Err((StatusCode::NOT_FOUND, format!("index not found: {}", req.index_name))),
+    };
+    if !profile.bound_datasets.contains(&index_meta.source) {
+        return Err((
+            StatusCode::FORBIDDEN,
+            format!(
+                "profile '{}' is not bound to '{}' — allowed bindings: {:?}",
+                profile.id, index_meta.source, profile.bound_datasets,
+            ),
+        ));
+    }
+
+    let top_k = req.top_k.unwrap_or(5);
+
+    // Embed the query.
+    let embed_resp = state
+        .ai_client
+        .embed(EmbedRequest { texts: vec![req.query.clone()], model: None })
+        .await
+        .map_err(|e| (StatusCode::BAD_GATEWAY, format!("embed: {e}")))?;
+    if embed_resp.embeddings.is_empty() {
+        return Err((StatusCode::BAD_GATEWAY, "no embedding returned".into()));
+    }
+    let query_vec: Vec<f32> = embed_resp.embeddings[0].iter().map(|&x| x as f32).collect();
+
+    // Prefer the hot HNSW path if the index is loaded, otherwise fall
+    // back to the brute-force path.
+    if state.hnsw_store.has_index(&req.index_name).await {
+        match state.hnsw_store.search(&req.index_name, &query_vec, top_k).await {
+            Ok(hits) => Ok(Json(serde_json::json!({
+                "profile": profile.id,
+                "source": index_meta.source,
+                "method": "hnsw",
+                "results": hits,
+            }))),
+            Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
+        }
+    } else {
+        let embeddings = state
+            .embedding_cache
+            .get_or_load(&req.index_name)
+            .await
+            .map_err(|e| (StatusCode::NOT_FOUND, format!("embeddings: {e}")))?;
+        let results = search::search(&query_vec, &embeddings, top_k);
+        Ok(Json(serde_json::json!({
+            "profile": profile.id,
+            "source": index_meta.source,
+            "method": "brute_force",
+            "results": results,
+        })))
+    }
+}