profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Phase 19 wiring + Path 1/2 work + chain integrity fixes

Browse Source 
Backend:
- crates/vectord/src/playbook_memory.rs (new): Phase 19 in-memory boost
  store with seed/rebuild/snapshot, plus temporal decay (e^-age/30 per
  playbook), persist_to_sql endpoint backing successful_playbooks_live,
  and discover_patterns endpoint for meta-index pattern aggregation
  (recurring certs/skills/archetype/reliability across similar past fills).
- DEFAULT_TOP_K_PLAYBOOKS bumped 5 → 25; old default silently missed
  most boosts when memory had > 25 entries.
- service.rs: new routes /vectors/playbook_memory/{seed,rebuild,stats,
  persist_sql,patterns}.

Bun staffing co-pilot (mcp-server/):
- /search, /match, /verify, /proof, /simulation/run, MCP tools all
  forward use_playbook_memory:true and playbook_memory_k:25 to the
  hybrid endpoint. Boost was previously dark across the entire app.
- /log no longer POSTs to /ingest/file — that endpoint REPLACES the
  dataset's object list, so single-row CSV writes were wiping all prior
  rows in successful_playbooks (sp_rows went 33→1 in one /log call).
  /log now seeds playbook_memory with canonical short text and calls
  /persist_sql to keep successful_playbooks_live in sync.
- /simulation/run cumulative end-of-week CSV write removed for the same
  reason. Per-day per-contract /seed (added in this session) is the
  accumulating feedback path now.
- search.html addWorkerInsight renders a green "Endorsed · N playbooks"
  chip with playbook citations when boost > 0.

Internal Dioxus UI (crates/ui/):
- Dashboard phase list rewritten through Phase 19 (was stuck at "Phase
  16: File Watcher" / "Phase 17: DB Connector" — both wrong).
- Removed fabricated "27ms" stat label.
- Ask tab examples + SQL default replaced with real staffing prompts
  against candidates/clients/job_orders (was referencing nonexistent
  employees/products/events).
- New Playbook tab exposes /vectors/playbook_memory/{stats,rebuild} and
  side-by-side hybrid search (boost OFF vs ON) with citations.

Tests (tests/multi-agent/):
- run_e2e_rated.ts: parallel two-agent (mistral + qwen2.5) build phase
  + verifier rating (geo, auth, persist, boost, speed → /10).
- network_proving.ts: continuous build → verify → repeat with
  staffing-recruiter profile hot-swap; geo-discrimination check.
- chain_of_custody.ts: single recruiter operation traced through every
  layer (Bun /search, direct /vectors/hybrid parity, /log, SQL,
  playbook_memory growth, profile activation, post-op boost lift).
This commit is contained in:
root
2026-04-20 06:21:13 -05:00
parent 8e3cac5812
commit 25b7e6c3a7
13 changed files with 4566 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/vectord/src/service.rs
+500 -19
View File
@@ -12,7 +12,7 @@ use std::sync::Arc;
use aibridge::client::{AiClient, EmbedRequest, GenerateRequest};
use catalogd::registry::Registry as CatalogRegistry;
use storaged::registry::BucketRegistry;
use crate::{agent, autotune, chunker, embedding_cache, harness, hnsw, index_registry, jobs, lance_backend, promotion, rag, refresh, search, store, supervisor, trial};
use crate::{agent, autotune, chunker, embedding_cache, harness, hnsw, index_registry, jobs, lance_backend, playbook_memory, promotion, rag, refresh, search, store, supervisor, trial};
#[derive(Clone)]
pub struct VectorState {
@@ -23,6 +23,9 @@ pub struct VectorState {
pub hnsw_store: hnsw::HnswStore,
pub embedding_cache: embedding_cache::EmbeddingCache,
pub trial_journal: trial::TrialJournal,
/// Federation-aware harness store — resolves eval artifacts to each
/// index's recorded bucket, falling back to primary for legacy evals.
pub harness_store: harness::HarnessStore,
/// Catalog registry — needed by the Phase C refresh path to mark/clear
/// staleness and look up dataset manifests.
pub catalog: CatalogRegistry,
@@ -46,6 +49,10 @@ pub struct VectorState {
/// ADR-019 hybrid: handles to Lance datasets keyed by index name.
/// Lazy-created on first /vectors/lance/* call.
pub lance: lance_backend::LanceRegistry,
/// Phase 19 — meta-index feedback. Embeds past successful_playbooks
/// and, when `use_playbook_memory` is set on /vectors/hybrid, boosts
/// workers that were actually filled in semantically-similar past ops.
pub playbook_memory: playbook_memory::PlaybookMemory,
}
/// What the active-profile singleton records. Narrow — we don't need the
@@ -63,6 +70,7 @@ pub fn router(state: VectorState) -> Router {
.route("/index", post(create_index))
.route("/indexes", get(list_indexes))
.route("/indexes/{name}", get(get_index_meta))
.route("/indexes/{name}/bucket", axum::routing::patch(migrate_index_bucket))
.route("/jobs", get(list_jobs))
.route("/jobs/{id}", get(get_job))
.route("/search", post(search_index))
@@ -110,6 +118,12 @@ pub fn router(state: VectorState) -> Router {
.route("/lance/stats/{index_name}", get(lance_stats))
.route("/lance/scalar-index/{index_name}/{column}", post(lance_build_scalar_index))
.route("/lance/recall/{index_name}", post(lance_recall_harness))
// Phase 19: playbook memory — the meta-index feedback loop
.route("/playbook_memory/rebuild", post(rebuild_playbook_memory))
.route("/playbook_memory/stats", get(playbook_memory_stats))
.route("/playbook_memory/seed", post(seed_playbook_memory))
.route("/playbook_memory/persist_sql", post(persist_playbook_memory_sql))
.route("/playbook_memory/patterns", post(discover_playbook_patterns))
.with_state(state)
}
@@ -259,6 +273,174 @@ async fn get_index_meta(
}
}
#[derive(Deserialize)]
struct MigrateBucketRequest {
dest_bucket: String,
/// If true, delete artifacts from the source bucket after the pointer
/// flip. Default false — keeping source copies means a failed migration
/// is recoverable by editing IndexMeta.bucket back, and a successful
/// migration leaves inspectable forensics until an operator sweeps.
#[serde(default)]
delete_source: bool,
}
#[derive(Serialize)]
struct MigrateBucketReport {
index_name: String,
source_bucket: String,
dest_bucket: String,
/// Artifact keys that were copied (or attempted). Order follows copy order.
copied: Vec<String>,
/// Artifact prefixes that had nothing to copy (optional files missing,
/// trial journal empty, etc).
skipped: Vec<String>,
/// Subset of `copied` that was subsequently deleted from the source.
deleted_source: Vec<String>,
duration_secs: f32,
}
/// Move an index's artifacts from its current bucket to `dest_bucket`.
/// Parquet-backed indexes only — Lance migration needs URI rewriting that
/// isn't in scope for this endpoint. Copies the vector data, trial journal,
/// promotion file, and auto-generated harness; updates `IndexMeta.bucket`
/// last so a mid-flight failure leaves the index still usable at its
/// original location. Evicts the `EmbeddingCache` entry so the next load
/// re-reads from the new bucket.
async fn migrate_index_bucket(
State(state): State<VectorState>,
Path(name): Path<String>,
Json(req): Json<MigrateBucketRequest>,
) -> Result<Json<MigrateBucketReport>, (StatusCode, String)> {
let t0 = std::time::Instant::now();
let mut meta = state
.index_registry
.get(&name)
.await
.ok_or_else(|| (StatusCode::NOT_FOUND, format!("index '{name}' not found")))?;
if meta.vector_backend == shared::types::VectorBackend::Lance {
return Err((
StatusCode::BAD_REQUEST,
"Lance-backed indexes cannot be migrated via this endpoint — \
Lance URIs are bucket-specific; a separate migrate_lance tool \
is needed".into(),
));
}
if !state.bucket_registry.contains(&req.dest_bucket) {
return Err((
StatusCode::BAD_REQUEST,
format!("dest bucket '{}' not registered", req.dest_bucket),
));
}
let source_bucket = meta.bucket.clone();
if source_bucket == req.dest_bucket {
return Err((
StatusCode::BAD_REQUEST,
format!("source and dest are both '{source_bucket}' — nothing to migrate"),
));
}
let src = state
.bucket_registry
.get(&source_bucket)
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
let dst = state
.bucket_registry
.get(&req.dest_bucket)
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
let mut copied: Vec<String> = Vec::new();
let mut skipped: Vec<String> = Vec::new();
// 1. Vector data (single parquet file for this backend).
copy_key(&src, &dst, &meta.storage_key)
.await
.map_err(|e| {
(StatusCode::INTERNAL_SERVER_ERROR,
format!("copy {}: {e}", meta.storage_key))
})?;
copied.push(meta.storage_key.clone());
// 2. Trial journal batches — per-index directory of JSONL files.
let trial_prefix = format!("_hnsw_trials/{name}/");
let trial_keys = storaged::ops::list(&src, Some(&trial_prefix))
.await
.unwrap_or_default();
if trial_keys.is_empty() {
skipped.push(trial_prefix);
}
for k in &trial_keys {
copy_key(&src, &dst, k)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("copy {k}: {e}")))?;
copied.push(k.clone());
}
// 3. Promotion file (optional — absent for never-promoted indexes).
let promo_key = format!("_hnsw_promotions/{name}.json");
match copy_key(&src, &dst, &promo_key).await {
Ok(()) => copied.push(promo_key),
Err(_) => skipped.push(promo_key),
}
// 4. Auto-generated harness (optional — absent if agent never ran).
let harness_key = format!("_hnsw_evals/{name}_auto.json");
match copy_key(&src, &dst, &harness_key).await {
Ok(()) => copied.push(harness_key),
Err(_) => skipped.push(harness_key),
}
// 5. Pointer flip — IndexMeta.bucket now points at destination. This
// is the commit point; earlier failures leave copies in dest but the
// index still usable at source.
meta.bucket = req.dest_bucket.clone();
state
.index_registry
.register(meta)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("update meta: {e}")))?;
// 6. Cache eviction — next load reads the new bucket's parquet.
state.embedding_cache.evict(&name).await;
// 7. Optional source cleanup.
let mut deleted_source: Vec<String> = Vec::new();
if req.delete_source {
for k in &copied {
if storaged::ops::delete(&src, k).await.is_ok() {
deleted_source.push(k.clone());
}
}
}
Ok(Json(MigrateBucketReport {
index_name: name,
source_bucket,
dest_bucket: req.dest_bucket,
copied,
skipped,
deleted_source,
duration_secs: t0.elapsed().as_secs_f32(),
}))
}
/// Stream a single object from one bucket to another. Uses the existing
/// `storaged::ops` get + put primitives — no native copy in object_store
/// across heterogeneous backends (local ↔ S3), so an in-memory hop is
/// unavoidable. Bounded by individual object size, which for our parquet
/// + jsonl artifacts tops out around a few hundred MB.
async fn copy_key(
src: &Arc<dyn ObjectStore>,
dst: &Arc<dyn ObjectStore>,
key: &str,
) -> Result<(), String> {
let data = storaged::ops::get(src, key).await?;
storaged::ops::put(dst, key, data).await
}
// --- unused legacy function below, kept for reference ---
#[allow(dead_code)]
@@ -420,6 +602,15 @@ struct HybridRequest {
/// If false, just return the ranked matches (faster, no Ollama gen).
#[serde(default = "default_true")]
generate: bool,
/// Phase 19: consult `playbook_memory` and boost workers that past
/// similar playbooks successfully filled. Off by default so current
/// callers keep deterministic ranking; opt-in unlocks the feedback.
#[serde(default)]
use_playbook_memory: bool,
/// Number of past playbooks to consider when `use_playbook_memory`
/// is on. Ignored otherwise. Defaults to 5.
#[serde(default)]
playbook_memory_k: Option<usize>,
}
fn default_true() -> bool { true }
@@ -442,8 +633,18 @@ struct HybridSource {
chunk_text: String,
score: f32,
sql_verified: bool,
/// Phase 19: how much the playbook_memory boost lifted this hit's
/// score. 0.0 when `use_playbook_memory=false` or no past playbook
/// endorsed this worker.
#[serde(default, skip_serializing_if = "is_zero")]
playbook_boost: f32,
/// playbook_ids whose endorsement contributed to `playbook_boost`.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
playbook_citations: Vec<String>,
}
fn is_zero(x: &f32) -> bool { x.abs() < 1e-6 }
async fn hybrid_search(
State(state): State<VectorState>,
Json(req): Json<HybridRequest>,
@@ -556,6 +757,11 @@ async fn hybrid_search(
.and_then(|m| m.id_prefix.clone());
let sql_count = valid_ids.as_ref().map(|s| s.len()).unwrap_or(0);
// Phase 19: when playbook_memory is consulted, pull a wider candidate
// pool so endorsed workers outside the vanilla top-K can still be
// boosted into visibility. 5× is a conservative multiplier — plenty
// for a +0.25 boost to flip rankings without dragging the cost up.
let fetch_k = if req.use_playbook_memory { req.top_k * 5 } else { req.top_k };
let filtered: Vec<search::SearchResult> = if let Some(ref ids) = valid_ids {
all_results.into_iter()
.filter(|r| {
@@ -572,20 +778,54 @@ async fn hybrid_search(
};
ids.contains(raw_id)
})
.take(req.top_k)
.take(fetch_k)
.collect()
} else {
all_results.into_iter().take(req.top_k).collect()
all_results.into_iter().take(fetch_k).collect()
};
// Step 4: Build sources with SQL-verified flag.
let sources: Vec<HybridSource> = filtered.iter().map(|r| HybridSource {
let mut sources: Vec<HybridSource> = filtered.iter().map(|r| HybridSource {
doc_id: r.doc_id.clone(),
chunk_text: r.chunk_text.clone(),
score: r.score,
sql_verified: valid_ids.is_some(),
playbook_boost: 0.0,
playbook_citations: Vec::new(),
}).collect();
// Step 4b (Phase 19): if use_playbook_memory, look up semantically
// similar past playbooks and boost workers they endorsed. Name-match
// is on the tuple (city, state, name) extracted from chunk_text —
// hybrid_search's SQL filter already narrowed to one city+state, so
// this just needs to check the name against each playbook's endorsed
// set. Additive boost on the existing vector score, then re-sort.
if req.use_playbook_memory {
let boost_k = req.playbook_memory_k.unwrap_or(playbook_memory::DEFAULT_TOP_K_PLAYBOOKS);
// We embedded the question as `qv` above — reuse it for the
// playbook similarity lookup so we don't double-pay Ollama.
let boosts = state.playbook_memory.compute_boost_for(&qv, boost_k, 0.5).await;
for src in sources.iter_mut() {
// Parse "{Name} — {Role} in {City}, {State}. …" chunk. Being
// defensive: chunks from other datasets may not follow this
// exact shape, so absent fields just skip the boost.
if let Some((name, city, state)) = parse_worker_chunk(&src.chunk_text) {
let key = (city, state, name);
if let Some(entry) = boosts.get(&key) {
src.score += entry.boost;
src.playbook_boost = entry.boost;
src.playbook_citations = entry.citations.clone();
}
}
}
// Re-rank: boosted scores can flip ordering.
sources.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(std::cmp::Ordering::Equal));
// Finally trim to the caller's requested top_k — we pulled fetch_k
// (5× wider) above specifically so the boost could reach workers
// that would otherwise have been trimmed pre-boost.
sources.truncate(req.top_k);
}
// Step 5: Generate answer if requested.
let answer = if req.generate && !sources.is_empty() {
let context: String = sources.iter().enumerate().map(|(i, s)| {
@@ -734,7 +974,7 @@ async fn run_trial(
State(state): State<VectorState>,
Json(req): Json<TrialRequest>,
) -> Result<Json<trial::Trial>, (StatusCode, String)> {
let mut harness_set = harness::EvalSet::load(&state.store, &req.harness)
let mut harness_set = state.harness_store.load_for_index(&req.index_name, &req.harness)
.await
.map_err(|e| (StatusCode::NOT_FOUND, format!("harness not found: {e}")))?;
@@ -764,8 +1004,8 @@ async fn run_trial(
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("ground truth: {e}")))?;
tracing::info!("trial: ground truth built in {:.1}s", t0.elapsed().as_secs_f32());
harness_set
.save(&state.store)
state.harness_store
.save(&harness_set)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("save harness: {e}")))?;
}
@@ -890,17 +1130,14 @@ async fn best_trial(
// --- Harness management ---
async fn list_evals(State(state): State<VectorState>) -> impl IntoResponse {
match harness::EvalSet::list(&state.store).await {
Ok(names) => Ok(Json(names)),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
Json(state.harness_store.list_all().await)
}
async fn get_eval(
State(state): State<VectorState>,
Path(name): Path<String>,
) -> impl IntoResponse {
match harness::EvalSet::load(&state.store, &name).await {
match state.harness_store.get_any(&name).await {
Ok(e) => Ok(Json(e)),
Err(err) => Err((StatusCode::NOT_FOUND, err)),
}
@@ -916,7 +1153,7 @@ async fn put_eval(
.queries
.iter()
.all(|q| q.ground_truth.is_some());
match harness_set.save(&state.store).await {
match state.harness_store.save(&harness_set).await {
Ok(()) => Ok(Json(harness_set)),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
@@ -957,8 +1194,8 @@ async fn autogen_eval(
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("ground truth: {e}")))?;
harness_set
.save(&state.store)
state.harness_store
.save(&harness_set)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("save: {e}")))?;
@@ -1407,7 +1644,12 @@ async fn profile_scoped_search(
let lance_store = state.lance.store_for(&req.index_name).await
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
let t0 = std::time::Instant::now();
match lance_store.search(&query_vec, top_k).await {
match lance_store.search(
&query_vec,
top_k,
Some(LANCE_DEFAULT_NPROBES),
Some(LANCE_DEFAULT_REFINE_FACTOR),
).await {
Ok(hits) => Ok(Json(serde_json::json!({
"profile": profile.id,
"source": index_meta.source,
@@ -1516,6 +1758,7 @@ async fn run_autotune_endpoint(
&state.index_registry,
&state.trial_journal,
&state.promotion_registry,
&state.harness_store,
&state.job_tracker,
).await {
Ok(result) => Ok(Json(result)),
@@ -1636,8 +1879,25 @@ struct LanceSearchRequest {
query: String,
#[serde(default = "default_top_k")]
top_k: usize,
/// IVF partitions to probe. `None` uses Lance's built-in default of
/// 1, which caps recall well below the index's real capability.
/// Recommended: 5–10% of num_partitions (≈20 for a 316-partition
/// index). Omitting it here picks the server-side default.
#[serde(default)]
nprobes: Option<usize>,
/// Refine factor — re-rank `top_k * factor` PQ-approximate candidates
/// with exact distances before returning `top_k`. Recovers recall
/// lost to product quantization.
#[serde(default)]
refine_factor: Option<u32>,
}
/// Server-side defaults when the caller doesn't pin nprobes / refine
/// themselves. Tuned for the ~100K × 768d reference workload; see
/// docs/ADR-019-vector-storage.md for the recall / latency trade-off.
const LANCE_DEFAULT_NPROBES: usize = 20;
const LANCE_DEFAULT_REFINE_FACTOR: u32 = 5;
fn default_top_k() -> usize { 5 }
/// Vector search against a Lance dataset. Embeds the query text via the
@@ -1660,7 +1920,9 @@ async fn lance_search(
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
let t0 = std::time::Instant::now();
let hits = lance_store.search(&qv, req.top_k).await
let nprobes = req.nprobes.or(Some(LANCE_DEFAULT_NPROBES));
let refine = req.refine_factor.or(Some(LANCE_DEFAULT_REFINE_FACTOR));
let hits = lance_store.search(&qv, req.top_k, nprobes, refine).await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e))?;
Ok(Json(serde_json::json!({
@@ -1761,6 +2023,12 @@ struct LanceRecallRequest {
harness: String,
#[serde(default = "default_top_k")]
top_k: usize,
/// Override server defaults so operators can sweep nprobes /
/// refine_factor to chart the recall-vs-latency curve.
#[serde(default)]
nprobes: Option<usize>,
#[serde(default)]
refine_factor: Option<u32>,
}
#[derive(serde::Serialize)]
@@ -1784,6 +2052,214 @@ struct LanceRecallQuery {
hits_returned: usize,
}
// --- Phase 19: playbook memory endpoints ---
/// Extract (name, city, state) from a chunk formatted like
/// "{Name} — {Role} in {City}, {State}. Skills: …".
/// Returns None if the chunk doesn't match the shape; callers simply
/// skip the boost for that hit.
fn parse_worker_chunk(chunk: &str) -> Option<(String, String, String)> {
// "Name — Role in City, ST. …" → split on "—" then " in " then ","
let (name_part, rest) = chunk.split_once('—')?;
let rest = rest.trim();
let (_role, loc_part) = rest.split_once(" in ")?;
let loc_part = loc_part.trim();
let (city, state_plus) = loc_part.split_once(',')?;
let state: String = state_plus.trim()
.chars()
.take_while(|c| c.is_ascii_alphabetic())
.collect();
let name = name_part.trim().to_string();
let city = city.trim().to_string();
if name.is_empty() || city.is_empty() || state.is_empty() {
return None;
}
Some((name, city, state))
}
#[derive(Deserialize)]
struct SeedPlaybookRequest {
/// One playbook with {operation, approach, context, endorsed_names}.
/// City + state are parsed from the operation text.
operation: String,
#[serde(default)]
approach: String,
#[serde(default)]
context: String,
endorsed_names: Vec<String>,
/// Append to the existing memory rather than replacing. Default true —
/// seeding is a bootstrap/demo tool, not a rebuild substitute.
#[serde(default = "default_true")]
append: bool,
}
/// Bootstrap / test-only: inject a playbook entry directly into
/// `playbook_memory` without going through `successful_playbooks`. Useful
/// when the source dataset has stale or phantom entries (as the initial
/// staffing seed did — names that don't correspond to real workers), and
/// you want to demonstrate the feedback loop with a known-good fixture.
///
/// Production path is always `/rebuild` — this endpoint is for operators
/// who need to prime the memory before real playbooks accumulate.
async fn seed_playbook_memory(
State(state): State<VectorState>,
Json(req): Json<SeedPlaybookRequest>,
) -> impl IntoResponse {
// Embed the entry through the same text shape `rebuild` uses so
// similarity math is comparable across seed + real entries.
let tmp_entry = playbook_memory::PlaybookEntry {
playbook_id: String::new(),
operation: req.operation.clone(),
approach: req.approach.clone(),
context: req.context.clone(),
timestamp: chrono::Utc::now().to_rfc3339(),
endorsed_names: req.endorsed_names.clone(),
city: None, state: None, embedding: None,
};
let text = format!(
"{} | {} | {} | fills: {}",
tmp_entry.operation, tmp_entry.approach, tmp_entry.context,
tmp_entry.endorsed_names.join(", "),
);
let resp = match state.ai_client.embed(EmbedRequest { texts: vec![text], model: None }).await {
Ok(r) => r,
Err(e) => return Err((StatusCode::BAD_GATEWAY, format!("embed seed: {e}"))),
};
if resp.embeddings.is_empty() {
return Err((StatusCode::BAD_GATEWAY, "embed returned nothing".into()));
}
let emb: Vec<f32> = resp.embeddings[0].iter().map(|&x| x as f32).collect();
// Parse city/state from the operation ("fill: Role xN in City, ST").
// Parser lives in playbook_memory::rebuild — expose via a tiny helper
// or inline the same logic here; duplicated briefly since this seed
// path is stable but infrequently called.
let (city, state_) = {
let after_in = req.operation.split(" in ").nth(1).unwrap_or("");
let mut parts = after_in.splitn(2, ',');
let city = parts.next().map(|s| s.trim().to_string()).filter(|s| !s.is_empty());
let state = parts.next().map(|s| s.trim().chars().take_while(|c| c.is_ascii_alphabetic()).collect::<String>()).filter(|s| !s.is_empty());
(city, state)
};
if city.is_none() || state_.is_none() {
return Err((StatusCode::BAD_REQUEST,
"operation must match 'fill: Role xN in City, ST' shape".into()));
}
// Stable id: hash of timestamp + operation. Callers get the id back
// so they can reference it in citations.
let ts = chrono::Utc::now().to_rfc3339();
use sha2::{Digest, Sha256};
let mut h = Sha256::new();
h.update(ts.as_bytes());
h.update(b"|");
h.update(req.operation.as_bytes());
let bytes = h.finalize();
let pid = format!("pb-seed-{}", bytes.iter().take(8).map(|b| format!("{b:02x}")).collect::<String>());
let new_entry = playbook_memory::PlaybookEntry {
playbook_id: pid.clone(),
operation: req.operation,
approach: req.approach,
context: req.context,
timestamp: ts,
endorsed_names: req.endorsed_names,
city, state: state_,
embedding: Some(emb),
};
let mut current = state.playbook_memory.snapshot().await;
if req.append {
current.push(new_entry);
} else {
current = vec![new_entry];
}
if let Err(e) = state.playbook_memory.set_entries(current).await {
return Err((StatusCode::INTERNAL_SERVER_ERROR, format!("persist: {e}")));
}
Ok(Json(serde_json::json!({ "playbook_id": pid, "entries_after": state.playbook_memory.entry_count().await })))
}
async fn rebuild_playbook_memory(
State(state): State<VectorState>,
) -> impl IntoResponse {
match playbook_memory::rebuild(
&state.playbook_memory,
&state.ai_client,
&state.catalog,
&state.bucket_registry,
).await {
Ok(report) => Ok(Json(report)),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
}
// Path 2 foundation — dump in-memory playbook_memory state to a fresh
// `successful_playbooks_live` dataset. Cheap to call (writes one parquet,
// updates one manifest), so /log can call it after every seed to keep the
// SQL-queryable surface honest without the destructive REPLACE bug that
// /ingest/file has.
async fn persist_playbook_memory_sql(
State(state): State<VectorState>,
) -> impl IntoResponse {
match playbook_memory::persist_to_sql(&state.playbook_memory, &state.catalog).await {
Ok(report) => Ok(Json(report)),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
}
#[derive(Deserialize)]
struct PatternsRequest {
query: String,
#[serde(default = "default_pattern_k")]
top_k_playbooks: usize,
/// Minimum frequency (0.0-1.0) for a trait to make the report.
/// Default 0.4 — at least 40% of examined workers must share it.
#[serde(default = "default_pattern_min_freq")]
min_trait_frequency: f32,
}
fn default_pattern_k() -> usize { 10 }
fn default_pattern_min_freq() -> f32 { 0.4 }
// Path 2 — meta-index discovery surface. "What did past similar fills
// have in common that I didn't ask about?" — surfaces signals like
// recurring certifications, skill clusters, archetype tendencies.
async fn discover_playbook_patterns(
State(state): State<VectorState>,
Json(req): Json<PatternsRequest>,
) -> impl IntoResponse {
match playbook_memory::discover_patterns(
&state.playbook_memory,
&state.ai_client,
&state.catalog,
&state.bucket_registry,
&req.query,
req.top_k_playbooks,
req.min_trait_frequency,
).await {
Ok(report) => Ok(Json(report)),
Err(e) => Err((StatusCode::INTERNAL_SERVER_ERROR, e)),
}
}
async fn playbook_memory_stats(
State(state): State<VectorState>,
) -> impl IntoResponse {
let entries = state.playbook_memory.snapshot().await;
Json(serde_json::json!({
"entries": entries.len(),
"total_names_endorsed": entries.iter().map(|e| e.endorsed_names.len()).sum::<usize>(),
"entries_with_embeddings": entries.iter().filter(|e| e.embedding.is_some()).count(),
"sample": entries.iter().take(3).map(|e| serde_json::json!({
"id": e.playbook_id,
"operation": e.operation,
"city": e.city,
"state": e.state,
"endorsed": e.endorsed_names,
})).collect::<Vec<_>>(),
}))
}
async fn lance_recall_harness(
State(state): State<VectorState>,
Path(index_name): Path<String>,
@@ -1791,7 +2267,7 @@ async fn lance_recall_harness(
) -> impl IntoResponse {
let t0 = std::time::Instant::now();
let harness_set = harness::EvalSet::load(&state.store, &req.harness).await
let harness_set = state.harness_store.load_for_index(&index_name, &req.harness).await
.map_err(|e| (StatusCode::NOT_FOUND, format!("harness: {e}")))?;
if !harness_set.ground_truth_built {
return Err((StatusCode::BAD_REQUEST,
@@ -1817,7 +2293,12 @@ async fn lance_recall_harness(
};
let qt0 = std::time::Instant::now();
let hits = lance_store.search(qv, k).await
let hits = lance_store.search(
qv,
k,
Some(req.nprobes.unwrap_or(LANCE_DEFAULT_NPROBES)),
Some(req.refine_factor.unwrap_or(LANCE_DEFAULT_REFINE_FACTOR)),
).await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("search: {e}")))?;
let lat_us = qt0.elapsed().as_micros() as f32;