Item 3 — geo-filtered playbook boost; diagnostic logging

ROOT CAUSE (found via instrumentation, not hunch):
After a 20-scenario corpus batch, only 6/40 successful (role, city)
combos ever triggered playbook_memory citations on subsequent runs.
Added `playbook_boost:` tracing::info! line in vectord::service to log
boost map size vs candidate pool vs match count. One query revealed:

  boosts=170 sources=50 parsed=50 matched=0

170 endorsed workers came back from compute_boost_for — but zero were
in the 50-candidate Toledo pool. The boost map was pulling globally-
ranked semantic neighbors (top-100 playbooks across ALL cities),
dominated by Kansas City / Chicago / Detroit forklift playbooks the
Toledo SQL filter would never admit. The mechanism was correct at the
per-playbook level; the problem was pool intersection.

FIX (surgical, not cap-tuning):
- playbook_memory::compute_boost_for_filtered(): accepts optional
  (city, state) filter. When set, skips playbooks from other geos
  BEFORE cosine-ranking, so top-k is within the target city.
- Backwards-compatible: compute_boost_for() calls the filtered variant
  with None — existing callers unchanged.
- service::hybrid_search(): extracts target (city, state) from the
  executor's SQL filter via a small parser (extract_target_geo),
  passes to compute_boost_for_filtered.

VERIFIED:
  Before fix: boosts=170 sources=50 parsed=50 matched=0   (0% hit)
  After fix:  boosts=36  sources=50 parsed=50 matched=11  (22% hit)
Top-k=10 now has 7/10 boosted workers with 2-3 citations each.
Boost values 0.075-0.113 on cosine scores 0.67-0.74 — meaningful
reorder without saturation.

scripts/kb_measure.py:
Aggregator that reads data/_kb/*.jsonl and playbooks/*/results.json,
reports fill rate, citation density, recommender confidence trend,
and zero-citation-ok combos (item 3 target signal). Used to measure
before/after on bigger batches.

Diagnostic logging stays — the class of "boosts computed but not
matched" bug can recur if the SQL filter format ever drifts, and
without the counter it's invisible. Every hybrid_search with
use_playbook_memory=true now logs its boost stats.
This commit is contained in:
root
2026-04-20 21:35:04 -05:00
parent 330cb90f99
commit a663698571
3 changed files with 283 additions and 1 deletions
+32
View File
@@ -213,6 +213,28 @@ impl PlaybookMemory {
query_embedding: &[f32],
top_k_playbooks: usize,
base_weight: f32,
) -> HashMap<(String, String, String), BoostEntry> {
self.compute_boost_for_filtered(query_embedding, top_k_playbooks, base_weight, None).await
}
/// Same as `compute_boost_for` but only considers playbooks whose
/// (city, state) matches the caller's target. This is the honest
/// fix for the "boosts=170 matched=0" pathology: globally-ranked
/// semantic neighbors include playbooks from every city the query
/// could never reach via its SQL filter. When the caller knows the
/// target geo, restricting here collapses noise and raises the
/// endorsed-worker hit rate. Pass None for the original behavior.
///
/// 2026-04-21 — added after a corpus-density batch of 25 runs
/// showed only 6/40 successful (role, city) combos ever triggered
/// a citation on subsequent runs. Diagnostic logging proved the
/// boost map had 170 keys but the 50-candidate pool matched 0.
pub async fn compute_boost_for_filtered(
&self,
query_embedding: &[f32],
top_k_playbooks: usize,
base_weight: f32,
target_geo: Option<(&str, &str)>,
) -> HashMap<(String, String, String), BoostEntry> {
let state = self.state.read().await;
let entries = state.entries.clone();
@@ -225,8 +247,18 @@ impl PlaybookMemory {
drop(state);
// Brute-force cosine. Empty / missing embeddings just skip.
// When target_geo is set, pre-filter to matching playbooks BEFORE
// cosine sort — that way top-k is within the city, not across
// all cities.
let mut scored: Vec<(f32, &PlaybookEntry)> = entries
.iter()
.filter(|e| match (target_geo, &e.city, &e.state) {
(None, _, _) => true,
(Some((tc, ts)), Some(ec), Some(es)) => {
ec.eq_ignore_ascii_case(tc) && es.eq_ignore_ascii_case(ts)
}
_ => false,
})
.filter_map(|e| e.embedding.as_ref().map(|v| (cosine(query_embedding, v), e)))
.collect();
scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal));