matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery

The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).

Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.

Result on playbook_lift_003 (Shape B + paraphrase pass):

  Verbatim discovery        6
  Verbatim lift             2 / 6
  **Paraphrase top-1**      **6 / 6**
  Paraphrase any-rank in K  6 / 6
  Mean Δ top-1 distance     -0.1637 (warm closer than cold)

Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.

Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.

Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op

Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 07:06:13 -05:00
co-authored by Claude Opus 4.7
parent e9822f025d
commit 154a72ea5e
5 changed files with 352 additions and 10 deletions
+72
View File
@@ -151,6 +151,78 @@ type PlaybookHit struct {
Entry PlaybookEntry `json:"entry"`
}
// InjectPlaybookMisses appends synthetic Results for playbook hits
// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
// This is "Shape B" from the doc comment at the top of this file:
// the v0 boost-only stance (ApplyPlaybookBoost) can't promote a
// recorded answer that wasn't already in the regular retrieval's
// top-K. Paraphrase queries broke this — different embedding ⇒
// different top-K ⇒ recorded answer drops out ⇒ no boost can save
// it. Reality test playbook_lift_002 showed 0/2 paraphrase top-1
// lifts because of exactly that.
//
// Synthetic distance = playbook_hit_distance × BoostFactor — same
// formula as ApplyPlaybookBoost, applied to the playbook hit's own
// distance instead of a result's. Lower playbook hit distance
// (current query is similar to recorded query) AND higher score
// (recorded outcome was strong) push the injection toward top-1.
//
// fetchPlaybookHits has already filtered hits to those within
// DefaultPlaybookMaxDistance (0.5), so injected results land in the
// same distance range as regular retrieval — they don't dominate
// top-K from out-of-distribution playbooks.
//
// Returns the (possibly extended) results slice and how many synthetic
// rows were appended. Caller MUST re-sort + truncate to K afterwards.
func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int) {
if len(hits) == 0 {
return results, 0
}
present := make(map[string]bool, len(results))
for _, r := range results {
present[r.Corpus+"|"+r.ID] = true
}
// For each (corpus, id) NOT in results, keep the playbook hit
// with the largest boost (lowest BoostFactor = highest score).
// Multiple hits to the same answer collapse to one injection.
bestForKey := make(map[string]PlaybookHit)
for _, h := range hits {
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
if present[key] {
continue
}
if existing, ok := bestForKey[key]; !ok || h.Entry.BoostFactor() < existing.Entry.BoostFactor() {
bestForKey[key] = h
}
}
for _, h := range bestForKey {
injectedDist := h.Distance * float32(h.Entry.BoostFactor())
// Synthesize metadata that flags the injection so callers
// (driver/UI/observer) can distinguish "regular retrieval"
// from "playbook injection." Production consumers needing
// the actual worker metadata can fetch from vectord by
// (Corpus, ID) — synthetic results carry only provenance.
meta, _ := json.Marshal(map[string]any{
"playbook_injected": true,
"playbook_id": h.PlaybookID,
"playbook_score": h.Entry.Score,
"playbook_query_text": h.Entry.QueryText,
"playbook_recorded_at_ns": h.Entry.RecordedAtNs,
"playbook_hit_distance": h.Distance,
})
results = append(results, Result{
ID: h.Entry.AnswerID,
Corpus: h.Entry.AnswerCorpus,
Distance: injectedDist,
Metadata: meta,
})
}
return results, len(bestForKey)
}
// ApplyPlaybookBoost re-ranks results in place using matched
// playbook hits. For each hit whose (AnswerID, AnswerCorpus)
// matches a result, multiply that result's distance by the hit's