matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery
The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).
Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.
Result on playbook_lift_003 (Shape B + paraphrase pass):
Verbatim discovery 6
Verbatim lift 2 / 6
**Paraphrase top-1** **6 / 6**
Paraphrase any-rank in K 6 / 6
Mean Δ top-1 distance -0.1637 (warm closer than cold)
Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.
Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.
Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op
Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -151,6 +151,78 @@ type PlaybookHit struct {
|
||||
Entry PlaybookEntry `json:"entry"`
|
||||
}
|
||||
|
||||
// InjectPlaybookMisses appends synthetic Results for playbook hits
|
||||
// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
|
||||
// This is "Shape B" from the doc comment at the top of this file:
|
||||
// the v0 boost-only stance (ApplyPlaybookBoost) can't promote a
|
||||
// recorded answer that wasn't already in the regular retrieval's
|
||||
// top-K. Paraphrase queries broke this — different embedding ⇒
|
||||
// different top-K ⇒ recorded answer drops out ⇒ no boost can save
|
||||
// it. Reality test playbook_lift_002 showed 0/2 paraphrase top-1
|
||||
// lifts because of exactly that.
|
||||
//
|
||||
// Synthetic distance = playbook_hit_distance × BoostFactor — same
|
||||
// formula as ApplyPlaybookBoost, applied to the playbook hit's own
|
||||
// distance instead of a result's. Lower playbook hit distance
|
||||
// (current query is similar to recorded query) AND higher score
|
||||
// (recorded outcome was strong) push the injection toward top-1.
|
||||
//
|
||||
// fetchPlaybookHits has already filtered hits to those within
|
||||
// DefaultPlaybookMaxDistance (0.5), so injected results land in the
|
||||
// same distance range as regular retrieval — they don't dominate
|
||||
// top-K from out-of-distribution playbooks.
|
||||
//
|
||||
// Returns the (possibly extended) results slice and how many synthetic
|
||||
// rows were appended. Caller MUST re-sort + truncate to K afterwards.
|
||||
func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int) {
|
||||
if len(hits) == 0 {
|
||||
return results, 0
|
||||
}
|
||||
present := make(map[string]bool, len(results))
|
||||
for _, r := range results {
|
||||
present[r.Corpus+"|"+r.ID] = true
|
||||
}
|
||||
|
||||
// For each (corpus, id) NOT in results, keep the playbook hit
|
||||
// with the largest boost (lowest BoostFactor = highest score).
|
||||
// Multiple hits to the same answer collapse to one injection.
|
||||
bestForKey := make(map[string]PlaybookHit)
|
||||
for _, h := range hits {
|
||||
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
|
||||
if present[key] {
|
||||
continue
|
||||
}
|
||||
if existing, ok := bestForKey[key]; !ok || h.Entry.BoostFactor() < existing.Entry.BoostFactor() {
|
||||
bestForKey[key] = h
|
||||
}
|
||||
}
|
||||
|
||||
for _, h := range bestForKey {
|
||||
injectedDist := h.Distance * float32(h.Entry.BoostFactor())
|
||||
// Synthesize metadata that flags the injection so callers
|
||||
// (driver/UI/observer) can distinguish "regular retrieval"
|
||||
// from "playbook injection." Production consumers needing
|
||||
// the actual worker metadata can fetch from vectord by
|
||||
// (Corpus, ID) — synthetic results carry only provenance.
|
||||
meta, _ := json.Marshal(map[string]any{
|
||||
"playbook_injected": true,
|
||||
"playbook_id": h.PlaybookID,
|
||||
"playbook_score": h.Entry.Score,
|
||||
"playbook_query_text": h.Entry.QueryText,
|
||||
"playbook_recorded_at_ns": h.Entry.RecordedAtNs,
|
||||
"playbook_hit_distance": h.Distance,
|
||||
})
|
||||
results = append(results, Result{
|
||||
ID: h.Entry.AnswerID,
|
||||
Corpus: h.Entry.AnswerCorpus,
|
||||
Distance: injectedDist,
|
||||
Metadata: meta,
|
||||
})
|
||||
}
|
||||
|
||||
return results, len(bestForKey)
|
||||
}
|
||||
|
||||
// ApplyPlaybookBoost re-ranks results in place using matched
|
||||
// playbook hits. For each hit whose (AnswerID, AnswerCorpus)
|
||||
// matches a result, multiply that result's distance by the hit's
|
||||
|
||||
Reference in New Issue
Block a user