matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery
The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).
Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.
Result on playbook_lift_003 (Shape B + paraphrase pass):
Verbatim discovery 6
Verbatim lift 2 / 6
**Paraphrase top-1** **6 / 6**
Paraphrase any-rank in K 6 / 6
Mean Δ top-1 distance -0.1637 (warm closer than cold)
Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.
Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.
Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op
Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -91,6 +91,11 @@ type SearchResponse struct {
|
||||
Results []Result `json:"results"`
|
||||
PerCorpusCounts map[string]int `json:"per_corpus_counts"`
|
||||
PlaybookBoosted int `json:"playbook_boosted,omitempty"`
|
||||
// PlaybookInjected is Shape B's per-query metric: synthetic
|
||||
// results inserted from playbook hits whose answer wasn't already
|
||||
// in the regular retrieval. Distinct from PlaybookBoosted (which
|
||||
// counts in-place re-ranks of results that WERE present).
|
||||
PlaybookInjected int `json:"playbook_injected,omitempty"`
|
||||
MetadataFilterDropped int `json:"metadata_filter_dropped,omitempty"`
|
||||
}
|
||||
|
||||
@@ -218,17 +223,30 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
|
||||
MetadataFilterDropped: dropped,
|
||||
}
|
||||
|
||||
// Playbook boost (component 5). Reuses the query vector — no
|
||||
// extra embed call. If the playbook corpus doesn't exist (first
|
||||
// search before any Record), the lookup gracefully no-ops.
|
||||
// Playbook (component 5) — both boost (re-rank existing) and
|
||||
// inject (Shape B: bring in answers that aren't in regular
|
||||
// retrieval). Reuses the query vector — no extra embed call.
|
||||
// Missing playbook corpus is a legitimate cold-start no-op.
|
||||
if req.UsePlaybook {
|
||||
hits, err := r.fetchPlaybookHits(ctx, qvec, req)
|
||||
if err != nil {
|
||||
// Don't fail the whole search on playbook errors — the
|
||||
// boost is opportunistic. Log + continue.
|
||||
slog.Warn("matrix: playbook lookup failed; skipping boost", "err", err)
|
||||
slog.Warn("matrix: playbook lookup failed; skipping boost+inject", "err", err)
|
||||
} else if len(hits) > 0 {
|
||||
resp.PlaybookBoosted = ApplyPlaybookBoost(resp.Results, hits)
|
||||
var injected int
|
||||
resp.Results, injected = InjectPlaybookMisses(resp.Results, hits)
|
||||
resp.PlaybookInjected = injected
|
||||
if injected > 0 {
|
||||
// Re-sort + truncate after injection. ApplyPlaybookBoost
|
||||
// already sorted, but injection appends past the end —
|
||||
// resort to merge, then enforce K.
|
||||
sort.SliceStable(resp.Results, func(i, j int) bool {
|
||||
return resp.Results[i].Distance < resp.Results[j].Distance
|
||||
})
|
||||
if len(resp.Results) > req.K {
|
||||
resp.Results = resp.Results[:req.K]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user