matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery
The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).
Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.
Result on playbook_lift_003 (Shape B + paraphrase pass):
Verbatim discovery 6
Verbatim lift 2 / 6
**Paraphrase top-1** **6 / 6**
Paraphrase any-rank in K 6 / 6
Mean Δ top-1 distance -0.1637 (warm closer than cold)
Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.
Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.
Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op
Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -85,10 +85,18 @@ type queryRun struct {
|
||||
// Paraphrase pass — only populated when --with-paraphrase. Tests
|
||||
// the playbook's actual learning property: does a recorded entry
|
||||
// for query Q help a similar-but-different query Q'?
|
||||
//
|
||||
// ParaphraseRecordedRank semantics:
|
||||
// nil = paraphrase pass didn't run for this query (no playbook
|
||||
// was recorded in cold pass, so nothing to test)
|
||||
// 0 = recorded answer landed at top-1
|
||||
// 1..K-1 = recorded answer present in top-K at that rank
|
||||
// -1 = recorded answer absent from top-K
|
||||
// Pointer (not int) so nil and rank-0 are distinguishable in JSON.
|
||||
ParaphraseQuery string `json:"paraphrase_query,omitempty"`
|
||||
ParaphraseTop1ID string `json:"paraphrase_top1_id,omitempty"`
|
||||
ParaphraseRecordedRank int `json:"paraphrase_recorded_rank,omitempty"` // -1 if not in top-K
|
||||
ParaphraseLift bool `json:"paraphrase_lift,omitempty"` // recorded answer at rank 0 for paraphrase
|
||||
ParaphraseRecordedRank *int `json:"paraphrase_recorded_rank,omitempty"`
|
||||
ParaphraseLift bool `json:"paraphrase_lift,omitempty"` // recorded answer at rank 0 for paraphrase
|
||||
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
@@ -273,7 +281,8 @@ func main() {
|
||||
resp, err := matrixSearch(hc, *gw, paraphrase, corpora, *k, true)
|
||||
if err != nil || len(resp.Results) == 0 {
|
||||
runs[i].Note = appendNote(runs[i].Note, fmt.Sprintf("paraphrase search failed: %v", err))
|
||||
runs[i].ParaphraseRecordedRank = -1
|
||||
missed := -1
|
||||
runs[i].ParaphraseRecordedRank = &missed
|
||||
continue
|
||||
}
|
||||
runs[i].ParaphraseTop1ID = resp.Results[0].ID
|
||||
@@ -284,7 +293,7 @@ func main() {
|
||||
break
|
||||
}
|
||||
}
|
||||
runs[i].ParaphraseRecordedRank = recordedRank
|
||||
runs[i].ParaphraseRecordedRank = &recordedRank
|
||||
if recordedRank == 0 {
|
||||
runs[i].ParaphraseLift = true
|
||||
paraphraseTop1Lifts++
|
||||
|
||||
Reference in New Issue
Block a user