matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery

The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).

Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.

Result on playbook_lift_003 (Shape B + paraphrase pass):

  Verbatim discovery        6
  Verbatim lift             2 / 6
  **Paraphrase top-1**      **6 / 6**
  Paraphrase any-rank in K  6 / 6
  Mean Δ top-1 distance     -0.1637 (warm closer than cold)

Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.

Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.

Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op

Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 07:06:13 -05:00
co-authored by Claude Opus 4.7
parent e9822f025d
commit 154a72ea5e
5 changed files with 352 additions and 10 deletions
+13 -4
View File
@@ -85,10 +85,18 @@ type queryRun struct {
// Paraphrase pass — only populated when --with-paraphrase. Tests
// the playbook's actual learning property: does a recorded entry
// for query Q help a similar-but-different query Q'?
//
// ParaphraseRecordedRank semantics:
// nil = paraphrase pass didn't run for this query (no playbook
// was recorded in cold pass, so nothing to test)
// 0 = recorded answer landed at top-1
// 1..K-1 = recorded answer present in top-K at that rank
// -1 = recorded answer absent from top-K
// Pointer (not int) so nil and rank-0 are distinguishable in JSON.
ParaphraseQuery string `json:"paraphrase_query,omitempty"`
ParaphraseTop1ID string `json:"paraphrase_top1_id,omitempty"`
ParaphraseRecordedRank int `json:"paraphrase_recorded_rank,omitempty"` // -1 if not in top-K
ParaphraseLift bool `json:"paraphrase_lift,omitempty"` // recorded answer at rank 0 for paraphrase
ParaphraseRecordedRank *int `json:"paraphrase_recorded_rank,omitempty"`
ParaphraseLift bool `json:"paraphrase_lift,omitempty"` // recorded answer at rank 0 for paraphrase
Note string `json:"note,omitempty"`
}
@@ -273,7 +281,8 @@ func main() {
resp, err := matrixSearch(hc, *gw, paraphrase, corpora, *k, true)
if err != nil || len(resp.Results) == 0 {
runs[i].Note = appendNote(runs[i].Note, fmt.Sprintf("paraphrase search failed: %v", err))
runs[i].ParaphraseRecordedRank = -1
missed := -1
runs[i].ParaphraseRecordedRank = &missed
continue
}
runs[i].ParaphraseTop1ID = resp.Results[0].ID
@@ -284,7 +293,7 @@ func main() {
break
}
}
runs[i].ParaphraseRecordedRank = recordedRank
runs[i].ParaphraseRecordedRank = &recordedRank
if recordedRank == 0 {
runs[i].ParaphraseLift = true
paraphraseTop1Lifts++