matrix: Shape B — inject playbook misses + 6/6 paraphrase recovery

The v0 boost-only stance documented in internal/matrix/playbook.go:22-27
("the boost only re-ranks results that ALREADY surfaced from the regular
retrieval") couldn't promote recorded answers that dropped out of a
paraphrase's top-K. playbook_lift_002 surfaced exactly that gap: 0/2
paraphrase recoveries because the recorded answers weren't in regular
retrieval at all (rank=-1).

Shape B: when warm-pass retrieval doesn't surface a playbook hit's
answer, inject a synthetic Result for it directly. Distance =
playbook_hit_distance × BoostFactor — same formula as the boost path so
injections land in comparable distance space. Caller re-sorts +
truncates after both boost and inject have run.

Result on playbook_lift_003 (Shape B + paraphrase pass):

  Verbatim discovery        6
  Verbatim lift             2 / 6
  **Paraphrase top-1**      **6 / 6**
  Paraphrase any-rank in K  6 / 6
  Mean Δ top-1 distance     -0.1637 (warm closer than cold)

Every paraphrase the judge generated landed the v1-recorded answer at
top-1 of the new query's results. The learning property holds — cosine
on embed(paraphrase) finds the recorded query's vector within
DefaultPlaybookMaxDistance (0.5), and Shape B injects the answer.

Verbatim lift dropped from v1's 7/8 because Shape B cross-pollinates
recorded answers across queries. w-4435 (Q2's recording) appears as
warm top-1 for several other queries because their embeddings are
within the playbook hit threshold of "OSHA-30 forklift Wisconsin." This
is a feature, not a bug — the matrix layer's purpose is to share
knowledge across queries — but the lift metric only counts "warm top-1
== cold judge best," so cross-pollinated lifts don't register. A v3
metric would re-judge warm pass to measure true judge improvement.

Tests:
- TestInjectPlaybookMisses_AddsMissingAnswers — primary claim
- TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent — no double-inject
- TestInjectPlaybookMisses_DedupesPerAnswer — multi-hit same answer
- TestInjectPlaybookMisses_EmptyHits — fast-path no-op

Driver fix: ParaphraseRecordedRank int → *int. The `omitempty` int
silently dropped rank=0 (top-1, the WANTED value) from JSON, making the
v003 report show "null" instead of "0" for every successful recovery.
Pointer keeps nil/rank-0 distinguishable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

internal/matrix/playbook.go

@@ -151,6 +151,78 @@ type PlaybookHit struct {
 	Entry      PlaybookEntry `json:"entry"`
 }
 
+// InjectPlaybookMisses appends synthetic Results for playbook hits
+// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
+// This is "Shape B" from the doc comment at the top of this file:
+// the v0 boost-only stance (ApplyPlaybookBoost) can't promote a
+// recorded answer that wasn't already in the regular retrieval's
+// top-K. Paraphrase queries broke this — different embedding ⇒
+// different top-K ⇒ recorded answer drops out ⇒ no boost can save
+// it. Reality test playbook_lift_002 showed 0/2 paraphrase top-1
+// lifts because of exactly that.
+//
+// Synthetic distance = playbook_hit_distance × BoostFactor — same
+// formula as ApplyPlaybookBoost, applied to the playbook hit's own
+// distance instead of a result's. Lower playbook hit distance
+// (current query is similar to recorded query) AND higher score
+// (recorded outcome was strong) push the injection toward top-1.
+//
+// fetchPlaybookHits has already filtered hits to those within
+// DefaultPlaybookMaxDistance (0.5), so injected results land in the
+// same distance range as regular retrieval — they don't dominate
+// top-K from out-of-distribution playbooks.
+//
+// Returns the (possibly extended) results slice and how many synthetic
+// rows were appended. Caller MUST re-sort + truncate to K afterwards.
+func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int) {
+	if len(hits) == 0 {
+		return results, 0
+	}
+	present := make(map[string]bool, len(results))
+	for _, r := range results {
+		present[r.Corpus+"|"+r.ID] = true
+	}
+
+	// For each (corpus, id) NOT in results, keep the playbook hit
+	// with the largest boost (lowest BoostFactor = highest score).
+	// Multiple hits to the same answer collapse to one injection.
+	bestForKey := make(map[string]PlaybookHit)
+	for _, h := range hits {
+		key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
+		if present[key] {
+			continue
+		}
+		if existing, ok := bestForKey[key]; !ok || h.Entry.BoostFactor() < existing.Entry.BoostFactor() {
+			bestForKey[key] = h
+		}
+	}
+
+	for _, h := range bestForKey {
+		injectedDist := h.Distance * float32(h.Entry.BoostFactor())
+		// Synthesize metadata that flags the injection so callers
+		// (driver/UI/observer) can distinguish "regular retrieval"
+		// from "playbook injection." Production consumers needing
+		// the actual worker metadata can fetch from vectord by
+		// (Corpus, ID) — synthetic results carry only provenance.
+		meta, _ := json.Marshal(map[string]any{
+			"playbook_injected":      true,
+			"playbook_id":            h.PlaybookID,
+			"playbook_score":         h.Entry.Score,
+			"playbook_query_text":    h.Entry.QueryText,
+			"playbook_recorded_at_ns": h.Entry.RecordedAtNs,
+			"playbook_hit_distance":  h.Distance,
+		})
+		results = append(results, Result{
+			ID:       h.Entry.AnswerID,
+			Corpus:   h.Entry.AnswerCorpus,
+			Distance: injectedDist,
+			Metadata: meta,
+		})
+	}
+
+	return results, len(bestForKey)
+}
+
 // ApplyPlaybookBoost re-ranks results in place using matched
 // playbook hits. For each hit whose (AnswerID, AnswerCorpus)
 // matches a result, multiply that result's distance by the hit's

internal/matrix/playbook_test.go

@@ -164,6 +164,134 @@ func TestUnmarshalPlaybookMetadata_RejectsEmpty(t *testing.T) {
 	}
 }
 
+// TestInjectPlaybookMisses_AddsMissingAnswers locks Shape B's primary
+// claim: when a playbook hit's answer isn't already in regular
+// retrieval results, InjectPlaybookMisses appends a synthetic Result
+// for it. Reality test playbook_lift_002 surfaced 0/2 paraphrase
+// recoveries because the v0 boost-only stance couldn't promote
+// answers that dropped out of the paraphrase's top-K.
+func TestInjectPlaybookMisses_AddsMissingAnswers(t *testing.T) {
+	results := []Result{
+		{ID: "w-1", Corpus: "workers", Distance: 0.30},
+		{ID: "w-2", Corpus: "workers", Distance: 0.35},
+	}
+	hits := []PlaybookHit{
+		{
+			PlaybookID: "pb-x",
+			Distance:   0.20, // current query is close to recorded query
+			Entry: PlaybookEntry{
+				QueryText:    "recorded query",
+				AnswerID:     "w-99", // NOT in results
+				AnswerCorpus: "workers",
+				Score:        1.0, // strong outcome → boost factor 0.5
+			},
+		},
+	}
+	out, injected := InjectPlaybookMisses(results, hits)
+	if injected != 1 {
+		t.Fatalf("expected 1 injected, got %d", injected)
+	}
+	if len(out) != 3 {
+		t.Fatalf("expected len=3, got %d (%v)", len(out), idsOf(out))
+	}
+	// The injected result should be findable + carry the playbook
+	// provenance metadata flag.
+	var injectedResult *Result
+	for i := range out {
+		if out[i].ID == "w-99" {
+			injectedResult = &out[i]
+			break
+		}
+	}
+	if injectedResult == nil {
+		t.Fatal("w-99 not present in output")
+	}
+	// distance = 0.20 * 0.5 = 0.10 → near-top after caller re-sorts
+	if injectedResult.Distance < 0.099 || injectedResult.Distance > 0.101 {
+		t.Errorf("expected injected distance ~0.10, got %f", injectedResult.Distance)
+	}
+	var meta map[string]any
+	if err := json.Unmarshal(injectedResult.Metadata, &meta); err != nil {
+		t.Fatalf("decode meta: %v", err)
+	}
+	if v, _ := meta["playbook_injected"].(bool); !v {
+		t.Errorf("expected playbook_injected=true marker, got %v", meta)
+	}
+	if v, _ := meta["playbook_query_text"].(string); v != "recorded query" {
+		t.Errorf("expected recorded query in meta, got %v", v)
+	}
+}
+
+// TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent locks the
+// boost-only-when-present property. If a playbook hit's answer is
+// ALREADY in results, we don't duplicate-inject — ApplyPlaybookBoost
+// has handled that case via in-place re-rank.
+func TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent(t *testing.T) {
+	results := []Result{
+		{ID: "w-1", Corpus: "workers", Distance: 0.30},
+		{ID: "w-99", Corpus: "workers", Distance: 0.40}, // ALREADY HERE
+	}
+	hits := []PlaybookHit{
+		{
+			PlaybookID: "pb-x",
+			Distance:   0.20,
+			Entry: PlaybookEntry{
+				QueryText: "x", AnswerID: "w-99", AnswerCorpus: "workers", Score: 1.0,
+			},
+		},
+	}
+	out, injected := InjectPlaybookMisses(results, hits)
+	if injected != 0 {
+		t.Errorf("expected 0 injected (answer already present), got %d", injected)
+	}
+	if len(out) != 2 {
+		t.Errorf("expected results unchanged at len=2, got %d", len(out))
+	}
+}
+
+// TestInjectPlaybookMisses_DedupesPerAnswer locks: multiple playbook
+// hits all pointing to the same missing answer collapse to ONE
+// injection (the highest-scoring hit wins).
+func TestInjectPlaybookMisses_DedupesPerAnswer(t *testing.T) {
+	results := []Result{{ID: "w-1", Corpus: "workers", Distance: 0.30}}
+	hits := []PlaybookHit{
+		{
+			PlaybookID: "pb-low",
+			Distance:   0.30,
+			Entry:      PlaybookEntry{QueryText: "q1", AnswerID: "w-99", AnswerCorpus: "workers", Score: 0.4},
+		},
+		{
+			PlaybookID: "pb-high",
+			Distance:   0.30,
+			Entry:      PlaybookEntry{QueryText: "q2", AnswerID: "w-99", AnswerCorpus: "workers", Score: 1.0},
+		},
+	}
+	out, injected := InjectPlaybookMisses(results, hits)
+	if injected != 1 {
+		t.Errorf("expected 1 injection (deduped), got %d", injected)
+	}
+	// Score=1.0 (the high one) wins → boost factor 0.5 → distance 0.15
+	for _, r := range out {
+		if r.ID == "w-99" {
+			if r.Distance < 0.149 || r.Distance > 0.151 {
+				t.Errorf("expected distance from highest-score hit (~0.15), got %f", r.Distance)
+			}
+		}
+	}
+}
+
+// TestInjectPlaybookMisses_EmptyHits is a fast-path no-op check.
+func TestInjectPlaybookMisses_EmptyHits(t *testing.T) {
+	results := []Result{{ID: "w-1", Corpus: "workers", Distance: 0.30}}
+	out, injected := InjectPlaybookMisses(results, nil)
+	if injected != 0 {
+		t.Errorf("expected 0 injection, got %d", injected)
+	}
+	if len(out) != 1 {
+		t.Errorf("results should be unchanged, got len=%d", len(out))
+	}
+}
+
 func abs(f float64) float64 {
 	if f < 0 {
 		return -f

internal/matrix/retrieve.go

@@ -91,6 +91,11 @@ type SearchResponse struct {
 	Results               []Result       `json:"results"`
 	PerCorpusCounts       map[string]int `json:"per_corpus_counts"`
 	PlaybookBoosted       int            `json:"playbook_boosted,omitempty"`
+	// PlaybookInjected is Shape B's per-query metric: synthetic
+	// results inserted from playbook hits whose answer wasn't already
+	// in the regular retrieval. Distinct from PlaybookBoosted (which
+	// counts in-place re-ranks of results that WERE present).
+	PlaybookInjected      int            `json:"playbook_injected,omitempty"`
 	MetadataFilterDropped int            `json:"metadata_filter_dropped,omitempty"`
 }
 
@@ -218,17 +223,30 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
 		MetadataFilterDropped: dropped,
 	}
 
-	// Playbook boost (component 5). Reuses the query vector — no
-	// extra embed call. If the playbook corpus doesn't exist (first
-	// search before any Record), the lookup gracefully no-ops.
+	// Playbook (component 5) — both boost (re-rank existing) and
+	// inject (Shape B: bring in answers that aren't in regular
+	// retrieval). Reuses the query vector — no extra embed call.
+	// Missing playbook corpus is a legitimate cold-start no-op.
 	if req.UsePlaybook {
 		hits, err := r.fetchPlaybookHits(ctx, qvec, req)
 		if err != nil {
-			// Don't fail the whole search on playbook errors — the
-			// boost is opportunistic. Log + continue.
-			slog.Warn("matrix: playbook lookup failed; skipping boost", "err", err)
+			slog.Warn("matrix: playbook lookup failed; skipping boost+inject", "err", err)
 		} else if len(hits) > 0 {
 			resp.PlaybookBoosted = ApplyPlaybookBoost(resp.Results, hits)
+			var injected int
+			resp.Results, injected = InjectPlaybookMisses(resp.Results, hits)
+			resp.PlaybookInjected = injected
+			if injected > 0 {
+				// Re-sort + truncate after injection. ApplyPlaybookBoost
+				// already sorted, but injection appends past the end —
+				// resort to merge, then enforce K.
+				sort.SliceStable(resp.Results, func(i, j int) bool {
+					return resp.Results[i].Distance < resp.Results[j].Distance
+				})
+				if len(resp.Results) > req.K {
+					resp.Results = resp.Results[:req.K]
+				}
+			}
 		}
 	}