profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

matrix: split boost / inject thresholds — kills Shape B cross-pollination

Browse Source 
Run #003 surfaced rampant cross-pollination: Q2's "OSHA-30 forklift
Wisconsin" recording (w-4435) became warm top-1 for Q19 (dental
hygienist), Q20 (RN), Q21 (software engineer), and 6 other unrelated
staffing queries. Cause: InjectPlaybookMisses inherited the same
DefaultPlaybookMaxDistance (0.5) as the boost path, but inject is
structurally riskier than boost — boost only re-ranks results that
already retrieved on their own merits, while inject FORCES a result
into top-K, so a loose match cross-pollinates wrong-domain answers.

Empirical motivation from v3:
  Implied playbook hit distances for cross-pollinated cases: 0.20-0.46
  Implied distances for the 6/6 paraphrase recoveries:        0.23-0.30
  Threshold of 0.20 should keep most paraphrases, kill the OOD bleed.

Implementation:
- New DefaultPlaybookMaxInjectDistance = 0.20 in playbook.go.
- New PlaybookMaxInjectDistance field on SearchRequest (override).
- InjectPlaybookMisses signature gains maxInjectDist param; hits whose
  Distance exceeds it are skipped (boost path may still re-rank them).
- TestInjectPlaybookMisses_RespectsInjectThreshold locks the contract
  with one tight + one loose hit, asserting only the tight one injects.
- Existing tests pass explicit threshold (0 = default for tight tests,
  0.5 for the dedupe test which uses 0.30 hits).

Run #004 result on identical queries with the split threshold:

  Verbatim discovery        8 (vs v3's 6 — judge variance, separate)
  Verbatim lift             6 / 8 (75%)
  Paraphrase top-1          6 / 8 (75%)
  Paraphrase any-rank in K  6 / 8

OOD queries Q19/Q20/Q21 ALL show warm top-1 = cold top-1 (no
injection) — cross-pollination eliminated where it was wrong-direction.
Mean Δ top-1 distance dropped from -0.164 (v3, distorted) to -0.071
(v4, comparable to v1's -0.053).

Two paraphrases missed in v4 (Q9, Q15) were ones where qwen2.5
rephrased liberally enough to drift past 0.20 — Q9: "Inventory
specialist..." → "Individual needed for inventory management..." and
Q15: "Engaged warehouse associate..." → "Warehouse associate currently
engaged with a robust history...". The system correctly refusing to
inject when it's not confident is the right product behavior; the
boost path still re-ranks recorded answers when they appear in regular
retrieval.

The Q6 ↔ Q7 cross-pollination ("Forklift-certified loader" ↔
"Hazmat warehouse worker") is legitimate — these are genuinely similar
staffing queries and the judge ranks both directions as plausible.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 07:24:55 -05:00
co-authored by Claude Opus 4.7
parent 94fc3b67ec
commit 67d1957b87
4 changed files with 217 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/matrix/playbook.go
+39 -1
View File
@@ -49,8 +49,31 @@ const DefaultPlaybookTopK = 3
// query is similar enough to count." 0.5 lets in genuinely related
// queries while excluding pure-coincidence neighbors. Caller can
// override per-request as we learn what works for staffing data.
//
// This threshold gates the BOOST path (re-rank in place), which is
// safe at loose thresholds because boost only modifies results already
// in regular retrieval. The INJECT path uses a tighter ceiling — see
// DefaultPlaybookMaxInjectDistance.
const DefaultPlaybookMaxDistance = 0.5
// DefaultPlaybookMaxInjectDistance is the SHAPE B cosine ceiling for
// "this past query is similar enough to FORCE its answer into the
// result set." Tighter than DefaultPlaybookMaxDistance because inject
// is structurally riskier than boost: it adds a result the embedding
// didn't surface, so a loose match can cross-pollinate the wrong
// answer into unrelated queries.
//
// Empirical motivation (playbook_lift_003): Q2's recording for an
// OSHA-30 forklift operator surfaced as warm top-1 for the dental
// hygienist / RN / software engineer OOD queries because their text
// vectors fell within 0.5 cosine of "OSHA-30 forklift Wisconsin."
// 0.20 would have rejected those (implied playbook distances 0.38-0.46)
// while keeping all 6 paraphrase recoveries (≤ 0.30 implied).
//
// Boost path stays at 0.5 — re-ranking results that already retrieved
// by their own merits is safe even when the playbook match is loose.
const DefaultPlaybookMaxInjectDistance = 0.20
// PlaybookEntry is what gets stored as metadata on each playbook
// vector. RecordedAt is captured at write time; callers should not
// set it (the recorder fills it in).
@@ -174,10 +197,18 @@ type PlaybookHit struct {
//
// Returns the (possibly extended) results slice and how many synthetic
// rows were appended. Caller MUST re-sort + truncate to K afterwards.
func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int) {
//
// maxInjectDist filters which hits qualify for injection — hits whose
// playbook-corpus cosine distance exceeds it are skipped (the boost
// path may still re-rank them in place). Pass 0 (or any non-positive
// value) to use DefaultPlaybookMaxInjectDistance.
func InjectPlaybookMisses(results []Result, hits []PlaybookHit, maxInjectDist float32) ([]Result, int) {
if len(hits) == 0 {
return results, 0
}
if maxInjectDist <= 0 {
maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
}
present := make(map[string]bool, len(results))
for _, r := range results {
present[r.Corpus+"|"+r.ID] = true
@@ -188,6 +219,13 @@ func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int)
// Multiple hits to the same answer collapse to one injection.
bestForKey := make(map[string]PlaybookHit)
for _, h := range hits {
// Inject-specific tighter threshold (boost path's threshold is
// looser; this prevents cross-pollination of wrong-domain
// answers into queries whose text happens to fall within
// boost-distance of an unrelated recording).
if h.Distance > maxInjectDist {
continue
}
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
if present[key] {
continue
internal/matrix/playbook_test.go
+45 -4
View File
@@ -187,7 +187,7 @@ func TestInjectPlaybookMisses_AddsMissingAnswers(t *testing.T) {
},
},
}
out, injected := InjectPlaybookMisses(results, hits)
out, injected := InjectPlaybookMisses(results, hits, 0)
if injected != 1 {
t.Fatalf("expected 1 injected, got %d", injected)
}
@@ -240,7 +240,7 @@ func TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent(t *testing.T) {
},
},
}
out, injected := InjectPlaybookMisses(results, hits)
out, injected := InjectPlaybookMisses(results, hits, 0)
if injected != 0 {
t.Errorf("expected 0 injected (answer already present), got %d", injected)
}
@@ -266,7 +266,7 @@ func TestInjectPlaybookMisses_DedupesPerAnswer(t *testing.T) {
Entry: PlaybookEntry{QueryText: "q2", AnswerID: "w-99", AnswerCorpus: "workers", Score: 1.0},
},
}
out, injected := InjectPlaybookMisses(results, hits)
out, injected := InjectPlaybookMisses(results, hits, 0.5) // explicit loose threshold so 0.30 hits qualify
if injected != 1 {
t.Errorf("expected 1 injection (deduped), got %d", injected)
}
@@ -280,10 +280,51 @@ func TestInjectPlaybookMisses_DedupesPerAnswer(t *testing.T) {
}
}
// TestInjectPlaybookMisses_RespectsInjectThreshold locks the
// cross-pollination defense added after run #003: hits whose playbook
// distance exceeds the inject threshold are skipped, preventing the
// "OSHA-30 forklift" recording from surfacing as warm top-1 for an
// unrelated dental-hygienist query just because their text vectors
// happened to fall within boost-threshold (0.5).
func TestInjectPlaybookMisses_RespectsInjectThreshold(t *testing.T) {
results := []Result{{ID: "w-1", Corpus: "workers", Distance: 0.30}}
// Two hits: one within tight inject threshold, one beyond it but
// within boost threshold. Only the tight one should inject.
hits := []PlaybookHit{
{
PlaybookID: "tight",
Distance: 0.10, // within inject (true paraphrase territory)
Entry: PlaybookEntry{QueryText: "q1", AnswerID: "w-tight", AnswerCorpus: "workers", Score: 1.0},
},
{
PlaybookID: "loose",
Distance: 0.40, // boost-eligible but inject-rejected
Entry: PlaybookEntry{QueryText: "q2", AnswerID: "w-loose", AnswerCorpus: "workers", Score: 1.0},
},
}
// Default threshold (0 → DefaultPlaybookMaxInjectDistance = 0.20)
out, injected := InjectPlaybookMisses(results, hits, 0)
if injected != 1 {
t.Errorf("expected 1 injection (only the tight hit qualifies), got %d", injected)
}
gotTight := false
for _, r := range out {
if r.ID == "w-tight" {
gotTight = true
}
if r.ID == "w-loose" {
t.Errorf("loose hit (distance > inject threshold) was injected anyway")
}
}
if !gotTight {
t.Error("tight hit should have been injected")
}
}
// TestInjectPlaybookMisses_EmptyHits is a fast-path no-op check.
func TestInjectPlaybookMisses_EmptyHits(t *testing.T) {
results := []Result{{ID: "w-1", Corpus: "workers", Distance: 0.30}}
out, injected := InjectPlaybookMisses(results, nil)
out, injected := InjectPlaybookMisses(results, nil, 0)
if injected != 0 {
t.Errorf("expected 0 injection, got %d", injected)
}
internal/matrix/retrieve.go
+16 -5
View File
@@ -53,8 +53,14 @@ type Result struct {
// PlaybookCorpus: index name; empty = DefaultPlaybookCorpus.
// PlaybookTopK: number of similar past queries to consider; 0 =
// DefaultPlaybookTopK.
// PlaybookMaxDistance: cosine ceiling for "similar enough"; 0 =
// DefaultPlaybookMaxDistance.
// PlaybookMaxDistance: cosine ceiling for "similar enough" on the
// BOOST path (re-rank in place); 0 = DefaultPlaybookMaxDistance.
// PlaybookMaxInjectDistance: tighter cosine ceiling for the SHAPE B
// INJECT path; 0 = DefaultPlaybookMaxInjectDistance. Splitting the
// two thresholds is intentional — boost is safe at loose thresholds
// because it only re-ranks results that already retrieved on their
// own merits, while inject forces results in and so cross-pollinates
// wrong-domain answers if the threshold is too loose.
//
// Metadata filter (post-retrieval structured gate):
// MetadataFilter: map of metadata-field → expected value. Results
@@ -76,8 +82,9 @@ type SearchRequest struct {
UsePlaybook bool `json:"use_playbook,omitempty"`
PlaybookCorpus string `json:"playbook_corpus,omitempty"`
PlaybookTopK int `json:"playbook_top_k,omitempty"`
PlaybookMaxDistance float64 `json:"playbook_max_distance,omitempty"`
MetadataFilter map[string]any `json:"metadata_filter,omitempty"`
PlaybookMaxDistance float64 `json:"playbook_max_distance,omitempty"`
PlaybookMaxInjectDistance float64 `json:"playbook_max_inject_distance,omitempty"`
MetadataFilter map[string]any `json:"metadata_filter,omitempty"`
}
// SearchResponse wraps the merged results plus per-corpus return
@@ -233,8 +240,12 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
slog.Warn("matrix: playbook lookup failed; skipping boost+inject", "err", err)
} else if len(hits) > 0 {
resp.PlaybookBoosted = ApplyPlaybookBoost(resp.Results, hits)
maxInjectDist := float32(req.PlaybookMaxInjectDistance)
if maxInjectDist <= 0 {
maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
}
var injected int
resp.Results, injected = InjectPlaybookMisses(resp.Results, hits)
resp.Results, injected = InjectPlaybookMisses(resp.Results, hits, maxInjectDist)
resp.PlaybookInjected = injected
if injected > 0 {
// Re-sort + truncate after injection. ApplyPlaybookBoost