matrix: split boost / inject thresholds — kills Shape B cross-pollination
Run #003 surfaced rampant cross-pollination: Q2's "OSHA-30 forklift Wisconsin" recording (w-4435) became warm top-1 for Q19 (dental hygienist), Q20 (RN), Q21 (software engineer), and 6 other unrelated staffing queries. Cause: InjectPlaybookMisses inherited the same DefaultPlaybookMaxDistance (0.5) as the boost path, but inject is structurally riskier than boost — boost only re-ranks results that already retrieved on their own merits, while inject FORCES a result into top-K, so a loose match cross-pollinates wrong-domain answers. Empirical motivation from v3: Implied playbook hit distances for cross-pollinated cases: 0.20-0.46 Implied distances for the 6/6 paraphrase recoveries: 0.23-0.30 Threshold of 0.20 should keep most paraphrases, kill the OOD bleed. Implementation: - New DefaultPlaybookMaxInjectDistance = 0.20 in playbook.go. - New PlaybookMaxInjectDistance field on SearchRequest (override). - InjectPlaybookMisses signature gains maxInjectDist param; hits whose Distance exceeds it are skipped (boost path may still re-rank them). - TestInjectPlaybookMisses_RespectsInjectThreshold locks the contract with one tight + one loose hit, asserting only the tight one injects. - Existing tests pass explicit threshold (0 = default for tight tests, 0.5 for the dedupe test which uses 0.30 hits). Run #004 result on identical queries with the split threshold: Verbatim discovery 8 (vs v3's 6 — judge variance, separate) Verbatim lift 6 / 8 (75%) Paraphrase top-1 6 / 8 (75%) Paraphrase any-rank in K 6 / 8 OOD queries Q19/Q20/Q21 ALL show warm top-1 = cold top-1 (no injection) — cross-pollination eliminated where it was wrong-direction. Mean Δ top-1 distance dropped from -0.164 (v3, distorted) to -0.071 (v4, comparable to v1's -0.053). Two paraphrases missed in v4 (Q9, Q15) were ones where qwen2.5 rephrased liberally enough to drift past 0.20 — Q9: "Inventory specialist..." → "Individual needed for inventory management..." and Q15: "Engaged warehouse associate..." → "Warehouse associate currently engaged with a robust history...". The system correctly refusing to inject when it's not confident is the right product behavior; the boost path still re-ranks recorded answers when they appear in regular retrieval. The Q6 ↔ Q7 cross-pollination ("Forklift-certified loader" ↔ "Hazmat warehouse worker") is legitimate — these are genuinely similar staffing queries and the judge ranks both directions as plausible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -49,8 +49,31 @@ const DefaultPlaybookTopK = 3
|
||||
// query is similar enough to count." 0.5 lets in genuinely related
|
||||
// queries while excluding pure-coincidence neighbors. Caller can
|
||||
// override per-request as we learn what works for staffing data.
|
||||
//
|
||||
// This threshold gates the BOOST path (re-rank in place), which is
|
||||
// safe at loose thresholds because boost only modifies results already
|
||||
// in regular retrieval. The INJECT path uses a tighter ceiling — see
|
||||
// DefaultPlaybookMaxInjectDistance.
|
||||
const DefaultPlaybookMaxDistance = 0.5
|
||||
|
||||
// DefaultPlaybookMaxInjectDistance is the SHAPE B cosine ceiling for
|
||||
// "this past query is similar enough to FORCE its answer into the
|
||||
// result set." Tighter than DefaultPlaybookMaxDistance because inject
|
||||
// is structurally riskier than boost: it adds a result the embedding
|
||||
// didn't surface, so a loose match can cross-pollinate the wrong
|
||||
// answer into unrelated queries.
|
||||
//
|
||||
// Empirical motivation (playbook_lift_003): Q2's recording for an
|
||||
// OSHA-30 forklift operator surfaced as warm top-1 for the dental
|
||||
// hygienist / RN / software engineer OOD queries because their text
|
||||
// vectors fell within 0.5 cosine of "OSHA-30 forklift Wisconsin."
|
||||
// 0.20 would have rejected those (implied playbook distances 0.38-0.46)
|
||||
// while keeping all 6 paraphrase recoveries (≤ 0.30 implied).
|
||||
//
|
||||
// Boost path stays at 0.5 — re-ranking results that already retrieved
|
||||
// by their own merits is safe even when the playbook match is loose.
|
||||
const DefaultPlaybookMaxInjectDistance = 0.20
|
||||
|
||||
// PlaybookEntry is what gets stored as metadata on each playbook
|
||||
// vector. RecordedAt is captured at write time; callers should not
|
||||
// set it (the recorder fills it in).
|
||||
@@ -174,10 +197,18 @@ type PlaybookHit struct {
|
||||
//
|
||||
// Returns the (possibly extended) results slice and how many synthetic
|
||||
// rows were appended. Caller MUST re-sort + truncate to K afterwards.
|
||||
func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int) {
|
||||
//
|
||||
// maxInjectDist filters which hits qualify for injection — hits whose
|
||||
// playbook-corpus cosine distance exceeds it are skipped (the boost
|
||||
// path may still re-rank them in place). Pass 0 (or any non-positive
|
||||
// value) to use DefaultPlaybookMaxInjectDistance.
|
||||
func InjectPlaybookMisses(results []Result, hits []PlaybookHit, maxInjectDist float32) ([]Result, int) {
|
||||
if len(hits) == 0 {
|
||||
return results, 0
|
||||
}
|
||||
if maxInjectDist <= 0 {
|
||||
maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
|
||||
}
|
||||
present := make(map[string]bool, len(results))
|
||||
for _, r := range results {
|
||||
present[r.Corpus+"|"+r.ID] = true
|
||||
@@ -188,6 +219,13 @@ func InjectPlaybookMisses(results []Result, hits []PlaybookHit) ([]Result, int)
|
||||
// Multiple hits to the same answer collapse to one injection.
|
||||
bestForKey := make(map[string]PlaybookHit)
|
||||
for _, h := range hits {
|
||||
// Inject-specific tighter threshold (boost path's threshold is
|
||||
// looser; this prevents cross-pollination of wrong-domain
|
||||
// answers into queries whose text happens to fall within
|
||||
// boost-distance of an unrelated recording).
|
||||
if h.Distance > maxInjectDist {
|
||||
continue
|
||||
}
|
||||
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
|
||||
if present[key] {
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user