matrix: split boost / inject thresholds — kills Shape B cross-pollination
Run #003 surfaced rampant cross-pollination: Q2's "OSHA-30 forklift Wisconsin" recording (w-4435) became warm top-1 for Q19 (dental hygienist), Q20 (RN), Q21 (software engineer), and 6 other unrelated staffing queries. Cause: InjectPlaybookMisses inherited the same DefaultPlaybookMaxDistance (0.5) as the boost path, but inject is structurally riskier than boost — boost only re-ranks results that already retrieved on their own merits, while inject FORCES a result into top-K, so a loose match cross-pollinates wrong-domain answers. Empirical motivation from v3: Implied playbook hit distances for cross-pollinated cases: 0.20-0.46 Implied distances for the 6/6 paraphrase recoveries: 0.23-0.30 Threshold of 0.20 should keep most paraphrases, kill the OOD bleed. Implementation: - New DefaultPlaybookMaxInjectDistance = 0.20 in playbook.go. - New PlaybookMaxInjectDistance field on SearchRequest (override). - InjectPlaybookMisses signature gains maxInjectDist param; hits whose Distance exceeds it are skipped (boost path may still re-rank them). - TestInjectPlaybookMisses_RespectsInjectThreshold locks the contract with one tight + one loose hit, asserting only the tight one injects. - Existing tests pass explicit threshold (0 = default for tight tests, 0.5 for the dedupe test which uses 0.30 hits). Run #004 result on identical queries with the split threshold: Verbatim discovery 8 (vs v3's 6 — judge variance, separate) Verbatim lift 6 / 8 (75%) Paraphrase top-1 6 / 8 (75%) Paraphrase any-rank in K 6 / 8 OOD queries Q19/Q20/Q21 ALL show warm top-1 = cold top-1 (no injection) — cross-pollination eliminated where it was wrong-direction. Mean Δ top-1 distance dropped from -0.164 (v3, distorted) to -0.071 (v4, comparable to v1's -0.053). Two paraphrases missed in v4 (Q9, Q15) were ones where qwen2.5 rephrased liberally enough to drift past 0.20 — Q9: "Inventory specialist..." → "Individual needed for inventory management..." and Q15: "Engaged warehouse associate..." → "Warehouse associate currently engaged with a robust history...". The system correctly refusing to inject when it's not confident is the right product behavior; the boost path still re-ranks recorded answers when they appear in regular retrieval. The Q6 ↔ Q7 cross-pollination ("Forklift-certified loader" ↔ "Hazmat warehouse worker") is legitimate — these are genuinely similar staffing queries and the judge ranks both directions as plausible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -53,8 +53,14 @@ type Result struct {
|
||||
// PlaybookCorpus: index name; empty = DefaultPlaybookCorpus.
|
||||
// PlaybookTopK: number of similar past queries to consider; 0 =
|
||||
// DefaultPlaybookTopK.
|
||||
// PlaybookMaxDistance: cosine ceiling for "similar enough"; 0 =
|
||||
// DefaultPlaybookMaxDistance.
|
||||
// PlaybookMaxDistance: cosine ceiling for "similar enough" on the
|
||||
// BOOST path (re-rank in place); 0 = DefaultPlaybookMaxDistance.
|
||||
// PlaybookMaxInjectDistance: tighter cosine ceiling for the SHAPE B
|
||||
// INJECT path; 0 = DefaultPlaybookMaxInjectDistance. Splitting the
|
||||
// two thresholds is intentional — boost is safe at loose thresholds
|
||||
// because it only re-ranks results that already retrieved on their
|
||||
// own merits, while inject forces results in and so cross-pollinates
|
||||
// wrong-domain answers if the threshold is too loose.
|
||||
//
|
||||
// Metadata filter (post-retrieval structured gate):
|
||||
// MetadataFilter: map of metadata-field → expected value. Results
|
||||
@@ -76,8 +82,9 @@ type SearchRequest struct {
|
||||
UsePlaybook bool `json:"use_playbook,omitempty"`
|
||||
PlaybookCorpus string `json:"playbook_corpus,omitempty"`
|
||||
PlaybookTopK int `json:"playbook_top_k,omitempty"`
|
||||
PlaybookMaxDistance float64 `json:"playbook_max_distance,omitempty"`
|
||||
MetadataFilter map[string]any `json:"metadata_filter,omitempty"`
|
||||
PlaybookMaxDistance float64 `json:"playbook_max_distance,omitempty"`
|
||||
PlaybookMaxInjectDistance float64 `json:"playbook_max_inject_distance,omitempty"`
|
||||
MetadataFilter map[string]any `json:"metadata_filter,omitempty"`
|
||||
}
|
||||
|
||||
// SearchResponse wraps the merged results plus per-corpus return
|
||||
@@ -233,8 +240,12 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
|
||||
slog.Warn("matrix: playbook lookup failed; skipping boost+inject", "err", err)
|
||||
} else if len(hits) > 0 {
|
||||
resp.PlaybookBoosted = ApplyPlaybookBoost(resp.Results, hits)
|
||||
maxInjectDist := float32(req.PlaybookMaxInjectDistance)
|
||||
if maxInjectDist <= 0 {
|
||||
maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
|
||||
}
|
||||
var injected int
|
||||
resp.Results, injected = InjectPlaybookMisses(resp.Results, hits)
|
||||
resp.Results, injected = InjectPlaybookMisses(resp.Results, hits, maxInjectDist)
|
||||
resp.PlaybookInjected = injected
|
||||
if injected > 0 {
|
||||
// Re-sort + truncate after injection. ApplyPlaybookBoost
|
||||
|
||||
Reference in New Issue
Block a user