matrix: cross-role playbook gate — closes real_001 bleed (OPEN #1)
real_001 surfaced same-client+city queries bleeding across roles:
Q#2 (Forklift Operator @ Beacon Freight Detroit) recorded e-6193
in the playbook corpus. Q#5 (Pickers same client+city) and Q#10
(CNC Operator same client+city) embedded within 0.13-0.18 cosine of
Q#2's query — well inside the 0.20 inject threshold — so e-6193
injected on both, demoting the cold-pass-correct workers.
Root cause: the inject distance threshold isn't tight enough on
the same-client+city cluster. Cosine collapses queries that share
city + client + count-token + time-token regardless of role. The
existing judge gate is per-injection at record time and doesn't
fire at retrieve time.
Fix: structural role gate in front of both Shape A boost and
Shape B inject. PlaybookEntry gains Role; SearchRequest gains
QueryRole. When both are non-empty and differ under roleEqual's
case+plural normalization, the entry is rejected before BoostFactor
or judge-gate logic runs.
Backward-compat: empty role on either side disables the gate —
preserves behavior for the lift suite's free-form multi-constraint
queries that have no clean single role. Caller-supplied (not
inferred), so existing recordings unaffected.
Wire-through:
- internal/matrix/playbook.go: Role field, NewPlaybookEntryWithRole,
roleEqual helper with plural+case normalization
- internal/matrix/retrieve.go: QueryRole on SearchRequest, threaded
to both ApplyPlaybookBoost + InjectPlaybookMisses
- cmd/matrixd/main.go: role on POST /matrix/playbooks/record + bulk
- scripts/playbook_lift/main.go: extractRoleFromNeed regex pulls
role from "Need N {role}{s} in" queries (the fill_events shape);
free-form queries fall back to empty (gate disabled)
Tests (5 new):
- TestInjectPlaybookMisses_RoleGateRejectsCrossRole: exact Q#10
scenario (distance 0.135, recorded "Forklift Operator", query
"CNC Operator") — locks the bleed at unit level
- TestInjectPlaybookMisses_RoleGateAllowsSameRole: Forklift Operator
recording fires on Forklift Operators query (plural normalization)
- TestInjectPlaybookMisses_RoleGateBackwardCompat: empty Role on
either side = gate disabled, preserves current behavior
- TestApplyPlaybookBoost_RoleGateRejectsCrossRole: Shape A defense
in depth — boost doesn't fire on cross-role even when answer is
in cold top-K
- TestRoleEqual_PluralAndCase: case + -s + -es plural normalization
Verification (real_002, same query set as real_001):
- Q#5 Pickers @ Beacon Freight: e-6193 → e-8499 (no bleed)
- Q#10 CNC Operator @ Beacon Freight: e-6193 → w-2404 (no bleed)
- Discoveries + lifts unchanged at 2 each (same-role lift still fires)
- Mean Δdist tightens from -0.127 to -0.040 (boosts no longer
pulling distances through the floor on cross-role mismatches)
Findings: reports/reality-tests/real_002_findings.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+112
-2
@@ -77,8 +77,20 @@ const DefaultPlaybookMaxInjectDistance = 0.20
|
||||
// PlaybookEntry is what gets stored as metadata on each playbook
|
||||
// vector. RecordedAt is captured at write time; callers should not
|
||||
// set it (the recorder fills it in).
|
||||
//
|
||||
// Role is an optional caller-provided hint that names the query's
|
||||
// staffing role at record time (e.g. "Forklift Operator", "CNC
|
||||
// Operator"). When set, the matrix retrieve layer gates Shape A boost
|
||||
// and Shape B inject so a recorded entry only fires for queries with
|
||||
// the same role. Empty Role = no gate (caller didn't know the role,
|
||||
// or the query has no clean role concept — e.g. lift-suite multi-
|
||||
// constraint queries). Reality test real_001 surfaced same-client+city
|
||||
// queries bleeding across roles when the embedder collapsed them
|
||||
// within the inject distance threshold; the role gate closes that
|
||||
// without rejecting legitimate intra-role boosts.
|
||||
type PlaybookEntry struct {
|
||||
QueryText string `json:"query_text"`
|
||||
Role string `json:"role,omitempty"`
|
||||
AnswerID string `json:"answer_id"`
|
||||
AnswerCorpus string `json:"answer_corpus"`
|
||||
Score float64 `json:"score"` // 0..1; higher = better outcome
|
||||
@@ -152,6 +164,10 @@ func UnmarshalPlaybookMetadata(raw json.RawMessage) (PlaybookEntry, error) {
|
||||
|
||||
// NewPlaybookEntry stamps RecordedAtNs to now and returns the entry.
|
||||
// Validation happens at storage; this is just construction.
|
||||
//
|
||||
// Role-aware overload is NewPlaybookEntryWithRole — callers that don't
|
||||
// know the role can keep using this constructor and the role gate
|
||||
// will degrade to current (un-gated) behavior.
|
||||
func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
|
||||
return PlaybookEntry{
|
||||
QueryText: query,
|
||||
@@ -163,6 +179,15 @@ func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags
|
||||
}
|
||||
}
|
||||
|
||||
// NewPlaybookEntryWithRole is the role-aware variant of NewPlaybookEntry.
|
||||
// role should be a stable role token (see roleEqual / role-token notes
|
||||
// in the cross-role gate); empty disables the gate for this entry.
|
||||
func NewPlaybookEntryWithRole(query, role, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
|
||||
e := NewPlaybookEntry(query, answerID, answerCorpus, score, tags)
|
||||
e.Role = role
|
||||
return e
|
||||
}
|
||||
|
||||
// PlaybookHit is one similarity-search result from the playbook
|
||||
// corpus, paired with its decoded entry. Distance is the cosine
|
||||
// distance between the current query and this past playbook's
|
||||
@@ -174,6 +199,64 @@ type PlaybookHit struct {
|
||||
Entry PlaybookEntry `json:"entry"`
|
||||
}
|
||||
|
||||
// roleEqual is the cross-role gate's match function. Returns true when
|
||||
// EITHER side is empty (gate disabled — current behavior preserved) OR
|
||||
// the role tokens compare equal under a normalization that ignores
|
||||
// case + trailing 's' (so "Forklift Operator" matches "forklift
|
||||
// operator" matches "Forklift Operators"). Robust enough for the
|
||||
// staffing-domain plurals seen in fill_events; tighter normalization
|
||||
// (Levenshtein, role-graph) is a separate concern and not blocking
|
||||
// the bleed fix.
|
||||
//
|
||||
// Why this lives near PlaybookEntry: the gate is a property of the
|
||||
// entry/query pair, not of the boost or inject mechanism — both call
|
||||
// sites in this file delegate to the same function so they can't
|
||||
// drift apart.
|
||||
func roleEqual(a, b string) bool {
|
||||
if a == "" || b == "" {
|
||||
return true
|
||||
}
|
||||
return roleNormalize(a) == roleNormalize(b)
|
||||
}
|
||||
|
||||
// roleNormalize lowercases + trims + strips a single trailing "s"
|
||||
// (or "es" after "x"/"s"/"ch"/"sh") to collapse common staffing
|
||||
// plurals without pulling in a real stemmer.
|
||||
func roleNormalize(s string) string {
|
||||
out := []rune{}
|
||||
// in-place lowercase + skip leading/trailing whitespace
|
||||
for _, r := range s {
|
||||
if r == ' ' || r == '\t' {
|
||||
if len(out) == 0 {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if r >= 'A' && r <= 'Z' {
|
||||
r += 32
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
// trim trailing whitespace
|
||||
for len(out) > 0 && (out[len(out)-1] == ' ' || out[len(out)-1] == '\t') {
|
||||
out = out[:len(out)-1]
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return ""
|
||||
}
|
||||
// crude singularization for staffing plurals
|
||||
n := len(out)
|
||||
if n >= 3 && out[n-1] == 's' && out[n-2] == 'e' &&
|
||||
(out[n-3] == 'x' || out[n-3] == 's' || out[n-3] == 'h') {
|
||||
// "boxes", "kisses", "dishes" → strip "es"
|
||||
out = out[:n-2]
|
||||
} else if n >= 2 && out[n-1] == 's' {
|
||||
// "operators" → "operator"; harmless on words ending in s if
|
||||
// callers mind their canonicalization (rare in role names).
|
||||
out = out[:n-1]
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
|
||||
// InjectPlaybookMisses appends synthetic Results for playbook hits
|
||||
// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
|
||||
// This is "Shape B" from the doc comment at the top of this file:
|
||||
@@ -219,7 +302,17 @@ type PlaybookHit struct {
|
||||
// it can score (query, candidate) pairs without re-deriving from
|
||||
// SearchRequest. Empty when the caller doesn't have it (gate
|
||||
// implementations should treat empty query as "skip judge, allow").
|
||||
func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
|
||||
//
|
||||
// queryRole, when non-empty, gates injection on role match: a playbook
|
||||
// hit whose Entry.Role is set and DIFFERENT from queryRole is rejected
|
||||
// before the judge gate runs. Closes the same-client+city cross-role
|
||||
// bleed (real_001 finding) where the inject distance threshold (0.20)
|
||||
// wasn't tight enough to prevent Forklift-Operator playbook entries
|
||||
// from injecting on CNC-Operator queries that happened to embed close
|
||||
// (same city + client + count-token + time-token dominate cosine).
|
||||
// Empty queryRole OR empty hit.Entry.Role = no gate (preserves
|
||||
// behavior for callers that don't supply role).
|
||||
func InjectPlaybookMisses(query, queryRole string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
|
||||
if len(hits) == 0 {
|
||||
return results, 0
|
||||
}
|
||||
@@ -243,6 +336,12 @@ func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, ma
|
||||
if h.Distance > maxInjectDist {
|
||||
continue
|
||||
}
|
||||
// Cross-role gate (real_001 bleed fix). Reject before the
|
||||
// judge gate so we don't burn a slow LLM call to immediately
|
||||
// reject the candidate on a structural mismatch.
|
||||
if !roleEqual(queryRole, h.Entry.Role) {
|
||||
continue
|
||||
}
|
||||
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
|
||||
if present[key] {
|
||||
continue
|
||||
@@ -316,13 +415,19 @@ func (f InjectGateFunc) Approve(q string, h PlaybookHit) bool { return f(q, h) }
|
||||
// BoostFactor. If multiple hits match the same result, the highest-
|
||||
// score one wins (greatest reduction in distance).
|
||||
//
|
||||
// queryRole, when non-empty, gates the boost on role match — same
|
||||
// semantics as InjectPlaybookMisses' queryRole. Defense in depth:
|
||||
// even if a recorded entry's answer happens to surface in the
|
||||
// regular retrieve top-K for a different-role query, role-mismatch
|
||||
// shouldn't fire a boost. Empty role on either side disables the gate.
|
||||
//
|
||||
// After applying boosts, results are re-sorted ascending by
|
||||
// distance.
|
||||
//
|
||||
// Returns the number of distinct results that received a boost.
|
||||
// Callers can log this as a signal of "how much the playbook
|
||||
// influenced this query."
|
||||
func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
|
||||
func ApplyPlaybookBoost(queryRole string, results []Result, hits []PlaybookHit) int {
|
||||
if len(hits) == 0 || len(results) == 0 {
|
||||
return 0
|
||||
}
|
||||
@@ -336,6 +441,11 @@ func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
|
||||
if h.Entry.AnswerID != r.ID || h.Entry.AnswerCorpus != r.Corpus {
|
||||
continue
|
||||
}
|
||||
// Cross-role gate (real_001 bleed fix). Same semantics as
|
||||
// InjectPlaybookMisses — see comment there.
|
||||
if !roleEqual(queryRole, h.Entry.Role) {
|
||||
continue
|
||||
}
|
||||
bf := h.Entry.BoostFactor()
|
||||
if cur, ok := bestBoost[i]; !ok || bf < cur {
|
||||
bestBoost[i] = bf
|
||||
|
||||
Reference in New Issue
Block a user