matrix: cross-role playbook gate — closes real_001 bleed (OPEN #1)

real_001 surfaced same-client+city queries bleeding across roles:
Q#2 (Forklift Operator @ Beacon Freight Detroit) recorded e-6193
in the playbook corpus. Q#5 (Pickers same client+city) and Q#10
(CNC Operator same client+city) embedded within 0.13-0.18 cosine of
Q#2's query — well inside the 0.20 inject threshold — so e-6193
injected on both, demoting the cold-pass-correct workers.

Root cause: the inject distance threshold isn't tight enough on
the same-client+city cluster. Cosine collapses queries that share
city + client + count-token + time-token regardless of role. The
existing judge gate is per-injection at record time and doesn't
fire at retrieve time.

Fix: structural role gate in front of both Shape A boost and
Shape B inject. PlaybookEntry gains Role; SearchRequest gains
QueryRole. When both are non-empty and differ under roleEqual's
case+plural normalization, the entry is rejected before BoostFactor
or judge-gate logic runs.

Backward-compat: empty role on either side disables the gate —
preserves behavior for the lift suite's free-form multi-constraint
queries that have no clean single role. Caller-supplied (not
inferred), so existing recordings unaffected.

Wire-through:
- internal/matrix/playbook.go: Role field, NewPlaybookEntryWithRole,
  roleEqual helper with plural+case normalization
- internal/matrix/retrieve.go: QueryRole on SearchRequest, threaded
  to both ApplyPlaybookBoost + InjectPlaybookMisses
- cmd/matrixd/main.go: role on POST /matrix/playbooks/record + bulk
- scripts/playbook_lift/main.go: extractRoleFromNeed regex pulls
  role from "Need N {role}{s} in" queries (the fill_events shape);
  free-form queries fall back to empty (gate disabled)

Tests (5 new):
- TestInjectPlaybookMisses_RoleGateRejectsCrossRole: exact Q#10
  scenario (distance 0.135, recorded "Forklift Operator", query
  "CNC Operator") — locks the bleed at unit level
- TestInjectPlaybookMisses_RoleGateAllowsSameRole: Forklift Operator
  recording fires on Forklift Operators query (plural normalization)
- TestInjectPlaybookMisses_RoleGateBackwardCompat: empty Role on
  either side = gate disabled, preserves current behavior
- TestApplyPlaybookBoost_RoleGateRejectsCrossRole: Shape A defense
  in depth — boost doesn't fire on cross-role even when answer is
  in cold top-K
- TestRoleEqual_PluralAndCase: case + -s + -es plural normalization

Verification (real_002, same query set as real_001):
- Q#5 Pickers @ Beacon Freight: e-6193 → e-8499 (no bleed)
- Q#10 CNC Operator @ Beacon Freight: e-6193 → w-2404 (no bleed)
- Discoveries + lifts unchanged at 2 each (same-role lift still fires)
- Mean Δdist tightens from -0.127 to -0.040 (boosts no longer
  pulling distances through the floor on cross-role mismatches)

Findings: reports/reality-tests/real_002_findings.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 20:34:10 -05:00
co-authored by Claude Opus 4.7
parent 7f2f112e6a
commit 997527be4d
8 changed files with 513 additions and 24 deletions
+112 -2
View File
@@ -77,8 +77,20 @@ const DefaultPlaybookMaxInjectDistance = 0.20
// PlaybookEntry is what gets stored as metadata on each playbook
// vector. RecordedAt is captured at write time; callers should not
// set it (the recorder fills it in).
//
// Role is an optional caller-provided hint that names the query's
// staffing role at record time (e.g. "Forklift Operator", "CNC
// Operator"). When set, the matrix retrieve layer gates Shape A boost
// and Shape B inject so a recorded entry only fires for queries with
// the same role. Empty Role = no gate (caller didn't know the role,
// or the query has no clean role concept — e.g. lift-suite multi-
// constraint queries). Reality test real_001 surfaced same-client+city
// queries bleeding across roles when the embedder collapsed them
// within the inject distance threshold; the role gate closes that
// without rejecting legitimate intra-role boosts.
type PlaybookEntry struct {
QueryText string `json:"query_text"`
Role string `json:"role,omitempty"`
AnswerID string `json:"answer_id"`
AnswerCorpus string `json:"answer_corpus"`
Score float64 `json:"score"` // 0..1; higher = better outcome
@@ -152,6 +164,10 @@ func UnmarshalPlaybookMetadata(raw json.RawMessage) (PlaybookEntry, error) {
// NewPlaybookEntry stamps RecordedAtNs to now and returns the entry.
// Validation happens at storage; this is just construction.
//
// Role-aware overload is NewPlaybookEntryWithRole — callers that don't
// know the role can keep using this constructor and the role gate
// will degrade to current (un-gated) behavior.
func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
return PlaybookEntry{
QueryText: query,
@@ -163,6 +179,15 @@ func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags
}
}
// NewPlaybookEntryWithRole is the role-aware variant of NewPlaybookEntry.
// role should be a stable role token (see roleEqual / role-token notes
// in the cross-role gate); empty disables the gate for this entry.
func NewPlaybookEntryWithRole(query, role, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
e := NewPlaybookEntry(query, answerID, answerCorpus, score, tags)
e.Role = role
return e
}
// PlaybookHit is one similarity-search result from the playbook
// corpus, paired with its decoded entry. Distance is the cosine
// distance between the current query and this past playbook's
@@ -174,6 +199,64 @@ type PlaybookHit struct {
Entry PlaybookEntry `json:"entry"`
}
// roleEqual is the cross-role gate's match function. Returns true when
// EITHER side is empty (gate disabled — current behavior preserved) OR
// the role tokens compare equal under a normalization that ignores
// case + trailing 's' (so "Forklift Operator" matches "forklift
// operator" matches "Forklift Operators"). Robust enough for the
// staffing-domain plurals seen in fill_events; tighter normalization
// (Levenshtein, role-graph) is a separate concern and not blocking
// the bleed fix.
//
// Why this lives near PlaybookEntry: the gate is a property of the
// entry/query pair, not of the boost or inject mechanism — both call
// sites in this file delegate to the same function so they can't
// drift apart.
func roleEqual(a, b string) bool {
if a == "" || b == "" {
return true
}
return roleNormalize(a) == roleNormalize(b)
}
// roleNormalize lowercases + trims + strips a single trailing "s"
// (or "es" after "x"/"s"/"ch"/"sh") to collapse common staffing
// plurals without pulling in a real stemmer.
func roleNormalize(s string) string {
out := []rune{}
// in-place lowercase + skip leading/trailing whitespace
for _, r := range s {
if r == ' ' || r == '\t' {
if len(out) == 0 {
continue
}
}
if r >= 'A' && r <= 'Z' {
r += 32
}
out = append(out, r)
}
// trim trailing whitespace
for len(out) > 0 && (out[len(out)-1] == ' ' || out[len(out)-1] == '\t') {
out = out[:len(out)-1]
}
if len(out) == 0 {
return ""
}
// crude singularization for staffing plurals
n := len(out)
if n >= 3 && out[n-1] == 's' && out[n-2] == 'e' &&
(out[n-3] == 'x' || out[n-3] == 's' || out[n-3] == 'h') {
// "boxes", "kisses", "dishes" → strip "es"
out = out[:n-2]
} else if n >= 2 && out[n-1] == 's' {
// "operators" → "operator"; harmless on words ending in s if
// callers mind their canonicalization (rare in role names).
out = out[:n-1]
}
return string(out)
}
// InjectPlaybookMisses appends synthetic Results for playbook hits
// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
// This is "Shape B" from the doc comment at the top of this file:
@@ -219,7 +302,17 @@ type PlaybookHit struct {
// it can score (query, candidate) pairs without re-deriving from
// SearchRequest. Empty when the caller doesn't have it (gate
// implementations should treat empty query as "skip judge, allow").
func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
//
// queryRole, when non-empty, gates injection on role match: a playbook
// hit whose Entry.Role is set and DIFFERENT from queryRole is rejected
// before the judge gate runs. Closes the same-client+city cross-role
// bleed (real_001 finding) where the inject distance threshold (0.20)
// wasn't tight enough to prevent Forklift-Operator playbook entries
// from injecting on CNC-Operator queries that happened to embed close
// (same city + client + count-token + time-token dominate cosine).
// Empty queryRole OR empty hit.Entry.Role = no gate (preserves
// behavior for callers that don't supply role).
func InjectPlaybookMisses(query, queryRole string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
if len(hits) == 0 {
return results, 0
}
@@ -243,6 +336,12 @@ func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, ma
if h.Distance > maxInjectDist {
continue
}
// Cross-role gate (real_001 bleed fix). Reject before the
// judge gate so we don't burn a slow LLM call to immediately
// reject the candidate on a structural mismatch.
if !roleEqual(queryRole, h.Entry.Role) {
continue
}
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
if present[key] {
continue
@@ -316,13 +415,19 @@ func (f InjectGateFunc) Approve(q string, h PlaybookHit) bool { return f(q, h) }
// BoostFactor. If multiple hits match the same result, the highest-
// score one wins (greatest reduction in distance).
//
// queryRole, when non-empty, gates the boost on role match — same
// semantics as InjectPlaybookMisses' queryRole. Defense in depth:
// even if a recorded entry's answer happens to surface in the
// regular retrieve top-K for a different-role query, role-mismatch
// shouldn't fire a boost. Empty role on either side disables the gate.
//
// After applying boosts, results are re-sorted ascending by
// distance.
//
// Returns the number of distinct results that received a boost.
// Callers can log this as a signal of "how much the playbook
// influenced this query."
func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
func ApplyPlaybookBoost(queryRole string, results []Result, hits []PlaybookHit) int {
if len(hits) == 0 || len(results) == 0 {
return 0
}
@@ -336,6 +441,11 @@ func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
if h.Entry.AnswerID != r.ID || h.Entry.AnswerCorpus != r.Corpus {
continue
}
// Cross-role gate (real_001 bleed fix). Same semantics as
// InjectPlaybookMisses — see comment there.
if !roleEqual(queryRole, h.Entry.Role) {
continue
}
bf := h.Entry.BoostFactor()
if cur, ok := bestBoost[i]; !ok || bf < cur {
bestBoost[i] = bf