profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

matrix: cross-role playbook gate — closes real_001 bleed (OPEN #1)

Browse Source 
real_001 surfaced same-client+city queries bleeding across roles:
Q#2 (Forklift Operator @ Beacon Freight Detroit) recorded e-6193
in the playbook corpus. Q#5 (Pickers same client+city) and Q#10
(CNC Operator same client+city) embedded within 0.13-0.18 cosine of
Q#2's query — well inside the 0.20 inject threshold — so e-6193
injected on both, demoting the cold-pass-correct workers.

Root cause: the inject distance threshold isn't tight enough on
the same-client+city cluster. Cosine collapses queries that share
city + client + count-token + time-token regardless of role. The
existing judge gate is per-injection at record time and doesn't
fire at retrieve time.

Fix: structural role gate in front of both Shape A boost and
Shape B inject. PlaybookEntry gains Role; SearchRequest gains
QueryRole. When both are non-empty and differ under roleEqual's
case+plural normalization, the entry is rejected before BoostFactor
or judge-gate logic runs.

Backward-compat: empty role on either side disables the gate —
preserves behavior for the lift suite's free-form multi-constraint
queries that have no clean single role. Caller-supplied (not
inferred), so existing recordings unaffected.

Wire-through:
- internal/matrix/playbook.go: Role field, NewPlaybookEntryWithRole,
  roleEqual helper with plural+case normalization
- internal/matrix/retrieve.go: QueryRole on SearchRequest, threaded
  to both ApplyPlaybookBoost + InjectPlaybookMisses
- cmd/matrixd/main.go: role on POST /matrix/playbooks/record + bulk
- scripts/playbook_lift/main.go: extractRoleFromNeed regex pulls
  role from "Need N {role}{s} in" queries (the fill_events shape);
  free-form queries fall back to empty (gate disabled)

Tests (5 new):
- TestInjectPlaybookMisses_RoleGateRejectsCrossRole: exact Q#10
  scenario (distance 0.135, recorded "Forklift Operator", query
  "CNC Operator") — locks the bleed at unit level
- TestInjectPlaybookMisses_RoleGateAllowsSameRole: Forklift Operator
  recording fires on Forklift Operators query (plural normalization)
- TestInjectPlaybookMisses_RoleGateBackwardCompat: empty Role on
  either side = gate disabled, preserves current behavior
- TestApplyPlaybookBoost_RoleGateRejectsCrossRole: Shape A defense
  in depth — boost doesn't fire on cross-role even when answer is
  in cold top-K
- TestRoleEqual_PluralAndCase: case + -s + -es plural normalization

Verification (real_002, same query set as real_001):
- Q#5 Pickers @ Beacon Freight: e-6193 → e-8499 (no bleed)
- Q#10 CNC Operator @ Beacon Freight: e-6193 → w-2404 (no bleed)
- Discoveries + lifts unchanged at 2 each (same-role lift still fires)
- Mean Δdist tightens from -0.127 to -0.040 (boosts no longer
  pulling distances through the floor on cross-role mismatches)

Findings: reports/reality-tests/real_002_findings.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 20:34:10 -05:00
co-authored by Claude Opus 4.7
parent 7f2f112e6a
commit 997527be4d
8 changed files with 513 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/matrix/playbook.go
+112 -2
View File
@@ -77,8 +77,20 @@ const DefaultPlaybookMaxInjectDistance = 0.20
// PlaybookEntry is what gets stored as metadata on each playbook
// vector. RecordedAt is captured at write time; callers should not
// set it (the recorder fills it in).
//
// Role is an optional caller-provided hint that names the query's
// staffing role at record time (e.g. "Forklift Operator", "CNC
// Operator"). When set, the matrix retrieve layer gates Shape A boost
// and Shape B inject so a recorded entry only fires for queries with
// the same role. Empty Role = no gate (caller didn't know the role,
// or the query has no clean role concept — e.g. lift-suite multi-
// constraint queries). Reality test real_001 surfaced same-client+city
// queries bleeding across roles when the embedder collapsed them
// within the inject distance threshold; the role gate closes that
// without rejecting legitimate intra-role boosts.
type PlaybookEntry struct {
QueryText string `json:"query_text"`
Role string `json:"role,omitempty"`
AnswerID string `json:"answer_id"`
AnswerCorpus string `json:"answer_corpus"`
Score float64 `json:"score"` // 0..1; higher = better outcome
@@ -152,6 +164,10 @@ func UnmarshalPlaybookMetadata(raw json.RawMessage) (PlaybookEntry, error) {
// NewPlaybookEntry stamps RecordedAtNs to now and returns the entry.
// Validation happens at storage; this is just construction.
//
// Role-aware overload is NewPlaybookEntryWithRole — callers that don't
// know the role can keep using this constructor and the role gate
// will degrade to current (un-gated) behavior.
func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
return PlaybookEntry{
QueryText: query,
@@ -163,6 +179,15 @@ func NewPlaybookEntry(query, answerID, answerCorpus string, score float64, tags
}
}
// NewPlaybookEntryWithRole is the role-aware variant of NewPlaybookEntry.
// role should be a stable role token (see roleEqual / role-token notes
// in the cross-role gate); empty disables the gate for this entry.
func NewPlaybookEntryWithRole(query, role, answerID, answerCorpus string, score float64, tags []string) PlaybookEntry {
e := NewPlaybookEntry(query, answerID, answerCorpus, score, tags)
e.Role = role
return e
}
// PlaybookHit is one similarity-search result from the playbook
// corpus, paired with its decoded entry. Distance is the cosine
// distance between the current query and this past playbook's
@@ -174,6 +199,64 @@ type PlaybookHit struct {
Entry PlaybookEntry `json:"entry"`
}
// roleEqual is the cross-role gate's match function. Returns true when
// EITHER side is empty (gate disabled — current behavior preserved) OR
// the role tokens compare equal under a normalization that ignores
// case + trailing 's' (so "Forklift Operator" matches "forklift
// operator" matches "Forklift Operators"). Robust enough for the
// staffing-domain plurals seen in fill_events; tighter normalization
// (Levenshtein, role-graph) is a separate concern and not blocking
// the bleed fix.
//
// Why this lives near PlaybookEntry: the gate is a property of the
// entry/query pair, not of the boost or inject mechanism — both call
// sites in this file delegate to the same function so they can't
// drift apart.
func roleEqual(a, b string) bool {
if a == "" || b == "" {
return true
}
return roleNormalize(a) == roleNormalize(b)
}
// roleNormalize lowercases + trims + strips a single trailing "s"
// (or "es" after "x"/"s"/"ch"/"sh") to collapse common staffing
// plurals without pulling in a real stemmer.
func roleNormalize(s string) string {
out := []rune{}
// in-place lowercase + skip leading/trailing whitespace
for _, r := range s {
if r == ' ' || r == '\t' {
if len(out) == 0 {
continue
}
}
if r >= 'A' && r <= 'Z' {
r += 32
}
out = append(out, r)
}
// trim trailing whitespace
for len(out) > 0 && (out[len(out)-1] == ' ' || out[len(out)-1] == '\t') {
out = out[:len(out)-1]
}
if len(out) == 0 {
return ""
}
// crude singularization for staffing plurals
n := len(out)
if n >= 3 && out[n-1] == 's' && out[n-2] == 'e' &&
(out[n-3] == 'x' || out[n-3] == 's' || out[n-3] == 'h') {
// "boxes", "kisses", "dishes" → strip "es"
out = out[:n-2]
} else if n >= 2 && out[n-1] == 's' {
// "operators" → "operator"; harmless on words ending in s if
// callers mind their canonicalization (rare in role names).
out = out[:n-1]
}
return string(out)
}
// InjectPlaybookMisses appends synthetic Results for playbook hits
// whose (AnswerCorpus, AnswerID) doesn't already appear in results.
// This is "Shape B" from the doc comment at the top of this file:
@@ -219,7 +302,17 @@ type PlaybookHit struct {
// it can score (query, candidate) pairs without re-deriving from
// SearchRequest. Empty when the caller doesn't have it (gate
// implementations should treat empty query as "skip judge, allow").
func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
//
// queryRole, when non-empty, gates injection on role match: a playbook
// hit whose Entry.Role is set and DIFFERENT from queryRole is rejected
// before the judge gate runs. Closes the same-client+city cross-role
// bleed (real_001 finding) where the inject distance threshold (0.20)
// wasn't tight enough to prevent Forklift-Operator playbook entries
// from injecting on CNC-Operator queries that happened to embed close
// (same city + client + count-token + time-token dominate cosine).
// Empty queryRole OR empty hit.Entry.Role = no gate (preserves
// behavior for callers that don't supply role).
func InjectPlaybookMisses(query, queryRole string, results []Result, hits []PlaybookHit, maxInjectDist float32, gate InjectGate) ([]Result, int) {
if len(hits) == 0 {
return results, 0
}
@@ -243,6 +336,12 @@ func InjectPlaybookMisses(query string, results []Result, hits []PlaybookHit, ma
if h.Distance > maxInjectDist {
continue
}
// Cross-role gate (real_001 bleed fix). Reject before the
// judge gate so we don't burn a slow LLM call to immediately
// reject the candidate on a structural mismatch.
if !roleEqual(queryRole, h.Entry.Role) {
continue
}
key := h.Entry.AnswerCorpus + "|" + h.Entry.AnswerID
if present[key] {
continue
@@ -316,13 +415,19 @@ func (f InjectGateFunc) Approve(q string, h PlaybookHit) bool { return f(q, h) }
// BoostFactor. If multiple hits match the same result, the highest-
// score one wins (greatest reduction in distance).
//
// queryRole, when non-empty, gates the boost on role match — same
// semantics as InjectPlaybookMisses' queryRole. Defense in depth:
// even if a recorded entry's answer happens to surface in the
// regular retrieve top-K for a different-role query, role-mismatch
// shouldn't fire a boost. Empty role on either side disables the gate.
//
// After applying boosts, results are re-sorted ascending by
// distance.
//
// Returns the number of distinct results that received a boost.
// Callers can log this as a signal of "how much the playbook
// influenced this query."
func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
func ApplyPlaybookBoost(queryRole string, results []Result, hits []PlaybookHit) int {
if len(hits) == 0 || len(results) == 0 {
return 0
}
@@ -336,6 +441,11 @@ func ApplyPlaybookBoost(results []Result, hits []PlaybookHit) int {
if h.Entry.AnswerID != r.ID || h.Entry.AnswerCorpus != r.Corpus {
continue
}
// Cross-role gate (real_001 bleed fix). Same semantics as
// InjectPlaybookMisses — see comment there.
if !roleEqual(queryRole, h.Entry.Role) {
continue
}
bf := h.Entry.BoostFactor()
if cur, ok := bestBoost[i]; !ok || bf < cur {
bestBoost[i] = bf
internal/matrix/playbook_test.go
+152 -12
View File
@@ -54,7 +54,7 @@ func TestApplyPlaybookBoost_NoHitsLeaveResultsAlone(t *testing.T) {
{ID: "a", Distance: 0.1, Corpus: "x"},
{ID: "b", Distance: 0.2, Corpus: "x"},
}
n := ApplyPlaybookBoost(results, nil)
n := ApplyPlaybookBoost("", results, nil)
if n != 0 {
t.Errorf("expected 0 boosted, got %d", n)
}
@@ -77,7 +77,7 @@ func TestApplyPlaybookBoost_BoostMovesResultUp(t *testing.T) {
AnswerID: "c", AnswerCorpus: "x", Score: 1.0,
}},
}
n := ApplyPlaybookBoost(results, hits)
n := ApplyPlaybookBoost("", results, hits)
if n != 1 {
t.Errorf("expected 1 boosted, got %d", n)
}
@@ -104,7 +104,7 @@ func TestApplyPlaybookBoost_HighestScoreWinsForSameAnswer(t *testing.T) {
AnswerID: "a", AnswerCorpus: "x", Score: 0.9,
}},
}
ApplyPlaybookBoost(results, hits)
ApplyPlaybookBoost("", results, hits)
wantDist := 0.30 * 0.55
if abs(float64(results[0].Distance)-wantDist) > 1e-6 {
t.Errorf("strong-score boost should win: want %.4f, got %.4f", wantDist, results[0].Distance)
@@ -125,7 +125,7 @@ func TestApplyPlaybookBoost_CorpusAttributionRespected(t *testing.T) {
AnswerID: "a", AnswerCorpus: "x", Score: 1.0,
}},
}
n := ApplyPlaybookBoost(results, hits)
n := ApplyPlaybookBoost("", results, hits)
if n != 0 {
t.Errorf("cross-corpus collision should not boost: got %d", n)
}
@@ -187,7 +187,7 @@ func TestInjectPlaybookMisses_AddsMissingAnswers(t *testing.T) {
},
},
}
out, injected := InjectPlaybookMisses("test query", results, hits, 0, nil)
out, injected := InjectPlaybookMisses("test query", "", results, hits, 0, nil)
if injected != 1 {
t.Fatalf("expected 1 injected, got %d", injected)
}
@@ -240,7 +240,7 @@ func TestInjectPlaybookMisses_SkipsAnswersAlreadyPresent(t *testing.T) {
},
},
}
out, injected := InjectPlaybookMisses("test query", results, hits, 0, nil)
out, injected := InjectPlaybookMisses("test query", "", results, hits, 0, nil)
if injected != 0 {
t.Errorf("expected 0 injected (answer already present), got %d", injected)
}
@@ -266,7 +266,7 @@ func TestInjectPlaybookMisses_DedupesPerAnswer(t *testing.T) {
Entry: PlaybookEntry{QueryText: "q2", AnswerID: "w-99", AnswerCorpus: "workers", Score: 1.0},
},
}
out, injected := InjectPlaybookMisses("test query", results, hits, 0.5, nil) // explicit loose threshold so 0.30 hits qualify
out, injected := InjectPlaybookMisses("test query", "", results, hits, 0.5, nil) // explicit loose threshold so 0.30 hits qualify
if injected != 1 {
t.Errorf("expected 1 injection (deduped), got %d", injected)
}
@@ -298,7 +298,7 @@ func TestInjectPlaybookMisses_GateRejectsCandidate(t *testing.T) {
},
}
rejectAll := InjectGateFunc(func(string, PlaybookHit) bool { return false })
out, injected := InjectPlaybookMisses("forklift loader query", results, hits, 0, rejectAll)
out, injected := InjectPlaybookMisses("forklift loader query", "", results, hits, 0, rejectAll)
if injected != 0 {
t.Errorf("rejectAll gate should skip injection, got %d injected", injected)
}
@@ -323,7 +323,7 @@ func TestInjectPlaybookMisses_GateApprovesCandidate(t *testing.T) {
},
}
approveAll := InjectGateFunc(func(string, PlaybookHit) bool { return true })
out, injected := InjectPlaybookMisses("test query", results, hits, 0, approveAll)
out, injected := InjectPlaybookMisses("test query", "", results, hits, 0, approveAll)
if injected != 1 {
t.Errorf("approveAll gate should inject, got %d", injected)
}
@@ -354,7 +354,7 @@ func TestInjectPlaybookMisses_GateSeesCorrectQuery(t *testing.T) {
seenRecordedQuery = h.Entry.QueryText
return true
})
_, _ = InjectPlaybookMisses("CURRENT", results, hits, 0, gate)
_, _ = InjectPlaybookMisses("CURRENT", "", results, hits, 0, gate)
if seenQuery != "CURRENT" {
t.Errorf("gate received query=%q, want CURRENT", seenQuery)
}
@@ -386,7 +386,7 @@ func TestInjectPlaybookMisses_RespectsInjectThreshold(t *testing.T) {
},
}
// Default threshold (0 → DefaultPlaybookMaxInjectDistance = 0.20)
out, injected := InjectPlaybookMisses("test query", results, hits, 0, nil)
out, injected := InjectPlaybookMisses("test query", "", results, hits, 0, nil)
if injected != 1 {
t.Errorf("expected 1 injection (only the tight hit qualifies), got %d", injected)
}
@@ -407,7 +407,7 @@ func TestInjectPlaybookMisses_RespectsInjectThreshold(t *testing.T) {
// TestInjectPlaybookMisses_EmptyHits is a fast-path no-op check.
func TestInjectPlaybookMisses_EmptyHits(t *testing.T) {
results := []Result{{ID: "w-1", Corpus: "workers", Distance: 0.30}}
out, injected := InjectPlaybookMisses("test query", results, nil, 0, nil)
out, injected := InjectPlaybookMisses("test query", "", results, nil, 0, nil)
if injected != 0 {
t.Errorf("expected 0 injection, got %d", injected)
}
@@ -416,6 +416,146 @@ func TestInjectPlaybookMisses_EmptyHits(t *testing.T) {
}
}
// Cross-role gate tests — reproduce the real_001 bleed scenario at
// the unit-test level so regressions can't ship without notice.
//
// real_001 scenario: Q#2 records playbook entry for e-6193 fitting
// "Forklift Operator at Beacon Freight Detroit MI". Q#10 (CNC Operator
// at same client + city) embeds within 0.135 cosine of Q#2's
// playbook entry — well inside the 0.20 inject threshold. Without the
// role gate, e-6193 injects on Q#10 at warm-top-1, demoting the
// cold-pass-correct w-3759.
func TestInjectPlaybookMisses_RoleGateRejectsCrossRole(t *testing.T) {
results := []Result{{ID: "w-3759", Corpus: "workers", Distance: 0.50}}
hits := []PlaybookHit{{
PlaybookID: "pb-q2",
Distance: 0.135, // tight; well within DefaultPlaybookMaxInjectDistance
Entry: PlaybookEntry{
QueryText: "Need 1 Forklift Operator in Detroit MI ...",
Role: "Forklift Operator",
AnswerID: "e-6193",
AnswerCorpus: "ethereal_workers",
Score: 1.0,
},
}}
out, injected := InjectPlaybookMisses(
"Need 1 CNC Operator in Detroit MI ...",
"CNC Operator", // queryRole differs
results, hits, 0, nil,
)
if injected != 0 {
t.Errorf("expected 0 cross-role injections, got %d", injected)
}
if len(out) != 1 {
t.Errorf("results should be unchanged when gate rejects, got len=%d", len(out))
}
}
func TestInjectPlaybookMisses_RoleGateAllowsSameRole(t *testing.T) {
results := []Result{{ID: "e-5617", Corpus: "ethereal_workers", Distance: 0.50}}
hits := []PlaybookHit{{
PlaybookID: "pb-q2",
Distance: 0.135,
Entry: PlaybookEntry{
QueryText: "Need 1 Forklift Operator in Detroit MI ...",
Role: "Forklift Operator",
AnswerID: "e-6193",
AnswerCorpus: "ethereal_workers",
Score: 1.0,
},
}}
out, injected := InjectPlaybookMisses(
"Need 2 Forklift Operators in Detroit MI ...",
"Forklift Operators", // plural — roleEqual normalizes to match
results, hits, 0, nil,
)
if injected != 1 {
t.Errorf("expected 1 same-role injection, got %d", injected)
}
if len(out) != 2 {
t.Errorf("results should be extended with the injection, got len=%d", len(out))
}
}
func TestInjectPlaybookMisses_RoleGateBackwardCompat(t *testing.T) {
// Empty role on either side disables the gate — preserves
// behavior for callers (lift suite, etc.) that don't supply role.
results := []Result{{ID: "w-3759", Corpus: "workers", Distance: 0.50}}
hits := []PlaybookHit{{
PlaybookID: "pb-q2",
Distance: 0.135,
Entry: PlaybookEntry{
QueryText: "...",
Role: "Forklift Operator", // entry has role
AnswerID: "e-6193",
AnswerCorpus: "ethereal_workers",
Score: 1.0,
},
}}
// Caller didn't pass queryRole — gate must NOT fire.
_, injected := InjectPlaybookMisses("...", "", results, hits, 0, nil)
if injected != 1 {
t.Errorf("expected gate disabled when queryRole empty (injected=1), got %d", injected)
}
// Inverse: queryRole set but entry has no role.
hits[0].Entry.Role = ""
_, injected = InjectPlaybookMisses("...", "Forklift Operator", results, hits, 0, nil)
if injected != 1 {
t.Errorf("expected gate disabled when entry.Role empty (injected=1), got %d", injected)
}
}
func TestApplyPlaybookBoost_RoleGateRejectsCrossRole(t *testing.T) {
results := []Result{
{ID: "w-3759", Corpus: "workers", Distance: 0.50},
{ID: "e-6193", Corpus: "ethereal_workers", Distance: 0.55},
}
hits := []PlaybookHit{{
PlaybookID: "pb-q2",
Distance: 0.18,
Entry: PlaybookEntry{
QueryText: "Forklift Operator query",
Role: "Forklift Operator",
AnswerID: "e-6193",
AnswerCorpus: "ethereal_workers",
Score: 1.0,
},
}}
// Different-role query → boost must NOT fire on e-6193 even though
// it's in the result set.
n := ApplyPlaybookBoost("CNC Operator", results, hits)
if n != 0 {
t.Errorf("expected 0 cross-role boosts, got %d", n)
}
if results[1].Distance != 0.55 {
t.Errorf("e-6193 distance should be unchanged (0.55), got %v", results[1].Distance)
}
}
func TestRoleEqual_PluralAndCase(t *testing.T) {
cases := []struct {
a, b string
want bool
}{
{"Forklift Operator", "forklift operator", true},
{"Forklift Operator", "Forklift Operators", true}, // plural
{"CNC Operator", "Forklift Operator", false},
{"Pickers", "Picker", true},
{"", "Forklift Operator", true}, // gate disabled
{"CNC Operator", "", true}, // gate disabled (other side empty)
{"", "", true}, // both empty = both disabled
{"Boxes", "Box", true}, // -es plural
}
for _, c := range cases {
got := roleEqual(c.a, c.b)
if got != c.want {
t.Errorf("roleEqual(%q, %q) = %v, want %v", c.a, c.b, got, c.want)
}
}
}
func abs(f float64) float64 {
if f < 0 {
return -f
internal/matrix/retrieve.go
+10 -2
View File
@@ -92,6 +92,14 @@ type SearchRequest struct {
JudgeURL string `json:"judge_url,omitempty"`
JudgeModel string `json:"judge_model,omitempty"`
JudgeMinRating int `json:"judge_min_rating,omitempty"`
// QueryRole is an optional role hint that gates Shape A boost +
// Shape B inject on role match (real_001 cross-role bleed fix).
// When set, a recorded playbook entry only fires for queries with
// the same role under roleEqual's normalization. Empty here OR
// empty on the recorded entry = gate disabled (preserves behavior
// for callers that don't supply role — e.g. lift-suite multi-
// constraint queries that have no clean single role).
QueryRole string `json:"query_role,omitempty"`
MetadataFilter map[string]any `json:"metadata_filter,omitempty"`
// ExcludeIDs filters out specific worker IDs post-retrieval.
// Real-world driver: a coordinator places 200 workers at a
@@ -292,7 +300,7 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
}
hits = keptHits
}
resp.PlaybookBoosted = ApplyPlaybookBoost(resp.Results, hits)
resp.PlaybookBoosted = ApplyPlaybookBoost(req.QueryRole, resp.Results, hits)
maxInjectDist := float32(req.PlaybookMaxInjectDistance)
if maxInjectDist <= 0 {
maxInjectDist = float32(DefaultPlaybookMaxInjectDistance)
@@ -304,7 +312,7 @@ func (r *Retriever) Search(ctx context.Context, req SearchRequest) (*SearchRespo
gate = g
}
var injected int
resp.Results, injected = InjectPlaybookMisses(req.QueryText, resp.Results, hits, maxInjectDist, gate)
resp.Results, injected = InjectPlaybookMisses(req.QueryText, req.QueryRole, resp.Results, hits, maxInjectDist, gate)
resp.PlaybookInjected = injected
if injected > 0 {
// Re-sort + truncate after injection. ApplyPlaybookBoost