reality_test real_003: 40-query paraphrase stress + extractor extension

Stress-tests the role gate with 40 queries (10 fill_events rows × 4
styles): need, client_first, looking, shorthand. Each row's role +
client + city stays the same; only the surface phrasing changes.

real_003 (original extractor) confirmed the shorthand-vs-shorthand
failure mode: CNC Operator shorthand recording leaked w-2404 onto
Forklift Operator shorthand query within the same Beacon Freight
Detroit cluster. Both record + query had empty role (extractor
returns "" for shorthand because there's no separator between role
and city), gate disabled, distance check passed, bleed fired.

Fix: extended extractRoleFromNeed to handle client_first
("{client} needs N {role} in...") and looking ("Looking for N
{role} at...") patterns. Shorthand left intentionally unmatched —
"Forklift Operator Detroit" is shape-indistinguishable from
"Forklift" + "Operator Detroit" without an LLM extractor or known-
cities lookup.

real_003b (extended extractor) verifies bleed closed across all 4
styles for this dataset. Forklift Operator queries keep w-2136 (the
cold-pass-correct match) regardless of which style the query came
in. Same-role boosts now fire correctly across styles — a CNC
Operator recording made in `looking` style boosts the CNC need-form
query.

scripts/cutover/gen_real_queries.go: added -styles flag with values
need|client_first|looking|shorthand|all (default need preserves
real_001/002 behavior). Tests/reality/real_coord_queries_v2.txt is
the 40-query stress file.

scripts/playbook_lift/main_test.go: 10 sub-tests lock the four
documented patterns + shorthand limitation + lift-suite-style
queries (no clean role, returns empty as expected).

Aggregate metrics:
- real_003  (original): disc=7,  lift=7,  boost=14, meanΔ=-0.108
- real_003b (extended): disc=11, lift=10, boost=31, meanΔ=-0.202
The growth reflects more LEGITIMATE same-role same-cluster transfer
firing across styles, not bleed (verified by per-cluster bleed
table — Forklift Operator queries unchanged across all 4 styles).

Known limitation documented in real_003_findings.md: same-cluster,
same-role queries in shorthand still embed close enough that a
shorthand recording could bleed onto a different-role shorthand
query if both record + query strip role. Closing this requires
LLM extraction or known-cities lookup at record + query time.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 21:42:02 -05:00
co-authored by Claude Opus 4.7
parent 997527be4d
commit 3263254f1c
8 changed files with 616 additions and 34 deletions
+42 -14
View File
@@ -631,25 +631,53 @@ func appendNote(existing, add string) string {
// refactor; harmless for now.
var _ = sort.Slice
// extractRoleFromNeed pulls the role out of "Need N {role}{s} in {city}"
// shape queries — the fill_events-derived form used by real_NNN runs.
// Returns "" for any query that doesn't match (free-form lift-suite
// queries fall back to empty, leaving the cross-role gate disabled).
// extractRoleFromNeed pulls the role out of staffing-shape queries.
// Returns "" for any query that doesn't match a known anchor pattern
// (free-form lift-suite queries + shorthand-style fall back to empty,
// leaving the cross-role gate disabled).
//
// The pattern is permissive: the count can be any digits, and the
// role is everything between the count and " in ". This catches
// "Need 5 Warehouse Associates in Kansas City" → "Warehouse Associates";
// roleEqual on the matrix side handles plurals + case.
// Patterns covered (in priority order):
// need: "Need N {role}{s} in {city} ..."
// client_first: "{client} needs N {role}{s} in {city} ..."
// looking: "Looking for N {role}{s} at {client} in {city} ..."
//
// Pattern explicitly NOT covered:
// shorthand: "N {role}{s} {city} {state} {at} {client}"
// Because there's no separator between role and city in shorthand
// ("Forklift Operator Detroit" is shape-indistinguishable from
// "Forklift" + "Operator Detroit"), a regex can't reliably extract
// role here. real_003 confirmed shorthand-vs-shorthand cross-role
// bleed: a CNC Operator shorthand recording leaked w-2404 onto a
// Forklift Operator shorthand query within the same Beacon Freight
// Detroit cluster. Closing that requires either an LLM extractor at
// record+query time or a known-cities lookup table.
//
// Lives here (not in internal/matrix) because role extraction from
// free-form text is a caller concern; matrix only consumes the
// already-resolved Role string. A future LLM-based extractor would
// replace this regex without changing matrix's gate logic.
// replace this function without changing matrix's gate logic.
func extractRoleFromNeed(query string) string {
re := regexp.MustCompile(`(?i)^Need\s+\d+\s+(.+?)\s+in\s+`)
m := re.FindStringSubmatch(query)
if len(m) < 2 {
return ""
for _, re := range roleExtractRegexes {
if m := re.FindStringSubmatch(query); len(m) >= 2 {
return strings.TrimSpace(m[1])
}
}
return strings.TrimSpace(m[1])
return ""
}
// roleExtractRegexes is ordered: more-specific anchors first so a
// "Looking for ..." query doesn't accidentally land in the "Need"
// pattern (impossible given the prefix, but guards against future
// pattern additions). Compiled once at package init via MustCompile.
var roleExtractRegexes = []*regexp.Regexp{
// "Need N {role} in ..." — the original real_001 form.
regexp.MustCompile(`(?i)^Need\s+\d+\s+(.+?)\s+in\s+`),
// "Looking for N {role} at ..." — the looking style. Anchor on
// "at" because the role is followed by client (preceded by "at"),
// not by city directly.
regexp.MustCompile(`(?i)^Looking\s+for\s+\d+\s+(.+?)\s+at\s+`),
// "{client} needs N {role} in ..." — the client_first style.
// Greedy on the client side via .+?, then "needs", then count,
// then role, then "in".
regexp.MustCompile(`(?i)^.+?\s+needs\s+\d+\s+(.+?)\s+in\s+`),
}