profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

playbook_lift: LLM-based role extractor closes shorthand bleed (real_004)

Browse Source 
real_003 left a known-weak hole: shorthand-style queries
("{count} {role} {city} {state} ...") have no separator between
role and city, so a regex can't reliably extract — leaving the
cross-role gate disabled when both record AND query are shorthand.

This commit adds a roleExtractor with regex-first + LLM fallback:

- Regex first (fast, deterministic) — handles need + client_first +
  looking from real_003b. ~75% of styles, no LLM cost paid.
- LLM fallback when regex returns empty AND model is configured —
  Ollama-shape /api/chat with format=json, schema-tight prompt,
  temperature 0. ~1-3s on local qwen2.5.
- Per-process cache — paraphrase + rejudge passes reuse the same
  query 4× per run; cache prevents 4× LLM cost.
- Off-by-default — opt-in via -llm-role-extract flag (CLI) and
  LLM_ROLE_EXTRACT=1 env var (harness wrapper). real_003b shipping
  config unchanged unless explicitly enabled.

8 new tests in scripts/playbook_lift/main_test.go:
- TestRoleExtractor_RegexFirst: LLM not called when regex matches
- TestRoleExtractor_LLMFallback: shorthand goes to LLM
- TestRoleExtractor_LLMOffLeavesEmpty: opt-in default preserved
- TestRoleExtractor_Cache: 3 calls = 1 LLM hit
- TestRoleExtractor_NilSafe: nil receiver runs regex only
- TestExtractRoleViaLLM_HTTPError + _BadJSON: failure paths
- TestRoleExtractor_ClosesCrossRoleShorthandBleed: synthetic
  witness for the real_003 scenario — both record + query are
  shorthand, regex returns "" for both, LLM produces DIFFERENT
  role tokens for CNC vs Forklift, so matrix gate's cross-role
  rejection (locked separately in
  TestInjectPlaybookMisses_RoleGateRejectsCrossRole) fires
  correctly. This is the load-bearing verification.

Reality test real_004 ran the same 40-query stress as real_003 with
LLM extraction on. Cross-style same-role boosts fired correctly
across all 4 styles for Loaders + Packers + Shipping Clerk clusters
(including shorthand → other-style transfer). No cross-role bleed
observed. The reality test alone can't be a clean "with vs without"
comparison (HNSW build is non-deterministic across runs, and
real_004 stochastics didn't trigger a shorthand recording at all),
which is why the unit-test witness exists.

Production note (in real_004_findings.md): LLM extraction is for
reality-test coverage of arbitrary query shapes. Production should
extract role at INGEST time (when the inbox parser already runs an
LLM) and pass already-resolved role through requests — same shape
as multi_coord_stress's existing Demand{Role: ...} model. The hot
path should never need the harness extractor's per-query LLM cost.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 22:51:27 -05:00
co-authored by Claude Opus 4.7
parent 3263254f1c
commit 0331288641
6 changed files with 572 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reports/reality-tests/playbook_lift_real_004.md
+116
View File
@@ -0,0 +1,116 @@
# Playbook-Lift Reality Test — Run real_004
**Generated:** 2026-05-01T03:48:39.932829485Z
**Judge:** `qwen2.5:latest` (Ollama, resolved from config [models].local_judge)
**Corpora:** `workers,ethereal_workers`
**Workers limit:** 5000
**Queries:** `tests/reality/real_coord_queries_v2.txt` (40 executed)
**K per pass:** 10
**Paraphrase pass:** disabled
**Re-judge pass:** disabled
**Evidence:** `reports/reality-tests/playbook_lift_real_004.json`
---
## Headline
| Metric | Value |
|---|---:|
| Total queries run | 40 |
| Cold-pass discoveries (judge-best ≠ top-1) | 5 |
| Warm-pass lifts (recorded playbook → top-1) | 4 |
| No change (judge-best already top-1, no playbook needed) | 36 |
| Playbook boosts triggered (warm pass) | 16 |
| Mean Δ top-1 distance (warm − cold) | -0.11025716 |
**Verbatim lift rate:** 4 of 5 discoveries became top-1 after warm pass.
---
## Per-query results
| # | Query | Cold top-1 | Cold judge-best (rank/rating) | Recorded? | Warm top-1 | Judge-best warm rank | Lift |
|---|---|---|---|---|---|---|---|
| 1 | Need 5 Warehouse Associates in Kansas City MO starting at 09 | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 2 | Parallel Machining needs 5 Warehouse Associates in Kansas Ci | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 3 | Looking for 5 Warehouse Associates at Parallel Machining in | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 4 | 5 Warehouse Associates Kansas City MO 09:00 Parallel Machini | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 5 | Need 1 Forklift Operator in Detroit MI starting at 15:00 for | w-2136 | 0/5 | — | w-2136 | 0 | no |
| 6 | Beacon Freight needs 1 Forklift Operator in Detroit MI at 15 | e-6723 | 2/2 | — | e-6723 | 2 | no |
| 7 | Looking for 1 Forklift Operator at Beacon Freight in Detroit | w-4766 | 0/5 | — | w-4766 | 0 | no |
| 8 | 1 Forklift Operator Detroit MI 15:00 Beacon Freight | e-5818 | 0/4 | — | e-5818 | 0 | no |
| 9 | Need 4 Loaders in Indianapolis IN starting at 12:00 for Midw | e-9877 | 0/4 | — | w-398 | 2 | no |
| 10 | Midway Distribution needs 4 Loaders in Indianapolis IN at 12 | e-4537 | 4/5 | ✓ w-2148 | w-398 | 1 | no |
| 11 | Looking for 4 Loaders at Midway Distribution in Indianapolis | w-962 | 6/4 | ✓ w-398 | w-398 | 0 | **YES** |
| 12 | 4 Loaders Indianapolis IN 12:00 Midway Distribution | e-9877 | 0/4 | — | w-2148 | 2 | no |
| 13 | Need 3 Warehouse Associates in Fort Wayne IN starting at 17: | w-2857 | 1/4 | ✓ w-1063 | w-1063 | 0 | **YES** |
| 14 | Cornerstone Fabrication needs 3 Warehouse Associates in Fort | w-4703 | 0/2 | — | w-1063 | 1 | no |
| 15 | Looking for 3 Warehouse Associates at Cornerstone Fabricatio | w-1784 | 0/3 | — | w-1063 | 1 | no |
| 16 | 3 Warehouse Associates Fort Wayne IN 17:30 Cornerstone Fabri | w-2754 | 2/3 | — | w-1063 | 3 | no |
| 17 | Need 4 Pickers in Detroit MI starting at 13:30 for Beacon Fr | e-2247 | 1/2 | — | e-2247 | 1 | no |
| 18 | Beacon Freight needs 4 Pickers in Detroit MI at 13:30 | e-2247 | 2/2 | — | e-2247 | 2 | no |
| 19 | Looking for 4 Pickers at Beacon Freight in Detroit MI for 13 | e-2247 | 1/2 | — | e-2247 | 1 | no |
| 20 | 4 Pickers Detroit MI 13:30 Beacon Freight | e-2247 | 0/3 | — | e-2247 | 0 | no |
| 21 | Need 2 Packers in Joliet IL starting at 09:30 for Parallel M | e-846 | 2/3 | — | e-2120 | 0 | no |
| 22 | Parallel Machining needs 2 Packers in Joliet IL at 09:30 | e-846 | 2/4 | ✓ e-2120 | e-2120 | 0 | **YES** |
| 23 | Looking for 2 Packers at Parallel Machining in Joliet IL for | e-846 | 1/2 | — | e-2120 | 2 | no |
| 24 | 2 Packers Joliet IL 09:30 Parallel Machining | e-846 | 2/3 | — | e-2120 | 0 | no |
| 25 | Need 3 Assemblers in Flint MI starting at 08:30 for Heritage | w-4124 | 9/3 | — | w-4124 | 9 | no |
| 26 | Heritage Foods needs 3 Assemblers in Flint MI at 08:30 | w-4124 | 3/3 | — | w-4124 | 3 | no |
| 27 | Looking for 3 Assemblers at Heritage Foods in Flint MI for 0 | w-4124 | 2/2 | — | w-4124 | 2 | no |
| 28 | 3 Assemblers Flint MI 08:30 Heritage Foods | w-4124 | 3/3 | — | w-4124 | 3 | no |
| 29 | Need 3 Packers in Flint MI starting at 12:30 for Parallel Ma | e-6019 | 8/2 | — | e-6019 | 8 | no |
| 30 | Parallel Machining needs 3 Packers in Flint MI at 12:30 | e-6019 | 9/2 | — | e-6019 | 9 | no |
| 31 | Looking for 3 Packers at Parallel Machining in Flint MI for | e-6019 | 0/1 | — | e-6019 | 0 | no |
| 32 | 3 Packers Flint MI 12:30 Parallel Machining | e-2006 | 0/3 | — | e-2006 | 0 | no |
| 33 | Need 1 Shipping Clerk in Flint MI starting at 17:00 for Pion | w-3988 | 2/4 | ✓ w-1367 | w-1367 | 0 | **YES** |
| 34 | Pioneer Assembly needs 1 Shipping Clerk in Flint MI at 17:00 | w-3988 | 1/3 | — | w-1367 | 2 | no |
| 35 | Looking for 1 Shipping Clerk at Pioneer Assembly in Flint MI | w-3988 | 0/2 | — | w-1367 | 1 | no |
| 36 | 1 Shipping Clerk Flint MI 17:00 Pioneer Assembly | e-4849 | 0/2 | — | w-1367 | 1 | no |
| 37 | Need 1 CNC Operator in Detroit MI starting at 17:30 for Beac | e-489 | 5/3 | — | e-489 | 5 | no |
| 38 | Beacon Freight needs 1 CNC Operator in Detroit MI at 17:30 | e-489 | 0/2 | — | e-489 | 0 | no |
| 39 | Looking for 1 CNC Operator at Beacon Freight in Detroit MI f | e-489 | 0/2 | — | e-489 | 0 | no |
| 40 | 1 CNC Operator Detroit MI 17:30 Beacon Freight | e-489 | 3/3 | — | e-489 | 3 | no |
---
## Honesty caveats
1. **Judge IS the ground truth proxy.** Without human-labeled relevance, the LLM
judge's verdict is what defines "best." If `` rates badly,
the lift number is meaningless. To validate the judge itself, sample 5–10
verdicts manually and check agreement.
2. **Score-1.0 boost = distance halved.** Playbook math is
`distance' = distance × (1 - 0.5 × score)`. Lift requires the judge-best
result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise
even halving doesn't promote it. Tight clusters → little visible lift.
3. **Verbatim vs paraphrase.** The verbatim lift rate (above) is the cheap
case — same query, recorded playbook, expected boost. The paraphrase
pass (when enabled) is the actual learning property: similar-but-different
queries hitting a recorded playbook. Compare verbatim and paraphrase
lift rates — paraphrase should be lower (semantic-distance gates some
playbook hits) but non-zero is the meaningful signal.
4. **Multi-corpus skew.** Default corpora=`workers,ethereal_workers` — if all judge-best
results land in one corpus, the matrix layer's purpose isn't being tested.
Check per-corpus distribution in the JSON.
5. **Judge resolution.** This run used `qwen2.5:latest` from
config [models].local_judge.
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
6. **Paraphrase generation also uses the judge.** The same model that rates
relevance also rephrases queries. A judge that's bad at rating staffing
queries is probably also bad at rephrasing them. Worth sanity-checking
a sample of `paraphrase_query` values in the JSON before trusting the
paraphrase lift number.
## Next moves
- If lift rate ≥ 50% of discoveries: matrix layer + playbook is doing real
work. Move to paraphrase queries + tag-based boost (currently ignored).
- If lift rate < 20%: investigate why — judge variance, distance gap too
wide, or playbook math too gentle. The score=1.0 / 0.5× formula may need
retuning.
- If discovery rate (cold judge-best ≠ top-1) is itself low: cosine is
already close to optimal on this query distribution. Either the corpus
is too narrow or the queries are too easy.
reports/reality-tests/real_004_findings.md
+104
View File
@@ -0,0 +1,104 @@
# Reality test real_004 — LLM-based role extractor closes shorthand hole
real_003 / real_003b had a documented limitation: the regex extractor
can't separate role from city in shorthand-style queries
(`{count} {role} {city} {state} ...`) because there's no anchor
between role and city. real_004 closes this with an LLM fallback —
qwen2.5 format=json, called only when the regex returns empty.
## Architecture
`roleExtractor` struct in `scripts/playbook_lift/main.go`:
1. **Regex first** (fast, deterministic) — handles need + client_first
+ looking patterns from real_003b.
2. **LLM fallback** when regex returns empty AND model is configured —
Ollama-shape `/api/chat` with format=json, schema-tight system
prompt, temperature 0.
3. **Per-process cache** — paraphrase + rejudge passes hit the same
query 4× per run; cache prevents 4× LLM cost per query.
4. **Off-by-default** — `-llm-role-extract` flag (CLI) +
`LLM_ROLE_EXTRACT=1` env var (harness) opt in. Default behavior
is unchanged from real_003b.
## Verification — what's proven
**Unit tests (8 new in `scripts/playbook_lift/main_test.go`):**
- `TestRoleExtractor_RegexFirst` — when regex matches, LLM is NOT
called (cost discipline preserved on the 75% of queries the regex
handles).
- `TestRoleExtractor_LLMFallback` — shorthand query goes to LLM,
result is used.
- `TestRoleExtractor_LLMOffLeavesEmpty` — without model configured,
shorthand returns empty (current default).
- `TestRoleExtractor_Cache` — 3 calls to same query = 1 LLM hit.
- `TestRoleExtractor_NilSafe` — nil receiver runs regex only;
matrixSearch + playbookRecord don't need a guard.
- `TestExtractRoleViaLLM_HTTPError` + `_BadJSON` — failure paths
surface error so caller can fall back cleanly.
- `TestRoleExtractor_ClosesCrossRoleShorthandBleed` — the synthetic
witness for the real_003 scenario: when both record AND query are
shorthand (regex returns "" for both), LLM produces DIFFERENT role
tokens for CNC Operator vs Forklift Operator queries → matrix
gate's cross-role rejection fires correctly. This is the load-bearing
verification — paired with `internal/matrix/playbook_test.go`'s
`TestInjectPlaybookMisses_RoleGateRejectsCrossRole` (which uses
the exact role tokens this test produces).
## Verification — what real_004 (live harness) shows
Same 40-query stress file as real_003. With `LLM_ROLE_EXTRACT=1`:
| Run | recordings | shorthand recordings | boosts | Detroit cluster bleed |
|---|---:|---:|---:|---|
| real_003 (regex only) | 7 | 1 (CNC) | 14 | **YES — w-2404 leaked** |
| real_003b (extended regex) | 11 | 1 (Pickers) | 31 | none observed |
| real_004 (regex + LLM) | 5 | 0 | 16 | none observed |
real_004's shorthand-recordings = 0 is stochastic (HNSW build is
non-deterministic across runs, so cold-pass top-1 varies and so does
which queries trigger discovery). The LLM is doing work — boosts
fired across all 4 styles for the Loaders + Packers + Shipping Clerk
clusters, including shorthand queries. But the dataset didn't
naturally produce a shorthand recording in this run, so we can't
read a clean "with-vs-without" reality-test signal on the bleed.
**This is why the unit-test witness exists.** The reality test confirms
no regressions; the unit test proves the extraction-layer correctness
on the exact failure mode real_003 surfaced. Together they cover the
fix.
## Cost note
LLM extraction adds ~1-3s per shorthand query on local qwen2.5. Per
real_004 timing the harness took ~10 minutes for 40 queries with LLM
on (vs ~2 min in real_003b). The cache makes paraphrase + rejudge
passes free; first-touch shorthand queries pay the LLM cost once.
For production hot paths (inbox parsing, retrieve), the LLM cost is
prohibitive. The right architecture there is to extract role at
INGEST time (when queries land in the inbox parser) and pass the
already-resolved role through the request — same pattern as
multi_coord_stress's existing `Demand{Role: ...}` shape. The harness
LLM extractor is for reality-test coverage of arbitrary query shapes;
production should never need it.
## Repro
```bash
# Generate the same 40-query stress file
go run scripts/cutover/gen_real_queries.go -limit 10 -styles all > tests/reality/real_coord_queries_v2.txt
# With LLM extractor (closes shorthand hole)
QUERIES_FILE=tests/reality/real_coord_queries_v2.txt RUN_ID=real_004 \
WITH_PARAPHRASE=0 WITH_REJUDGE=0 LLM_ROLE_EXTRACT=1 \
./scripts/playbook_lift.sh
# Without LLM extractor (regex-only — current default)
QUERIES_FILE=tests/reality/real_coord_queries_v2.txt RUN_ID=real_004_regex_only \
WITH_PARAPHRASE=0 WITH_REJUDGE=0 \
./scripts/playbook_lift.sh
```
Evidence: `reports/reality-tests/playbook_lift_real_004.{json,md}`.