real_006: distribution-shift test on rows 10-59 of fill_events
Methodology fix: gen_real_queries.go gains -offset N flag. Every prior real_NNN test sourced queries from rows 0-9 of fill_events.parquet (default -limit 10), so the substrate's published "8/10 cold-pass top-1 = judge-best" was measured on a memorized slice, not held-out data. real_006 samples 50 fresh rows (offset 10, never seen by the workers or ethereal_workers corpora). Same harness, same local qwen2.5:latest judge, same K=10. ~14 min wall total. Local-only, no cloud calls. Headline findings: - Cold-pass top-1 = judge-best (rank match): 41/50 (82%) vs real_001's 8/10 (80%) — substrate generalizes at rank level. - Strict (rating ≥ 2): 34/50 (68%) — 12-point drop from real_001's 80%. ~7 of 41 "no-discovery" queries had cold top-1 the judge rated 1; the corpus has gaps for some role-city combos in the v3 slice. - Verbatim lift: 9/9 discoveries → warm top-1 (clean, matches real_001 2/2) - Paraphrase recovery: 6/9 → top-1, 9/9 any-rank - Quality regressed: 3/50 — Q43 is the structural one Q43 (Packer at Midway Distribution / Chicago IL) regressed from rating 5 to rating 2 on warm pass with `warm_boosted_count=0` and `playbook_recorded=false`. Q18 (Shipping Clerks at the same client+city) recorded a playbook entry. The regression suggests Q18's recording leaked into Q43 via the warm-pass playbook corpus retrieval surface even though the role gate from real_002 should have blocked it. Three possible paths: extractor failed on one query, gate fires on boost path but not Shape B inject, or cosine drift puts the recorded worker close enough to Q43's embedding that warm-pass retrieval picks it up directly. Diagnosis is the next move. Three same-(client, city) clusters tested: - Heritage Foods Gary IN × 3 distinct roles: clean, distinct workers - Riverfront Steel Columbus OH × 4: cosine-level confusion (Q9/Q25 surface same worker w-281 for Assemblers vs Quality Techs at cold- pass), but no playbook bleed - Midway Distribution Chicago IL × 3: Q43 regression as above What this confirms: substrate works on the fresh distribution at the rank level, verbatim lift is real, paraphrase recovery is real. What this falsifies: real_002's role-gate fix is not structurally airtight. The bleed pattern can still fire under conditions the prior tests didn't reach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,7 @@ import (
|
||||
func main() {
|
||||
src := flag.String("src", "/home/profit/lakehouse/data/datasets/fill_events.parquet", "fill_events parquet path")
|
||||
limit := flag.Int("limit", 10, "number of source rows to read")
|
||||
offset := flag.Int("offset", 0, "skip the first N rows (lets reality tests sample beyond the memorized real_001 slice)")
|
||||
styles := flag.String("styles", "need", "comma-separated styles to emit per row (need|client_first|looking|shorthand|all)")
|
||||
flag.Parse()
|
||||
|
||||
@@ -58,16 +59,24 @@ func main() {
|
||||
at := tbl.Column(10).Data().Chunk(0)
|
||||
deadline := tbl.Column(12).Data().Chunk(0)
|
||||
|
||||
n := int(tbl.NumRows())
|
||||
if *limit < n {
|
||||
n = *limit
|
||||
totalRows := int(tbl.NumRows())
|
||||
start := *offset
|
||||
if start < 0 {
|
||||
start = 0
|
||||
}
|
||||
if start > totalRows {
|
||||
start = totalRows
|
||||
}
|
||||
end := start + *limit
|
||||
if end > totalRows {
|
||||
end = totalRows
|
||||
}
|
||||
|
||||
stylesList := parseStyles(*styles)
|
||||
|
||||
fmt.Println("# Real-shape coordinator queries — generated from fill_events.parquet")
|
||||
fmt.Println("# (real-shape demand data; queries built mechanically from event rows).")
|
||||
fmt.Printf("# Source: %s (%d rows total, %d emitted, styles=%v)\n", *src, tbl.NumRows(), n, stylesList)
|
||||
fmt.Printf("# Source: %s (%d rows total, rows [%d,%d) emitted, styles=%v)\n", *src, totalRows, start, end, stylesList)
|
||||
fmt.Println("#")
|
||||
fmt.Println("# Styles:")
|
||||
fmt.Println("# need: 'Need N {role}{s} in {city} {state} starting at {at} for {client}'")
|
||||
@@ -80,7 +89,7 @@ func main() {
|
||||
fmt.Println("# substrate's bleed behavior when the role gate is silently disabled.")
|
||||
fmt.Println()
|
||||
|
||||
for i := 0; i < n; i++ {
|
||||
for i := start; i < end; i++ {
|
||||
ev := event{
|
||||
client: client.ValueStr(i),
|
||||
city: city.ValueStr(i),
|
||||
|
||||
Reference in New Issue
Block a user