profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

playbook_lift: harness expansion + reality test #001 (7/8 lift, 87.5%)

Browse Source 
The 5-loop substrate's load-bearing gate is verified — playbook +
matrix indexer give the results we're looking for. Per the report's
rubric, lift ≥ 50% of discoveries means matrix is doing real work;
7/8 = 87.5% blew through that.

Harness was structurally hiding bugs behind a 5-daemon stripped boot.
Expanding to the full 10-daemon prod stack surfaced 7 fixes in cascade:

1. driver→matrixd: {"query": ...} → {"query_text": ...} field name
2. harness temp toml missing [s3] → wrong default bucket → catalogd
   rehydrate 500 on first call
3. harness→queryd SQL probe: {"q": ...} → {"sql": ...} field name
4. expand boot from 5 → 10 daemons in dep-ordered launch
5. add SQL surface probe (3-row CSV ingest → COUNT(*)=3 assertion)
6. candidates corpus was synthetic SWE-tech (Swift/iOS, Scala/Spark) —
   wrong domain for staffing queries; replaced with ethereal_workers
   (10K rows, real staffing schema, "e-" id prefix to avoid collision
   with workers' "w-"). staffing_workers driver gains -index-name +
   -id-prefix flags so the same binary serves both corpora
7. local_judge qwen3.5:latest is a vision-SSM 256K-ctx build running
   ~30s per judge call against the lift loop; reverted to
   qwen2.5:latest (~1s/call, 30× faster, held lift theory)

Each contract drift (1, 3) is now locked into a cmd/<bin>/main_test.go
so future drift fires in `go test`, not in a reality run. R-005 closed:

- cmd/matrixd/main_test.go (new) — playbook record drift detector +
  score bounds + 6 routes mounted
- cmd/queryd/main_test.go — wrong-field-name drift detector
- cmd/pathwayd/main_test.go (new) — 9 routes + add round-trip + retire
- cmd/observerd/main_test.go (new) — 4 routes + invalid-op + unknown-mode

`go test ./cmd/{matrixd,queryd,pathwayd,observerd}` all green.

Reality test results (reports/reality-tests/playbook_lift_001.{json,md}):
  Queries              21 (staffing-domain, 7 categories)
  Discoveries          8 (judge ≠ cosine top-1)
  Lifts                7/8 (87.5%)
  Boosts triggered     9
  Mean Δ distance      -0.053 (warm closer than cold)
  OOD honesty          dental/RN/SWE rated 1, no fake matches
  Cross-corpus boosts  confirmed (e- ↔ w- swaps in lifts)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 06:22:21 -05:00
co-authored by Claude Opus 4.7
parent 740eb0d00c
commit b2e45f7f26
11 changed files with 699 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/playbook_lift.sh
+125 -16
View File
@@ -4,11 +4,20 @@
# raw cosine on staffing queries.
#
# Pipeline:
# 1. Boot the Go stack (storaged, embedd, vectord, matrixd, gateway)
# 2. Ingest workers (default 5000) + candidates corpora
# 3. Run the playbook_lift driver: cold pass → judge → record →
# 1. Boot the full Go HTTP stack (storaged, catalogd, ingestd, queryd,
# embedd, vectord, pathwayd, observerd, matrixd, gateway). Earlier
# versions booted only the 5 daemons matrix.search needs, which
# gave a falsely clean "everything works" signal — we now exercise
# the prod-realistic daemon graph so daemons that observe (observerd)
# or persist (pathwayd) are actually in the loop.
# 2. SQL surface probe — ingest a 3-row CSV via /v1/ingest (catalogd
# → ingestd → queryd refresh), assert SELECT COUNT(*)=3. Proves the
# ingestd→catalogd→queryd path is wired even though the lift driver
# itself is vector-only retrieval.
# 3. Ingest workers (default 5000) + candidates corpora into vectord
# 4. Run the playbook_lift driver: cold pass → judge → record →
# warm pass → measure
# 4. Generate markdown report from the JSON evidence
# 5. Generate markdown report from the JSON evidence
#
# Output:
# reports/reality-tests/playbook_lift_<N>.json — raw evidence
@@ -34,7 +43,7 @@ RUN_ID="${RUN_ID:-001}"
JUDGE_MODEL="${JUDGE_MODEL:-}"
WORKERS_LIMIT="${WORKERS_LIMIT:-5000}"
QUERIES_FILE="${QUERIES_FILE:-tests/reality/playbook_lift_queries.txt}"
CORPORA="${CORPORA:-workers,candidates}"
CORPORA="${CORPORA:-workers,ethereal_workers}"
K="${K:-10}"
CONFIG_PATH="${CONFIG_PATH:-lakehouse.toml}"
@@ -62,11 +71,15 @@ fi
echo "[lift] judge resolved to: $EFFECTIVE_JUDGE (from ${JUDGE_MODEL:+env}${JUDGE_MODEL:-config})"
echo "[lift] building binaries..."
go build -o bin/ ./cmd/storaged ./cmd/embedd ./cmd/vectord ./cmd/matrixd ./cmd/gateway \
go build -o bin/ ./cmd/storaged ./cmd/catalogd ./cmd/ingestd ./cmd/queryd \
./cmd/embedd ./cmd/vectord ./cmd/pathwayd ./cmd/observerd \
./cmd/matrixd ./cmd/gateway \
./scripts/staffing_workers ./scripts/staffing_candidates \
./scripts/playbook_lift
pkill -f "bin/(storaged|embedd|vectord|matrixd|gateway)" 2>/dev/null || true
# Anchor pkill to bin/<name>$ so we don't accidentally hit unrelated
# binaries — and exclude chatd (independent of retrieval, stays up).
pkill -f "bin/(storaged|catalogd|ingestd|queryd|embedd|vectord|pathwayd|observerd|matrixd|gateway)" 2>/dev/null || true
sleep 0.3
PIDS=()
@@ -81,6 +94,17 @@ cleanup() {
trap cleanup EXIT INT TERM
cat > "$CFG" <<EOF
# [s3] tells storaged which bucket to talk to. Without it, defaults
# resolve to "lakehouse-primary" (no -go-) which doesn't exist on this
# box and catalogd's rehydrate fails with NoSuchBucket. Access keys
# come from the secrets file (storaged -secrets defaults to
# /etc/lakehouse/secrets-go.toml), not this temp toml.
[s3]
endpoint = "http://localhost:9000"
region = "us-east-1"
bucket = "lakehouse-go-primary"
use_path_style = true
[gateway]
bind = "127.0.0.1:3110"
storaged_url = "http://127.0.0.1:3211"
@@ -91,11 +115,46 @@ vectord_url = "http://127.0.0.1:3215"
embedd_url = "http://127.0.0.1:3216"
pathwayd_url = "http://127.0.0.1:3217"
matrixd_url = "http://127.0.0.1:3218"
observerd_url = "http://127.0.0.1:3219"
[storaged]
bind = "127.0.0.1:3211"
[catalogd]
bind = "127.0.0.1:3212"
storaged_url = "http://127.0.0.1:3211"
[ingestd]
bind = "127.0.0.1:3213"
storaged_url = "http://127.0.0.1:3211"
catalogd_url = "http://127.0.0.1:3212"
max_ingest_bytes = 268435456
[queryd]
bind = "127.0.0.1:3214"
catalogd_url = "http://127.0.0.1:3212"
secrets_path = "/etc/lakehouse/secrets-go.toml"
# Aggressive refresh so the SQL probe table appears within ~1s of
# ingestd registering it, instead of the prod default 30s.
refresh_every = "1s"
[embedd]
bind = "127.0.0.1:3216"
provider_url = "http://localhost:11434"
default_model = "nomic-embed-text"
[vectord]
bind = "127.0.0.1:3215"
storaged_url = ""
[pathwayd]
bind = "127.0.0.1:3217"
persist_path = ""
[observerd]
bind = "127.0.0.1:3219"
persist_path = ""
[matrixd]
bind = "127.0.0.1:3218"
embedd_url = "http://127.0.0.1:3216"
@@ -111,26 +170,76 @@ poll_health() {
return 1
}
echo "[lift] launching stack..."
./bin/storaged -config "$CFG" > /tmp/storaged.log 2>&1 & PIDS+=($!)
echo "[lift] launching stack (10 daemons; chatd stays up independently)..."
# Order respects dependencies: storaged → catalogd (needs storaged) →
# ingestd (needs storaged+catalogd) → queryd (needs catalogd) → embedd →
# vectord → pathwayd → observerd → matrixd (needs embedd+vectord) →
# gateway (needs all of them).
./bin/storaged -config "$CFG" > /tmp/storaged.log 2>&1 & PIDS+=($!)
poll_health 3211 || { echo "storaged failed"; exit 1; }
./bin/embedd -config "$CFG" > /tmp/embedd.log 2>&1 & PIDS+=($!)
./bin/catalogd -config "$CFG" > /tmp/catalogd.log 2>&1 & PIDS+=($!)
poll_health 3212 || { echo "catalogd failed"; exit 1; }
./bin/ingestd -config "$CFG" > /tmp/ingestd.log 2>&1 & PIDS+=($!)
poll_health 3213 || { echo "ingestd failed"; exit 1; }
./bin/queryd -config "$CFG" > /tmp/queryd.log 2>&1 & PIDS+=($!)
poll_health 3214 || { echo "queryd failed"; exit 1; }
./bin/embedd -config "$CFG" > /tmp/embedd.log 2>&1 & PIDS+=($!)
poll_health 3216 || { echo "embedd failed"; exit 1; }
./bin/vectord -config "$CFG" > /tmp/vectord.log 2>&1 & PIDS+=($!)
./bin/vectord -config "$CFG" > /tmp/vectord.log 2>&1 & PIDS+=($!)
poll_health 3215 || { echo "vectord failed"; exit 1; }
./bin/matrixd -config "$CFG" > /tmp/matrixd.log 2>&1 & PIDS+=($!)
./bin/pathwayd -config "$CFG" > /tmp/pathwayd.log 2>&1 & PIDS+=($!)
poll_health 3217 || { echo "pathwayd failed"; exit 1; }
./bin/observerd -config "$CFG" > /tmp/observerd.log 2>&1 & PIDS+=($!)
poll_health 3219 || { echo "observerd failed"; exit 1; }
./bin/matrixd -config "$CFG" > /tmp/matrixd.log 2>&1 & PIDS+=($!)
poll_health 3218 || { echo "matrixd failed"; exit 1; }
./bin/gateway -config "$CFG" > /tmp/gateway.log 2>&1 & PIDS+=($!)
./bin/gateway -config "$CFG" > /tmp/gateway.log 2>&1 & PIDS+=($!)
poll_health 3110 || { echo "gateway failed"; exit 1; }
echo
echo "[lift] SQL surface probe — ingest 3-row CSV, assert SELECT COUNT(*)=3..."
PROBE_CSV="$TMP/sql_probe.csv"
cat > "$PROBE_CSV" <<CSVEOF
id,name,role
1,Alice,Forklift Operator
2,Bob,Production Worker
3,Charlie,Warehouse Associate
CSVEOF
INGEST_RESP="$(curl -sS -F "file=@$PROBE_CSV" "http://127.0.0.1:3110/v1/ingest?name=lift_sql_probe")"
echo "[lift] ingest response: $INGEST_RESP"
# queryd refresh_every=1s — give it a couple ticks to discover the new manifest.
sleep 2.5
SQL_RESP="$(curl -sS -X POST http://127.0.0.1:3110/v1/sql \
-H 'content-type: application/json' \
-d '{"sql":"SELECT COUNT(*) FROM lift_sql_probe"}')"
PROBE_COUNT="$(echo "$SQL_RESP" | jq -r '.rows[0][0] // "ERR"' 2>/dev/null || echo "ERR")"
if [ "$PROBE_COUNT" = "3" ]; then
echo "[lift] ✓ SQL surface probe passed (rowcount=3)"
else
echo "[lift] ✗ SQL surface probe FAILED (got: $SQL_RESP)"
exit 1
fi
echo
echo "[lift] ingest workers (limit=$WORKERS_LIMIT)..."
./bin/staffing_workers -limit "$WORKERS_LIMIT"
echo
echo "[lift] ingest candidates..."
./bin/staffing_candidates -skip-populate=false -query "warmup" 2>&1 \
| grep -v "^\[candidates\]\(matrix\|reality\)" || true
echo "[lift] ingest ethereal_workers (10K, second staffing-domain corpus)..."
# ethereal_workers is the right second corpus for staffing-domain reality
# tests: same schema as workers_500k but a different population (Material
# Handlers, Admin Assistants, etc.) so the matrix layer's multi-corpus
# retrieve+merge actually has TWO relevant corpora to compose against.
# Earlier versions used scripts/staffing_candidates against the SWE-tech
# candidates parquet (Swift/iOS, Scala/Spark, Rust/DataFusion) — wrong
# domain for staffing queries; effectively dead-corpus noise.
# id-prefix "e-" prevents collisions with workers' "w-" since both files
# count worker_id from 1.
./bin/staffing_workers \
-parquet "/home/profit/lakehouse/data/datasets/ethereal_workers.parquet" \
-index-name ethereal_workers \
-id-prefix "e-" \
-limit 0
echo
echo "[lift] running driver — judge=$EFFECTIVE_JUDGE · queries=$QUERIES_FILE · k=$K"
scripts/playbook_lift/main.go
+1 -1
View File
@@ -292,7 +292,7 @@ func matrixSearch(hc *http.Client, gw, query string, corpora []string, k int, us
func playbookRecord(hc *http.Client, gw, query, answerID, answerCorpus string, score float64) error {
body := map[string]any{
"query": query,
"query_text": query,
"answer_id": answerID,
"answer_corpus": answerCorpus,
"score": score,
scripts/staffing_workers/main.go
+16 -14
View File
@@ -39,8 +39,7 @@ import (
)
const (
indexName = "workers"
dim = 768
dim = 768
)
// workersSource implements corpusingest.Source over an in-memory
@@ -52,8 +51,9 @@ type workersSource struct {
workerID *chunkedInt64
name, role, city, state, skills, certs, archetype, resume, comm *chunkedString
}
n int64
cur int64
n int64
cur int64
idPrefix string // "w-" for workers, "e-" for ethereal_workers, etc.
}
// chunkedString lets per-row access work whether the table came back
@@ -120,7 +120,7 @@ func (c *chunkedInt64) At(row int64) int64 {
return 0
}
func newWorkersSource(path string) (*workersSource, func(), error) {
func newWorkersSource(path, idPrefix string) (*workersSource, func(), error) {
f, err := os.Open(path)
if err != nil {
return nil, nil, fmt.Errorf("open parquet: %w", err)
@@ -143,7 +143,7 @@ func newWorkersSource(path string) (*workersSource, func(), error) {
return nil, nil, fmt.Errorf("read table: %w", err)
}
src := &workersSource{n: table.NumRows()}
src := &workersSource{n: table.NumRows(), idPrefix: idPrefix}
schema := table.Schema()
stringCol := func(name string) (*chunkedString, error) {
@@ -248,7 +248,7 @@ func (s *workersSource) Next() (corpusingest.Row, error) {
text := b.String()
return corpusingest.Row{
ID: fmt.Sprintf("w-%d", workerID),
ID: fmt.Sprintf("%s%d", s.idPrefix, workerID),
Text: text,
Metadata: map[string]any{
"worker_id": workerID,
@@ -267,15 +267,17 @@ func main() {
var (
gateway = flag.String("gateway", "http://127.0.0.1:3110", "gateway base URL")
parquetPath = flag.String("parquet", "/home/profit/lakehouse/data/datasets/workers_500k.parquet", "workers parquet")
limit = flag.Int("limit", 5000, "limit rows (0 = all 500K — usually not what you want here)")
drop = flag.Bool("drop", true, "DELETE workers index before populate")
indexName = flag.String("index-name", "workers", "vector index name (e.g. workers, ethereal_workers)")
idPrefix = flag.String("id-prefix", "w-", "ID prefix to disambiguate worker_id collisions across corpora (e.g. w-, e-)")
limit = flag.Int("limit", 5000, "limit rows (0 = all rows; default suits multi-corpus reality testing, not stress)")
drop = flag.Bool("drop", true, "DELETE the index before populate")
)
flag.Parse()
hc := &http.Client{Timeout: 5 * time.Minute}
ctx := context.Background()
src, cleanup, err := newWorkersSource(*parquetPath)
src, cleanup, err := newWorkersSource(*parquetPath, *idPrefix)
if err != nil {
log.Fatalf("open workers source: %v", err)
}
@@ -283,7 +285,7 @@ func main() {
stats, err := corpusingest.Run(ctx, corpusingest.Config{
GatewayURL: *gateway,
IndexName: indexName,
IndexName: *indexName,
Dimension: dim,
Distance: "cosine",
EmbedBatch: 16,
@@ -296,13 +298,13 @@ func main() {
}, src)
if err != nil {
if errors.Is(err, corpusingest.ErrPartialFailure) {
fmt.Printf("[workers] WARN partial failure: %v\n", err)
fmt.Printf("[%s] WARN partial failure: %v\n", *indexName, err)
} else {
log.Fatalf("ingest: %v", err)
}
}
fmt.Printf("[workers] populate: scanned=%d embedded=%d added=%d failed=%d wall=%v\n",
stats.Scanned, stats.Embedded, stats.Added, stats.FailedBatches,
fmt.Printf("[%s] populate: scanned=%d embedded=%d added=%d failed=%d wall=%v\n",
*indexName, stats.Scanned, stats.Embedded, stats.Added, stats.FailedBatches,
stats.Wall.Round(time.Millisecond))
}