profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

playbook_lift: harness expansion + reality test #001 (7/8 lift, 87.5%)

Browse Source 
The 5-loop substrate's load-bearing gate is verified — playbook +
matrix indexer give the results we're looking for. Per the report's
rubric, lift ≥ 50% of discoveries means matrix is doing real work;
7/8 = 87.5% blew through that.

Harness was structurally hiding bugs behind a 5-daemon stripped boot.
Expanding to the full 10-daemon prod stack surfaced 7 fixes in cascade:

1. driver→matrixd: {"query": ...} → {"query_text": ...} field name
2. harness temp toml missing [s3] → wrong default bucket → catalogd
   rehydrate 500 on first call
3. harness→queryd SQL probe: {"q": ...} → {"sql": ...} field name
4. expand boot from 5 → 10 daemons in dep-ordered launch
5. add SQL surface probe (3-row CSV ingest → COUNT(*)=3 assertion)
6. candidates corpus was synthetic SWE-tech (Swift/iOS, Scala/Spark) —
   wrong domain for staffing queries; replaced with ethereal_workers
   (10K rows, real staffing schema, "e-" id prefix to avoid collision
   with workers' "w-"). staffing_workers driver gains -index-name +
   -id-prefix flags so the same binary serves both corpora
7. local_judge qwen3.5:latest is a vision-SSM 256K-ctx build running
   ~30s per judge call against the lift loop; reverted to
   qwen2.5:latest (~1s/call, 30× faster, held lift theory)

Each contract drift (1, 3) is now locked into a cmd/<bin>/main_test.go
so future drift fires in `go test`, not in a reality run. R-005 closed:

- cmd/matrixd/main_test.go (new) — playbook record drift detector +
  score bounds + 6 routes mounted
- cmd/queryd/main_test.go — wrong-field-name drift detector
- cmd/pathwayd/main_test.go (new) — 9 routes + add round-trip + retire
- cmd/observerd/main_test.go (new) — 4 routes + invalid-op + unknown-mode

`go test ./cmd/{matrixd,queryd,pathwayd,observerd}` all green.

Reality test results (reports/reality-tests/playbook_lift_001.{json,md}):
  Queries              21 (staffing-domain, 7 categories)
  Discoveries          8 (judge ≠ cosine top-1)
  Lifts                7/8 (87.5%)
  Boosts triggered     9
  Mean Δ distance      -0.053 (warm closer than cold)
  OOD honesty          dental/RN/SWE rated 1, no fake matches
  Cross-corpus boosts  confirmed (e- ↔ w- swaps in lifts)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-30 06:22:21 -05:00
co-authored by Claude Opus 4.7
parent 740eb0d00c
commit b2e45f7f26
11 changed files with 699 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATE_OF_PLAY.md
+45 -4
View File
@@ -1,7 +1,7 @@
# STATE OF PLAY — Lakehouse-Go
**Last verified:** 2026-04-30 ~01:00 CDT
**Verified by:** live probes + `just verify` PASS, not memory.
**Last verified:** 2026-04-30 ~05:50 CDT
**Verified by:** live probes + `just verify` PASS + reality test PASS (7/8 lift), not memory.
> **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo); the Rust legacy at `/home/profit/lakehouse/` is maintenance-only. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
@@ -95,6 +95,47 @@ Callers read `cfg.Models.LocalJudge` etc. instead of literal strings. `playbook_
Composite **50/60** at scrum2 head `c7e3124` (was 35 baseline → 43 R1 → 50 R2). Today's chatd wave reviewed by Opus + Kimi + Qwen3-coder via the chatd's own `/v1/chat`; **2 BLOCKs + 2 WARNs landed as fixes** (`0efc736`); reusable driver at `scripts/scrum_review.sh`.
### Reality test PASSED — `playbook_lift_001` (2026-04-30 ~05:50 CDT)
The 5-loop substrate's load-bearing gate (per `project_small_model_pipeline_vision.md`: *"the playbook + matrix indexer must give the results we're looking for"*) is verified.
| Metric | Value |
|---|---:|
| Queries | 21 (staffing-domain, 7 categories) |
| Cold-pass discoveries (judge-best ≠ top-1) | 8 |
| **Warm-pass lifts** (recorded playbook → top-1) | **7 / 8 (87.5%)** |
| Boosts triggered | 9 |
| Mean Δ top-1 distance | -0.053 (warm consistently closer) |
| OOD honesty (dental/RN/SWE queries) | rated 1, no fake matches |
| Cross-corpus boosts | confirmed (e- ↔ w- swaps in lifts) |
Evidence: `reports/reality-tests/playbook_lift_001.{json,md}`. Per the report's rubric (lift ≥ 50% = matrix doing real work), 87.5% means we're well past validation.
### Harness expansion (2026-04-30 ~05:30 CDT)
`scripts/playbook_lift.sh` rewritten from a 5-daemon stripped harness to the **full 10-daemon prod-realistic stack** (chatd stays up independently). The 5-daemon version was structurally hiding bugs; expanding the daemon set surfaced 7 distinct fixes:
| # | Fix | Lock |
|---|---|---|
| 1 | driver→matrixd: `query` → `query_text` field name | `cmd/matrixd/main_test.go` TestPlaybookRecord_OldFieldNameRejected |
| 2 | harness toml missing `[s3]` block | inline comment in `scripts/playbook_lift.sh` |
| 3 | harness→queryd: `q` → `sql` field name | `cmd/queryd/main_test.go` TestHandleSQL_WrongFieldName_400 |
| 4 | 5→10 daemon boot order | inline comment + dep-ordered launch |
| 5 | SQL surface probe (3-row CSV → COUNT=3) | `[lift] ✓ SQL surface probe passed` assertion |
| 6 | `candidates` corpus was SWE-tech, not staffing | swapped to `ethereal_workers.parquet` (10K rows, real staffing schema, "e-" id prefix) |
| 7 | `qwen3.5:latest` is vision-SSM 256K-ctx → 30s/judge | reverted `local_judge` to `qwen2.5:latest` (1s/judge, 30× faster) |
### R-005 closed (2026-04-30 ~05:35 CDT)
Four new `cmd/<bin>/main_test.go` files — chi router-level contract tests:
- `cmd/matrixd/main_test.go` (123 lines) — playbook record drift detector + score bounds + 6 routes mounted
- `cmd/queryd/main_test.go` (extended) — wrong-field-name drift detector
- `cmd/pathwayd/main_test.go` (102 lines) — 9 routes + add round-trip + retire-nonexistent
- `cmd/observerd/main_test.go` (98 lines) — 4 routes + invalid-op 400 + unknown-mode 400
`go test ./cmd/{matrixd,queryd,pathwayd,observerd}` all green. R-005 from prior STATE OPEN list is closed.
---
## DO NOT RELITIGATE
@@ -135,8 +176,8 @@ Verbatim verdicts at `reports/scrum/_evidence/2026-04-30/verdicts/`. Disposition
| Item | What | When to act |
|---|---|---|
| **Reality test for the 5-loop substrate** | `playbook_lift_001.json` exists at `reports/reality-tests/` but the harness hasn't been run against real queries yet (J held it). Driver: `scripts/playbook_lift.sh`. Needs J's 20+ staffing queries in `tests/reality/playbook_lift_queries.txt` first (5 placeholders shipped). | When J supplies queries OR explicitly green-lights running with placeholders. |
| **`cmd/{matrixd,observerd,pathwayd}/main_test.go` absent** | 3 new daemons each mount ≥4 routes with no wiring test. Original 6 binaries all closed via `0f79bce`. New gap reopens R-005. | ~1 hr pattern-match against `cmd/storaged/main_test.go`. Cheap. |
| **Reality test v2: paraphrase queries** | The 21 verbatim queries in `tests/reality/playbook_lift_queries.txt` exercise verbatim replay only. The interesting case is *similar but not identical* queries hitting a recorded playbook — does the cosine on `query_text` find the playbook hit? Add a paraphrase pass and measure. | After J wants to push the harness past v1 baseline. |
| **Q15 boost-math edge case** | "Engaged warehouse associate with strong safety compliance" — judge picked rank-9 result; score=1.0 boost halves distance but rank-9 was >2× top-1 distance, so not promoted. Documented in caveat #2. Either (a) accept the math limit, or (b) tier scores so judge-best-found-deep gets score>1.0. Open design call. | When a second reality run shows the same edge case persisting. |
| **Sprint 4 — deployment** | No `REPLICATION.md`, `secrets-go.toml.example`, `deploy/systemd/<bin>.service`, `Dockerfile`. Largest open Sprint. Required input for any G5 cutover plan. | When G5 cutover is on the table. |
| **ADR-005 — observer fail-safe semantics** | Observer ported but the upstream "verdict:accept on crash" anti-pattern still has no Go-side decision locked. Doc-only, ~30 min. | Before observer is wired into production paths. |
| **ADR-006 — auth posture for non-loopback deploy** | Locks R-001 + R-007 from "opt-in middleware exists" to "wired-by-default for X, opt-in for Y." Doc-only, ~1 hr. | Required before any Go binary binds non-loopback in prod. |