reality-tests: playbook-lift harness — does the 5-loop substrate beat raw cosine?

First reality test driver. Two-pass design:
- Pass 1 (cold): matrix.search use_playbook=false → small-model judge
  rates top-K → record playbook entry pointing at the highest-rated
  result (which may NOT be top-1 by distance — that's the discovery).
- Pass 2 (warm): same queries with use_playbook=true → measure
  ranking shift. Lift = real if recorded answer becomes top-1.

Files:
- scripts/playbook_lift/main.go         driver (391 LoC)
- scripts/playbook_lift.sh              stack-bring-up + report gen
- tests/reality/playbook_lift_queries.txt  query corpus (5 placeholders;
                                            J writes real 20+)
- reports/reality-tests/README.md       framework + interpretation
- .gitignore                            track reports/reality-tests/
                                        but ignore per-run JSON evidence

This answers the gate from project_small_model_pipeline_vision.md:
"the playbook + matrix indexer must give the results we're looking
for." Without ground-truth labels, the LLM judge is the proxy — the
same small-model thesis applied to evaluation. Honest about that
limitation in the generated reports.

Driver compiles clean; full run requires Ollama + workers/candidates
ingest. Skips cleanly if Ollama absent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 23:22:36 -05:00
co-authored by Claude Opus 4.7
parent 8278eb9a87
commit 3dd7d9fe30
5 changed files with 715 additions and 0 deletions
+4
View File
@@ -39,10 +39,14 @@ vendor/
# Use /reports/* + un-ignore so git can traverse into reports/.
/reports/*
!/reports/scrum/
!/reports/reality-tests/
# Inside the audit directory, the per-run _evidence/ dump (smoke logs,
# command output) IS runtime — track the dir, ignore its contents.
/reports/scrum/_evidence/*
!/reports/scrum/_evidence/.gitkeep
# Reality-test JSON evidence is runtime — track the dir + MD reports
# (committed deliberately as outcome record), ignore per-run JSON.
/reports/reality-tests/*.json
# Proof harness runtime output — same pattern as reports/scrum/_evidence.
# Track the directory but ignore per-run subdirs.