reality-tests: playbook-lift harness — does the 5-loop substrate beat raw cosine?
First reality test driver. Two-pass design:
- Pass 1 (cold): matrix.search use_playbook=false → small-model judge
rates top-K → record playbook entry pointing at the highest-rated
result (which may NOT be top-1 by distance — that's the discovery).
- Pass 2 (warm): same queries with use_playbook=true → measure
ranking shift. Lift = real if recorded answer becomes top-1.
Files:
- scripts/playbook_lift/main.go driver (391 LoC)
- scripts/playbook_lift.sh stack-bring-up + report gen
- tests/reality/playbook_lift_queries.txt query corpus (5 placeholders;
J writes real 20+)
- reports/reality-tests/README.md framework + interpretation
- .gitignore track reports/reality-tests/
but ignore per-run JSON evidence
This answers the gate from project_small_model_pipeline_vision.md:
"the playbook + matrix indexer must give the results we're looking
for." Without ground-truth labels, the LLM judge is the proxy — the
same small-model thesis applied to evaluation. Honest about that
limitation in the generated reports.
Driver compiles clean; full run requires Ollama + workers/candidates
ingest. Skips cleanly if Ollama absent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,10 +39,14 @@ vendor/
|
||||
# Use /reports/* + un-ignore so git can traverse into reports/.
|
||||
/reports/*
|
||||
!/reports/scrum/
|
||||
!/reports/reality-tests/
|
||||
# Inside the audit directory, the per-run _evidence/ dump (smoke logs,
|
||||
# command output) IS runtime — track the dir, ignore its contents.
|
||||
/reports/scrum/_evidence/*
|
||||
!/reports/scrum/_evidence/.gitkeep
|
||||
# Reality-test JSON evidence is runtime — track the dir + MD reports
|
||||
# (committed deliberately as outcome record), ignore per-run JSON.
|
||||
/reports/reality-tests/*.json
|
||||
|
||||
# Proof harness runtime output — same pattern as reports/scrum/_evidence.
|
||||
# Track the directory but ignore per-run subdirs.
|
||||
|
||||
Reference in New Issue
Block a user