phase 3: playbook_lift harness reads judge from config

migrate the reality-test harness's judge-model default from a
hardcoded "qwen3.5:latest" string to cfg.Models.LocalJudge.

resolution priority: explicit -judge flag > $JUDGE_MODEL env >
cfg.Models.LocalJudge from lakehouse.toml > hardcoded fallback.

bumping the judge for run #N+1 now means editing one line in
lakehouse.toml [models].local_judge — no Go file or shell script
edits required.

changes:
- scripts/playbook_lift/main.go: -config flag added, judge default
  flips to "" so resolution chain runs. Imports internal/shared for
  config loader.
- scripts/playbook_lift.sh: JUDGE_MODEL no longer defaulted in bash;
  EFFECTIVE_JUDGE resolved by mirror-of-the-Go-chain (env > config
  grep > qwen3.5:latest fallback). Used for the Ollama presence
  check + report header. Pre-flight grep avoids requiring jq just
  to read the toml.
- reports/reality-tests/README.md: documents the 4-step priority
  chain.

verified all 4 paths produce the expected judge:
- config (no env): qwen3.5:latest (from lakehouse.toml)
- env override:    env wins
- flag override:   flag wins over env
- missing config:  DefaultConfig fallback still gives qwen3.5:latest

just verify PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 23:57:28 -05:00
co-authored by Claude Opus 4.7
parent 622e124b8f
commit 848cbf5fef
3 changed files with 59 additions and 9 deletions
+16 -3
View File
@@ -34,17 +34,30 @@ See the run reports for honesty caveats — chiefly that the LLM judge IS the gr
## Running a reality test
```bash
# Defaults: judge=qwen3.5:latest, workers limit 5000, run id 001
# Defaults: judge resolved from lakehouse.toml [models].local_judge,
# workers limit 5000, run id 001
./scripts/playbook_lift.sh
# Re-run with a different judge to check inter-judge agreement
JUDGE_MODEL=qwen2.5:latest RUN_ID=002 ./scripts/playbook_lift.sh
# (env JUDGE_MODEL overrides the config tier)
JUDGE_MODEL=qwen3:latest RUN_ID=002 ./scripts/playbook_lift.sh
# Smaller scale for fast iteration
WORKERS_LIMIT=1000 K=5 RUN_ID=dev ./scripts/playbook_lift.sh
```
Requires: Ollama on `:11434` with `nomic-embed-text` + the chosen judge model loaded. Skips cleanly (exit 0) if Ollama is absent.
**Judge resolution priority** (Phase 3, 2026-04-29):
1. `-judge` flag on the Go driver (explicit override)
2. `JUDGE_MODEL` env var (operator override)
3. `lakehouse.toml [models].local_judge` (default)
4. Hardcoded `qwen3.5:latest` (last-resort fallback if config missing)
This means model bumps land in `lakehouse.toml`, not in this script or
the Go driver. Bumping `local_judge` to a stronger local model (e.g.
when qwen4 ships) takes one line.
Requires: Ollama on `:11434` with `nomic-embed-text` + the resolved judge
model loaded. Skips cleanly (exit 0) if Ollama is absent.
---