phase 3: playbook_lift harness reads judge from config
migrate the reality-test harness's judge-model default from a hardcoded "qwen3.5:latest" string to cfg.Models.LocalJudge. resolution priority: explicit -judge flag > $JUDGE_MODEL env > cfg.Models.LocalJudge from lakehouse.toml > hardcoded fallback. bumping the judge for run #N+1 now means editing one line in lakehouse.toml [models].local_judge — no Go file or shell script edits required. changes: - scripts/playbook_lift/main.go: -config flag added, judge default flips to "" so resolution chain runs. Imports internal/shared for config loader. - scripts/playbook_lift.sh: JUDGE_MODEL no longer defaulted in bash; EFFECTIVE_JUDGE resolved by mirror-of-the-Go-chain (env > config grep > qwen3.5:latest fallback). Used for the Ollama presence check + report header. Pre-flight grep avoids requiring jq just to read the toml. - reports/reality-tests/README.md: documents the 4-step priority chain. verified all 4 paths produce the expected judge: - config (no env): qwen3.5:latest (from lakehouse.toml) - env override: env wins - flag override: flag wins over env - missing config: DefaultConfig fallback still gives qwen3.5:latest just verify PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -34,17 +34,30 @@ See the run reports for honesty caveats — chiefly that the LLM judge IS the gr
|
||||
## Running a reality test
|
||||
|
||||
```bash
|
||||
# Defaults: judge=qwen3.5:latest, workers limit 5000, run id 001
|
||||
# Defaults: judge resolved from lakehouse.toml [models].local_judge,
|
||||
# workers limit 5000, run id 001
|
||||
./scripts/playbook_lift.sh
|
||||
|
||||
# Re-run with a different judge to check inter-judge agreement
|
||||
JUDGE_MODEL=qwen2.5:latest RUN_ID=002 ./scripts/playbook_lift.sh
|
||||
# (env JUDGE_MODEL overrides the config tier)
|
||||
JUDGE_MODEL=qwen3:latest RUN_ID=002 ./scripts/playbook_lift.sh
|
||||
|
||||
# Smaller scale for fast iteration
|
||||
WORKERS_LIMIT=1000 K=5 RUN_ID=dev ./scripts/playbook_lift.sh
|
||||
```
|
||||
|
||||
Requires: Ollama on `:11434` with `nomic-embed-text` + the chosen judge model loaded. Skips cleanly (exit 0) if Ollama is absent.
|
||||
**Judge resolution priority** (Phase 3, 2026-04-29):
|
||||
1. `-judge` flag on the Go driver (explicit override)
|
||||
2. `JUDGE_MODEL` env var (operator override)
|
||||
3. `lakehouse.toml [models].local_judge` (default)
|
||||
4. Hardcoded `qwen3.5:latest` (last-resort fallback if config missing)
|
||||
|
||||
This means model bumps land in `lakehouse.toml`, not in this script or
|
||||
the Go driver. Bumping `local_judge` to a stronger local model (e.g.
|
||||
when qwen4 ships) takes one line.
|
||||
|
||||
Requires: Ollama on `:11434` with `nomic-embed-text` + the resolved judge
|
||||
model loaded. Skips cleanly (exit 0) if Ollama is absent.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user