phase 3: playbook_lift harness reads judge from config
migrate the reality-test harness's judge-model default from a hardcoded "qwen3.5:latest" string to cfg.Models.LocalJudge. resolution priority: explicit -judge flag > $JUDGE_MODEL env > cfg.Models.LocalJudge from lakehouse.toml > hardcoded fallback. bumping the judge for run #N+1 now means editing one line in lakehouse.toml [models].local_judge — no Go file or shell script edits required. changes: - scripts/playbook_lift/main.go: -config flag added, judge default flips to "" so resolution chain runs. Imports internal/shared for config loader. - scripts/playbook_lift.sh: JUDGE_MODEL no longer defaulted in bash; EFFECTIVE_JUDGE resolved by mirror-of-the-Go-chain (env > config grep > qwen3.5:latest fallback). Used for the Ollama presence check + report header. Pre-flight grep avoids requiring jq just to read the toml. - reports/reality-tests/README.md: documents the 4-step priority chain. verified all 4 paths produce the expected judge: - config (no env): qwen3.5:latest (from lakehouse.toml) - env override: env wins - flag override: flag wins over env - missing config: DefaultConfig fallback still gives qwen3.5:latest just verify PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -29,11 +29,14 @@ cd "$(dirname "$0")/.."
|
||||
export PATH="$PATH:/usr/local/go/bin"
|
||||
|
||||
RUN_ID="${RUN_ID:-001}"
|
||||
JUDGE_MODEL="${JUDGE_MODEL:-qwen3.5:latest}"
|
||||
# JUDGE_MODEL: empty means "let the Go driver resolve from
|
||||
# lakehouse.toml [models].local_judge". Set explicitly to override.
|
||||
JUDGE_MODEL="${JUDGE_MODEL:-}"
|
||||
WORKERS_LIMIT="${WORKERS_LIMIT:-5000}"
|
||||
QUERIES_FILE="${QUERIES_FILE:-tests/reality/playbook_lift_queries.txt}"
|
||||
CORPORA="${CORPORA:-workers,candidates}"
|
||||
K="${K:-10}"
|
||||
CONFIG_PATH="${CONFIG_PATH:-lakehouse.toml}"
|
||||
|
||||
OUT_JSON="reports/reality-tests/playbook_lift_${RUN_ID}.json"
|
||||
OUT_MD="reports/reality-tests/playbook_lift_${RUN_ID}.md"
|
||||
@@ -43,11 +46,20 @@ if ! curl -sS --max-time 3 http://localhost:11434/api/tags >/dev/null 2>&1; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! curl -sS http://localhost:11434/api/tags | jq -e --arg m "$JUDGE_MODEL" \
|
||||
# Resolve judge from config when not set explicitly — needed for the
|
||||
# Ollama model-presence check below. Mirrors the Go driver's priority.
|
||||
EFFECTIVE_JUDGE="$JUDGE_MODEL"
|
||||
if [ -z "$EFFECTIVE_JUDGE" ] && [ -f "$CONFIG_PATH" ]; then
|
||||
EFFECTIVE_JUDGE="$(grep -E '^local_judge\s*=' "$CONFIG_PATH" | head -1 | sed -E 's/.*=\s*"([^"]+)".*/\1/')"
|
||||
fi
|
||||
EFFECTIVE_JUDGE="${EFFECTIVE_JUDGE:-qwen3.5:latest}"
|
||||
|
||||
if ! curl -sS http://localhost:11434/api/tags | jq -e --arg m "$EFFECTIVE_JUDGE" \
|
||||
'.models[] | select(.name == $m)' >/dev/null 2>&1; then
|
||||
echo "[lift] judge model '$JUDGE_MODEL' not loaded in Ollama — pull it first"
|
||||
echo "[lift] judge model '$EFFECTIVE_JUDGE' not loaded in Ollama — pull it first"
|
||||
exit 1
|
||||
fi
|
||||
echo "[lift] judge resolved to: $EFFECTIVE_JUDGE (from ${JUDGE_MODEL:+env}${JUDGE_MODEL:-config})"
|
||||
|
||||
echo "[lift] building binaries..."
|
||||
go build -o bin/ ./cmd/storaged ./cmd/embedd ./cmd/vectord ./cmd/matrixd ./cmd/gateway \
|
||||
@@ -121,8 +133,12 @@ echo "[lift] ingest candidates..."
|
||||
| grep -v "^\[candidates\]\(matrix\|reality\)" || true
|
||||
|
||||
echo
|
||||
echo "[lift] running driver — judge=$JUDGE_MODEL · queries=$QUERIES_FILE · k=$K"
|
||||
echo "[lift] running driver — judge=$EFFECTIVE_JUDGE · queries=$QUERIES_FILE · k=$K"
|
||||
# Pass empty -judge so the Go driver runs its config resolution chain
|
||||
# unless the operator explicitly set JUDGE_MODEL (in which case we
|
||||
# pass it through, taking priority over config).
|
||||
./bin/playbook_lift \
|
||||
-config "$CONFIG_PATH" \
|
||||
-gateway "http://127.0.0.1:3110" \
|
||||
-ollama "http://localhost:11434" \
|
||||
-queries "$QUERIES_FILE" \
|
||||
@@ -148,7 +164,7 @@ generate_md() {
|
||||
# Playbook-Lift Reality Test — Run ${RUN_ID}
|
||||
|
||||
**Generated:** ${gen_at}
|
||||
**Judge:** \`${JUDGE_MODEL}\` (Ollama)
|
||||
**Judge:** \`${EFFECTIVE_JUDGE}\` (Ollama, resolved from ${JUDGE_MODEL:+env JUDGE_MODEL}${JUDGE_MODEL:-config [models].local_judge})
|
||||
**Corpora:** \`${CORPORA}\`
|
||||
**Workers limit:** ${WORKERS_LIMIT}
|
||||
**Queries:** \`${QUERIES_FILE}\` (${total} executed)
|
||||
@@ -211,6 +227,9 @@ MDEOF
|
||||
4. **Multi-corpus skew.** Default corpora=\`${CORPORA}\` — if all judge-best
|
||||
results land in one corpus, the matrix layer's purpose isn't being tested.
|
||||
Check per-corpus distribution in the JSON.
|
||||
5. **Judge resolution.** This run used \`${EFFECTIVE_JUDGE}\` from
|
||||
${JUDGE_MODEL:+env JUDGE_MODEL override}${JUDGE_MODEL:-the lakehouse.toml [models].local_judge tier}.
|
||||
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
|
||||
|
||||
## Next moves
|
||||
|
||||
|
||||
Reference in New Issue
Block a user