profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase 3: playbook_lift harness reads judge from config

Browse Source 
migrate the reality-test harness's judge-model default from a
hardcoded "qwen3.5:latest" string to cfg.Models.LocalJudge.

resolution priority: explicit -judge flag > $JUDGE_MODEL env >
cfg.Models.LocalJudge from lakehouse.toml > hardcoded fallback.

bumping the judge for run #N+1 now means editing one line in
lakehouse.toml [models].local_judge — no Go file or shell script
edits required.

changes:
- scripts/playbook_lift/main.go: -config flag added, judge default
  flips to "" so resolution chain runs. Imports internal/shared for
  config loader.
- scripts/playbook_lift.sh: JUDGE_MODEL no longer defaulted in bash;
  EFFECTIVE_JUDGE resolved by mirror-of-the-Go-chain (env > config
  grep > qwen3.5:latest fallback). Used for the Ollama presence
  check + report header. Pre-flight grep avoids requiring jq just
  to read the toml.
- reports/reality-tests/README.md: documents the 4-step priority
  chain.

verified all 4 paths produce the expected judge:
- config (no env): qwen3.5:latest (from lakehouse.toml)
- env override:    env wins
- flag override:   flag wins over env
- missing config:  DefaultConfig fallback still gives qwen3.5:latest

just verify PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 23:57:28 -05:00
co-authored by Claude Opus 4.7
parent 622e124b8f
commit 848cbf5fef
3 changed files with 59 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/playbook_lift.sh
+24 -5
View File
@@ -29,11 +29,14 @@ cd "$(dirname "$0")/.."
export PATH="$PATH:/usr/local/go/bin"
RUN_ID="${RUN_ID:-001}"
JUDGE_MODEL="${JUDGE_MODEL:-qwen3.5:latest}"
# JUDGE_MODEL: empty means "let the Go driver resolve from
# lakehouse.toml [models].local_judge". Set explicitly to override.
JUDGE_MODEL="${JUDGE_MODEL:-}"
WORKERS_LIMIT="${WORKERS_LIMIT:-5000}"
QUERIES_FILE="${QUERIES_FILE:-tests/reality/playbook_lift_queries.txt}"
CORPORA="${CORPORA:-workers,candidates}"
K="${K:-10}"
CONFIG_PATH="${CONFIG_PATH:-lakehouse.toml}"
OUT_JSON="reports/reality-tests/playbook_lift_${RUN_ID}.json"
OUT_MD="reports/reality-tests/playbook_lift_${RUN_ID}.md"
@@ -43,11 +46,20 @@ if ! curl -sS --max-time 3 http://localhost:11434/api/tags >/dev/null 2>&1; then
exit 0
fi
if ! curl -sS http://localhost:11434/api/tags | jq -e --arg m "$JUDGE_MODEL" \
# Resolve judge from config when not set explicitly — needed for the
# Ollama model-presence check below. Mirrors the Go driver's priority.
EFFECTIVE_JUDGE="$JUDGE_MODEL"
if [ -z "$EFFECTIVE_JUDGE" ] && [ -f "$CONFIG_PATH" ]; then
EFFECTIVE_JUDGE="$(grep -E '^local_judge\s*=' "$CONFIG_PATH" | head -1 | sed -E 's/.*=\s*"([^"]+)".*/\1/')"
fi
EFFECTIVE_JUDGE="${EFFECTIVE_JUDGE:-qwen3.5:latest}"
if ! curl -sS http://localhost:11434/api/tags | jq -e --arg m "$EFFECTIVE_JUDGE" \
'.models[] | select(.name == $m)' >/dev/null 2>&1; then
echo "[lift] judge model '$JUDGE_MODEL' not loaded in Ollama — pull it first"
echo "[lift] judge model '$EFFECTIVE_JUDGE' not loaded in Ollama — pull it first"
exit 1
fi
echo "[lift] judge resolved to: $EFFECTIVE_JUDGE (from ${JUDGE_MODEL:+env}${JUDGE_MODEL:-config})"
echo "[lift] building binaries..."
go build -o bin/ ./cmd/storaged ./cmd/embedd ./cmd/vectord ./cmd/matrixd ./cmd/gateway \
@@ -121,8 +133,12 @@ echo "[lift] ingest candidates..."
| grep -v "^\[candidates\]\(matrix\|reality\)" || true
echo
echo "[lift] running driver — judge=$JUDGE_MODEL · queries=$QUERIES_FILE · k=$K"
echo "[lift] running driver — judge=$EFFECTIVE_JUDGE · queries=$QUERIES_FILE · k=$K"
# Pass empty -judge so the Go driver runs its config resolution chain
# unless the operator explicitly set JUDGE_MODEL (in which case we
# pass it through, taking priority over config).
./bin/playbook_lift \
-config "$CONFIG_PATH" \
-gateway "http://127.0.0.1:3110" \
-ollama "http://localhost:11434" \
-queries "$QUERIES_FILE" \
@@ -148,7 +164,7 @@ generate_md() {
# Playbook-Lift Reality Test — Run ${RUN_ID}
**Generated:** ${gen_at}
**Judge:** \`${JUDGE_MODEL}\` (Ollama)
**Judge:** \`${EFFECTIVE_JUDGE}\` (Ollama, resolved from ${JUDGE_MODEL:+env JUDGE_MODEL}${JUDGE_MODEL:-config [models].local_judge})
**Corpora:** \`${CORPORA}\`
**Workers limit:** ${WORKERS_LIMIT}
**Queries:** \`${QUERIES_FILE}\` (${total} executed)
@@ -211,6 +227,9 @@ MDEOF
4. **Multi-corpus skew.** Default corpora=\`${CORPORA}\` — if all judge-best
results land in one corpus, the matrix layer's purpose isn't being tested.
Check per-corpus distribution in the JSON.
5. **Judge resolution.** This run used \`${EFFECTIVE_JUDGE}\` from
${JUDGE_MODEL:+env JUDGE_MODEL override}${JUDGE_MODEL:-the lakehouse.toml [models].local_judge tier}.
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
## Next moves
scripts/playbook_lift/main.go
+19 -1
View File
@@ -40,6 +40,8 @@ import (
"sort"
"strings"
"time"
"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
)
type matrixResult struct {
@@ -93,15 +95,31 @@ type summary struct {
}
func main() {
configPath := flag.String("config", "lakehouse.toml", "path to TOML config (provides judge default from [models].local_judge)")
gw := flag.String("gateway", "http://127.0.0.1:3110", "Go gateway base URL")
ollama := flag.String("ollama", "http://127.0.0.1:11434", "Ollama base URL for LLM judge")
queries := flag.String("queries", "tests/reality/playbook_lift_queries.txt", "query corpus path")
corporaCSV := flag.String("corpora", "workers,candidates", "comma-separated matrix corpora")
judge := flag.String("judge", "qwen3.5:latest", "Ollama model for relevance judging")
// Empty default — resolved below from (priority): flag > env > config > hardcoded.
judge := flag.String("judge", "", "Ollama model for relevance judging (empty = read from config [models].local_judge)")
k := flag.Int("k", 10, "top-k from matrix.search per pass")
out := flag.String("out", "reports/reality-tests/playbook_lift_001.json", "output JSONL path")
flag.Parse()
// Judge resolution priority: explicit flag > $JUDGE_MODEL env >
// cfg.Models.LocalJudge > hardcoded fallback. Phase 3 wired this
// up so model bumps land in lakehouse.toml, not in this driver.
if *judge == "" {
if env := strings.TrimSpace(os.Getenv("JUDGE_MODEL")); env != "" {
*judge = env
} else if cfg, err := shared.LoadConfig(*configPath); err == nil && cfg.Models.LocalJudge != "" {
*judge = cfg.Models.LocalJudge
} else {
*judge = "qwen3.5:latest"
log.Printf("[lift] warn: no judge model from flag/env/config; falling back to %q", *judge)
}
}
corpora := strings.Split(*corporaCSV, ",")
qs, err := loadQueries(*queries)