profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase 3: playbook_lift harness reads judge from config

Browse Source 
migrate the reality-test harness's judge-model default from a
hardcoded "qwen3.5:latest" string to cfg.Models.LocalJudge.

resolution priority: explicit -judge flag > $JUDGE_MODEL env >
cfg.Models.LocalJudge from lakehouse.toml > hardcoded fallback.

bumping the judge for run #N+1 now means editing one line in
lakehouse.toml [models].local_judge — no Go file or shell script
edits required.

changes:
- scripts/playbook_lift/main.go: -config flag added, judge default
  flips to "" so resolution chain runs. Imports internal/shared for
  config loader.
- scripts/playbook_lift.sh: JUDGE_MODEL no longer defaulted in bash;
  EFFECTIVE_JUDGE resolved by mirror-of-the-Go-chain (env > config
  grep > qwen3.5:latest fallback). Used for the Ollama presence
  check + report header. Pre-flight grep avoids requiring jq just
  to read the toml.
- reports/reality-tests/README.md: documents the 4-step priority
  chain.

verified all 4 paths produce the expected judge:
- config (no env): qwen3.5:latest (from lakehouse.toml)
- env override:    env wins
- flag override:   flag wins over env
- missing config:  DefaultConfig fallback still gives qwen3.5:latest

just verify PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-29 23:57:28 -05:00
co-authored by Claude Opus 4.7
parent 622e124b8f
commit 848cbf5fef
3 changed files with 59 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/playbook_lift/main.go
+19 -1
View File
@@ -40,6 +40,8 @@ import (
"sort"
"strings"
"time"
"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
)
type matrixResult struct {
@@ -93,15 +95,31 @@ type summary struct {
}
func main() {
configPath := flag.String("config", "lakehouse.toml", "path to TOML config (provides judge default from [models].local_judge)")
gw := flag.String("gateway", "http://127.0.0.1:3110", "Go gateway base URL")
ollama := flag.String("ollama", "http://127.0.0.1:11434", "Ollama base URL for LLM judge")
queries := flag.String("queries", "tests/reality/playbook_lift_queries.txt", "query corpus path")
corporaCSV := flag.String("corpora", "workers,candidates", "comma-separated matrix corpora")
judge := flag.String("judge", "qwen3.5:latest", "Ollama model for relevance judging")
// Empty default — resolved below from (priority): flag > env > config > hardcoded.
judge := flag.String("judge", "", "Ollama model for relevance judging (empty = read from config [models].local_judge)")
k := flag.Int("k", 10, "top-k from matrix.search per pass")
out := flag.String("out", "reports/reality-tests/playbook_lift_001.json", "output JSONL path")
flag.Parse()
// Judge resolution priority: explicit flag > $JUDGE_MODEL env >
// cfg.Models.LocalJudge > hardcoded fallback. Phase 3 wired this
// up so model bumps land in lakehouse.toml, not in this driver.
if *judge == "" {
if env := strings.TrimSpace(os.Getenv("JUDGE_MODEL")); env != "" {
*judge = env
} else if cfg, err := shared.LoadConfig(*configPath); err == nil && cfg.Models.LocalJudge != "" {
*judge = cfg.Models.LocalJudge
} else {
*judge = "qwen3.5:latest"
log.Printf("[lift] warn: no judge model from flag/env/config; falling back to %q", *judge)
}
}
corpora := strings.Split(*corporaCSV, ",")
qs, err := loadQueries(*queries)