profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Model matrix — 5 tiers, local hard workers + cloud overseers

Browse Source 
config/models.json is the authoritative catalog. Hot path (T1/T2) stays
local; cloud is consulted only for overview (T3), strategic (T4), and
gatekeeper (T5) calls. J named qwen3.5 + newer models (minimax-m2.7,
glm-5, qwen3-next) specifically — all mapped with real reachable IDs
verified against ollama.com/api/tags.

Tier shape:
- t1_hot     mistral + qwen2.5 local       — 50-200 calls/scenario
- t2_review  qwen2.5 + qwen3 local         — 5-14 calls/event
- t3_overview gpt-oss:120b cloud           — 1-3 calls/scenario
- t4_strategic qwen3.5:397b + glm-4.7      — 1-10 calls/day
- t5_gatekeeper kimi-k2-thinking           — 1-5 calls/day, audit-logged

Rate budgets are declared in-config — Ollama Cloud paid tier is generous
but we cap overview/strategic/gatekeeper so no single rogue scenario can
blow the day's quota.

Experimental rotation list wired but disabled by default. When enabled,
T4 randomly routes 10% of calls to a rotating minimax/GLM/qwen-next/
deepseek/nemotron/cogito/mistral-large candidate, logs comparisons, and
auto-promotes after 3 rotations of wins.

Playbook versioning SPEC embedded under `playbook_versioning` key: every
seed gets version + parent_id + retired_at + architecture_snapshot, so
when a schema migration breaks a playbook we can pinpoint which change
retired it. Implementation flagged for next sprint (touches gateway +
catalogd + mcp-server) — not wired here.

- scenario.ts now loads config/models.json at init, env vars still override
- mcp-server exposes /models/matrix read-only so UI can render it
This commit is contained in:
root
2026-04-20 19:24:41 -05:00
parent e4ae5b646e
commit 03d723e7e6
3 changed files with 157 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/multi-agent/scenario.ts
+22 -1
View File
@@ -53,8 +53,29 @@ const DRAFT_MODEL = "qwen2.5:latest"; // artifact generation; short outputs
// every N events, and (A) cross-day lesson after all events complete.
// gpt-oss:20b is a thinking model: it spends tokens in a hidden reasoning
// block before emitting `response`. Budget accordingly — never under 400.
// Model matrix — config/models.json is authoritative. Env vars override.
// Loaded at module init so we can log the tier shape at scenario start.
interface ModelTier {
primary: { model: string; provider: string };
local_fallback?: { model: string; provider: string };
max_tokens?: number;
temperature?: number;
env_flag?: string;
}
let MODEL_MATRIX: { tiers: Record<string, ModelTier> } = { tiers: {} };
try {
MODEL_MATRIX = JSON.parse(await Bun.file("config/models.json").text());
} catch {
// Config optional — env vars alone work too. Silent: the per-tier
// logging below will show "default" if matrix is empty.
}
const T3_TIER = MODEL_MATRIX.tiers?.t3_overview;
const OVERVIEW_CLOUD = process.env.LH_OVERVIEW_CLOUD === "1";
const OVERVIEW_MODEL = process.env.LH_OVERVIEW_MODEL ?? (OVERVIEW_CLOUD ? "gpt-oss:120b" : "gpt-oss:20b");
const OVERVIEW_MODEL = process.env.LH_OVERVIEW_MODEL
?? (OVERVIEW_CLOUD
? (T3_TIER?.primary.model ?? "gpt-oss:120b")
: (T3_TIER?.local_fallback?.model ?? "gpt-oss:20b"));
const T3_CHECKPOINT_EVERY = Number(process.env.LH_T3_CHECKPOINT_EVERY ?? 3);
const T3_DISABLED = process.env.LH_T3_DISABLE === "1";