qwen3.5 executor + continuation primitive + think:false
Three coupled fixes that together turned the Riverfront Steel scenario
from 0/5 (mistral) to 4/5 (qwen3.5) with T3 flagging real staffing
concerns rather than linter advice.
MODEL SWAP
- Executor: mistral → qwen3.5:latest (9.7B, 262K ctx, thinking).
mistral's decoder emitted malformed JSON on complex SQL filters
regardless of prompt; J called it — stop using mistral.
- Reviewer: qwen2.5 → qwen3:latest (40K ctx)
- Applied to scenario.ts, orchestrator.ts, network_proving.ts,
run_e2e_rated.ts
CONTINUATION PRIMITIVE (agent.ts)
- generateContinuable(): empty-response → geometric backoff retry;
truncated-JSON → continue from partial as scratchpad; bounded by
budget cap + max_continuations. No more "bump max_tokens until it
stops truncating" tourniquet.
- generateTreeSplit(): map-reduce for oversized input corpora with
running scratchpad digest, reduce pass for final synthesis.
- Empty text no longer throws — it's a signal to continuable that
thinking ate the budget.
think:false FOR HOT PATH
- qwen3.5 burned ~650 tokens of hidden thinking for trivial JSON
emission. For executor/reviewer/draft: think:false. For T3/T4/T5
overseers: thinking stays on (that's the point).
- Sidecar generate endpoint accepts `think` bool, passes through to
Ollama's /api/generate.
VERIFIED OUTCOMES
Riverfront Steel 2026-04-21, qwen3.5+continuable+think:false:
08:00 baseline_fill 3/3 4 turns
10:30 recurring 2/2 3 turns (1 playbook citation)
12:15 expansion 0/5 drift-aborted (5-fill orchestration
problem, separate work)
14:00 emergency 4/4 3 turns (1 citation)
15:45 misplacement 1/1 3 turns
→ T3 caught Patrick Ross double-booking across events
→ T3 flagged forklift cert drift on the event that failed
→ Cross-day lesson proposed "maintain buffer of ≥3 emergency
candidates, pre-fetch certs for expansion, booking system
cross-check" — real staffing advice, not generic linter output
PRD PHASE 21 rewritten to reflect the actual primitive shape (two-
call map-reduce with scratchpad glue) instead of the tourniquet
approach originally documented. Rust port queued for next sprint.
scripts/ab_t3_test.sh: A/B harness that chains B→C→D runs and emits
tests/multi-agent/playbooks/ab_scorecard.json.
This commit is contained in:
root
@@ -29,6 +29,7 @@ import {
|
||||
sqlQuery,
|
||||
generate,
|
||||
generateCloud,
|
||||
generateContinuable,
|
||||
parseAction,
|
||||
executorPrompt,
|
||||
reviewerPrompt,
|
||||
@@ -37,15 +38,14 @@ import {
|
||||
import { mkdir, writeFile, appendFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
// 2026-04-20 — reverted to mistral executor after trying qwen2.5.
|
||||
// qwen2.5 emits malformed JSON (trailing `)` garbage, unterminated
|
||||
// strings) when asked for tool calls. mistral drops fields occasionally
|
||||
// but produces valid JSON. With optional `question` default + lean
|
||||
// prompt + schema lock, mistral seals baseline + recurring reliably.
|
||||
// Complex scenarios (5-fill, emergency, misplacement) remain flaky —
|
||||
// real Phase 20+ problem (larger model or constrained decoding needed).
|
||||
const EXECUTOR_MODEL = "mistral:latest";
|
||||
const REVIEWER_MODEL = "qwen2.5:latest";
|
||||
// 2026-04-21 — executor is now qwen3.5:latest (9.7B, 262K context,
|
||||
// thinking model, emits clean JSON). Replaces mistral, which produced
|
||||
// malformed JSON on complex SQL filters (bare IN-clause identifiers,
|
||||
// unclosed braces) regardless of prompt — decoder-level bug that all
|
||||
// 5 events hit across 4 A/B test runs. qwen3.5 tested clean on first
|
||||
// try with 800 max_tokens.
|
||||
const EXECUTOR_MODEL = "qwen3.5:latest";
|
||||
const REVIEWER_MODEL = "qwen3:latest";
|
||||
const DRAFT_MODEL = "qwen2.5:latest"; // artifact generation; short outputs
|
||||
|
||||
// T3 overview tier. Called sparingly — NOT per tool call. Two insertion
|
||||
@@ -81,9 +81,16 @@ const T3_DISABLED = process.env.LH_T3_DISABLE === "1";
|
||||
|
||||
// Dispatcher: route T3 calls to local sidecar or Ollama Cloud depending
|
||||
// on the LH_OVERVIEW_CLOUD flag. Hot-path T1/T2 always stay local.
|
||||
// T3 outputs are free-form prose (lesson/hint), so shape=text — the
|
||||
// continuation primitive treats any non-empty response as complete.
|
||||
async function overviewGenerate(prompt: string, opts: { temperature?: number; max_tokens?: number } = {}): Promise<string> {
|
||||
if (OVERVIEW_CLOUD) return generateCloud(OVERVIEW_MODEL, prompt, opts);
|
||||
return generate(OVERVIEW_MODEL, prompt, opts);
|
||||
return generateContinuable(OVERVIEW_MODEL, prompt, {
|
||||
temperature: opts.temperature,
|
||||
max_tokens: opts.max_tokens ?? 1000,
|
||||
shape: "text",
|
||||
max_continuations: 2,
|
||||
cloud: OVERVIEW_CLOUD,
|
||||
});
|
||||
}
|
||||
|
||||
const MAX_TURNS = 14;
|
||||
@@ -365,10 +372,26 @@ async function runAgentFill(
|
||||
while (turn < MAX_TURNS && !sealed) {
|
||||
turn += 1;
|
||||
|
||||
const execRaw = await generate(
|
||||
// generateContinuable: if the model truncates mid-JSON (thinking
|
||||
// ate the budget, or payload was just long), auto-continue with the
|
||||
// partial as scratchpad until braces balance and JSON parses.
|
||||
// No more "bump max_tokens until it stops truncating" tourniquet.
|
||||
// think:false — executor emits structured JSON, doesn't need hidden
|
||||
// reasoning. Burning ~650 thinking tokens on a 400-token JSON was
|
||||
// exactly the bug we just solved.
|
||||
const execRaw = await generateContinuable(
|
||||
EXECUTOR_MODEL,
|
||||
withExtras(executorPrompt(task, log)),
|
||||
{ temperature: 0.2, max_tokens: 600 },
|
||||
{
|
||||
temperature: 0.2,
|
||||
max_tokens: 800,
|
||||
shape: "json",
|
||||
max_continuations: 3,
|
||||
think: false,
|
||||
on_continuation: (n, len) =>
|
||||
append({ turn, role: "executor", model: EXECUTOR_MODEL, kind: "note",
|
||||
content: { continuation: n, combined_chars: len } }),
|
||||
},
|
||||
);
|
||||
let execAction: Action;
|
||||
try {
|
||||
@@ -419,10 +442,19 @@ async function runAgentFill(
|
||||
}
|
||||
}
|
||||
|
||||
const revRaw = await generate(
|
||||
const revRaw = await generateContinuable(
|
||||
REVIEWER_MODEL,
|
||||
withExtras(reviewerPrompt(task, log)),
|
||||
{ temperature: 0.1, max_tokens: 400 },
|
||||
{
|
||||
temperature: 0.1,
|
||||
max_tokens: 600,
|
||||
shape: "json",
|
||||
max_continuations: 3,
|
||||
think: false,
|
||||
on_continuation: (n, len) =>
|
||||
append({ turn, role: "reviewer", model: REVIEWER_MODEL, kind: "note",
|
||||
content: { continuation: n, combined_chars: len } }),
|
||||
},
|
||||
);
|
||||
let revAction: Action;
|
||||
try {
|
||||
@@ -867,7 +899,10 @@ HINT: <hint>`;
|
||||
|
||||
let text = "";
|
||||
try {
|
||||
text = await overviewGenerate(prompt, { temperature: 0.2, max_tokens: 600 });
|
||||
// overviewGenerate routes through generateContinuable — if thinking
|
||||
// ate the initial budget, it auto-continues rather than requiring
|
||||
// us to guess a safe max_tokens upfront.
|
||||
text = await overviewGenerate(prompt, { temperature: 0.2, max_tokens: 800 });
|
||||
} catch (e) {
|
||||
return {
|
||||
after_event: event.at,
|
||||
|
||||
Reference in New Issue
Block a user