qwen3.5 executor + continuation primitive + think:false
Three coupled fixes that together turned the Riverfront Steel scenario
from 0/5 (mistral) to 4/5 (qwen3.5) with T3 flagging real staffing
concerns rather than linter advice.
MODEL SWAP
- Executor: mistral → qwen3.5:latest (9.7B, 262K ctx, thinking).
mistral's decoder emitted malformed JSON on complex SQL filters
regardless of prompt; J called it — stop using mistral.
- Reviewer: qwen2.5 → qwen3:latest (40K ctx)
- Applied to scenario.ts, orchestrator.ts, network_proving.ts,
run_e2e_rated.ts
CONTINUATION PRIMITIVE (agent.ts)
- generateContinuable(): empty-response → geometric backoff retry;
truncated-JSON → continue from partial as scratchpad; bounded by
budget cap + max_continuations. No more "bump max_tokens until it
stops truncating" tourniquet.
- generateTreeSplit(): map-reduce for oversized input corpora with
running scratchpad digest, reduce pass for final synthesis.
- Empty text no longer throws — it's a signal to continuable that
thinking ate the budget.
think:false FOR HOT PATH
- qwen3.5 burned ~650 tokens of hidden thinking for trivial JSON
emission. For executor/reviewer/draft: think:false. For T3/T4/T5
overseers: thinking stays on (that's the point).
- Sidecar generate endpoint accepts `think` bool, passes through to
Ollama's /api/generate.
VERIFIED OUTCOMES
Riverfront Steel 2026-04-21, qwen3.5+continuable+think:false:
08:00 baseline_fill 3/3 4 turns
10:30 recurring 2/2 3 turns (1 playbook citation)
12:15 expansion 0/5 drift-aborted (5-fill orchestration
problem, separate work)
14:00 emergency 4/4 3 turns (1 citation)
15:45 misplacement 1/1 3 turns
→ T3 caught Patrick Ross double-booking across events
→ T3 flagged forklift cert drift on the event that failed
→ Cross-day lesson proposed "maintain buffer of ≥3 emergency
candidates, pre-fetch certs for expansion, booking system
cross-check" — real staffing advice, not generic linter output
PRD PHASE 21 rewritten to reflect the actual primitive shape (two-
call map-reduce with scratchpad glue) instead of the tourniquet
approach originally documented. Rust port queued for next sprint.
scripts/ab_t3_test.sh: A/B harness that chains B→C→D runs and emits
tests/multi-agent/playbooks/ab_scorecard.json.
This commit is contained in:
root
+225
-8
@@ -22,6 +22,214 @@ export function estimateTokens(text: string): number {
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// Scratchpad + tree-split continuation
|
||||
// ============================================================
|
||||
//
|
||||
// Core problem: when a prompt OR response would exceed a model's window
|
||||
// — e.g. 200 playbooks pasted in, or a long propose_done payload — we
|
||||
// cannot just raise max_tokens forever. That's the tourniquet approach
|
||||
// J explicitly rejected. The right answer is: split into sub-calls
|
||||
// glued by a persistent scratchpad, so no context is ever lost.
|
||||
//
|
||||
// Two primitives:
|
||||
//
|
||||
// 1. generateContinuable(): for OUTPUT overflow. If the model returns
|
||||
// a structurally-incomplete JSON (unmatched braces, truncated
|
||||
// mid-value), auto-continue with the partial as scratchpad until
|
||||
// the JSON parses or max_continuations is hit.
|
||||
//
|
||||
// 2. generateTreeSplit(): for INPUT overflow. If prompt + system +
|
||||
// max_tokens exceeds window, shard the input at semantic boundaries,
|
||||
// run each shard through the model with a running scratchpad digest,
|
||||
// then run a final reduce pass to combine. This is map-reduce with
|
||||
// glue.
|
||||
//
|
||||
// Both are tier-agnostic — they wrap generate() and generateCloud().
|
||||
|
||||
// Check structural completeness of a response. For non-JSON responses
|
||||
// (lesson text, checkpoint hints) we treat "non-empty" as complete.
|
||||
// For JSON-shaped responses (executor/reviewer actions) we check that
|
||||
// braces match AND that JSON.parse succeeds on the first {...} block.
|
||||
function isStructurallyComplete(text: string, shape: "json" | "text"): boolean {
|
||||
if (!text || !text.trim()) return false;
|
||||
if (shape === "text") return true;
|
||||
// Strip ``` fences
|
||||
let s = text.trim();
|
||||
if (s.startsWith("```")) s = s.replace(/^```(?:json)?\n?/, "").replace(/```$/, "").trim();
|
||||
const start = s.indexOf("{");
|
||||
const end = s.lastIndexOf("}");
|
||||
if (start < 0 || end <= start) return false;
|
||||
const slice = s.slice(start, end + 1);
|
||||
// Balance check — cheaper than JSON.parse and catches truncated nests
|
||||
let depth = 0, inStr = false, esc = false;
|
||||
for (const c of slice) {
|
||||
if (esc) { esc = false; continue; }
|
||||
if (c === "\\") { esc = true; continue; }
|
||||
if (c === '"') { inStr = !inStr; continue; }
|
||||
if (inStr) continue;
|
||||
if (c === "{") depth++;
|
||||
else if (c === "}") depth--;
|
||||
}
|
||||
if (depth !== 0) return false;
|
||||
try { JSON.parse(slice); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
// Continue a truncated response. We do NOT ask the model to start over —
|
||||
// we ask it to continue from exactly where it stopped. The partial goes
|
||||
// in as scratchpad so it knows what's already committed.
|
||||
async function continueResponse(
|
||||
model: string,
|
||||
originalPrompt: string,
|
||||
partial: string,
|
||||
opts: { max_tokens: number; temperature: number; system?: string; cloud: boolean; think?: boolean },
|
||||
): Promise<string> {
|
||||
const continuationPrompt = `${originalPrompt}
|
||||
|
||||
PARTIAL RESPONSE SO FAR (continue from here — do NOT restart, do NOT repeat what's already there, emit ONLY the remaining tokens to close the structure):
|
||||
${partial}`;
|
||||
const fn = opts.cloud ? generateCloud : generate;
|
||||
const rest = await fn(model, continuationPrompt, {
|
||||
max_tokens: opts.max_tokens,
|
||||
temperature: opts.temperature,
|
||||
system: opts.system,
|
||||
bypass_budget: true, // the caller already checked; continuation doesn't double-count
|
||||
think: opts.think,
|
||||
});
|
||||
return partial + rest;
|
||||
}
|
||||
|
||||
// Output-overflow handler. Handles two distinct failure modes:
|
||||
// (a) EMPTY response — thinking model ate the whole budget. Retry the
|
||||
// ORIGINAL prompt with 2x the budget (geometric backoff up to cap).
|
||||
// (b) TRUNCATED non-empty — model got most of the way there but hit
|
||||
// max_tokens before closing. Continue with the partial as
|
||||
// scratchpad.
|
||||
// The two modes need different repair: (a) needs more budget, not
|
||||
// continuation from ""; (b) needs scratchpad-glued continuation.
|
||||
export async function generateContinuable(
|
||||
model: string,
|
||||
prompt: string,
|
||||
opts: {
|
||||
max_tokens?: number;
|
||||
temperature?: number;
|
||||
system?: string;
|
||||
shape?: "json" | "text";
|
||||
max_continuations?: number;
|
||||
cloud?: boolean;
|
||||
think?: boolean;
|
||||
on_continuation?: (n: number, combined_len: number) => void;
|
||||
} = {},
|
||||
): Promise<string> {
|
||||
const shape = opts.shape ?? "json";
|
||||
const initialMax = opts.max_tokens ?? 800;
|
||||
const maxConts = opts.max_continuations ?? 3;
|
||||
const cloud = opts.cloud ?? false;
|
||||
const budgetCap = 8000; // don't geometric-backoff forever
|
||||
const fn = cloud ? generateCloud : generate;
|
||||
|
||||
let combined = "";
|
||||
let currentMax = initialMax;
|
||||
|
||||
// Initial call + empty-response backoff loop.
|
||||
for (let retry = 0; retry < 3; retry++) {
|
||||
const out = await fn(model, prompt, {
|
||||
max_tokens: currentMax,
|
||||
temperature: opts.temperature,
|
||||
system: opts.system,
|
||||
think: opts.think,
|
||||
});
|
||||
if (out.trim().length > 0) { combined = out; break; }
|
||||
// Empty — thinking model ate the budget. Double it and retry.
|
||||
if (opts.on_continuation) opts.on_continuation(retry + 1, 0);
|
||||
currentMax = Math.min(currentMax * 2, budgetCap);
|
||||
}
|
||||
|
||||
// Structural completion loop (continuation from partial).
|
||||
for (let i = 0; i < maxConts; i++) {
|
||||
if (isStructurallyComplete(combined, shape)) return combined;
|
||||
if (opts.on_continuation) opts.on_continuation(i + 1, combined.length);
|
||||
combined = await continueResponse(model, prompt, combined, {
|
||||
max_tokens: Math.min(currentMax, budgetCap),
|
||||
temperature: opts.temperature ?? 0.3,
|
||||
system: opts.system,
|
||||
cloud,
|
||||
think: opts.think,
|
||||
});
|
||||
}
|
||||
// Last-resort: return combined even if incomplete; caller's parser
|
||||
// will throw with the raw text for forensics rather than silently
|
||||
// truncating.
|
||||
return combined;
|
||||
}
|
||||
|
||||
// Input-overflow handler. When the input corpus exceeds the window,
|
||||
// shard → map → reduce with a running scratchpad digest.
|
||||
//
|
||||
// shards: array of input chunks the caller already split at semantic
|
||||
// boundaries (paragraphs, records, playbook entries).
|
||||
// map_prompt: fn taking (shard, running_scratchpad) → prompt for a
|
||||
// single map call. Must fit within window.
|
||||
// reduce_prompt: fn taking (combined_scratchpad) → final prompt. Also
|
||||
// must fit window; if the scratchpad itself overflows, this triggers
|
||||
// a recursive tree-split.
|
||||
//
|
||||
// Result: the reduce call's response.
|
||||
export async function generateTreeSplit(
|
||||
model: string,
|
||||
shards: string[],
|
||||
map_prompt: (shard: string, scratchpad: string) => string,
|
||||
reduce_prompt: (scratchpad: string) => string,
|
||||
opts: {
|
||||
max_tokens?: number;
|
||||
temperature?: number;
|
||||
system?: string;
|
||||
cloud?: boolean;
|
||||
on_shard?: (i: number, total: number) => void;
|
||||
scratchpad_budget?: number;
|
||||
} = {},
|
||||
): Promise<{ response: string; scratchpad: string; shards_processed: number }> {
|
||||
const cloud = opts.cloud ?? false;
|
||||
const scratchpadBudget = opts.scratchpad_budget ?? 6000;
|
||||
let scratchpad = "";
|
||||
|
||||
for (let i = 0; i < shards.length; i++) {
|
||||
if (opts.on_shard) opts.on_shard(i + 1, shards.length);
|
||||
const shardPrompt = map_prompt(shards[i], scratchpad);
|
||||
// If the per-shard prompt alone exceeds window, the caller sharded
|
||||
// too coarsely — bubble up rather than silently truncating.
|
||||
assertContextBudget(model, shardPrompt, {
|
||||
system: opts.system,
|
||||
max_tokens: opts.max_tokens,
|
||||
bypass: false,
|
||||
});
|
||||
const shardOut = await generateContinuable(model, shardPrompt, {
|
||||
max_tokens: opts.max_tokens ?? 800,
|
||||
temperature: opts.temperature,
|
||||
system: opts.system,
|
||||
shape: "text",
|
||||
cloud,
|
||||
});
|
||||
// Append to scratchpad; truncate oldest if over budget.
|
||||
scratchpad = (scratchpad + `\n— shard ${i + 1}/${shards.length} digest —\n` + shardOut.trim()).slice(-scratchpadBudget * 4);
|
||||
}
|
||||
|
||||
const reducePrompt = reduce_prompt(scratchpad);
|
||||
assertContextBudget(model, reducePrompt, {
|
||||
system: opts.system,
|
||||
max_tokens: opts.max_tokens,
|
||||
bypass: false,
|
||||
});
|
||||
const response = await generateContinuable(model, reducePrompt, {
|
||||
max_tokens: opts.max_tokens ?? 1500,
|
||||
temperature: opts.temperature,
|
||||
system: opts.system,
|
||||
shape: "text",
|
||||
cloud,
|
||||
});
|
||||
return { response, scratchpad, shards_processed: shards.length };
|
||||
}
|
||||
|
||||
// Known context windows — matches crates/../config/models.json. Kept in
|
||||
// code as a fallback so the test harness doesn't crash if the config is
|
||||
// missing. Production path should read from models.json.
|
||||
@@ -29,10 +237,14 @@ export const CONTEXT_WINDOWS: Record<string, number> = {
|
||||
"mistral:latest": 32768,
|
||||
"qwen2.5:latest": 32768,
|
||||
"qwen3:latest": 40960,
|
||||
"qwen3.5:latest": 262144, // local 9.7B — new executor
|
||||
"qwen3-embedding": 32768, // local embedding model
|
||||
"nomic-embed-text-v2-moe": 2048, // local embedding, MoE
|
||||
"gpt-oss:20b": 131072,
|
||||
"gpt-oss:120b": 131072,
|
||||
"qwen3.5:397b": 131072,
|
||||
"kimi-k2-thinking": 200000,
|
||||
"kimi-k2.6": 200000, // cloud — new T4 candidate
|
||||
"kimi-k2:1t": 1048576,
|
||||
"deepseek-v3.1:671b": 131072,
|
||||
"glm-4.7": 131072,
|
||||
@@ -89,6 +301,7 @@ export interface LogEntry {
|
||||
| "critique"
|
||||
| "propose_done"
|
||||
| "consensus_done"
|
||||
| "note"
|
||||
| "error";
|
||||
content: any;
|
||||
}
|
||||
@@ -150,6 +363,7 @@ export async function generate(model: string, prompt: string, opts: {
|
||||
temperature?: number;
|
||||
system?: string;
|
||||
bypass_budget?: boolean;
|
||||
think?: boolean;
|
||||
} = {}): Promise<string> {
|
||||
assertContextBudget(model, prompt, {
|
||||
system: opts.system,
|
||||
@@ -163,11 +377,13 @@ export async function generate(model: string, prompt: string, opts: {
|
||||
max_tokens: opts.max_tokens ?? 800,
|
||||
};
|
||||
if (opts.system) body.system = opts.system;
|
||||
if (opts.think !== undefined) body.think = opts.think;
|
||||
const r = await http<any>("POST", `${SIDECAR}/generate`, body);
|
||||
const text = r.text ?? "";
|
||||
if (!text || typeof text !== "string") {
|
||||
throw new Error(`generate returned empty text from ${model}: ${JSON.stringify(r).slice(0, 200)}`);
|
||||
}
|
||||
const text = typeof r.text === "string" ? r.text : "";
|
||||
// Do NOT throw on empty. Thinking models (gpt-oss, qwen3.5) burn the
|
||||
// max_tokens budget on hidden reasoning and emit "" when budget was
|
||||
// too tight. generateContinuable detects empty + continues with more
|
||||
// budget. Callers that expected non-empty can check themselves.
|
||||
return text;
|
||||
}
|
||||
|
||||
@@ -181,6 +397,7 @@ export async function generateCloud(model: string, prompt: string, opts: {
|
||||
temperature?: number;
|
||||
system?: string;
|
||||
bypass_budget?: boolean;
|
||||
think?: boolean;
|
||||
} = {}): Promise<string> {
|
||||
if (!OLLAMA_CLOUD_KEY) {
|
||||
throw new Error("OLLAMA_CLOUD_KEY not set; cannot reach Ollama Cloud");
|
||||
@@ -200,6 +417,7 @@ export async function generateCloud(model: string, prompt: string, opts: {
|
||||
},
|
||||
};
|
||||
if (opts.system) body.system = opts.system;
|
||||
if (opts.think !== undefined) body.think = opts.think;
|
||||
const resp = await fetch(`${OLLAMA_CLOUD_URL}/api/generate`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
@@ -212,10 +430,9 @@ export async function generateCloud(model: string, prompt: string, opts: {
|
||||
throw new Error(`Ollama Cloud ${resp.status}: ${await resp.text().catch(() => "?")}`);
|
||||
}
|
||||
const data: any = await resp.json();
|
||||
const text = data.response ?? "";
|
||||
if (!text) {
|
||||
throw new Error(`Ollama Cloud returned empty response for ${model}: ${JSON.stringify(data).slice(0, 200)}`);
|
||||
}
|
||||
const text = typeof data.response === "string" ? data.response : "";
|
||||
// Same non-throw policy as local generate() — empty text is a valid
|
||||
// signal that thinking ate the budget. Let generateContinuable retry.
|
||||
return text;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user