Lift k cap, drop ornamental reason field, scenario generator

ITEM 1 — k CAP + REASON FIELD
The hybrid_search default k was hard-coded to 10. For multi-fill events
(5× expansion, 4× emergency) that's pool=10 → propose 5-of-10, half
the candidates become the answer with no room for rejection. Executor
prompt now instructs k to scale with target_count: k = max(count*5, 20),
cap 80. Default helper bumped 10 → 20.

Fill.reason dropped from required to optional. Nothing downstream ever
consumed it — resolveWorkerIds, sealSale, retrospective all use
candidate_id and name. Models loved to write 100-150 char justifications
per fill; on 4+ fills that blew the JSON budget before the structure
closed. Test 1 run result after this change: FIRST EVER 5/5 on the
Riverfront Steel scenario, 13 total turns across 5 events. The event
that failed last run (emergency 4×Loader with truncated reason-field
continuation) now clears in 2 turns.

Progression:
  mistral baseline:                  0/5
  qwen3.5 + continuation + think:false: 4/5
  qwen3.5 + k=20 + no-reason:        5/5 ✓

ITEM 2 — SCENARIO GENERATOR (NOT YET TESTED E2E)
tests/multi-agent/gen_scenarios.ts emits N deterministic ScenarioSpecs
with varied clients (15 companies), cities (20 Midwest cities known
to exist in workers_500k), role mixes (14 industrial staffing roles,
weighted realistic), and event sequences. Each gets a unique sig_hash
so the KB populates with distinct neighbor signatures.

scripts/run_kb_batch.sh runs all generated specs sequentially against
scenario.ts, logs per-scenario outcomes, and reports KB state at the
end. Each run takes ~2-4min; 20-30 scenarios = 1-2hr unattended.

Next: test the generator+batch on a small N (3-5) to verify KB
populates correctly and pathway recommendations start getting neighbor
signal instead of cold-starts. Then item 3 (Rust re-weighting of
hybrid_search by playbook_memory success).
This commit is contained in:
root
2026-04-20 20:31:34 -05:00
parent 9c1400d738
commit 330cb90f99
3 changed files with 246 additions and 5 deletions
+16 -5
View File
@@ -318,7 +318,11 @@ export type Action =
export interface Fill {
candidate_id: string;
name: string;
reason: string;
reason?: string; // optional — the schema used to require it; nothing
// downstream consumed it, and qwen3.5 would emit
// 100-150 char justifications per fill that blew
// the JSON budget on 4+ fills. Kept optional so
// models that still emit it don't break parsing.
}
// --- HTTP helpers (fail-fast) ---
@@ -348,7 +352,10 @@ export async function callTool(tool: string, args: Record<string, any>): Promise
});
}
export async function hybridSearch(sql_filter: string, question: string, k = 10): Promise<any> {
// Default k=20 is a floor, not a ceiling — executor prompt instructs
// models to scale k to 5× target_count (cap 80) so multi-fill events
// get a meaningfully deep pool to rank within.
export async function hybridSearch(sql_filter: string, question: string, k = 20): Promise<any> {
return http("POST", `${GATEWAY}/vectors/hybrid`, { sql_filter, question, k });
}
@@ -448,7 +455,11 @@ Available tools (each takes a JSON "args" object):
{"index_name":"workers_500k_v1",
"sql_filter":"role = 'Forklift Operator' AND city = 'Toledo' AND state = 'OH' AND CAST(availability AS DOUBLE) > 0.5",
"question":"reliable forklift operator Toledo",
"k":10}
"k":40}
→ k should scale with target_count: roughly 5× the number of fills
needed, floor 20, cap 80. For 1-2 fills use k=20. For 5 fills use
k=40. A deep pool lets the ranker discriminate across a larger
candidate set; k=10 was too tight for multi-fill events.
- sql(query: string)
→ Raw read-only SELECT. Use for verification (confirm a worker exists,
@@ -580,8 +591,8 @@ Your next action MUST be a JSON object matching one of these shapes:
— use on turn 1 to outline your approach. Steps must be concrete.
{"kind":"tool_call","tool":"...","args":{...},"rationale":"why"}
— call a tool and see its result next turn.
{"kind":"propose_done","fills":[{"candidate_id":"...","name":"First Last","reason":"why them"}],"rationale":"..."}
— propose you've met the target. fills MUST have EXACTLY ${task.target_count} entries — count twice before emitting.
{"kind":"propose_done","fills":[{"candidate_id":"...","name":"First Last"}],"rationale":"..."}
— propose you've met the target. fills MUST have EXACTLY ${task.target_count} entries — count twice before emitting. Each fill is ONLY {candidate_id, name} — no reason field, no scores, no commentary.
Strategy tip: once "CANDIDATES SURFACED SO FAR" has ≥ ${task.target_count} entries in ${task.target_city}, ${task.target_state} matching ${task.target_role}, verify ONE via the sql tool (to satisfy the reviewer's SQL-verification criterion) and then propose_done with the top ${task.target_count}. Don't keep re-searching.