profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Phase 23 — contract terms + staffer identity + competence-weighted retrieval

Browse Source 
Matrix-index the "who handled this" dimension so top staffers become
the training signal and juniors inherit their playbooks automatically
via the boost pipeline. Auto-discovered indicators emerge from
comparing trajectories across staffers on similar contracts — that was
always the architectural point; this wires the last piece.

ContractTerms:
- deadline, budget_total_usd, budget_per_hour_max, local_bonus_per_hour,
  local_bonus_radius_mi, fill_requirement ("paramount" | "preferred")
- Attached to ScenarioSpec, propagated into T3 checkpoint + cloud
  rescue prompts so cloud reasons about trade-offs (pivot within bonus
  radius first; respect per-hour cap; split across cities when
  fill_requirement=paramount).

Staffer:
- {id, name, tenure_months, role: senior|mid|junior|trainee}
- On ScenarioSpec; logged at scenario start; attached to KB outcome
- Recomputed StafferStats written to data/_kb/staffers.jsonl after
  every run: total_runs, fill_rate, avg_turns, avg_citations,
  rescue_rate, competence_score.
- Competence formula: 0.45*fill_rate + 0.20*turn_efficiency +
  0.20*citation_density + 0.15*rescue_rate. Normalized to 0..1.

findNeighbors now returns weighted_score = cosine × best_staffer_competence
(floored at 0.3 so high-similarity low-competence neighbors still
surface). pathway_recommender prompt shows the top staffer's identity
so cloud knows WHOSE playbook it's synthesizing from.

Demo infrastructure:
- tests/multi-agent/gen_staffer_demo.ts: 4 personas (Maria senior,
  James mid, Sam junior, Alex trainee) × 3 contracts (Nashville Welder,
  Joliet Warehouse, Indianapolis Assembly). 12 scenarios total.
- scripts/run_staffer_demo.sh: runs the 12 sequentially with
  LH_OVERVIEW_CLOUD=1. Post-run calls kb_staffer_report.py.
- scripts/kb_staffer_report.py: leaderboard + cross-staffer worker
  overlap (names endorsed by ≥2 staffers → auto-discovered high-value
  workers). Top vs bottom differential.

gen_scenarios.ts (Phase 22 generator) also now emits contract terms
on 70% of generated specs — future KB batches populate with realistic
constraint patterns instead of bare role+city+count.

Stress scenario from item A intentionally NOT the production test.
Real staffing has constraints; Nashville contract + staffer demo is
the honest test of whether the architecture produces measurable
differential between coordinator skill levels.

Demo batch launched — 12 runs × ~3min each ≈ 40min unattended. Report
emitted after batch.
This commit is contained in:
root
2026-04-20 22:16:09 -05:00
parent a7fc8e2256
commit 6b71c8e9b2
41 changed files with 2316 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/multi-agent/scenario.ts
+85 -27
View File
@@ -122,10 +122,40 @@ interface FillEvent {
replaces_event?: string; // misplacement back-ref for reporting
}
// Contract terms describe the business constraints around a scenario:
// how much can the client pay, by when, and how is a local candidate
// bonused vs an out-of-area one. T3 checkpoint + cloud rescue pass
// these into the overseer's prompt so pivots respect budget (not just
// geography). Optional — legacy scenarios without a contract degrade
// to the original "nearest-city" pivot logic.
interface ContractTerms {
deadline: string; // ISO date — "2026-05-05"
budget_total_usd?: number; // optional total cap across the contract
budget_per_hour_max?: number; // per-worker hourly cap
local_bonus_per_hour?: number; // extra $/hr for in-radius candidates
local_bonus_radius_mi?: number; // radius from the contract city
fill_requirement?: "paramount" | "preferred"; // "paramount" = must fill;
// "preferred" = best-effort
}
// Phase 23 — staffer identity. When a scenario is assigned to a
// specific coordinator, we record their trajectory so KB can
// weight future retrieval by competence. Junior staffers inherit
// Senior staffers' playbooks; top staffers become the training
// signal the system auto-discovers.
interface Staffer {
id: string; // "S-001"
name: string; // "Maria Chen"
tenure_months: number;
role: "senior" | "mid" | "junior" | "trainee";
}
interface ScenarioSpec {
client: string;
date: string;
events: FillEvent[];
contract?: ContractTerms;
staffer?: Staffer;
}
interface EventResult {
@@ -167,6 +197,7 @@ interface EventResult {
interface CloudRemediation {
retry: boolean;
new_city?: string;
new_state?: string;
new_role?: string;
new_count?: number;
rationale: string;
@@ -957,10 +988,25 @@ function extractDiagnostics(log: LogEntry[] | undefined): {
return { sql_filters, hybrid_row_counts, sql_errors, drift_reasons };
}
// Render contract terms into a prompt-friendly block. Empty string when
// no contract is attached — keeps legacy scenarios unaffected.
function contractBlock(contract: ContractTerms | undefined): string {
if (!contract) return "";
const parts: string[] = [];
parts.push(`deadline=${contract.deadline}`);
if (contract.fill_requirement) parts.push(`fill=${contract.fill_requirement}`);
if (contract.budget_per_hour_max) parts.push(`budget/hr max=$${contract.budget_per_hour_max}`);
if (contract.budget_total_usd) parts.push(`total budget=$${contract.budget_total_usd}`);
if (contract.local_bonus_per_hour) parts.push(`local bonus=+$${contract.local_bonus_per_hour}/hr`);
if (contract.local_bonus_radius_mi) parts.push(`local radius=${contract.local_bonus_radius_mi}mi`);
return `\nCONTRACT TERMS: ${parts.join(", ")}`;
}
async function runOverviewCheckpoint(
event: FillEvent,
result: EventResult,
prior: EventResult[],
contract?: ContractTerms,
): Promise<OverviewCheckpoint | null> {
if (T3_DISABLED) return null;
const start = Date.now();
@@ -985,13 +1031,13 @@ async function runOverviewCheckpoint(
+ `Gap signals: ${result.gap_signals.join("; ") || "none"}.`;
const prompt = `You are the overview reviewer for a staffing coordinator agent system. A mid-day checkpoint has been triggered.
${contractBlock(contract)}
Recent events (most recent last):
${priorSummary || "(no prior events)"}
${thisOne}${diagBlock}
Your job: emit ONE risk flag (≤8 words) and ONE actionable hint (≤40 words) for the NEXT event. Be concrete: name the role, city, worker class, OR a geographic pivot (e.g. "pivot Gary IN → Chicago IL, 40min drive"). Do not restate what happened. If the failure was zero-supply, your hint MUST propose a specific alternative city or role. Think step by step, then output strictly as:
Your job: emit ONE risk flag (≤8 words) and ONE actionable hint (≤40 words) for the NEXT event. Be concrete: name the role, city, worker class, OR a geographic pivot (e.g. "pivot to {city}, {state} ({distance}mi) — still in bonus radius"). Do not restate what happened. If the failure was zero-supply, your hint MUST propose a specific alternative city or role. If contract has local_bonus_radius_mi, prefer alternates within that radius before proposing farther pivots. Think step by step, then output strictly as:
RISK: <flag>
HINT: <hint>`;
@@ -1036,6 +1082,7 @@ HINT: <hint>`;
async function requestCloudRemediation(
event: FillEvent,
result: EventResult,
contract?: ContractTerms,
): Promise<{ remediation: CloudRemediation; duration_secs: number } | null> {
if (T3_DISABLED) return null;
const start = Date.now();
@@ -1054,6 +1101,7 @@ FAILED EVENT:
outcome: ${result.error ?? "unknown failure"}
turns used: ${result.turns}
pool surfaced: ${result.pool_size ?? "n/a"}
${contractBlock(contract)}
RAW DIAGNOSTICS (what the agent actually saw):
${diagBlock}
@@ -1062,6 +1110,7 @@ Respond with a JSON object (NOTHING else, no prose before or after, no markdown)
{
"retry": true | false,
"new_city": "string (same as original if no pivot)",
"new_state": "XX (two-letter abbreviation)",
"new_role": "string (same as original if no pivot)",
"new_count": <integer (same as original if no pivot)>,
"rationale": "2-3 sentences explaining the fix or why retry is futile"
@@ -1072,7 +1121,9 @@ RULES:
- If the city has genuine zero supply, pivot to the NEAREST alternate city with comparable labor pool (name a specific one).
- If the role is uniquely scarce, either broaden to a synonym role OR reduce count to something achievable.
- If no pivot seems viable, set retry=false — wasting a retry is worse than declaring impossible.
- Keep new_count realistic. Don't propose 5× in a city that clearly only has 2 workers.`;
- Keep new_count realistic. Don't propose 5× in a city that clearly only has 2 workers.
- new_city is the CITY name only (e.g. "Clarksville"). Put the two-letter state in new_state. Don't concat them.
${contract ? `- CONTRACT AWARENESS: fill_requirement=${contract.fill_requirement ?? "preferred"}. ${contract.fill_requirement === "paramount" ? "MUST fill — splitting across more cities is acceptable if one city can't cover headcount." : "Best-effort — declining is OK if budget can't absorb pivot premium."} ${contract.local_bonus_radius_mi ? `Local bonus radius is ${contract.local_bonus_radius_mi}mi from ${event.city}, ${event.state}; prefer pivots WITHIN that radius even if they have slightly less supply, since those candidates earn the local bonus premium.` : ""} ${contract.budget_per_hour_max ? `Per-hour budget max is $${contract.budget_per_hour_max}.` : ""}` : ""}`;
let raw = "";
try {
@@ -1108,7 +1159,7 @@ async function runCrossDayLesson(ctx: ScenarioContext, checkpoints: OverviewChec
const prompt = `You are the end-of-day lesson writer for a staffing coordinator agent system. The day is done. Distill it.
Client: ${ctx.spec.client} Date: ${ctx.spec.date}
Client: ${ctx.spec.client} Date: ${ctx.spec.date}${contractBlock(ctx.spec.contract)}
Events that ran:
${eventDigest}
@@ -1326,6 +1377,13 @@ async function main() {
const checkpoints: OverviewCheckpoint[] = [];
console.log(`▶ scenario: ${spec.client}, ${spec.date}, ${spec.events.length} events`);
if (spec.staffer) {
console.log(`▶ staffer: ${spec.staffer.id} ${spec.staffer.name} (${spec.staffer.role}, ${spec.staffer.tenure_months}mo)`);
}
if (spec.contract) {
const c = spec.contract;
console.log(`▶ contract: deadline=${c.deadline} fill=${c.fill_requirement ?? "preferred"}${c.budget_per_hour_max ? ` budget=$${c.budget_per_hour_max}/hr` : ""}${c.local_bonus_radius_mi ? ` local_radius=${c.local_bonus_radius_mi}mi+$${c.local_bonus_per_hour ?? 0}` : ""}`);
}
console.log(`▶ models: exec=${EXECUTOR_MODEL} review=${REVIEWER_MODEL} overview=${T3_DISABLED ? "disabled" : OVERVIEW_MODEL + (OVERVIEW_CLOUD ? " (cloud)" : "")}`);
console.log(`▶ out: ${out_dir}\n`);
@@ -1363,26 +1421,21 @@ async function main() {
// scenarios.
if (!result.ok && RETRY_ON_FAIL && !T3_DISABLED) {
console.log(` ▶ cloud rescue requested for ${event.at} ${event.kind}…`);
const rescue = await requestCloudRemediation(event, result);
const rescue = await requestCloudRemediation(event, result, spec.contract);
if (rescue && rescue.remediation.retry) {
const r = rescue.remediation;
// Sanitize cloud's fields — model sometimes emits "Hammond, IN"
// as new_city and "IN" as new_state, producing "Hammond, IN, IN"
// downstream. Split on comma and take the first token for city.
const sanitizeCity = (c: string | undefined) => (c ?? "").split(",")[0].trim();
const sanitizeState = (c: string | undefined, stateFromCity: string) => {
const explicit = (c ?? "").trim();
// If explicit state is empty or matches original, try to
// extract from the city string if it had a trailing ", XX".
return explicit || stateFromCity || event.state;
};
const cityRaw = r.new_city ?? event.city;
const cityClean = sanitizeCity(cityRaw);
const stateFromCity = (cityRaw.match(/,\s*([A-Z]{2})/) ?? [])[1] ?? "";
// Sanitize cloud's fields — model sometimes still packs state
// into city ("Hammond, IN") even when prompted to split; guard
// by stripping comma+ from new_city and harvesting the trailing
// state if new_state is missing.
const cityRaw = (r.new_city ?? event.city).trim();
const cityParts = cityRaw.split(",").map(s => s.trim());
const cityClean = cityParts[0] || event.city;
const stateFromCity = cityParts[1]?.match(/^([A-Z]{2})/i)?.[1]?.toUpperCase() ?? "";
const newEvent: FillEvent = {
...event,
city: cityClean || event.city,
state: sanitizeState(undefined, stateFromCity) || event.state,
city: cityClean,
state: (r.new_state?.trim() || stateFromCity || event.state).toUpperCase(),
role: r.new_role ?? event.role,
count: r.new_count ?? event.count,
scenario_note: `[cloud-rescue ${rescue.duration_secs.toFixed(1)}s] ${r.rationale}`,
@@ -1455,7 +1508,7 @@ async function main() {
const nthHit = T3_CHECKPOINT_EVERY > 0 && ((i + 1) % T3_CHECKPOINT_EVERY === 0);
const shouldCheckpoint = !T3_DISABLED && (event.kind === "misplacement" || nthHit || isLast);
if (shouldCheckpoint) {
const cp = await runOverviewCheckpoint(event, result, ctx.results.slice(0, -1));
const cp = await runOverviewCheckpoint(event, result, ctx.results.slice(0, -1), spec.contract);
if (cp) {
checkpoints.push(cp);
await appendFile(join(out_dir, "checkpoints.jsonl"), JSON.stringify(cp) + "\n");
@@ -1532,12 +1585,17 @@ async function main() {
// run ends → KB updates → next run reads rec at startup.
try {
const elapsed = (Date.now() - runStart) / 1000;
const { sig_hash } = await indexRun(out_dir, spec, {
executor: EXECUTOR_MODEL,
reviewer: REVIEWER_MODEL,
overview: OVERVIEW_MODEL,
overview_cloud: OVERVIEW_CLOUD,
}, elapsed);
const { sig_hash } = await indexRun(
out_dir,
{ client: spec.client, date: spec.date, events: spec.events, staffer: spec.staffer },
{
executor: EXECUTOR_MODEL,
reviewer: REVIEWER_MODEL,
overview: OVERVIEW_MODEL,
overview_cloud: OVERVIEW_CLOUD,
},
elapsed,
);
console.log(`▶ KB indexed: sig=${sig_hash} (${elapsed.toFixed(1)}s)`);
const newRec = await recommendFor(spec, {
overview_model: OVERVIEW_MODEL,