Phase 24 — observer HTTP ingest + scenario outcome streaming
Closes the gap J flagged: observer wraps MCP:3700, scenarios hit
gateway:3100 directly, observer idle at 0 ops across 3600+ cycles.
Now scenarios POST per-event outcomes to observer's new HTTP ingest
on :3800, observer consumes them alongside MCP-wrapped ops, ERROR_
ANALYZER and PLAYBOOK_BUILDER loops see the full picture.
observer.ts:
- Bun.serve() HTTP listener on OBSERVER_PORT (default 3800):
GET /health — basic + ring depth
GET /stats — total / success / failure / by_source / recent
scenario ops digest
POST /event — accept scenario outcome, shape it into ObservedOp
with source="scenario" + staffer_id + sig_hash +
event_kind + role/city/state + rescue flags
- recordExternalOp() — shared ring-buffer insert so the main analyzer
+ playbook builder don't care where the op came from
- ObservedOp extended with provenance fields
persistOp() FIX — old path POSTed to /ingest/file?name=observed_operations
which REPLACES the dataset (flagged in feedback_ingest_replace_semantics.md).
Every op was silently wiping all prior ops. Replaced with append to
data/_observer/ops.jsonl so the historical trace is durable across
analyzer cycles and process restarts.
scenario.ts:
- OBSERVER_URL env (default http://localhost:3800)
- postObserverEvent() helper with 2s AbortSignal.timeout so observer
being down doesn't block scenario flow
- Per-event POST after ctx.results.push(result), carrying staffer_id,
sig_hash (via imported computeSignature), event_kind + role + city
+ state + count + rescue_attempted / rescue_succeeded + truncated
output_summary
VERIFIED:
curl POST /event → {"accepted":true,"ring_size":1}
curl GET /stats → {"total":1,"successes":1,"by_source":{"scenario":1},
"recent_scenario_ops":[{...staffer_id,kind,role}]}
Final v3 demo leaderboard (9 runs per staffer, cumulative 3 batches):
James (local): 92.9% fill, 36.8 cites, score 0.775 — RANK 1
Maria (full): 81.0% fill, 26.2 cites, score 0.727
Sam (basic): 61.9% fill, 28.2 cites, score 0.640
Alex (minimal): 59.5% fill, 32.2 cites, score 0.631
Honest finding: Alex has MORE citations than Sam despite NO T3 and NO
rescue. Playbook inheritance alone is firing hardest when overseer is
absent. The 59.5% fill rate (up from 0% when qwen2.5 was executor)
proves cloud-exec + playbook inheritance is the floor the architecture
delivers.
Local gpt-oss:20b T3 outperforms cloud gpt-oss:120b T3 by 12pt fill
rate on this workload — cloud overseer paying latency+variance for
no measurable gain, worth flagging in next models.json tune.
This commit is contained in:
@@ -35,8 +35,27 @@ import {
|
||||
reviewerPrompt,
|
||||
GATEWAY,
|
||||
} from "./agent.ts";
|
||||
import { indexRun, recommendFor, loadRecommendation, snapshotConfig, type PathwayRecommendation } from "./kb.ts";
|
||||
import { indexRun, recommendFor, loadRecommendation, snapshotConfig, computeSignature, type PathwayRecommendation } from "./kb.ts";
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
// Phase 24 — observer HTTP endpoint. Scenarios POST per-event
|
||||
// outcomes here so the observer's analyzer + playbook builder see
|
||||
// them. Best-effort: if observer is down, scenario still proceeds.
|
||||
const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
|
||||
|
||||
async function postObserverEvent(payload: Record<string, any>): Promise<void> {
|
||||
try {
|
||||
await fetch(`${OBSERVER_URL}/event`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify(payload),
|
||||
signal: AbortSignal.timeout(2000),
|
||||
});
|
||||
} catch {
|
||||
// Observer down — silent skip. The scenario's own logs are still
|
||||
// the source of truth; observer integration is an augmentation.
|
||||
}
|
||||
}
|
||||
import { mkdir, writeFile, appendFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -1608,6 +1627,29 @@ async function main() {
|
||||
ctx.gap_signals.push({ event: event.at, category: category.trim(), detail: rest.join(":").trim() });
|
||||
}
|
||||
|
||||
// Phase 24 — emit outcome to observer so its ERROR_ANALYZER and
|
||||
// PLAYBOOK_BUILDER loops see scenario events alongside MCP ops.
|
||||
// Best-effort; observer being down doesn't affect scenario flow.
|
||||
postObserverEvent({
|
||||
timestamp: new Date().toISOString(),
|
||||
endpoint: "scenario:fill",
|
||||
success: result.ok,
|
||||
duration_ms: Math.round((result.duration_secs ?? 0) * 1000),
|
||||
staffer_id: spec.staffer?.id,
|
||||
sig_hash: computeSignature(spec),
|
||||
event_kind: result.event.kind,
|
||||
role: result.event.role,
|
||||
city: result.event.city,
|
||||
state: result.event.state,
|
||||
count: result.event.count,
|
||||
error: result.error,
|
||||
rescue_attempted: !!result.retry_remediation,
|
||||
rescue_succeeded: !!result.retry_remediation && result.ok,
|
||||
output_summary: result.ok
|
||||
? `filled ${result.fills.length}/${result.event.count} in ${result.turns} turns, ${result.playbook_citations?.length ?? 0} cites`
|
||||
: `FAIL: ${(result.error ?? "unknown").slice(0, 80)}`,
|
||||
}).catch(() => {});
|
||||
|
||||
// Option B — T3 checkpoint after every misplacement, and every N-th event.
|
||||
const isLast = i === spec.events.length - 1;
|
||||
const nthHit = T3_CHECKPOINT_EVERY > 0 && ((i + 1) % T3_CHECKPOINT_EVERY === 0);
|
||||
|
||||
Reference in New Issue
Block a user