Phase 24 — observer HTTP ingest + scenario outcome streaming

Closes the gap J flagged: observer wraps MCP:3700, scenarios hit
gateway:3100 directly, observer idle at 0 ops across 3600+ cycles.
Now scenarios POST per-event outcomes to observer's new HTTP ingest
on :3800, observer consumes them alongside MCP-wrapped ops, ERROR_
ANALYZER and PLAYBOOK_BUILDER loops see the full picture.

observer.ts:
- Bun.serve() HTTP listener on OBSERVER_PORT (default 3800):
  GET /health    — basic + ring depth
  GET /stats     — total / success / failure / by_source / recent
                   scenario ops digest
  POST /event    — accept scenario outcome, shape it into ObservedOp
                   with source="scenario" + staffer_id + sig_hash +
                   event_kind + role/city/state + rescue flags
- recordExternalOp() — shared ring-buffer insert so the main analyzer
  + playbook builder don't care where the op came from
- ObservedOp extended with provenance fields

persistOp() FIX — old path POSTed to /ingest/file?name=observed_operations
which REPLACES the dataset (flagged in feedback_ingest_replace_semantics.md).
Every op was silently wiping all prior ops. Replaced with append to
data/_observer/ops.jsonl so the historical trace is durable across
analyzer cycles and process restarts.

scenario.ts:
- OBSERVER_URL env (default http://localhost:3800)
- postObserverEvent() helper with 2s AbortSignal.timeout so observer
  being down doesn't block scenario flow
- Per-event POST after ctx.results.push(result), carrying staffer_id,
  sig_hash (via imported computeSignature), event_kind + role + city
  + state + count + rescue_attempted / rescue_succeeded + truncated
  output_summary

VERIFIED:
  curl POST /event → {"accepted":true,"ring_size":1}
  curl GET /stats → {"total":1,"successes":1,"by_source":{"scenario":1},
    "recent_scenario_ops":[{...staffer_id,kind,role}]}

Final v3 demo leaderboard (9 runs per staffer, cumulative 3 batches):
  James (local):   92.9% fill, 36.8 cites, score 0.775 — RANK 1
  Maria (full):    81.0% fill, 26.2 cites, score 0.727
  Sam (basic):     61.9% fill, 28.2 cites, score 0.640
  Alex (minimal):  59.5% fill, 32.2 cites, score 0.631
Honest finding: Alex has MORE citations than Sam despite NO T3 and NO
rescue. Playbook inheritance alone is firing hardest when overseer is
absent. The 59.5% fill rate (up from 0% when qwen2.5 was executor)
proves cloud-exec + playbook inheritance is the floor the architecture
delivers.

Local gpt-oss:20b T3 outperforms cloud gpt-oss:120b T3 by 12pt fill
rate on this workload — cloud overseer paying latency+variance for
no measurable gain, worth flagging in next models.json tune.
This commit is contained in:
root
2026-04-20 23:49:30 -05:00
parent 137aed64fb
commit b95dd86556
2 changed files with 151 additions and 9 deletions
+43 -1
View File
@@ -35,8 +35,27 @@ import {
reviewerPrompt,
GATEWAY,
} from "./agent.ts";
import { indexRun, recommendFor, loadRecommendation, snapshotConfig, type PathwayRecommendation } from "./kb.ts";
import { indexRun, recommendFor, loadRecommendation, snapshotConfig, computeSignature, type PathwayRecommendation } from "./kb.ts";
import { createHash } from "node:crypto";
// Phase 24 — observer HTTP endpoint. Scenarios POST per-event
// outcomes here so the observer's analyzer + playbook builder see
// them. Best-effort: if observer is down, scenario still proceeds.
const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
async function postObserverEvent(payload: Record<string, any>): Promise<void> {
try {
await fetch(`${OBSERVER_URL}/event`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(2000),
});
} catch {
// Observer down — silent skip. The scenario's own logs are still
// the source of truth; observer integration is an augmentation.
}
}
import { mkdir, writeFile, appendFile } from "node:fs/promises";
import { join } from "node:path";
@@ -1608,6 +1627,29 @@ async function main() {
ctx.gap_signals.push({ event: event.at, category: category.trim(), detail: rest.join(":").trim() });
}
// Phase 24 — emit outcome to observer so its ERROR_ANALYZER and
// PLAYBOOK_BUILDER loops see scenario events alongside MCP ops.
// Best-effort; observer being down doesn't affect scenario flow.
postObserverEvent({
timestamp: new Date().toISOString(),
endpoint: "scenario:fill",
success: result.ok,
duration_ms: Math.round((result.duration_secs ?? 0) * 1000),
staffer_id: spec.staffer?.id,
sig_hash: computeSignature(spec),
event_kind: result.event.kind,
role: result.event.role,
city: result.event.city,
state: result.event.state,
count: result.event.count,
error: result.error,
rescue_attempted: !!result.retry_remediation,
rescue_succeeded: !!result.retry_remediation && result.ok,
output_summary: result.ok
? `filled ${result.fills.length}/${result.event.count} in ${result.turns} turns, ${result.playbook_citations?.length ?? 0} cites`
: `FAIL: ${(result.error ?? "unknown").slice(0, 80)}`,
}).catch(() => {});
// Option B — T3 checkpoint after every misplacement, and every N-th event.
const isLast = i === spec.events.length - 1;
const nthHit = T3_CHECKPOINT_EVERY > 0 && ((i + 1) % T3_CHECKPOINT_EVERY === 0);