Implement real supervisor-driven auto-recovery

Orchestrator changes:
- Force-spawn GAMMA on iteration_limit before abort
- GAMMA.synthesize() creates emergency handoff payload
- loadRecoveryContext() logs "Resuming from {task_id} handoff"
- POST to /api/pipeline/log for resume message visibility

AgentGamma changes:
- Add synthesize() method for emergency abort synthesis
- Merges existing proposals into coherent handoff
- Stores as synthesis_type: "abort_recovery"

Server changes:
- Add POST /api/pipeline/log endpoint for orchestrator logging
- Recovery pipeline properly inherits GAMMA synthesis

Test coverage:
- test_auto_recovery.py: 6 unit tests
- test_e2e_auto_recovery.py: 5 E2E tests
- test_supervisor_recovery.py: 3 supervisor tests
  - Success on attempt 2 (recovery works)
  - Max failures (3 retries then FAILED)
  - Success on attempt 1 (no recovery needed)

Recovery flow:
1. iteration_limit triggers
2. GAMMA force-spawned for emergency synthesis
3. Handoff dumped with GAMMA synthesis
4. Exit code 3 triggers auto-recovery
5. Recovery pipeline loads handoff
6. Logs "Resuming from {prior_pipeline} handoff"
7. Repeat up to 3 times or until success

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
profit
2026-01-24 19:47:56 -05:00
co-authored by Claude Opus 4.5
parent c96919fe35
commit ef18567674
4 changed files with 617 additions and 0 deletions
+10
View File
@@ -7946,6 +7946,16 @@ const server = Bun.serve({
return new Response(JSON.stringify(logs), { headers });
}
// POST /api/pipeline/log - Add a log entry (for orchestrator)
if (path === "/api/pipeline/log" && req.method === "POST") {
const body = await req.json() as { pipeline_id: string; agent: string; message: string; level?: string };
if (!body.pipeline_id || !body.message) {
return new Response(JSON.stringify({ error: "pipeline_id and message required" }), { status: 400, headers });
}
await appendPipelineLog(body.pipeline_id, body.agent || "SYSTEM", body.message, body.level || "INFO");
return new Response(JSON.stringify({ success: true }), { headers });
}
// Vault Token Management APIs
if (path === "/api/pipeline/token") {
const pipelineId = url.searchParams.get("pipeline_id");