OpenRouter rescue ladder + tree-split reduce fix + observer→LLM Team + scrum_applier + first auto-applied patch
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "journal event verified live (total_events_created 0→1 after probe)."

## Infrastructure (scrum loop hardening)

crates/gateway/src/v1/openrouter.rs — new OpenRouter provider
  Direct HTTPS to openrouter.ai/api/v1/chat/completions with OpenAI-compatible shape.
  Key resolution: OPENROUTER_API_KEY env → /home/profit/.env → /root/llm_team_config.json
  (shares LLM Team UI's quota). Added after iter 5 hit repeated Ollama Cloud 502s on
  kimi-k2:1t — different provider backbone as rescue rung. Unit tests pin the URL
  stripping and OpenAI wire shape.

crates/gateway/src/v1/mod.rs + main.rs
  Added `"openrouter" | "openrouter_free"` arm to /v1/chat dispatch.
  V1State.openrouter_key loaded at startup via openrouter::resolve_openrouter_key()
  mirroring the Ollama Cloud pattern. Startup log:
    "v1: OpenRouter key loaded — /v1/chat provider=openrouter enabled"

tests/real-world/scrum_master_pipeline.ts
  * 9-rung ladder — kimi-k2:1t → qwen3-coder:480b → deepseek-v3.1:671b →
    mistral-large-3:675b → gpt-oss:120b → qwen3.5:397b → openrouter/gpt-oss-120b:free
    → openrouter/gemma-3-27b-it:free → local qwen3.5:latest.
    Added qwen3-coder:480b as rung 2 after live probes confirmed it rescues
    kimi-k2:1t 502s cleanly (0.9s latency, substantive reviews).
    Dropped devstral-2 (displaced by qwen3-coder); dropped kimi-k2.6 (not available);
    dropped minimax-m2.7 (returned 0 chars / 400 thinking tokens).
    Local fallback promoted qwen3.5:latest per J's direction 2026-04-24.
  * MAX_ATTEMPTS bumped 6 → 9 to accommodate the rescue tier.
  * Tree-split scratchpad fixed — was concatenating shard markers directly
    into the reviewer input, causing kimi-k2:1t to write titles like
    "Forensic Audit Report – file.rs (shard 3)". Now uses internal §N§
    markers during accumulation and runs a proper reduce step that
    collapses per-shard digests into ONE coherent file-level synthesis
    with markers stripped. Matches the Phase 21 aibridge::tree_split
    map→reduce design. Fallback to stripped scratchpad if reducer returns thin.

tests/real-world/scrum_applier.ts — NEW (737 lines)
  The auto-apply pipeline. Reads scrum_reviews.jsonl, filters rows where
  gradient_tier ∈ {auto, dry_run} AND confidence_avg ≥ MIN_CONF (default 90),
  asks the reviewer model for concrete old_string/new_string patch JSON,
  applies via text replacement, runs cargo check after each file, commits
  if green and reverts if red. Deny-list: /etc/, config/, ops/, auditor/,
  docs/, data/, mcp-server/, ui/, sidecar/, scripts/. Hard caps: per-patch
  confidence ≥ MIN_CONF, old_string must be exactly unique, max 20 lines per
  patch. Never runs on main without explicit LH_APPLIER_BRANCH override.
  Audit trail in data/_kb/auto_apply.jsonl.

  Empirical behavior (dry-run over iter 4 reviews):
    5 eligible files → 1 green commit-ready, 2 build-red reverts, 2 all-rejected
  The build-green gate caught 2 bad patches before they'd have merged.

mcp-server/observer.ts — LLM Team code_review escalation
  When a sig_hash accumulates ≥3 failures (ESCALATION_THRESHOLD), fire-and-forget
  POST /api/run?mode=code_review at localhost:5000 with the failure cluster context.
  Parses facts/entities/relationships/file_hints from the response. Writes to a
  new data/_kb/observer_escalations.jsonl surface. Answers J's vision of the
  observer triggering richer LLM Team calls when failures pile up.
  Non-blocking: runs parallel to existing qwen2.5 analyzer, never replaces it.
  Tracks escalated sig_hashes in a session-local Set to avoid re-hammering
  LLM Team when a cluster persists across observer cycles.

crates/aibridge/src/context.rs
  First auto-applied patch produced by scrum_applier.ts (dry-run path —
  applier writes files in dry-run mode but doesn't commit; bug noted for
  iter 6 fix). Adds #[deprecated] annotation to the inline estimate_tokens
  helper pointing callers to the centralized shared::model_matrix::ModelMatrix
  entry point (P21-002 — duplicate token-estimator surfaces). Cargo check
  passes with the annotation (verified by applier's own build gate).

## Visual Control Plane (UI)

ui/server.ts — Bun.serve on :3950 with /data/* fan-out:
  /data/services, /data/reviews, /data/metrics, /data/trust, /data/overrides,
  /data/findings, /data/outcomes, /data/audit_facts, /data/file/:path,
  /data/refactor_signals, /data/search?q=, /data/signal_classes,
  /data/logs/:svc (journalctl tail per systemd unit), /data/scrum_log.
  Bug fix: tryFetch always attempts JSON.parse before falling back to text
  — observer's Bun.serve returns JSON without application/json content-type,
  which was displaying stats as a raw string ("0 ops" on map) before.

ui/index.html + ui.css — dark neo-brutalist shell. 6 views:
  MAP (D3 force-graph + overlays) / TRACE (per-file iter history) /
  TRAJECTORY (signal-class cards + refactor-signals table + reverse-index
  search box) / METRICS (every card has SOURCE + GOOD lines explaining
  where the number comes from and what target trajectory means) /
  KB (card grid with tooltips on every field) / CONSOLE (per-service
  journalctl tabs).

ui/ui.js — polling client, D3 wiring, signal-class panel, refactor-signals
  table, reverse-index search, per-service console tabs. Bug fix:
  renderNodeContext had Object.entries() iterating string characters when
  /health returned a plain string — now guards with typeof check so
  "lakehouse ok" renders as one row instead of "0 l / 1 a / 2 k / ...".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-24 03:45:35 -05:00
co-authored by Claude Opus 4.7
parent 39a2856851
commit 8b77d67c9c
10 changed files with 937 additions and 30 deletions
+90
View File
@@ -103,6 +103,93 @@ const REFACTOR_PHRASES = [
"split this file", "too large",
];
// Signal-class classifier — per file, given 2+ consecutive iterations'
// reviews, tag the file's behavior:
// CONVERGING — resolved > novel, score ↑
// LOOPING — 3+ same findings repeat, novel = 0, score flat
// ORBITING — novel findings each iter, no resolved (healthy depth)
// PLATEAU — score flat + findings flat (diminishing returns)
// MIXED — partial/unclear
// This is the foundation for iter-6+ auto-routing: each class gets a
// different sub-pipeline (specialist model, reviewer rotation, etc).
const SIGNAL_PHRASES = [
"pseudocode", "placeholder", "stub", "unwired", "missing", "dead code", "orphaned",
"duplicate", "redundant", "refactor", "rewrite", "remove", "unused", "unnecessary",
];
async function signalClasses(): Promise<any> {
const runsDir = `${REPO}/tests/real-world/runs`;
// Load every review, group by file, sort by timestamp
const perFile: Record<string, Array<{run: string, phrases: Set<string>, score: number | null, conf_avg: number | null, findings: number, ts: number}>> = {};
try {
const dirs = await Array.fromAsync(new Bun.Glob("scrum_*").scan({ cwd: runsDir, onlyFiles: false }));
for (const d of dirs) {
const files = await Array.fromAsync(new Bun.Glob("review_*.json").scan({ cwd: `${runsDir}/${d}` }));
for (const f of files) {
try {
const p = `${runsDir}/${d}/${f}`;
const j = JSON.parse(await Bun.file(p).text());
const key = j.file?.replace("/home/profit/lakehouse/", "") ?? "?";
const sug = (j.suggestions ?? "").toLowerCase();
const phrases = new Set<string>();
for (const ph of SIGNAL_PHRASES) if (sug.includes(ph)) phrases.add(ph);
const scoreMatch = sug.match(/(\d(?:\.\d)?)\s*\/\s*10\b/);
const score = scoreMatch ? parseFloat(scoreMatch[1]) : null;
const mconf = [...sug.matchAll(/(?:confidence[*:\s]*\s*|\|\s*)(\d{1,3})\s*%/gi)].map(m=>parseInt(m[1],10));
const jconf = [...sug.matchAll(/"confidence"\s*:\s*(\d{1,3})(?!\d)/gi)].map(m=>parseInt(m[1],10));
const all = [...mconf, ...jconf].filter(x => 0 <= x && x <= 100);
const conf_avg = all.length ? Math.round(all.reduce((a,b)=>a+b,0)/all.length) : null;
const ts = (await Bun.file(p).stat()).mtime.getTime();
(perFile[key] ??= []).push({ run: d, phrases, score, conf_avg, findings: all.length, ts });
} catch {}
}
}
} catch (e) {
return { error: String(e), classes: {} };
}
const classes: Record<string, any> = {};
for (const [file, runs] of Object.entries(perFile)) {
runs.sort((a, b) => a.ts - b.ts);
if (runs.length < 2) { classes[file] = { cls: "NEW", runs: runs.length }; continue; }
const last = runs[runs.length - 1];
const prev = runs[runs.length - 2];
const novel = [...last.phrases].filter(p => !prev.phrases.has(p));
const resolved = [...prev.phrases].filter(p => !last.phrases.has(p));
const looping = [...prev.phrases].filter(p => last.phrases.has(p));
const dScore = (last.score != null && prev.score != null) ? last.score - prev.score : null;
const dConf = (last.conf_avg != null && prev.conf_avg != null) ? last.conf_avg - prev.conf_avg : null;
const dFindings = last.findings - prev.findings;
let cls: string;
if (dScore != null && dScore > 0 && resolved.length > novel.length) cls = "CONVERGING";
else if (looping.length >= 3 && novel.length === 0 && (dScore == null || Math.abs(dScore) < 0.5)) cls = "LOOPING";
else if (novel.length >= 2 && resolved.length === 0) cls = "ORBITING";
else if (Math.abs(dFindings) <= 1 && (dScore == null || Math.abs(dScore) < 0.5)) cls = "PLATEAU";
else cls = "MIXED";
classes[file] = {
cls,
runs: runs.length,
iter_span: `${runs[0].run}…${last.run}`,
prev_score: prev.score,
last_score: last.score,
delta_score: dScore,
delta_conf: dConf,
delta_findings: dFindings,
novel,
resolved,
looping,
};
}
// Summary counts
const counts: Record<string, number> = {};
for (const v of Object.values(classes)) counts[v.cls] = (counts[v.cls] ?? 0) + 1;
return { generated_at: new Date().toISOString(), counts, classes };
}
async function refactorSignals(): Promise<any> {
// Walk every accepted review across all scrum runs. For each file,
// count how many times its suggestions mention a refactor phrase.
@@ -269,6 +356,9 @@ Bun.serve({
if (path === "/data/refactor_signals") {
return Response.json(await refactorSignals());
}
if (path === "/data/signal_classes") {
return Response.json(await signalClasses());
}
if (path === "/data/search") {
const q = url.searchParams.get("q") ?? "";
return Response.json(await reverseIndex(q, 30));