Cohesion: Python inventory + integration plan + Phase A verdict indexing
lakehouse/auditor all checks passed (3 findings, all info)

Three artifacts in one PR:

1. docs/PYTHON_INVENTORY.md — every .py file in the repo classified:
   Production (sidecar routers + 3 systemd services), Documented
   (kb_measure, kb_staffer_report), Manual (one-off tools), Dead
   (sidecar/sidecar/lab_ui.py + pipeline_lab.py are genuinely
   not imported anywhere).

2. docs/COHESION_INTEGRATION_PLAN.md — the "smarter DB" loop J
   called out as missing. Six phases A-F. Phase A ships here; B-F
   are named + sequenced for follow-up PRs. Each phase adds ONE
   wire of the loop; no single PR does them all.

3. Phase A wire (auditor verdicts → observer + KB):
   - auditor/audit.ts: after assembleVerdict, fire-and-forget POST
     to :3800/event with source="auditor" AND append to
     data/_kb/outcomes.jsonl with kind="audit". Errors log + drop
     — the verdict is still on disk at _auditor/verdicts/.
   - mcp-server/observer.ts: extend source union to include
     "auditor" | "bot" (was "mcp" | "scenario" only, which silently
     coerced my first auditor POST to source="scenario"). Accept
     body.ok OR body.success. Accept body.audit_duration_ms as a
     fallback for duration_ms. Uses body.one_liner as
     output_summary when set.

Live-verified after observer restart:
   re-audit PR #6 → verdict=request_changes, 4 findings (1 warn)
     observer: by_source={'auditor': 1}  (previously coerced to 'scenario')
     _kb/outcomes.jsonl tail: kind=audit sig=pr6-7fe47bab
       pr=6 overall=request_changes

The shape of the loop is now visible to downstream consumers. Phase
B (auditor's kb_query check reads these audit rows for history)
lands in a follow-up PR. Phase C-F similar.

NOT in this PR:
- Actually deleting lab_ui.py + pipeline_lab.py (operator decision,
  called out in the inventory doc)
- Cleaning up the 5 overlapping Python scripts (same)
- Phases B-F of the cohesion plan (separate PRs per wire)
- Integration test that asserts "smarter DB" across runs (Phase F)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
profit
2026-04-22 17:22:42 -05:00
co-authored by Claude Opus 4.7
parent 6d7b251607
commit 6e39d8778f
4 changed files with 298 additions and 6 deletions
+72
View File
@@ -80,6 +80,16 @@ export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<
await persistVerdict(verdict);
// Phase A of the cohesion plan (docs/COHESION_INTEGRATION_PLAN.md):
// make every audit verdict visible to the observer + KB. Enables
// future Phase B (kb_query sees prior audit history) without a
// separate backfill. Fire-and-forget: observer/KB failures don't
// block the Gitea post.
indexVerdictToObserver(verdict).catch(e =>
console.error(`[auditor] observer indexing failed: ${(e as Error).message}`));
appendVerdictToKbOutcomes(verdict).catch(e =>
console.error(`[auditor] kb outcomes append failed: ${(e as Error).message}`));
if (!opts.dry_run) {
await postToGitea(verdict);
}
@@ -87,6 +97,68 @@ export async function auditPr(pr: PrSnapshot, opts: AuditOptions = {}): Promise<
return verdict;
}
// Phase A — verdict indexing.
//
// Two destinations, both append-only + non-blocking:
// 1. observer :3800/event — ring buffer + data/_observer/ops.jsonl
// 2. data/_kb/outcomes.jsonl — same file scenarios write to, with
// kind:"audit" so readers can filter
//
// Errors log + drop. The verdict is still on disk at
// _auditor/verdicts/{pr}-{sha}.json; observer + KB are a convenience
// surface, not a source of truth.
const OBSERVER_URL = process.env.LH_OBSERVER_URL ?? "http://localhost:3800";
const KB_OUTCOMES = "/home/profit/lakehouse/data/_kb/outcomes.jsonl";
async function indexVerdictToObserver(v: Verdict): Promise<void> {
const payload = {
source: "auditor",
event_kind: "audit",
ok: v.overall === "approve",
sig_hash: `pr${v.pr_number}-${v.head_sha.slice(0, 8)}`,
pr_number: v.pr_number,
head_sha: v.head_sha,
overall: v.overall,
one_liner: v.one_liner,
findings_block: v.metrics.findings_block,
findings_warn: v.metrics.findings_warn,
audit_duration_ms: v.metrics.audit_duration_ms,
audited_at: v.audited_at,
};
const r = await fetch(`${OBSERVER_URL}/event`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(3000),
});
if (!r.ok) throw new Error(`observer ${r.status}: ${await r.text()}`);
}
async function appendVerdictToKbOutcomes(v: Verdict): Promise<void> {
const { appendFile, mkdir } = await import("node:fs/promises");
const { dirname } = await import("node:path");
await mkdir(dirname(KB_OUTCOMES), { recursive: true });
const row = {
kind: "audit",
sig_hash: `pr${v.pr_number}-${v.head_sha.slice(0, 8)}`,
audited_at: v.audited_at,
pr_number: v.pr_number,
head_sha: v.head_sha,
overall: v.overall,
one_liner: v.one_liner,
ok_events: v.overall === "approve" ? 1 : 0,
total_events: 1,
findings: {
block: v.metrics.findings_block,
warn: v.metrics.findings_warn,
info: v.metrics.findings_info,
},
elapsed_secs: (v.metrics.audit_duration_ms ?? 0) / 1000,
};
await appendFile(KB_OUTCOMES, JSON.stringify(row) + "\n");
}
async function persistVerdict(v: Verdict): Promise<void> {
await mkdir(VERDICTS_DIR, { recursive: true });
const filename = `${v.pr_number}-${v.head_sha.slice(0, 12)}.json`;